From greg.ewing at canterbury.ac.nz Sun Mar 1 00:18:41 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 01 Mar 2009 12:18:41 +1300 Subject: [Cython] strict aliasing problem with subclass method In-Reply-To: References: <49A814C1.5050302@behnel.de> <49A83D94.8050004@behnel.de> <49A88B29.3090506@canterbury.ac.nz> Message-ID: <49A9C651.4010708@canterbury.ac.nz> Dag Sverre Seljebotn wrote: > There are certain things the > compiler simply cannot do automatically for you; FORTRAN "solves" this > problem by being a very restricted language, while C has taken a different > route and tries to keep the flexibility while still going in a direction > which makes it as efficient. Well, I think it's a mistake to try to turn C into a language that's all things to all people. Those with Fortran-envy should either use Fortran or design a new language that gives them what they want -- not stuff C up for the rest of us. -- Greg From robertwb at math.washington.edu Sun Mar 1 13:25:06 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 1 Mar 2009 04:25:06 -0800 Subject: [Cython] Final fix for #143 before a release? In-Reply-To: <49A8413B.4030204@behnel.de> References: <49A8413B.4030204@behnel.de> Message-ID: On Feb 27, 2009, at 11:38 AM, Stefan Behnel wrote: > Hi, > > the last remaining ticket for 0.11 is #143. I think the right fix > for this > release is to disable the typing of locals in .pxd files, and to > change > #143 into a feature request. Fixed. The site is temporarily down, so I can't push right now. - Robert From wstein at gmail.com Sun Mar 1 16:13:14 2009 From: wstein at gmail.com (William Stein) Date: Sun, 1 Mar 2009 10:13:14 -0500 Subject: [Cython] [sage-support] Re: Small groups library is missing In-Reply-To: References: <9405b309-c562-4e73-a7ea-9503cb4331e8@s20g2000yqh.googlegroups.com> <8cf963450902270337g12ae7ff8l36a0b95331c98798@mail.gmail.com> <79e81e94-f8e5-4e70-a6cb-594152aa7433@w9g2000yqa.googlegroups.com> Message-ID: <85e81ba30903010713v1c7239fft20d0485305a4b32a@mail.gmail.com> > sagemath and trac are down now anyway, so I'm not sure if this can be > fixed today. > I can start working on it after I finish grading an exam tomorrow and > Almost definitely will be free to work on it a large chunk of the day > Tuesday. > They (=sagemath, cython, etc.) are all back up now. Apache used up all memory on sagemath for unknown reasons... and we're now logging all output of top and have adjusted a bunch of parameters for apache2, so we'll see what happens. William From hoytak at cs.ubc.ca Sun Mar 1 18:51:09 2009 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Sun, 1 Mar 2009 09:51:09 -0800 Subject: [Cython] PyBool_CheckExact not found Message-ID: <4db580fd0903010951r1ef5b9d6gfd322383cabfcc95@mail.gmail.com> Hello, I'm using the latest pull from cython-dev, 1800:e8b5a8e5da3d, with python 2.5.4, and some of my code calls PyBool_CheckExact. However, this doesn't seem to be defined anywhere. When I compile it with gcc, it warns that the symbol is defined implicitly, and trying to load it results in an import error saying it's an undefined symbol. I'll try to get a specific example together where it happens and file a bug report, but it seems like a simple-to-fix version issue here. --Hoyt -- ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From hoytak at cs.ubc.ca Sun Mar 1 19:26:33 2009 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Sun, 1 Mar 2009 10:26:33 -0800 Subject: [Cython] PyBool_CheckExact not found In-Reply-To: <4db580fd0903010951r1ef5b9d6gfd322383cabfcc95@mail.gmail.com> References: <4db580fd0903010951r1ef5b9d6gfd322383cabfcc95@mail.gmail.com> Message-ID: <4db580fd0903011026p20d09b8fo20be756f4b20dd77@mail.gmail.com> It's now Ticket #227 (http://trac.cython.org/cython_trac/ticket/227). The quick fix is to not use bool, but use bint instead (of course). I had accidentally typed bool instead of bint in one place in my code, and fixing that got my code working. Still, it's a bug, just more of a corner case. --Hoyt On Sun, Mar 1, 2009 at 9:51 AM, Hoyt Koepke wrote: > Hello, > > I'm using the latest pull from cython-dev, 1800:e8b5a8e5da3d, with > python 2.5.4, and some of my code calls PyBool_CheckExact. ?However, > this doesn't seem to be defined anywhere. ?When I compile it with gcc, > it warns that the symbol is defined implicitly, and trying to load it > results in an import error saying it's an undefined symbol. > > I'll try to get a specific example together where it happens and file > a bug report, but it seems like a simple-to-fix version issue here. > > --Hoyt > > > -- > > ++++++++++++++++++++++++++++++++++++++++++++++++ > + Hoyt Koepke > + University of Washington Department of Statistics > + http://www.stat.washington.edu/~hoytak/ > + hoytak at gmail.com > ++++++++++++++++++++++++++++++++++++++++++ > -- ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From stefan_ml at behnel.de Sun Mar 1 21:29:18 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 01 Mar 2009 21:29:18 +0100 Subject: [Cython] PATCH: use function pointer arguments in import/export api functions In-Reply-To: <49A8595B.8040505@behnel.de> References: <49A8595B.8040505@behnel.de> Message-ID: <49AAF01E.6040005@behnel.de> Stefan Behnel wrote: > Lisandro Dalcin wrote: >> Just to prevent this: >> warning: ISO C forbids conversion of function pointer to object pointer type > > with this patch applied, I now get a warning on this line: > > p = *(void **)&f; > > src/lxml/lxml.etree.c: In function '__Pyx_ExportFunction': > src/lxml/lxml.etree.c:148416: warning: dereferencing type-punned pointer > will break strict-aliasing rules > > This needs fixing, I guess. That said, I'm more in favour of enabling clean > strict-aliasing than to be strictly C89 compliant, unless we can have both. :) I reverted the patch for now. Lisandro, if you can come up with a fix that does not break strict aliasing (or a partial patch that only adds consts and the like), please give it another try. Stefan From stefan_ml at behnel.de Sun Mar 1 21:34:32 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 01 Mar 2009 21:34:32 +0100 Subject: [Cython] PyBool_CheckExact not found In-Reply-To: <4db580fd0903011026p20d09b8fo20be756f4b20dd77@mail.gmail.com> References: <4db580fd0903010951r1ef5b9d6gfd322383cabfcc95@mail.gmail.com> <4db580fd0903011026p20d09b8fo20be756f4b20dd77@mail.gmail.com> Message-ID: <49AAF158.7020505@behnel.de> Hoyt Koepke wrote: > It's now Ticket #227 (http://trac.cython.org/cython_trac/ticket/227). Fixed. Stefan From zachary.pincus at yale.edu Sun Mar 1 21:36:42 2009 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Sun, 1 Mar 2009 15:36:42 -0500 Subject: [Cython] cython / numpy / mac(?) issue Message-ID: <074811C8-98C4-44FC-A6FF-DAE6DA57DE5C@yale.edu> Hello all, I've run into a problem using some cython+numpy code that some folks recently made available. After some back and forth with the original authors, we've determined that the problem appears to be with cython on my box. I've attached the offending code and will post a test-case below, but first some configuration details: I'm using OS X 10.5.6 and a python 2.5.2 that's installed as a "framework" (the standard OS X way of installing Python) -- the only major difference is that the python directories live in /Library/ Frameworks/[etc.] instead of /usr/local. First, I should note that on my system, using both the latest stable Cython release as well as the tip of cython-devel from hg, cython's numpy tests fail utterly, due to some sort of error with include paths: hg clone http://hg.cython.org/cython-devel/ cd cython-devel make local make test [...output...] compiling (c) and running numpy_test ... numpy_test.c: 129:31:numpy_test.c:129:31: error: error: numpy/arrayobject.h: No such file or directorynumpy/arrayobject.h: No such file or directory The error using cython 0.10.3 is pretty much the same, except with the spacing errors fixed: compiling (c) and running numpy_test ... numpy_test.c:112:31: error: numpy/arrayobject.h: No such file or directory Also, I should point out that the default OS X filesystem is case- insensitive, so using a directory named 'BUILD' for building the tests is not a great idea, as it is the same as the python 'build' directory, so the latter gets nuked by the 'rm -rf BUILD' in 'make test'. Anyhow, if I install cython despite these issues, I can build numpy- using projects fine... for some reason, the right include directories get pulled in. However, thereafter I have further issues. I've attached the troublesome code; it's pretty simple but I don't really know enough cython yet to pull it into a minimal test case. (I suspect from the nature of the errors that this should be easy.) Anyhow, here's the test case and error: cd stefanv-bilateral python setup.py build_ext -i python Python 2.5.2 (r252:60911, Feb 22 2008, 07:57:53) [GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> print numpy.version.version 1.3.0.dev6359 >>> import Cython.Compiler.Version >>> print Cython.Compiler.Version.version 0.11.beta >>> import bilateral >>> bilateral.bilateral(numpy.zeros((4,4)), 4, 4) Traceback (most recent call last): File "", line 1, in File "bilateral/bilateral.py", line 38, in bilateral filter_fcn = _BB.Bilat_fcn(xy_sig, z_sig, filter_size) File "bilateral_base.pyx", line 40, in bilateral.bilateral_base.Bilat_fcn.__init__ (bilateral/ bilateral_base.c:752) NameError: np Now, bilateral_base.pyx has the line 'cimport numpy as np' at the top, but somehow a NameError gets thrown the first time that 'np' is used, at the end of this snippet: def __init__(self, spat_sig, inten_sig, filter_size=None): if filter_size is not None and filter_size >= 2: self.xy_size = int(filter_size) else: self.xy_size = int(round(spat_sig*4)) # Make filter size odd self.xy_size += 1-self.xy_size%2 x = np.arange(self.xy_size, dtype=float) Just for fun, I tried adding 'import numpy as np' to bilateral_base.pyx after the cimport line (this is what numpy_text.pyx does in the cython tests). Here then is the error that I get: >>> bilateral.bilateral(numpy.zeros((4,4)), 4, 4) Traceback (most recent call last): File "", line 1, in File "bilateral/bilateral.py", line 40, in bilateral return generic_filter(mat, filter_fcn.cfilter, size=size, mode=mode) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/scipy/ndimage/filters.py", line 992, in generic_filter cval, origins, extra_arguments, extra_keywords) File "bilateral_base.pyx", line 73, in bilateral.bilateral_base.Bilat_fcn.cfilter (bilateral/bilateral_base.c: 1443) AttributeError: 'numpy.ndarray' object has no attribute 'dimensions' This corresponds to the last line in this snippet: def cfilter(self, np.ndarray data): 'An optimized implementation' cdef np.ndarray kernel = self.xy_ker cdef double sigma = self.inten_sig cdef double weight_i, weight, result, centre, dat_i cdef double *pdata=data.data, *pker=kernel.data cdef int i, dim = data.dimensions[0] Now, numpy arrays don't expose a dimensions attribute in python, but the array struct has such a member, which I suppose should be exposed to cython via 'cimport numpy'. So, clearly something is going wrong with cython's numpy support on my system -- the tests fail, as well as actual use of numpy, but the failures are different. The authors of this code can't reproduce this with the same versions of cython and numpy on their linux systems, so I suspect that this might be a mac issue, perhaps relating to some assumptions about where the numpy headers live? On a framework install, Python.h lives in: /Library/Frameworks/Python.framework/Versions/2.5/include but numpy/arrayobject.h is in: /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site- packages/numpy/core/include Can anyone offer any suggestions here? Thanks, Zach Pincus Postdoctoral Fellow, lab of Frank Slack Molecular, Cellular and Developmental Biology Yale University -------------- next part -------------- A non-text attachment was scrubbed... Name: stefanv-bilateral.zip Type: application/zip Size: 3225 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090301/ced4ab19/attachment-0001.zip From dagss at student.matnat.uio.no Sun Mar 1 22:13:00 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 01 Mar 2009 22:13:00 +0100 Subject: [Cython] cython / numpy / mac(?) issue Message-ID: <3318790436.424141@smtp.netcom.no> All of these are expected, I don't see any issues here (except perhaps lack of documentation; feel free to submit improvements). 1) You should set the INCLUDE environment variable prior to running runtests.py. 2) cimport and import are different things; you need both. 3) The "dimensions" field is in Cython renamed "shape" to be closer to the Python interface. This is done in Cython/Includes/numpy.pxd Dag Sverre Seljebotn -----Original Message----- From: Zachary Pincus Date: Sunday, Mar 1, 2009 9:35 pm Subject: [Cython] cython / numpy / mac(?) issue To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net Hello all, > >I've run into a problem using some cython+numpy code that some folks >recently made available. After some back and forth with the original >authors, we've determined that the problem appears to be with cython >on my box. I've attached the offending code and will post a test-case >below, but first some configuration details: > >I'm using OS X 10.5.6 and a python 2.5.2 that's installed as a >"framework" (the standard OS X way of installing Python) -- the only >major difference is that the python directories live in /Library/ >Frameworks/[etc.] instead of /usr/local. > >First, I should note that on my system, using both the latest stable >Cython release as well as the tip of cython-devel from hg, cython's >numpy tests fail utterly, due to some sort of error with include paths: hg clone http://hg.cython.org/cython-devel/ >cd cython-devel >make local >make test >[...output...] >compiling (c) and running numpy_test ... numpy_test.c: >129:31:numpy_test.c:129:31: error: error: numpy/arrayobject.h: No >such file or directorynumpy/arrayobject.h: No such file or directory > >The error using cython 0.10.3 is pretty much the same, except with the >spacing errors fixed: >compiling (c) and running numpy_test ... numpy_test.c:112:31: error: >numpy/arrayobject.h: No such file or directory > >Also, I should point out that the default OS X filesystem is case- >insensitive, so using a directory named 'BUILD' for building the tests >is not a great idea, as it is the same as the python 'build' >directory, so the latter gets nuked by the 'rm -rf BUILD' in 'make >test'. > >Anyhow, if I install cython despite these issues, I can build numpy- >using projects fine... for some reason, the right include directories >get pulled in. However, thereafter I have further issues. > >I've attached the troublesome code; it's pretty simple but I don't >really know enough cython yet to pull it into a minimal test case. (I >suspect from the nature of the errors that this should be easy.) > >Anyhow, here's the test case and error: >cd stefanv-bilateral >python setup.py build_ext -i >python > >Python 2.5.2 (r252:60911, Feb 22 2008, 07:57:53) >[GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin >Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy > >>> print numpy.version.version >1.3.0.dev6359 > >>> import Cython.Compiler.Version > >>> print Cython.Compiler.Version.version >0.11.beta > >>> import bilateral > >>> bilateral.bilateral(numpy.zeros((4,4)), 4, 4) >Traceback (most recent call last): > File "", line 1, in > File "bilateral/bilateral.py", line 38, in bilateral > filter_fcn = _BB.Bilat_fcn(xy_sig, z_sig, filter_size) > File "bilateral_base.pyx", line 40, in >bilateral.bilateral_base.Bilat_fcn.__init__ (bilateral/ >bilateral_base.c:752) >NameError: np > >Now, bilateral_base.pyx has the line 'cimport numpy as np' at the top, >but somehow a NameError gets thrown the first time that 'np' is used, >at the end of this snippet: > def __init__(self, spat_sig, inten_sig, filter_size=None): > if filter_size is not None and filter_size >= 2: > self.xy_size = int(filter_size) > else: > self.xy_size = int(round(spat_sig*4)) > # Make filter size odd > self.xy_size += 1-self.xy_size%2 > x = np.arange(self.xy_size, dtype=float) > > >Just for fun, I tried adding 'import numpy as np' to >bilateral_base.pyx after the cimport line (this is what numpy_text.pyx >does in the cython tests). Here then is the error that I get: > > >>> bilateral.bilateral(numpy.zeros((4,4)), 4, 4) >Traceback (most recent call last): > File "", line 1, in > File "bilateral/bilateral.py", line 40, in bilateral > return generic_filter(mat, filter_fcn.cfilter, size=size, mode=mode) > File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ >python2.5/site-packages/scipy/ndimage/filters.py", line 992, in >generic_filter > cval, origins, extra_arguments, extra_keywords) > File "bilateral_base.pyx", line 73, in >bilateral.bilateral_base.Bilat_fcn.cfilter (bilateral/bilateral_base.c: >1443) >AttributeError: 'numpy.ndarray' object has no attribute 'dimensions' > >This corresponds to the last line in this snippet: > def cfilter(self, np.ndarray data): > 'An optimized implementation' > cdef np.ndarray kernel = self.xy_ker > cdef double sigma = self.inten_sig > cdef double weight_i, weight, result, centre, dat_i > cdef double *pdata=data.data, *pker=*>kernel.data > cdef int i, dim = data.dimensions[0] > >Now, numpy arrays don't expose a dimensions attribute in python, but >the array struct has such a member, which I suppose should be exposed >to cython via 'cimport numpy'. > >So, clearly something is going wrong with cython's numpy support on my >system -- the tests fail, as well as actual use of numpy, but the >failures are different. The authors of this code can't reproduce this >with the same versions of cython and numpy on their linux systems, so >I suspect that this might be a mac issue, perhaps relating to some >assumptions about where the numpy headers live? > >On a framework install, Python.h lives in: >/Library/Frameworks/Python.framework/Versions/2.5/include >but numpy/arrayobject.h is in: >/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site- >packages/numpy/core/include > > >Can anyone offer any suggestions here? > >Thanks, > > >Zach Pincus >Postdoctoral Fellow, lab of Frank Slack >Molecular, Cellular and Developmental Biology >Yale University > > > > > From zachary.pincus at yale.edu Sun Mar 1 22:43:02 2009 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Sun, 1 Mar 2009 16:43:02 -0500 Subject: [Cython] cython / numpy / mac(?) issue In-Reply-To: <3318790436.424141@smtp.netcom.no> References: <3318790436.424141@smtp.netcom.no> Message-ID: <810D63F3-17E0-4B5C-B25C-6A8A3A38009C@yale.edu> Thanks, Dag! I appreciate your help -- that indeed fixed everything. The only surprising thing here is that the original developers of the code I'm using report no difficulties with that code as-is, using similar versions of cython. Perhaps they have some old cruft laying around from previous installs... Anyhow, your help is appreciated, as are your efforts on integrating numpy support in cython -- I'm excited to start using this tool more! Zach On Mar 1, 2009, at 4:13 PM, Dag Sverre Seljebotn wrote: > All of these are expected, I don't see any issues here (except > perhaps lack of documentation; feel free to submit improvements). > > 1) You should set the INCLUDE environment variable prior to running > runtests.py. > > 2) cimport and import are different things; you need both. > > 3) The "dimensions" field is in Cython renamed "shape" to be closer > to the Python interface. This is done in Cython/Includes/numpy.pxd > > Dag Sverre Seljebotn > -----Original Message----- > From: Zachary Pincus > Date: Sunday, Mar 1, 2009 9:35 pm > Subject: [Cython] cython / numpy / mac(?) issue > To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net > > Hello all, >> >> I've run into a problem using some cython+numpy code that some folks >> recently made available. After some back and forth with the original >> authors, we've determined that the problem appears to be with cython >> on my box. I've attached the offending code and will post a test-case >> below, but first some configuration details: >> >> I'm using OS X 10.5.6 and a python 2.5.2 that's installed as a >> "framework" (the standard OS X way of installing Python) -- the only >> major difference is that the python directories live in /Library/ >> Frameworks/[etc.] instead of /usr/local. >> >> First, I should note that on my system, using both the latest stable >> Cython release as well as the tip of cython-devel from hg, cython's >> numpy tests fail utterly, due to some sort of error with include >> paths: hg clone http://hg.cython.org/cython-devel/ >> cd cython-devel >> make local >> make test >> [...output...] >> compiling (c) and running numpy_test ... numpy_test.c: >> 129:31:numpy_test.c:129:31: error: error: numpy/arrayobject.h: No >> such file or directorynumpy/arrayobject.h: No such file or directory >> >> The error using cython 0.10.3 is pretty much the same, except with >> the >> spacing errors fixed: >> compiling (c) and running numpy_test ... numpy_test.c:112:31: error: >> numpy/arrayobject.h: No such file or directory >> >> Also, I should point out that the default OS X filesystem is case- >> insensitive, so using a directory named 'BUILD' for building the >> tests >> is not a great idea, as it is the same as the python 'build' >> directory, so the latter gets nuked by the 'rm -rf BUILD' in 'make >> test'. >> >> Anyhow, if I install cython despite these issues, I can build numpy- >> using projects fine... for some reason, the right include directories >> get pulled in. However, thereafter I have further issues. >> >> I've attached the troublesome code; it's pretty simple but I don't >> really know enough cython yet to pull it into a minimal test case. (I >> suspect from the nature of the errors that this should be easy.) >> >> Anyhow, here's the test case and error: >> cd stefanv-bilateral >> python setup.py build_ext -i >> python >> >> Python 2.5.2 (r252:60911, Feb 22 2008, 07:57:53) >> [GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin >> Type "help", "copyright", "credits" or "license" for more >> information. >>>>> import numpy >>>>> print numpy.version.version >> 1.3.0.dev6359 >>>>> import Cython.Compiler.Version >>>>> print Cython.Compiler.Version.version >> 0.11.beta >>>>> import bilateral >>>>> bilateral.bilateral(numpy.zeros((4,4)), 4, 4) >> Traceback (most recent call last): >> File "", line 1, in >> File "bilateral/bilateral.py", line 38, in bilateral >> filter_fcn = _BB.Bilat_fcn(xy_sig, z_sig, filter_size) >> File "bilateral_base.pyx", line 40, in >> bilateral.bilateral_base.Bilat_fcn.__init__ (bilateral/ >> bilateral_base.c:752) >> NameError: np >> >> Now, bilateral_base.pyx has the line 'cimport numpy as np' at the >> top, >> but somehow a NameError gets thrown the first time that 'np' is used, >> at the end of this snippet: >> def __init__(self, spat_sig, inten_sig, filter_size=None): >> if filter_size is not None and filter_size >= 2: >> self.xy_size = int(filter_size) >> else: >> self.xy_size = int(round(spat_sig*4)) >> # Make filter size odd >> self.xy_size += 1-self.xy_size%2 >> x = np.arange(self.xy_size, dtype=float) >> >> >> Just for fun, I tried adding 'import numpy as np' to >> bilateral_base.pyx after the cimport line (this is what >> numpy_text.pyx >> does in the cython tests). Here then is the error that I get: >> >>>>> bilateral.bilateral(numpy.zeros((4,4)), 4, 4) >> Traceback (most recent call last): >> File "", line 1, in >> File "bilateral/bilateral.py", line 40, in bilateral >> return generic_filter(mat, filter_fcn.cfilter, size=size, >> mode=mode) >> File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ >> python2.5/site-packages/scipy/ndimage/filters.py", line 992, in >> generic_filter >> cval, origins, extra_arguments, extra_keywords) >> File "bilateral_base.pyx", line 73, in >> bilateral.bilateral_base.Bilat_fcn.cfilter (bilateral/ >> bilateral_base.c: >> 1443) >> AttributeError: 'numpy.ndarray' object has no attribute 'dimensions' >> >> This corresponds to the last line in this snippet: >> def cfilter(self, np.ndarray data): >> 'An optimized implementation' >> cdef np.ndarray kernel = self.xy_ker >> cdef double sigma = self.inten_sig >> cdef double weight_i, weight, result, centre, dat_i >> cdef double *pdata=data.data, *pker=> *>kernel.data >> cdef int i, dim = data.dimensions[0] >> >> Now, numpy arrays don't expose a dimensions attribute in python, but >> the array struct has such a member, which I suppose should be exposed >> to cython via 'cimport numpy'. >> >> So, clearly something is going wrong with cython's numpy support on >> my >> system -- the tests fail, as well as actual use of numpy, but the >> failures are different. The authors of this code can't reproduce this >> with the same versions of cython and numpy on their linux systems, so >> I suspect that this might be a mac issue, perhaps relating to some >> assumptions about where the numpy headers live? >> >> On a framework install, Python.h lives in: >> /Library/Frameworks/Python.framework/Versions/2.5/include >> but numpy/arrayobject.h is in: >> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site- >> packages/numpy/core/include >> >> >> Can anyone offer any suggestions here? >> >> Thanks, >> >> >> Zach Pincus >> Postdoctoral Fellow, lab of Frank Slack >> Molecular, Cellular and Developmental Biology >> Yale University >> >> >> >> >> > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From dalcinl at gmail.com Mon Mar 2 15:00:29 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 2 Mar 2009 12:00:29 -0200 Subject: [Cython] PATCH: use function pointer arguments in import/export api functions In-Reply-To: <49A8595B.8040505@behnel.de> References: <49A8595B.8040505@behnel.de> Message-ID: Stefan, is this the only warning you get related to strict aliasing? No warnings at all on those lines where __Pyx_ExportFunction is called?? If that is the only warning, they I can write a better patch. On Fri, Feb 27, 2009 at 7:21 PM, Stefan Behnel wrote: > Hi, > > Lisandro Dalcin wrote: >> Just to prevent this: >> warning: ISO C forbids conversion of function pointer to object pointer type > > with this patch applied, I now get a warning on this line: > > ? ?p = *(void **)&f; > > src/lxml/lxml.etree.c: In function '__Pyx_ExportFunction': > > > src/lxml/lxml.etree.c:148416: warning: dereferencing type-punned pointer > will break strict-aliasing rules > > > This needs fixing, I guess. That said, I'm more in favour of enabling clean > strict-aliasing than to be strictly C89 compliant, unless we can have both. :) > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Mon Mar 2 15:08:16 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 2 Mar 2009 15:08:16 +0100 (CET) Subject: [Cython] PATCH: use function pointer arguments in import/export api functions In-Reply-To: References: <49A8595B.8040505@behnel.de> Message-ID: <43034.213.61.181.86.1236002896.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Lisandro Dalcin wrote: > Stefan, is this the only warning you get related to strict aliasing? > No warnings at all on those lines where __Pyx_ExportFunction is > called?? Yes, the only one that is left under Py3. > If that is the only warning, they I can write a better patch. Please do. Stefan From dalcinl at gmail.com Mon Mar 2 22:08:34 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 2 Mar 2009 19:08:34 -0200 Subject: [Cython] PyErr_Format, bare "%s" format specifier, and Py_TYPE(o)->tp_name Message-ID: After a fast view to Stefan's fix for #227, I noticed that Cython uses bare "%s" format specifiers when using PyErr_Format() to print Py_TYPE(o)->tp_name. Perhaps we should follow core CPython convention of using "%.200s" ?? . Anyway, core CPython does not follow this rule at all places ;-) -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Mon Mar 2 22:21:30 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 2 Mar 2009 19:21:30 -0200 Subject: [Cython] PATCH: use function pointer arguments in import/export api functions In-Reply-To: <43034.213.61.181.86.1236002896.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49A8595B.8040505@behnel.de> <43034.213.61.181.86.1236002896.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: Stefan, here you have the new patch... Check GCC manpages about -fstrict-aliasing, this should work. Moreover, this pattern is used on Open MPI codebase, likely for the same issues. On Mon, Mar 2, 2009 at 12:08 PM, Stefan Behnel wrote: > Lisandro Dalcin wrote: >> Stefan, is this the only warning you get related to strict aliasing? >> No warnings at all on those lines where ?__Pyx_ExportFunction is >> called?? > > Yes, the only one that is left under Py3. > > >> If that is the only warning, they I can write a better patch. > > Please do. > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 -------------- next part -------------- A non-text attachment was scrubbed... Name: IMP_EXP_API.diff Type: application/octet-stream Size: 4379 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090302/5393b9a6/attachment-0001.obj From dalcinl at gmail.com Thu Mar 5 23:58:30 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 5 Mar 2009 20:58:30 -0200 Subject: [Cython] Why is 'with gil' disabled in the parser? Any good reason? Message-ID: Error converting Pyrex file to C: ------------------------------------------------------------ ... cdef int f1(): cdef int i = 0 with gil: ^ ------------------------------------------------------------ /u/dalcinl/tmp/gil2.pyx:3:3: undeclared name not builtin: gil Of course, this one-line patch would enable it. diff -r a0ebfe3b1cc2 Cython/Compiler/Parsing.py --- a/Cython/Compiler/Parsing.py Tue Mar 03 20:44:15 2009 +0100 +++ b/Cython/Compiler/Parsing.py Thu Mar 05 20:54:59 2009 -0200 @@ -1363,8 +1363,7 @@ def p_with_statement(s): pos = s.position() s.next() # 'with' -# if s.sy == 'IDENT' and s.systring in ('gil', 'nogil'): - if s.sy == 'IDENT' and s.systring == 'nogil': + if s.sy == 'IDENT' and s.systring in ('gil', 'nogil'): state = s.systring s.next() body = p_suite(s) BTW, If the above is enabled, the following patch would be needed diff -r a0ebfe3b1cc2 Cython/Compiler/Nodes.py --- a/Cython/Compiler/Nodes.py Tue Mar 03 20:44:15 2009 +0100 +++ b/Cython/Compiler/Nodes.py Thu Mar 05 20:53:29 2009 -0200 @@ -4566,7 +4566,7 @@ def generate_execution_code(self, code): if self.state == 'gil': - code.putln("PyGILState_Release();") + code.putln("PyGILState_Release(_save);") else: code.putln("Py_BLOCK_THREADS") -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From ondrej at certik.cz Fri Mar 6 06:39:40 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Fri, 6 Mar 2009 00:39:40 -0500 Subject: [Cython] new cython fails to compile a.(key=hash) Message-ID: <85b5c3130903052139p1a073aadi7956184df26fc937@mail.gmail.com> Hi, the new cython in Debian fails to compile my old code: $ cython sympy_pyx.pyx Error converting Pyrex file to C: ------------------------------------------------------------ ... # this is faster: cdef list a a = list(self._args) #a.sort(key=hash) a.sort(None, hash) ^ ------------------------------------------------------------ /home/ondra/repos/sympyx/sympy_pyx.pyx:330:14: Call with wrong number of arguments (expected 1, got 3) Error converting Pyrex file to C: But it should work. Any ideas? Originally, I used the commented out thing before and now it also doesn't work: $ cython sympy_pyx.pyx Error converting Pyrex file to C: ------------------------------------------------------------ ... #h = hash(self._args_set) # this is faster: cdef list a a = list(self._args) a.sort(key=hash) ^ ------------------------------------------------------------ /home/ondra/repos/sympyx/sympy_pyx.pyx:329:9: Cannot convert 'int (object) except -1' to Python object Error converting Pyrex file to C: Any ideas? Is this supposed to work? Ondrej From greg.ewing at canterbury.ac.nz Fri Mar 6 07:23:00 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 06 Mar 2009 19:23:00 +1300 Subject: [Cython] Why is 'with gil' disabled in the parser? Any good reason? In-Reply-To: References: Message-ID: <49B0C144.4070107@canterbury.ac.nz> Lisandro Dalcin wrote: > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > cdef int f1(): > cdef int i = 0 > with gil: > ^ > ------------------------------------------------------------ I started to add this to Pyrex, but half way through I realised it was going to be much more complicated to implement than I had thought, and backed out of the idea in favour of having a 'with gil' clause on the whole function instead, e.g. def myfunc() with gil: ... > Of course, this one-line patch would enable it. > > - if s.sy == 'IDENT' and s.systring == 'nogil': > + if s.sy == 'IDENT' and s.systring in ('gil', 'nogil'): > > BTW, If the above is enabled, the following patch would be needed > > def generate_execution_code(self, code): > if self.state == 'gil': > - code.putln("PyGILState_Release();") > + code.putln("PyGILState_Release(_save);") You'd have to do quite a lot more than that. I don't remember all the details, but essentially if you want to use Python stuff anywhere in the function, there's setup code that needs to run with the GIL held before you even get into the user's code. It's not easy to just turn it on half way through. -- Greg From ondrej at certik.cz Fri Mar 6 07:21:26 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Fri, 6 Mar 2009 01:21:26 -0500 Subject: [Cython] new cython fails to compile a.(key=hash) In-Reply-To: <85b5c3130903052139p1a073aadi7956184df26fc937@mail.gmail.com> References: <85b5c3130903052139p1a073aadi7956184df26fc937@mail.gmail.com> Message-ID: <85b5c3130903052221x19e01b00u410520947e8775ea@mail.gmail.com> On Fri, Mar 6, 2009 at 12:39 AM, Ondrej Certik wrote: > Hi, > > the new cython in Debian fails to compile my old code: > > $ cython ?sympy_pyx.pyx > > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > > ? ? ? ?# this is faster: > ? ? ? ?cdef list a ^^^ I removed the above line and things started to work. I think it was a problem on my side. I don't understand the reason, but I think all is ok now, so don't loose time with this. Ondrej From hoytak at cs.ubc.ca Fri Mar 6 08:09:26 2009 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Thu, 5 Mar 2009 23:09:26 -0800 Subject: [Cython] Best way to display parse tree Message-ID: <4db580fd0903052309r73680d27g2d09d637245f699a@mail.gmail.com> Hello, I'm trying to get into cython development, and I'm having difficulty understanding the internals of the parse tree and how it works. Being able to print or display the parse tree would really help in this effort, as it's really tedious to work through the code trying to figure this stuff out. What is the best way to see the parse tree of a given pyx file / function / code block? Is there an easy way to do it that I don't know about? Thanks! --Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From dalcinl at gmail.com Fri Mar 6 09:24:15 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 6 Mar 2009 05:24:15 -0300 Subject: [Cython] Why is 'with gil' disabled in the parser? Any good reason? In-Reply-To: <49B0C144.4070107@canterbury.ac.nz> References: <49B0C144.4070107@canterbury.ac.nz> Message-ID: On Fri, Mar 6, 2009 at 3:23 AM, Greg Ewing wrote: > > I started to add this to Pyrex, but half way through > I realised it was going to be much more complicated to > implement than I had thought, and backed out of the idea > in favour of having a 'with gil' clause on the whole > function instead, e.g. Yes, I understand :-) ... >> Of course, this one-line patch would enable it. >> >> - ? ?if s.sy == 'IDENT' and s.systring == 'nogil': >> + ? ?if s.sy == 'IDENT' and s.systring in ('gil', 'nogil'): >> > > You'd have to do quite a lot more than that. I don't > remember all the details, but essentially if you want to > use Python stuff anywhere in the function, there's setup > code that needs to run with the GIL held before you even > get into the user's code. It's not easy to just turn it > on half way through. > Yes, of course... but I was thinking on taking advantage of this for getting some king of locking in pure C functions.... Supose you have a a pure C functions (I mean, no Py stuff), perhaps even declared 'with gil' in its signature... Now suppose that function access some global variables, then we would be able to get some sort of (global) locking using a 'with gil' block. Am I being clear? Do this make sense? Would this be useful? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Fri Mar 6 09:31:17 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 6 Mar 2009 09:31:17 +0100 (CET) Subject: [Cython] new cython fails to compile a.(key=hash) In-Reply-To: <85b5c3130903052139p1a073aadi7956184df26fc937@mail.gmail.com> References: <85b5c3130903052139p1a073aadi7956184df26fc937@mail.gmail.com> Message-ID: <34788.213.61.181.86.1236328277.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Ondrej Certik wrote: > the new cython in Debian fails to compile my old code: > > $ cython sympy_pyx.pyx > > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > > # this is faster: > cdef list a > a = list(self._args) > #a.sort(key=hash) > a.sort(None, hash) > ^ > ------------------------------------------------------------ > > /home/ondra/repos/sympyx/sympy_pyx.pyx:330:14: Call with wrong number > of arguments (expected 1, got 3) This is a known bug that apparently wasn't fixed in the 0.10 releases. The (trivial) fix is here: http://hg.cython.org/cython-devel/rev/4bbc6166052f Stefan From greg.ewing at canterbury.ac.nz Fri Mar 6 10:45:46 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 06 Mar 2009 22:45:46 +1300 Subject: [Cython] Why is 'with gil' disabled in the parser? Any good reason? In-Reply-To: References: <49B0C144.4070107@canterbury.ac.nz> Message-ID: <49B0F0CA.8080602@canterbury.ac.nz> Lisandro Dalcin wrote: > Supose you have a a pure C functions (I mean, no Py stuff), perhaps > even declared 'with gil' in its signature... Now suppose that > function access some global variables, then we would be able to get > some sort of (global) locking using a 'with gil' block. Am I being > clear? Do this make sense? Would this be useful? Hmmm... you'd be creating a third kind of state, as well as 'gil held' and 'gil not held', there would be 'gil held but you're still not allowed to mess with Python stuff in this function'. Sounds a bit messy to me. It would be better to create an intermediate function declared 'with gil' and call that from the nogil function. Then the intermediate function will run with the gil held. Also, I wouldn't like to build in a language feature whose only purpose in life is to enable abusing the gil for non-python purposes. A 'with gil' block ought to deliver all that it seems to promise, or not be there. -- Greg From dalcinl at gmail.com Fri Mar 6 11:25:58 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 6 Mar 2009 07:25:58 -0300 Subject: [Cython] Why is 'with gil' disabled in the parser? Any good reason? In-Reply-To: <49B0F0CA.8080602@canterbury.ac.nz> References: <49B0C144.4070107@canterbury.ac.nz> <49B0F0CA.8080602@canterbury.ac.nz> Message-ID: Well, yes, you are totally right... On Fri, Mar 6, 2009 at 6:45 AM, Greg Ewing wrote: > Lisandro Dalcin wrote: > >> Supose you have a a pure C functions (I mean, no Py stuff), perhaps >> even declared ?'with gil' in its signature... Now suppose that >> function access some global variables, then we would be able to get >> some sort of (global) locking using a 'with gil' block. Am I being >> clear? Do this make sense? Would this be useful? > > Hmmm... you'd be creating a third kind of state, as well > as 'gil held' and 'gil not held', there would be 'gil > held but you're still not allowed to mess with Python > stuff in this function'. Sounds a bit messy to me. > > It would be better to create an intermediate function > declared 'with gil' and call that from the nogil > function. Then the intermediate function will run > with the gil held. > > Also, I wouldn't like to build in a language feature > whose only purpose in life is to enable abusing the > gil for non-python purposes. A 'with gil' block ought > to deliver all that it seems to promise, or not be > there. > > -- > Greg > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Fri Mar 6 11:55:25 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 6 Mar 2009 11:55:25 +0100 (CET) Subject: [Cython] Best way to display parse tree In-Reply-To: <4db580fd0903052309r73680d27g2d09d637245f699a@mail.gmail.com> References: <4db580fd0903052309r73680d27g2d09d637245f699a@mail.gmail.com> Message-ID: <33209.213.61.181.86.1236336925.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Hoyt Koepke wrote: > I'm trying to get into cython development You are very welcome. > and I'm having difficulty > understanding the internals of the parse tree and how it works. Being > able to print or display the parse tree would really help in this > effort, as it's really tedious to work through the code trying to > figure this stuff out. What is the best way to see the parse tree of > a given pyx file / function / code block? Is there an easy way to do > it that I don't know about? Glad you asked. :) Yes, there is. Look into Main.py, there is a method called create_pipeline(). It imports a class called PrintTree at the top and builds the compiler pipeline at the end. If you throw an instance of that class into the pipeline at any point, it will dump out the current tree for you. Note that the output can be pretty long, so you may want to start with a short example (such as one of the test cases). Stefan From robertwb at math.washington.edu Fri Mar 6 12:43:49 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 6 Mar 2009 03:43:49 -0800 Subject: [Cython] possible problems with temp allocation? In-Reply-To: References: Message-ID: Is this still an issue? On Feb 2, 2009, at 10:18 AM, Lisandro Dalcin wrote: > See the attached files. The line 452 in generated foo.c (attached just > for your convenience) seems to be wrong. Furthermore, valgrind pointed > me to that line. > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0) > 342-451.1594__________________________________________ > _____ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From stefan_ml at behnel.de Fri Mar 6 12:59:58 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 6 Mar 2009 12:59:58 +0100 (CET) Subject: [Cython] possible problems with temp allocation? In-Reply-To: References: Message-ID: <36495.213.61.181.86.1236340798.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Robert Bradshaw wrote: > Is this still an issue? > > On Feb 2, 2009, at 10:18 AM, Lisandro Dalcin wrote: > >> See the attached files. The line 452 in generated foo.c (attached just >> for your convenience) seems to be wrong. Furthermore, valgrind pointed >> me to that line. There was a fix by Dag in reply to that e-mail. Should be ok now. Stefan From sturla at molden.no Fri Mar 6 14:48:55 2009 From: sturla at molden.no (Sturla Molden) Date: Fri, 06 Mar 2009 14:48:55 +0100 Subject: [Cython] Why is 'with gil' disabled in the parser? Any good reason? In-Reply-To: <49B0C144.4070107@canterbury.ac.nz> References: <49B0C144.4070107@canterbury.ac.nz> Message-ID: <49B129C7.2080207@molden.no> On 3/6/2009 7:23 AM, Greg Ewing wrote: > You'd have to do quite a lot more than that. I don't > remember all the details, There is also the simplified GIL API, which only requires two function calls. This is e.g. how ctypes makes sure the GIL is held in callbacks to Python. A 'with gil' block could just expand to: { PyGILState_STATE _gilstate = PyGILState_Ensure(); /* now we have the GIL */ PyGILState_Release(_gilstate); } Nothing else is required. The simplified GIL API makes sure the function does not deadlock if the caller already has the GIL. The only requirement is that a call to PyGILState_Ensure should always have a matching call to PyGILState_Release. Sturla Molden From sturla at molden.no Fri Mar 6 15:04:28 2009 From: sturla at molden.no (Sturla Molden) Date: Fri, 06 Mar 2009 15:04:28 +0100 Subject: [Cython] Why is 'with gil' disabled in the parser? Any good reason? In-Reply-To: <49B129C7.2080207@molden.no> References: <49B0C144.4070107@canterbury.ac.nz> <49B129C7.2080207@molden.no> Message-ID: <49B12D6C.8090607@molden.no> On 3/6/2009 2:48 PM, Sturla Molden wrote: > { > PyGILState_STATE _gilstate = PyGILState_Ensure(); > > /* now we have the GIL */ > > PyGILState_Release(_gilstate); > } What this means is PyGILState_Release does not release the GIL. It just resets the GIL to whatever it was before PyGILState_Ensure was called. So, { /* We might or might not hold the GIL. */ PyGILState_STATE _gilstate = PyGILState_Ensure(); /* We have the GIL. Now its safe to use the rest of Python's C API. */ PyGILState_Release(_gilstate); /* We might or might not hold the GIL. */ } Thus, with nogil: with gil: with gil: with nogil: Would be Py_BEGIN_ALLOW_THREADS { PyGILState_STATE _gilstate = PyGILState_Ensure(); { PyGILState_STATE _gilstate = PyGILState_Ensure(); { Py_BEGIN_ALLOW_THREADS { /*suite*/ } Py_END_ALLOW_THREADS } PyGILState_Release(_gilstate); } PyGILState_Release(_gilstate); } Py_END_ALLOW_THREADS S.M. From stefan_ml at behnel.de Fri Mar 6 17:32:10 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 6 Mar 2009 17:32:10 +0100 (CET) Subject: [Cython] Why is 'with gil' disabled in the parser? Any good reason? In-Reply-To: <49B129C7.2080207@molden.no> References: <49B0C144.4070107@canterbury.ac.nz> <49B129C7.2080207@molden.no> Message-ID: <52217.213.61.181.86.1236357130.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Sturla Molden wrote: > On 3/6/2009 7:23 AM, Greg Ewing wrote: > >> You'd have to do quite a lot more than that. I don't >> remember all the details, > > There is also the simplified GIL API, which only requires two function > calls. This is e.g. how ctypes makes sure the GIL is held in callbacks > to Python. > > A 'with gil' block could just expand to: > > { > PyGILState_STATE _gilstate = PyGILState_Ensure(); > > /* now we have the GIL */ > > PyGILState_Release(_gilstate); > } > > Nothing else is required. For pure C code, that might be true. However, in the current Pyrex/Cython language (and in Python, BTW), function local variables are global to the function and not local to a block. And supporting a "with gil" block would require to restrict all ref-counting related operations (e.g. variable initialisations and function cleanup) to the specific block that uses them. That's the hard part. Stefan From dagss at student.matnat.uio.no Fri Mar 6 17:36:00 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 06 Mar 2009 17:36:00 +0100 Subject: [Cython] Best way to display parse tree Message-ID: <3319205783.355064@smtp.netcom.no> There's also another way for more local debugging: print node.dump() (see Node in Nodes.py). Practical for e.g. print self.dump() within node methods. Also the dumptree function in Main.py can be used in the same way as Stefan mentions but prints the tree in the dump()-format. this should probably be consolidated -- I implemented what Stefan talked about first, and then I found that I often wanted to simply print a small part of the tree and added another way... Dag Sverre Seljebotn -----Original Message----- From: "Stefan Behnel" Date: Friday, Mar 6, 2009 11:55 am Subject: Re: [Cython] Best way to display parse tree To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net Hoyt Koepke wrote: > I'm trying to get into cython development > >You are very welcome. > > >> and I'm having difficulty > understanding the internals of the parse tree and how it works. Being > able to print or display the parse tree would really help in this > effort, as it's really tedious to work through the code trying to > figure this stuff out. What is the best way to see the parse tree of > a given pyx file / function / code block? Is there an easy way to do > it that I don't know about? > >Glad you asked. :) > >Yes, there is. Look into Main.py, there is a method called >create_pipeline(). It imports a class called PrintTree at the top and builds the compiler pipeline at the end. If you throw an instance of that class into the pipeline at any point, it will dump out the current tree for you. Note that the output can be pretty long, so you may want to start with a short example (such as one of the test cases). > >Stefan > >_______________________________________________ >Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > From dalcinl at gmail.com Fri Mar 6 18:41:57 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 6 Mar 2009 15:41:57 -0200 Subject: [Cython] Why is 'with gil' disabled in the parser? Any good reason? In-Reply-To: <49B129C7.2080207@molden.no> References: <49B0C144.4070107@canterbury.ac.nz> <49B129C7.2080207@molden.no> Message-ID: On Fri, Mar 6, 2009 at 11:48 AM, Sturla Molden wrote: > On 3/6/2009 7:23 AM, Greg Ewing wrote: > > There is also the simplified GIL API, which only requires two function > calls. This is e.g. how ctypes makes sure the GIL is held in callbacks > to Python. > Cython already used the simplified GIL API for pure-C functions requiring GIL acquisition. However, AFAIK the simplified GIL API does not suport multiple interpreters, right? Could we do it better? > A 'with gil' block could just expand to: > > { > ? ?PyGILState_STATE _gilstate = PyGILState_Ensure(); > > ? ?/* now we have the GIL */ > > ? ?PyGILState_Release(_gilstate); > } > > Nothing else is required. The simplified GIL API makes sure the function > does not deadlock if the caller already has the GIL. The only > requirement is that a call to PyGILState_Ensure should always have a > matching call to PyGILState_Release. > > > > Sturla Molden > > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From sturla at molden.no Fri Mar 6 19:05:53 2009 From: sturla at molden.no (Sturla Molden) Date: Fri, 6 Mar 2009 19:05:53 +0100 (CET) Subject: [Cython] Why is 'with gil' disabled in the parser? Any good reason? In-Reply-To: References: <49B0C144.4070107@canterbury.ac.nz> <49B129C7.2080207@molden.no> Message-ID: <747612f3e927c6d7892f663b193b4f44.squirrel@webmail.uio.no> > On Fri, Mar 6, 2009 at 11:48 AM, Sturla Molden wrote: > However, AFAIK the simplified GIL API does not suport multiple > interpreters, right? > > Could we do it better? That is correct. It does not support multiple interpreters. Sub-interpreters are a misfeature of CPython anyway. All sorts of weird things can happen when they are used. They are not completely isolated, and many extension modules do not honor them. Sub-interpreters are not useful like .NET appdomains, Java isolates, or tcl threads. Their only raison d'etre seems to be the internals of Apache's mod_python. Sturla Molden From stefan_ml at behnel.de Fri Mar 6 19:26:01 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 06 Mar 2009 19:26:01 +0100 Subject: [Cython] Why is 'with gil' disabled in the parser? Any good reason? In-Reply-To: <747612f3e927c6d7892f663b193b4f44.squirrel@webmail.uio.no> References: <49B0C144.4070107@canterbury.ac.nz> <49B129C7.2080207@molden.no> <747612f3e927c6d7892f663b193b4f44.squirrel@webmail.uio.no> Message-ID: <49B16AB9.9070603@behnel.de> Sturla Molden wrote: >> On Fri, Mar 6, 2009 at 11:48 AM, Sturla Molden wrote: >> However, AFAIK the simplified GIL API does not suport multiple >> interpreters, right? >> >> Could we do it better? > > That is correct. It does not support multiple interpreters. > > Sub-interpreters are a misfeature of CPython anyway. All sorts of weird > things can happen when they are used. They are not completely isolated, > and many extension modules do not honor them. Sub-interpreters are not > useful like .NET appdomains, Java isolates, or tcl threads. That's mainly because the whole infrastructure for making them useful is only available in Py3. http://trac.cython.org/cython_trac/ticket/173 Stefan From dalcinl at gmail.com Fri Mar 6 21:08:33 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 6 Mar 2009 18:08:33 -0200 Subject: [Cython] =?windows-1252?q?GCC=2C_warning_=91=5F=5FPyx=5FPrint=92_?= =?windows-1252?q?defined_but_not_used_=28Python_2=2EX=29?= Message-ID: classmethod.c:1229: warning: ?__Pyx_Print? defined but not used Not sure how to fix it. Could PrintOne also call Print for Py2? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Fri Mar 6 21:11:41 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 6 Mar 2009 12:11:41 -0800 Subject: [Cython] =?windows-1252?q?GCC=2C_warning_=91=5F=5FPyx=5FPrint=92_?= =?windows-1252?q?defined_but_not_used_=28Python_2=2EX=29?= In-Reply-To: References: Message-ID: <6247A7DE-C5DD-474C-BD5D-836FC508607E@math.washington.edu> On Mar 6, 2009, at 12:08 PM, Lisandro Dalcin wrote: > classmethod.c:1229: warning: ?__Pyx_Print? defined but not used > > Not sure how to fix it. Could PrintOne also call Print for Py2? The whole point of PrintOne was to avoid the overhead of calling Print. If we call it after we return, or "if (0) __Pyx_Print(...)", does that get rid of the warning even though it's optimized away? - Robert From dalcinl at gmail.com Fri Mar 6 21:20:25 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 6 Mar 2009 18:20:25 -0200 Subject: [Cython] =?windows-1252?q?GCC=2C_warning_=91=5F=5FPyx=5FPrint=92_?= =?windows-1252?q?defined_but_not_used_=28Python_2=2EX=29?= In-Reply-To: <6247A7DE-C5DD-474C-BD5D-836FC508607E@math.washington.edu> References: <6247A7DE-C5DD-474C-BD5D-836FC508607E@math.washington.edu> Message-ID: On Fri, Mar 6, 2009 at 6:11 PM, Robert Bradshaw wrote: > > The whole point of PrintOne was to avoid the overhead of calling > Print. Robert, I believe some times we complicate implementations too much for the sake of optimization... :-) > If we call it after we return, or "if (0) __Pyx_Print(...)", > does that get rid of the warning even though it's optimized away? > OK, I'll push that as a quick fix, getting those warnings when running the testsuite is really annoying -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Fri Mar 6 21:29:33 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 6 Mar 2009 12:29:33 -0800 Subject: [Cython] =?windows-1252?q?GCC=2C_warning_=91=5F=5FPyx=5FPrint=92_?= =?windows-1252?q?defined_but_not_used_=28Python_2=2EX=29?= In-Reply-To: References: <6247A7DE-C5DD-474C-BD5D-836FC508607E@math.washington.edu> Message-ID: <5026DF3C-4C02-4EDB-9AA3-CD471E283A8D@math.washington.edu> On Mar 6, 2009, at 12:20 PM, Lisandro Dalcin wrote: > On Fri, Mar 6, 2009 at 6:11 PM, Robert Bradshaw > wrote: >> >> The whole point of PrintOne was to avoid the overhead of calling >> Print. > > Robert, I believe some times we complicate implementations too much > for the sake of optimization... :-) This was motivated by some code where printing was the dominant step in the runtime. Unfortunately, PySys_GetObject((char *)"stdout") (which must be called every time) still takes some time. - Robert From dalcinl at gmail.com Fri Mar 6 21:56:48 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 6 Mar 2009 18:56:48 -0200 Subject: [Cython] =?windows-1252?q?GCC=2C_warning_=91=5F=5FPyx=5FPrint=92_?= =?windows-1252?q?defined_but_not_used_=28Python_2=2EX=29?= In-Reply-To: <5026DF3C-4C02-4EDB-9AA3-CD471E283A8D@math.washington.edu> References: <6247A7DE-C5DD-474C-BD5D-836FC508607E@math.washington.edu> <5026DF3C-4C02-4EDB-9AA3-CD471E283A8D@math.washington.edu> Message-ID: On Fri, Mar 6, 2009 at 6:29 PM, Robert Bradshaw wrote: > On Mar 6, 2009, at 12:20 PM, Lisandro Dalcin wrote: >> Robert, I believe some times we complicate implementations too much >> for the sake of optimization... :-) > > This was motivated by some code where printing was the dominant step > in the runtime. Unfortunately, PySys_GetObject((char *)"stdout") > (which must be called every time) still takes some time. > Mmm.. Are you sure that's the only cause? Every thing you need to print involve calls to PyObject_Repr, and they involve creating string objetcs... I would never expect that 'print' could run fast retaining its generality... Anyway, I've pushed a fix calling __Pyx_Print() after return... Well, other compilers (Intel) would warn about unreachable code... Anyway Cython generates a lot of unreachable code :-) ... some day far in the future.... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Fri Mar 6 22:09:18 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 6 Mar 2009 13:09:18 -0800 Subject: [Cython] =?windows-1252?q?GCC=2C_warning_=91=5F=5FPyx=5FPrint=92_?= =?windows-1252?q?defined_but_not_used_=28Python_2=2EX=29?= In-Reply-To: References: <6247A7DE-C5DD-474C-BD5D-836FC508607E@math.washington.edu> <5026DF3C-4C02-4EDB-9AA3-CD471E283A8D@math.washington.edu> Message-ID: <8F5B4DB7-5A1B-4C81-9DD7-1A8411494E24@math.washington.edu> On Mar 6, 2009, at 12:56 PM, Lisandro Dalcin wrote: > On Fri, Mar 6, 2009 at 6:29 PM, Robert Bradshaw > wrote: >> On Mar 6, 2009, at 12:20 PM, Lisandro Dalcin wrote: >>> Robert, I believe some times we complicate implementations too much >>> for the sake of optimization... :-) >> >> This was motivated by some code where printing was the dominant step >> in the runtime. Unfortunately, PySys_GetObject((char *)"stdout") >> (which must be called every time) still takes some time. >> > > Mmm.. Are you sure that's the only cause? Every thing you need to > print involve calls to PyObject_Repr, and they involve creating string > objetcs... I would never expect that 'print' could run fast retaining > its generality... Well, I was printing strings. > Anyway, I've pushed a fix calling __Pyx_Print() after return... Well, > other compilers (Intel) would warn about unreachable code... Anyway > Cython generates a lot of unreachable code :-) ... some day far in the > future.... Yeah, there's a lot of unreachable code. The logic to not output unreachable code may exceed disk space saved...it's an optimization that so far I've been happy to leave to the C compiler. - Robert From dalcinl at gmail.com Sat Mar 7 01:13:35 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 6 Mar 2009 21:13:35 -0300 Subject: [Cython] getting rid of C' stlib strcmp() in generated sources Message-ID: There would be any objections? I've already have a (tested) patch for this... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Sat Mar 7 02:23:50 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 6 Mar 2009 17:23:50 -0800 Subject: [Cython] getting rid of C' stlib strcmp() in generated sources In-Reply-To: References: Message-ID: <61C7B9B2-9B05-4AC3-ADB3-1DA85F922B3F@math.washington.edu> WIth strncmp(), or are you thinking of something else? On Mar 6, 2009, at 4:13 PM, Lisandro Dalcin wrote: > There would be any objections? I've already have a (tested) patch > for this... > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From dalcinl at gmail.com Sat Mar 7 03:32:33 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 6 Mar 2009 23:32:33 -0300 Subject: [Cython] getting rid of C' stlib strcmp() in generated sources In-Reply-To: <61C7B9B2-9B05-4AC3-ADB3-1DA85F922B3F@math.washington.edu> References: <61C7B9B2-9B05-4AC3-ADB3-1DA85F922B3F@math.washington.edu> Message-ID: I'm thinking on removing dependencies on the C stdlib... Then, I'm proposing to use a custom implementation (utility code) for the few places were strcmp() is used... strcmp() (at least for mpi4py) is the only (link-time) dependency on the C stdlib... Why I'm asking this? Just because a guy trying to build mpi4py in a supercomputing environment (SGI Altix/Itanium machines, Intel compilers, SGI MPI implementation, and system Python 2.4 built with some GCC version) had so many headaches to make it work... Perhaps the problem was that the build environment was a bit broken, but the only way to make it work was to fix mpi4py to not use ANY C stdlib call... The only missing bit is strcmp(), as Cython generates code using it... As strcmp() is trivial to implement, Cython uses it in just a few places, and mainly for work around some deficiencies in the Python C-API, then I what to remove this dependency from Cython generated code. IMHO, it is a really good idea to avoid any link-time dependencies in the generated codes... right now, I'm even moved in my own code from using malloc()/free() to use PyMem_Malloc()/PyMem_Free()... On Fri, Mar 6, 2009 at 10:23 PM, Robert Bradshaw wrote: > WIth strncmp(), or are you thinking of something else? > > On Mar 6, 2009, at 4:13 PM, Lisandro Dalcin wrote: > >> There would be any objections? I've already have a (tested) patch >> for this... >> >> >> -- >> Lisandro Dalc?n >> --------------- >> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) >> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) >> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) >> PTLC - G?emes 3450, (3000) Santa Fe, Argentina >> Tel/Fax: +54-(0)342-451.1594 >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Sat Mar 7 03:38:42 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 6 Mar 2009 18:38:42 -0800 Subject: [Cython] getting rid of C' stlib strcmp() in generated sources In-Reply-To: References: <61C7B9B2-9B05-4AC3-ADB3-1DA85F922B3F@math.washington.edu> Message-ID: <88B66FA3-3A41-46AE-8988-5E8FEFFE9AC2@math.washington.edu> On Mar 6, 2009, at 6:32 PM, Lisandro Dalcin wrote: > I'm thinking on removing dependencies on the C stdlib... Then, I'm > proposing to use a custom implementation (utility code) for the few > places were strcmp() is used... strcmp() (at least for mpi4py) is the > only (link-time) dependency on the C stdlib... Interesting. > Why I'm asking this? Just because a guy trying to build mpi4py in a > supercomputing environment (SGI Altix/Itanium machines, Intel > compilers, SGI MPI implementation, and system Python 2.4 built with > some GCC version) had so many headaches to make it work... Perhaps the > problem was that the build environment was a bit broken, but the only > way to make it work was to fix mpi4py to not use ANY C stdlib call... > The only missing bit is strcmp(), as Cython generates code using it... > > As strcmp() is trivial to implement, Cython uses it in just a few > places, and mainly for work around some deficiencies in the Python > C-API, then I what to remove this dependency from Cython generated > code. Sure, I'm fine with that. > - Robert From stefan_ml at behnel.de Sat Mar 7 08:06:15 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 07 Mar 2009 08:06:15 +0100 Subject: [Cython] getting rid of C' stlib strcmp() in generated sources In-Reply-To: References: <61C7B9B2-9B05-4AC3-ADB3-1DA85F922B3F@math.washington.edu> Message-ID: <49B21CE7.4070102@behnel.de> Lisandro Dalcin wrote: > I'm thinking on removing dependencies on the C stdlib... > [...] > As strcmp() is trivial to implement Is it really? It does sound trivial, but I would assume that many platforms have highly optimised implementations for it. At least, I know that libxml2 uses memcmp on some platforms and strcmp on others, depending on what's known to be faster. I wouldn't want to copy this kind of code into Cython, just to work around one broken platform. Honestly, how broken must a platform be to not support strcmp()? > Cython uses it in just a few > places, and mainly for work around some deficiencies in the Python > C-API, then I what to remove this dependency from Cython generated > code. I have no idea where that's currently used and if those places are in any way performance critical, so I can't comment without looking into this. I might even have added a couple of those myself. > IMHO, it is a really good idea to avoid any link-time dependencies in > the generated codes... right now, I'm even moved in my own code from > using malloc()/free() to use PyMem_Malloc()/PyMem_Free()... That by itself is perfectly ok and makes sense (although I don't know if you can use PyMem_*() in nogil functions/blocks). Stefan From robertwb at math.washington.edu Sat Mar 7 08:58:15 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 6 Mar 2009 23:58:15 -0800 Subject: [Cython] getting rid of C' stlib strcmp() in generated sources In-Reply-To: <49B21CE7.4070102@behnel.de> References: <61C7B9B2-9B05-4AC3-ADB3-1DA85F922B3F@math.washington.edu> <49B21CE7.4070102@behnel.de> Message-ID: <15D6E1AF-8B34-48CB-96EB-BEFEFC086D78@math.washington.edu> On Mar 6, 2009, at 11:06 PM, Stefan Behnel wrote: > Lisandro Dalcin wrote: >> I'm thinking on removing dependencies on the C stdlib... >> [...] >> As strcmp() is trivial to implement > > Is it really? It does sound trivial, but I would assume that many > platforms > have highly optimised implementations for it. At least, I know that > libxml2 > uses memcmp on some platforms and strcmp on others, depending on > what's > known to be faster. We could wrap it in an #ifndef strcmp > I wouldn't want to copy this kind of code into Cython, > just to work around one broken platform. > > Honestly, how broken must a platform be to not support strcmp()? I wouldn't say such a platform is broken, but it would certainly be an interesting one. Perhaps embedded devices and the likes. >> Cython uses it in just a few >> places, and mainly for work around some deficiencies in the Python >> C-API, then I what to remove this dependency from Cython generated >> code. > > I have no idea where that's currently used and if those places are > in any > way performance critical, so I can't comment without looking into > this. I > might even have added a couple of those myself. Most places it's used to (1) verify types on importing, (2) verify C api matches, and (3) to verify un-exported types in classmethod and cpdef functions, and (4) in keyword parsing code. The first two don't need to be fast, the third should be optimized anyways (at least the cpdef methods) so I'm going to do that after Lisandro's patch, and the last I'm not sure about. >> IMHO, it is a really good idea to avoid any link-time dependencies in >> the generated codes... right now, I'm even moved in my own code from >> using malloc()/free() to use PyMem_Malloc()/PyMem_Free()... > > That by itself is perfectly ok and makes sense (although I don't > know if > you can use PyMem_*() in nogil functions/blocks). I am pretty sure it's not treadsafe. - Robert From hoytak at cs.ubc.ca Sat Mar 7 09:43:15 2009 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Sat, 7 Mar 2009 00:43:15 -0800 Subject: [Cython] Best way to display parse tree In-Reply-To: <3319205783.355064@smtp.netcom.no> References: <3319205783.355064@smtp.netcom.no> Message-ID: <4db580fd0903070043k7923b8ddgb913daef27c9f620@mail.gmail.com> Thanks, everyone, for the help. I've been busy today but will try to look into it more this weekend. -- Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From stefan_ml at behnel.de Sat Mar 7 15:21:17 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 07 Mar 2009 15:21:17 +0100 Subject: [Cython] getting rid of C' stlib strcmp() in generated sources In-Reply-To: <15D6E1AF-8B34-48CB-96EB-BEFEFC086D78@math.washington.edu> References: <61C7B9B2-9B05-4AC3-ADB3-1DA85F922B3F@math.washington.edu> <49B21CE7.4070102@behnel.de> <15D6E1AF-8B34-48CB-96EB-BEFEFC086D78@math.washington.edu> Message-ID: <49B282DD.6070506@behnel.de> Robert Bradshaw wrote: > On Mar 6, 2009, at 11:06 PM, Stefan Behnel wrote: >> Lisandro Dalcin wrote: >>> Cython uses it in just a few >>> places, and mainly for work around some deficiencies in the Python >>> C-API, then I what to remove this dependency from Cython generated >>> code. >> I have no idea where that's currently used and if those places are >> in any >> way performance critical, so I can't comment without looking into >> this. I might even have added a couple of those myself. > > Most places it's used to (1) verify types on importing, (2) verify C > api matches, and (3) to verify un-exported types in classmethod and > cpdef functions, and (4) in keyword parsing code. The first two don't > need to be fast, the third should be optimized anyways (at least the > cpdef methods) so I'm going to do that after Lisandro's patch, and > the last I'm not sure about. 4) isn't performance critical either. The generic keyword parsing utility function is no longer called for keyword-only arguments or normally declared keyword arguments, as long as you do not use **kwargs. And if it's needed, I would expect the PyDict_Next() loop to dominate any string comparisons. And in the extremely likely case that we are dealing with interned strings, we skip the content comparison all together. I guess I'm +0 on this then. Stefan From dalcinl at gmail.com Sat Mar 7 19:37:48 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sat, 7 Mar 2009 15:37:48 -0300 Subject: [Cython] getting rid of C' stlib strcmp() in generated sources In-Reply-To: <15D6E1AF-8B34-48CB-96EB-BEFEFC086D78@math.washington.edu> References: <61C7B9B2-9B05-4AC3-ADB3-1DA85F922B3F@math.washington.edu> <49B21CE7.4070102@behnel.de> <15D6E1AF-8B34-48CB-96EB-BEFEFC086D78@math.washington.edu> Message-ID: On Sat, Mar 7, 2009 at 4:58 AM, Robert Bradshaw wrote: > On Mar 6, 2009, at 11:06 PM, Stefan Behnel wrote: > >> >> That by itself is perfectly ok and makes sense (although I don't >> know if >> you can use PyMem_*() in nogil functions/blocks). > > I am pretty sure it's not treadsafe. > First of all, AFAIK Cython does not emit any malloc()/free() call, so there are not issues regarding this. If you look at core CPython sources, you will discover that PyMem_Malloc() (implemented in "Objects/object.c") actually used the macro PyMem_MALLOC(). This one is defined in "Include/pymem.h". If "PYMALLOC_DEBUG" is not #define'd (should be the case for non-debug Python builds, right?), then PyMem_MALLOC actually calls system malloc(). So in this case using PyMem_Malloc() (note: the function, not the macro) should be as thread-safe as the system malloc() is. If "PYMALLOC_DEBUG" do is #define'd, then yes, propably PyMem_Malloc() is not thread-safe -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Sat Mar 7 20:03:51 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sat, 7 Mar 2009 16:03:51 -0300 Subject: [Cython] PATCH: removing strcmp() in favor of custom impl or other fixes Message-ID: Here you have the patch... Implemented as an (inline) function in a utility code that anyway is alwas emited (as it is INLINE, the GCC does not warn). Two things worth to point out: * The API import function uses a different thing, just a while loop. I do not want to complicate things in generated API headers for something that can be done with two auxiliar variables and a one-line while loop. * In the keyword handling code, I've used _PyString_Eq(), which is available in the C-API for Py2. If Python's C-API provides what we need, let's use it. Moreover, just in case, please see the implementation of _PyString_Eq(), it is smarter (and likely faster) than a strcmp() call ;-). -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 -------------- next part -------------- A non-text attachment was scrubbed... Name: STRCMP.diff Type: application/octet-stream Size: 5971 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090307/74e3eac6/attachment.obj From stefan_ml at behnel.de Sat Mar 7 20:47:23 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 07 Mar 2009 20:47:23 +0100 Subject: [Cython] PATCH: removing strcmp() in favor of custom impl or other fixes In-Reply-To: References: Message-ID: <49B2CF4B.4010901@behnel.de> Lisandro Dalcin wrote: > Here you have the patch... Implemented as an (inline) function in a > utility code Since it's never used for anything but string equality checks, it would be better to restrict it to that (i.e. 1/0), instead of using the normal strcmp() -1,0,1 return values and additionally checking for "== 0" everywhere in the code. Maybe call it "__Pyx_StrEqual()". > * The API import function uses a different thing, just a while loop. I > do not want to complicate things in generated API headers for > something that can be done with two auxiliar variables and a one-line > while loop. Sure, good idea. > * In the keyword handling code, I've used _PyString_Eq(), which is > available in the C-API for Py2. If Python's C-API provides what we > need, let's use it. Moreover, just in case, please see the > implementation of _PyString_Eq(), it is smarter (and likely faster) > than a strcmp() call ;-). Note that I already used (almost) the same optimisation in the code, which now becomes superfluous. Although I expect it to be a lot faster to check the length inside the loop before calling into the Python's C-API, these checks should be rare enough to just make that if (_PyString_Eq(**name, key)) ... The real fast-path is the loop with plain string pointer equality checks before all that. (BTW, note that this function is only available in Py2, so it's perfectly ok to use it here but not in general). Stefan From robertwb at math.washington.edu Sat Mar 7 21:18:20 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 7 Mar 2009 12:18:20 -0800 Subject: [Cython] Google summer of code Message-ID: Last summer we participated in Google's Summer of Code, which was a great thing for the Cython project. I just wanted to start a thread to coordinate our participation this next year. The timeline for mentoring organizations to apply is this this next week--should we to try applying as a separate organization? Note due to the economy the scale of GSoC is going to be smaller this year (less organizations and less students). Also, who is willing and able to be a mentor this year? Dag put up a great list of projects up at http://wiki.cython.org/ codeprojects . An older (less refined) list is up at http:// wiki.cython.org/enhancements . If anyone else has ideas, it would be good to add to this site (preferably the codeprojects one, which is better oriented to GSoC students. - Robert From robertwb at math.washington.edu Sat Mar 7 21:22:54 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 7 Mar 2009 12:22:54 -0800 Subject: [Cython] PATCH: removing strcmp() in favor of custom impl or other fixes In-Reply-To: <49B2CF4B.4010901@behnel.de> References: <49B2CF4B.4010901@behnel.de> Message-ID: On Mar 7, 2009, at 11:47 AM, Stefan Behnel wrote: > Lisandro Dalcin wrote: >> Here you have the patch... Implemented as an (inline) function in a >> utility code > > Since it's never used for anything but string equality checks, it > would be > better to restrict it to that (i.e. 1/0), instead of using the normal > strcmp() -1,0,1 return values and additionally checking for "== 0" > everywhere in the code. Maybe call it "__Pyx_StrEqual()". +1 >> * The API import function uses a different thing, just a while >> loop. I >> do not want to complicate things in generated API headers for >> something that can be done with two auxiliar variables and a one-line >> while loop. > > Sure, good idea. > > >> * In the keyword handling code, I've used _PyString_Eq(), which is >> available in the C-API for Py2. If Python's C-API provides what we >> need, let's use it. Moreover, just in case, please see the >> implementation of _PyString_Eq(), it is smarter (and likely faster) >> than a strcmp() call ;-). > > Note that I already used (almost) the same optimisation in the > code, which > now becomes superfluous. Although I expect it to be a lot faster to > check > the length inside the loop before calling into the Python's C-API, > these > checks should be rare enough to just make that > > if (_PyString_Eq(**name, key)) ... > > The real fast-path is the loop with plain string pointer equality > checks > before all that. (BTW, note that this function is only available in > Py2, so > it's perfectly ok to use it here but not in general). > > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From dalcinl at gmail.com Sat Mar 7 21:28:09 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sat, 7 Mar 2009 17:28:09 -0300 Subject: [Cython] PATCH: removing strcmp() in favor of custom impl or other fixes In-Reply-To: <49B2CF4B.4010901@behnel.de> References: <49B2CF4B.4010901@behnel.de> Message-ID: On Sat, Mar 7, 2009 at 4:47 PM, Stefan Behnel wrote: > > Since it's never used for anything but string equality checks, it would be > better to restrict it to that (i.e. 1/0), instead of using the normal > strcmp() -1,0,1 return values and additionally checking for "== 0" > everywhere in the code. Maybe call it "__Pyx_StrEqual()". > OK. At first, I was going to impement it like that, but changed my mind (for no good reason) in the process. +1. > > Note that I already used (almost) the same optimisation in the code, which > now becomes superfluous. Although I expect it to be a lot faster to check > the length inside the loop before calling into the Python's C-API, these > checks should be rare enough to just make that > > ? ? ? ?if (_PyString_Eq(**name, key)) ... > OK. I did not give a full review of that part of the code. So then this _PyString_Eq() call will unlikely be ever made, right? > The real fast-path is the loop with plain string pointer equality checks > before all that. (BTW, note that this function is only available in Py2, so > it's perfectly ok to use it here but not in general). > Of couse. I've decided to use _PyString_Eq() just because you had already special-cased Python major version. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Sat Mar 7 21:31:40 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 07 Mar 2009 21:31:40 +0100 Subject: [Cython] PATCH: use function pointer arguments in import/export api functions In-Reply-To: References: <49A8595B.8040505@behnel.de> <43034.213.61.181.86.1236002896.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49B2D9AC.5050204@behnel.de> Hi Lisandro, sorry for the late reply, I didn't see your e-mail until now. Lisandro Dalcin wrote: > Stefan, here you have the new patch... This looks totally hackish, but ... > Check GCC manpages about -fstrict-aliasing, this should work. I did, they are actually presenting exactly this as an example that works. I guess that's the way to do it, then. Anyway, lxml compiles without aliasing warnings with your patch, so I'm ok with applying it. Stefan From stefan_ml at behnel.de Sat Mar 7 21:39:17 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 07 Mar 2009 21:39:17 +0100 Subject: [Cython] PATCH: removing strcmp() in favor of custom impl or other fixes In-Reply-To: References: <49B2CF4B.4010901@behnel.de> Message-ID: <49B2DB75.3050701@behnel.de> Lisandro Dalcin wrote: > On Sat, Mar 7, 2009 at 4:47 PM, Stefan Behnel wrote: >> Note that I already used (almost) the same optimisation in the code, which >> now becomes superfluous. Although I expect it to be a lot faster to check >> the length inside the loop before calling into the Python's C-API, these >> checks should be rare enough to just make that >> >> if (_PyString_Eq(**name, key)) ... > > OK. I did not give a full review of that part of the code. So then > this _PyString_Eq() call will unlikely be ever made, right? Yes. The most common cases are handled directly inside the function's own init code, so it's unlikely already that this function will be called in a time critical context. Then, most keyword names are inlined, so the fast-path at the beginning of the loop body will hit. If it does not, it's either the rare case that a non-inlined keyword name was passed, or one of the possible error cases. None of these it performance critical. Stefan From stefan_ml at behnel.de Sat Mar 7 22:22:18 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 07 Mar 2009 22:22:18 +0100 Subject: [Cython] Google summer of code In-Reply-To: References: Message-ID: <49B2E58A.9090900@behnel.de> Robert Bradshaw wrote: > Last summer we participated in Google's Summer of Code, which was a > great thing for the Cython project. I just wanted to start a thread > to coordinate our participation this next year. > > The timeline for mentoring organizations to apply is this this next > week--should we to try applying as a separate organization? I don't have much personal experience here, so it's hard to comment on what's better. I also don't think my personal life schedule permits mentorship. > Dag put up a great list of projects up at http://wiki.cython.org/ > codeprojects . An older (less refined) list is up at http:// > wiki.cython.org/enhancements . If anyone else has ideas, it would be > good to add to this site I further split up the "more numerical features" section. There is at least one other section in there that is described by ticket #211 and this page: http://wiki.cython.org/enhancements/parallel It's about parallel execution support. Making arrays a first-class type in Cython is one way of doing that. To me, that totally sounds like a very interesting (and not so small) project in its own right. Stefan From dagss at student.matnat.uio.no Sat Mar 7 23:24:42 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 7 Mar 2009 23:24:42 +0100 (CET) Subject: [Cython] Google summer of code In-Reply-To: <49B2E58A.9090900@behnel.de> References: <49B2E58A.9090900@behnel.de> Message-ID: <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> Stefan Behnel wrote: > > Robert Bradshaw wrote: >> Last summer we participated in Google's Summer of Code, which was a >> great thing for the Cython project. I just wanted to start a thread >> to coordinate our participation this next year. >> >> The timeline for mentoring organizations to apply is this this next >> week--should we to try applying as a separate organization? > > I don't have much personal experience here, so it's hard to comment on > what's better. I also don't think my personal life schedule permits > mentorship. Disadvantages: There's the work of submitting the actual application, which one shouldn't take on unless we know we want to participate. The number of possible mentors play a strong role here. Myself I was hoping to wait a month and see before I decide on whether I want to mentor. I hope (though nothing is decided!) to be able to mentor a student for a year-long university project on Cython here in Oslo, and mentoring for a full year of work will definitely have higher priority for me than two months (as the investment overhead in getting somebody to know coding in Cython is likely comparable in the two cases). That taken in consideration I now have second thoughts on mentoring in GSoC; the same amount of mentorship in autumn can potentially pay off more in amount of code written. So to get things declared, I'm unfortunately only a "maybe" on the mentorship side, at least right now. Dag Sverre From dagss at student.matnat.uio.no Sat Mar 7 23:37:11 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 7 Mar 2009 23:37:11 +0100 (CET) Subject: [Cython] Buffer syntax/array primitive type Message-ID: <22684a329c8ff577025d36e76274817f.squirrel@webmail.uio.no> It seems to be the right time to raise this issue now as we're talking future projects. I actually raised the issue on the NumPy list first last week so that some feedback from the numerical community is ready for whoever is interested right away. I think the sum-up is that something like this is needed if we want to take up a real fight with Fortran (though whether that is one of Cython's goals is always debatable). Anyway, I feel myself that this has a better feel than a new array type, as it makes use of the buffer interface and so bridges better with Python-land. My proposal is here: http://wiki.cython.org/enhancements/buffersyntax And the thread from the numpy list here: http://thread.gmane.org/gmane.comp.python.numeric.general/28439 Dag Sverre From robertwb at math.washington.edu Sun Mar 8 00:06:00 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 7 Mar 2009 15:06:00 -0800 Subject: [Cython] Buffer syntax/array primitive type In-Reply-To: <22684a329c8ff577025d36e76274817f.squirrel@webmail.uio.no> References: <22684a329c8ff577025d36e76274817f.squirrel@webmail.uio.no> Message-ID: On Mar 7, 2009, at 2:37 PM, Dag Sverre Seljebotn wrote: > It seems to be the right time to raise this issue now as we're talking > future projects. > > I actually raised the issue on the NumPy list first last week so > that some > feedback from the numerical community is ready for whoever is > interested > right away. I think the sum-up is that something like this is > needed if we > want to take up a real fight with Fortran (though whether that is > one of > Cython's goals is always debatable). > > Anyway, I feel myself that this has a better feel than a new array > type, > as it makes use of the buffer interface and so bridges better with > Python-land. > > My proposal is here: > http://wiki.cython.org/enhancements/buffersyntax > > And the thread from the numpy list here: > http://thread.gmane.org/gmane.comp.python.numeric.general/28439 Very interesting, and I like the syntax for the most part (int [:,:,"fortran"] seems a bit odd to me, perhaps mode should be a keyword). Can you pass them around/have them assigned to global/class-level variables? Will they be garbage collected? Also, how much of numpy will you be duplicating, and how easy will it be to turn one of these bare buffers into a numpy array? (I do like the easier int* <-> int [:] relation though.) - Robert From robertwb at math.washington.edu Sun Mar 8 00:11:59 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 7 Mar 2009 15:11:59 -0800 Subject: [Cython] Google summer of code In-Reply-To: <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> References: <49B2E58A.9090900@behnel.de> <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> Message-ID: <993FE1B3-30D3-45C7-9E40-B8C420E2492C@math.washington.edu> On Mar 7, 2009, at 2:24 PM, Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> >> Robert Bradshaw wrote: >>> Last summer we participated in Google's Summer of Code, which was a >>> great thing for the Cython project. I just wanted to start a thread >>> to coordinate our participation this next year. >>> >>> The timeline for mentoring organizations to apply is this this next >>> week--should we to try applying as a separate organization? >> >> I don't have much personal experience here, so it's hard to >> comment on >> what's better. I also don't think my personal life schedule permits >> mentorship. > > Disadvantages: There's the work of submitting the actual application, > which one shouldn't take on unless we know we want to participate. > > The number of possible mentors play a strong role here. > > Myself I was hoping to wait a month and see before I decide on > whether I > want to mentor. I hope (though nothing is decided!) to be able to > mentor a > student for a year-long university project on Cython here in Oslo, and > mentoring for a full year of work will definitely have higher > priority for > me than two months (as the investment overhead in getting somebody > to know > coding in Cython is likely comparable in the two cases). That taken in > consideration I now have second thoughts on mentoring in GSoC; the > same > amount of mentorship in autumn can potentially pay off more in > amount of > code written. > > So to get things declared, I'm unfortunately only a "maybe" on the > mentorship side, at least right now. I'm a "probably," but it's sounding like at this point we don't have enough potential mentors to be a full mentoring organization unless someone else stands up. On the numerical side of things, perhaps a NumPy person would be willing. On the other hand, I think we're higher on Python's radar than last year, so I'm hopeful of at least getting one (maybe two) spots for them. The quality of the students and projects will play a huge role here--one thing I learned at the summit last year is that it's not even worth you time to go after sub-par projects (though good students are an extremely good return on the investment of mentoring them). - Robert From dalcinl at gmail.com Sun Mar 8 00:15:03 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sat, 7 Mar 2009 20:15:03 -0300 Subject: [Cython] PATCH: use function pointer arguments in import/export api functions In-Reply-To: <49B2D9AC.5050204@behnel.de> References: <49A8595B.8040505@behnel.de> <43034.213.61.181.86.1236002896.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B2D9AC.5050204@behnel.de> Message-ID: Yes, it seems hackish, but I do not know of any other way, and other codes do also use this hackery... Please, could you push the fixes? On Sat, Mar 7, 2009 at 5:31 PM, Stefan Behnel wrote: > Hi Lisandro, > > sorry for the late reply, I didn't see your e-mail until now. > > Lisandro Dalcin wrote: >> Stefan, here you have the new patch... > > This looks totally hackish, but ... > >> Check GCC manpages about -fstrict-aliasing, this should work. > > I did, they are actually presenting exactly this as an example that works. > I guess that's the way to do it, then. > > Anyway, lxml compiles without aliasing warnings with your patch, so I'm ok > with applying it. > - Show quoted text - > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Sun Mar 8 00:40:09 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 7 Mar 2009 15:40:09 -0800 Subject: [Cython] PATCH: removing strcmp() in favor of custom impl or other fixes In-Reply-To: <49B2DB75.3050701@behnel.de> References: <49B2CF4B.4010901@behnel.de> <49B2DB75.3050701@behnel.de> Message-ID: On Mar 7, 2009, at 12:39 PM, Stefan Behnel wrote: > Lisandro Dalcin wrote: >> On Sat, Mar 7, 2009 at 4:47 PM, Stefan Behnel >> wrote: >>> Note that I already used (almost) the same optimisation in the >>> code, which >>> now becomes superfluous. Although I expect it to be a lot faster >>> to check >>> the length inside the loop before calling into the Python's C- >>> API, these >>> checks should be rare enough to just make that >>> >>> if (_PyString_Eq(**name, key)) ... >> >> OK. I did not give a full review of that part of the code. So then >> this _PyString_Eq() call will unlikely be ever made, right? > > Yes. The most common cases are handled directly inside the > function's own > init code, so it's unlikely already that this function will be > called in a > time critical context. Then, most keyword names are inlined, so the > fast-path at the beginning of the loop body will hit. If it does > not, it's > either the rare case that a non-inlined keyword name was passed, or > one of > the possible error cases. None of these it performance critical. Note I removed a strcmp() at http://hg.cython.org/cython-devel/rev/ cd51476651c6 which will make one chunk from your patch unnecessary (just a heads up when you merge). - Robert From dalcinl at gmail.com Sun Mar 8 02:06:20 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sat, 7 Mar 2009 22:06:20 -0300 Subject: [Cython] allowing archive downloads of cython-devel repo Message-ID: In case it works (not shure how cython-devel is actually served), could this be added to .hg/hgrc in cython-devel (under [web] section, as shown)? [web] allow_archive = gz, zip, bz2 -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Sun Mar 8 07:30:18 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 7 Mar 2009 22:30:18 -0800 Subject: [Cython] PyErr_Format, bare "%s" format specifier, and Py_TYPE(o)->tp_name In-Reply-To: References: Message-ID: I'm not opposed to this, though the only place I could see it being very useful is in case Py_TYPE(o)->tp_name doesn't really point to a string because something really got messed up. On Mar 2, 2009, at 1:08 PM, Lisandro Dalcin wrote: > After a fast view to Stefan's fix for #227, I noticed that Cython uses > bare "%s" format specifiers when using PyErr_Format() to print > Py_TYPE(o)->tp_name. > > Perhaps we should follow core CPython convention of using "%.200s" ?? > . Anyway, core CPython does not follow this rule at all places ;-) > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From prabhu at aero.iitb.ac.in Sun Mar 8 07:30:27 2009 From: prabhu at aero.iitb.ac.in (Prabhu Ramachandran) Date: Sun, 08 Mar 2009 12:00:27 +0530 Subject: [Cython] Buffer syntax/array primitive type In-Reply-To: <22684a329c8ff577025d36e76274817f.squirrel@webmail.uio.no> References: <22684a329c8ff577025d36e76274817f.squirrel@webmail.uio.no> Message-ID: <49B36603.6060108@aero.iitb.ac.in> On 03/08/09 04:07, Dag Sverre Seljebotn wrote: > Anyway, I feel myself that this has a better feel than a new array type, > as it makes use of the buffer interface and so bridges better with > Python-land. > > My proposal is here: > http://wiki.cython.org/enhancements/buffersyntax This looks really nice. I'm just a little curious about one little feature, would this support resizing the buffer? i.e. could I do def f(int[:] arr): cdef int i # ... arr.append(i) say? My guess is no but this would be really handy. I sometimes need a fast resizable container for basic types and find that using a list is about 4-7 times slower than a simple class exposing a block of memory with suitable resizing. Creating such a container is easy for one type but without a templating system this gets too messy when I want support for ints, longs, floats and doubles. cheers, prabhu From robertwb at math.washington.edu Sun Mar 8 07:38:52 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 7 Mar 2009 22:38:52 -0800 Subject: [Cython] allowing archive downloads of cython-devel repo In-Reply-To: References: Message-ID: Sure. Done. On Mar 7, 2009, at 5:06 PM, Lisandro Dalcin wrote: > In case it works (not shure how cython-devel is actually served), > could this be added to .hg/hgrc in cython-devel (under [web] section, > as shown)? > > [web] > allow_archive = gz, zip, bz2 > > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From robertwb at math.washington.edu Sun Mar 8 07:41:06 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 7 Mar 2009 22:41:06 -0800 Subject: [Cython] improving Py <-> C conversion of numeric types In-Reply-To: References: Message-ID: <2A716CBD-4D2E-488A-8FEF-AE14AA688C7A@math.washington.edu> On Feb 26, 2009, at 10:52 AM, Lisandro Dalcin wrote: > I'm working on a patch related to de subject of this mail, but I will > need any of you to make some coments on the above .... > > In [1]: from Cython.Compiler import PyrexTypes > > In [2]: PyrexTypes.c_anon_enum_type.rank > Out[2]: -1 > > In [3]: PyrexTypes.c_anon_enum_type.sign_and_name() > Out[3]: 'long double' > > > [2] seems fine, but what about [3] ?? Should we specialize the method > and make it return 'int' ?? Yes. Actually, isn't "sign_and_name" supposed to return the actual name of the enum? - Robert From stefan_ml at behnel.de Sun Mar 8 07:53:32 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 08 Mar 2009 07:53:32 +0100 Subject: [Cython] PyErr_Format, bare "%s" format specifier, and Py_TYPE(o)->tp_name In-Reply-To: References: Message-ID: <49B36B6C.2000300@behnel.de> Robert Bradshaw wrote: > On Mar 2, 2009, at 1:08 PM, Lisandro Dalcin wrote: > >> After a fast view to Stefan's fix for #227, I noticed that Cython uses >> bare "%s" format specifiers when using PyErr_Format() to print >> Py_TYPE(o)->tp_name. >> >> Perhaps we should follow core CPython convention of using "%.200s" ?? >> . Anyway, core CPython does not follow this rule at all places ;-) > > I'm not opposed to this, though the only place I could see it being > very useful is in case Py_TYPE(o)->tp_name doesn't really point to a > string because something really got messed up. Fine with me, too. Stefan From poonaatsoc at gmail.com Sun Mar 8 07:56:32 2009 From: poonaatsoc at gmail.com (Anoop Saldanha) Date: Sun, 8 Mar 2009 12:26:32 +0530 Subject: [Cython] Google summer of code In-Reply-To: <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> References: <49B2E58A.9090900@behnel.de> <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> Message-ID: On Sun, Mar 8, 2009 at 3:54 AM, Dag Sverre Seljebotn < dagss at student.matnat.uio.no> wrote: > Stefan Behnel wrote: > > > > Robert Bradshaw wrote: > >> Last summer we participated in Google's Summer of Code, which was a > >> great thing for the Cython project. I just wanted to start a thread > >> to coordinate our participation this next year. > >> > >> The timeline for mentoring organizations to apply is this this next > >> week--should we to try applying as a separate organization? > > > > I don't have much personal experience here, so it's hard to comment on > > what's better. I also don't think my personal life schedule permits > > mentorship. > > Disadvantages: There's the work of submitting the actual application, > which one shouldn't take on unless we know we want to participate. > > The number of possible mentors play a strong role here. > > Myself I was hoping to wait a month and see before I decide on whether I > want to mentor. I hope (though nothing is decided!) to be able to mentor a > student for a year-long university project on Cython here in Oslo, and > mentoring for a full year of work will definitely have higher priority for > me than two months (as the investment overhead in getting somebody to know > coding in Cython is likely comparable in the two cases). That taken in > consideration I now have second thoughts on mentoring in GSoC; the same > amount of mentorship in autumn can potentially pay off more in amount of > code written. > I was digging into cython currently, to work full time(as a developer) on it. I am not a student, but I would like to have your help in understanding cython better so that I can start contributing to cython. > > So to get things declared, I'm unfortunately only a "maybe" on the > mentorship side, at least right now. > > Dag Sverre > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Regards, Anoop S. People who says, "It's not winning or losing that matters, but participation", probably lost. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090308/aa6ad268/attachment-0001.htm From stefan_ml at behnel.de Sun Mar 8 08:04:05 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 08 Mar 2009 08:04:05 +0100 Subject: [Cython] Google summer of code In-Reply-To: References: <49B2E58A.9090900@behnel.de> <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> Message-ID: <49B36DE5.8010404@behnel.de> Hi, Anoop Saldanha wrote: > I was digging into cython currently, to work full time(as a developer) on > it. I am not a student, but I would like to have your help in understanding > cython better so that I can start contributing to cython. Help is always appreciated. Do you have any special interest in doing so? Any idea what part of Cython you would like to contribute to? That would make it easier to point you to a good entry point. Stefan From robertwb at math.washington.edu Sun Mar 8 08:17:46 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 7 Mar 2009 23:17:46 -0800 Subject: [Cython] Google summer of code In-Reply-To: References: <49B2E58A.9090900@behnel.de> <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> Message-ID: <7B956834-3610-4895-A215-786AF07980AF@math.washington.edu> On Mar 7, 2009, at 10:56 PM, Anoop Saldanha wrote: > I was digging into cython currently, to work full time(as a > developer) on it. I am not a student, but I would like to have > your help in understanding cython better so that I can start > contributing to cython. Very cool. What is your background? We'd be glad to help out, though none of us have the liberty of being full time developers :). - Robert From poonaatsoc at gmail.com Sun Mar 8 08:27:07 2009 From: poonaatsoc at gmail.com (Anoop Saldanha) Date: Sun, 8 Mar 2009 12:57:07 +0530 Subject: [Cython] Google summer of code In-Reply-To: <7B956834-3610-4895-A215-786AF07980AF@math.washington.edu> References: <49B2E58A.9090900@behnel.de> <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> <7B956834-3610-4895-A215-786AF07980AF@math.washington.edu> Message-ID: I am currently working for a company(to clear up the word working full-time :) ). Apart from my office work I can shell out around 8 hours everyday on my other work, which currently is digging into cython. I am an ex google summer of coder, though I doubt it really matters. On the coding front I am currently working with c and python. On Sun, Mar 8, 2009 at 12:47 PM, Robert Bradshaw < robertwb at math.washington.edu> wrote: > On Mar 7, 2009, at 10:56 PM, Anoop Saldanha wrote: > > > I was digging into cython currently, to work full time(as a > > developer) on it. I am not a student, but I would like to have > > your help in understanding cython better so that I can start > > contributing to cython. > > Very cool. What is your background? We'd be glad to help out, though > none of us have the liberty of being full time developers :). > > - Robert > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Regards, Anoop S. People who says, "It's not winning or losing that matters, but participation", probably lost. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090308/523ab16f/attachment.htm From stefan_ml at behnel.de Sun Mar 8 08:37:21 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 08 Mar 2009 08:37:21 +0100 Subject: [Cython] PATCH: use function pointer arguments in import/export api functions In-Reply-To: References: <49A8595B.8040505@behnel.de> <43034.213.61.181.86.1236002896.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B2D9AC.5050204@behnel.de> Message-ID: <49B375B1.7020200@behnel.de> Lisandro Dalcin wrote: > Yes, it seems hackish, but I do not know of any other way, and other > codes do also use this hackery... > > Please, could you push the fixes? It's up. Stefan From robertwb at math.washington.edu Sun Mar 8 10:00:12 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 8 Mar 2009 01:00:12 -0800 Subject: [Cython] Cython 0.11 release candidate Message-ID: Finally, it looks like we're on the brink of releasing 0.11. Try out the release candidate at http://cython.org/Cython-0.11.rc.tar.gz - Robert From dagss at student.matnat.uio.no Sun Mar 8 11:31:41 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 8 Mar 2009 11:31:41 +0100 (CET) Subject: [Cython] Buffer syntax/array primitive type In-Reply-To: References: <22684a329c8ff577025d36e76274817f.squirrel@webmail.uio.no> Message-ID: <1b5d42aff49cb2e508b937cdad2d0771.squirrel@webmail.uio.no> Robert Bradshaw wrote: > On Mar 7, 2009, at 2:37 PM, Dag Sverre Seljebotn wrote: >> My proposal is here: >> http://wiki.cython.org/enhancements/buffersyntax >> >> And the thread from the numpy list here: >> http://thread.gmane.org/gmane.comp.python.numeric.general/28439 > > Very interesting, and I like the syntax for the most part (int > [:,:,"fortran"] seems a bit odd to me, perhaps mode should be a > keyword). Consider though that @cython.locals(a=cython.int[:,:,"fortran"]) def foo(): ... is legal Python. Dropping Python compatability on a new syntax like this seems like a high price to pay. Perhaps cython.int[:,:,cython.buffer.FORTRAN_MODE] or similar? > Can you pass them around/have them assigned to global/class-level > variables? Will they be garbage collected? Also, how much of numpy Global/class-level was always the intention for current buffers as well, just never got around to it. So yes, it would just need to be done. Implementation side they can be considered (smaller variations on) Py_buffer structs. The variables themselves will (up to optimization) be structs which are copied by value, however they point to Python objects which will be refcounted in the process. a) They will hold a refcounted reference to an underlying object owning the memory, and b) they will (sometimes) need a seperate "acquisition refcount" so that when making a slice, the buffer will not be released until the last slice goes out of scope. All in all a happy mixture of copying by value and reference counting. > will you be duplicating, and how easy will it be to turn one of these > bare buffers into a numpy array? (I do like the easier int* <-> int > [:] relation though.) myarr = numpy.array(mybuf) This is currently used for conversion from lists etc. Implementation-wise this is rather simple (NumPy can support memoryview in newer releases/Python versions, and until then we can subclass memoryview to provide the NumPy __toarray__ protocol which numpy.array supports). How much duplication: Well at first nothing, but suppose the whole wanted-list is done (which depends on bringing in e.g. a student): All in all, it's more a matter of reimplementing small parts of Fortran than reimplementing NumPy. (I am actually toying with the idea of outputting and linking in Fortran code, but I don't think it would buy us much). a) Slices is pretty much duplication of functionality, but doing it compile-time, for speed, means it must be coded in a different way than in NumPy. b) The same goes for arithmetic. The NumPy one-operation-at-the-time approach isn't the most efficient one and if we decide to do this ourselves, it would be in order to do it in a different way, i.e. turn "a = b + sqrt(c)" into a full inline loop. c) Functions (log, sqrt, sin, etc.). Currently there's NumPy which only takes a whole array and C which only takes a single element. What one needs is functions which can be used on both levels, so that "a = sqrt(c)" can be used on buffer level but be turned by Cython into a loop calling sqrt on all elements individually. All in all, I think arithmetic on buffers needs pluggability to handle different backends anyway (does one have the commercial Intel MKL library? And so on.) At the cost of a very strong NumPy dependency (if this feature is used), one could of course implement this through NumPy at first, perhaps as one "backend". I.e. Cython-generated modules with buffer arithmetic would at first implicitly depend on NumPy (even if all you're trying to do is sum the contents of two C arrays). But I'm still thinking about possible solutions for the arithmetic part, it's a tricky part. Dag Sverre From dagss at student.matnat.uio.no Sun Mar 8 11:38:43 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 8 Mar 2009 11:38:43 +0100 (CET) Subject: [Cython] Buffer syntax/array primitive type In-Reply-To: <49B36603.6060108@aero.iitb.ac.in> References: <22684a329c8ff577025d36e76274817f.squirrel@webmail.uio.no> <49B36603.6060108@aero.iitb.ac.in> Message-ID: Prabhu Ramachandran wrote: > On 03/08/09 04:07, Dag Sverre Seljebotn wrote: >> Anyway, I feel myself that this has a better feel than a new array type, >> as it makes use of the buffer interface and so bridges better with >> Python-land. >> >> My proposal is here: >> http://wiki.cython.org/enhancements/buffersyntax > > This looks really nice. I'm just a little curious about one little > feature, would this support resizing the buffer? i.e. could I do > > def f(int[:] arr): > cdef int i > # ... > arr.append(i) No, this would never happen. This is defined as syntax candy for PEP 3118, which exports a view to allocated memory but doesn't say anything about how that memory is allocated or deallocated. I believe we must solve this, but in a different way. Given that we get a "fast list", wouldn't we then want a "fast dict"? Adding a type templating mechanism to Cython is not out of the question I think, there was some talk 3/4 year ago. But now I myself believe the most promising option is to wrap C++ in a very good way, so that one can do something along the lines of cdef cpp.vector[int] arr = cpp.vector[int]() cdef cpp.map[str, cpp.vector[int]] map = cpp.map[str, cpp.vector[int]]() arr.push_back(3) map[40] = arr (Yes, those lines raises about 10 questions about reference counting vs. copy-by-value semantics; making C++ access feel Pythonic is part of such a challenge IMO.) Dag Sverre From dagss at student.matnat.uio.no Sun Mar 8 11:43:17 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 8 Mar 2009 11:43:17 +0100 (CET) Subject: [Cython] Buffer syntax/array primitive type In-Reply-To: References: <22684a329c8ff577025d36e76274817f.squirrel@webmail.uio.no> <49B36603.6060108@aero.iitb.ac.in> Message-ID: <0351b79ec08c54e9c50dc2acb7c50abb.squirrel@webmail.uio.no> Dag Sverre Seljebotn wrote: > Adding a type templating mechanism to Cython is not out of the question I > think, there was some talk 3/4 year ago. But now I myself believe the most > promising option is to wrap C++ in a very good way, so that one can do > something along the lines of > > cdef cpp.vector[int] arr = cpp.vector[int]() > cdef cpp.map[str, cpp.vector[int]] map = cpp.map[str, cpp.vector[int]]() > arr.push_back(3) > map[40] = arr Sorry, that should have been map["mystring"] = arr. Dag Sverre From prabhu at aero.iitb.ac.in Sun Mar 8 12:15:05 2009 From: prabhu at aero.iitb.ac.in (Prabhu Ramachandran) Date: Sun, 08 Mar 2009 16:45:05 +0530 Subject: [Cython] Buffer syntax/array primitive type In-Reply-To: References: <22684a329c8ff577025d36e76274817f.squirrel@webmail.uio.no> <49B36603.6060108@aero.iitb.ac.in> Message-ID: <49B3A8B9.4030208@aero.iitb.ac.in> On 03/08/09 16:08, Dag Sverre Seljebotn wrote: >> def f(int[:] arr): >> cdef int i >> # ... >> arr.append(i) > > No, this would never happen. This is defined as syntax candy for PEP 3118, > which exports a view to allocated memory but doesn't say anything about > how that memory is allocated or deallocated. That is what I had imagined but hope springs eternal. > I believe we must solve this, but in a different way. Given that we get a > "fast list", wouldn't we then want a "fast dict"? Indeed, that would be the next obvious step. :) > Adding a type templating mechanism to Cython is not out of the question I > think, there was some talk 3/4 year ago. But now I myself believe the most > promising option is to wrap C++ in a very good way, so that one can do > something along the lines of > > cdef cpp.vector[int] arr = cpp.vector[int]() > cdef cpp.map[str, cpp.vector[int]] map = cpp.map[str, cpp.vector[int]]() > arr.push_back(3) > map[40] = arr I guess that would work too. > (Yes, those lines raises about 10 questions about reference counting vs. > copy-by-value semantics; making C++ access feel Pythonic is part of such a > challenge IMO.) Indeed and all those would have to be handled very carefully. A cheap templating system would be of course to do string substitutions and spit out the template instances into a pyx file using Python -- along the lines of numpy's own templating with the .src files. A more integrated solution would of course be really convenient. cheers, prabhu From stefan_ml at behnel.de Sun Mar 8 14:33:57 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 08 Mar 2009 14:33:57 +0100 Subject: [Cython] Cython 0.11 release candidate In-Reply-To: References: Message-ID: <49B3C945.7020102@behnel.de> Hi, Robert Bradshaw wrote: > Finally, it looks like we're on the brink of releasing 0.11. Try out > the release candidate at http://cython.org/Cython-0.11.rc.tar.gz If this represents cython-devel up to the last fix (838a6b7cae62) for Lisandros warnings, I'm fine with the release. I just cleaned up a couple of more tickets in trac and added some test cases for them. I don't care if they go in or not, but it would be good to have them in the release branch at least, as we might fix some of them in the 0.11.x life cycle. Stefan From ndbecker2 at gmail.com Sun Mar 8 15:19:20 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Sun, 08 Mar 2009 10:19:20 -0400 Subject: [Cython] Cython 0.11 release candidate References: Message-ID: Robert Bradshaw wrote: > Finally, it looks like we're on the brink of releasing 0.11. Try out > the release candidate at http://cython.org/Cython-0.11.rc.tar.gz > > - Robert Running on Fedora f10: running tests against Cython 0.11.rc Python 2.5.2 (r252:60911, Sep 30 2008, 15:42:03) [GCC 4.3.2 20080917 (Red Hat 4.3.2-4)] Traceback (most recent call last): File "runtests.py", line 559, in pyxbuild_dir=os.path.join(WORKDIR, "support")) File "/home/nbecker/RPM/BUILD/Cython-0.11.rc/pyximport/pyxbuild.py", line 26, in pyx_to_dll assert os.path.exists(filename) AssertionError From psp.reachable at gmail.com Sun Mar 8 16:10:29 2009 From: psp.reachable at gmail.com (Prajwal Suhas P) Date: Sun, 8 Mar 2009 20:40:29 +0530 Subject: [Cython] checking out the repo Message-ID: I need to checkout the cython trunk from the repository. I have checked out the check-devel repo hg clone http://hg.cython.org/cython-devel/ check-devel Is the check-devel repo, the trunk for cython devel? I saw 3-4 repos for cython. I saw a similar mail in the mailing list archive, but found no reply to that. Have I checked out the right repo? -- Regards, Prajwal S. "There is little difference in people,but this little difference makes a big difference.The little difference is attitude,the big difference is whether its positive or negative. " -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090308/718f3e6b/attachment.htm From stefan_ml at behnel.de Sun Mar 8 16:38:49 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 08 Mar 2009 16:38:49 +0100 Subject: [Cython] checking out the repo In-Reply-To: References: Message-ID: <49B3E689.2050708@behnel.de> Prajwal Suhas P wrote: > I need to checkout the cython trunk from the repository. I have checked out > the check-devel repo That's the right one. There's "cython", which has the latest official release (and possibly some additional fixes), and "cython-devel", which has the latest development version (i.e. the "trunk"). "cython-docs" has the official documentation. Other repositories come up at need. Stefan From dalcinl at gmail.com Mon Mar 9 01:13:45 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sun, 8 Mar 2009 21:13:45 -0300 Subject: [Cython] improving Py <-> C conversion of numeric types In-Reply-To: <2A716CBD-4D2E-488A-8FEF-AE14AA688C7A@math.washington.edu> References: <2A716CBD-4D2E-488A-8FEF-AE14AA688C7A@math.washington.edu> Message-ID: On Sun, Mar 8, 2009 at 3:41 AM, Robert Bradshaw wrote: > On Feb 26, 2009, at 10:52 AM, Lisandro Dalcin wrote: >> >> [2] seems fine, but what about [3] ?? Should we specialize the method >> and make it return 'int' ?? > > Yes. Actually, isn't "sign_and_name" supposed to return the actual > name of the enum? > But an anon enum has no name! -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From sturla at molden.no Mon Mar 9 13:01:58 2009 From: sturla at molden.no (Sturla Molden) Date: Mon, 09 Mar 2009 13:01:58 +0100 Subject: [Cython] Buffer syntax/array primitive type In-Reply-To: References: <22684a329c8ff577025d36e76274817f.squirrel@webmail.uio.no> Message-ID: <49B50536.9020708@molden.no> On 3/8/2009 12:06 AM, Robert Bradshaw wrote: > Very interesting, and I like the syntax for the most part (int > [:,:,"fortran"] seems a bit odd to me, perhaps mode should be a > keyword). I wonder if you would consider cdef int[::,:] arr to indicate that a dimension might have strides in the first dimension different from 1. And similar cdef int[:,::] arr cdef int[::,::] arr That would allow you to ignore strides and speed up indexing for most arrays. The most common case is unit strides. S.M. From stefan_ml at behnel.de Mon Mar 9 13:17:51 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 9 Mar 2009 13:17:51 +0100 (CET) Subject: [Cython] Better call optimisation Message-ID: <36891.213.61.181.86.1236601071.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Hi, there are a couple of tickets related to call optimisations. http://trac.cython.org/cython_trac/ticket/3 http://trac.cython.org/cython_trac/ticket/131 http://trac.cython.org/cython_trac/ticket/166 The getattr3() problem also relates to other cases, such as the fact that we can't currently optimise cdef list l = [...] l.sort() into a call to PyList_Sort(), as this would break calls like l.sort(key=...) My suggestion is to remove the current support for optimised methods of builtins, and to write a dedicated transform that intercepts the tree on call nodes and that can then handle all cases in explicit code. This can include, for example, adding a None check if required, or handling different call signatures only some of which can be optimised, maybe even by changing keyword arguments into positional arguments of a matching C-API function. I imagine that it would use some kind of dispatch mechanism on (obj_type, obj_attribute_name) and on builtin function names. I also think that the current generic "X.append()" optimisation (i.e. not only the one for typed lists) should go there. This might(!) depend on (or at least benefit from) splitting up the type analysis phase (as discussed before, looks like we still need a ticket for this), so that we can recognise the declared/builtin names when running the transform, and can handle the final return values appropriately (which are sometimes boolean ints instead of (None) Python objects, for example) *after* switching to the corresponding C-API function. Opinions? Stefan From dalcinl at gmail.com Mon Mar 9 13:24:14 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 9 Mar 2009 10:24:14 -0200 Subject: [Cython] Better call optimisation In-Reply-To: <36891.213.61.181.86.1236601071.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <36891.213.61.181.86.1236601071.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: On Mon, Mar 9, 2009 at 10:17 AM, Stefan Behnel wrote: > > My suggestion is to remove the current support for optimised methods of > builtins, and to write a dedicated transform that intercepts the tree on > call nodes and that can then handle all cases in explicit code. > > Opinions? > Definitely +1 for me ... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Mon Mar 9 13:50:19 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 9 Mar 2009 10:50:19 -0200 Subject: [Cython] too late to review and push patch for better handling of integral C types ? Message-ID: Sending again, former message bounced because of 40k limit. The patch is almost ready, just waiting for review. The only thing I did not take is to remove the usage of PyNumber_Int(). Anyway, Cython already uses it, so the "brokeness" of my patch is similar to the current one :-) .. As I've introduced a __Pyx_PyNumber_Int() function, that is the only place that we should change to implement things as we want. I would really love to get this in for 0.11 ... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 -------------- next part -------------- A non-text attachment was scrubbed... Name: NUMERICTYPES.diff Type: application/octet-stream Size: 18755 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090309/a5fe7d7f/attachment-0001.obj From dalcinl at gmail.com Mon Mar 9 13:51:20 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 9 Mar 2009 10:51:20 -0200 Subject: [Cython] too late to review and push patch for better handling of integral C types ? In-Reply-To: References: Message-ID: And here you have the testcase... On Mon, Mar 9, 2009 at 10:50 AM, Lisandro Dalcin wrote: > Sending again, former message bounced because of 40k limit. > > The patch is almost ready, just waiting for review. The only thing I > did not take is to remove the usage of PyNumber_Int(). Anyway, Cython > already uses it, so the "brokeness" of my patch is similar to the > current one :-) .. As I've introduced a __Pyx_PyNumber_Int() function, > that is the only place that we should change to implement things as we > want. > > I would really love to get this in for 0.11 ... > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 -------------- next part -------------- A non-text attachment was scrubbed... Name: c_int_types.pyx Type: application/octet-stream Size: 16020 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090309/f5db079c/attachment.obj From dagss at student.matnat.uio.no Mon Mar 9 14:49:51 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 09 Mar 2009 14:49:51 +0100 Subject: [Cython] Buffer syntax/array primitive type In-Reply-To: <49B50536.9020708@molden.no> References: <22684a329c8ff577025d36e76274817f.squirrel@webmail.uio.no> <49B50536.9020708@molden.no> Message-ID: <49B51E7F.9090409@student.matnat.uio.no> Sturla Molden wrote: > On 3/8/2009 12:06 AM, Robert Bradshaw wrote: > > >> Very interesting, and I like the syntax for the most part (int >> [:,:,"fortran"] seems a bit odd to me, perhaps mode should be a >> keyword). >> > > I wonder if you would consider > > cdef int[::,:] arr > > to indicate that a dimension might have strides in the first dimension > different from 1. And similar > > cdef int[:,::] arr > cdef int[::,::] arr > > That would allow you to ignore strides and speed up indexing for most > arrays. The most common case is unit strides. > Hmm. Nice idea. I would turn it the other way around though; cdef int[:,::1] for C ordering. Note the explicit "1" stride; I think "::" should always mean the same as ":" or there will be confusion. I think the "faster, more special case" route should always require more syntax knowledge than the "marginally slower generic case". The typical fast access will then look like cdef int[0:,0::1] Next: Would it be possible to incorporate other modes in a similar way? cdef int[::contiguous,::contiguous,::1] could mean C-contiguous 3D-array? What about full mode (i.e. where one on an arbitrary dimension can have pointer dereferences...) perhaps int[::(1,ptr),::contiguous,::1] to get a layout where the first dimension in a contiguous array of pointers to C-ordered 2D contiguous data? :-) All in all, the buffer PEP contains huge flexibility, and locking that flexibility down means more speed. Whether anyone will use it is another issue :-) I suppose int[:,::1] is enough for now. Long-term wishful thinking digression: In typical applications it may be that the compiler can make the decision as to whether one should copy or access in strided mode. For instance: def foo(int[:,:] arr): ... foo(myarr) # Contiguous use foo(myarr[:,::-2]) # Non-contiguous use Here, the compiler has essentially three options: a) "What it says" b) Wrap foo in a copy-in, copy-out wrapper, and also skip that wrapper in the middle line, and indeed if all uses use a contiguous array, skip the wrapper c) Generate one contiguous and one non-contiguous This is just to show the lengths it would be possible to go to. It also gives me an eerie feeling that replacing Fortran (which had had ages to think about stuff like this) will not be easy... Dag Sverre From stefan_ml at behnel.de Mon Mar 9 15:21:07 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 9 Mar 2009 15:21:07 +0100 (CET) Subject: [Cython] too late to review and push patch for better handling of integral C types ? In-Reply-To: References: Message-ID: <35193.213.61.181.86.1236608467.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Lisandro Dalcin wrote: > The patch is almost ready, just waiting for review. > [...] > I would really love to get this in for 0.11 ... Without having looked at it: we are past the RC, so unless someone finds the time for a thorough review and test, I would hate breaking 0.11 right with the first release. Stefan From dagss at student.matnat.uio.no Mon Mar 9 18:34:00 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 09 Mar 2009 18:34:00 +0100 Subject: [Cython] too late to review and push patch for better handling of integral C types ? Message-ID: <3319468485.617624@smtp.netcom.no> +1; there's plenty of bugs which are not fixed and will not be fixed for 0.11; the RC should only point out regressions, things that used to work. There's nothing stopping us from putting out 0.11.1 in another week's time. Dag Sverre Seljebotn -----Original Message----- From: "Stefan Behnel" Date: Monday, Mar 9, 2009 3:21 pm Subject: Re: [Cython] too late to review and push patch for better handling of integral C types ? To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net Lisandro Dalcin wrote: > The patch is almost ready, just waiting for review. > [...] > I would really love to get this in for 0.11 ... > >Without having looked at it: we are past the RC, so unless someone finds the time for a thorough review and test, I would hate breaking 0.11 right with the first release. > >Stefan > >_______________________________________________ >Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > From robertwb at math.washington.edu Mon Mar 9 18:39:33 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 9 Mar 2009 10:39:33 -0700 Subject: [Cython] too late to review and push patch for better handling of integral C types ? In-Reply-To: <35193.213.61.181.86.1236608467.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <35193.213.61.181.86.1236608467.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: On Mar 9, 2009, at 7:21 AM, Stefan Behnel wrote: > Lisandro Dalcin wrote: >> The patch is almost ready, just waiting for review. >> [...] >> I would really love to get this in for 0.11 ... > > Without having looked at it: we are past the RC, so unless someone > finds > the time for a thorough review and test, I would hate breaking 0.11 > right > with the first release. This could be the first patch for 0.11.1, but let's finally get 0.11 out the door. Unless someone reports in the next 24 hours or so that somethings horribly broken, that's the plan. - Robert From stefan_ml at behnel.de Mon Mar 9 18:50:42 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 09 Mar 2009 18:50:42 +0100 Subject: [Cython] Cython 0.11 release candidate In-Reply-To: References: Message-ID: <49B556F2.1020008@behnel.de> Hi, thanks for the report. Neal Becker wrote: > Running on Fedora f10: > > running tests against Cython 0.11.rc > Python 2.5.2 (r252:60911, Sep 30 2008, 15:42:03) > [GCC 4.3.2 20080917 (Red Hat 4.3.2-4)] > > Traceback (most recent call last): > File "runtests.py", line 559, in > pyxbuild_dir=os.path.join(WORKDIR, "support")) > File "/home/nbecker/RPM/BUILD/Cython-0.11.rc/pyximport/pyxbuild.py", line > 26, in pyx_to_dll > assert os.path.exists(filename) > AssertionError I don't think this qualifies as release critical. Could you file a bug report against 0.11.1 and provide some more information? You seem to be using an RPM, is this during the RPM build (running the tests for some reason) or did you start the tests after installing the RPM, maybe from a different directory? Please add a couple things like that to make this reproducible or at least traceable. You may need a login for trac, just send a mail to Robert Bradshaw containing a htpasswd entry. Stefan From dagss at student.matnat.uio.no Mon Mar 9 19:47:52 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 09 Mar 2009 19:47:52 +0100 Subject: [Cython] Better call optimisation In-Reply-To: <36891.213.61.181.86.1236601071.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <36891.213.61.181.86.1236601071.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49B56458.4030505@student.matnat.uio.no> Stefan Behnel wrote: > Hi, > > there are a couple of tickets related to call optimisations. > > http://trac.cython.org/cython_trac/ticket/3 > http://trac.cython.org/cython_trac/ticket/131 > http://trac.cython.org/cython_trac/ticket/166 > > The getattr3() problem also relates to other cases, such as the fact that > we can't currently optimise > > cdef list l = [...] > l.sort() > > into a call to PyList_Sort(), as this would break calls like > > l.sort(key=...) > > My suggestion is to remove the current support for optimised methods of > builtins, and to write a dedicated transform that intercepts the tree on > call nodes and that can then handle all cases in explicit code. This can > include, for example, adding a None check if required, or handling > different call signatures only some of which can be optimised, maybe even > by changing keyword arguments into positional arguments of a matching > C-API function. > > I imagine that it would use some kind of dispatch mechanism on > > (obj_type, obj_attribute_name) > > and on builtin function names. I also think that the current generic > "X.append()" optimisation (i.e. not only the one for typed lists) should > go there. Sounds good! One thing: I think it would be good if the output is something very targeted (i.e. not simply new SimpleCallNodes or similar). From the refnanny it seemed that very many CPython API calls used the same pattern, so perhaps something along the lines of a PythonApiCall-node, perhaps with a NoneCheckNode as its "subject", and so on. (Some of the resulting operations could warrant their own node, while others are similar enough to not warrant new classes for each -- the point is that I think it should be more "targeted" than new SimpleCallNodes). > This might(!) depend on (or at least benefit from) splitting up the type > analysis phase (as discussed before, looks like we still need a ticket for > this), so that we can recognise the declared/builtin names when running > the transform, and can handle the final return values appropriately (which > are sometimes boolean ints instead of (None) Python objects, for example) > *after* switching to the corresponding C-API function. This is a going to cause at least as much trouble as the new temps (though that is going to be worth it in the end, and so could this). At least myself I am not decided on how I'd like to see such a split -- I know I advocated it a year ago, but now "first names, then expressions" seems a bit suboptimal to me. "First declare possible types, then transform, then resolve and insert coercion nodes and raise any errors" seems promising but I'm still unsure about it. If there's a chance of a Cython workshop during the coming year I'd say these kind of matters (and the actual coding) can be resolved more easily that way. Otherwise it seems to stand between one person with a bright idea doing it all alone, or one of those 100-emails-threads converging very slowly towards the right solution... But I may be overestimating the complexity, if you have a clear idea then I won't object to it. -- Dag Sverre From ndbecker2 at gmail.com Mon Mar 9 19:49:05 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 09 Mar 2009 14:49:05 -0400 Subject: [Cython] Cython 0.11 release candidate References: <49B556F2.1020008@behnel.de> Message-ID: Stefan Behnel wrote: > Hi, > > thanks for the report. > > Neal Becker wrote: >> Running on Fedora f10: >> >> running tests against Cython 0.11.rc >> Python 2.5.2 (r252:60911, Sep 30 2008, 15:42:03) >> [GCC 4.3.2 20080917 (Red Hat 4.3.2-4)] >> >> Traceback (most recent call last): >> File "runtests.py", line 559, in >> pyxbuild_dir=os.path.join(WORKDIR, "support")) >> File "/home/nbecker/RPM/BUILD/Cython-0.11.rc/pyximport/pyxbuild.py", >> line >> 26, in pyx_to_dll >> assert os.path.exists(filename) >> AssertionError > > I don't think this qualifies as release critical. Could you file a bug > report against 0.11.1 and provide some more information? You seem to be > using an RPM, is this during the RPM build (running the tests for some > reason) or did you start the tests after installing the RPM, maybe from a > different directory? Please add a couple things like that to make this > reproducible or at least traceable. > > You may need a login for trac, just send a mail to Robert Bradshaw > containing a htpasswd entry. > > Stefan I am the maintainer for cython on fedora. This was during the check phase (%check). If the test doesn't pass, at least I'll have to turn off the checking, which would be unfortunate. Are the tests expected to pass? From robertwb at math.washington.edu Mon Mar 9 19:56:52 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 9 Mar 2009 11:56:52 -0700 Subject: [Cython] Cython 0.11 release candidate In-Reply-To: References: <49B556F2.1020008@behnel.de> Message-ID: <994EFBF5-7848-4929-A5B4-B3A050386C6D@math.washington.edu> It's this test here: out_fname = pyx_to_dll("foo.pyx") It looks like we don't even ship a foo.pyx to test with, though I'm not sure why I haven't seen this error before... On Mar 9, 2009, at 11:49 AM, Neal Becker wrote: > Stefan Behnel wrote: > >> Hi, >> >> thanks for the report. >> >> Neal Becker wrote: >>> Running on Fedora f10: >>> >>> running tests against Cython 0.11.rc >>> Python 2.5.2 (r252:60911, Sep 30 2008, 15:42:03) >>> [GCC 4.3.2 20080917 (Red Hat 4.3.2-4)] >>> >>> Traceback (most recent call last): >>> File "runtests.py", line 559, in >>> pyxbuild_dir=os.path.join(WORKDIR, "support")) >>> File "/home/nbecker/RPM/BUILD/Cython-0.11.rc/pyximport/ >>> pyxbuild.py", >>> line >>> 26, in pyx_to_dll >>> assert os.path.exists(filename) >>> AssertionError >> >> I don't think this qualifies as release critical. Could you file a >> bug >> report against 0.11.1 and provide some more information? You seem >> to be >> using an RPM, is this during the RPM build (running the tests for >> some >> reason) or did you start the tests after installing the RPM, maybe >> from a >> different directory? Please add a couple things like that to make >> this >> reproducible or at least traceable. >> >> You may need a login for trac, just send a mail to Robert Bradshaw >> containing a htpasswd entry. >> >> Stefan > > I am the maintainer for cython on fedora. This was during the > check phase > (%check). If the test doesn't pass, at least I'll have to turn off > the > checking, which would be unfortunate. > > Are the tests expected to pass? > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From stefan_ml at behnel.de Mon Mar 9 19:58:11 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 09 Mar 2009 19:58:11 +0100 Subject: [Cython] Cython 0.11 release candidate In-Reply-To: References: <49B556F2.1020008@behnel.de> Message-ID: <49B566C3.8010203@behnel.de> Neal Becker wrote: > I am the maintainer for cython on fedora. Ah, ok, that makes it more important. > This was during the check phase (%check). I assume this happens after building? > If the test doesn't pass, at least I'll have to turn off the > checking, which would be unfortunate. > > Are the tests expected to pass? Yes. This looks like the refnanny can't be built. You can disable it using "--no-refannny", although I would prefer a real fix. Did you check if the directory in which you run the tests has a file "Cython/Runtime/refnanny.pyx"? Actually, when Cython is tested after building, the refnanny should already be ready for use, so there shouldn't be a reason to build it again. Stefan From stefan_ml at behnel.de Mon Mar 9 20:02:19 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 09 Mar 2009 20:02:19 +0100 Subject: [Cython] Cython 0.11 release candidate In-Reply-To: <49B566C3.8010203@behnel.de> References: <49B556F2.1020008@behnel.de> <49B566C3.8010203@behnel.de> Message-ID: <49B567BB.3020102@behnel.de> Stefan Behnel wrote: > Did you check if the > directory in which you run the tests has a file "Cython/Runtime/refnanny.pyx"? Ah, does this patch fix it? diff -r 474a2405291b MANIFEST.in --- a/MANIFEST.in Sun Mar 08 14:18:27 2009 +0100 +++ b/MANIFEST.in Mon Mar 09 20:01:52 2009 +0100 @@ -7,6 +7,7 @@ include cython.py include Cython/Compiler/Lexicon.pickle include Cython/Includes/*.pxd +recursive-include Cython *.pyx include Doc/* include Demos/*.pyx Stefan From dagss at student.matnat.uio.no Mon Mar 9 20:19:25 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 09 Mar 2009 20:19:25 +0100 Subject: [Cython] Cython 0.11 release candidate In-Reply-To: <49B566C3.8010203@behnel.de> References: <49B556F2.1020008@behnel.de> <49B566C3.8010203@behnel.de> Message-ID: <49B56BBD.5010303@student.matnat.uio.no> Stefan Behnel wrote: > Actually, when Cython is tested after building, the refnanny should already > be ready for use, so there shouldn't be a reason to build it again. Unless something's changed since I did it, runtests.py always maintains its own compilation of refnanny in BUILD (to make refnanny development easier). It doesn't rebuild it if it is already present in BUILD with a newer timestamp than source though. -- Dag Sverre From robertwb at math.washington.edu Mon Mar 9 20:22:59 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 9 Mar 2009 12:22:59 -0700 Subject: [Cython] Better call optimisation In-Reply-To: <36891.213.61.181.86.1236601071.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <36891.213.61.181.86.1236601071.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: On Mar 9, 2009, at 5:17 AM, Stefan Behnel wrote: > Hi, > > there are a couple of tickets related to call optimisations. > > http://trac.cython.org/cython_trac/ticket/3 > http://trac.cython.org/cython_trac/ticket/131 > http://trac.cython.org/cython_trac/ticket/166 > > The getattr3() problem also relates to other cases, such as the > fact that > we can't currently optimise > > cdef list l = [...] > l.sort() > > into a call to PyList_Sort(), as this would break calls like > > l.sort(key=...) > > My suggestion is to remove the current support for optimised > methods of > builtins, and to write a dedicated transform that intercepts the > tree on > call nodes and that can then handle all cases in explicit code. > This can > include, for example, adding a None check if required, or handling > different call signatures only some of which can be optimised, > maybe even > by changing keyword arguments into positional arguments of a matching > C-API function. This sounds like a good idea. Another option would be to support polymorphism (which would be good to do, at least for external C functions). > I imagine that it would use some kind of dispatch mechanism on > > (obj_type, obj_attribute_name) > > and on builtin function names. I also think that the current generic > "X.append()" optimisation (i.e. not only the one for typed lists) > should > go there. > > This might(!) depend on (or at least benefit from) splitting up the > type > analysis phase (as discussed before, looks like we still need a > ticket for > this), so that we can recognise the declared/builtin names when > running > the transform, and can handle the final return values appropriately > (which > are sometimes boolean ints instead of (None) Python objects, for > example) > *after* switching to the corresponding C-API function. > > Opinions? My one concern is that it would make type inference significantly more difficult by deferring this to a later phase (especially thinking in terms of methods that don't return PyObject*). - Robert From ndbecker2 at gmail.com Mon Mar 9 20:26:18 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 09 Mar 2009 15:26:18 -0400 Subject: [Cython] Cython 0.11 release candidate References: <49B556F2.1020008@behnel.de> <49B566C3.8010203@behnel.de> <49B56BBD.5010303@student.matnat.uio.no> Message-ID: Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Actually, when Cython is tested after building, the refnanny should >> already be ready for use, so there shouldn't be a reason to build it >> again. > > Unless something's changed since I did it, runtests.py always maintains > its own compilation of refnanny in BUILD (to make refnanny development > easier). It doesn't rebuild it if it is already present in BUILD with a > newer timestamp than source though. > One thing that might be a clue: The rpm build procedure will do: python setup.py build python setup.py install -O1 --skip-build --root $RPM_BUILD_ROOT python runtests.py From stefan_ml at behnel.de Mon Mar 9 20:30:52 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 09 Mar 2009 20:30:52 +0100 Subject: [Cython] Better call optimisation In-Reply-To: <49B56458.4030505@student.matnat.uio.no> References: <36891.213.61.181.86.1236601071.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B56458.4030505@student.matnat.uio.no> Message-ID: <49B56E6C.8000404@behnel.de> Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: > One thing: I think it would be good if the output is something very > targeted (i.e. not simply new SimpleCallNodes or similar). From the > refnanny it seemed that very many CPython API calls used the same > pattern, so perhaps something along the lines of a PythonApiCall-node, > perhaps with a NoneCheckNode as its "subject", and so on. Fine with me. I also thought about a "NoneCheckNode" when I wrote this. That would be a very simple and explicit thing, and also very easy to handle (or optimise away) in later transformations and during code generation. A PythonApiCall node might be a good idea (needs some more thinking) and would relieve us from having to generate all sorts of Entry's & friends. > Some of the resulting operations could warrant their own node Yes, I gave an example with the int vs. None return value. There's also the issue that many methods of builtins have their own specific exceptions when passing None values. This needs to be handled, and a PythonApiCall node would be a good place to do it. >> This might(!) depend on (or at least benefit from) splitting up the type >> analysis phase (as discussed before, looks like we still need a ticket for >> this), so that we can recognise the declared/builtin names when running >> the transform, and can handle the final return values appropriately (which >> are sometimes boolean ints instead of (None) Python objects, for example) >> *after* switching to the corresponding C-API function. > > This is a going to cause at least as much trouble as the new temps > (though that is going to be worth it in the end, and so could this). At > least myself I am not decided on how I'd like to see such a split -- I > know I advocated it a year ago, but now "first names, then expressions" > seems a bit suboptimal to me. "First declare possible types, then > transform, then resolve and insert coercion nodes and raise any errors" > seems promising but I'm still unsure about it. Yes, this needs some discussion. For 0.12, we may just try the optimisation and see how far it takes us without this split. That will also make it clearer if, why and how this needs to be done. > If there's a chance of a Cython workshop during the coming year I'd say > these kind of matters (and the actual coding) can be resolved more > easily that way. Otherwise it seems to stand between one person with a > bright idea doing it all alone, or one of those 100-emails-threads > converging very slowly towards the right solution... ;) Certainly not a task for a GSoC student. Stefan From stefan_ml at behnel.de Mon Mar 9 20:32:26 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 09 Mar 2009 20:32:26 +0100 Subject: [Cython] Cython 0.11 release candidate In-Reply-To: References: <49B556F2.1020008@behnel.de> <49B566C3.8010203@behnel.de> <49B56BBD.5010303@student.matnat.uio.no> Message-ID: <49B56ECA.3040106@behnel.de> Neal Becker wrote: > Dag Sverre Seljebotn wrote: > >> Stefan Behnel wrote: >>> Actually, when Cython is tested after building, the refnanny should >>> already be ready for use, so there shouldn't be a reason to build it >>> again. >> Unless something's changed since I did it, runtests.py always maintains >> its own compilation of refnanny in BUILD (to make refnanny development >> easier). It doesn't rebuild it if it is already present in BUILD with a >> newer timestamp than source though. > > One thing that might be a clue: > > The rpm build procedure will do: > > python setup.py build > python setup.py install -O1 --skip-build --root $RPM_BUILD_ROOT > python runtests.py Yes, I think the patch that I just pushed to cython-devel might fix this. Please give it a try. Stefan From stefan_ml at behnel.de Mon Mar 9 22:05:48 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 09 Mar 2009 22:05:48 +0100 Subject: [Cython] too late to review and push patch for better handling of integral C types ? In-Reply-To: References: Message-ID: <49B584AC.6040506@behnel.de> Lisandro Dalcin wrote: > Sending again, former message bounced because of 40k limit. > > The patch is almost ready, just waiting for review. The only thing I > did not take is to remove the usage of PyNumber_Int(). Anyway, Cython > already uses it, so the "brokeness" of my patch is similar to the > current one :-) .. As I've introduced a __Pyx_PyNumber_Int() function, > that is the only place that we should change to implement things as we > want. I skipped through your patch a little. It's a fairly involved change that requires some testing, especially on different architectures and platforms. It might well be that the user experience is positive in that it helps in finding bugs and making user code more portable. My current gut feeling is that code that depends on the old behaviour here is broken anyway and just wasn't fixed yet because no-one noticed. But in any case, I would like to avoid breaking code for a micro (bug-fix!) release, unless there is at least some evidence that users appreciate it. So, if anyone has code that does a bit of integer conversion between Python and C space, especially when working on a non-Intel or 64 bit platform, please give this patch a try. Stefan From ndbecker2 at gmail.com Tue Mar 10 00:12:23 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 09 Mar 2009 19:12:23 -0400 Subject: [Cython] Cython 0.11 release candidate References: <49B556F2.1020008@behnel.de> <49B566C3.8010203@behnel.de> <49B56BBD.5010303@student.matnat.uio.no> <49B56ECA.3040106@behnel.de> Message-ID: Stefan Behnel wrote: > > Neal Becker wrote: >> Dag Sverre Seljebotn wrote: >> >>> Stefan Behnel wrote: >>>> Actually, when Cython is tested after building, the refnanny should >>>> already be ready for use, so there shouldn't be a reason to build it >>>> again. >>> Unless something's changed since I did it, runtests.py always maintains >>> its own compilation of refnanny in BUILD (to make refnanny development >>> easier). It doesn't rebuild it if it is already present in BUILD with a >>> newer timestamp than source though. >> >> One thing that might be a clue: >> >> The rpm build procedure will do: >> >> python setup.py build >> python setup.py install -O1 --skip-build --root $RPM_BUILD_ROOT >> python runtests.py > > Yes, I think the patch that I just pushed to cython-devel might fix this. > Please give it a try. > > Stefan Sorry, I don't follow devel that closely. How do I get the patch, or an updated source? From ndbecker2 at gmail.com Tue Mar 10 02:06:34 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 09 Mar 2009 21:06:34 -0400 Subject: [Cython] Cython 0.11 release candidate References: <49B556F2.1020008@behnel.de> <49B566C3.8010203@behnel.de> <49B567BB.3020102@behnel.de> Message-ID: Stefan Behnel wrote: > > Stefan Behnel wrote: >> Did you check if the >> directory in which you run the tests has a file >> "Cython/Runtime/refnanny.pyx"? > > Ah, does this patch fix it? > > diff -r 474a2405291b MANIFEST.in > --- a/MANIFEST.in Sun Mar 08 14:18:27 2009 +0100 > +++ b/MANIFEST.in Mon Mar 09 20:01:52 2009 +0100 > @@ -7,6 +7,7 @@ > include cython.py > include Cython/Compiler/Lexicon.pickle > include Cython/Includes/*.pxd > +recursive-include Cython *.pyx > include Doc/* > > include Demos/*.pyx > > Stefan No, this patch made no difference. python setup.py install -O1 --skip-build --root ~/RPM/BUILDROOT/Cython-0.11-1rc.fc1\ 0.x86_64/ Compiling module Cython.Plex.Scanners ... Compiling module Cython.Compiler.Scanning ... Compiling module Cython.Compiler.Parsing ... Compiling module Cython.Compiler.Visitor ... Compiling module Cython.Runtime.refpython setup.py build Compiling module Cython.Plex.Scanners ... Compiling module Cython.Compiler.Scanning ... Compiling module Cython.Compiler.Parsing ... Compiling module Cython.Compiler.Visitor ... Compiling module Cython.Runtime.refnanny ... ERROR: [Errno 2] No such file or directory: u'/home/nbecker/RPM/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx' Extension module compilation failed, using plain Python implementation running build running build_py running build_scripts python setup.py install -O1 --skip-build --root ~/RPM/BUILDROOT/Cython-0.11-1rc.fc10.x86_64/ Compiling module Cython.Plex.Scanners ... nanny ... ERROR: [Errno 2] No such file or directory: u'/home/nbecker/RPM/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx' Extension module compilation failed, using plain Python implementation running install running install_lib writing byte-compilation script '/tmp/tmpTN0AmD.py' /usr/bin/python -O /tmp/tmpTN0AmD.py removing /tmp/tmpTN0AmD.py running install_scripts changing mode of /home/nbecker/RPM/BUILDROOT/Cython-0.11-1rc.fc10.x86_64/usr/bin/cython to 755 running install_egg_info Removing /home/nbecker/RPM/BUILDROOT/Cython-0.11-1rc.fc10.x86_64/usr/lib/python2.5/site- packages/Cython-0.11.rc-py2.5.egg-info Writing /home/nbecker/RPM/BUILDROOT/Cython-0.11-1rc.fc10.x86_64/usr/lib/python2.5/site- packages/Cython-0.11.rc-py2.5.egg-info $ python runtests.py Running tests against Cython 0.11.rc Python 2.5.2 (r252:60911, Sep 30 2008, 15:42:03) [GCC 4.3.2 20080917 (Red Hat 4.3.2-4)] Traceback (most recent call last): File "runtests.py", line 559, in pyxbuild_dir=os.path.join(WORKDIR, "support")) File "/home/nbecker/RPM/BUILD/Cython-0.11.rc/pyximport/pyxbuild.py", line 26, in pyx_to_dll assert os.path.exists(filename) AssertionError From robertwb at math.washington.edu Tue Mar 10 03:50:33 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 9 Mar 2009 19:50:33 -0700 Subject: [Cython] Cython 0.11 release candidate In-Reply-To: References: Message-ID: <872A0AAF-4DBA-4471-A081-9013738CA234@math.washington.edu> On Mar 8, 2009, at 7:19 AM, Neal Becker wrote: > Robert Bradshaw wrote: > >> Finally, it looks like we're on the brink of releasing 0.11. Try out >> the release candidate at http://cython.org/Cython-0.11.rc.tar.gz >> >> - Robert > > Running on Fedora f10: > > running tests against Cython 0.11.rc > Python 2.5.2 (r252:60911, Sep 30 2008, 15:42:03) > [GCC 4.3.2 20080917 (Red Hat 4.3.2-4)] > > Traceback (most recent call last): > File "runtests.py", line 559, in > pyxbuild_dir=os.path.join(WORKDIR, "support")) > File "/home/nbecker/RPM/BUILD/Cython-0.11.rc/pyximport/ > pyxbuild.py", line > 26, in pyx_to_dll > assert os.path.exists(filename) > AssertionError OK, could you try pulling from http://hg.cython.org/cython-devel/ again? (Or alternatively I've updated http://cython.org/ Cython-0.11.rc.tar.gz ). It won't fix your error, but should help diagnose it better. - Robert From stefan_ml at behnel.de Tue Mar 10 10:23:19 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 10 Mar 2009 10:23:19 +0100 Subject: [Cython] Cython 0.11 release candidate In-Reply-To: References: <49B556F2.1020008@behnel.de> <49B566C3.8010203@behnel.de> <49B567BB.3020102@behnel.de> Message-ID: <49B63187.2060009@behnel.de> Neal Becker wrote: > python setup.py install -O1 --skip-build --root > ~/RPM/BUILDROOT/Cython-0.11-1rc.fc1\ > 0.x86_64/ > Compiling module Cython.Plex.Scanners ... > Compiling module Cython.Compiler.Scanning ... > Compiling module Cython.Compiler.Parsing ... > Compiling module Cython.Compiler.Visitor ... > Compiling module Cython.Runtime.refpython setup.py build This looks weird, are you sure you copied this correctly? > Compiling module Cython.Plex.Scanners ... > Compiling module Cython.Compiler.Scanning ... > Compiling module Cython.Compiler.Parsing ... > Compiling module Cython.Compiler.Visitor ... > Compiling module Cython.Runtime.refnanny ... > ERROR: [Errno 2] No such file or directory: > u'/home/nbecker/RPM/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx' > Extension module compilation failed, using plain Python implementation > running build > running build_py > running build_scripts > > python setup.py install -O1 --skip-build --root > ~/RPM/BUILDROOT/Cython-0.11-1rc.fc10.x86_64/ > Compiling module Cython.Plex.Scanners ... And this seems to be where things continue: > nanny ... > ERROR: [Errno 2] No such file or directory: > u'/home/nbecker/RPM/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx' > Extension module compilation failed, using plain Python implementation > running install > [...] I actually think that it's the "install" that hits us here. The refnanny.pyx isn't copied on installation, as we expect the .so file to be built already. However, you call "install" with the "--skip-build" option, which prevents the build of the shared library, thus preventing the refnanny from being available at all. We might just declare the refnanny.pyx file as package data, so that it's copied into the installation. It doesn't hurt to have it lying next to the refnanny.so file. Here's another patch. BTW, you can pass the "--no-cython-compile" option to your "setup.py install" call to keep it from compiling certain source files multiple times. It's well enough if it does that once on "setup.py build". Stefan -------------- next part -------------- A non-text attachment was scrubbed... Name: package_data.patch Type: text/x-patch Size: 1479 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090310/a338b292/attachment.bin From ndbecker2 at gmail.com Tue Mar 10 13:51:32 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 10 Mar 2009 08:51:32 -0400 Subject: [Cython] Cython 0.11 release candidate References: <49B556F2.1020008@behnel.de> <49B566C3.8010203@behnel.de> <49B567BB.3020102@behnel.de> <49B63187.2060009@behnel.de> Message-ID: Both patches applied. Still no good. I am sending the whole build transcript. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: cython.build Url: http://codespeak.net/pipermail/cython-dev/attachments/20090310/878404a2/attachment-0001.diff From stefan_ml at behnel.de Tue Mar 10 14:29:31 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 10 Mar 2009 14:29:31 +0100 (CET) Subject: [Cython] Cython 0.11 release candidate In-Reply-To: References: <49B556F2.1020008@behnel.de> <49B566C3.8010203@behnel.de> <49B567BB.3020102@behnel.de> <49B63187.2060009@behnel.de> Message-ID: <40045.213.61.181.86.1236691771.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Hi, Neal Becker wrote: > Both patches applied. Still no good. I am sending the whole build > transcript. can't check right now, but did you verify that the tar ball has the file Cython/Runtime/refnanny.pyx? The first patch might have prevented it from getting into the release. Robert, in case it's missing, could you provide Neal with a new tar ball from the latest cython-devel? Stefan From stefan_ml at behnel.de Tue Mar 10 14:31:13 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 10 Mar 2009 14:31:13 +0100 (CET) Subject: [Cython] Cython 0.11 release candidate In-Reply-To: <40045.213.61.181.86.1236691771.squirrel@groupware.dvs.informatik.tu-d armstadt.de> References: <49B556F2.1020008@behnel.de> <49B566C3.8010203@behnel.de> <49B567BB.3020102@behnel.de> <49B63187.2060009@behnel.de> <40045.213.61.181.86.1236691771.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <41995.213.61.181.86.1236691873.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Stefan Behnel wrote: > Cython/Runtime/refnanny.pyx? The first patch might have prevented it from > getting into the release. ... actually not "the first patch", but the bug that I patched with my first change... Stefan From dagss at student.matnat.uio.no Tue Mar 10 14:33:50 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 10 Mar 2009 14:33:50 +0100 Subject: [Cython] Cython 0.11 release candidate In-Reply-To: <40045.213.61.181.86.1236691771.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49B556F2.1020008@behnel.de> <49B566C3.8010203@behnel.de> <49B567BB.3020102@behnel.de> <49B63187.2060009@behnel.de> <40045.213.61.181.86.1236691771.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49B66C3E.6090508@student.matnat.uio.no> Stefan Behnel wrote: > Hi, > > Neal Becker wrote: > >> Both patches applied. Still no good. I am sending the whole build >> transcript. >> > > can't check right now, but did you verify that the tar ball has the file > Cython/Runtime/refnanny.pyx? The first patch might have prevented it from > getting into the release. > The RC tarball has refnanny.pyx present in the right location. Dag Sverre From ndbecker2 at gmail.com Tue Mar 10 15:11:08 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 10 Mar 2009 10:11:08 -0400 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) Message-ID: I grabbed a NEW tar from http://cython.org/Cython-0.11.rc.tar.gz. (I won't comment on the confusion caused by changing the file without changing the name :) Now testing at least starts, but seems to have some errors: (Ignore the RPM build errors: Installed (but unpackaged) file(s) found:, I'll fix those) -------------- next part -------------- A non-text attachment was scrubbed... Name: cython.build.gz Type: application/x-gzip Size: 12174 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090310/5b5a8f35/attachment.bin From dagss at student.matnat.uio.no Tue Mar 10 15:18:11 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 10 Mar 2009 15:18:11 +0100 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: References: Message-ID: <49B676A3.3090405@student.matnat.uio.no> Neal Becker wrote: > I grabbed a NEW tar from http://cython.org/Cython-0.11.rc.tar.gz. (I won't > comment on the confusion caused by changing the file without changing the > name :) > > Now testing at least starts, but seems to have some errors: > (Ignore the > RPM build errors: > Installed (but unpackaged) file(s) found:, > I'll fix those) > The problem is that NumPy appears to be installed on your system (import numpy succeeds), yet the NumPy C header files is not present in the include path. Either set an extra include-path using the INCLUDE environment variable: INCLUDE=/path/to/numpy/headers python runtests.py ... or, drop the NumPy tests from the test suite (as I suppose your Cython RPM doesn't depend on NumPy -- it shouldn't!): python runtests.py ... -x numpy Dag Sverre From dalcinl at gmail.com Tue Mar 10 15:18:08 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 10 Mar 2009 12:18:08 -0200 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: References: Message-ID: I really recomend Cython to include a "setup.cfg" file in the root dir (I mean, alongside the setup.py script) whit at least the lines below: [sdist] force_manifest=1 [install] optimize = 1 - The fist would solve common gotchas of out-of-date MANIFEST and the rpm build problems... - The second would solve the "Installed (but unpackaged) file(s) found", wich is related to the generation of *.pyo files... On Tue, Mar 10, 2009 at 12:11 PM, Neal Becker wrote: > I grabbed a NEW tar from http://cython.org/Cython-0.11.rc.tar.gz. ?(I won't > comment on the confusion caused by changing the file without changing the > name :) > > Now testing at least starts, but seems to have some errors: > (Ignore the > RPM build errors: > ? ?Installed (but unpackaged) file(s) found:, > I'll fix those) > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Tue Mar 10 15:21:54 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 10 Mar 2009 12:21:54 -0200 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: <49B676A3.3090405@student.matnat.uio.no> References: <49B676A3.3090405@student.matnat.uio.no> Message-ID: Dag, from very long time (as far as I can remember), numpy provided a canonical way to get the include files import numpy numpy_include = numpy.get_include() I remember some mails in this list where (Stefan or you?) resisted on automating this... But this has worked even before 1.0.X numpy released... IMHO, this should be automated ... On Tue, Mar 10, 2009 at 12:18 PM, Dag Sverre Seljebotn wrote: > Neal Becker wrote: >> I grabbed a NEW tar from http://cython.org/Cython-0.11.rc.tar.gz. ?(I won't >> comment on the confusion caused by changing the file without changing the >> name :) >> >> Now testing at least starts, but seems to have some errors: >> (Ignore the >> RPM build errors: >> ? ? Installed (but unpackaged) file(s) found:, >> I'll fix those) >> > > The problem is that NumPy appears to be installed on your system (import > numpy succeeds), yet the NumPy C header files is not present in the > include path. > > Either set an extra include-path using the INCLUDE environment variable: > > INCLUDE=/path/to/numpy/headers python runtests.py ... > > or, drop the NumPy tests from the test suite (as I suppose your Cython > RPM doesn't depend on NumPy -- it shouldn't!): > > python runtests.py ... -x numpy > > Dag Sverre > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From ndbecker2 at gmail.com Tue Mar 10 15:22:34 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 10 Mar 2009 10:22:34 -0400 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) References: <49B676A3.3090405@student.matnat.uio.no> Message-ID: Dag Sverre Seljebotn wrote: > Neal Becker wrote: >> I grabbed a NEW tar from http://cython.org/Cython-0.11.rc.tar.gz. (I >> won't comment on the confusion caused by changing the file without >> changing the name :) >> >> Now testing at least starts, but seems to have some errors: >> (Ignore the >> RPM build errors: >> Installed (but unpackaged) file(s) found:, >> I'll fix those) >> > > The problem is that NumPy appears to be installed on your system (import > numpy succeeds), yet the NumPy C header files is not present in the > include path. > > Either set an extra include-path using the INCLUDE environment variable: > > INCLUDE=/path/to/numpy/headers python runtests.py ... > > or, drop the NumPy tests from the test suite (as I suppose your Cython > RPM doesn't depend on NumPy -- it shouldn't!): > > python runtests.py ... -x numpy > > Dag Sverre Here is code used to find the 'pyublas' includes. Something like this could be used to find the numpy includes: def pyublas_inc_path(): from imp import find_module file, pathname, descr = find_module("pyublas") from os.path import join return join(pathname, "..", "include") From stefan_ml at behnel.de Tue Mar 10 15:36:53 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 10 Mar 2009 15:36:53 +0100 (CET) Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: References: Message-ID: <52637.213.61.181.86.1236695813.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Neal Becker wrote: > I grabbed a NEW tar from http://cython.org/Cython-0.11.rc.tar.gz. (I > won't > comment on the confusion caused by changing the file without changing the > name :) Since you don't do it, I will: it's usually a very bad idea to replace a release (even a release candidate) with a modified version without changing the name, as it tends to generate a lot of confusion as to what people are reporting bugs about. I actually know what I'm talking about. ;-) > Now testing at least starts, but seems to have some errors: * warning: left shift count >= width of type I don't get these locally. The problem is in __Pyx_GetItemGeneric, second line: "1ULL << (sizeof(Py_ssize_t)*8)". Looks like that's not 64bit clean. * arrayassign.c: In function "__pyx_pf_11arrayassign_test_literal_list_slice_start_param": arrayassign.c:775: warning: format "%d" expects type "int", but argument 4 has type "Py_ssize_t" arrayassign.cpp: In function "PyObject* __pyx_pf_11arrayassign_test_literal_list_slice_start_end_param(PyObject*, PyObject*, PyObject*)": arrayassign.cpp:1056: warning: format "%d" expects type "int", but argument 4 has type "long int" Same thing, 64 bit issues IMHO. * Error converting Pyrex file to C: ------------------------------------------------------------ ... cdef object[foo2=short unsigned int] x ^ ------------------------------------------------------------ Cython_Compiler_Tests_TestBuffer_TestBufferParsing_test_notype_as_expr1:1:23: Expected: expression Error converting Pyrex file to C: ------------------------------------------------------------ ... cdef object[int, short unsigned int] x ^ ------------------------------------------------------------ Cython_Compiler_Tests_TestBuffer_TestBufferParsing_test_notype_as_expr2:1:23: Expected: expression Error converting Pyrex file to C: ------------------------------------------------------------ ... cdef object[foo=1, 2] x ^ ------------------------------------------------------------ Cython_Compiler_Tests_TestBuffer_TestBufferParsing_test_pos_after_key:1:19: Non-keyword arg following keyword arg *** I remember seeing these, but they went away when rebuilding Cython. That's why I'm surprised to see them in a clean build now... Dag, any idea where those might come from? Stefan From dagss at student.matnat.uio.no Tue Mar 10 15:37:20 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 10 Mar 2009 15:37:20 +0100 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: References: <49B676A3.3090405@student.matnat.uio.no> Message-ID: <49B67B20.7060807@student.matnat.uio.no> Neal Becker wrote: > Dag Sverre Seljebotn wrote: > > >> Neal Becker wrote: >> >>> I grabbed a NEW tar from http://cython.org/Cython-0.11.rc.tar.gz. (I >>> won't comment on the confusion caused by changing the file without >>> changing the name :) >>> >>> Now testing at least starts, but seems to have some errors: >>> (Ignore the >>> RPM build errors: >>> Installed (but unpackaged) file(s) found:, >>> I'll fix those) >>> >>> >> The problem is that NumPy appears to be installed on your system (import >> numpy succeeds), yet the NumPy C header files is not present in the >> include path. >> >> Either set an extra include-path using the INCLUDE environment variable: >> >> INCLUDE=/path/to/numpy/headers python runtests.py ... >> >> or, drop the NumPy tests from the test suite (as I suppose your Cython >> RPM doesn't depend on NumPy -- it shouldn't!): >> >> python runtests.py ... -x numpy >> >> Dag Sverre >> > > Here is code used to find the 'pyublas' includes. Something like this could > be used to find the numpy includes: > Yep, but it won't happen in this release. See my upcoming response to Lisandro. But I think the Cython RPM shouldn't depend on the NumPy RPM, or be tested differently if NumPy is installed or not in the RPM build? Cython doesn't depend on NumPy in any way, there's just a test case sitting there more for convenience than anything else...the testcase doesn't add anything to correctness testing of Cython either. Dag Sverre From stefan_ml at behnel.de Tue Mar 10 15:44:22 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 10 Mar 2009 15:44:22 +0100 (CET) Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: References: Message-ID: <61769.213.61.181.86.1236696262.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Neal Becker wrote: > I grabbed a NEW tar from http://cython.org/Cython-0.11.rc.tar.gz. > [...] > Now testing at least starts, but seems to have some errors: BTW, Neal, getting the RPM build working on a 64bit machine appears to be the perfect test harness for the 0.11 release. Thanks for speaking up in the right moment! Stefan From dagss at student.matnat.uio.no Tue Mar 10 15:45:36 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 10 Mar 2009 15:45:36 +0100 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: References: <49B676A3.3090405@student.matnat.uio.no> Message-ID: <49B67D10.4060308@student.matnat.uio.no> Lisandro Dalcin wrote: > Dag, from very long time (as far as I can remember), numpy provided a > canonical way to get the include files > > import numpy > numpy_include = numpy.get_include() > > I remember some mails in this list where (Stefan or you?) resisted on > automating this... But this has worked even before 1.0.X numpy > released... > > IMHO, this should be automated ... > I'll resist again, but I don't have a strong opinion, so if you can persuade Stefan and provide a patch I'm ok with it. a) One could add it to runtests.py. But that is primarily a tool for development, and the most important thing is that things are nice and explicit and that nothing magic happens under the hood. It's not supposed to "just work", rather the opposite! :-) b) One could add it to Cython distutils, so that if NumPy is installed, Cython modules always have its include path added. I don't really have an opinion here, but my feeling is that that would be a rather strange and unexpected special treatment of this one library. It would be better to push for a *proper* solution: Either have NumPy install a symlink in /usr/include or the Python headers, or have Python and/or distutils keep som kind of include path registry for include paths to installed extensions. Dag Sverre From stefan_ml at behnel.de Tue Mar 10 15:53:38 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 10 Mar 2009 15:53:38 +0100 (CET) Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: <49B67D10.4060308@student.matnat.uio.no> References: <49B676A3.3090405@student.matnat.uio.no> <49B67D10.4060308@student.matnat.uio.no> Message-ID: <40279.213.61.181.86.1236696818.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: >> Dag, from very long time (as far as I can remember), numpy provided a >> canonical way to get the include files >> >> import numpy >> numpy_include = numpy.get_include() >> >> I remember some mails in this list where (Stefan or you?) resisted on >> automating this... But this has worked even before 1.0.X numpy >> released... >> >> IMHO, this should be automated ... >> > I'll resist again, but I don't have a strong opinion, so if you can > persuade Stefan and provide a patch I'm ok with it. > > a) One could add it to runtests.py. But that is primarily a tool for > development, and the most important thing is that things are nice and > explicit and that nothing magic happens under the hood. It's not > supposed to "just work", rather the opposite! :-) I'm fine with using it in runtest.py. We test for NumPy anyway when setting up the test suite, so we can just as well go all the way. Requiring a user to set up a specific build environment just to run the test suite is wrong. I'm against adding any special handling to the distutils modules, though. Having NumPy installed doesn't imply that you want to use its header files. Stefan From ndbecker2 at gmail.com Tue Mar 10 15:56:35 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 10 Mar 2009 10:56:35 -0400 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) References: <49B676A3.3090405@student.matnat.uio.no> <49B67B20.7060807@student.matnat.uio.no> Message-ID: Dag Sverre Seljebotn wrote: > Neal Becker wrote: >> Dag Sverre Seljebotn wrote: >> >> >>> Neal Becker wrote: >>> >>>> I grabbed a NEW tar from http://cython.org/Cython-0.11.rc.tar.gz. (I >>>> won't comment on the confusion caused by changing the file without >>>> changing the name :) >>>> >>>> Now testing at least starts, but seems to have some errors: >>>> (Ignore the >>>> RPM build errors: >>>> Installed (but unpackaged) file(s) found:, >>>> I'll fix those) >>>> >>>> >>> The problem is that NumPy appears to be installed on your system (import >>> numpy succeeds), yet the NumPy C header files is not present in the >>> include path. >>> >>> Either set an extra include-path using the INCLUDE environment variable: >>> >>> INCLUDE=/path/to/numpy/headers python runtests.py ... >>> >>> or, drop the NumPy tests from the test suite (as I suppose your Cython >>> RPM doesn't depend on NumPy -- it shouldn't!): >>> >>> python runtests.py ... -x numpy >>> >>> Dag Sverre >>> >> >> Here is code used to find the 'pyublas' includes. Something like this >> could be used to find the numpy includes: >> > Yep, but it won't happen in this release. See my upcoming response to > Lisandro. > > But I think the Cython RPM shouldn't depend on the NumPy RPM, or be > tested differently if NumPy is installed or not in the RPM build? > > Cython doesn't depend on NumPy in any way, there's just a test case > sitting there more for convenience than anything else...the testcase > doesn't add anything to correctness testing of Cython either. > > Dag Sverre OK, I agree. I will use -x numpy and we avoid BuildRequires: numpy. With this, the build succeeds (on x86_64 F10). Anyone test this yet on gcc-4.4 (which is used for F11)? From dagss at student.matnat.uio.no Tue Mar 10 16:00:00 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 10 Mar 2009 16:00:00 +0100 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: <52637.213.61.181.86.1236695813.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <52637.213.61.181.86.1236695813.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49B68070.6040509@student.matnat.uio.no> Stefan Behnel wrote: > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > cdef object[foo=1, 2] x > ^ > ------------------------------------------------------------ > > Cython_Compiler_Tests_TestBuffer_TestBufferParsing_test_pos_after_key:1:19: > Non-keyword arg following keyword arg > *** > > I remember seeing these, but they went away when rebuilding Cython. That's > why I'm surprised to see them in a clean build now... > > Dag, any idea where those might come from? > Hmm. At least it does not indicate errors outside of the test framework. These are *expected and required* errors, from Cython.Compiler.Tests.TestBuffer.TestBufferParsing. What is checked is that these errors are raised. The problem is that obviously the parser logs the exception somewhere in this case; probably in addition to raising the exception which is probably properly trapped. On my machine this is done cleanly: dagss at corcaroli ~/cython/devel $ python runtests.py TestBuffer -vv Running tests against Cython 0.11.rc Python 2.5.2 (r252:60911, Jul 22 2008, 12:44:54) [GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] test_basic (Cython.Compiler.Tests.TestBuffer.TestBufferParsing) ... ok test_notype_as_expr1 (Cython.Compiler.Tests.TestBuffer.TestBufferParsing) ... ok test_notype_as_expr2 (Cython.Compiler.Tests.TestBuffer.TestBufferParsing) ... ok test_pos_after_key (Cython.Compiler.Tests.TestBuffer.TestBufferParsing) ... ok test_type_keyword (Cython.Compiler.Tests.TestBuffer.TestBufferParsing) ... ok test_type_pos (Cython.Compiler.Tests.TestBuffer.TestBufferParsing) ... ok ---------------------------------------------------------------------- Ran 6 tests in 0.012s OK Might be unittest version and logging policy differences? Don't have time looking deeper into it yet. Dag Sverre From dalcinl at gmail.com Tue Mar 10 16:23:38 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 10 Mar 2009 13:23:38 -0200 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: <40279.213.61.181.86.1236696818.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49B676A3.3090405@student.matnat.uio.no> <49B67D10.4060308@student.matnat.uio.no> <40279.213.61.181.86.1236696818.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: I just asked for automating the numpy include in runtest.py !! I have numpy installed, I run the testsuite, and almost all times I forget to setenv INCLUDE, thus the numpy test fails... If the numpy test is enabled by default depending on numpy being installed, why should manually setup my environment? That's an nonsense, and even more when numpy provides a canonical, long-time supported way to get their include directory ... Regarding hacking distutils for the heck of numpy support, please no! That would be a very bad thing... Anyway, anyone using numpy can all "numpy.get_include()" to easily get the numpy include directory, it is really easy.... On Tue, Mar 10, 2009 at 12:53 PM, Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> Lisandro Dalcin wrote: >>> Dag, from very long time (as far as I can remember), numpy provided a >>> canonical way to get the include files >>> >>> import numpy >>> numpy_include = numpy.get_include() >>> >>> I remember some mails in this list where (Stefan or you?) resisted on >>> automating this... But this has worked even before 1.0.X numpy >>> released... >>> >>> IMHO, this should be automated ... >>> >> I'll resist again, but I don't have a strong opinion, so if you can >> persuade Stefan and provide a patch I'm ok with it. >> >> a) One could add it to runtests.py. But that is primarily a tool for >> development, and the most important thing is that things are nice and >> explicit and that nothing magic happens under the hood. It's not >> supposed to "just work", rather the opposite! :-) > > I'm fine with using it in runtest.py. We test for NumPy anyway when > setting up the test suite, so we can just as well go all the way. > Requiring a user to set up a specific build environment just to run the > test suite is wrong. > > I'm against adding any special handling to the distutils modules, though. > Having NumPy installed doesn't imply that you want to use its header > files. > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Tue Mar 10 16:29:20 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 10 Mar 2009 16:29:20 +0100 (CET) Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: References: <49B676A3.3090405@student.matnat.uio.no> <49B67D10.4060308@student.matnat.uio.no> <40279.213.61.181.86.1236696818.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <50816.213.61.181.86.1236698960.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Lisandro Dalcin wrote: > On Tue, Mar 10, 2009 at 12:53 PM, Stefan Behnel wrote: >> I'm fine with using it in runtest.py. We test for NumPy anyway when >> setting up the test suite, so we can just as well go all the way. >> Requiring a user to set up a specific build environment just to run the >> test suite is wrong. >> >> I'm against adding any special handling to the distutils modules, >> though. >> Having NumPy installed doesn't imply that you want to use its header >> files. > > I just asked for automating the numpy include in runtest.py !! I have > numpy installed, I run the testsuite, and almost all times I forget > to setenv INCLUDE, thus the numpy test fails... If the numpy test is > enabled by default depending on numpy being installed, why should > manually setup my environment? That's an nonsense, and even more when > numpy provides a canonical, long-time supported way to get their > include directory ... > > Regarding hacking distutils for the heck of numpy support, please no! > That would be a very bad thing... Anyway, anyone using numpy can all > "numpy.get_include()" to easily get the numpy include directory, it is > really easy.... Thanks for rephrasing my thoughts. :) Please supply a fix for runtests.py. Stefan From ndbecker2 at gmail.com Tue Mar 10 16:27:12 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 10 Mar 2009 11:27:12 -0400 Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 Message-ID: This is on fedora rawhide, x86_64: + /usr/bin/python runtests.py -x numpy Running tests against Cython 0.11.rc Python 2.6 (r26:66714, Feb 26 2009, 17:32:04) [GCC 4.4.0 20090219 (Red Hat 4.4.0-0.21)] warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:62:6: 'report_unraisable' redeclared warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:84:17: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:87:18: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:87:32: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:87:47: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:96:13: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:98:13: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:98:33: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:101:18: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:101:32: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:101:47: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:110:25: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:112:25: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:112:45: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:115:18: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:115:32: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:115:47: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:122:34: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:127:38: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:135:18: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:136:15: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:136:42: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:142:14: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:144:18: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:144:32: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:144:47: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:90:5: Unraisable exception in function 'refnanny.GOTREF'. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:103:5: Unraisable exception in function 'refnanny.GIVEREF_and_report'. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:129:5: Unraisable exception in function 'refnanny.FinishContext'. Error converting Pyrex file to C: ------------------------------------------------------------ ... cdef object[foo2=short unsigned int] x ^ ------------------------------------------------------------ Cython_Compiler_Tests_TestBuffer_TestBufferParsing_test_notype_as_expr1:1:23: Expected: expression Error converting Pyrex file to C: ------------------------------------------------------------ ... cdef object[int, short unsigned int] x ^ ------------------------------------------------------------ Cython_Compiler_Tests_TestBuffer_TestBufferParsing_test_notype_as_expr2:1:23: Expected: expression Error converting Pyrex file to C: ------------------------------------------------------------ ... cdef object[foo=1, 2] x ^ ------------------------------------------------------------ Cython_Compiler_Tests_TestBuffer_TestBufferParsing_test_pos_after_key:1:19: Non-keyword arg following keyword arg del.cpp: In function ?int __Pyx_DelItemInt(PyObject*, Py_ssize_t, int)?: del.cpp:274: warning: left shift count >= width of type index.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: index.cpp:271: warning: left shift count >= width of type index.cpp: In function ?int __Pyx_SetItemInt(PyObject*, Py_ssize_t, PyObject*, int)?: index.cpp:339: warning: left shift count >= width of type indices.cpp: In function ?int __Pyx_SetItemInt(PyObject*, Py_ssize_t, PyObject*, int)?: indices.cpp:272: warning: left shift count >= width of type indices.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: indices.cpp:283: warning: left shift count >= width of type jiba5.cpp: In function ?int __Pyx_DelItemInt(PyObject*, Py_ssize_t, int)?: jiba5.cpp:266: warning: left shift count >= width of type johnson1.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: johnson1.cpp:263: warning: left shift count >= width of type specmethdocstring.c:303: warning: ?__pyx_doc_17specmethdocstring_1C___init__? defined but not used specmethdocstring.c:325: warning: ?__pyx_doc_17specmethdocstring_1C_3foo___get__? defined but not used specmethdocstring.c:344: warning: ?__pyx_doc_17specmethdocstring_1C_3foo___set__? defined but not used specmethdocstring.cpp:303: warning: ?__pyx_doc_17specmethdocstring_1C___init__? defined but not used specmethdocstring.cpp:325: warning: ?__pyx_doc_17specmethdocstring_1C_3foo___get__? defined but not used specmethdocstring.cpp:344: warning: ?__pyx_doc_17specmethdocstring_1C_3foo___set__? defined but not used tryexcept.cpp: In function ?int __Pyx_SetItemInt(PyObject*, Py_ssize_t, PyObject*, int)?: tryexcept.cpp:280: warning: left shift count >= width of type typecast.c: In function ?__pyx_f_8typecast_f?: typecast.c:307: warning: cast to pointer from integer of different size arrayassign.c: In function ?__pyx_pf_11arrayassign_test_literal_list_slice_start_param?: arrayassign.c:775: warning: format ?%d? expects type ?int?, but argument 4 has type ?Py_ssize_t? arrayassign.c: In function ?__pyx_pf_11arrayassign_test_literal_list_slice_end_param?: arrayassign.c:893: warning: format ?%d? expects type ?int?, but argument 4 has type ?Py_ssize_t? arrayassign.c: In function ?__pyx_pf_11arrayassign_test_literal_list_slice_start_end_param?: arrayassign.c:1056: warning: format ?%d? expects type ?int?, but argument 4 has type ?Py_ssize_t? arrayassign.cpp: In function ?PyObject* __pyx_pf_11arrayassign_test_literal_list_slice_start_param(PyObject*, PyObject*)?: arrayassign.cpp:775: warning: format ?%d? expects type ?int?, but argument 4 has type ?Py_ssize_t? arrayassign.cpp: In function ?PyObject* __pyx_pf_11arrayassign_test_literal_list_slice_end_param(PyObject*, PyObject*)?: arrayassign.cpp:893: warning: format ?%d? expects type ?int?, but argument 4 has type ?Py_ssize_t? arrayassign.cpp: In function ?PyObject* __pyx_pf_11arrayassign_test_literal_list_slice_start_end_param(PyObject*, PyObject*, PyObject*)?: arrayassign.cpp:1056: warning: format ?%d? expects type ?int?, but argument 4 has type ?Py_ssize_t? big_indices.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: big_indices.cpp:263: warning: left shift count >= width of type bufaccess.c: In function ?__pyx_pf_9bufaccess_printbuf?: bufaccess.c:1525: warning: unused variable ?__pyx_boffset_1_buf? bufaccess.c:1524: warning: unused variable ?__pyx_boffset_0_buf? bufaccess.c:1523: warning: unused variable ?__pyx_bshape_1_buf? bufaccess.c:1522: warning: unused variable ?__pyx_bshape_0_buf? bufaccess.c:1521: warning: unused variable ?__pyx_bstride_1_buf? bufaccess.c:1520: warning: unused variable ?__pyx_bstride_0_buf? bufaccess.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: bufaccess.cpp:311: warning: left shift count >= width of type bufaccess.cpp: In function ?PyObject* __pyx_pf_9bufaccess_printbuf(PyObject*, PyObject*)?: bufaccess.cpp:1520: warning: unused variable ?__pyx_bstride_0_buf? bufaccess.cpp:1521: warning: unused variable ?__pyx_bstride_1_buf? bufaccess.cpp:1522: warning: unused variable ?__pyx_bshape_0_buf? bufaccess.cpp:1523: warning: unused variable ?__pyx_bshape_1_buf? bufaccess.cpp:1524: warning: unused variable ?__pyx_boffset_0_buf? bufaccess.cpp:1525: warning: unused variable ?__pyx_boffset_1_buf? buffer.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: buffer.cpp:263: warning: left shift count >= width of type callargs.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: callargs.cpp:274: warning: left shift count >= width of type cdef_opt.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: cdef_opt.cpp:271: warning: left shift count >= width of type charencoding.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: charencoding.cpp:263: warning: left shift count >= width of type charescape.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: charescape.cpp:263: warning: left shift count >= width of type classbody_exec.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: classbody_exec.cpp:263: warning: left shift count >= width of type concatcstrings.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: concatcstrings.cpp:263: warning: left shift count >= width of type cstringmeth.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: cstringmeth.cpp:263: warning: left shift count >= width of type ct_DEF.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: ct_DEF.cpp:263: warning: left shift count >= width of type delete.cpp: In function ?int __Pyx_DelItemInt(PyObject*, Py_ssize_t, int)?: delete.cpp:268: warning: left shift count >= width of type dictintindex.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: dictintindex.cpp:263: warning: left shift count >= width of type dictintindex.cpp: In function ?int __Pyx_DelItemInt(PyObject*, Py_ssize_t, int)?: dictintindex.cpp:325: warning: left shift count >= width of type forfrom.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: forfrom.cpp:263: warning: left shift count >= width of type funcexcept.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: funcexcept.cpp:263: warning: left shift count >= width of type funcexceptchained.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: funcexceptchained.cpp:263: warning: left shift count >= width of type funcexceptcypy.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: funcexceptcypy.cpp:271: warning: left shift count >= width of type funcexceptreturn.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: funcexceptreturn.cpp:263: warning: left shift count >= width of type future_unicode_literals.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: future_unicode_literals.cpp:263: warning: left shift count >= width of type index.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: index.cpp:271: warning: left shift count >= width of type inplace.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: inplace.cpp:277: warning: left shift count >= width of type inplace.cpp: In function ?int __Pyx_SetItemInt(PyObject*, Py_ssize_t, PyObject*, int)?: inplace.cpp:345: warning: left shift count >= width of type int_literals.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: int_literals.cpp:263: warning: left shift count >= width of type ishimoto2.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: ishimoto2.cpp:271: warning: left shift count >= width of type isinstance.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: isinstance.cpp:263: warning: left shift count >= width of type jarausch1.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: jarausch1.cpp:263: warning: left shift count >= width of type kostyrka2.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: kostyrka2.cpp:263: warning: left shift count >= width of type kwargproblems.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: kwargproblems.cpp:269: warning: left shift count >= width of type lepage_1.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: lepage_1.cpp:266: warning: left shift count >= width of type literal_lists.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: literal_lists.cpp:263: warning: left shift count >= width of type literalslice.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: literalslice.cpp:263: warning: left shift count >= width of type locals.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: locals.cpp:271: warning: left shift count >= width of type longlongindex.cpp: In function ?int __Pyx_SetItemInt(PyObject*, Py_ssize_t, PyObject*, int)?: longlongindex.cpp:280: warning: left shift count >= width of type modbody.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: modbody.cpp:263: warning: left shift count >= width of type modop.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: modop.cpp:271: warning: left shift count >= width of type multass.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: multass.cpp:263: warning: left shift count >= width of type new_style_exceptions.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: new_style_exceptions.cpp:263: warning: left shift count >= width of type powop.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: powop.cpp:271: warning: left shift count >= width of type r_addint.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: r_addint.cpp:271: warning: left shift count >= width of type r_barbieri1.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: r_barbieri1.cpp:269: warning: left shift count >= width of type r_bowden1.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: r_bowden1.cpp:263: warning: left shift count >= width of type r_docstrings.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: r_docstrings.cpp:265: warning: left shift count >= width of type r_hordijk1.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: r_hordijk1.cpp:269: warning: left shift count >= width of type r_huss3.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: r_huss3.cpp:263: warning: left shift count >= width of type r_jeff_epler_1.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: r_jeff_epler_1.cpp:263: warning: left shift count >= width of type r_mang1.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: r_mang1.cpp:263: warning: left shift count >= width of type r_mitch_chapman_2.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: r_mitch_chapman_2.cpp:263: warning: left shift count >= width of type r_pythonapi.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: r_pythonapi.cpp:264: warning: left shift count >= width of type r_uintindex.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: r_uintindex.cpp:271: warning: left shift count >= width of type set.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: set.cpp:351: warning: left shift count >= width of type simpcall.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: simpcall.cpp:271: warning: left shift count >= width of type size_t.c: In function ?__pyx_pf_6size_t_1A___init__?: size_t.c:389: warning: unused variable ?__pyx_arg_a? size_t.cpp: In function ?int __pyx_pf_6size_t_1A___init__(PyObject*, PyObject*, PyObject*)?: size_t.cpp:389: warning: unused variable ?__pyx_arg_a? slice_charptr.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: slice_charptr.cpp:271: warning: left shift count >= width of type strconstinclass.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: strconstinclass.cpp:263: warning: left shift count >= width of type strescapes.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: strescapes.cpp:263: warning: left shift count >= width of type strfunction.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: strfunction.cpp:263: warning: left shift count >= width of type strliterals.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: strliterals.cpp:263: warning: left shift count >= width of type unicodefunction.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: unicodefunction.cpp:263: warning: left shift count >= width of type unicodeliterals.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: unicodeliterals.cpp:263: warning: left shift count >= width of type unicodeliteralsdefault.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: unicodeliteralsdefault.cpp:263: warning: left shift count >= width of type unicodeliteralslatin1.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: unicodeliteralslatin1.cpp:263: warning: left shift count >= width of type unsigned.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: unsigned.cpp:263: warning: left shift count >= width of type wundram1.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: wundram1.cpp:263: warning: left shift count >= width of type ---------------------------------------------------------------------- Ran 1404 tests in 263.628s OK Following tests excluded because of missing dependencies on your system: run.numpy_test From dalcinl at gmail.com Tue Mar 10 16:30:26 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 10 Mar 2009 13:30:26 -0200 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: <52637.213.61.181.86.1236695813.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <52637.213.61.181.86.1236695813.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: >> Now testing at least starts, but seems to have some errors: > > * warning: left shift count >= width of type > > I don't get these locally. The problem is in __Pyx_GetItemGeneric, second > line: "1ULL << (sizeof(Py_ssize_t)*8)". Looks like that's not 64bit clean. > PyInt_FromSsize_t() should be used here.... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Tue Mar 10 16:38:43 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 10 Mar 2009 16:38:43 +0100 (CET) Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: References: <52637.213.61.181.86.1236695813.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <61316.213.61.181.86.1236699523.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Lisandro Dalcin wrote: >>> Now testing at least starts, but seems to have some errors: >> >> * warning: left shift count >= width of type >> >> I don't get these locally. The problem is in __Pyx_GetItemGeneric, >> second >> line: "1ULL << (sizeof(Py_ssize_t)*8)". Looks like that's not 64bit >> clean. >> > > PyInt_FromSsize_t() should be used here.... Could you come up with a fix? The other functions are affected, too: inplace.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: inplace.cpp:277: warning: left shift count >= width of type inplace.cpp: In function ?int __Pyx_SetItemInt(PyObject*, Py_ssize_t, PyObject*, int)?: inplace.cpp:345: warning: left shift count >= width of type dictintindex.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: dictintindex.cpp:263: warning: left shift count >= width of type Stefan From ndbecker2 at gmail.com Tue Mar 10 16:48:49 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 10 Mar 2009 11:48:49 -0400 Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 References: Message-ID: Built succeeds on fedora rawhide on all supported archs. From stefan_ml at behnel.de Tue Mar 10 16:54:17 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 10 Mar 2009 16:54:17 +0100 (CET) Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 In-Reply-To: References: Message-ID: <44706.213.61.181.86.1236700457.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Neal Becker wrote: > Built succeeds on fedora rawhide on all supported archs. Thanks for testing. However, we will need a retest when we have fixed the discovered 64bit issues. You can get the latest cython-devel from here: http://hg.cython.org/cython-devel Either install hg and do a checkout of that URL, or use the direct archive download links on the left of the web page. The latest version from there shouldn't require any of the patches I posted, but it does not yet have the 64bit fix. Stefan From dalcinl at gmail.com Tue Mar 10 16:55:02 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 10 Mar 2009 13:55:02 -0200 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: <61316.213.61.181.86.1236699523.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <52637.213.61.181.86.1236695813.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <61316.213.61.181.86.1236699523.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: sure, wait me a bit, working on the runtests+numpy patch :-) On Tue, Mar 10, 2009 at 1:38 PM, Stefan Behnel wrote: > Lisandro Dalcin wrote: >>>> Now testing at least starts, but seems to have some errors: >>> >>> * warning: left shift count >= width of type >>> >>> I don't get these locally. The problem is in __Pyx_GetItemGeneric, >>> second >>> line: "1ULL << (sizeof(Py_ssize_t)*8)". Looks like that's not 64bit >>> clean. >>> >> >> PyInt_FromSsize_t() should be used here.... > > Could you come up with a fix? The other functions are affected, too: > > inplace.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, > Py_ssize_t, int)?: > inplace.cpp:277: warning: left shift count >= width of type > > inplace.cpp: In function ?int __Pyx_SetItemInt(PyObject*, Py_ssize_t, > PyObject*, int)?: > inplace.cpp:345: warning: left shift count >= width of type > > dictintindex.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, > Py_ssize_t, int)?: > dictintindex.cpp:263: warning: left shift count >= width of type > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Tue Mar 10 16:58:38 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 10 Mar 2009 13:58:38 -0200 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: <50816.213.61.181.86.1236698960.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49B676A3.3090405@student.matnat.uio.no> <49B67D10.4060308@student.matnat.uio.no> <40279.213.61.181.86.1236696818.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <50816.213.61.181.86.1236698960.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: On Tue, Mar 10, 2009 at 1:29 PM, Stefan Behnel wrote: > > Thanks for rephrasing my thoughts. :) > You are welcome... > > Please supply a fix for runtests.py. > Attached... Hope you like it.... feel free to modify and push ... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 -------------- next part -------------- A non-text attachment was scrubbed... Name: RUNTESTS_NUMPY.diff Type: application/octet-stream Size: 1107 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090310/eff6ec60/attachment.obj From dalcinl at gmail.com Tue Mar 10 17:19:53 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 10 Mar 2009 14:19:53 -0200 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: <61316.213.61.181.86.1236699523.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <52637.213.61.181.86.1236695813.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <61316.213.61.181.86.1236699523.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: Here you have a trivial patch... However, all this part smells so bad... On Tue, Mar 10, 2009 at 1:38 PM, Stefan Behnel wrote: > Lisandro Dalcin wrote: >>>> Now testing at least starts, but seems to have some errors: >>> >>> * warning: left shift count >= width of type >>> >>> I don't get these locally. The problem is in __Pyx_GetItemGeneric, >>> second >>> line: "1ULL << (sizeof(Py_ssize_t)*8)". Looks like that's not 64bit >>> clean. >>> >> >> PyInt_FromSsize_t() should be used here.... > > Could you come up with a fix? The other functions are affected, too: > > inplace.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, > Py_ssize_t, int)?: > inplace.cpp:277: warning: left shift count >= width of type > > inplace.cpp: In function ?int __Pyx_SetItemInt(PyObject*, Py_ssize_t, > PyObject*, int)?: > inplace.cpp:345: warning: left shift count >= width of type > > dictintindex.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, > Py_ssize_t, int)?: > dictintindex.cpp:263: warning: left shift count >= width of type > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 -------------- next part -------------- A non-text attachment was scrubbed... Name: PATCH.diff Type: application/octet-stream Size: 1680 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090310/50e17e36/attachment.obj From dalcinl at gmail.com Tue Mar 10 17:53:17 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 10 Mar 2009 14:53:17 -0200 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: References: <52637.213.61.181.86.1236695813.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <61316.213.61.181.86.1236699523.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: Indeed, this smells really bad... the former implementation seems really hackish... my patch will not work. On Tue, Mar 10, 2009 at 2:19 PM, Lisandro Dalcin wrote: > Here you have a trivial patch... However, all this part smells so bad... > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From ndbecker2 at gmail.com Tue Mar 10 18:10:55 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 10 Mar 2009 13:10:55 -0400 Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 References: <44706.213.61.181.86.1236700457.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: Stefan Behnel wrote: > Neal Becker wrote: >> Built succeeds on fedora rawhide on all supported archs. > > Thanks for testing. However, we will need a retest when we have fixed the > discovered 64bit issues. You can get the latest cython-devel from here: > > http://hg.cython.org/cython-devel > > Either install hg and do a checkout of that URL, or use the direct archive > download links on the left of the web page. > > The latest version from there shouldn't require any of the patches I > posted, but it does not yet have the 64bit fix. > > Stefan Can we have an rc2 with all proposed fixes? I'll plan to retest then. From stefan_ml at behnel.de Tue Mar 10 18:15:06 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 10 Mar 2009 18:15:06 +0100 Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 In-Reply-To: References: <44706.213.61.181.86.1236700457.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49B6A01A.8050904@behnel.de> Neal Becker wrote: > Can we have an rc2 with all proposed fixes? I'll plan to retest then. Fine with me, although it (hopefully) won't take long. Stefan From robertwb at math.washington.edu Tue Mar 10 18:44:54 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 10 Mar 2009 10:44:54 -0700 Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 In-Reply-To: <49B6A01A.8050904@behnel.de> References: <44706.213.61.181.86.1236700457.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B6A01A.8050904@behnel.de> Message-ID: On Mar 10, 2009, at 10:15 AM, Stefan Behnel wrote: > > Neal Becker wrote: >> Can we have an rc2 with all proposed fixes? I'll plan to retest >> then. > > Fine with me, although it (hopefully) won't take long. http://cython.org/Cython-0.11.rc2.tar.gz From stefan_ml at behnel.de Tue Mar 10 18:50:20 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 10 Mar 2009 18:50:20 +0100 Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 In-Reply-To: References: <44706.213.61.181.86.1236700457.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B6A01A.8050904@behnel.de> Message-ID: <49B6A85C.2090301@behnel.de> Robert Bradshaw wrote: > On Mar 10, 2009, at 10:15 AM, Stefan Behnel wrote: > >> Neal Becker wrote: >>> Can we have an rc2 with all proposed fixes? I'll plan to retest >>> then. >> Fine with me, although it (hopefully) won't take long. > > http://cython.org/Cython-0.11.rc2.tar.gz Ah, ok, but this doesn't include a fix for the __Pyx_GetItemInt() issues on 64bit platforms. Looks like we need another RC for that... Stefan From robertwb at math.washington.edu Tue Mar 10 18:51:09 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 10 Mar 2009 10:51:09 -0700 Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 In-Reply-To: References: <44706.213.61.181.86.1236700457.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B6A01A.8050904@behnel.de> Message-ID: On Mar 10, 2009, at 10:44 AM, Robert Bradshaw wrote: > On Mar 10, 2009, at 10:15 AM, Stefan Behnel wrote: > >> >> Neal Becker wrote: >>> Can we have an rc2 with all proposed fixes? I'll plan to retest >>> then. >> >> Fine with me, although it (hopefully) won't take long. > > http://cython.org/Cython-0.11.rc2.tar.gz BTW, this does not have the 64-bit issues fixed. (Still catching up on the other thread). Sorry about the naming for rc1, that wasn't intentional. - Robert From robertwb at math.washington.edu Tue Mar 10 19:08:32 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 10 Mar 2009 11:08:32 -0700 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: References: <52637.213.61.181.86.1236695813.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <61316.213.61.181.86.1236699523.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <5770E6DE-385E-46BE-9070-0AD57466C5A1@math.washington.edu> I'll fix this. The problem is that when passing via the inline function, the size and signness of the original index gets lost. The very common and much faster case is to use PyInt_FromLong, but of course we have to handle bigger indices as well. - Robert On Mar 10, 2009, at 9:19 AM, Lisandro Dalcin wrote: > Here you have a trivial patch... However, all this part smells so > bad... > > > On Tue, Mar 10, 2009 at 1:38 PM, Stefan Behnel > wrote: >> Lisandro Dalcin wrote: >>>>> Now testing at least starts, but seems to have some errors: >>>> >>>> * warning: left shift count >= width of type >>>> >>>> I don't get these locally. The problem is in __Pyx_GetItemGeneric, >>>> second >>>> line: "1ULL << (sizeof(Py_ssize_t)*8)". Looks like that's not 64bit >>>> clean. >>>> >>> >>> PyInt_FromSsize_t() should be used here.... >> >> Could you come up with a fix? The other functions are affected, too: >> >> inplace.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic >> (PyObject*, >> Py_ssize_t, int)?: >> inplace.cpp:277: warning: left shift count >= width of type >> >> inplace.cpp: In function ?int __Pyx_SetItemInt(PyObject*, Py_ssize_t, >> PyObject*, int)?: >> inplace.cpp:345: warning: left shift count >= width of type >> >> dictintindex.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic >> (PyObject*, >> Py_ssize_t, int)?: >> dictintindex.cpp:263: warning: left shift count >= width of type >> >> Stefan >> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0) > 342-451.1594______________________________________________ > _ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From stefan_ml at behnel.de Tue Mar 10 20:57:59 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 10 Mar 2009 20:57:59 +0100 Subject: [Cython] Remove "parsetuple_format" from PyrexTypes? Message-ID: <49B6C647.9090807@behnel.de> Hi, I just fixed an unused variable problem when using (Py_s)size_t as function argument types (fix can wait for 0.11.1). Doing this, I noticed that the type indication character that each type stores in "parsetuple_format" is no longer used anywhere since PyArg_ParseTupleAndKeywords() was replaced by our own code. Before I rip this attribute straight out, can anyone still see a use case for this? Maybe in combination with buffer content types? Stefan From dagss at student.matnat.uio.no Tue Mar 10 21:17:49 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 10 Mar 2009 21:17:49 +0100 Subject: [Cython] Remove "parsetuple_format" from PyrexTypes? In-Reply-To: <49B6C647.9090807@behnel.de> References: <49B6C647.9090807@behnel.de> Message-ID: <49B6CAED.9080400@student.matnat.uio.no> Stefan Behnel wrote: > Hi, > > I just fixed an unused variable problem when using (Py_s)size_t as function > argument types (fix can wait for 0.11.1). Doing this, I noticed that the > type indication character that each type stores in "parsetuple_format" is > no longer used anywhere since PyArg_ParseTupleAndKeywords() was replaced by > our own code. Before I rip this attribute straight out, can anyone still > see a use case for this? Maybe in combination with buffer content types? The buffers typestring checks do not use this. -- Dag Sverre From hazelnusse at gmail.com Tue Mar 10 22:33:02 2009 From: hazelnusse at gmail.com (Luke) Date: Tue, 10 Mar 2009 14:33:02 -0700 Subject: [Cython] Update on: Incomplete C++ example Message-ID: <99214b470903101433w2cb1a2f4x896fd11203f97a14@mail.gmail.com> I'm trying to wrap some basic C++ code. I tried to follow the directions on the wiki, but I found them to be unclear and incomplete. I also noticed a thread about this on Cython-dev, but it doesn't seem to have been resolved. As a total rookie with Cython and wrapping code in general, it would be nice to have a step by step example of the process that is illustrative of how to wrap a C++ library, including: 1) Source of the header file (Rectangle.h) 2) Source of the implementation file (Rectangle.cpp) 3) Source of the setup file (setup.py) 4) Source of the ??? file (Rectangle.pyx) 5) shell commands necessary prior to being able to import the Rectangle class into python 6) example of using the wrapped C++ library inside the python interpreter Things that were not helpful in the setup.py were things like: include_dirs=[...], # usual stuff libraries=[...], # ditto extra_link_args=[...], # if needed As an example of how to wrap a C++ class, I don't think extras that aren't well explained help the situation. From the wiki, it wasn't clear what exactly belonged in the .pyx file. It would be nice if this could be clarified. I think I can piece together steps 1), 2), 3) from the wiki and from the previous thread on the subject, but steps 4), 5), and 6) are still unclear to me. It would be a great help if these steps could be shown explicitly. Can anybody help me out on how to do this? Thanks, ~Luke From bpederse at gmail.com Tue Mar 10 22:51:30 2009 From: bpederse at gmail.com (Brent Pedersen) Date: Tue, 10 Mar 2009 14:51:30 -0700 Subject: [Cython] Update on: Incomplete C++ example In-Reply-To: <99214b470903101433w2cb1a2f4x896fd11203f97a14@mail.gmail.com> References: <99214b470903101433w2cb1a2f4x896fd11203f97a14@mail.gmail.com> Message-ID: i'm sure it doesnt answer all your questions, but here's how i built and used the cpp example, it required some code changes. i uploaded to wiki, but didnt link to it directly. it has a Makefile and example usage script. http://wiki.cython.org/WrappingCPlusPlus?action=AttachFile&do=get&target=cythoncpp.tgz -brent On Tue, Mar 10, 2009 at 2:33 PM, Luke wrote: > I'm trying to wrap some basic C++ code. ?I tried to follow the > directions on the wiki, but I found them to be unclear and incomplete. > ?I also noticed a thread about this on Cython-dev, but it doesn't seem > to have been resolved. ?As a total rookie with Cython and wrapping > code in general, it would be nice to have a step by step example of > the process that is illustrative of how to wrap a C++ library, > including: > > 1) ?Source of the header file (Rectangle.h) > 2) ?Source of the implementation file (Rectangle.cpp) > 3) ?Source of the setup file (setup.py) > 4) ?Source of the ??? file (Rectangle.pyx) > 5) ?shell commands necessary prior to being able to import the > Rectangle class into python > 6) ?example of using the wrapped C++ library inside the python interpreter > > Things that were not helpful in the setup.py were things like: > ? ?include_dirs=[...], ? ? ? ? ?# usual stuff > ? ?libraries=[...], ? ? ? ? ? ? # ditto > ? ?extra_link_args=[...], ? ? ? # if needed > > As an example of how to wrap a C++ class, I don't think extras that > aren't well explained help the situation. ?From the wiki, it wasn't > clear what exactly belonged in the .pyx file. ?It would be nice if > this could be clarified. > > I think I can piece together steps 1), 2), 3) from the wiki and from > the previous thread on the subject, but steps 4), 5), and 6) are still > unclear to me. ?It would be a great help if these steps could be shown > explicitly. ?Can anybody help me out on how to do this? > > Thanks, > ~Luke > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From hazelnusse at gmail.com Tue Mar 10 23:31:46 2009 From: hazelnusse at gmail.com (Luke) Date: Tue, 10 Mar 2009 15:31:46 -0700 Subject: [Cython] Update on: Incomplete C++ example In-Reply-To: References: <99214b470903101433w2cb1a2f4x896fd11203f97a14@mail.gmail.com> Message-ID: <99214b470903101531j5666939dv9c7e8f6f47ae74e4@mail.gmail.com> Brent, Thanks for the quick response. That helped a lot. What would I need to do differently if instead of having the header file and the implementation file (cpp_rect.h & cpp_rect.cpp), I instead had the header file and the object file (cpp_rect.h and cpp_rect.o)? Thanks, ~Luke On Tue, Mar 10, 2009 at 2:51 PM, Brent Pedersen wrote: > i'm sure it doesnt answer all your questions, but here's how i built > and used the cpp example, it required some code changes. > i uploaded to wiki, but didnt link to it directly. > it has a Makefile and example usage script. > > http://wiki.cython.org/WrappingCPlusPlus?action=AttachFile&do=get&target=cythoncpp.tgz > > -brent > > On Tue, Mar 10, 2009 at 2:33 PM, Luke wrote: >> I'm trying to wrap some basic C++ code. ?I tried to follow the >> directions on the wiki, but I found them to be unclear and incomplete. >> ?I also noticed a thread about this on Cython-dev, but it doesn't seem >> to have been resolved. ?As a total rookie with Cython and wrapping >> code in general, it would be nice to have a step by step example of >> the process that is illustrative of how to wrap a C++ library, >> including: >> >> 1) ?Source of the header file (Rectangle.h) >> 2) ?Source of the implementation file (Rectangle.cpp) >> 3) ?Source of the setup file (setup.py) >> 4) ?Source of the ??? file (Rectangle.pyx) >> 5) ?shell commands necessary prior to being able to import the >> Rectangle class into python >> 6) ?example of using the wrapped C++ library inside the python interpreter >> >> Things that were not helpful in the setup.py were things like: >> ? ?include_dirs=[...], ? ? ? ? ?# usual stuff >> ? ?libraries=[...], ? ? ? ? ? ? # ditto >> ? ?extra_link_args=[...], ? ? ? # if needed >> >> As an example of how to wrap a C++ class, I don't think extras that >> aren't well explained help the situation. ?From the wiki, it wasn't >> clear what exactly belonged in the .pyx file. ?It would be nice if >> this could be clarified. >> >> I think I can piece together steps 1), 2), 3) from the wiki and from >> the previous thread on the subject, but steps 4), 5), and 6) are still >> unclear to me. ?It would be a great help if these steps could be shown >> explicitly. ?Can anybody help me out on how to do this? >> >> Thanks, >> ~Luke >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From dalcinl at gmail.com Tue Mar 10 23:48:35 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 10 Mar 2009 20:48:35 -0200 Subject: [Cython] Update on: Incomplete C++ example In-Reply-To: <99214b470903101531j5666939dv9c7e8f6f47ae74e4@mail.gmail.com> References: <99214b470903101433w2cb1a2f4x896fd11203f97a14@mail.gmail.com> <99214b470903101531j5666939dv9c7e8f6f47ae74e4@mail.gmail.com> Message-ID: Well, here you have... a full tarball package (generated with "python setup.py sdist") with the minimum required package metadata for the sake of PyPI upload... the build.sh script install it in $PWD/tmp, next run the test.sh script, it just adjusts $PYTHONPATH and runs example.py I hope you can make a useful contribution to Cython's Wiki after this ;-) On Tue, Mar 10, 2009 at 8:31 PM, Luke wrote: > Brent, > ?Thanks for the quick response. ?That helped a lot. ?What would I > need to do differently if instead of having the header file and the > implementation file (cpp_rect.h & cpp_rect.cpp), I instead had the > header file and the object file (cpp_rect.h and cpp_rect.o)? > Thanks, > ~Luke > > On Tue, Mar 10, 2009 at 2:51 PM, Brent Pedersen wrote: >> i'm sure it doesnt answer all your questions, but here's how i built >> and used the cpp example, it required some code changes. >> i uploaded to wiki, but didnt link to it directly. >> it has a Makefile and example usage script. >> >> http://wiki.cython.org/WrappingCPlusPlus?action=AttachFile&do=get&target=cythoncpp.tgz >> >> -brent >> >> On Tue, Mar 10, 2009 at 2:33 PM, Luke wrote: >>> I'm trying to wrap some basic C++ code. ?I tried to follow the >>> directions on the wiki, but I found them to be unclear and incomplete. >>> ?I also noticed a thread about this on Cython-dev, but it doesn't seem >>> to have been resolved. ?As a total rookie with Cython and wrapping >>> code in general, it would be nice to have a step by step example of >>> the process that is illustrative of how to wrap a C++ library, >>> including: >>> >>> 1) ?Source of the header file (Rectangle.h) >>> 2) ?Source of the implementation file (Rectangle.cpp) >>> 3) ?Source of the setup file (setup.py) >>> 4) ?Source of the ??? file (Rectangle.pyx) >>> 5) ?shell commands necessary prior to being able to import the >>> Rectangle class into python >>> 6) ?example of using the wrapped C++ library inside the python interpreter >>> >>> Things that were not helpful in the setup.py were things like: >>> ? ?include_dirs=[...], ? ? ? ? ?# usual stuff >>> ? ?libraries=[...], ? ? ? ? ? ? # ditto >>> ? ?extra_link_args=[...], ? ? ? # if needed >>> >>> As an example of how to wrap a C++ class, I don't think extras that >>> aren't well explained help the situation. ?From the wiki, it wasn't >>> clear what exactly belonged in the .pyx file. ?It would be nice if >>> this could be clarified. >>> >>> I think I can piece together steps 1), 2), 3) from the wiki and from >>> the previous thread on the subject, but steps 4), 5), and 6) are still >>> unclear to me. ?It would be a great help if these steps could be shown >>> explicitly. ?Can anybody help me out on how to do this? >>> >>> Thanks, >>> ~Luke >>> _______________________________________________ >>> Cython-dev mailing list >>> Cython-dev at codespeak.net >>> http://codespeak.net/mailman/listinfo/cython-dev >>> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 -------------- next part -------------- A non-text attachment was scrubbed... Name: rectangle-0.0.1.tar.gz Type: application/x-gzip Size: 1663 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090310/0c07a6bc/attachment.bin From hazelnusse at gmail.com Wed Mar 11 00:34:20 2009 From: hazelnusse at gmail.com (Luke) Date: Tue, 10 Mar 2009 16:34:20 -0700 Subject: [Cython] Wrapping C++ functions that pass by reference Message-ID: <99214b470903101634i16f79657pe008fd1bc75a9533@mail.gmail.com> I have a C++ function like this: void constraint(double& x, double& dxdt) { dxdt[0] = ... dxdt[1] = ... .... dxdt[n] = ... } How would I properly wrap this function in the .pyx file? And how would calling it in Python work? I've tried to find examples for this type of thing but have been unable. If anybody knows of any, I'd greatly appreciate it. Thanks, ~Luke From carl.witty at gmail.com Wed Mar 11 03:35:39 2009 From: carl.witty at gmail.com (Carl Witty) Date: Tue, 10 Mar 2009 18:35:39 -0800 Subject: [Cython] Semantics of cdef int division and %? Message-ID: I just reported http://trac.cython.org/cython_trac/ticket/229, which points out that division and % on cdef ints have C rather than Python semantics (for instance, (-1 % 16) is -1 in C, and 15 in Python). I'm not sure what the right thing to do here is. Using C semantics gives faster code. Using Python semantics is nicer (whenever I care about the difference, I always want Python semantics) but slower, and would be confusing to somebody who was coming from C. But whatever happens should be prominently documented; I couldn't find any mention of the issue in the current docs (although maybe I wasn't looking in the right place). Carl From hazelnusse at gmail.com Wed Mar 11 05:46:03 2009 From: hazelnusse at gmail.com (Luke) Date: Tue, 10 Mar 2009 21:46:03 -0700 Subject: [Cython] Update on: Incomplete C++ example In-Reply-To: References: <99214b470903101433w2cb1a2f4x896fd11203f97a14@mail.gmail.com> <99214b470903101531j5666939dv9c7e8f6f47ae74e4@mail.gmail.com> Message-ID: <99214b470903102146q442c5406s2c3ed265c65f02e4@mail.gmail.com> Lisandro, Thank you! ~Luke On Tue, Mar 10, 2009 at 3:48 PM, Lisandro Dalcin wrote: > Well, here you have... a full tarball package (generated with "python > setup.py sdist") with the minimum required package metadata for the > sake of PyPI upload... the build.sh script install it in $PWD/tmp, > next run the test.sh script, it just adjusts $PYTHONPATH and runs > example.py > > I hope you can make a useful contribution to Cython's Wiki after this ;-) > > > On Tue, Mar 10, 2009 at 8:31 PM, Luke wrote: >> Brent, >> ?Thanks for the quick response. ?That helped a lot. ?What would I >> need to do differently if instead of having the header file and the >> implementation file (cpp_rect.h & cpp_rect.cpp), I instead had the >> header file and the object file (cpp_rect.h and cpp_rect.o)? >> Thanks, >> ~Luke >> >> On Tue, Mar 10, 2009 at 2:51 PM, Brent Pedersen wrote: >>> i'm sure it doesnt answer all your questions, but here's how i built >>> and used the cpp example, it required some code changes. >>> i uploaded to wiki, but didnt link to it directly. >>> it has a Makefile and example usage script. >>> >>> http://wiki.cython.org/WrappingCPlusPlus?action=AttachFile&do=get&target=cythoncpp.tgz >>> >>> -brent >>> >>> On Tue, Mar 10, 2009 at 2:33 PM, Luke wrote: >>>> I'm trying to wrap some basic C++ code. ?I tried to follow the >>>> directions on the wiki, but I found them to be unclear and incomplete. >>>> ?I also noticed a thread about this on Cython-dev, but it doesn't seem >>>> to have been resolved. ?As a total rookie with Cython and wrapping >>>> code in general, it would be nice to have a step by step example of >>>> the process that is illustrative of how to wrap a C++ library, >>>> including: >>>> >>>> 1) ?Source of the header file (Rectangle.h) >>>> 2) ?Source of the implementation file (Rectangle.cpp) >>>> 3) ?Source of the setup file (setup.py) >>>> 4) ?Source of the ??? file (Rectangle.pyx) >>>> 5) ?shell commands necessary prior to being able to import the >>>> Rectangle class into python >>>> 6) ?example of using the wrapped C++ library inside the python interpreter >>>> >>>> Things that were not helpful in the setup.py were things like: >>>> ? ?include_dirs=[...], ? ? ? ? ?# usual stuff >>>> ? ?libraries=[...], ? ? ? ? ? ? # ditto >>>> ? ?extra_link_args=[...], ? ? ? # if needed >>>> >>>> As an example of how to wrap a C++ class, I don't think extras that >>>> aren't well explained help the situation. ?From the wiki, it wasn't >>>> clear what exactly belonged in the .pyx file. ?It would be nice if >>>> this could be clarified. >>>> >>>> I think I can piece together steps 1), 2), 3) from the wiki and from >>>> the previous thread on the subject, but steps 4), 5), and 6) are still >>>> unclear to me. ?It would be a great help if these steps could be shown >>>> explicitly. ?Can anybody help me out on how to do this? >>>> >>>> Thanks, >>>> ~Luke >>>> _______________________________________________ >>>> Cython-dev mailing list >>>> Cython-dev at codespeak.net >>>> http://codespeak.net/mailman/listinfo/cython-dev >>>> >>> _______________________________________________ >>> Cython-dev mailing list >>> Cython-dev at codespeak.net >>> http://codespeak.net/mailman/listinfo/cython-dev >>> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > From dagss at student.matnat.uio.no Wed Mar 11 08:31:35 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 08:31:35 +0100 Subject: [Cython] Wrapping C++ functions that pass by reference In-Reply-To: <99214b470903101634i16f79657pe008fd1bc75a9533@mail.gmail.com> References: <99214b470903101634i16f79657pe008fd1bc75a9533@mail.gmail.com> Message-ID: <49B768D7.4030903@student.matnat.uio.no> Luke wrote: > I have a C++ function like this: > > void constraint(double& x, double& dxdt) > { > dxdt[0] = ... > dxdt[1] = ... > .... > dxdt[n] = ... > } > > How would I properly wrap this function in the .pyx file? And how > would calling it in Python work? I've tried to find examples for this > type of thing but have been unable. If anybody knows of any, I'd > greatly appreciate it. This is C++ syntax candy for passing pointers. You declare and use it in Cython like this: cdef extern from "yourheader.h": void constraint(double* x, double* dxdt) def foo(): cdef double x = 3.2, dxdt = 0.01 constraint(&x, &dxdt) -- Dag Sverre From dagss at student.matnat.uio.no Wed Mar 11 08:34:19 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 08:34:19 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: Message-ID: <49B7697B.5090304@student.matnat.uio.no> Carl Witty wrote: > I just reported http://trac.cython.org/cython_trac/ticket/229, which > points out that division and % on cdef ints have C rather than Python > semantics (for instance, (-1 % 16) is -1 in C, and 15 in Python). > > I'm not sure what the right thing to do here is. Using C semantics > gives faster code. Using Python semantics is nicer (whenever I care > about the difference, I always want Python semantics) but slower, and > would be confusing to somebody who was coming from C. I'm +1 on Python semantics. We could make the C version available under some obscure name for any case when one really wants the speed, like cython.fast_c_mod(-1, 16). -- Dag Sverre From hoytak at cs.ubc.ca Wed Mar 11 08:35:40 2009 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Wed, 11 Mar 2009 00:35:40 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49B7697B.5090304@student.matnat.uio.no> References: <49B7697B.5090304@student.matnat.uio.no> Message-ID: <4db580fd0903110035n2c76f5f6u1b9b7b804564566d@mail.gmail.com> > I'm +1 on Python semantics. We could make the C version available under > some obscure name for any case when one really wants the speed, like > cython.fast_c_mod(-1, 16). Couldn't we just use the same trick as for buffer arrays -- i.e. different checks are performed when its unsigned vs. signed? I don't see why unsigned integers are not identical in behavior here to C. I've also never come across a situation where one wants -1 % 16 to be -1 and not 15, but maybe there is some obscure use case. --Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From dagss at student.matnat.uio.no Wed Mar 11 09:23:04 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 09:23:04 +0100 (CET) Subject: [Cython] Integer power operator Message-ID: <1286d6e2177fb46620ab4a1f1fa7d7f3.squirrel@webmail.uio.no> While we're discussing arithmetic operations anyway...: I happen to routinely write (some complex integer expression)**2 in Python, and it is tedious to manually store the expression in a temporary etc. in Cython. I understand the reasons this is disabled from the FAQ; are there any reasons I can't create a Cython-specific utility code implementation of integer power though? The powers are usually very small here so I'm thinking just a for-loop with O(log n) multiplications... Or if the power is a smallish literal (my **2 an **3), I suppose just inline the multiplication operations... Dag Sverre From dagss at student.matnat.uio.no Wed Mar 11 10:27:22 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 10:27:22 +0100 Subject: [Cython] Beginner-friendly tickets Message-ID: <49B783FA.5070309@student.matnat.uio.no> Since there's been some interest in how to get started coding on Cython on the mailing list lately, I thought I'd call attention to these two tickets: http://trac.cython.org/cython_trac/ticket/158 http://trac.cython.org/cython_trac/ticket/203 These seems to fit the sweet spot of not being too hard to fix for beginners (with some mentoring), but not being completely trivial either. I'm happy to provide an attack strategy/small amount of mentoring for these tickets if anybody's interested. -- Dag Sverre From stefan_ml at behnel.de Wed Mar 11 10:13:39 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 11 Mar 2009 10:13:39 +0100 (CET) Subject: [Cython] Integer power operator In-Reply-To: <1286d6e2177fb46620ab4a1f1fa7d7f3.squirrel@webmail.uio.no> References: <1286d6e2177fb46620ab4a1f1fa7d7f3.squirrel@webmail.uio.no> Message-ID: <39764.213.61.181.86.1236762819.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > I happen to routinely write (some complex integer expression)**2 in > Python, and it is tedious to manually store the expression in a temporary > etc. in Cython. > > I understand the reasons this is disabled from the FAQ; are there any > reasons I can't create a Cython-specific utility code implementation of > integer power though? The powers are usually very small here so I'm > thinking just a for-loop with O(log n) multiplications... Fine with me, see http://trac.cython.org/cython_trac/ticket/127 > Or if the power is a smallish literal (my **2 an **3), I suppose just > inline the multiplication operations... For constant expressions, that would even be part of the constant folding transformation. There are two problems here: 1) IIRC, the error is currently raised too early in the code, before running into constant folding, so even cdef int someval = 3**2 doesn't work. 2) it would be weird to have an operator work in some cases and not in others. For example, you don't know the size of a "long" at translation time, so you can't know if cdef long someval = 65536**2 overflows or not. So, where would you draw the line? If you can come up with a sensible utility function that handles the different C compile time and runtime overflow cases nicely, I'm all for it. Stefan From dagss at student.matnat.uio.no Wed Mar 11 10:37:36 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 10:37:36 +0100 Subject: [Cython] Integer power operator In-Reply-To: <39764.213.61.181.86.1236762819.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <1286d6e2177fb46620ab4a1f1fa7d7f3.squirrel@webmail.uio.no> <39764.213.61.181.86.1236762819.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49B78660.8060904@student.matnat.uio.no> Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> I happen to routinely write (some complex integer expression)**2 in >> Python, and it is tedious to manually store the expression in a temporary >> etc. in Cython. > > If you can come up with a sensible utility function that handles the > different C compile time and runtime overflow cases nicely, I'm all for > it. So you are "+1 if I include an overflow check"? I was just thinking of letting it silently overflow, the way addition and multiplication already silently overflows. -- Dag Sverre From dagss at student.matnat.uio.no Wed Mar 11 10:49:34 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 10:49:34 +0100 Subject: [Cython] Integer power operator In-Reply-To: <49B78660.8060904@student.matnat.uio.no> References: <1286d6e2177fb46620ab4a1f1fa7d7f3.squirrel@webmail.uio.no> <39764.213.61.181.86.1236762819.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B78660.8060904@student.matnat.uio.no> Message-ID: <49B7892E.2010901@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Dag Sverre Seljebotn wrote: >>> I happen to routinely write (some complex integer expression)**2 in >>> Python, and it is tedious to manually store the expression in a temporary >>> etc. in Cython. > > >> If you can come up with a sensible utility function that handles the >> different C compile time and runtime overflow cases nicely, I'm all for >> it. > > So you are "+1 if I include an overflow check"? > > I was just thinking of letting it silently overflow, the way addition > and multiplication already silently overflows. > To be clear, I'm in favour of overflow checks, I just think they in that case should apply in *all* situations, not simply in this one operator. Otherwise Cython will be inconsistent and it makes the language harder to learn. So I'm thinking this: 1) Add integer ** without overflow check 2) Add overflowcheck compiler directive, similar to the nonecheck directive, which checks for overflows in /all/ situations (which defaults to off though) -- Dag Sverre From stefan_ml at behnel.de Wed Mar 11 12:01:40 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 11 Mar 2009 12:01:40 +0100 (CET) Subject: [Cython] Integer power operator In-Reply-To: <49B78660.8060904@student.matnat.uio.no> References: <1286d6e2177fb46620ab4a1f1fa7d7f3.squirrel@webmail.uio.no> <39764.213.61.181.86.1236762819.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B78660.8060904@student.matnat.uio.no> Message-ID: <44766.213.61.181.86.1236769300.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Dag Sverre Seljebotn wrote: >>> I happen to routinely write (some complex integer expression)**2 in >>> Python, and it is tedious to manually store the expression in a >>> temporary >>> etc. in Cython. > >> If you can come up with a sensible utility function that handles the >> different C compile time and runtime overflow cases nicely, I'm all for >> it. > > So you are "+1 if I include an overflow check"? > > I was just thinking of letting it silently overflow, the way addition > and multiplication already silently overflows. Hmm, yes, you are right. When a user writes cdef int tinyint = 2**2000 it's not Cython's fault if it overflows. So I guess I'm actually +1 for a fix that provides Python integer power semantics modulo C overflows/wrap-arounds. I guess that would be the semantics with the lowest probability of surprise. Would you say that this makes sense as a utility function? I mean, we could always write out the bit-shifting code for a constant exponent... http://c2.com/cgi/wiki?IntegerPowerAlgorithm Stefan From sturla at molden.no Wed Mar 11 13:30:55 2009 From: sturla at molden.no (Sturla Molden) Date: Wed, 11 Mar 2009 13:30:55 +0100 Subject: [Cython] Wrapping C++ functions that pass by reference In-Reply-To: <99214b470903101634i16f79657pe008fd1bc75a9533@mail.gmail.com> References: <99214b470903101634i16f79657pe008fd1bc75a9533@mail.gmail.com> Message-ID: <49B7AEFF.1000302@molden.no> Luke wrote: > I have a C++ function like this: > > void constraint(double& x, double& dxdt) > { > dxdt[0] = ... > dxdt[1] = ... > .... > dxdt[n] = ... > } > > How would I properly wrap this function in the .pyx file? And how > would calling it in Python work? I think you can just say cdef extern from "constraint.h": void constraint(double x, double dxdt) as the generated C++ should be identical. S.M. From dalcinl at gmail.com Wed Mar 11 14:01:38 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 11 Mar 2009 11:01:38 -0200 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49B7697B.5090304@student.matnat.uio.no> References: <49B7697B.5090304@student.matnat.uio.no> Message-ID: On Wed, Mar 11, 2009 at 5:34 AM, Dag Sverre Seljebotn wrote: > Carl Witty wrote: >> I just reported http://trac.cython.org/cython_trac/ticket/229, which >> points out that division and % on cdef ints have C rather than Python >> semantics (for instance, (-1 % 16) is -1 in C, and 15 in Python). >> >> I'm not sure what the right thing to do here is. ?Using C semantics >> gives faster code. ?Using Python semantics is nicer (whenever I care >> about the difference, I always want Python semantics) but slower, and >> would be confusing to somebody who was coming from C. > > I'm +1 on Python semantics. We could make the C version available under > some obscure name for any case when one really wants the speed, like > cython.fast_c_mod(-1, 16). > I'm no so sure... C is C, Python is Python, and Cython is in the middle, but a cdef int is IMHO on C side... Are we going to upcast '1/2' to a float? Of course, I could live if Cython does this: - Python semantics for '%' - C semantics for an all-new operator '%%'. Just thinking on what Python has done with '//' And again... Are we going to upcast '1/2' to a float next? Still not sure about this... Dag, some long time ago, when I asked to suport s"abc" string literals, you told me that such feature is easily implemented on users side with a tiny function and doing s("abc") ... Well, you were right, but I believe so I am, users can always implement c = mod(a,b) to return what they want ;-) -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Wed Mar 11 14:05:11 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 14:05:11 +0100 Subject: [Cython] Wrapping C++ functions that pass by reference In-Reply-To: <49B7AEFF.1000302@molden.no> References: <99214b470903101634i16f79657pe008fd1bc75a9533@mail.gmail.com> <49B7AEFF.1000302@molden.no> Message-ID: <49B7B707.4000303@student.matnat.uio.no> Sturla Molden wrote: > Luke wrote: > >> I have a C++ function like this: >> >> void constraint(double& x, double& dxdt) >> { >> dxdt[0] = ... >> dxdt[1] = ... >> .... >> dxdt[n] = ... >> } >> >> How would I properly wrap this function in the .pyx file? And how >> would calling it in Python work? >> > I think you can just say > > cdef extern from "constraint.h": > void constraint(double x, double dxdt) > > as the generated C++ should be identical. > Good point! Please disregard my previous mail about this, I was wrong, and this is right. Dag Sverre From dagss at student.matnat.uio.no Wed Mar 11 14:13:58 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 14:13:58 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: <49B7697B.5090304@student.matnat.uio.no> Message-ID: <49B7B916.3080101@student.matnat.uio.no> Lisandro Dalcin wrote: > On Wed, Mar 11, 2009 at 5:34 AM, Dag Sverre Seljebotn > wrote: > >> Carl Witty wrote: >> >>> I just reported http://trac.cython.org/cython_trac/ticket/229, which >>> points out that division and % on cdef ints have C rather than Python >>> semantics (for instance, (-1 % 16) is -1 in C, and 15 in Python). >>> >>> I'm not sure what the right thing to do here is. Using C semantics >>> gives faster code. Using Python semantics is nicer (whenever I care >>> about the difference, I always want Python semantics) but slower, and >>> would be confusing to somebody who was coming from C. >>> >> I'm +1 on Python semantics. We could make the C version available under >> some obscure name for any case when one really wants the speed, like >> cython.fast_c_mod(-1, 16). >> >> > > I'm no so sure... C is C, Python is Python, and Cython is in the > middle, but a cdef int is IMHO on C side... Are we going to upcast > '1/2' to a float? > I think we should follow Python convention, i.e. "from __future__ import division" means this behaviour, and if the code (in the future) is Python 3-level, then that should be the default (as you say there's always the //). > Still not sure about this... > > Dag, some long time ago, when I asked to suport s"abc" string > literals, you told me that such feature is easily implemented on users > side with a tiny function and doing s("abc") ... Well, you were right, > but I believe so I am, users can always implement c = mod(a,b) to > return what they want ;-) > Sure :-) Syntax is mostly a matter of taste in the end. But the deeper reason behind my arguments is this: A rather typical usecase of Cython, which I believe we should encourage, is to a) first implement things in Python (or at least in untyped Cython) b) profile (or at least experience that things are too slow) c) find out which functions you bother to type, and incrementally type your program "bottom up". Now, if we suddenly change the meaning of operators depending on whether they operate on Python ints or C ints, it means that an algorithm that worked perfectly in step a) will break in step c). Which just means more hassle, and that step c) become a bigger job than it needs to be. Therefore I believe that the "C" part of Cython should mean "it is easy to call C and work with C types", but that the operator semantics should be as close to Python as possible. (Modulo overflowing, but as I said I hope for a overflowcheck compiler directive as well to help while doing step c), which can then be turned off). Dag Sverre From stefan_ml at behnel.de Wed Mar 11 14:18:06 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 11 Mar 2009 14:18:06 +0100 (CET) Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: <49B7697B.5090304@student.matnat.uio.no> Message-ID: <56541.213.61.181.86.1236777486.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Lisandro Dalcin wrote: > On Wed, Mar 11, 2009 at 5:34 AM, Dag Sverre Seljebotn > wrote: >> Carl Witty wrote: >>> I just reported http://trac.cython.org/cython_trac/ticket/229, which >>> points out that division and % on cdef ints have C rather than Python >>> semantics (for instance, (-1 % 16) is -1 in C, and 15 in Python). >>> >>> I'm not sure what the right thing to do here is. ?Using C semantics >>> gives faster code. ?Using Python semantics is nicer (whenever I care >>> about the difference, I always want Python semantics) but slower, and >>> would be confusing to somebody who was coming from C. >> >> I'm +1 on Python semantics. We could make the C version available under >> some obscure name for any case when one really wants the speed, like >> cython.fast_c_mod(-1, 16). > > I'm no so sure... C is C, Python is Python, and Cython is in the > middle, but a cdef int is IMHO on C side... Are we going to upcast > '1/2' to a float? I agree that this is one of those corner cases where there isn't a clear case for either Python and C semantics. I do see an interest in having Python semantics by default (even for C ints), but that might be different for people coming more from a C background. So there will always be surprises. What about supporting an explicit cast like this: i = -1 % 16 or at whatever part of that expression it looks better. Maybe cdef unsigned int i = -1 % 16 could behave like that, although it does feel weird that the result of an expression might depend on what it's assigned to... Stefan From dagss at student.matnat.uio.no Wed Mar 11 14:19:27 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 14:19:27 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: <49B7697B.5090304@student.matnat.uio.no> Message-ID: <49B7BA5F.9050107@student.matnat.uio.no> Lisandro Dalcin wrote: > Of course, I could live if Cython does this: > > - Python semantics for '%' > - C semantics for an all-new operator '%%'. Just thinking on what > Python has done with '//' > BTW I'm +0 on a new %%. As long as there is also a function in the cython module to do the same while having Python-compatible syntax. And I'm definitely +1 on having "a % b" mean "a %% b". Dag Sverre From stefan_ml at behnel.de Wed Mar 11 14:36:11 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 11 Mar 2009 14:36:11 +0100 (CET) Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49B7BA5F.9050107@student.matnat.uio.no> References: <49B7697B.5090304@student.matnat.uio.no> <49B7BA5F.9050107@student.matnat.uio.no> Message-ID: <44325.213.61.181.86.1236778571.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: >> Of course, I could live if Cython does this: >> >> - Python semantics for '%' >> - C semantics for an all-new operator '%%'. Just thinking on what >> Python has done with '//' >> > BTW I'm +0 on a new %%. As long as there is also a function in the > cython module to do the same while having Python-compatible syntax. I vote for being careful with new operators. If Python ever decides that it needs a '%%' on its own for whatever reason, we'd better be in the stdlib first and have a good eye on python-dev at that point. I'm rather for considering "-1%16==-1" an odd corner case that's not worth being the default in Cython (as you said, the idea is to come from Python and step down into C at need). Having a special "cython.signed_mod()" function seems enough to support it. > And I'm definitely +1 on having "a % b" mean "a %% b". Fair enough, but my guess is that the C compiler would do that for us. Stefan From stefan_ml at behnel.de Wed Mar 11 14:41:32 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 11 Mar 2009 14:41:32 +0100 (CET) Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49B7B916.3080101@student.matnat.uio.no> References: <49B7697B.5090304@student.matnat.uio.no> <49B7B916.3080101@student.matnat.uio.no> Message-ID: <50632.213.61.181.86.1236778892.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > a) first implement things in Python (or at least in untyped Cython) > b) profile (or at least experience that things are too slow) > c) find out which functions you bother to type, and incrementally type > your program "bottom up". > > Now, if we suddenly change the meaning of operators depending on whether > they operate on Python ints or C ints, it means that an algorithm that > worked perfectly in step a) will break in step c). Which just means more > hassle, and that step c) become a bigger job than it needs to be. That's also where I see the difference to things like overflows: you have to think about them anyway when you restrict the value range through typing (e.g. a Python long into a C long). But having an operator change its result just because you just typed one of its input variables seems like a rather dangerous trap that's hard to track down. Stefan From dalcinl at gmail.com Wed Mar 11 15:31:23 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 11 Mar 2009 12:31:23 -0200 Subject: [Cython] PATCH: the right way to use PyErr_Format with Py_ssize_t arguments Message-ID: Likely Dag should review this... If you agree on the patch, please push ... IMHO, this fix should go to 0.11 While testing things on Linux 64, I had to implement this to do things the right way... Should work on 32/64 bits and Py 2.3 to 3.0 ... BTW, the only possible problems are the warnings below (they appear on 32 and also 64 bits)... Do not remember right now C semantics about that., perhaps this does not harm too much index.cpp: In function ?PyObject* __pyx_pf_5index_test_unsigned_long(PyObject*, PyObject*)?: index.cpp:714: warning: comparison between signed and unsigned integer expressions index.cpp:750: warning: comparison between signed and unsigned integer expressions -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 -------------- next part -------------- A non-text attachment was scrubbed... Name: FMT_SIZE_T.diff Type: application/octet-stream Size: 1413 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090311/e4a06445/attachment.obj From dagss at student.matnat.uio.no Wed Mar 11 15:43:02 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 15:43:02 +0100 Subject: [Cython] PATCH: the right way to use PyErr_Format with Py_ssize_t arguments In-Reply-To: References: Message-ID: <49B7CDF6.3020401@student.matnat.uio.no> Lisandro Dalcin wrote: > Likely Dag should review this... If you agree on the patch, please > push ... IMHO, this fix should go to 0.11 > I don't think I have anything to do with that code (though there's likely other instances of the same problem in my Buffer.py). a) Saying that this should go into 0.11 means saying that this specific bug is more important than everything else that sits in milestone 0.11.1 in Trac. We probably put out 0.10 with the same bug and the Cython project survived! We need to have some disipline and get things released -- release early, release often. b) I don't know if I'm the right person to review, I don't know too much about the Python API or size_t in this respect; it looks OK to me. (If I made the API in the first place I'd make the macro contain the entire string, i.e. "d" and "ld", and not simply the "l" or nothing. Some C versions have a "z" for size_t IIRC?) Dag Sverre From carl.witty at gmail.com Wed Mar 11 16:52:20 2009 From: carl.witty at gmail.com (Carl Witty) Date: Wed, 11 Mar 2009 07:52:20 -0800 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: Message-ID: On Tue, Mar 10, 2009 at 6:35 PM, Carl Witty wrote: > I just reported http://trac.cython.org/cython_trac/ticket/229, which > points out that division and % on cdef ints have C rather than Python > semantics (for instance, (-1 % 16) is -1 in C, and 15 in Python). One point I alluded to above, but maybe should emphasize: IMHO we should preserve that a == (a//b)*b + a%b. This means that we should either change both division and modulo, or neither. Carl From dalcinl at gmail.com Wed Mar 11 17:01:07 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 11 Mar 2009 14:01:07 -0200 Subject: [Cython] PATCH: the right way to use PyErr_Format with Py_ssize_t arguments In-Reply-To: <49B7CDF6.3020401@student.matnat.uio.no> References: <49B7CDF6.3020401@student.matnat.uio.no> Message-ID: On Wed, Mar 11, 2009 at 12:43 PM, Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: >> Likely Dag should review this... If you agree on the patch, please >> push ... IMHO, this fix should go to 0.11 >> > I don't think I have anything to do with that code (though there's > likely other instances of the same problem in my Buffer.py). > OK, sorry... For some reason I though that this was related to numpy support... > a) Saying that this should go into 0.11 means saying that this specific > bug is more important than everything else that sits in milestone 0.11.1 > in Trac. We probably put out 0.10 with the same bug and the Cython > project survived! We need to have some disipline and get things released > -- release early, release often. > I understand that... I just wanted to point that this fix likely does not have high impact and fix 64 bits issues, more or less like recent Robert's pushes did. > b) I don't know if I'm the right person to review, I don't know too much > about the Python API or size_t in this respect; it looks OK to me. Sorry again... > (If I > made the API in the first place I'd make the macro contain the entire > string, i.e. "d" and "ld", and not simply the "l" or nothing. My patch just mimics what CPython >= 2.5 does... > Some C versions have a "z" for size_t IIRC?) > In Py<2.5, Py_ssize_t is bare 'int' and the 'z' specifier is not handled in PyErr_Format(). -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Wed Mar 11 17:07:29 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 17:07:29 +0100 Subject: [Cython] PATCH: the right way to use PyErr_Format with Py_ssize_t arguments In-Reply-To: References: <49B7CDF6.3020401@student.matnat.uio.no> Message-ID: <49B7E1C1.7040601@student.matnat.uio.no> Lisandro Dalcin wrote: >> a) Saying that this should go into 0.11 means saying that this specific >> bug is more important than everything else that sits in milestone 0.11.1 >> in Trac. We probably put out 0.10 with the same bug and the Cython >> project survived! We need to have some disipline and get things released >> -- release early, release often. >> >> > > I understand that... I just wanted to point that this fix likely does > not have high impact and fix 64 bits issues, more or less like recent > Robert's pushes did. > Yes, sorry about the outburst, requesting this into 0.11 isn't all that unreasonable, you are right that it is low impact. Anyway, this is Robert's call in the end. >> b) I don't know if I'm the right person to review, I don't know too much >> about the Python API or size_t in this respect; it looks OK to me. >> > > Sorry again... > No problem. Thank you for cleaning up all of these 64-bit issues BTW. I'll try to remember to see whether Buffer.py has any... Dag Sverre From sturla at molden.no Wed Mar 11 17:08:48 2009 From: sturla at molden.no (Sturla Molden) Date: Wed, 11 Mar 2009 17:08:48 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: Message-ID: <49B7E210.1010007@molden.no> Carl Witty wrote: > > One point I alluded to above, but maybe should emphasize: > > IMHO we should preserve that a == (a//b)*b + a%b. This means that we > should either change both division and modulo, or neither. Please let simple C types behave like simple C types and Python objects like Python objects. Even if it means 1/1 will return 1.0 i Python 3 and 1 in Cython. You could let numerical literals be Python objects unless an explicit cast to C is made. Thus, "1" would a Python integer object and " 1" would be a C integer constant. That would ensure maximum similarity between Python and Cython, but be very annoying when writing numerical code in Cython. Sturla Molden From dagss at student.matnat.uio.no Wed Mar 11 17:19:35 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 17:19:35 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49B7E210.1010007@molden.no> References: <49B7E210.1010007@molden.no> Message-ID: <49B7E497.4050007@student.matnat.uio.no> Sturla Molden wrote: > Carl Witty wrote: > >> One point I alluded to above, but maybe should emphasize: >> >> IMHO we should preserve that a == (a//b)*b + a%b. This means that we >> should either change both division and modulo, or neither. >> > > Please let simple C types behave like simple C types and Python objects > like Python objects. Even if it means 1/1 will return 1.0 i Python 3 and > 1 in Cython. > Could you provide some specific examples where this is more convenient? I already argued the opposite from the "don't break algorithms when adding types for speed"-perspective (which is a very common way of using Cython). (I.e. it's fine that *your* knowledge of C is in place and that you expect Cython to behave like C here, but what will give least surprise to new Python user wanting to speed up his or her code? My point is that Cython users come from different backgrounds and will expect different things. When specific usecases are presented, we can bring this discussion beyond personal taste and what whomever uses Cython right now "already knows"). (BTW Python semantics for division means that Cython has Python 2 semantics for "/" unless from __future__ import division is specified.) Dag Sverre From psp.reachable at gmail.com Wed Mar 11 19:12:01 2009 From: psp.reachable at gmail.com (Prajwal Suhas P) Date: Wed, 11 Mar 2009 23:42:01 +0530 Subject: [Cython] Beginner-friendly tickets In-Reply-To: <49B783FA.5070309@student.matnat.uio.no> References: <49B783FA.5070309@student.matnat.uio.no> Message-ID: Hi Dag, I am currently looking at the tickets. I was looking at this - cdef int i, n n = 10 for i in range(n): if i == 5: n *= 2 print i The generated c code for this is - __pyx_v_5test1_n = 10; for (__pyx_v_5test1_i = 0; __pyx_v_5test1_i < __pyx_v_5test1_n; __pyx_v_5test1_i+=1) { __pyx_1 = (__pyx_v_5test1_i == 5); if (__pyx_1) { __pyx_v_5test1_n *= 2; goto __pyx_L4; } __pyx_L4:; } Can the var 'n' be held in another temp variable? Something like this - __pyx_v_5test1_n = 10; __pyx_v_5test1_const_n = 10; for (__pyx_v_5test1_i = 0; __pyx_v_5test1_i < __pyx_v_5test1_const_n; __pyx_v_5test1_i+=1) { __pyx_1 = (__pyx_v_5test1_i == 5); if (__pyx_1) { __pyx_v_5test1_n *= 2; goto __pyx_L4; } __pyx_L4:; } i--; I haven't looked at the cython code which parses the loop and outputs the c code. I would really appreciate if you can give me a head start into this and correct me if I am wrong. On Wed, Mar 11, 2009 at 2:57 PM, Dag Sverre Seljebotn < dagss at student.matnat.uio.no> wrote: > Since there's been some interest in how to get started coding on Cython > on the mailing list lately, I thought I'd call attention to these two > tickets: > > http://trac.cython.org/cython_trac/ticket/158 > http://trac.cython.org/cython_trac/ticket/203 > > These seems to fit the sweet spot of not being too hard to fix for > beginners (with some mentoring), but not being completely trivial either. > > I'm happy to provide an attack strategy/small amount of mentoring for > these tickets if anybody's interested. > > -- > Dag Sverre > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Regards, Prajwal S. "There is little difference in people,but this little difference makes a big difference.The little difference is attitude,the big difference is whether its positive or negative. " -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090311/e0ff98cc/attachment.htm From dagss at student.matnat.uio.no Wed Mar 11 19:44:19 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 19:44:19 +0100 Subject: [Cython] Beginner-friendly tickets In-Reply-To: References: <49B783FA.5070309@student.matnat.uio.no> Message-ID: <49B80683.4090800@student.matnat.uio.no> Prajwal Suhas P wrote: > Hi Dag, > > I am currently looking at the tickets. > > I was looking at this - > > cdef int i, n > n = 10 > for i in range(n): > if i == 5: > n *= 2 > print i > > > The generated c code for this is - > > __pyx_v_5test1_n = 10; > > for (__pyx_v_5test1_i = 0; __pyx_v_5test1_i < __pyx_v_5test1_n; > __pyx_v_5test1_i+=1) { > > __pyx_1 = (__pyx_v_5test1_i == 5); > if (__pyx_1) { > __pyx_v_5test1_n *= 2; > goto __pyx_L4; > } > __pyx_L4:; > } > > Can the var 'n' be held in another temp variable? > > Something like this - > > __pyx_v_5test1_n = 10; > __pyx_v_5test1_const_n = 10; > > for (__pyx_v_5test1_i = 0; __pyx_v_5test1_i < __pyx_v_5test1_const_n; > __pyx_v_5test1_i+=1) { > > __pyx_1 = (__pyx_v_5test1_i == 5); > if (__pyx_1) { > __pyx_v_5test1_n *= 2; > goto __pyx_L4; > } > __pyx_L4:; > } > i--; > > I haven't looked at the cython code which parses the loop and outputs > the c code. I would really appreciate if you can give me a head start > into this and correct me if I am wrong. You are correct, this is what you need to do. Some suggestions: a) Add a flag to ForFromStatNode in Nodes.py, say, "freeze_endpoints". b) IterationTransform in Optimize.py is what takes a "for i in range(...)" and turns it into a ForFromStatNode. Here you should turn the freeze_endpoints flag on when the ForFromStatNode is constructed. c) If this flag is set, then ForFromStatNode.generate_execution_code must put the start or end points (or both, depending on the case) into temporary variables. d) You can get hold of a temporary variable to use (like __pyx_t2) by calling code.funcstate.allocate_temp/release_temp. See Code.py for docs. (Stefan: Yes, this can be done by temp handling nodes in the transform as well, but since the canonical way of doing that is a bit unclear/a mess at the moment I don't want to encourage that. This works as well, and is likely less code in the end.) Another for-loop bug was also fixed with a bit of assistance on the mailing list, you may want to read through: http://thread.gmane.org/gmane.comp.python.cython.devel/4195 Also please make sure you make a testcase in tests/bugs/something_T203.pyx (then run this with python runtests.py -T203 while developing). -- Dag Sverre From dagss at student.matnat.uio.no Wed Mar 11 19:52:50 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 19:52:50 +0100 Subject: [Cython] Beginner-friendly tickets In-Reply-To: References: <49B783FA.5070309@student.matnat.uio.no> Message-ID: <49B80882.9030608@student.matnat.uio.no> Prajwal Suhas P wrote: > Something like this - > > __pyx_v_5test1_n = 10; > __pyx_v_5test1_const_n = 10; Acutally, it is more like __pyx_v_5test1_n = 10; __pyx_t1 = __pyx_v_5test1_n ... A better example than the one in the ticket is: """ >>> f() g called 0 1 2 2 """ cdef int g(): print "g called" return 3 def f(): cdef int i for i in range(g()): print i print i Here, "g called" will be printed three times if compiled in Cython; which is obviously not right. You should probably work with this example as well right away. Great that you're looking at this BTW. -- Dag Sverre From stefan_ml at behnel.de Wed Mar 11 20:01:51 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 11 Mar 2009 20:01:51 +0100 Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 In-Reply-To: References: <44706.213.61.181.86.1236700457.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49B80A9F.1000200@behnel.de> Hi Neal, Neal Becker wrote: > Can we have an rc2 with all proposed fixes? I'll plan to retest then. There's an rc3 now: http://cython.org/Cython-0.11.rc3.tar.gz Could you give it a try? If all works fine, this will become Cython 0.11 final. Thanks, Stefan From kwmsmith at gmail.com Wed Mar 11 21:22:51 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Wed, 11 Mar 2009 15:22:51 -0500 Subject: [Cython] Beginner-friendly tickets In-Reply-To: <49B783FA.5070309@student.matnat.uio.no> References: <49B783FA.5070309@student.matnat.uio.no> Message-ID: On Wed, Mar 11, 2009 at 4:27 AM, Dag Sverre Seljebotn wrote: > Since there's been some interest in how to get started coding on Cython > on the mailing list lately, I thought I'd call attention to these two > tickets: > > http://trac.cython.org/cython_trac/ticket/158 > http://trac.cython.org/cython_trac/ticket/203 > > These seems to fit the sweet spot of not being too hard to fix for > beginners (with some mentoring), but not being completely trivial either. > > I'm happy to provide an attack strategy/small amount of mentoring for > these tickets if anybody's interested. Hi Dag, I'm interested in working on #158 if it isn't already claimed. In the generated c code for these python functions: def nocrash1(): cdef str s = "Test" print "%s" % s def crash1(): print "%s" % s cdef str s = "Test" They differ only in that each line in the function body is converted into C in a different order. The internal string s is assigned to a non-null python object (the string "Test") in nocrash before being printed, while 's' is assiged to "Test" after trying to print in crash1. One solution (as mentioned in the ticket comments) would be to raise a compilation error since the s variable is declared after it's used. I imagine some checks would be in order to ensure this, but I don't have ideas about where to start. This function: def nocrash2(): print "%s" % s cdef str s Prints "None", although it should raise an exception since the analogous python code raises a "NameError." The generated C code puts the 'cdef str s' line before the 'print' line and sets s to None. This would be solved by the fix suggested above. The other solution involving flow analysis would be way beyond my abilities (or a bug fixable by a beginner). Kurt From dagss at student.matnat.uio.no Wed Mar 11 21:56:56 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 21:56:56 +0100 Subject: [Cython] Beginner-friendly tickets In-Reply-To: References: <49B783FA.5070309@student.matnat.uio.no> Message-ID: <49B82598.3000303@student.matnat.uio.no> Kurt Smith wrote: > On Wed, Mar 11, 2009 at 4:27 AM, Dag Sverre Seljebotn > wrote: >> Since there's been some interest in how to get started coding on Cython >> on the mailing list lately, I thought I'd call attention to these two >> tickets: >> >> http://trac.cython.org/cython_trac/ticket/158 >> http://trac.cython.org/cython_trac/ticket/203 >> >> These seems to fit the sweet spot of not being too hard to fix for >> beginners (with some mentoring), but not being completely trivial either. >> >> I'm happy to provide an attack strategy/small amount of mentoring for >> these tickets if anybody's interested. > > Hi Dag, > > I'm interested in working on #158 if it isn't already claimed. Nice! I haven't heard anything on #158 yet, so consider it yours. I've assigned both tickets now in trac. You may want to get Trac accounts if you haven't already BTW, send an .htpasswd-file to Robert for that. > One solution (as mentioned in the ticket comments) would be to raise a > compilation error since the s variable is declared after it's used. I > imagine some checks would be in order to ensure this, but I don't have > ideas about where to start. Yep, that's the solution I'm thinking about. The Cython "parse tree" is processed in stages. Two of those are analyse_declarations and analyse_expressions. The problem here is that "cdef str s" is handled in the analyse_declarations step and then disappears; while the analyse_expressions happen afterwards, and cannot know at which point in the source the declaration happened. So the analyse declarations phase must do this check. You will need to do your work in AnalyseDeclarationsTransform, in ParseTreeTransforms.py. It is a "filter" on the parse tree, and each method is called as you go according to the type of the node. Read some of the different transforms in that file to get the idea (to see the parse tree, insert e.g. "print node.dump()" at the beginning of visit_ModuleNode). Now: a) Keep a dict in self containing the names that has been referenced up to that point in the given scope (i.e. in visit_ModuleNode and visit_FuncDefNode you need to push/pop which dict one is using before/after processing the contents). b) Everywhere a name is used, a NameNode sits in the parse tree. So you create a visit_NameNode to record that the variable was used (it doesn't really matter whether it was assigned to or read; any usage of the variable before a cdef should be illegal). c) Finally, in visit_CVarDefNode you can now check whether the name has already been seen in the module/function, and if so there's a problem and you raise an exception (see other transforms for the convention here). Please ask again if/when you get stuck or anything was too brief. -- Dag Sverre From dagss at student.matnat.uio.no Wed Mar 11 21:59:40 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 21:59:40 +0100 Subject: [Cython] Beginner-friendly tickets In-Reply-To: <49B82598.3000303@student.matnat.uio.no> References: <49B783FA.5070309@student.matnat.uio.no> <49B82598.3000303@student.matnat.uio.no> Message-ID: <49B8263C.6090706@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Kurt Smith wrote: >> On Wed, Mar 11, 2009 at 4:27 AM, Dag Sverre Seljebotn >> wrote: >>> Since there's been some interest in how to get started coding on Cython >>> on the mailing list lately, I thought I'd call attention to these two >>> tickets: >>> >>> http://trac.cython.org/cython_trac/ticket/158 >>> http://trac.cython.org/cython_trac/ticket/203 >>> >>> These seems to fit the sweet spot of not being too hard to fix for >>> beginners (with some mentoring), but not being completely trivial either. >>> >>> I'm happy to provide an attack strategy/small amount of mentoring for >>> these tickets if anybody's interested. >> Hi Dag, >> >> I'm interested in working on #158 if it isn't already claimed. > > Nice! I haven't heard anything on #158 yet, so consider it yours. I've > assigned both tickets now in trac. > > You may want to get Trac accounts if you haven't already BTW, send an > .htpasswd-file to Robert for that. > >> One solution (as mentioned in the ticket comments) would be to raise a >> compilation error since the s variable is declared after it's used. I >> imagine some checks would be in order to ensure this, but I don't have >> ideas about where to start. > > Yep, that's the solution I'm thinking about. > > The Cython "parse tree" is processed in stages. Two of those are > analyse_declarations and analyse_expressions. The problem here is that > "cdef str s" is handled in the analyse_declarations step and then > disappears; while the analyse_expressions happen afterwards, and cannot > know at which point in the source the declaration happened. > > So the analyse declarations phase must do this check. > > You will need to do your work in AnalyseDeclarationsTransform, in > ParseTreeTransforms.py. It is a "filter" on the parse tree, and each > method is called as you go according to the type of the node. Read some > of the different transforms in that file to get the idea (to see the > parse tree, insert e.g. "print node.dump()" at the beginning of > visit_ModuleNode). > > Now: > > a) Keep a dict in self containing the names that has been referenced up > to that point in the given scope (i.e. in visit_ModuleNode and > visit_FuncDefNode you need to push/pop which dict one is using > before/after processing the contents). Actually I meant to say "set" everywhere I said "dict". -- Dag Sverre From greg.ewing at canterbury.ac.nz Wed Mar 11 22:35:37 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 12 Mar 2009 10:35:37 +1300 Subject: [Cython] Update on: Incomplete C++ example In-Reply-To: <99214b470903101433w2cb1a2f4x896fd11203f97a14@mail.gmail.com> References: <99214b470903101433w2cb1a2f4x896fd11203f97a14@mail.gmail.com> Message-ID: <49B82EA9.2030306@canterbury.ac.nz> Luke wrote: > it would be nice to have a step by step example of > the process that is illustrative of how to wrap a C++ library Generally agreed, but > Things that were not helpful in the setup.py were things like: > include_dirs=[...], # usual stuff > libraries=[...], # ditto > extra_link_args=[...], # if needed None of that is specific to wrapping C++, it's all just general distutils knowledge. I don't think it's reasonable to expect a C++ wrapping tutorial to include complete handholding on all aspects of distutils. A self-contained example wouldn't need any of those options, anyway. -- Greg From robertwb at math.washington.edu Wed Mar 11 23:51:26 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Mar 2009 15:51:26 -0700 Subject: [Cython] too late to review and push patch for better handling of integral C types ? In-Reply-To: <49B584AC.6040506@behnel.de> References: <49B584AC.6040506@behnel.de> Message-ID: On Mar 9, 2009, at 2:05 PM, Stefan Behnel wrote: > Lisandro Dalcin wrote: >> Sending again, former message bounced because of 40k limit. >> >> The patch is almost ready, just waiting for review. The only thing I >> did not take is to remove the usage of PyNumber_Int(). Anyway, Cython >> already uses it, so the "brokeness" of my patch is similar to the >> current one :-) .. As I've introduced a __Pyx_PyNumber_Int() >> function, >> that is the only place that we should change to implement things >> as we >> want. > > I skipped through your patch a little. It's a fairly involved > change that > requires some testing, especially on different architectures and > platforms. > It might well be that the user experience is positive in that it > helps in > finding bugs and making user code more portable. My current gut > feeling is > that code that depends on the old behaviour here is broken anyway > and just > wasn't fixed yet because no-one noticed. > > But in any case, I would like to avoid breaking code for a micro > (bug-fix!) > release, unless there is at least some evidence that users > appreciate it. > > So, if anyone has code that does a bit of integer conversion > between Python > and C space, especially when working on a non-Intel or 64 bit > platform, > please give this patch a try. Sage does this :). It's too late for 0.11 though. Is there a ticket up? - Robert From robertwb at math.washington.edu Wed Mar 11 23:54:29 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Mar 2009 15:54:29 -0700 Subject: [Cython] Update on: Incomplete C++ example In-Reply-To: <49B82EA9.2030306@canterbury.ac.nz> References: <99214b470903101433w2cb1a2f4x896fd11203f97a14@mail.gmail.com> <49B82EA9.2030306@canterbury.ac.nz> Message-ID: <613B0D07-81E4-4CBE-8BE5-388DFF0DDF21@math.washington.edu> On Mar 11, 2009, at 2:35 PM, Greg Ewing wrote: > Luke wrote: > >> it would be nice to have a step by step example of >> the process that is illustrative of how to wrap a C++ library > > Generally agreed, but > >> Things that were not helpful in the setup.py were things like: >> include_dirs=[...], # usual stuff >> libraries=[...], # ditto >> extra_link_args=[...], # if needed > > None of that is specific to wrapping C++, it's all > just general distutils knowledge. I don't think it's > reasonable to expect a C++ wrapping tutorial to > include complete handholding on all aspects of > distutils. > > A self-contained example wouldn't need any of those > options, anyway. What we should have is a link to the distutils documentation here. Part of "how to wrap a C++ library" is "how to wrap a library." - Robert From jasone at canonware.com Wed Mar 11 23:57:14 2009 From: jasone at canonware.com (Jason Evans) Date: Wed, 11 Mar 2009 15:57:14 -0700 Subject: [Cython] Possible blocker for 0.11 Message-ID: <49B841CA.8000805@canonware.com> I hit what appears to be a Cython compiler bug this afternoon (related to temp allocation?) that might be important to fix before 0.11 is released. I haven't had time yet to reduce it to a small test case. What is the tentative release schedule? Thanks, Jason From robertwb at math.washington.edu Thu Mar 12 00:01:13 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Mar 2009 16:01:13 -0700 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: <5770E6DE-385E-46BE-9070-0AD57466C5A1@math.washington.edu> References: <52637.213.61.181.86.1236695813.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <61316.213.61.181.86.1236699523.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5770E6DE-385E-46BE-9070-0AD57466C5A1@math.washington.edu> Message-ID: On Mar 10, 2009, at 11:08 AM, Robert Bradshaw wrote: > I'll fix this. The problem is that when passing via the inline > function, the size and signness of the original index gets lost. The > very common and much faster case is to use PyInt_FromLong, but of > course we have to handle bigger indices as well. > > - Robert > > > On Mar 10, 2009, at 9:19 AM, Lisandro Dalcin wrote: > >> Here you have a trivial patch... However, all this part smells so >> bad... >> >> >> On Tue, Mar 10, 2009 at 1:38 PM, Stefan Behnel >> wrote: >>> Lisandro Dalcin wrote: >>>>>> Now testing at least starts, but seems to have some errors: >>>>> >>>>> * warning: left shift count >= width of type Note that this is only called when sizeof(Py_ssize_t) < sizeof (unsigned long long). >>>>> I don't get these locally. The problem is in __Pyx_GetItemGeneric, >>>>> second >>>>> line: "1ULL << (sizeof(Py_ssize_t)*8)". Looks like that's not >>>>> 64bit >>>>> clean. >>>> >>>> PyInt_FromSsize_t() should be used here.... Well, it's a bit more complicated than this. However, I will admit, though I tried to cover all the cases, the original code was quite hackish. I've come up with a better solution that doesn't sacrifice speed for the common cases. RC3 is up for testing. - Robert From robertwb at math.washington.edu Thu Mar 12 00:04:50 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Mar 2009 16:04:50 -0700 Subject: [Cython] Possible blocker for 0.11 In-Reply-To: <49B841CA.8000805@canonware.com> References: <49B841CA.8000805@canonware.com> Message-ID: On Mar 11, 2009, at 3:57 PM, Jason Evans wrote: > I hit what appears to be a Cython compiler bug this afternoon (related > to temp allocation?) that might be important to fix before 0.11 is > released. I haven't had time yet to reduce it to a small test case. > What is the tentative release schedule? We're hoping that the latest rc fixes Neal's issues with building/ testing RMPs, and rc3 will be the final release. That being said, please provide an example and make a trac ticket. Unless it's really bad, we'll probably wait until 0.11.1, which I hope will be released on a much shorter timescale. - Robert From jasone at canonware.com Thu Mar 12 00:10:32 2009 From: jasone at canonware.com (Jason Evans) Date: Wed, 11 Mar 2009 16:10:32 -0700 Subject: [Cython] Possible blocker for 0.11 In-Reply-To: References: <49B841CA.8000805@canonware.com> Message-ID: <49B844E8.8050301@canonware.com> Robert Bradshaw wrote: > We're hoping that the latest rc fixes Neal's issues with building/ > testing RMPs, and rc3 will be the final release. > > That being said, please provide an example and make a trac ticket. Okay, I'm working on it now. > Unless it's really bad, we'll probably wait until 0.11.1, which I > hope will be released on a much shorter timescale. It's causing a reference to an uninitialized temp variable (which should contain the result of a conditional branch test), which causes my app to crash, so this particular manifestation is pretty bad. However, it isn't yet clear whether this is a common problem, or a corner case due to the structure of this particular (rather simple) method. Thanks, Jason From robertwb at math.washington.edu Thu Mar 12 00:17:21 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Mar 2009 16:17:21 -0700 Subject: [Cython] PATCH: the right way to use PyErr_Format with Py_ssize_t arguments In-Reply-To: References: <49B7CDF6.3020401@student.matnat.uio.no> Message-ID: <2E671135-5B33-4551-82FB-183486183E88@math.washington.edu> On Mar 11, 2009, at 9:01 AM, Lisandro Dalcin wrote: > On Wed, Mar 11, 2009 at 12:43 PM, Dag Sverre Seljebotn > wrote: >> Lisandro Dalcin wrote: >>> Likely Dag should review this... If you agree on the patch, please >>> push ... IMHO, this fix should go to 0.11 The patch looks good, can you make a ticket for it? Also, can you use mercurial to output diffs so we get your name and a comment with it as well? >> a) Saying that this should go into 0.11 means saying that this >> specific >> bug is more important than everything else that sits in milestone >> 0.11.1 >> in Trac. We probably put out 0.10 with the same bug and the Cython >> project survived! We need to have some disipline and get things >> released >> -- release early, release often. Yes, we need to return to the "release early, release often" schedule. We can start putting patches up and getting reviews for them on trac for 0.11.1 right away. I've added milestone 0.11.2, everything that doesn't have a patch or easy fix for 0.11.1 will get moved there so we can release 0.11.1 shortly. > I understand that... I just wanted to point that this fix likely does > not have high impact and fix 64 bits issues, more or less like recent > Robert's pushes did. The last push was to facilitate getting cython 0.11 into the rpm repo, though in retrospect I guess it's lower priority than some other stuff. At this point, unless there's a major bug, the only thing I can see adding is perhaps testcases. - Robert From jasone at canonware.com Thu Mar 12 00:36:18 2009 From: jasone at canonware.com (Jason Evans) Date: Wed, 11 Mar 2009 16:36:18 -0700 Subject: [Cython] Possible blocker for 0.11 In-Reply-To: <49B841CA.8000805@canonware.com> References: <49B841CA.8000805@canonware.com> Message-ID: <49B84AF2.7010300@canonware.com> Jason Evans wrote: > I hit what appears to be a Cython compiler bug this afternoon (related > to temp allocation?) that might be important to fix before 0.11 is > released. False alarm; there's a bug in my program. I misinterpreted valgrind output, then misread Cython-generated code. Cython is dutifully giving me the wrong answer I asked for. Jason From robertwb at math.washington.edu Thu Mar 12 00:38:51 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Mar 2009 16:38:51 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: Message-ID: On Mar 10, 2009, at 7:35 PM, Carl Witty wrote: > I just reported http://trac.cython.org/cython_trac/ticket/229, which > points out that division and % on cdef ints have C rather than Python > semantics (for instance, (-1 % 16) is -1 in C, and 15 in Python). > > I'm not sure what the right thing to do here is. Using C semantics > gives faster code. Using Python semantics is nicer (whenever I care > about the difference, I always want Python semantics) but slower, and > would be confusing to somebody who was coming from C. Changing this to use Python semantics would actually slow down real code in this case. I'm thinking specifically of linear algebra mod n (for word sized n) in Sage, but I'm sure there are many other places as well. So I'm generally -1 for such a change. I also don't like the idea of an new operator (%%), and I think Carl Witty has a good point about needing to change the semantics of // as well. Also, I think making literals into Python objects by default will be way cumbersome. > But whatever happens should be prominently documented; I couldn't find > any mention of the issue in the current docs (although maybe I wasn't > looking in the right place). +10 I think this is neither the first nor last issue that will come up like this. What I think we should do is have a "-pedantic" flag which obeys Python semantics exactly, at the expense of speed. This applies to %, //, **, etc. (Overflowing is a separate issue, and I think there should be a flag for that too.) Compilation of .py files would be, by default, with -pedantic enabled, and compilation of .pyx files with it disabled. If one suspects that something subtle like this is changing your algorithm, you can compile with -pedantic and confirm (or deny) your suspicions. Also, everything that differs will query this flag in the code, making it easy to find and *document* the discrepancies. In fact, the annotate feature could even make note of such places. Thoughts? - Robert P.S. As an implementation aside, it can be much faster to write r = a % b; r += (r < 0) * b than r = a % b; if r < 0: r += b especially if one is doing this in a loop with randomly distributed (pos/neg) a. From robertwb at math.washington.edu Thu Mar 12 00:42:11 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Mar 2009 16:42:11 -0700 Subject: [Cython] Possible blocker for 0.11 In-Reply-To: <49B84AF2.7010300@canonware.com> References: <49B841CA.8000805@canonware.com> <49B84AF2.7010300@canonware.com> Message-ID: <8800314B-D309-4D09-AEA5-4FC6C67135D8@math.washington.edu> On Mar 11, 2009, at 4:36 PM, Jason Evans wrote: > Jason Evans wrote: >> I hit what appears to be a Cython compiler bug this afternoon >> (related >> to temp allocation?) that might be important to fix before 0.11 is >> released. > > False alarm; there's a bug in my program. I misinterpreted valgrind > output, then misread Cython-generated code. Cython is dutifully > giving > me the wrong answer I asked for. Well that's good news from my point of view :). Thanks for the follow up. - Robert From robertwb at math.washington.edu Thu Mar 12 00:42:38 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Mar 2009 16:42:38 -0700 Subject: [Cython] Wrapping C++ functions that pass by reference In-Reply-To: <49B7B707.4000303@student.matnat.uio.no> References: <99214b470903101634i16f79657pe008fd1bc75a9533@mail.gmail.com> <49B7AEFF.1000302@molden.no> <49B7B707.4000303@student.matnat.uio.no> Message-ID: On Mar 11, 2009, at 6:05 AM, Dag Sverre Seljebotn wrote: > Sturla Molden wrote: >> Luke wrote: >> >>> I have a C++ function like this: >>> >>> void constraint(double& x, double& dxdt) >>> { >>> dxdt[0] = ... >>> dxdt[1] = ... >>> .... >>> dxdt[n] = ... >>> } >>> >>> How would I properly wrap this function in the .pyx file? And how >>> would calling it in Python work? >>> >> I think you can just say >> >> cdef extern from "constraint.h": >> void constraint(double x, double dxdt) >> >> as the generated C++ should be identical. >> > Good point! Please disregard my previous mail about this, I was wrong, > and this is right. Is this documented in the wiki anywhere? From robertwb at math.washington.edu Thu Mar 12 00:43:18 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Mar 2009 16:43:18 -0700 Subject: [Cython] Remove "parsetuple_format" from PyrexTypes? In-Reply-To: <49B6C647.9090807@behnel.de> References: <49B6C647.9090807@behnel.de> Message-ID: On Mar 10, 2009, at 12:57 PM, Stefan Behnel wrote: > Hi, > > I just fixed an unused variable problem when using (Py_s)size_t as > function > argument types (fix can wait for 0.11.1). Doing this, I noticed > that the > type indication character that each type stores in > "parsetuple_format" is > no longer used anywhere since PyArg_ParseTupleAndKeywords() was > replaced by > our own code. Before I rip this attribute straight out, can anyone > still > see a use case for this? Maybe in combination with buffer content > types? Yes, I think we can get rid of this. - Robert From robertwb at math.washington.edu Thu Mar 12 00:56:00 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Mar 2009 16:56:00 -0700 Subject: [Cython] Integer power operator In-Reply-To: <49B7892E.2010901@student.matnat.uio.no> References: <1286d6e2177fb46620ab4a1f1fa7d7f3.squirrel@webmail.uio.no> <39764.213.61.181.86.1236762819.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B78660.8060904@student.matnat.uio.no> <49B7892E.2010901@student.matnat.uio.no> Message-ID: On Mar 11, 2009, at 2:49 AM, Dag Sverre Seljebotn wrote: > Dag Sverre Seljebotn wrote: >> Stefan Behnel wrote: >>> Dag Sverre Seljebotn wrote: >>>> I happen to routinely write (some complex integer expression)**2 in >>>> Python, and it is tedious to manually store the expression in a >>>> temporary >>>> etc. in Cython. >>> >>> If you can come up with a sensible utility function that handles the >>> different C compile time and runtime overflow cases nicely, I'm >>> all for >>> it. >> >> So you are "+1 if I include an overflow check"? >> >> I was just thinking of letting it silently overflow, the way addition >> and multiplication already silently overflows. >> > > To be clear, I'm in favour of overflow checks, I just think they in > that > case should apply in *all* situations, not simply in this one > operator. > Otherwise Cython will be inconsistent and it makes the language harder > to learn. > > So I'm thinking this: > 1) Add integer ** without overflow check > 2) Add overflowcheck compiler directive, similar to the nonecheck > directive, which checks for overflows in /all/ situations (which > defaults to off though) +1 We should special constant case base 2 (or perhaps base power of 2) and constant exponent up to, say, 5 or so. - Robert From carl.witty at gmail.com Thu Mar 12 01:14:31 2009 From: carl.witty at gmail.com (Carl Witty) Date: Wed, 11 Mar 2009 17:14:31 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: Message-ID: On Wed, Mar 11, 2009 at 4:38 PM, Robert Bradshaw wrote: > I think this is neither the first nor last issue that will come up > like this. What I think we should do is have a "-pedantic" flag which > obeys Python semantics exactly, at the expense of speed. This applies > to %, //, **, etc. (Overflowing is a separate issue, and I think > there should be a flag for that too.) Compilation of .py files would > be, by default, with -pedantic enabled, and compilation of .pyx files > with it disabled. > > If one suspects that something subtle like this is changing your > algorithm, you can compile with -pedantic and confirm (or deny) your > suspicions. Also, everything that differs will query this flag in the > code, making it easy to find and *document* the discrepancies. In > fact, the annotate feature could even make note of such places. > > Thoughts? This doesn't feel like quite the right plan... Consider a file which is edited by both an expert (defined as "somebody who happens to remember that C and Python have different semantics for signed division") and a novice (defined as "somebody who doesn't remember about any signed division issues"; note that the same person can be an expert one day and a novice the next). If the novice is going to be switching -pedantic on and off, doesn't that mean that the expert had better write code that works either way? That seems worse than being able to depend on one behavior or the other. Maybe this would be OK combined with builtins ("floordiv"/"floormod" and "truncdiv"/"truncmod", perhaps) that were guaranteed not to change with -pedantic. Carl From robertwb at math.washington.edu Thu Mar 12 03:32:28 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Mar 2009 19:32:28 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: Message-ID: On Mar 11, 2009, at 5:14 PM, Carl Witty wrote: > On Wed, Mar 11, 2009 at 4:38 PM, Robert Bradshaw > wrote: >> I think this is neither the first nor last issue that will come up >> like this. What I think we should do is have a "-pedantic" flag which >> obeys Python semantics exactly, at the expense of speed. This applies >> to %, //, **, etc. (Overflowing is a separate issue, and I think >> there should be a flag for that too.) Compilation of .py files would >> be, by default, with -pedantic enabled, and compilation of .pyx files >> with it disabled. >> >> If one suspects that something subtle like this is changing your >> algorithm, you can compile with -pedantic and confirm (or deny) your >> suspicions. Also, everything that differs will query this flag in the >> code, making it easy to find and *document* the discrepancies. In >> fact, the annotate feature could even make note of such places. >> >> Thoughts? > > This doesn't feel like quite the right plan... Consider a file which > is edited by both an expert (defined as "somebody who happens to > remember that C and Python have different semantics for signed > division") and a novice (defined as "somebody who doesn't remember > about any signed division issues"; note that the same person can be an > expert one day and a novice the next). If the novice is going to be > switching -pedantic on and off, doesn't that mean that the expert had > better write code that works either way? That seems worse than being > able to depend on one behavior or the other. I wasn't imagining something people would be switch on and off... it would be more of a useful tool for when people are converting Python code to Cython code. We could make this a pragma too. > Maybe this would be OK combined with builtins ("floordiv"/"floormod" > and "truncdiv"/"truncmod", perhaps) that were guaranteed not to change > with -pedantic. Yes, I think we could provide such functions as well, though it may be cumbersome to use them. Surrounding functions/blocks by pragmas would do the trick as well, in case one needed to depend on a certain behavior. Most of this stuff is corner cases that are just slower to test/special case for, and people wouldn't be using Cython if they didn't care about speed. One of the primary advantages of this approach is that it would facilitate very good documentation about exactly what (and why) there are any discrepancies between pure Python and Cython code--and yes I'm making the assumption that sometimes it's worth doing some things differently with C types for efficiency reasons. - Robert From stefan_ml at behnel.de Thu Mar 12 07:13:43 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 12 Mar 2009 07:13:43 +0100 Subject: [Cython] too late to review and push patch for better handling of integral C types ? In-Reply-To: References: <49B584AC.6040506@behnel.de> Message-ID: <49B8A817.1090609@behnel.de> Robert Bradshaw wrote: > It's too late for 0.11 though. Is there a ticket up? Don't think so, but while we're at it, I'd like to have a ticket for any change going into 0.11.1. Lisandro, could you open one and post the patch there? Stefan From stefan_ml at behnel.de Thu Mar 12 07:20:31 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 12 Mar 2009 07:20:31 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: Message-ID: <49B8A9AF.6060806@behnel.de> Robert Bradshaw wrote: > I think this is neither the first nor last issue that will come up > like this. What I think we should do is have a "-pedantic" flag which > obeys Python semantics exactly, at the expense of speed. This applies > to %, //, **, etc. (Overflowing is a separate issue, and I think > there should be a flag for that too.) Compilation of .py files would > be, by default, with -pedantic enabled, and compilation of .pyx files > with it disabled. Wouldn't it be better to have such a flag emit warnings (preferably with a hint how to write better code) instead of changing the semantics of operators? I think having operators change their semantics without a clear change in the user code (such as a __future__ import) is a rather dangerous 'feature'. Stefan From stefan_ml at behnel.de Thu Mar 12 07:39:20 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 12 Mar 2009 07:39:20 +0100 Subject: [Cython] Trac usage and bug handling for 0.11.x Message-ID: <49B8AE18.9040109@behnel.de> Hi all, I would like to encourage the following bug fixing procedure for the future 0.11.x releases. 1) Every patch that goes into a bug-fix release of Cython must be backed by a trac ticket. To get a trac account, please send a htpasswd entry to Robert Bradshaw. 2) Every trac ticket must have a bug test case associated with it. This means that there must be a failing test case in tests/bugs/ named "nicely_descriptive_name_here_Txyz.pyx" (where 'xyz' is the ticket number), or a patch in trac that adds this test case. See http://wiki.cython.org/HackerGuide Please try to do this even for the tricky cases that feel like there isn't a good test case. Reproducing a bug is critical for fixing it, and having a test case is critical for not breaking it in the future. 3) A working patch attached to the ticket will definitely accelerate the mainline bug fixing, although if you are unsure where to get started, it's usually better to ask on the mailing list before getting lost. 4) The Cython developers will assign priorities and milestones to trac tickets. Tickets with working patches should be handled with priority and should at least receive a timely review. 5) When a bug is fixed, the test case must be copied from tests/bugs/ to tests/run/ and the test suite must pass. Can everyone agree on this? Stefan From robertwb at math.washington.edu Thu Mar 12 07:46:41 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Mar 2009 23:46:41 -0700 Subject: [Cython] Trac usage and bug handling for 0.11.x In-Reply-To: <49B8AE18.9040109@behnel.de> References: <49B8AE18.9040109@behnel.de> Message-ID: <3BE155F1-FE7E-41AA-99D8-E53A7188E70F@math.washington.edu> On Mar 11, 2009, at 11:39 PM, Stefan Behnel wrote: > Hi all, > > I would like to encourage the following bug fixing procedure for > the future > 0.11.x releases. > > 1) Every patch that goes into a bug-fix release of Cython must be > backed by > a trac ticket. To get a trac account, please send a htpasswd entry to > Robert Bradshaw. > > 2) Every trac ticket must have a bug test case associated with it. > This > means that there must be a failing test case in tests/bugs/ named > "nicely_descriptive_name_here_Txyz.pyx" (where 'xyz' is the ticket > number), > or a patch in trac that adds this test case. See > > http://wiki.cython.org/HackerGuide > > Please try to do this even for the tricky cases that feel like > there isn't > a good test case. Reproducing a bug is critical for fixing it, and > having a > test case is critical for not breaking it in the future. > > 3) A working patch attached to the ticket will definitely > accelerate the > mainline bug fixing, although if you are unsure where to get > started, it's > usually better to ask on the mailing list before getting lost. > > 4) The Cython developers will assign priorities and milestones to trac > tickets. Tickets with working patches should be handled with > priority and > should at least receive a timely review. > > 5) When a bug is fixed, the test case must be copied from tests/ > bugs/ to > tests/run/ and the test suite must pass. +1. In fact, I think even non-bugfixes should all be assigned a ticket on trac. (Some features may, of course, span many patches.) The only thing I would say is that I'd rather not clutter tests/run/-- perhaps we could put them somewhere else (or alternatively have runtests test all but a static list of known open tickets). - Robert From carl.witty at gmail.com Thu Mar 12 07:53:52 2009 From: carl.witty at gmail.com (Carl Witty) Date: Wed, 11 Mar 2009 23:53:52 -0700 Subject: [Cython] Trac usage and bug handling for 0.11.x In-Reply-To: <49B8AE18.9040109@behnel.de> References: <49B8AE18.9040109@behnel.de> Message-ID: On Wed, Mar 11, 2009 at 11:39 PM, Stefan Behnel wrote: > 2) Every trac ticket must have a bug test case associated with it. This > means that there must be a failing test case in tests/bugs/ named > "nicely_descriptive_name_here_Txyz.pyx" (where 'xyz' is the ticket number), > or a patch in trac that adds this test case. See > > http://wiki.cython.org/HackerGuide > > Please try to do this even for the tricky cases that feel like there isn't > a good test case. Reproducing a bug is critical for fixing it, and having a > test case is critical for not breaking it in the future. Does that mean I can't report bugs to trac anymore, without constructing a test case in the form of a Cython patch? If so, then -0. (I probably just wouldn't bother reporting bugs any more under such a rule.) If you just mean that every patch that purports to fix a bug must also include a test case, then +1 to that (and to the rest of your list). Carl From robertwb at math.washington.edu Thu Mar 12 07:56:02 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Mar 2009 23:56:02 -0700 Subject: [Cython] Google summer of code In-Reply-To: <993FE1B3-30D3-45C7-9E40-B8C420E2492C@math.washington.edu> References: <49B2E58A.9090900@behnel.de> <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> <993FE1B3-30D3-45C7-9E40-B8C420E2492C@math.washington.edu> Message-ID: <253B6623-3D04-47F7-B870-A74E420C7D6E@math.washington.edu> On Mar 7, 2009, at 3:11 PM, Robert Bradshaw wrote: > I'm a "probably," but it's sounding like at this point we don't have > enough potential mentors to be a full mentoring organization unless > someone else stands up. On the numerical side of things, perhaps a > NumPy person would be willing. > > On the other hand, I think we're higher on Python's radar than last > year, so I'm hopeful of at least getting one (maybe two) spots for > them. The quality of the students and projects will play a huge role > here--one thing I learned at the summit last year is that it's not > even worth you time to go after sub-par projects (though good > students are an extremely good return on the investment of mentoring > them). Just a reminder, the organization app is due Friday if we decide to do it. Another advantage of being a mentor organization is that we get $500/ student. I'm not exactly sure what Cython would do with the money at this point (we don't have infrastructure to support, it's not enough to fund a get-together) I would propose it would be divvied out to the mentors to help offset the time they put into it. (They Python Foundation keeps the $500/student, which I'm fine with as they're big enough to have actual costs.) But being under an umbrella organization worked out well last time and is certainly less work and more flexibility on our part. - Robert From dagss at student.matnat.uio.no Thu Mar 12 07:59:50 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 12 Mar 2009 07:59:50 +0100 (CET) Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: Message-ID: Robert Bradshaw wrote: > On Mar 10, 2009, at 7:35 PM, Carl Witty wrote: > >> I just reported http://trac.cython.org/cython_trac/ticket/229, which >> points out that division and % on cdef ints have C rather than Python >> semantics (for instance, (-1 % 16) is -1 in C, and 15 in Python). >> >> I'm not sure what the right thing to do here is. Using C semantics >> gives faster code. Using Python semantics is nicer (whenever I care >> about the difference, I always want Python semantics) but slower, and >> would be confusing to somebody who was coming from C. > > Changing this to use Python semantics would actually slow down real > code in this case. I'm thinking specifically of linear algebra mod n > (for word sized n) in Sage, but I'm sure there are many other places > as well. So I'm generally -1 for such a change. I also don't like the > idea of an new operator (%%), and I think Carl Witty has a good point > about needing to change the semantics of // as well. Also, I think > making literals into Python objects by default will be way cumbersome. > >> But whatever happens should be prominently documented; I couldn't find >> any mention of the issue in the current docs (although maybe I wasn't >> looking in the right place). > > +10 > > I think this is neither the first nor last issue that will come up > like this. What I'd think we should do is have a "-pedantic" flag which > obeys Python semantics exactly, at the expense of speed. This applies > to %, //, **, etc. (Overflowing is a separate issue, and I think > there should be a flag for that too.) Compilation of .py files would > be, by default, with -pedantic enabled, and compilation of .pyx files > with it disabled. Like Carl I have problem with a pedantic flag because it gives two "modes" to worry about (which mode is this code written in? etc.) and just headache in general. So I'm much more in favor of a specific pragma for this (cython.fastdivision), which can be inserted in source visibly as a decorator (as much for documentation purposes as anything else). (Then the disagreement boils down to what should be the default value of that pragma, I suppose.). Dag Sverre From robertwb at math.washington.edu Thu Mar 12 08:02:09 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 12 Mar 2009 00:02:09 -0700 Subject: [Cython] Trac usage and bug handling for 0.11.x In-Reply-To: References: <49B8AE18.9040109@behnel.de> Message-ID: <77B03456-43CB-4ED8-A087-922B85F260DC@math.washington.edu> On Mar 11, 2009, at 11:53 PM, Carl Witty wrote: > On Wed, Mar 11, 2009 at 11:39 PM, Stefan Behnel > wrote: >> 2) Every trac ticket must have a bug test case associated with it. >> This >> means that there must be a failing test case in tests/bugs/ named >> "nicely_descriptive_name_here_Txyz.pyx" (where 'xyz' is the ticket >> number), >> or a patch in trac that adds this test case. See >> >> http://wiki.cython.org/HackerGuide >> >> Please try to do this even for the tricky cases that feel like >> there isn't >> a good test case. Reproducing a bug is critical for fixing it, and >> having a >> test case is critical for not breaking it in the future. > > Does that mean I can't report bugs to trac anymore, without > constructing a test case in the form of a Cython patch? If so, then > -0. (I probably just wouldn't bother reporting bugs any more under > such a rule.) My interpretation is that bugs won't get assigned a release milestone without a clear testcase. This will probably impact how much it's worked on as well. The testcase does not need to be in the form of a patch (until we actually commit it). If I work on a trac ticket with an "inline" testcase, I'll probably go ahead and add it to the /tests/ bugs directory myself. > If you just mean that every patch that purports to fix a bug must also > include a test case, then +1 to that (and to the rest of your list). Yes, for sure on this point. - Robert From stefan_ml at behnel.de Thu Mar 12 08:03:41 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 12 Mar 2009 08:03:41 +0100 Subject: [Cython] Trac usage and bug handling for 0.11.x In-Reply-To: References: <49B8AE18.9040109@behnel.de> Message-ID: <49B8B3CD.4020308@behnel.de> Carl Witty wrote: > On Wed, Mar 11, 2009 at 11:39 PM, Stefan Behnel wrote: >> 2) Every trac ticket must have a bug test case associated with it. This >> means that there must be a failing test case in tests/bugs/ named >> "nicely_descriptive_name_here_Txyz.pyx" (where 'xyz' is the ticket number), >> or a patch in trac that adds this test case. See >> >> http://wiki.cython.org/HackerGuide >> >> Please try to do this even for the tricky cases that feel like there isn't >> a good test case. Reproducing a bug is critical for fixing it, and having a >> test case is critical for not breaking it in the future. > > Does that mean I can't report bugs to trac anymore, without > constructing a test case in the form of a Cython patch? If so, then > -0. (I probably just wouldn't bother reporting bugs any more under > such a rule.) No, reporting bugs is open to everyone. The above doesn't mean the reporter of a bug *must* be the one to provide the test case. I just wanted to stress the fact that we need a test case that must go in before the fix, and that a test case makes it easier for us to reproduce and fix the bug. So there is a reason to prioritise tickets that contain a test case, as tickets without a test case are more work. I'll clarify that on a Wiki page later today. Stefan From dagss at student.matnat.uio.no Thu Mar 12 08:07:36 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 12 Mar 2009 08:07:36 +0100 (CET) Subject: [Cython] Google summer of code In-Reply-To: <253B6623-3D04-47F7-B870-A74E420C7D6E@math.washington.edu> References: <49B2E58A.9090900@behnel.de> <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> <993FE1B3-30D3-45C7-9E40-B8C420E2492C@math.washington.edu> <253B6623-3D04-47F7-B870-A74E420C7D6E@math.washington.edu> Message-ID: <6528b00c5b8779160ffe2dc3ec1f2cfb.squirrel@webmail.uio.no> > Robert Bradshaw wrote: > On Mar 7, 2009, at 3:11 PM, Robert Bradshaw wrote: > >> I'm a "probably," but it's sounding like at this point we don't have >> enough potential mentors to be a full mentoring organization unless >> someone else stands up. On the numerical side of things, perhaps a >> NumPy person would be willing. >> >> On the other hand, I think we're higher on Python's radar than last >> year, so I'm hopeful of at least getting one (maybe two) spots for >> them. The quality of the students and projects will play a huge role >> here--one thing I learned at the summit last year is that it's not >> even worth you time to go after sub-par projects (though good >> students are an extremely good return on the investment of mentoring >> them). > > Just a reminder, the organization app is due Friday if we decide to > do it. > > Another advantage of being a mentor organization is that we get $500/ > student. I'm not exactly sure what Cython would do with the money at > this point (we don't have infrastructure to support, it's not enough > to fund a get-together) I would propose it would be divvied out to > the mentors to help offset the time they put into it. (They Python > Foundation keeps the $500/student, which I'm fine with as they're big > enough to have actual costs.) But being under an umbrella > organization worked out well last time and is certainly less work and > more flexibility on our part. This was up on the NumPy list -- they basically decided to go with the PSF for lack of manpower to put together an application (though there were arguments in favor of staying with the PSF anyway). Anyway, if one still wants to apply one could consider doing it together with NumPy in order to have a larger mentor base. I can't help that much myself though (polishing the project list and so on I could always do, but not submitting the actual application). Dag Sverre From dagss at student.matnat.uio.no Thu Mar 12 08:12:06 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 12 Mar 2009 08:12:06 +0100 (CET) Subject: [Cython] Trac usage and bug handling for 0.11.x In-Reply-To: <49B8B3CD.4020308@behnel.de> References: <49B8AE18.9040109@behnel.de> <49B8B3CD.4020308@behnel.de> Message-ID: Stefan wrote: > > Carl Witty wrote: >> On Wed, Mar 11, 2009 at 11:39 PM, Stefan Behnel >> wrote: >>> 2) Every trac ticket must have a bug test case associated with it. This >>> means that there must be a failing test case in tests/bugs/ named >>> "nicely_descriptive_name_here_Txyz.pyx" (where 'xyz' is the ticket >>> number), >>> or a patch in trac that adds this test case. See >>> >>> http://wiki.cython.org/HackerGuide >>> >>> Please try to do this even for the tricky cases that feel like there >>> isn't >>> a good test case. Reproducing a bug is critical for fixing it, and >>> having a >>> test case is critical for not breaking it in the future. >> >> Does that mean I can't report bugs to trac anymore, without >> constructing a test case in the form of a Cython patch? If so, then >> -0. (I probably just wouldn't bother reporting bugs any more under >> such a rule.) > > No, reporting bugs is open to everyone. The above doesn't mean the > reporter > of a bug *must* be the one to provide the test case. I just wanted to > stress the fact that we need a test case that must go in before the fix, > and that a test case makes it easier for us to reproduce and fix the bug. > So there is a reason to prioritise tickets that contain a test case, as > tickets without a test case are more work. > > I'll clarify that on a Wiki page later today. Could we also put in something about which repositories to use for the various releases? I.e. some questions: - Will we use cython-devel or cython for 0.11.1? - If I do something that's just cleanup and not bugfix (e.g. convert NameNode to new temps), should that go into 0.12 or 0.11.1? I'm in favor of 0.11.x going in cython and NOT contain cleanup, only bugfixes, but I'm really OK with either. Dag Sverre From robertwb at math.washington.edu Thu Mar 12 08:12:49 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 12 Mar 2009 00:12:49 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49B8A9AF.6060806@behnel.de> References: <49B8A9AF.6060806@behnel.de> Message-ID: <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> On Mar 11, 2009, at 11:20 PM, Stefan Behnel wrote: > Robert Bradshaw wrote: >> I think this is neither the first nor last issue that will come up >> like this. What I think we should do is have a "-pedantic" flag which >> obeys Python semantics exactly, at the expense of speed. This applies >> to %, //, **, etc. (Overflowing is a separate issue, and I think >> there should be a flag for that too.) Compilation of .py files would >> be, by default, with -pedantic enabled, and compilation of .pyx files >> with it disabled. > > Wouldn't it be better to have such a flag emit warnings (preferably > with a > hint how to write better code) instead of changing the semantics of > operators? It won't be very useful to emit a warning on, for example, every % operator. Also, I really want to be able to write code values speed over perfect Python semantics, and at the same time take a .py file and not have to worry about subtle corner-case changes. The problem is that both modes are very defendable as the default. > I think having operators change their semantics without a clear > change in > the user code (such as a __future__ import) is a rather dangerous > 'feature'. I'm convinced pragmas are a better thing to use than a compiler flag. One could also imagine adding more granularity with pragmas as well (e.g. only use c semantics for % and //, but make sure everything else is the same). I'm not sure if this is a good thing... - Robert From robertwb at math.washington.edu Thu Mar 12 08:19:04 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 12 Mar 2009 00:19:04 -0700 Subject: [Cython] Google summer of code In-Reply-To: <6528b00c5b8779160ffe2dc3ec1f2cfb.squirrel@webmail.uio.no> References: <49B2E58A.9090900@behnel.de> <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> <993FE1B3-30D3-45C7-9E40-B8C420E2492C@math.washington.edu> <253B6623-3D04-47F7-B870-A74E420C7D6E@math.washington.edu> <6528b00c5b8779160ffe2dc3ec1f2cfb.squirrel@webmail.uio.no> Message-ID: On Mar 12, 2009, at 12:07 AM, Dag Sverre Seljebotn wrote: >> Robert Bradshaw wrote: >> On Mar 7, 2009, at 3:11 PM, Robert Bradshaw wrote: >> >>> I'm a "probably," but it's sounding like at this point we don't have >>> enough potential mentors to be a full mentoring organization unless >>> someone else stands up. On the numerical side of things, perhaps a >>> NumPy person would be willing. >>> >>> On the other hand, I think we're higher on Python's radar than last >>> year, so I'm hopeful of at least getting one (maybe two) spots for >>> them. The quality of the students and projects will play a huge role >>> here--one thing I learned at the summit last year is that it's not >>> even worth you time to go after sub-par projects (though good >>> students are an extremely good return on the investment of mentoring >>> them). >> >> Just a reminder, the organization app is due Friday if we decide to >> do it. >> >> Another advantage of being a mentor organization is that we get $500/ >> student. I'm not exactly sure what Cython would do with the money at >> this point (we don't have infrastructure to support, it's not enough >> to fund a get-together) I would propose it would be divvied out to >> the mentors to help offset the time they put into it. (They Python >> Foundation keeps the $500/student, which I'm fine with as they're big >> enough to have actual costs.) But being under an umbrella >> organization worked out well last time and is certainly less work and >> more flexibility on our part. > > This was up on the NumPy list -- they basically decided to go with > the PSF > for lack of manpower to put together an application (though there were > arguments in favor of staying with the PSF anyway). Anyway, if one > still > wants to apply one could consider doing it together with NumPy in > order to > have a larger mentor base. I can't help that much myself though > (polishing > the project list and so on I could always do, but not submitting the > actual application). Going together on it with NumPy is an interesting idea, though it doesn't elevate all the cons. Submitting the actual application doesn't seem too laborious (though it is a bit of work) but the main concern for me is that we have to commit to a certain level of involvement. One really nice thing about being with Python is that we can wait and see what kind of students and proposals we get, and then decide what to do about it. The downside is that we'll probably only get 1, or at most 2, slots (which isn't much of a downside given the response so far). It's a lot easier to get motivated once we have actual students and projects in place, assuming we'll get at least some of the caliber we want. - Robert From robertwb at math.washington.edu Thu Mar 12 08:25:09 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 12 Mar 2009 00:25:09 -0700 Subject: [Cython] Trac usage and bug handling for 0.11.x In-Reply-To: References: <49B8AE18.9040109@behnel.de> <49B8B3CD.4020308@behnel.de> Message-ID: <1140A710-E200-4633-BA6F-53B435580F13@math.washington.edu> On Mar 12, 2009, at 12:12 AM, Dag Sverre Seljebotn wrote: > Could we also put in something about which repositories to use for the > various releases? I.e. some questions: > > - Will we use cython-devel or cython for 0.11.1? > - If I do something that's just cleanup and not bugfix (e.g. convert > NameNode to new temps), should that go into 0.12 or 0.11.1? > > I'm in favor of 0.11.x going in cython and NOT contain cleanup, only > bugfixes, but I'm really OK with either. I'm actually not a huge fan of bugfix-only releases being parallel with main releases--I'd rather get the new features out as soon as possible too. I am willing to try that approach for this next release cycle if people things it's a good idea though. (That being said, I do like to be relatively "safe" with minor releases.) 0.11.1 should be the low-hanging fruit, e.g. some of the stuff that's already been implemented but didn't make it in in time. - Robert From carl.witty at gmail.com Thu Mar 12 08:26:30 2009 From: carl.witty at gmail.com (Carl Witty) Date: Thu, 12 Mar 2009 00:26:30 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> Message-ID: On Thu, Mar 12, 2009 at 12:12 AM, Robert Bradshaw wrote: > On Mar 11, 2009, at 11:20 PM, Stefan Behnel wrote: >> Wouldn't it be better to have such a flag emit warnings (preferably >> with a >> hint how to write better code) instead of changing the semantics of >> operators? > > It won't be very useful to emit a warning on, for example, every % > operator. Also, I really want to be able to write code values speed > over perfect Python semantics, and at the same time take a .py file > and not have to worry about subtle corner-case changes. The problem > is that both modes are very defendable as the default. How about a flag that generates runtime warnings on a%b and a//b whenever a and b have different sign? Carl From robertwb at math.washington.edu Thu Mar 12 08:49:45 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 12 Mar 2009 00:49:45 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> Message-ID: <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> On Mar 12, 2009, at 12:26 AM, Carl Witty wrote: > On Thu, Mar 12, 2009 at 12:12 AM, Robert Bradshaw > wrote: >> On Mar 11, 2009, at 11:20 PM, Stefan Behnel wrote: >>> Wouldn't it be better to have such a flag emit warnings (preferably >>> with a >>> hint how to write better code) instead of changing the semantics of >>> operators? >> >> It won't be very useful to emit a warning on, for example, every % >> operator. Also, I really want to be able to write code values speed >> over perfect Python semantics, and at the same time take a .py file >> and not have to worry about subtle corner-case changes. The problem >> is that both modes are very defendable as the default. > > How about a flag that generates runtime warnings on a%b and a//b > whenever a and b have different sign? This could be very useful for debugging things, but it implies there's a single, correct way that the % and // operators behave. The problem is that sometimes I want to run code with Python semantics (e.g. I'm quickly cythonizing a file) and sometimes I want to run code with C semantics (e.g. I'm doing linear algebra mod p, and don't want the overhead of fixing the sign). And perhaps I'm to demanding, but I want to be able to use % in both places rather than know some obscure function call. - Robert From dagss at student.matnat.uio.no Thu Mar 12 09:46:24 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 12 Mar 2009 09:46:24 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> Message-ID: <49B8CBE0.5050305@student.matnat.uio.no> Robert Bradshaw wrote: > On Mar 12, 2009, at 12:26 AM, Carl Witty wrote: > > >> On Thu, Mar 12, 2009 at 12:12 AM, Robert Bradshaw >> wrote: >> >>> On Mar 11, 2009, at 11:20 PM, Stefan Behnel wrote: >>> >>>> Wouldn't it be better to have such a flag emit warnings (preferably >>>> with a >>>> hint how to write better code) instead of changing the semantics of >>>> operators? >>>> >>> It won't be very useful to emit a warning on, for example, every % >>> operator. Also, I really want to be able to write code values speed >>> over perfect Python semantics, and at the same time take a .py file >>> and not have to worry about subtle corner-case changes. The problem >>> is that both modes are very defendable as the default. >>> >> How about a flag that generates runtime warnings on a%b and a//b >> whenever a and b have different sign? >> > > This could be very useful for debugging things, but it implies > there's a single, correct way that the % and // operators behave. > > The problem is that sometimes I want to run code with Python > semantics (e.g. I'm quickly cythonizing a file) and sometimes I want > to run code with C semantics (e.g. I'm doing linear algebra mod p, > and don't want the overhead of fixing the sign). And perhaps I'm to > demanding, but I want to be able to use % in both places rather than > know some obscure function call. > Perhaps you could ask the Sage list to get some more input of what the typical expectations to Cython are? (Also if you work in Z_p, could you not use unsigned ints? Then you don't get the overhead? Though I'm avoiding unsigned types like the plague myself after I figured out that range(-n, n) is empty if n is unsigned :-)) Dag Sverre From wstein at gmail.com Thu Mar 12 10:15:37 2009 From: wstein at gmail.com (William Stein) Date: Thu, 12 Mar 2009 02:15:37 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49B8CBE0.5050305@student.matnat.uio.no> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> Message-ID: <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> On Thu, Mar 12, 2009 at 1:46 AM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On Mar 12, 2009, at 12:26 AM, Carl Witty wrote: >> >> >>> On Thu, Mar 12, 2009 at 12:12 AM, Robert Bradshaw >>> wrote: >>> >>>> On Mar 11, 2009, at 11:20 PM, Stefan Behnel wrote: >>>> >>>>> Wouldn't it be better to have such a flag emit warnings (preferably >>>>> with a >>>>> hint how to write better code) instead of changing the semantics of >>>>> operators? >>>>> >>>> It won't be very useful to emit a warning on, for example, every % >>>> operator. Also, I really want to be able to write code values speed >>>> over perfect Python semantics, and at the same time take a .py file >>>> and not have to worry about subtle corner-case changes. The problem >>>> is that both modes are very defendable as the default. >>>> >>> How about a flag that generates runtime warnings on a%b and a//b >>> whenever a and b have different sign? >>> >> >> This could be very useful for debugging things, but it implies >> there's a single, correct way that the % and // operators behave. >> >> The problem is that sometimes I want to run code with Python >> semantics (e.g. I'm quickly cythonizing a file) and sometimes I want >> to run code with C semantics (e.g. I'm doing linear algebra mod p, >> and don't want the overhead of fixing the sign). And perhaps I'm to >> demanding, but I want to be able to use % in both places rather than >> know some obscure function call. >> > Perhaps you could ask the Sage list to get some more input of what the > typical expectations to Cython are? > > (Also if you work in Z_p, could you not use unsigned ints? Then you > don't get the overhead? Though I'm avoiding unsigned types like the > plague myself after I figured out that range(-n, n) is empty if n is > unsigned :-)) If you compute the sum of x and y mod p by doing "(x+y)%p", then the time is dominated by the % operation, which can easily be nearly 10 times longer than +. Now imagine doing linear algebra, where you take dot products, etc., and do lots of mod p arithmetic. If you dot two vectors with n entries in the naive way you end up doing n %p's, which is very expensive. Now imagine that you represent the entries of your vectors as C ints between -p/2 and p/2. Then you can very often do much of the dot product and only have to do very few mods, since frequently when you add up a bunch of numbers between -(p/2)^2 and (p/2)^2, they are likely to not overflow. The sum will be close to 0 because of cancellation. > Perhaps you could ask the Sage list to get some more input of what the > typical expectations to Cython are? I often think of Cython primarily as a "Python compiler", and from that perspective (-1 % 16) being different in Cython and Python is worrisome, since it will surely lead to subtle bugs. For me, Cython is all about writing fast code that is easy to use from Python. Cython is not about writing C code; if I want to write C code I write it in C. Cython is not the C language after all. Also, I think many current and potential users of Cython (at least through Sage) don't even know C, or if they do, they are using Cython so they don't have to use C. So I guess I prefer Python semantics for %, and an example in the documentation that explains how to get C semantics for % (i.e., via some macro or something). For those who view Cython as a way to write C code with a Python-like syntax, then C semantics would be preferable for %. I just don't see Cython that way, personally. -- William From stefan_ml at behnel.de Thu Mar 12 07:58:28 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 12 Mar 2009 07:58:28 +0100 Subject: [Cython] Trac usage and bug handling for 0.11.x In-Reply-To: <3BE155F1-FE7E-41AA-99D8-E53A7188E70F@math.washington.edu> References: <49B8AE18.9040109@behnel.de> <3BE155F1-FE7E-41AA-99D8-E53A7188E70F@math.washington.edu> Message-ID: <49B8B294.7020909@behnel.de> Robert Bradshaw wrote: > In fact, I think even non-bugfixes should all be assigned a > ticket on trac. (Some features may, of course, span many patches.) Sure, fine with me. > The only thing I would say is that I'd rather not clutter tests/run/-- > perhaps we could put them somewhere else (or alternatively have > runtests test all but a static list of known open tickets). If the trac based bug tickets all follow the ...Txyz.pyx convention (which I really like as it points directly to the background of the problem), it's easy to move them into a new "trac" or "fixed" directory (or whatever) when we feel like it. I'd just like to reserve tests/bugs/ for things that do not currently work (and don't care so much about the rest for now). Stefan From stefan_ml at behnel.de Thu Mar 12 10:14:31 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 12 Mar 2009 10:14:31 +0100 (CET) Subject: [Cython] Trac usage and bug handling for 0.11.x Message-ID: <61799.213.61.181.86.1236849271.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Robert Bradshaw wrote: > On Mar 12, 2009, at 12:12 AM, Dag Sverre Seljebotn wrote: >> Could we also put in something about which repositories to use for the >> various releases? I.e. some questions: >> >> - Will we use cython-devel or cython for 0.11.1? >> - If I do something that's just cleanup and not bugfix (e.g. convert >> NameNode to new temps), should that go into 0.12 or 0.11.1? Depends on how much we break with it. ;) >> I'm in favor of 0.11.x going in cython and NOT contain cleanup, only >> bugfixes, but I'm really OK with either. > > I'm actually not a huge fan of bugfix-only releases being parallel > with main releases ... also because it already happened that we forgot to push a fix into "cython". That said, it's sometimes easier to put trivial fixes into cython and pulling them over. > --I'd rather get the new features out as soon as > possible too. I am willing to try that approach for this next release > cycle if people things it's a good idea though. (That being said, I > do like to be relatively "safe" with minor releases.) > > 0.11.1 should be the low-hanging fruit, e.g. some of the stuff that's > already been implemented but didn't make it in in time. +1 and I think we should actively use trac's milestones to decide which fixes and features go where. However, the above doesn't answer the question if we should distinguish between potentially dangerous changes (that we implement because we can or because we need them, and have to put somewhere) and those that are safe enough to go into the next release. I would like to have something like a stable release series of Cython that other projects could depend on. I was having a pretty hard time in the past telling people that they must build lxml with /this/ Cython version instead of /that/ one. Being able to hold certain changes back will definitely help us in getting releases out a lot faster than 0.11. What about a separate cython-0.12 branch only for non-0.11 changes, next to a 'somewhat' safe cython-devel? When a major change is required, we can push it there instead of cython-devel, start working on it and have others test it while we can still release 0.11.x from cython-devel. It doesn't need to be up-to-date with cython-devel all the time (just when we push changes), and we can always import certain changes into cython-devel if we think we can't wait for 0.12. Or maybe "cython-unstable" would be a more generic name. Also, it would be good to pile up a set of test projects (sage and lxml as we do now, but also some others) that we regularly test before any release. Sort of a continuous integration test setup, like Python's buildbots. Stefan From dagss at student.matnat.uio.no Thu Mar 12 10:43:28 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 12 Mar 2009 10:43:28 +0100 Subject: [Cython] Trac usage and bug handling for 0.11.x In-Reply-To: <61799.213.61.181.86.1236849271.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <61799.213.61.181.86.1236849271.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49B8D940.3090202@student.matnat.uio.no> Stefan Behnel wrote: > However, the above doesn't answer the question if we should distinguish > between potentially dangerous changes (that we implement because we can or > because we need them, and have to put somewhere) and those that are safe > enough to go into the next release. I would like to have something like a > stable release series of Cython that other projects could depend on. I was > having a pretty hard time in the past telling people that they must build > lxml with /this/ Cython version instead of /that/ one. > That's my concern too. > Being able to hold certain changes back will definitely help us in getting > releases out a lot faster than 0.11. What about a separate cython-0.12 > branch only for non-0.11 changes, next to a 'somewhat' safe cython-devel? > When a major change is required, we can push it there instead of > cython-devel, start working on it and have others test it while we can > still release 0.11.x from cython-devel. It doesn't need to be up-to-date > with cython-devel all the time (just when we push changes), and we can > always import certain changes into cython-devel if we think we can't wait > for 0.12. Or maybe "cython-unstable" would be a more generic name. > +1 to cython-devel and cython-unstable. Dag Sverre From stefan_ml at behnel.de Thu Mar 12 11:00:42 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 12 Mar 2009 11:00:42 +0100 (CET) Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> Message-ID: <50507.213.61.181.86.1236852042.squirrel@groupware.dvs.informatik.tu-darmstadt.de> William Stein wrote: > On Thu, Mar 12, 2009 at 1:46 AM, Dag Sverre Seljebotn > wrote: >> Robert Bradshaw wrote: >>> On Mar 12, 2009, at 12:26 AM, Carl Witty wrote: >>>> How about a flag that generates runtime warnings on a%b and a//b >>>> whenever a and b have different sign? >>> >>> This could be very useful for debugging things, but it implies >>> there's a single, correct way that the % and // operators behave. At least, there *must* be a single way that the % and // operators behave. That's the only thing related to "correctness" in this debate. > So I guess I prefer Python semantics for %, and an example in the > documentation that explains how to get C semantics for % (i.e., via > some macro or something). ... that, and by optimising away the adaptation on unsigned numbers (or at least by checking that gcc does it for us). > For those who view Cython as a way to write C code with a Python-like > syntax, then C semantics would be preferable for %. I just don't see > Cython that way, personally. +1 Stefan From robertwb at math.washington.edu Thu Mar 12 11:03:01 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 12 Mar 2009 03:03:01 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> Message-ID: <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> On Mar 12, 2009, at 2:15 AM, William Stein wrote: > On Thu, Mar 12, 2009 at 1:46 AM, Dag Sverre Seljebotn > wrote: >>> >>> This could be very useful for debugging things, but it implies >>> there's a single, correct way that the % and // operators behave. >>> >>> The problem is that sometimes I want to run code with Python >>> semantics (e.g. I'm quickly cythonizing a file) and sometimes I want >>> to run code with C semantics (e.g. I'm doing linear algebra mod p, >>> and don't want the overhead of fixing the sign). And perhaps I'm to >>> demanding, but I want to be able to use % in both places rather than >>> know some obscure function call. >>> >> Perhaps you could ask the Sage list to get some more input of what >> the >> typical expectations to Cython are? >> >> (Also if you work in Z_p, could you not use unsigned ints? Then you >> don't get the overhead? Though I'm avoiding unsigned types like the >> plague myself after I figured out that range(-n, n) is empty if n is >> unsigned :-)) > > If you compute the sum of x and y mod p by doing "(x+y)%p", then the > time is dominated by the % operation, which can easily be nearly 10 > times longer than +. Now imagine doing linear algebra, where you take > dot products, etc., and do lots of mod p arithmetic. If you dot two > vectors with n entries in the naive way you end up doing n %p's, which > is very expensive. Now imagine that you represent the entries of your > vectors as C ints between -p/2 and p/2. Then you can very often do > much of the dot product and only have to do very few mods, since > frequently when you add up a bunch of numbers between -(p/2)^2 and > (p/2)^2, they are likely to not overflow. The sum will be close to 0 > because of cancellation. I was about to answer the very same thing then this email came in. Also, as you mentioned unsigned ints have subtle semantics, like the above. Another bad (easy--I've corrected it several times) mistake is when one writes (a-b)%p. If a and b are unsigned, this could be totally wrong. Of course, the main goal is to avoid using any division as much as possible. >> Perhaps you could ask the Sage list to get some more input of what >> the >> typical expectations to Cython are? > > I often think of Cython primarily as a "Python compiler", and from > that perspective (-1 % 16) being different in Cython and Python is > worrisome, since it will surely lead to subtle bugs. For me, > Cython is all about writing fast code that is easy to use from Python. > Cython is not about writing C code; if I want to write C code I write > it in C. Cython is not the C language after all. Also, I think many > current and potential users of Cython (at least through Sage) don't > even know C, or if they do, they are using Cython so they don't have > to use C. Good point. I'm going to start a thread there just to get a straw poll. It's the largest community of Cython users I know of... > So I guess I prefer Python semantics for %, and an example in the > documentation that explains how to get C semantics for % (i.e., via > some macro or something). One of the things that drives a lot of my optimizations in Cython has been the maxim that the obvious way should be the fast way, and I especially want to avoid having to resort to tricks to "fool" the compiler to get maximum speed. (Of course in this case we're talking about only a 5% or so difference, assuming no branching). I like the pragma idea because it makes it easy to switch a block/file of code at once (while forcing one to consciously think about it). > For those who view Cython as a way to write C code with a Python-like > syntax, then C semantics would be preferable for %. I just don't see > Cython that way, personally. I see it both ways, which I guess is why I want to have my cake and eat it too :). - Robert From stefan_ml at behnel.de Thu Mar 12 13:12:26 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 12 Mar 2009 13:12:26 +0100 (CET) Subject: [Cython] Google summer of code In-Reply-To: References: <49B2E58A.9090900@behnel.de> <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> <993FE1B3-30D3-45C7-9E40-B8C420E2492C@math.washington.edu> <253B6623-3D04-47F7-B870-A74E420C7D6E@math.washington.edu> <6528b00c5b8779160ffe2dc3ec1f2cfb.squirrel@webmail.uio.no> Message-ID: <43348.213.61.181.86.1236859946.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Robert Bradshaw wrote: > Submitting the actual application doesn't seem too laborious (though > it is a bit of work) but the main concern for me is that we have to > commit to a certain level of involvement. Just a quick note that my "involvement" in this will not be as an official mentor. I appreciate the GSoC, but I can't give any guarantees about the time I can invest. So I'll keep myself out of the decision about the way we participate. Stefan From poonaatsoc at gmail.com Thu Mar 12 13:30:25 2009 From: poonaatsoc at gmail.com (Anoop Saldanha) Date: Thu, 12 Mar 2009 18:00:25 +0530 Subject: [Cython] Google summer of code In-Reply-To: <43348.213.61.181.86.1236859946.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49B2E58A.9090900@behnel.de> <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> <993FE1B3-30D3-45C7-9E40-B8C420E2492C@math.washington.edu> <253B6623-3D04-47F7-B870-A74E420C7D6E@math.washington.edu> <6528b00c5b8779160ffe2dc3ec1f2cfb.squirrel@webmail.uio.no> <43348.213.61.181.86.1236859946.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: I would like to be mentor, but the issue is right now I don't know the internal technical details of the cython compiler, but I am going through the compiler code currently. As well as the tickets raised in the last couple of days have given me some insight on the inner working. I think I should be up to speed and should be amongst the tickets in sometime. In case I become a mentor, it will be more of a collaboration between me and the student rather than a pure mentor-student relationship and I will have to work more closely with the student(have no issue with it). On Thu, Mar 12, 2009 at 5:42 PM, Stefan Behnel wrote: > Robert Bradshaw wrote: > > Submitting the actual application doesn't seem too laborious (though > > it is a bit of work) but the main concern for me is that we have to > > commit to a certain level of involvement. > > Just a quick note that my "involvement" in this will not be as an official > mentor. I appreciate the GSoC, but I can't give any guarantees about the > time I can invest. So I'll keep myself out of the decision about the way > we participate. > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Regards, Anoop S. People who says, "It's not winning or losing that matters, but participation", probably lost. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090312/3e2dee40/attachment.htm From robertwb at math.washington.edu Thu Mar 12 19:06:34 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 12 Mar 2009 11:06:34 -0700 Subject: [Cython] Google summer of code In-Reply-To: References: <49B2E58A.9090900@behnel.de> <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> <993FE1B3-30D3-45C7-9E40-B8C420E2492C@math.washington.edu> <253B6623-3D04-47F7-B870-A74E420C7D6E@math.washington.edu> <6528b00c5b8779160ffe2dc3ec1f2cfb.squirrel@webmail.uio.no> <43348.213.61.181.86.1236859946.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <48D41E38-20B4-4082-8205-DAF49D9FB452@math.washington.edu> On Mar 12, 2009, at 5:30 AM, Anoop Saldanha wrote: > I would like to be mentor, but the issue is right now I don't know > the internal technical details of the cython compiler, but I am > going through the compiler code currently. As well as the tickets > raised in the last couple of days have given me some insight on the > inner working. I think I should be up to speed and should be > amongst the tickets in sometime. In case I become a mentor, it > will be more of a collaboration between me and the student rather > than a pure mentor-student relationship and I will have to work > more closely with the student(have no issue with it). I think that's just fine, and I bet it would be a very productive experience which is the primary goal. If you can mentor, and there's a good project/student, please do. - Robert From robertwb at math.washington.edu Thu Mar 12 19:11:51 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 12 Mar 2009 11:11:51 -0700 Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 In-Reply-To: <49B80A9F.1000200@behnel.de> References: <44706.213.61.181.86.1236700457.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B80A9F.1000200@behnel.de> Message-ID: <4875B432-09E8-46F3-BEF0-87644F9C709D@math.washington.edu> On Mar 11, 2009, at 12:01 PM, Stefan Behnel wrote: > Hi Neal, > > Neal Becker wrote: >> Can we have an rc2 with all proposed fixes? I'll plan to retest >> then. > > There's an rc3 now: > > http://cython.org/Cython-0.11.rc3.tar.gz > > Could you give it a try? I'm assuming nothing bad came up. The last word was that it did build, even if it had gcc warnings. > If all works fine, this will become Cython 0.11 final. Sage 3.4 was just released. I'll verify that it passes all tests (was fine with 3.3 so I don't anticipate issues) and then lets release. - Robert From robertwb at math.washington.edu Thu Mar 12 19:14:22 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 12 Mar 2009 11:14:22 -0700 Subject: [Cython] Trac usage and bug handling for 0.11.x In-Reply-To: <49B8D940.3090202@student.matnat.uio.no> References: <61799.213.61.181.86.1236849271.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B8D940.3090202@student.matnat.uio.no> Message-ID: <4DB56201-278E-4068-A185-4AB1947EC239@math.washington.edu> On Mar 12, 2009, at 2:43 AM, Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> However, the above doesn't answer the question if we should >> distinguish >> between potentially dangerous changes (that we implement because >> we can or >> because we need them, and have to put somewhere) and those that >> are safe >> enough to go into the next release. I would like to have something >> like a >> stable release series of Cython that other projects could depend >> on. I was >> having a pretty hard time in the past telling people that they >> must build >> lxml with /this/ Cython version instead of /that/ one. >> > That's my concern too. We don't have this issue with Sage as we ship Cython as a component. (We even ship and build our own Python...) However, modulo any bug fixes, I think the 0.x range should all be compatible. >> Being able to hold certain changes back will definitely help us in >> getting >> releases out a lot faster than 0.11. What about a separate >> cython-0.12 >> branch only for non-0.11 changes, next to a 'somewhat' safe cython- >> devel? >> When a major change is required, we can push it there instead of >> cython-devel, start working on it and have others test it while we >> can >> still release 0.11.x from cython-devel. It doesn't need to be up- >> to-date >> with cython-devel all the time (just when we push changes), and we >> can >> always import certain changes into cython-devel if we think we >> can't wait >> for 0.12. Or maybe "cython-unstable" would be a more generic name. >> > +1 to cython-devel and cython-unstable. Yes, I'll make such branches. I like the idea of "safe" stuff (either fixes or minor improvements) being in cython-devel, and anything more drastic (e.g. moving all the temp stuff over to the new model) in cython-unstable. A much better division than "bugfix" and "other." - Robert From stefan_ml at behnel.de Thu Mar 12 19:22:03 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 12 Mar 2009 19:22:03 +0100 Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 In-Reply-To: <4875B432-09E8-46F3-BEF0-87644F9C709D@math.washington.edu> References: <44706.213.61.181.86.1236700457.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B80A9F.1000200@behnel.de> <4875B432-09E8-46F3-BEF0-87644F9C709D@math.washington.edu> Message-ID: <49B952CB.2080802@behnel.de> Robert Bradshaw wrote: > Sage 3.4 was just released. I'll verify that it passes all tests (was > fine with 3.3 so I don't anticipate issues) and then lets release. Yes, let's. (finally!) Stefan From robertwb at math.washington.edu Fri Mar 13 00:06:06 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 12 Mar 2009 16:06:06 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> Message-ID: <5A295F2F-EA79-40B6-92F6-139A5D0FA808@math.washington.edu> What about ** for doubles? flotobject.c:float_pow has a lot of comments about that. - Robert From wstein at gmail.com Fri Mar 13 00:18:31 2009 From: wstein at gmail.com (William Stein) Date: Thu, 12 Mar 2009 16:18:31 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> Message-ID: <85e81ba30903121618q2506d10fxb9ab6b5899032a4f@mail.gmail.com> On Thu, Mar 12, 2009 at 3:03 AM, Robert Bradshaw wrote: > On Mar 12, 2009, at 2:15 AM, William Stein wrote: > >> On Thu, Mar 12, 2009 at 1:46 AM, Dag Sverre Seljebotn >> wrote: >>>> >>>> This could be very useful for debugging things, but it implies >>>> there's a single, correct way that the % and // operators behave. >>>> >>>> The problem is that sometimes I want to run code with Python >>>> semantics (e.g. I'm quickly cythonizing a file) and sometimes I want >>>> to run code with C semantics (e.g. I'm doing linear algebra mod p, >>>> and don't want the overhead of fixing the sign). And perhaps I'm to >>>> demanding, but I want to be able to use % in both places rather than >>>> know some obscure function call. >>>> >>> Perhaps you could ask the Sage list to get some more input of what >>> the >>> typical expectations to Cython are? >>> >>> (Also if you work in Z_p, could you not use unsigned ints? Then you >>> don't get the overhead? Though I'm avoiding unsigned types like the >>> plague myself after I figured out that range(-n, n) is empty if n is >>> unsigned :-)) >> >> If you compute the sum of x and y mod p by doing "(x+y)%p", then the >> time is dominated by the % operation, which can easily be nearly 10 >> times longer than +. ?Now imagine doing linear algebra, where you take >> dot products, etc., and do lots of mod p arithmetic. ?If you dot two >> vectors with n entries in the naive way you end up doing n %p's, which >> is very expensive. ?Now imagine that you represent the entries of your >> vectors as C ints between -p/2 and p/2. ?Then you can very often do >> much of the dot product and only have to do very few mods, since >> frequently when you add up a bunch of numbers between -(p/2)^2 and >> (p/2)^2, they are likely to not overflow. ?The sum will be close to 0 >> because of cancellation. > > I was about to answer the very same thing then this email came in. That's probably because you explained it to me once :-) > One of the things that drives a lot of my optimizations in Cython has > been the maxim that the obvious way should be the fast way, and I > especially want to avoid having to resort to tricks to "fool" the > compiler to get maximum speed. (Of course in this case we're talking > about only a 5% or so difference, assuming no branching). I like the Are you sure it's a difference of 5%? I just tried these two programs: main() { int n = -1, m = 16, k, a=0; for(k=1;k<100000000;k++) { a = n%k + (a<0)?k:0; } } and main() { int n = -1, m = 16, k, a=0; for(k=1;k<100000000;k++) { a = n%k; } } and the first takes 22% longer than the second. -- William From santiago.aguiar at gmail.com Fri Mar 13 01:05:18 2009 From: santiago.aguiar at gmail.com (Santiago Aguiar) Date: Thu, 12 Mar 2009 21:05:18 -0300 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... Message-ID: <49B9A33E.2080405@gmail.com> Hello, I've started to use Cython just yesterday, and I'm facing some difficulties ;). I'm sorry to ask this on the dev list, but I didn't found a user specific one... I have a struct-dense C source code that defines the PDU of a network protocol and functions to read/write from a buffer struct. First, I don't know if I'm doing this the right way, but I had to define by hand all my structs again on Cython to use them. I think it's the way to do it, but I did feel like repeating myself over and over ;). Is there any tool to create mappings like these automatically? (and handle common cases like wrapping structs with inner structs, uint8_t *, defining macros like #define VAL 1 as dummy enums, etc.). I saw some for C++ (using gcc-xml) but I'm stuck with pure C. My main objective on doing these bindings are to reuse the C code, having an efficient implementation is secondary. Second, at this extremely un-proficient time, for each of the structs on my pdu.pyx I have something like this: cdef class Msg1(Message): cdef public ... (one declaration for each field present in the struct) def __init__(self, ... (one keyword parameter for each field provided by the struct): self.fieldN = ... (this repeated for every field) cdef void from_c(self, msg1_c_struct *msg1): self.fieldN = msg1.fieldN (repeated for every field, with some additional mappings in case the field is a struct) And then I have a message parser that uses the C functions to read the PDU and wrap it on Msg1: class Parser: def read_pdu(self, str_data): cdef msg_struct msg .... pdu_read(data, &msg) .... if msg.header.type == MSG_1_TYPE: return Msg1().from_c(&msg.data.msg_1) Where msg.data.msg_1 is of type msg1_c_struct. On that code, I get an error message saying that: 'Cannot convert 'msg1_c_struct *' to Python object' where I do the Msg1().from_c. I don't understand why I'm getting the error. If I call a cdef function instead of a cdef method, it works OK; but shouldn't I be able to call a cdef method and passing a C type instead of a Python type? Otherwise, I will need to add an external cdef function for each class to create the object from the struct... I must be doing something wrong :). Thanks, and sorry for the basic questions! Best regards, Santiago. From robertwb at math.washington.edu Fri Mar 13 02:33:00 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 12 Mar 2009 18:33:00 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <85e81ba30903121618q2506d10fxb9ab6b5899032a4f@mail.gmail.com> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <85e81ba30903121618q2506d10fxb9ab6b5899032a4f@mail.gmail.com> Message-ID: On Mar 12, 2009, at 4:18 PM, William Stein wrote: > >> One of the things that drives a lot of my optimizations in Cython has >> been the maxim that the obvious way should be the fast way, and I >> especially want to avoid having to resort to tricks to "fool" the >> compiler to get maximum speed. (Of course in this case we're talking >> about only a 5% or so difference, assuming no branching). I like the > > Are you sure it's a difference of 5%? > > I just tried these two programs: > > main() { > int n = -1, m = 16, k, a=0; > for(k=1;k<100000000;k++) { > a = n%k + (a<0)?k:0; > } > } > > and > > main() { > int n = -1, m = 16, k, a=0; > for(k=1;k<100000000;k++) { > a = n%k; > } > } > > and the first takes 22% longer than the second. Thanks for the timings--5% was a conservative estimate, but I didn't realize it was that conservative. - Robert From stefan_ml at behnel.de Fri Mar 13 08:53:07 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 13 Mar 2009 08:53:07 +0100 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: <49B9A33E.2080405@gmail.com> References: <49B9A33E.2080405@gmail.com> Message-ID: <49BA10E3.6090409@behnel.de> Santiago Aguiar wrote: > I've started to use Cython just yesterday, and I'm facing some > difficulties ;). I'm sorry to ask this on the dev list, but I didn't > found a user specific one... > > I have a struct-dense C source code that defines the PDU of a network > protocol and functions to read/write from a buffer struct. > > First, I don't know if I'm doing this the right way, but I had to define > by hand all my structs again on Cython to use them. I think it's the way > to do it, but I did feel like repeating myself over and over ;). That's the way to do it, though. You only go through that once for each struct you use (and your code that uses the struct is likely longer than the struct itself), so it's not too much work. Note that you do not need to declare the complete struct. Just write down the fields you need. Cython doesn't need to know the others and the C compiler sees the struct anyway. That also allows you to ignore any externally referenced complex types that you do not use yourself. > Second, at this extremely un-proficient time, for each of the structs on > my pdu.pyx I have something like this: > > cdef class Msg1(Message): > cdef public ... (one declaration for each field present in the struct) Why don't you reference the struct itself here? Do you really need these fields to be public? Some methods for setting up common use cases are often a lot better than just having users set everything by hand, especially when there are non-trivial type conversions involved, or when there are interdependencies between fields that are better checked for validity. For example, Cython cannot check arbitrary integer values for you that the user writes into an enum type attribute. > def __init__(self, ... (one keyword parameter for each field > provided by the struct): > self.fieldN = ... (this repeated for every field) > cdef void from_c(self, msg1_c_struct *msg1): > self.fieldN = msg1.fieldN (repeated for every field, with some > additional mappings in case the field is a struct) There is some special support for struct assignments from dicts, you can try that. I have no idea where that's documented, though. Robert? > And then I have a message parser that uses the C functions to read the > PDU and wrap it on Msg1: > > class Parser: > def read_pdu(self, str_data): > cdef msg_struct msg > .... > pdu_read(data, &msg) > .... > if msg.header.type == MSG_1_TYPE: > return Msg1().from_c(&msg.data.msg_1) > > Where msg.data.msg_1 is of type msg1_c_struct. > > On that code, I get an error message saying that: 'Cannot convert > 'msg1_c_struct *' to Python object' where I do the Msg1().from_c. You have to assign the result of Msg1() to a typed variable. Otherwise, Cython doesn't know that the thing it returns is really of type Msg1 (and not a generic Python object) and that it does have a C typed from_c() method. > I don't understand why I'm getting the error. If I call a cdef function > instead of a cdef method, it works OK; but shouldn't I be able to call a > cdef method and passing a C type instead of a Python type? Otherwise, I > will need to add an external cdef function for each class to create the > object from the struct... I must be doing something wrong :). I consider this a deficiency of Cython. It should know the type of the result when you call an extension type. http://trac.cython.org/cython_trac/ticket/232 Stefan From greg.ewing at canterbury.ac.nz Fri Mar 13 11:19:05 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 13 Mar 2009 23:19:05 +1300 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: <49BA10E3.6090409@behnel.de> References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> Message-ID: <49BA3319.1020600@canterbury.ac.nz> Stefan Behnel wrote: > Santiago Aguiar wrote: > >> return Msg1().from_c(&msg.data.msg_1) >> >> I get an error message saying that: 'Cannot convert >>'msg1_c_struct *' to Python object' where I do the Msg1().from_c. > > I consider this a deficiency of Cython. It should know the type of the > result when you call an extension type. This does currently work in Pyrex, btw. -- Greg From ndbecker2 at gmail.com Fri Mar 13 12:50:24 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 13 Mar 2009 07:50:24 -0400 Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 References: <44706.213.61.181.86.1236700457.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B80A9F.1000200@behnel.de> <4875B432-09E8-46F3-BEF0-87644F9C709D@math.washington.edu> <49B952CB.2080802@behnel.de> Message-ID: Stefan Behnel wrote: > > Robert Bradshaw wrote: >> Sage 3.4 was just released. I'll verify that it passes all tests (was >> fine with 3.3 so I don't anticipate issues) and then lets release. > > Yes, let's. (finally!) > > Stefan 0.11rc3 built OK (with warnings). From stefan_ml at behnel.de Fri Mar 13 13:17:32 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 13 Mar 2009 13:17:32 +0100 (CET) Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 In-Reply-To: References: <44706.213.61.181.86.1236700457.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B80A9F.1000200@behnel.de> <4875B432-09E8-46F3-BEF0-87644F9C709D@math.washington.edu> <49B952CB.2080802@behnel.de> Message-ID: <45933.213.61.181.86.1236946652.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Neal Becker wrote: > 0.11rc3 built OK (with warnings). Such as? Stefan From ndbecker2 at gmail.com Fri Mar 13 13:34:03 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 13 Mar 2009 08:34:03 -0400 Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 References: <44706.213.61.181.86.1236700457.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B80A9F.1000200@behnel.de> <4875B432-09E8-46F3-BEF0-87644F9C709D@math.washington.edu> <49B952CB.2080802@behnel.de> <45933.213.61.181.86.1236946652.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: Stefan Behnel wrote: > Neal Becker wrote: >> 0.11rc3 built OK (with warnings). > > Such as? > > Stefan The build log (for x86_64) can be seen here: http://koji.fedoraproject.org/koji/getfile?taskID=1239381&name=build.log From poonaatsoc at gmail.com Fri Mar 13 14:43:29 2009 From: poonaatsoc at gmail.com (Anoop Saldanha) Date: Fri, 13 Mar 2009 19:13:29 +0530 Subject: [Cython] Google summer of code In-Reply-To: <48D41E38-20B4-4082-8205-DAF49D9FB452@math.washington.edu> References: <49B2E58A.9090900@behnel.de> <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> <993FE1B3-30D3-45C7-9E40-B8C420E2492C@math.washington.edu> <253B6623-3D04-47F7-B870-A74E420C7D6E@math.washington.edu> <6528b00c5b8779160ffe2dc3ec1f2cfb.squirrel@webmail.uio.no> <43348.213.61.181.86.1236859946.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <48D41E38-20B4-4082-8205-DAF49D9FB452@math.washington.edu> Message-ID: Nice. Thanks. Do let me know, if you want me as a mentor. I saw the enhancements section for cython, and I will pick one of those and start digging into it. On Thu, Mar 12, 2009 at 11:36 PM, Robert Bradshaw < robertwb at math.washington.edu> wrote: > On Mar 12, 2009, at 5:30 AM, Anoop Saldanha wrote: > > > I would like to be mentor, but the issue is right now I don't know > > the internal technical details of the cython compiler, but I am > > going through the compiler code currently. As well as the tickets > > raised in the last couple of days have given me some insight on the > > inner working. I think I should be up to speed and should be > > amongst the tickets in sometime. In case I become a mentor, it > > will be more of a collaboration between me and the student rather > > than a pure mentor-student relationship and I will have to work > > more closely with the student(have no issue with it). > > I think that's just fine, and I bet it would be a very productive > experience which is the primary goal. If you can mentor, and there's > a good project/student, please do. > > - Robert > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Regards, Anoop S. People who says, "It's not winning or losing that matters, but participation", probably lost. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090313/c92fb0d2/attachment.htm From stefan_ml at behnel.de Fri Mar 13 14:43:48 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 13 Mar 2009 14:43:48 +0100 (CET) Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 In-Reply-To: References: <44706.213.61.181.86.1236700457.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B80A9F.1000200@behnel.de> <4875B432-09E8-46F3-BEF0-87644F9C709D@math.washington.edu> <49B952CB.2080802@behnel.de> <45933.213.61.181.86.1236946652.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <41449.213.61.181.86.1236951828.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Neal Becker wrote: > Stefan Behnel wrote: > >> Neal Becker wrote: >>> 0.11rc3 built OK (with warnings). >> >> Such as? >> >> Stefan > > The build log (for x86_64) can be seen here: > http://koji.fedoraproject.org/koji/getfile?taskID=1239381&name=build.log Looks perfectly fine to me. The rest are all (minor) known problems. Thanks! Stefan From carl.witty at gmail.com Fri Mar 13 15:42:29 2009 From: carl.witty at gmail.com (Carl Witty) Date: Fri, 13 Mar 2009 07:42:29 -0700 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: <49BA10E3.6090409@behnel.de> References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> Message-ID: On Fri, Mar 13, 2009 at 12:53 AM, Stefan Behnel wrote: >> I don't understand why I'm getting the error. If I call a cdef function >> instead of a cdef method, it works OK; but shouldn't I be able to call a >> cdef method and passing a C type instead of a Python type? Otherwise, I >> will need to add an external cdef function for each class to create the >> object from the struct... I must be doing something wrong :). > > I consider this a deficiency of Cython. It should know the type of the > result when you call an extension type. > > http://trac.cython.org/cython_trac/ticket/232 Hmm... I hope that eventually Cython will allow overriding __new__, at which point calling an extension type won't necessarily return a value of that type. (I see that Robert already made this point in a comment on the ticket.) Carl From stefan_ml at behnel.de Fri Mar 13 15:46:03 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 13 Mar 2009 15:46:03 +0100 (CET) Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> Message-ID: <43407.213.61.181.86.1236955563.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Carl Witty wrote: > On Fri, Mar 13, 2009 at 12:53 AM, Stefan Behnel wrote: >>> I don't understand why I'm getting the error. If I call a cdef function >>> instead of a cdef method, it works OK; but shouldn't I be able to call >>> a >>> cdef method and passing a C type instead of a Python type? Otherwise, I >>> will need to add an external cdef function for each class to create the >>> object from the struct... I must be doing something wrong :). >> >> I consider this a deficiency of Cython. It should know the type of the >> result when you call an extension type. >> >> http://trac.cython.org/cython_trac/ticket/232 > > Hmm... I hope that eventually Cython will allow overriding __new__, at > which point calling an extension type won't necessarily return a value > of that type. I'm not sure. This should certainly be supported for Python classes, but for extension types? What would be the use case? Stefan From dalcinl at gmail.com Fri Mar 13 17:19:45 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 13 Mar 2009 14:19:45 -0200 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> Message-ID: On Fri, Mar 13, 2009 at 12:42 PM, Carl Witty wrote: > On Fri, Mar 13, 2009 at 12:53 AM, Stefan Behnel wrote: >>> I don't understand why I'm getting the error. If I call a cdef function >>> instead of a cdef method, it works OK; but shouldn't I be able to call a >>> cdef method and passing a C type instead of a Python type? Otherwise, I >>> will need to add an external cdef function for each class to create the >>> object from the struct... I must be doing something wrong :). >> >> I consider this a deficiency of Cython. It should know the type of the >> result when you call an extension type. >> >> http://trac.cython.org/cython_trac/ticket/232 > > Hmm... I hope that eventually Cython will allow overriding __new__, at > which point calling an extension type won't necessarily return a value > of that type. ?(I see that Robert already made this point in a comment > on the ticket.) > But if a typecheck is done, we could dispatch a (polymorfic) C call of a Python call... Am I missing something? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Fri Mar 13 17:25:29 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 13 Mar 2009 14:25:29 -0200 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: <43407.213.61.181.86.1236955563.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> <43407.213.61.181.86.1236955563.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: On Fri, Mar 13, 2009 at 12:46 PM, Stefan Behnel wrote: > Carl Witty wrote: >> Hmm... I hope that eventually Cython will allow overriding __new__, at >> which point calling an extension type won't necessarily return a value >> of that type. > > I'm not sure. This should certainly be supported for Python classes, but > for extension types? What would be the use case? > Well, take for granted that if this were available, I would use it in mpi4py. For example, some instances of cdef classes could be specially handled, and I could easily implement things like in the code below: comm = MPI.Comm(MPI.COMM_WORLD) assert comm is MPI.COMM_WORLD and the assertion will succeed... Moreover, any cdef class whose instances are intended to be immutable (like int, str, float, tuple) could benefit from this feature, such in the same way that core CPython does. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Fri Mar 13 17:27:02 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 13 Mar 2009 17:27:02 +0100 (CET) Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> Message-ID: <47474.213.61.181.86.1236961622.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Lisandro Dalcin wrote: > On Fri, Mar 13, 2009 at 12:42 PM, Carl Witty wrote: >> On Fri, Mar 13, 2009 at 12:53 AM, Stefan Behnel >> wrote: >>>> I don't understand why I'm getting the error. If I call a cdef >>>> function >>>> instead of a cdef method, it works OK; but shouldn't I be able to call >>>> a >>>> cdef method and passing a C type instead of a Python type? Otherwise, >>>> I >>>> will need to add an external cdef function for each class to create >>>> the >>>> object from the struct... I must be doing something wrong :). >>> >>> I consider this a deficiency of Cython. It should know the type of the >>> result when you call an extension type. >>> >>> http://trac.cython.org/cython_trac/ticket/232 >> >> Hmm... I hope that eventually Cython will allow overriding __new__, at >> which point calling an extension type won't necessarily return a value >> of that type. ?(I see that Robert already made this point in a comment >> on the ticket.) > > But if a typecheck is done, we could dispatch a (polymorfic) C call of > a Python call... Am I missing something? A typecheck would only catch the case that you return the exact type, not a subtype (which works perfectly fine otherwise). And a full fledged isinstance() sounds a bit too heavy for this. Stefan From stefan_ml at behnel.de Fri Mar 13 17:39:47 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 13 Mar 2009 17:39:47 +0100 (CET) Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> <43407.213.61.181.86.1236955563.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <54183.213.61.181.86.1236962387.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Lisandro Dalcin wrote: > On Fri, Mar 13, 2009 at 12:46 PM, Stefan Behnel wrote: >> Carl Witty wrote: >>> Hmm... I hope that eventually Cython will allow overriding __new__, at >>> which point calling an extension type won't necessarily return a value >>> of that type. >> >> I'm not sure. This should certainly be supported for Python classes, but >> for extension types? What would be the use case? > > Well, take for granted that if this were available, I would use it in > mpi4py. > > For example, some instances of cdef classes could be specially > handled, and I could easily implement things like in the code below: > > comm = MPI.Comm(MPI.COMM_WORLD) > assert comm is MPI.COMM_WORLD > > and the assertion will succeed... Am I right in guessing that Comm is an extension type here and that COMM_WORLD is an instance? So you want to use Comm as both a class and a factory, right? So, what was your use case again? > Moreover, any cdef class whose instances are intended to be immutable > (like int, str, float, tuple) could benefit from this feature, such in > the same way that core CPython does. Here, I assume you are hinting towards singletons and caching, right? lxml actually does this with dedicated factories, but I can see that some APIs may benefit from such a behaviour. So, can't we just require that ExtType.__new__() returns an instance of ExtType (or its subtypes)? Or is there a use case that would benefit from returning arbitrary things? Stefan From carl.witty at gmail.com Fri Mar 13 17:56:15 2009 From: carl.witty at gmail.com (Carl Witty) Date: Fri, 13 Mar 2009 09:56:15 -0700 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: <54183.213.61.181.86.1236962387.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> <43407.213.61.181.86.1236955563.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <54183.213.61.181.86.1236962387.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: On Fri, Mar 13, 2009 at 9:39 AM, Stefan Behnel wrote: > So, can't we just require that ExtType.__new__() returns an instance of > ExtType (or its subtypes)? Or is there a use case that would benefit from > returning arbitrary things? I have code (currently in pure Python, but if I ever transition the code to Cython I'd like to do the same thing) where B and C both inherit from A, and B's .__new__() can sometimes return an instance of C. The details are a bit mathematical, but in case you're curious: B is ANExtensionElement, which represents an arbitrary number field element. C is ANRational, which represents a rational number. When I try to construct an ANExtensionElement that's actually a rational number, I return an ANRational instead. Carl From santiago.aguiar at gmail.com Fri Mar 13 18:08:10 2009 From: santiago.aguiar at gmail.com (Santiago Aguiar) Date: Fri, 13 Mar 2009 14:08:10 -0300 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: <49BA10E3.6090409@behnel.de> References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> Message-ID: <49BA92FA.2070801@gmail.com> Stefan Behnel wrote: > Why don't you reference the struct itself here? Do you really need these > fields to be public? > I could, but in that case I would need to write getters/setters for each attribute. In my code, almost all the struct fields are relevant, since they represent fields in the protocol PDU which I need to provide to python code. > There is some special support for struct assignments from dicts, you can > try that. I have no idea where that's documented, though. Robert? > Any pointers will be welcomed! Thanks! Best regards, Santiago. From dalcinl at gmail.com Fri Mar 13 18:11:33 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 13 Mar 2009 15:11:33 -0200 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: <54183.213.61.181.86.1236962387.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> <43407.213.61.181.86.1236955563.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <54183.213.61.181.86.1236962387.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: On Fri, Mar 13, 2009 at 2:39 PM, Stefan Behnel wrote: >> >> comm = MPI.Comm(MPI.COMM_WORLD) >> assert comm is MPI.COMM_WORLD >> >> and the assertion will succeed... > > Am I right in guessing that Comm is an extension type here and that > COMM_WORLD is an instance? > Yes, you are right... > So you want to use Comm as both a class and a factory, right? So, what was > your use case again? > MPI.COMM_WORLD is a predefined, immutable, live-forever instance of a class 'Intracomm', a derived class of 'Comm'. Comm should be seen as an abstract base class in C++ terms... Comm is a class, but it is also a factory in the sense that if you call Comm(comm) it returns a new Comm instance referencing the underlying C handle... in mpi4py, all cdef classes are actually proxies to C handles.... I would like to maintain if possible this invariant... if the C handle is MPI_COMM_WORLD (the C side name), then the Python-level instance is always MPI.COMM_WORLD .... > >> Moreover, any cdef class whose instances are intended to be immutable >> (like int, str, float, tuple) could benefit from this feature, such in >> the same way that core CPython does. > > Here, I assume you are hinting towards singletons and caching, right? lxml > actually does this with dedicated factories, but I can see that some APIs > may benefit from such a behaviour. > At least singletons... In all the above discussion, MPI.COMM_WORLD is similar to a singleton, I do not really know how to call it :-) > So, can't we just require that ExtType.__new__() returns an instance of > ExtType (or its subtypes)? As a first approach, it could be like that... ? Or is there a use case that would benefit from > returning arbitrary things? > Well, Python supports this, we could in case it is possible.... > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Fri Mar 13 18:17:09 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 13 Mar 2009 15:17:09 -0200 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: <49BA92FA.2070801@gmail.com> References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> <49BA92FA.2070801@gmail.com> Message-ID: Santiago, have you considered writing a simple script to generate Cython code :-) ?? You could parse a simple txt file with metadata about your C-level structs (name, field types, fieldname, and even default field values), then you could easily dump Cython code automating the member, __cinit__() methods and all the 'property' you need. As Stefan commented, you should just store your struct in your cdef class, and use 'property' to expose the field values with tiny get/set funcions... Remember that struct member assignments should work in any C89 compiler... No point to complicate your from_c()/ to_c() cdef methods ... On Fri, Mar 13, 2009 at 3:08 PM, Santiago Aguiar wrote: > Stefan Behnel wrote: >> Why don't you reference the struct itself here? Do you really need these >> fields to be public? >> > I could, but in that case I would need to write getters/setters for each > attribute. In my code, almost all the struct fields are relevant, since > they represent fields in the protocol PDU which I need to provide to > python code. >> There is some special support for struct assignments from dicts, you can >> try that. I have no idea where that's documented, though. Robert? >> > Any pointers will be welcomed! > > Thanks! > > Best regards, > > Santiago. > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From carl.witty at gmail.com Fri Mar 13 18:24:10 2009 From: carl.witty at gmail.com (Carl Witty) Date: Fri, 13 Mar 2009 10:24:10 -0700 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> <43407.213.61.181.86.1236955563.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <54183.213.61.181.86.1236962387.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: On Fri, Mar 13, 2009 at 10:11 AM, Lisandro Dalcin wrote: > On Fri, Mar 13, 2009 at 2:39 PM, Stefan Behnel wrote: >> So, can't we just require that ExtType.__new__() returns an instance of >> ExtType (or its subtypes)? > > As a first approach, it could be like that... > > ? Or is there a use case that would benefit from >> returning arbitrary things? >> > > Well, Python supports this, we could in case it is possible.... How about this? In the hypothetical future version of Cython that allows overriding __new__, say that the overrides have to be declared in the .pxd file, with cdef TYPE __new__(self, foo, bar) (where TYPE defaults to the class being declared, if omitted). So when generating code, if the class doesn't override __new__, you know that calling the class returns an instance; if the class does override __new__, then the return type is explicitly given. This could be a parent class, for my use case where I want to return an instance of a sibling class; or it could be "object", if anybody ever has a use case for having __new__ return an arbitrary object. Carl From robertwb at math.washington.edu Fri Mar 13 18:41:40 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 13 Mar 2009 10:41:40 -0700 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> <43407.213.61.181.86.1236955563.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <54183.213.61.181.86.1236962387.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: On Mar 13, 2009, at 10:24 AM, Carl Witty wrote: > On Fri, Mar 13, 2009 at 10:11 AM, Lisandro Dalcin > wrote: >> On Fri, Mar 13, 2009 at 2:39 PM, Stefan Behnel >> wrote: >>> So, can't we just require that ExtType.__new__() returns an >>> instance of >>> ExtType (or its subtypes)? >> >> As a first approach, it could be like that... >> >> ? Or is there a use case that would benefit from >>> returning arbitrary things? >>> >> >> Well, Python supports this, we could in case it is possible.... > > How about this? In the hypothetical future version of Cython that > allows overriding __new__, say that the overrides have to be declared > in the .pxd file, with > > cdef TYPE __new__(self, foo, bar) > > (where TYPE defaults to the class being declared, if omitted). So > when generating code, if the class doesn't override __new__, you know > that calling the class returns an instance; if the class does override > __new__, then the return type is explicitly given. This could be a > parent class, for my use case where I want to return an instance of a > sibling class; or it could be "object", if anybody ever has a use case > for having __new__ return an arbitrary object. +1. This will make it possible to optimize the common case while providing (future) flexibility. - Robert From robertwb at math.washington.edu Fri Mar 13 19:01:54 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 13 Mar 2009 11:01:54 -0700 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: <49BA10E3.6090409@behnel.de> References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> Message-ID: <5993C518-1C51-45F4-AA6C-7771F50729B0@math.washington.edu> On Mar 13, 2009, at 12:53 AM, Stefan Behnel wrote: > > Santiago Aguiar wrote: >> I've started to use Cython just yesterday, and I'm facing some >> difficulties ;). I'm sorry to ask this on the dev list, but I didn't >> found a user specific one... >> >> I have a struct-dense C source code that defines the PDU of a network >> protocol and functions to read/write from a buffer struct. >> >> First, I don't know if I'm doing this the right way, but I had to >> define >> by hand all my structs again on Cython to use them. I think it's >> the way >> to do it, but I did feel like repeating myself over and over ;). > > That's the way to do it, though. You only go through that once for > each > struct you use (and your code that uses the struct is likely longer > than > the struct itself), so it's not too much work. > > Note that you do not need to declare the complete struct. Just > write down > the fields you need. Cython doesn't need to know the others and the C > compiler sees the struct anyway. That also allows you to ignore any > externally referenced complex types that you do not use yourself. > > >> Second, at this extremely un-proficient time, for each of the >> structs on >> my pdu.pyx I have something like this: >> >> cdef class Msg1(Message): >> cdef public ... (one declaration for each field present in the >> struct) > > Why don't you reference the struct itself here? Do you really need > these > fields to be public? Some methods for setting up common use cases > are often > a lot better than just having users set everything by hand, > especially when > there are non-trivial type conversions involved, or when there are > interdependencies between fields that are better checked for > validity. For > example, Cython cannot check arbitrary integer values for you that > the user > writes into an enum type attribute. I would second this--you'll have to make properties for each of the fields, but your internal code will be much easier if you can just pass structs around. >> def __init__(self, ... (one keyword parameter for each field >> provided by the struct): >> self.fieldN = ... (this repeated for every field) >> cdef void from_c(self, msg1_c_struct *msg1): >> self.fieldN = msg1.fieldN (repeated for every field, with some >> additional mappings in case the field is a struct) > > There is some special support for struct assignments from dicts, > you can > try that. I have no idea where that's documented, though. Robert? It's not except for at http://hg.cython.org/cython-devel/file/ f2a2d6e68de2/tests/run/struct_conversion.pyx Simply put, if one has cdef struct Point: double x double y One can write "Point(x=1, y=2)" to get a literal struct, and conversion from a struct to a Python object creates a dict. I intended to write conversion the other direction too (from a dict to a struct in the obvious way) but haven't gotten around to it yet. Personally, I would write a simple script that auto-generates this kind of stuff for you in your specific case. - Robert From stefan_ml at behnel.de Fri Mar 13 19:07:00 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 13 Mar 2009 19:07:00 +0100 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> <43407.213.61.181.86.1236955563.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <54183.213.61.181.86.1236962387.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49BAA0C4.8060208@behnel.de> Robert Bradshaw wrote: > On Mar 13, 2009, at 10:24 AM, Carl Witty wrote: >> How about this? In the hypothetical future version of Cython that >> allows overriding __new__, say that the overrides have to be declared >> in the .pxd file, with >> >> cdef TYPE __new__(self, foo, bar) >> >> (where TYPE defaults to the class being declared, if omitted). So >> when generating code, if the class doesn't override __new__, you know >> that calling the class returns an instance; if the class does override >> __new__, then the return type is explicitly given. This could be a >> parent class, for my use case where I want to return an instance of a >> sibling class; or it could be "object", if anybody ever has a use case >> for having __new__ return an arbitrary object. > > +1. This will make it possible to optimize the common case while > providing (future) flexibility. +1 sounds like a good way to do it. Carl, can you write a comment for the ticket so that it has your name in it? I'll change it into a feature request. http://trac.cython.org/cython_trac/ticket/232 Stefan From santiago.aguiar at gmail.com Fri Mar 13 20:39:54 2009 From: santiago.aguiar at gmail.com (Santiago Aguiar) Date: Fri, 13 Mar 2009 16:39:54 -0300 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> <49BA92FA.2070801@gmail.com> Message-ID: <49BAB68A.7060303@gmail.com> Lisandro Dalcin wrote: > Santiago, have you considered writing a simple script to generate > Cython code :-) ?? You could parse a simple txt file with metadata > about your C-level structs (name, field types, fieldname, and even > default field values), then you could easily dump Cython code > automating the member, __cinit__() methods and all the 'property' you > need. > I did, and I'm usually on for codegen, but since I was on a schedule I wanted to avoid having to code the generator, and hoped that something already existed.... besides, the metadata will still have to mirror the C defs... or I would need to parse the .h which... well I want to avoid if at all possible! However, it still could save me some time. > As Stefan commented, you should just store your struct in your cdef > class, and use 'property' to expose the field values with tiny get/set > funcions... Remember that struct member assignments should work in any > C89 compiler... No point to complicate your from_c()/ to_c() cdef > methods ... > OK, I'm not terribly convinced that I will save some time by writing the get/set instead of the from_c to_c, but I'm going to try it! Thanks for the comments!! Regards, Santiago. From stefan_ml at behnel.de Fri Mar 13 20:59:12 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 13 Mar 2009 20:59:12 +0100 Subject: [Cython] Trac usage and bug handling for 0.11.x In-Reply-To: <4DB56201-278E-4068-A185-4AB1947EC239@math.washington.edu> References: <61799.213.61.181.86.1236849271.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B8D940.3090202@student.matnat.uio.no> <4DB56201-278E-4068-A185-4AB1947EC239@math.washington.edu> Message-ID: <49BABB10.8070909@behnel.de> Robert Bradshaw wrote: > On Mar 12, 2009, at 2:43 AM, Dag Sverre Seljebotn wrote: >> Stefan Behnel wrote: >>> Or maybe "cython-unstable" would be a more generic name. >>> >> +1 to cython-devel and cython-unstable. > > Yes, I'll make such branches. > > I like the idea of "safe" stuff (either fixes or minor improvements) > being in cython-devel, and anything more drastic (e.g. moving all the > temp stuff over to the new model) in cython-unstable. A much better > division than "bugfix" and "other." I put an overview of trac usage and branch designation here: http://wiki.cython.org/HackerGuide Stefan From kallacomer at gmail.com Fri Mar 13 21:12:36 2009 From: kallacomer at gmail.com (comer c) Date: Sat, 14 Mar 2009 01:42:36 +0530 Subject: [Cython] [Help]Hacking the cython parser Code Message-ID: <78a091340903131312k222ad46s9de599683bdeeb30@mail.gmail.com> I was looking at the cython compiler code, after sometime i got lost tracing it. What is the best way to trace/analyze the code?For example if i want to see the different phases of the compiler like the tokenizer,parser and code generator?Can you provide me the proper way to begin with code analzing. Much help appreciated in this regard. Thanks, comer -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090314/4c1aee42/attachment.htm From stefan_ml at behnel.de Fri Mar 13 21:45:53 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 13 Mar 2009 21:45:53 +0100 Subject: [Cython] [Help]Hacking the cython parser Code In-Reply-To: <78a091340903131312k222ad46s9de599683bdeeb30@mail.gmail.com> References: <78a091340903131312k222ad46s9de599683bdeeb30@mail.gmail.com> Message-ID: <49BAC601.7040007@behnel.de> comer c wrote: > I was looking at the cython compiler code, after sometime i got lost tracing > it. Yes, there is a lot of code, and it's not always trivial to trace into it. > What is the best way to trace/analyze the code?For example if i want to see > the different phases of the compiler like the tokenizer,parser and code > generator?Can you provide me the proper way to begin with code analzing. > Much help appreciated in this regard. Have you looked into the compiler pipeline in Main.py? (~line 110). It runs after the parser, but the most interesting stuff happens there, actually. Just set a break point into a class that looks interesting and see what it does with the tree. To print the tree, call "node.dump()" at any time, or put a "PrintTree()" into the pipeline at any point. Stefan From robertwb at math.washington.edu Sat Mar 14 02:19:18 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 13 Mar 2009 18:19:18 -0700 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: <49BAB68A.7060303@gmail.com> References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> <49BA92FA.2070801@gmail.com> <49BAB68A.7060303@gmail.com> Message-ID: On Mar 13, 2009, at 12:39 PM, Santiago Aguiar wrote: > Lisandro Dalcin wrote: >> Santiago, have you considered writing a simple script to generate >> Cython code :-) ?? You could parse a simple txt file with metadata >> about your C-level structs (name, field types, fieldname, and even >> default field values), then you could easily dump Cython code >> automating the member, __cinit__() methods and all the 'property' you >> need. >> > I did, and I'm usually on for codegen, but since I was on a schedule I > wanted to avoid having to code the generator, and hoped that something > already existed.... besides, the metadata will still have to mirror > the > C defs... or I would need to parse the .h which... well I want to > avoid > if at all possible! However, it still could save me some time. There was a project (called ccython) that proposed to somewhat automated this kind of stuff. >> As Stefan commented, you should just store your struct in your cdef >> class, and use 'property' to expose the field values with tiny get/ >> set >> funcions... Remember that struct member assignments should work in >> any >> C89 compiler... No point to complicate your from_c()/ to_c() cdef >> methods ... >> > OK, I'm not terribly convinced that I will save some time by > writing the > get/set instead of the from_c to_c, but I'm going to try it! At least I think it would be a more natural interface on the Python side. > Thanks for the comments!! No problem. Thanks for using Cython, and the feedback. - Robert From robertwb at math.washington.edu Sat Mar 14 02:22:05 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 13 Mar 2009 18:22:05 -0700 Subject: [Cython] [Help]Hacking the cython parser Code In-Reply-To: <49BAC601.7040007@behnel.de> References: <78a091340903131312k222ad46s9de599683bdeeb30@mail.gmail.com> <49BAC601.7040007@behnel.de> Message-ID: On Mar 13, 2009, at 1:45 PM, Stefan Behnel wrote: > comer c wrote: >> I was looking at the cython compiler code, after sometime i got >> lost tracing >> it. > > Yes, there is a lot of code, and it's not always trivial to trace > into it. > > >> What is the best way to trace/analyze the code?For example if i >> want to see >> the different phases of the compiler like the tokenizer,parser and >> code >> generator?Can you provide me the proper way to begin with code >> analzing. >> Much help appreciated in this regard. > > Have you looked into the compiler pipeline in Main.py? (~line 110). > It runs > after the parser, but the most interesting stuff happens there, > actually. > Just set a break point into a class that looks interesting and see > what it > does with the tree. To print the tree, call "node.dump()" at any > time, or > put a "PrintTree()" into the pipeline at any point. For the code generation phase, you can uncomment the lines # Uncomment this for debugging. # __metaclass__ = VerboseCodeWriter In Node.py:119-120, which will very verbosely annotate the output (though it hasn't been fully debugged--sometimes the resulting file won't compile). - Robert From robertwb at math.washington.edu Sat Mar 14 08:08:09 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 14 Mar 2009 00:08:09 -0700 Subject: [Cython] cython-support or cython-users Message-ID: <821530A1-A6BA-4143-88D9-6E455CB091D4@math.washington.edu> Should we start a new list for cython support, aimed at end users (and hopefully lower volume) distinct from the devel list? I'd set one on up google groups, but some people have issues with their terms of service, and it'd probably be good to have it be at codespeak like this one. - Robert From stefan_ml at behnel.de Sat Mar 14 08:27:18 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 14 Mar 2009 08:27:18 +0100 Subject: [Cython] cython-support or cython-users In-Reply-To: <821530A1-A6BA-4143-88D9-6E455CB091D4@math.washington.edu> References: <821530A1-A6BA-4143-88D9-6E455CB091D4@math.washington.edu> Message-ID: <49BB5C56.2030707@behnel.de> Robert Bradshaw wrote: > Should we start a new list for cython support, aimed at end users > (and hopefully lower volume) distinct from the devel list? I do think that many of the developer topics are at least superficially interesting to users (Cython users are also Python developers and commonly care about the generated code). However, "superficially" already implies that the volume argument has a certain weight here. I can ask on codespeak.net, in case there is some demand by actual users. Stefan From robertwb at math.washington.edu Sat Mar 14 08:56:19 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 14 Mar 2009 00:56:19 -0700 Subject: [Cython] Cython 0.11 release announcement Message-ID: After much delay, Cython 0.11 is finally released! We would like to thank Stefan Behnel, Robert Bradshaw, Ondrej Certik, Lisandro Dalcin, Jason Evans, Magnus Lie Hetland, Hoyt Koepke, and Dag Sverre Seljebotn who contributed code to this release, as well as the many others who contributed ideas, bug reports, and feedback. The majority of changes in this version are internal, the largest of which is moving most temp allocation to the code generation phase which has several advantages and will greatly facilitate future development. In conjunction with this, Dag Sverre Seljebotn wrote a "reference counting nanny" which, when enabled, warns of potential double-frees and memory-leaks. Though it will not find memory leaks in user code (malloc is just as dangerous as ever) it is a great tool for verifying that Cython itself is outputting correct code and is an invaluable consistency check during the many changes we've been making to temporary variable handling. There have also been many improvements for developers--for example, the parse tree is printed on compiler crashes, the testing framework has been enhanced, and there is a mode to produce output files interlaced with the compiler's call stacks to more easily trace what's going on. The most visible new user-level features are size_t as a native type (as part of Lisandro Dalcin's cleanup and improvement of Python <-> C type conversions) and proper Python semantics for optimized range() loops (Magnus Lie Hetland). C(p)def functions can now take decorators, though only @cython.locals is currently supported, and inline functions can appear in .pxd files. There were also numerious bug fixes, optimizations, and general code cleanups that happened in this release too. A list of closed tickets can be found at http://trac.cython.org/ cython_trac/query?group=component&milestone=0.11 . Though not every change is associated to a trac ticket, we plan to be more stringent about this in the future. We have several patches already done that didn't make it into 0.11, and so 0.11.1 will be out soon. - Robert From robertwb at math.washington.edu Sat Mar 14 09:17:49 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 14 Mar 2009 01:17:49 -0700 (PDT) Subject: [Cython] Cython 0.11 Released Message-ID: <4da1608f-43b0-4ced-846a-cc329d581327@s9g2000prg.googlegroups.com> The Python compiler Cython 0.11 was just released. We would like to thank Stefan Behnel, Robert Bradshaw, Ondrej Certik, Lisandro Dalcin, Jason Evans, Magnus Lie Hetland, Hoyt Koepke, and Dag Sverre Seljebotn who contributed code to this release, as well as the many others who contributed ideas, bug reports, and feedback. What is Cython? ===================== Cython is a language that makes writing C extensions for the Python language as easy as Python itself. Cython is based on the well-known Pyrex, but supports more cutting edge functionality and optimizations. The Cython language is very close to the Python language, but Cython additionally supports calling C functions and declaring C types on variables and class attributes. This allows the compiler to generate very efficient C code from Cython code. This makes Cython the ideal language for wrapping for external C libraries, and for fast C modules that speed up the execution of Python code. Where to get it? ===================== http://cython.org/ http://pypi.python.org/pypi/Cython/ Release Notes ===================== The majority of changes in this version are internal, the largest of which is moving most temp allocation to the code generation phase which has several advantages and will greatly facilitate future development. In conjunction with this, Dag Sverre Seljebotn wrote a "reference counting nanny" which, when enabled, warns of potential double-frees and memory-leaks. Though it will not find memory leaks in user code (malloc is just as dangerous as ever) it is a great tool for verifying that Cython itself is outputting correct code and is an invaluable consistency check during the many changes we've been making to temporary variable handling. There have also been many improvements for developers--for example, the parse tree is printed on compiler crashes, the testing framework has been enhanced, and there is a mode to produce output files interlaced with the compiler's call stacks to more easily trace what's going on. The most visible new user-level features are size_t as a native type (as part of Lisandro Dalcin's cleanup and improvement of Python <-> C type conversions) and proper Python semantics for optimized range() loops (Magnus Lie Hetland). C(p)def functions can now take decorators, though only @cython.locals is currently supported, and inline functions can appear in .pxd files. There were also numerious bug fixes, optimizations, and general code cleanups that happened in this release too. A list of closed tickets can be found at http://trac.cython.org/ cython_trac/query?group=component&milestone=0.11 . Though not every change is associated to a trac ticket, we plan to be more stringent about this in the future. We have several patches already done that didn't make it into 0.11, and so 0.11.1 will be out soon. From kallacomer at gmail.com Sat Mar 14 20:55:54 2009 From: kallacomer at gmail.com (comer c) Date: Sun, 15 Mar 2009 01:25:54 +0530 Subject: [Cython] [Help]Hacking the cython parser Code In-Reply-To: References: <78a091340903131312k222ad46s9de599683bdeeb30@mail.gmail.com> <49BAC601.7040007@behnel.de> Message-ID: <78a091340903141255y65d110a0r204e6886461e6143@mail.gmail.com> I uncommented this line, and I tried with both these commands. Cython dummy.pyx Cython -v dummy.pyx and it did not generate any verbose output.The output was same as before, without uncommenting that line. Am i missing something ? On Sat, Mar 14, 2009 at 6:52 AM, Robert Bradshaw < robertwb at math.washington.edu> wrote: > On Mar 13, 2009, at 1:45 PM, Stefan Behnel wrote: > > > comer c wrote: > >> I was looking at the cython compiler code, after sometime i got > >> lost tracing > >> it. > > > > Yes, there is a lot of code, and it's not always trivial to trace > > into it. > > > > > >> What is the best way to trace/analyze the code?For example if i > >> want to see > >> the different phases of the compiler like the tokenizer,parser and > >> code > >> generator?Can you provide me the proper way to begin with code > >> analzing. > >> Much help appreciated in this regard. > > > > Have you looked into the compiler pipeline in Main.py? (~line 110). > > It runs > > after the parser, but the most interesting stuff happens there, > > actually. > > Just set a break point into a class that looks interesting and see > > what it > > does with the tree. To print the tree, call "node.dump()" at any > > time, or > > put a "PrintTree()" into the pipeline at any point. > > For the code generation phase, you can uncomment the lines > > # Uncomment this for debugging. > # __metaclass__ = VerboseCodeWriter > > In Node.py:119-120, which will very verbosely annotate the output > (though it hasn't been fully debugged--sometimes the resulting file > won't compile). > > - Robert > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090315/d1be8b4c/attachment.htm From robertwb at math.washington.edu Sun Mar 15 00:57:07 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 14 Mar 2009 16:57:07 -0700 Subject: [Cython] [Help]Hacking the cython parser Code In-Reply-To: <78a091340903141255y65d110a0r204e6886461e6143@mail.gmail.com> References: <78a091340903131312k222ad46s9de599683bdeeb30@mail.gmail.com> <49BAC601.7040007@behnel.de> <78a091340903141255y65d110a0r204e6886461e6143@mail.gmail.com> Message-ID: <05DD5AEC-37F4-4670-94B5-9E8FBA75DD9B@math.washington.edu> On Mar 14, 2009, at 12:55 PM, comer c wrote: > I uncommented this line, and I tried with both these commands. > Cython dummy.pyx Cython -v dummy.pyx and it did not generate any > verbose output.The output was same as before, without uncommenting > that line. > Am i missing something ? Look at the .c file. You should see stuff like /* -> ExprStatNode.generate_execution_code (18, 25) */ /* -> SimpleCallNode.generate_evaluation_code (18, 25) */ /* -> SimpleCallNode.generate_subexpr_evaluation_code (18, 25) */ /* -> CoerceToPyTypeNode.generate_evaluation_code (18, 27) */ / * -> CoerceToPyTypeNode.allocate_temp_result (18, 27) */ /* __pyx_t_1 allocated */ / * <- CoerceToPyTypeNode.allocate_temp_result (18, 27) */ / * -> CoerceToPyTypeNode.generate_result_code (18, 27) */ __pyx_t_1 = PyInt_FromLong(__pyx_v_n); if (unlikely(! __pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 18; __pyx_clineno = __LINE__; goto __pyx_L1_error;} __Pyx_GOTREF(__pyx_t_1); / * <- CoerceToPyTypeNode.generate_result_code (18, 27) */ /* <- CoerceToPyTypeNode.generate_evaluation_code (18, 27) */ /* <- SimpleCallNode.generate_subexpr_evaluation_code (18, 25) */ /* -> SimpleCallNode.allocate_temp_result (18, 25) */ /* __pyx_t_3 allocated */ /* <- SimpleCallNode.allocate_temp_result (18, 25) */ - Robert From robertwb at math.washington.edu Sun Mar 15 01:01:41 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 14 Mar 2009 17:01:41 -0700 Subject: [Cython] Trac usage and bug handling for 0.11.x In-Reply-To: <49BABB10.8070909@behnel.de> References: <61799.213.61.181.86.1236849271.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B8D940.3090202@student.matnat.uio.no> <4DB56201-278E-4068-A185-4AB1947EC239@math.washington.edu> <49BABB10.8070909@behnel.de> Message-ID: On Mar 13, 2009, at 12:59 PM, Stefan Behnel wrote: > Robert Bradshaw wrote: >> On Mar 12, 2009, at 2:43 AM, Dag Sverre Seljebotn wrote: >>> Stefan Behnel wrote: >>>> Or maybe "cython-unstable" would be a more generic name. >>>> >>> +1 to cython-devel and cython-unstable. >> >> Yes, I'll make such branches. >> >> I like the idea of "safe" stuff (either fixes or minor improvements) >> being in cython-devel, and anything more drastic (e.g. moving all the >> temp stuff over to the new model) in cython-unstable. A much better >> division than "bugfix" and "other." > > I put an overview of trac usage and branch designation here: > > http://wiki.cython.org/HackerGuide Thanks for the good writeup. - Robert From robertwb at math.washington.edu Sun Mar 15 06:24:28 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 14 Mar 2009 22:24:28 -0700 Subject: [Cython] Integer power operator In-Reply-To: <44766.213.61.181.86.1236769300.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <1286d6e2177fb46620ab4a1f1fa7d7f3.squirrel@webmail.uio.no> <39764.213.61.181.86.1236762819.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B78660.8060904@student.matnat.uio.no> <44766.213.61.181.86.1236769300.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <66738701-AD7B-4153-974C-67C94D5698EC@math.washington.edu> On Mar 11, 2009, at 4:01 AM, Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> Stefan Behnel wrote: >>> Dag Sverre Seljebotn wrote: >>>> I happen to routinely write (some complex integer expression)**2 in >>>> Python, and it is tedious to manually store the expression in a >>>> temporary >>>> etc. in Cython. >> >>> If you can come up with a sensible utility function that handles the >>> different C compile time and runtime overflow cases nicely, I'm >>> all for >>> it. >> >> So you are "+1 if I include an overflow check"? >> >> I was just thinking of letting it silently overflow, the way addition >> and multiplication already silently overflows. > > Hmm, yes, you are right. When a user writes > > cdef int tinyint = 2**2000 > > it's not Cython's fault if it overflows. > > So I guess I'm actually +1 for a fix that provides Python integer > power > semantics modulo C overflows/wrap-arounds. I guess that would be the > semantics with the lowest probability of surprise. > > Would you say that this makes sense as a utility function? > > I mean, we could always write out the bit-shifting code for a constant > exponent... http://trac.cython.org/cython_trac/ticket/127 I only special-cased 0-3, the rest is generic (repeated squaring) code. - Robert From jasone at canonware.com Sun Mar 15 06:47:32 2009 From: jasone at canonware.com (Jason Evans) Date: Sat, 14 Mar 2009 22:47:32 -0700 Subject: [Cython] Integer power operator In-Reply-To: <66738701-AD7B-4153-974C-67C94D5698EC@math.washington.edu> References: <1286d6e2177fb46620ab4a1f1fa7d7f3.squirrel@webmail.uio.no> <39764.213.61.181.86.1236762819.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B78660.8060904@student.matnat.uio.no> <44766.213.61.181.86.1236769300.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <66738701-AD7B-4153-974C-67C94D5698EC@math.washington.edu> Message-ID: <49BC9674.4090905@canonware.com> Robert Bradshaw wrote: > http://trac.cython.org/cython_trac/ticket/127 I only special-cased > 0-3, the rest is generic (repeated squaring) code. Here's some code I wrote a while back that uses binary decomposition to improve performance. I don't know how much it really matters... Jason // Use binary decomposition to reduce the number of multiplication operations // when calculating x^n. See "Hackers Delight", pp 212-213 for further // information. LkmpInline int64_t LkpIntExp(int64_t x, int64_t n) { int64_t ret, p; bool neg = (n < 0); // The main loop only works for non-negative exponents, so transform // negative exponents outside the loop, and clean up at the end with a // division. if (neg) { // This can overflow if n is -2^63, but the results are explicitly // undefined on underflow/overflow, so there is no need to check. n = -n; } for (ret = 1, p = x; n != 0; n >>= 1, p *= p) { if (n & 1) { ret *= p; } } if (neg) { ret = 1 / ret; } return ret; From dagss at student.matnat.uio.no Sun Mar 15 07:14:31 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 15 Mar 2009 07:14:31 +0100 Subject: [Cython] Integer power operator In-Reply-To: <66738701-AD7B-4153-974C-67C94D5698EC@math.washington.edu> References: <1286d6e2177fb46620ab4a1f1fa7d7f3.squirrel@webmail.uio.no> <39764.213.61.181.86.1236762819.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B78660.8060904@student.matnat.uio.no> <44766.213.61.181.86.1236769300.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <66738701-AD7B-4153-974C-67C94D5698EC@math.washington.edu> Message-ID: <49BC9CC7.2050103@student.matnat.uio.no> Robert Bradshaw wrote: > On Mar 11, 2009, at 4:01 AM, Stefan Behnel wrote: > >> Dag Sverre Seljebotn wrote: >>> Stefan Behnel wrote: >>>> Dag Sverre Seljebotn wrote: >>>>> I happen to routinely write (some complex integer expression)**2 in >>>>> Python, and it is tedious to manually store the expression in a >>>>> temporary >>>>> etc. in Cython. >>>> If you can come up with a sensible utility function that handles the >>>> different C compile time and runtime overflow cases nicely, I'm >>>> all for >>>> it. >>> So you are "+1 if I include an overflow check"? >>> >>> I was just thinking of letting it silently overflow, the way addition >>> and multiplication already silently overflows. >> Hmm, yes, you are right. When a user writes >> >> cdef int tinyint = 2**2000 >> >> it's not Cython's fault if it overflows. >> >> So I guess I'm actually +1 for a fix that provides Python integer >> power >> semantics modulo C overflows/wrap-arounds. I guess that would be the >> semantics with the lowest probability of surprise. >> >> Would you say that this makes sense as a utility function? >> >> I mean, we could always write out the bit-shifting code for a constant >> exponent... > > http://trac.cython.org/cython_trac/ticket/127 I only special-cased > 0-3, the rest is generic (repeated squaring) code. Great! -- Dag Sverre From robertwb at math.washington.edu Sun Mar 15 07:09:58 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 14 Mar 2009 23:09:58 -0700 Subject: [Cython] Integer power operator In-Reply-To: <49BC9674.4090905@canonware.com> References: <1286d6e2177fb46620ab4a1f1fa7d7f3.squirrel@webmail.uio.no> <39764.213.61.181.86.1236762819.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B78660.8060904@student.matnat.uio.no> <44766.213.61.181.86.1236769300.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <66738701-AD7B-4153-974C-67C94D5698EC@math.washington.edu> <49BC9674.4090905@canonware.com> Message-ID: <89C94664-180F-4C10-A169-A1F4A85D47DF@math.washington.edu> On Mar 14, 2009, at 10:47 PM, Jason Evans wrote: > Robert Bradshaw wrote: >> http://trac.cython.org/cython_trac/ticket/127 I only special-cased >> 0-3, the rest is generic (repeated squaring) code. > > Here's some code I wrote a while back that uses binary > decomposition to > improve performance. I don't know how much it really matters... > > Jason > > // Use binary decomposition to reduce the number of multiplication > operations > // when calculating x^n. See "Hackers Delight", pp 212-213 for > further > // information. > LkmpInline int64_t > LkpIntExp(int64_t x, int64_t n) { > int64_t ret, p; > bool neg = (n < 0); > > // The main loop only works for non-negative exponents, so > transform > // negative exponents outside the loop, and clean up at the > end with a > // division. > if (neg) { > // This can overflow if n is -2^63, but the results are > explicitly > // undefined on underflow/overflow, so there is no need to > check. > n = -n; > } > > for (ret = 1, p = x; > n != 0; > n >>= 1, p *= p) { > if (n & 1) { > ret *= p; > } > } > > if (neg) { > ret = 1 / ret; > } > > return ret; Yep, this is basically what I implemented. The switch at the top is handy for small exponents, and is primarily for constant exponents. The messy expression at the top of the loop is to avoid branching (may or may not be worth it--depends on how randomly the bits are set in the exponent). ---------- static INLINE %(type)s %(func_name)s(%(type)s b, %(type)s e) { %(type)s t = b; switch (e) { case 3: t *= b; case 2: t *= b; case 1: return t; case 0: return 1; } if (unlikely(e<0)) return 0; t = 1; while (likely(e)) { t *= (b * (e&1)) | ((~e)&1); /* 1 or b */ b *= b; e >>= 1; } return t; From stefan_ml at behnel.de Sun Mar 15 09:17:40 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 15 Mar 2009 09:17:40 +0100 Subject: [Cython] [Help]Hacking the cython parser Code In-Reply-To: References: <78a091340903131312k222ad46s9de599683bdeeb30@mail.gmail.com> <49BAC601.7040007@behnel.de> Message-ID: <49BCB9A4.3050307@behnel.de> Robert Bradshaw wrote: > For the code generation phase, you can uncomment the lines > > # Uncomment this for debugging. > # __metaclass__ = VerboseCodeWriter > > In Node.py:119-120, which will very verbosely annotate the output I moved the "uncommenting" step into the DebugFlags module as "debug_trace_code_generation" and added a section to the Hacker-Guide about this: http://wiki.cython.org/HackerGuide#Gettingstarted Stefan From m.gysel at gmail.com Sun Mar 15 13:50:59 2009 From: m.gysel at gmail.com (Martin Gysel) Date: Sun, 15 Mar 2009 13:50:59 +0100 Subject: [Cython] cdef class understanding Message-ID: <49BCF9B3.1030601@gmail.com> Hi I'm quite new to cython and try to bind some c code to python. Now I'm not sure if I understood the cython doc correctly since what I'm trying do do doesn't work :-( It looks like this: cdef class Cobj: cdef a_c_struct_t *loop def __init__(self): self.loop = c_lib_call() c_lib_call_reg_callback(self.callback, NULL) cdef void callback(self, void *userdata) with gil: # but here, self.loop seems to be NULL instead of something # to test this: # if self.loop == NULL: print "isNull" doSomething(self.loop) in the callback self.loop is NULL but in the init it wasn't. is it possible to access self and its members in a c callback? I also tried to use __cinit__ or __new__ instead of __init__ but always the same behavior. thanks fro your comments martin From ggellner at uoguelph.ca Sun Mar 15 18:52:15 2009 From: ggellner at uoguelph.ca (Gabriel Gellner) Date: Sun, 15 Mar 2009 13:52:15 -0400 Subject: [Cython] Optional arguments question Message-ID: <20090315175215.GA10289@encolpuis> I can't seem to get optional arguments for a char variable to work in a cdef function, is this a bug or am I missing something simple? Any help would, as always, be greatly appreciated. For example: cdef void test_args(int number, char letter='C'): print letter def test(): test_args(3, letter='F') Gives a cryptic compile compile error (It doesn't work with c'F' either). Gabriel From greg.ewing at canterbury.ac.nz Sun Mar 15 22:48:45 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 16 Mar 2009 09:48:45 +1200 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: <43407.213.61.181.86.1236955563.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> <43407.213.61.181.86.1236955563.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49BD77BD.9050609@canterbury.ac.nz> Stefan Behnel wrote: > Carl Witty wrote: > >>Hmm... I hope that eventually Cython will allow overriding __new__, at >>which point calling an extension type won't necessarily return a value >>of that type. > > I'm not sure. This should certainly be supported for Python classes, but > for extension types? What would be the use case? If it is supported, it could perhaps be handled by allowing the return type of the __new__ method to be declared explicitly, defaulting to the class. E.g. given cdef class Foo: def __new__(cls, ..): ... then Foo() is assumed to return something of type Foo, but given cdef class Blarg: def object __new__(cls, ..): ... then Blarg() could return any type of object. -- Greg From psp.reachable at gmail.com Mon Mar 16 04:53:37 2009 From: psp.reachable at gmail.com (Prajwal Suhas P) Date: Mon, 16 Mar 2009 09:23:37 +0530 Subject: [Cython] Beginner-friendly tickets In-Reply-To: <49B8263C.6090706@student.matnat.uio.no> References: <49B783FA.5070309@student.matnat.uio.no> <49B82598.3000303@student.matnat.uio.no> <49B8263C.6090706@student.matnat.uio.no> Message-ID: I changed the code to cdef int g(): print "g called" return 3 def f(): for i in range(g()): print i print i we get the right output here g called 0 1 2 2 the c output changes(and is wrong) when we declare a "cdef int i" before the for loop.i missed this part initially. On Thu, Mar 12, 2009 at 2:29 AM, Dag Sverre Seljebotn < dagss at student.matnat.uio.no> wrote: > Dag Sverre Seljebotn wrote: > > Kurt Smith wrote: > >> On Wed, Mar 11, 2009 at 4:27 AM, Dag Sverre Seljebotn > >> wrote: > >>> Since there's been some interest in how to get started coding on Cython > >>> on the mailing list lately, I thought I'd call attention to these two > >>> tickets: > >>> > >>> http://trac.cython.org/cython_trac/ticket/158 > >>> http://trac.cython.org/cython_trac/ticket/203 > >>> > >>> These seems to fit the sweet spot of not being too hard to fix for > >>> beginners (with some mentoring), but not being completely trivial > either. > >>> > >>> I'm happy to provide an attack strategy/small amount of mentoring for > >>> these tickets if anybody's interested. > >> Hi Dag, > >> > >> I'm interested in working on #158 if it isn't already claimed. > > > > Nice! I haven't heard anything on #158 yet, so consider it yours. I've > > assigned both tickets now in trac. > > > > You may want to get Trac accounts if you haven't already BTW, send an > > .htpasswd-file to Robert for that. > > > >> One solution (as mentioned in the ticket comments) would be to raise a > >> compilation error since the s variable is declared after it's used. I > >> imagine some checks would be in order to ensure this, but I don't have > >> ideas about where to start. > > > > Yep, that's the solution I'm thinking about. > > > > The Cython "parse tree" is processed in stages. Two of those are > > analyse_declarations and analyse_expressions. The problem here is that > > "cdef str s" is handled in the analyse_declarations step and then > > disappears; while the analyse_expressions happen afterwards, and cannot > > know at which point in the source the declaration happened. > > > > So the analyse declarations phase must do this check. > > > > You will need to do your work in AnalyseDeclarationsTransform, in > > ParseTreeTransforms.py. It is a "filter" on the parse tree, and each > > method is called as you go according to the type of the node. Read some > > of the different transforms in that file to get the idea (to see the > > parse tree, insert e.g. "print node.dump()" at the beginning of > > visit_ModuleNode). > > > > Now: > > > > a) Keep a dict in self containing the names that has been referenced up > > to that point in the given scope (i.e. in visit_ModuleNode and > > visit_FuncDefNode you need to push/pop which dict one is using > > before/after processing the contents). > > Actually I meant to say "set" everywhere I said "dict". > > -- > Dag Sverre > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Regards, Prajwal S. "There is little difference in people,but this little difference makes a big difference.The little difference is attitude,the big difference is whether its positive or negative. " -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090316/908dcd6d/attachment.htm From kwmsmith at gmail.com Mon Mar 16 06:26:49 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Mon, 16 Mar 2009 00:26:49 -0500 Subject: [Cython] Beginner-friendly tickets In-Reply-To: <49B8263C.6090706@student.matnat.uio.no> References: <49B783FA.5070309@student.matnat.uio.no> <49B82598.3000303@student.matnat.uio.no> <49B8263C.6090706@student.matnat.uio.no> Message-ID: Thanks for the guidance -- I'm making progress but I can't figure out one issue: when I add the following method to AnalyzeDeclarationsTransform everything works as expected: def visit_NameNode(self, node): print node.dump() return node It is called for every name node in the parse tree. but this one is not called for every matching node: def visit_CNameDeclaratorNode(self, node): print node.dump() return node Specifically, it is only called when the CNameDeclaratorNode is part of (a descendant of) a CArgDeclNode. When it's part of (a descendant of) a CVarDefNode it is not called. I'm happy to give an example if it would help. In general, if I add a visit_(self, node) method in one of the Transform classes that forms the pipeline, should I expect that node type to be visited provided it exists in the tree at that point in the pipeline? The bug is partially fixed, but if I can resolve this issue it will help fix some corner cases. Thanks, Kurt From dagss at student.matnat.uio.no Mon Mar 16 08:25:50 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 16 Mar 2009 08:25:50 +0100 Subject: [Cython] Beginner-friendly tickets In-Reply-To: References: <49B783FA.5070309@student.matnat.uio.no> <49B82598.3000303@student.matnat.uio.no> <49B8263C.6090706@student.matnat.uio.no> Message-ID: <49BDFEFE.70805@student.matnat.uio.no> Prajwal Suhas P wrote: > I changed the code to > > cdef int g(): > print "g called" > return 3 > > def f(): > for i in range(g()): > print i > print i > > we get the right output here > > g called > 0 > 1 > 2 > 2 > > the c output changes(and is wrong) when we declare a "cdef int i" before > the for loop.i missed this part initially. Yes, when "i" is not typed as int, the Python "range" function is called, which gives the right behaviour. The task is to make Cython do the right thing when i is declared "cdef int" as well. -- Dag Sverre From m.gysel at gmail.com Mon Mar 16 08:32:10 2009 From: m.gysel at gmail.com (Martin Gysel) Date: Mon, 16 Mar 2009 08:32:10 +0100 Subject: [Cython] cdef class understanding, 2nd Message-ID: <49BE007A.4050004@gmail.com> Hi It seems to me like my post yesterday didn't make it to the subscriber, at least not back to me although I should receive my own posts... which is listed in the list archive. So I repost me concern and hope I'll get a copy this time. If everyone out there already got my first post, sorry for that. I'm quite new to cython and try to bind some c code to python. Now I'm not sure if I understood the cython doc correctly since what I'm trying do do doesn't work :-( It looks like this: cdef class Cobj: cdef a_c_struct_t *loop def __init__(self): self.loop = c_lib_call() c_lib_call_reg_callback(self.callback, NULL) cdef void callback(self, void *userdata) with gil: # but here, self.loop seems to be NULL instead of something # to test this: # if self.loop == NULL: print "isNull" doSomething(self.loop) in the callback self.loop is NULL but in the init it wasn't. is it possible to access self and its members in a c callback? I also tried to use __cinit__ or __new__ instead of __init__ but always the same behavior. thanks fro your comments martin From stefan_ml at behnel.de Mon Mar 16 08:37:23 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 16 Mar 2009 08:37:23 +0100 Subject: [Cython] Beginner-friendly tickets In-Reply-To: References: <49B783FA.5070309@student.matnat.uio.no> <49B82598.3000303@student.matnat.uio.no> <49B8263C.6090706@student.matnat.uio.no> Message-ID: <49BE01B3.9070907@behnel.de> Kurt Smith wrote: > Thanks for the guidance -- I'm making progress but I can't figure out one issue: > > when I add the following method to AnalyzeDeclarationsTransform > everything works as expected: > > def visit_NameNode(self, node): > print node.dump() > return node > > It is called for every name node in the parse tree. > > but this one is not called for every matching node: > > def visit_CNameDeclaratorNode(self, node): > print node.dump() > return node > > Specifically, it is only called when the CNameDeclaratorNode is part > of (a descendant of) a CArgDeclNode. When it's part of (a descendant > of) a CVarDefNode it is not called. > > I'm happy to give an example if it would help. > > In general, if I add a visit_(self, node) method in one of > the Transform classes that forms the pipeline, should I expect that > node type to be visited provided it exists in the tree at that point > in the pipeline? There are two issues that might hit here. One is that (IIRC) declarators are removed from the tree at some point in the pipeline as they have done their duty after type analysis and would just increase the overall runtime of the remaining tree operations. Note the IIRC. At least, I'm sure about the fact that there is a transformation base class that ignores them explicitly. So watch out for the class hierarchy of your transformation. The second thing is that a method "visit_SomeNode()" only matches if there is not a more specific method match. E.g., "visit_Node()" will not get called for an expression node if there is a "visit_ExprNode()" sitting in the same class. Does that help? Stefan From dagss at student.matnat.uio.no Mon Mar 16 09:40:51 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 16 Mar 2009 09:40:51 +0100 Subject: [Cython] [Fwd: Re: Beginner-friendly tickets] Message-ID: <49BE1093.7080505@student.matnat.uio.no> Ahh; this was sent to my private address as well and so cython-dev in its wisdom decided not to send me a copy from the mailing list. Forwarding my response to the mailing list as well then. -------- Original Message -------- Subject: Re: [Cython] Beginner-friendly tickets Date: Mon, 16 Mar 2009 08:23:26 +0100 From: Dag Sverre Seljebotn To: Kurt Smith References: <49B783FA.5070309 at student.matnat.uio.no> <49B82598.3000303 at student.matnat.uio.no> <49B8263C.6090706 at student.matnat.uio.no> Kurt Smith wrote: > Thanks for the guidance -- I'm making progress but I can't figure out one issue: > > when I add the following method to AnalyzeDeclarationsTransform > everything works as expected: > > def visit_NameNode(self, node): > print node.dump() > return node > > It is called for every name node in the parse tree. > > but this one is not called for every matching node: > > def visit_CNameDeclaratorNode(self, node): > print node.dump() > return node > > Specifically, it is only called when the CNameDeclaratorNode is part > of (a descendant of) a CArgDeclNode. When it's part of (a descendant > of) a CVarDefNode it is not called. Great that you're making progress. Yes, there is something I'd rather tell you about this :-) The transform works by recursive descent, and when a node type is intercepted, one needs to actively visit the children as well. This is done by self.visitchildren(node); see in visit_FuncDefNode for an example. Since visitchildren is never called in CVarDefNode, the children of CVarDefNode are never visited. Also, the node will be replaced with whatever it returns. visit_CVarDefNode returns None so that the entire declaration is removed from the tree, as it is no longer needed (which is why one isn't analysing the children as well). (Depending on what you need this for, you could be able to extract the information from the attributes of the enclosing FuncDefNode; I think there's a Scope there somewhere which you can inspect the entries of.) -- Dag Sverre -- Dag Sverre From dagss at student.matnat.uio.no Mon Mar 16 10:35:55 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 16 Mar 2009 10:35:55 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> Message-ID: <49BE1D7B.7090009@student.matnat.uio.no> Robert Bradshaw wrote: > On Mar 12, 2009, at 2:15 AM, William Stein wrote: > >> On Thu, Mar 12, 2009 at 1:46 AM, Dag Sverre Seljebotn >> wrote: >>>> This could be very useful for debugging things, but it implies >>>> there's a single, correct way that the % and // operators behave. >>>> >>>> The problem is that sometimes I want to run code with Python >>>> semantics (e.g. I'm quickly cythonizing a file) and sometimes I want >>>> to run code with C semantics (e.g. I'm doing linear algebra mod p, >>>> and don't want the overhead of fixing the sign). And perhaps I'm to >>>> demanding, but I want to be able to use % in both places rather than >>>> know some obscure function call. >>>> >>> Perhaps you could ask the Sage list to get some more input of what >>> the >>> typical expectations to Cython are? >>> >>> (Also if you work in Z_p, could you not use unsigned ints? Then you >>> don't get the overhead? Though I'm avoiding unsigned types like the >>> plague myself after I figured out that range(-n, n) is empty if n is >>> unsigned :-)) >> If you compute the sum of x and y mod p by doing "(x+y)%p", then the >> time is dominated by the % operation, which can easily be nearly 10 >> times longer than +. Now imagine doing linear algebra, where you take >> dot products, etc., and do lots of mod p arithmetic. If you dot two >> vectors with n entries in the naive way you end up doing n %p's, which >> is very expensive. Now imagine that you represent the entries of your >> vectors as C ints between -p/2 and p/2. Then you can very often do >> much of the dot product and only have to do very few mods, since >> frequently when you add up a bunch of numbers between -(p/2)^2 and >> (p/2)^2, they are likely to not overflow. The sum will be close to 0 >> because of cancellation. > > I was about to answer the very same thing then this email came in. > Also, as you mentioned unsigned ints have subtle semantics, like the > above. Another bad (easy--I've corrected it several times) mistake is > when one writes (a-b)%p. If a and b are unsigned, this could be > totally wrong. Of course, the main goal is to avoid using any > division as much as possible. > >>> Perhaps you could ask the Sage list to get some more input of what >>> the >>> typical expectations to Cython are? >> I often think of Cython primarily as a "Python compiler", and from >> that perspective (-1 % 16) being different in Cython and Python is >> worrisome, since it will surely lead to subtle bugs. For me, >> Cython is all about writing fast code that is easy to use from Python. >> Cython is not about writing C code; if I want to write C code I write >> it in C. Cython is not the C language after all. Also, I think many >> current and potential users of Cython (at least through Sage) don't >> even know C, or if they do, they are using Cython so they don't have >> to use C. > > Good point. I'm going to start a thread there just to get a straw > poll. It's the largest community of Cython users I know of... I did the same for NumPy. The results are here: http://thread.gmane.org/gmane.comp.python.numeric.general/28726 (And sage-devel thread is here: http://groups.google.com/group/sage-devel/browse_thread/thread/be9e2ef6a9745575 ) Both communities seems to mirror the discussion we've already had here very closely. There's one group who sees Cython as "fast Python" who want Python semantics and another who sees it as "mixed Python and C" who want the C semantics. (BTW NumPy uses the C semantics both for ints and floats, i.e. when doing % or // on arrays.) -- Dag Sverre From stefan_ml at behnel.de Mon Mar 16 11:59:15 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 16 Mar 2009 11:59:15 +0100 (CET) Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49BE1D7B.7090009@student.matnat.uio.no> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> Message-ID: <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > Both communities seems to mirror the discussion we've already had here > very closely. There's one group who sees Cython as "fast Python" who > want Python semantics and another who sees it as "mixed Python and C" > who want the C semantics. I was also rather unsure at the beginning, but when I think of the consequences for totally Python-looking Cython source code like [ y % 5 for y in some_iterable ] where you have to a) look up the type of "y" *and* b) know the associated semantics of the % operator, before you even know what the numeric range of the result will be - I think that having C semantics here is just too dangerous to be the default. We should not forget that the syntax we write in is Python, so it's a sure trap to assign C semantics to obvious looking things like the above. My clear vote is for Python-semantics on "x%y" and for a helper function "cython.cmod(x,y)" that boils down to a straight and fast C mod operator without value fixing. Stefan PS: Dag, I read your comment on "(2147483647 + 2147483647)" wrapping around in current Cython. I never thought about that, simply since I never ran into it. However, when I write py_large_val = 2147483647 + 2147483647 it must never wrap around as it clearly requests Python semantics, and when I write cdef uint64 c_large_val = 2147483647 + 2147483647 it must not wrap around either. I'm fine with a wrap-around in cdef int c_large_val = 2147483647 + 2147483647 though, because that's what the user a) requested or at least b) must be aware of when explicitly assigning types. So that's clearly different from a discussion on operator semantics. From stefan_ml at behnel.de Mon Mar 16 12:23:15 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 16 Mar 2009 12:23:15 +0100 Subject: [Cython] cdef class understanding In-Reply-To: <49BCF9B3.1030601@gmail.com> References: <49BCF9B3.1030601@gmail.com> Message-ID: <49BE36A3.7000500@behnel.de> Hi, welcome to Cython. Martin Gysel wrote: > I'm quite new to cython and try to bind some c code to python. Now I'm > not sure if I understood the cython doc correctly since what I'm trying > do do doesn't work :-( > It looks like this: > > cdef class Cobj: > cdef a_c_struct_t *loop > > def __init__(self): > self.loop = c_lib_call() > c_lib_call_reg_callback(self.callback, NULL) > > cdef void callback(self, void *userdata) with gil: > # but here, self.loop seems to be NULL instead of something > # to test this: > # if self.loop == NULL: print "isNull" > doSomething(self.loop) > > in the callback self.loop is NULL but in the init it wasn't. is it > possible to access self and its members in a c callback? Yes. However, since "callback" is a C function (and not a Python method), it cannot remember what instance it belongs to. You will have to pass along the pointer to its Python object yourself. This is what the "userdata" pointer is there for. It should work to do this: cdef class Cobj: cdef a_c_struct_t *loop def __init__(self): self.loop = c_lib_call() c_lib_call_reg_callback(self.callback, self) cdef void callback(self) with gil: if self.loop == NULL: print "isNull" doSomething(self.loop) There is also a callback example under "Demos/callback/" in the Cython sources. Stefan From m.gysel at gmail.com Mon Mar 16 14:08:25 2009 From: m.gysel at gmail.com (Martin Gysel) Date: Mon, 16 Mar 2009 14:08:25 +0100 Subject: [Cython] cdef class understanding In-Reply-To: <49BE36A3.7000500@behnel.de> References: <49BCF9B3.1030601@gmail.com> <49BE36A3.7000500@behnel.de> Message-ID: <49BE4F49.8090005@gmail.com> Stefan Behnel schrieb: > Hi, > > welcome to Cython. > > Martin Gysel wrote: >> I'm quite new to cython and try to bind some c code to python. Now I'm >> not sure if I understood the cython doc correctly since what I'm trying >> do do doesn't work :-( >> It looks like this: >> >> cdef class Cobj: >> cdef a_c_struct_t *loop >> >> def __init__(self): >> self.loop = c_lib_call() >> c_lib_call_reg_callback(self.callback, NULL) >> >> cdef void callback(self, void *userdata) with gil: >> # but here, self.loop seems to be NULL instead of something >> # to test this: >> # if self.loop == NULL: print "isNull" >> doSomething(self.loop) >> >> in the callback self.loop is NULL but in the init it wasn't. is it >> possible to access self and its members in a c callback? > > Yes. However, since "callback" is a C function (and not a Python method), > it cannot remember what instance it belongs to. You will have to pass along > the pointer to its Python object yourself. This is what the "userdata" > pointer is there for. It should work to do this: > > > cdef class Cobj: > cdef a_c_struct_t *loop > > def __init__(self): > self.loop = c_lib_call() > c_lib_call_reg_callback(self.callback, self) > > cdef void callback(self) with gil: > if self.loop == NULL: print "isNull" > doSomething(self.loop) thanks Stefan, but now it is a little bit more complex as my callback expects more than one argument to register: c_lib_call_reg_callback(mycontext, self.callback, self) and the callback (as plain c fuction) void callback(context *c, void *userdata) according to your code above, the first argument where the callback gets called is the first argument of the declared callback (in code above, cb(userdata) -> callback(self): userdata->self) so I assume if I have more arguments it works in the same way but then self would be context instead of userdata (at this point code seems to get kind of ugly): cdef void callback(self, void *userdata) now I'm asking myself what's the benefit to declare a c callback in the class rather then outside and and call another class method from there, like this: cdef void callback(context *c, void *userdata): doWhatHasToBeDone(c) anotherOperation((userdata).loop) itsDone((userdata).cbDone()) what's the best way to do such things? /martin From stefan_ml at behnel.de Mon Mar 16 14:41:49 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 16 Mar 2009 14:41:49 +0100 (CET) Subject: [Cython] cdef class understanding In-Reply-To: <49BE4F49.8090005@gmail.com> References: <49BCF9B3.1030601@gmail.com> <49BE36A3.7000500@behnel.de> <49BE4F49.8090005@gmail.com> Message-ID: <33575.213.61.181.86.1237210909.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Martin Gysel wrote: > it is a little bit more complex as my callback > expects more than one argument > to register: > c_lib_call_reg_callback(mycontext, self.callback, self) > > and the callback (as plain c fuction) > void callback(context *c, void *userdata) > > according to your code above, the first argument where the callback gets > called is the first argument of the declared callback (in code above, > cb(userdata) -> callback(self): userdata->self) Yes. That's a special case that would have worked in your original example. > now I'm asking myself what's the benefit to declare a c callback in the > class rather then outside and and call another class method from there, > like this: > cdef void callback(context *c, void *userdata): > doWhatHasToBeDone(c) > anotherOperation((userdata).loop) > itsDone((userdata).cbDone()) > > what's the best way to do such things? It's perfectly ok (and very common, due to signature constraints) to define a callback function outside of a class and to call back into some class instance from inside the callback (just as you do above). In many cases, your callback context/user-data will be a Python object anyway. It really just depends on what your callback does. Stefan From dagss at student.matnat.uio.no Mon Mar 16 15:54:23 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 16 Mar 2009 15:54:23 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49BE681F.8080005@student.matnat.uio.no> Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: > >> Both communities seems to mirror the discussion we've already had here >> very closely. There's one group who sees Cython as "fast Python" who >> want Python semantics and another who sees it as "mixed Python and C" >> who want the C semantics. >> > > I was also rather unsure at the beginning, but when I think of the > consequences for totally Python-looking Cython source code like > > [ y % 5 for y in some_iterable ] > > where you have to a) look up the type of "y" *and* b) know the associated > semantics of the % operator, before you even know what the numeric range > of the result will be - I think that having C semantics here is just too > dangerous to be the default. We should not forget that the syntax we write > in is Python, so it's a sure trap to assign C semantics to obvious looking > things like the above. > > My clear vote is for Python-semantics on "x%y" and for a helper function > "cython.cmod(x,y)" that boils down to a straight and fast C mod operator > without value fixing. > Agreed...but like Robert I'm worried about the usability of requiring to use cython.cmod and cython.cdiv (the latter being kist as important here); especially as one would typically use them in lots of situations (where one is only dividing positive integers anyway) to avoid the speed penalty (the 20% penalty reported by William is a bit much!). So I'm myself leaning towards new operators (in addition to cython.cmod/cdiv) for the C semantics (e.g. /- and %-, or ^/ and ^%, or similar -- as long as we stay away from /~ etc which is proposed in some PEPs). Yes, this is ugly! -- but I see them as less ugly than the alternatives. It seems that tomorrow I may have a different opinion though, so I may just let the matter rest for some days for what I'm concerned. Dag Sverre From stefan_ml at behnel.de Mon Mar 16 14:28:34 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 16 Mar 2009 14:28:34 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <85e81ba30903121618q2506d10fxb9ab6b5899032a4f@mail.gmail.com> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <85e81ba30903121618q2506d10fxb9ab6b5899032a4f@mail.gmail.com> Message-ID: <49BE5402.7050009@behnel.de> Hi, William Stein wrote: > I just tried these two programs: > > main() { > int n = -1, m = 16, k, a=0; > for(k=1;k<100000000;k++) { > a = n%k + (a<0)?k:0; > } > } Note that this does not have the proposed semantics. > and > > main() { > int n = -1, m = 16, k, a=0; > for(k=1;k<100000000;k++) { > a = n%k; > } > } > > and the first takes 22% longer than the second. Here's what I tried. I added another factor of 10 to the loop to get readable timings. ------------------------------------ /* cmod.c */ main() { int n = -1, m = 16, k, a=0; for(k=1;k<1000000000;k++) { a += n%k; n = -n; } printf("%i", a); } /* py2mod.c */ main() { int n = -1, m = 16, k, a=0; for(k=1;k<1000000000;k++) { int mod = n%k; if (mod<0) mod+=k; a += mod; n = -n; } printf("%i", a); } /* pymod.c */ main() { int n = -1, m = 16, k, a=0; for(k=1;k<1000000000;k++) { int mod = n%k; mod += (mod<0)*k; a += mod; n = -n; } printf("%i", a); } ---------------------------------- ./cmod 0 real 0m3.789s user 0m3.704s sys 0m0.000s ./py2mod -371654657 real 0m4.893s user 0m4.856s sys 0m0.004s ./pymod -371654657 real 0m5.623s user 0m5.452s sys 0m0.004s So, yes, there is a performance difference of up to 30% even for the fastest (BTW, branching) implementation. For a constant power-of-2 divisor (m=16), the difference is about 17% for me: ./cmod -1 real 0m3.316s user 0m3.268s sys 0m0.000s ./py2mod -589934593 real 0m3.880s user 0m3.868s sys 0m0.000s ./pymod -589934593 real 0m4.634s user 0m4.580s sys 0m0.000s When I switch to "unsigned int" for "mod" (i.e. for "unsigned int n=3"), I end up with about the same numbers as for cmod (signed or unsigned), which means that gcc drops the if-branch as expected. Looking at the absolute numbers, a speed penalty of even 30% does not sound too bad for me. If better performance is really required (i.e. inside a tight loop), you can switch to "unsigned int" (if you know that you never use negative values), or use a dedicated helper function in the "cython" namespace. Anyway, given that I'm not a heavy numerics user of '%' I guess it's better to take my weight out of this discussion. The (artificially determined) 30% might still be too much for some people, when it's the default. Stefan From sturla at molden.no Mon Mar 16 16:05:37 2009 From: sturla at molden.no (Sturla Molden) Date: Mon, 16 Mar 2009 16:05:37 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49BE6AC1.3000303@molden.no> On 3/16/2009 11:59 AM, Stefan Behnel wrote: > My clear vote is for Python-semantics on "x%y" and for a helper function > "cython.cmod(x,y)" that boils down to a straight and fast C mod operator > without value fixing. As I see it, cdef int a, b, c is a request to "step into C" when using these variables. Congratulation on your efforts to produce a broken C. "If it ain't broke, don't fix it." Sturla Molden From stefan_ml at behnel.de Mon Mar 16 16:40:17 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 16 Mar 2009 16:40:17 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49BE6AC1.3000303@molden.no> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> Message-ID: <49BE72E1.608@behnel.de> Sturla Molden wrote: > On 3/16/2009 11:59 AM, Stefan Behnel wrote: > >> My clear vote is for Python-semantics on "x%y" and for a helper function >> "cython.cmod(x,y)" that boils down to a straight and fast C mod operator >> without value fixing. > > As I see it, > > cdef int a, b, c > > is a request to "step into C" when using these variables. It's hard to have an argument about personal impressions (as in "as I see it"), so I won't go there. But let me clarify that cdef int a, b, c means: "use C type 'int' for a, b and c". This changes the value range of the variables and makes various operators overflow if the value range is too tight. That's what users must expect when they assign a numeric type with a restricted value range. It's also ok if an operator behaves different for an "int" and an "unsigned int", because both are explicitly declared as such by the user in order to make exactly the distinction between an unsigned and a signed type. It's a very different thing if an operator changes its target value range because of a type declaration that is perfectly suited to match the expected target range. A C "int" can represent all values that "some_pyint%5" can take. The same applies to a Python int target and a C "unsigned int" target because of the semantics of the % operator in Python. All three are equivalent from the point of view of this operation. Here, it is simply not true that the user "must" expect the semantics to change when typing the variable. Instead, it comes as a side effect, because it's not the variable with its assigned value range that imposes these semantics, but the operator itself, which simply behaves different when working on exactly the same input. And it behaves different in a place that can be very hard to find, as it's extremely unlikely that users back this kind of code by unit tests. Personally, I think Cython should not be designed to enforce unexpected code behaviour, and definitely not by default. Stefan From hoytak at cs.ubc.ca Mon Mar 16 17:03:51 2009 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Mon, 16 Mar 2009 09:03:51 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49BE6AC1.3000303@molden.no> References: <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> Message-ID: <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> Hello, I'm +1 on preserving python semantics wherever possible. In my field (statistics), many people don't have formal training in anything but R. I grew up programming in C, so keeping all these issues in mind isn't hard for me, but trying to teach people small relevant smatterings of C while showing them how to cythonize some python code can be a pain. It can also convey the idea that there are dozens of other corner cases to keep in mind, and that fear can be kinda discouraging. If we say that writing 'cdef int' is a request for c semantics, and that is the way you have to view it to write correct code, then the implication is that you have to know C to understand Cython. That puts an extra barrier in front of programmers that don't know C, and there are probably a lot more people in that camp than most of us realize. My understanding of the philosophy of cython is that it is trying to be as semantically python-like as possible (at least that's how it's been presented), and I don't really want that to change. Also, adding type information to code when cythonizing it shouldn't change the behavior of the code in subtle ways (as has been argued). Departing from python semantics with cython code is implicitly saying that the proper workflow is just to write in cython first instead of first writing python code and converting it to cython. I kinda like the idea of having two types that are only really different in corner cases like these, e.g. cint , clong, (maybe int, long), and that these are an explicit request for c semantics in every way. Other types like int (maybe py_int?) are a request for python sematics, even if it means a slight speed penalty over c semantics. Making this distinction explicit would seem to make everyone happy -- both the python coders who want speed and the c coders who work in python -- though it would probably mean more work for the cython developers. It also seems like more of a long-term solution, as there are other cases this would resolve (like overflows in c but not python). --Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From sturla at molden.no Mon Mar 16 18:01:22 2009 From: sturla at molden.no (Sturla Molden) Date: Mon, 16 Mar 2009 18:01:22 +0100 (CET) Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> References: <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> Message-ID: <608839d782db7ef5110e9c71b34f041e.squirrel@webmail.uio.no> > I kinda like the idea of having two types that are only really > different in corner cases like these, e.g. cint , clong, (maybe int, > long), and that these are an explicit request for c semantics in every > way. Other types like int (maybe py_int?) are a request for python > sematics, even if it means a slight speed penalty over c semantics. > Making this distinction explicit would seem to make everyone happy There is a big source of inconsistency here. First, there is no such thing as an unsigned Python int, a Python char or a Python short. There is no such thing as a Python pointer. And a Python float is a C double, and a C float does not exist in Python. What should this mean? cdef unsigned int a, b, c cdef char *a, b # C char, Python bytes, Python string? # a signed Python int that holds a pointer? cdef float a, b # single or double precision? who can tell? cdef double a cdef short b cdef long c # what's this? a Python 2 long? a C long? # C long in Python 3 and Python long in 2.5? cdef bytes c # Python 3 bytes, C typedef, whatever? cdef dict d # a dict right? or a ctypedef perhaps? Explicitely declaring Python types is fine. But then this should be restricted to actual Python types! Not try to make 25% of C types Python-like, and leve the rest to behave as in C. Now we get a complete mess of C and Python semantics, never knowing what we get. There should be a clear distinction between Python objects and primitive C types. Further, Python as an official way to make type annotation. If Cython wants to be as Python-like as possible, use the f*cking Python 3 type annotations - don't use cdefs. Making a statically typed Python and braking C is very different. And the Cython team is breaking C. Sturla Molden From carl.witty at gmail.com Mon Mar 16 18:20:12 2009 From: carl.witty at gmail.com (Carl Witty) Date: Mon, 16 Mar 2009 10:20:12 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49BE5402.7050009@behnel.de> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <85e81ba30903121618q2506d10fxb9ab6b5899032a4f@mail.gmail.com> <49BE5402.7050009@behnel.de> Message-ID: On Mon, Mar 16, 2009 at 6:28 AM, Stefan Behnel wrote: > So, yes, there is a performance difference of up to 30% even for the > fastest (BTW, branching) implementation. For a constant power-of-2 divisor > (m=16), the difference is about 17% for me: > > ./cmod > -1 > real ? ?0m3.316s > user ? ?0m3.268s > sys ? ? 0m0.000s > ./py2mod > -589934593 > real ? ?0m3.880s > user ? ?0m3.868s > sys ? ? 0m0.000s > ./pymod > -589934593 > real ? ?0m4.634s > user ? ?0m4.580s > sys ? ? 0m0.000s But note that for constant power-of-2 divisors, getting Python semantics by using a bitmask is actually faster than getting C semantics: cwitty at magnetar:/tmp$ time ./cmod -1 real 0m1.875s user 0m1.872s sys 0m0.004s cwitty at magnetar:/tmp$ time ./py3mod -589934593 real 0m1.214s user 0m1.212s sys 0m0.000s This is because Python semantics is a single AND instruction, but C semantics uses AND plus some fixups to get the negative result. (So your pymod and py2mod above are using an AND, subtracting to get the negative result, and then adding to undo the subtractions.) Carl From stefan_ml at behnel.de Mon Mar 16 18:26:11 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 16 Mar 2009 18:26:11 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <608839d782db7ef5110e9c71b34f041e.squirrel@webmail.uio.no> References: <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> <608839d782db7ef5110e9c71b34f041e.squirrel@webmail.uio.no> Message-ID: <49BE8BB3.9070400@behnel.de> Sturla Molden wrote: > There is a big source of inconsistency here. First, there is no such thing > as an unsigned Python int, a Python char or a Python short. No-one is questioning that. Please read the subject of this thread. > Explicitely declaring Python types is fine. But then this should be > restricted to actual Python types! Not try to make 25% of C types > Python-like, and leve the rest to behave as in C. Now we get a complete > mess of C and Python semantics, never knowing what we get. There should be > a clear distinction between Python objects and primitive C types. There is a very clear distinction between all types, including Python types and C types. The internals are not always simple, but the syntax is and the semantics tries to be, which is the thing that matters. This discussion is not about right or wrong, it's about good semantics that matches the expectations that users have of the language. Semantics that keeps users from getting things wrong because of bad language design. And it's about a trade-off between expected semantics and performance, which is what makes this tricky. It's also a discussion about the design goals of Cython, which, I assume, is the reason why people are having very strong opinions on this. > Further, Python as an official way to make type annotation. The annotations in Python 3 are by no means as powerful as the ones in Cython. So this argument is not very convincing. > If Cython > wants to be as Python-like as possible, use the f*cking Python 3 type > annotations - don't use cdefs. May I ask you to rethink your wording? > Making a statically typed Python and braking C is very different. And the > Cython team is breaking C. The correct word is "generating" here. Stefan From carl.witty at gmail.com Mon Mar 16 18:36:40 2009 From: carl.witty at gmail.com (Carl Witty) Date: Mon, 16 Mar 2009 10:36:40 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> References: <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> Message-ID: On Mon, Mar 16, 2009 at 9:03 AM, Hoyt Koepke wrote: > I kinda like the idea of having two types that are only really > different in corner cases like these, e.g. cint , clong, (maybe int, > long), and that these are an explicit request for c semantics in every > way. Other types like int (maybe py_int?) are a request for python > sematics, even if it means a slight speed penalty over c semantics. > Making this distinction explicit would seem to make everyone happy -- > both the python coders who want speed and the c coders who work in > python -- though it would probably mean more work for the cython > developers. ?It also seems like more of a long-term solution, as there > are other cases this would resolve (like overflows in c but not > python). Ooh, I definitely like this idea. (Although it certainly has its own set of complications, like deciding which semantics and return type "wins" if you divide a C-variant machine int by a Python-variant machine int.) It feels much nicer to me than any of the other "have it both ways" suggestions (compiler flags, compiler pragmas, multiple operator names, built-in functions). One minor note: I don't like the name py_int, because my first thought on seeing "cdef py_int i" would be that i holds a PyIntObject* (the C type that corresponds to the Python-language int type). Carl From sturla at molden.no Mon Mar 16 18:43:22 2009 From: sturla at molden.no (Sturla Molden) Date: Mon, 16 Mar 2009 18:43:22 +0100 (CET) Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49BE8BB3.9070400@behnel.de> References: <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> <608839d782db7ef5110e9c71b34f041e.squirrel@webmail.uio.no> <49BE8BB3.9070400@behnel.de> Message-ID: > There is a very clear distinction between all types, including Python > types > and C types. The internals are not always simple, but the syntax is and > the > semantics tries to be, which is the thing that matters. You are adding Python syntax to some but not all C types, with the argument that you are trying to make Cython an optionally statically typed Python. So 'cdef int' gives the behaviour of a Python int, whilst 'cdef float' gives the behaviour of a C float. > This discussion is not about right or wrong, it's about good semantics > that > matches the expectations that users have of the language. What does we expect cdef float a to mean? Single or double precision? Sturla Molden From sturla at molden.no Mon Mar 16 18:46:43 2009 From: sturla at molden.no (Sturla Molden) Date: Mon, 16 Mar 2009 18:46:43 +0100 (CET) Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> Message-ID: <98e90d5f7f0efa2cdd80e99572744339.squirrel@webmail.uio.no> > On Mon, Mar 16, 2009 at 9:03 AM, Hoyt Koepke wrote: > One minor note: I don't like the name py_int, because my first thought > on seeing "cdef py_int i" would be that i holds a PyIntObject* (the C > type that corresponds to the Python-language int type). What about a different keyword for declaring Python types? pdef? cpdef? Or simply use Python 3 syntax for type annotation? From hoytak at cs.ubc.ca Mon Mar 16 18:50:49 2009 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Mon, 16 Mar 2009 10:50:49 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> Message-ID: <4db580fd0903161050m7e4baf48wa793bb43a85b3195@mail.gmail.com> >> I kinda like the idea of having two types that are only really >> different in corner cases like these, e.g. cint , clong, (maybe int, >> long), and that these are an explicit request for c semantics in every >> way. Other types like int (maybe py_int?) are a request for python >> sematics, even if it means a slight speed penalty over c semantics. >> Making this distinction explicit would seem to make everyone happy -- >> both the python coders who want speed and the c coders who work in >> python -- though it would probably mean more work for the cython >> developers. ?It also seems like more of a long-term solution, as there >> are other cases this would resolve (like overflows in c but not >> python). > > Ooh, I definitely like this idea. Thanks. It seems like the only places where this matters are with integers. Thus if cython had one type, say py_integer (to distinguish it from PyIntObject*, like you mentioned), which in the C code would be a long or long long, but the semantics in the generated code would always be determined by python syntax. > (Although it certainly has its own > set of complications, like deciding which semantics and return type > "wins" if you divide a C-variant machine int by a Python-variant > machine int.) Maybe, though I see this as a fall-back default for people who don't want to work through making sure the C types works. Thus defaulting to py_integer whenever there's a question would make sense. The only gotchas here are with integer division (upcast to double) and overflow, and these issues may make this option more difficult (unless you go with the matlab way of making everything a double and downcasting to integers as appropriate.) As for floats, just alias py_float with double. With this in place, cdef int, long, etc. could be as in c, and I'd have a way to teach people to cythonize their python code without worrying about weird corner cases. -- Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From sturla at molden.no Mon Mar 16 18:59:08 2009 From: sturla at molden.no (Sturla Molden) Date: Mon, 16 Mar 2009 18:59:08 +0100 (CET) Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <4db580fd0903161050m7e4baf48wa793bb43a85b3195@mail.gmail.com> References: <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> <4db580fd0903161050m7e4baf48wa793bb43a85b3195@mail.gmail.com> Message-ID: <8257bb86d0e7b8f86cff4e60ecf4ba96.squirrel@webmail.uio.no> > It seems like the only places where this matters are with integers. 0.0/0.0 gives a NAN in C and raises an exception in Python. > As for floats, just alias py_float with double. So a cdef int s a py_int, but a cdef float is not a py_float? Sturla Molden From stefan_ml at behnel.de Mon Mar 16 19:02:47 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 16 Mar 2009 19:02:47 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> <608839d782db7ef5110e9c71b34f041e.squirrel@webmail.uio.no> <49BE8BB3.9070400@behnel.de> Message-ID: <49BE9447.4080107@behnel.de> Sturla Molden wrote: >> There is a very clear distinction between all types, including Python >> types >> and C types. The internals are not always simple, but the syntax is and >> the semantics tries to be, which is the thing that matters. > > You are adding Python syntax to some but not all C types I assume you meant semantics here. > with the > argument that you are trying to make Cython an optionally statically typed > Python. It's actually more complex than that. There are constraints that come from the compatibility to Py2.x and Py3, from language features in C and Python, from different C standards and compilers, ... Once all of that's been considered, the remaining design decisions are rarely black-or-white. This very thread is just one example. > So 'cdef int' gives the behaviour of a Python int, whilst 'cdef > float' gives the behaviour of a C float. This is wrong, please read what I posted already. >> This discussion is not about right or wrong, it's about good semantics >> that matches the expectations that users have of the language. > > What does we expect > > cdef float a > > to mean? Single or double precision? "float" is a C type. You do the math. Stefan From stefan_ml at behnel.de Mon Mar 16 19:13:24 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 16 Mar 2009 19:13:24 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <8257bb86d0e7b8f86cff4e60ecf4ba96.squirrel@webmail.uio.no> References: <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> <4db580fd0903161050m7e4baf48wa793bb43a85b3195@mail.gmail.com> <8257bb86d0e7b8f86cff4e60ecf4ba96.squirrel@webmail.uio.no> Message-ID: <49BE96C4.5030404@behnel.de> Sturla Molden wrote: >> It seems like the only places where this matters are with integers. > > 0.0/0.0 gives a NAN in C and raises an exception in Python. On the Python side, this has little to do with typing, as the same applies to integers. On the C side, typing will give you the required semantics now, no change required. >> As for floats, just alias py_float with double. > > So a cdef int s a py_int, but a cdef float is not a py_float? No, that's not what he wrote, and it's not how things work either. Maybe you should try to understand the existing semantics first, before taking the step to proposing improvements. Stefan From hoytak at cs.ubc.ca Mon Mar 16 19:15:42 2009 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Mon, 16 Mar 2009 11:15:42 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <8257bb86d0e7b8f86cff4e60ecf4ba96.squirrel@webmail.uio.no> References: <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> <4db580fd0903161050m7e4baf48wa793bb43a85b3195@mail.gmail.com> <8257bb86d0e7b8f86cff4e60ecf4ba96.squirrel@webmail.uio.no> Message-ID: <4db580fd0903161115g273c09b3s2ed60189e637b443@mail.gmail.com> > So a cdef int s a py_int, but a cdef float is not a py_float? No. Clarifying my above idea -- to the programmer, a cdef int would be a c int, a cdef float would be a c float, a py_int would be a python int, and a py_float would be a python float. In the generated c code, a cdef py_int could be a c long or c long long with python operations and a py_float would be a c double. To step back -- if the only way to drop into fast compiled code is to learn c semantics, it will cause problems for people who just want to cythonize their python code. If you use python semantics, it will cause problems for people who are c programmers using cython to write mixed python-c code. So doing this seems to give everyone something they like. I'm also thinking of proposed steps in the cython development such as automatically determining types (e.g. of a in 'a = 5'). Having a python integer would make that easier. --Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From sturla at molden.no Mon Mar 16 19:23:20 2009 From: sturla at molden.no (Sturla Molden) Date: Mon, 16 Mar 2009 19:23:20 +0100 (CET) Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49BE9447.4080107@behnel.de> References: <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> <608839d782db7ef5110e9c71b34f041e.squirrel@webmail.uio.no> <49BE8BB3.9070400@behnel.de> <49BE9447.4080107@behnel.de> Message-ID: > >> What does we expect >> >> cdef float a >> >> to mean? Single or double precision? > > "float" is a C type. You do the math. float is also a Python type. Shouldn't we users expect it to have the semantics of a Python float? That is precisely the same argument you are using for int. What about attribue lookup? cdef int a = 1234 print a.id Not keeping C and Python types apart only creates a big mess. Sturla Molden From sturla at molden.no Mon Mar 16 19:30:46 2009 From: sturla at molden.no (Sturla Molden) Date: Mon, 16 Mar 2009 19:30:46 +0100 (CET) Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <4db580fd0903161115g273c09b3s2ed60189e637b443@mail.gmail.com> References: <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> <4db580fd0903161050m7e4baf48wa793bb43a85b3195@mail.gmail.com> <8257bb86d0e7b8f86cff4e60ecf4ba96.squirrel@webmail.uio.no> <4db580fd0903161115g273c09b3s2ed60189e637b443@mail.gmail.com> Message-ID: <06a6d90b687537d63da82468a91f2c77.squirrel@webmail.uio.no> > No. Clarifying my above idea -- to the programmer, a cdef int would > be a c int, a cdef float would be a c float, a py_int would be a > python int, and a py_float would be a python float. Yes, that is nice. And now the semantics should be different, depending on the type. A numerical litteral could default to the Python type, unless a cast i made. > To step back -- if the only way to drop into fast compiled code is to > learn c semantics, it will cause problems for people who just want to > cythonize their python code. If you use python semantics, it will > cause problems for people who are c programmers using cython to write > mixed python-c code. So doing this seems to give everyone something > they like. Exactly. That is what I have been trying to communicate. Keep C and Python types apart. Don't ad Python semantics to a subset of the C types. Add a completely new set of types that directly corresponds to the Python equivalent. Sturla MOlden From sturla at molden.no Mon Mar 16 19:30:47 2009 From: sturla at molden.no (Sturla Molden) Date: Mon, 16 Mar 2009 19:30:47 +0100 (CET) Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <4db580fd0903161115g273c09b3s2ed60189e637b443@mail.gmail.com> References: <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> <4db580fd0903161050m7e4baf48wa793bb43a85b3195@mail.gmail.com> <8257bb86d0e7b8f86cff4e60ecf4ba96.squirrel@webmail.uio.no> <4db580fd0903161115g273c09b3s2ed60189e637b443@mail.gmail.com> Message-ID: <304684db9a8ae9971b3e9df41929d988.squirrel@webmail.uio.no> > No. Clarifying my above idea -- to the programmer, a cdef int would > be a c int, a cdef float would be a c float, a py_int would be a > python int, and a py_float would be a python float. Yes, that is nice. And now the semantics should be different, depending on the type. A numerical litteral could default to the Python type, unless a cast i made. > To step back -- if the only way to drop into fast compiled code is to > learn c semantics, it will cause problems for people who just want to > cythonize their python code. If you use python semantics, it will > cause problems for people who are c programmers using cython to write > mixed python-c code. So doing this seems to give everyone something > they like. Exactly. That is what I have been trying to communicate. Keep C and Python types apart. Don't ad Python semantics to a subset of the C types. Add a completely new set of types that directly corresponds to the Python equivalent. Sturla Molden From stefan_ml at behnel.de Mon Mar 16 19:42:03 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 16 Mar 2009 19:42:03 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> <608839d782db7ef5110e9c71b34f041e.squirrel@webmail.uio.no> <49BE8BB3.9070400@behnel.de> <49BE9447.4080107@behnel.de> Message-ID: <49BE9D7B.3030603@behnel.de> Sturla Molden wrote: >>> What does we expect >>> >>> cdef float a >>> >>> to mean? Single or double precision? >> "float" is a C type. You do the math. > > float is also a Python type. Correct. However, not in this context. > Shouldn't we users expect it to have the semantics of a Python float? Well, that's not how it works. A C float is not a Python float. That said, I wouldn't mind seeing patches that add the support for fake Python methods on the corresponding C types, so that you could do cdef float f = 0.5 print f.is_integer() But that would not change the semantics of a C float in C arithmetic, for example. It would just be syntactic sugar. > That is precisely the same argument you are using for int. No, it's not. "cdef float f" defines f as a C float with C semantics. You may want to read the distinction that I made in an earlier post regarding the explicit semantics of a variable and the semantics of an operator, which may or may not be explicit. > Not keeping C and Python types apart only creates a big mess. Not necessarily. It's all a matter of careful consideration and language design. Stefan From stefan_ml at behnel.de Mon Mar 16 19:54:18 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 16 Mar 2009 19:54:18 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <06a6d90b687537d63da82468a91f2c77.squirrel@webmail.uio.no> References: <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> <4db580fd0903161050m7e4baf48wa793bb43a85b3195@mail.gmail.com> <8257bb86d0e7b8f86cff4e60ecf4ba96.squirrel@webmail.uio.no> <4db580fd0903161115g273c09b3s2ed60189e637b443@mail.gmail.com> <06a6d90b687537d63da82468a91f2c77.squirrel@webmail.uio.no> Message-ID: <49BEA05A.8040205@behnel.de> Sturla Molden wrote: > A numerical litteral could default to the Python type, unless a cast i made. I doubt that people want that. Performance is one of the design goals of Cython. If you had to cast every literal in order to move it to C speed, that would be worse than programming in plain C. Stefan From stefan_ml at behnel.de Mon Mar 16 19:59:15 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 16 Mar 2009 19:59:15 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <4db580fd0903161050m7e4baf48wa793bb43a85b3195@mail.gmail.com> References: <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> <4db580fd0903161050m7e4baf48wa793bb43a85b3195@mail.gmail.com> Message-ID: <49BEA183.6060801@behnel.de> Hoyt Koepke wrote: >>> I kinda like the idea of having two types that are only really >>> different in corner cases like these, e.g. cint , clong, (maybe int, >>> long), and that these are an explicit request for c semantics in every >>> way. Other types like int (maybe py_int?) are a request for python >>> sematics, even if it means a slight speed penalty over c semantics. >>> Making this distinction explicit would seem to make everyone happy -- >>> both the python coders who want speed and the c coders who work in >>> python -- though it would probably mean more work for the cython >>> developers. It also seems like more of a long-term solution, as there >>> are other cases this would resolve (like overflows in c but not >>> python). >> Ooh, I definitely like this idea. > > Thanks. > > It seems like the only places where this matters are with integers. > Thus if cython had one type, say py_integer (to distinguish it from > PyIntObject*, like you mentioned), which in the C code would be a long > or long long, but the semantics in the generated code would always be > determined by python syntax. But you can't emulate the Python int semantics without also adding arbitrary size values. So, where would you draw the line? And how would you explain to users why this type has part X of Python's int semantics but not part Y, and why this type exists at all? I'm afraid of the complexity that this duplication of type names adds to the language. Plus, it does not solve the problem at hand, which is a problem with the semantics of an operator and not with the semantics of a type. Stefan From robertwb at math.washington.edu Mon Mar 16 20:15:52 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 16 Mar 2009 12:15:52 -0700 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: <49BD77BD.9050609@canterbury.ac.nz> References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> <43407.213.61.181.86.1236955563.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BD77BD.9050609@canterbury.ac.nz> Message-ID: On Mar 15, 2009, at 2:48 PM, Greg Ewing wrote: > Stefan Behnel wrote: >> Carl Witty wrote: >> >>> Hmm... I hope that eventually Cython will allow overriding >>> __new__, at >>> which point calling an extension type won't necessarily return a >>> value >>> of that type. >> >> I'm not sure. This should certainly be supported for Python >> classes, but >> for extension types? What would be the use case? > > If it is supported, it could perhaps be handled by allowing > the return type of the __new__ method to be declared explicitly, > defaulting to the class. E.g. given > > cdef class Foo: > def __new__(cls, ..): > ... > > then Foo() is assumed to return something of type Foo, but > given > > cdef class Blarg: > def object __new__(cls, ..): > ... > > then Blarg() could return any type of object. This sounds like a good syntax. - Robert From stefan_ml at behnel.de Mon Mar 16 20:32:41 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 16 Mar 2009 20:32:41 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49BE9D7B.3030603@behnel.de> References: <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> <608839d782db7ef5110e9c71b34f041e.squirrel@webmail.uio.no> <49BE8BB3.9070400@behnel.de> <49BE9447.4080107@behnel.de> <49BE9D7B.3030603@behnel.de> Message-ID: <49BEA959.4010204@behnel.de> Stefan Behnel wrote: > I wouldn't mind seeing patches that add the support for fake Python methods > on the corresponding C types, so that you could do > > cdef float f = 0.5 > print f.is_integer() This is now ticket 236: http://trac.cython.org/cython_trac/ticket/236 Stefan From sturla at molden.no Mon Mar 16 20:33:15 2009 From: sturla at molden.no (Sturla Molden) Date: Mon, 16 Mar 2009 20:33:15 +0100 (CET) Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49BEA183.6060801@behnel.de> References: <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> <4db580fd0903161050m7e4baf48wa793bb43a85b3195@mail.gmail.com> <49BEA183.6060801@behnel.de> Message-ID: <48a0102447f556dd8c10590520a60ec8.squirrel@webmail.uio.no> > But you can't emulate the Python int semantics without also adding > arbitrary size values. So, where would you draw the line? That's even better. Make sure the semantics neither complies with C nor Python. I'm back to using Pyrex. From robertwb at math.washington.edu Mon Mar 16 20:37:44 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 16 Mar 2009 12:37:44 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: On Mar 16, 2009, at 3:59 AM, Stefan Behnel wrote: > PS: Dag, I read your comment on "(2147483647 + 2147483647)" wrapping > around in current Cython. I never thought about that, simply since > I never > ran into it. However, when I write > > py_large_val = 2147483647 + 2147483647 > > it must never wrap around as it clearly requests Python semantics, and > when I write > > cdef uint64 c_large_val = 2147483647 + 2147483647 > > it must not wrap around either. I'm fine with a wrap-around in > > cdef int c_large_val = 2147483647 + 2147483647 > > though, because that's what the user a) requested or at least b) > must be > aware of when explicitly assigning types. So that's clearly > different from > a discussion on operator semantics. Very interesting--this only happens with constant literal expressions, right? In that case constant folding should take care of it. - Robert From dagss at student.matnat.uio.no Mon Mar 16 21:17:03 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 16 Mar 2009 21:17:03 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <48a0102447f556dd8c10590520a60ec8.squirrel@webmail.uio.no> References: <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> <4db580fd0903161050m7e4baf48wa793bb43a85b3195@mail.gmail.com> <49BEA183.6060801@behnel.de> <48a0102447f556dd8c10590520a60ec8.squirrel@webmail.uio.no> Message-ID: <49BEB3BF.1000701@student.matnat.uio.no> Sturla Molden wrote: >> But you can't emulate the Python int semantics without also adding >> arbitrary size values. So, where would you draw the line? > > That's even better. > > Make sure the semantics neither complies with C nor Python. > > I'm back to using Pyrex. Is this supposed to be a threat? If Pyrex fits your goals better, then great! Pyrex and Cython have different sets of goals (though partly overlapping). Your goal seems to be "a language where a poweruser who knows both C and Python well can go seamlessly between the two"; and then Pyrex seems to be a good match for you. It seems like you impose your own assumed goals for Cython as the only valid ones, and then, when you find something you don't like in what others think, instantly jump to the conclusion that there's somehow a flaw in their reasoning. Why not try something different, like asking: Are your goals the same? Is somebody else's position understandable by looking at the other persons goals and pre-assumptions? It's not a matter of somebody being stupid or not simply "seeing the obvious"; it's a matter of having different starting assumptions and different goals. Given that, different conclusions follow naturally. -- Dag Sverre From dagss at student.matnat.uio.no Mon Mar 16 21:24:48 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 16 Mar 2009 21:24:48 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49BEA183.6060801@behnel.de> References: <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> <4db580fd0903161050m7e4baf48wa793bb43a85b3195@mail.gmail.com> <49BEA183.6060801@behnel.de> Message-ID: <49BEB590.9050204@student.matnat.uio.no> Stefan Behnel wrote: > Hoyt Koepke wrote: >>>> I kinda like the idea of having two types that are only really >>>> different in corner cases like these, e.g. cint , clong, (maybe int, >>>> long), and that these are an explicit request for c semantics in every >>>> way. Other types like int (maybe py_int?) are a request for python >>>> sematics, even if it means a slight speed penalty over c semantics. >>>> Making this distinction explicit would seem to make everyone happy -- >>>> both the python coders who want speed and the c coders who work in >>>> python -- though it would probably mean more work for the cython >>>> developers. It also seems like more of a long-term solution, as there >>>> are other cases this would resolve (like overflows in c but not >>>> python). >>> Ooh, I definitely like this idea. >> Thanks. >> >> It seems like the only places where this matters are with integers. >> Thus if cython had one type, say py_integer (to distinguish it from >> PyIntObject*, like you mentioned), which in the C code would be a long >> or long long, but the semantics in the generated code would always be >> determined by python syntax. > > But you can't emulate the Python int semantics without also adding > arbitrary size values. So, where would you draw the line? And how would you > explain to users why this type has part X of Python's int semantics but not > part Y, and why this type exists at all? > > I'm afraid of the complexity that this duplication of type names adds to > the language. > > Plus, it does not solve the problem at hand, which is a problem with the > semantics of an operator and not with the semantics of a type. Well, strictly speaking, Python % etc. means "invoke __mod__"; and different classes can do different things (indeed NumPy uses C semantics; for speed I presume). So it's not true that types and operators are orthogonal in language design. To make this more concrete, we already have a syntax aimed more at Python users speeding up code: @cython.locals(x=cython.int) def f(x): return x % 3 So "cython.int" and friends could assume Python behaviour. BTW I'm not proposing that the @locals has anything to do with it, def f(cython.int x): return x % 3 should have Python semantics as well, it is just mixing coding styles and not what one is likely to see... Then one could add cython.cint to correspond directly to a C int. -- Dag Sverre From stefan_ml at behnel.de Mon Mar 16 21:10:59 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 16 Mar 2009 21:10:59 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49BEB253.3050704@behnel.de> Robert Bradshaw wrote: > On Mar 16, 2009, at 3:59 AM, Stefan Behnel wrote: > >> PS: Dag, I read your comment on "(2147483647 + 2147483647)" wrapping >> around in current Cython. I never thought about that, simply since >> I never >> ran into it. However, when I write >> >> py_large_val = 2147483647 + 2147483647 >> >> it must never wrap around as it clearly requests Python semantics, and >> when I write >> >> cdef uint64 c_large_val = 2147483647 + 2147483647 >> >> it must not wrap around either. I'm fine with a wrap-around in >> >> cdef int c_large_val = 2147483647 + 2147483647 >> >> though, because that's what the user a) requested or at least b) >> must be >> aware of when explicitly assigning types. So that's clearly >> different from >> a discussion on operator semantics. > > Very interesting--this only happens with constant literal > expressions, right? In that case constant folding should take care of > it. Well, yes, sort of. At least it does something, although not to the better. http://trac.cython.org/cython_trac/ticket/237 Stefan From stefan_ml at behnel.de Mon Mar 16 21:23:59 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 16 Mar 2009 21:23:59 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49BEB590.9050204@student.matnat.uio.no> References: <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> <4db580fd0903161050m7e4baf48wa793bb43a85b3195@mail.gmail.com> <49BEA183.6060801@behnel.de> <49BEB590.9050204@student.matnat.uio.no> Message-ID: <49BEB55F.3070505@behnel.de> Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Plus, it does not solve the problem at hand, which is a problem with the >> semantics of an operator and not with the semantics of a type. > > Well, strictly speaking, Python % etc. means "invoke __mod__"; and > different classes can do different things (indeed NumPy uses C > semantics; for speed I presume). So it's not true that types and > operators are orthogonal in language design. I didn't say that they are "orthogonal in language design". I just said that this specific problem "at hand" is a problem with operator semantics, not with type semantics. Here's one place where I tried to make that clear: http://permalink.gmane.org/gmane.comp.python.cython.devel/4907 Stefan From sturla at molden.no Mon Mar 16 21:27:00 2009 From: sturla at molden.no (Sturla Molden) Date: Mon, 16 Mar 2009 21:27:00 +0100 (CET) Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49BEB3BF.1000701@student.matnat.uio.no> References: <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> <4db580fd0903161050m7e4baf48wa793bb43a85b3195@mail.gmail.com> <49BEA183.6060801@behnel.de> <48a0102447f556dd8c10590520a60ec8.squirrel@webmail.uio.no> <49BEB3BF.1000701@student.matnat.uio.no> Message-ID: <25d710adde72a605845128b1acd27f00.squirrel@webmail.uio.no> > Is this supposed to be a threat? No, I don't make threats. It was just stating a conclusion. > Is somebody else's position understandable by > looking at the other persons goals and pre-assumptions? That would force me to conclude you are doing this just to be mean. But in this case I think Hanlon's razor indicates a simpler explanation. I'll just quote Albert Einstein: "Only two things are infinite, the universe and human stupidity, and I'm not sure about the universe." :) Sturla Molden From robert.kern at gmail.com Mon Mar 16 21:29:49 2009 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 16 Mar 2009 15:29:49 -0500 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49BEB55F.3070505@behnel.de> References: <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> <4db580fd0903161050m7e4baf48wa793bb43a85b3195@mail.gmail.com> <49BEA183.6060801@behnel.de> <49BEB590.9050204@student.matnat.uio.no> <49BEB55F.3070505@behnel.de> Message-ID: On 2009-03-16 15:23, Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> Stefan Behnel wrote: >>> Plus, it does not solve the problem at hand, which is a problem with the >>> semantics of an operator and not with the semantics of a type. >> Well, strictly speaking, Python % etc. means "invoke __mod__"; and >> different classes can do different things (indeed NumPy uses C >> semantics; for speed I presume). So it's not true that types and >> operators are orthogonal in language design. > > I didn't say that they are "orthogonal in language design". I just said > that this specific problem "at hand" is a problem with operator semantics, > not with type semantics. Here's one place where I tried to make that clear: > > http://permalink.gmane.org/gmane.comp.python.cython.devel/4907 I think Dag's point is that "operator semantics" is not something that you can distinguish from "type semantics". You *are* claiming that the two are orthogonal if you think that the issue at hand can be just one and not the other. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robertwb at math.washington.edu Mon Mar 16 21:31:41 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 16 Mar 2009 13:31:41 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <25d710adde72a605845128b1acd27f00.squirrel@webmail.uio.no> References: <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> <4db580fd0903161050m7e4baf48wa793bb43a85b3195@mail.gmail.com> <49BEA183.6060801@behnel.de> <48a0102447f556dd8c10590520a60ec8.squirrel@webmail.uio.no> <49BEB3BF.1000701@student.matnat.uio.no> <25d710adde72a605845128b1acd27f00.squirrel@webmail.uio.no> Message-ID: <3C4E03E8-E574-4F96-9A94-DF0657FFAD7D@math.washington.edu> On Mar 16, 2009, at 1:27 PM, Sturla Molden wrote: >> Is this supposed to be a threat? > > No, I don't make threats. It was just stating a conclusion. Greg hasn't spoken up on this thread yet (I'd be really curious to see what he says) but I would hazard a guess that Pyrex will lean to C types being raw C types, C semantics and all. >> Is somebody else's position understandable by >> looking at the other persons goals and pre-assumptions? > > That would force me to conclude you are doing this just to be mean. > But in this case I think Hanlon's razor indicates a simpler > explanation. I'll > just quote Albert Einstein: "Only two things are infinite, the > universe > and human stupidity, and I'm not sure about the universe." I don't think any meanness was intended, and hopefully we can keep it that way. This thread is long and controversial enough as it is :). - Robert From stefan_ml at behnel.de Mon Mar 16 21:53:28 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 16 Mar 2009 21:53:28 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> <4db580fd0903161050m7e4baf48wa793bb43a85b3195@mail.gmail.com> <49BEA183.6060801@behnel.de> <49BEB590.9050204@student.matnat.uio.no> <49BEB55F.3070505@behnel.de> Message-ID: <49BEBC48.5030801@behnel.de> Robert Kern wrote: > On 2009-03-16 15:23, Stefan Behnel wrote: >> Dag Sverre Seljebotn wrote: >>> Stefan Behnel wrote: >>>> Plus, it does not solve the problem at hand, which is a problem with the >>>> semantics of an operator and not with the semantics of a type. >>> Well, strictly speaking, Python % etc. means "invoke __mod__"; and >>> different classes can do different things (indeed NumPy uses C >>> semantics; for speed I presume). So it's not true that types and >>> operators are orthogonal in language design. >> I didn't say that they are "orthogonal in language design". I just said >> that this specific problem "at hand" is a problem with operator semantics, >> not with type semantics. Here's one place where I tried to make that clear: >> >> http://permalink.gmane.org/gmane.comp.python.cython.devel/4907 > > I think Dag's point is that "operator semantics" is not something that you can > distinguish from "type semantics". You *are* claiming that the two are > orthogonal if you think that the issue at hand can be just one and not the other. I agree that part of the problem is that the type semantics are not clear, so this is not orthogonal, even not in this case. What I was trying to bring across was that solving this by extending the type system is (IMHO) the wrong place to start with. This is a small problem that involves the semantics of one, maybe two, operators that interact with integer types. Adding a new integer type will require its definition for all operators and all kinds of interactions with other types, which will let the type system explode just to solve two tiny ambiguities. We will end up explaining to users that they should normally use type X, except in case Y and Z, where they must be aware that P applies (although most people will not care about P, but will have to learn about type X anyway to be able to understand other people's programs). This is a horrible violation of the OWTDI principle. The simple and easy to explain solution is to give the operators the however-determined well-defined Cython default semantics, and to provide a replacement for power users who require a specific feature for performance reasons. Because performance is really the whole issue here. Users can implement any kind of semantics at the one place in their code where they actually really need it, but using an operator leads to the shortest and presumably fastest code. Stefan From robertwb at math.washington.edu Mon Mar 16 22:07:38 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 16 Mar 2009 14:07:38 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49BE681F.8080005@student.matnat.uio.no> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE681F.8080005@student.matnat.uio.no> Message-ID: <2F1DAAEC-6D59-46A8-A40E-15E0372805D6@math.washington.edu> On Mar 16, 2009, at 7:54 AM, Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Dag Sverre Seljebotn wrote: >> >>> Both communities seems to mirror the discussion we've already had >>> here >>> very closely. There's one group who sees Cython as "fast Python" who >>> want Python semantics and another who sees it as "mixed Python >>> and C" >>> who want the C semantics. >>> >> >> I was also rather unsure at the beginning, but when I think of the >> consequences for totally Python-looking Cython source code like >> >> [ y % 5 for y in some_iterable ] >> >> where you have to a) look up the type of "y" *and* b) know the >> associated >> semantics of the % operator, before you even know what the numeric >> range >> of the result will be - I think that having C semantics here is >> just too >> dangerous to be the default. We should not forget that the syntax >> we write >> in is Python, so it's a sure trap to assign C semantics to obvious >> looking >> things like the above. >> >> My clear vote is for Python-semantics on "x%y" After reading all the discussion on all the lists (thanks Dag for asking on NumPy as well), my vote is also for the default semantics for "x%y" to be Python semantics. This is primarily to follow the path of least surprise, especially for Python users that don't really know C just trying to make (existing) Python code faster and/or call C libraries (which I view as our main audience). In particular, I think C semantics would be much more surprising to those expecting Python semantics than Python semantics would be to someone expecting C semantics. I have a hunch that the latter group will be more capable of creating a workaround. >> and for a helper function >> "cython.cmod(x,y)" that boils down to a straight and fast C mod >> operator >> without value fixing. > > Agreed...but like Robert I'm worried about the usability of > requiring to > use cython.cmod and cython.cdiv (the latter being kist as important > here); especially as one would typically use them in lots of > situations > (where one is only dividing positive integers anyway) to avoid the > speed > penalty (the 20% penalty reported by William is a bit much!). > > So I'm myself leaning towards new operators (in addition to > cython.cmod/cdiv) for the C semantics (e.g. /- and %-, or ^/ and ^ > %, or > similar -- as long as we stay away from /~ etc which is proposed in > some > PEPs). Yes, this is ugly! -- but I see them as less ugly than the > alternatives. I'm less sure about how to expose the C operations, but just accepting a 20-30% slowdown is not acceptable. We should provide cython.cmod/cdiv, but I feel strongly we should provide an infix option as well. I'm wary of introducing new operators, but the %-, /-, and //- seems like the best options proposed. It is a bit deceptive though, as we would not force the truncating rounding mode for negative numbers, just pass it on to the C compiler (are there any that don't make this choice?). Another option is to do something very like c(%) or c_op(%), which is currently unused invalid syntax. This is very explicit and could generalized to other situations, but it's not as readable and extending the parser should be done with caution. The option I'm leaning towards (which is probably the most controversial) is having a pragma like "cython.cdivision" that controls what the semantics are. This really isn't any more invasive than the from __future__ import division, and we can do it on a more local scale. One reason I like this option are that usually when one wants to use it in one place, one wants to use it consistently throughout a whole block of code, and introducing a new operator is hard to remember to use everywhere. Also, it makes it easy to convert existing code (e.g. the Sage library) and appease those who really want C semantics. I'm not a fan of introducing a new set of types (cint, cfloat, ...) and I think this will make Cython more confusing, especially as they a priori have no indication they have anything to do with division. - Robert From dalcinl at gmail.com Mon Mar 16 22:21:16 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 16 Mar 2009 18:21:16 -0300 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <2F1DAAEC-6D59-46A8-A40E-15E0372805D6@math.washington.edu> References: <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE681F.8080005@student.matnat.uio.no> <2F1DAAEC-6D59-46A8-A40E-15E0372805D6@math.washington.edu> Message-ID: On Mon, Mar 16, 2009 at 6:07 PM, Robert Bradshaw wrote: > On Mar 16, 2009, at 7:54 AM, Dag Sverre Seljebotn wrote: > > After reading all the discussion on all the lists (thanks Dag for > asking on NumPy as well), my vote is also for the default semantics > for "x%y" to be Python semantics. Well, at this point in the thread I feel like my opinion would be irrelevant, but I'm still -1. > > I'm less sure about how to expose the C operations, but just > accepting a 20-30% slowdown is not acceptable. We should provide > cython.cmod/cdiv, but I feel strongly we should provide an infix > option as well. If Python semantics is chosen, I would also ask for a compiler directive (perhaps comand line disabled) to select C semantics. In all this thread, backward compatibility with Cython itself and Pyrex was not taken into account. A compiler directive will ease a lot user code updates for the new semantics. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From greg.ewing at canterbury.ac.nz Mon Mar 16 22:58:14 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 17 Mar 2009 09:58:14 +1200 Subject: [Cython] cdef class understanding In-Reply-To: <49BE4F49.8090005@gmail.com> References: <49BCF9B3.1030601@gmail.com> <49BE36A3.7000500@behnel.de> <49BE4F49.8090005@gmail.com> Message-ID: <49BECB76.1090505@canterbury.ac.nz> Martin Gysel wrote: > now I'm asking myself what's the benefit to declare a c callback in the > class There isn't any. Just make it a top-level C function. -- Greg From greg.ewing at canterbury.ac.nz Tue Mar 17 00:15:26 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 17 Mar 2009 11:15:26 +1200 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49BE6AC1.3000303@molden.no> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> Message-ID: <49BEDD8E.1000007@canterbury.ac.nz> Sturla Molden wrote: > As I see it, > > cdef int a, b, c > > is a request to "step into C" when using these variables. How about this: 1) By default, all operators have C semantics when applied to C data types. 2) To get Python semantics for / and %, put this at the top of the module: from __python__ import idiv, imod 3) You are allowed to feed a ".py" file directly to Cython, in which case it is compiled as though it begins with an implicit from __python__ import * This mechanism could be extended to handle any other areas where there is conflicting desire for C and Python compatibility. -- Greg From greg.ewing at canterbury.ac.nz Tue Mar 17 00:22:18 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 17 Mar 2009 11:22:18 +1200 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49BE72E1.608@behnel.de> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <49BE72E1.608@behnel.de> Message-ID: <49BEDF2A.3020201@canterbury.ac.nz> Stefan Behnel wrote: > Here, it > is simply not true that the user "must" expect the semantics to change when > typing the variable. There's a precedent for this in C already: the / operator represents different operations when applied to ints vs. floats. So while it may not be true that a user "must" expect an operator to change semantics depending on the operand types, they can't expect it *not* to change, either. -- Greg From greg.ewing at canterbury.ac.nz Tue Mar 17 00:58:28 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 17 Mar 2009 11:58:28 +1200 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <3C4E03E8-E574-4F96-9A94-DF0657FFAD7D@math.washington.edu> References: <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> <4db580fd0903161050m7e4baf48wa793bb43a85b3195@mail.gmail.com> <49BEA183.6060801@behnel.de> <48a0102447f556dd8c10590520a60ec8.squirrel@webmail.uio.no> <49BEB3BF.1000701@student.matnat.uio.no> <25d710adde72a605845128b1acd27f00.squirrel@webmail.uio.no> <3C4E03E8-E574-4F96-9A94-DF0657FFAD7D@math.washington.edu> Message-ID: <49BEE7A4.1010906@canterbury.ac.nz> Robert Bradshaw wrote: > Greg hasn't spoken up on this thread yet (I'd be really curious to > see what he says) but I would hazard a guess that Pyrex will lean to > C types being raw C types, C semantics and all. I haven't thought it through in detail yet, but yes, probably. There's already a // operator I can use to get Python-like int division. I may add a %% operator to do the same for mod. To get efficient operations when you don't care about negative numbers, you can use unsigned types with %% and //. -- Greg From robertwb at math.washington.edu Tue Mar 17 01:02:58 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 16 Mar 2009 17:02:58 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE681F.8080005@student.matnat.uio.no> <2F1DAAEC-6D59-46A8-A40E-15E0372805D6@math.washington.edu> Message-ID: <4D9C0B11-D039-44B0-86EA-2127D678D615@math.washington.edu> On Mar 16, 2009, at 2:21 PM, Lisandro Dalcin wrote: > On Mon, Mar 16, 2009 at 6:07 PM, Robert Bradshaw > wrote: >> On Mar 16, 2009, at 7:54 AM, Dag Sverre Seljebotn wrote: >> >> After reading all the discussion on all the lists (thanks Dag for >> asking on NumPy as well), my vote is also for the default semantics >> for "x%y" to be Python semantics. > > Well, at this point in the thread I feel like my opinion would be > irrelevant, but I'm still -1. Don't worry, your input is valued a lot higher on this list than most :) That reminds me, I was adding your name to http://cython.org/ and I remembered that you had some accents in your last name, but couldn't remember which way they went for sure (or how to get them in HTML). Is it Dalcín? - Robert From robertwb at math.washington.edu Tue Mar 17 01:13:09 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 16 Mar 2009 17:13:09 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49BEDD8E.1000007@canterbury.ac.nz> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <49BEDD8E.1000007@canterbury.ac.nz> Message-ID: <501A9CFB-978E-4EB0-A093-7A495E6F750C@math.washington.edu> On Mar 16, 2009, at 4:15 PM, Greg Ewing wrote: > Sturla Molden wrote: > >> As I see it, >> >> cdef int a, b, c >> >> is a request to "step into C" when using these variables. > > How about this: > > 1) By default, all operators have C semantics when > applied to C data types. > > 2) To get Python semantics for / and %, put this at > the top of the module: > > from __python__ import idiv, imod > > 3) You are allowed to feed a ".py" file directly to > Cython, in which case it is compiled as though it > begins with an implicit > > from __python__ import * > > This mechanism could be extended to handle any other > areas where there is conflicting desire for C and > Python compatibility. Interestingly enough, this was almost exactly my original proposal too from a week ago (the only difference being from __python__ import ..." being much cleaner than a compiler flag, and I agree an in-file directive is better). - Robert From greg.ewing at canterbury.ac.nz Tue Mar 17 02:08:33 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 17 Mar 2009 13:08:33 +1200 Subject: [Cython] cdef class understanding, 2nd In-Reply-To: <49BE007A.4050004@gmail.com> References: <49BE007A.4050004@gmail.com> Message-ID: <49BEF811.8070407@canterbury.ac.nz> Martin Gysel wrote: > cdef class Cobj: > cdef a_c_struct_t *loop > > def __init__(self): > self.loop = c_lib_call() > c_lib_call_reg_callback(self.callback, NULL) ^^^^^^^^^^^^^ I don't know what Cython does with this, but it's an error in Pyrex -- you can't obtain a bound C method, you can only call it. -- Greg From hoytak at cs.ubc.ca Tue Mar 17 04:15:02 2009 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Mon, 16 Mar 2009 20:15:02 -0700 Subject: [Cython] patch to make lexicon pickle loading more robust Message-ID: <4db580fd0903162015o2fac5a29t8bafbb8a47dc2bea@mail.gmail.com> Hello, When upgrading the last cython, it seems that loading the lexicon pickle failed. I think I had an incompatible version around somewhere. Anyway, the error I got was this: Compiling module Cython.Plex.Scanners ... ERROR: ('__init__() takes at least 2 arguments (1 given)', , ()) Extension module compilation failed, using plain Python implementation running install The error occurred while unpickling the lexicon. Under this condition, it seemed like a good idea to just regenerate the pickle file if there's an error loading it. This is what the following patch does: # HG changeset patch # User Hoyt Koepke hoytak at gmail.com # Date 1237259414 25200 # Node ID 2ec368843599a624c88fa76735824bb7c1c3ede4 # Parent 137b3d0a0c601f810e36df7530734848cb9bb425 Made lexicon unpickling more robust; on error it regenerates it. diff -r 137b3d0a0c60 -r 2ec368843599 Cython/Compiler/Scanning.py --- a/Cython/Compiler/Scanning.py Mon Mar 16 21:33:46 2009 +0100 +++ b/Cython/Compiler/Scanning.py Mon Mar 16 20:10:14 2009 -0700 @@ -102,7 +103,14 @@ if notify_lexicon_unpickling: t0 = time() print("Unpickling lexicon...") - lexicon = pickle.load(f) + + try: + lexicon = pickle.load(f) + except Exception, e: + print "WARNING: Exception while loading lexicon pickle, regenerating" + print e + lexicon = None + f.close() if notify_lexicon_unpickling: t1 = time() -- Hoyt +++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ -------------- next part -------------- A non-text attachment was scrubbed... Name: robust_lexicon_unpickling.diff Type: text/x-patch Size: 931 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090316/81b362da/attachment.bin From kwmsmith at gmail.com Tue Mar 17 06:31:36 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Tue, 17 Mar 2009 00:31:36 -0500 Subject: [Cython] [Fwd: Re: Beginner-friendly tickets] In-Reply-To: <49BE1093.7080505@student.matnat.uio.no> References: <49BE1093.7080505@student.matnat.uio.no> Message-ID: On Mon, Mar 16, 2009 at 3:40 AM, Dag Sverre Seljebotn wrote: [snip] > > Great that you're making progress. > > Yes, there is something I'd rather tell you about this :-) The transform > works by recursive descent, and when a node type is intercepted, one > needs to actively visit the children as well. > > This is done by self.visitchildren(node); see in visit_FuncDefNode for > an example. Since visitchildren is never called in CVarDefNode, the > children of CVarDefNode are never visited. > > Also, the node will be replaced with whatever it returns. > visit_CVarDefNode returns None so that the entire declaration is removed > from the tree, as it is no longer needed (which is why one isn't > analysing the children as well). > > (Depending on what you need this for, you could be able to extract the > information from the attributes of the enclosing FuncDefNode; I think > there's a Scope there somewhere which you can inspect the entries of.) Thanks again for the guidance. I have a patch for review -- it involves adding self.visitchildren(node) in some methods and adding a visit_CNameDeclaratorNode() method along with the discussed visit_NameNode() method. It also adds a declareafteruse_T158.pyx file to tests/errors. Let me know if this along the lines of what you were thinking & how it can be improved. (Normally I'd attach this patch to the trac ticket but I thought some feedback would be good.) Kurt -------------- next part -------------- A non-text attachment was scrubbed... Name: ticket-158.patch Type: text/x-diff Size: 4789 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090317/843f947f/attachment.bin From robertwb at math.washington.edu Tue Mar 17 06:46:03 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 16 Mar 2009 22:46:03 -0700 Subject: [Cython] patch to make lexicon pickle loading more robust In-Reply-To: <4db580fd0903162015o2fac5a29t8bafbb8a47dc2bea@mail.gmail.com> References: <4db580fd0903162015o2fac5a29t8bafbb8a47dc2bea@mail.gmail.com> Message-ID: <8B415CD2-E3B7-4682-ADE3-519ED8B861FB@math.washington.edu> I noticed this too. The patch looks good (on sight, haven't applied it yet). http://trac.cython.org/cython_trac/ticket/239 On Mar 16, 2009, at 8:15 PM, Hoyt Koepke wrote: > Hello, > > When upgrading the last cython, it seems that loading the lexicon > pickle failed. I think I had an incompatible version around > somewhere. Anyway, the error I got was this: > > Compiling module Cython.Plex.Scanners ... > ERROR: ('__init__() takes at least 2 arguments (1 given)', 'Cython.Plex.Lexicons.Lexicon'>, ()) > Extension module compilation failed, using plain Python implementation > running install > > The error occurred while unpickling the lexicon. Under this condition, > it seemed like a good idea to just regenerate the pickle file if > there's an error loading it. This is what the following patch does: > > # HG changeset patch > # User Hoyt Koepke hoytak at gmail.com > # Date 1237259414 25200 > # Node ID 2ec368843599a624c88fa76735824bb7c1c3ede4 > # Parent 137b3d0a0c601f810e36df7530734848cb9bb425 > Made lexicon unpickling more robust; on error it regenerates it. > > diff -r 137b3d0a0c60 -r 2ec368843599 Cython/Compiler/Scanning.py > --- a/Cython/Compiler/Scanning.py Mon Mar 16 21:33:46 2009 +0100 > +++ b/Cython/Compiler/Scanning.py Mon Mar 16 20:10:14 2009 -0700 > @@ -102,7 +103,14 @@ > if notify_lexicon_unpickling: > t0 = time() > print("Unpickling lexicon...") > - lexicon = pickle.load(f) > + > + try: > + lexicon = pickle.load(f) > + except Exception, e: > + print "WARNING: Exception while loading lexicon pickle, > regenerating" > + print e > + lexicon = None > + > f.close() > if notify_lexicon_unpickling: > t1 = time() > > -- Hoyt > > > > +++++++++++++++++++++++++++++++++++++++++++++++ > + Hoyt Koepke > + University of Washington Department of Statistics > + http://www.stat.washington.edu/~hoytak/ > + hoytak at gmail.com > +++++++++++++++++++++++++++++++++++++++++ > +_____________________________________ > __________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From dagss at student.matnat.uio.no Tue Mar 17 07:37:20 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 17 Mar 2009 07:37:20 +0100 Subject: [Cython] #158 In-Reply-To: References: <49BE1093.7080505@student.matnat.uio.no> Message-ID: <49BF4520.6040100@student.matnat.uio.no> Kurt Smith wrote: > On Mon, Mar 16, 2009 at 3:40 AM, Dag Sverre Seljebotn > wrote: > [snip] >> Great that you're making progress. >> >> Yes, there is something I'd rather tell you about this :-) The transform >> works by recursive descent, and when a node type is intercepted, one >> needs to actively visit the children as well. >> >> This is done by self.visitchildren(node); see in visit_FuncDefNode for >> an example. Since visitchildren is never called in CVarDefNode, the >> children of CVarDefNode are never visited. >> >> Also, the node will be replaced with whatever it returns. >> visit_CVarDefNode returns None so that the entire declaration is removed >> from the tree, as it is no longer needed (which is why one isn't >> analysing the children as well). >> >> (Depending on what you need this for, you could be able to extract the >> information from the attributes of the enclosing FuncDefNode; I think >> there's a Scope there somewhere which you can inspect the entries of.) > > Thanks again for the guidance. > > I have a patch for review -- it involves adding > self.visitchildren(node) in some methods and adding a > visit_CNameDeclaratorNode() method along with the discussed > visit_NameNode() method. It also adds a declareafteruse_T158.pyx file > to tests/errors. > > Let me know if this along the lines of what you were thinking & how it > can be improved. Positive review -- I don't think it can be improved :-) Thanks! I've committed it to cython-devel. (I made two small changes to the error message) -- Dag Sverre From stefan_ml at behnel.de Tue Mar 17 07:25:49 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 17 Mar 2009 07:25:49 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49BEDD8E.1000007@canterbury.ac.nz> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <49BEDD8E.1000007@canterbury.ac.nz> Message-ID: <49BF426D.7010005@behnel.de> Greg Ewing wrote: > 2) To get Python semantics for / and %, put this at > the top of the module: > > from __python__ import idiv, imod > > 3) You are allowed to feed a ".py" file directly to > Cython, in which case it is compiled as though it > begins with an implicit > > from __python__ import * I thought about 3), too. If provided together with 2) it might be ok. Otherwise, it would mean that we break programs simply by letting users rename the file (e.g. because they want to start using "cdef" syntax). Such differences between .py file compilation and .pyx file compilation should be documented somewhere. Stefan From stefan_ml at behnel.de Tue Mar 17 08:05:07 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 17 Mar 2009 08:05:07 +0100 Subject: [Cython] [Fwd: Re: Beginner-friendly tickets] In-Reply-To: <49BE1093.7080505@student.matnat.uio.no> References: <49BE1093.7080505@student.matnat.uio.no> Message-ID: <49BF4BA3.3080302@behnel.de> Dag Sverre Seljebotn wrote: > Ahh; this was sent to my private address as well and so cython-dev in > its wisdom decided not to send me a copy from the mailing list. > Forwarding my response to the mailing list as well then. Not sure I mentioned it before, but that's one of the reasons not to break "reply/reply-all" in a mailing list. *wink* Stefan From dagss at student.matnat.uio.no Tue Mar 17 08:59:45 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 17 Mar 2009 08:59:45 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49BF426D.7010005@behnel.de> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <49BEDD8E.1000007@canterbury.ac.nz> <49BF426D.7010005@behnel.de> Message-ID: <49BF5871.4040404@student.matnat.uio.no> Stefan Behnel wrote: > Greg Ewing wrote: >> 2) To get Python semantics for / and %, put this at >> the top of the module: >> >> from __python__ import idiv, imod >> >> 3) You are allowed to feed a ".py" file directly to >> Cython, in which case it is compiled as though it >> begins with an implicit >> >> from __python__ import * > > I thought about 3), too. If provided together with 2) it might be ok. > Otherwise, it would mean that we break programs simply by letting users > rename the file (e.g. because they want to start using "cdef" syntax). Such > differences between .py file compilation and .pyx file compilation should > be documented somewhere. It means "one more thing to consider" though. E.g. to know what "//" does one needs to know not only the types it is operating on and which imports are used, but possibly whether one is compiling a .pyx or .py. I'm -1 on this, and +1 on always defining .py compilation in Cython as a strict subset of .pyx compilation (exception: Making e.g. "cdef" a valid variable name in .py files). Much easier to explain and remember. -- Dag Sverre From stefan_ml at behnel.de Tue Mar 17 09:48:25 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 17 Mar 2009 09:48:25 +0100 (CET) Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49BF5871.4040404@student.matnat.uio.no> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <49BEDD8E.1000007@canterbury.ac.nz> <49BF426D.7010005@behnel.de> <49BF5871.4040404@student.matnat.uio.no> Message-ID: <50876.213.61.181.86.1237279705.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Greg Ewing wrote: >>> 2) To get Python semantics for / and %, put this at >>> the top of the module: >>> >>> from __python__ import idiv, imod >>> >>> 3) You are allowed to feed a ".py" file directly to >>> Cython, in which case it is compiled as though it >>> begins with an implicit >>> >>> from __python__ import * >> >> I thought about 3), too. If provided together with 2) it might be ok. >> Otherwise, it would mean that we break programs simply by letting users >> rename the file (e.g. because they want to start using "cdef" syntax). >> Such >> differences between .py file compilation and .pyx file compilation >> should be documented somewhere. > > It means "one more thing to consider" though. E.g. to know what "//" > does one needs to know not only the types it is operating on and which > imports are used, but possibly whether one is compiling a .pyx or .py. > > I'm -1 on this, and +1 on always defining .py compilation in Cython as a > strict subset of .pyx compilation (exception: Making e.g. "cdef" a valid > variable name in .py files). Much easier to explain and remember. My thoughts, too, and a reason to stick as close to Python semantics as it seems sensible. My only concern by now is that we might break existing code that already depends on C semantics here. (although one might argue that there is only a constant amount of existing code and an infinite amount of unwritten code ;) I think this topic merits a CEP by now. There are good arguments on both sides, and the decision should be documented. Stefan From dagss at student.matnat.uio.no Tue Mar 17 10:15:53 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 17 Mar 2009 10:15:53 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <2F1DAAEC-6D59-46A8-A40E-15E0372805D6@math.washington.edu> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE681F.8080005@student.matnat.uio.no> <2F1DAAEC-6D59-46A8-A40E-15E0372805D6@math.washington.edu> Message-ID: <49BF6A49.6080506@student.matnat.uio.no> Robert Bradshaw wrote: > I'm less sure about how to expose the C operations, but just > accepting a 20-30% slowdown is not acceptable. We should provide > cython.cmod/cdiv, but I feel strongly we should provide an infix > option as well. I'm wary of introducing new operators, but the > %-, /-, and //- seems like the best options proposed. It is a bit > deceptive though, as we would not force the truncating rounding mode > for negative numbers, just pass it on to the C compiler (are there > any that don't make this choice?). Another option is to do something > very like c(%) or c_op(%), which is currently unused invalid syntax. How about this: cdef int a = -5, b = 3 print a.cmod(b), a.cdiv(b) -- Dag Sverre From dagss at student.matnat.uio.no Tue Mar 17 10:23:04 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 17 Mar 2009 10:23:04 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <50876.213.61.181.86.1237279705.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <49BEDD8E.1000007@canterbury.ac.nz> <49BF426D.7010005@behnel.de> <49BF5871.4040404@student.matnat.uio.no> <50876.213.61.181.86.1237279705.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49BF6BF8.1040906@student.matnat.uio.no> Stefan Behnel wrote: > My only concern by now is that we might break existing code that already > depends on C semantics here. (although one might argue that there is only > a constant amount of existing code and an infinite amount of unwritten > code ;) We can always print out warnings for a couple of releases before switching. -- Dag Sverre From stefan_ml at behnel.de Tue Mar 17 10:56:37 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 17 Mar 2009 10:56:37 +0100 (CET) Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49BF6A49.6080506@student.matnat.uio.no> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE681F.8080005@student.matnat.uio.no> <2F1DAAEC-6D59-46A8-A40E-15E0372805D6@math.washington.edu> <49BF6A49.6080506@student.matnat.uio.no> Message-ID: <39362.213.61.181.86.1237283797.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> I'm less sure about how to expose the C operations, but just >> accepting a 20-30% slowdown is not acceptable. We should provide >> cython.cmod/cdiv, but I feel strongly we should provide an infix >> option as well. I'm wary of introducing new operators, but the >> %-, /-, and //- seems like the best options proposed. It is a bit >> deceptive though, as we would not force the truncating rounding mode >> for negative numbers, just pass it on to the C compiler (are there >> any that don't make this choice?). Another option is to do something >> very like c(%) or c_op(%), which is currently unused invalid syntax. > > How about this: > > cdef int a = -5, b = 3 > > print a.cmod(b), a.cdiv(b) Ah, that was easy, now that there's a ticket for these things already... ;) http://trac.cython.org/cython_trac/ticket/236 But, sure, why not. Looks like a simple enough syntax to me, and it's pretty close to an infix operator. Stefan From greg.ewing at canterbury.ac.nz Tue Mar 17 11:45:56 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 17 Mar 2009 22:45:56 +1200 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49BF426D.7010005@behnel.de> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <49BEDD8E.1000007@canterbury.ac.nz> <49BF426D.7010005@behnel.de> Message-ID: <49BF7F64.3080000@canterbury.ac.nz> Stefan Behnel wrote: > I thought about 3), too. If provided together with 2) it might be ok. > Otherwise, it would mean that we break programs simply by letting users > rename the file By changing the extension, you're declaring it to be in a different language. I don't think it's very surprising if things break when you do that without changing the contents of the file to match. -- Greg From robertwb at math.washington.edu Tue Mar 17 18:56:09 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 17 Mar 2009 10:56:09 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <50876.213.61.181.86.1237279705.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <49BEDD8E.1000007@canterbury.ac.nz> <49BF426D.7010005@behnel.de> <49BF5871.4040404@student.matnat.uio.no> <50876.213.61.181.86.1237279705.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <232E962E-D2AE-4604-B3D6-5D4912D0D870@math.washington.edu> On Mar 17, 2009, at 1:48 AM, Stefan Behnel wrote: > I think this topic merits a CEP by now. There are good arguments on > both > sides, and the decision should be documented. +1 Any volunteers? From psp.reachable at gmail.com Tue Mar 17 20:16:42 2009 From: psp.reachable at gmail.com (Prajwal Suhas P) Date: Wed, 18 Mar 2009 00:46:42 +0530 Subject: [Cython] [Fwd: Re: Beginner-friendly tickets] In-Reply-To: <49BF4BA3.3080302@behnel.de> References: <49BE1093.7080505@student.matnat.uio.no> <49BF4BA3.3080302@behnel.de> Message-ID: This is the patch for ticket #203. I came out with this, but i did not understand the need for freeze_endpoints.I still haven't supplied the complete patch with all the tests.This is for confirming whether i'm in the right direction. diff -r e2365a6d00b8 Cython/Compiler/Nodes.py --- a/Cython/Compiler/Nodes.py Tue Mar 17 07:32:51 2009 +0100 +++ b/Cython/Compiler/Nodes.py Wed Mar 18 00:20:19 2009 +0530 @@ -3971,11 +3971,13 @@ code.putln("if (%s%s %s %s) {" % ( self.bound1.result(), offset, self.relation2, self.bound2.result() )) + temp_for_loop_bound = code.funcstate.allocate_temp(PyrexTypes.c_char_ptr_type, manage_ref=False) + code.putln("%s = %s;" % (temp_for_loop_bound, self.bound2.result())) code.putln( "for (%s = %s%s; %s %s %s; %s%s) {" % ( loopvar_name, self.bound1.result(), offset, - loopvar_name, self.relation2, self.bound2.result(), + loopvar_name, self.relation2, temp_for_loop_bound, loopvar_name, incop)) if self.py_loopvar_node: self.py_loopvar_node.generate_evaluation_code(code) @@ -3989,6 +3991,7 @@ code.putln("} /* end if */") else: code.putln("}") + code.funcstate.release_temp(temp_for_loop_bound) break_label = code.break_label code.set_loop_labels(old_loop_labels) if self.else_clause: -- Regards, Prajwal S. "There is little difference in people,but this little difference makes a big difference.The little difference is attitude,the big difference is whether its positive or negative. " -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090318/456c031e/attachment.htm From dagss at student.matnat.uio.no Tue Mar 17 20:38:42 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 17 Mar 2009 20:38:42 +0100 Subject: [Cython] [Fwd: Re: Beginner-friendly tickets] In-Reply-To: References: <49BE1093.7080505@student.matnat.uio.no> <49BF4BA3.3080302@behnel.de> Message-ID: <49BFFC42.5090800@student.matnat.uio.no> Prajwal Suhas P wrote: > This is the patch for ticket #203. Great! You appear to be on the right track. Here's some feedback. > I came out with this, but i did not understand the need for > freeze_endpoints.I still haven't supplied the complete patch with all > the tests.This is for confirming whether i'm in the right direction. ForFromStatNode is used in two situations in Cython code; in the Cython-specific syntax cdef int i for i from start <= i < end [by step]: ... and in normal syntax cdef int i for i in range([start,] end, [step]): ... In the former case, end should NOT be frozen (i.e. if it is a function call it should be called again for every iteration, if it is a variable then changes to it in the loop body should modify the number of iterations, etc.). Therefore, it is necesarry to have a flag which is set in the latter case but not in the former case. See my original post for when and how the flag should be set. > diff -r e2365a6d00b8 Cython/Compiler/Nodes.py > --- a/Cython/Compiler/Nodes.py Tue Mar 17 07:32:51 2009 +0100 > +++ b/Cython/Compiler/Nodes.py Wed Mar 18 00:20:19 2009 +0530 > @@ -3971,11 +3971,13 @@ > code.putln("if (%s%s %s %s) {" % ( > self.bound1.result(), offset, self.relation2, > self.bound2.result() > )) > + temp_for_loop_bound = > code.funcstate.allocate_temp(PyrexTypes.c_char_ptr_type, manage_ref=False) Why do you use c_char_ptr_type here? You must use the right type, which you can look up from the "entry" attributes in the corresponding bounds nodes (do a "print self.dump()" to see the tree; and then find the types you need there). > + code.putln("%s = %s;" % (temp_for_loop_bound, > self.bound2.result())) This only tackles the endpoint. What if the start is a function call, or step is a function call? -- Dag Sverre From greg.ewing at canterbury.ac.nz Tue Mar 17 22:35:10 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 18 Mar 2009 09:35:10 +1200 Subject: [Cython] [Fwd: Re: Beginner-friendly tickets] In-Reply-To: <49BFFC42.5090800@student.matnat.uio.no> References: <49BE1093.7080505@student.matnat.uio.no> <49BF4BA3.3080302@behnel.de> <49BFFC42.5090800@student.matnat.uio.no> Message-ID: <49C0178E.60908@canterbury.ac.nz> Dag Sverre Seljebotn wrote: > ForFromStatNode is used in two situations in Cython code; in the > Cython-specific syntax > > cdef int i > for i from start <= i < end [by step]: > ... > > In the former case, end should NOT be frozen (i.e. if it is a function > call it should be called again for every iteration Has Cython changed this? It's not how the for-from statement works in Pyrex -- the limits are only evaluated once. -- Greg From robertwb at math.washington.edu Tue Mar 17 23:03:11 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 17 Mar 2009 15:03:11 -0700 Subject: [Cython] [Fwd: Re: Beginner-friendly tickets] In-Reply-To: <49C0178E.60908@canterbury.ac.nz> References: <49BE1093.7080505@student.matnat.uio.no> <49BF4BA3.3080302@behnel.de> <49BFFC42.5090800@student.matnat.uio.no> <49C0178E.60908@canterbury.ac.nz> Message-ID: <2A4C178D-E9F6-4F65-8240-CF680C082C53@math.washington.edu> On Mar 17, 2009, at 2:35 PM, Greg Ewing wrote: > Dag Sverre Seljebotn wrote: > >> ForFromStatNode is used in two situations in Cython code; in the >> Cython-specific syntax >> >> cdef int i >> for i from start <= i < end [by step]: >> ... >> >> In the former case, end should NOT be frozen (i.e. if it is a >> function >> call it should be called again for every iteration > > Has Cython changed this? It's not how the for-from > statement works in Pyrex -- the limits are only > evaluated once. No, Cython has not changed this, and I'm not sure we should. But there is some inconsistency: %cython cdef int get_bound(int m): print "get_bound(%s)"%m return m def test_var(int n, int m): cdef int i for i from 0 <= i < n: print "at", i n = m return i def test_func(int n): cdef int i for i from 0 <= i < get_bound(n): print "at", i return i sage: test_var(10, 5) at 0 at 1 at 2 at 3 at 4 5 sage: test_func(5) get_bound(5) at 0 at 1 at 2 at 3 at 4 5 - Robert From robertwb at math.washington.edu Wed Mar 18 04:55:24 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 17 Mar 2009 20:55:24 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <50876.213.61.181.86.1237279705.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <49BEDD8E.1000007@canterbury.ac.nz> <49BF426D.7010005@behnel.de> <49BF5871.4040404@student.matnat.uio.no> <50876.213.61.181.86.1237279705.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: I started a CEP up at http://wiki.cython.org/enhancements/division which attempts to summarize the issues brought up on this thread. - Robert From robertwb at math.washington.edu Wed Mar 18 05:02:59 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 17 Mar 2009 21:02:59 -0700 Subject: [Cython] Optional arguments question In-Reply-To: <20090315175215.GA10289@encolpuis> References: <20090315175215.GA10289@encolpuis> Message-ID: On Mar 15, 2009, at 10:52 AM, Gabriel Gellner wrote: > I can't seem to get optional arguments for a char variable to work > in a cdef > function, is this a bug or am I missing something simple? Any help > would, as > always, be greatly appreciated. > > For example: > > cdef void test_args(int number, char letter='C'): > print letter > > def test(): > test_args(3, letter='F') This is because cdef functions can't take keyword arguments. The error 11:13: Cannot convert 'void (int, struct __pyx__test_args *__pyx_optional_args)' to Python object should be better... http://trac.cython.org/cython_trac/ticket/241 - Robert From poonaatsoc at gmail.com Wed Mar 18 05:32:59 2009 From: poonaatsoc at gmail.com (Anoop Saldanha) Date: Wed, 18 Mar 2009 10:02:59 +0530 Subject: [Cython] [Fwd: Re: Beginner-friendly tickets] In-Reply-To: <2A4C178D-E9F6-4F65-8240-CF680C082C53@math.washington.edu> References: <49BE1093.7080505@student.matnat.uio.no> <49BF4BA3.3080302@behnel.de> <49BFFC42.5090800@student.matnat.uio.no> <49C0178E.60908@canterbury.ac.nz> <2A4C178D-E9F6-4F65-8240-CF680C082C53@math.washington.edu> Message-ID: Does it make any difference, if the start point in a range() is a function call or even if it is modified to a new value inside the loop? Even if it is modified, the start would be evaluated only once, isn't it? I defined a start function and supplied it to the range() and from the output it looked like start is called once, and also the c output seems to agree with this. In this case if the start is a function call, then it doesn't have to be handled right? I am not sure if I have caught the topic right! Or am i missing something here? On Wed, Mar 18, 2009 at 3:33 AM, Robert Bradshaw < robertwb at math.washington.edu> wrote: > On Mar 17, 2009, at 2:35 PM, Greg Ewing wrote: > > > Dag Sverre Seljebotn wrote: > > > >> ForFromStatNode is used in two situations in Cython code; in the > >> Cython-specific syntax > >> > >> cdef int i > >> for i from start <= i < end [by step]: > >> ... > >> > >> In the former case, end should NOT be frozen (i.e. if it is a > >> function > >> call it should be called again for every iteration > > > > Has Cython changed this? It's not how the for-from > > statement works in Pyrex -- the limits are only > > evaluated once. > > No, Cython has not changed this, and I'm not sure we should. But > there is some inconsistency: > > %cython > cdef int get_bound(int m): > print "get_bound(%s)"%m > return m > > def test_var(int n, int m): > cdef int i > for i from 0 <= i < n: > print "at", i > n = m > return i > > def test_func(int n): > cdef int i > for i from 0 <= i < get_bound(n): > print "at", i > return i > > sage: test_var(10, 5) > at 0 > at 1 > at 2 > at 3 > at 4 > 5 > sage: test_func(5) > get_bound(5) > at 0 > at 1 > at 2 > at 3 > at 4 > 5 > > - Robert > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Regards, Anoop S. People who says, "It's not winning or losing that matters, but participation", probably lost. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090318/c7a2f4e9/attachment-0001.htm From dagss at student.matnat.uio.no Wed Mar 18 09:01:10 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 18 Mar 2009 09:01:10 +0100 Subject: [Cython] [Fwd: Re: Beginner-friendly tickets] In-Reply-To: <2A4C178D-E9F6-4F65-8240-CF680C082C53@math.washington.edu> References: <49BE1093.7080505@student.matnat.uio.no> <49BF4BA3.3080302@behnel.de> <49BFFC42.5090800@student.matnat.uio.no> <49C0178E.60908@canterbury.ac.nz> <2A4C178D-E9F6-4F65-8240-CF680C082C53@math.washington.edu> Message-ID: <49C0AA46.4080408@student.matnat.uio.no> Robert Bradshaw wrote: > On Mar 17, 2009, at 2:35 PM, Greg Ewing wrote: > >> Dag Sverre Seljebotn wrote: >> >>> ForFromStatNode is used in two situations in Cython code; in the >>> Cython-specific syntax >>> >>> cdef int i >>> for i from start <= i < end [by step]: >>> ... >>> >>> In the former case, end should NOT be frozen (i.e. if it is a >>> function >>> call it should be called again for every iteration >> Has Cython changed this? It's not how the for-from >> statement works in Pyrex -- the limits are only >> evaluated once. > > No, Cython has not changed this, and I'm not sure we should. But > there is some inconsistency: Ahh, wasn't aware of this, I leapt to conclusions. So the way we should go then is fix the endpoint also in for-from if it is a variable? > %cython > cdef int get_bound(int m): > print "get_bound(%s)"%m > return m > > def test_func(int n): > cdef int i > for i from 0 <= i < get_bound(n): > print "at", i > return i > > sage: test_func(5) > get_bound(5) > at 0 > at 1 > at 2 > at 3 > at 4 > 5 Ahh, that's strange. So the reason this happens is that the transformation from the for-in node doesn't create the same thing that a for-from written manually would give then... changing your code to cdef int get_bound(int m): print "get_bound(%s)"%m return m def test_func(int n): cdef int i for i in range(get_bound(n)): print "at", i return i gives get_bound(5) get_bound(5) at 0 get_bound(5) at 1 get_bound(5) at 2 get_bound(5) at 3 get_bound(5) at 4 get_bound(5) 4 Which is rather suboptimal. -- Dag Sverre From dagss at student.matnat.uio.no Wed Mar 18 09:04:17 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 18 Mar 2009 09:04:17 +0100 Subject: [Cython] [Fwd: Re: Beginner-friendly tickets] In-Reply-To: References: <49BE1093.7080505@student.matnat.uio.no> <49BF4BA3.3080302@behnel.de> <49BFFC42.5090800@student.matnat.uio.no> <49C0178E.60908@canterbury.ac.nz> <2A4C178D-E9F6-4F65-8240-CF680C082C53@math.washington.edu> Message-ID: <49C0AB01.9080401@student.matnat.uio.no> Anoop Saldanha wrote: > Does it make any difference, if the start point in a range() is a > function call or even if it is modified to a new value inside the loop? > Even if it is modified, the start would be evaluated only once, isn't > it? I defined a start function and supplied it to the range() and from > the output it looked like start is called once, and also the c output > seems to agree with this. In this case if the start is a function > call, then it doesn't have to be handled right? I am not sure if I have > caught the topic right! Or am i missing something here? No, of course, you are right. And actually I think Magnus already fixed "step". Testcases should reveal whether he did, there ought to be something in the ticket he fixed: http://trac.cython.org/cython_trac/ticket/196 So the patch posted here was likely correct except for the type of the temp variable. -- Dag Sverre From stefan_ml at behnel.de Wed Mar 18 09:11:11 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 18 Mar 2009 09:11:11 +0100 Subject: [Cython] Beginner-friendly tickets In-Reply-To: References: <49BE1093.7080505@student.matnat.uio.no> <49BF4BA3.3080302@behnel.de> Message-ID: <49C0AC9F.4080102@behnel.de> Prajwal Suhas P wrote: > This is the patch for ticket #203. Thanks. I noticed that the ticket lacks a test case, though. It was agreed on that tickets must have a corresponding (failing!) test case before patches get applied. Could you please provide one? That will also make it easier for you to understand the implications of your change and to figure out corner cases. http://wiki.cython.org/HackerGuide Stefan From poonaatsoc at gmail.com Wed Mar 18 09:50:52 2009 From: poonaatsoc at gmail.com (Anoop Saldanha) Date: Wed, 18 Mar 2009 14:20:52 +0530 Subject: [Cython] Beginner-friendly tickets In-Reply-To: <49C0AC9F.4080102@behnel.de> References: <49BE1093.7080505@student.matnat.uio.no> <49BF4BA3.3080302@behnel.de> <49C0AC9F.4080102@behnel.de> Message-ID: I think he/she has to supply the patch with the right type for the temp variable. Should the type of the temp variable be of PyrexTypes.py_object_type? From what I can see, the code is currently using PyrexTypes.py_object_type. On Wed, Mar 18, 2009 at 1:41 PM, Stefan Behnel wrote: > > Prajwal Suhas P wrote: > > This is the patch for ticket #203. > > Thanks. I noticed that the ticket lacks a test case, though. It was agreed > on that tickets must have a corresponding (failing!) test case before > patches get applied. Could you please provide one? That will also make it > easier for you to understand the implications of your change and to figure > out corner cases. > > http://wiki.cython.org/HackerGuide > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Regards, Anoop S. People who says, "It's not winning or losing that matters, but participation", probably lost. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090318/f7feb1a0/attachment.htm From dagss at student.matnat.uio.no Wed Mar 18 10:14:59 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 18 Mar 2009 10:14:59 +0100 Subject: [Cython] Beginner-friendly tickets In-Reply-To: References: <49BE1093.7080505@student.matnat.uio.no> <49BF4BA3.3080302@behnel.de> <49C0AC9F.4080102@behnel.de> Message-ID: <49C0BB93.40503@student.matnat.uio.no> Anoop Saldanha wrote: > I think he/she has to supply the patch with the right type for the temp > variable. > > Should the type of the temp variable be of PyrexTypes.py_object_type? > From what I can see, the code is currently using PyrexTypes.py_object_type. First off, I was wrong in my previous email; step is NOT put into a temporary, and it probably should be. Secondly, the type to use for temps should be the same as the nodes containing the expressions in question. E.g. self.step.type, self.bound1.type, and so on, contains references to the right type objects. (Inspect the tree of a for-loop by "print somenode.dump()"/"print self.dump()" to see how it works -- note that e.g. the type of IntNodes doesn't show up here as it is a hard-coded attribute for that class.) -- Dag Sverre From greg.ewing at canterbury.ac.nz Wed Mar 18 11:51:56 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 18 Mar 2009 22:51:56 +1200 Subject: [Cython] [Fwd: Re: Beginner-friendly tickets] In-Reply-To: <2A4C178D-E9F6-4F65-8240-CF680C082C53@math.washington.edu> References: <49BE1093.7080505@student.matnat.uio.no> <49BF4BA3.3080302@behnel.de> <49BFFC42.5090800@student.matnat.uio.no> <49C0178E.60908@canterbury.ac.nz> <2A4C178D-E9F6-4F65-8240-CF680C082C53@math.washington.edu> Message-ID: <49C0D24C.9020803@canterbury.ac.nz> Robert Bradshaw wrote: > No, Cython has not changed this, and I'm not sure we should. But > there is some inconsistency: Yes, I noticed that too. It's a bug -- a temp should always be used unless the limit expression is constant. -- Greg From poonaatsoc at gmail.com Wed Mar 18 16:27:53 2009 From: poonaatsoc at gmail.com (Anoop Saldanha) Date: Wed, 18 Mar 2009 20:57:53 +0530 Subject: [Cython] Beginner-friendly tickets In-Reply-To: <49C0BB93.40503@student.matnat.uio.no> References: <49BE1093.7080505@student.matnat.uio.no> <49BF4BA3.3080302@behnel.de> <49C0AC9F.4080102@behnel.de> <49C0BB93.40503@student.matnat.uio.no> Message-ID: To create a temp of the right type for an expression, one can do a code.funcstate.allocate_temp(self.bound2.type, manage_ref=False), where self.bound2 is the expresssion? Also is there an irc channel for cython? Another question that I had in mind is, when would ForInStatNode and ForFromStatNode be used? My understanding was that the former would be used in case of "for in" and the latter would be used in case of "for from" loops. But cdef int g(): print "g called" return 5 def f(): cdef int i for i in range(g()): print i print i uses a ForFromStatNode for the "for in" loop! On Wed, Mar 18, 2009 at 2:44 PM, Dag Sverre Seljebotn < dagss at student.matnat.uio.no> wrote: > Anoop Saldanha wrote: > > I think he/she has to supply the patch with the right type for the temp > > variable. > > > > Should the type of the temp variable be of PyrexTypes.py_object_type? > > From what I can see, the code is currently using > PyrexTypes.py_object_type. > > First off, I was wrong in my previous email; step is NOT put into a > temporary, and it probably should be. > > Secondly, the type to use for temps should be the same as the nodes > containing the expressions in question. E.g. self.step.type, > self.bound1.type, and so on, contains references to the right type > objects. (Inspect the tree of a for-loop by "print > somenode.dump()"/"print self.dump()" to see how it works -- note that > e.g. the type of IntNodes doesn't show up here as it is a hard-coded > attribute for that class.) > > -- > Dag Sverre > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Regards, Anoop S. People who says, "It's not winning or losing that matters, but participation", probably lost. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090318/54271ece/attachment.htm From kwmsmith at gmail.com Wed Mar 18 17:29:49 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Wed, 18 Mar 2009 11:29:49 -0500 Subject: [Cython] Beginner-friendly tickets In-Reply-To: References: <49BE1093.7080505@student.matnat.uio.no> <49BF4BA3.3080302@behnel.de> <49C0AC9F.4080102@behnel.de> <49C0BB93.40503@student.matnat.uio.no> Message-ID: 2009/3/18 Anoop Saldanha : [snip] > > But > > cdef int g(): > ???? print "g called" > ???? return 5 > > def f(): > ??? cdef int i > ??? for i in range(g()): > ??????? print i > ??? print i > > uses a ForFromStatNode for the "for in" loop! I'm not a core developer, so this is open to correction, but putting in some PrintTree() calls into Cython/Compiler/Main.py:create_pipeline() shows that it starts out as a ForInStatNode and is changed to a ForFromStatNode. Perhaps because cython inferrs that g returns a C int and it triggers the 'range' optimization. Before any transformations: - (root): ModuleNode(pos=(foo.pyx:1:0)) - body: StatListNode(pos=(foo.pyx:1:0)) - stats[0]: CFuncDefNode(pos=(foo.pyx:1:5)) [removing cdef int g() stuff] - stats[1]: DefNode(name="f") - body: StatListNode(pos=(foo.pyx:6:4)) - stats[0]: CVarDefNode(pos=(foo.pyx:6:9)) - base_type: CSimpleBaseTypeNode(pos=(foo.pyx:6:9)) - declarators[0]: CNameDeclaratorNode(pos=(foo.pyx:6:13)) - stats[1]: ForInStatNode(pos=(foo.pyx:8:4)) <== starts out as as ForInStatNode [snip rest of def f() function body] After all transformations: - (root): ModuleNode(pos=(foo.pyx:1:0)) - body: StatListNode(pos=(foo.pyx:1:0)) - stats[0]: CFuncDefNode(pos=(foo.pyx:1:5)) [snip func body] - stats[1]: DefNode(name="f") - body: StatListNode(pos=(foo.pyx:6:4)) - stats[0]: ForFromStatNode(pos=(foo.pyx:8:4)) <== Changed to a ForFromStatNode. - target: NameNode(type=, name="i") - bound1: IntNode(type=) - bound2: SimpleCallNode(type=) - function: NameNode(type= []>, name="g") - step: IntNode(type=) - body: StatListNode(pos=(foo.pyx:9:8)) - stats[0]: PrintStatNode(pos=(foo.pyx:9:8)) - arg_tuple: TupleNode(type=) - args[0]: CoerceToPyTypeNode(type=) - arg: NameNode(type=, name="i") - stats[1]: PrintStatNode(pos=(foo.pyx:10:4)) - arg_tuple: TupleNode(type=) - args[0]: CoerceToPyTypeNode(type=) - arg: NameNode(type=, name="i") Kurt From carl.witty at gmail.com Wed Mar 18 17:36:11 2009 From: carl.witty at gmail.com (Carl Witty) Date: Wed, 18 Mar 2009 09:36:11 -0700 Subject: [Cython] Cython header ordering bug (forwarded from Ryan Hinton) Message-ID: I'm forwarding the following mail from Ryan Hinton (rwh4s at virginia.edu), because he's not subscribed to the list: [begin forwarded email] I'm having problems with C header #include order, and maybe something more. The easiest way I know to reproduce the bug is to add from sage.rings.polynomial.polynomial_gf2x cimport Polynomial_GF2X near the top of a random .pyx file in the Sage source. I used sage/finance/fractal.pyx and got $ sage -b ---------------------------------------------------------- sage: Building and installing modified Sage library files. Installing c_lib scons: `install' is up to date. Updating Cython code.... Building modified file sage/finance/fractal.pyx. python2.5 `which cython` --embed-positions --incref-local-binop -I/home/ryan/sage/devel/sage-sage-patches -o sage/finance/fractal.c sage/finance/fractal.pyx sage/finance/fractal.pyx --> /home/ryan/sage/local//lib/python/site-packages//sage/finance/fractal.pyx Time to execute 1 commands: 7.48708200455 seconds Finished compiling Cython code (time = 8.25148797035 seconds) running install running build running build_py running build_ext building 'sage.finance.fractal' extension gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/ryan/sage/local//include -I/home/ryan/sage/local//include/csage -I/home/ryan/sage/devel//sage/sage/ext -I/home/ryan/sage/local/include/python2.5 -c sage/finance/fractal.c -o build/temp.linux-i686-2.5/sage/finance/fractal.o -w In file included from sage/finance/fractal.c:112: /home/ryan/sage/local//include/csage/ntl_wrap.h:47: error: expected ?)? before ?*? token /home/ryan/sage/local//include/csage/ntl_wrap.h:48: error: expected ?;?, ?,? or ?)? before ?*? token /home/ryan/sage/local//include/csage/ntl_wrap.h:90: error: expected ?)? before ?*? token sage/finance/fractal.c:397: error: field ?x? has incomplete type error: command 'gcc' failed with exit status 1 sage: There was an error installing modified sage library code. Looking at the generated C code, it appears that ntl_wrap.h is included before gmp.h, but ntl_wrap.h uses the mpz_t type defined in gmp.h. The second error I have since fixed -- it requires the #clang c++ directive. Please copy me on any follow-ups as I am not (yet) subscribed to the list. Enjoy! --- Ryan Hinton PhD candidate, electrical engineering University of Virginia From poonaatsoc at gmail.com Wed Mar 18 17:53:18 2009 From: poonaatsoc at gmail.com (Anoop Saldanha) Date: Wed, 18 Mar 2009 22:23:18 +0530 Subject: [Cython] Beginner-friendly tickets In-Reply-To: References: <49BE1093.7080505@student.matnat.uio.no> <49BF4BA3.3080302@behnel.de> <49C0AC9F.4080102@behnel.de> <49C0BB93.40503@student.matnat.uio.no> Message-ID: Thanks Kurt. That was helpful. Looks like the optimization is happening at IterationTransform() in the pipeline. On Wed, Mar 18, 2009 at 9:59 PM, Kurt Smith wrote: > 2009/3/18 Anoop Saldanha : > > [snip] > > > > > But > > > > cdef int g(): > > print "g called" > > return 5 > > > > def f(): > > cdef int i > > for i in range(g()): > > print i > > print i > > > > uses a ForFromStatNode for the "for in" loop! > > I'm not a core developer, so this is open to correction, but putting > in some PrintTree() calls into > Cython/Compiler/Main.py:create_pipeline() shows that it starts out as > a ForInStatNode and is changed to a ForFromStatNode. Perhaps because > cython inferrs that g returns a C int and it triggers the 'range' > optimization. > > Before any transformations: > > - (root): ModuleNode(pos=(foo.pyx:1:0)) > - body: StatListNode(pos=(foo.pyx:1:0)) > - stats[0]: CFuncDefNode(pos=(foo.pyx:1:5)) > [removing cdef int g() stuff] > - stats[1]: DefNode(name="f") > - body: StatListNode(pos=(foo.pyx:6:4)) > - stats[0]: CVarDefNode(pos=(foo.pyx:6:9)) > - base_type: CSimpleBaseTypeNode(pos=(foo.pyx:6:9)) > - declarators[0]: CNameDeclaratorNode(pos=(foo.pyx:6:13)) > - stats[1]: ForInStatNode(pos=(foo.pyx:8:4)) <== starts out as > as ForInStatNode > [snip rest of def f() function body] > > After all transformations: > > - (root): ModuleNode(pos=(foo.pyx:1:0)) > - body: StatListNode(pos=(foo.pyx:1:0)) > - stats[0]: CFuncDefNode(pos=(foo.pyx:1:5)) > [snip func body] > - stats[1]: DefNode(name="f") > - body: StatListNode(pos=(foo.pyx:6:4)) > - stats[0]: ForFromStatNode(pos=(foo.pyx:8:4)) <== Changed to > a ForFromStatNode. > - target: NameNode(type=, name="i") > - bound1: IntNode(type=) > - bound2: SimpleCallNode(type=) > - function: NameNode(type= > []>, name="g") > - step: IntNode(type=) > - body: StatListNode(pos=(foo.pyx:9:8)) > - stats[0]: PrintStatNode(pos=(foo.pyx:9:8)) > - arg_tuple: TupleNode(type=) > - args[0]: CoerceToPyTypeNode(type=) > - arg: NameNode(type=, name="i") > - stats[1]: PrintStatNode(pos=(foo.pyx:10:4)) > - arg_tuple: TupleNode(type=) > - args[0]: CoerceToPyTypeNode(type=) > - arg: NameNode(type=, name="i") > > Kurt > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Regards, Anoop S. People who says, "It's not winning or losing that matters, but participation", probably lost. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090318/bfe9034b/attachment.htm From kwmsmith at gmail.com Wed Mar 18 18:16:23 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Wed, 18 Mar 2009 12:16:23 -0500 Subject: [Cython] Fix for ticket #135 Message-ID: Ticket 135 (return statement outside a function body crashes cython) can be fixed with a simple assignment of None to self.return_type in the Scope __init__ function. The crash happens when there is a return statement in the class body as well as at the module level. Every function object explicitly assigns something else to the scope's return_type, so the function objects are not affected by this change. Here's the diff with a test case included. I included this on the trac page, but thought it would be easiest to get it pushed here. diff -r e2365a6d00b8 -r f153d2da27b0 Cython/Compiler/Symtab.py --- a/Cython/Compiler/Symtab.py Tue Mar 17 07:32:51 2009 +0100 +++ b/Cython/Compiler/Symtab.py Tue Mar 17 15:10:14 2009 -0500 @@ -235,6 +235,7 @@ class Scope(object): self.pystring_entries = [] self.buffer_entries = [] self.control_flow = ControlFlow.LinearControlFlow() + self.return_type = None def start_branching(self, pos): self.control_flow = self.control_flow.start_branch(pos) diff -r e2365a6d00b8 -r f153d2da27b0 tests/errors/return_outside_function_T135.pyx --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/tests/errors/return_outside_function_T135.pyx Tue Mar 17 15:10:14 2009 -0500 @@ -0,0 +1,12 @@ +return 'bar' +class A: + return None + +cdef class B: + return None + +_ERRORS = u''' +1:0: Return not inside a function body +3:4: Return not inside a function body +6:4: Return not inside a function body +''' From stefan_ml at behnel.de Wed Mar 18 18:22:33 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 18 Mar 2009 18:22:33 +0100 Subject: [Cython] Cython header ordering bug (forwarded from Ryan Hinton) In-Reply-To: References: Message-ID: <49C12DD9.6040002@behnel.de> Hi, Carl Witty wrote: > I'm forwarding the following mail from Ryan Hinton > (rwh4s at virginia.edu), because he's not subscribed to the list: > > [begin forwarded email] > > I'm having problems with C header #include order, and maybe something > more. The easiest way I know to reproduce the bug is to add > > from sage.rings.polynomial.polynomial_gf2x cimport Polynomial_GF2X > [...] > Looking at the generated C code, it appears that ntl_wrap.h is > included before gmp.h, but ntl_wrap.h uses the mpz_t type defined in > gmp.h. I don't have sage to test this, but is this using the latest Cython? I remember that Robert fixed a couple of things regarding header include order, and it might have been after the 0.10 releases. Stefan From poonaatsoc at gmail.com Wed Mar 18 18:29:51 2009 From: poonaatsoc at gmail.com (Anoop Saldanha) Date: Wed, 18 Mar 2009 22:59:51 +0530 Subject: [Cython] Beginner-friendly tickets In-Reply-To: References: <49BE1093.7080505@student.matnat.uio.no> <49BF4BA3.3080302@behnel.de> <49C0AC9F.4080102@behnel.de> <49C0BB93.40503@student.matnat.uio.no> Message-ID: Looks like it was available here http://docs.cython.org/docs/language_basics.html#integer-for-loops and http://docs.cython.org/docs/pyrex_differences.html#automatic-range-conversion On Wed, Mar 18, 2009 at 10:23 PM, Anoop Saldanha wrote: > Thanks Kurt. That was helpful. Looks like the optimization is happening > at IterationTransform() in the pipeline. > > > On Wed, Mar 18, 2009 at 9:59 PM, Kurt Smith wrote: > >> 2009/3/18 Anoop Saldanha : >> >> [snip] >> >> > >> > But >> > >> > cdef int g(): >> > print "g called" >> > return 5 >> > >> > def f(): >> > cdef int i >> > for i in range(g()): >> > print i >> > print i >> > >> > uses a ForFromStatNode for the "for in" loop! >> >> I'm not a core developer, so this is open to correction, but putting >> in some PrintTree() calls into >> Cython/Compiler/Main.py:create_pipeline() shows that it starts out as >> a ForInStatNode and is changed to a ForFromStatNode. Perhaps because >> cython inferrs that g returns a C int and it triggers the 'range' >> optimization. >> >> Before any transformations: >> >> - (root): ModuleNode(pos=(foo.pyx:1:0)) >> - body: StatListNode(pos=(foo.pyx:1:0)) >> - stats[0]: CFuncDefNode(pos=(foo.pyx:1:5)) >> [removing cdef int g() stuff] >> - stats[1]: DefNode(name="f") >> - body: StatListNode(pos=(foo.pyx:6:4)) >> - stats[0]: CVarDefNode(pos=(foo.pyx:6:9)) >> - base_type: CSimpleBaseTypeNode(pos=(foo.pyx:6:9)) >> - declarators[0]: CNameDeclaratorNode(pos=(foo.pyx:6:13)) >> - stats[1]: ForInStatNode(pos=(foo.pyx:8:4)) <== starts out as >> as ForInStatNode >> [snip rest of def f() function body] >> >> After all transformations: >> >> - (root): ModuleNode(pos=(foo.pyx:1:0)) >> - body: StatListNode(pos=(foo.pyx:1:0)) >> - stats[0]: CFuncDefNode(pos=(foo.pyx:1:5)) >> [snip func body] >> - stats[1]: DefNode(name="f") >> - body: StatListNode(pos=(foo.pyx:6:4)) >> - stats[0]: ForFromStatNode(pos=(foo.pyx:8:4)) <== Changed to >> a ForFromStatNode. >> - target: NameNode(type=, name="i") >> - bound1: IntNode(type=) >> - bound2: SimpleCallNode(type=) >> - function: NameNode(type= >> []>, name="g") >> - step: IntNode(type=) >> - body: StatListNode(pos=(foo.pyx:9:8)) >> - stats[0]: PrintStatNode(pos=(foo.pyx:9:8)) >> - arg_tuple: TupleNode(type=) >> - args[0]: CoerceToPyTypeNode(type=) >> - arg: NameNode(type=, name="i") >> - stats[1]: PrintStatNode(pos=(foo.pyx:10:4)) >> - arg_tuple: TupleNode(type=) >> - args[0]: CoerceToPyTypeNode(type=) >> - arg: NameNode(type=, name="i") >> >> Kurt >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > > > -- > Regards, > Anoop S. > > People who says, "It's not winning or losing that matters, but > participation", probably lost. > -- Regards, Anoop S. People who says, "It's not winning or losing that matters, but participation", probably lost. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090318/787ab8e2/attachment-0001.htm From stefan_ml at behnel.de Wed Mar 18 18:32:52 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 18 Mar 2009 18:32:52 +0100 Subject: [Cython] Fix for ticket #135 In-Reply-To: References: Message-ID: <49C13044.2090808@behnel.de> Hi, Kurt Smith wrote: > Ticket 135 (return statement outside a function body crashes cython) > can be fixed with a simple assignment of None to self.return_type in > the Scope __init__ function. The crash happens when there is a return > statement in the class body as well as at the module level. > > Every function object explicitly assigns something else to the scope's > return_type, so the function objects are not affected by this change. > > Here's the diff with a test case included. I included this on the > trac page, but thought it would be easiest to get it pushed here. > > diff -r e2365a6d00b8 -r f153d2da27b0 Cython/Compiler/Symtab.py > --- a/Cython/Compiler/Symtab.py Tue Mar 17 07:32:51 2009 +0100 > +++ b/Cython/Compiler/Symtab.py Tue Mar 17 15:10:14 2009 -0500 > @@ -235,6 +235,7 @@ class Scope(object): > self.pystring_entries = [] > self.buffer_entries = [] > self.control_flow = ControlFlow.LinearControlFlow() > + self.return_type = None > > def start_branching(self, pos): > self.control_flow = self.control_flow.start_branch(pos) Thanks for looking into this. Without actually knowing the details, I'm not sure yet that this is a good way to do it. I do not have high confidence in the current handling of the return_type, so setting it globally in a class might cover incorrect assumptions in other parts of the code that just expect that the return_type is set at some point in the pipeline where it normally wouldn't be set (i.e. long before code generation). I ran into tons of problems like these since I started working on ticket 144. It's actually possible that this will be fixed when I'm done with that ticket. BTW, I'm working in cython-unstable with these things. Stefan From kwmsmith at gmail.com Wed Mar 18 18:36:26 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Wed, 18 Mar 2009 12:36:26 -0500 Subject: [Cython] Fix for ticket #135 In-Reply-To: <49C13044.2090808@behnel.de> References: <49C13044.2090808@behnel.de> Message-ID: On Wed, Mar 18, 2009 at 12:32 PM, Stefan Behnel wrote: > > Thanks for looking into this. Without actually knowing the details, I'm not > sure yet that this is a good way to do it. I do not have high confidence in > the current handling of the return_type, so setting it globally in a class > might cover incorrect assumptions in other parts of the code that just > expect that the return_type is set at some point in the pipeline where it > normally wouldn't be set (i.e. long before code generation). I ran into > tons of problems like these since I started working on ticket 144. It's > actually possible that this will be fixed when I'm done with that ticket. > > BTW, I'm working in cython-unstable with these things. Good points -- thanks for the feedback. I've still got lots to learn about Cython ;-) Kurt From dagss at student.matnat.uio.no Wed Mar 18 18:59:39 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 18 Mar 2009 18:59:39 +0100 Subject: [Cython] Fix for ticket #135 In-Reply-To: <49C13044.2090808@behnel.de> References: <49C13044.2090808@behnel.de> Message-ID: <49C1368B.3010007@student.matnat.uio.no> Stefan Behnel wrote: > Hi, > > Kurt Smith wrote: >> Ticket 135 (return statement outside a function body crashes cython) >> can be fixed with a simple assignment of None to self.return_type in >> the Scope __init__ function. The crash happens when there is a return >> statement in the class body as well as at the module level. >> >> Every function object explicitly assigns something else to the scope's >> return_type, so the function objects are not affected by this change. >> >> Here's the diff with a test case included. I included this on the >> trac page, but thought it would be easiest to get it pushed here. >> >> diff -r e2365a6d00b8 -r f153d2da27b0 Cython/Compiler/Symtab.py >> --- a/Cython/Compiler/Symtab.py Tue Mar 17 07:32:51 2009 +0100 >> +++ b/Cython/Compiler/Symtab.py Tue Mar 17 15:10:14 2009 -0500 >> @@ -235,6 +235,7 @@ class Scope(object): >> self.pystring_entries = [] >> self.buffer_entries = [] >> self.control_flow = ControlFlow.LinearControlFlow() >> + self.return_type = None >> >> def start_branching(self, pos): >> self.control_flow = self.control_flow.start_branch(pos) > > Thanks for looking into this. Without actually knowing the details, I'm not > sure yet that this is a good way to do it. I do not have high confidence in > the current handling of the return_type, so setting it globally in a class > might cover incorrect assumptions in other parts of the code that just > expect that the return_type is set at some point in the pipeline where it > normally wouldn't be set (i.e. long before code generation). I ran into > tons of problems like these since I started working on ticket 144. It's > actually possible that this will be fixed when I'm done with that ticket. If not, PostParse (in ParseTreeTransforms.py) is a very safe spot to do things like this. One can trivially implement visit_ReturnStatNode and raise syntax errors if one is not within a function, and it means that if there's an error, one simply stops before getting to somewhere it can cause trouble. IMO more error checking (e.g. for most syntax errors) should go into there. -- Dag Sverre From carl.witty at gmail.com Wed Mar 18 18:49:15 2009 From: carl.witty at gmail.com (Carl Witty) Date: Wed, 18 Mar 2009 10:49:15 -0700 Subject: [Cython] Cython header ordering bug (forwarded from Ryan Hinton) In-Reply-To: <49C12DD9.6040002@behnel.de> References: <49C12DD9.6040002@behnel.de> Message-ID: On Wed, Mar 18, 2009 at 10:22 AM, Stefan Behnel wrote: > I don't have sage to test this, but is this using the latest Cython? I > remember that Robert fixed a couple of things regarding header include > order, and it might have been after the 0.10 releases. This would have been with Cython 0.10.3. I'll check this evening to see if the problem persists with Cython 0.11, if nobody beats me to it. Carl From dagss at student.matnat.uio.no Wed Mar 18 19:19:50 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 18 Mar 2009 19:19:50 +0100 Subject: [Cython] Cython header ordering bug (forwarded from Ryan Hinton) In-Reply-To: References: <49C12DD9.6040002@behnel.de> Message-ID: <49C13B46.3020807@student.matnat.uio.no> Carl Witty wrote: > On Wed, Mar 18, 2009 at 10:22 AM, Stefan Behnel wrote: >> I don't have sage to test this, but is this using the latest Cython? I >> remember that Robert fixed a couple of things regarding header include >> order, and it might have been after the 0.10 releases. > > This would have been with Cython 0.10.3. I'll check this evening to > see if the problem persists with Cython 0.11, if nobody beats me to > it. Here's the ticket: http://trac.cython.org/cython_trac/ticket/188 It's for 0.11, so your chances are good. -- Dag Sverre From sturla at molden.no Wed Mar 18 19:13:22 2009 From: sturla at molden.no (Sturla Molden) Date: Wed, 18 Mar 2009 19:13:22 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <49BEDD8E.1000007@canterbury.ac.nz> <49BF426D.7010005@behnel.de> <49BF5871.4040404@student.matnat.uio.no> <50876.213.61.181.86.1237279705.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49C139C2.1020500@molden.no> On 3/18/2009 4:55 AM, Robert Bradshaw wrote: > I started a CEP up at http://wiki.cython.org/enhancements/division > which attempts to summarize the issues brought up on this thread. Regardless of opinion in the matter, this is a very good summary. S.M. From robertwb at math.washington.edu Wed Mar 18 19:57:45 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 18 Mar 2009 11:57:45 -0700 Subject: [Cython] Cython header ordering bug (forwarded from Ryan Hinton) In-Reply-To: References: <49C12DD9.6040002@behnel.de> Message-ID: <477DA1E4-D83E-4AB8-8B55-A0A319CC562A@math.washington.edu> On Mar 18, 2009, at 10:49 AM, Carl Witty wrote: > On Wed, Mar 18, 2009 at 10:22 AM, Stefan Behnel > wrote: >> I don't have sage to test this, but is this using the latest >> Cython? I >> remember that Robert fixed a couple of things regarding header >> include >> order, and it might have been after the 0.10 releases. > > This would have been with Cython 0.10.3. I'll check this evening to > see if the problem persists with Cython 0.11, if nobody beats me to > it. In Cython 0.11, the headers appear exactly in the order that they are given in the source files, without duplication. If I include or import a file, all (recursively) included .h files are included at that point in the sequence. An example should make things clearer. --------------------- car.pyx ---------------- cdef extern from "a.h": pass cdef extern from "b.h": pass cimport engine cdef extern from "z.h": pass cdef extern from "y.h": pass -------------- engine.pxd ----------------- cdef extern from "a.h": pass cdef extern from "c.h": pass cdef extern from "y.h": pass ---------------------------------- Will result in # include "a.h" # include "b.h" # include "c.h" # include "y.h" # include "z.h" - Robert From robertwb at math.washington.edu Wed Mar 18 19:59:49 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 18 Mar 2009 11:59:49 -0700 Subject: [Cython] Fix for ticket #135 In-Reply-To: <49C1368B.3010007@student.matnat.uio.no> References: <49C13044.2090808@behnel.de> <49C1368B.3010007@student.matnat.uio.no> Message-ID: <13046D31-4E52-4DEB-9288-108070EDC5F4@math.washington.edu> On Mar 18, 2009, at 10:59 AM, Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Hi, >> >> Kurt Smith wrote: >>> Ticket 135 (return statement outside a function body crashes cython) >>> can be fixed with a simple assignment of None to self.return_type in >>> the Scope __init__ function. The crash happens when there is a >>> return >>> statement in the class body as well as at the module level. >>> >>> Every function object explicitly assigns something else to the >>> scope's >>> return_type, so the function objects are not affected by this >>> change. >>> >>> Here's the diff with a test case included. I included this on the >>> trac page, but thought it would be easiest to get it pushed here. >>> >>> diff -r e2365a6d00b8 -r f153d2da27b0 Cython/Compiler/Symtab.py >>> --- a/Cython/Compiler/Symtab.py Tue Mar 17 07:32:51 2009 +0100 >>> +++ b/Cython/Compiler/Symtab.py Tue Mar 17 15:10:14 2009 -0500 >>> @@ -235,6 +235,7 @@ class Scope(object): >>> self.pystring_entries = [] >>> self.buffer_entries = [] >>> self.control_flow = ControlFlow.LinearControlFlow() >>> + self.return_type = None >>> >>> def start_branching(self, pos): >>> self.control_flow = self.control_flow.start_branch(pos) >> >> Thanks for looking into this. Without actually knowing the >> details, I'm not >> sure yet that this is a good way to do it. I do not have high >> confidence in >> the current handling of the return_type, so setting it globally in >> a class >> might cover incorrect assumptions in other parts of the code that >> just >> expect that the return_type is set at some point in the pipeline >> where it >> normally wouldn't be set (i.e. long before code generation). I ran >> into >> tons of problems like these since I started working on ticket 144. >> It's >> actually possible that this will be fixed when I'm done with that >> ticket. > > If not, PostParse (in ParseTreeTransforms.py) is a very safe spot > to do > things like this. One can trivially implement visit_ReturnStatNode and > raise syntax errors if one is not within a function, and it means that > if there's an error, one simply stops before getting to somewhere > it can > cause trouble. > > IMO more error checking (e.g. for most syntax errors) should go > into there. +1 to this idea. It's a lot clearer what's going on. - Robert From robertwb at math.washington.edu Wed Mar 18 20:02:14 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 18 Mar 2009 12:02:14 -0700 Subject: [Cython] [Fwd: Re: Beginner-friendly tickets] In-Reply-To: <49C0D24C.9020803@canterbury.ac.nz> References: <49BE1093.7080505@student.matnat.uio.no> <49BF4BA3.3080302@behnel.de> <49BFFC42.5090800@student.matnat.uio.no> <49C0178E.60908@canterbury.ac.nz> <2A4C178D-E9F6-4F65-8240-CF680C082C53@math.washington.edu> <49C0D24C.9020803@canterbury.ac.nz> Message-ID: On Mar 18, 2009, at 3:51 AM, Greg Ewing wrote: > Robert Bradshaw wrote: > >> No, Cython has not changed this, and I'm not sure we should. But >> there is some inconsistency: > > Yes, I noticed that too. It's a bug -- a temp > should always be used unless the limit expression > is constant. So in Pyrex, the endpoint will always be evaluated once? (This is a departure from the actual C for loop, but IMHO a good one.) - Robert From robertwb at math.washington.edu Wed Mar 18 20:04:52 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 18 Mar 2009 12:04:52 -0700 Subject: [Cython] IRC In-Reply-To: References: <49BE1093.7080505@student.matnat.uio.no> <49BF4BA3.3080302@behnel.de> <49C0AC9F.4080102@behnel.de> <49C0BB93.40503@student.matnat.uio.no> Message-ID: <254C9D4B-AB73-46F2-B32F-4C6BC128F402@math.washington.edu> On Mar 18, 2009, at 8:27 AM, Anoop Saldanha wrote: > Also is there an irc channel for cython? There was a year ago, but it wasn't very populated back then. Perhaps there's enough of a community to start one up again. - Robert From stefan_ml at behnel.de Wed Mar 18 20:08:24 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 18 Mar 2009 20:08:24 +0100 Subject: [Cython] Fix for ticket #135 In-Reply-To: <49C1368B.3010007@student.matnat.uio.no> References: <49C13044.2090808@behnel.de> <49C1368B.3010007@student.matnat.uio.no> Message-ID: <49C146A8.3010306@behnel.de> Dag Sverre Seljebotn wrote: > PostParse (in ParseTreeTransforms.py) is a very safe spot to do > things like this. One can trivially implement visit_ReturnStatNode and > raise syntax errors if one is not within a function, and it means that > if there's an error, one simply stops before getting to somewhere it can > cause trouble. > > IMO more error checking (e.g. for most syntax errors) should go into there. Yes, good idea. Then we could prevent jumping into the code generation phase completely whenever an error occurred in an earlier phase, which will keep the code generation code simple. Stefan From stefan_ml at behnel.de Wed Mar 18 20:44:12 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 18 Mar 2009 20:44:12 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <49BEDD8E.1000007@canterbury.ac.nz> <49BF426D.7010005@behnel.de> <49BF5871.4040404@student.matnat.uio.no> <50876.213.61.181.86.1237279705.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49C14F0C.4030000@behnel.de> Robert Bradshaw wrote: > I started a CEP up at http://wiki.cython.org/enhancements/division > which attempts to summarize the issues brought up on this thread. I added the pros and cons that I could come up with to each solution. Maybe others can add to that. Stefan From robertwb at math.washington.edu Wed Mar 18 20:50:05 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 18 Mar 2009 12:50:05 -0700 Subject: [Cython] loops Message-ID: There's a lot of work going on with loops right now, I've also discovered http://trac.cython.org/cython_trac/ticket/243 . Another problem is that we haven't nailed down exactly what the semantics for the for...from loop should be. Should it be a C loop (where the endpoint and step are re-evaluated every time, and changing the index variable from the loop body modifies the loop behavior) or simply alternative syntax for the for...in range() loop. I would say the latter, (use a while loop if one wants the other behavior) but certainly not some hybrid like we have now. - Robert From dagss at student.matnat.uio.no Wed Mar 18 21:14:50 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 18 Mar 2009 21:14:50 +0100 Subject: [Cython] Fix for ticket #135 In-Reply-To: <49C146A8.3010306@behnel.de> References: <49C13044.2090808@behnel.de> <49C1368B.3010007@student.matnat.uio.no> <49C146A8.3010306@behnel.de> Message-ID: <49C1563A.4020803@student.matnat.uio.no> Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> PostParse (in ParseTreeTransforms.py) is a very safe spot to do >> things like this. One can trivially implement visit_ReturnStatNode and >> raise syntax errors if one is not within a function, and it means that >> if there's an error, one simply stops before getting to somewhere it can >> cause trouble. >> >> IMO more error checking (e.g. for most syntax errors) should go into there. > > Yes, good idea. Then we could prevent jumping into the code generation > phase completely whenever an error occurred in an earlier phase, which will > keep the code generation code simple. I've lobbied earlier for moving all error checking out of code generation, because it makes writing transforms much simpler (it's difficult to think about the various ways the tree can be broken when writing a transform, and so one ends up with compiler crash bugs due to transform code). I think that ideally there should be a point in the pipeline where you can say "all errors have now been reported"; and all the optimization and/or feature-implementing transforms could happen after. It's not at all a priority, but at least it is something to keep in mind when new error checking is added. -- Dag Sverre From poonaatsoc at gmail.com Wed Mar 18 21:07:39 2009 From: poonaatsoc at gmail.com (Anoop Saldanha) Date: Thu, 19 Mar 2009 01:37:39 +0530 Subject: [Cython] loops In-Reply-To: References: Message-ID: On Thu, Mar 19, 2009 at 1:20 AM, Robert Bradshaw < robertwb at math.washington.edu> wrote: > There's a lot of work going on with loops right now, I've also > discovered http://trac.cython.org/cython_trac/ticket/243 . Another > problem is that we haven't nailed down exactly what the semantics for > the for...from loop should be. Should it be a C loop (where the > endpoint and step are re-evaluated every time, and changing the index > variable from the loop body modifies the loop behavior) or simply > alternative syntax for the for...in range() loop. I would say the > latter, (use a while loop if one wants the other behavior) but > certainly not some hybrid like we have now. > > - Robert > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > Won't ticket 203 solution apply for this too? As in using a temp variable? And like Dag told you can use freeze_endpoints, so that we can confine it only for range optimizations. -- Regards, Anoop S. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090319/2af38533/attachment.htm From dagss at student.matnat.uio.no Wed Mar 18 21:26:22 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 18 Mar 2009 21:26:22 +0100 Subject: [Cython] loops In-Reply-To: References: Message-ID: <49C158EE.6000502@student.matnat.uio.no> Robert Bradshaw wrote: > There's a lot of work going on with loops right now, I've also > discovered http://trac.cython.org/cython_trac/ticket/243 . Another > problem is that we haven't nailed down exactly what the semantics for > the for...from loop should be. Should it be a C loop (where the > endpoint and step are re-evaluated every time, and changing the index > variable from the loop body modifies the loop behavior) or simply > alternative syntax for the for...in range() loop. I would say the > latter, (use a while loop if one wants the other behavior) but > certainly not some hybrid like we have now. At least changing the value the iterator variable has after the loop "from C to Python semantics" (to use a phrase that's awfully familiar) would potentially break code. And then the semantics differ between the two anyway. I think a case could be made that "for-from is the C for loop", and always reevaluate end/step. However, myself I consider for-from present mostly for backwards compatability anyway, and so I don't really have an opinion about "the right" behaviour here. -- Dag Sverre From psp.reachable at gmail.com Wed Mar 18 21:12:07 2009 From: psp.reachable at gmail.com (Prajwal Suhas P) Date: Thu, 19 Mar 2009 01:42:07 +0530 Subject: [Cython] loops In-Reply-To: References: Message-ID: Yes, a little modification for the solution of ticket#203 should apply for this ticket too.So, can this be assigned to me so i can handle both the cases. 2009/3/19 Anoop Saldanha > On Thu, Mar 19, 2009 at 1:20 AM, Robert Bradshaw < > robertwb at math.washington.edu> wrote: > >> There's a lot of work going on with loops right now, I've also >> discovered http://trac.cython.org/cython_trac/ticket/243 . Another >> problem is that we haven't nailed down exactly what the semantics for >> the for...from loop should be. Should it be a C loop (where the >> endpoint and step are re-evaluated every time, and changing the index >> variable from the loop body modifies the loop behavior) or simply >> alternative syntax for the for...in range() loop. I would say the >> latter, (use a while loop if one wants the other behavior) but >> certainly not some hybrid like we have now. >> >> - Robert >> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > Won't ticket 203 solution apply for this too? As in using a temp > variable? And like Dag told you can use freeze_endpoints, so that we can > confine it only for range optimizations. > > -- > Regards, > Anoop S. > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > -- Regards, Prajwal S. "There is little difference in people,but this little difference makes a big difference.The little difference is attitude,the big difference is whether its positive or negative. " -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090319/f9d03620/attachment.htm From robertwb at math.washington.edu Wed Mar 18 21:12:36 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 18 Mar 2009 13:12:36 -0700 Subject: [Cython] loops In-Reply-To: References: Message-ID: <7EBFBC8A-48A3-4C15-B70A-BE106CBD97EB@math.washington.edu> On Mar 18, 2009, at 1:07 PM, Anoop Saldanha wrote: > On Thu, Mar 19, 2009 at 1:20 AM, Robert Bradshaw > wrote: > There's a lot of work going on with loops right now, I've also > discovered http://trac.cython.org/cython_trac/ticket/243 . Another > problem is that we haven't nailed down exactly what the semantics for > the for...from loop should be. Should it be a C loop (where the > endpoint and step are re-evaluated every time, and changing the index > variable from the loop body modifies the loop behavior) or simply > alternative syntax for the for...in range() loop. I would say the > latter, (use a while loop if one wants the other behavior) but > certainly not some hybrid like we have now. > > - Robert > > Won't ticket 203 solution apply for this too? As in using a temp > variable? And like Dag told you can use freeze_endpoints, so that > we can confine it only for range optimizations. Yes, #203 is very relevant, which is why I'm not just jumping in and fixing it and duplicating effort. My points are that freeze_endpoints should always be true (i.e. it shouldn't even be an option) and that the loop variable itself should be a temp, not exposed to the user. - Robert From poonaatsoc at gmail.com Wed Mar 18 21:12:41 2009 From: poonaatsoc at gmail.com (Anoop Saldanha) Date: Thu, 19 Mar 2009 01:42:41 +0530 Subject: [Cython] IRC In-Reply-To: <254C9D4B-AB73-46F2-B32F-4C6BC128F402@math.washington.edu> References: <49BE1093.7080505@student.matnat.uio.no> <49BF4BA3.3080302@behnel.de> <49C0AC9F.4080102@behnel.de> <49C0BB93.40503@student.matnat.uio.no> <254C9D4B-AB73-46F2-B32F-4C6BC128F402@math.washington.edu> Message-ID: On Thu, Mar 19, 2009 at 12:34 AM, Robert Bradshaw < robertwb at math.washington.edu> wrote: > On Mar 18, 2009, at 8:27 AM, Anoop Saldanha wrote: > > > Also is there an irc channel for cython? > > There was a year ago, but it wasn't very populated back then. Perhaps > there's enough of a community to start one up again. > > - Robert > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > Maybe we can have a #cython on freenode. If it is okay, I can open the channel or if anyone else wants to open the channel, they can go ahead and do so. -- Regards, Anoop S. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090319/8ed410df/attachment.htm From wstein at gmail.com Wed Mar 18 21:14:58 2009 From: wstein at gmail.com (William Stein) Date: Wed, 18 Mar 2009 13:14:58 -0700 Subject: [Cython] IRC In-Reply-To: References: <49BE1093.7080505@student.matnat.uio.no> <49BF4BA3.3080302@behnel.de> <49C0AC9F.4080102@behnel.de> <49C0BB93.40503@student.matnat.uio.no> <254C9D4B-AB73-46F2-B32F-4C6BC128F402@math.washington.edu> Message-ID: <85e81ba30903181314m2ce92b4ex8b27610b91d162dd@mail.gmail.com> 2009/3/18 Anoop Saldanha : > > On Thu, Mar 19, 2009 at 12:34 AM, Robert Bradshaw > wrote: >> >> On Mar 18, 2009, at 8:27 AM, Anoop Saldanha wrote: >> >> > Also is there an irc channel for cython? >> >> There was a year ago, but it wasn't very populated back then. Perhaps >> there's enough of a community to start one up again. >> >> - Robert > > Maybe we can have a #cython on freenode.? If it is okay, I can open the > channel or if anyone else wants to open the channel, they can go ahead and > do so. +1 - I was going to suggest exactly the same thing. And just for the record #sage-devel on irc.freenode.net is also a good place to find people hanging out who have a lot of cython experience, since we use Cython so much in the Sage project. -- William From poonaatsoc at gmail.com Wed Mar 18 21:29:00 2009 From: poonaatsoc at gmail.com (Anoop Saldanha) Date: Thu, 19 Mar 2009 01:59:00 +0530 Subject: [Cython] IRC In-Reply-To: <85e81ba30903181314m2ce92b4ex8b27610b91d162dd@mail.gmail.com> References: <49BE1093.7080505@student.matnat.uio.no> <49BF4BA3.3080302@behnel.de> <49C0AC9F.4080102@behnel.de> <49C0BB93.40503@student.matnat.uio.no> <254C9D4B-AB73-46F2-B32F-4C6BC128F402@math.washington.edu> <85e81ba30903181314m2ce92b4ex8b27610b91d162dd@mail.gmail.com> Message-ID: On Thu, Mar 19, 2009 at 1:44 AM, William Stein wrote: > 2009/3/18 Anoop Saldanha : > > > > On Thu, Mar 19, 2009 at 12:34 AM, Robert Bradshaw > > wrote: > >> > >> On Mar 18, 2009, at 8:27 AM, Anoop Saldanha wrote: > >> > >> > Also is there an irc channel for cython? > >> > >> There was a year ago, but it wasn't very populated back then. Perhaps > >> there's enough of a community to start one up again. > >> > >> - Robert > > > > Maybe we can have a #cython on freenode. If it is okay, I can open the > > channel or if anyone else wants to open the channel, they can go ahead > and > > do so. > > +1 - I was going to suggest exactly the same thing. > > And just for the record #sage-devel on irc.freenode.net is also a good > place to find people hanging out who have a lot of cython experience, > since we use Cython so much in the Sage project. > > -- William > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > Looks like a #cython has already been registred by Arc a year ago. -- Regards, Anoop S. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090319/70659f3f/attachment.htm From wstein at gmail.com Wed Mar 18 21:30:42 2009 From: wstein at gmail.com (William Stein) Date: Wed, 18 Mar 2009 13:30:42 -0700 Subject: [Cython] IRC In-Reply-To: References: <49BE1093.7080505@student.matnat.uio.no> <49C0AC9F.4080102@behnel.de> <49C0BB93.40503@student.matnat.uio.no> <254C9D4B-AB73-46F2-B32F-4C6BC128F402@math.washington.edu> <85e81ba30903181314m2ce92b4ex8b27610b91d162dd@mail.gmail.com> Message-ID: <85e81ba30903181330v76fe0632j28f8d069ddfdf5e@mail.gmail.com> 2009/3/18 Anoop Saldanha : > On Thu, Mar 19, 2009 at 1:44 AM, William Stein wrote: >> >> 2009/3/18 Anoop Saldanha : >> > >> > On Thu, Mar 19, 2009 at 12:34 AM, Robert Bradshaw >> > wrote: >> >> >> >> On Mar 18, 2009, at 8:27 AM, Anoop Saldanha wrote: >> >> >> >> > Also is there an irc channel for cython? >> >> >> >> There was a year ago, but it wasn't very populated back then. Perhaps >> >> there's enough of a community to start one up again. >> >> >> >> - Robert >> > >> > Maybe we can have a #cython on freenode.? If it is okay, I can open the >> > channel or if anyone else wants to open the channel, they can go ahead >> > and >> > do so. >> >> +1 ?- I was going to suggest exactly the same thing. >> >> And just for the record #sage-devel on irc.freenode.net is also a good >> place to find people hanging out who have a lot of cython experience, >> since we use Cython so much in the Sage project. >> >> ?-- William >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev > > Looks like a #cython has already been registred by Arc a year ago. > What does this mean? Does this mean we should instead use "#cython-dev"? That would be nice since it is the same as the name of the mailing list. William From poonaatsoc at gmail.com Wed Mar 18 21:35:53 2009 From: poonaatsoc at gmail.com (Anoop Saldanha) Date: Thu, 19 Mar 2009 02:05:53 +0530 Subject: [Cython] IRC In-Reply-To: <85e81ba30903181330v76fe0632j28f8d069ddfdf5e@mail.gmail.com> References: <49BE1093.7080505@student.matnat.uio.no> <49C0AC9F.4080102@behnel.de> <49C0BB93.40503@student.matnat.uio.no> <254C9D4B-AB73-46F2-B32F-4C6BC128F402@math.washington.edu> <85e81ba30903181314m2ce92b4ex8b27610b91d162dd@mail.gmail.com> <85e81ba30903181330v76fe0632j28f8d069ddfdf5e@mail.gmail.com> Message-ID: On Thu, Mar 19, 2009 at 2:00 AM, William Stein wrote: > 2009/3/18 Anoop Saldanha : > > On Thu, Mar 19, 2009 at 1:44 AM, William Stein wrote: > >> > >> 2009/3/18 Anoop Saldanha : > >> > > >> > On Thu, Mar 19, 2009 at 12:34 AM, Robert Bradshaw > >> > wrote: > >> >> > >> >> On Mar 18, 2009, at 8:27 AM, Anoop Saldanha wrote: > >> >> > >> >> > Also is there an irc channel for cython? > >> >> > >> >> There was a year ago, but it wasn't very populated back then. Perhaps > >> >> there's enough of a community to start one up again. > >> >> > >> >> - Robert > >> > > >> > Maybe we can have a #cython on freenode. If it is okay, I can open > the > >> > channel or if anyone else wants to open the channel, they can go ahead > >> > and > >> > do so. > >> > >> +1 - I was going to suggest exactly the same thing. > >> > >> And just for the record #sage-devel on irc.freenode.net is also a good > >> place to find people hanging out who have a lot of cython experience, > >> since we use Cython so much in the Sage project. > >> > >> -- William > >> _______________________________________________ > >> Cython-dev mailing list > >> Cython-dev at codespeak.net > >> http://codespeak.net/mailman/listinfo/cython-dev > > > > Looks like a #cython has already been registred by Arc a year ago. > > > > What does this mean? > > Does this mean we should instead use "#cython-dev"? That would be > nice since it is the same as the name of the mailing list. > > William > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > No idea as to who Arc(the person who registered #cython) is. A whois on him directed me to #python. -- Regards, Anoop S. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090319/09728ec3/attachment.htm From stefan_ml at behnel.de Wed Mar 18 21:52:20 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 18 Mar 2009 21:52:20 +0100 Subject: [Cython] constant handling refactored and up on cython-unstable (ticket 144) Message-ID: <49C15F04.2020504@behnel.de> Hi, I just pushed the constant allocation refactoring to cython-unstable. There are still a few failing test cases, especially regarding struct coercion. The main problem was that some places in the code expected the result code to be available for constants during the analysis phase. Where I could not move it to the code generation phase, I had to replace this with a call to "compile_time_value()" for now, not sure if this is the right thing to do... The change can clearly benefit from some review and some additional bug fixing. I hope someone beats me to it. :) Stefan From greg.ewing at canterbury.ac.nz Wed Mar 18 22:00:34 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 19 Mar 2009 09:00:34 +1200 Subject: [Cython] [Fwd: Re: Beginner-friendly tickets] In-Reply-To: References: <49BE1093.7080505@student.matnat.uio.no> <49BF4BA3.3080302@behnel.de> <49BFFC42.5090800@student.matnat.uio.no> <49C0178E.60908@canterbury.ac.nz> <2A4C178D-E9F6-4F65-8240-CF680C082C53@math.washington.edu> <49C0D24C.9020803@canterbury.ac.nz> Message-ID: <49C160F2.6000803@canterbury.ac.nz> Robert Bradshaw wrote: > So in Pyrex, the endpoint will always be evaluated once? (This is a > departure from the actual C for loop, but IMHO a good one.) That's the intention, yes. If you want a dynamic endpoint (a rare case in my experience) you just have to use a while loop. -- Greg From greg.ewing at canterbury.ac.nz Wed Mar 18 22:08:47 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 19 Mar 2009 09:08:47 +1200 Subject: [Cython] loops In-Reply-To: <49C158EE.6000502@student.matnat.uio.no> References: <49C158EE.6000502@student.matnat.uio.no> Message-ID: <49C162DF.7020708@canterbury.ac.nz> Dag Sverre Seljebotn wrote: > However, myself I consider for-from present mostly for backwards > compatability anyway, Efficiency wasn't the only reason I added the for-from loop to Pyrex. Equally important, perhaps more so, is that it provides a concise and readable way of specifying inclusion or exclusion of each endpoint. As for the value of the loop variable after the loop, I'd be willing to consider it undefined, so that whatever behaviour is most efficiently implementable can be used. -- Greg From m.gysel at gmail.com Wed Mar 18 22:17:32 2009 From: m.gysel at gmail.com (Martin Gysel) Date: Wed, 18 Mar 2009 22:17:32 +0100 Subject: [Cython] callback/segfault Message-ID: <49C164EC.9040807@gmail.com> Hi I have a 'cython' class which interfaces c code. therefor I need some callback functions which I declared outside of the class. Over a userdata argument pointer this function gets the reference of my class. Callback registration and also the callback itself work. The problem is, as soon as the callback starts to execute, I'll get a segfault (from time to time it also works...). So I debugged the c code, the segfault occurs while getting the sdtout for the print statement. So I removed all print statements but from my callback I want to call either a function of my class or some python callback (which needs to be registered first). For me it seems the segfault occurs when accessing some python objects. I also tried to play around with 'which gil' and 'nogil' but no luck. When using 'which gil' it segfaults immediately the callback gets called. What do I wrong? I assume it should be possible to have print in callback functions and also to have access to python objects. thx /martin From m.gysel at gmail.com Wed Mar 18 22:21:20 2009 From: m.gysel at gmail.com (Martin Gysel) Date: Wed, 18 Mar 2009 22:21:20 +0100 Subject: [Cython] IRC In-Reply-To: References: <49BE1093.7080505@student.matnat.uio.no> <49BF4BA3.3080302@behnel.de> <49C0AC9F.4080102@behnel.de> <49C0BB93.40503@student.matnat.uio.no> <254C9D4B-AB73-46F2-B32F-4C6BC128F402@math.washington.edu> Message-ID: <49C165D0.8030409@gmail.com> Anoop Saldanha schrieb: > On Thu, Mar 19, 2009 at 12:34 AM, Robert Bradshaw < > robertwb at math.washington.edu> wrote: > >> On Mar 18, 2009, at 8:27 AM, Anoop Saldanha wrote: >> >>> Also is there an irc channel for cython? >>> >> There was a year ago, but it wasn't very populated back then. Perhaps >> there's enough of a community to start one up again. >> >> - Robert >> >> > > Maybe we can have a #cython on freenode. If it is okay, I can open the > channel or if anyone else wants to open the channel, they can go ahead and > do so. > +1 if it's called cython or cython-dev or what ever I love the idea... /martin From robertwb at math.washington.edu Wed Mar 18 22:32:05 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 18 Mar 2009 14:32:05 -0700 Subject: [Cython] IRC In-Reply-To: <49C165D0.8030409@gmail.com> References: <49BE1093.7080505@student.matnat.uio.no> <49BF4BA3.3080302@behnel.de> <49C0AC9F.4080102@behnel.de> <49C0BB93.40503@student.matnat.uio.no> <254C9D4B-AB73-46F2-B32F-4C6BC128F402@math.washington.edu> <49C165D0.8030409@gmail.com> Message-ID: <293F708B-BB62-4E07-9D99-18E50313EF5D@math.washington.edu> On Mar 18, 2009, at 2:21 PM, Martin Gysel wrote: > Anoop Saldanha schrieb: >> On Thu, Mar 19, 2009 at 12:34 AM, Robert Bradshaw < >> robertwb at math.washington.edu> wrote: >> >>> On Mar 18, 2009, at 8:27 AM, Anoop Saldanha wrote: >>> >>>> Also is there an irc channel for cython? >>>> >>> There was a year ago, but it wasn't very populated back then. >>> Perhaps >>> there's enough of a community to start one up again. >>> >>> - Robert >>> >>> >> >> Maybe we can have a #cython on freenode. If it is okay, I can >> open the >> channel or if anyone else wants to open the channel, they can go >> ahead and >> do so. >> > +1 if it's called cython or cython-dev or what ever I love the idea... I think just plain #cython would be nice if we could get it--I'd rather not intimidate non-developers from joining in. - Robert From amirnntp at gmail.com Thu Mar 19 00:38:24 2009 From: amirnntp at gmail.com (Amir) Date: Wed, 18 Mar 2009 23:38:24 +0000 (UTC) Subject: [Cython] numpy calls without Python overhead? Message-ID: The end of the numpy section of the cython docs suggests the possibility to call numpy/scipy functions without Python call overhead. How can this be done? A test script at bottom is 1.8 times faster when I expand numpy calls into simple for loops (n,m = 1000,1500). weave.inline is 2.7 times faster. Looking at the cython -a output, not sure where most of that time is lost. Looks like strides generate many more calls and dot products are done using Python calls for multiplications, for example. Replacing numpy calls with explicit code even for simple things as matrix multiplication will end up being bad as the code wont take advantage of optimizations like blas on multicore. Sorry if this has been discussed already. from numpy import (random, diag, dot, empty, sign, abs) from numpy cimport ndarray cimport cython @cython.boundscheck(False) def f(ndarray[double, ndim=1] y, ndarray[double, ndim=2] X, double lam, int maxit=10000, double eta=1e-9): cdef int n = X.shape[0], m = X.shape[1] cdef ndarray[double, ndim=1] theta = random.normal(0, 1, m) cdef ndarray[double, ndim=1] v = diag(dot(X.T, X)) cdef unsigned int i = 0, j = 0 cdef double alpha cdef double chg = 1. cdef ndarray[double, ndim=1] theta_old = empty((m,)) cdef ndarray[double, ndim=1] Xtheta = dot(X, theta) while i < maxit and chg > eta: i += 1 theta_old = theta.copy() for j in range(m): if theta[j] != 0.: Xtheta -= theta[j] * X[:,j] alpha = dot(X[:,j].T, y - Xtheta) if abs(alpha) > lam: theta[j] = sign(alpha) * (abs(alpha) - lam) / v[j] Xtheta += theta[j] * X[:,j] else: theta[j] = 0. chg = sum((theta - theta_old)**2) / sum(theta_old**2) return theta From carl.witty at gmail.com Thu Mar 19 02:58:15 2009 From: carl.witty at gmail.com (Carl Witty) Date: Wed, 18 Mar 2009 18:58:15 -0700 Subject: [Cython] Cython header ordering bug (forwarded from Ryan Hinton) In-Reply-To: References: <49C12DD9.6040002@behnel.de> Message-ID: On Wed, Mar 18, 2009 at 10:49 AM, Carl Witty wrote: > On Wed, Mar 18, 2009 at 10:22 AM, Stefan Behnel wrote: >> I don't have sage to test this, but is this using the latest Cython? I >> remember that Robert fixed a couple of things regarding header include >> order, and it might have been after the 0.10 releases. > > This would have been with Cython 0.10.3. ?I'll check this evening to > see if the problem persists with Cython 0.11, if nobody beats me to > it. OK, the problem still occurs with Cython 0.11. I can fix the ntl_wrap.h problems by adding in the appropriate "cdef extern from ...", so I don't think there's anything for Cython to fix there; but this problem remains: sage/finance/fractal.c:536: error: field ?x? has incomplete type And sure enough, 'x' does have incomplete type when I look at fractal.c -- the type is forward-declared as a struct, but there's never a definition of the type. This comes from Sage's polynomial template mechanism, implemented by Robert Bradshaw and Martin Albrecht -- if only I knew where to get hold of one of these people :) Robert, could you take a look? You're probably the best person to figure out whether this is a bug in Cython or in Sage. To reproduce the problem, add this line to the top of sage/finance/fractal.pyx: from sage.rings.polynomial.polynomial_gf2x cimport Polynomial_GF2X and to get rid of the ntl_wrap errors, add the following lines to the top of sage/libs/ntl/ntl_GF2_decl.pxd: cdef extern from "gmp.h": pass Carl From carl.witty at gmail.com Thu Mar 19 04:07:45 2009 From: carl.witty at gmail.com (Carl Witty) Date: Wed, 18 Mar 2009 20:07:45 -0700 Subject: [Cython] Cython header ordering bug (forwarded from Ryan Hinton) In-Reply-To: References: <49C12DD9.6040002@behnel.de> Message-ID: On Wed, Mar 18, 2009 at 6:58 PM, Carl Witty wrote: ... a bunch of garbage. Can I just cancel that last email, please? :) Ryan points out that he already figured out the "incomplete type" part of the problem; you need to compile the file as C++, because the type declarations are guarded by "#ifdef __cplusplus" in the header file. And I stared at the relevant line in fractal.c without understanding it, so I didn't understand where the problem was at all. For some reason, I find it hard to read code that uses 60-character type names that start __pyx_... :) Oops. Carl From poonaatsoc at gmail.com Thu Mar 19 05:57:11 2009 From: poonaatsoc at gmail.com (Anoop Saldanha) Date: Thu, 19 Mar 2009 10:27:11 +0530 Subject: [Cython] IRC In-Reply-To: <293F708B-BB62-4E07-9D99-18E50313EF5D@math.washington.edu> References: <49BE1093.7080505@student.matnat.uio.no> <49C0AC9F.4080102@behnel.de> <49C0BB93.40503@student.matnat.uio.no> <254C9D4B-AB73-46F2-B32F-4C6BC128F402@math.washington.edu> <49C165D0.8030409@gmail.com> <293F708B-BB62-4E07-9D99-18E50313EF5D@math.washington.edu> Message-ID: On Thu, Mar 19, 2009 at 3:02 AM, Robert Bradshaw < robertwb at math.washington.edu> wrote: > On Mar 18, 2009, at 2:21 PM, Martin Gysel wrote: > > > Anoop Saldanha schrieb: > >> On Thu, Mar 19, 2009 at 12:34 AM, Robert Bradshaw < > >> robertwb at math.washington.edu> wrote: > >> > >>> On Mar 18, 2009, at 8:27 AM, Anoop Saldanha wrote: > >>> > >>>> Also is there an irc channel for cython? > >>>> > >>> There was a year ago, but it wasn't very populated back then. > >>> Perhaps > >>> there's enough of a community to start one up again. > >>> > >>> - Robert > >>> > >>> > >> > >> Maybe we can have a #cython on freenode. If it is okay, I can > >> open the > >> channel or if anyone else wants to open the channel, they can go > >> ahead and > >> do so. > >> > > +1 if it's called cython or cython-dev or what ever I love the idea... > > I think just plain #cython would be nice if we could get it--I'd > rather not intimidate non-developers from joining in. > > - Robert > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > Will get in touch with Arc, and ask him if he can transfer the ownership(#cython has already been registered by Arc a year ago, but no one seems to be using it) to one of us here. -- Regards, Anoop S. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090319/a6ea37fd/attachment.htm From robertwb at math.washington.edu Thu Mar 19 06:20:36 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 18 Mar 2009 22:20:36 -0700 Subject: [Cython] Cython header ordering bug (forwarded from Ryan Hinton) In-Reply-To: References: <49C12DD9.6040002@behnel.de> Message-ID: <6296B2E7-DDAB-4756-9B69-A1E78902B64B@math.washington.edu> On Mar 18, 2009, at 8:07 PM, Carl Witty wrote: > On Wed, Mar 18, 2009 at 6:58 PM, Carl Witty > wrote: > ... > > a bunch of garbage. Can I just cancel that last email, please? :) :). I'm glad to hear that, it sounded scary... > Ryan points out that he already figured out the "incomplete type" part > of the problem; you need to compile the file as C++, because the type > declarations are guarded by "#ifdef __cplusplus" in the header file. > And I stared at the relevant line in fractal.c without understanding > it, so I didn't understand where the problem was at all. For some > reason, I find it hard to read code that uses 60-character type names > that start __pyx_... :) I've toyed with the idea of making it output much less verbose but still unique names, but I'm used to it by now. In any case, the C output isn't meant to be read by mere mortals... :). - Robert From robertwb at math.washington.edu Thu Mar 19 06:24:32 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 18 Mar 2009 22:24:32 -0700 Subject: [Cython] loops In-Reply-To: <49C162DF.7020708@canterbury.ac.nz> References: <49C158EE.6000502@student.matnat.uio.no> <49C162DF.7020708@canterbury.ac.nz> Message-ID: <6B1E8348-1E34-4218-90CB-58F521A43C22@math.washington.edu> On Mar 18, 2009, at 2:08 PM, Greg Ewing wrote: > Dag Sverre Seljebotn wrote: > >> However, myself I consider for-from present mostly for backwards >> compatability anyway, Me too, though I actually do like the syntax. Lets let the endpoints always be fixed for consistency between Cython and Pyrex. > Efficiency wasn't the only reason I added the > for-from loop to Pyrex. Equally important, perhaps > more so, is that it provides a concise and readable > way of specifying inclusion or exclusion of each > endpoint. > > As for the value of the loop variable after the > loop, I'd be willing to consider it undefined, so > that whatever behaviour is most efficiently > implementable can be used. What about the behavior of for i from 0 <= i < 10: print i i += 5 ? - Robert From robertwb at math.washington.edu Thu Mar 19 06:29:54 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 18 Mar 2009 22:29:54 -0700 Subject: [Cython] callback/segfault In-Reply-To: <49C164EC.9040807@gmail.com> References: <49C164EC.9040807@gmail.com> Message-ID: On Mar 18, 2009, at 2:17 PM, Martin Gysel wrote: > Hi > > I have a 'cython' class which interfaces c code. therefor I need some > callback functions which I declared outside of the class. Over a > userdata argument pointer this function gets the reference of my > class. > Callback registration and also the callback itself work. The > problem is, > as soon as the callback starts to execute, I'll get a segfault (from > time to time it also works...). So I debugged the c code, the segfault > occurs while getting the sdtout for the print statement. So I removed > all print statements but from my callback I want to call either a > function of my class or some python callback (which needs to be > registered first). For me it seems the segfault occurs when accessing > some python objects. I also tried to play around with 'which gil' and > 'nogil' but no luck. When using 'which gil' it segfaults > immediately the > callback gets called. > What do I wrong? I assume it should be possible to have print in > callback functions and also to have access to python objects. Yes, that should be possible. It does sound like you're having threading/gil issues. What is your callback function signature? When you say you've tried "with gil" are you doing cdef void my_callback(void *data) with gil: ... - Robert From robertwb at math.washington.edu Thu Mar 19 07:17:12 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 18 Mar 2009 23:17:12 -0700 Subject: [Cython] patch to make lexicon pickle loading more robust In-Reply-To: <8B415CD2-E3B7-4682-ADE3-519ED8B861FB@math.washington.edu> References: <4db580fd0903162015o2fac5a29t8bafbb8a47dc2bea@mail.gmail.com> <8B415CD2-E3B7-4682-ADE3-519ED8B861FB@math.washington.edu> Message-ID: <10E52A01-5D4E-4ADF-8660-B2FDB65AE888@math.washington.edu> Fixed. On Mar 16, 2009, at 10:46 PM, Robert Bradshaw wrote: > I noticed this too. The patch looks good (on sight, haven't applied > it yet). > > http://trac.cython.org/cython_trac/ticket/239 > From robertwb at math.washington.edu Thu Mar 19 07:18:35 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 18 Mar 2009 23:18:35 -0700 Subject: [Cython] improving Py <-> C conversion of numeric types In-Reply-To: References: <2A716CBD-4D2E-488A-8FEF-AE14AA688C7A@math.washington.edu> Message-ID: On Mar 8, 2009, at 5:13 PM, Lisandro Dalcin wrote: > On Sun, Mar 8, 2009 at 3:41 AM, Robert Bradshaw > wrote: >> On Feb 26, 2009, at 10:52 AM, Lisandro Dalcin wrote: >>> >>> [2] seems fine, but what about [3] ?? Should we specialize the >>> method >>> and make it return 'int' ?? >> >> Yes. Actually, isn't "sign_and_name" supposed to return the actual >> name of the enum? >> > > But an anon enum has no name! Oh, now I see what you're asking about. Yes, let's just return "int" in this special case. - Robert From nicdumz at gmail.com Thu Mar 19 07:35:51 2009 From: nicdumz at gmail.com (Nicolas Dumazet) Date: Thu, 19 Mar 2009 15:35:51 +0900 Subject: [Cython] Can't build, what am I doing wrong ? =) Message-ID: <1f077e770903182335i5d718443v3e980be86b3f892f@mail.gmail.com> Hello Cython community! Following the announcement of accepted organizations for the Google Summer of Code of this year, I discovered cython =) Curious about this project, I wanted to take a look into it. After reading http://wiki.cython.org/HackerGuide I checked out cython-devel branch and tried building it. Following INSTALL instructions, I tried "python setup.py". This, however seems to fail when building 'Cython.Plex.Scanners' extension : Plex/Scanners.c can't find Python.h nor structmember.h for inclusion, and building fails. I first thought that the dev branch was broken at the moment, and tried building cython main branch instead, but got the same errors =) What did I misunderstand? ^_^ My current box is an Unix system. Please ask if you need more input from me! Thanks! -- Nicolas Dumazet ? NicDumZ [ n?k.d?ymz ] From nicdumz at gmail.com Thu Mar 19 07:41:00 2009 From: nicdumz at gmail.com (Nicolas Dumazet) Date: Thu, 19 Mar 2009 15:41:00 +0900 Subject: [Cython] Can't build, what am I doing wrong ? =) In-Reply-To: <1f077e770903182335i5d718443v3e980be86b3f892f@mail.gmail.com> References: <1f077e770903182335i5d718443v3e980be86b3f892f@mail.gmail.com> Message-ID: <1f077e770903182341m68783f93k640e4e6d66e2c95c@mail.gmail.com> psshhht, forget this. Solved, sorry for the naive question =) 2009/3/19 Nicolas Dumazet : > Hello Cython community! > > Following the announcement of accepted organizations for the Google > Summer of Code of this year, I discovered cython =) > > Curious about this project, I wanted to take a look into it. After > reading http://wiki.cython.org/HackerGuide I checked out cython-devel > branch and tried building it. > > Following INSTALL instructions, I tried "python setup.py". > This, however seems to fail when building 'Cython.Plex.Scanners' > extension : Plex/Scanners.c can't find Python.h nor structmember.h for > inclusion, and building fails. > > I first thought that the dev branch was broken at the moment, and > tried building cython main branch instead, but got the same errors =) > > What did I misunderstand? ^_^ > > My current box is an Unix system. Please ask if you need more input from me! > > Thanks! > -- > Nicolas Dumazet ? NicDumZ [ n?k.d?ymz ] > -- Nicolas Dumazet ? NicDumZ [ n?k.d?ymz ] From robertwb at math.washington.edu Thu Mar 19 07:50:08 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 18 Mar 2009 23:50:08 -0700 Subject: [Cython] Can't build, what am I doing wrong ? =) In-Reply-To: <1f077e770903182341m68783f93k640e4e6d66e2c95c@mail.gmail.com> References: <1f077e770903182335i5d718443v3e980be86b3f892f@mail.gmail.com> <1f077e770903182341m68783f93k640e4e6d66e2c95c@mail.gmail.com> Message-ID: <0D271A1B-2AC9-4D40-904B-5ACAB795EA7E@math.washington.edu> Let me guess, you didn't have a development version of Python (with the requisite headers)? In any case, welcome to Cython! On Mar 18, 2009, at 11:41 PM, Nicolas Dumazet wrote: > psshhht, forget this. Solved, sorry for the naive question =) > > 2009/3/19 Nicolas Dumazet : >> Hello Cython community! >> >> Following the announcement of accepted organizations for the Google >> Summer of Code of this year, I discovered cython =) >> >> Curious about this project, I wanted to take a look into it. After >> reading http://wiki.cython.org/HackerGuide I checked out cython-devel >> branch and tried building it. >> >> Following INSTALL instructions, I tried "python setup.py". >> This, however seems to fail when building 'Cython.Plex.Scanners' >> extension : Plex/Scanners.c can't find Python.h nor structmember.h >> for >> inclusion, and building fails. >> >> I first thought that the dev branch was broken at the moment, and >> tried building cython main branch instead, but got the same errors =) >> >> What did I misunderstand? ^_^ >> >> My current box is an Unix system. Please ask if you need more >> input from me! >> >> Thanks! >> -- >> Nicolas Dumazet ? NicDumZ [ n?k.d?ymz ] >> > > > > -- > Nicolas Dumazet ? NicDumZ [ n?k.d?ymz ] > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From m.gysel at gmail.com Thu Mar 19 08:44:06 2009 From: m.gysel at gmail.com (Martin Gysel) Date: Thu, 19 Mar 2009 08:44:06 +0100 Subject: [Cython] callback/segfault In-Reply-To: References: <49C164EC.9040807@gmail.com> Message-ID: <49C1F7C6.6090506@gmail.com> Robert Bradshaw schrieb: > On Mar 18, 2009, at 2:17 PM, Martin Gysel wrote: > > >> Hi >> >> I have a 'cython' class which interfaces c code. therefor I need some >> callback functions which I declared outside of the class. Over a >> userdata argument pointer this function gets the reference of my >> class. >> Callback registration and also the callback itself work. The >> problem is, >> as soon as the callback starts to execute, I'll get a segfault (from >> time to time it also works...). So I debugged the c code, the segfault >> occurs while getting the sdtout for the print statement. So I removed >> all print statements but from my callback I want to call either a >> function of my class or some python callback (which needs to be >> registered first). For me it seems the segfault occurs when accessing >> some python objects. I also tried to play around with 'which gil' and >> 'nogil' but no luck. When using 'which gil' it segfaults >> immediately the >> callback gets called. >> What do I wrong? I assume it should be possible to have print in >> callback functions and also to have access to python objects. >> > > Yes, that should be possible. It does sound like you're having > threading/gil issues. What is your callback function signature? When > you say you've tried "with gil" are you doing > > cdef void my_callback(void *data) with gil: > ... > yes exactly (well some more arguments...). I forgot to mention that the callback gets called from another thread as the library which I want to interface starts such a thread. Probably thats the point. But how can I assure that another thread can safely call my code? /martin From robertwb at math.washington.edu Thu Mar 19 08:49:14 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 19 Mar 2009 00:49:14 -0700 Subject: [Cython] callback/segfault In-Reply-To: <49C1F7C6.6090506@gmail.com> References: <49C164EC.9040807@gmail.com> <49C1F7C6.6090506@gmail.com> Message-ID: <8D564F47-9FAA-43FE-8944-10F89269F0CA@math.washington.edu> On Mar 19, 2009, at 12:44 AM, Martin Gysel wrote: > Robert Bradshaw schrieb: >> On Mar 18, 2009, at 2:17 PM, Martin Gysel wrote: >> >> >>> Hi >>> >>> I have a 'cython' class which interfaces c code. therefor I need >>> some >>> callback functions which I declared outside of the class. Over a >>> userdata argument pointer this function gets the reference of my >>> class. >>> Callback registration and also the callback itself work. The >>> problem is, >>> as soon as the callback starts to execute, I'll get a segfault (from >>> time to time it also works...). So I debugged the c code, the >>> segfault >>> occurs while getting the sdtout for the print statement. So I >>> removed >>> all print statements but from my callback I want to call either a >>> function of my class or some python callback (which needs to be >>> registered first). For me it seems the segfault occurs when >>> accessing >>> some python objects. I also tried to play around with 'which gil' >>> and >>> 'nogil' but no luck. When using 'which gil' it segfaults >>> immediately the >>> callback gets called. >>> What do I wrong? I assume it should be possible to have print in >>> callback functions and also to have access to python objects. >>> >> >> Yes, that should be possible. It does sound like you're having >> threading/gil issues. What is your callback function signature? When >> you say you've tried "with gil" are you doing >> >> cdef void my_callback(void *data) with gil: >> ... >> > yes exactly (well some more arguments...). I forgot to mention that > the > callback gets called from another thread as the library which I > want to > interface starts such a thread. Probably thats the point. But how > can I > assure that another thread can safely call my code? Hmm... I thought that this was the way to do it. I've never written code that has to deal with the gil myself, so hopefully someone more knowledgeable on this particular topic will speak up. - Robert From stefan_ml at behnel.de Thu Mar 19 09:11:18 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 19 Mar 2009 09:11:18 +0100 (CET) Subject: [Cython] callback/segfault In-Reply-To: <49C1F7C6.6090506@gmail.com> References: <49C164EC.9040807@gmail.com> <49C1F7C6.6090506@gmail.com> Message-ID: <36656.213.61.181.86.1237450278.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Martin Gysel wrote: > Robert Bradshaw schrieb: >> On Mar 18, 2009, at 2:17 PM, Martin Gysel wrote: >> >> >>> Hi >>> >>> I have a 'cython' class which interfaces c code. therefor I need some >>> callback functions which I declared outside of the class. Over a >>> userdata argument pointer this function gets the reference of my >>> class. >>> Callback registration and also the callback itself work. The >>> problem is, >>> as soon as the callback starts to execute, I'll get a segfault (from >>> time to time it also works...). So I debugged the c code, the segfault >>> occurs while getting the sdtout for the print statement. So I removed >>> all print statements but from my callback I want to call either a >>> function of my class or some python callback (which needs to be >>> registered first). For me it seems the segfault occurs when accessing >>> some python objects. I also tried to play around with 'which gil' and >>> 'nogil' but no luck. When using 'which gil' it segfaults >>> immediately the >>> callback gets called. >>> What do I wrong? I assume it should be possible to have print in >>> callback functions and also to have access to python objects. >>> >> >> Yes, that should be possible. It does sound like you're having >> threading/gil issues. What is your callback function signature? When >> you say you've tried "with gil" are you doing >> >> cdef void my_callback(void *data) with gil: >> ... >> > yes exactly (well some more arguments...). I forgot to mention that the > callback gets called from another thread as the library which I want to > interface starts such a thread. Probably thats the point. But how can I > assure that another thread can safely call my code? The above should work. Can you tell us a bit more about the C code that calls your callback? Is there a direct call chain like your code -> C function with callback pointer -> ... -> callback -> your code ... <- your code or do you just register your callback and it gets called at some later point? In the first case, you need to release the GIL by wrapping the C function call in a "with nogil:" block, so that the callback can acquire it. The second case should just work, but you have to make sure that you release the GIL from time to time to allow the callback to run (the Python interpreter does that, but only when it's running Python code). Stefan From dagss at student.matnat.uio.no Thu Mar 19 09:56:34 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 19 Mar 2009 09:56:34 +0100 Subject: [Cython] numpy calls without Python overhead? In-Reply-To: References: Message-ID: <49C208C2.3060209@student.matnat.uio.no> Amir wrote: > The end of the numpy section of the cython docs suggests the possibility to call > numpy/scipy functions without Python call overhead. How can this be done? > > A test script at bottom is 1.8 times faster when I expand numpy calls into > simple for loops (n,m = 1000,1500). weave.inline is 2.7 times faster. Looking at > the cython -a output, not sure where most of that time is lost. Looks like > strides generate many more calls and dot products are done using Python calls > for multiplications, for example. Yes, unfortunately that's what the status is; the only thing that is optimized by Cython is element indexing (i.e. your theta[j] and v[j]). This is where you'd really remove a bottleneck in some code, but it means that "mixed" code like yours doesn't benefit that much. Remember though that in your case, as n and m goes to infinity, the Python overhead will be rather small. If you want to, you could have a look at calling e.g. dot using the NumPy C API defined in the NumPy header files. Then you can supply an implementation in Cython/Includes/numpy.pxd like this: cdef inline dot(a, b): ... Then that would remove a (quite small) part of the overhead. Further improvements would require changes to NumPy. A full implementation of things alluded to in http://wiki.cython.org/enhancements/buffersyntax could potentially fix this as well, but the status is still uncertain and the timeframe if it happens about a year from now. -- Dag Sverre From m.gysel at gmail.com Thu Mar 19 10:08:52 2009 From: m.gysel at gmail.com (Martin Gysel) Date: Thu, 19 Mar 2009 10:08:52 +0100 Subject: [Cython] callback/segfault In-Reply-To: <36656.213.61.181.86.1237450278.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49C164EC.9040807@gmail.com> <49C1F7C6.6090506@gmail.com> <36656.213.61.181.86.1237450278.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49C20BA4.6070800@gmail.com> Stefan Behnel schrieb: > Martin Gysel wrote: > >> Robert Bradshaw schrieb: >> >>> On Mar 18, 2009, at 2:17 PM, Martin Gysel wrote: >>> >>> >>> >>>> Hi >>>> >>>> I have a 'cython' class which interfaces c code. therefor I need some >>>> callback functions which I declared outside of the class. Over a >>>> userdata argument pointer this function gets the reference of my >>>> class. >>>> Callback registration and also the callback itself work. The >>>> problem is, >>>> as soon as the callback starts to execute, I'll get a segfault (from >>>> time to time it also works...). So I debugged the c code, the segfault >>>> occurs while getting the sdtout for the print statement. So I removed >>>> all print statements but from my callback I want to call either a >>>> function of my class or some python callback (which needs to be >>>> registered first). For me it seems the segfault occurs when accessing >>>> some python objects. I also tried to play around with 'which gil' and >>>> 'nogil' but no luck. When using 'which gil' it segfaults >>>> immediately the >>>> callback gets called. >>>> What do I wrong? I assume it should be possible to have print in >>>> callback functions and also to have access to python objects. >>>> >>>> >>> Yes, that should be possible. It does sound like you're having >>> threading/gil issues. What is your callback function signature? When >>> you say you've tried "with gil" are you doing >>> >>> cdef void my_callback(void *data) with gil: >>> ... >>> >>> >> yes exactly (well some more arguments...). I forgot to mention that the >> callback gets called from another thread as the library which I want to >> interface starts such a thread. Probably thats the point. But how can I >> assure that another thread can safely call my code? >> > > The above should work. Can you tell us a bit more about the C code that > calls your callback? Is there a direct call chain like > > your code > -> C function with callback pointer > -> ... > -> callback > -> your code > ... > <- > your code > > or do you just register your callback and it gets called at some later point? > > In the first case, you need to release the GIL by wrapping the C function > call in a "with nogil:" block, so that the callback can acquire it. The > second case should just work, but you have to make sure that you release > the GIL from time to time to allow the callback to run (the Python > interpreter does that, but only when it's running Python code). > It's the second case, I register it and it gets called at some time. As already said, the library starts a new thread... at the time the callback gets called no cython code is supposed to run except the callback itself of course. 1. register callback 2. start library thread 3. ... (at the moment I just do nothing apart of waiting) 4. callback gets called (here I want to deal with either the library or my cython object) 5. segfault :-( really strange for me is it segfaults immediately if I use with gil. is it possible the gil is held by another function? /martin From stefan_ml at behnel.de Thu Mar 19 10:26:58 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 19 Mar 2009 10:26:58 +0100 (CET) Subject: [Cython] callback/segfault In-Reply-To: <49C20BA4.6070800@gmail.com> References: <49C164EC.9040807@gmail.com> <49C1F7C6.6090506@gmail.com> <36656.213.61.181.86.1237450278.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C20BA4.6070800@gmail.com> Message-ID: <57265.213.61.181.86.1237454818.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Martin Gysel wrote: > Stefan Behnel wrote: >> do you just register your callback and it gets called at some later >> point? > > I register it and it gets called at some time. As > already said, the library starts a new thread... at the time the > callback gets called no cython code is supposed to run except the > callback itself of course. > > 1. register callback > 2. start library thread > 3. ... (at the moment I just do nothing apart of waiting) How do you wait? using time.wait()? > 4. callback gets called (here I want to deal with either the library or > my cython object) > 5. segfault :-( > > really strange for me is it segfaults immediately if I use with gil. is > it possible the gil is held by another function? No. The GIL is an exclusive lock. When you get hold of it, you are the only one who has it. Declaring your callback function "with gil" is exactly the right thing to do here. BTW, are you sure that threading support has been initialised in your program? Just a wild guess, but maybe "import threading" at module level helps? If so, we may be able to fix that. It would be good if you could come up with a small test program that shows exactly the code you use. Also, so far, we do not know which library you are interfacing with. Is the library thread based on pthreads (which Python uses)? Stefan From stefan_ml at behnel.de Thu Mar 19 08:37:50 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 19 Mar 2009 08:37:50 +0100 Subject: [Cython] Cython header ordering bug (forwarded from Ryan Hinton) In-Reply-To: <6296B2E7-DDAB-4756-9B69-A1E78902B64B@math.washington.edu> References: <49C12DD9.6040002@behnel.de> <6296B2E7-DDAB-4756-9B69-A1E78902B64B@math.washington.edu> Message-ID: <49C1F64E.1090300@behnel.de> Robert Bradshaw wrote: > I've toyed with the idea of making it output much less verbose but > still unique names, but I'm used to it by now. In any case, the C > output isn't meant to be read by mere mortals... :). Oh, but don't tell them that. I still stick to saying "generating readable C code is a goal of the Cython project", mostly in response to people saying that other projects (namely PyBindGen) generate exactly the code that a programmer would write (except that writing a plain wrapper isn't really something a programmer would want to write, but I rarely add that comment in order to avoid sounding rude ;). Stefan From stefan_ml at behnel.de Thu Mar 19 11:40:08 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 19 Mar 2009 11:40:08 +0100 Subject: [Cython] PATH: force threads initialization for Py2.3, please review In-Reply-To: References: Message-ID: <49C22108.5010400@behnel.de> Lisandro Dalcin wrote: > Some time ago, I've posted some mails about thread issues in Python > 2.3. Unfortunately, those post did not receive any attention. > > Currently, mpi4py on Python 2.3 makes the interpreter segfault. I > could easily fix this issue myself for mpi4py, but perhaps it do make > sense to alleviate other Py2.3 users of this really low-level issue. > > The problem: If at the point PyGILState_Release() is called, the GIL > whas not yet created, then the Python (2.3) interpreter segfautls. > > The solution: Call PyEval_InitThread() in the module init function. > But this could have performance impacts for non threaded applications > using a Cython-generated extension module, so this thread > initialization should be done only if needed. A simple approach is to > call PyEval_InitThread() only if the Cython module ever > released/aquired the GIL > > The attached patch implement all the ideas discussed above. Any chance > to this being accepted? Any better idea? Note that PyEval_InitThread() > is a no-op if the GIL is already created. Just a quick note that I applied your patch. http://hg.cython.org/cython-devel/rev/9a1f6f7c260a http://hg.cython.org/cython-devel/rev/0bd3bfd487a5 Stefan From greg.ewing at canterbury.ac.nz Thu Mar 19 12:10:58 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 19 Mar 2009 23:10:58 +1200 Subject: [Cython] loops In-Reply-To: <6B1E8348-1E34-4218-90CB-58F521A43C22@math.washington.edu> References: <49C158EE.6000502@student.matnat.uio.no> <49C162DF.7020708@canterbury.ac.nz> <6B1E8348-1E34-4218-90CB-58F521A43C22@math.washington.edu> Message-ID: <49C22842.7030205@canterbury.ac.nz> Robert Bradshaw wrote: > What about the behavior of > > for i from 0 <= i < 10: > print i > i += 5 Undefined also. -- Greg From dagss at student.matnat.uio.no Thu Mar 19 14:19:53 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 19 Mar 2009 14:19:53 +0100 Subject: [Cython] constant handling refactored and up on cython-unstable (ticket 144) In-Reply-To: <49C15F04.2020504@behnel.de> References: <49C15F04.2020504@behnel.de> Message-ID: <49C24679.9080300@student.matnat.uio.no> Stefan Behnel wrote: > Hi, > > I just pushed the constant allocation refactoring to cython-unstable. There > are still a few failing test cases, especially regarding struct coercion. > > The main problem was that some places in the code expected the result code > to be available for constants during the analysis phase. Where I could not > move it to the code generation phase, I had to replace this with a call to > "compile_time_value()" for now, not sure if this is the right thing to > do... The change can clearly benefit from some review and some additional > bug fixing. I hope someone beats me to it. :) > Unfortunately I don't think I'll beat you to it; I just wanted to say THANK YOU for doing this! It looks very nice. (The places where result_code is used during analysis should go anyway; after the temp refactoring is complete we'll hopefully remove result_code from pre-code-generation entirely.) Dag Sverre From dalcinl at gmail.com Thu Mar 19 14:34:43 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 19 Mar 2009 10:34:43 -0300 Subject: [Cython] PATH: force threads initialization for Py2.3, please review In-Reply-To: <49C22108.5010400@behnel.de> References: <49C22108.5010400@behnel.de> Message-ID: On Thu, Mar 19, 2009 at 7:40 AM, Stefan Behnel wrote: > > Lisandro Dalcin wrote: >> Some time ago, I've posted some mails about thread issues in Python >> 2.3. Unfortunately, those post did not receive any attention. >> > > Just a quick note that I applied your patch. > > http://hg.cython.org/cython-devel/rev/9a1f6f7c260a > http://hg.cython.org/cython-devel/rev/0bd3bfd487a5 > Ok. It seems the bug is also in some 2.4 releases... Then , two comments: * I would suggest changing the name of the utility code from "py23_init_threads_utility_code" to "force_init_threads_utility_code"... Just to avoid confusion. * When I wrote the patch, I took the precaution of protecting "#ifndef __PYX_FORCE_INIT_THREADS" just in case some users would have the need to decide themselves what to do... If this feature is maintained (I would), then perhaps the macro should have a more "public" name, like "CYTHON_FORCE_INIT_THREADS"... Please note that I'm +0 on this, it is unlikely that users will need tho hack on this... So, I'll leave the final decision on your hands ... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Thu Mar 19 14:57:05 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 19 Mar 2009 10:57:05 -0300 Subject: [Cython] improving Py <-> C conversion of numeric types In-Reply-To: References: <2A716CBD-4D2E-488A-8FEF-AE14AA688C7A@math.washington.edu> Message-ID: On Thu, Mar 19, 2009 at 3:18 AM, Robert Bradshaw wrote: >> >> But an anon enum has no name! > > Oh, now I see what you're asking about. Yes, let's just return "int" > in this special case. > Let's leave this for the time I restart my work on C<->Py conversions of integral types... Anon enums are currently the only way to enforce "constness" on external declararions... I have fear of braking user code. This is yet other more thing we have to improve... "const" keyword should be accepted and handled at least in external declarations, in such a way that we can write cdef extern from ""someheader.h" const long LONG_MAX const size_t SIZEOF_SOMETHING const double PI or perhaps: cdef extern from ""someheader.h" const: long LONG_MAX size_t SIZEOF_SOMETHING double PI Unfortunately, I doubt I could implement this myself... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Thu Mar 19 15:01:12 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 19 Mar 2009 15:01:12 +0100 Subject: [Cython] improving Py <-> C conversion of numeric types In-Reply-To: References: <2A716CBD-4D2E-488A-8FEF-AE14AA688C7A@math.washington.edu> Message-ID: <49C25028.4020203@student.matnat.uio.no> Lisandro Dalcin wrote: > On Thu, Mar 19, 2009 at 3:18 AM, Robert Bradshaw > wrote: > >>> But an anon enum has no name! >>> >> Oh, now I see what you're asking about. Yes, let's just return "int" >> in this special case. >> >> > > Let's leave this for the time I restart my work on C<->Py conversions > of integral types... > > Anon enums are currently the only way to enforce "constness" on > external declararions... I have fear of braking user code. > > This is yet other more thing we have to improve... "const" keyword > should be accepted and handled at least in external declarations, in > such a way that we can write > When/if this is done, we should also add support for the C99 "restrict" keyword in my opinion. Dag Sverre From dsurviver at gmail.com Thu Mar 19 15:07:05 2009 From: dsurviver at gmail.com (Danilo Freitas) Date: Thu, 19 Mar 2009 11:07:05 -0300 Subject: [Cython] [HELP-INSTALL] Problem with setup.py Message-ID: <45239150903190707l7f3ab2f7k631af5a9fd467a4e@mail.gmail.com> Hi. I'm trying to install Cython here, but I'm with problems. When I put "python setup.py install" on terminal, it begins the installation, but then some problems occurs. When in file "Cython/Plex/Scanners.c" it gets lots of syntax errors on file, and I think it shouldn't occur. How can I fix this problem? Thanks Danilo From stefan_ml at behnel.de Thu Mar 19 15:15:25 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 19 Mar 2009 15:15:25 +0100 (CET) Subject: [Cython] constant handling refactored and up on cython-unstable (ticket 144) In-Reply-To: <49C24679.9080300@student.matnat.uio.no> References: <49C15F04.2020504@behnel.de> <49C24679.9080300@student.matnat.uio.no> Message-ID: <65193.213.61.181.86.1237472125.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > The places where result_code is used during analysis should go anyway; I agree. It's not trivial in all cases, though. Cython has a couple of optimisations and enhancements that rely on replacing a node by a different one, which may then need a constant value at creation time in some cases. The exception value that is used in CFuncDeclaratorNode to set up the function type is one example that I'm currently trying to work around. Another example is IntNode, which is expected to return a plain constant result all the time, but actually requires the compile time code writer to cache Python integers before it knows its result code. This would be easier to solve if the pipeline went type annotation -> constant folding -> type analysis instead of the current type annotation and analysis -> constant folding as it would allow us to just read the constant value in a generic way. I've been lobbying for this for a while now, but I must admit that its a lot of work with low priority. It could also be argued that most modifications to the tree that create new nodes should run in a transform instead of type analysis anyway, and could thus get moved behind the constant folding step. BTW, Pyrex also still uses "result()" in a couple of places during the analysis phase. It's just luckily escaping crashes because integer constants are really just plain string valued C constants there. > after the temp refactoring is complete we'll hopefully remove > result_code from pre-code-generation entirely. Sure thing, a clear 0.12 goal. Stefan From stefan_ml at behnel.de Thu Mar 19 15:19:09 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 19 Mar 2009 15:19:09 +0100 (CET) Subject: [Cython] [HELP-INSTALL] Problem with setup.py In-Reply-To: <45239150903190707l7f3ab2f7k631af5a9fd467a4e@mail.gmail.com> References: <45239150903190707l7f3ab2f7k631af5a9fd467a4e@mail.gmail.com> Message-ID: <36098.213.61.181.86.1237472349.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Danilo Freitas wrote: > I'm trying to install Cython here, but I'm with problems. "here" being what kind of system and Python version? > When I put "python setup.py install" on terminal, it begins the > installation, but then some problems occurs. > > When in file "Cython/Plex/Scanners.c" it gets lots of syntax errors on > file, and I think it shouldn't occur. You can either install a (slower) pure Python version of Cython by passing --no-cython-compile to setup.py, or you can install the Python development files (i.e. header files etc.) so that you can build the faster binary version. Note that you will need them anyway if you want to use Cython (to build extension modules). Stefan From dagss at student.matnat.uio.no Thu Mar 19 15:23:59 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 19 Mar 2009 15:23:59 +0100 Subject: [Cython] constant handling refactored and up on cython-unstable (ticket 144) In-Reply-To: <65193.213.61.181.86.1237472125.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49C15F04.2020504@behnel.de> <49C24679.9080300@student.matnat.uio.no> <65193.213.61.181.86.1237472125.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49C2557F.3090402@student.matnat.uio.no> Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: > >> The places where result_code is used during analysis should go anyway; >> > > I agree. It's not trivial in all cases, though. Cython has a couple of > optimisations and enhancements that rely on replacing a node by a > different one, which may then need a constant value at creation time in > some cases. The exception value that is used in CFuncDeclaratorNode to set > up the function type is one example that I'm currently trying to work > around. > > Another example is IntNode, which is expected to return a plain constant > result all the time, but actually requires the compile time code writer to > cache Python integers before it knows its result code. This would be > easier to solve if the pipeline went > > type annotation -> constant folding -> type analysis > > instead of the current > > type annotation and analysis -> constant folding > Is it impossible for constant folding to happen before type annotation? After all, the type should be hard-coded in the class for constants? (Or at least, a seperate "type constants" phase first is much easier than a full refactor here.) Dag Sverre From dsurviver at gmail.com Thu Mar 19 15:27:08 2009 From: dsurviver at gmail.com (Danilo Freitas) Date: Thu, 19 Mar 2009 11:27:08 -0300 Subject: [Cython] [HELP-INSTALL] Problem with setup.py In-Reply-To: <36098.213.61.181.86.1237472349.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <45239150903190707l7f3ab2f7k631af5a9fd467a4e@mail.gmail.com> <36098.213.61.181.86.1237472349.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <45239150903190727o6a6ade2ei118bd8729a9845c7@mail.gmail.com> I'm on Ubuntu 8.04 and Python 2.5, to be more specific I used the --no-cython-compile and it did with no problems. But, I can normally use Cython this way? Thanks Danilo 2009/3/19 Stefan Behnel : > Danilo Freitas wrote: >> I'm trying to install Cython here, but I'm with problems. > > "here" being what kind of system and Python version? > > >> When I put "python setup.py install" on terminal, it begins the >> installation, but then some problems occurs. >> >> When in file "Cython/Plex/Scanners.c" it gets lots of syntax errors on >> file, and I think it shouldn't occur. > > You can either install a (slower) pure Python version of Cython by passing > > ? ?--no-cython-compile > > to setup.py, or you can install the Python development files (i.e. header > files etc.) so that you can build the faster binary version. Note that you > will need them anyway if you want to use Cython (to build extension > modules). > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From nicdumz at gmail.com Thu Mar 19 15:35:07 2009 From: nicdumz at gmail.com (Nicolas Dumazet) Date: Thu, 19 Mar 2009 23:35:07 +0900 Subject: [Cython] [HELP-INSTALL] Problem with setup.py In-Reply-To: <45239150903190727o6a6ade2ei118bd8729a9845c7@mail.gmail.com> References: <45239150903190707l7f3ab2f7k631af5a9fd467a4e@mail.gmail.com> <36098.213.61.181.86.1237472349.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <45239150903190727o6a6ade2ei118bd8729a9845c7@mail.gmail.com> Message-ID: <1f077e770903190735o2ef90ba8g137af6be6e42b0f7@mail.gmail.com> Hello Danilo! sudo apt-get install python-dev should fix your problem =) 2009/3/19 Danilo Freitas : > I'm on Ubuntu 8.04 and Python 2.5, to be more specific > > I used the --no-cython-compile and it did with no problems. > But, I can normally use Cython this way? > > Thanks > > Danilo > > 2009/3/19 Stefan Behnel : >> Danilo Freitas wrote: >>> I'm trying to install Cython here, but I'm with problems. >> >> "here" being what kind of system and Python version? >> >> >>> When I put "python setup.py install" on terminal, it begins the >>> installation, but then some problems occurs. >>> >>> When in file "Cython/Plex/Scanners.c" it gets lots of syntax errors on >>> file, and I think it shouldn't occur. >> >> You can either install a (slower) pure Python version of Cython by passing >> >> ? ?--no-cython-compile >> >> to setup.py, or you can install the Python development files (i.e. header >> files etc.) so that you can build the faster binary version. Note that you >> will need them anyway if you want to use Cython (to build extension >> modules). >> >> Stefan >> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Nicolas Dumazet ? NicDumZ [ n?k.d?ymz ] From dsurviver at gmail.com Thu Mar 19 16:23:04 2009 From: dsurviver at gmail.com (Danilo Freitas) Date: Thu, 19 Mar 2009 12:23:04 -0300 Subject: [Cython] [HELP-INSTALL] Problem with setup.py In-Reply-To: <1f077e770903190735o2ef90ba8g137af6be6e42b0f7@mail.gmail.com> References: <45239150903190707l7f3ab2f7k631af5a9fd467a4e@mail.gmail.com> <36098.213.61.181.86.1237472349.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <45239150903190727o6a6ade2ei118bd8729a9845c7@mail.gmail.com> <1f077e770903190735o2ef90ba8g137af6be6e42b0f7@mail.gmail.com> Message-ID: <45239150903190823k63651252h220f8306a5a99f27@mail.gmail.com> Thanks a lot, Nicolas! I did it and installed normally :D thanks you two Danilo 2009/3/19 Nicolas Dumazet : > Hello Danilo! > > sudo apt-get install python-dev > > should fix your problem =) > > 2009/3/19 Danilo Freitas : >> I'm on Ubuntu 8.04 and Python 2.5, to be more specific >> >> I used the --no-cython-compile and it did with no problems. >> But, I can normally use Cython this way? >> >> Thanks >> >> Danilo >> >> 2009/3/19 Stefan Behnel : >>> Danilo Freitas wrote: >>>> I'm trying to install Cython here, but I'm with problems. >>> >>> "here" being what kind of system and Python version? >>> >>> >>>> When I put "python setup.py install" on terminal, it begins the >>>> installation, but then some problems occurs. >>>> >>>> When in file "Cython/Plex/Scanners.c" it gets lots of syntax errors on >>>> file, and I think it shouldn't occur. >>> >>> You can either install a (slower) pure Python version of Cython by passing >>> >>> ? ?--no-cython-compile >>> >>> to setup.py, or you can install the Python development files (i.e. header >>> files etc.) so that you can build the faster binary version. Note that you >>> will need them anyway if you want to use Cython (to build extension >>> modules). >>> >>> Stefan >>> >>> _______________________________________________ >>> Cython-dev mailing list >>> Cython-dev at codespeak.net >>> http://codespeak.net/mailman/listinfo/cython-dev >>> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > > > -- > Nicolas Dumazet ? NicDumZ [ n?k.d?ymz ] > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From stefan_ml at behnel.de Thu Mar 19 16:33:40 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 19 Mar 2009 16:33:40 +0100 (CET) Subject: [Cython] constant handling refactored and up on cython-unstable (ticket 144) In-Reply-To: <49C2557F.3090402@student.matnat.uio.no> References: <49C15F04.2020504@behnel.de> <49C24679.9080300@student.matnat.uio.no> <65193.213.61.181.86.1237472125.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C2557F.3090402@student.matnat.uio.no> Message-ID: <49093.213.61.181.86.1237476820.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > Is it impossible for constant folding to happen before type annotation? > After all, the type should be hard-coded in the class for constants? Not sure. At least, it needs scope and target type information in its current way of working, as it does type coercions. It might be possible to remove that completely, and to determine the widest target type of two operands instead. Want to have fun giving it a try? :) > Or at least, a seperate "type constants" phase first is much easier > than a full refactor here. Let's try first if the default type associated with a constant is enough for plain constant folding. BTW, I'm not sure how type casts should be handled. What about something like this: cdef int x = 3 + (2**10) ? Whereas cdef int x = 3 + (2**10) would simplify just fine... Stefan From dagss at student.matnat.uio.no Thu Mar 19 16:46:57 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 19 Mar 2009 16:46:57 +0100 Subject: [Cython] constant handling refactored and up on cython-unstable (ticket 144) In-Reply-To: <49093.213.61.181.86.1237476820.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49C15F04.2020504@behnel.de> <49C24679.9080300@student.matnat.uio.no> <65193.213.61.181.86.1237472125.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C2557F.3090402@student.matnat.uio.no> <49093.213.61.181.86.1237476820.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49C268F1.2010704@student.matnat.uio.no> Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: > >> Is it impossible for constant folding to happen before type annotation? >> After all, the type should be hard-coded in the class for constants? >> > > Not sure. At least, it needs scope and target type information in its > current way of working, as it does type coercions. It might be possible to > remove that completely, and to determine the widest target type of two > operands instead. Want to have fun giving it a try? :) > Unfortunately email activity == procrastination != real opportunity for Cython activity; at least for the rest of this week. (And unfortunately procrastination doesn't work the way that one can simply "bottle it up" and use it for real coding :-) ). (I think that when I get a peaceful Cython moment, I'll focus on e.g. getting more nodes over to new temps; the constant folding has a learning curve and I should focus on the 0.12 stuff which doesn't involve a learning curve first; more effective use of manpower that way.) > > BTW, I'm not sure how type casts should be handled. What about something > like this: > > cdef int x = 3 + (2**10) > This is asking for a wraparound if you ask me. I'd say 2**10 folds, then leave it to runtime, because we cannot know the type sizes (at least of long). > ? Whereas > > cdef int x = 3 + (2**10) > > would simplify just fine... > However, cdef long x = 3 + (2**34) would simplify fine on non-Windows 64-bit but not otherwise. I'd be reluctant to simplify casts at all because the only intention I can find in code like that is to do a "& MAX_VALUE_OF_TYPE", and that stuff is much better left to the compiler. Dag Sverre From stefan_ml at behnel.de Thu Mar 19 16:55:45 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 19 Mar 2009 16:55:45 +0100 (CET) Subject: [Cython] constant handling refactored and up on cython-unstable (ticket 144) In-Reply-To: <49C268F1.2010704@student.matnat.uio.no> References: <49C15F04.2020504@behnel.de> <49C24679.9080300@student.matnat.uio.no> <65193.213.61.181.86.1237472125.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C2557F.3090402@student.matnat.uio.no> <49093.213.61.181.86.1237476820.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C268F1.2010704@student.matnat.uio.no> Message-ID: <40298.213.61.181.86.1237478145.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Dag Sverre Seljebotn wrote: >> >>> Is it impossible for constant folding to happen before type annotation? >>> After all, the type should be hard-coded in the class for constants? >> >> Not sure. At least, it needs scope and target type information in its >> current way of working, as it does type coercions. It might be possible >> to >> remove that completely, and to determine the widest target type of two >> operands instead. Want to have fun giving it a try? :) >> > Unfortunately email activity == procrastination != real opportunity for > Cython activity; at least for the rest of this week. (And unfortunately > procrastination doesn't work the way that one can simply "bottle it up" > and use it for real coding :-) ). > > (I think that when I get a peaceful Cython moment, I'll focus on e.g. > getting more nodes over to new temps; the constant folding has a > learning curve and I should focus on the 0.12 stuff which doesn't > involve a learning curve first; more effective use of manpower that way.) Sure, good idea. >> BTW, I'm not sure how type casts should be handled. What about something >> like this: >> >> cdef int x = 3 + (2**10) >> > This is asking for a wraparound if you ask me. I'd say 2**10 folds, then > leave it to runtime, because we cannot know the type sizes (at least of > long). >> ? Whereas >> >> cdef int x = 3 + (2**10) >> >> would simplify just fine... >> > However, > cdef long x = 3 + (2**34) > > would simplify fine on non-Windows 64-bit but not otherwise. I'd be > reluctant to simplify casts at all because the only intention I can find > in code like that is to do a "& MAX_VALUE_OF_TYPE", and that stuff is > much better left to the compiler. Yep. I also think that it's rather unlikely that users will put a type cast before constants in places where compile time values are actually interesting (such as loop ranges/steps). Stefan From stefan_ml at behnel.de Thu Mar 19 16:58:03 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 19 Mar 2009 16:58:03 +0100 (CET) Subject: [Cython] PATH: force threads initialization for Py2.3, please review In-Reply-To: References: <49C22108.5010400@behnel.de> Message-ID: <42255.213.61.181.86.1237478283.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Lisandro Dalcin wrote: > On Thu, Mar 19, 2009 at 7:40 AM, Stefan Behnel wrote: >> >> Lisandro Dalcin wrote: >>> Some time ago, I've posted some mails about thread issues in Python >>> 2.3. Unfortunately, those post did not receive any attention. >>> >> >> Just a quick note that I applied your patch. >> >> http://hg.cython.org/cython-devel/rev/9a1f6f7c260a >> http://hg.cython.org/cython-devel/rev/0bd3bfd487a5 >> > > Ok. It seems the bug is also in some 2.4 releases... Then , two comments: > > * I would suggest changing the name of the utility code from > "py23_init_threads_utility_code" to > "force_init_threads_utility_code"... Just to avoid confusion. Makes sense. Done. > * When I wrote the patch, I took the precaution of protecting "#ifndef > __PYX_FORCE_INIT_THREADS" just in case some users would have the need > to decide themselves what to do... If this feature is maintained (I > would), then perhaps the macro should have a more "public" name, like > "CYTHON_FORCE_INIT_THREADS"... Please note that I'm +0 on this, it is > unlikely that users will need tho hack on this... So, I'll leave the > final decision on your hands ... I'll leave it as is for now. This is for old Python versions only, so users will most likely not care anyway and instead be happy that it just works. Stefan From poonaatsoc at gmail.com Thu Mar 19 18:35:18 2009 From: poonaatsoc at gmail.com (Anoop Saldanha) Date: Thu, 19 Mar 2009 23:05:18 +0530 Subject: [Cython] IRC In-Reply-To: References: <49BE1093.7080505@student.matnat.uio.no> <49C0AC9F.4080102@behnel.de> <49C0BB93.40503@student.matnat.uio.no> <254C9D4B-AB73-46F2-B32F-4C6BC128F402@math.washington.edu> <49C165D0.8030409@gmail.com> <293F708B-BB62-4E07-9D99-18E50313EF5D@math.washington.edu> Message-ID: On Thu, Mar 19, 2009 at 10:27 AM, Anoop Saldanha wrote: > > > On Thu, Mar 19, 2009 at 3:02 AM, Robert Bradshaw < > robertwb at math.washington.edu> wrote: > >> On Mar 18, 2009, at 2:21 PM, Martin Gysel wrote: >> >> > Anoop Saldanha schrieb: >> >> On Thu, Mar 19, 2009 at 12:34 AM, Robert Bradshaw < >> >> robertwb at math.washington.edu> wrote: >> >> >> >>> On Mar 18, 2009, at 8:27 AM, Anoop Saldanha wrote: >> >>> >> >>>> Also is there an irc channel for cython? >> >>>> >> >>> There was a year ago, but it wasn't very populated back then. >> >>> Perhaps >> >>> there's enough of a community to start one up again. >> >>> >> >>> - Robert >> >>> >> >>> >> >> >> >> Maybe we can have a #cython on freenode. If it is okay, I can >> >> open the >> >> channel or if anyone else wants to open the channel, they can go >> >> ahead and >> >> do so. >> >> >> > +1 if it's called cython or cython-dev or what ever I love the idea... >> >> I think just plain #cython would be nice if we could get it--I'd >> rather not intimidate non-developers from joining in. >> >> - Robert >> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > Will get in touch with Arc, and ask him if he can transfer the > ownership(#cython has already been registered by Arc a year ago, but no one > seems to be using it) to one of us here. > > -- > Regards, > Anoop S. > > That should have gone to the mailing list. I spoke to Arc. He asked one of the cython developers whom he can verify, to mail him, and he will have the ownership of the channel transferred to him. -- Regards, Anoop S. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090319/18acfdf5/attachment.htm From arcriley at gmail.com Thu Mar 19 18:55:51 2009 From: arcriley at gmail.com (Arc Riley) Date: Thu, 19 Mar 2009 13:55:51 -0400 Subject: [Cython] IRC In-Reply-To: References: <49BE1093.7080505@student.matnat.uio.no> <49C0BB93.40503@student.matnat.uio.no> <254C9D4B-AB73-46F2-B32F-4C6BC128F402@math.washington.edu> <49C165D0.8030409@gmail.com> <293F708B-BB62-4E07-9D99-18E50313EF5D@math.washington.edu> Message-ID: > I spoke to Arc. He asked one of the cython developers whom he can verify, > to mail him, and he will have the ownership of the channel transferred to > him. Or have Robert/etc send me a message on IRC. I regged it last year to ensure it wouldn't be squatted before we decided not to go with Cython. You guys have always "owned" the channel on Freenode, the chanserv founder status really has no teeth beside usually making it easier for the freenode staff. I just need to know who I'm transfering it to is actually part of the project. You should of course also register as an organization on www.freenode.net, even if you don't want the @cython/developer/foo masks it helps the freenode staff when there's an official contact point for the group. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090319/893c7fee/attachment.htm From robertwb at math.washington.edu Thu Mar 19 19:36:36 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 19 Mar 2009 11:36:36 -0700 Subject: [Cython] improving Py <-> C conversion of numeric types In-Reply-To: References: <2A716CBD-4D2E-488A-8FEF-AE14AA688C7A@math.washington.edu> Message-ID: On Mar 19, 2009, at 6:57 AM, Lisandro Dalcin wrote: > On Thu, Mar 19, 2009 at 3:18 AM, Robert Bradshaw > wrote: >>> >>> But an anon enum has no name! >> >> Oh, now I see what you're asking about. Yes, let's just return "int" >> in this special case. >> > > Let's leave this for the time I restart my work on C<->Py conversions > of integral types... > > Anon enums are currently the only way to enforce "constness" on > external declararions... I have fear of braking user code. > > This is yet other more thing we have to improve... "const" keyword > should be accepted and handled at least in external declarations, in > such a way that we can write > > cdef extern from ""someheader.h" > const long LONG_MAX > const size_t SIZEOF_SOMETHING > const double PI > > or perhaps: > > cdef extern from ""someheader.h" > const: > long LONG_MAX > size_t SIZEOF_SOMETHING > double PI > > Unfortunately, I doubt I could implement this myself... Yes, this is an oft requested feature. http://trac.cython.org/cython_trac/ticket/246 - Robert From robertwb at math.washington.edu Thu Mar 19 20:01:41 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 19 Mar 2009 12:01:41 -0700 Subject: [Cython] [HELP-INSTALL] Problem with setup.py In-Reply-To: <45239150903190823k63651252h220f8306a5a99f27@mail.gmail.com> References: <45239150903190707l7f3ab2f7k631af5a9fd467a4e@mail.gmail.com> <36098.213.61.181.86.1237472349.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <45239150903190727o6a6ade2ei118bd8729a9845c7@mail.gmail.com> <1f077e770903190735o2ef90ba8g137af6be6e42b0f7@mail.gmail.com> <45239150903190823k63651252h220f8306a5a99f27@mail.gmail.com> Message-ID: I just posted http://trac.cython.org/cython_trac/ticket/247 - Robert On Mar 19, 2009, at 8:23 AM, Danilo Freitas wrote: > Thanks a lot, Nicolas! > I did it and installed normally > :D > > thanks you two > > Danilo > > 2009/3/19 Nicolas Dumazet : >> Hello Danilo! >> >> sudo apt-get install python-dev >> >> should fix your problem =) >> >> 2009/3/19 Danilo Freitas : >>> I'm on Ubuntu 8.04 and Python 2.5, to be more specific >>> >>> I used the --no-cython-compile and it did with no problems. >>> But, I can normally use Cython this way? >>> >>> Thanks >>> >>> Danilo >>> >>> 2009/3/19 Stefan Behnel : >>>> Danilo Freitas wrote: >>>>> I'm trying to install Cython here, but I'm with problems. >>>> >>>> "here" being what kind of system and Python version? >>>> >>>> >>>>> When I put "python setup.py install" on terminal, it begins the >>>>> installation, but then some problems occurs. >>>>> >>>>> When in file "Cython/Plex/Scanners.c" it gets lots of syntax >>>>> errors on >>>>> file, and I think it shouldn't occur. >>>> >>>> You can either install a (slower) pure Python version of Cython >>>> by passing >>>> >>>> --no-cython-compile >>>> >>>> to setup.py, or you can install the Python development files >>>> (i.e. header >>>> files etc.) so that you can build the faster binary version. >>>> Note that you >>>> will need them anyway if you want to use Cython (to build extension >>>> modules). >>>> >>>> Stefan >>>> >>>> _______________________________________________ >>>> Cython-dev mailing list >>>> Cython-dev at codespeak.net >>>> http://codespeak.net/mailman/listinfo/cython-dev >>>> >>> _______________________________________________ >>> Cython-dev mailing list >>> Cython-dev at codespeak.net >>> http://codespeak.net/mailman/listinfo/cython-dev >>> >> >> >> >> -- >> Nicolas Dumazet ? NicDumZ [ n?k.d?ymz ] >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From amirnntp at gmail.com Thu Mar 19 20:57:04 2009 From: amirnntp at gmail.com (Amir) Date: Thu, 19 Mar 2009 19:57:04 +0000 (UTC) Subject: [Cython] numpy calls without Python overhead? References: <49C208C2.3060209@student.matnat.uio.no> Message-ID: Dag Sverre Seljebotn writes: > > Amir wrote: > > A test script at bottom is 1.8 times faster when I expand numpy > > calls into simple for loops (n,m = 1000,1500). weave.inline is 2.7 > > times faster. Looking at the cython -a output, not sure where most > > of that time is lost. Looks like strides generate many more calls > > and dot products are done using Python calls for multiplications, > > for example. > > Yes, unfortunately that's what the status is; the only thing that is > optimized by Cython is element indexing (i.e. your theta[j] and > v[j]). This is where you'd really remove a bottleneck in some code, > but it means that "mixed" code like yours doesn't benefit that much. > > Remember though that in your case, as n and m goes to infinity, the > Python overhead will be rather small. > I see. Well, it's great that it can understand regular numpy code. If I only use pointers to ndarray.data in my inner loop and no buffer striding, I get a more than factor 3 speedup. The only difference in the generated code is __Pyx_BufPtrStrided1d and __Pyx_BufPtrStrided2d calls. These should be very fast. Do these cost that much more than using direct pointers? > If you want to, you could have a look at calling e.g. dot using the > NumPy C API defined in the NumPy header files. Then you can supply > an implementation in Cython/Includes/numpy.pxd like this: > > cdef inline dot(a, b): > ... > > Then that would remove a (quite small) part of the overhead. Further > improvements would require changes to NumPy. The numpy dot (_dotblas.c) has to spend a lot of time checking the types, dimensions, and row/column ordering of the arguments. I could replace the numpy dot with the appropriate cblas call for my particular case. But it seems it would be difficult to make this automatically work in all its generality at compile time by cython. I assume this is what you mean. From stefan_ml at behnel.de Thu Mar 19 22:09:12 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 19 Mar 2009 22:09:12 +0100 Subject: [Cython] [HELP-INSTALL] Problem with setup.py In-Reply-To: References: <45239150903190707l7f3ab2f7k631af5a9fd467a4e@mail.gmail.com> <36098.213.61.181.86.1237472349.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <45239150903190727o6a6ade2ei118bd8729a9845c7@mail.gmail.com> <1f077e770903190735o2ef90ba8g137af6be6e42b0f7@mail.gmail.com> <45239150903190823k63651252h220f8306a5a99f27@mail.gmail.com> Message-ID: <49C2B478.8040407@behnel.de> Robert Bradshaw wrote: > I just posted http://trac.cython.org/cython_trac/ticket/247 I also thought about how to make this clearer, but this is a very smart way of doing it. Stefan From dagss at student.matnat.uio.no Thu Mar 19 22:10:39 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 19 Mar 2009 22:10:39 +0100 (CET) Subject: [Cython] numpy calls without Python overhead? In-Reply-To: References: <49C208C2.3060209@student.matnat.uio.no> Message-ID: <7a2c78052f976519350fb93f939de3ca.squirrel@webmail.uio.no> Amir wrote: > Dag Sverre Seljebotn writes: >> >> Amir wrote: >> > A test script at bottom is 1.8 times faster when I expand numpy >> > calls into simple for loops (n,m = 1000,1500). weave.inline is 2.7 >> > times faster. Looking at the cython -a output, not sure where most >> > of that time is lost. Looks like strides generate many more calls >> > and dot products are done using Python calls for multiplications, >> > for example. >> >> Yes, unfortunately that's what the status is; the only thing that is >> optimized by Cython is element indexing (i.e. your theta[j] and >> v[j]). This is where you'd really remove a bottleneck in some code, >> but it means that "mixed" code like yours doesn't benefit that much. >> >> Remember though that in your case, as n and m goes to infinity, the >> Python overhead will be rather small. >> > > I see. Well, it's great that it can understand regular numpy code. > > If I only use pointers to ndarray.data in my inner loop and no buffer > striding, I get a more than factor 3 speedup. The only difference in > the generated code is __Pyx_BufPtrStrided1d and __Pyx_BufPtrStrided2d > calls. These should be very fast. Do these cost that much more than > using direct pointers? You can try adding mode="c", i.e. cdef ndarray[double, ndim=1, mode="c"] theta_old = empty((m,)) This saves one multiplication per access with an unknown variable (and instead always multiplies with 8...). The requirement of course is that the buffer is contiguous. Dag Sverre From psp.reachable at gmail.com Thu Mar 19 22:31:00 2009 From: psp.reachable at gmail.com (Prajwal Suhas P) Date: Fri, 20 Mar 2009 03:01:00 +0530 Subject: [Cython] [Fwd: Re: Beginner-friendly tickets] In-Reply-To: <49C160F2.6000803@canterbury.ac.nz> References: <49BE1093.7080505@student.matnat.uio.no> <49BF4BA3.3080302@behnel.de> <49BFFC42.5090800@student.matnat.uio.no> <49C0178E.60908@canterbury.ac.nz> <2A4C178D-E9F6-4F65-8240-CF680C082C53@math.washington.edu> <49C0D24C.9020803@canterbury.ac.nz> <49C160F2.6000803@canterbury.ac.nz> Message-ID: This is the patch for the ticket#203. -- Regards, Prajwal S. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090320/3034b8a4/attachment.htm -------------- next part -------------- A non-text attachment was scrubbed... Name: ticket203.patch Type: application/octet-stream Size: 5328 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090320/3034b8a4/attachment.obj From m.gysel at gmail.com Thu Mar 19 23:25:33 2009 From: m.gysel at gmail.com (Martin Gysel) Date: Thu, 19 Mar 2009 23:25:33 +0100 Subject: [Cython] callback/segfault In-Reply-To: <57265.213.61.181.86.1237454818.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49C164EC.9040807@gmail.com> <49C1F7C6.6090506@gmail.com> <36656.213.61.181.86.1237450278.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C20BA4.6070800@gmail.com> <57265.213.61.181.86.1237454818.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49C2C65D.4010507@gmail.com> Stefan Behnel schrieb: > Martin Gysel wrote: > >> Stefan Behnel wrote: >> >> I register it and it gets called at some time. As >> already said, the library starts a new thread... at the time the >> callback gets called no cython code is supposed to run except the >> callback itself of course. >> >> 1. register callback >> 2. start library thread >> 3. ... (at the moment I just do nothing apart of waiting) >> > > How do you wait? using time.wait()? > no, for now I just use the console and wait for some printouts... > >> 4. callback gets called (here I want to deal with either the library or >> my cython object) >> 5. segfault :-( >> >> really strange for me is it segfaults immediately if I use with gil. is >> it possible the gil is held by another function? >> > > No. The GIL is an exclusive lock. When you get hold of it, you are the > only one who has it. Declaring your callback function "with gil" is > exactly the right thing to do here. > > BTW, are you sure that threading support has been initialised in your > program? Just a wild guess, but maybe "import threading" at module level > helps? If so, we may be able to fix that. > > It would be good if you could come up with a small test program that shows > exactly the code you use. Also, so far, we do not know which library you > are interfacing with. Is the library thread based on pthreads (which > Python uses)? > my cython code is available here [1]. It interfaces with PulseAudio [2]. I don't know which library pulseaudio uses... I tried different combinations, with and without gil on the functions named *_cb(...) .... to compile it simply type: gcc -g -shared -pthread -fPIC -fwrapv -O2 -Wall -fno-strict-aliasing -I/usr/include/python2.5 -lpulse -o pulse.so pulse.c (you need pulseaudio to be installed) /martin [1] https://bitbucket.org/bearsh/alpaka/raw/tip/contents/code/pulse.pyx [2] http://www.pulseaudio.org From ggellner at uoguelph.ca Fri Mar 20 00:40:54 2009 From: ggellner at uoguelph.ca (Gabriel Gellner) Date: Thu, 19 Mar 2009 19:40:54 -0400 Subject: [Cython] Subclassing ndarray Message-ID: <20090319234054.GA13791@encolpuis> Has anyone subclassed a numpy.ndarray in Cython? Or is there a way to do something like this to get method calls fast for a derived type? I am thinking of things like speeding up MaskedArray's or simpler subclasses like adding some metadata, but wanting all the other ndarray semantic in place. It is not obvious to me that this is possible (due to the __new__ overloading needed, or if it would actually make things faster). If, on the other hand this has been done somewhere, I would love to look at the source! As always, thanks for any tips. Gabriel From dalcinl at gmail.com Fri Mar 20 01:28:11 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 19 Mar 2009 21:28:11 -0300 Subject: [Cython] Cython header ordering bug (forwarded from Ryan Hinton) In-Reply-To: <49C1F64E.1090300@behnel.de> References: <49C12DD9.6040002@behnel.de> <6296B2E7-DDAB-4756-9B69-A1E78902B64B@math.washington.edu> <49C1F64E.1090300@behnel.de> Message-ID: On Thu, Mar 19, 2009 at 4:37 AM, Stefan Behnel wrote: > > Oh, but don't tell them that. I still stick to saying "generating readable > C code is a goal of the Cython project", > In line with such spirit, what do you think about generating all the argument unpacking code in a static inline function emitted right before the actual function body? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Fri Mar 20 08:17:25 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 20 Mar 2009 08:17:25 +0100 Subject: [Cython] Cython header ordering bug (forwarded from Ryan Hinton) In-Reply-To: References: <49C12DD9.6040002@behnel.de> <6296B2E7-DDAB-4756-9B69-A1E78902B64B@math.washington.edu> <49C1F64E.1090300@behnel.de> Message-ID: <49C34305.1080509@behnel.de> Lisandro Dalcin wrote: > On Thu, Mar 19, 2009 at 4:37 AM, Stefan Behnel wrote: >> Oh, but don't tell them that. I still stick to saying "generating readable >> C code is a goal of the Cython project", > > In line with such spirit, what do you think about generating all the > argument unpacking code in a static inline function emitted right > before the actual function body? Sounds like a good idea. Can you open a wishlist ticket? Stefan From stefan_ml at behnel.de Fri Mar 20 08:18:51 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 20 Mar 2009 08:18:51 +0100 Subject: [Cython] loops In-Reply-To: <49C22842.7030205@canterbury.ac.nz> References: <49C158EE.6000502@student.matnat.uio.no> <49C162DF.7020708@canterbury.ac.nz> <6B1E8348-1E34-4218-90CB-58F521A43C22@math.washington.edu> <49C22842.7030205@canterbury.ac.nz> Message-ID: <49C3435B.2060609@behnel.de> Greg Ewing wrote: > Robert Bradshaw wrote: > >> What about the behavior of >> >> for i from 0 <= i < 10: >> print i >> i += 5 > > Undefined also. Saying "undefined" sounds a bit too simple here. Users will write this kind of code and intuitively expect it to work. Stefan From psp.reachable at gmail.com Fri Mar 20 10:25:24 2009 From: psp.reachable at gmail.com (Prajwal Suhas P) Date: Fri, 20 Mar 2009 14:55:24 +0530 Subject: [Cython] [Fwd: Re: Beginner-friendly tickets] In-Reply-To: References: <49BE1093.7080505@student.matnat.uio.no> <49BF4BA3.3080302@behnel.de> <49BFFC42.5090800@student.matnat.uio.no> <49C0178E.60908@canterbury.ac.nz> <2A4C178D-E9F6-4F65-8240-CF680C082C53@math.washington.edu> <49C0D24C.9020803@canterbury.ac.nz> <49C160F2.6000803@canterbury.ac.nz> Message-ID: Can someone review this patch for ticket#203? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090320/1eabb8dd/attachment.htm -------------- next part -------------- A non-text attachment was scrubbed... Name: ticket203.patch Type: application/octet-stream Size: 5328 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090320/1eabb8dd/attachment.obj From stefan_ml at behnel.de Fri Mar 20 13:44:57 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 20 Mar 2009 13:44:57 +0100 (CET) Subject: [Cython] [Fwd: Re: Beginner-friendly tickets] In-Reply-To: References: <49BE1093.7080505@student.matnat.uio.no> <49BF4BA3.3080302@behnel.de> <49BFFC42.5090800@student.matnat.uio.no> <49C0178E.60908@canterbury.ac.nz> <2A4C178D-E9F6-4F65-8240-CF680C082C53@math.washington.edu> <49C0D24C.9020803@canterbury.ac.nz> <49C160F2.6000803@canterbury.ac.nz> Message-ID: <37547.213.61.181.86.1237553097.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Prajwal Suhas P wrote: > Can someone review this patch for ticket#203? Quick comment: could you please send patches as (plain) text attachment instead of application/octet-stream? It's easier to have a quick look when the mail program shows the patch directly. Stefan From lists at cheimes.de Fri Mar 20 13:55:16 2009 From: lists at cheimes.de (Christian Heimes) Date: Fri, 20 Mar 2009 13:55:16 +0100 Subject: [Cython] callback/segfault In-Reply-To: <57265.213.61.181.86.1237454818.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49C164EC.9040807@gmail.com> <49C1F7C6.6090506@gmail.com> <36656.213.61.181.86.1237450278.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C20BA4.6070800@gmail.com> <57265.213.61.181.86.1237454818.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: Stefan Behnel wrote: > BTW, are you sure that threading support has been initialised in your > program? Just a wild guess, but maybe "import threading" at module level > helps? If so, we may be able to fix that. I suspect the thread doesn't have a valid thread state. Python needs a PyThreadState for every thread that runs Python code. You have to call PyThreadState_New() inside the thread exactly once. The code in Modules/threadmodule.c shows you how. Christian From stefan_ml at behnel.de Fri Mar 20 14:24:49 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 20 Mar 2009 14:24:49 +0100 (CET) Subject: [Cython] [Fwd: Re: Beginner-friendly tickets] In-Reply-To: References: <49BE1093.7080505@student.matnat.uio.no> <49BF4BA3.3080302@behnel.de> <49BFFC42.5090800@student.matnat.uio.no> <49C0178E.60908@canterbury.ac.nz> <2A4C178D-E9F6-4F65-8240-CF680C082C53@math.washington.edu> <49C0D24C.9020803@canterbury.ac.nz> <49C160F2.6000803@canterbury.ac.nz> Message-ID: <44276.213.61.181.86.1237555489.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Prajwal Suhas P wrote: > Can someone review this patch for ticket#203? Looking at the patch, I noticed that it duplicates a lot of code. Could you try to limit this to the parts that are really different? This would make the code a lot easier to read and the patch easier to review. For example, one section uses "temp_range_bound", where the other spells "self.bound2.result()". Assigning "self.bound2.result()" to "temp_range_bound" in the "else" case would make the first code output identical in both conditional branches, so that it could happen once outside of the blocks. Stefan From psp.reachable at gmail.com Fri Mar 20 17:27:20 2009 From: psp.reachable at gmail.com (Prajwal Suhas P) Date: Fri, 20 Mar 2009 21:57:20 +0530 Subject: [Cython] [Fwd: Re: Beginner-friendly tickets] In-Reply-To: <44276.213.61.181.86.1237555489.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49BE1093.7080505@student.matnat.uio.no> <49BFFC42.5090800@student.matnat.uio.no> <49C0178E.60908@canterbury.ac.nz> <2A4C178D-E9F6-4F65-8240-CF680C082C53@math.washington.edu> <49C0D24C.9020803@canterbury.ac.nz> <49C160F2.6000803@canterbury.ac.nz> <44276.213.61.181.86.1237555489.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: I have restructured and removed the redundant code. Can someone review this patch. Regards, Prajwal -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090320/308a80e1/attachment.htm -------------- next part -------------- A non-text attachment was scrubbed... Name: ticket203.patch Type: application/octet-stream Size: 3098 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090320/308a80e1/attachment.obj From stefan_ml at behnel.de Fri Mar 20 17:27:48 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 20 Mar 2009 17:27:48 +0100 Subject: [Cython] callback/segfault In-Reply-To: References: <49C164EC.9040807@gmail.com> <49C1F7C6.6090506@gmail.com> <36656.213.61.181.86.1237450278.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C20BA4.6070800@gmail.com> <57265.213.61.181.86.1237454818.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49C3C404.3090708@behnel.de> Christian Heimes wrote: > Stefan Behnel wrote: >> BTW, are you sure that threading support has been initialised in your >> program? Just a wild guess, but maybe "import threading" at module level >> helps? If so, we may be able to fix that. > > I suspect the thread doesn't have a valid thread state. Python needs a > PyThreadState for every thread that runs Python code. You have to call > PyThreadState_New() inside the thread exactly once. The code in > Modules/threadmodule.c shows you how. Ah, yes, that's possible. So the idea would be to write a nogil function as callback, to do all the thread-state initialisation and cleanup manually in there, and then you can call into a with-gil function in between. Although that might look somewhat hackish all in all. I don't think Cython can do much to improve this, as it cannot know the interpreter reference to use for tread-state creation. That has to be done in user code. Stefan From stefan_ml at behnel.de Fri Mar 20 18:00:44 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 20 Mar 2009 18:00:44 +0100 Subject: [Cython] [Fwd: Re: Beginner-friendly tickets] In-Reply-To: References: <49BE1093.7080505@student.matnat.uio.no> <49BFFC42.5090800@student.matnat.uio.no> <49C0178E.60908@canterbury.ac.nz> <2A4C178D-E9F6-4F65-8240-CF680C082C53@math.washington.edu> <49C0D24C.9020803@canterbury.ac.nz> <49C160F2.6000803@canterbury.ac.nz> <44276.213.61.181.86.1237555489.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49C3CBBC.5030805@behnel.de> Prajwal Suhas P wrote: > I have restructured and removed the redundant code. Can someone review this > patch. I cleaned it up a bit in the way I described in my last e-mail. Now it looks ok to me. One problem I noticed is that I get a gcc warning about this code: -------------------------------- def f(): cdef int i for i in range(g()): print i print i warning: '__pyx_v_i' may be used uninitialized in this function -------------------------------- which is true, because "i" may not have been initialised at the end, depending on the return value of g(). Since this is exactly what Python will tell you also, I'm fine with leaving it as it is, though: -------------------------------- >>> for i in range(0): print i ... >>> i Traceback (most recent call last): File "", line 1, in NameError: name 'i' is not defined -------------------------------- So I fixed the test case also by initialising i. Is this ok with you? Stefan From psp.reachable at gmail.com Fri Mar 20 18:47:03 2009 From: psp.reachable at gmail.com (Prajwal Suhas P) Date: Fri, 20 Mar 2009 23:17:03 +0530 Subject: [Cython] [Fwd: Re: Beginner-friendly tickets] In-Reply-To: <49C3CBBC.5030805@behnel.de> References: <49BE1093.7080505@student.matnat.uio.no> <2A4C178D-E9F6-4F65-8240-CF680C082C53@math.washington.edu> <49C0D24C.9020803@canterbury.ac.nz> <49C160F2.6000803@canterbury.ac.nz> <44276.213.61.181.86.1237555489.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C3CBBC.5030805@behnel.de> Message-ID: On Fri, Mar 20, 2009 at 10:30 PM, Stefan Behnel wrote: > > Prajwal Suhas P wrote: > > I have restructured and removed the redundant code. Can someone review > this > > patch. > > I cleaned it up a bit in the way I described in my last e-mail. Now it > looks ok to me. One problem I noticed is that I get a gcc warning about > this code: > > -------------------------------- > def f(): > cdef int i > for i in range(g()): > print i > print i > > warning: '__pyx_v_i' may be used uninitialized in this function > -------------------------------- > > which is true, because "i" may not have been initialised at the end, > depending on the return value of g(). Since this is exactly what Python > will tell you also, I'm fine with leaving it as it is, though: > > -------------------------------- > >>> for i in range(0): print i > ... > >>> i > Traceback (most recent call last): > File "", line 1, in > NameError: name 'i' is not defined > -------------------------------- > > So I fixed the test case also by initialising i. > > Is this ok with you? > > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > Perfect. No issues. I initialized "i" to zero and it worked as expected. -- Regards, Prajwal S. "There is little difference in people,but this little difference makes a big difference.The little difference is attitude,the big difference is whether its positive or negative. " -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090320/9cefb0da/attachment.htm From dalcinl at gmail.com Fri Mar 20 20:04:10 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 20 Mar 2009 16:04:10 -0300 Subject: [Cython] A Trove classifier for Cython? Message-ID: Hey, could we ask for a trove classifier? $ python setup.py register running register Using PyPI login from /u/dalcinl/.pypirc Server response (400): Invalid classifier "Programming Language :: Cython" -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Fri Mar 20 20:59:55 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 20 Mar 2009 20:59:55 +0100 (CET) Subject: [Cython] [Fwd: Re: Beginner-friendly tickets] In-Reply-To: References: <49BE1093.7080505@student.matnat.uio.no> <49BFFC42.5090800@student.matnat.uio.no> <49C0178E.60908@canterbury.ac.nz> <2A4C178D-E9F6-4F65-8240-CF680C082C53@math.washington.edu> <49C0D24C.9020803@canterbury.ac.nz> <49C160F2.6000803@canterbury.ac.nz> <44276.213.61.181.86.1237555489.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <5d7a85e992ed2a63b100875c371e1b0f.squirrel@webmail.uio.no> Prajwal wrote: > I have restructured and removed the redundant code. Can someone review > this > patch. It seems that the step is still missing, i.e. if you do "for i in range(0, 10, somefunc())". Of course that could be done in another patch though, we must just remember to not close the ticket until that is done. Dag Dverre From stefan_ml at behnel.de Fri Mar 20 21:57:44 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 20 Mar 2009 21:57:44 +0100 Subject: [Cython] A Trove classifier for Cython? In-Reply-To: References: Message-ID: <49C40348.3000104@behnel.de> Lisandro Dalcin wrote: > Hey, could we ask for a trove classifier? > > $ python setup.py register > running register > Using PyPI login from /u/dalcinl/.pypirc > Server response (400): Invalid classifier "Programming Language :: Cython" Sure, please do. :) Stefan From dalcinl at gmail.com Sat Mar 21 00:00:57 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 20 Mar 2009 20:00:57 -0300 Subject: [Cython] A Trove classifier for Cython? In-Reply-To: <49C40348.3000104@behnel.de> References: <49C40348.3000104@behnel.de> Message-ID: On Fri, Mar 20, 2009 at 5:57 PM, Stefan Behnel wrote: > > Lisandro Dalcin wrote: >> Hey, could we ask for a trove classifier? >> >> $ python setup.py register >> running register >> Using PyPI login from /u/dalcinl/.pypirc >> Server response (400): Invalid classifier "Programming Language :: Cython" > > Sure, please do. :) > Well, I have no idea where to ask!! -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From m.gysel at gmail.com Sat Mar 21 00:21:09 2009 From: m.gysel at gmail.com (Martin Gysel) Date: Sat, 21 Mar 2009 00:21:09 +0100 Subject: [Cython] callback/segfault In-Reply-To: <49C3C404.3090708@behnel.de> References: <49C164EC.9040807@gmail.com> <49C1F7C6.6090506@gmail.com> <36656.213.61.181.86.1237450278.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C20BA4.6070800@gmail.com> <57265.213.61.181.86.1237454818.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C3C404.3090708@behnel.de> Message-ID: <49C424E5.2060200@gmail.com> thanks you guys for taking the time Stefan Behnel schrieb: > Christian Heimes wrote: > >> Stefan Behnel wrote: >> >>> BTW, are you sure that threading support has been initialised in your >>> program? Just a wild guess, but maybe "import threading" at module level >>> helps? If so, we may be able to fix that. >>> >> I suspect the thread doesn't have a valid thread state. Python needs a >> PyThreadState for every thread that runs Python code. You have to call >> PyThreadState_New() inside the thread exactly once. The code in >> Modules/threadmodule.c shows you how. >> > > Ah, yes, that's possible. So the idea would be to write a nogil function as > callback, to do all the thread-state initialisation and cleanup manually in > there, and then you can call into a with-gil function in between. Although > that might look somewhat hackish all in all. > According to [1] a new state can be declared by either do it manually by calling PyThreadState_New() (and so on), by using some macros (Py_BEGIN_ALLOW_THREADS, ...) of by calling PyGILState_Ensure (is this what 'with gil' does?). But all of them do basically the same. For now (not much testing has been done...) the problem seems to be fixed by simply call PyEval_InitThreads() in my __init__ function, which seems to be mandatory as soon as you use threads (Stefan is that what you meant with "import threading", should this generally impose something like this?) As I really need these callbacks I'll see it this really works in the future ;-) > I don't think Cython can do much to improve this, as it cannot know the > interpreter reference to use for tread-state creation. That has to be done > in user code. > If the problem lays in this missing call some documentation should be added to the guide... cheers martin From dalcinl at gmail.com Sat Mar 21 00:35:21 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 20 Mar 2009 20:35:21 -0300 Subject: [Cython] callback/segfault In-Reply-To: <49C424E5.2060200@gmail.com> References: <49C164EC.9040807@gmail.com> <49C1F7C6.6090506@gmail.com> <36656.213.61.181.86.1237450278.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C20BA4.6070800@gmail.com> <57265.213.61.181.86.1237454818.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C3C404.3090708@behnel.de> <49C424E5.2060200@gmail.com> Message-ID: On Fri, Mar 20, 2009 at 8:21 PM, Martin Gysel wrote: > thanks you guys for taking the time > > For now (not much testing has been done...) the problem seems to be > fixed by simply call PyEval_InitThreads() in my __init__ function, > Could you tell me please what exact Python version are you using? Stefan recently pushed a patch for Py < 2.4.2, with requires explicit calls to PyEval_InitThreads() at the module init function to enforce the creation of the GIL in advance... If you want a vile hack for making it work anyway with latest Cython 0.11 release, look at the very end of this file, just put these bit in some header, and include it with "cdef extern from ...": http://code.google.com/p/mpi4py/source/browse/trunk/src/atimport.h -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Sat Mar 21 00:40:45 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 20 Mar 2009 20:40:45 -0300 Subject: [Cython] constant handling refactored and up on cython-unstable (ticket 144) In-Reply-To: <40298.213.61.181.86.1237478145.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49C15F04.2020504@behnel.de> <49C24679.9080300@student.matnat.uio.no> <65193.213.61.181.86.1237472125.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C2557F.3090402@student.matnat.uio.no> <49093.213.61.181.86.1237476820.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C268F1.2010704@student.matnat.uio.no> <40298.213.61.181.86.1237478145.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: Are you saying with all this that cdef object CHAR_MAX = ((-1)>>1) will not give me in CHAR_MAX a (Python) integer with the maximum value representable in a C char? On Thu, Mar 19, 2009 at 12:55 PM, Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> Stefan Behnel wrote: >>> Dag Sverre Seljebotn wrote: >>> >>>> Is it impossible for constant folding to happen before type annotation? >>>> After all, the type should be hard-coded in the class for constants? >>> >>> Not sure. At least, it needs scope and target type information in its >>> current way of working, as it does type coercions. It might be possible >>> to >>> remove that completely, and to determine the widest target type of two >>> operands instead. Want to have fun giving it a try? :) >>> >> Unfortunately email activity == procrastination != real opportunity for >> Cython activity; at least for the rest of this week. (And unfortunately >> procrastination doesn't work the way that one can simply "bottle it up" >> and use it for real coding :-) ). >> >> (I think that when I get a peaceful Cython moment, I'll focus on e.g. >> getting more nodes over to new temps; the constant folding has a >> learning curve and I should focus on the 0.12 stuff which doesn't >> involve a learning curve first; more effective use of manpower that way.) > > Sure, good idea. > > >>> BTW, I'm not sure how type casts should be handled. What about something >>> like this: >>> >>> ? ? cdef int x = 3 + (2**10) >>> >> This is asking for a wraparound if you ask me. I'd say 2**10 folds, then >> leave it to runtime, because we cannot know the type sizes (at least of >> long). >>> ? Whereas >>> >>> ? ? cdef int x = 3 + (2**10) >>> >>> would simplify just fine... >>> >> However, >> cdef long x = 3 + (2**34) >> >> would simplify fine on non-Windows 64-bit but not otherwise. I'd be >> reluctant to simplify casts at all because the only intention I can find >> in code like that is to do a "& MAX_VALUE_OF_TYPE", and that stuff is >> much better left to the compiler. > > Yep. I also think that it's rather unlikely that users will put a type > cast before constants in places where compile time values are actually > interesting (such as loop ranges/steps). > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From lists at cheimes.de Sat Mar 21 00:56:53 2009 From: lists at cheimes.de (Christian Heimes) Date: Sat, 21 Mar 2009 00:56:53 +0100 Subject: [Cython] callback/segfault In-Reply-To: <49C424E5.2060200@gmail.com> References: <49C164EC.9040807@gmail.com> <49C1F7C6.6090506@gmail.com> <36656.213.61.181.86.1237450278.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C20BA4.6070800@gmail.com> <57265.213.61.181.86.1237454818.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C3C404.3090708@behnel.de> <49C424E5.2060200@gmail.com> Message-ID: Martin Gysel wrote: > According to [1] a new state can be declared by either do it manually by > calling PyThreadState_New() (and so on), by using some macros > (Py_BEGIN_ALLOW_THREADS, ...) of by calling PyGILState_Ensure (is this > what 'with gil' does?). But all of them do basically the same. > For now (not much testing has been done...) the problem seems to be > fixed by simply call PyEval_InitThreads() in my __init__ function, which > seems to be mandatory as soon as you use threads (Stefan is that what > you meant with "import threading", should this generally impose > something like this?) > As I really need these callbacks I'll see it this really works in the > future ;-) You are confusing the API functions. Py_BEGIN_ALLOW_THREADS doesn't creating a new thread state. Only PyThreadState_New() creates a new thread state. PyGILState_Ensure() and PyGILState_Release() are convenient wrappers that create and destroy thread states for a thread. It's important to know that they don't work with subinterpreters as used by mod_python and mod_wsgi. The release() method also destroys the thread state object when the internal counter reaches zero. That may be a speed issue when a short running callback is called often. Christian From robertwb at math.washington.edu Sat Mar 21 00:59:35 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 20 Mar 2009 16:59:35 -0700 Subject: [Cython] A Trove classifier for Cython? In-Reply-To: References: <49C40348.3000104@behnel.de> Message-ID: On Mar 20, 2009, at 4:00 PM, Lisandro Dalcin wrote: > On Fri, Mar 20, 2009 at 5:57 PM, Stefan Behnel > wrote: >> >> Lisandro Dalcin wrote: >>> Hey, could we ask for a trove classifier? >>> >>> $ python setup.py register >>> running register >>> Using PyPI login from /u/dalcinl/.pypirc >>> Server response (400): Invalid classifier "Programming >>> Language :: Cython" >> >> Sure, please do. :) > > Well, I have no idea where to ask!! It looks like we ask on catalog-sig at python.org. - Robert From robertwb at math.washington.edu Sat Mar 21 01:01:21 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 20 Mar 2009 17:01:21 -0700 Subject: [Cython] A Trove classifier for Cython? In-Reply-To: References: <49C40348.3000104@behnel.de> Message-ID: <04ED9E4A-1030-44A0-B5AB-AA2EEE0B4D29@math.washington.edu> Could we get a trove classifier for the programing language Cython? Programming Language :: Cython Thanks, Robert Bradshaw From robertwb at math.washington.edu Sat Mar 21 01:03:42 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 20 Mar 2009 17:03:42 -0700 Subject: [Cython] constant handling refactored and up on cython-unstable (ticket 144) In-Reply-To: References: <49C15F04.2020504@behnel.de> <49C24679.9080300@student.matnat.uio.no> <65193.213.61.181.86.1237472125.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C2557F.3090402@student.matnat.uio.no> <49093.213.61.181.86.1237476820.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C268F1.2010704@student.matnat.uio.no> <40298.213.61.181.86.1237478145.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <28DCA01B-8F5A-4AA4-B02E-937383E34508@math.washington.edu> On Mar 20, 2009, at 4:40 PM, Lisandro Dalcin wrote: > Are you saying with all this that > > cdef object CHAR_MAX = ((-1)>>1) > > will not give me in CHAR_MAX a (Python) integer with the maximum value > representable in a C char? Under my understanding, this will behave as it always did. >>>> BTW, I'm not sure how type casts should be handled. What about >>>> something >>>> like this: >>>> >>>> cdef int x = 3 + (2**10) >>>> >>> This is asking for a wraparound if you ask me. I'd say 2**10 >>> folds, then >>> leave it to runtime, because we cannot know the type sizes (at >>> least of >>> long). >>>> ? Whereas >>>> >>>> cdef int x = 3 + (2**10) >>>> >>>> would simplify just fine... >>>> >>> However, >>> cdef long x = 3 + (2**34) >>> >>> would simplify fine on non-Windows 64-bit but not otherwise. I'd be >>> reluctant to simplify casts at all because the only intention I >>> can find >>> in code like that is to do a "& MAX_VALUE_OF_TYPE", and that >>> stuff is >>> much better left to the compiler. >> >> Yep. I also think that it's rather unlikely that users will put a >> type >> cast before constants in places where compile time values are >> actually >> interesting (such as loop ranges/steps). Yes, we shouldn't fold casts. - Robert From robertwb at math.washington.edu Sat Mar 21 01:13:08 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 20 Mar 2009 17:13:08 -0700 Subject: [Cython] loops In-Reply-To: <49C3435B.2060609@behnel.de> References: <49C158EE.6000502@student.matnat.uio.no> <49C162DF.7020708@canterbury.ac.nz> <6B1E8348-1E34-4218-90CB-58F521A43C22@math.washington.edu> <49C22842.7030205@canterbury.ac.nz> <49C3435B.2060609@behnel.de> Message-ID: <3AD6A0CE-BD54-433E-BCCF-BCEE8E6A89F2@math.washington.edu> On Mar 20, 2009, at 12:18 AM, Stefan Behnel wrote: > Greg Ewing wrote: >> Robert Bradshaw wrote: >> >>> What about the behavior of >>> >>> for i from 0 <= i < 10: >>> print i >>> i += 5 >> >> Undefined also. > > Saying "undefined" sounds a bit too simple here. Users will write > this kind > of code and intuitively expect it to work. I'd rather specify the behavior. How will people expect it to work? Currently, the loop will exit early if i is a cdef int, and not if it's a python object. Anyone have an opinion on this? (I'd be inclined to specify the former behavior just to preserve backwards compatibility. - Robert From robertwb at math.washington.edu Sat Mar 21 01:15:48 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 20 Mar 2009 17:15:48 -0700 Subject: [Cython] Subclassing ndarray In-Reply-To: <20090319234054.GA13791@encolpuis> References: <20090319234054.GA13791@encolpuis> Message-ID: <87D132BA-8816-4B72-8A78-135BF32DEA0D@math.washington.edu> On Mar 19, 2009, at 4:40 PM, Gabriel Gellner wrote: > Has anyone subclassed a numpy.ndarray in Cython? Or is there a way > to do > something like this to get method calls fast for a derived type? I > am thinking > of things like speeding up MaskedArray's or simpler subclasses like > adding some metadata, but wanting all the other ndarray semantic in > place. > > It is not obvious to me that this is possible (due to the __new__ > overloading > needed, or if it would actually make things faster). If, on the > other hand this > has been done somewhere, I would love to look at the source! > > As always, thanks for any tips. I'm not aware of anyone that has done this, but you might try asking on the Python lists about the implications of subclassing this just in Python. I'd imagine the issues would be largely the same. - Robert From dalcinl at gmail.com Sat Mar 21 01:30:23 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 20 Mar 2009 21:30:23 -0300 Subject: [Cython] Subclassing ndarray In-Reply-To: <20090319234054.GA13791@encolpuis> References: <20090319234054.GA13791@encolpuis> Message-ID: On Thu, Mar 19, 2009 at 8:40 PM, Gabriel Gellner wrote: > Has anyone subclassed a numpy.ndarray in Cython? Or is there a way to do > something like this to get method calls fast for a derived type? What do you mean for fast methods calls? Skipping the lookup in the type's dict? In such case, I can imagine some easy and some not so easy (though a bit faster) ways of doing this -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dsurviver at gmail.com Sat Mar 21 03:27:28 2009 From: dsurviver at gmail.com (Danilo Freitas) Date: Fri, 20 Mar 2009 23:27:28 -0300 Subject: [Cython] Specific IDE Message-ID: <45239150903201927n5c547465r3aaf3999fe419aa@mail.gmail.com> Hi, all I'd like to know if there is a specific IDE for Cython like the IDLE for Python thanks Danilo From ellisonbg.net at gmail.com Sat Mar 21 03:56:32 2009 From: ellisonbg.net at gmail.com (Brian Granger) Date: Fri, 20 Mar 2009 19:56:32 -0700 Subject: [Cython] Subclassing ndarray In-Reply-To: References: <20090319234054.GA13791@encolpuis> Message-ID: <6ce0ac130903201956u469655dcneaa4da557275f622@mail.gmail.com> > What do you mean for fast methods calls? Skipping the lookup in the > type's dict? In such case, I can imagine some easy and some not so > easy (though a bit faster) ways of doing this In certain cases, I would love to be able to avoid this overhead. What ideas do you have in mind? Brian From dalcinl at gmail.com Sat Mar 21 05:06:11 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sat, 21 Mar 2009 01:06:11 -0300 Subject: [Cython] Subclassing ndarray In-Reply-To: <6ce0ac130903201956u469655dcneaa4da557275f622@mail.gmail.com> References: <20090319234054.GA13791@encolpuis> <6ce0ac130903201956u469655dcneaa4da557275f622@mail.gmail.com> Message-ID: What about creating a cache of pre-loaded methods from the ndarray type? As long as these methods are the tiny objects wrapping the PyCFunction pointer, the only overhead you will have is : 1- Cython creating a tuple to call the FUNC(args) # Cython sees FUNC as a normal func. 2- Python enters tp_call slot in the method type, and then does some stuff in C to split args in two: self<-args[0] and otherargs<-args[1:], plus dispatching the final C call cfunc(self, otherargs) taking into accout some flags... 4- NumPy (C) code doing tuple arg unpacking... But all the above steps are in C, and you save the dict lookup... A more cute implementation could even skip step (2) by catching the PyCFunction pointer and some flags from the PyMethodDef struct... Though I doubt we could gain too much... Of course, I'm just throwing wild ideas.... Never tried to subclass numpy, though I'm pretty sure we could found a way to solve the __new__ problem (in the case we actually need to solve something) And of course, all this would perhaps require some script generating (Cython) code, and perhaps some pure-C utility code... On Fri, Mar 20, 2009 at 11:56 PM, Brian Granger wrote: >> What do you mean for fast methods calls? Skipping the lookup in the >> type's dict? In such case, I can imagine some easy and some not so >> easy (though a bit faster) ways of doing this > > In certain cases, I would love to be able to avoid this overhead. > What ideas do you have in mind? > > Brian > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Sat Mar 21 05:33:28 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 20 Mar 2009 21:33:28 -0700 Subject: [Cython] Specific IDE In-Reply-To: <45239150903201927n5c547465r3aaf3999fe419aa@mail.gmail.com> References: <45239150903201927n5c547465r3aaf3999fe419aa@mail.gmail.com> Message-ID: <3A9E3F7A-9DF0-4A96-9B70-6945548ED617@math.washington.edu> On Mar 20, 2009, at 7:27 PM, Danilo Freitas wrote: > Hi, all > > I'd like to know if there is a specific IDE for Cython > like the IDLE for Python Not that I know of, but an IDE that understands Python works pretty well for editing Cython files as well. There's also a specific cython- mode.el for emacs. - Robert From dsurviver at gmail.com Sat Mar 21 06:20:17 2009 From: dsurviver at gmail.com (Danilo Freitas) Date: Sat, 21 Mar 2009 02:20:17 -0300 Subject: [Cython] Specific IDE In-Reply-To: <3A9E3F7A-9DF0-4A96-9B70-6945548ED617@math.washington.edu> References: <45239150903201927n5c547465r3aaf3999fe419aa@mail.gmail.com> <3A9E3F7A-9DF0-4A96-9B70-6945548ED617@math.washington.edu> Message-ID: <45239150903202220mfa57csdbec745e38f852c7@mail.gmail.com> I'm using IDLE of Python but it doesn't recognize Cython keywords, like cdef I just wanted to correct this.. I tried to configurate IDLE, but I found nothing about it :/ but it is not necessary for programming.. it's just a little "help" :D Thanks 2009/3/21 Robert Bradshaw : > On Mar 20, 2009, at 7:27 PM, Danilo Freitas wrote: > >> Hi, all >> >> I'd like to know if there is a specific IDE for Cython >> like the IDLE for Python > > Not that I know of, but an IDE that understands Python works pretty > well for editing Cython files as well. There's also a specific cython- > mode.el for emacs. > > - Robert > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From m.gysel at gmail.com Sat Mar 21 07:18:04 2009 From: m.gysel at gmail.com (Martin Gysel) Date: Sat, 21 Mar 2009 07:18:04 +0100 Subject: [Cython] Specific IDE In-Reply-To: <45239150903201927n5c547465r3aaf3999fe419aa@mail.gmail.com> References: <45239150903201927n5c547465r3aaf3999fe419aa@mail.gmail.com> Message-ID: <49C4869C.7010006@gmail.com> Danilo Freitas schrieb: > Hi, all > > I'd like to know if there is a specific IDE for Cython > like the IDLE for Python > if you use KDE's kate (also possible on windows) look at this: http://bearsh.blogspot.com/2009/03/cython-syntax-highlighting-for-kate.html I know kate is not an IDE but a bloody good editor... /martin From chrandr at binf.ku.dk Sat Mar 21 09:09:39 2009 From: chrandr at binf.ku.dk (Christian Andreetta) Date: Sat, 21 Mar 2009 09:09:39 +0100 Subject: [Cython] access a C++ class private *thisptr from another class/function? Message-ID: <300f68560903210109s994db01r277ba999de99f25e@mail.gmail.com> Hello, is there a way to access a C++ class private *thisptr from another class/function? something like (from wiki.cython.org/WrappingCPlusPlus): cdef class Rectangle: cdef c_Rectangle *thisptr def __cinit__(self): self.thisptr = new_Rectangle() # rest of implementation cdef class Obj_mangler: cdef c_Obj_mangler *thisptr def __cinit__(self): self.thisptr = new_Obj_mangler() def do_something_on_Polygon(self,Rectangle_list): # Rectangle_list is a list of Python objects wrappers # do_something is a void() c++ function accepting object pointers self.thisptr.do_something(Rectangle_list[0].thisptr) Thanks! From poonaatsoc at gmail.com Sat Mar 21 10:58:36 2009 From: poonaatsoc at gmail.com (Anoop Saldanha) Date: Sat, 21 Mar 2009 15:28:36 +0530 Subject: [Cython] loops In-Reply-To: <3AD6A0CE-BD54-433E-BCCF-BCEE8E6A89F2@math.washington.edu> References: <49C158EE.6000502@student.matnat.uio.no> <49C162DF.7020708@canterbury.ac.nz> <6B1E8348-1E34-4218-90CB-58F521A43C22@math.washington.edu> <49C22842.7030205@canterbury.ac.nz> <49C3435B.2060609@behnel.de> <3AD6A0CE-BD54-433E-BCCF-BCEE8E6A89F2@math.washington.edu> Message-ID: On Sat, Mar 21, 2009 at 5:43 AM, Robert Bradshaw < robertwb at math.washington.edu> wrote: > On Mar 20, 2009, at 12:18 AM, Stefan Behnel wrote: > > > Greg Ewing wrote: > >> Robert Bradshaw wrote: > >> > >>> What about the behavior of > >>> > >>> for i from 0 <= i < 10: > >>> print i > >>> i += 5 > >> > >> Undefined also. > > > > Saying "undefined" sounds a bit too simple here. Users will write > > this kind > > of code and intuitively expect it to work. > > I'd rather specify the behavior. How will people expect it to work? > Currently, the loop will exit early if i is a cdef int, and not if > it's a python object. Anyone have an opinion on this? (I'd be > inclined to specify the former behavior just to preserve backwards > compatibility. > > - Robert > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > If I am not wrong a "for from" resolves to a pure c loop, because of which I'd expect i to hold on to the modified value from inside the loop irrespective of i being a python object or a cdef int type. Can't we fall back to the "cdef int i" kind of loop behaviour for both the cases? Personally feel that it is more intuitive. -- Regards, Anoop S. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090321/51c0fa34/attachment.htm From stefan_ml at behnel.de Sat Mar 21 13:24:20 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 21 Mar 2009 13:24:20 +0100 Subject: [Cython] loops In-Reply-To: <3AD6A0CE-BD54-433E-BCCF-BCEE8E6A89F2@math.washington.edu> References: <49C158EE.6000502@student.matnat.uio.no> <49C162DF.7020708@canterbury.ac.nz> <6B1E8348-1E34-4218-90CB-58F521A43C22@math.washington.edu> <49C22842.7030205@canterbury.ac.nz> <49C3435B.2060609@behnel.de> <3AD6A0CE-BD54-433E-BCCF-BCEE8E6A89F2@math.washington.edu> Message-ID: <49C4DC74.2060002@behnel.de> Robert Bradshaw wrote: > On Mar 20, 2009, at 12:18 AM, Stefan Behnel wrote: > >> Greg Ewing wrote: >>> Robert Bradshaw wrote: >>> >>>> What about the behavior of >>>> >>>> for i from 0 <= i < 10: >>>> print i >>>> i += 5 >>> Undefined also. >> Saying "undefined" sounds a bit too simple here. Users will write >> this kind >> of code and intuitively expect it to work. > > I'd rather specify the behavior. How will people expect it to work? > Currently, the loop will exit early if i is a cdef int, and not if > it's a python object. Anyone have an opinion on this? (I'd be > inclined to specify the former behavior just to preserve backwards > compatibility. When I see the above code, the only intuitive expectation I can come up with is to have it exit early. I think the syntax is sufficiently far from a for-in-range loop that no-one would expect i to be taken from an iterable here. It's much more intuitive to have a more or less direct mapping to a C loop that defines a start value, an boundary condition, and a step. So I'm +1 for officially allowing the above and making it exit early. Letting Cython basically ignore the "i += 5" would be a bug. Stefan From m.gysel at gmail.com Sat Mar 21 13:51:54 2009 From: m.gysel at gmail.com (Martin Gysel) Date: Sat, 21 Mar 2009 13:51:54 +0100 Subject: [Cython] callback/segfault In-Reply-To: References: <49C164EC.9040807@gmail.com> <49C1F7C6.6090506@gmail.com> <36656.213.61.181.86.1237450278.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C20BA4.6070800@gmail.com> <57265.213.61.181.86.1237454818.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C3C404.3090708@behnel.de> <49C424E5.2060200@gmail.com> Message-ID: <49C4E2EA.9010804@gmail.com> Lisandro Dalcin schrieb: > Could you tell me please what exact Python version are you using? > Python 2.5.4 (r254:67916, Feb 22 2009, 18:08:01) > Stefan recently pushed a patch for Py < 2.4.2, with requires explicit > calls to PyEval_InitThreads() at the module init function to enforce > the creation of the GIL in advance... > > If you want a vile hack for making it work anyway with latest Cython > 0.11 release, look at the very end of this file, just put these bit in > some header, and include it with "cdef extern from ...": > > http://code.google.com/p/mpi4py/source/browse/trunk/src/atimport.h > > /martin From tseaver at palladion.com Sat Mar 21 14:00:52 2009 From: tseaver at palladion.com (Tres Seaver) Date: Sat, 21 Mar 2009 09:00:52 -0400 Subject: [Cython] A Trove classifier for Cython? In-Reply-To: <49C48896.5040308@v.loewis.de> References: <49C40348.3000104@behnel.de> <04ED9E4A-1030-44A0-B5AB-AA2EEE0B4D29@math.washington.edu> <49C48896.5040308@v.loewis.de> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Martin v. L?wis wrote: > Robert Bradshaw wrote: >> Could we get a trove classifier for the programing language Cython? >> >> Programming Language :: Cython > > What packages would be classified under this classifier (both > specific, and in principle)? One example: - - 'lxml' uses Cython to implement its wrappers of libxml2 and libxslt; it can be installed without Cython (the generated C code is part of the released package), but requires Cython for a build after a 'distclean'. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJxOUE+gerLs4ltQ4RAl8SAKCth7L6S1lKwhZ0tYtfkhc2nNLuSwCg1zmb 2p997QN0+m3ePdVevpo8Oto= =bugK -----END PGP SIGNATURE----- From dalcinl at gmail.com Sat Mar 21 15:36:24 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sat, 21 Mar 2009 11:36:24 -0300 Subject: [Cython] A Trove classifier for Cython? In-Reply-To: References: <49C40348.3000104@behnel.de> <04ED9E4A-1030-44A0-B5AB-AA2EEE0B4D29@math.washington.edu> <49C48896.5040308@v.loewis.de> Message-ID: On Sat, Mar 21, 2009 at 10:00 AM, Tres Seaver wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Martin v. L?wis wrote: >> Robert Bradshaw wrote: >>> Could we get a trove classifier for the programing language Cython? >>> >>> Programming Language :: Cython >> >> What packages would be classified under this classifier (both >> specific, and in principle)? > More examples, from my side (all of them at googlecode.comm): 1) mpi4py 2) petsc4py 3) slepc4py 4) tao4py -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Sat Mar 21 15:44:21 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sat, 21 Mar 2009 11:44:21 -0300 Subject: [Cython] access a C++ class private *thisptr from another class/function? In-Reply-To: <300f68560903210109s994db01r277ba999de99f25e@mail.gmail.com> References: <300f68560903210109s994db01r277ba999de99f25e@mail.gmail.com> Message-ID: The private/public distinction in Cython is not as in C++, it is just related to exposing C/C++ stuff to Python land... Inside a module coded in Cython, everything is (in the C++ sense) public . Yur do_something_on_Polygon() is missing a "cast", or Cython does not know that someting is actually a cdef class. def do_something_on_Polygon(self,Rectangle_list): cdef Rectangle r = Rectangle_list[0] self.thisptr.do_something(r.thisptr) On Sat, Mar 21, 2009 at 5:09 AM, Christian Andreetta wrote: > Hello, > > is there a way to access a C++ class private *thisptr from another > class/function? > something like (from wiki.cython.org/WrappingCPlusPlus): > > cdef class Rectangle: > ? ?cdef c_Rectangle *thisptr > ? ?def __cinit__(self): > ? ? ? ?self.thisptr = new_Rectangle() > ? ?# rest of implementation > > cdef class Obj_mangler: > ? ?cdef c_Obj_mangler *thisptr > ? ?def __cinit__(self): > ? ? ? ?self.thisptr = new_Obj_mangler() > ? ?def do_something_on_Polygon(self,Rectangle_list): > ? ? ? ?# Rectangle_list is a list of Python objects wrappers > ? ? ? ?# do_something is a void() c++ function accepting object pointers > ? ? ? ?self.thisptr.do_something(Rectangle_list[0].thisptr) > > Thanks! > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Sat Mar 21 15:55:20 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sat, 21 Mar 2009 11:55:20 -0300 Subject: [Cython] loops In-Reply-To: <49C4DC74.2060002@behnel.de> References: <49C158EE.6000502@student.matnat.uio.no> <49C162DF.7020708@canterbury.ac.nz> <6B1E8348-1E34-4218-90CB-58F521A43C22@math.washington.edu> <49C22842.7030205@canterbury.ac.nz> <49C3435B.2060609@behnel.de> <3AD6A0CE-BD54-433E-BCCF-BCEE8E6A89F2@math.washington.edu> <49C4DC74.2060002@behnel.de> Message-ID: On Sat, Mar 21, 2009 at 9:24 AM, Stefan Behnel wrote: >>>>> >>>>> for i from 0 <= i < 10: >>>>> ? ? ?print i >>>>> ? ? ?i += 5 > > So I'm +1 for officially allowing the above and making it exit early. > Letting Cython basically ignore the "i += 5" would be a bug. > Me too, +1. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Sat Mar 21 20:31:41 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 21 Mar 2009 12:31:41 -0700 Subject: [Cython] A Trove classifier for Cython? In-Reply-To: References: <49C40348.3000104@behnel.de> <04ED9E4A-1030-44A0-B5AB-AA2EEE0B4D29@math.washington.edu> <49C48896.5040308@v.loewis.de> Message-ID: <0EA97A4E-D300-4698-AAFC-85E1AF0D65DD@math.washington.edu> On Mar 21, 2009, at 7:36 AM, Lisandro Dalcin wrote: > On Sat, Mar 21, 2009 at 10:00 AM, Tres Seaver > wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> Martin v. L?wis wrote: >>> Robert Bradshaw wrote: >>>> Could we get a trove classifier for the programing language Cython? >>>> >>>> Programming Language :: Cython >>> >>> What packages would be classified under this classifier (both >>> specific, and in principle)? >> > > More examples, from my side (all of them at googlecode.comm): > > 1) mpi4py > 2) petsc4py > 3) slepc4py > 4) tao4py Note that these are also indexed on pypi. Some other packages using Cython are http://pypi.python.org/pypi/PyAMF http://pypi.python.org/pypi/cogent http://pypi.python.org/pypi/PyYAML http://pypi.python.org/pypi/jwp_ri http://pypi.python.org/pypi/line_profiler http://pypi.python.org/pypi/python-ctags http://pypi.python.org/pypi/python-ecore http://pypi.python.org/pypi/python-edje http://pypi.python.org/pypi/python-emotion http://pypi.python.org/pypi/python-epsilon http://pypi.python.org/pypi/python-evas http://pypi.python.org/pypi/scikits.audiolab http://pypi.python.org/pypi/mwlib http://pypi.python.org/pypi/TailSpin http://pypi.python.org/pypi/BIP http://pypi.python.org/pypi/WorldMill The project I'm most involved in, http://sagemath.org also has 200,000 + lines of Cython code. In principle, any project could use Cython, but the most likely candidates are wrappers of external C/C++ libraries (Cython makes this really easy) and scientific code (where speed is highly valued). - Robert From dsurviver at gmail.com Sun Mar 22 00:34:50 2009 From: dsurviver at gmail.com (Danilo Freitas) Date: Sat, 21 Mar 2009 20:34:50 -0300 Subject: [Cython] Specific IDE In-Reply-To: <49C4869C.7010006@gmail.com> References: <45239150903201927n5c547465r3aaf3999fe419aa@mail.gmail.com> <49C4869C.7010006@gmail.com> Message-ID: <45239150903211634i3b6e9a14wb5621453b4ec0b35@mail.gmail.com> Thanks, Martin it's working ok, here :D 2009/3/21 Martin Gysel : > Danilo Freitas schrieb: >> Hi, all >> >> I'd like to know if there is a specific IDE for Cython >> like the IDLE for Python >> > if you use KDE's kate (also possible on windows) look at this: > http://bearsh.blogspot.com/2009/03/cython-syntax-highlighting-for-kate.html > I know kate is not an IDE but a bloody good editor... > > /martin > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From wesmckinn at gmail.com Sun Mar 22 05:36:38 2009 From: wesmckinn at gmail.com (Wes McKinney) Date: Sun, 22 Mar 2009 00:36:38 -0400 Subject: [Cython] Specific IDE In-Reply-To: <45239150903211634i3b6e9a14wb5621453b4ec0b35@mail.gmail.com> References: <45239150903201927n5c547465r3aaf3999fe419aa@mail.gmail.com> <49C4869C.7010006@gmail.com> <45239150903211634i3b6e9a14wb5621453b4ec0b35@mail.gmail.com> Message-ID: <689CBD74-ADB8-45BA-B771-ABD3FC163B79@gmail.com> The commercial Wing IDE has Cython / Pyrex syntax highlighting in the new (3.2) beta, it's a pretty great IDE for python, but probably won't provide much help in Cython other than highlighting. On Mar 21, 2009, at 7:34 PM, Danilo Freitas wrote: > Thanks, Martin > > it's working ok, here :D > > 2009/3/21 Martin Gysel : >> Danilo Freitas schrieb: >>> Hi, all >>> >>> I'd like to know if there is a specific IDE for Cython >>> like the IDLE for Python >>> >> if you use KDE's kate (also possible on windows) look at this: >> http://bearsh.blogspot.com/2009/03/cython-syntax-highlighting-for- >> kate.html >> I know kate is not an IDE but a bloody good editor... >> >> /martin >> >> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From chrandr at binf.ku.dk Sun Mar 22 11:11:58 2009 From: chrandr at binf.ku.dk (Christian Andreetta) Date: Sun, 22 Mar 2009 11:11:58 +0100 Subject: [Cython] access a C++ class private *thisptr from another class/function? In-Reply-To: References: <300f68560903210109s994db01r277ba999de99f25e@mail.gmail.com> Message-ID: <300f68560903220311m2b343e99g372e706cbc195ff6@mail.gmail.com> the *casting of the classes*!! thanks!! the situation was quite more complicate, with overloadings, typedefs over templates and multiple inheritances (the best of stl c++ and the nemesis of the interfaces!), but with a proper chain of casts a lot of obstacles are easily surpassed. I've struggled with undocumented features and parser bugs in boost.python for a couple of weeks, trying to do what I did in 2 days of cython! thanks to the cython devs for their great work!! On Sat, Mar 21, 2009 at 3:44 PM, Lisandro Dalcin wrote: > The private/public distinction in Cython is not as in C++, it is just > related to exposing C/C++ stuff to Python land... Inside a module > coded in Cython, everything is (in the C++ sense) public . > > Yur do_something_on_Polygon() is missing a "cast", or Cython does not > know that someting is actually a cdef class. > > def do_something_on_Polygon(self,Rectangle_list): > ? ?cdef Rectangle r = Rectangle_list[0] > ? ?self.thisptr.do_something(r.thisptr) > > > On Sat, Mar 21, 2009 at 5:09 AM, Christian Andreetta wrote: >> Hello, >> >> is there a way to access a C++ class private *thisptr from another >> class/function? >> something like (from wiki.cython.org/WrappingCPlusPlus): >> >> cdef class Rectangle: >> ? ?cdef c_Rectangle *thisptr >> ? ?def __cinit__(self): >> ? ? ? ?self.thisptr = new_Rectangle() >> ? ?# rest of implementation >> >> cdef class Obj_mangler: >> ? ?cdef c_Obj_mangler *thisptr >> ? ?def __cinit__(self): >> ? ? ? ?self.thisptr = new_Obj_mangler() >> ? ?def do_something_on_Polygon(self,Rectangle_list): >> ? ? ? ?# Rectangle_list is a list of Python objects wrappers >> ? ? ? ?# do_something is a void() c++ function accepting object pointers >> ? ? ? ?self.thisptr.do_something(Rectangle_list[0].thisptr) >> >> Thanks! >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From poonaatsoc at gmail.com Sun Mar 22 16:48:45 2009 From: poonaatsoc at gmail.com (Anoop Saldanha) Date: Sun, 22 Mar 2009 21:18:45 +0530 Subject: [Cython] loops In-Reply-To: References: <49C158EE.6000502@student.matnat.uio.no> <49C162DF.7020708@canterbury.ac.nz> <6B1E8348-1E34-4218-90CB-58F521A43C22@math.washington.edu> <49C22842.7030205@canterbury.ac.nz> <49C3435B.2060609@behnel.de> <3AD6A0CE-BD54-433E-BCCF-BCEE8E6A89F2@math.washington.edu> <49C4DC74.2060002@behnel.de> Message-ID: 2009/3/21 Lisandro Dalcin > On Sat, Mar 21, 2009 at 9:24 AM, Stefan Behnel > wrote: > >>>>> > >>>>> for i from 0 <= i < 10: > >>>>> print i > >>>>> i += 5 > > > > So I'm +1 for officially allowing the above and making it exit early. > > Letting Cython basically ignore the "i += 5" would be a bug. > > > > Me too, +1. > > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > I looked at a range optimization example cdef int g(): print "g called" return 10 def f(): cdef int i for i in range(0, g(), 2): print i i += 2 This gives the following output - >>> g called 0 4 8 Is this the output we are expecting? I for one expected i to hold the next value from the list supplied by range every time it iterates over the loop, inspite of i being defined as a cdef int type. I created a temp variable inside Nodes.py. Every time i enter the loop, i assign the loop counter var to the temp. Inside the loop, there on I can modify the loop counter, but at the end of the loop i reassign the temp back to the loop counter var. This is the code change i made diff -r 651a88137be7 Cython/Compiler/Nodes.py --- a/Cython/Compiler/Nodes.py Fri Mar 20 17:51:46 2009 +0100 +++ b/Cython/Compiler/Nodes.py Sun Mar 22 21:05:41 2009 +0530 @@ -3968,6 +3968,7 @@ loopvar_name = self.loopvar_node.result() if from_range: range_bound = code.funcstate.allocate_temp(self.bound2.type, manage_ref=False) + temp_loop_counter = code.funcstate.allocate_temp(self.loopvar_node.type, manage_ref=False) code.putln("%s = %s;" % (range_bound, self.bound2.result())) # Skip the loop entirely (and avoid assigning to the loopvar) if # the loop is empty: @@ -3976,18 +3977,22 @@ )) else: range_bound = self.bound2.result() code.putln( "for (%s = %s%s; %s %s %s; %s%s) {" % ( loopvar_name, self.bound1.result(), offset, loopvar_name, self.relation2, range_bound, loopvar_name, incop)) + if from_range: + code.putln("%s = %s;" % (temp_loop_counter, loopvar_name)) if self.py_loopvar_node: self.py_loopvar_node.generate_evaluation_code(code) self.target.generate_assignment_code(self.py_loopvar_node, code) self.body.generate_execution_code(code) code.put_label(code.continue_label) if from_range: + code.putln("%s = %s;" % (loopvar_name, temp_loop_counter)) # Undo last increment to maintain Python semantics: code.putln("} %s%s;" % (loopvar_name, decop)) # End the outer if statement: With that I get the following output- g called 0 2 4 6 8 Shouldn't this be the expected output? -- Regards, Anoop S. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090322/b4277630/attachment.htm From psp.reachable at gmail.com Sun Mar 22 17:04:40 2009 From: psp.reachable at gmail.com (Prajwal Suhas P) Date: Sun, 22 Mar 2009 21:34:40 +0530 Subject: [Cython] [Fwd: Re: Beginner-friendly tickets] In-Reply-To: <5d7a85e992ed2a63b100875c371e1b0f.squirrel@webmail.uio.no> References: <49BE1093.7080505@student.matnat.uio.no> <2A4C178D-E9F6-4F65-8240-CF680C082C53@math.washington.edu> <49C0D24C.9020803@canterbury.ac.nz> <49C160F2.6000803@canterbury.ac.nz> <44276.213.61.181.86.1237555489.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5d7a85e992ed2a63b100875c371e1b0f.squirrel@webmail.uio.no> Message-ID: Here, i tested the 3 cases <1> for i in range(0, somefunc( ) , 1): <2> for i in range(0, 10, somefunc( )): <3> for i in range(somefunc( ), 10, 1): all work perfectly fine. So from this, i feel step need not be assigned to temp. cdef int g(): print "g called" return 3 cdef f(): cdef int i for i in range(0, 10, g()): ----------> statement print i print i -- Regards, Prajwal S. "There is little difference in people,but this little difference makes a big difference.The little difference is attitude,the big difference is whether its positive or negative. " -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090322/f6389c68/attachment.htm From stefan_ml at behnel.de Sun Mar 22 17:30:57 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 22 Mar 2009 17:30:57 +0100 Subject: [Cython] [Fwd: Re: Beginner-friendly tickets] In-Reply-To: References: <49BE1093.7080505@student.matnat.uio.no> <2A4C178D-E9F6-4F65-8240-CF680C082C53@math.washington.edu> <49C0D24C.9020803@canterbury.ac.nz> <49C160F2.6000803@canterbury.ac.nz> <44276.213.61.181.86.1237555489.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5d7a85e992ed2a63b100875c371e1b0f.squirrel@webmail.uio.no> Message-ID: <49C667C1.8020703@behnel.de> Hi, Prajwal Suhas P wrote: > Here, i tested the 3 cases > <1> for i in range(0, somefunc( ) , 1): <2> for i in range(0, 10, somefunc( > )): > <3> for i in range(somefunc( ), 10, 1): > all work perfectly fine. So from this, i feel step need not be assigned to > temp. > > cdef int g(): > print "g called" > return 3 > > cdef f(): > cdef int i > for i in range(0, 10, g()): ----------> statement > print i > print i Could you please make sure those three cases are covered by the test case? Thanks, Stefan From psp.reachable at gmail.com Sun Mar 22 19:48:59 2009 From: psp.reachable at gmail.com (Prajwal Suhas P) Date: Mon, 23 Mar 2009 00:18:59 +0530 Subject: [Cython] [Fwd: Re: Beginner-friendly tickets] In-Reply-To: <49C667C1.8020703@behnel.de> References: <49BE1093.7080505@student.matnat.uio.no> <49C160F2.6000803@canterbury.ac.nz> <44276.213.61.181.86.1237555489.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5d7a85e992ed2a63b100875c371e1b0f.squirrel@webmail.uio.no> <49C667C1.8020703@behnel.de> Message-ID: On Sun, Mar 22, 2009 at 10:00 PM, Stefan Behnel wrote: > Hi, > > Prajwal Suhas P wrote: > > Here, i tested the 3 cases > > <1> for i in range(0, somefunc( ) , 1): <2> for i in range(0, 10, > somefunc( > > )): > > <3> for i in range(somefunc( ), 10, 1): > > all work perfectly fine. So from this, i feel step need not be assigned > to > > temp. > > > > cdef int g(): > > print "g called" > > return 3 > > > > cdef f(): > > cdef int i > > for i in range(0, 10, g()): ----------> statement > > print i > > print i > > Could you please make sure those three cases are covered by the test case? > > Thanks, > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > The old patch i supplied works perfectly fine for cases---> >for i in range(0, 10, somefunc( )):--->STEP case and >for i in range(0, somefunc( ) , 1):---->Loop Bound case but it doesn't work as expected when we invoke a function in start of loop as like, for i in range(somefunc( ), 10, 1):. I have updated nodes.py and also the test cases. This is the new patch for ticket #203.But this time solves the start case for the range. -- Regards, Prajwal S. "There is little difference in people,but this little difference makes a big difference.The little difference is attitude,the big difference is whether its positive or negative. " -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090323/ef753ee4/attachment.htm -------------- next part -------------- A non-text attachment was scrubbed... Name: ticket203.patch Type: application/octet-stream Size: 3008 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090323/ef753ee4/attachment.obj From vic.kelson at gmail.com Mon Mar 23 03:46:07 2009 From: vic.kelson at gmail.com (Vic Kelson) Date: Sun, 22 Mar 2009 22:46:07 -0400 Subject: [Cython] PyCon? Message-ID: Hi Everyone, I plan to attend PyCon in Chicago next weekend, and I'm wondering what Cython experts might be there to help me jump-start an application I'm working on. I've been a Python programmer since 1998 (I , and I've written several extensions in C and Fortran. I'm well-acquainted with the ins and outs of Python since 1.4. This will be my fifth PyCon, and I always try to go there with a project in mind. I develop hydrologic models in Fortran and C/C++, but more and more, I do the majority of my work in Python. Most of the Python work is pre- and post- processing for Fortran groundwater flow model codes. I've experimented with a number of approaches to getting a suitably high- performance model coderunning in a Pythonic way, but there's always one or two things that stop me from getting a full-featured code working. Until now. I'm convinced that Cython offers me the ability to develop a Cython/Python codebase that will allow me to achieve suitable performance for a general-purpose code. However, I can use some assistance with a few issues, and I'd like to see just how fast I can make this code go. Issues are: (1) many of the fundamental analytic functions are already carefully coded in Fortran 90 (no pointers) and can readily connect to Python via F2PY. I don't want to re-code these because "they're both nasty" and coded using complex arithmetic. So, my optimal Cython module should call some Fortran routines. (2) I already have a solid object-oriented design in Python, and a small working development code (a colleague and I wrote it for a paper we've recently submitted to the journal Ground Water). It will need a change or two to make it Cython-able. In particular, it uses a mix-in class to achieve some functionality. These features can be easily delegated to a helper object that is instantiated separately. I will refactor that before the conference. (3) The "killer" feature that has stopped me from a Python computational code is the ability to implement particle tracking in the model. The tracking algorithm is computationally intensive, including checking for intersections between line segments and proximity to model features. I believe that Cython will allow me to overcome this hurdle. The test code I mentioned above does not support this feature, but I will try to implement it before PyCon. So, is there anyone who will be in Chicago who would be willing to assist me in this adventure? I have Python 2.5.4, Cython, gfortran, numpy, and the whole shebang installed on my Macbook, and I'm willing to provide tasty malt beverages... Please drop me a line if you'd be willing to help. THANKS! Vic Kelson -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090322/681d8954/attachment-0001.htm From dagss at student.matnat.uio.no Mon Mar 23 09:10:20 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 23 Mar 2009 09:10:20 +0100 Subject: [Cython] [Fwd: Re: Beginner-friendly tickets] In-Reply-To: References: <49BE1093.7080505@student.matnat.uio.no> <49C160F2.6000803@canterbury.ac.nz> <44276.213.61.181.86.1237555489.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5d7a85e992ed2a63b100875c371e1b0f.squirrel@webmail.uio.no> <49C667C1.8020703@behnel.de> Message-ID: <49C743EC.10300@student.matnat.uio.no> Prajwal Suhas P wrote: > The old patch i supplied works perfectly fine for cases---> > >for i in range(0, 10, somefunc( )):--->STEP case and > >for i in range(0, somefunc( ) , 1):---->Loop Bound case > > but it doesn't work as expected when we invoke a function in start of > loop as like, for i in range(somefunc( ), 10, 1):. Ah right; that problem was introduced when the wrapping if-test was added so that empty loops doesn't assign to "i" (as both the if-test and the loop checks the start-point) Once "i" is put in a temporary as well (so that modifying "i" within the loop doesn't affect looping) this should not be necesarry as the startpoint will only be called once. So I'd recommend that strategy instead of this patch; it should be done anyway before a release. For that reason I'm -1 on applying this. (Then one would go back to the old behaviour with for-from I suppose, i.e. so that "for i from 10 <= i < 7: pass" would assign 10 to i). BTW one should ideally test with negative steps as well, though I expect the result to be the same. I.e. "for i in range(10, somefunc(), -1)" and so on. -- Dag Sverre From stefan_ml at behnel.de Mon Mar 23 09:31:39 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 23 Mar 2009 09:31:39 +0100 (CET) Subject: [Cython] loop semantics (was: Beginner-friendly tickets) In-Reply-To: <49C743EC.10300@student.matnat.uio.no> References: <49BE1093.7080505@student.matnat.uio.no> <49C160F2.6000803@canterbury.ac.nz> <44276.213.61.181.86.1237555489.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5d7a85e992ed2a63b100875c371e1b0f.squirrel@webmail.uio.no> <49C667C1.8020703@behnel.de> <49C743EC.10300@student.matnat.uio.no> Message-ID: <63760.213.61.181.86.1237797099.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > Once "i" is put in a temporary as well (so that modifying "i" within the > loop doesn't affect looping) this should not be necesarry as the > startpoint will only be called once. ... and the assignment to "i" will happen inside the loop, so that loops that never run do not touch the value of "i". That's prefect Python behaviour. > (Then one would go back to the old behaviour with for-from I suppose, > i.e. so that "for i from 10 <= i < 7: pass" would assign 10 to i). Hmmm, did we change that? I can't remember discussing this bit. Looks like we really need a Wiki page here that clearly specifies the semantics of both loops in all corner cases... > BTW one should ideally test with negative steps as well, though I expect > the result to be the same. I.e. "for i in range(10, somefunc(), -1)" and > so on. Yes, that's missing from the test cases, but it's an important point. Stefan From poonaatsoc at gmail.com Mon Mar 23 10:11:16 2009 From: poonaatsoc at gmail.com (Anoop Saldanha) Date: Mon, 23 Mar 2009 14:41:16 +0530 Subject: [Cython] loops In-Reply-To: References: <49C158EE.6000502@student.matnat.uio.no> <49C162DF.7020708@canterbury.ac.nz> <6B1E8348-1E34-4218-90CB-58F521A43C22@math.washington.edu> <49C22842.7030205@canterbury.ac.nz> <49C3435B.2060609@behnel.de> <3AD6A0CE-BD54-433E-BCCF-BCEE8E6A89F2@math.washington.edu> <49C4DC74.2060002@behnel.de> Message-ID: On Sun, Mar 22, 2009 at 9:18 PM, Anoop Saldanha wrote: > > > 2009/3/21 Lisandro Dalcin > > On Sat, Mar 21, 2009 at 9:24 AM, Stefan Behnel >> wrote: >> >>>>> >> >>>>> for i from 0 <= i < 10: >> >>>>> print i >> >>>>> i += 5 >> > >> > So I'm +1 for officially allowing the above and making it exit early. >> > Letting Cython basically ignore the "i += 5" would be a bug. >> > >> >> Me too, +1. >> >> >> >> -- >> Lisandro Dalc?n >> --------------- >> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) >> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) >> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) >> PTLC - G?emes 3450, (3000) Santa Fe, Argentina >> Tel/Fax: +54-(0)342-451.1594 >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > I looked at a range optimization example > > cdef int g(): > print "g called" > return 10 > > def f(): > cdef int i > for i in range(0, g(), 2): > print i > i += 2 > > This gives the following output - > > >>> > g called > 0 > 4 > 8 > > Is this the output we are expecting? I for one expected i to hold the next > value from the list supplied by range every time it iterates over the loop, > inspite of i being defined as a cdef int type. > > I created a temp variable inside Nodes.py. Every time i enter the loop, i > assign the loop counter var to the temp. Inside the loop, there on I can > modify the loop counter, but at the end of the loop i reassign the temp back > to the loop counter var. > > This is the code change i made > > diff -r 651a88137be7 Cython/Compiler/Nodes.py > --- a/Cython/Compiler/Nodes.py Fri Mar 20 17:51:46 2009 +0100 > +++ b/Cython/Compiler/Nodes.py Sun Mar 22 21:05:41 2009 +0530 > @@ -3968,6 +3968,7 @@ > loopvar_name = self.loopvar_node.result() > if from_range: > range_bound = code.funcstate.allocate_temp(self.bound2.type, > manage_ref=False) > + temp_loop_counter = > code.funcstate.allocate_temp(self.loopvar_node.type, manage_ref=False) > code.putln("%s = %s;" % (range_bound, self.bound2.result())) > # Skip the loop entirely (and avoid assigning to the loopvar) > if > # the loop is empty: > @@ -3976,18 +3977,22 @@ > )) > else: > range_bound = self.bound2.result() > code.putln( > "for (%s = %s%s; %s %s %s; %s%s) {" % ( > loopvar_name, > self.bound1.result(), offset, > loopvar_name, self.relation2, range_bound, > loopvar_name, incop)) > + if from_range: > + code.putln("%s = %s;" % (temp_loop_counter, loopvar_name)) > if self.py_loopvar_node: > self.py_loopvar_node.generate_evaluation_code(code) > self.target.generate_assignment_code(self.py_loopvar_node, > code) > self.body.generate_execution_code(code) > code.put_label(code.continue_label) > if from_range: > + code.putln("%s = %s;" % (loopvar_name, temp_loop_counter)) > # Undo last increment to maintain Python semantics: > code.putln("} %s%s;" % (loopvar_name, decop)) > # End the outer if statement: > > With that I get the following output- > g called > 0 > 2 > 4 > 6 > 8 > > Shouldn't this be the expected output? > > -- > Regards, > Anoop S. > > And for a "for in range", shouldn't we fall back to the way the python "for in range" works, even though a range optimization happens in case of a cython "for in range"? For example the following code: def f(): cdef int i i = 0 for i in range(10): print i i += 3 print "modify loop counter >> ", i print "outside the loop", i we currently get this output - >> calling f() 0 modify loop counter >> 3 4 modify loop counter >> 7 8 modify loop counter >> 11 outside the loop 11 But wouldn't the output be more intuitive if we get the following output? - 0 modify loop counter >> 3 1 modify loop counter >> 4 2 modify loop counter >> 5 3 modify loop counter >> 6 4 modify loop counter >> 7 5 modify loop counter >> 8 6 modify loop counter >> 9 7 modify loop counter >> 10 8 modify loop counter >> 11 9 modify loop counter >> 12 outside the loop 9 We can fall back to exiting the loop early in case of a "for from", but in case of a "for in range()", I feel the second output above is more intuitive, since that is how a python "for in range()" behaves. -- Regards, Anoop S. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090323/dd0db42f/attachment.htm From stefan_ml at behnel.de Mon Mar 23 10:25:23 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 23 Mar 2009 10:25:23 +0100 (CET) Subject: [Cython] loops In-Reply-To: References: <49C158EE.6000502@student.matnat.uio.no> <49C162DF.7020708@canterbury.ac.nz> <6B1E8348-1E34-4218-90CB-58F521A43C22@math.washington.edu> <49C22842.7030205@canterbury.ac.nz> <49C3435B.2060609@behnel.de> <3AD6A0CE-BD54-433E-BCCF-BCEE8E6A89F2@math.washington.edu> <49C4DC74.2060002@behnel.de> Message-ID: <37085.213.61.181.86.1237800323.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Anoop Saldanha wrote: > We can fall back to exiting the loop early in case of a "for from", but in > case of a "for in range()", I feel the second output above is more > intuitive, since that is how a python "for in range()" behaves. This was discussed in a separate thread. The solution is to keep the loop variable in a temp variable and to assign that to the real loop variable inside the loop. The C compiler should be smart enough to alias the two variables in many cases. Stefan From psp.reachable at gmail.com Mon Mar 23 14:35:33 2009 From: psp.reachable at gmail.com (Prajwal Suhas P) Date: Mon, 23 Mar 2009 19:05:33 +0530 Subject: [Cython] Control flow analysis Message-ID: I was reading about "control flow analysis".I couldn't catch much from the enhancements page description.Can someone please briefly explain as to what it means? Regards, Prajwal S. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090323/409d4d14/attachment.htm From stefan_ml at behnel.de Mon Mar 23 14:38:44 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 23 Mar 2009 14:38:44 +0100 (CET) Subject: [Cython] Control flow analysis In-Reply-To: References: Message-ID: <63043.213.61.181.86.1237815524.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Prajwal Suhas P wrote: > I was reading about "control flow analysis".I couldn't catch much from the > enhancements page description.Can someone please briefly explain as to > what it means? Does this help? http://en.wikipedia.org/wiki/Control_flow_analysis http://en.wikipedia.org/wiki/Control_flow_graph Stefan From stefan_ml at behnel.de Mon Mar 23 14:53:09 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 23 Mar 2009 14:53:09 +0100 (CET) Subject: [Cython] Wikipedia article on compiler oprimisation Message-ID: <46779.213.61.181.86.1237816389.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Hi, I just noticed that the Wikipedia article on compiler optimisation is actually pretty good as an entry point for people who want to learn about some of the concepts in this area. It might be worth linking to it (or its subtopics) from a couple of Cython wiki pages. http://en.wikipedia.org/wiki/Compiler_optimization Stefan From dagss at student.matnat.uio.no Mon Mar 23 16:21:42 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 23 Mar 2009 16:21:42 +0100 Subject: [Cython] About GSoC 2009 Message-ID: <49C7A906.6040502@student.matnat.uio.no> Part A: I'm starting this thread for information more directed towards potential GSoC students. The student application round starts today and lasts until April 3rd. How it works: Cython can mentor students through the Python Software Foundation. Application happens through Google, and then who gets mentored is in part up to Google (number of slots for PSF), in part up to PSF (especially how many, but I suppose this also depends on the strength of the applications), and in part up to the potential Cython mentors. However, if you haven't been in touch with the Cython list and/or a potential mentor in Cython before submitting your application, it is much more unlikely that you will get your application accepted. Again, some possible projects are at http://wiki.cython.org/codeprojects Part B: About myself as a mentor an GSoC I've come to the conclusion that a Fortran integration project is something I'm interested enough in, and which is small enough and feasible enough, that I could become a mentor. If you are interested in doing Fortran integration, please read my notes on http://wiki.cython.org/enhancements/fortran and provide your own thoughts on my attack plan *in private* (as I'm more interested in evaluating and talking to potential candidates at this point than to resolve the final vote on the mailing list on the issue). It's especially important to contact me early in this case because the addition of Fortran integration should be voted on on the mailing list before I can accept a student for this (and I won't bother to do this until something is more ready for voting). Constructs for parallell execution might attract me as well, though I'd like to find a co-mentor with a stronger background in OpenMP. Everything else has a much smaller chance for my part. Dag Sverre From dagss at student.matnat.uio.no Mon Mar 23 16:30:16 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 23 Mar 2009 16:30:16 +0100 Subject: [Cython] About GSoC 2009 In-Reply-To: <49C7A906.6040502@student.matnat.uio.no> References: <49C7A906.6040502@student.matnat.uio.no> Message-ID: <49C7AB08.10906@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > I've come to the conclusion that a Fortran integration project is > something I'm interested enough in, and which is small enough and > feasible enough, that I could become a mentor. If you are interested in > doing Fortran integration, please read my notes on > http://wiki.cython.org/enhancements/fortran and provide your own > thoughts on my attack plan *in private* (as I'm more interested in > evaluating and talking to potential candidates at this point than to > resolve the final vote on the mailing list on the issue). > I hasten to add: It is not necesarry to immediately understand everything on that link, especially the part about strided arrays. The important thing is that a potential student gets the big lines and understands and can comment on some parts of it; also good questions can impress like nothing else. (The page was written in a hurry and things could definitely be explained a lot better.) Dag Sverre From psp.reachable at gmail.com Mon Mar 23 17:42:17 2009 From: psp.reachable at gmail.com (Prajwal Suhas P) Date: Mon, 23 Mar 2009 22:12:17 +0530 Subject: [Cython] About GSoC 2009 In-Reply-To: <49C7AB08.10906@student.matnat.uio.no> References: <49C7A906.6040502@student.matnat.uio.no> <49C7AB08.10906@student.matnat.uio.no> Message-ID: I'm interested in both "control flow analysis" and the "python 3" compatibility for gsoc 2009. I was reading about "control flow analysis" and i mainly have it in my mind over python 3 compatibility and it interested me more than python3 compatibility. If someone can mentor me for this it would be very nice. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090323/06d9bb88/attachment.htm From kwmsmith at gmail.com Mon Mar 23 18:37:44 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Mon, 23 Mar 2009 12:37:44 -0500 Subject: [Cython] Wikipedia article on compiler oprimisation In-Reply-To: <46779.213.61.181.86.1237816389.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <46779.213.61.181.86.1237816389.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: On Mon, Mar 23, 2009 at 8:53 AM, Stefan Behnel wrote: > Hi, > > I just noticed that the Wikipedia article on compiler optimisation is > actually pretty good as an entry point for people who want to learn about > some of the concepts in this area. It might be worth linking to it (or its > subtopics) from a couple of Cython wiki pages. > > http://en.wikipedia.org/wiki/Compiler_optimization Thanks for taking the time to post the link -- helps a lot. Kurt From kwmsmith at gmail.com Mon Mar 23 18:44:41 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Mon, 23 Mar 2009 12:44:41 -0500 Subject: [Cython] Fix for ticket #135 In-Reply-To: <49C1368B.3010007@student.matnat.uio.no> References: <49C13044.2090808@behnel.de> <49C1368B.3010007@student.matnat.uio.no> Message-ID: On Wed, Mar 18, 2009 at 12:59 PM, Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Hi, >> >> Kurt Smith wrote: >>> Ticket 135 (return statement outside a function body crashes cython) >>> can be fixed with a simple assignment of None to self.return_type in >>> the Scope __init__ function. ?The crash happens when there is a return >>> statement in the class body as well as at the module level. >>> [snip] >> >> Thanks for looking into this. Without actually knowing the details, I'm not >> sure yet that this is a good way to do it. I do not have high confidence in >> the current handling of the return_type, so setting it globally in a class >> might cover incorrect assumptions in other parts of the code that just >> expect that the return_type is set at some point in the pipeline where it >> normally wouldn't be set (i.e. long before code generation). I ran into >> tons of problems like these since I started working on ticket 144. It's >> actually possible that this will be fixed when I'm done with that ticket. > > If not, PostParse (in ParseTreeTransforms.py) is a very safe spot to do > things like this. One can trivially implement visit_ReturnStatNode and > raise syntax errors if one is not within a function, and it means that > if there's an error, one simply stops before getting to somewhere it can > cause trouble. > > IMO more error checking (e.g. for most syntax errors) should go into there. Would it be worthwhile to pursue a fix for #135 by modifying code in the PostParse phase as Dag mentions here, or is the fix going to be subsumed by 'fixing' ticket #144? If the former, I'll take a stab at it. Thanks, Kurt From psp.reachable at gmail.com Mon Mar 23 18:48:46 2009 From: psp.reachable at gmail.com (Prajwal Suhas P) Date: Mon, 23 Mar 2009 23:18:46 +0530 Subject: [Cython] Control flow analysis In-Reply-To: <63043.213.61.181.86.1237815524.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <63043.213.61.181.86.1237815524.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: On Mon, Mar 23, 2009 at 7:08 PM, Stefan Behnel wrote: > Prajwal Suhas P wrote: > > I was reading about "control flow analysis".I couldn't catch much from > the > > enhancements page description.Can someone please briefly explain as to > > what it means? > > Does this help? > > http://en.wikipedia.org/wiki/Control_flow_analysis > http://en.wikipedia.org/wiki/Control_flow_graph > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > Thanks that was very useful. -- Regards, Prajwal S. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090323/cef5e766/attachment.htm From poonaatsoc at gmail.com Mon Mar 23 20:08:59 2009 From: poonaatsoc at gmail.com (Anoop Saldanha) Date: Tue, 24 Mar 2009 00:38:59 +0530 Subject: [Cython] About GSoC 2009 In-Reply-To: References: <49C7A906.6040502@student.matnat.uio.no> <49C7AB08.10906@student.matnat.uio.no> Message-ID: 2009/3/23 Prajwal Suhas P > I'm interested in both "control flow analysis" and the "python 3" > compatibility for gsoc 2009. I was reading about "control flow analysis" and > i mainly have it in my mind over python 3 compatibility and it interested me > more than python3 compatibility. If someone can mentor me for this it would > be very nice. > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > I would like to mentor you for control flow analysis, but I don't have much idea on how this works inside cython. Looking into control flow analysis for quite sometime now, since I wanted to have a go at this project, but i have still a lot to understand. With a co-mentor I should be able to collaborate in a much better way with you. It would also get me started on understanding cython better. If someone wants to solely mentor you it is fine with me. Otherwise I am okay with being a mentor, with a co-mentor alongside. -- Regards, Anoop S. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090324/d0fc51ad/attachment.htm From robertwb at math.washington.edu Mon Mar 23 21:32:51 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 23 Mar 2009 13:32:51 -0700 Subject: [Cython] About GSoC 2009 In-Reply-To: References: <49C7A906.6040502@student.matnat.uio.no> <49C7AB08.10906@student.matnat.uio.no> Message-ID: On Mar 23, 2009, at 12:08 PM, Anoop Saldanha wrote: > 2009/3/23 Prajwal Suhas P > I'm interested in both "control flow analysis" and the "python 3" > compatibility for gsoc 2009. I was reading about "control flow > analysis" and i mainly have it in my mind over python 3 > compatibility and it interested me more than python3 compatibility. > If someone can mentor me for this it would be very nice. > > I would like to mentor you for control flow analysis, but I don't > have much idea on how this works inside cython. Right now there isn't any (well, there isn't any that's worth keeping). > Looking into control flow analysis for quite sometime now, since I > wanted to have a go at this project, but i have still a lot to > understand. With a co-mentor I should be able to collaborate in a > much better way with you. It would also get me started on > understanding cython better. If someone wants to solely mentor you > it is fine with me. Otherwise I am okay with being a mentor, with > a co-mentor alongside. I could probably co-mentor you for a project like this. - Robert From robertwb at math.washington.edu Mon Mar 23 21:37:30 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 23 Mar 2009 13:37:30 -0700 Subject: [Cython] Fix for ticket #135 In-Reply-To: References: <49C13044.2090808@behnel.de> <49C1368B.3010007@student.matnat.uio.no> Message-ID: <02DFEEFA-4143-4F00-971F-30FD91D92324@math.washington.edu> On Mar 23, 2009, at 10:44 AM, Kurt Smith wrote: > On Wed, Mar 18, 2009 at 12:59 PM, Dag Sverre Seljebotn > wrote: >> Stefan Behnel wrote: >>> Hi, >>> >>> Kurt Smith wrote: >>>> Ticket 135 (return statement outside a function body crashes >>>> cython) >>>> can be fixed with a simple assignment of None to >>>> self.return_type in >>>> the Scope __init__ function. The crash happens when there is a >>>> return >>>> statement in the class body as well as at the module level. >>>> > [snip] >>> >>> Thanks for looking into this. Without actually knowing the >>> details, I'm not >>> sure yet that this is a good way to do it. I do not have high >>> confidence in >>> the current handling of the return_type, so setting it globally >>> in a class >>> might cover incorrect assumptions in other parts of the code that >>> just >>> expect that the return_type is set at some point in the pipeline >>> where it >>> normally wouldn't be set (i.e. long before code generation). I >>> ran into >>> tons of problems like these since I started working on ticket >>> 144. It's >>> actually possible that this will be fixed when I'm done with that >>> ticket. >> >> If not, PostParse (in ParseTreeTransforms.py) is a very safe spot >> to do >> things like this. One can trivially implement visit_ReturnStatNode >> and >> raise syntax errors if one is not within a function, and it means >> that >> if there's an error, one simply stops before getting to somewhere >> it can >> cause trouble. >> >> IMO more error checking (e.g. for most syntax errors) should go >> into there. > > Would it be worthwhile to pursue a fix for #135 by modifying code in > the PostParse phase as Dag mentions here, or is the fix going to be > subsumed by 'fixing' ticket #144? > > If the former, I'll take a stab at it. I think it's worth doing the former, in place of the current patch up at #135. I would say the current patch covers up bad code, while modifying PostParse will avoid it (which is both cleaner and leads to less potential confusion/unintended effects later). - Robert From robertwb at math.washington.edu Mon Mar 23 21:49:17 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 23 Mar 2009 13:49:17 -0700 Subject: [Cython] loop semantics (was: Beginner-friendly tickets) In-Reply-To: <63760.213.61.181.86.1237797099.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49BE1093.7080505@student.matnat.uio.no> <49C160F2.6000803@canterbury.ac.nz> <44276.213.61.181.86.1237555489.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5d7a85e992ed2a63b100875c371e1b0f.squirrel@webmail.uio.no> <49C667C1.8020703@behnel.de> <49C743EC.10300@student.matnat.uio.no> <63760.213.61.181.86.1237797099.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: On Mar 23, 2009, at 1:31 AM, Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> Once "i" is put in a temporary as well (so that modifying "i" >> within the >> loop doesn't affect looping) this should not be necesarry as the >> startpoint will only be called once. > > ... and the assignment to "i" will happen inside the loop, so that > loops > that never run do not touch the value of "i". That's prefect Python > behaviour. > > >> (Then one would go back to the old behaviour with for-from I suppose, >> i.e. so that "for i from 10 <= i < 7: pass" would assign 10 to i). > > Hmmm, did we change that? I can't remember discussing this bit. > > Looks like we really need a Wiki page here that clearly specifies the > semantics of both loops in all corner cases... http://wiki.cython.org/loops , according to how I think of them. >> BTW one should ideally test with negative steps as well, though I >> expect >> the result to be the same. I.e. "for i in range(10, somefunc(), >> -1)" and >> so on. > > Yes, that's missing from the test cases, but it's an important point. > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From kwmsmith at gmail.com Mon Mar 23 22:46:55 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Mon, 23 Mar 2009 16:46:55 -0500 Subject: [Cython] Fix for ticket #135 In-Reply-To: <02DFEEFA-4143-4F00-971F-30FD91D92324@math.washington.edu> References: <49C13044.2090808@behnel.de> <49C1368B.3010007@student.matnat.uio.no> <02DFEEFA-4143-4F00-971F-30FD91D92324@math.washington.edu> Message-ID: On Mon, Mar 23, 2009 at 3:37 PM, Robert Bradshaw wrote: [...] >> >> Would it be worthwhile to pursue a fix for #135 by modifying code in >> the PostParse phase as Dag mentions here, or is the fix going to be >> subsumed by 'fixing' ticket #144? >> >> If the former, I'll take a stab at it. > > I think it's worth doing the former, in place of the current patch up > at #135. I would say the current patch covers up bad code, while > modifying PostParse will avoid it (which is both cleaner and leads to > less potential confusion/unintended effects later). I'll get working on it; thanks. Kurt From robertwb at math.washington.edu Tue Mar 24 03:44:27 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 23 Mar 2009 19:44:27 -0700 Subject: [Cython] PyCon? In-Reply-To: References: Message-ID: <7F45D784-D392-426C-B6E2-2902285277D8@math.washington.edu> On Mar 22, 2009, at 7:46 PM, Vic Kelson wrote: > Hi Everyone, > > I plan to attend PyCon in Chicago next weekend, and I'm wondering > what Cython experts might be there to help me jump-start an > application I'm working on. I've been a Python programmer since > 1998 (I , and I've written several extensions in C and Fortran. > I'm well-acquainted with the ins and outs of Python since 1.4. This > will be my fifth PyCon, and I always try to go there with a project > in mind. I won't be there, but hopefully someone familiar with Cython may. If not, Cython should be relatively easy to learn for someone already versed in Python, Fortran, and C. > I develop hydrologic models in Fortran and C/C++, but more and > more, I do the majority of my work in Python. Most of the Python > work is pre- and post- processing for Fortran groundwater flow > model codes. I've experimented with a number of approaches to > getting a suitably high-performance model coderunning in a Pythonic > way, but there's always one or two things that stop me from getting > a full-featured code working. > > Until now. I'm convinced that Cython offers me the ability to > develop a Cython/Python codebase that will allow me to achieve > suitable performance for a general-purpose code. However, I can use > some assistance with a few issues, and I'd like to see just how > fast I can make this code go. > > Issues are: > > (1) many of the fundamental analytic functions are already > carefully coded in Fortran 90 (no pointers) and can readily connect > to Python via F2PY. I don't want to re-code these because "they're > both nasty" and coded using complex arithmetic. So, my optimal > Cython module should call some Fortran routines. There may be a GSoC project on making direct calls to Fortran easier, but for now you'd have to do it manually via C. Depending on how messy it is, it may or may not be worth re-writing this portion of your code. > (2) I already have a solid object-oriented design in Python, and a > small working development code (a colleague and I wrote it for a > paper we've recently submitted to the journal Ground Water). It > will need a change or two to make it Cython-able. In particular, it > uses a mix-in class to achieve some functionality. These features > can be easily delegated to a helper object that is instantiated > separately. I will refactor that before the conference. Cython does support multiple inheritance for non-cdef classes. > (3) The "killer" feature that has stopped me from a Python > computational code is the ability to implement particle tracking in > the model. The tracking algorithm is computationally intensive, > including checking for intersections between line segments and > proximity to model features. I believe that Cython will allow me to > overcome this hurdle. The test code I mentioned above does not > support this feature, but I will try to implement it before PyCon. This is the kind of stuff Cython excels at, so I'm optimistic this will work well for you. > So, is there anyone who will be in Chicago who would be willing to > assist me in this adventure? I have Python 2.5.4, Cython, gfortran, > numpy, and the whole shebang installed on my Macbook, and I'm > willing to provide tasty malt beverages... > > Please drop me a line if you'd be willing to help. > > THANKS! > Vic Kelson > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From robertwb at math.washington.edu Tue Mar 24 04:05:20 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 23 Mar 2009 20:05:20 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49C14F0C.4030000@behnel.de> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <49BEDD8E.1000007@canterbury.ac.nz> <49BF426D.7010005@behnel.de> <49BF5871.4040404@student.matnat.uio.no> <50876.213.61.181.86.1237279705.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C14F0C.4030000@behnel.de> Message-ID: On Mar 18, 2009, at 12:44 PM, Stefan Behnel wrote: > Robert Bradshaw wrote: >> I started a CEP up at http://wiki.cython.org/enhancements/division >> which attempts to summarize the issues brought up on this thread. > > I added the pros and cons that I could come up with to each > solution. Maybe > others can add to that. I think we've waited long enough for everyone to give input. I say we change the default to python-style division, implementing a compiler directive and cython.cdiv/cmod methods to obtain the old behavior. This is a big enough change it should probably wait 'till 0.12, but we should maybe start printing out a warning in the next release. - Robert From stefan_ml at behnel.de Tue Mar 24 07:23:22 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 24 Mar 2009 07:23:22 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <49BEDD8E.1000007@canterbury.ac.nz> <49BF426D.7010005@behnel.de> <49BF5871.4040404@student.matnat.uio.no> <50876.213.61.181.86.1237279705.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C14F0C.4030000@behnel.de> Message-ID: <49C87C5A.80607@behnel.de> Robert Bradshaw wrote: > On Mar 18, 2009, at 12:44 PM, Stefan Behnel wrote: >> Robert Bradshaw wrote: >>> I started a CEP up at http://wiki.cython.org/enhancements/division >>> which attempts to summarize the issues brought up on this thread. >> I added the pros and cons that I could come up with to each >> solution. Maybe >> others can add to that. > > I think we've waited long enough for everyone to give input. I say we > change the default to python-style division, implementing a compiler > directive and cython.cdiv/cmod methods to obtain the old behavior. +1 > This is a big enough change it should probably wait 'till 0.12 It should still go into cython-unstable as soon as possible so that users can test their code with it. > we should maybe start printing out a warning in the next release. A warning will be somewhat cumbersome to users. The change won't affect, say, 90% of the code using "%", but it will require everyone to care about it. I'm not saying we shouldn't do that, just that it's a heavy thing to do (that, is: if we don't kill performance entirely and check the value at runtime to emit a warning only when it's necessary). Stefan From stefan_ml at behnel.de Tue Mar 24 07:34:52 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 24 Mar 2009 07:34:52 +0100 Subject: [Cython] Fix for ticket #135 In-Reply-To: References: <49C13044.2090808@behnel.de> <49C1368B.3010007@student.matnat.uio.no> Message-ID: <49C87F0C.6070302@behnel.de> Hi, Kurt Smith wrote: > Would it be worthwhile to pursue a fix for #135 by modifying code in > the PostParse phase as Dag mentions here, or is the fix going to be > subsumed by 'fixing' ticket #144? Sorry, I totally misread "return_type" as "result_value" (head too deep in ticket 144, I guess). So this won't be fixed by #144 and setting "return_type" to None where you did is actually a viable solution. Using PostParse sounds like a better approach, though. Stefan From stefan_ml at behnel.de Tue Mar 24 07:43:42 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 24 Mar 2009 07:43:42 +0100 Subject: [Cython] loop semantics In-Reply-To: References: <49BE1093.7080505@student.matnat.uio.no> <49C160F2.6000803@canterbury.ac.nz> <44276.213.61.181.86.1237555489.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5d7a85e992ed2a63b100875c371e1b0f.squirrel@webmail.uio.no> <49C667C1.8020703@behnel.de> <49C743EC.10300@student.matnat.uio.no> <63760.213.61.181.86.1237797099.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49C8811E.2080405@behnel.de> Robert Bradshaw wrote: > On Mar 23, 2009, at 1:31 AM, Stefan Behnel wrote: >> Looks like we really need a Wiki page here that clearly specifies the >> semantics of both loops in all corner cases... > > http://wiki.cython.org/loops , according to how I think of them. Yep, that sounds about right. Stefan From stefan_ml at behnel.de Mon Mar 23 12:40:55 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 23 Mar 2009 12:40:55 +0100 Subject: [Cython] constant handling refactored and up on cython-unstable (ticket 144) In-Reply-To: <49C2557F.3090402@student.matnat.uio.no> References: <49C15F04.2020504@behnel.de> <49C24679.9080300@student.matnat.uio.no> <65193.213.61.181.86.1237472125.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C2557F.3090402@student.matnat.uio.no> Message-ID: <49C77547.4010702@behnel.de> Dag Sverre Seljebotn wrote: > Is it impossible for constant folding to happen before type annotation? Fxied. Soon to be up on cython-unstable. Stefan From stefan_ml at behnel.de Mon Mar 23 13:07:58 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 23 Mar 2009 13:07:58 +0100 Subject: [Cython] too late to review and push patch for better handling of integral C types ? In-Reply-To: <49B8A817.1090609@behnel.de> References: <49B584AC.6040506@behnel.de> <49B8A817.1090609@behnel.de> Message-ID: <49C77B9E.40504@behnel.de> Stefan Behnel wrote: > Robert Bradshaw wrote: >> It's too late for 0.11 though. Is there a ticket up? > > Don't think so, but while we're at it, I'd like to have a ticket for any > change going into 0.11.1. Lisandro, could you open one and post the patch > there? Since this patch is very well backed by a test (much better than many other patches that went in), I'm ok with committing it for 0.11.1. Very nice work, Lisandro. Could you give it a ticket and push patch and test case to cython-devel? Thanks, Stefan From robertwb at math.washington.edu Tue Mar 24 07:52:31 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 23 Mar 2009 23:52:31 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49C87C5A.80607@behnel.de> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <49BEDD8E.1000007@canterbury.ac.nz> <49BF426D.7010005@behnel.de> <49BF5871.4040404@student.matnat.uio.no> <50876.213.61.181.86.1237279705.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C14F0C.4030000@behnel.de> <49C87C5A.80607@behnel.de> Message-ID: On Mar 23, 2009, at 11:23 PM, Stefan Behnel wrote: > > Robert Bradshaw wrote: >> On Mar 18, 2009, at 12:44 PM, Stefan Behnel wrote: >>> Robert Bradshaw wrote: >>>> I started a CEP up at http://wiki.cython.org/enhancements/division >>>> which attempts to summarize the issues brought up on this thread. >>> I added the pros and cons that I could come up with to each >>> solution. Maybe >>> others can add to that. >> >> I think we've waited long enough for everyone to give input. I say we >> change the default to python-style division, implementing a compiler >> directive and cython.cdiv/cmod methods to obtain the old behavior. > > +1 > > >> This is a big enough change it should probably wait 'till 0.12 > > It should still go into cython-unstable as soon as possible so that > users > can test their code with it. Yep, will do. >> we should maybe start printing out a warning in the next release. > > A warning will be somewhat cumbersome to users. The change won't > affect, > say, 90% of the code using "%", but it will require everyone to > care about > it. I'm not saying we shouldn't do that, just that it's a heavy > thing to do > (that, is: if we don't kill performance entirely and check the > value at > runtime to emit a warning only when it's necessary). I'm thinking a single (compile time) warning if % is used on cdef ints. - Robert From dagss at student.matnat.uio.no Tue Mar 24 14:30:47 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 24 Mar 2009 14:30:47 +0100 Subject: [Cython] Quick Fortran support poll Message-ID: <49C8E087.4080505@student.matnat.uio.no> OK I got some response on my GSoC Fortran project call, so in interest of that I'd like to have a quick poll (preferably without going into details at this stage): Is it OK to have nice and convenient wrapping of Fortran libraries as an additional feature of core Cython? [ ] Yes, that would be grand [ ] No, that would bloat the language and/or the compiler/project too much, I'd prefer to see this in a standalone tool/plugin The most basic examples could be done at a "C library level", however delegating this to a C library or tool one wouldn't be able to simply pass an "object[int, ndim=2]" to Fortran and have Fortran access it through the buffer interface (as it lacks a canonical C representation). So making the Python buffer interface and Fortran array types play nicely together is a reason to have Cython-specific support for this. Dag Sverre From dsurviver at gmail.com Tue Mar 24 14:41:31 2009 From: dsurviver at gmail.com (Danilo Freitas) Date: Tue, 24 Mar 2009 10:41:31 -0300 Subject: [Cython] About GSoC 2009 In-Reply-To: References: <49C7A906.6040502@student.matnat.uio.no> <49C7AB08.10906@student.matnat.uio.no> Message-ID: <45239150903240641x2334ebf3xf7b5a659e78eb5f3@mail.gmail.com> Even with a know knowledge about Cython, I'm interested in participate of GSoC. I thought it would be interesting work on C++ Support, even why there are many important features in C++ that could be used to improve Cython, just like STL. If I get a mentor for it, I'd be grateful Danilo 2009/3/23 Robert Bradshaw : > On Mar 23, 2009, at 12:08 PM, Anoop Saldanha wrote: > >> 2009/3/23 Prajwal Suhas P >> I'm interested in both "control flow analysis" and the "python 3" >> compatibility for gsoc 2009. I was reading about "control flow >> analysis" and i mainly have it in my mind over python 3 >> compatibility and it interested me more than python3 compatibility. >> If someone can mentor me for this it would be very nice. >> >> I would like to mentor you for control flow analysis, but I don't >> have much idea on how this works inside cython. > > Right now there isn't any (well, there isn't any that's worth keeping). > >> Looking into control flow analysis for quite sometime now, since I >> wanted to have a go at this project, but i have still a lot to >> understand. ?With a co-mentor I should be able to collaborate in a >> much better way with you. ?It would also get me started on >> understanding cython better. ?If someone wants to solely mentor you >> it is fine with me. ?Otherwise I am okay with being a mentor, with >> a co-mentor alongside. > > > I could probably co-mentor you for a project like this. > > - Robert > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From ilmarw at simula.no Tue Mar 24 14:50:13 2009 From: ilmarw at simula.no (Ilmar Wilbers) Date: Tue, 24 Mar 2009 14:50:13 +0100 Subject: [Cython] Quick Fortran support poll In-Reply-To: <49C8E087.4080505@student.matnat.uio.no> References: <49C8E087.4080505@student.matnat.uio.no> Message-ID: <49C8E515.9010301@simula.no> > [X] Yes, that would be grand > IMHO. ilmar From vic.kelson at gmail.com Tue Mar 24 14:52:04 2009 From: vic.kelson at gmail.com (Vic Kelson) Date: Tue, 24 Mar 2009 09:52:04 -0400 Subject: [Cython] Quick Fortran support poll In-Reply-To: <49C8E087.4080505@student.matnat.uio.no> References: <49C8E087.4080505@student.matnat.uio.no> Message-ID: > > [X] Yes, that would be grand > Thanks for asking! Vic Kelson -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090324/a8e8cce3/attachment.htm From stefan_ml at behnel.de Tue Mar 24 15:11:57 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 24 Mar 2009 15:11:57 +0100 (CET) Subject: [Cython] Quick Fortran support poll In-Reply-To: <49C8E087.4080505@student.matnat.uio.no> References: <49C8E087.4080505@student.matnat.uio.no> Message-ID: <6e449ed14def4abf26ebf4448966eff0.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > Is it OK to have nice and convenient wrapping of Fortran libraries as an > additional feature of core Cython? > > [ ] Yes, that would be grand > [ ] No, that would bloat the language and/or the compiler/project too > much, I'd prefer to see this in a standalone tool/plugin [X] It depends on how it's done. Cython is about generating fast C code from Python code. Interfacing with Fortran code is a worthy goal (even beyond the GCC world), but letting users write Fortran code in Cython is not quite within the project scope, IMHO. If the core feature is like "well, we have all those Fortran types, so we should make them first class types in Cython, like fint and ffloat, and then overload all operators for them, do all the coercion from and to C types under the hood, and then we need an 'fdef' keyword to define Fortran functions, and maybe some other stuff, too", then I would consider that akin to language bloat. Stefan From dagss at student.matnat.uio.no Tue Mar 24 15:32:12 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 24 Mar 2009 15:32:12 +0100 Subject: [Cython] Quick Fortran support poll In-Reply-To: <6e449ed14def4abf26ebf4448966eff0.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49C8E087.4080505@student.matnat.uio.no> <6e449ed14def4abf26ebf4448966eff0.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49C8EEEC.1070005@student.matnat.uio.no> Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: > >> Is it OK to have nice and convenient wrapping of Fortran libraries as an >> additional feature of core Cython? >> >> [ ] Yes, that would be grand >> [ ] No, that would bloat the language and/or the compiler/project too >> much, I'd prefer to see this in a standalone tool/plugin >> > > [X] It depends on how it's done. > OK, some details then. This is a scetch only! First of all, this is about calling into Fortran (i.e. calling conventions) only, not about writing Fortran code using Cython. 1) A "fortran" calling convention on external function declarations, and perhaps a "module"/"namespace" declaration (which one could reuse for C++): cdef extern: module mymodule: # reusable for C++ wrapping, could say "namespace" instead cdef fortran sometype myfunc(sometype arg) When "fortran" is in effect, one may accept some extra syntax for the arguments, i.e. "in" or "inout" (but "const" and no const could serve the same purpose). 2) Types -- I am thinking of a "cython.fortran" psuedomodule with Fortran types (i.e. cython.fortran.real(kind=8)); these would simply map down to C typedefs. But one could get by with a simple pxd with no compiler support as well. 3) Potentially parsing of Fortran files to autogenerate pxd files, but this could be a seperate tool and/or be pluggable using an API that could also be used for header files. Implementation-wise, there would be a transform which would deal with calls to functions using the Fortran calling convention, and generates a mymodule.f90 wrapper to supplement mymodule.c -- however this is a very thin wrapper to retype the Fortran functions from Fortran-side to use some C conventions and some Cython-specific conventions, and most of the things would be C-side. The argument for doing this in Cython is that both Cython/Python and Fortran support a higher-level construct which C doesn't (buffers/strided arrays). How does that make you feel? Alternatives: - Plug it in through a plugin framework - Do it through a preprocessing step which generates the necesarry .pyx and .f90 files. Dag Sverre From ggellner at uoguelph.ca Tue Mar 24 15:48:41 2009 From: ggellner at uoguelph.ca (Gabriel Gellner) Date: Tue, 24 Mar 2009 10:48:41 -0400 Subject: [Cython] Quick Fortran support poll In-Reply-To: <49C8E087.4080505@student.matnat.uio.no> References: <49C8E087.4080505@student.matnat.uio.no> Message-ID: <20090324144841.GA17117@encolpuis> > [X] No, that would bloat the language and/or the compiler/project too > much, I'd prefer to see this in a standalone tool/plugin > Just my opinion, but f2py rocks at this, so it seems like a distraction to me. Gabriel From vic.kelson at gmail.com Tue Mar 24 15:58:39 2009 From: vic.kelson at gmail.com (Vic Kelson) Date: Tue, 24 Mar 2009 10:58:39 -0400 Subject: [Cython] Quick Fortran support poll In-Reply-To: <20090324144841.GA17117@encolpuis> References: <49C8E087.4080505@student.matnat.uio.no> <20090324144841.GA17117@encolpuis> Message-ID: To me, the problem with using Cython with F2PY extensions is that Cython would need to construct Python objects that would then be unpacked into Fortran-callable functions by the F2PY-generated module. Since the purpose of the programmer who is trying to call Fortran routines with Cython is to improve code performance, this is an obvious bottleneck. I like the proposed approach, and I'm not sure at all that we need the Fortran scanner (unless Cython scans an INTERFACE block), because it's a lot of work. The proposed approach offers the programmer the ability to call Fortran from Cython without having to deal with all the ins and outs of C/Fortran integration. All I'd like to be able to do is to have some syntactical cues that I can use to tell Cython that I have a function in Fortran that takes some arguments and returns something, and have Cython handle the dirty work. I may be wrong, but that seems a lot less intensive than F2PY, and takes advantage of the power of Cython. Thanks. --v On Tue, Mar 24, 2009 at 10:48 AM, Gabriel Gellner wrote: > > [X] No, that would bloat the language and/or the compiler/project too > > much, I'd prefer to see this in a standalone tool/plugin > > > Just my opinion, but f2py rocks at this, so it seems like a distraction to > me. > > Gabriel > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Vic Kelson vic.kelson at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090324/d67d34bd/attachment.htm From dagss at student.matnat.uio.no Tue Mar 24 16:00:55 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 24 Mar 2009 16:00:55 +0100 Subject: [Cython] Quick Fortran support poll In-Reply-To: <20090324144841.GA17117@encolpuis> References: <49C8E087.4080505@student.matnat.uio.no> <20090324144841.GA17117@encolpuis> Message-ID: <49C8F5A7.8040302@student.matnat.uio.no> Gabriel Gellner wrote: >> [X] No, that would bloat the language and/or the compiler/project too >> much, I'd prefer to see this in a standalone tool/plugin >> >> > Just my opinion, but f2py rocks at this, so it seems like a distraction to me. > That's certainly a valid argument against this. In my opinion, using f2py is a bit like using SWIG or ctypes; though we are still using Cython? When I'm using Cython I'd like Fortran access to be closer to the metal and not do the Fortran call through the entire Python stack. (Also f2py needs some work to wrap the libraries I work with (using assumed-shape arrays), and doesn't pass non-contiguous buffers and non-NumPy PEP-3118 buffers. As far as I understand anyway.) Dag Sverre From stefan_ml at behnel.de Tue Mar 24 16:02:56 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 24 Mar 2009 16:02:56 +0100 (CET) Subject: [Cython] Quick Fortran support poll In-Reply-To: <49C8EEEC.1070005@student.matnat.uio.no> References: <49C8E087.4080505@student.matnat.uio.no> <6e449ed14def4abf26ebf4448966eff0.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C8EEEC.1070005@student.matnat.uio.no> Message-ID: <4ee28a1e6aaad9b714d54b78cfdaf2d0.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > 1) A "fortran" calling convention on external function declarations, and > perhaps a "module"/"namespace" declaration (which one could reuse for > C++): > > cdef extern: > module mymodule: # reusable for C++ wrapping, could say "namespace" > instead > cdef fortran sometype myfunc(sometype arg) Fine with me. What about cdef extern from "myheader.h": for C and cdef extern from module "mymodule": for Fortran and cdef extern from namespace "mynamespace": for C++, which would automatically enable C, C++ and Fortran conventions for their body? (i.e. without a "fortran" or "module" keyword). > When "fortran" is in effect, one may accept some extra syntax for the > arguments, i.e. "in" or "inout" (but "const" and no const could serve > the same purpose). I'm for using in/inout (maybe as a Py3 annotation or whatever) rather than overloading "const". > 2) Types -- I am thinking of a "cython.fortran" psuedomodule with > Fortran types (i.e. cython.fortran.real(kind=8)); these would simply map > down to C typedefs. But one could get by with a simple pxd with no > compiler support as well. I'm not sure what would be a good way to handle this. If it's an external .pxd, I don't mind. But requiring compiler support here may open a can of worms. > 3) Potentially parsing of Fortran files to autogenerate pxd files, but > this could be a seperate tool and/or be pluggable using an API that > could also be used for header files. Sounds like a separate tool to me. > Implementation-wise, there would be a transform which would deal with > calls to functions using the Fortran calling convention, and generates a > mymodule.f90 wrapper to supplement mymodule.c -- however this is a very > thin wrapper to retype the Fortran functions from Fortran-side to use > some C conventions and some Cython-specific conventions, and most of the > things would be C-side. Wouldn't the external .pxd generator tool be the right place to also generate the .f90 wrapper? I don't see Cython generate a Fortran module while compiling a Cython module to C code that *calls* the module. Maybe a second backend to F2Py would do the trick already? Stefan From dagss at student.matnat.uio.no Tue Mar 24 16:04:28 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 24 Mar 2009 16:04:28 +0100 Subject: [Cython] Quick Fortran support poll In-Reply-To: References: <49C8E087.4080505@student.matnat.uio.no> <20090324144841.GA17117@encolpuis> Message-ID: <49C8F67C.4060106@student.matnat.uio.no> Vic Kelson wrote: > > I like the proposed approach, and I'm not sure at all that we need the > Fortran scanner (unless Cython scans an INTERFACE block), because it's > a lot of work. The proposed approach offers the programmer the ability > to call Fortran from Cython without having to deal with all the ins > and outs of C/Fortran integration. Indeed, with native Fortran calling convention in Cython, one could reuse the f2py Fortran scanner and generate the necesarry Cython pxd files from it (i.e. as a non-Cython utility script). Though I've seen one or two projects which seem to have higher-quality Fortran scanners than f2py that may be a better starting point. Dag Sverre From dagss at student.matnat.uio.no Tue Mar 24 16:17:25 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 24 Mar 2009 16:17:25 +0100 Subject: [Cython] Quick Fortran support poll In-Reply-To: <4ee28a1e6aaad9b714d54b78cfdaf2d0.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49C8E087.4080505@student.matnat.uio.no> <6e449ed14def4abf26ebf4448966eff0.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C8EEEC.1070005@student.matnat.uio.no> <4ee28a1e6aaad9b714d54b78cfdaf2d0.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49C8F985.3030102@student.matnat.uio.no> Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: > >> 1) A "fortran" calling convention on external function declarations, and >> perhaps a "module"/"namespace" declaration (which one could reuse for >> C++): >> >> cdef extern: >> module mymodule: # reusable for C++ wrapping, could say "namespace" >> instead >> cdef fortran sometype myfunc(sometype arg) >> > > Fine with me. What about > > cdef extern from "myheader.h": > > for C and > > cdef extern from module "mymodule": > > for Fortran and > > cdef extern from namespace "mynamespace": > > for C++, which would automatically enable C, C++ and Fortran conventions > for their body? (i.e. without a "fortran" or "module" keyword). > Comments: I like the explicitness of the "fortran" keyword, and C++ requires a header file to be included as well as a namespace. >> When "fortran" is in effect, one may accept some extra syntax for the >> arguments, i.e. "in" or "inout" (but "const" and no const could serve >> the same purpose). >> > > I'm for using in/inout (maybe as a Py3 annotation or whatever) rather than > overloading "const". > Actually, I slipped here. Pointers would be "inout", i.e. cdef fortran void myfunc(int* foo) # foo is inout argument I would *not* like to have Cython code where "myfunc(x)" can change the value of x :-) But for array/buffer arguments, "in"/"out"/"inout" would be nice to have. >> Implementation-wise, there would be a transform which would deal with >> calls to functions using the Fortran calling convention, and generates a >> mymodule.f90 wrapper to supplement mymodule.c -- however this is a very >> thin wrapper to retype the Fortran functions from Fortran-side to use >> some C conventions and some Cython-specific conventions, and most of the >> things would be C-side. >> > > Wouldn't the external .pxd generator tool be the right place to also > generate the .f90 wrapper? I don't see Cython generate a Fortran module > while compiling a Cython module to C code that *calls* the module. > I was hoping to do this without ever touching Fortran parsing at first, the way one does it with wrapping C. This would loose the possibility of working from a manually written pxd at first. Also the f90 wrapper would insert Cython-specific structs in the call signature to marshal PEP-3118 buffers across. So it doesn't seem to make so much sense to decouple the two (from an end-user perspective), though it could certainly be done that way. > Maybe a second backend to F2Py would do the trick already? > F2py does the parsing-from-Fortran-source stuff, but that is not at all the focus here. Whatever happens, noone will rewrite a Fortran parser :-) Dag Sverre From carl.witty at gmail.com Tue Mar 24 16:46:10 2009 From: carl.witty at gmail.com (Carl Witty) Date: Tue, 24 Mar 2009 08:46:10 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: <49BE6AC1.3000303@molden.no> <49BEDD8E.1000007@canterbury.ac.nz> <49BF426D.7010005@behnel.de> <49BF5871.4040404@student.matnat.uio.no> <50876.213.61.181.86.1237279705.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C14F0C.4030000@behnel.de> <49C87C5A.80607@behnel.de> Message-ID: On Mon, Mar 23, 2009 at 11:52 PM, Robert Bradshaw wrote: >> A warning will be somewhat cumbersome to users. The change won't >> affect, >> say, 90% of the code using "%", but it will require everyone to >> care about >> it. I'm not saying we shouldn't do that, just that it's a heavy >> thing to do >> (that, is: if we don't kill performance entirely and check the >> value at >> runtime to emit a warning only when it's necessary). > > I'm thinking a single (compile time) warning if % is used on cdef ints. It would be nice if the "kill performance entirely and check the value at runtime to emit a warning only when it's necessary" option were also available (under the control of a compiler flag, probably, and definitely not by default). Then somebody could compile Sage with that flag and see if doctesting reveals any problems. Carl From vic.kelson at gmail.com Tue Mar 24 16:51:56 2009 From: vic.kelson at gmail.com (Vic Kelson) Date: Tue, 24 Mar 2009 11:51:56 -0400 Subject: [Cython] Quick Fortran support poll In-Reply-To: <49C8F985.3030102@student.matnat.uio.no> References: <49C8E087.4080505@student.matnat.uio.no> <6e449ed14def4abf26ebf4448966eff0.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C8EEEC.1070005@student.matnat.uio.no> <4ee28a1e6aaad9b714d54b78cfdaf2d0.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C8F985.3030102@student.matnat.uio.no> Message-ID: > I would *not* like to have Cython code where "myfunc(x)" can change the > value of x :-) F2PY handles intent(out) arguments by returning them as a tuple from the function. That's one way to handle it. --v -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090324/6e90d5c1/attachment.htm From cournape at gmail.com Tue Mar 24 16:52:41 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 25 Mar 2009 00:52:41 +0900 Subject: [Cython] Quick Fortran support poll In-Reply-To: <49C8F5A7.8040302@student.matnat.uio.no> References: <49C8E087.4080505@student.matnat.uio.no> <20090324144841.GA17117@encolpuis> <49C8F5A7.8040302@student.matnat.uio.no> Message-ID: <5b8d13220903240852n777cebe0k30f1a4c404e04f64@mail.gmail.com> On Wed, Mar 25, 2009 at 12:00 AM, Dag Sverre Seljebotn wrote: > Gabriel Gellner wrote: >>> ?[X] No, that would bloat the language and/or the compiler/project too >>> much, I'd prefer to see this in a standalone tool/plugin >>> >>> >> Just my opinion, but f2py rocks at this, so it seems like a distraction to me. >> > That's certainly a valid argument against this. One potentially big advantage of cython is the ability to generate C code compatible with python 3. I don't know how much work would be required for f2py to do this, but if that's a non trivial task, then it definitely make sense for cython to do this. I don't know enough about cython architecture to know whether a plugin or first-class support is better. cheers, David From stefan_ml at behnel.de Tue Mar 24 17:02:52 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 24 Mar 2009 17:02:52 +0100 (CET) Subject: [Cython] Quick Fortran support poll In-Reply-To: <49C8F985.3030102@student.matnat.uio.no> References: <49C8E087.4080505@student.matnat.uio.no> <6e449ed14def4abf26ebf4448966eff0.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C8EEEC.1070005@student.matnat.uio.no> <4ee28a1e6aaad9b714d54b78cfdaf2d0.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C8F985.3030102@student.matnat.uio.no> Message-ID: <8d39e0611b58d306ccabb4df75084d61.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Dag Sverre Seljebotn wrote: >> >>> 1) A "fortran" calling convention on external function declarations, >>> and >>> perhaps a "module"/"namespace" declaration (which one could reuse for >>> C++): >>> >>> cdef extern: >>> module mymodule: # reusable for C++ wrapping, could say "namespace" >>> instead >>> cdef fortran sometype myfunc(sometype arg) >>> >> >> Fine with me. What about >> >> cdef extern from "myheader.h": >> >> for C and >> >> cdef extern from module "mymodule": >> >> for Fortran and >> >> cdef extern from namespace "mynamespace": >> >> for C++, which would automatically enable C, C++ and Fortran conventions >> for their body? (i.e. without a "fortran" or "module" keyword). >> > Comments: I like the explicitness of the "fortran" keyword, and C++ > requires a header file to be included as well as a namespace. cdef extern from fortran module "mymodule": cdef extern from "cppheader.H" namespace "mynamespace": ? I'd really like to keep the syntax simple here. You will never have anything but Fortran conventions in a Fortran module, so requiring a sub-block complicates things unnecessarily. >>> When "fortran" is in effect, one may accept some extra syntax for the >>> arguments, i.e. "in" or "inout" (but "const" and no const could serve >>> the same purpose). >> >> I'm for using in/inout (maybe as a Py3 annotation or whatever) rather >> than overloading "const". >> > Actually, I slipped here. Pointers would be "inout", i.e. > > cdef fortran void myfunc(int* foo) # foo is inout argument > > I would *not* like to have Cython code where "myfunc(x)" can change the > value of x :-) > > But for array/buffer arguments, "in"/"out"/"inout" would be nice to have. In that case, could they be declared as part of the array/buffer declaration (inside the []), instead of becoming an external part of the variable declaration? >>> Implementation-wise, there would be a transform which would deal with >>> calls to functions using the Fortran calling convention, and generates a >>> mymodule.f90 wrapper to supplement mymodule.c -- however this is a very >>> thin wrapper to retype the Fortran functions from Fortran-side to use >>> some C conventions and some Cython-specific conventions, and most of >>> the things would be C-side. >> >> Wouldn't the external .pxd generator tool be the right place to also >> generate the .f90 wrapper? I don't see Cython generate a Fortran module >> while compiling a Cython module to C code that *calls* the module. >> > I was hoping to do this without ever touching Fortran parsing at first, > the way one does it with wrapping C. This would loose the possibility of > working from a manually written pxd at first. Why would you write a .pxd by hand if you can just run a little tool on your Fortran module? :-) That said, generating the .f90 module from the .pxd should be no different from generating it from the .pyx sources, right? So it doesn't matter where you get your .pxd from. The only question is what generates the F90 wrapper, and I don't think it's the compiler that compiles the Cython sources. In most use cases I can see, you will generate the wrapper once (or once in a while), but your Cython code many, many times during development. So this really is a different tool that generates different code (from the same .pxd file). > Also the f90 wrapper would insert Cython-specific structs in the call > signature to marshal PEP-3118 buffers across. How is that something that must be done during the compilation of the *calling* code? It's the *called* code that is affected here. Stefan From dalcinl at gmail.com Tue Mar 24 17:05:06 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 24 Mar 2009 13:05:06 -0300 Subject: [Cython] too late to review and push patch for better handling of integral C types ? In-Reply-To: <49C77B9E.40504@behnel.de> References: <49B584AC.6040506@behnel.de> <49B8A817.1090609@behnel.de> <49C77B9E.40504@behnel.de> Message-ID: Stefan, I'm currently very busy trying to get a new release of two of my projects... My original patch still needs a bit of discussion, and there is room for improvement, as I want to extend that machinery to "extern ctypedef" integrals in such a way that Cython stop assuming a base type and instead emit a custom conversor. So (if you can) open the ticket for me, upload the curent patch, and wait me a bit until I can handle further work on this. On Mon, Mar 23, 2009 at 9:07 AM, Stefan Behnel wrote: > > Stefan Behnel wrote: >> Robert Bradshaw wrote: >>> It's too late for 0.11 though. Is there a ticket up? >> >> Don't think so, but while we're at it, I'd like to have a ticket for any >> change going into 0.11.1. Lisandro, could you open one and post the patch >> there? > > Since this patch is very well backed by a test (much better than many other > patches that went in), I'm ok with committing it for 0.11.1. > > Very nice work, Lisandro. Could you give it a ticket and push patch and > test case to cython-devel? > > Thanks, > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From kwmsmith at gmail.com Tue Mar 24 17:21:43 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Tue, 24 Mar 2009 11:21:43 -0500 Subject: [Cython] Fix for ticket #135 In-Reply-To: References: <49C13044.2090808@behnel.de> <49C1368B.3010007@student.matnat.uio.no> <02DFEEFA-4143-4F00-971F-30FD91D92324@math.washington.edu> Message-ID: Here's the addition of a test case. I added a visit_ReturnStatNode in PostParse (not in the patch below), but it seems there is some issue with error handling. If I raise a PostParseError, everything is fine and the compiler stops there. But when there are multiple tests in the test case (see below) the later errors (3:4 and 6:4) aren't tested. It seems that the best route is to call the error(pos, message) function and have it record the errors; this works for other stages in the pipeline (AnalyseDeclarationsTransform, e.g.) but fails at the PostParse stage (the compiler barfs). I reckon something like a standardized error reporting/handling would be a good thing to do, but I'm not sure why the current error handling is in place. PostParse raises PostParseError which will stop the compiler in its tracks, while other stages call error() to allow all errors to be reported after the pipeline is run. # HG changeset patch # User Kurt Smith # Date 1237857818 18000 # Node ID 346d8d3955293edb37498231d76799f0926c561a # Parent e2365a6d00b8d0cdb665699e840a8f052d471e37 Added test case for ticket 135 in tests/bugs. diff -r e2365a6d00b8 -r 346d8d395529 tests/bugs/return_outside_function_T135.pyx --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/tests/bugs/return_outside_function_T135.pyx Mon Mar 23 20:23:38 2009 -0500 @@ -0,0 +1,12 @@ +return 'bar' +class A: + return None + +cdef class B: + return None + +_ERRORS = u''' +1:0: Return not inside a function body +3:4: Return not inside a function body +6:4: Return not inside a function body +''' From dagss at student.matnat.uio.no Tue Mar 24 19:15:04 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 24 Mar 2009 19:15:04 +0100 Subject: [Cython] Quick Fortran support poll In-Reply-To: <8d39e0611b58d306ccabb4df75084d61.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49C8E087.4080505@student.matnat.uio.no> <6e449ed14def4abf26ebf4448966eff0.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C8EEEC.1070005@student.matnat.uio.no> <4ee28a1e6aaad9b714d54b78cfdaf2d0.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C8F985.3030102@student.matnat.uio.no> <8d39e0611b58d306ccabb4df75084d61.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49C92328.2040504@student.matnat.uio.no> Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> Actually, I slipped here. Pointers would be "inout", i.e. >> >> cdef fortran void myfunc(int* foo) # foo is inout argument >> >> I would *not* like to have Cython code where "myfunc(x)" can change the >> value of x :-) >> >> But for array/buffer arguments, "in"/"out"/"inout" would be nice to have. > > In that case, could they be declared as part of the array/buffer > declaration (inside the []), instead of becoming an external part of the > variable declaration? Yes, I think so. Actually, I think passing acquired buffers to/from C code would be nice to have, and could be done using the same mechanism. in/out/inout is needed then as well, so this is not Fortran-specific. > The only question is what generates the F90 wrapper, and I don't think > it's the compiler that compiles the Cython sources. In most use cases I > can see, you will generate the wrapper once (or once in a while), but your > Cython code many, many times during development. So this really is a > different tool that generates different code (from the same .pxd file). Yes, I can see where you are coming from and it is certainly something to consider. On one end, work on f2py to in addition generate Cython wrapper around a generated C interface for Fortran. On the other end, add Cython syntax candy for passing acquired buffers to C code. Taken together, they should get us there without specific Fortran support in Cython. I think I'll/we'll need to think more about it and perhaps CEP it with pros and cons and get back to it. I'm getting more ideas as I write here. Thanks for stimulating feedback. -- Dag Sverre From dagss at student.matnat.uio.no Tue Mar 24 19:20:23 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 24 Mar 2009 19:20:23 +0100 Subject: [Cython] Fix for ticket #135 In-Reply-To: References: <49C13044.2090808@behnel.de> <49C1368B.3010007@student.matnat.uio.no> <02DFEEFA-4143-4F00-971F-30FD91D92324@math.washington.edu> Message-ID: <49C92467.5050201@student.matnat.uio.no> Kurt Smith wrote: > Here's the addition of a test case. > > I added a visit_ReturnStatNode in PostParse (not in the patch below), > but it seems there is some issue with error handling. If I raise a > PostParseError, everything is fine and the compiler stops there. But > when there are multiple tests in the test case (see below) the later > errors (3:4 and 6:4) aren't tested. It seems that the best route is > to call the error(pos, message) function and have it record the > errors; this works for other stages in the pipeline > (AnalyseDeclarationsTransform, e.g.) but fails at the PostParse stage > (the compiler barfs). Indeed, this is one of the unclean things (of many) in Cython where different people have used different conventions. Myself I would like to never call Errors.error from transforms, because then you get full Inversion of Control and can much more easily unit-test. I suppose we should just agree on something (in fact we may have agreed on using Errors.error, I can't remember). Stefan, Robert? For now, try my preferred IoC way; self.context.nonfatal_error(PostParseError(...)). Also, proceed to "return None" which will delete the node from the tree before further processing. -- Dag Sverre From dagss at student.matnat.uio.no Tue Mar 24 19:45:35 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 24 Mar 2009 19:45:35 +0100 Subject: [Cython] Quick Fortran support poll In-Reply-To: <5b8d13220903240852n777cebe0k30f1a4c404e04f64@mail.gmail.com> References: <49C8E087.4080505@student.matnat.uio.no> <20090324144841.GA17117@encolpuis> <49C8F5A7.8040302@student.matnat.uio.no> <5b8d13220903240852n777cebe0k30f1a4c404e04f64@mail.gmail.com> Message-ID: <49C92A4F.3010605@student.matnat.uio.no> David Cournapeau wrote: > On Wed, Mar 25, 2009 at 12:00 AM, Dag Sverre Seljebotn > wrote: >> Gabriel Gellner wrote: >>>> [X] No, that would bloat the language and/or the compiler/project too >>>> much, I'd prefer to see this in a standalone tool/plugin >>>> >>>> >>> Just my opinion, but f2py rocks at this, so it seems like a distraction to me. >>> >> That's certainly a valid argument against this. > > One potentially big advantage of cython is the ability to generate C > code compatible with python 3. I don't know how much work would be > required for f2py to do this, but if that's a non trivial task, then > it definitely make sense for cython to do this. I don't know about f2py w/ respect to Py3 either. But in general, there's certain overlap in implementation between Cython and f2py (basically f2py does directly much of what Cython provides), which is a waste of developer resources anyway (well, if Cython is gearing up for more direct Fortran support, one way or another, over summer). I can certainly see Cython replacing or becoming an additional backend; so that f2py long-term becomes a tool which parses Fortran and generates Cython and Fortran sources instead of C and Fortran sources. We should keep in touch on GSoC projects related to this, I think a joint Cython/f2py project could be fruitful. (Is there anybody but you I should ping over at the SciPy camp about this?) -- Dag Sverre From dagss at student.matnat.uio.no Tue Mar 24 19:49:32 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 24 Mar 2009 19:49:32 +0100 Subject: [Cython] too late to review and push patch for better handling of integral C types ? In-Reply-To: References: <49B584AC.6040506@behnel.de> <49B8A817.1090609@behnel.de> <49C77B9E.40504@behnel.de> Message-ID: <49C92B3C.6040506@student.matnat.uio.no> Lisandro Dalcin wrote: > Stefan, I'm currently very busy trying to get a new release of two of > my projects... > > My original patch still needs a bit of discussion, and there is room > for improvement, as I want to extend that machinery to "extern > ctypedef" integrals in such a way that Cython stop assuming a base > type and instead emit a custom conversor. Excellent! As you probably know the buffer interface already does not make such assumptions. So basically, cdef extern: ctypedef short a ctypedef long b here, a and b should always be treated in the exact same way by Cython? +1! I think we should include floats though, so that the size of float vs. long double isn't assumed for external typedefs as well. -- Dag Sverre From dagss at student.matnat.uio.no Tue Mar 24 20:01:40 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 24 Mar 2009 20:01:40 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> <49BEDD8E.1000007@canterbury.ac.nz> <49BF426D.7010005@behnel.de> <49BF5871.4040404@student.matnat.uio.no> <50876.213.61.181.86.1237279705.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C14F0C.4030000@behnel.de> Message-ID: <49C92E14.1040008@student.matnat.uio.no> Robert Bradshaw wrote: > On Mar 18, 2009, at 12:44 PM, Stefan Behnel wrote: > >> Robert Bradshaw wrote: >>> I started a CEP up at http://wiki.cython.org/enhancements/division >>> which attempts to summarize the issues brought up on this thread. >> I added the pros and cons that I could come up with to each >> solution. Maybe >> others can add to that. > > I think we've waited long enough for everyone to give input. I say we > change the default to python-style division, implementing a compiler > directive and cython.cdiv/cmod methods to obtain the old behavior. +1 -- Dag Sverre From dagss at student.matnat.uio.no Tue Mar 24 20:16:29 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 24 Mar 2009 20:16:29 +0100 Subject: [Cython] Fix for ticket #135 In-Reply-To: <49C92467.5050201@student.matnat.uio.no> References: <49C13044.2090808@behnel.de> <49C1368B.3010007@student.matnat.uio.no> <02DFEEFA-4143-4F00-971F-30FD91D92324@math.washington.edu> <49C92467.5050201@student.matnat.uio.no> Message-ID: <49C9318D.60904@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Kurt Smith wrote: >> Here's the addition of a test case. >> >> I added a visit_ReturnStatNode in PostParse (not in the patch below), >> but it seems there is some issue with error handling. If I raise a >> PostParseError, everything is fine and the compiler stops there. But >> when there are multiple tests in the test case (see below) the later >> errors (3:4 and 6:4) aren't tested. It seems that the best route is >> to call the error(pos, message) function and have it record the >> errors; this works for other stages in the pipeline >> (AnalyseDeclarationsTransform, e.g.) but fails at the PostParse stage >> (the compiler barfs). > > Indeed, this is one of the unclean things (of many) in Cython where > different people have used different conventions. Myself I would like to > never call Errors.error from transforms, because then you get full > Inversion of Control and can much more easily unit-test. > > I suppose we should just agree on something (in fact we may have agreed > on using Errors.error, I can't remember). Stefan, Robert? > > For now, try my preferred IoC way; > > self.context.nonfatal_error(PostParseError(...)). That was very bad advice, as it does the same as Errors.error :-) I'll try to have a look and get back here. -- Dag Sverre From dagss at student.matnat.uio.no Tue Mar 24 20:31:25 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 24 Mar 2009 20:31:25 +0100 Subject: [Cython] Fix for ticket #135 In-Reply-To: References: <49C13044.2090808@behnel.de> <49C1368B.3010007@student.matnat.uio.no> <02DFEEFA-4143-4F00-971F-30FD91D92324@math.washington.edu> Message-ID: <49C9350D.6090409@student.matnat.uio.no> Kurt Smith wrote: > I reckon something like a standardized error reporting/handling would > be a good thing to do, but I'm not sure why the current error handling > is in place. PostParse raises PostParseError which will stop the > compiler in its tracks, while other stages call error() to allow all > errors to be reported after the pipeline is run. OK, here's a better answer: A CVarDefNode cannot make sense without some of its components. So if a node inside a CVarDefNode causes an error, the whole CVarDefNode should not be processed further -- however following CVarDefNodes should be processed. Exceptions are the obvious way to tackle this. So what happens is that nodes that sits *inside* CVarDefNode raises PostParseError, then visit_CVarDefNode catches these and reports the error and removes the node from the tree (by returning None) and moves on. So this really is a different usecase. The inconsistency that's left is between self.context.nonfatal_error and error(pos, msg). They currently do the same thing; the latter is the real Cython way of doing things, while the former is something I wanted to introduce for transforms to get IoC but never got around to discussing/documenting/follow up. BTW calling error(pos, msg) in PostParse worked well for me now, so I don't know what that is about. It should work. (However I couldn't compile the refnanny in the current -devel?) -- Dag Sverre From robertwb at math.washington.edu Tue Mar 24 21:17:01 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 24 Mar 2009 13:17:01 -0700 Subject: [Cython] Fix for ticket #135 In-Reply-To: <49C92467.5050201@student.matnat.uio.no> References: <49C13044.2090808@behnel.de> <49C1368B.3010007@student.matnat.uio.no> <02DFEEFA-4143-4F00-971F-30FD91D92324@math.washington.edu> <49C92467.5050201@student.matnat.uio.no> Message-ID: <185204E1-506D-4897-BF7E-6978660E8107@math.washington.edu> On Mar 24, 2009, at 11:20 AM, Dag Sverre Seljebotn wrote: > Kurt Smith wrote: >> Here's the addition of a test case. >> >> I added a visit_ReturnStatNode in PostParse (not in the patch below), >> but it seems there is some issue with error handling. If I raise a >> PostParseError, everything is fine and the compiler stops there. But >> when there are multiple tests in the test case (see below) the later >> errors (3:4 and 6:4) aren't tested. It seems that the best route is >> to call the error(pos, message) function and have it record the >> errors; this works for other stages in the pipeline >> (AnalyseDeclarationsTransform, e.g.) but fails at the PostParse stage >> (the compiler barfs). > > Indeed, this is one of the unclean things (of many) in Cython where > different people have used different conventions. Myself I would > like to > never call Errors.error from transforms, because then you get full > Inversion of Control and can much more easily unit-test. > > I suppose we should just agree on something (in fact we may have > agreed > on using Errors.error, I can't remember). Stefan, Robert? One of my pet peeves is when I'm compiling something, and it gives me a single error, which i then fix, and it gives me one more, etc. This is why, when it's possible to continue, I think it's better to use Errors.error. Surprising it doesn't work though--was this some special code you put in Dag? > For now, try my preferred IoC way; > > self.context.nonfatal_error(PostParseError(...)). > > Also, proceed to "return None" which will delete the node from the > tree > before further processing. You make a good point about CVarDefNode, but I think this case (and most statements) are much simpler. - Robert From robertwb at math.washington.edu Tue Mar 24 21:17:59 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 24 Mar 2009 13:17:59 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: <49BE6AC1.3000303@molden.no> <49BEDD8E.1000007@canterbury.ac.nz> <49BF426D.7010005@behnel.de> <49BF5871.4040404@student.matnat.uio.no> <50876.213.61.181.86.1237279705.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C14F0C.4030000@behnel.de> <49C87C5A.80607@behnel.de> Message-ID: <1D4BF5BC-A491-4CE1-B965-D9D5CAD91D9C@math.washington.edu> On Mar 24, 2009, at 8:46 AM, Carl Witty wrote: > On Mon, Mar 23, 2009 at 11:52 PM, Robert Bradshaw > wrote: >>> A warning will be somewhat cumbersome to users. The change won't >>> affect, >>> say, 90% of the code using "%", but it will require everyone to >>> care about >>> it. I'm not saying we shouldn't do that, just that it's a heavy >>> thing to do >>> (that, is: if we don't kill performance entirely and check the >>> value at >>> runtime to emit a warning only when it's necessary). >> >> I'm thinking a single (compile time) warning if % is used on cdef >> ints. > > It would be nice if the "kill performance entirely and check the > value at > runtime to emit a warning only when it's necessary" option were also > available (under the control of a compiler flag, probably, and > definitely not by default). Then somebody could compile Sage with > that flag and see if doctesting reveals any problems. Sure. We have cython.cdivision(True), cython.cdivision(False), and ??? - Robert From dagss at student.matnat.uio.no Tue Mar 24 21:37:52 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 24 Mar 2009 21:37:52 +0100 Subject: [Cython] Fix for ticket #135 In-Reply-To: <185204E1-506D-4897-BF7E-6978660E8107@math.washington.edu> References: <49C13044.2090808@behnel.de> <49C1368B.3010007@student.matnat.uio.no> <02DFEEFA-4143-4F00-971F-30FD91D92324@math.washington.edu> <49C92467.5050201@student.matnat.uio.no> <185204E1-506D-4897-BF7E-6978660E8107@math.washington.edu> Message-ID: <49C944A0.5080203@student.matnat.uio.no> Robert Bradshaw wrote: > On Mar 24, 2009, at 11:20 AM, Dag Sverre Seljebotn wrote: > >> Kurt Smith wrote: >>> Here's the addition of a test case. >>> >>> I added a visit_ReturnStatNode in PostParse (not in the patch below), >>> but it seems there is some issue with error handling. If I raise a >>> PostParseError, everything is fine and the compiler stops there. But >>> when there are multiple tests in the test case (see below) the later >>> errors (3:4 and 6:4) aren't tested. It seems that the best route is >>> to call the error(pos, message) function and have it record the >>> errors; this works for other stages in the pipeline >>> (AnalyseDeclarationsTransform, e.g.) but fails at the PostParse stage >>> (the compiler barfs). >> Indeed, this is one of the unclean things (of many) in Cython where >> different people have used different conventions. Myself I would >> like to >> never call Errors.error from transforms, because then you get full >> Inversion of Control and can much more easily unit-test. >> >> I suppose we should just agree on something (in fact we may have >> agreed >> on using Errors.error, I can't remember). Stefan, Robert? > > One of my pet peeves is when I'm compiling something, and it gives me > a single error, which i then fix, and it gives me one more, etc. This > is why, when it's possible to continue, I think it's better to use > Errors.error. I think this misrepresents the issue. Currently error(...) and self.context.nonfatal_error(PostParseError(...)) does almost the EXACT SAME thing, especially no exception is raised. The difference is that the self.context indirection allows one to plug in a different error reciever for a transform in some situations. As transforms are otherwise completely isolated, I don't like breaking IoC for this one case. But it's not important. But either context.nonfatal_error should be removed, or all calls to "error" in transforms should be replaced with context.nonfatal_error. -- Dag Sverre From robertwb at math.washington.edu Tue Mar 24 21:24:55 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 24 Mar 2009 13:24:55 -0700 Subject: [Cython] too late to review and push patch for better handling of integral C types ? In-Reply-To: References: <49B584AC.6040506@behnel.de> <49B8A817.1090609@behnel.de> <49C77B9E.40504@behnel.de> Message-ID: <5FE99EF5-3055-4A62-AA20-91A8CB4ED435@math.washington.edu> On Mar 24, 2009, at 9:05 AM, Lisandro Dalcin wrote: > Stefan, I'm currently very busy trying to get a new release of two of > my projects... > > My original patch still needs a bit of discussion, and there is room > for improvement, as I want to extend that machinery to "extern > ctypedef" integrals in such a way that Cython stop assuming a base > type and instead emit a custom conversor. That would be great. BTW, I'm more in favor of auto-generated code than huge macros. See the specialize() method of http://hg.cython.org/cython-devel/file/ 8db1388bb844/Cython/Utils.py which is very handy for stuff like that. > So (if you can) open the ticket for me, upload the curent patch, and > wait me a bit until I can handle further work on this. http://trac.cython.org/cython_trac/ticket/255 Are you having trouble making tickets? > On Mon, Mar 23, 2009 at 9:07 AM, Stefan Behnel > wrote: >> >> Stefan Behnel wrote: >>> Robert Bradshaw wrote: >>>> It's too late for 0.11 though. Is there a ticket up? >>> >>> Don't think so, but while we're at it, I'd like to have a ticket >>> for any >>> change going into 0.11.1. Lisandro, could you open one and post >>> the patch >>> there? >> >> Since this patch is very well backed by a test (much better than >> many other >> patches that went in), I'm ok with committing it for 0.11.1. >> >> Very nice work, Lisandro. Could you give it a ticket and push >> patch and >> test case to cython-devel? >> >> Thanks, >> Stefan >> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From carl.witty at gmail.com Tue Mar 24 21:39:40 2009 From: carl.witty at gmail.com (Carl Witty) Date: Tue, 24 Mar 2009 13:39:40 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <1D4BF5BC-A491-4CE1-B965-D9D5CAD91D9C@math.washington.edu> References: <49BF426D.7010005@behnel.de> <49BF5871.4040404@student.matnat.uio.no> <50876.213.61.181.86.1237279705.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C14F0C.4030000@behnel.de> <49C87C5A.80607@behnel.de> <1D4BF5BC-A491-4CE1-B965-D9D5CAD91D9C@math.washington.edu> Message-ID: On Tue, Mar 24, 2009 at 1:17 PM, Robert Bradshaw wrote: > On Mar 24, 2009, at 8:46 AM, Carl Witty wrote: >> It would be nice if the "kill performance entirely and check the >> value at >> runtime to emit a warning only when it's necessary" option were also >> available (under the control of a compiler flag, probably, and >> definitely not by default). ?Then somebody could compile Sage with >> that flag and see if doctesting reveals any problems. > > Sure. We have cython.cdivision(True), cython.cdivision(False), and ??? Well, it should be orthogonal to these (you may want the warning flag regardless of whether the current code defaults to C or Python division mode). Maybe cdivision(c_mode, warn_if_mode_matters=False), so you can use cython.cdivision(True, True) or cython.cdivision(False, True). Or cython.division_mode('C'), cython.division_mode('Python', True), etc. Carl From robertwb at math.washington.edu Tue Mar 24 21:42:27 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 24 Mar 2009 13:42:27 -0700 Subject: [Cython] Quick Fortran support poll In-Reply-To: <49C8EEEC.1070005@student.matnat.uio.no> References: <49C8E087.4080505@student.matnat.uio.no> <6e449ed14def4abf26ebf4448966eff0.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C8EEEC.1070005@student.matnat.uio.no> Message-ID: <77BC19C8-E7C8-4DA0-91DF-9A36199D4BBE@math.washington.edu> On Mar 24, 2009, at 7:32 AM, Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Dag Sverre Seljebotn wrote: >> >>> Is it OK to have nice and convenient wrapping of Fortran >>> libraries as an >>> additional feature of core Cython? >>> >>> [ ] Yes, that would be grand >>> [ ] No, that would bloat the language and/or the compiler/ >>> project too >>> much, I'd prefer to see this in a standalone tool/plugin >>> >> >> [X] It depends on how it's done. >> > OK, some details then. This is a scetch only! Given these details (though there's still a lot to hammer out), I'd vote [X] Yes, that would be grand as well. > First of all, this is about calling into Fortran (i.e. calling > conventions) only, not about writing Fortran code using Cython. > > 1) A "fortran" calling convention on external function > declarations, and > perhaps a "module"/"namespace" declaration (which one could reuse > for C++): > > cdef extern: > module mymodule: # reusable for C++ wrapping, could say > "namespace" > instead > cdef fortran sometype myfunc(sometype arg) I like the cdef extern from fortran module "mymodule" syntax (as well as cdef extern from "cppheader.H" namespace "mynamespace"). > When "fortran" is in effect, one may accept some extra syntax for the > arguments, i.e. "in" or "inout" (but "const" and no const could serve > the same purpose). I'll let people who are more familiar with this hash it out, but if it's semantically different than const, lets not overload (especially since we don't even support the standard const yet). > 2) Types -- I am thinking of a "cython.fortran" psuedomodule with > Fortran types (i.e. cython.fortran.real(kind=8)); these would > simply map > down to C typedefs. But one could get by with a simple pxd with no > compiler support as well. +1 > 3) Potentially parsing of Fortran files to autogenerate pxd files, but > this could be a seperate tool and/or be pluggable using an API that > could also be used for header files. Yes. I could see us shipping a set of header -> pxd scripts. Actually, ran across one for .h files already. > Implementation-wise, there would be a transform which would deal with > calls to functions using the Fortran calling convention, and > generates a > mymodule.f90 wrapper to supplement mymodule.c -- however this is a > very > thin wrapper to retype the Fortran functions from Fortran-side to use > some C conventions and some Cython-specific conventions, and most > of the > things would be C-side. > > The argument for doing this in Cython is that both Cython/Python and > Fortran support a higher-level construct which C doesn't > (buffers/strided arrays). > > How does that make you feel? Alternatives: > - Plug it in through a plugin framework > - Do it through a preprocessing step which generates the > necesarry .pyx > and .f90 files. I think it makes the most sense to have a .f90 file correspond to the library .pxd file (either generated from the .pxd, or along side the .pxd), but I don't have much experience with fortran. If two separate .pyx files use the same library, how (if at all) would their .f90 wrappers differ? Also, what would the compilation process then be like? - Robert From dagss at student.matnat.uio.no Tue Mar 24 22:03:48 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 24 Mar 2009 22:03:48 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: <49BF426D.7010005@behnel.de> <49BF5871.4040404@student.matnat.uio.no> <50876.213.61.181.86.1237279705.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C14F0C.4030000@behnel.de> <49C87C5A.80607@behnel.de> <1D4BF5BC-A491-4CE1-B965-D9D5CAD91D9C@math.washington.edu> Message-ID: <49C94AB4.2070604@student.matnat.uio.no> Carl Witty wrote: > On Tue, Mar 24, 2009 at 1:17 PM, Robert Bradshaw > wrote: >> On Mar 24, 2009, at 8:46 AM, Carl Witty wrote: >>> It would be nice if the "kill performance entirely and check the >>> value at >>> runtime to emit a warning only when it's necessary" option were also >>> available (under the control of a compiler flag, probably, and >>> definitely not by default). Then somebody could compile Sage with >>> that flag and see if doctesting reveals any problems. >> Sure. We have cython.cdivision(True), cython.cdivision(False), and ??? > > Well, it should be orthogonal to these (you may want the warning flag > regardless of whether the current code defaults to C or Python > division mode). > > Maybe cdivision(c_mode, warn_if_mode_matters=False), so you can use > cython.cdivision(True, True) or cython.cdivision(False, True). A seperate cython.warning_if_division_mode_matters will save us 20 minutes of developer time, as keywords are currently not supported for directives :-) -- Dag Sverre From carl.witty at gmail.com Tue Mar 24 22:15:18 2009 From: carl.witty at gmail.com (Carl Witty) Date: Tue, 24 Mar 2009 14:15:18 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49C94AB4.2070604@student.matnat.uio.no> References: <50876.213.61.181.86.1237279705.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C14F0C.4030000@behnel.de> <49C87C5A.80607@behnel.de> <1D4BF5BC-A491-4CE1-B965-D9D5CAD91D9C@math.washington.edu> <49C94AB4.2070604@student.matnat.uio.no> Message-ID: On Tue, Mar 24, 2009 at 2:03 PM, Dag Sverre Seljebotn wrote: > A seperate cython.warning_if_division_mode_matters will save us 20 > minutes of developer time, as keywords are currently not supported for > directives :-) That would be fine with me :) Will this automatically create a corresponding command-line compiler argument? Carl From dalcinl at gmail.com Tue Mar 24 22:19:21 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 24 Mar 2009 18:19:21 -0300 Subject: [Cython] too late to review and push patch for better handling of integral C types ? In-Reply-To: <49C92B3C.6040506@student.matnat.uio.no> References: <49B584AC.6040506@behnel.de> <49B8A817.1090609@behnel.de> <49C77B9E.40504@behnel.de> <49C92B3C.6040506@student.matnat.uio.no> Message-ID: On Tue, Mar 24, 2009 at 3:49 PM, Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: >> Stefan, I'm currently very busy trying to get a new release of two of >> my projects... >> >> My original patch still needs a bit of discussion, and there is room >> for improvement, as I want to extend that machinery to "extern >> ctypedef" integrals in such a way that Cython stop assuming a base >> type and instead emit a custom conversor. > > Excellent! As you probably know the buffer interface already does not > make such assumptions. > Yes, of course :-) > So basically, > > cdef extern: > ? ? ctypedef short a > ? ? ctypedef long b > > here, a and b should always be treated in the exact same way by Cython? +1! > Yes, but you still will need to specify signed(default)/unsigned. > I think we should include floats though, so that the size of float vs. > long double isn't assumed for external typedefs as well. > Well... I believe floats are different beasts. Python can only handle 'double', there is no a flexible floating like PyLong for integers... I do not want to enter "Decimal" terrain. We should discuss at some point what to do with all this... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Tue Mar 24 22:22:09 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 24 Mar 2009 18:22:09 -0300 Subject: [Cython] too late to review and push patch for better handling of integral C types ? In-Reply-To: <5FE99EF5-3055-4A62-AA20-91A8CB4ED435@math.washington.edu> References: <49B584AC.6040506@behnel.de> <49B8A817.1090609@behnel.de> <49C77B9E.40504@behnel.de> <5FE99EF5-3055-4A62-AA20-91A8CB4ED435@math.washington.edu> Message-ID: On Tue, Mar 24, 2009 at 5:24 PM, Robert Bradshaw wrote: > On Mar 24, 2009, at 9:05 AM, Lisandro Dalcin wrote: > >> Stefan, I'm currently very busy trying to get a new release of two of >> my projects... >> >> My original patch still needs a bit of discussion, and there is room >> for improvement, as I want to extend that machinery to "extern >> ctypedef" integrals in such a way that Cython stop assuming a base >> type and instead emit a custom conversor. > > That would be great. > But I'll likely need help for the ctypedef part... > BTW, I'm more in favor of auto-generated code than huge macros. See > the specialize() method of http://hg.cython.org/cython-devel/file/ > 8db1388bb844/Cython/Utils.py which is very handy for stuff like that. > OK, I'm fine with that... >> So (if you can) open the ticket for me, upload the curent patch, and >> wait me a bit until I can handle further work on this. > > http://trac.cython.org/cython_trac/ticket/255 > > Are you having trouble making tickets? > No, no problems... Just really busy working on my personal projects... I really need them to get released... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Tue Mar 24 22:27:08 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 24 Mar 2009 18:27:08 -0300 Subject: [Cython] Quick Fortran support poll In-Reply-To: <77BC19C8-E7C8-4DA0-91DF-9A36199D4BBE@math.washington.edu> References: <49C8E087.4080505@student.matnat.uio.no> <6e449ed14def4abf26ebf4448966eff0.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C8EEEC.1070005@student.matnat.uio.no> <77BC19C8-E7C8-4DA0-91DF-9A36199D4BBE@math.washington.edu> Message-ID: Have all you taken into account the ISO_C_BINDING stuff from Fortran 2003 standard? On Tue, Mar 24, 2009 at 5:42 PM, Robert Bradshaw wrote: > On Mar 24, 2009, at 7:32 AM, Dag Sverre Seljebotn wrote: > >> Stefan Behnel wrote: >>> Dag Sverre Seljebotn wrote: >>> >>>> Is it OK to have nice and convenient wrapping of Fortran >>>> libraries as an >>>> additional feature of core Cython? >>>> >>>> ?[ ] Yes, that would be grand >>>> ?[ ] No, that would bloat the language and/or the compiler/ >>>> project too >>>> much, I'd prefer to see this in a standalone tool/plugin >>>> >>> >>> ? [X] It depends on how it's done. >>> >> OK, some details then. This is a scetch only! > > Given these details (though there's still a lot to hammer out), I'd > vote [X] Yes, that would be grand as well. > >> First of all, this is about calling into Fortran (i.e. calling >> conventions) only, not about writing Fortran code using Cython. >> >> 1) A "fortran" calling convention on external function >> declarations, and >> perhaps a "module"/"namespace" declaration (which one could reuse >> for C++): >> >> cdef extern: >> ? ? module mymodule: # reusable for C++ wrapping, could say >> "namespace" >> instead >> ? ? ? ? cdef fortran sometype myfunc(sometype arg) > > I like the cdef extern from fortran module "mymodule" syntax (as well > as cdef extern from "cppheader.H" namespace "mynamespace"). > >> When "fortran" is in effect, one may accept some extra syntax for the >> arguments, i.e. "in" or "inout" (but "const" and no const could serve >> the same purpose). > > I'll let people who are more familiar with this hash it out, but if > it's semantically different than const, lets not overload (especially > since we don't even support the standard const yet). > >> 2) Types -- I am thinking of a "cython.fortran" psuedomodule with >> Fortran types (i.e. cython.fortran.real(kind=8)); these would >> simply map >> down to C typedefs. But one could get by with a simple pxd with no >> compiler support as well. > > +1 > >> 3) Potentially parsing of Fortran files to autogenerate pxd files, but >> this could be a seperate tool and/or be pluggable using an API that >> could also be used for header files. > > Yes. I could see us shipping a set of header -> pxd scripts. > Actually, ran across one for .h files already. > >> Implementation-wise, there would be a transform which would deal with >> calls to functions using the Fortran calling convention, and >> generates a >> mymodule.f90 wrapper to supplement mymodule.c -- however this is a >> very >> thin wrapper to retype the Fortran functions from Fortran-side to use >> some C conventions and some Cython-specific conventions, and most >> of the >> things would be C-side. >> >> The argument for doing this in Cython is that both Cython/Python and >> Fortran support a higher-level construct which C doesn't >> (buffers/strided arrays). >> >> How does that make you feel? Alternatives: >> - Plug it in through a plugin framework >> - Do it through a preprocessing step which generates the >> necesarry .pyx >> and .f90 files. > > I think it makes the most sense to have a .f90 file correspond to the > library .pxd file (either generated from the .pxd, or along side > the .pxd), but I don't have much experience with fortran. If two > separate .pyx files use the same library, how (if at all) would > their .f90 wrappers differ? > > Also, what would the compilation process then be like? > > - Robert > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Tue Mar 24 22:48:36 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 24 Mar 2009 22:48:36 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: <50876.213.61.181.86.1237279705.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C14F0C.4030000@behnel.de> <49C87C5A.80607@behnel.de> <1D4BF5BC-A491-4CE1-B965-D9D5CAD91D9C@math.washington.edu> <49C94AB4.2070604@student.matnat.uio.no> Message-ID: <49C95534.8090905@student.matnat.uio.no> Carl Witty wrote: > On Tue, Mar 24, 2009 at 2:03 PM, Dag Sverre Seljebotn > wrote: >> A seperate cython.warning_if_division_mode_matters will save us 20 >> minutes of developer time, as keywords are currently not supported for >> directives :-) > > That would be fine with me :) > > Will this automatically create a corresponding command-line compiler argument? Yes. -- Dag Sverre From dagss at student.matnat.uio.no Tue Mar 24 22:53:01 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 24 Mar 2009 22:53:01 +0100 Subject: [Cython] Quick Fortran support poll In-Reply-To: References: <49C8E087.4080505@student.matnat.uio.no> <6e449ed14def4abf26ebf4448966eff0.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C8EEEC.1070005@student.matnat.uio.no> <77BC19C8-E7C8-4DA0-91DF-9A36199D4BBE@math.washington.edu> Message-ID: <49C9563D.4050207@student.matnat.uio.no> Lisandro Dalcin wrote: > Have all you taken into account the ISO_C_BINDING stuff from Fortran > 2003 standard? Yes; that's already up at http://wiki.cython.org/enhancements/fortran But things are quickly moving a bit beyond that proposal now. We will get back to the list sometime later with more concrete proposals, taking into account the feedback. -- Dag Sverre From dalcinl at gmail.com Tue Mar 24 23:46:19 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 24 Mar 2009 19:46:19 -0300 Subject: [Cython] PyCon? In-Reply-To: References: Message-ID: 2009/3/22 Vic Kelson : > (1) many of the fundamental analytic functions are already carefully coded > in Fortran 90 (no pointers) and can readily connect to Python via F2PY. I > don't want to re-code these because "they're both nasty" and coded using > complex arithmetic. So, my optimal Cython module should call some Fortran > routines. As long as you add the nasty mangling underscores, you should be able to call Fortran functions from Cython. Complex type arguments would require a bit of manual code, as Cython do not (currently) handle complex C/C++ types. > (3) The "killer" feature that has stopped me from a Python computational > code is the ability to implement particle tracking in the model. The > tracking algorithm is computationally intensive, including checking for > intersections between line segments and proximity to model features. I > believe that Cython will allow me to overcome this hurdle. The test code I > mentioned above does not support this feature, but I will try to implement > it before PyCon. Let me tell you a story. A grad student doing some work with a coworker was coding Python for some computational geometry stuff, with GUI API?s, OpenGL and all that... But at some point, Python was a bit slow for managing some stuff, like "Point" classes. Then I told the guy: use Cython, just use "cdef" were appropriate (for the Point class and instance members), and the student got the code working with impressive speedups in a morning... So I bet you will not have any problems understanding and using Cython... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From brett.calcott at gmail.com Wed Mar 25 05:20:35 2009 From: brett.calcott at gmail.com (Brett Calcott) Date: Wed, 25 Mar 2009 15:20:35 +1100 Subject: [Cython] optimising numpy code Message-ID: Hi, I've just whipped up my first cython-based code to try and speed up some operations on some numpy-based code I'm writing. It works, but I'd like to know how to make it go faster. It basically does something like bincount (though bincount does not appear to work on more than 1-dimension). I've looked at the resulting C code and cannot understand why (in some cases) there are still some pythonic calls being made. Any help would be appreciated. Some questions are in the code (and yes, it assumes that bounds checking is done before the call) import numpy as np cimport numpy as np cimport cython @cython.boundscheck(False) # turn of bounds-checking for entire function def table_bincount(int max_elements, np.ndarray[long, ndim=2] table): cdef long rows = table.shape[0] cdef long columns = table.shape[1] cdef long i, j cdef np.ndarray[double, ndim=2] retval = np.zeros( (rows, max_elements), dtype=np.float64) for 0 <= i < rows: # Is there any way of typing these intermediate values so that they # do not get turned into python code? current_in_row = table[i] current_out_row = retval[i] for 0 <= j < columns: current_out_row[current_in_row[j]] += 1 # Notice that even if do this instead : # retval[i][table[i, j]] += 1 # ... it still generates some pythonic stuff like # PyNumberAddInplace return retval Thanks in advance, Brett Calcott From hoytak at cs.ubc.ca Wed Mar 25 05:37:17 2009 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Tue, 24 Mar 2009 21:37:17 -0700 Subject: [Cython] optimising numpy code In-Reply-To: References: Message-ID: <4db580fd0903242137r6936a0dasfcf09c4de8df9f9f@mail.gmail.com> Hi Brett, There are a few small ways to speed up the code (I'm not thinking about algorithmic ones; there may be some of those too) > @cython.boundscheck(False) # turn of bounds-checking for entire function > def table_bincount(int max_elements, np.ndarray[long, ndim=2] table): > ? ?cdef long rows = table.shape[0] > ? ?cdef long columns = table.shape[1] > ? ?cdef long i, j Use size_t (in latest cython version) or unsigned long instead of long where appropriate. If it's a signed type, cython checks to see if it's negative so things like a[-1] can be supported. Thus unsigned types are faster. > ? ?cdef np.ndarray[double, ndim=2] retval = np.zeros( > ? ? ? ? ? ?(rows, max_elements), dtype=np.float64) you can add mode="c" into this array as well, since you're creating it. This means it's a contiguously aligned array, so the last stride is assumed to be one. This can speed up internal loops. So the new version is > cdef np.ndarray[double, ndim=2, mode="c"] retval = np.zeros( > (rows, max_elements), dtype=np.float64) > ? ?for 0 <= i < rows: > ? ? ? ?# Is there any way of typing these intermediate values so that they > ? ? ? ?# do not get turned into python code? > current_in_row = table[i] > current_out_row = retval[i] Any reason you can't use table[i, j] and retval[i,j] instead of creating intermediate slices? Note that you should usually do A[i,j] instead of A[i][j] in regular python as well. > ? ? ? ? ? ?current_out_row[current_in_row[j]] += 1 Can't this just be something like: retval[i, table[i,j]] += 1 Hope this all helps! --Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From robertwb at math.washington.edu Wed Mar 25 10:59:19 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 25 Mar 2009 02:59:19 -0700 Subject: [Cython] optimising numpy code In-Reply-To: References: Message-ID: <74E12139-A3CF-42AB-B6A2-3B3213E443A3@math.washington.edu> On Mar 24, 2009, at 9:20 PM, Brett Calcott wrote: > Hi, > > I've just whipped up my first cython-based code to try and speed up > some operations on some numpy-based code I'm writing. It works, but > I'd like to know how to make it go faster. It basically does something > like bincount (though bincount does not appear to work on more than > 1-dimension). > > I've looked at the resulting C code and cannot understand why (in some > cases) there are still some pythonic calls being made. Any help would > be appreciated. Some questions are in the code (and yes, it assumes > that bounds checking is done before the call) > > import numpy as np > cimport numpy as np > cimport cython > > @cython.boundscheck(False) # turn of bounds-checking for entire > function > def table_bincount(int max_elements, np.ndarray[long, ndim=2] table): > cdef long rows = table.shape[0] > cdef long columns = table.shape[1] > cdef long i, j > cdef np.ndarray[double, ndim=2] retval = np.zeros( > (rows, max_elements), dtype=np.float64) > > for 0 <= i < rows: > # Is there any way of typing these intermediate values so > that they > # do not get turned into python code? > current_in_row = table[i] > current_out_row = retval[i] > for 0 <= j < columns: > current_out_row[current_in_row[j]] += 1 > # Notice that even if do this instead : > # retval[i][table[i, j]] += 1 > # ... it still generates some pythonic stuff like > # PyNumberAddInplace > > return retval This is because you're writing retval[i][table[i, j]] += 1 rather than retval[i, table[i, j]] += 1. (I think this was already mentioned.) - Robert From simone at tekno-soft.it Wed Mar 25 19:17:30 2009 From: simone at tekno-soft.it (Simone Bisogni) Date: Wed, 25 Mar 2009 19:17:30 +0100 Subject: [Cython] C++ exception mapping problems Message-ID: <49CA753A.4010302@tekno-soft.it> Hi guys, I have a problem building a python c++ extension and, in particular, the problem occurs when I try to map C++ exceptions to python exceptions. Here is a simple example: ======================================================================== ### Cpp_Rectangle.pxd cdef extern from "Rectangle.h": ctypedef struct cpp_Rectangle "Rectangle": int raiseException() except + cpp_Rectangle* new_Rectangle "new Rectangle"( int x0, int y0, int x1, int y1 ) void delete_Rectangle "delete"( cpp_Rectangle* rect ) ======================================================================== ### PyRectangle.pyx cimport Cpp_Rectangle cdef class Rectangle: cdef Cpp_Rectangle.cpp_Rectangle* thisptr def __init__( self, int x0, int y0, int x1, int y1 ): self.thisptr = Cpp_Rectangle.new_Rectangle( x0, y0, x1, y1 ) def __del__( self ): Cpp_Rectangle.delete_Rectangle( self.thisptr ) def raiseException( self ): self.thisptr.raiseException() ========================================================================= How you can see in the Cpp_Rectangle.pyd, I am trying to use the "except +" feature to map the exceptions generated by the raiseException function into a Python RuntimeError. The cython --cplus runs fine, but when I try to compile the PyRectangle.cpp then I got the following errors: ========================================================================= [simone at spider Cython_Test]$ g++ -shared -fPIC -I/usr/include/python2.5 -o PyRectangle.so PyRectangle.cpp -L /usr/lib64 -L. -lRectangle -lpython2.5 PyRectangle.cpp: In function 'void __Pyx_CppExn2PyErr()': PyRectangle.cpp:277: error: expected unqualified-id before '&' token PyRectangle.cpp:277: error: ISO C++ forbids declaration of 'type name' with no type PyRectangle.cpp:277: error: expected `)' before '&' token PyRectangle.cpp:277: error: expected `{' before '&' token PyRectangle.cpp:277: error: 'exn' was not declared in this scope PyRectangle.cpp:277: error: expected `;' before ')' token PyRectangle.cpp:280: error: expected primary-expression before 'catch' PyRectangle.cpp:280: error: expected `;' before 'catch' PyRectangle.cpp:283: error: expected primary-expression before 'catch' PyRectangle.cpp:283: error: expected `;' before 'catch' ========================================================================== The errors occur because the the inclusion of the stdexcept header is missing. Is it a cython bug or it is up to the user to add the inclusion? Actually, I have solved the problem adding the following rows to my Cpp_Rectangle.pxd file, but I don't know if it is the right way to procede: ========================================================================== cdef extern from "stdexcept": pass ========================================================================== Another error happens when I change my Cpp_Rectangle.pxd like this: ========================================================================== ### Cpp_Rectangle.pxd cdef extern from "stdexcept": pass cdef extern from "Rectangle.h": ctypedef struct cpp_Rectangle "Rectangle": int raiseException() except +MemoryError cpp_Rectangle* new_Rectangle "new Rectangle"( int x0, int y0, int x1, int y1 ) void delete_Rectangle "delete"( cpp_Rectangle* rect ) ========================================================================= Basically I am trying now to map each exception thrown by the raiseException to a python MemoryError exception. The cython compilation runs fine again, but when I try to compile the .cpp file I got the errors: ========================================================================= [simone at spider Cython_Test]$ g++ -shared -fPIC -I/usr/include/python2.5 -o PyRectangle.so PyRectangle.cpp -L /usr/lib64 -L. -lRectangle -lpython2.5 PyRectangle.cpp: In function 'PyObject* __pyx_pf_11PyRectangle_9Rectangle_raiseException(PyObject*, PyObject*)': PyRectangle.cpp:454: error: '__pyx_builtin_MemoryError' was not declared in this scope ========================================================================== The accused piece of code is the following: ========================================================================== /* __pyx_t_1 allocated */ try { __pyx_t_1 = ((struct__pyx_obj_11PyRectangle_Rectangle*)__pyx_v_self)->thisptr->raiseException();} catch(...) {PyErr_SetString(__pyx_builtin_MemoryError, ""); {__pyx_filename = __pyx_f[0]; __pyx_lineno = 14; __pyx_clineno = __LINE__; goto __pyx_L1_error;}} ============================================================================= The variable __pyx_builtin_MemoryError is not declared anywhere. Should the __pyx_builtin_MemoryError be substituted with PyExc_MemoryError? The version of cython I am using is the 0.11 Thanks in advance and sorry for my long email. Ciao, Simone -------------- next part -------------- A non-text attachment was scrubbed... Name: simone.vcf Type: text/x-vcard Size: 2272 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090325/ac551a1c/attachment.vcf From stefan_ml at behnel.de Wed Mar 25 20:37:34 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 25 Mar 2009 20:37:34 +0100 Subject: [Cython] Fix for ticket #135 In-Reply-To: References: <49C13044.2090808@behnel.de> <49C1368B.3010007@student.matnat.uio.no> <02DFEEFA-4143-4F00-971F-30FD91D92324@math.washington.edu> Message-ID: <49CA87FE.7080500@behnel.de> Kurt Smith wrote: > Here's the addition of a test case. I pushed it to cython-devel and extended it a bit to test other block contexts (same for "break" and "continue", which already work nicely today). Stefan From stefan_ml at behnel.de Wed Mar 25 21:06:45 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 25 Mar 2009 21:06:45 +0100 Subject: [Cython] Cython's way into the stdlib? Message-ID: <49CA8ED5.7030002@behnel.de> Hi all, seeing that Cython's latest release has brought it a major step forwards, I wonder if it isn't time to knock at python-dev and ask for making Cython an integral part of CPython's standard library. This has actually been a project goal since the beginning, but in an active project, it's always hard to find a point where something's really "ready enough" for inclusion. My feeling is that it would be great to have Cython in the stdlib rather sooner than later, even if there will be further enhancements and optimisations being added to Cython outside of CPython. I think the core non-Python parts of the language are pretty stable by now. Having it in would make it a lot easier for people to depend on Cython being available on the user side, it would lead to a better integration of Cython into CPython as a general purpose Python code compiler, and it would allow CPython's core and stdlib developers to write their modules in Cython rather than C, which would make them a lot more accessible for normal Python users. And it would certainly underline the great match that CPython and Cython represent. What do the others think? Should we ask on python-dev, and then target Cython 0.12 (or maybe really 0.13) as a release for the stlib? Stefan From dagss at student.matnat.uio.no Wed Mar 25 21:32:12 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 25 Mar 2009 21:32:12 +0100 Subject: [Cython] Cython's way into the stdlib? In-Reply-To: <49CA8ED5.7030002@behnel.de> References: <49CA8ED5.7030002@behnel.de> Message-ID: <49CA94CC.3080203@student.matnat.uio.no> Stefan Behnel wrote: > Hi all, > > seeing that Cython's latest release has brought it a major step forwards, I > wonder if it isn't time to knock at python-dev and ask for making Cython an > integral part of CPython's standard library. This has actually been a > project goal since the beginning, but in an active project, it's always > hard to find a point where something's really "ready enough" for inclusion. > > My feeling is that it would be great to have Cython in the stdlib rather > sooner than later, even if there will be further enhancements and > optimisations being added to Cython outside of CPython. I think the core > non-Python parts of the language are pretty stable by now. Comment about non-Python parts of language: As I've mentioned I've got serious doubts about the current buffer syntax (http://wiki.cython.org/enhancements/buffersyntax). Options: a) Do not change the syntax b) Add new one, but do not deprecate the current one c) Add new one and do deprecate the current one My preferred solution is c), but to have a really long deprecation cycle (say two years). Of course, as the new syntax isn't even voted on yet, I guess the only right answer here can be that this should not impact inclusion of Cython into stdlib in anyway. The question is whether that should spur me into further working on the syntax and getting it voted for and implemented sooner rather than later as well. Anyway, I'm +1 to an inclusion in stdlib around 0.12. -- Dag Sverre From vic.kelson at gmail.com Wed Mar 25 21:22:13 2009 From: vic.kelson at gmail.com (Vic Kelson) Date: Wed, 25 Mar 2009 16:22:13 -0400 Subject: [Cython] Syntax highlighting for Cython? Message-ID: Hi All, Has anyone implemented syntax highlighting for Cython in any popular editor? I frequently use Eclipse/PyDev, Komodo Edit, and a few others, but I haven't found a syntax highlighter yet. It would be nice if it was integrated into PyDev, which highlights detected errors. Or maybe Cython's not mature enough yet? --v -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090325/b94a69ad/attachment.htm From robertwb at math.washington.edu Wed Mar 25 21:22:19 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 25 Mar 2009 13:22:19 -0700 Subject: [Cython] Cython's way into the stdlib? In-Reply-To: <49CA8ED5.7030002@behnel.de> References: <49CA8ED5.7030002@behnel.de> Message-ID: <52D0D5CB-5E2B-4AE6-A2EA-7CF7ACD71BFD@math.washington.edu> On Mar 25, 2009, at 1:06 PM, Stefan Behnel wrote: > Hi all, > > seeing that Cython's latest release has brought it a major step > forwards, I > wonder if it isn't time to knock at python-dev and ask for making > Cython an > integral part of CPython's standard library. This has actually been a > project goal since the beginning, but in an active project, it's > always > hard to find a point where something's really "ready enough" for > inclusion. > > My feeling is that it would be great to have Cython in the stdlib > rather > sooner than later, even if there will be further enhancements and > optimisations being added to Cython outside of CPython. I think the > core > non-Python parts of the language are pretty stable by now. > > Having it in would make it a lot easier for people to depend on Cython > being available on the user side, it would lead to a better > integration of > Cython into CPython as a general purpose Python code compiler, and > it would > allow CPython's core and stdlib developers to write their modules > in Cython > rather than C, which would make them a lot more accessible for normal > Python users. > > And it would certainly underline the great match that CPython and > Cython > represent. > > What do the others think? I tend to have a hard time calling things "ready enough," but I think that we're getting there, and I think we should at least start the process. (Note that inclusion in the standard Python is less beneficial to me personally, as I use it mostly for Sage which ships Cython, so I'm trying to ignore my bias here.) > Should we ask on python-dev, and then target > Cython 0.12 (or maybe really 0.13) as a release for the stlib? I think perhaps we should target 2.7/3.2, rather than a specific Cython release. It would be a great idea to ask of python-dev and see what they think is needed/desired for inclusion with the standard library. - Robert From stefan_ml at behnel.de Wed Mar 25 21:26:03 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 25 Mar 2009 21:26:03 +0100 Subject: [Cython] Syntax highlighting for Cython? In-Reply-To: References: Message-ID: <49CA935B.70601@behnel.de> Vic Kelson wrote: > Has anyone implemented syntax highlighting for Cython in any popular editor? > I frequently use Eclipse/PyDev, Komodo Edit, and a few others, but I haven't > found a syntax highlighter yet. Regarding syntax highlighting, Cython is supported in Pygments and there is a pretty good Emacs mode for it. Didn't have to care about anything else so far. Stefan From robertwb at math.washington.edu Wed Mar 25 21:26:36 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 25 Mar 2009 13:26:36 -0700 Subject: [Cython] Syntax highlighting for Cython? In-Reply-To: References: Message-ID: <63D46A36-BF6D-4F47-A944-EF7B1FD02E60@math.washington.edu> On Mar 25, 2009, at 1:22 PM, Vic Kelson wrote: > Hi All, > > Has anyone implemented syntax highlighting for Cython in any > popular editor? I frequently use Eclipse/PyDev, Komodo Edit, and a > few others, but I haven't found a syntax highlighter yet. Yes, there's a Cython mode for emacs. > It would be nice if it was integrated into PyDev, which highlights > detected errors. That would be cool. > Or maybe Cython's not mature enough yet? The language itself is stable enough, and close enough to Python, that I think it should be feasible. Most of the work tends to be non- syntactical changes (bufferes are the exception, but a distinct syntax is justified here). - Robert From vic.kelson at gmail.com Wed Mar 25 21:31:11 2009 From: vic.kelson at gmail.com (Vic Kelson) Date: Wed, 25 Mar 2009 16:31:11 -0400 Subject: [Cython] Syntax highlighting for Cython? In-Reply-To: <63D46A36-BF6D-4F47-A944-EF7B1FD02E60@math.washington.edu> References: <63D46A36-BF6D-4F47-A944-EF7B1FD02E60@math.washington.edu> Message-ID: I'll write to the PyDev developers. Who knows? --v On Wed, Mar 25, 2009 at 4:26 PM, Robert Bradshaw < robertwb at math.washington.edu> wrote: > On Mar 25, 2009, at 1:22 PM, Vic Kelson wrote: > > > Hi All, > > > > Has anyone implemented syntax highlighting for Cython in any > > popular editor? I frequently use Eclipse/PyDev, Komodo Edit, and a > > few others, but I haven't found a syntax highlighter yet. > > Yes, there's a Cython mode for emacs. > > > It would be nice if it was integrated into PyDev, which highlights > > detected errors. > > That would be cool. > > > Or maybe Cython's not mature enough yet? > > The language itself is stable enough, and close enough to Python, > that I think it should be feasible. Most of the work tends to be non- > syntactical changes (bufferes are the exception, but a distinct > syntax is justified here). > > - Robert > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Vic Kelson vic.kelson at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090325/f1130b56/attachment.htm From m.gysel at gmail.com Wed Mar 25 21:42:25 2009 From: m.gysel at gmail.com (Martin Gysel) Date: Wed, 25 Mar 2009 21:42:25 +0100 Subject: [Cython] Syntax highlighting for Cython? In-Reply-To: References: Message-ID: <49CA9731.60309@gmail.com> Vic Kelson schrieb: > Hi All, > > Has anyone implemented syntax highlighting for Cython in any popular editor? > I frequently use Eclipse/PyDev, Komodo Edit, and a few others, but I haven't > found a syntax highlighter yet. > I use kate (KDE) with this highlight definition (afaik works in every app which use katepart as editor, even on windows): http://bearsh.blogspot.com/2009/03/cython-syntax-highlighting-for-kate.html /martin From dalcinl at gmail.com Wed Mar 25 22:17:37 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 25 Mar 2009 18:17:37 -0300 Subject: [Cython] Syntax highlighting for Cython? In-Reply-To: <63D46A36-BF6D-4F47-A944-EF7B1FD02E60@math.washington.edu> References: <63D46A36-BF6D-4F47-A944-EF7B1FD02E60@math.washington.edu> Message-ID: On Wed, Mar 25, 2009 at 5:26 PM, Robert Bradshaw wrote: > On Mar 25, 2009, at 1:22 PM, Vic Kelson wrote: > > > Yes, there's a Cython mode for emacs. > But it is a mess. I mean, I'm not satisfied at all with it :-(... I'm always tempted to fork from "python-mode.el" and write a "cython-mode.el" adding/changing stuff as appropriate. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From poonaatsoc at gmail.com Thu Mar 26 04:02:03 2009 From: poonaatsoc at gmail.com (Anoop Saldanha) Date: Thu, 26 Mar 2009 08:32:03 +0530 Subject: [Cython] Cython's way into the stdlib? In-Reply-To: <52D0D5CB-5E2B-4AE6-A2EA-7CF7ACD71BFD@math.washington.edu> References: <49CA8ED5.7030002@behnel.de> <52D0D5CB-5E2B-4AE6-A2EA-7CF7ACD71BFD@math.washington.edu> Message-ID: On Thu, Mar 26, 2009 at 1:52 AM, Robert Bradshaw < robertwb at math.washington.edu> wrote: > On Mar 25, 2009, at 1:06 PM, Stefan Behnel wrote: > > > Hi all, > > > > seeing that Cython's latest release has brought it a major step > > forwards, I > > wonder if it isn't time to knock at python-dev and ask for making > > Cython an > > integral part of CPython's standard library. This has actually been a > > project goal since the beginning, but in an active project, it's > > always > > hard to find a point where something's really "ready enough" for > > inclusion. > > > > My feeling is that it would be great to have Cython in the stdlib > > rather > > sooner than later, even if there will be further enhancements and > > optimisations being added to Cython outside of CPython. I think the > > core > > non-Python parts of the language are pretty stable by now. > > > > Having it in would make it a lot easier for people to depend on Cython > > being available on the user side, it would lead to a better > > integration of > > Cython into CPython as a general purpose Python code compiler, and > > it would > > allow CPython's core and stdlib developers to write their modules > > in Cython > > rather than C, which would make them a lot more accessible for normal > > Python users. > > > > And it would certainly underline the great match that CPython and > > Cython > > represent. > > > > What do the others think? > > I tend to have a hard time calling things "ready enough," but I think > that we're getting there, and I think we should at least start the > process. (Note that inclusion in the standard Python is less > beneficial to me personally, as I use it mostly for Sage which ships > Cython, so I'm trying to ignore my bias here.) > > > Should we ask on python-dev, and then target > > Cython 0.12 (or maybe really 0.13) as a release for the stlib? > > > I think perhaps we should target 2.7/3.2, rather than a specific > Cython release. It would be a great idea to ask of python-dev and see > what they think is needed/desired for inclusion with the standard > library. > > - Robert > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > Right now, python is trying to gain some support for its 3.0 version and AFAIK they want 3.0 to replace the current versions of python by mid 2010-late 2010. So if there is any substantial work pending, on providing better Python 3 compatibility in cython, that can be done first, and maybe we can use that as the selling point while talking to python for integration into their stdlib. -- Regards, Anoop S. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090326/193da1cc/attachment-0001.htm From brett.calcott at gmail.com Thu Mar 26 04:39:20 2009 From: brett.calcott at gmail.com (Brett Calcott) Date: Thu, 26 Mar 2009 03:39:20 +0000 (UTC) Subject: [Cython] optimising numpy code References: <74E12139-A3CF-42AB-B6A2-3B3213E443A3@math.washington.edu> Message-ID: Hi Robert, > > # Notice that even if do this instead : > > # retval[i][table[i, j]] += 1 > > # ... it still generates some pythonic stuff like > > # PyNumberAddInplace > > This is because you're writing retval[i][table[i, j]] += 1 rather than > retval[i, table[i, j]] += 1. (I think this was already mentioned.) > Ah, now I see. That makes all the difference. Now the C Code only has Pyrex #defines in it. Excellent! Thanks, Brett From brett.calcott at gmail.com Thu Mar 26 11:35:08 2009 From: brett.calcott at gmail.com (Brett Calcott) Date: Thu, 26 Mar 2009 10:35:08 +0000 (UTC) Subject: [Cython] optimising numpy code References: <4db580fd0903242137r6936a0dasfcf09c4de8df9f9f@mail.gmail.com> Message-ID: Hi Hoyt, > > Hope this all helps! It does indeed. Thanks very much. Numpy + Cython is such a great combination. Brett From hoytak at cs.ubc.ca Fri Mar 27 00:40:34 2009 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Thu, 26 Mar 2009 16:40:34 -0700 Subject: [Cython] Calling Python C-API functions having PyObject** arguments Message-ID: <4db580fd0903261640y5c8cb148y22da543a4a348485@mail.gmail.com> Hello, What would be the best way to call a python C-API function that takes PyObject** arguments from my cython code? Unless I've missed something, there's no way inside of cython to take &a, where a is type "object" without some fancy casting that I don't know about. I've thought about doing a hack with C macros that take the address of a PyObject* and call the C-API function and import them into cython as functions, but I'd think there's a better way. Thanks! --Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From chrandr at binf.ku.dk Fri Mar 27 01:22:28 2009 From: chrandr at binf.ku.dk (Christian Andreetta) Date: Fri, 27 Mar 2009 01:22:28 +0100 Subject: [Cython] polymorphism/dynamic casting? Message-ID: <300f68560903261722gd1e3a54r56fe4ee98a0ac28a@mail.gmail.com> Hi! First of all, thanks to everybody for the *great* tool called cython th question: is there a way to declare a dynamic_cast in cython? Practically, a polymorphism: I would like to address the same memory space with two different object types. The C++ lib has this function: Node* base() { return dynamic_cast(this); } // "this" is an instance of the class SuperDuperNode(Node) Some helper functions can of course go around this problem, but I was wondering if there was a way of letting cython know that: SuperDuperNode_this and Node_this=SuperDuperNode_this.base() are actually referring to the same memory address. Thanks!! From robertwb at math.washington.edu Fri Mar 27 01:44:17 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 26 Mar 2009 17:44:17 -0700 Subject: [Cython] Calling Python C-API functions having PyObject** arguments In-Reply-To: <4db580fd0903261640y5c8cb148y22da543a4a348485@mail.gmail.com> References: <4db580fd0903261640y5c8cb148y22da543a4a348485@mail.gmail.com> Message-ID: <1FA15B57-102C-41D8-958F-519F2CD3D1DE@math.washington.edu> On Mar 26, 2009, at 4:40 PM, Hoyt Koepke wrote: > Hello, > > What would be the best way to call a python C-API function that takes > PyObject** arguments from my cython code? Unless I've missed > something, there's no way inside of cython to take &a, where a is type > "object" without some fancy casting that I don't know about. > > I've thought about doing a hack with C macros that take the address of > a PyObject* and call the C-API function and import them into cython as > functions, but I'd think there's a better way. You can do a, so do &(a) to get a PyObject** - Robert From robertwb at math.washington.edu Fri Mar 27 01:54:53 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 26 Mar 2009 17:54:53 -0700 Subject: [Cython] polymorphism/dynamic casting? In-Reply-To: <300f68560903261722gd1e3a54r56fe4ee98a0ac28a@mail.gmail.com> References: <300f68560903261722gd1e3a54r56fe4ee98a0ac28a@mail.gmail.com> Message-ID: On Mar 26, 2009, at 5:22 PM, Christian Andreetta wrote: > Hi! > > First of all, thanks to everybody for the *great* tool called cython Glad you're liking it. > th question: is there a way to declare a dynamic_cast in cython? > Practically, a polymorphism: I would like to address the same memory > space with two different object types. The C++ lib has this function: > > Node* base() { return dynamic_cast(this); } // "this" is an > instance of the class SuperDuperNode(Node) > > Some helper functions can of course go around this problem, but I was > wondering if there was a way of letting cython know that: > > SuperDuperNode_this and Node_this=SuperDuperNode_this.base() > > are actually referring to the same memory address. No, there is no way to do this in pure Cython. What you could do is write it as a macro, then use "cdef extern from" to declare it as a function in the Cython scope. BTW, I don't think they're always the same memory address (e.g. in the case of multiple inheritance). - Robert From kwmsmith at gmail.com Fri Mar 27 02:36:27 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Thu, 26 Mar 2009 20:36:27 -0500 Subject: [Cython] Passing python buffers to external code, CEP is up Message-ID: I wrote up the ideas Dag and I have been knocking around (and the good input from others) about passing python buffers to external code. The implementation of this CEP would be one part of the GSoC project we're proposing. You can find it here: http://wiki.cython.org/enhancements/buffer_external It is fairly broad right now, please help us fill in the gaps. Any input is welcome -- thanks for the feedback so far! Kurt From dalcinl at gmail.com Fri Mar 27 02:41:56 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 26 Mar 2009 22:41:56 -0300 Subject: [Cython] Calling Python C-API functions having PyObject** arguments In-Reply-To: <1FA15B57-102C-41D8-958F-519F2CD3D1DE@math.washington.edu> References: <4db580fd0903261640y5c8cb148y22da543a4a348485@mail.gmail.com> <1FA15B57-102C-41D8-958F-519F2CD3D1DE@math.washington.edu> Message-ID: On Thu, Mar 26, 2009 at 9:44 PM, Robert Bradshaw wrote: > On Mar 26, 2009, at 4:40 PM, Hoyt Koepke wrote: > >> Hello, >> >> What would be the best way to call a python C-API function that takes >> PyObject** arguments from my cython code? >> >> I've thought about doing a hack with C macros that take the address of >> a PyObject* and call the C-API function and import them into cython as >> functions, but I'd think there's a better way. > > You can do a, so do &(a) to get a PyObject** > Mmm, I believe a temporary var would be safer (think about refnanny!): cdef PyObject *tmp = NULL call_c_func(&tmp); a = tmp; but take into account that now 'a' is a new ref (Am I right?), so you may need to Py_DECREF(tmp) if the returned 'tmp' is not a borrowed ref... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Fri Mar 27 03:12:53 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 26 Mar 2009 19:12:53 -0700 Subject: [Cython] Calling Python C-API functions having PyObject** arguments In-Reply-To: References: <4db580fd0903261640y5c8cb148y22da543a4a348485@mail.gmail.com> <1FA15B57-102C-41D8-958F-519F2CD3D1DE@math.washington.edu> Message-ID: <3FA9524C-77DF-401D-B5A6-8DB19930ADD1@math.washington.edu> On Mar 26, 2009, at 6:41 PM, Lisandro Dalcin wrote: > On Thu, Mar 26, 2009 at 9:44 PM, Robert Bradshaw > wrote: >> On Mar 26, 2009, at 4:40 PM, Hoyt Koepke wrote: >> >>> Hello, >>> >>> What would be the best way to call a python C-API function that >>> takes >>> PyObject** arguments from my cython code? >>> >>> I've thought about doing a hack with C macros that take the >>> address of >>> a PyObject* and call the C-API function and import them into >>> cython as >>> functions, but I'd think there's a better way. >> >> You can do a, so do &(a) to get a PyObject** >> > > Mmm, I believe a temporary var would be safer (think about refnanny!): Yep, that would make things clearer. > cdef PyObject *tmp = NULL > call_c_func(&tmp); > a = tmp; > > but take into account that now 'a' is a new ref (Am I right?), so you > may need to Py_DECREF(tmp) if the returned 'tmp' is not a borrowed > ref... Yes, if you're dealing with PyObject*, you'll need to worry about borrowed/new references yourself. tmp will incref tmp. - RObert From stefan_ml at behnel.de Fri Mar 27 06:48:05 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 27 Mar 2009 06:48:05 +0100 Subject: [Cython] Calling Python C-API functions having PyObject** arguments In-Reply-To: <4db580fd0903261640y5c8cb148y22da543a4a348485@mail.gmail.com> References: <4db580fd0903261640y5c8cb148y22da543a4a348485@mail.gmail.com> Message-ID: <49CC6895.9060708@behnel.de> Hoyt Koepke wrote: > What would be the best way to call a python C-API function that takes > PyObject** arguments from my cython code? The best way would be to let Cython do it for you, instead of calling the function yourself. ;) What function do you mean? For example, Cython uses PyDict_Next() when looping over a dictionary, so there is no need to call that function yourself. Other use cases may get optimised into a direct C-API call in a similar fashion. Stefan From stefan_ml at behnel.de Fri Mar 27 07:24:48 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 27 Mar 2009 07:24:48 +0100 Subject: [Cython] Enable for-in-range optimisation also for untyped loop vars? Message-ID: <49CC7130.5020805@behnel.de> Hi, now that we can guarantee Python semantics for the optimised for-in-range loop, should we enable the optimisation also when the loop variable is not explicitly typed, but when the range is constant and clearly within int bounds? We might consider waiting with these things until we have flow analysis in place to make sure the loop variable isn't assigned non-int values inside the loop, but I doubt that this will be a common case. Stefan From hoytak at cs.ubc.ca Fri Mar 27 07:54:48 2009 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Thu, 26 Mar 2009 23:54:48 -0700 Subject: [Cython] Calling Python C-API functions having PyObject** arguments In-Reply-To: <49CC6895.9060708@behnel.de> References: <4db580fd0903261640y5c8cb148y22da543a4a348485@mail.gmail.com> <49CC6895.9060708@behnel.de> Message-ID: <4db580fd0903262354i6ff86d2bk44793753d5ba5ac5@mail.gmail.com> > The best way would be to let Cython do it for you, instead of calling the > function yourself. ;) I definitely agree, but here I couldn't find any other way to do it. > What function do you mean? > > For example, Cython uses PyDict_Next() when looping over a dictionary, so > there is no need to call that function yourself. Other use cases may get > optimised into a direct C-API call in a similar fashion. That's exactly the function I'm using. My use case is iterating through a tree-like structure, where each node holds a dict of values and/or branches. When I encounter a branch, I iterate recursively over that branch before continuing iteration locally. The best way of implementing this, at least that I found, is to use a stack of node dictionaries references + position variables and invoke PyDict_Next() while pushing/popping as needed. I now implemented it using the intermediate variable method given above. It works great. I assume you still have to have the "ctypedef void PyObject" in the extern block; correct? BTW, this solution: > You can do a, so do &(a) to get a PyObject** gives me the error message /home/hoytak/workspace/paramtree/paramtree/paramtree.pyx:648:31: Taking address of non-lvalue Is that a bug? Thanks for everyones help! --Hoyt From stefan_ml at behnel.de Fri Mar 27 09:14:54 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 27 Mar 2009 09:14:54 +0100 (CET) Subject: [Cython] Calling Python C-API functions having PyObject** arguments In-Reply-To: <4db580fd0903262354i6ff86d2bk44793753d5ba5ac5@mail.gmail.com> References: <4db580fd0903261640y5c8cb148y22da543a4a348485@mail.gmail.com> <49CC6895.9060708@behnel.de> <4db580fd0903262354i6ff86d2bk44793753d5ba5ac5@mail.gmail.com> Message-ID: Hoyt Koepke wrote: >> The best way would be to let Cython do it for you, instead of calling >> the function yourself. ;) > > I definitely agree, but here I couldn't find any other way to do it. > >> What function do you mean? >> >> For example, Cython uses PyDict_Next() when looping over a dictionary, so >> there is no need to call that function yourself. Other use cases may get >> optimised into a direct C-API call in a similar fashion. > > That's exactly the function I'm using. My use case is iterating > through a tree-like structure, where each node holds a dict of values > and/or branches. If the nodes are cdef classes, it should be enough to type the attribute as "cdef dict" and use the iter*() dict methods in your code. Especially the improvement compared to the plain (untyped) iteritems() method call (which requires redundant tuple packing and unpacking) should be substantial. > When I encounter a branch, I iterate recursively > over that branch before continuing iteration locally. The best way of > implementing this, at least that I found, is to use a stack of node > dictionaries references + position variables and invoke PyDict_Next() > while pushing/popping as needed. That sounds like you are not using a recursive function here. If so, why not? > I now implemented it using the intermediate variable method given > above. It works great. That's why I implemented that loop optimisation. :) > I assume you still have to have the "ctypedef > void PyObject" in the extern block; correct? No, why? You just call iter*() and get plain Python objects back. No explict PyObject* involved. > BTW, this solution: > >> You can do a, so do &(a) to get a PyObject** > > gives me the error message > > /home/hoytak/workspace/paramtree/paramtree/paramtree.pyx:648:31: > Taking address of non-lvalue > > Is that a bug? Hmm, I don't see an obvious problem with this, so it looks like a bug. Could you file a bug report with a test case that shows how to reproduce this? Stefan From dagss at student.matnat.uio.no Fri Mar 27 10:16:14 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 27 Mar 2009 10:16:14 +0100 Subject: [Cython] Enable for-in-range optimisation also for untyped loop vars? In-Reply-To: <49CC7130.5020805@behnel.de> References: <49CC7130.5020805@behnel.de> Message-ID: <49CC995E.6080500@student.matnat.uio.no> Stefan Behnel wrote: > Hi, > > now that we can guarantee Python semantics for the optimised for-in-range > loop, should we enable the optimisation also when the loop variable is not > explicitly typed, but when the range is constant and clearly within int bounds? If this is done, I'd go even further and always do it. At least under Python 2: >>> for i in xrange(10**90): pass ... Traceback (most recent call last): File "", line 1, in OverflowError: long int too large to convert to int But I really thing that range() is such a common use, and super-ints so seldom, that I'd propose this as the one place it would make sense to break Python compatability also if Py3 fixes this (but keep a cython.bigrange which isn't optimized -- Sage has a corresponding srange I think). > We might consider waiting with these things until we have flow analysis in > place to make sure the loop variable isn't assigned non-int values inside > the loop, but I doubt that this will be a common case. My instinct is to wait until we can start implementing some kind of simplistic type inference transform, with this as the first simple case to deal with, rather than specific code for the for-in-range. That would depend on flow analysis though. So my instinct is to wait, but practicality might push this special case ahead as you say. I'm not sure. -- Dag Sverre From dagss at student.matnat.uio.no Fri Mar 27 10:27:55 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 27 Mar 2009 10:27:55 +0100 Subject: [Cython] Passing python buffers to external code, CEP is up In-Reply-To: References: Message-ID: <49CC9C1B.2010000@student.matnat.uio.no> Kurt Smith wrote: > I wrote up the ideas Dag and I have been knocking around (and the good > input from others) about passing python buffers to external code. The > implementation of this CEP would be one part of the GSoC project we're > proposing. You can find it here: > > http://wiki.cython.org/enhancements/buffer_external > > It is fairly broad right now, please help us fill in the gaps. > > Any input is welcome -- thanks for the feedback so far! I added two sections, implementation and syntax. I) I think cdef extern double func(object[int, ndim=2, mode="strided"] arr) really makes it seem like a PyObject* is passed, which is not the case. The wiki page now has my alternatives in it. But this question doesn't need to be settled before the application is submitted. (Note that in the current buffer syntax when "object" is the container type, the mode defaults to "full". ndarray makes mode default to "strided" as NumPy arrays are always in that format. See PEP 3118 for more information on the buffer modes.) II) I think we/you should start writing the project description/plan; and then I think much of the Fortran part of the story should be moved there. Then the project description should be the first entry point for people interested in your project and this CEP only for discussing quite targeted Cython<->C points. But it's fine as it is until a project description is in place. -- Dag Sverre From dagss at student.matnat.uio.no Fri Mar 27 10:30:19 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 27 Mar 2009 10:30:19 +0100 Subject: [Cython] Quick Fortran support poll In-Reply-To: <77BC19C8-E7C8-4DA0-91DF-9A36199D4BBE@math.washington.edu> References: <49C8E087.4080505@student.matnat.uio.no> <6e449ed14def4abf26ebf4448966eff0.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C8EEEC.1070005@student.matnat.uio.no> <77BC19C8-E7C8-4DA0-91DF-9A36199D4BBE@math.washington.edu> Message-ID: <49CC9CAB.3090107@student.matnat.uio.no> Robert Bradshaw wrote: > On Mar 24, 2009, at 7:32 AM, Dag Sverre Seljebotn wrote: >> How does that make you feel? Alternatives: >> - Plug it in through a plugin framework >> - Do it through a preprocessing step which generates the >> necesarry .pyx >> and .f90 files. > > I think it makes the most sense to have a .f90 file correspond to the > library .pxd file (either generated from the .pxd, or along side > the .pxd), but I don't have much experience with fortran. If two > separate .pyx files use the same library, how (if at all) would > their .f90 wrappers differ? > > Also, what would the compilation process then be like? Here's the current plan which should answer these questions. I've already posted on the SciPy list about this, so I just paste that: 1) Add a Cython syntax and API for passing acquired PEP-3118 buffers/NumPy arrays to external C functions (i.e. as a struct with the necesarry information (pointer, shape, strides)). This simply means defining a syntax for passing information that Cython already has to an external C function. 2) Create a new tool which uses the parser part of f2py (with any necesarry improvements) but adds a different backend which generates a C interface to the given Fortran module, along with a Cython pxd file for it. (Adding a C .h file target, to get "f2c" functionality, would be trivial.) This will be done using the Fortran 2003 C bindings. So a .f90 file is generated which compiles to a C interface to the library. Array parameters will be passed as the PEP-3118-like structs we define in 1), and so the functions will be callable directly with e.g. NumPy arrays from Cython. Copy-in/out might be necesarry for Fortran to be able to work with the arrays, if so this will happen in the Fortran wrapper generated by this new tool. 3) One could then add a feature to Cython to automatically export external functions as Python functions so that one doesn't have to write wrapper stubs. This should bring the functionality to a level comparable to current f2py. -- Dag Sverre From stefan_ml at behnel.de Fri Mar 27 11:21:30 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 27 Mar 2009 11:21:30 +0100 (CET) Subject: [Cython] Enable for-in-range optimisation also for untyped loop vars? In-Reply-To: <49CC995E.6080500@student.matnat.uio.no> References: <49CC7130.5020805@behnel.de> <49CC995E.6080500@student.matnat.uio.no> Message-ID: <70e6887dd9601a9bc8e9011155ea8bba.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> now that we can guarantee Python semantics for the optimised for-in-range >> loop, should we enable the optimisation also when the loop variable is >> not explicitly typed, but when the range is constant and clearly within >> int bounds? > > If this is done, I'd go even further and always do it. At least under > Python 2: > > >>> for i in xrange(10**90): pass > ... > Traceback (most recent call last): > File "", line 1, in > OverflowError: long int too large to convert to int Well, yes, that might be the case in Py2.x. Py3 allows you to do that - although it likely takes at least a couple of hours to run. I bet the same thing in Cython would just be optimised away and run in no-time. ;) We could auto-type the (hidden!) temp loop variable to "unsigned long long int" if the value range is not known at compile time (i.e. having non-constant, untyped range parameters). That would still be a lot faster than iterating through Python int/long objects, but it wouldn't break for-loops that just slip slightly over the 2**31 border (which might not be that uncommon anymore these days). > But I really thing that range() is such a common use, and super-ints so > seldom, that I'd propose this as the one place it would make sense to > break Python compatability also if Py3 fixes this (but keep a > cython.bigrange which isn't optimized -- Sage has a corresponding srange > I think). For now, you can do cdef object large_loop_range = xrange(10**90) for i in large_loop_range: pass It'll be a while until Cython optimises that... This might also work: for i in xrange(10**90): pass and I even find it kind of explicit. > My instinct is to wait until we can start implementing some kind of > simplistic type inference transform, with this as the first simple case > to deal with, rather than specific code for the for-in-range. That would > depend on flow analysis though. Not necessarily. It's actually extremely common to have a single point of assignment to a variable in a function, e.g. when building a list, or when running a single "for i in ..." loop. It shouldn't be too hard to optimise just this specific case in a transform, and it would give you a simplistic kind of type inference that should hit in quite a number of cases. Even running it before type analysis would catch code like "l = [...]". It would also lead to funny effects, such as having a loop inside a function run slower when you add a second one that uses the same loop variable :) However, this could only be fixed by live variable analysis (which isn't all that trivial for Python, given that e.g. "for i in range(X)" may not assign to i!) JIC: http://en.wikipedia.org/wiki/Live_variable_analysis Stefan From jorgen at devsoft.no Fri Mar 27 13:48:37 2009 From: jorgen at devsoft.no (=?UTF-8?B?IkrDuHJnZW4gUC4gVGplcm7DuCI=?=) Date: Fri, 27 Mar 2009 13:48:37 +0100 Subject: [Cython] Shorthand for getter properties? Message-ID: <49CCCB25.1050203@devsoft.no> Hi there! I'm currently using Cython to provide some bindings to a C api. I'm using classes to "wrap" the various C structs, but they're VERY repetetive to write, since many of the attribute I expose are in the form: property title: def __get__(self): return self.track.title Is there any shorthand for this? :-) Also, I want to initialize them with a C struct pointer (they're only ever initialized by my Cython code, not by the user themselves), is there a better way than: cdef class Artist: def __init__(self): raise TypeError("This class cannot be instantiated from Python") cdef extern from "foo.h": cdef Artist NEW_ARTIST "PY_NEW" (object t) # #define PY_NEW(T) \ # (((PyTypeObject*)(T))->tp_new( \ # (PyTypeObject*)(T), __pyx_empty_tuple, NULL)) cdef Artist _create_artist(artist* artist, bool take_owner=False): cdef Artist instance instance = NEW_TRACK(Artist) instance.artist = artist instance.take_owner = take_owner return instance -- Kindest regards, J?rgen P. Tjern? From stephane.drouard at st.com Fri Mar 27 14:08:31 2009 From: stephane.drouard at st.com (Stephane DROUARD) Date: Fri, 27 Mar 2009 14:08:31 +0100 Subject: [Cython] Missing stdexcept inclusion In-Reply-To: <49CCCB25.1050203@devsoft.no> Message-ID: <002701c9aedd$2275f110$b9ad810a@gnb.st.com> Hello, Cython does not include stdexcept when we ask it to catch exceptions. lib.h ===== void foo(); good.pyx ======== cdef extern from "lib.h": void foo() foo() Everything's OK. But with: bad.pyx ======= cdef extern from "lib.h": void foo() except + foo() g++ reports: bad.c: In function `void __Pyx_CppExn2PyErr()': bad.c:189: error: expected unqualified-id before '&' token bad.c:189: error: ISO C++ forbids declaration of `type name' with no type bad.c:189: error: expected `)' before '&' token bad.c:189: error: expected `{' before '&' token bad.c:189: error: `exn' was not declared in this scope bad.c:189: error: expected `;' before ')' token bad.c:192: error: expected primary-expression before "catch" bad.c:192: error: expected `;' before "catch" bad.c:195: error: expected primary-expression before "catch" bad.c:195: error: expected `;' before "catch" bad.c:189: warning: unused variable 'exn' Here is bad.c extract: 182 #ifndef __Pyx_CppExn2PyErr 183 static void __Pyx_CppExn2PyErr() { 184 try { 185 if (PyErr_Occurred()) 186 ; // let the latest Python exn pass through and ignore the current one 187 else 188 throw; 189 } catch (const std::out_of_range& exn) { 190 // catch out_of_range explicitly so the proper Python exn may be raised 191 PyErr_SetString(PyExc_IndexError, exn.what()); 192 } catch (const std::exception& exn) { 193 PyErr_SetString(PyExc_RuntimeError, exn.what()); 194 } 195 catch (...) 196 { 197 PyErr_SetString(PyExc_RuntimeError, "Unknown exception"); 198 } 199 } 200 #endif To fix that I need to add in bad.pyx: cdef extern from "stdexcept": pass Cheers, Stephane From dagss at student.matnat.uio.no Fri Mar 27 14:29:40 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 27 Mar 2009 14:29:40 +0100 Subject: [Cython] Missing stdexcept inclusion In-Reply-To: <002701c9aedd$2275f110$b9ad810a@gnb.st.com> References: <002701c9aedd$2275f110$b9ad810a@gnb.st.com> Message-ID: <49CCD4C4.7090809@student.matnat.uio.no> Stephane DROUARD wrote: > Hello, > > Cython does not include stdexcept when we ask it to catch exceptions. Thanks, this is now http://trac.cython.org/cython_trac/ticket/261 -- Dag Sverre From stefan_ml at behnel.de Fri Mar 27 14:34:59 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 27 Mar 2009 14:34:59 +0100 (CET) Subject: [Cython] Unladen Swallow Message-ID: Looks like the competition has just started: http://code.google.com/p/unladen-swallow/wiki/ProjectPlan Stefan From dalcinl at gmail.com Fri Mar 27 14:58:12 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 27 Mar 2009 10:58:12 -0300 Subject: [Cython] Calling Python C-API functions having PyObject** arguments In-Reply-To: References: <4db580fd0903261640y5c8cb148y22da543a4a348485@mail.gmail.com> <49CC6895.9060708@behnel.de> <4db580fd0903262354i6ff86d2bk44793753d5ba5ac5@mail.gmail.com> Message-ID: On Fri, Mar 27, 2009 at 5:14 AM, Stefan Behnel wrote: > Hoyt Koepke wrote: > >> BTW, this solution: >> >>> You can do a, so do &(a) to get a PyObject** >> >> gives me the error message >> >> /home/hoytak/workspace/paramtree/paramtree/paramtree.pyx:648:31: >> Taking address of non-lvalue >> >> Is that a bug? > > Hmm, I don't see an obvious problem with this, so it looks like a bug. > Could you file a bug report with a test case that shows how to reproduce > this? > But in any case, please DO NOT let code being able to do &(a). I'm almost sure this will completely confuse the refnanny machinery. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From jorgen at devsoft.no Fri Mar 27 15:58:49 2009 From: jorgen at devsoft.no (=?UTF-8?B?IkrDuHJnZW4gUC4gVGplcm7DuCI=?=) Date: Fri, 27 Mar 2009 15:58:49 +0100 Subject: [Cython] gcc warning: implicit declaration of PyBool_CheckExact Message-ID: <49CCE9A9.6010201@devsoft.no> The following python is generating cdef class Track: cdef Artist _create_artist(self, artist* artist, bool take_owner=False): # Do stuff to create an Artist object. property artists: def __get__(self): if self._artists is None: self._artists = [self._create_artist(some_c_api(self.ds, a.id), True) for a in artists] return self._artists This code, specifically the list comprehension, generates C code that gives me: warning: implicit declaration of function ?PyBool_CheckExact? Any ideas why? Should I care at all? This is Python 2.5.2, Cython 0.10.3. :-) -- Kindest regards, From jasone at canonware.com Fri Mar 27 16:01:06 2009 From: jasone at canonware.com (Jason Evans) Date: Fri, 27 Mar 2009 08:01:06 -0700 Subject: [Cython] gcc warning: implicit declaration of PyBool_CheckExact In-Reply-To: <49CCE9A9.6010201@devsoft.no> References: <49CCE9A9.6010201@devsoft.no> Message-ID: <49CCEA32.9020008@canonware.com> J?rgen P. Tjern? wrote: > The following python is generating > > cdef class Track: > cdef Artist _create_artist(self, artist* artist, bool take_owner=False): > # Do stuff to create an Artist object. > > property artists: > def __get__(self): > if self._artists is None: > self._artists = [self._create_artist(some_c_api(self.ds, a.id), > True) for a in artists] > return self._artists > > This code, specifically the list comprehension, generates C code that > gives me: > warning: implicit declaration of function ?PyBool_CheckExact? > > Any ideas why? Should I care at all? > > This is Python 2.5.2, Cython 0.10.3. :-) Use 'bint' instead of 'bool'. (I've also tripped on this at various times.) Jason From jorgen at devsoft.no Fri Mar 27 16:04:18 2009 From: jorgen at devsoft.no (=?UTF-8?B?IkrDuHJnZW4gUC4gVGplcm7DuCI=?=) Date: Fri, 27 Mar 2009 16:04:18 +0100 Subject: [Cython] Wrong ordering of functions (function being implicitly defined) Message-ID: <49CCEAF2.2020606@devsoft.no> Hi! Cython 0.10.3 is generating some C-code that has the wrong ordering. Not sure exactly what part of my code is doing this, but it generates the C code from the Cython code without any warnings. It's a 500 line Cython project, and after some recent (big) changes I get an error when building. It seems that the "constructor" function for the SessionStruct class (which Album inherits from, both are defined in Cython as cdef classes) is referenced in the Album "constructor" function, but is declared and defined AFTER the Album "constructor" function. (I use "constructor" because I'm not sure if that's the correct term for the function) The implicit declaration does not match the explicit one, and it fails. This is the implicit declaration warning: src/spytify.c: In function ?__pyx_tp_new_7spytify_Album?: src/spytify.c:4024: warning: implicit declaration of function ?__pyx_tp_new_7spytify_SessionStruct? This is the conflicting types warning for the explictio declaration: src/spytify.c:5128: error: conflicting types for ?__pyx_tp_new_7spytify_SessionStruct? src/spytify.c:4024: error: previous implicit declaration of ?__pyx_tp_new_7spytify_SessionStruct? was here Any idea why Cython doesn't order it correctly? If I move the new and dealloc functions (in the .c file) to right before the new function for Album, it compiles fine. :-) -- Kindest regards, J?rgen P. Tjern?. From jorgen at devsoft.no Fri Mar 27 16:08:35 2009 From: jorgen at devsoft.no (=?UTF-8?B?IkrDuHJnZW4gUC4gVGplcm7DuCI=?=) Date: Fri, 27 Mar 2009 16:08:35 +0100 Subject: [Cython] gcc warning: implicit declaration of PyBool_CheckExact In-Reply-To: <49CCEA32.9020008@canonware.com> References: <49CCE9A9.6010201@devsoft.no> <49CCEA32.9020008@canonware.com> Message-ID: <49CCEBF3.1010606@devsoft.no> Jason Evans wrote: > J?rgen P. Tjern? wrote: >> This code, specifically the list comprehension, generates C code that >> gives me: >> warning: implicit declaration of function ?PyBool_CheckExact? > Use 'bint' instead of 'bool'. (I've also tripped on this at various times.) Ah, thanks a lot. That fixes it nicely! -- Kindest regards, J?rgen P. Tjern?. From dagss at student.matnat.uio.no Fri Mar 27 16:09:10 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 27 Mar 2009 16:09:10 +0100 Subject: [Cython] Wrong ordering of functions (function being implicitly defined) In-Reply-To: <49CCEAF2.2020606@devsoft.no> References: <49CCEAF2.2020606@devsoft.no> Message-ID: <49CCEC16.8020508@student.matnat.uio.no> J?rgen P. Tjern? wrote: > Hi! > > Cython 0.10.3 is generating some C-code that has the wrong ordering. Not > In general you should test with the latest released version before reporting problems. That way the Cython developers can be more sure to spend their Cython time focusing on real issues. (I don't know whether this would be fixed now or not.) Dag Sverre From jorgen at devsoft.no Fri Mar 27 16:17:36 2009 From: jorgen at devsoft.no (=?UTF-8?B?IkrDuHJnZW4gUC4gVGplcm7DuCI=?=) Date: Fri, 27 Mar 2009 16:17:36 +0100 Subject: [Cython] Wrong ordering of functions (function being implicitly defined) In-Reply-To: <49CCEC16.8020508@student.matnat.uio.no> References: <49CCEAF2.2020606@devsoft.no> <49CCEC16.8020508@student.matnat.uio.no> Message-ID: <49CCEE10.8040700@devsoft.no> Dag Sverre Seljebotn wrote: > In general you should test with the latest released version before > reporting problems. That way the Cython developers can be more sure to > spend their Cython time focusing on real issues. Ah, I wasn't aware that 0.11 was released. Pi-day release and everything! ;) I've upgraded now, and I experience the same problem. /* Generated by Cython 0.11 on Fri Mar 27 16:15:44 2009 */ -- Kindest regards, J?rgen P. Tjern?. From jorgen at devsoft.no Fri Mar 27 16:38:51 2009 From: jorgen at devsoft.no (=?UTF-8?B?IkrDuHJnZW4gUC4gVGplcm7DuCI=?=) Date: Fri, 27 Mar 2009 16:38:51 +0100 Subject: [Cython] Wrong ordering of functions (function being implicitly defined) In-Reply-To: <49CCEE10.8040700@devsoft.no> References: <49CCEAF2.2020606@devsoft.no> <49CCEC16.8020508@student.matnat.uio.no> <49CCEE10.8040700@devsoft.no> Message-ID: <49CCF30B.90204@devsoft.no> J?rgen P. Tjern? wrote: > Dag Sverre Seljebotn wrote: >> In general you should test with the latest released version before >> reporting problems. That way the Cython developers can be more sure to >> spend their Cython time focusing on real issues. > Ah, I wasn't aware that 0.11 was released. Pi-day release and everything! ;) > > I've upgraded now, and I experience the same problem. > /* Generated by Cython 0.11 on Fri Mar 27 16:15:44 2009 */ > This also seems to affect the ordering of code in PyMODINIT_FUNC: __pyx_vtable_7spytify_Album.__pyx_base = *__pyx_vtabptr_7spytify_SessionStruct this line is before __pyx_vtabptr_7spytify_SessionStruct = &__pyx_vtable_7spytify_SessionStruct; So it dereferences a NULL-pointer and segfaults python on import of the extension. If I reorder the type init code so that the code for SessionStruct is first, then it builds and imports fine. :) I think it might be related to this structure: cdef class Album cdef class SessionStruct: cdef Album _create_album(self, album* album, bint take_owner=?) cdef class Album(SessionStruct): pass I declare the class before SessionStruct, but it also inherits from SessionStruct -- so I guess this is confusing Cython? Is there a workaround? Is this the "wrong" way to do it? :) (Another class, which inherits from SessionStruct, but is not forward declared before SessionStruct, seems to have its code ordered properly, both type init code and "constructor" function) -- Kindest regards, J?rgen P. Tjern?. From jorgen at devsoft.no Fri Mar 27 16:52:54 2009 From: jorgen at devsoft.no (=?UTF-8?B?IkrDuHJnZW4gUC4gVGplcm7DuCI=?=) Date: Fri, 27 Mar 2009 16:52:54 +0100 Subject: [Cython] Wrong ordering of functions (function being implicitly defined) In-Reply-To: <49CCF30B.90204@devsoft.no> References: <49CCEAF2.2020606@devsoft.no> <49CCEC16.8020508@student.matnat.uio.no> <49CCEE10.8040700@devsoft.no> <49CCF30B.90204@devsoft.no> Message-ID: <49CCF656.5090402@devsoft.no> J?rgen P. Tjern? wrote: > I declare the class before SessionStruct, but it also inherits from > SessionStruct -- so I guess this is confusing Cython? Is there a > workaround? Is this the "wrong" way to do it? :) If I forward declare SessionStruct _before_ I forward declare the classes inheriting from SessionStruct (without referring to the inheritance in the forward declaration), it orders the code correctly. That is, this works: cdef class SessionStruct cdef class Album cdef class SessionStruct: cdef Album foo(self): pass cdef class Album(SessionStruct): pass But this does not (that is, it generates C code, but the C code doesn't build): cdef class Album cdef class SessionStruct: cdef Album foo(self): pass cdef class Album(SessionStruct): pass -- Kindest regards, J?rgen P. Tjern? From hoytak at cs.ubc.ca Fri Mar 27 17:00:52 2009 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Fri, 27 Mar 2009 09:00:52 -0700 Subject: [Cython] Calling Python C-API functions having PyObject** arguments In-Reply-To: References: <4db580fd0903261640y5c8cb148y22da543a4a348485@mail.gmail.com> <49CC6895.9060708@behnel.de> <4db580fd0903262354i6ff86d2bk44793753d5ba5ac5@mail.gmail.com> Message-ID: <4db580fd0903270900g32225d0u3ec8a06cf0512e14@mail.gmail.com> >> That's exactly the function I'm using. ?My use case is iterating >> through a tree-like structure, where each node holds a dict of values >> and/or branches. > > If the nodes are cdef classes, it should be enough to type the attribute > as "cdef dict" and use the iter*() dict methods in your code. Especially > the improvement compared to the plain (untyped) iteritems() method call > (which requires redundant tuple packing and unpacking) should be > substantial. > >> When I encounter a branch, I iterate recursively >> over that branch before continuing iteration locally. ?The best way of >> implementing this, at least that I found, is to use a stack of node >> dictionaries references + position variables and invoke PyDict_Next() >> while pushing/popping as needed. > > That sounds like you are not using a recursive function here. If so, why not? Sorry, I should have been clearer. I'm writing an iterator class used to iterate through the tree, so I need to preserve state at every level between returns of __next__(). This stack approach is what I immediately thought of, and while I could use recursion to traverse the state stack, I ended up just using a loop. >> I now implemented it using the intermediate variable method given >> above. ?It works great. > > That's why I implemented that loop optimisation. :) Thanks! I use that all over the place in my code. >> BTW, this solution: >> >>> You can do a, so do &(a) to get a PyObject** >> >> gives me the error message >> >> /home/hoytak/workspace/paramtree/paramtree/paramtree.pyx:648:31: >> Taking address of non-lvalue >> >> Is that a bug? > > Hmm, I don't see an obvious problem with this, so it looks like a bug. > Could you file a bug report with a test case that shows how to reproduce > this? Okay, I'll do that, though I don't have time til later today. --Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From stefan_ml at behnel.de Fri Mar 27 17:16:20 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 27 Mar 2009 17:16:20 +0100 Subject: [Cython] gcc warning: implicit declaration of PyBool_CheckExact In-Reply-To: <49CCEBF3.1010606@devsoft.no> References: <49CCE9A9.6010201@devsoft.no> <49CCEA32.9020008@canonware.com> <49CCEBF3.1010606@devsoft.no> Message-ID: <49CCFBD4.8030500@behnel.de> J?rgen P. Tjern? wrote: > Jason Evans wrote: >> J?rgen P. Tjern? wrote: >>> This code, specifically the list comprehension, generates C code that >>> gives me: >>> warning: implicit declaration of function ?PyBool_CheckExact? That's a known bug in 0.10.3. It's fixed in 0.11. >> Use 'bint' instead of 'bool'. (I've also tripped on this at various times.) > Ah, thanks a lot. That fixes it nicely! That's the right thing to do anyway. "bint" is a C type, "bool" is a Python type. Stefan From stefan_ml at behnel.de Fri Mar 27 17:30:32 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 27 Mar 2009 17:30:32 +0100 Subject: [Cython] Wrong ordering of functions (function being implicitly defined) In-Reply-To: <49CCF30B.90204@devsoft.no> References: <49CCEAF2.2020606@devsoft.no> <49CCEC16.8020508@student.matnat.uio.no> <49CCEE10.8040700@devsoft.no> <49CCF30B.90204@devsoft.no> Message-ID: <49CCFF28.1050509@behnel.de> J?rgen P. Tjern? wrote: > cdef class Album > > cdef class SessionStruct: > cdef Album _create_album(self, album* album, bint take_owner=?) > > cdef class Album(SessionStruct): > pass This is some unusual code. Letting a superclass know its subtypes is rarely a good design. Anyway, I consider it a bug that it doesn't raise an error because of the missing base class in the pre-declaration. http://trac.cython.org/cython_trac/ticket/262 It should be declared like this: cdef class SessionStruct cdef class Album(SessionStruct) cdef class SessionStruct: cdef Album _create_album(self, album* album, bint take_owner=?) cdef class Album(SessionStruct): pass Stefan From jorgen at devsoft.no Fri Mar 27 17:40:21 2009 From: jorgen at devsoft.no (=?UTF-8?B?IkrDuHJnZW4gUC4gVGplcm7DuCI=?=) Date: Fri, 27 Mar 2009 17:40:21 +0100 Subject: [Cython] Wrong ordering of functions (function being implicitly defined) In-Reply-To: <49CCFF28.1050509@behnel.de> References: <49CCEAF2.2020606@devsoft.no> <49CCEC16.8020508@student.matnat.uio.no> <49CCEE10.8040700@devsoft.no> <49CCF30B.90204@devsoft.no> <49CCFF28.1050509@behnel.de> Message-ID: <49CD0175.7000905@devsoft.no> Stefan Behnel wrote: > J?rgen P. Tjern? wrote: >> cdef class SessionStruct: >> cdef Album _create_album(self, album* album, bint take_owner=?) >> >> cdef class Album(SessionStruct): >> pass > > This is some unusual code. Letting a superclass know its subtypes is rarely > a good design. True. It's more of a "gather state and methods that all these classes need in a single place", so it could just as well be represented as a "has-a" instead of a "is-a". :-) > Anyway, I consider it a bug that it doesn't raise an error because of the > missing base class in the pre-declaration. Ah, great. :) > It should be declared like this: > > cdef class SessionStruct > cdef class Album(SessionStruct) I'll update my code to reflect this. Thanks! -- Kindest regards, J?rgen P. Tjern?. From stefan_ml at behnel.de Fri Mar 27 18:40:06 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 27 Mar 2009 18:40:06 +0100 Subject: [Cython] program analysis using Datalog and binary decision diagrams Message-ID: <49CD0F76.2050406@behnel.de> Hi, I found this article on using Datalog for implementing program analysis algorithms: http://people.csail.mit.edu/mcarbin/papers/aplas05.pdf Here is a quick blog intro: http://blog.robjsoftware.org/2007/10/bddbddb-pointer-alias-analysis-comes-of.html I don't know if this fits into the Cython project at all. The tool they are using, bddbddb, is written in Java, for example, and it currently only works on Java and C. http://bddbddb.sourceforge.net/ I just found it a good idea to use binary decision diagrams together with Datalog for this, and I think it's something to keep in mind for the GSoC project. It reads very sound and efficient. BTW, another thing probably worth keeping in mind is PyPy. Stefan From stefan_ml at behnel.de Fri Mar 27 19:54:19 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 27 Mar 2009 19:54:19 +0100 Subject: [Cython] Shorthand for getter properties? In-Reply-To: <49CCCB25.1050203@devsoft.no> References: <49CCCB25.1050203@devsoft.no> Message-ID: <49CD20DB.5060907@behnel.de> Hi, J?rgen P. Tjern? wrote: > I'm currently using Cython to provide some bindings to a C api. I'm > using classes to "wrap" the various C structs, but they're VERY > repetetive to write, since many of the attribute I expose are in the form: > > property title: > def __get__(self): > return self.track.title > > Is there any shorthand for this? :-) No. We could potentially transform @property def title(self): return self.track.title into the above code, but that wouldn't be that much shorter either. http://trac.cython.org/cython_trac/ticket/264 Note that there is a shortcut for direct class attributes, though, so you could write class SomeClass: cdef readonly object some_attribute to get a read-only property "some_attribute" that is still accessed efficiently (and is writable) from Cython code. But this does not work for your case above. > Also, I want to initialize them with a C struct pointer (they're only > ever initialized by my Cython code, not by the user themselves), is > there a better way than: > > cdef class Artist: > def __init__(self): > raise TypeError("This class cannot be instantiated from Python") > > cdef extern from "foo.h": > cdef Artist NEW_ARTIST "PY_NEW" (object t) > > # #define PY_NEW(T) \ > # (((PyTypeObject*)(T))->tp_new( \ > # (PyTypeObject*)(T), __pyx_empty_tuple, NULL)) > > cdef Artist _create_artist(artist* artist, bool take_owner=False): > cdef Artist instance > > instance = NEW_TRACK(Artist) > instance.artist = artist > instance.take_owner = take_owner > return instance Again, we could potentially do some things here. http://trac.cython.org/cython_trac/ticket/263 However, this is a pretty common and also very fast way of doing this, so you can't currently do better. Stefan From robertwb at math.washington.edu Fri Mar 27 20:38:44 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 27 Mar 2009 12:38:44 -0700 Subject: [Cython] Unladen Swallow In-Reply-To: References: Message-ID: <2947E74B-6553-4872-B985-845C991AC2B3@math.washington.edu> On Mar 27, 2009, at 6:34 AM, Stefan Behnel wrote: > Looks like the competition has just started: > > http://code.google.com/p/unladen-swallow/wiki/ProjectPlan This came up on the sage mailiinglist too. Not sure what to make of it yet--we'll see. I think the idea has great potential, but I'm skeptical of the 5x goal. - Robert From robertwb at math.washington.edu Fri Mar 27 20:59:11 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 27 Mar 2009 12:59:11 -0700 Subject: [Cython] C++ exception mapping problems In-Reply-To: <49CA753A.4010302@tekno-soft.it> References: <49CA753A.4010302@tekno-soft.it> Message-ID: On Mar 25, 2009, at 11:17 AM, Simone Bisogni wrote: > Hi guys, > > I have a problem building a python c++ extension and, in > particular, the > problem occurs when I try to map C++ exceptions to python exceptions. > Here is a simple example: [...] > The errors occur because the the inclusion of the stdexcept header is > missing. > > Is it a cython bug or it is up to the user to add the inclusion? http://trac.cython.org/cython_trac/ticket/265 > Actually, I have solved the problem adding the following rows to my > Cpp_Rectangle.pxd file, but I don't know if it is the right way to > procede: > > cdef extern from "stdexcept": > pass Yes, that should work well for now. > ====================================================================== > ==== > > Another error happens when I change my Cpp_Rectangle.pxd like this: > > ====================================================================== > ==== > > ### Cpp_Rectangle.pxd > > cdef extern from "stdexcept": > pass > > cdef extern from "Rectangle.h": > ctypedef struct cpp_Rectangle "Rectangle": > > int raiseException() except +MemoryError > > cpp_Rectangle* new_Rectangle "new Rectangle"( int x0, int y0, int > x1, > int y1 ) > void delete_Rectangle "delete"( cpp_Rectangle* rect ) > > ====================================================================== > === > > Basically I am trying now to map each exception thrown by the > raiseException to a python MemoryError exception. > > The cython compilation runs fine again, but when I try to compile the > .cpp file I got the errors: > > ====================================================================== > === > > [simone at spider Cython_Test]$ g++ -shared -fPIC -I/usr/include/ > python2.5 > -o PyRectangle.so PyRectangle.cpp -L /usr/lib64 -L. -lRectangle - > lpython2.5 > PyRectangle.cpp: In function 'PyObject* > __pyx_pf_11PyRectangle_9Rectangle_raiseException(PyObject*, > PyObject*)': > PyRectangle.cpp:454: error: '__pyx_builtin_MemoryError' was not > declared > in this scope > > ====================================================================== > ==== > > The accused piece of code is the following: > > ====================================================================== > ==== > > /* __pyx_t_1 allocated */ > try { > __pyx_t_1 = > ((struct__pyx_obj_11PyRectangle_Rectangle*)__pyx_v_self)->thisptr- > >raiseException();} > > > catch(...) {PyErr_SetString(__pyx_builtin_MemoryError, ""); > {__pyx_filename = __pyx_f[0]; __pyx_lineno = 14; __pyx_clineno = > __LINE__; goto __pyx_L1_error;}} > > ====================================================================== > ======= > > The variable __pyx_builtin_MemoryError is not declared anywhere. > Should the __pyx_builtin_MemoryError be substituted with > PyExc_MemoryError? http://trac.cython.org/cython_trac/ticket/266 Perhaps writing foo = MemoryError somewhere in your file will be a temporary fix. > The version of cython I am using is the 0.11 > > Thanks in advance and sorry for my long email. Thanks for the bug reports. - Robert From robertwb at math.washington.edu Fri Mar 27 21:05:34 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 27 Mar 2009 13:05:34 -0700 Subject: [Cython] gcc warning: implicit declaration of PyBool_CheckExact In-Reply-To: <49CCFBD4.8030500@behnel.de> References: <49CCE9A9.6010201@devsoft.no> <49CCEA32.9020008@canonware.com> <49CCEBF3.1010606@devsoft.no> <49CCFBD4.8030500@behnel.de> Message-ID: <93576ABB-DA05-4D52-83CE-B4EB736706E9@math.washington.edu> On Mar 27, 2009, at 9:16 AM, Stefan Behnel wrote: > J?rgen P. Tjern? wrote: >> Jason Evans wrote: >>> J?rgen P. Tjern? wrote: >>>> This code, specifically the list comprehension, generates C code >>>> that >>>> gives me: >>>> warning: implicit declaration of function ?PyBool_CheckExact? > > That's a known bug in 0.10.3. It's fixed in 0.11. > > >>> Use 'bint' instead of 'bool'. (I've also tripped on this at >>> various times.) >> Ah, thanks a lot. That fixes it nicely! > > That's the right thing to do anyway. "bint" is a C type, "bool" is > a Python > type. To be more specific, bool is a C++ type, but has different semantics than bint. In Cython, bint is a c int (and may have several distinct values, e.g. 2 != 1. The only difference is that the conversion routines from Python objects go via PyObject_IsTrue and it becomes True and False when going to a Python object. - Robert From robertwb at math.washington.edu Fri Mar 27 21:07:49 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 27 Mar 2009 13:07:49 -0700 Subject: [Cython] Calling Python C-API functions having PyObject** arguments In-Reply-To: References: <4db580fd0903261640y5c8cb148y22da543a4a348485@mail.gmail.com> <49CC6895.9060708@behnel.de> <4db580fd0903262354i6ff86d2bk44793753d5ba5ac5@mail.gmail.com> Message-ID: On Mar 27, 2009, at 6:58 AM, Lisandro Dalcin wrote: > On Fri, Mar 27, 2009 at 5:14 AM, Stefan Behnel > wrote: >> Hoyt Koepke wrote: >> >>> BTW, this solution: >>> >>>> You can do a, so do &(a) to get a PyObject** >>> >>> gives me the error message >>> >>> /home/hoytak/workspace/paramtree/paramtree/paramtree.pyx:648:31: >>> Taking address of non-lvalue >>> >>> Is that a bug? >> >> Hmm, I don't see an obvious problem with this, so it looks like a >> bug. >> Could you file a bug report with a test case that shows how to >> reproduce >> this? >> > > But in any case, please DO NOT let code being able to do & > (a). Actually, this is a non-lvalue, so we do need a temp here. > I'm almost sure this will completely confuse the refnanny machinery. Anytime you mess with PyObject* you're able to completely confuse the refnanny machinery. - Robert From robertwb at math.washington.edu Fri Mar 27 21:15:55 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 27 Mar 2009 13:15:55 -0700 Subject: [Cython] Enable for-in-range optimisation also for untyped loop vars? In-Reply-To: <49CC995E.6080500@student.matnat.uio.no> References: <49CC7130.5020805@behnel.de> <49CC995E.6080500@student.matnat.uio.no> Message-ID: <02C4DF05-B7AA-4A8F-806D-F5B19E2EE543@math.washington.edu> On Mar 27, 2009, at 2:16 AM, Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Hi, >> >> now that we can guarantee Python semantics for the optimised for- >> in-range >> loop, should we enable the optimisation also when the loop >> variable is not >> explicitly typed, but when the range is constant and clearly >> within int bounds? > > If this is done, I'd go even further and always do it. At least under > Python 2: > >>>> for i in xrange(10**90): pass > ... > Traceback (most recent call last): > File "", line 1, in > OverflowError: long int too large to convert to int > > But I really thing that range() is such a common use, and super- > ints so > seldom, that I'd propose this as the one place it would make sense to > break Python compatability also if Py3 fixes this (but keep a > cython.bigrange which isn't optimized -- Sage has a corresponding > srange > I think). >>> range(10**30, 10**30+10) [1000000000000000000000000000000L, 1000000000000000000000000000001L, 1000000000000000000000000000002L, 1000000000000000000000000000003L, 1000000000000000000000000000004L, 1000000000000000000000000000005L, 1000000000000000000000000000006L, 1000000000000000000000000000007L, 1000000000000000000000000000008L, 1000000000000000000000000000009L] but xrange is buggy to not be able to handle this (IMHO). >> We might consider waiting with these things until we have flow >> analysis in >> place to make sure the loop variable isn't assigned non-int values >> inside >> the loop, but I doubt that this will be a common case. > > My instinct is to wait until we can start implementing some kind of > simplistic type inference transform, with this as the first simple > case > to deal with, rather than specific code for the for-in-range. That > would > depend on flow analysis though. > > So my instinct is to wait, but practicality might push this special > case > ahead as you say. I'm not sure. I felt the same, but on the other hand it is a super common case. def do_range(N): for i in range(N): pass def do_from(N): for i from 0 <= i < N: pass sage: time do_range(10**7) CPU times: user 0.44 s, sys: 0.05 s, total: 0.49 s Wall time: 0.49 s sage: time do_from(10**7) CPU times: user 0.15 s, sys: 0.00 s, total: 0.15 s Wall time: 0.15 s seems to justify it. - Robert From stefan_ml at behnel.de Fri Mar 27 21:23:51 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 27 Mar 2009 21:23:51 +0100 Subject: [Cython] Calling Python C-API functions having PyObject** arguments In-Reply-To: References: <4db580fd0903261640y5c8cb148y22da543a4a348485@mail.gmail.com> <49CC6895.9060708@behnel.de> <4db580fd0903262354i6ff86d2bk44793753d5ba5ac5@mail.gmail.com> Message-ID: <49CD35D7.5030306@behnel.de> Robert Bradshaw wrote: >> On Fri, Mar 27, 2009 at 5:14 AM, Stefan Behnel wrote: >>> Hoyt Koepke wrote: >>>>> You can do a, so do &(a) to get a PyObject** >>>> gives me the error message >>>> >>>> /home/hoytak/workspace/paramtree/paramtree/paramtree.pyx:648:31: >>>> Taking address of non-lvalue >>>> >>>> Is that a bug? >>> Hmm, I don't see an obvious problem with this, so it looks like a >>> bug. > > Actually, this is a non-lvalue, so we do need a temp here. Ah, right, I got confused. Actually, it's not even that bad that this uses a temp variable - although it wouldn't be strictly necessary for Python objects. This actually makes it clear to users that they are doing dangerous stuff that needs special consideration. Stefan From wstein at gmail.com Fri Mar 27 22:02:29 2009 From: wstein at gmail.com (William Stein) Date: Fri, 27 Mar 2009 14:02:29 -0700 Subject: [Cython] [sage-devel] Re: Python Performance Improvement In-Reply-To: <836d65f2-fb78-48a1-a630-944673ec3035@e38g2000yqa.googlegroups.com> References: <0afffcf4-d2b7-49d6-9f18-312f955000ce@z15g2000yqm.googlegroups.com> <85e81ba30903262059t2c424acei30270f4efb8735ce@mail.gmail.com> <836d65f2-fb78-48a1-a630-944673ec3035@e38g2000yqa.googlegroups.com> Message-ID: <85e81ba30903271402t6ddbee6byc75d067c526a1b40@mail.gmail.com> (In reference to http://arstechnica.com/open-source/news/2009/03/google-launches-project-to-boost-python-performance-by-5x.ars) On Fri, Mar 27, 2009 at 11:37 AM, Bill Hart wrote: > > Could you be more specific about what you think these guys don't > understand? > > To a non-python expert type there isn't much difference between the > article and the wiki information. > > I was very impressed at the goals as spelled out on the wiki and they > seem to have a decent roadmap up. > > Is it the goals themselves which you feel are bollucks or do you think > they don't understand the technical workings of python? Perhaps, as a > non-expert, I won't understand, but I am interested to hear, if you > have the time to explain. All I wrote before was "there's not much on that wiki that gives me confidence the project members know what they are doing." I mean that literally, i.e., that there wasn't anything that gives me confidence that they know what they're doing. I didn't mean to say "I think they don't know what they are doing," since I don't know if they do or not. Why should a a few guys who read some papers be able to massively beat (a factor of 5!?) the work of the entire Python development community over 15 years? I certainly didn't see anything to convince me that they can pull that off. I'm certainly not saying they won't or can't. In fact, I really hope they do since it would benefit us all. It's just that with Python every few months it seems like another project pops up with the goal to speed up Python by a factor of 5-10, and they fall by the wayside after a few years... except Cython, which actually succeeds. -- William -- William Stein Associate Professor of Mathematics University of Washington http://wstein.org From robertwb at math.washington.edu Fri Mar 27 22:28:20 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 27 Mar 2009 14:28:20 -0700 Subject: [Cython] About GSoC 2009 In-Reply-To: <45239150903240641x2334ebf3xf7b5a659e78eb5f3@mail.gmail.com> References: <49C7A906.6040502@student.matnat.uio.no> <49C7AB08.10906@student.matnat.uio.no> <45239150903240641x2334ebf3xf7b5a659e78eb5f3@mail.gmail.com> Message-ID: <1EBB4B4B-45F0-4305-88A4-8FCD1C3353C4@math.washington.edu> On Mar 24, 2009, at 6:41 AM, Danilo Freitas wrote: > Even with a know knowledge about Cython, I'm interested in > participate of GSoC. > I thought it would be interesting work on C++ Support, even why there > are many important features in C++ that could be used to improve > Cython, just like STL. > > If I get a mentor for it, I'd be grateful Please write up a proposal. One of the first tasks is getting a good syntax--it would be nice to get this nailed down so by the time GSoC starts you could start coding right away. - Robert From robertwb at math.washington.edu Fri Mar 27 22:31:26 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 27 Mar 2009 14:31:26 -0700 Subject: [Cython] A Trove classifier for Cython? In-Reply-To: <0EA97A4E-D300-4698-AAFC-85E1AF0D65DD@math.washington.edu> References: <49C40348.3000104@behnel.de> <04ED9E4A-1030-44A0-B5AB-AA2EEE0B4D29@math.washington.edu> <49C48896.5040308@v.loewis.de> <0EA97A4E-D300-4698-AAFC-85E1AF0D65DD@math.washington.edu> Message-ID: <9866BAA7-8BB2-4D60-8EBD-B61C3614459F@math.washington.edu> On Mar 21, 2009, at 12:31 PM, Robert Bradshaw wrote: > On Mar 21, 2009, at 7:36 AM, Lisandro Dalcin wrote: > >> On Sat, Mar 21, 2009 at 10:00 AM, Tres Seaver >> wrote: >>> -----BEGIN PGP SIGNED MESSAGE----- >>> Hash: SHA1 >>> >>> Martin v. L?wis wrote: >>>> Robert Bradshaw wrote: >>>>> Could we get a trove classifier for the programing language >>>>> Cython? >>>>> >>>>> Programming Language :: Cython >>>> >>>> What packages would be classified under this classifier (both >>>> specific, and in principle)? >>> >> To follow up, looks like it's been added http://pypi.python.org/pypi?%3Aaction=list_classifiers Thanks. - Robert From dalcinl at gmail.com Fri Mar 27 22:33:07 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 27 Mar 2009 18:33:07 -0300 Subject: [Cython] Calling Python C-API functions having PyObject** arguments In-Reply-To: References: <4db580fd0903261640y5c8cb148y22da543a4a348485@mail.gmail.com> <49CC6895.9060708@behnel.de> <4db580fd0903262354i6ff86d2bk44793753d5ba5ac5@mail.gmail.com> Message-ID: On Fri, Mar 27, 2009 at 5:07 PM, Robert Bradshaw wrote: > > Anytime you mess with PyObject* you're able to completely confuse the > refnanny machinery. > Are you sure? I believe It is rather hard to break it... Of course, as long as you do not actually force deallocation of some PyObject*, and in that case you'll likely segfault Python instead of confusing the refnanny. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Fri Mar 27 22:48:38 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 27 Mar 2009 14:48:38 -0700 Subject: [Cython] Calling Python C-API functions having PyObject** arguments In-Reply-To: References: <4db580fd0903261640y5c8cb148y22da543a4a348485@mail.gmail.com> <49CC6895.9060708@behnel.de> <4db580fd0903262354i6ff86d2bk44793753d5ba5ac5@mail.gmail.com> Message-ID: On Mar 27, 2009, at 2:33 PM, Lisandro Dalcin wrote: > On Fri, Mar 27, 2009 at 5:07 PM, Robert Bradshaw > wrote: >> >> Anytime you mess with PyObject* you're able to completely confuse the >> refnanny machinery. >> > > Are you sure? I believe It is rather hard to break it... Of course, as > long as you do not actually force deallocation of some PyObject*, and > in that case you'll likely segfault Python instead of confusing the > refnanny. If you incref, decref, or otherwise call into the C/API that steals/ returns new references, it messes it up. Just passing it around won't hurt of course. - Robert From stefan_ml at behnel.de Fri Mar 27 23:17:46 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 27 Mar 2009 23:17:46 +0100 Subject: [Cython] [sage-devel] Re: Python Performance Improvement In-Reply-To: <85e81ba30903271402t6ddbee6byc75d067c526a1b40@mail.gmail.com> References: <0afffcf4-d2b7-49d6-9f18-312f955000ce@z15g2000yqm.googlegroups.com> <85e81ba30903262059t2c424acei30270f4efb8735ce@mail.gmail.com> <836d65f2-fb78-48a1-a630-944673ec3035@e38g2000yqa.googlegroups.com> <85e81ba30903271402t6ddbee6byc75d067c526a1b40@mail.gmail.com> Message-ID: <49CD508A.7010700@behnel.de> William Stein wrote: > Why should a a few guys who read some papers be able to massively beat > (a factor of 5!?) the work of the entire Python development community > over 15 years? I certainly didn't see anything to convince me that > they can pull that off. Those were my doubts, too. It's a project that was started without even involving the core CPython developers. But they seem to be good in marketing. ;) I know that technical revolutions are possible, but I also know that "removing the GIL and ref-counting" has serious implications that will necessarily break things, especially for extension modules. > I'm certainly not saying they won't or > can't. In fact, I really hope they do since it would benefit us all. > It's just that with Python every few months it seems like another > project pops up with the goal to speed up Python by a factor of 5-10, > and they fall by the wayside after a few years... except Cython, which > actually succeeds. Cython is far from giving you a factor of 5 for 'normal' Python code, i.e. without any type annotations, but it can give you a factor of 100 and more if you get things right. We know that there is quite some potential in type inference, but you have to do a lot more program analysis than we have today to really know when you can safely move things into C space (and how). I think the main reason why Cython works so well is that it can improve gradually. Everything that it doesn't generate nicely looking code for is a bug, and it just gets better every day. Removing GIL and ref-counting from CPython is a major project that it's not that satisfying to do 10% of. Stefan From dagss at student.matnat.uio.no Sat Mar 28 11:07:01 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 28 Mar 2009 11:07:01 +0100 Subject: [Cython] Changes to compiler directive rules? Message-ID: <49CDF6C5.5040705@student.matnat.uio.no> Now that compiler directives will become really important for some folks we should solve this. First, it seems that "--directive" can be used to set compiler directives from the command line; however there's no shortcut (despite the command line documentation, the -X switch is used for some Mac OS X experimental stuff -- can we remove this and use -X for the directives? --link could still be used for the Mac OS X thing). Secondly, I propose a change of priorities. Currently a directive is set by these precedence rules (i.e. start at the top of the list and continue until a setting is found): 1) with statement and decorators w/ block level 2) command line switch --directive 3) #cython comment at top of file 4) builtin defaults Now that we see what kind of things this can be used for (i.e. cdivision), I propose that the 2) and 3) are exchanged, so that #cython comments take precedence before command line. If there's a need for the current behaviour, we could introduce #cython default: nonecheck=True or similar as another mechanism to be checked after the command line. -- Dag Sverre From dalcinl at gmail.com Sat Mar 28 17:27:50 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sat, 28 Mar 2009 13:27:50 -0300 Subject: [Cython] Changes to compiler directive rules? In-Reply-To: <49CDF6C5.5040705@student.matnat.uio.no> References: <49CDF6C5.5040705@student.matnat.uio.no> Message-ID: On Sat, Mar 28, 2009 at 7:07 AM, Dag Sverre Seljebotn wrote: > > 1) with statement and decorators w/ block level > 2) command line switch --directive > 3) #cython comment at top of file > 4) builtin defaults > > Now that we see what kind of things this can be used for (i.e. > cdivision), I propose that the 2) and 3) are exchanged, so that #cython > comments take precedence before command line. > Definitely +1 for me. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Sat Mar 28 18:58:35 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 28 Mar 2009 18:58:35 +0100 Subject: [Cython] Better call optimisation In-Reply-To: <49B56458.4030505@student.matnat.uio.no> References: <36891.213.61.181.86.1236601071.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B56458.4030505@student.matnat.uio.no> Message-ID: <49CE654B.6010808@behnel.de> Just a quick follow-up that this is ticket #233. http://trac.cython.org/cython_trac/ticket/233 Stefan Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> there are a couple of tickets related to call optimisations. >> >> http://trac.cython.org/cython_trac/ticket/3 >> http://trac.cython.org/cython_trac/ticket/131 >> http://trac.cython.org/cython_trac/ticket/166 >> >> The getattr3() problem also relates to other cases, such as the fact that >> we can't currently optimise >> >> cdef list l = [...] >> l.sort() >> >> into a call to PyList_Sort(), as this would break calls like >> >> l.sort(key=...) >> >> My suggestion is to remove the current support for optimised methods of >> builtins, and to write a dedicated transform that intercepts the tree on >> call nodes and that can then handle all cases in explicit code. This can >> include, for example, adding a None check if required, or handling >> different call signatures only some of which can be optimised, maybe even >> by changing keyword arguments into positional arguments of a matching >> C-API function. >> >> I imagine that it would use some kind of dispatch mechanism on >> >> (obj_type, obj_attribute_name) >> >> and on builtin function names. I also think that the current generic >> "X.append()" optimisation (i.e. not only the one for typed lists) should >> go there. > > Sounds good! > > One thing: I think it would be good if the output is something very > targeted (i.e. not simply new SimpleCallNodes or similar). From the > refnanny it seemed that very many CPython API calls used the same > pattern, so perhaps something along the lines of a PythonApiCall-node, > perhaps with a NoneCheckNode as its "subject", and so on. (Some of the > resulting operations could warrant their own node, while others are > similar enough to not warrant new classes for each -- the point is that > I think it should be more "targeted" than new SimpleCallNodes). From robertwb at math.washington.edu Sat Mar 28 20:33:35 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 28 Mar 2009 12:33:35 -0700 Subject: [Cython] Changes to compiler directive rules? In-Reply-To: References: <49CDF6C5.5040705@student.matnat.uio.no> Message-ID: <76C42D1E-99D9-48A8-BF34-3D812163CCC7@math.washington.edu> On Mar 28, 2009, at 9:27 AM, Lisandro Dalcin wrote: > On Sat, Mar 28, 2009 at 7:07 AM, Dag Sverre Seljebotn > wrote: >> >> 1) with statement and decorators w/ block level >> 2) command line switch --directive >> 3) #cython comment at top of file >> 4) builtin defaults >> >> Now that we see what kind of things this can be used for (i.e. >> cdivision), I propose that the 2) and 3) are exchanged, so that >> #cython >> comments take precedence before command line. >> > > Definitely +1 for me. +1 from me too. - Robert From robertwb at math.washington.edu Sat Mar 28 21:02:24 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 28 Mar 2009 13:02:24 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49C94AB4.2070604@student.matnat.uio.no> References: <49BF426D.7010005@behnel.de> <49BF5871.4040404@student.matnat.uio.no> <50876.213.61.181.86.1237279705.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49C14F0C.4030000@behnel.de> <49C87C5A.80607@behnel.de> <1D4BF5BC-A491-4CE1-B965-D9D5CAD91D9C@math.washington.edu> <49C94AB4.2070604@student.matnat.uio.no> Message-ID: <4EA7123B-D0EB-4A12-8B88-4A45F3AAB5A9@math.washington.edu> CEP 516 (http://wiki.cython.org/enhancements/division) is fully implemented and committed to the cython-devel branch. In 0.12, we will change the default to be Python semantics. - Robert From dagss at student.matnat.uio.no Sun Mar 29 12:50:19 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 29 Mar 2009 12:50:19 +0200 Subject: [Cython] 0.11.1 release timeframe? Message-ID: <49CF526B.6070305@student.matnat.uio.no> It appears we had a regression in 0.11; making the Cython compiler crash on code like this: f() if g() else h() A fix is up. It's strange that this escaped us but I guess it is a rather new Cython feature and so wasn't used that much in user code. I believe a regression like this is a case for releasing 0.11.1 sooner rather than later, and moving a lot of those tickets over to 0.11.2 ? (as they are not regressions). We've also had the for loops update which it would be nice to release sooner rather than later. -- Dag Sverre From tiagomdp at gmail.com Sun Mar 29 13:12:52 2009 From: tiagomdp at gmail.com (Tiago Pereira) Date: Sun, 29 Mar 2009 22:12:52 +1100 Subject: [Cython] Passing numpy arrays to C and vice-versa (again...) Message-ID: <49CF57B4.4020505@gmail.com> Hello, I know this topic has been discussed quite a few times in this mailing list. I have read and tried many of the suggestions given on this list, but nevertheless I'm still stuck. Here's my problem: I want to use a C function that takes a two dimensional array, float **a. How to I pass a numpy 2d array to **float? For a 1D array case I have managed to make it work by passing a.data to a C function (where a is a numpy 1D array). But for the 2D case, if I pass a.data python either segfaults or gives Bus Error. And if I try to pass only a.data, it says it can't assign type 'float *' 'to float **'. I have also used something of the kind (based on an example from this mailing list): cdef inline void iarray_d(ndarray a, float **data): if a.dtype != float32: raise TypeError("The array must have the dtype=float32.") if data!=NULL: data[0] = (a.data) But I still can't make it to work. Maybe this is because my ignorance in C arrays and pointers. My function requires a **float, but is there any way to pass the data as *float? Or shall I use another way in Cython to convert a numpy 2D array to **float? Or this is not yet supported in Cython? I remember a while ago using C directly to build some numpy extensions, and to convert the numpy 2D array to C array pointers (I can implement that in C, but I don't know how to do it in Cython...). Any obvious thing I'm doing wrong? Thanks! Tiago From robertwb at math.washington.edu Sun Mar 29 13:18:21 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 29 Mar 2009 04:18:21 -0700 Subject: [Cython] 0.11.1 release timeframe? In-Reply-To: <49CF526B.6070305@student.matnat.uio.no> References: <49CF526B.6070305@student.matnat.uio.no> Message-ID: On Mar 29, 2009, at 3:50 AM, Dag Sverre Seljebotn wrote: > It appears we had a regression in 0.11; making the Cython compiler > crash > on code like this: > > f() if g() else h() > > A fix is up. It's strange that this escaped us but I guess it is a > rather new Cython feature and so wasn't used that much in user code. http://trac.cython.org/cython_trac/ticket/267, right? Good catch. Also, I noticed you moved a bunch of other types over. > I believe a regression like this is a case for releasing 0.11.1 sooner > rather than later, Yep, I was actually thinking about this recently too. Maybe an RC next week, and a release soon after (assuming no big issues arise). > and moving a lot of those tickets over to 0.11.2 ? > (as they are not regressions). We've also had the for loops update > which > it would be nice to release sooner rather than later. My only hesitance about loops is that it changes (for the better) the semantics of some loops. Hopefully people haven't been relying on the old behavior, but is that too much for a 0.11.x release? Granted, it would only affect people abusing wrong corner cases. - Robert From cournape at gmail.com Sun Mar 29 13:22:18 2009 From: cournape at gmail.com (David Cournapeau) Date: Sun, 29 Mar 2009 20:22:18 +0900 Subject: [Cython] Passing numpy arrays to C and vice-versa (again...) In-Reply-To: <49CF57B4.4020505@gmail.com> References: <49CF57B4.4020505@gmail.com> Message-ID: <5b8d13220903290422q30c226c2m537fe63e7b26bbaa@mail.gmail.com> On Sun, Mar 29, 2009 at 8:12 PM, Tiago Pereira wrote: > Hello, > > I know this topic has been discussed quite a few times in this mailing > list. I have read and tried many of the suggestions given on this list, > but nevertheless I'm still stuck. > > Here's my problem: I want to use a C function that takes a two > dimensional array, float **a. How to I pass a numpy 2d array to **float? > For a 1D array case I have managed to make it work by passing *>a.data to a C function (where a is a numpy 1D array). But for the 2D > case, if I pass a.data python either segfaults or gives Bus > Error. And if I try to pass only a.data, it says it can't > assign type 'float *' 'to float **'. Yes, that's for your own good :) It depends on your C function, but generally, when a function expects a 2d array as a float**, it means it expects a pointer to an array which keeps the *address* (hence float** -> pointer to an array of pointers) of each row. Which means you have to create such an array first (those kind of arrays are also called ragged arrays, because each row can potentially be of a different size). Something like (pseudo code): # a is the original array, n rows/m rows, in C order, as a float*, b is an array of pointers, that is defined as float** b for i in range(nrows): b[i] = &a[i*m] cheers, David From tiagomdp at gmail.com Sun Mar 29 13:49:53 2009 From: tiagomdp at gmail.com (Tiago Pereira) Date: Sun, 29 Mar 2009 22:49:53 +1100 Subject: [Cython] Passing numpy arrays to C and vice-versa (again...) In-Reply-To: <5b8d13220903290422q30c226c2m537fe63e7b26bbaa@mail.gmail.com> References: <49CF57B4.4020505@gmail.com> <5b8d13220903290422q30c226c2m537fe63e7b26bbaa@mail.gmail.com> Message-ID: <49CF6061.5@gmail.com> Thanks for your quick reply. David Cournapeau wrote: > # a is the original array, n rows/m rows, in C order, as a float*, b > is an array of pointers, that is defined as float** b > for i in range(nrows): > b[i] = &a[i*m] I've tried doing something like this, but I still get a bus error. Maybe I'm doing something very dumb. Here's what I have. For simplicity let's start with a square array (N,N), and a C function called sum1 that basically sums all the elements in the array. Here's what my pyx looks like: import numpy as np cimport numpy as np DTYPE = np.float32 ctypedef np.float32_t DTYPE_t cdef extern float sum1(float **a,int m) def matmul1(np.ndarray[DTYPE_t, ndim=2] a): cdef int N = a.shape[0] cdef float *c_arr cdef float **b c_arr = a.data for i in range(N): b[i] = &c_arr[i*N] return sum1(b,N) Compiling and running this in python with a 2D array of the type np.float32 gives a bus error. Any ideas? Thanks, Tiago From robertwb at math.washington.edu Sun Mar 29 13:56:40 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 29 Mar 2009 04:56:40 -0700 Subject: [Cython] Passing numpy arrays to C and vice-versa (again...) In-Reply-To: <49CF6061.5@gmail.com> References: <49CF57B4.4020505@gmail.com> <5b8d13220903290422q30c226c2m537fe63e7b26bbaa@mail.gmail.com> <49CF6061.5@gmail.com> Message-ID: <8D74ED18-0E87-4C4F-A006-8A335DB4C976@math.washington.edu> On Mar 29, 2009, at 4:49 AM, Tiago Pereira wrote: > Thanks for your quick reply. > > David Cournapeau wrote: > >> # a is the original array, n rows/m rows, in C order, as a float*, b >> is an array of pointers, that is defined as float** b >> for i in range(nrows): >> b[i] = &a[i*m] > > I've tried doing something like this, but I still get a bus error. > Maybe > I'm doing something very dumb. Here's what I have. For simplicity > let's > start with a square array (N,N), and a C function called sum1 that > basically sums all the elements in the array. Here's what my pyx > looks like: > > import numpy as np > cimport numpy as np > DTYPE = np.float32 > ctypedef np.float32_t DTYPE_t > > cdef extern float sum1(float **a,int m) > > def matmul1(np.ndarray[DTYPE_t, ndim=2] a): > cdef int N = a.shape[0] > cdef float *c_arr > cdef float **b > > c_arr = a.data > > for i in range(N): > b[i] = &c_arr[i*N] > > return sum1(b,N) > > > Compiling and running this in python with a 2D array of the type > np.float32 gives a bus error. Any ideas? You have to manually allocate b. Also, it's not guaranteed that the array is contiguous--you may need to look at the stride information. - Robert From tiagomdp at gmail.com Sun Mar 29 14:24:24 2009 From: tiagomdp at gmail.com (Tiago Pereira) Date: Sun, 29 Mar 2009 23:24:24 +1100 Subject: [Cython] Passing numpy arrays to C and vice-versa (again...) In-Reply-To: <8D74ED18-0E87-4C4F-A006-8A335DB4C976@math.washington.edu> References: <49CF57B4.4020505@gmail.com> <5b8d13220903290422q30c226c2m537fe63e7b26bbaa@mail.gmail.com> <49CF6061.5@gmail.com> <8D74ED18-0E87-4C4F-A006-8A335DB4C976@math.washington.edu> Message-ID: <49CF6878.4080707@gmail.com> Robert Bradshaw wrote: > You have to manually allocate b. Also, it's not guaranteed that the > array is contiguous--you may need to look at the stride information. I understand the part about contiguous, and I've seen examples on how to deal with it. So if I manually allocate, I guess I'll have to free it later too. But forgive me my ignorance, how do I do that in cython? In C I would do: b = (float **) malloc(N * sizeof(float*)); for(i = 0; i < N; i++) b[i] = (float *) malloc(N * sizeof(float)); and to free: for(i=0;i References: <49CF57B4.4020505@gmail.com> <5b8d13220903290422q30c226c2m537fe63e7b26bbaa@mail.gmail.com> <49CF6061.5@gmail.com> <8D74ED18-0E87-4C4F-A006-8A335DB4C976@math.washington.edu> <49CF6878.4080707@gmail.com> Message-ID: <49CF7614.6060900@student.matnat.uio.no> Tiago Pereira wrote: > Robert Bradshaw wrote: > >> You have to manually allocate b. Also, it's not guaranteed that the >> array is contiguous--you may need to look at the stride information. > > I understand the part about contiguous, and I've seen examples on how to > deal with it. So if I manually allocate, I guess I'll have to free it > later too. But forgive me my ignorance, how do I do that in cython? In C > I would do: > > b = (float **) malloc(N * sizeof(float*)); > for(i = 0; i < N; i++) > b[i] = (float *) malloc(N * sizeof(float)); No, only the first line. You assign b[i] to point to memory in your contiguous array; which is already allocated. Anyway, you do the same in Cython: from stdlib cimport free, malloc -- Dag Sverre From tiagomdp at gmail.com Sun Mar 29 15:53:05 2009 From: tiagomdp at gmail.com (Tiago Pereira) Date: Mon, 30 Mar 2009 00:53:05 +1100 Subject: [Cython] Passing numpy arrays to C and vice-versa (again...) In-Reply-To: <49CF7614.6060900@student.matnat.uio.no> References: <49CF57B4.4020505@gmail.com> <5b8d13220903290422q30c226c2m537fe63e7b26bbaa@mail.gmail.com> <49CF6061.5@gmail.com> <8D74ED18-0E87-4C4F-A006-8A335DB4C976@math.washington.edu> <49CF6878.4080707@gmail.com> <49CF7614.6060900@student.matnat.uio.no> Message-ID: <49CF7D41.9010905@gmail.com> Dag Sverre Seljebotn wrote: > No, only the first line. You assign b[i] to point to memory in your > contiguous array; which is already allocated. > > Anyway, you do the same in Cython: > > from stdlib cimport free, malloc > Thanks! It now works. I had it working before but just using a C interface function to do what now is done in Cython: def matmul1(np.ndarray[DTYPE_t, ndim=2] a): cdef int N = a.shape[0] cdef float **b cdef float res b = malloc(N*sizeof(float*)) for i in range(N): b[i] = &(a.data)[i*N] res = sum1(b,N) free(b) return res Now, just another question... How do I do the reverse? Ie., convert from **float to numpy? I looked in previous emails and I saw something using np.memcpy. Adapting for 2d it would be something like this (assuming a float **res): cdef np.ndarray[DTYPE_t,ndim=2]result = np.zeros((N,N),dtype=DTYPE) if data != NULL: np.memcpy(result.data,res,N*N*sizeof(float)) The problem is res is an array of pointers (so I guess this does not work), but cython also complains that it can't convert float **res to Python type. So it seems memcpy is expecting a Python type, which means this example would never work... I can always use brute force and do a couple of loops for setting result[i,j]=res[i][j]. But I wonder if there is a more elegant way of doing this. Thanks again, Tiago From dagss at student.matnat.uio.no Sun Mar 29 16:06:50 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 29 Mar 2009 16:06:50 +0200 (CEST) Subject: [Cython] Passing numpy arrays to C and vice-versa (again...) In-Reply-To: <49CF7D41.9010905@gmail.com> References: <49CF57B4.4020505@gmail.com> <5b8d13220903290422q30c226c2m537fe63e7b26bbaa@mail.gmail.com> <49CF6061.5@gmail.com> <8D74ED18-0E87-4C4F-A006-8A335DB4C976@math.washington.edu> <49CF6878.4080707@gmail.com> <49CF7614.6060900@student.matnat.uio.no> <49CF7D41.9010905@gmail.com> Message-ID: Tiago Pereira wrote: > Now, just another question... How do I do the reverse? Ie., convert from > **float to numpy? I looked in previous emails and I saw something using > np.memcpy. Adapting for 2d it would be something like this (assuming a > float **res): > > cdef np.ndarray[DTYPE_t,ndim=2]result = np.zeros((N,N),dtype=DTYPE) > if data != NULL: np.memcpy(result.data,res,N*N*sizeof(float)) > > The problem is res is an array of pointers (so I guess this does not > work), but cython also complains that it can't convert float **res to > Python type. So it seems memcpy is expecting a Python type, which means > this example would never work... You should use memcpy from C instead to work with pointers. Then, do the same thing as before (loop over the rows of the array) but doing a memcpy instead for each row. > I can always use brute force and do a couple of loops for setting > result[i,j]=res[i][j]. But I wonder if there is a more elegant way of > doing this. Well, there's no way a float** will fit within the NumPy memory model, so the memory must be copied somehow. Using a memcpy per row will likely be faster but is fundamentally the same thing. Dag Sverre From cournape at gmail.com Sun Mar 29 16:08:38 2009 From: cournape at gmail.com (David Cournapeau) Date: Sun, 29 Mar 2009 23:08:38 +0900 Subject: [Cython] Passing numpy arrays to C and vice-versa (again...) In-Reply-To: <49CF7D41.9010905@gmail.com> References: <49CF57B4.4020505@gmail.com> <5b8d13220903290422q30c226c2m537fe63e7b26bbaa@mail.gmail.com> <49CF6061.5@gmail.com> <8D74ED18-0E87-4C4F-A006-8A335DB4C976@math.washington.edu> <49CF6878.4080707@gmail.com> <49CF7614.6060900@student.matnat.uio.no> <49CF7D41.9010905@gmail.com> Message-ID: <5b8d13220903290708h611a05e1mb1f85db5065f579f@mail.gmail.com> On Sun, Mar 29, 2009 at 10:53 PM, Tiago Pereira wrote: > > Now, just another question... How do I do the reverse? Ie., convert from > **float to numpy? I looked in previous emails and I saw something using > np.memcpy. Adapting for 2d it would be something like this (assuming a > float **res): > > cdef np.ndarray[DTYPE_t,ndim=2]result = np.zeros((N,N),dtype=DTYPE) > if data != NULL: np.memcpy(result.data,res,N*N*sizeof(float)) You have to think about how things are layered in memory, otherwise you will keep making those mistakes. That's the fun of C :) A float** is like this: a[0] -> address of first row: float* row0 = a[0], then row0, row0+1, etc... give you the items of the first row. ... To create a numpy array from those kind of arrays, you have to memcpy every row: float *dest; for i in range(nrows): mempcy(dest + i * ncols, a[i], number of bytes in a[i]) and then use dest to create a numpy array. In C, using ragged arrays is generally not a good idea - of course, if your externaly library uses it, you have no choice, but for your own code, you are often better using plain float*. cheers, David From dagss at student.matnat.uio.no Sun Mar 29 20:41:56 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 29 Mar 2009 20:41:56 +0200 Subject: [Cython] 0.11.1 release timeframe? In-Reply-To: References: <49CF526B.6070305@student.matnat.uio.no> Message-ID: <49CFC0F4.4070500@student.matnat.uio.no> Robert Bradshaw wrote: >> and moving a lot of those tickets over to 0.11.2 ? >> (as they are not regressions). We've also had the for loops update >> which >> it would be nice to release sooner rather than later. > > My only hesitance about loops is that it changes (for the better) the > semantics of some loops. Hopefully people haven't been relying on the > old behavior, but is that too much for a 0.11.x release? Granted, it > would only affect people abusing wrong corner cases. I think this shouldn't affect the timeframe though. We always have the option of releasing cython-devel as 0.12 because the semantics has changed. (If you do so, rename the milestones rather than move the tickets :-) ). (I think the other option, backing out the loop changes and move them to cython-unstable, a waste of valuable developer time in this case). -- Dag Sverre From m.gysel at gmail.com Mon Mar 30 00:18:51 2009 From: m.gysel at gmail.com (Martin Gysel) Date: Mon, 30 Mar 2009 00:18:51 +0200 Subject: [Cython] code documentation tool Message-ID: <49CFF3CB.2000408@gmail.com> Hi Can anybody recommend a code documentation tool which works good with cython. I tried doxygen but I'm rather disappointed. I see the cython project uses sphinx for the doc. But is sphinx the best tool to generate api docs? Is there something which can analyze cython code and do what doxygen does with c or java? /martin From cournape at gmail.com Mon Mar 30 04:34:50 2009 From: cournape at gmail.com (David Cournapeau) Date: Mon, 30 Mar 2009 11:34:50 +0900 Subject: [Cython] code documentation tool In-Reply-To: <49CFF3CB.2000408@gmail.com> References: <49CFF3CB.2000408@gmail.com> Message-ID: <5b8d13220903291934w69a8cac9y1af6e1722eb4ddca@mail.gmail.com> On Mon, Mar 30, 2009 at 7:18 AM, Martin Gysel wrote: > Hi > > Can anybody recommend a code documentation tool which works good with > cython. > I tried doxygen but I'm rather disappointed. I see the cython project > uses sphinx for the doc. But is sphinx the best tool to generate api > docs? It is the best I know for python, at least. > Is there something which can analyze cython code and do what > doxygen does with c or java? Sphinx API doc works by introspection, so it does not matter so much whether the underlying code is in python or C (cython is "just" a tool to generate C in that case). You can see a simple example here I made for audiolab: http://www.ar.media.kyoto-u.ac.jp/members/david/softwares/audiolab/sphinx/fullapi.html And the corresponding source file: http://github.com/cournape/audiolab/blob/4afd4853dbb273305ab8468e860bf2e651883c20/scikits/audiolab/pysndfile/_sndfile.pyx David From sturla at molden.no Mon Mar 30 05:35:43 2009 From: sturla at molden.no (Sturla Molden) Date: Mon, 30 Mar 2009 05:35:43 +0200 Subject: [Cython] Passing numpy arrays to C and vice-versa (again...) In-Reply-To: References: <49CF57B4.4020505@gmail.com> <5b8d13220903290422q30c226c2m537fe63e7b26bbaa@mail.gmail.com> <49CF6061.5@gmail.com> <8D74ED18-0E87-4C4F-A006-8A335DB4C976@math.washington.edu> <49CF6878.4080707@gmail.com> <49CF7614.6060900@student.matnat.uio.no> <49CF7D41.9010905@gmail.com> Message-ID: <49D03E0F.3020508@molden.no> Dag Sverre Seljebotn wrote: > > Well, there's no way a float** will fit within the NumPy memory model, so > the memory must be copied somehow. Using a memcpy per row will likely be > faster but is fundamentally the same thing. There is, if a contiguous block is allocated instead of individual rows. double **array = malloc(nrows * sizeof(double*)); double *array_buffer = malloc(nrows * ncols * sizeof(double)); for (int i=0; i References: <49CF57B4.4020505@gmail.com> <5b8d13220903290422q30c226c2m537fe63e7b26bbaa@mail.gmail.com> <49CF6061.5@gmail.com> <8D74ED18-0E87-4C4F-A006-8A335DB4C976@math.washington.edu> <49CF6878.4080707@gmail.com> <49CF7614.6060900@student.matnat.uio.no> <49CF7D41.9010905@gmail.com> <49D03E0F.3020508@molden.no> Message-ID: <49D042EA.2090900@molden.no> Sturla Molden wrote: > Dag Sverre Seljebotn wrote: > >> Well, there's no way a float** will fit within the NumPy memory model, so >> the memory must be copied somehow. Using a memcpy per row will likely be >> faster but is fundamentally the same thing. >> > There is, if a contiguous block is allocated instead of individual rows. > > double **array = malloc(nrows * sizeof(double*)); > double *array_buffer = malloc(nrows * ncols * sizeof(double)); > for (int i=0; i array[i] = array_buffer + i * ncols; > Actually, one could improve this even further, allocating everything into one block: void *array_buffer = malloc(nrows * ncols * sizeof(double) + nrows * sizeof(double*)); double **array = (double **)((char *)array_buffer + nrows * ncols * sizeof(double)); for (int i=0; i References: <49CF57B4.4020505@gmail.com> <5b8d13220903290422q30c226c2m537fe63e7b26bbaa@mail.gmail.com> <49CF6061.5@gmail.com> <8D74ED18-0E87-4C4F-A006-8A335DB4C976@math.washington.edu> <49CF6878.4080707@gmail.com> <49CF7614.6060900@student.matnat.uio.no> <49CF7D41.9010905@gmail.com> <5b8d13220903290708h611a05e1mb1f85db5065f579f@mail.gmail.com> Message-ID: <49D04E69.5040303@gmail.com> David Cournapeau wrote: > > float *dest; > for i in range(nrows): > mempcy(dest + i * ncols, a[i], number of bytes in a[i]) > > and then use dest to create a numpy array. Thanks guys for all your help! I've managed to get it working. I now use the following functions to convert from float ** to 2D numpy and vice-versa: cdef inline float **npy2c_float(np.ndarray a): cdef int m = a.shape[0] cdef int n = a.shape[1] cdef int i cdef float **data data = malloc(m*sizeof(float*)) for i in range(m): data[i] = &(a.data)[i*n] return data cdef inline np.ndarray c2npy_float(float **a, int n, int m): cdef np.ndarray[DTYPE_t,ndim=2]result = np.zeros((m,n),dtype=DTYPE) cdef float *dest cdef int i dest = malloc(m*n*sizeof(float*)) for i in range(m): memcpy(dest + i*n,a[i],m*sizeof(float*)) free(a[i]) memcpy(result.data,dest,m*n*sizeof(float*)) free(dest) free(a) return result This of course assumes a is contiguous. > In C, using ragged arrays is generally not a good idea - of course, if > your externaly library uses it, you have no choice, but for your own > code, you are often better using plain float*. I see. And it seems that float* is also more flexible for n-d arrays. But, another dumb C question, if you have an array as float*, can you use it (in C) as array[i][j] or you need to refer to it as array[i+ncols*j]? Thanks again for all the help and for making the great piece of software that Cython is! Cheers, Tiago From cournape at gmail.com Mon Mar 30 09:01:08 2009 From: cournape at gmail.com (David Cournapeau) Date: Mon, 30 Mar 2009 16:01:08 +0900 Subject: [Cython] Passing numpy arrays to C and vice-versa (again...) In-Reply-To: <49D04E69.5040303@gmail.com> References: <49CF57B4.4020505@gmail.com> <5b8d13220903290422q30c226c2m537fe63e7b26bbaa@mail.gmail.com> <49CF6061.5@gmail.com> <8D74ED18-0E87-4C4F-A006-8A335DB4C976@math.washington.edu> <49CF6878.4080707@gmail.com> <49CF7614.6060900@student.matnat.uio.no> <49CF7D41.9010905@gmail.com> <5b8d13220903290708h611a05e1mb1f85db5065f579f@mail.gmail.com> <49D04E69.5040303@gmail.com> Message-ID: <5b8d13220903300001q5b4724d6p19efef3eb9779ad3@mail.gmail.com> On Mon, Mar 30, 2009 at 1:45 PM, Tiago Pereira wrote: > > I see. And it seems that float* is also more flexible for n-d arrays. Yes. That's maybe the strongest argument for float*, actually. > But, another dumb C question, if you have an array as float*, can you > use it (in C) as array[i][j] or you need to refer to it as array[i+ncols*j]? Yes. Just create the corresponding array to get the address: float **m; ... allocate m for nrows pointers... for i in range(nrows): m[i] = array + i * ncols cheers, David From cournape at gmail.com Mon Mar 30 10:59:15 2009 From: cournape at gmail.com (David Cournapeau) Date: Mon, 30 Mar 2009 17:59:15 +0900 Subject: [Cython] Support for arbitrary rank numpy array Message-ID: <5b8d13220903300159p14fe478eg6895a28d5350241@mail.gmail.com> Hi, I saw on the wiki that support for iterating over arbitrary rank array is considered in cython. Has there been any work on it ? thanks, David From dagss at student.matnat.uio.no Mon Mar 30 11:59:18 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 30 Mar 2009 11:59:18 +0200 (CEST) Subject: [Cython] Support for arbitrary rank numpy array In-Reply-To: <5b8d13220903300159p14fe478eg6895a28d5350241@mail.gmail.com> References: <5b8d13220903300159p14fe478eg6895a28d5350241@mail.gmail.com> Message-ID: David Cournapeau wrote: > Hi, > > I saw on the wiki that support for iterating over arbitrary rank array > is considered in cython. Has there been any work on it ? No, unfortunately. There's no lack of good ideas for numerics in Cython, but there's a certain lack of time to implement them :-( It could be possible to use the NumPy C API arbitrary rank iterators though. Not as nice looking but should do the job. Patches for getting that into numpy.pxd is welcome. Dag Sverre From cournape at gmail.com Mon Mar 30 12:19:22 2009 From: cournape at gmail.com (David Cournapeau) Date: Mon, 30 Mar 2009 19:19:22 +0900 Subject: [Cython] Support for arbitrary rank numpy array In-Reply-To: References: <5b8d13220903300159p14fe478eg6895a28d5350241@mail.gmail.com> Message-ID: <5b8d13220903300319g47589375ua7350a2bda36a22e@mail.gmail.com> On Mon, Mar 30, 2009 at 6:59 PM, Dag Sverre Seljebotn wrote: > David Cournapeau wrote: >> Hi, >> >> I saw on the wiki that support for iterating over arbitrary rank array >> is considered in cython. Has there been any work on it ? > > No, unfortunately. There's no lack of good ideas for numerics in Cython, > but there's a certain lack of time to implement them :-( That's a well known problem that even python has not been able to solve :) > Not as nice looking but should do the job. Patches for getting > that into numpy.pxd is welcome. General support in cython is beyond my capability, but numpy.pxd is certainly possible, cheers, David From jorgen at devsoft.no Mon Mar 30 15:46:41 2009 From: jorgen at devsoft.no (=?UTF-8?B?IkrDuHJnZW4gUC4gVGplcm7DuCI=?=) Date: Mon, 30 Mar 2009 15:46:41 +0200 Subject: [Cython] Mapping python objects -> C objects Message-ID: <49D0CD41.8010703@devsoft.no> Okay, the subject isn't very clear, but I found it hard to explain it in a concise and understandable fashion in a single sentence. What I have are a set of classes that "wrap" access to C structs, and are created by handing them a pointer. (Either they take ownership of it, meaning they free it in __dealloc__, or they just borrow it, but that's beside the point) What I want a way is to make sure that one pointer = one specific python object, so that trying to instantiate it again would return the same object. What's a good way to do this? A module-level dict of pointer -> object, "memoizing" the python objects? Would access to such a dict be thread-safe, or do I need some kind of explicit locking? -- Kindest regards, J?rgen P. Tjern?. From stefan_ml at behnel.de Mon Mar 30 16:14:49 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 30 Mar 2009 16:14:49 +0200 (CEST) Subject: [Cython] Mapping python objects -> C objects In-Reply-To: <49D0CD41.8010703@devsoft.no> References: <49D0CD41.8010703@devsoft.no> Message-ID: "J??rgen P. Tjern??" wrote: > What I want a way is to make sure that one pointer = one specific python > object, so that trying to instantiate it again would return the same > object. If you are lucky, the struct has a field reserved for the user that you can use to keep pointers to the corresponding Python object. If you are a bit less lucky, and you have control over how the structs are malloced, you can create your own struct instead, that adds such a back pointer at the end. Everything else is rather slow and messy, but please ask back if you think there's no way to achieve the above. Stefan From jorgen at devsoft.no Mon Mar 30 17:28:34 2009 From: jorgen at devsoft.no (=?UTF-8?B?IkrDuHJnZW4gUC4gVGplcm7DuCI=?=) Date: Mon, 30 Mar 2009 17:28:34 +0200 Subject: [Cython] Storing a list of C pointers Message-ID: <49D0E522.7070606@devsoft.no> Hi! I'd like to store (not for the world to see, just my code) a list of pointers to C structs. When I do: cdef list lists and then instance.lists = [some_list] It says: Cannot convert 'playlist *' to Python object Is there any way to have a ?vector?-esque datastructure in Cython? (dynamically sized array) -- Kindest regards, J?rgen P. Tjern?. From sturla at molden.no Mon Mar 30 17:52:19 2009 From: sturla at molden.no (Sturla Molden) Date: Mon, 30 Mar 2009 17:52:19 +0200 Subject: [Cython] Storing a list of C pointers In-Reply-To: <49D0E522.7070606@devsoft.no> References: <49D0E522.7070606@devsoft.no> Message-ID: <49D0EAB3.2050702@molden.no> J?rgen P. Tjern? wrote: > > It says: > Cannot convert 'playlist *' to Python object Then cast the pointer to a C long (or Py_intptr_t), and create a Python int from that. cdef playlist *list instance.lists = [ int( list) ] list = ( instance.lists[0]) S.M. From stefan_ml at behnel.de Mon Mar 30 09:28:47 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 30 Mar 2009 09:28:47 +0200 Subject: [Cython] 0.11.1 release timeframe? In-Reply-To: <49CFC0F4.4070500@student.matnat.uio.no> References: <49CF526B.6070305@student.matnat.uio.no> <49CFC0F4.4070500@student.matnat.uio.no> Message-ID: <49D074AF.10602@behnel.de> Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >>> and moving a lot of those tickets over to 0.11.2 ? >>> (as they are not regressions). We've also had the for loops update >>> which it would be nice to release sooner rather than later. >> >> My only hesitance about loops is that it changes (for the better) the >> semantics of some loops. Hopefully people haven't been relying on the >> old behavior, but is that too much for a 0.11.x release? Granted, it >> would only affect people abusing wrong corner cases. I'm not sure either. We might think about moving the change to 0.12 entirely, but I actually consider it a bug that gets fixed. The sooner, the better. Fixing it right in 0.11.1 would make that clear. Stefan From stefan_ml at behnel.de Mon Mar 30 09:55:25 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 30 Mar 2009 09:55:25 +0200 Subject: [Cython] code documentation tool In-Reply-To: <49CFF3CB.2000408@gmail.com> References: <49CFF3CB.2000408@gmail.com> Message-ID: <49D07AED.7020905@behnel.de> Martin Gysel wrote: > Can anybody recommend a code documentation tool which works good with > cython. > I tried doxygen but I'm rather disappointed. I see the cython project > uses sphinx for the doc. But is sphinx the best tool to generate api > docs? Sphinx is good in generating documentation. It's not that satisfying for generating API docs from sources or through module introspection. > Is there something which can analyze cython code and do what > doxygen does with c or java? Nothing I know of. Epydoc can read the embedded signatures, as long as you stick with Python 2.x syntax and avoid fancy default values. It will fail hopelessly (reg-exps, that is) for type annotations in whatever form. I considered fixing it several times, but I guess when I start investing this time, I'd rather fix up Sphinx' API doc generation instead. It just looks better. Stefan From robertwb at math.washington.edu Mon Mar 30 21:15:43 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 30 Mar 2009 12:15:43 -0700 Subject: [Cython] Passing numpy arrays to C and vice-versa (again...) In-Reply-To: <49D04E69.5040303@gmail.com> References: <49CF57B4.4020505@gmail.com> <5b8d13220903290422q30c226c2m537fe63e7b26bbaa@mail.gmail.com> <49CF6061.5@gmail.com> <8D74ED18-0E87-4C4F-A006-8A335DB4C976@math.washington.edu> <49CF6878.4080707@gmail.com> <49CF7614.6060900@student.matnat.uio.no> <49CF7D41.9010905@gmail.com> <5b8d13220903290708h611a05e1mb1f85db5065f579f@mail.gmail.com> <49D04E69.5040303@gmail.com> Message-ID: On Mar 29, 2009, at 9:45 PM, Tiago Pereira wrote: > David Cournapeau wrote: >> >> float *dest; >> for i in range(nrows): >> mempcy(dest + i * ncols, a[i], number of bytes in a[i]) >> >> and then use dest to create a numpy array. > > Thanks guys for all your help! I've managed to get it working. I > now use > the following functions to convert from float ** to 2D numpy and > vice-versa: > > cdef inline float **npy2c_float(np.ndarray a): > cdef int m = a.shape[0] > cdef int n = a.shape[1] > cdef int i > cdef float **data > data = malloc(m*sizeof(float*)) > for i in range(m): > data[i] = &(a.data)[i*n] > return data > > cdef inline np.ndarray c2npy_float(float **a, int n, int m): > cdef np.ndarray[DTYPE_t,ndim=2]result = np.zeros > ((m,n),dtype=DTYPE) > cdef float *dest > cdef int i > dest = malloc(m*n*sizeof(float*)) > for i in range(m): > memcpy(dest + i*n,a[i],m*sizeof(float*)) > free(a[i]) > memcpy(result.data,dest,m*n*sizeof(float*)) > free(dest) > free(a) > return result > > This of course assumes a is contiguous. BTW, you can make sure the array is contiguous by checking that a.strides == ( n*sizeof(float), sizeof(float) ) >> In C, using ragged arrays is generally not a good idea - of >> course, if >> your externaly library uses it, you have no choice, but for your own >> code, you are often better using plain float*. > > I see. And it seems that float* is also more flexible for n-d arrays. > But, another dumb C question, if you have an array as float*, can you > use it (in C) as array[i][j] or you need to refer to it as array[i > +ncols*j]? No, you can't do array[i][j] because C wouldn't have a way of getting ncols. float** is literally an list of pointers. > Thanks again for all the help and for making the great piece of > software > that Cython is! On behalf of the many people who've contributed, you're welcome. Glad you're finding it as useful as we have :) - Robert From dagss at student.matnat.uio.no Mon Mar 30 21:39:25 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 30 Mar 2009 21:39:25 +0200 Subject: [Cython] Passing numpy arrays to C and vice-versa (again...) In-Reply-To: References: <49CF57B4.4020505@gmail.com> <5b8d13220903290422q30c226c2m537fe63e7b26bbaa@mail.gmail.com> <49CF6061.5@gmail.com> <8D74ED18-0E87-4C4F-A006-8A335DB4C976@math.washington.edu> <49CF6878.4080707@gmail.com> <49CF7614.6060900@student.matnat.uio.no> <49CF7D41.9010905@gmail.com> <5b8d13220903290708h611a05e1mb1f85db5065f579f@mail.gmail.com> <49D04E69.5040303@gmail.com> Message-ID: <49D11FED.8020200@student.matnat.uio.no> Robert Bradshaw wrote: > On Mar 29, 2009, at 9:45 PM, Tiago Pereira wrote: > >> David Cournapeau wrote: >>> float *dest; >>> for i in range(nrows): >>> mempcy(dest + i * ncols, a[i], number of bytes in a[i]) >>> >>> and then use dest to create a numpy array. >> Thanks guys for all your help! I've managed to get it working. I >> now use >> the following functions to convert from float ** to 2D numpy and >> vice-versa: >> >> cdef inline float **npy2c_float(np.ndarray a): >> cdef int m = a.shape[0] >> cdef int n = a.shape[1] >> cdef int i >> cdef float **data >> data = malloc(m*sizeof(float*)) >> for i in range(m): >> data[i] = &(a.data)[i*n] >> return data >> >> cdef inline np.ndarray c2npy_float(float **a, int n, int m): >> cdef np.ndarray[DTYPE_t,ndim=2]result = np.zeros >> ((m,n),dtype=DTYPE) >> cdef float *dest >> cdef int i >> dest = malloc(m*n*sizeof(float*)) >> for i in range(m): >> memcpy(dest + i*n,a[i],m*sizeof(float*)) >> free(a[i]) >> memcpy(result.data,dest,m*n*sizeof(float*)) >> free(dest) >> free(a) >> return result >> >> This of course assumes a is contiguous. > > BTW, you can make sure the array is contiguous by checking that > a.strides == ( n*sizeof(float), sizeof(float) ) Actually, no: a.strides is what amounts to a size_t* since a is typed to ndarray. A long-standing usability problem. So, check a.strides[0] == n*sizeof(float), a.strides[1] == sizeof(float). The usual way is to check a.flags['C_CONTIGUOUS'], although this tends to be a bit too conservative sometimes (arrays which are contiguous are not flagged as such; that's a problem with NumPy). -- Dag Sverre From robertwb at math.washington.edu Mon Mar 30 21:31:21 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 30 Mar 2009 12:31:21 -0700 Subject: [Cython] 0.11.1 release timeframe? In-Reply-To: <49D074AF.10602@behnel.de> References: <49CF526B.6070305@student.matnat.uio.no> <49CFC0F4.4070500@student.matnat.uio.no> <49D074AF.10602@behnel.de> Message-ID: On Mar 30, 2009, at 12:28 AM, Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> Robert Bradshaw wrote: >>>> and moving a lot of those tickets over to 0.11.2 ? >>>> (as they are not regressions). We've also had the for loops update >>>> which it would be nice to release sooner rather than later. >>> >>> My only hesitance about loops is that it changes (for the better) >>> the >>> semantics of some loops. Hopefully people haven't been relying on >>> the >>> old behavior, but is that too much for a 0.11.x release? Granted, it >>> would only affect people abusing wrong corner cases. > > I'm not sure either. We might think about moving the change to 0.12 > entirely, but I actually consider it a bug that gets fixed. The > sooner, the > better. Fixing it right in 0.11.1 would make that clear. I consider it a bug too, the only hesitance is the philosophy that any code that works with 0.11 should also work unchanged with 0.11.1. Given that it's already in the devel branch, I'll move my changes fixes there too. Is there anything else in -unstable that's deemed stable enough to be released? Also, if there's anything inhttp:// trac.cython.org/cython_trac/milestone/0.11.1 that anyone thinks should really go in this next release (if so, mark it as critical or blocker). - Robert From stefan_ml at behnel.de Mon Mar 30 21:46:57 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 30 Mar 2009 21:46:57 +0200 Subject: [Cython] 0.11.1 release timeframe? In-Reply-To: References: <49CF526B.6070305@student.matnat.uio.no> <49CFC0F4.4070500@student.matnat.uio.no> <49D074AF.10602@behnel.de> Message-ID: <49D121B1.4030902@behnel.de> Robert Bradshaw wrote: > I consider it a bug too, the only hesitance is the philosophy that > any code that works with 0.11 should also work unchanged with 0.11.1. > Given that it's already in the devel branch, I'll move my changes > fixes there too. Is there anything else in -unstable that's deemed > stable enough to be released? Nothing for 0.11.1 from my side. Maybe 0.11.2, after some more testing. Or just release 0.12 in a couple of weeks. :) > Also, if there's anything inhttp:// > trac.cython.org/cython_trac/milestone/0.11.1 that anyone thinks > should really go in this next release (if so, mark it as critical or > blocker). No, nothing that looks critical to me. Stefan From dalcinl at gmail.com Mon Mar 30 22:30:19 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 30 Mar 2009 17:30:19 -0300 Subject: [Cython] code documentation tool In-Reply-To: <49D07AED.7020905@behnel.de> References: <49CFF3CB.2000408@gmail.com> <49D07AED.7020905@behnel.de> Message-ID: On Mon, Mar 30, 2009 at 4:55 AM, Stefan Behnel wrote: > > Martin Gysel wrote: >> Can anybody recommend a code documentation tool which works good with >> cython. > > Sphinx is good in generating documentation. It's not that satisfying for > generating API docs from sources or through module introspection. > Indeed. >> Is there something which can analyze cython code and do what >> doxygen does with c or java? > > Nothing I know of. Epydoc can read the embedded signatures, as long as you > stick with Python 2.x syntax and avoid fancy default values. It will fail > hopelessly (reg-exps, that is) for type annotations in whatever form. > However, take a look at the script in the below (works as a cmd line replacement for epydoc) Of course you have to remove stuff at the very beginning and at the end (the 'epydoc.cfg' rc file stuff) ... Moreover, this script also monkeypatch the issue of Epydoc trying to generate GIF files in favor of PNG, as Linux distros (Fedora, in my case) disable that format for the famous patents problems http://code.google.com/p/mpi4py/source/browse/trunk/conf/epydocify.py Wait a minute... I've pushed something to Cython a long time ago (Tools/cython-epydoc.py)... But it is likely outdated... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Mon Mar 30 22:40:55 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 30 Mar 2009 17:40:55 -0300 Subject: [Cython] 0.11.1 release timeframe? In-Reply-To: References: <49CF526B.6070305@student.matnat.uio.no> <49CFC0F4.4070500@student.matnat.uio.no> <49D074AF.10602@behnel.de> Message-ID: Well, Robert, I you could add those changes we discussed in private mail for enabling Cython run from the ZIP file, I would be great... This can give anyone the freedom of easily "downgrade/upgrade" Cython just for the purpose of cythonizing some codebase that requires older/newer versions of Cython of the one installed in the system. The only issue that remains problem of handling "resource" files like these *.pxd in "Include/" ... On Mon, Mar 30, 2009 at 4:31 PM, Robert Bradshaw wrote: > On Mar 30, 2009, at 12:28 AM, Stefan Behnel wrote: > >> Dag Sverre Seljebotn wrote: >>> Robert Bradshaw wrote: >>>>> and moving a lot of those tickets over to 0.11.2 ? >>>>> (as they are not regressions). We've also had the for loops update >>>>> which it would be nice to release sooner rather than later. >>>> >>>> My only hesitance about loops is that it changes (for the better) >>>> the >>>> semantics of some loops. Hopefully people haven't been relying on >>>> the >>>> old behavior, but is that too much for a 0.11.x release? Granted, it >>>> would only affect people abusing wrong corner cases. >> >> I'm not sure either. We might think about moving the change to 0.12 >> entirely, but I actually consider it a bug that gets fixed. The >> sooner, the >> better. Fixing it right in 0.11.1 would make that clear. > > > I consider it a bug too, the only hesitance is the philosophy that > any code that works with 0.11 should also work unchanged with 0.11.1. > Given that it's already in the devel branch, I'll move my changes > fixes there too. Is there anything else in -unstable that's deemed > stable enough to be released? Also, if there's anything inhttp:// > trac.cython.org/cython_trac/milestone/0.11.1 that anyone thinks > should really go in this next release (if so, mark it as critical or > blocker). > > - Robert > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Mon Mar 30 23:21:16 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 30 Mar 2009 23:21:16 +0200 Subject: [Cython] Package names Message-ID: <49D137CC.8070707@student.matnat.uio.no> Since Stefan talked about aim at getting 0.12.x into Python stdlib, I think we should discuss package names. 1) Currently there's "Cython" for the package with the compiler and "cython" for the "shadow" module importable from pure Python. I feel this is both confusing and error-prone. A solution is to rename the former to e.g. "cythoncompiler". (However, unless more renaming is done, one would have "cythoncompiler.compiler". Perhaps "cythoncompiler.core"). 2) Should all the package names be converted to lowercase? 3) Should one perhaps distribute Cython under two seperate package names, so that a standalone version can coexist with the one in stdlib? This would facilitate easy installation and usage of non-stdlib Cython releases. Can't come up with any good names though. -- Dag Sverre From robertwb at math.washington.edu Mon Mar 30 23:50:22 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 30 Mar 2009 14:50:22 -0700 Subject: [Cython] code documentation tool In-Reply-To: References: <49CFF3CB.2000408@gmail.com> <49D07AED.7020905@behnel.de> Message-ID: On Mar 30, 2009, at 1:30 PM, Lisandro Dalcin wrote: > On Mon, Mar 30, 2009 at 4:55 AM, Stefan Behnel > wrote: >> >> Martin Gysel wrote: >>> Can anybody recommend a code documentation tool which works good >>> with >>> cython. >> >> Sphinx is good in generating documentation. It's not that >> satisfying for >> generating API docs from sources or through module introspection. >> > > Indeed. > >>> Is there something which can analyze cython code and do what >>> doxygen does with c or java? >> >> Nothing I know of. Epydoc can read the embedded signatures, as >> long as you >> stick with Python 2.x syntax and avoid fancy default values. It >> will fail >> hopelessly (reg-exps, that is) for type annotations in whatever form. >> > > However, take a look at the script in the below (works as a cmd line > replacement for epydoc) > Of course you have to remove stuff at the very beginning and at the > end (the 'epydoc.cfg' rc file stuff) ... > Moreover, this script also monkeypatch the issue of Epydoc trying to > generate GIF files in favor of PNG, as Linux distros (Fedora, in my > case) disable that format for the famous patents problems > > http://code.google.com/p/mpi4py/source/browse/trunk/conf/epydocify.py > > Wait a minute... I've pushed something to Cython a long time ago > (Tools/cython-epydoc.py)... But it is likely outdated... We got the cython patch in for embedding epydoc signature data, right? - Robert From robertwb at math.washington.edu Mon Mar 30 23:53:30 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 30 Mar 2009 14:53:30 -0700 Subject: [Cython] 0.11.1 release timeframe? In-Reply-To: References: <49CF526B.6070305@student.matnat.uio.no> <49CFC0F4.4070500@student.matnat.uio.no> <49D074AF.10602@behnel.de> Message-ID: <99C9207D-413E-4B53-8ED3-76D50F3EA47B@math.washington.edu> On Mar 30, 2009, at 1:40 PM, Lisandro Dalcin wrote: > Well, Robert, I you could add those changes we discussed in private > mail for enabling Cython run from the ZIP file, I would be great... > This can give anyone the freedom of easily "downgrade/upgrade" Cython > just for the purpose of cythonizing some codebase that requires > older/newer versions of Cython of the one installed in the system. The > only issue that remains problem of handling "resource" files like > these *.pxd in "Include/" ... Sure. I pushed my changes to devel, and the pickling/unpickling stuff looks safe enough for me if you want to push it. > > > On Mon, Mar 30, 2009 at 4:31 PM, Robert Bradshaw > wrote: >> On Mar 30, 2009, at 12:28 AM, Stefan Behnel wrote: >> >>> Dag Sverre Seljebotn wrote: >>>> Robert Bradshaw wrote: >>>>>> and moving a lot of those tickets over to 0.11.2 ? >>>>>> (as they are not regressions). We've also had the for loops >>>>>> update >>>>>> which it would be nice to release sooner rather than later. >>>>> >>>>> My only hesitance about loops is that it changes (for the better) >>>>> the >>>>> semantics of some loops. Hopefully people haven't been relying on >>>>> the >>>>> old behavior, but is that too much for a 0.11.x release? >>>>> Granted, it >>>>> would only affect people abusing wrong corner cases. >>> >>> I'm not sure either. We might think about moving the change to 0.12 >>> entirely, but I actually consider it a bug that gets fixed. The >>> sooner, the >>> better. Fixing it right in 0.11.1 would make that clear. >> >> >> I consider it a bug too, the only hesitance is the philosophy that >> any code that works with 0.11 should also work unchanged with 0.11.1. >> Given that it's already in the devel branch, I'll move my changes >> fixes there too. Is there anything else in -unstable that's deemed >> stable enough to be released? Also, if there's anything inhttp:// >> trac.cython.org/cython_trac/milestone/0.11.1 that anyone thinks >> should really go in this next release (if so, mark it as critical or >> blocker). >> >> - Robert >> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From dalcinl at gmail.com Tue Mar 31 00:34:04 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 30 Mar 2009 19:34:04 -0300 Subject: [Cython] code documentation tool In-Reply-To: References: <49CFF3CB.2000408@gmail.com> <49D07AED.7020905@behnel.de> Message-ID: On Mon, Mar 30, 2009 at 6:50 PM, Robert Bradshaw wrote: > On Mar 30, 2009, at 1:30 PM, Lisandro Dalcin wrote: > > We got the cython patch in for embedding epydoc signature data, right? > Not sure what patch are you talking about... The auto-generation of signature is working from long ago, but it adds types to the func args... but epydoc cannot parse it... so the monkeypatching I'm talking about... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Tue Mar 31 01:00:41 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 30 Mar 2009 20:00:41 -0300 Subject: [Cython] Package names In-Reply-To: <49D137CC.8070707@student.matnat.uio.no> References: <49D137CC.8070707@student.matnat.uio.no> Message-ID: On Mon, Mar 30, 2009 at 6:21 PM, Dag Sverre Seljebotn wrote: > Since Stefan talked about aim at getting 0.12.x into Python stdlib, I > think we should discuss package names. > > 1) Currently there's "Cython" for the package with the compiler and > "cython" for the "shadow" module importable from pure Python. I feel > this is both confusing and error-prone. A solution is to rename the > former to e.g. "cythoncompiler". (However, unless more renaming is done, > one would have "cythoncompiler.compiler". Perhaps "cythoncompiler.core"). > I would just do the renaming Cython -> Core > 2) Should all the package names be converted to lowercase? > Perhaps we should follow PEP 8 ? Package and Module Names Modules should have short, all-lowercase names. Underscores can be used in the module name if it improves readability. Python packages should also have short, all-lowercase names, although the use of underscores is discouraged. Since module names are mapped to file names, and some file systems are case insensitive and truncate long names, it is important that module names be chosen to be fairly short -- this won't be a problem on Unix, but it may be a problem when the code is transported to older Mac or Windows versions, or DOS. > 3) Should one perhaps distribute Cython under two seperate package > names, so that a standalone version can coexist with the one in stdlib? > This would facilitate easy installation and usage of non-stdlib Cython > releases. Can't come up with any good names though. > 1) cython 2) cython.core 3) cython.pyximport Regarding (1), I would make it have in the cython/__init__.py define __version__, and also have cython/run.py for cmd line work with -m switch, and make __init__py use that for newer Python versions that let you run packages. cython/__init__.py should also manage the shadow stuff Regarding (2), that's what I've previously suggested. Regarding (3), not a strong feeling. All this plus good support for making Cython work from a ZIP would make it really easy to switch to Cython's other than the system one... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Tue Mar 31 02:28:54 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 30 Mar 2009 17:28:54 -0700 Subject: [Cython] Package names In-Reply-To: References: <49D137CC.8070707@student.matnat.uio.no> Message-ID: On Mar 30, 2009, at 4:00 PM, Lisandro Dalcin wrote: > On Mon, Mar 30, 2009 at 6:21 PM, Dag Sverre Seljebotn > wrote: >> Since Stefan talked about aim at getting 0.12.x into Python stdlib, I >> think we should discuss package names. >> >> 1) Currently there's "Cython" for the package with the compiler and >> "cython" for the "shadow" module importable from pure Python. I feel >> this is both confusing and error-prone. A solution is to rename the >> former to e.g. "cythoncompiler". (However, unless more renaming is >> done, >> one would have "cythoncompiler.compiler". Perhaps >> "cythoncompiler.core"). >> > > I would just do the renaming > > Cython -> Core > > >> 2) Should all the package names be converted to lowercase? >> > > Perhaps we should follow PEP 8 ? > > Package and Module Names > > Modules should have short, all-lowercase names. Underscores > can be used > in the module name if it improves readability. Python > packages should > also have short, all-lowercase names, although the use of > underscores is > discouraged. > > Since module names are mapped to file names, and some file > systems are > case insensitive and truncate long names, it is important > that module > names be chosen to be fairly short -- this won't be a problem > on Unix, > but it may be a problem when the code is transported to older > Mac or > Windows versions, or DOS. Yes. >> 3) Should one perhaps distribute Cython under two seperate package >> names, so that a standalone version can coexist with the one in >> stdlib? >> This would facilitate easy installation and usage of non-stdlib >> Cython >> releases. Can't come up with any good names though. >> > > 1) cython > 2) cython.core > 3) cython.pyximport Meaning we'd have cython.core.nodes, etc. > > Regarding (1), I would make it have in the cython/__init__.py define > __version__, and also have cython/run.py for cmd line work with -m > switch, and make __init__py use that for newer Python versions that > let you run packages. cython/__init__.py should also manage the shadow > stuff +1 > Regarding (2), that's what I've previously suggested. > > Regarding (3), not a strong feeling. > > All this plus good support for making Cython work from a ZIP would > make it really easy to switch to Cython's other than the system one... I would like (3) to be easy too. I'm short on ideas for names too. - Robert From tiagomdp at gmail.com Tue Mar 31 02:40:46 2009 From: tiagomdp at gmail.com (Tiago Pereira) Date: Tue, 31 Mar 2009 11:40:46 +1100 Subject: [Cython] Passing numpy arrays to C and vice-versa (again...) In-Reply-To: <49D11FED.8020200@student.matnat.uio.no> References: <49CF57B4.4020505@gmail.com> <5b8d13220903290422q30c226c2m537fe63e7b26bbaa@mail.gmail.com> <49CF6061.5@gmail.com> <8D74ED18-0E87-4C4F-A006-8A335DB4C976@math.washington.edu> <49CF6878.4080707@gmail.com> <49CF7614.6060900@student.matnat.uio.no> <49CF7D41.9010905@gmail.com> <5b8d13220903290708h611a05e1mb1f85db5065f579f@mail.gmail.com> <49D04E69.5040303@gmail.com> <49D11FED.8020200@student.matnat.uio.no> Message-ID: <49D1668E.9080908@gmail.com> Dag Sverre Seljebotn wrote: > The usual way is to check a.flags['C_CONTIGUOUS'], although this tends > to be a bit too conservative sometimes (arrays which are contiguous are > not flagged as such; that's a problem with NumPy). You keep referring to a.flags['C_CONTIGUOUS'] but it seems to me it should be (a).flags["C_CONTIGUOUS"]. Using a.flags['C_CONTIGUOUS'] will give an invalid index error since a.flags is stored as an int (at least in my Cython 0.11 version). Btw, to enforce contiguous arrays what I've done was to place this condition in my Cython function: if not (a).flags["C_CONTIGUOUS"]: a = a.copy('C') Cheers, Tiago From nicdumz at gmail.com Tue Mar 31 04:15:16 2009 From: nicdumz at gmail.com (Nicolas Dumazet) Date: Tue, 31 Mar 2009 11:15:16 +0900 Subject: [Cython] Naive build system Message-ID: <1f077e770903301915l47cb09efsc53660b13c0ec6ed@mail.gmail.com> Hello! I opened a few days ago ticket 249 ( http://trac.cython.org/cython_trac/ticket/249 ) about the build system which basically recompiles everything on each build, even if nothing has changed. I submitted a patch, which adds a date check on the cython module files to avoid recompiling the cython parts of the module if the whole module hasnt been updated since the last generation of the .c files Today I'm looking for reviews and comments =) Thanks, -- Nicolas Dumazet ? NicDumZ [ n?k.d?ymz ] From dagss at student.matnat.uio.no Tue Mar 31 12:18:32 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 31 Mar 2009 12:18:32 +0200 Subject: [Cython] Passing numpy arrays to C and vice-versa (again...) In-Reply-To: <49D1668E.9080908@gmail.com> References: <49CF57B4.4020505@gmail.com> <5b8d13220903290422q30c226c2m537fe63e7b26bbaa@mail.gmail.com> <49CF6061.5@gmail.com> <8D74ED18-0E87-4C4F-A006-8A335DB4C976@math.washington.edu> <49CF6878.4080707@gmail.com> <49CF7614.6060900@student.matnat.uio.no> <49CF7D41.9010905@gmail.com> <5b8d13220903290708h611a05e1mb1f85db5065f579f@mail.gmail.com> <49D04E69.5040303@gmail.com> <49D11FED.8020200@student.matnat.uio.no> <49D1668E.9080908@gmail.com> Message-ID: <49D1EDF8.1000807@student.matnat.uio.no> Tiago Pereira wrote: > Dag Sverre Seljebotn wrote: > >> The usual way is to check a.flags['C_CONTIGUOUS'], although this tends >> to be a bit too conservative sometimes (arrays which are contiguous are >> not flagged as such; that's a problem with NumPy). > > You keep referring to a.flags['C_CONTIGUOUS'] but it seems to me it > should be (a).flags["C_CONTIGUOUS"]. Ah yes, I went into the exact same trap as I was saying Robert went into :-) -- Dag Sverre From sturla at molden.no Tue Mar 31 12:37:23 2009 From: sturla at molden.no (Sturla Molden) Date: Tue, 31 Mar 2009 12:37:23 +0200 Subject: [Cython] Passing numpy arrays to C and vice-versa (again...) In-Reply-To: <49D1EDF8.1000807@student.matnat.uio.no> References: <49CF57B4.4020505@gmail.com> <5b8d13220903290422q30c226c2m537fe63e7b26bbaa@mail.gmail.com> <49CF6061.5@gmail.com> <8D74ED18-0E87-4C4F-A006-8A335DB4C976@math.washington.edu> <49CF6878.4080707@gmail.com> <49CF7614.6060900@student.matnat.uio.no> <49CF7D41.9010905@gmail.com> <5b8d13220903290708h611a05e1mb1f85db5065f579f@mail.gmail.com> <49D04E69.5040303@gmail.com> <49D11FED.8020200@student.matnat.uio.no> <49D1668E.9080908@gmail.com> <49D1EDF8.1000807@student.matnat.uio.no> Message-ID: <49D1F263.8070801@molden.no> Dag Sverre Seljebotn wrote: > Tiago Pereira wrote: >> You keep referring to a.flags['C_CONTIGUOUS'] but it seems to me it >> should be (a).flags["C_CONTIGUOUS"]. > > Ah yes, I went into the exact same trap as I was saying Robert went into I believe a.flags & numpy.NPY_C_CONTIGUOUS avoids some overhead. Sturla Molden From cournape at gmail.com Tue Mar 31 12:58:41 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 31 Mar 2009 19:58:41 +0900 Subject: [Cython] Passing numpy arrays to C and vice-versa (again...) In-Reply-To: <49D11FED.8020200@student.matnat.uio.no> References: <49CF57B4.4020505@gmail.com> <49CF6061.5@gmail.com> <8D74ED18-0E87-4C4F-A006-8A335DB4C976@math.washington.edu> <49CF6878.4080707@gmail.com> <49CF7614.6060900@student.matnat.uio.no> <49CF7D41.9010905@gmail.com> <5b8d13220903290708h611a05e1mb1f85db5065f579f@mail.gmail.com> <49D04E69.5040303@gmail.com> <49D11FED.8020200@student.matnat.uio.no> Message-ID: <5b8d13220903310358s7964ca30mdf8a751dd493ce60@mail.gmail.com> On Tue, Mar 31, 2009 at 4:39 AM, Dag Sverre Seljebotn wrote: > The usual way is to check a.flags['C_CONTIGUOUS'], although this tends > to be a bit too conservative sometimes (arrays which are contiguous are > not flagged as such; that's a problem with NumPy). Do you mean that some arrays which are C contiguous are not flagged as such (known problem) or something else ? cheers, David From dagss at student.matnat.uio.no Tue Mar 31 13:18:43 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 31 Mar 2009 13:18:43 +0200 Subject: [Cython] Passing numpy arrays to C and vice-versa (again...) In-Reply-To: <5b8d13220903310358s7964ca30mdf8a751dd493ce60@mail.gmail.com> References: <49CF57B4.4020505@gmail.com> <49CF6061.5@gmail.com> <8D74ED18-0E87-4C4F-A006-8A335DB4C976@math.washington.edu> <49CF6878.4080707@gmail.com> <49CF7614.6060900@student.matnat.uio.no> <49CF7D41.9010905@gmail.com> <5b8d13220903290708h611a05e1mb1f85db5065f579f@mail.gmail.com> <49D04E69.5040303@gmail.com> <49D11FED.8020200@student.matnat.uio.no> <5b8d13220903310358s7964ca30mdf8a751dd493ce60@mail.gmail.com> Message-ID: <49D1FC13.1090009@student.matnat.uio.no> David Cournapeau wrote: > On Tue, Mar 31, 2009 at 4:39 AM, Dag Sverre Seljebotn > wrote: > >> The usual way is to check a.flags['C_CONTIGUOUS'], although this tends >> to be a bit too conservative sometimes (arrays which are contiguous are >> not flagged as such; that's a problem with NumPy). > > Do you mean that some arrays which are C contiguous are not flagged as > such (known problem) or something else ? I meant what you say, the known problem. -- Dag Sverre From dagss at student.matnat.uio.no Tue Mar 31 15:08:47 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 31 Mar 2009 15:08:47 +0200 Subject: [Cython] Package names In-Reply-To: References: <49D137CC.8070707@student.matnat.uio.no> Message-ID: <49D215DF.5060801@student.matnat.uio.no> Robert Bradshaw wrote: >>> 3) Should one perhaps distribute Cython under two seperate package >>> names, so that a standalone version can coexist with the one in >>> stdlib? >>> This would facilitate easy installation and usage of non-stdlib >>> Cython >>> releases. Can't come up with any good names though. > > I would like (3) to be easy too. I'm short on ideas for names too. How about having the version names in releases made from cython.org? I.e. the Cython version bundled with Python is "cython", while the newest one is "cython012", "cython013". Minor releases wouldn't be included. Alternatively, "cythonedge" or similar (as in bleeding edge). -- Dag Sverre From cournape at gmail.com Tue Mar 31 15:26:36 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 31 Mar 2009 22:26:36 +0900 Subject: [Cython] Passing numpy arrays to C and vice-versa (again...) In-Reply-To: <49D1FC13.1090009@student.matnat.uio.no> References: <49CF57B4.4020505@gmail.com> <49CF6878.4080707@gmail.com> <49CF7614.6060900@student.matnat.uio.no> <49CF7D41.9010905@gmail.com> <5b8d13220903290708h611a05e1mb1f85db5065f579f@mail.gmail.com> <49D04E69.5040303@gmail.com> <49D11FED.8020200@student.matnat.uio.no> <5b8d13220903310358s7964ca30mdf8a751dd493ce60@mail.gmail.com> <49D1FC13.1090009@student.matnat.uio.no> Message-ID: <5b8d13220903310626p30be4d93y41eb3956ef5e7457@mail.gmail.com> On Tue, Mar 31, 2009 at 8:18 PM, Dag Sverre Seljebotn wrote: > David Cournapeau wrote: >> On Tue, Mar 31, 2009 at 4:39 AM, Dag Sverre Seljebotn >> wrote: >> >>> The usual way is to check a.flags['C_CONTIGUOUS'], although this tends >>> to be a bit too conservative sometimes (arrays which are contiguous are >>> not flagged as such; that's a problem with NumPy). >> >> Do you mean that some arrays which are C contiguous are not flagged as >> such (known problem) or something else ? > > I meant what you say, the known problem. This bothered me several times too. I created a ticket on numpy trac to track this: http://projects.scipy.org/numpy/ticket/1075 David From dalcinl at gmail.com Tue Mar 31 16:16:47 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 31 Mar 2009 11:16:47 -0300 Subject: [Cython] Naive build system In-Reply-To: <1f077e770903301915l47cb09efsc53660b13c0ec6ed@mail.gmail.com> References: <1f077e770903301915l47cb09efsc53660b13c0ec6ed@mail.gmail.com> Message-ID: Nicolas, your fix seems good... however, it is still sub-optimal IMHO... setup.py is intended for more things than just building/installing packages. It also should also serve to get package metadata (blame distutils!, but these are the rules). For example, you should be able to do (took one of my projects as an example). $ python setup.py --fullname petsc4py-1.0.1 $ python setup.py --description PETSc for Python $ python setup.py --author Lisandro Dalcin $ python setup.py --url http://petsc4py.googlecode.com/ So setup.py should not do a lot of work. We have two ways to make this possible. 1) push your patch, but let future Cython releases zip/tarballs ship C sources. 2) implement this the right way, more or less in the spirit of numpy.distutils build_src command. Just to be clear, I'm +1 on your patch. However, I wanted to point out that this still is not the definitive solution. On Mon, Mar 30, 2009 at 11:15 PM, Nicolas Dumazet wrote: > Hello! > > I opened a few days ago ticket 249 ( > http://trac.cython.org/cython_trac/ticket/249 ) about the build system > which basically recompiles everything on each build, even if nothing > has changed. > > I submitted a patch, which adds a date check on the cython module > files to avoid recompiling the cython parts of the module if the whole > module hasnt been updated since the last generation of the .c files > > Today I'm looking for reviews and comments =) > > Thanks, > > -- > Nicolas Dumazet ? NicDumZ [ n?k.d?ymz ] > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From nicdumz at gmail.com Tue Mar 31 17:14:21 2009 From: nicdumz at gmail.com (Nicolas Dumazet) Date: Wed, 1 Apr 2009 00:14:21 +0900 Subject: [Cython] Naive build system In-Reply-To: References: <1f077e770903301915l47cb09efsc53660b13c0ec6ed@mail.gmail.com> Message-ID: <1f077e770903310814u14f1c074kf5771f70539a96e3@mail.gmail.com> 2009/3/31 Lisandro Dalcin : > So setup.py should not do a lot of work. We have two ways to make this possible. > > 1) push your patch, but let future Cython releases zip/tarballs ship C sources. > 2) implement this the right way, more or less in the spirit of > numpy.distutils build_src command. For other readers, source is here : http://projects.scipy.org/numpy/browser/trunk/numpy/distutils/command/build_src.py > Just to be clear, I'm +1 on your patch. However, I wanted to point out > that this still is not the definitive solution. And I couldn't agree more with you. I looked at ways to avoid the extra setup.py work, it seems there's no "easy way" here. I thought about extending the distutils build command to build the c files before the module build, but it's definitely not simple: if the C files are correctly generated, one would need to register the C extensions on the fly, once the build process has already started; and I'm not sure that this is possible. And yes, if it is not possible, one would need to write a complex build system handling the whole compilation chain. In short, I agree, but I don't feel confident enough with distutils to build a ? la numpy build_src command :) -- Nicolas Dumazet ? NicDumZ [ n?k.d?ymz ] From nicdumz at gmail.com Tue Mar 31 17:20:16 2009 From: nicdumz at gmail.com (Nicolas Dumazet) Date: Wed, 1 Apr 2009 00:20:16 +0900 Subject: [Cython] Naive build system In-Reply-To: <1f077e770903310814u14f1c074kf5771f70539a96e3@mail.gmail.com> References: <1f077e770903301915l47cb09efsc53660b13c0ec6ed@mail.gmail.com> <1f077e770903310814u14f1c074kf5771f70539a96e3@mail.gmail.com> Message-ID: <1f077e770903310820p73c3d014pc7b70ff29f4fb6c7@mail.gmail.com> 2009/4/1 Nicolas Dumazet : > I looked at ways to avoid the extra setup.py work, it seems there's no > "easy way" here. > I thought about extending the distutils build command to build the c > files before the module build, but it's definitely not simple: if the > C files are correctly generated, one would need to register the C > extensions on the fly, once the build process has already started; and > I'm not sure that this is possible. > And yes, if it is not possible, one would need to write a complex > build system handling the whole compilation chain. Or... the setup.py behavior could be modified: When --no-cython-compile is not provided, if the system is not able to generate the .c, it bails out, spitting out an error, instead of trying to back out "intelligently" to the pure python implementation by itself. This would make the setup.py implementation much more simple: * If --no-cython-compile is here, there's almost nothing to do. * If it's not here, add the dependences on the C extensions, extend the build_ext distutils command to generate the C files before building the cython module, (this is actually easy to do) and... voil?! Just exit with an error if you can't generate the C files, suggest to try --no-cython-compile instead. -- Nicolas Dumazet ? NicDumZ [ n?k.d?ymz ] From dagss at student.matnat.uio.no Tue Mar 31 17:39:06 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 31 Mar 2009 17:39:06 +0200 Subject: [Cython] Naive build system In-Reply-To: <1f077e770903310814u14f1c074kf5771f70539a96e3@mail.gmail.com> References: <1f077e770903301915l47cb09efsc53660b13c0ec6ed@mail.gmail.com> <1f077e770903310814u14f1c074kf5771f70539a96e3@mail.gmail.com> Message-ID: <49D2391A.4030102@student.matnat.uio.no> Nicolas Dumazet wrote: > 2009/3/31 Lisandro Dalcin : >> So setup.py should not do a lot of work. We have two ways to make this possible. >> >> 1) push your patch, but let future Cython releases zip/tarballs ship C sources. >> 2) implement this the right way, more or less in the spirit of >> numpy.distutils build_src command. > > For other readers, source is here : > http://projects.scipy.org/numpy/browser/trunk/numpy/distutils/command/build_src.py > >> Just to be clear, I'm +1 on your patch. However, I wanted to point out >> that this still is not the definitive solution. > > And I couldn't agree more with you. > > I looked at ways to avoid the extra setup.py work, it seems there's no > "easy way" here. From what I hear, the usual opinion is that distutils is messed up beyond redemption for anything but simple Python package installation anyway and if one is serious about approaching optimal incremental builds, one should just switch to another tool; sooner rather than later. David Cournapeau knows a lot more about this, as he's working on numscons: SCons builds for NumPy. Perhaps we should ship a Cython plugin to e.g. SCons...David, do you know of any? -- Dag Sverre From cournape at gmail.com Tue Mar 31 17:30:15 2009 From: cournape at gmail.com (David Cournapeau) Date: Wed, 1 Apr 2009 00:30:15 +0900 Subject: [Cython] Naive build system In-Reply-To: <49D2391A.4030102@student.matnat.uio.no> References: <1f077e770903301915l47cb09efsc53660b13c0ec6ed@mail.gmail.com> <1f077e770903310814u14f1c074kf5771f70539a96e3@mail.gmail.com> <49D2391A.4030102@student.matnat.uio.no> Message-ID: <5b8d13220903310830y5df703cal37a52dd0750e062a@mail.gmail.com> On Wed, Apr 1, 2009 at 12:39 AM, Dag Sverre Seljebotn wrote: > Nicolas Dumazet wrote: >> 2009/3/31 Lisandro Dalcin : >>> So setup.py should not do a lot of work. We have two ways to make this possible. >>> >>> 1) push your patch, but let future Cython releases zip/tarballs ship C sources. >>> 2) implement this the right way, more or less in the spirit of >>> numpy.distutils build_src command. >> >> For other readers, source is here : >> http://projects.scipy.org/numpy/browser/trunk/numpy/distutils/command/build_src.py >> >>> Just to be clear, I'm +1 on your patch. However, I wanted to point out >>> that this still is not the definitive solution. >> >> And I couldn't agree more with you. >> >> I looked at ways to avoid the extra setup.py work, it seems there's no >> "easy way" here. > > ?From what I hear, the usual opinion is that distutils is messed up > beyond redemption for anything but simple Python package installation > anyway and if one is serious about approaching optimal incremental > builds, one should just switch to another tool; sooner rather than later. > > David Cournapeau knows a lot more about this, as he's working on > numscons: SCons builds for NumPy. Perhaps we should ship a Cython plugin > to e.g. SCons...David, do you know of any? I implemented basic support last year during Robert's talk at scipy: http://github.com/cournape/numscons/blob/bb3724e021c526f19eb6285492a5d2fb091ddd97/numscons/tools/cython.py With an example there: http://github.com/cournape/numscons/tree/bb3724e021c526f19eb6285492a5d2fb091ddd97/tests/examples/cython I have never used it, though, cheers, David From wstein at gmail.com Tue Mar 31 18:28:13 2009 From: wstein at gmail.com (William Stein) Date: Tue, 31 Mar 2009 09:28:13 -0700 Subject: [Cython] code documentation tool In-Reply-To: References: <49CFF3CB.2000408@gmail.com> <49D07AED.7020905@behnel.de> Message-ID: <85e81ba30903310928x711f2272o914a0348da6332a2@mail.gmail.com> On Mon, Mar 30, 2009 at 1:30 PM, Lisandro Dalcin wrote: > On Mon, Mar 30, 2009 at 4:55 AM, Stefan Behnel wrote: >> >> Martin Gysel wrote: >>> Can anybody recommend a code documentation tool which works good with >>> cython. >> >> Sphinx is good in generating documentation. It's not that satisfying for >> generating API docs from sources or through module introspection. >> > > Indeed. > >>> Is there something which can analyze cython code and do what >>> doxygen does with c or java? >> >> Nothing I know of. Epydoc can read the embedded signatures, as long as you >> stick with Python 2.x syntax and avoid fancy default values. It will fail >> hopelessly (reg-exps, that is) for type annotations in whatever form. >> > > However, take a look at the script in the below (works as a cmd line > replacement for epydoc) > Of course you have to remove stuff at the very beginning and at the > end (the 'epydoc.cfg' rc file stuff) ... > Moreover, this script also monkeypatch the issue of Epydoc trying to > generate GIF files in favor of PNG, as Linux distros (Fedora, in my > case) disable that format for the famous patents problems This is off topic, but for the record "all the relevant patents have expired." This is according to both Wikipedia: http://en.wikipedia.org/wiki/Graphics_Interchange_Format and FSF http://www.gnu.org/philosophy/gif.html William > > http://code.google.com/p/mpi4py/source/browse/trunk/conf/epydocify.py > > Wait a minute... I've pushed something to Cython a long time ago > (Tools/cython-epydoc.py)... But it is likely outdated... > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- William Stein Associate Professor of Mathematics University of Washington http://wstein.org From stefan_ml at behnel.de Tue Mar 31 19:25:00 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 31 Mar 2009 19:25:00 +0200 Subject: [Cython] Package names In-Reply-To: References: <49D137CC.8070707@student.matnat.uio.no> Message-ID: <49D251EC.2070501@behnel.de> Lisandro Dalcin wrote: > All this plus good support for making Cython work from a ZIP would > make it really easy to switch to Cython's other than the system one... But if you keep Cython in a ZIP file, you cannot compile it to C extensions anymore. I doubt that that's worth it. Stefan From stefan_ml at behnel.de Tue Mar 31 20:17:46 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 31 Mar 2009 20:17:46 +0200 Subject: [Cython] Package names In-Reply-To: References: <49D137CC.8070707@student.matnat.uio.no> Message-ID: <49D25E4A.8040309@behnel.de> Lisandro Dalcin wrote: >> 2) Should all the package names be converted to lowercase? > > Perhaps we should follow PEP 8 ? Sure. When we break stuff, make sure we only have to do it once. That also includes asking on python-dev what a suitable target package would be for the stdlib. >> 3) Should one perhaps distribute Cython under two seperate package >> names, so that a standalone version can coexist with the one in stdlib? >> This would facilitate easy installation and usage of non-stdlib Cython >> releases. Can't come up with any good names though. > > 1) cython > 2) cython.core > 3) cython.pyximport Yes, one package for everything. So, what you mean is: rename the current Cython package to "cython.core"? Would that be for the stdlib or for the external package? > Regarding (1), I would make it have in the cython/__init__.py define > __version__, Sure. > and also have cython/run.py for cmd line work with -m > switch, and make __init__py use that for newer Python versions that > let you run packages. Note that those aren't that "new" anymore. What do people think about dropping the "cython" frontend alltogether, and making python -m cython the official way to start the Cython compiler? I doubt that there are still a lot of developers who do not have access to Python 2.4 or later in one way or another. And not being able to run Cython in Python 2.3 wouldn't hurt anyone either, I guess, as long as the generated code keeps compiling. > cython/__init__.py should also manage the shadow stuff Which it already does now. Stefan From dagss at student.matnat.uio.no Tue Mar 31 21:39:48 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 31 Mar 2009 21:39:48 +0200 Subject: [Cython] Refcount question: SetAttrString Message-ID: <49D27184.9020103@student.matnat.uio.no> While ripping out the old temp system (yay!), I discovered the following code in CEnumDefNode. I don't know much about the CPython API, but there's no note in the docs that PyObject_SetAttrString eats a reference. Is this a leak? (It's using a local temporary variable which is not refnannied) for item in self.entry.enum_values: code.putln("%s = PyInt_FromLong(%s); %s" % ( temp, item.cname, code.error_goto_if_null(temp, item.pos))) code.putln('if (__Pyx_SetAttrString(%s, "%s", %s) < 0) %s' % ( Naming.module_cname, item.name, temp, code.error_goto(item.pos))) code.putln("%s = 0;" % temp) -- Dag Sverre From stefan_ml at behnel.de Tue Mar 31 21:42:58 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 31 Mar 2009 21:42:58 +0200 Subject: [Cython] Refcount question: SetAttrString In-Reply-To: <49D27184.9020103@student.matnat.uio.no> References: <49D27184.9020103@student.matnat.uio.no> Message-ID: <49D27242.8090709@behnel.de> Dag Sverre Seljebotn wrote: > While ripping out the old temp system (yay!), I discovered the following > code in CEnumDefNode. I don't know much about the CPython API, but > there's no note in the docs that PyObject_SetAttrString eats a reference. > > Is this a leak? (It's using a local temporary variable which is not > refnannied) > > for item in self.entry.enum_values: > code.putln("%s = PyInt_FromLong(%s); %s" % ( > temp, > item.cname, > code.error_goto_if_null(temp, item.pos))) > code.putln('if (__Pyx_SetAttrString(%s, "%s", %s) < 0) > %s' % ( > Naming.module_cname, > item.name, > temp, > code.error_goto(item.pos))) > code.putln("%s = 0;" % temp) PyObject_SetAttr() maps to PyDict_SetItem(), which INCREFs the value. So we need to care about the reference ourselves here. Stefan From dalcinl at gmail.com Tue Mar 31 22:48:22 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 31 Mar 2009 17:48:22 -0300 Subject: [Cython] Package names In-Reply-To: <49D25E4A.8040309@behnel.de> References: <49D137CC.8070707@student.matnat.uio.no> <49D25E4A.8040309@behnel.de> Message-ID: On Tue, Mar 31, 2009 at 3:17 PM, Stefan Behnel wrote: > > Lisandro Dalcin wrote: >>> 2) Should all the package names be converted to lowercase? >> >> Perhaps we should follow PEP 8 ? > > Sure. When we break stuff, make sure we only have to do it once. That also > includes asking on python-dev what a suitable target package would be for > the stdlib. > > >>> 3) Should one perhaps distribute Cython under two seperate package >>> names, so that a standalone version can coexist with the one in stdlib? >>> This would facilitate easy installation and usage of non-stdlib Cython >>> releases. Can't come up with any good names though. >> >> 1) cython >> 2) cython.core >> 3) cython.pyximport > > Yes, one package for everything. So, what you mean is: rename the current > Cython package to "cython.core"? Would that be for the stdlib or for the > external package? > Both stdlib and external :-) ... but doing 'import cython' should not have the side-effect of importing 'cython.core' ... Then we could decide that stdlib cython.__version__ is old for a project, and hot-fix "sys.path" to get a newer Cython, perhaps from a ZIP file. > >> Regarding (1), I would make it have in the cython/__init__.py define >> __version__, > > Sure. > OK. > >> ?and also have cython/run.py for cmd line work with -m >> switch, and make __init__py use that for newer Python versions that >> let you run packages. > > Note that those aren't that "new" anymore. What do people think about > dropping the "cython" frontend alltogether, and making > > ? ?python -m cython > > the official way to start the Cython compiler? I doubt that there are still > a lot of developers who do not have access to Python 2.4 or later in one > way or another. Stefan, the -m switch is know to be broken in Python 2.4. It does not work for packages. See yourself: $ python2.5 -m distutils # the output is nothing $ python2.4 -m distutils python2.4: module distutils has no associated file that is, 2.4 does not handle __init__.py as the associated file for a package.. > >> cython/__init__.py should also manage the shadow stuff > > Which it already does now. > Ah!! This makes me realize that my original way will not work with the shadow stuff... I'll have to think a bit more... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From fperez.net at gmail.com Tue Mar 31 23:02:24 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 31 Mar 2009 14:02:24 -0700 Subject: [Cython] code documentation tool In-Reply-To: <49D07AED.7020905@behnel.de> References: <49CFF3CB.2000408@gmail.com> <49D07AED.7020905@behnel.de> Message-ID: On Mon, Mar 30, 2009 at 12:55 AM, Stefan Behnel wrote: > > Martin Gysel wrote: >> Can anybody recommend a code documentation tool which works good with >> cython. >> I tried doxygen but I'm rather disappointed. I see the cython project >> uses sphinx for the doc. But is sphinx the best tool to generate api >> docs? > > Sphinx is good in generating documentation. It's not that satisfying for > generating API docs from sources or through module introspection. I'm curious: why do you say that? This doesn't seem so bad to me: http://neuroimaging.scipy.org/site/doc/manual/html/api/index.html e.g. http://neuroimaging.scipy.org/site/doc/manual/html/api/generated/neuroimaging.algorithms.statistics.rft.html In the above our math wasn't using the proper rest markup so it doesn't render right, but if you mark it up correctly, math is fine: http://neuroimaging.scipy.org/site/doc/manual/html/users/glm_spec.html#experimental-model and so are code examples (which can have links to your api docs if you want, which is not so clean if you generate your api docs with some other tool): http://neuroimaging.scipy.org/site/doc/manual/html/users/coordinate_map.html as well as plots: http://neuroimaging.scipy.org/site/doc/manual/html/devel/guidelines/sphinx_helpers.html#inserting-matplotlib-plots In all, we're pretty happy with the sphinx-based workflow, even if we've had to write/improve some of the support plugins ourselves. Cheers, f ps - the above docs have historically had a lot of different conventions, so I cherry-picked pages that rendered OK. The ones that don't are typically our fault, not a problem with sphinx, and we're updating them as we go now that we're happy with the core machinery. From dagss at student.matnat.uio.no Tue Mar 31 23:30:29 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 31 Mar 2009 23:30:29 +0200 Subject: [Cython] Refcount question: SetAttrString In-Reply-To: <49D27242.8090709@behnel.de> References: <49D27184.9020103@student.matnat.uio.no> <49D27242.8090709@behnel.de> Message-ID: <49D28B75.8050205@student.matnat.uio.no> Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> While ripping out the old temp system (yay!), I discovered the following >> code in CEnumDefNode. I don't know much about the CPython API, but >> there's no note in the docs that PyObject_SetAttrString eats a reference. >> >> Is this a leak? (It's using a local temporary variable which is not >> refnannied) >> >> for item in self.entry.enum_values: >> code.putln("%s = PyInt_FromLong(%s); %s" % ( >> temp, >> item.cname, >> code.error_goto_if_null(temp, item.pos))) >> code.putln('if (__Pyx_SetAttrString(%s, "%s", %s) < 0) >> %s' % ( >> Naming.module_cname, >> item.name, >> temp, >> code.error_goto(item.pos))) >> code.putln("%s = 0;" % temp) > > PyObject_SetAttr() maps to PyDict_SetItem(), which INCREFs the value. So we > need to care about the reference ourselves here. OK, this is now [1]. However, I don't know how to make a public enum and am so unable to produce a testcode invoking the code, so someone who knows how to actually make a public enum should follow this one up. [1] http://trac.cython.org/cython_trac/ticket/270 -- Dag Sverre From m.gysel at gmail.com Tue Mar 31 23:51:10 2009 From: m.gysel at gmail.com (Martin Gysel) Date: Tue, 31 Mar 2009 23:51:10 +0200 Subject: [Cython] code documentation tool In-Reply-To: <5b8d13220903291934w69a8cac9y1af6e1722eb4ddca@mail.gmail.com> References: <49CFF3CB.2000408@gmail.com> <5b8d13220903291934w69a8cac9y1af6e1722eb4ddca@mail.gmail.com> Message-ID: <49D2904E.2020702@gmail.com> David Cournapeau schrieb: > Sphinx API doc works by introspection, so it does not matter so much > whether the underlying code is in python or C (cython is "just" a tool > to generate C in that case). > > You can see a simple example here I made for audiolab: > > http://www.ar.media.kyoto-u.ac.jp/members/david/softwares/audiolab/sphinx/fullapi.html > > And the corresponding source file: > > http://github.com/cournape/audiolab/blob/4afd4853dbb273305ab8468e860bf2e651883c20/scikits/audiolab/pysndfile/_sndfile.pyx > I looked through you code and sphinx files, the output seems to be very nice, but I can't reproduce such an output with my code/configuration. AFAIK sphinx gets the info for autodoc from the docstring but how does this work for you because my docstrings are removed in the module (according the cython doc, that's the normal behavior). did I overlook something in your configuration? cheers martin