From greg.ewing at canterbury.ac.nz Sun Mar 1 00:18:41 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 01 Mar 2009 12:18:41 +1300 Subject: [Cython] strict aliasing problem with subclass method In-Reply-To: References: <49A814C1.5050302@behnel.de> <49A83D94.8050004@behnel.de> <49A88B29.3090506@canterbury.ac.nz> Message-ID: <49A9C651.4010708@canterbury.ac.nz> Dag Sverre Seljebotn wrote: > There are certain things the > compiler simply cannot do automatically for you; FORTRAN "solves" this > problem by being a very restricted language, while C has taken a different > route and tries to keep the flexibility while still going in a direction > which makes it as efficient. Well, I think it's a mistake to try to turn C into a language that's all things to all people. Those with Fortran-envy should either use Fortran or design a new language that gives them what they want -- not stuff C up for the rest of us. -- Greg From robertwb at math.washington.edu Sun Mar 1 13:25:06 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 1 Mar 2009 04:25:06 -0800 Subject: [Cython] Final fix for #143 before a release? In-Reply-To: <49A8413B.4030204@behnel.de> References: <49A8413B.4030204@behnel.de> Message-ID: On Feb 27, 2009, at 11:38 AM, Stefan Behnel wrote: > Hi, > > the last remaining ticket for 0.11 is #143. I think the right fix > for this > release is to disable the typing of locals in .pxd files, and to > change > #143 into a feature request. Fixed. The site is temporarily down, so I can't push right now. - Robert From wstein at gmail.com Sun Mar 1 16:13:14 2009 From: wstein at gmail.com (William Stein) Date: Sun, 1 Mar 2009 10:13:14 -0500 Subject: [Cython] [sage-support] Re: Small groups library is missing In-Reply-To: References: <9405b309-c562-4e73-a7ea-9503cb4331e8@s20g2000yqh.googlegroups.com> <8cf963450902270337g12ae7ff8l36a0b95331c98798@mail.gmail.com> <79e81e94-f8e5-4e70-a6cb-594152aa7433@w9g2000yqa.googlegroups.com> Message-ID: <85e81ba30903010713v1c7239fft20d0485305a4b32a@mail.gmail.com> > sagemath and trac are down now anyway, so I'm not sure if this can be > fixed today. > I can start working on it after I finish grading an exam tomorrow and > Almost definitely will be free to work on it a large chunk of the day > Tuesday. > They (=sagemath, cython, etc.) are all back up now. Apache used up all memory on sagemath for unknown reasons... and we're now logging all output of top and have adjusted a bunch of parameters for apache2, so we'll see what happens. William From hoytak at cs.ubc.ca Sun Mar 1 18:51:09 2009 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Sun, 1 Mar 2009 09:51:09 -0800 Subject: [Cython] PyBool_CheckExact not found Message-ID: <4db580fd0903010951r1ef5b9d6gfd322383cabfcc95@mail.gmail.com> Hello, I'm using the latest pull from cython-dev, 1800:e8b5a8e5da3d, with python 2.5.4, and some of my code calls PyBool_CheckExact. However, this doesn't seem to be defined anywhere. When I compile it with gcc, it warns that the symbol is defined implicitly, and trying to load it results in an import error saying it's an undefined symbol. I'll try to get a specific example together where it happens and file a bug report, but it seems like a simple-to-fix version issue here. --Hoyt -- ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From hoytak at cs.ubc.ca Sun Mar 1 19:26:33 2009 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Sun, 1 Mar 2009 10:26:33 -0800 Subject: [Cython] PyBool_CheckExact not found In-Reply-To: <4db580fd0903010951r1ef5b9d6gfd322383cabfcc95@mail.gmail.com> References: <4db580fd0903010951r1ef5b9d6gfd322383cabfcc95@mail.gmail.com> Message-ID: <4db580fd0903011026p20d09b8fo20be756f4b20dd77@mail.gmail.com> It's now Ticket #227 (http://trac.cython.org/cython_trac/ticket/227). The quick fix is to not use bool, but use bint instead (of course). I had accidentally typed bool instead of bint in one place in my code, and fixing that got my code working. Still, it's a bug, just more of a corner case. --Hoyt On Sun, Mar 1, 2009 at 9:51 AM, Hoyt Koepke wrote: > Hello, > > I'm using the latest pull from cython-dev, 1800:e8b5a8e5da3d, with > python 2.5.4, and some of my code calls PyBool_CheckExact. ?However, > this doesn't seem to be defined anywhere. ?When I compile it with gcc, > it warns that the symbol is defined implicitly, and trying to load it > results in an import error saying it's an undefined symbol. > > I'll try to get a specific example together where it happens and file > a bug report, but it seems like a simple-to-fix version issue here. > > --Hoyt > > > -- > > ++++++++++++++++++++++++++++++++++++++++++++++++ > + Hoyt Koepke > + University of Washington Department of Statistics > + http://www.stat.washington.edu/~hoytak/ > + hoytak at gmail.com > ++++++++++++++++++++++++++++++++++++++++++ > -- ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From stefan_ml at behnel.de Sun Mar 1 21:29:18 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 01 Mar 2009 21:29:18 +0100 Subject: [Cython] PATCH: use function pointer arguments in import/export api functions In-Reply-To: <49A8595B.8040505@behnel.de> References: <49A8595B.8040505@behnel.de> Message-ID: <49AAF01E.6040005@behnel.de> Stefan Behnel wrote: > Lisandro Dalcin wrote: >> Just to prevent this: >> warning: ISO C forbids conversion of function pointer to object pointer type > > with this patch applied, I now get a warning on this line: > > p = *(void **)&f; > > src/lxml/lxml.etree.c: In function '__Pyx_ExportFunction': > src/lxml/lxml.etree.c:148416: warning: dereferencing type-punned pointer > will break strict-aliasing rules > > This needs fixing, I guess. That said, I'm more in favour of enabling clean > strict-aliasing than to be strictly C89 compliant, unless we can have both. :) I reverted the patch for now. Lisandro, if you can come up with a fix that does not break strict aliasing (or a partial patch that only adds consts and the like), please give it another try. Stefan From stefan_ml at behnel.de Sun Mar 1 21:34:32 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 01 Mar 2009 21:34:32 +0100 Subject: [Cython] PyBool_CheckExact not found In-Reply-To: <4db580fd0903011026p20d09b8fo20be756f4b20dd77@mail.gmail.com> References: <4db580fd0903010951r1ef5b9d6gfd322383cabfcc95@mail.gmail.com> <4db580fd0903011026p20d09b8fo20be756f4b20dd77@mail.gmail.com> Message-ID: <49AAF158.7020505@behnel.de> Hoyt Koepke wrote: > It's now Ticket #227 (http://trac.cython.org/cython_trac/ticket/227). Fixed. Stefan From zachary.pincus at yale.edu Sun Mar 1 21:36:42 2009 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Sun, 1 Mar 2009 15:36:42 -0500 Subject: [Cython] cython / numpy / mac(?) issue Message-ID: <074811C8-98C4-44FC-A6FF-DAE6DA57DE5C@yale.edu> Hello all, I've run into a problem using some cython+numpy code that some folks recently made available. After some back and forth with the original authors, we've determined that the problem appears to be with cython on my box. I've attached the offending code and will post a test-case below, but first some configuration details: I'm using OS X 10.5.6 and a python 2.5.2 that's installed as a "framework" (the standard OS X way of installing Python) -- the only major difference is that the python directories live in /Library/ Frameworks/[etc.] instead of /usr/local. First, I should note that on my system, using both the latest stable Cython release as well as the tip of cython-devel from hg, cython's numpy tests fail utterly, due to some sort of error with include paths: hg clone http://hg.cython.org/cython-devel/ cd cython-devel make local make test [...output...] compiling (c) and running numpy_test ... numpy_test.c: 129:31:numpy_test.c:129:31: error: error: numpy/arrayobject.h: No such file or directorynumpy/arrayobject.h: No such file or directory The error using cython 0.10.3 is pretty much the same, except with the spacing errors fixed: compiling (c) and running numpy_test ... numpy_test.c:112:31: error: numpy/arrayobject.h: No such file or directory Also, I should point out that the default OS X filesystem is case- insensitive, so using a directory named 'BUILD' for building the tests is not a great idea, as it is the same as the python 'build' directory, so the latter gets nuked by the 'rm -rf BUILD' in 'make test'. Anyhow, if I install cython despite these issues, I can build numpy- using projects fine... for some reason, the right include directories get pulled in. However, thereafter I have further issues. I've attached the troublesome code; it's pretty simple but I don't really know enough cython yet to pull it into a minimal test case. (I suspect from the nature of the errors that this should be easy.) Anyhow, here's the test case and error: cd stefanv-bilateral python setup.py build_ext -i python Python 2.5.2 (r252:60911, Feb 22 2008, 07:57:53) [GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> print numpy.version.version 1.3.0.dev6359 >>> import Cython.Compiler.Version >>> print Cython.Compiler.Version.version 0.11.beta >>> import bilateral >>> bilateral.bilateral(numpy.zeros((4,4)), 4, 4) Traceback (most recent call last): File "", line 1, in File "bilateral/bilateral.py", line 38, in bilateral filter_fcn = _BB.Bilat_fcn(xy_sig, z_sig, filter_size) File "bilateral_base.pyx", line 40, in bilateral.bilateral_base.Bilat_fcn.__init__ (bilateral/ bilateral_base.c:752) NameError: np Now, bilateral_base.pyx has the line 'cimport numpy as np' at the top, but somehow a NameError gets thrown the first time that 'np' is used, at the end of this snippet: def __init__(self, spat_sig, inten_sig, filter_size=None): if filter_size is not None and filter_size >= 2: self.xy_size = int(filter_size) else: self.xy_size = int(round(spat_sig*4)) # Make filter size odd self.xy_size += 1-self.xy_size%2 x = np.arange(self.xy_size, dtype=float) Just for fun, I tried adding 'import numpy as np' to bilateral_base.pyx after the cimport line (this is what numpy_text.pyx does in the cython tests). Here then is the error that I get: >>> bilateral.bilateral(numpy.zeros((4,4)), 4, 4) Traceback (most recent call last): File "", line 1, in File "bilateral/bilateral.py", line 40, in bilateral return generic_filter(mat, filter_fcn.cfilter, size=size, mode=mode) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ python2.5/site-packages/scipy/ndimage/filters.py", line 992, in generic_filter cval, origins, extra_arguments, extra_keywords) File "bilateral_base.pyx", line 73, in bilateral.bilateral_base.Bilat_fcn.cfilter (bilateral/bilateral_base.c: 1443) AttributeError: 'numpy.ndarray' object has no attribute 'dimensions' This corresponds to the last line in this snippet: def cfilter(self, np.ndarray data): 'An optimized implementation' cdef np.ndarray kernel = self.xy_ker cdef double sigma = self.inten_sig cdef double weight_i, weight, result, centre, dat_i cdef double *pdata=data.data, *pker=kernel.data cdef int i, dim = data.dimensions[0] Now, numpy arrays don't expose a dimensions attribute in python, but the array struct has such a member, which I suppose should be exposed to cython via 'cimport numpy'. So, clearly something is going wrong with cython's numpy support on my system -- the tests fail, as well as actual use of numpy, but the failures are different. The authors of this code can't reproduce this with the same versions of cython and numpy on their linux systems, so I suspect that this might be a mac issue, perhaps relating to some assumptions about where the numpy headers live? On a framework install, Python.h lives in: /Library/Frameworks/Python.framework/Versions/2.5/include but numpy/arrayobject.h is in: /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site- packages/numpy/core/include Can anyone offer any suggestions here? Thanks, Zach Pincus Postdoctoral Fellow, lab of Frank Slack Molecular, Cellular and Developmental Biology Yale University -------------- next part -------------- A non-text attachment was scrubbed... Name: stefanv-bilateral.zip Type: application/zip Size: 3225 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090301/ced4ab19/attachment-0001.zip From dagss at student.matnat.uio.no Sun Mar 1 22:13:00 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 01 Mar 2009 22:13:00 +0100 Subject: [Cython] cython / numpy / mac(?) issue Message-ID: <3318790436.424141@smtp.netcom.no> All of these are expected, I don't see any issues here (except perhaps lack of documentation; feel free to submit improvements). 1) You should set the INCLUDE environment variable prior to running runtests.py. 2) cimport and import are different things; you need both. 3) The "dimensions" field is in Cython renamed "shape" to be closer to the Python interface. This is done in Cython/Includes/numpy.pxd Dag Sverre Seljebotn -----Original Message----- From: Zachary Pincus Date: Sunday, Mar 1, 2009 9:35 pm Subject: [Cython] cython / numpy / mac(?) issue To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net Hello all, > >I've run into a problem using some cython+numpy code that some folks >recently made available. After some back and forth with the original >authors, we've determined that the problem appears to be with cython >on my box. I've attached the offending code and will post a test-case >below, but first some configuration details: > >I'm using OS X 10.5.6 and a python 2.5.2 that's installed as a >"framework" (the standard OS X way of installing Python) -- the only >major difference is that the python directories live in /Library/ >Frameworks/[etc.] instead of /usr/local. > >First, I should note that on my system, using both the latest stable >Cython release as well as the tip of cython-devel from hg, cython's >numpy tests fail utterly, due to some sort of error with include paths: hg clone http://hg.cython.org/cython-devel/ >cd cython-devel >make local >make test >[...output...] >compiling (c) and running numpy_test ... numpy_test.c: >129:31:numpy_test.c:129:31: error: error: numpy/arrayobject.h: No >such file or directorynumpy/arrayobject.h: No such file or directory > >The error using cython 0.10.3 is pretty much the same, except with the >spacing errors fixed: >compiling (c) and running numpy_test ... numpy_test.c:112:31: error: >numpy/arrayobject.h: No such file or directory > >Also, I should point out that the default OS X filesystem is case- >insensitive, so using a directory named 'BUILD' for building the tests >is not a great idea, as it is the same as the python 'build' >directory, so the latter gets nuked by the 'rm -rf BUILD' in 'make >test'. > >Anyhow, if I install cython despite these issues, I can build numpy- >using projects fine... for some reason, the right include directories >get pulled in. However, thereafter I have further issues. > >I've attached the troublesome code; it's pretty simple but I don't >really know enough cython yet to pull it into a minimal test case. (I >suspect from the nature of the errors that this should be easy.) > >Anyhow, here's the test case and error: >cd stefanv-bilateral >python setup.py build_ext -i >python > >Python 2.5.2 (r252:60911, Feb 22 2008, 07:57:53) >[GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin >Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy > >>> print numpy.version.version >1.3.0.dev6359 > >>> import Cython.Compiler.Version > >>> print Cython.Compiler.Version.version >0.11.beta > >>> import bilateral > >>> bilateral.bilateral(numpy.zeros((4,4)), 4, 4) >Traceback (most recent call last): > File "", line 1, in > File "bilateral/bilateral.py", line 38, in bilateral > filter_fcn = _BB.Bilat_fcn(xy_sig, z_sig, filter_size) > File "bilateral_base.pyx", line 40, in >bilateral.bilateral_base.Bilat_fcn.__init__ (bilateral/ >bilateral_base.c:752) >NameError: np > >Now, bilateral_base.pyx has the line 'cimport numpy as np' at the top, >but somehow a NameError gets thrown the first time that 'np' is used, >at the end of this snippet: > def __init__(self, spat_sig, inten_sig, filter_size=None): > if filter_size is not None and filter_size >= 2: > self.xy_size = int(filter_size) > else: > self.xy_size = int(round(spat_sig*4)) > # Make filter size odd > self.xy_size += 1-self.xy_size%2 > x = np.arange(self.xy_size, dtype=float) > > >Just for fun, I tried adding 'import numpy as np' to >bilateral_base.pyx after the cimport line (this is what numpy_text.pyx >does in the cython tests). Here then is the error that I get: > > >>> bilateral.bilateral(numpy.zeros((4,4)), 4, 4) >Traceback (most recent call last): > File "", line 1, in > File "bilateral/bilateral.py", line 40, in bilateral > return generic_filter(mat, filter_fcn.cfilter, size=size, mode=mode) > File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ >python2.5/site-packages/scipy/ndimage/filters.py", line 992, in >generic_filter > cval, origins, extra_arguments, extra_keywords) > File "bilateral_base.pyx", line 73, in >bilateral.bilateral_base.Bilat_fcn.cfilter (bilateral/bilateral_base.c: >1443) >AttributeError: 'numpy.ndarray' object has no attribute 'dimensions' > >This corresponds to the last line in this snippet: > def cfilter(self, np.ndarray data): > 'An optimized implementation' > cdef np.ndarray kernel = self.xy_ker > cdef double sigma = self.inten_sig > cdef double weight_i, weight, result, centre, dat_i > cdef double *pdata=data.data, *pker=*>kernel.data > cdef int i, dim = data.dimensions[0] > >Now, numpy arrays don't expose a dimensions attribute in python, but >the array struct has such a member, which I suppose should be exposed >to cython via 'cimport numpy'. > >So, clearly something is going wrong with cython's numpy support on my >system -- the tests fail, as well as actual use of numpy, but the >failures are different. The authors of this code can't reproduce this >with the same versions of cython and numpy on their linux systems, so >I suspect that this might be a mac issue, perhaps relating to some >assumptions about where the numpy headers live? > >On a framework install, Python.h lives in: >/Library/Frameworks/Python.framework/Versions/2.5/include >but numpy/arrayobject.h is in: >/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site- >packages/numpy/core/include > > >Can anyone offer any suggestions here? > >Thanks, > > >Zach Pincus >Postdoctoral Fellow, lab of Frank Slack >Molecular, Cellular and Developmental Biology >Yale University > > > > > From zachary.pincus at yale.edu Sun Mar 1 22:43:02 2009 From: zachary.pincus at yale.edu (Zachary Pincus) Date: Sun, 1 Mar 2009 16:43:02 -0500 Subject: [Cython] cython / numpy / mac(?) issue In-Reply-To: <3318790436.424141@smtp.netcom.no> References: <3318790436.424141@smtp.netcom.no> Message-ID: <810D63F3-17E0-4B5C-B25C-6A8A3A38009C@yale.edu> Thanks, Dag! I appreciate your help -- that indeed fixed everything. The only surprising thing here is that the original developers of the code I'm using report no difficulties with that code as-is, using similar versions of cython. Perhaps they have some old cruft laying around from previous installs... Anyhow, your help is appreciated, as are your efforts on integrating numpy support in cython -- I'm excited to start using this tool more! Zach On Mar 1, 2009, at 4:13 PM, Dag Sverre Seljebotn wrote: > All of these are expected, I don't see any issues here (except > perhaps lack of documentation; feel free to submit improvements). > > 1) You should set the INCLUDE environment variable prior to running > runtests.py. > > 2) cimport and import are different things; you need both. > > 3) The "dimensions" field is in Cython renamed "shape" to be closer > to the Python interface. This is done in Cython/Includes/numpy.pxd > > Dag Sverre Seljebotn > -----Original Message----- > From: Zachary Pincus > Date: Sunday, Mar 1, 2009 9:35 pm > Subject: [Cython] cython / numpy / mac(?) issue > To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net > > Hello all, >> >> I've run into a problem using some cython+numpy code that some folks >> recently made available. After some back and forth with the original >> authors, we've determined that the problem appears to be with cython >> on my box. I've attached the offending code and will post a test-case >> below, but first some configuration details: >> >> I'm using OS X 10.5.6 and a python 2.5.2 that's installed as a >> "framework" (the standard OS X way of installing Python) -- the only >> major difference is that the python directories live in /Library/ >> Frameworks/[etc.] instead of /usr/local. >> >> First, I should note that on my system, using both the latest stable >> Cython release as well as the tip of cython-devel from hg, cython's >> numpy tests fail utterly, due to some sort of error with include >> paths: hg clone http://hg.cython.org/cython-devel/ >> cd cython-devel >> make local >> make test >> [...output...] >> compiling (c) and running numpy_test ... numpy_test.c: >> 129:31:numpy_test.c:129:31: error: error: numpy/arrayobject.h: No >> such file or directorynumpy/arrayobject.h: No such file or directory >> >> The error using cython 0.10.3 is pretty much the same, except with >> the >> spacing errors fixed: >> compiling (c) and running numpy_test ... numpy_test.c:112:31: error: >> numpy/arrayobject.h: No such file or directory >> >> Also, I should point out that the default OS X filesystem is case- >> insensitive, so using a directory named 'BUILD' for building the >> tests >> is not a great idea, as it is the same as the python 'build' >> directory, so the latter gets nuked by the 'rm -rf BUILD' in 'make >> test'. >> >> Anyhow, if I install cython despite these issues, I can build numpy- >> using projects fine... for some reason, the right include directories >> get pulled in. However, thereafter I have further issues. >> >> I've attached the troublesome code; it's pretty simple but I don't >> really know enough cython yet to pull it into a minimal test case. (I >> suspect from the nature of the errors that this should be easy.) >> >> Anyhow, here's the test case and error: >> cd stefanv-bilateral >> python setup.py build_ext -i >> python >> >> Python 2.5.2 (r252:60911, Feb 22 2008, 07:57:53) >> [GCC 4.0.1 (Apple Computer, Inc. build 5363)] on darwin >> Type "help", "copyright", "credits" or "license" for more >> information. >>>>> import numpy >>>>> print numpy.version.version >> 1.3.0.dev6359 >>>>> import Cython.Compiler.Version >>>>> print Cython.Compiler.Version.version >> 0.11.beta >>>>> import bilateral >>>>> bilateral.bilateral(numpy.zeros((4,4)), 4, 4) >> Traceback (most recent call last): >> File "", line 1, in >> File "bilateral/bilateral.py", line 38, in bilateral >> filter_fcn = _BB.Bilat_fcn(xy_sig, z_sig, filter_size) >> File "bilateral_base.pyx", line 40, in >> bilateral.bilateral_base.Bilat_fcn.__init__ (bilateral/ >> bilateral_base.c:752) >> NameError: np >> >> Now, bilateral_base.pyx has the line 'cimport numpy as np' at the >> top, >> but somehow a NameError gets thrown the first time that 'np' is used, >> at the end of this snippet: >> def __init__(self, spat_sig, inten_sig, filter_size=None): >> if filter_size is not None and filter_size >= 2: >> self.xy_size = int(filter_size) >> else: >> self.xy_size = int(round(spat_sig*4)) >> # Make filter size odd >> self.xy_size += 1-self.xy_size%2 >> x = np.arange(self.xy_size, dtype=float) >> >> >> Just for fun, I tried adding 'import numpy as np' to >> bilateral_base.pyx after the cimport line (this is what >> numpy_text.pyx >> does in the cython tests). Here then is the error that I get: >> >>>>> bilateral.bilateral(numpy.zeros((4,4)), 4, 4) >> Traceback (most recent call last): >> File "", line 1, in >> File "bilateral/bilateral.py", line 40, in bilateral >> return generic_filter(mat, filter_fcn.cfilter, size=size, >> mode=mode) >> File "/Library/Frameworks/Python.framework/Versions/2.5/lib/ >> python2.5/site-packages/scipy/ndimage/filters.py", line 992, in >> generic_filter >> cval, origins, extra_arguments, extra_keywords) >> File "bilateral_base.pyx", line 73, in >> bilateral.bilateral_base.Bilat_fcn.cfilter (bilateral/ >> bilateral_base.c: >> 1443) >> AttributeError: 'numpy.ndarray' object has no attribute 'dimensions' >> >> This corresponds to the last line in this snippet: >> def cfilter(self, np.ndarray data): >> 'An optimized implementation' >> cdef np.ndarray kernel = self.xy_ker >> cdef double sigma = self.inten_sig >> cdef double weight_i, weight, result, centre, dat_i >> cdef double *pdata=data.data, *pker=> *>kernel.data >> cdef int i, dim = data.dimensions[0] >> >> Now, numpy arrays don't expose a dimensions attribute in python, but >> the array struct has such a member, which I suppose should be exposed >> to cython via 'cimport numpy'. >> >> So, clearly something is going wrong with cython's numpy support on >> my >> system -- the tests fail, as well as actual use of numpy, but the >> failures are different. The authors of this code can't reproduce this >> with the same versions of cython and numpy on their linux systems, so >> I suspect that this might be a mac issue, perhaps relating to some >> assumptions about where the numpy headers live? >> >> On a framework install, Python.h lives in: >> /Library/Frameworks/Python.framework/Versions/2.5/include >> but numpy/arrayobject.h is in: >> /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site- >> packages/numpy/core/include >> >> >> Can anyone offer any suggestions here? >> >> Thanks, >> >> >> Zach Pincus >> Postdoctoral Fellow, lab of Frank Slack >> Molecular, Cellular and Developmental Biology >> Yale University >> >> >> >> >> > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From dalcinl at gmail.com Mon Mar 2 15:00:29 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 2 Mar 2009 12:00:29 -0200 Subject: [Cython] PATCH: use function pointer arguments in import/export api functions In-Reply-To: <49A8595B.8040505@behnel.de> References: <49A8595B.8040505@behnel.de> Message-ID: Stefan, is this the only warning you get related to strict aliasing? No warnings at all on those lines where __Pyx_ExportFunction is called?? If that is the only warning, they I can write a better patch. On Fri, Feb 27, 2009 at 7:21 PM, Stefan Behnel wrote: > Hi, > > Lisandro Dalcin wrote: >> Just to prevent this: >> warning: ISO C forbids conversion of function pointer to object pointer type > > with this patch applied, I now get a warning on this line: > > ? ?p = *(void **)&f; > > src/lxml/lxml.etree.c: In function '__Pyx_ExportFunction': > > > src/lxml/lxml.etree.c:148416: warning: dereferencing type-punned pointer > will break strict-aliasing rules > > > This needs fixing, I guess. That said, I'm more in favour of enabling clean > strict-aliasing than to be strictly C89 compliant, unless we can have both. :) > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Mon Mar 2 15:08:16 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 2 Mar 2009 15:08:16 +0100 (CET) Subject: [Cython] PATCH: use function pointer arguments in import/export api functions In-Reply-To: References: <49A8595B.8040505@behnel.de> Message-ID: <43034.213.61.181.86.1236002896.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Lisandro Dalcin wrote: > Stefan, is this the only warning you get related to strict aliasing? > No warnings at all on those lines where __Pyx_ExportFunction is > called?? Yes, the only one that is left under Py3. > If that is the only warning, they I can write a better patch. Please do. Stefan From dalcinl at gmail.com Mon Mar 2 22:08:34 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 2 Mar 2009 19:08:34 -0200 Subject: [Cython] PyErr_Format, bare "%s" format specifier, and Py_TYPE(o)->tp_name Message-ID: After a fast view to Stefan's fix for #227, I noticed that Cython uses bare "%s" format specifiers when using PyErr_Format() to print Py_TYPE(o)->tp_name. Perhaps we should follow core CPython convention of using "%.200s" ?? . Anyway, core CPython does not follow this rule at all places ;-) -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Mon Mar 2 22:21:30 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 2 Mar 2009 19:21:30 -0200 Subject: [Cython] PATCH: use function pointer arguments in import/export api functions In-Reply-To: <43034.213.61.181.86.1236002896.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49A8595B.8040505@behnel.de> <43034.213.61.181.86.1236002896.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: Stefan, here you have the new patch... Check GCC manpages about -fstrict-aliasing, this should work. Moreover, this pattern is used on Open MPI codebase, likely for the same issues. On Mon, Mar 2, 2009 at 12:08 PM, Stefan Behnel wrote: > Lisandro Dalcin wrote: >> Stefan, is this the only warning you get related to strict aliasing? >> No warnings at all on those lines where ?__Pyx_ExportFunction is >> called?? > > Yes, the only one that is left under Py3. > > >> If that is the only warning, they I can write a better patch. > > Please do. > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 -------------- next part -------------- A non-text attachment was scrubbed... Name: IMP_EXP_API.diff Type: application/octet-stream Size: 4379 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090302/5393b9a6/attachment-0001.obj From dalcinl at gmail.com Thu Mar 5 23:58:30 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 5 Mar 2009 20:58:30 -0200 Subject: [Cython] Why is 'with gil' disabled in the parser? Any good reason? Message-ID: Error converting Pyrex file to C: ------------------------------------------------------------ ... cdef int f1(): cdef int i = 0 with gil: ^ ------------------------------------------------------------ /u/dalcinl/tmp/gil2.pyx:3:3: undeclared name not builtin: gil Of course, this one-line patch would enable it. diff -r a0ebfe3b1cc2 Cython/Compiler/Parsing.py --- a/Cython/Compiler/Parsing.py Tue Mar 03 20:44:15 2009 +0100 +++ b/Cython/Compiler/Parsing.py Thu Mar 05 20:54:59 2009 -0200 @@ -1363,8 +1363,7 @@ def p_with_statement(s): pos = s.position() s.next() # 'with' -# if s.sy == 'IDENT' and s.systring in ('gil', 'nogil'): - if s.sy == 'IDENT' and s.systring == 'nogil': + if s.sy == 'IDENT' and s.systring in ('gil', 'nogil'): state = s.systring s.next() body = p_suite(s) BTW, If the above is enabled, the following patch would be needed diff -r a0ebfe3b1cc2 Cython/Compiler/Nodes.py --- a/Cython/Compiler/Nodes.py Tue Mar 03 20:44:15 2009 +0100 +++ b/Cython/Compiler/Nodes.py Thu Mar 05 20:53:29 2009 -0200 @@ -4566,7 +4566,7 @@ def generate_execution_code(self, code): if self.state == 'gil': - code.putln("PyGILState_Release();") + code.putln("PyGILState_Release(_save);") else: code.putln("Py_BLOCK_THREADS") -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From ondrej at certik.cz Fri Mar 6 06:39:40 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Fri, 6 Mar 2009 00:39:40 -0500 Subject: [Cython] new cython fails to compile a.(key=hash) Message-ID: <85b5c3130903052139p1a073aadi7956184df26fc937@mail.gmail.com> Hi, the new cython in Debian fails to compile my old code: $ cython sympy_pyx.pyx Error converting Pyrex file to C: ------------------------------------------------------------ ... # this is faster: cdef list a a = list(self._args) #a.sort(key=hash) a.sort(None, hash) ^ ------------------------------------------------------------ /home/ondra/repos/sympyx/sympy_pyx.pyx:330:14: Call with wrong number of arguments (expected 1, got 3) Error converting Pyrex file to C: But it should work. Any ideas? Originally, I used the commented out thing before and now it also doesn't work: $ cython sympy_pyx.pyx Error converting Pyrex file to C: ------------------------------------------------------------ ... #h = hash(self._args_set) # this is faster: cdef list a a = list(self._args) a.sort(key=hash) ^ ------------------------------------------------------------ /home/ondra/repos/sympyx/sympy_pyx.pyx:329:9: Cannot convert 'int (object) except -1' to Python object Error converting Pyrex file to C: Any ideas? Is this supposed to work? Ondrej From greg.ewing at canterbury.ac.nz Fri Mar 6 07:23:00 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 06 Mar 2009 19:23:00 +1300 Subject: [Cython] Why is 'with gil' disabled in the parser? Any good reason? In-Reply-To: References: Message-ID: <49B0C144.4070107@canterbury.ac.nz> Lisandro Dalcin wrote: > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > cdef int f1(): > cdef int i = 0 > with gil: > ^ > ------------------------------------------------------------ I started to add this to Pyrex, but half way through I realised it was going to be much more complicated to implement than I had thought, and backed out of the idea in favour of having a 'with gil' clause on the whole function instead, e.g. def myfunc() with gil: ... > Of course, this one-line patch would enable it. > > - if s.sy == 'IDENT' and s.systring == 'nogil': > + if s.sy == 'IDENT' and s.systring in ('gil', 'nogil'): > > BTW, If the above is enabled, the following patch would be needed > > def generate_execution_code(self, code): > if self.state == 'gil': > - code.putln("PyGILState_Release();") > + code.putln("PyGILState_Release(_save);") You'd have to do quite a lot more than that. I don't remember all the details, but essentially if you want to use Python stuff anywhere in the function, there's setup code that needs to run with the GIL held before you even get into the user's code. It's not easy to just turn it on half way through. -- Greg From ondrej at certik.cz Fri Mar 6 07:21:26 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Fri, 6 Mar 2009 01:21:26 -0500 Subject: [Cython] new cython fails to compile a.(key=hash) In-Reply-To: <85b5c3130903052139p1a073aadi7956184df26fc937@mail.gmail.com> References: <85b5c3130903052139p1a073aadi7956184df26fc937@mail.gmail.com> Message-ID: <85b5c3130903052221x19e01b00u410520947e8775ea@mail.gmail.com> On Fri, Mar 6, 2009 at 12:39 AM, Ondrej Certik wrote: > Hi, > > the new cython in Debian fails to compile my old code: > > $ cython ?sympy_pyx.pyx > > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > > ? ? ? ?# this is faster: > ? ? ? ?cdef list a ^^^ I removed the above line and things started to work. I think it was a problem on my side. I don't understand the reason, but I think all is ok now, so don't loose time with this. Ondrej From hoytak at cs.ubc.ca Fri Mar 6 08:09:26 2009 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Thu, 5 Mar 2009 23:09:26 -0800 Subject: [Cython] Best way to display parse tree Message-ID: <4db580fd0903052309r73680d27g2d09d637245f699a@mail.gmail.com> Hello, I'm trying to get into cython development, and I'm having difficulty understanding the internals of the parse tree and how it works. Being able to print or display the parse tree would really help in this effort, as it's really tedious to work through the code trying to figure this stuff out. What is the best way to see the parse tree of a given pyx file / function / code block? Is there an easy way to do it that I don't know about? Thanks! --Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From dalcinl at gmail.com Fri Mar 6 09:24:15 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 6 Mar 2009 05:24:15 -0300 Subject: [Cython] Why is 'with gil' disabled in the parser? Any good reason? In-Reply-To: <49B0C144.4070107@canterbury.ac.nz> References: <49B0C144.4070107@canterbury.ac.nz> Message-ID: On Fri, Mar 6, 2009 at 3:23 AM, Greg Ewing wrote: > > I started to add this to Pyrex, but half way through > I realised it was going to be much more complicated to > implement than I had thought, and backed out of the idea > in favour of having a 'with gil' clause on the whole > function instead, e.g. Yes, I understand :-) ... >> Of course, this one-line patch would enable it. >> >> - ? ?if s.sy == 'IDENT' and s.systring == 'nogil': >> + ? ?if s.sy == 'IDENT' and s.systring in ('gil', 'nogil'): >> > > You'd have to do quite a lot more than that. I don't > remember all the details, but essentially if you want to > use Python stuff anywhere in the function, there's setup > code that needs to run with the GIL held before you even > get into the user's code. It's not easy to just turn it > on half way through. > Yes, of course... but I was thinking on taking advantage of this for getting some king of locking in pure C functions.... Supose you have a a pure C functions (I mean, no Py stuff), perhaps even declared 'with gil' in its signature... Now suppose that function access some global variables, then we would be able to get some sort of (global) locking using a 'with gil' block. Am I being clear? Do this make sense? Would this be useful? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Fri Mar 6 09:31:17 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 6 Mar 2009 09:31:17 +0100 (CET) Subject: [Cython] new cython fails to compile a.(key=hash) In-Reply-To: <85b5c3130903052139p1a073aadi7956184df26fc937@mail.gmail.com> References: <85b5c3130903052139p1a073aadi7956184df26fc937@mail.gmail.com> Message-ID: <34788.213.61.181.86.1236328277.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Ondrej Certik wrote: > the new cython in Debian fails to compile my old code: > > $ cython sympy_pyx.pyx > > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > > # this is faster: > cdef list a > a = list(self._args) > #a.sort(key=hash) > a.sort(None, hash) > ^ > ------------------------------------------------------------ > > /home/ondra/repos/sympyx/sympy_pyx.pyx:330:14: Call with wrong number > of arguments (expected 1, got 3) This is a known bug that apparently wasn't fixed in the 0.10 releases. The (trivial) fix is here: http://hg.cython.org/cython-devel/rev/4bbc6166052f Stefan From greg.ewing at canterbury.ac.nz Fri Mar 6 10:45:46 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 06 Mar 2009 22:45:46 +1300 Subject: [Cython] Why is 'with gil' disabled in the parser? Any good reason? In-Reply-To: References: <49B0C144.4070107@canterbury.ac.nz> Message-ID: <49B0F0CA.8080602@canterbury.ac.nz> Lisandro Dalcin wrote: > Supose you have a a pure C functions (I mean, no Py stuff), perhaps > even declared 'with gil' in its signature... Now suppose that > function access some global variables, then we would be able to get > some sort of (global) locking using a 'with gil' block. Am I being > clear? Do this make sense? Would this be useful? Hmmm... you'd be creating a third kind of state, as well as 'gil held' and 'gil not held', there would be 'gil held but you're still not allowed to mess with Python stuff in this function'. Sounds a bit messy to me. It would be better to create an intermediate function declared 'with gil' and call that from the nogil function. Then the intermediate function will run with the gil held. Also, I wouldn't like to build in a language feature whose only purpose in life is to enable abusing the gil for non-python purposes. A 'with gil' block ought to deliver all that it seems to promise, or not be there. -- Greg From dalcinl at gmail.com Fri Mar 6 11:25:58 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 6 Mar 2009 07:25:58 -0300 Subject: [Cython] Why is 'with gil' disabled in the parser? Any good reason? In-Reply-To: <49B0F0CA.8080602@canterbury.ac.nz> References: <49B0C144.4070107@canterbury.ac.nz> <49B0F0CA.8080602@canterbury.ac.nz> Message-ID: Well, yes, you are totally right... On Fri, Mar 6, 2009 at 6:45 AM, Greg Ewing wrote: > Lisandro Dalcin wrote: > >> Supose you have a a pure C functions (I mean, no Py stuff), perhaps >> even declared ?'with gil' in its signature... Now suppose that >> function access some global variables, then we would be able to get >> some sort of (global) locking using a 'with gil' block. Am I being >> clear? Do this make sense? Would this be useful? > > Hmmm... you'd be creating a third kind of state, as well > as 'gil held' and 'gil not held', there would be 'gil > held but you're still not allowed to mess with Python > stuff in this function'. Sounds a bit messy to me. > > It would be better to create an intermediate function > declared 'with gil' and call that from the nogil > function. Then the intermediate function will run > with the gil held. > > Also, I wouldn't like to build in a language feature > whose only purpose in life is to enable abusing the > gil for non-python purposes. A 'with gil' block ought > to deliver all that it seems to promise, or not be > there. > > -- > Greg > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Fri Mar 6 11:55:25 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 6 Mar 2009 11:55:25 +0100 (CET) Subject: [Cython] Best way to display parse tree In-Reply-To: <4db580fd0903052309r73680d27g2d09d637245f699a@mail.gmail.com> References: <4db580fd0903052309r73680d27g2d09d637245f699a@mail.gmail.com> Message-ID: <33209.213.61.181.86.1236336925.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Hoyt Koepke wrote: > I'm trying to get into cython development You are very welcome. > and I'm having difficulty > understanding the internals of the parse tree and how it works. Being > able to print or display the parse tree would really help in this > effort, as it's really tedious to work through the code trying to > figure this stuff out. What is the best way to see the parse tree of > a given pyx file / function / code block? Is there an easy way to do > it that I don't know about? Glad you asked. :) Yes, there is. Look into Main.py, there is a method called create_pipeline(). It imports a class called PrintTree at the top and builds the compiler pipeline at the end. If you throw an instance of that class into the pipeline at any point, it will dump out the current tree for you. Note that the output can be pretty long, so you may want to start with a short example (such as one of the test cases). Stefan From robertwb at math.washington.edu Fri Mar 6 12:43:49 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 6 Mar 2009 03:43:49 -0800 Subject: [Cython] possible problems with temp allocation? In-Reply-To: References: Message-ID: Is this still an issue? On Feb 2, 2009, at 10:18 AM, Lisandro Dalcin wrote: > See the attached files. The line 452 in generated foo.c (attached just > for your convenience) seems to be wrong. Furthermore, valgrind pointed > me to that line. > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0) > 342-451.1594__________________________________________ > _____ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From stefan_ml at behnel.de Fri Mar 6 12:59:58 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 6 Mar 2009 12:59:58 +0100 (CET) Subject: [Cython] possible problems with temp allocation? In-Reply-To: References: Message-ID: <36495.213.61.181.86.1236340798.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Robert Bradshaw wrote: > Is this still an issue? > > On Feb 2, 2009, at 10:18 AM, Lisandro Dalcin wrote: > >> See the attached files. The line 452 in generated foo.c (attached just >> for your convenience) seems to be wrong. Furthermore, valgrind pointed >> me to that line. There was a fix by Dag in reply to that e-mail. Should be ok now. Stefan From sturla at molden.no Fri Mar 6 14:48:55 2009 From: sturla at molden.no (Sturla Molden) Date: Fri, 06 Mar 2009 14:48:55 +0100 Subject: [Cython] Why is 'with gil' disabled in the parser? Any good reason? In-Reply-To: <49B0C144.4070107@canterbury.ac.nz> References: <49B0C144.4070107@canterbury.ac.nz> Message-ID: <49B129C7.2080207@molden.no> On 3/6/2009 7:23 AM, Greg Ewing wrote: > You'd have to do quite a lot more than that. I don't > remember all the details, There is also the simplified GIL API, which only requires two function calls. This is e.g. how ctypes makes sure the GIL is held in callbacks to Python. A 'with gil' block could just expand to: { PyGILState_STATE _gilstate = PyGILState_Ensure(); /* now we have the GIL */ PyGILState_Release(_gilstate); } Nothing else is required. The simplified GIL API makes sure the function does not deadlock if the caller already has the GIL. The only requirement is that a call to PyGILState_Ensure should always have a matching call to PyGILState_Release. Sturla Molden From sturla at molden.no Fri Mar 6 15:04:28 2009 From: sturla at molden.no (Sturla Molden) Date: Fri, 06 Mar 2009 15:04:28 +0100 Subject: [Cython] Why is 'with gil' disabled in the parser? Any good reason? In-Reply-To: <49B129C7.2080207@molden.no> References: <49B0C144.4070107@canterbury.ac.nz> <49B129C7.2080207@molden.no> Message-ID: <49B12D6C.8090607@molden.no> On 3/6/2009 2:48 PM, Sturla Molden wrote: > { > PyGILState_STATE _gilstate = PyGILState_Ensure(); > > /* now we have the GIL */ > > PyGILState_Release(_gilstate); > } What this means is PyGILState_Release does not release the GIL. It just resets the GIL to whatever it was before PyGILState_Ensure was called. So, { /* We might or might not hold the GIL. */ PyGILState_STATE _gilstate = PyGILState_Ensure(); /* We have the GIL. Now its safe to use the rest of Python's C API. */ PyGILState_Release(_gilstate); /* We might or might not hold the GIL. */ } Thus, with nogil: with gil: with gil: with nogil: Would be Py_BEGIN_ALLOW_THREADS { PyGILState_STATE _gilstate = PyGILState_Ensure(); { PyGILState_STATE _gilstate = PyGILState_Ensure(); { Py_BEGIN_ALLOW_THREADS { /*suite*/ } Py_END_ALLOW_THREADS } PyGILState_Release(_gilstate); } PyGILState_Release(_gilstate); } Py_END_ALLOW_THREADS S.M. From stefan_ml at behnel.de Fri Mar 6 17:32:10 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 6 Mar 2009 17:32:10 +0100 (CET) Subject: [Cython] Why is 'with gil' disabled in the parser? Any good reason? In-Reply-To: <49B129C7.2080207@molden.no> References: <49B0C144.4070107@canterbury.ac.nz> <49B129C7.2080207@molden.no> Message-ID: <52217.213.61.181.86.1236357130.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Sturla Molden wrote: > On 3/6/2009 7:23 AM, Greg Ewing wrote: > >> You'd have to do quite a lot more than that. I don't >> remember all the details, > > There is also the simplified GIL API, which only requires two function > calls. This is e.g. how ctypes makes sure the GIL is held in callbacks > to Python. > > A 'with gil' block could just expand to: > > { > PyGILState_STATE _gilstate = PyGILState_Ensure(); > > /* now we have the GIL */ > > PyGILState_Release(_gilstate); > } > > Nothing else is required. For pure C code, that might be true. However, in the current Pyrex/Cython language (and in Python, BTW), function local variables are global to the function and not local to a block. And supporting a "with gil" block would require to restrict all ref-counting related operations (e.g. variable initialisations and function cleanup) to the specific block that uses them. That's the hard part. Stefan From dagss at student.matnat.uio.no Fri Mar 6 17:36:00 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 06 Mar 2009 17:36:00 +0100 Subject: [Cython] Best way to display parse tree Message-ID: <3319205783.355064@smtp.netcom.no> There's also another way for more local debugging: print node.dump() (see Node in Nodes.py). Practical for e.g. print self.dump() within node methods. Also the dumptree function in Main.py can be used in the same way as Stefan mentions but prints the tree in the dump()-format. this should probably be consolidated -- I implemented what Stefan talked about first, and then I found that I often wanted to simply print a small part of the tree and added another way... Dag Sverre Seljebotn -----Original Message----- From: "Stefan Behnel" Date: Friday, Mar 6, 2009 11:55 am Subject: Re: [Cython] Best way to display parse tree To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net Hoyt Koepke wrote: > I'm trying to get into cython development > >You are very welcome. > > >> and I'm having difficulty > understanding the internals of the parse tree and how it works. Being > able to print or display the parse tree would really help in this > effort, as it's really tedious to work through the code trying to > figure this stuff out. What is the best way to see the parse tree of > a given pyx file / function / code block? Is there an easy way to do > it that I don't know about? > >Glad you asked. :) > >Yes, there is. Look into Main.py, there is a method called >create_pipeline(). It imports a class called PrintTree at the top and builds the compiler pipeline at the end. If you throw an instance of that class into the pipeline at any point, it will dump out the current tree for you. Note that the output can be pretty long, so you may want to start with a short example (such as one of the test cases). > >Stefan > >_______________________________________________ >Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > From dalcinl at gmail.com Fri Mar 6 18:41:57 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 6 Mar 2009 15:41:57 -0200 Subject: [Cython] Why is 'with gil' disabled in the parser? Any good reason? In-Reply-To: <49B129C7.2080207@molden.no> References: <49B0C144.4070107@canterbury.ac.nz> <49B129C7.2080207@molden.no> Message-ID: On Fri, Mar 6, 2009 at 11:48 AM, Sturla Molden wrote: > On 3/6/2009 7:23 AM, Greg Ewing wrote: > > There is also the simplified GIL API, which only requires two function > calls. This is e.g. how ctypes makes sure the GIL is held in callbacks > to Python. > Cython already used the simplified GIL API for pure-C functions requiring GIL acquisition. However, AFAIK the simplified GIL API does not suport multiple interpreters, right? Could we do it better? > A 'with gil' block could just expand to: > > { > ? ?PyGILState_STATE _gilstate = PyGILState_Ensure(); > > ? ?/* now we have the GIL */ > > ? ?PyGILState_Release(_gilstate); > } > > Nothing else is required. The simplified GIL API makes sure the function > does not deadlock if the caller already has the GIL. The only > requirement is that a call to PyGILState_Ensure should always have a > matching call to PyGILState_Release. > > > > Sturla Molden > > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From sturla at molden.no Fri Mar 6 19:05:53 2009 From: sturla at molden.no (Sturla Molden) Date: Fri, 6 Mar 2009 19:05:53 +0100 (CET) Subject: [Cython] Why is 'with gil' disabled in the parser? Any good reason? In-Reply-To: References: <49B0C144.4070107@canterbury.ac.nz> <49B129C7.2080207@molden.no> Message-ID: <747612f3e927c6d7892f663b193b4f44.squirrel@webmail.uio.no> > On Fri, Mar 6, 2009 at 11:48 AM, Sturla Molden wrote: > However, AFAIK the simplified GIL API does not suport multiple > interpreters, right? > > Could we do it better? That is correct. It does not support multiple interpreters. Sub-interpreters are a misfeature of CPython anyway. All sorts of weird things can happen when they are used. They are not completely isolated, and many extension modules do not honor them. Sub-interpreters are not useful like .NET appdomains, Java isolates, or tcl threads. Their only raison d'etre seems to be the internals of Apache's mod_python. Sturla Molden From stefan_ml at behnel.de Fri Mar 6 19:26:01 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 06 Mar 2009 19:26:01 +0100 Subject: [Cython] Why is 'with gil' disabled in the parser? Any good reason? In-Reply-To: <747612f3e927c6d7892f663b193b4f44.squirrel@webmail.uio.no> References: <49B0C144.4070107@canterbury.ac.nz> <49B129C7.2080207@molden.no> <747612f3e927c6d7892f663b193b4f44.squirrel@webmail.uio.no> Message-ID: <49B16AB9.9070603@behnel.de> Sturla Molden wrote: >> On Fri, Mar 6, 2009 at 11:48 AM, Sturla Molden wrote: >> However, AFAIK the simplified GIL API does not suport multiple >> interpreters, right? >> >> Could we do it better? > > That is correct. It does not support multiple interpreters. > > Sub-interpreters are a misfeature of CPython anyway. All sorts of weird > things can happen when they are used. They are not completely isolated, > and many extension modules do not honor them. Sub-interpreters are not > useful like .NET appdomains, Java isolates, or tcl threads. That's mainly because the whole infrastructure for making them useful is only available in Py3. http://trac.cython.org/cython_trac/ticket/173 Stefan From dalcinl at gmail.com Fri Mar 6 21:08:33 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 6 Mar 2009 18:08:33 -0200 Subject: [Cython] =?windows-1252?q?GCC=2C_warning_=91=5F=5FPyx=5FPrint=92_?= =?windows-1252?q?defined_but_not_used_=28Python_2=2EX=29?= Message-ID: classmethod.c:1229: warning: ?__Pyx_Print? defined but not used Not sure how to fix it. Could PrintOne also call Print for Py2? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Fri Mar 6 21:11:41 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 6 Mar 2009 12:11:41 -0800 Subject: [Cython] =?windows-1252?q?GCC=2C_warning_=91=5F=5FPyx=5FPrint=92_?= =?windows-1252?q?defined_but_not_used_=28Python_2=2EX=29?= In-Reply-To: References: Message-ID: <6247A7DE-C5DD-474C-BD5D-836FC508607E@math.washington.edu> On Mar 6, 2009, at 12:08 PM, Lisandro Dalcin wrote: > classmethod.c:1229: warning: ?__Pyx_Print? defined but not used > > Not sure how to fix it. Could PrintOne also call Print for Py2? The whole point of PrintOne was to avoid the overhead of calling Print. If we call it after we return, or "if (0) __Pyx_Print(...)", does that get rid of the warning even though it's optimized away? - Robert From dalcinl at gmail.com Fri Mar 6 21:20:25 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 6 Mar 2009 18:20:25 -0200 Subject: [Cython] =?windows-1252?q?GCC=2C_warning_=91=5F=5FPyx=5FPrint=92_?= =?windows-1252?q?defined_but_not_used_=28Python_2=2EX=29?= In-Reply-To: <6247A7DE-C5DD-474C-BD5D-836FC508607E@math.washington.edu> References: <6247A7DE-C5DD-474C-BD5D-836FC508607E@math.washington.edu> Message-ID: On Fri, Mar 6, 2009 at 6:11 PM, Robert Bradshaw wrote: > > The whole point of PrintOne was to avoid the overhead of calling > Print. Robert, I believe some times we complicate implementations too much for the sake of optimization... :-) > If we call it after we return, or "if (0) __Pyx_Print(...)", > does that get rid of the warning even though it's optimized away? > OK, I'll push that as a quick fix, getting those warnings when running the testsuite is really annoying -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Fri Mar 6 21:29:33 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 6 Mar 2009 12:29:33 -0800 Subject: [Cython] =?windows-1252?q?GCC=2C_warning_=91=5F=5FPyx=5FPrint=92_?= =?windows-1252?q?defined_but_not_used_=28Python_2=2EX=29?= In-Reply-To: References: <6247A7DE-C5DD-474C-BD5D-836FC508607E@math.washington.edu> Message-ID: <5026DF3C-4C02-4EDB-9AA3-CD471E283A8D@math.washington.edu> On Mar 6, 2009, at 12:20 PM, Lisandro Dalcin wrote: > On Fri, Mar 6, 2009 at 6:11 PM, Robert Bradshaw > wrote: >> >> The whole point of PrintOne was to avoid the overhead of calling >> Print. > > Robert, I believe some times we complicate implementations too much > for the sake of optimization... :-) This was motivated by some code where printing was the dominant step in the runtime. Unfortunately, PySys_GetObject((char *)"stdout") (which must be called every time) still takes some time. - Robert From dalcinl at gmail.com Fri Mar 6 21:56:48 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 6 Mar 2009 18:56:48 -0200 Subject: [Cython] =?windows-1252?q?GCC=2C_warning_=91=5F=5FPyx=5FPrint=92_?= =?windows-1252?q?defined_but_not_used_=28Python_2=2EX=29?= In-Reply-To: <5026DF3C-4C02-4EDB-9AA3-CD471E283A8D@math.washington.edu> References: <6247A7DE-C5DD-474C-BD5D-836FC508607E@math.washington.edu> <5026DF3C-4C02-4EDB-9AA3-CD471E283A8D@math.washington.edu> Message-ID: On Fri, Mar 6, 2009 at 6:29 PM, Robert Bradshaw wrote: > On Mar 6, 2009, at 12:20 PM, Lisandro Dalcin wrote: >> Robert, I believe some times we complicate implementations too much >> for the sake of optimization... :-) > > This was motivated by some code where printing was the dominant step > in the runtime. Unfortunately, PySys_GetObject((char *)"stdout") > (which must be called every time) still takes some time. > Mmm.. Are you sure that's the only cause? Every thing you need to print involve calls to PyObject_Repr, and they involve creating string objetcs... I would never expect that 'print' could run fast retaining its generality... Anyway, I've pushed a fix calling __Pyx_Print() after return... Well, other compilers (Intel) would warn about unreachable code... Anyway Cython generates a lot of unreachable code :-) ... some day far in the future.... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Fri Mar 6 22:09:18 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 6 Mar 2009 13:09:18 -0800 Subject: [Cython] =?windows-1252?q?GCC=2C_warning_=91=5F=5FPyx=5FPrint=92_?= =?windows-1252?q?defined_but_not_used_=28Python_2=2EX=29?= In-Reply-To: References: <6247A7DE-C5DD-474C-BD5D-836FC508607E@math.washington.edu> <5026DF3C-4C02-4EDB-9AA3-CD471E283A8D@math.washington.edu> Message-ID: <8F5B4DB7-5A1B-4C81-9DD7-1A8411494E24@math.washington.edu> On Mar 6, 2009, at 12:56 PM, Lisandro Dalcin wrote: > On Fri, Mar 6, 2009 at 6:29 PM, Robert Bradshaw > wrote: >> On Mar 6, 2009, at 12:20 PM, Lisandro Dalcin wrote: >>> Robert, I believe some times we complicate implementations too much >>> for the sake of optimization... :-) >> >> This was motivated by some code where printing was the dominant step >> in the runtime. Unfortunately, PySys_GetObject((char *)"stdout") >> (which must be called every time) still takes some time. >> > > Mmm.. Are you sure that's the only cause? Every thing you need to > print involve calls to PyObject_Repr, and they involve creating string > objetcs... I would never expect that 'print' could run fast retaining > its generality... Well, I was printing strings. > Anyway, I've pushed a fix calling __Pyx_Print() after return... Well, > other compilers (Intel) would warn about unreachable code... Anyway > Cython generates a lot of unreachable code :-) ... some day far in the > future.... Yeah, there's a lot of unreachable code. The logic to not output unreachable code may exceed disk space saved...it's an optimization that so far I've been happy to leave to the C compiler. - Robert From dalcinl at gmail.com Sat Mar 7 01:13:35 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 6 Mar 2009 21:13:35 -0300 Subject: [Cython] getting rid of C' stlib strcmp() in generated sources Message-ID: There would be any objections? I've already have a (tested) patch for this... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Sat Mar 7 02:23:50 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 6 Mar 2009 17:23:50 -0800 Subject: [Cython] getting rid of C' stlib strcmp() in generated sources In-Reply-To: References: Message-ID: <61C7B9B2-9B05-4AC3-ADB3-1DA85F922B3F@math.washington.edu> WIth strncmp(), or are you thinking of something else? On Mar 6, 2009, at 4:13 PM, Lisandro Dalcin wrote: > There would be any objections? I've already have a (tested) patch > for this... > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From dalcinl at gmail.com Sat Mar 7 03:32:33 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 6 Mar 2009 23:32:33 -0300 Subject: [Cython] getting rid of C' stlib strcmp() in generated sources In-Reply-To: <61C7B9B2-9B05-4AC3-ADB3-1DA85F922B3F@math.washington.edu> References: <61C7B9B2-9B05-4AC3-ADB3-1DA85F922B3F@math.washington.edu> Message-ID: I'm thinking on removing dependencies on the C stdlib... Then, I'm proposing to use a custom implementation (utility code) for the few places were strcmp() is used... strcmp() (at least for mpi4py) is the only (link-time) dependency on the C stdlib... Why I'm asking this? Just because a guy trying to build mpi4py in a supercomputing environment (SGI Altix/Itanium machines, Intel compilers, SGI MPI implementation, and system Python 2.4 built with some GCC version) had so many headaches to make it work... Perhaps the problem was that the build environment was a bit broken, but the only way to make it work was to fix mpi4py to not use ANY C stdlib call... The only missing bit is strcmp(), as Cython generates code using it... As strcmp() is trivial to implement, Cython uses it in just a few places, and mainly for work around some deficiencies in the Python C-API, then I what to remove this dependency from Cython generated code. IMHO, it is a really good idea to avoid any link-time dependencies in the generated codes... right now, I'm even moved in my own code from using malloc()/free() to use PyMem_Malloc()/PyMem_Free()... On Fri, Mar 6, 2009 at 10:23 PM, Robert Bradshaw wrote: > WIth strncmp(), or are you thinking of something else? > > On Mar 6, 2009, at 4:13 PM, Lisandro Dalcin wrote: > >> There would be any objections? I've already have a (tested) patch >> for this... >> >> >> -- >> Lisandro Dalc?n >> --------------- >> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) >> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) >> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) >> PTLC - G?emes 3450, (3000) Santa Fe, Argentina >> Tel/Fax: +54-(0)342-451.1594 >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Sat Mar 7 03:38:42 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 6 Mar 2009 18:38:42 -0800 Subject: [Cython] getting rid of C' stlib strcmp() in generated sources In-Reply-To: References: <61C7B9B2-9B05-4AC3-ADB3-1DA85F922B3F@math.washington.edu> Message-ID: <88B66FA3-3A41-46AE-8988-5E8FEFFE9AC2@math.washington.edu> On Mar 6, 2009, at 6:32 PM, Lisandro Dalcin wrote: > I'm thinking on removing dependencies on the C stdlib... Then, I'm > proposing to use a custom implementation (utility code) for the few > places were strcmp() is used... strcmp() (at least for mpi4py) is the > only (link-time) dependency on the C stdlib... Interesting. > Why I'm asking this? Just because a guy trying to build mpi4py in a > supercomputing environment (SGI Altix/Itanium machines, Intel > compilers, SGI MPI implementation, and system Python 2.4 built with > some GCC version) had so many headaches to make it work... Perhaps the > problem was that the build environment was a bit broken, but the only > way to make it work was to fix mpi4py to not use ANY C stdlib call... > The only missing bit is strcmp(), as Cython generates code using it... > > As strcmp() is trivial to implement, Cython uses it in just a few > places, and mainly for work around some deficiencies in the Python > C-API, then I what to remove this dependency from Cython generated > code. Sure, I'm fine with that. > - Robert From stefan_ml at behnel.de Sat Mar 7 08:06:15 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 07 Mar 2009 08:06:15 +0100 Subject: [Cython] getting rid of C' stlib strcmp() in generated sources In-Reply-To: References: <61C7B9B2-9B05-4AC3-ADB3-1DA85F922B3F@math.washington.edu> Message-ID: <49B21CE7.4070102@behnel.de> Lisandro Dalcin wrote: > I'm thinking on removing dependencies on the C stdlib... > [...] > As strcmp() is trivial to implement Is it really? It does sound trivial, but I would assume that many platforms have highly optimised implementations for it. At least, I know that libxml2 uses memcmp on some platforms and strcmp on others, depending on what's known to be faster. I wouldn't want to copy this kind of code into Cython, just to work around one broken platform. Honestly, how broken must a platform be to not support strcmp()? > Cython uses it in just a few > places, and mainly for work around some deficiencies in the Python > C-API, then I what to remove this dependency from Cython generated > code. I have no idea where that's currently used and if those places are in any way performance critical, so I can't comment without looking into this. I might even have added a couple of those myself. > IMHO, it is a really good idea to avoid any link-time dependencies in > the generated codes... right now, I'm even moved in my own code from > using malloc()/free() to use PyMem_Malloc()/PyMem_Free()... That by itself is perfectly ok and makes sense (although I don't know if you can use PyMem_*() in nogil functions/blocks). Stefan From robertwb at math.washington.edu Sat Mar 7 08:58:15 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 6 Mar 2009 23:58:15 -0800 Subject: [Cython] getting rid of C' stlib strcmp() in generated sources In-Reply-To: <49B21CE7.4070102@behnel.de> References: <61C7B9B2-9B05-4AC3-ADB3-1DA85F922B3F@math.washington.edu> <49B21CE7.4070102@behnel.de> Message-ID: <15D6E1AF-8B34-48CB-96EB-BEFEFC086D78@math.washington.edu> On Mar 6, 2009, at 11:06 PM, Stefan Behnel wrote: > Lisandro Dalcin wrote: >> I'm thinking on removing dependencies on the C stdlib... >> [...] >> As strcmp() is trivial to implement > > Is it really? It does sound trivial, but I would assume that many > platforms > have highly optimised implementations for it. At least, I know that > libxml2 > uses memcmp on some platforms and strcmp on others, depending on > what's > known to be faster. We could wrap it in an #ifndef strcmp > I wouldn't want to copy this kind of code into Cython, > just to work around one broken platform. > > Honestly, how broken must a platform be to not support strcmp()? I wouldn't say such a platform is broken, but it would certainly be an interesting one. Perhaps embedded devices and the likes. >> Cython uses it in just a few >> places, and mainly for work around some deficiencies in the Python >> C-API, then I what to remove this dependency from Cython generated >> code. > > I have no idea where that's currently used and if those places are > in any > way performance critical, so I can't comment without looking into > this. I > might even have added a couple of those myself. Most places it's used to (1) verify types on importing, (2) verify C api matches, and (3) to verify un-exported types in classmethod and cpdef functions, and (4) in keyword parsing code. The first two don't need to be fast, the third should be optimized anyways (at least the cpdef methods) so I'm going to do that after Lisandro's patch, and the last I'm not sure about. >> IMHO, it is a really good idea to avoid any link-time dependencies in >> the generated codes... right now, I'm even moved in my own code from >> using malloc()/free() to use PyMem_Malloc()/PyMem_Free()... > > That by itself is perfectly ok and makes sense (although I don't > know if > you can use PyMem_*() in nogil functions/blocks). I am pretty sure it's not treadsafe. - Robert From hoytak at cs.ubc.ca Sat Mar 7 09:43:15 2009 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Sat, 7 Mar 2009 00:43:15 -0800 Subject: [Cython] Best way to display parse tree In-Reply-To: <3319205783.355064@smtp.netcom.no> References: <3319205783.355064@smtp.netcom.no> Message-ID: <4db580fd0903070043k7923b8ddgb913daef27c9f620@mail.gmail.com> Thanks, everyone, for the help. I've been busy today but will try to look into it more this weekend. -- Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From stefan_ml at behnel.de Sat Mar 7 15:21:17 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 07 Mar 2009 15:21:17 +0100 Subject: [Cython] getting rid of C' stlib strcmp() in generated sources In-Reply-To: <15D6E1AF-8B34-48CB-96EB-BEFEFC086D78@math.washington.edu> References: <61C7B9B2-9B05-4AC3-ADB3-1DA85F922B3F@math.washington.edu> <49B21CE7.4070102@behnel.de> <15D6E1AF-8B34-48CB-96EB-BEFEFC086D78@math.washington.edu> Message-ID: <49B282DD.6070506@behnel.de> Robert Bradshaw wrote: > On Mar 6, 2009, at 11:06 PM, Stefan Behnel wrote: >> Lisandro Dalcin wrote: >>> Cython uses it in just a few >>> places, and mainly for work around some deficiencies in the Python >>> C-API, then I what to remove this dependency from Cython generated >>> code. >> I have no idea where that's currently used and if those places are >> in any >> way performance critical, so I can't comment without looking into >> this. I might even have added a couple of those myself. > > Most places it's used to (1) verify types on importing, (2) verify C > api matches, and (3) to verify un-exported types in classmethod and > cpdef functions, and (4) in keyword parsing code. The first two don't > need to be fast, the third should be optimized anyways (at least the > cpdef methods) so I'm going to do that after Lisandro's patch, and > the last I'm not sure about. 4) isn't performance critical either. The generic keyword parsing utility function is no longer called for keyword-only arguments or normally declared keyword arguments, as long as you do not use **kwargs. And if it's needed, I would expect the PyDict_Next() loop to dominate any string comparisons. And in the extremely likely case that we are dealing with interned strings, we skip the content comparison all together. I guess I'm +0 on this then. Stefan From dalcinl at gmail.com Sat Mar 7 19:37:48 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sat, 7 Mar 2009 15:37:48 -0300 Subject: [Cython] getting rid of C' stlib strcmp() in generated sources In-Reply-To: <15D6E1AF-8B34-48CB-96EB-BEFEFC086D78@math.washington.edu> References: <61C7B9B2-9B05-4AC3-ADB3-1DA85F922B3F@math.washington.edu> <49B21CE7.4070102@behnel.de> <15D6E1AF-8B34-48CB-96EB-BEFEFC086D78@math.washington.edu> Message-ID: On Sat, Mar 7, 2009 at 4:58 AM, Robert Bradshaw wrote: > On Mar 6, 2009, at 11:06 PM, Stefan Behnel wrote: > >> >> That by itself is perfectly ok and makes sense (although I don't >> know if >> you can use PyMem_*() in nogil functions/blocks). > > I am pretty sure it's not treadsafe. > First of all, AFAIK Cython does not emit any malloc()/free() call, so there are not issues regarding this. If you look at core CPython sources, you will discover that PyMem_Malloc() (implemented in "Objects/object.c") actually used the macro PyMem_MALLOC(). This one is defined in "Include/pymem.h". If "PYMALLOC_DEBUG" is not #define'd (should be the case for non-debug Python builds, right?), then PyMem_MALLOC actually calls system malloc(). So in this case using PyMem_Malloc() (note: the function, not the macro) should be as thread-safe as the system malloc() is. If "PYMALLOC_DEBUG" do is #define'd, then yes, propably PyMem_Malloc() is not thread-safe -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Sat Mar 7 20:03:51 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sat, 7 Mar 2009 16:03:51 -0300 Subject: [Cython] PATCH: removing strcmp() in favor of custom impl or other fixes Message-ID: Here you have the patch... Implemented as an (inline) function in a utility code that anyway is alwas emited (as it is INLINE, the GCC does not warn). Two things worth to point out: * The API import function uses a different thing, just a while loop. I do not want to complicate things in generated API headers for something that can be done with two auxiliar variables and a one-line while loop. * In the keyword handling code, I've used _PyString_Eq(), which is available in the C-API for Py2. If Python's C-API provides what we need, let's use it. Moreover, just in case, please see the implementation of _PyString_Eq(), it is smarter (and likely faster) than a strcmp() call ;-). -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 -------------- next part -------------- A non-text attachment was scrubbed... Name: STRCMP.diff Type: application/octet-stream Size: 5971 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090307/74e3eac6/attachment.obj From stefan_ml at behnel.de Sat Mar 7 20:47:23 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 07 Mar 2009 20:47:23 +0100 Subject: [Cython] PATCH: removing strcmp() in favor of custom impl or other fixes In-Reply-To: References: Message-ID: <49B2CF4B.4010901@behnel.de> Lisandro Dalcin wrote: > Here you have the patch... Implemented as an (inline) function in a > utility code Since it's never used for anything but string equality checks, it would be better to restrict it to that (i.e. 1/0), instead of using the normal strcmp() -1,0,1 return values and additionally checking for "== 0" everywhere in the code. Maybe call it "__Pyx_StrEqual()". > * The API import function uses a different thing, just a while loop. I > do not want to complicate things in generated API headers for > something that can be done with two auxiliar variables and a one-line > while loop. Sure, good idea. > * In the keyword handling code, I've used _PyString_Eq(), which is > available in the C-API for Py2. If Python's C-API provides what we > need, let's use it. Moreover, just in case, please see the > implementation of _PyString_Eq(), it is smarter (and likely faster) > than a strcmp() call ;-). Note that I already used (almost) the same optimisation in the code, which now becomes superfluous. Although I expect it to be a lot faster to check the length inside the loop before calling into the Python's C-API, these checks should be rare enough to just make that if (_PyString_Eq(**name, key)) ... The real fast-path is the loop with plain string pointer equality checks before all that. (BTW, note that this function is only available in Py2, so it's perfectly ok to use it here but not in general). Stefan From robertwb at math.washington.edu Sat Mar 7 21:18:20 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 7 Mar 2009 12:18:20 -0800 Subject: [Cython] Google summer of code Message-ID: Last summer we participated in Google's Summer of Code, which was a great thing for the Cython project. I just wanted to start a thread to coordinate our participation this next year. The timeline for mentoring organizations to apply is this this next week--should we to try applying as a separate organization? Note due to the economy the scale of GSoC is going to be smaller this year (less organizations and less students). Also, who is willing and able to be a mentor this year? Dag put up a great list of projects up at http://wiki.cython.org/ codeprojects . An older (less refined) list is up at http:// wiki.cython.org/enhancements . If anyone else has ideas, it would be good to add to this site (preferably the codeprojects one, which is better oriented to GSoC students. - Robert From robertwb at math.washington.edu Sat Mar 7 21:22:54 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 7 Mar 2009 12:22:54 -0800 Subject: [Cython] PATCH: removing strcmp() in favor of custom impl or other fixes In-Reply-To: <49B2CF4B.4010901@behnel.de> References: <49B2CF4B.4010901@behnel.de> Message-ID: On Mar 7, 2009, at 11:47 AM, Stefan Behnel wrote: > Lisandro Dalcin wrote: >> Here you have the patch... Implemented as an (inline) function in a >> utility code > > Since it's never used for anything but string equality checks, it > would be > better to restrict it to that (i.e. 1/0), instead of using the normal > strcmp() -1,0,1 return values and additionally checking for "== 0" > everywhere in the code. Maybe call it "__Pyx_StrEqual()". +1 >> * The API import function uses a different thing, just a while >> loop. I >> do not want to complicate things in generated API headers for >> something that can be done with two auxiliar variables and a one-line >> while loop. > > Sure, good idea. > > >> * In the keyword handling code, I've used _PyString_Eq(), which is >> available in the C-API for Py2. If Python's C-API provides what we >> need, let's use it. Moreover, just in case, please see the >> implementation of _PyString_Eq(), it is smarter (and likely faster) >> than a strcmp() call ;-). > > Note that I already used (almost) the same optimisation in the > code, which > now becomes superfluous. Although I expect it to be a lot faster to > check > the length inside the loop before calling into the Python's C-API, > these > checks should be rare enough to just make that > > if (_PyString_Eq(**name, key)) ... > > The real fast-path is the loop with plain string pointer equality > checks > before all that. (BTW, note that this function is only available in > Py2, so > it's perfectly ok to use it here but not in general). > > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From dalcinl at gmail.com Sat Mar 7 21:28:09 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sat, 7 Mar 2009 17:28:09 -0300 Subject: [Cython] PATCH: removing strcmp() in favor of custom impl or other fixes In-Reply-To: <49B2CF4B.4010901@behnel.de> References: <49B2CF4B.4010901@behnel.de> Message-ID: On Sat, Mar 7, 2009 at 4:47 PM, Stefan Behnel wrote: > > Since it's never used for anything but string equality checks, it would be > better to restrict it to that (i.e. 1/0), instead of using the normal > strcmp() -1,0,1 return values and additionally checking for "== 0" > everywhere in the code. Maybe call it "__Pyx_StrEqual()". > OK. At first, I was going to impement it like that, but changed my mind (for no good reason) in the process. +1. > > Note that I already used (almost) the same optimisation in the code, which > now becomes superfluous. Although I expect it to be a lot faster to check > the length inside the loop before calling into the Python's C-API, these > checks should be rare enough to just make that > > ? ? ? ?if (_PyString_Eq(**name, key)) ... > OK. I did not give a full review of that part of the code. So then this _PyString_Eq() call will unlikely be ever made, right? > The real fast-path is the loop with plain string pointer equality checks > before all that. (BTW, note that this function is only available in Py2, so > it's perfectly ok to use it here but not in general). > Of couse. I've decided to use _PyString_Eq() just because you had already special-cased Python major version. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Sat Mar 7 21:31:40 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 07 Mar 2009 21:31:40 +0100 Subject: [Cython] PATCH: use function pointer arguments in import/export api functions In-Reply-To: References: <49A8595B.8040505@behnel.de> <43034.213.61.181.86.1236002896.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49B2D9AC.5050204@behnel.de> Hi Lisandro, sorry for the late reply, I didn't see your e-mail until now. Lisandro Dalcin wrote: > Stefan, here you have the new patch... This looks totally hackish, but ... > Check GCC manpages about -fstrict-aliasing, this should work. I did, they are actually presenting exactly this as an example that works. I guess that's the way to do it, then. Anyway, lxml compiles without aliasing warnings with your patch, so I'm ok with applying it. Stefan From stefan_ml at behnel.de Sat Mar 7 21:39:17 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 07 Mar 2009 21:39:17 +0100 Subject: [Cython] PATCH: removing strcmp() in favor of custom impl or other fixes In-Reply-To: References: <49B2CF4B.4010901@behnel.de> Message-ID: <49B2DB75.3050701@behnel.de> Lisandro Dalcin wrote: > On Sat, Mar 7, 2009 at 4:47 PM, Stefan Behnel wrote: >> Note that I already used (almost) the same optimisation in the code, which >> now becomes superfluous. Although I expect it to be a lot faster to check >> the length inside the loop before calling into the Python's C-API, these >> checks should be rare enough to just make that >> >> if (_PyString_Eq(**name, key)) ... > > OK. I did not give a full review of that part of the code. So then > this _PyString_Eq() call will unlikely be ever made, right? Yes. The most common cases are handled directly inside the function's own init code, so it's unlikely already that this function will be called in a time critical context. Then, most keyword names are inlined, so the fast-path at the beginning of the loop body will hit. If it does not, it's either the rare case that a non-inlined keyword name was passed, or one of the possible error cases. None of these it performance critical. Stefan From stefan_ml at behnel.de Sat Mar 7 22:22:18 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 07 Mar 2009 22:22:18 +0100 Subject: [Cython] Google summer of code In-Reply-To: References: Message-ID: <49B2E58A.9090900@behnel.de> Robert Bradshaw wrote: > Last summer we participated in Google's Summer of Code, which was a > great thing for the Cython project. I just wanted to start a thread > to coordinate our participation this next year. > > The timeline for mentoring organizations to apply is this this next > week--should we to try applying as a separate organization? I don't have much personal experience here, so it's hard to comment on what's better. I also don't think my personal life schedule permits mentorship. > Dag put up a great list of projects up at http://wiki.cython.org/ > codeprojects . An older (less refined) list is up at http:// > wiki.cython.org/enhancements . If anyone else has ideas, it would be > good to add to this site I further split up the "more numerical features" section. There is at least one other section in there that is described by ticket #211 and this page: http://wiki.cython.org/enhancements/parallel It's about parallel execution support. Making arrays a first-class type in Cython is one way of doing that. To me, that totally sounds like a very interesting (and not so small) project in its own right. Stefan From dagss at student.matnat.uio.no Sat Mar 7 23:24:42 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 7 Mar 2009 23:24:42 +0100 (CET) Subject: [Cython] Google summer of code In-Reply-To: <49B2E58A.9090900@behnel.de> References: <49B2E58A.9090900@behnel.de> Message-ID: <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> Stefan Behnel wrote: > > Robert Bradshaw wrote: >> Last summer we participated in Google's Summer of Code, which was a >> great thing for the Cython project. I just wanted to start a thread >> to coordinate our participation this next year. >> >> The timeline for mentoring organizations to apply is this this next >> week--should we to try applying as a separate organization? > > I don't have much personal experience here, so it's hard to comment on > what's better. I also don't think my personal life schedule permits > mentorship. Disadvantages: There's the work of submitting the actual application, which one shouldn't take on unless we know we want to participate. The number of possible mentors play a strong role here. Myself I was hoping to wait a month and see before I decide on whether I want to mentor. I hope (though nothing is decided!) to be able to mentor a student for a year-long university project on Cython here in Oslo, and mentoring for a full year of work will definitely have higher priority for me than two months (as the investment overhead in getting somebody to know coding in Cython is likely comparable in the two cases). That taken in consideration I now have second thoughts on mentoring in GSoC; the same amount of mentorship in autumn can potentially pay off more in amount of code written. So to get things declared, I'm unfortunately only a "maybe" on the mentorship side, at least right now. Dag Sverre From dagss at student.matnat.uio.no Sat Mar 7 23:37:11 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 7 Mar 2009 23:37:11 +0100 (CET) Subject: [Cython] Buffer syntax/array primitive type Message-ID: <22684a329c8ff577025d36e76274817f.squirrel@webmail.uio.no> It seems to be the right time to raise this issue now as we're talking future projects. I actually raised the issue on the NumPy list first last week so that some feedback from the numerical community is ready for whoever is interested right away. I think the sum-up is that something like this is needed if we want to take up a real fight with Fortran (though whether that is one of Cython's goals is always debatable). Anyway, I feel myself that this has a better feel than a new array type, as it makes use of the buffer interface and so bridges better with Python-land. My proposal is here: http://wiki.cython.org/enhancements/buffersyntax And the thread from the numpy list here: http://thread.gmane.org/gmane.comp.python.numeric.general/28439 Dag Sverre From robertwb at math.washington.edu Sun Mar 8 00:06:00 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 7 Mar 2009 15:06:00 -0800 Subject: [Cython] Buffer syntax/array primitive type In-Reply-To: <22684a329c8ff577025d36e76274817f.squirrel@webmail.uio.no> References: <22684a329c8ff577025d36e76274817f.squirrel@webmail.uio.no> Message-ID: On Mar 7, 2009, at 2:37 PM, Dag Sverre Seljebotn wrote: > It seems to be the right time to raise this issue now as we're talking > future projects. > > I actually raised the issue on the NumPy list first last week so > that some > feedback from the numerical community is ready for whoever is > interested > right away. I think the sum-up is that something like this is > needed if we > want to take up a real fight with Fortran (though whether that is > one of > Cython's goals is always debatable). > > Anyway, I feel myself that this has a better feel than a new array > type, > as it makes use of the buffer interface and so bridges better with > Python-land. > > My proposal is here: > http://wiki.cython.org/enhancements/buffersyntax > > And the thread from the numpy list here: > http://thread.gmane.org/gmane.comp.python.numeric.general/28439 Very interesting, and I like the syntax for the most part (int [:,:,"fortran"] seems a bit odd to me, perhaps mode should be a keyword). Can you pass them around/have them assigned to global/class-level variables? Will they be garbage collected? Also, how much of numpy will you be duplicating, and how easy will it be to turn one of these bare buffers into a numpy array? (I do like the easier int* <-> int [:] relation though.) - Robert From robertwb at math.washington.edu Sun Mar 8 00:11:59 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 7 Mar 2009 15:11:59 -0800 Subject: [Cython] Google summer of code In-Reply-To: <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> References: <49B2E58A.9090900@behnel.de> <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> Message-ID: <993FE1B3-30D3-45C7-9E40-B8C420E2492C@math.washington.edu> On Mar 7, 2009, at 2:24 PM, Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> >> Robert Bradshaw wrote: >>> Last summer we participated in Google's Summer of Code, which was a >>> great thing for the Cython project. I just wanted to start a thread >>> to coordinate our participation this next year. >>> >>> The timeline for mentoring organizations to apply is this this next >>> week--should we to try applying as a separate organization? >> >> I don't have much personal experience here, so it's hard to >> comment on >> what's better. I also don't think my personal life schedule permits >> mentorship. > > Disadvantages: There's the work of submitting the actual application, > which one shouldn't take on unless we know we want to participate. > > The number of possible mentors play a strong role here. > > Myself I was hoping to wait a month and see before I decide on > whether I > want to mentor. I hope (though nothing is decided!) to be able to > mentor a > student for a year-long university project on Cython here in Oslo, and > mentoring for a full year of work will definitely have higher > priority for > me than two months (as the investment overhead in getting somebody > to know > coding in Cython is likely comparable in the two cases). That taken in > consideration I now have second thoughts on mentoring in GSoC; the > same > amount of mentorship in autumn can potentially pay off more in > amount of > code written. > > So to get things declared, I'm unfortunately only a "maybe" on the > mentorship side, at least right now. I'm a "probably," but it's sounding like at this point we don't have enough potential mentors to be a full mentoring organization unless someone else stands up. On the numerical side of things, perhaps a NumPy person would be willing. On the other hand, I think we're higher on Python's radar than last year, so I'm hopeful of at least getting one (maybe two) spots for them. The quality of the students and projects will play a huge role here--one thing I learned at the summit last year is that it's not even worth you time to go after sub-par projects (though good students are an extremely good return on the investment of mentoring them). - Robert From dalcinl at gmail.com Sun Mar 8 00:15:03 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sat, 7 Mar 2009 20:15:03 -0300 Subject: [Cython] PATCH: use function pointer arguments in import/export api functions In-Reply-To: <49B2D9AC.5050204@behnel.de> References: <49A8595B.8040505@behnel.de> <43034.213.61.181.86.1236002896.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B2D9AC.5050204@behnel.de> Message-ID: Yes, it seems hackish, but I do not know of any other way, and other codes do also use this hackery... Please, could you push the fixes? On Sat, Mar 7, 2009 at 5:31 PM, Stefan Behnel wrote: > Hi Lisandro, > > sorry for the late reply, I didn't see your e-mail until now. > > Lisandro Dalcin wrote: >> Stefan, here you have the new patch... > > This looks totally hackish, but ... > >> Check GCC manpages about -fstrict-aliasing, this should work. > > I did, they are actually presenting exactly this as an example that works. > I guess that's the way to do it, then. > > Anyway, lxml compiles without aliasing warnings with your patch, so I'm ok > with applying it. > - Show quoted text - > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Sun Mar 8 00:40:09 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 7 Mar 2009 15:40:09 -0800 Subject: [Cython] PATCH: removing strcmp() in favor of custom impl or other fixes In-Reply-To: <49B2DB75.3050701@behnel.de> References: <49B2CF4B.4010901@behnel.de> <49B2DB75.3050701@behnel.de> Message-ID: On Mar 7, 2009, at 12:39 PM, Stefan Behnel wrote: > Lisandro Dalcin wrote: >> On Sat, Mar 7, 2009 at 4:47 PM, Stefan Behnel >> wrote: >>> Note that I already used (almost) the same optimisation in the >>> code, which >>> now becomes superfluous. Although I expect it to be a lot faster >>> to check >>> the length inside the loop before calling into the Python's C- >>> API, these >>> checks should be rare enough to just make that >>> >>> if (_PyString_Eq(**name, key)) ... >> >> OK. I did not give a full review of that part of the code. So then >> this _PyString_Eq() call will unlikely be ever made, right? > > Yes. The most common cases are handled directly inside the > function's own > init code, so it's unlikely already that this function will be > called in a > time critical context. Then, most keyword names are inlined, so the > fast-path at the beginning of the loop body will hit. If it does > not, it's > either the rare case that a non-inlined keyword name was passed, or > one of > the possible error cases. None of these it performance critical. Note I removed a strcmp() at http://hg.cython.org/cython-devel/rev/ cd51476651c6 which will make one chunk from your patch unnecessary (just a heads up when you merge). - Robert From dalcinl at gmail.com Sun Mar 8 02:06:20 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sat, 7 Mar 2009 22:06:20 -0300 Subject: [Cython] allowing archive downloads of cython-devel repo Message-ID: In case it works (not shure how cython-devel is actually served), could this be added to .hg/hgrc in cython-devel (under [web] section, as shown)? [web] allow_archive = gz, zip, bz2 -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Sun Mar 8 07:30:18 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 7 Mar 2009 22:30:18 -0800 Subject: [Cython] PyErr_Format, bare "%s" format specifier, and Py_TYPE(o)->tp_name In-Reply-To: References: Message-ID: I'm not opposed to this, though the only place I could see it being very useful is in case Py_TYPE(o)->tp_name doesn't really point to a string because something really got messed up. On Mar 2, 2009, at 1:08 PM, Lisandro Dalcin wrote: > After a fast view to Stefan's fix for #227, I noticed that Cython uses > bare "%s" format specifiers when using PyErr_Format() to print > Py_TYPE(o)->tp_name. > > Perhaps we should follow core CPython convention of using "%.200s" ?? > . Anyway, core CPython does not follow this rule at all places ;-) > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From prabhu at aero.iitb.ac.in Sun Mar 8 07:30:27 2009 From: prabhu at aero.iitb.ac.in (Prabhu Ramachandran) Date: Sun, 08 Mar 2009 12:00:27 +0530 Subject: [Cython] Buffer syntax/array primitive type In-Reply-To: <22684a329c8ff577025d36e76274817f.squirrel@webmail.uio.no> References: <22684a329c8ff577025d36e76274817f.squirrel@webmail.uio.no> Message-ID: <49B36603.6060108@aero.iitb.ac.in> On 03/08/09 04:07, Dag Sverre Seljebotn wrote: > Anyway, I feel myself that this has a better feel than a new array type, > as it makes use of the buffer interface and so bridges better with > Python-land. > > My proposal is here: > http://wiki.cython.org/enhancements/buffersyntax This looks really nice. I'm just a little curious about one little feature, would this support resizing the buffer? i.e. could I do def f(int[:] arr): cdef int i # ... arr.append(i) say? My guess is no but this would be really handy. I sometimes need a fast resizable container for basic types and find that using a list is about 4-7 times slower than a simple class exposing a block of memory with suitable resizing. Creating such a container is easy for one type but without a templating system this gets too messy when I want support for ints, longs, floats and doubles. cheers, prabhu From robertwb at math.washington.edu Sun Mar 8 07:38:52 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 7 Mar 2009 22:38:52 -0800 Subject: [Cython] allowing archive downloads of cython-devel repo In-Reply-To: References: Message-ID: Sure. Done. On Mar 7, 2009, at 5:06 PM, Lisandro Dalcin wrote: > In case it works (not shure how cython-devel is actually served), > could this be added to .hg/hgrc in cython-devel (under [web] section, > as shown)? > > [web] > allow_archive = gz, zip, bz2 > > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From robertwb at math.washington.edu Sun Mar 8 07:41:06 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 7 Mar 2009 22:41:06 -0800 Subject: [Cython] improving Py <-> C conversion of numeric types In-Reply-To: References: Message-ID: <2A716CBD-4D2E-488A-8FEF-AE14AA688C7A@math.washington.edu> On Feb 26, 2009, at 10:52 AM, Lisandro Dalcin wrote: > I'm working on a patch related to de subject of this mail, but I will > need any of you to make some coments on the above .... > > In [1]: from Cython.Compiler import PyrexTypes > > In [2]: PyrexTypes.c_anon_enum_type.rank > Out[2]: -1 > > In [3]: PyrexTypes.c_anon_enum_type.sign_and_name() > Out[3]: 'long double' > > > [2] seems fine, but what about [3] ?? Should we specialize the method > and make it return 'int' ?? Yes. Actually, isn't "sign_and_name" supposed to return the actual name of the enum? - Robert From stefan_ml at behnel.de Sun Mar 8 07:53:32 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 08 Mar 2009 07:53:32 +0100 Subject: [Cython] PyErr_Format, bare "%s" format specifier, and Py_TYPE(o)->tp_name In-Reply-To: References: Message-ID: <49B36B6C.2000300@behnel.de> Robert Bradshaw wrote: > On Mar 2, 2009, at 1:08 PM, Lisandro Dalcin wrote: > >> After a fast view to Stefan's fix for #227, I noticed that Cython uses >> bare "%s" format specifiers when using PyErr_Format() to print >> Py_TYPE(o)->tp_name. >> >> Perhaps we should follow core CPython convention of using "%.200s" ?? >> . Anyway, core CPython does not follow this rule at all places ;-) > > I'm not opposed to this, though the only place I could see it being > very useful is in case Py_TYPE(o)->tp_name doesn't really point to a > string because something really got messed up. Fine with me, too. Stefan From poonaatsoc at gmail.com Sun Mar 8 07:56:32 2009 From: poonaatsoc at gmail.com (Anoop Saldanha) Date: Sun, 8 Mar 2009 12:26:32 +0530 Subject: [Cython] Google summer of code In-Reply-To: <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> References: <49B2E58A.9090900@behnel.de> <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> Message-ID: On Sun, Mar 8, 2009 at 3:54 AM, Dag Sverre Seljebotn < dagss at student.matnat.uio.no> wrote: > Stefan Behnel wrote: > > > > Robert Bradshaw wrote: > >> Last summer we participated in Google's Summer of Code, which was a > >> great thing for the Cython project. I just wanted to start a thread > >> to coordinate our participation this next year. > >> > >> The timeline for mentoring organizations to apply is this this next > >> week--should we to try applying as a separate organization? > > > > I don't have much personal experience here, so it's hard to comment on > > what's better. I also don't think my personal life schedule permits > > mentorship. > > Disadvantages: There's the work of submitting the actual application, > which one shouldn't take on unless we know we want to participate. > > The number of possible mentors play a strong role here. > > Myself I was hoping to wait a month and see before I decide on whether I > want to mentor. I hope (though nothing is decided!) to be able to mentor a > student for a year-long university project on Cython here in Oslo, and > mentoring for a full year of work will definitely have higher priority for > me than two months (as the investment overhead in getting somebody to know > coding in Cython is likely comparable in the two cases). That taken in > consideration I now have second thoughts on mentoring in GSoC; the same > amount of mentorship in autumn can potentially pay off more in amount of > code written. > I was digging into cython currently, to work full time(as a developer) on it. I am not a student, but I would like to have your help in understanding cython better so that I can start contributing to cython. > > So to get things declared, I'm unfortunately only a "maybe" on the > mentorship side, at least right now. > > Dag Sverre > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Regards, Anoop S. People who says, "It's not winning or losing that matters, but participation", probably lost. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090308/aa6ad268/attachment-0001.htm From stefan_ml at behnel.de Sun Mar 8 08:04:05 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 08 Mar 2009 08:04:05 +0100 Subject: [Cython] Google summer of code In-Reply-To: References: <49B2E58A.9090900@behnel.de> <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> Message-ID: <49B36DE5.8010404@behnel.de> Hi, Anoop Saldanha wrote: > I was digging into cython currently, to work full time(as a developer) on > it. I am not a student, but I would like to have your help in understanding > cython better so that I can start contributing to cython. Help is always appreciated. Do you have any special interest in doing so? Any idea what part of Cython you would like to contribute to? That would make it easier to point you to a good entry point. Stefan From robertwb at math.washington.edu Sun Mar 8 08:17:46 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 7 Mar 2009 23:17:46 -0800 Subject: [Cython] Google summer of code In-Reply-To: References: <49B2E58A.9090900@behnel.de> <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> Message-ID: <7B956834-3610-4895-A215-786AF07980AF@math.washington.edu> On Mar 7, 2009, at 10:56 PM, Anoop Saldanha wrote: > I was digging into cython currently, to work full time(as a > developer) on it. I am not a student, but I would like to have > your help in understanding cython better so that I can start > contributing to cython. Very cool. What is your background? We'd be glad to help out, though none of us have the liberty of being full time developers :). - Robert From poonaatsoc at gmail.com Sun Mar 8 08:27:07 2009 From: poonaatsoc at gmail.com (Anoop Saldanha) Date: Sun, 8 Mar 2009 12:57:07 +0530 Subject: [Cython] Google summer of code In-Reply-To: <7B956834-3610-4895-A215-786AF07980AF@math.washington.edu> References: <49B2E58A.9090900@behnel.de> <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> <7B956834-3610-4895-A215-786AF07980AF@math.washington.edu> Message-ID: I am currently working for a company(to clear up the word working full-time :) ). Apart from my office work I can shell out around 8 hours everyday on my other work, which currently is digging into cython. I am an ex google summer of coder, though I doubt it really matters. On the coding front I am currently working with c and python. On Sun, Mar 8, 2009 at 12:47 PM, Robert Bradshaw < robertwb at math.washington.edu> wrote: > On Mar 7, 2009, at 10:56 PM, Anoop Saldanha wrote: > > > I was digging into cython currently, to work full time(as a > > developer) on it. I am not a student, but I would like to have > > your help in understanding cython better so that I can start > > contributing to cython. > > Very cool. What is your background? We'd be glad to help out, though > none of us have the liberty of being full time developers :). > > - Robert > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Regards, Anoop S. People who says, "It's not winning or losing that matters, but participation", probably lost. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090308/523ab16f/attachment.htm From stefan_ml at behnel.de Sun Mar 8 08:37:21 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 08 Mar 2009 08:37:21 +0100 Subject: [Cython] PATCH: use function pointer arguments in import/export api functions In-Reply-To: References: <49A8595B.8040505@behnel.de> <43034.213.61.181.86.1236002896.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B2D9AC.5050204@behnel.de> Message-ID: <49B375B1.7020200@behnel.de> Lisandro Dalcin wrote: > Yes, it seems hackish, but I do not know of any other way, and other > codes do also use this hackery... > > Please, could you push the fixes? It's up. Stefan From robertwb at math.washington.edu Sun Mar 8 10:00:12 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 8 Mar 2009 01:00:12 -0800 Subject: [Cython] Cython 0.11 release candidate Message-ID: Finally, it looks like we're on the brink of releasing 0.11. Try out the release candidate at http://cython.org/Cython-0.11.rc.tar.gz - Robert From dagss at student.matnat.uio.no Sun Mar 8 11:31:41 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 8 Mar 2009 11:31:41 +0100 (CET) Subject: [Cython] Buffer syntax/array primitive type In-Reply-To: References: <22684a329c8ff577025d36e76274817f.squirrel@webmail.uio.no> Message-ID: <1b5d42aff49cb2e508b937cdad2d0771.squirrel@webmail.uio.no> Robert Bradshaw wrote: > On Mar 7, 2009, at 2:37 PM, Dag Sverre Seljebotn wrote: >> My proposal is here: >> http://wiki.cython.org/enhancements/buffersyntax >> >> And the thread from the numpy list here: >> http://thread.gmane.org/gmane.comp.python.numeric.general/28439 > > Very interesting, and I like the syntax for the most part (int > [:,:,"fortran"] seems a bit odd to me, perhaps mode should be a > keyword). Consider though that @cython.locals(a=cython.int[:,:,"fortran"]) def foo(): ... is legal Python. Dropping Python compatability on a new syntax like this seems like a high price to pay. Perhaps cython.int[:,:,cython.buffer.FORTRAN_MODE] or similar? > Can you pass them around/have them assigned to global/class-level > variables? Will they be garbage collected? Also, how much of numpy Global/class-level was always the intention for current buffers as well, just never got around to it. So yes, it would just need to be done. Implementation side they can be considered (smaller variations on) Py_buffer structs. The variables themselves will (up to optimization) be structs which are copied by value, however they point to Python objects which will be refcounted in the process. a) They will hold a refcounted reference to an underlying object owning the memory, and b) they will (sometimes) need a seperate "acquisition refcount" so that when making a slice, the buffer will not be released until the last slice goes out of scope. All in all a happy mixture of copying by value and reference counting. > will you be duplicating, and how easy will it be to turn one of these > bare buffers into a numpy array? (I do like the easier int* <-> int > [:] relation though.) myarr = numpy.array(mybuf) This is currently used for conversion from lists etc. Implementation-wise this is rather simple (NumPy can support memoryview in newer releases/Python versions, and until then we can subclass memoryview to provide the NumPy __toarray__ protocol which numpy.array supports). How much duplication: Well at first nothing, but suppose the whole wanted-list is done (which depends on bringing in e.g. a student): All in all, it's more a matter of reimplementing small parts of Fortran than reimplementing NumPy. (I am actually toying with the idea of outputting and linking in Fortran code, but I don't think it would buy us much). a) Slices is pretty much duplication of functionality, but doing it compile-time, for speed, means it must be coded in a different way than in NumPy. b) The same goes for arithmetic. The NumPy one-operation-at-the-time approach isn't the most efficient one and if we decide to do this ourselves, it would be in order to do it in a different way, i.e. turn "a = b + sqrt(c)" into a full inline loop. c) Functions (log, sqrt, sin, etc.). Currently there's NumPy which only takes a whole array and C which only takes a single element. What one needs is functions which can be used on both levels, so that "a = sqrt(c)" can be used on buffer level but be turned by Cython into a loop calling sqrt on all elements individually. All in all, I think arithmetic on buffers needs pluggability to handle different backends anyway (does one have the commercial Intel MKL library? And so on.) At the cost of a very strong NumPy dependency (if this feature is used), one could of course implement this through NumPy at first, perhaps as one "backend". I.e. Cython-generated modules with buffer arithmetic would at first implicitly depend on NumPy (even if all you're trying to do is sum the contents of two C arrays). But I'm still thinking about possible solutions for the arithmetic part, it's a tricky part. Dag Sverre From dagss at student.matnat.uio.no Sun Mar 8 11:38:43 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 8 Mar 2009 11:38:43 +0100 (CET) Subject: [Cython] Buffer syntax/array primitive type In-Reply-To: <49B36603.6060108@aero.iitb.ac.in> References: <22684a329c8ff577025d36e76274817f.squirrel@webmail.uio.no> <49B36603.6060108@aero.iitb.ac.in> Message-ID: Prabhu Ramachandran wrote: > On 03/08/09 04:07, Dag Sverre Seljebotn wrote: >> Anyway, I feel myself that this has a better feel than a new array type, >> as it makes use of the buffer interface and so bridges better with >> Python-land. >> >> My proposal is here: >> http://wiki.cython.org/enhancements/buffersyntax > > This looks really nice. I'm just a little curious about one little > feature, would this support resizing the buffer? i.e. could I do > > def f(int[:] arr): > cdef int i > # ... > arr.append(i) No, this would never happen. This is defined as syntax candy for PEP 3118, which exports a view to allocated memory but doesn't say anything about how that memory is allocated or deallocated. I believe we must solve this, but in a different way. Given that we get a "fast list", wouldn't we then want a "fast dict"? Adding a type templating mechanism to Cython is not out of the question I think, there was some talk 3/4 year ago. But now I myself believe the most promising option is to wrap C++ in a very good way, so that one can do something along the lines of cdef cpp.vector[int] arr = cpp.vector[int]() cdef cpp.map[str, cpp.vector[int]] map = cpp.map[str, cpp.vector[int]]() arr.push_back(3) map[40] = arr (Yes, those lines raises about 10 questions about reference counting vs. copy-by-value semantics; making C++ access feel Pythonic is part of such a challenge IMO.) Dag Sverre From dagss at student.matnat.uio.no Sun Mar 8 11:43:17 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 8 Mar 2009 11:43:17 +0100 (CET) Subject: [Cython] Buffer syntax/array primitive type In-Reply-To: References: <22684a329c8ff577025d36e76274817f.squirrel@webmail.uio.no> <49B36603.6060108@aero.iitb.ac.in> Message-ID: <0351b79ec08c54e9c50dc2acb7c50abb.squirrel@webmail.uio.no> Dag Sverre Seljebotn wrote: > Adding a type templating mechanism to Cython is not out of the question I > think, there was some talk 3/4 year ago. But now I myself believe the most > promising option is to wrap C++ in a very good way, so that one can do > something along the lines of > > cdef cpp.vector[int] arr = cpp.vector[int]() > cdef cpp.map[str, cpp.vector[int]] map = cpp.map[str, cpp.vector[int]]() > arr.push_back(3) > map[40] = arr Sorry, that should have been map["mystring"] = arr. Dag Sverre From prabhu at aero.iitb.ac.in Sun Mar 8 12:15:05 2009 From: prabhu at aero.iitb.ac.in (Prabhu Ramachandran) Date: Sun, 08 Mar 2009 16:45:05 +0530 Subject: [Cython] Buffer syntax/array primitive type In-Reply-To: References: <22684a329c8ff577025d36e76274817f.squirrel@webmail.uio.no> <49B36603.6060108@aero.iitb.ac.in> Message-ID: <49B3A8B9.4030208@aero.iitb.ac.in> On 03/08/09 16:08, Dag Sverre Seljebotn wrote: >> def f(int[:] arr): >> cdef int i >> # ... >> arr.append(i) > > No, this would never happen. This is defined as syntax candy for PEP 3118, > which exports a view to allocated memory but doesn't say anything about > how that memory is allocated or deallocated. That is what I had imagined but hope springs eternal. > I believe we must solve this, but in a different way. Given that we get a > "fast list", wouldn't we then want a "fast dict"? Indeed, that would be the next obvious step. :) > Adding a type templating mechanism to Cython is not out of the question I > think, there was some talk 3/4 year ago. But now I myself believe the most > promising option is to wrap C++ in a very good way, so that one can do > something along the lines of > > cdef cpp.vector[int] arr = cpp.vector[int]() > cdef cpp.map[str, cpp.vector[int]] map = cpp.map[str, cpp.vector[int]]() > arr.push_back(3) > map[40] = arr I guess that would work too. > (Yes, those lines raises about 10 questions about reference counting vs. > copy-by-value semantics; making C++ access feel Pythonic is part of such a > challenge IMO.) Indeed and all those would have to be handled very carefully. A cheap templating system would be of course to do string substitutions and spit out the template instances into a pyx file using Python -- along the lines of numpy's own templating with the .src files. A more integrated solution would of course be really convenient. cheers, prabhu From stefan_ml at behnel.de Sun Mar 8 14:33:57 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 08 Mar 2009 14:33:57 +0100 Subject: [Cython] Cython 0.11 release candidate In-Reply-To: References: Message-ID: <49B3C945.7020102@behnel.de> Hi, Robert Bradshaw wrote: > Finally, it looks like we're on the brink of releasing 0.11. Try out > the release candidate at http://cython.org/Cython-0.11.rc.tar.gz If this represents cython-devel up to the last fix (838a6b7cae62) for Lisandros warnings, I'm fine with the release. I just cleaned up a couple of more tickets in trac and added some test cases for them. I don't care if they go in or not, but it would be good to have them in the release branch at least, as we might fix some of them in the 0.11.x life cycle. Stefan From ndbecker2 at gmail.com Sun Mar 8 15:19:20 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Sun, 08 Mar 2009 10:19:20 -0400 Subject: [Cython] Cython 0.11 release candidate References: Message-ID: Robert Bradshaw wrote: > Finally, it looks like we're on the brink of releasing 0.11. Try out > the release candidate at http://cython.org/Cython-0.11.rc.tar.gz > > - Robert Running on Fedora f10: running tests against Cython 0.11.rc Python 2.5.2 (r252:60911, Sep 30 2008, 15:42:03) [GCC 4.3.2 20080917 (Red Hat 4.3.2-4)] Traceback (most recent call last): File "runtests.py", line 559, in pyxbuild_dir=os.path.join(WORKDIR, "support")) File "/home/nbecker/RPM/BUILD/Cython-0.11.rc/pyximport/pyxbuild.py", line 26, in pyx_to_dll assert os.path.exists(filename) AssertionError From psp.reachable at gmail.com Sun Mar 8 16:10:29 2009 From: psp.reachable at gmail.com (Prajwal Suhas P) Date: Sun, 8 Mar 2009 20:40:29 +0530 Subject: [Cython] checking out the repo Message-ID: I need to checkout the cython trunk from the repository. I have checked out the check-devel repo hg clone http://hg.cython.org/cython-devel/ check-devel Is the check-devel repo, the trunk for cython devel? I saw 3-4 repos for cython. I saw a similar mail in the mailing list archive, but found no reply to that. Have I checked out the right repo? -- Regards, Prajwal S. "There is little difference in people,but this little difference makes a big difference.The little difference is attitude,the big difference is whether its positive or negative. " -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090308/718f3e6b/attachment.htm From stefan_ml at behnel.de Sun Mar 8 16:38:49 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 08 Mar 2009 16:38:49 +0100 Subject: [Cython] checking out the repo In-Reply-To: References: Message-ID: <49B3E689.2050708@behnel.de> Prajwal Suhas P wrote: > I need to checkout the cython trunk from the repository. I have checked out > the check-devel repo That's the right one. There's "cython", which has the latest official release (and possibly some additional fixes), and "cython-devel", which has the latest development version (i.e. the "trunk"). "cython-docs" has the official documentation. Other repositories come up at need. Stefan From dalcinl at gmail.com Mon Mar 9 01:13:45 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sun, 8 Mar 2009 21:13:45 -0300 Subject: [Cython] improving Py <-> C conversion of numeric types In-Reply-To: <2A716CBD-4D2E-488A-8FEF-AE14AA688C7A@math.washington.edu> References: <2A716CBD-4D2E-488A-8FEF-AE14AA688C7A@math.washington.edu> Message-ID: On Sun, Mar 8, 2009 at 3:41 AM, Robert Bradshaw wrote: > On Feb 26, 2009, at 10:52 AM, Lisandro Dalcin wrote: >> >> [2] seems fine, but what about [3] ?? Should we specialize the method >> and make it return 'int' ?? > > Yes. Actually, isn't "sign_and_name" supposed to return the actual > name of the enum? > But an anon enum has no name! -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From sturla at molden.no Mon Mar 9 13:01:58 2009 From: sturla at molden.no (Sturla Molden) Date: Mon, 09 Mar 2009 13:01:58 +0100 Subject: [Cython] Buffer syntax/array primitive type In-Reply-To: References: <22684a329c8ff577025d36e76274817f.squirrel@webmail.uio.no> Message-ID: <49B50536.9020708@molden.no> On 3/8/2009 12:06 AM, Robert Bradshaw wrote: > Very interesting, and I like the syntax for the most part (int > [:,:,"fortran"] seems a bit odd to me, perhaps mode should be a > keyword). I wonder if you would consider cdef int[::,:] arr to indicate that a dimension might have strides in the first dimension different from 1. And similar cdef int[:,::] arr cdef int[::,::] arr That would allow you to ignore strides and speed up indexing for most arrays. The most common case is unit strides. S.M. From stefan_ml at behnel.de Mon Mar 9 13:17:51 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 9 Mar 2009 13:17:51 +0100 (CET) Subject: [Cython] Better call optimisation Message-ID: <36891.213.61.181.86.1236601071.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Hi, there are a couple of tickets related to call optimisations. http://trac.cython.org/cython_trac/ticket/3 http://trac.cython.org/cython_trac/ticket/131 http://trac.cython.org/cython_trac/ticket/166 The getattr3() problem also relates to other cases, such as the fact that we can't currently optimise cdef list l = [...] l.sort() into a call to PyList_Sort(), as this would break calls like l.sort(key=...) My suggestion is to remove the current support for optimised methods of builtins, and to write a dedicated transform that intercepts the tree on call nodes and that can then handle all cases in explicit code. This can include, for example, adding a None check if required, or handling different call signatures only some of which can be optimised, maybe even by changing keyword arguments into positional arguments of a matching C-API function. I imagine that it would use some kind of dispatch mechanism on (obj_type, obj_attribute_name) and on builtin function names. I also think that the current generic "X.append()" optimisation (i.e. not only the one for typed lists) should go there. This might(!) depend on (or at least benefit from) splitting up the type analysis phase (as discussed before, looks like we still need a ticket for this), so that we can recognise the declared/builtin names when running the transform, and can handle the final return values appropriately (which are sometimes boolean ints instead of (None) Python objects, for example) *after* switching to the corresponding C-API function. Opinions? Stefan From dalcinl at gmail.com Mon Mar 9 13:24:14 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 9 Mar 2009 10:24:14 -0200 Subject: [Cython] Better call optimisation In-Reply-To: <36891.213.61.181.86.1236601071.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <36891.213.61.181.86.1236601071.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: On Mon, Mar 9, 2009 at 10:17 AM, Stefan Behnel wrote: > > My suggestion is to remove the current support for optimised methods of > builtins, and to write a dedicated transform that intercepts the tree on > call nodes and that can then handle all cases in explicit code. > > Opinions? > Definitely +1 for me ... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Mon Mar 9 13:50:19 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 9 Mar 2009 10:50:19 -0200 Subject: [Cython] too late to review and push patch for better handling of integral C types ? Message-ID: Sending again, former message bounced because of 40k limit. The patch is almost ready, just waiting for review. The only thing I did not take is to remove the usage of PyNumber_Int(). Anyway, Cython already uses it, so the "brokeness" of my patch is similar to the current one :-) .. As I've introduced a __Pyx_PyNumber_Int() function, that is the only place that we should change to implement things as we want. I would really love to get this in for 0.11 ... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 -------------- next part -------------- A non-text attachment was scrubbed... Name: NUMERICTYPES.diff Type: application/octet-stream Size: 18755 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090309/a5fe7d7f/attachment-0001.obj From dalcinl at gmail.com Mon Mar 9 13:51:20 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 9 Mar 2009 10:51:20 -0200 Subject: [Cython] too late to review and push patch for better handling of integral C types ? In-Reply-To: References: Message-ID: And here you have the testcase... On Mon, Mar 9, 2009 at 10:50 AM, Lisandro Dalcin wrote: > Sending again, former message bounced because of 40k limit. > > The patch is almost ready, just waiting for review. The only thing I > did not take is to remove the usage of PyNumber_Int(). Anyway, Cython > already uses it, so the "brokeness" of my patch is similar to the > current one :-) .. As I've introduced a __Pyx_PyNumber_Int() function, > that is the only place that we should change to implement things as we > want. > > I would really love to get this in for 0.11 ... > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 -------------- next part -------------- A non-text attachment was scrubbed... Name: c_int_types.pyx Type: application/octet-stream Size: 16020 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090309/f5db079c/attachment.obj From dagss at student.matnat.uio.no Mon Mar 9 14:49:51 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 09 Mar 2009 14:49:51 +0100 Subject: [Cython] Buffer syntax/array primitive type In-Reply-To: <49B50536.9020708@molden.no> References: <22684a329c8ff577025d36e76274817f.squirrel@webmail.uio.no> <49B50536.9020708@molden.no> Message-ID: <49B51E7F.9090409@student.matnat.uio.no> Sturla Molden wrote: > On 3/8/2009 12:06 AM, Robert Bradshaw wrote: > > >> Very interesting, and I like the syntax for the most part (int >> [:,:,"fortran"] seems a bit odd to me, perhaps mode should be a >> keyword). >> > > I wonder if you would consider > > cdef int[::,:] arr > > to indicate that a dimension might have strides in the first dimension > different from 1. And similar > > cdef int[:,::] arr > cdef int[::,::] arr > > That would allow you to ignore strides and speed up indexing for most > arrays. The most common case is unit strides. > Hmm. Nice idea. I would turn it the other way around though; cdef int[:,::1] for C ordering. Note the explicit "1" stride; I think "::" should always mean the same as ":" or there will be confusion. I think the "faster, more special case" route should always require more syntax knowledge than the "marginally slower generic case". The typical fast access will then look like cdef int[0:,0::1] Next: Would it be possible to incorporate other modes in a similar way? cdef int[::contiguous,::contiguous,::1] could mean C-contiguous 3D-array? What about full mode (i.e. where one on an arbitrary dimension can have pointer dereferences...) perhaps int[::(1,ptr),::contiguous,::1] to get a layout where the first dimension in a contiguous array of pointers to C-ordered 2D contiguous data? :-) All in all, the buffer PEP contains huge flexibility, and locking that flexibility down means more speed. Whether anyone will use it is another issue :-) I suppose int[:,::1] is enough for now. Long-term wishful thinking digression: In typical applications it may be that the compiler can make the decision as to whether one should copy or access in strided mode. For instance: def foo(int[:,:] arr): ... foo(myarr) # Contiguous use foo(myarr[:,::-2]) # Non-contiguous use Here, the compiler has essentially three options: a) "What it says" b) Wrap foo in a copy-in, copy-out wrapper, and also skip that wrapper in the middle line, and indeed if all uses use a contiguous array, skip the wrapper c) Generate one contiguous and one non-contiguous This is just to show the lengths it would be possible to go to. It also gives me an eerie feeling that replacing Fortran (which had had ages to think about stuff like this) will not be easy... Dag Sverre From stefan_ml at behnel.de Mon Mar 9 15:21:07 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 9 Mar 2009 15:21:07 +0100 (CET) Subject: [Cython] too late to review and push patch for better handling of integral C types ? In-Reply-To: References: Message-ID: <35193.213.61.181.86.1236608467.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Lisandro Dalcin wrote: > The patch is almost ready, just waiting for review. > [...] > I would really love to get this in for 0.11 ... Without having looked at it: we are past the RC, so unless someone finds the time for a thorough review and test, I would hate breaking 0.11 right with the first release. Stefan From dagss at student.matnat.uio.no Mon Mar 9 18:34:00 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 09 Mar 2009 18:34:00 +0100 Subject: [Cython] too late to review and push patch for better handling of integral C types ? Message-ID: <3319468485.617624@smtp.netcom.no> +1; there's plenty of bugs which are not fixed and will not be fixed for 0.11; the RC should only point out regressions, things that used to work. There's nothing stopping us from putting out 0.11.1 in another week's time. Dag Sverre Seljebotn -----Original Message----- From: "Stefan Behnel" Date: Monday, Mar 9, 2009 3:21 pm Subject: Re: [Cython] too late to review and push patch for better handling of integral C types ? To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net Lisandro Dalcin wrote: > The patch is almost ready, just waiting for review. > [...] > I would really love to get this in for 0.11 ... > >Without having looked at it: we are past the RC, so unless someone finds the time for a thorough review and test, I would hate breaking 0.11 right with the first release. > >Stefan > >_______________________________________________ >Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > From robertwb at math.washington.edu Mon Mar 9 18:39:33 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 9 Mar 2009 10:39:33 -0700 Subject: [Cython] too late to review and push patch for better handling of integral C types ? In-Reply-To: <35193.213.61.181.86.1236608467.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <35193.213.61.181.86.1236608467.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: On Mar 9, 2009, at 7:21 AM, Stefan Behnel wrote: > Lisandro Dalcin wrote: >> The patch is almost ready, just waiting for review. >> [...] >> I would really love to get this in for 0.11 ... > > Without having looked at it: we are past the RC, so unless someone > finds > the time for a thorough review and test, I would hate breaking 0.11 > right > with the first release. This could be the first patch for 0.11.1, but let's finally get 0.11 out the door. Unless someone reports in the next 24 hours or so that somethings horribly broken, that's the plan. - Robert From stefan_ml at behnel.de Mon Mar 9 18:50:42 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 09 Mar 2009 18:50:42 +0100 Subject: [Cython] Cython 0.11 release candidate In-Reply-To: References: Message-ID: <49B556F2.1020008@behnel.de> Hi, thanks for the report. Neal Becker wrote: > Running on Fedora f10: > > running tests against Cython 0.11.rc > Python 2.5.2 (r252:60911, Sep 30 2008, 15:42:03) > [GCC 4.3.2 20080917 (Red Hat 4.3.2-4)] > > Traceback (most recent call last): > File "runtests.py", line 559, in > pyxbuild_dir=os.path.join(WORKDIR, "support")) > File "/home/nbecker/RPM/BUILD/Cython-0.11.rc/pyximport/pyxbuild.py", line > 26, in pyx_to_dll > assert os.path.exists(filename) > AssertionError I don't think this qualifies as release critical. Could you file a bug report against 0.11.1 and provide some more information? You seem to be using an RPM, is this during the RPM build (running the tests for some reason) or did you start the tests after installing the RPM, maybe from a different directory? Please add a couple things like that to make this reproducible or at least traceable. You may need a login for trac, just send a mail to Robert Bradshaw containing a htpasswd entry. Stefan From dagss at student.matnat.uio.no Mon Mar 9 19:47:52 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 09 Mar 2009 19:47:52 +0100 Subject: [Cython] Better call optimisation In-Reply-To: <36891.213.61.181.86.1236601071.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <36891.213.61.181.86.1236601071.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49B56458.4030505@student.matnat.uio.no> Stefan Behnel wrote: > Hi, > > there are a couple of tickets related to call optimisations. > > http://trac.cython.org/cython_trac/ticket/3 > http://trac.cython.org/cython_trac/ticket/131 > http://trac.cython.org/cython_trac/ticket/166 > > The getattr3() problem also relates to other cases, such as the fact that > we can't currently optimise > > cdef list l = [...] > l.sort() > > into a call to PyList_Sort(), as this would break calls like > > l.sort(key=...) > > My suggestion is to remove the current support for optimised methods of > builtins, and to write a dedicated transform that intercepts the tree on > call nodes and that can then handle all cases in explicit code. This can > include, for example, adding a None check if required, or handling > different call signatures only some of which can be optimised, maybe even > by changing keyword arguments into positional arguments of a matching > C-API function. > > I imagine that it would use some kind of dispatch mechanism on > > (obj_type, obj_attribute_name) > > and on builtin function names. I also think that the current generic > "X.append()" optimisation (i.e. not only the one for typed lists) should > go there. Sounds good! One thing: I think it would be good if the output is something very targeted (i.e. not simply new SimpleCallNodes or similar). From the refnanny it seemed that very many CPython API calls used the same pattern, so perhaps something along the lines of a PythonApiCall-node, perhaps with a NoneCheckNode as its "subject", and so on. (Some of the resulting operations could warrant their own node, while others are similar enough to not warrant new classes for each -- the point is that I think it should be more "targeted" than new SimpleCallNodes). > This might(!) depend on (or at least benefit from) splitting up the type > analysis phase (as discussed before, looks like we still need a ticket for > this), so that we can recognise the declared/builtin names when running > the transform, and can handle the final return values appropriately (which > are sometimes boolean ints instead of (None) Python objects, for example) > *after* switching to the corresponding C-API function. This is a going to cause at least as much trouble as the new temps (though that is going to be worth it in the end, and so could this). At least myself I am not decided on how I'd like to see such a split -- I know I advocated it a year ago, but now "first names, then expressions" seems a bit suboptimal to me. "First declare possible types, then transform, then resolve and insert coercion nodes and raise any errors" seems promising but I'm still unsure about it. If there's a chance of a Cython workshop during the coming year I'd say these kind of matters (and the actual coding) can be resolved more easily that way. Otherwise it seems to stand between one person with a bright idea doing it all alone, or one of those 100-emails-threads converging very slowly towards the right solution... But I may be overestimating the complexity, if you have a clear idea then I won't object to it. -- Dag Sverre From ndbecker2 at gmail.com Mon Mar 9 19:49:05 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 09 Mar 2009 14:49:05 -0400 Subject: [Cython] Cython 0.11 release candidate References: <49B556F2.1020008@behnel.de> Message-ID: Stefan Behnel wrote: > Hi, > > thanks for the report. > > Neal Becker wrote: >> Running on Fedora f10: >> >> running tests against Cython 0.11.rc >> Python 2.5.2 (r252:60911, Sep 30 2008, 15:42:03) >> [GCC 4.3.2 20080917 (Red Hat 4.3.2-4)] >> >> Traceback (most recent call last): >> File "runtests.py", line 559, in >> pyxbuild_dir=os.path.join(WORKDIR, "support")) >> File "/home/nbecker/RPM/BUILD/Cython-0.11.rc/pyximport/pyxbuild.py", >> line >> 26, in pyx_to_dll >> assert os.path.exists(filename) >> AssertionError > > I don't think this qualifies as release critical. Could you file a bug > report against 0.11.1 and provide some more information? You seem to be > using an RPM, is this during the RPM build (running the tests for some > reason) or did you start the tests after installing the RPM, maybe from a > different directory? Please add a couple things like that to make this > reproducible or at least traceable. > > You may need a login for trac, just send a mail to Robert Bradshaw > containing a htpasswd entry. > > Stefan I am the maintainer for cython on fedora. This was during the check phase (%check). If the test doesn't pass, at least I'll have to turn off the checking, which would be unfortunate. Are the tests expected to pass? From robertwb at math.washington.edu Mon Mar 9 19:56:52 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 9 Mar 2009 11:56:52 -0700 Subject: [Cython] Cython 0.11 release candidate In-Reply-To: References: <49B556F2.1020008@behnel.de> Message-ID: <994EFBF5-7848-4929-A5B4-B3A050386C6D@math.washington.edu> It's this test here: out_fname = pyx_to_dll("foo.pyx") It looks like we don't even ship a foo.pyx to test with, though I'm not sure why I haven't seen this error before... On Mar 9, 2009, at 11:49 AM, Neal Becker wrote: > Stefan Behnel wrote: > >> Hi, >> >> thanks for the report. >> >> Neal Becker wrote: >>> Running on Fedora f10: >>> >>> running tests against Cython 0.11.rc >>> Python 2.5.2 (r252:60911, Sep 30 2008, 15:42:03) >>> [GCC 4.3.2 20080917 (Red Hat 4.3.2-4)] >>> >>> Traceback (most recent call last): >>> File "runtests.py", line 559, in >>> pyxbuild_dir=os.path.join(WORKDIR, "support")) >>> File "/home/nbecker/RPM/BUILD/Cython-0.11.rc/pyximport/ >>> pyxbuild.py", >>> line >>> 26, in pyx_to_dll >>> assert os.path.exists(filename) >>> AssertionError >> >> I don't think this qualifies as release critical. Could you file a >> bug >> report against 0.11.1 and provide some more information? You seem >> to be >> using an RPM, is this during the RPM build (running the tests for >> some >> reason) or did you start the tests after installing the RPM, maybe >> from a >> different directory? Please add a couple things like that to make >> this >> reproducible or at least traceable. >> >> You may need a login for trac, just send a mail to Robert Bradshaw >> containing a htpasswd entry. >> >> Stefan > > I am the maintainer for cython on fedora. This was during the > check phase > (%check). If the test doesn't pass, at least I'll have to turn off > the > checking, which would be unfortunate. > > Are the tests expected to pass? > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From stefan_ml at behnel.de Mon Mar 9 19:58:11 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 09 Mar 2009 19:58:11 +0100 Subject: [Cython] Cython 0.11 release candidate In-Reply-To: References: <49B556F2.1020008@behnel.de> Message-ID: <49B566C3.8010203@behnel.de> Neal Becker wrote: > I am the maintainer for cython on fedora. Ah, ok, that makes it more important. > This was during the check phase (%check). I assume this happens after building? > If the test doesn't pass, at least I'll have to turn off the > checking, which would be unfortunate. > > Are the tests expected to pass? Yes. This looks like the refnanny can't be built. You can disable it using "--no-refannny", although I would prefer a real fix. Did you check if the directory in which you run the tests has a file "Cython/Runtime/refnanny.pyx"? Actually, when Cython is tested after building, the refnanny should already be ready for use, so there shouldn't be a reason to build it again. Stefan From stefan_ml at behnel.de Mon Mar 9 20:02:19 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 09 Mar 2009 20:02:19 +0100 Subject: [Cython] Cython 0.11 release candidate In-Reply-To: <49B566C3.8010203@behnel.de> References: <49B556F2.1020008@behnel.de> <49B566C3.8010203@behnel.de> Message-ID: <49B567BB.3020102@behnel.de> Stefan Behnel wrote: > Did you check if the > directory in which you run the tests has a file "Cython/Runtime/refnanny.pyx"? Ah, does this patch fix it? diff -r 474a2405291b MANIFEST.in --- a/MANIFEST.in Sun Mar 08 14:18:27 2009 +0100 +++ b/MANIFEST.in Mon Mar 09 20:01:52 2009 +0100 @@ -7,6 +7,7 @@ include cython.py include Cython/Compiler/Lexicon.pickle include Cython/Includes/*.pxd +recursive-include Cython *.pyx include Doc/* include Demos/*.pyx Stefan From dagss at student.matnat.uio.no Mon Mar 9 20:19:25 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 09 Mar 2009 20:19:25 +0100 Subject: [Cython] Cython 0.11 release candidate In-Reply-To: <49B566C3.8010203@behnel.de> References: <49B556F2.1020008@behnel.de> <49B566C3.8010203@behnel.de> Message-ID: <49B56BBD.5010303@student.matnat.uio.no> Stefan Behnel wrote: > Actually, when Cython is tested after building, the refnanny should already > be ready for use, so there shouldn't be a reason to build it again. Unless something's changed since I did it, runtests.py always maintains its own compilation of refnanny in BUILD (to make refnanny development easier). It doesn't rebuild it if it is already present in BUILD with a newer timestamp than source though. -- Dag Sverre From robertwb at math.washington.edu Mon Mar 9 20:22:59 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 9 Mar 2009 12:22:59 -0700 Subject: [Cython] Better call optimisation In-Reply-To: <36891.213.61.181.86.1236601071.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <36891.213.61.181.86.1236601071.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: On Mar 9, 2009, at 5:17 AM, Stefan Behnel wrote: > Hi, > > there are a couple of tickets related to call optimisations. > > http://trac.cython.org/cython_trac/ticket/3 > http://trac.cython.org/cython_trac/ticket/131 > http://trac.cython.org/cython_trac/ticket/166 > > The getattr3() problem also relates to other cases, such as the > fact that > we can't currently optimise > > cdef list l = [...] > l.sort() > > into a call to PyList_Sort(), as this would break calls like > > l.sort(key=...) > > My suggestion is to remove the current support for optimised > methods of > builtins, and to write a dedicated transform that intercepts the > tree on > call nodes and that can then handle all cases in explicit code. > This can > include, for example, adding a None check if required, or handling > different call signatures only some of which can be optimised, > maybe even > by changing keyword arguments into positional arguments of a matching > C-API function. This sounds like a good idea. Another option would be to support polymorphism (which would be good to do, at least for external C functions). > I imagine that it would use some kind of dispatch mechanism on > > (obj_type, obj_attribute_name) > > and on builtin function names. I also think that the current generic > "X.append()" optimisation (i.e. not only the one for typed lists) > should > go there. > > This might(!) depend on (or at least benefit from) splitting up the > type > analysis phase (as discussed before, looks like we still need a > ticket for > this), so that we can recognise the declared/builtin names when > running > the transform, and can handle the final return values appropriately > (which > are sometimes boolean ints instead of (None) Python objects, for > example) > *after* switching to the corresponding C-API function. > > Opinions? My one concern is that it would make type inference significantly more difficult by deferring this to a later phase (especially thinking in terms of methods that don't return PyObject*). - Robert From ndbecker2 at gmail.com Mon Mar 9 20:26:18 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 09 Mar 2009 15:26:18 -0400 Subject: [Cython] Cython 0.11 release candidate References: <49B556F2.1020008@behnel.de> <49B566C3.8010203@behnel.de> <49B56BBD.5010303@student.matnat.uio.no> Message-ID: Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Actually, when Cython is tested after building, the refnanny should >> already be ready for use, so there shouldn't be a reason to build it >> again. > > Unless something's changed since I did it, runtests.py always maintains > its own compilation of refnanny in BUILD (to make refnanny development > easier). It doesn't rebuild it if it is already present in BUILD with a > newer timestamp than source though. > One thing that might be a clue: The rpm build procedure will do: python setup.py build python setup.py install -O1 --skip-build --root $RPM_BUILD_ROOT python runtests.py From stefan_ml at behnel.de Mon Mar 9 20:30:52 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 09 Mar 2009 20:30:52 +0100 Subject: [Cython] Better call optimisation In-Reply-To: <49B56458.4030505@student.matnat.uio.no> References: <36891.213.61.181.86.1236601071.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B56458.4030505@student.matnat.uio.no> Message-ID: <49B56E6C.8000404@behnel.de> Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: > One thing: I think it would be good if the output is something very > targeted (i.e. not simply new SimpleCallNodes or similar). From the > refnanny it seemed that very many CPython API calls used the same > pattern, so perhaps something along the lines of a PythonApiCall-node, > perhaps with a NoneCheckNode as its "subject", and so on. Fine with me. I also thought about a "NoneCheckNode" when I wrote this. That would be a very simple and explicit thing, and also very easy to handle (or optimise away) in later transformations and during code generation. A PythonApiCall node might be a good idea (needs some more thinking) and would relieve us from having to generate all sorts of Entry's & friends. > Some of the resulting operations could warrant their own node Yes, I gave an example with the int vs. None return value. There's also the issue that many methods of builtins have their own specific exceptions when passing None values. This needs to be handled, and a PythonApiCall node would be a good place to do it. >> This might(!) depend on (or at least benefit from) splitting up the type >> analysis phase (as discussed before, looks like we still need a ticket for >> this), so that we can recognise the declared/builtin names when running >> the transform, and can handle the final return values appropriately (which >> are sometimes boolean ints instead of (None) Python objects, for example) >> *after* switching to the corresponding C-API function. > > This is a going to cause at least as much trouble as the new temps > (though that is going to be worth it in the end, and so could this). At > least myself I am not decided on how I'd like to see such a split -- I > know I advocated it a year ago, but now "first names, then expressions" > seems a bit suboptimal to me. "First declare possible types, then > transform, then resolve and insert coercion nodes and raise any errors" > seems promising but I'm still unsure about it. Yes, this needs some discussion. For 0.12, we may just try the optimisation and see how far it takes us without this split. That will also make it clearer if, why and how this needs to be done. > If there's a chance of a Cython workshop during the coming year I'd say > these kind of matters (and the actual coding) can be resolved more > easily that way. Otherwise it seems to stand between one person with a > bright idea doing it all alone, or one of those 100-emails-threads > converging very slowly towards the right solution... ;) Certainly not a task for a GSoC student. Stefan From stefan_ml at behnel.de Mon Mar 9 20:32:26 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 09 Mar 2009 20:32:26 +0100 Subject: [Cython] Cython 0.11 release candidate In-Reply-To: References: <49B556F2.1020008@behnel.de> <49B566C3.8010203@behnel.de> <49B56BBD.5010303@student.matnat.uio.no> Message-ID: <49B56ECA.3040106@behnel.de> Neal Becker wrote: > Dag Sverre Seljebotn wrote: > >> Stefan Behnel wrote: >>> Actually, when Cython is tested after building, the refnanny should >>> already be ready for use, so there shouldn't be a reason to build it >>> again. >> Unless something's changed since I did it, runtests.py always maintains >> its own compilation of refnanny in BUILD (to make refnanny development >> easier). It doesn't rebuild it if it is already present in BUILD with a >> newer timestamp than source though. > > One thing that might be a clue: > > The rpm build procedure will do: > > python setup.py build > python setup.py install -O1 --skip-build --root $RPM_BUILD_ROOT > python runtests.py Yes, I think the patch that I just pushed to cython-devel might fix this. Please give it a try. Stefan From stefan_ml at behnel.de Mon Mar 9 22:05:48 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 09 Mar 2009 22:05:48 +0100 Subject: [Cython] too late to review and push patch for better handling of integral C types ? In-Reply-To: References: Message-ID: <49B584AC.6040506@behnel.de> Lisandro Dalcin wrote: > Sending again, former message bounced because of 40k limit. > > The patch is almost ready, just waiting for review. The only thing I > did not take is to remove the usage of PyNumber_Int(). Anyway, Cython > already uses it, so the "brokeness" of my patch is similar to the > current one :-) .. As I've introduced a __Pyx_PyNumber_Int() function, > that is the only place that we should change to implement things as we > want. I skipped through your patch a little. It's a fairly involved change that requires some testing, especially on different architectures and platforms. It might well be that the user experience is positive in that it helps in finding bugs and making user code more portable. My current gut feeling is that code that depends on the old behaviour here is broken anyway and just wasn't fixed yet because no-one noticed. But in any case, I would like to avoid breaking code for a micro (bug-fix!) release, unless there is at least some evidence that users appreciate it. So, if anyone has code that does a bit of integer conversion between Python and C space, especially when working on a non-Intel or 64 bit platform, please give this patch a try. Stefan From ndbecker2 at gmail.com Tue Mar 10 00:12:23 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 09 Mar 2009 19:12:23 -0400 Subject: [Cython] Cython 0.11 release candidate References: <49B556F2.1020008@behnel.de> <49B566C3.8010203@behnel.de> <49B56BBD.5010303@student.matnat.uio.no> <49B56ECA.3040106@behnel.de> Message-ID: Stefan Behnel wrote: > > Neal Becker wrote: >> Dag Sverre Seljebotn wrote: >> >>> Stefan Behnel wrote: >>>> Actually, when Cython is tested after building, the refnanny should >>>> already be ready for use, so there shouldn't be a reason to build it >>>> again. >>> Unless something's changed since I did it, runtests.py always maintains >>> its own compilation of refnanny in BUILD (to make refnanny development >>> easier). It doesn't rebuild it if it is already present in BUILD with a >>> newer timestamp than source though. >> >> One thing that might be a clue: >> >> The rpm build procedure will do: >> >> python setup.py build >> python setup.py install -O1 --skip-build --root $RPM_BUILD_ROOT >> python runtests.py > > Yes, I think the patch that I just pushed to cython-devel might fix this. > Please give it a try. > > Stefan Sorry, I don't follow devel that closely. How do I get the patch, or an updated source? From ndbecker2 at gmail.com Tue Mar 10 02:06:34 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 09 Mar 2009 21:06:34 -0400 Subject: [Cython] Cython 0.11 release candidate References: <49B556F2.1020008@behnel.de> <49B566C3.8010203@behnel.de> <49B567BB.3020102@behnel.de> Message-ID: Stefan Behnel wrote: > > Stefan Behnel wrote: >> Did you check if the >> directory in which you run the tests has a file >> "Cython/Runtime/refnanny.pyx"? > > Ah, does this patch fix it? > > diff -r 474a2405291b MANIFEST.in > --- a/MANIFEST.in Sun Mar 08 14:18:27 2009 +0100 > +++ b/MANIFEST.in Mon Mar 09 20:01:52 2009 +0100 > @@ -7,6 +7,7 @@ > include cython.py > include Cython/Compiler/Lexicon.pickle > include Cython/Includes/*.pxd > +recursive-include Cython *.pyx > include Doc/* > > include Demos/*.pyx > > Stefan No, this patch made no difference. python setup.py install -O1 --skip-build --root ~/RPM/BUILDROOT/Cython-0.11-1rc.fc1\ 0.x86_64/ Compiling module Cython.Plex.Scanners ... Compiling module Cython.Compiler.Scanning ... Compiling module Cython.Compiler.Parsing ... Compiling module Cython.Compiler.Visitor ... Compiling module Cython.Runtime.refpython setup.py build Compiling module Cython.Plex.Scanners ... Compiling module Cython.Compiler.Scanning ... Compiling module Cython.Compiler.Parsing ... Compiling module Cython.Compiler.Visitor ... Compiling module Cython.Runtime.refnanny ... ERROR: [Errno 2] No such file or directory: u'/home/nbecker/RPM/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx' Extension module compilation failed, using plain Python implementation running build running build_py running build_scripts python setup.py install -O1 --skip-build --root ~/RPM/BUILDROOT/Cython-0.11-1rc.fc10.x86_64/ Compiling module Cython.Plex.Scanners ... nanny ... ERROR: [Errno 2] No such file or directory: u'/home/nbecker/RPM/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx' Extension module compilation failed, using plain Python implementation running install running install_lib writing byte-compilation script '/tmp/tmpTN0AmD.py' /usr/bin/python -O /tmp/tmpTN0AmD.py removing /tmp/tmpTN0AmD.py running install_scripts changing mode of /home/nbecker/RPM/BUILDROOT/Cython-0.11-1rc.fc10.x86_64/usr/bin/cython to 755 running install_egg_info Removing /home/nbecker/RPM/BUILDROOT/Cython-0.11-1rc.fc10.x86_64/usr/lib/python2.5/site- packages/Cython-0.11.rc-py2.5.egg-info Writing /home/nbecker/RPM/BUILDROOT/Cython-0.11-1rc.fc10.x86_64/usr/lib/python2.5/site- packages/Cython-0.11.rc-py2.5.egg-info $ python runtests.py Running tests against Cython 0.11.rc Python 2.5.2 (r252:60911, Sep 30 2008, 15:42:03) [GCC 4.3.2 20080917 (Red Hat 4.3.2-4)] Traceback (most recent call last): File "runtests.py", line 559, in pyxbuild_dir=os.path.join(WORKDIR, "support")) File "/home/nbecker/RPM/BUILD/Cython-0.11.rc/pyximport/pyxbuild.py", line 26, in pyx_to_dll assert os.path.exists(filename) AssertionError From robertwb at math.washington.edu Tue Mar 10 03:50:33 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 9 Mar 2009 19:50:33 -0700 Subject: [Cython] Cython 0.11 release candidate In-Reply-To: References: Message-ID: <872A0AAF-4DBA-4471-A081-9013738CA234@math.washington.edu> On Mar 8, 2009, at 7:19 AM, Neal Becker wrote: > Robert Bradshaw wrote: > >> Finally, it looks like we're on the brink of releasing 0.11. Try out >> the release candidate at http://cython.org/Cython-0.11.rc.tar.gz >> >> - Robert > > Running on Fedora f10: > > running tests against Cython 0.11.rc > Python 2.5.2 (r252:60911, Sep 30 2008, 15:42:03) > [GCC 4.3.2 20080917 (Red Hat 4.3.2-4)] > > Traceback (most recent call last): > File "runtests.py", line 559, in > pyxbuild_dir=os.path.join(WORKDIR, "support")) > File "/home/nbecker/RPM/BUILD/Cython-0.11.rc/pyximport/ > pyxbuild.py", line > 26, in pyx_to_dll > assert os.path.exists(filename) > AssertionError OK, could you try pulling from http://hg.cython.org/cython-devel/ again? (Or alternatively I've updated http://cython.org/ Cython-0.11.rc.tar.gz ). It won't fix your error, but should help diagnose it better. - Robert From stefan_ml at behnel.de Tue Mar 10 10:23:19 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 10 Mar 2009 10:23:19 +0100 Subject: [Cython] Cython 0.11 release candidate In-Reply-To: References: <49B556F2.1020008@behnel.de> <49B566C3.8010203@behnel.de> <49B567BB.3020102@behnel.de> Message-ID: <49B63187.2060009@behnel.de> Neal Becker wrote: > python setup.py install -O1 --skip-build --root > ~/RPM/BUILDROOT/Cython-0.11-1rc.fc1\ > 0.x86_64/ > Compiling module Cython.Plex.Scanners ... > Compiling module Cython.Compiler.Scanning ... > Compiling module Cython.Compiler.Parsing ... > Compiling module Cython.Compiler.Visitor ... > Compiling module Cython.Runtime.refpython setup.py build This looks weird, are you sure you copied this correctly? > Compiling module Cython.Plex.Scanners ... > Compiling module Cython.Compiler.Scanning ... > Compiling module Cython.Compiler.Parsing ... > Compiling module Cython.Compiler.Visitor ... > Compiling module Cython.Runtime.refnanny ... > ERROR: [Errno 2] No such file or directory: > u'/home/nbecker/RPM/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx' > Extension module compilation failed, using plain Python implementation > running build > running build_py > running build_scripts > > python setup.py install -O1 --skip-build --root > ~/RPM/BUILDROOT/Cython-0.11-1rc.fc10.x86_64/ > Compiling module Cython.Plex.Scanners ... And this seems to be where things continue: > nanny ... > ERROR: [Errno 2] No such file or directory: > u'/home/nbecker/RPM/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx' > Extension module compilation failed, using plain Python implementation > running install > [...] I actually think that it's the "install" that hits us here. The refnanny.pyx isn't copied on installation, as we expect the .so file to be built already. However, you call "install" with the "--skip-build" option, which prevents the build of the shared library, thus preventing the refnanny from being available at all. We might just declare the refnanny.pyx file as package data, so that it's copied into the installation. It doesn't hurt to have it lying next to the refnanny.so file. Here's another patch. BTW, you can pass the "--no-cython-compile" option to your "setup.py install" call to keep it from compiling certain source files multiple times. It's well enough if it does that once on "setup.py build". Stefan -------------- next part -------------- A non-text attachment was scrubbed... Name: package_data.patch Type: text/x-patch Size: 1479 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090310/a338b292/attachment.bin From ndbecker2 at gmail.com Tue Mar 10 13:51:32 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 10 Mar 2009 08:51:32 -0400 Subject: [Cython] Cython 0.11 release candidate References: <49B556F2.1020008@behnel.de> <49B566C3.8010203@behnel.de> <49B567BB.3020102@behnel.de> <49B63187.2060009@behnel.de> Message-ID: Both patches applied. Still no good. I am sending the whole build transcript. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: cython.build Url: http://codespeak.net/pipermail/cython-dev/attachments/20090310/878404a2/attachment-0001.diff From stefan_ml at behnel.de Tue Mar 10 14:29:31 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 10 Mar 2009 14:29:31 +0100 (CET) Subject: [Cython] Cython 0.11 release candidate In-Reply-To: References: <49B556F2.1020008@behnel.de> <49B566C3.8010203@behnel.de> <49B567BB.3020102@behnel.de> <49B63187.2060009@behnel.de> Message-ID: <40045.213.61.181.86.1236691771.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Hi, Neal Becker wrote: > Both patches applied. Still no good. I am sending the whole build > transcript. can't check right now, but did you verify that the tar ball has the file Cython/Runtime/refnanny.pyx? The first patch might have prevented it from getting into the release. Robert, in case it's missing, could you provide Neal with a new tar ball from the latest cython-devel? Stefan From stefan_ml at behnel.de Tue Mar 10 14:31:13 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 10 Mar 2009 14:31:13 +0100 (CET) Subject: [Cython] Cython 0.11 release candidate In-Reply-To: <40045.213.61.181.86.1236691771.squirrel@groupware.dvs.informatik.tu-d armstadt.de> References: <49B556F2.1020008@behnel.de> <49B566C3.8010203@behnel.de> <49B567BB.3020102@behnel.de> <49B63187.2060009@behnel.de> <40045.213.61.181.86.1236691771.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <41995.213.61.181.86.1236691873.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Stefan Behnel wrote: > Cython/Runtime/refnanny.pyx? The first patch might have prevented it from > getting into the release. ... actually not "the first patch", but the bug that I patched with my first change... Stefan From dagss at student.matnat.uio.no Tue Mar 10 14:33:50 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 10 Mar 2009 14:33:50 +0100 Subject: [Cython] Cython 0.11 release candidate In-Reply-To: <40045.213.61.181.86.1236691771.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49B556F2.1020008@behnel.de> <49B566C3.8010203@behnel.de> <49B567BB.3020102@behnel.de> <49B63187.2060009@behnel.de> <40045.213.61.181.86.1236691771.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49B66C3E.6090508@student.matnat.uio.no> Stefan Behnel wrote: > Hi, > > Neal Becker wrote: > >> Both patches applied. Still no good. I am sending the whole build >> transcript. >> > > can't check right now, but did you verify that the tar ball has the file > Cython/Runtime/refnanny.pyx? The first patch might have prevented it from > getting into the release. > The RC tarball has refnanny.pyx present in the right location. Dag Sverre From ndbecker2 at gmail.com Tue Mar 10 15:11:08 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 10 Mar 2009 10:11:08 -0400 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) Message-ID: I grabbed a NEW tar from http://cython.org/Cython-0.11.rc.tar.gz. (I won't comment on the confusion caused by changing the file without changing the name :) Now testing at least starts, but seems to have some errors: (Ignore the RPM build errors: Installed (but unpackaged) file(s) found:, I'll fix those) -------------- next part -------------- A non-text attachment was scrubbed... Name: cython.build.gz Type: application/x-gzip Size: 12174 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090310/5b5a8f35/attachment.bin From dagss at student.matnat.uio.no Tue Mar 10 15:18:11 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 10 Mar 2009 15:18:11 +0100 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: References: Message-ID: <49B676A3.3090405@student.matnat.uio.no> Neal Becker wrote: > I grabbed a NEW tar from http://cython.org/Cython-0.11.rc.tar.gz. (I won't > comment on the confusion caused by changing the file without changing the > name :) > > Now testing at least starts, but seems to have some errors: > (Ignore the > RPM build errors: > Installed (but unpackaged) file(s) found:, > I'll fix those) > The problem is that NumPy appears to be installed on your system (import numpy succeeds), yet the NumPy C header files is not present in the include path. Either set an extra include-path using the INCLUDE environment variable: INCLUDE=/path/to/numpy/headers python runtests.py ... or, drop the NumPy tests from the test suite (as I suppose your Cython RPM doesn't depend on NumPy -- it shouldn't!): python runtests.py ... -x numpy Dag Sverre From dalcinl at gmail.com Tue Mar 10 15:18:08 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 10 Mar 2009 12:18:08 -0200 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: References: Message-ID: I really recomend Cython to include a "setup.cfg" file in the root dir (I mean, alongside the setup.py script) whit at least the lines below: [sdist] force_manifest=1 [install] optimize = 1 - The fist would solve common gotchas of out-of-date MANIFEST and the rpm build problems... - The second would solve the "Installed (but unpackaged) file(s) found", wich is related to the generation of *.pyo files... On Tue, Mar 10, 2009 at 12:11 PM, Neal Becker wrote: > I grabbed a NEW tar from http://cython.org/Cython-0.11.rc.tar.gz. ?(I won't > comment on the confusion caused by changing the file without changing the > name :) > > Now testing at least starts, but seems to have some errors: > (Ignore the > RPM build errors: > ? ?Installed (but unpackaged) file(s) found:, > I'll fix those) > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Tue Mar 10 15:21:54 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 10 Mar 2009 12:21:54 -0200 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: <49B676A3.3090405@student.matnat.uio.no> References: <49B676A3.3090405@student.matnat.uio.no> Message-ID: Dag, from very long time (as far as I can remember), numpy provided a canonical way to get the include files import numpy numpy_include = numpy.get_include() I remember some mails in this list where (Stefan or you?) resisted on automating this... But this has worked even before 1.0.X numpy released... IMHO, this should be automated ... On Tue, Mar 10, 2009 at 12:18 PM, Dag Sverre Seljebotn wrote: > Neal Becker wrote: >> I grabbed a NEW tar from http://cython.org/Cython-0.11.rc.tar.gz. ?(I won't >> comment on the confusion caused by changing the file without changing the >> name :) >> >> Now testing at least starts, but seems to have some errors: >> (Ignore the >> RPM build errors: >> ? ? Installed (but unpackaged) file(s) found:, >> I'll fix those) >> > > The problem is that NumPy appears to be installed on your system (import > numpy succeeds), yet the NumPy C header files is not present in the > include path. > > Either set an extra include-path using the INCLUDE environment variable: > > INCLUDE=/path/to/numpy/headers python runtests.py ... > > or, drop the NumPy tests from the test suite (as I suppose your Cython > RPM doesn't depend on NumPy -- it shouldn't!): > > python runtests.py ... -x numpy > > Dag Sverre > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From ndbecker2 at gmail.com Tue Mar 10 15:22:34 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 10 Mar 2009 10:22:34 -0400 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) References: <49B676A3.3090405@student.matnat.uio.no> Message-ID: Dag Sverre Seljebotn wrote: > Neal Becker wrote: >> I grabbed a NEW tar from http://cython.org/Cython-0.11.rc.tar.gz. (I >> won't comment on the confusion caused by changing the file without >> changing the name :) >> >> Now testing at least starts, but seems to have some errors: >> (Ignore the >> RPM build errors: >> Installed (but unpackaged) file(s) found:, >> I'll fix those) >> > > The problem is that NumPy appears to be installed on your system (import > numpy succeeds), yet the NumPy C header files is not present in the > include path. > > Either set an extra include-path using the INCLUDE environment variable: > > INCLUDE=/path/to/numpy/headers python runtests.py ... > > or, drop the NumPy tests from the test suite (as I suppose your Cython > RPM doesn't depend on NumPy -- it shouldn't!): > > python runtests.py ... -x numpy > > Dag Sverre Here is code used to find the 'pyublas' includes. Something like this could be used to find the numpy includes: def pyublas_inc_path(): from imp import find_module file, pathname, descr = find_module("pyublas") from os.path import join return join(pathname, "..", "include") From stefan_ml at behnel.de Tue Mar 10 15:36:53 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 10 Mar 2009 15:36:53 +0100 (CET) Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: References: Message-ID: <52637.213.61.181.86.1236695813.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Neal Becker wrote: > I grabbed a NEW tar from http://cython.org/Cython-0.11.rc.tar.gz. (I > won't > comment on the confusion caused by changing the file without changing the > name :) Since you don't do it, I will: it's usually a very bad idea to replace a release (even a release candidate) with a modified version without changing the name, as it tends to generate a lot of confusion as to what people are reporting bugs about. I actually know what I'm talking about. ;-) > Now testing at least starts, but seems to have some errors: * warning: left shift count >= width of type I don't get these locally. The problem is in __Pyx_GetItemGeneric, second line: "1ULL << (sizeof(Py_ssize_t)*8)". Looks like that's not 64bit clean. * arrayassign.c: In function "__pyx_pf_11arrayassign_test_literal_list_slice_start_param": arrayassign.c:775: warning: format "%d" expects type "int", but argument 4 has type "Py_ssize_t" arrayassign.cpp: In function "PyObject* __pyx_pf_11arrayassign_test_literal_list_slice_start_end_param(PyObject*, PyObject*, PyObject*)": arrayassign.cpp:1056: warning: format "%d" expects type "int", but argument 4 has type "long int" Same thing, 64 bit issues IMHO. * Error converting Pyrex file to C: ------------------------------------------------------------ ... cdef object[foo2=short unsigned int] x ^ ------------------------------------------------------------ Cython_Compiler_Tests_TestBuffer_TestBufferParsing_test_notype_as_expr1:1:23: Expected: expression Error converting Pyrex file to C: ------------------------------------------------------------ ... cdef object[int, short unsigned int] x ^ ------------------------------------------------------------ Cython_Compiler_Tests_TestBuffer_TestBufferParsing_test_notype_as_expr2:1:23: Expected: expression Error converting Pyrex file to C: ------------------------------------------------------------ ... cdef object[foo=1, 2] x ^ ------------------------------------------------------------ Cython_Compiler_Tests_TestBuffer_TestBufferParsing_test_pos_after_key:1:19: Non-keyword arg following keyword arg *** I remember seeing these, but they went away when rebuilding Cython. That's why I'm surprised to see them in a clean build now... Dag, any idea where those might come from? Stefan From dagss at student.matnat.uio.no Tue Mar 10 15:37:20 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 10 Mar 2009 15:37:20 +0100 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: References: <49B676A3.3090405@student.matnat.uio.no> Message-ID: <49B67B20.7060807@student.matnat.uio.no> Neal Becker wrote: > Dag Sverre Seljebotn wrote: > > >> Neal Becker wrote: >> >>> I grabbed a NEW tar from http://cython.org/Cython-0.11.rc.tar.gz. (I >>> won't comment on the confusion caused by changing the file without >>> changing the name :) >>> >>> Now testing at least starts, but seems to have some errors: >>> (Ignore the >>> RPM build errors: >>> Installed (but unpackaged) file(s) found:, >>> I'll fix those) >>> >>> >> The problem is that NumPy appears to be installed on your system (import >> numpy succeeds), yet the NumPy C header files is not present in the >> include path. >> >> Either set an extra include-path using the INCLUDE environment variable: >> >> INCLUDE=/path/to/numpy/headers python runtests.py ... >> >> or, drop the NumPy tests from the test suite (as I suppose your Cython >> RPM doesn't depend on NumPy -- it shouldn't!): >> >> python runtests.py ... -x numpy >> >> Dag Sverre >> > > Here is code used to find the 'pyublas' includes. Something like this could > be used to find the numpy includes: > Yep, but it won't happen in this release. See my upcoming response to Lisandro. But I think the Cython RPM shouldn't depend on the NumPy RPM, or be tested differently if NumPy is installed or not in the RPM build? Cython doesn't depend on NumPy in any way, there's just a test case sitting there more for convenience than anything else...the testcase doesn't add anything to correctness testing of Cython either. Dag Sverre From stefan_ml at behnel.de Tue Mar 10 15:44:22 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 10 Mar 2009 15:44:22 +0100 (CET) Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: References: Message-ID: <61769.213.61.181.86.1236696262.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Neal Becker wrote: > I grabbed a NEW tar from http://cython.org/Cython-0.11.rc.tar.gz. > [...] > Now testing at least starts, but seems to have some errors: BTW, Neal, getting the RPM build working on a 64bit machine appears to be the perfect test harness for the 0.11 release. Thanks for speaking up in the right moment! Stefan From dagss at student.matnat.uio.no Tue Mar 10 15:45:36 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 10 Mar 2009 15:45:36 +0100 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: References: <49B676A3.3090405@student.matnat.uio.no> Message-ID: <49B67D10.4060308@student.matnat.uio.no> Lisandro Dalcin wrote: > Dag, from very long time (as far as I can remember), numpy provided a > canonical way to get the include files > > import numpy > numpy_include = numpy.get_include() > > I remember some mails in this list where (Stefan or you?) resisted on > automating this... But this has worked even before 1.0.X numpy > released... > > IMHO, this should be automated ... > I'll resist again, but I don't have a strong opinion, so if you can persuade Stefan and provide a patch I'm ok with it. a) One could add it to runtests.py. But that is primarily a tool for development, and the most important thing is that things are nice and explicit and that nothing magic happens under the hood. It's not supposed to "just work", rather the opposite! :-) b) One could add it to Cython distutils, so that if NumPy is installed, Cython modules always have its include path added. I don't really have an opinion here, but my feeling is that that would be a rather strange and unexpected special treatment of this one library. It would be better to push for a *proper* solution: Either have NumPy install a symlink in /usr/include or the Python headers, or have Python and/or distutils keep som kind of include path registry for include paths to installed extensions. Dag Sverre From stefan_ml at behnel.de Tue Mar 10 15:53:38 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 10 Mar 2009 15:53:38 +0100 (CET) Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: <49B67D10.4060308@student.matnat.uio.no> References: <49B676A3.3090405@student.matnat.uio.no> <49B67D10.4060308@student.matnat.uio.no> Message-ID: <40279.213.61.181.86.1236696818.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: >> Dag, from very long time (as far as I can remember), numpy provided a >> canonical way to get the include files >> >> import numpy >> numpy_include = numpy.get_include() >> >> I remember some mails in this list where (Stefan or you?) resisted on >> automating this... But this has worked even before 1.0.X numpy >> released... >> >> IMHO, this should be automated ... >> > I'll resist again, but I don't have a strong opinion, so if you can > persuade Stefan and provide a patch I'm ok with it. > > a) One could add it to runtests.py. But that is primarily a tool for > development, and the most important thing is that things are nice and > explicit and that nothing magic happens under the hood. It's not > supposed to "just work", rather the opposite! :-) I'm fine with using it in runtest.py. We test for NumPy anyway when setting up the test suite, so we can just as well go all the way. Requiring a user to set up a specific build environment just to run the test suite is wrong. I'm against adding any special handling to the distutils modules, though. Having NumPy installed doesn't imply that you want to use its header files. Stefan From ndbecker2 at gmail.com Tue Mar 10 15:56:35 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 10 Mar 2009 10:56:35 -0400 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) References: <49B676A3.3090405@student.matnat.uio.no> <49B67B20.7060807@student.matnat.uio.no> Message-ID: Dag Sverre Seljebotn wrote: > Neal Becker wrote: >> Dag Sverre Seljebotn wrote: >> >> >>> Neal Becker wrote: >>> >>>> I grabbed a NEW tar from http://cython.org/Cython-0.11.rc.tar.gz. (I >>>> won't comment on the confusion caused by changing the file without >>>> changing the name :) >>>> >>>> Now testing at least starts, but seems to have some errors: >>>> (Ignore the >>>> RPM build errors: >>>> Installed (but unpackaged) file(s) found:, >>>> I'll fix those) >>>> >>>> >>> The problem is that NumPy appears to be installed on your system (import >>> numpy succeeds), yet the NumPy C header files is not present in the >>> include path. >>> >>> Either set an extra include-path using the INCLUDE environment variable: >>> >>> INCLUDE=/path/to/numpy/headers python runtests.py ... >>> >>> or, drop the NumPy tests from the test suite (as I suppose your Cython >>> RPM doesn't depend on NumPy -- it shouldn't!): >>> >>> python runtests.py ... -x numpy >>> >>> Dag Sverre >>> >> >> Here is code used to find the 'pyublas' includes. Something like this >> could be used to find the numpy includes: >> > Yep, but it won't happen in this release. See my upcoming response to > Lisandro. > > But I think the Cython RPM shouldn't depend on the NumPy RPM, or be > tested differently if NumPy is installed or not in the RPM build? > > Cython doesn't depend on NumPy in any way, there's just a test case > sitting there more for convenience than anything else...the testcase > doesn't add anything to correctness testing of Cython either. > > Dag Sverre OK, I agree. I will use -x numpy and we avoid BuildRequires: numpy. With this, the build succeeds (on x86_64 F10). Anyone test this yet on gcc-4.4 (which is used for F11)? From dagss at student.matnat.uio.no Tue Mar 10 16:00:00 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 10 Mar 2009 16:00:00 +0100 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: <52637.213.61.181.86.1236695813.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <52637.213.61.181.86.1236695813.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49B68070.6040509@student.matnat.uio.no> Stefan Behnel wrote: > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > cdef object[foo=1, 2] x > ^ > ------------------------------------------------------------ > > Cython_Compiler_Tests_TestBuffer_TestBufferParsing_test_pos_after_key:1:19: > Non-keyword arg following keyword arg > *** > > I remember seeing these, but they went away when rebuilding Cython. That's > why I'm surprised to see them in a clean build now... > > Dag, any idea where those might come from? > Hmm. At least it does not indicate errors outside of the test framework. These are *expected and required* errors, from Cython.Compiler.Tests.TestBuffer.TestBufferParsing. What is checked is that these errors are raised. The problem is that obviously the parser logs the exception somewhere in this case; probably in addition to raising the exception which is probably properly trapped. On my machine this is done cleanly: dagss at corcaroli ~/cython/devel $ python runtests.py TestBuffer -vv Running tests against Cython 0.11.rc Python 2.5.2 (r252:60911, Jul 22 2008, 12:44:54) [GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] test_basic (Cython.Compiler.Tests.TestBuffer.TestBufferParsing) ... ok test_notype_as_expr1 (Cython.Compiler.Tests.TestBuffer.TestBufferParsing) ... ok test_notype_as_expr2 (Cython.Compiler.Tests.TestBuffer.TestBufferParsing) ... ok test_pos_after_key (Cython.Compiler.Tests.TestBuffer.TestBufferParsing) ... ok test_type_keyword (Cython.Compiler.Tests.TestBuffer.TestBufferParsing) ... ok test_type_pos (Cython.Compiler.Tests.TestBuffer.TestBufferParsing) ... ok ---------------------------------------------------------------------- Ran 6 tests in 0.012s OK Might be unittest version and logging policy differences? Don't have time looking deeper into it yet. Dag Sverre From dalcinl at gmail.com Tue Mar 10 16:23:38 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 10 Mar 2009 13:23:38 -0200 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: <40279.213.61.181.86.1236696818.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49B676A3.3090405@student.matnat.uio.no> <49B67D10.4060308@student.matnat.uio.no> <40279.213.61.181.86.1236696818.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: I just asked for automating the numpy include in runtest.py !! I have numpy installed, I run the testsuite, and almost all times I forget to setenv INCLUDE, thus the numpy test fails... If the numpy test is enabled by default depending on numpy being installed, why should manually setup my environment? That's an nonsense, and even more when numpy provides a canonical, long-time supported way to get their include directory ... Regarding hacking distutils for the heck of numpy support, please no! That would be a very bad thing... Anyway, anyone using numpy can all "numpy.get_include()" to easily get the numpy include directory, it is really easy.... On Tue, Mar 10, 2009 at 12:53 PM, Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> Lisandro Dalcin wrote: >>> Dag, from very long time (as far as I can remember), numpy provided a >>> canonical way to get the include files >>> >>> import numpy >>> numpy_include = numpy.get_include() >>> >>> I remember some mails in this list where (Stefan or you?) resisted on >>> automating this... But this has worked even before 1.0.X numpy >>> released... >>> >>> IMHO, this should be automated ... >>> >> I'll resist again, but I don't have a strong opinion, so if you can >> persuade Stefan and provide a patch I'm ok with it. >> >> a) One could add it to runtests.py. But that is primarily a tool for >> development, and the most important thing is that things are nice and >> explicit and that nothing magic happens under the hood. It's not >> supposed to "just work", rather the opposite! :-) > > I'm fine with using it in runtest.py. We test for NumPy anyway when > setting up the test suite, so we can just as well go all the way. > Requiring a user to set up a specific build environment just to run the > test suite is wrong. > > I'm against adding any special handling to the distutils modules, though. > Having NumPy installed doesn't imply that you want to use its header > files. > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Tue Mar 10 16:29:20 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 10 Mar 2009 16:29:20 +0100 (CET) Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: References: <49B676A3.3090405@student.matnat.uio.no> <49B67D10.4060308@student.matnat.uio.no> <40279.213.61.181.86.1236696818.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <50816.213.61.181.86.1236698960.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Lisandro Dalcin wrote: > On Tue, Mar 10, 2009 at 12:53 PM, Stefan Behnel wrote: >> I'm fine with using it in runtest.py. We test for NumPy anyway when >> setting up the test suite, so we can just as well go all the way. >> Requiring a user to set up a specific build environment just to run the >> test suite is wrong. >> >> I'm against adding any special handling to the distutils modules, >> though. >> Having NumPy installed doesn't imply that you want to use its header >> files. > > I just asked for automating the numpy include in runtest.py !! I have > numpy installed, I run the testsuite, and almost all times I forget > to setenv INCLUDE, thus the numpy test fails... If the numpy test is > enabled by default depending on numpy being installed, why should > manually setup my environment? That's an nonsense, and even more when > numpy provides a canonical, long-time supported way to get their > include directory ... > > Regarding hacking distutils for the heck of numpy support, please no! > That would be a very bad thing... Anyway, anyone using numpy can all > "numpy.get_include()" to easily get the numpy include directory, it is > really easy.... Thanks for rephrasing my thoughts. :) Please supply a fix for runtests.py. Stefan From ndbecker2 at gmail.com Tue Mar 10 16:27:12 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 10 Mar 2009 11:27:12 -0400 Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 Message-ID: This is on fedora rawhide, x86_64: + /usr/bin/python runtests.py -x numpy Running tests against Cython 0.11.rc Python 2.6 (r26:66714, Feb 26 2009, 17:32:04) [GCC 4.4.0 20090219 (Red Hat 4.4.0-0.21)] warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:62:6: 'report_unraisable' redeclared warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:84:17: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:87:18: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:87:32: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:87:47: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:96:13: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:98:13: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:98:33: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:101:18: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:101:32: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:101:47: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:110:25: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:112:25: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:112:45: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:115:18: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:115:32: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:115:47: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:122:34: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:127:38: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:135:18: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:136:15: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:136:42: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:142:14: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:144:18: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:144:32: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:144:47: No conversion from python_exc.PyObject * to Python object, python object pointer used. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:90:5: Unraisable exception in function 'refnanny.GOTREF'. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:103:5: Unraisable exception in function 'refnanny.GIVEREF_and_report'. warning: /builddir/build/BUILD/Cython-0.11.rc/Cython/Runtime/refnanny.pyx:129:5: Unraisable exception in function 'refnanny.FinishContext'. Error converting Pyrex file to C: ------------------------------------------------------------ ... cdef object[foo2=short unsigned int] x ^ ------------------------------------------------------------ Cython_Compiler_Tests_TestBuffer_TestBufferParsing_test_notype_as_expr1:1:23: Expected: expression Error converting Pyrex file to C: ------------------------------------------------------------ ... cdef object[int, short unsigned int] x ^ ------------------------------------------------------------ Cython_Compiler_Tests_TestBuffer_TestBufferParsing_test_notype_as_expr2:1:23: Expected: expression Error converting Pyrex file to C: ------------------------------------------------------------ ... cdef object[foo=1, 2] x ^ ------------------------------------------------------------ Cython_Compiler_Tests_TestBuffer_TestBufferParsing_test_pos_after_key:1:19: Non-keyword arg following keyword arg del.cpp: In function ?int __Pyx_DelItemInt(PyObject*, Py_ssize_t, int)?: del.cpp:274: warning: left shift count >= width of type index.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: index.cpp:271: warning: left shift count >= width of type index.cpp: In function ?int __Pyx_SetItemInt(PyObject*, Py_ssize_t, PyObject*, int)?: index.cpp:339: warning: left shift count >= width of type indices.cpp: In function ?int __Pyx_SetItemInt(PyObject*, Py_ssize_t, PyObject*, int)?: indices.cpp:272: warning: left shift count >= width of type indices.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: indices.cpp:283: warning: left shift count >= width of type jiba5.cpp: In function ?int __Pyx_DelItemInt(PyObject*, Py_ssize_t, int)?: jiba5.cpp:266: warning: left shift count >= width of type johnson1.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: johnson1.cpp:263: warning: left shift count >= width of type specmethdocstring.c:303: warning: ?__pyx_doc_17specmethdocstring_1C___init__? defined but not used specmethdocstring.c:325: warning: ?__pyx_doc_17specmethdocstring_1C_3foo___get__? defined but not used specmethdocstring.c:344: warning: ?__pyx_doc_17specmethdocstring_1C_3foo___set__? defined but not used specmethdocstring.cpp:303: warning: ?__pyx_doc_17specmethdocstring_1C___init__? defined but not used specmethdocstring.cpp:325: warning: ?__pyx_doc_17specmethdocstring_1C_3foo___get__? defined but not used specmethdocstring.cpp:344: warning: ?__pyx_doc_17specmethdocstring_1C_3foo___set__? defined but not used tryexcept.cpp: In function ?int __Pyx_SetItemInt(PyObject*, Py_ssize_t, PyObject*, int)?: tryexcept.cpp:280: warning: left shift count >= width of type typecast.c: In function ?__pyx_f_8typecast_f?: typecast.c:307: warning: cast to pointer from integer of different size arrayassign.c: In function ?__pyx_pf_11arrayassign_test_literal_list_slice_start_param?: arrayassign.c:775: warning: format ?%d? expects type ?int?, but argument 4 has type ?Py_ssize_t? arrayassign.c: In function ?__pyx_pf_11arrayassign_test_literal_list_slice_end_param?: arrayassign.c:893: warning: format ?%d? expects type ?int?, but argument 4 has type ?Py_ssize_t? arrayassign.c: In function ?__pyx_pf_11arrayassign_test_literal_list_slice_start_end_param?: arrayassign.c:1056: warning: format ?%d? expects type ?int?, but argument 4 has type ?Py_ssize_t? arrayassign.cpp: In function ?PyObject* __pyx_pf_11arrayassign_test_literal_list_slice_start_param(PyObject*, PyObject*)?: arrayassign.cpp:775: warning: format ?%d? expects type ?int?, but argument 4 has type ?Py_ssize_t? arrayassign.cpp: In function ?PyObject* __pyx_pf_11arrayassign_test_literal_list_slice_end_param(PyObject*, PyObject*)?: arrayassign.cpp:893: warning: format ?%d? expects type ?int?, but argument 4 has type ?Py_ssize_t? arrayassign.cpp: In function ?PyObject* __pyx_pf_11arrayassign_test_literal_list_slice_start_end_param(PyObject*, PyObject*, PyObject*)?: arrayassign.cpp:1056: warning: format ?%d? expects type ?int?, but argument 4 has type ?Py_ssize_t? big_indices.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: big_indices.cpp:263: warning: left shift count >= width of type bufaccess.c: In function ?__pyx_pf_9bufaccess_printbuf?: bufaccess.c:1525: warning: unused variable ?__pyx_boffset_1_buf? bufaccess.c:1524: warning: unused variable ?__pyx_boffset_0_buf? bufaccess.c:1523: warning: unused variable ?__pyx_bshape_1_buf? bufaccess.c:1522: warning: unused variable ?__pyx_bshape_0_buf? bufaccess.c:1521: warning: unused variable ?__pyx_bstride_1_buf? bufaccess.c:1520: warning: unused variable ?__pyx_bstride_0_buf? bufaccess.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: bufaccess.cpp:311: warning: left shift count >= width of type bufaccess.cpp: In function ?PyObject* __pyx_pf_9bufaccess_printbuf(PyObject*, PyObject*)?: bufaccess.cpp:1520: warning: unused variable ?__pyx_bstride_0_buf? bufaccess.cpp:1521: warning: unused variable ?__pyx_bstride_1_buf? bufaccess.cpp:1522: warning: unused variable ?__pyx_bshape_0_buf? bufaccess.cpp:1523: warning: unused variable ?__pyx_bshape_1_buf? bufaccess.cpp:1524: warning: unused variable ?__pyx_boffset_0_buf? bufaccess.cpp:1525: warning: unused variable ?__pyx_boffset_1_buf? buffer.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: buffer.cpp:263: warning: left shift count >= width of type callargs.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: callargs.cpp:274: warning: left shift count >= width of type cdef_opt.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: cdef_opt.cpp:271: warning: left shift count >= width of type charencoding.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: charencoding.cpp:263: warning: left shift count >= width of type charescape.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: charescape.cpp:263: warning: left shift count >= width of type classbody_exec.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: classbody_exec.cpp:263: warning: left shift count >= width of type concatcstrings.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: concatcstrings.cpp:263: warning: left shift count >= width of type cstringmeth.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: cstringmeth.cpp:263: warning: left shift count >= width of type ct_DEF.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: ct_DEF.cpp:263: warning: left shift count >= width of type delete.cpp: In function ?int __Pyx_DelItemInt(PyObject*, Py_ssize_t, int)?: delete.cpp:268: warning: left shift count >= width of type dictintindex.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: dictintindex.cpp:263: warning: left shift count >= width of type dictintindex.cpp: In function ?int __Pyx_DelItemInt(PyObject*, Py_ssize_t, int)?: dictintindex.cpp:325: warning: left shift count >= width of type forfrom.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: forfrom.cpp:263: warning: left shift count >= width of type funcexcept.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: funcexcept.cpp:263: warning: left shift count >= width of type funcexceptchained.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: funcexceptchained.cpp:263: warning: left shift count >= width of type funcexceptcypy.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: funcexceptcypy.cpp:271: warning: left shift count >= width of type funcexceptreturn.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: funcexceptreturn.cpp:263: warning: left shift count >= width of type future_unicode_literals.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: future_unicode_literals.cpp:263: warning: left shift count >= width of type index.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: index.cpp:271: warning: left shift count >= width of type inplace.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: inplace.cpp:277: warning: left shift count >= width of type inplace.cpp: In function ?int __Pyx_SetItemInt(PyObject*, Py_ssize_t, PyObject*, int)?: inplace.cpp:345: warning: left shift count >= width of type int_literals.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: int_literals.cpp:263: warning: left shift count >= width of type ishimoto2.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: ishimoto2.cpp:271: warning: left shift count >= width of type isinstance.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: isinstance.cpp:263: warning: left shift count >= width of type jarausch1.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: jarausch1.cpp:263: warning: left shift count >= width of type kostyrka2.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: kostyrka2.cpp:263: warning: left shift count >= width of type kwargproblems.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: kwargproblems.cpp:269: warning: left shift count >= width of type lepage_1.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: lepage_1.cpp:266: warning: left shift count >= width of type literal_lists.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: literal_lists.cpp:263: warning: left shift count >= width of type literalslice.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: literalslice.cpp:263: warning: left shift count >= width of type locals.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: locals.cpp:271: warning: left shift count >= width of type longlongindex.cpp: In function ?int __Pyx_SetItemInt(PyObject*, Py_ssize_t, PyObject*, int)?: longlongindex.cpp:280: warning: left shift count >= width of type modbody.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: modbody.cpp:263: warning: left shift count >= width of type modop.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: modop.cpp:271: warning: left shift count >= width of type multass.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: multass.cpp:263: warning: left shift count >= width of type new_style_exceptions.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: new_style_exceptions.cpp:263: warning: left shift count >= width of type powop.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: powop.cpp:271: warning: left shift count >= width of type r_addint.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: r_addint.cpp:271: warning: left shift count >= width of type r_barbieri1.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: r_barbieri1.cpp:269: warning: left shift count >= width of type r_bowden1.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: r_bowden1.cpp:263: warning: left shift count >= width of type r_docstrings.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: r_docstrings.cpp:265: warning: left shift count >= width of type r_hordijk1.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: r_hordijk1.cpp:269: warning: left shift count >= width of type r_huss3.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: r_huss3.cpp:263: warning: left shift count >= width of type r_jeff_epler_1.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: r_jeff_epler_1.cpp:263: warning: left shift count >= width of type r_mang1.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: r_mang1.cpp:263: warning: left shift count >= width of type r_mitch_chapman_2.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: r_mitch_chapman_2.cpp:263: warning: left shift count >= width of type r_pythonapi.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: r_pythonapi.cpp:264: warning: left shift count >= width of type r_uintindex.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: r_uintindex.cpp:271: warning: left shift count >= width of type set.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: set.cpp:351: warning: left shift count >= width of type simpcall.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: simpcall.cpp:271: warning: left shift count >= width of type size_t.c: In function ?__pyx_pf_6size_t_1A___init__?: size_t.c:389: warning: unused variable ?__pyx_arg_a? size_t.cpp: In function ?int __pyx_pf_6size_t_1A___init__(PyObject*, PyObject*, PyObject*)?: size_t.cpp:389: warning: unused variable ?__pyx_arg_a? slice_charptr.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: slice_charptr.cpp:271: warning: left shift count >= width of type strconstinclass.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: strconstinclass.cpp:263: warning: left shift count >= width of type strescapes.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: strescapes.cpp:263: warning: left shift count >= width of type strfunction.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: strfunction.cpp:263: warning: left shift count >= width of type strliterals.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: strliterals.cpp:263: warning: left shift count >= width of type unicodefunction.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: unicodefunction.cpp:263: warning: left shift count >= width of type unicodeliterals.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: unicodeliterals.cpp:263: warning: left shift count >= width of type unicodeliteralsdefault.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: unicodeliteralsdefault.cpp:263: warning: left shift count >= width of type unicodeliteralslatin1.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: unicodeliteralslatin1.cpp:263: warning: left shift count >= width of type unsigned.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: unsigned.cpp:263: warning: left shift count >= width of type wundram1.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: wundram1.cpp:263: warning: left shift count >= width of type ---------------------------------------------------------------------- Ran 1404 tests in 263.628s OK Following tests excluded because of missing dependencies on your system: run.numpy_test From dalcinl at gmail.com Tue Mar 10 16:30:26 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 10 Mar 2009 13:30:26 -0200 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: <52637.213.61.181.86.1236695813.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <52637.213.61.181.86.1236695813.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: >> Now testing at least starts, but seems to have some errors: > > * warning: left shift count >= width of type > > I don't get these locally. The problem is in __Pyx_GetItemGeneric, second > line: "1ULL << (sizeof(Py_ssize_t)*8)". Looks like that's not 64bit clean. > PyInt_FromSsize_t() should be used here.... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Tue Mar 10 16:38:43 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 10 Mar 2009 16:38:43 +0100 (CET) Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: References: <52637.213.61.181.86.1236695813.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <61316.213.61.181.86.1236699523.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Lisandro Dalcin wrote: >>> Now testing at least starts, but seems to have some errors: >> >> * warning: left shift count >= width of type >> >> I don't get these locally. The problem is in __Pyx_GetItemGeneric, >> second >> line: "1ULL << (sizeof(Py_ssize_t)*8)". Looks like that's not 64bit >> clean. >> > > PyInt_FromSsize_t() should be used here.... Could you come up with a fix? The other functions are affected, too: inplace.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: inplace.cpp:277: warning: left shift count >= width of type inplace.cpp: In function ?int __Pyx_SetItemInt(PyObject*, Py_ssize_t, PyObject*, int)?: inplace.cpp:345: warning: left shift count >= width of type dictintindex.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, Py_ssize_t, int)?: dictintindex.cpp:263: warning: left shift count >= width of type Stefan From ndbecker2 at gmail.com Tue Mar 10 16:48:49 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 10 Mar 2009 11:48:49 -0400 Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 References: Message-ID: Built succeeds on fedora rawhide on all supported archs. From stefan_ml at behnel.de Tue Mar 10 16:54:17 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 10 Mar 2009 16:54:17 +0100 (CET) Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 In-Reply-To: References: Message-ID: <44706.213.61.181.86.1236700457.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Neal Becker wrote: > Built succeeds on fedora rawhide on all supported archs. Thanks for testing. However, we will need a retest when we have fixed the discovered 64bit issues. You can get the latest cython-devel from here: http://hg.cython.org/cython-devel Either install hg and do a checkout of that URL, or use the direct archive download links on the left of the web page. The latest version from there shouldn't require any of the patches I posted, but it does not yet have the 64bit fix. Stefan From dalcinl at gmail.com Tue Mar 10 16:55:02 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 10 Mar 2009 13:55:02 -0200 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: <61316.213.61.181.86.1236699523.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <52637.213.61.181.86.1236695813.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <61316.213.61.181.86.1236699523.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: sure, wait me a bit, working on the runtests+numpy patch :-) On Tue, Mar 10, 2009 at 1:38 PM, Stefan Behnel wrote: > Lisandro Dalcin wrote: >>>> Now testing at least starts, but seems to have some errors: >>> >>> * warning: left shift count >= width of type >>> >>> I don't get these locally. The problem is in __Pyx_GetItemGeneric, >>> second >>> line: "1ULL << (sizeof(Py_ssize_t)*8)". Looks like that's not 64bit >>> clean. >>> >> >> PyInt_FromSsize_t() should be used here.... > > Could you come up with a fix? The other functions are affected, too: > > inplace.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, > Py_ssize_t, int)?: > inplace.cpp:277: warning: left shift count >= width of type > > inplace.cpp: In function ?int __Pyx_SetItemInt(PyObject*, Py_ssize_t, > PyObject*, int)?: > inplace.cpp:345: warning: left shift count >= width of type > > dictintindex.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, > Py_ssize_t, int)?: > dictintindex.cpp:263: warning: left shift count >= width of type > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Tue Mar 10 16:58:38 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 10 Mar 2009 13:58:38 -0200 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: <50816.213.61.181.86.1236698960.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49B676A3.3090405@student.matnat.uio.no> <49B67D10.4060308@student.matnat.uio.no> <40279.213.61.181.86.1236696818.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <50816.213.61.181.86.1236698960.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: On Tue, Mar 10, 2009 at 1:29 PM, Stefan Behnel wrote: > > Thanks for rephrasing my thoughts. :) > You are welcome... > > Please supply a fix for runtests.py. > Attached... Hope you like it.... feel free to modify and push ... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 -------------- next part -------------- A non-text attachment was scrubbed... Name: RUNTESTS_NUMPY.diff Type: application/octet-stream Size: 1107 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090310/eff6ec60/attachment.obj From dalcinl at gmail.com Tue Mar 10 17:19:53 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 10 Mar 2009 14:19:53 -0200 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: <61316.213.61.181.86.1236699523.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <52637.213.61.181.86.1236695813.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <61316.213.61.181.86.1236699523.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: Here you have a trivial patch... However, all this part smells so bad... On Tue, Mar 10, 2009 at 1:38 PM, Stefan Behnel wrote: > Lisandro Dalcin wrote: >>>> Now testing at least starts, but seems to have some errors: >>> >>> * warning: left shift count >= width of type >>> >>> I don't get these locally. The problem is in __Pyx_GetItemGeneric, >>> second >>> line: "1ULL << (sizeof(Py_ssize_t)*8)". Looks like that's not 64bit >>> clean. >>> >> >> PyInt_FromSsize_t() should be used here.... > > Could you come up with a fix? The other functions are affected, too: > > inplace.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, > Py_ssize_t, int)?: > inplace.cpp:277: warning: left shift count >= width of type > > inplace.cpp: In function ?int __Pyx_SetItemInt(PyObject*, Py_ssize_t, > PyObject*, int)?: > inplace.cpp:345: warning: left shift count >= width of type > > dictintindex.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic(PyObject*, > Py_ssize_t, int)?: > dictintindex.cpp:263: warning: left shift count >= width of type > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 -------------- next part -------------- A non-text attachment was scrubbed... Name: PATCH.diff Type: application/octet-stream Size: 1680 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090310/50e17e36/attachment.obj From dalcinl at gmail.com Tue Mar 10 17:53:17 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 10 Mar 2009 14:53:17 -0200 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: References: <52637.213.61.181.86.1236695813.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <61316.213.61.181.86.1236699523.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: Indeed, this smells really bad... the former implementation seems really hackish... my patch will not work. On Tue, Mar 10, 2009 at 2:19 PM, Lisandro Dalcin wrote: > Here you have a trivial patch... However, all this part smells so bad... > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From ndbecker2 at gmail.com Tue Mar 10 18:10:55 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 10 Mar 2009 13:10:55 -0400 Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 References: <44706.213.61.181.86.1236700457.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: Stefan Behnel wrote: > Neal Becker wrote: >> Built succeeds on fedora rawhide on all supported archs. > > Thanks for testing. However, we will need a retest when we have fixed the > discovered 64bit issues. You can get the latest cython-devel from here: > > http://hg.cython.org/cython-devel > > Either install hg and do a checkout of that URL, or use the direct archive > download links on the left of the web page. > > The latest version from there shouldn't require any of the patches I > posted, but it does not yet have the 64bit fix. > > Stefan Can we have an rc2 with all proposed fixes? I'll plan to retest then. From stefan_ml at behnel.de Tue Mar 10 18:15:06 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 10 Mar 2009 18:15:06 +0100 Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 In-Reply-To: References: <44706.213.61.181.86.1236700457.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49B6A01A.8050904@behnel.de> Neal Becker wrote: > Can we have an rc2 with all proposed fixes? I'll plan to retest then. Fine with me, although it (hopefully) won't take long. Stefan From robertwb at math.washington.edu Tue Mar 10 18:44:54 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 10 Mar 2009 10:44:54 -0700 Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 In-Reply-To: <49B6A01A.8050904@behnel.de> References: <44706.213.61.181.86.1236700457.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B6A01A.8050904@behnel.de> Message-ID: On Mar 10, 2009, at 10:15 AM, Stefan Behnel wrote: > > Neal Becker wrote: >> Can we have an rc2 with all proposed fixes? I'll plan to retest >> then. > > Fine with me, although it (hopefully) won't take long. http://cython.org/Cython-0.11.rc2.tar.gz From stefan_ml at behnel.de Tue Mar 10 18:50:20 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 10 Mar 2009 18:50:20 +0100 Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 In-Reply-To: References: <44706.213.61.181.86.1236700457.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B6A01A.8050904@behnel.de> Message-ID: <49B6A85C.2090301@behnel.de> Robert Bradshaw wrote: > On Mar 10, 2009, at 10:15 AM, Stefan Behnel wrote: > >> Neal Becker wrote: >>> Can we have an rc2 with all proposed fixes? I'll plan to retest >>> then. >> Fine with me, although it (hopefully) won't take long. > > http://cython.org/Cython-0.11.rc2.tar.gz Ah, ok, but this doesn't include a fix for the __Pyx_GetItemInt() issues on 64bit platforms. Looks like we need another RC for that... Stefan From robertwb at math.washington.edu Tue Mar 10 18:51:09 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 10 Mar 2009 10:51:09 -0700 Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 In-Reply-To: References: <44706.213.61.181.86.1236700457.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B6A01A.8050904@behnel.de> Message-ID: On Mar 10, 2009, at 10:44 AM, Robert Bradshaw wrote: > On Mar 10, 2009, at 10:15 AM, Stefan Behnel wrote: > >> >> Neal Becker wrote: >>> Can we have an rc2 with all proposed fixes? I'll plan to retest >>> then. >> >> Fine with me, although it (hopefully) won't take long. > > http://cython.org/Cython-0.11.rc2.tar.gz BTW, this does not have the 64-bit issues fixed. (Still catching up on the other thread). Sorry about the naming for rc1, that wasn't intentional. - Robert From robertwb at math.washington.edu Tue Mar 10 19:08:32 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 10 Mar 2009 11:08:32 -0700 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: References: <52637.213.61.181.86.1236695813.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <61316.213.61.181.86.1236699523.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <5770E6DE-385E-46BE-9070-0AD57466C5A1@math.washington.edu> I'll fix this. The problem is that when passing via the inline function, the size and signness of the original index gets lost. The very common and much faster case is to use PyInt_FromLong, but of course we have to handle bigger indices as well. - Robert On Mar 10, 2009, at 9:19 AM, Lisandro Dalcin wrote: > Here you have a trivial patch... However, all this part smells so > bad... > > > On Tue, Mar 10, 2009 at 1:38 PM, Stefan Behnel > wrote: >> Lisandro Dalcin wrote: >>>>> Now testing at least starts, but seems to have some errors: >>>> >>>> * warning: left shift count >= width of type >>>> >>>> I don't get these locally. The problem is in __Pyx_GetItemGeneric, >>>> second >>>> line: "1ULL << (sizeof(Py_ssize_t)*8)". Looks like that's not 64bit >>>> clean. >>>> >>> >>> PyInt_FromSsize_t() should be used here.... >> >> Could you come up with a fix? The other functions are affected, too: >> >> inplace.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic >> (PyObject*, >> Py_ssize_t, int)?: >> inplace.cpp:277: warning: left shift count >= width of type >> >> inplace.cpp: In function ?int __Pyx_SetItemInt(PyObject*, Py_ssize_t, >> PyObject*, int)?: >> inplace.cpp:345: warning: left shift count >= width of type >> >> dictintindex.cpp: In function ?PyObject* __Pyx_GetItemInt_Generic >> (PyObject*, >> Py_ssize_t, int)?: >> dictintindex.cpp:263: warning: left shift count >= width of type >> >> Stefan >> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0) > 342-451.1594______________________________________________ > _ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From stefan_ml at behnel.de Tue Mar 10 20:57:59 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 10 Mar 2009 20:57:59 +0100 Subject: [Cython] Remove "parsetuple_format" from PyrexTypes? Message-ID: <49B6C647.9090807@behnel.de> Hi, I just fixed an unused variable problem when using (Py_s)size_t as function argument types (fix can wait for 0.11.1). Doing this, I noticed that the type indication character that each type stores in "parsetuple_format" is no longer used anywhere since PyArg_ParseTupleAndKeywords() was replaced by our own code. Before I rip this attribute straight out, can anyone still see a use case for this? Maybe in combination with buffer content types? Stefan From dagss at student.matnat.uio.no Tue Mar 10 21:17:49 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 10 Mar 2009 21:17:49 +0100 Subject: [Cython] Remove "parsetuple_format" from PyrexTypes? In-Reply-To: <49B6C647.9090807@behnel.de> References: <49B6C647.9090807@behnel.de> Message-ID: <49B6CAED.9080400@student.matnat.uio.no> Stefan Behnel wrote: > Hi, > > I just fixed an unused variable problem when using (Py_s)size_t as function > argument types (fix can wait for 0.11.1). Doing this, I noticed that the > type indication character that each type stores in "parsetuple_format" is > no longer used anywhere since PyArg_ParseTupleAndKeywords() was replaced by > our own code. Before I rip this attribute straight out, can anyone still > see a use case for this? Maybe in combination with buffer content types? The buffers typestring checks do not use this. -- Dag Sverre From hazelnusse at gmail.com Tue Mar 10 22:33:02 2009 From: hazelnusse at gmail.com (Luke) Date: Tue, 10 Mar 2009 14:33:02 -0700 Subject: [Cython] Update on: Incomplete C++ example Message-ID: <99214b470903101433w2cb1a2f4x896fd11203f97a14@mail.gmail.com> I'm trying to wrap some basic C++ code. I tried to follow the directions on the wiki, but I found them to be unclear and incomplete. I also noticed a thread about this on Cython-dev, but it doesn't seem to have been resolved. As a total rookie with Cython and wrapping code in general, it would be nice to have a step by step example of the process that is illustrative of how to wrap a C++ library, including: 1) Source of the header file (Rectangle.h) 2) Source of the implementation file (Rectangle.cpp) 3) Source of the setup file (setup.py) 4) Source of the ??? file (Rectangle.pyx) 5) shell commands necessary prior to being able to import the Rectangle class into python 6) example of using the wrapped C++ library inside the python interpreter Things that were not helpful in the setup.py were things like: include_dirs=[...], # usual stuff libraries=[...], # ditto extra_link_args=[...], # if needed As an example of how to wrap a C++ class, I don't think extras that aren't well explained help the situation. From the wiki, it wasn't clear what exactly belonged in the .pyx file. It would be nice if this could be clarified. I think I can piece together steps 1), 2), 3) from the wiki and from the previous thread on the subject, but steps 4), 5), and 6) are still unclear to me. It would be a great help if these steps could be shown explicitly. Can anybody help me out on how to do this? Thanks, ~Luke From bpederse at gmail.com Tue Mar 10 22:51:30 2009 From: bpederse at gmail.com (Brent Pedersen) Date: Tue, 10 Mar 2009 14:51:30 -0700 Subject: [Cython] Update on: Incomplete C++ example In-Reply-To: <99214b470903101433w2cb1a2f4x896fd11203f97a14@mail.gmail.com> References: <99214b470903101433w2cb1a2f4x896fd11203f97a14@mail.gmail.com> Message-ID: i'm sure it doesnt answer all your questions, but here's how i built and used the cpp example, it required some code changes. i uploaded to wiki, but didnt link to it directly. it has a Makefile and example usage script. http://wiki.cython.org/WrappingCPlusPlus?action=AttachFile&do=get&target=cythoncpp.tgz -brent On Tue, Mar 10, 2009 at 2:33 PM, Luke wrote: > I'm trying to wrap some basic C++ code. ?I tried to follow the > directions on the wiki, but I found them to be unclear and incomplete. > ?I also noticed a thread about this on Cython-dev, but it doesn't seem > to have been resolved. ?As a total rookie with Cython and wrapping > code in general, it would be nice to have a step by step example of > the process that is illustrative of how to wrap a C++ library, > including: > > 1) ?Source of the header file (Rectangle.h) > 2) ?Source of the implementation file (Rectangle.cpp) > 3) ?Source of the setup file (setup.py) > 4) ?Source of the ??? file (Rectangle.pyx) > 5) ?shell commands necessary prior to being able to import the > Rectangle class into python > 6) ?example of using the wrapped C++ library inside the python interpreter > > Things that were not helpful in the setup.py were things like: > ? ?include_dirs=[...], ? ? ? ? ?# usual stuff > ? ?libraries=[...], ? ? ? ? ? ? # ditto > ? ?extra_link_args=[...], ? ? ? # if needed > > As an example of how to wrap a C++ class, I don't think extras that > aren't well explained help the situation. ?From the wiki, it wasn't > clear what exactly belonged in the .pyx file. ?It would be nice if > this could be clarified. > > I think I can piece together steps 1), 2), 3) from the wiki and from > the previous thread on the subject, but steps 4), 5), and 6) are still > unclear to me. ?It would be a great help if these steps could be shown > explicitly. ?Can anybody help me out on how to do this? > > Thanks, > ~Luke > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From hazelnusse at gmail.com Tue Mar 10 23:31:46 2009 From: hazelnusse at gmail.com (Luke) Date: Tue, 10 Mar 2009 15:31:46 -0700 Subject: [Cython] Update on: Incomplete C++ example In-Reply-To: References: <99214b470903101433w2cb1a2f4x896fd11203f97a14@mail.gmail.com> Message-ID: <99214b470903101531j5666939dv9c7e8f6f47ae74e4@mail.gmail.com> Brent, Thanks for the quick response. That helped a lot. What would I need to do differently if instead of having the header file and the implementation file (cpp_rect.h & cpp_rect.cpp), I instead had the header file and the object file (cpp_rect.h and cpp_rect.o)? Thanks, ~Luke On Tue, Mar 10, 2009 at 2:51 PM, Brent Pedersen wrote: > i'm sure it doesnt answer all your questions, but here's how i built > and used the cpp example, it required some code changes. > i uploaded to wiki, but didnt link to it directly. > it has a Makefile and example usage script. > > http://wiki.cython.org/WrappingCPlusPlus?action=AttachFile&do=get&target=cythoncpp.tgz > > -brent > > On Tue, Mar 10, 2009 at 2:33 PM, Luke wrote: >> I'm trying to wrap some basic C++ code. ?I tried to follow the >> directions on the wiki, but I found them to be unclear and incomplete. >> ?I also noticed a thread about this on Cython-dev, but it doesn't seem >> to have been resolved. ?As a total rookie with Cython and wrapping >> code in general, it would be nice to have a step by step example of >> the process that is illustrative of how to wrap a C++ library, >> including: >> >> 1) ?Source of the header file (Rectangle.h) >> 2) ?Source of the implementation file (Rectangle.cpp) >> 3) ?Source of the setup file (setup.py) >> 4) ?Source of the ??? file (Rectangle.pyx) >> 5) ?shell commands necessary prior to being able to import the >> Rectangle class into python >> 6) ?example of using the wrapped C++ library inside the python interpreter >> >> Things that were not helpful in the setup.py were things like: >> ? ?include_dirs=[...], ? ? ? ? ?# usual stuff >> ? ?libraries=[...], ? ? ? ? ? ? # ditto >> ? ?extra_link_args=[...], ? ? ? # if needed >> >> As an example of how to wrap a C++ class, I don't think extras that >> aren't well explained help the situation. ?From the wiki, it wasn't >> clear what exactly belonged in the .pyx file. ?It would be nice if >> this could be clarified. >> >> I think I can piece together steps 1), 2), 3) from the wiki and from >> the previous thread on the subject, but steps 4), 5), and 6) are still >> unclear to me. ?It would be a great help if these steps could be shown >> explicitly. ?Can anybody help me out on how to do this? >> >> Thanks, >> ~Luke >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From dalcinl at gmail.com Tue Mar 10 23:48:35 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 10 Mar 2009 20:48:35 -0200 Subject: [Cython] Update on: Incomplete C++ example In-Reply-To: <99214b470903101531j5666939dv9c7e8f6f47ae74e4@mail.gmail.com> References: <99214b470903101433w2cb1a2f4x896fd11203f97a14@mail.gmail.com> <99214b470903101531j5666939dv9c7e8f6f47ae74e4@mail.gmail.com> Message-ID: Well, here you have... a full tarball package (generated with "python setup.py sdist") with the minimum required package metadata for the sake of PyPI upload... the build.sh script install it in $PWD/tmp, next run the test.sh script, it just adjusts $PYTHONPATH and runs example.py I hope you can make a useful contribution to Cython's Wiki after this ;-) On Tue, Mar 10, 2009 at 8:31 PM, Luke wrote: > Brent, > ?Thanks for the quick response. ?That helped a lot. ?What would I > need to do differently if instead of having the header file and the > implementation file (cpp_rect.h & cpp_rect.cpp), I instead had the > header file and the object file (cpp_rect.h and cpp_rect.o)? > Thanks, > ~Luke > > On Tue, Mar 10, 2009 at 2:51 PM, Brent Pedersen wrote: >> i'm sure it doesnt answer all your questions, but here's how i built >> and used the cpp example, it required some code changes. >> i uploaded to wiki, but didnt link to it directly. >> it has a Makefile and example usage script. >> >> http://wiki.cython.org/WrappingCPlusPlus?action=AttachFile&do=get&target=cythoncpp.tgz >> >> -brent >> >> On Tue, Mar 10, 2009 at 2:33 PM, Luke wrote: >>> I'm trying to wrap some basic C++ code. ?I tried to follow the >>> directions on the wiki, but I found them to be unclear and incomplete. >>> ?I also noticed a thread about this on Cython-dev, but it doesn't seem >>> to have been resolved. ?As a total rookie with Cython and wrapping >>> code in general, it would be nice to have a step by step example of >>> the process that is illustrative of how to wrap a C++ library, >>> including: >>> >>> 1) ?Source of the header file (Rectangle.h) >>> 2) ?Source of the implementation file (Rectangle.cpp) >>> 3) ?Source of the setup file (setup.py) >>> 4) ?Source of the ??? file (Rectangle.pyx) >>> 5) ?shell commands necessary prior to being able to import the >>> Rectangle class into python >>> 6) ?example of using the wrapped C++ library inside the python interpreter >>> >>> Things that were not helpful in the setup.py were things like: >>> ? ?include_dirs=[...], ? ? ? ? ?# usual stuff >>> ? ?libraries=[...], ? ? ? ? ? ? # ditto >>> ? ?extra_link_args=[...], ? ? ? # if needed >>> >>> As an example of how to wrap a C++ class, I don't think extras that >>> aren't well explained help the situation. ?From the wiki, it wasn't >>> clear what exactly belonged in the .pyx file. ?It would be nice if >>> this could be clarified. >>> >>> I think I can piece together steps 1), 2), 3) from the wiki and from >>> the previous thread on the subject, but steps 4), 5), and 6) are still >>> unclear to me. ?It would be a great help if these steps could be shown >>> explicitly. ?Can anybody help me out on how to do this? >>> >>> Thanks, >>> ~Luke >>> _______________________________________________ >>> Cython-dev mailing list >>> Cython-dev at codespeak.net >>> http://codespeak.net/mailman/listinfo/cython-dev >>> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 -------------- next part -------------- A non-text attachment was scrubbed... Name: rectangle-0.0.1.tar.gz Type: application/x-gzip Size: 1663 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090310/0c07a6bc/attachment.bin From hazelnusse at gmail.com Wed Mar 11 00:34:20 2009 From: hazelnusse at gmail.com (Luke) Date: Tue, 10 Mar 2009 16:34:20 -0700 Subject: [Cython] Wrapping C++ functions that pass by reference Message-ID: <99214b470903101634i16f79657pe008fd1bc75a9533@mail.gmail.com> I have a C++ function like this: void constraint(double& x, double& dxdt) { dxdt[0] = ... dxdt[1] = ... .... dxdt[n] = ... } How would I properly wrap this function in the .pyx file? And how would calling it in Python work? I've tried to find examples for this type of thing but have been unable. If anybody knows of any, I'd greatly appreciate it. Thanks, ~Luke From carl.witty at gmail.com Wed Mar 11 03:35:39 2009 From: carl.witty at gmail.com (Carl Witty) Date: Tue, 10 Mar 2009 18:35:39 -0800 Subject: [Cython] Semantics of cdef int division and %? Message-ID: I just reported http://trac.cython.org/cython_trac/ticket/229, which points out that division and % on cdef ints have C rather than Python semantics (for instance, (-1 % 16) is -1 in C, and 15 in Python). I'm not sure what the right thing to do here is. Using C semantics gives faster code. Using Python semantics is nicer (whenever I care about the difference, I always want Python semantics) but slower, and would be confusing to somebody who was coming from C. But whatever happens should be prominently documented; I couldn't find any mention of the issue in the current docs (although maybe I wasn't looking in the right place). Carl From hazelnusse at gmail.com Wed Mar 11 05:46:03 2009 From: hazelnusse at gmail.com (Luke) Date: Tue, 10 Mar 2009 21:46:03 -0700 Subject: [Cython] Update on: Incomplete C++ example In-Reply-To: References: <99214b470903101433w2cb1a2f4x896fd11203f97a14@mail.gmail.com> <99214b470903101531j5666939dv9c7e8f6f47ae74e4@mail.gmail.com> Message-ID: <99214b470903102146q442c5406s2c3ed265c65f02e4@mail.gmail.com> Lisandro, Thank you! ~Luke On Tue, Mar 10, 2009 at 3:48 PM, Lisandro Dalcin wrote: > Well, here you have... a full tarball package (generated with "python > setup.py sdist") with the minimum required package metadata for the > sake of PyPI upload... the build.sh script install it in $PWD/tmp, > next run the test.sh script, it just adjusts $PYTHONPATH and runs > example.py > > I hope you can make a useful contribution to Cython's Wiki after this ;-) > > > On Tue, Mar 10, 2009 at 8:31 PM, Luke wrote: >> Brent, >> ?Thanks for the quick response. ?That helped a lot. ?What would I >> need to do differently if instead of having the header file and the >> implementation file (cpp_rect.h & cpp_rect.cpp), I instead had the >> header file and the object file (cpp_rect.h and cpp_rect.o)? >> Thanks, >> ~Luke >> >> On Tue, Mar 10, 2009 at 2:51 PM, Brent Pedersen wrote: >>> i'm sure it doesnt answer all your questions, but here's how i built >>> and used the cpp example, it required some code changes. >>> i uploaded to wiki, but didnt link to it directly. >>> it has a Makefile and example usage script. >>> >>> http://wiki.cython.org/WrappingCPlusPlus?action=AttachFile&do=get&target=cythoncpp.tgz >>> >>> -brent >>> >>> On Tue, Mar 10, 2009 at 2:33 PM, Luke wrote: >>>> I'm trying to wrap some basic C++ code. ?I tried to follow the >>>> directions on the wiki, but I found them to be unclear and incomplete. >>>> ?I also noticed a thread about this on Cython-dev, but it doesn't seem >>>> to have been resolved. ?As a total rookie with Cython and wrapping >>>> code in general, it would be nice to have a step by step example of >>>> the process that is illustrative of how to wrap a C++ library, >>>> including: >>>> >>>> 1) ?Source of the header file (Rectangle.h) >>>> 2) ?Source of the implementation file (Rectangle.cpp) >>>> 3) ?Source of the setup file (setup.py) >>>> 4) ?Source of the ??? file (Rectangle.pyx) >>>> 5) ?shell commands necessary prior to being able to import the >>>> Rectangle class into python >>>> 6) ?example of using the wrapped C++ library inside the python interpreter >>>> >>>> Things that were not helpful in the setup.py were things like: >>>> ? ?include_dirs=[...], ? ? ? ? ?# usual stuff >>>> ? ?libraries=[...], ? ? ? ? ? ? # ditto >>>> ? ?extra_link_args=[...], ? ? ? # if needed >>>> >>>> As an example of how to wrap a C++ class, I don't think extras that >>>> aren't well explained help the situation. ?From the wiki, it wasn't >>>> clear what exactly belonged in the .pyx file. ?It would be nice if >>>> this could be clarified. >>>> >>>> I think I can piece together steps 1), 2), 3) from the wiki and from >>>> the previous thread on the subject, but steps 4), 5), and 6) are still >>>> unclear to me. ?It would be a great help if these steps could be shown >>>> explicitly. ?Can anybody help me out on how to do this? >>>> >>>> Thanks, >>>> ~Luke >>>> _______________________________________________ >>>> Cython-dev mailing list >>>> Cython-dev at codespeak.net >>>> http://codespeak.net/mailman/listinfo/cython-dev >>>> >>> _______________________________________________ >>> Cython-dev mailing list >>> Cython-dev at codespeak.net >>> http://codespeak.net/mailman/listinfo/cython-dev >>> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > From dagss at student.matnat.uio.no Wed Mar 11 08:31:35 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 08:31:35 +0100 Subject: [Cython] Wrapping C++ functions that pass by reference In-Reply-To: <99214b470903101634i16f79657pe008fd1bc75a9533@mail.gmail.com> References: <99214b470903101634i16f79657pe008fd1bc75a9533@mail.gmail.com> Message-ID: <49B768D7.4030903@student.matnat.uio.no> Luke wrote: > I have a C++ function like this: > > void constraint(double& x, double& dxdt) > { > dxdt[0] = ... > dxdt[1] = ... > .... > dxdt[n] = ... > } > > How would I properly wrap this function in the .pyx file? And how > would calling it in Python work? I've tried to find examples for this > type of thing but have been unable. If anybody knows of any, I'd > greatly appreciate it. This is C++ syntax candy for passing pointers. You declare and use it in Cython like this: cdef extern from "yourheader.h": void constraint(double* x, double* dxdt) def foo(): cdef double x = 3.2, dxdt = 0.01 constraint(&x, &dxdt) -- Dag Sverre From dagss at student.matnat.uio.no Wed Mar 11 08:34:19 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 08:34:19 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: Message-ID: <49B7697B.5090304@student.matnat.uio.no> Carl Witty wrote: > I just reported http://trac.cython.org/cython_trac/ticket/229, which > points out that division and % on cdef ints have C rather than Python > semantics (for instance, (-1 % 16) is -1 in C, and 15 in Python). > > I'm not sure what the right thing to do here is. Using C semantics > gives faster code. Using Python semantics is nicer (whenever I care > about the difference, I always want Python semantics) but slower, and > would be confusing to somebody who was coming from C. I'm +1 on Python semantics. We could make the C version available under some obscure name for any case when one really wants the speed, like cython.fast_c_mod(-1, 16). -- Dag Sverre From hoytak at cs.ubc.ca Wed Mar 11 08:35:40 2009 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Wed, 11 Mar 2009 00:35:40 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49B7697B.5090304@student.matnat.uio.no> References: <49B7697B.5090304@student.matnat.uio.no> Message-ID: <4db580fd0903110035n2c76f5f6u1b9b7b804564566d@mail.gmail.com> > I'm +1 on Python semantics. We could make the C version available under > some obscure name for any case when one really wants the speed, like > cython.fast_c_mod(-1, 16). Couldn't we just use the same trick as for buffer arrays -- i.e. different checks are performed when its unsigned vs. signed? I don't see why unsigned integers are not identical in behavior here to C. I've also never come across a situation where one wants -1 % 16 to be -1 and not 15, but maybe there is some obscure use case. --Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From dagss at student.matnat.uio.no Wed Mar 11 09:23:04 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 09:23:04 +0100 (CET) Subject: [Cython] Integer power operator Message-ID: <1286d6e2177fb46620ab4a1f1fa7d7f3.squirrel@webmail.uio.no> While we're discussing arithmetic operations anyway...: I happen to routinely write (some complex integer expression)**2 in Python, and it is tedious to manually store the expression in a temporary etc. in Cython. I understand the reasons this is disabled from the FAQ; are there any reasons I can't create a Cython-specific utility code implementation of integer power though? The powers are usually very small here so I'm thinking just a for-loop with O(log n) multiplications... Or if the power is a smallish literal (my **2 an **3), I suppose just inline the multiplication operations... Dag Sverre From dagss at student.matnat.uio.no Wed Mar 11 10:27:22 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 10:27:22 +0100 Subject: [Cython] Beginner-friendly tickets Message-ID: <49B783FA.5070309@student.matnat.uio.no> Since there's been some interest in how to get started coding on Cython on the mailing list lately, I thought I'd call attention to these two tickets: http://trac.cython.org/cython_trac/ticket/158 http://trac.cython.org/cython_trac/ticket/203 These seems to fit the sweet spot of not being too hard to fix for beginners (with some mentoring), but not being completely trivial either. I'm happy to provide an attack strategy/small amount of mentoring for these tickets if anybody's interested. -- Dag Sverre From stefan_ml at behnel.de Wed Mar 11 10:13:39 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 11 Mar 2009 10:13:39 +0100 (CET) Subject: [Cython] Integer power operator In-Reply-To: <1286d6e2177fb46620ab4a1f1fa7d7f3.squirrel@webmail.uio.no> References: <1286d6e2177fb46620ab4a1f1fa7d7f3.squirrel@webmail.uio.no> Message-ID: <39764.213.61.181.86.1236762819.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > I happen to routinely write (some complex integer expression)**2 in > Python, and it is tedious to manually store the expression in a temporary > etc. in Cython. > > I understand the reasons this is disabled from the FAQ; are there any > reasons I can't create a Cython-specific utility code implementation of > integer power though? The powers are usually very small here so I'm > thinking just a for-loop with O(log n) multiplications... Fine with me, see http://trac.cython.org/cython_trac/ticket/127 > Or if the power is a smallish literal (my **2 an **3), I suppose just > inline the multiplication operations... For constant expressions, that would even be part of the constant folding transformation. There are two problems here: 1) IIRC, the error is currently raised too early in the code, before running into constant folding, so even cdef int someval = 3**2 doesn't work. 2) it would be weird to have an operator work in some cases and not in others. For example, you don't know the size of a "long" at translation time, so you can't know if cdef long someval = 65536**2 overflows or not. So, where would you draw the line? If you can come up with a sensible utility function that handles the different C compile time and runtime overflow cases nicely, I'm all for it. Stefan From dagss at student.matnat.uio.no Wed Mar 11 10:37:36 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 10:37:36 +0100 Subject: [Cython] Integer power operator In-Reply-To: <39764.213.61.181.86.1236762819.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <1286d6e2177fb46620ab4a1f1fa7d7f3.squirrel@webmail.uio.no> <39764.213.61.181.86.1236762819.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49B78660.8060904@student.matnat.uio.no> Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> I happen to routinely write (some complex integer expression)**2 in >> Python, and it is tedious to manually store the expression in a temporary >> etc. in Cython. > > If you can come up with a sensible utility function that handles the > different C compile time and runtime overflow cases nicely, I'm all for > it. So you are "+1 if I include an overflow check"? I was just thinking of letting it silently overflow, the way addition and multiplication already silently overflows. -- Dag Sverre From dagss at student.matnat.uio.no Wed Mar 11 10:49:34 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 10:49:34 +0100 Subject: [Cython] Integer power operator In-Reply-To: <49B78660.8060904@student.matnat.uio.no> References: <1286d6e2177fb46620ab4a1f1fa7d7f3.squirrel@webmail.uio.no> <39764.213.61.181.86.1236762819.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B78660.8060904@student.matnat.uio.no> Message-ID: <49B7892E.2010901@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Dag Sverre Seljebotn wrote: >>> I happen to routinely write (some complex integer expression)**2 in >>> Python, and it is tedious to manually store the expression in a temporary >>> etc. in Cython. > > >> If you can come up with a sensible utility function that handles the >> different C compile time and runtime overflow cases nicely, I'm all for >> it. > > So you are "+1 if I include an overflow check"? > > I was just thinking of letting it silently overflow, the way addition > and multiplication already silently overflows. > To be clear, I'm in favour of overflow checks, I just think they in that case should apply in *all* situations, not simply in this one operator. Otherwise Cython will be inconsistent and it makes the language harder to learn. So I'm thinking this: 1) Add integer ** without overflow check 2) Add overflowcheck compiler directive, similar to the nonecheck directive, which checks for overflows in /all/ situations (which defaults to off though) -- Dag Sverre From stefan_ml at behnel.de Wed Mar 11 12:01:40 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 11 Mar 2009 12:01:40 +0100 (CET) Subject: [Cython] Integer power operator In-Reply-To: <49B78660.8060904@student.matnat.uio.no> References: <1286d6e2177fb46620ab4a1f1fa7d7f3.squirrel@webmail.uio.no> <39764.213.61.181.86.1236762819.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B78660.8060904@student.matnat.uio.no> Message-ID: <44766.213.61.181.86.1236769300.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Dag Sverre Seljebotn wrote: >>> I happen to routinely write (some complex integer expression)**2 in >>> Python, and it is tedious to manually store the expression in a >>> temporary >>> etc. in Cython. > >> If you can come up with a sensible utility function that handles the >> different C compile time and runtime overflow cases nicely, I'm all for >> it. > > So you are "+1 if I include an overflow check"? > > I was just thinking of letting it silently overflow, the way addition > and multiplication already silently overflows. Hmm, yes, you are right. When a user writes cdef int tinyint = 2**2000 it's not Cython's fault if it overflows. So I guess I'm actually +1 for a fix that provides Python integer power semantics modulo C overflows/wrap-arounds. I guess that would be the semantics with the lowest probability of surprise. Would you say that this makes sense as a utility function? I mean, we could always write out the bit-shifting code for a constant exponent... http://c2.com/cgi/wiki?IntegerPowerAlgorithm Stefan From sturla at molden.no Wed Mar 11 13:30:55 2009 From: sturla at molden.no (Sturla Molden) Date: Wed, 11 Mar 2009 13:30:55 +0100 Subject: [Cython] Wrapping C++ functions that pass by reference In-Reply-To: <99214b470903101634i16f79657pe008fd1bc75a9533@mail.gmail.com> References: <99214b470903101634i16f79657pe008fd1bc75a9533@mail.gmail.com> Message-ID: <49B7AEFF.1000302@molden.no> Luke wrote: > I have a C++ function like this: > > void constraint(double& x, double& dxdt) > { > dxdt[0] = ... > dxdt[1] = ... > .... > dxdt[n] = ... > } > > How would I properly wrap this function in the .pyx file? And how > would calling it in Python work? I think you can just say cdef extern from "constraint.h": void constraint(double x, double dxdt) as the generated C++ should be identical. S.M. From dalcinl at gmail.com Wed Mar 11 14:01:38 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 11 Mar 2009 11:01:38 -0200 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49B7697B.5090304@student.matnat.uio.no> References: <49B7697B.5090304@student.matnat.uio.no> Message-ID: On Wed, Mar 11, 2009 at 5:34 AM, Dag Sverre Seljebotn wrote: > Carl Witty wrote: >> I just reported http://trac.cython.org/cython_trac/ticket/229, which >> points out that division and % on cdef ints have C rather than Python >> semantics (for instance, (-1 % 16) is -1 in C, and 15 in Python). >> >> I'm not sure what the right thing to do here is. ?Using C semantics >> gives faster code. ?Using Python semantics is nicer (whenever I care >> about the difference, I always want Python semantics) but slower, and >> would be confusing to somebody who was coming from C. > > I'm +1 on Python semantics. We could make the C version available under > some obscure name for any case when one really wants the speed, like > cython.fast_c_mod(-1, 16). > I'm no so sure... C is C, Python is Python, and Cython is in the middle, but a cdef int is IMHO on C side... Are we going to upcast '1/2' to a float? Of course, I could live if Cython does this: - Python semantics for '%' - C semantics for an all-new operator '%%'. Just thinking on what Python has done with '//' And again... Are we going to upcast '1/2' to a float next? Still not sure about this... Dag, some long time ago, when I asked to suport s"abc" string literals, you told me that such feature is easily implemented on users side with a tiny function and doing s("abc") ... Well, you were right, but I believe so I am, users can always implement c = mod(a,b) to return what they want ;-) -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Wed Mar 11 14:05:11 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 14:05:11 +0100 Subject: [Cython] Wrapping C++ functions that pass by reference In-Reply-To: <49B7AEFF.1000302@molden.no> References: <99214b470903101634i16f79657pe008fd1bc75a9533@mail.gmail.com> <49B7AEFF.1000302@molden.no> Message-ID: <49B7B707.4000303@student.matnat.uio.no> Sturla Molden wrote: > Luke wrote: > >> I have a C++ function like this: >> >> void constraint(double& x, double& dxdt) >> { >> dxdt[0] = ... >> dxdt[1] = ... >> .... >> dxdt[n] = ... >> } >> >> How would I properly wrap this function in the .pyx file? And how >> would calling it in Python work? >> > I think you can just say > > cdef extern from "constraint.h": > void constraint(double x, double dxdt) > > as the generated C++ should be identical. > Good point! Please disregard my previous mail about this, I was wrong, and this is right. Dag Sverre From dagss at student.matnat.uio.no Wed Mar 11 14:13:58 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 14:13:58 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: <49B7697B.5090304@student.matnat.uio.no> Message-ID: <49B7B916.3080101@student.matnat.uio.no> Lisandro Dalcin wrote: > On Wed, Mar 11, 2009 at 5:34 AM, Dag Sverre Seljebotn > wrote: > >> Carl Witty wrote: >> >>> I just reported http://trac.cython.org/cython_trac/ticket/229, which >>> points out that division and % on cdef ints have C rather than Python >>> semantics (for instance, (-1 % 16) is -1 in C, and 15 in Python). >>> >>> I'm not sure what the right thing to do here is. Using C semantics >>> gives faster code. Using Python semantics is nicer (whenever I care >>> about the difference, I always want Python semantics) but slower, and >>> would be confusing to somebody who was coming from C. >>> >> I'm +1 on Python semantics. We could make the C version available under >> some obscure name for any case when one really wants the speed, like >> cython.fast_c_mod(-1, 16). >> >> > > I'm no so sure... C is C, Python is Python, and Cython is in the > middle, but a cdef int is IMHO on C side... Are we going to upcast > '1/2' to a float? > I think we should follow Python convention, i.e. "from __future__ import division" means this behaviour, and if the code (in the future) is Python 3-level, then that should be the default (as you say there's always the //). > Still not sure about this... > > Dag, some long time ago, when I asked to suport s"abc" string > literals, you told me that such feature is easily implemented on users > side with a tiny function and doing s("abc") ... Well, you were right, > but I believe so I am, users can always implement c = mod(a,b) to > return what they want ;-) > Sure :-) Syntax is mostly a matter of taste in the end. But the deeper reason behind my arguments is this: A rather typical usecase of Cython, which I believe we should encourage, is to a) first implement things in Python (or at least in untyped Cython) b) profile (or at least experience that things are too slow) c) find out which functions you bother to type, and incrementally type your program "bottom up". Now, if we suddenly change the meaning of operators depending on whether they operate on Python ints or C ints, it means that an algorithm that worked perfectly in step a) will break in step c). Which just means more hassle, and that step c) become a bigger job than it needs to be. Therefore I believe that the "C" part of Cython should mean "it is easy to call C and work with C types", but that the operator semantics should be as close to Python as possible. (Modulo overflowing, but as I said I hope for a overflowcheck compiler directive as well to help while doing step c), which can then be turned off). Dag Sverre From stefan_ml at behnel.de Wed Mar 11 14:18:06 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 11 Mar 2009 14:18:06 +0100 (CET) Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: <49B7697B.5090304@student.matnat.uio.no> Message-ID: <56541.213.61.181.86.1236777486.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Lisandro Dalcin wrote: > On Wed, Mar 11, 2009 at 5:34 AM, Dag Sverre Seljebotn > wrote: >> Carl Witty wrote: >>> I just reported http://trac.cython.org/cython_trac/ticket/229, which >>> points out that division and % on cdef ints have C rather than Python >>> semantics (for instance, (-1 % 16) is -1 in C, and 15 in Python). >>> >>> I'm not sure what the right thing to do here is. ?Using C semantics >>> gives faster code. ?Using Python semantics is nicer (whenever I care >>> about the difference, I always want Python semantics) but slower, and >>> would be confusing to somebody who was coming from C. >> >> I'm +1 on Python semantics. We could make the C version available under >> some obscure name for any case when one really wants the speed, like >> cython.fast_c_mod(-1, 16). > > I'm no so sure... C is C, Python is Python, and Cython is in the > middle, but a cdef int is IMHO on C side... Are we going to upcast > '1/2' to a float? I agree that this is one of those corner cases where there isn't a clear case for either Python and C semantics. I do see an interest in having Python semantics by default (even for C ints), but that might be different for people coming more from a C background. So there will always be surprises. What about supporting an explicit cast like this: i = -1 % 16 or at whatever part of that expression it looks better. Maybe cdef unsigned int i = -1 % 16 could behave like that, although it does feel weird that the result of an expression might depend on what it's assigned to... Stefan From dagss at student.matnat.uio.no Wed Mar 11 14:19:27 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 14:19:27 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: <49B7697B.5090304@student.matnat.uio.no> Message-ID: <49B7BA5F.9050107@student.matnat.uio.no> Lisandro Dalcin wrote: > Of course, I could live if Cython does this: > > - Python semantics for '%' > - C semantics for an all-new operator '%%'. Just thinking on what > Python has done with '//' > BTW I'm +0 on a new %%. As long as there is also a function in the cython module to do the same while having Python-compatible syntax. And I'm definitely +1 on having "a % b" mean "a %% b". Dag Sverre From stefan_ml at behnel.de Wed Mar 11 14:36:11 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 11 Mar 2009 14:36:11 +0100 (CET) Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49B7BA5F.9050107@student.matnat.uio.no> References: <49B7697B.5090304@student.matnat.uio.no> <49B7BA5F.9050107@student.matnat.uio.no> Message-ID: <44325.213.61.181.86.1236778571.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: >> Of course, I could live if Cython does this: >> >> - Python semantics for '%' >> - C semantics for an all-new operator '%%'. Just thinking on what >> Python has done with '//' >> > BTW I'm +0 on a new %%. As long as there is also a function in the > cython module to do the same while having Python-compatible syntax. I vote for being careful with new operators. If Python ever decides that it needs a '%%' on its own for whatever reason, we'd better be in the stdlib first and have a good eye on python-dev at that point. I'm rather for considering "-1%16==-1" an odd corner case that's not worth being the default in Cython (as you said, the idea is to come from Python and step down into C at need). Having a special "cython.signed_mod()" function seems enough to support it. > And I'm definitely +1 on having "a % b" mean "a %% b". Fair enough, but my guess is that the C compiler would do that for us. Stefan From stefan_ml at behnel.de Wed Mar 11 14:41:32 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 11 Mar 2009 14:41:32 +0100 (CET) Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49B7B916.3080101@student.matnat.uio.no> References: <49B7697B.5090304@student.matnat.uio.no> <49B7B916.3080101@student.matnat.uio.no> Message-ID: <50632.213.61.181.86.1236778892.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > a) first implement things in Python (or at least in untyped Cython) > b) profile (or at least experience that things are too slow) > c) find out which functions you bother to type, and incrementally type > your program "bottom up". > > Now, if we suddenly change the meaning of operators depending on whether > they operate on Python ints or C ints, it means that an algorithm that > worked perfectly in step a) will break in step c). Which just means more > hassle, and that step c) become a bigger job than it needs to be. That's also where I see the difference to things like overflows: you have to think about them anyway when you restrict the value range through typing (e.g. a Python long into a C long). But having an operator change its result just because you just typed one of its input variables seems like a rather dangerous trap that's hard to track down. Stefan From dalcinl at gmail.com Wed Mar 11 15:31:23 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 11 Mar 2009 12:31:23 -0200 Subject: [Cython] PATCH: the right way to use PyErr_Format with Py_ssize_t arguments Message-ID: Likely Dag should review this... If you agree on the patch, please push ... IMHO, this fix should go to 0.11 While testing things on Linux 64, I had to implement this to do things the right way... Should work on 32/64 bits and Py 2.3 to 3.0 ... BTW, the only possible problems are the warnings below (they appear on 32 and also 64 bits)... Do not remember right now C semantics about that., perhaps this does not harm too much index.cpp: In function ?PyObject* __pyx_pf_5index_test_unsigned_long(PyObject*, PyObject*)?: index.cpp:714: warning: comparison between signed and unsigned integer expressions index.cpp:750: warning: comparison between signed and unsigned integer expressions -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 -------------- next part -------------- A non-text attachment was scrubbed... Name: FMT_SIZE_T.diff Type: application/octet-stream Size: 1413 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090311/e4a06445/attachment.obj From dagss at student.matnat.uio.no Wed Mar 11 15:43:02 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 15:43:02 +0100 Subject: [Cython] PATCH: the right way to use PyErr_Format with Py_ssize_t arguments In-Reply-To: References: Message-ID: <49B7CDF6.3020401@student.matnat.uio.no> Lisandro Dalcin wrote: > Likely Dag should review this... If you agree on the patch, please > push ... IMHO, this fix should go to 0.11 > I don't think I have anything to do with that code (though there's likely other instances of the same problem in my Buffer.py). a) Saying that this should go into 0.11 means saying that this specific bug is more important than everything else that sits in milestone 0.11.1 in Trac. We probably put out 0.10 with the same bug and the Cython project survived! We need to have some disipline and get things released -- release early, release often. b) I don't know if I'm the right person to review, I don't know too much about the Python API or size_t in this respect; it looks OK to me. (If I made the API in the first place I'd make the macro contain the entire string, i.e. "d" and "ld", and not simply the "l" or nothing. Some C versions have a "z" for size_t IIRC?) Dag Sverre From carl.witty at gmail.com Wed Mar 11 16:52:20 2009 From: carl.witty at gmail.com (Carl Witty) Date: Wed, 11 Mar 2009 07:52:20 -0800 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: Message-ID: On Tue, Mar 10, 2009 at 6:35 PM, Carl Witty wrote: > I just reported http://trac.cython.org/cython_trac/ticket/229, which > points out that division and % on cdef ints have C rather than Python > semantics (for instance, (-1 % 16) is -1 in C, and 15 in Python). One point I alluded to above, but maybe should emphasize: IMHO we should preserve that a == (a//b)*b + a%b. This means that we should either change both division and modulo, or neither. Carl From dalcinl at gmail.com Wed Mar 11 17:01:07 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 11 Mar 2009 14:01:07 -0200 Subject: [Cython] PATCH: the right way to use PyErr_Format with Py_ssize_t arguments In-Reply-To: <49B7CDF6.3020401@student.matnat.uio.no> References: <49B7CDF6.3020401@student.matnat.uio.no> Message-ID: On Wed, Mar 11, 2009 at 12:43 PM, Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: >> Likely Dag should review this... If you agree on the patch, please >> push ... IMHO, this fix should go to 0.11 >> > I don't think I have anything to do with that code (though there's > likely other instances of the same problem in my Buffer.py). > OK, sorry... For some reason I though that this was related to numpy support... > a) Saying that this should go into 0.11 means saying that this specific > bug is more important than everything else that sits in milestone 0.11.1 > in Trac. We probably put out 0.10 with the same bug and the Cython > project survived! We need to have some disipline and get things released > -- release early, release often. > I understand that... I just wanted to point that this fix likely does not have high impact and fix 64 bits issues, more or less like recent Robert's pushes did. > b) I don't know if I'm the right person to review, I don't know too much > about the Python API or size_t in this respect; it looks OK to me. Sorry again... > (If I > made the API in the first place I'd make the macro contain the entire > string, i.e. "d" and "ld", and not simply the "l" or nothing. My patch just mimics what CPython >= 2.5 does... > Some C versions have a "z" for size_t IIRC?) > In Py<2.5, Py_ssize_t is bare 'int' and the 'z' specifier is not handled in PyErr_Format(). -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Wed Mar 11 17:07:29 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 17:07:29 +0100 Subject: [Cython] PATCH: the right way to use PyErr_Format with Py_ssize_t arguments In-Reply-To: References: <49B7CDF6.3020401@student.matnat.uio.no> Message-ID: <49B7E1C1.7040601@student.matnat.uio.no> Lisandro Dalcin wrote: >> a) Saying that this should go into 0.11 means saying that this specific >> bug is more important than everything else that sits in milestone 0.11.1 >> in Trac. We probably put out 0.10 with the same bug and the Cython >> project survived! We need to have some disipline and get things released >> -- release early, release often. >> >> > > I understand that... I just wanted to point that this fix likely does > not have high impact and fix 64 bits issues, more or less like recent > Robert's pushes did. > Yes, sorry about the outburst, requesting this into 0.11 isn't all that unreasonable, you are right that it is low impact. Anyway, this is Robert's call in the end. >> b) I don't know if I'm the right person to review, I don't know too much >> about the Python API or size_t in this respect; it looks OK to me. >> > > Sorry again... > No problem. Thank you for cleaning up all of these 64-bit issues BTW. I'll try to remember to see whether Buffer.py has any... Dag Sverre From sturla at molden.no Wed Mar 11 17:08:48 2009 From: sturla at molden.no (Sturla Molden) Date: Wed, 11 Mar 2009 17:08:48 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: Message-ID: <49B7E210.1010007@molden.no> Carl Witty wrote: > > One point I alluded to above, but maybe should emphasize: > > IMHO we should preserve that a == (a//b)*b + a%b. This means that we > should either change both division and modulo, or neither. Please let simple C types behave like simple C types and Python objects like Python objects. Even if it means 1/1 will return 1.0 i Python 3 and 1 in Cython. You could let numerical literals be Python objects unless an explicit cast to C is made. Thus, "1" would a Python integer object and " 1" would be a C integer constant. That would ensure maximum similarity between Python and Cython, but be very annoying when writing numerical code in Cython. Sturla Molden From dagss at student.matnat.uio.no Wed Mar 11 17:19:35 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 17:19:35 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49B7E210.1010007@molden.no> References: <49B7E210.1010007@molden.no> Message-ID: <49B7E497.4050007@student.matnat.uio.no> Sturla Molden wrote: > Carl Witty wrote: > >> One point I alluded to above, but maybe should emphasize: >> >> IMHO we should preserve that a == (a//b)*b + a%b. This means that we >> should either change both division and modulo, or neither. >> > > Please let simple C types behave like simple C types and Python objects > like Python objects. Even if it means 1/1 will return 1.0 i Python 3 and > 1 in Cython. > Could you provide some specific examples where this is more convenient? I already argued the opposite from the "don't break algorithms when adding types for speed"-perspective (which is a very common way of using Cython). (I.e. it's fine that *your* knowledge of C is in place and that you expect Cython to behave like C here, but what will give least surprise to new Python user wanting to speed up his or her code? My point is that Cython users come from different backgrounds and will expect different things. When specific usecases are presented, we can bring this discussion beyond personal taste and what whomever uses Cython right now "already knows"). (BTW Python semantics for division means that Cython has Python 2 semantics for "/" unless from __future__ import division is specified.) Dag Sverre From psp.reachable at gmail.com Wed Mar 11 19:12:01 2009 From: psp.reachable at gmail.com (Prajwal Suhas P) Date: Wed, 11 Mar 2009 23:42:01 +0530 Subject: [Cython] Beginner-friendly tickets In-Reply-To: <49B783FA.5070309@student.matnat.uio.no> References: <49B783FA.5070309@student.matnat.uio.no> Message-ID: Hi Dag, I am currently looking at the tickets. I was looking at this - cdef int i, n n = 10 for i in range(n): if i == 5: n *= 2 print i The generated c code for this is - __pyx_v_5test1_n = 10; for (__pyx_v_5test1_i = 0; __pyx_v_5test1_i < __pyx_v_5test1_n; __pyx_v_5test1_i+=1) { __pyx_1 = (__pyx_v_5test1_i == 5); if (__pyx_1) { __pyx_v_5test1_n *= 2; goto __pyx_L4; } __pyx_L4:; } Can the var 'n' be held in another temp variable? Something like this - __pyx_v_5test1_n = 10; __pyx_v_5test1_const_n = 10; for (__pyx_v_5test1_i = 0; __pyx_v_5test1_i < __pyx_v_5test1_const_n; __pyx_v_5test1_i+=1) { __pyx_1 = (__pyx_v_5test1_i == 5); if (__pyx_1) { __pyx_v_5test1_n *= 2; goto __pyx_L4; } __pyx_L4:; } i--; I haven't looked at the cython code which parses the loop and outputs the c code. I would really appreciate if you can give me a head start into this and correct me if I am wrong. On Wed, Mar 11, 2009 at 2:57 PM, Dag Sverre Seljebotn < dagss at student.matnat.uio.no> wrote: > Since there's been some interest in how to get started coding on Cython > on the mailing list lately, I thought I'd call attention to these two > tickets: > > http://trac.cython.org/cython_trac/ticket/158 > http://trac.cython.org/cython_trac/ticket/203 > > These seems to fit the sweet spot of not being too hard to fix for > beginners (with some mentoring), but not being completely trivial either. > > I'm happy to provide an attack strategy/small amount of mentoring for > these tickets if anybody's interested. > > -- > Dag Sverre > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Regards, Prajwal S. "There is little difference in people,but this little difference makes a big difference.The little difference is attitude,the big difference is whether its positive or negative. " -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090311/e0ff98cc/attachment.htm From dagss at student.matnat.uio.no Wed Mar 11 19:44:19 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 19:44:19 +0100 Subject: [Cython] Beginner-friendly tickets In-Reply-To: References: <49B783FA.5070309@student.matnat.uio.no> Message-ID: <49B80683.4090800@student.matnat.uio.no> Prajwal Suhas P wrote: > Hi Dag, > > I am currently looking at the tickets. > > I was looking at this - > > cdef int i, n > n = 10 > for i in range(n): > if i == 5: > n *= 2 > print i > > > The generated c code for this is - > > __pyx_v_5test1_n = 10; > > for (__pyx_v_5test1_i = 0; __pyx_v_5test1_i < __pyx_v_5test1_n; > __pyx_v_5test1_i+=1) { > > __pyx_1 = (__pyx_v_5test1_i == 5); > if (__pyx_1) { > __pyx_v_5test1_n *= 2; > goto __pyx_L4; > } > __pyx_L4:; > } > > Can the var 'n' be held in another temp variable? > > Something like this - > > __pyx_v_5test1_n = 10; > __pyx_v_5test1_const_n = 10; > > for (__pyx_v_5test1_i = 0; __pyx_v_5test1_i < __pyx_v_5test1_const_n; > __pyx_v_5test1_i+=1) { > > __pyx_1 = (__pyx_v_5test1_i == 5); > if (__pyx_1) { > __pyx_v_5test1_n *= 2; > goto __pyx_L4; > } > __pyx_L4:; > } > i--; > > I haven't looked at the cython code which parses the loop and outputs > the c code. I would really appreciate if you can give me a head start > into this and correct me if I am wrong. You are correct, this is what you need to do. Some suggestions: a) Add a flag to ForFromStatNode in Nodes.py, say, "freeze_endpoints". b) IterationTransform in Optimize.py is what takes a "for i in range(...)" and turns it into a ForFromStatNode. Here you should turn the freeze_endpoints flag on when the ForFromStatNode is constructed. c) If this flag is set, then ForFromStatNode.generate_execution_code must put the start or end points (or both, depending on the case) into temporary variables. d) You can get hold of a temporary variable to use (like __pyx_t2) by calling code.funcstate.allocate_temp/release_temp. See Code.py for docs. (Stefan: Yes, this can be done by temp handling nodes in the transform as well, but since the canonical way of doing that is a bit unclear/a mess at the moment I don't want to encourage that. This works as well, and is likely less code in the end.) Another for-loop bug was also fixed with a bit of assistance on the mailing list, you may want to read through: http://thread.gmane.org/gmane.comp.python.cython.devel/4195 Also please make sure you make a testcase in tests/bugs/something_T203.pyx (then run this with python runtests.py -T203 while developing). -- Dag Sverre From dagss at student.matnat.uio.no Wed Mar 11 19:52:50 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 19:52:50 +0100 Subject: [Cython] Beginner-friendly tickets In-Reply-To: References: <49B783FA.5070309@student.matnat.uio.no> Message-ID: <49B80882.9030608@student.matnat.uio.no> Prajwal Suhas P wrote: > Something like this - > > __pyx_v_5test1_n = 10; > __pyx_v_5test1_const_n = 10; Acutally, it is more like __pyx_v_5test1_n = 10; __pyx_t1 = __pyx_v_5test1_n ... A better example than the one in the ticket is: """ >>> f() g called 0 1 2 2 """ cdef int g(): print "g called" return 3 def f(): cdef int i for i in range(g()): print i print i Here, "g called" will be printed three times if compiled in Cython; which is obviously not right. You should probably work with this example as well right away. Great that you're looking at this BTW. -- Dag Sverre From stefan_ml at behnel.de Wed Mar 11 20:01:51 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 11 Mar 2009 20:01:51 +0100 Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 In-Reply-To: References: <44706.213.61.181.86.1236700457.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49B80A9F.1000200@behnel.de> Hi Neal, Neal Becker wrote: > Can we have an rc2 with all proposed fixes? I'll plan to retest then. There's an rc3 now: http://cython.org/Cython-0.11.rc3.tar.gz Could you give it a try? If all works fine, this will become Cython 0.11 final. Thanks, Stefan From kwmsmith at gmail.com Wed Mar 11 21:22:51 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Wed, 11 Mar 2009 15:22:51 -0500 Subject: [Cython] Beginner-friendly tickets In-Reply-To: <49B783FA.5070309@student.matnat.uio.no> References: <49B783FA.5070309@student.matnat.uio.no> Message-ID: On Wed, Mar 11, 2009 at 4:27 AM, Dag Sverre Seljebotn wrote: > Since there's been some interest in how to get started coding on Cython > on the mailing list lately, I thought I'd call attention to these two > tickets: > > http://trac.cython.org/cython_trac/ticket/158 > http://trac.cython.org/cython_trac/ticket/203 > > These seems to fit the sweet spot of not being too hard to fix for > beginners (with some mentoring), but not being completely trivial either. > > I'm happy to provide an attack strategy/small amount of mentoring for > these tickets if anybody's interested. Hi Dag, I'm interested in working on #158 if it isn't already claimed. In the generated c code for these python functions: def nocrash1(): cdef str s = "Test" print "%s" % s def crash1(): print "%s" % s cdef str s = "Test" They differ only in that each line in the function body is converted into C in a different order. The internal string s is assigned to a non-null python object (the string "Test") in nocrash before being printed, while 's' is assiged to "Test" after trying to print in crash1. One solution (as mentioned in the ticket comments) would be to raise a compilation error since the s variable is declared after it's used. I imagine some checks would be in order to ensure this, but I don't have ideas about where to start. This function: def nocrash2(): print "%s" % s cdef str s Prints "None", although it should raise an exception since the analogous python code raises a "NameError." The generated C code puts the 'cdef str s' line before the 'print' line and sets s to None. This would be solved by the fix suggested above. The other solution involving flow analysis would be way beyond my abilities (or a bug fixable by a beginner). Kurt From dagss at student.matnat.uio.no Wed Mar 11 21:56:56 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 21:56:56 +0100 Subject: [Cython] Beginner-friendly tickets In-Reply-To: References: <49B783FA.5070309@student.matnat.uio.no> Message-ID: <49B82598.3000303@student.matnat.uio.no> Kurt Smith wrote: > On Wed, Mar 11, 2009 at 4:27 AM, Dag Sverre Seljebotn > wrote: >> Since there's been some interest in how to get started coding on Cython >> on the mailing list lately, I thought I'd call attention to these two >> tickets: >> >> http://trac.cython.org/cython_trac/ticket/158 >> http://trac.cython.org/cython_trac/ticket/203 >> >> These seems to fit the sweet spot of not being too hard to fix for >> beginners (with some mentoring), but not being completely trivial either. >> >> I'm happy to provide an attack strategy/small amount of mentoring for >> these tickets if anybody's interested. > > Hi Dag, > > I'm interested in working on #158 if it isn't already claimed. Nice! I haven't heard anything on #158 yet, so consider it yours. I've assigned both tickets now in trac. You may want to get Trac accounts if you haven't already BTW, send an .htpasswd-file to Robert for that. > One solution (as mentioned in the ticket comments) would be to raise a > compilation error since the s variable is declared after it's used. I > imagine some checks would be in order to ensure this, but I don't have > ideas about where to start. Yep, that's the solution I'm thinking about. The Cython "parse tree" is processed in stages. Two of those are analyse_declarations and analyse_expressions. The problem here is that "cdef str s" is handled in the analyse_declarations step and then disappears; while the analyse_expressions happen afterwards, and cannot know at which point in the source the declaration happened. So the analyse declarations phase must do this check. You will need to do your work in AnalyseDeclarationsTransform, in ParseTreeTransforms.py. It is a "filter" on the parse tree, and each method is called as you go according to the type of the node. Read some of the different transforms in that file to get the idea (to see the parse tree, insert e.g. "print node.dump()" at the beginning of visit_ModuleNode). Now: a) Keep a dict in self containing the names that has been referenced up to that point in the given scope (i.e. in visit_ModuleNode and visit_FuncDefNode you need to push/pop which dict one is using before/after processing the contents). b) Everywhere a name is used, a NameNode sits in the parse tree. So you create a visit_NameNode to record that the variable was used (it doesn't really matter whether it was assigned to or read; any usage of the variable before a cdef should be illegal). c) Finally, in visit_CVarDefNode you can now check whether the name has already been seen in the module/function, and if so there's a problem and you raise an exception (see other transforms for the convention here). Please ask again if/when you get stuck or anything was too brief. -- Dag Sverre From dagss at student.matnat.uio.no Wed Mar 11 21:59:40 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 11 Mar 2009 21:59:40 +0100 Subject: [Cython] Beginner-friendly tickets In-Reply-To: <49B82598.3000303@student.matnat.uio.no> References: <49B783FA.5070309@student.matnat.uio.no> <49B82598.3000303@student.matnat.uio.no> Message-ID: <49B8263C.6090706@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Kurt Smith wrote: >> On Wed, Mar 11, 2009 at 4:27 AM, Dag Sverre Seljebotn >> wrote: >>> Since there's been some interest in how to get started coding on Cython >>> on the mailing list lately, I thought I'd call attention to these two >>> tickets: >>> >>> http://trac.cython.org/cython_trac/ticket/158 >>> http://trac.cython.org/cython_trac/ticket/203 >>> >>> These seems to fit the sweet spot of not being too hard to fix for >>> beginners (with some mentoring), but not being completely trivial either. >>> >>> I'm happy to provide an attack strategy/small amount of mentoring for >>> these tickets if anybody's interested. >> Hi Dag, >> >> I'm interested in working on #158 if it isn't already claimed. > > Nice! I haven't heard anything on #158 yet, so consider it yours. I've > assigned both tickets now in trac. > > You may want to get Trac accounts if you haven't already BTW, send an > .htpasswd-file to Robert for that. > >> One solution (as mentioned in the ticket comments) would be to raise a >> compilation error since the s variable is declared after it's used. I >> imagine some checks would be in order to ensure this, but I don't have >> ideas about where to start. > > Yep, that's the solution I'm thinking about. > > The Cython "parse tree" is processed in stages. Two of those are > analyse_declarations and analyse_expressions. The problem here is that > "cdef str s" is handled in the analyse_declarations step and then > disappears; while the analyse_expressions happen afterwards, and cannot > know at which point in the source the declaration happened. > > So the analyse declarations phase must do this check. > > You will need to do your work in AnalyseDeclarationsTransform, in > ParseTreeTransforms.py. It is a "filter" on the parse tree, and each > method is called as you go according to the type of the node. Read some > of the different transforms in that file to get the idea (to see the > parse tree, insert e.g. "print node.dump()" at the beginning of > visit_ModuleNode). > > Now: > > a) Keep a dict in self containing the names that has been referenced up > to that point in the given scope (i.e. in visit_ModuleNode and > visit_FuncDefNode you need to push/pop which dict one is using > before/after processing the contents). Actually I meant to say "set" everywhere I said "dict". -- Dag Sverre From greg.ewing at canterbury.ac.nz Wed Mar 11 22:35:37 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 12 Mar 2009 10:35:37 +1300 Subject: [Cython] Update on: Incomplete C++ example In-Reply-To: <99214b470903101433w2cb1a2f4x896fd11203f97a14@mail.gmail.com> References: <99214b470903101433w2cb1a2f4x896fd11203f97a14@mail.gmail.com> Message-ID: <49B82EA9.2030306@canterbury.ac.nz> Luke wrote: > it would be nice to have a step by step example of > the process that is illustrative of how to wrap a C++ library Generally agreed, but > Things that were not helpful in the setup.py were things like: > include_dirs=[...], # usual stuff > libraries=[...], # ditto > extra_link_args=[...], # if needed None of that is specific to wrapping C++, it's all just general distutils knowledge. I don't think it's reasonable to expect a C++ wrapping tutorial to include complete handholding on all aspects of distutils. A self-contained example wouldn't need any of those options, anyway. -- Greg From robertwb at math.washington.edu Wed Mar 11 23:51:26 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Mar 2009 15:51:26 -0700 Subject: [Cython] too late to review and push patch for better handling of integral C types ? In-Reply-To: <49B584AC.6040506@behnel.de> References: <49B584AC.6040506@behnel.de> Message-ID: On Mar 9, 2009, at 2:05 PM, Stefan Behnel wrote: > Lisandro Dalcin wrote: >> Sending again, former message bounced because of 40k limit. >> >> The patch is almost ready, just waiting for review. The only thing I >> did not take is to remove the usage of PyNumber_Int(). Anyway, Cython >> already uses it, so the "brokeness" of my patch is similar to the >> current one :-) .. As I've introduced a __Pyx_PyNumber_Int() >> function, >> that is the only place that we should change to implement things >> as we >> want. > > I skipped through your patch a little. It's a fairly involved > change that > requires some testing, especially on different architectures and > platforms. > It might well be that the user experience is positive in that it > helps in > finding bugs and making user code more portable. My current gut > feeling is > that code that depends on the old behaviour here is broken anyway > and just > wasn't fixed yet because no-one noticed. > > But in any case, I would like to avoid breaking code for a micro > (bug-fix!) > release, unless there is at least some evidence that users > appreciate it. > > So, if anyone has code that does a bit of integer conversion > between Python > and C space, especially when working on a non-Intel or 64 bit > platform, > please give this patch a try. Sage does this :). It's too late for 0.11 though. Is there a ticket up? - Robert From robertwb at math.washington.edu Wed Mar 11 23:54:29 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Mar 2009 15:54:29 -0700 Subject: [Cython] Update on: Incomplete C++ example In-Reply-To: <49B82EA9.2030306@canterbury.ac.nz> References: <99214b470903101433w2cb1a2f4x896fd11203f97a14@mail.gmail.com> <49B82EA9.2030306@canterbury.ac.nz> Message-ID: <613B0D07-81E4-4CBE-8BE5-388DFF0DDF21@math.washington.edu> On Mar 11, 2009, at 2:35 PM, Greg Ewing wrote: > Luke wrote: > >> it would be nice to have a step by step example of >> the process that is illustrative of how to wrap a C++ library > > Generally agreed, but > >> Things that were not helpful in the setup.py were things like: >> include_dirs=[...], # usual stuff >> libraries=[...], # ditto >> extra_link_args=[...], # if needed > > None of that is specific to wrapping C++, it's all > just general distutils knowledge. I don't think it's > reasonable to expect a C++ wrapping tutorial to > include complete handholding on all aspects of > distutils. > > A self-contained example wouldn't need any of those > options, anyway. What we should have is a link to the distutils documentation here. Part of "how to wrap a C++ library" is "how to wrap a library." - Robert From jasone at canonware.com Wed Mar 11 23:57:14 2009 From: jasone at canonware.com (Jason Evans) Date: Wed, 11 Mar 2009 15:57:14 -0700 Subject: [Cython] Possible blocker for 0.11 Message-ID: <49B841CA.8000805@canonware.com> I hit what appears to be a Cython compiler bug this afternoon (related to temp allocation?) that might be important to fix before 0.11 is released. I haven't had time yet to reduce it to a small test case. What is the tentative release schedule? Thanks, Jason From robertwb at math.washington.edu Thu Mar 12 00:01:13 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Mar 2009 16:01:13 -0700 Subject: [Cython] cython-0.11.rc: Updated tar, test fails (but gets further) In-Reply-To: <5770E6DE-385E-46BE-9070-0AD57466C5A1@math.washington.edu> References: <52637.213.61.181.86.1236695813.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <61316.213.61.181.86.1236699523.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <5770E6DE-385E-46BE-9070-0AD57466C5A1@math.washington.edu> Message-ID: On Mar 10, 2009, at 11:08 AM, Robert Bradshaw wrote: > I'll fix this. The problem is that when passing via the inline > function, the size and signness of the original index gets lost. The > very common and much faster case is to use PyInt_FromLong, but of > course we have to handle bigger indices as well. > > - Robert > > > On Mar 10, 2009, at 9:19 AM, Lisandro Dalcin wrote: > >> Here you have a trivial patch... However, all this part smells so >> bad... >> >> >> On Tue, Mar 10, 2009 at 1:38 PM, Stefan Behnel >> wrote: >>> Lisandro Dalcin wrote: >>>>>> Now testing at least starts, but seems to have some errors: >>>>> >>>>> * warning: left shift count >= width of type Note that this is only called when sizeof(Py_ssize_t) < sizeof (unsigned long long). >>>>> I don't get these locally. The problem is in __Pyx_GetItemGeneric, >>>>> second >>>>> line: "1ULL << (sizeof(Py_ssize_t)*8)". Looks like that's not >>>>> 64bit >>>>> clean. >>>> >>>> PyInt_FromSsize_t() should be used here.... Well, it's a bit more complicated than this. However, I will admit, though I tried to cover all the cases, the original code was quite hackish. I've come up with a better solution that doesn't sacrifice speed for the common cases. RC3 is up for testing. - Robert From robertwb at math.washington.edu Thu Mar 12 00:04:50 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Mar 2009 16:04:50 -0700 Subject: [Cython] Possible blocker for 0.11 In-Reply-To: <49B841CA.8000805@canonware.com> References: <49B841CA.8000805@canonware.com> Message-ID: On Mar 11, 2009, at 3:57 PM, Jason Evans wrote: > I hit what appears to be a Cython compiler bug this afternoon (related > to temp allocation?) that might be important to fix before 0.11 is > released. I haven't had time yet to reduce it to a small test case. > What is the tentative release schedule? We're hoping that the latest rc fixes Neal's issues with building/ testing RMPs, and rc3 will be the final release. That being said, please provide an example and make a trac ticket. Unless it's really bad, we'll probably wait until 0.11.1, which I hope will be released on a much shorter timescale. - Robert From jasone at canonware.com Thu Mar 12 00:10:32 2009 From: jasone at canonware.com (Jason Evans) Date: Wed, 11 Mar 2009 16:10:32 -0700 Subject: [Cython] Possible blocker for 0.11 In-Reply-To: References: <49B841CA.8000805@canonware.com> Message-ID: <49B844E8.8050301@canonware.com> Robert Bradshaw wrote: > We're hoping that the latest rc fixes Neal's issues with building/ > testing RMPs, and rc3 will be the final release. > > That being said, please provide an example and make a trac ticket. Okay, I'm working on it now. > Unless it's really bad, we'll probably wait until 0.11.1, which I > hope will be released on a much shorter timescale. It's causing a reference to an uninitialized temp variable (which should contain the result of a conditional branch test), which causes my app to crash, so this particular manifestation is pretty bad. However, it isn't yet clear whether this is a common problem, or a corner case due to the structure of this particular (rather simple) method. Thanks, Jason From robertwb at math.washington.edu Thu Mar 12 00:17:21 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Mar 2009 16:17:21 -0700 Subject: [Cython] PATCH: the right way to use PyErr_Format with Py_ssize_t arguments In-Reply-To: References: <49B7CDF6.3020401@student.matnat.uio.no> Message-ID: <2E671135-5B33-4551-82FB-183486183E88@math.washington.edu> On Mar 11, 2009, at 9:01 AM, Lisandro Dalcin wrote: > On Wed, Mar 11, 2009 at 12:43 PM, Dag Sverre Seljebotn > wrote: >> Lisandro Dalcin wrote: >>> Likely Dag should review this... If you agree on the patch, please >>> push ... IMHO, this fix should go to 0.11 The patch looks good, can you make a ticket for it? Also, can you use mercurial to output diffs so we get your name and a comment with it as well? >> a) Saying that this should go into 0.11 means saying that this >> specific >> bug is more important than everything else that sits in milestone >> 0.11.1 >> in Trac. We probably put out 0.10 with the same bug and the Cython >> project survived! We need to have some disipline and get things >> released >> -- release early, release often. Yes, we need to return to the "release early, release often" schedule. We can start putting patches up and getting reviews for them on trac for 0.11.1 right away. I've added milestone 0.11.2, everything that doesn't have a patch or easy fix for 0.11.1 will get moved there so we can release 0.11.1 shortly. > I understand that... I just wanted to point that this fix likely does > not have high impact and fix 64 bits issues, more or less like recent > Robert's pushes did. The last push was to facilitate getting cython 0.11 into the rpm repo, though in retrospect I guess it's lower priority than some other stuff. At this point, unless there's a major bug, the only thing I can see adding is perhaps testcases. - Robert From jasone at canonware.com Thu Mar 12 00:36:18 2009 From: jasone at canonware.com (Jason Evans) Date: Wed, 11 Mar 2009 16:36:18 -0700 Subject: [Cython] Possible blocker for 0.11 In-Reply-To: <49B841CA.8000805@canonware.com> References: <49B841CA.8000805@canonware.com> Message-ID: <49B84AF2.7010300@canonware.com> Jason Evans wrote: > I hit what appears to be a Cython compiler bug this afternoon (related > to temp allocation?) that might be important to fix before 0.11 is > released. False alarm; there's a bug in my program. I misinterpreted valgrind output, then misread Cython-generated code. Cython is dutifully giving me the wrong answer I asked for. Jason From robertwb at math.washington.edu Thu Mar 12 00:38:51 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Mar 2009 16:38:51 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: Message-ID: On Mar 10, 2009, at 7:35 PM, Carl Witty wrote: > I just reported http://trac.cython.org/cython_trac/ticket/229, which > points out that division and % on cdef ints have C rather than Python > semantics (for instance, (-1 % 16) is -1 in C, and 15 in Python). > > I'm not sure what the right thing to do here is. Using C semantics > gives faster code. Using Python semantics is nicer (whenever I care > about the difference, I always want Python semantics) but slower, and > would be confusing to somebody who was coming from C. Changing this to use Python semantics would actually slow down real code in this case. I'm thinking specifically of linear algebra mod n (for word sized n) in Sage, but I'm sure there are many other places as well. So I'm generally -1 for such a change. I also don't like the idea of an new operator (%%), and I think Carl Witty has a good point about needing to change the semantics of // as well. Also, I think making literals into Python objects by default will be way cumbersome. > But whatever happens should be prominently documented; I couldn't find > any mention of the issue in the current docs (although maybe I wasn't > looking in the right place). +10 I think this is neither the first nor last issue that will come up like this. What I think we should do is have a "-pedantic" flag which obeys Python semantics exactly, at the expense of speed. This applies to %, //, **, etc. (Overflowing is a separate issue, and I think there should be a flag for that too.) Compilation of .py files would be, by default, with -pedantic enabled, and compilation of .pyx files with it disabled. If one suspects that something subtle like this is changing your algorithm, you can compile with -pedantic and confirm (or deny) your suspicions. Also, everything that differs will query this flag in the code, making it easy to find and *document* the discrepancies. In fact, the annotate feature could even make note of such places. Thoughts? - Robert P.S. As an implementation aside, it can be much faster to write r = a % b; r += (r < 0) * b than r = a % b; if r < 0: r += b especially if one is doing this in a loop with randomly distributed (pos/neg) a. From robertwb at math.washington.edu Thu Mar 12 00:42:11 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Mar 2009 16:42:11 -0700 Subject: [Cython] Possible blocker for 0.11 In-Reply-To: <49B84AF2.7010300@canonware.com> References: <49B841CA.8000805@canonware.com> <49B84AF2.7010300@canonware.com> Message-ID: <8800314B-D309-4D09-AEA5-4FC6C67135D8@math.washington.edu> On Mar 11, 2009, at 4:36 PM, Jason Evans wrote: > Jason Evans wrote: >> I hit what appears to be a Cython compiler bug this afternoon >> (related >> to temp allocation?) that might be important to fix before 0.11 is >> released. > > False alarm; there's a bug in my program. I misinterpreted valgrind > output, then misread Cython-generated code. Cython is dutifully > giving > me the wrong answer I asked for. Well that's good news from my point of view :). Thanks for the follow up. - Robert From robertwb at math.washington.edu Thu Mar 12 00:42:38 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Mar 2009 16:42:38 -0700 Subject: [Cython] Wrapping C++ functions that pass by reference In-Reply-To: <49B7B707.4000303@student.matnat.uio.no> References: <99214b470903101634i16f79657pe008fd1bc75a9533@mail.gmail.com> <49B7AEFF.1000302@molden.no> <49B7B707.4000303@student.matnat.uio.no> Message-ID: On Mar 11, 2009, at 6:05 AM, Dag Sverre Seljebotn wrote: > Sturla Molden wrote: >> Luke wrote: >> >>> I have a C++ function like this: >>> >>> void constraint(double& x, double& dxdt) >>> { >>> dxdt[0] = ... >>> dxdt[1] = ... >>> .... >>> dxdt[n] = ... >>> } >>> >>> How would I properly wrap this function in the .pyx file? And how >>> would calling it in Python work? >>> >> I think you can just say >> >> cdef extern from "constraint.h": >> void constraint(double x, double dxdt) >> >> as the generated C++ should be identical. >> > Good point! Please disregard my previous mail about this, I was wrong, > and this is right. Is this documented in the wiki anywhere? From robertwb at math.washington.edu Thu Mar 12 00:43:18 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Mar 2009 16:43:18 -0700 Subject: [Cython] Remove "parsetuple_format" from PyrexTypes? In-Reply-To: <49B6C647.9090807@behnel.de> References: <49B6C647.9090807@behnel.de> Message-ID: On Mar 10, 2009, at 12:57 PM, Stefan Behnel wrote: > Hi, > > I just fixed an unused variable problem when using (Py_s)size_t as > function > argument types (fix can wait for 0.11.1). Doing this, I noticed > that the > type indication character that each type stores in > "parsetuple_format" is > no longer used anywhere since PyArg_ParseTupleAndKeywords() was > replaced by > our own code. Before I rip this attribute straight out, can anyone > still > see a use case for this? Maybe in combination with buffer content > types? Yes, I think we can get rid of this. - Robert From robertwb at math.washington.edu Thu Mar 12 00:56:00 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Mar 2009 16:56:00 -0700 Subject: [Cython] Integer power operator In-Reply-To: <49B7892E.2010901@student.matnat.uio.no> References: <1286d6e2177fb46620ab4a1f1fa7d7f3.squirrel@webmail.uio.no> <39764.213.61.181.86.1236762819.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B78660.8060904@student.matnat.uio.no> <49B7892E.2010901@student.matnat.uio.no> Message-ID: On Mar 11, 2009, at 2:49 AM, Dag Sverre Seljebotn wrote: > Dag Sverre Seljebotn wrote: >> Stefan Behnel wrote: >>> Dag Sverre Seljebotn wrote: >>>> I happen to routinely write (some complex integer expression)**2 in >>>> Python, and it is tedious to manually store the expression in a >>>> temporary >>>> etc. in Cython. >>> >>> If you can come up with a sensible utility function that handles the >>> different C compile time and runtime overflow cases nicely, I'm >>> all for >>> it. >> >> So you are "+1 if I include an overflow check"? >> >> I was just thinking of letting it silently overflow, the way addition >> and multiplication already silently overflows. >> > > To be clear, I'm in favour of overflow checks, I just think they in > that > case should apply in *all* situations, not simply in this one > operator. > Otherwise Cython will be inconsistent and it makes the language harder > to learn. > > So I'm thinking this: > 1) Add integer ** without overflow check > 2) Add overflowcheck compiler directive, similar to the nonecheck > directive, which checks for overflows in /all/ situations (which > defaults to off though) +1 We should special constant case base 2 (or perhaps base power of 2) and constant exponent up to, say, 5 or so. - Robert From carl.witty at gmail.com Thu Mar 12 01:14:31 2009 From: carl.witty at gmail.com (Carl Witty) Date: Wed, 11 Mar 2009 17:14:31 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: Message-ID: On Wed, Mar 11, 2009 at 4:38 PM, Robert Bradshaw wrote: > I think this is neither the first nor last issue that will come up > like this. What I think we should do is have a "-pedantic" flag which > obeys Python semantics exactly, at the expense of speed. This applies > to %, //, **, etc. (Overflowing is a separate issue, and I think > there should be a flag for that too.) Compilation of .py files would > be, by default, with -pedantic enabled, and compilation of .pyx files > with it disabled. > > If one suspects that something subtle like this is changing your > algorithm, you can compile with -pedantic and confirm (or deny) your > suspicions. Also, everything that differs will query this flag in the > code, making it easy to find and *document* the discrepancies. In > fact, the annotate feature could even make note of such places. > > Thoughts? This doesn't feel like quite the right plan... Consider a file which is edited by both an expert (defined as "somebody who happens to remember that C and Python have different semantics for signed division") and a novice (defined as "somebody who doesn't remember about any signed division issues"; note that the same person can be an expert one day and a novice the next). If the novice is going to be switching -pedantic on and off, doesn't that mean that the expert had better write code that works either way? That seems worse than being able to depend on one behavior or the other. Maybe this would be OK combined with builtins ("floordiv"/"floormod" and "truncdiv"/"truncmod", perhaps) that were guaranteed not to change with -pedantic. Carl From robertwb at math.washington.edu Thu Mar 12 03:32:28 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Mar 2009 19:32:28 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: Message-ID: On Mar 11, 2009, at 5:14 PM, Carl Witty wrote: > On Wed, Mar 11, 2009 at 4:38 PM, Robert Bradshaw > wrote: >> I think this is neither the first nor last issue that will come up >> like this. What I think we should do is have a "-pedantic" flag which >> obeys Python semantics exactly, at the expense of speed. This applies >> to %, //, **, etc. (Overflowing is a separate issue, and I think >> there should be a flag for that too.) Compilation of .py files would >> be, by default, with -pedantic enabled, and compilation of .pyx files >> with it disabled. >> >> If one suspects that something subtle like this is changing your >> algorithm, you can compile with -pedantic and confirm (or deny) your >> suspicions. Also, everything that differs will query this flag in the >> code, making it easy to find and *document* the discrepancies. In >> fact, the annotate feature could even make note of such places. >> >> Thoughts? > > This doesn't feel like quite the right plan... Consider a file which > is edited by both an expert (defined as "somebody who happens to > remember that C and Python have different semantics for signed > division") and a novice (defined as "somebody who doesn't remember > about any signed division issues"; note that the same person can be an > expert one day and a novice the next). If the novice is going to be > switching -pedantic on and off, doesn't that mean that the expert had > better write code that works either way? That seems worse than being > able to depend on one behavior or the other. I wasn't imagining something people would be switch on and off... it would be more of a useful tool for when people are converting Python code to Cython code. We could make this a pragma too. > Maybe this would be OK combined with builtins ("floordiv"/"floormod" > and "truncdiv"/"truncmod", perhaps) that were guaranteed not to change > with -pedantic. Yes, I think we could provide such functions as well, though it may be cumbersome to use them. Surrounding functions/blocks by pragmas would do the trick as well, in case one needed to depend on a certain behavior. Most of this stuff is corner cases that are just slower to test/special case for, and people wouldn't be using Cython if they didn't care about speed. One of the primary advantages of this approach is that it would facilitate very good documentation about exactly what (and why) there are any discrepancies between pure Python and Cython code--and yes I'm making the assumption that sometimes it's worth doing some things differently with C types for efficiency reasons. - Robert From stefan_ml at behnel.de Thu Mar 12 07:13:43 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 12 Mar 2009 07:13:43 +0100 Subject: [Cython] too late to review and push patch for better handling of integral C types ? In-Reply-To: References: <49B584AC.6040506@behnel.de> Message-ID: <49B8A817.1090609@behnel.de> Robert Bradshaw wrote: > It's too late for 0.11 though. Is there a ticket up? Don't think so, but while we're at it, I'd like to have a ticket for any change going into 0.11.1. Lisandro, could you open one and post the patch there? Stefan From stefan_ml at behnel.de Thu Mar 12 07:20:31 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 12 Mar 2009 07:20:31 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: Message-ID: <49B8A9AF.6060806@behnel.de> Robert Bradshaw wrote: > I think this is neither the first nor last issue that will come up > like this. What I think we should do is have a "-pedantic" flag which > obeys Python semantics exactly, at the expense of speed. This applies > to %, //, **, etc. (Overflowing is a separate issue, and I think > there should be a flag for that too.) Compilation of .py files would > be, by default, with -pedantic enabled, and compilation of .pyx files > with it disabled. Wouldn't it be better to have such a flag emit warnings (preferably with a hint how to write better code) instead of changing the semantics of operators? I think having operators change their semantics without a clear change in the user code (such as a __future__ import) is a rather dangerous 'feature'. Stefan From stefan_ml at behnel.de Thu Mar 12 07:39:20 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 12 Mar 2009 07:39:20 +0100 Subject: [Cython] Trac usage and bug handling for 0.11.x Message-ID: <49B8AE18.9040109@behnel.de> Hi all, I would like to encourage the following bug fixing procedure for the future 0.11.x releases. 1) Every patch that goes into a bug-fix release of Cython must be backed by a trac ticket. To get a trac account, please send a htpasswd entry to Robert Bradshaw. 2) Every trac ticket must have a bug test case associated with it. This means that there must be a failing test case in tests/bugs/ named "nicely_descriptive_name_here_Txyz.pyx" (where 'xyz' is the ticket number), or a patch in trac that adds this test case. See http://wiki.cython.org/HackerGuide Please try to do this even for the tricky cases that feel like there isn't a good test case. Reproducing a bug is critical for fixing it, and having a test case is critical for not breaking it in the future. 3) A working patch attached to the ticket will definitely accelerate the mainline bug fixing, although if you are unsure where to get started, it's usually better to ask on the mailing list before getting lost. 4) The Cython developers will assign priorities and milestones to trac tickets. Tickets with working patches should be handled with priority and should at least receive a timely review. 5) When a bug is fixed, the test case must be copied from tests/bugs/ to tests/run/ and the test suite must pass. Can everyone agree on this? Stefan From robertwb at math.washington.edu Thu Mar 12 07:46:41 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Mar 2009 23:46:41 -0700 Subject: [Cython] Trac usage and bug handling for 0.11.x In-Reply-To: <49B8AE18.9040109@behnel.de> References: <49B8AE18.9040109@behnel.de> Message-ID: <3BE155F1-FE7E-41AA-99D8-E53A7188E70F@math.washington.edu> On Mar 11, 2009, at 11:39 PM, Stefan Behnel wrote: > Hi all, > > I would like to encourage the following bug fixing procedure for > the future > 0.11.x releases. > > 1) Every patch that goes into a bug-fix release of Cython must be > backed by > a trac ticket. To get a trac account, please send a htpasswd entry to > Robert Bradshaw. > > 2) Every trac ticket must have a bug test case associated with it. > This > means that there must be a failing test case in tests/bugs/ named > "nicely_descriptive_name_here_Txyz.pyx" (where 'xyz' is the ticket > number), > or a patch in trac that adds this test case. See > > http://wiki.cython.org/HackerGuide > > Please try to do this even for the tricky cases that feel like > there isn't > a good test case. Reproducing a bug is critical for fixing it, and > having a > test case is critical for not breaking it in the future. > > 3) A working patch attached to the ticket will definitely > accelerate the > mainline bug fixing, although if you are unsure where to get > started, it's > usually better to ask on the mailing list before getting lost. > > 4) The Cython developers will assign priorities and milestones to trac > tickets. Tickets with working patches should be handled with > priority and > should at least receive a timely review. > > 5) When a bug is fixed, the test case must be copied from tests/ > bugs/ to > tests/run/ and the test suite must pass. +1. In fact, I think even non-bugfixes should all be assigned a ticket on trac. (Some features may, of course, span many patches.) The only thing I would say is that I'd rather not clutter tests/run/-- perhaps we could put them somewhere else (or alternatively have runtests test all but a static list of known open tickets). - Robert From carl.witty at gmail.com Thu Mar 12 07:53:52 2009 From: carl.witty at gmail.com (Carl Witty) Date: Wed, 11 Mar 2009 23:53:52 -0700 Subject: [Cython] Trac usage and bug handling for 0.11.x In-Reply-To: <49B8AE18.9040109@behnel.de> References: <49B8AE18.9040109@behnel.de> Message-ID: On Wed, Mar 11, 2009 at 11:39 PM, Stefan Behnel wrote: > 2) Every trac ticket must have a bug test case associated with it. This > means that there must be a failing test case in tests/bugs/ named > "nicely_descriptive_name_here_Txyz.pyx" (where 'xyz' is the ticket number), > or a patch in trac that adds this test case. See > > http://wiki.cython.org/HackerGuide > > Please try to do this even for the tricky cases that feel like there isn't > a good test case. Reproducing a bug is critical for fixing it, and having a > test case is critical for not breaking it in the future. Does that mean I can't report bugs to trac anymore, without constructing a test case in the form of a Cython patch? If so, then -0. (I probably just wouldn't bother reporting bugs any more under such a rule.) If you just mean that every patch that purports to fix a bug must also include a test case, then +1 to that (and to the rest of your list). Carl From robertwb at math.washington.edu Thu Mar 12 07:56:02 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 11 Mar 2009 23:56:02 -0700 Subject: [Cython] Google summer of code In-Reply-To: <993FE1B3-30D3-45C7-9E40-B8C420E2492C@math.washington.edu> References: <49B2E58A.9090900@behnel.de> <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> <993FE1B3-30D3-45C7-9E40-B8C420E2492C@math.washington.edu> Message-ID: <253B6623-3D04-47F7-B870-A74E420C7D6E@math.washington.edu> On Mar 7, 2009, at 3:11 PM, Robert Bradshaw wrote: > I'm a "probably," but it's sounding like at this point we don't have > enough potential mentors to be a full mentoring organization unless > someone else stands up. On the numerical side of things, perhaps a > NumPy person would be willing. > > On the other hand, I think we're higher on Python's radar than last > year, so I'm hopeful of at least getting one (maybe two) spots for > them. The quality of the students and projects will play a huge role > here--one thing I learned at the summit last year is that it's not > even worth you time to go after sub-par projects (though good > students are an extremely good return on the investment of mentoring > them). Just a reminder, the organization app is due Friday if we decide to do it. Another advantage of being a mentor organization is that we get $500/ student. I'm not exactly sure what Cython would do with the money at this point (we don't have infrastructure to support, it's not enough to fund a get-together) I would propose it would be divvied out to the mentors to help offset the time they put into it. (They Python Foundation keeps the $500/student, which I'm fine with as they're big enough to have actual costs.) But being under an umbrella organization worked out well last time and is certainly less work and more flexibility on our part. - Robert From dagss at student.matnat.uio.no Thu Mar 12 07:59:50 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 12 Mar 2009 07:59:50 +0100 (CET) Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: Message-ID: Robert Bradshaw wrote: > On Mar 10, 2009, at 7:35 PM, Carl Witty wrote: > >> I just reported http://trac.cython.org/cython_trac/ticket/229, which >> points out that division and % on cdef ints have C rather than Python >> semantics (for instance, (-1 % 16) is -1 in C, and 15 in Python). >> >> I'm not sure what the right thing to do here is. Using C semantics >> gives faster code. Using Python semantics is nicer (whenever I care >> about the difference, I always want Python semantics) but slower, and >> would be confusing to somebody who was coming from C. > > Changing this to use Python semantics would actually slow down real > code in this case. I'm thinking specifically of linear algebra mod n > (for word sized n) in Sage, but I'm sure there are many other places > as well. So I'm generally -1 for such a change. I also don't like the > idea of an new operator (%%), and I think Carl Witty has a good point > about needing to change the semantics of // as well. Also, I think > making literals into Python objects by default will be way cumbersome. > >> But whatever happens should be prominently documented; I couldn't find >> any mention of the issue in the current docs (although maybe I wasn't >> looking in the right place). > > +10 > > I think this is neither the first nor last issue that will come up > like this. What I'd think we should do is have a "-pedantic" flag which > obeys Python semantics exactly, at the expense of speed. This applies > to %, //, **, etc. (Overflowing is a separate issue, and I think > there should be a flag for that too.) Compilation of .py files would > be, by default, with -pedantic enabled, and compilation of .pyx files > with it disabled. Like Carl I have problem with a pedantic flag because it gives two "modes" to worry about (which mode is this code written in? etc.) and just headache in general. So I'm much more in favor of a specific pragma for this (cython.fastdivision), which can be inserted in source visibly as a decorator (as much for documentation purposes as anything else). (Then the disagreement boils down to what should be the default value of that pragma, I suppose.). Dag Sverre From robertwb at math.washington.edu Thu Mar 12 08:02:09 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 12 Mar 2009 00:02:09 -0700 Subject: [Cython] Trac usage and bug handling for 0.11.x In-Reply-To: References: <49B8AE18.9040109@behnel.de> Message-ID: <77B03456-43CB-4ED8-A087-922B85F260DC@math.washington.edu> On Mar 11, 2009, at 11:53 PM, Carl Witty wrote: > On Wed, Mar 11, 2009 at 11:39 PM, Stefan Behnel > wrote: >> 2) Every trac ticket must have a bug test case associated with it. >> This >> means that there must be a failing test case in tests/bugs/ named >> "nicely_descriptive_name_here_Txyz.pyx" (where 'xyz' is the ticket >> number), >> or a patch in trac that adds this test case. See >> >> http://wiki.cython.org/HackerGuide >> >> Please try to do this even for the tricky cases that feel like >> there isn't >> a good test case. Reproducing a bug is critical for fixing it, and >> having a >> test case is critical for not breaking it in the future. > > Does that mean I can't report bugs to trac anymore, without > constructing a test case in the form of a Cython patch? If so, then > -0. (I probably just wouldn't bother reporting bugs any more under > such a rule.) My interpretation is that bugs won't get assigned a release milestone without a clear testcase. This will probably impact how much it's worked on as well. The testcase does not need to be in the form of a patch (until we actually commit it). If I work on a trac ticket with an "inline" testcase, I'll probably go ahead and add it to the /tests/ bugs directory myself. > If you just mean that every patch that purports to fix a bug must also > include a test case, then +1 to that (and to the rest of your list). Yes, for sure on this point. - Robert From stefan_ml at behnel.de Thu Mar 12 08:03:41 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 12 Mar 2009 08:03:41 +0100 Subject: [Cython] Trac usage and bug handling for 0.11.x In-Reply-To: References: <49B8AE18.9040109@behnel.de> Message-ID: <49B8B3CD.4020308@behnel.de> Carl Witty wrote: > On Wed, Mar 11, 2009 at 11:39 PM, Stefan Behnel wrote: >> 2) Every trac ticket must have a bug test case associated with it. This >> means that there must be a failing test case in tests/bugs/ named >> "nicely_descriptive_name_here_Txyz.pyx" (where 'xyz' is the ticket number), >> or a patch in trac that adds this test case. See >> >> http://wiki.cython.org/HackerGuide >> >> Please try to do this even for the tricky cases that feel like there isn't >> a good test case. Reproducing a bug is critical for fixing it, and having a >> test case is critical for not breaking it in the future. > > Does that mean I can't report bugs to trac anymore, without > constructing a test case in the form of a Cython patch? If so, then > -0. (I probably just wouldn't bother reporting bugs any more under > such a rule.) No, reporting bugs is open to everyone. The above doesn't mean the reporter of a bug *must* be the one to provide the test case. I just wanted to stress the fact that we need a test case that must go in before the fix, and that a test case makes it easier for us to reproduce and fix the bug. So there is a reason to prioritise tickets that contain a test case, as tickets without a test case are more work. I'll clarify that on a Wiki page later today. Stefan From dagss at student.matnat.uio.no Thu Mar 12 08:07:36 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 12 Mar 2009 08:07:36 +0100 (CET) Subject: [Cython] Google summer of code In-Reply-To: <253B6623-3D04-47F7-B870-A74E420C7D6E@math.washington.edu> References: <49B2E58A.9090900@behnel.de> <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> <993FE1B3-30D3-45C7-9E40-B8C420E2492C@math.washington.edu> <253B6623-3D04-47F7-B870-A74E420C7D6E@math.washington.edu> Message-ID: <6528b00c5b8779160ffe2dc3ec1f2cfb.squirrel@webmail.uio.no> > Robert Bradshaw wrote: > On Mar 7, 2009, at 3:11 PM, Robert Bradshaw wrote: > >> I'm a "probably," but it's sounding like at this point we don't have >> enough potential mentors to be a full mentoring organization unless >> someone else stands up. On the numerical side of things, perhaps a >> NumPy person would be willing. >> >> On the other hand, I think we're higher on Python's radar than last >> year, so I'm hopeful of at least getting one (maybe two) spots for >> them. The quality of the students and projects will play a huge role >> here--one thing I learned at the summit last year is that it's not >> even worth you time to go after sub-par projects (though good >> students are an extremely good return on the investment of mentoring >> them). > > Just a reminder, the organization app is due Friday if we decide to > do it. > > Another advantage of being a mentor organization is that we get $500/ > student. I'm not exactly sure what Cython would do with the money at > this point (we don't have infrastructure to support, it's not enough > to fund a get-together) I would propose it would be divvied out to > the mentors to help offset the time they put into it. (They Python > Foundation keeps the $500/student, which I'm fine with as they're big > enough to have actual costs.) But being under an umbrella > organization worked out well last time and is certainly less work and > more flexibility on our part. This was up on the NumPy list -- they basically decided to go with the PSF for lack of manpower to put together an application (though there were arguments in favor of staying with the PSF anyway). Anyway, if one still wants to apply one could consider doing it together with NumPy in order to have a larger mentor base. I can't help that much myself though (polishing the project list and so on I could always do, but not submitting the actual application). Dag Sverre From dagss at student.matnat.uio.no Thu Mar 12 08:12:06 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 12 Mar 2009 08:12:06 +0100 (CET) Subject: [Cython] Trac usage and bug handling for 0.11.x In-Reply-To: <49B8B3CD.4020308@behnel.de> References: <49B8AE18.9040109@behnel.de> <49B8B3CD.4020308@behnel.de> Message-ID: Stefan wrote: > > Carl Witty wrote: >> On Wed, Mar 11, 2009 at 11:39 PM, Stefan Behnel >> wrote: >>> 2) Every trac ticket must have a bug test case associated with it. This >>> means that there must be a failing test case in tests/bugs/ named >>> "nicely_descriptive_name_here_Txyz.pyx" (where 'xyz' is the ticket >>> number), >>> or a patch in trac that adds this test case. See >>> >>> http://wiki.cython.org/HackerGuide >>> >>> Please try to do this even for the tricky cases that feel like there >>> isn't >>> a good test case. Reproducing a bug is critical for fixing it, and >>> having a >>> test case is critical for not breaking it in the future. >> >> Does that mean I can't report bugs to trac anymore, without >> constructing a test case in the form of a Cython patch? If so, then >> -0. (I probably just wouldn't bother reporting bugs any more under >> such a rule.) > > No, reporting bugs is open to everyone. The above doesn't mean the > reporter > of a bug *must* be the one to provide the test case. I just wanted to > stress the fact that we need a test case that must go in before the fix, > and that a test case makes it easier for us to reproduce and fix the bug. > So there is a reason to prioritise tickets that contain a test case, as > tickets without a test case are more work. > > I'll clarify that on a Wiki page later today. Could we also put in something about which repositories to use for the various releases? I.e. some questions: - Will we use cython-devel or cython for 0.11.1? - If I do something that's just cleanup and not bugfix (e.g. convert NameNode to new temps), should that go into 0.12 or 0.11.1? I'm in favor of 0.11.x going in cython and NOT contain cleanup, only bugfixes, but I'm really OK with either. Dag Sverre From robertwb at math.washington.edu Thu Mar 12 08:12:49 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 12 Mar 2009 00:12:49 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49B8A9AF.6060806@behnel.de> References: <49B8A9AF.6060806@behnel.de> Message-ID: <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> On Mar 11, 2009, at 11:20 PM, Stefan Behnel wrote: > Robert Bradshaw wrote: >> I think this is neither the first nor last issue that will come up >> like this. What I think we should do is have a "-pedantic" flag which >> obeys Python semantics exactly, at the expense of speed. This applies >> to %, //, **, etc. (Overflowing is a separate issue, and I think >> there should be a flag for that too.) Compilation of .py files would >> be, by default, with -pedantic enabled, and compilation of .pyx files >> with it disabled. > > Wouldn't it be better to have such a flag emit warnings (preferably > with a > hint how to write better code) instead of changing the semantics of > operators? It won't be very useful to emit a warning on, for example, every % operator. Also, I really want to be able to write code values speed over perfect Python semantics, and at the same time take a .py file and not have to worry about subtle corner-case changes. The problem is that both modes are very defendable as the default. > I think having operators change their semantics without a clear > change in > the user code (such as a __future__ import) is a rather dangerous > 'feature'. I'm convinced pragmas are a better thing to use than a compiler flag. One could also imagine adding more granularity with pragmas as well (e.g. only use c semantics for % and //, but make sure everything else is the same). I'm not sure if this is a good thing... - Robert From robertwb at math.washington.edu Thu Mar 12 08:19:04 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 12 Mar 2009 00:19:04 -0700 Subject: [Cython] Google summer of code In-Reply-To: <6528b00c5b8779160ffe2dc3ec1f2cfb.squirrel@webmail.uio.no> References: <49B2E58A.9090900@behnel.de> <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> <993FE1B3-30D3-45C7-9E40-B8C420E2492C@math.washington.edu> <253B6623-3D04-47F7-B870-A74E420C7D6E@math.washington.edu> <6528b00c5b8779160ffe2dc3ec1f2cfb.squirrel@webmail.uio.no> Message-ID: On Mar 12, 2009, at 12:07 AM, Dag Sverre Seljebotn wrote: >> Robert Bradshaw wrote: >> On Mar 7, 2009, at 3:11 PM, Robert Bradshaw wrote: >> >>> I'm a "probably," but it's sounding like at this point we don't have >>> enough potential mentors to be a full mentoring organization unless >>> someone else stands up. On the numerical side of things, perhaps a >>> NumPy person would be willing. >>> >>> On the other hand, I think we're higher on Python's radar than last >>> year, so I'm hopeful of at least getting one (maybe two) spots for >>> them. The quality of the students and projects will play a huge role >>> here--one thing I learned at the summit last year is that it's not >>> even worth you time to go after sub-par projects (though good >>> students are an extremely good return on the investment of mentoring >>> them). >> >> Just a reminder, the organization app is due Friday if we decide to >> do it. >> >> Another advantage of being a mentor organization is that we get $500/ >> student. I'm not exactly sure what Cython would do with the money at >> this point (we don't have infrastructure to support, it's not enough >> to fund a get-together) I would propose it would be divvied out to >> the mentors to help offset the time they put into it. (They Python >> Foundation keeps the $500/student, which I'm fine with as they're big >> enough to have actual costs.) But being under an umbrella >> organization worked out well last time and is certainly less work and >> more flexibility on our part. > > This was up on the NumPy list -- they basically decided to go with > the PSF > for lack of manpower to put together an application (though there were > arguments in favor of staying with the PSF anyway). Anyway, if one > still > wants to apply one could consider doing it together with NumPy in > order to > have a larger mentor base. I can't help that much myself though > (polishing > the project list and so on I could always do, but not submitting the > actual application). Going together on it with NumPy is an interesting idea, though it doesn't elevate all the cons. Submitting the actual application doesn't seem too laborious (though it is a bit of work) but the main concern for me is that we have to commit to a certain level of involvement. One really nice thing about being with Python is that we can wait and see what kind of students and proposals we get, and then decide what to do about it. The downside is that we'll probably only get 1, or at most 2, slots (which isn't much of a downside given the response so far). It's a lot easier to get motivated once we have actual students and projects in place, assuming we'll get at least some of the caliber we want. - Robert From robertwb at math.washington.edu Thu Mar 12 08:25:09 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 12 Mar 2009 00:25:09 -0700 Subject: [Cython] Trac usage and bug handling for 0.11.x In-Reply-To: References: <49B8AE18.9040109@behnel.de> <49B8B3CD.4020308@behnel.de> Message-ID: <1140A710-E200-4633-BA6F-53B435580F13@math.washington.edu> On Mar 12, 2009, at 12:12 AM, Dag Sverre Seljebotn wrote: > Could we also put in something about which repositories to use for the > various releases? I.e. some questions: > > - Will we use cython-devel or cython for 0.11.1? > - If I do something that's just cleanup and not bugfix (e.g. convert > NameNode to new temps), should that go into 0.12 or 0.11.1? > > I'm in favor of 0.11.x going in cython and NOT contain cleanup, only > bugfixes, but I'm really OK with either. I'm actually not a huge fan of bugfix-only releases being parallel with main releases--I'd rather get the new features out as soon as possible too. I am willing to try that approach for this next release cycle if people things it's a good idea though. (That being said, I do like to be relatively "safe" with minor releases.) 0.11.1 should be the low-hanging fruit, e.g. some of the stuff that's already been implemented but didn't make it in in time. - Robert From carl.witty at gmail.com Thu Mar 12 08:26:30 2009 From: carl.witty at gmail.com (Carl Witty) Date: Thu, 12 Mar 2009 00:26:30 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> Message-ID: On Thu, Mar 12, 2009 at 12:12 AM, Robert Bradshaw wrote: > On Mar 11, 2009, at 11:20 PM, Stefan Behnel wrote: >> Wouldn't it be better to have such a flag emit warnings (preferably >> with a >> hint how to write better code) instead of changing the semantics of >> operators? > > It won't be very useful to emit a warning on, for example, every % > operator. Also, I really want to be able to write code values speed > over perfect Python semantics, and at the same time take a .py file > and not have to worry about subtle corner-case changes. The problem > is that both modes are very defendable as the default. How about a flag that generates runtime warnings on a%b and a//b whenever a and b have different sign? Carl From robertwb at math.washington.edu Thu Mar 12 08:49:45 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 12 Mar 2009 00:49:45 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> Message-ID: <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> On Mar 12, 2009, at 12:26 AM, Carl Witty wrote: > On Thu, Mar 12, 2009 at 12:12 AM, Robert Bradshaw > wrote: >> On Mar 11, 2009, at 11:20 PM, Stefan Behnel wrote: >>> Wouldn't it be better to have such a flag emit warnings (preferably >>> with a >>> hint how to write better code) instead of changing the semantics of >>> operators? >> >> It won't be very useful to emit a warning on, for example, every % >> operator. Also, I really want to be able to write code values speed >> over perfect Python semantics, and at the same time take a .py file >> and not have to worry about subtle corner-case changes. The problem >> is that both modes are very defendable as the default. > > How about a flag that generates runtime warnings on a%b and a//b > whenever a and b have different sign? This could be very useful for debugging things, but it implies there's a single, correct way that the % and // operators behave. The problem is that sometimes I want to run code with Python semantics (e.g. I'm quickly cythonizing a file) and sometimes I want to run code with C semantics (e.g. I'm doing linear algebra mod p, and don't want the overhead of fixing the sign). And perhaps I'm to demanding, but I want to be able to use % in both places rather than know some obscure function call. - Robert From dagss at student.matnat.uio.no Thu Mar 12 09:46:24 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 12 Mar 2009 09:46:24 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> Message-ID: <49B8CBE0.5050305@student.matnat.uio.no> Robert Bradshaw wrote: > On Mar 12, 2009, at 12:26 AM, Carl Witty wrote: > > >> On Thu, Mar 12, 2009 at 12:12 AM, Robert Bradshaw >> wrote: >> >>> On Mar 11, 2009, at 11:20 PM, Stefan Behnel wrote: >>> >>>> Wouldn't it be better to have such a flag emit warnings (preferably >>>> with a >>>> hint how to write better code) instead of changing the semantics of >>>> operators? >>>> >>> It won't be very useful to emit a warning on, for example, every % >>> operator. Also, I really want to be able to write code values speed >>> over perfect Python semantics, and at the same time take a .py file >>> and not have to worry about subtle corner-case changes. The problem >>> is that both modes are very defendable as the default. >>> >> How about a flag that generates runtime warnings on a%b and a//b >> whenever a and b have different sign? >> > > This could be very useful for debugging things, but it implies > there's a single, correct way that the % and // operators behave. > > The problem is that sometimes I want to run code with Python > semantics (e.g. I'm quickly cythonizing a file) and sometimes I want > to run code with C semantics (e.g. I'm doing linear algebra mod p, > and don't want the overhead of fixing the sign). And perhaps I'm to > demanding, but I want to be able to use % in both places rather than > know some obscure function call. > Perhaps you could ask the Sage list to get some more input of what the typical expectations to Cython are? (Also if you work in Z_p, could you not use unsigned ints? Then you don't get the overhead? Though I'm avoiding unsigned types like the plague myself after I figured out that range(-n, n) is empty if n is unsigned :-)) Dag Sverre From wstein at gmail.com Thu Mar 12 10:15:37 2009 From: wstein at gmail.com (William Stein) Date: Thu, 12 Mar 2009 02:15:37 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49B8CBE0.5050305@student.matnat.uio.no> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> Message-ID: <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> On Thu, Mar 12, 2009 at 1:46 AM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On Mar 12, 2009, at 12:26 AM, Carl Witty wrote: >> >> >>> On Thu, Mar 12, 2009 at 12:12 AM, Robert Bradshaw >>> wrote: >>> >>>> On Mar 11, 2009, at 11:20 PM, Stefan Behnel wrote: >>>> >>>>> Wouldn't it be better to have such a flag emit warnings (preferably >>>>> with a >>>>> hint how to write better code) instead of changing the semantics of >>>>> operators? >>>>> >>>> It won't be very useful to emit a warning on, for example, every % >>>> operator. Also, I really want to be able to write code values speed >>>> over perfect Python semantics, and at the same time take a .py file >>>> and not have to worry about subtle corner-case changes. The problem >>>> is that both modes are very defendable as the default. >>>> >>> How about a flag that generates runtime warnings on a%b and a//b >>> whenever a and b have different sign? >>> >> >> This could be very useful for debugging things, but it implies >> there's a single, correct way that the % and // operators behave. >> >> The problem is that sometimes I want to run code with Python >> semantics (e.g. I'm quickly cythonizing a file) and sometimes I want >> to run code with C semantics (e.g. I'm doing linear algebra mod p, >> and don't want the overhead of fixing the sign). And perhaps I'm to >> demanding, but I want to be able to use % in both places rather than >> know some obscure function call. >> > Perhaps you could ask the Sage list to get some more input of what the > typical expectations to Cython are? > > (Also if you work in Z_p, could you not use unsigned ints? Then you > don't get the overhead? Though I'm avoiding unsigned types like the > plague myself after I figured out that range(-n, n) is empty if n is > unsigned :-)) If you compute the sum of x and y mod p by doing "(x+y)%p", then the time is dominated by the % operation, which can easily be nearly 10 times longer than +. Now imagine doing linear algebra, where you take dot products, etc., and do lots of mod p arithmetic. If you dot two vectors with n entries in the naive way you end up doing n %p's, which is very expensive. Now imagine that you represent the entries of your vectors as C ints between -p/2 and p/2. Then you can very often do much of the dot product and only have to do very few mods, since frequently when you add up a bunch of numbers between -(p/2)^2 and (p/2)^2, they are likely to not overflow. The sum will be close to 0 because of cancellation. > Perhaps you could ask the Sage list to get some more input of what the > typical expectations to Cython are? I often think of Cython primarily as a "Python compiler", and from that perspective (-1 % 16) being different in Cython and Python is worrisome, since it will surely lead to subtle bugs. For me, Cython is all about writing fast code that is easy to use from Python. Cython is not about writing C code; if I want to write C code I write it in C. Cython is not the C language after all. Also, I think many current and potential users of Cython (at least through Sage) don't even know C, or if they do, they are using Cython so they don't have to use C. So I guess I prefer Python semantics for %, and an example in the documentation that explains how to get C semantics for % (i.e., via some macro or something). For those who view Cython as a way to write C code with a Python-like syntax, then C semantics would be preferable for %. I just don't see Cython that way, personally. -- William From stefan_ml at behnel.de Thu Mar 12 07:58:28 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 12 Mar 2009 07:58:28 +0100 Subject: [Cython] Trac usage and bug handling for 0.11.x In-Reply-To: <3BE155F1-FE7E-41AA-99D8-E53A7188E70F@math.washington.edu> References: <49B8AE18.9040109@behnel.de> <3BE155F1-FE7E-41AA-99D8-E53A7188E70F@math.washington.edu> Message-ID: <49B8B294.7020909@behnel.de> Robert Bradshaw wrote: > In fact, I think even non-bugfixes should all be assigned a > ticket on trac. (Some features may, of course, span many patches.) Sure, fine with me. > The only thing I would say is that I'd rather not clutter tests/run/-- > perhaps we could put them somewhere else (or alternatively have > runtests test all but a static list of known open tickets). If the trac based bug tickets all follow the ...Txyz.pyx convention (which I really like as it points directly to the background of the problem), it's easy to move them into a new "trac" or "fixed" directory (or whatever) when we feel like it. I'd just like to reserve tests/bugs/ for things that do not currently work (and don't care so much about the rest for now). Stefan From stefan_ml at behnel.de Thu Mar 12 10:14:31 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 12 Mar 2009 10:14:31 +0100 (CET) Subject: [Cython] Trac usage and bug handling for 0.11.x Message-ID: <61799.213.61.181.86.1236849271.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Robert Bradshaw wrote: > On Mar 12, 2009, at 12:12 AM, Dag Sverre Seljebotn wrote: >> Could we also put in something about which repositories to use for the >> various releases? I.e. some questions: >> >> - Will we use cython-devel or cython for 0.11.1? >> - If I do something that's just cleanup and not bugfix (e.g. convert >> NameNode to new temps), should that go into 0.12 or 0.11.1? Depends on how much we break with it. ;) >> I'm in favor of 0.11.x going in cython and NOT contain cleanup, only >> bugfixes, but I'm really OK with either. > > I'm actually not a huge fan of bugfix-only releases being parallel > with main releases ... also because it already happened that we forgot to push a fix into "cython". That said, it's sometimes easier to put trivial fixes into cython and pulling them over. > --I'd rather get the new features out as soon as > possible too. I am willing to try that approach for this next release > cycle if people things it's a good idea though. (That being said, I > do like to be relatively "safe" with minor releases.) > > 0.11.1 should be the low-hanging fruit, e.g. some of the stuff that's > already been implemented but didn't make it in in time. +1 and I think we should actively use trac's milestones to decide which fixes and features go where. However, the above doesn't answer the question if we should distinguish between potentially dangerous changes (that we implement because we can or because we need them, and have to put somewhere) and those that are safe enough to go into the next release. I would like to have something like a stable release series of Cython that other projects could depend on. I was having a pretty hard time in the past telling people that they must build lxml with /this/ Cython version instead of /that/ one. Being able to hold certain changes back will definitely help us in getting releases out a lot faster than 0.11. What about a separate cython-0.12 branch only for non-0.11 changes, next to a 'somewhat' safe cython-devel? When a major change is required, we can push it there instead of cython-devel, start working on it and have others test it while we can still release 0.11.x from cython-devel. It doesn't need to be up-to-date with cython-devel all the time (just when we push changes), and we can always import certain changes into cython-devel if we think we can't wait for 0.12. Or maybe "cython-unstable" would be a more generic name. Also, it would be good to pile up a set of test projects (sage and lxml as we do now, but also some others) that we regularly test before any release. Sort of a continuous integration test setup, like Python's buildbots. Stefan From dagss at student.matnat.uio.no Thu Mar 12 10:43:28 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 12 Mar 2009 10:43:28 +0100 Subject: [Cython] Trac usage and bug handling for 0.11.x In-Reply-To: <61799.213.61.181.86.1236849271.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <61799.213.61.181.86.1236849271.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49B8D940.3090202@student.matnat.uio.no> Stefan Behnel wrote: > However, the above doesn't answer the question if we should distinguish > between potentially dangerous changes (that we implement because we can or > because we need them, and have to put somewhere) and those that are safe > enough to go into the next release. I would like to have something like a > stable release series of Cython that other projects could depend on. I was > having a pretty hard time in the past telling people that they must build > lxml with /this/ Cython version instead of /that/ one. > That's my concern too. > Being able to hold certain changes back will definitely help us in getting > releases out a lot faster than 0.11. What about a separate cython-0.12 > branch only for non-0.11 changes, next to a 'somewhat' safe cython-devel? > When a major change is required, we can push it there instead of > cython-devel, start working on it and have others test it while we can > still release 0.11.x from cython-devel. It doesn't need to be up-to-date > with cython-devel all the time (just when we push changes), and we can > always import certain changes into cython-devel if we think we can't wait > for 0.12. Or maybe "cython-unstable" would be a more generic name. > +1 to cython-devel and cython-unstable. Dag Sverre From stefan_ml at behnel.de Thu Mar 12 11:00:42 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 12 Mar 2009 11:00:42 +0100 (CET) Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> Message-ID: <50507.213.61.181.86.1236852042.squirrel@groupware.dvs.informatik.tu-darmstadt.de> William Stein wrote: > On Thu, Mar 12, 2009 at 1:46 AM, Dag Sverre Seljebotn > wrote: >> Robert Bradshaw wrote: >>> On Mar 12, 2009, at 12:26 AM, Carl Witty wrote: >>>> How about a flag that generates runtime warnings on a%b and a//b >>>> whenever a and b have different sign? >>> >>> This could be very useful for debugging things, but it implies >>> there's a single, correct way that the % and // operators behave. At least, there *must* be a single way that the % and // operators behave. That's the only thing related to "correctness" in this debate. > So I guess I prefer Python semantics for %, and an example in the > documentation that explains how to get C semantics for % (i.e., via > some macro or something). ... that, and by optimising away the adaptation on unsigned numbers (or at least by checking that gcc does it for us). > For those who view Cython as a way to write C code with a Python-like > syntax, then C semantics would be preferable for %. I just don't see > Cython that way, personally. +1 Stefan From robertwb at math.washington.edu Thu Mar 12 11:03:01 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 12 Mar 2009 03:03:01 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> Message-ID: <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> On Mar 12, 2009, at 2:15 AM, William Stein wrote: > On Thu, Mar 12, 2009 at 1:46 AM, Dag Sverre Seljebotn > wrote: >>> >>> This could be very useful for debugging things, but it implies >>> there's a single, correct way that the % and // operators behave. >>> >>> The problem is that sometimes I want to run code with Python >>> semantics (e.g. I'm quickly cythonizing a file) and sometimes I want >>> to run code with C semantics (e.g. I'm doing linear algebra mod p, >>> and don't want the overhead of fixing the sign). And perhaps I'm to >>> demanding, but I want to be able to use % in both places rather than >>> know some obscure function call. >>> >> Perhaps you could ask the Sage list to get some more input of what >> the >> typical expectations to Cython are? >> >> (Also if you work in Z_p, could you not use unsigned ints? Then you >> don't get the overhead? Though I'm avoiding unsigned types like the >> plague myself after I figured out that range(-n, n) is empty if n is >> unsigned :-)) > > If you compute the sum of x and y mod p by doing "(x+y)%p", then the > time is dominated by the % operation, which can easily be nearly 10 > times longer than +. Now imagine doing linear algebra, where you take > dot products, etc., and do lots of mod p arithmetic. If you dot two > vectors with n entries in the naive way you end up doing n %p's, which > is very expensive. Now imagine that you represent the entries of your > vectors as C ints between -p/2 and p/2. Then you can very often do > much of the dot product and only have to do very few mods, since > frequently when you add up a bunch of numbers between -(p/2)^2 and > (p/2)^2, they are likely to not overflow. The sum will be close to 0 > because of cancellation. I was about to answer the very same thing then this email came in. Also, as you mentioned unsigned ints have subtle semantics, like the above. Another bad (easy--I've corrected it several times) mistake is when one writes (a-b)%p. If a and b are unsigned, this could be totally wrong. Of course, the main goal is to avoid using any division as much as possible. >> Perhaps you could ask the Sage list to get some more input of what >> the >> typical expectations to Cython are? > > I often think of Cython primarily as a "Python compiler", and from > that perspective (-1 % 16) being different in Cython and Python is > worrisome, since it will surely lead to subtle bugs. For me, > Cython is all about writing fast code that is easy to use from Python. > Cython is not about writing C code; if I want to write C code I write > it in C. Cython is not the C language after all. Also, I think many > current and potential users of Cython (at least through Sage) don't > even know C, or if they do, they are using Cython so they don't have > to use C. Good point. I'm going to start a thread there just to get a straw poll. It's the largest community of Cython users I know of... > So I guess I prefer Python semantics for %, and an example in the > documentation that explains how to get C semantics for % (i.e., via > some macro or something). One of the things that drives a lot of my optimizations in Cython has been the maxim that the obvious way should be the fast way, and I especially want to avoid having to resort to tricks to "fool" the compiler to get maximum speed. (Of course in this case we're talking about only a 5% or so difference, assuming no branching). I like the pragma idea because it makes it easy to switch a block/file of code at once (while forcing one to consciously think about it). > For those who view Cython as a way to write C code with a Python-like > syntax, then C semantics would be preferable for %. I just don't see > Cython that way, personally. I see it both ways, which I guess is why I want to have my cake and eat it too :). - Robert From stefan_ml at behnel.de Thu Mar 12 13:12:26 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 12 Mar 2009 13:12:26 +0100 (CET) Subject: [Cython] Google summer of code In-Reply-To: References: <49B2E58A.9090900@behnel.de> <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> <993FE1B3-30D3-45C7-9E40-B8C420E2492C@math.washington.edu> <253B6623-3D04-47F7-B870-A74E420C7D6E@math.washington.edu> <6528b00c5b8779160ffe2dc3ec1f2cfb.squirrel@webmail.uio.no> Message-ID: <43348.213.61.181.86.1236859946.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Robert Bradshaw wrote: > Submitting the actual application doesn't seem too laborious (though > it is a bit of work) but the main concern for me is that we have to > commit to a certain level of involvement. Just a quick note that my "involvement" in this will not be as an official mentor. I appreciate the GSoC, but I can't give any guarantees about the time I can invest. So I'll keep myself out of the decision about the way we participate. Stefan From poonaatsoc at gmail.com Thu Mar 12 13:30:25 2009 From: poonaatsoc at gmail.com (Anoop Saldanha) Date: Thu, 12 Mar 2009 18:00:25 +0530 Subject: [Cython] Google summer of code In-Reply-To: <43348.213.61.181.86.1236859946.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49B2E58A.9090900@behnel.de> <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> <993FE1B3-30D3-45C7-9E40-B8C420E2492C@math.washington.edu> <253B6623-3D04-47F7-B870-A74E420C7D6E@math.washington.edu> <6528b00c5b8779160ffe2dc3ec1f2cfb.squirrel@webmail.uio.no> <43348.213.61.181.86.1236859946.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: I would like to be mentor, but the issue is right now I don't know the internal technical details of the cython compiler, but I am going through the compiler code currently. As well as the tickets raised in the last couple of days have given me some insight on the inner working. I think I should be up to speed and should be amongst the tickets in sometime. In case I become a mentor, it will be more of a collaboration between me and the student rather than a pure mentor-student relationship and I will have to work more closely with the student(have no issue with it). On Thu, Mar 12, 2009 at 5:42 PM, Stefan Behnel wrote: > Robert Bradshaw wrote: > > Submitting the actual application doesn't seem too laborious (though > > it is a bit of work) but the main concern for me is that we have to > > commit to a certain level of involvement. > > Just a quick note that my "involvement" in this will not be as an official > mentor. I appreciate the GSoC, but I can't give any guarantees about the > time I can invest. So I'll keep myself out of the decision about the way > we participate. > > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Regards, Anoop S. People who says, "It's not winning or losing that matters, but participation", probably lost. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090312/3e2dee40/attachment.htm From robertwb at math.washington.edu Thu Mar 12 19:06:34 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 12 Mar 2009 11:06:34 -0700 Subject: [Cython] Google summer of code In-Reply-To: References: <49B2E58A.9090900@behnel.de> <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> <993FE1B3-30D3-45C7-9E40-B8C420E2492C@math.washington.edu> <253B6623-3D04-47F7-B870-A74E420C7D6E@math.washington.edu> <6528b00c5b8779160ffe2dc3ec1f2cfb.squirrel@webmail.uio.no> <43348.213.61.181.86.1236859946.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <48D41E38-20B4-4082-8205-DAF49D9FB452@math.washington.edu> On Mar 12, 2009, at 5:30 AM, Anoop Saldanha wrote: > I would like to be mentor, but the issue is right now I don't know > the internal technical details of the cython compiler, but I am > going through the compiler code currently. As well as the tickets > raised in the last couple of days have given me some insight on the > inner working. I think I should be up to speed and should be > amongst the tickets in sometime. In case I become a mentor, it > will be more of a collaboration between me and the student rather > than a pure mentor-student relationship and I will have to work > more closely with the student(have no issue with it). I think that's just fine, and I bet it would be a very productive experience which is the primary goal. If you can mentor, and there's a good project/student, please do. - Robert From robertwb at math.washington.edu Thu Mar 12 19:11:51 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 12 Mar 2009 11:11:51 -0700 Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 In-Reply-To: <49B80A9F.1000200@behnel.de> References: <44706.213.61.181.86.1236700457.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B80A9F.1000200@behnel.de> Message-ID: <4875B432-09E8-46F3-BEF0-87644F9C709D@math.washington.edu> On Mar 11, 2009, at 12:01 PM, Stefan Behnel wrote: > Hi Neal, > > Neal Becker wrote: >> Can we have an rc2 with all proposed fixes? I'll plan to retest >> then. > > There's an rc3 now: > > http://cython.org/Cython-0.11.rc3.tar.gz > > Could you give it a try? I'm assuming nothing bad came up. The last word was that it did build, even if it had gcc warnings. > If all works fine, this will become Cython 0.11 final. Sage 3.4 was just released. I'll verify that it passes all tests (was fine with 3.3 so I don't anticipate issues) and then lets release. - Robert From robertwb at math.washington.edu Thu Mar 12 19:14:22 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 12 Mar 2009 11:14:22 -0700 Subject: [Cython] Trac usage and bug handling for 0.11.x In-Reply-To: <49B8D940.3090202@student.matnat.uio.no> References: <61799.213.61.181.86.1236849271.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B8D940.3090202@student.matnat.uio.no> Message-ID: <4DB56201-278E-4068-A185-4AB1947EC239@math.washington.edu> On Mar 12, 2009, at 2:43 AM, Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> However, the above doesn't answer the question if we should >> distinguish >> between potentially dangerous changes (that we implement because >> we can or >> because we need them, and have to put somewhere) and those that >> are safe >> enough to go into the next release. I would like to have something >> like a >> stable release series of Cython that other projects could depend >> on. I was >> having a pretty hard time in the past telling people that they >> must build >> lxml with /this/ Cython version instead of /that/ one. >> > That's my concern too. We don't have this issue with Sage as we ship Cython as a component. (We even ship and build our own Python...) However, modulo any bug fixes, I think the 0.x range should all be compatible. >> Being able to hold certain changes back will definitely help us in >> getting >> releases out a lot faster than 0.11. What about a separate >> cython-0.12 >> branch only for non-0.11 changes, next to a 'somewhat' safe cython- >> devel? >> When a major change is required, we can push it there instead of >> cython-devel, start working on it and have others test it while we >> can >> still release 0.11.x from cython-devel. It doesn't need to be up- >> to-date >> with cython-devel all the time (just when we push changes), and we >> can >> always import certain changes into cython-devel if we think we >> can't wait >> for 0.12. Or maybe "cython-unstable" would be a more generic name. >> > +1 to cython-devel and cython-unstable. Yes, I'll make such branches. I like the idea of "safe" stuff (either fixes or minor improvements) being in cython-devel, and anything more drastic (e.g. moving all the temp stuff over to the new model) in cython-unstable. A much better division than "bugfix" and "other." - Robert From stefan_ml at behnel.de Thu Mar 12 19:22:03 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 12 Mar 2009 19:22:03 +0100 Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 In-Reply-To: <4875B432-09E8-46F3-BEF0-87644F9C709D@math.washington.edu> References: <44706.213.61.181.86.1236700457.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B80A9F.1000200@behnel.de> <4875B432-09E8-46F3-BEF0-87644F9C709D@math.washington.edu> Message-ID: <49B952CB.2080802@behnel.de> Robert Bradshaw wrote: > Sage 3.4 was just released. I'll verify that it passes all tests (was > fine with 3.3 so I don't anticipate issues) and then lets release. Yes, let's. (finally!) Stefan From robertwb at math.washington.edu Fri Mar 13 00:06:06 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 12 Mar 2009 16:06:06 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> Message-ID: <5A295F2F-EA79-40B6-92F6-139A5D0FA808@math.washington.edu> What about ** for doubles? flotobject.c:float_pow has a lot of comments about that. - Robert From wstein at gmail.com Fri Mar 13 00:18:31 2009 From: wstein at gmail.com (William Stein) Date: Thu, 12 Mar 2009 16:18:31 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> Message-ID: <85e81ba30903121618q2506d10fxb9ab6b5899032a4f@mail.gmail.com> On Thu, Mar 12, 2009 at 3:03 AM, Robert Bradshaw wrote: > On Mar 12, 2009, at 2:15 AM, William Stein wrote: > >> On Thu, Mar 12, 2009 at 1:46 AM, Dag Sverre Seljebotn >> wrote: >>>> >>>> This could be very useful for debugging things, but it implies >>>> there's a single, correct way that the % and // operators behave. >>>> >>>> The problem is that sometimes I want to run code with Python >>>> semantics (e.g. I'm quickly cythonizing a file) and sometimes I want >>>> to run code with C semantics (e.g. I'm doing linear algebra mod p, >>>> and don't want the overhead of fixing the sign). And perhaps I'm to >>>> demanding, but I want to be able to use % in both places rather than >>>> know some obscure function call. >>>> >>> Perhaps you could ask the Sage list to get some more input of what >>> the >>> typical expectations to Cython are? >>> >>> (Also if you work in Z_p, could you not use unsigned ints? Then you >>> don't get the overhead? Though I'm avoiding unsigned types like the >>> plague myself after I figured out that range(-n, n) is empty if n is >>> unsigned :-)) >> >> If you compute the sum of x and y mod p by doing "(x+y)%p", then the >> time is dominated by the % operation, which can easily be nearly 10 >> times longer than +. ?Now imagine doing linear algebra, where you take >> dot products, etc., and do lots of mod p arithmetic. ?If you dot two >> vectors with n entries in the naive way you end up doing n %p's, which >> is very expensive. ?Now imagine that you represent the entries of your >> vectors as C ints between -p/2 and p/2. ?Then you can very often do >> much of the dot product and only have to do very few mods, since >> frequently when you add up a bunch of numbers between -(p/2)^2 and >> (p/2)^2, they are likely to not overflow. ?The sum will be close to 0 >> because of cancellation. > > I was about to answer the very same thing then this email came in. That's probably because you explained it to me once :-) > One of the things that drives a lot of my optimizations in Cython has > been the maxim that the obvious way should be the fast way, and I > especially want to avoid having to resort to tricks to "fool" the > compiler to get maximum speed. (Of course in this case we're talking > about only a 5% or so difference, assuming no branching). I like the Are you sure it's a difference of 5%? I just tried these two programs: main() { int n = -1, m = 16, k, a=0; for(k=1;k<100000000;k++) { a = n%k + (a<0)?k:0; } } and main() { int n = -1, m = 16, k, a=0; for(k=1;k<100000000;k++) { a = n%k; } } and the first takes 22% longer than the second. -- William From santiago.aguiar at gmail.com Fri Mar 13 01:05:18 2009 From: santiago.aguiar at gmail.com (Santiago Aguiar) Date: Thu, 12 Mar 2009 21:05:18 -0300 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... Message-ID: <49B9A33E.2080405@gmail.com> Hello, I've started to use Cython just yesterday, and I'm facing some difficulties ;). I'm sorry to ask this on the dev list, but I didn't found a user specific one... I have a struct-dense C source code that defines the PDU of a network protocol and functions to read/write from a buffer struct. First, I don't know if I'm doing this the right way, but I had to define by hand all my structs again on Cython to use them. I think it's the way to do it, but I did feel like repeating myself over and over ;). Is there any tool to create mappings like these automatically? (and handle common cases like wrapping structs with inner structs, uint8_t *, defining macros like #define VAL 1 as dummy enums, etc.). I saw some for C++ (using gcc-xml) but I'm stuck with pure C. My main objective on doing these bindings are to reuse the C code, having an efficient implementation is secondary. Second, at this extremely un-proficient time, for each of the structs on my pdu.pyx I have something like this: cdef class Msg1(Message): cdef public ... (one declaration for each field present in the struct) def __init__(self, ... (one keyword parameter for each field provided by the struct): self.fieldN = ... (this repeated for every field) cdef void from_c(self, msg1_c_struct *msg1): self.fieldN = msg1.fieldN (repeated for every field, with some additional mappings in case the field is a struct) And then I have a message parser that uses the C functions to read the PDU and wrap it on Msg1: class Parser: def read_pdu(self, str_data): cdef msg_struct msg .... pdu_read(data, &msg) .... if msg.header.type == MSG_1_TYPE: return Msg1().from_c(&msg.data.msg_1) Where msg.data.msg_1 is of type msg1_c_struct. On that code, I get an error message saying that: 'Cannot convert 'msg1_c_struct *' to Python object' where I do the Msg1().from_c. I don't understand why I'm getting the error. If I call a cdef function instead of a cdef method, it works OK; but shouldn't I be able to call a cdef method and passing a C type instead of a Python type? Otherwise, I will need to add an external cdef function for each class to create the object from the struct... I must be doing something wrong :). Thanks, and sorry for the basic questions! Best regards, Santiago. From robertwb at math.washington.edu Fri Mar 13 02:33:00 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 12 Mar 2009 18:33:00 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <85e81ba30903121618q2506d10fxb9ab6b5899032a4f@mail.gmail.com> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <85e81ba30903121618q2506d10fxb9ab6b5899032a4f@mail.gmail.com> Message-ID: On Mar 12, 2009, at 4:18 PM, William Stein wrote: > >> One of the things that drives a lot of my optimizations in Cython has >> been the maxim that the obvious way should be the fast way, and I >> especially want to avoid having to resort to tricks to "fool" the >> compiler to get maximum speed. (Of course in this case we're talking >> about only a 5% or so difference, assuming no branching). I like the > > Are you sure it's a difference of 5%? > > I just tried these two programs: > > main() { > int n = -1, m = 16, k, a=0; > for(k=1;k<100000000;k++) { > a = n%k + (a<0)?k:0; > } > } > > and > > main() { > int n = -1, m = 16, k, a=0; > for(k=1;k<100000000;k++) { > a = n%k; > } > } > > and the first takes 22% longer than the second. Thanks for the timings--5% was a conservative estimate, but I didn't realize it was that conservative. - Robert From stefan_ml at behnel.de Fri Mar 13 08:53:07 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 13 Mar 2009 08:53:07 +0100 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: <49B9A33E.2080405@gmail.com> References: <49B9A33E.2080405@gmail.com> Message-ID: <49BA10E3.6090409@behnel.de> Santiago Aguiar wrote: > I've started to use Cython just yesterday, and I'm facing some > difficulties ;). I'm sorry to ask this on the dev list, but I didn't > found a user specific one... > > I have a struct-dense C source code that defines the PDU of a network > protocol and functions to read/write from a buffer struct. > > First, I don't know if I'm doing this the right way, but I had to define > by hand all my structs again on Cython to use them. I think it's the way > to do it, but I did feel like repeating myself over and over ;). That's the way to do it, though. You only go through that once for each struct you use (and your code that uses the struct is likely longer than the struct itself), so it's not too much work. Note that you do not need to declare the complete struct. Just write down the fields you need. Cython doesn't need to know the others and the C compiler sees the struct anyway. That also allows you to ignore any externally referenced complex types that you do not use yourself. > Second, at this extremely un-proficient time, for each of the structs on > my pdu.pyx I have something like this: > > cdef class Msg1(Message): > cdef public ... (one declaration for each field present in the struct) Why don't you reference the struct itself here? Do you really need these fields to be public? Some methods for setting up common use cases are often a lot better than just having users set everything by hand, especially when there are non-trivial type conversions involved, or when there are interdependencies between fields that are better checked for validity. For example, Cython cannot check arbitrary integer values for you that the user writes into an enum type attribute. > def __init__(self, ... (one keyword parameter for each field > provided by the struct): > self.fieldN = ... (this repeated for every field) > cdef void from_c(self, msg1_c_struct *msg1): > self.fieldN = msg1.fieldN (repeated for every field, with some > additional mappings in case the field is a struct) There is some special support for struct assignments from dicts, you can try that. I have no idea where that's documented, though. Robert? > And then I have a message parser that uses the C functions to read the > PDU and wrap it on Msg1: > > class Parser: > def read_pdu(self, str_data): > cdef msg_struct msg > .... > pdu_read(data, &msg) > .... > if msg.header.type == MSG_1_TYPE: > return Msg1().from_c(&msg.data.msg_1) > > Where msg.data.msg_1 is of type msg1_c_struct. > > On that code, I get an error message saying that: 'Cannot convert > 'msg1_c_struct *' to Python object' where I do the Msg1().from_c. You have to assign the result of Msg1() to a typed variable. Otherwise, Cython doesn't know that the thing it returns is really of type Msg1 (and not a generic Python object) and that it does have a C typed from_c() method. > I don't understand why I'm getting the error. If I call a cdef function > instead of a cdef method, it works OK; but shouldn't I be able to call a > cdef method and passing a C type instead of a Python type? Otherwise, I > will need to add an external cdef function for each class to create the > object from the struct... I must be doing something wrong :). I consider this a deficiency of Cython. It should know the type of the result when you call an extension type. http://trac.cython.org/cython_trac/ticket/232 Stefan From greg.ewing at canterbury.ac.nz Fri Mar 13 11:19:05 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 13 Mar 2009 23:19:05 +1300 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: <49BA10E3.6090409@behnel.de> References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> Message-ID: <49BA3319.1020600@canterbury.ac.nz> Stefan Behnel wrote: > Santiago Aguiar wrote: > >> return Msg1().from_c(&msg.data.msg_1) >> >> I get an error message saying that: 'Cannot convert >>'msg1_c_struct *' to Python object' where I do the Msg1().from_c. > > I consider this a deficiency of Cython. It should know the type of the > result when you call an extension type. This does currently work in Pyrex, btw. -- Greg From ndbecker2 at gmail.com Fri Mar 13 12:50:24 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 13 Mar 2009 07:50:24 -0400 Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 References: <44706.213.61.181.86.1236700457.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B80A9F.1000200@behnel.de> <4875B432-09E8-46F3-BEF0-87644F9C709D@math.washington.edu> <49B952CB.2080802@behnel.de> Message-ID: Stefan Behnel wrote: > > Robert Bradshaw wrote: >> Sage 3.4 was just released. I'll verify that it passes all tests (was >> fine with 3.3 so I don't anticipate issues) and then lets release. > > Yes, let's. (finally!) > > Stefan 0.11rc3 built OK (with warnings). From stefan_ml at behnel.de Fri Mar 13 13:17:32 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 13 Mar 2009 13:17:32 +0100 (CET) Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 In-Reply-To: References: <44706.213.61.181.86.1236700457.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B80A9F.1000200@behnel.de> <4875B432-09E8-46F3-BEF0-87644F9C709D@math.washington.edu> <49B952CB.2080802@behnel.de> Message-ID: <45933.213.61.181.86.1236946652.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Neal Becker wrote: > 0.11rc3 built OK (with warnings). Such as? Stefan From ndbecker2 at gmail.com Fri Mar 13 13:34:03 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 13 Mar 2009 08:34:03 -0400 Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 References: <44706.213.61.181.86.1236700457.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B80A9F.1000200@behnel.de> <4875B432-09E8-46F3-BEF0-87644F9C709D@math.washington.edu> <49B952CB.2080802@behnel.de> <45933.213.61.181.86.1236946652.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: Stefan Behnel wrote: > Neal Becker wrote: >> 0.11rc3 built OK (with warnings). > > Such as? > > Stefan The build log (for x86_64) can be seen here: http://koji.fedoraproject.org/koji/getfile?taskID=1239381&name=build.log From poonaatsoc at gmail.com Fri Mar 13 14:43:29 2009 From: poonaatsoc at gmail.com (Anoop Saldanha) Date: Fri, 13 Mar 2009 19:13:29 +0530 Subject: [Cython] Google summer of code In-Reply-To: <48D41E38-20B4-4082-8205-DAF49D9FB452@math.washington.edu> References: <49B2E58A.9090900@behnel.de> <7d15b0ff5873d4c37fd080293f94c3a4.squirrel@webmail.uio.no> <993FE1B3-30D3-45C7-9E40-B8C420E2492C@math.washington.edu> <253B6623-3D04-47F7-B870-A74E420C7D6E@math.washington.edu> <6528b00c5b8779160ffe2dc3ec1f2cfb.squirrel@webmail.uio.no> <43348.213.61.181.86.1236859946.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <48D41E38-20B4-4082-8205-DAF49D9FB452@math.washington.edu> Message-ID: Nice. Thanks. Do let me know, if you want me as a mentor. I saw the enhancements section for cython, and I will pick one of those and start digging into it. On Thu, Mar 12, 2009 at 11:36 PM, Robert Bradshaw < robertwb at math.washington.edu> wrote: > On Mar 12, 2009, at 5:30 AM, Anoop Saldanha wrote: > > > I would like to be mentor, but the issue is right now I don't know > > the internal technical details of the cython compiler, but I am > > going through the compiler code currently. As well as the tickets > > raised in the last couple of days have given me some insight on the > > inner working. I think I should be up to speed and should be > > amongst the tickets in sometime. In case I become a mentor, it > > will be more of a collaboration between me and the student rather > > than a pure mentor-student relationship and I will have to work > > more closely with the student(have no issue with it). > > I think that's just fine, and I bet it would be a very productive > experience which is the primary goal. If you can mentor, and there's > a good project/student, please do. > > - Robert > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Regards, Anoop S. People who says, "It's not winning or losing that matters, but participation", probably lost. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090313/c92fb0d2/attachment.htm From stefan_ml at behnel.de Fri Mar 13 14:43:48 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 13 Mar 2009 14:43:48 +0100 (CET) Subject: [Cython] cython-0.11rc result on python-2.6, gcc-4.4 In-Reply-To: References: <44706.213.61.181.86.1236700457.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B80A9F.1000200@behnel.de> <4875B432-09E8-46F3-BEF0-87644F9C709D@math.washington.edu> <49B952CB.2080802@behnel.de> <45933.213.61.181.86.1236946652.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <41449.213.61.181.86.1236951828.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Neal Becker wrote: > Stefan Behnel wrote: > >> Neal Becker wrote: >>> 0.11rc3 built OK (with warnings). >> >> Such as? >> >> Stefan > > The build log (for x86_64) can be seen here: > http://koji.fedoraproject.org/koji/getfile?taskID=1239381&name=build.log Looks perfectly fine to me. The rest are all (minor) known problems. Thanks! Stefan From carl.witty at gmail.com Fri Mar 13 15:42:29 2009 From: carl.witty at gmail.com (Carl Witty) Date: Fri, 13 Mar 2009 07:42:29 -0700 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: <49BA10E3.6090409@behnel.de> References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> Message-ID: On Fri, Mar 13, 2009 at 12:53 AM, Stefan Behnel wrote: >> I don't understand why I'm getting the error. If I call a cdef function >> instead of a cdef method, it works OK; but shouldn't I be able to call a >> cdef method and passing a C type instead of a Python type? Otherwise, I >> will need to add an external cdef function for each class to create the >> object from the struct... I must be doing something wrong :). > > I consider this a deficiency of Cython. It should know the type of the > result when you call an extension type. > > http://trac.cython.org/cython_trac/ticket/232 Hmm... I hope that eventually Cython will allow overriding __new__, at which point calling an extension type won't necessarily return a value of that type. (I see that Robert already made this point in a comment on the ticket.) Carl From stefan_ml at behnel.de Fri Mar 13 15:46:03 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 13 Mar 2009 15:46:03 +0100 (CET) Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> Message-ID: <43407.213.61.181.86.1236955563.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Carl Witty wrote: > On Fri, Mar 13, 2009 at 12:53 AM, Stefan Behnel wrote: >>> I don't understand why I'm getting the error. If I call a cdef function >>> instead of a cdef method, it works OK; but shouldn't I be able to call >>> a >>> cdef method and passing a C type instead of a Python type? Otherwise, I >>> will need to add an external cdef function for each class to create the >>> object from the struct... I must be doing something wrong :). >> >> I consider this a deficiency of Cython. It should know the type of the >> result when you call an extension type. >> >> http://trac.cython.org/cython_trac/ticket/232 > > Hmm... I hope that eventually Cython will allow overriding __new__, at > which point calling an extension type won't necessarily return a value > of that type. I'm not sure. This should certainly be supported for Python classes, but for extension types? What would be the use case? Stefan From dalcinl at gmail.com Fri Mar 13 17:19:45 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 13 Mar 2009 14:19:45 -0200 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> Message-ID: On Fri, Mar 13, 2009 at 12:42 PM, Carl Witty wrote: > On Fri, Mar 13, 2009 at 12:53 AM, Stefan Behnel wrote: >>> I don't understand why I'm getting the error. If I call a cdef function >>> instead of a cdef method, it works OK; but shouldn't I be able to call a >>> cdef method and passing a C type instead of a Python type? Otherwise, I >>> will need to add an external cdef function for each class to create the >>> object from the struct... I must be doing something wrong :). >> >> I consider this a deficiency of Cython. It should know the type of the >> result when you call an extension type. >> >> http://trac.cython.org/cython_trac/ticket/232 > > Hmm... I hope that eventually Cython will allow overriding __new__, at > which point calling an extension type won't necessarily return a value > of that type. ?(I see that Robert already made this point in a comment > on the ticket.) > But if a typecheck is done, we could dispatch a (polymorfic) C call of a Python call... Am I missing something? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Fri Mar 13 17:25:29 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 13 Mar 2009 14:25:29 -0200 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: <43407.213.61.181.86.1236955563.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> <43407.213.61.181.86.1236955563.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: On Fri, Mar 13, 2009 at 12:46 PM, Stefan Behnel wrote: > Carl Witty wrote: >> Hmm... I hope that eventually Cython will allow overriding __new__, at >> which point calling an extension type won't necessarily return a value >> of that type. > > I'm not sure. This should certainly be supported for Python classes, but > for extension types? What would be the use case? > Well, take for granted that if this were available, I would use it in mpi4py. For example, some instances of cdef classes could be specially handled, and I could easily implement things like in the code below: comm = MPI.Comm(MPI.COMM_WORLD) assert comm is MPI.COMM_WORLD and the assertion will succeed... Moreover, any cdef class whose instances are intended to be immutable (like int, str, float, tuple) could benefit from this feature, such in the same way that core CPython does. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Fri Mar 13 17:27:02 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 13 Mar 2009 17:27:02 +0100 (CET) Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> Message-ID: <47474.213.61.181.86.1236961622.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Lisandro Dalcin wrote: > On Fri, Mar 13, 2009 at 12:42 PM, Carl Witty wrote: >> On Fri, Mar 13, 2009 at 12:53 AM, Stefan Behnel >> wrote: >>>> I don't understand why I'm getting the error. If I call a cdef >>>> function >>>> instead of a cdef method, it works OK; but shouldn't I be able to call >>>> a >>>> cdef method and passing a C type instead of a Python type? Otherwise, >>>> I >>>> will need to add an external cdef function for each class to create >>>> the >>>> object from the struct... I must be doing something wrong :). >>> >>> I consider this a deficiency of Cython. It should know the type of the >>> result when you call an extension type. >>> >>> http://trac.cython.org/cython_trac/ticket/232 >> >> Hmm... I hope that eventually Cython will allow overriding __new__, at >> which point calling an extension type won't necessarily return a value >> of that type. ?(I see that Robert already made this point in a comment >> on the ticket.) > > But if a typecheck is done, we could dispatch a (polymorfic) C call of > a Python call... Am I missing something? A typecheck would only catch the case that you return the exact type, not a subtype (which works perfectly fine otherwise). And a full fledged isinstance() sounds a bit too heavy for this. Stefan From stefan_ml at behnel.de Fri Mar 13 17:39:47 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 13 Mar 2009 17:39:47 +0100 (CET) Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> <43407.213.61.181.86.1236955563.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <54183.213.61.181.86.1236962387.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Lisandro Dalcin wrote: > On Fri, Mar 13, 2009 at 12:46 PM, Stefan Behnel wrote: >> Carl Witty wrote: >>> Hmm... I hope that eventually Cython will allow overriding __new__, at >>> which point calling an extension type won't necessarily return a value >>> of that type. >> >> I'm not sure. This should certainly be supported for Python classes, but >> for extension types? What would be the use case? > > Well, take for granted that if this were available, I would use it in > mpi4py. > > For example, some instances of cdef classes could be specially > handled, and I could easily implement things like in the code below: > > comm = MPI.Comm(MPI.COMM_WORLD) > assert comm is MPI.COMM_WORLD > > and the assertion will succeed... Am I right in guessing that Comm is an extension type here and that COMM_WORLD is an instance? So you want to use Comm as both a class and a factory, right? So, what was your use case again? > Moreover, any cdef class whose instances are intended to be immutable > (like int, str, float, tuple) could benefit from this feature, such in > the same way that core CPython does. Here, I assume you are hinting towards singletons and caching, right? lxml actually does this with dedicated factories, but I can see that some APIs may benefit from such a behaviour. So, can't we just require that ExtType.__new__() returns an instance of ExtType (or its subtypes)? Or is there a use case that would benefit from returning arbitrary things? Stefan From carl.witty at gmail.com Fri Mar 13 17:56:15 2009 From: carl.witty at gmail.com (Carl Witty) Date: Fri, 13 Mar 2009 09:56:15 -0700 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: <54183.213.61.181.86.1236962387.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> <43407.213.61.181.86.1236955563.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <54183.213.61.181.86.1236962387.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: On Fri, Mar 13, 2009 at 9:39 AM, Stefan Behnel wrote: > So, can't we just require that ExtType.__new__() returns an instance of > ExtType (or its subtypes)? Or is there a use case that would benefit from > returning arbitrary things? I have code (currently in pure Python, but if I ever transition the code to Cython I'd like to do the same thing) where B and C both inherit from A, and B's .__new__() can sometimes return an instance of C. The details are a bit mathematical, but in case you're curious: B is ANExtensionElement, which represents an arbitrary number field element. C is ANRational, which represents a rational number. When I try to construct an ANExtensionElement that's actually a rational number, I return an ANRational instead. Carl From santiago.aguiar at gmail.com Fri Mar 13 18:08:10 2009 From: santiago.aguiar at gmail.com (Santiago Aguiar) Date: Fri, 13 Mar 2009 14:08:10 -0300 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: <49BA10E3.6090409@behnel.de> References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> Message-ID: <49BA92FA.2070801@gmail.com> Stefan Behnel wrote: > Why don't you reference the struct itself here? Do you really need these > fields to be public? > I could, but in that case I would need to write getters/setters for each attribute. In my code, almost all the struct fields are relevant, since they represent fields in the protocol PDU which I need to provide to python code. > There is some special support for struct assignments from dicts, you can > try that. I have no idea where that's documented, though. Robert? > Any pointers will be welcomed! Thanks! Best regards, Santiago. From dalcinl at gmail.com Fri Mar 13 18:11:33 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 13 Mar 2009 15:11:33 -0200 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: <54183.213.61.181.86.1236962387.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> <43407.213.61.181.86.1236955563.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <54183.213.61.181.86.1236962387.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: On Fri, Mar 13, 2009 at 2:39 PM, Stefan Behnel wrote: >> >> comm = MPI.Comm(MPI.COMM_WORLD) >> assert comm is MPI.COMM_WORLD >> >> and the assertion will succeed... > > Am I right in guessing that Comm is an extension type here and that > COMM_WORLD is an instance? > Yes, you are right... > So you want to use Comm as both a class and a factory, right? So, what was > your use case again? > MPI.COMM_WORLD is a predefined, immutable, live-forever instance of a class 'Intracomm', a derived class of 'Comm'. Comm should be seen as an abstract base class in C++ terms... Comm is a class, but it is also a factory in the sense that if you call Comm(comm) it returns a new Comm instance referencing the underlying C handle... in mpi4py, all cdef classes are actually proxies to C handles.... I would like to maintain if possible this invariant... if the C handle is MPI_COMM_WORLD (the C side name), then the Python-level instance is always MPI.COMM_WORLD .... > >> Moreover, any cdef class whose instances are intended to be immutable >> (like int, str, float, tuple) could benefit from this feature, such in >> the same way that core CPython does. > > Here, I assume you are hinting towards singletons and caching, right? lxml > actually does this with dedicated factories, but I can see that some APIs > may benefit from such a behaviour. > At least singletons... In all the above discussion, MPI.COMM_WORLD is similar to a singleton, I do not really know how to call it :-) > So, can't we just require that ExtType.__new__() returns an instance of > ExtType (or its subtypes)? As a first approach, it could be like that... ? Or is there a use case that would benefit from > returning arbitrary things? > Well, Python supports this, we could in case it is possible.... > Stefan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Fri Mar 13 18:17:09 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 13 Mar 2009 15:17:09 -0200 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: <49BA92FA.2070801@gmail.com> References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> <49BA92FA.2070801@gmail.com> Message-ID: Santiago, have you considered writing a simple script to generate Cython code :-) ?? You could parse a simple txt file with metadata about your C-level structs (name, field types, fieldname, and even default field values), then you could easily dump Cython code automating the member, __cinit__() methods and all the 'property' you need. As Stefan commented, you should just store your struct in your cdef class, and use 'property' to expose the field values with tiny get/set funcions... Remember that struct member assignments should work in any C89 compiler... No point to complicate your from_c()/ to_c() cdef methods ... On Fri, Mar 13, 2009 at 3:08 PM, Santiago Aguiar wrote: > Stefan Behnel wrote: >> Why don't you reference the struct itself here? Do you really need these >> fields to be public? >> > I could, but in that case I would need to write getters/setters for each > attribute. In my code, almost all the struct fields are relevant, since > they represent fields in the protocol PDU which I need to provide to > python code. >> There is some special support for struct assignments from dicts, you can >> try that. I have no idea where that's documented, though. Robert? >> > Any pointers will be welcomed! > > Thanks! > > Best regards, > > Santiago. > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From carl.witty at gmail.com Fri Mar 13 18:24:10 2009 From: carl.witty at gmail.com (Carl Witty) Date: Fri, 13 Mar 2009 10:24:10 -0700 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> <43407.213.61.181.86.1236955563.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <54183.213.61.181.86.1236962387.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: On Fri, Mar 13, 2009 at 10:11 AM, Lisandro Dalcin wrote: > On Fri, Mar 13, 2009 at 2:39 PM, Stefan Behnel wrote: >> So, can't we just require that ExtType.__new__() returns an instance of >> ExtType (or its subtypes)? > > As a first approach, it could be like that... > > ? Or is there a use case that would benefit from >> returning arbitrary things? >> > > Well, Python supports this, we could in case it is possible.... How about this? In the hypothetical future version of Cython that allows overriding __new__, say that the overrides have to be declared in the .pxd file, with cdef TYPE __new__(self, foo, bar) (where TYPE defaults to the class being declared, if omitted). So when generating code, if the class doesn't override __new__, you know that calling the class returns an instance; if the class does override __new__, then the return type is explicitly given. This could be a parent class, for my use case where I want to return an instance of a sibling class; or it could be "object", if anybody ever has a use case for having __new__ return an arbitrary object. Carl From robertwb at math.washington.edu Fri Mar 13 18:41:40 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 13 Mar 2009 10:41:40 -0700 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> <43407.213.61.181.86.1236955563.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <54183.213.61.181.86.1236962387.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: On Mar 13, 2009, at 10:24 AM, Carl Witty wrote: > On Fri, Mar 13, 2009 at 10:11 AM, Lisandro Dalcin > wrote: >> On Fri, Mar 13, 2009 at 2:39 PM, Stefan Behnel >> wrote: >>> So, can't we just require that ExtType.__new__() returns an >>> instance of >>> ExtType (or its subtypes)? >> >> As a first approach, it could be like that... >> >> ? Or is there a use case that would benefit from >>> returning arbitrary things? >>> >> >> Well, Python supports this, we could in case it is possible.... > > How about this? In the hypothetical future version of Cython that > allows overriding __new__, say that the overrides have to be declared > in the .pxd file, with > > cdef TYPE __new__(self, foo, bar) > > (where TYPE defaults to the class being declared, if omitted). So > when generating code, if the class doesn't override __new__, you know > that calling the class returns an instance; if the class does override > __new__, then the return type is explicitly given. This could be a > parent class, for my use case where I want to return an instance of a > sibling class; or it could be "object", if anybody ever has a use case > for having __new__ return an arbitrary object. +1. This will make it possible to optimize the common case while providing (future) flexibility. - Robert From robertwb at math.washington.edu Fri Mar 13 19:01:54 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 13 Mar 2009 11:01:54 -0700 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: <49BA10E3.6090409@behnel.de> References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> Message-ID: <5993C518-1C51-45F4-AA6C-7771F50729B0@math.washington.edu> On Mar 13, 2009, at 12:53 AM, Stefan Behnel wrote: > > Santiago Aguiar wrote: >> I've started to use Cython just yesterday, and I'm facing some >> difficulties ;). I'm sorry to ask this on the dev list, but I didn't >> found a user specific one... >> >> I have a struct-dense C source code that defines the PDU of a network >> protocol and functions to read/write from a buffer struct. >> >> First, I don't know if I'm doing this the right way, but I had to >> define >> by hand all my structs again on Cython to use them. I think it's >> the way >> to do it, but I did feel like repeating myself over and over ;). > > That's the way to do it, though. You only go through that once for > each > struct you use (and your code that uses the struct is likely longer > than > the struct itself), so it's not too much work. > > Note that you do not need to declare the complete struct. Just > write down > the fields you need. Cython doesn't need to know the others and the C > compiler sees the struct anyway. That also allows you to ignore any > externally referenced complex types that you do not use yourself. > > >> Second, at this extremely un-proficient time, for each of the >> structs on >> my pdu.pyx I have something like this: >> >> cdef class Msg1(Message): >> cdef public ... (one declaration for each field present in the >> struct) > > Why don't you reference the struct itself here? Do you really need > these > fields to be public? Some methods for setting up common use cases > are often > a lot better than just having users set everything by hand, > especially when > there are non-trivial type conversions involved, or when there are > interdependencies between fields that are better checked for > validity. For > example, Cython cannot check arbitrary integer values for you that > the user > writes into an enum type attribute. I would second this--you'll have to make properties for each of the fields, but your internal code will be much easier if you can just pass structs around. >> def __init__(self, ... (one keyword parameter for each field >> provided by the struct): >> self.fieldN = ... (this repeated for every field) >> cdef void from_c(self, msg1_c_struct *msg1): >> self.fieldN = msg1.fieldN (repeated for every field, with some >> additional mappings in case the field is a struct) > > There is some special support for struct assignments from dicts, > you can > try that. I have no idea where that's documented, though. Robert? It's not except for at http://hg.cython.org/cython-devel/file/ f2a2d6e68de2/tests/run/struct_conversion.pyx Simply put, if one has cdef struct Point: double x double y One can write "Point(x=1, y=2)" to get a literal struct, and conversion from a struct to a Python object creates a dict. I intended to write conversion the other direction too (from a dict to a struct in the obvious way) but haven't gotten around to it yet. Personally, I would write a simple script that auto-generates this kind of stuff for you in your specific case. - Robert From stefan_ml at behnel.de Fri Mar 13 19:07:00 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 13 Mar 2009 19:07:00 +0100 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> <43407.213.61.181.86.1236955563.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <54183.213.61.181.86.1236962387.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49BAA0C4.8060208@behnel.de> Robert Bradshaw wrote: > On Mar 13, 2009, at 10:24 AM, Carl Witty wrote: >> How about this? In the hypothetical future version of Cython that >> allows overriding __new__, say that the overrides have to be declared >> in the .pxd file, with >> >> cdef TYPE __new__(self, foo, bar) >> >> (where TYPE defaults to the class being declared, if omitted). So >> when generating code, if the class doesn't override __new__, you know >> that calling the class returns an instance; if the class does override >> __new__, then the return type is explicitly given. This could be a >> parent class, for my use case where I want to return an instance of a >> sibling class; or it could be "object", if anybody ever has a use case >> for having __new__ return an arbitrary object. > > +1. This will make it possible to optimize the common case while > providing (future) flexibility. +1 sounds like a good way to do it. Carl, can you write a comment for the ticket so that it has your name in it? I'll change it into a feature request. http://trac.cython.org/cython_trac/ticket/232 Stefan From santiago.aguiar at gmail.com Fri Mar 13 20:39:54 2009 From: santiago.aguiar at gmail.com (Santiago Aguiar) Date: Fri, 13 Mar 2009 16:39:54 -0300 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> <49BA92FA.2070801@gmail.com> Message-ID: <49BAB68A.7060303@gmail.com> Lisandro Dalcin wrote: > Santiago, have you considered writing a simple script to generate > Cython code :-) ?? You could parse a simple txt file with metadata > about your C-level structs (name, field types, fieldname, and even > default field values), then you could easily dump Cython code > automating the member, __cinit__() methods and all the 'property' you > need. > I did, and I'm usually on for codegen, but since I was on a schedule I wanted to avoid having to code the generator, and hoped that something already existed.... besides, the metadata will still have to mirror the C defs... or I would need to parse the .h which... well I want to avoid if at all possible! However, it still could save me some time. > As Stefan commented, you should just store your struct in your cdef > class, and use 'property' to expose the field values with tiny get/set > funcions... Remember that struct member assignments should work in any > C89 compiler... No point to complicate your from_c()/ to_c() cdef > methods ... > OK, I'm not terribly convinced that I will save some time by writing the get/set instead of the from_c to_c, but I'm going to try it! Thanks for the comments!! Regards, Santiago. From stefan_ml at behnel.de Fri Mar 13 20:59:12 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 13 Mar 2009 20:59:12 +0100 Subject: [Cython] Trac usage and bug handling for 0.11.x In-Reply-To: <4DB56201-278E-4068-A185-4AB1947EC239@math.washington.edu> References: <61799.213.61.181.86.1236849271.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B8D940.3090202@student.matnat.uio.no> <4DB56201-278E-4068-A185-4AB1947EC239@math.washington.edu> Message-ID: <49BABB10.8070909@behnel.de> Robert Bradshaw wrote: > On Mar 12, 2009, at 2:43 AM, Dag Sverre Seljebotn wrote: >> Stefan Behnel wrote: >>> Or maybe "cython-unstable" would be a more generic name. >>> >> +1 to cython-devel and cython-unstable. > > Yes, I'll make such branches. > > I like the idea of "safe" stuff (either fixes or minor improvements) > being in cython-devel, and anything more drastic (e.g. moving all the > temp stuff over to the new model) in cython-unstable. A much better > division than "bugfix" and "other." I put an overview of trac usage and branch designation here: http://wiki.cython.org/HackerGuide Stefan From kallacomer at gmail.com Fri Mar 13 21:12:36 2009 From: kallacomer at gmail.com (comer c) Date: Sat, 14 Mar 2009 01:42:36 +0530 Subject: [Cython] [Help]Hacking the cython parser Code Message-ID: <78a091340903131312k222ad46s9de599683bdeeb30@mail.gmail.com> I was looking at the cython compiler code, after sometime i got lost tracing it. What is the best way to trace/analyze the code?For example if i want to see the different phases of the compiler like the tokenizer,parser and code generator?Can you provide me the proper way to begin with code analzing. Much help appreciated in this regard. Thanks, comer -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090314/4c1aee42/attachment.htm From stefan_ml at behnel.de Fri Mar 13 21:45:53 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 13 Mar 2009 21:45:53 +0100 Subject: [Cython] [Help]Hacking the cython parser Code In-Reply-To: <78a091340903131312k222ad46s9de599683bdeeb30@mail.gmail.com> References: <78a091340903131312k222ad46s9de599683bdeeb30@mail.gmail.com> Message-ID: <49BAC601.7040007@behnel.de> comer c wrote: > I was looking at the cython compiler code, after sometime i got lost tracing > it. Yes, there is a lot of code, and it's not always trivial to trace into it. > What is the best way to trace/analyze the code?For example if i want to see > the different phases of the compiler like the tokenizer,parser and code > generator?Can you provide me the proper way to begin with code analzing. > Much help appreciated in this regard. Have you looked into the compiler pipeline in Main.py? (~line 110). It runs after the parser, but the most interesting stuff happens there, actually. Just set a break point into a class that looks interesting and see what it does with the tree. To print the tree, call "node.dump()" at any time, or put a "PrintTree()" into the pipeline at any point. Stefan From robertwb at math.washington.edu Sat Mar 14 02:19:18 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 13 Mar 2009 18:19:18 -0700 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: <49BAB68A.7060303@gmail.com> References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> <49BA92FA.2070801@gmail.com> <49BAB68A.7060303@gmail.com> Message-ID: On Mar 13, 2009, at 12:39 PM, Santiago Aguiar wrote: > Lisandro Dalcin wrote: >> Santiago, have you considered writing a simple script to generate >> Cython code :-) ?? You could parse a simple txt file with metadata >> about your C-level structs (name, field types, fieldname, and even >> default field values), then you could easily dump Cython code >> automating the member, __cinit__() methods and all the 'property' you >> need. >> > I did, and I'm usually on for codegen, but since I was on a schedule I > wanted to avoid having to code the generator, and hoped that something > already existed.... besides, the metadata will still have to mirror > the > C defs... or I would need to parse the .h which... well I want to > avoid > if at all possible! However, it still could save me some time. There was a project (called ccython) that proposed to somewhat automated this kind of stuff. >> As Stefan commented, you should just store your struct in your cdef >> class, and use 'property' to expose the field values with tiny get/ >> set >> funcions... Remember that struct member assignments should work in >> any >> C89 compiler... No point to complicate your from_c()/ to_c() cdef >> methods ... >> > OK, I'm not terribly convinced that I will save some time by > writing the > get/set instead of the from_c to_c, but I'm going to try it! At least I think it would be a more natural interface on the Python side. > Thanks for the comments!! No problem. Thanks for using Cython, and the feedback. - Robert From robertwb at math.washington.edu Sat Mar 14 02:22:05 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 13 Mar 2009 18:22:05 -0700 Subject: [Cython] [Help]Hacking the cython parser Code In-Reply-To: <49BAC601.7040007@behnel.de> References: <78a091340903131312k222ad46s9de599683bdeeb30@mail.gmail.com> <49BAC601.7040007@behnel.de> Message-ID: On Mar 13, 2009, at 1:45 PM, Stefan Behnel wrote: > comer c wrote: >> I was looking at the cython compiler code, after sometime i got >> lost tracing >> it. > > Yes, there is a lot of code, and it's not always trivial to trace > into it. > > >> What is the best way to trace/analyze the code?For example if i >> want to see >> the different phases of the compiler like the tokenizer,parser and >> code >> generator?Can you provide me the proper way to begin with code >> analzing. >> Much help appreciated in this regard. > > Have you looked into the compiler pipeline in Main.py? (~line 110). > It runs > after the parser, but the most interesting stuff happens there, > actually. > Just set a break point into a class that looks interesting and see > what it > does with the tree. To print the tree, call "node.dump()" at any > time, or > put a "PrintTree()" into the pipeline at any point. For the code generation phase, you can uncomment the lines # Uncomment this for debugging. # __metaclass__ = VerboseCodeWriter In Node.py:119-120, which will very verbosely annotate the output (though it hasn't been fully debugged--sometimes the resulting file won't compile). - Robert From robertwb at math.washington.edu Sat Mar 14 08:08:09 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 14 Mar 2009 00:08:09 -0700 Subject: [Cython] cython-support or cython-users Message-ID: <821530A1-A6BA-4143-88D9-6E455CB091D4@math.washington.edu> Should we start a new list for cython support, aimed at end users (and hopefully lower volume) distinct from the devel list? I'd set one on up google groups, but some people have issues with their terms of service, and it'd probably be good to have it be at codespeak like this one. - Robert From stefan_ml at behnel.de Sat Mar 14 08:27:18 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 14 Mar 2009 08:27:18 +0100 Subject: [Cython] cython-support or cython-users In-Reply-To: <821530A1-A6BA-4143-88D9-6E455CB091D4@math.washington.edu> References: <821530A1-A6BA-4143-88D9-6E455CB091D4@math.washington.edu> Message-ID: <49BB5C56.2030707@behnel.de> Robert Bradshaw wrote: > Should we start a new list for cython support, aimed at end users > (and hopefully lower volume) distinct from the devel list? I do think that many of the developer topics are at least superficially interesting to users (Cython users are also Python developers and commonly care about the generated code). However, "superficially" already implies that the volume argument has a certain weight here. I can ask on codespeak.net, in case there is some demand by actual users. Stefan From robertwb at math.washington.edu Sat Mar 14 08:56:19 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 14 Mar 2009 00:56:19 -0700 Subject: [Cython] Cython 0.11 release announcement Message-ID: After much delay, Cython 0.11 is finally released! We would like to thank Stefan Behnel, Robert Bradshaw, Ondrej Certik, Lisandro Dalcin, Jason Evans, Magnus Lie Hetland, Hoyt Koepke, and Dag Sverre Seljebotn who contributed code to this release, as well as the many others who contributed ideas, bug reports, and feedback. The majority of changes in this version are internal, the largest of which is moving most temp allocation to the code generation phase which has several advantages and will greatly facilitate future development. In conjunction with this, Dag Sverre Seljebotn wrote a "reference counting nanny" which, when enabled, warns of potential double-frees and memory-leaks. Though it will not find memory leaks in user code (malloc is just as dangerous as ever) it is a great tool for verifying that Cython itself is outputting correct code and is an invaluable consistency check during the many changes we've been making to temporary variable handling. There have also been many improvements for developers--for example, the parse tree is printed on compiler crashes, the testing framework has been enhanced, and there is a mode to produce output files interlaced with the compiler's call stacks to more easily trace what's going on. The most visible new user-level features are size_t as a native type (as part of Lisandro Dalcin's cleanup and improvement of Python <-> C type conversions) and proper Python semantics for optimized range() loops (Magnus Lie Hetland). C(p)def functions can now take decorators, though only @cython.locals is currently supported, and inline functions can appear in .pxd files. There were also numerious bug fixes, optimizations, and general code cleanups that happened in this release too. A list of closed tickets can be found at http://trac.cython.org/ cython_trac/query?group=component&milestone=0.11 . Though not every change is associated to a trac ticket, we plan to be more stringent about this in the future. We have several patches already done that didn't make it into 0.11, and so 0.11.1 will be out soon. - Robert From robertwb at math.washington.edu Sat Mar 14 09:17:49 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 14 Mar 2009 01:17:49 -0700 (PDT) Subject: [Cython] Cython 0.11 Released Message-ID: <4da1608f-43b0-4ced-846a-cc329d581327@s9g2000prg.googlegroups.com> The Python compiler Cython 0.11 was just released. We would like to thank Stefan Behnel, Robert Bradshaw, Ondrej Certik, Lisandro Dalcin, Jason Evans, Magnus Lie Hetland, Hoyt Koepke, and Dag Sverre Seljebotn who contributed code to this release, as well as the many others who contributed ideas, bug reports, and feedback. What is Cython? ===================== Cython is a language that makes writing C extensions for the Python language as easy as Python itself. Cython is based on the well-known Pyrex, but supports more cutting edge functionality and optimizations. The Cython language is very close to the Python language, but Cython additionally supports calling C functions and declaring C types on variables and class attributes. This allows the compiler to generate very efficient C code from Cython code. This makes Cython the ideal language for wrapping for external C libraries, and for fast C modules that speed up the execution of Python code. Where to get it? ===================== http://cython.org/ http://pypi.python.org/pypi/Cython/ Release Notes ===================== The majority of changes in this version are internal, the largest of which is moving most temp allocation to the code generation phase which has several advantages and will greatly facilitate future development. In conjunction with this, Dag Sverre Seljebotn wrote a "reference counting nanny" which, when enabled, warns of potential double-frees and memory-leaks. Though it will not find memory leaks in user code (malloc is just as dangerous as ever) it is a great tool for verifying that Cython itself is outputting correct code and is an invaluable consistency check during the many changes we've been making to temporary variable handling. There have also been many improvements for developers--for example, the parse tree is printed on compiler crashes, the testing framework has been enhanced, and there is a mode to produce output files interlaced with the compiler's call stacks to more easily trace what's going on. The most visible new user-level features are size_t as a native type (as part of Lisandro Dalcin's cleanup and improvement of Python <-> C type conversions) and proper Python semantics for optimized range() loops (Magnus Lie Hetland). C(p)def functions can now take decorators, though only @cython.locals is currently supported, and inline functions can appear in .pxd files. There were also numerious bug fixes, optimizations, and general code cleanups that happened in this release too. A list of closed tickets can be found at http://trac.cython.org/ cython_trac/query?group=component&milestone=0.11 . Though not every change is associated to a trac ticket, we plan to be more stringent about this in the future. We have several patches already done that didn't make it into 0.11, and so 0.11.1 will be out soon. From kallacomer at gmail.com Sat Mar 14 20:55:54 2009 From: kallacomer at gmail.com (comer c) Date: Sun, 15 Mar 2009 01:25:54 +0530 Subject: [Cython] [Help]Hacking the cython parser Code In-Reply-To: References: <78a091340903131312k222ad46s9de599683bdeeb30@mail.gmail.com> <49BAC601.7040007@behnel.de> Message-ID: <78a091340903141255y65d110a0r204e6886461e6143@mail.gmail.com> I uncommented this line, and I tried with both these commands. Cython dummy.pyx Cython -v dummy.pyx and it did not generate any verbose output.The output was same as before, without uncommenting that line. Am i missing something ? On Sat, Mar 14, 2009 at 6:52 AM, Robert Bradshaw < robertwb at math.washington.edu> wrote: > On Mar 13, 2009, at 1:45 PM, Stefan Behnel wrote: > > > comer c wrote: > >> I was looking at the cython compiler code, after sometime i got > >> lost tracing > >> it. > > > > Yes, there is a lot of code, and it's not always trivial to trace > > into it. > > > > > >> What is the best way to trace/analyze the code?For example if i > >> want to see > >> the different phases of the compiler like the tokenizer,parser and > >> code > >> generator?Can you provide me the proper way to begin with code > >> analzing. > >> Much help appreciated in this regard. > > > > Have you looked into the compiler pipeline in Main.py? (~line 110). > > It runs > > after the parser, but the most interesting stuff happens there, > > actually. > > Just set a break point into a class that looks interesting and see > > what it > > does with the tree. To print the tree, call "node.dump()" at any > > time, or > > put a "PrintTree()" into the pipeline at any point. > > For the code generation phase, you can uncomment the lines > > # Uncomment this for debugging. > # __metaclass__ = VerboseCodeWriter > > In Node.py:119-120, which will very verbosely annotate the output > (though it hasn't been fully debugged--sometimes the resulting file > won't compile). > > - Robert > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090315/d1be8b4c/attachment.htm From robertwb at math.washington.edu Sun Mar 15 00:57:07 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 14 Mar 2009 16:57:07 -0700 Subject: [Cython] [Help]Hacking the cython parser Code In-Reply-To: <78a091340903141255y65d110a0r204e6886461e6143@mail.gmail.com> References: <78a091340903131312k222ad46s9de599683bdeeb30@mail.gmail.com> <49BAC601.7040007@behnel.de> <78a091340903141255y65d110a0r204e6886461e6143@mail.gmail.com> Message-ID: <05DD5AEC-37F4-4670-94B5-9E8FBA75DD9B@math.washington.edu> On Mar 14, 2009, at 12:55 PM, comer c wrote: > I uncommented this line, and I tried with both these commands. > Cython dummy.pyx Cython -v dummy.pyx and it did not generate any > verbose output.The output was same as before, without uncommenting > that line. > Am i missing something ? Look at the .c file. You should see stuff like /* -> ExprStatNode.generate_execution_code (18, 25) */ /* -> SimpleCallNode.generate_evaluation_code (18, 25) */ /* -> SimpleCallNode.generate_subexpr_evaluation_code (18, 25) */ /* -> CoerceToPyTypeNode.generate_evaluation_code (18, 27) */ / * -> CoerceToPyTypeNode.allocate_temp_result (18, 27) */ /* __pyx_t_1 allocated */ / * <- CoerceToPyTypeNode.allocate_temp_result (18, 27) */ / * -> CoerceToPyTypeNode.generate_result_code (18, 27) */ __pyx_t_1 = PyInt_FromLong(__pyx_v_n); if (unlikely(! __pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 18; __pyx_clineno = __LINE__; goto __pyx_L1_error;} __Pyx_GOTREF(__pyx_t_1); / * <- CoerceToPyTypeNode.generate_result_code (18, 27) */ /* <- CoerceToPyTypeNode.generate_evaluation_code (18, 27) */ /* <- SimpleCallNode.generate_subexpr_evaluation_code (18, 25) */ /* -> SimpleCallNode.allocate_temp_result (18, 25) */ /* __pyx_t_3 allocated */ /* <- SimpleCallNode.allocate_temp_result (18, 25) */ - Robert From robertwb at math.washington.edu Sun Mar 15 01:01:41 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 14 Mar 2009 17:01:41 -0700 Subject: [Cython] Trac usage and bug handling for 0.11.x In-Reply-To: <49BABB10.8070909@behnel.de> References: <61799.213.61.181.86.1236849271.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B8D940.3090202@student.matnat.uio.no> <4DB56201-278E-4068-A185-4AB1947EC239@math.washington.edu> <49BABB10.8070909@behnel.de> Message-ID: On Mar 13, 2009, at 12:59 PM, Stefan Behnel wrote: > Robert Bradshaw wrote: >> On Mar 12, 2009, at 2:43 AM, Dag Sverre Seljebotn wrote: >>> Stefan Behnel wrote: >>>> Or maybe "cython-unstable" would be a more generic name. >>>> >>> +1 to cython-devel and cython-unstable. >> >> Yes, I'll make such branches. >> >> I like the idea of "safe" stuff (either fixes or minor improvements) >> being in cython-devel, and anything more drastic (e.g. moving all the >> temp stuff over to the new model) in cython-unstable. A much better >> division than "bugfix" and "other." > > I put an overview of trac usage and branch designation here: > > http://wiki.cython.org/HackerGuide Thanks for the good writeup. - Robert From robertwb at math.washington.edu Sun Mar 15 06:24:28 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 14 Mar 2009 22:24:28 -0700 Subject: [Cython] Integer power operator In-Reply-To: <44766.213.61.181.86.1236769300.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <1286d6e2177fb46620ab4a1f1fa7d7f3.squirrel@webmail.uio.no> <39764.213.61.181.86.1236762819.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B78660.8060904@student.matnat.uio.no> <44766.213.61.181.86.1236769300.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <66738701-AD7B-4153-974C-67C94D5698EC@math.washington.edu> On Mar 11, 2009, at 4:01 AM, Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> Stefan Behnel wrote: >>> Dag Sverre Seljebotn wrote: >>>> I happen to routinely write (some complex integer expression)**2 in >>>> Python, and it is tedious to manually store the expression in a >>>> temporary >>>> etc. in Cython. >> >>> If you can come up with a sensible utility function that handles the >>> different C compile time and runtime overflow cases nicely, I'm >>> all for >>> it. >> >> So you are "+1 if I include an overflow check"? >> >> I was just thinking of letting it silently overflow, the way addition >> and multiplication already silently overflows. > > Hmm, yes, you are right. When a user writes > > cdef int tinyint = 2**2000 > > it's not Cython's fault if it overflows. > > So I guess I'm actually +1 for a fix that provides Python integer > power > semantics modulo C overflows/wrap-arounds. I guess that would be the > semantics with the lowest probability of surprise. > > Would you say that this makes sense as a utility function? > > I mean, we could always write out the bit-shifting code for a constant > exponent... http://trac.cython.org/cython_trac/ticket/127 I only special-cased 0-3, the rest is generic (repeated squaring) code. - Robert From jasone at canonware.com Sun Mar 15 06:47:32 2009 From: jasone at canonware.com (Jason Evans) Date: Sat, 14 Mar 2009 22:47:32 -0700 Subject: [Cython] Integer power operator In-Reply-To: <66738701-AD7B-4153-974C-67C94D5698EC@math.washington.edu> References: <1286d6e2177fb46620ab4a1f1fa7d7f3.squirrel@webmail.uio.no> <39764.213.61.181.86.1236762819.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B78660.8060904@student.matnat.uio.no> <44766.213.61.181.86.1236769300.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <66738701-AD7B-4153-974C-67C94D5698EC@math.washington.edu> Message-ID: <49BC9674.4090905@canonware.com> Robert Bradshaw wrote: > http://trac.cython.org/cython_trac/ticket/127 I only special-cased > 0-3, the rest is generic (repeated squaring) code. Here's some code I wrote a while back that uses binary decomposition to improve performance. I don't know how much it really matters... Jason // Use binary decomposition to reduce the number of multiplication operations // when calculating x^n. See "Hackers Delight", pp 212-213 for further // information. LkmpInline int64_t LkpIntExp(int64_t x, int64_t n) { int64_t ret, p; bool neg = (n < 0); // The main loop only works for non-negative exponents, so transform // negative exponents outside the loop, and clean up at the end with a // division. if (neg) { // This can overflow if n is -2^63, but the results are explicitly // undefined on underflow/overflow, so there is no need to check. n = -n; } for (ret = 1, p = x; n != 0; n >>= 1, p *= p) { if (n & 1) { ret *= p; } } if (neg) { ret = 1 / ret; } return ret; From dagss at student.matnat.uio.no Sun Mar 15 07:14:31 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sun, 15 Mar 2009 07:14:31 +0100 Subject: [Cython] Integer power operator In-Reply-To: <66738701-AD7B-4153-974C-67C94D5698EC@math.washington.edu> References: <1286d6e2177fb46620ab4a1f1fa7d7f3.squirrel@webmail.uio.no> <39764.213.61.181.86.1236762819.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B78660.8060904@student.matnat.uio.no> <44766.213.61.181.86.1236769300.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <66738701-AD7B-4153-974C-67C94D5698EC@math.washington.edu> Message-ID: <49BC9CC7.2050103@student.matnat.uio.no> Robert Bradshaw wrote: > On Mar 11, 2009, at 4:01 AM, Stefan Behnel wrote: > >> Dag Sverre Seljebotn wrote: >>> Stefan Behnel wrote: >>>> Dag Sverre Seljebotn wrote: >>>>> I happen to routinely write (some complex integer expression)**2 in >>>>> Python, and it is tedious to manually store the expression in a >>>>> temporary >>>>> etc. in Cython. >>>> If you can come up with a sensible utility function that handles the >>>> different C compile time and runtime overflow cases nicely, I'm >>>> all for >>>> it. >>> So you are "+1 if I include an overflow check"? >>> >>> I was just thinking of letting it silently overflow, the way addition >>> and multiplication already silently overflows. >> Hmm, yes, you are right. When a user writes >> >> cdef int tinyint = 2**2000 >> >> it's not Cython's fault if it overflows. >> >> So I guess I'm actually +1 for a fix that provides Python integer >> power >> semantics modulo C overflows/wrap-arounds. I guess that would be the >> semantics with the lowest probability of surprise. >> >> Would you say that this makes sense as a utility function? >> >> I mean, we could always write out the bit-shifting code for a constant >> exponent... > > http://trac.cython.org/cython_trac/ticket/127 I only special-cased > 0-3, the rest is generic (repeated squaring) code. Great! -- Dag Sverre From robertwb at math.washington.edu Sun Mar 15 07:09:58 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 14 Mar 2009 23:09:58 -0700 Subject: [Cython] Integer power operator In-Reply-To: <49BC9674.4090905@canonware.com> References: <1286d6e2177fb46620ab4a1f1fa7d7f3.squirrel@webmail.uio.no> <39764.213.61.181.86.1236762819.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49B78660.8060904@student.matnat.uio.no> <44766.213.61.181.86.1236769300.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <66738701-AD7B-4153-974C-67C94D5698EC@math.washington.edu> <49BC9674.4090905@canonware.com> Message-ID: <89C94664-180F-4C10-A169-A1F4A85D47DF@math.washington.edu> On Mar 14, 2009, at 10:47 PM, Jason Evans wrote: > Robert Bradshaw wrote: >> http://trac.cython.org/cython_trac/ticket/127 I only special-cased >> 0-3, the rest is generic (repeated squaring) code. > > Here's some code I wrote a while back that uses binary > decomposition to > improve performance. I don't know how much it really matters... > > Jason > > // Use binary decomposition to reduce the number of multiplication > operations > // when calculating x^n. See "Hackers Delight", pp 212-213 for > further > // information. > LkmpInline int64_t > LkpIntExp(int64_t x, int64_t n) { > int64_t ret, p; > bool neg = (n < 0); > > // The main loop only works for non-negative exponents, so > transform > // negative exponents outside the loop, and clean up at the > end with a > // division. > if (neg) { > // This can overflow if n is -2^63, but the results are > explicitly > // undefined on underflow/overflow, so there is no need to > check. > n = -n; > } > > for (ret = 1, p = x; > n != 0; > n >>= 1, p *= p) { > if (n & 1) { > ret *= p; > } > } > > if (neg) { > ret = 1 / ret; > } > > return ret; Yep, this is basically what I implemented. The switch at the top is handy for small exponents, and is primarily for constant exponents. The messy expression at the top of the loop is to avoid branching (may or may not be worth it--depends on how randomly the bits are set in the exponent). ---------- static INLINE %(type)s %(func_name)s(%(type)s b, %(type)s e) { %(type)s t = b; switch (e) { case 3: t *= b; case 2: t *= b; case 1: return t; case 0: return 1; } if (unlikely(e<0)) return 0; t = 1; while (likely(e)) { t *= (b * (e&1)) | ((~e)&1); /* 1 or b */ b *= b; e >>= 1; } return t; From stefan_ml at behnel.de Sun Mar 15 09:17:40 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 15 Mar 2009 09:17:40 +0100 Subject: [Cython] [Help]Hacking the cython parser Code In-Reply-To: References: <78a091340903131312k222ad46s9de599683bdeeb30@mail.gmail.com> <49BAC601.7040007@behnel.de> Message-ID: <49BCB9A4.3050307@behnel.de> Robert Bradshaw wrote: > For the code generation phase, you can uncomment the lines > > # Uncomment this for debugging. > # __metaclass__ = VerboseCodeWriter > > In Node.py:119-120, which will very verbosely annotate the output I moved the "uncommenting" step into the DebugFlags module as "debug_trace_code_generation" and added a section to the Hacker-Guide about this: http://wiki.cython.org/HackerGuide#Gettingstarted Stefan From m.gysel at gmail.com Sun Mar 15 13:50:59 2009 From: m.gysel at gmail.com (Martin Gysel) Date: Sun, 15 Mar 2009 13:50:59 +0100 Subject: [Cython] cdef class understanding Message-ID: <49BCF9B3.1030601@gmail.com> Hi I'm quite new to cython and try to bind some c code to python. Now I'm not sure if I understood the cython doc correctly since what I'm trying do do doesn't work :-( It looks like this: cdef class Cobj: cdef a_c_struct_t *loop def __init__(self): self.loop = c_lib_call() c_lib_call_reg_callback(self.callback, NULL) cdef void callback(self, void *userdata) with gil: # but here, self.loop seems to be NULL instead of something # to test this: # if self.loop == NULL: print "isNull" doSomething(self.loop) in the callback self.loop is NULL but in the init it wasn't. is it possible to access self and its members in a c callback? I also tried to use __cinit__ or __new__ instead of __init__ but always the same behavior. thanks fro your comments martin From ggellner at uoguelph.ca Sun Mar 15 18:52:15 2009 From: ggellner at uoguelph.ca (Gabriel Gellner) Date: Sun, 15 Mar 2009 13:52:15 -0400 Subject: [Cython] Optional arguments question Message-ID: <20090315175215.GA10289@encolpuis> I can't seem to get optional arguments for a char variable to work in a cdef function, is this a bug or am I missing something simple? Any help would, as always, be greatly appreciated. For example: cdef void test_args(int number, char letter='C'): print letter def test(): test_args(3, letter='F') Gives a cryptic compile compile error (It doesn't work with c'F' either). Gabriel From greg.ewing at canterbury.ac.nz Sun Mar 15 22:48:45 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 16 Mar 2009 09:48:45 +1200 Subject: [Cython] User Question - Questions and 'Cannot convert to Python Object'... In-Reply-To: <43407.213.61.181.86.1236955563.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49B9A33E.2080405@gmail.com> <49BA10E3.6090409@behnel.de> <43407.213.61.181.86.1236955563.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49BD77BD.9050609@canterbury.ac.nz> Stefan Behnel wrote: > Carl Witty wrote: > >>Hmm... I hope that eventually Cython will allow overriding __new__, at >>which point calling an extension type won't necessarily return a value >>of that type. > > I'm not sure. This should certainly be supported for Python classes, but > for extension types? What would be the use case? If it is supported, it could perhaps be handled by allowing the return type of the __new__ method to be declared explicitly, defaulting to the class. E.g. given cdef class Foo: def __new__(cls, ..): ... then Foo() is assumed to return something of type Foo, but given cdef class Blarg: def object __new__(cls, ..): ... then Blarg() could return any type of object. -- Greg From psp.reachable at gmail.com Mon Mar 16 04:53:37 2009 From: psp.reachable at gmail.com (Prajwal Suhas P) Date: Mon, 16 Mar 2009 09:23:37 +0530 Subject: [Cython] Beginner-friendly tickets In-Reply-To: <49B8263C.6090706@student.matnat.uio.no> References: <49B783FA.5070309@student.matnat.uio.no> <49B82598.3000303@student.matnat.uio.no> <49B8263C.6090706@student.matnat.uio.no> Message-ID: I changed the code to cdef int g(): print "g called" return 3 def f(): for i in range(g()): print i print i we get the right output here g called 0 1 2 2 the c output changes(and is wrong) when we declare a "cdef int i" before the for loop.i missed this part initially. On Thu, Mar 12, 2009 at 2:29 AM, Dag Sverre Seljebotn < dagss at student.matnat.uio.no> wrote: > Dag Sverre Seljebotn wrote: > > Kurt Smith wrote: > >> On Wed, Mar 11, 2009 at 4:27 AM, Dag Sverre Seljebotn > >> wrote: > >>> Since there's been some interest in how to get started coding on Cython > >>> on the mailing list lately, I thought I'd call attention to these two > >>> tickets: > >>> > >>> http://trac.cython.org/cython_trac/ticket/158 > >>> http://trac.cython.org/cython_trac/ticket/203 > >>> > >>> These seems to fit the sweet spot of not being too hard to fix for > >>> beginners (with some mentoring), but not being completely trivial > either. > >>> > >>> I'm happy to provide an attack strategy/small amount of mentoring for > >>> these tickets if anybody's interested. > >> Hi Dag, > >> > >> I'm interested in working on #158 if it isn't already claimed. > > > > Nice! I haven't heard anything on #158 yet, so consider it yours. I've > > assigned both tickets now in trac. > > > > You may want to get Trac accounts if you haven't already BTW, send an > > .htpasswd-file to Robert for that. > > > >> One solution (as mentioned in the ticket comments) would be to raise a > >> compilation error since the s variable is declared after it's used. I > >> imagine some checks would be in order to ensure this, but I don't have > >> ideas about where to start. > > > > Yep, that's the solution I'm thinking about. > > > > The Cython "parse tree" is processed in stages. Two of those are > > analyse_declarations and analyse_expressions. The problem here is that > > "cdef str s" is handled in the analyse_declarations step and then > > disappears; while the analyse_expressions happen afterwards, and cannot > > know at which point in the source the declaration happened. > > > > So the analyse declarations phase must do this check. > > > > You will need to do your work in AnalyseDeclarationsTransform, in > > ParseTreeTransforms.py. It is a "filter" on the parse tree, and each > > method is called as you go according to the type of the node. Read some > > of the different transforms in that file to get the idea (to see the > > parse tree, insert e.g. "print node.dump()" at the beginning of > > visit_ModuleNode). > > > > Now: > > > > a) Keep a dict in self containing the names that has been referenced up > > to that point in the given scope (i.e. in visit_ModuleNode and > > visit_FuncDefNode you need to push/pop which dict one is using > > before/after processing the contents). > > Actually I meant to say "set" everywhere I said "dict". > > -- > Dag Sverre > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Regards, Prajwal S. "There is little difference in people,but this little difference makes a big difference.The little difference is attitude,the big difference is whether its positive or negative. " -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090316/908dcd6d/attachment.htm From kwmsmith at gmail.com Mon Mar 16 06:26:49 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Mon, 16 Mar 2009 00:26:49 -0500 Subject: [Cython] Beginner-friendly tickets In-Reply-To: <49B8263C.6090706@student.matnat.uio.no> References: <49B783FA.5070309@student.matnat.uio.no> <49B82598.3000303@student.matnat.uio.no> <49B8263C.6090706@student.matnat.uio.no> Message-ID: Thanks for the guidance -- I'm making progress but I can't figure out one issue: when I add the following method to AnalyzeDeclarationsTransform everything works as expected: def visit_NameNode(self, node): print node.dump() return node It is called for every name node in the parse tree. but this one is not called for every matching node: def visit_CNameDeclaratorNode(self, node): print node.dump() return node Specifically, it is only called when the CNameDeclaratorNode is part of (a descendant of) a CArgDeclNode. When it's part of (a descendant of) a CVarDefNode it is not called. I'm happy to give an example if it would help. In general, if I add a visit_(self, node) method in one of the Transform classes that forms the pipeline, should I expect that node type to be visited provided it exists in the tree at that point in the pipeline? The bug is partially fixed, but if I can resolve this issue it will help fix some corner cases. Thanks, Kurt From dagss at student.matnat.uio.no Mon Mar 16 08:25:50 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 16 Mar 2009 08:25:50 +0100 Subject: [Cython] Beginner-friendly tickets In-Reply-To: References: <49B783FA.5070309@student.matnat.uio.no> <49B82598.3000303@student.matnat.uio.no> <49B8263C.6090706@student.matnat.uio.no> Message-ID: <49BDFEFE.70805@student.matnat.uio.no> Prajwal Suhas P wrote: > I changed the code to > > cdef int g(): > print "g called" > return 3 > > def f(): > for i in range(g()): > print i > print i > > we get the right output here > > g called > 0 > 1 > 2 > 2 > > the c output changes(and is wrong) when we declare a "cdef int i" before > the for loop.i missed this part initially. Yes, when "i" is not typed as int, the Python "range" function is called, which gives the right behaviour. The task is to make Cython do the right thing when i is declared "cdef int" as well. -- Dag Sverre From m.gysel at gmail.com Mon Mar 16 08:32:10 2009 From: m.gysel at gmail.com (Martin Gysel) Date: Mon, 16 Mar 2009 08:32:10 +0100 Subject: [Cython] cdef class understanding, 2nd Message-ID: <49BE007A.4050004@gmail.com> Hi It seems to me like my post yesterday didn't make it to the subscriber, at least not back to me although I should receive my own posts... which is listed in the list archive. So I repost me concern and hope I'll get a copy this time. If everyone out there already got my first post, sorry for that. I'm quite new to cython and try to bind some c code to python. Now I'm not sure if I understood the cython doc correctly since what I'm trying do do doesn't work :-( It looks like this: cdef class Cobj: cdef a_c_struct_t *loop def __init__(self): self.loop = c_lib_call() c_lib_call_reg_callback(self.callback, NULL) cdef void callback(self, void *userdata) with gil: # but here, self.loop seems to be NULL instead of something # to test this: # if self.loop == NULL: print "isNull" doSomething(self.loop) in the callback self.loop is NULL but in the init it wasn't. is it possible to access self and its members in a c callback? I also tried to use __cinit__ or __new__ instead of __init__ but always the same behavior. thanks fro your comments martin From stefan_ml at behnel.de Mon Mar 16 08:37:23 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 16 Mar 2009 08:37:23 +0100 Subject: [Cython] Beginner-friendly tickets In-Reply-To: References: <49B783FA.5070309@student.matnat.uio.no> <49B82598.3000303@student.matnat.uio.no> <49B8263C.6090706@student.matnat.uio.no> Message-ID: <49BE01B3.9070907@behnel.de> Kurt Smith wrote: > Thanks for the guidance -- I'm making progress but I can't figure out one issue: > > when I add the following method to AnalyzeDeclarationsTransform > everything works as expected: > > def visit_NameNode(self, node): > print node.dump() > return node > > It is called for every name node in the parse tree. > > but this one is not called for every matching node: > > def visit_CNameDeclaratorNode(self, node): > print node.dump() > return node > > Specifically, it is only called when the CNameDeclaratorNode is part > of (a descendant of) a CArgDeclNode. When it's part of (a descendant > of) a CVarDefNode it is not called. > > I'm happy to give an example if it would help. > > In general, if I add a visit_(self, node) method in one of > the Transform classes that forms the pipeline, should I expect that > node type to be visited provided it exists in the tree at that point > in the pipeline? There are two issues that might hit here. One is that (IIRC) declarators are removed from the tree at some point in the pipeline as they have done their duty after type analysis and would just increase the overall runtime of the remaining tree operations. Note the IIRC. At least, I'm sure about the fact that there is a transformation base class that ignores them explicitly. So watch out for the class hierarchy of your transformation. The second thing is that a method "visit_SomeNode()" only matches if there is not a more specific method match. E.g., "visit_Node()" will not get called for an expression node if there is a "visit_ExprNode()" sitting in the same class. Does that help? Stefan From dagss at student.matnat.uio.no Mon Mar 16 09:40:51 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 16 Mar 2009 09:40:51 +0100 Subject: [Cython] [Fwd: Re: Beginner-friendly tickets] Message-ID: <49BE1093.7080505@student.matnat.uio.no> Ahh; this was sent to my private address as well and so cython-dev in its wisdom decided not to send me a copy from the mailing list. Forwarding my response to the mailing list as well then. -------- Original Message -------- Subject: Re: [Cython] Beginner-friendly tickets Date: Mon, 16 Mar 2009 08:23:26 +0100 From: Dag Sverre Seljebotn To: Kurt Smith References: <49B783FA.5070309 at student.matnat.uio.no> <49B82598.3000303 at student.matnat.uio.no> <49B8263C.6090706 at student.matnat.uio.no> Kurt Smith wrote: > Thanks for the guidance -- I'm making progress but I can't figure out one issue: > > when I add the following method to AnalyzeDeclarationsTransform > everything works as expected: > > def visit_NameNode(self, node): > print node.dump() > return node > > It is called for every name node in the parse tree. > > but this one is not called for every matching node: > > def visit_CNameDeclaratorNode(self, node): > print node.dump() > return node > > Specifically, it is only called when the CNameDeclaratorNode is part > of (a descendant of) a CArgDeclNode. When it's part of (a descendant > of) a CVarDefNode it is not called. Great that you're making progress. Yes, there is something I'd rather tell you about this :-) The transform works by recursive descent, and when a node type is intercepted, one needs to actively visit the children as well. This is done by self.visitchildren(node); see in visit_FuncDefNode for an example. Since visitchildren is never called in CVarDefNode, the children of CVarDefNode are never visited. Also, the node will be replaced with whatever it returns. visit_CVarDefNode returns None so that the entire declaration is removed from the tree, as it is no longer needed (which is why one isn't analysing the children as well). (Depending on what you need this for, you could be able to extract the information from the attributes of the enclosing FuncDefNode; I think there's a Scope there somewhere which you can inspect the entries of.) -- Dag Sverre -- Dag Sverre From dagss at student.matnat.uio.no Mon Mar 16 10:35:55 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 16 Mar 2009 10:35:55 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> Message-ID: <49BE1D7B.7090009@student.matnat.uio.no> Robert Bradshaw wrote: > On Mar 12, 2009, at 2:15 AM, William Stein wrote: > >> On Thu, Mar 12, 2009 at 1:46 AM, Dag Sverre Seljebotn >> wrote: >>>> This could be very useful for debugging things, but it implies >>>> there's a single, correct way that the % and // operators behave. >>>> >>>> The problem is that sometimes I want to run code with Python >>>> semantics (e.g. I'm quickly cythonizing a file) and sometimes I want >>>> to run code with C semantics (e.g. I'm doing linear algebra mod p, >>>> and don't want the overhead of fixing the sign). And perhaps I'm to >>>> demanding, but I want to be able to use % in both places rather than >>>> know some obscure function call. >>>> >>> Perhaps you could ask the Sage list to get some more input of what >>> the >>> typical expectations to Cython are? >>> >>> (Also if you work in Z_p, could you not use unsigned ints? Then you >>> don't get the overhead? Though I'm avoiding unsigned types like the >>> plague myself after I figured out that range(-n, n) is empty if n is >>> unsigned :-)) >> If you compute the sum of x and y mod p by doing "(x+y)%p", then the >> time is dominated by the % operation, which can easily be nearly 10 >> times longer than +. Now imagine doing linear algebra, where you take >> dot products, etc., and do lots of mod p arithmetic. If you dot two >> vectors with n entries in the naive way you end up doing n %p's, which >> is very expensive. Now imagine that you represent the entries of your >> vectors as C ints between -p/2 and p/2. Then you can very often do >> much of the dot product and only have to do very few mods, since >> frequently when you add up a bunch of numbers between -(p/2)^2 and >> (p/2)^2, they are likely to not overflow. The sum will be close to 0 >> because of cancellation. > > I was about to answer the very same thing then this email came in. > Also, as you mentioned unsigned ints have subtle semantics, like the > above. Another bad (easy--I've corrected it several times) mistake is > when one writes (a-b)%p. If a and b are unsigned, this could be > totally wrong. Of course, the main goal is to avoid using any > division as much as possible. > >>> Perhaps you could ask the Sage list to get some more input of what >>> the >>> typical expectations to Cython are? >> I often think of Cython primarily as a "Python compiler", and from >> that perspective (-1 % 16) being different in Cython and Python is >> worrisome, since it will surely lead to subtle bugs. For me, >> Cython is all about writing fast code that is easy to use from Python. >> Cython is not about writing C code; if I want to write C code I write >> it in C. Cython is not the C language after all. Also, I think many >> current and potential users of Cython (at least through Sage) don't >> even know C, or if they do, they are using Cython so they don't have >> to use C. > > Good point. I'm going to start a thread there just to get a straw > poll. It's the largest community of Cython users I know of... I did the same for NumPy. The results are here: http://thread.gmane.org/gmane.comp.python.numeric.general/28726 (And sage-devel thread is here: http://groups.google.com/group/sage-devel/browse_thread/thread/be9e2ef6a9745575 ) Both communities seems to mirror the discussion we've already had here very closely. There's one group who sees Cython as "fast Python" who want Python semantics and another who sees it as "mixed Python and C" who want the C semantics. (BTW NumPy uses the C semantics both for ints and floats, i.e. when doing % or // on arrays.) -- Dag Sverre From stefan_ml at behnel.de Mon Mar 16 11:59:15 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 16 Mar 2009 11:59:15 +0100 (CET) Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49BE1D7B.7090009@student.matnat.uio.no> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> Message-ID: <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > Both communities seems to mirror the discussion we've already had here > very closely. There's one group who sees Cython as "fast Python" who > want Python semantics and another who sees it as "mixed Python and C" > who want the C semantics. I was also rather unsure at the beginning, but when I think of the consequences for totally Python-looking Cython source code like [ y % 5 for y in some_iterable ] where you have to a) look up the type of "y" *and* b) know the associated semantics of the % operator, before you even know what the numeric range of the result will be - I think that having C semantics here is just too dangerous to be the default. We should not forget that the syntax we write in is Python, so it's a sure trap to assign C semantics to obvious looking things like the above. My clear vote is for Python-semantics on "x%y" and for a helper function "cython.cmod(x,y)" that boils down to a straight and fast C mod operator without value fixing. Stefan PS: Dag, I read your comment on "(2147483647 + 2147483647)" wrapping around in current Cython. I never thought about that, simply since I never ran into it. However, when I write py_large_val = 2147483647 + 2147483647 it must never wrap around as it clearly requests Python semantics, and when I write cdef uint64 c_large_val = 2147483647 + 2147483647 it must not wrap around either. I'm fine with a wrap-around in cdef int c_large_val = 2147483647 + 2147483647 though, because that's what the user a) requested or at least b) must be aware of when explicitly assigning types. So that's clearly different from a discussion on operator semantics. From stefan_ml at behnel.de Mon Mar 16 12:23:15 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 16 Mar 2009 12:23:15 +0100 Subject: [Cython] cdef class understanding In-Reply-To: <49BCF9B3.1030601@gmail.com> References: <49BCF9B3.1030601@gmail.com> Message-ID: <49BE36A3.7000500@behnel.de> Hi, welcome to Cython. Martin Gysel wrote: > I'm quite new to cython and try to bind some c code to python. Now I'm > not sure if I understood the cython doc correctly since what I'm trying > do do doesn't work :-( > It looks like this: > > cdef class Cobj: > cdef a_c_struct_t *loop > > def __init__(self): > self.loop = c_lib_call() > c_lib_call_reg_callback(self.callback, NULL) > > cdef void callback(self, void *userdata) with gil: > # but here, self.loop seems to be NULL instead of something > # to test this: > # if self.loop == NULL: print "isNull" > doSomething(self.loop) > > in the callback self.loop is NULL but in the init it wasn't. is it > possible to access self and its members in a c callback? Yes. However, since "callback" is a C function (and not a Python method), it cannot remember what instance it belongs to. You will have to pass along the pointer to its Python object yourself. This is what the "userdata" pointer is there for. It should work to do this: cdef class Cobj: cdef a_c_struct_t *loop def __init__(self): self.loop = c_lib_call() c_lib_call_reg_callback(self.callback, self) cdef void callback(self) with gil: if self.loop == NULL: print "isNull" doSomething(self.loop) There is also a callback example under "Demos/callback/" in the Cython sources. Stefan From m.gysel at gmail.com Mon Mar 16 14:08:25 2009 From: m.gysel at gmail.com (Martin Gysel) Date: Mon, 16 Mar 2009 14:08:25 +0100 Subject: [Cython] cdef class understanding In-Reply-To: <49BE36A3.7000500@behnel.de> References: <49BCF9B3.1030601@gmail.com> <49BE36A3.7000500@behnel.de> Message-ID: <49BE4F49.8090005@gmail.com> Stefan Behnel schrieb: > Hi, > > welcome to Cython. > > Martin Gysel wrote: >> I'm quite new to cython and try to bind some c code to python. Now I'm >> not sure if I understood the cython doc correctly since what I'm trying >> do do doesn't work :-( >> It looks like this: >> >> cdef class Cobj: >> cdef a_c_struct_t *loop >> >> def __init__(self): >> self.loop = c_lib_call() >> c_lib_call_reg_callback(self.callback, NULL) >> >> cdef void callback(self, void *userdata) with gil: >> # but here, self.loop seems to be NULL instead of something >> # to test this: >> # if self.loop == NULL: print "isNull" >> doSomething(self.loop) >> >> in the callback self.loop is NULL but in the init it wasn't. is it >> possible to access self and its members in a c callback? > > Yes. However, since "callback" is a C function (and not a Python method), > it cannot remember what instance it belongs to. You will have to pass along > the pointer to its Python object yourself. This is what the "userdata" > pointer is there for. It should work to do this: > > > cdef class Cobj: > cdef a_c_struct_t *loop > > def __init__(self): > self.loop = c_lib_call() > c_lib_call_reg_callback(self.callback, self) > > cdef void callback(self) with gil: > if self.loop == NULL: print "isNull" > doSomething(self.loop) thanks Stefan, but now it is a little bit more complex as my callback expects more than one argument to register: c_lib_call_reg_callback(mycontext, self.callback, self) and the callback (as plain c fuction) void callback(context *c, void *userdata) according to your code above, the first argument where the callback gets called is the first argument of the declared callback (in code above, cb(userdata) -> callback(self): userdata->self) so I assume if I have more arguments it works in the same way but then self would be context instead of userdata (at this point code seems to get kind of ugly): cdef void callback(self, void *userdata) now I'm asking myself what's the benefit to declare a c callback in the class rather then outside and and call another class method from there, like this: cdef void callback(context *c, void *userdata): doWhatHasToBeDone(c) anotherOperation((userdata).loop) itsDone((userdata).cbDone()) what's the best way to do such things? /martin From stefan_ml at behnel.de Mon Mar 16 14:41:49 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 16 Mar 2009 14:41:49 +0100 (CET) Subject: [Cython] cdef class understanding In-Reply-To: <49BE4F49.8090005@gmail.com> References: <49BCF9B3.1030601@gmail.com> <49BE36A3.7000500@behnel.de> <49BE4F49.8090005@gmail.com> Message-ID: <33575.213.61.181.86.1237210909.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Martin Gysel wrote: > it is a little bit more complex as my callback > expects more than one argument > to register: > c_lib_call_reg_callback(mycontext, self.callback, self) > > and the callback (as plain c fuction) > void callback(context *c, void *userdata) > > according to your code above, the first argument where the callback gets > called is the first argument of the declared callback (in code above, > cb(userdata) -> callback(self): userdata->self) Yes. That's a special case that would have worked in your original example. > now I'm asking myself what's the benefit to declare a c callback in the > class rather then outside and and call another class method from there, > like this: > cdef void callback(context *c, void *userdata): > doWhatHasToBeDone(c) > anotherOperation((userdata).loop) > itsDone((userdata).cbDone()) > > what's the best way to do such things? It's perfectly ok (and very common, due to signature constraints) to define a callback function outside of a class and to call back into some class instance from inside the callback (just as you do above). In many cases, your callback context/user-data will be a Python object anyway. It really just depends on what your callback does. Stefan From dagss at student.matnat.uio.no Mon Mar 16 15:54:23 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 16 Mar 2009 15:54:23 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49BE681F.8080005@student.matnat.uio.no> Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: > >> Both communities seems to mirror the discussion we've already had here >> very closely. There's one group who sees Cython as "fast Python" who >> want Python semantics and another who sees it as "mixed Python and C" >> who want the C semantics. >> > > I was also rather unsure at the beginning, but when I think of the > consequences for totally Python-looking Cython source code like > > [ y % 5 for y in some_iterable ] > > where you have to a) look up the type of "y" *and* b) know the associated > semantics of the % operator, before you even know what the numeric range > of the result will be - I think that having C semantics here is just too > dangerous to be the default. We should not forget that the syntax we write > in is Python, so it's a sure trap to assign C semantics to obvious looking > things like the above. > > My clear vote is for Python-semantics on "x%y" and for a helper function > "cython.cmod(x,y)" that boils down to a straight and fast C mod operator > without value fixing. > Agreed...but like Robert I'm worried about the usability of requiring to use cython.cmod and cython.cdiv (the latter being kist as important here); especially as one would typically use them in lots of situations (where one is only dividing positive integers anyway) to avoid the speed penalty (the 20% penalty reported by William is a bit much!). So I'm myself leaning towards new operators (in addition to cython.cmod/cdiv) for the C semantics (e.g. /- and %-, or ^/ and ^%, or similar -- as long as we stay away from /~ etc which is proposed in some PEPs). Yes, this is ugly! -- but I see them as less ugly than the alternatives. It seems that tomorrow I may have a different opinion though, so I may just let the matter rest for some days for what I'm concerned. Dag Sverre From stefan_ml at behnel.de Mon Mar 16 14:28:34 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 16 Mar 2009 14:28:34 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <85e81ba30903121618q2506d10fxb9ab6b5899032a4f@mail.gmail.com> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <85e81ba30903121618q2506d10fxb9ab6b5899032a4f@mail.gmail.com> Message-ID: <49BE5402.7050009@behnel.de> Hi, William Stein wrote: > I just tried these two programs: > > main() { > int n = -1, m = 16, k, a=0; > for(k=1;k<100000000;k++) { > a = n%k + (a<0)?k:0; > } > } Note that this does not have the proposed semantics. > and > > main() { > int n = -1, m = 16, k, a=0; > for(k=1;k<100000000;k++) { > a = n%k; > } > } > > and the first takes 22% longer than the second. Here's what I tried. I added another factor of 10 to the loop to get readable timings. ------------------------------------ /* cmod.c */ main() { int n = -1, m = 16, k, a=0; for(k=1;k<1000000000;k++) { a += n%k; n = -n; } printf("%i", a); } /* py2mod.c */ main() { int n = -1, m = 16, k, a=0; for(k=1;k<1000000000;k++) { int mod = n%k; if (mod<0) mod+=k; a += mod; n = -n; } printf("%i", a); } /* pymod.c */ main() { int n = -1, m = 16, k, a=0; for(k=1;k<1000000000;k++) { int mod = n%k; mod += (mod<0)*k; a += mod; n = -n; } printf("%i", a); } ---------------------------------- ./cmod 0 real 0m3.789s user 0m3.704s sys 0m0.000s ./py2mod -371654657 real 0m4.893s user 0m4.856s sys 0m0.004s ./pymod -371654657 real 0m5.623s user 0m5.452s sys 0m0.004s So, yes, there is a performance difference of up to 30% even for the fastest (BTW, branching) implementation. For a constant power-of-2 divisor (m=16), the difference is about 17% for me: ./cmod -1 real 0m3.316s user 0m3.268s sys 0m0.000s ./py2mod -589934593 real 0m3.880s user 0m3.868s sys 0m0.000s ./pymod -589934593 real 0m4.634s user 0m4.580s sys 0m0.000s When I switch to "unsigned int" for "mod" (i.e. for "unsigned int n=3"), I end up with about the same numbers as for cmod (signed or unsigned), which means that gcc drops the if-branch as expected. Looking at the absolute numbers, a speed penalty of even 30% does not sound too bad for me. If better performance is really required (i.e. inside a tight loop), you can switch to "unsigned int" (if you know that you never use negative values), or use a dedicated helper function in the "cython" namespace. Anyway, given that I'm not a heavy numerics user of '%' I guess it's better to take my weight out of this discussion. The (artificially determined) 30% might still be too much for some people, when it's the default. Stefan From sturla at molden.no Mon Mar 16 16:05:37 2009 From: sturla at molden.no (Sturla Molden) Date: Mon, 16 Mar 2009 16:05:37 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <49BE6AC1.3000303@molden.no> On 3/16/2009 11:59 AM, Stefan Behnel wrote: > My clear vote is for Python-semantics on "x%y" and for a helper function > "cython.cmod(x,y)" that boils down to a straight and fast C mod operator > without value fixing. As I see it, cdef int a, b, c is a request to "step into C" when using these variables. Congratulation on your efforts to produce a broken C. "If it ain't broke, don't fix it." Sturla Molden From stefan_ml at behnel.de Mon Mar 16 16:40:17 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 16 Mar 2009 16:40:17 +0100 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49BE6AC1.3000303@molden.no> References: <49B8A9AF.6060806@behnel.de> <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> Message-ID: <49BE72E1.608@behnel.de> Sturla Molden wrote: > On 3/16/2009 11:59 AM, Stefan Behnel wrote: > >> My clear vote is for Python-semantics on "x%y" and for a helper function >> "cython.cmod(x,y)" that boils down to a straight and fast C mod operator >> without value fixing. > > As I see it, > > cdef int a, b, c > > is a request to "step into C" when using these variables. It's hard to have an argument about personal impressions (as in "as I see it"), so I won't go there. But let me clarify that cdef int a, b, c means: "use C type 'int' for a, b and c". This changes the value range of the variables and makes various operators overflow if the value range is too tight. That's what users must expect when they assign a numeric type with a restricted value range. It's also ok if an operator behaves different for an "int" and an "unsigned int", because both are explicitly declared as such by the user in order to make exactly the distinction between an unsigned and a signed type. It's a very different thing if an operator changes its target value range because of a type declaration that is perfectly suited to match the expected target range. A C "int" can represent all values that "some_pyint%5" can take. The same applies to a Python int target and a C "unsigned int" target because of the semantics of the % operator in Python. All three are equivalent from the point of view of this operation. Here, it is simply not true that the user "must" expect the semantics to change when typing the variable. Instead, it comes as a side effect, because it's not the variable with its assigned value range that imposes these semantics, but the operator itself, which simply behaves different when working on exactly the same input. And it behaves different in a place that can be very hard to find, as it's extremely unlikely that users back this kind of code by unit tests. Personally, I think Cython should not be designed to enforce unexpected code behaviour, and definitely not by default. Stefan From hoytak at cs.ubc.ca Mon Mar 16 17:03:51 2009 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Mon, 16 Mar 2009 09:03:51 -0700 Subject: [Cython] Semantics of cdef int division and %? In-Reply-To: <49BE6AC1.3000303@molden.no> References: <6C62B083-C7C8-4691-8B6D-935799885E3B@math.washington.edu> <02CF70CC-54F7-430E-A03B-98808254740E@math.washington.edu> <49B8CBE0.5050305@student.matnat.uio.no> <85e81ba30903120215o651ee059tb9bffbd3bb80ee18@mail.gmail.com> <819E0F37-016F-45FC-930E-426E23A9C0A3@math.washington.edu> <49BE1D7B.7090009@student.matnat.uio.no> <58486.213.61.181.86.1237201155.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49BE6AC1.3000303@molden.no> Message-ID: <4db580fd0903160903w4a898eb0j876735653e4482ae@mail.gmail.com> Hello, I'm +1 on preserving python semantics wherever possible. In my field (statistics), many people do