From rasky at develer.com Sun Aug 2 17:51:15 2009 From: rasky at develer.com (Giovanni Bajo) Date: Sun, 2 Aug 2009 15:51:15 +0000 (UTC) Subject: [Cython] Include paths for pyximport (continued) References: <1249059202.5648.19.camel@leonidas> Message-ID: On Fri, 31 Jul 2009 18:53:22 +0200, Cyrille Rosset wrote: > Hi all, > > I am using cython and numpy and looked for a way to get cython modules > using numpy compiled with pyximport. > I found on the list the following message > (http://codespeak.net/pipermail/cython-dev/2009-May/005289.html) but > didn't find any modification in the cython repository in this direction. > > Following Lisandro Dalc?n's suggestion, I made a patch that adds a > ext_opts option to pyximport.install() which cascades to > pyximport.get_distutils_extension() and is used as a dictionnary with > options for distutils.Extension(). > Only the dictionnary type is tested (I assume Extension will tell if an > option is not good). > I also added a pyximport.unsinstall() function (so it's possible to > change options of the importer). Can't you simply use **kwargs instead of ext_opts? -- Giovanni Bajo Develer S.r.l. http://www.develer.com From jonovik at gmail.com Tue Aug 4 20:26:19 2009 From: jonovik at gmail.com (Jon Olav Vik) Date: Tue, 4 Aug 2009 18:26:19 +0000 (UTC) Subject: [Cython] Problems compiling C code that #include's my public Cython function Message-ID: Background: I'd like to compute second derivatives of a Python-implemented function with the help of a Fortran library called NDL [1]. The function to be differentiated can be implemented in C (an "external f" statement in the main Fortran program will make available the functions in the "f.o" object file). Now, I'm hoping to generate the requisite C code from a thin Cython wrapper around my function, as in [2]. As a first step, I have declared a Cython function "cdef public", compiled it (command-line output of compilation follows at the end), and am trying to call it from C. However, the compilation fails at the line '#include "fun.h"' and I don't understand why. bash-3.2$ gcc usefun.c In file included from usefun.c:2: fun.h: In function 'DL_IMPORT': fun.h:11: error: expected declaration specifiers before 'fun' fun.h:15: error: expected declaration specifiers before 'PyMODINIT_FUNC' In file included from /usr/include/stdio.h:34, from usefun.c:3: /usr/lib/gcc/x86_64-redhat-linux/4.1.2/include/stddef.h:214: error: storage class specified for parameter 'size_t' In file included from /usr/include/stdio.h:36, from usefun.c:3: /usr/include/bits/types.h:34: error: storage class specified for parameter '__u_char' /usr/include/bits/types.h:35: error: storage class specified for parameter '__u_short' [...etc etc] /usr/include/stdio.h:821: error: expected ')' before '*' token usefun.c:4: error: expected '=', ',', ';', 'asm' or '__attribute__' before '{' token usefun.c:7: error: old-style parameter declarations in prototyped function definition fun.h:11: error: parameter name omitted usefun.c:7: error: expected '{' at end of input Any hints? I am a novice in the worlds of C and Fortran alike, so no explanation is too basic 8-) A couple of related questions/remarks: * Assuming I get this to work, can the Fortran library call back to my Cython function without having to re-initialize the Cython module for every callback? * I see that there's an ongoing Fortran-to-Cython project that looks very relevant [3]. If that could make the NDL Fortran library callable from Cython, I'd be happy to provide a test case. (However, I only started coding Fortran today, and the examples I've seen are in F77. Thus, the array handling issues are a bit above my head. One-dimensional arrays of double will suffice for me.) * The Fortran example (not shown, as I'm not quite there yet) links to an *.o file produced by "gcc -c". I'm guessing that the *.o file that Cython puts in the build/temp.linux-x86_64-2.5/ directory could be used in the same way? Thank you in advance for any help. Jon Olav Vik == fun.pyx == """ Cython function that can be called from C, and hopefully Fortran. Compile with "python cysetup.py fun", see below. """ from math import * print "This is a Cython module" # time-consuming initialization goes here cdef public double fun(double *x, int n): return x[0]*cos(x[1])+x[1]*cos(x[0]) == usefun.c == /* compile with "gcc usefun.c" */ #include "fun.h" // note to self: "" not <> #include int main() { double x[2] = {1.0, 1.1}; // initialize array printf("fun(x) = %4.2f", fun(&x, 2)); // call include'd function } == cysetup.py == """ Compile Cython file with distutils.setup() This is just a hack to save me from having to write separate setup.py files """ from distutils.core import setup from distutils.extension import Extension from Cython.Distutils import build_ext import numpy as np from sys import argv import os modulename = argv.pop(1) for filename in modulename + ".c", modulename + ".so", "build": try: os.remove(filename) except OSError: pass argv += ["build_ext", "--inplace"] setup( cmdclass = {'build_ext': build_ext}, ext_modules = [ Extension(modulename, [modulename + ".pyx"], include_dirs=['/usit/titan/u1/jonvi/usr/include', np.get_include ()], library_dirs=['/usit/titan/u1/jonvi/usr/lib'], # include_dirs=['/site/VERSIONS/sundials-2.3.0/include', np.get_include ()], # library_dirs=['/site/VERSIONS/sundials-2.3.0/lib'], libraries=['sundials_cvode', 'sundials_nvecserial'], # extra_compile_args=["-g"], extra_link_args=["-g"], ), ] ) __import__(modulename) == Compiling the Cython module == bash-3.2$ python cysetup.py fun running build_ext cythoning fun.pyx to fun.c building 'fun' extension gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict- prototypes -fPIC -I/usit/titan/u1/jonvi/usr/include -I/site/VERSIONS/ compython-2.5/Linux/lib/python2.5/site-packages/numpy/core/include -I/site/ VERSIONS/compython-2.5/Linux/include/python2.5 -c fun.c -o build/temp.linux- x86_64-2.5/fun.o gcc -pthread -shared build/temp.linux-x86_64-2.5/fun.o -L/usit/titan/u1/jonvi/ usr/lib -L/site/VERSIONS/compython-2.5/Linux/lib -lsundials_cvode - lsundials_nvecserial -lpython2.5 -o fun.so This is a Cython module == Testing that it works on its own == bash-3.2$ python -c "import fun" This is a Cython module [1] C. Voglis et al., A numerical differentiation library exploiting parallel architectures, Computer Physics Communications (2009), doi:10.1016/j.cpc.2009.02.004 http://www.cs.uoi.gr/~voglis/ [2] http://docs.cython.org/docs/external_C_code.html#using-cython-declarations- from-c [3] http://wiki.cython.org/enhancements/fortran and links therein From muogoro at gmail.com Wed Aug 5 11:18:14 2009 From: muogoro at gmail.com (Daniele Pianu) Date: Wed, 5 Aug 2009 11:18:14 +0200 Subject: [Cython] Reference count on object assignment Message-ID: <66a8c45d0908050218o493871a9p36ddfcf8c8cd9afe@mail.gmail.com> Hi all, I've a simple question about the reference count automatically performed by Cython/Pyrex. I've a cdef function in an ipotetic extension type: ...... cdef TakeObject( self, void* obj ): self.obj = obj Py_INCREF( self.obj ) ....... The function receives a void pointer that I know it's always a pointer to a Python object. The extension type must mantain a reference to this object in its obj attribute, so, I perform the assignment "self.obj = obj". The reference should be valid after the object pointed by the obj method argument is deref-counted. For this reason, I increment manually the reference count of the same object (now pointed by the obj instance attribute too). The question is: in an assignment where a cast is performed, is the reference count automatically handled? Or do I need to manually increment-decrement the reference count as in the example above? And, if self.obj previously pointed to another object whose reference counted was incremented, do I have to decrement the reference count to avoid a possible memory leak? I've checked a bit the code and it seems the, first than the assignment is done, the self.obj ref count is decremented, but I've not understood if it's incremented too after the new assignment. Thanks for any advice, Daniele From dagss at student.matnat.uio.no Wed Aug 5 12:46:00 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 05 Aug 2009 12:46:00 +0200 Subject: [Cython] Reference count on object assignment Message-ID: <3332321215.689292@smtp.netcom.no> I'm not sure of this, but I know how I'd find out: Compile the pyx with the -a switch, open the generated html, and click on the line in question. Dag Sverre Seljebotn -----Original Message----- From: Daniele Pianu Date: Wednesday, Aug 5, 2009 11:26 am Subject: [Cython] Reference count on object assignment To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net Hi all, >I've a simple question about the reference count automatically >performed by Cython/Pyrex. I've a cdef function in an ipotetic >extension type: > >...... > >cdef TakeObject( self, void* obj ): > self.obj = obj > Py_INCREF( self.obj ) > >....... > >The function receives a void pointer that I know it's always a pointer to a Python object. The extension type must mantain a reference to >this object in its obj attribute, so, I perform >the assignment "self.obj = obj". The reference should be valid after the object pointed by the obj method argument is deref-counted. For this reason, I increment manually the reference count of the same object (now pointed by the obj instance attribute too). The question >is: in an assignment where a cast is performed, is the >reference count automatically handled? Or do I need to manually >increment-decrement the reference count as in the example above? And, if self.obj previously pointed to another object whose reference >counted was incremented, do I have to decrement the reference count to avoid a possible memory leak? I've checked a bit the code and it seems the, first than the assignment is done, the self.obj ref count is >decremented, but I've not understood if it's incremented too after the new assignment. > >Thanks for any advice, >Daniele >_______________________________________________ >Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > From jonovik at gmail.com Wed Aug 5 15:47:31 2009 From: jonovik at gmail.com (Jon Olav Vik) Date: Wed, 5 Aug 2009 13:47:31 +0000 (UTC) Subject: [Cython] =?utf-8?q?Problems_compiling_C_code_that_=23include=27s_?= =?utf-8?q?my_public_Cython=09function?= References: Message-ID: Jon Olav Vik writes: > Background: I'd like to compute second derivatives of a Python-implemented > function with the help of a Fortran library called NDL [1]. The function to be > differentiated can be implemented in C (an "external f" statement in the main > Fortran program will make available the functions in the "f.o" object file). > Now, I'm hoping to generate the requisite C code from a thin Cython wrapper > around my function, as in [2]. > > As a first step, I have declared a Cython function "cdef public", compiled it > (command-line output of compilation follows at the end), and am trying to call > it from C. However, the compilation fails at the line '#include "fun.h"' and I > don't understand why. [...] Maybe a step in a relevant direction: While Googling for examples of how to call Cython code from pure C, I stumbled across this posting: http://article.gmane.org/gmane.comp.python.cython.devel/5515 It describes how to "embed the interpreter and statically link your module", and allows a C program to run the main() function of a Cython module called mylib.pyx. In deciding whether to spend more time on this, it would be nice to know: 1) Is this how I need to proceed to call a Cython function from a C program? 2) Can a function implemented in Cython be used as a callback for a C or Fortran optimization or differentiation routine? 3) If 2), can I have once-only initialization in the Cython module that's not repeated with every function evaluation? Best regards, Jon Olav PS. I tried compiling the example in the other post (the C program is quoted below), and didn't quite get there. Details follow in case anyone is interested. After adjusting the -I (include dir) parameter to my system, the first few lines of compilation went okay: cython mylib.pyx gcc -fPIC -g -O2 -I /usr/include/python2.5 -c -o main.o main.c gcc -fPIC -g -O2 -I /usr/include/python2.5 -c -o mylib.o mylib.c However, the final step failed: gcc -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions \ main.o mylib.o /usr/lib/python2.5/config/libpython2.5.a \ -lm -ldl -pthread -lutil \ -o main I changed the path to libpython2.5.a and added the -I switch, but got /usr/bin/ld: unrecognized option '-Bsymbolic-functions' /usr/bin/ld: use the --help option for usage information collect2: ld returned 1 exit status Removing the -Bsymbolic-functions or replacing it with -Bsymbolic gave the error: /xanadu/site/common/VERSIONS/compython-2.5/Linux/lib/python2.5/config/ libpython2.5.a(posixmodule.o): In function `posix_tmpnam': /site/VERSIONS/compython-2.5/src/python/Python-2.5.2/./Modules/ posixmodule.c:6862: warning: the use of `tmpnam_r' is dangerous, better use `mkstemp' /xanadu/site/common/VERSIONS/compython-2.5/Linux/lib/python2.5/config/ libpython2.5.a(posixmodule.o): In function `posix_tempnam': /site/VERSIONS/compython-2.5/src/python/Python-2.5.2/./Modules/ posixmodule.c:6817: warning: the use of `tempnam' is dangerous, better use `mkstemp' /xanadu/site/common/VERSIONS/compython-2.5/Linux/lib/python2.5/config/ libpython2.5.a(zlibmodule.o): In function `PyZlib_compress': /site/VERSIONS/compython-2.5/src/python/Python-2.5.2/./Modules/ zlibmodule.c:146: undefined reference to `deflateInit_' etc etc ----------------------------------- main.c --------------------------- #include // For each Cython module you want to embed, you must declare an // init function, like so: PyMODINIT_FUNC initmylib(void); int main(int argc, char *argv[]) { // The first step is to set up the Python interpreter: Py_Initialize(); PySys_SetArgv(argc, argv); // Next, we need to tell Python that our module exists. Call each // of the functions you declared above. initmylib(); // Now do some Python stuff. The easiest thing to do is to give // the interpreter a string of Python code that imports your // module and calls it. PyRun_SimpleString("from mylib import main\n" "main()\n"); // When we're done, tell Python to clean up. Py_Finalize(); return 0; } From dalcinl at gmail.com Wed Aug 5 16:13:00 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 5 Aug 2009 11:13:00 -0300 Subject: [Cython] Reference count on object assignment In-Reply-To: <3332321215.689292@smtp.netcom.no> References: <3332321215.689292@smtp.netcom.no> Message-ID: Yes, in that case Cython manages the reference count the right way. You do NOT NEED to INCREF/DECREF. In fact, your code is introducing a reference leak. On Wed, Aug 5, 2009 at 7:46 AM, Dag Sverre Seljebotn wrote: > I'm not sure of this, but I know how I'd find out: Compile the pyx with the -a switch, open the generated html, and click on the line in question. > > Dag Sverre Seljebotn > -----Original Message----- > From: Daniele Pianu > Date: Wednesday, Aug 5, 2009 11:26 am > Subject: [Cython] Reference count on object assignment > To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net > > Hi all, >>I've a simple question about the reference count automatically >>performed by Cython/Pyrex. I've a cdef function in an ipotetic >>extension type: >> >>...... >> >>cdef TakeObject( self, void* obj ): >> ?self.obj = obj >> ?Py_INCREF( self.obj ) >> >>....... >> >>The function receives a void pointer that I know it's always a pointer to a Python object. The extension type must mantain a reference to >>this object in its obj attribute, so, I perform >>the assignment "self.obj = obj". The reference should be valid after the object pointed by the obj method argument is deref-counted. For this reason, I increment manually the reference count of the same object (now pointed by the obj instance attribute too). The question >>is: in an assignment where a cast is performed, is the >>reference count automatically handled? Or do I need to manually >>increment-decrement the reference count as in the example above? And, if self.obj previously pointed to another object whose reference >>counted was incremented, do I have to decrement the reference count to avoid a possible memory leak? I've checked a bit the code and it seems the, first than the assignment is done, the self.obj ref count is >>decremented, but I've not understood if it's incremented too after the new assignment. >> >>Thanks for any advice, >>Daniele >>_______________________________________________ >>Cython-dev mailing list >>Cython-dev at codespeak.net >>http://codespeak.net/mailman/listinfo/cython-dev >> > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Wed Aug 5 16:19:47 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 5 Aug 2009 11:19:47 -0300 Subject: [Cython] Problems compiling C code that #include's my public Cython function In-Reply-To: References: Message-ID: Could you tell me what language are you going to use for your "main" application? I mean, are you coding in Python, and then want to reuse the Fortran library? Or perhaps you are coding in C/Fortran, but want to be able to write your functions in Python? Depending on that, the approach is quite different (at least from my POV). On Wed, Aug 5, 2009 at 10:47 AM, Jon Olav Vik wrote: > Jon Olav Vik writes: >> Background: I'd like to compute second derivatives of a Python-implemented >> function with the help of a Fortran library called NDL [1]. The function to > be >> differentiated can be implemented in C (an "external f" statement in the main >> Fortran program will make available the functions in the "f.o" object file). >> Now, I'm hoping to generate the requisite C code from a thin Cython wrapper >> around my function, as in [2]. >> >> As a first step, I have declared a Cython function "cdef public", compiled it >> (command-line output of compilation follows at the end), and am trying to > call >> it from C. However, the compilation fails at the line '#include "fun.h"' and > I >> don't understand why. > [...] > > Maybe a step in a relevant direction: > > While Googling for examples of how to call Cython code from pure C, I stumbled > across this posting: > http://article.gmane.org/gmane.comp.python.cython.devel/5515 > It describes how to "embed the interpreter and statically link your module", > and allows a C program to run the main() function of a Cython module called > mylib.pyx. > > In deciding whether to spend more time on this, it would be nice to know: > 1) Is this how I need to proceed to call a Cython function from a C program? > 2) Can a function implemented in Cython be used as a callback for a C or > Fortran optimization or differentiation routine? > 3) If 2), can I have once-only initialization in the Cython module that's not > repeated with every function evaluation? > > Best regards, > Jon Olav > > PS. I tried compiling the example in the other post (the C program is quoted > below), and didn't quite get there. Details follow in case anyone is interested. > > After adjusting the -I (include dir) parameter to my system, the first few > lines of compilation went okay: > > > cython ?mylib.pyx > gcc -fPIC -g -O2 -I /usr/include/python2.5 ?-c -o main.o main.c > gcc -fPIC -g -O2 -I /usr/include/python2.5 ?-c -o mylib.o mylib.c > > > However, the final step failed: > > gcc -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions \ > ? ?main.o mylib.o /usr/lib/python2.5/config/libpython2.5.a \ > ? ?-lm -ldl -pthread -lutil \ > ? ?-o main > > I changed the path to libpython2.5.a and added the -I switch, but got > > /usr/bin/ld: unrecognized option '-Bsymbolic-functions' > /usr/bin/ld: use the --help option for usage information > collect2: ld returned 1 exit status > > Removing the -Bsymbolic-functions or replacing it with -Bsymbolic gave the > error: > > /xanadu/site/common/VERSIONS/compython-2.5/Linux/lib/python2.5/config/ > libpython2.5.a(posixmodule.o): In function `posix_tmpnam': > /site/VERSIONS/compython-2.5/src/python/Python-2.5.2/./Modules/ > posixmodule.c:6862: warning: the use of `tmpnam_r' is dangerous, better use > `mkstemp' > /xanadu/site/common/VERSIONS/compython-2.5/Linux/lib/python2.5/config/ > libpython2.5.a(posixmodule.o): In function `posix_tempnam': > /site/VERSIONS/compython-2.5/src/python/Python-2.5.2/./Modules/ > posixmodule.c:6817: warning: the use of `tempnam' is dangerous, better use > `mkstemp' > /xanadu/site/common/VERSIONS/compython-2.5/Linux/lib/python2.5/config/ > libpython2.5.a(zlibmodule.o): In function `PyZlib_compress': > /site/VERSIONS/compython-2.5/src/python/Python-2.5.2/./Modules/ > zlibmodule.c:146: undefined reference to `deflateInit_' > etc etc > > > ----------------------------------- main.c --------------------------- > #include > // For each Cython module you want to embed, you must declare an > // init function, like so: > PyMODINIT_FUNC initmylib(void); > int > main(int argc, char *argv[]) > { > ? ?// The first step is to set up the Python interpreter: > ? ?Py_Initialize(); > ? ?PySys_SetArgv(argc, argv); > ? ?// Next, we need to tell Python that our module exists. ?Call each > ? ?// of the functions you declared above. > ? ?initmylib(); > ? ?// Now do some Python stuff. ?The easiest thing to do is to give > ? ?// the interpreter a string of Python code that imports your > ? ?// module and calls it. > ? ?PyRun_SimpleString("from mylib import main\n" > ? ? ? ? ? ? ? ? ? ? ? "main()\n"); > ? ?// When we're done, tell Python to clean up. > ? ?Py_Finalize(); > ? ?return 0; > } > > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From jonovik at gmail.com Wed Aug 5 19:52:55 2009 From: jonovik at gmail.com (Jon Olav Vik) Date: Wed, 5 Aug 2009 17:52:55 +0000 (UTC) Subject: [Cython] =?utf-8?q?Problems_compiling_C_code_that_=23include=27s_?= =?utf-8?q?my_public=09Cython_function?= References: Message-ID: Lisandro Dalcin writes: > Could you tell me what language are you going to use for your "main" > application? I mean, are you coding in Python, and then want to reuse > the Fortran library? Or perhaps you are coding in C/Fortran, but want > to be able to write your functions in Python? Depending on that, the > approach is quite different (at least from my POV). * Python is my main language. * C is used for a rate function in a differential equation (provided to me from elsewhere). * Fortran has the numerical differentiation library, which I'd like to use for the following reasons: - Don't have to code it myself 8-) - Presumably robust, efficient and numerically accurate. - Citable publication. - Parallel implementations are available (though I'm unsure how that will play with a Cython callback) I'd be very grateful if you could recommend a strategy for this use case, or suggest other options. I include some more detail below in case you're interested. As for the mixed-language programming, I can see the following scenarios: == Ideal case: f2cy lets me call Fortran library from Cython == (f2cy is described at fortrancython.wordpress.com.) Then I could do, in principle: cimport f2cy import my_model # heavy initialization goes here ndl = f2cy("NDL") # make Fortran library callable from Cython cdef double phenotype(double* parameters, int number_of_parameters): """The function I want to differentiate, with an argument list matching the NDL specification.""" return my_model.phenotype(parameters) # expensive function evaluation # calling an NDL function which makes callbacks to my function: print ndl.hessian(phenotype, ...) # returns a 2-d array of double (In real life, the Fortran names are six characters long and devoid of vowels, of course.) == "Practicality beats elegance" case: do whatever it takes... == "cdef double phenotype..." as above, but calling phenotype() from Fortran (supposedly as easy as calling it from C). Then the main program will have to be in Fortran: ! This is how I make compiled C functions in my_model_cy.o ! available to Fortran. Incredibly simple: external my_model_cy ! Would need to figure out a way ! to make the answer available for further processing... print *, hessian(phenotype, ...) (Maybe I could even make a reusable, flexible Fortran function that could be called from Cython, but I wouldn't know how to pass function pointers to it...and if I could, I should be able to call the NDL library directly.) Thanks a lot for your help! Jon Olav == Details about my use case == My overall goal is to estimate parameter interaction in a detailed model of heart cell physiology. The parameters represent conductances of cellular ion channels and lots of other stuff. The model simulates the change over time in transmembrane voltage and concentrations of various ions in the cell. I will aggregate this to a single scalar-valued cellular phenotype, e.g. a measure of the duration of the voltage peak. This gives me a scalar function H of an n- dimensional parameter vector p. The derivative d(H)/d(p_i) is a measure of the "sensitivity" of the chosen phenotype to each model parameter p_j. However, this sensitivity may depend on other parameters p_j. The second derivatives d^2 H / [d(p_j) d(p_i)] is a measure of how a (small) change in parameter p_j modifies the effect of a (small) change in parameter p_j on the chosen phenotype. I will evaluate these derivatives at different points in parameter space and study various phenotypic measures, orchestrating it all in Python/Numpy: iterating over parameter scenarios, computing aggregate phenotypic measures, storing results, making graphs (Matplotlib) and exporting for statistical analysis (rpy2). Also, I'll want to concentrate on just a few parameters p_i and p_j; this could be neatly done with ad hoc wrappers like: p = ... # Full n-dimensional parameter vector i = ... # Indices to the parameters I'll be looking at def f(x, n): p0 = p # copy of original parameter values p0[i] = x # substitute the given parameters return phenotype(p0) # call with full parameter vector Having to code each of these in Fortran would probably make me cry. Because the function evaluations are rather expensive, I'd like to use an efficient, robust, and perhaps parallel library for the differentation. From kwmsmith at gmail.com Wed Aug 5 20:20:51 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Wed, 5 Aug 2009 13:20:51 -0500 Subject: [Cython] Problems compiling C code that #include's my public Cython function In-Reply-To: References: Message-ID: On Wed, Aug 5, 2009 at 12:52 PM, Jon Olav Vik wrote: > Lisandro Dalcin writes: > >> Could you tell me what language are you going to use for your "main" >> application? I mean, are you coding in Python, and then want to reuse >> the Fortran library? Or perhaps you are coding in C/Fortran, but want >> to be able to write your functions in Python? Depending on that, the >> approach is quite different (at least from my POV). > > * Python is my main language. > * C is used for a rate function in a differential equation (provided to me from > elsewhere). > * Fortran has the numerical differentiation library, which I'd like to use for > the following reasons: > - Don't have to code it myself 8-) > - Presumably robust, efficient and numerically accurate. > - Citable publication. > - Parallel implementations are available (though I'm unsure how that will play > with a Cython callback) > > I'd be very grateful if you could recommend a strategy for this use case, or > suggest other options. I include some more detail below in case you're > interested. As for the mixed-language programming, I can see the following > scenarios: > > == Ideal case: f2cy lets me call Fortran library from Cython == > (f2cy is described at fortrancython.wordpress.com.) > Then I could do, in principle: Hi, Jon. I'm the 'maintaner' of fwrap, which is the new name of 'f2cy' (I clearly need to put an update post on the blog!). It's still in development, and will have basic functionality ready to go in a few weeks. I don't know if we'll have C (i.e. cdef) callbacks implemented by then, but it will be high on the priority list. It will certainly help you wrap a Fortran library in Cython, as you desire, although the api is similar to f2py where the wrapping is automated, rather than the ctypes-like example you have below. If you can't wait that long then you'll have to find another way. I'm sorry that I can't devote the time to your particular mixed language programming questions, since I have to put all my time into fwrap. If I can carve out a chunk of time in the next few days I'll take a look. Kurt > > cimport f2cy > import my_model # heavy initialization goes here > ndl = f2cy("NDL") # make Fortran library callable from Cython > cdef double phenotype(double* parameters, int number_of_parameters): > ? """The function I want to differentiate, > ? with an argument list matching the NDL specification.""" > ? return my_model.phenotype(parameters) # expensive function evaluation > # calling an NDL function which makes callbacks to my function: > print ndl.hessian(phenotype, ...) # returns a 2-d array of double > > (In real life, the Fortran names are six characters long and devoid of vowels, > of course.) > > == "Practicality beats elegance" case: do whatever it takes... == > "cdef double phenotype..." as above, but calling phenotype() from Fortran > (supposedly as easy as calling it from C). Then the main program will have to > be in Fortran: > > ! This is how I make compiled C functions in my_model_cy.o > ! available to Fortran. Incredibly simple: > external my_model_cy > ! Would need to figure out a way > ! to make the answer available for further processing... > print *, hessian(phenotype, ...) > > (Maybe I could even make a reusable, flexible Fortran function that could be > called from Cython, but I wouldn't know how to pass function pointers to > it...and if I could, I should be able to call the NDL library directly.) > > > Thanks a lot for your help! > > Jon Olav > > > == Details about my use case == > > My overall goal is to estimate parameter interaction in a detailed model of > heart cell physiology. The parameters represent conductances of cellular ion > channels and lots of other stuff. The model simulates the change over time in > transmembrane voltage and concentrations of various ions in the cell. I will > aggregate this to a single scalar-valued cellular phenotype, e.g. a measure of > the duration of the voltage peak. This gives me a scalar function H of an n- > dimensional parameter vector p. > > The derivative d(H)/d(p_i) is a measure of the "sensitivity" of the chosen > phenotype to each model parameter p_j. However, this sensitivity may depend on > other parameters p_j. The second derivatives d^2 H / [d(p_j) d(p_i)] is a > measure of how a (small) change in parameter p_j modifies the effect of a > (small) change in parameter p_j on the chosen phenotype. > > I will evaluate these derivatives at different points in parameter space and > study various phenotypic measures, orchestrating it all in Python/Numpy: > iterating over parameter scenarios, computing aggregate phenotypic measures, > storing results, making graphs (Matplotlib) and exporting for statistical > analysis (rpy2). Also, I'll want to concentrate on just a few parameters p_i > and p_j; this could be neatly done with ad hoc wrappers like: > > p = ... # Full n-dimensional parameter vector > i = ... # Indices to the parameters I'll be looking at > def f(x, n): > ? p0 = p # copy of original parameter values > ? p0[i] = x # substitute the given parameters > ? return phenotype(p0) # call with full parameter vector > > Having to code each of these in Fortran would probably make me cry. > > > Because the function evaluations are rather expensive, I'd like to use an > efficient, robust, and perhaps parallel library for the differentation. > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From jonovik at gmail.com Wed Aug 5 21:12:03 2009 From: jonovik at gmail.com (Jon Olav Vik) Date: Wed, 5 Aug 2009 19:12:03 +0000 (UTC) Subject: [Cython] =?utf-8?q?Problems_compiling_C_code_that_=23include=27s_?= =?utf-8?q?my_public=09Cython_function?= References: Message-ID: Kurt Smith writes: > Hi, Jon. I'm the 'maintaner' of fwrap, which is the new name of > 'f2cy' (I clearly need to put an update post on the blog!). It's > still in development, and will have basic functionality ready to go in > a few weeks. I don't know if we'll have C (i.e. cdef) callbacks > implemented by then, but it will be high on the priority list. It > will certainly help you wrap a Fortran library in Cython, as you > desire, although the api is similar to f2py where the wrapping is > automated, Looking at http://www.scipy.org/F2py (it was down the last time I researched this), it certainly looks promising. What is it that fwrap offers that f2py doesn't? Direct access from C(ython), rather than the overhead of a Python extension module? Is there a separate mailing list for f2py? How would I pass a function pointer (callback) to the f2py-generated extension module? > rather than the ctypes-like example you have below. I'm not really familiar with ctypes; I tried to imitate the Cython coding patterns I have acquired in the last few months 8-) Could you tell me what is ctypes-like about it? > If you > can't wait that long then you'll have to find another way. I'm sorry > that I can't devote the time to your particular mixed language > programming questions, since I have to put all my time into fwrap. No problem. I was just thinking that maybe the development version was available for download somewhere. If it has docstrings and some examples/ unittests/doctests, I could have a look. Thank you for your help! Regards, Jon Olav From robertwb at math.washington.edu Fri Aug 7 06:04:12 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 6 Aug 2009 21:04:12 -0700 Subject: [Cython] Reference count on object assignment In-Reply-To: References: <3332321215.689292@smtp.netcom.no> Message-ID: Yep, an INCREF is always generated on assignment to an object variable, the fact that there's a cast here is irrelevant. - Robert On Aug 5, 2009, at 7:13 AM, Lisandro Dalcin wrote: > Yes, in that case Cython manages the reference count the right way. > You do NOT NEED to INCREF/DECREF. In fact, your code is introducing a > reference leak. > > On Wed, Aug 5, 2009 at 7:46 AM, Dag Sverre > Seljebotn wrote: >> I'm not sure of this, but I know how I'd find out: Compile the pyx >> with the -a switch, open the generated html, and click on the line >> in question. >> >> Dag Sverre Seljebotn >> -----Original Message----- >> From: Daniele Pianu >> Date: Wednesday, Aug 5, 2009 11:26 am >> Subject: [Cython] Reference count on object assignment >> To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net >> >> Hi all, >>> I've a simple question about the reference count automatically >>> performed by Cython/Pyrex. I've a cdef function in an ipotetic >>> extension type: >>> >>> ...... >>> >>> cdef TakeObject( self, void* obj ): >>> self.obj = obj >>> Py_INCREF( self.obj ) >>> >>> ....... >>> >>> The function receives a void pointer that I know it's always a >>> pointer to a Python object. The extension type must mantain a >>> reference to >>> this object in its obj attribute, so, I perform >>> the assignment "self.obj = obj". The reference should be >>> valid after the object pointed by the obj method argument is >>> deref-counted. For this reason, I increment manually the >>> reference count of the same object (now pointed by the obj >>> instance attribute too). The question >>> is: in an assignment where a cast is performed, is the >>> reference count automatically handled? Or do I need to manually >>> increment-decrement the reference count as in the example above? >>> And, if self.obj previously pointed to another object whose >>> reference >>> counted was incremented, do I have to decrement the reference >>> count to avoid a possible memory leak? I've checked a bit the >>> code and it seems the, first than the assignment is done, the >>> self.obj ref count is >>> decremented, but I've not understood if it's incremented too >>> after the new assignment. >>> >>> Thanks for any advice, >>> Daniele >>> _______________________________________________ >>> Cython-dev mailing list >>> Cython-dev at codespeak.net >>> http://codespeak.net/mailman/listinfo/cython-dev >>> >> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From greg.ewing at canterbury.ac.nz Fri Aug 7 11:18:43 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 07 Aug 2009 21:18:43 +1200 Subject: [Cython] Reference count on object assignment In-Reply-To: References: <3332321215.689292@smtp.netcom.no> Message-ID: <4A7BF173.6030506@canterbury.ac.nz> Robert Bradshaw wrote: > Yep, an INCREF is always generated on assignment to an object > variable Well, it's not quite as simple as that, because the result of an expression that returns a new reference isn't increfed on assignment. But a value being cast to object is considered a borrowed reference, so an incref is generated in that case. -- Greg From robertwb at math.washington.edu Fri Aug 7 18:12:43 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 7 Aug 2009 09:12:43 -0700 Subject: [Cython] Reference count on object assignment In-Reply-To: <4A7BF173.6030506@canterbury.ac.nz> References: <3332321215.689292@smtp.netcom.no> <4A7BF173.6030506@canterbury.ac.nz> Message-ID: On Aug 7, 2009, at 2:18 AM, Greg Ewing wrote: > Robert Bradshaw wrote: >> Yep, an INCREF is always generated on assignment to an object >> variable > > Well, it's not quite as simple as that, because the > result of an expression that returns a new reference > isn't increfed on assignment. But a value being > cast to object is considered a borrowed reference, > so an incref is generated in that case. True. The point I was trying to make is the cast doesn't cause the incref, the assignment does. - Robert From Chris.Barker at noaa.gov Sat Aug 8 02:09:02 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 07 Aug 2009 17:09:02 -0700 Subject: [Cython] Checking for numpy arrays and None Message-ID: <4A7CC21E.10209@noaa.gov> Hi all, I just accidentally passed None in to a cython function that is expecting a numpy array, and got a hard crash. I boiled it down to a trivial example: import cython import numpy as np cimport numpy as np def none_test(np.ndarray[np.uint8_t, ndim=1, mode="c"] a): return a when I test it, it does raise exceptions if I pass in a string, or a numpy array of the wrong shape or dtype, but not when I pass in None. I think I saw something in the docs about None no being well behaved, but it seems that a check for it in this sort of case would be a good idea -- None turns up in the darndest places in Python... -Chris Attached some test code... -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- A non-text attachment was scrubbed... Name: test_cython_none.py Type: application/x-python Size: 751 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090807/f227d2db/attachment.bin -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: cython_nonetest.pyx Url: http://codespeak.net/pipermail/cython-dev/attachments/20090807/f227d2db/attachment.diff From greg.ewing at canterbury.ac.nz Sat Aug 8 03:25:22 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 08 Aug 2009 13:25:22 +1200 Subject: [Cython] Checking for numpy arrays and None In-Reply-To: <4A7CC21E.10209@noaa.gov> References: <4A7CC21E.10209@noaa.gov> Message-ID: <4A7CD402.1090003@canterbury.ac.nz> Christopher Barker wrote: > I just accidentally passed None in to a cython function that is > expecting a numpy array, and got a hard crash. > > def none_test(np.ndarray[np.uint8_t, ndim=1, mode="c"] a): > return a Variables declared as referring to extension types also accept the value None, because you need some way of representing a null value. The type checking code generated for function arguments also accepts None by default, but you can override this by declaring the argument as "not None", e.g. def none_test(np.ndarray[np.uint8_t, ndim=1, mode="c"] a not None): You will then get an exception if you try to pass None to this function. (In Pyrex I'm thinking about changing this so that the default is *not* to accept None for an argument value, because this seems to trip up a lot of people.) -- Greg From chrisde88 at yahoo.de Sat Aug 8 04:46:17 2009 From: chrisde88 at yahoo.de (Christian) Date: Sat, 08 Aug 2009 04:46:17 +0200 Subject: [Cython] malloc(), free() via include "stdlib.pxi" Message-ID: <4A7CE6F9.8060807@yahoo.de> Hello, I try to implement a dynamic array in cython with malloc and free. My current problem is, that these functions are not builtins. After a little research in the documentation I have found the following wiki article: http://wiki.cython.org/DynamicMemoryAllocation My problem seems to be solved by including "stdlib.pxi" But when I add this line at the top of my pyx file and try to compile it. I get the following error: 'stdlib.pxi' not found I tried to add the lib to my setup.py as an additional source file, but nothing changes. I get the same error. How do I include the stdlib file correctly? Thank you in advance Christian From bpederse at gmail.com Sat Aug 8 05:14:04 2009 From: bpederse at gmail.com (Brent Pedersen) Date: Fri, 7 Aug 2009 20:14:04 -0700 Subject: [Cython] malloc(), free() via include "stdlib.pxi" In-Reply-To: <4A7CE6F9.8060807@yahoo.de> References: <4A7CE6F9.8060807@yahoo.de> Message-ID: On Fri, Aug 7, 2009 at 7:46 PM, Christian wrote: > Hello, > > I try to implement a dynamic array in cython with malloc and free. > My current problem is, that these functions are not builtins. After a > little research in the documentation I have found the following wiki > article: > > http://wiki.cython.org/DynamicMemoryAllocation > > My problem seems to be solved by including "stdlib.pxi" > But when I add this line at the top of my pyx file and try to compile > it. I get the following error: > > 'stdlib.pxi' not found > > I tried to add the lib to my setup.py as an additional source file, but > nothing changes. I get the same error. > > How do I include the stdlib file correctly? > > > > Thank you in advance > > Christian > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > hi, stdlib is included with cython. so i think you can just do something like: cimport stdlib cdef int* a = stdlib.malloc(N * stdlib.sizeof(int)) a[2] = 1 stdlib.free(a) From dagss at student.matnat.uio.no Sat Aug 8 10:49:00 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 08 Aug 2009 10:49:00 +0200 Subject: [Cython] malloc(), free() via include "stdlib.pxi" Message-ID: <3332573366.564242@smtp.netcom.no> Note that sizeof is a builtin keyword though (so no stdlib there). Dag Sverre Seljebotn -----Original Message----- From: Brent Pedersen Date: Saturday, Aug 8, 2009 5:22 am Subject: Re: [Cython] malloc(), free() via include "stdlib.pxi" To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net On Fri, Aug 7, 2009 at 7:46 PM, Christian wrote: > Hello, > >> I try to implement a dynamic array in cython with malloc and free. > My current problem is, that these functions are not builtins. After a > little research in the documentation I have found the following wiki > article: > >> http://wiki.cython.org/DynamicMemoryAllocation > >> My problem seems to be solved by including "stdlib.pxi" > But when I add this line at the top of my pyx file and try to compile > it. I get the following error: > >> 'stdlib.pxi' not found > >> I tried to add the lib to my setup.py as an additional source file, but > nothing changes. I get the same error. > >> How do I include the stdlib file correctly? > >> > >> Thank you in advance > >> Christian > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > >hi, stdlib is included with cython. so i think you can just do something like: > >cimport stdlib >cdef int* a = stdlib.malloc(N * stdlib.sizeof(int)) >a[2] = 1 >stdlib.free(a) >_______________________________________________ >Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > From dagss at student.matnat.uio.no Sat Aug 8 11:13:00 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 08 Aug 2009 11:13:00 +0200 Subject: [Cython] Checking for numpy arrays and None Message-ID: <3332574805.609777@smtp.netcom.no> Note also that in Cython there is a "nonecheck" compiler directive. If you write #cython: nonecheck=True at the top of your file, an exception will be raised instead. The will slow down all accesses to cdef classes somewhat though. wiki.cython.org/enhancements/compilerdirectives Dag Sverre Seljebotn -----Original Message----- From: Greg Ewing Date: Saturday, Aug 8, 2009 3:25 am Subject: Re: [Cython] Checking for numpy arrays and None To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net Christopher Barker wrote: > >> I just accidentally passed None in to a cython function that is > expecting a numpy array, and got a hard crash. > > def none_test(np.ndarray[np.uint8_t, ndim=1, mode="c"] a): > return a > >Variables declared as referring to extension types also >accept the value None, because you need some way of >representing a null value. > >The type checking code generated for function arguments >also accepts None by default, but you can override this >by declaring the argument as "not None", e.g. > > def none_test(np.ndarray[np.uint8_t, ndim=1, mode="c"] a not None): > >You will then get an exception if you try to pass None >to this function. > >(In Pyrex I'm thinking about changing this so that the >default is *not* to accept None for an argument value, >because this seems to trip up a lot of people.) > >-- >Greg >_______________________________________________ >Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > From Chris.Barker at noaa.gov Mon Aug 10 19:06:57 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 10 Aug 2009 10:06:57 -0700 Subject: [Cython] Checking for numpy arrays and None In-Reply-To: <4A7CD402.1090003@canterbury.ac.nz> References: <4A7CC21E.10209@noaa.gov> <4A7CD402.1090003@canterbury.ac.nz> Message-ID: <4A8053B1.8010003@noaa.gov> Greg Ewing wrote: > Variables declared as referring to extension types also > accept the value None, because you need some way of > representing a null value. I see - that makes sense. > you can override this > by declaring the argument as "not None", e.g. > > def none_test(np.ndarray[np.uint8_t, ndim=1, mode="c"] a not None): > > You will then get an exception if you try to pass None > to this function. That works, thanks. > (In Pyrex I'm thinking about changing this so that the > default is *not* to accept None for an argument value, > because this seems to trip up a lot of people.) I agree -- I think that makes more sense as the default -- if a function accepts None, there will almost always have to be special case case to handle it anyway. Dag Sverre Seljebotn wrote: > Note also that in Cython there is a "nonecheck" compiler directive. If you write > > #cython: nonecheck=True > > at the top of your file, an exception will be raised instead. Not for me: this seems to have no effect. Nor does using it as a decorator: @cython.nonecheck(True) Cython 0.11.2 unrelated note: The cython module doesn't seem to have a "__version__" attribute. Perhaps it should, this has become pretty much a standard, and it's the first place I tried to look to see what version I was running. Thanks, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: cython_nonetest.pyx Url: http://codespeak.net/pipermail/cython-dev/attachments/20090810/9e37e8e1/attachment.diff -------------- next part -------------- A non-text attachment was scrubbed... Name: test_cython_none.py Type: application/x-python Size: 870 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090810/9e37e8e1/attachment.bin From dagss at student.matnat.uio.no Mon Aug 10 19:19:21 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 10 Aug 2009 19:19:21 +0200 Subject: [Cython] Checking for numpy arrays and None In-Reply-To: <4A8053B1.8010003@noaa.gov> References: <4A7CC21E.10209@noaa.gov> <4A7CD402.1090003@canterbury.ac.nz> <4A8053B1.8010003@noaa.gov> Message-ID: <4A805699.5010402@student.matnat.uio.no> Christopher Barker wrote: > Greg Ewing wrote: >> Variables declared as referring to extension types also >> accept the value None, because you need some way of >> representing a null value. > > I see - that makes sense. > >> you can override this >> by declaring the argument as "not None", e.g. >> >> def none_test(np.ndarray[np.uint8_t, ndim=1, mode="c"] a not None): >> >> You will then get an exception if you try to pass None >> to this function. > > That works, thanks. > >> (In Pyrex I'm thinking about changing this so that the >> default is *not* to accept None for an argument value, >> because this seems to trip up a lot of people.) > > I agree -- I think that makes more sense as the default -- if a function > accepts None, there will almost always have to be special case case to > handle it anyway. > > Dag Sverre Seljebotn wrote: >> Note also that in Cython there is a "nonecheck" compiler directive. If >> you write >> >> #cython: nonecheck=True >> >> at the top of your file, an exception will be raised instead. > > Not for me: this seems to have no effect. Nor does using it as a decorator: > > @cython.nonecheck(True) > > Cython 0.11.2 Your example code is wrong; you need to actually try to use the object; THEN you will get an exception (where you would otherwise get a crash). Returning a None value is perfectly OK. From the docs I pointed you to (I'm sure it could be written clearer): nonecheck: If set to False, Cython is free to assume that native field accesses on variables typed as an extension type, or buffer accesses on a buffer variable, never occurs when the variable is set to None. Otherwise a check is inserted and the appropriate exception is raised. This is off by default for performance reasons. -- Dag Sverre From Chris.Barker at noaa.gov Mon Aug 10 19:42:45 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 10 Aug 2009 10:42:45 -0700 Subject: [Cython] Checking for numpy arrays and None In-Reply-To: <4A805699.5010402@student.matnat.uio.no> References: <4A7CC21E.10209@noaa.gov> <4A7CD402.1090003@canterbury.ac.nz> <4A8053B1.8010003@noaa.gov> <4A805699.5010402@student.matnat.uio.no> Message-ID: <4A805C15.8060206@noaa.gov> Dag Sverre Seljebotn wrote: > Your example code is wrong; you need to actually try to use the object; > THEN you will get an exception (where you would otherwise get a crash). > Returning a None value is perfectly OK. hmmm -- I misunderstood when the check was happening. However, I actually first tested this on real code, and code and got a hard crash when I passed in None. I'll try that test again, and post the actually example if it still fails for me. I also see why this is a performance issue -- the check needs to be done at every access, not just when the function starts. Which makes me think that Greg's suggestion is really what I want. I wonder why anyone would want to check on access -- this is different than bounds checking -- either the code can handle a None, or it can't. It's a type check, rather than a value check. That's what I get for boiling down the problem to too simple and example! > From the docs I pointed you to (I'm sure it could be written clearer): maybe, though I"m not sure how -- my problem is probably that I'm trying to learn from example, but haven't read enough of the docs to grok how cython works -- so I didn't understand the docs. thanks, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From dagss at student.matnat.uio.no Mon Aug 10 20:29:21 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 10 Aug 2009 20:29:21 +0200 Subject: [Cython] Checking for numpy arrays and None In-Reply-To: <4A805C15.8060206@noaa.gov> References: <4A7CC21E.10209@noaa.gov> <4A7CD402.1090003@canterbury.ac.nz> <4A8053B1.8010003@noaa.gov> <4A805699.5010402@student.matnat.uio.no> <4A805C15.8060206@noaa.gov> Message-ID: <4A806701.5070503@student.matnat.uio.no> Christopher Barker wrote: > Dag Sverre Seljebotn wrote: >> Your example code is wrong; you need to actually try to use the object; >> THEN you will get an exception (where you would otherwise get a crash). >> Returning a None value is perfectly OK. > > hmmm -- I misunderstood when the check was happening. However, I > actually first tested this on real code, and code and got a hard crash > when I passed in None. I'll try that test again, and post the actually > example if it still fails for me. > > I also see why this is a performance issue -- the check needs to be done > at every access, not just when the function starts. Which makes me think > that Greg's suggestion is really what I want. > > I wonder why anyone would want to check on access -- this is different > than bounds checking -- either the code can handle a None, or it can't. > It's a type check, rather than a value check. It's for use during debugging -- also this is the behaviour Python gives. -- Dag Sverre From Chris.Barker at noaa.gov Mon Aug 10 21:15:16 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Mon, 10 Aug 2009 12:15:16 -0700 Subject: [Cython] Checking for numpy arrays and None In-Reply-To: <4A806701.5070503@student.matnat.uio.no> References: <4A7CC21E.10209@noaa.gov> <4A7CD402.1090003@canterbury.ac.nz> <4A8053B1.8010003@noaa.gov> <4A805699.5010402@student.matnat.uio.no> <4A805C15.8060206@noaa.gov> <4A806701.5070503@student.matnat.uio.no> Message-ID: <4A8071C4.3060608@noaa.gov> Dag Sverre Seljebotn wrote: > It's for use during debugging -- also this is the behaviour Python gives. Got it. However, this doesn't seem to work for me: #cython: nonecheck=True I get a bus error crash. If I use the decorator instead, it does work. real, useful code enclosed. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: cython_resize.pyx Url: http://codespeak.net/pipermail/cython-dev/attachments/20090810/5b3044fa/attachment.diff -------------- next part -------------- A non-text attachment was scrubbed... Name: test_cython_none2.py Type: application/x-python Size: 373 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090810/5b3044fa/attachment.bin From dagss at student.matnat.uio.no Mon Aug 10 21:22:07 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 10 Aug 2009 21:22:07 +0200 Subject: [Cython] Checking for numpy arrays and None In-Reply-To: <4A8071C4.3060608@noaa.gov> References: <4A7CC21E.10209@noaa.gov> <4A7CD402.1090003@canterbury.ac.nz> <4A8053B1.8010003@noaa.gov> <4A805699.5010402@student.matnat.uio.no> <4A805C15.8060206@noaa.gov> <4A806701.5070503@student.matnat.uio.no> <4A8071C4.3060608@noaa.gov> Message-ID: <4A80735F.50400@student.matnat.uio.no> Christopher Barker wrote: > Dag Sverre Seljebotn wrote: >> It's for use during debugging -- also this is the behaviour Python gives. > > Got it. > > However, this doesn't seem to work for me: > > #cython: nonecheck=True The comment should be at the very top of the file. Perhaps we should emit an error on such comments in other places... Dag Sverre From greg.ewing at canterbury.ac.nz Tue Aug 11 02:45:22 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 11 Aug 2009 12:45:22 +1200 Subject: [Cython] Checking for numpy arrays and None In-Reply-To: <4A805C15.8060206@noaa.gov> References: <4A7CC21E.10209@noaa.gov> <4A7CD402.1090003@canterbury.ac.nz> <4A8053B1.8010003@noaa.gov> <4A805699.5010402@student.matnat.uio.no> <4A805C15.8060206@noaa.gov> Message-ID: <4A80BF22.8020305@canterbury.ac.nz> Christopher Barker wrote: > I wonder why anyone would want to check on access -- this is different > than bounds checking -- either the code can handle a None, or it can't. To catch bugs. You're right that correct code shouldn't need it (unless you catch the exception and do something else, but I wouldn't recommend that -- testing for None explicitly beforehand will be much more efficient). -- Greg From dalcinl at gmail.com Tue Aug 11 18:25:10 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 11 Aug 2009 13:25:10 -0300 Subject: [Cython] Checking for numpy arrays and None In-Reply-To: <4A80BF22.8020305@canterbury.ac.nz> References: <4A7CC21E.10209@noaa.gov> <4A7CD402.1090003@canterbury.ac.nz> <4A8053B1.8010003@noaa.gov> <4A805699.5010402@student.matnat.uio.no> <4A805C15.8060206@noaa.gov> <4A80BF22.8020305@canterbury.ac.nz> Message-ID: On Mon, Aug 10, 2009 at 9:45 PM, Greg Ewing wrote: > Christopher Barker wrote: > >> I wonder why anyone would want to check on access -- this is different >> than bounds checking -- either the code can handle a None, or it can't. > > To catch bugs. You're right that correct code shouldn't > need it (unless you catch the exception and do something > else, but I wouldn't recommend that -- testing for None > explicitly beforehand will be much more efficient). > I'm 100% on Greg's side... It's so easy to forget the "... not None" bit ... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Tue Aug 11 18:46:33 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 11 Aug 2009 18:46:33 +0200 Subject: [Cython] Checking for numpy arrays and None In-Reply-To: References: <4A7CC21E.10209@noaa.gov> <4A7CD402.1090003@canterbury.ac.nz> <4A8053B1.8010003@noaa.gov> <4A805699.5010402@student.matnat.uio.no> <4A805C15.8060206@noaa.gov> <4A80BF22.8020305@canterbury.ac.nz> Message-ID: <4A81A069.20205@student.matnat.uio.no> Lisandro Dalcin wrote: > On Mon, Aug 10, 2009 at 9:45 PM, Greg Ewing wrote: >> Christopher Barker wrote: >> >>> I wonder why anyone would want to check on access -- this is different >>> than bounds checking -- either the code can handle a None, or it can't. >> To catch bugs. You're right that correct code shouldn't >> need it (unless you catch the exception and do something >> else, but I wouldn't recommend that -- testing for None >> explicitly beforehand will be much more efficient). >> > > I'm 100% on Greg's side... It's so easy to forget the "... not None" bit ... This discussion (i.e. should "not None" be the default) was done to death earlier in here -- I strongly recommend creating a CEP for this from the earlier discussion before any more emails are spent on this. The sumup of the majority view was: - With decent control flow analysis nonecheck can be made the default, which is a better solution (as it is closer to Python) - Since there's a better AND backwards-compatible way available in the future, let's not break backwards compatability in the meantime (which changing the default to "not None" would do). -- Dag Sverre From robertwb at math.washington.edu Tue Aug 11 19:06:29 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 11 Aug 2009 10:06:29 -0700 (PDT) Subject: [Cython] Checking for numpy arrays and None In-Reply-To: <4A81A069.20205@student.matnat.uio.no> References: <4A7CC21E.10209@noaa.gov> <4A7CD402.1090003@canterbury.ac.nz> <4A8053B1.8010003@noaa.gov> <4A805699.5010402@student.matnat.uio.no> <4A805C15.8060206@noaa.gov> <4A80BF22.8020305@canterbury.ac.nz> <4A81A069.20205@student.matnat.uio.no> Message-ID: On Tue, 11 Aug 2009, Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: >> On Mon, Aug 10, 2009 at 9:45 PM, Greg Ewing wrote: >>> Christopher Barker wrote: >>> >>>> I wonder why anyone would want to check on access -- this is different >>>> than bounds checking -- either the code can handle a None, or it can't. >>> To catch bugs. You're right that correct code shouldn't >>> need it (unless you catch the exception and do something >>> else, but I wouldn't recommend that -- testing for None >>> explicitly beforehand will be much more efficient). >>> >> >> I'm 100% on Greg's side... It's so easy to forget the "... not None" bit ... > > This discussion (i.e. should "not None" be the default) was done to > death earlier in here -- I strongly recommend creating a CEP for this > from the earlier discussion before any more emails are spent on this. +1 > The sumup of the majority view was: > - With decent control flow analysis nonecheck can be made the default, > which is a better solution (as it is closer to Python) > - Since there's a better AND backwards-compatible way available in the > future, let's not break backwards compatability in the meantime (which > changing the default to "not None" would do). Two other key points were, for whoever wants to write up the CEP: - If we were to change the default to not None, there'd be no safe way of ever going back. - This handles function arguments. Extension type fields, local variables, and global variables still have the same issue that a coder needs to be aware of (for the moment). - Robert From greg.ewing at canterbury.ac.nz Wed Aug 12 02:13:52 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 12 Aug 2009 12:13:52 +1200 Subject: [Cython] Checking for numpy arrays and None In-Reply-To: References: <4A7CC21E.10209@noaa.gov> <4A7CD402.1090003@canterbury.ac.nz> <4A8053B1.8010003@noaa.gov> <4A805699.5010402@student.matnat.uio.no> <4A805C15.8060206@noaa.gov> <4A80BF22.8020305@canterbury.ac.nz> Message-ID: <4A820940.3080006@canterbury.ac.nz> Lisandro Dalcin wrote: > I'm 100% on Greg's side... It's so easy to forget the "... not None" bit ... That's good to hear. Last time I raised the idea on the Cython list it met with a rather unenthusiastic response. Looks like I should learn to trust my instincts more! -- Greg From greg.ewing at canterbury.ac.nz Wed Aug 12 02:22:49 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 12 Aug 2009 12:22:49 +1200 Subject: [Cython] Checking for numpy arrays and None In-Reply-To: <4A81A069.20205@student.matnat.uio.no> References: <4A7CC21E.10209@noaa.gov> <4A7CD402.1090003@canterbury.ac.nz> <4A8053B1.8010003@noaa.gov> <4A805699.5010402@student.matnat.uio.no> <4A805C15.8060206@noaa.gov> <4A80BF22.8020305@canterbury.ac.nz> <4A81A069.20205@student.matnat.uio.no> Message-ID: <4A820B59.2020503@canterbury.ac.nz> Dag Sverre Seljebotn wrote: > - With decent control flow analysis nonecheck can be made the default, > which is a better solution (as it is closer to Python) It looks like Pyrex and Cython may end up diverging on this point then, because I don't envisage adding control flow analysis to Pyrex any time soon. -- Greg From greg.ewing at canterbury.ac.nz Wed Aug 12 02:33:44 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 12 Aug 2009 12:33:44 +1200 Subject: [Cython] Checking for numpy arrays and None In-Reply-To: References: <4A7CC21E.10209@noaa.gov> <4A7CD402.1090003@canterbury.ac.nz> <4A8053B1.8010003@noaa.gov> <4A805699.5010402@student.matnat.uio.no> <4A805C15.8060206@noaa.gov> <4A80BF22.8020305@canterbury.ac.nz> <4A81A069.20205@student.matnat.uio.no> Message-ID: <4A820DE8.6000401@canterbury.ac.nz> Robert Bradshaw wrote: > - This handles function arguments. Extension type fields, local > variables, and global variables still have the same issue that a coder > needs to be aware of (for the moment). The Pyrex philosophy on this is that the None checks are for the purpose of protecting against abuse from Python code. Once inside Pyrex, it's the Pyrex programmer's responsibility to avoid dereferencing None values. Since extension-typed attributes can't be written to directly from Python, there's no equivalent of the "not None" mechanism for them. Instead you can use a property whose setter function declares its argument appropriately. -- Greg From robertwb at math.washington.edu Wed Aug 12 05:10:20 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 11 Aug 2009 20:10:20 -0700 Subject: [Cython] Checking for numpy arrays and None In-Reply-To: <4A820DE8.6000401@canterbury.ac.nz> References: <4A7CC21E.10209@noaa.gov> <4A7CD402.1090003@canterbury.ac.nz> <4A8053B1.8010003@noaa.gov> <4A805699.5010402@student.matnat.uio.no> <4A805C15.8060206@noaa.gov> <4A80BF22.8020305@canterbury.ac.nz> <4A81A069.20205@student.matnat.uio.no> <4A820DE8.6000401@canterbury.ac.nz> Message-ID: <81676F1C-2B2B-4DC6-905E-9793BE93C789@math.washington.edu> On Aug 11, 2009, at 5:33 PM, Greg Ewing wrote: > Robert Bradshaw wrote: > >> - This handles function arguments. Extension type fields, local >> variables, and global variables still have the same issue that a >> coder >> needs to be aware of (for the moment). > > The Pyrex philosophy on this is that the None checks are > for the purpose of protecting against abuse from Python > code. Once inside Pyrex, it's the Pyrex programmer's > responsibility to avoid dereferencing None values. Yep, this is one of the areas where we have different philosophies. The Cython philosophy is that one should be able to take a piece of Python code, compile it with Cython, maybe sprinkle in some static type declarations, and the behavior should be essentially the same (except it should run faster of course :). Programming Cython should be as easy as programming Python, and we'd like to minimize the overhead of "shifting gears" when going back and forth between the two. We're not there yet. - Robert From Chris.Barker at noaa.gov Wed Aug 12 18:04:53 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 12 Aug 2009 09:04:53 -0700 Subject: [Cython] Checking for numpy arrays and None In-Reply-To: <81676F1C-2B2B-4DC6-905E-9793BE93C789@math.washington.edu> References: <4A7CC21E.10209@noaa.gov> <4A7CD402.1090003@canterbury.ac.nz> <4A8053B1.8010003@noaa.gov> <4A805699.5010402@student.matnat.uio.no> <4A805C15.8060206@noaa.gov> <4A80BF22.8020305@canterbury.ac.nz> <4A81A069.20205@student.matnat.uio.no> <4A820DE8.6000401@canterbury.ac.nz> <81676F1C-2B2B-4DC6-905E-9793BE93C789@math.washington.edu> Message-ID: <4A82E825.4000800@noaa.gov> Robert Bradshaw wrote: > The Cython philosophy is that one should be able to take a piece of > Python code, compile it with Cython, maybe sprinkle in some static > type declarations, and the behavior should be essentially the same I, for one, like this approach, which is why I was surprised by the None issue. If I write a python function that expects a numpy array, and I pass a None into it, I'll get an exception. If I write a Cython function that expects a numpy array and declare that in the function signature, then pass a None it, I get a bus error (or who knows what other ugly crash). I do like the idea that one can pass a None in to signify "no value", or "False" or whatever, but there is very little chance that code will handle it properly if it wasn't written with that in mind. This is why it seems to me that the "not None" behavior should clearly be the default. I understand the need for backward compatibility, so I'll just try to remember to put "not None" in my declarations. NOTE: it would be a good idea to add that to examples and tutorials, too. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Wed Aug 12 18:31:41 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 12 Aug 2009 09:31:41 -0700 Subject: [Cython] Checking for numpy arrays and None In-Reply-To: <4A82E825.4000800@noaa.gov> References: <4A7CC21E.10209@noaa.gov> <4A7CD402.1090003@canterbury.ac.nz> <4A8053B1.8010003@noaa.gov> <4A805699.5010402@student.matnat.uio.no> <4A805C15.8060206@noaa.gov> <4A80BF22.8020305@canterbury.ac.nz> <4A81A069.20205@student.matnat.uio.no> <4A820DE8.6000401@canterbury.ac.nz> <81676F1C-2B2B-4DC6-905E-9793BE93C789@math.washington.edu> <4A82E825.4000800@noaa.gov> Message-ID: <4A82EE6D.8020904@noaa.gov> Christopher Barker wrote: > NOTE: it would be a good idea to add that to examples and tutorials, too. In the spirit of putting my time where my mouth is, I just updated the numpy tutorial with "non None" information: http://wiki.cython.org/tutorials/numpy please edit, or roll back if you don't like it. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From kwmsmith at gmail.com Wed Aug 12 21:20:58 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Wed, 12 Aug 2009 14:20:58 -0500 Subject: [Cython] Question about packaging external fortran parser with Cython for fwrap Message-ID: Hi, As my GSoC winds up, we need to decide whether we'll package the fortran parser, 'fparser' (located as a subpackage of the G3 F2PY project http://code.google.com/p/f2py/) with Cython, or if it will remain an external dependency. Dag Sverre and I had a short email thread with Pearu Peterson about this, and inclusion was left as a possibility. I want to check here and see if inclusion would be rejected straight away, before pursuing the matter further with Pearu. First, I believe there to be no licensing issues, although I'm not 100% certain here -- Cython is under the Apache license, right? And G3 F2PY is under the Numpy-license. If these are incompatible then that takes care of the issue. Otherwise -- In favor of including fparser: * Ease of use, one fewer barrier to entry. * Dag and I both have commit access to fparser; we (principally me) can fix bugs in the Cython branch of fparser. * fparser is a self-contained package, but its ultimate design is a part of another package (G3 F2PY) that is developing. Since users would have to deal with the G3 F2PY level for download, etc, and G3 F2PY is unfinished but for fparser, this would be confusing. It would be great to hide these details completely. Against including fparser: * Maintenance & support -- responsibility for upkeep falls more on Cython's shoulders (via yours truly). * Bloat. It includes a large package in Cython not central to Cython's core philosophy. I'm neutral in this. If the devs decide against inclusion, then I'd strongly consider implementing a smaller fortran parser (focused just on the bits we need for fwrap) that *would* be included. Thanks, Kurt From dagss at student.matnat.uio.no Wed Aug 12 22:00:38 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 12 Aug 2009 22:00:38 +0200 Subject: [Cython] Question about packaging external fortran parser with Cython for fwrap In-Reply-To: References: Message-ID: <4A831F66.3090901@student.matnat.uio.no> Kurt Smith wrote: > Hi, > > As my GSoC winds up, we need to decide whether we'll package the > fortran parser, 'fparser' (located as a subpackage of the G3 F2PY > project http://code.google.com/p/f2py/) with Cython, or if it will > remain an external dependency. > > Dag Sverre and I had a short email thread with Pearu Peterson about > this, and inclusion was left as a possibility. I want to check here > and see if inclusion would be rejected straight away, before pursuing > the matter further with Pearu. > > First, I believe there to be no licensing issues, although I'm not > 100% certain here -- Cython is under the Apache license, right? And > G3 F2PY is under the Numpy-license. If these are incompatible then > that takes care of the issue. > > Otherwise -- > > In favor of including fparser: > > * Ease of use, one fewer barrier to entry. > * Dag and I both have commit access to fparser; we (principally me) > can fix bugs in the Cython branch of fparser. > * fparser is a self-contained package, but its ultimate design is a > part of another package (G3 F2PY) that is developing. Since users > would have to deal with the G3 F2PY level for download, etc, and G3 > F2PY is unfinished but for fparser, this would be confusing. It would > be great to hide these details completely. > > Against including fparser: > > * Maintenance & support -- responsibility for upkeep falls more on > Cython's shoulders (via yours truly). > * Bloat. It includes a large package in Cython not central to > Cython's core philosophy. Question: How large? Say, in bytes (and compressed bytes, if you bother to check). (I suppose it would be "hidden away" in the Tools folder, so download size seems to be the major concern.) -- Dag Sverre From kwmsmith at gmail.com Wed Aug 12 22:27:42 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Wed, 12 Aug 2009 15:27:42 -0500 Subject: [Cython] Question about packaging external fortran parser with Cython for fwrap In-Reply-To: <4A831F66.3090901@student.matnat.uio.no> References: <4A831F66.3090901@student.matnat.uio.no> Message-ID: On Wed, Aug 12, 2009 at 3:00 PM, Dag Sverre Seljebotn wrote: > Kurt Smith wrote: >> Hi, >> >> As my GSoC winds up, we need to decide whether we'll package the >> fortran parser, 'fparser' (located as a subpackage of the G3 F2PY >> project http://code.google.com/p/f2py/) with Cython, or if it will >> remain an external dependency. >> >> Dag Sverre and I had a short email thread with Pearu Peterson about >> this, and inclusion was left as a possibility. ?I want to check here >> and see if inclusion would be rejected straight away, before pursuing >> the matter further with Pearu. >> >> First, I believe there to be no licensing issues, although I'm not >> 100% certain here -- Cython is under the Apache license, right? ?And >> G3 F2PY is under the Numpy-license. ?If these are incompatible then >> that takes care of the issue. >> >> Otherwise -- >> >> In favor of including fparser: >> >> ?* Ease of use, one fewer barrier to entry. >> ?* Dag and I both have commit access to fparser; we (principally me) >> can fix bugs in the Cython branch of fparser. >> ?* fparser is a self-contained package, but its ultimate design is a >> part of another package (G3 F2PY) that is developing. ?Since users >> would have to deal with the G3 F2PY level for download, etc, and G3 >> F2PY is unfinished but for fparser, this would be confusing. ?It would >> be great to hide these details completely. >> >> Against including fparser: >> >> ?* Maintenance & support -- responsibility for upkeep falls more on >> Cython's shoulders (via yours truly). >> ?* Bloat. ?It includes a large package in Cython not central to >> Cython's core philosophy. > > Question: How large? Say, in bytes (and compressed bytes, if you bother > to check). > > (I suppose it would be "hidden away" in the Tools folder, so download > size seems to be the major concern.) About 600K for the .py source files, so we're not breaking the bank here. From kwmsmith at gmail.com Wed Aug 12 22:31:05 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Wed, 12 Aug 2009 15:31:05 -0500 Subject: [Cython] Question about packaging external fortran parser with Cython for fwrap In-Reply-To: References: <4A831F66.3090901@student.matnat.uio.no> Message-ID: On Wed, Aug 12, 2009 at 3:27 PM, Kurt Smith wrote: > On Wed, Aug 12, 2009 at 3:00 PM, Dag Sverre > Seljebotn wrote: >> Kurt Smith wrote: >>> Hi, >>> >>> As my GSoC winds up, we need to decide whether we'll package the >>> fortran parser, 'fparser' (located as a subpackage of the G3 F2PY >>> project http://code.google.com/p/f2py/) with Cython, or if it will >>> remain an external dependency. >>> >>> Dag Sverre and I had a short email thread with Pearu Peterson about >>> this, and inclusion was left as a possibility. ?I want to check here >>> and see if inclusion would be rejected straight away, before pursuing >>> the matter further with Pearu. >>> >>> First, I believe there to be no licensing issues, although I'm not >>> 100% certain here -- Cython is under the Apache license, right? ?And >>> G3 F2PY is under the Numpy-license. ?If these are incompatible then >>> that takes care of the issue. >>> >>> Otherwise -- >>> >>> In favor of including fparser: >>> >>> ?* Ease of use, one fewer barrier to entry. >>> ?* Dag and I both have commit access to fparser; we (principally me) >>> can fix bugs in the Cython branch of fparser. >>> ?* fparser is a self-contained package, but its ultimate design is a >>> part of another package (G3 F2PY) that is developing. ?Since users >>> would have to deal with the G3 F2PY level for download, etc, and G3 >>> F2PY is unfinished but for fparser, this would be confusing. ?It would >>> be great to hide these details completely. >>> >>> Against including fparser: >>> >>> ?* Maintenance & support -- responsibility for upkeep falls more on >>> Cython's shoulders (via yours truly). >>> ?* Bloat. ?It includes a large package in Cython not central to >>> Cython's core philosophy. >> >> Question: How large? Say, in bytes (and compressed bytes, if you bother >> to check). >> >> (I suppose it would be "hidden away" in the Tools folder, so download >> size seems to be the major concern.) > > About 600K for the .py source files, so we're not breaking the bank here. ~150K compressed. From dagss at student.matnat.uio.no Thu Aug 13 09:44:13 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 13 Aug 2009 09:44:13 +0200 Subject: [Cython] Cython 0.11.2 win32 installer In-Reply-To: <96de71860908121538k98dd1e3ldaf6a567ad474b19@mail.gmail.com> References: <96de71860908121538k98dd1e3ldaf6a567ad474b19@mail.gmail.com> Message-ID: <4A83C44D.9040107@student.matnat.uio.no> Hi cython-dev, Fernando Perez wrote: > Hi Dag, > > Dave Cournapeau kindly made for us win32 installers of the newest > cython, since you guys don't have them on your site: > > https://cirl.berkeley.edu/fperez/tmp/Cython-0.11.2.win32-py2.5.exe > https://cirl.berkeley.edu/fperez/tmp/Cython-0.11.2.win32-py2.6.exe Great, thanks! Is there somebody on the list who could more easily than myself give these a test ride? (In addition to running the installer, a C compiler must be set up properly, see wiki.cython.org/Installation and the link to Windows installation there. An alternative to creating distutils.cfg (for mingw) is to pass "-c mingw" to setup.py.) I'll take this opportunity to come with a request: If somebody with easy access to Windows would like to be responsible for Windows packaging of Cython that would be great (as none of the core devs run Windows) -- it seems that David Cournapeau has already done the dirty work, so that it would likely just be about running his scripts for every Cython release and test that things work properly. -- Dag Sverre From Fernando.Perez at berkeley.edu Thu Aug 13 10:33:47 2009 From: Fernando.Perez at berkeley.edu (Fernando Perez) Date: Thu, 13 Aug 2009 01:33:47 -0700 Subject: [Cython] Cython 0.11.2 win32 installer In-Reply-To: <4A83C44D.9040107@student.matnat.uio.no> References: <96de71860908121538k98dd1e3ldaf6a567ad474b19@mail.gmail.com> <4A83C44D.9040107@student.matnat.uio.no> Message-ID: <96de71860908130133t2f97ed34he8ff5ba3eeabd517@mail.gmail.com> Hi Dag, On Thu, Aug 13, 2009 at 12:44 AM, Dag Sverre Seljebotn wrote: > Hi cython-dev, > > Fernando Perez wrote: >> >> Hi Dag, >> >> Dave Cournapeau kindly made for us win32 installers of the newest >> cython, since you guys don't have them on your site: >> >> https://cirl.berkeley.edu/fperez/tmp/Cython-0.11.2.win32-py2.5.exe >> https://cirl.berkeley.edu/fperez/tmp/Cython-0.11.2.win32-py2.6.exe > > Great, thanks! > > Is there somebody on the list who could more easily than myself give these a > test ride? > > (In addition to running the installer, a C compiler must be set up properly, > see > > wiki.cython.org/Installation > > and the link to Windows installation there. An alternative to creating > distutils.cfg (for mingw) is to pass "-c mingw" to setup.py.) I just tested this on top of a bare EPD install, and once I commented out the old cython egg path from the easy-install.pth file, it worked just fine. So tomorrow I'll publicize it to the attendees. > I'll take this opportunity to come with a request: If somebody with easy > access to Windows would like to be responsible for Windows packaging of > Cython that would be great (as none of the core devs run Windows) -- it > seems that David Cournapeau has already done the dirty work, so that it > would likely just be about running his scripts for every Cython release and > test that things work properly. By the way, David did this on a wine setup he has on his Mac laptop in just a couple of minutes! So it seems that once you have the right setup, it's not a huge deal. Dag, since you'll get to meet David next week at scipy, perhaps you can get this set up as well so any of you can pump these out easily at release time, without needing a windows box (it can apparently be done on either Macs or linux, though I have yet to do it myself). In any case, thanks again to David C for his help with this! Cheers, f From robertwb at math.washington.edu Thu Aug 13 10:34:44 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 13 Aug 2009 01:34:44 -0700 Subject: [Cython] Question about packaging external fortran parser with Cython for fwrap In-Reply-To: References: <4A831F66.3090901@student.matnat.uio.no> Message-ID: <19BF669A-FB71-4FB9-A4C9-DA6B23A2605C@math.washington.edu> On Aug 12, 2009, at 1:31 PM, Kurt Smith wrote: > On Wed, Aug 12, 2009 at 3:27 PM, Kurt Smith wrote: >> On Wed, Aug 12, 2009 at 3:00 PM, Dag Sverre >> Seljebotn wrote: >>> Kurt Smith wrote: >>>> Hi, >>>> >>>> As my GSoC winds up, we need to decide whether we'll package the >>>> fortran parser, 'fparser' (located as a subpackage of the G3 F2PY >>>> project http://code.google.com/p/f2py/) with Cython, or if it will >>>> remain an external dependency. >>>> >>>> Dag Sverre and I had a short email thread with Pearu Peterson about >>>> this, and inclusion was left as a possibility. I want to check >>>> here >>>> and see if inclusion would be rejected straight away, before >>>> pursuing >>>> the matter further with Pearu. >>>> >>>> First, I believe there to be no licensing issues, although I'm not >>>> 100% certain here -- Cython is under the Apache license, right? >>>> And >>>> G3 F2PY is under the Numpy-license. If these are incompatible then >>>> that takes care of the issue. >>>> >>>> Otherwise -- >>>> >>>> In favor of including fparser: >>>> >>>> * Ease of use, one fewer barrier to entry. >>>> * Dag and I both have commit access to fparser; we (principally >>>> me) >>>> can fix bugs in the Cython branch of fparser. >>>> * fparser is a self-contained package, but its ultimate design >>>> is a >>>> part of another package (G3 F2PY) that is developing. Since users >>>> would have to deal with the G3 F2PY level for download, etc, and G3 >>>> F2PY is unfinished but for fparser, this would be confusing. It >>>> would >>>> be great to hide these details completely. >>>> >>>> Against including fparser: >>>> >>>> * Maintenance & support -- responsibility for upkeep falls more on >>>> Cython's shoulders (via yours truly). >>>> * Bloat. It includes a large package in Cython not central to >>>> Cython's core philosophy. >>> >>> Question: How large? Say, in bytes (and compressed bytes, if you >>> bother >>> to check). >>> >>> (I suppose it would be "hidden away" in the Tools folder, so >>> download >>> size seems to be the major concern.) >> >> About 600K for the .py source files, so we're not breaking the >> bank here. > > ~150K compressed. That seems acceptable to me. Given that you've volunteered to maintain it, and numerics certainly is an important audience of Cython, it seems to make sense to bundle it. Any other opinions? - Robert From robertwb at math.washington.edu Thu Aug 13 10:42:12 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 13 Aug 2009 01:42:12 -0700 Subject: [Cython] Cython 0.11.2 win32 installer In-Reply-To: <96de71860908130133t2f97ed34he8ff5ba3eeabd517@mail.gmail.com> References: <96de71860908121538k98dd1e3ldaf6a567ad474b19@mail.gmail.com> <4A83C44D.9040107@student.matnat.uio.no> <96de71860908130133t2f97ed34he8ff5ba3eeabd517@mail.gmail.com> Message-ID: <7EBD264E-B8E0-4E90-B4CD-26507123D5F8@math.washington.edu> On Aug 13, 2009, at 1:33 AM, Fernando Perez wrote: > Hi Dag, > > On Thu, Aug 13, 2009 at 12:44 AM, Dag Sverre > Seljebotn wrote: >> Hi cython-dev, >> >> Fernando Perez wrote: >>> >>> Hi Dag, >>> >>> Dave Cournapeau kindly made for us win32 installers of the newest >>> cython, since you guys don't have them on your site: >>> >>> https://cirl.berkeley.edu/fperez/tmp/Cython-0.11.2.win32-py2.5.exe >>> https://cirl.berkeley.edu/fperez/tmp/Cython-0.11.2.win32-py2.6.exe >> >> Great, thanks! >> >> Is there somebody on the list who could more easily than myself >> give these a >> test ride? >> >> (In addition to running the installer, a C compiler must be set up >> properly, >> see >> >> wiki.cython.org/Installation >> >> and the link to Windows installation there. An alternative to >> creating >> distutils.cfg (for mingw) is to pass "-c mingw" to setup.py.) > > I just tested this on top of a bare EPD install, and once I commented > out the old cython egg path from the easy-install.pth file, it worked > just fine. So tomorrow I'll publicize it to the attendees. > >> I'll take this opportunity to come with a request: If somebody >> with easy >> access to Windows would like to be responsible for Windows >> packaging of >> Cython that would be great (as none of the core devs run Windows) >> -- it >> seems that David Cournapeau has already done the dirty work, so >> that it >> would likely just be about running his scripts for every Cython >> release and >> test that things work properly. > > By the way, David did this on a wine setup he has on his Mac laptop in > just a couple of minutes! So it seems that once you have the right > setup, it's not a huge deal. Dag, since you'll get to meet David next > week at scipy, perhaps you can get this set up as well so any of you > can pump these out easily at release time, without needing a windows > box (it can apparently be done on either Macs or linux, though I have > yet to do it myself). > > In any case, thanks again to David C for his help with this! Yes, that's very nice. (My preferred way to distribute cython is via Sage, but that doesn't work well for windows users yet.:-) - Robert From fperez.net at gmail.com Thu Aug 13 10:48:23 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 13 Aug 2009 01:48:23 -0700 Subject: [Cython] Cython 0.11.2 win32 installer In-Reply-To: <7EBD264E-B8E0-4E90-B4CD-26507123D5F8@math.washington.edu> References: <96de71860908121538k98dd1e3ldaf6a567ad474b19@mail.gmail.com> <4A83C44D.9040107@student.matnat.uio.no> <96de71860908130133t2f97ed34he8ff5ba3eeabd517@mail.gmail.com> <7EBD264E-B8E0-4E90-B4CD-26507123D5F8@math.washington.edu> Message-ID: On Thu, Aug 13, 2009 at 1:42 AM, Robert Bradshaw wrote: > Yes, that's very nice. (My preferred way to distribute cython is via > Sage, but that doesn't work well for windows users yet.:-) > There's also the fact that Cython can be of interest to many people who might want to use it as a standalone tool in their own projects on top of their normal Python toolchain, thus for whom Sage may be too large a dependency (even if it was native for windows). Providing self-contained native windows installers is always a big plus for a project in terms of gaining acceptance, even if it is already packaged as part of some other toolchain (say Sage, EPD or PythonXY for example, all of which ship Cython). In any case, keep up the good work! I look forward to Dag's presentation, and I hope you'll be able to make it next year to Scipy :) Cheers, f From robertwb at math.washington.edu Thu Aug 13 12:18:08 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 13 Aug 2009 03:18:08 -0700 Subject: [Cython] Cython 0.11.2 win32 installer In-Reply-To: References: <96de71860908121538k98dd1e3ldaf6a567ad474b19@mail.gmail.com> <4A83C44D.9040107@student.matnat.uio.no> <96de71860908130133t2f97ed34he8ff5ba3eeabd517@mail.gmail.com> <7EBD264E-B8E0-4E90-B4CD-26507123D5F8@math.washington.edu> Message-ID: <6EE3F3FF-E71A-4237-80EB-482D291942E2@math.washington.edu> On Aug 13, 2009, at 1:48 AM, Fernando Perez wrote: > On Thu, Aug 13, 2009 at 1:42 AM, Robert > Bradshaw wrote: >> Yes, that's very nice. (My preferred way to distribute cython is via >> Sage, but that doesn't work well for windows users yet.:-) > > There's also the fact that Cython can be of interest to many people > who might want to use it as a standalone tool in their own projects on > top of their normal Python toolchain, thus for whom Sage may be too > large a dependency (even if it was native for windows). For sure! (My emoticon was supposed to apply to the entire parenthetical statement--as much as I'd like to plug Sage, Cython has a much wider potential audience.) > Providing self-contained native windows installers is always a big > plus for a project in terms of gaining acceptance, even if it is > already packaged as part of some other toolchain (say Sage, EPD or > PythonXY for example, all of which ship Cython). I totally agree. > In any case, keep up the good work! I look forward to Dag's > presentation, and I hope you'll be able to make it next year to Scipy > :) > > Cheers, > > f > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From dagss at student.matnat.uio.no Thu Aug 13 18:54:12 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 13 Aug 2009 18:54:12 +0200 Subject: [Cython] Cython 0.11.2 win32 installer In-Reply-To: References: <96de71860908121538k98dd1e3ldaf6a567ad474b19@mail.gmail.com> <4A83C44D.9040107@student.matnat.uio.no> <96de71860908130133t2f97ed34he8ff5ba3eeabd517@mail.gmail.com> <7EBD264E-B8E0-4E90-B4CD-26507123D5F8@math.washington.edu> Message-ID: Fernando Perez wrote: > On Thu, Aug 13, 2009 at 1:42 AM, Robert > Bradshaw wrote: >> Yes, that's very nice. (My preferred way to distribute cython is via >> Sage, but that doesn't work well for windows users yet.:-) >> > > There's also the fact that Cython can be of interest to many people > who might want to use it as a standalone tool in their own projects on > top of their normal Python toolchain, thus for whom Sage may be too > large a dependency (even if it was native for windows). > > Providing self-contained native windows installers is always a big > plus for a project in terms of gaining acceptance, even if it is > already packaged as part of some other toolchain (say Sage, EPD or > PythonXY for example, all of which ship Cython). I'll be happy to talk with David about it and see what can be done. That said: I think the benefit for Cython is less than it would appear at first glance, since Cython is next to useless without a C compiler and Python C extension compilation support. PythonXY update Cython regularily, so the usecase that seems to remain for an .exe is to update EPD. (If you got a C compiler working outside of a distro, odds are you're more comfortable installing Cython from the egg/zip/etc. anyway?) A Cython installer bundling MinGW and setting it up properly would be something though! Dag Sverre From Chris.Barker at noaa.gov Thu Aug 13 19:01:40 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 13 Aug 2009 10:01:40 -0700 Subject: [Cython] Cython 0.11.2 win32 installer In-Reply-To: References: <96de71860908121538k98dd1e3ldaf6a567ad474b19@mail.gmail.com> <4A83C44D.9040107@student.matnat.uio.no> <96de71860908130133t2f97ed34he8ff5ba3eeabd517@mail.gmail.com> <7EBD264E-B8E0-4E90-B4CD-26507123D5F8@math.washington.edu> Message-ID: <4A8446F4.80200@noaa.gov> Dag Sverre Seljebotn wrote: > I think the benefit for Cython is less than it would appear at first > glance, since Cython is next to useless without a C compiler and Python C > extension compilation support. I agree -- getting that set up is a much bigger hurdle -- really, Windows installers are critical for packages that have to be compiled, so that people without a compiler can use them. > PythonXY update Cython regularily, so the usecase that seems to remain for > an .exe is to update EPD. Doesn't Enthought build their own anyway? > A Cython installer bundling MinGW and setting it up properly would be > something though! Now that would be great! I don't know why, but installing MinGW always seems to be far more of a pain than it should be. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From dagss at student.matnat.uio.no Thu Aug 13 19:33:53 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 13 Aug 2009 19:33:53 +0200 Subject: [Cython] Question about packaging external fortran parser with Cython for fwrap In-Reply-To: <19BF669A-FB71-4FB9-A4C9-DA6B23A2605C@math.washington.edu> References: <4A831F66.3090901@student.matnat.uio.no> <19BF669A-FB71-4FB9-A4C9-DA6B23A2605C@math.washington.edu> Message-ID: <0323dfabb1a602e3d00e75822c1e203e.squirrel@webmail.uio.no> Robert Bradshaw wrote: > On Aug 12, 2009, at 1:31 PM, Kurt Smith wrote: > >> On Wed, Aug 12, 2009 at 3:27 PM, Kurt Smith wrote: >>> On Wed, Aug 12, 2009 at 3:00 PM, Dag Sverre >>> Seljebotn wrote: >>>> Question: How large? Say, in bytes (and compressed bytes, if you >>>> bother >>>> to check). >>>> >>>> (I suppose it would be "hidden away" in the Tools folder, so >>>> download >>>> size seems to be the major concern.) >>> >>> About 600K for the .py source files, so we're not breaking the >>> bank here. >> >> ~150K compressed. > > That seems acceptable to me. Given that you've volunteered to > maintain it, and numerics certainly is an important audience of > Cython, it seems to make sense to bundle it. If anyone was wondering, I'm +1. Dag Sverre From josef.pktd at gmail.com Thu Aug 13 20:03:06 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Thu, 13 Aug 2009 14:03:06 -0400 Subject: [Cython] Cython 0.11.2 win32 installer In-Reply-To: <4A8446F4.80200@noaa.gov> References: <96de71860908121538k98dd1e3ldaf6a567ad474b19@mail.gmail.com> <4A83C44D.9040107@student.matnat.uio.no> <96de71860908130133t2f97ed34he8ff5ba3eeabd517@mail.gmail.com> <7EBD264E-B8E0-4E90-B4CD-26507123D5F8@math.washington.edu> <4A8446F4.80200@noaa.gov> Message-ID: <1cd32cbb0908131103q6c28afccvf67a54ceec821e58@mail.gmail.com> On Thu, Aug 13, 2009 at 1:01 PM, Christopher Barker wrote: > Dag Sverre Seljebotn wrote: >> I think the benefit for Cython is less than it would appear at first >> glance, since Cython is next to useless without a C compiler and Python C >> extension compilation support. > > I agree -- getting that set up is a much bigger hurdle -- really, > Windows installers are critical for packages that have to be compiled, > so that people without a compiler can use them. > >> PythonXY update Cython regularily, so the usecase that seems to remain for >> an .exe is to update EPD. > > Doesn't Enthought build their own anyway? > >> A Cython installer bundling MinGW and setting it up properly would be >> something though! > > Now that would be great! I don't know why, but installing MinGW always > seems to be far more of a pain than it should be. > > -Chris I don't think bundling MinGW with cython for Windows is a great idea. shedskin had mingw bundled and it works ok for getting started, but it's not useful for customization, adding libraries and so on. But to get MinGW properly installed in "program files" or whichever location the user wants, and setting it up properly seems a lot of work, which might be more appropriate for pythonxy, or other distributers, especially, since it has to be done only once, and doesn't need to be updated with cython. And with several copies of MinGW installed, it's always a pain to make sure the right one is on the Windows path. I don't think I had any problems when I used the official MinGW/msys package installers the last time. (I would be happy enough when the options in pyximport pick the right compiler so that the current hacks are not necessary anymore.) Josef > > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R ? ? ? ? ? ?(206) 526-6959 ? voice > 7600 Sand Point Way NE ? (206) 526-6329 ? fax > Seattle, WA ?98115 ? ? ? (206) 526-6317 ? main reception > > Chris.Barker at noaa.gov > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From kwmsmith at gmail.com Thu Aug 13 20:42:46 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Thu, 13 Aug 2009 13:42:46 -0500 Subject: [Cython] Question about packaging external fortran parser with Cython for fwrap In-Reply-To: <0323dfabb1a602e3d00e75822c1e203e.squirrel@webmail.uio.no> References: <4A831F66.3090901@student.matnat.uio.no> <19BF669A-FB71-4FB9-A4C9-DA6B23A2605C@math.washington.edu> <0323dfabb1a602e3d00e75822c1e203e.squirrel@webmail.uio.no> Message-ID: On Thu, Aug 13, 2009 at 12:33 PM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On Aug 12, 2009, at 1:31 PM, Kurt Smith wrote: >> >>> On Wed, Aug 12, 2009 at 3:27 PM, Kurt Smith wrote: >>>> On Wed, Aug 12, 2009 at 3:00 PM, Dag Sverre >>>> Seljebotn wrote: >>>>> Question: How large? Say, in bytes (and compressed bytes, if you >>>>> bother >>>>> to check). >>>>> >>>>> (I suppose it would be "hidden away" in the Tools folder, so >>>>> download >>>>> size seems to be the major concern.) >>>> >>>> About 600K for the .py source files, so we're not breaking the >>>> bank here. >>> >>> ~150K compressed. >> >> That seems acceptable to me. Given that you've volunteered to >> maintain it, and numerics certainly is an important audience of >> Cython, it seems to make sense to bundle it. > Good -- I sent an email to Pearu Peterson on the f2py-dev list to get his official approval, to check on licensing compatibilites, and to sort out the nuts and bolts of versioning fparser once it stabilizes, etc. If other Cython devs not-yet-heard-from have reservations, please speak up. Otherwise your silence will be interpreted as consent ;-) Kurt > If anyone was wondering, I'm +1. From kwmsmith at gmail.com Thu Aug 13 23:25:26 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Thu, 13 Aug 2009 16:25:26 -0500 Subject: [Cython] Question about packaging external fortran parser with Cython for fwrap In-Reply-To: References: <4A831F66.3090901@student.matnat.uio.no> <19BF669A-FB71-4FB9-A4C9-DA6B23A2605C@math.washington.edu> <0323dfabb1a602e3d00e75822c1e203e.squirrel@webmail.uio.no> Message-ID: On Thu, Aug 13, 2009 at 1:42 PM, Kurt Smith wrote: > On Thu, Aug 13, 2009 at 12:33 PM, Dag Sverre > Seljebotn wrote: >> Robert Bradshaw wrote: >>>> >>>> ~150K compressed. >>> >>> That seems acceptable to me. Given that you've volunteered to >>> maintain it, and numerics certainly is an important audience of >>> Cython, it seems to make sense to bundle it. >> > > Good -- I sent an email to Pearu Peterson on the f2py-dev list to get > his official approval, to check on licensing compatibilites, and to > sort out the nuts and bolts of versioning fparser once it stabilizes, > etc. I have Pearu's permission to include fparser in Cython. Fparser is BSD licensed, and I'll be reading through it to see what requirements there are. If the following site is authoritative: http://www.dwheeler.com/essays/floss-license-slide.html then it looks like we won't have licensing issues. Kurt > > If other Cython devs not-yet-heard-from have reservations, please > speak up. ?Otherwise your silence will be interpreted as consent ;-) From robert.kern at gmail.com Fri Aug 14 01:55:32 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 13 Aug 2009 18:55:32 -0500 Subject: [Cython] Question about packaging external fortran parser with Cython for fwrap In-Reply-To: References: <4A831F66.3090901@student.matnat.uio.no> <19BF669A-FB71-4FB9-A4C9-DA6B23A2605C@math.washington.edu> <0323dfabb1a602e3d00e75822c1e203e.squirrel@webmail.uio.no> Message-ID: On 2009-08-13 16:25 PM, Kurt Smith wrote: > On Thu, Aug 13, 2009 at 1:42 PM, Kurt Smith wrote: >> On Thu, Aug 13, 2009 at 12:33 PM, Dag Sverre >> Seljebotn wrote: >>> Robert Bradshaw wrote: > >>>>> >>>>> ~150K compressed. >>>> >>>> That seems acceptable to me. Given that you've volunteered to >>>> maintain it, and numerics certainly is an important audience of >>>> Cython, it seems to make sense to bundle it. >>> >> >> Good -- I sent an email to Pearu Peterson on the f2py-dev list to get >> his official approval, to check on licensing compatibilites, and to >> sort out the nuts and bolts of versioning fparser once it stabilizes, >> etc. > > I have Pearu's permission to include fparser in Cython. Fparser is > BSD licensed, and I'll be reading through it to see what requirements > there are. If the following site is authoritative: > > http://www.dwheeler.com/essays/floss-license-slide.html > > then it looks like we won't have licensing issues. It would be worth asking Pearu to explicitly give his code to you under the Apache license. He doesn't have to relicense fparser in general, but just give it to you under that license so you can correctly claim that all of Cython is Apache licensed. It's an annoying process for substantially similar licenses, but it will make a few things easier down the line. In particular, if Cython is still serious about inclusion into the standard library, this is a necessity since the PSF will accept Apache-licensed code and not BSD-licensed code. I believe this was one of the original reasons for using the Apache license for Cython to begin with. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From wstein at gmail.com Fri Aug 14 02:05:35 2009 From: wstein at gmail.com (William Stein) Date: Thu, 13 Aug 2009 17:05:35 -0700 Subject: [Cython] Question about packaging external fortran parser with Cython for fwrap In-Reply-To: References: <4A831F66.3090901@student.matnat.uio.no> <19BF669A-FB71-4FB9-A4C9-DA6B23A2605C@math.washington.edu> <0323dfabb1a602e3d00e75822c1e203e.squirrel@webmail.uio.no> Message-ID: <85e81ba30908131705w6f885e5fm678b759a974ebb69@mail.gmail.com> On Thu, Aug 13, 2009 at 4:55 PM, Robert Kern wrote: > On 2009-08-13 16:25 PM, Kurt Smith wrote: >> On Thu, Aug 13, 2009 at 1:42 PM, Kurt Smith ?wrote: >>> On Thu, Aug 13, 2009 at 12:33 PM, Dag Sverre >>> Seljebotn ?wrote: >>>> Robert Bradshaw wrote: >> >>>>>> >>>>>> ~150K compressed. >>>>> >>>>> That seems acceptable to me. Given that you've volunteered to >>>>> maintain it, and numerics certainly is an important audience of >>>>> Cython, it seems to make sense to bundle it. >>>> >>> >>> Good -- I sent an email to Pearu Peterson on the f2py-dev list to get >>> his official approval, to check on licensing compatibilites, and to >>> sort out the nuts and bolts of versioning fparser once it stabilizes, >>> etc. >> >> I have Pearu's permission to include fparser in Cython. ?Fparser is >> BSD licensed, and I'll be reading through it to see what requirements >> there are. ?If the following site is authoritative: >> >> http://www.dwheeler.com/essays/floss-license-slide.html >> >> then it looks like we won't have licensing issues. > > It would be worth asking Pearu to explicitly give his code to you under the > Apache license. He doesn't have to relicense fparser in general, but just give > it to you under that license so you can correctly claim that all of Cython is > Apache licensed. > > It's an annoying process for substantially similar licenses, but it will make a > few things easier down the line. In particular, if Cython is still serious about > inclusion into the standard library, this is a necessity since the PSF will > accept Apache-licensed code and not BSD-licensed code. I'm just curious. Why does the PSF reject BSD-licensed code, but accept Apache-licensed code, given that the Apache license is vastly more restrictive than BSD? I can understand this, but find it surprising. The Apache license isn't GPL v2 compatible according to the FSF by the way, but it is GPL v3 compatible (http://www.apache.org/licenses/GPL-compatibility.html). This is relevant to another thread on sage-devel right now, because we claim jsmath is the only Apache licensed code in Sage. In fact Cython is another Apache-licensed component of Sage. In our COPYING.txt file we write: cython GPL (pyrex has no restrictions) which is clearly very wrong. -- William > I believe this was one of > the original reasons for using the Apache license for Cython to begin with. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless enigma > ?that is made terrible by our own mad attempt to interpret it as though it had > ?an underlying truth." > ? -- Umberto Eco > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- William Stein Associate Professor of Mathematics University of Washington http://wstein.org From fperez.net at gmail.com Fri Aug 14 03:48:18 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 13 Aug 2009 18:48:18 -0700 Subject: [Cython] [sage-devel] Re: Question about packaging external fortran parser with Cython for fwrap In-Reply-To: <85e81ba30908131705w6f885e5fm678b759a974ebb69@mail.gmail.com> References: <4A831F66.3090901@student.matnat.uio.no> <19BF669A-FB71-4FB9-A4C9-DA6B23A2605C@math.washington.edu> <0323dfabb1a602e3d00e75822c1e203e.squirrel@webmail.uio.no> <85e81ba30908131705w6f885e5fm678b759a974ebb69@mail.gmail.com> Message-ID: On Thu, Aug 13, 2009 at 5:05 PM, William Stein wrote: > I'm just curious. ?Why does the PSF reject BSD-licensed code, but > accept Apache-licensed code, given that the Apache license is vastly > more restrictive than BSD? ?I can understand this, but find it > surprising. > I'm also scratching my head a little on that one, and would love to hear the reasons behind it... Cheers, f From robertwb at math.washington.edu Fri Aug 14 06:51:21 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 13 Aug 2009 21:51:21 -0700 Subject: [Cython] [sage-devel] Re: Question about packaging external fortran parser with Cython for fwrap In-Reply-To: References: <4A831F66.3090901@student.matnat.uio.no> <19BF669A-FB71-4FB9-A4C9-DA6B23A2605C@math.washington.edu> <0323dfabb1a602e3d00e75822c1e203e.squirrel@webmail.uio.no> <85e81ba30908131705w6f885e5fm678b759a974ebb69@mail.gmail.com> Message-ID: <2895B654-4992-4DB8-864F-0C62CCE3008E@math.washington.edu> On Aug 13, 2009, at 6:48 PM, Fernando Perez wrote: > On Thu, Aug 13, 2009 at 5:05 PM, William Stein > wrote: >> I'm just curious. Why does the PSF reject BSD-licensed code, but >> accept Apache-licensed code, given that the Apache license is vastly >> more restrictive than BSD? I can understand this, but find it >> surprising. >> > > I'm also scratching my head a little on that one, and would love to > hear the reasons behind it... Yes, me too. The page in question is here: http://www.python.org/psf/ contrib/ . I think it would be much simpler to just use the BSD and be done with it. On the other hand, ctypes is MIT license, and is now included with Python, so maybe the above is not set in stone. - Robert From ondrej at certik.cz Fri Aug 14 08:42:22 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Fri, 14 Aug 2009 00:42:22 -0600 Subject: [Cython] [sage-devel] Re: Question about packaging external fortran parser with Cython for fwrap In-Reply-To: <2895B654-4992-4DB8-864F-0C62CCE3008E@math.washington.edu> References: <19BF669A-FB71-4FB9-A4C9-DA6B23A2605C@math.washington.edu> <0323dfabb1a602e3d00e75822c1e203e.squirrel@webmail.uio.no> <85e81ba30908131705w6f885e5fm678b759a974ebb69@mail.gmail.com> <2895B654-4992-4DB8-864F-0C62CCE3008E@math.washington.edu> Message-ID: <85b5c3130908132342u6c45ae5ev791d863e87988e99@mail.gmail.com> On Thu, Aug 13, 2009 at 10:51 PM, Robert Bradshaw wrote: > > On Aug 13, 2009, at 6:48 PM, Fernando Perez wrote: > >> On Thu, Aug 13, 2009 at 5:05 PM, William Stein >> wrote: >>> I'm just curious. ?Why does the PSF reject BSD-licensed code, but >>> accept Apache-licensed code, given that the Apache license is vastly >>> more restrictive than BSD? ?I can understand this, but find it >>> surprising. >>> >> >> I'm also scratching my head a little on that one, and would love to >> hear the reasons behind it... > > Yes, me too. The page in question is here: http://www.python.org/psf/ > contrib/ . I think it would be much simpler to just use the BSD and > be done with it. > > On the other hand, ctypes is MIT license, and is now included with > Python, so maybe the above is not set in stone. What stops you from taking a BSD code and relicense it to Apache? You don't need any permission, do you? Ondrej From robertwb at math.washington.edu Fri Aug 14 18:10:35 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 14 Aug 2009 09:10:35 -0700 Subject: [Cython] [sage-devel] Re: Question about packaging external fortran parser with Cython for fwrap In-Reply-To: <85b5c3130908132342u6c45ae5ev791d863e87988e99@mail.gmail.com> References: <19BF669A-FB71-4FB9-A4C9-DA6B23A2605C@math.washington.edu> <0323dfabb1a602e3d00e75822c1e203e.squirrel@webmail.uio.no> <85e81ba30908131705w6f885e5fm678b759a974ebb69@mail.gmail.com> <2895B654-4992-4DB8-864F-0C62CCE3008E@math.washington.edu> <85b5c3130908132342u6c45ae5ev791d863e87988e99@mail.gmail.com> Message-ID: <2BFA1D69-A3A4-47DE-80BF-BE74CA401A33@math.washington.edu> On Aug 13, 2009, at 11:42 PM, Ondrej Certik wrote: > On Thu, Aug 13, 2009 at 10:51 PM, Robert > Bradshaw wrote: >> >> On Aug 13, 2009, at 6:48 PM, Fernando Perez wrote: >> >>> On Thu, Aug 13, 2009 at 5:05 PM, William Stein >>> wrote: >>>> I'm just curious. Why does the PSF reject BSD-licensed code, but >>>> accept Apache-licensed code, given that the Apache license is >>>> vastly >>>> more restrictive than BSD? I can understand this, but find it >>>> surprising. >>>> >>> >>> I'm also scratching my head a little on that one, and would love to >>> hear the reasons behind it... >> >> Yes, me too. The page in question is here: http://www.python.org/psf/ >> contrib/ . I think it would be much simpler to just use the BSD and >> be done with it. >> >> On the other hand, ctypes is MIT license, and is now included with >> Python, so maybe the above is not set in stone. > > What stops you from taking a BSD code and relicense it to Apache? > > You don't need any permission, do you? No one's been giving me their copyrights, so I (personally) can't just change the license as far as I understand. - Robert From robert.kern at gmail.com Fri Aug 14 18:15:14 2009 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 14 Aug 2009 11:15:14 -0500 Subject: [Cython] Question about packaging external fortran parser with Cython for fwrap In-Reply-To: <85e81ba30908131705w6f885e5fm678b759a974ebb69@mail.gmail.com> References: <4A831F66.3090901@student.matnat.uio.no> <19BF669A-FB71-4FB9-A4C9-DA6B23A2605C@math.washington.edu> <0323dfabb1a602e3d00e75822c1e203e.squirrel@webmail.uio.no> <85e81ba30908131705w6f885e5fm678b759a974ebb69@mail.gmail.com> Message-ID: On 2009-08-13 19:05 PM, William Stein wrote: > > On Thu, Aug 13, 2009 at 4:55 PM, Robert Kern wrote: >> On 2009-08-13 16:25 PM, Kurt Smith wrote: >>> On Thu, Aug 13, 2009 at 1:42 PM, Kurt Smith wrote: >>>> On Thu, Aug 13, 2009 at 12:33 PM, Dag Sverre >>>> Seljebotn wrote: >>>>> Robert Bradshaw wrote: >>> >>>>>>> >>>>>>> ~150K compressed. >>>>>> >>>>>> That seems acceptable to me. Given that you've volunteered to >>>>>> maintain it, and numerics certainly is an important audience of >>>>>> Cython, it seems to make sense to bundle it. >>>>> >>>> >>>> Good -- I sent an email to Pearu Peterson on the f2py-dev list to get >>>> his official approval, to check on licensing compatibilites, and to >>>> sort out the nuts and bolts of versioning fparser once it stabilizes, >>>> etc. >>> >>> I have Pearu's permission to include fparser in Cython. Fparser is >>> BSD licensed, and I'll be reading through it to see what requirements >>> there are. If the following site is authoritative: >>> >>> http://www.dwheeler.com/essays/floss-license-slide.html >>> >>> then it looks like we won't have licensing issues. >> >> It would be worth asking Pearu to explicitly give his code to you under the >> Apache license. He doesn't have to relicense fparser in general, but just give >> it to you under that license so you can correctly claim that all of Cython is >> Apache licensed. >> >> It's an annoying process for substantially similar licenses, but it will make a >> few things easier down the line. In particular, if Cython is still serious about >> inclusion into the standard library, this is a necessity since the PSF will >> accept Apache-licensed code and not BSD-licensed code. > > I'm just curious. Why does the PSF reject BSD-licensed code, but > accept Apache-licensed code, given that the Apache license is vastly > more restrictive than BSD? I can understand this, but find it > surprising. http://www.python.org/psf/contrib/ """ The choice of initial license originates from the advice of our lawyer: the initial licenses should be open source, they should be clear and explicit in what they permit, and they should not only include a copyright grant (in particular to reproduce and prepare derivative works), but also a license grant to all relevant patents that the contributor holds. As most contributors do not hold any patents (at least none relevant for the contribution), the patent grant is likely irrelevant for most contributors. """ If you want to be cynical about it, it basically boils down to the personal preference of the PSF's legal counsel, Larry Rosen, not coincidentally the author of the Academic Free License, the other accepted license. Something that the Cython team may want to consider while the contributor list is manageable is to get agreements from all contributors to dual-license their code for BSD and Apache. The BSD license grant allows GPLv2 mixing and the Apache license grant allows eventual stdlib inclusion, if that is still on the agenda. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Fri Aug 14 18:18:22 2009 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 14 Aug 2009 11:18:22 -0500 Subject: [Cython] Question about packaging external fortran parser with Cython for fwrap In-Reply-To: <85b5c3130908132342u6c45ae5ev791d863e87988e99@mail.gmail.com> References: <19BF669A-FB71-4FB9-A4C9-DA6B23A2605C@math.washington.edu> <0323dfabb1a602e3d00e75822c1e203e.squirrel@webmail.uio.no> <85e81ba30908131705w6f885e5fm678b759a974ebb69@mail.gmail.com> <2895B654-4992-4DB8-864F-0C62CCE3008E@math.washington.edu> <85b5c3130908132342u6c45ae5ev791d863e87988e99@mail.gmail.com> Message-ID: On 2009-08-14 01:42 AM, Ondrej Certik wrote: > > On Thu, Aug 13, 2009 at 10:51 PM, Robert > Bradshaw wrote: >> >> On Aug 13, 2009, at 6:48 PM, Fernando Perez wrote: >> >>> On Thu, Aug 13, 2009 at 5:05 PM, William Stein >>> wrote: >>>> I'm just curious. Why does the PSF reject BSD-licensed code, but >>>> accept Apache-licensed code, given that the Apache license is vastly >>>> more restrictive than BSD? I can understand this, but find it >>>> surprising. >>>> >>> >>> I'm also scratching my head a little on that one, and would love to >>> hear the reasons behind it... >> >> Yes, me too. The page in question is here: http://www.python.org/psf/ >> contrib/ . I think it would be much simpler to just use the BSD and >> be done with it. >> >> On the other hand, ctypes is MIT license, and is now included with >> Python, so maybe the above is not set in stone. > > What stops you from taking a BSD code and relicense it to Apache? > > You don't need any permission, do you? If you're being a stickler about licenses, you don't need permission to re-release someone else's software under the Apache license, but that doesn't actually make the software fully under the Apache license. The original author never agreed to the terms of the Apache license and never made the guarantees in those terms, and it is those guarantees that the PSF wants. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Fri Aug 14 18:21:51 2009 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 14 Aug 2009 11:21:51 -0500 Subject: [Cython] Question about packaging external fortran parser with Cython for fwrap In-Reply-To: <85b5c3130908132342u6c45ae5ev791d863e87988e99@mail.gmail.com> References: <19BF669A-FB71-4FB9-A4C9-DA6B23A2605C@math.washington.edu> <0323dfabb1a602e3d00e75822c1e203e.squirrel@webmail.uio.no> <85e81ba30908131705w6f885e5fm678b759a974ebb69@mail.gmail.com> <2895B654-4992-4DB8-864F-0C62CCE3008E@math.washington.edu> <85b5c3130908132342u6c45ae5ev791d863e87988e99@mail.gmail.com> Message-ID: On 2009-08-14 01:42 AM, Ondrej Certik wrote: > > On Thu, Aug 13, 2009 at 10:51 PM, Robert > Bradshaw wrote: >> >> On Aug 13, 2009, at 6:48 PM, Fernando Perez wrote: >> >>> On Thu, Aug 13, 2009 at 5:05 PM, William Stein >>> wrote: >>>> I'm just curious. Why does the PSF reject BSD-licensed code, but >>>> accept Apache-licensed code, given that the Apache license is vastly >>>> more restrictive than BSD? I can understand this, but find it >>>> surprising. >>>> >>> >>> I'm also scratching my head a little on that one, and would love to >>> hear the reasons behind it... >> >> Yes, me too. The page in question is here: http://www.python.org/psf/ >> contrib/ . I think it would be much simpler to just use the BSD and >> be done with it. >> >> On the other hand, ctypes is MIT license, and is now included with >> Python, so maybe the above is not set in stone. > > What stops you from taking a BSD code and relicense it to Apache? > > You don't need any permission, do you? In particular, the Apache license includes language that explicitly allows you to sublicense. The BSD license does not, so BSD->PSF or BSD->Apache->PSF is murkier than Apache->PSF. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dagss at student.matnat.uio.no Fri Aug 14 18:37:55 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 14 Aug 2009 18:37:55 +0200 Subject: [Cython] Question about packaging external fortran parser with Cython for fwrap In-Reply-To: References: <19BF669A-FB71-4FB9-A4C9-DA6B23A2605C@math.washington.edu> <0323dfabb1a602e3d00e75822c1e203e.squirrel@webmail.uio.no> <85e81ba30908131705w6f885e5fm678b759a974ebb69@mail.gmail.com> <2895B654-4992-4DB8-864F-0C62CCE3008E@math.washington.edu> <85b5c3130908132342u6c45ae5ev791d863e87988e99@mail.gmail.com> Message-ID: <4A8592E3.5050909@student.matnat.uio.no> Robert Kern wrote: > On 2009-08-14 01:42 AM, Ondrej Certik wrote: >> On Thu, Aug 13, 2009 at 10:51 PM, Robert >> Bradshaw wrote: >>> On Aug 13, 2009, at 6:48 PM, Fernando Perez wrote: >>> >>>> On Thu, Aug 13, 2009 at 5:05 PM, William Stein >>>> wrote: >>>>> I'm just curious. Why does the PSF reject BSD-licensed code, but >>>>> accept Apache-licensed code, given that the Apache license is vastly >>>>> more restrictive than BSD? I can understand this, but find it >>>>> surprising. >>>>> >>>> I'm also scratching my head a little on that one, and would love to >>>> hear the reasons behind it... >>> Yes, me too. The page in question is here: http://www.python.org/psf/ >>> contrib/ . I think it would be much simpler to just use the BSD and >>> be done with it. >>> >>> On the other hand, ctypes is MIT license, and is now included with >>> Python, so maybe the above is not set in stone. >> What stops you from taking a BSD code and relicense it to Apache? >> >> You don't need any permission, do you? > > If you're being a stickler about licenses, you don't need permission to > re-release someone else's software under the Apache license, but that doesn't > actually make the software fully under the Apache license. The original author > never agreed to the terms of the Apache license and never made the guarantees in > those terms, and it is those guarantees that the PSF wants. As for fparser, I think we can keep PSF out of it for now: fwrap depends on Cython and fparser, but Cython itself will never depend on either At least for the time being it's more a matter of shipping fparser and fwrap with the default Cython download (in the Tools dir). I can definitely see Cython itself being included into Python's stdlib without fwrap following along (as nobody outside of science use Fortran anyway). -- Dag Sverre From ondrej at certik.cz Fri Aug 14 18:50:45 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Fri, 14 Aug 2009 10:50:45 -0600 Subject: [Cython] Question about packaging external fortran parser with Cython for fwrap In-Reply-To: References: <0323dfabb1a602e3d00e75822c1e203e.squirrel@webmail.uio.no> <85e81ba30908131705w6f885e5fm678b759a974ebb69@mail.gmail.com> <2895B654-4992-4DB8-864F-0C62CCE3008E@math.washington.edu> <85b5c3130908132342u6c45ae5ev791d863e87988e99@mail.gmail.com> Message-ID: <85b5c3130908140950x6c4406dbu776ddc873dc22b03@mail.gmail.com> On Fri, Aug 14, 2009 at 10:21 AM, Robert Kern wrote: > On 2009-08-14 01:42 AM, Ondrej Certik wrote: >> >> On Thu, Aug 13, 2009 at 10:51 PM, Robert >> Bradshaw ?wrote: >>> >>> On Aug 13, 2009, at 6:48 PM, Fernando Perez wrote: >>> >>>> On Thu, Aug 13, 2009 at 5:05 PM, William Stein >>>> wrote: >>>>> I'm just curious. ?Why does the PSF reject BSD-licensed code, but >>>>> accept Apache-licensed code, given that the Apache license is vastly >>>>> more restrictive than BSD? ?I can understand this, but find it >>>>> surprising. >>>>> >>>> >>>> I'm also scratching my head a little on that one, and would love to >>>> hear the reasons behind it... >>> >>> Yes, me too. The page in question is here: http://www.python.org/psf/ >>> contrib/ . I think it would be much simpler to just use the BSD and >>> be done with it. >>> >>> On the other hand, ctypes is MIT license, and is now included with >>> Python, so maybe the above is not set in stone. >> >> What stops you from taking a BSD code and relicense it to Apache? >> >> You don't need any permission, do you? > > In particular, the Apache license includes language that explicitly allows you > to sublicense. The BSD license does not, so BSD->PSF or BSD->Apache->PSF is > murkier than Apache->PSF. Ok, I thought that BSD is a safe bet for everything. I was wrong. Ondrej From robert.kern at gmail.com Fri Aug 14 19:00:22 2009 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 14 Aug 2009 12:00:22 -0500 Subject: [Cython] Question about packaging external fortran parser with Cython for fwrap In-Reply-To: <4A8592E3.5050909@student.matnat.uio.no> References: <19BF669A-FB71-4FB9-A4C9-DA6B23A2605C@math.washington.edu> <0323dfabb1a602e3d00e75822c1e203e.squirrel@webmail.uio.no> <85e81ba30908131705w6f885e5fm678b759a974ebb69@mail.gmail.com> <2895B654-4992-4DB8-864F-0C62CCE3008E@math.washington.edu> <85b5c3130908132342u6c45ae5ev791d863e87988e99@mail.gmail.com> <4A8592E3.5050909@student.matnat.uio.no> Message-ID: On 2009-08-14 11:37 AM, Dag Sverre Seljebotn wrote: > As for fparser, I think we can keep PSF out of it for now: > > fwrap depends on Cython and fparser, but Cython itself will never depend > on either > > At least for the time being it's more a matter of shipping fparser and > fwrap with the default Cython download (in the Tools dir). I can > definitely see Cython itself being included into Python's stdlib without > fwrap following along (as nobody outside of science use Fortran anyway). Hmmm. Are you sure you don't just want to distribute fwrap as a separate project, then? Personally, I will always prefer a second package over an optional component. If I had a project that used fwrap to build, I would want my prerequisite installation instructions to be this: 1. Install Cython. http://pypi.python.org/pypi/Cython 2. Install fwrap. http://pypi.python.org/pypi/fwrap rather than this: 1. Download the source tarball of Cython. http://pypi.python.org/pypi/Cython 2. Install Cython using "python setup.py install" 3. cd Tools/fwrap/ 4. Install fwrap using "python setup.py install" With the latter, there will inevitably be some Linux distribution that will put both Cython and fwrap into their python-cython package and some who won't. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robert.kern at gmail.com Fri Aug 14 19:02:40 2009 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 14 Aug 2009 12:02:40 -0500 Subject: [Cython] Question about packaging external fortran parser with Cython for fwrap In-Reply-To: <85b5c3130908140950x6c4406dbu776ddc873dc22b03@mail.gmail.com> References: <0323dfabb1a602e3d00e75822c1e203e.squirrel@webmail.uio.no> <85e81ba30908131705w6f885e5fm678b759a974ebb69@mail.gmail.com> <2895B654-4992-4DB8-864F-0C62CCE3008E@math.washington.edu> <85b5c3130908132342u6c45ae5ev791d863e87988e99@mail.gmail.com> <85b5c3130908140950x6c4406dbu776ddc873dc22b03@mail.gmail.com> Message-ID: On 2009-08-14 11:50 AM, Ondrej Certik wrote: > > On Fri, Aug 14, 2009 at 10:21 AM, Robert Kern wrote: >> On 2009-08-14 01:42 AM, Ondrej Certik wrote: >>> >>> On Thu, Aug 13, 2009 at 10:51 PM, Robert >>> Bradshaw wrote: >>>> >>>> On Aug 13, 2009, at 6:48 PM, Fernando Perez wrote: >>>> >>>>> On Thu, Aug 13, 2009 at 5:05 PM, William Stein >>>>> wrote: >>>>>> I'm just curious. Why does the PSF reject BSD-licensed code, but >>>>>> accept Apache-licensed code, given that the Apache license is vastly >>>>>> more restrictive than BSD? I can understand this, but find it >>>>>> surprising. >>>>>> >>>>> >>>>> I'm also scratching my head a little on that one, and would love to >>>>> hear the reasons behind it... >>>> >>>> Yes, me too. The page in question is here: http://www.python.org/psf/ >>>> contrib/ . I think it would be much simpler to just use the BSD and >>>> be done with it. >>>> >>>> On the other hand, ctypes is MIT license, and is now included with >>>> Python, so maybe the above is not set in stone. >>> >>> What stops you from taking a BSD code and relicense it to Apache? >>> >>> You don't need any permission, do you? >> >> In particular, the Apache license includes language that explicitly allows you >> to sublicense. The BSD license does not, so BSD->PSF or BSD->Apache->PSF is >> murkier than Apache->PSF. > > Ok, I thought that BSD is a safe bet for everything. I was wrong. If you don't expect to get lawyers involved, it's a pretty good choice. If you do get lawyers involved, the Apache license has all of the verbiage to make them happy and shut up. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From kwmsmith at gmail.com Fri Aug 14 20:42:45 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Fri, 14 Aug 2009 13:42:45 -0500 Subject: [Cython] Question about packaging external fortran parser with Cython for fwrap In-Reply-To: References: <85e81ba30908131705w6f885e5fm678b759a974ebb69@mail.gmail.com> <2895B654-4992-4DB8-864F-0C62CCE3008E@math.washington.edu> <85b5c3130908132342u6c45ae5ev791d863e87988e99@mail.gmail.com> <4A8592E3.5050909@student.matnat.uio.no> Message-ID: On Fri, Aug 14, 2009 at 12:00 PM, Robert Kern wrote: > On 2009-08-14 11:37 AM, Dag Sverre Seljebotn wrote: > >> As for fparser, I think we can keep PSF out of it for now: >> >> fwrap depends on Cython and fparser, but Cython itself will never depend >> on either >> >> At least for the time being it's more a matter of shipping fparser and >> fwrap with the default Cython download (in the Tools dir). I can >> definitely see Cython itself being included into Python's stdlib without >> fwrap following along (as nobody outside of science use Fortran anyway). > > Hmmm. Are you sure you don't just want to distribute fwrap as a separate > project, then? Personally, I will always prefer a second package over an > optional component. If I had a project that used fwrap to build, I would want my > prerequisite installation instructions to be this: > > ? 1. Install Cython. http://pypi.python.org/pypi/Cython > ? 2. Install fwrap. http://pypi.python.org/pypi/fwrap > > rather than this: > > ? 1. Download the source tarball of Cython. http://pypi.python.org/pypi/Cython > ? 2. Install Cython using "python setup.py install" > ? 3. cd Tools/fwrap/ > ? 4. Install fwrap using "python setup.py install" > > With the latter, there will inevitably be some Linux distribution that will put > both Cython and fwrap into their python-cython package and some who won't. Good points. Dag and I talked about packaging fwrap separately once it stabilizes. Perhaps we should just make it its own package from the start. This would alleviate the licensing issues and 'core Cython' could be pure Apache (presuming Pearu doesn't want to relicense fparser for fwrap under Apache). Thoughts? > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless enigma > ?that is made terrible by our own mad attempt to interpret it as though it had > ?an underlying truth." > ? -- Umberto Eco > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From robert.kern at gmail.com Fri Aug 14 21:17:36 2009 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 14 Aug 2009 14:17:36 -0500 Subject: [Cython] Question about packaging external fortran parser with Cython for fwrap In-Reply-To: References: <85e81ba30908131705w6f885e5fm678b759a974ebb69@mail.gmail.com> <2895B654-4992-4DB8-864F-0C62CCE3008E@math.washington.edu> <85b5c3130908132342u6c45ae5ev791d863e87988e99@mail.gmail.com> <4A8592E3.5050909@student.matnat.uio.no> Message-ID: On 2009-08-14 13:42 PM, Kurt Smith wrote: > On Fri, Aug 14, 2009 at 12:00 PM, Robert Kern wrote: >> On 2009-08-14 11:37 AM, Dag Sverre Seljebotn wrote: >> >>> As for fparser, I think we can keep PSF out of it for now: >>> >>> fwrap depends on Cython and fparser, but Cython itself will never depend >>> on either >>> >>> At least for the time being it's more a matter of shipping fparser and >>> fwrap with the default Cython download (in the Tools dir). I can >>> definitely see Cython itself being included into Python's stdlib without >>> fwrap following along (as nobody outside of science use Fortran anyway). >> >> Hmmm. Are you sure you don't just want to distribute fwrap as a separate >> project, then? Personally, I will always prefer a second package over an >> optional component. If I had a project that used fwrap to build, I would want my >> prerequisite installation instructions to be this: >> >> 1. Install Cython. http://pypi.python.org/pypi/Cython >> 2. Install fwrap. http://pypi.python.org/pypi/fwrap >> >> rather than this: >> >> 1. Download the source tarball of Cython. http://pypi.python.org/pypi/Cython >> 2. Install Cython using "python setup.py install" >> 3. cd Tools/fwrap/ >> 4. Install fwrap using "python setup.py install" >> >> With the latter, there will inevitably be some Linux distribution that will put >> both Cython and fwrap into their python-cython package and some who won't. > > Good points. Dag and I talked about packaging fwrap separately once > it stabilizes. Perhaps we should just make it its own package from > the start. This would alleviate the licensing issues and 'core > Cython' could be pure Apache (presuming Pearu doesn't want to > relicense fparser for fwrap under Apache). > > Thoughts? I would think that it would be better to start off separate while it stabilizes. If you don't intend for a Cython release to contain it, why bother putting it in the Cython repo in the first place? Mercurial repos are cheap. :-) But I'm certainly not doing any work. Do what you think is best. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From Chris.Barker at noaa.gov Fri Aug 14 21:22:09 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 14 Aug 2009 12:22:09 -0700 Subject: [Cython] Question about packaging external fortran parser with Cython for fwrap In-Reply-To: References: <85e81ba30908131705w6f885e5fm678b759a974ebb69@mail.gmail.com> <2895B654-4992-4DB8-864F-0C62CCE3008E@math.washington.edu> <85b5c3130908132342u6c45ae5ev791d863e87988e99@mail.gmail.com> <4A8592E3.5050909@student.matnat.uio.no> Message-ID: <4A85B961.2080101@noaa.gov> It may be too late now, but if you're building a package that is only useful when used with with c-python, and particularly if you think you may want it included in the standard library some day, why not use the Python License from the beginning? Unless you DO want to be more restrictive (ala GPL), but I don't think that's the case here. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From robert.kern at gmail.com Fri Aug 14 21:34:57 2009 From: robert.kern at gmail.com (Robert Kern) Date: Fri, 14 Aug 2009 14:34:57 -0500 Subject: [Cython] Question about packaging external fortran parser with Cython for fwrap In-Reply-To: <4A85B961.2080101@noaa.gov> References: <85e81ba30908131705w6f885e5fm678b759a974ebb69@mail.gmail.com> <2895B654-4992-4DB8-864F-0C62CCE3008E@math.washington.edu> <85b5c3130908132342u6c45ae5ev791d863e87988e99@mail.gmail.com> <4A8592E3.5050909@student.matnat.uio.no> <4A85B961.2080101@noaa.gov> Message-ID: On 2009-08-14 14:22 PM, Christopher Barker wrote: > It may be too late now, but if you're building a package that is only > useful when used with with c-python, and particularly if you think you > may want it included in the standard library some day, why not use the > Python License from the beginning? Because the PSF does not accept code under the PSF license. http://www.python.org/psf/contrib/ -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From kwmsmith at gmail.com Fri Aug 14 21:57:30 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Fri, 14 Aug 2009 14:57:30 -0500 Subject: [Cython] Question about packaging external fortran parser with Cython for fwrap In-Reply-To: References: <85e81ba30908131705w6f885e5fm678b759a974ebb69@mail.gmail.com> <2895B654-4992-4DB8-864F-0C62CCE3008E@math.washington.edu> <85b5c3130908132342u6c45ae5ev791d863e87988e99@mail.gmail.com> <4A8592E3.5050909@student.matnat.uio.no> Message-ID: On Fri, Aug 14, 2009 at 2:17 PM, Robert Kern wrote: > On 2009-08-14 13:42 PM, Kurt Smith wrote: >> On Fri, Aug 14, 2009 at 12:00 PM, Robert Kern ?wrote: >>> On 2009-08-14 11:37 AM, Dag Sverre Seljebotn wrote: >>> >>>> As for fparser, I think we can keep PSF out of it for now: >>>> >>>> fwrap depends on Cython and fparser, but Cython itself will never depend >>>> on either >>>> >>>> At least for the time being it's more a matter of shipping fparser and >>>> fwrap with the default Cython download (in the Tools dir). I can >>>> definitely see Cython itself being included into Python's stdlib without >>>> fwrap following along (as nobody outside of science use Fortran anyway). >>> >>> Hmmm. Are you sure you don't just want to distribute fwrap as a separate >>> project, then? Personally, I will always prefer a second package over an >>> optional component. If I had a project that used fwrap to build, I would want my >>> prerequisite installation instructions to be this: >>> >>> ? ?1. Install Cython. http://pypi.python.org/pypi/Cython >>> ? ?2. Install fwrap. http://pypi.python.org/pypi/fwrap >>> >>> rather than this: >>> >>> ? ?1. Download the source tarball of Cython. http://pypi.python.org/pypi/Cython >>> ? ?2. Install Cython using "python setup.py install" >>> ? ?3. cd Tools/fwrap/ >>> ? ?4. Install fwrap using "python setup.py install" >>> >>> With the latter, there will inevitably be some Linux distribution that will put >>> both Cython and fwrap into their python-cython package and some who won't. >> >> Good points. ?Dag and I talked about packaging fwrap separately once >> it stabilizes. ?Perhaps we should just make it its own package from >> the start. ?This would alleviate the licensing issues and 'core >> Cython' could be pure Apache (presuming Pearu doesn't want to >> relicense fparser for fwrap under Apache). >> >> Thoughts? > > I would think that it would be better to start off separate while it stabilizes. > If you don't intend for a Cython release to contain it, why bother putting it in > the Cython repo in the first place? Mercurial repos are cheap. :-) This aspect of things (where fwrap belongs) has been evolving over the summer, as has fwrap's 'intent'. Is it primarily a tool to use with Cython, allowing one to wrap external Fortran code along with C, as initially intended? Or is it its own self-standing thing, like an updated f2py? fwrap is progressing towards the latter, and this thread is good to thresh out the best solution. fparser has been a tremendous help, but it needs to stabilize, too. If we knew then what we know now, and all that... > > But I'm certainly not doing any work. Do what you think is best. > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless enigma > ?that is made terrible by our own mad attempt to interpret it as though it had > ?an underlying truth." > ? -- Umberto Eco > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From jonovik at gmail.com Fri Aug 14 23:17:04 2009 From: jonovik at gmail.com (Jon Olav Vik) Date: Fri, 14 Aug 2009 21:17:04 +0000 (UTC) Subject: [Cython] Problems compiling C code that #include's my public Cython function References: Message-ID: Jon Olav Vik writes: Answering a few of my own questions, in case anyone else is interested: > Kurt Smith writes: > > Hi, Jon. I'm the 'maintaner' of fwrap, which is the new name of > > 'f2cy' (I clearly need to put an update post on the blog!). It's > > still in development, and will have basic functionality ready to go in > > a few weeks. I don't know if we'll have C (i.e. cdef) callbacks > > implemented by then, but it will be high on the priority list. It > > will certainly help you wrap a Fortran library in Cython, as you > > desire, although the api is similar to f2py where the wrapping is > > automated, > > Looking at http://www.scipy.org/F2py (it was down the last time I researched > this), it certainly looks promising. What is it that fwrap offers that f2py > doesn't? Direct access from C(ython), rather than the overhead of a Python > extension module? Nice overview at http://conference.scipy.org/abstract?id=19 > Is there a separate mailing list for f2py? http://news.gmane.org/gmane.comp.python.f2py.user > How would I pass a function pointer (callback) to the f2py-generated > extension module? Chapters 5 and 9 of this book have detailed examples: http://www.amazon.com/Python-Scripting-Computational-Science-Engineering/ dp/3540739157/ From chrisde88 at yahoo.de Mon Aug 17 05:44:32 2009 From: chrisde88 at yahoo.de (Christian) Date: Mon, 17 Aug 2009 05:44:32 +0200 Subject: [Cython] Import time from time.h Message-ID: <4A88D220.50206@yahoo.de> Hello, I am trying to import the time function from time.h as following: cdef extern from "time.h" nogil: ctypedef long time_t time_t time(time_t*) I am using MinGW as compiler and the time function is defined in time.h as following: #ifndef _TIME_T_DEFINED typedef long time_t; #define _TIME_T_DEFINED #endif _CRTIMP clock_t __cdecl __MINGW_NOTHROW clock (void); _CRTIMP time_t __cdecl __MINGW_NOTHROW time (time_t*); _CRTIMP double __cdecl __MINGW_NOTHROW difftime (time_t, time_t); _CRTIMP time_t __cdecl __MINGW_NOTHROW mktime (struct tm*); but when I try to call the function: print time(0) I get the following error: ImportError: DLL load failed: The specified procedure could not be found. What is the problem? Thank you in advance. Christian From sccolbert at gmail.com Mon Aug 17 09:46:04 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Mon, 17 Aug 2009 03:46:04 -0400 Subject: [Cython] Import time from time.h In-Reply-To: <4A88D220.50206@yahoo.de> References: <4A88D220.50206@yahoo.de> Message-ID: <7f014ea60908170046n2c36cb75ked416a66af18e59f@mail.gmail.com> did you link to libc? (i think that's where time.h is implemented) On 8/16/09, Christian wrote: > Hello, > > I am trying to import the time function from time.h as following: > > cdef extern from "time.h" nogil: > ctypedef long time_t > time_t time(time_t*) > > > I am using MinGW as compiler and the time function is defined in time.h > as following: > > #ifndef _TIME_T_DEFINED > typedef long time_t; > #define _TIME_T_DEFINED > #endif > > _CRTIMP clock_t __cdecl __MINGW_NOTHROW clock (void); > _CRTIMP time_t __cdecl __MINGW_NOTHROW time (time_t*); > _CRTIMP double __cdecl __MINGW_NOTHROW difftime (time_t, time_t); > _CRTIMP time_t __cdecl __MINGW_NOTHROW mktime (struct tm*); > > > but when I try to call the function: > > print time(0) > > > I get the following error: > > ImportError: DLL load failed: The specified procedure could not be found. > > > > What is the problem? > > > > > Thank you in advance. > > Christian > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From chrisde88 at yahoo.de Mon Aug 17 18:01:24 2009 From: chrisde88 at yahoo.de (Christian) Date: Mon, 17 Aug 2009 18:01:24 +0200 Subject: [Cython] Import time from time.h In-Reply-To: <7f014ea60908170046n2c36cb75ked416a66af18e59f@mail.gmail.com> References: <4A88D220.50206@yahoo.de> <7f014ea60908170046n2c36cb75ked416a66af18e59f@mail.gmail.com> Message-ID: <4A897ED4.9050005@yahoo.de> No, how do I link against libc? I added libraries=['libc'] to my setup.py file, but then dllwrap couldn't find llibc. Or do I have to add a special library path? Chris Colbert wrote: > did you link to libc? (i think that's where time.h is implemented) > > From robertwb at math.washington.edu Mon Aug 17 18:15:50 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 17 Aug 2009 09:15:50 -0700 Subject: [Cython] Import time from time.h In-Reply-To: <7f014ea60908170046n2c36cb75ked416a66af18e59f@mail.gmail.com> References: <4A88D220.50206@yahoo.de> <7f014ea60908170046n2c36cb75ked416a66af18e59f@mail.gmail.com> Message-ID: On Aug 17, 2009, at 12:46 AM, Chris Colbert wrote: > did you link to libc? (i think that's where time.h is implemented) I know nothing about linking on Windows, but I'll second that this sounds like a linking issue. You might be able to google around for how people link with distutils+MinGW without using Cython, and if you find something that works please post here. > > On 8/16/09, Christian wrote: >> Hello, >> >> I am trying to import the time function from time.h as following: >> >> cdef extern from "time.h" nogil: >> ctypedef long time_t >> time_t time(time_t*) >> >> >> I am using MinGW as compiler and the time function is defined in >> time.h >> as following: >> >> #ifndef _TIME_T_DEFINED >> typedef long time_t; >> #define _TIME_T_DEFINED >> #endif >> >> _CRTIMP clock_t __cdecl __MINGW_NOTHROW clock (void); >> _CRTIMP time_t __cdecl __MINGW_NOTHROW time (time_t*); >> _CRTIMP double __cdecl __MINGW_NOTHROW difftime (time_t, time_t); >> _CRTIMP time_t __cdecl __MINGW_NOTHROW mktime (struct tm*); >> >> >> but when I try to call the function: >> >> print time(0) >> >> >> I get the following error: >> >> ImportError: DLL load failed: The specified procedure could not be >> found. >> >> >> >> What is the problem? >> >> >> >> >> Thank you in advance. >> >> Christian >> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From chrisde88 at yahoo.de Mon Aug 17 19:31:14 2009 From: chrisde88 at yahoo.de (Christian) Date: Mon, 17 Aug 2009 19:31:14 +0200 Subject: [Cython] Import time from time.h In-Reply-To: References: <4A88D220.50206@yahoo.de> <7f014ea60908170046n2c36cb75ked416a66af18e59f@mail.gmail.com> Message-ID: <4A8993E2.9090400@yahoo.de> There seems to be a serious problem with MinGW under Windows and with versioned MSVCR.dll's as you can read at: http://bugs.python.org/issue3308 A workaround would be: cdef extern from "time.h" nogil: ctypedef long __time64_t __time64_t _time64(__time64_t*) If someone has a another idea to this topic I would appreciate it. Christian From plotnikoff at gmail.com Tue Aug 18 12:27:34 2009 From: plotnikoff at gmail.com (Andrey Plotnikov) Date: Tue, 18 Aug 2009 18:27:34 +0800 Subject: [Cython] PATCH: code object support in exec statement Message-ID: <9a3577710908180327k6e8d8887he441a48032153976@mail.gmail.com> Is it right place for posting patches? diff -r adbef02d669f Cython/Compiler/Builtin.py --- a/Cython/Compiler/Builtin.py Sat Aug 08 20:58:14 2009 +0200 +++ b/Cython/Compiler/Builtin.py Tue Aug 18 18:19:08 2009 +0800 @@ -182,27 +182,34 @@ globals = locals; } - if (PyUnicode_Check(o)) { - s = PyUnicode_AsUTF8String(o); - if (!s) goto bad; - o = s; - #if PY_MAJOR_VERSION >= 3 - } else if (!PyBytes_Check(o)) { - #else - } else if (!PyString_Check(o)) { - #endif - /* FIXME: support file objects and code objects */ - PyErr_SetString(PyExc_TypeError, - "exec currently requires a string as code input."); - goto bad; + if (PyDict_GetItemString(globals, "__builtins__") == NULL) { + PyDict_SetItemString(globals, "__builtins__", PyEval_GetBuiltins()); } - - #if PY_MAJOR_VERSION >= 3 - code = PyBytes_AS_STRING(o); - #else - code = PyString_AS_STRING(o); - #endif - result = PyRun_String(code, Py_file_input, globals, locals); + + if (PyCode_Check(o)) { + result = PyEval_EvalCode((PyCodeObject *)o, globals, locals); + } + else { + if (PyUnicode_Check(o)) { + s = PyUnicode_AsUTF8String(o); + if (!s) goto bad; + o = s; + #if PY_MAJOR_VERSION >= 3 + } else if (!PyBytes_Check(o)) { + #else + } else if (!PyString_Check(o)) { + #endif + PyErr_SetString(PyExc_TypeError, + "exec: arg 1 must be string or code object"); + goto bad; + } + #if PY_MAJOR_VERSION >= 3 + code = PyBytes_AS_STRING(o); + #else + code = PyString_AS_STRING(o); + #endif + result = PyRun_String(code, Py_file_input, globals, locals); + } Py_XDECREF(s); return result; -- Andrey Plotnikov From dalcinl at gmail.com Tue Aug 18 15:14:06 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 18 Aug 2009 10:14:06 -0300 Subject: [Cython] PATCH: code object support in exec statement In-Reply-To: <9a3577710908180327k6e8d8887he441a48032153976@mail.gmail.com> References: <9a3577710908180327k6e8d8887he441a48032153976@mail.gmail.com> Message-ID: Perhaps the error message should be this one (taken from Python 3)?: "exec: arg 1 must be string, bytes or code object" On Tue, Aug 18, 2009 at 7:27 AM, Andrey Plotnikov wrote: > Is it right place for posting patches? > > diff -r adbef02d669f Cython/Compiler/Builtin.py > --- a/Cython/Compiler/Builtin.py ? ? ? ?Sat Aug 08 20:58:14 2009 +0200 > +++ b/Cython/Compiler/Builtin.py ? ? ? ?Tue Aug 18 18:19:08 2009 +0800 > @@ -182,27 +182,34 @@ > ? ? ? ? globals = locals; > ? ? } > > - ? ?if (PyUnicode_Check(o)) { > - ? ? ? ?s = PyUnicode_AsUTF8String(o); > - ? ? ? ?if (!s) goto bad; > - ? ? ? ?o = s; > - ? ?#if PY_MAJOR_VERSION >= 3 > - ? ?} else if (!PyBytes_Check(o)) { > - ? ?#else > - ? ?} else if (!PyString_Check(o)) { > - ? ?#endif > - ? ? ? ?/* FIXME: support file objects and code objects */ > - ? ? ? ?PyErr_SetString(PyExc_TypeError, > - ? ? ? ? ? ?"exec currently requires a string as code input."); > - ? ? ? ?goto bad; > + ? ?if (PyDict_GetItemString(globals, "__builtins__") == NULL) { > + ? ? ? PyDict_SetItemString(globals, "__builtins__", PyEval_GetBuiltins()); > ? ? } > - > - ? ?#if PY_MAJOR_VERSION >= 3 > - ? ?code = PyBytes_AS_STRING(o); > - ? ?#else > - ? ?code = PyString_AS_STRING(o); > - ? ?#endif > - ? ?result = PyRun_String(code, Py_file_input, globals, locals); > + > + ? ?if (PyCode_Check(o)) { > + ? ? ? result = PyEval_EvalCode((PyCodeObject *)o, globals, locals); > + ? ?} > + ? ?else { > + ? ? ? if (PyUnicode_Check(o)) { > + ? ? ? ? ? s = PyUnicode_AsUTF8String(o); > + ? ? ? ? ? if (!s) goto bad; > + ? ? ? ? ? o = s; > + ? ? ? #if PY_MAJOR_VERSION >= 3 > + ? ? ? } else if (!PyBytes_Check(o)) { > + ? ? ? #else > + ? ? ? } else if (!PyString_Check(o)) { > + ? ? ? #endif > + ? ? ? ? ? PyErr_SetString(PyExc_TypeError, > + ? ? ? ? ? ? ? "exec: arg 1 must be string or code object"); > + ? ? ? ? ? goto bad; > + ? ? ? } > + ? ? ? #if PY_MAJOR_VERSION >= 3 > + ? ? ? code = PyBytes_AS_STRING(o); > + ? ? ? #else > + ? ? ? code = PyString_AS_STRING(o); > + ? ? ? #endif > + ? ? ? result = PyRun_String(code, Py_file_input, globals, locals); > + ? ?} > > ? ? Py_XDECREF(s); > ? ? return result; > > -- > Andrey Plotnikov > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From johan.gronqvist at gmail.com Tue Aug 18 16:02:56 2009 From: johan.gronqvist at gmail.com (=?ISO-8859-1?Q?Johan_Gr=F6nqvist?=) Date: Tue, 18 Aug 2009 16:02:56 +0200 Subject: [Cython] Question: Extension type wrapping a cdef struct Message-ID: Hi, I am trying to define a c struct and wrap it in a python object, but fail to make cython accept my .pyx files. My intention is to define c structs that are passed to cdefs (for more efficient function calls) and wrap them in extension types that allow for greater flexibility and higher level programming. I use three files: vector.pxd and vector.pyx define the cdef struct and wrapper extension type vector_test.pyx defines a variable whose type is the extension type, extracts the c struct, and attempts an operation on it. My files contain: File vector.pxd: ================ cdef struct vector_c: double x, y, z cdef class vector: cdef: vector_c data vector_c get_data(self) ================ File vector.pyx ================ cdef class vector: def __init__(self, double x, double y, double z): self.data.x = x self.data.y = y self.data.z = z cdef vector_c get_data(vector self): return self.data def __getitem__(self, ix): if ix==0: return self.data.x elif ix == 1: return self.data.y elif ix == 2: return self.data.z else: raise "Error" ============================== File vector_test.pyx ============================== import vector cimport vector v = vector.vector(1.0, 1.0, 0.0) cdef: vector.vector_c w = (v.get_data()) double x = v[0] double xx = w.x print x print xx ============================== The first two files can be converted to c-code by issuing "cython vector.pyx". When running "cython vector_test.pyx", I get the error message ============================================= Error converting Pyrex file to C: ------------------------------------------------------------ ... import vector cimport vector v = vector.vector(1.0, 1.0, 0.0) cdef: vector.vector_c w = (v.get_data()) ^ ------------------------------------------------------------ /home/johan/Work/Elasticity/Program/Python/vector_test.pyx:6:35: Cannot convert Python object to 'vector_c' ============================================= I am surprised by this message, as I think that no conversion should be necessary. When I look at the previously generated vector.c I see that the function get_data should return a pure c struct and not a PyObject. Grateful for any help. / johan From fuzzy.wombat.iii at gmail.com Tue Aug 18 20:41:17 2009 From: fuzzy.wombat.iii at gmail.com (Fuzzy Wombat) Date: Tue, 18 Aug 2009 13:41:17 -0500 Subject: [Cython] restrict keyword support Message-ID: <6c25fd500908181141se368dc5u318e567d2941922a@mail.gmail.com> I'm attempting to write some bindings to the functions in regex.h (I know only applicable on POSIX systems) and I'm running across an issue, that not withstanding I think my problem is the 'restrict' keyword, which I cannot find any documentation on as far as cython's usage. My question then would be: Is this supported? and if so, what is the appropriate syntax for using it? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090818/4b8d4cda/attachment.htm From robertwb at math.washington.edu Tue Aug 18 21:57:25 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 18 Aug 2009 12:57:25 -0700 (PDT) Subject: [Cython] restrict keyword support In-Reply-To: <6c25fd500908181141se368dc5u318e567d2941922a@mail.gmail.com> References: <6c25fd500908181141se368dc5u318e567d2941922a@mail.gmail.com> Message-ID: On Tue, 18 Aug 2009, Fuzzy Wombat wrote: > I'm attempting to write some bindings to the functions in regex.h (I know > only applicable on POSIX systems) > and I'm running across an issue, that not withstanding I think my problem is > the 'restrict' keyword, which I cannot > find any documentation on as far as cython's usage. My question then would > be: Is this supported? and if so, what > is the appropriate syntax for using it? No, Cython does not know about the restrict keyword. I bet using the same tricks one does for const would work here as well, if you need it. - Robert From jonovik at gmail.com Tue Aug 18 22:10:07 2009 From: jonovik at gmail.com (Jon Olav Vik) Date: Tue, 18 Aug 2009 22:10:07 +0200 Subject: [Cython] [Resolved] Problems compiling C code that #include's my public Cython function References: Message-ID: On Fri, 14 Aug 2009 23:17:04 +0200, Jon Olav Vik wrote: > Jon Olav Vik writes: > > Answering a few of my own questions, in case anyone else is interested: > >> Kurt Smith writes: >> > Hi, Jon. I'm the 'maintaner' of fwrap, which is the new name of >> > 'f2cy' (I clearly need to put an update post on the blog!). It's >> > still in development, and will have basic functionality ready to go in >> > a few weeks. I don't know if we'll have C (i.e. cdef) callbacks >> > implemented by then, but it will be high on the priority list. It >> > will certainly help you wrap a Fortran library in Cython, as you >> > desire, although the api is similar to f2py where the wrapping is >> > automated, >> >> Looking at http://www.scipy.org/F2py, it certainly looks promising. [...] > Chapters 5 and 9 of this book have detailed examples: > http://www.amazon.com/Python-Scripting-Computational-Science-Engineering/dp/3540739157/ To wrap this up (pardon the pun), f2py turned out to be a perfect fit in this case (expensive function evaluations). I spent a day on the examples in Langtangen's book, studying the f2py user guide, and some manual preprocessing of the source files to be wrapped. I wrote a dummy .f file to provide signatures for the functions to be differentiated, then autogenerated a .pyf header file with f2py. With a moderate amount of careful editing of the .pyf file, the wrapper ended up being very flexible and pythonic. The wrapper simplifies the interface by: * Allowing lists etc for vectors "x"; these are autoconverted to Numpy arrays. * Using Python callables for callback functions "f(x)". * Possibility of passing extra parameters to f(x, ...). * Hiding parameters for array dimensions (these are obtained from the Numpy arrays). * Providing defaults for technical parameters such as the order of accuracy. * Returning results as function values instead of using output arguments. The first four are done automatically by f2py; the remaining ones are done by tweaking individual lines of the .pyf header file. These convenience features save me a lot of effort compared to having a more low-level, true-to-Fortran wrapper. I lose a little speed because callbacks go through Python and because of argument checking/conversion, but that doesn't matter so much when the function evaluations themselves are expensive. (The functions in question are the differential equation simulations I posted about previously. Using Cython for the right-hand-side function gave a 10x speedup for the whole integration; 100x for the right-hand-side, but keeping the Pysundials wrapper which passes callbacks through Python. I gave up wrapping CVODE directly at this time; it ended up requiring too much manual work for my current level of (in)expertise.) Thanks to everyone who helped! Best regards, Jon Olav From robertwb at math.washington.edu Wed Aug 19 04:56:34 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 18 Aug 2009 19:56:34 -0700 Subject: [Cython] Question: Extension type wrapping a cdef struct In-Reply-To: References: Message-ID: <0B5F358B-4198-489F-88DA-05A750082790@math.washington.edu> On Aug 18, 2009, at 7:02 AM, Johan Gr?nqvist wrote: > Hi, > > I am trying to define a c struct and wrap it in a python object, but > fail to make cython accept my .pyx files. > > My intention is to define c structs that are passed to cdefs (for more > efficient function calls) and wrap them in extension types that allow > for greater flexibility and higher level programming. > > > > I use three files: > > vector.pxd and vector.pyx define the cdef struct and wrapper > extension type > > vector_test.pyx defines a variable whose type is the extension type, > extracts the c struct, and attempts an operation on it. > > My files contain: > > File vector.pxd: > ================ > cdef struct vector_c: > double x, y, z > > cdef class vector: > cdef: > vector_c data > vector_c get_data(self) > ================ > > File vector.pyx > ================ > cdef class vector: > def __init__(self, double x, double y, double z): > self.data.x = x > self.data.y = y > self.data.z = z > > cdef vector_c get_data(vector self): > return self.data > > def __getitem__(self, ix): > if ix==0: > return self.data.x > elif ix == 1: > return self.data.y > elif ix == 2: > return self.data.z > else: > raise "Error" > ============================== > > > File vector_test.pyx > ============================== > import vector > cimport vector > > v = vector.vector(1.0, 1.0, 0.0) # v is (implicitly) declared to be a Python object. You need to declare it to be a vector. > cdef: > vector.vector_c w = (v.get_data()) # v.get_data is treated as a Python function, because it thinks v is a Python object. > > double x = v[0] > double xx = w.x > > print x > print xx > ============================== > > > > The first two files can be converted to c-code by issuing "cython > vector.pyx". When running "cython vector_test.pyx", I get the error > message > > ============================================= > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > import vector > cimport vector > > v = vector.vector(1.0, 1.0, 0.0) > cdef: > vector.vector_c w = (v.get_data()) > ^ > ------------------------------------------------------------ > > /home/johan/Work/Elasticity/Program/Python/vector_test.pyx:6:35: > Cannot > convert Python object to 'vector_c' > ============================================= > > I am surprised by this message, as I think that no conversion > should be > necessary. When I look at the previously generated vector.c I see that > the function get_data should return a pure c struct and not a > PyObject. > > Grateful for any help. > > / johan > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From robertwb at math.washington.edu Wed Aug 19 05:03:08 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 18 Aug 2009 20:03:08 -0700 Subject: [Cython] PATCH: code object support in exec statement In-Reply-To: <9a3577710908180327k6e8d8887he441a48032153976@mail.gmail.com> References: <9a3577710908180327k6e8d8887he441a48032153976@mail.gmail.com> Message-ID: <4EAE15D3-339C-48E4-AAC6-DA1B3D138E4D@math.washington.edu> On Aug 18, 2009, at 3:27 AM, Andrey Plotnikov wrote: > Is it right place for posting patches? Yes, thanks. Even better is trac http://trac.cython.org/cython_trac/ (send an email too). Also, we prefer mercurial (or even git) patches, as they have authorship and a commit message, but we can definitely use this as it is too. - Robert From ondrej at certik.cz Wed Aug 19 08:19:16 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Tue, 18 Aug 2009 23:19:16 -0700 Subject: [Cython] PATCH: code object support in exec statement In-Reply-To: <4EAE15D3-339C-48E4-AAC6-DA1B3D138E4D@math.washington.edu> References: <9a3577710908180327k6e8d8887he441a48032153976@mail.gmail.com> <4EAE15D3-339C-48E4-AAC6-DA1B3D138E4D@math.washington.edu> Message-ID: <85b5c3130908182319p1ed06067le9f74aaa01aaa39a@mail.gmail.com> On Tue, Aug 18, 2009 at 8:03 PM, Robert Bradshaw wrote: > On Aug 18, 2009, at 3:27 AM, Andrey Plotnikov wrote: > >> Is it right place for posting patches? > > Yes, thanks. Even better is trac http://trac.cython.org/cython_trac/ > (send an email too). Also, we prefer mercurial (or even git) patches, > as they have authorship and a commit message, but we can definitely > use this as it is too. Cool, during my tutorial here at scipy09, I asked how many people use hg and how many git, and it seems it's about half and half --- and that's a python conference, so I assume lots of people use hg because it's in python. So git is growing, I like to see that. :) Ondrej From asmeurer at gmail.com Wed Aug 19 08:46:59 2009 From: asmeurer at gmail.com (Aaron S. Meurer) Date: Wed, 19 Aug 2009 00:46:59 -0600 Subject: [Cython] Using Cython with SymPy Message-ID: Hi. I am a developer for the project SymPy [0], which is a computer algebra system written in Python. We have been considering moving part or all of the code base into Cython for speed reasons. Ondrej Certik, the project's leader, recently gave a tutorial at SciPy09 (there should be a video up soon), which included a bit about increasing the speed of an operation by about 20x by using the cython.locals() decorator to convert some of the variables into cython ints. You can pull from his github account [1]. If you run: >>> from sympy import * >>> d = divisors(5*10**7) The function will run about 20x to 30x faster, as I mentioned. But if you run the ntheory test, which uses divisors, using $./bin/test sympy/ntheory it runs slower or at least as slow as without cython. We figured out that the reason is that when the multiple imports in the tests are called, they run about 2x slower with cython, because it calls the decorator each time. You can see this by running In [1]: %timeit from sympy import * with and without cython in IPython. Is there any way around this? This could be a show stopper for us, because the increased import times levels out or makes worse the total times for things. This is not exclusive to the test suite. It comes from multiple imports in each file. We know that we can use .pxd files, but that would require to handle both .pxd files and .py files instead of maintaining one file with everything. Aaron Meurer [0] - http://code.google.com/p/sympy/ [1] - http://github.com/certik/sympy/tree/cython From robertwb at math.washington.edu Wed Aug 19 09:32:13 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 19 Aug 2009 00:32:13 -0700 Subject: [Cython] Using Cython with SymPy In-Reply-To: References: Message-ID: On Aug 18, 2009, at 11:46 PM, Aaron S. Meurer wrote: > Hi. I am a developer for the project SymPy [0], which is a computer > algebra system written in Python. We have been considering moving > part or all of the code base into Cython for speed reasons. Ondrej > Certik, the project's leader, recently gave a tutorial at SciPy09 > (there should be a video up soon), which included a bit about > increasing the speed of an operation by about 20x by using the > cython.locals() decorator to convert some of the variables into cython > ints. You can pull from his github account [1]. If you run: > >>>> from sympy import * >>>> d = divisors(5*10**7) > > The function will run about 20x to 30x faster, as I mentioned. But if > you run the ntheory test, which uses divisors, using > > $./bin/test sympy/ntheory > > it runs slower or at least as slow as without cython. We figured out > that the reason is that when the multiple imports in the tests are > called, they run about 2x slower with cython, because it calls the > decorator each time. > > You can see this by running > > In [1]: %timeit from sympy import * > > with and without cython in IPython. > > Is there any way around this? This could be a show stopper for us, > because the increased import times levels out or makes worse the total > times for things. This is not exclusive to the test suite. It comes > from multiple imports in each file. > > We know that we can use .pxd files, but that would require to handle > both .pxd files and .py files instead of maintaining one file with > everything. Eventually I want to get type inference, which could help here. However, I'm not following how decorating these functions makes everything slower. Sure, it's a bit slower on the first import, but the decorator is no re-called on each import. Clearly I'm missing something here, could you explain why the decorator is being repeatedly called? In the worst case, maintaining separate .pxd files seems a small price to pay to get a 20-30x speedup. Also, if you want to have cdef classes, especially across various modules, you'll have to do that anyways right now. (Patches for fixing this, e.g. via class decorators, would be highly appreciated). - Robert ------------------------- foo.py ----------------------- def decorate(f): print "here" return f @decorate def f(x): return x ---------------------------------------------------------- sage: from foo import * here sage: from foo import * sage: from foo import * From robertwb at math.washington.edu Wed Aug 19 09:55:36 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 19 Aug 2009 00:55:36 -0700 Subject: [Cython] Checking for numpy arrays and None In-Reply-To: <4A82EE6D.8020904@noaa.gov> References: <4A7CC21E.10209@noaa.gov> <4A7CD402.1090003@canterbury.ac.nz> <4A8053B1.8010003@noaa.gov> <4A805699.5010402@student.matnat.uio.no> <4A805C15.8060206@noaa.gov> <4A80BF22.8020305@canterbury.ac.nz> <4A81A069.20205@student.matnat.uio.no> <4A820DE8.6000401@canterbury.ac.nz> <81676F1C-2B2B-4DC6-905E-9793BE93C789@math.washington.edu> <4A82E825.4000800@noaa.gov> <4A82EE6D.8020904@noaa.gov> Message-ID: <9FC2EA37-D027-4ECE-8AA4-E244DF4688E9@math.washington.edu> On Aug 12, 2009, at 9:04 AM, Christopher Barker wrote: > Robert Bradshaw wrote: >> The Cython philosophy is that one should be able to take a piece of >> Python code, compile it with Cython, maybe sprinkle in some static >> type declarations, and the behavior should be essentially the same > > I, for one, like this approach, which is why I was surprised by the > None > issue. > > If I write a python function that expects a numpy array, and I pass a > None into it, I'll get an exception. > > If I write a Cython function that expects a numpy array and declare > that > in the function signature, then pass a None it, I get a bus error (or > who knows what other ugly crash). The difference in behavior is that an exception is raised as soon as the function is entered, vs. somewhere down the line. Of course, a hard crash vs. a Python exception (at the wrong spot or not) is pretty glaring, but the plan is to change that. On Aug 12, 2009, at 9:31 AM, Christopher Barker wrote: > Christopher Barker wrote: > >> NOTE: it would be a good idea to add that to examples and >> tutorials, too. > > In the spirit of putting my time where my mouth is, I just updated the > numpy tutorial with "non None" information: > > http://wiki.cython.org/tutorials/numpy > > please edit, or roll back if you don't like it. Thanks! - Robert From robertwb at math.washington.edu Wed Aug 19 09:57:05 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 19 Aug 2009 00:57:05 -0700 Subject: [Cython] Question about packaging external fortran parser with Cython for fwrap In-Reply-To: References: <85e81ba30908131705w6f885e5fm678b759a974ebb69@mail.gmail.com> <2895B654-4992-4DB8-864F-0C62CCE3008E@math.washington.edu> <85b5c3130908132342u6c45ae5ev791d863e87988e99@mail.gmail.com> <4A8592E3.5050909@student.matnat.uio.no> Message-ID: <6BCB659E-F50B-40F0-8BB2-F58F5740E237@math.washington.edu> On Aug 14, 2009, at 12:57 PM, Kurt Smith wrote: > On Fri, Aug 14, 2009 at 2:17 PM, Robert Kern > wrote: >> On 2009-08-14 13:42 PM, Kurt Smith wrote: >>> On Fri, Aug 14, 2009 at 12:00 PM, Robert >>> Kern wrote: >>>> On 2009-08-14 11:37 AM, Dag Sverre Seljebotn wrote: >>>> >>>>> As for fparser, I think we can keep PSF out of it for now: >>>>> >>>>> fwrap depends on Cython and fparser, but Cython itself will >>>>> never depend >>>>> on either >>>>> >>>>> At least for the time being it's more a matter of shipping >>>>> fparser and >>>>> fwrap with the default Cython download (in the Tools dir). I can >>>>> definitely see Cython itself being included into Python's >>>>> stdlib without >>>>> fwrap following along (as nobody outside of science use Fortran >>>>> anyway). >>>> >>>> Hmmm. Are you sure you don't just want to distribute fwrap as a >>>> separate >>>> project, then? Personally, I will always prefer a second package >>>> over an >>>> optional component. If I had a project that used fwrap to build, >>>> I would want my >>>> prerequisite installation instructions to be this: >>>> >>>> 1. Install Cython. http://pypi.python.org/pypi/Cython >>>> 2. Install fwrap. http://pypi.python.org/pypi/fwrap >>>> >>>> rather than this: >>>> >>>> 1. Download the source tarball of Cython. http:// >>>> pypi.python.org/pypi/Cython >>>> 2. Install Cython using "python setup.py install" >>>> 3. cd Tools/fwrap/ >>>> 4. Install fwrap using "python setup.py install" >>>> >>>> With the latter, there will inevitably be some Linux >>>> distribution that will put >>>> both Cython and fwrap into their python-cython package and some >>>> who won't. >>> >>> Good points. Dag and I talked about packaging fwrap separately once >>> it stabilizes. Perhaps we should just make it its own package from >>> the start. This would alleviate the licensing issues and 'core >>> Cython' could be pure Apache (presuming Pearu doesn't want to >>> relicense fparser for fwrap under Apache). >>> >>> Thoughts? >> >> I would think that it would be better to start off separate while >> it stabilizes. >> If you don't intend for a Cython release to contain it, why bother >> putting it in >> the Cython repo in the first place? Mercurial repos are cheap. :-) > > This aspect of things (where fwrap belongs) has been evolving over the > summer, as has fwrap's 'intent'. Is it primarily a tool to use with > Cython, allowing one to wrap external Fortran code along with C, as > initially intended? Or is it its own self-standing thing, like an > updated f2py? fwrap is progressing towards the latter, and this > thread is good to thresh out the best solution. > > fparser has been a tremendous help, but it needs to stabilize, too. > If we knew then what we know now, and all that... Perhaps the question would be better answered if I had a better idea of what, exactly, fwrap was. I assume you have slides from your scipy talk? - Robert From stefan_ml at behnel.de Wed Aug 19 10:25:46 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 19 Aug 2009 10:25:46 +0200 Subject: [Cython] Using Cython with SymPy In-Reply-To: References: Message-ID: <4A8BB70A.6040607@behnel.de> Robert Bradshaw wrote: > However, I'm not following how decorating these functions makes > everything slower. Sure, it's a bit slower on the first import, but > the decorator is no re-called on each import. Clearly I'm missing > something here, could you explain why the decorator is being > repeatedly called? It shouldn't, at least for Cython 0.11 code (without inner functions). Decorators are evaluated at import time, so as long as the module isn't unloaded (which doesn't really work for extension modules in Py2 anyway), they are evaluated only once, and when compiled, the Cython builtins are actually evaluated at compile time. > Also, if you want to have cdef > classes, especially across various modules, you'll have to do that > anyways right now. (Patches for fixing this, e.g. via class > decorators, would be highly appreciated). Given that Cython supports class decorators (ok, not in a release version, but that will obviously change one day ;), I think the main obstacle here is that Py2 doesn't support them, so there's not that much to gain from using them in pure Python mode - unless you specifically target Cython and Py3.x only. Stefan From robertwb at math.washington.edu Wed Aug 19 10:35:24 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 19 Aug 2009 01:35:24 -0700 Subject: [Cython] Using Cython with SymPy In-Reply-To: <4A8BB70A.6040607@behnel.de> References: <4A8BB70A.6040607@behnel.de> Message-ID: <6C122BF7-63E8-4757-B249-C395B9C43D3D@math.washington.edu> On Aug 19, 2009, at 1:25 AM, Stefan Behnel wrote: > Robert Bradshaw wrote: >> However, I'm not following how decorating these functions makes >> everything slower. Sure, it's a bit slower on the first import, but >> the decorator is no re-called on each import. Clearly I'm missing >> something here, could you explain why the decorator is being >> repeatedly called? > > It shouldn't, at least for Cython 0.11 code (without inner functions). > Decorators are evaluated at import time, so as long as the module > isn't > unloaded (which doesn't really work for extension modules in Py2 > anyway), > they are evaluated only once, and when compiled, the Cython > builtins are > actually evaluated at compile time. > > >> Also, if you want to have cdef >> classes, especially across various modules, you'll have to do that >> anyways right now. (Patches for fixing this, e.g. via class >> decorators, would be highly appreciated). > > Given that Cython supports class decorators (ok, not in a release > version, > but that will obviously change one day ;), I think the main > obstacle here > is that Py2 doesn't support them, so there's not that much to gain > from > using them in pure Python mode - unless you specifically target > Cython and > Py3.x only. I believe that 2.6 has them as well, but it is still a limited audience. No reason not to support it though... Perhaps we could also support cdef class A: a = cython.declare(int) (FYI, this is an alternative to the @cython.locals, but of course in a function would get run every time.) - Robert From stefan_ml at behnel.de Wed Aug 19 12:57:42 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 19 Aug 2009 12:57:42 +0200 Subject: [Cython] Using Cython with SymPy In-Reply-To: <6C122BF7-63E8-4757-B249-C395B9C43D3D@math.washington.edu> References: <4A8BB70A.6040607@behnel.de> <6C122BF7-63E8-4757-B249-C395B9C43D3D@math.washington.edu> Message-ID: <4A8BDAA6.1020202@behnel.de> Robert Bradshaw wrote: > On Aug 19, 2009, at 1:25 AM, Stefan Behnel wrote: >> I think the main obstacle here >> is that Py2 doesn't support them, so there's not that much to gain >> from using them in pure Python mode - unless you specifically target >> Cython and Py3.x only. > > I believe that 2.6 has them as well Interesting. I didn't notice they backported this feature. > but it is still a limited > audience. No reason not to support it though... Sure. Note that cdef classes don't currently support them, but a) that's not a problem for pure Python mode and b) it would be easy to enable cdef class decorators for Cython builtin declarations. Now that I write that, I actually think that the current test inside the parser is too early to catch the pure Python mode anyway, so maybe it actually just works. ;) > Perhaps we could also support > > cdef class A: > a = cython.declare(int) Fine with me - I thought this was already supported? Stefan From dalcinl at gmail.com Wed Aug 19 15:29:06 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 19 Aug 2009 10:29:06 -0300 Subject: [Cython] PATCH: code object support in exec statement In-Reply-To: <85b5c3130908182319p1ed06067le9f74aaa01aaa39a@mail.gmail.com> References: <9a3577710908180327k6e8d8887he441a48032153976@mail.gmail.com> <4EAE15D3-339C-48E4-AAC6-DA1B3D138E4D@math.washington.edu> <85b5c3130908182319p1ed06067le9f74aaa01aaa39a@mail.gmail.com> Message-ID: On Wed, Aug 19, 2009 at 3:19 AM, Ondrej Certik wrote: > > Cool, during my tutorial here at scipy09, I asked how many people use > hg and how many git, and it seems it's about half and half --- and > that's a python conference, so I assume lots of people use hg because > it's in python. So git is growing, I like to see that. :) > Well, I do not think that my preference for hg is related to it being written in Python. I would say that is more or less like the preference that I have for Python comparing to Perl ;-). Of course, the workflow of my projects is extremely simple, I'm basically the only one that make all the commits. Perhaps for larger projects, git is better... I do not know... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From ondrej at certik.cz Wed Aug 19 16:49:31 2009 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 19 Aug 2009 07:49:31 -0700 Subject: [Cython] PATCH: code object support in exec statement In-Reply-To: References: <9a3577710908180327k6e8d8887he441a48032153976@mail.gmail.com> <4EAE15D3-339C-48E4-AAC6-DA1B3D138E4D@math.washington.edu> <85b5c3130908182319p1ed06067le9f74aaa01aaa39a@mail.gmail.com> Message-ID: <85b5c3130908190749x53b9ac5n4a6264413505d28b@mail.gmail.com> On Wed, Aug 19, 2009 at 6:29 AM, Lisandro Dalcin wrote: > On Wed, Aug 19, 2009 at 3:19 AM, Ondrej Certik wrote: >> >> Cool, during my tutorial here at scipy09, I asked how many people use >> hg and how many git, and it seems it's about half and half --- and >> that's a python conference, so I assume lots of people use hg because >> it's in python. So git is growing, I like to see that. :) >> > > Well, I do not think that my preference for hg is related to it being > written in Python. I would say that is more or less like the > preference that I have for Python comparing to Perl ;-). Of course, > the workflow of my projects is extremely simple, I'm basically the > only one that make all the commits. Perhaps for larger projects, git > is better... I do not know... Yes, I think you summed it up more nicely why python people are more inclined to hg. I also used to think that git is like perl (=bad), but in fact it's pretty cool. Anyway, this is offtopic, sorry for that. Ondrej From stefan_ml at behnel.de Wed Aug 19 16:56:59 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 19 Aug 2009 16:56:59 +0200 Subject: [Cython] PATCH: code object support in exec statement In-Reply-To: <9a3577710908180327k6e8d8887he441a48032153976@mail.gmail.com> References: <9a3577710908180327k6e8d8887he441a48032153976@mail.gmail.com> Message-ID: <4A8C12BB.7000900@behnel.de> Andrey Plotnikov wrote: > Is it right place for posting patches? Thanks for the patch. I reworked the overall implementation a little and committed it to cython-devel. Stefan From plotnikoff at gmail.com Wed Aug 19 18:09:06 2009 From: plotnikoff at gmail.com (Andrey Plotnikov) Date: Thu, 20 Aug 2009 00:09:06 +0800 Subject: [Cython] PATCH: code object support in exec statement In-Reply-To: <4A8C12BB.7000900@behnel.de> References: <9a3577710908180327k6e8d8887he441a48032153976@mail.gmail.com> <4A8C12BB.7000900@behnel.de> Message-ID: <9a3577710908190909r567a531ek703eb3f24b56fb6d@mail.gmail.com> > Thanks for the patch. I reworked the overall implementation a little and > committed it to cython-devel. Thank you!. It's mandatory functionality for my project which uses cython. -- Andrey Plotnikov From robertwb at math.washington.edu Wed Aug 19 18:29:44 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 19 Aug 2009 09:29:44 -0700 Subject: [Cython] Using Cython with SymPy In-Reply-To: <4A8BDAA6.1020202@behnel.de> References: <4A8BB70A.6040607@behnel.de> <6C122BF7-63E8-4757-B249-C395B9C43D3D@math.washington.edu> <4A8BDAA6.1020202@behnel.de> Message-ID: <1A0844F4-F3F0-4A05-9CDF-4927F6B8BC28@math.washington.edu> On Aug 19, 2009, at 3:57 AM, Stefan Behnel wrote: >> Perhaps we could also support >> >> cdef class A: >> a = cython.declare(int) > > Fine with me - I thought this was already supported? Yes, we do support that. What I meant to say was that if we do class A: a = cython.declare(int) it automatically becomes a cdef class. Same if any (decorator- indicated) cdef methods are defined. - Robert From robertwb at math.washington.edu Wed Aug 19 18:33:33 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 19 Aug 2009 09:33:33 -0700 Subject: [Cython] PATCH: code object support in exec statement In-Reply-To: <85b5c3130908190749x53b9ac5n4a6264413505d28b@mail.gmail.com> References: <9a3577710908180327k6e8d8887he441a48032153976@mail.gmail.com> <4EAE15D3-339C-48E4-AAC6-DA1B3D138E4D@math.washington.edu> <85b5c3130908182319p1ed06067le9f74aaa01aaa39a@mail.gmail.com> <85b5c3130908190749x53b9ac5n4a6264413505d28b@mail.gmail.com> Message-ID: On Aug 19, 2009, at 7:49 AM, Ondrej Certik wrote: > On Wed, Aug 19, 2009 at 6:29 AM, Lisandro Dalcin > wrote: >> On Wed, Aug 19, 2009 at 3:19 AM, Ondrej Certik >> wrote: >>> >>> Cool, during my tutorial here at scipy09, I asked how many people >>> use >>> hg and how many git, and it seems it's about half and half --- and >>> that's a python conference, so I assume lots of people use hg >>> because >>> it's in python. So git is growing, I like to see that. :) >>> >> >> Well, I do not think that my preference for hg is related to it being >> written in Python. I would say that is more or less like the >> preference that I have for Python comparing to Perl ;-). Of course, >> the workflow of my projects is extremely simple, I'm basically the >> only one that make all the commits. Perhaps for larger projects, git >> is better... I do not know... > > Yes, I think you summed it up more nicely why python people are more > inclined to hg. > > I also used to think that git is like perl (=bad), but in fact it's > pretty cool. Anyway, this is offtopic, sorry for that. Yes, I use git for some stuff too, though I also prefer hg. However, as long as git+hg takes over from cvs+svn, things are moving in the right direction... - Robert From fuzzy.wombat.iii at gmail.com Wed Aug 19 19:56:39 2009 From: fuzzy.wombat.iii at gmail.com (Fuzzy Wombat) Date: Wed, 19 Aug 2009 12:56:39 -0500 Subject: [Cython] restrict keyword support In-Reply-To: References: <6c25fd500908181141se368dc5u318e567d2941922a@mail.gmail.com> Message-ID: <6c25fd500908191056s55652eb7ob975475cf8270e07@mail.gmail.com> On Tue, Aug 18, 2009 at 2:57 PM, Robert Bradshaw < robertwb at math.washington.edu> wrote: > On Tue, 18 Aug 2009, Fuzzy Wombat wrote: > > > I'm attempting to write some bindings to the functions in regex.h (I know > > only applicable on POSIX systems) > > and I'm running across an issue, that not withstanding I think my problem > is > > the 'restrict' keyword, which I cannot > > find any documentation on as far as cython's usage. My question then > would > > be: Is this supported? and if so, what > > is the appropriate syntax for using it? > > No, Cython does not know about the restrict keyword. I bet using the same > tricks one does for const would work here as well, if you need it. > > - Robert I had thought that originally, but could not come up with an incarnation of it that would work for restrict qualifiers. In the end I did solve the problem, I wrote a pure C function that did the work for me and took types that cython supported, and wrapped that with cython, then linked them both together to make my module. It worked with very little code and effort, however may not be the cleanest of solutions. If interested I can post my code somewhere for people to see. Mike "Fuzzy" Partin -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090819/829aa37c/attachment.htm From robertwb at math.washington.edu Wed Aug 19 20:01:29 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 19 Aug 2009 11:01:29 -0700 (PDT) Subject: [Cython] restrict keyword support In-Reply-To: <6c25fd500908191056s55652eb7ob975475cf8270e07@mail.gmail.com> References: <6c25fd500908181141se368dc5u318e567d2941922a@mail.gmail.com> <6c25fd500908191056s55652eb7ob975475cf8270e07@mail.gmail.com> Message-ID: On Wed, 19 Aug 2009, Fuzzy Wombat wrote: > On Tue, Aug 18, 2009 at 2:57 PM, Robert Bradshaw < > robertwb at math.washington.edu> wrote: > >> On Tue, 18 Aug 2009, Fuzzy Wombat wrote: >> >>> I'm attempting to write some bindings to the functions in regex.h (I know >>> only applicable on POSIX systems) >>> and I'm running across an issue, that not withstanding I think my problem >> is >>> the 'restrict' keyword, which I cannot >>> find any documentation on as far as cython's usage. My question then >> would >>> be: Is this supported? and if so, what >>> is the appropriate syntax for using it? >> >> No, Cython does not know about the restrict keyword. I bet using the same >> tricks one does for const would work here as well, if you need it. >> >> - Robert > > > I had thought that originally, but could not come up with an incarnation of > it that would work for restrict qualifiers. Hmm... I actually had never heard about the restrict keyword before this email so I don't know much about it (though I can see how it could be very useful). > In the end I did solve the problem, I wrote a pure C function > that did the work for me and took > types that cython supported, and wrapped that with cython, then linked them > both together to make my module. Yep, that'd work. > It worked with very little code and effort, however may not be the > cleanest of solutions. If interested > I can post my code somewhere for people to see. Could you put it in the FAQ on the wiki? http://wiki.cython.org/ - Robert From fuzzy.wombat.iii at gmail.com Wed Aug 19 20:25:54 2009 From: fuzzy.wombat.iii at gmail.com (Fuzzy Wombat) Date: Wed, 19 Aug 2009 13:25:54 -0500 Subject: [Cython] restrict keyword support In-Reply-To: References: <6c25fd500908181141se368dc5u318e567d2941922a@mail.gmail.com> <6c25fd500908191056s55652eb7ob975475cf8270e07@mail.gmail.com> Message-ID: <6c25fd500908191125v56618eb4m66b4f890336d9983@mail.gmail.com> > > > It worked with very little code and effort, however may not be the > > cleanest of solutions. If interested > > I can post my code somewhere for people to see. > > Could you put it in the FAQ on the wiki? http://wiki.cython.org/ > Done and done. http://wiki.cython.org/FAQ#HowtowrapCcodethatusestherestrictqualifier.3F -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090819/d163102e/attachment.htm From asmeurer at gmail.com Wed Aug 19 20:53:00 2009 From: asmeurer at gmail.com (Aaron S. Meurer) Date: Wed, 19 Aug 2009 12:53:00 -0600 Subject: [Cython] Using Cython with SymPy In-Reply-To: References: Message-ID: <52CDFB26-0817-493D-8347-DF85E9EF4FEE@gmail.com> Well, I don't know for a fact that this is truly what is happening. It is just Ondrej's insight, plus the fact that when I comment out the decorator, the import time is the same as it is in the master branch without Cythonization, but when it is there, it is twice as slow for repeated imports. Clearly something is slowing it down because the tests run a little slower. Maybe Cythonizing more code would produce speed increases that would overcome any slowdowns from imports, or maybe it would level out. I haven't tried much. Ondrej is the one who is doing most of this, so he will probably know more here. Aaron Meurer On Aug 19, 2009, at 1:32 AM, Robert Bradshaw wrote: > On Aug 18, 2009, at 11:46 PM, Aaron S. Meurer wrote: > >> Hi. I am a developer for the project SymPy [0], which is a computer >> algebra system written in Python. We have been considering moving >> part or all of the code base into Cython for speed reasons. Ondrej >> Certik, the project's leader, recently gave a tutorial at SciPy09 >> (there should be a video up soon), which included a bit about >> increasing the speed of an operation by about 20x by using the >> cython.locals() decorator to convert some of the variables into >> cython >> ints. You can pull from his github account [1]. If you run: >> >>>>> from sympy import * >>>>> d = divisors(5*10**7) >> >> The function will run about 20x to 30x faster, as I mentioned. But >> if >> you run the ntheory test, which uses divisors, using >> >> $./bin/test sympy/ntheory >> >> it runs slower or at least as slow as without cython. We figured out >> that the reason is that when the multiple imports in the tests are >> called, they run about 2x slower with cython, because it calls the >> decorator each time. >> >> You can see this by running >> >> In [1]: %timeit from sympy import * >> >> with and without cython in IPython. >> >> Is there any way around this? This could be a show stopper for us, >> because the increased import times levels out or makes worse the >> total >> times for things. This is not exclusive to the test suite. It comes >> from multiple imports in each file. >> >> We know that we can use .pxd files, but that would require to handle >> both .pxd files and .py files instead of maintaining one file with >> everything. > > Eventually I want to get type inference, which could help here. > However, I'm not following how decorating these functions makes > everything slower. Sure, it's a bit slower on the first import, but > the decorator is no re-called on each import. Clearly I'm missing > something here, could you explain why the decorator is being > repeatedly called? > > In the worst case, maintaining separate .pxd files seems a small > price to pay to get a 20-30x speedup. Also, if you want to have cdef > classes, especially across various modules, you'll have to do that > anyways right now. (Patches for fixing this, e.g. via class > decorators, would be highly appreciated). > > - Robert > > > ------------------------- foo.py ----------------------- > > def decorate(f): > print "here" > return f > > @decorate > def f(x): > return x > > ---------------------------------------------------------- > > sage: from foo import * > here > sage: from foo import * > sage: from foo import * > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From stefan_ml at behnel.de Thu Aug 20 16:31:35 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 20 Aug 2009 16:31:35 +0200 Subject: [Cython] anyone using Py++ with Cython? Message-ID: <4A8D5E47.3030207@behnel.de> Hi, I just stumbled over the Py++ project that reads C++ code using pygccxml and comes with wrapper code generators for Boost.Python and ctypes. http://www.language-binding.net/pyplusplus/pyplusplus.html >From a first glance, it seems not too hard to add a third generator for Cython code that could write out the corresponding .pxd files and a simple 1:1 API skeleton. Has anyone looked into this already? This would be a wonderful tool for those who want to wrap larger libraries. Stefan From cournape at gmail.com Fri Aug 21 01:42:09 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 20 Aug 2009 16:42:09 -0700 Subject: [Cython] Cython 0.11.2 win32 installer In-Reply-To: <1cd32cbb0908131103q6c28afccvf67a54ceec821e58@mail.gmail.com> References: <96de71860908121538k98dd1e3ldaf6a567ad474b19@mail.gmail.com> <4A83C44D.9040107@student.matnat.uio.no> <96de71860908130133t2f97ed34he8ff5ba3eeabd517@mail.gmail.com> <7EBD264E-B8E0-4E90-B4CD-26507123D5F8@math.washington.edu> <4A8446F4.80200@noaa.gov> <1cd32cbb0908131103q6c28afccvf67a54ceec821e58@mail.gmail.com> Message-ID: <5b8d13220908201642g4ec9f3e7o1c98fe50695ad32b@mail.gmail.com> On Thu, Aug 13, 2009 at 11:03 AM, wrote: > On Thu, Aug 13, 2009 at 1:01 PM, Christopher > Barker wrote: >> Dag Sverre Seljebotn wrote: >>> I think the benefit for Cython is less than it would appear at first >>> glance, since Cython is next to useless without a C compiler and Python C >>> extension compilation support. >> >> I agree -- getting that set up is a much bigger hurdle -- really, >> Windows installers are critical for packages that have to be compiled, >> so that people without a compiler can use them. >> >>> PythonXY update Cython regularily, so the usecase that seems to remain for >>> an .exe is to update EPD. >> >> Doesn't Enthought build their own anyway? >> >>> A Cython installer bundling MinGW and setting it up properly would be >>> something though! >> >> Now that would be great! I don't know why, but installing MinGW always >> seems to be far more of a pain than it should be. >> >> -Chris > > I don't think bundling MinGW with cython for Windows is a great idea. > shedskin had mingw bundled and it works ok for getting started, but > it's not useful for customization, adding libraries and so on. I agree - mingw is easy to install, but the mingw webpage is awful, and the download link almost hidden in their main page. The only thing which may be usefully added to the installer is to automatically add mingw to the PATH if asked - I am sure it would be relatively easy to add if someone send them the patch. cheers, David From cournape at gmail.com Fri Aug 21 01:44:40 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 20 Aug 2009 16:44:40 -0700 Subject: [Cython] Cython 0.11.2 win32 installer In-Reply-To: <96de71860908130133t2f97ed34he8ff5ba3eeabd517@mail.gmail.com> References: <96de71860908121538k98dd1e3ldaf6a567ad474b19@mail.gmail.com> <4A83C44D.9040107@student.matnat.uio.no> <96de71860908130133t2f97ed34he8ff5ba3eeabd517@mail.gmail.com> Message-ID: <5b8d13220908201644u3697c534v262ff2fc778f1664@mail.gmail.com> On Thu, Aug 13, 2009 at 1:33 AM, Fernando Perez wrote: > Hi Dag, > > On Thu, Aug 13, 2009 at 12:44 AM, Dag Sverre > Seljebotn wrote: >> Hi cython-dev, >> >> Fernando Perez wrote: >>> >>> Hi Dag, >>> >>> Dave Cournapeau kindly made for us win32 installers of the newest >>> cython, since you guys don't have them on your site: >>> >>> https://cirl.berkeley.edu/fperez/tmp/Cython-0.11.2.win32-py2.5.exe >>> https://cirl.berkeley.edu/fperez/tmp/Cython-0.11.2.win32-py2.6.exe >> >> Great, thanks! >> >> Is there somebody on the list who could more easily than myself give these a >> test ride? >> >> (In addition to running the installer, a C compiler must be set up properly, >> see >> >> wiki.cython.org/Installation >> >> and the link to Windows installation there. An alternative to creating >> distutils.cfg (for mingw) is to pass "-c mingw" to setup.py.) > > I just tested this on top of a bare EPD install, and once I commented > out the old cython egg path from the easy-install.pth file, it worked > just fine. ?So tomorrow I'll publicize it to the attendees. > >> I'll take this opportunity to come with a request: If somebody with easy >> access to Windows would like to be responsible for Windows packaging of >> Cython that would be great (as none of the core devs run Windows) -- it >> seems that David Cournapeau has already done the dirty work, so that it >> would likely just be about running his scripts for every Cython release and >> test that things work properly. > > By the way, David did this on a wine setup he has on his Mac laptop in > just a couple of minutes! ?So it seems that once you have the right > setup, it's not a huge deal. It is quite easy to set up: you can install wine for mac os x (darwine), install python + mingw into wine, and that's it. I have for example an alist winepy="WINEPATH $HOME/.wine/drive_c/Python25/python.exe", and then making the binary is just an issue of: winepy setup.py bdist_wininst Numpy and scipy binaries are actually entirely built under wine as well nowadays, it is a viable alternative to the endless windows pain. David From fperez.net at gmail.com Fri Aug 21 08:36:41 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Thu, 20 Aug 2009 23:36:41 -0700 Subject: [Cython] Cython 0.11.2 win32 installer In-Reply-To: <5b8d13220908201644u3697c534v262ff2fc778f1664@mail.gmail.com> References: <96de71860908121538k98dd1e3ldaf6a567ad474b19@mail.gmail.com> <4A83C44D.9040107@student.matnat.uio.no> <96de71860908130133t2f97ed34he8ff5ba3eeabd517@mail.gmail.com> <5b8d13220908201644u3697c534v262ff2fc778f1664@mail.gmail.com> Message-ID: On Thu, Aug 20, 2009 at 4:44 PM, David Cournapeau wrote: > On Thu, Aug 13, 2009 at 1:33 AM, Fernando > Perez wrote: >> By the way, David did this on a wine setup he has on his Mac laptop in >> just a couple of minutes! ?So it seems that once you have the right >> setup, it's not a huge deal. > > It is quite easy to set up: you can install wine for mac os x > (darwine), install python + mingw into wine, and that's it. > > I have for example an alist winepy="WINEPATH > $HOME/.wine/drive_c/Python25/python.exe", and then making the binary > is just an issue of: > > winepy setup.py bdist_wininst > > Numpy and scipy binaries are actually entirely built under wine as > well nowadays, it is a viable alternative to the endless windows pain. I should add that Stefan vdW pointed out to me that the detailed tutorial on how to set this wine machinery lives in fact on the Cython wiki :) http://wiki.cython.org/BuildingWindowsInstaller I did it a few days ago, and I can now build a Cython installer in no time flat: maqroll[Cython-0.11.2]> uname -a Linux maqroll 2.6.28-15-generic #49-Ubuntu SMP Tue Aug 18 18:40:08 UTC 2009 i686 GNU/Linux maqroll[Cython-0.11.2]> time wine python setup.py bdist_wininst Compiling module Cython.Plex.Scanners ... Compiling module Cython.Compiler.Scanning ... Compiling module Cython.Compiler.Parsing ... Compiling module Cython.Compiler.Visitor ... Compiling module Cython.Runtime.refnanny ... running bdist_wininst [... snip long output ] adding 'PLATLIB\pyximport\pyxbuild.py' adding 'SCRIPTS\cython.py' creating dist removing 'build\bdist.win32\wininst' (and everything under it) real 0m20.775s user 0m5.764s sys 0m0.500s This is really great: the setup takes a few minutes, and now I don't even have to fire up VirtualBox and spend any time in the godforsaken hell that is Windows to build installers for code with C extensions! Cheers, f From josef.pktd at gmail.com Fri Aug 21 13:05:27 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 21 Aug 2009 07:05:27 -0400 Subject: [Cython] Cython 0.11.2 win32 installer In-Reply-To: References: <96de71860908121538k98dd1e3ldaf6a567ad474b19@mail.gmail.com> <4A83C44D.9040107@student.matnat.uio.no> <96de71860908130133t2f97ed34he8ff5ba3eeabd517@mail.gmail.com> <5b8d13220908201644u3697c534v262ff2fc778f1664@mail.gmail.com> Message-ID: <1cd32cbb0908210405r39eaff1dyb6cfd15ba1a52e78@mail.gmail.com> On Fri, Aug 21, 2009 at 2:36 AM, Fernando Perez wrote: > On Thu, Aug 20, 2009 at 4:44 PM, David Cournapeau wrote: >> On Thu, Aug 13, 2009 at 1:33 AM, Fernando >> Perez wrote: > >>> By the way, David did this on a wine setup he has on his Mac laptop in >>> just a couple of minutes! ?So it seems that once you have the right >>> setup, it's not a huge deal. >> >> It is quite easy to set up: you can install wine for mac os x >> (darwine), install python + mingw into wine, and that's it. >> >> I have for example an alist winepy="WINEPATH >> $HOME/.wine/drive_c/Python25/python.exe", and then making the binary >> is just an issue of: >> >> winepy setup.py bdist_wininst >> >> Numpy and scipy binaries are actually entirely built under wine as >> well nowadays, it is a viable alternative to the endless windows pain. > > I should add that Stefan vdW pointed out to me that the detailed > tutorial on how to set this wine machinery lives in fact on the Cython > wiki :) > > http://wiki.cython.org/BuildingWindowsInstaller > > I did it a few days ago, and I can now build a Cython installer in no time flat: > > maqroll[Cython-0.11.2]> uname -a > Linux maqroll 2.6.28-15-generic #49-Ubuntu SMP Tue Aug 18 18:40:08 UTC > 2009 i686 GNU/Linux > maqroll[Cython-0.11.2]> time wine python setup.py bdist_wininst > Compiling module Cython.Plex.Scanners ... > Compiling module Cython.Compiler.Scanning ... > Compiling module Cython.Compiler.Parsing ... > Compiling module Cython.Compiler.Visitor ... > Compiling module Cython.Runtime.refnanny ... > running bdist_wininst > > [... snip long output ] > > adding 'PLATLIB\pyximport\pyxbuild.py' > adding 'SCRIPTS\cython.py' > creating dist > removing 'build\bdist.win32\wininst' (and everything under it) > > real ? ?0m20.775s > user ? ?0m5.764s > sys ? ? 0m0.500s > > This is really great: the setup takes a few minutes, and now I don't > even have to fire up VirtualBox and spend any time in the godforsaken > hell that is Windows to build installers for code with C extensions! There is no reason for this language, there are a lot of us who like Windows, and if you look at the numpy/scipy mailinglist then the only build issues are with the hundreds of versions of unix/linux. I find it extremely difficult to do anything in a bash shell, because they screwed up every command and shortcut key compared to windows. Josef > > Cheers, > > f > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From cournape at gmail.com Fri Aug 21 17:36:56 2009 From: cournape at gmail.com (David Cournapeau) Date: Fri, 21 Aug 2009 08:36:56 -0700 Subject: [Cython] Cython 0.11.2 win32 installer In-Reply-To: <1cd32cbb0908210405r39eaff1dyb6cfd15ba1a52e78@mail.gmail.com> References: <96de71860908121538k98dd1e3ldaf6a567ad474b19@mail.gmail.com> <4A83C44D.9040107@student.matnat.uio.no> <96de71860908130133t2f97ed34he8ff5ba3eeabd517@mail.gmail.com> <5b8d13220908201644u3697c534v262ff2fc778f1664@mail.gmail.com> <1cd32cbb0908210405r39eaff1dyb6cfd15ba1a52e78@mail.gmail.com> Message-ID: <5b8d13220908210836g6eed2176u496fc73dbd075967@mail.gmail.com> On Fri, Aug 21, 2009 at 4:05 AM, wrote: > On Fri, Aug 21, 2009 at 2:36 AM, Fernando Perez wrote: >> On Thu, Aug 20, 2009 at 4:44 PM, David Cournapeau wrote: >>> On Thu, Aug 13, 2009 at 1:33 AM, Fernando >>> Perez wrote: >> >>>> By the way, David did this on a wine setup he has on his Mac laptop in >>>> just a couple of minutes! ?So it seems that once you have the right >>>> setup, it's not a huge deal. >>> >>> It is quite easy to set up: you can install wine for mac os x >>> (darwine), install python + mingw into wine, and that's it. >>> >>> I have for example an alist winepy="WINEPATH >>> $HOME/.wine/drive_c/Python25/python.exe", and then making the binary >>> is just an issue of: >>> >>> winepy setup.py bdist_wininst >>> >>> Numpy and scipy binaries are actually entirely built under wine as >>> well nowadays, it is a viable alternative to the endless windows pain. >> >> I should add that Stefan vdW pointed out to me that the detailed >> tutorial on how to set this wine machinery lives in fact on the Cython >> wiki :) >> >> http://wiki.cython.org/BuildingWindowsInstaller >> >> I did it a few days ago, and I can now build a Cython installer in no time flat: >> >> maqroll[Cython-0.11.2]> uname -a >> Linux maqroll 2.6.28-15-generic #49-Ubuntu SMP Tue Aug 18 18:40:08 UTC >> 2009 i686 GNU/Linux >> maqroll[Cython-0.11.2]> time wine python setup.py bdist_wininst >> Compiling module Cython.Plex.Scanners ... >> Compiling module Cython.Compiler.Scanning ... >> Compiling module Cython.Compiler.Parsing ... >> Compiling module Cython.Compiler.Visitor ... >> Compiling module Cython.Runtime.refnanny ... >> running bdist_wininst >> >> [... snip long output ] >> >> adding 'PLATLIB\pyximport\pyxbuild.py' >> adding 'SCRIPTS\cython.py' >> creating dist >> removing 'build\bdist.win32\wininst' (and everything under it) >> >> real ? ?0m20.775s >> user ? ?0m5.764s >> sys ? ? 0m0.500s >> >> This is really great: the setup takes a few minutes, and now I don't >> even have to fire up VirtualBox and spend any time in the godforsaken >> hell that is Windows to build installers for code with C extensions! > > There is no reason for this language, there are a lot of us who like > Windows, and if you look at the numpy/scipy mailinglist then the only > build issues are with the hundreds of versions of unix/linux. There are no build issues on windows because nobody builds it, and because some people in numpy/scipy waste days if not weeks working on windows specificities. David From sturla at molden.no Sat Aug 22 04:28:15 2009 From: sturla at molden.no (Sturla Molden) Date: Fri, 21 Aug 2009 19:28:15 -0700 Subject: [Cython] Cython 0.11.2 win32 installer In-Reply-To: <5b8d13220908201642g4ec9f3e7o1c98fe50695ad32b@mail.gmail.com> References: <96de71860908121538k98dd1e3ldaf6a567ad474b19@mail.gmail.com> <4A83C44D.9040107@student.matnat.uio.no> <96de71860908130133t2f97ed34he8ff5ba3eeabd517@mail.gmail.com> <7EBD264E-B8E0-4E90-B4CD-26507123D5F8@math.washington.edu> <4A8446F4.80200@noaa.gov> <1cd32cbb0908131103q6c28afccvf67a54ceec821e58@mail.gmail.com> <5b8d13220908201642g4ec9f3e7o1c98fe50695ad32b@mail.gmail.com> Message-ID: <4A8F57BF.4090301@molden.no> David Cournapeau skrev: > I agree - mingw is easy to install, but the mingw webpage is awful, > and the download link almost hidden in their main page. > > I tend to like the binary installer from Equation.com better (currently GCC 4.4.1): http://www.equation.com/servlet/equation.cmd?call=fortran gfortran's wiki also has binary installers for Windows (GCC 4.5.x), but it does not include C++. http://gcc.gnu.org/wiki/GFortranBinaries MSYS from MinGW is nice if you need to run configure scripts. But getting all the right packages from MinGW's sourceforge site a nightmare. They have also made sure that all packages are gzipped tarballs, not zip-files that Windows undertands natively. So to install you first have to find a way to unpack the tarballs (e.g. pay for WinZip), which can be a challenge on Windows. No... use the binary from Equation.com instead. You get C, C++ and Fortran compilers installed without any hassle. Sturla Molden From josef.pktd at gmail.com Fri Aug 21 20:15:52 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Fri, 21 Aug 2009 14:15:52 -0400 Subject: [Cython] Cython 0.11.2 win32 installer In-Reply-To: <4A8F57BF.4090301@molden.no> References: <96de71860908121538k98dd1e3ldaf6a567ad474b19@mail.gmail.com> <4A83C44D.9040107@student.matnat.uio.no> <96de71860908130133t2f97ed34he8ff5ba3eeabd517@mail.gmail.com> <7EBD264E-B8E0-4E90-B4CD-26507123D5F8@math.washington.edu> <4A8446F4.80200@noaa.gov> <1cd32cbb0908131103q6c28afccvf67a54ceec821e58@mail.gmail.com> <5b8d13220908201642g4ec9f3e7o1c98fe50695ad32b@mail.gmail.com> <4A8F57BF.4090301@molden.no> Message-ID: <1cd32cbb0908211115h67459efemad6a4c2af811858@mail.gmail.com> On Fri, Aug 21, 2009 at 10:28 PM, Sturla Molden wrote: > David Cournapeau skrev: >> I agree - mingw is easy to install, but the mingw webpage is awful, >> and the download link almost hidden in their main page. >> >> > I tend to like the binary installer from Equation.com better (currently > GCC 4.4.1): > > http://www.equation.com/servlet/equation.cmd?call=fortran > > gfortran's wiki also has binary installers for Windows (GCC 4.5.x), but > it does not include C++. > > http://gcc.gnu.org/wiki/GFortranBinaries > > MSYS from MinGW is nice if you need to run configure scripts. > > But getting all the right packages from MinGW's sourceforge site a > nightmare. They have also made sure that all packages are gzipped > tarballs, not zip-files that Windows undertands natively. So to install > you first have to find a way to unpack the tarballs (e.g. pay for > WinZip), which can be a challenge on Windows. For the official version, MingW has an installer script to pick the individual pieces and update them in one run, although they are cautious and slow with the official updates. For archive files, I use either WinRar or 7zip, and I haven't yet found an archive type that I couldn't open. (I also turned off automatic/native zip support in Windows, to avoid long searches in archive files.) Josef > > No... use the binary from Equation.com instead. You get C, C++ and > Fortran compilers installed without any hassle. > > Sturla Molden > > > > > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From fperez.net at gmail.com Fri Aug 21 20:17:57 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Fri, 21 Aug 2009 11:17:57 -0700 Subject: [Cython] Cython 0.11.2 win32 installer In-Reply-To: <1cd32cbb0908210405r39eaff1dyb6cfd15ba1a52e78@mail.gmail.com> References: <96de71860908121538k98dd1e3ldaf6a567ad474b19@mail.gmail.com> <4A83C44D.9040107@student.matnat.uio.no> <96de71860908130133t2f97ed34he8ff5ba3eeabd517@mail.gmail.com> <5b8d13220908201644u3697c534v262ff2fc778f1664@mail.gmail.com> <1cd32cbb0908210405r39eaff1dyb6cfd15ba1a52e78@mail.gmail.com> Message-ID: On Fri, Aug 21, 2009 at 4:05 AM, wrote: > There is no reason for this language, there are a lot of us who like > Windows, and if you look at the numpy/scipy mailinglist then the only > build issues are with the hundreds of versions of unix/linux. > You're right, my apology: I shouldn't email when I'm that tired (too many days at the scipy conference with too much to do and too little sleep). Windows is a sensitive topic to me because it has given us in ipython nothing but headaches and cost a disproportionate amount of developer time, and has created special-case code that I'd be happier without. So for *me*, windows is indeed a problem. But people are free to use the tools that best suit their needs, and we should help them out as best we can, within the limits of our available resources. The benefit of this wine-based setup is that it lets *nix-based developers help windows users with less expenditure of their time, the scarcest of resources. And that is certainly a net win. So I apologize if it offended you, that was certainly not my intention. Regards, f From robertwb at math.washington.edu Sat Aug 22 05:37:30 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 21 Aug 2009 20:37:30 -0700 Subject: [Cython] anyone using Py++ with Cython? In-Reply-To: <4A8D5E47.3030207@behnel.de> References: <4A8D5E47.3030207@behnel.de> Message-ID: <7B7D9369-761E-4EBB-870C-2C51B8A8BADA@math.washington.edu> On Aug 20, 2009, at 7:31 AM, Stefan Behnel wrote: > Hi, > > I just stumbled over the Py++ project that reads C++ code using > pygccxml > and comes with wrapper code generators for Boost.Python and ctypes. > > http://www.language-binding.net/pyplusplus/pyplusplus.html > >> From a first glance, it seems not too hard to add a third >> generator for > Cython code that could write out the corresponding .pxd files and a > simple > 1:1 API skeleton. > > Has anyone looked into this already? This would be a wonderful tool > for > those who want to wrap larger libraries. Yes, that would be cool. Just using gcc-xml to generate .pxd files (for c or c++) would be very useful and something I've thought about before, but this is a step beyond that. - Robert From krunk7 at gmail.com Sat Aug 22 17:43:29 2009 From: krunk7 at gmail.com (James Kyle) Date: Sat, 22 Aug 2009 08:43:29 -0700 Subject: [Cython] first stab at a c++ api wrapper Message-ID: <2A61DFC7-5830-4AAA-A0E8-126837C3D283@gmail.com> Morning, I'm attempting to make a c++ api wrapper to an existing library. I'm having some trouble getting past the basics. My first stop was the manual, but I could not get the Rectangle Demo to compile on my system. Some searching of the mailing list found a thread where someone was kind enough to post corrections to the wiki and manual entries (http://codespeak.net/pipermail/cython-dev/2008-April/000516.html ). With this I was able to get the example to compile, but quickly ran into issues of my own when trying to duplicate it for my library. The library is installed into /opt/local with headers in /opt/local/ include/mp4v2 and libs in /opt/local/lib. I was going to first attempt to duplicate the examples provided by the project. The first step would be to declare a MP4FileHandle object and open it with MP4Modify(). It would look something like this: > #include > #include > #include > #include > > int main (int argc, char const **argv) > { > MP4FileHandle file = MP4Modify(argv[1], MP4_DETAILS_ERROR, 0); > if(file == MP4_INVALID_FILE_HANDLE) { > printf("MP4Modify failed\n"); > return 1; > } > > MP4Close(file); > > return 0; > } Obviously, the MP4* definitions are found in mp4v2/mp4v2.h. Compiling is also straight forward: > g++ -I/opt/local/include -L/opt/local/lib -lmp4v2 main.cpp Now, on to duplicating or declaring these in a cython interface file. I put together a setup.py like so: > from distutils.core import setup > from distutils.extension import Extension > from Cython.Distutils import build_ext > > utils = Extension( > "utils", > sources=["pymp4v2/utils.pyx"], > include_dirs=["/opt/lcoal/include/"], > libraries=["mp4v2"], > library_dirs=['/opt/local/lib'], > language="c++", > define_macros=[('MACOSX_DEPLOYMENT_TARGET', 10.5)], > ) > > setup( > ext_modules = [utils], > cmdclass={'build_ext':build_ext}, > > > ) I then created the following pyx file and attempted build. I tried various other combos e.g. cdef extern from "mp4v2.mp4v2.h" or "mp4v2/ mp4v2.h". any guidance is greatly appreciated. I'm sure I'm missing some section of the documentation that explains this, my apologies if that's the case. -james > """ > Python Wrapper Interface for libmp4v2 methods. > """ > cdef extern from "mp4v2.h": > void* MP4FileHandle > MP4FileHandle MP4Modify(char* fileName, int verbosity=0, int > flags = 0) > running build_ext > cythoning pymp4v2/utils.pyx to pymp4v2/utils.cpp > > Error converting Pyrex file to C: > ------------------------------------------------------------ > ... > """ > Python Wrapper Interface for libmp4v2 methods. > """ > cdef extern from "mp4v2.h": > void* MP4FileHandle > MP4FileHandle MP4Modify(char* fileName, int verbosity=0, int > flags = 0) ^ > ------------------------------------------------------------ > > /Users/jkyle/Projects/pymp4v2.env/src/python/pymp4v2/utils.pyx:6:4: > 'MP4FileHandle' is not a type identifier > building 'utils' extension > /usr/bin/gcc-4.0 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG - > g -fwrapv -O3 -Wall -Wstrict-prototypes - > DMACOSX_DEPLOYMENT_TARGET=10.5 -I/opt/lcoal/include/ -I/opt/local/ > Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 - > c pymp4v2/utils.cpp -o build/temp.macosx-10.3-i386-2.6/pymp4v2/utils.o > cc1plus: warning: command line option "-Wstrict-prototypes" is valid > for C/ObjC but not for C++ > pymp4v2/utils.cpp:1:2: error: #error Do not use this file, it is the > result of a failed Cython compilation. > error: command '/usr/bin/gcc-4.0' failed with exit status 1 > From adam.ginsburg at colorado.edu Sun Aug 23 04:28:07 2009 From: adam.ginsburg at colorado.edu (Adam Ginsburg) Date: Sat, 22 Aug 2009 20:28:07 -0600 Subject: [Cython] Linking error - 64 bit problem? Message-ID: Hi folks, I'm trying to turn some numpy code into cython. In pure python it's very slow, and I don't believe the task can be vectorized easily, so I've turned to you. The code imported fine before I tried adding the cython modifications to it, but it ran slower than before. Now, it fails with this error: $ python-64 setup.py build running build running build_ext building 'cplfit' extension creating build creating build/temp.macosx-10.3-universal-2.6 gcc -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/ -I/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include -I. -I/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c cplfit.c -o build/temp.macosx-10.3-universal-2.6/cplfit.o cplfit.c:854: warning: '__pyx_k_3' defined but not used cplfit.c:855: warning: '__pyx_k_4' defined but not used cplfit.c:875: warning: '__pyx_k_24' defined but not used cplfit.c:881: warning: '__pyx_k_26' defined but not used cplfit.c:882: warning: '__pyx_k_27' defined but not used i686-apple-darwin9-gcc-4.0.1: /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/: linker input file unused because linking not done creating build/lib.macosx-10.3-universal-2.6 gcc -arch i386 -arch ppc -arch ppc64 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.5.sdk -bundle -undefined dynamic_lookup /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/ build/temp.macosx-10.3-universal-2.6/cplfit.o -o build/lib.macosx-10.3-universal-2.6/cplfit.so ld: in /Developer/SDKs/MacOSX10.5.sdk/Library/Frameworks/Python.framework/ldVersions/:2.6 /inlib //Developer/ldSDKs:python2.6 in/MacOSX10.5.sdk ///siteLibrary-/Developer/packagesFrameworks/SDKsnumpy//Python.frameworkcore///MacOSX10.5.sdkVersions/include//,Library /can2.6Frameworks'//tlibPython.framework //mappython2.6Versions //filesite2.6-/,packageslib //errnonumpypython2.6/ldsite=-/core/:22 include for architecturepackages/numpy/ ppc64 ,in // coreDevelopercan//'includeSDKs/tMacOSX10.5.sdk/ Library/map, /file, Frameworks canerrno't/= map collect2: 22 filefor architectureld returned 1 exit status ,Python.frameworki386 / errnoVersions/=2.6/22lib /forpython2.6 architecture /x86_64 site-packages/numpy/core/include/, can't map file, errno=22 for architecture ppccollect2: ld returned 1 exit status collect2: ld returned 1 exit status collect2: ld returned 1 exit status lipo: can't open input file: /var/folders/ni/ni+DtdqFGMeSMH13AvkNkU+++TI/-Tmp-//ccl3U2RI.out (No such file or directory) error: command 'gcc' failed with exit status 1 My setup.py: from distutils.core import setup from distutils.extension import Extension from Cython.Distutils import build_ext import numpy setup( ext_modules = [ Extension("cplfit", ["cplfit.pyx"], include_dirs = [numpy.get_include(),'.']) ], cmdclass = {'build_ext': build_ext}, ) Am I passing the wrong flags / setting the wrong variables? Or is there some fundamental incompatibility with the 64 bit setup I have? I can distribute the python & cython code if that helps. Thanks, Adam From adam.ginsburg at colorado.edu Sun Aug 23 04:40:18 2009 From: adam.ginsburg at colorado.edu (Adam Ginsburg) Date: Sat, 22 Aug 2009 20:40:18 -0600 Subject: [Cython] Linking error - 64 bit problem? In-Reply-To: References: Message-ID: I reinstalled cython from source (instead of easy_install-ing it) and my code compiled successfully. Sorry about the previous e-mail. Unfortunately, my code runs slower with cython than without, but at least now I know I can compile. In case anyone wants to offer assistance, the code I'm trying to optimize is: http://code.google.com/p/agpy/source/browse/trunk/plfit.py http://code.google.com/p/agpy/source/browse/trunk/cplfit.pyx Thanks, Adam On Sat, Aug 22, 2009 at 8:28 PM, Adam Ginsburg wrote: > Hi folks, > ?I'm trying to turn some numpy code into cython. ?In pure python > it's very slow, and I don't believe the task can be vectorized easily, > so I've turned to you. > > ?The code imported fine before I tried adding the cython > modifications to it, but it ran slower than before. ?Now, it fails > with this error: > > $ python-64 setup.py build > running build > running build_ext > building 'cplfit' extension > creating build > creating build/temp.macosx-10.3-universal-2.6 > gcc -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes > /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/ > -I/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include > -I. -I/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 > -c cplfit.c -o build/temp.macosx-10.3-universal-2.6/cplfit.o > cplfit.c:854: warning: '__pyx_k_3' defined but not used > cplfit.c:855: warning: '__pyx_k_4' defined but not used > cplfit.c:875: warning: '__pyx_k_24' defined but not used > cplfit.c:881: warning: '__pyx_k_26' defined but not used > cplfit.c:882: warning: '__pyx_k_27' defined but not used > i686-apple-darwin9-gcc-4.0.1: > /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/: > linker input file unused because linking not done > creating build/lib.macosx-10.3-universal-2.6 > gcc -arch i386 -arch ppc -arch ppc64 -arch x86_64 -isysroot > /Developer/SDKs/MacOSX10.5.sdk -bundle -undefined dynamic_lookup > /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/numpy/core/include/ > build/temp.macosx-10.3-universal-2.6/cplfit.o -o > build/lib.macosx-10.3-universal-2.6/cplfit.so > ld: in /Developer/SDKs/MacOSX10.5.sdk/Library/Frameworks/Python.framework/ldVersions/:2.6 > /inlib //Developer/ldSDKs:python2.6 in/MacOSX10.5.sdk > ///siteLibrary-/Developer/packagesFrameworks/SDKsnumpy//Python.frameworkcore///MacOSX10.5.sdkVersions/include//,Library > /can2.6Frameworks'//tlibPython.framework //mappython2.6Versions > //filesite2.6-/,packageslib //errnonumpypython2.6/ldsite=-/core/:22 > include for architecturepackages/numpy/ ppc64 > ,in // coreDevelopercan//'includeSDKs/tMacOSX10.5.sdk/ Library/map, > /file, Frameworks canerrno't/= map collect2: 22 filefor architectureld > returned 1 exit status > ?,Python.frameworki386 / > errnoVersions/=2.6/22lib /forpython2.6 architecture /x86_64 > site-packages/numpy/core/include/, can't map file, errno=22 for > architecture ppccollect2: > ld returned 1 exit status > collect2: ld returned 1 exit status > collect2: ld returned 1 exit status > lipo: can't open input file: > /var/folders/ni/ni+DtdqFGMeSMH13AvkNkU+++TI/-Tmp-//ccl3U2RI.out (No > such file or directory) > error: command 'gcc' failed with exit status 1 > > My setup.py: > from distutils.core import setup > from distutils.extension import Extension > from Cython.Distutils import build_ext > import numpy > > setup( > ? ext_modules = [ > ? ? Extension("cplfit", ["cplfit.pyx"], > ? ? include_dirs = [numpy.get_include(),'.']) > ? ? ], > ? cmdclass = {'build_ext': build_ext}, > ) > > > Am I passing the wrong flags / setting the wrong variables? ?Or is > there some fundamental incompatibility with the 64 bit setup I have? > > I can distribute the python & cython code if that helps. > > Thanks, > Adam > From robertwb at math.washington.edu Sun Aug 23 05:34:29 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 22 Aug 2009 20:34:29 -0700 Subject: [Cython] Linking error - 64 bit problem? In-Reply-To: References: Message-ID: On Aug 22, 2009, at 7:40 PM, Adam Ginsburg wrote: > I reinstalled cython from source (instead of easy_install-ing it) and > my code compiled successfully. Sorry about the previous e-mail. Great--I saw your previous email and started to get worried. I thought easy_install was a source install (we didn't ship any binaries until recently, and then just windows installers). > Unfortunately, my code runs slower with cython than without, but at > least now I know I can compile. In case anyone wants to offer > assistance, the code I'm trying to optimize is: > http://code.google.com/p/agpy/source/browse/trunk/plfit.py > http://code.google.com/p/agpy/source/browse/trunk/cplfit.pyx My first piece of advice is to run cython -a over your code. This will produce an html file which is most helpful in diagnosing where you're wasting time. Just in a quick glance at your code I noticed: - You're calling Python's log, sqrt, max, abs, sum. This would negate any gains you may hope to see. - You're calling len a lot, either store it as a int somewhere, or use .shape[0]. - Unless n is huge, arange(n)/float(n) is probably way to expensive. If the above don't help, try unrolling this. I bet there's a lot more performance that one could ring out of your code, but the above should take you far. > Thanks, > Adam > > On Sat, Aug 22, 2009 at 8:28 PM, Adam > Ginsburg wrote: >> Hi folks, >> I'm trying to turn some numpy code into cython. In pure python >> it's very slow, and I don't believe the task can be vectorized >> easily, >> so I've turned to you. >> >> The code imported fine before I tried adding the cython >> modifications to it, but it ran slower than before. Now, it fails >> with this error: >> >> $ python-64 setup.py build >> running build >> running build_ext >> building 'cplfit' extension >> creating build >> creating build/temp.macosx-10.3-universal-2.6 >> gcc -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes >> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/ >> site-packages/numpy/core/include/ >> -I/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/ >> site-packages/numpy/core/include >> -I. -I/Library/Frameworks/Python.framework/Versions/2.6/include/ >> python2.6 >> -c cplfit.c -o build/temp.macosx-10.3-universal-2.6/cplfit.o >> cplfit.c:854: warning: '__pyx_k_3' defined but not used >> cplfit.c:855: warning: '__pyx_k_4' defined but not used >> cplfit.c:875: warning: '__pyx_k_24' defined but not used >> cplfit.c:881: warning: '__pyx_k_26' defined but not used >> cplfit.c:882: warning: '__pyx_k_27' defined but not used >> i686-apple-darwin9-gcc-4.0.1: >> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/ >> site-packages/numpy/core/include/: >> linker input file unused because linking not done >> creating build/lib.macosx-10.3-universal-2.6 >> gcc -arch i386 -arch ppc -arch ppc64 -arch x86_64 -isysroot >> /Developer/SDKs/MacOSX10.5.sdk -bundle -undefined dynamic_lookup >> /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/ >> site-packages/numpy/core/include/ >> build/temp.macosx-10.3-universal-2.6/cplfit.o -o >> build/lib.macosx-10.3-universal-2.6/cplfit.so >> ld: in /Developer/SDKs/MacOSX10.5.sdk/Library/Frameworks/ >> Python.framework/ldVersions/:2.6 >> /inlib //Developer/ldSDKs:python2.6 in/MacOSX10.5.sdk >> ///siteLibrary-/Developer/packagesFrameworks/SDKsnumpy// >> Python.frameworkcore///MacOSX10.5.sdkVersions/include//,Library >> /can2.6Frameworks'//tlibPython.framework //mappython2.6Versions >> //filesite2.6-/,packageslib //errnonumpypython2.6/ldsite=-/core/:22 >> include for architecturepackages/numpy/ ppc64 >> ,in // coreDevelopercan//'includeSDKs/tMacOSX10.5.sdk/ Library/map, >> /file, Frameworks canerrno't/= map collect2: 22 filefor >> architectureld >> returned 1 exit status >> ,Python.frameworki386 / >> errnoVersions/=2.6/22lib /forpython2.6 architecture /x86_64 >> site-packages/numpy/core/include/, can't map file, errno=22 for >> architecture ppccollect2: >> ld returned 1 exit status >> collect2: ld returned 1 exit status >> collect2: ld returned 1 exit status >> lipo: can't open input file: >> /var/folders/ni/ni+DtdqFGMeSMH13AvkNkU+++TI/-Tmp-//ccl3U2RI.out (No >> such file or directory) >> error: command 'gcc' failed with exit status 1 >> >> My setup.py: >> from distutils.core import setup >> from distutils.extension import Extension >> from Cython.Distutils import build_ext >> import numpy >> >> setup( >> ext_modules = [ >> Extension("cplfit", ["cplfit.pyx"], >> include_dirs = [numpy.get_include(),'.']) >> ], >> cmdclass = {'build_ext': build_ext}, >> ) >> >> >> Am I passing the wrong flags / setting the wrong variables? Or is >> there some fundamental incompatibility with the 64 bit setup I have? >> >> I can distribute the python & cython code if that helps. >> >> Thanks, >> Adam >> > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From seb.binet at gmail.com Mon Aug 24 15:16:15 2009 From: seb.binet at gmail.com (Sebastien Binet) Date: Mon, 24 Aug 2009 15:16:15 +0200 Subject: [Cython] anyone using Py++ with Cython? In-Reply-To: <4A8D5E47.3030207@behnel.de> References: <4A8D5E47.3030207@behnel.de> Message-ID: <200908241516.15870.binet@cern.ch> On Thursday 20 August 2009 16:31:35 Stefan Behnel wrote: > Hi, > > I just stumbled over the Py++ project that reads C++ code using pygccxml > and comes with wrapper code generators for Boost.Python and ctypes. > > http://www.language-binding.net/pyplusplus/pyplusplus.html > > >From a first glance, it seems not too hard to add a third generator for > > Cython code that could write out the corresponding .pxd files and a simple > 1:1 API skeleton. > > Has anyone looked into this already? This would be a wonderful tool for > those who want to wrap larger libraries. I believe David threw a bunch of things together in that direction: http://cournape.wordpress.com/2008/12/01/cython-codegen-cython-code-generator- from-gccxml-files/ http://pypi.python.org/pypi/cython-codegen/ the only problem with all of this is that gcc-xml is ill-maintained and is based on the C++ parser of GCC. so all is fine as long as what you parse is correct C++ code, but not all C code is valid C++... FYI, the CERN team (and its Reflex library) used to rely on gcc-xml to extract such meta-data from source code, but it is now migrating towards using LLVM/CLang. cheers, sebastien. -- ######################################### # Dr. Sebastien Binet # Laboratoire de l'Accelerateur Lineaire # Universite Paris-Sud XI # Batiment 200 # 91898 Orsay ######################################### From sccolbert at gmail.com Mon Aug 24 21:18:36 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Mon, 24 Aug 2009 15:18:36 -0400 Subject: [Cython] citing Cython Message-ID: <7f014ea60908241218ud33876dma30c388e2a853382@mail.gmail.com> I'm working on a paper and need to include a citation for Cython, is there a preferred citation, or should I use this entry I found in the sage archives? @manual{cython, > Author = {Stefan Behnel, Robert Bradshaw, and Dag Sverre Seljebotn }, > Title = {Cython: C-Extensions for Python}, > note = {{\tt http://www.cython.org}}, > year = {2008} > } cheers! Chris From robertwb at math.washington.edu Mon Aug 24 22:13:13 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 24 Aug 2009 13:13:13 -0700 (PDT) Subject: [Cython] citing Cython In-Reply-To: <7f014ea60908241218ud33876dma30c388e2a853382@mail.gmail.com> References: <7f014ea60908241218ud33876dma30c388e2a853382@mail.gmail.com> Message-ID: On Mon, 24 Aug 2009, Chris Colbert wrote: > I'm working on a paper and need to include a citation for Cython, is > there a preferred citation, or should I use this entry I found in the > sage archives? > > @manual{cython, >> Author = {Stefan Behnel, Robert Bradshaw, and Dag Sverre Seljebotn }, >> Title = {Cython: C-Extensions for Python}, >> note = {{\tt http://www.cython.org}}, >> year = {2008} >> } I would put an et al. in the author field, as there are many other contributors. Also, I think Greg Ewing's name belongs in there too, as he contributed the original codebase. Thoughts? - Robert From sccolbert at gmail.com Mon Aug 24 22:46:05 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Mon, 24 Aug 2009 16:46:05 -0400 Subject: [Cython] citing Cython In-Reply-To: References: <7f014ea60908241218ud33876dma30c388e2a853382@mail.gmail.com> Message-ID: <7f014ea60908241346q2a910de3w2a101fb979f35e9a@mail.gmail.com> no prob. I'll add Greg, and I already have the et al. my citation now looks like this: @Misc{cython, Author = {{Stefan Behnel} and {Robert Bradshaw} and {Dag Sverre Seljebotn} and {Greg Ewing} and others}, Title = {Cython: {C}-{E}xtensions for {P}ython}, url = {http://www.cython.org}, year = 2009 } On Mon, Aug 24, 2009 at 4:13 PM, Robert Bradshaw wrote: > On Mon, 24 Aug 2009, Chris Colbert wrote: > >> I'm working on a paper ?and need to include a citation for Cython, is >> there a preferred citation, or should I use this entry I found in the >> sage archives? >> >> @manual{cython, >>> Author = {Stefan Behnel, Robert Bradshaw, and Dag Sverre Seljebotn }, >>> Title = {Cython: C-Extensions for Python}, >>> note = {{\tt http://www.cython.org}}, >>> year = {2008} >>> } > > I would put an et al. in the author field, as there are many other > contributors. Also, I think Greg Ewing's name belongs in there too, as he > contributed the original codebase. > > Thoughts? > > - Robert > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From saul at gfz-potsdam.de Mon Aug 24 22:26:00 2009 From: saul at gfz-potsdam.de (Joachim Saul) Date: Mon, 24 Aug 2009 22:26:00 +0200 Subject: [Cython] ctypedef public class Message-ID: <4A92F758.7020204@gfz-potsdam.de> Hello all, as a happy Pyrex user for more than six years I recently got interested in Cython as well. So far adopting codes to Cython has in most cases been quite straightforward. In fact it even resulted in improvements due to some errors, which Pyrex had not complained about and which were detected by Cython. However, for the following minimalistic example the resulting C code doesn't compile: #### _stypes.pxd #### ctypedef public class Time [type MyTime_Type, object MyTimeObject]: cdef public double seconds ctypedef public class Event [type MyEvent_Type, object MyEventObject]: cdef public Time time #### _stypes.pyx #### ctypedef public class Time [type MyTime_Type, object MyTimeObject]: def __init__(self, seconds): self.seconds = seconds ctypedef public class Event [type MyEvent_Type, object MyEventObject]: def __init__(self, Time time): self.time = time #### Compilation to C works without complaint, but the compilation of the resulting C code gives errors like _stypes.c:319: error: expected specifier-qualifier-list before 'MyTimeObject' _stypes.c: In function '__pyx_pf_7_stypes_5Event___init__': _stypes.c:463: error: 'MyEventObject' has no member named 'time' [The last error repeated several times] It turns out that in _stypes.c, the declaration of MyEventObject precedes that of MyTimeObject. Here's the relevant code snippet: /* Type declarations */ /* "/home/saul/Cython-Test/_stypes.pxd":5 * cdef public double seconds * * ctypedef public class Event [type MyEvent_Type, object MyEventObject]: # <<<<<<<<<<<<<< * cdef public Time time */ typedef struct { PyObject_HEAD MyTimeObject *time; } MyEventObject; /* "/home/saul/Cython-Test/_stypes.pxd":1 * ctypedef public class Time [type MyTime_Type, object MyTimeObject]: # <<<<<<<<<<<<<< * cdef public double seconds */ typedef struct { PyObject_HEAD double seconds; } MyTimeObject; /* Module declarations from _stypes */ I.e. MyTimeObject is declared *after* MyEventObject, which I wouldn't have expected. Is there any reason for that? At this point I am stuck, although I suspect that I overlooked something very simple. Any hints would be appreciated. By the way, the C code Pyrex produces from the above pyx/pxd compiles and works properly. Cython 0.11.2 and Python 2.6 under Ubuntu 9.04. Cheers, Joachim From robertwb at math.washington.edu Tue Aug 25 09:27:46 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 25 Aug 2009 00:27:46 -0700 Subject: [Cython] ctypedef public class In-Reply-To: <4A92F758.7020204@gfz-potsdam.de> References: <4A92F758.7020204@gfz-potsdam.de> Message-ID: On Aug 24, 2009, at 1:26 PM, Joachim Saul wrote: > Hello all, > > as a happy Pyrex user for more than six years I recently got > interested > in Cython as well. So far adopting codes to Cython has in most cases > been quite straightforward. In fact it even resulted in > improvements due > to some errors, which Pyrex had not complained about and which were > detected by Cython. > > However, for the following minimalistic example the resulting C code > doesn't compile: > > #### _stypes.pxd #### > ctypedef public class Time [type MyTime_Type, object MyTimeObject]: > cdef public double seconds > > ctypedef public class Event [type MyEvent_Type, object MyEventObject]: > cdef public Time time > > #### _stypes.pyx #### > ctypedef public class Time [type MyTime_Type, object MyTimeObject]: > def __init__(self, seconds): > self.seconds = seconds > > ctypedef public class Event [type MyEvent_Type, object MyEventObject]: > def __init__(self, Time time): > self.time = time > > #### > > Compilation to C works without complaint, but the compilation of the > resulting C code gives errors like [...] Thanks for the bug report! I bet our header-ordering code lacks some constraint, but I'm not sure how hard it will be to fix. We are tracking it at http://trac.cython.org/cython_trac/ticket/355 - Robert From robertwb at math.washington.edu Tue Aug 25 09:32:15 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 25 Aug 2009 00:32:15 -0700 Subject: [Cython] first stab at a c++ api wrapper In-Reply-To: <2A61DFC7-5830-4AAA-A0E8-126837C3D283@gmail.com> References: <2A61DFC7-5830-4AAA-A0E8-126837C3D283@gmail.com> Message-ID: On Aug 22, 2009, at 8:43 AM, James Kyle wrote: > Morning, > > I'm attempting to make a c++ api wrapper to an existing library. I'm > having some trouble getting past the basics. [...] >> cdef extern from "mp4v2.h": >> void* MP4FileHandle What you've declared here is a variable of type void* named MP4FileHandle. You probably want ctypedef void *MP4FileHandle >> MP4FileHandle MP4Modify(char* fileName, int verbosity=0, int >> flags = 0) ^ >> ------------------------------------------------------------ >> >> /Users/jkyle/Projects/pymp4v2.env/src/python/pymp4v2/utils.pyx:6:4: >> 'MP4FileHandle' is not a type identifier Let us know if this solves your issue. - Robert From dagss at student.matnat.uio.no Tue Aug 25 11:49:00 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 25 Aug 2009 11:49:00 +0200 Subject: [Cython] Cython 0.11.2 win32 installer In-Reply-To: <1cd32cbb0908210405r39eaff1dyb6cfd15ba1a52e78@mail.gmail.com> References: <96de71860908121538k98dd1e3ldaf6a567ad474b19@mail.gmail.com> <4A83C44D.9040107@student.matnat.uio.no> <96de71860908130133t2f97ed34he8ff5ba3eeabd517@mail.gmail.com> <5b8d13220908201644u3697c534v262ff2fc778f1664@mail.gmail.com> <1cd32cbb0908210405r39eaff1dyb6cfd15ba1a52e78@mail.gmail.com> Message-ID: <4A93B38C.9010001@student.matnat.uio.no> josef.pktd at gmail.com wrote: > On Fri, Aug 21, 2009 at 2:36 AM, Fernando Perez wrote: >> On Thu, Aug 20, 2009 at 4:44 PM, David Cournapeau wrote: >>> On Thu, Aug 13, 2009 at 1:33 AM, Fernando >>> Perez wrote: >>>> By the way, David did this on a wine setup he has on his Mac laptop in >>>> just a couple of minutes! So it seems that once you have the right >>>> setup, it's not a huge deal. >>> It is quite easy to set up: you can install wine for mac os x >>> (darwine), install python + mingw into wine, and that's it. >>> >>> I have for example an alist winepy="WINEPATH >>> $HOME/.wine/drive_c/Python25/python.exe", and then making the binary >>> is just an issue of: >>> >>> winepy setup.py bdist_wininst >>> >>> Numpy and scipy binaries are actually entirely built under wine as >>> well nowadays, it is a viable alternative to the endless windows pain. >> I should add that Stefan vdW pointed out to me that the detailed >> tutorial on how to set this wine machinery lives in fact on the Cython >> wiki :) >> >> http://wiki.cython.org/BuildingWindowsInstaller >> >> I did it a few days ago, and I can now build a Cython installer in no time flat: >> >> maqroll[Cython-0.11.2]> uname -a >> Linux maqroll 2.6.28-15-generic #49-Ubuntu SMP Tue Aug 18 18:40:08 UTC >> 2009 i686 GNU/Linux >> maqroll[Cython-0.11.2]> time wine python setup.py bdist_wininst >> Compiling module Cython.Plex.Scanners ... >> Compiling module Cython.Compiler.Scanning ... >> Compiling module Cython.Compiler.Parsing ... >> Compiling module Cython.Compiler.Visitor ... >> Compiling module Cython.Runtime.refnanny ... >> running bdist_wininst >> >> [... snip long output ] >> >> adding 'PLATLIB\pyximport\pyxbuild.py' >> adding 'SCRIPTS\cython.py' >> creating dist >> removing 'build\bdist.win32\wininst' (and everything under it) >> >> real 0m20.775s >> user 0m5.764s >> sys 0m0.500s >> >> This is really great: the setup takes a few minutes, and now I don't >> even have to fire up VirtualBox and spend any time in the godforsaken >> hell that is Windows to build installers for code with C extensions! > > There is no reason for this language, there are a lot of us who like > Windows, and if you look at the numpy/scipy mailinglist then the only > build issues are with the hundreds of versions of unix/linux. Your statement here in turn provokes me. I'd like to try to stop short of a flamewar, but I have to say something: Like David said, this is simply because Windows-users don't build their own software!! A lot of time in open source is spent on Windows-support, it's just that few of the typical mailing list participants build it themselves! (Implication: They can't contribute to development versions!) Windows support takes a good deal of time from open source development -- while my feeling is relatively few Windows users take the trouble to contribute back (with some honorable exceptions, of course; here on the list the recent pyximport patches have come from Windows users for instance). (One quote in support of this heard during SciPy 09: All SymPy devs use Linux or Mac. Yet the number of Windows downloads are much higher.) I'm always tempted to say that I believe Cython should simply say that we do not support Windows, or Visual C, until we can AT THE VERY LEAST find one user who actually use Windows on a daily basis who volunteers to do the relatively trivial task of testing new releases and packaging it for exe distribution about four times a year. -- Dag Sverre From cournape at gmail.com Tue Aug 25 15:39:34 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 25 Aug 2009 08:39:34 -0500 Subject: [Cython] anyone using Py++ with Cython? In-Reply-To: <200908241516.15870.binet@cern.ch> References: <4A8D5E47.3030207@behnel.de> <200908241516.15870.binet@cern.ch> Message-ID: <5b8d13220908250639h2b85c5d3od3168fd1d2946562@mail.gmail.com> On Mon, Aug 24, 2009 at 8:16 AM, Sebastien Binet wrote: > On Thursday 20 August 2009 16:31:35 Stefan Behnel wrote: >> Hi, >> >> I just stumbled over the Py++ project that reads C++ code using pygccxml >> and comes with wrapper code generators for Boost.Python and ctypes. >> >> http://www.language-binding.net/pyplusplus/pyplusplus.html >> >> >From a first glance, it seems not too hard to add a third generator for >> >> Cython code that could write out the corresponding .pxd files and a simple >> 1:1 API skeleton. >> >> Has anyone looked into this already? This would be a wonderful tool for >> those who want to wrap larger libraries. > > I believe David threw a bunch of things together in that direction: > http://cournape.wordpress.com/2008/12/01/cython-codegen-cython-code-generator- > from-gccxml-files/ "Bunch of things" is a good description, it is nothing more than a hack. >, but it is now migrating towards using > LLVM/CLang. I wanted to use clang as well - what is the status of the C parsing ? It is usable ? cheers, David From dagss at student.matnat.uio.no Tue Aug 25 16:40:01 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 25 Aug 2009 16:40:01 +0200 Subject: [Cython] anyone using Py++ with Cython? In-Reply-To: <5b8d13220908250639h2b85c5d3od3168fd1d2946562@mail.gmail.com> References: <4A8D5E47.3030207@behnel.de> <200908241516.15870.binet@cern.ch> <5b8d13220908250639h2b85c5d3od3168fd1d2946562@mail.gmail.com> Message-ID: <4A93F7C1.8030906@student.matnat.uio.no> David Cournapeau wrote: > On Mon, Aug 24, 2009 at 8:16 AM, Sebastien Binet wrote: >> On Thursday 20 August 2009 16:31:35 Stefan Behnel wrote: >>> Hi, >>> >>> I just stumbled over the Py++ project that reads C++ code using pygccxml >>> and comes with wrapper code generators for Boost.Python and ctypes. >>> >>> http://www.language-binding.net/pyplusplus/pyplusplus.html >>> >>> >From a first glance, it seems not too hard to add a third generator for >>> >>> Cython code that could write out the corresponding .pxd files and a simple >>> 1:1 API skeleton. >>> >>> Has anyone looked into this already? This would be a wonderful tool for >>> those who want to wrap larger libraries. >> I believe David threw a bunch of things together in that direction: >> http://cournape.wordpress.com/2008/12/01/cython-codegen-cython-code-generator- >> from-gccxml-files/ > > "Bunch of things" is a good description, it is nothing more than a hack. > >> , but it is now migrating towards using >> LLVM/CLang. > > I wanted to use clang as well - what is the status of the C parsing ? > It is usable ? I didn't try it, but from the website: """ Clang is still under heavy development. Clang is considered to be a production quality C and Objective-C compiler when targetting X86-32 and X86-64 (other targets may have caveats, but are usually easy to fix). If you are looking for source analysis or source-to-source transformation tools, clang is probably a great solution for you. If you are interested in C++, full support is still way off. """ CPU support/output in general doesn't matter for us of course, so it seems like clang should be a workable solution for pure C code. -- Dag Sverre From josef.pktd at gmail.com Tue Aug 25 16:51:50 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 25 Aug 2009 10:51:50 -0400 Subject: [Cython] Cython 0.11.2 win32 installer In-Reply-To: <4A93B38C.9010001@student.matnat.uio.no> References: <96de71860908121538k98dd1e3ldaf6a567ad474b19@mail.gmail.com> <4A83C44D.9040107@student.matnat.uio.no> <96de71860908130133t2f97ed34he8ff5ba3eeabd517@mail.gmail.com> <5b8d13220908201644u3697c534v262ff2fc778f1664@mail.gmail.com> <1cd32cbb0908210405r39eaff1dyb6cfd15ba1a52e78@mail.gmail.com> <4A93B38C.9010001@student.matnat.uio.no> Message-ID: <1cd32cbb0908250751u55d6876du573863913287ccc6@mail.gmail.com> On Tue, Aug 25, 2009 at 5:49 AM, Dag Sverre Seljebotn wrote: > josef.pktd at gmail.com wrote: >> On Fri, Aug 21, 2009 at 2:36 AM, Fernando Perez wrote: >>> On Thu, Aug 20, 2009 at 4:44 PM, David Cournapeau wrote: >>>> On Thu, Aug 13, 2009 at 1:33 AM, Fernando >>>> Perez wrote: >>>>> By the way, David did this on a wine setup he has on his Mac laptop in >>>>> just a couple of minutes! ?So it seems that once you have the right >>>>> setup, it's not a huge deal. >>>> It is quite easy to set up: you can install wine for mac os x >>>> (darwine), install python + mingw into wine, and that's it. >>>> >>>> I have for example an alist winepy="WINEPATH >>>> $HOME/.wine/drive_c/Python25/python.exe", and then making the binary >>>> is just an issue of: >>>> >>>> winepy setup.py bdist_wininst >>>> >>>> Numpy and scipy binaries are actually entirely built under wine as >>>> well nowadays, it is a viable alternative to the endless windows pain. >>> I should add that Stefan vdW pointed out to me that the detailed >>> tutorial on how to set this wine machinery lives in fact on the Cython >>> wiki :) >>> >>> http://wiki.cython.org/BuildingWindowsInstaller >>> >>> I did it a few days ago, and I can now build a Cython installer in no time flat: >>> >>> maqroll[Cython-0.11.2]> uname -a >>> Linux maqroll 2.6.28-15-generic #49-Ubuntu SMP Tue Aug 18 18:40:08 UTC >>> 2009 i686 GNU/Linux >>> maqroll[Cython-0.11.2]> time wine python setup.py bdist_wininst >>> Compiling module Cython.Plex.Scanners ... >>> Compiling module Cython.Compiler.Scanning ... >>> Compiling module Cython.Compiler.Parsing ... >>> Compiling module Cython.Compiler.Visitor ... >>> Compiling module Cython.Runtime.refnanny ... >>> running bdist_wininst >>> >>> [... snip long output ] >>> >>> adding 'PLATLIB\pyximport\pyxbuild.py' >>> adding 'SCRIPTS\cython.py' >>> creating dist >>> removing 'build\bdist.win32\wininst' (and everything under it) >>> >>> real ? ?0m20.775s >>> user ? ?0m5.764s >>> sys ? ? 0m0.500s >>> >>> This is really great: the setup takes a few minutes, and now I don't >>> even have to fire up VirtualBox and spend any time in the godforsaken >>> hell that is Windows to build installers for code with C extensions! >> >> There is no reason for this language, there are a lot of us who like >> Windows, and if you look at the numpy/scipy mailinglist then the only >> build issues are with the hundreds of versions of unix/linux. > > Your statement here in turn provokes me. I'd like to try to stop short > of a flamewar, but I have to say something: > > Like David said, this is simply because Windows-users don't build their > own software!! A lot of time in open source is spent on Windows-support, > it's just that few of the typical mailing list participants build it > themselves! (Implication: They can't contribute to development versions!) > > Windows support takes a good deal of time from open source development > -- while my feeling is relatively few Windows users take the trouble to > contribute back (with some honorable exceptions, of course; here on the > list the recent pyximport patches have come from Windows users for > instance). > > (One quote in support of this heard during SciPy 09: All SymPy devs use > Linux or Mac. Yet the number of Windows downloads are much higher.) > > I'm always tempted to say that I believe Cython should simply say that > we do not support Windows, or Visual C, until we can AT THE VERY LEAST > find one user who actually use Windows on a daily basis who volunteers > to do the relatively trivial task of testing new releases and packaging > it for exe distribution about four times a year. > > -- > Dag Sverre I really appreciate all the work that non-Windows developers (especially David) are doing to support our platform, and I'm sorry if I sounded too harsh and thank you Fernando. a disclaimer and a few impressions from a Windows user: A disclaimer, I know very little about C, nothing about cross-platform differences in C, and I'm keeping track of cython mostly for future use in scipy but currently don't use it myself, except for some experimentation. I'm more interested in application programming than compiler and build issues and my main programming languages are matlab and python (and gauss). Additionally, I have been involved in any open source project for less than a year. And I'm using Windows since Windows95 came out and crashed on me every half hour with a nice blue screen. My main motivation for my initial reply and for comments in other scipy related mailing lists, is to avoid the impression that it is very difficult or impossible to build on Windows. With the mingw setup, it is relatively easy to build many SciPy related packages for Windows32. (As I said, I don't know anything about the underlying c code.) I think Windows users should be less worried or scared about building the packages themselves. I was when I tried to build scipy and numpy the first few times. It requires a one-time setup, but then it's just a call to setup.py. But the impression on the mailing lists is that, if there is a c extension, then a Windows user doesn't even need to try this impossible task. In spring, I helped to debug and fix the build and install problems for nipy on Windows with mingw, which was almost only a question of getting the infrastructure with mingw and numpy correctly set up (except for the bugs in nipy). If cython were to drop Windows support, it would be impossible for me to work on any cython code in scipy, and I expect that the share of cython code in numpy/scipy will increase over time. Other reasons for the lack of windows developers, I think, are, that windows user are proportionally more interested in application than low-level programming, and the chicken-and-egg problem, the preferred development style for Windows and Posix users is not exactly the same. (I'm a bit, but not too surprised about the computer choice of sympy developers, since git is not very windows friendly, but that's a different debate.) Thank you for the improvements in the numerical support in cython, I hope to be using it soon. Josef From adam.ginsburg at colorado.edu Tue Aug 25 17:19:44 2009 From: adam.ginsburg at colorado.edu (Adam Ginsburg) Date: Tue, 25 Aug 2009 09:19:44 -0600 Subject: [Cython] Linking error - 64 bit problem? Message-ID: > From: Robert Bradshaw > Subject: Re: [Cython] Linking error - 64 bit problem? > > On Aug 22, 2009, at 7:40 PM, Adam Ginsburg wrote: > >> I reinstalled cython from source (instead of easy_install-ing it) and >> my code compiled successfully. Sorry about the previous e-mail. > > Great--I saw your previous email and started to get worried. I > thought easy_install was a source install (we didn't ship any > binaries until recently, and then just windows installers). I'm not entirely sure what happened, but I have at least 3 python installs that I use routinely, so it was probably just a confusion issue on my end. >> Unfortunately, my code runs slower with cython than without, but at >> least now I know I can compile. In case anyone wants to offer >> assistance, the code I'm trying to optimize is: >> http://code.google.com/p/agpy/source/browse/trunk/plfit.py >> http://code.google.com/p/agpy/source/browse/trunk/cplfit.pyx > > My first piece of advice is to run cython -a over your code. This > will produce an html file which is most helpful in diagnosing where > you're wasting time. Just in a quick glance at your code I noticed: > > - You're calling Python's log, sqrt, max, abs, sum. This would negate > any gains you may hope to see. > - You're calling len a lot, either store it as a int somewhere, or > use .shape[0]. > - Unless n is huge, arange(n)/float(n) is probably way to expensive. > If the above don't help, try unrolling this. > > I bet there's a lot more performance that one could ring out of your > code, but the above should take you far. Much appreciated. I guess the various levels of yellow in the html file indicate the slow lines? I tried getting rid of all of my numpy calls in the loop by rewriting them as loops, but that hasn't improved speed at all, and in fact appears to have become slower. Right now a fortran (f2py) version goes ~75% faster and pure python goes ~25% faster, so I must be doing something wrong. Example: I changed this: dat[xm] = numpy.max(numpy.abs(cf-cx)) to this: for i from 0 <= i < n: val = abs(cf[i]-cx[i]) if dat[xm] < val: dat[xm] = val Is that the right approach for removing max/sum ? And is this: cdef extern from "math.h": double log(double theta) double sqrt(double theta) double abs(double theta) the right way to get rid of numpy log/abs/sqrt calls? If you'd rather see the whole code, it's been updated on the google code site. Thanks, Adam From sccolbert at gmail.com Tue Aug 25 17:54:08 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Tue, 25 Aug 2009 11:54:08 -0400 Subject: [Cython] Linking error - 64 bit problem? In-Reply-To: References: Message-ID: <7f014ea60908250854l13e913abjd24c42ed79c24ab5@mail.gmail.com> I'm no expert, but I would type your input arguments and use the float versions of the math.h functions since your dtypes are floats. On Tue, Aug 25, 2009 at 11:19 AM, Adam Ginsburg wrote: >> From: Robert Bradshaw >> Subject: Re: [Cython] Linking error - 64 bit problem? >> >> On Aug 22, 2009, at 7:40 PM, Adam Ginsburg wrote: >> >>> I reinstalled cython from source (instead of easy_install-ing it) and >>> my code compiled successfully. ?Sorry about the previous e-mail. >> >> Great--I saw your previous email and started to get worried. I >> thought easy_install was a source install (we didn't ship any >> binaries until recently, and then just windows installers). > > I'm not entirely sure what happened, but I have at least 3 python > installs that I use routinely, so it was probably just a confusion > issue on my end. > >>> Unfortunately, my code runs slower with cython than without, but at >>> least now I know I can compile. ?In case anyone wants to offer >>> assistance, the code I'm trying to optimize is: >>> http://code.google.com/p/agpy/source/browse/trunk/plfit.py >>> http://code.google.com/p/agpy/source/browse/trunk/cplfit.pyx >> >> My first piece of advice is to run cython -a over your code. This >> will produce an html file which is most helpful in diagnosing where >> you're wasting time. Just in a quick glance at your code I noticed: >> >> - You're calling Python's log, sqrt, max, abs, sum. This would negate >> any gains you may hope to see. >> - You're calling len a lot, either store it as a int somewhere, or >> use .shape[0]. >> - Unless n is huge, arange(n)/float(n) is probably way to expensive. >> If the above don't help, try unrolling this. >> >> I bet there's a lot more performance that one could ring out of your >> code, but the above should take you far. > > > Much appreciated. ?I guess the various levels of yellow in the html > file indicate the slow lines? ?I tried getting rid of all of my numpy > calls in the loop by rewriting them as loops, but that hasn't improved > speed at all, and in fact appears to have become slower. ?Right now a > fortran (f2py) version goes ~75% faster and pure python goes ~25% > faster, so I must be doing something wrong. > > Example: > I changed this: > ? ? ? dat[xm] = numpy.max(numpy.abs(cf-cx)) > > to this: > ? ? ? for i from 0 <= i < n: > ? ? ? ? ? val = abs(cf[i]-cx[i]) > ? ? ? ? ? if dat[xm] < val: > ? ? ? ? ? ? ? dat[xm] = val > > Is that the right approach for removing max/sum ? > > And is this: > cdef extern from "math.h": > ? double log(double theta) > ? double sqrt(double theta) > ? double abs(double theta) > > the right way to get rid of numpy log/abs/sqrt calls? > > If you'd rather see the whole code, it's been updated on the google code site. > > Thanks, > Adam > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From robertwb at math.washington.edu Tue Aug 25 18:23:10 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 25 Aug 2009 09:23:10 -0700 Subject: [Cython] Fwd: Type checking bug when passing None instead of a Cython object References: <4A93CC39.6020903@ebi.ac.uk> Message-ID: <995A6088-6BAC-4DF9-8360-733B2D88EFC7@math.washington.edu> > +--- > | Dr. Simon Anders, Dipl.-Phys. > | European Molecular Biology Laboratory (EMBL), Heidelberg > | office phone +49-6221-387-8632 > | preferred (permanent) e-mail: sanders at fs.tum.de > > > From: Simon Anders > Date: August 25, 2009 3:46:52 AM PDT > To: cython-dev at codespeak.net > Subject: Type checking bug when passing None instead of a Cython > object > > > Hi, > > I'm a rather new user of Cython but if I am not completely > mistaken, I've found a bug in the type checking of function arguments. > > Consider the following simple Cython module: > > ---8<--- test1.pyx --------- > > cdef class foo: > cdef public int value > def __init__( self, value ): > self.value = value > > def getValue( foo obj ): > return obj.value > > ---8<----------------------- > > Now observe how the function 'getValue' correctly accepts an > argument of class 'foo', refuses an argument of type 'str', but > fails to reject a 'None': > > ---8<----------------------- > > Python 2.5.2 (r252:60911, Oct 5 2008, 19:29:17) > [GCC 4.3.2] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > > >>> import test1 > > >>> f = test1.foo( 3 ) > > >>> test1.getValue( f ) > 3 > > >>> test1.getValue( "A" ) > Traceback (most recent call last): > File "", line 1, in > File "test1.pyx", line 8, in test1.getValue (test1.c:412) > def getValue( foo obj ): > TypeError: Argument 'obj' has incorrect type (expected test1.foo, > got str) > > >>> test1.getValue( None ) > 3 > > ---8<----------------------- > > > As you can see, 'None' was treated as an object of type 'foo', was > dereferenced and the slot 'value' accessed. By chance, we found a > 3, but we might as well have gotten a segfault (which is how I > stumbled over this in the first instance.) > > I've played a bit more with it, and it seems that a Cython function > accepts 'None' whenever it expects an object of a class that was > cdef'ed in the Cython module, while it correctly rejects 'None' > when it expects a native Python type. > > > Version information: The module was translated to C with Cython > version 0.11.2 (called without further arguments), compiled with > 'gcc `python-config --cflags` test1.c -shared -fPIC -o > test1.so' (using GCC 4.3.2) on Ubuntu Intrepid x86_64. > > > Thank a lot, by the way, for Cython; it is an incredible useful tool. > > > Best regards > Simon > > > > > +--- > | Dr. Simon Anders, Dipl.-Phys. > | European Molecular Biology Laboratory (EMBL), Heidelberg > | office phone +49-6221-387-8632 > | preferred (permanent) e-mail: sanders at fs.tum.de > > > > From robertwb at math.washington.edu Tue Aug 25 18:35:25 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 25 Aug 2009 09:35:25 -0700 Subject: [Cython] Cython 0.11.2 win32 installer In-Reply-To: <4A93B38C.9010001@student.matnat.uio.no> References: <96de71860908121538k98dd1e3ldaf6a567ad474b19@mail.gmail.com> <4A83C44D.9040107@student.matnat.uio.no> <96de71860908130133t2f97ed34he8ff5ba3eeabd517@mail.gmail.com> <5b8d13220908201644u3697c534v262ff2fc778f1664@mail.gmail.com> <1cd32cbb0908210405r39eaff1dyb6cfd15ba1a52e78@mail.gmail.com> <4A93B38C.9010001@student.matnat.uio.no> Message-ID: <5C142374-B981-47F5-AB1C-42D88EF91BE1@math.washington.edu> On Aug 25, 2009, at 2:49 AM, Dag Sverre Seljebotn wrote: >>> This is really great: the setup takes a few minutes, and now I don't >>> even have to fire up VirtualBox and spend any time in the >>> godforsaken >>> hell that is Windows to build installers for code with C extensions! >> >> There is no reason for this language, there are a lot of us who like >> Windows, and if you look at the numpy/scipy mailinglist then the only >> build issues are with the hundreds of versions of unix/linux. > > Your statement here in turn provokes me. I'd like to try to stop short > of a flamewar, but I have to say something: Maybe we need a cython-flames list. On a more serious note, thanks to all for keeping the discussion relatively civil. I think the main point above is the excitement that the important target audience using Windows can be more easily supported by those who don't enjoy using it. > I'm always tempted to say that I believe Cython should simply say that > we do not support Windows, or Visual C, until we can AT THE VERY LEAST > find one user who actually use Windows on a daily basis who volunteers > to do the relatively trivial task of testing new releases and > packaging > it for exe distribution about four times a year. I don't think we have to outright drop support, as it seems to mostly work most of the time (the messiness of distuitls is not our issue, as long as we get the .c files right). But it is untested and I would love for someone who knows and uses windows to step up and do testing an packaging. Perhaps much could be automated as above, but I wouldn't even know what I'm looking for. - Robert From robertwb at math.washington.edu Tue Aug 25 18:38:49 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 25 Aug 2009 09:38:49 -0700 Subject: [Cython] Linking error - 64 bit problem? In-Reply-To: References: Message-ID: <0CB2D49E-A536-4C4A-80C9-F9A84D1B3CBE@math.washington.edu> On Aug 25, 2009, at 8:19 AM, Adam Ginsburg wrote: >> From: Robert Bradshaw >> Subject: Re: [Cython] Linking error - 64 bit problem? >> >> On Aug 22, 2009, at 7:40 PM, Adam Ginsburg wrote: >> >>> I reinstalled cython from source (instead of easy_install-ing it) >>> and >>> my code compiled successfully. Sorry about the previous e-mail. >> >> Great--I saw your previous email and started to get worried. I >> thought easy_install was a source install (we didn't ship any >> binaries until recently, and then just windows installers). > > I'm not entirely sure what happened, but I have at least 3 python > installs that I use routinely, so it was probably just a confusion > issue on my end. > >>> Unfortunately, my code runs slower with cython than without, but at >>> least now I know I can compile. In case anyone wants to offer >>> assistance, the code I'm trying to optimize is: >>> http://code.google.com/p/agpy/source/browse/trunk/plfit.py >>> http://code.google.com/p/agpy/source/browse/trunk/cplfit.pyx >> >> My first piece of advice is to run cython -a over your code. This >> will produce an html file which is most helpful in diagnosing where >> you're wasting time. Just in a quick glance at your code I noticed: >> >> - You're calling Python's log, sqrt, max, abs, sum. This would negate >> any gains you may hope to see. >> - You're calling len a lot, either store it as a int somewhere, or >> use .shape[0]. >> - Unless n is huge, arange(n)/float(n) is probably way to expensive. >> If the above don't help, try unrolling this. >> >> I bet there's a lot more performance that one could ring out of your >> code, but the above should take you far. > > > Much appreciated. I guess the various levels of yellow in the html > file indicate the slow lines? Sort-of. Yellow indicates Python/C API calls, which may or may not be worth the overhead. > I tried getting rid of all of my numpy > calls in the loop by rewriting them as loops, but that hasn't improved > speed at all, and in fact appears to have become slower. Right now a > fortran (f2py) version goes ~75% faster and pure python goes ~25% > faster, so I must be doing something wrong. Don't have time to look at your code now, but you shouldn't be getting slower. > Example: > I changed this: > dat[xm] = numpy.max(numpy.abs(cf-cx)) > > to this: > for i from 0 <= i < n: > val = abs(cf[i]-cx[i]) > if dat[xm] < val: > dat[xm] = val > > Is that the right approach for removing max/sum ? Using numpy.max instead of Python's max is a clear win. Both should be tight C loops, so if your array is long enough than it's not a clear win. > And is this: > cdef extern from "math.h": > double log(double theta) > double sqrt(double theta) > double abs(double theta) > > the right way to get rid of numpy log/abs/sqrt calls? Yes. > If you'd rather see the whole code, it's been updated on the google > code site. > > Thanks, > Adam > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From cournape at gmail.com Tue Aug 25 18:55:24 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 25 Aug 2009 11:55:24 -0500 Subject: [Cython] Cython 0.11.2 win32 installer In-Reply-To: <5C142374-B981-47F5-AB1C-42D88EF91BE1@math.washington.edu> References: <96de71860908121538k98dd1e3ldaf6a567ad474b19@mail.gmail.com> <4A83C44D.9040107@student.matnat.uio.no> <96de71860908130133t2f97ed34he8ff5ba3eeabd517@mail.gmail.com> <5b8d13220908201644u3697c534v262ff2fc778f1664@mail.gmail.com> <1cd32cbb0908210405r39eaff1dyb6cfd15ba1a52e78@mail.gmail.com> <4A93B38C.9010001@student.matnat.uio.no> <5C142374-B981-47F5-AB1C-42D88EF91BE1@math.washington.edu> Message-ID: <5b8d13220908250955x1f9b8017j24a01f23d9f0853d@mail.gmail.com> On Tue, Aug 25, 2009 at 11:35 AM, Robert Bradshaw wrote: > On Aug 25, 2009, at 2:49 AM, Dag Sverre Seljebotn wrote: > >>>> This is really great: the setup takes a few minutes, and now I don't >>>> even have to fire up VirtualBox and spend any time in the >>>> godforsaken >>>> hell that is Windows to build installers for code with C extensions! >>> >>> There is no reason for this language, there are a lot of us who like >>> Windows, and if you look at the numpy/scipy mailinglist then the only >>> build issues are with the hundreds of versions of unix/linux. >> >> Your statement here in turn provokes me. I'd like to try to stop short >> of a flamewar, but I have to say something: > > Maybe we need a cython-flames list. On a more serious note, thanks to > all for keeping the discussion relatively civil. I think the main > point above is the excitement that the important target audience > using Windows can be more easily supported by those who don't enjoy > using it. > >> I'm always tempted to say that I believe Cython should simply say that >> we do not support Windows, or Visual C, until we can AT THE VERY LEAST >> find one user who actually use Windows on a daily basis who volunteers >> to do the relatively trivial task of testing new releases and >> packaging >> it for exe distribution about four times a year. > > > I don't think we have to outright drop support, as it seems to mostly > work most of the time (the messiness of distuitls is not our issue, > as long as we get the .c files right). But it is untested and I would > love for someone who knows and uses windows to step up and do testing > an packaging. Perhaps much could be automated as above, but I > wouldn't even know what I'm looking for. as long as you are looking for: - purely command line - don't care about compilers other than mingw - don't care about windows x64 wine is really the best solution IMHO for people who don't enjoy using windows. You can test things, install python and quite a few dependencies, generate binaries. You should always test the final product on windows in the end, though. cheers, David From cournape at gmail.com Tue Aug 25 18:56:53 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 25 Aug 2009 11:56:53 -0500 Subject: [Cython] Cython 0.11.2 win32 installer In-Reply-To: <5b8d13220908250955x1f9b8017j24a01f23d9f0853d@mail.gmail.com> References: <96de71860908121538k98dd1e3ldaf6a567ad474b19@mail.gmail.com> <4A83C44D.9040107@student.matnat.uio.no> <96de71860908130133t2f97ed34he8ff5ba3eeabd517@mail.gmail.com> <5b8d13220908201644u3697c534v262ff2fc778f1664@mail.gmail.com> <1cd32cbb0908210405r39eaff1dyb6cfd15ba1a52e78@mail.gmail.com> <4A93B38C.9010001@student.matnat.uio.no> <5C142374-B981-47F5-AB1C-42D88EF91BE1@math.washington.edu> <5b8d13220908250955x1f9b8017j24a01f23d9f0853d@mail.gmail.com> Message-ID: <5b8d13220908250956r2fb7847cib8352c833dcac4be@mail.gmail.com> On Tue, Aug 25, 2009 at 11:55 AM, David Cournapeau wrote: > > wine is really the best solution IMHO for people who don't enjoy using > windows. I forgot to add: wine can be scripted easily from bash, which is really a big help, since native windows scripting abilities are extremely weak. David From dagss at student.matnat.uio.no Tue Aug 25 19:01:32 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 25 Aug 2009 19:01:32 +0200 Subject: [Cython] Fwd: Type checking bug when passing None instead of a Cython object In-Reply-To: <995A6088-6BAC-4DF9-8360-733B2D88EFC7@math.washington.edu> References: <4A93CC39.6020903@ebi.ac.uk> <995A6088-6BAC-4DF9-8360-733B2D88EFC7@math.washington.edu> Message-ID: <4A9418EC.4050709@student.matnat.uio.no> Robert Bradshaw wrote: >> +--- >> | Dr. Simon Anders, Dipl.-Phys. >> | European Molecular Biology Laboratory (EMBL), Heidelberg >> | office phone +49-6221-387-8632 >> | preferred (permanent) e-mail: sanders at fs.tum.de >> >> >> From: Simon Anders >> Date: August 25, 2009 3:46:52 AM PDT >> To: cython-dev at codespeak.net >> Subject: Type checking bug when passing None instead of a Cython >> object >> >> >> Hi, >> >> I'm a rather new user of Cython but if I am not completely >> mistaken, I've found a bug in the type checking of function arguments. >> >> Consider the following simple Cython module: >> >> ---8<--- test1.pyx --------- >> >> cdef class foo: >> cdef public int value >> def __init__( self, value ): >> self.value = value >> >> def getValue( foo obj ): >> return obj.value >> >> ---8<----------------------- >> >> Now observe how the function 'getValue' correctly accepts an >> argument of class 'foo', refuses an argument of type 'str', but >> fails to reject a 'None': >> >> ---8<----------------------- >> >> Python 2.5.2 (r252:60911, Oct 5 2008, 19:29:17) >> [GCC 4.3.2] on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >> >>>>> import test1 >>>>> f = test1.foo( 3 ) >>>>> test1.getValue( f ) >> 3 >> >>>>> test1.getValue( "A" ) >> Traceback (most recent call last): >> File "", line 1, in >> File "test1.pyx", line 8, in test1.getValue (test1.c:412) >> def getValue( foo obj ): >> TypeError: Argument 'obj' has incorrect type (expected test1.foo, >> got str) >> >>>>> test1.getValue( None ) >> 3 >> >> ---8<----------------------- >> >> >> As you can see, 'None' was treated as an object of type 'foo', was >> dereferenced and the slot 'value' accessed. By chance, we found a >> 3, but we might as well have gotten a segfault (which is how I >> stumbled over this in the first instance.) Actually, this is documented and relied-upon behaviour (None is used as a "blank", "no object", in much control flow). See e.g. http://thread.gmane.org/gmane.comp.python.cython.devel/7041 The short story is, you can put #cython: nonecheck=True at the top of your file and all access will be checked for None, but it will slow things down. Alternatively, def foo(MyClass a not None): <...> is a short-hand for def foo(MyClass a): if a is None: raise <...> <...> (It would be great if this was added to the FAQ, it tends to come up a lot. Perhaps we should switch to making nonecheck the default?) -- Dag Sverre From dagss at student.matnat.uio.no Tue Aug 25 19:07:19 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 25 Aug 2009 19:07:19 +0200 Subject: [Cython] Cython 0.11.2 win32 installer In-Reply-To: <5b8d13220908250955x1f9b8017j24a01f23d9f0853d@mail.gmail.com> References: <96de71860908121538k98dd1e3ldaf6a567ad474b19@mail.gmail.com> <4A83C44D.9040107@student.matnat.uio.no> <96de71860908130133t2f97ed34he8ff5ba3eeabd517@mail.gmail.com> <5b8d13220908201644u3697c534v262ff2fc778f1664@mail.gmail.com> <1cd32cbb0908210405r39eaff1dyb6cfd15ba1a52e78@mail.gmail.com> <4A93B38C.9010001@student.matnat.uio.no> <5C142374-B981-47F5-AB1C-42D88EF91BE1@math.washington.edu> <5b8d13220908250955x1f9b8017j24a01f23d9f0853d@mail.gmail.com> Message-ID: <4A941A47.3090803@student.matnat.uio.no> David Cournapeau wrote: > On Tue, Aug 25, 2009 at 11:35 AM, Robert > Bradshaw wrote: >> On Aug 25, 2009, at 2:49 AM, Dag Sverre Seljebotn wrote: >> >>>>> This is really great: the setup takes a few minutes, and now I don't >>>>> even have to fire up VirtualBox and spend any time in the >>>>> godforsaken >>>>> hell that is Windows to build installers for code with C extensions! >>>> There is no reason for this language, there are a lot of us who like >>>> Windows, and if you look at the numpy/scipy mailinglist then the only >>>> build issues are with the hundreds of versions of unix/linux. >>> Your statement here in turn provokes me. I'd like to try to stop short >>> of a flamewar, but I have to say something: >> Maybe we need a cython-flames list. On a more serious note, thanks to >> all for keeping the discussion relatively civil. I think the main >> point above is the excitement that the important target audience >> using Windows can be more easily supported by those who don't enjoy >> using it. >> >>> I'm always tempted to say that I believe Cython should simply say that >>> we do not support Windows, or Visual C, until we can AT THE VERY LEAST >>> find one user who actually use Windows on a daily basis who volunteers >>> to do the relatively trivial task of testing new releases and >>> packaging >>> it for exe distribution about four times a year. >> >> I don't think we have to outright drop support, as it seems to mostly >> work most of the time (the messiness of distuitls is not our issue, >> as long as we get the .c files right). But it is untested and I would >> love for someone who knows and uses windows to step up and do testing >> an packaging. Perhaps much could be automated as above, but I >> wouldn't even know what I'm looking for. > > as long as you are looking for: > - purely command line > - don't care about compilers other than mingw > - don't care about windows x64 Hmm. If you wanted to start writing parts of NumPy in Cython, you'd need for Cython to support Windows x64 and Visual C though? (Background note: There's been some talk about that, although no decision has been made to my knowledge. scipy's got some Cython code in it though.) -- Dag Sverre From Chris.Barker at noaa.gov Tue Aug 25 20:00:06 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 25 Aug 2009 11:00:06 -0700 Subject: [Cython] Fwd: Type checking bug when passing None instead of a Cython object In-Reply-To: <4A9418EC.4050709@student.matnat.uio.no> References: <4A93CC39.6020903@ebi.ac.uk> <995A6088-6BAC-4DF9-8360-733B2D88EFC7@math.washington.edu> <4A9418EC.4050709@student.matnat.uio.no> Message-ID: <4A9426A6.4050007@noaa.gov> Dag Sverre Seljebotn wrote: > (It would be great if this was added to the FAQ, it tends to come up a > lot. Done: http://wiki.cython.org/FAQ#WhydoesafunctionthatIhaveprovidedcdef.27dparametertoacceptNone.3F Could someone check for accuracy? > Perhaps we should switch to making nonecheck the default?) I'd rather have "not None" the default, but there is backwards compatibility to consider. Does the Cython project have any policy about deprecation procedures? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From dagss at student.matnat.uio.no Tue Aug 25 20:05:16 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 25 Aug 2009 20:05:16 +0200 Subject: [Cython] Fwd: Type checking bug when passing None instead of a Cython object In-Reply-To: <4A9426A6.4050007@noaa.gov> References: <4A93CC39.6020903@ebi.ac.uk> <995A6088-6BAC-4DF9-8360-733B2D88EFC7@math.washington.edu> <4A9418EC.4050709@student.matnat.uio.no> <4A9426A6.4050007@noaa.gov> Message-ID: <4A9427DC.6060307@student.matnat.uio.no> Christopher Barker wrote: > Dag Sverre Seljebotn wrote: >> (It would be great if this was added to the FAQ, it tends to come up a >> lot. > > Done: > > http://wiki.cython.org/FAQ#WhydoesafunctionthatIhaveprovidedcdef.27dparametertoacceptNone.3F > > Could someone check for accuracy? > >> Perhaps we should switch to making nonecheck the default?) > > I'd rather have "not None" the default, but there is backwards > compatibility to consider. Does the Cython project have any policy about > deprecation procedures? (Well, at least two people (myself and Robert) has opposed this being the default (and not for reasons of backwards compatability!), so I don't think we're at the stage where we can discuss deprecation just yet :-)) I think it's done on a case-by-case basis. As an example though, we're currently, in 0.11, deprecating the current integer division semantics by printing warnings; they will then be changed in 0.12. -- Dag Sverre From dagss at student.matnat.uio.no Tue Aug 25 20:10:54 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 25 Aug 2009 20:10:54 +0200 Subject: [Cython] Fwd: Type checking bug when passing None instead of a Cython object In-Reply-To: <4A9427DC.6060307@student.matnat.uio.no> References: <4A93CC39.6020903@ebi.ac.uk> <995A6088-6BAC-4DF9-8360-733B2D88EFC7@math.washington.edu> <4A9418EC.4050709@student.matnat.uio.no> <4A9426A6.4050007@noaa.gov> <4A9427DC.6060307@student.matnat.uio.no> Message-ID: <4A94292E.7020907@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Christopher Barker wrote: >> Dag Sverre Seljebotn wrote: >>> (It would be great if this was added to the FAQ, it tends to come up a >>> lot. >> Done: >> >> http://wiki.cython.org/FAQ#WhydoesafunctionthatIhaveprovidedcdef.27dparametertoacceptNone.3F >> >> Could someone check for accuracy? >> >>> Perhaps we should switch to making nonecheck the default?) >> I'd rather have "not None" the default, but there is backwards >> compatibility to consider. Does the Cython project have any policy about >> deprecation procedures? > > (Well, at least two people (myself and Robert) has opposed this being > the default (and not for reasons of backwards compatability!), so I > don't think we're at the stage where we can discuss deprecation just yet > :-)) In fact, I think I remember me and Robert discussing the possibility of deprecating the "not None" feature from Cython instead at some point (though I might be mistaken here). If a faster "nonecheck" was added then "not None" would be a really redundant part of the language. So I wouldn't phrase the FAQ entry just like that myself, although the important thing is that something got added :-( Thanks! (If something comes out of this discussion and there's consensus for whether we should endorse "not None" or not in examples we can look at changing it, but I'll leave it as is now. Myself I tend to advocate adding an if-test in the function instead.) -- Dag Sverre From wstein at gmail.com Tue Aug 25 20:02:29 2009 From: wstein at gmail.com (William Stein) Date: Tue, 25 Aug 2009 11:02:29 -0700 Subject: [Cython] Cython 0.11.2 win32 installer In-Reply-To: <5b8d13220908250955x1f9b8017j24a01f23d9f0853d@mail.gmail.com> References: <96de71860908121538k98dd1e3ldaf6a567ad474b19@mail.gmail.com> <4A83C44D.9040107@student.matnat.uio.no> <96de71860908130133t2f97ed34he8ff5ba3eeabd517@mail.gmail.com> <5b8d13220908201644u3697c534v262ff2fc778f1664@mail.gmail.com> <1cd32cbb0908210405r39eaff1dyb6cfd15ba1a52e78@mail.gmail.com> <4A93B38C.9010001@student.matnat.uio.no> <5C142374-B981-47F5-AB1C-42D88EF91BE1@math.washington.edu> <5b8d13220908250955x1f9b8017j24a01f23d9f0853d@mail.gmail.com> Message-ID: <85e81ba30908251102r5798fc1ds20423055e068bb57@mail.gmail.com> On Tue, Aug 25, 2009 at 9:55 AM, David Cournapeau wrote: > On Tue, Aug 25, 2009 at 11:35 AM, Robert > Bradshaw wrote: >> On Aug 25, 2009, at 2:49 AM, Dag Sverre Seljebotn wrote: >> >>>>> This is really great: the setup takes a few minutes, and now I don't >>>>> even have to fire up VirtualBox and spend any time in the >>>>> godforsaken >>>>> hell that is Windows to build installers for code with C extensions! >>>> >>>> There is no reason for this language, there are a lot of us who like >>>> Windows, and if you look at the numpy/scipy mailinglist then the only >>>> build issues are with the hundreds of versions of unix/linux. >>> >>> Your statement here in turn provokes me. I'd like to try to stop short >>> of a flamewar, but I have to say something: >> >> Maybe we need a cython-flames list. On a more serious note, thanks to >> all for keeping the discussion relatively civil. I think the main >> point above is the excitement that the important target audience >> using Windows can be more easily supported by those who don't enjoy >> using it. >> >>> I'm always tempted to say that I believe Cython should simply say that >>> we do not support Windows, or Visual C, until we can AT THE VERY LEAST >>> find one user who actually use Windows on a daily basis who volunteers >>> to do the relatively trivial task of testing new releases and >>> packaging >>> it for exe distribution about four times a year. >> >> >> I don't think we have to outright drop support, as it seems to mostly >> work most of the time (the messiness of distuitls is not our issue, >> as long as we get the .c files right). But it is untested and I would >> love for someone who knows and uses windows to step up and do testing >> an packaging. Perhaps much could be automated as above, but I >> wouldn't even know what I'm looking for. > > as long as you are looking for: > ?- ?purely command line > ?- don't care about compilers other than mingw > ?- don't care about windows x64 > > wine is really the best solution IMHO for people who don't enjoy using > windows. You can test things, install python and quite a few > dependencies, generate binaries. You should always test the final > product on windows in the end, though. Is it possible these days to build Python from source using mingw + wine? You suggest that above, and this makes me very excited, because it used to be very hard to do that. William From cournape at gmail.com Tue Aug 25 20:09:57 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 25 Aug 2009 13:09:57 -0500 Subject: [Cython] Cython 0.11.2 win32 installer In-Reply-To: <4A941A47.3090803@student.matnat.uio.no> References: <96de71860908121538k98dd1e3ldaf6a567ad474b19@mail.gmail.com> <4A83C44D.9040107@student.matnat.uio.no> <96de71860908130133t2f97ed34he8ff5ba3eeabd517@mail.gmail.com> <5b8d13220908201644u3697c534v262ff2fc778f1664@mail.gmail.com> <1cd32cbb0908210405r39eaff1dyb6cfd15ba1a52e78@mail.gmail.com> <4A93B38C.9010001@student.matnat.uio.no> <5C142374-B981-47F5-AB1C-42D88EF91BE1@math.washington.edu> <5b8d13220908250955x1f9b8017j24a01f23d9f0853d@mail.gmail.com> <4A941A47.3090803@student.matnat.uio.no> Message-ID: <5b8d13220908251109t493d0db4u18c21f58ddf9a5c4@mail.gmail.com> On Tue, Aug 25, 2009 at 12:07 PM, Dag Sverre Seljebotn wrote: > > Hmm. If you wanted to start writing parts of NumPy in Cython, you'd need > for Cython to support Windows x64 and Visual C though? Yes, but that's a bit different I believe: you want cython to output C code which is "visual studio compliant", but you don't need visual studio to build cython or even output compatible C code. Wine does not help you for the first part (at least as long as VS does not work on wine), but is very helpful for the second part. > > (Background note: There's been some talk about that, although no > decision has been made to my knowledge. scipy's got some Cython code in > it though.) Numpy random is generated from cython now instead of pyrex - that's how we discovered a few "bugs" in cython related to MS compilers compliance. cheers, David From dagss at student.matnat.uio.no Tue Aug 25 20:12:23 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 25 Aug 2009 20:12:23 +0200 Subject: [Cython] Fwd: Type checking bug when passing None instead of a Cython object In-Reply-To: <4A94292E.7020907@student.matnat.uio.no> References: <4A93CC39.6020903@ebi.ac.uk> <995A6088-6BAC-4DF9-8360-733B2D88EFC7@math.washington.edu> <4A9418EC.4050709@student.matnat.uio.no> <4A9426A6.4050007@noaa.gov> <4A9427DC.6060307@student.matnat.uio.no> <4A94292E.7020907@student.matnat.uio.no> Message-ID: <4A942987.1080903@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Dag Sverre Seljebotn wrote: >> Christopher Barker wrote: >>> Dag Sverre Seljebotn wrote: >>>> (It would be great if this was added to the FAQ, it tends to come up a >>>> lot. >>> Done: >>> >>> http://wiki.cython.org/FAQ#WhydoesafunctionthatIhaveprovidedcdef.27dparametertoacceptNone.3F >>> >>> Could someone check for accuracy? >>> >>>> Perhaps we should switch to making nonecheck the default?) >>> I'd rather have "not None" the default, but there is backwards >>> compatibility to consider. Does the Cython project have any policy about >>> deprecation procedures? >> (Well, at least two people (myself and Robert) has opposed this being >> the default (and not for reasons of backwards compatability!), so I >> don't think we're at the stage where we can discuss deprecation just yet >> :-)) > > In fact, I think I remember me and Robert discussing the possibility of > deprecating the "not None" feature from Cython instead at some point > (though I might be mistaken here). If a faster "nonecheck" was added > then "not None" would be a really redundant part of the language. > > So I wouldn't phrase the FAQ entry just like that myself, although the > important thing is that something got added :-( Thanks! (If something Whoops. I got the smiley the wrong way, it should be ":-)" !! Your effort is appriciated! -- Dag Sverre From cournape at gmail.com Tue Aug 25 20:12:52 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 25 Aug 2009 13:12:52 -0500 Subject: [Cython] Cython 0.11.2 win32 installer In-Reply-To: <85e81ba30908251102r5798fc1ds20423055e068bb57@mail.gmail.com> References: <96de71860908121538k98dd1e3ldaf6a567ad474b19@mail.gmail.com> <4A83C44D.9040107@student.matnat.uio.no> <96de71860908130133t2f97ed34he8ff5ba3eeabd517@mail.gmail.com> <5b8d13220908201644u3697c534v262ff2fc778f1664@mail.gmail.com> <1cd32cbb0908210405r39eaff1dyb6cfd15ba1a52e78@mail.gmail.com> <4A93B38C.9010001@student.matnat.uio.no> <5C142374-B981-47F5-AB1C-42D88EF91BE1@math.washington.edu> <5b8d13220908250955x1f9b8017j24a01f23d9f0853d@mail.gmail.com> <85e81ba30908251102r5798fc1ds20423055e068bb57@mail.gmail.com> Message-ID: <5b8d13220908251112q104e245ft485e1eff6d23627a@mail.gmail.com> On Tue, Aug 25, 2009 at 1:02 PM, William Stein wrote: > On Tue, Aug 25, 2009 at 9:55 AM, David Cournapeau wrote: >> On Tue, Aug 25, 2009 at 11:35 AM, Robert >> Bradshaw wrote: >>> On Aug 25, 2009, at 2:49 AM, Dag Sverre Seljebotn wrote: >>> >>>>>> This is really great: the setup takes a few minutes, and now I don't >>>>>> even have to fire up VirtualBox and spend any time in the >>>>>> godforsaken >>>>>> hell that is Windows to build installers for code with C extensions! >>>>> >>>>> There is no reason for this language, there are a lot of us who like >>>>> Windows, and if you look at the numpy/scipy mailinglist then the only >>>>> build issues are with the hundreds of versions of unix/linux. >>>> >>>> Your statement here in turn provokes me. I'd like to try to stop short >>>> of a flamewar, but I have to say something: >>> >>> Maybe we need a cython-flames list. On a more serious note, thanks to >>> all for keeping the discussion relatively civil. I think the main >>> point above is the excitement that the important target audience >>> using Windows can be more easily supported by those who don't enjoy >>> using it. >>> >>>> I'm always tempted to say that I believe Cython should simply say that >>>> we do not support Windows, or Visual C, until we can AT THE VERY LEAST >>>> find one user who actually use Windows on a daily basis who volunteers >>>> to do the relatively trivial task of testing new releases and >>>> packaging >>>> it for exe distribution about four times a year. >>> >>> >>> I don't think we have to outright drop support, as it seems to mostly >>> work most of the time (the messiness of distuitls is not our issue, >>> as long as we get the .c files right). But it is untested and I would >>> love for someone who knows and uses windows to step up and do testing >>> an packaging. Perhaps much could be automated as above, but I >>> wouldn't even know what I'm looking for. >> >> as long as you are looking for: >> ?- ?purely command line >> ?- don't care about compilers other than mingw >> ?- don't care about windows x64 >> >> wine is really the best solution IMHO for people who don't enjoy using >> windows. You can test things, install python and quite a few >> dependencies, generate binaries. You should always test the final >> product on windows in the end, though. > > Is it possible these days to build Python from source using mingw + wine? This I don't know - I just install python from python.org, install mingw in wine, and build numpy/scipy from it. It is literally as easy as downloading the python/mingw binaries, installing them, and modifying one wine config file for mingw to be in wine path. I would avoid building python with mingw, unless you are ready to build everything, as extensions start to be incompatible in quite subtle ways (it will import ok but will crash, for example). cheers, David From cournape at gmail.com Tue Aug 25 20:21:43 2009 From: cournape at gmail.com (David Cournapeau) Date: Tue, 25 Aug 2009 13:21:43 -0500 Subject: [Cython] Cython 0.11.2 win32 installer In-Reply-To: <5b8d13220908251109t493d0db4u18c21f58ddf9a5c4@mail.gmail.com> References: <96de71860908121538k98dd1e3ldaf6a567ad474b19@mail.gmail.com> <96de71860908130133t2f97ed34he8ff5ba3eeabd517@mail.gmail.com> <5b8d13220908201644u3697c534v262ff2fc778f1664@mail.gmail.com> <1cd32cbb0908210405r39eaff1dyb6cfd15ba1a52e78@mail.gmail.com> <4A93B38C.9010001@student.matnat.uio.no> <5C142374-B981-47F5-AB1C-42D88EF91BE1@math.washington.edu> <5b8d13220908250955x1f9b8017j24a01f23d9f0853d@mail.gmail.com> <4A941A47.3090803@student.matnat.uio.no> <5b8d13220908251109t493d0db4u18c21f58ddf9a5c4@mail.gmail.com> Message-ID: <5b8d13220908251121u40ba72f8yfa5e4395e524c5b8@mail.gmail.com> On Tue, Aug 25, 2009 at 1:09 PM, David Cournapeau wrote: > On Tue, Aug 25, 2009 at 12:07 PM, Dag Sverre > Seljebotn wrote: > >> >> Hmm. If you wanted to start writing parts of NumPy in Cython, you'd need >> for Cython to support Windows x64 and Visual C though? > > Yes, but that's a bit different I believe: you want cython to output C > code which is "visual studio compliant", but you don't need visual > studio to build cython or even output compatible C code. Wine does not > help you for the first part (at least as long as VS does not work on > wine), but is very helpful for the second part. Of course, I meant the contrary: wine helps you with building binaries for cython and supporting windows users much better, but is not so helpful if you need to test things w.r.t. a typical windows *dev* environment, as Visual Studio cannot be installed (yet) in wine. cheers, David From seb.binet at gmail.com Tue Aug 25 20:25:09 2009 From: seb.binet at gmail.com (Sebastien Binet) Date: Tue, 25 Aug 2009 20:25:09 +0200 Subject: [Cython] scipy'09 cython BoF Message-ID: <200908252025.09211.binet@cern.ch> hi there, IIRC there was a cython oriented BoF at last week's scipy conference. anything interesting ? (lame attempt at having a good soul summarizing or even just outlining what happened for the poor guys from the other side of the big pond) cheers, sebastien. -- ######################################### # Dr. Sebastien Binet # Laboratoire de l'Accelerateur Lineaire # Universite Paris-Sud XI # Batiment 200 # 91898 Orsay ######################################### From Chris.Barker at noaa.gov Tue Aug 25 20:36:51 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Tue, 25 Aug 2009 11:36:51 -0700 Subject: [Cython] Fwd: Type checking bug when passing None instead of a Cython object In-Reply-To: <4A942987.1080903@student.matnat.uio.no> References: <4A93CC39.6020903@ebi.ac.uk> <995A6088-6BAC-4DF9-8360-733B2D88EFC7@math.washington.edu> <4A9418EC.4050709@student.matnat.uio.no> <4A9426A6.4050007@noaa.gov> <4A9427DC.6060307@student.matnat.uio.no> <4A94292E.7020907@student.matnat.uio.no> <4A942987.1080903@student.matnat.uio.no> Message-ID: <4A942F43.30004@noaa.gov> Dag Sverre Seljebotn wrote: >>> (Well, at least two people (myself and Robert) has opposed this being >>> the default (and not for reasons of backwards compatability!), so I >>> don't think we're at the stage where we can discuss deprecation just yet >>> :-)) My point was that IF any backward incompatible changes were made, there should be a reasonable deprecation procedure. Whether the change should be made or not I'll leave to others -- what seems obvious to me could well be a function of my very limited use of Cython so far. What I do know is that the current behavior is a surprise to newbies -- a bit more documentation may be all we need. >> If a faster "nonecheck" was added >> then "not None" would be a really redundant part of the language. Is that possible (well, not "faster", but fast enough not to notice). My understanding of nonecheck is that a check has to be done on every object access (Or index, or...). In a tight loop that indexes through a numpy array and does very little, that just doesn't seem practical. Or could you put the check outside the loop, but not at the beginning of the function? >> So I wouldn't phrase the FAQ entry just like that myself, feel free to edit, of course. >> Myself I tend to advocate adding an if-test in the function instead.) Why is that? I like "not none" it's clear, and it seems belongs in a type declaration. But anything that works... > Whoops. I got the smiley the wrong way, it should be ":-)" no problem -- I read it as a smiley anyway. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From dagss at student.matnat.uio.no Tue Aug 25 20:45:26 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 25 Aug 2009 20:45:26 +0200 Subject: [Cython] scipy'09 cython BoF In-Reply-To: <200908252025.09211.binet@cern.ch> References: <200908252025.09211.binet@cern.ch> Message-ID: <4A943146.5060306@student.matnat.uio.no> Sebastien Binet wrote: > hi there, > > IIRC there was a cython oriented BoF at last week's scipy conference. > > anything interesting ? > (lame attempt at having a good soul summarizing or even just outlining what > happened for the poor guys from the other side of the big pond) I meant to get around to summarize SciPy 09 w.r.t. Cython for all of you who weren't there, just give me another day to let it all sink in :-) -- Dag Sverre From kwmsmith at gmail.com Tue Aug 25 21:08:30 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Tue, 25 Aug 2009 14:08:30 -0500 Subject: [Cython] scipy'09 cython BoF In-Reply-To: <200908252025.09211.binet@cern.ch> References: <200908252025.09211.binet@cern.ch> Message-ID: On Tue, Aug 25, 2009 at 1:25 PM, Sebastien Binet wrote: > hi there, > > IIRC there was a cython oriented BoF at last week's scipy conference. > > anything interesting ? > (lame attempt at having a good soul summarizing or even just outlining what > happened for the poor guys from the other side of the big pond) I thought so, although I was privy to just a part of the conversation. Dag or others, feel free to fill in parts that I'm missing. Rough outline of BoF topics (more of a brain-dump, actually): Memoryviews & Memoryviewslices (see: http://wiki.cython.org/enhancements/array) Dag had an extended conversation with the attendees about the semantics of these arrays -- when copying should occur, when views should be taken, etc. SIMD operations on Cython arrays / Memoryviews My impression was that there was general support for this feature, although this isn't surprising since we were at a numerical computing conference ;-) Dag discussed a few possibilities for how it would work, noting that this particular aspect of Cython arrays would need a vote & acceptance in the Cython community and core-devs. A numexpr backend is one possibility. Others brought up the possibility of incoporating OpenMP directives in Cython somehow. Another thread in this topic was support for generalized ufuncs in Cython (see: http://projects.scipy.org/numpy/browser/trunk/doc/neps/generalized-ufuncs.rst), which very easily extend to SIMD operations. Dag, you'll have to fill in here. fwrap: A few of us off in the corner discussed fwrap and what remains to be done to get the first release out. The fellas I spoke to work on Clawpack at the U-Washington, and they're very interested in getting callbacks working well. They even offered to pitch in. So that's encouraging. Topics listed but not discussed at the BoF (but certainly worth mentioning ;-): C++ & Cython integration -- Damian Eads has a nice C++ Container <-> Python container tool called Convert-xy (see: http://code.google.com/p/convert-xy/). If integrated in Cython, it could ease the use of STL containers in Cython. Weave-like support in Cython -- Fernando Perez (not at the BoF) suggested that Cython could assume the same functionality of scipy.weave (see: http://www.scipy.org/Weave), since scipy.weave is poorly documented and could use more maintenance than it is currently getting. I'm personally very interested in this -- it seems that there is much overlap between scipy.weave and the combination of pyximport & the pure-python mode of Cython. It seems that with a bit of work from an interested user and some guidance from yours truly, we could improve pyximport & Cython's pure-python mode to incorporate the good stuff from scipy.weave, but better ;-) I intend to write-up a CEP with my thoughts sometime in the near future. I know there are other topics & significant details that I'm forgetting. Kurt > > cheers, > sebastien. > -- > ######################################### > # Dr. Sebastien Binet > # Laboratoire de l'Accelerateur Lineaire > # Universite Paris-Sud XI > # Batiment 200 > # 91898 Orsay > ######################################### > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From kwmsmith at gmail.com Tue Aug 25 21:18:32 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Tue, 25 Aug 2009 14:18:32 -0500 Subject: [Cython] Giving fwrap a home Message-ID: I think its best that fwrap be its own package, distributed separately from Cython. That was the vibe I got from those at the SciPy 2009 conference and from the recent thread on Fwrap's licensing. The added benefit is that fwrap won't weigh Cython down w.r.t. licensing issues or be an impediment to Cython's acceptance into the Python std. lib. Presuming that everyone here agrees with the above (or doesn't care), the remaining question is where to host it. Since fwrap is still closely linked to Cython, I think a natural spot for the mercurial repo would be on Cython's servers. No strong feelings here -- I'm just as happy putting it on bitbucket. Fwrap would *not* clutter up Cython's trac, or Cython's wiki. These would be elsewhere. Kurt From kwmsmith at gmail.com Tue Aug 25 21:20:45 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Tue, 25 Aug 2009 14:20:45 -0500 Subject: [Cython] Giving fwrap a home In-Reply-To: References: Message-ID: BTW, if you would like more info on just what fwrap is (or will be by December), here's a blog post with links to the SciPy 2009 presentation. http://tinyurl.com/ns4by8 Kurt On Tue, Aug 25, 2009 at 2:18 PM, Kurt Smith wrote: > I think its best that fwrap be its own package, distributed separately > from Cython. ?That was the vibe I got from those at the SciPy 2009 > conference and from the recent thread on Fwrap's licensing. ?The added > benefit is that fwrap won't weigh Cython down w.r.t. licensing issues > or be an impediment to Cython's acceptance into the Python std. lib. > > Presuming that everyone here agrees with the above (or doesn't care), > the remaining question is where to host it. ?Since fwrap is still > closely linked to Cython, I think a natural spot for the mercurial > repo would be on Cython's servers. ?No strong feelings here -- I'm > just as happy putting it on bitbucket. ?Fwrap would *not* clutter up > Cython's trac, or Cython's wiki. ?These would be elsewhere. > > Kurt > From kwmsmith at gmail.com Tue Aug 25 22:11:16 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Tue, 25 Aug 2009 15:11:16 -0500 Subject: [Cython] Linking error - 64 bit problem? In-Reply-To: References: Message-ID: On Tue, Aug 25, 2009 at 10:19 AM, Adam Ginsburg wrote: >> From: Robert Bradshaw >> Subject: Re: [Cython] Linking error - 64 bit problem? >> >> On Aug 22, 2009, at 7:40 PM, Adam Ginsburg wrote: >> >>> I reinstalled cython from source (instead of easy_install-ing it) and >>> my code compiled successfully. ?Sorry about the previous e-mail. >> >> Great--I saw your previous email and started to get worried. I >> thought easy_install was a source install (we didn't ship any >> binaries until recently, and then just windows installers). > > I'm not entirely sure what happened, but I have at least 3 python > installs that I use routinely, so it was probably just a confusion > issue on my end. > >>> Unfortunately, my code runs slower with cython than without, but at >>> least now I know I can compile. ?In case anyone wants to offer >>> assistance, the code I'm trying to optimize is: >>> http://code.google.com/p/agpy/source/browse/trunk/plfit.py >>> http://code.google.com/p/agpy/source/browse/trunk/cplfit.pyx >> >> My first piece of advice is to run cython -a over your code. This >> will produce an html file which is most helpful in diagnosing where >> you're wasting time. Just in a quick glance at your code I noticed: >> >> - You're calling Python's log, sqrt, max, abs, sum. This would negate >> any gains you may hope to see. >> - You're calling len a lot, either store it as a int somewhere, or >> use .shape[0]. >> - Unless n is huge, arange(n)/float(n) is probably way to expensive. >> If the above don't help, try unrolling this. >> >> I bet there's a lot more performance that one could ring out of your >> code, but the above should take you far. > > > Much appreciated. ?I guess the various levels of yellow in the html > file indicate the slow lines? ?I tried getting rid of all of my numpy > calls in the loop by rewriting them as loops, but that hasn't improved > speed at all, and in fact appears to have become slower. ?Right now a > fortran (f2py) version goes ~75% faster and pure python goes ~25% > faster, so I must be doing something wrong. > > Example: > I changed this: > ? ? ? dat[xm] = numpy.max(numpy.abs(cf-cx)) > > to this: > ? ? ? for i from 0 <= i < n: > ? ? ? ? ? val = abs(cf[i]-cx[i]) > ? ? ? ? ? if dat[xm] < val: > ? ? ? ? ? ? ? dat[xm] = val > > Is that the right approach for removing max/sum ? > > And is this: > cdef extern from "math.h": > ? double log(double theta) > ? double sqrt(double theta) > ? double abs(double theta) > > the right way to get rid of numpy log/abs/sqrt calls? > > If you'd rather see the whole code, it's been updated on the google code site. Some possible speedups: I'd add a mode='c' option to all the cnp.ndarray's -- this will speed up access. A future optimization would be the @cython.boundscheck(False) directive. In the inner loops get rid of *all* python operations. For example: line 63: z = z[z>=xmin] this is heavy on numpy operations (allocates & discards a temp boolean array every time, etc) and might kill performance. line 64: n = float(...) ==> n = (...) # replace a Python cast with a C-level cast line 68: a += float(n) / (log(z[i]/xmin)) ==> a += n / (log(z[i]/xmin)) line 76: dat = dat[:xm] These slicing operations are done through the Python-level, involving the creation & destruction of python objects, etc. You might be better off here with a for loop. line 79: ibid. line 80: cf = 1-(xmin/z)**a This is again heavy on Python -- you might do better with a loop over z and use pow from math.h. Hope this helps (and let us know if it doesn't). Kurt > > Thanks, > Adam > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From dagss at student.matnat.uio.no Tue Aug 25 22:28:19 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 25 Aug 2009 22:28:19 +0200 Subject: [Cython] Linking error - 64 bit problem? In-Reply-To: References: Message-ID: <4A944963.1020403@student.matnat.uio.no> Adam Ginsburg wrote: > Much appreciated. I guess the various levels of yellow in the html > file indicate the slow lines? I tried getting rid of all of my numpy > calls in the loop by rewriting them as loops, but that hasn't improved > speed at all, and in fact appears to have become slower. Right now a > fortran (f2py) version goes ~75% faster and pure python goes ~25% > faster, so I must be doing something wrong. Not necesarrily. If your Fortran code is only 2x faster than Python there's usually not much Cython can do. Cython is for the times when Fortran is 1000x-2000x faster than Python! Another way of putting it: Cython is able to speed up operations on single array elements only. If you have lots of full-array operations, there's really not that much to gain. Cython shouldn't be *slower*, so something is a little strange. Did you try it with Cython before adding *any* types? Was it the act of compiling it in Cython, or adding the types, which slowed it down? Replacing array operations with loops can definitely slow things down over a straight NumPy function call, especially if boundscheck/wraparound directives are not set. -- Dag Sverre From dagss at student.matnat.uio.no Tue Aug 25 22:36:12 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 25 Aug 2009 22:36:12 +0200 Subject: [Cython] Linking error - 64 bit problem? In-Reply-To: <4A944963.1020403@student.matnat.uio.no> References: <4A944963.1020403@student.matnat.uio.no> Message-ID: > Adam Ginsburg wrote: >> Much appreciated. I guess the various levels of yellow in the html >> file indicate the slow lines? I tried getting rid of all of my numpy >> calls in the loop by rewriting them as loops, but that hasn't improved >> speed at all, and in fact appears to have become slower. Right now a >> fortran (f2py) version goes ~75% faster and pure python goes ~25% >> faster, so I must be doing something wrong. > > Not necesarrily. > > If your Fortran code is only 2x faster than Python there's usually not > much Cython can do. Cython is for the times when Fortran is 1000x-2000x > faster than Python! Sorry about exaggerating. That should have been about 200x to 2000x. Still two orders of magnitude from what you appear to be looking at, I'd be surprised if Cython can give you any real speedups here. (What's the size of your test data though? If f2py overhead comes into play then Fortran could really be faster, meaning more of a potential for Cython. I didn't really look at your code though.) Dag Sverre From fperez.net at gmail.com Tue Aug 25 23:21:35 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 25 Aug 2009 14:21:35 -0700 Subject: [Cython] scipy'09 cython BoF In-Reply-To: References: <200908252025.09211.binet@cern.ch> Message-ID: Hi folks, On Tue, Aug 25, 2009 at 12:08 PM, Kurt Smith wrote: > ?Weave-like support in Cython -- Fernando Perez (not at the BoF) > suggested that Cython could assume the same functionality of > scipy.weave (see: http://www.scipy.org/Weave), since scipy.weave is > poorly documented and could use more maintenance than it is currently > getting. ?I'm personally very interested in this -- it seems that > there is much overlap between scipy.weave and the combination of > pyximport & the pure-python mode of Cython. ?It seems that with a bit > of work from an interested user and some guidance from yours truly, we > could improve pyximport & Cython's pure-python mode to incorporate the > good stuff from scipy.weave, but better ;-) ?I intend to write-up a > CEP with my thoughts sometime in the near future. > Kurt, thanks for resurrecting this comment. Indeed, weave is an extremely useful piece of functionality but one that has been somewhat neglected for years. There's a lot of good in there though, and the need for it is significant. It's *really* cool to do things like: def prod(m, v): """Matrix-vector multiply via weave/blitz++""" nrows, ncolumns = m.shape assert v.ndim==1 and ncolumns==v.shape[0],"Shape mismatch in prod" res = np.zeros(nrows, float) code = r""" for (int i=0; i References: <200908252025.09211.binet@cern.ch> Message-ID: <7f014ea60908251449q27713ad5t756cdcdb264c4ff7@mail.gmail.com> with regards to replacing weave, would it be possible to just use a function decorator? i.e something like: @cythonize def trivial_func(int i): cdef int j j = i + 1 return j I particularly like the thought of this over weave so that I don't have to "switch gears" in brain from reading python to reading C. Cheers, Chris On Tue, Aug 25, 2009 at 5:21 PM, Fernando Perez wrote: > Hi folks, > > On Tue, Aug 25, 2009 at 12:08 PM, Kurt Smith wrote: >> ?Weave-like support in Cython -- Fernando Perez (not at the BoF) >> suggested that Cython could assume the same functionality of >> scipy.weave (see: http://www.scipy.org/Weave), since scipy.weave is >> poorly documented and could use more maintenance than it is currently >> getting. ?I'm personally very interested in this -- it seems that >> there is much overlap between scipy.weave and the combination of >> pyximport & the pure-python mode of Cython. ?It seems that with a bit >> of work from an interested user and some guidance from yours truly, we >> could improve pyximport & Cython's pure-python mode to incorporate the >> good stuff from scipy.weave, but better ;-) ?I intend to write-up a >> CEP with my thoughts sometime in the near future. >> > > Kurt, thanks for resurrecting this comment. ?Indeed, weave is an > extremely useful piece of functionality but one that has been somewhat > neglected for years. ?There's a lot of good in there though, and the > need for it is significant. It's *really* cool to do things like: > > def prod(m, v): > ? ?"""Matrix-vector multiply via weave/blitz++""" > ? ?nrows, ncolumns = m.shape > ? ?assert v.ndim==1 and ncolumns==v.shape[0],"Shape mismatch in prod" > > ? ?res = np.zeros(nrows, float) > ? ?code = r""" > ? ?for (int i=0; i ? ?{ > ? ? ? ?for (int j=0; j ? ? ? ?{ > ? ? ? ? ? ?res(i) += m(i,j)*v(j); > ? ? ? ?} > ? ?} > ? ?""" > ? ?err = inline(code,['nrows', 'ncolumns', 'res', 'm', 'v'], verbose=2, > ? ? ? ? ? ? ? ? type_converters=converters.blitz) > ? ?return res > > This makes interactive experimentation and the writing of > loop/indexing-intensive code really easy. ?But by itself, weave is in > serious danger of bitrot, and right now Cython has the momentum and > concentrated effort for this problem domain. > > Everyone is extremely impressed with the amazing work you've all done, > and rooting for cython to continue to improve, as you are ?solving a > number of key problems to make python a really complete platform for > scientific computing. ?It's been lots of ?fun to see cython steadily > attack a number of problems that have been a real issue for years. > > So if Cython absorbs/extends/integrates/develops/whatever the good > ideas and code from weave, so that this type of ?workflow is also > supported, it would be great. ? I am really happy to see f2py grow > into fwrap, and the weave machinery (or at least parts of it) seems > like a logical addition to the cython-based toolkit. > > One particular use case for which weave.inline was invaluable to me in > the past was the generation of fast looping code for various > dimensions. ? I had to write function evaluation codes where the user > could supply a C expression (as a string, that would #define NDIM to > indicate the dimensionality of the problem). ?I could then easily > generate a few methods for D=1..6 (the range I needed to cover) that > would do certain key operations very fast. ?The bulk of the code used > numpy arrays to be dimension agnostic, but a few methods really needed > explicit loops and with weave, it was made completely transparent to > the user, all ?of it being done at runtime (for us, needing gcc at > runtime was OK). ?I mention this to give you some feedback on the use > cases for weave (there are others), but I hope that in the end, all of > this class of functionality will be available through a unified entry > point, and if that's cython, all the better. > > Thanks again to the team for your great work! > > Cheers, > > f > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From sccolbert at gmail.com Tue Aug 25 23:50:51 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Tue, 25 Aug 2009 17:50:51 -0400 Subject: [Cython] scipy'09 cython BoF In-Reply-To: <7f014ea60908251449q27713ad5t756cdcdb264c4ff7@mail.gmail.com> References: <200908252025.09211.binet@cern.ch> <7f014ea60908251449q27713ad5t756cdcdb264c4ff7@mail.gmail.com> Message-ID: <7f014ea60908251450i4c62b81axaadabd15fb770bed@mail.gmail.com> wait, scratch that idea, that would throw python syntax errors all over the place. I was too quick on the send button... On Tue, Aug 25, 2009 at 5:49 PM, Chris Colbert wrote: > with regards to replacing weave, would it be possible to just use a > function decorator? > > i.e something like: > > @cythonize > def trivial_func(int i): > ? ? ?cdef int j > ? ? ?j = i + 1 > ? ? ?return j > > > I particularly like the thought of this over weave so that I don't > have to "switch gears" in brain from reading python to reading C. > > Cheers, > > Chris > > On Tue, Aug 25, 2009 at 5:21 PM, Fernando Perez wrote: >> Hi folks, >> >> On Tue, Aug 25, 2009 at 12:08 PM, Kurt Smith wrote: >>> ?Weave-like support in Cython -- Fernando Perez (not at the BoF) >>> suggested that Cython could assume the same functionality of >>> scipy.weave (see: http://www.scipy.org/Weave), since scipy.weave is >>> poorly documented and could use more maintenance than it is currently >>> getting. ?I'm personally very interested in this -- it seems that >>> there is much overlap between scipy.weave and the combination of >>> pyximport & the pure-python mode of Cython. ?It seems that with a bit >>> of work from an interested user and some guidance from yours truly, we >>> could improve pyximport & Cython's pure-python mode to incorporate the >>> good stuff from scipy.weave, but better ;-) ?I intend to write-up a >>> CEP with my thoughts sometime in the near future. >>> >> >> Kurt, thanks for resurrecting this comment. ?Indeed, weave is an >> extremely useful piece of functionality but one that has been somewhat >> neglected for years. ?There's a lot of good in there though, and the >> need for it is significant. It's *really* cool to do things like: >> >> def prod(m, v): >> ? ?"""Matrix-vector multiply via weave/blitz++""" >> ? ?nrows, ncolumns = m.shape >> ? ?assert v.ndim==1 and ncolumns==v.shape[0],"Shape mismatch in prod" >> >> ? ?res = np.zeros(nrows, float) >> ? ?code = r""" >> ? ?for (int i=0; i> ? ?{ >> ? ? ? ?for (int j=0; j> ? ? ? ?{ >> ? ? ? ? ? ?res(i) += m(i,j)*v(j); >> ? ? ? ?} >> ? ?} >> ? ?""" >> ? ?err = inline(code,['nrows', 'ncolumns', 'res', 'm', 'v'], verbose=2, >> ? ? ? ? ? ? ? ? type_converters=converters.blitz) >> ? ?return res >> >> This makes interactive experimentation and the writing of >> loop/indexing-intensive code really easy. ?But by itself, weave is in >> serious danger of bitrot, and right now Cython has the momentum and >> concentrated effort for this problem domain. >> >> Everyone is extremely impressed with the amazing work you've all done, >> and rooting for cython to continue to improve, as you are ?solving a >> number of key problems to make python a really complete platform for >> scientific computing. ?It's been lots of ?fun to see cython steadily >> attack a number of problems that have been a real issue for years. >> >> So if Cython absorbs/extends/integrates/develops/whatever the good >> ideas and code from weave, so that this type of ?workflow is also >> supported, it would be great. ? I am really happy to see f2py grow >> into fwrap, and the weave machinery (or at least parts of it) seems >> like a logical addition to the cython-based toolkit. >> >> One particular use case for which weave.inline was invaluable to me in >> the past was the generation of fast looping code for various >> dimensions. ? I had to write function evaluation codes where the user >> could supply a C expression (as a string, that would #define NDIM to >> indicate the dimensionality of the problem). ?I could then easily >> generate a few methods for D=1..6 (the range I needed to cover) that >> would do certain key operations very fast. ?The bulk of the code used >> numpy arrays to be dimension agnostic, but a few methods really needed >> explicit loops and with weave, it was made completely transparent to >> the user, all ?of it being done at runtime (for us, needing gcc at >> runtime was OK). ?I mention this to give you some feedback on the use >> cases for weave (there are others), but I hope that in the end, all of >> this class of functionality will be available through a unified entry >> point, and if that's cython, all the better. >> >> Thanks again to the team for your great work! >> >> Cheers, >> >> f >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > From robert.kern at gmail.com Wed Aug 26 00:17:49 2009 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 25 Aug 2009 17:17:49 -0500 Subject: [Cython] scipy'09 cython BoF In-Reply-To: <7f014ea60908251450i4c62b81axaadabd15fb770bed@mail.gmail.com> References: <200908252025.09211.binet@cern.ch> <7f014ea60908251449q27713ad5t756cdcdb264c4ff7@mail.gmail.com> <7f014ea60908251450i4c62b81axaadabd15fb770bed@mail.gmail.com> Message-ID: On 2009-08-25 16:50 PM, Chris Colbert wrote: > wait, scratch that idea, that would throw python syntax errors all > over the place. You know, if anyone is really up for some fun, they could try to port Cython's grammar to be a modification of python4ply: http://www.dalkescientific.com/Python/python4ply.html One could write .pyx files in the restricted subset supported by the pure Python mode. pyximport could be changed to try to compile the .pyx file as Cython if possible. If a compiler is not available, or if the user has told pyximport to use pure Python mode, then it will just compile the file to Python bytecode as supported by python4ply. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From adam.ginsburg at colorado.edu Wed Aug 26 01:02:48 2009 From: adam.ginsburg at colorado.edu (Adam Ginsburg) Date: Tue, 25 Aug 2009 17:02:48 -0600 Subject: [Cython] Linking error - 64 bit problem? In-Reply-To: References: <4A944963.1020403@student.matnat.uio.no> Message-ID: >> Adam Ginsburg wrote: >>> Much appreciated. I guess the various levels of yellow in the html >>> file indicate the slow lines? I tried getting rid of all of my numpy >>> calls in the loop by rewriting them as loops, but that hasn't improved >>> speed at all, and in fact appears to have become slower. Right now a >>> fortran (f2py) version goes ~75% faster and pure python goes ~25% >>> faster, so I must be doing something wrong. >> >> Not necesarrily. >> >> If your Fortran code is only 2x faster than Python there's usually not >> much Cython can do. Cython is for the times when Fortran is 1000x-2000x >> faster than Python! That makes sense. I was very surprised to find that the fortran code was only a few times faster than the python code; I suppose the problem I'm trying to address must be intrinsically slow. I have lots of full-array operations, but they are being done within long for loops (arrays of size ~n done ~n times...). > (What's the size of your test data though? If f2py overhead comes into > play then Fortran could really be faster, meaning more of a potential for > Cython. I didn't really look at your code though.) My test data set is 10^4 elements, which is typical for what I expect to deal with but it could go up an order of magnitude. Of course, I need to do 10^4 element sets ~10^4 times each... > I'd add a mode='c' option to all the cnp.ndarray's -- this will speed up access. > > A future optimization would be the @cython.boundscheck(False) directive. Do you mean future as in "don't try to use it now" or "should use it if it's safe to proceed without boundary checking"? Also, this looks like a decorator to me, but I couldn't compile when I put it in front of my function definition. > In the inner loops get rid of *all* python operations. For example: > > line 63: z = z[z>=xmin] > this is heavy on numpy operations (allocates & discards a temp > boolean array every time, etc) and might kill performance. > > line 64: n = float(...) ==> n = (...) # replace a Python > cast with a C-level cast I think I came up with a way around these both. didn't work, though - I received errors when I tried it. > line 80: cf = 1-(xmin/z)**a > This is again heavy on Python -- you might do better with a loop > over z and use pow from math.h. OK, I put cx and cf into loops and switched from ** to pow. > Hope this helps (and let us know if it doesn't). I found a factor of 2-3 improvement. In particular, I left one python float() in, and that made python and cython go ~the same speed. When I changed it to , it dropped the cython time by 2. Now execution times are: n=2e4 python~3x fortran cython~1.5x fortran So this may be as fast as I can get. I'm a little confused about fortran getting slower relative to python as n gets larger, but that is probably some sort of failure on my part. Thanks for the help! Adam From wstein at gmail.com Wed Aug 26 01:31:10 2009 From: wstein at gmail.com (William Stein) Date: Tue, 25 Aug 2009 16:31:10 -0700 Subject: [Cython] Giving fwrap a home In-Reply-To: References: Message-ID: <85e81ba30908251631n78de4e2dk849826cf5a86593d@mail.gmail.com> On Tue, Aug 25, 2009 at 12:18 PM, Kurt Smith wrote: > I think its best that fwrap be its own package, distributed separately > from Cython. ?That was the vibe I got from those at the SciPy 2009 > conference and from the recent thread on Fwrap's licensing. ?The added > benefit is that fwrap won't weigh Cython down w.r.t. licensing issues > or be an impediment to Cython's acceptance into the Python std. lib. > > Presuming that everyone here agrees with the above (or doesn't care), > the remaining question is where to host it. ?Since fwrap is still > closely linked to Cython, I think a natural spot for the mercurial > repo would be on Cython's servers. ?No strong feelings here -- I'm > just as happy putting it on bitbucket. ?Fwrap would *not* clutter up > Cython's trac, or Cython's wiki. ?These would be elsewhere. > As the own of Cython's servers, I hereby certainly offer you hosting space. -- William From robertwb at math.washington.edu Wed Aug 26 01:35:22 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 25 Aug 2009 16:35:22 -0700 (PDT) Subject: [Cython] Linking error - 64 bit problem? In-Reply-To: References: <4A944963.1020403@student.matnat.uio.no> Message-ID: On Tue, 25 Aug 2009, Adam Ginsburg wrote: >>> Adam Ginsburg wrote: > > My test data set is 10^4 elements, which is typical for what I expect > to deal with but it could go up an order of magnitude. Of course, I > need to do 10^4 element sets ~10^4 times each... > > >> I'd add a mode='c' option to all the cnp.ndarray's -- this will speed up access. >> >> A future optimization would be the @cython.boundscheck(False) directive. > > Do you mean future as in "don't try to use it now" or "should use it > if it's safe to proceed without boundary checking"? Also, this looks > like a decorator to me, but I couldn't compile when I put it in front > of my function definition. Did you import cython? >> In the inner loops get rid of *all* python operations. For example: >> >> line 63: z = z[z>=xmin] >> this is heavy on numpy operations (allocates & discards a temp >> boolean array every time, etc) and might kill performance. >> >> line 64: n = float(...) ==> n = (...) # replace a Python >> cast with a C-level cast > > I think I came up with a way around these both. didn't work, > though - I received errors when I tried it. > >> line 80: cf = 1-(xmin/z)**a >> This is again heavy on Python -- you might do better with a loop >> over z and use pow from math.h. > > OK, I put cx and cf into loops and switched from ** to pow. > >> Hope this helps (and let us know if it doesn't). > > I found a factor of 2-3 improvement. In particular, I left one python > float() in, and that made python and cython go ~the same speed. When > I changed it to , it dropped the cython time by 2. This is actually a common occurance--sometimes that last Python operation takes 10x as long as everything else you're doing, so until you eliminate them all you won't see the huge jump in speed. > Now execution times are: > n=2e4 > python~3x fortran > cython~1.5x fortran 10^4 still isn't a huge array, but I would be curious how much a speed increase you could get without unrolling all your loops (i.e. a minimal modification to your code). Might be 1.6, might not be. > So this may be as fast as I can get. I'm a little confused about > fortran getting slower relative to python as n gets larger, but that > is probably some sort of failure on my part. This is due to the fact that the array processing happens at near the same speed, so as the array gets larger, the (relative) significance of the Python function call overhead goes away. - Robert From kwmsmith at gmail.com Wed Aug 26 03:16:58 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Tue, 25 Aug 2009 20:16:58 -0500 Subject: [Cython] scipy'09 cython BoF In-Reply-To: References: <200908252025.09211.binet@cern.ch> Message-ID: On Tue, Aug 25, 2009 at 4:21 PM, Fernando Perez wrote: > Hi folks, > > On Tue, Aug 25, 2009 at 12:08 PM, Kurt Smith wrote: >> ?Weave-like support in Cython -- Fernando Perez (not at the BoF) >> suggested that Cython could assume the same functionality of >> scipy.weave (see: http://www.scipy.org/Weave), since scipy.weave is >> poorly documented and could use more maintenance than it is currently >> getting. ?I'm personally very interested in this -- it seems that >> there is much overlap between scipy.weave and the combination of >> pyximport & the pure-python mode of Cython. ?It seems that with a bit >> of work from an interested user and some guidance from yours truly, we >> could improve pyximport & Cython's pure-python mode to incorporate the >> good stuff from scipy.weave, but better ;-) ?I intend to write-up a >> CEP with my thoughts sometime in the near future. >> > > Kurt, thanks for resurrecting this comment. ?Indeed, weave is an > extremely useful piece of functionality but one that has been somewhat > neglected for years. ?There's a lot of good in there though, and the > need for it is significant. It's *really* cool to do things like: > > def prod(m, v): > ? ?"""Matrix-vector multiply via weave/blitz++""" > ? ?nrows, ncolumns = m.shape > ? ?assert v.ndim==1 and ncolumns==v.shape[0],"Shape mismatch in prod" > > ? ?res = np.zeros(nrows, float) > ? ?code = r""" > ? ?for (int i=0; i ? ?{ > ? ? ? ?for (int j=0; j ? ? ? ?{ > ? ? ? ? ? ?res(i) += m(i,j)*v(j); > ? ? ? ?} > ? ?} > ? ?""" > ? ?err = inline(code,['nrows', 'ncolumns', 'res', 'm', 'v'], verbose=2, > ? ? ? ? ? ? ? ? type_converters=converters.blitz) > ? ?return res Here's what I'm envisioning for the above usecase -- I admit to not using the pure-python mode that much, so there may be errors (and some of what I'm using isn't implemented yet). But it shows how much is already there in Cython's pure-python mode. @cython.compile # this is new & where most of the work is required! @cython.locals(m=cython.float[:,::1], v=cython.float[::1]) def prod(m, v): cython.declare( i=cython.int, j=cython.int, nrows=cython.int, ncolumns=cython.int) nrows = m.shape[0]; ncolumns = m.shape[1] cython.declare(res=cython.float[::1]) res = np.zeros(nrows, np.float) for i in range(nrows): for j in range(ncolumns): res[i] += m[i,j] * v[j] return res Once the new array type / memoryview slice is merged into cython-devel and we get it integrated into pure-python mode, this will be close to working. If you just want things fast in the function body (what weave gives you currently) then the above is all you need. The hard part is getting things to compile & run automatically for just this function (the cython.compile decorator). This overlaps with pyximport a good amount, and it will take some thought as to how it will all work together. Perhaps the core devs won't like so much going on behind the scenes -- cython is principally a utility for generating a C extension module, not compiling & loading the extension module. Weave compiles individual snippets into their own extension module, whereas Cython has always compiled entire modules into extension modules, and requires a separate compilation step (whether through the commandline or through pyximport). The cython.compile decorator would be different, if it were to be just like weave -- it would compile a single function into its own extension module, without an explicit compilation step. Much more is automated. This seems to be the main thing that weave offers -- fast code only where you want it, without the explicit compilation step. Pyximport overlaps here, but pyximport still compiles an entire module, not just a part. So those are some of my thoughts. They'll be in a CEP sometime soon :-) > > This makes interactive experimentation and the writing of > loop/indexing-intensive code really easy. ?But by itself, weave is in > serious danger of bitrot, and right now Cython has the momentum and > concentrated effort for this problem domain. > > Everyone is extremely impressed with the amazing work you've all done, > and rooting for cython to continue to improve, as you are ?solving a > number of key problems to make python a really complete platform for > scientific computing. ?It's been lots of ?fun to see cython steadily > attack a number of problems that have been a real issue for years. > > So if Cython absorbs/extends/integrates/develops/whatever the good > ideas and code from weave, so that this type of ?workflow is also > supported, it would be great. ? I am really happy to see f2py grow > into fwrap, and the weave machinery (or at least parts of it) seems > like a logical addition to the cython-based toolkit. > > One particular use case for which weave.inline was invaluable to me in > the past was the generation of fast looping code for various > dimensions. ? I had to write function evaluation codes where the user > could supply a C expression (as a string, that would #define NDIM to > indicate the dimensionality of the problem). ?I could then easily > generate a few methods for D=1..6 (the range I needed to cover) that > would do certain key operations very fast. ?The bulk of the code used > numpy arrays to be dimension agnostic, but a few methods really needed > explicit loops and with weave, it was made completely transparent to > the user, all ?of it being done at runtime (for us, needing gcc at > runtime was OK). ?I mention this to give you some feedback on the use > cases for weave (there are others), but I hope that in the end, all of > this class of functionality will be available through a unified entry > point, and if that's cython, all the better. > Thanks for the encouragement, and for the examples. I don't have time to absorb all this just yet, but I'll return to it. And get other usecases for weave from you and others. Kurt > Thanks again to the team for your great work! > > Cheers, > > f > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From kwmsmith at gmail.com Wed Aug 26 03:37:23 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Tue, 25 Aug 2009 20:37:23 -0500 Subject: [Cython] Giving fwrap a home In-Reply-To: <85e81ba30908251631n78de4e2dk849826cf5a86593d@mail.gmail.com> References: <85e81ba30908251631n78de4e2dk849826cf5a86593d@mail.gmail.com> Message-ID: On Tue, Aug 25, 2009 at 6:31 PM, William Stein wrote: > On Tue, Aug 25, 2009 at 12:18 PM, Kurt Smith wrote: >> I think its best that fwrap be its own package, distributed separately >> from Cython. ?That was the vibe I got from those at the SciPy 2009 >> conference and from the recent thread on Fwrap's licensing. ?The added >> benefit is that fwrap won't weigh Cython down w.r.t. licensing issues >> or be an impediment to Cython's acceptance into the Python std. lib. >> >> Presuming that everyone here agrees with the above (or doesn't care), >> the remaining question is where to host it. ?Since fwrap is still >> closely linked to Cython, I think a natural spot for the mercurial >> repo would be on Cython's servers. ?No strong feelings here -- I'm >> just as happy putting it on bitbucket. ?Fwrap would *not* clutter up >> Cython's trac, or Cython's wiki. ?These would be elsewhere. >> > > As the own of Cython's servers, I hereby certainly offer you hosting space. Thanks! I'm evaluating bitbucket & googlecode to see what they offer -- apparently googlecode has mercurial support, so that is a big plus. I'll decide soon & let you know. Kurt > > ?-- William > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From fperez.net at gmail.com Wed Aug 26 04:22:45 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 25 Aug 2009 19:22:45 -0700 Subject: [Cython] scipy'09 cython BoF In-Reply-To: References: <200908252025.09211.binet@cern.ch> Message-ID: On Tue, Aug 25, 2009 at 6:16 PM, Kurt Smith wrote: > Thanks for the encouragement, and for the examples. ?I don't have time > to absorb all this just yet, but I'll return to it. ?And get other > usecases for weave from you and others. Kurt, I have to run now, but this page is a great summary of approaches to python performance that Prabhu wrote years ago: http://www.scipy.org/PerformancePython It has weave examples, and it would be also great to have (possibly more than one) Cython approaches explained as well. I hope it's a useful reference as you move forward. Cheers, f From josef.pktd at gmail.com Wed Aug 26 04:28:16 2009 From: josef.pktd at gmail.com (josef.pktd at gmail.com) Date: Tue, 25 Aug 2009 22:28:16 -0400 Subject: [Cython] scipy'09 cython BoF In-Reply-To: References: <200908252025.09211.binet@cern.ch> Message-ID: <1cd32cbb0908251928tb49f03ep4c2c0c7e44958e7c@mail.gmail.com> On Tue, Aug 25, 2009 at 9:16 PM, Kurt Smith wrote: > On Tue, Aug 25, 2009 at 4:21 PM, Fernando Perez wrote: >> Hi folks, >> >> On Tue, Aug 25, 2009 at 12:08 PM, Kurt Smith wrote: >>> ?Weave-like support in Cython -- Fernando Perez (not at the BoF) >>> suggested that Cython could assume the same functionality of >>> scipy.weave (see: http://www.scipy.org/Weave), since scipy.weave is >>> poorly documented and could use more maintenance than it is currently >>> getting. ?I'm personally very interested in this -- it seems that >>> there is much overlap between scipy.weave and the combination of >>> pyximport & the pure-python mode of Cython. ?It seems that with a bit >>> of work from an interested user and some guidance from yours truly, we >>> could improve pyximport & Cython's pure-python mode to incorporate the >>> good stuff from scipy.weave, but better ;-) ?I intend to write-up a >>> CEP with my thoughts sometime in the near future. >>> >> >> Kurt, thanks for resurrecting this comment. ?Indeed, weave is an >> extremely useful piece of functionality but one that has been somewhat >> neglected for years. ?There's a lot of good in there though, and the >> need for it is significant. It's *really* cool to do things like: >> >> def prod(m, v): >> ? ?"""Matrix-vector multiply via weave/blitz++""" >> ? ?nrows, ncolumns = m.shape >> ? ?assert v.ndim==1 and ncolumns==v.shape[0],"Shape mismatch in prod" >> >> ? ?res = np.zeros(nrows, float) >> ? ?code = r""" >> ? ?for (int i=0; i> ? ?{ >> ? ? ? ?for (int j=0; j> ? ? ? ?{ >> ? ? ? ? ? ?res(i) += m(i,j)*v(j); >> ? ? ? ?} >> ? ?} >> ? ?""" >> ? ?err = inline(code,['nrows', 'ncolumns', 'res', 'm', 'v'], verbose=2, >> ? ? ? ? ? ? ? ? type_converters=converters.blitz) >> ? ?return res > > Here's what I'm envisioning for the above usecase -- I admit to not > using the pure-python mode that much, so there may be errors (and some > of what I'm using isn't implemented yet). ?But it shows how much is > already there in Cython's pure-python mode. > > @cython.compile # this is new & where most of the work is required! > @cython.locals(m=cython.float[:,::1], v=cython.float[::1]) > def prod(m, v): > ? ?cython.declare( > ? ? ? ? ? ?i=cython.int, > ? ? ? ? ? ?j=cython.int, > ? ? ? ? ? ?nrows=cython.int, > ? ? ? ? ? ?ncolumns=cython.int) > ? ?nrows = m.shape[0]; ncolumns = m.shape[1] > ? ?cython.declare(res=cython.float[::1]) > ? ?res = np.zeros(nrows, np.float) > ? ?for i in range(nrows): > ? ? ? ?for j in range(ncolumns): > ? ? ? ? ? ?res[i] += m[i,j] * v[j] > ? ?return res > > Once the new array type / memoryview slice is merged into cython-devel > and we get it integrated into pure-python mode, this will be close to > working. ?If you just want things fast in the function body (what > weave gives you currently) then the above is all you need. > > The hard part is getting things to compile & run automatically for > just this function (the cython.compile decorator). ?This overlaps with > pyximport a good amount, and it will take some thought as to how it > will all work together. ?Perhaps the core devs won't like so much > going on behind the scenes -- cython is principally a utility for > generating a C extension module, not compiling & loading the extension > module. > > Weave compiles individual snippets into their own extension module, > whereas Cython has always compiled entire modules into extension > modules, and requires a separate compilation step (whether through the > commandline or through pyximport). ? The cython.compile decorator > would be different, if it were to be just like weave -- it would > compile a single function into its own extension module, without an > explicit compilation step. ?Much more is automated. > > This seems to be the main thing that weave offers -- fast code only > where you want it, without the explicit compilation step. ?Pyximport > overlaps here, but pyximport still compiles an entire module, not just > a part. > > So those are some of my thoughts. ?They'll be in a CEP sometime soon :-) > >> >> This makes interactive experimentation and the writing of >> loop/indexing-intensive code really easy. ?But by itself, weave is in >> serious danger of bitrot, and right now Cython has the momentum and >> concentrated effort for this problem domain. >> >> Everyone is extremely impressed with the amazing work you've all done, >> and rooting for cython to continue to improve, as you are ?solving a >> number of key problems to make python a really complete platform for >> scientific computing. ?It's been lots of ?fun to see cython steadily >> attack a number of problems that have been a real issue for years. >> >> So if Cython absorbs/extends/integrates/develops/whatever the good >> ideas and code from weave, so that this type of ?workflow is also >> supported, it would be great. ? I am really happy to see f2py grow >> into fwrap, and the weave machinery (or at least parts of it) seems >> like a logical addition to the cython-based toolkit. >> >> One particular use case for which weave.inline was invaluable to me in >> the past was the generation of fast looping code for various >> dimensions. ? I had to write function evaluation codes where the user >> could supply a C expression (as a string, that would #define NDIM to >> indicate the dimensionality of the problem). ?I could then easily >> generate a few methods for D=1..6 (the range I needed to cover) that >> would do certain key operations very fast. ?The bulk of the code used >> numpy arrays to be dimension agnostic, but a few methods really needed >> explicit loops and with weave, it was made completely transparent to >> the user, all ?of it being done at runtime (for us, needing gcc at >> runtime was OK). ?I mention this to give you some feedback on the use >> cases for weave (there are others), but I hope that in the end, all of >> this class of functionality will be available through a unified entry >> point, and if that's cython, all the better. >> > > Thanks for the encouragement, and for the examples. ?I don't have time > to absorb all this just yet, but I'll return to it. ?And get other > usecases for weave from you and others. > > Kurt > >> Thanks again to the team for your great work! >> >> Cheers, >> >> f (A comment from a non-expert) A more literal approach to imitating weave inline compilation, seems already possible with cython and pyximport. To try out cython and to reduce the time in debugging my (many) mistakes, a added the compilation step directly to the script. This seems closer to the usage of weave that Fernando was deswcribing and what I saw of weave examples. Cheers, Josef -------------- next part -------------- code = ''' import numpy as np cimport numpy as np __all__ = ["hist3d"] DTYPE = ${imgtype} DTYPE32 = ${outtype} ctypedef ${imgtype}_t DTYPE_t ctypedef ${outtype}_t DTYPE_t32 def hist3d(np.ndarray[DTYPE_t, ndim=3] img, int nbins): # a variation on code by Chris Colbert #if nbins are not a divisor of 256, all extra high values are added to the last bin cdef int x = img.shape[0] cdef int y = img.shape[1] #cdef int z = img.shape[2] #I don't know how to get the correct width cdef int bdist = np.max(img) / nbins #np.ceil(256 / (1.0* nbins)) cdef int addx cdef int addy cdef int addz cdef np.ndarray[DTYPE_t32, ndim=3] out = np.zeros([nbins, nbins, nbins], dtype=DTYPE32) cdef int i, j, v0, v1, v2 for i in range(x): for j in range(y): v0 = img[i, j, 0] v1 = img[i, j, 1] v2 = img[i, j, 2] addx = min((v0 - (v0 % bdist)) / bdist, nbins-1) addy = min((v1 - (v1 % bdist)) / bdist, nbins-1) addz = min((v2 - (v2 % bdist)) / bdist, nbins-1) #print addx, addy, addz out[addx, addy, addz] += 1 return out ''' from string import Template s = Template(code) src = s.substitute({'imgtype': 'np.int', 'outtype': 'np.float64'}) open('histrgbintfl06.pyx','w').write(src) import pyximport; pyximport.install() import histrgbintfl06 import numpy as np factors = np.random.randint(256,size=(480, 630, 3)) h = histrgbintfl06.hist3d(factors, 4)#.astype(np.uint8)) print h[:,:,0] h = histrgbintfl06.hist3d(factors, 5) From dagss at student.matnat.uio.no Wed Aug 26 06:50:00 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 26 Aug 2009 06:50:00 +0200 Subject: [Cython] scipy'09 cython BoF Message-ID: <3334114249.448558@smtp.netcom.no> Yep -- building something like that directly into pyximport as a general utility would be the first step; then we could see after that. There's already unmerged patches sitting for pyximport though. The real problem here is that pyximport doesn't have a testsuite, which makes making changes to it seem overwhelmingly daunting... Adding that would definitely be my first step in all of this. Since pyximport is expected to run in pure Python, I think it is really just about writing the tests; no test-compilation framework needed. Any volunteers? :-) Dag Sverre Seljebotn -----Original Message----- From: josef.pktd at gmail.com Date: Wednesday, Aug 26, 2009 4:28 am Subject: Re: [Cython] scipy'09 cython BoF To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net On Tue, Aug 25, 2009 at 9:16 PM, Kurt Smith wrote: >> On Tue, Aug 25, 2009 at 4:21 PM, Fernando Perez wrote: >>> Hi folks, >>> >>> On Tue, Aug 25, 2009 at 12:08 PM, Kurt Smith wrote: >>>> ?Weave-like support in Cython -- Fernando Perez (not at the BoF) >>>> suggested that Cython could assume the same functionality of >>>> scipy.weave (see: http://www.scipy.org/Weave), since scipy.weave is >>>> poorly documented and could use more maintenance than it is currently >>>> getting. ?I'm personally very interested in this -- it seems that >>>> there is much overlap between scipy.weave and the combination of >>>> pyximport & the pure-python mode of Cython. ?It seems that with a bit >>>> of work from an interested user and some guidance from yours truly, we >>>> could improve pyximport & Cython's pure-python mode to incorporate the >>>> good stuff from scipy.weave, but better ;-) ?I intend to write-up a >>>> CEP with my thoughts sometime in the near future. >>>> >>> >>> Kurt, thanks for resurrecting this comment. ?Indeed, weave is an >>> extremely useful piece of functionality but one that has been somewhat >>> neglected for years. ?There's a lot of good in there though, and the >>> need for it is significant. It's *really* cool to do things like: >>> >>> def prod(m, v): >>> ? ?"""Matrix-vector multiply via weave/blitz++""" >>> ? ?nrows, ncolumns = m.shape >>> ? ?assert v.ndim==1 and ncolumns==v.shape[0],"Shape mismatch in prod" >>> >>> ? ?res = np.zeros(nrows, float) >>> ? ?code = r""" >>> ? ?for (int i=0; i>> ? ?{ >>> ? ? ? ?for (int j=0; j>> ? ? ? ?{ >>> ? ? ? ? ? ?res(i) += m(i,j)*v(j); >>> ? ? ? ?} >>> ? ?} >>> ? ?""" >>> ? ?err = inline(code,['nrows', 'ncolumns', 'res', 'm', 'v'], verbose=2, >>> ? ? ? ? ? ? ? ? type_converters=converters.blitz) >>> ? ?return res >> >> Here's what I'm envisioning for the above usecase -- I admit to not >> using the pure-python mode that much, so there may be errors (and some >> of what I'm using isn't implemented yet). ?But it shows how much is >> already there in Cython's pure-python mode. >> >> @cython.compile # this is new & where most of the work is required! >> @cython.locals(m=cython.float[:,::1], v=cython.float[::1]) >> def prod(m, v): >> ? ?cython.declare( >> ? ? ? ? ? ?i=cython.int, >> ? ? ? ? ? ?j=cython.int, >> ? ? ? ? ? ?nrows=cython.int, >> ? ? ? ? ? ?ncolumns=cython.int) >> ? ?nrows = m.shape[0]; ncolumns = m.shape[1] >> ? ?cython.declare(res=cython.float[::1]) >> ? ?res = np.zeros(nrows, np.float) >> ? ?for i in range(nrows): >> ? ? ? ?for j in range(ncolumns): >> ? ? ? ? ? ?res[i] += m[i,j] * v[j] >> ? ?return res >> >> Once the new array type / memoryview slice is merged into cython-devel >> and we get it integrated into pure-python mode, this will be close to >> working. ?If you just want things fast in the function body (what >> weave gives you currently) then the above is all you need. >> >> The hard part is getting things to compile & run automatically for >> just this function (the cython.compile decorator). ?This overlaps with >> pyximport a good amount, and it will take some thought as to how it >> will all work together. ?Perhaps the core devs won't like so much >> going on behind the scenes -- cython is principally a utility for >> generating a C extension module, not compiling & loading the extension >> module. >> >> Weave compiles individual snippets into their own extension module, >> whereas Cython has always compiled entire modules into extension >> modules, and requires a separate compilation step (whether through the >> commandline or through pyximport). ? The cython.compile decorator >> would be different, if it were to be just like weave -- it would >> compile a single function into its own extension module, without an >> explicit compilation step. ?Much more is automated. >> >> This seems to be the main thing that weave offers -- fast code only >> where you want it, without the explicit compilation step. ?Pyximport >> overlaps here, but pyximport still compiles an entire module, not just >> a part. >> >> So those are some of my thoughts. ?They'll be in a CEP sometime soon :-) >> >>> >>> This makes interactive experimentation and the writing of >>> loop/indexing-intensive code really easy. ?But by itself, weave is in >>> serious danger of bitrot, and right now Cython has the momentum and >>> concentrated effort for this problem domain. >>> >>> Everyone is extremely impressed with the amazing work you've all done, >>> and rooting for cython to continue to improve, as you are ?solving a >>> number of key problems to make python a really complete platform for >>> scientific computing. ?It's been lots of ?fun to see cython steadily >>> attack a number of problems that have been a real issue for years. >>> >>> So if Cython absorbs/extends/integrates/develops/whatever the good >>> ideas and code from weave, so that this type of ?workflow is also >>> supported, it would be great. ? I am really happy to see f2py grow >>> into fwrap, and the weave machinery (or at least parts of it) seems >>> like a logical addition to the cython-based toolkit. >>> >>> One particular use case for which weave.inline was invaluable to me in >>> the past was the generation of fast looping code for various >>> dimensions. ? I had to write function evaluation codes where the user >>> could supply a C expression (as a string, that would #define NDIM to >>> indicate the dimensionality of the problem). ?I could then easily >>> generate a few methods for D=1..6 (the range I needed to cover) that >>> would do certain key operations very fast. ?The bulk of the code used >>> numpy arrays to be dimension agnostic, but a few methods really needed >>> explicit loops and with weave, it was made completely transparent to >>> the user, all ?of it being done at runtime (for us, needing gcc at >>> runtime was OK). ?I mention this to give you some feedback on the use >>> cases for weave (there are others), but I hope that in the end, all of >>> this class of functionality will be available through a unified entry >>> point, and if that's cython, all the better. >>> >> >> Thanks for the encouragement, and for the examples. ?I don't have time >> to absorb all this just yet, but I'll return to it. ?And get other >> usecases for weave from you and others. >> >> Kurt >> >>> Thanks again to the team for your great work! >>> >>> Cheers, >>> >>> f > >(A comment from a non-expert) > >A more literal approach to imitating weave inline compilation, seems >already possible with cython and pyximport. To try out cython and to >reduce the time in debugging my (many) mistakes, a added the >compilation step directly to the script. This seems closer to the >usage of weave that Fernando was deswcribing and what I saw of weave >examples. > >Cheers, > >Josef > From robertwb at math.washington.edu Wed Aug 26 07:18:54 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 25 Aug 2009 22:18:54 -0700 Subject: [Cython] Giving fwrap a home In-Reply-To: References: <85e81ba30908251631n78de4e2dk849826cf5a86593d@mail.gmail.com> Message-ID: <9E2BDBD3-4D7C-45C9-8F43-A3F647CF14D1@math.washington.edu> On Aug 25, 2009, at 6:37 PM, Kurt Smith wrote: > On Tue, Aug 25, 2009 at 6:31 PM, William Stein > wrote: >> On Tue, Aug 25, 2009 at 12:18 PM, Kurt Smith >> wrote: >>> I think its best that fwrap be its own package, distributed >>> separately >>> from Cython. That was the vibe I got from those at the SciPy 2009 >>> conference and from the recent thread on Fwrap's licensing. The >>> added >>> benefit is that fwrap won't weigh Cython down w.r.t. licensing >>> issues >>> or be an impediment to Cython's acceptance into the Python std. lib. >>> >>> Presuming that everyone here agrees with the above (or doesn't >>> care), >>> the remaining question is where to host it. Since fwrap is still >>> closely linked to Cython, I think a natural spot for the mercurial >>> repo would be on Cython's servers. No strong feelings here -- I'm >>> just as happy putting it on bitbucket. Fwrap would *not* clutter up >>> Cython's trac, or Cython's wiki. These would be elsewhere. >>> >> >> As the own of Cython's servers, I hereby certainly offer you >> hosting space. > > Thanks! I'm evaluating bitbucket & googlecode to see what they offer > -- apparently googlecode has mercurial support, so that is a big plus. > I'll decide soon & let you know. I wouldn't consider it "clutter" to have an fwrap section of the wiki, nor host another porject on the cython trac server. - Robert From robertwb at math.washington.edu Wed Aug 26 07:29:51 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 25 Aug 2009 22:29:51 -0700 Subject: [Cython] scipy'09 cython BoF In-Reply-To: References: <200908252025.09211.binet@cern.ch> Message-ID: <2B3398DD-9714-4E61-B390-49E0771C66E9@math.washington.edu> On Aug 25, 2009, at 12:08 PM, Kurt Smith wrote: > On Tue, Aug 25, 2009 at 1:25 PM, Sebastien > Binet wrote: >> hi there, >> >> IIRC there was a cython oriented BoF at last week's scipy conference. >> >> anything interesting ? >> (lame attempt at having a good soul summarizing or even just >> outlining what >> happened for the poor guys from the other side of the big pond) > > I thought so, although I was privy to just a part of the conversation. > Dag or others, feel free to fill in parts that I'm missing. Thanks for the summary. I'll be looking forward to Dag's take on all this too. > Rough outline of BoF topics (more of a brain-dump, actually): > > Memoryviews & Memoryviewslices (see: http://wiki.cython.org/ > enhancements/array) > Dag had an extended conversation with the attendees about the > semantics of these arrays -- when copying should occur, when views > should be taken, etc. Good to hear that's getting nailed down, and by those who will actually be using them. > SIMD operations on Cython arrays / Memoryviews > My impression was that there was general support for this feature, > although this isn't surprising since we were at a numerical computing > conference ;-) Dag discussed a few possibilities for how it would > work, noting that this particular aspect of Cython arrays would need a > vote & acceptance in the Cython community and core-devs. A numexpr > backend is one possibility. Others brought up the possibility of > incoporating OpenMP directives in Cython somehow. Another thread in > this topic was support for generalized ufuncs in Cython (see: > http://projects.scipy.org/numpy/browser/trunk/doc/neps/generalized- > ufuncs.rst), > which very easily extend to SIMD operations. Dag, you'll have to fill > in here. We'll see how it plays out, but I think well-thought out, natively supported arrays like this are a killer feature (and I don't usually depart from focusing on Cython as mostly an optimizing Python compiler). > fwrap: > A few of us off in the corner discussed fwrap and what remains to > be done to get the first release out. The fellas I spoke to work on > Clawpack at the U-Washington, and they're very interested in getting > callbacks working well. They even offered to pitch in. So that's > encouraging. Actually, I've been meaning to ask you if you thought fwrap would be a good fit for the Clawpack project--glad to see you've already met up. > Topics listed but not discussed at the BoF (but certainly worth > mentioning ;-): > C++ & Cython integration -- Damian Eads has a nice C++ Container > <-> Python container tool called Convert-xy (see: > http://code.google.com/p/convert-xy/). If integrated in Cython, it > could ease the use of STL containers in Cython. Hmm.. That could be useful. > Weave-like support in Cython -- Fernando Perez (not at the BoF) > suggested that Cython could assume the same functionality of > scipy.weave (see: http://www.scipy.org/Weave), since scipy.weave is > poorly documented and could use more maintenance than it is currently > getting. I'm personally very interested in this -- it seems that > there is much overlap between scipy.weave and the combination of > pyximport & the pure-python mode of Cython. It seems that with a bit > of work from an interested user and some guidance from yours truly, we > could improve pyximport & Cython's pure-python mode to incorporate the > good stuff from scipy.weave, but better ;-) I intend to write-up a > CEP with my thoughts sometime in the near future. More comments on this later. - Robert From robertwb at math.washington.edu Wed Aug 26 07:38:22 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 25 Aug 2009 22:38:22 -0700 Subject: [Cython] scipy'09 cython BoF In-Reply-To: References: <200908252025.09211.binet@cern.ch> Message-ID: On Aug 25, 2009, at 7:22 PM, Fernando Perez wrote: > On Tue, Aug 25, 2009 at 6:16 PM, Kurt Smith wrote: >> Thanks for the encouragement, and for the examples. I don't have >> time >> to absorb all this just yet, but I'll return to it. And get other >> usecases for weave from you and others. > > Kurt, I have to run now, but this page is a great summary of > approaches to python performance that Prabhu wrote years ago: > > http://www.scipy.org/PerformancePython > > It has weave examples, and it would be also great to have (possibly > more than one) Cython approaches explained as well. I hope it's a > useful reference as you move forward. Funny you should mention this--I actually wrote up a Cython version of that. It's as fast as the Pyrex version, but only requires the addition of a couple of lines lines to the raw Python and no other changes to the source. - Robert -------------- next part -------------- A non-text attachment was scrubbed... Name: laplace.pyx Type: application/octet-stream Size: 4448 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090825/cafac165/attachment.obj From robertwb at math.washington.edu Wed Aug 26 07:52:45 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 25 Aug 2009 22:52:45 -0700 Subject: [Cython] scipy'09 cython BoF In-Reply-To: References: <200908252025.09211.binet@cern.ch> Message-ID: On Aug 25, 2009, at 6:16 PM, Kurt Smith wrote: > On Tue, Aug 25, 2009 at 4:21 PM, Fernando > Perez wrote: >> Hi folks, >> >> On Tue, Aug 25, 2009 at 12:08 PM, Kurt Smith >> wrote: >>> Weave-like support in Cython -- Fernando Perez (not at the BoF) >>> suggested that Cython could assume the same functionality of >>> scipy.weave (see: http://www.scipy.org/Weave), since scipy.weave is >>> poorly documented and could use more maintenance than it is >>> currently >>> getting. I'm personally very interested in this -- it seems that >>> there is much overlap between scipy.weave and the combination of >>> pyximport & the pure-python mode of Cython. It seems that with a >>> bit >>> of work from an interested user and some guidance from yours >>> truly, we >>> could improve pyximport & Cython's pure-python mode to >>> incorporate the >>> good stuff from scipy.weave, but better ;-) I intend to write-up a >>> CEP with my thoughts sometime in the near future. >>> >> >> Kurt, thanks for resurrecting this comment. Indeed, weave is an >> extremely useful piece of functionality but one that has been >> somewhat >> neglected for years. There's a lot of good in there though, and the >> need for it is significant. It's *really* cool to do things like: >> >> def prod(m, v): >> """Matrix-vector multiply via weave/blitz++""" >> nrows, ncolumns = m.shape >> assert v.ndim==1 and ncolumns==v.shape[0],"Shape mismatch in prod" >> >> res = np.zeros(nrows, float) >> code = r""" >> for (int i=0; i> { >> for (int j=0; j> { >> res(i) += m(i,j)*v(j); >> } >> } >> """ >> err = inline(code,['nrows', 'ncolumns', 'res', 'm', 'v'], >> verbose=2, >> type_converters=converters.blitz) >> return res > > Here's what I'm envisioning for the above usecase -- I admit to not > using the pure-python mode that much, so there may be errors (and some > of what I'm using isn't implemented yet). But it shows how much is > already there in Cython's pure-python mode. > > @cython.compile # this is new & where most of the work is required! > @cython.locals(m=cython.float[:,::1], v=cython.float[::1]) > def prod(m, v): > cython.declare( > i=cython.int, > j=cython.int, > nrows=cython.int, > ncolumns=cython.int) > nrows = m.shape[0]; ncolumns = m.shape[1] > cython.declare(res=cython.float[::1]) > res = np.zeros(nrows, np.float) > for i in range(nrows): > for j in range(ncolumns): > res[i] += m[i,j] * v[j] > return res > > Once the new array type / memoryview slice is merged into cython-devel > and we get it integrated into pure-python mode, this will be close to > working. If you just want things fast in the function body (what > weave gives you currently) then the above is all you need. You beat me too it--this is almost exactly what I was going to write. (Just a note, it does understand the less verbose cython.declare (i=int, j=int, ...)). > The hard part is getting things to compile & run automatically for > just this function (the cython.compile decorator). This overlaps with > pyximport a good amount, and it will take some thought as to how it > will all work together. Perhaps the core devs won't like so much > going on behind the scenes -- cython is principally a utility for > generating a C extension module, not compiling & loading the extension > module. Personally, I think this is an awsome way to use Cython. Also, from the notebook one can create a Cython cell that does the same (compiles, loads, and spits the function right into your local namespace right from your browsers) which is really handy. Of course being able to make full modules with persistent globals and C functions that call each other is more powerful than just this. Even more, if cython is not available (or disabled for easier debugging, or no gcc was present) this could become a no-op, and would work just fine (though not as fast). > Weave compiles individual snippets into their own extension module, > whereas Cython has always compiled entire modules into extension > modules, and requires a separate compilation step (whether through the > commandline or through pyximport). The cython.compile decorator > would be different, if it were to be just like weave -- it would > compile a single function into its own extension module, without an > explicit compilation step. Much more is automated. > > This seems to be the main thing that weave offers -- fast code only > where you want it, without the explicit compilation step. Pyximport > overlaps here, but pyximport still compiles an entire module, not just > a part. > > So those are some of my thoughts. They'll be in a CEP sometime > soon :-) I have to say the one thing I don't like about weave is its use of strings to contain opaque code. OK, there's two things--the context switch between C and Fortran and Python. Nonetheless, I can see how it's really handy and a clever hack. I really want to avoid Cython allowing snippets of C and Fortran scattered in valid Cython sources if at all possible. (I would not be opposed to a cython.eval or cython.exec that would invoke the complier on a string). - Robert From robertwb at math.washington.edu Wed Aug 26 07:54:37 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 25 Aug 2009 22:54:37 -0700 Subject: [Cython] Using Cython with SymPy In-Reply-To: <52CDFB26-0817-493D-8347-DF85E9EF4FEE@gmail.com> References: <52CDFB26-0817-493D-8347-DF85E9EF4FEE@gmail.com> Message-ID: <238E9081-D863-461A-8503-FF4756DB027B@math.washington.edu> On Aug 19, 2009, at 11:53 AM, Aaron S. Meurer wrote: > Well, I don't know for a fact that this is truly what is happening. > It is just Ondrej's insight, plus the fact that when I comment out the > decorator, the import time is the same as it is in the master branch > without Cythonization, but when it is there, it is twice as slow for > repeated imports. > > Clearly something is slowing it down because the tests run a little > slower. Maybe Cythonizing more code would produce speed increases > that would overcome any slowdowns from imports, or maybe it would > level out. I haven't tried much. Ondrej is the one who is doing most > of this, so he will probably know more here. Any more insight on what is going on here? I would love to see SymPy on Cython (and from Sage of course). > > Aaron Meurer > On Aug 19, 2009, at 1:32 AM, Robert Bradshaw wrote: > >> On Aug 18, 2009, at 11:46 PM, Aaron S. Meurer wrote: >> >>> Hi. I am a developer for the project SymPy [0], which is a computer >>> algebra system written in Python. We have been considering moving >>> part or all of the code base into Cython for speed reasons. Ondrej >>> Certik, the project's leader, recently gave a tutorial at SciPy09 >>> (there should be a video up soon), which included a bit about >>> increasing the speed of an operation by about 20x by using the >>> cython.locals() decorator to convert some of the variables into >>> cython >>> ints. You can pull from his github account [1]. If you run: >>> >>>>>> from sympy import * >>>>>> d = divisors(5*10**7) >>> >>> The function will run about 20x to 30x faster, as I mentioned. But >>> if >>> you run the ntheory test, which uses divisors, using >>> >>> $./bin/test sympy/ntheory >>> >>> it runs slower or at least as slow as without cython. We figured >>> out >>> that the reason is that when the multiple imports in the tests are >>> called, they run about 2x slower with cython, because it calls the >>> decorator each time. >>> >>> You can see this by running >>> >>> In [1]: %timeit from sympy import * >>> >>> with and without cython in IPython. >>> >>> Is there any way around this? This could be a show stopper for us, >>> because the increased import times levels out or makes worse the >>> total >>> times for things. This is not exclusive to the test suite. It >>> comes >>> from multiple imports in each file. >>> >>> We know that we can use .pxd files, but that would require to handle >>> both .pxd files and .py files instead of maintaining one file with >>> everything. >> >> Eventually I want to get type inference, which could help here. >> However, I'm not following how decorating these functions makes >> everything slower. Sure, it's a bit slower on the first import, but >> the decorator is no re-called on each import. Clearly I'm missing >> something here, could you explain why the decorator is being >> repeatedly called? >> >> In the worst case, maintaining separate .pxd files seems a small >> price to pay to get a 20-30x speedup. Also, if you want to have cdef >> classes, especially across various modules, you'll have to do that >> anyways right now. (Patches for fixing this, e.g. via class >> decorators, would be highly appreciated). >> >> - Robert >> >> >> ------------------------- foo.py ----------------------- >> >> def decorate(f): >> print "here" >> return f >> >> @decorate >> def f(x): >> return x >> >> ---------------------------------------------------------- >> >> sage: from foo import * >> here >> sage: from foo import * >> sage: from foo import * >> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From robertwb at math.washington.edu Wed Aug 26 08:05:55 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 25 Aug 2009 23:05:55 -0700 Subject: [Cython] Fwd: Type checking bug when passing None instead of a Cython object In-Reply-To: <4A942F43.30004@noaa.gov> References: <4A93CC39.6020903@ebi.ac.uk> <995A6088-6BAC-4DF9-8360-733B2D88EFC7@math.washington.edu> <4A9418EC.4050709@student.matnat.uio.no> <4A9426A6.4050007@noaa.gov> <4A9427DC.6060307@student.matnat.uio.no> <4A94292E.7020907@student.matnat.uio.no> <4A942987.1080903@student.matnat.uio.no> <4A942F43.30004@noaa.gov> Message-ID: <70337FE3-457E-4672-ABDA-CC0B4126BBD3@math.washington.edu> On Aug 25, 2009, at 11:36 AM, Christopher Barker wrote: > Dag Sverre Seljebotn wrote: >>>> (Well, at least two people (myself and Robert) has opposed this >>>> being >>>> the default (and not for reasons of backwards compatability!), so I >>>> don't think we're at the stage where we can discuss deprecation >>>> just yet >>>> :-)) > > My point was that IF any backward incompatible changes were made, > there > should be a reasonable deprecation procedure. Whether the change > should > be made or not I'll leave to others -- what seems obvious to me could > well be a function of my very limited use of Cython so far. > > What I do know is that the current behavior is a surprise to > newbies -- > a bit more documentation may be all we need. Yes, thanks! Maybe there should even be a "common pitfalls" page for newbies. I would be in favor of adding a compiler directive for this behavior, so that people who wanted "not None" everywhere could just stick it in one spot. >>> If a faster "nonecheck" was added >>> then "not None" would be a really redundant part of the language. > > Is that possible (well, not "faster", but fast enough not to > notice). My > understanding of nonecheck is that a check has to be done on every > object access (Or index, or...). In a tight loop that indexes > through a > numpy array and does very little, that just doesn't seem practical. Or > could you put the check outside the loop, but not at the beginning of > the function? With control flow analysis, one could statically eliminate almost all checks (in this example, if the argument were never assigned to, it would probably only need to be checked on the first access). - Robert From fperez.net at gmail.com Wed Aug 26 08:23:38 2009 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 25 Aug 2009 23:23:38 -0700 Subject: [Cython] scipy'09 cython BoF In-Reply-To: References: <200908252025.09211.binet@cern.ch> Message-ID: On Tue, Aug 25, 2009 at 10:52 PM, Robert Bradshaw wrote: > I have to say the one thing I don't like about weave is its use of > strings to contain opaque code. OK, there's two things--the context > switch between C and Fortran and Python. Nonetheless, I can see how > it's really handy and a clever hack. I really want to avoid Cython > allowing snippets of C and Fortran scattered in valid Cython sources > if at all possible. (I would not be opposed to a cython.eval or > cython.exec that would invoke the complier on a string). I should add that I'm not asking for *the* weave approach, just for that overall level of functionality and flexibility. When weave was written, nothing like cython's underlying machinery was even in the horizon, so weave was already a phenomenal tool to have. But if the equivalent functionality can be built, more cleanly on top of cython, great! I'm just trying to provide you guys with context on the kinds of things I've used weave for in the past and found it useful, not to push for any particular implementation details. Regards, f From glenn at tarbox.org Wed Aug 26 17:41:41 2009 From: glenn at tarbox.org (Glenn Tarbox, PhD) Date: Wed, 26 Aug 2009 08:41:41 -0700 Subject: [Cython] PySide to replace PyQt - Cython could play a major role Message-ID: "Mr President, we have a situation" I posted the following to the IPython and Enthought-dev lists last night. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sooner or later, something was gonna need to happen WRT Riverbank and the PyQt licensing. I had hoped that an entirely new project wasn't going to be necessary.... but apparently it is. PySide Released to the Wild: http://labs.trolltech.com/blogs/2009/08/25/pyside-released-to-the-wild/ >From the PySide FAQ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> What about PyQt? Nokia?s initial research into Python bindings for Qt involved speaking with Riverbank Computing, the makers of PyQt. We had several discussions with them to see if it was possible to use PyQt to achieve our goals. Unfortunately, a common agreement could not be found , so in the end we decided to proceed with PySide. We will however maintain API compatibility with PyQt (you can use the same method names but can?t inter-operate with PyQt), at least for the initial release. To import PySide you have to use ?import PySide? instead of ?import PyQt4?. <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< I didn't know where to post this. PySide needs to mature a bit (support more than Linux for example) but both Matplotlib and Enthought are affected. PyQt will likely need to be replaced in both packages once PySide becomes more mature as the licensing of PyQt is problematic now that Qt is LGPL. Its also likely that with Nokia's backing, the PySide API will eventually dominate. Hopefully, the above statement regarding the similarity of the API will make moving over easy. Personally I'd like to see a focus on Qt vs Wx by Enthought as I believe it to be much more powerful... but thats my personal opinion and what I use. As a side note, I've successfully nailed my C++ Qt code to IPython using a Cython shim. The PyQt event loop is available and all seems to work great. <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Eric Jones responded on Enthought-dev >>>>>>>>>>>>>>>>>>>>>>>> 1. We are switching our focus to Qt and away from wx for all new work. That happened when the Qt LGPL was announced. We aren't abandoning wx in the near future, though. To much of our legacy stuff relies on it. <<<<<<<<<<<<<<<<<<<<<<<< My guess is there are relatively few Gui / native graphics folks in the Cython community but it seems that there is, potentially, an opportunity to pull a number of critical pieces together with Cython as a modification of the current yet early PySide strategy. The key, other than Cython being a better way to nail Python to C/C++, is the recent Cython Numpy work. If Enthought (hence Scipy) is going to focus on Qt, and the Python <-> Qt integration is in play, perhaps this is an opportunity to do this right. One of the issues being worked with PySide is the parsing of C++ header files to generate Python interfaces. I know Cython has been eying this issue for a while. Throw on the Numpy enhancements and perhaps things Cython become even more compelling to Enthought / Scipy. While I haven't dug deeply enough, perhaps Matplotlib is in the middle of this as well. Qt has features which makes it a better (IMHO) back end than may of the alternatives. One example is the OpenGL back-end which improves performance without changing the 2D API. In my case, I got huge speedups just by swapping back-ends (4 lines of C++ code) because X11 was no longer involved in plotting. I went fom a pegged cpu supporting X to, essentially 0% because X involvement goes away. This is a bigger deal than one might think when working with large data sets and comes virtually for free. PySide appears to be fairly immature yet Nokia is behind it and its gonna happen. The announcement posted above was yesterday. Perhaps that was the first "official" announcement once the Riverbank discussions failed. Seems to me that explaining how Cython could make everyone's (including Nokia's) life much easier, providing things important to IPython / Scipy / Enthought, and potentially enhancing Cython with a capability to parse C/C++ headers could be a pretty big win all around. -glenn -- Glenn H. Tarbox, PhD || 206-274-6919 http://www.tarbox.org -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090826/1309d7b0/attachment.htm From adam.ginsburg at colorado.edu Wed Aug 26 18:31:35 2009 From: adam.ginsburg at colorado.edu (Adam Ginsburg) Date: Wed, 26 Aug 2009 10:31:35 -0600 Subject: [Cython] Linking error - 64 bit problem? In-Reply-To: References: <4A944963.1020403@student.matnat.uio.no> Message-ID: >>> A future optimization would be the @cython.boundscheck(False) directive. >> >> Do you mean future as in "don't try to use it now" or "should use it >> if it's safe to proceed without boundary checking"? ?Also, this looks >> like a decorator to me, but I couldn't compile when I put it in front >> of my function definition. > > Did you import cython? You'd think I could catch something like that.... from now on, coffee before code. > 10^4 still isn't a huge array, but I would be curious how much a speed > increase you could get without unrolling all your loops (i.e. a minimal > modification to your code). Might be 1.6, might not be. I've now rewritten the cython code so that it is identical to the fortran, and now cython is ~10% slower than fortran but much cleaner-looking. Thanks. > This is due to the fact that the array processing happens at near the same > speed, so as the array gets larger, the (relative) significance of the > Python function call overhead goes away. That makes sense; the python overhead is increasing proportional to n and the overall execution time proportional to n^2. Thanks everyone, I think I now understand cython well enough to make some code without help next time. I was slow to figure out that cython really is c/fortran loop-style coding with python syntax. Adam From kwmsmith at gmail.com Wed Aug 26 18:33:29 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Wed, 26 Aug 2009 11:33:29 -0500 Subject: [Cython] Giving fwrap a home In-Reply-To: <9E2BDBD3-4D7C-45C9-8F43-A3F647CF14D1@math.washington.edu> References: <85e81ba30908251631n78de4e2dk849826cf5a86593d@mail.gmail.com> <9E2BDBD3-4D7C-45C9-8F43-A3F647CF14D1@math.washington.edu> Message-ID: On Wed, Aug 26, 2009 at 12:18 AM, Robert Bradshaw wrote: > On Aug 25, 2009, at 6:37 PM, Kurt Smith wrote: > >> On Tue, Aug 25, 2009 at 6:31 PM, William Stein >> wrote: >>> On Tue, Aug 25, 2009 at 12:18 PM, Kurt Smith >>> wrote: >>>> I think its best that fwrap be its own package, distributed >>>> separately >>>> from Cython. ?That was the vibe I got from those at the SciPy 2009 >>>> conference and from the recent thread on Fwrap's licensing. ?The >>>> added >>>> benefit is that fwrap won't weigh Cython down w.r.t. licensing >>>> issues >>>> or be an impediment to Cython's acceptance into the Python std. lib. >>>> >>>> Presuming that everyone here agrees with the above (or doesn't >>>> care), >>>> the remaining question is where to host it. ?Since fwrap is still >>>> closely linked to Cython, I think a natural spot for the mercurial >>>> repo would be on Cython's servers. ?No strong feelings here -- I'm >>>> just as happy putting it on bitbucket. ?Fwrap would *not* clutter up >>>> Cython's trac, or Cython's wiki. ?These would be elsewhere. >>>> >>> >>> As the own of Cython's servers, I hereby certainly offer you >>> hosting space. >> >> Thanks! ?I'm evaluating bitbucket & googlecode to see what they offer >> -- apparently googlecode has mercurial support, so that is a big plus. >> ?I'll decide soon & let you know. > > I wouldn't consider it "clutter" to have an fwrap section of the > wiki, nor host another porject on the cython trac server. Well, this changes things significantly -- thanks for the offer! Since I'd rather work with you folks than some external organization, I'll keep fwrap on Cython's servers, wiki & trac. Thanks for the generosity! I'll start setting things up over the weekend. What's the next step(s)? Kurt From robertwb at math.washington.edu Wed Aug 26 19:21:05 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 26 Aug 2009 10:21:05 -0700 (PDT) Subject: [Cython] Giving fwrap a home In-Reply-To: References: <85e81ba30908251631n78de4e2dk849826cf5a86593d@mail.gmail.com> <9E2BDBD3-4D7C-45C9-8F43-A3F647CF14D1@math.washington.edu> Message-ID: On Wed, 26 Aug 2009, Kurt Smith wrote: > On Wed, Aug 26, 2009 at 12:18 AM, Robert > Bradshaw wrote: >> On Aug 25, 2009, at 6:37 PM, Kurt Smith wrote: >> >>> On Tue, Aug 25, 2009 at 6:31 PM, William Stein >>> wrote: >>>> On Tue, Aug 25, 2009 at 12:18 PM, Kurt Smith >>>> wrote: >>>>> I think its best that fwrap be its own package, distributed >>>>> separately >>>>> from Cython. ?That was the vibe I got from those at the SciPy 2009 >>>>> conference and from the recent thread on Fwrap's licensing. ?The >>>>> added >>>>> benefit is that fwrap won't weigh Cython down w.r.t. licensing >>>>> issues >>>>> or be an impediment to Cython's acceptance into the Python std. lib. >>>>> >>>>> Presuming that everyone here agrees with the above (or doesn't >>>>> care), >>>>> the remaining question is where to host it. ?Since fwrap is still >>>>> closely linked to Cython, I think a natural spot for the mercurial >>>>> repo would be on Cython's servers. ?No strong feelings here -- I'm >>>>> just as happy putting it on bitbucket. ?Fwrap would *not* clutter up >>>>> Cython's trac, or Cython's wiki. ?These would be elsewhere. >>>>> >>>> >>>> As the own of Cython's servers, I hereby certainly offer you >>>> hosting space. >>> >>> Thanks! ?I'm evaluating bitbucket & googlecode to see what they offer >>> -- apparently googlecode has mercurial support, so that is a big plus. >>> ?I'll decide soon & let you know. >> >> I wouldn't consider it "clutter" to have an fwrap section of the >> wiki, nor host another porject on the cython trac server. > > Well, this changes things significantly -- thanks for the offer! > > Since I'd rather work with you folks than some external organization, > I'll keep fwrap on Cython's servers, wiki & trac. Thanks for the > generosity! > > I'll start setting things up over the weekend. What's the next step(s)? If you point me to your repository, I'll put it up next to ours. I'll also set up a trac project for you, and for the wiki I think wiki.cython.org/fwrap should be good enough. - Robert From dagss at student.matnat.uio.no Wed Aug 26 20:23:10 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 26 Aug 2009 20:23:10 +0200 Subject: [Cython] Thoughts after SciPy 09 Message-ID: <4A957D8E.7070003@student.matnat.uio.no> As many of you know me and Kurt attended SciPy 09. Four Cython-related events were held: - An introductory tutorial to Cython (by me) - A talk about Cython for numerics (by me again) - A talk on Fwrap (by Kurt) - A Cython BoF You can find links to slides and videos for the three first on http://conference.scipy.org. An intensive week like that makes me reflect on what Cython is good about, lacking, etc. etc. First of all, there seems to be quite a lot of interest in Cython, many thinks it is excellent, and many thanked me personally for our efforts. One thing that's also very interesting to me personally is that there's some talk of porting parts of NumPy over to Cython for easier Python 3 support. Beyond that, I've got a nice list of topics for further improvement. For instance one thing that is very possible to fix was a real dealbreaker that some complained about, and in one case stopped somebody from recommending it to co-workers. It's always nice to get the "outside" perspective that I get when I present Cython to lots of people. BUT, I'm going to wait a little with presenting that list. It simply feels too depressing to discuss lots of new ideas etc. without dealing properly with how such ideas can ever be brought to life. In the end, what matters right now is to stabilize -unstable, get the GSoC work merged in, and get 0.12 released; which at the current pace should keep us busy for some months to come. On that note though: It seems to me that many has the impression that a) Cython is complicated technology which takes much work b) A lot of effort is put into steadily improving it BUT, I feel the reality is that a) Core developers can implement new features or fix bugs rather quickly b) Relatively little time is spent in total on Cython, compared to some other projects I don't mean to belittle the efforts Robert and Stefan and others have put into Cython; I just feel that the amount of work going on in Cython is really less than people think it is. Or put another way: Putting relatively little in can, at least at this point in Cython's development, yield high returns. Example: Profiling was a feature many at SciPy was anxious about getting and was asking about a lot. That's in trunk now, mainly because Robert had an intercontinental flight (!!). (That admittedly might say a lot more about Robert than Cython, but still.) In a year things might have changed; I think this is a perfect time to evaluate whether Cython should solicit help of some form. The best would be CS master students ready to work on Cython (perhaps they can come to Oslo for their MSc :-)). Donations towards employing one of the core devs (Robert or Stefan, if at all possible...) for one day every week could do wonders for the whole project and likely energize everyone else as well as one sees bugs fixed and releases made, but is probably not realistic? Cython can thrive without this too though! Looking at the coming half-to-three-quarter year, here's what I'm guessing will happen: - I might get the new memoryviews from summer finished and merged with trunk - Cython might run properly in Python 3 (w/ 2to3) - Get -unstable stabilized and released (significant portion) - Closures - Fwrap released - Better C++ support merged - Perhaps some pyximport improvements Not bad at all! But, there's also a long list of projects we already badly want to have done that we can't possibly reach now, IMO: - Fix the bugs, complete the test suite - Speed up compilation speed, break up compilation units/utility code - Control flow analysis! - Convenient debugging, line-by-line profiling - SIMD - Many rather low-hanging fruit CEPs which would make using Cython a nicer experience - Full Python semantics compatability for untyped code - Type inference/a better pure Python mode It is this last list I don't want to make longer just now by coming with all the ideas and input I got during SciPy 09 :-) Thoughts on the picture I'm painting? -- Dag Sverre From rkern at enthought.com Wed Aug 26 20:27:12 2009 From: rkern at enthought.com (Robert Kern) Date: Wed, 26 Aug 2009 11:27:12 -0700 Subject: [Cython] [Enthought-Dev] PySide to replace PyQt - Cython could play a major role In-Reply-To: References: Message-ID: <3d375d730908261127s535f9404n9c58d9a267c3bec6@mail.gmail.com> On Wed, Aug 26, 2009 at 10:45, Matthieu Brucher wrote: > I don't think there is a chance of PySide using another toolbox. I'll > explain myself. > PySide uses Boost.Python, and thus has access to a complete excellent > parser (Spirit) and other tools to automate its build (it was proved > with Py++). The bindings generator and the bindings themselves are separate things. The bindings generator can use Boost all it likes for parsing and graph manipulations internally without using Boost.Python for the generated code. The PySide team is looking at other strategies for the generated code. There is a branch under weigh that uses just the Python C API. > Another point is that Cython generates C code, and not C++ > code. Boost and Qt are known to be C++ frameworks, and heavily rely on > this. If Cython cannot output C++ code, I don't know if it can compete > with Boost... Cython can output C++ code and wrap C++ libraries. However, I don't think it supports everything that Boost.Python does, particularly overriding virtual methods from the Python side. I doubt Cython would help them much over just using the Python C API, but it is a viable option. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From dagss at student.matnat.uio.no Wed Aug 26 20:41:37 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 26 Aug 2009 20:41:37 +0200 Subject: [Cython] PySide to replace PyQt - Cython could play a major role In-Reply-To: References: Message-ID: <4A9581E1.707@student.matnat.uio.no> Glenn Tarbox, PhD wrote: > Seems to me that explaining how Cython could make everyone's (including > Nokia's) life much easier, providing things important to IPython / > Scipy / Enthought, and potentially enhancing Cython with a capability to > parse C/C++ headers could be a pretty big win all around. Enthought certainly already know about and appriciate Cython. But they likely has neither money nor manpower to spare now (not that I really know anything about it, but there's this crisis and so on). I don't really see them taking on developing Cython in addition to everything else they are developing, just to make switch from PyQT in a "cleaner" fashion. -- Dag Sverre From dagss at student.matnat.uio.no Wed Aug 26 20:49:10 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 26 Aug 2009 20:49:10 +0200 Subject: [Cython] PySide to replace PyQt - Cython could play a major role In-Reply-To: <4A9581E1.707@student.matnat.uio.no> References: <4A9581E1.707@student.matnat.uio.no> Message-ID: <4A9583A6.1070805@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Glenn Tarbox, PhD wrote: >> Seems to me that explaining how Cython could make everyone's (including >> Nokia's) life much easier, providing things important to IPython / >> Scipy / Enthought, and potentially enhancing Cython with a capability to >> parse C/C++ headers could be a pretty big win all around. > > Enthought certainly already know about and appriciate Cython. But they > likely has neither money nor manpower to spare now (not that I really > know anything about it, but there's this crisis and so on). I don't > really see them taking on developing Cython in addition to everything > else they are developing, just to make switch from PyQT in a "cleaner" > fashion. Hmm. Perhaps this isn't what you meant at all. But at any rate the point seems to be moot, as discussed by Robert K. and Mattieu -- and I definitely see their point, I'd seriously doubt using Cython for wrapping QT myself at this point, since QT is so C++-centric. -- Dag Sverre From dagss at student.matnat.uio.no Wed Aug 26 21:44:04 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 26 Aug 2009 21:44:04 +0200 Subject: [Cython] Thoughts after SciPy 09 In-Reply-To: <4A957D8E.7070003@student.matnat.uio.no> References: <4A957D8E.7070003@student.matnat.uio.no> Message-ID: <4A959084.5000600@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > As many of you know me and Kurt attended SciPy 09. Four Cython-related > events were held: > > - An introductory tutorial to Cython (by me) > - A talk about Cython for numerics (by me again) > - A talk on Fwrap (by Kurt) > - A Cython BoF > > You can find links to slides and videos for the three first on > http://conference.scipy.org. > > An intensive week like that makes me reflect on what Cython is good > about, lacking, etc. etc. > > First of all, there seems to be quite a lot of interest in Cython, many > thinks it is excellent, and many thanked me personally for our efforts. One other thought from SciPy 09: It emphasised to me that one of the most important things missing in Cython, perhaps more than any single feature, is solid, updated introductory-level documentation. (docs.cython.org is more of a reference...) Luckily we hope to be able to do something about this shortly though, in the conference proceedings. -- Dag Sverre From robertwb at math.washington.edu Thu Aug 27 05:12:33 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 26 Aug 2009 20:12:33 -0700 Subject: [Cython] Thoughts after SciPy 09 In-Reply-To: <4A959084.5000600@student.matnat.uio.no> References: <4A957D8E.7070003@student.matnat.uio.no> <4A959084.5000600@student.matnat.uio.no> Message-ID: <82AAB96C-260F-4105-B557-761B078824CE@math.washington.edu> On Aug 26, 2009, at 12:44 PM, Dag Sverre Seljebotn wrote: > Dag Sverre Seljebotn wrote: >> As many of you know me and Kurt attended SciPy 09. Four Cython- >> related >> events were held: >> >> - An introductory tutorial to Cython (by me) >> - A talk about Cython for numerics (by me again) >> - A talk on Fwrap (by Kurt) >> - A Cython BoF >> >> You can find links to slides and videos for the three first on >> http://conference.scipy.org. >> >> An intensive week like that makes me reflect on what Cython is good >> about, lacking, etc. etc. >> >> First of all, there seems to be quite a lot of interest in Cython, >> many >> thinks it is excellent, and many thanked me personally for our >> efforts. > > One other thought from SciPy 09: It emphasised to me that one of the > most important things missing in Cython, perhaps more than any single > feature, is solid, updated introductory-level documentation. > (docs.cython.org is more of a reference...) I've been toying with the idea of making a "Dive into Cython" by basically taking the "Dive into Python" tutorial and re-writing it for Cython (which chapters like "Not Everything's an Object.") > Luckily we hope to be able to do something about this shortly > though, in > the conference proceedings. Yep. - Robert From robertwb at math.washington.edu Thu Aug 27 05:35:15 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 26 Aug 2009 20:35:15 -0700 Subject: [Cython] Thoughts after SciPy 09 In-Reply-To: <4A957D8E.7070003@student.matnat.uio.no> References: <4A957D8E.7070003@student.matnat.uio.no> Message-ID: <3344F9B0-DED7-4403-9BC9-AC2A4063B49C@math.washington.edu> On Aug 26, 2009, at 11:23 AM, Dag Sverre Seljebotn wrote: > As many of you know me and Kurt attended SciPy 09. Four Cython-related > events were held: > > - An introductory tutorial to Cython (by me) > - A talk about Cython for numerics (by me again) > - A talk on Fwrap (by Kurt) > - A Cython BoF > > You can find links to slides and videos for the three first on > http://conference.scipy.org. > > An intensive week like that makes me reflect on what Cython is good > about, lacking, etc. etc. > > First of all, there seems to be quite a lot of interest in Cython, > many > thinks it is excellent, and many thanked me personally for our > efforts. > > One thing that's also very interesting to me personally is that > there's > some talk of porting parts of NumPy over to Cython for easier Python 3 > support. Sounds like things went really well. > Beyond that, I've got a nice list of topics for further > improvement. For > instance one thing that is very possible to fix was a real dealbreaker > that some complained about, and in one case stopped somebody from > recommending it to co-workers. It's always nice to get the "outside" > perspective that I get when I present Cython to lots of people. > > BUT, I'm going to wait a little with presenting that list. It simply > feels too depressing to discuss lots of new ideas etc. without dealing > properly with how such ideas can ever be brought to life. In the end, > what matters right now is to stabilize -unstable, get the GSoC work > merged in, and get 0.12 released; which at the current pace should > keep > us busy for some months to come. > > On that note though: It seems to me that many has the impression that > > a) Cython is complicated technology which takes much work > b) A lot of effort is put into steadily improving it > > BUT, I feel the reality is that > > a) Core developers can implement new features or fix bugs rather > quickly > b) Relatively little time is spent in total on Cython, compared to > some > other projects > > I don't mean to belittle the efforts Robert and Stefan and others have > put into Cython; I just feel that the amount of work going on in > Cython > is really less than people think it is. > > Or put another way: Putting relatively little in can, at least at this > point in Cython's development, yield high returns. [...] > Thoughts on the picture I'm painting? I think you've hit the nail on the head here. Personally, there are tons of things I'd love to do with Cython (many of which you've listed), and I think it's headed in a very cool direction, but what time I have is mostly going to higher-priority tasks like work and school and 6 month old twin girls. (Also, for the moment, as fun as hacking on Cython is, it unfortunately doesn't directly contribute to thesis or job.) I'm confident that in the long term (say, looking at the next year or maybe two) - Robert From wstein at gmail.com Thu Aug 27 06:02:43 2009 From: wstein at gmail.com (William Stein) Date: Wed, 26 Aug 2009 21:02:43 -0700 Subject: [Cython] Thoughts after SciPy 09 In-Reply-To: <82AAB96C-260F-4105-B557-761B078824CE@math.washington.edu> References: <4A957D8E.7070003@student.matnat.uio.no> <4A959084.5000600@student.matnat.uio.no> <82AAB96C-260F-4105-B557-761B078824CE@math.washington.edu> Message-ID: <85e81ba30908262102j732c5c7ao5066468c2570e438@mail.gmail.com> On Wed, Aug 26, 2009 at 8:12 PM, Robert Bradshaw wrote: > I've been toying with the idea of making a "Dive into Cython" by > basically taking the "Dive into Python" tutorial and re-writing it > for Cython (which chapters like "Not Everything's an Object.") > That's an intriguing idea. Last week a very similar idea came up at the Sage seminar, where it was suggested that somebody write a "Dive into Sage" by rewriting the Dive into Python book. Has anybody ever rewritten that book to make a "Dive into X?" William From robertwb at math.washington.edu Thu Aug 27 06:31:30 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 26 Aug 2009 21:31:30 -0700 Subject: [Cython] Thoughts after SciPy 09 In-Reply-To: <3344F9B0-DED7-4403-9BC9-AC2A4063B49C@math.washington.edu> References: <4A957D8E.7070003@student.matnat.uio.no> <3344F9B0-DED7-4403-9BC9-AC2A4063B49C@math.washington.edu> Message-ID: On Aug 26, 2009, at 8:35 PM, Robert Bradshaw wrote: > On Aug 26, 2009, at 11:23 AM, Dag Sverre Seljebotn wrote: > >> As many of you know me and Kurt attended SciPy 09. Four Cython- >> related >> events were held: >> >> - An introductory tutorial to Cython (by me) >> - A talk about Cython for numerics (by me again) >> - A talk on Fwrap (by Kurt) >> - A Cython BoF >> >> You can find links to slides and videos for the three first on >> http://conference.scipy.org. >> >> An intensive week like that makes me reflect on what Cython is good >> about, lacking, etc. etc. >> >> First of all, there seems to be quite a lot of interest in Cython, >> many >> thinks it is excellent, and many thanked me personally for our >> efforts. >> >> One thing that's also very interesting to me personally is that >> there's >> some talk of porting parts of NumPy over to Cython for easier >> Python 3 >> support. > > Sounds like things went really well. > >> Beyond that, I've got a nice list of topics for further >> improvement. For >> instance one thing that is very possible to fix was a real >> dealbreaker >> that some complained about, and in one case stopped somebody from >> recommending it to co-workers. It's always nice to get the "outside" >> perspective that I get when I present Cython to lots of people. >> >> BUT, I'm going to wait a little with presenting that list. It simply >> feels too depressing to discuss lots of new ideas etc. without >> dealing >> properly with how such ideas can ever be brought to life. In the end, >> what matters right now is to stabilize -unstable, get the GSoC work >> merged in, and get 0.12 released; which at the current pace should >> keep >> us busy for some months to come. >> >> On that note though: It seems to me that many has the impression that >> >> a) Cython is complicated technology which takes much work >> b) A lot of effort is put into steadily improving it >> >> BUT, I feel the reality is that >> >> a) Core developers can implement new features or fix bugs rather >> quickly >> b) Relatively little time is spent in total on Cython, compared to >> some >> other projects >> >> I don't mean to belittle the efforts Robert and Stefan and others >> have >> put into Cython; I just feel that the amount of work going on in >> Cython >> is really less than people think it is. >> >> Or put another way: Putting relatively little in can, at least at >> this >> point in Cython's development, yield high returns. > > [...] > >> Thoughts on the picture I'm painting? > > I think you've hit the nail on the head here. > > Personally, there are tons of things I'd love to do with Cython (many > of which you've listed), and I think it's headed in a very cool > direction, but what time I have is mostly going to higher-priority > tasks like work and school and 6 month old twin girls. (Also, for the > moment, as fun as hacking on Cython is, it unfortunately doesn't > directly contribute to thesis or job.) I'm confident that in the long > term (say, looking at the next year or maybe two) Oops. I didn't finish my sentence. I'm confident that that we can hit most of the points on that feature list in the next year or two. - Robert From dagss at student.matnat.uio.no Thu Aug 27 07:21:47 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 27 Aug 2009 07:21:47 +0200 Subject: [Cython] Thoughts after SciPy 09 In-Reply-To: References: <4A957D8E.7070003@student.matnat.uio.no> <3344F9B0-DED7-4403-9BC9-AC2A4063B49C@math.washington.edu> Message-ID: <4A9617EB.10701@student.matnat.uio.no> Robert Bradshaw wrote: > On Aug 26, 2009, at 8:35 PM, Robert Bradshaw wrote: > >> On Aug 26, 2009, at 11:23 AM, Dag Sverre Seljebotn wrote: >>> On that note though: It seems to me that many has the impression that >>> >>> a) Cython is complicated technology which takes much work >>> b) A lot of effort is put into steadily improving it >>> >>> BUT, I feel the reality is that >>> >>> a) Core developers can implement new features or fix bugs rather >>> quickly >>> b) Relatively little time is spent in total on Cython, compared to >>> some >>> other projects >>> >>> I don't mean to belittle the efforts Robert and Stefan and others >>> have >>> put into Cython; I just feel that the amount of work going on in >>> Cython >>> is really less than people think it is. >>> >>> Or put another way: Putting relatively little in can, at least at >>> this >>> point in Cython's development, yield high returns. >> [...] >> >>> Thoughts on the picture I'm painting? >> I think you've hit the nail on the head here. >> >> Personally, there are tons of things I'd love to do with Cython (many >> of which you've listed), and I think it's headed in a very cool >> direction, but what time I have is mostly going to higher-priority >> tasks like work and school and 6 month old twin girls. (Also, for the >> moment, as fun as hacking on Cython is, it unfortunately doesn't >> directly contribute to thesis or job.) I'm confident that in the long >> term (say, looking at the next year or maybe two) > > > Oops. I didn't finish my sentence. I'm confident that that we can hit > most of the points on that feature list in the next year or two. OK. You're more confident than I am then :-) Anyway, I don't have much time between school and 12 month old girl either. I'm hoping to be able to put in about 2 hours a week the coming year after September...which is rather less than in the previous year. (Perhaps I should simply start by being less active on the mailing list.) -- Dag Sverre From seb.binet at gmail.com Thu Aug 27 09:28:17 2009 From: seb.binet at gmail.com (Sebastien Binet) Date: Thu, 27 Aug 2009 09:28:17 +0200 Subject: [Cython] Thoughts after SciPy 09 In-Reply-To: <4A959084.5000600@student.matnat.uio.no> References: <4A957D8E.7070003@student.matnat.uio.no> <4A959084.5000600@student.matnat.uio.no> Message-ID: <200908270928.17936.binet@cern.ch> On Wednesday 26 August 2009 21:44:04 Dag Sverre Seljebotn wrote: > Dag Sverre Seljebotn wrote: > > As many of you know me and Kurt attended SciPy 09. Four Cython-related > > events were held: > > > > - An introductory tutorial to Cython (by me) > > - A talk about Cython for numerics (by me again) > > - A talk on Fwrap (by Kurt) > > - A Cython BoF > > > > You can find links to slides and videos for the three first on > > http://conference.scipy.org. > > > > An intensive week like that makes me reflect on what Cython is good > > about, lacking, etc. etc. > > > > First of all, there seems to be quite a lot of interest in Cython, many > > thinks it is excellent, and many thanked me personally for our efforts. > > One other thought from SciPy 09: It emphasised to me that one of the > most important things missing in Cython, perhaps more than any single > feature, is solid, updated introductory-level documentation. > (docs.cython.org is more of a reference...) and probably a howto for developers: cython's citymap, the 10,000 ft view of the overall organization of cython's code, the various concepts, etc... right now I have just been playing on the shore w/o the ability (nor the time, to be frank) to play further with the code. cheers, sebastien. -- ######################################### # Dr. Sebastien Binet # Laboratoire de l'Accelerateur Lineaire # Universite Paris-Sud XI # Batiment 200 # 91898 Orsay ######################################### From dagss at student.matnat.uio.no Thu Aug 27 09:42:28 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 27 Aug 2009 09:42:28 +0200 Subject: [Cython] Thoughts after SciPy 09 In-Reply-To: <200908270928.17936.binet@cern.ch> References: <4A957D8E.7070003@student.matnat.uio.no> <4A959084.5000600@student.matnat.uio.no> <200908270928.17936.binet@cern.ch> Message-ID: <4A9638E4.3070709@student.matnat.uio.no> Sebastien Binet wrote: > On Wednesday 26 August 2009 21:44:04 Dag Sverre Seljebotn wrote: >> Dag Sverre Seljebotn wrote: >>> As many of you know me and Kurt attended SciPy 09. Four Cython-related >>> events were held: >>> >>> - An introductory tutorial to Cython (by me) >>> - A talk about Cython for numerics (by me again) >>> - A talk on Fwrap (by Kurt) >>> - A Cython BoF >>> >>> You can find links to slides and videos for the three first on >>> http://conference.scipy.org. >>> >>> An intensive week like that makes me reflect on what Cython is good >>> about, lacking, etc. etc. >>> >>> First of all, there seems to be quite a lot of interest in Cython, many >>> thinks it is excellent, and many thanked me personally for our efforts. >> One other thought from SciPy 09: It emphasised to me that one of the >> most important things missing in Cython, perhaps more than any single >> feature, is solid, updated introductory-level documentation. >> (docs.cython.org is more of a reference...) > and probably a howto for developers: cython's citymap, the 10,000 ft view of > the overall organization of cython's code, the various concepts, etc... > > right now I have just been playing on the shore w/o the ability (nor the time, > to be frank) to play further with the code. I'm not too sure about this. It's always a question whether the investment is worth it. I have a strong feeling that developing on Cython will for the foreseeable future happen through mentoring. Trying to upfront come up with all the potential questions one might ask about the code base, or just writing down the 10 000 ft organization of the code, is very time consuming. (Especially since Cython's code organization isn't the nicest -- but I'd rather spend time to very gradually fix that than to document it...) So: When you get the time to contribute to Cython, just state what you are interested in fixing, and the existing developers will come along and tell you what you need to know. There's been several threads on the mailing list that's basically "how to fix this bug" (over the summer I mentored Kurt over Skype which worked out excellently.) Perhaps we should state just that somewhere visible though. (BTW, for anyone reading this, a nice, rather low-hanging fruit right now is just to write testcases for pyximport and merge in the existing patches for it.) -- Dag Sverre From dagss at student.matnat.uio.no Thu Aug 27 09:53:42 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 27 Aug 2009 09:53:42 +0200 Subject: [Cython] Thoughts after SciPy 09 In-Reply-To: <4A9638E4.3070709@student.matnat.uio.no> References: <4A957D8E.7070003@student.matnat.uio.no> <4A959084.5000600@student.matnat.uio.no> <200908270928.17936.binet@cern.ch> <4A9638E4.3070709@student.matnat.uio.no> Message-ID: <4A963B86.3010408@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Sebastien Binet wrote: >> On Wednesday 26 August 2009 21:44:04 Dag Sverre Seljebotn wrote: >>> Dag Sverre Seljebotn wrote: >>>> As many of you know me and Kurt attended SciPy 09. Four Cython-related >>>> events were held: >>>> >>>> - An introductory tutorial to Cython (by me) >>>> - A talk about Cython for numerics (by me again) >>>> - A talk on Fwrap (by Kurt) >>>> - A Cython BoF >>>> >>>> You can find links to slides and videos for the three first on >>>> http://conference.scipy.org. >>>> >>>> An intensive week like that makes me reflect on what Cython is good >>>> about, lacking, etc. etc. >>>> >>>> First of all, there seems to be quite a lot of interest in Cython, many >>>> thinks it is excellent, and many thanked me personally for our efforts. >>> One other thought from SciPy 09: It emphasised to me that one of the >>> most important things missing in Cython, perhaps more than any single >>> feature, is solid, updated introductory-level documentation. >>> (docs.cython.org is more of a reference...) >> and probably a howto for developers: cython's citymap, the 10,000 ft view of >> the overall organization of cython's code, the various concepts, etc... >> >> right now I have just been playing on the shore w/o the ability (nor the time, >> to be frank) to play further with the code. > > I'm not too sure about this. It's always a question whether the > investment is worth it. > > I have a strong feeling that developing on Cython will for the > foreseeable future happen through mentoring. Trying to upfront come up > with all the potential questions one might ask about the code base, or > just writing down the 10 000 ft organization of the code, is very time > consuming. > > (Especially since Cython's code organization isn't the nicest -- but I'd > rather spend time to very gradually fix that than to document it...) PS. If you disagree to this view, and writing such an overview is a way to get you involved in Cython development, I (or somebody else) might still do it :-) Just offering an alternative take on this. > > So: When you get the time to contribute to Cython, just state what you > are interested in fixing, and the existing developers will come along > and tell you what you need to know. There's been several threads on the > mailing list that's basically "how to fix this bug" (over the summer I > mentored Kurt over Skype which worked out excellently.) > > Perhaps we should state just that somewhere visible though. > > (BTW, for anyone reading this, a nice, rather low-hanging fruit right > now is just to write testcases for pyximport and merge in the existing > patches for it.) > > > -- Dag Sverre From Chris.Barker at noaa.gov Thu Aug 27 17:57:12 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 27 Aug 2009 08:57:12 -0700 Subject: [Cython] pyximport and numpy In-Reply-To: <4A9638E4.3070709@student.matnat.uio.no> References: <4A957D8E.7070003@student.matnat.uio.no> <4A959084.5000600@student.matnat.uio.no> <200908270928.17936.binet@cern.ch> <4A9638E4.3070709@student.matnat.uio.no> Message-ID: <4A96ACD8.8040301@noaa.gov> Dag Sverre Seljebotn wrote: > (BTW, for anyone reading this, a nice, rather low-hanging fruit right > now is just to write testcases for pyximport and merge in the existing > patches for it.) A note on that: What kept pyximport working for me was the need to add the numpy headers to the compilation. I can't remember why the numpy headers are in an odd place, but I think there are good reasons. anyway, as numpy is a common use-case, we could probably just add: numpy.get_numpy_include() somewhere, but it might be better to have a more general mechanism to add such things. If there is one already, tell me about it, and I'll add it to the docs somewhere. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From shamrin at gmail.com Thu Aug 27 20:24:42 2009 From: shamrin at gmail.com (Alexey Shamrin) Date: Thu, 27 Aug 2009 22:24:42 +0400 Subject: [Cython] Extension types: allocation issues Message-ID: <28d42ea70908271124gae2f280n8ba00464821c3d1d@mail.gmail.com> Hello! I am wrapping an open source library with Cython. Hope to release some code soon... First of all, Cython works great for this! It took some time to get used to, but soon it became fun to work on a C-Python border! :-) And now the question. The C library has structures that I'm wrapping with extension types. Here's the code (simplified and without error handling): cdef extern from "library.h": ctypedef struct lib_obj: int n int allocate_obj(lib_obj **obj) void deallocate_obj(lib_obj *obj) cdef class Object(object): cdef lib_obj *obj def __cinit__(self, n): allocate_obj(&self.obj) self.obj.n = n def __dealloc__(self): if self.obj: deallocate_obj(self.obj) Everything is nice so far. But then I had to wrap two more functions from "library.h": cdef extern from "library.h": ... # these functions transfer deallocation responsibility to the caller int copy_obj(lib_obj **dst, lib_obj *src) lib_obj *load_some_obj() As you can see, I needed to create Object instances not only from scratch, but reusing existing lib_obj pointer and skipping memory allocation. Plus a nice copy_obj wrapper. After some trial-and-error the code became: cdef class Object(object): cdef lib_obj *obj def __cinit__(self, n=None, allocate=True): if allocate: allocate_obj(&self.obj) self.obj.n = n def __dealloc__(self): if self.obj: deallocate_obj(self.obj) def copy(self): cdef lib_obj *obj copy_obj(&obj, self.obj) o = Object(allocate=False) o.obj = obj return o cdef Object wrap(lib_obj *obj): Object o o = Object(allocate=False) o.obj = obj return obj def load_Object(): cdef lib_obj *obj = load_some_obj() return wrap(obj) Well, it works ;-) But can you suggest a nicer way to accomplish this? Problems: 1. Now I have to handle __cinit__ `allocate` argument in every `Object` subclass. It's easy to forget about it -- I'm writing many such subclasses. 2. Notice how I forgot to check for `n` value in __cinit__. One more thing to forget... 3. I couldn't make `wrap` function a part of `Object` definition. I tried to make it a cdef `fromobj` method decorated with @classmethod or @staticmethod, but it didn't work. Any suggestions? Or hidden problems? Thanks! From shamrin at gmail.com Thu Aug 27 20:11:22 2009 From: shamrin at gmail.com (Alexey Shamrin) Date: Thu, 27 Aug 2009 18:11:22 +0000 (UTC) Subject: [Cython] Thoughts after SciPy 09 References: <4A957D8E.7070003@student.matnat.uio.no> <4A959084.5000600@student.matnat.uio.no> Message-ID: Dag Sverre Seljebotn writes: > One other thought from SciPy 09: It emphasised to me that one of the > most important things missing in Cython, perhaps more than any single > feature, is solid, updated introductory-level documentation. > (docs.cython.org is more of a reference...) > > Luckily we hope to be able to do something about this shortly though, in > the conference proceedings. > I would like to comment on this as a Cython newcomer. I am using Cython for about a month to wrap C library. Documentation at docs.cython.org, while you say is more of a reference, is great! I like it. It's completely usable as a way to learn Cython. What I liked most is its attention to detail. And I have a feeling that it's somewhat dangerous to skip some parts of it. Maybe I'm wrong about it - not sure. Thank you all for Cython! :-) From ianlangmore at gmail.com Thu Aug 27 21:46:46 2009 From: ianlangmore at gmail.com (Ian Langmore) Date: Thu, 27 Aug 2009 15:46:46 -0400 Subject: [Cython] Function pointers in cython Message-ID: <4A96E2A6.9010409@gmail.com> I'd like to pass function pointers as in e.g. c. In python of course function pointers are automatically passed. I can't find documentation on this. I tried both c and python-like syntax and neither worked. Below is my "c-syntax" (failed) attempt. ------------------------------------------------------------------------------------------------------------ cdef float Plus (float a, float b): return a+b ############ # This block will not compile cdef use_fun_ptr(float a, float b, float (*pt2Func) (float, float)): cdef float result result = pt2Func(a,b) return result ############ def main(): cdef float use_fun_ptr(float, float, float (float, float)) # This line works print "Using Plus directly, " + str(Plus(2,5)) # print "Using use_fun_ptr, " + str( use_fun_ptr(2, 5, &Plus) ) ------------------------------------------------------------------------------------------------------------ From robertwb at math.washington.edu Thu Aug 27 22:23:59 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 27 Aug 2009 13:23:59 -0700 (PDT) Subject: [Cython] Function pointers in cython In-Reply-To: <4A96E2A6.9010409@gmail.com> References: <4A96E2A6.9010409@gmail.com> Message-ID: On Thu, 27 Aug 2009, Ian Langmore wrote: > I'd like to pass function pointers as in e.g. c. In python of course > function pointers are automatically passed. I can't find documentation > on this. I tried both c and python-like syntax and neither worked. > Below is my "c-syntax" (failed) attempt. > ------------------------------------------------------------------------------------------------------------ > cdef float Plus (float a, float b): > return a+b > > ############ > # This block will not compile > cdef use_fun_ptr(float a, float b, float (*pt2Func) (float, float)): > cdef float result > result = pt2Func(a,b) > return result > ############ Works for me, what version of Cython are you using? > def main(): > cdef float use_fun_ptr(float, float, float (float, float)) Here you're declaring a new variable, are you sure that's what you want to do? > # This line works > print "Using Plus directly, " + str(Plus(2,5)) > # > print "Using use_fun_ptr, " + str( use_fun_ptr(2, 5, &Plus) ) > ------------------------------------------------------------------------------------------------------------ > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From ianlangmore at gmail.com Thu Aug 27 22:58:30 2009 From: ianlangmore at gmail.com (Ian Langmore) Date: Thu, 27 Aug 2009 16:58:30 -0400 Subject: [Cython] Function pointers in cython In-Reply-To: References: <4A96E2A6.9010409@gmail.com> Message-ID: <4A96F376.9050308@gmail.com> Thanks Robert, I hope colloquium tea at UW still brings good food. When you say I was "declaring a new variable", I think you're referring to cdef float use_fun_ptr(float, float, float (float, float)) Right? I was declaring function in the same way that variables are declared at the top of c-code. This "function prototype" isn't necessary always, but it doesn't hurt, does it? (I'm a novice) As for the compile error, I have python 2.5 and cython 0.9.6.14-1 on my Ubuntu Intrepid machine. The following block of code gives me the following error (code, error, and setup.py all pasted below). --------------------------------- example_cython.pyx -------------------------- # This block will not compile cdef float use_fun_ptr(float a, float b, float (*pt2Func) (float, float)): cdef float result result = pt2Func(a,b) return result ---------------------------------- compile error ------------------------------------ $ python setup.py build_ext --inplace running build_ext cythoning example_cython.pyx to example_cython.c Traceback (most recent call last): File "setup.py", line 12, in ext_modules = [Extension("example_cython", ["example_cython.pyx"])] File "/usr/lib/python2.5/distutils/core.py", line 151, in setup dist.run_commands() File "/usr/lib/python2.5/distutils/dist.py", line 974, in run_commands self.run_command(cmd) File "/usr/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/usr/lib/python2.5/distutils/command/build_ext.py", line 290, in run self.build_extensions() File "/var/lib/python-support/python2.5/Cython/Distutils/build_ext.py", line 81, in build_extensions ext.sources = self.cython_sources(ext.sources, ext) File "/var/lib/python-support/python2.5/Cython/Distutils/build_ext.py", line 196, in cython_sources full_module_name=module_name) File "/var/lib/python-support/python2.5/Cython/Compiler/Main.py", line 316, in compile return context.compile(source, options, full_module_name) File "/var/lib/python-support/python2.5/Cython/Compiler/Main.py", line 213, in compile tree.process_implementation(scope, options, result) File "/var/lib/python-support/python2.5/Cython/Compiler/ModuleNode.py", line 48, in process_implementation self.analyse_declarations(env) File "/var/lib/python-support/python2.5/Cython/Compiler/ModuleNode.py", line 45, in analyse_declarations self.body.analyse_declarations(env) File "/var/lib/python-support/python2.5/Cython/Compiler/Nodes.py", line 1046, in analyse_declarations name_declarator, type = self.declarator.analyse(base_type, env, self.body is not None) File "/var/lib/python-support/python2.5/Cython/Compiler/Nodes.py", line 480, in analyse name_declarator, type = arg_node.analyse(env, nonempty = nonempty) File "/var/lib/python-support/python2.5/Cython/Compiler/Nodes.py", line 580, in analyse return self.declarator.analyse(base_type, env, nonempty = nonempty) File "/var/lib/python-support/python2.5/Cython/Compiler/Nodes.py", line 480, in analyse name_declarator, type = arg_node.analyse(env, nonempty = nonempty) File "/var/lib/python-support/python2.5/Cython/Compiler/Nodes.py", line 580, in analyse return self.declarator.analyse(base_type, env, nonempty = nonempty) File "/var/lib/python-support/python2.5/Cython/Compiler/Nodes.py", line 390, in analyse self.name = base_type.name AttributeError: CFloatType instance has no attribute 'name' ---------------------------------- setup.py --------------------------------------------- from distutils.core import setup from distutils.extension import Extension from Cython.Distutils import build_ext setup( cmdclass = {"build_ext": build_ext}, ext_modules = [Extension("example_cython", ["example_cython.pyx"])] ) ------------------------------------------------------------------------------------------------- Thanks, Ian Robert Bradshaw wrote: > On Thu, 27 Aug 2009, Ian Langmore wrote: > > >> I'd like to pass function pointers as in e.g. c. In python of course >> function pointers are automatically passed. I can't find documentation >> on this. I tried both c and python-like syntax and neither worked. >> Below is my "c-syntax" (failed) attempt. >> ------------------------------------------------------------------------------------------------------------ >> cdef float Plus (float a, float b): >> return a+b >> >> ############ >> # This block will not compile >> cdef use_fun_ptr(float a, float b, float (*pt2Func) (float, float)): >> cdef float result >> result = pt2Func(a,b) >> return result >> ############ >> > > Works for me, what version of Cython are you using? > > >> def main(): >> cdef float use_fun_ptr(float, float, float (float, float)) >> > > Here you're declaring a new variable, are you sure that's what you want to > do? > > >> # This line works >> print "Using Plus directly, " + str(Plus(2,5)) >> # >> print "Using use_fun_ptr, " + str( use_fun_ptr(2, 5, &Plus) ) >> ------------------------------------------------------------------------------------------------------------ >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> >> > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- ****************************************************************** Ian Langmore Postdoctoral Researcher, Columbia Univ. Applied Mathematics 415-272-6321 www.columbia.edu/~il2176 ****************************************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090827/596e5246/attachment.htm From robertwb at math.washington.edu Thu Aug 27 23:21:59 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 27 Aug 2009 14:21:59 -0700 (PDT) Subject: [Cython] Function pointers in cython In-Reply-To: <4A96F376.9050308@gmail.com> References: <4A96E2A6.9010409@gmail.com> <4A96F376.9050308@gmail.com> Message-ID: On Thu, 27 Aug 2009, Ian Langmore wrote: > Thanks Robert, I hope colloquium tea at UW still brings good food. Most of the time :) > When you say I was "declaring a new variable", I think you're referring to > > cdef float use_fun_ptr(float, float, float (float, float)) > > Right? I was declaring function in the same way that variables are declared > at the top of c-code. This "function prototype" isn't necessary always, but > it doesn't hurt, does it? (I'm a novice) When you do that inside of a function (inside your main() here) you're declaring a new local variable. In terms of top-level declarations, you should never need prototypes in Cython. > As for the compile error, I have python 2.5 and cython 0.9.6.14-1 on my > Ubuntu Intrepid machine. Probably a bug in an old version of Cython that has since been fixed, I recommend upgrading. - Robert From ianlangmore at gmail.com Thu Aug 27 23:56:17 2009 From: ianlangmore at gmail.com (Ian Langmore) Date: Thu, 27 Aug 2009 17:56:17 -0400 Subject: [Cython] Function pointers in cython In-Reply-To: References: <4A96E2A6.9010409@gmail.com> <4A96F376.9050308@gmail.com> Message-ID: <4A970101.7050104@gmail.com> Thanks! As it turns out, installing Cython-0.11.2 was the answer. I thought I had the newest version, since Ubuntu's "Synaptic Package Manager" said I did. I'll be more careful in the future. -Ian Robert Bradshaw wrote: > On Thu, 27 Aug 2009, Ian Langmore wrote: > > >> Thanks Robert, I hope colloquium tea at UW still brings good food. >> > > Most of the time :) > > >> When you say I was "declaring a new variable", I think you're referring to >> >> cdef float use_fun_ptr(float, float, float (float, float)) >> >> Right? I was declaring function in the same way that variables are declared >> at the top of c-code. This "function prototype" isn't necessary always, but >> it doesn't hurt, does it? (I'm a novice) >> > > When you do that inside of a function (inside your main() here) you're > declaring a new local variable. > > In terms of top-level declarations, you should never need prototypes in > Cython. > > >> As for the compile error, I have python 2.5 and cython 0.9.6.14-1 on my >> Ubuntu Intrepid machine. >> > > Probably a bug in an old version of Cython that has since been fixed, I > recommend upgrading. > > - Robert > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- ****************************************************************** Ian Langmore Postdoctoral Researcher, Columbia Univ. Applied Mathematics 415-272-6321 www.columbia.edu/~il2176 ****************************************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090827/0fdfd0e3/attachment.htm From e-huss at netmeridian.com Fri Aug 28 01:38:08 2009 From: e-huss at netmeridian.com (Eric Huss) Date: Thu, 27 Aug 2009 16:38:08 -0700 (PDT) Subject: [Cython] setuptools and Cython Message-ID: I've run into a problem when using setuptools with Cython. If you don't have Pyrex installed, then you cannot get setuptools to compile the .pyx file to .c. I see there was a discussion on the distutils list here: http://mail.python.org/pipermail/distutils-sig/2007-September/008204.html But I don't see much of an outcome. What other options are there? -Eric From grant at osafoundation.org Fri Aug 28 04:06:22 2009 From: grant at osafoundation.org (Grant Baillie) Date: Thu, 27 Aug 2009 19:06:22 -0700 Subject: [Cython] cdef public enum values not exported Message-ID: Hi, cython-dev I think I found a regression in Cython 0.11. Basically, I was attempting to follow to export values from a public enum to python. However, after building the following .pyx with Cython 0.11.2: cdef public enum FOO: BAR = 3 I found that BAR isn't exported from the resulting extension module. According to that thread, this worked back in 0.9.6.14. I've attached my attempt at a patch, as well as an extra test case that reproduces the problem. (I admit I'm not exactly an expert on what ParseTreeTransforms.py is doing :o). --Grant -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: public_enum.patch.txt Url: http://codespeak.net/pipermail/cython-dev/attachments/20090827/db901f38/attachment.txt From lev at columbia.edu Fri Aug 28 04:25:59 2009 From: lev at columbia.edu (Lev Givon) Date: Thu, 27 Aug 2009 22:25:59 -0400 Subject: [Cython] setuptools and Cython In-Reply-To: References: Message-ID: <20090828022559.GB8166@localhost.ee.columbia.edu> Received from Eric Huss on Thu, Aug 27, 2009 at 07:38:08PM EDT: > I've run into a problem when using setuptools with Cython. If you don't > have Pyrex installed, then you cannot get setuptools to compile the .pyx > file to .c. I see there was a discussion on the distutils list here: > > http://mail.python.org/pipermail/distutils-sig/2007-September/008204.html > > But I don't see much of an outcome. What other options are there? > > -Eric I haven't tried this, but it appears to have been inspired by the above discussion: http://pypi.python.org/pypi/setuptools_cython/ L.G. From stefan_ml at behnel.de Fri Aug 28 06:44:49 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 28 Aug 2009 06:44:49 +0200 Subject: [Cython] setuptools and Cython In-Reply-To: References: Message-ID: <4A9760C1.4030005@behnel.de> Eric Huss wrote: > I've run into a problem when using setuptools with Cython. If you don't > have Pyrex installed, then you cannot get setuptools to compile the .pyx > file to .c. I see there was a discussion on the distutils list here: > > http://mail.python.org/pipermail/distutils-sig/2007-September/008204.html > > But I don't see much of an outcome. What other options are there? http://mail.python.org/pipermail/python-list/2009-August/722917.html Stefan From robertwb at math.washington.edu Fri Aug 28 07:32:41 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 27 Aug 2009 22:32:41 -0700 Subject: [Cython] cdef public enum values not exported In-Reply-To: References: Message-ID: <7A5031E8-9234-4306-86F1-F32347EDF471@math.washington.edu> On Aug 27, 2009, at 7:06 PM, Grant Baillie wrote: > Hi, cython-dev > > I think I found a regression in Cython 0.11. Basically, I was > attempting to follow > > > > to export values from a public enum to python. However, after > building the following .pyx with Cython 0.11.2: > > cdef public enum FOO: > BAR = 3 > > I found that BAR isn't exported from the resulting extension > module. According to that thread, this worked back in 0.9.6.14. > > I've attached my attempt at a patch, as well as an extra test case > that reproduces the problem. That did the trick Thanks! Pushed to cython-devel. > (I admit I'm not exactly an expert on what ParseTreeTransforms.py > is doing :o). That part of the transform is pruning the unneeded declaration code (to speed up walking over the tree), but as you pointed out we did need the public enums later on. - Robert From robertwb at math.washington.edu Fri Aug 28 07:41:55 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 27 Aug 2009 22:41:55 -0700 Subject: [Cython] Extension types: allocation issues In-Reply-To: <28d42ea70908271124gae2f280n8ba00464821c3d1d@mail.gmail.com> References: <28d42ea70908271124gae2f280n8ba00464821c3d1d@mail.gmail.com> Message-ID: On Aug 27, 2009, at 11:24 AM, Alexey Shamrin wrote: > Hello! > > I am wrapping an open source library with Cython. Hope to release some > code soon... > > First of all, Cython works great for this! It took some time to get > used to, but soon it became fun to work on a C-Python border! :-) > > And now the question. > > The C library has structures that I'm wrapping with extension types. > Here's the code (simplified and without error handling): > > cdef extern from "library.h": > ctypedef struct lib_obj: > int n > > int allocate_obj(lib_obj **obj) > void deallocate_obj(lib_obj *obj) > > cdef class Object(object): > cdef lib_obj *obj > > def __cinit__(self, n): > allocate_obj(&self.obj) > self.obj.n = n > > def __dealloc__(self): > if self.obj: > deallocate_obj(self.obj) > > Everything is nice so far. But then I had to wrap two more functions > from "library.h": > > cdef extern from "library.h": > ... > # these functions transfer deallocation responsibility to the > caller > int copy_obj(lib_obj **dst, lib_obj *src) > lib_obj *load_some_obj() > > As you can see, I needed to create Object instances not only from > scratch, but reusing existing lib_obj pointer and skipping memory > allocation. Plus a nice copy_obj wrapper. After some trial-and-error > the code became: > > cdef class Object(object): > cdef lib_obj *obj > > def __cinit__(self, n=None, allocate=True): > if allocate: > allocate_obj(&self.obj) > self.obj.n = n > > def __dealloc__(self): > if self.obj: > deallocate_obj(self.obj) > > def copy(self): > cdef lib_obj *obj > copy_obj(&obj, self.obj) > o = Object(allocate=False) > o.obj = obj > return o > > cdef Object wrap(lib_obj *obj): > Object o > o = Object(allocate=False) > o.obj = obj > return obj > > def load_Object(): > cdef lib_obj *obj = load_some_obj() > return wrap(obj) > > Well, it works ;-) But can you suggest a nicer way to accomplish > this? Problems: > > 1. Now I have to handle __cinit__ `allocate` argument in every > `Object` subclass. It's easy to forget about it -- I'm writing many > such subclasses. I don't see how you'd get around this--somewhere you have to give that information to the __cinit__ method. Does each subclass have to do something extra in __cinit__? It could be wasteful, but the logic might be simpler to either always assume it's allocated and deallocate in copy and wrap. I don't know what you're doing with these things, but if you never wrap None you could use n=None as a sentinel to not allocate, only allocating if you're actually wrapping something. It's really hard to tell without understanding how it's going to be used. > 2. Notice how I forgot to check for `n` value in __cinit__. One more > thing to forget... > 3. I couldn't make `wrap` function a part of `Object` definition. I > tried to make it a cdef `fromobj` method decorated with @classmethod > or @staticmethod, but it didn't work. cdef methods can't (yet) have decorators like this, but a module- level function should work fine. - Robert From mziulu at gmail.com Fri Aug 28 09:45:55 2009 From: mziulu at gmail.com (Mattia Ziulu) Date: Fri, 28 Aug 2009 09:45:55 +0200 Subject: [Cython] Expose a *float element to Python namespace Message-ID: <8833ea530908280045m62106dc1n943c9919fd94c3ba@mail.gmail.com> Hello everybody! I present you this little problem that is slowing down a project of mine (I have actually already found a way to bypass it, but it is kinda hack-ish). So, I'm wrapping a C library using Cython, in order to expose its functionalities to Python via a simple from [module] import *, and until now everything is looking pretty good. The underlying library is a simple matrix library, declared as typedef struct MatrixFloat{ int rows; int columns; float *data; } MatrixFloat; As you probably can see, the main headache is the actual data access. In C, if I have a MatrixFloat instance named foo, I can access its data via conventional offset-based indexing, so e.g. foo.data[0][0] would actually be foo.data[0 * columns + 0] and so on and so forth. Now, let's say that I don't want this for my Python interface, but I'd like instead to do something like the conventional foo.data[0][0] (because of the goodies that Python provides, such as slicing and row assignment and the likes). After some thought and research (I am new to both Cython and Python) I found out that the common (and perhaps only?) way to expose the data contained in the wrapped struct is to use properties, and so I did something like this: cdef class MatrixFloat: cdef cMatrixFloat.MatrixFloat *ptr def __init__( self ): self.ptr = cMatrixFloat.MatrixFloat_new() property rows: def __get__(self): return self.ptr.rows def __set__(self, n): self.ptr.rows = n property columns: def __get__(self): return self.ptr.columns def __set__(self, n): self.ptr.columns = n property data: def __get__(self): a = [] for i in xrange( self.rows*self.columns ): a.append( self.ptr.data[i] ) data = [ [ None for _ in range(self.columns) ] for _ in range(self.rows) ] for i in xrange(self.columns): if i < self.rows: data[i] = a[ i*self.columns : i*self.columns+self.columns] return data [... plus other methods like MatrixFloat_new() used above, MatrixFloat_init() to initialize the fields and MatrixFloat_randPopulate() to populate the matrix with random values ...] This is good, and provides the functionality what I want, except it's *really* slow (and it should be, since there are, after all, 3 or 4 for loops). I am somewhat limited in my choices. For example, I can't simply fill a list / list of lists / array during the initialization phase because usually the matrices are created using the methods outlined in the snippet above. It doesn't help also that the methods available for the properties (namely __get__ and __set__ ) don't accept more than one parameter. I partially solved this situation using the methods exposed by my library, like MatrixFloat_setElem() and _getElem(), but they do not provide anything fancy like slicing, row assignement or even sheer performance, and also foo.data[0][0] = n (which I think I can't do using properties) is easier than MatrixFloat_setElem( foo, 0, 0, n). I was wondering if anyone had to deal with a case like this one before and if so if he/she had found a simpler and more elegant solution! ?Greenspun's Tenth Rule of Programming: any sufficiently complicated C or Fortran program contains an ad hoc informally-specified bug-ridden slow implementation of half of Common Lisp.? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090828/d4ea114f/attachment.htm From dagss at student.matnat.uio.no Fri Aug 28 15:29:00 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 28 Aug 2009 15:29:00 +0200 Subject: [Cython] Expose a *float element to Python namespace Message-ID: <3334318183.1248402@smtp.netcom.no> Two answers: 1) In about half a year, there will likely be a released version of Cython with built in capability to do this kind of wrapping. 2) NumPy NumPy NumPy! Doing this without using NumPy is a waste of time IMO. Use the PyArray_New... function in the C api to create array backed by your C struct. (I remember posting a howto to Jon Olav Vik here on the list around June). Dag Sverre Seljebotn -----Original Message----- From: Mattia Ziulu Date: Friday, Aug 28, 2009 9:55 am Subject: [Cython] Expose a *float element to Python namespace To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net Hello everybody! >I present you this little problem that is slowing down a project of mine >(I have actually already found a way to bypass it, but it is kinda hack-ish). >So, I'm wrapping a C library using Cython, in order to expose its > functionalities to Python via a simple from [module] import *, and until now >everything is looking pretty good. The underlying library is a simple matrix >library, declared as > >typedef struct MatrixFloat{ >??? int rows; >??? int columns; >??? float *data; >} MatrixFloat; > >As you probably can see, the main headache is the actual data access. In C, >if I have a MatrixFloat instance named foo, I can access its data via > conventional offset-based indexing, so e.g. foo.data[0][0] would actually >be foo.data[0 * columns + 0] and so on and so forth. >Now, let's say that I don't want this for my Python interface, but I'd like > instead to do something like the conventional foo.data[0][0] (because of the >goodies that Python provides, such as slicing and row assignment and the likes). >After some thought and research (I am new to both Cython and Python) I found out > that the common (and perhaps only?) way to expose the data contained in the >wrapped struct is to use properties, and so I did something like this: > >cdef class MatrixFloat: >??? cdef cMatrixFloat.MatrixFloat *ptr >??? >??? def __init__( self ): >??????? self.ptr = cMatrixFloat.MatrixFloat_new() >??????? >??? property rows: >??????? def __get__(self): >??????????? return self.ptr.rows > >??????? def __set__(self, n): >???????????? self.ptr.rows = n > >??? property columns: >???????? def __get__(self): >??????????? return self.ptr.columns > >??????? def __set__(self, n): >???????????? self.ptr.columns = n > >??? property data: >??????? def __get__(self): >???????????? a = [] >???????????? for i in xrange( self.rows*self.columns ): >???????????????? a.append( self.ptr.data[i] ) >??????????? data = [ [ None for _ in range(self.columns) ] for _ in range(self.rows) ] >??????????? for i?? in xrange(self.columns): >??????????????? if i < self.rows: >???????????????????? data[i] = a[ i*self.columns : i*self.columns+self.columns] >???????????? return data > >[... plus other methods like MatrixFloat_new() used above, MatrixFloat_init() to initialize >???? the fields and MatrixFloat_randPopulate() to populate the matrix with random >???? values ...] > >This is good, and provides the functionality what I want, except it's *really* slow >(and it should be, since there are, after all, 3 or 4 for loops). > I am somewhat limited in my choices. For example, I can't simply fill a list / list >of lists / array during the initialization phase because usually the matrices are >created using the methods outlined in the snippet above. It doesn't help also that > the methods available for the properties (namely __get__ and __set__ ) don't accept >more than one parameter. >I partially solved this situation using the methods exposed by my library, like >MatrixFloat_setElem() and _getElem(), but they do not provide anything fancy like > slicing, row assignement or even sheer performance, and also foo.data[0][0] = n (which >I think I can't do using properties) is easier than MatrixFloat_setElem( foo, 0, 0, n). >I was wondering if anyone had to deal with a case like this one before and if so if > he/she had found a simpler and more elegant solution! > > - Greenspun's Tenth Rule of Programming: any sufficiently complicated C or Fortran program contains an ad hoc informally-specified bug-ridden slow implementation of half of Common Lisp. - > From mziulu at gmail.com Fri Aug 28 15:49:48 2009 From: mziulu at gmail.com (Mattia Ziulu) Date: Fri, 28 Aug 2009 15:49:48 +0200 Subject: [Cython] Expose a *float element to Python namespace In-Reply-To: <3334318183.1248402@smtp.netcom.no> References: <3334318183.1248402@smtp.netcom.no> Message-ID: <8833ea530908280649n562234e6s6dfb013ba8251117@mail.gmail.com> On Fri, Aug 28, 2009 at 3:29 PM, Dag Sverre Seljebotn < dagss at student.matnat.uio.no> wrote: > Two answers: > 1) In about half a year, there will likely be a released version of Cython > with built in capability to do this kind of wrapping. > This is awesome! > > 2) NumPy NumPy NumPy! Doing this without using NumPy is a waste of time > IMO. Use the PyArray_New... function in the C api to create array backed by > your C struct. (I remember posting a howto to Jon Olav Vik here on the list > around June). > > No can do. Or better yet, yes could do, but the whole point here is that my MatrixFloat gimmick is all a big test for something a lot more complicated and generalized, where Numpy wouldn't be useful at all. Let's just say that I'm using this "project" to wrap my mind around this whole (and for me new) world of wrapping and exposing and whatnot, and being a matrix class something pretty easy to set up (in a fairly basic way anyway) I choose to implement this. I've looked at numpy for some inspiration though, both docs and source code, but there are not a lot of useful things in it that could be useful in this context too, I'm afraid. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090828/a3d8514b/attachment.htm From dagss at student.matnat.uio.no Fri Aug 28 16:09:00 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 28 Aug 2009 16:09:00 +0200 Subject: [Cython] Expose a *float element to Python namespace Message-ID: <3334320574.1343798@smtp.netcom.no> Ah I see. Well, Cython's coming support won't be much help either then...(if you want to, say, wrap a sparse matrix lib or similar). Anyway: - Don't do arr[i][j], do arr[i,j]. - Do override __getitem__! (if necesarry in a new object returned from a proprty, but in this case I don't see any need for properties) - Slices etc are no problem, you can check the index parameter to __getitem__ for slice objects. Yes, you must manually implement slices etc., but if NumPy doesn't cover your case this follows anyway! -- it's not like Python or Cython would be able to guess what a slice should represent or how it should be implemented in your more complex problems. Dag Sverre Seljebotn -----Original Message----- From: Mattia Ziulu Date: Friday, Aug 28, 2009 3:50 pm Subject: Re: [Cython] Expose a *float element to Python namespace To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net >On Fri, Aug 28, 2009 at 3:29 PM, Dag Sverre Seljebotn wrote: > Two answers: > ?1) In about half a year, there will likely be a released version of Cython with built in capability to do this kind of wrapping. > > >This is awesome!? > > ?2) NumPy NumPy NumPy! Doing this without using NumPy is a waste of time IMO. Use the PyArray_New... function in the C api to create array backed by your C struct. (I remember posting a howto to Jon Olav Vik here on the list around June). > > >No can do. Or better yet, yes could do, but the whole point here is that my MatrixFloat gimmick is all a big test for something a lot more complicated and generalized, where Numpy wouldn't be useful at all. Let's just say that I'm using this 'project' to wrap my mind around this whole (and for me new) world of wrapping and exposing and whatnot, and being a matrix class something pretty easy to set up (in a fairly basic way anyway) I choose to implement this. > I've looked at numpy for some inspiration though, both docs and source code, but there are not a lot of useful things in it that could be useful in this context too, I'm afraid. > From mziulu at gmail.com Fri Aug 28 17:29:52 2009 From: mziulu at gmail.com (Mattia Ziulu) Date: Fri, 28 Aug 2009 17:29:52 +0200 Subject: [Cython] Expose a *float element to Python namespace In-Reply-To: <3334320574.1343798@smtp.netcom.no> References: <3334320574.1343798@smtp.netcom.no> Message-ID: <8833ea530908280829o4190ffa4lc977cb6dccc8d38d@mail.gmail.com> Incidentally, this doesn't work: def getdata(self, x, y): return self.ptr.data[x * self.ptr.columns + y] def setdata(self, x, y, n): self.ptr.data[x * self.ptr.columns + y] = n data = property( getdata, setdata ) No matter how I access foo.data it just doesn't pass the parameters. I guess it's intended behavior. Anyway, thanks Dag, I'll see if I can hack together something, but probably I'll leave it as it is, which is gonna be fine enough for now! Tusen takk! ?Greenspun's Tenth Rule of Programming: any sufficiently complicated C or Fortran program contains an ad hoc informally-specified bug-ridden slow implementation of half of Common Lisp.? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090828/99bbbdc1/attachment.htm From robertwb at math.washington.edu Fri Aug 28 17:43:08 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 28 Aug 2009 08:43:08 -0700 Subject: [Cython] Expose a *float element to Python namespace In-Reply-To: <8833ea530908280045m62106dc1n943c9919fd94c3ba@mail.gmail.com> References: <8833ea530908280045m62106dc1n943c9919fd94c3ba@mail.gmail.com> Message-ID: <53B91074-F7A5-4DEF-882D-7FD804631F8F@math.washington.edu> On Aug 28, 2009, at 12:45 AM, Mattia Ziulu wrote: > Hello everybody! > I present you this little problem that is slowing down a project of > mine > (I have actually already found a way to bypass it, but it is kinda > hack-ish). > So, I'm wrapping a C library using Cython, in order to expose its > functionalities to Python via a simple from [module] import *, and > until now > everything is looking pretty good. The underlying library is a > simple matrix > library, declared as > > typedef struct MatrixFloat{ > int rows; > int columns; > float *data; > } MatrixFloat; > > As you probably can see, the main headache is the actual data > access. In C, > if I have a MatrixFloat instance named foo, I can access its data via > conventional offset-based indexing, so e.g. foo.data[0][0] would > actually > be foo.data[0 * columns + 0] and so on and so forth. > Now, let's say that I don't want this for my Python interface, but > I'd like > instead to do something like the conventional foo.data[0][0] > (because of the > goodies that Python provides, such as slicing and row assignment > and the likes). > After some thought and research (I am new to both Cython and > Python) I found out > that the common (and perhaps only?) way to expose the data > contained in the > wrapped struct is to use properties, and so I did something like this: > > cdef class MatrixFloat: > cdef cMatrixFloat.MatrixFloat *ptr > > def __init__( self ): > self.ptr = cMatrixFloat.MatrixFloat_new() > > property rows: > def __get__(self): > return self.ptr.rows > > def __set__(self, n): > self.ptr.rows = n > > property columns: > def __get__(self): > return self.ptr.columns > > def __set__(self, n): > self.ptr.columns = n > > property data: > def __get__(self): > a = [] > for i in xrange( self.rows*self.columns ): > a.append( self.ptr.data[i] ) > data = [ [ None for _ in range(self.columns) ] for _ in > range(self.rows) ] > for i in xrange(self.columns): > if i < self.rows: > data[i] = a[ i*self.columns : i*self.columns > +self.columns] > return data First, implement __getitem__ and __setitem__, so you can do foo[1,5] = 17. Second, how about property data: def __get__(self): cdef int i,j return [ [ data[i+self.columns*j ] for i in range (self.columns) ] for j in range(self.rows) ] Which should be faster at least (though you're still doing a lot of float -> Python object conversions. > > > [... plus other methods like MatrixFloat_new() used above, > MatrixFloat_init() to initialize > the fields and MatrixFloat_randPopulate() to populate the > matrix with random > values ...] > > This is good, and provides the functionality what I want, except > it's *really* slow > (and it should be, since there are, after all, 3 or 4 for loops). > I am somewhat limited in my choices. For example, I can't simply > fill a list / list > of lists / array during the initialization phase because usually > the matrices are > created using the methods outlined in the snippet above. It doesn't > help also that > the methods available for the properties (namely __get__ and > __set__ ) don't accept > more than one parameter. > I partially solved this situation using the methods exposed by my > library, like > MatrixFloat_setElem() and _getElem(), but they do not provide > anything fancy like > slicing, row assignement or even sheer performance, and also > foo.data[0][0] = n (which > I think I can't do using properties) is easier than > MatrixFloat_setElem( foo, 0, 0, n). > I was wondering if anyone had to deal with a case like this one > before and if so if > he/she had found a simpler and more elegant solution! > > ?Greenspun's Tenth Rule of Programming: any sufficiently > complicated C or Fortran program contains an ad hoc informally- > specified bug-ridden slow implementation of half of Common Lisp.? > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From dagss at student.matnat.uio.no Fri Aug 28 17:59:00 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 28 Aug 2009 17:59:00 +0200 Subject: [Cython] Expose a *float element to Python namespace Message-ID: <3334327167.1434048@smtp.netcom.no> Again I don't see what you are attempting. Python has no concept of "passing arguments to a property", there's no reason why this should work, and I don't know what it would mean if it did? Why is it that you cannot simply implement __getitem__? If you want "x.y[z]", do cdef class YAttr: def __getitem__(self, z): ... cdef class XObject: cdef public YAttr data Dag Sverre Seljebotn -----Original Message----- From: Mattia Ziulu Date: Friday, Aug 28, 2009 5:30 pm Subject: Re: [Cython] Expose a *float element to Python namespace To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net Incidentally, this doesn't work: > >???? def getdata(self, x, y): >???????? return self.ptr.data[x * self.ptr.columns + y] >???? def setdata(self, x, y, n): >???????? self.ptr.data[x * self.ptr.columns + y] = n >???? data = property( getdata, setdata ) > >No matter how I access foo.data it just doesn't pass the parameters. I guess it's intended behavior. >Anyway, thanks Dag, I'll see if I can hack together something, but probably I'll leave it as it is, which is gonna be fine enough for now! Tusen takk! > > - Greenspun's Tenth Rule of Programming: any sufficiently complicated C or Fortran program contains an ad hoc informally-specified bug-ridden slow implementation of half of Common Lisp. - > > > From mziulu at gmail.com Fri Aug 28 18:02:08 2009 From: mziulu at gmail.com (Mattia Ziulu) Date: Fri, 28 Aug 2009 18:02:08 +0200 Subject: [Cython] Expose a *float element to Python namespace In-Reply-To: <53B91074-F7A5-4DEF-882D-7FD804631F8F@math.washington.edu> References: <8833ea530908280045m62106dc1n943c9919fd94c3ba@mail.gmail.com> <53B91074-F7A5-4DEF-882D-7FD804631F8F@math.washington.edu> Message-ID: <8833ea530908280902p4270450aq558b4f0354730f12@mail.gmail.com> Mmh, if I call foo[0,0], passing the indexes like a tuple that is, if I got it right, it works correctly. At this point though a properties is redundant, isn't it, since I already access the internal data via __getitem__, right? It could be useful for the "matrix-like" representation, though, which I'm not getting using my implementation __getitem__...gonna play with it a little bit more. On Fri, Aug 28, 2009 at 5:43 PM, Robert Bradshaw < robertwb at math.washington.edu> wrote: > > First, implement __getitem__ and __setitem__, so you can do foo[1,5] > = 17. Second, how about > > property data: > def __get__(self): > cdef int i,j > return [ [ data[i+self.columns*j ] for i in range > (self.columns) ] for j in range(self.rows) ] > > Which should be faster at least (though you're still doing a lot of > float -> Python object conversions. > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090828/5202120d/attachment.htm From dagss at student.matnat.uio.no Fri Aug 28 18:01:00 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: 28 Aug 2009 18:01:00 +0200 Subject: [Cython] Expose a *float element to Python namespace Message-ID: <3334327315.1448830@smtp.netcom.no> Yes, the properties likely are redundant. Dag Sverre Seljebotn -----Original Message----- From: Mattia Ziulu Date: Friday, Aug 28, 2009 6:03 pm Subject: Re: [Cython] Expose a *float element to Python namespace To: cython-dev at codespeak.netReply-To: cython-dev at codespeak.net Mmh, if I call foo[0,0], passing the indexes like a tuple that is, if I got it right, it works correctly. >At this point though a properties is redundant, isn't it, since I already access the internal data via __getitem__, right? It could be useful for the 'matrix-like' representation, though, which I'm not getting using my implementation __getitem__...gonna play with it a little bit more. > >On Fri, Aug 28, 2009 at 5:43 PM, Robert Bradshaw wrote: > > > >First, implement __getitem__ and __setitem__, so you can do foo[1,5] >= 17. Second, how about > > ? ? property data: > ? ? ? ? def __get__(self): > ? ? ? ? ? ? cdef int i,j > ? ? ? ? ? ? return [ [ data[i+self.columns*j ] for i in range >(self.columns) ] for j in range(self.rows) ] > > Which should be faster at least (though you're still doing a lot of > float -> Python object conversions. > >_______________________________________________ > Cython-dev mailing list >Cython-dev at codespeak.net >http://codespeak.net/mailman/listinfo/cython-dev > > From Chris.Barker at noaa.gov Fri Aug 28 18:30:34 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Fri, 28 Aug 2009 09:30:34 -0700 Subject: [Cython] Expose a *float element to Python namespace In-Reply-To: <8833ea530908280649n562234e6s6dfb013ba8251117@mail.gmail.com> References: <3334318183.1248402@smtp.netcom.no> <8833ea530908280649n562234e6s6dfb013ba8251117@mail.gmail.com> Message-ID: <4A98062A.2030003@noaa.gov> Mattia Ziulu wrote: > > wrote: > 2) NumPy NumPy NumPy! Doing this without using NumPy is a waste of > time IMO. > No can do. Or better yet, yes could do, but the whole point here is that > my MatrixFloat gimmick is all a big test for something a lot more > complicated and generalized, where Numpy wouldn't be useful at all. I wouldn't be so sure. Even if you can't directly map your data structure to a numpy array, you need to be able return parts of your data structure as Python types of some sort. If they involve a bunch of numbers then numpy arrays are natural. If you really don't want to use numpy, you could use the std lib array.array objects, but numpy arrays give you so much more. Let's say you are wrapping a sparse matrix object, as DAG suggested. You're right, you can't directly wrap it with a numy array, but you may want: MySparseArray[i, :] or MySparseArray[:, j] to return a numpy array for the row or column. Unless you want a sparse row or column, but I think you'll get my point. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From kwmsmith at gmail.com Fri Aug 28 19:29:13 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Fri, 28 Aug 2009 12:29:13 -0500 Subject: [Cython] Giving fwrap a home In-Reply-To: References: <85e81ba30908251631n78de4e2dk849826cf5a86593d@mail.gmail.com> <9E2BDBD3-4D7C-45C9-8F43-A3F647CF14D1@math.washington.edu> Message-ID: On Wed, Aug 26, 2009 at 12:21 PM, Robert Bradshaw wrote: > On Wed, 26 Aug 2009, Kurt Smith wrote: > >> On Wed, Aug 26, 2009 at 12:18 AM, Robert >> Bradshaw wrote: >>> >>> On Aug 25, 2009, at 6:37 PM, Kurt Smith wrote: >>> >>>> On Tue, Aug 25, 2009 at 6:31 PM, William Stein >>>> wrote: >>>>> >>>>> On Tue, Aug 25, 2009 at 12:18 PM, Kurt Smith >>>>> wrote: >>>>>> >>>>>> I think its best that fwrap be its own package, distributed >>>>>> separately >>>>>> from Cython. ?That was the vibe I got from those at the SciPy 2009 >>>>>> conference and from the recent thread on Fwrap's licensing. ?The >>>>>> added >>>>>> benefit is that fwrap won't weigh Cython down w.r.t. licensing >>>>>> issues >>>>>> or be an impediment to Cython's acceptance into the Python std. lib. >>>>>> >>>>>> Presuming that everyone here agrees with the above (or doesn't >>>>>> care), >>>>>> the remaining question is where to host it. ?Since fwrap is still >>>>>> closely linked to Cython, I think a natural spot for the mercurial >>>>>> repo would be on Cython's servers. ?No strong feelings here -- I'm >>>>>> just as happy putting it on bitbucket. ?Fwrap would *not* clutter up >>>>>> Cython's trac, or Cython's wiki. ?These would be elsewhere. >>>>>> >>>>> >>>>> As the own of Cython's servers, I hereby certainly offer you >>>>> hosting space. >>>> >>>> Thanks! ?I'm evaluating bitbucket & googlecode to see what they offer >>>> -- apparently googlecode has mercurial support, so that is a big plus. >>>> ?I'll decide soon & let you know. >>> >>> I wouldn't consider it "clutter" to have an fwrap section of the >>> wiki, nor host another porject on the cython trac server. >> >> Well, this changes things significantly -- thanks for the offer! >> >> Since I'd rather work with you folks than some external organization, >> I'll keep fwrap on Cython's servers, wiki & trac. ?Thanks for the >> generosity! >> >> I'll start setting things up over the weekend. ?What's the next step(s)? > > If you point me to your repository, I'll put it up next to ours. I'll also > set up a trac project for you, and for the wiki I think > wiki.cython.org/fwrap should be good enough. I've failed to successfully extract fwrap from hg.cython.org/gsoc-kurt using 'hg convert'. Do you (or anyone else) have experience extracting a directory from a hg repo into its own repo? We could always just copy the directory (gsoc-kurt/Tools/fwrap) and make a new repo with no history, I guess. Kurt From dagss at student.matnat.uio.no Fri Aug 28 20:04:29 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 28 Aug 2009 20:04:29 +0200 Subject: [Cython] Giving fwrap a home In-Reply-To: References: <85e81ba30908251631n78de4e2dk849826cf5a86593d@mail.gmail.com> <9E2BDBD3-4D7C-45C9-8F43-A3F647CF14D1@math.washington.edu> Message-ID: <4A981C2D.5000006@student.matnat.uio.no> Kurt Smith wrote: > On Wed, Aug 26, 2009 at 12:21 PM, Robert > Bradshaw wrote: > >> On Wed, 26 Aug 2009, Kurt Smith wrote: >> >> >>> On Wed, Aug 26, 2009 at 12:18 AM, Robert >>> Bradshaw wrote: >>> >>>> On Aug 25, 2009, at 6:37 PM, Kurt Smith wrote: >>>> >>>> >>>>> On Tue, Aug 25, 2009 at 6:31 PM, William Stein >>>>> wrote: >>>>> >>>>>> On Tue, Aug 25, 2009 at 12:18 PM, Kurt Smith >>>>>> wrote: >>>>>> >>>>>>> I think its best that fwrap be its own package, distributed >>>>>>> separately >>>>>>> from Cython. That was the vibe I got from those at the SciPy 2009 >>>>>>> conference and from the recent thread on Fwrap's licensing. The >>>>>>> added >>>>>>> benefit is that fwrap won't weigh Cython down w.r.t. licensing >>>>>>> issues >>>>>>> or be an impediment to Cython's acceptance into the Python std. lib. >>>>>>> >>>>>>> Presuming that everyone here agrees with the above (or doesn't >>>>>>> care), >>>>>>> the remaining question is where to host it. Since fwrap is still >>>>>>> closely linked to Cython, I think a natural spot for the mercurial >>>>>>> repo would be on Cython's servers. No strong feelings here -- I'm >>>>>>> just as happy putting it on bitbucket. Fwrap would *not* clutter up >>>>>>> Cython's trac, or Cython's wiki. These would be elsewhere. >>>>>>> >>>>>>> >>>>>> As the own of Cython's servers, I hereby certainly offer you >>>>>> hosting space. >>>>>> >>>>> Thanks! I'm evaluating bitbucket & googlecode to see what they offer >>>>> -- apparently googlecode has mercurial support, so that is a big plus. >>>>> I'll decide soon & let you know. >>>>> >>>> I wouldn't consider it "clutter" to have an fwrap section of the >>>> wiki, nor host another porject on the cython trac server. >>>> >>> Well, this changes things significantly -- thanks for the offer! >>> >>> Since I'd rather work with you folks than some external organization, >>> I'll keep fwrap on Cython's servers, wiki & trac. Thanks for the >>> generosity! >>> >>> I'll start setting things up over the weekend. What's the next step(s)? >>> >> If you point me to your repository, I'll put it up next to ours. I'll also >> set up a trac project for you, and for the wiki I think >> wiki.cython.org/fwrap should be good enough. >> > > I've failed to successfully extract fwrap from hg.cython.org/gsoc-kurt > using 'hg convert'. Do you (or anyone else) have experience > extracting a directory from a hg repo into its own repo? We could > always just copy the directory (gsoc-kurt/Tools/fwrap) and make a new > repo with no history, I guess. > Would it really matter if you loose history? That's what I'd go for, then just keep gsoc-kurt around for a year just in case. Failing everything else, it should be possible to export every relevant changeset to file and reimport them again into a new repository with a smallish shell script. You likely need to be doing stuff like that anyway in order to submit your code to Google, right? (At least if you want to submit changes to the existing files.) Dag Sverre From greg.ewing at canterbury.ac.nz Sat Aug 29 01:26:52 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 29 Aug 2009 11:26:52 +1200 Subject: [Cython] Expose a *float element to Python namespace In-Reply-To: <8833ea530908280649n562234e6s6dfb013ba8251117@mail.gmail.com> References: <3334318183.1248402@smtp.netcom.no> <8833ea530908280649n562234e6s6dfb013ba8251117@mail.gmail.com> Message-ID: <4A9867BC.7050401@canterbury.ac.nz> The trick to supporting things like foo[i][j] is to give foo a __getitem__ method that returns an intermediate object representing a row. The row object holds a reference to the original matrix object, and enough information to locate the row of interest within it. You then give the row object __getitem__ and __setitem__ methods that access an element of the row. It's a fair amount of work to set up, but it's the only way if you can't make use of something like numpy that already does it for you. -- Greg From greg.ewing at canterbury.ac.nz Sat Aug 29 01:49:03 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 29 Aug 2009 11:49:03 +1200 Subject: [Cython] Function pointers in cython In-Reply-To: References: <4A96E2A6.9010409@gmail.com> <4A96F376.9050308@gmail.com> Message-ID: <4A986CEF.4030000@canterbury.ac.nz> Robert Bradshaw wrote: > On Thu, 27 Aug 2009, Ian Langmore wrote: > >>When you say I was "declaring a new variable", I think you're referring to >> >>cdef float use_fun_ptr(float, float, float (float, float)) > > When you do that inside of a function (inside your main() here) you're > declaring a new local variable. Don't know about Cython, but in Pyrex this is an error -- it thinks you've forward-declared a function without defining it. > In terms of top-level declarations, you should never need prototypes in > Cython. You may sometimes need forward declarations, but you don't ever need to put them inside functions in either Pyrex or Cython. -- Greg From mziulu at gmail.com Mon Aug 31 08:40:22 2009 From: mziulu at gmail.com (Mattia Ziulu) Date: Mon, 31 Aug 2009 08:40:22 +0200 Subject: [Cython] Expose a *float element to Python namespace In-Reply-To: <4A9867BC.7050401@canterbury.ac.nz> References: <3334318183.1248402@smtp.netcom.no> <8833ea530908280649n562234e6s6dfb013ba8251117@mail.gmail.com> <4A9867BC.7050401@canterbury.ac.nz> Message-ID: <8833ea530908302340r43e456bhb986b6abc6d96e57@mail.gmail.com> The point was, I didn't actually know how to properly implement __getitem__, I thought that I had to pass it as arguments the indexes (note the plural) of the element I wanted. I made some progress passing it a tuple, calling that is foo[1,2], and this worked. A coworker of mine (my superior, actually) then told me that I could implement __getitem__ to work recursively, and right now that's how i have it working. Only caveat, it doesn't return a row if I ask for foo[n], but that's probably something I can come up with some thought, as Greg points out. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090831/22a5caa3/attachment.htm From robertwb at math.washington.edu Mon Aug 31 18:11:07 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 31 Aug 2009 09:11:07 -0700 Subject: [Cython] Expose a *float element to Python namespace In-Reply-To: <8833ea530908302340r43e456bhb986b6abc6d96e57@mail.gmail.com> References: <3334318183.1248402@smtp.netcom.no> <8833ea530908280649n562234e6s6dfb013ba8251117@mail.gmail.com> <4A9867BC.7050401@canterbury.ac.nz> <8833ea530908302340r43e456bhb986b6abc6d96e57@mail.gmail.com> Message-ID: <626422D9-6C9B-42C0-B802-CB7AFA290889@math.washington.edu> On Aug 30, 2009, at 11:40 PM, Mattia Ziulu wrote: > The point was, I didn't actually know how to properly implement > __getitem__, I thought that I had to pass it as arguments the > indexes (note the plural) of the element I wanted. I made some > progress passing it a tuple, calling that is foo[1,2], and this > worked. A coworker of mine (my superior, actually) then told me > that I could implement __getitem__ to work recursively, and right > now that's how i have it working. Only caveat, it doesn't return a > row if I ask for foo[n], but that's probably something I can come > up with some thought, as Greg points out. Yep, though there's really little benefit to doing it recursively (that's just how C does it because there's no notion of multi- dimensional arrays, just arrays of arrays) and it may have a performance penalty. - Robert