From stefan_ml at behnel.de Sat May 1 23:11:19 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 01 May 2010 23:11:19 +0200 Subject: [Cython] coercion of char/Py_UNICODE to Python objects - string or integer? In-Reply-To: <4BD3FD96.7080803@behnel.de> References: <4BCFE07F.70307@behnel.de> <6F4F2615-4B5B-4DAC-9089-15776CFECE29@math.washington.edu> <4BD3D308.6090509@behnel.de> <4BD3FD96.7080803@behnel.de> Message-ID: <4BDC98F7.7080405@behnel.de> Stefan Behnel, 25.04.2010 10:30: > Stefan Behnel, 25.04.2010 07:28: >> the question is just if we want >> >> py_int_val = uval >> py_ustr_val =uval >> >> or >> >> py_int_val =uval >> py_ustr_val = uval >> >> My gut feeling is that the coercion to strings would be more straight >> forward. It would also clean up the compiler code a bit as implicit >> coercions (e.g. for comparisons) would then work out-of-the-box in both >> ways. Currently, "Py_UNICODE in unicode" must be special cased (which it >> still would in the future, but only for optimisation purposes, not to make >> it work at all). > > Given that only char->bytes breaks backwards compatibility, since > Py_UNICODE wasn't supported at all until now - could we agree on making > Py_UNICODE->unicode the default behaviour and leaving char->PyInt as is for > the time being? We can still decide to break backwards compatibility later > on, and we can always support the explicit > > py_s =char_val > py_i =pyunicode_val > > safely, so users can just rely on that if they need safe behaviour. > > The 'reasoning' would be that a plain 'char' is a bit too generic to map it > to a Python bytes string (note that 'char*' isn't), whereas Py_UNICODE is > something that only really makes sense in the context of a Python unicode > string, so it's a lot less surprising if it also maps to a Python unicode > string. ... and this even matches the behaviour of the 'bytes' type in Python 3 which really returns integer values on indexing/iteration, whereas the 'str' (i.e. 'unicode') type returns substrings. I've implemented the explicit (char)->bytes and default Py_UNICODE->unicode coercions and updated the documentation chapter on string handling accordingly. http://hg.cython.org/cython-docs/rev/e712d9647f47 Stefan From dalcinl at gmail.com Sun May 2 23:49:04 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sun, 2 May 2010 18:49:04 -0300 Subject: [Cython] ssize_t_T399 test failing in Py2.4 In-Reply-To: <4BDC60A0.6020701@behnel.de> References: <4BDC60A0.6020701@behnel.de> Message-ID: On 1 May 2010 14:10, Stefan Behnel wrote: > Hi Lisandro, > > I get this in Py2.4 after your last commit. Can you take a look? > Is that machine a 64bits one? In such case, the test is broken, because of this line: SSIZE_T_MAX = ((-1)>>1) I've pushed a quick fix: http://hg.cython.org/cython-devel/rev/66bc28d5c751 PS: Can you see now why I was no so happy about using Py_ssize_t? Do you think that my original way of defining SSIZE_T_MAX was a nonsense? We should not enforce ssize_t meaning Py_ssize_t, but in case we do it anyway, we should provide users a last chance to change our decision (i.e, add an indirection and #define __Pyx_ssize_t Py_ssize_t). > > > Traceback (most recent call last): > ?File > "/home/scoder/hudson/jobs/py24-svn/workspace/INSTALL/py24/lib/python2.4/doctest.py", > line 2157, in runTest > ? ?raise self.failureException(self.format_failure(new.getvalue())) > AssertionError: Failed doctest test for ssize_t_T399 > ?File > "/mnt/usb1/scratch/hudson/jobs/cython-devel-tests-py24-c/workspace/BUILD/run/c/ssize_t_T399.so", > line 27, in ssize_t_T399 > > ---------------------------------------------------------------------- > File > "/mnt/usb1/scratch/hudson/jobs/cython-devel-tests-py24-c/workspace/BUILD/run/c/ssize_t_T399.so", > line 45, in ssize_t_T399 > Failed example: > ? ?test(SSIZE_T_MAX+1) #doctest: +ELLIPSIS > Expected: > ? ?Traceback (most recent call last): > ? ? ? ?... > ? ?OverflowError: ... > Got: > ? ?0 > ---------------------------------------------------------------------- > File > "/mnt/usb1/scratch/hudson/jobs/cython-devel-tests-py24-c/workspace/BUILD/run/c/ssize_t_T399.so", > line 49, in ssize_t_T399 > Failed example: > ? ?test(SSIZE_T_MIN-1) #doctest: +ELLIPSIS > Expected: > ? ?Traceback (most recent call last): > ? ? ? ?... > ? ?OverflowError: ... > Got: > ? ?-1 > > -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From dalcinl at gmail.com Mon May 3 00:45:11 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sun, 2 May 2010 19:45:11 -0300 Subject: [Cython] coercion of char/Py_UNICODE to Python objects - string or integer? In-Reply-To: <4BDC98F7.7080405@behnel.de> References: <4BCFE07F.70307@behnel.de> <6F4F2615-4B5B-4DAC-9089-15776CFECE29@math.washington.edu> <4BD3D308.6090509@behnel.de> <4BD3FD96.7080803@behnel.de> <4BDC98F7.7080405@behnel.de> Message-ID: On 1 May 2010 18:11, Stefan Behnel wrote: > ... and this even matches the behaviour of the 'bytes' type in Python 3 > which really returns integer values on indexing/iteration, whereas the > 'str' (i.e. 'unicode') type returns substrings. VERY good point... -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From baihaoyu at gmail.com Mon May 3 12:19:40 2010 From: baihaoyu at gmail.com (Haoyu Bai) Date: Mon, 3 May 2010 18:19:40 +0800 Subject: [Cython] Haoyu Bai's application accepted for GSoC In-Reply-To: References: <4BD86899.6090402@behnel.de> Message-ID: On Thu, Apr 29, 2010 at 1:14 AM, Craig Citro wrote: >>> The list of accepted applications for Google Summer of Code 2010 were >>> just announced yesterday, and Haoyu Bai's Cython application was >>> officially accepted. Congratulations, and we're looking forward to a >>> lot of great stuff happing this summer. >> >> Happy to read this. Congratulations! >> > > Ditto -- between this and several of us having other responsibilities > off our shoulders, it should be an exciting summer for Cython. ;) > > -cc Thank you all! I'm excited about the summer, too! :) Craig and I have discussed about the workflow. I'll use some local hg branches to maintain the patches and submit the patches to trac. Hopefully I can start to work on Cython sooner. -- Haoyu BAI School of Computing, National University of Singapore. From kwmsmith at gmail.com Mon May 3 17:34:02 2010 From: kwmsmith at gmail.com (Kurt Smith) Date: Mon, 3 May 2010 10:34:02 -0500 Subject: [Cython] Haoyu Bai's application accepted for GSoC In-Reply-To: References: <4BD86899.6090402@behnel.de> Message-ID: On Mon, May 3, 2010 at 5:19 AM, Haoyu Bai wrote: > On Thu, Apr 29, 2010 at 1:14 AM, Craig Citro wrote: >>>> The list of accepted applications for Google Summer of Code 2010 were >>>> just announced yesterday, and Haoyu Bai's Cython application was >>>> officially accepted. Congratulations, and we're looking forward to a >>>> lot of great stuff happing this summer. >>> >>> Happy to read this. Congratulations! >>> >> >> Ditto -- between this and several of us having other responsibilities >> off our shoulders, it should be an exciting summer for Cython. ;) >> >> -cc > > Thank you all! I'm excited about the summer, too! :) > > Craig and I have discussed about the workflow. I'll use some local hg > branches to maintain the patches and submit the patches to trac. > Hopefully I can start to work on Cython sooner. Just a suggestion, take it or leave it: For my GSoC work I found google's codereview to be a very helpful tool. http://code.google.com/p/rietveld/wiki/CodeReviewHelp It allows you to submit patches that are presented in an app-engine GUI. Reviewers can make comments & suggestions that are local to each change in the patch. Perhaps it would be easier to use for interactive patch review than trac. (It works seamlessly with mercurial, too). Good luck! Kurt From craigcitro at gmail.com Mon May 3 17:40:32 2010 From: craigcitro at gmail.com (Craig Citro) Date: Mon, 3 May 2010 08:40:32 -0700 Subject: [Cython] Haoyu Bai's application accepted for GSoC In-Reply-To: References: <4BD86899.6090402@behnel.de> Message-ID: > Just a suggestion, take it or leave it: > > For my GSoC work I found google's codereview to be a very helpful tool. > > http://code.google.com/p/rietveld/wiki/CodeReviewHelp > > It allows you to submit patches that are presented in an app-engine > GUI. ?Reviewers can make comments & suggestions that are local to each > change in the patch. > > Perhaps it would be easier to use for interactive patch review than > trac. (It works seamlessly with mercurial, too). > +10 -- we should *totally* start using rietveld. (Plus, in a month or two, Robert and I will be totally in love with it.) -cc From robertwb at math.washington.edu Mon May 3 19:58:59 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 3 May 2010 10:58:59 -0700 Subject: [Cython] Haoyu Bai's application accepted for GSoC In-Reply-To: References: <4BD86899.6090402@behnel.de> Message-ID: <1FBF84E4-2FDE-41C3-A009-8A95BD693053@math.washington.edu> On May 3, 2010, at 8:40 AM, Craig Citro wrote: >> Just a suggestion, take it or leave it: >> >> For my GSoC work I found google's codereview to be a very helpful >> tool. >> >> http://code.google.com/p/rietveld/wiki/CodeReviewHelp >> >> It allows you to submit patches that are presented in an app-engine >> GUI. Reviewers can make comments & suggestions that are local to >> each >> change in the patch. >> >> Perhaps it would be easier to use for interactive patch review than >> trac. (It works seamlessly with mercurial, too). >> > > +10 -- we should *totally* start using rietveld. (Plus, in a month or > two, Robert and I will be totally in love with it.) I already am :). It may be good to put up a repo on hg.cython.org (so it's easy for people to sync up with what you have to try it out and play with it). Also, did you set up a blog yet? - Robert From marien.zwart at gmail.com Mon May 3 20:22:30 2010 From: marien.zwart at gmail.com (Marien Zwart) Date: Mon, 03 May 2010 20:22:30 +0200 Subject: [Cython] Missed optimization for list.pop Message-ID: <1272910950.5314.29.camel@cyclops> Hi, http://trac.cython.org/cython_trac/ticket/447 describes an added optimization for calls to list.pop(). Somewhat counter-intuitively this optimization is only applied if pop() is called on things of type "object", not if it is called on things explicitly typed as "list". Compare the following (silly) examples: def f(): cdef list l = [] l.append(0) l.pop() return l def f(): cdef object l = [] l.append(0) l.pop() return l Using either Cython 0.12.1 or cython-devel the second one generates calls to __Pyx_Object_Append and __Pyx_Object_Pop, which take fast paths if called on a list. The first one generates a direct to PyList_Append, but then PyObject_GetAttr and PyObject_Call for pop(). It might be nice to add: _handle_simple_method_list_pop = _handle_simple_method_object_pop somewhere in Cython/Compiler/Optimize.py, which at least makes Cython not generate worse code for the more restrictively typed list (tested it briefly, the resulting code looked reasonable and compiled). From stefan_ml at behnel.de Tue May 4 07:30:30 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 04 May 2010 07:30:30 +0200 Subject: [Cython] Missed optimization for list.pop In-Reply-To: <1272910950.5314.29.camel@cyclops> References: <1272910950.5314.29.camel@cyclops> Message-ID: <4BDFB0F6.9030801@behnel.de> Marien Zwart, 03.05.2010 20:22: > http://trac.cython.org/cython_trac/ticket/447 describes an added > optimization for calls to list.pop(). Somewhat counter-intuitively this > optimization is only applied if pop() is called on things of type > "object", not if it is called on things explicitly typed as "list". > Compare the following (silly) examples: > > def f(): > cdef list l = [] > l.append(0) > l.pop() > return l > > def f(): > cdef object l = [] > l.append(0) > l.pop() > return l > > Using either Cython 0.12.1 or cython-devel the second one generates > calls to __Pyx_Object_Append and __Pyx_Object_Pop, which take fast paths > if called on a list. The first one generates a direct to PyList_Append, > but then PyObject_GetAttr and PyObject_Call for pop(). Yes, I stumbled over that, too, but didn't care to fix it so far. I'll look into it. Stefan From stefan_ml at behnel.de Tue May 4 07:57:28 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 04 May 2010 07:57:28 +0200 Subject: [Cython] Missed optimization for list.pop In-Reply-To: <1272910950.5314.29.camel@cyclops> References: <1272910950.5314.29.camel@cyclops> Message-ID: <4BDFB748.1070605@behnel.de> Marien Zwart, 03.05.2010 20:22: > It might be nice to add: > _handle_simple_method_list_pop = _handle_simple_method_object_pop > > somewhere in Cython/Compiler/Optimize.py, which at least makes Cython > not generate worse code for the more restrictively typed list (tested it > briefly, the resulting code looked reasonable and compiled). That's also the right thing to do. The resulting code is just about as fast as a specialised function for typed lists, so it's not worth doing anything more than that. Thanks! Stefan From stefan_ml at behnel.de Tue May 4 11:16:52 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 04 May 2010 11:16:52 +0200 Subject: [Cython] 'private' extension classes (ticket 263) Message-ID: <4BDFE604.3030708@behnel.de> Hi, the last discussion on this dates back to 2008, and I'd like to finally get this finished up. What would be a good syntax for extension types that are not visible from Python space? I guess a class decorator would make sense, so what about @cython.python_visible(False) cdef class MyType: ... ? Or should we extend the proposed 'cclass' decorator in pure mode? @cython.cclass(python_visible=False) class MyType: ... or both? Stefan From dalcinl at gmail.com Tue May 4 14:47:35 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 4 May 2010 09:47:35 -0300 Subject: [Cython] 'private' extension classes (ticket 263) In-Reply-To: <4BDFE604.3030708@behnel.de> References: <4BDFE604.3030708@behnel.de> Message-ID: On 4 May 2010 06:16, Stefan Behnel wrote: > > @cython.cclass(python_visible=False) > class MyType: > ? ? ... > I like the second, because could even think about extending it to: @cython.cclass(export=True, api=True) class MyType: ... or something on that line, in order to fix the overloaded meaning of "public" and also turn on API generation. However, the concept of "python_visible" could also apply to enum's, and even struct's (provided we can ever manage to fully automate Py<->C conversion). So perhaps we should think about something more generic than "cclass" ? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From gabriele.lanaro at gmail.com Tue May 4 15:36:07 2010 From: gabriele.lanaro at gmail.com (Gabriele Lanaro) Date: Tue, 4 May 2010 15:36:07 +0200 Subject: [Cython] building cython modules in another position Message-ID: Hi! In the project I'm working on there is a splitting between extension source files and pure python source files. I've some problems in compiling correctly the extensions: Src/cython/ one.pyx one.pxd two.pyx two.pxd ... Package/ __init__.py stuff.py ... one,two etc are dependent each other ("two" has a cimport statement in it) When compiling the situation should be: Package/ __init__.py stuff.py one.so two.so ... When importing two.so there's an import error like this (name are different but the concept is the same): File "primitive_gto.pxd", line 18, in contracted_gto (Src/PyQuante/contracted_gto.c:1843) ImportError: No module named primitive_gto If I move the .pyx and .pxd files in Package the compilation goes ok. How can I obtain a correct compilation using the first package structure? (I really need that, it's not my choice) - Gabriele -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20100504/4dd1f57f/attachment.htm From kwmsmith at gmail.com Tue May 4 17:22:14 2010 From: kwmsmith at gmail.com (Kurt Smith) Date: Tue, 4 May 2010 10:22:14 -0500 Subject: [Cython] building cython modules in another position In-Reply-To: References: Message-ID: On Tue, May 4, 2010 at 8:36 AM, Gabriele Lanaro wrote: > Hi! > In the project I'm working on there is a splitting between extension source > files and pure python source files. I've some problems in compiling > correctly the extensions: > Src/cython/ > ? ? ? ?? one.pyx > ? ? ? ?? one.pxd > ? ? ? ?? two.pyx > ???????? two.pxd > ???????? ... > Package/ > ???????? __init__.py > ???????? stuff.py > ???????? ... > > one,two etc are dependent each other ("two" has a cimport statement in it) > When compiling the situation should be: > > Package/ > ????? __init__.py > ????? stuff.py > ????? one.so > ????? two.so > ????? ... > > When importing two.so there's an import error like this (name are different > but the concept is the same): > > File "primitive_gto.pxd", line 18, in contracted_gto > (Src/PyQuante/contracted_gto.c:1843) > ImportError: No module named primitive_gto > > If I move the .pyx and .pxd files in Package the compilation goes ok. How > can I obtain a correct compilation using the first package structure? > > (I really need that, it's not my choice) Looks like Cython (and Cython's distutils) needs to know about the package structure at compile time to get everything setup right. One solution would be the following. Arrange your directory structure like this: . |-- Makefile |-- Package | |-- __init__.py | `-- stuff.py `-- src |-- Package | |-- __init__.py | |-- one.pxd | |-- one.pyx | |-- two.pxd | `-- two.pyx `-- setup.py Essentially you create a parallel package structure with the cython sources, and tell distutils to put the compiled extension modules in the right place. Here's the contents of the makefile: $ cat Makefile all: (cd src && python setup.py build_ext -b ..) clean: -rm -r src/build -rm Package/*.so -rm src/Package/*.c I've attached this solution to the email -- hopefully it gives you the idea. Kurt -------------- next part -------------- A non-text attachment was scrubbed... Name: test_cython_compile.tgz Type: application/x-gzip Size: 1082 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20100504/22758ff4/attachment.bin From gabriele.lanaro at gmail.com Tue May 4 18:36:32 2010 From: gabriele.lanaro at gmail.com (Gabriele Lanaro) Date: Tue, 4 May 2010 18:36:32 +0200 Subject: [Cython] building cython modules in another position In-Reply-To: References: Message-ID: Thank you very much for pointing this out and giving me the example, now I've solved my problem. It also work by putting setup.py script one level up. I also want to reinforce that the __init__.py file in src/Package is required (for other people searching for a solution). Thanks again, - Gabriele 2010/5/4 Kurt Smith > On Tue, May 4, 2010 at 8:36 AM, Gabriele Lanaro > wrote: > > Hi! > > In the project I'm working on there is a splitting between extension > source > > files and pure python source files. I've some problems in compiling > > correctly the extensions: > > Src/cython/ > > one.pyx > > one.pxd > > two.pyx > > two.pxd > > ... > > Package/ > > __init__.py > > stuff.py > > ... > > > > one,two etc are dependent each other ("two" has a cimport statement in > it) > > When compiling the situation should be: > > > > Package/ > > __init__.py > > stuff.py > > one.so > > two.so > > ... > > > > When importing two.so there's an import error like this (name are > different > > but the concept is the same): > > > > File "primitive_gto.pxd", line 18, in contracted_gto > > (Src/PyQuante/contracted_gto.c:1843) > > ImportError: No module named primitive_gto > > > > If I move the .pyx and .pxd files in Package the compilation goes ok. How > > can I obtain a correct compilation using the first package structure? > > > > (I really need that, it's not my choice) > > Looks like Cython (and Cython's distutils) needs to know about the > package structure at compile time to get everything setup right. > > One solution would be the following. Arrange your directory structure > like this: > > . > |-- Makefile > |-- Package > | |-- __init__.py > | `-- stuff.py > `-- src > |-- Package > | |-- __init__.py > | |-- one.pxd > | |-- one.pyx > | |-- two.pxd > | `-- two.pyx > `-- setup.py > > Essentially you create a parallel package structure with the cython > sources, and tell distutils to put the compiled extension modules in > the right place. > > Here's the contents of the makefile: > > $ cat Makefile > all: > (cd src && python setup.py build_ext -b ..) > > clean: > -rm -r src/build > -rm Package/*.so > -rm src/Package/*.c > > I've attached this solution to the email -- hopefully it gives you the > idea. > > Kurt > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20100504/738b081f/attachment-0001.htm From binet at cern.ch Tue May 4 20:43:09 2010 From: binet at cern.ch (Sebastien Binet) Date: Tue, 4 May 2010 20:43:09 +0200 Subject: [Cython] json to .pxd - autogen cython declaration code In-Reply-To: References: <1271407753-sup-9628@farnsworth> <1271760467-sup-7955@farnsworth> Message-ID: <1272998114-sup-4328@farnsworth> Excerpts from David Cournapeau's message of 2010-04-22 05:09:43 +0200: > On Tue, Apr 20, 2010 at 7:53 PM, Sebastien Binet wrote: > > Excerpts from David Cournapeau's message of 2010-04-20 12:06:17 +0200: > >> On Tue, Apr 20, 2010 at 12:57 AM, Haoyu Bai wrote: > >> > >> > Hi, > >> > > >> > I just found a project on GitHub which might do the thing you want: > >> > > >> > http://github.com/cournape/cython-codegen > >> > >> That code is horrible though :) > > > > don't worry about that, I am dealing with high energy physicists (C++ > > and python) code on a daily basis :) > > I love the researchers/C++ combination, always full of surprises :) > > I mostly meant that something reusable should not be based on this > code at all - neither the implementation nor the idea of reusing > gccxml makes much sense for something meant to be used in the long > term. Gccxml is not that great, because it is not really maintained, a > PITA to build, and you cannot force C mode (which was the biggest > issue in my case, and I guess an issue for many cython users). > > > > > thanks for the link. > > llvm/clang was on my radar (I played a bit w/ llvm-py) but this > > tutorial will most definitely help ! > > Yes, unfortunately, there is no python API, and llvm is not as mature > as I had hoped (the API has significantly changed since this tutorial, > for example). OTOH, for C++, llvm code is relatively readable, and > there are also an ocaml interface and a haskell one > (http://augustss.blogspot.com/) - maybe a good excuse to learn > haskell. > > It took me half a day to be able to implement a basic pre-processor in > C++ with llvm, so anyone more familiar with C++ (and parsing > techniques) should be able to implement something useful in a short > amount of time, ok... let me push something out of the door: http://bitbucket.org/binet/cylon/src/tip/ext/clang/genjson/ the code is still *very* crappy and it seems I managed to make the clang plugin loader crash when STL headers are included in someone's source code... but it would be a start. in that directory you'll find a dumb Makefile which should: - compile the CLang plugin library - run the plugin on the utest.cxx file (through run.py) - which will create a utest_ast.py file - and then create a utest_ast.json file from it now that I know a bit more my way around CLang's source code, I shall re-do the exercize using the ast-print-xml plugin as inspiration (for some reason, that plugin crashes or just creates corrupted xml files...) I mainly put the utest_ast.{py,json} as a tentative start for feedback on what and how types,fcts,... should be exposed. (a proper python API to manipulate those ast/json structures should also be devised) cheers, sebastien. -- ######################################### # Dr. Sebastien Binet # Laboratoire de l'Accelerateur Lineaire # Universite Paris-Sud XI # Batiment 200 # 91898 Orsay ######################################### From dalcinl at gmail.com Wed May 5 17:23:08 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 5 May 2010 12:23:08 -0300 Subject: [Cython] legacy pxd's in Cython/Includes Message-ID: How are we going to manage these old, deprecated pxd's? I would Like to move all them to other place (let say Cython/IncludesOld) and add that dir to the default search path for a few releases. No idea how to generate a deprecation warning, though. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From dalcinl at gmail.com Wed May 5 19:02:21 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 5 May 2010 14:02:21 -0300 Subject: [Cython] going to add POSIX pxd's, needing a little help from your side Message-ID: I'm in the process of adding some pxd's in Cython/Includes for POSIX (up to now, I've implemented 'fcntl' and 'unistd'). I want to add some tests, I've started implementing a context manager for temporarily stdout redirection to /dev/null in order to call a noisy, external C function with lots of hard-wired 'printf' calls. But these tests will not run on Windows. Could any of you fix runtests.py to filter testcases with 'posix' in their name if os.name()!='posix' ? I could do it myself after some research, but I bet Stefan/Robert can implement it in 5 seconds :-) After that, I could push my POSIX stuff. PS: Cython's testsuite is running on Windows (Vista+MSVC|MinGW) with 0(zero) failures! And I do not want to push things that could change that ;-) -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From dagss at student.matnat.uio.no Wed May 5 19:06:40 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 05 May 2010 19:06:40 +0200 Subject: [Cython] going to add POSIX pxd's, needing a little help from your side In-Reply-To: References: Message-ID: <4BE1A5A0.2030405@student.matnat.uio.no> Lisandro Dalcin wrote: > I'm in the process of adding some pxd's in Cython/Includes for POSIX > (up to now, I've implemented 'fcntl' and 'unistd'). I want to add some > tests, I've started implementing a context manager for temporarily > stdout redirection to /dev/null in order to call a noisy, external C > function with lots of hard-wired 'printf' calls. > > But these tests will not run on Windows. Could any of you fix > runtests.py to filter testcases with 'posix' in their name if > os.name()!='posix' ? I could do it myself after some research, but I > bet Stefan/Robert can implement it in 5 seconds :-) After that, I > could push my POSIX stuff. I don't have time now, but you could add the pattern to EXT_DEP_MODULES at the top of runtests.py; conditional on the import of the fcntl module. -- Dag Sverre From dagss at student.matnat.uio.no Wed May 5 19:08:55 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 05 May 2010 19:08:55 +0200 Subject: [Cython] going to add POSIX pxd's, needing a little help from your side In-Reply-To: <4BE1A5A0.2030405@student.matnat.uio.no> References: <4BE1A5A0.2030405@student.matnat.uio.no> Message-ID: <4BE1A627.1020500@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: >> I'm in the process of adding some pxd's in Cython/Includes for POSIX >> (up to now, I've implemented 'fcntl' and 'unistd'). I want to add some >> tests, I've started implementing a context manager for temporarily >> stdout redirection to /dev/null in order to call a noisy, external C >> function with lots of hard-wired 'printf' calls. >> >> But these tests will not run on Windows. Could any of you fix >> runtests.py to filter testcases with 'posix' in their name if >> os.name()!='posix' ? I could do it myself after some research, but I >> bet Stefan/Robert can implement it in 5 seconds :-) After that, I >> could push my POSIX stuff. > > I don't have time now, but you could add the pattern to EXT_DEP_MODULES > at the top of runtests.py; conditional on the import of the fcntl module. > Also, going offtopic for the benefit of anyone who might struggle with the fcntl module: flock is bad. lockf is good. At least in the context of programming on a cluster. (Yes, there are two of them, and I wasted some hours before I discovered that.) -- Dag Sverre From dalcinl at gmail.com Wed May 5 19:42:20 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 5 May 2010 14:42:20 -0300 Subject: [Cython] going to add POSIX pxd's, needing a little help from your side In-Reply-To: <4BE1A627.1020500@student.matnat.uio.no> References: <4BE1A5A0.2030405@student.matnat.uio.no> <4BE1A627.1020500@student.matnat.uio.no> Message-ID: On 5 May 2010 14:08, Dag Sverre Seljebotn wrote: > > Also, going offtopic for the benefit of anyone who might struggle with > the fcntl module: > > flock is bad. lockf is good. At least in the context of programming on a > cluster. (Yes, there are two of them, and I wasted some hours before I > discovered that.) > So MPI_File_xxx API's are not useful for your? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From dalcinl at gmail.com Wed May 5 19:49:12 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 5 May 2010 14:49:12 -0300 Subject: [Cython] going to add POSIX pxd's, needing a little help from your side In-Reply-To: <4BE1A5A0.2030405@student.matnat.uio.no> References: <4BE1A5A0.2030405@student.matnat.uio.no> Message-ID: On 5 May 2010 14:06, Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: >> I'm in the process of adding some pxd's in Cython/Includes for POSIX >> (up to now, I've implemented 'fcntl' and 'unistd'). I want to add some >> tests, I've started implementing a context manager for temporarily >> stdout redirection to /dev/null in order to call a noisy, external C >> function with lots of hard-wired 'printf' calls. >> >> But these tests will not run on Windows. Could any of you fix >> runtests.py to filter testcases with 'posix' in their name if >> os.name()!='posix' ? I could do it myself after some research, but I >> bet Stefan/Robert can implement it in 5 seconds :-) ?After that, I >> could push my POSIX stuff. > > I don't have time now, but you could add the pattern to EXT_DEP_MODULES > at the top of runtests.py; conditional on the import of the fcntl module. > Yes, I now see that's the way, though I'll make it conditional on the import of the 'posix' module. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From dalcinl at gmail.com Wed May 5 20:52:47 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 5 May 2010 15:52:47 -0300 Subject: [Cython] going to add POSIX pxd's, needing a little help from your side In-Reply-To: References: Message-ID: On 5 May 2010 14:02, Lisandro Dalcin wrote: > I'm in the process of adding some pxd's in Cython/Includes for POSIX > (up to now, I've implemented 'fcntl' and 'unistd'). http://hg.cython.org/cython-devel/rev/9dc7e1a2d56a -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From dagss at student.matnat.uio.no Wed May 5 21:10:12 2010 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 05 May 2010 21:10:12 +0200 Subject: [Cython] going to add POSIX pxd's, needing a little help from your side In-Reply-To: References: <4BE1A5A0.2030405@student.matnat.uio.no> <4BE1A627.1020500@student.matnat.uio.no> Message-ID: <4BE1C294.4060207@student.matnat.uio.no> Lisandro Dalcin wrote: > On 5 May 2010 14:08, Dag Sverre Seljebotn wrote: >> Also, going offtopic for the benefit of anyone who might struggle with >> the fcntl module: >> >> flock is bad. lockf is good. At least in the context of programming on a >> cluster. (Yes, there are two of them, and I wasted some hours before I >> discovered that.) >> > > So MPI_File_xxx API's are not useful for your? Well, if I can avoid writing at that low level I do so (and I haven't had the need yet). I prefer working with a self-annotating file format of some sort, either HDF5 or FITS. HDF5 does support MPI-IO, but the available Python wrappers doesn't follow through currently... Luckily my applications are not anywhere close to be filesystem IO bound. Locking with lockf then opening the file using h5py or PyTables is convenient for some isolated purposes (but the occasions IO is needed must be few, or else the processes will just sit and wait for locks...). -- Dag Sverre From wing1127aishi at gmail.com Wed May 5 23:46:48 2010 From: wing1127aishi at gmail.com (Leon Sit) Date: Wed, 5 May 2010 16:46:48 -0500 Subject: [Cython] Status of closure Message-ID: Hi all, I have been following the dev mailing list and got the impression that closure will likely to be merged to the next release. Is it still the case? If this is the case, what branch should I pull to test-run the closure code? I have some mount of code that make heavy use of itertools and would like to test the new features against my code. Thanks. Leon From craigcitro at gmail.com Thu May 6 00:19:16 2010 From: craigcitro at gmail.com (Craig Citro) Date: Wed, 5 May 2010 15:19:16 -0700 Subject: [Cython] Status of closure In-Reply-To: References: Message-ID: Hi Leon, > I have been following the dev mailing list and got the impression that > closure will likely to be merged to the next release. Is it still the > case? If this is the case, what branch should I pull to test-run the > closure code? I have some mount of code that make heavy use of > itertools and would like to test the new features against my code. > The plan is still to include closures in 0.13 -- it's currently long-delayed, but it will happen. I'm the release manager, but I've been horribly busy with the rest of life. In the interim, you can just clone the closures branch: http://hg.cython.org/cython-closures/ which also has the most recent cython-devel merged, too. Give it a go, and definitely let us know if you run into any troubles. Warning: closures are supported, but generators/yield are still not, unless Stefan's been doing more than I think. ;) -cc From robertwb at math.washington.edu Sat May 8 21:39:29 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 8 May 2010 12:39:29 -0700 Subject: [Cython] legacy pxd's in Cython/Includes In-Reply-To: References: Message-ID: <621B3AAC-F69A-4BED-83EA-5016A7F57EDC@math.washington.edu> On May 5, 2010, at 8:23 AM, Lisandro Dalcin wrote: > How are we going to manage these old, deprecated pxd's? I would Like > to move all them to other place (let say Cython/IncludesOld) and add > that dir to the default search path for a few releases. Or maybe even Cython/Includes/deprecated (where one would still be able to look them up as if they were top level with a warning 'till their gone). > No idea how to generate a deprecation warning, though. Maybe in the lookup module code, where it tries to locate the pxd, if all else fails it could try that directory (hard coded in) and raise a deprecation warning before failing. Somehow it'd be nice for the message to tell it where to actually look. - Robert From robertwb at math.washington.edu Sat May 8 23:34:15 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 8 May 2010 14:34:15 -0700 Subject: [Cython] Fwd: AttributeError: 'CppClassScope' object has no attribute 'add_include_file' References: <87ljbuyvkn.fsf@tiny.univ-orleans.fr> Message-ID: <055E14CF-0761-470C-83BF-7A3F68BA5C2C@math.washington.edu> Here's a message from a user who had his email bounce because he's not signed up to the list. > From: Denys Duchier > Subject: AttributeError: 'CppClassScope' object has no attribute > 'add_include_file' > To: cython-dev at codespeak.net > Date: Sat, 08 May 2010 20:56:50 +0200 > > not sure where to submit this bug report/fix, so here goes: > > I am using cython-devel, and "extern +" annotations on methods of a > "cdef cppclass" declaration dont currently compile due to the > following > error: > > File "/home/denys/lib/python/Cython/Compiler/Nodes.py", line 575, > in analyse > env.add_include_file('stdexcept') > AttributeError: 'CppClassScope' object has no attribute > 'add_include_file' > > I managed to make it compile happily with the following patch: > > diff -r 85ae5e1b7fb9 Cython/Compiler/Symtab.py > --- a/Cython/Compiler/Symtab.py Thu May 06 08:27:37 2010 +0200 > +++ b/Cython/Compiler/Symtab.py Sat May 08 20:44:51 2010 +0200 > @@ -643,6 +643,9 @@ > else: > return outer.is_cpp() > > + def add_include_file(self, filename): > + self.outer_scope.add_include_file(filename) > + > class PreImportScope(Scope): > > namespace_cname = Naming.preimport_cname > > Cheers, > > --Denys > ---------- From dalcinl at gmail.com Sun May 9 00:30:33 2010 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sat, 8 May 2010 19:30:33 -0300 Subject: [Cython] Fwd: AttributeError: 'CppClassScope' object has no attribute 'add_include_file' In-Reply-To: <055E14CF-0761-470C-83BF-7A3F68BA5C2C@math.washington.edu> References: <87ljbuyvkn.fsf@tiny.univ-orleans.fr> <055E14CF-0761-470C-83BF-7A3F68BA5C2C@math.washington.edu> Message-ID: On 8 May 2010 18:34, Robert Bradshaw wrote: > Here's a message from a user who had his email bounce because he's not > signed up to the list. > >> From: Denys Duchier >> Subject: AttributeError: 'CppClassScope' object has no attribute >> 'add_include_file' >> To: cython-dev at codespeak.net >> Date: Sat, 08 May 2010 20:56:50 +0200 >> >> not sure where to submit this bug report/fix, so here goes: >> >> I am using cython-devel, and "extern +" annotations on methods of a >> "cdef cppclass" declaration dont currently compile due to the >> following >> error: >> >> ?File "/home/denys/lib/python/Cython/Compiler/Nodes.py", line 575, >> in analyse >> ? ?env.add_include_file('stdexcept') >> AttributeError: 'CppClassScope' object has no attribute >> 'add_include_file' >> >> I managed to make it compile happily with the following patch: >> >> diff -r 85ae5e1b7fb9 Cython/Compiler/Symtab.py >> --- a/Cython/Compiler/Symtab.py ? ? ? Thu May 06 08:27:37 2010 +0200 >> +++ b/Cython/Compiler/Symtab.py ? ? ? Sat May 08 20:44:51 2010 +0200 >> @@ -643,6 +643,9 @@ >> ? ? ? ? else: >> ? ? ? ? ? ? return outer.is_cpp() >> >> + ? ?def add_include_file(self, filename): >> + ? ? ? ?self.outer_scope.add_include_file(filename) >> + >> class PreImportScope(Scope): >> >> ? ? namespace_cname = Naming.preimport_cname >> >> Cheers, >> >> --Denys >> ---------- > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > A slightly different approach (add the missing method to CppClassScope), and a few tests: http://hg.cython.org/cython-devel/rev/e14fe9894ca9 Many thanks to Denys for reporting the issue. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From ademan555 at gmail.com Sun May 9 08:12:58 2010 From: ademan555 at gmail.com (Dan Roberts) Date: Sat, 8 May 2010 23:12:58 -0700 Subject: [Cython] __Pyx_GetException() and friends In-Reply-To: References: Message-ID: The code that Cython generates for numpy/random/mtrand/mtrand.pyx contains several __Pyx_*() functions that muck about in the PyThreadState object (which isn't standard) but each seems to have a sanctioned equivalent provided in the official C API, listed here: http://docs.python.org/c-api/exceptions.html This is important because non-CPython interpreters likely won't expose that struct, and indeed PyPy does not, and it would be cumbersome to do so. So I suppose my question, then, is, is there a reason why those functions are implemented? I'm asking here before filing a bug report in case there's a good reason for this that I'm oblivious to. Me being oblivious is not unheard of... Cheers, Dan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20100508/ba9b2285/attachment.htm From stefan_ml at behnel.de Sun May 9 12:48:52 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 09 May 2010 12:48:52 +0200 Subject: [Cython] __Pyx_GetException() and friends In-Reply-To: References: Message-ID: <4BE69314.1030607@behnel.de> Dan Roberts, 09.05.2010 08:12: > The code that Cython generates for numpy/random/mtrand/mtrand.pyx contains > several __Pyx_*() functions that muck about in the PyThreadState object > (which isn't standard) Well, it's not an object, first of all, but IMHO, it's pretty much standard for CPython. And it's not Cython's fault that other runtimes refuse to implement it. > but each seems to have a sanctioned equivalent > provided in the official C API, listed here: > http://docs.python.org/c-api/exceptions.html I think "seems" is the right word here. They are either not exactly equivalent or have CPython version portability constrains, or they're simply not fast enough. Without looking closer, any subset of those three properties will likely hold for each of them. Note that exception handling in Py3 is handled very differently from that in Py2, simply because Py3 has suitable C-API functions that Py2 lacks, or that behave sufficiently different in Py2 to render them useless to Cython. > This is important because non-CPython interpreters likely won't expose that > struct, and indeed PyPy does not, and it would be cumbersome to do so. I don't expect much Cython code to run on PyPy anyway. The exception handling is only one reason why this won't easily work. That being said, I think this is absolutely worth a closer look. I wouldn't mind adding compatibility #define/#ifdef blocks to Cython to support PyPy or IronPython explicitly, as long as this doesn't get too ugly. Especially for the case of exception handling, the respective code may look yet another bit different for other runtimes. We have all sorts of portability code for various CPython versions in there anyway, and most of the relevant code is written down in separate utility functions instead of being generated line by line, so this shouldn't be too hard to integrate. Any volunteers? Stefan From ademan555 at gmail.com Mon May 10 23:12:08 2010 From: ademan555 at gmail.com (Dan Roberts) Date: Mon, 10 May 2010 14:12:08 -0700 Subject: [Cython] __Pyx_GetException() and friends In-Reply-To: References: <4BE69314.1030607@behnel.de> Message-ID: Hi Stefan, Thanks for getting back to me. I've been digging pretty deep to try and see what's so special about __Pyx_GetException() and the only thing I see is that it copies curexc_* fields into exc_* fields. I've dug around in the CPython source and I'm having a hard time figuring out the importance of exc_* If we could establish that, we could build a new __Pyx_GetException() that only relies on PyErr_Fetch(). (The only other thing is __Pyx_GetException() deals with refcounts too, but that could be added easily) I'm glad you're open to having multiple implementations, that way everyone is happy :-) I suppose since I've brought this up, and I'm most immediately affected, it falls to me to implement. That's fine as long as it doesn't take my whole summer :-p. And hopefully someone will be willing to guide me and answer my possibly dumb questions. I see that cython has a healthy heaping of tests too, which should help in determining compliance. I think I'm up to the challenge... I think... Thanks, Dan On May 9, 2010 3:49 AM, "Stefan Behnel" wrote: Dan Roberts, 09.05.2010 08:12: > The code that Cython generates for numpy/random/mtrand/mtrand.pyx contains > several __Pyx_*() fun... Well, it's not an object, first of all, but IMHO, it's pretty much standard for CPython. And it's not Cython's fault that other runtimes refuse to implement it. > but each seems to have a sanctioned equivalent > provided in the official C API, listed here: > ... I think "seems" is the right word here. They are either not exactly equivalent or have CPython version portability constrains, or they're simply not fast enough. Without looking closer, any subset of those three properties will likely hold for each of them. Note that exception handling in Py3 is handled very differently from that in Py2, simply because Py3 has suitable C-API functions that Py2 lacks, or that behave sufficiently different in Py2 to render them useless to Cython. > This is important because non-CPython interpreters likely won't expose that > struct, and indeed... I don't expect much Cython code to run on PyPy anyway. The exception handling is only one reason why this won't easily work. That being said, I think this is absolutely worth a closer look. I wouldn't mind adding compatibility #define/#ifdef blocks to Cython to support PyPy or IronPython explicitly, as long as this doesn't get too ugly. Especially for the case of exception handling, the respective code may look yet another bit different for other runtimes. We have all sorts of portability code for various CPython versions in there anyway, and most of the relevant code is written down in separate utility functions instead of being generated line by line, so this shouldn't be too hard to integrate. Any volunteers? Stefan _______________________________________________ Cython-dev mailing list Cython-dev at codespeak.net http://codespeak.net/mailman/listinfo/cython-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20100510/643d6e3d/attachment-0001.htm From pfh at logarithmic.net Tue May 11 02:27:58 2010 From: pfh at logarithmic.net (Paul Harrison) Date: Tue, 11 May 2010 10:27:58 +1000 Subject: [Cython] Problem with string literals containing newline in Cython 12.1 Message-ID: Hello, I found a problem in Cython 12.1 (with Python 2.6.1). If you try to compile the following: str1 = 'a' str2 = 'a\n' It produces the following code, which will not compile: static char __pyx_k__a[] = "a"; static char __pyx_k__a [] = "a\n"; Also, I find I can no longer cast str to char* (again with Python 2.6.1). The documentation seems to indicate this should only be a problem in Python 3, or have I missed something? regards, Paul Harrison Victorian Bioinformatics Consortium / Monash University pfh at logarithmic.net jabber pfh at gristle.org http://www.logarithmic.net/pfh/ From robertwb at math.washington.edu Tue May 11 06:16:01 2010 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 10 May 2010 21:16:01 -0700 Subject: [Cython] Problem with string literals containing newline in Cython 12.1 In-Reply-To: References: Message-ID: On May 10, 2010, at 5:27 PM, Paul Harrison wrote: > Hello, > > I found a problem in Cython 12.1 (with Python 2.6.1). If you try to > compile the following: > > str1 = 'a' > str2 = 'a\n' > > It produces the following code, which will not compile: > > static char __pyx_k__a[] = "a"; > static char __pyx_k__a > [] = "a\n"; I thought C was whitespace agnostic. What compiler are you using? In any case, it's probably a trivial fix. > Also, I find I can no longer cast str to char* (again with Python > 2.6.1). The documentation seems to indicate this should only be a > problem in Python 3, or have I missed something? Cython is super strict about this right now even if you're only using Py2 (too strict in my opinion, but that's a whole can of worms there that we're not going to re-visit at this time...) Cast to an object first, i.e. if x is a str, do char* s =