From michael.abshoff at googlemail.com Fri May 1 00:53:11 2009 From: michael.abshoff at googlemail.com (Michael Abshoff) Date: Thu, 30 Apr 2009 15:53:11 -0700 Subject: [Cython] C -- Variable-sized array on stack? In-Reply-To: <49F9FE8C.60707@student.matnat.uio.no> References: <49F9F714.70907@student.matnat.uio.no> <49F9FDAE.20301@canonware.com> <49F9FE8C.60707@student.matnat.uio.no> Message-ID: <49FA2BD7.2050503@gmail.com> Dag Sverre Seljebotn wrote: > Jason Evans wrote: >> Dag Sverre Seljebotn wrote: >>> Will I get away with code like this in Cython? >>> >>> void foo(int n) { >>> int stack[n]; >>> ... >>> } >>> >>> I.e. what C language level are we targeting exactly? Also, does the >>> above cause a proper stack allocation? >> I think this is a C99 feature that happens to be part of gnu89 as well. >> I seem to recall that MSVC++ does not support the feature, which is an >> important consideration here. And MSVC did that not in order to spite everyone, but because it opens a giant can of worms including security issues. At 23C3 I saw a demo on how this "feature" can be used to do many bad, bad things with gcc since it did not do any checking at all if the stack was large enough, etc. > Sure, I'm convinced not to use it now. Good :) Cheers, Michael From robertwb at math.washington.edu Fri May 1 07:12:06 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 30 Apr 2009 22:12:06 -0700 Subject: [Cython] Path towards Python completeness In-Reply-To: <49F9D371.3060209@behnel.de> References: <49ED5CE8.6090708@behnel.de> <80500050-599D-4F3A-9FA1-0291E43B3B99@math.washington.edu> <380c38b6623e9a1a6d12a649dee43ec4.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <2038f7c55d17516c79c8b3b2a9385fb1.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49F76991.5010303@behnel.de> <49F9D371.3060209@behnel.de> Message-ID: <140BC717-E6BC-4188-A5B6-42AC9D529B29@math.washington.edu> On Apr 30, 2009, at 9:36 AM, Stefan Behnel wrote: > Stefan Behnel wrote: >> Stefan Behnel wrote: >>> Robert Bradshaw wrote: >>>> http://hg.cython.org/cython-closures/file/c4ee199f08a2/tests/run/ >>>> closures_T82.pyx >>>> >>>> Before you get your hopes up too much, I want to add a big >>>> disclaimer that though I tried to be careful, this is very first- >>>> draft code and there are bound to be issues (e.g. I think I've run >>>> into some with respect to Python arguments). >>> I'll give it a try during the next days. >> >> I exercised it a bit and added more test cases. That revealed a >> couple of >> problems, although I must say that it works pretty well in general. >> >> I added comments to the tests as far as I got with analysing them. >> One >> segfaults pretty badly, which makes it harder to figure out. > > Ok, that was due to ref-counting problems. I was hoping that's all it was (not that tracking down ref-counting issues is always easy...) > All of these work for me now: > > http://hg.cython.org/cython-closures/file/626e754738b6/tests/run/ > closures_T82.pyx > > I find that pretty impressive. Very good work, Robert! Thanks. > This looks like it could go right into 0.12 rather than waiting for > 0.13. Yep, I think that's very feasible too. - Robert From robertwb at math.washington.edu Fri May 1 07:19:32 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 30 Apr 2009 22:19:32 -0700 Subject: [Cython] hg repo doesn't seem to work? In-Reply-To: References: <85b5c3130904291146v1395cd1en81ec1415da1b9c58@mail.gmail.com> Message-ID: <8F3A5997-2642-4789-A80E-7C2D2D1F0E34@math.washington.edu> On Apr 29, 2009, at 12:33 PM, Robert Bradshaw wrote: > On Apr 29, 2009, at 11:46 AM, Ondrej Certik wrote: > >> Hi, >> >> I did: >> >> $ hg clone http://hg.cython.org/cython/ >> destination directory: cython >> requesting all changes >> adding changesets >> adding manifests >> adding file changes >> transaction abort! >> rollback completed >> abort: 00changelog.i at 696e20737973: no node! >> >> am I doing something wrong? > > Hmm... I'm getting that too. I can't look into this now, but will try > to do so later. You can clone from the others right now. OK, it was missing a file in the repo (?). I think something got lost when we packaged it with distuitils. Resolved. - Robert From dagss at student.matnat.uio.no Fri May 1 14:28:25 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 01 May 2009 14:28:25 +0200 Subject: [Cython] Please vote: wraparound directive Message-ID: <49FAEAE9.7000908@student.matnat.uio.no> A new directive which allows Cython to assume that integers passed to [] are not negative. Currently buffers have the negative_indices option: object[int, negative_indices=False] arr = ... but this can be cumbersome and less flexible. The rationale for not doing this in the first place was that it changes semantics during normal running rather than under exceptional circumstances, but now @cdivision has crossed that line anyway. Also note that this could potentially apply to typed lists and tuples as well, if there's a benefit. -- Dag Sverre From dagss at student.matnat.uio.no Fri May 1 14:53:32 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 01 May 2009 14:53:32 +0200 Subject: [Cython] 0.11.2 release schedule? Message-ID: <49FAF0CC.5070108@student.matnat.uio.no> Like I've been saying on some occasions, I'm going show off Cython 18th of May, and have had a tendency to scratch some itches rather than planning how to explain them... so I'd really like a release to happen before then. (Though I also have some fixes/rewrites which are not done yet which will go in next week.) If it helps I can play release manager this time and do the groundwork for making it build Sage etc. (especially if that gets us complex floats in time :-)). Would it be possible to e.g. do a full stop feature freeze next Friday 8th, and then a release the week after? -- Dag Sverre From kwmsmith at gmail.com Fri May 1 15:50:01 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Fri, 1 May 2009 08:50:01 -0500 Subject: [Cython] Please vote: wraparound directive In-Reply-To: <49FAEAE9.7000908@student.matnat.uio.no> References: <49FAEAE9.7000908@student.matnat.uio.no> Message-ID: On Fri, May 1, 2009 at 7:28 AM, Dag Sverre Seljebotn wrote: > A new directive which allows Cython to assume that integers passed to [] > are not negative. > > Currently buffers have the negative_indices option: > > ? object[int, negative_indices=False] arr = ... > > but this can be cumbersome and less flexible. The rationale for not > doing this in the first place was that it changes semantics during > normal running rather than under exceptional circumstances, but now > @cdivision has crossed that line anyway. +1 for me. Since directives can be module, function or block scoped they're pretty flexible, and it makes it clear & explicit that different semantics are going on. The current state allows the use of two different negative_indices type buffers in the same expression: arr_neg[a,b] = arr_neg[a,b] + arr_no_neg[abs(a), abs(b)] While that would be impossible with the proposed directive. I can't easily think of cases when I'd like to mix negative & non_negative indexing in the same expression, or when I couldn't just revert to regular indexing for these cases when required -- presumably I'm giving the arr_no_neg 'negative_indices=False' since I want to speed things up, and I should do the same for the other arrays. I like that it lifts the semantics out of the individual arrays and puts it in a directive. Kurt > > Also note that this could potentially apply to typed lists and tuples as > well, if there's a benefit. > > -- > Dag Sverre > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From stefan_ml at behnel.de Fri May 1 18:18:00 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 01 May 2009 18:18:00 +0200 Subject: [Cython] 0.11.2 release schedule? In-Reply-To: <49FAF0CC.5070108@student.matnat.uio.no> References: <49FAF0CC.5070108@student.matnat.uio.no> Message-ID: <49FB20B8.8090506@behnel.de> Dag Sverre Seljebotn wrote: > Like I've been saying on some occasions, I'm going show off Cython 18th > of May, and have had a tendency to scratch some itches rather than > planning how to explain them... so I'd really like a release to happen > before then. (Though I also have some fixes/rewrites which are not done > yet which will go in next week.) > > If it helps I can play release manager this time and do the groundwork > for making it build Sage etc. (especially if that gets us complex floats > in time :-)). > > Would it be possible to e.g. do a full stop feature freeze next Friday > 8th, and then a release the week after? I'm not currently working on cython-devel anyway, so I'm glad to hear that you volunteer as QA manager. :) There are tons of open issues assigned to 0.11.2, though. I don't expect many more releases for 0.11.x (maybe a .3, but not necessarily a .4), so they may have to get moved to 0.12. At least, there are no really critical issues. Some even seem to be so low priority that we should remove their milestone all together. Even some 'wishlist' bugs sound more important than some of the 0.11.2 tickets. A minor bug in 0.11 doesn't mean it needs fixing in a 0.11.x release, especially if the fix is not trivial. Stefan From simon.king at uni-jena.de Fri May 1 19:15:59 2009 From: simon.king at uni-jena.de (Simon King) Date: Fri, 1 May 2009 19:15:59 +0200 Subject: [Cython] Problems with creating a package of extension modules Message-ID: <1241198159.49fb2e4f76162@webmail.uni-jena.de> Dear Cython-Dev, currently I try to create a Cython package comprising couple of extension modules for usage in Sage. It somehow works, but apparently the extension classes in the modules don't know where they belong to. In my setup.py, I have setup( name = "pGroupCohomology", version = "1.0", author = "Simon A. King", author_email = "simon.king at uni-jena.de", url = "http://users.minet.uni-jena.de/~king/cohomology/", description = "Modular Cohomology Rings of Groups of Prime Power Order", packages=["pGroupCohomology"], package_dir={"pGroupCohomology":"CohoSrc"}, ext_package="pGroupCohomology", ext_modules=[ Extension("mtx", sources = ["CohoSrc/mtx.pyx", ...], ...), Extension(...)] cmdclass = {'build_ext': build_ext} ) I learned from some documentation that providing the optional argument 'ext_package' should make the Extensions modules in that package, so, it should be Extension("mtx",...), not Extension("pGroupCohomology.mtx",...). Running it, I get modules that I can import into sage: sage: from pGroupCohomology.mtx import MTX sage: M=MTX('some_data_file') This works, and I can do some computations. But apparently, the class MTX does not know where it lives: sage: M.__class__ Shouldn't it say ? The callable class responsible for unpickling MTX has the same identity crisis. sage: save(M,'Test') --------------------------------------------------------------------------- PicklingError Traceback (most recent call last) ... PicklingError: Can't pickle mtx.MTX_unpickle_class: it's not the same object as mtx.MTX_unpickle_class So, MTX_unpickle_class is not identical with itself! Removing the ext_package parameter and instead writing Extension("pGroupCohomology.mtx",...) did not help. What goes wrong here? Is there a way to tell a module that it belongs to a package? Best regards, Simon ---------------------------------------------------------------- This mail was sent through http://webmail.uni-jena.de From robertwb at math.washington.edu Fri May 1 20:43:20 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 1 May 2009 11:43:20 -0700 Subject: [Cython] Please vote: wraparound directive In-Reply-To: <49FAEAE9.7000908@student.matnat.uio.no> References: <49FAEAE9.7000908@student.matnat.uio.no> Message-ID: On May 1, 2009, at 5:28 AM, Dag Sverre Seljebotn wrote: > A new directive which allows Cython to assume that integers passed > to [] > are not negative. > > Currently buffers have the negative_indices option: > > object[int, negative_indices=False] arr = ... > > but this can be cumbersome and less flexible. The rationale for not > doing this in the first place was that it changes semantics during > normal running rather than under exceptional circumstances, but now > @cdivision has crossed that line anyway. Yes, I like this. I think directives are more explicit and easier to follow than extra data attached to types. > Also note that this could potentially apply to typed lists and > tuples as > well, if there's a benefit. Yes, there is a potential benefit here. - Robert From robertwb at math.washington.edu Fri May 1 20:46:19 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 1 May 2009 11:46:19 -0700 Subject: [Cython] Problems with creating a package of extension modules In-Reply-To: <1241198159.49fb2e4f76162@webmail.uni-jena.de> References: <1241198159.49fb2e4f76162@webmail.uni-jena.de> Message-ID: <922BECD5-7D73-46BC-A541-C76081EAC703@math.washington.edu> On May 1, 2009, at 10:15 AM, Simon King wrote: > Dear Cython-Dev, > > currently I try to create a Cython package comprising couple of > extension > modules for usage in Sage. It somehow works, but apparently the > extension > classes in the modules don't know where they belong to. > > In my setup.py, I have > setup( > name = "pGroupCohomology", > version = "1.0", > author = "Simon A. King", > author_email = "simon.king at uni-jena.de", > url = "http://users.minet.uni-jena.de/~king/cohomology/", > description = "Modular Cohomology Rings of Groups of Prime Power > Order", > packages=["pGroupCohomology"], > package_dir={"pGroupCohomology":"CohoSrc"}, > ext_package="pGroupCohomology", > ext_modules=[ > Extension("mtx", > sources = ["CohoSrc/mtx.pyx", > ...], > ...), > Extension(...)] > cmdclass = {'build_ext': build_ext} > ) > > I learned from some documentation that providing the optional argument > 'ext_package' should make the Extensions modules in that package, > so, it should > be Extension("mtx",...), not Extension("pGroupCohomology.mtx",...). Hmm... could you point me to that bit of documentation? I think it's wrong (or at least not implemented as explained), we do Extension ("package.module",...) all the time in sage and that seems to mix well with pickling. - Robert From robertwb at math.washington.edu Fri May 1 21:07:36 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 1 May 2009 12:07:36 -0700 Subject: [Cython] 0.11.2 release schedule? In-Reply-To: <49FB20B8.8090506@behnel.de> References: <49FAF0CC.5070108@student.matnat.uio.no> <49FB20B8.8090506@behnel.de> Message-ID: On May 1, 2009, at 9:18 AM, Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> Like I've been saying on some occasions, I'm going show off Cython >> 18th >> of May, and have had a tendency to scratch some itches rather than >> planning how to explain them... so I'd really like a release to >> happen >> before then. (Though I also have some fixes/rewrites which are not >> done >> yet which will go in next week.) >> >> If it helps I can play release manager this time and do the >> groundwork >> for making it build Sage etc. (especially if that gets us complex >> floats >> in time :-)). >> >> Would it be possible to e.g. do a full stop feature freeze next >> Friday >> 8th, and then a release the week after? > > I'm not currently working on cython-devel anyway, so I'm glad to > hear that > you volunteer as QA manager. :) Yes, that would be great if you would do the release managing-- definitely would make a mid-May release more feasible. > There are tons of open issues assigned to 0.11.2, though. I don't > expect > many more releases for 0.11.x (maybe a .3, but not necessarily a . > 4), so > they may have to get moved to 0.12. At least, there are no really > critical > issues. Some even seem to be so low priority that we should remove > their > milestone all together. Even some 'wishlist' bugs sound more > important than > some of the 0.11.2 tickets. A minor bug in 0.11 doesn't mean it needs > fixing in a 0.11.x release, especially if the fix is not trivial. In my view, non-critical tickets attached to unreleased milestones (especially 0.x.y ones) are fair game for moving around. Please go through the list and move everything you don't think we need by the release in two weeks. - Robert From simon.king at uni-jena.de Fri May 1 21:30:25 2009 From: simon.king at uni-jena.de (Simon King) Date: Fri, 1 May 2009 21:30:25 +0200 Subject: [Cython] Problems with creating a package of extension modules In-Reply-To: <922BECD5-7D73-46BC-A541-C76081EAC703@math.washington.edu> References: <1241198159.49fb2e4f76162@webmail.uni-jena.de> <922BECD5-7D73-46BC-A541-C76081EAC703@math.washington.edu> Message-ID: <1241206225.49fb4dd161f7c@webmail.uni-jena.de> Dear Robert, Zitat von Robert Bradshaw : > Hmm... could you point me to that bit of documentation? I think it's > wrong (or at least not implemented as explained), we do Extension > ("package.module",...) all the time in sage and that seems to mix > well with pickling. Admittedly it is not Cython-documentation, but Python: At http://docs.python.org/distutils/setupscript.html, they say: --------------------- If you have a number of extensions all in the same package (or all under the same base package), use the ext_package keyword argument to setup(). For example, setup(..., ext_package='pkg', ext_modules=[Extension('foo', ['foo.c']), Extension('subpkg.bar', ['bar.c'])], ) will compile foo.c to the extension pkg.foo, and bar.c to pkg.subpkg.bar. --------------------- Anyway. As I mentioned in my original post, I also tried it *without* ext_package but with explicit package names. Five minutes ago I found that (as a leftover of a previous built) I had mtx.so not only in the pGroupCohomology folder, but also one level up. Perhaps this was confusing for Sage. A propos confusing: It is still not clear to me what the role of the setup-options "name", "packages", and "ext_package". In order to learn (one of the main advantages of open source, imho), I was looking at the setup.py of the optional pil-package for sage. They have name="PIL", package_dir={"": "PIL"}, packages=[""], ext_modules = [Extension("_imaging", ["_imaging.c"])], # dummy and apparently it results in _imaging.so in the folder PIL. If I do, analogously, name='pGroupCohomology', packages=[""], package_dir={"":"CohoSrc"}, Extension("mtx", then my (dummy) __init__.py is copied at the top level (folder SAGE_LOCAL/lib/python2.5/site-packages/), and in the same folder mtx.so shows up -- I hope this did not destroy something important. Anyway, I wonder why the PIL package works. If I do name='pGroupCohomology' packages=[""], package_dir={"":"CohoSrc"}, Extension("pGroupCohomology.mtx", then still my __init__.py runs amok. mtx.so seems in the correct place, but of course I can't import mtx if there is no __init__.py in the package folder. If I do name='pGroupCohomology' packages=["pGroupCohomology"], package_dir={"pGroupCohomology":"CohoSrc"}, ext_package="pGroupCohomology", Extension("mtx", then everything seems in the right place, in accordance with the above-mentioned documentation. I can import mtx sage: from pGroupCohomology.mtx import MTX and construct some instance M, but again sage: M.__class__ and sage: save(M,'Test') --------------------------------------------------------------------------- PicklingError Traceback (most recent call last) ... PicklingError: Can't pickle mtx.MTX_unpickle_class: import of module mtx failed Note that the error is slightly different from my original post. Apparently, at that time, I had a copy of mtx.so in the site-packages top folder, so that the import of mtx worked, but was of course not the same as pGroupCohomology/mtx.so. Conclusion: - I have a folder pGroupCohomology in the site-packages, - it contains __init__.py, hence, gives rise to a package - it contains the module mtx.so - but mtx.so believes that it is not contained in a package. Why? Cheers, Simon ---------------------------------------------------------------- This mail was sent through http://webmail.uni-jena.de From dagss at student.matnat.uio.no Fri May 1 22:00:06 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 01 May 2009 22:00:06 +0200 Subject: [Cython] 0.11.2 release schedule? In-Reply-To: References: <49FAF0CC.5070108@student.matnat.uio.no> <49FB20B8.8090506@behnel.de> Message-ID: <49FB54C6.4020009@student.matnat.uio.no> Robert Bradshaw wrote: > On May 1, 2009, at 9:18 AM, Stefan Behnel wrote: > >> Dag Sverre Seljebotn wrote: >>> Like I've been saying on some occasions, I'm going show off Cython >>> 18th >>> of May, and have had a tendency to scratch some itches rather than >>> planning how to explain them... so I'd really like a release to >>> happen >>> before then. (Though I also have some fixes/rewrites which are not >>> done >>> yet which will go in next week.) >>> >>> If it helps I can play release manager this time and do the >>> groundwork >>> for making it build Sage etc. (especially if that gets us complex >>> floats >>> in time :-)). >>> >>> Would it be possible to e.g. do a full stop feature freeze next >>> Friday >>> 8th, and then a release the week after? >> I'm not currently working on cython-devel anyway, so I'm glad to >> hear that >> you volunteer as QA manager. :) > > Yes, that would be great if you would do the release managing-- > definitely would make a mid-May release more feasible. I'll give it a try. >> There are tons of open issues assigned to 0.11.2, though. I don't >> expect >> many more releases for 0.11.x (maybe a .3, but not necessarily a . >> 4), so >> they may have to get moved to 0.12. At least, there are no really >> critical >> issues. Some even seem to be so low priority that we should remove >> their >> milestone all together. Even some 'wishlist' bugs sound more >> important than >> some of the 0.11.2 tickets. A minor bug in 0.11 doesn't mean it needs >> fixing in a 0.11.x release, especially if the fix is not trivial. > > > In my view, non-critical tickets attached to unreleased milestones > (especially 0.x.y ones) are fair game for moving around. Please go > through the list and move everything you don't think we need by the > release in two weeks. Yes, I share this view -- I tend to think of it in the direction of time-based release cycles and no so much what it says in trac. However I like the split between 0.11.x and 0.12 because it helps get some kind of sense of what needs to go in which repository -- i.e. does it make things unstable, or require something which already went into unstable? I like filing new trivial bugs to the "-devel"-release for this reason. About "wishlist", I think that just a lot of those bugs should be moved out of "wishlist"; feature requests should go there IMO. Anyway, this is not very important, but I think I'll create a 0.11.3 rather than mix things into 0.12. -- Dag Sverre From robertwb at math.washington.edu Fri May 1 23:35:35 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 1 May 2009 14:35:35 -0700 Subject: [Cython] 0.11.2 release schedule? In-Reply-To: <49FB54C6.4020009@student.matnat.uio.no> References: <49FAF0CC.5070108@student.matnat.uio.no> <49FB20B8.8090506@behnel.de> <49FB54C6.4020009@student.matnat.uio.no> Message-ID: <6D3AEDC6-8BC7-4775-83C6-16ACA1E468B2@math.washington.edu> On May 1, 2009, at 1:00 PM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On May 1, 2009, at 9:18 AM, Stefan Behnel wrote: >> >> Yes, that would be great if you would do the release managing-- >> definitely would make a mid-May release more feasible. > > I'll give it a try. Great. Thanks. >>> There are tons of open issues assigned to 0.11.2, though. I don't >>> expect >>> many more releases for 0.11.x (maybe a .3, but not necessarily a . >>> 4), so >>> they may have to get moved to 0.12. At least, there are no really >>> critical >>> issues. Some even seem to be so low priority that we should remove >>> their >>> milestone all together. Even some 'wishlist' bugs sound more >>> important than >>> some of the 0.11.2 tickets. A minor bug in 0.11 doesn't mean it >>> needs >>> fixing in a 0.11.x release, especially if the fix is not trivial. >> >> >> In my view, non-critical tickets attached to unreleased milestones >> (especially 0.x.y ones) are fair game for moving around. Please go >> through the list and move everything you don't think we need by the >> release in two weeks. > > Yes, I share this view -- I tend to think of it in the direction of > time-based release cycles and no so much what it says in trac. > > However I like the split between 0.11.x and 0.12 because it helps get > some kind of sense of what needs to go in which repository -- i.e. > does > it make things unstable, or require something which already went into > unstable? > > I like filing new trivial bugs to the "-devel"-release for this > reason. Yep, I think this is the right place for them. Also, as part of a release, I often see if there's any trivial bugs to knock off. > About "wishlist", I think that just a lot of those bugs should be > moved > out of "wishlist"; feature requests should go there IMO. I've been using this as a "it's would be nice to have, but I don't see anyone working on it in the near term." > Anyway, this is not very important, but I think I'll create a 0.11.3 > rather than mix things into 0.12. Sound good. - Robert From jusa.sj at gmail.com Sat May 2 14:19:25 2009 From: jusa.sj at gmail.com (Juha Salo) Date: Sat, 2 May 2009 15:19:25 +0300 Subject: [Cython] How to produce Python 3.0.1 compatible DLLs? Message-ID: <20db9a7d0905020519r3daba354l5bb044b1996a3718@mail.gmail.com> Hi, I've been struggling with this for a day now. I'm trying to get the Primes example from the Cython tutorial to work with Python 3.0.1 on Windows Vista. So far I've managed to produce a primes.pyd file by executing the setup.py file with Python 2.6. The primes module works when I import it in Python 2.6 but importing the module in Python 3.0.1 gives me the following: "ImportError: Module use of python26.dll conflicts with this version of Python." I haven't found instructions for using cython with Python 3. The only things I've found are the brief mention of Python 3 support in the cython docs and a forum post explaining that cython modules should be build with Python 3 headers before they can be used with Python 3 but I have no idea how to do it. Could anyone offer me some help with this? If there's a tutorial somewhere on the net, I'd love to hear about it. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090502/b27ecd5d/attachment.htm From dagss at student.matnat.uio.no Sat May 2 15:34:46 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 02 May 2009 15:34:46 +0200 Subject: [Cython] How to produce Python 3.0.1 compatible DLLs? In-Reply-To: <20db9a7d0905020519r3daba354l5bb044b1996a3718@mail.gmail.com> References: <20db9a7d0905020519r3daba354l5bb044b1996a3718@mail.gmail.com> Message-ID: <49FC4BF6.50907@student.matnat.uio.no> Juha Salo wrote: > Hi, > > I've been struggling with this for a day now. I'm trying to get the > Primes example from the Cython tutorial to work with Python 3.0.1 on > Windows Vista. So far I've managed to produce a primes.pyd file by > executing the setup.py file with Python 2.6. The primes module works > when I import it in Python 2.6 but importing the module in Python 3.0.1 > gives me the following: > > "ImportError: Module use of python26.dll conflicts with this version of > Python." > > I haven't found instructions for using cython with Python 3. The only > things I've found are the brief mention of Python 3 support in the > cython docs and a forum post explaining that cython modules should be > build with Python 3 headers before they can be used with Python 3 but I > have no idea how to do it. Could anyone offer me some help with this? If > there's a tutorial somewhere on the net, I'd love to hear about it. You need to execute setup.py using Python 3.0 to build for Python 3. Details: The C file generated by Cython is the same for all Python versions, however it must be compiled with a different include path for the Python headers depending on the Python version. Executing setup.py with the right Python version takes care of this automatically. You can see this by Py3> from distutils import sysconfig Py3> sysconfig.get_python_inc() '/home/dagss/opt/python3.0/include/python3.0' which should be passed as an include path to the C compiler. Feel free to add this to the wiki in a place you would have found it if you get it to work. -- Dag Sverre From dagss at student.matnat.uio.no Sat May 2 15:41:43 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 02 May 2009 15:41:43 +0200 Subject: [Cython] How to produce Python 3.0.1 compatible DLLs? In-Reply-To: <49FC4BF6.50907@student.matnat.uio.no> References: <20db9a7d0905020519r3daba354l5bb044b1996a3718@mail.gmail.com> <49FC4BF6.50907@student.matnat.uio.no> Message-ID: <49FC4D97.5070500@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Juha Salo wrote: >> Hi, >> >> I've been struggling with this for a day now. I'm trying to get the >> Primes example from the Cython tutorial to work with Python 3.0.1 on >> Windows Vista. So far I've managed to produce a primes.pyd file by >> executing the setup.py file with Python 2.6. The primes module works >> when I import it in Python 2.6 but importing the module in Python 3.0.1 >> gives me the following: >> >> "ImportError: Module use of python26.dll conflicts with this version of >> Python." >> >> I haven't found instructions for using cython with Python 3. The only >> things I've found are the brief mention of Python 3 support in the >> cython docs and a forum post explaining that cython modules should be >> build with Python 3 headers before they can be used with Python 3 but I >> have no idea how to do it. Could anyone offer me some help with this? If >> there's a tutorial somewhere on the net, I'd love to hear about it. > > You need to execute setup.py using Python 3.0 to build for Python 3. Heh. Actually, that is likely not possible as the current release of Cython doesn't run on Python 3 (it will not too far off though). Anyway, what *should* work is a) executing Cython manually to get a C file, and b) executing the C compiler manually with the appropriate flags. You can figure out the flags to pass to the C compiler like this (I'm not an expert on this and there might be other options to retrieve which would be slightly more appropriate...) In [1]: from distutils import sysconfig In [7]: sysconfig.get_config_var("CFLAGS_FOR_SHARED") In [8]: sysconfig.get_config_var("CFLAGSFORSHARED") Out[8]: '-fPIC' In [9]: sysconfig.get_config_var("CFLAGS") Out[9]: '-fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes' In [10]: sysconfig.get_python_inc() Out[10]: '/usr/include/python2.5' Your output will be different on Windows. -- Dag Sverre From jusa.sj at gmail.com Sat May 2 15:59:52 2009 From: jusa.sj at gmail.com (Juha Salo) Date: Sat, 2 May 2009 16:59:52 +0300 Subject: [Cython] How to produce Python 3.0.1 compatible DLLs? In-Reply-To: <49FC4D97.5070500@student.matnat.uio.no> References: <20db9a7d0905020519r3daba354l5bb044b1996a3718@mail.gmail.com> <49FC4BF6.50907@student.matnat.uio.no> <49FC4D97.5070500@student.matnat.uio.no> Message-ID: <20db9a7d0905020659s1e5a3f0aibc4ad491b1dd560e@mail.gmail.com> Ok, thanks. I'll try that. 2009/5/2 Dag Sverre Seljebotn > Dag Sverre Seljebotn wrote: > > Juha Salo wrote: > >> Hi, > >> > >> I've been struggling with this for a day now. I'm trying to get the > >> Primes example from the Cython tutorial to work with Python 3.0.1 on > >> Windows Vista. So far I've managed to produce a primes.pyd file by > >> executing the setup.py file with Python 2.6. The primes module works > >> when I import it in Python 2.6 but importing the module in Python 3.0.1 > >> gives me the following: > >> > >> "ImportError: Module use of python26.dll conflicts with this version of > >> Python." > >> > >> I haven't found instructions for using cython with Python 3. The only > >> things I've found are the brief mention of Python 3 support in the > >> cython docs and a forum post explaining that cython modules should be > >> build with Python 3 headers before they can be used with Python 3 but I > >> have no idea how to do it. Could anyone offer me some help with this? If > >> there's a tutorial somewhere on the net, I'd love to hear about it. > > > > You need to execute setup.py using Python 3.0 to build for Python 3. > > Heh. Actually, that is likely not possible as the current release of > Cython doesn't run on Python 3 (it will not too far off though). > > Anyway, what *should* work is a) executing Cython manually to get a C > file, and b) executing the C compiler manually with the appropriate flags. > > You can figure out the flags to pass to the C compiler like this (I'm > not an expert on this and there might be other options to retrieve which > would be slightly more appropriate...) > > In [1]: from distutils import sysconfig > > In [7]: sysconfig.get_config_var("CFLAGS_FOR_SHARED") > > In [8]: sysconfig.get_config_var("CFLAGSFORSHARED") > Out[8]: '-fPIC' > > In [9]: sysconfig.get_config_var("CFLAGS") > Out[9]: '-fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall > -Wstrict-prototypes' > > In [10]: sysconfig.get_python_inc() > Out[10]: '/usr/include/python2.5' > > Your output will be different on Windows. > > -- > Dag Sverre > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090502/fc65d77d/attachment.htm From stefan at sun.ac.za Mon May 4 20:33:21 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 4 May 2009 20:33:21 +0200 Subject: [Cython] List type, link on Wiki front-page Message-ID: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> Hi all, Is there a way to tell Cython if you are using a homogeneous list? E.g. cdef list[int] my_list my_list.append(1) my_list.append(2) I don't have access to edit the Wiki, but the link to WritingFastPyrexCode should be http://wiki.sagemath.org/WritingFastPyrexCode Thanks, St?fan From robertwb at math.washington.edu Mon May 4 21:37:33 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 4 May 2009 12:37:33 -0700 Subject: [Cython] List type, link on Wiki front-page In-Reply-To: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> Message-ID: On May 4, 2009, at 11:33 AM, St?fan van der Walt wrote: > Hi all, > > Is there a way to tell Cython if you are using a homogeneous list? > E.g. > > cdef list[int] my_list > > my_list.append(1) > my_list.append(2) > No, there's not. It would be nice, but it would be hard to control what people stick in the list (e.g. if you pass it off to another routine). You might want to look at using NumPy arrays, which can store lists of actual c ints, etc. - Robert From kwmsmith at gmail.com Mon May 4 21:50:08 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Mon, 4 May 2009 14:50:08 -0500 Subject: [Cython] List type, link on Wiki front-page In-Reply-To: References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> Message-ID: On Mon, May 4, 2009 at 2:37 PM, Robert Bradshaw wrote: > On May 4, 2009, at 11:33 AM, St?fan van der Walt wrote: > >> Hi all, >> >> Is there a way to tell Cython if you are using a homogeneous list? >> E.g. >> >> cdef list[int] my_list >> >> my_list.append(1) >> my_list.append(2) >> > > No, there's not. It would be nice, but it would be hard to control > what people stick in the list (e.g. if you pass it off to another > routine). You might want to look at using NumPy arrays, which can > store lists of actual c ints, etc. In addition to NumPy arrays, you might consider using array.array which support 'append' and other list methods. They're basically homogeneous lists within the Python standard lib. A bit more flexible but not as powerful as NumPy arrays. Kurt From dalcinl at gmail.com Mon May 4 21:51:05 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 4 May 2009 16:51:05 -0300 Subject: [Cython] possible regression on cython-devel Message-ID: After hg update'd cython-devel, and now the code below generates a compiler error: cdef extern from "compat.h": pass ^ ------------------------------------------------------------ /u/dalcinl/Devel/petsc4py-dev/src/PETSc/PETSc.pyx:67:4: Expected an identifier, found 'pass' IIRC, that is a documented way to force inclusion of C headers... What this intentional? BTW... "tests/broken/cdefexternempty.pyx" has a tescase for this... For some reason, it never got promoted as a working test... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Mon May 4 22:22:00 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 04 May 2009 22:22:00 +0200 Subject: [Cython] possible regression on cython-devel In-Reply-To: References: Message-ID: <49FF4E68.3050100@behnel.de> Lisandro Dalcin wrote: > After hg update'd cython-devel, and now the code below generates a > compiler error: > > cdef extern from "compat.h": > pass > ^ > ------------------------------------------------------------ > > /u/dalcinl/Devel/petsc4py-dev/src/PETSc/PETSc.pyx:67:4: Expected an > identifier, found 'pass' Hmm, looks like I've broken that. http://hg.cython.org/cython-devel/rev/6515c3757571 The grammar of the cdef statement is pretty bizarre. It previously allowed you to write "cdef pass" for a "pass" statement, for example. It looks like the change in p_c_simple_base_type() is responsible here, although I suspect the real problem is in p_statement(). It parses a statement differently when the cdef_flag is set, also externally, i.e. when the statement itself is not preceded by a "cdef". This makes cdef extern from "compat.h": pass and cdef extern from "compat.h": int a possible, but also allows interpreting "pass" as type name. That shouldn't be allowed. > IIRC, that is a documented way to force inclusion of C headers... What > this intentional? BTW... "tests/broken/cdefexternempty.pyx" has a > tescase for this... For some reason, it never got promoted as a > working test... Yes, some of those are just there because they didn't make sense in the test runner, e.g. because the resulting C code wouldn't compile, or the module wouldn't load into CPython. Greg's test suite doesn't actually C-compile stuff, but only checks that the C code comes out as expected. That might work for a one-person-project, but definitely not for Cython. Could you try to come up with a compilable test case for this? I pushed an untested fix. http://hg.cython.org/cython-devel/rev/e6f5fc96fb42 Stefan From dagss at student.matnat.uio.no Mon May 4 22:26:16 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 04 May 2009 22:26:16 +0200 Subject: [Cython] List type, link on Wiki front-page In-Reply-To: References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> Message-ID: <49FF4F68.2000506@student.matnat.uio.no> Kurt Smith wrote: > On Mon, May 4, 2009 at 2:37 PM, Robert Bradshaw > wrote: >> On May 4, 2009, at 11:33 AM, St?fan van der Walt wrote: >> >>> Hi all, >>> >>> Is there a way to tell Cython if you are using a homogeneous list? >>> E.g. >>> >>> cdef list[int] my_list >>> >>> my_list.append(1) >>> my_list.append(2) >>> >> No, there's not. It would be nice, but it would be hard to control >> what people stick in the list (e.g. if you pass it off to another >> routine). You might want to look at using NumPy arrays, which can >> store lists of actual c ints, etc. > > In addition to NumPy arrays, you might consider using array.array > which support 'append' and other list methods. They're basically > homogeneous lists within the Python standard lib. A bit more flexible > but not as powerful as NumPy arrays. The question then becomes: Do we support "array.array[int]"? And I think the answer is that we actually might in Python 2.6+ due to PEP 3118 if you make a proper pxd for the array module, but as far as I know noone has tested it. We could have basic support for this in older Python versions as well, somebody just has to write the pxd. Using append at the same time would cause a lot of trouble though (but again, they might have changed append to raise an exception if a buffer is acquired in Py2.6+ -- I don't know). -- Dag Sverre From dalcinl at gmail.com Mon May 4 22:41:56 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 4 May 2009 17:41:56 -0300 Subject: [Cython] possible regression on cython-devel In-Reply-To: <49FF4E68.3050100@behnel.de> References: <49FF4E68.3050100@behnel.de> Message-ID: On Mon, May 4, 2009 at 5:22 PM, Stefan Behnel wrote: > > > Could you try to come up with a compilable test case for this? I pushed an > untested fix. > > http://hg.cython.org/cython-devel/rev/e6f5fc96fb42 > Moving the testcase from test/broken to test/compile and adding a empty header now works for me (as well as the code in my own projects)... $ hg status A tests/compile/cdefexternempty.pyx A tests/compile/cheese.h R tests/broken/cdefexternempty.pyx IIUC, these test are C-compiled... so this should be enough, right? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Mon May 4 22:42:13 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 04 May 2009 22:42:13 +0200 Subject: [Cython] main-embedding problem Message-ID: <49FF5325.7020705@behnel.de> Hi, I just tried to use the new main() embedding feature to compile a multi-module Python program. This doesn't seem to work as expected. What I would like to see is that when I say $ cython --embed somemain.py other1.py other2.py ... Cython should generate a .c file for each .py file and add a main() function only to the first module. This main() function should then register all other modules that were compiled at the same run, so that the resulting main program can become self-contained by simply compiling all .c files into a single executable. Since this is not how it currently works (instead, all .c files get their own main() function), we might want to at least disable the --embed option for multiple compilation in 0.11.2, so that people do not start relying on this. Stefan From stefan_ml at behnel.de Mon May 4 22:47:27 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 04 May 2009 22:47:27 +0200 Subject: [Cython] possible regression on cython-devel In-Reply-To: References: <49FF4E68.3050100@behnel.de> Message-ID: <49FF545F.4010006@behnel.de> Lisandro Dalcin wrote: > On Mon, May 4, 2009 at 5:22 PM, Stefan Behnel wrote: >> >> Could you try to come up with a compilable test case for this? I pushed an >> untested fix. >> >> http://hg.cython.org/cython-devel/rev/e6f5fc96fb42 > > Moving the testcase from test/broken to test/compile and adding a > empty header now works for me (as well as the code in my own > projects)... > > $ hg status > A tests/compile/cdefexternempty.pyx > A tests/compile/cheese.h > R tests/broken/cdefexternempty.pyx > > IIUC, these test are C-compiled... so this should be enough, right? Yes, please push this to cython-devel, nothing more to test here. Stefan From dalcinl at gmail.com Mon May 4 22:53:48 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 4 May 2009 17:53:48 -0300 Subject: [Cython] main-embedding problem In-Reply-To: <49FF5325.7020705@behnel.de> References: <49FF5325.7020705@behnel.de> Message-ID: On Mon, May 4, 2009 at 5:42 PM, Stefan Behnel wrote: > Hi, > > I just tried to use the new main() embedding feature to compile a > multi-module Python program. This doesn't seem to work as expected. What I > would like to see is that when I say > > ? ?$ cython --embed somemain.py other1.py other2.py ... > > Cython should generate a .c file for each .py file and add a main() > function only to the first module. This main() function should then > register all other modules that were compiled at the same run, so that the > resulting main program can become self-contained by simply compiling all .c > files into a single executable. Since this is not how it currently works > (instead, all .c files get their own main() function), we might want to at > least disable the --embed option for multiple compilation in 0.11.2, so > that people do not start relying on this. > Unfortunatelly, I have to agree, for different reasons... As an starter, the current code does not handle Py3K on Windows (because of the new wchar_t based API) -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Mon May 4 22:59:26 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 4 May 2009 17:59:26 -0300 Subject: [Cython] possible regression on cython-devel In-Reply-To: <49FF545F.4010006@behnel.de> References: <49FF4E68.3050100@behnel.de> <49FF545F.4010006@behnel.de> Message-ID: On Mon, May 4, 2009 at 5:47 PM, Stefan Behnel wrote: >> >> $ hg status >> A tests/compile/cdefexternempty.pyx >> A tests/compile/cheese.h >> R tests/broken/cdefexternempty.pyx >> >> IIUC, these test are C-compiled... so this should be enough, right? > > Yes, please push this to cython-devel, nothing more to test here. > Pushed. BTW, could I use the same trick (I mean, add header files as appropriate) to make some testcases pass the linking step on Windows? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From lodatom at gmail.com Mon May 4 23:18:26 2009 From: lodatom at gmail.com (Mark Lodato) Date: Mon, 4 May 2009 17:18:26 -0400 Subject: [Cython] main-embedding problem In-Reply-To: <49FF5325.7020705@behnel.de> References: <49FF5325.7020705@behnel.de> Message-ID: What about using something like my script? http://www.mail-archive.com/cython-dev at codespeak.net/msg04989.html I think this solution would be much cleaner - to create a *new* C file in addition to everything on the command line. That way you can compile all your modules to object files, and then either make them into shared objects or compile them with the extra C file into a standalone. Mark On Mon, May 4, 2009 at 4:42 PM, Stefan Behnel wrote: > Hi, > > I just tried to use the new main() embedding feature to compile a > multi-module Python program. This doesn't seem to work as expected. What I > would like to see is that when I say > > ? ?$ cython --embed somemain.py other1.py other2.py ... > > Cython should generate a .c file for each .py file and add a main() > function only to the first module. This main() function should then > register all other modules that were compiled at the same run, so that the > resulting main program can become self-contained by simply compiling all .c > files into a single executable. Since this is not how it currently works > (instead, all .c files get their own main() function), we might want to at > least disable the --embed option for multiple compilation in 0.11.2, so > that people do not start relying on this. > > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From stefan at sun.ac.za Mon May 4 23:33:56 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 4 May 2009 23:33:56 +0200 Subject: [Cython] List type, link on Wiki front-page In-Reply-To: <49FF4F68.2000506@student.matnat.uio.no> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> Message-ID: <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> 2009/5/4 Dag Sverre Seljebotn : > Using append at the same time would cause a lot of trouble though (but > again, they might have changed append to raise an exception if a buffer > is acquired in Py2.6+ -- I don't know). This is my main use-case: I need a homogenous container that automatically resizes when it is full. Maybe the performance impact of Python lists isn't even so bad -- I was only worried because I saw lots of yellow in my annotated pyx :) Cheers St?fan From robertwb at math.washington.edu Mon May 4 23:41:18 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 4 May 2009 14:41:18 -0700 Subject: [Cython] List type, link on Wiki front-page In-Reply-To: <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> Message-ID: <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> On May 4, 2009, at 2:33 PM, St?fan van der Walt wrote: > 2009/5/4 Dag Sverre Seljebotn : >> Using append at the same time would cause a lot of trouble though >> (but >> again, they might have changed append to raise an exception if a >> buffer >> is acquired in Py2.6+ -- I don't know). > > This is my main use-case: I need a homogenous container that > automatically resizes when it is full. Maybe the performance impact > of Python lists isn't even so bad -- I was only worried because I saw > lots of yellow in my annotated pyx :) Lists are actually rather fast. As a rule of thumb, yellow != slow, it just means that you're using the Python/C API (which is still often a good place to look for slowness). - Robert From stefan at sun.ac.za Mon May 4 23:43:00 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 4 May 2009 23:43:00 +0200 Subject: [Cython] Storing many, many instances Message-ID: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> Hi all, I have code that generates about 40000 instances of a certain cdef'd class. I would like to know: 1) If I define my cdef class in my_class.pyx, and then, in main.pyx do: from my_class import MyClass [.. instantiate many, many MyClass objects ..] Does this happen in C, or am I making Python calls along the way? 2) Is there a standard pattern for lowering the cost of instance creation? I think I may have to switch to structs instead of classes, but then I no longer have convenient methods or array members. Maybe I should rewrite my class to have many static methods that operate on a struct, and then collect the structs as data instead of the full instances. Any input much appreciated, Regards St?fan From dalcinl at gmail.com Mon May 4 23:45:32 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 4 May 2009 18:45:32 -0300 Subject: [Cython] main-embedding problem In-Reply-To: References: <49FF5325.7020705@behnel.de> Message-ID: On Mon, May 4, 2009 at 6:18 PM, Mark Lodato wrote: > What about using something like my script? > http://www.mail-archive.com/cython-dev at codespeak.net/msg04989.html > > I think this solution would be much cleaner - to create a *new* C file > in addition to everything on the command line. ?That way you can > compile all your modules to object files, and then either make them > into shared objects or compile them with the extra C file into a > standalone. > Looks good, however... * Not sure if the usage of PyMODINIT_FUNC is fine in the Windows case... (because of the __dclspec stuff) * In Py3K, "main()"" should be "wmain()" using wchar_t API. * Instead of using C stdlib calls for reporting failures and exiting, I would use Py_FatalError() Up to now, I'm still -1, just because: >>> import this ... Now is better than never. Although never is often better than *right* now. ... Anyway, the inittab extending stuff seems the way to go... > Mark > > On Mon, May 4, 2009 at 4:42 PM, Stefan Behnel wrote: >> Hi, >> >> I just tried to use the new main() embedding feature to compile a >> multi-module Python program. This doesn't seem to work as expected. What I >> would like to see is that when I say >> >> ? ?$ cython --embed somemain.py other1.py other2.py ... >> >> Cython should generate a .c file for each .py file and add a main() >> function only to the first module. This main() function should then >> register all other modules that were compiled at the same run, so that the >> resulting main program can become self-contained by simply compiling all .c >> files into a single executable. Since this is not how it currently works >> (instead, all .c files get their own main() function), we might want to at >> least disable the --embed option for multiple compilation in 0.11.2, so >> that people do not start relying on this. >> >> Stefan >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Mon May 4 23:49:15 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 4 May 2009 14:49:15 -0700 Subject: [Cython] main-embedding problem In-Reply-To: References: <49FF5325.7020705@behnel.de> Message-ID: On May 4, 2009, at 2:18 PM, Mark Lodato wrote: > What about using something like my script? > http://www.mail-archive.com/cython-dev at codespeak.net/msg04989.html > > I think this solution would be much cleaner - to create a *new* C file > in addition to everything on the command line. That way you can > compile all your modules to object files, and then either make them > into shared objects or compile them with the extra C file into a > standalone. If you want to include many files, this is probably a more obvious way to go. It's also nice because one can run the Cython compiler ahead of time and use the same output for .so files. > > On Mon, May 4, 2009 at 4:42 PM, Stefan Behnel > wrote: >> Hi, >> >> I just tried to use the new main() embedding feature to compile a >> multi-module Python program. This doesn't seem to work as >> expected. What I >> would like to see is that when I say >> >> $ cython --embed somemain.py other1.py other2.py ... >> >> Cython should generate a .c file for each .py file and add a main() >> function only to the first module. This main() function should then >> register all other modules that were compiled at the same run, so >> that the >> resulting main program can become self-contained by simply >> compiling all .c >> files into a single executable. Since this is not how it currently >> works >> (instead, all .c files get their own main() function), we might >> want to at >> least disable the --embed option for multiple compilation in >> 0.11.2, so >> that people do not start relying on this. Yes, lets disable it for multiple compilation for now at least. - Robert From robertwb at math.washington.edu Tue May 5 00:10:09 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 4 May 2009 15:10:09 -0700 Subject: [Cython] main-embedding problem In-Reply-To: References: <49FF5325.7020705@behnel.de> Message-ID: <7B8637C0-61E3-40D6-8BE4-04E635F78682@math.washington.edu> On May 4, 2009, at 1:53 PM, Lisandro Dalcin wrote: > On Mon, May 4, 2009 at 5:42 PM, Stefan Behnel > wrote: >> Hi, >> >> I just tried to use the new main() embedding feature to compile a >> multi-module Python program. This doesn't seem to work as >> expected. What I >> would like to see is that when I say >> >> $ cython --embed somemain.py other1.py other2.py ... >> >> Cython should generate a .c file for each .py file and add a main() >> function only to the first module. This main() function should then >> register all other modules that were compiled at the same run, so >> that the >> resulting main program can become self-contained by simply >> compiling all .c >> files into a single executable. Since this is not how it currently >> works >> (instead, all .c files get their own main() function), we might >> want to at >> least disable the --embed option for multiple compilation in >> 0.11.2, so >> that people do not start relying on this. >> > > Unfortunatelly, I have to agree, for different reasons... As an > starter, the current code does not handle Py3K on Windows (because of > the new wchar_t based API) This should be sufficient, right? http://hg.cython.org/cython-devel/rev/2e35d5caac86 - Robert From robertwb at math.washington.edu Tue May 5 00:15:10 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 4 May 2009 15:15:10 -0700 Subject: [Cython] Storing many, many instances In-Reply-To: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> Message-ID: <644465C5-4A7B-4FED-BED6-26D97C7C3CDF@math.washington.edu> On May 4, 2009, at 2:43 PM, St?fan van der Walt wrote: > Hi all, > > I have code that generates about 40000 instances of a certain > cdef'd class. > > I would like to know: > > 1) If I define my cdef class in my_class.pyx, and then, in main.pyx > do: > > from my_class import MyClass > > [.. instantiate many, many MyClass objects ..] > > Does this happen in C, or am I making Python calls along the way? You have to cimport to use C calls from another module. However, it will still call __init__ with Python semantics. The more arguments you pass (especially keyword arguments), the slower it'll get. > 2) Is there a standard pattern for lowering the cost of instance > creation? > > I think I may have to switch to structs instead of classes, but then I > no longer have convenient methods or array members. Maybe I should > rewrite my class to have many static methods that operate on a struct, > and then collect the structs as data instead of the full instances. > > Any input much appreciated, You could look into PY_NEW that we use in Sage. There was a thread on this earlier. There is inherently overhead in using Python objects, but it's usually worth the convenience. Another option would be to write a single class that provides access to a large set your objects. It all depends on your use case. - Robert From stefan at sun.ac.za Tue May 5 00:36:51 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 5 May 2009 00:36:51 +0200 Subject: [Cython] Storing many, many instances In-Reply-To: <644465C5-4A7B-4FED-BED6-26D97C7C3CDF@math.washington.edu> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <644465C5-4A7B-4FED-BED6-26D97C7C3CDF@math.washington.edu> Message-ID: <9457e7c80905041536r19777c3el8caf576b48a60845@mail.gmail.com> 2009/5/5 Robert Bradshaw : >> Does this happen in C, or am I making Python calls along the way? > > You have to cimport to use C calls from another module. However, it > will still call __init__ with Python semantics. The more arguments > you pass (especially keyword arguments), the slower it'll get. I feel really silly that I can't get this working, but Cython complains that MyClass.pxd was not found, and then says: Name 'MyClass' not declared in module 'mymodule.myclass' (but it is defined in myclass.pxd and importable in Python as myclass.MyClass). It doesn't look like sage.rings.Integer has a .pxd file, so I'm not sure why mine is failing. > You could look into PY_NEW that we use in Sage. There was a thread on > this earlier. There is inherently overhead in using Python objects, > but it's usually worth the convenience. Another option would be to > write a single class that provides access to a large set your > objects. It all depends on your use case. Thank you, I'll have a look! Cheers St?fan From greg.ewing at canterbury.ac.nz Tue May 5 02:18:51 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 05 May 2009 12:18:51 +1200 Subject: [Cython] possible regression on cython-devel In-Reply-To: <49FF4E68.3050100@behnel.de> References: <49FF4E68.3050100@behnel.de> Message-ID: <49FF85EB.6010401@canterbury.ac.nz> Stefan Behnel wrote: > The grammar of the cdef statement is pretty bizarre. Talking about "the cdef statement" is a bit misleading. The way I think of it in Pyrex, there isn't a single cdef statement. Rather, there is a cdef prefix that can be applied to declarations or suites of declarations to give them a different meaning. Since 'pass' is a valid declaration statement (it needs to be so that you can have empty suites of declarations), 'cdef pass' becomes a slightly fancier way of writing an empty declaration. It might look a bit odd, but it doesn't do any harm to allow it, IMO. Another thing you need to be careful not to break is cdef: where all the statements in the suite are treated as though they were prefixed with cdef (including pass statements!). -- Greg From greg.ewing at canterbury.ac.nz Tue May 5 02:48:00 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 05 May 2009 12:48:00 +1200 Subject: [Cython] List type, link on Wiki front-page In-Reply-To: <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> Message-ID: <49FF8CC0.7020907@canterbury.ac.nz> Robert Bradshaw wrote: > Lists are actually rather fast. As a rule of thumb, yellow != slow, > it just means that you're using the Python/C API (which is still > often a good place to look for slowness). What's likely to be more of a problem is converting between C and Python ints a lot, rather than the list appends themselves. -- Greg From brett.calcott at gmail.com Tue May 5 05:08:21 2009 From: brett.calcott at gmail.com (Brett Calcott) Date: Tue, 5 May 2009 13:08:21 +1000 Subject: [Cython] Include paths for pyximport Message-ID: Hi all, I'm using pyimport and writing some extensions to work numpy. To get things to happen magically the compiler needs to be able to find the header files. I've got it working on my mac, by adding a line to my .bashrc export C_INCLUDE_PATH=/Library/Python/2.5/site-packages/numpy/core/include I'd like to get it working on windows. I've added the equivalent path in my windows env (set INCLUDE=...), but I'm getting an error telling me it cannot find the "include/arrayobject.h". I've tried setting the right directory in the GUI too, but to no avail. Can anyone tell me the easy/right way to add include paths for pyximport on windows. Is there a better and portable way to do this? Thanks, Brett ps. pyximport rocks! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090505/318288a9/attachment.htm From hoytak at cs.ubc.ca Tue May 5 05:26:26 2009 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Mon, 4 May 2009 20:26:26 -0700 Subject: [Cython] List type, link on Wiki front-page In-Reply-To: <49FF8CC0.7020907@canterbury.ac.nz> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> Message-ID: <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> This is a common use case for me as well. What I've done (for the case of ints, doubles, etc.) is have a separate struct that emulates the vector<> in C++; there are associated functions that resize it, create it, etc. Like vector<>, it allocates more space than needed for new members, and uses malloc and realloc as appropriate. I then have a .d structure member, holding a pointer to the data, so in my code I just use v.d[i] to access the i'th element. However, this approach is far from as elegant as I'd like it to be. Out of curiosity, how difficult would it be to implement a new container type that emulates python's list but only holds one time and would work with both PyObject* and c types? Say vector[type] to distinguish it from list. Here would be my wishlist, given my current use cases, in order of desirability. 1. Very fast c level element access using []. Specifically, as fast as C array access if it holds a C data type and as fast as list if holding a python type. 2. All memory management issues handled automatically. 3. List-like methods -- append, pop, del, etc. -- that avoid any python overhead except possible ref counting stuff. 4. Ability to convert to and from python list in interfacing with python code (doesn't need to be the fastest, but would have in-bulk type checking / C--PyObject* conversions where needed). 5. Slicing that works like list's slicing. I suspect there are tons of hidden issues behind each of these, and I also understand it would be good to do get other issues out of the way in this. I am quite busy currently due to classes, research, and an upcoming prelim exam, but would be happy to put some time into this if someone can mentor me. I've been meaning to get more into cython development for a long time but am always too busy, so I might as well start sometime. --Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From dagss at student.matnat.uio.no Tue May 5 06:35:59 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 5 May 2009 06:35:59 +0200 (CEST) Subject: [Cython] Storing many, many instances In-Reply-To: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> Message-ID: <8f5db410ca5009dd2aec60dce437cd23.squirrel@webmail.uio.no> St?fan van der Walt wrote: > I think I may have to switch to structs instead of classes, but then I > no longer have convenient methods or array members. Maybe I should > rewrite my class to have many static methods that operate on a struct, > and then collect the structs as data instead of the full instances. Technically it would not be too difficult to allow cdef struct MyStruct: void foo(self): # self is pure MyStruct, perhaps stack-allocated in Cython, and transform mystructinstance.foo() into __mangled_MyStruct_foo(mystructinstance) The question is whether we want it or not, perhaps it would just be confusing to provide OO-like syntax on something that's neither reference counted nor polymorphic. Also the line has to be drawn somewhere before we rewrite C++ :-) So I don't know what I think about it myself, but it's not too difficult technically. Dag Sverre From dagss at student.matnat.uio.no Tue May 5 06:39:26 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 5 May 2009 06:39:26 +0200 (CEST) Subject: [Cython] Storing many, many instances In-Reply-To: <8f5db410ca5009dd2aec60dce437cd23.squirrel@webmail.uio.no> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <8f5db410ca5009dd2aec60dce437cd23.squirrel@webmail.uio.no> Message-ID: <7c940a5d130e97a4d739efc6f3343bff.squirrel@webmail.uio.no> > St?fan van der Walt wrote: >> I think I may have to switch to structs instead of classes, but then I >> no longer have convenient methods or array members. Maybe I should >> rewrite my class to have many static methods that operate on a struct, >> and then collect the structs as data instead of the full instances. > > Technically it would not be too difficult to allow > > cdef struct MyStruct: > void foo(self): > # self is pure MyStruct, perhaps stack-allocated > > in Cython, and transform > > mystructinstance.foo() > > into > > __mangled_MyStruct_foo(mystructinstance) Sorry, it should be MyStruct* as self, and one would transform a call into __mangled_MyStruct_foo(&mystructinstance) Dag Sverre From dagss at student.matnat.uio.no Tue May 5 06:52:36 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 5 May 2009 06:52:36 +0200 (CEST) Subject: [Cython] List type, link on Wiki front-page In-Reply-To: <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> Message-ID: Hoyt Koepke wrote: > This is a common use case for me as well. What I've done (for the > case of ints, doubles, etc.) is have a separate struct that emulates > the vector<> in C++; there are associated functions that resize it, > create it, etc. Like vector<>, it allocates more space than needed > for new members, and uses malloc and realloc as appropriate. I then > have a .d structure member, holding a pointer to the data, so in my > code I just use v.d[i] to access the i'th element. However, this > approach is far from as elegant as I'd like it to be. > > Out of curiosity, how difficult would it be to implement a new > container type that emulates python's list but only holds one time and > would work with both PyObject* and c types? Say vector[type] to > distinguish it from list. Here would be my wishlist, given my current > use cases, in order of desirability. I don't think such a type should be built into Cython itself, but should be written as just another class in pyx and pxd files (which can be shipped with Cython perhaps). However, it seems to require templates/generics in Cython. So that would be the place to start working. Unfortunately that would probably be some work (on the order of a GSoC I think?), but I think that would be a much better approach than hard-coding in a brand new vector type in Cython. (This is when I usually state that I think it is better if we can properly wrap STL though, which will hopefully be done something about in summer, since the list of data structures which it would be nice to have doesn't at all stop with vectors.) Dag Sverre From hoytak at cs.ubc.ca Tue May 5 07:17:57 2009 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Mon, 4 May 2009 22:17:57 -0700 Subject: [Cython] List type, link on Wiki front-page In-Reply-To: References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> Message-ID: <4db580fd0905042217v30ce34dal615cf55bdc586b73@mail.gmail.com> >> Out of curiosity, how difficult would it be to implement a new >> container type that emulates python's list but only holds one time and >> would work with both PyObject* and c types? ?Say vector[type] to >> distinguish it from list. ?Here would be my wishlist, given my current >> use cases, in order of desirability. > > I don't think such a type should be built into Cython itself, but should > be written as just another class in pyx and pxd files (which can be > shipped with Cython perhaps). Yes, I agree, that makes sense. Being able to do these operations for general stuff, beyond just something like vector[int], would be well worth implementing, at least in my book. I would use them frequently. > However, it seems to require templates/generics in Cython. So that would > be the place to start working. Unfortunately that would probably be some > work (on the order of a GSoC I think?), but I think that would be a much > better approach than hard-coding in a brand new vector type in Cython. Yes, I see; well I would love to help, but grad school has a way of sucking up one's free time, and there are probably better ways for me to learn the cython code base. > (This is when I usually state that I think it is better if we can properly > wrap STL though, which will hopefully be done something about in summer, > since the list of data structures which it would be nice to have doesn't > at all stop with vectors.) I totally agree; STL-type maps of int to double, etc. would be awesome and used often. --Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From stefan_ml at behnel.de Tue May 5 09:50:22 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 May 2009 09:50:22 +0200 Subject: [Cython] template types In-Reply-To: References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> Message-ID: <49FFEFBE.6010800@behnel.de> Dag Sverre Seljebotn wrote: > Hoyt Koepke wrote: >> Out of curiosity, how difficult would it be to implement a new >> container type that emulates python's list but only holds one time and >> would work with both PyObject* and c types? Say vector[type] to >> distinguish it from list. Here would be my wishlist, given my current >> use cases, in order of desirability. > > I don't think such a type should be built into Cython itself, but should > be written as just another class in pyx and pxd files (which can be > shipped with Cython perhaps). Agreed. > However, it seems to require templates/generics in Cython. So that would > be the place to start working. Unfortunately that would probably be some > work (on the order of a GSoC I think?), but I think that would be a much > better approach than hard-coding in a brand new vector type in Cython. > > (This is when I usually state that I think it is better if we can properly > wrap STL though, which will hopefully be done something about in summer, > since the list of data structures which it would be nice to have doesn't > at all stop with vectors.) Wrapping the STL sounds like a different feature. I don't think we should *require* C++ here. Writing type templates into plain C would be a lot more useful. It would be more work, sure, but it shouldn't be too hard. It would mainly require smarter type checks (i.e. MyType != MyType) and code duplication in the generated C code. Maybe it's not even that hard to hack that into the current code. Imagine you could write cdef class MyType(object): # type T gets defined in the class scope to make the parser happy cdef T* my_attribute def __init__(self, T value): self.my_attribute = malloc(sizeof(T)*1000) self.my_attribute[100] = value def T do_work(self, T input): return input + self.my_attribute[100] o = MyType(20) # first occurrence creates/registers the type o.do_work(18) ctypedef MyType MyIntType # reuses the type and provides a name cdef MyIntType ot = MyIntType(25) print ot.do_work(19) ctypedef MyType MyDType # creates and defines a new type I'm not sure if a common base class would make sense here, something that would just implement everything that doesn't depend on T. But maybe that's overkill already. And most stuff *will* We might be able to implement this completely textually, i.e. the parser would declare the specialised types on the fly (as it does now for a ctypedef or extension type, i.e. new C type names). The resulting types would really be named "MyType" internally, and we'd just do some name mangling on the way out. It would just be about as 'powerful' as Java's compile-time generics (minus being limited to types), and not more. That doesn't sound like a GSoC to me. Even template functions would be out of scope at the beginning (as they would require knowing the type of a value/name that you pass, which isn't available at parse time, i.e. doesn't fit the current type declaration scheme). Stefan From dagss at student.matnat.uio.no Tue May 5 10:22:54 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 5 May 2009 10:22:54 +0200 (CEST) Subject: [Cython] template types In-Reply-To: <49FFEFBE.6010800@behnel.de> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> Message-ID: <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> However, it seems to require templates/generics in Cython. So that would >> be the place to start working. Unfortunately that would probably be some >> work (on the order of a GSoC I think?), but I think that would be a much >> better approach than hard-coding in a brand new vector type in Cython. >> >> (This is when I usually state that I think it is better if we can >> properly >> wrap STL though, which will hopefully be done something about in summer, >> since the list of data structures which it would be nice to have doesn't >> at all stop with vectors.) > > Wrapping the STL sounds like a different feature. I don't think we should > *require* C++ here. Writing type templates into plain C would be a lot > more > useful. It would be more work, sure, but it shouldn't be too hard. It > would > mainly require smarter type checks (i.e. MyType != MyType) > and > code duplication in the generated C code. Maybe it's not even that hard to > hack that into the current code. > > Imagine you could write > > cdef class MyType(object): > # type T gets defined in the class scope to make the parser happy > cdef T* my_attribute > def __init__(self, T value): > self.my_attribute = malloc(sizeof(T)*1000) > self.my_attribute[100] = value > def T do_work(self, T input): > return input + self.my_attribute[100] > > o = MyType(20) # first occurrence creates/registers the type > o.do_work(18) > > ctypedef MyType MyIntType # reuses the type and provides a name > > cdef MyIntType ot = MyIntType(25) > print ot.do_work(19) > > ctypedef MyType MyDType # creates and defines a new type > > I'm not sure if a common base class would make sense here, something that > would just implement everything that doesn't depend on T. But maybe that's > overkill already. And most stuff *will* Hmm. Yes, you are right, there's lots of stuff which doesn't need to be supported at first. Somebody still has to do it though :-) Before the C++ syntax sticks though I'd like highlight some benefits of the [] syntax: - It is legal Python to do "vector[cython.int]()". (Which leaves the way open for providing a meta-class doing run-time type-checking of arguments and return values, which would be the Python run-time compatability layer in the shadow module. That's far off and perhaps too little useful, but I like leaving the way open.). - Guido once used the [] syntax for something similar (which never became anything) [1] - Using < and > makes the parser more complicated, while [] must already be parsed. (in C++ there's the mess of having to write "vector >", note the space, because there would be some problem with the >> operator and the parser otherwise). [1] http://www.artima.com/weblogs/viewpost.jsp?thread=86641 I even think one should use [] for wrapping STL personally. Dag Sverre From stefan_ml at behnel.de Tue May 5 13:07:35 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 5 May 2009 13:07:35 +0200 (CEST) Subject: [Cython] template types In-Reply-To: <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> Message-ID: <75369f0f2977f46a3f1c2a8be7952925.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > Before the C++ syntax sticks I actually think it makes sense. In declarations, the parameter can appear only behind type names, so parsing them would be trivial - except for cases where the type name appears in an expression: isinstance(x, MyType) type_ref = MyType vs. type_ref = SomeValue < something > 5 Not sure if type_ref = MyType[int] makes this any simpler - it may at least simplify the parser, although there might be further ambiguities with buffers, array sizes and indexing. I also considered keyword arguments to the type (like Py3's "metaclass"), but didn't find an obvious way to keep that information in the type name when referring to it later on. cdef class MyType(SomeParent, templates=[T,V]): ... cdef MyType(T=int, V=list) my_instance my_instance = MyType(1,2,3,4, T=int, V=list) ... does that look readable? > I'd like highlight some benefits of the [] syntax: > > - It is legal Python to do "vector[cython.int]()". (Which leaves the way > open for providing a meta-class doing run-time type-checking of arguments > and return values, which would be the Python run-time compatability layer > in the shadow module. That's far off and perhaps too little useful, but I > like leaving the way open.). IMHO not a major goal, though. Plus, the type definition itself would still not match Python's syntax in that case, so the gain is rather marginal. > - Guido once used the [] syntax for something similar (which never became > anything) [1] > [1] http://www.artima.com/weblogs/viewpost.jsp?thread=86641 Yes, although I wouldn't count on that. If you read the link at the top of that post (i.e. what became of it), you'll see that that syntax is basically gone before being examined. > - Using < and > makes the parser more complicated, while [] must already > be parsed. (in C++ there's the mess of having to write "vector >>", note the space, because there would be some problem with the >> > operator and the parser otherwise). That's sick. Looks like a scanner problem, though, not a parser problem. The parser would know that '>>' doesn't make any sense at all at that position (ok, it's C++, but let's assume a sane kind of 'sense' here). Stefan From hoytak at cs.ubc.ca Tue May 5 17:40:40 2009 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Tue, 5 May 2009 08:40:40 -0700 Subject: [Cython] template types In-Reply-To: <75369f0f2977f46a3f1c2a8be7952925.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <75369f0f2977f46a3f1c2a8be7952925.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <4db580fd0905050840q56f1a25cx5f9496fbb0d1475d@mail.gmail.com> On Tue, May 5, 2009 at 4:07 AM, Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> Before the C++ syntax sticks > > I actually think it makes sense. In declarations, the parameter can appear > only behind type names, so parsing them would be trivial - except for > cases where the type name appears in an expression: > > ? ?isinstance(x, MyType) > ? ?type_ref = MyType ?vs. ?type_ref = SomeValue < something > 5 > > Not sure if > > ? ?type_ref = MyType[int] > > makes this any simpler - it may at least simplify the parser, although > there might be further ambiguities with buffers, array sizes and indexing. One other thing that you all might want to consider is D's syntax for templates (http://www.digitalmars.com/d/2.0/templates-revisited.html). As much as I am familiar with C++ template stuff, I think it is much better thought out than C++. I know it's completely tangential to python/C, but might be worth considering. --Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From dagss at student.matnat.uio.no Tue May 5 17:47:59 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 05 May 2009 17:47:59 +0200 Subject: [Cython] template types In-Reply-To: <75369f0f2977f46a3f1c2a8be7952925.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <75369f0f2977f46a3f1c2a8be7952925.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <4A005FAF.4080107@student.matnat.uio.no> Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> Before the C++ syntax sticks > > I actually think it makes sense. In declarations, the parameter can appear Well it certainly has the advantage that C++ and Java knows what it means right away. > only behind type names, so parsing them would be trivial - except for > cases where the type name appears in an expression: > > isinstance(x, MyType) > type_ref = MyType vs. type_ref = SomeValue < something > 5 In general, foo(baz) can mean two things, and you don't know until you know what kind of expression foo is. This makes it virtually guaranteed that if Python gets generic interfaces somehow [1] this will not be the syntax of choice, as Guido hates these kinds of things even if it is technically possible to seperate it, possible to require that you write a<(b>a) or similar. Also in Python (and in Cython with metaprogramming/macro support!), one could envisage doing def f(): if wierd: return FooTemplate else: return BarTemplate f()(baz) # operators or template instantiation? won't know 'till runtime [1] Well, I suppose 3rd party libraries can already add interfaces, including templated ones, using metaclasses > Not sure if > > type_ref = MyType[int] > > makes this any simpler - it may at least simplify the parser, although > there might be further ambiguities with buffers, array sizes and indexing. Well, there's already ambiguities here (and still a regression compared to before I mixed things up with buffers) making the current parser approach suboptimal. One day when I get time I think I'll just introduce BracketOperatorNode, and leave deciding what it does to later transforms. One thing I really dislike in current Cython code is that the parser has to know whether things are types or not in a given context. (Although I think Robert fixed at least some of this.) BTW one can look at a template as a collection of types, and template instantiation as fetching the right type in the collection. > I also considered keyword arguments to the type (like Py3's "metaclass"), > but didn't find an obvious way to keep that information in the type name > when referring to it later on. > > cdef class MyType(SomeParent, templates=[T,V]): > ... > > cdef MyType(T=int, V=list) my_instance > my_instance = MyType(1,2,3,4, T=int, V=list) > > ... does that look readable? Hmm. It's an alternative to consider although I must admit my guts prefer MyType to this. > That's sick. Looks like a scanner problem, though, not a parser problem. > The parser would know that '>>' doesn't make any sense at all at that > position (ok, it's C++, but let's assume a sane kind of 'sense' here). Well it is the same kind of problem as above; basically you don't know whether a> d; is doing arithmetic or instantiating a variable until you can fully resolve what kind of identifiers a, b, c and d are. Just makes everything more complicated, and it is harder to parse for humans as well. The difference with a[b](c) is that you at least know how to structure the parse tree and the operator precedence just by looking at it without knowing anything about a, b and c. -- Dag Sverre From seb.binet at gmail.com Tue May 5 17:48:11 2009 From: seb.binet at gmail.com (Sebastien Binet) Date: Tue, 5 May 2009 17:48:11 +0200 Subject: [Cython] template types In-Reply-To: <4db580fd0905050840q56f1a25cx5f9496fbb0d1475d@mail.gmail.com> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <75369f0f2977f46a3f1c2a8be7952925.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4db580fd0905050840q56f1a25cx5f9496fbb0d1475d@mail.gmail.com> Message-ID: <200905051748.11645.binet@cern.ch> hi, lurking_mode=off On Tuesday 05 May 2009 17:40:40 Hoyt Koepke wrote: > On Tue, May 5, 2009 at 4:07 AM, Stefan Behnel wrote: > > Dag Sverre Seljebotn wrote: > >> Before the C++ syntax sticks > > > > I actually think it makes sense. In declarations, the parameter can > > appear only behind type names, so parsing them would be trivial - except > > for cases where the type name appears in an expression: > > > > isinstance(x, MyType) > > type_ref = MyType vs. type_ref = SomeValue < something > 5 > > > > Not sure if > > > > type_ref = MyType[int] > > > > makes this any simpler - it may at least simplify the parser, although > > there might be further ambiguities with buffers, array sizes and > > indexing. > > One other thing that you all might want to consider is D's syntax for > templates (http://www.digitalmars.com/d/2.0/templates-revisited.html). > As much as I am familiar with C++ template stuff, I think it is much > better thought out than C++. I know it's completely tangential to > python/C, but might be worth considering. it is a bit disturbing though: auto foo = Bar !(3, Baz, 42); auto bar = Foo !(z); it is hard to not think of these statements as negating something (probably especially the second one ?) I wish D had chosen @ or $ instead of !. cheers, sebastien. -- ######################################### # Dr. Sebastien Binet # Laboratoire de l'Accelerateur Lineaire # Universite Paris-Sud XI # Batiment 200 # 91898 Orsay ######################################### From stefan at sun.ac.za Tue May 5 18:03:46 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 5 May 2009 18:03:46 +0200 Subject: [Cython] Storing many, many instances In-Reply-To: <9457e7c80905041536r19777c3el8caf576b48a60845@mail.gmail.com> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <644465C5-4A7B-4FED-BED6-26D97C7C3CDF@math.washington.edu> <9457e7c80905041536r19777c3el8caf576b48a60845@mail.gmail.com> Message-ID: <9457e7c80905050903u63e65f69j2c0ec0cbe3f9f98f@mail.gmail.com> 2009/5/5 St?fan van der Walt : >> You have to cimport to use C calls from another module. However, it >> will still call __init__ with Python semantics. The more arguments >> you pass (especially keyword arguments), the slower it'll get. > > I feel really silly that I can't get this working, but Cython > complains that MyClass.pxd was not found, and then says: > > Name 'MyClass' not declared in module 'mymodule.myclass' (but it is > defined in myclass.pxd and importable in Python as myclass.MyClass). > > It doesn't look like sage.rings.Integer has a .pxd file, so I'm not > sure why mine is failing. In case someone comes across this thread: Sage *does* have .pxd files to the integer class, otherwise this wouldn't work. The procedure is: 1. Define your class in myclass.pyx: cdef class MyClass: cdef test(self): print "hello!" 2. Define a pxd "header" file: cdef class MyClass: cdef test(self) 3. Use *both* cimport and normal import: from myclass cimport MyClass as MyClassT from myclass import MyClass cdef MyClassT m = MyClass() m.test() Hope that saves someone some time along the way. Cheers St?fan From stefan_ml at behnel.de Tue May 5 18:57:39 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 May 2009 18:57:39 +0200 Subject: [Cython] Storing many, many instances In-Reply-To: <9457e7c80905050903u63e65f69j2c0ec0cbe3f9f98f@mail.gmail.com> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <644465C5-4A7B-4FED-BED6-26D97C7C3CDF@math.washington.edu> <9457e7c80905041536r19777c3el8caf576b48a60845@mail.gmail.com> <9457e7c80905050903u63e65f69j2c0ec0cbe3f9f98f@mail.gmail.com> Message-ID: <4A007003.7090708@behnel.de> St?fan van der Walt wrote: > 1. Define your class in myclass.pyx: > > cdef class MyClass: > cdef test(self): > print "hello!" > > 2. Define a pxd "header" file: > > cdef class MyClass: > cdef test(self) > > 3. Use *both* cimport and normal import: > > from myclass cimport MyClass as MyClassT > from myclass import MyClass > > cdef MyClassT m = MyClass() Note that this will generate unnecessary type checks in the code as Cython cannot know that both types are identical. There should be no reason to import both, just stick with the cimport. Stefan From hoytak at cs.ubc.ca Tue May 5 20:42:06 2009 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Tue, 5 May 2009 11:42:06 -0700 Subject: [Cython] template types In-Reply-To: <200905051748.11645.binet@cern.ch> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <75369f0f2977f46a3f1c2a8be7952925.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4db580fd0905050840q56f1a25cx5f9496fbb0d1475d@mail.gmail.com> <200905051748.11645.binet@cern.ch> Message-ID: <4db580fd0905051142r69076d82re4f9301c54bb51d@mail.gmail.com> > it is a bit disturbing though: > ?auto foo = Bar !(3, Baz, 42); > ?auto bar = Foo !(z); > > it is hard to not think of these statements as negating something (probably > especially the second one ?) > ?I wish D had chosen @ or $ instead of !. Perhaps, but isn't it true that every language has slightly confusing stuff when first considered? Whitespace in python, < > for template stuff in C++, [] notation for buffers in cython (I've gotten used to it fine, and don't mind it now, but it did make me to do a few double-takes at first). What I like about D's syntax is that it's easy to pick out !( ) as type arguments to a function, class, method, etc. and ( ) for the regular arguments. What I have found to be nice (and granted, I haven't done much D programming recently) is that type parameters and regular parameters follow similar syntax, by intention. I've found that templating stuff in D is an amazingly nice experience (C++ just stinks in comparison) as once I've got my head around what can and can't be done, the syntax is very intuitive. Just my 2 cents, but I'm far from being experienced enough to deside what would work best here. -- Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From stefan_ml at behnel.de Tue May 5 20:48:26 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 May 2009 20:48:26 +0200 Subject: [Cython] template types In-Reply-To: <4db580fd0905050840q56f1a25cx5f9496fbb0d1475d@mail.gmail.com> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <75369f0f2977f46a3f1c2a8be7952925.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4db580fd0905050840q56f1a25cx5f9496fbb0d1475d@mail.gmail.com> Message-ID: <4A0089FA.6060403@behnel.de> Hoyt Koepke wrote: > On Tue, May 5, 2009 at 4:07 AM, Stefan Behnel wrote: >> Dag Sverre Seljebotn wrote: >>> Before the C++ syntax sticks >> I actually think it makes sense. In declarations, the parameter can appear >> only behind type names, so parsing them would be trivial - except for >> cases where the type name appears in an expression: >> >> isinstance(x, MyType) >> type_ref = MyType vs. type_ref = SomeValue < something > 5 >> >> Not sure if >> >> type_ref = MyType[int] >> >> makes this any simpler - it may at least simplify the parser, although >> there might be further ambiguities with buffers, array sizes and indexing. > > One other thing that you all might want to consider is D's syntax for > templates (http://www.digitalmars.com/d/2.0/templates-revisited.html). > As much as I am familiar with C++ template stuff, I think it is much > better thought out than C++. I know it's completely tangential to > python/C, but might be worth considering. Hmm, the '!' might make sense in the D (and maybe C) corner, but for Python and Cython? Stefan From stefan_ml at behnel.de Tue May 5 10:08:10 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 May 2009 10:08:10 +0200 Subject: [Cython] possible regression on cython-devel In-Reply-To: <49FF85EB.6010401@canterbury.ac.nz> References: <49FF4E68.3050100@behnel.de> <49FF85EB.6010401@canterbury.ac.nz> Message-ID: <49FFF3EA.6050606@behnel.de> Greg Ewing wrote: > Talking about "the cdef statement" is a bit misleading. > The way I think of it in Pyrex, there isn't a single > cdef statement. Rather, there is a cdef prefix that > can be applied to declarations or suites of declarations > to give them a different meaning. That makes it clearer, although the parser function is still called "p_cdef_statement". > Since 'pass' is a valid declaration statement (it needs > to be so that you can have empty suites of declarations), > 'cdef pass' becomes a slightly fancier way of writing > an empty declaration. It might look a bit odd, but it > doesn't do any harm to allow it, IMO. It's perfectly sufficient to allow "pass" in a cdef block, rather than in any cdef context. That's what I did in my last fix. > Another thing you need to be careful not to break is > > cdef: > > > where all the statements in the suite are treated as > though they were prefixed with cdef (including pass > statements!). Thanks, I just checked, and there is a test case for that (tests/compile/ia_cdefblock.pyx). I still think it makes a lot more sense to support cdef: pass than to allow cdef pass explicitly in the grammar. Stefan From stefan_ml at behnel.de Tue May 5 13:53:34 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 May 2009 13:53:34 +0200 Subject: [Cython] Storing many, many instances In-Reply-To: <7c940a5d130e97a4d739efc6f3343bff.squirrel@webmail.uio.no> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <8f5db410ca5009dd2aec60dce437cd23.squirrel@webmail.uio.no> <7c940a5d130e97a4d739efc6f3343bff.squirrel@webmail.uio.no> Message-ID: <4A0028BE.9030903@behnel.de> Dag Sverre Seljebotn wrote: >> St?fan van der Walt wrote: >>> I think I may have to switch to structs instead of classes, but then I >>> no longer have convenient methods or array members. Maybe I should >>> rewrite my class to have many static methods that operate on a struct, >>> and then collect the structs as data instead of the full instances. >> Technically it would not be too difficult to allow >> >> cdef struct MyStruct: >> void foo(self): >> # self is pure MyStruct, perhaps stack-allocated >> >> in Cython, and transform >> >> mystructinstance.foo() >> >> into >> >> __mangled_MyStruct_foo(mystructinstance) > > Sorry, it should be MyStruct* as self, and one would transform a call into > > __mangled_MyStruct_foo(&mystructinstance) I wouldn't mind. We have a lot of convenience stuff in structs already (e.g. dict coercion and keyword arguments), so adding methods with the above semantics wouldn't hurt. It's not rewriting C++ either, as long as it makes sense in Cython as a language. Stefan From seb.binet at gmail.com Tue May 5 20:59:23 2009 From: seb.binet at gmail.com (Sebastien Binet) Date: Tue, 5 May 2009 20:59:23 +0200 Subject: [Cython] template types In-Reply-To: <4db580fd0905051142r69076d82re4f9301c54bb51d@mail.gmail.com> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <200905051748.11645.binet@cern.ch> <4db580fd0905051142r69076d82re4f9301c54bb51d@mail.gmail.com> Message-ID: <200905052059.24065.binet@cern.ch> On Tuesday 05 May 2009 20:42:06 Hoyt Koepke wrote: > > it is a bit disturbing though: > > auto foo = Bar !(3, Baz, 42); > > auto bar = Foo !(z); > > > > it is hard to not think of these statements as negating something > > (probably especially the second one ?) > > I wish D had chosen @ or $ instead of !. > > Perhaps, but isn't it true that every language has slightly confusing > stuff when first considered? Whitespace in python, < > for template > stuff in C++, [] notation for buffers in cython (I've gotten used to > it fine, and don't mind it now, but it did make me to do a few > double-takes at first). What I like about D's syntax is that it's > easy to pick out !( ) as type arguments to a function, class, method, > etc. and ( ) for the regular arguments. What I have found to be nice > (and granted, I haven't done much D programming recently) is that type > parameters and regular parameters follow similar syntax, by intention. > I've found that templating stuff in D is an amazingly nice experience > (C++ just stinks in comparison) as once I've got my head around what > can and can't be done, the syntax is very intuitive. in fact, I am not really sure I am buying that this: auto foo = Foo !(T1, T2, T3)(arg1, arg2); is really cleaner/clearer than: auto foo = Foo (T1, T2, T3)(arg1, arg2); that is what we do in python/PyROOT (the main python-to-C/C++ module we use in my field (High Energy Physics)): import ROOT cls =ROOT.std.vector(double) # instantiates std::vector data = cls() assert hasattr(data, 'size') my_data = ROOT.std.vector('MyNS::MyClass')() my_data.push_back(...) assert isinstance(my_data[0], ROOT.MyNS.MyClass) ROOT.std.vector being just a function returning classes, by type or by (C++) class name... cheers, sebastien. -- ######################################### # Dr. Sebastien Binet # Laboratoire de l'Accelerateur Lineaire # Universite Paris-Sud XI # Batiment 200 # 91898 Orsay ######################################### From stefan_ml at behnel.de Tue May 5 21:10:18 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 May 2009 21:10:18 +0200 Subject: [Cython] template types In-Reply-To: <200905052059.24065.binet@cern.ch> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <200905051748.11645.binet@cern.ch> <4db580fd0905051142r69076d82re4f9301c54bb51d@mail.gmail.com> <200905052059.24065.binet@cern.ch> Message-ID: <4A008F1A.2070400@behnel.de> Sebastien Binet wrote: > On Tuesday 05 May 2009 20:42:06 Hoyt Koepke wrote: >>> it is a bit disturbing though: >>> auto foo = Bar !(3, Baz, 42); >>> auto bar = Foo !(z); >>> >>> it is hard to not think of these statements as negating something >>> (probably especially the second one ?) >>> I wish D had chosen @ or $ instead of !. >> Perhaps, but isn't it true that every language has slightly confusing >> stuff when first considered? Whitespace in python, < > for template >> stuff in C++, [] notation for buffers in cython (I've gotten used to >> it fine, and don't mind it now, but it did make me to do a few >> double-takes at first). What I like about D's syntax is that it's >> easy to pick out !( ) as type arguments to a function, class, method, >> etc. and ( ) for the regular arguments. What I have found to be nice >> (and granted, I haven't done much D programming recently) is that type >> parameters and regular parameters follow similar syntax, by intention. >> I've found that templating stuff in D is an amazingly nice experience >> (C++ just stinks in comparison) as once I've got my head around what >> can and can't be done, the syntax is very intuitive. > > in fact, I am not really sure I am buying that this: > auto foo = Foo !(T1, T2, T3)(arg1, arg2); > > is really cleaner/clearer than: > auto foo = Foo (T1, T2, T3)(arg1, arg2); > > that is what we do in python/PyROOT (the main python-to-C/C++ module we use in > my field (High Energy Physics)): > import ROOT > cls =ROOT.std.vector(double) # instantiates std::vector > data = cls() > assert hasattr(data, 'size') > > my_data = ROOT.std.vector('MyNS::MyClass')() > my_data.push_back(...) > assert isinstance(my_data[0], ROOT.MyNS.MyClass) > > ROOT.std.vector being just a function returning classes, by type or by (C++) > class name... That looks a bit like partial(), and similar to what I 'proposed' earlier with the keyword arguments. It might make sense to stay in Python-syntax completely here. Maybe there are more ideas on how to achieve this nicely. Stefan From stefan_ml at behnel.de Tue May 5 21:23:57 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 May 2009 21:23:57 +0200 Subject: [Cython] template types In-Reply-To: <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> Message-ID: <4A00924D.9070202@behnel.de> Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Imagine you could write >> >> cdef class MyType(object): >> # type T gets defined in the class scope to make the parser happy >> cdef T* my_attribute >> def __init__(self, T value): >> self.my_attribute = malloc(sizeof(T)*1000) >> self.my_attribute[100] = value >> def T do_work(self, T input): >> return input + self.my_attribute[100] >> >> o = MyType(20) # first occurrence creates/registers the type >> o.do_work(18) >> >> ctypedef MyType MyIntType # reuses the type and provides a name >> >> cdef MyIntType ot = MyIntType(25) >> print ot.do_work(19) >> >> ctypedef MyType MyDType # creates and defines a new type >> >> I'm not sure if a common base class would make sense here, something that >> would just implement everything that doesn't depend on T. But maybe that's >> overkill already. And most stuff *will* > > - Using < and > makes the parser more complicated, while [] must already > be parsed. (in C++ there's the mess of having to write "vector >> ", note the space, because there would be some problem with the >> > operator and the parser otherwise). What about avoiding this problem all-together? We could require a ctypedef in the beginning, which would alleviate the need for a syntax that works well in expressions. Stefan From dagss at student.matnat.uio.no Tue May 5 21:59:19 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 05 May 2009 21:59:19 +0200 Subject: [Cython] template types In-Reply-To: <4A00924D.9070202@behnel.de> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> Message-ID: <4A009A97.50608@student.matnat.uio.no> Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> Stefan Behnel wrote: >>> Imagine you could write >>> >>> cdef class MyType(object): >>> # type T gets defined in the class scope to make the parser happy >>> cdef T* my_attribute >>> def __init__(self, T value): >>> self.my_attribute = malloc(sizeof(T)*1000) >>> self.my_attribute[100] = value >>> def T do_work(self, T input): >>> return input + self.my_attribute[100] >>> >>> o = MyType(20) # first occurrence creates/registers the type >>> o.do_work(18) >>> >>> ctypedef MyType MyIntType # reuses the type and provides a name >>> >>> cdef MyIntType ot = MyIntType(25) >>> print ot.do_work(19) >>> >>> ctypedef MyType MyDType # creates and defines a new type >>> >>> I'm not sure if a common base class would make sense here, something that >>> would just implement everything that doesn't depend on T. But maybe that's >>> overkill already. And most stuff *will* >> - Using < and > makes the parser more complicated, while [] must already >> be parsed. (in C++ there's the mess of having to write "vector >>> ", note the space, because there would be some problem with the >> >> operator and the parser otherwise). > > What about avoiding this problem all-together? We could require a ctypedef > in the beginning, which would alleviate the need for a syntax that works > well in expressions. I would feel it was inconvenient to use a seperate line to define IntVector, then remember what name one gave it etc., rather than just express what one wants. Templates quickly becomes second-nature in C++ and Java and this seems to work against that. OTOH, this brings up another interesting question: What happens with exporting templates to Python-land? If there was indeed something very like typedef, but which also made the templated type available Python-side... I guess IntVector = Vector would currently achieve that without any further magic though. With [] or () or somesuch one could export some wrapper type for "Vector" and look up the Vector[cython.int] type from Python-space, however one must know compile-time which templates to instantiate for Python access anyway. The template types to always instantiate and export to Python could perhaps be embedded in the template declaration syntax. (I'd say that's a valid usecase eventually, but of course not something to support at first. But it /does/ impact the chosen syntax quite heavily as <> is ruled straight out.) -- Dag Sverre From dalcinl at gmail.com Tue May 5 22:41:55 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 5 May 2009 17:41:55 -0300 Subject: [Cython] template types In-Reply-To: <4A009A97.50608@student.matnat.uio.no> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> Message-ID: On Tue, May 5, 2009 at 4:59 PM, Dag Sverre Seljebotn wrote: > > OTOH, this brings up another interesting question: What happens with > exporting templates to Python-land? > SWIG supports templates and exposing templates to Python side... but having to choose a name for the "instantiation" always bothered me... At some point, I've started to use some hackery via a registry and a custom class abusing of __getitem__() to support on Python side: myinst = MyClass[sometype](args) At first, I was tempted to use "MyClass(sometype)(args)", but almost immediately I ruled it out because that was too much confusing. So, I'm +1 on "[]" for templates... > > (I'd say that's a valid usecase eventually, but of course not something > to support at first. > Why not? IMHO, that would be easiest part to implement from the whole beast (assuming that a registry and a companion type implementing dispatching via __getitem__() would be enough). > But it /does/ impact the chosen syntax quite > heavily as <> is ruled straight out.) > Indeed. That's the reason I support "[]" for templates over the C++/D way. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Tue May 5 22:54:02 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 5 May 2009 17:54:02 -0300 Subject: [Cython] Storing many, many instances In-Reply-To: <4A0028BE.9030903@behnel.de> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <8f5db410ca5009dd2aec60dce437cd23.squirrel@webmail.uio.no> <7c940a5d130e97a4d739efc6f3343bff.squirrel@webmail.uio.no> <4A0028BE.9030903@behnel.de> Message-ID: On Tue, May 5, 2009 at 8:53 AM, Stefan Behnel wrote: > > Dag Sverre Seljebotn wrote: >>> St?fan van der Walt wrote: >>>> I think I may have to switch to structs instead of classes, but then I >>>> no longer have convenient methods or array members. ?Maybe I should >>>> rewrite my class to have many static methods that operate on a struct, >>>> and then collect the structs as data instead of the full instances. >>> Technically it would not be too difficult to allow >>> >>> cdef struct MyStruct: >>> ? ? void foo(self): >>> ? ? ? ? # self is pure MyStruct, perhaps stack-allocated >>> > > I wouldn't mind. We have a lot of convenience stuff in structs already > (e.g. dict coercion and keyword arguments), so adding methods with the > above semantics wouldn't hurt. > I agree... > It's not rewriting C++ either, as long as it makes sense in Cython as a > language. > But them someone will ask for polimorphism, and then the struct instances could have a pointer to a struct-specific vtable, and then you are more or less rewriting C++. And that would not bother me, currently cdef methods do work as a sort C++, right?. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Tue May 5 23:11:10 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 May 2009 23:11:10 +0200 Subject: [Cython] template types In-Reply-To: <4A009A97.50608@student.matnat.uio.no> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> Message-ID: <4A00AB6E.90909@behnel.de> Dag Sverre Seljebotn wrote: > OTOH, this brings up another interesting question: What happens with > exporting templates to Python-land? > > If there was indeed something very like typedef, but which also made the > templated type available Python-side... > > I guess > > IntVector = Vector > > would currently achieve that without any further magic though. Sure. > With [] or () or somesuch one could export some wrapper type for > "Vector" and look up the Vector[cython.int] type from Python-space, > however one must know compile-time which templates to instantiate for > Python access anyway. The template types to always instantiate and > export to Python could perhaps be embedded in the template declaration > syntax. Why? If you want to provide a type to Python space, you can just instantiate it somewhere in your program, e.g. by adding it to a name->type mapping. No magic whatsoever involved. > (I'd say that's a valid usecase eventually, but of course not something > to support at first. But it /does/ impact the chosen syntax quite > heavily as <> is ruled straight out.) The syntax would be Cython-only, no need to move it into Python. Stefan From stefan_ml at behnel.de Tue May 5 23:18:59 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 May 2009 23:18:59 +0200 Subject: [Cython] template types In-Reply-To: References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> Message-ID: <4A00AD43.1050709@behnel.de> Lisandro Dalcin wrote: > At some point, I've started to use some hackery via a registry and a > custom class abusing of __getitem__() to support on Python side: > > myinst = MyClass[sometype](args) Actually, now that I see this - it actually makes sense to think of a parametrised type as a meta-type that maps types to types, so a mapping syntax makes sense here. If you have more than one type parameter, it'd look like this: myinst = MyClass[(sometype, someothertype)](*args) and the concrete type would basically be 'looked up' at compile time. It doesn't directly map to an 'obvious' declaration syntax, but I guess cdef MyType(object) [(sometype, someothertype)]: ... works well enough. So I'm +1 on this, too. Stefan From stefan_ml at behnel.de Tue May 5 23:28:17 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 May 2009 23:28:17 +0200 Subject: [Cython] template types In-Reply-To: <4A00AD43.1050709@behnel.de> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> Message-ID: <4A00AF71.7010103@behnel.de> Stefan Behnel schrieb: > Lisandro Dalcin wrote: >> At some point, I've started to use some hackery via a registry and a >> custom class abusing of __getitem__() to support on Python side: >> >> myinst = MyClass[sometype](args) > > Actually, now that I see this - it actually makes sense to think of a > parametrised type as a meta-type that maps types to types, so a mapping > syntax makes sense here. If you have more than one type parameter, it'd > look like this: > > myinst = MyClass[(sometype, someothertype)](*args) > > and the concrete type would basically be 'looked up' at compile time. > > It doesn't directly map to an 'obvious' declaration syntax, but I guess > > cdef MyType(object) [(sometype, someothertype)]: > ... > > works well enough. I meant cdef MyType(object) [(T,V)]: here, although I now noticed that there is already the "private type" syntax: cdef public class _Document [ type LxmlDocumentType, object LxmlDocument ]: ... so this won't work straight away either... Stefan From dalcinl at gmail.com Tue May 5 23:28:30 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 5 May 2009 18:28:30 -0300 Subject: [Cython] template types In-Reply-To: <4A00AD43.1050709@behnel.de> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> Message-ID: On Tue, May 5, 2009 at 6:18 PM, Stefan Behnel wrote: > > Lisandro Dalcin wrote: >> At some point, I've started to use some hackery via a registry and a >> custom class abusing of __getitem__() to support on Python side: >> >> myinst = MyClass[sometype](args) > > Actually, now that I see this - it actually makes sense to think of a > parametrised type as a meta-type that maps types to types, so a mapping > syntax makes sense here. Nice! You just added another point for me to like "[]", the mapping concept fits well IMHO... > If you have more than one type parameter, it'd > look like this: > > ? ?myinst = MyClass[(sometype, someothertype)](*args) > Why the tuple? This should work just fine: myinst = MyClass[sometype, someothertype](*args) thoug the MyClass.__getitem__() will receive a tuple actually... (sugar added to Python slice syntax long ago following request of Numeric) > and the concrete type would basically be 'looked up' at compile time. > > It doesn't directly map to an 'obvious' declaration syntax, but I guess > > ? ?cdef MyType(object) [(sometype, someothertype)]: > ? ? ?... I would prefer: cdef MyType[sometype, someothertype](object): -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Tue May 5 23:33:16 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 May 2009 23:33:16 +0200 Subject: [Cython] template types In-Reply-To: References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> Message-ID: <4A00B09C.4030606@behnel.de> Lisandro Dalcin wrote: > On Tue, May 5, 2009 at 6:18 PM, Stefan Behnel wrote: >> Lisandro Dalcin wrote: >>> At some point, I've started to use some hackery via a registry and a >>> custom class abusing of __getitem__() to support on Python side: >>> >>> myinst = MyClass[sometype](args) > >> Actually, now that I see this - it actually makes sense to think of a >> parametrised type as a meta-type that maps types to types, so a mapping >> syntax makes sense here. > > Nice! You just added another point for me to like "[]", the mapping > concept fits well IMHO... > >> If you have more than one type parameter, it'd >> look like this: >> >> myinst = MyClass[(sometype, someothertype)](*args) >> > > Why the tuple? This should work just fine: > > myinst = MyClass[sometype, someothertype](*args) > > thoug the MyClass.__getitem__() will receive a tuple actually... > (sugar added to Python slice syntax long ago following request of > Numeric) > > >> and the concrete type would basically be 'looked up' at compile time. >> >> It doesn't directly map to an 'obvious' declaration syntax, but I guess >> >> cdef MyType(object) [(sometype, someothertype)]: >> ... > > I would prefer: > > cdef MyType[sometype, someothertype](object): ... which also wouldn't be ambiguous with public types. Looks CEP-able to me. Stefan From dalcinl at gmail.com Tue May 5 23:33:56 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 5 May 2009 18:33:56 -0300 Subject: [Cython] template types In-Reply-To: <4A00AF71.7010103@behnel.de> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> <4A00AF71.7010103@behnel.de> Message-ID: On Tue, May 5, 2009 at 6:28 PM, Stefan Behnel wrote: > I meant > > ? ?cdef MyType(object) [(T,V)]: > > here, although I now noticed that there is already the "private type" syntax: > > cdef public class _Document [ type LxmlDocumentType, object LxmlDocument ]: > ? ?... > > so this won't work straight away either... > But this would work anyway, right? Perhaps you could even support {} for the [type ..., object ...] part... cdef public class MyBaseClass[T,V] [type MyBaseClass_Type, object MyBaseClassObject ]: cdef public class MyClass[T,V](MyBaseClass)[ type MyClass_Type, object MyClassObject ]: -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Wed May 6 00:04:51 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 5 May 2009 19:04:51 -0300 Subject: [Cython] main-embedding problem In-Reply-To: <7B8637C0-61E3-40D6-8BE4-04E635F78682@math.washington.edu> References: <49FF5325.7020705@behnel.de> <7B8637C0-61E3-40D6-8BE4-04E635F78682@math.washington.edu> Message-ID: On Mon, May 4, 2009 at 7:10 PM, Robert Bradshaw wrote: >> Unfortunatelly, I have to agree, for different reasons... As an >> starter, the current code does not handle Py3K on Windows (because of >> the new wchar_t based API) > > This should be sufficient, right? > > http://hg.cython.org/cython-devel/rev/2e35d5caac86 > No, unfortunately it is not enough... I didn't tell you the whole story... Py3K has a wchar_t-based API for all platforms. On Windows, the fix is easy, you just use wmain(). On POSIX, not so easy... you have to use char-based, traditional main(), and use mbrtowc() to convent all the argument one by one... IMHO, core CPython is doing it wrong here... See the nightmare in py3k/Modules/python.c ... IMHO, Python should provide a both char-based and wchar_t-based API's, and internally implement the char-based ones as wrappers, managing the conversion char->wchat_t and then call the wchar_t-based API's. If this is not done, any one trying to embed python have to start converting arguments (and managing the tmp mem blocks), and have to cope with the apparent brokenness of mbrtowc() on some platforms... I've already had to implement all that crap in mpi4py, see http://code.google.com/p/mpi4py/source/browse/trunk/src/python.c (borrow code from Py3_Main_GetArgs() if you like it). Fortunately, that file is not frequently required for end-users, but I'm not even sure if it works on the many Unix-like systems out-there (current supercomputer systems hardly have python3 available for end-users) I do not have the energy to go to Python-Dev to discuss all this... Do any of you have it :-)? Stefan? Greg? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Wed May 6 07:15:42 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 06 May 2009 07:15:42 +0200 Subject: [Cython] template types In-Reply-To: References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> <4A00AF71.7010103@behnel.de> Message-ID: <4A011CFE.7070006@behnel.de> Lisandro Dalcin wrote: > On Tue, May 5, 2009 at 6:28 PM, Stefan Behnel wrote: >> I meant >> >> cdef MyType(object) [(T,V)]: >> >> here, although I now noticed that there is already the "private type" syntax: >> >> cdef public class _Document [ type LxmlDocumentType, object LxmlDocument ]: >> ... >> >> so this won't work straight away either... >> > > But this would work anyway, right? Not that easily. You could have both [] parts next to each other, or just any one of them. So the parser would fail if you decided to call your template variable "type", which isn't that a bad name in this context. > Perhaps you could even support {} > for the [type ..., object ...] part... > > cdef public class MyBaseClass[T,V] [type MyBaseClass_Type, object > MyBaseClassObject ]: > > cdef public class MyClass[T,V](MyBaseClass)[ type MyClass_Type, object > MyClassObject ]: Well, if we get to change *that* syntax (which I'd do straight away if it wasn't for compatibility - it really hurt my eyes the first time I saw it), then this seems a lot better: cdef public class MyClass[T,V](MyBaseClass, public_ctype="MyClass_Type", public_cobject="MyClassObject"): The parser can easily know and check all valid keywords here, so there'd be no change to the rest of the compiler. And using strings as arguments makes it a lot clearer (at least to me) that this is only configuration that is supposed to end up in the generated C code, and not meant for real use in the program. Stefan From dagss at student.matnat.uio.no Wed May 6 07:17:54 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 6 May 2009 07:17:54 +0200 (CEST) Subject: [Cython] template types In-Reply-To: <4A00AD43.1050709@behnel.de> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> Message-ID: <899fcc678c0606bb152339924fa2c459.squirrel@webmail.uio.no> Stefan Behnel wrote: > Lisandro Dalcin wrote: >> At some point, I've started to use some hackery via a registry and a >> custom class abusing of __getitem__() to support on Python side: >> >> myinst = MyClass[sometype](args) > > Actually, now that I see this - it actually makes sense to think of a > parametrised type as a meta-type that maps types to types, so a mapping > syntax makes sense here. If you have more than one type parameter, it'd > look like this: > > myinst = MyClass[(sometype, someothertype)](*args) Well, in Python there's no way to seperate that from myinst = MyClass[sometype, someothertype](*args) which is syntax candy for the same. Of course we can in Cython, but why? Dag Sverre From stefan_ml at behnel.de Wed May 6 07:21:21 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 06 May 2009 07:21:21 +0200 Subject: [Cython] template types In-Reply-To: <899fcc678c0606bb152339924fa2c459.squirrel@webmail.uio.no> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> <899fcc678c0606bb152339924fa2c459.squirrel@webmail.uio.no> Message-ID: <4A011E51.5080609@behnel.de> Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Lisandro Dalcin wrote: >>> At some point, I've started to use some hackery via a registry and a >>> custom class abusing of __getitem__() to support on Python side: >>> >>> myinst = MyClass[sometype](args) >> Actually, now that I see this - it actually makes sense to think of a >> parametrised type as a meta-type that maps types to types, so a mapping >> syntax makes sense here. If you have more than one type parameter, it'd >> look like this: >> >> myinst = MyClass[(sometype, someothertype)](*args) > > Well, in Python there's no way to seperate that from > > myinst = MyClass[sometype, someothertype](*args) > > which is syntax candy for the same. Fine with me. Stefan From dagss at student.matnat.uio.no Wed May 6 07:23:39 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 6 May 2009 07:23:39 +0200 (CEST) Subject: [Cython] template types In-Reply-To: <4A011CFE.7070006@behnel.de> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> <4A00AF71.7010103@behnel.de> <4A011CFE.7070006@behnel.de> Message-ID: <3439af420a85daac11ba6238c254c18f.squirrel@webmail.uio.no> Stefan Behnel wrote: > Lisandro Dalcin wrote: >> On Tue, May 5, 2009 at 6:28 PM, Stefan Behnel wrote: >>> I meant >>> >>> cdef MyType(object) [(T,V)]: >>> >>> here, although I now noticed that there is already the "private type" >>> syntax: >>> >>> cdef public class _Document [ type LxmlDocumentType, object >>> LxmlDocument ]: >>> ... >>> >>> so this won't work straight away either... >>> >> >> But this would work anyway, right? > > Not that easily. You could have both [] parts next to each other, or just > any one of them. So the parser would fail if you decided to call your > template variable "type", which isn't that a bad name in this context. How about cdef class A: T, V = cython.template_args(type, type) ? (The reason for the arguments: In C++ you can have stuff like ints in template arguments as well, might as well leave the possibility open even if only passing type is allowed for now). This even puts T and V in the proper scope for @cython.locals declarations, e.g. this passes through (and could even be made to do the checks) in pure Python: @cython.cdef class A: T, V = template_args(type, type) @cython.locals(a=T, b=v) def foo(self, a, b): ... Dag Sverre From robertwb at math.washington.edu Wed May 6 09:02:52 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 00:02:52 -0700 Subject: [Cython] template types In-Reply-To: <3439af420a85daac11ba6238c254c18f.squirrel@webmail.uio.no> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> <4A00AF71.7010103@behnel.de> <4A011CFE.7070006@behnel.de> <3439af420a85daac11ba6238c254c18f.squirrel@webmail.uio.no> Message-ID: <8B75D598-FF6D-4986-8E3D-0167BA5CE778@math.washington.edu> On May 5, 2009, at 10:23 PM, Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Lisandro Dalcin wrote: >>> On Tue, May 5, 2009 at 6:28 PM, Stefan Behnel wrote: >>>> I meant >>>> >>>> cdef MyType(object) [(T,V)]: >>>> >>>> here, although I now noticed that there is already the "private >>>> type" >>>> syntax: >>>> >>>> cdef public class _Document [ type LxmlDocumentType, object >>>> LxmlDocument ]: >>>> ... >>>> >>>> so this won't work straight away either... >>>> >>> >>> But this would work anyway, right? >> >> Not that easily. You could have both [] parts next to each other, >> or just >> any one of them. So the parser would fail if you decided to call your >> template variable "type", which isn't that a bad name in this >> context. > > How about > > cdef class A: > T, V = cython.template_args(type, type) > > ? I prefer the MyType[T,V] syntax. It encapsulates the "meta-type" idea, and also the same syntax can be used both in declaring the type and instantiating the type. > (The reason for the arguments: In C++ you can have stuff like ints in > template arguments as well, might as well leave the possibility > open even > if only passing type is allowed for now). > > This even puts T and V in the proper scope for @cython.locals > declarations, e.g. this passes through (and could even be made to > do the > checks) in pure Python: > > @cython.cdef > class A: > T, V = template_args(type, type) > @cython.locals(a=T, b=v) > def foo(self, a, b): ... I'd rather see the pure Python mode decorate the class than have a magic method, but I don't think decorators get applied soon enough to inject variables into the scope. - Robert From stefan_ml at behnel.de Wed May 6 09:40:51 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 6 May 2009 09:40:51 +0200 (CEST) Subject: [Cython] template types In-Reply-To: <8B75D598-FF6D-4986-8E3D-0167BA5CE778@math.washington.edu> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> <4A00AF71.7010103@behnel.de> <4A011CFE.7070006@behnel.de> <3439af420a85daac11ba6238c254c18f.squirrel@webmail.uio.no> <8B75D598-FF6D-4986-8E3D-0167BA5CE778@math.washington.edu> Message-ID: <1409138872f9c0172f9467e4d3380dfe.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Robert Bradshaw wrote: > On May 5, 2009, at 10:23 PM, Dag Sverre Seljebotn wrote: >> How about >> >> cdef class A: >> T, V = cython.template_args(type, type) > > I prefer the MyType[T,V] syntax. It encapsulates the "meta-type" > idea, and also the same syntax can be used both in declaring the type > and instantiating the type. +1 >> (The reason for the arguments: In C++ you can have stuff like ints in >> template arguments as well, might as well leave the possibility >> open even >> if only passing type is allowed for now). >> >> This even puts T and V in the proper scope for @cython.locals >> declarations, e.g. this passes through (and could even be made to >> do the checks) in pure Python: >> >> @cython.cdef >> class A: >> T, V = template_args(type, type) >> @cython.locals(a=T, b=v) >> def foo(self, a, b): ... > > I'd rather see the pure Python mode decorate the class than have a > magic method, but I don't think decorators get applied soon enough to > inject variables into the scope. I wouldn't care so much about that for now. Whatever declaration we choose, it's easily added when the feature itself is there, and I think parametrised types are such an advanced feature that users can well wait for it to become available in pure Python mode. The priorities should be to make the feature simple and well readable in Cython code, and then 'somehow' available in Python code. Stefan From dagss at student.matnat.uio.no Wed May 6 09:42:53 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 May 2009 09:42:53 +0200 Subject: [Cython] template types In-Reply-To: <8B75D598-FF6D-4986-8E3D-0167BA5CE778@math.washington.edu> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> <4A00AF71.7010103@behnel.de> <4A011CFE.7070006@behnel.de> <3439af420a85daac11ba6238c254c18f.squirrel@webmail.uio.no> <8B75D598-FF6D-4986-8E3D-0167BA5CE778@math.washington.edu> Message-ID: <4A013F7D.9070101@student.matnat.uio.no> Robert Bradshaw wrote: > On May 5, 2009, at 10:23 PM, Dag Sverre Seljebotn wrote: > >> Stefan Behnel wrote: >>> Lisandro Dalcin wrote: >>>> On Tue, May 5, 2009 at 6:28 PM, Stefan Behnel wrote: >>>>> I meant >>>>> >>>>> cdef MyType(object) [(T,V)]: >>>>> >>>>> here, although I now noticed that there is already the "private >>>>> type" >>>>> syntax: >>>>> >>>>> cdef public class _Document [ type LxmlDocumentType, object >>>>> LxmlDocument ]: >>>>> ... >>>>> >>>>> so this won't work straight away either... >>>>> >>>> But this would work anyway, right? >>> Not that easily. You could have both [] parts next to each other, >>> or just >>> any one of them. So the parser would fail if you decided to call your >>> template variable "type", which isn't that a bad name in this >>> context. >> How about >> >> cdef class A: >> T, V = cython.template_args(type, type) >> >> ? > > I prefer the MyType[T,V] syntax. It encapsulates the "meta-type" > idea, and also the same syntax can be used both in declaring the type > and instantiating the type. > >> (The reason for the arguments: In C++ you can have stuff like ints in >> template arguments as well, might as well leave the possibility >> open even >> if only passing type is allowed for now). >> >> This even puts T and V in the proper scope for @cython.locals >> declarations, e.g. this passes through (and could even be made to >> do the >> checks) in pure Python: >> >> @cython.cdef >> class A: >> T, V = template_args(type, type) >> @cython.locals(a=T, b=v) >> def foo(self, a, b): ... > > I'd rather see the pure Python mode decorate the class than have a > magic method, but I don't think decorators get applied soon enough to > inject variables into the scope. That is correct. The whole advantage was that T and V gets injected in the scope in the right place. A better example is once we start to support argument decorators: class A: T, V = template_args(type, type) def foo(self, a: T, b: V): ... This currently passes just fine in Python 3. We could even allow referring to T and V as self.T and self.V within functions (in addition or as the only way -- it is more Pythonic as it doesn't invent a template-only scope). -- Dag Sverre From dagss at student.matnat.uio.no Wed May 6 09:47:15 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 May 2009 09:47:15 +0200 Subject: [Cython] template types In-Reply-To: <1409138872f9c0172f9467e4d3380dfe.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> <4A00AF71.7010103@behnel.de> <4A011CFE.7070006@behnel.de> <3439af420a85daac11ba6238c254c18f.squirrel@webmail.uio.no> <8B75D598-FF6D-4986-8E3D-0167BA5CE778@math.washington.edu> <1409138872f9c0172f9467e4d3380dfe.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <4A014083.1030500@student.matnat.uio.no> Stefan Behnel wrote: > I wouldn't care so much about that for now. Whatever declaration we > choose, it's easily added when the feature itself is there, and I think > parametrised types are such an advanced feature that users can well wait > for it to become available in pure Python mode. > > The priorities should be to make the feature simple and well readable in > Cython code, and then 'somehow' available in Python code. I disagree with this principle. (I guess it might be a matter of taste in the end, so I'll just state my opinion and try not to drag this on for too long.) In perhaps about a year, with Python 3 argument decorators (def foo(x: int) -> float) and a bit of type inference, "pure Python mode" could actually become as convenient as Cython syntax and the preferred syntax we'd teach new users in my opinion. And anyway, I'm in favor of avoiding to have to invent two kinds of syntax for new features. Especially when the benefits are so small! (One uses a template a lot more than writing them.) It's just a mess to have to tell users "this one is the Cython syntax but you can also use this one for Python compatability" etc.; that problem is there to stay but I don't see the sense of keeping adding to it. -- Dag Sverre From robertwb at math.washington.edu Wed May 6 09:57:08 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 00:57:08 -0700 Subject: [Cython] template types In-Reply-To: <4A014083.1030500@student.matnat.uio.no> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> <4A00AF71.7010103@behnel.de> <4A011CFE.7070006@behnel.de> <3439af420a85daac11ba6238c254c18f.squirrel@webmail.uio.no> <8B75D598-FF6D-4986-8E3D-0167BA5CE778@math.washington.edu> <1409138872f9c0172f9467e4d3380dfe.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A014083.1030500@student.matnat.uio.no> Message-ID: <4D8338A7-3164-423A-89F4-A52FDCCDDF22@math.washington.edu> On May 6, 2009, at 12:47 AM, Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> I wouldn't care so much about that for now. Whatever declaration we >> choose, it's easily added when the feature itself is there, and I >> think >> parametrised types are such an advanced feature that users can >> well wait >> for it to become available in pure Python mode. >> >> The priorities should be to make the feature simple and well >> readable in >> Cython code, and then 'somehow' available in Python code. > > I disagree with this principle. (I guess it might be a matter of taste > in the end, so I'll just state my opinion and try not to drag this on > for too long.) > > In perhaps about a year, with Python 3 argument decorators (def foo(x: > int) -> float) and a bit of type inference, "pure Python mode" could > actually become as convenient as Cython syntax and the preferred > syntax > we'd teach new users in my opinion. I think "cdef int i,j,k" is in always going to be easier than i = cython.declare(int) j = cython.declare(int) k = cython.declare(int) but in general I agree with this philosophy. > And anyway, I'm in favor of avoiding to have to invent two kinds of > syntax for new features. Especially when the benefits are so small! > (One > uses a template a lot more than writing them.) I'm guessing you mean you're *not* in favor? > It's just a mess to have to tell users "this one is the Cython syntax > but you can also use this one for Python compatability" etc.; that > problem is there to stay but I don't see the sense of keeping > adding to it. Only if there's a non-pure Python syntax that's a lot cleaner (which I think the class Foo[T] is). - Robert From brett.calcott at gmail.com Wed May 6 10:09:44 2009 From: brett.calcott at gmail.com (Brett Calcott) Date: Wed, 6 May 2009 18:09:44 +1000 Subject: [Cython] Passing numpy buffers into a cdef function Message-ID: Hi, I am very happy with the cython based extensions I have written to operate on numpy arrays. I'd like to refactor some code that is in these functions, putting it into a common cdef function that they can each call. Something like this: cdef common_stuff(np.ndarray[double, ndim=2] elements): # Tight loop here ... def fun1(np.ndarray[double, ndim=2] elements): # Do something ... common_stuff(elements) def fun2(np.ndarray[double, ndim=2] elements): # Do something different ... common_stuff(elements) But I get an error on compilation, that the parser "Expected ']'" where ndim is defined. What is the right way to do this? Thanks Brett From dagss at student.matnat.uio.no Wed May 6 10:11:55 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 May 2009 10:11:55 +0200 Subject: [Cython] template types In-Reply-To: <4D8338A7-3164-423A-89F4-A52FDCCDDF22@math.washington.edu> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> <4A00AF71.7010103@behnel.de> <4A011CFE.7070006@behnel.de> <3439af420a85daac11ba6238c254c18f.squirrel@webmail.uio.no> <8B75D598-FF6D-4986-8E3D-0167BA5CE778@math.washington.edu> <1409138872f9c0172f9467e4d3380dfe.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A014083.1030500@student.matnat.uio.no> <4D8338A7-3164-423A-89F4-A52FDCCDDF22@math.washington.edu> Message-ID: <4A01464B.7060506@student.matnat.uio.no> Robert Bradshaw wrote: > On May 6, 2009, at 12:47 AM, Dag Sverre Seljebotn wrote: > >> Stefan Behnel wrote: >>> I wouldn't care so much about that for now. Whatever declaration we >>> choose, it's easily added when the feature itself is there, and I >>> think >>> parametrised types are such an advanced feature that users can >>> well wait >>> for it to become available in pure Python mode. >>> >>> The priorities should be to make the feature simple and well >>> readable in >>> Cython code, and then 'somehow' available in Python code. >> I disagree with this principle. (I guess it might be a matter of taste >> in the end, so I'll just state my opinion and try not to drag this on >> for too long.) >> >> In perhaps about a year, with Python 3 argument decorators (def foo(x: >> int) -> float) and a bit of type inference, "pure Python mode" could >> actually become as convenient as Cython syntax and the preferred >> syntax >> we'd teach new users in my opinion. > > I think "cdef int i,j,k" is in always going to be easier than > > i = cython.declare(int) > j = cython.declare(int) > k = cython.declare(int) > > but in general I agree with this philosophy. Yes, like I said, type inference would be needed. So perhaps more than a year. But type inference wouldn't need to be that great, just at a minimum pick up manual hints like i = cython.int(func_returning_any_type()) and if i is never assigned to anything than known cython.int instances, then type it accordingly. I've also been thinking about doing things like def foo(x: int, y: int, _, localvar1: float, localvar2: float): ... but it is too ugly :-( In Py3 we could also abuse the new def f(x, *, kwonly): ... syntax in the same way, but it already has a use so probvably not...but I'm still playing with variations on the theme. > Only if there's a non-pure Python syntax that's a lot cleaner (which > I think the class Foo[T] is). Yes, I was basing what I said on writing a template to be much rarer (and done by a lot fewer users) and so not "needing" a clean syntax; I agree that it is cleaner. (I'm still concerned about adding Yet Another Scope Rule (I haven't even learned the current ones fully yet!) for this, but mostly on the philosophical level, in practice it would work as the user expects.) -- Dag Sverre From dagss at student.matnat.uio.no Wed May 6 10:13:24 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 May 2009 10:13:24 +0200 Subject: [Cython] Passing numpy buffers into a cdef function In-Reply-To: References: Message-ID: <4A0146A4.106@student.matnat.uio.no> Brett Calcott wrote: > Hi, > > I am very happy with the cython based extensions I have written to > operate on numpy arrays. I'd like to refactor some code that is in these > functions, putting it into a common cdef function that they can each > call. Something like this: > > cdef common_stuff(np.ndarray[double, ndim=2] elements): > # Tight loop here > ... > > def fun1(np.ndarray[double, ndim=2] elements): > # Do something > ... > common_stuff(elements) > > def fun2(np.ndarray[double, ndim=2] elements): > # Do something different > ... > common_stuff(elements) > > > But I get an error on compilation, that the parser "Expected ']'" where > ndim is defined. > > What is the right way to do this? You can't. You have to do cdef func(elements_) cdef np.ndarray[double, ndim=2] elements = elements_ and there's a certain overhead. This will likely be fixed during summer. http://trac.cython.org/cython_trac/ticket/177 -- Dag Sverre From brett.calcott at gmail.com Wed May 6 10:21:11 2009 From: brett.calcott at gmail.com (Brett Calcott) Date: Wed, 6 May 2009 18:21:11 +1000 Subject: [Cython] Passing numpy buffers into a cdef function In-Reply-To: <4A0146A4.106@student.matnat.uio.no> References: <4A0146A4.106@student.matnat.uio.no> Message-ID: 2009/5/6 Dag Sverre Seljebotn : > > You can't. You have to do > > cdef func(elements_) > ? ?cdef np.ndarray[double, ndim=2] elements = elements_ > > and there's a certain overhead. > > This will likely be fixed during summer. > > http://trac.cython.org/cython_trac/ticket/177 > Hi Dag, thanks for the info. I'll just stick with some ugly code for now then :) Cheers, Brett From dagss at student.matnat.uio.no Wed May 6 10:31:43 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 May 2009 10:31:43 +0200 Subject: [Cython] template types In-Reply-To: <8B75D598-FF6D-4986-8E3D-0167BA5CE778@math.washington.edu> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> <4A00AF71.7010103@behnel.de> <4A011CFE.7070006@behnel.de> <3439af420a85daac11ba6238c254c18f.squirrel@webmail.uio.no> <8B75D598-FF6D-4986-8E3D-0167BA5CE778@math.washington.edu> Message-ID: <4A014AEF.7030901@student.matnat.uio.no> Robert Bradshaw wrote: > On May 5, 2009, at 10:23 PM, Dag Sverre Seljebotn wrote: > >> Stefan Behnel wrote: >>> Lisandro Dalcin wrote: >>>> On Tue, May 5, 2009 at 6:28 PM, Stefan Behnel wrote: >>>>> I meant >>>>> >>>>> cdef MyType(object) [(T,V)]: >>>>> >>>>> here, although I now noticed that there is already the "private >>>>> type" >>>>> syntax: >>>>> >>>>> cdef public class _Document [ type LxmlDocumentType, object >>>>> LxmlDocument ]: >>>>> ... >>>>> >>>>> so this won't work straight away either... >>>>> >>>> But this would work anyway, right? >>> Not that easily. You could have both [] parts next to each other, >>> or just >>> any one of them. So the parser would fail if you decided to call your >>> template variable "type", which isn't that a bad name in this >>> context. >> How about >> >> cdef class A: >> T, V = cython.template_args(type, type) >> >> ? > > I prefer the MyType[T,V] syntax. It encapsulates the "meta-type" > idea, and also the same syntax can be used both in declaring the type > and instantiating the type. BTW will this discussion also be used for C++ templates, so that the consensus seem to be some_cpp_class[int] for template instantiation? -- Dag Sverre From robertwb at math.washington.edu Wed May 6 10:37:13 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 01:37:13 -0700 Subject: [Cython] template types In-Reply-To: <4A014AEF.7030901@student.matnat.uio.no> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> <4A00AF71.7010103@behnel.de> <4A011CFE.7070006@behnel.de> <3439af420a85daac11ba6238c254c18f.squirrel@webmail.uio.no> <8B75D598-FF6D-4986-8E3D-0167BA5CE778@math.washington.edu> <4A014AEF.7030901@student.matnat.uio.no> Message-ID: On May 6, 2009, at 1:31 AM, Dag Sverre Seljebotn wrote: > BTW will this discussion also be used for C++ templates, so that the > consensus seem to be > > some_cpp_class[int] > > for template instantiation? I think so. Speak now or forever hold your peace... :) - Robert From robertwb at math.washington.edu Wed May 6 11:02:15 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 02:02:15 -0700 Subject: [Cython] main-embedding problem In-Reply-To: References: <49FF5325.7020705@behnel.de> <7B8637C0-61E3-40D6-8BE4-04E635F78682@math.washington.edu> Message-ID: On May 5, 2009, at 3:04 PM, Lisandro Dalcin wrote: > On Mon, May 4, 2009 at 7:10 PM, Robert Bradshaw > wrote: >>> Unfortunatelly, I have to agree, for different reasons... As an >>> starter, the current code does not handle Py3K on Windows >>> (because of >>> the new wchar_t based API) >> >> This should be sufficient, right? >> >> http://hg.cython.org/cython-devel/rev/2e35d5caac86 >> > > No, unfortunately it is not enough... I didn't tell you the whole > story... Py3K has a wchar_t-based API for all platforms. On Windows, > the fix is easy, you just use wmain(). On POSIX, not so easy... you > have to use char-based, traditional main(), and use mbrtowc() to > convent all the argument one by one... > > IMHO, core CPython is doing it wrong here... See the nightmare in > py3k/Modules/python.c ... IMHO, Python should provide a both > char-based and wchar_t-based API's, and internally implement the > char-based ones as wrappers, managing the conversion char->wchat_t and > then call the wchar_t-based API's. > > If this is not done, any one trying to embed python have to start > converting arguments (and managing the tmp mem blocks), and have to > cope with the apparent brokenness of mbrtowc() on some platforms... > I've already had to implement all that crap in mpi4py, see > http://code.google.com/p/mpi4py/source/browse/trunk/src/python.c > (borrow code from Py3_Main_GetArgs() if you like it). Fortunately, > that file is not frequently required for end-users, but I'm not even > sure if it works on the many Unix-like systems out-there (current > supercomputer systems hardly have python3 available for end-users) > > I do not have the energy to go to Python-Dev to discuss all this... Do > any of you have it :-)? Stefan? Greg? :( That is just ugly. I am with you that CPython should be fixed, but don't have the time to look into hacking around this myself. - Robert From robertwb at math.washington.edu Wed May 6 11:08:50 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 02:08:50 -0700 Subject: [Cython] Storing many, many instances In-Reply-To: References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <8f5db410ca5009dd2aec60dce437cd23.squirrel@webmail.uio.no> <7c940a5d130e97a4d739efc6f3343bff.squirrel@webmail.uio.no> <4A0028BE.9030903@behnel.de> Message-ID: On May 5, 2009, at 1:54 PM, Lisandro Dalcin wrote: > On Tue, May 5, 2009 at 8:53 AM, Stefan Behnel > wrote: >> >> Dag Sverre Seljebotn wrote: >>>> St?fan van der Walt wrote: >>>>> I think I may have to switch to structs instead of classes, but >>>>> then I >>>>> no longer have convenient methods or array members. Maybe I >>>>> should >>>>> rewrite my class to have many static methods that operate on a >>>>> struct, >>>>> and then collect the structs as data instead of the full >>>>> instances. >>>> Technically it would not be too difficult to allow >>>> >>>> cdef struct MyStruct: >>>> void foo(self): >>>> # self is pure MyStruct, perhaps stack-allocated >>>> >> >> I wouldn't mind. We have a lot of convenience stuff in structs >> already >> (e.g. dict coercion and keyword arguments), so adding methods with >> the >> above semantics wouldn't hurt. > > I agree... I wouldn't be opposed... > >> It's not rewriting C++ either, as long as it makes sense in Cython >> as a >> language. >> > > But them someone will ask for polimorphism, and then the struct > instances could have a pointer to a struct-specific vtable, and then > you are more or less rewriting C++. And that would not bother me, > currently cdef methods do work as a sort C++, right?. Yeah, I'm not sure how far one wants to take this though. We should look into making instantiating (cdef) classes faster by default as well. - Robert From robertwb at math.washington.edu Wed May 6 11:10:28 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 02:10:28 -0700 Subject: [Cython] Problems with creating a package of extension modules In-Reply-To: <1241206225.49fb4dd161f7c@webmail.uni-jena.de> References: <1241198159.49fb2e4f76162@webmail.uni-jena.de> <922BECD5-7D73-46BC-A541-C76081EAC703@math.washington.edu> <1241206225.49fb4dd161f7c@webmail.uni-jena.de> Message-ID: <09C1EF75-876B-4775-A8AB-552ABCF9882D@math.washington.edu> On May 1, 2009, at 12:30 PM, Simon King wrote: > Dear Robert, > > Zitat von Robert Bradshaw : >> Hmm... could you point me to that bit of documentation? I think it's >> wrong (or at least not implemented as explained), we do Extension >> ("package.module",...) all the time in sage and that seems to mix >> well with pickling. > > Admittedly it is not Cython-documentation, but Python: > At http://docs.python.org/distutils/setupscript.html, they say: > > --------------------- > If you have a number of extensions all in the same package (or all > under the > same base package), use the ext_package keyword argument to setup > (). For > example, > > setup(..., > ext_package='pkg', > ext_modules=[Extension('foo', ['foo.c']), > Extension('subpkg.bar', ['bar.c'])], > ) > > will compile foo.c to the extension pkg.foo, and bar.c to > pkg.subpkg.bar. To wrap this thread up, it turns out that the ext_package option is incompatible with using .pxd files. Without the ext_package it works as expected (One complication--when trying stuff like this out, one has to make sure to delete the .c file as well as the build directory.) - Robert From robertwb at math.washington.edu Wed May 6 11:39:06 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 02:39:06 -0700 Subject: [Cython] New idea for C++ stack allocation In-Reply-To: <49F6A8E5.2040006@student.matnat.uio.no> References: <49F6A8E5.2040006@student.matnat.uio.no> Message-ID: <759D34EC-9F78-4E80-8818-987F7551AA07@math.washington.edu> On Apr 27, 2009, at 11:57 PM, Dag Sverre Seljebotn wrote: > I got another idea for how using C++ objects without no-arg > constructors > could work. It means enforcing C++ variable scoping through compile- > time > errors, but keeping Python syntax. I think for our first pass at least, we should allocate everything on the heap. We could then try to (transparently?) move things to the stack when we can (this would have implications for passing things around though). > The transform necesarry to enforce these rules should not be too > complicated. I don't think we need full analysis, just track > assignments > and deletions and usages on a per-name basis. > > Finally, the C++ output is made by adding a block, i.e. > > cdef CppObject obj > print 1 > obj = bar > obj.func() > del obj > print 2 > > Results in > > print 1 > { > CppObject obj = bar; > obj.func() > } # del > print 2 What about cdef CppObject a, b a = bar b = bar del a del b ? - Robert From stefan_ml at behnel.de Wed May 6 12:07:35 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 6 May 2009 12:07:35 +0200 (CEST) Subject: [Cython] Storing many, many instances In-Reply-To: References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <8f5db410ca5009dd2aec60dce437cd23.squirrel@webmail.uio.no> <7c940a5d130e97a4d739efc6f3343bff.squirrel@webmail.uio.no> <4A0028BE.9030903@behnel.de> Message-ID: <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Robert Bradshaw wrote: > On May 5, 2009, at 1:54 PM, Lisandro Dalcin wrote: >> But them someone will ask for polimorphism, and then the struct >> instances could have a pointer to a struct-specific vtable, and then >> you are more or less rewriting C++. And that would not bother me, >> currently cdef methods do work as a sort C++, right?. > > Yeah, I'm not sure how far one wants to take this though. I think a good point to stop is where we have to change the struct itself. Adding a vtable to it would be rather surprising for users as it could break stuff. We can do with cdef classes what we want, but C structs should stay the way they are defined in user code. Adding 'methods' to them is a rather straight transform in Cython that does not impact the generated C code, so that's just fine. > We should > look into making instantiating (cdef) classes faster by default as well. Different topic, but surely worth it (and worth a ticket). It might work to always use PY_NEW() for instantiation and then generate a separate (and direct) call to __init__() only if the type defines it. Stefan From dagss at student.matnat.uio.no Wed May 6 14:10:55 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 May 2009 14:10:55 +0200 Subject: [Cython] Storing many, many instances In-Reply-To: <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <8f5db410ca5009dd2aec60dce437cd23.squirrel@webmail.uio.no> <7c940a5d130e97a4d739efc6f3343bff.squirrel@webmail.uio.no> <4A0028BE.9030903@behnel.de> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <4A017E4F.5050800@student.matnat.uio.no> Stefan Behnel wrote: > Robert Bradshaw wrote: >> We should >> look into making instantiating (cdef) classes faster by default as well. > > Different topic, but surely worth it (and worth a ticket). It might work > to always use PY_NEW() for instantiation and then generate a separate (and > direct) call to __init__() only if the type defines it. Should one consider allocating arrays of cdef class objects as one big memory block? I would think that allocating first and then run a constructor N times would be faster than N allocations. -- Dag Sverre From dagss at student.matnat.uio.no Wed May 6 14:23:03 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 May 2009 14:23:03 +0200 Subject: [Cython] New idea for C++ stack allocation In-Reply-To: <759D34EC-9F78-4E80-8818-987F7551AA07@math.washington.edu> References: <49F6A8E5.2040006@student.matnat.uio.no> <759D34EC-9F78-4E80-8818-987F7551AA07@math.washington.edu> Message-ID: <4A018127.4090208@student.matnat.uio.no> Robert Bradshaw wrote: > On Apr 27, 2009, at 11:57 PM, Dag Sverre Seljebotn wrote: > >> I got another idea for how using C++ objects without no-arg >> constructors >> could work. It means enforcing C++ variable scoping through compile- >> time >> errors, but keeping Python syntax. > > I think for our first pass at least, we should allocate everything on > the heap. We could then try to (transparently?) move things to the > stack when we can (this would have implications for passing things > around though). I agree to this. If I understand you correctly it means that C++ classes would work very much like cdef classes (like in the wiki proposal). Some further notes: Passing things around shouldn't be such a problem; I don't think it would be much more complicated to do the heap allocation through a wrapper Python object (it could be opaque at first, only feature would be type-checking on coercion to a typed C++ class and perhaps store the C++ class name for its repr). Conversion of "CppClass" (refcounted wrapper) to "CppClass*" could be allowed; but probably not the other way (at least without using a function saying whether ownership of the pointer should be taken over or not). All of this *could* have been done for C structs as well, and in C++ there's no real distinction between a struct and a classes. I suppose a distinction must be done in Cython though because of code like cdef MyStruct a, b a.field = ... b = a # copies value, not reference (Note that the /only/ difference between the struct keyword and class keyword in C++ is whether private or public is the default visibility. A vtable is added if and only if a function is declared as virtual, whether it is declared as struct or not.) See also the other discussion on methods in structs. > What about > > cdef CppObject a, b > a = bar > b = bar > del a > del b > > ? Yes, I thought of that after sending it. -- Dag Sverre From dagss at student.matnat.uio.no Wed May 6 14:33:59 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 May 2009 14:33:59 +0200 Subject: [Cython] New idea for C++ stack allocation In-Reply-To: <759D34EC-9F78-4E80-8818-987F7551AA07@math.washington.edu> References: <49F6A8E5.2040006@student.matnat.uio.no> <759D34EC-9F78-4E80-8818-987F7551AA07@math.washington.edu> Message-ID: <4A0183B7.6080508@student.matnat.uio.no> Robert Bradshaw wrote: > What about > > cdef CppObject a, b > a = bar > b = bar > del a > del b > > ? Actually this *can* be supported by overloading operator new. I'll change the example to cdef CppObject a, b a = CppObject(3) b = CppObject(4) del a del b to avoid an unrelated and orthogonal issue with operator=. Then you can do: struct {} CythonOperators; void* operator new(size_t bytes, char* arg, CythonOperators foo) { return arg; } void operator delete(void* arg, CythonOperators foo) {/*noop*/} void f() { char a[sizeof(CppObject)]; char b[sizeof(CppObject)]; new(a, CythonOperators) CppObject(3); new(b, CythonOperators) CppObject(4); delete(CythonOperators) a; delete(CythonOperators) b; } And /yes/, this is real C++ code... *sigh* :-) (though I'm not sure if I got the details 100% right). I wouldn't really recommend going this route though. -- Dag Sverre From stefan_ml at behnel.de Wed May 6 14:34:52 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 6 May 2009 14:34:52 +0200 (CEST) Subject: [Cython] Storing many, many instances In-Reply-To: <4A017E4F.5050800@student.matnat.uio.no> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <8f5db410ca5009dd2aec60dce437cd23.squirrel@webmail.uio.no> <7c940a5d130e97a4d739efc6f3343bff.squirrel@webmail.uio.no> <4A0028BE.9030903@behnel.de> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A017E4F.5050800@student.matnat.uio.no> Message-ID: <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Robert Bradshaw wrote: >>> We should >>> look into making instantiating (cdef) classes faster by default as >>> well. >> >> Different topic, but surely worth it (and worth a ticket). It might work >> to always use PY_NEW() for instantiation and then generate a separate >> (and direct) call to __init__() only if the type defines it. > > Should one consider allocating arrays of cdef class objects as one big > memory block? We do not control the allocation, CPython handles that. > I would think that allocating first and then run a > constructor N times would be faster than N allocations. How would you instantiate arrays of classes in a single step? Stefan From dagss at student.matnat.uio.no Wed May 6 14:55:48 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 May 2009 14:55:48 +0200 Subject: [Cython] Storing many, many instances In-Reply-To: <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <8f5db410ca5009dd2aec60dce437cd23.squirrel@webmail.uio.no> <7c940a5d130e97a4d739efc6f3343bff.squirrel@webmail.uio.no> <4A0028BE.9030903@behnel.de> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A017E4F.5050800@student.matnat.uio.no> <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <4A0188D4.1050208@student.matnat.uio.no> Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> Stefan Behnel wrote: >>> Robert Bradshaw wrote: >>>> We should >>>> look into making instantiating (cdef) classes faster by default as >>>> well. >>> Different topic, but surely worth it (and worth a ticket). It might work >>> to always use PY_NEW() for instantiation and then generate a separate >>> (and direct) call to __init__() only if the type defines it. >> Should one consider allocating arrays of cdef class objects as one big >> memory block? > > We do not control the allocation, CPython handles that. OK. I didn't know exactly how fine-grained the CPython API was and whether it was possible to run a constructor on already allocated memory (like you can do in e.g. C++). It just seemed like that could be one of the major bottlenecks. -- Dag Sverre From stefan_ml at behnel.de Wed May 6 15:23:59 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 6 May 2009 15:23:59 +0200 (CEST) Subject: [Cython] Storing many, many instances In-Reply-To: <4A0188D4.1050208@student.matnat.uio.no> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <8f5db410ca5009dd2aec60dce437cd23.squirrel@webmail.uio.no> <7c940a5d130e97a4d739efc6f3343bff.squirrel@webmail.uio.no> <4A0028BE.9030903@behnel.de> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A017E4F.5050800@student.matnat.uio.no> <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A0188D4.1050208@student.matnat.uio.no> Message-ID: Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Dag Sverre Seljebotn wrote: >>> Stefan Behnel wrote: >>>> It might work >>>> to always use PY_NEW() for instantiation and then generate a separate >>>> (and direct) call to __init__() only if the type defines it. >>>> >>> Should one consider allocating arrays of cdef class objects as one big >>> memory block? >> >> We do not control the allocation, CPython handles that. > > OK. I didn't know exactly how fine-grained the CPython API was and > whether it was possible to run a constructor on already allocated memory > (like you can do in e.g. C++). It just seemed like that could be one of > the major bottlenecks. What we tend to call PY_NEW here is a direct call to a type's tp_new(), which handles both the allocation and the basic initialisation of an instance. Cython could do a tiny bit better here, since we know exactly what function a cdef class has in its tp_new slot. However, that function will always recursively run into the base type's tp_new and at the end of the chain into the one of Python's 'type' type, and there is not much we can do to improve this situation in general. Stefan From dalcinl at gmail.com Wed May 6 15:57:15 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 6 May 2009 10:57:15 -0300 Subject: [Cython] Include paths for pyximport In-Reply-To: References: Message-ID: On Tue, May 5, 2009 at 12:08 AM, Brett Calcott wrote: > Hi all, > I'm using pyimport and writing some extensions to work numpy. To get things > to happen magically the compiler needs to be able to find the header files. That's a very valid use case... > I've got it working on my mac, by adding a line to my .bashrc > export C_INCLUDE_PATH=/Library/Python/2.5/site-packages/numpy/core/include > I'd like to get it working on windows. Though it works on your Mac, it is not clear if this works on other POSIX systems... > I've added the equivalent path in my > windows env (set INCLUDE=...), but I'm getting an error telling me it cannot > find the "include/arrayobject.h". I've tried setting the right directory in > the GUI too, but to no avail. On Windows, distutils hardly inspect the environment, so playing with it is not a general solution. > Can anyone tell me the easy/right way to add include paths for pyximport on > windows. Is there a better and portable way to do this? AFAIK, pyximport does not support this at all... However, it would be really easy to enhance it. I would add a 'options' or 'config' (or whatever name) at the end of "pyximport.install()" (or perhaps provide a new call for this task) accepting a dictionary. That dictionary should contain stuff to pass as kwargs to the distutils Extension, the more important ones would be include_dirs define_macros undef_macros library_dirs libraries runtime_library_dirs That stuff should be (after checking and perhaps sanitizing the dict contents) passed to Extension() in "pyximport.get_distutils_extension()". Unfortunately, I'm really busy to implement something like that for review. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Wed May 6 16:21:23 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 6 May 2009 11:21:23 -0300 Subject: [Cython] asking for review of my patch for #287 Message-ID: I want to raise your attention to ticket #287, as my patch could be controversial. I even changed my mind, and fixed my previous work on all this to prefer 'nb_int' over 'nb_long' in __Pyx_PyNumber_Int(). This change was not a quick hack to get things solved, but rather a hard to take decision after diving core CPython sources from 2.3 to 2.7. IMHO, the 'nb_long' slot should have been deprecated long, long ago in core CPython. I would like this to be discussed a bit more before next release. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From brett.calcott at gmail.com Wed May 6 16:43:36 2009 From: brett.calcott at gmail.com (Brett Calcott) Date: Thu, 7 May 2009 00:43:36 +1000 Subject: [Cython] Include paths for pyximport In-Reply-To: References: Message-ID: 2009/5/6 Lisandro Dalcin : > > --- Snip --- > >> Can anyone tell me the easy/right way to add include paths for pyximport on >> windows. Is there a better and portable way to do this? > > > AFAIK, pyximport does not support this at all... However, it would be > really easy to enhance it. I would add a 'options' or 'config' (or > whatever name) at the end of "pyximport.install()" (or perhaps provide > a new call for this task) accepting a dictionary. That dictionary > should contain stuff to pass as kwargs to the distutils Extension, the > more important ones would be > > include_dirs > define_macros > undef_macros > library_dirs > libraries > runtime_library_dirs > > That stuff should be (after checking and perhaps sanitizing the dict > contents) passed to Extension() in > "pyximport.get_distutils_extension()". > > Unfortunately, I'm really busy to implement something like that for review. > Hi Lisandro. I've just had a look at the pyximport files, and what you are suggesting makes sense to me. I'll give it a go myself, and then get back to the list with any progress. Thanks, Brett -- Brett Calcott Philosophy Program, Research School of Social Sciences, AND Centre for Macroevolution & Macroecology, School of Botany & Zoology, AUSTRALIAN NATIONAL UNIVERSITY, Canberra, ACT, 0200, AUSTRALIA From carl.witty at gmail.com Wed May 6 17:56:27 2009 From: carl.witty at gmail.com (Carl Witty) Date: Wed, 6 May 2009 08:56:27 -0700 Subject: [Cython] Storing many, many instances In-Reply-To: References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <7c940a5d130e97a4d739efc6f3343bff.squirrel@webmail.uio.no> <4A0028BE.9030903@behnel.de> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A017E4F.5050800@student.matnat.uio.no> <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A0188D4.1050208@student.matnat.uio.no> Message-ID: On Wed, May 6, 2009 at 6:23 AM, Stefan Behnel wrote: > What we tend to call PY_NEW here is a direct call to a type's tp_new(), > which handles both the allocation and the basic initialisation of an > instance. Cython could do a tiny bit better here, since we know exactly > what function a cdef class has in its tp_new slot. However, that function > will always recursively run into the base type's tp_new and at the end of > the chain into the one of Python's 'type' type, and there is not much we > can do to improve this situation in general. Well, you could optimize tp_new. You could have it call the base type's tp_new directly, instead of through a vtable (at least if the base type is a Cython type); or better yet, inline the base type's tp_new (for Cython base types). (Gary Furnish complained that calling through the tp_new chain was a major expense if you had a deep inheritance hierarchy; I haven't confirmed this myself, but it sounds plausible.) Carl From ellisonbg.net at gmail.com Wed May 6 18:08:37 2009 From: ellisonbg.net at gmail.com (Brian Granger) Date: Wed, 6 May 2009 09:08:37 -0700 Subject: [Cython] Include paths for pyximport In-Reply-To: References: Message-ID: <6ce0ac130905060908wf3542d1m8148afda3bec9a9@mail.gmail.com> > AFAIK, pyximport does not support this at all... However, it would be > really easy to enhance it. I would add a 'options' or 'config' (or > whatever name) at the end of "pyximport.install()" (or perhaps provide > a new call for this task) accepting a dictionary. That dictionary > should contain stuff to pass as kwargs to the distutils Extension, the > more important ones would be > > include_dirs > define_macros > undef_macros > library_dirs > libraries > runtime_library_dirs > > That stuff should be (after checking and perhaps sanitizing the dict > contents) passed to Extension() in > "pyximport.get_distutils_extension()". +1, great idea From dagss at student.matnat.uio.no Wed May 6 18:31:06 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 May 2009 18:31:06 +0200 Subject: [Cython] Buffer passing implementation details Message-ID: <4A01BB4A.1060003@student.matnat.uio.no> (Kurt is the major recipient of this.) I've been thinking some more on buffer passing. In the end (and perhaps even in summer; I could perhaps have a go at it working alongside you) the scenario we are looking at is something like external_set_foo_array(foo_handle, self.arr[2::2]) with fast operations all the way. That is a) Efficient slicing without Python overhead (#178) b) Storing acquired buffers in cdef class fields (#301) c) External functions can keep a reference to the buffer (not really necesarry, but it is necesarry for internal Cython "cdef" functions, and it would be nice to treat them the same) d) The base pointer may have to be moved (again slicing can do this) This seems to make it clear that a) The Py_buffer is not suitable as the primary vessel of our buffer data, e.g. to the external function. It can't work that well with slicing as we must maintain the original Py_buffer data when releasing it. b) We need a reference count. E.g. doing a slice, or assigning to self.arr, would increase this count. This must go on the heap; and so we might as well put the entire Py_buffer on the heap. I have made a new ticket outlining the thoughts in detail in #311 and added a comment in #299. #311 is not really in your direct interest for GSoC but it is very tightly coupled with #299 so it would be good to keep in the back of your mind anyway. Thoughts? It sheems a shame to let go of a neatly PEP-defined Py_buffer for passing to external functions, but I think it won't be too bad if we with each Cython version ship nice C header and Fortran include files containing the appropriate structs and access macros. -- Dag Sverre From dagss at student.matnat.uio.no Wed May 6 19:21:23 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 May 2009 19:21:23 +0200 Subject: [Cython] endianness bug fix in numpy_test.pyx In-Reply-To: References: <49E991D7.5000809@student.matnat.uio.no> <49E9CE21.8040105@student.matnat.uio.no> Message-ID: <4A01C713.7080302@student.matnat.uio.no> Kurt Smith wrote: > On Sat, Apr 18, 2009 at 7:57 AM, Dag Sverre Seljebotn > wrote: >> Dag Sverre Seljebotn wrote: >>> Kurt Smith wrote: >>>> The current numpy_test.pyx file fails for PowerPC macs due to >>>> endianness issues in the dtype. This is a small fix to make it work >>>> (and make all tests pass on my machine). It also adds an explicit >>>> big-endian test to the doctest. >>>> >>>> diff -r fc73225aaea1 tests/run/numpy_test.pyx >>>> --- a/tests/run/numpy_test.pyx Fri Apr 17 09:11:16 2009 +0200 >>>> +++ b/tests/run/numpy_test.pyx Fri Apr 17 15:43:11 2009 -0500 >>>> @@ -132,13 +132,20 @@ >>>> >>> test_recordarray() >>>> >>>> >>> test_nested_dtypes(np.zeros((3,), dtype=np.dtype([\ >>>> - ('a', np.dtype('i,i')),\ >>>> - ('b', np.dtype('i,i'))\ >>>> + ('a', np.dtype('>>> + ('b', np.dtype('>>> ]))) >>>> array([((0, 0), (0, 0)), ((1, 2), (1, 4)), ((1, 2), (1, 4))], >>>> dtype=[('a', [('f0', '>>> '>> Any ideas on why this particular case actually passes on your machine? >>> The Cython code alwayws works with big-endian on your machine, so I >>> wouldn't expect to see ((1, 2), (1, 4)) there, but the byteswapped >>> versions of those numbers... >>> >> Ahh right; the only value read from the input array (in the test >> function) is zero, which is the same in both endians, and you added >> explicit byteswap on the output. > > Sorry -- can't devote the time to address these questions right now > but I'll get to them as soon as I can (Monday?). Could you try again with current cython-devel at some point? I believe this should be fixed now but obviously it should be tested on a PPC. -- Dag Sverre From robertwb at math.washington.edu Wed May 6 19:30:46 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 10:30:46 -0700 Subject: [Cython] New idea for C++ stack allocation In-Reply-To: <4A0183B7.6080508@student.matnat.uio.no> References: <49F6A8E5.2040006@student.matnat.uio.no> <759D34EC-9F78-4E80-8818-987F7551AA07@math.washington.edu> <4A0183B7.6080508@student.matnat.uio.no> Message-ID: On May 6, 2009, at 5:33 AM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> What about >> >> cdef CppObject a, b >> a = bar >> b = bar >> del a >> del b >> >> ? > > Actually this *can* be supported by overloading operator new. > > I'll change the example to > > cdef CppObject a, b > a = CppObject(3) > b = CppObject(4) > del a > del b > > to avoid an unrelated and orthogonal issue with operator=. Then you > can do: > > struct {} CythonOperators; > > void* operator new(size_t bytes, char* arg, CythonOperators foo) { > return arg; } > void operator delete(void* arg, CythonOperators foo) {/*noop*/} > > void f() { > char a[sizeof(CppObject)]; > char b[sizeof(CppObject)]; > > new(a, CythonOperators) CppObject(3); > new(b, CythonOperators) CppObject(4); > > delete(CythonOperators) a; > delete(CythonOperators) b; > } > > And /yes/, this is real C++ code... *sigh* :-) (though I'm not sure > if I > got the details 100% right). Actually, the new operator already supports passing in an address to be used for memory, but I wasn't sure enough of how it worked to bring it up. There's an issue of alignment as well. > I wouldn't really recommend going this route though. I like it better than trying to use blocks to accomplish stack-based allocation though. - Robert From robertwb at math.washington.edu Wed May 6 19:38:53 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 10:38:53 -0700 Subject: [Cython] Storing many, many instances In-Reply-To: References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <7c940a5d130e97a4d739efc6f3343bff.squirrel@webmail.uio.no> <4A0028BE.9030903@behnel.de> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A017E4F.5050800@student.matnat.uio.no> <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A0188D4.1050208@student.matnat.uio.no> Message-ID: <11E61C9F-3175-4473-8B39-20277284E9D1@math.washington.edu> On May 6, 2009, at 8:56 AM, Carl Witty wrote: > On Wed, May 6, 2009 at 6:23 AM, Stefan Behnel > wrote: >> What we tend to call PY_NEW here is a direct call to a type's >> tp_new(), >> which handles both the allocation and the basic initialisation of an >> instance. Cython could do a tiny bit better here, since we know >> exactly >> what function a cdef class has in its tp_new slot. However, that >> function >> will always recursively run into the base type's tp_new and at the >> end of >> the chain into the one of Python's 'type' type, and there is not >> much we >> can do to improve this situation in general. > > Well, you could optimize tp_new. You could have it call the base > type's tp_new directly, instead of through a vtable (at least if the > base type is a Cython type); I already implemented this optimization, though only for cdef classes within the same module. (If they're small enough, they may get even inlined by the C compiler.) I'm not sure how possible this would be to do across modules, but one difficulty is that the contents of tp_new can change based on the .pyx file, not just the .pxd file. That being said, one does have control over allocation via the tp_alloc method. Allocating an array of objects as a big chuck is messy though--how would one handle deallocating them one by one? Or would one try to reference-count them as a group? I think it could be done, but it would be a non-trivial interface. - Robert From dagss at student.matnat.uio.no Wed May 6 19:56:02 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 May 2009 19:56:02 +0200 Subject: [Cython] New idea for C++ stack allocation In-Reply-To: References: <49F6A8E5.2040006@student.matnat.uio.no> <759D34EC-9F78-4E80-8818-987F7551AA07@math.washington.edu> <4A0183B7.6080508@student.matnat.uio.no> Message-ID: <4A01CF32.8050207@student.matnat.uio.no> Robert Bradshaw wrote: > On May 6, 2009, at 5:33 AM, Dag Sverre Seljebotn wrote: > >> Robert Bradshaw wrote: >>> What about >>> >>> cdef CppObject a, b >>> a = bar >>> b = bar >>> del a >>> del b >>> >>> ? >> Actually this *can* be supported by overloading operator new. >> >> I'll change the example to >> >> cdef CppObject a, b >> a = CppObject(3) >> b = CppObject(4) >> del a >> del b >> >> to avoid an unrelated and orthogonal issue with operator=. Then you >> can do: >> >> struct {} CythonOperators; >> >> void* operator new(size_t bytes, char* arg, CythonOperators foo) { >> return arg; } >> void operator delete(void* arg, CythonOperators foo) {/*noop*/} >> >> void f() { >> char a[sizeof(CppObject)]; >> char b[sizeof(CppObject)]; >> >> new(a, CythonOperators) CppObject(3); >> new(b, CythonOperators) CppObject(4); >> >> delete(CythonOperators) a; >> delete(CythonOperators) b; >> } >> >> And /yes/, this is real C++ code... *sigh* :-) (though I'm not sure >> if I >> got the details 100% right). > > Actually, the new operator already supports passing in an address to > be used for memory, but I wasn't sure enough of how it worked to > bring it up. There's an issue of alignment as well. Hmm. Interesting. Looked it up properly and tried it out, attaching for any future reference (though I suppose stack allocation is far down the line and after heap then). Some memory is wasted for the alignment issue. #include #include using namespace std; class CppObject { int data; public: CppObject(int d) : data(d) { cout << data << " construct" << endl; } ~CppObject() { cout << data << " destruct" << endl; } }; int main () { void* __pyx_mem_a[(sizeof(CppObject) / sizeof(void*))+1]; CppObject& __pyx_v_a = *(CppObject*)buf; new(buf) CppObject(4); // use __pyx_v_a as any stack allocated object co.~CppObject(); cout << "done" << endl; } -- Dag Sverre From dagss at student.matnat.uio.no Wed May 6 20:12:10 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 May 2009 20:12:10 +0200 Subject: [Cython] New idea for C++ stack allocation In-Reply-To: <4A01CF32.8050207@student.matnat.uio.no> References: <49F6A8E5.2040006@student.matnat.uio.no> <759D34EC-9F78-4E80-8818-987F7551AA07@math.washington.edu> <4A0183B7.6080508@student.matnat.uio.no> <4A01CF32.8050207@student.matnat.uio.no> Message-ID: <4A01D2FA.400@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On May 6, 2009, at 5:33 AM, Dag Sverre Seljebotn wrote: >> >>> Robert Bradshaw wrote: >>>> What about >>>> >>>> cdef CppObject a, b >>>> a = bar >>>> b = bar >>>> del a >>>> del b >>>> >>>> ? >>> Actually this *can* be supported by overloading operator new. >>> >>> I'll change the example to >>> >>> cdef CppObject a, b >>> a = CppObject(3) >>> b = CppObject(4) >>> del a >>> del b >>> >>> to avoid an unrelated and orthogonal issue with operator=. Then you >>> can do: >>> >>> struct {} CythonOperators; >>> >>> void* operator new(size_t bytes, char* arg, CythonOperators foo) { >>> return arg; } >>> void operator delete(void* arg, CythonOperators foo) {/*noop*/} >>> >>> void f() { >>> char a[sizeof(CppObject)]; >>> char b[sizeof(CppObject)]; >>> >>> new(a, CythonOperators) CppObject(3); >>> new(b, CythonOperators) CppObject(4); >>> >>> delete(CythonOperators) a; >>> delete(CythonOperators) b; >>> } >>> >>> And /yes/, this is real C++ code... *sigh* :-) (though I'm not sure >>> if I >>> got the details 100% right). >> Actually, the new operator already supports passing in an address to >> be used for memory, but I wasn't sure enough of how it worked to >> bring it up. There's an issue of alignment as well. > > Hmm. Interesting. Looked it up properly and tried it out, attaching for > any future reference (though I suppose stack allocation is far down the > line and after heap then). /Honestly/... This time without changing any variables names while in email mode... #include #include using namespace std; class CppObject { int data; public: CppObject(int d) : data(d) { cout << data << " construct" << endl; } ~CppObject() { cout << data << " destruct" << endl; } }; int main () { void* __pyx_mem_myvar[(sizeof(CppObject) / sizeof(void*))+1]; CppObject& __pyx_v_myvar = *(CppObject*)__pyx_mem_myvar; new(__pyx_mem_myvar) CppObject(4); // use __pyx_v_myvar as normal CppObject on stack __pyx_v_myvar.~CppObject(); cout << "done" << endl; } -- Dag Sverre From carl.witty at gmail.com Wed May 6 20:18:59 2009 From: carl.witty at gmail.com (Carl Witty) Date: Wed, 6 May 2009 11:18:59 -0700 Subject: [Cython] Storing many, many instances In-Reply-To: <11E61C9F-3175-4473-8B39-20277284E9D1@math.washington.edu> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A017E4F.5050800@student.matnat.uio.no> <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A0188D4.1050208@student.matnat.uio.no> <11E61C9F-3175-4473-8B39-20277284E9D1@math.washington.edu> Message-ID: On Wed, May 6, 2009 at 10:38 AM, Robert Bradshaw wrote: > I already implemented this optimization, though only for cdef classes > within the same module. (If they're small enough, they may get even > inlined by the C compiler.) I'm not sure how possible this would be > to do across modules, but one difficulty is that the contents of > tp_new can change based on the .pyx file, not just the .pxd file. Really? What in the .pyx file can change tp_new? (I'm mostly just curious.) Across modules, you should at least be able to call the tp_new function directly instead of going through the vtable. (I assume from the .pxd file you can at least determine the mangled name of the base's tp_new function?) Carl From kwmsmith at gmail.com Wed May 6 20:50:27 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Wed, 6 May 2009 13:50:27 -0500 Subject: [Cython] endianness bug fix in numpy_test.pyx In-Reply-To: <4A01C713.7080302@student.matnat.uio.no> References: <49E991D7.5000809@student.matnat.uio.no> <49E9CE21.8040105@student.matnat.uio.no> <4A01C713.7080302@student.matnat.uio.no> Message-ID: On Wed, May 6, 2009 at 12:21 PM, Dag Sverre Seljebotn wrote: > Could you try again with current cython-devel at some point? I believe > this should be fixed now but obviously it should be tested on a PPC. Using changeset 2028:1377b8aff3a1 from cython-devel. $ uname -a Darwin ksmith.physics.wisc.edu 8.11.0 Darwin Kernel Version 8.11.0: Wed Oct 10 18:26:00 PDT 2007; root:xnu-792.24.17~1/RELEASE_PPC Power Macintosh powerpc $ python -V Python 2.5.2 $ python runtests.py -vv Yields 3 failures and 7 errors. None seem related to the endianness issues; the 7 errors are related to recent fixes to Buffer.py. I'm attaching the full output -- copied here are just the failure summaries: ====================================================================== FAIL: Doctest: cdef_setitem_T284 ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/doctest.py", line 2128, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for cdef_setitem_T284 File "/Users/ksmith/Devel/cython/cython-devel/BUILD/run/c/cdef_setitem_T284.so", line unknown line number, in cdef_setitem_T284 ---------------------------------------------------------------------- File "/Users/ksmith/Devel/cython/cython-devel/BUILD/run/c/cdef_setitem_T284.so", line ?, in cdef_setitem_T284 Failed example: test_list(range(11), "invalid index", None) Expected: Traceback (most recent call last): ... TypeError: list indices must be integers, not str Got: Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/doctest.py", line 1228, in __run compileflags, 1) in test.globs File "", line 1, in test_list(range(11), "invalid index", None) File "cdef_setitem_T284.pyx", line 26, in cdef_setitem_T284.test_list (cdef_setitem_T284.c:606) TypeError: list indices must be integers ====================================================================== FAIL: Doctest: cdef_setitem_T284 ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/doctest.py", line 2128, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for cdef_setitem_T284 File "/Users/ksmith/Devel/cython/cython-devel/BUILD/run/cpp/cdef_setitem_T284.so", line unknown line number, in cdef_setitem_T284 ---------------------------------------------------------------------- File "/Users/ksmith/Devel/cython/cython-devel/BUILD/run/cpp/cdef_setitem_T284.so", line ?, in cdef_setitem_T284 Failed example: test_list(range(11), "invalid index", None) Expected: Traceback (most recent call last): ... TypeError: list indices must be integers, not str Got: Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/doctest.py", line 1228, in __run compileflags, 1) in test.globs File "", line 1, in test_list(range(11), "invalid index", None) File "cdef_setitem_T284.pyx", line 26, in cdef_setitem_T284.test_list (cdef_setitem_T284.cpp:606) TypeError: list indices must be integers ====================================================================== FAIL: Doctest: numpy_test ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/doctest.py", line 2128, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for numpy_test File "/Users/ksmith/Devel/cython/cython-devel/BUILD/run/c/numpy_test.so", line 159, in numpy_test ---------------------------------------------------------------------- File "/Users/ksmith/Devel/cython/cython-devel/BUILD/run/c/numpy_test.so", line 326, in numpy_test Failed example: test_bad_cast() Expected: Traceback (most recent call last): ... ValueError: Item size of buffer (1 byte) does not match size of 'long' (8 bytes) Got: Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/doctest.py", line 1228, in __run compileflags, 1) in test.globs File "", line 1, in test_bad_cast() File "numpy_test.pyx", line 353, in numpy_test.test_bad_cast (numpy_test.c:5123) ValueError: Item size of buffer (1 byte) does not match size of 'long' (4 bytes) ---------------------------------------------------------------------- -------------- next part -------------- A non-text attachment was scrubbed... Name: runtests.tgz Type: application/x-gzip Size: 10564 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090506/c2601bcd/attachment.bin From dagss at student.matnat.uio.no Wed May 6 21:16:31 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 May 2009 21:16:31 +0200 Subject: [Cython] endianness bug fix in numpy_test.pyx In-Reply-To: References: <49E991D7.5000809@student.matnat.uio.no> <49E9CE21.8040105@student.matnat.uio.no> <4A01C713.7080302@student.matnat.uio.no> Message-ID: <4A01E20F.2080506@student.matnat.uio.no> Kurt Smith wrote: > On Wed, May 6, 2009 at 12:21 PM, Dag Sverre Seljebotn > wrote: > >> Could you try again with current cython-devel at some point? I believe >> this should be fixed now but obviously it should be tested on a PPC. > > Using changeset 2028:1377b8aff3a1 from cython-devel. > > $ uname -a > Darwin ksmith.physics.wisc.edu 8.11.0 Darwin Kernel Version 8.11.0: > Wed Oct 10 18:26:00 PDT 2007; root:xnu-792.24.17~1/RELEASE_PPC Power > Macintosh powerpc > > $ python -V > Python 2.5.2 > > $ python runtests.py -vv > > Yields 3 failures and 7 errors. None seem related to the endianness > issues; the 7 errors are related to recent fixes to Buffer.py. Doh; I forgot to test buffers on C++! Anyway your log was helpful in other areas too, thanks! -- Dag Sverre From robertwb at math.washington.edu Wed May 6 21:24:13 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 12:24:13 -0700 Subject: [Cython] Storing many, many instances In-Reply-To: References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A017E4F.5050800@student.matnat.uio.no> <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A0188D4.1050208@student.matnat.uio.no> <11E61C9F-3175-4473-8B39-20277284E9D1@math.washington.edu> Message-ID: On May 6, 2009, at 11:18 AM, Carl Witty wrote: > On Wed, May 6, 2009 at 10:38 AM, Robert Bradshaw > wrote: >> I already implemented this optimization, though only for cdef classes >> within the same module. (If they're small enough, they may get even >> inlined by the C compiler.) I'm not sure how possible this would be >> to do across modules, but one difficulty is that the contents of >> tp_new can change based on the .pyx file, not just the .pxd file. > > Really? What in the .pyx file can change tp_new? (I'm mostly just > curious.) The __cinit__ (aka __new__) function, if any, get's invoked. > Across modules, you should at least be able to call the tp_new > function directly instead of going through the vtable. (I assume from > the .pxd file you can at least determine the mangled name of the > base's tp_new function?) This is the optimization I was referring too--look at the generated sources for, e.g., sage.structure.element.pyx - Robert From robertwb at math.washington.edu Wed May 6 21:30:17 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 12:30:17 -0700 Subject: [Cython] Syntax to declare a c++ class? Message-ID: Just trying to get a syntax, how about cdef extern from "header.h": cdef cclass MyClass[TemplateParamers](BaseCPPClass): int member int method(int) The only concern I have is whether or not "cdef cclass" is to similar to "cdef class" (probably not). Any other ideas? - Robert From dalcinl at gmail.com Wed May 6 21:46:49 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 6 May 2009 16:46:49 -0300 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: Message-ID: On Wed, May 6, 2009 at 4:30 PM, Robert Bradshaw wrote: > Just trying to get a syntax, how about > > cdef extern from "header.h": > ? ? cdef cclass MyClass[TemplateParamers](BaseCPPClass): > ? ? ? ? int member > ? ? ? ? int method(int) > > The only concern I have is whether or not "cdef cclass" is to similar > to "cdef class" (probably not). Any other ideas? > Any chance you do not need 'cclass' and 'struct' would be enough (perhaps with ctypedef instead of cdef)? Cython will not be able to access private stuff from C++ classes... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Wed May 6 21:48:42 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 6 May 2009 21:48:42 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: Message-ID: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> Robert Bradshaw wrote: > Just trying to get a syntax, how about > > cdef extern from "header.h": > cdef cclass MyClass[TemplateParamers](BaseCPPClass): > int member > int method(int) > > The only concern I have is whether or not "cdef cclass" is to similar > to "cdef class" (probably not). Any other ideas? cdef cppclass? Yes, I think cclass is to similar to class. Anyway, when I proposed a Fortran syntax earlier you and Stefan made a good point about the language specification belonging in the "cdef extern" part, as, after all, the "header.h" file contains only C++ code. Also consider that C++ needs namespace specifications, for instance cdef extern ...: namespace std: cdef cppclass vector[T]: ... or cdef cclass std::vector[T]: ... Dag Sverre From carl.witty at gmail.com Wed May 6 22:08:46 2009 From: carl.witty at gmail.com (Carl Witty) Date: Wed, 6 May 2009 13:08:46 -0700 Subject: [Cython] Storing many, many instances In-Reply-To: References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A017E4F.5050800@student.matnat.uio.no> <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A0188D4.1050208@student.matnat.uio.no> <11E61C9F-3175-4473-8B39-20277284E9D1@math.washington.edu> Message-ID: On Wed, May 6, 2009 at 12:24 PM, Robert Bradshaw wrote: > On May 6, 2009, at 11:18 AM, Carl Witty wrote: > >> On Wed, May 6, 2009 at 10:38 AM, Robert Bradshaw >> wrote: >>> I already implemented this optimization, though only for cdef classes >>> within the same module. (If they're small enough, they may get even >>> inlined by the C compiler.) I'm not sure how possible this would be >>> to do across modules, but one difficulty is that the contents of >>> tp_new can change based on the .pyx file, not just the .pxd file. >> >> Really? ?What in the .pyx file can change tp_new? ?(I'm mostly just >> curious.) > > The __cinit__ (aka __new__) function, if any, get's invoked. So if we decided that the optimization was important enough, we could fix this by requiring that __cinit__ functions be declared in the .pxd file. >> Across modules, you should at least be able to call the tp_new >> function directly instead of going through the vtable. ?(I assume from >> the .pxd file you can at least determine the mangled name of the >> base's tp_new function?) > > This is the optimization I was referring too--look at the generated > sources for, e.g., sage.structure.element.pyx Yes, but you said that you only implement it within a single module (which is what I see in element.c); I'm saying that it should also be possible to implement across modules (just make tp_new functions non-static, add an extern declaration in the second module, and call the function directly). Carl From robertwb at math.washington.edu Wed May 6 22:12:26 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 13:12:26 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> Message-ID: On May 6, 2009, at 12:48 PM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> Just trying to get a syntax, how about >> >> cdef extern from "header.h": >> cdef cclass MyClass[TemplateParamers](BaseCPPClass): >> int member >> int method(int) >> >> The only concern I have is whether or not "cdef cclass" is to similar >> to "cdef class" (probably not). Any other ideas? > > cdef cppclass? I don't like this one, but I think just for aesthetic reasons. Nor cpp_class. > Yes, I think cclass is to similar to class. > > Anyway, when I proposed a Fortran syntax earlier you and Stefan made a > good point about the language specification belonging in the "cdef > extern" > part, as, after all, the "header.h" file contains only C++ code. Will the compiler need to know that header.h is C++ code? If not, I see no reason to declare it. > Also consider that C++ needs namespace specifications, for instance > > cdef extern ...: > namespace std: > cdef cppclass vector[T]: ... > > or > > cdef cclass std::vector[T]: ... Yep. That too. I'm a fan of both declarations. (Or even rolling the "namespace" into the cdef extern statement.) - Robert From robertwb at math.washington.edu Wed May 6 22:17:07 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 13:17:07 -0700 Subject: [Cython] Storing many, many instances In-Reply-To: References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A017E4F.5050800@student.matnat.uio.no> <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A0188D4.1050208@student.matnat.uio.no> <11E61C9F-3175-4473-8B39-20277284E9D1@math.washington.edu> Message-ID: <4B519262-77AC-4BC4-A62B-026B4E254FAA@math.washington.edu> On May 6, 2009, at 1:08 PM, Carl Witty wrote: > On Wed, May 6, 2009 at 12:24 PM, Robert Bradshaw > wrote: >> On May 6, 2009, at 11:18 AM, Carl Witty wrote: >> >>> On Wed, May 6, 2009 at 10:38 AM, Robert Bradshaw >>> wrote: >>>> I already implemented this optimization, though only for cdef >>>> classes >>>> within the same module. (If they're small enough, they may get even >>>> inlined by the C compiler.) I'm not sure how possible this would be >>>> to do across modules, but one difficulty is that the contents of >>>> tp_new can change based on the .pyx file, not just the .pxd file. >>> >>> Really? What in the .pyx file can change tp_new? (I'm mostly just >>> curious.) >> >> The __cinit__ (aka __new__) function, if any, get's invoked. > > So if we decided that the optimization was important enough, we could > fix this by requiring that __cinit__ functions be declared in the .pxd > file. > >>> Across modules, you should at least be able to call the tp_new >>> function directly instead of going through the vtable. (I assume >>> from >>> the .pxd file you can at least determine the mangled name of the >>> base's tp_new function?) >> >> This is the optimization I was referring too--look at the generated >> sources for, e.g., sage.structure.element.pyx > > Yes, but you said that you only implement it within a single module > (which is what I see in element.c); I'm saying that it should also be > possible to implement across modules (just make tp_new functions > non-static, add an extern declaration in the second module, and call > the function directly). Yes, I we could do this. We could also support exported cdef functions this way, instead of the current function import mechanism. (Or was there some reason it wasn't done this way in the first place? Certainly it results in "kinder" errors.) - Robert From dagss at student.matnat.uio.no Wed May 6 22:33:42 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 6 May 2009 22:33:42 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> Message-ID: <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> Robert Bradshaw wrote: > On May 6, 2009, at 12:48 PM, Dag Sverre Seljebotn wrote: > >> Robert Bradshaw wrote: >>> Just trying to get a syntax, how about >>> >>> cdef extern from "header.h": >>> cdef cclass MyClass[TemplateParamers](BaseCPPClass): >>> int member >>> int method(int) >>> >>> The only concern I have is whether or not "cdef cclass" is to similar >>> to "cdef class" (probably not). Any other ideas? >> >> cdef cppclass? > > I don't like this one, but I think just for aesthetic reasons. Nor > cpp_class. Myself I'm used to a .cpp suffix on C++ files; I suppose people who are used to the competing .cc will have stronger associations with your proposal. "cclass" gives me no associations at all initially. >> Yes, I think cclass is to similar to class. >> >> Anyway, when I proposed a Fortran syntax earlier you and Stefan made a >> good point about the language specification belonging in the "cdef >> extern" >> part, as, after all, the "header.h" file contains only C++ code. > > Will the compiler need to know that header.h is C++ code? If not, I > see no reason to declare it. No -- it's just the fact that usually all classes within the extern block will be a C++ class. I was thinking just plain "cdef class" could then change meaning, which would have a happy association with the heap-allocation thing (which C++ classes would share with existing cdef classes). cdef extern "C++" from "myheader.h": cdef class A: ... OTOH, if heap-allocation-and-refcounted semantics isn't selected, one should use "cdef struct" instead, like Lisandro suggests. In the unlikely event that you have both a C++ class and a Python extension class in the same header you can always repeat the extern statement. After all, C++ uses extern "C" { } to interface with C. > >> Also consider that C++ needs namespace specifications, for instance >> >> cdef extern ...: >> namespace std: >> cdef cppclass vector[T]: ... >> >> or >> >> cdef cclass std::vector[T]: ... > > Yep. That too. I'm a fan of both declarations. (Or even rolling the > "namespace" into the cdef extern statement.) FWIW I don't like the second one. I like rolling it into the extern statement. A single header file can declare things for many namespaces, but it is not common and you can always use two cdef extern blocks for the same header in such cases. Dag Sverre From kwmsmith at gmail.com Wed May 6 23:41:12 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Wed, 6 May 2009 16:41:12 -0500 Subject: [Cython] Buffer passing implementation details In-Reply-To: <4A01BB4A.1060003@student.matnat.uio.no> References: <4A01BB4A.1060003@student.matnat.uio.no> Message-ID: On Wed, May 6, 2009 at 11:31 AM, Dag Sverre Seljebotn wrote: > (Kurt is the major recipient of this.) > > I've been thinking some more on buffer passing. In the end (and perhaps > even in summer; I could perhaps have a go at it working alongside you) > the scenario we are looking at is something like > > external_set_foo_array(foo_handle, self.arr[2::2]) > > with fast operations all the way. > > That is > a) Efficient slicing without Python overhead (#178) > b) Storing acquired buffers in cdef class fields (#301) > c) External functions can keep a reference to the buffer (not really > necesarry, but it is necesarry for internal Cython "cdef" functions, and > it would be nice to treat them the same) > d) The base pointer may have to be moved (again slicing can do this) > > This seems to make it clear that > a) The Py_buffer is not suitable as the primary vessel of our buffer > data, e.g. to the external function. It can't work that well with > slicing as we must maintain the original Py_buffer data when releasing it. > b) We need a reference count. E.g. doing a slice, or assigning to > self.arr, would increase this count. This must go on the heap; and so we > might as well put the entire Py_buffer on the heap. Doing the slicing manually on the buffer data and having a refcounted heap-allocated Py_buffer would address the object-Py_buffer data synchronization problem, right? The above modifications would have that, once the buffer is acquired, all modifications to it are done at the buffer level. Although, what about when we require a contiguous copy (or any mode, for that matter) of a buffer? Would we set the top-level object reference to None, as suggested in a previous thread? That would prevent slicing at the Python API level, though, right? > > I have made a new ticket outlining the thoughts in detail in #311 and > added a comment in #299. #311 is not really in your direct interest for > GSoC but it is very tightly coupled with #299 so it would be good to > keep in the back of your mind anyway. > > Thoughts? General question: how difficult would it be for the Cython-side programmer to get the underlying char *data inside the Py_buffer, in case it needs to be accessed? It seems there are more than a few 'layers' -- are they all necessary? The justifications I see for each are the following: typedef struct { void *buf; PyObject *obj; Py_ssize_t len; Py_ssize_t itemsize; int readonly; int ndim; char *format; Py_ssize_t *shape; Py_ssize_t *strides; Py_ssize_t *suboffsets; void *internal; } Py_buffer; The "base layer" -- interface structure for tp_getbuffer from objects, etc. typedef struct { size_t refcount; Py_buffer bufinfo; } __Pyx_Buffer; The Cython wrapper around Py_buffer -- includes cython-managed refcounting upon assignment/slicing (other cases?) typedef struct { __Pyx_buffer* bufinfo; char* data; Py_ssize_t shape0, stride0; } __Pyx_StridedBuf_1D; The "int[:] buf" buffer -- exists to make the new buffer syntax more transparent (not part of GSoC, but putting in framework for it). char *data pointer to allow fast slicing w/o Python API (e.g. arr[10:]), shape0 & stride0 copied from buffer for optimizations. typedef struct { __Pyx_StridedBuf_1D buffer; PyObject* object; } __Pyx_StridedBuf_1D_Obj; The "object[int, mode='strided'] buf" object buffer. This is the top level and holds a reference to the original PyObject from which the buffer was acquired. Object reference required when a python-level access/assignment is required. So, initially, if one passes a 'buffer' to a function like so: external_func(arr[2::2]) What is the slicing modifying/updating? The top-level through the Python API layer via the PyObject? Or is it being done to the __Pyx_StridedBuf_1D members -- offsetting the char *data pointer by 2, and changing the shape0 and stride0 members? (Would it be the top layer object at first & for the GSoC, then when faster, low-level slicing is implemented, directly on the lower-level?) Besides Python API-level slicing, why is the top-level object reference required? Is it for buffer reacquisition through tp_getbuffer? That will be handled at the next lower level through a struct copy by value and increase of the refcount. It isn't for buffer release, either, since that doesn't require the original object reference explicitly. (There is an object reference at the Py_buffer level). Basically, nothing Cython-side would require the PyObject from which the original buffer is acquired once all these are implemented, right? (Again, not all part of GSoC, but at least putting in the framework.) This is something of me thinking out loud, so let me know confusing parts, etc. > It sheems a shame to let go of a neatly PEP-defined Py_buffer for > passing to external functions, but I think it won't be too bad if we > with each Cython version ship nice C header and Fortran include files > containing the appropriate structs and access macros. I guess the access macros would make it easy for the external functions to get to the important stuff, so my previous concern isn't so essential (removing the 'top level' layer from the buffer struct heirarchy). Kurt From dsurviver at gmail.com Wed May 6 23:53:33 2009 From: dsurviver at gmail.com (Danilo Freitas) Date: Wed, 6 May 2009 18:53:33 -0300 Subject: [Cython] template types In-Reply-To: References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <4A00AD43.1050709@behnel.de> <4A00AF71.7010103@behnel.de> <4A011CFE.7070006@behnel.de> <3439af420a85daac11ba6238c254c18f.squirrel@webmail.uio.no> <8B75D598-FF6D-4986-8E3D-0167BA5CE778@math.washington.edu> <4A014AEF.7030901@student.matnat.uio.no> Message-ID: <45239150905061453s62c8dc63p45fa1d483c731256@mail.gmail.com> Hey. I got a few late here. I think the template syntax needs to be simple, "readable" and easy to use and implement (I think this is obvious). I saw here some options really unreadable. I vote for the [] syntax. It's easy to understand and the __getitem__ could be used here. I think it will be easier to choose one syntax when somebody try to implement it (is anybody doing it?). This way, we get more concrete solutions (and problems) for it. Whatever, it's just my opinion :) From robertwb at math.washington.edu Thu May 7 00:52:47 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 15:52:47 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> Message-ID: <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> On May 6, 2009, at 1:33 PM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On May 6, 2009, at 12:48 PM, Dag Sverre Seljebotn wrote: >> >>> Robert Bradshaw wrote: >>>> Just trying to get a syntax, how about >>>> >>>> cdef extern from "header.h": >>>> cdef cclass MyClass[TemplateParamers](BaseCPPClass): >>>> int member >>>> int method(int) >>>> >>>> The only concern I have is whether or not "cdef cclass" is to >>>> similar >>>> to "cdef class" (probably not). Any other ideas? >>> >>> cdef cppclass? >> >> I don't like this one, but I think just for aesthetic reasons. Nor >> cpp_class. > > Myself I'm used to a .cpp suffix on C++ files; I suppose people who > are > used to the competing .cc will have stronger associations with your > proposal. "cclass" gives me no associations at all initially. I could get used to cppclass, but I'm hoping someone comes up with a name that's obviously the "right" one :). >>> Yes, I think cclass is to similar to class. >>> >>> Anyway, when I proposed a Fortran syntax earlier you and Stefan >>> made a >>> good point about the language specification belonging in the "cdef >>> extern" >>> part, as, after all, the "header.h" file contains only C++ code. >> >> Will the compiler need to know that header.h is C++ code? If not, I >> see no reason to declare it. > > No -- it's just the fact that usually all classes within the extern > block > will be a C++ class. I was thinking just plain "cdef class" could then > change meaning, which would have a happy association with the > heap-allocation thing (which C++ classes would share with existing > cdef > classes). > > cdef extern "C++" from "myheader.h": > cdef class A: ... I'm -1 on having "cdef class" mean two very different things. > OTOH, if heap-allocation-and-refcounted semantics isn't selected, one > should use "cdef struct" instead, like Lisandro suggests. Structs are also something different (well, there's a lot of similarity, but the idea of inheritance is a big one), especially if we're looking at heap allocated classes for the first iteration. (I don't think we want to try to do our own refcounting, let the user use new and delete directly if they want to use C++.) - Robert > From greg.ewing at canterbury.ac.nz Thu May 7 03:55:46 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 07 May 2009 13:55:46 +1200 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> Message-ID: <4A023FA2.3040001@canterbury.ac.nz> Robert Bradshaw wrote: > On May 6, 2009, at 1:33 PM, Dag Sverre Seljebotn wrote: >>cdef extern "C++" from "myheader.h": >> cdef class A: ... For Pyrex I'm thinking about using "cdef+" as a general prefix for C++-related declarations, e.g. cdef+ extern from "myheader.hpp": ... cdef+ struct Foo: ... -- Greg From dsurviver at gmail.com Thu May 7 04:26:13 2009 From: dsurviver at gmail.com (Danilo Freitas) Date: Wed, 6 May 2009 23:26:13 -0300 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <4A023FA2.3040001@canterbury.ac.nz> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <4A023FA2.3040001@canterbury.ac.nz> Message-ID: <45239150905061926g58c5cb97hd770669552e7b48a@mail.gmail.com> 2009/5/6 Greg Ewing : > Robert Bradshaw wrote: >> On May 6, 2009, at 1:33 PM, Dag Sverre Seljebotn wrote: >>>cdef extern "C++" from "myheader.h": >>> ? ?cdef class A: ... > > For Pyrex I'm thinking about using "cdef+" as a > general prefix for C++-related declarations, e.g. > > ? cdef+ extern from "myheader.hpp": > ? ? ... > > ? cdef+ struct Foo: > ? ? ... I think this is weird. But it's a solution. I don't see many problems with cppdef. From robertwb at math.washington.edu Thu May 7 04:37:12 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 19:37:12 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <45239150905061926g58c5cb97hd770669552e7b48a@mail.gmail.com> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <4A023FA2.3040001@canterbury.ac.nz> <45239150905061926g58c5cb97hd770669552e7b48a@mail.gmail.com> Message-ID: <40EB49CE-6598-46BA-912D-231D05295E06@math.washington.edu> On May 6, 2009, at 7:26 PM, Danilo Freitas wrote: > 2009/5/6 Greg Ewing : >> Robert Bradshaw wrote: >>> On May 6, 2009, at 1:33 PM, Dag Sverre Seljebotn wrote: >>>> cdef extern "C++" from "myheader.h": >>>> cdef class A: ... >> >> For Pyrex I'm thinking about using "cdef+" as a >> general prefix for C++-related declarations, e.g. >> >> cdef+ extern from "myheader.hpp": >> ... >> >> cdef+ struct Foo: >> ... > I think this is weird. But it's a solution. I don't see many problems > with cppdef. It's too close to cpdef I think. I asked about this during our sage dev meeting today and people seemed to like cdef cppclass Foo: .... - Robert From dsurviver at gmail.com Thu May 7 04:52:35 2009 From: dsurviver at gmail.com (Danilo Freitas) Date: Wed, 6 May 2009 23:52:35 -0300 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <40EB49CE-6598-46BA-912D-231D05295E06@math.washington.edu> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <4A023FA2.3040001@canterbury.ac.nz> <45239150905061926g58c5cb97hd770669552e7b48a@mail.gmail.com> <40EB49CE-6598-46BA-912D-231D05295E06@math.washington.edu> Message-ID: <45239150905061952r56d831d3md0e980a43af2909d@mail.gmail.com> > > It's too close to cpdef I think. I I agree. It may confuse a little. From dagss at student.matnat.uio.no Thu May 7 07:06:15 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 7 May 2009 07:06:15 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> Message-ID: <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> Robert Bradshaw wrote: > On May 6, 2009, at 1:33 PM, Dag Sverre Seljebotn wrote: >> OTOH, if heap-allocation-and-refcounted semantics isn't selected, one >> should use "cdef struct" instead, like Lisandro suggests. > > Structs are also something different (well, there's a lot of > similarity, but the idea of inheritance is a big one), especially if > we're looking at heap allocated classes for the first iteration. (I > don't think we want to try to do our own refcounting, let the user > use new and delete directly if they want to use C++.) Well, they are not something different in C++ :-) If you don't want to try your own refcounting, please, please tell me that the only allowed form of usage of the class will be cdef CppClass* obj Allowing cdef CppClass obj in any form and requiring a manual delete would be very confusing and difficult to remember. (That's what I mean by refcounting -- I imagined the difference between these two syntaxes would be that the latter of these would be refcounted; and subject for automatic stack-allocation-optimization.) Dag Sverre From stefan_ml at behnel.de Thu May 7 08:05:18 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 07 May 2009 08:05:18 +0200 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <4A023FA2.3040001@canterbury.ac.nz> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <4A023FA2.3040001@canterbury.ac.nz> Message-ID: <4A027A1E.7020604@behnel.de> Greg Ewing wrote: > Robert Bradshaw wrote: >> On May 6, 2009, at 1:33 PM, Dag Sverre Seljebotn wrote: >>> cdef extern "C++" from "myheader.h": >>> cdef class A: ... > > For Pyrex I'm thinking about using "cdef+" as a > general prefix for C++-related declarations, e.g. > > cdef+ extern from "myheader.hpp": > ... > > cdef+ struct Foo: > ... ... and if we ever support the D language, we'd need a 'ddef' or 'cdefd'? What about using some kind of qualifier as above, but more like cdef[c++] extern ... or cdef(c++) extern ... or even cdef("c++") extern ... to come a bit closer to pure Python code. That would be easily extensible to cdef[fortran] ... cdef[D] and would simply default to cdef[C] if no qualifier is provided in a plain "cdef". Stefan From stefan_ml at behnel.de Thu May 7 08:23:14 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 07 May 2009 08:23:14 +0200 Subject: [Cython] Storing many, many instances In-Reply-To: <4B519262-77AC-4BC4-A62B-026B4E254FAA@math.washington.edu> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A017E4F.5050800@student.matnat.uio.no> <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A0188D4.1050208@student.matnat.uio.no> <11E61C9F-3175-4473-8B39-20277284E9D1@math.washington.edu> <4B519262-77AC-4BC4-A62B-026B4E254FAA@math.washington.edu> Message-ID: <4A027E52.9070003@behnel.de> Robert Bradshaw wrote: > On May 6, 2009, at 1:08 PM, Carl Witty wrote: >> Yes, but you said that you only implement it within a single module >> (which is what I see in element.c); I'm saying that it should also be >> possible to implement across modules (just make tp_new functions >> non-static, add an extern declaration in the second module, and call >> the function directly). > > Yes, I we could do this. We could also support exported cdef > functions this way, instead of the current function import mechanism. > (Or was there some reason it wasn't done this way in the first place? There are two cdef function export mechanisms: "public" and "api". The second is targeted towards calling functions between separate extension modules (through "__pyx_capi"), while "public" (and declaration in the module .pxd file IIRC) simply removes the name mangling and makes functions non-static for use in other C files linked into the same extension module. So this would mimic the "public" mechanism in a way, although it'd be better to do the normal name mangling for tp_new() even if we export it. Stefan From robertwb at math.washington.edu Thu May 7 08:36:06 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 23:36:06 -0700 Subject: [Cython] Storing many, many instances In-Reply-To: <4A027E52.9070003@behnel.de> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A017E4F.5050800@student.matnat.uio.no> <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A0188D4.1050208@student.matnat.uio.no> <11E61C9F-3175-4473-8B39-20277284E9D1@math.washington.edu> <4B519262-77AC-4BC4-A62B-026B4E254FAA@math.washington.edu> <4A027E52.9070003@behnel.de> Message-ID: <8C4356C0-2011-4519-A5EB-4476E63EA971@math.washington.edu> On May 6, 2009, at 11:23 PM, Stefan Behnel wrote: > Robert Bradshaw wrote: >> On May 6, 2009, at 1:08 PM, Carl Witty wrote: >>> Yes, but you said that you only implement it within a single module >>> (which is what I see in element.c); I'm saying that it should >>> also be >>> possible to implement across modules (just make tp_new functions >>> non-static, add an extern declaration in the second module, and call >>> the function directly). >> >> Yes, I we could do this. We could also support exported cdef >> functions this way, instead of the current function import mechanism. >> (Or was there some reason it wasn't done this way in the first place? > > There are two cdef function export mechanisms: "public" and "api". The > second is targeted towards calling functions between separate > extension > modules (through "__pyx_capi"), while "public" (and declaration in the > module .pxd file IIRC) simply removes the name mangling and makes > functions > non-static for use in other C files linked into the same extension > module. > > So this would mimic the "public" mechanism in a way, although it'd be > better to do the normal name mangling for tp_new() even if we > export it. Yes, I'm thinking of "api" functions, mangled and all, but without the __pyx_capi object. Actually, I don't know if this would work, as the .so files may not be loaded in the right order. - Robert From robertwb at math.washington.edu Thu May 7 08:50:49 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 23:50:49 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <4A027A1E.7020604@behnel.de> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <4A023FA2.3040001@canterbury.ac.nz> <4A027A1E.7020604@behnel.de> Message-ID: On May 6, 2009, at 11:05 PM, Stefan Behnel wrote: > > Greg Ewing wrote: >> Robert Bradshaw wrote: >>> On May 6, 2009, at 1:33 PM, Dag Sverre Seljebotn wrote: >>>> cdef extern "C++" from "myheader.h": >>>> cdef class A: ... >> >> For Pyrex I'm thinking about using "cdef+" as a >> general prefix for C++-related declarations, e.g. >> >> cdef+ extern from "myheader.hpp": >> ... >> >> cdef+ struct Foo: >> ... > > ... and if we ever support the D language, we'd need a 'ddef' or > 'cdefd'? > > What about using some kind of qualifier as above, but more like > > cdef[c++] extern ... > > or > > cdef(c++) extern ... > > or even > > cdef("c++") extern ... > > to come a bit closer to pure Python code. > > That would be easily extensible to > > cdef[fortran] ... > cdef[D] > > and would simply default to > > cdef[C] > > if no qualifier is provided in a plain "cdef". Parameterized keywords? Sorry, but that hurts my eyes... :( The only time we'd need to declare a language is for external declarations. I'd rather have cdef extern from "foo.h" language "cpp": ... or cdef extern from cpp "foo.h": ... Actually, the whole cdef keyword here is redundant, maybe one could just write "extern from ..." (though I do like the fact that nearly all non-Python syntax/blocks begin with a cdef). I think this is somewhat tangential to the question of how to declare classes, unless one wants "class" or "struct" to have new meanings in different contexts (which I don't like). - Robert From robertwb at math.washington.edu Thu May 7 09:03:02 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 May 2009 00:03:02 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> Message-ID: <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> On May 6, 2009, at 10:06 PM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On May 6, 2009, at 1:33 PM, Dag Sverre Seljebotn wrote: >>> OTOH, if heap-allocation-and-refcounted semantics isn't selected, >>> one >>> should use "cdef struct" instead, like Lisandro suggests. >> >> Structs are also something different (well, there's a lot of >> similarity, but the idea of inheritance is a big one), especially if >> we're looking at heap allocated classes for the first iteration. (I >> don't think we want to try to do our own refcounting, let the user >> use new and delete directly if they want to use C++.) > > Well, they are not something different in C++ :-) A C++ class is a glorified struct, but one still has plain old structs too. > If you don't want to try your own refcounting, please, please tell > me that > the only allowed form of usage of the class will be > > cdef CppClass* obj This is what I imagined (for now at least). > Allowing > > cdef CppClass obj > > in any form and requiring a manual delete would be very confusing and > difficult to remember. > > (That's what I mean by refcounting -- I imagined the difference > between > these two syntaxes would be that the latter of these would be > refcounted; > and subject for automatic stack-allocation-optimization.) By "refcounting", are you implying global garbage collection of some sort, just like Python objects, or something that emulates being on the stack (e.g. objects are deleted when the scope exits, and all assignments are done by value)? - Robert From dagss at student.matnat.uio.no Thu May 7 09:12:26 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 07 May 2009 09:12:26 +0200 Subject: [Cython] Storing many, many instances In-Reply-To: <8C4356C0-2011-4519-A5EB-4476E63EA971@math.washington.edu> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A017E4F.5050800@student.matnat.uio.no> <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A0188D4.1050208@student.matnat.uio.no> <11E61C9F-3175-4473-8B39-20277284E9D1@math.washington.edu> <4B519262-77AC-4BC4-A62B-026B4E254FAA@math.washington.edu> <4A027E52.9070003@behnel.de> <8C4356C0-2011-4519-A5EB-4476E63EA971@math.washington.edu> Message-ID: <4A0289DA.4000009@student.matnat.uio.no> Robert Bradshaw wrote: > On May 6, 2009, at 11:23 PM, Stefan Behnel wrote: > >> Robert Bradshaw wrote: >>> On May 6, 2009, at 1:08 PM, Carl Witty wrote: >>>> Yes, but you said that you only implement it within a single module >>>> (which is what I see in element.c); I'm saying that it should >>>> also be >>>> possible to implement across modules (just make tp_new functions >>>> non-static, add an extern declaration in the second module, and call >>>> the function directly). >>> Yes, I we could do this. We could also support exported cdef >>> functions this way, instead of the current function import mechanism. >>> (Or was there some reason it wasn't done this way in the first place? >> There are two cdef function export mechanisms: "public" and "api". The >> second is targeted towards calling functions between separate >> extension >> modules (through "__pyx_capi"), while "public" (and declaration in the >> module .pxd file IIRC) simply removes the name mangling and makes >> functions >> non-static for use in other C files linked into the same extension >> module. >> >> So this would mimic the "public" mechanism in a way, although it'd be >> better to do the normal name mangling for tp_new() even if we >> export it. > > Yes, I'm thinking of "api" functions, mangled and all, but without > the __pyx_capi object. Actually, I don't know if this would work, as > the .so files may not be loaded in the right order. And by default, the .so files are loaded with RTLD_PRIVATE, not RTLD_GLOBAL, so that symbols in one isn't available in another. I think you'd have a hard time getting it to work reliably; but if you do it would be cool as it would speed up cross-Cython-module code (?). One could do a combination approach where __pyx_capi is available but a custom Cython classloader (like pyximport) would load the .so as RTLD_GLOBAL. A bit convoluted though, and I wonder whether it should be a priority... In the end, there are "inlineable" functions which are so small that the overhead matters, and "non-inlineable" which do so much anyway that it doesn't matter. The former can be put in inline __init__ code in pxd files, while the latter should be able to live with __pyx_capi. -- Dag Sverre From dagss at student.matnat.uio.no Thu May 7 09:24:52 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 07 May 2009 09:24:52 +0200 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> Message-ID: <4A028CC4.6040903@student.matnat.uio.no> Robert Bradshaw wrote: > On May 6, 2009, at 10:06 PM, Dag Sverre Seljebotn wrote: > >> Robert Bradshaw wrote: >>> On May 6, 2009, at 1:33 PM, Dag Sverre Seljebotn wrote: >>>> OTOH, if heap-allocation-and-refcounted semantics isn't selected, >>>> one >>>> should use "cdef struct" instead, like Lisandro suggests. >>> Structs are also something different (well, there's a lot of >>> similarity, but the idea of inheritance is a big one), especially if >>> we're looking at heap allocated classes for the first iteration. (I >>> don't think we want to try to do our own refcounting, let the user >>> use new and delete directly if they want to use C++.) >> Well, they are not something different in C++ :-) > > A C++ class is a glorified struct, but one still has plain old > structs too. ...which is the same as a class with only public fields in them :-) What I'm getting here at is that we are creating a distinction in Cython which actually does not exist in C++ (rather C++ adds member methods, constructors, private members, and vtables all as orthogonal and independent extensions). But I think that is how it should be. >> If you don't want to try your own refcounting, please, please tell >> me that >> the only allowed form of usage of the class will be >> >> cdef CppClass* obj > > This is what I imagined (for now at least). > >> Allowing >> >> cdef CppClass obj >> >> in any form and requiring a manual delete would be very confusing and >> difficult to remember. >> >> (That's what I mean by refcounting -- I imagined the difference >> between >> these two syntaxes would be that the latter of these would be >> refcounted; >> and subject for automatic stack-allocation-optimization.) > > By "refcounting", are you implying global garbage collection of some > sort, just like Python objects, or something that emulates being on > the stack (e.g. objects are deleted when the scope exits, and all > assignments are done by value)? I was referring to something which have the full semantics of Python objects (after the result of the "New idea for C++ stack allocation" thread, which expored the opposite idea). So cdef CppClass obj = None obj.do_something() # Illegal, it is None obj = CppClass(3) obj.do_something() # OK ...and so on, just like cdef classes. Then stack allocation would be a pure transparent optimization (an important one; you don't really want to be heap-allocating STL iterators for each iteration!) where semantics can be preserved. When interfacing with external code; like with my favourite example: cdef extern from "vector" namespace std: cdef cppclass vector[T]: cdef cppclass iterator: # next/prev operator overloading somehow, # we still disagree here I guess iterator begin() then when one does cdef vector[int].iterator it it = myvec.begin() it would mean getting hold of new std::vector::iterator(myvec.begin()) and refcounting it -- however, since begin() returns by value, it would be legal to instead allocate on stack, as long as semantics are preserved. (Perhaps just changing this to using placement new and skip refcounting.) The alternative to this seems to be the ideas you already rejected. And I must say I like emulating cdef classes better. -- Dag Sverre From stefan_ml at behnel.de Thu May 7 09:39:09 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 7 May 2009 09:39:09 +0200 (CEST) Subject: [Cython] Storing many, many instances In-Reply-To: <8C4356C0-2011-4519-A5EB-4476E63EA971@math.washington.edu> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A017E4F.5050800@student.matnat.uio.no> <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A0188D4.1050208@student.matnat.uio.no> <11E61C9F-3175-4473-8B39-20277284E9D1@math.washington.edu> <4B519262-77AC-4BC4-A62B-026B4E254FAA@math.washington.edu> <4A027E52.9070003@behnel.de> <8C4356C0-2011-4519-A5EB-4476E63EA971@math.washington.edu> Message-ID: <6c13a7cafc2a4a8deff53300fbffb1a9.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Robert Bradshaw wrote: > Yes, I'm thinking of "api" functions, mangled and all, but without > the __pyx_capi object. Actually, I don't know if this would work, as > the .so files may not be loaded in the right order. Well, it wouldn't work due to symbol export restrictions (as Dag noted). Plus, you'd loose a lot for basically no gain. The current mechanism is arbitrarily portable, almost as fast as a direct call, and provides a simple form of module encapsulation to support future API extensions without breaking existing modules. If you want inlined functions and/or direct calls, use .pxd inline functions and/or include or link your file into the module, instead of using separate modules. Stefan From stefan_ml at behnel.de Thu May 7 09:42:00 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 7 May 2009 09:42:00 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <4A023FA2.3040001@canterbury.ac.nz> <4A027A1E.7020604@behnel.de> Message-ID: Robert Bradshaw wrote: > On May 6, 2009, at 11:05 PM, Stefan Behnel wrote: >> cdef[c++] extern ... > > Parameterized keywords? Sorry, but that hurts my eyes... :( Ok, not on keywords then. What about cdef extern[c++] ... ? Stefan From robertwb at math.washington.edu Thu May 7 09:42:45 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 May 2009 00:42:45 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <4A028CC4.6040903@student.matnat.uio.no> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> Message-ID: On May 7, 2009, at 12:24 AM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On May 6, 2009, at 10:06 PM, Dag Sverre Seljebotn wrote: >> >>> Robert Bradshaw wrote: >>>> On May 6, 2009, at 1:33 PM, Dag Sverre Seljebotn wrote: >>>>> OTOH, if heap-allocation-and-refcounted semantics isn't selected, >>>>> one >>>>> should use "cdef struct" instead, like Lisandro suggests. >>>> Structs are also something different (well, there's a lot of >>>> similarity, but the idea of inheritance is a big one), >>>> especially if >>>> we're looking at heap allocated classes for the first iteration. (I >>>> don't think we want to try to do our own refcounting, let the user >>>> use new and delete directly if they want to use C++.) >>> Well, they are not something different in C++ :-) >> >> A C++ class is a glorified struct, but one still has plain old >> structs too. > > ...which is the same as a class with only public fields in them :-) > > What I'm getting here at is that we are creating a distinction in > Cython > which actually does not exist in C++ (rather C++ adds member methods, > constructors, private members, and vtables all as orthogonal and > independent extensions). > > But I think that is how it should be. Yep, if C++ considers them different enough to allocate two separate keywords for them, than so should we. (I think it'll also be less confusing). >>> If you don't want to try your own refcounting, please, please tell >>> me that >>> the only allowed form of usage of the class will be >>> >>> cdef CppClass* obj >> >> This is what I imagined (for now at least). >> >>> Allowing >>> >>> cdef CppClass obj >>> >>> in any form and requiring a manual delete would be very confusing >>> and >>> difficult to remember. >>> >>> (That's what I mean by refcounting -- I imagined the difference >>> between >>> these two syntaxes would be that the latter of these would be >>> refcounted; >>> and subject for automatic stack-allocation-optimization.) >> >> By "refcounting", are you implying global garbage collection of some >> sort, just like Python objects, or something that emulates being on >> the stack (e.g. objects are deleted when the scope exits, and all >> assignments are done by value)? > > I was referring to something which have the full semantics of Python > objects (after the result of the "New idea for C++ stack allocation" > thread, which expored the opposite idea). > > So > > cdef CppClass obj = None > obj.do_something() # Illegal, it is None > obj = CppClass(3) > obj.do_something() # OK > > ...and so on, just like cdef classes. > > Then stack allocation would be a pure transparent optimization (an > important one; you don't really want to be heap-allocating STL > iterators > for each iteration!) where semantics can be preserved. I agree it makes stuff like STL iterators nice to work with, but can one do cdef CppClass x(): obj = CppClass(3) return obj Or will it be freed on exit unlike a Python object? Also, what about def y(): cdef CppClass a = CppClass(5) cdef CppClass b = a return # don't free a twice... Or perhaps you're thinking about wrapping C++ class instances inside actual objects? - Robert From robertwb at math.washington.edu Thu May 7 09:46:20 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 May 2009 00:46:20 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <4A023FA2.3040001@canterbury.ac.nz> <4A027A1E.7020604@behnel.de> Message-ID: <47F3AE68-2D7B-44DD-90E7-14587EEB0D33@math.washington.edu> On May 7, 2009, at 12:42 AM, Stefan Behnel wrote: > Robert Bradshaw wrote: >> On May 6, 2009, at 11:05 PM, Stefan Behnel wrote: >>> cdef[c++] extern ... >> >> Parameterized keywords? Sorry, but that hurts my eyes... :( > > Ok, not on keywords then. What about > > cdef extern[c++] ... "extern" feels a lot like a keyword too. - Robert From dagss at student.matnat.uio.no Thu May 7 09:58:11 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 7 May 2009 09:58:11 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> Message-ID: Robert Bradshaw wrote: > On May 7, 2009, at 12:24 AM, Dag Sverre Seljebotn wrote: >> What I'm getting here at is that we are creating a distinction in >> Cython >> which actually does not exist in C++ (rather C++ adds member methods, >> constructors, private members, and vtables all as orthogonal and >> independent extensions). >> >> But I think that is how it should be. > > Yep, if C++ considers them different enough to allocate two separate > keywords for them, than so should we. (I think it'll also be less > confusing). But C++ doesn't!! class Foo { int x; public: int y; void z(); }; is EXACTLY the same as struct Foo { int y; void z() private: int x; }; Whatever the distinction is supposed to be in Cython, it cannot be the default private/public, as only the public stuff is available to us anyway! I've been using the distinction to mean "these structs should be reference-counted, not have stack-allocated semantics like we use for C structs". If one doesn't plan to ever go for a reference counted scheme, we should just use plain struct. >>>> If you don't want to try your own refcounting, please, please tell >>>> me that >>>> the only allowed form of usage of the class will be >>>> >>>> cdef CppClass* obj >>> >>> This is what I imagined (for now at least). >>> >>>> Allowing >>>> >>>> cdef CppClass obj >>>> >>>> in any form and requiring a manual delete would be very confusing >>>> and >>>> difficult to remember. >>>> >>>> (That's what I mean by refcounting -- I imagined the difference >>>> between >>>> these two syntaxes would be that the latter of these would be >>>> refcounted; >>>> and subject for automatic stack-allocation-optimization.) >>> >>> By "refcounting", are you implying global garbage collection of some >>> sort, just like Python objects, or something that emulates being on >>> the stack (e.g. objects are deleted when the scope exits, and all >>> assignments are done by value)? >> >> I was referring to something which have the full semantics of Python >> objects (after the result of the "New idea for C++ stack allocation" >> thread, which expored the opposite idea). >> >> So >> >> cdef CppClass obj = None >> obj.do_something() # Illegal, it is None >> obj = CppClass(3) >> obj.do_something() # OK >> >> ...and so on, just like cdef classes. >> >> Then stack allocation would be a pure transparent optimization (an >> important one; you don't really want to be heap-allocating STL >> iterators >> for each iteration!) where semantics can be preserved. > > I agree it makes stuff like STL iterators nice to work with, but can > one do > > cdef CppClass x(): > obj = CppClass(3) > return obj > > Or will it be freed on exit unlike a Python object? Also, what about > > def y(): > cdef CppClass a = CppClass(5) > cdef CppClass b = a > return # don't free a twice... > > Or perhaps you're thinking about wrapping C++ class instances inside > actual objects? Yes! Allocation would happen through an actual (as light-weight as possible, and probably mostly opaque, perhaps doing as much as printing the C++ name of the class in its repr) Python object; and assigning it to "cdef CppClass a" would basically make Cython automatically do all accesses (except passing around references) on the contained pointer. Again -- this is a larger debate with huge consequences for how one should do operator overloading etc. For instance, += would have to be implemented like it += 1 would mean it = please_refcount_iteratorclassname(new std::vector::iterator(it + 1)) However, this kind of semantics ensures that we avoid that cdef vector[int].iterator a, b a = vec.begin() b = a a += 1 # also changes b which seems to be the alternative (where a and b are just kept on the stack and we somehow circumvent the constructor business). This is, BTW, a good reason why I proposed calling operators by "rewrapping" them, because then a = vec.begin() b = a a.next() # also changes b, through calling a++ makes perfect sense! I'm happy that only "CppClass*" will be allowed at first, which takes much of this worry away. However note that this might have consequences for how operators should be defined (I haven't seen a concrete proposal for what you are thinking there so I don't know how it would fare). Dag Sverre From stefan_ml at behnel.de Thu May 7 10:10:34 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 7 May 2009 10:10:34 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <47F3AE68-2D7B-44DD-90E7-14587EEB0D33@math.washington.edu> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <4A023FA2.3040001@canterbury.ac.nz> <4A027A1E.7020604@behnel.de> <47F3AE68-2D7B-44DD-90E7-14587EEB0D33@math.washington.edu> Message-ID: Robert Bradshaw wrote: > On May 7, 2009, at 12:42 AM, Stefan Behnel wrote: > >> Robert Bradshaw wrote: >>> On May 6, 2009, at 11:05 PM, Stefan Behnel wrote: >>>> cdef[c++] extern ... >>> >>> Parameterized keywords? Sorry, but that hurts my eyes... :( >> >> Ok, not on keywords then. What about >> >> cdef extern[c++] ... > > "extern" feels a lot like a keyword too. Fine, but you get the idea, right? In a way, the declarations *are* parametrised in that they come from different languages. I think it makes sense to add that to the "extern" 'keyword' to say: "this external declaration obeys C++ semantics". Stefan From stefan_ml at behnel.de Thu May 7 10:24:02 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 7 May 2009 10:24:02 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> Message-ID: <472918d332225688ad7bf5e86e1c9d2a.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On May 7, 2009, at 12:24 AM, Dag Sverre Seljebotn wrote: >>> What I'm getting here at is that we are creating a distinction in >>> Cython >>> which actually does not exist in C++ (rather C++ adds member methods, >>> constructors, private members, and vtables all as orthogonal and >>> independent extensions). >>> >>> But I think that is how it should be. >> >> Yep, if C++ considers them different enough to allocate two separate >> keywords for them, than so should we. (I think it'll also be less >> confusing). > > But C++ doesn't!! > > class Foo { > int x; > public: > int y; > void z(); > }; > > is EXACTLY the same as > > struct Foo { > int y; > void z() > private: > int x; > }; > > Whatever the distinction is supposed to be in Cython, it cannot be the > default private/public, as only the public stuff is available to us > anyway! > > I've been using the distinction to mean "these structs should be > reference-counted, not have stack-allocated semantics like we use for C > structs". If one doesn't plan to ever go for a reference counted scheme, > we should just use plain struct. Ok, so these are orthogonal goals, IIUC. In that case, I'd vote for plain "struct" to avoid the ambiguity with Python's "class". Then, struct methods would behave different for C and C++ semantics in that the C++ code would call the class method, while the C code would use the transformed plain function call. C struct methods can only be defined in Cython (.pyx) code, whereas C++ struct methods can only be defined in external C++ declarations. Note that we can always switch to "class" or whatever other declaration later on if we want to enable ref-counting and consider it similar enough to Python's class behaviour. That would be an additional feature. Stefan From dagss at student.matnat.uio.no Thu May 7 10:35:38 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 7 May 2009 10:35:38 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <472918d332225688ad7bf5e86e1c9d2a.squirrel@groupware.dvs.informatik.tu -darmstadt.de> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <472918d332225688ad7bf5e86e1c9d2a.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <3292c0f02874a62289df377e877c024b.squirrel@webmail.uio.no> Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> Robert Bradshaw wrote: >>> On May 7, 2009, at 12:24 AM, Dag Sverre Seljebotn wrote: >>>> What I'm getting here at is that we are creating a distinction in >>>> Cython >>>> which actually does not exist in C++ (rather C++ adds member methods, >>>> constructors, private members, and vtables all as orthogonal and >>>> independent extensions). >>>> >>>> But I think that is how it should be. >>> >>> Yep, if C++ considers them different enough to allocate two separate >>> keywords for them, than so should we. (I think it'll also be less >>> confusing). >> >> But C++ doesn't!! >> >> class Foo { >> int x; >> public: >> int y; >> void z(); >> }; >> >> is EXACTLY the same as >> >> struct Foo { >> int y; >> void z() >> private: >> int x; >> }; >> >> Whatever the distinction is supposed to be in Cython, it cannot be the >> default private/public, as only the public stuff is available to us >> anyway! >> >> I've been using the distinction to mean "these structs should be >> reference-counted, not have stack-allocated semantics like we use for C >> structs". If one doesn't plan to ever go for a reference counted scheme, >> we should just use plain struct. > > Ok, so these are orthogonal goals, IIUC. In that case, I'd vote for plain > "struct" to avoid the ambiguity with Python's "class". > > Then, struct methods would behave different for C and C++ semantics in > that the C++ code would call the class method, while the C code would use > the transformed plain function call. C struct methods can only be defined > in Cython (.pyx) code, whereas C++ struct methods can only be defined in > external C++ declarations. That's a very constructive idea. But couldn't we just use cdef extern from "myheader.h": cdef struct MyStruct: # External declaration int cppfunc(self, float) # Struct convenience wrapper int cythonwrappingfunc(self): print self.a (Here cppfunc has "self" just for visual consistency, it could be dropped.) > Note that we can always switch to "class" or whatever other declaration > later on if we want to enable ref-counting and consider it similar enough > to Python's class behaviour. That would be an additional feature. I'm a bit wary of encouraging people to start writing cdef CppClass a which would then be possible (if and only if the constructor has no arguments -- which is rather rare anyway in my experience); because it would start calling constructors and destructors, interfering with our knowledge of control flow. A feature of "cppclass" could be that it disallowed code like cdef CppClass a with "Error: Cannot allocate C++ object on stack". Dag Sverre From dagss at student.matnat.uio.no Thu May 7 10:37:30 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 7 May 2009 10:37:30 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> Message-ID: <5305e3422c242075b014430f42fc75ca.squirrel@webmail.uio.no> > Robert Bradshaw wrote: >> On May 7, 2009, at 12:24 AM, Dag Sverre Seljebotn wrote: >>> What I'm getting here at is that we are creating a distinction in >>> Cython >>> which actually does not exist in C++ (rather C++ adds member methods, >>> constructors, private members, and vtables all as orthogonal and >>> independent extensions). >>> >>> But I think that is how it should be. >> >> Yep, if C++ considers them different enough to allocate two separate >> keywords for them, than so should we. (I think it'll also be less >> confusing). > > But C++ doesn't!! > > class Foo { > int x; > public: > int y; > void z(); > }; > > is EXACTLY the same as > > struct Foo { > int y; > void z() > private: > int x; > }; > > Whatever the distinction is supposed to be in Cython, it cannot be the > default private/public, as only the public stuff is available to us > anyway! > > I've been using the distinction to mean "these structs should be > reference-counted, not have stack-allocated semantics like we use for C > structs". If one doesn't plan to ever go for a reference counted scheme, > we should just use plain struct. > >>>>> If you don't want to try your own refcounting, please, please tell >>>>> me that >>>>> the only allowed form of usage of the class will be >>>>> >>>>> cdef CppClass* obj >>>> >>>> This is what I imagined (for now at least). >>>> >>>>> Allowing >>>>> >>>>> cdef CppClass obj >>>>> >>>>> in any form and requiring a manual delete would be very confusing >>>>> and >>>>> difficult to remember. >>>>> >>>>> (That's what I mean by refcounting -- I imagined the difference >>>>> between >>>>> these two syntaxes would be that the latter of these would be >>>>> refcounted; >>>>> and subject for automatic stack-allocation-optimization.) >>>> >>>> By "refcounting", are you implying global garbage collection of some >>>> sort, just like Python objects, or something that emulates being on >>>> the stack (e.g. objects are deleted when the scope exits, and all >>>> assignments are done by value)? >>> >>> I was referring to something which have the full semantics of Python >>> objects (after the result of the "New idea for C++ stack allocation" >>> thread, which expored the opposite idea). >>> >>> So >>> >>> cdef CppClass obj = None >>> obj.do_something() # Illegal, it is None >>> obj = CppClass(3) >>> obj.do_something() # OK >>> >>> ...and so on, just like cdef classes. >>> >>> Then stack allocation would be a pure transparent optimization (an >>> important one; you don't really want to be heap-allocating STL >>> iterators >>> for each iteration!) where semantics can be preserved. >> >> I agree it makes stuff like STL iterators nice to work with, but can >> one do >> >> cdef CppClass x(): >> obj = CppClass(3) >> return obj >> >> Or will it be freed on exit unlike a Python object? Also, what about >> >> def y(): >> cdef CppClass a = CppClass(5) >> cdef CppClass b = a >> return # don't free a twice... >> >> Or perhaps you're thinking about wrapping C++ class instances inside >> actual objects? > > Yes! Allocation would happen through an actual (as light-weight as > possible, and probably mostly opaque, perhaps doing as much as printing > the C++ name of the class in its repr) Python object; and assigning it to > "cdef CppClass a" would basically make Cython automatically do all > accesses (except passing around references) on the contained pointer. > > Again -- this is a larger debate with huge consequences for how one should > do operator overloading etc. For instance, += would have to be implemented > like > > it += 1 > > would mean > > it = please_refcount_iteratorclassname(new std::vector::iterator(it + > 1)) > > However, this kind of semantics ensures that we avoid that > > cdef vector[int].iterator a, b > a = vec.begin() > b = a > a += 1 # also changes b > > which seems to be the alternative (where a and b are just kept on the > stack and we somehow circumvent the constructor business). Sorry, it's too early in the morning :-) please disregard this last bit, it doesn't make sense .. "b = a" make a copy if stack-allocation, which makes sure that "a += 1" only modifies a. Dag Sverre From stefan_ml at behnel.de Thu May 7 10:39:02 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 7 May 2009 10:39:02 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> Message-ID: <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > Allocation would happen through an actual (as light-weight as > possible, and probably mostly opaque, perhaps doing as much as printing > the C++ name of the class in its repr) Python object That wouldn't work, though. Imagine you'd return the C++ object pointer from a function. What would happen to the Python object that holds it? How would you keep the ref-counting context for it? If we enable ref-counting for C++ objects, we should do it ourselves, rather than relying on Python object creation. Not sure if that is doable at all, but Python objects simply don't work here. Stefan From robertwb at math.washington.edu Thu May 7 10:44:07 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 May 2009 01:44:07 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> Message-ID: <16C56E2D-7A3A-405D-986C-377D043E7D8F@math.washington.edu> On May 7, 2009, at 12:58 AM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On May 7, 2009, at 12:24 AM, Dag Sverre Seljebotn wrote: >>> What I'm getting here at is that we are creating a distinction in >>> Cython >>> which actually does not exist in C++ (rather C++ adds member >>> methods, >>> constructors, private members, and vtables all as orthogonal and >>> independent extensions). >>> >>> But I think that is how it should be. >> >> Yep, if C++ considers them different enough to allocate two separate >> keywords for them, than so should we. (I think it'll also be less >> confusing). > > But C++ doesn't!! > > class Foo { > int x; > public: > int y; > void z(); > }; > > is EXACTLY the same as > > struct Foo { > int y; > void z() > private: > int x; > }; I agree it's a tiny distinction, but it *was* enough to convince the authors of C++ that a new keyword was warranted. > Whatever the distinction is supposed to be in Cython, it cannot be the > default private/public, as only the public stuff is available to us > anyway! > > I've been using the distinction to mean "these structs should be > reference-counted, not have stack-allocated semantics like we use > for C > structs". If one doesn't plan to ever go for a reference counted > scheme, > we should just use plain struct. Hmm... I've been thinking of structs as plain old dumb C structs, as they always have been in Cython, and classes as the C++ version with all the goodies like inheritance, polymorphism, operator overloading, etc. BTW, Not ruling out stack/automanaged memory management down the road. >> Or perhaps you're thinking about wrapping C++ class instances inside >> actual objects? > > Yes! OK, that changes everything. It still has its issues though. I'd assume you'd try to delete the C++ object on deallocation? But what if two distinct Python objects refer to the same C++ object? Or the C+ + object (pointer) gets passed and stored in some library somewhere? I don't see a way to ensure a one-to-one relationship between C++ objects and Python objects--short of having stack allocated semantics (i.e. it goes out of scope and is deconstructed when the function exits) I don't see how one can automatically refcount C++ classes. > Allocation would happen through an actual (as light-weight as > possible, and probably mostly opaque, perhaps doing as much as > printing > the C++ name of the class in its repr) Python object; and assigning > it to > "cdef CppClass a" would basically make Cython automatically do all > accesses (except passing around references) on the contained pointer. You know people are then going to want to have access to all it's methods and members, so that just declaring the C++ class will be creating a full-fledged wrapper around it. I can just see the expectations now :). [...] > I'm happy that only "CppClass*" will be allowed at first, which > takes much > of this worry away. One drawback is that it makes not having a good referencing operator a pain. One will have to write a lot of code like c[0] = a[0] + b[0] * c[0] Cython does automatically promote a.b to a->b when a is a pointer type, and it'd be really nice to do something similar, but, e.g., CppClass* + int already has meaning. > However note that this might have consequences for how > operators should be defined (I haven't seen a concrete proposal for > what > you are thinking there so I don't know how it would fare). http://www.mail-archive.com/cython-dev at codespeak.net/msg05140.html - Robert From dagss at student.matnat.uio.no Thu May 7 10:50:28 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 7 May 2009 10:50:28 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu -darmstadt.de> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> > Dag Sverre Seljebotn wrote: >> Allocation would happen through an actual (as light-weight as >> possible, and probably mostly opaque, perhaps doing as much as printing >> the C++ name of the class in its repr) Python object > > That wouldn't work, though. Imagine you'd return the C++ object pointer > from a function. What would happen to the Python object that holds it? How > would you keep the ref-counting context for it? There would be no coercion between pointers and refcounted variables. I.e. if you have cdef extern CppObject* foo() you can NOT do cdef CppObject obj = foo() It would only work when C++ returned an object *by value*: cdef extern CppObject foo() cdef CppObject obj = foo() This would then heap-allocate a CppObject, passing in the object that is returned to the copy constructor. We *could* perhaps add manual conversion from pointers, with a "managed" boolean keyword etc., but that is something else entirely. Robert, this answers your problems with it as well I believe. Dag Sverre From dagss at student.matnat.uio.no Thu May 7 11:03:34 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 7 May 2009 11:03:34 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <16C56E2D-7A3A-405D-986C-377D043E7D8F@math.washington.edu> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <16C56E2D-7A3A-405D-986C-377D043E7D8F@math.washington.edu> Message-ID: <05801ec60573b09b3f0bc28c72764050.squirrel@webmail.uio.no> > On May 7, 2009, at 12:58 AM, Dag Sverre Seljebotn wrote: > >> Robert Bradshaw wrote: >>> On May 7, 2009, at 12:24 AM, Dag Sverre Seljebotn wrote: >>>> What I'm getting here at is that we are creating a distinction in >>>> Cython >>>> which actually does not exist in C++ (rather C++ adds member >>>> methods, >>>> constructors, private members, and vtables all as orthogonal and >>>> independent extensions). >>>> >>>> But I think that is how it should be. >>> >>> Yep, if C++ considers them different enough to allocate two separate >>> keywords for them, than so should we. (I think it'll also be less >>> confusing). >> >> But C++ doesn't!! >> >> class Foo { >> int x; >> public: >> int y; >> void z(); >> }; >> >> is EXACTLY the same as >> >> struct Foo { >> int y; >> void z() >> private: >> int x; >> }; > > I agree it's a tiny distinction, but it *was* enough to convince the > authors of C++ that a new keyword was warranted. > >> Whatever the distinction is supposed to be in Cython, it cannot be the >> default private/public, as only the public stuff is available to us >> anyway! >> >> I've been using the distinction to mean "these structs should be >> reference-counted, not have stack-allocated semantics like we use >> for C >> structs". If one doesn't plan to ever go for a reference counted >> scheme, >> we should just use plain struct. > > Hmm... I've been thinking of structs as plain old dumb C structs, as > they always have been in Cython, and classes as the C++ version with > all the goodies like inheritance, polymorphism, operator overloading, > etc. > > BTW, Not ruling out stack/automanaged memory management down the road. > >>> Or perhaps you're thinking about wrapping C++ class instances inside >>> actual objects? >> >> Yes! > > OK, that changes everything. It still has its issues though. I'd > assume you'd try to delete the C++ object on deallocation? But what > if two distinct Python objects refer to the same C++ object? Or the C+ > + object (pointer) gets passed and stored in some library somewhere? All crossings between Cython and C++ would pass the object by value. You'd still need to use & to get a pointer, which would be similar to str -> char*. > I don't see a way to ensure a one-to-one relationship between C++ > objects and Python objects--short of having stack allocated semantics > (i.e. it goes out of scope and is deconstructed when the function > exits) I don't see how one can automatically refcount C++ classes. I don't see the problem, sorry. > You know people are then going to want to have access to all it's > methods and members, so that just declaring the C++ class will be > creating a full-fledged wrapper around it. I can just see the > expectations now :). Well, obviously that is down the pipeline, but no need to go there now. I just want *some* way to avoid having to write it[0][0] to dereference an STL iterator. Everything else is secondary really :-) > [...] > >> I'm happy that only "CppClass*" will be allowed at first, which >> takes much >> of this worry away. > > One drawback is that it makes not having a good referencing operator > a pain. One will have to write a lot of code like > > c[0] = a[0] + b[0] * c[0] > > Cython does automatically promote a.b to a->b when a is a pointer > type, and it'd be really nice to do something similar, but, e.g., > CppClass* + int already has meaning. Yes, that's the problem my Python-object-solution was designed to take away. > >> However note that this might have consequences for how >> operators should be defined (I haven't seen a concrete proposal for >> what >> you are thinking there so I don't know how it would fare). > > http://www.mail-archive.com/cython-dev at codespeak.net/msg05140.html Hmm. So how would e.g. += be treated? Dag Sverre From dagss at student.matnat.uio.no Thu May 7 11:12:23 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 7 May 2009 11:12:23 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> Message-ID: >> Dag Sverre Seljebotn wrote: >>> Allocation would happen through an actual (as light-weight as >>> possible, and probably mostly opaque, perhaps doing as much as printing >>> the C++ name of the class in its repr) Python object >> >> That wouldn't work, though. Imagine you'd return the C++ object pointer >> from a function. What would happen to the Python object that holds it? >> How >> would you keep the ref-counting context for it? > > There would be no coercion between pointers and refcounted variables. I.e. > if you have > > cdef extern CppObject* foo() > > you can NOT do > > cdef CppObject obj = foo() > > It would only work when C++ returned an object *by value*: The deeper point here is that Cython has no concept of objects by value, which C++ has, and I'm trying to avoid that from creeping in. For instance you have CppObject a, b; b = a; a.set_foo(4); // does not set foo of b which would look strange in Cython. So I have two goals: - convenient STL iterator syntax - avoiding the above semantics in Cython and everything else, though it might seem complicated, kind of flows from that (and it's not hard to implement). Dag Sverre From stefan_ml at behnel.de Thu May 7 11:14:26 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 7 May 2009 11:14:26 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> Message-ID: <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: >> Dag Sverre Seljebotn wrote: >>> Allocation would happen through an actual (as light-weight as >>> possible, and probably mostly opaque, perhaps doing as much as printing >>> the C++ name of the class in its repr) Python object >> >> That wouldn't work, though. Imagine you'd return the C++ object pointer >> from a function. What would happen to the Python object that holds it? >> How would you keep the ref-counting context for it? > > There would be no coercion between pointers and refcounted variables. I.e. > if you have > > cdef extern CppObject* foo() > > you can NOT do > > cdef CppObject obj = foo() > > It would only work when C++ returned an object *by value*: > > cdef extern CppObject foo() > cdef CppObject obj = foo() > > This would then heap-allocate a CppObject, passing in the object that is > returned to the copy constructor. Ah, ok, got it. So they'd basically be distinct types and you cannot (implicitly) coerce a C++ object to a pointer. That should work then. Can C++ objects override their own allocation, or could you use a PyVarObject to allocate both the object and the Cython wrapper at once? (targeting heap allocation here, I assume that stack allocated objects do not need ref-counting anyway?) Stefan From robertwb at math.washington.edu Thu May 7 11:27:12 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 May 2009 02:27:12 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <05801ec60573b09b3f0bc28c72764050.squirrel@webmail.uio.no> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <16C56E2D-7A3A-405D-986C-377D043E7D8F@math.washington.edu> <05801ec60573b09b3f0bc28c72764050.squirrel@webmail.uio.no> Message-ID: <38519F5C-179F-4292-8DD0-32AAF2DBB404@math.washington.edu> On May 7, 2009, at 2:03 AM, Dag Sverre Seljebotn wrote: >> OK, that changes everything. It still has its issues though. I'd >> assume you'd try to delete the C++ object on deallocation? But what >> if two distinct Python objects refer to the same C++ object? Or >> the C+ >> + object (pointer) gets passed and stored in some library somewhere? > > All crossings between Cython and C++ would pass the object by > value. You'd > still need to use & to get a pointer, which would be similar to str -> > char*. > >> I don't see a way to ensure a one-to-one relationship between C++ >> objects and Python objects--short of having stack allocated semantics >> (i.e. it goes out of scope and is deconstructed when the function >> exits) I don't see how one can automatically refcount C++ classes. > > I don't see the problem, sorry. I hadn't seen your retraction of "also changes b" so I thought you were trying to pass things around by reference. Then what does b = a actually do? Construct a new Python object, then copy by value into it, and assign to be? So a is b is now False? >> You know people are then going to want to have access to all it's >> methods and members, so that just declaring the C++ class will be >> creating a full-fledged wrapper around it. I can just see the >> expectations now :). > > Well, obviously that is down the pipeline, but no need to go there > now. > > I just want *some* way to avoid having to write it[0][0] to > dereference an > STL iterator. Everything else is secondary really :-) Yeah, me too :-) > >> >>> However note that this might have consequences for how >>> operators should be defined (I haven't seen a concrete proposal for >>> what >>> you are thinking there so I don't know how it would fare). >> >> http://www.mail-archive.com/cython-dev at codespeak.net/msg05140.html > > Hmm. So how would e.g. += be treated? Perhaps we could declare it as having an __iadd__ operator, but realistically it might make sense to disallow all assignment operators for the first pass. - Robert From robertwb at math.washington.edu Thu May 7 11:39:53 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 May 2009 02:39:53 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: On May 7, 2009, at 2:14 AM, Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >>> Dag Sverre Seljebotn wrote: >>>> Allocation would happen through an actual (as light-weight as >>>> possible, and probably mostly opaque, perhaps doing as much as >>>> printing >>>> the C++ name of the class in its repr) Python object >>> >>> That wouldn't work, though. Imagine you'd return the C++ object >>> pointer >>> from a function. What would happen to the Python object that >>> holds it? >>> How would you keep the ref-counting context for it? >> >> There would be no coercion between pointers and refcounted >> variables. I.e. >> if you have >> >> cdef extern CppObject* foo() >> >> you can NOT do >> >> cdef CppObject obj = foo() >> >> It would only work when C++ returned an object *by value*: >> >> cdef extern CppObject foo() >> cdef CppObject obj = foo() >> >> This would then heap-allocate a CppObject, passing in the object >> that is >> returned to the copy constructor. > > Ah, ok, got it. So they'd basically be distinct types and you cannot > (implicitly) coerce a C++ object to a pointer. That should work then. Will dereferencing a CppObject pointer always create a Python object then? You could take the address of a CppObject and get a volitile pointer (just as with bytes -> char*), right? > Can C++ objects override their own allocation, or could you use a > PyVarObject to allocate both the object and the Cython wrapper at > once? Yes, you can, we do that in Sage. I don't think one can declare a struct member to be an object without a no-args constructor without extra work though, so one would have to do the same hackery that was proposed for stack-allocated objects. > (targeting heap allocation here, I assume that stack allocated > objects do > not need ref-counting anyway?) Yep, stack-allocated objects disappear with the stack. - Robert From dagss at student.matnat.uio.no Thu May 7 11:47:05 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 7 May 2009 11:47:05 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <38519F5C-179F-4292-8DD0-32AAF2DBB404@math.washington.edu> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <16C56E2D-7A3A-405D-986C-377D043E7D8F@math.washington.edu> <05801ec60573b09b3f0bc28c72764050.squirrel@webmail.uio.no> <38519F5C-179F-4292-8DD0-32AAF2DBB404@math.washington.edu> Message-ID: <30eeafae0e895c6e1aca708240db3cc3.squirrel@webmail.uio.no> Robert wrote: > On May 7, 2009, at 2:03 AM, Dag Sverre Seljebotn wrote: > >>> OK, that changes everything. It still has its issues though. I'd >>> assume you'd try to delete the C++ object on deallocation? But what >>> if two distinct Python objects refer to the same C++ object? Or >>> the C+ >>> + object (pointer) gets passed and stored in some library somewhere? >> >> All crossings between Cython and C++ would pass the object by >> value. You'd >> still need to use & to get a pointer, which would be similar to str -> >> char*. >> >>> I don't see a way to ensure a one-to-one relationship between C++ >>> objects and Python objects--short of having stack allocated semantics >>> (i.e. it goes out of scope and is deconstructed when the function >>> exits) I don't see how one can automatically refcount C++ classes. >> >> I don't see the problem, sorry. > > I hadn't seen your retraction of "also changes b" so I thought you > were trying to pass things around by reference. Then what does That whole explanation was a mess, as my daughter waked up from her sleep at the time before I could improve it :-) > b = a > > actually do? Construct a new Python object, then copy by value into > it, and assign to be? So > > a is b > > is now False? I'll just forget about earlier explanations. But what I am advocating is to just have a regular Python reference transfer, no C++ operators involved. With that approach, cdef CppObject a += 2 would mean the same as a = CppObject(a + 2) i.e. a new Python wrapper passing in a+2 as the argument. (C++ classes almost always has this "copy constructor", or else they can't be passed by value in the first place anyway). I am agreeing with Stefan that this is probably orthogonal. It is something you might want to support for C structs as well, so that cdef CWidget a, b b = a a.show_caption = True # also changes b would hold, i.e. an ability to give C structs Python semantics. This would especially be useful given the recent method-on-struct idea, as b = a a.set_value(3) # does not modify b is a "new" wierd exception from Python semantics. I still believe it should be discussed now because a) it heavily impacts how one thinks about C++ operators b) it is a way to solve the construction/destruction problem with C++ classes but as a feature, it could be mostly orthogonal. Dag Sverre From dagss at student.matnat.uio.no Thu May 7 11:56:33 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 7 May 2009 11:56:33 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <716739d959ac4aa061bfe5a55f1b64fe.squirrel@webmail.uio.no> Robert wrote: > On May 7, 2009, at 2:14 AM, Stefan Behnel wrote: > >> Dag Sverre Seljebotn wrote: >>>> Dag Sverre Seljebotn wrote: >>>>> Allocation would happen through an actual (as light-weight as >>>>> possible, and probably mostly opaque, perhaps doing as much as >>>>> printing >>>>> the C++ name of the class in its repr) Python object >>>> >>>> That wouldn't work, though. Imagine you'd return the C++ object >>>> pointer >>>> from a function. What would happen to the Python object that >>>> holds it? >>>> How would you keep the ref-counting context for it? >>> >>> There would be no coercion between pointers and refcounted >>> variables. I.e. >>> if you have >>> >>> cdef extern CppObject* foo() >>> >>> you can NOT do >>> >>> cdef CppObject obj = foo() >>> >>> It would only work when C++ returned an object *by value*: >>> >>> cdef extern CppObject foo() >>> cdef CppObject obj = foo() >>> >>> This would then heap-allocate a CppObject, passing in the object >>> that is >>> returned to the copy constructor. >> >> Ah, ok, got it. So they'd basically be distinct types and you cannot >> (implicitly) coerce a C++ object to a pointer. That should work then. > > Will dereferencing a CppObject pointer always create a Python object > then? You could take the address of a CppObject and get a volitile > pointer (just as with bytes -> char*), right? &obj would be like bytes -> char*, same caveats ptr[0] would (in my proposal anyway) return a Python object; BUT if that is immedeately sent off to a C++ function by value it is just a middle-person that can be left out safely, as an optimization. > Yes, you can, we do that in Sage. I don't think one can declare a > struct member to be an object without a no-args constructor without > extra work though, so one would have to do the same hackery that was > proposed for stack-allocated objects. I don't know enough CPython, but another (cleaner) option could be to allocate the whole PyObject using C++ new (which would pass on the constructor arguments to the field using the "MyClass(int arg) : myfield(arg) {}" syntax), and override tp_alloc to return that object? If it is possible to tell tp_alloc about that somehow. Another idea BTW is that we can use C++ templates to generate these Python wrappers, so we only need to output one (manually written) template to our code, and can then use CppPyWrapper and it could instantiate the right type structure through more template programming etc. Dag Sverre From dagss at student.matnat.uio.no Thu May 7 12:00:57 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 7 May 2009 12:00:57 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <30eeafae0e895c6e1aca708240db3cc3.squirrel@webmail.uio.no> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <16C56E2D-7A3A-405D-986C-377D043E7D8F@math.washington.edu> <05801ec60573b09b3f0bc28c72764050.squirrel@webmail.uio.no> <38519F5C-179F-4292-8DD0-32AAF2DBB404@math.washington.edu> <30eeafae0e895c6e1aca708240db3cc3.squirrel@webmail.uio.no> Message-ID: > Robert wrote: >> On May 7, 2009, at 2:03 AM, Dag Sverre Seljebotn wrote: >> >>>> OK, that changes everything. It still has its issues though. I'd >>>> assume you'd try to delete the C++ object on deallocation? But what >>>> if two distinct Python objects refer to the same C++ object? Or >>>> the C+ >>>> + object (pointer) gets passed and stored in some library somewhere? >>> >>> All crossings between Cython and C++ would pass the object by >>> value. You'd >>> still need to use & to get a pointer, which would be similar to str -> >>> char*. >>> >>>> I don't see a way to ensure a one-to-one relationship between C++ >>>> objects and Python objects--short of having stack allocated semantics >>>> (i.e. it goes out of scope and is deconstructed when the function >>>> exits) I don't see how one can automatically refcount C++ classes. >>> >>> I don't see the problem, sorry. >> >> I hadn't seen your retraction of "also changes b" so I thought you >> were trying to pass things around by reference. Then what does > > That whole explanation was a mess, as my daughter waked up from her sleep > at the time before I could improve it :-) > >> b = a >> >> actually do? Construct a new Python object, then copy by value into >> it, and assign to be? So >> >> a is b >> >> is now False? > > I'll just forget about earlier explanations. > > But what I am advocating is to just have a regular Python reference > transfer, no C++ operators involved. > > With that approach, > > cdef CppObject > a += 2 > > would mean the same as > > a = CppObject(a + 2) > > i.e. a new Python wrapper passing in a+2 as the argument. (C++ classes > almost always has this "copy constructor", or else they can't be passed by > value in the first place anyway). > > I am agreeing with Stefan that this is probably orthogonal. It is > something you might want to support for C structs as well, so that > > cdef CWidget a, b > b = a > a.show_caption = True # also changes b > > would hold, i.e. an ability to give C structs Python semantics. I keep making mistakes. cdef CWidget a, b # a and b are None a = CWidget() b = a a.show_caption = True # also changes b (Obviously you'd change the syntax for declaring the CWidget struct heavily, probably call it something with class). Dag Sverre From dagss at student.matnat.uio.no Thu May 7 12:10:09 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 7 May 2009 12:10:09 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <716739d959ac4aa061bfe5a55f1b64fe.squirrel@webmail.uio.no> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <716739d959ac4aa061bfe5a55f1b64fe.squirrel@webmail.uio.no> Message-ID: I wrote: > Robert wrote: >> Will dereferencing a CppObject pointer always create a Python object >> then? You could take the address of a CppObject and get a volitile >> pointer (just as with bytes -> char*), right? > > &obj would be like bytes -> char*, same caveats > > ptr[0] would (in my proposal anyway) return a Python object; BUT if that > is immedeately sent off to a C++ function by value it is just a > middle-person that can be left out safely, as an optimization. I think the right way to think about this is ptr[0] returning a C++ CppObject by value, but what then happens to it? If it is coerced to object then obviously one is constructed, likewise if it assigned a a Cython refcounted "cdef CppObject". But passing to C++ code would happen by value. Dag Sverre From robertwb at math.washington.edu Thu May 7 12:22:58 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 May 2009 03:22:58 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <30eeafae0e895c6e1aca708240db3cc3.squirrel@webmail.uio.no> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <16C56E2D-7A3A-405D-986C-377D043E7D8F@math.washington.edu> <05801ec60573b09b3f0bc28c72764050.squirrel@webmail.uio.no> <38519F5C-179F-4292-8DD0-32AAF2DBB404@math.washington.edu> <30eeafae0e895c6e1aca708240db3cc3.squirrel@webmail.uio.no> Message-ID: On May 7, 2009, at 2:47 AM, Dag Sverre Seljebotn wrote: > That whole explanation was a mess, as my daughter waked up from her > sleep > at the time before I could improve it :-) Don't worry, I *totally* understand. :-) >> b = a >> >> actually do? Construct a new Python object, then copy by value into >> it, and assign to be? So >> >> a is b >> >> is now False? > > I'll just forget about earlier explanations. > > But what I am advocating is to just have a regular Python reference > transfer, no C++ operators involved. [...] > I am agreeing with Stefan that this is probably orthogonal. It is > something you might want to support for C structs as well, so that > > cdef CWidget a, b > b = a > a.show_caption = True # also changes b > > would hold, i.e. an ability to give C structs Python semantics. > This would > especially be useful given the recent method-on-struct idea, as > > b = a > a.set_value(3) # does not modify b > > is a "new" wierd exception from Python semantics. This has very little advantage over just creating a simple cdef class... if one doesn't mind the overhead (which is usually the reason to go with structs in the first place). This is almost sounding like a proposal to do cdef managed CWidget which would be an object wrapping a CWidget. > I still believe it should be discussed now because > a) it heavily impacts how one thinks about C++ operators Yes. > b) it is a way to solve the construction/destruction problem with C+ > + classes I'm not sure it does. - Robert From dagss at student.matnat.uio.no Thu May 7 12:39:07 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 07 May 2009 12:39:07 +0200 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <16C56E2D-7A3A-405D-986C-377D043E7D8F@math.washington.edu> <05801ec60573b09b3f0bc28c72764050.squirrel@webmail.uio.no> <38519F5C-179F-4292-8DD0-32AAF2DBB404@math.washington.edu> <30eeafae0e895c6e1aca708240db3cc3.squirrel@webmail.uio.no> Message-ID: <4A02BA4B.5080302@student.matnat.uio.no> Robert Bradshaw wrote: > On May 7, 2009, at 2:47 AM, Dag Sverre Seljebotn wrote: > reason to go with structs in the first place). This is almost > sounding like a proposal to do > > cdef managed CWidget > > which would be an object wrapping a CWidget. Perhaps that is a better way to think about it. Still, introducing that as the direct way of wrapping C++ classes sidesteps the issue of figuring out how C++ semantics should work in Cython (just in order to be wrapped). >> I still believe it should be discussed now because >> a) it heavily impacts how one thinks about C++ operators > > Yes. > >> b) it is a way to solve the construction/destruction problem with C+ >> + classes > > I'm not sure it does. I am referring to having to do it[0][0], which is needed because one cannot do cdef vector[int].iterator it currently. With my proposal you can do cdef vector[int].iterator it print it is None # True it = vec.begin() while it != vec.end(): # != would go through to wrapped objects print deref(it) # it.get() with my operator proposal inc(it) # it.next() with my operator proposal which is a "solved" in my book. (I've let inc operate in-place here, so that "b = a; inc(a)" would modify b as well. As if inc is calling a method on a. If you did while it != vec.end(): ... it += 1 then you would start craving stack-allocation optimizations as this would have an malloc per iteration; but that's not too hard, just notice that "it" is never referenced directly anywhere (only used in operations) and so it can go on the stack instead). -- Dag Sverre From robertwb at math.washington.edu Thu May 7 12:41:06 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 May 2009 03:41:06 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <716739d959ac4aa061bfe5a55f1b64fe.squirrel@webmail.uio.no> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <716739d959ac4aa061bfe5a55f1b64fe.squirrel@webmail.uio.no> Message-ID: <0AE19092-B472-47D8-B0BA-E590768F9E5E@math.washington.edu> On May 7, 2009, at 2:56 AM, Dag Sverre Seljebotn wrote: >> Yes, you can, we do that in Sage. I don't think one can declare a >> struct member to be an object without a no-args constructor without >> extra work though, so one would have to do the same hackery that was >> proposed for stack-allocated objects. > > I don't know enough CPython, but another (cleaner) option could be to > allocate the whole PyObject using C++ new (which would pass on the > constructor arguments to the field using the "MyClass(int arg) : > myfield(arg) {}" syntax), and override tp_alloc to return that > object? If > it is possible to tell tp_alloc about that somehow. PyObject *tp_alloc(PyTypeObject *self, int nitems) I wouldn't call it cleaner, but one could have tp_alloc/tp_new raise an error, and then always construct the objects manually. Feels like we're abusing something there. > Another idea BTW is that we can use C++ templates to generate these > Python > wrappers, so we only need to output one (manually written) template > to our > code, and can then use > > CppPyWrapper > > and it could instantiate the right type structure through more > template > programming etc. One would have to have struct CppPyWrapper { PyObject_HEAD MyCppClass; CppPyWrapper() : MyCppClass(default_args) { }; }; which is no better than simply trying to put MyCppClass on the stack, so sticking the memory chunk in a PyObject* doesn't really save you anything. (Well, it lets you pass it out of the scope, and gives reference assignment semantics, but I'm not sure either of these are desirable for something that looks like its allocated on the stack, and one could provide this in an orthogonal way (e.g. with the "managed" keyword idea). Also, giving it reference semantics means that we could never go and optimize it by placing things actually on the stack. - Robert From robertwb at math.washington.edu Thu May 7 12:52:58 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 May 2009 03:52:58 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <4A02BA4B.5080302@student.matnat.uio.no> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <16C56E2D-7A3A-405D-986C-377D043E7D8F@math.washington.edu> <05801ec60573b09b3f0bc28c72764050.squirrel@webmail.uio.no> <38519F5C-179F-4292-8DD0-32AAF2DBB404@math.washington.edu> <30eeafae0e895c6e1aca708240db3cc3.squirrel@webmail.uio.no> <4A02BA4B.5080302@student.matnat.uio.no> Message-ID: <14F264DD-8789-4D99-9F94-63CAAE9FF157@math.washington.edu> On May 7, 2009, at 3:39 AM, Dag Sverre Seljebotn wrote: >>> b) it is a way to solve the construction/destruction problem with C+ >>> + classes >> >> I'm not sure it does. > > I am referring to having to do it[0][0], which is needed because one > cannot do > > cdef vector[int].iterator it > > currently. > > With my proposal you can do > > cdef vector[int].iterator it > print it is None # True > it = vec.begin() > while it != vec.end(): # != would go through to wrapped objects > print deref(it) # it.get() with my operator proposal > inc(it) # it.next() with my operator proposal To clarify, your operator proposal is the inline C++ code chunk strings? > which is a "solved" in my book. No, it's pushed the problem of allocating vector[int].iterator on the stack to allocating vector[int].iterator as a member of a struct. - Robert From dagss at student.matnat.uio.no Thu May 7 12:56:57 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 07 May 2009 12:56:57 +0200 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <14F264DD-8789-4D99-9F94-63CAAE9FF157@math.washington.edu> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <16C56E2D-7A3A-405D-986C-377D043E7D8F@math.washington.edu> <05801ec60573b09b3f0bc28c72764050.squirrel@webmail.uio.no> <38519F5C-179F-4292-8DD0-32AAF2DBB404@math.washington.edu> <30eeafae0e895c6e1aca708240db3cc3.squirrel@webmail.uio.no> <4A02BA4B.5080302@student.matnat.uio.no> <14F264DD-8789-4D99-9F94-63CAAE9FF157@math.washington.edu> Message-ID: <4A02BE79.2050304@student.matnat.uio.no> Robert Bradshaw wrote: > On May 7, 2009, at 3:39 AM, Dag Sverre Seljebotn wrote: > >>>> b) it is a way to solve the construction/destruction problem with C+ >>>> + classes >>> I'm not sure it does. >> I am referring to having to do it[0][0], which is needed because one >> cannot do >> >> cdef vector[int].iterator it >> >> currently. >> >> With my proposal you can do >> >> cdef vector[int].iterator it >> print it is None # True >> it = vec.begin() >> while it != vec.end(): # != would go through to wrapped objects >> print deref(it) # it.get() with my operator proposal >> inc(it) # it.next() with my operator proposal > > To clarify, your operator proposal is the inline C++ code chunk strings? > >> which is a "solved" in my book. > > No, it's pushed the problem of allocating vector[int].iterator on the > stack to allocating vector[int].iterator as a member of a struct. My problem isn't at all with the technical difficulties. I'm fine with using placement new for stack allocation if that is the problem. My problem is how the user semantics should look like. Python semantics has a concept of constructors/destructors which I'm borrowing here to fix the semantics part. The alternative is somehow invent a syntax to call constructors on explicitly stack-allocated variables, and you seemed to reject that in the thread I started about it. -- Dag Sverre From dagss at student.matnat.uio.no Thu May 7 12:57:29 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 07 May 2009 12:57:29 +0200 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <0AE19092-B472-47D8-B0BA-E590768F9E5E@math.washington.edu> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <716739d959ac4aa061bfe5a55f1b64fe.squirrel@webmail.uio.no> <0AE19092-B472-47D8-B0BA-E590768F9E5E@math.washington.edu> Message-ID: <4A02BE99.3060608@student.matnat.uio.no> Robert Bradshaw wrote: > which is no better than simply trying to put MyCppClass on the stack, > so sticking the memory chunk in a PyObject* doesn't really save you > anything. (Well, it lets you pass it out of the scope, and gives > reference assignment semantics, but I'm not sure either of these are > desirable for something that looks like its allocated on the stack, > and one could provide this in an orthogonal way (e.g. with the > "managed" keyword idea). Also, giving it reference semantics means > that we could never go and optimize it by placing things actually on > the stack. We're hitting the core of the matter here! I can see why one would like to stay "close to the metal", so to speak. I was mainly (mis)interpreting your lack of support in the "C++ stack allocation" thread as being in favour of this other proposal... Still, with direct stack allocation one needs a way to do cdef CppObject a, b and then run their constructors before they are used. With structs you can do cdef CObject a a.x = 3 which just doesn't fly with C++ with most classes. My thoughts were in the other thread, but I guess it boils down to making sure that "a" is assigned to before it is used. You are not rejecting that then? That would bring the whole C++ semantics into Cython, so that b = a would invoke operator=, b = a a.set_foo(3) # does not modify b and so on. I guess I don't have to like it to support it, I can see why you'd do it. Keeping "managed" strictly orthogonal does make a lot of sense. -- Dag Sverre From dagss at student.matnat.uio.no Thu May 7 13:00:02 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 07 May 2009 13:00:02 +0200 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <14F264DD-8789-4D99-9F94-63CAAE9FF157@math.washington.edu> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <16C56E2D-7A3A-405D-986C-377D043E7D8F@math.washington.edu> <05801ec60573b09b3f0bc28c72764050.squirrel@webmail.uio.no> <38519F5C-179F-4292-8DD0-32AAF2DBB404@math.washington.edu> <30eeafae0e895c6e1aca708240db3cc3.squirrel@webmail.uio.no> <4A02BA4B.5080302@student.matnat.uio.no> <14F264DD-8789-4D99-9F94-63CAAE9FF157@math.washington.edu> Message-ID: <4A02BF32.7050908@student.matnat.uio.no> Robert Bradshaw wrote: > On May 7, 2009, at 3:39 AM, Dag Sverre Seljebotn wrote: > >>>> b) it is a way to solve the construction/destruction problem with C+ >>>> + classes >>> I'm not sure it does. >> I am referring to having to do it[0][0], which is needed because one >> cannot do >> >> cdef vector[int].iterator it >> >> currently. >> >> With my proposal you can do >> >> cdef vector[int].iterator it >> print it is None # True >> it = vec.begin() >> while it != vec.end(): # != would go through to wrapped objects >> print deref(it) # it.get() with my operator proposal >> inc(it) # it.next() with my operator proposal > > To clarify, your operator proposal is the inline C++ code chunk strings? Yes. -- Dag Sverre From dagss at student.matnat.uio.no Thu May 7 13:28:05 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 07 May 2009 13:28:05 +0200 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <4A02BE99.3060608@student.matnat.uio.no> References: <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <716739d959ac4aa061bfe5a55f1b64fe.squirrel@webmail.uio.no> <0AE19092-B472-47D8-B0BA-E590768F9E5E@math.washington.edu> <4A02BE99.3060608@student.matnat.uio.no> Message-ID: <4A02C5C5.4000509@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> which is no better than simply trying to put MyCppClass on the stack, >> so sticking the memory chunk in a PyObject* doesn't really save you >> anything. (Well, it lets you pass it out of the scope, and gives >> reference assignment semantics, but I'm not sure either of these are >> desirable for something that looks like its allocated on the stack, >> and one could provide this in an orthogonal way (e.g. with the >> "managed" keyword idea). Also, giving it reference semantics means >> that we could never go and optimize it by placing things actually on >> the stack. > > We're hitting the core of the matter here! > > I can see why one would like to stay "close to the metal", so to speak. > I was mainly (mis)interpreting your lack of support in the "C++ stack > allocation" thread as being in favour of this other proposal... OK I'm sorry, my C++ stack allocation thread was way more far-reaching. Just scratch that reference. -- Dag Sverre From dagss at student.matnat.uio.no Thu May 7 13:32:03 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 07 May 2009 13:32:03 +0200 Subject: [Cython] [Numpy-discussion] efficient 3d histogram creation In-Reply-To: <9457e7c80905070411x6d51c768g20f4dd0990b843a8@mail.gmail.com> References: <7f014ea60905031715h635a69faof6a06e10c3621ba7@mail.gmail.com> <91cf711d0905041218p70bb44ct35a601844c8c262b@mail.gmail.com> <7f014ea60905041300y49b48055i6df8d5e598d0fe80@mail.gmail.com> <1cd32cbb0905041318g11dea0b9oa61f08bcc380144d@mail.gmail.com> <91cf711d0905050646q7652eepc55d1aed17b1d1ed@mail.gmail.com> <7f014ea60905061506o7253a315p940d6cbd2cc2b420@mail.gmail.com> <1cd32cbb0905061630t1e73e8a5i4bd454f789e22714@mail.gmail.com> <7f014ea60905061639k21dd12b5j4ee6c2738758284@mail.gmail.com> <1cd32cbb0905061721y233aa4c7uf580ad1d99539b1c@mail.gmail.com> <7f014ea60905061734y3dfcb299w14681ff0020a4de1@mail.gmail.com> <9457e7c80905070411x6d51c768g20f4dd0990b843a8@mail.gmail.com> Message-ID: <4A02C6B3.3030404@student.matnat.uio.no> St?fan van der Walt wrote: > 2009/5/7 Chris Colbert : >> This was really my first attempt at doing anything constructive with Cython. >> It was actually unbelievably easy to work with. I think i spent less time >> working on this, than I did trying to find an optimized solution using pure >> numpy and python. > > One aspect we often overlook is how easy it is to write a for-loop in > comparison to vectorisation. Besides, for-loops are sometimes easier > to read as well! > > I think the Cython guys are planning some sort of templating, but I'll > CC Dag so that he can tell us more. We were discussing how it would/should look like, but noone's committed to implementing it so it's pretty much up in the blue I think -- someone might jump in and do it next week, or it might go another year, I can't tell. While I'm here, also note in that code Chris wrote that you want to pay attention to the change of default division semantics on Cython 0.12 (especially for speed). http://wiki.cython.org/enhancements/division -- Dag Sverre From robertwb at math.washington.edu Thu May 7 13:37:05 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 May 2009 04:37:05 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <4A02BE99.3060608@student.matnat.uio.no> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <716739d959ac4aa061bfe5a55f1b64fe.squirrel@webmail.uio.no> <0AE19092-B472-47D8-B0BA-E590768F9E5E@math.washington.edu>! <4A02BE99.3060608@student.matnat.uio.no> Message-ID: On May 7, 2009, at 3:57 AM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> which is no better than simply trying to put MyCppClass on the stack, >> so sticking the memory chunk in a PyObject* doesn't really save you >> anything. (Well, it lets you pass it out of the scope, and gives >> reference assignment semantics, but I'm not sure either of these are >> desirable for something that looks like its allocated on the stack, >> and one could provide this in an orthogonal way (e.g. with the >> "managed" keyword idea). Also, giving it reference semantics means >> that we could never go and optimize it by placing things actually on >> the stack. > > We're hitting the core of the matter here! > > I can see why one would like to stay "close to the metal", so to > speak. > I was mainly (mis)interpreting your lack of support in the "C++ stack > allocation" thread as being in favour of this other proposal... > > Still, with direct stack allocation one needs a way to do > > cdef CppObject a, b > > and then run their constructors before they are used. With structs you > can do > > cdef CObject a > a.x = 3 > > which just doesn't fly with C++ with most classes. > > My thoughts were in the other thread, but I guess it boils down to > making sure that "a" is assigned to before it is used. > > You are not rejecting that then? No. Have I been critical of it? Yes, but I have been of every potential solution so far (including my own). Honestly, I'm still feeling my way around here...and learning a lot about C++ as I go :) > That would bring the whole C++ semantics into Cython, so that > > b = a > > would invoke operator=, > > b = a > a.set_foo(3) # does not modify b > > and so on. Yep. > I guess I don't have to like it to support it, I can see why you'd do > it. Keeping "managed" strictly orthogonal does make a lot of sense. Originally, I thought we could keep things simple by only supporting CppClass*, but the pain of having to manually dereference everything, plus cumbersome dealings with objects that return objects by value, started to convince me otherwise. With very few exceptions (the new division semantics being one of them), I think it's valuable that code manipulating all C types translates nearly directly to C with very little extra magic or layers. One is then free to write as nice or raw of an interface on top of that as one wishes. I think applying the same principles to C+ + is a good idea. - Robert From dagss at student.matnat.uio.no Thu May 7 14:03:46 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 07 May 2009 14:03:46 +0200 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <716739d959ac4aa061bfe5a55f1b64fe.squirrel@webmail.uio.no> <0AE19092-B472-47D8-B0BA-E590768F9E5E@math.washington.edu>! <4A02BE99.3060608@student.matnat.uio.no> Message-ID: <4A02CE22.8050005@student.matnat.uio.no> Robert Bradshaw wrote: > With very few exceptions (the new division semantics being one of > them), I think it's valuable that code manipulating all C types > translates nearly directly to C with very little extra magic or > layers. One is then free to write as nice or raw of an interface on > top of that as one wishes. I think applying the same principles to C+ > + is a good idea. OK. Variable scoping is still a problem. I'll just dump my current, much clearer thoughts for the record. 1) The "with" clause has C variable scope semantics but would be kind of a pain to use. with vector[int].iterator(vec.begin()) as it: # cdefs the "it" in block while it != vec.end(): ... 2) Explicit construct/destruct is a possibility: cdef vector[int].iterator it ... construct(it, vec.begin()) destruct(it) ... But, ugly. If you construct twice in a function or forget to destruct you get to keep the pieces (often nothing bad will happen though). 3) Just tracking the first assignment, and destruct at end of function, would kind of work, but there's a problem: Basically the first assignment must invoke placement new, while subsequent assignments must use "=". This could be dealt with with a seperate flag though! So you have a flag "is_constructed", and if it is not, use placement new for assignment, otherwise use "=". As for eliminating such a flag, it would be easy to support cdef iterator it it = vec1.begin() # placement new if not foo: it = vec2.begin() # "=" but much more difficult to support if foo: it = vec1.begin() # placement new else: it = vec2.begin() # placement new and impossible to do if foo: it = vec1.begin() it = vec2.begin() # placement new or "="? need a flag Still, using such a flag in all situations before we have control flow analysis is a lot better than refcounting until we have control flow analysis :-) Finally, we could still have an explicit destruct(it) which would also toggle the flag back, making the next assignment reconstruct the object. This would actually work :-) (assuming void*[] is all the alignment we need). -- Dag Sverre From dagss at student.matnat.uio.no Thu May 7 14:10:02 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 07 May 2009 14:10:02 +0200 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <4A02CE22.8050005@student.matnat.uio.no> References: <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <716739d959ac4aa061bfe5a55f1b64fe.squirrel@webmail.uio.no> <0AE19092-B472-47D8-B0BA-E590768F9E5E@math.washington.edu>! <4A02BE99.3060608@student.matnat.uio.no> <4A02CE22.8050005@student.matnat.uio.no> Message-ID: <4A02CF9A.8090502@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > 3) Just tracking the first assignment, and destruct at end of function, > would kind of work, but there's a problem: Basically the first > assignment must invoke placement new, while subsequent assignments must > use "=". If this was unintuitive, consider the fact that when doing a = b what really happens is a.~ClassName(); // run destructor a.ClassName(b); // run copy constructor Thus we simply need to "achieve parity" with this, so to speak. -- Dag Sverre From dagss at student.matnat.uio.no Thu May 7 14:16:22 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 07 May 2009 14:16:22 +0200 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <4A02CF9A.8090502@student.matnat.uio.no> References: <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <716739d959ac4aa061bfe5a55f1b64fe.squirrel@webmail.uio.no> <0AE19092-B472-47D8-B0BA-E590768F9E5E@math.washington.edu>! <4A02BE99.3060608@student.matnat.uio.no> <4A02CE22.8050005@student.matnat.uio.no> <4A02CF9A.8090502@student.matnat.uio.no> Message-ID: <4A02D116.4010308@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Dag Sverre Seljebotn wrote: >> 3) Just tracking the first assignment, and destruct at end of function, >> would kind of work, but there's a problem: Basically the first >> assignment must invoke placement new, while subsequent assignments must >> use "=". > > If this was unintuitive, consider the fact that when doing > > a = b > > what really happens is > > a.~ClassName(); // run destructor > a.ClassName(b); // run copy constructor Wrong, wrong, wrong!! operator= is invoked! I'll just let this be thread be for now :-) -- Dag Sverre From kxroberto at googlemail.com Thu May 7 16:10:41 2009 From: kxroberto at googlemail.com (Robert) Date: Thu, 07 May 2009 16:10:41 +0200 Subject: [Cython] pyximport ignors [build] compiler = mingw32 Message-ID: On Windows in distutils/distutils.cfg MingW is set instead of VC default [build] compiler = mingw32 works well with extensions, including setup.py-method for building Cython modules. But it fails with pyximport method. Is this a bug - or how to tell using the MingW compiler? Robert >>> import pyximport >>> pyximport.install() >>> import cyex Traceback (most recent call last): File "", line 1, in File "C:\Python26\lib\site-packages\pyximport\pyximport.py", line 288, in load_module self.pyxbuild_dir) File "C:\Python26\lib\site-packages\pyximport\pyximport.py", line 154, in load_module raise ImportError("Building module failed: %s" % e) ImportError: Building module failed: DistutilsPlatformError('Unable to find vcvarsall.bat',) - File "C:\Python26\lib\distutils\dist.py", line 975, in run_commands self.run_command(cmd) File "C:\Python26\lib\distutils\dist.py", line 995, in run_command cmd_obj.run() File "C:\Python26\lib\distutils\command\build_ext.py", line 345, in run self.build_extensions() File "C:\Python26\lib\site-packages\Cython\Distutils\build_ext.py", line 83, in build_extensions self.build_extension(ext) File "C:\Python26\lib\distutils\command\build_ext.py", line 536, in build_extension depends=ext.depends) File "C:\Python26\lib\distutils\msvc9compiler.py", line 448, in compile self.initialize() File "C:\Python26\lib\distutils\msvc9compiler.py", line 358, in initialize vc_env = query_vcvarsall(VERSION, plat_spec) File "C:\Python26\lib\distutils\msvc9compiler.py", line 250, in query_vcvarsall raise DistutilsPlatformError("Unable to find vcvarsall.bat") DistutilsPlatformError: Unable to find vcvarsall.bat From dalcinl at gmail.com Thu May 7 18:44:24 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 7 May 2009 13:44:24 -0300 Subject: [Cython] pyximport ignors [build] compiler = mingw32 In-Reply-To: References: Message-ID: On Thu, May 7, 2009 at 11:10 AM, Robert wrote: > On Windows in distutils/distutils.cfg MingW is set instead of VC > default > > [build] > compiler = mingw32 > > > works well with extensions, including setup.py-method for building > Cython modules. But it fails with pyximport method. > Is this a bug - or how to tell using the MingW compiler? > I would not say it is a bug, but rather a consequence of the implementation of pyximport. Likely a call to "dist.parse_config_files()" should be added just before "dist.parse_command_line()" in pyximport/pyxbuild.py However, if you do this, a "setup.cfg" file will be parse if present in $PWD... This could be seen as a gotcha, or as a feature... > Robert > > > ?>>> import pyximport > ?>>> pyximport.install() > ?>>> import cyex > Traceback (most recent call last): > ? File "", line 1, in > ? File "C:\Python26\lib\site-packages\pyximport\pyximport.py", > line 288, in load_module > ? ? self.pyxbuild_dir) > ? File "C:\Python26\lib\site-packages\pyximport\pyximport.py", > line 154, in load_module > ? ? raise ImportError("Building module failed: %s" % e) > ImportError: Building module failed: > DistutilsPlatformError('Unable to find vcvarsall.bat',) > > > - > > ? File "C:\Python26\lib\distutils\dist.py", line 975, in run_commands > ? ? self.run_command(cmd) > ? File "C:\Python26\lib\distutils\dist.py", line 995, in run_command > ? ? cmd_obj.run() > ? File "C:\Python26\lib\distutils\command\build_ext.py", line > 345, in run > ? ? self.build_extensions() > ? File > "C:\Python26\lib\site-packages\Cython\Distutils\build_ext.py", > line 83, in build_extensions > ? ? self.build_extension(ext) > ? File "C:\Python26\lib\distutils\command\build_ext.py", line > 536, in build_extension > ? ? depends=ext.depends) > ? File "C:\Python26\lib\distutils\msvc9compiler.py", line 448, in > compile > ? ? self.initialize() > ? File "C:\Python26\lib\distutils\msvc9compiler.py", line 358, in > initialize > ? ? vc_env = query_vcvarsall(VERSION, plat_spec) > ? File "C:\Python26\lib\distutils\msvc9compiler.py", line 250, in > query_vcvarsall > ? ? raise DistutilsPlatformError("Unable to find vcvarsall.bat") > DistutilsPlatformError: Unable to find vcvarsall.bat > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Thu May 7 21:17:55 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 May 2009 12:17:55 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <4A02CE22.8050005@student.matnat.uio.no> References: <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <716739d959ac4aa061bfe5a55f1b64fe.squirrel@webmail.uio.no> <0AE19092-B472-47D8-B0BA-E590768F9E5E@math.washington.edu>! <4A02BE99.3060608@student.matnat.uio.no> <4A02CE22.805000! 5@student.matnat.uio.no> Message-ID: On May 7, 2009, at 5:03 AM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> With very few exceptions (the new division semantics being one of >> them), I think it's valuable that code manipulating all C types >> translates nearly directly to C with very little extra magic or >> layers. One is then free to write as nice or raw of an interface on >> top of that as one wishes. I think applying the same principles to C+ >> + is a good idea. > > OK. Variable scoping is still a problem. I'll just dump my current, > much > clearer thoughts for the record. > > 1) The "with" clause has C variable scope semantics but would be > kind of > a pain to use. > > with vector[int].iterator(vec.begin()) as it: # cdefs the "it" in > block > while it != vec.end(): > ... > > 2) Explicit construct/destruct is a possibility: > > cdef vector[int].iterator it > ... > construct(it, vec.begin()) > > destruct(it) > ... > > But, ugly. If you construct twice in a function or forget to destruct > you get to keep the pieces (often nothing bad will happen though). > > 3) Just tracking the first assignment, and destruct at end of > function, > would kind of work, but there's a problem: Basically the first > assignment must invoke placement new, while subsequent assignments > must > use "=". > > This could be dealt with with a seperate flag though! So you have a > flag > "is_constructed", and if it is not, use placement new for assignment, > otherwise use "=". > > As for eliminating such a flag, it would be easy to support > > cdef iterator it > it = vec1.begin() # placement new > if not foo: > it = vec2.begin() # "=" > > but much more difficult to support > > if foo: > it = vec1.begin() # placement new > else: > it = vec2.begin() # placement new > > and impossible to do > > if foo: > it = vec1.begin() > it = vec2.begin() # placement new or "="? need a flag > > Still, using such a flag in all situations before we have control flow > analysis is a lot better than refcounting until we have control flow > analysis :-) > > Finally, we could still have an explicit > > destruct(it) > > which would also toggle the flag back, making the next assignment > reconstruct the object. > > This would actually work :-) (assuming void*[] is all the alignment we > need). I much prefer option (3), that seems both feasible and obvious to use. - Robert From dagss at student.matnat.uio.no Thu May 7 21:32:41 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 07 May 2009 21:32:41 +0200 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <716739d959ac4aa061bfe5a55f1b64fe.squirrel@webmail.uio.no> <0AE19092-B472-47D8-B0BA-E590768F9E5E@math.washington.edu>! <4A02BE99.3060608@student.matnat.uio.no> <4A02CE22.805000! 5@student.matnat.uio.no> Message-ID: <4A033759.7060007@student.matnat.uio.no> Robert Bradshaw wrote: > On May 7, 2009, at 5:03 AM, Dag Sverre Seljebotn wrote: > >> Robert Bradshaw wrote: >>> With very few exceptions (the new division semantics being one of >>> them), I think it's valuable that code manipulating all C types >>> translates nearly directly to C with very little extra magic or >>> layers. One is then free to write as nice or raw of an interface on >>> top of that as one wishes. I think applying the same principles to C+ >>> + is a good idea. >> OK. Variable scoping is still a problem. I'll just dump my current, >> much >> clearer thoughts for the record. >> >> 1) The "with" clause has C variable scope semantics but would be >> kind of >> a pain to use. >> >> with vector[int].iterator(vec.begin()) as it: # cdefs the "it" in >> block >> while it != vec.end(): >> ... >> >> 2) Explicit construct/destruct is a possibility: >> >> cdef vector[int].iterator it >> ... >> construct(it, vec.begin()) >> >> destruct(it) >> ... >> >> But, ugly. If you construct twice in a function or forget to destruct >> you get to keep the pieces (often nothing bad will happen though). >> >> 3) Just tracking the first assignment, and destruct at end of >> function, >> would kind of work, but there's a problem: Basically the first >> assignment must invoke placement new, while subsequent assignments >> must >> use "=". >> >> This could be dealt with with a seperate flag though! So you have a >> flag >> "is_constructed", and if it is not, use placement new for assignment, >> otherwise use "=". >> >> As for eliminating such a flag, it would be easy to support >> >> cdef iterator it >> it = vec1.begin() # placement new >> if not foo: >> it = vec2.begin() # "=" >> >> but much more difficult to support >> >> if foo: >> it = vec1.begin() # placement new >> else: >> it = vec2.begin() # placement new >> >> and impossible to do >> >> if foo: >> it = vec1.begin() >> it = vec2.begin() # placement new or "="? need a flag >> >> Still, using such a flag in all situations before we have control flow >> analysis is a lot better than refcounting until we have control flow >> analysis :-) >> >> Finally, we could still have an explicit >> >> destruct(it) >> >> which would also toggle the flag back, making the next assignment >> reconstruct the object. >> >> This would actually work :-) (assuming void*[] is all the alignment we >> need). > > I much prefer option (3), that seems both feasible and obvious to use. Sounds like a plan. (I guess a @notassignedtoyetcheck directive is in order too then). I'd feel it would be natural to allow f() = 3 and f(a) # a is modified as well then -- why stop at 90%... I'd like to eventually have (1) available as well a more obvious alternative in case you *need* explicit control over when the destructor is called, or the object is a bit big. with DbConnection("http://...") as db: ... seems very natural, and this is a very common idiom in C++ (they basically use stack allocation for both the "normal" situations, *and* for situations where one would use "with" in Python). It would then be natural tohave this in general, so that you could do with cython.int(3) as a: # locally scoped "cdef int a = 3" -- Dag Sverre From stefan_ml at behnel.de Thu May 7 21:46:26 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 07 May 2009 21:46:26 +0200 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <716739d959ac4aa061bfe5a55f1b64fe.squirrel@webmail.uio.no> <0AE19092-B472-47D8-B0BA-E590768F9E5E@math.washington.edu>! <4A02BE99.3060608@student.matnat.uio.no> <4A02CE22.805000! 5@student.matnat.uio.no> Message-ID: <4A033A92.3000100@behnel.de> Robert Bradshaw wrote: > On May 7, 2009, at 5:03 AM, Dag Sverre Seljebotn wrote: >> 3) Just tracking the first assignment, and destruct at end of >> function, >> would kind of work, but there's a problem: Basically the first >> assignment must invoke placement new, while subsequent assignments >> must >> use "=". >> >> This could be dealt with with a seperate flag though! So you have a >> flag >> "is_constructed", and if it is not, use placement new for assignment, >> otherwise use "=". >> >> As for eliminating such a flag, it would be easy to support >> >> cdef iterator it >> it = vec1.begin() # placement new >> if not foo: >> it = vec2.begin() # "=" >> >> but much more difficult to support >> >> if foo: >> it = vec1.begin() # placement new >> else: >> it = vec2.begin() # placement new >> >> and impossible to do >> >> if foo: >> it = vec1.begin() >> it = vec2.begin() # placement new or "="? need a flag >> >> Still, using such a flag in all situations before we have control flow >> analysis is a lot better than refcounting until we have control flow >> analysis :-) >> >> Finally, we could still have an explicit >> >> destruct(it) >> >> which would also toggle the flag back, making the next assignment >> reconstruct the object. >> >> This would actually work :-) (assuming void*[] is all the alignment we >> need). > > I much prefer option (3), that seems both feasible and obvious to use. +1, makes sense to me. Stefan From dagss at student.matnat.uio.no Thu May 7 22:09:13 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 07 May 2009 22:09:13 +0200 Subject: [Cython] Buffer passing implementation details In-Reply-To: References: <4A01BB4A.1060003@student.matnat.uio.no> Message-ID: <4A033FE9.9060001@student.matnat.uio.no> Seems I got too caught up in C++ stuff :-) Here's an answer. Kurt Smith wrote: > On Wed, May 6, 2009 at 11:31 AM, Dag Sverre Seljebotn > wrote: >> (Kurt is the major recipient of this.) >> >> I've been thinking some more on buffer passing. In the end (and perhaps >> even in summer; I could perhaps have a go at it working alongside you) >> the scenario we are looking at is something like >> >> external_set_foo_array(foo_handle, self.arr[2::2]) >> >> with fast operations all the way. >> >> That is >> a) Efficient slicing without Python overhead (#178) >> b) Storing acquired buffers in cdef class fields (#301) >> c) External functions can keep a reference to the buffer (not really >> necesarry, but it is necesarry for internal Cython "cdef" functions, and >> it would be nice to treat them the same) >> d) The base pointer may have to be moved (again slicing can do this) >> >> This seems to make it clear that >> a) The Py_buffer is not suitable as the primary vessel of our buffer >> data, e.g. to the external function. It can't work that well with >> slicing as we must maintain the original Py_buffer data when releasing it. >> b) We need a reference count. E.g. doing a slice, or assigning to >> self.arr, would increase this count. This must go on the heap; and so we >> might as well put the entire Py_buffer on the heap. > > Doing the slicing manually on the buffer data and having a refcounted > heap-allocated Py_buffer would address the object-Py_buffer data > synchronization problem, right? The above modifications would have > that, once the buffer is acquired, all modifications to it are done at > the buffer level. Not sure what you mean, but: Once the buffer is required, all new slices etc. is done in the local structs and the Py_buffer just sits there for querying information that doesn't change and for use release time. > Although, what about when we require a contiguous copy (or any mode, > for that matter) of a buffer? Would we set the top-level object > reference to None, as suggested in a previous thread? That would > prevent slicing at the Python API level, though, right? Hmm. I'm still thinking about contiguous copies. Let's deal with that in the next iteration. > General question: how difficult would it be for the Cython-side > programmer to get the underlying char *data inside the Py_buffer, in > case it needs to be accessed? It seems there are more than a few > 'layers' -- are they all necessary? The justifications I see for each > are the following: > > typedef struct { > void *buf; > PyObject *obj; > Py_ssize_t len; > Py_ssize_t itemsize; > int readonly; > int ndim; > char *format; > Py_ssize_t *shape; > Py_ssize_t *strides; > Py_ssize_t *suboffsets; > void *internal; > } Py_buffer; I think the presence of the obj field here is a bit unsatisfactory. It seems that after months (not to say years) of buffer PEP discussion, someone just added it in a bugfix *sigh* Anyway it is not to my knowledge documented anywhere but there's a long discussion about it here: http://bugs.python.org/issue3139 (You don't really need to read it though.) Things seem a bit unclear due to the lack of official documentation, but at least for now I don't want to assume that the PyObject filled in here is the same as the object you acquire the buffer from. The obj field is filled in by the tp_getbuffer implementation which could do things like make a dedicated copy of the buffer into a new object and then pass that new object into the obj field. Anyway, the idea (I gather from reading CPython sources) is that the obj is who should recieve the release call, which may or may not be the object you acquire on. So really, obj here is just present as the recipient of a release. It might even be set to NULL if there's no need to release the buffer. > The "base layer" -- interface structure for tp_getbuffer from objects, etc. > > typedef struct { > size_t refcount; > Py_buffer bufinfo; > } __Pyx_Buffer; > > The Cython wrapper around Py_buffer -- includes cython-managed > refcounting upon assignment/slicing (other cases?) Basically whenever the __Pyx_buffer* is copied. > typedef struct { > __Pyx_buffer* bufinfo; > char* data; > Py_ssize_t shape0, stride0; > } __Pyx_StridedBuf_1D; > > The "int[:] buf" buffer -- exists to make the new buffer syntax more > transparent (not part of GSoC, but putting in framework for it). char > *data pointer to allow fast slicing w/o Python API (e.g. arr[10:]), > shape0 & stride0 copied from buffer for optimizations. Yep. (And shape0 and stride0 also to allow fast slicing; we don't want to be modifying the shape/stride arrays passed in the Py_buffer) To be pedantic (and make sure things are understood correctly) there's not a Python API for slicing, only a NumPy one. > > typedef struct { > __Pyx_StridedBuf_1D buffer; > PyObject* object; > } __Pyx_StridedBuf_1D_Obj; > > The "object[int, mode='strided'] buf" object buffer. This is the top > level and holds a reference to the original PyObject from which the > buffer was acquired. Object reference required when a python-level > access/assignment is required. Yes. a) Since this doesn't have to be the same as the on in Py_buffer, a seperate reference seems necesarry, b) storing a function local variable inside a pointer will make accesses to the object part of the variable slightly slower. > So, initially, if one passes a 'buffer' to a function like so: > > external_func(arr[2::2]) > > What is the slicing modifying/updating? The top-level through the > Python API layer via the PyObject? Or is it being done to the > __Pyx_StridedBuf_1D members -- offsetting the char *data pointer by 2, > and changing the shape0 and stride0 members? (Would it be the top > layer object at first & for the GSoC, then when faster, low-level > slicing is implemented, directly on the lower-level?) Yes. First & for GSoC, we wouldn't know anything about "arr[2::2]", it would just be "some Python expression". It would (if arr is a NumPy object) return a new NumPy object from which we can acquire a correct Py_buffer. When slicing is optimized, we don't have that luxury, and must store the info in shape0/stride0. > > Besides Python API-level slicing, why is the top-level object > reference required? Is it for buffer reacquisition through > tp_getbuffer? That will be handled at the next lower level through a > struct copy by value and increase of the refcount. It isn't for > buffer release, either, since that doesn't require the original object > reference explicitly. (There is an object reference at the Py_buffer > level). It is needed because when you do cdef np.ndarray[int] obj = ... obj.fooblargh() on which object do you execute "fooblargh()"? (And we can't rely on the Py_buffer level one.) > Basically, nothing Cython-side would require the PyObject from which > the original buffer is acquired once all these are implemented, right? > (Again, not all part of GSoC, but at least putting in the framework.) Yes, that would happen at the same rate that the cdef object[int] a syntax is deprecated in favor of cdef int[:] a This is really just linked to syntax. >> It sheems a shame to let go of a neatly PEP-defined Py_buffer for >> passing to external functions, but I think it won't be too bad if we >> with each Cython version ship nice C header and Fortran include files >> containing the appropriate structs and access macros. > > I guess the access macros would make it easy for the external > functions to get to the important stuff, so my previous concern isn't > so essential (removing the 'top level' layer from the buffer struct > heirarchy). Well, if you think it is easier we can just introduce the new syntax only in a few limited spots for your GSoC and thus drop the top level. Your pick. I don't think it makes much of a difference. -- Dag Sverre From bblais at gmail.com Fri May 8 01:10:23 2009 From: bblais at gmail.com (Brian Blais) Date: Thu, 07 May 2009 19:10:23 -0400 Subject: [Cython] class versus struct...optimization Message-ID: Hello, I am trying to write a neuron simulator, and I have the need for speed. :) Unfortunately, I seem to be taking a serious performance hit by writing things as classes, as opposed to c-structs. I am including the code below (I hope it's not too much) of the smallest meaningful example I could write (although it still doesn't do anything real. :) ). The first is an all-python version, which is easy to follow but slow. In [1]:%timeit py1.runit() 10 loops, best of 3: 606 ms per loop The second is a cython version, which makes extension classes to replace the python ones, and I get a nice speed up of about 30x: In [2]:%timeit cpy2.runit() 10 loops, best of 3: 20.9 ms per loop The final one is ugly code, making an array of structures instead of lists of class instances. Where I would have more classes in the second version, I'd pile those variables all into this one big struct so I can make an array of them easily. It's ugly, but much faster: In [3]:%timeit cpy3.runit() 1000 loops, best of 3: 1.29 ms per loop My question is, can I have the best of both worlds (i.e. am I doing something obviously stupid), or is this the price for cleaner object- oriented code? Are the the method look-ups killing me in the object- oriented way? In the larger code I am working on, the difference between the class versus struct versions is about a factor of 5, not the factor of 20 above, but there is more going on. Still, a factor of 5 makes a big difference if the simulation may take a day to run! thanks, Brian Blais -- Brian Blais bblais at gmail.com http://web.bryant.edu/~bblais #================================================ #==================PYTHON VERSION================ #================================================ from numpy import ones,zeros,prod class Neuron(object): """docstring for Neuron""" def __init__(self, N): super(Neuron, self).__init__() self.N = N self.type=1 self.shape=(N,) self.spikes=zeros(N,int) self.old_spikes=zeros(N,int) self.V=zeros(N) self.connections_to=[] def update(self,t): for c in self.connections_to: V=self.__getattribute__('V') w=c.weights V+=w[:,c.incell.spikes.nonzero()[0]].sum(axis=1) V=self.V self.V-=self.V/100.0 self.spikes[:]=1.0 class Connection(object): """docstring for Connection""" def __init__(self, incell,outcell): super(Connection, self).__init__() self.incell=incell self.outcell=outcell self.shape=(outcell.N,incell.N) self.outcell.connections_to.append(self) self.weights=ones((outcell.N,incell.N),float) def update(self,t): pass def run_sim( duration,neurons,connections): t=0.0 while t<=duration: for n in neurons: n.update(t) # for c in connections: # c.update(t) t+=1.0 def runit(): n1=Neuron(5) n2=Neuron(3) c=Connection(n1,n2) run_sim(10000,[n1,n2],[c]) #================================================ #==================CYTHON VERSION WITH CLASSES================ #================================================ import numpy as np cimport numpy as np DTYPE=np.float64 ctypedef np.float64_t DTYPE_t DTYPE_I=np.int ctypedef np.int_t DTYPE_I_t from numpy import ones,zeros,prod cdef class Neuron: """docstring for Neuron""" cdef readonly int N cdef public np.ndarray spikes cdef public np.ndarray V cdef public object connections_to def __getitem__(self,key): return self.__getattribute__(key) def __init__(self, N): self.N = N self.spikes=zeros(N,int) self.V=zeros(N) self.connections_to=[] cdef void _update(self,double t): cdef int __i,__j cdef np.ndarray[DTYPE_I_t,ndim=1] spikes cdef np.ndarray[DTYPE_t,ndim=1] V cdef np.ndarray[DTYPE_t,ndim=2] _W cdef int k for c in self.connections_to: V=self.__getattribute__('V') _W=c.weights spikes=c.incell.spikes k=c.weights.shape[1] for __i in range(self.N): for __j in range(k): if spikes[__j]: V[__i]+=_W[__i,__j] V=self.V spikes=self.spikes for __i in range(self.N): V[__i]-=V[__i]/100.0 spikes[__i]=1 def update(self,double t): self._update(t) cdef class Connection: """docstring for Connection""" cdef public object incell,outcell cdef public object shape cdef public np.ndarray weights def __init__(self, incell,outcell): self.incell=incell self.outcell=outcell self.shape=(outcell.N,incell.N) self.outcell.connections_to.append(self) self.weights=ones((outcell.N,incell.N),float) def update(self,t): pass def run_sim(duration,neurons,connections): t=0.0 while t<=duration: for n in neurons: n.update(t) # for c in connections: # c.update(t) t+=1.0 #================================================ #==================CYTHON VERSION WITH STRUCTS================ #================================================ import numpy as np cimport numpy as np DTYPE=np.float64 ctypedef np.float64_t DTYPE_t cdef double* DoubleData(np.ndarray M): return M.data cdef char* CharData(np.ndarray M): return M.data cdef int* IntData(np.ndarray M): return M.data cdef struct Neuron_Group_struct: int N int *spikes double *V int type int number_of_connections_to # hard-coded maximum numbers...yuck! int *c_spikes[31] int c_num_incell[31],c_num_outcell[31] double *c_weights[31] # function to initialize the struct cdef Neuron_Group_struct init_Neuron_Group(object self): cdef Neuron_Group_struct s s.type=self.type s.N=self.N s.spikes=IntData(self.spikes) s.V=DoubleData(self.V) s.number_of_connections_to=len(self.connections_to) cdef int cg for cg from 0<=cg References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A017E4F.5050800@student.matnat.uio.no> <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A0188D4.1050208@student.matnat.uio.no> <11E61C9F-3175-4473-8B39-20277284E9D1@math.washington.edu> <4B519262-77AC-4BC4-A62B-026B4E254FAA@math.washington.edu> <4A027E52.9070003@behnel.de> Message-ID: <4A037454.4010805@canterbury.ac.nz> Stefan Behnel wrote: > So this would mimic the "public" mechanism in a way, although it'd be > better to do the normal name mangling for tp_new() even if we export it. Yes, the name should definitely still be mangled, since that's the only way of distinguishing between classes having the same name in different modules. And there's no reason not to mangle, because the programmer won't be calling it directly. -- Greg From greg.ewing at canterbury.ac.nz Fri May 8 02:18:51 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 08 May 2009 12:18:51 +1200 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <4A023FA2.3040001@canterbury.ac.nz> <4A027A1E.7020604@behnel.de> <47F3AE68-2D7B-44DD-90E7-14587EEB0D33@math.washington.edu> Message-ID: <4A037A6B.9000002@canterbury.ac.nz> Stefan Behnel wrote: > I think it makes > sense to add that to the "extern" 'keyword' to say: "this external > declaration obeys C++ semantics". External declarations aren't necessarily the only place you might want to declare something as having C++ semantics, though. The advantage of cdef+ (or any variation that modifies the cdef keyword) is that it's easily applied to any existing C declaration, and there's no need to invent a new syntax for each case. -- Greg From dalcinl at gmail.com Fri May 8 02:42:50 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 7 May 2009 21:42:50 -0300 Subject: [Cython] class versus struct...optimization In-Reply-To: References: Message-ID: Brian, in the Cython version, class Neuron, do this: instead of "def update(...)", please write "cpdef update(...)" and time your code again. If this make it faster (it should be near the C-struct version), please consider first removing your original "def update", and replace your "cdef _update()" by "cpdef update()". On Thu, May 7, 2009 at 8:10 PM, Brian Blais wrote: > Hello, > I am trying to write a neuron simulator, and I have the need for speed. ?:) > ?Unfortunately, I seem to be taking a serious performance hit by writing > things as classes, as opposed to c-structs. ?I am including the code below > (I hope it's not too much) of the smallest meaningful example I could write > (although it still doesn't do anything real. :) ?). ?The first is an > all-python version, which is easy to follow but slow. > In [1]:%timeit py1.runit() > 10 loops, best of 3: 606 ms per loop > > The second is a cython version, which makes extension classes to replace the > python ones, and I get a nice speed up of about 30x: > In [2]:%timeit cpy2.runit() > 10 loops, best of 3: 20.9 ms per loop > > The final one is ugly code, making an array of structures instead of lists > of class instances. ?Where I would have more classes in the second version, > I'd pile those variables all into this one big struct so I can make an array > of them easily. ?It's ugly, but much faster: > In [3]:%timeit cpy3.runit() > 1000 loops, best of 3: 1.29 ms per loop > > My question is, can I have the best of both worlds (i.e. am I doing > something obviously stupid), or is this the price for cleaner > object-oriented code? ?Are the the method look-ups killing me in the > object-oriented way? ?In the larger code I am working on, the difference > between the class versus struct versions is about a factor of 5, not the > factor of 20 above, but there is more going on. ?Still, a factor of 5 makes > a big difference if the simulation may take a day to run! > thanks, > Brian Blais > > > -- > Brian Blais > bblais at gmail.com > http://web.bryant.edu/~bblais > > #================================================ > #==================PYTHON VERSION================ > #================================================ > from numpy import ones,zeros,prod > class Neuron(object): > ?? ?"""docstring for Neuron""" > ?? ?def __init__(self, N): > ?? ? ? ?super(Neuron, self).__init__() > ?? ? ? ?self.N = N > ?? ? ? ?self.type=1 > ?? ? ? ?self.shape=(N,) > ?? ? ? ?self.spikes=zeros(N,int) > ?? ? ? ?self.old_spikes=zeros(N,int) > ?? ? ? ?self.V=zeros(N) > ?? ? ? ?self.connections_to=[] > > ?? ?def update(self,t): > > ?? ? ? ?for c in self.connections_to: > ?? ? ? ? ? ?V=self.__getattribute__('V') > ?? ? ? ? ? ?w=c.weights > ?? ? ? ? ? ?V+=w[:,c.incell.spikes.nonzero()[0]].sum(axis=1) > ?? ? ? ?V=self.V > ?? ? ? ?self.V-=self.V/100.0 > ?? ? ? ?self.spikes[:]=1.0 > > > class Connection(object): > ?? ?"""docstring for Connection""" > ?? ?def __init__(self, incell,outcell): > ?? ? ? ?super(Connection, self).__init__() > ?? ? ? ?self.incell=incell > ?? ? ? ?self.outcell=outcell > ?? ? ? ?self.shape=(outcell.N,incell.N) > ?? ? ? ?self.outcell.connections_to.append(self) > ?? ? ? ?self.weights=ones((outcell.N,incell.N),float) > > > ?? ?def update(self,t): > ?? ? ? ?pass > > def run_sim( duration,neurons,connections): > > ?? ?t=0.0 > ?? ?while t<=duration: > ?? ? ? ?for n in neurons: > ?? ? ? ? ? ?n.update(t) > > ?? ? ? ?# for c in connections: > ?? ? ? ?# ? ? c.update(t) > ?? ? ? ?t+=1.0 > def runit(): > ?? ?n1=Neuron(5) > ?? ?n2=Neuron(3) > > ?? ?c=Connection(n1,n2) > > ?? ?run_sim(10000,[n1,n2],[c]) > > > > #================================================ > #==================CYTHON VERSION WITH CLASSES================ > #================================================ > import numpy as np > cimport numpy as np > > DTYPE=np.float64 > ctypedef np.float64_t DTYPE_t > DTYPE_I=np.int > ctypedef np.int_t DTYPE_I_t > from numpy import ones,zeros,prod > cdef class Neuron: > ?? ?"""docstring for Neuron""" > > ?? ?cdef readonly int N > ?? ?cdef public np.ndarray spikes > ?? ?cdef public np.ndarray V > ?? ?cdef public object connections_to > ?? ?def __getitem__(self,key): > ?? ? ? ?return self.__getattribute__(key) > > > ?? ?def __init__(self, N): > ?? ? ? ?self.N = N > ?? ? ? ?self.spikes=zeros(N,int) > ?? ? ? ?self.V=zeros(N) > ?? ? ? ?self.connections_to=[] > > ?? ?cdef void _update(self,double t): > ?? ? ? ?cdef int __i,__j > > ?? ? ? ?cdef np.ndarray[DTYPE_I_t,ndim=1] spikes > ?? ? ? ?cdef np.ndarray[DTYPE_t,ndim=1] V > ?? ? ? ?cdef np.ndarray[DTYPE_t,ndim=2] _W > ?? ? ? ?cdef int k > > > ?? ? ? ?for c in self.connections_to: > ?? ? ? ? ? ?V=self.__getattribute__('V') > ?? ? ? ? ? ?_W=c.weights > ?? ? ? ? ? ?spikes=c.incell.spikes > ?? ? ? ? ? ?k=c.weights.shape[1] > ?? ? ? ? ? ?for __i in range(self.N): > ?? ? ? ? ? ? ? ?for __j in range(k): > ?? ? ? ? ? ? ? ? ? ?if spikes[__j]: > ?? ? ? ? ? ? ? ? ? ? ? ?V[__i]+=_W[__i,__j] > > ?? ? ? ?V=self.V > ?? ? ? ?spikes=self.spikes > ?? ? ? ?for __i in range(self.N): > ?? ? ? ? ? ?V[__i]-=V[__i]/100.0 > ?? ? ? ? ? ?spikes[__i]=1 > > > ?? ?def update(self,double t): > ?? ? ? ?self._update(t) > > > cdef class Connection: > ?? ?"""docstring for Connection""" > ?? ?cdef public object incell,outcell > ?? ?cdef public object shape > ?? ?cdef public np.ndarray weights > > ?? ?def __init__(self, incell,outcell): > ?? ? ? ?self.incell=incell > ?? ? ? ?self.outcell=outcell > ?? ? ? ?self.shape=(outcell.N,incell.N) > ?? ? ? ?self.outcell.connections_to.append(self) > ?? ? ? ?self.weights=ones((outcell.N,incell.N),float) > > > ?? ?def update(self,t): > ?? ? ? ?pass > > def run_sim(duration,neurons,connections): > > ?? ?t=0.0 > ?? ?while t<=duration: > ?? ? ? ?for n in neurons: > ?? ? ? ? ? ?n.update(t) > > ?? ? ? ?# for c in connections: > ?? ? ? ?# ? ? c.update(t) > ?? ? ? ?t+=1.0 > > #================================================ > #==================CYTHON VERSION WITH STRUCTS================ > #================================================ > import numpy as np > cimport numpy as np > > DTYPE=np.float64 > ctypedef np.float64_t DTYPE_t > > cdef double* DoubleData(np.ndarray M): > ?? ?return M.data > cdef char* CharData(np.ndarray M): > ?? ?return M.data > > cdef int* IntData(np.ndarray M): > ?? ?return M.data > cdef struct Neuron_Group_struct: > ?? ?int N > ?? ?int *spikes > ?? ?double *V > ?? ?int type > ?? ?int number_of_connections_to > > ?? ?# hard-coded maximum numbers...yuck! > ?? ?int *c_spikes[31] > ?? ?int c_num_incell[31],c_num_outcell[31] > ?? ?double *c_weights[31] > > > # function to initialize the struct > cdef Neuron_Group_struct init_Neuron_Group(object self): > ?? ?cdef Neuron_Group_struct s > ?? ?s.type=self.type > ?? ?s.N=self.N > ?? ?s.spikes=IntData(self.spikes) > ?? ?s.V=DoubleData(self.V) > ?? ?s.number_of_connections_to=len(self.connections_to) > > ?? ?cdef int cg > > ?? ?for cg from 0<=cg ?? ? ? ?s.c_spikes[cg]=IntData(self.connections_to[cg].incell.spikes) > ?? ? ? ?s.c_num_incell[cg]=self.connections_to[cg].incell.N > ?? ? ? ?s.c_num_outcell[cg]=self.connections_to[cg].outcell.N > > ?? ? ? ?s.c_weights[cg]=DoubleData(self.connections_to[cg].weights) > > cdef void Test_update(Neuron_Group_struct *s,object self,double t): > ?? ?cdef int __i,__j > ?? ?cdef int number_of_connections_to > ?? ?number_of_connections_to=s.number_of_connections_to > ?? ?cdef int *spikes > ?? ?cdef int num_incell,num_outcell > ?? ?cdef int cg,ni,no > ?? ?cdef double *weights > ?? ?cdef double *V > > ?? ?V=s.V > > ?? ?for cg from 0<=cg ?? ? ? ?spikes=s.c_spikes[cg] > ?? ? ? ?num_incell=s.c_num_incell[cg] > ?? ? ? ?num_outcell=s.c_num_outcell[cg] > > ?? ? ? ?weights=s.c_weights[cg] > ?? ? ? ?for ni from 0<=ni ?? ? ? ? ? ?if spikes[ni]: > ?? ? ? ? ? ? ? ?for no from 0<=no ?? ? ? ? ? ? ? ? ? V[ni]+=weights[no+ni*num_outcell] > ?? ?V=s.V > ?? ?spikes=s.spikes > ?? ?for no from 0<=no ?? ? ? ?V[no]-=V[no]/100.0 > ?? ? ? ?spikes[no]=1 > > cdef nupdate(Neuron_Group_struct *s,object self,double t): > ?? ?if s.type==-1: > ?? ? ? ?self.update(t) > ?? ? ? ?return > ?? ?if s.type==0: # Silent Neuron > ?? ? ? ?pass > ?? ?elif s.type==1: # Constant Fixed > ?? ? ? ?Test_update(s,self,t) > > def run_sim(duration,neurons,connections): > ?? ?cdef long long t > ?? ?cdef int i > ?? ?cdef int ln,lc > ?? ?ln=len(neurons) > ?? ?lc=len(connections) > ?? ?cdef long long num_iter > ?? ?num_iter=duration > > ?? ?cdef Neuron_Group_struct Sn[31] > ?? ?for i from 0<=i ?? ? ? ?Sn[i]=init_Neuron_Group(neurons[i]) > ?? ?for t from 0<=t ?? ? ? ?for i from 0<=i ?? ? ? ? ? ?nupdate(&Sn[i],neurons[i],t) > > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From josef.pktd at gmail.com Fri May 8 02:37:58 2009 From: josef.pktd at gmail.com (josef) Date: Fri, 8 May 2009 00:37:58 +0000 (UTC) Subject: [Cython] pyximport ignors [build] compiler = mingw32 References: Message-ID: Robert writes: > > On Windows in distutils/distutils.cfg MingW is set instead of VC > default > > [build] > compiler = mingw32 > > works well with extensions, including setup.py-method for building > Cython modules. But it fails with pyximport method. > Is this a bug - or how to tell using the MingW compiler? > I just hard coded the required include path and compiler in my installed pyxbuild.py, as a quick solution, since I didn't find any options to do it otherwise. args = [quiet, "build_ext", "--compiler=mingw32", r"-Ic:\programs\python25\lib\site-packages\numpy\core\include"] Josef From bblais at gmail.com Fri May 8 12:08:08 2009 From: bblais at gmail.com (Brian Blais) Date: Fri, 08 May 2009 06:08:08 -0400 Subject: [Cython] class versus struct...optimization In-Reply-To: References: Message-ID: On May 7, 2009, at 20:42 , Lisandro Dalcin wrote: > Brian, in the Cython version, class Neuron, do this: instead of "def > update(...)", please write "cpdef update(...)" and time your code > again. If this make it faster (it should be near the C-struct > version), please consider first removing your original "def update", > and replace your "cdef _update()" by "cpdef update()". > thanks, but this didn't change the time much at all (something like . 5 ms out of 20ms). I was following the example in the cython docs Early Binding for Speed section, but now I see it got updated to cpdef! I'll keep hacking away at it. Perhaps I'll try a smaller example, to see the difference between lookups in classes and in structs, if that really is the issue. Is there a better way to find speed bottlenecks than the "comment out and re-compile" method? :) thanks, bb -- Brian Blais bblais at gmail.com http://web.bryant.edu/~bblais -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090508/9889e5e3/attachment.htm From robertwb at math.washington.edu Fri May 8 12:22:17 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 8 May 2009 03:22:17 -0700 Subject: [Cython] class versus struct...optimization In-Reply-To: References: Message-ID: On May 8, 2009, at 3:08 AM, Brian Blais wrote: > On May 7, 2009, at 20:42 , Lisandro Dalcin wrote: > >> Brian, in the Cython version, class Neuron, do this: instead of "def >> update(...)", please write "cpdef update(...)" and time your code >> again. If this make it faster (it should be near the C-struct >> version), please consider first removing your original "def update", >> and replace your "cdef _update()" by "cpdef update()". >> > > thanks, but this didn't change the time much at all (something > like .5 ms out of 20ms). I was following the example in the cython > docs Early Binding for Speed section, but now I see it got updated > to cpdef! > > I'll keep hacking away at it. Perhaps I'll try a smaller example, > to see the difference between lookups in classes and in structs, if > that really is the issue. > > Is there a better way to find speed bottlenecks than the "comment > out and re-compile" method? :) That's of course the only sure way to do it... Try compiling with "cython -a" and then opening up the resulting .html file. Yellow means Python/C API calls, which are often an indication of unintended inefficiencies. - Robert From kxroberto at googlemail.com Fri May 8 12:25:46 2009 From: kxroberto at googlemail.com (Robert) Date: Fri, 08 May 2009 12:25:46 +0200 Subject: [Cython] pyximport ignors [build] compiler = mingw32 In-Reply-To: References: Message-ID: Lisandro Dalcin wrote: > On Thu, May 7, 2009 at 11:10 AM, Robert wrote: >> On Windows in distutils/distutils.cfg MingW is set instead of VC >> default >> >> [build] >> compiler = mingw32 >> >> >> works well with extensions, including setup.py-method for building >> Cython modules. But it fails with pyximport method. >> Is this a bug - or how to tell using the MingW compiler? >> > > I would not say it is a bug, but rather a consequence of the > implementation of pyximport. > > Likely a call to "dist.parse_config_files()" should be added just > before "dist.parse_command_line()" in pyximport/pyxbuild.py > > However, if you do this, a "setup.cfg" file will be parse if present > in $PWD... This could be seen as a gotcha, or as a feature... > Thanks, in attachment my patch so far to: * only load the global distutils.cfg * extend the .pyxbld scheme for enabling custom setup() args * speed up import pyximport;pyximport.install() for use in auto startup files (by late imports). It was quite slowing down python startup crucially. Maybe its interesting for sb else too, or for general. Didn't find into http://trac.cython.org/cython_trac/ so far. Robert -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: pyxi.patch Url: http://codespeak.net/pipermail/cython-dev/attachments/20090508/6645f137/attachment.diff From robertwb at math.washington.edu Fri May 8 12:29:22 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 8 May 2009 03:29:22 -0700 Subject: [Cython] pyximport ignors [build] compiler = mingw32 In-Reply-To: References: Message-ID: <1F3DF633-6EF5-4AE9-B109-F3D481E55F68@math.washington.edu> On May 8, 2009, at 3:25 AM, Robert wrote: > Lisandro Dalcin wrote: >> On Thu, May 7, 2009 at 11:10 AM, Robert >> wrote: >>> On Windows in distutils/distutils.cfg MingW is set instead of VC >>> default >>> >>> [build] >>> compiler = mingw32 >>> >>> >>> works well with extensions, including setup.py-method for building >>> Cython modules. But it fails with pyximport method. >>> Is this a bug - or how to tell using the MingW compiler? >>> >> I would not say it is a bug, but rather a consequence of the >> implementation of pyximport. >> Likely a call to "dist.parse_config_files()" should be added just >> before "dist.parse_command_line()" in pyximport/pyxbuild.py >> However, if you do this, a "setup.cfg" file will be parse if present >> in $PWD... This could be seen as a gotcha, or as a feature... > > Thanks, > > in attachment my patch so far to: > > * only load the global distutils.cfg > * extend the .pyxbld scheme for enabling custom setup() args > * speed up import pyximport;pyximport.install() for use in auto > startup files (by late imports). It was quite slowing down python > startup crucially. > > Maybe its interesting for sb else too, or for general. > Didn't find into http://trac.cython.org/cython_trac/ so far. I don't use Windows, and nor do I use pyximport much, but I think these changes are valuable. Please add a ticket (mail me offlist if you don't yet have an account). - Robert From bblais at gmail.com Fri May 8 15:51:02 2009 From: bblais at gmail.com (Brian Blais) Date: Fri, 08 May 2009 09:51:02 -0400 Subject: [Cython] c-array of numpy ndarrays? Message-ID: Hello, In my question for optimization of my routines, I am trying to pre- fill a static c-array from a list of numpy ndarrays. Previously, I did something like: cdef double *w[30] cdef int r[30],c[30] and then I'd run through my list, and get the shape info and the data pointer, and fill in these arrays. This works, but then I have to do all my 2D -> 1D indexing, like: w[k][i+c[k]*j] which is ugly, and prone to typos. I'd like to do: cdef np.ndarray w[30] so I can do the nicer: w[k][i,j] I tried making an array of buffer objects, but I can only make local versions of that. c-arrays of python objects are forbidden. Is there a c-version of the ndarray? thanks, Brian Blais -- Brian Blais bblais at gmail.com http://web.bryant.edu/~bblais -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090508/2e27801a/attachment.htm From dagss at student.matnat.uio.no Fri May 8 16:27:11 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 08 May 2009 16:27:11 +0200 Subject: [Cython] c-array of numpy ndarrays? In-Reply-To: References: Message-ID: <4A04413F.6050903@student.matnat.uio.no> Brian Blais wrote: > Hello, > > In my question for optimization of my routines, I am trying to pre-fill > a static c-array from a list of numpy ndarrays. Previously, I did > something like: > > cdef double *w[30] > cdef int r[30],c[30] > > and then I'd run through my list, and get the shape info and the data > pointer, and fill in these arrays. This works, but then I have to do > all my 2D -> 1D indexing, like: > > w[k][i+c[k]*j] > > which is ugly, and prone to typos. I'd like to do: So the underlying problem here is that each array has a (widely different) size? > cdef np.ndarray w[30] No, it will be a long while before you can do this. Perhaps in a year one /might/ be able to use e.g. a C++ vector of buffers, but no promises. You can always wrap the solution above into e.g. a cdef class, so that you can write yourspecialarrayobject.get(k, i, j) -- Dag Sverre From dalcinl at gmail.com Fri May 8 17:02:19 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 8 May 2009 12:02:19 -0300 Subject: [Cython] pyximport ignors [build] compiler = mingw32 In-Reply-To: References: Message-ID: Robert, IMHO, you should load pydistutils.cfg, only remove 'setup.cfg'. pydistutils.cfg could be the only way to configure things for users without administrative rights... This should do the trick: cfgfiles = dist.find_config_files() try: cfgfiles.remove('setup.cfg') except ValueError: pass dist.parse_config_files(cfgfiles) On Fri, May 8, 2009 at 7:25 AM, Robert wrote: > Lisandro Dalcin wrote: >> >> On Thu, May 7, 2009 at 11:10 AM, Robert wrote: >>> >>> On Windows in distutils/distutils.cfg MingW is set instead of VC >>> default >>> >>> [build] >>> compiler = mingw32 >>> >>> >>> works well with extensions, including setup.py-method for building >>> Cython modules. But it fails with pyximport method. >>> Is this a bug - or how to tell using the MingW compiler? >>> >> >> I would not say it is a bug, but rather a consequence of the >> implementation of pyximport. >> >> Likely a call to ?"dist.parse_config_files()" should be added just >> before "dist.parse_command_line()" in pyximport/pyxbuild.py >> >> However, if you do this, a "setup.cfg" file will be parse if present >> in $PWD... This could be seen as a gotcha, or as a feature... >> > > Thanks, > > in attachment my patch so far to: > > * only load the global distutils.cfg > * extend the .pyxbld scheme for enabling custom setup() args > * speed up import pyximport;pyximport.install() for use in auto startup > files (by late imports). It was quite slowing down python startup crucially. > > Maybe its interesting for sb else too, or for general. > Didn't find into http://trac.cython.org/cython_trac/ so far. > > Robert > > Only in .: __init__.pyc > diff -u ../pyximport.orig\pyxbuild.py .\pyxbuild.py > --- ../pyximport.orig\pyxbuild.py ? ? ? Fri Apr 03 01:52:00 2009 > +++ .\pyxbuild.py ? ? ? Fri May 08 11:39:52 2009 > @@ -6,7 +6,6 @@ > ?import os > ?import sys > > -import distutils > ?from distutils.dist import Distribution > ?from distutils.errors import DistutilsArgError, DistutilsError, > CCompilerError > ?from distutils.extension import Extension > @@ -16,11 +15,10 @@ > ? ? HAS_CYTHON = True > ?except ImportError: > ? ? HAS_CYTHON = False > -import shutil > > ?DEBUG = 0 > ?def pyx_to_dll(filename, ext = None, force_rebuild = 0, > - ? ? ? ? ? ? ? build_in_temp=False, pyxbuild_dir=None): > + ? ? ? ? ? ? ? build_in_temp=False, pyxbuild_dir=None, setup_args={}): > ? ? """Compile a PYX file to a DLL and return the name of the generated .so > ? ? ? ?or .dll .""" > ? ? assert os.path.exists(filename), "Could not find %s" % > os.path.abspath(filename) > @@ -46,7 +44,11 @@ > ? ? ? ? args.append("--force") > ? ? if HAS_CYTHON and build_in_temp: > ? ? ? ? args.append("--pyrex-c-in-temp") > - ? ?dist = Distribution({"script_name": None, "script_args": args}) > + ? ?sargs=setup_args.copy() > + ? ?sargs.update( > + ? ? ? ?{"script_name": None, > + ? ? ? ? "script_args": args + setup_args.get("script_args",[])} ) > + ? ?dist = Distribution(sargs) > ? ? if not dist.ext_modules: > ? ? ? ? dist.ext_modules = [] > ? ? dist.ext_modules.append(ext) > @@ -55,6 +57,13 @@ > ? ? build = dist.get_command_obj('build') > ? ? build.build_base = pyxbuild_dir > > + ? ?# use global distutils.cfg only > + ? ?cfgfiles = [] > + ? ?sys_dir = os.path.dirname(sys.modules['distutils'].__file__) > + ? ?sys_file = os.path.join(sys_dir, "distutils.cfg") > + ? ?if os.path.isfile(sys_file): > + ? ? ? ?cfgfiles.append(sys_file) > + ? ?dist.parse_config_files(cfgfiles) ? # distutils/distutils.cfg; not > ./pydistutils.cfg; ./setup.cfg; > ? ? try: > ? ? ? ? ok = dist.parse_command_line() > ? ? except DistutilsArgError: > Only in .: pyxbuild.py.opt > Only in .: pyxbuild.pyc > Only in .: pyxi.patch > diff -u ../pyximport.orig\pyximport.py .\pyximport.py > --- ../pyximport.orig\pyximport.py ? ? ?Fri Apr 03 01:52:00 2009 > +++ .\pyximport.py ? ? ?Fri May 08 12:03:42 2009 > @@ -14,6 +14,22 @@ > ?sitecustomize.py with only those two lines at > ?/usr/local/lib/python2.3/site-packages/sitecustomize.py . > > +A custom distutils.core.Extension instance and setup() args > +(Distribution) for for the build can be defined by a .pyxbld > +file like: > + > +# examplemod.pyxbdl > +def make_ext(modname, pyxfilename): > + ? ?from distutils.extension import Extension > + ? ?return Extension(name = modname, > + ? ? ? ? ? ? ? ? ? ? sources=[pyxfilename, 'hello.c'], > + ? ? ? ? ? ? ? ? ? ? include_dirs=['/myinclude'] ) > +def make_setup_args(): > + ? ?return dict(script_args=["--compiler=mingw32"]) > + > +Extra dependencies can be defined by a .pyxdep . > +See README. > + > ?Since Cython 0.11, the :mod:`pyximport` module also has experimental > ?compilation support for normal Python modules. ?This allows you to > ?automatically run Cython on every .pyx and .py module that Python > @@ -34,14 +50,9 @@ > ?import os > ?import glob > ?import imp > -import pyxbuild > -from distutils.dep_util import newer > -from distutils.extension import Extension > - > -try: > - ? ?import hashlib > -except ImportError: > - ? ?import md5 as hashlib > +##import pyxbuild ?# changed: late import for speed > +##from distutils.dep_util import newer > +##from distutils.extension import Extension > > ?mod_name = "pyximport" > > @@ -64,30 +75,43 @@ > ? ? "Load a pyrex file given a name and filename." > > ?def get_distutils_extension(modname, pyxfilename): > - ? ?extra = "_" + hashlib.md5(open(pyxfilename).read()).hexdigest() > +# ? ?try: > +# ? ? ? ?import hashlib > +# ? ?except ImportError: > +# ? ? ? ?import md5 as hashlib > +# ? ?extra = "_" + hashlib.md5(open(pyxfilename).read()).hexdigest() > ?# ? ?modname = modname + extra > - ? ?extension_mod = handle_special_build(modname, pyxfilename) > + ? ?extension_mod,setup_args = handle_special_build(modname, pyxfilename) > ? ? if not extension_mod: > + ? ? ? ?from distutils.extension import Extension > ? ? ? ? extension_mod = Extension(name = modname, sources=[pyxfilename]) > - ? ?return extension_mod > + ? ?return extension_mod,setup_args > > ?def handle_special_build(modname, pyxfilename): > ? ? special_build = os.path.splitext(pyxfilename)[0] + PYXBLD_EXT > - > - ? ?if not os.path.exists(special_build): > - ? ? ? ?ext = None > - ? ?else: > - ? ? ? ?globls = {} > - ? ? ? ?locs = {} > + ? ?ext = None > + ? ?setup_args={} > + ? ?if os.path.exists(special_build): > + ? ? ? ?# globls = {} > + ? ? ? ?# locs = {} > ? ? ? ? # execfile(special_build, globls, locs) > ? ? ? ? # ext = locs["make_ext"](modname, pyxfilename) > ? ? ? ? mod = imp.load_source("XXXX", special_build, open(special_build)) > - ? ? ? ?ext = mod.make_ext(modname, pyxfilename) > - ? ? ? ?assert ext and ext.sources, ("make_ext in %s did not return > Extension" > - ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? % special_build) > + ? ? ? ?make_ext = getattr(mod,'make_ext') > + ? ? ? ?if make_ext: > + ? ? ? ? ? ?ext = make_ext(modname, pyxfilename) > + ? ? ? ? ? ?assert ext and ext.sources, ("make_ext in %s did not return > Extension" > + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? % special_build) > + ? ? ? ?make_setup_args = getattr(mod,'make_setup_args') > + ? ? ? ?if make_setup_args: > + ? ? ? ? ? ?setup_args = make_setup_args() > + ? ? ? ? ? ?assert isinstance(setup_args,dict), ("make_setup_args in %s did > not return a dict" > + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? % special_build) > + ? ? ? ?assert set or setup_args, ("neither make_ext nor make_setup_args > %s" > + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? % special_build) > ? ? ? ? ext.sources = [os.path.join(os.path.dirname(special_build), source) > ? ? ? ? ? ? ? ? ? ? ? ?for source in ext.sources] > - ? ?return ext > + ? ?return ext, setup_args > > ?def handle_dependencies(pyxfilename): > ? ? dependfile = os.path.splitext(pyxfilename)[0] + PYXDEP_EXT > @@ -116,6 +140,7 @@ > ? ? ? ? # the pyx file, 'touch' the pyx file so that distutils will > ? ? ? ? # be tricked into rebuilding it. > ? ? ? ? for file in files: > + ? ? ? ? ? ?from distutils.dep_util import newer > ? ? ? ? ? ? if newer(file, pyxfilename): > ? ? ? ? ? ? ? ? print "Rebuilding because of ", file > ? ? ? ? ? ? ? ? filetime = os.path.getmtime(file) > @@ -127,11 +152,13 @@ > ? ? ? ? "Path does not exist: %s" % pyxfilename) > ? ? handle_dependencies(pyxfilename) > > - ? ?extension_mod = get_distutils_extension(name, pyxfilename) > + ? ?extension_mod,setup_args = get_distutils_extension(name, pyxfilename) > > + ? ?import pyxbuild > ? ? so_path = pyxbuild.pyx_to_dll(pyxfilename, extension_mod, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? build_in_temp=True, > - ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?pyxbuild_dir=pyxbuild_dir) > + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?pyxbuild_dir=pyxbuild_dir, > + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?setup_args=setup_args ) > ? ? assert os.path.exists(so_path), "Cannot find: %s" % so_path > > ? ? junkpath = os.path.join(os.path.dirname(so_path), name+"_*") > @@ -171,10 +198,10 @@ > ? ? ? ? ? ? print "SEARCHING", fullname, package_path > ? ? ? ? if '.' in fullname: > ? ? ? ? ? ? mod_parts = fullname.split('.') > - ? ? ? ? ? ?package = '.'.join(mod_parts[:-1]) > + ? ? ? ? ? ?#package = '.'.join(mod_parts[:-1]) > ? ? ? ? ? ? module_name = mod_parts[-1] > ? ? ? ? else: > - ? ? ? ? ? ?package = None > + ? ? ? ? ? ?#package = None > ? ? ? ? ? ? module_name = fullname > ? ? ? ? pyx_module_name = module_name + self.extension > ? ? ? ? # this may work, but it returns the file content, not its path > Only in .: pyximport.py.opt > Only in .: pyximport.pyc > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From bblais at gmail.com Fri May 8 17:26:31 2009 From: bblais at gmail.com (Brian Blais) Date: Fri, 08 May 2009 11:26:31 -0400 Subject: [Cython] c-array of numpy ndarrays? In-Reply-To: <4A04413F.6050903@student.matnat.uio.no> References: <4A04413F.6050903@student.matnat.uio.no> Message-ID: On May 8, 2009, at 10:27 , Dag Sverre Seljebotn wrote: > Brian Blais wrote: >> Hello, >> >> In my question for optimization of my routines, I am trying to pre- >> fill >> a static c-array from a list of numpy ndarrays. Previously, I did >> something like: >> >> cdef double *w[30] >> cdef int r[30],c[30] >> >> and then I'd run through my list, and get the shape info and the data >> pointer, and fill in these arrays. This works, but then I have to do >> all my 2D -> 1D indexing, like: >> >> w[k][i+c[k]*j] >> >> which is ugly, and prone to typos. I'd like to do: > > So the underlying problem here is that each array has a (widely > different) size? the underlying problem is that the difference between the following two pieces is about a factor of 20: # using a list of objects, getting an array (called weights) from each one # and doing some calculations cdef np.ndarray[DTYPE_t,ndim=2] weights for c in self.connections_to: num_incell=c.incell.N num_outcell=c.outcell.N weights=c.weights for __i in range(num_outcell): for __j in range(num_incell): a+=weights[__i,__j] and # using an array of structures, prefilled from a list of objects, # getting an pointer to the data (called weights) from each one # and doing some calculations cdef double *weights for cg from 0<=cg > You can always wrap the solution above into e.g. a cdef class, so that > you can write > > yourspecialarrayobject.get(k, i, j) that's a nice idea... thanks. bb -- Brian Blais bblais at gmail.com http://web.bryant.edu/~bblais -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090508/0077772d/attachment.htm From dalcinl at gmail.com Fri May 8 21:20:22 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 8 May 2009 16:20:22 -0300 Subject: [Cython] class versus struct...optimization In-Reply-To: References: Message-ID: On Fri, May 8, 2009 at 7:08 AM, Brian Blais wrote: > On May 7, 2009, at 20:42 , Lisandro Dalcin wrote: > > Brian, in the Cython version, class Neuron, do this: instead of "def > update(...)", please write "cpdef update(...)" and time your code > again. If this make it faster (it should be near the C-struct > version), please consider first removing your original "def update", > and replace your "cdef _update()" by "cpdef update()". > > thanks, but this didn't change the time much at all (something like .5 ms > out of 20ms). ?I was following the example in the cython docs Early Binding > for Speed section, but now I see it got updated to cpdef! > I'll keep hacking away at it. ?Perhaps I'll try a smaller example, to see > the difference between lookups in classes and in structs, if that really is > the issue. I'm almos sure that's the issue :-) Looking closer to your code, it seems the big offender is your Connection class, as it attributes are bare "object" instances... > Is there a better way to find speed bottlenecks than the "comment out and > re-compile" method? ?:) In general, any time to cdef'ine something as "object", you are going to pay the cost of Python attribute lookup via a dict... Try do get rid of these definitions in favor of explicit types... In short, write Cython code as if you were writing C++, where you have to explicitly type everything. > thanks, > > bb > > -- > Brian Blais > bblais at gmail.com > http://web.bryant.edu/~bblais > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Fri May 8 21:27:45 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 8 May 2009 12:27:45 -0700 Subject: [Cython] class versus struct...optimization In-Reply-To: References: Message-ID: <7E1F3DFD-4BDC-46D8-8BE3-A922602A3850@math.washington.edu> On May 8, 2009, at 12:20 PM, Lisandro Dalcin wrote: > On Fri, May 8, 2009 at 7:08 AM, Brian Blais wrote: >> On May 7, 2009, at 20:42 , Lisandro Dalcin wrote: >> >> Brian, in the Cython version, class Neuron, do this: instead of "def >> update(...)", please write "cpdef update(...)" and time your code >> again. If this make it faster (it should be near the C-struct >> version), please consider first removing your original "def update", >> and replace your "cdef _update()" by "cpdef update()". >> >> thanks, but this didn't change the time much at all (something >> like .5 ms >> out of 20ms). I was following the example in the cython docs >> Early Binding >> for Speed section, but now I see it got updated to cpdef! > > >> I'll keep hacking away at it. Perhaps I'll try a smaller example, >> to see >> the difference between lookups in classes and in structs, if that >> really is >> the issue. > > I'm almos sure that's the issue :-) > > Looking closer to your code, it seems the big offender is your > Connection class, as it attributes are bare "object" instances... > >> Is there a better way to find speed bottlenecks than the "comment >> out and >> re-compile" method? :) > > In general, any time to cdef'ine something as "object", you are going > to pay the cost of Python attribute lookup via a dict... Try do get > rid of these definitions in favor of explicit types... In short, write > Cython code as if you were writing C++, where you have to explicitly > type everything. Well, the good thing is that one you only have to type the critical parts of the code, and can use the convienience of Python in not caring for the (usually) bulk of the code linewise that only takes 1% of the runtime. - Robert From dalcinl at gmail.com Fri May 8 21:40:50 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 8 May 2009 16:40:50 -0300 Subject: [Cython] Calling conventions on Windows Message-ID: I'm having a hard time trying to make mpi4py compatible with Microsoft MPI. All the MPI calls are annotated with __stdcall, and callbacks functions (in my case written in Cython) should also be. But other MPI implementations (MPICH2 and DeinoMPI) does different, they require __cdecl. I cannot figure out how to solve that without the help of Cython.... I'm thinking about ADDITIONALLY supporting the following sintax: cde int "CALL_SPEC" somefunc(): then I would be able to pass -DCALL_SPEC=__xxx or even conditionally "#define CALL_SPEC __xxx" in some header... This could have other uses as well, as the string literal could provide other annotations (like __attribute__ stuff in GCC). I've already modified the parser and all seems to work. The only thing left is to make Cython not so strict about the signatures I mean, if I use a bare string literal for annotating the function, then the function types int "ABC" foo() int "XYZ" bar() would be treated as equivalent to the unannotated: inf fun() What do you think? BTW, we should add support for __fastcall calling conventions, like this int __fastcall func() - -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From sccolbert at gmail.com Fri May 8 22:07:41 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Fri, 8 May 2009 16:07:41 -0400 Subject: [Cython] newbie question concerning scope Message-ID: <7f014ea60905081307w1676bb38u5105c274b691a469@mail.gmail.com> Hi all, Just recently got into working with cython for speeding up some calculation with numpy. this question, however, has nothing to do with numpy. The following code is valid python: var = 0 def count(n): for i in range(n): var += i return var Translating to cython results in run-time errors: cdef double var = 0.0 def count(int n): cdef int i for i in range(n): var += i return var However this works (as it should, it's the better way to do it): def count(int n): cdef int i cdef double var = 0.0 for i in range(n): var += i return var now, i'm not debating that anyone would ever want to use the second method, but regardless, in that version, why isn't var found in the module scope? The runtime error i get is 'can't assign nonetype to int' Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090508/40486362/attachment.htm From dagss at student.matnat.uio.no Fri May 8 22:28:55 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 08 May 2009 22:28:55 +0200 Subject: [Cython] Calling conventions on Windows In-Reply-To: References: Message-ID: <4A049607.5040805@student.matnat.uio.no> Lisandro Dalcin wrote: > I'm having a hard time trying to make mpi4py compatible with Microsoft > MPI. All the MPI calls are annotated with __stdcall, and callbacks > functions (in my case written in Cython) should also be. But other MPI > implementations (MPICH2 and DeinoMPI) does different, they require > __cdecl. I cannot figure out how to solve that without the help of > Cython.... > > I'm thinking about ADDITIONALLY supporting the following sintax: > > cde int "CALL_SPEC" somefunc(): > > then I would be able to pass -DCALL_SPEC=__xxx or even conditionally > "#define CALL_SPEC __xxx" in some header... > > This could have other uses as well, as the string literal could > provide other annotations (like __attribute__ stuff in GCC). > > I've already modified the parser and all seems to work. The only thing > left is to make Cython not so strict about the signatures Could we have a decorator instead, perhaps? Or some way of declaring a new callspec? ccallspec __stdcall ... I'm just thinking about the cdef extern unsigned int public inline __stdcall foo "realfoo" () expect+: ... > > I mean, if I use a bare string literal for annotating the function, > then the function types > > int "ABC" foo() > > int "XYZ" bar() > > would be treated as equivalent to the unannotated: > > inf fun() > > What do you think? > > BTW, we should add support for __fastcall calling conventions, like this > > int __fastcall func() > > - > -- Dag Sverre From dagss at student.matnat.uio.no Fri May 8 22:35:33 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 08 May 2009 22:35:33 +0200 Subject: [Cython] Calling conventions on Windows In-Reply-To: <4A049607.5040805@student.matnat.uio.no> References: <4A049607.5040805@student.matnat.uio.no> Message-ID: <4A049795.2090506@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: >> I'm having a hard time trying to make mpi4py compatible with Microsoft >> MPI. All the MPI calls are annotated with __stdcall, and callbacks >> functions (in my case written in Cython) should also be. But other MPI >> implementations (MPICH2 and DeinoMPI) does different, they require >> __cdecl. I cannot figure out how to solve that without the help of >> Cython.... >> >> I'm thinking about ADDITIONALLY supporting the following sintax: >> >> cde int "CALL_SPEC" somefunc(): >> >> then I would be able to pass -DCALL_SPEC=__xxx or even conditionally >> "#define CALL_SPEC __xxx" in some header... >> >> This could have other uses as well, as the string literal could >> provide other annotations (like __attribute__ stuff in GCC). >> >> I've already modified the parser and all seems to work. The only thing >> left is to make Cython not so strict about the signatures > > Could we have a decorator instead, perhaps? Or some way of declaring a > new callspec? > > ccallspec __stdcall > > ... > > I'm just thinking about the > > cdef extern unsigned int public inline __stdcall foo "realfoo" () > expect+: ... Sorry, accidentally hit send. The order is wrong but you get my point (and I forgot to add "api"). >> I mean, if I use a bare string literal for annotating the function, >> then the function types >> >> int "ABC" foo() Well consider cdef extern: int "__fastcall" foo "realfoo" () It's not a very readable syntax (did the real name go before or after? With this change, the compiler won't complain if you try the wrong one first) If one instead had a flexible mechanism for declaring "__fastcall" and friends so that the quotes can disappear, it looks much more readable: cdef extern: int __fastcall foo "realfoo" () -- Dag Sverre From stefan_ml at behnel.de Sat May 9 05:50:29 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 09 May 2009 05:50:29 +0200 Subject: [Cython] newbie question concerning scope In-Reply-To: <7f014ea60905081307w1676bb38u5105c274b691a469@mail.gmail.com> References: <7f014ea60905081307w1676bb38u5105c274b691a469@mail.gmail.com> Message-ID: <4A04FD85.3050908@behnel.de> Chris Colbert wrote: > Just recently got into working with cython for speeding up some calculation > with numpy. > > this question, however, has nothing to do with numpy. > > The following code is valid python: > > var = 0 > > def count(n): > for i in range(n): > var += i > > return var Valid only in the sense that it passes the parser. >>> count(5) Traceback (most recent call last): File "", line 1, in File "", line 3, in count UnboundLocalError: local variable 'var' referenced before assignment > Translating to cython results in run-time errors: > > cdef double var = 0.0 > > def count(int n): > cdef int i > for i in range(n): > var += i > > return var Just like in Python. Cython currently isn't smart enough to figure out at compile time that the local variable var is never initialised in your example. It therefore assigns None to it on initialisation to make sure things don't break at runtime. As in Python, you need to use the "global" statement to make the above work. > The runtime error i get is 'can't assign nonetype to int' Didn't try, but I doubt that that's the exact error message. Stefan From dsurviver at gmail.com Sat May 9 09:43:10 2009 From: dsurviver at gmail.com (Danilo Freitas) Date: Sat, 9 May 2009 04:43:10 -0300 Subject: [Cython] newbie question concerning scope In-Reply-To: <7f014ea60905081307w1676bb38u5105c274b691a469@mail.gmail.com> References: <7f014ea60905081307w1676bb38u5105c274b691a469@mail.gmail.com> Message-ID: <45239150905090043o1326a8b1oa1a4195a64cabe9f@mail.gmail.com> As Stefan said... try: > cdef double var = 0.0 > > def count(int n): > ??? cdef int i global var > ??? for i in range(n): > ??????? var += i > > ??? return var I don't know if this is valid in Cython... but you may try it. I don't how Cython parser works with the "global" keyworld :/ If it isn't valid, somebody may try to fix it (if possible :) ). I don't have enought knowledge about it. The only way to discover it is trying. When you do it, please post here. (I'm speaking only to myself... I don't know if this is a already known problem, and if there is a solution). From kxroberto at googlemail.com Sat May 9 17:19:32 2009 From: kxroberto at googlemail.com (Robert) Date: Sat, 09 May 2009 17:19:32 +0200 Subject: [Cython] pyximport ignors [build] compiler = mingw32 In-Reply-To: References: Message-ID: Lisandro Dalcin wrote: > Robert, IMHO, you should load pydistutils.cfg, only remove > 'setup.cfg'. pydistutils.cfg could be the only way to configure things > for users without administrative rights... > > This should do the trick: > > cfgfiles = dist.find_config_files() > try: cfgfiles.remove('setup.cfg') > except ValueError: pass > dist.parse_config_files(cfgfiles) > > yes, just ./setup.cfg seems to be questionable this and more related cleanup, speedup of importer, --inplace --verbose reload() .. here now: http://trac.cython.org/cython_trac/ticket/312 From sccolbert at gmail.com Sun May 10 00:19:34 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Sat, 9 May 2009 18:19:34 -0400 Subject: [Cython] newbie question concerning scope In-Reply-To: <45239150905090043o1326a8b1oa1a4195a64cabe9f@mail.gmail.com> References: <7f014ea60905081307w1676bb38u5105c274b691a469@mail.gmail.com> <45239150905090043o1326a8b1oa1a4195a64cabe9f@mail.gmail.com> Message-ID: <7f014ea60905091519o5d763a63gfadd3b1a6c8f137@mail.gmail.com> I tested that and it works, thanks. Chris On Sat, May 9, 2009 at 3:43 AM, Danilo Freitas wrote: > As Stefan said... > try: > > > cdef double var = 0.0 > > > > def count(int n): > > cdef int i > global var > > for i in range(n): > > var += i > > > > return var > > I don't know if this is valid in Cython... > but you may try it. > I don't how Cython parser works with the "global" keyworld :/ > > If it isn't valid, somebody may try to fix it (if possible :) ). I > don't have enought knowledge about it. The only way to discover it is > trying. > When you do it, please post here. (I'm speaking only to myself... I > don't know if this is a already known problem, and if there is a > solution). > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090509/e0895f61/attachment.htm From dsurviver at gmail.com Sun May 10 02:05:39 2009 From: dsurviver at gmail.com (Danilo Freitas) Date: Sat, 9 May 2009 21:05:39 -0300 Subject: [Cython] newbie question concerning scope In-Reply-To: <7f014ea60905091519o5d763a63gfadd3b1a6c8f137@mail.gmail.com> References: <7f014ea60905081307w1676bb38u5105c274b691a469@mail.gmail.com> <45239150905090043o1326a8b1oa1a4195a64cabe9f@mail.gmail.com> <7f014ea60905091519o5d763a63gfadd3b1a6c8f137@mail.gmail.com> Message-ID: <45239150905091705u3f07dbf9p46df9968e5538aad@mail.gmail.com> 2009/5/9 Chris Colbert : > I tested that and it works, thanks. > > Chris > It was nothing :D Thanks you too. From greg.ewing at canterbury.ac.nz Sun May 10 03:00:41 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 10 May 2009 13:00:41 +1200 Subject: [Cython] Calling conventions on Windows In-Reply-To: <4A049795.2090506@student.matnat.uio.no> References: <4A049607.5040805@student.matnat.uio.no> <4A049795.2090506@student.matnat.uio.no> Message-ID: <4A062739.2010606@canterbury.ac.nz> Dag Sverre Seljebotn wrote: > cdef extern: > int "__fastcall" foo "realfoo" () Why aren't you getting this from a header file? -- Greg From bblais at gmail.com Mon May 11 12:01:32 2009 From: bblais at gmail.com (Brian Blais) Date: Mon, 11 May 2009 06:01:32 -0400 Subject: [Cython] #define, or equivalent? Message-ID: Hello, I have a number of arrays that need to be some maximum size, and I was wondering if there is some way to define that size at the top of the code, the equivalent of: #define MAXSIZE 10 I haven't seen any examples of this. So far I have just put the number in manually, like: cdef a[10],b[10],c[10] but I'd rather have: cdef a[MAXSIZE],b[MAXSIZE],c[MAXSIZE] is there a way of doing that? thanks, Brian Blais -- Brian Blais bblais at gmail.com http://web.bryant.edu/~bblais -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090511/3a02dd3f/attachment.htm From dalcinl at gmail.com Mon May 11 15:33:15 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 11 May 2009 10:33:15 -0300 Subject: [Cython] Calling conventions on Windows In-Reply-To: <4A062739.2010606@canterbury.ac.nz> References: <4A049607.5040805@student.matnat.uio.no> <4A049795.2090506@student.matnat.uio.no> <4A062739.2010606@canterbury.ac.nz> Message-ID: As Greg pointed out, this has nothing to do with external declarations. The actual issue is when you need to implement in Cython code a callback function. In my particular issue, these callbacks have to respect a standardized API. And the problem is that different implementations of that standardized API use different calling conventions... Then I want my Cython code to work with any of these implementation, but I only see two solutions: 1) Implement all my callbacks in C and include them with 'cdef extern'. But then I've just lost the power of Cython for implementing C functions, particularly when Python objects are involved and the GIL has to be properly managed. 2) Modify Cython to accept any string literal instead of just __stdcall/__cdecl/__fastcall. Then the contents of that literal could be a macro and I would be able to control its actual contents via the C preprocessor. Can any of you figure out an alternative to (2) to save me (or any other with similar issues) from (1) ?? -- On Sat, May 9, 2009 at 10:00 PM, Greg Ewing wrote: > Dag Sverre Seljebotn wrote: > >> cdef extern: >> ? ? ?int "__fastcall" foo "realfoo" () > > Why aren't you getting this from a header file? > > -- > Greg > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Mon May 11 16:45:09 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 11 May 2009 16:45:09 +0200 (CEST) Subject: [Cython] Calling conventions on Windows In-Reply-To: References: <4A049607.5040805@student.matnat.uio.no> <4A049795.2090506@student.matnat.uio.no> <4A062739.2010606@canterbury.ac.nz> Message-ID: <0ade16b730dabdc7433ffa8453e8f0ef.squirrel@webmail.uio.no> Lisandro wrote: > As Greg pointed out, this has nothing to do with external > declarations. The actual issue is when you need to implement in Cython > code a callback function. Sure, you will typically not need cdef extern "declspec" func "realname" () because you can then use a header file. However, it is about creating a syntax in Cython which is easy to learn and remember! And that means making each feature working in any situation where one would "guess" it would work and not put in arbitrary limits (I should know, the number of complaints I get on the buffer-in-function-only restriction :-) ). And making lots of special cases (this syntax is not allowed in this case and this syntax is not allowed in that case) complicates the language a lot. > In my particular issue, these callbacks have to respect a standardized > API. And the problem is that different implementations of that > standardized API use different calling conventions... Then I want my > Cython code to work with any of these implementation, but I only see > two solutions: > > 1) Implement all my callbacks in C and include them with 'cdef > extern'. But then I've just lost the power of Cython for implementing > C functions, particularly when Python objects are involved and the GIL > has to be properly managed. > > 2) Modify Cython to accept any string literal instead of just > __stdcall/__cdecl/__fastcall. Then the contents of that literal could > be a macro and I would be able to control its actual contents via the > C preprocessor. > > Can any of you figure out an alternative to (2) to save me (or any > other with similar issues) from (1) ?? Yes, as I proposed, some kind of syntax to declare your own calling convention; e.g. cdef callspec __mycall "__mycall" cdef __mycall int foo(): return 3 This will avoid cdef extern: int "realfunc" func() from silently compiling but producing unexpected code, without having to arbitrarily disable parts of the language in a perfectly valid corner case. Dag Sverre From dagss at student.matnat.uio.no Mon May 11 16:52:33 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 11 May 2009 16:52:33 +0200 (CEST) Subject: [Cython] Calling conventions on Windows In-Reply-To: <0ade16b730dabdc7433ffa8453e8f0ef.squirrel@webmail.uio.no> References: <4A049607.5040805@student.matnat.uio.no> <4A049795.2090506@student.matnat.uio.no> <4A062739.2010606@canterbury.ac.nz> <0ade16b730dabdc7433ffa8453e8f0ef.squirrel@webmail.uio.no> Message-ID: <7945881b0549cbd48f825565d33060bb.squirrel@webmail.uio.no> Dag wrote: > Lisandro wrote: >> API. And the problem is that different implementations of that >> standardized API use different calling conventions... Then I want my >> Cython code to work with any of these implementation, but I only see >> two solutions: >> >> 1) Implement all my callbacks in C and include them with 'cdef >> extern'. But then I've just lost the power of Cython for implementing >> C functions, particularly when Python objects are involved and the GIL >> has to be properly managed. >> >> 2) Modify Cython to accept any string literal instead of just >> __stdcall/__cdecl/__fastcall. Then the contents of that literal could >> be a macro and I would be able to control its actual contents via the >> C preprocessor. >> >> Can any of you figure out an alternative to (2) to save me (or any >> other with similar issues) from (1) ?? > > Yes, as I proposed, some kind of syntax to declare your own calling > convention; e.g. > > cdef callspec __mycall "__mycall" > > cdef __mycall int foo(): return 3 Alternatively: from cython import callconvention @callconvention("__mycall") cdef int foo(): ... though it might be too long? (Shorter name possible on decorator too.) Similarily one could have @gccattribute("...") which I'd prefer to hacking this into a callconvention mechanism. Dag Sverre From dalcinl at gmail.com Mon May 11 17:54:59 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 11 May 2009 12:54:59 -0300 Subject: [Cython] Calling conventions on Windows In-Reply-To: <7945881b0549cbd48f825565d33060bb.squirrel@webmail.uio.no> References: <4A049607.5040805@student.matnat.uio.no> <4A049795.2090506@student.matnat.uio.no> <4A062739.2010606@canterbury.ac.nz> <0ade16b730dabdc7433ffa8453e8f0ef.squirrel@webmail.uio.no> <7945881b0549cbd48f825565d33060bb.squirrel@webmail.uio.no> Message-ID: On Mon, May 11, 2009 at 11:52 AM, Dag Sverre Seljebotn wrote: > Dag wrote: >> >> Yes, as I proposed, some kind of syntax to declare your own calling >> convention; e.g. >> >> cdef callspec __mycall "__mycall" >> >> cdef __mycall int foo(): return 3 > I really like this, but I also understand that it is going to be a burden to implement. > Alternatively: > > from cython import callconvention > > @callconvention("__mycall") > cdef int foo(): ... > > though it might be too long? (Shorter name possible on decorator too.) @callspec("__mycall") seems short and explicit enough for me. > Similarily one could have > > @gccattribute("...") > > which I'd prefer to hacking this into a callconvention mechanism. > > Dag Sverre OK, even better, though I would call it "@attribute(...)" or something like that, as the "gcc" bit seems not appropriate (because the the syntax could target special features of other compilers, not just GCC) -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Mon May 11 20:18:24 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 11 May 2009 20:18:24 +0200 Subject: [Cython] Calling conventions on Windows In-Reply-To: References: <4A049607.5040805@student.matnat.uio.no> <4A049795.2090506@student.matnat.uio.no> <4A062739.2010606@canterbury.ac.nz> <0ade16b730dabdc7433ffa8453e8f0ef.squirrel@webmail.uio.no> <7945881b0549cbd48f825565d33060bb.squirrel@webmail.uio.no> Message-ID: <4A086BF0.5070007@student.matnat.uio.no> Lisandro Dalcin wrote: > On Mon, May 11, 2009 at 11:52 AM, Dag Sverre Seljebotn > wrote: >> Dag wrote: >>> Yes, as I proposed, some kind of syntax to declare your own calling >>> convention; e.g. >>> >>> cdef callspec __mycall "__mycall" >>> >>> cdef __mycall int foo(): return 3 > > I really like this, but I also understand that it is going to be a > burden to implement. > >> Alternatively: >> >> from cython import callconvention >> >> @callconvention("__mycall") >> cdef int foo(): ... >> >> though it might be too long? (Shorter name possible on decorator too.) > > @callspec("__mycall") seems short and explicit enough for me. +1. Especially since no change in the parser is required :-) (Should we have a +1 from Robert or Stefan as well you think?) If you end up doing this, here's one (of many) strategies: Compiler directives (Options.py and somewhere in ParseTreeTransforms.py) must be extended to accept string arguments and possibly extended with a mechanism to make directives only valid in certain contexts (no "with callspec(...):" :-) ). (Technically this is perhaps not a compiler directive, but it's an easy way of implementing it and not *too* far off...) -- Dag Sverre From robertwb at math.washington.edu Mon May 11 20:37:06 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 11 May 2009 11:37:06 -0700 Subject: [Cython] Calling conventions on Windows In-Reply-To: <4A086BF0.5070007@student.matnat.uio.no> References: <4A049607.5040805@student.matnat.uio.no> <4A049795.2090506@student.matnat.uio.no> <4A062739.2010606@canterbury.ac.nz> <0ade16b730dabdc7433ffa8453e8f0ef.squirrel@webmail.uio.no> <7945881b0549cbd48f825565d33060bb.squirrel@webmail.uio.no> <4A086BF0.5070007@student.matnat.uio.no> Message-ID: On May 11, 2009, at 11:18 AM, Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: >> On Mon, May 11, 2009 at 11:52 AM, Dag Sverre Seljebotn >> wrote: >>> Dag wrote: >>>> Yes, as I proposed, some kind of syntax to declare your own calling >>>> convention; e.g. >>>> >>>> cdef callspec __mycall "__mycall" >>>> >>>> cdef __mycall int foo(): return 3 >> >> I really like this, but I also understand that it is going to be a >> burden to implement. >> >>> Alternatively: >>> >>> from cython import callconvention >>> >>> @callconvention("__mycall") >>> cdef int foo(): ... >>> >>> though it might be too long? (Shorter name possible on decorator >>> too.) >> >> @callspec("__mycall") seems short and explicit enough for me. > > +1. Especially since no change in the parser is required :-) I like @callspec("__mycall"") too, and the preferred way to do things (rather than the ones we support so far). An @attribute decorators would be useful too, but orthogonal (I'd rather have it just be for __attribute__.). - Robert From dagss at student.matnat.uio.no Mon May 11 20:49:52 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 11 May 2009 20:49:52 +0200 Subject: [Cython] Calling conventions on Windows In-Reply-To: References: <4A049607.5040805@student.matnat.uio.no> <4A049795.2090506@student.matnat.uio.no> <4A062739.2010606@canterbury.ac.nz> <0ade16b730dabdc7433ffa8453e8f0ef.squirrel@webmail.uio.no> <7945881b0549cbd48f825565d33060bb.squirrel@webmail.uio.no> Message-ID: <4A087350.1000907@student.matnat.uio.no> Lisandro Dalcin wrote: > On Mon, May 11, 2009 at 11:52 AM, Dag Sverre Seljebotn >> Similarily one could have >> >> @gccattribute("...") >> >> which I'd prefer to hacking this into a callconvention mechanism. >> >> Dag Sverre > > OK, even better, though I would call it "@attribute(...)" or something > like that, as the "gcc" bit seems not appropriate (because the the > syntax could target special features of other compilers, not just GCC) > The reason I proposed to call it "gccattribute" is that AFAIK GCC is the compiler which introduces these and that is how they are referred to (although I think icc supports a lot of them as well). I think "attribute" is far to generic; "cattribute" is slightly better but still these are not standard C. (In most cases we are better off implementing higher-level support anyway, like I did with packed structs, where differenct extensions from more than one compiler is used through #ifdefs. I'm +0 on adding gccattribute, I just responded to Lisandro's ideas on it.) -- Dag Sverre From robertwb at math.washington.edu Mon May 11 21:00:58 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 11 May 2009 12:00:58 -0700 Subject: [Cython] Calling conventions on Windows In-Reply-To: <4A087350.1000907@student.matnat.uio.no> References: <4A049607.5040805@student.matnat.uio.no> <4A049795.2090506@student.matnat.uio.no> <4A062739.2010606@canterbury.ac.nz> <0ade16b730dabdc7433ffa8453e8f0ef.squirrel@webmail.uio.no> <7945881b0549cbd48f825565d33060bb.squirrel@webmail.uio.no> <4A087350.1000907@student.matnat.uio.no> Message-ID: <6AFC4F3E-C3B3-45C0-A1F8-517C74672CB8@math.washington.edu> On May 11, 2009, at 11:49 AM, Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: >> On Mon, May 11, 2009 at 11:52 AM, Dag Sverre Seljebotn >>> Similarily one could have >>> >>> @gccattribute("...") >>> >>> which I'd prefer to hacking this into a callconvention mechanism. >>> >>> Dag Sverre >> >> OK, even better, though I would call it "@attribute(...)" or >> something >> like that, as the "gcc" bit seems not appropriate (because the the >> syntax could target special features of other compilers, not just >> GCC) >> > > The reason I proposed to call it "gccattribute" is that AFAIK GCC > is the > compiler which introduces these and that is how they are referred to > (although I think icc supports a lot of them as well). > > I think "attribute" is far to generic; "cattribute" is slightly better > but still these are not standard C. Yeah, gccattribute or cattribute are more clear. > (In most cases we are better off implementing higher-level support > anyway, like I did with packed structs, where differenct extensions > from > more than one compiler is used through #ifdefs. I'm +0 on adding > gccattribute, I just responded to Lisandro's ideas on it.) This opens the option for every obscure attribute one might want to use, whether/before we support it at a higher level. I don't think it's high priority though. - Robert From lrhazi at gmail.com Tue May 12 03:51:13 2009 From: lrhazi at gmail.com (Mohamed Lrhazi) Date: Mon, 11 May 2009 21:51:13 -0400 Subject: [Cython] How do I pass opaque pointers back and forth? Message-ID: Hello, Is it possible in Cython to pass a void pointer to a python function, then have the latter send it back as a parameter to a function defined in Cython, then be able to reuse it? Basically, I am using a C lib that sends back these pointers, pointing to structures whose definitions in the header file are undefined/incomplete. I would like to send them as is to Python, but then be able to recover them back and resend them to lib. I can keep them in Cython as global variables, but was wondering if I could transparently pass them back and forth instead. I thought maybe I could pass the actual address, as in: &pointer[0] but could not figure out how to make a new pointer in Cython assigning the recovered address to it. Thanks a lot, Mohamed. From lrhazi at gmail.com Tue May 12 04:13:18 2009 From: lrhazi at gmail.com (Mohamed Lrhazi) Date: Mon, 11 May 2009 22:13:18 -0400 Subject: [Cython] How do I pass opaque pointers back and forth? In-Reply-To: References: Message-ID: It seems I can make it work passing the address as &pointer[0], then recovering t as: cdef void *new_pointer = recovered_value Is this correct? or is it just luck that my program is not crashing yet :) Thanks, Mohamed. On Mon, May 11, 2009 at 9:51 PM, Mohamed Lrhazi wrote: > Hello, > > Is it possible in Cython to pass a void pointer to a python function, > then have the latter send it back as a parameter to a ?function > defined in Cython, then be able to reuse it? > > Basically, I am using a C lib that sends back these pointers, pointing > to structures whose definitions in the header file are > undefined/incomplete. I would like to send them as is to Python, but > then be able to recover them back and resend them to lib. > > I can keep them in Cython as global variables, but was wondering if I > could transparently pass them back and forth instead. > > I thought maybe I could pass the actual address, as in: &pointer[0] > > but could not figure out how to make a new pointer in Cython assigning > the recovered address to it. > > Thanks a lot, > Mohamed. > -- " Logic merely sanctions the conquests of the intuition." Jacques Hadamard From dalcinl at gmail.com Tue May 12 04:25:30 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 11 May 2009 23:25:30 -0300 Subject: [Cython] How do I pass opaque pointers back and forth? In-Reply-To: References: Message-ID: On Mon, May 11, 2009 at 11:13 PM, Mohamed Lrhazi wrote: > It seems I can make it work passing the address as > &pointer[0], then recovering t as: > > cdef void *new_pointer = recovered_value > > Is this correct? or is it just luck that my program is not crashing yet :) > It is correct as long as you pass back the "correct" integer value... If you ever make a mistake, you will likely have a segfault... Depending on your original C API, you could consider designing an more object oriented, pythonic API where your pointers are (cdef) attributes of a (cdef) class, and their methods provide access to the many function calls of the lib you are wrapping... > Thanks, > Mohamed. > > On Mon, May 11, 2009 at 9:51 PM, Mohamed Lrhazi wrote: >> Hello, >> >> Is it possible in Cython to pass a void pointer to a python function, >> then have the latter send it back as a parameter to a ?function >> defined in Cython, then be able to reuse it? >> >> Basically, I am using a C lib that sends back these pointers, pointing >> to structures whose definitions in the header file are >> undefined/incomplete. I would like to send them as is to Python, but >> then be able to recover them back and resend them to lib. >> >> I can keep them in Cython as global variables, but was wondering if I >> could transparently pass them back and forth instead. >> >> I thought maybe I could pass the actual address, as in: &pointer[0] >> >> but could not figure out how to make a new pointer in Cython assigning >> the recovered address to it. >> >> Thanks a lot, >> Mohamed. >> > > > > -- > " Logic merely sanctions the conquests of the intuition." > Jacques Hadamard > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Tue May 12 06:21:15 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 11 May 2009 21:21:15 -0700 Subject: [Cython] How do I pass opaque pointers back and forth? In-Reply-To: References: Message-ID: <852A1B6B-7694-42C1-A1FD-AB3D3B499B0C@math.washington.edu> On May 11, 2009, at 7:25 PM, Lisandro Dalcin wrote: > On Mon, May 11, 2009 at 11:13 PM, Mohamed Lrhazi > wrote: >> It seems I can make it work passing the address as >> &pointer[0], then recovering t as: >> >> cdef void *new_pointer = recovered_value >> >> Is this correct? or is it just luck that my program is not >> crashing yet :) >> > > It is correct as long as you pass back the "correct" integer value... > If you ever make a mistake, you will likely have a segfault... > > Depending on your original C API, you could consider designing an more > object oriented, pythonic API where your pointers are (cdef) > attributes of a (cdef) class, and their methods provide access to the > many function calls of the lib you are wrapping... See also http://docs.python.org/c-api/cobject.html which is made for this kind of thing. - Robert From dagss at student.matnat.uio.no Tue May 12 09:50:42 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 12 May 2009 09:50:42 +0200 Subject: [Cython] Notes on code streams refactoring Message-ID: <4A092A52.2010309@student.matnat.uio.no> I found myself in a corner where the utility code mechanism was not fine-grained enough. Rather than adding yet-another-special-case I decided to refactor some stuff. Reviews welcome; I can of course change it back (though hopefully not until after 0.11.2). a) In Code.py: GlobalState, there's now a list code_layout: code_layout = [ 'h_code', 'utility_code_proto', 'type_declarations', 'module_declarations', 'typeinfo', 'before_global_var', 'global_var', 'all_the_rest', 'utility_code_def' ] This identifies code stream blocks that are set up initially (in that order). b) At any point, your code instance represents the default location, but you can also output to other places: otherplace = code.globalstate['typeinfo'] otherplace.putln(...) If just the right location isn't available, just split up an existing stream in (a). c) An alternative to the classic utility code is now this: if your_id not in code.globalstate.utility_codes: code.globalstate.utility_codes.add(your_id) somestream = code.globalstate['somestream'] somestream.putln(...) I.e. the presence of "some utility code" is now fully orthogonal to which code stream/location in the output the code ends up at. (The UtilityCode class is a convenience for this with a couple of hard-coded streams.) http://hg.cython.org/cython-devel/rev/b86a7b374a95 http://hg.cython.org/cython-devel/rev/fd6cc8a5998d http://hg.cython.org/cython-devel/rev/df8dd466bc4a etc. -- Dag Sverre From Dave.Cross at cdl.co.uk Tue May 12 14:47:11 2009 From: Dave.Cross at cdl.co.uk (Dave Cross) Date: Tue, 12 May 2009 13:47:11 +0100 Subject: [Cython] Problem importing Cython built module from within embedded interpreter. Message-ID: <3C1CBB29A3D34C479B570500C7A73F2907BCD36B@cdlms2.cheshdatasys.co.uk> Hi, We have embedded python 2.6 within our C++ app and it runs quite substantial Python scripts operating on the C++ data perfectly OK. I have just compiled a small module (foo) using Cython and it works fine standalone and within the main script under normal Python. >From within the embedded interpreter, however, the main script reports ImportError: No module named foo despite me specifically appending the source folder to sys.path just prior to the import. Not sure if this is related but I need to do from foo import * in order to get the imported python class name into my local namespace. Any ideas anybody? Dave.

**********************************************************************
Please consider the environment - do you really need to print this email?

This email is intended only for the person(s) named above and may contain private and confidential information. If it has come to you in error, please destroy and permanently delete any copy in your possession and contact us on +44 (0) 161 480 4420. The information in this email is copyright © CDL Group Holdings Limited. We cannot accept any liability for any loss or damage sustained as a result of software viruses. It is your responsibility to carry out such virus checking as is necessary before opening any attachment.
Cheshire Datasystems Limited uses software which automatically screens incoming emails for inappropriate content and attachments. If the software identifies such content or attachment, the email will be forwarded to our Technology Department for checking. You should be aware that any email which you send to Cheshire Datasystems Limited is subject to this procedure.
Cheshire Datasystems Limited, Strata House, Kings Reach Road, Stockport SK4 2HD
Registered in England and Wales with Company Number 3991057
VAT registration: 727 1188 33

 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090512/c5c155ef/attachment.htm 

From dagss at student.matnat.uio.no  Tue May 12 17:58:05 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 12 May 2009 17:58:05 +0200
Subject: [Cython] asking for review of my patch for #287
In-Reply-To: 
References: 
Message-ID: <4A099C8D.2000407@student.matnat.uio.no>

Lisandro Dalcin wrote:
> I want to raise your attention to ticket #287, as my patch could be
> controversial. I even changed my mind, and fixed my previous work on
> all this to prefer 'nb_int' over 'nb_long' in __Pyx_PyNumber_Int().
> This change was not a quick hack to get things solved, but rather a
> hard to take decision after diving core CPython sources from 2.3 to
> 2.7. IMHO, the 'nb_long' slot should have been deprecated long, long
> ago in core CPython. I would like this to be discussed a bit more
> before next release.

I don't think I'm qualified to comment on this. Do you consider it safe 
enough yourself for inclusion in Cython 0.11.2, or is it blocked pending 
review by Stefan or Robert?

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Tue May 12 18:04:28 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 12 May 2009 18:04:28 +0200
Subject: [Cython] #303 + simplifying type system
Message-ID: <4A099E0C.9000909@student.matnat.uio.no>

A)

Ticket #303 manifests itself like this:

cdef extern:
     ctypedef float footype # really double!

cdef class A:
     cdef public footype myproperty

My suggestion for a fix here is to replace all T_INT, T_UINT etc. with 
simply three types: __Pyx_T_SIGNED, __Pyx_T_UNSIGNED, __Pyx_T_FLOAT. 
These would use sizeof to determine the right type to flag the extension 
type property as (like Lisandro did with T_SIZET already does).

I can then change T_SIZET to __Pyx_T_UNSIGNED, right? (Basically that 
would be a rename...) Or it T_SIZET is defined in any Python version, 
I'll just throw it in on the list of possible "outputs" from the macro 
if it is defined.

B)

Given this change (which could happen in -devel), is there any obstacles 
to getting rid of the detailed type system in PyrexTypes? Basically 
Cython would only deal with "signed", "unsigned" and "float" (and in 
time "complex").

Doing

cdef short i

would basically flag i as "signed"; the "short-ness" would only be 
present as the cname of the type.


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Tue May 12 18:07:35 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 12 May 2009 18:07:35 +0200
Subject: [Cython] #303 + simplifying type system
In-Reply-To: <4A099E0C.9000909@student.matnat.uio.no>
References: <4A099E0C.9000909@student.matnat.uio.no>
Message-ID: <4A099EC7.8040109@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> A)
> 
> Ticket #303 manifests itself like this:
> 
> cdef extern:
>      ctypedef float footype # really double!
> 
> cdef class A:
>      cdef public footype myproperty
> 
> My suggestion for a fix here is to replace all T_INT, T_UINT etc. with 
> simply three types: __Pyx_T_SIGNED, __Pyx_T_UNSIGNED, __Pyx_T_FLOAT. 
> These would use sizeof to determine the right type to flag the extension 
> type property as (like Lisandro did with T_SIZET already does).
> 
> I can then change T_SIZET to __Pyx_T_UNSIGNED, right? (Basically that 
> would be a rename...) Or it T_SIZET is defined in any Python version, 
> I'll just throw it in on the list of possible "outputs" from the macro 
> if it is defined.
> 
> B)
> 
> Given this change (which could happen in -devel), is there any obstacles 
> to getting rid of the detailed type system in PyrexTypes? Basically 
> Cython would only deal with "signed", "unsigned" and "float" (and in 
> time "complex").
> 
> Doing
> 
> cdef short i
> 
> would basically flag i as "signed"; the "short-ness" would only be 
> present as the cname of the type.
> 

Note: I see B) happening in -unstable. (A) in -devel).


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Tue May 12 18:19:17 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 12 May 2009 18:19:17 +0200
Subject: [Cython] Buffer passing implementation details
In-Reply-To: <4A033FE9.9060001@student.matnat.uio.no>
References: <4A01BB4A.1060003@student.matnat.uio.no>	
	<4A033FE9.9060001@student.matnat.uio.no>
Message-ID: <4A09A185.2020008@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Kurt Smith wrote:
>> Although, what about when we require a contiguous copy (or any mode,
>> for that matter) of a buffer?  Would we set the top-level object
>> reference to None, as suggested in a previous thread?  That would
>> prevent slicing at the Python API level, though, right?
> 
> Hmm. I'm still thinking about contiguous copies. Let's deal with that in 
> the next iteration.

OK I've been thinking about copy-making:

This really seems to be linked to in/out/inout specifiers as part of the 
function signature, more than anything else. There's basically two kind 
of functions:

a) The ones that pass in and out data. These can use in/out/inout 
specifiers and copying can happen automatically.

b) The ones that need to hold on to the buffer after returning 
("obj.set_data_store"). Copying here will cause trouble.

So I propose:

cdef void dataonly(inout int[::1] arr) # requires contiguous
cdef void keepsref(int[::1] arr)       # requires contiguous
cdef void keepsref_strided(int[:] arr)


int[:] discontiguous_buf = ...

dataonly(discontiguous_buf) # allowed, copying happen
keepsref(discontiguous_buf) # disallowed, compile-time error
keepsref_strided(discontiguous_buf) # allowed

keepsref(discontiguous_buf.copy()) # allowed

where I envision the copy method as being a way of manually creating a 
contiguous copy.

As for pure Python syntax for in/out/inout modifiers, I'm thinking 
something like this:

@cython.locals(arr=cython.inout & cython.int[:])
def foo(arr):
     ...

or in time

def foo(arr: cython.inout & cython.int[:]):
     ...

(The point here is: Relying on modifiers is not scary, we can use "&" in 
pure Python mode to get it across.)

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Tue May 12 18:21:58 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 12 May 2009 18:21:58 +0200
Subject: [Cython] Buffer passing implementation details
In-Reply-To: <4A09A185.2020008@student.matnat.uio.no>
References: <4A01BB4A.1060003@student.matnat.uio.no>		<4A033FE9.9060001@student.matnat.uio.no>
	<4A09A185.2020008@student.matnat.uio.no>
Message-ID: <4A09A226.9040105@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Dag Sverre Seljebotn wrote:
>> Kurt Smith wrote:
>>> Although, what about when we require a contiguous copy (or any mode,
>>> for that matter) of a buffer?  Would we set the top-level object
>>> reference to None, as suggested in a previous thread?  That would
>>> prevent slicing at the Python API level, though, right?
>> Hmm. I'm still thinking about contiguous copies. Let's deal with that in 
>> the next iteration.
> 
> OK I've been thinking about copy-making:
> 
> This really seems to be linked to in/out/inout specifiers as part of the 
> function signature, more than anything else. There's basically two kind 
> of functions:
> 
> a) The ones that pass in and out data. These can use in/out/inout 
> specifiers and copying can happen automatically.
> 
> b) The ones that need to hold on to the buffer after returning 
> ("obj.set_data_store"). Copying here will cause trouble.
> 
> So I propose:
> 
> cdef void dataonly(inout int[::1] arr) # requires contiguous
> cdef void keepsref(int[::1] arr)       # requires contiguous
> cdef void keepsref_strided(int[:] arr)
> 
> 
> int[:] discontiguous_buf = ...
> 
> dataonly(discontiguous_buf) # allowed, copying happen
> keepsref(discontiguous_buf) # disallowed, compile-time error

Actually, let's keep this a runtime error -- as strided buffers can be 
contiguous (it needs to be runtime anyway for passing Python objects). 
It could hint to "use the .copy() method".

-- 
Dag Sverre

From stefan_ml at behnel.de  Tue May 12 08:45:19 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 12 May 2009 08:45:19 +0200
Subject: [Cython] Cython question
In-Reply-To: <89b937320905110100h1c068eb8ke31a01674ba2f6de@mail.gmail.com>
References: <89b937320905110100h1c068eb8ke31a01674ba2f6de@mail.gmail.com>
Message-ID: <4A091AFF.8070006@behnel.de>

Hi,

I forwarded your question to the Cython mailing list, which is the right
place to ask these things. Please subscribe to it if you are interested in
further discussion.

Jan Svec wrote:
> I have one question regarding the Cython project. Can I create a generic
> dynamically linked library in Cython? In other words I want to export my
> Python functions as DLL.

I'm not sure what you mean. Python's extension modules *are* shared
libraries, so a DLL is always what you get.

If with "generic" you mean: not only callable from Python, but exporting C
functions that other programs can call, then the answer is: I never tried.
It might work to link statically against libpython to create a DLL that
does not depend on the Python library. However, there is some
initialisation code involved that is called from the Python runtime. My
guess is that you'd at least have to add some hand-written code that
imports your module into an embedded Python interpreter so that it gets
properly set up. Some example code for embedding the Python interpreter is
here:

http://www.freenet.org.nz/python/embeddingpyrex/

Recent developer versions of Cython accept an "--embed" flag that will
generate an appropriate main() function (i.e. not exactly what you need,
but the code should be similar).

So I think the answer to your question is: probably yes, but not without a
bit of work on your side.

Stefan


From stefan_ml at behnel.de  Tue May 12 18:36:06 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 12 May 2009 18:36:06 +0200
Subject: [Cython] #define, or equivalent?
In-Reply-To: 
References: 
Message-ID: <4A09A576.4040604@behnel.de>


Brian Blais wrote:
> I have a number of arrays that need to be some maximum size, and I was
> wondering if there is some way to define that size at the top of the
> code, the equivalent of:
> 
> #define MAXSIZE 10
> 
> I haven't seen any examples of this.  So far I have just put the number
> in manually, like:
> 
> cdef a[10],b[10],c[10]
> 
> but I'd rather have:
> 
> cdef a[MAXSIZE],b[MAXSIZE],c[MAXSIZE]
> 
> is there a way of doing that?

You might be happy with the "DEF" statement (capital letters).

Stefan


From dalcinl at gmail.com  Tue May 12 19:58:13 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 12 May 2009 14:58:13 -0300
Subject: [Cython] asking for review of my patch for #287
In-Reply-To: <4A099C8D.2000407@student.matnat.uio.no>
References: 
	<4A099C8D.2000407@student.matnat.uio.no>
Message-ID: 

On Tue, May 12, 2009 at 12:58 PM, Dag Sverre Seljebotn
 wrote:
> Lisandro Dalcin wrote:
>> I want to raise your attention to ticket #287, as my patch could be
>> controversial. I even changed my mind, and fixed my previous work on
>> all this to prefer 'nb_int' over 'nb_long' in __Pyx_PyNumber_Int().
>> This change was not a quick hack to get things solved, but rather a
>> hard to take decision after diving core CPython sources from 2.3 to
>> 2.7. IMHO, the 'nb_long' slot should have been deprecated long, long
>> ago in core CPython. I would like this to be discussed a bit more
>> before next release.
>
> I don't think I'm qualified to comment on this. Do you consider it safe
> enough yourself for inclusion in Cython 0.11.2, or is it blocked pending
> review by Stefan or Robert?
>

IMHO, it is ready for 0.11.2, but I would like to know other opinions,
just in case... As I said, the patch could be controversial, specially
the change for Py2 in Pyx_PyNumber_Int(), where the "nb_int" slot is
preferred over "nb_long" slot.

If Robert/Stefan does not comment on this on a couple of days (and
provide an example showing the brokenness of my approach ;-) ), I
would just push the patch.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Tue May 12 20:12:24 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 12 May 2009 11:12:24 -0700
Subject: [Cython] 0.11.2 release schedule?
In-Reply-To: <49FB54C6.4020009@student.matnat.uio.no>
References: <49FAF0CC.5070108@student.matnat.uio.no>	<49FB20B8.8090506@behnel.de>
	
	<49FB54C6.4020009@student.matnat.uio.no>
Message-ID: <1A7D9422-4220-4C7B-852B-42BE15F93F6F@math.washington.edu>

On May 1, 2009, at 1:00 PM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On May 1, 2009, at 9:18 AM, Stefan Behnel wrote:
>>
>>> Dag Sverre Seljebotn wrote:
>>>> Like I've been saying on some occasions, I'm going show off Cython
>>>> 18th
>>>> of May, and have had a tendency to scratch some itches rather than
>>>> planning how to explain them... so I'd really like a release to
>>>> happen
>>>> before then. (Though I also have some fixes/rewrites which are not
>>>> done
>>>> yet which will go in next week.)
>>>>
>>>> If it helps I can play release manager this time and do the
>>>> groundwork
>>>> for making it build Sage etc. (especially if that gets us complex
>>>> floats
>>>> in time :-)).
>>>>
>>>> Would it be possible to e.g. do a full stop feature freeze next
>>>> Friday
>>>> 8th, and then a release the week after?
>>> I'm not currently working on cython-devel anyway, so I'm glad to
>>> hear that
>>> you volunteer as QA manager. :)
>>
>> Yes, that would be great if you would do the release managing--
>> definitely would make a mid-May release more feasible.
>
> I'll give it a try.

Dag, any chance on a beta in the near future? Finishing up complex  
support is #2 on my todo list, so it'll get pushed today.

- Robert


From robertwb at math.washington.edu  Tue May 12 20:17:15 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 12 May 2009 11:17:15 -0700
Subject: [Cython] asking for review of my patch for #287
In-Reply-To: 
References: 
	<4A099C8D.2000407@student.matnat.uio.no>
	
Message-ID: <3620E7AB-10E1-4A90-A1FB-85E65C8E1758@math.washington.edu>

On May 12, 2009, at 10:58 AM, Lisandro Dalcin wrote:

> On Tue, May 12, 2009 at 12:58 PM, Dag Sverre Seljebotn
>  wrote:
>> Lisandro Dalcin wrote:
>>> I want to raise your attention to ticket #287, as my patch could be
>>> controversial. I even changed my mind, and fixed my previous work on
>>> all this to prefer 'nb_int' over 'nb_long' in __Pyx_PyNumber_Int().
>>> This change was not a quick hack to get things solved, but rather a
>>> hard to take decision after diving core CPython sources from 2.3 to
>>> 2.7. IMHO, the 'nb_long' slot should have been deprecated long, long
>>> ago in core CPython. I would like this to be discussed a bit more
>>> before next release.
>>
>> I don't think I'm qualified to comment on this. Do you consider it  
>> safe
>> enough yourself for inclusion in Cython 0.11.2, or is it blocked  
>> pending
>> review by Stefan or Robert?
>>
>
> IMHO, it is ready for 0.11.2, but I would like to know other opinions,
> just in case... As I said, the patch could be controversial, specially
> the change for Py2 in Pyx_PyNumber_Int(), where the "nb_int" slot is
> preferred over "nb_long" slot.
>
> If Robert/Stefan does not comment on this on a couple of days (and
> provide an example showing the brokenness of my approach ;-) ), I
> would just push the patch.

Sorry, most of my "dev" time has been focused on the immanent Sage  
4.0 release, which is why I haven't had time to comment on many of  
the things that have shown up here.

- Robert


From dagss at student.matnat.uio.no  Tue May 12 20:20:41 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 12 May 2009 20:20:41 +0200
Subject: [Cython] 0.11.2 release schedule?
In-Reply-To: <1A7D9422-4220-4C7B-852B-42BE15F93F6F@math.washington.edu>
References: <49FAF0CC.5070108@student.matnat.uio.no>	<49FB20B8.8090506@behnel.de>		<49FB54C6.4020009@student.matnat.uio.no>
	<1A7D9422-4220-4C7B-852B-42BE15F93F6F@math.washington.edu>
Message-ID: <4A09BDF9.8070109@student.matnat.uio.no>

Robert Bradshaw wrote:
> On May 1, 2009, at 1:00 PM, Dag Sverre Seljebotn wrote:
> 
>> Robert Bradshaw wrote:
>>> On May 1, 2009, at 9:18 AM, Stefan Behnel wrote:
>>>
>>>> Dag Sverre Seljebotn wrote:
>>>>> Like I've been saying on some occasions, I'm going show off Cython
>>>>> 18th
>>>>> of May, and have had a tendency to scratch some itches rather than
>>>>> planning how to explain them... so I'd really like a release to
>>>>> happen
>>>>> before then. (Though I also have some fixes/rewrites which are not
>>>>> done
>>>>> yet which will go in next week.)
>>>>>
>>>>> If it helps I can play release manager this time and do the
>>>>> groundwork
>>>>> for making it build Sage etc. (especially if that gets us complex
>>>>> floats
>>>>> in time :-)).
>>>>>
>>>>> Would it be possible to e.g. do a full stop feature freeze next
>>>>> Friday
>>>>> 8th, and then a release the week after?
>>>> I'm not currently working on cython-devel anyway, so I'm glad to
>>>> hear that
>>>> you volunteer as QA manager. :)
>>> Yes, that would be great if you would do the release managing--
>>> definitely would make a mid-May release more feasible.
>> I'll give it a try.
> 
> Dag, any chance on a beta in the near future? Finishing up complex  
> support is #2 on my todo list, so it'll get pushed today.

Yes, sorry, I got sidetracked for a few days (and left the tree in a 
defect state over the weekend as a bonus...). But I'm doing "make check" 
on Sage as I write this and things are looking good.

I was planning on doing it in a few hours, though if you're pushing 
complex support soon I might wait until after then.

(Any news on server access? I can find another location for it, no problem.)

-- 
Dag Sverre

From robertwb at math.washington.edu  Tue May 12 20:34:40 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 12 May 2009 11:34:40 -0700
Subject: [Cython] 0.11.2 release schedule?
In-Reply-To: <4A09BDF9.8070109@student.matnat.uio.no>
References: <49FAF0CC.5070108@student.matnat.uio.no>	<49FB20B8.8090506@behnel.de>		<49FB54C6.4020009@student.matnat.uio.no>
	<1A7D9422-4220-4C7B-852B-42BE15F93F6F@math.washington.edu>
	<4A09BDF9.8070109@student.matnat.uio.no>
Message-ID: 

On May 12, 2009, at 11:20 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On May 1, 2009, at 1:00 PM, Dag Sverre Seljebotn wrote:
>>
>>> Robert Bradshaw wrote:
>>>> On May 1, 2009, at 9:18 AM, Stefan Behnel wrote:
>>>>
>>>>> Dag Sverre Seljebotn wrote:
>>>>>> Like I've been saying on some occasions, I'm going show off  
>>>>>> Cython
>>>>>> 18th
>>>>>> of May, and have had a tendency to scratch some itches rather  
>>>>>> than
>>>>>> planning how to explain them... so I'd really like a release to
>>>>>> happen
>>>>>> before then. (Though I also have some fixes/rewrites which are  
>>>>>> not
>>>>>> done
>>>>>> yet which will go in next week.)
>>>>>>
>>>>>> If it helps I can play release manager this time and do the
>>>>>> groundwork
>>>>>> for making it build Sage etc. (especially if that gets us complex
>>>>>> floats
>>>>>> in time :-)).
>>>>>>
>>>>>> Would it be possible to e.g. do a full stop feature freeze next
>>>>>> Friday
>>>>>> 8th, and then a release the week after?
>>>>> I'm not currently working on cython-devel anyway, so I'm glad to
>>>>> hear that
>>>>> you volunteer as QA manager. :)
>>>> Yes, that would be great if you would do the release managing--
>>>> definitely would make a mid-May release more feasible.
>>> I'll give it a try.
>>
>> Dag, any chance on a beta in the near future? Finishing up complex
>> support is #2 on my todo list, so it'll get pushed today.
>
> Yes, sorry, I got sidetracked for a few days (and left the tree in a
> defect state over the weekend as a bonus...). But I'm doing "make  
> check"
> on Sage as I write this and things are looking good.

No problem. As much as we like Cython, it's not our full time job for  
any of us :).

> I was planning on doing it in a few hours, though if you're pushing
> complex support soon I might wait until after then.

Don't hold up for it--often little stuff falls out so we'll need  
another round for release anyways.

> (Any news on server access? I can find another location for it, no  
> problem.)


My plan is to make a "cython" account on the server and move  
everything over to there (giving both of us access)--just haven't  
done it yet. I'll be happy to push to there myself until then, you  
can just stick it in http://sage.math.washington.edu/home/dagss/.

- Robert


From stefan_ml at behnel.de  Tue May 12 20:52:07 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 12 May 2009 20:52:07 +0200
Subject: [Cython] 0.11.2 release schedule?
In-Reply-To: <4A09BDF9.8070109@student.matnat.uio.no>
References: <49FAF0CC.5070108@student.matnat.uio.no>	<49FB20B8.8090506@behnel.de>		<49FB54C6.4020009@student.matnat.uio.no>	<1A7D9422-4220-4C7B-852B-42BE15F93F6F@math.washington.edu>
	<4A09BDF9.8070109@student.matnat.uio.no>
Message-ID: <4A09C557.6050103@behnel.de>


Dag Sverre Seljebotn wrote:
> Robert Bradshaw wrote:
>> Dag, any chance on a beta in the near future? Finishing up complex  
>> support is #2 on my todo list, so it'll get pushed today.
> 
> Yes, sorry, I got sidetracked for a few days (and left the tree in a 
> defect state over the weekend as a bonus...). But I'm doing "make check" 
> on Sage as I write this and things are looking good.
> 
> I was planning on doing it in a few hours, though if you're pushing 
> complex support soon I might wait until after then.

Just a quick note that lxml builds and tests nicely with cython-devel on
Py2.5 and 3.1. So a beta is ok over here.

Stefan


From dagss at student.matnat.uio.no  Tue May 12 21:53:23 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 12 May 2009 21:53:23 +0200
Subject: [Cython] 0.11.2 release schedule?
In-Reply-To: 
References: <49FAF0CC.5070108@student.matnat.uio.no>	<49FB20B8.8090506@behnel.de>		<49FB54C6.4020009@student.matnat.uio.no>	<1A7D9422-4220-4C7B-852B-42BE15F93F6F@math.washington.edu>	<4A09BDF9.8070109@student.matnat.uio.no>
	
Message-ID: <4A09D3B3.5090902@student.matnat.uio.no>

Robert Bradshaw wrote:
> On May 12, 2009, at 11:20 AM, Dag Sverre Seljebotn wrote:
> 
>> Robert Bradshaw wrote:
>>> On May 1, 2009, at 1:00 PM, Dag Sverre Seljebotn wrote:
>>>
>>>> Robert Bradshaw wrote:
>>>>> On May 1, 2009, at 9:18 AM, Stefan Behnel wrote:
>>>>>
>>>>>> Dag Sverre Seljebotn wrote:
>>>>>>> Like I've been saying on some occasions, I'm going show off  
>>>>>>> Cython
>>>>>>> 18th
>>>>>>> of May, and have had a tendency to scratch some itches rather  
>>>>>>> than
>>>>>>> planning how to explain them... so I'd really like a release to
>>>>>>> happen
>>>>>>> before then. (Though I also have some fixes/rewrites which are  
>>>>>>> not
>>>>>>> done
>>>>>>> yet which will go in next week.)
>>>>>>>
>>>>>>> If it helps I can play release manager this time and do the
>>>>>>> groundwork
>>>>>>> for making it build Sage etc. (especially if that gets us complex
>>>>>>> floats
>>>>>>> in time :-)).
>>>>>>>
>>>>>>> Would it be possible to e.g. do a full stop feature freeze next
>>>>>>> Friday
>>>>>>> 8th, and then a release the week after?
>>>>>> I'm not currently working on cython-devel anyway, so I'm glad to
>>>>>> hear that
>>>>>> you volunteer as QA manager. :)
>>>>> Yes, that would be great if you would do the release managing--
>>>>> definitely would make a mid-May release more feasible.
>>>> I'll give it a try.
>>> Dag, any chance on a beta in the near future? Finishing up complex
>>> support is #2 on my todo list, so it'll get pushed today.
>> Yes, sorry, I got sidetracked for a few days (and left the tree in a
>> defect state over the weekend as a bonus...). But I'm doing "make  
>> check"
>> on Sage as I write this and things are looking good.

Status on Sage is that these fail:

         sage -t  "devel/sage/sage/libs/ntl/ntl_ZZ_pX.pyx"
         sage -t  "devel/sage/sage/rings/integer.pyx"
         sage -t  "devel/sage/sage/plot/plot.py"
         sage -t  "devel/sage/sage/interfaces/maxima.py"

The last two seem to be because of fragile doctests. The first two are 
because of changed Cython semantics:

     OverflowError: can't convert negative value to unsigned int

In the cases it is about going from "long" via "object" to "unsigned 
long" (with, confirmed, a negative integer), so I'd say the mistake is 
in Sage here.

Question is again whether this is too much semantics change for a minor 
release, but I'm +1.

-- 
Dag Sverre

From michael.abshoff at googlemail.com  Tue May 12 22:04:33 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Tue, 12 May 2009 13:04:33 -0700
Subject: [Cython] 0.11.2 release schedule?
In-Reply-To: <4A09D3B3.5090902@student.matnat.uio.no>
References: <49FAF0CC.5070108@student.matnat.uio.no>	<49FB20B8.8090506@behnel.de>		<49FB54C6.4020009@student.matnat.uio.no>	<1A7D9422-4220-4C7B-852B-42BE15F93F6F@math.washington.edu>	<4A09BDF9.8070109@student.matnat.uio.no>	
	<4A09D3B3.5090902@student.matnat.uio.no>
Message-ID: <4A09D651.7090309@gmail.com>

Dag Sverre Seljebotn wrote:
> Robert Bradshaw wrote:



Hi Dag,

> Status on Sage is that these fail:
> 
>          sage -t  "devel/sage/sage/libs/ntl/ntl_ZZ_pX.pyx"
>          sage -t  "devel/sage/sage/rings/integer.pyx"
>          sage -t  "devel/sage/sage/plot/plot.py"
>          sage -t  "devel/sage/sage/interfaces/maxima.py"
> 
> The last two seem to be because of fragile doctests. 

Can you post the first doctest failure from each failing file?

Cheers,

Michael

From dagss at student.matnat.uio.no  Tue May 12 22:21:06 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 12 May 2009 22:21:06 +0200
Subject: [Cython] 0.11.2 release schedule?
In-Reply-To: <4A09D651.7090309@gmail.com>
References: <49FAF0CC.5070108@student.matnat.uio.no>	<49FB20B8.8090506@behnel.de>		<49FB54C6.4020009@student.matnat.uio.no>	<1A7D9422-4220-4C7B-852B-42BE15F93F6F@math.washington.edu>	<4A09BDF9.8070109@student.matnat.uio.no>		<4A09D3B3.5090902@student.matnat.uio.no>
	<4A09D651.7090309@gmail.com>
Message-ID: <4A09DA32.2040607@student.matnat.uio.no>

Michael Abshoff wrote:
> Dag Sverre Seljebotn wrote:
>> Robert Bradshaw wrote:
> 
> 
> 
> Hi Dag,
> 
>> Status on Sage is that these fail:
>>
>>          sage -t  "devel/sage/sage/libs/ntl/ntl_ZZ_pX.pyx"
>>          sage -t  "devel/sage/sage/rings/integer.pyx"
>>          sage -t  "devel/sage/sage/plot/plot.py"
>>          sage -t  "devel/sage/sage/interfaces/maxima.py"
>>
>> The last two seem to be because of fragile doctests. 
> 
> Can you post the first doctest failure from each failing file?

This is on Sage 3.4.1 (my own build).

sage -t  "devel/sage/sage/libs/ntl/ntl_ZZ_pX.pyx"
**********************************************************************
File 
"/home/dagss/sage/sage-3.4.1/devel/sage/sage/libs/ntl/ntl_ZZ_pX.pyx", 
line 751:
     sage: f.leading_coefficient()
Exception raised:
     Traceback (most recent call last):
       File "/home/dagss/sage/sage-3.4.1/local/bin/ncadoctest.py", line 
1231, in run_one_test
         self.run_one_example(test, example, filename, compileflags)
       File "/home/dagss/sage/sage-3.4.1/local/bin/sagedoctest.py", line 
38, in run_one_example
         OrigDocTestRunner.run_one_example(self, test, example, 
filename, compileflags)
       File "/home/dagss/sage/sage-3.4.1/local/bin/ncadoctest.py", line 
1172, in run_one_example
         compileflags, 1) in test.globs
       File "", line 1, in 
         f.leading_coefficient()###line 751:
     sage: f.leading_coefficient()
       File "ntl_ZZ_pX.pyx", line 758, in 
sage.libs.ntl.ntl_ZZ_pX.ntl_ZZ_pX.leading_coefficient 
(sage/libs/ntl/ntl_ZZ_pX.cpp:6208)
       File "ntl_ZZ_pX.pyx", line 255, in 
sage.libs.ntl.ntl_ZZ_pX.ntl_ZZ_pX.__getitem__ 
(sage/libs/ntl/ntl_ZZ_pX.cpp:3853)
     OverflowError: can't convert negative value to unsigned int
**********************************************************************


sage -t  "devel/sage/sage/rings/integer.pyx"
**********************************************************************
File "/home/dagss/sage/sage-3.4.1/devel/sage/sage/rings/integer.pyx", 
line 4388:
     sage: w.test_bit(-1)
Exception raised:
     Traceback (most recent call last):
       File "/home/dagss/sage/sage-3.4.1/local/bin/ncadoctest.py", line 
1231, in run_one_test
         self.run_one_example(test, example, filename, compileflags)
       File "/home/dagss/sage/sage-3.4.1/local/bin/sagedoctest.py", line 
38, in run_one_example
         OrigDocTestRunner.run_one_example(self, test, example, 
filename, compileflags)
       File "/home/dagss/sage/sage-3.4.1/local/bin/ncadoctest.py", line 
1172, in run_one_example
         compileflags, 1) in test.globs
       File "", line 1, in 
         w.test_bit(-Integer(1))###line 4388:
     sage: w.test_bit(-1)
       File "integer.pyx", line 4392, in 
sage.rings.integer.Integer.test_bit (sage/rings/integer.c:25745)
     OverflowError: can't convert negative value to unsigned long

sage -t  "devel/sage/sage/plot/plot.py"
**********************************************************************
File "/home/dagss/sage/sage-3.4.1/devel/sage/sage/plot/plot.py", line 206:
     sage: savefig('sage.png')
Exception raised:
     Traceback (most recent call last):
       File "/home/dagss/sage/sage-3.4.1/local/bin/ncadoctest.py", line 
1231, in run_one_test
         self.run_one_example(test, example, filename, compileflags)
       File "/home/dagss/sage/sage-3.4.1/local/bin/sagedoctest.py", line 
38, in run_one_example
         OrigDocTestRunner.run_one_example(self, test, example, 
filename, compileflags)
       File "/home/dagss/sage/sage-3.4.1/local/bin/ncadoctest.py", line 
1172, in run_one_example
         compileflags, 1) in test.globs
       File "", line 1, in 
         savefig('sage.png')###line 206:
     sage: savefig('sage.png')
       File 
"/home/dagss/sage/sage-3.4.1/local/lib/python2.5/site-packages/matplotlib/pyplot.py", 
line 346, in savefig
         return fig.savefig(*args, **kwargs)
       File 
"/home/dagss/sage/sage-3.4.1/local/lib/python2.5/site-packages/matplotlib/figure.py", 
line 1001, in savefig
         self.canvas.print_figure(*args, **kwargs)
       File 
"/home/dagss/sage/sage-3.4.1/local/lib/python2.5/site-packages/matplotlib/backend_bases.py", 
line 1411, in print_figure
         '%s.' % (format, ', '.join(formats)))
     ValueError: Format "png" is not supported.
     Supported formats: eps, ps.
**********************************************************************

sage -t  "devel/sage/sage/interfaces/maxima.py"
**********************************************************************
File "/home/dagss/sage/sage-3.4.1/devel/sage/sage/interfaces/maxima.py", 
line 822:
     sage: maxima._command_runner('describe', 'gcd')
Expected:
     -- Function: gcd (, , , ...)
     ...
Got:
     ;; Loading file /home/dagss/.clisprc ...
     ;; Loaded file /home/dagss/.clisprc
     
      -- Function: gcd (, , , ...)
          Returns the greatest common divisor of  and .  The flag
          `gcd' determines which algorithm is employed.  Setting `gcd' to
          `ez', `subres', `red', or `spmod' selects the `ezgcd',
          subresultant `prs', reduced, or modular algorithm, respectively.
          If `gcd' `false' then `gcd (, , )' always returns 1
          for all .  Many functions (e.g.  `ratsimp', `factor', etc.)
          cause gcd's to be taken implicitly.  For homogeneous polynomials
          it is recommended that `gcd' equal to `subres' be used.  To take
          the gcd when an algebraic is present, e.g., `gcd (^2 -
          2*sqrt(2)* + 2,  - sqrt(2))', `algebraic' must be `true' and
          `gcd' must not be `ez'.
     
          The `gcd' flag, default: `spmod', if `false' will also prevent the
          greatest common divisor from being taken when expressions are
          converted to canonical rational expression (CRE) form.  This will
          sometimes speed the calculation if gcds are not required.
     
     
       There are also some inexact matches for `gcd'.
       Try `?? gcd' to see them.
     
                                          true
     
**********************************************************************


-- 
Dag Sverre

From cswiercz at gmail.com  Tue May 12 22:19:53 2009
From: cswiercz at gmail.com (Chris Swierczewski)
Date: Tue, 12 May 2009 13:19:53 -0700
Subject: [Cython] Converting Bound Methods to Unbound Methods
Message-ID: 

Hello,

I have a cdef'ed class with a cdef'ed method that looks something like this:

cdef class MyClass:
   cdef void myfunc(self, double *x):
       ... (function contents) ...

   def execute():
       call_func( &(self.myfunc) )

I'm linking to a function in an external library that accepts a
function pointer of similar signature

cdef extern from 'somelibrary.h':
   cdef void call_func(void (*func)(double *x))

However, I receive an error at compile time that a function of type

void (*)(MyClass, double *)

cannot be converted to a function of type

void (*)(double *)

I was wondering there is a way to unbind this myfunc class method so
that it can be properly sent to this external function. I know that I
can simply create a function pointer attribute as part of the class
but it would seem cleaner for a user to subclass and override myfunc
instead of subclassing and setting the value of function pointer
attribute from within an overridden __cdef__ method.

Thank you in advance for any hints or suggestions!

-- 
Chris Swierczewski
cswiercz at gmail.com
blog: cswiercz.wordpress.com

From stefan_ml at behnel.de  Tue May 12 22:24:41 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 12 May 2009 22:24:41 +0200
Subject: [Cython] Converting Bound Methods to Unbound Methods
In-Reply-To: 
References: 
Message-ID: <4A09DB09.1020400@behnel.de>


Chris Swierczewski wrote:
> I have a cdef'ed class with a cdef'ed method that looks something like this:
> 
> cdef class MyClass:
>    cdef void myfunc(self, double *x):
>        ... (function contents) ...
> 
>    def execute():
>        call_func( &(self.myfunc) )
> 
> I'm linking to a function in an external library that accepts a
> function pointer of similar signature
> 
> cdef extern from 'somelibrary.h':
>    cdef void call_func(void (*func)(double *x))
> 
> However, I receive an error at compile time that a function of type
> 
> void (*)(MyClass, double *)
> 
> cannot be converted to a function of type
> 
> void (*)(double *)
> 
> I was wondering there is a way to unbind this myfunc class method so
> that it can be properly sent to this external function.

No you can't, the signatures are incompatible at the C level.


> I know that I
> can simply create a function pointer attribute as part of the class
> but it would seem cleaner for a user to subclass and override myfunc
> instead of subclassing and setting the value of function pointer
> attribute from within an overridden __cdef__ method.

The question is: if the method makes sense without the class, why is it a
method in the first place? Maybe you should make it a function and just
call it from the class method?

Stefan

From dalcinl at gmail.com  Tue May 12 22:29:05 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 12 May 2009 17:29:05 -0300
Subject: [Cython] about _USE_MATH_DEFINES
Message-ID: 

In this changeset

changeset:   1024:ce19a1f80694
user:        JimKleckner
date:        Fri Aug 15 01:56:03 2008 -0700
summary:     Windos __stdcall and _USE_MATH_DEFINES

Jim added this line (and other #define's for compatibility of calling
conventions on non-Win32)

code.putln("  #define _USE_MATH_DEFINES")

but it seems (looking at the full change) that this #define will not
be active for Windows 64.

Is this corrent? Should this be fixed for the Win64 case?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From michael.abshoff at googlemail.com  Tue May 12 22:30:53 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Tue, 12 May 2009 13:30:53 -0700
Subject: [Cython] 0.11.2 release schedule?
In-Reply-To: <4A09DA32.2040607@student.matnat.uio.no>
References: <49FAF0CC.5070108@student.matnat.uio.no>	<49FB20B8.8090506@behnel.de>		<49FB54C6.4020009@student.matnat.uio.no>	<1A7D9422-4220-4C7B-852B-42BE15F93F6F@math.washington.edu>	<4A09BDF9.8070109@student.matnat.uio.no>		<4A09D3B3.5090902@student.matnat.uio.no>	<4A09D651.7090309@gmail.com>
	<4A09DA32.2040607@student.matnat.uio.no>
Message-ID: <4A09DC7D.9010307@gmail.com>

Dag Sverre Seljebotn wrote:
> Michael Abshoff wrote:
>> Dag Sverre Seljebotn wrote:
>>> Robert Bradshaw wrote:
>> 
>>
>> Hi Dag,
>>
>>> Status on Sage is that these fail:
>>>
>>>          sage -t  "devel/sage/sage/libs/ntl/ntl_ZZ_pX.pyx"
>>>          sage -t  "devel/sage/sage/rings/integer.pyx"
>>>          sage -t  "devel/sage/sage/plot/plot.py"
>>>          sage -t  "devel/sage/sage/interfaces/maxima.py"
>>>
>>> The last two seem to be because of fragile doctests. 
>> Can you post the first doctest failure from each failing file?
> 
> This is on Sage 3.4.1 (my own build).

Ok. Which compiler and host? 3.4.2 is out by the way and 4.0.alpha0 
should drop tonight :)


> sage -t  "devel/sage/sage/libs/ntl/ntl_ZZ_pX.pyx"
> **********************************************************************



>      OverflowError: can't convert negative value to unsigned int
> **********************************************************************

Ok, changing semantics like that makes me queasy, but I assume you guys 
know what you are doing. It seems like a rather deep fundamental change 
for 0.11.1->0.11.2.

> 
> sage -t  "devel/sage/sage/rings/integer.pyx"
> **********************************************************************
> File "/home/dagss/sage/sage-3.4.1/devel/sage/sage/rings/integer.



>      OverflowError: can't convert negative value to unsigned long

Ok.

> sage -t  "devel/sage/sage/plot/plot.py"
> **********************************************************************
> File "/home/dagss/sage/sage-3.4.1/devel/sage/sage/plot/plot.py", line 206:
>      sage: savefig('sage.png')



>      ValueError: Format "png" is not supported.
>      Supported formats: eps, ps.
> **********************************************************************

I have never seen this and I am pretty sure it is unrelated to Cython. 
If you could send me a link to the compressed install.log I could take a 
look. This indicates a problem with libpng.

> sage -t  "devel/sage/sage/interfaces/maxima.py"
> **********************************************************************
> File "/home/dagss/sage/sage-3.4.1/devel/sage/sage/interfaces/maxima.py", 
> line 822:
>      sage: maxima._command_runner('describe', 'gcd')



>        There are also some inexact matches for `gcd'.
>        Try `?? gcd' to see them.
>      
>                                           true
>      
> **********************************************************************

In case you have a $HOME/.maxima can you move that out of the way? If 
you do this might explain this problem.

Sooner or later it might be time to take this to sage-devel ;)

Cheers,

Michael

From cswiercz at gmail.com  Tue May 12 22:36:02 2009
From: cswiercz at gmail.com (Chris Swierczewski)
Date: Tue, 12 May 2009 13:36:02 -0700
Subject: [Cython] Converting Bound Methods to Unbound Methods
In-Reply-To: <4A09DB09.1020400@behnel.de>
References: 
	<4A09DB09.1020400@behnel.de>
Message-ID: 

> No you can't, the signatures are incompatible at the C level.

Drat.

> The question is: if the method makes sense without the class, why is it a
> method in the first place? Maybe you should make it a function and just
> call it from the class method?

Thanks for bringing that to mind. I guess I should share that
possibility with the other developers for the project I'm working on.
We thought of encapsulating some of the functionality in a class just
for organizational purposes but perhaps bumping it up to module-level
would be better.

Thanks,
Chris

From dagss at student.matnat.uio.no  Tue May 12 22:40:29 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 12 May 2009 22:40:29 +0200
Subject: [Cython] 0.11.2 release schedule?
In-Reply-To: <4A09DC7D.9010307@gmail.com>
References: <49FAF0CC.5070108@student.matnat.uio.no>	<49FB20B8.8090506@behnel.de>		<49FB54C6.4020009@student.matnat.uio.no>	<1A7D9422-4220-4C7B-852B-42BE15F93F6F@math.washington.edu>	<4A09BDF9.8070109@student.matnat.uio.no>		<4A09D3B3.5090902@student.matnat.uio.no>	<4A09D651.7090309@gmail.com>	<4A09DA32.2040607@student.matnat.uio.no>
	<4A09DC7D.9010307@gmail.com>
Message-ID: <4A09DEBD.9040006@student.matnat.uio.no>

Michael Abshoff wrote:
> Dag Sverre Seljebotn wrote:
>> Michael Abshoff wrote:
>>> Dag Sverre Seljebotn wrote:
>>>> Robert Bradshaw wrote:
>>> 
>>>
>>> Hi Dag,
>>>
>>>> Status on Sage is that these fail:
>>>>
>>>>          sage -t  "devel/sage/sage/libs/ntl/ntl_ZZ_pX.pyx"
>>>>          sage -t  "devel/sage/sage/rings/integer.pyx"
>>>>          sage -t  "devel/sage/sage/plot/plot.py"
>>>>          sage -t  "devel/sage/sage/interfaces/maxima.py"
>>>>
>>>> The last two seem to be because of fragile doctests. 
>>> Can you post the first doctest failure from each failing file?
>> This is on Sage 3.4.1 (my own build).
> 
> Ok. Which compiler and host? 3.4.2 is out by the way and 4.0.alpha0 
> should drop tonight :)

dagss at boksen:~/sage/sage-3.4.1$ uname -a
Linux boksen 2.6.24-19-generic #1 SMP Fri Jul 11 21:01:46 UTC 2008 
x86_64 GNU/Linux

dagss at boksen:~/sage/sage-3.4.1$ gcc -v
Using built-in specs.
Target: x86_64-linux-gnu
Configured with: ../src/configure -v 
--enable-languages=c,c++,fortran,objc,obj-c++,treelang --prefix=/usr 
--enable-shared --with-system-zlib --libexecdir=/usr/lib 
--without-included-gettext --enable-threads=posix --enable-nls 
--with-gxx-include-dir=/usr/include/c++/4.2 --program-suffix=-4.2 
--enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc 
--enable-mpfr --enable-checking=release --build=x86_64-linux-gnu 
--host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7)

> Ok, changing semantics like that makes me queasy, but I assume you guys 
> know what you are doing. It seems like a rather deep fundamental change 
> for 0.11.1->0.11.2.

I think there's no way around it now -- unless Robert wants to make this 
0.12 because of this change alone, it will have to be remedied by a big 
warning in the release notes only.

> I have never seen this and I am pretty sure it is unrelated to Cython. 
> If you could send me a link to the compressed install.log I could take a 
> look. This indicates a problem with libpng.

Well the real problem is if Cython does something strange (could by 
anything, like truncating a list or whatever) leading to this. I will 
have to run with the original Cython first and then I'll file the issue 
properly if it is Cython-unrelated.

> In case you have a $HOME/.maxima can you move that out of the way? If 
> you do this might explain this problem.

I don't. (I get startup timeouts too BTW.)

-- 
Dag Sverre

From michael.abshoff at googlemail.com  Tue May 12 22:45:18 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Tue, 12 May 2009 13:45:18 -0700
Subject: [Cython] 0.11.2 release schedule?
In-Reply-To: <4A09DEBD.9040006@student.matnat.uio.no>
References: <49FAF0CC.5070108@student.matnat.uio.no>	<49FB20B8.8090506@behnel.de>		<49FB54C6.4020009@student.matnat.uio.no>	<1A7D9422-4220-4C7B-852B-42BE15F93F6F@math.washington.edu>	<4A09BDF9.8070109@student.matnat.uio.no>		<4A09D3B3.5090902@student.matnat.uio.no>	<4A09D651.7090309@gmail.com>	<4A09DA32.2040607@student.matnat.uio.no>	<4A09DC7D.9010307@gmail.com>
	<4A09DEBD.9040006@student.matnat.uio.no>
Message-ID: <4A09DFDE.2080802@gmail.com>

Dag Sverre Seljebotn wrote:
> Michael Abshoff wrote:
>> Dag Sverre Seljebotn wrote:
>>> Michael Abshoff wrote:
>>>> Dag Sverre Seljebotn wrote:
>>>>> Robert Bradshaw wrote:
>>>> 
>>>>
>>>> Hi Dag,
>>>>
>>>>> Status on Sage is that these fail:
>>>>>
>>>>>          sage -t  "devel/sage/sage/libs/ntl/ntl_ZZ_pX.pyx"
>>>>>          sage -t  "devel/sage/sage/rings/integer.pyx"
>>>>>          sage -t  "devel/sage/sage/plot/plot.py"
>>>>>          sage -t  "devel/sage/sage/interfaces/maxima.py"
>>>>>
>>>>> The last two seem to be because of fragile doctests. 
>>>> Can you post the first doctest failure from each failing file?
>>> This is on Sage 3.4.1 (my own build).
>> Ok. Which compiler and host? 3.4.2 is out by the way and 4.0.alpha0 
>> should drop tonight :)
> 
> dagss at boksen:~/sage/sage-3.4.1$ uname -a
> Linux boksen 2.6.24-19-generic #1 SMP Fri Jul 11 21:01:46 UTC 2008 
> x86_64 GNU/Linux
> 
> dagss at boksen:~/sage/sage-3.4.1$ gcc -v
> Using built-in specs.
> Target: x86_64-linux-gnu
> Configured with: ../src/configure -v 
> --enable-languages=c,c++,fortran,objc,obj-c++,treelang --prefix=/usr 
> --enable-shared --with-system-zlib --libexecdir=/usr/lib 
> --without-included-gettext --enable-threads=posix --enable-nls 
> --with-gxx-include-dir=/usr/include/c++/4.2 --program-suffix=-4.2 
> --enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc 
> --enable-mpfr --enable-checking=release --build=x86_64-linux-gnu 
> --host=x86_64-linux-gnu --target=x86_64-linux-gnu
> Thread model: posix
> gcc version 4.2.3 (Ubuntu 4.2.3-2ubuntu7)

ok, noting unusual so far.

>> Ok, changing semantics like that makes me queasy, but I assume you guys 
>> know what you are doing. It seems like a rather deep fundamental change 
>> for 0.11.1->0.11.2.
> 
> I think there's no way around it now -- unless Robert wants to make this 
> 0.12 because of this change alone, it will have to be remedied by a big 
> warning in the release notes only.

I did not mean to imply that this discussion should be reopened, just 
that some people might be a little surprised that a "minor" bug fix 
breaks their code :)

>> I have never seen this and I am pretty sure it is unrelated to Cython. 
>> If you could send me a link to the compressed install.log I could take a 
>> look. This indicates a problem with libpng.
> 
> Well the real problem is if Cython does something strange (could by 
> anything, like truncating a list or whatever) leading to this. I will 
> have to run with the original Cython first and then I'll file the issue 
> properly if it is Cython-unrelated.

Still, MPL claims that png isn't supported. If this is a Cython issue 
somewhat color me surprised.

>> In case you have a $HOME/.maxima can you move that out of the way? If 
>> you do this might explain this problem.
> 
> I don't. (I get startup timeouts too BTW.)

Ok. 3.4.1 shipped with the new clisp 2.47 which has been working out 
rather well, but there is always a compiler that breaks it. Sage 4.0 
will switch to ecl and use Maxima a whole lot less. Eventually Maxima 
won't be used for anything in Sage, but that is still some time to go.

If downgrading to the old cython does not fix your problem I am more 
than happy to look into install.log from your box to see if I can find 
the cause for the MPL png issue for example.

Cheers,

Michael



From dagss at student.matnat.uio.no  Tue May 12 22:52:35 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 12 May 2009 22:52:35 +0200
Subject: [Cython] Cython 0.11.2 beta
Message-ID: <4A09E193.20001@student.matnat.uio.no>

Please try it out and report any problems:

http://sage.math.washington.edu/home/dagss/Cython-0.11.2.beta.tar.gz

(This is the same trunk as was discussed earlier today, I poked around a 
bit more and didn't find more problems.)

NOTE that the semantics for int conversion has changed in this version, 
so that converting a negative Python int to an unsigned C type will 
raise an OverflowException.

-- 
Dag Sverre

From robertwb at math.washington.edu  Tue May 12 22:54:55 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 12 May 2009 13:54:55 -0700
Subject: [Cython] 0.11.2 release schedule?
In-Reply-To: <4A09DC7D.9010307@gmail.com>
References: <49FAF0CC.5070108@student.matnat.uio.no>	<49FB20B8.8090506@behnel.de>		<49FB54C6.4020009@student.matnat.uio.no>	<1A7D9422-4220-4C7B-852B-42BE15F93F6F@math.washington.edu>	<4A09BDF9.8070109@student.matnat.uio.no>		<4A09D3B3.5090902@student.matnat.uio.no>	<4A09D651.7090309@gmail.com>
	<4A09DA32.2040607@student.matnat.uio.no>
	<4A09DC7D.9010307@gmail.com>
Message-ID: <3C620D24-BF64-48F0-9E73-E87EBE13AFDF@math.washington.edu>

On May 12, 2009, at 1:30 PM, Michael Abshoff wrote:

> Dag Sverre Seljebotn wrote:
>> Michael Abshoff wrote:
>>> Dag Sverre Seljebotn wrote:
>>>> Robert Bradshaw wrote:
>>> 
>>>
>>> Hi Dag,
>>>
>>>> Status on Sage is that these fail:
>>>>
>>>>          sage -t  "devel/sage/sage/libs/ntl/ntl_ZZ_pX.pyx"
>>>>          sage -t  "devel/sage/sage/rings/integer.pyx"
>>>>          sage -t  "devel/sage/sage/plot/plot.py"
>>>>          sage -t  "devel/sage/sage/interfaces/maxima.py"
>>>>
>>>> The last two seem to be because of fragile doctests.
>>> Can you post the first doctest failure from each failing file?
>>
>> This is on Sage 3.4.1 (my own build).
>
> Ok. Which compiler and host? 3.4.2 is out by the way and 4.0.alpha0
> should drop tonight :)
>
>
>> sage -t  "devel/sage/sage/libs/ntl/ntl_ZZ_pX.pyx"
>> ********************************************************************* 
>> *
>
> 
>
>>      OverflowError: can't convert negative value to unsigned int
>> ********************************************************************* 
>> *
>
> Ok, changing semantics like that makes me queasy, but I assume you  
> guys
> know what you are doing. It seems like a rather deep fundamental  
> change
> for 0.11.1->0.11.2.

I consider it a bug that we used to allow this--what it's doing now  
is converting -1 to 2^64-1.

The patch for this is already in Sage: http://trac.sagemath.org/ 
sage_trac/attachment/ticket/5803/5803-cython-0.11.1.patch

- Robert



From robertwb at math.washington.edu  Tue May 12 22:58:04 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 12 May 2009 13:58:04 -0700
Subject: [Cython] Converting Bound Methods to Unbound Methods
In-Reply-To: 
References: 
	<4A09DB09.1020400@behnel.de>
	
Message-ID: <7319CA41-2B68-42C7-AE18-1B954FA51B03@math.washington.edu>

On May 12, 2009, at 1:36 PM, Chris Swierczewski wrote:

>> No you can't, the signatures are incompatible at the C level.
>
> Drat.
>
>> The question is: if the method makes sense without the class, why  
>> is it a
>> method in the first place? Maybe you should make it a function and  
>> just
>> call it from the class method?
>
> Thanks for bringing that to mind. I guess I should share that
> possibility with the other developers for the project I'm working on.
> We thought of encapsulating some of the functionality in a class just
> for organizational purposes but perhaps bumping it up to module-level
> would be better.


You can make a callback functions like

cdef my_callback(double x, void* params):
     (params).callback_method(x)

if the overhead isn't too great.

- Robert


From cswiercz at gmail.com  Wed May 13 00:33:18 2009
From: cswiercz at gmail.com (Chris Swierczewski)
Date: Tue, 12 May 2009 15:33:18 -0700
Subject: [Cython] Converting Bound Methods to Unbound Methods
In-Reply-To: <7319CA41-2B68-42C7-AE18-1B954FA51B03@math.washington.edu>
References: 
	<4A09DB09.1020400@behnel.de>
	
	<7319CA41-2B68-42C7-AE18-1B954FA51B03@math.washington.edu>
Message-ID: 

Robert,

> You can make a callback functions like
>
> cdef my_callback(double x, void* params):
> ? ? (params).callback_method(x)
>
> if the overhead isn't too great.

Sorry but I'm not sure what's going on in your example. Could you
elaborate a bit or is there a Cython docs page you could direct me to?
Thanks.

-- 
Chris Swierczewski
cswiercz at gmail.com
blog: cswiercz.wordpress.com

From greg.ewing at canterbury.ac.nz  Wed May 13 03:45:27 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 13 May 2009 13:45:27 +1200
Subject: [Cython] Converting Bound Methods to Unbound Methods
In-Reply-To: <4A09DB09.1020400@behnel.de>
References: 
	<4A09DB09.1020400@behnel.de>
Message-ID: <4A0A2637.1070706@canterbury.ac.nz>

Stefan Behnel wrote:

>>cdef class MyClass:
>>   cdef void myfunc(self, double *x):
>>       ... (function contents) ...
>>
>>   def execute():
>>       call_func( &(self.myfunc) )

Cython and Pyrex should probably disallow taking the
address of a C method of an instance, since it doesn't
do what it looks like it does.

-- 
Greg

From lrhazi at gmail.com  Wed May 13 05:48:32 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Tue, 12 May 2009 23:48:32 -0400
Subject: [Cython] ctypes Vs Cython
Message-ID: 

Hello,

How fast/slow is using ctypes to talk to a C lib, compared to doing it
directly form Cython?

While debugging an issue using ctypes, I found myself writing code
that was mixing the two methods... the Cython module was passing a
pointer to my Python code, which then uses ctypes functions to talk
the C lib, passing the pointer along the way, and it seems to work
fine... So, my question is, when I have the option to do some
interaction with the C lib in Cython or in ctypes, which should I go
with?

ctypes is more pythonic with its Structures concepts.. but is it
really slow? would I gain from doing it the harder way, in Cython?

Thanks a lot,
Mohamed.

From sccolbert at gmail.com  Wed May 13 06:31:35 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Wed, 13 May 2009 00:31:35 -0400
Subject: [Cython] ctypes Vs Cython
In-Reply-To: 
References: 
Message-ID: <7f014ea60905122131q3de90a6cv55387813c11fac9d@mail.gmail.com>

What follows is my 0.02, and i'm really new at these two libs so keep in
mind I could be completely off base. I'm sure someone will jump in and
correct my errors.

I think the speed is really going to depend on the nature of your calls into
the c library. From the code i've examined, there will probably be more
overhead for the function call when using ctypes vs cython, but cython
generated source code is somewhat less than maintainable and I would wager
also less optimized than something you would write yourself.

If your making lots of rapid calls to short running functions in the
C-library, then you may start to feel the ctypes overhead.

Chris
On Tue, May 12, 2009 at 11:48 PM, Mohamed Lrhazi  wrote:

> Hello,
>
> How fast/slow is using ctypes to talk to a C lib, compared to doing it
> directly form Cython?
>
> While debugging an issue using ctypes, I found myself writing code
> that was mixing the two methods... the Cython module was passing a
> pointer to my Python code, which then uses ctypes functions to talk
> the C lib, passing the pointer along the way, and it seems to work
> fine... So, my question is, when I have the option to do some
> interaction with the C lib in Cython or in ctypes, which should I go
> with?
>
> ctypes is more pythonic with its Structures concepts.. but is it
> really slow? would I gain from doing it the harder way, in Cython?
>
> Thanks a lot,
> Mohamed.
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090513/1e049e53/attachment.htm 

From sccolbert at gmail.com  Wed May 13 06:34:23 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Wed, 13 May 2009 00:34:23 -0400
Subject: [Cython] ctypes Vs Cython
In-Reply-To: <7f014ea60905122131q3de90a6cv55387813c11fac9d@mail.gmail.com>
References: 
	<7f014ea60905122131q3de90a6cv55387813c11fac9d@mail.gmail.com>
Message-ID: <7f014ea60905122134n77ac06efy97d0e57618377bc3@mail.gmail.com>

I should also state, that its a helluva lot easier to get something up and
running using cython vs ctypes. So if your not doing application programming
intented for distribution, my vote goes to Cython.

I can write my whole script and have it execute to completion using Cython,
before i'm even done compiling a C dll.

Chris

On Wed, May 13, 2009 at 12:31 AM, Chris Colbert  wrote:

> What follows is my 0.02, and i'm really new at these two libs so keep in
> mind I could be completely off base. I'm sure someone will jump in and
> correct my errors.
>
> I think the speed is really going to depend on the nature of your calls
> into the c library. From the code i've examined, there will probably be more
> overhead for the function call when using ctypes vs cython, but cython
> generated source code is somewhat less than maintainable and I would wager
> also less optimized than something you would write yourself.
>
> If your making lots of rapid calls to short running functions in the
> C-library, then you may start to feel the ctypes overhead.
>
> Chris
>   On Tue, May 12, 2009 at 11:48 PM, Mohamed Lrhazi wrote:
>
>> Hello,
>>
>> How fast/slow is using ctypes to talk to a C lib, compared to doing it
>> directly form Cython?
>>
>> While debugging an issue using ctypes, I found myself writing code
>> that was mixing the two methods... the Cython module was passing a
>> pointer to my Python code, which then uses ctypes functions to talk
>> the C lib, passing the pointer along the way, and it seems to work
>> fine... So, my question is, when I have the option to do some
>> interaction with the C lib in Cython or in ctypes, which should I go
>> with?
>>
>> ctypes is more pythonic with its Structures concepts.. but is it
>> really slow? would I gain from doing it the harder way, in Cython?
>>
>> Thanks a lot,
>> Mohamed.
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090513/d4a8b4c8/attachment.htm 

From lrhazi at gmail.com  Wed May 13 06:47:20 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Wed, 13 May 2009 00:47:20 -0400
Subject: [Cython] ctypes Vs Cython
In-Reply-To: <7f014ea60905122131q3de90a6cv55387813c11fac9d@mail.gmail.com>
References: 
	<7f014ea60905122131q3de90a6cv55387813c11fac9d@mail.gmail.com>
Message-ID: 

On Wed, May 13, 2009 at 12:31 AM, Chris Colbert  wrote:
> If your making lots of rapid calls to short running functions in the
> C-library, then you may start to feel the ctypes overhead.

That's what I was afraid to hear.. What I was hoping to hear is "Oh
no, ctypes is all C anyways, and will perform just the same as Cython"

Thanks a lot, I'll work on doing it all in Cython.

Mohamed.

From robertwb at math.washington.edu  Wed May 13 08:35:27 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 12 May 2009 23:35:27 -0700
Subject: [Cython] ctypes Vs Cython
In-Reply-To: 
References: 
	<7f014ea60905122131q3de90a6cv55387813c11fac9d@mail.gmail.com>
	
Message-ID: <56FD54E9-515B-4D08-9650-96D8A7EE24AF@math.washington.edu>

On May 12, 2009, at 9:47 PM, Mohamed Lrhazi wrote:

> On Wed, May 13, 2009 at 12:31 AM, Chris Colbert  
>  wrote:
>> If your making lots of rapid calls to short running functions in the
>> C-library, then you may start to feel the ctypes overhead.
>
> That's what I was afraid to hear..

Hopefully after using Cython a bit, you're fears will quickly go  
away :).

> What I was hoping to hear is "Oh
> no, ctypes is all C anyways, and will perform just the same as Cython"

A simple benchmark:

import ctypes
libm = ctypes.cdll.LoadLibrary("libm.dylib") # platform dependent...
def ctypes_sum(N):
     lib_sqrt = libm.sqrt
     lib_sqrt.argtypes = (ctypes.c_double,)
     lib_sqrt.restype = ctypes.c_double
     s = 0
     for i in range(N):
         s += lib_sqrt(i)
     return s

%cython
cdef extern from "math.h":
     double sqrt(double)

def cython_sum(long N):
     cdef int i
     cdef double s=0
     for i in range(N):
         s += sqrt(i)
     return s


 >>> time ctypes_sum(10**6)
666666166.4588418
Time: CPU 1.13 s, Wall: 1.14 s

time cython_sum(10**6)
666666166.4588418
Time: CPU 0.03 s, Wall: 0.03 s

Of course if you're calling a c function, waiting 1 second for the  
result, and then looking at the return value, the overhead won't make  
a difference at all. On the other extreme, as in the example above,  
the overhead will absolutely kill you.

That being said, ctypes is a very cool module and does have its place.

- Robert

From dagss at student.matnat.uio.no  Wed May 13 09:16:34 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 13 May 2009 09:16:34 +0200
Subject: [Cython] Project: Buffer arithmetic operations
In-Reply-To: <4db580fd0905121314j142a30c7s915e4eb59d772b71@mail.gmail.com>
References: <4db580fd0905121314j142a30c7s915e4eb59d772b71@mail.gmail.com>
Message-ID: <4A0A73D2.6070105@student.matnat.uio.no>

Hoyt Koepke wrote:
> Hello Robert & Dag,
> 
> I'm sending this to you guys first; feel free to bump it to the whole
> list if you think it's appropriate.

Doing so; I think at least it should be available to Kurt as well.

> I'm planning on making time this weekend to attend part of the
> upcoming sage days, and I definitely want to participate in the coding
> sprint and get my hands dirty with some cython development.
> 
> What I'd like to work on, which would be quite relevant in my research
> code, is to write an optimization routine that unrolls numpy array
> expressions involving slices, array operators like +-/*, and scalar
> values into a set of nested loops that avoids the use of any temporary
> array copies.  Thus something like:
> 
> A[0, a:b] = B[0, a:b]*C[a:2*b:2] + D + n
> 
> would get turned into some code that checked broadcasting requirements
> and then into a loop that ran from a to b and looped over the
> appropriate vertices.   Thus we would basically replicate what blitz++
> does with templates or fortran 90 does (so I've heard).  I've sketched
> out an approach to actually coding it up and it seems doable for a
> sage days coding project.

Great that you want to do this :-)

I think doing it for a Sage days project is a bit ambitious, but you 
might be able to get a good start on it. It also depends on what route 
is taken, see below.

> I guess how I see this coming out then is that I'd create a routine
> that takes the cython AST for this expression and a few configuration
> parameters and returns the code for an inline function that accepts as
> parameters all the needed buffer parameters (data ptr, strides, shape,
> and slice parameters if there are some needed (e.g. a, b above),
> constants, etc. ),  and does the appropriate looping and copying.
> Part of the reason I want to do the inline function approach is that I
> want this code to be minimally wrapped up in cython's internals, as a
> tested routine to generate such code might be more generally
> applicable.  Though if there's a stronger argument in favor of the
> other way, that's okay too.

I have given this a lot of thought as well. My thoughts diverge from 
yours in that I want to improve Cython numerics in lot of areas, and 
that the above would come more as a natural consequence than the first 
step. What I'll scetch below is thus a more "patient" route.

My experience is that I'm not sure how good results one can get if one 
is set from the beginning on getting tangible results during a 
conference. A concrete goal like the above seems great for things like a 
GSoC going over one or two months, but perhaps not a week -- the danger 
is that one skips important parts of the design etc. just in order to 
get some example working, but in the end nothing but the example 
works.... (I hope I don't kill your enthusiasm off!, I just want the 
perspective noted :-) ).

Also I should not that I don't have much time the rest of this week to 
be involved in designing this.

Anyway, here's my thoughts. It's pretty much designed to give the 
feeling of "oh no not all this, I just want to do something simple and 
get this done" -- that's OK, just read it and write your rejoinder :-)

A) There's a clash with existing semantics to be resolved. For instance 
if you have a NumPy "matrix" instance, then * does matrix multiplication 
instead, and just implementing this now would break code.

http://wiki.cython.org/enhancements/buffersyntax
http://thread.gmane.org/gmane.comp.python.numeric.general/28439

contains my thoughts on how this could be resolved. I'll use this syntax 
below.

B) I think the feature could be broken down in smaller parts:

1) Efficient slicing (#178):

cdef int[:,:] a
cdef int[:] b = a[3,:]

perhaps also broadcasting?

cdef int[:,:,:] c = a # ?

2) Efficient ndenumerate for-loop construct. This is the "engine" of 
what you want to achieve.

from cython import ndenumerate
cdef int[:,:,:] a = ...
cdef int[:,:,:,:] b = ...

# a[...] = 2 * b[3,...]
for idx, value in ndenumerate(a, b[3,...]):
     a[idx] = 2 * b[3, idx]

Or, perhaps

for a_item, b_item in ndenumerate(a, b[3,...]):
     a_item.set(2*b_item.get())

or similar. The code making sure that the dimension with the smallest 
strides is iterated innermost would be generated here (likely one would 
have a while loop around a for loop; no matter how many dimensions there 
are; see also NumPy's ND-generic iterators... we could also allow this 
to work for "cdef int[...] a", i.e. no declared number of dimensions).

3) Transform "a[...] = 2 * b[3, ...]" into the construct in 2). This is 
easily done using a transform once 2) is done.

3) Ufuncs. I was imagining it like this:

cdef int multiply(int x, int y): return x * y

...
cdef int[:,:] a = ..., b = ..
a[...] = multiply(a, b) # Calls for each item!

So basically "every function with primitive arguments is an ufunc". 
(Fortran has a similar concept, but you need to declare that the 
function has no side-effects -- I say let's just make N invocations part 
of the spec of how this works.)

Obviously this is very easily implemented once 2) is in place.

4) New buffer results (you didn't mention this). Most of this will 
likely be done in Kurt's GSoC. Basically allow Cython to allocate new 
memory, so that you can do

a = 2 * b[3,...]

that is, allocate a new memory area rather than overwrite what is 
already allocated in a.


So..where does this leave your Sage project? Of course you don't need to 
heed this, your patch might get accepted anyway :-) but what I'd suggest 
is to have a crack at doing 2). This can also be done without having a 
new syntax as supporting ndenumerate for the current arrays wouldn't 
hurt at all. If you agree we can move to discussing 2) in more detail.

(BTW if you find time, it being a Sage conference after all, to make 
Sage play more nicely with NumPy so that NumPy would be somewhat usable 
without reassigning Integer and RealNumber I'd be very grateful :-) ).

> Next, what requirements and considerations should I keep in mind when
> designing such a routine?  Here's what I have so far.

Hmm, let's see if I covered it.

> 
> 1.  Since for buffers, the number of dimensions and things like
> contiguity are known at compile time, I want to have these things hard
> coded in.
> 2.  The looping should be ordered so that the smallest dimension is
> copied first (which would have to be determined at runtime, with the
> possible simplification of contiguous arrays).

These two apply directly to 2) above.

> 3.  Ufuncs like exp, cos, sin, etc.

3)

> 3.  Considerations from the current GsoC work ??????

Kurt might change how buffers will be stored in the local scope, but 
nothing very significant. You might want to read over

http://trac.cython.org/cython_trac/query?status=new&status=assigned&status=reopened&order=priority&keywords=~kurtgsoc

(If that is wrapped: Just hit "candidate tickets for Kurt's GSoC" on the 
trac front page)

> 4.  Type escalation stuff.
> 
> Finally, down the line, I'd also like to support functions that do
> reductions like A.sum(axis=0) or A.sum().

I haven't thought much about this. Assuming a new syntax so that we 
don't have to keep NumPy semantics, I think we don't support methods for 
this, but add a decorator on functions:

@cython.buffer.reduction
cdef sum[T](T a, T b): # using future templates as well
     return a + b

and the reduction decorator would add arguments like "axis" etc. and 
just do the right thing.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Wed May 13 09:48:52 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 13 May 2009 09:48:52 +0200
Subject: [Cython] 0.11.2 release schedule?
In-Reply-To: <4A09DFDE.2080802@gmail.com>
References: <49FAF0CC.5070108@student.matnat.uio.no>	<49FB20B8.8090506@behnel.de>		<49FB54C6.4020009@student.matnat.uio.no>	<1A7D9422-4220-4C7B-852B-42BE15F93F6F@math.washington.edu>	<4A09BDF9.8070109@student.matnat.uio.no>		<4A09D3B3.5090902@student.matnat.uio.no>	<4A09D651.7090309@gmail.com>	<4A09DA32.2040607@student.matnat.uio.no>	<4A09DC7D.9010307@gmail.com>	<4A09DEBD.9040006@student.matnat.uio.no>
	<4A09DFDE.2080802@gmail.com>
Message-ID: <4A0A7B64.9030006@student.matnat.uio.no>

Michael Abshoff wrote:
> Dag Sverre Seljebotn wrote:
>> Michael Abshoff wrote:
>>> Ok, changing semantics like that makes me queasy, but I assume you guys 
>>> know what you are doing. It seems like a rather deep fundamental change 
>>> for 0.11.1->0.11.2.
>> I think there's no way around it now -- unless Robert wants to make this 
>> 0.12 because of this change alone, it will have to be remedied by a big 
>> warning in the release notes only.
> 
> I did not mean to imply that this discussion should be reopened, just 
> that some people might be a little surprised that a "minor" bug fix 
> breaks their code :)

Well, there's been some discussions about this and in the end it I looks 
like it was decided that minor releases are /not/ simply 
backwards-compatible bugfix releases. Currently they more reflect 
whether the -unstable tree has been merged since the last time or not 
:-), and basically "how significant the upgrade is".

The reason this isn't a problem is because there are seldom bugs of the 
kind which forces users to upgrade in the first place :-)

If something like a critical bug which forces users to upgrade comes 
along and this *does* become a problem I suppose the solution is to 
backport the fix on the 0.11.1 tree and make a 0.11.1.1 release

>>> I have never seen this and I am pretty sure it is unrelated to Cython. 
>>> If you could send me a link to the compressed install.log I could take a 
>>> look. This indicates a problem with libpng.
>> Well the real problem is if Cython does something strange (could by 
>> anything, like truncating a list or whatever) leading to this. I will 
>> have to run with the original Cython first and then I'll file the issue 
>> properly if it is Cython-unrelated.
> 
> Still, MPL claims that png isn't supported. If this is a Cython issue 
> somewhat color me surprised.

Ah, I didn't notice that there's no Cython code in that stack trace. So 
I won't even bother downgrading Cython (it literally takes hours), my 
install.log is here:

http://sage.math.washington.edu/home/dagss/install.log.gz

-- 
Dag Sverre

From seb.binet at gmail.com  Wed May 13 10:17:07 2009
From: seb.binet at gmail.com (Sebastien Binet)
Date: Wed, 13 May 2009 10:17:07 +0200
Subject: [Cython] ctypes Vs Cython
In-Reply-To: <56FD54E9-515B-4D08-9650-96D8A7EE24AF@math.washington.edu>
References: 
	
	<56FD54E9-515B-4D08-9650-96D8A7EE24AF@math.washington.edu>
Message-ID: <200905131017.07652.binet@cern.ch>

hi,

On Wednesday 13 May 2009 08:35:27 Robert Bradshaw wrote:
> On May 12, 2009, at 9:47 PM, Mohamed Lrhazi wrote:
> > On Wed, May 13, 2009 at 12:31 AM, Chris Colbert
> >
> >  wrote:
> >> If your making lots of rapid calls to short running functions in the
> >> C-library, then you may start to feel the ctypes overhead.
> >
> > That's what I was afraid to hear..
>
> Hopefully after using Cython a bit, you're fears will quickly go
> away :).
>
> > What I was hoping to hear is "Oh
> > no, ctypes is all C anyways, and will perform just the same as Cython"
>
> A simple benchmark:
>
> import ctypes
> libm = ctypes.cdll.LoadLibrary("libm.dylib") # platform dependent...
> def ctypes_sum(N):
>      lib_sqrt = libm.sqrt
>      lib_sqrt.argtypes = (ctypes.c_double,)
>      lib_sqrt.restype = ctypes.c_double
>      s = 0
>      for i in range(N):
>          s += lib_sqrt(i)
>      return s
>
> %cython
> cdef extern from "math.h":
>      double sqrt(double)
>
> def cython_sum(long N):
>      cdef int i
>      cdef double s=0
>      for i in range(N):
>          s += sqrt(i)
>      return s
>
>  >>> time ctypes_sum(10**6)
>
> 666666166.4588418
> Time: CPU 1.13 s, Wall: 1.14 s
>
> time cython_sum(10**6)
> 666666166.4588418
> Time: CPU 0.03 s, Wall: 0.03 s

interesting simple minded benchmark :)

how would this translate into the pure-python mode ? (I couldn't seem to be 
able to declare the C-sqrt function using the pure-python mode of cython:
http://wiki.cython.org/pure wasn't helpful)

cheers,
sebastien.
-- 
#########################################
# Dr. Sebastien Binet
# Laboratoire de l'Accelerateur Lineaire
# Universite Paris-Sud XI
# Batiment 200
# 91898 Orsay
#########################################


From dagss at student.matnat.uio.no  Wed May 13 10:36:04 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 13 May 2009 10:36:04 +0200 (CEST)
Subject: [Cython] ctypes Vs Cython
In-Reply-To: <200905131017.07652.binet@cern.ch>
References: 
	
	<56FD54E9-515B-4D08-9650-96D8A7EE24AF@math.washington.edu>
	<200905131017.07652.binet@cern.ch>
Message-ID: 

Sebastian Pinet wrote:
> how would this translate into the pure-python mode ? (I couldn't seem to
> be
> able to declare the C-sqrt function using the pure-python mode of cython:
> http://wiki.cython.org/pure wasn't helpful)

You can't. Calling C functions directly is a fundamental consequence of
compiling with Cython and even if there was a pure-python syntax for it
your code couldn't run in pure Python.

What one could do (but which noone has attempted yet) is to make the
Cython "shadow module" (which is used when running the code in pure
Python) simply wrap ctypes, so that ctypes is used when not compiling and
Cython is used when compiling. The hardest part here is actually writing
the shadow module (Cython/Shadow.py) so that one can make Cython
declarations and have them translated to ctypes declarations.

(This would require to have some calls like

cython.loadlibrary(...)

which would just be ignored in Cython.)

Dag Sverre


From cournape at gmail.com  Wed May 13 10:37:12 2009
From: cournape at gmail.com (David Cournapeau)
Date: Wed, 13 May 2009 17:37:12 +0900
Subject: [Cython] ctypes Vs Cython
In-Reply-To: <7f014ea60905122131q3de90a6cv55387813c11fac9d@mail.gmail.com>
References: 
	<7f014ea60905122131q3de90a6cv55387813c11fac9d@mail.gmail.com>
Message-ID: <5b8d13220905130137y52c8ae92n7177160166ba8311@mail.gmail.com>

On Wed, May 13, 2009 at 1:31 PM, Chris Colbert  wrote:
> What follows is my 0.02, and i'm really new at these two libs so keep in
> mind I could be completely off base. I'm sure someone will jump in and
> correct my errors.
>
> I think the speed is really going to depend on the nature of your calls into
> the c library. From the code i've examined, there will probably be more
> overhead for the function call when using ctypes vs cython, but cython
> generated source code is somewhat less than maintainable and I would wager
> also less optimized than something you would write yourself.
>
> If your making lots of rapid calls to short running functions in the
> C-library, then you may start to feel the ctypes overhead.

That's my experience as well: ctypes overhead is quite noticeable. For
my own usage (wrapping C api), I think cython is better, and I stopped
using ctypes entirely for that usage. But ctypes still shines when you
want to try things with some C libraries, as you can do everything
from python (no cython/compile cycle).

David

From robertwb at math.washington.edu  Wed May 13 10:52:26 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 13 May 2009 01:52:26 -0700
Subject: [Cython] ctypes Vs Cython
In-Reply-To: <200905131017.07652.binet@cern.ch>
References: 
	
	<56FD54E9-515B-4D08-9650-96D8A7EE24AF@math.washington.edu>
	<200905131017.07652.binet@cern.ch>
Message-ID: 

On May 13, 2009, at 1:17 AM, Sebastien Binet wrote:

> hi,
>
> On Wednesday 13 May 2009 08:35:27 Robert Bradshaw wrote:
>> On May 12, 2009, at 9:47 PM, Mohamed Lrhazi wrote:
>>> On Wed, May 13, 2009 at 12:31 AM, Chris Colbert
>>>
>>>  wrote:
>>>> If your making lots of rapid calls to short running functions in  
>>>> the
>>>> C-library, then you may start to feel the ctypes overhead.
>>>
>>> That's what I was afraid to hear..
>>
>> Hopefully after using Cython a bit, you're fears will quickly go
>> away :).
>>
>>> What I was hoping to hear is "Oh
>>> no, ctypes is all C anyways, and will perform just the same as  
>>> Cython"
>>
>> A simple benchmark:
>>
>> import ctypes
>> libm = ctypes.cdll.LoadLibrary("libm.dylib") # platform dependent...
>> def ctypes_sum(N):
>>      lib_sqrt = libm.sqrt
>>      lib_sqrt.argtypes = (ctypes.c_double,)
>>      lib_sqrt.restype = ctypes.c_double
>>      s = 0
>>      for i in range(N):
>>          s += lib_sqrt(i)
>>      return s
>>
>> %cython
>> cdef extern from "math.h":
>>      double sqrt(double)
>>
>> def cython_sum(long N):
>>      cdef int i
>>      cdef double s=0
>>      for i in range(N):
>>          s += sqrt(i)
>>      return s
>>
>>>>> time ctypes_sum(10**6)
>>
>> 666666166.4588418
>> Time: CPU 1.13 s, Wall: 1.14 s
>>
>> time cython_sum(10**6)
>> 666666166.4588418
>> Time: CPU 0.03 s, Wall: 0.03 s
>
> interesting simple minded benchmark :)
>
> how would this translate into the pure-python mode ?

Pure Python:

def python_sum(N):
     from math import sqrt
     s = 0
     for i in range(N):
         s += sqrt(i)
     return s

 >>> time python_sum(10**6)
666666166.4588418
Time: CPU 0.47 s, Wall: 0.47 s

So for such a tiny call, ctypes is slower. (Since math.sqrt is a  
static wrapper around libm's sqrt, no surprise here.)

> (I couldn't seem to be
> able to declare the C-sqrt function using the pure-python mode of  
> cython:
> http://wiki.cython.org/pure wasn't helpful)

The pure-python mode should be exactly the same as the above, but  
there's no way to declare functions in it (yet).

- Robert



From robertwb at math.washington.edu  Wed May 13 10:53:28 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 13 May 2009 01:53:28 -0700
Subject: [Cython] ctypes Vs Cython
In-Reply-To: <200905131050.06546.binet@cern.ch>
References: 
	<200905131017.07652.binet@cern.ch>
	<30DDBB3F-7219-4506-B93C-BCC07650044B@math.washington.edu>
	<200905131050.06546.binet@cern.ch>
Message-ID: <1F4ECFF5-22E4-4712-8EC2-647F5B321A56@math.washington.edu>

On May 13, 2009, at 1:50 AM, Sebastien Binet wrote:

> On Wednesday 13 May 2009 10:41:43 Robert Bradshaw wrote:
>> On May 13, 2009, at 1:17 AM, Sebastien Binet wrote:
>>> hi,
>>>
>>> On Wednesday 13 May 2009 08:35:27 Robert Bradshaw wrote:
>>>> On May 12, 2009, at 9:47 PM, Mohamed Lrhazi wrote:
>>>>> On Wed, May 13, 2009 at 12:31 AM, Chris Colbert
>>>>>
>>>>>  wrote:
>>>>>> If your making lots of rapid calls to short running functions in
>>>>>> the
>>>>>> C-library, then you may start to feel the ctypes overhead.
>>>>>
>>>>> That's what I was afraid to hear..
>>>>
>>>> Hopefully after using Cython a bit, you're fears will quickly go
>>>> away :).
>>>>
>>>>> What I was hoping to hear is "Oh
>>>>> no, ctypes is all C anyways, and will perform just the same as
>>>>> Cython"
>>>>
>>>> A simple benchmark:
>>>>
>>>> import ctypes
>>>> libm = ctypes.cdll.LoadLibrary("libm.dylib") # platform  
>>>> dependent...
>>>> def ctypes_sum(N):
>>>>      lib_sqrt = libm.sqrt
>>>>      lib_sqrt.argtypes = (ctypes.c_double,)
>>>>      lib_sqrt.restype = ctypes.c_double
>>>>      s = 0
>>>>      for i in range(N):
>>>>          s += lib_sqrt(i)
>>>>      return s
>>>>
>>>> %cython
>>>> cdef extern from "math.h":
>>>>      double sqrt(double)
>>>>
>>>> def cython_sum(long N):
>>>>      cdef int i
>>>>      cdef double s=0
>>>>      for i in range(N):
>>>>          s += sqrt(i)
>>>>      return s
>>>>
>>>>>>> time ctypes_sum(10**6)
>>>>
>>>> 666666166.4588418
>>>> Time: CPU 1.13 s, Wall: 1.14 s
>>>>
>>>> time cython_sum(10**6)
>>>> 666666166.4588418
>>>> Time: CPU 0.03 s, Wall: 0.03 s
>>>
>>> interesting simple minded benchmark :)
>>>
>>> how would this translate into the pure-python mode ?
>>
>> Pure Python:
>>
>> def python_sum(N):
>>      from math import sqrt
>>      s = 0
>>      for i in range(N):
>>          s += sqrt(i)
>>      return s
>>
>>>>> time python_sum(10**6)
>>
>> 666666166.4588418
>> Time: CPU 0.47 s, Wall: 0.47 s
>>
>> So for such a tiny call, ctypes is slower. (Since math.sqrt is a
>> static wrapper around libm's sqrt, no surprise here.)
>>
>>> (I couldn't seem to be
>>> able to declare the C-sqrt function using the pure-python mode of
>>> cython:
>>> http://wiki.cython.org/pure wasn't helpful)
>>
>> The pure-python mode should be exactly the same as the above, but
>> there's no way to declare functions in it (yet).
> but if you pyximport import it, you should get the performance back  
> (assuming
> the function declaration were here), right ?

Yes, that's right.

There'd have to be some way of specifying "here's a C function, but  
in pure python mode, use this python function instead."

- Robert


From kxroberto at googlemail.com  Wed May 13 11:21:32 2009
From: kxroberto at googlemail.com (Robert)
Date: Wed, 13 May 2009 11:21:32 +0200
Subject: [Cython] pyximport ignors [build] compiler = mingw32
In-Reply-To: 
References: 			
	
Message-ID: 

Robert wrote:
> 
> this and more related cleanup, speedup of importer, --inplace 
> --verbose reload() .. here now:
> 
> http://trac.cython.org/cython_trac/ticket/312
> 

maybe sb likes to test it further:

* support for dynamic "reload()" added: e.g change of 
the Cython code without stopping & relaunching a big app. 
.reloadNN files may arise, as previously loaded files are 
blocked.


From kxroberto at googlemail.com  Wed May 13 16:46:32 2009
From: kxroberto at googlemail.com (Robert)
Date: Wed, 13 May 2009 16:46:32 +0200
Subject: [Cython] array.array and PIL directly ?
Message-ID: 

How to deal with Python's array.array directly - as with numpy.pxd 
is there a array.pxd ?

When not wanting the numpy overhead in an application, I use 
Python's array.array . And entering/exiting Cython I do around 
with .buffer_info() etc on Python level so far like:

assert pyarray.typecode=='f'  # single float array
cdef unsigned adr, length
adr,length=pyarray.buffer_info()
cdef float* a = adr
..
.__copy__()
.fromstring()
etc...

somewhat inelegant for frequent usage and with overhead.
Moving around automatic arrays intra Cython code (instead of bare 
pointers would be also more easy with auto refcounting ..

didn't find an example online so far. And also not the 
corresponding Python object in the Python C-API


Similar Question regarding a good way of using PIL images directly 
(without copying) ? Are there examples?


From dagss at student.matnat.uio.no  Wed May 13 17:18:55 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 13 May 2009 17:18:55 +0200
Subject: [Cython] array.array and PIL directly ?
In-Reply-To: 
References: 
Message-ID: <4A0AE4DF.20209@student.matnat.uio.no>

Robert wrote:
> How to deal with Python's array.array directly - as with numpy.pxd 
> is there a array.pxd ?

Which Python version?

Under Python 3 this should probably happen automatically, try:

cdef object[int, ndim=1, mode="c"] arr = yourarray

Not sure about Python 2.6+

For Python 2.5- an array.pxd must be written. It is not difficult, one 
simply follows the pattern in numpy.pxd (by implementing __getbuffer__ 
and filling in the Py_buffer struct).

If somebody ends up doing this, please submit it for inclusion in 
Python. Here's some docs:

http://wiki.cython.org/docs/specialmethods
http://wiki.cython.org/enhancements/buffer
http://docs.python.org/c-api/buffer.html

> Similar Question regarding a good way of using PIL images directly 
> (without copying) ? Are there examples?

I don't think so. I believe the story is the same (one must make an 
Image.pxd implementing __getbuffer__ on the behalf of PIL). However IIRC 
the "full" mode must be used (i.e. images are stored as arrays of 
pointers to arrays), so it is slightly more complicated to get the 
details rest.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Wed May 13 17:20:17 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 13 May 2009 17:20:17 +0200
Subject: [Cython] array.array and PIL directly ?
In-Reply-To: <4A0AE4DF.20209@student.matnat.uio.no>
References:  <4A0AE4DF.20209@student.matnat.uio.no>
Message-ID: <4A0AE531.9020203@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> If somebody ends up doing this, please submit it for inclusion in 
> Python. Here's some docs:

s/P/C/

-- 
Dag Sverre

From sccolbert at gmail.com  Wed May 13 17:42:01 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Wed, 13 May 2009 11:42:01 -0400
Subject: [Cython] creating for-loops with no gil?
Message-ID: <7f014ea60905130842m4147901aw777663d1b6be583d@mail.gmail.com>

I'm trying to test writing extensions that release the gil. This simple
example is failing cython compilation with errors stating that the count()
function requires the gil.

Can anyone point me in the right direction?

Thanks,

Chris


##### code ##########

cdef double count(int n) nogil:
    cdef int i, j, k
    cdef double out
    for i in range(n):
        for j in range(n):
            out = 0
            for k in range(n):
                out += k

    return out


def myinterface():
    cdef int n = 10000000
    cdef double out
    out = count(n)
    return out
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090513/09c1484a/attachment.htm 

From dagss at student.matnat.uio.no  Wed May 13 17:48:48 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 13 May 2009 17:48:48 +0200
Subject: [Cython] creating for-loops with no gil?
In-Reply-To: <7f014ea60905130842m4147901aw777663d1b6be583d@mail.gmail.com>
References: <7f014ea60905130842m4147901aw777663d1b6be583d@mail.gmail.com>
Message-ID: <4A0AEBE0.2000401@student.matnat.uio.no>

Chris Colbert wrote:
> I'm trying to test writing extensions that release the gil. This simple 
> example is failing cython compilation with errors stating that the 
> count() function requires the gil.
> 
> Can anyone point me in the right direction?

http://trac.cython.org/cython_trac/ticket/205

A fix is scheduled at "not before the 0.12 release".

Until then, you must use the

for i from 0 <= i < n:
     ...

syntax.

-- 
Dag Sverre

From dalcinl at gmail.com  Wed May 13 17:52:03 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 13 May 2009 12:52:03 -0300
Subject: [Cython] Win32 calling conventions, second round
Message-ID: 

I've partially implemented the @cython.callspec("something") stuff. It
actually solves my particular needs, but it not all the possible
needs. The problem is that the calling convention spec is part of the
type of the function, and a general solution should handle the code
below:

@cython.callspec("something")
cdef void myfunc(): pass

@cython.callspec("something")
cdef void (*p)()

p = myfunc

But then the decoration on the function pointer "p" IMHO deviates too
much for the normal Python meaning of decorators...


The only possibility I can imagine up to now is something like:

cdef void callspec("something") myfunc(): pass

cdef void (callspec("something") *p)()

p = myfunc


Of course, this approach requires modification to the parser, and the
syntax could be controversial. However, the addition of callspec() as
a declarator modifier (like inline or extern) taking a single arg
would remove all the ambiguities of my original proposal of using a
bare string literal...

What do you think? We could even use a more obfuscated
__callspec("something") ...



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From sccolbert at gmail.com  Wed May 13 17:58:54 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Wed, 13 May 2009 11:58:54 -0400
Subject: [Cython] creating for-loops with no gil?
In-Reply-To: <4A0AEBE0.2000401@student.matnat.uio.no>
References: <7f014ea60905130842m4147901aw777663d1b6be583d@mail.gmail.com>
	<4A0AEBE0.2000401@student.matnat.uio.no>
Message-ID: <7f014ea60905130858h71d70910qa8e1bf9132f84624@mail.gmail.com>

thanks

On Wed, May 13, 2009 at 11:48 AM, Dag Sverre Seljebotn <
dagss at student.matnat.uio.no> wrote:

> Chris Colbert wrote:
> > I'm trying to test writing extensions that release the gil. This simple
> > example is failing cython compilation with errors stating that the
> > count() function requires the gil.
> >
> > Can anyone point me in the right direction?
>
> http://trac.cython.org/cython_trac/ticket/205
>
> A fix is scheduled at "not before the 0.12 release".
>
> Until then, you must use the
>
> for i from 0 <= i < n:
>     ...
>
> syntax.
>
> --
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090513/fccbf580/attachment.htm 

From dagss at student.matnat.uio.no  Wed May 13 18:06:51 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 13 May 2009 18:06:51 +0200
Subject: [Cython] Win32 calling conventions, second round
In-Reply-To: 
References: 
Message-ID: <4A0AF01B.6070106@student.matnat.uio.no>

Lisandro Dalcin wrote:
> I've partially implemented the @cython.callspec("something") stuff. It
> actually solves my particular needs, but it not all the possible
> needs. The problem is that the calling convention spec is part of the
> type of the function, and a general solution should handle the code
> below:
> 
> @cython.callspec("something")
> cdef void myfunc(): pass
> 
> @cython.callspec("something")
> cdef void (*p)()
> 
> p = myfunc
> 
> But then the decoration on the function pointer "p" IMHO deviates too
> much for the normal Python meaning of decorators...
> 
> 
> The only possibility I can imagine up to now is something like:
> 
> cdef void callspec("something") myfunc(): pass
> 
> cdef void (callspec("something") *p)()
> 
> p = myfunc

+1 from me (if the decorator is also available). As long as I'm not the 
one who has to make the parser accept it :-)

-- 
Dag Sverre

From sccolbert at gmail.com  Wed May 13 18:09:34 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Wed, 13 May 2009 12:09:34 -0400
Subject: [Cython] creating for-loops with no gil?
In-Reply-To: <7f014ea60905130858h71d70910qa8e1bf9132f84624@mail.gmail.com>
References: <7f014ea60905130842m4147901aw777663d1b6be583d@mail.gmail.com>
	<4A0AEBE0.2000401@student.matnat.uio.no>
	<7f014ea60905130858h71d70910qa8e1bf9132f84624@mail.gmail.com>
Message-ID: <7f014ea60905130909vb1f6513rbfa0d417c122787d@mail.gmail.com>

alright i made the change and here is the code:

cdef double count(int n) nogil:
    cdef int i, j, k
    cdef double out
    for i from 0 <= i < n:
        for j from 0 <= j < n:
            out = 0
            for k from 0 <= k < n:
                out += k

    return out


def myinterface():
    cdef int n = 10000000
    cdef double out
    out = count(n)
    return out


this compiles but when i test with this python script I hang on the function
call:

import threading
from giltest import myinterface

class NoGil(threading.Thread):

    def __init__(self):
        super(NoGil, self).__init__()

    def run(self):
        a = myinterface()
        print 'non gil', a


class Gil():

    def run(self):
        print 'gil thread'

if __name__=='__main__':
    a = NoGil()
    b = Gil()
    a.start()
    b.run()
    raw_input()



i presume this is because myinterface() doesn't release the gil, and thus
waits for count() to return.

How do I work this so that I can run a c-function outside the gil and thus
make use of the multi-cores on my machine?

Thanks!

Chris


On Wed, May 13, 2009 at 11:58 AM, Chris Colbert  wrote:

> thanks
>
>
> On Wed, May 13, 2009 at 11:48 AM, Dag Sverre Seljebotn <
> dagss at student.matnat.uio.no> wrote:
>
>> Chris Colbert wrote:
>> > I'm trying to test writing extensions that release the gil. This simple
>> > example is failing cython compilation with errors stating that the
>> > count() function requires the gil.
>> >
>> > Can anyone point me in the right direction?
>>
>> http://trac.cython.org/cython_trac/ticket/205
>>
>> A fix is scheduled at "not before the 0.12 release".
>>
>> Until then, you must use the
>>
>> for i from 0 <= i < n:
>>     ...
>>
>> syntax.
>>
>> --
>> Dag Sverre
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090513/5b8a4800/attachment.htm 

From dagss at student.matnat.uio.no  Wed May 13 18:16:59 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 13 May 2009 18:16:59 +0200
Subject: [Cython] creating for-loops with no gil?
In-Reply-To: <7f014ea60905130909vb1f6513rbfa0d417c122787d@mail.gmail.com>
References: <7f014ea60905130842m4147901aw777663d1b6be583d@mail.gmail.com>	<4A0AEBE0.2000401@student.matnat.uio.no>	<7f014ea60905130858h71d70910qa8e1bf9132f84624@mail.gmail.com>
	<7f014ea60905130909vb1f6513rbfa0d417c122787d@mail.gmail.com>
Message-ID: <4A0AF27B.4030400@student.matnat.uio.no>

Chris Colbert wrote:
> alright i made the change and here is the code:
> 
> cdef double count(int n) nogil:
>     cdef int i, j, k
>     cdef double out
>     for i from 0 <= i < n:
>         for j from 0 <= j < n:
>             out = 0
>             for k from 0 <= k < n:
>                 out += k
>    
>     return out
>                
> 
> def myinterface():
>     cdef int n = 10000000
>     cdef double out
>     out = count(n)
>     return out

> i presume this is because myinterface() doesn't release the gil, and 
> thus waits for count() to return.
> 
> How do I work this so that I can run a c-function outside the gil and 
> thus make use of the multi-cores on my machine?

When you do

cdef double count(int n) nogil

it merely declared that the function /can/ be run without the GIL, but 
it doesn't explicitly release it. Do this:

def myinterface():
     ...
     with nogil:
         out = count(n)
     return out

--
Dag Sverre

From sccolbert at gmail.com  Wed May 13 18:20:04 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Wed, 13 May 2009 12:20:04 -0400
Subject: [Cython] creating for-loops with no gil?
In-Reply-To: <4A0AF27B.4030400@student.matnat.uio.no>
References: <7f014ea60905130842m4147901aw777663d1b6be583d@mail.gmail.com>
	<4A0AEBE0.2000401@student.matnat.uio.no>
	<7f014ea60905130858h71d70910qa8e1bf9132f84624@mail.gmail.com>
	<7f014ea60905130909vb1f6513rbfa0d417c122787d@mail.gmail.com>
	<4A0AF27B.4030400@student.matnat.uio.no>
Message-ID: <7f014ea60905130920k700e383dycbde8b3bc515345@mail.gmail.com>

Excellent! that did the trick!

Thanks Dag!

Chris

On Wed, May 13, 2009 at 12:16 PM, Dag Sverre Seljebotn <
dagss at student.matnat.uio.no> wrote:

> Chris Colbert wrote:
> > alright i made the change and here is the code:
> >
> > cdef double count(int n) nogil:
> >     cdef int i, j, k
> >     cdef double out
> >     for i from 0 <= i < n:
> >         for j from 0 <= j < n:
> >             out = 0
> >             for k from 0 <= k < n:
> >                 out += k
> >
> >     return out
> >
> >
> > def myinterface():
> >     cdef int n = 10000000
> >     cdef double out
> >     out = count(n)
> >     return out
> 
> > i presume this is because myinterface() doesn't release the gil, and
> > thus waits for count() to return.
> >
> > How do I work this so that I can run a c-function outside the gil and
> > thus make use of the multi-cores on my machine?
>
> When you do
>
> cdef double count(int n) nogil
>
> it merely declared that the function /can/ be run without the GIL, but
> it doesn't explicitly release it. Do this:
>
> def myinterface():
>     ...
>     with nogil:
>          out = count(n)
>     return out
>
> --
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090513/9dff1209/attachment-0001.htm 

From dalcinl at gmail.com  Wed May 13 18:51:34 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 13 May 2009 13:51:34 -0300
Subject: [Cython] Win32 calling conventions, second round
In-Reply-To: <4A0AF01B.6070106@student.matnat.uio.no>
References: 
	<4A0AF01B.6070106@student.matnat.uio.no>
Message-ID: 

On Wed, May 13, 2009 at 1:06 PM, Dag Sverre Seljebotn
 wrote:
> Lisandro Dalcin wrote:
>>
>> The only possibility I can imagine up to now is something like:
>>
>> cdef void callspec("something") myfunc(): pass
>>
>> cdef void (callspec("something") *p)()
>>
>> p = myfunc
>
> +1 from me (if the decorator is also available).

Then, what should happen with this code

@cython.callspec("foo")
cdef void callspec("bar") myfunc(): pass

It seems it should fail, right ? However, perhaps it should pass if
the callspec's are the same ?


> As long as I'm not the
> one who has to make the parser accept it :-)
>

In general, the problem is not to make the Cython parser accept and
and do the right thing with VALID syntax... the hard part is to make
it fail nicely (I mean, inform about the errors and not crashing) for
INVALID syntax...



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Wed May 13 19:06:57 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 13 May 2009 19:06:57 +0200 (CEST)
Subject: [Cython] Win32 calling conventions, second round
In-Reply-To: 
References: 
	<4A0AF01B.6070106@student.matnat.uio.no>
	
Message-ID: <204d08c125894ade6f30521798a9339d.squirrel@webmail.uio.no>

Lisandro wrote:
> On Wed, May 13, 2009 at 1:06 PM, Dag Sverre Seljebotn
>  wrote:
>> Lisandro Dalcin wrote:
>>>
>>> The only possibility I can imagine up to now is something like:
>>>
>>> cdef void callspec("something") myfunc(): pass
>>>
>>> cdef void (callspec("something") *p)()
>>>
>>> p = myfunc
>>
>> +1 from me (if the decorator is also available).
>
> Then, what should happen with this code
>
> @cython.callspec("foo")
> cdef void callspec("bar") myfunc(): pass
>
> It seems it should fail, right ? However, perhaps it should pass if
> the callspec's are the same ?

Either is fine with me.

> In general, the problem is not to make the Cython parser accept and
> and do the right thing with VALID syntax... the hard part is to make
> it fail nicely (I mean, inform about the errors and not crashing) for
> INVALID syntax...

Yes :-)

BTW if you refer to the situation above, I believe one should pass on both
declarations from the parser and raise an error in PostParse? That way one
doesn't need to change the parser at all for the decorator.

Dag Sverre


From sccolbert at gmail.com  Wed May 13 19:30:06 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Wed, 13 May 2009 13:30:06 -0400
Subject: [Cython] ctypes Vs Cython
In-Reply-To: <1F4ECFF5-22E4-4712-8EC2-647F5B321A56@math.washington.edu>
References: 
	<200905131017.07652.binet@cern.ch>
	<30DDBB3F-7219-4506-B93C-BCC07650044B@math.washington.edu>
	<200905131050.06546.binet@cern.ch>
	<1F4ECFF5-22E4-4712-8EC2-647F5B321A56@math.washington.edu>
Message-ID: <7f014ea60905131030n136ec8a9m54014cf79e238842@mail.gmail.com>

I ran another benchmark showing the opposite case: a single call to a long
running c function. The results are surprising, Cython is much much faster.
(this may be do my inability to compile an optimized dll).

Anyway, the following is the code I tested:


##### Cython script (.pyx) #########

cdef double count(int n) nogil:
    cdef int i, j, k
    cdef double out
    for i from 0 <= i < n:
        for j from 0 <= j < n:
            out = 0
            for k from 0 <= k < n:
                out += k

    return out

def myinterface(int a):
    cdef int n = a
    cdef double out
    with nogil:
        out = count(n)
    return out

####### Ctypes scripts (pure .c) #########

#include "ctypesgiltest.h"

EXPORT double count(int n){
    int i, j, k;
    int a, b, c;
    double out;

    a = n;
    for (i=0; i>>
Starting Threads
>>> no GIL cython:  Value: 1999000.000000 Time: 8.076570 s
no GIL ctypes:  Value: 1999000.000000 Time: 38.927630 s


That's a BIG advantage to Cython! Kudos to the developers.

Looking into the Cython generated code, the two c files are virtually
identical. I assume the performance is down to compiler optimization.

Using minGW-32, I issued the following commands to build the DLL for ctypes:

>gcc -c -DBUILD_DLL ctypesgiltest.c
>gcc -shared -o ctypesgiltest.dll ctypesgiltest.o




Chris




On Wed, May 13, 2009 at 4:53 AM, Robert Bradshaw <
robertwb at math.washington.edu> wrote:

> On May 13, 2009, at 1:50 AM, Sebastien Binet wrote:
>
> > On Wednesday 13 May 2009 10:41:43 Robert Bradshaw wrote:
> >> On May 13, 2009, at 1:17 AM, Sebastien Binet wrote:
> >>> hi,
> >>>
> >>> On Wednesday 13 May 2009 08:35:27 Robert Bradshaw wrote:
> >>>> On May 12, 2009, at 9:47 PM, Mohamed Lrhazi wrote:
> >>>>> On Wed, May 13, 2009 at 12:31 AM, Chris Colbert
> >>>>>
> >>>>>  wrote:
> >>>>>> If your making lots of rapid calls to short running functions in
> >>>>>> the
> >>>>>> C-library, then you may start to feel the ctypes overhead.
> >>>>>
> >>>>> That's what I was afraid to hear..
> >>>>
> >>>> Hopefully after using Cython a bit, you're fears will quickly go
> >>>> away :).
> >>>>
> >>>>> What I was hoping to hear is "Oh
> >>>>> no, ctypes is all C anyways, and will perform just the same as
> >>>>> Cython"
> >>>>
> >>>> A simple benchmark:
> >>>>
> >>>> import ctypes
> >>>> libm = ctypes.cdll.LoadLibrary("libm.dylib") # platform
> >>>> dependent...
> >>>> def ctypes_sum(N):
> >>>>      lib_sqrt = libm.sqrt
> >>>>      lib_sqrt.argtypes = (ctypes.c_double,)
> >>>>      lib_sqrt.restype = ctypes.c_double
> >>>>      s = 0
> >>>>      for i in range(N):
> >>>>          s += lib_sqrt(i)
> >>>>      return s
> >>>>
> >>>> %cython
> >>>> cdef extern from "math.h":
> >>>>      double sqrt(double)
> >>>>
> >>>> def cython_sum(long N):
> >>>>      cdef int i
> >>>>      cdef double s=0
> >>>>      for i in range(N):
> >>>>          s += sqrt(i)
> >>>>      return s
> >>>>
> >>>>>>> time ctypes_sum(10**6)
> >>>>
> >>>> 666666166.4588418
> >>>> Time: CPU 1.13 s, Wall: 1.14 s
> >>>>
> >>>> time cython_sum(10**6)
> >>>> 666666166.4588418
> >>>> Time: CPU 0.03 s, Wall: 0.03 s
> >>>
> >>> interesting simple minded benchmark :)
> >>>
> >>> how would this translate into the pure-python mode ?
> >>
> >> Pure Python:
> >>
> >> def python_sum(N):
> >>      from math import sqrt
> >>      s = 0
> >>      for i in range(N):
> >>          s += sqrt(i)
> >>      return s
> >>
> >>>>> time python_sum(10**6)
> >>
> >> 666666166.4588418
> >> Time: CPU 0.47 s, Wall: 0.47 s
> >>
> >> So for such a tiny call, ctypes is slower. (Since math.sqrt is a
> >> static wrapper around libm's sqrt, no surprise here.)
> >>
> >>> (I couldn't seem to be
> >>> able to declare the C-sqrt function using the pure-python mode of
> >>> cython:
> >>> http://wiki.cython.org/pure wasn't helpful)
> >>
> >> The pure-python mode should be exactly the same as the above, but
> >> there's no way to declare functions in it (yet).
> > but if you pyximport import it, you should get the performance back
> > (assuming
> > the function declaration were here), right ?
>
> Yes, that's right.
>
> There'd have to be some way of specifying "here's a C function, but
> in pure python mode, use this python function instead."
>
> - Robert
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090513/6dfad689/attachment.htm 

From ilmarw at simula.no  Wed May 13 19:44:36 2009
From: ilmarw at simula.no (Ilmar Wilbers)
Date: Wed, 13 May 2009 19:44:36 +0200
Subject: [Cython] Paper about Cython
Message-ID: <4A0B0704.1010205@simula.no>

Hi list,

I wanted to make you aware of a paper[1] that we have written at Simula 
Research Laboratory in Oslo, Norway. The paper will be presented at 
MekIT 09 - Conference on Computational Mechanics in Trondheim in a 
couple of weeks. In this paper we compare Cython with similar packages 
with regard to execution time.

It must be noted that the time for the arithmetic mean for Cython has 
decreased since the paper was accepted, as Dag Sverre pointed me to the 
additional argument 'mode' for the arrays. This reduces the time for 
Cython from 1.6 to 1.3 compared to the pure Fortran implementation. I 
apologize for any mistakes concerning Cython in the paper, as I'm sure 
the developers have some comments about stuff I've missed.

We will be following the development of Cython further (in particular 
the Fortran integration) thanks for an excellent package so far!

Ilmar Wilbers
Simula Research Laboratory

[1]: 
http://simula.no/research/scientific/publications/Simula.SC.578/simula_pdf_file

From roland at utk.edu  Wed May 13 19:50:28 2009
From: roland at utk.edu (Roland Schulz)
Date: Wed, 13 May 2009 13:50:28 -0400
Subject: [Cython] Project: Buffer arithmetic operations
In-Reply-To: <4A0A73D2.6070105@student.matnat.uio.no>
References: <4db580fd0905121314j142a30c7s915e4eb59d772b71@mail.gmail.com> 
	<4A0A73D2.6070105@student.matnat.uio.no>
Message-ID: 

On Wed, May 13, 2009 at 3:16 AM, Dag Sverre Seljebotn <
dagss at student.matnat.uio.no> wrote:

> > I'm planning on making time this weekend to attend part of the
> > upcoming sage days, and I definitely want to participate in the coding
> > sprint and get my hands dirty with some cython development.
> >
> > What I'd like to work on, which would be quite relevant in my research
> > code, is to write an optimization routine that unrolls numpy array
> > expressions involving slices, array operators like +-/*, and scalar
> > values into a set of nested loops that avoids the use of any temporary
> > array copies.  Thus something like:
> >
> > A[0, a:b] = B[0, a:b]*C[a:2*b:2] + D + n
> >
> > would get turned into some code that checked broadcasting requirements
> > and then into a loop that ran from a to b and looped over the
> > appropriate vertices.   Thus we would basically replicate what blitz++
> > does with templates or fortran 90 does (so I've heard).  I've sketched
> > out an approach to actually coding it up and it seems doable for a
> > sage days coding project.


I think this would be very very useful. But it seems reinventing the wheel
to implement the actual vector math. It is not that trivial. Just as one
exmaple: sometimes intermediate buffers are faster so you need to have a
heuristic to decide that.
Why not using a C++ template library for that? One could use Blitz++ but I
think Eigen (eigen.tuxfamily.org) is newer and significant faster (e.g.
automatically uses SSE). Since everything is in a include files no external
libary requirement is added.


> A) There's a clash with existing semantics to be resolved. For instance
> if you have a NumPy "matrix" instance, then * does matrix multiplication
> instead, and just implementing this now would break code.
>
> http://wiki.cython.org/enhancements/buffersyntax
> http://thread.gmane.org/gmane.comp.python.numeric.general/28439
>
> contains my thoughts on how this could be resolved. I'll use this syntax
> below.


I think this could solved by allowing to define custom mapping of
operators/methods of python types/classes to C functions. With that one
could define that for Numpy ndarray the element wise multiplication is used
and for NumPy matrix the matrix multiplication is used. And both could be
translated into calls of Eigen or some other vector library.

This would allow to directly transform normal NumPy, which tries to avoid
using loops, into highly optimized C with SSE without any intermediate
buffers.

Roland


-- 
ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
865-241-1537, ORNL PO BOX 2008 MS6309
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090513/e0be2682/attachment.htm 

From robertwb at math.washington.edu  Wed May 13 20:08:16 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 13 May 2009 11:08:16 -0700
Subject: [Cython] Win32 calling conventions, second round
In-Reply-To: 
References: 
Message-ID: <8C4044F3-BEB5-4811-AA76-1606E9762429@math.washington.edu>

On May 13, 2009, at 8:52 AM, Lisandro Dalcin wrote:

> I've partially implemented the @cython.callspec("something") stuff. It
> actually solves my particular needs, but it not all the possible
> needs. The problem is that the calling convention spec is part of the
> type of the function, and a general solution should handle the code
> below:
>
> @cython.callspec("something")
> cdef void myfunc(): pass
>
> @cython.callspec("something")
> cdef void (*p)()
>
> p = myfunc
>
> But then the decoration on the function pointer "p" IMHO deviates too
> much for the normal Python meaning of decorators...

How so? A decorator takes an object and returns a new object (in this  
case, with a new type).

> The only possibility I can imagine up to now is something like:
>
> cdef void callspec("something") myfunc(): pass
>
> cdef void (callspec("something") *p)()
>
> p = myfunc
>
>
> Of course, this approach requires modification to the parser, and the
> syntax could be controversial. However, the addition of callspec() as
> a declarator modifier (like inline or extern) taking a single arg
> would remove all the ambiguities of my original proposal of using a
> bare string literal...

I like this better than the original proposal, but still prefer the  
decorator approach.

- Robert



From dagss at student.matnat.uio.no  Wed May 13 20:34:21 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 13 May 2009 20:34:21 +0200
Subject: [Cython] Project: Buffer arithmetic operations
In-Reply-To: 
References: <4db580fd0905121314j142a30c7s915e4eb59d772b71@mail.gmail.com>
	<4A0A73D2.6070105@student.matnat.uio.no>
	
Message-ID: <4A0B12AD.2060301@student.matnat.uio.no>

Roland Schulz wrote:
> 
> 
> On Wed, May 13, 2009 at 3:16 AM, Dag Sverre Seljebotn 
> > wrote:
> 
>      > I'm planning on making time this weekend to attend part of the
>      > upcoming sage days, and I definitely want to participate in the
>     coding
>      > sprint and get my hands dirty with some cython development.
>      >
>      > What I'd like to work on, which would be quite relevant in my
>     research
>      > code, is to write an optimization routine that unrolls numpy array
>      > expressions involving slices, array operators like +-/*, and scalar
>      > values into a set of nested loops that avoids the use of any
>     temporary
>      > array copies.  Thus something like:
>      >
>      > A[0, a:b] = B[0, a:b]*C[a:2*b:2] + D + n
>      >
>      > would get turned into some code that checked broadcasting
>     requirements
>      > and then into a loop that ran from a to b and looped over the
>      > appropriate vertices.   Thus we would basically replicate what
>     blitz++
>      > does with templates or fortran 90 does (so I've heard).  I've
>     sketched
>      > out an approach to actually coding it up and it seems doable for a
>      > sage days coding project.
> 
> 
> I think this would be very very useful. But it seems reinventing the 
> wheel to implement the actual vector math. It is not that trivial. Just 
> as one exmaple: sometimes intermediate buffers are faster so you need to 
> have a heuristic to decide that.
> Why not using a C++ template library for that? One could use Blitz++ but 
> I think Eigen (eigen.tuxfamily.org ) is 
> newer and significant faster (e.g. automatically uses SSE). Since 
> everything is in a include files no external libary requirement is added.

Thanks for pitching in; that's some very interesting thoughts!

Here's my immediate reaction:
  - We should avoid reinventing the wheel at all costs. Heuristics for 
intermediate buffers in Cython is probably a no-go.
  - Still, relying explicitly on one specific C++ library makes me uneasy
  - So perhaps what is needed here is some kind of plugin system.

Getting the basics working (just unoptimized, "what you would have 
written in C") is more trivial, and would get us up fast and allow 
operation without an external library.

Then, if there was a good protocol for it, one would ideally be able to 
easily add support for other backends as well. (Which would all be 
required to have the same result, they would just differ in 
implementation detail.)

On one hand, one shouldn't over-design, while on the other hand, a 
vector math library is a fairly sophisticated thing and one could argue 
that those should be made swappable if possible.

>     A) There's a clash with existing semantics to be resolved. For instance
>     if you have a NumPy "matrix" instance, then * does matrix multiplication
>     instead, and just implementing this now would break code.
> 
>     http://wiki.cython.org/enhancements/buffersyntax
>     http://thread.gmane.org/gmane.comp.python.numeric.general/28439
> 
>     contains my thoughts on how this could be resolved. I'll use this syntax
>     below.
> 
> 
> I think this could solved by allowing to define custom mapping of 
> operators/methods of python types/classes to C functions. With that one 
> could define that for Numpy ndarray the element wise multiplication is 
> used and for NumPy matrix the matrix multiplication is used. And both 
> could be translated into calls of Eigen or some other vector library.
> 
> This would allow to directly transform normal NumPy, which tries to 
> avoid using loops, into highly optimized C with SSE without any 
> intermediate buffers.

Well, if you write

   def f(np.ndarray[int] arr):
     print arr * arr

   f(np.matrix(...)) # subclass, so allowed

then the Python version of that would be np.dot but the Cython version 
np.multiply.

The problem is that we really need to know the behaviour compile-time. 
Should we disallow subclasses when a buffer is declared perhaps?

Another perspective is that it is nice to be able to have a consistent 
way of working with any Python 3 buffer. This advantage is not obvious 
yet as there are very few such buffers, but I think in time Python might 
contain more and more of what NumPy does now (i.e. the functions in the 
math module might become "ufuncs" able to operate on any PEP 3118 buffer 
and so on).

But I'm definitely not certain here; more input welcome!

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Wed May 13 20:57:27 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 13 May 2009 20:57:27 +0200
Subject: [Cython] Project: Buffer arithmetic operations
In-Reply-To: <4A0B12AD.2060301@student.matnat.uio.no>
References: <4db580fd0905121314j142a30c7s915e4eb59d772b71@mail.gmail.com>	<4A0A73D2.6070105@student.matnat.uio.no>	
	<4A0B12AD.2060301@student.matnat.uio.no>
Message-ID: <4A0B1817.1020206@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Roland Schulz wrote:
>>
>> On Wed, May 13, 2009 at 3:16 AM, Dag Sverre Seljebotn 
>>     A) There's a clash with existing semantics to be resolved. For instance
>>     if you have a NumPy "matrix" instance, then * does matrix multiplication
>>     instead, and just implementing this now would break code.
>>
>>     http://wiki.cython.org/enhancements/buffersyntax
>>     http://thread.gmane.org/gmane.comp.python.numeric.general/28439
>>
>>     contains my thoughts on how this could be resolved. I'll use this syntax
>>     below.
>>
>>
>> I think this could solved by allowing to define custom mapping of 
>> operators/methods of python types/classes to C functions. With that one 
>> could define that for Numpy ndarray the element wise multiplication is 
>> used and for NumPy matrix the matrix multiplication is used. And both 
>> could be translated into calls of Eigen or some other vector library.
>>
>> This would allow to directly transform normal NumPy, which tries to 
>> avoid using loops, into highly optimized C with SSE without any 
>> intermediate buffers.
> 

> Another perspective is that it is nice to be able to have a consistent 
> way of working with any Python 3 buffer. This advantage is not obvious 
> yet as there are very few such buffers, but I think in time Python might 
> contain more and more of what NumPy does now (i.e. the functions in the 
> math module might become "ufuncs" able to operate on any PEP 3118 buffer 
> and so on).

Note that one could implement this in "shadow module" mode as well. I.e.

@cython.locals(arr=cython.int[:])
def f(arr):
     return arr * arr + 2

could be made to work in pure Python mode by coercing the argument to an 
appropriate NumPy subclass emulating the explicit buffers.

So I think it is primarily about what kind of syntax it is natural to 
work with etc; the one "transparently" optimizing some parts of what one 
is used to writing in Python, or the one shouting "syntax error" 
whenever you do something which is not an optimized, fast operation.

I find I now (after nearly a year) wish I had gone for the latter 
instead :-) as users are still caught by surprise that e.g. "arr[2][3]" 
isn't as fast as "arr[2,3]", but there's no compiler complaints, and so on.

-- 
Dag Sverre

From kwmsmith at gmail.com  Wed May 13 21:34:11 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Wed, 13 May 2009 14:34:11 -0500
Subject: [Cython] Project: Buffer arithmetic operations
In-Reply-To: <4A0A73D2.6070105@student.matnat.uio.no>
References: <4db580fd0905121314j142a30c7s915e4eb59d772b71@mail.gmail.com>
	<4A0A73D2.6070105@student.matnat.uio.no>
Message-ID: 

On Wed, May 13, 2009 at 2:16 AM, Dag Sverre Seljebotn
 wrote:
> Hoyt Koepke wrote:
>> Hello Robert & Dag,
>>
>> I'm sending this to you guys first; feel free to bump it to the whole
>> list if you think it's appropriate.
>
> Doing so; I think at least it should be available to Kurt as well.

Thanks -- glad to see more interest in improving numerics in Cython.
I don't have much to comment on other than a small pot-shot below :-)

>
>> I'm planning on making time this weekend to attend part of the
>> upcoming sage days, and I definitely want to participate in the coding
>> sprint and get my hands dirty with some cython development.
>>
>> What I'd like to work on, which would be quite relevant in my research
>> code, is to write an optimization routine that unrolls numpy array
>> expressions involving slices, array operators like +-/*, and scalar
>> values into a set of nested loops that avoids the use of any temporary
>> array copies. ?Thus something like:
>>
>> A[0, a:b] = B[0, a:b]*C[a:2*b:2] + D + n
>>
>> would get turned into some code that checked broadcasting requirements
>> and then into a loop that ran from a to b and looped over the
>> appropriate vertices. ? Thus we would basically replicate what blitz++
>> does with templates or fortran 90 does (so I've heard). ?I've sketched
>> out an approach to actually coding it up and it seems doable for a
>> sage days coding project.
>
> Great that you want to do this :-)
>
> I think doing it for a Sage days project is a bit ambitious, but you
> might be able to get a good start on it. It also depends on what route
> is taken, see below.
>
>> I guess how I see this coming out then is that I'd create a routine
>> that takes the cython AST for this expression and a few configuration
>> parameters and returns the code for an inline function that accepts as
>> parameters all the needed buffer parameters (data ptr, strides, shape,
>> and slice parameters if there are some needed (e.g. a, b above),
>> constants, etc. ), ?and does the appropriate looping and copying.
>> Part of the reason I want to do the inline function approach is that I
>> want this code to be minimally wrapped up in cython's internals, as a
>> tested routine to generate such code might be more generally
>> applicable. ?Though if there's a stronger argument in favor of the
>> other way, that's okay too.
>
> I have given this a lot of thought as well. My thoughts diverge from
> yours in that I want to improve Cython numerics in lot of areas, and
> that the above would come more as a natural consequence than the first
> step. What I'll scetch below is thus a more "patient" route.
>
> My experience is that I'm not sure how good results one can get if one
> is set from the beginning on getting tangible results during a
> conference. A concrete goal like the above seems great for things like a
> GSoC going over one or two months, but perhaps not a week -- the danger
> is that one skips important parts of the design etc. just in order to
> get some example working, but in the end nothing but the example
> works.... (I hope I don't kill your enthusiasm off!, I just want the
> perspective noted :-) ).
>
> Also I should not that I don't have much time the rest of this week to
> be involved in designing this.
>
> Anyway, here's my thoughts. It's pretty much designed to give the
> feeling of "oh no not all this, I just want to do something simple and
> get this done" -- that's OK, just read it and write your rejoinder :-)
>
> A) There's a clash with existing semantics to be resolved. For instance
> if you have a NumPy "matrix" instance, then * does matrix multiplication
> instead, and just implementing this now would break code.
>
> http://wiki.cython.org/enhancements/buffersyntax
> http://thread.gmane.org/gmane.comp.python.numeric.general/28439
>
> contains my thoughts on how this could be resolved. I'll use this syntax
> below.
>
> B) I think the feature could be broken down in smaller parts:
>
> 1) Efficient slicing (#178):
>
> cdef int[:,:] a
> cdef int[:] b = a[3,:]
>
> perhaps also broadcasting?
>
> cdef int[:,:,:] c = a # ?
>
> 2) Efficient ndenumerate for-loop construct. This is the "engine" of
> what you want to achieve.
>
> from cython import ndenumerate
> cdef int[:,:,:] a = ...
> cdef int[:,:,:,:] b = ...
>
> # a[...] = 2 * b[3,...]
> for idx, value in ndenumerate(a, b[3,...]):
> ? ? a[idx] = 2 * b[3, idx]
>
> Or, perhaps
>
> for a_item, b_item in ndenumerate(a, b[3,...]):
> ? ? a_item.set(2*b_item.get())
>
> or similar. The code making sure that the dimension with the smallest
> strides is iterated innermost would be generated here (likely one would
> have a while loop around a for loop; no matter how many dimensions there
> are; see also NumPy's ND-generic iterators... we could also allow this
> to work for "cdef int[...] a", i.e. no declared number of dimensions).
>
> 3) Transform "a[...] = 2 * b[3, ...]" into the construct in 2). This is
> easily done using a transform once 2) is done.
>
> 3) Ufuncs. I was imagining it like this:
>
> cdef int multiply(int x, int y): return x * y
>
> ...
> cdef int[:,:] a = ..., b = ..
> a[...] = multiply(a, b) # Calls for each item!
>
> So basically "every function with primitive arguments is an ufunc".
> (Fortran has a similar concept, but you need to declare that the
> function has no side-effects -- I say let's just make N invocations part
> of the spec of how this works.)
>
> Obviously this is very easily implemented once 2) is in place.

The above strikes me as a bit too much magic under the hood -- perhaps
I can get used to it.  My initial preference would be to have some
indication that this is a 'ufuncable' function, or is intended to be
so; Cython could then check to be sure the dummy arguments & return
type are primitive.  (Although I reckon this check would be done
during compilation once someone tries to pass a numpy array to a cdef
function with primitive-type arguments.)

Kurt

From kwmsmith at gmail.com  Wed May 13 21:41:34 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Wed, 13 May 2009 14:41:34 -0500
Subject: [Cython] Cython 0.11.2 beta
In-Reply-To: <4A09E193.20001@student.matnat.uio.no>
References: <4A09E193.20001@student.matnat.uio.no>
Message-ID: 

On Tue, May 12, 2009 at 3:52 PM, Dag Sverre Seljebotn
 wrote:
> Please try it out and report any problems:

2 failures and 2 errors on PPC G5 Mac, 10.4.11.  Error/failure output
below; I can attach the entire runtests.py output if helpful.

[105]$ uname -a
Darwin ksmith.physics.wisc.edu 8.11.0 Darwin Kernel Version 8.11.0:
Wed Oct 10 18:26:00 PDT 2007; root:xnu-792.24.17~1/RELEASE_PPC Power
Macintosh powerpc

======================================================================
ERROR: compiling (c) and running buffmt
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 345, in run
    doctest.DocTestSuite(self.module).run(result)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/doctest.py",
line 2266, in DocTestSuite
    module = _normalize_module(module)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/doctest.py",
line 200, in _normalize_module
    return __import__(module, globals(), locals(), ["*"])
  File "buffmt.pyx", line 30, in buffmt (buffmt.c:3143)
RuntimeError: Alignment or size of double is 4 on this system, please
report to cython-dev for a testcase fix

======================================================================
ERROR: compiling (cpp) and running buffmt
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 345, in run
    doctest.DocTestSuite(self.module).run(result)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/doctest.py",
line 2266, in DocTestSuite
    module = _normalize_module(module)
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/doctest.py",
line 200, in _normalize_module
    return __import__(module, globals(), locals(), ["*"])
  File "buffmt.pyx", line 30, in buffmt (buffmt.cpp:3143)
RuntimeError: Alignment or size of double is 4 on this system, please
report to cython-dev for a testcase fix

======================================================================
FAIL: Doctest: typedfieldbug_T303
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/doctest.py",
line 2128, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for typedfieldbug_T303
  File "/Users/ksmith/Downloads/Cython-0.11.2.beta/BUILD/run/c/typedfieldbug_T303.so",
line unknown line number, in typedfieldbug_T303

----------------------------------------------------------------------
File "/Users/ksmith/Downloads/Cython-0.11.2.beta/BUILD/run/c/typedfieldbug_T303.so",
line ?, in typedfieldbug_T303
Failed example:
    f()
Expected:
    42.0 42.0
Got:
    42.0 3.078125


======================================================================
FAIL: Doctest: typedfieldbug_T303
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/doctest.py",
line 2128, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for typedfieldbug_T303
  File "/Users/ksmith/Downloads/Cython-0.11.2.beta/BUILD/run/cpp/typedfieldbug_T303.so",
line unknown line number, in typedfieldbug_T303

----------------------------------------------------------------------
File "/Users/ksmith/Downloads/Cython-0.11.2.beta/BUILD/run/cpp/typedfieldbug_T303.so",
line ?, in typedfieldbug_T303
Failed example:
    f()
Expected:
    42.0 42.0
Got:
    42.0 3.078125


----------------------------------------------------------------------
Ran 1482 tests in 4090.720s

FAILED (failures=2, errors=2)

>
> http://sage.math.washington.edu/home/dagss/Cython-0.11.2.beta.tar.gz
>
> (This is the same trunk as was discussed earlier today, I poked around a
> bit more and didn't find more problems.)
>
> NOTE that the semantics for int conversion has changed in this version,
> so that converting a negative Python int to an unsigned C type will
> raise an OverflowException.
>
> --
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From hoytak at cs.ubc.ca  Wed May 13 21:43:56 2009
From: hoytak at cs.ubc.ca (Hoyt Koepke)
Date: Wed, 13 May 2009 12:43:56 -0700
Subject: [Cython] Project: Buffer arithmetic operations
In-Reply-To: <4A0B12AD.2060301@student.matnat.uio.no>
References: <4db580fd0905121314j142a30c7s915e4eb59d772b71@mail.gmail.com> 
	<4A0A73D2.6070105@student.matnat.uio.no>
	 
	<4A0B12AD.2060301@student.matnat.uio.no>
Message-ID: <4db580fd0905131243p5721a92em12f838059ae97d2c@mail.gmail.com>

>> I think this would be very very useful. But it seems reinventing the
>> wheel to implement the actual vector math. It is not that trivial. Just
>> as one exmaple: sometimes intermediate buffers are faster so you need to
>> have a heuristic to decide that.
>> Why not using a C++ template library for that? One could use Blitz++ but
>> I think Eigen (eigen.tuxfamily.org ) is
>> newer and significant faster (e.g. automatically uses SSE). Since
>> everything is in a include files no external libary requirement is added.
>
> Thanks for pitching in; that's some very interesting thoughts!
>
> Here's my immediate reaction:
> ?- We should avoid reinventing the wheel at all costs. Heuristics for
> intermediate buffers in Cython is probably a no-go.

I also agree.  Getting the basic algorithm working wouldn't be too big
a deal, but that is not being familiar with cython's internals and
abstracting the problem away.
However, I'm realizing that getting such a thing to play nicely in all
the corner cases is tricky.

> ?- Still, relying explicitly on one specific C++ library makes me uneasy
> ?- So perhaps what is needed here is some kind of plugin system.

>From the above arguments, I'm inclined to think that optional,
plugin-based optimization is probably the way to go.
It seems there are two reasons for not including something like eigen
or blitz++ in cython natively; first, license issues (a BSD version of
Blitz++ is included in scipy, but eigen does seem nicer and handling
matrices is super), and second it raises the compiler requirement from
that of CPython to C++.

I'm not familiar with how cython does optimizations, but the way I'd
imagine an ideal system to work would be something like this:

1. Plugins are registered somehow with hooks determining which node
types in the AST they are run on.
2. Plugins take such a node and return a modified node (or the same)
plus some other info (such as new headers to include or a new function
to stick somewhere).

Such plugin optimizers would be enabled using something like
"cython.enablePlugin('myplugin.py')"  or "with
cython.plugin('myplugin.py'):"    Then, when cython is run, the plugin
is loaded if it's found and a warning is printed if it's not.  (This,
of course, brings the requirement that such plugins don't change the
ultimate behavior.)

In addition, if such a system were in place, you could have dummy
"optimizing" plugins that did nothing but warn about code that could
be better written, such as A[0][1] vs A[0,1].

Now as I mentioned, I'm speaking out of ignorance about how cython
actually works in this regard.  It seems to me, though, that if
something like this was in place, expressions like the example I gave
previously could be detected and replaced with a block of code that
implemented the operation in eigen.

I am convinced that something like this would be a better way to go in
the long run. In 99% of the cases, I think that those who care about
the extra speed would likely be willing to deal with the extra effort
of another dependency and compiling in c++.

> Getting the basics working (just unoptimized, "what you would have
> written in C") is more trivial, and would get us up fast and allow
> operation without an external library.

> Then, if there was a good protocol for it, one would ideally be able to
> easily add support for other backends as well. (Which would all be
> required to have the same result, they would just differ in
> implementation detail.)

> On one hand, one shouldn't over-design, while on the other hand, a
> vector math library is a fairly sophisticated thing and one could argue
> that those should be made swappable if possible.

I think a plugin system like the one previously described is the way
to go.  Fully swappable is probably not the answer; you'll easily run
into corner cases that kill things.  For instance, blitz++ breaks if
the types of index arguments are not "int" (e.g. unsigned long) due to
how it implements the templates.  But a plugin system should handle
all this.

>> ? ? A) There's a clash with existing semantics to be resolved. For instance
>> ? ? if you have a NumPy "matrix" instance, then * does matrix multiplication
>> ? ? instead, and just implementing this now would break code.
>>
>> ? ? http://wiki.cython.org/enhancements/buffersyntax
>> ? ? http://thread.gmane.org/gmane.comp.python.numeric.general/28439
>>
>> ? ? contains my thoughts on how this could be resolved. I'll use this syntax
>> ? ? below.
>>

Ah, yes, I hadn't seen that earlier or
http://trac.cython.org/cython_trac/ticket/178.  That makes sense -- I
should have done a bit more searching before jumping in on this :-/.
I've been quite busy lately, so I didn't really follow that thread.

>> I think this could solved by allowing to define custom mapping of
>> operators/methods of python types/classes to C functions. With that one
>> could define that for Numpy ndarray the element wise multiplication is
>> used and for NumPy matrix the matrix multiplication is used. And both
>> could be translated into calls of Eigen or some other vector library.
>>
>> This would allow to directly transform normal NumPy, which tries to
>> avoid using loops, into highly optimized C with SSE without any
>> intermediate buffers.
>
> Well, if you write
>
> ? def f(np.ndarray[int] arr):
> ? ? print arr * arr
>
> ? f(np.matrix(...)) # subclass, so allowed
>
> then the Python version of that would be np.dot but the Cython version
> np.multiply.
>
> The problem is that we really need to know the behaviour compile-time.
> Should we disallow subclasses when a buffer is declared perhaps?

With a plugin optimization system, it seems that one could handle this
with flags in the AST nodes; e.g. one saying it is exactly of type
ndarray.  Then the optimizer would reject if the flags were not good.
Of course more cases (such as matrix) could be handled after the
initial, conservative cases were working.

Just my 2 cents.  I'm still willing to help out on this, though I'd
need more mentoring.

--Hoyt

++++++++++++++++++++++++++++++++++++++++++++++++
+ Hoyt Koepke
+ University of Washington Department of Statistics
+ http://www.stat.washington.edu/~hoytak/
+ hoytak at gmail.com
++++++++++++++++++++++++++++++++++++++++++

From hoytak at cs.ubc.ca  Wed May 13 21:56:43 2009
From: hoytak at cs.ubc.ca (Hoyt Koepke)
Date: Wed, 13 May 2009 12:56:43 -0700
Subject: [Cython] Project: Buffer arithmetic operations
In-Reply-To: <4db580fd0905131253x4e18b782kc978b6894a5f7c5e@mail.gmail.com>
References: <4db580fd0905121314j142a30c7s915e4eb59d772b71@mail.gmail.com> 
	<4A0A73D2.6070105@student.matnat.uio.no>
	 
	<4db580fd0905131253x4e18b782kc978b6894a5f7c5e@mail.gmail.com>
Message-ID: <4db580fd0905131256s66327119s981ce9648c97ad63@mail.gmail.com>

> The above strikes me as a bit too much magic under the hood -- perhaps
> I can get used to it.  My initial preference would be to have some
> indication that this is a 'ufuncable' function, or is intended to be
> so; Cython could then check to be sure the dummy arguments & return
> type are primitive.  (Although I reckon this check would be done
> during compilation once someone tries to pass a numpy array to a cdef
> function with primitive-type arguments.)

How difficult would it be to have a way to add general flags to
functions (like nogil?), so one could write

cdef extern from "math.h":
   double cos(double) ufunc

and then optimizing plugins like the ones I mentioned earlier could
pick up these flags and treat it appropriately.

-- Hoyt

P.S. I know that every open source project absolutely loves people
with more suggestions than willingness to look at the code , but I do think cython is very well designed and I'd like to
follow what's there.  I just get excited about small things I can get
my head around and think I might be able to do, so as I get to learn
the cython code base more I hope to be able to contribute suggestions
more productively.


++++++++++++++++++++++++++++++++++++++++++++++++
+ Hoyt Koepke
+ University of Washington Department of Statistics
+ http://www.stat.washington.edu/~hoytak/
+ hoytak at gmail.com
++++++++++++++++++++++++++++++++++++++++++

From sccolbert at gmail.com  Wed May 13 21:57:54 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Wed, 13 May 2009 15:57:54 -0400
Subject: [Cython] ctypes Vs Cython
In-Reply-To: <7f014ea60905131030n136ec8a9m54014cf79e238842@mail.gmail.com>
References: 
	<200905131017.07652.binet@cern.ch>
	<30DDBB3F-7219-4506-B93C-BCC07650044B@math.washington.edu>
	<200905131050.06546.binet@cern.ch>
	<1F4ECFF5-22E4-4712-8EC2-647F5B321A56@math.washington.edu>
	<7f014ea60905131030n136ec8a9m54014cf79e238842@mail.gmail.com>
Message-ID: <7f014ea60905131257g4e3b8239s86dd8365c34080fe@mail.gmail.com>

so my intitial guess that the problem was with my inability to properly
build a dll was correct.

After reading (partially) the gcc manual on compiling optimizations i get
the following result:

Starting Threads
>>> no GIL cython:  Value: 1999000.000000 Time: 8.076570 s
no GIL ctypes:  Value: 1999000.000000 Time: 8.083144 s

this was using the following compiler directives:

>gcc -c -O3 -DBUILD_DLL ctypesgiltest.c
>gcc -shared -o ctypesgiltest.dll ctypesgiltest.o


so, the two ended up being roughly equal (for the time scale in question)
and I learned how to compile a dll.

Cheers,

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090513/93d5722b/attachment.htm 

From dagss at student.matnat.uio.no  Wed May 13 22:11:57 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 13 May 2009 22:11:57 +0200 (CEST)
Subject: [Cython] Project: Buffer arithmetic operations
In-Reply-To: <4db580fd0905131256s66327119s981ce9648c97ad63@mail.gmail.com>
References: <4db580fd0905121314j142a30c7s915e4eb59d772b71@mail.gmail.com>
	<4A0A73D2.6070105@student.matnat.uio.no>
	
	<4db580fd0905131253x4e18b782kc978b6894a5f7c5e@mail.gmail.com>
	<4db580fd0905131256s66327119s981ce9648c97ad63@mail.gmail.com>
Message-ID: <1703a22623e40eb7bf68365fef4b4c91.squirrel@webmail.uio.no>

Hoyt Koepke wrote:
> P.S. I know that every open source project absolutely loves people
> with more suggestions than willingness to look at the code  sarcasm>, but I do think cython is very well designed and I'd like to
> follow what's there.  I just get excited about small things I can get
> my head around and think I might be able to do, so as I get to learn
> the cython code base more I hope to be able to contribute suggestions
> more productively.

Well in that case I am a bigger sinner than you -- I do write some code,
but I've easily started discussions on ten times more suggestions than
there's time to implement :-)

I think it spawned a very helpful discussion. In the end, everybody wants
to be able to "kill off Fortran", but it might be a long road to get there
with contributions from many.

As you've seen there's probably too many open issues to get something
usable by next week. Do you plan to work more on Cython after that?
(Answer in private if you wish). I'm happy to mentor you over summer if
you have time for something in that direction.

As for an isolated Sage days project, I still think (2) in my initial
email (some form of efficient ndenumerate loop) would make a very good
stepping stone project and is something which would be useful in many
places regardless of how things turn out with Blitz++ etc. (At the very
least, it would mean more efficient copying to and from contiguous memory,
which would be useful everywhere, also in Kurt's GSoC or when using C++
libraries for vectorization).

However I don't have time to *really* mentor until Tuesday, which may be a
bit late.

Dag Sverre


From robertwb at math.washington.edu  Wed May 13 22:15:07 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 13 May 2009 13:15:07 -0700
Subject: [Cython] #303 + simplifying type system
In-Reply-To: <4A099EC7.8040109@student.matnat.uio.no>
References: <4A099E0C.9000909@student.matnat.uio.no>
	<4A099EC7.8040109@student.matnat.uio.no>
Message-ID: 

On May 12, 2009, at 9:07 AM, Dag Sverre Seljebotn wrote:

> Dag Sverre Seljebotn wrote:
>> A)
>>
>> Ticket #303 manifests itself like this:
>>
>> cdef extern:
>>      ctypedef float footype # really double!
>>
>> cdef class A:
>>      cdef public footype myproperty
>>
>> My suggestion for a fix here is to replace all T_INT, T_UINT etc.  
>> with
>> simply three types: __Pyx_T_SIGNED, __Pyx_T_UNSIGNED, __Pyx_T_FLOAT.
>> These would use sizeof to determine the right type to flag the  
>> extension
>> type property as (like Lisandro did with T_SIZET already does).
>>
>> I can then change T_SIZET to __Pyx_T_UNSIGNED, right? (Basically that
>> would be a rename...) Or it T_SIZET is defined in any Python version,
>> I'll just throw it in on the list of possible "outputs" from the  
>> macro
>> if it is defined.

This sounds like a good idea.

>> B)
>>
>> Given this change (which could happen in -devel), is there any  
>> obstacles
>> to getting rid of the detailed type system in PyrexTypes? Basically
>> Cython would only deal with "signed", "unsigned" and "float" (and in
>> time "complex").
>>
>> Doing
>>
>> cdef short i
>>
>> would basically flag i as "signed"; the "short-ness" would only be
>> present as the cname of the type.

We use the relative ranking to determine the type of arithmetic  
operations. This will become even more important when type inference  
is used.

- Robert



From robertwb at math.washington.edu  Wed May 13 22:16:33 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 13 May 2009 13:16:33 -0700
Subject: [Cython] Notes on code streams refactoring
In-Reply-To: <4A092A52.2010309@student.matnat.uio.no>
References: <4A092A52.2010309@student.matnat.uio.no>
Message-ID: <7AC765CE-531D-4146-B100-E9203FF38CED@math.washington.edu>

On May 12, 2009, at 12:50 AM, Dag Sverre Seljebotn wrote:

> I found myself in a corner where the utility code mechanism was not
> fine-grained enough. Rather than adding yet-another-special-case I
> decided to refactor some stuff. Reviews welcome; I can of course  
> change
> it back (though hopefully not until after 0.11.2).
>
> a) In Code.py: GlobalState, there's now a list code_layout:
>      code_layout = [
>          'h_code',
>          'utility_code_proto',
>          'type_declarations',
>          'module_declarations',
>          'typeinfo',
>          'before_global_var',
>          'global_var',
>          'all_the_rest',
>          'utility_code_def'
>      ]
>
> This identifies code stream blocks that are set up initially (in that
> order).
>
> b) At any point, your code instance represents the default  
> location, but
> you can also output to other places:
>
> otherplace = code.globalstate['typeinfo']
> otherplace.putln(...)
>
> If just the right location isn't available, just split up an existing
> stream in (a).
>
> c) An alternative to the classic utility code is now this:
>
> if your_id not in code.globalstate.utility_codes:
>      code.globalstate.utility_codes.add(your_id)
>      somestream = code.globalstate['somestream']
>      somestream.putln(...)
>
> I.e. the presence of "some utility code" is now fully orthogonal to
> which code stream/location in the output the code ends up at. (The
> UtilityCode class is a convenience for this with a couple of hard- 
> coded
> streams.)
>
> http://hg.cython.org/cython-devel/rev/b86a7b374a95
> http://hg.cython.org/cython-devel/rev/fd6cc8a5998d
> http://hg.cython.org/cython-devel/rev/df8dd466bc4a

Looks fine to me after a cursory glance.

- Robert



From robertwb at math.washington.edu  Wed May 13 22:17:38 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 13 May 2009 13:17:38 -0700
Subject: [Cython] ctypes Vs Cython
In-Reply-To: <7f014ea60905131257g4e3b8239s86dd8365c34080fe@mail.gmail.com>
References: 
	<200905131017.07652.binet@cern.ch>
	<30DDBB3F-7219-4506-B93C-BCC07650044B@math.washington.edu>
	<200905131050.06546.binet@cern.ch>
	<1F4ECFF5-22E4-4712-8EC2-647F5B321A56@math.washington.edu>
	<7f014ea60905131030n136ec8a9m54014cf79e238842@mail.gmail.com>
	<7f014ea60905131257g4e3b8239s86dd8365c34080fe@mail.gmail.com>
Message-ID: <0195B9F6-9BCD-4157-9942-9AF21B0B7D42@math.washington.edu>

On May 13, 2009, at 12:57 PM, Chris Colbert wrote:

> so my intitial guess that the problem was with my inability to  
> properly build a dll was correct.
>
> After reading (partially) the gcc manual on compiling optimizations  
> i get the following result:
>
> Starting Threads
> >>> no GIL cython:  Value: 1999000.000000 Time: 8.076570 s
> no GIL ctypes:  Value: 1999000.000000 Time: 8.083144 s
>
> this was using the following compiler directives:
>
> >gcc -c -O3 -DBUILD_DLL ctypesgiltest.c
> >gcc -shared -o ctypesgiltest.dll ctypesgiltest.o
>
>
> so, the two ended up being roughly equal (for the time scale in  
> question) and I learned how to compile a dll.

That's a fairer comparison and more like what I would have expected.  
Moral of the story: don't compile (release build) without any  
optimizations if you care about speed.

- Robert


From roland at utk.edu  Wed May 13 22:27:04 2009
From: roland at utk.edu (Roland Schulz)
Date: Wed, 13 May 2009 16:27:04 -0400
Subject: [Cython] Project: Buffer arithmetic operations
In-Reply-To: <4A0B1817.1020206@student.matnat.uio.no>
References: <4db580fd0905121314j142a30c7s915e4eb59d772b71@mail.gmail.com> 
	<4A0A73D2.6070105@student.matnat.uio.no>
	 
	<4A0B12AD.2060301@student.matnat.uio.no>
	<4A0B1817.1020206@student.matnat.uio.no>
Message-ID: 

On Wed, May 13, 2009 at 2:57 PM, Dag Sverre Seljebotn <
dagss at student.matnat.uio.no> wrote:

> Dag Sverre Seljebotn wrote:
>
>
> Note that one could implement this in "shadow module" mode as well. I.e.
>
> @cython.locals(arr=cython.int[:])
> def f(arr):
>     return arr * arr + 2
>
> could be made to work in pure Python mode by coercing the argument to an
> appropriate NumPy subclass emulating the explicit buffers.
>
> So I think it is primarily about what kind of syntax it is natural to
> work with etc; the one "transparently" optimizing some parts of what one
> is used to writing in Python, or the one shouting "syntax error"
> whenever you do something which is not an optimized, fast operation.
>
> I find I now (after nearly a year) wish I had gone for the latter
> instead :-) as users are still caught by surprise that e.g. "arr[2][3]"
> isn't as fast as "arr[2,3]", but there's no compiler complaints, and so on.
>
>
Why not give warning when certain constructors are likely slower than
expected? At least when running with something like -Wall.

Also to me the strength of cython is that I can take a python function and
add only minimal information (a few cdefs for variables) and get a lot of
speedup. I wouldn't start a project with cython.Iif  know from the beginning
it is to slow in python, then I do that directly in C/C+.

The problem is that cython does not help really for code heavily using numpy
arrays without loops. So it would be great to be able to define a numpy
array as a blitz/eigen array and some plugin conversion rules convert the
calls. Sure it would prohibit to use subclasses. But to me that would be
similar to changing an Python variable to a cython int. There one also has
to make sure that e.g. there is no overflow.

A general way to work on buffers would probably be nice, but it would not
help so much in being easily able to speed up numpy code with cython, if it
is not using the numpy syntax.

Roland
-- 
ORNL/UT Center for Molecular Biophysics cmb.ornl.gov
865-241-1537, ORNL PO BOX 2008 MS6309
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090513/2beea1e8/attachment.htm 

From dagss at student.matnat.uio.no  Wed May 13 22:31:59 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 13 May 2009 22:31:59 +0200 (CEST)
Subject: [Cython] #303 + simplifying type system
In-Reply-To: 
References: <4A099E0C.9000909@student.matnat.uio.no>
	<4A099EC7.8040109@student.matnat.uio.no>
	
Message-ID: <8e8be90c33e92cc1cca8b77d1d0ce407.squirrel@webmail.uio.no>

Robert wrote:
>>> B)
>>>
>>> Given this change (which could happen in -devel), is there any
>>> obstacles
>>> to getting rid of the detailed type system in PyrexTypes? Basically
>>> Cython would only deal with "signed", "unsigned" and "float" (and in
>>> time "complex").
>>>
>>> Doing
>>>
>>> cdef short i
>>>
>>> would basically flag i as "signed"; the "short-ness" would only be
>>> present as the cname of the type.
>
> We use the relative ranking to determine the type of arithmetic
> operations. This will become even more important when type inference
> is used.

Hmm. Yes...what are those used for for which a size-less system isn't
sufficient? I can think of declaration of temps, but nothing else.

But that means there's still a real problem even if I fix #303; if we
don't know the exact type for external types then e.g. temporaries from
expressions of those could loose precision.

Would it work to always allocate temps at the widest possible precision?

Dag Sverre


From hoytak at cs.ubc.ca  Wed May 13 22:41:32 2009
From: hoytak at cs.ubc.ca (Hoyt Koepke)
Date: Wed, 13 May 2009 13:41:32 -0700
Subject: [Cython] Project: Buffer arithmetic operations
In-Reply-To: <1703a22623e40eb7bf68365fef4b4c91.squirrel@webmail.uio.no>
References: <4db580fd0905121314j142a30c7s915e4eb59d772b71@mail.gmail.com> 
	<4A0A73D2.6070105@student.matnat.uio.no>
	 
	<4db580fd0905131253x4e18b782kc978b6894a5f7c5e@mail.gmail.com> 
	<4db580fd0905131256s66327119s981ce9648c97ad63@mail.gmail.com> 
	<1703a22623e40eb7bf68365fef4b4c91.squirrel@webmail.uio.no>
Message-ID: <4db580fd0905131341n43a00289lbb36b6c29da0306d@mail.gmail.com>

> I think it spawned a very helpful discussion. In the end, everybody wants
> to be able to "kill off Fortran", but it might be a long road to get there
> with contributions from many.
>
> As you've seen there's probably too many open issues to get something
> usable by next week. Do you plan to work more on Cython after that?
> (Answer in private if you wish). I'm happy to mentor you over summer if
> you have time for something in that direction.

I would definitely like to continue trying to develop cython in an
ongoing way, and my intent was to try to use the sage days coding
sprint as a way to gain some familiarity with the cython code base.  I
suggested the previous project in part because there are a lot of
aspects that I could do on my own so I don't monopolize other people's
time with questions.  In general, though, I am still balancing
classes, prelims, and research, which is the only reason I haven't got
more involved with cython til now, and I don't think things will let
up until the fall.

> As for an isolated Sage days project, I still think (2) in my initial
> email (some form of efficient ndenumerate loop) would make a very good
> stepping stone project and is something which would be useful in many
> places regardless of how things turn out with Blitz++ etc. (At the very
> least, it would mean more efficient copying to and from contiguous memory,
> which would be useful everywhere, also in Kurt's GSoC or when using C++
> libraries for vectorization).

This definitely seems like something useful, and I'd be happy to do
it, but I would need some mentoring to get started.

--Hoyt

++++++++++++++++++++++++++++++++++++++++++++++++
+ Hoyt Koepke
+ University of Washington Department of Statistics
+ http://www.stat.washington.edu/~hoytak/
+ hoytak at gmail.com
++++++++++++++++++++++++++++++++++++++++++

From carl.witty at gmail.com  Wed May 13 22:42:01 2009
From: carl.witty at gmail.com (Carl Witty)
Date: Wed, 13 May 2009 13:42:01 -0700
Subject: [Cython] #303 + simplifying type system
In-Reply-To: <8e8be90c33e92cc1cca8b77d1d0ce407.squirrel@webmail.uio.no>
References: <4A099E0C.9000909@student.matnat.uio.no>
	<4A099EC7.8040109@student.matnat.uio.no> 
	 
	<8e8be90c33e92cc1cca8b77d1d0ce407.squirrel@webmail.uio.no>
Message-ID: 

On Wed, May 13, 2009 at 1:31 PM, Dag Sverre Seljebotn
 wrote:
> But that means there's still a real problem even if I fix #303; if we
> don't know the exact type for external types then e.g. temporaries from
> expressions of those could loose precision.
>
> Would it work to always allocate temps at the widest possible precision?

That doesn't sound like a good idea.  The widest possible precisions
would presumably be "long long" (for integer) or "long double" (for
float) types; but on many platforms, "long long" is much slower than
"int" and "long double" is much slower than "double".

(And I'm pretty sure it's theoretically possible (allowed by the C
standard) to have integral types wider than "long long" and/or
floating-point types wider than "long double".)

Carl

From dagss at student.matnat.uio.no  Wed May 13 22:49:19 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 13 May 2009 22:49:19 +0200 (CEST)
Subject: [Cython] #303 + simplifying type system
In-Reply-To: 
References: <4A099E0C.9000909@student.matnat.uio.no>
	<4A099EC7.8040109@student.matnat.uio.no>
	
	<8e8be90c33e92cc1cca8b77d1d0ce407.squirrel@webmail.uio.no>
	
Message-ID: 

Carl Witty wrote:
> On Wed, May 13, 2009 at 1:31 PM, Dag Sverre Seljebotn
>  wrote:
>> But that means there's still a real problem even if I fix #303; if we
>> don't know the exact type for external types then e.g. temporaries from
>> expressions of those could loose precision.
>>
>> Would it work to always allocate temps at the widest possible precision?
>
> That doesn't sound like a good idea.  The widest possible precisions
> would presumably be "long long" (for integer) or "long double" (for
> float) types; but on many platforms, "long long" is much slower than
> "int" and "long double" is much slower than "double".

Yes, but these are only used in temporaries which are presumably read just
after. Wouldn't the compiler be able to optimize it? E.g.

short a = ..., b = ..., c;
long long long long int temp;

temp = a + b;
c = temp;

It seems plausible that the compiler could decide that temp could be made
a short (?). But it definitely needs investigation to see exactly which
sort of situations the knowledge about the size is used in.

The alternative (which doesn't make everything mixed up and confusing,
anyway) seems to be to require pxd's to be exact; which means that Cython
would pretty much require #ifdef and being passed defines from makefiles
etc.

Dag Sverre


From carl.witty at gmail.com  Wed May 13 23:38:08 2009
From: carl.witty at gmail.com (Carl Witty)
Date: Wed, 13 May 2009 14:38:08 -0700
Subject: [Cython] #303 + simplifying type system
In-Reply-To: 
References: <4A099E0C.9000909@student.matnat.uio.no>
	<4A099EC7.8040109@student.matnat.uio.no> 
	 
	<8e8be90c33e92cc1cca8b77d1d0ce407.squirrel@webmail.uio.no> 
	 
	
Message-ID: 

On Wed, May 13, 2009 at 1:49 PM, Dag Sverre Seljebotn
 wrote:
> Yes, but these are only used in temporaries which are presumably read just
> after. Wouldn't the compiler be able to optimize it? E.g.
>
> short a = ..., b = ..., c;
> long long long long int temp;
>
> temp = a + b;
> c = temp;
>
> It seems plausible that the compiler could decide that temp could be made
> a short (?). But it definitely needs investigation to see exactly which
> sort of situations the knowledge about the size is used in.

This doesn't work in more complicated situations.

unsigned int a=..., b=..., c=..., d = ..., result;
unsigned long long temp1, temp2, temp3, temp4, temp5, temp6;

temp1 = a+1;
temp2 = b+1;
temp3 = c+1;
temp4 = d+1;
temp5 = temp1*temp2;
temp6 = temp3*temp4;
result = temp5/temp6;

And assume that int is 32 bits, long long is 64 bits.

Here C semantics guarantee that temp5 and temp6 are computed to full
64-bit resolution, and that the division is done in 64 bits (and then
truncated to 32 bits).  But if the temporaries were unsigned int,
instead, then temp5 and temp6 would be truncated to 32 bits before
doing the division.  So the C compiler can't possibly optimize temp5
and temp6 to unsigned int.  (I believe it could optimize temp1 ..
temp4 to unsigned int, as long as it still used 32x32->64 multiplies
to compute temp5 and temp6.)

I certainly have written C code that depends on the compiler exactly
following the C semantics for overflow on unsigned types.  I don't
remember if I've written such code in Cython (probably not).

Carl

From dalcinl at gmail.com  Thu May 14 00:08:25 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 13 May 2009 19:08:25 -0300
Subject: [Cython] #303 + simplifying type system
In-Reply-To: <4A099E0C.9000909@student.matnat.uio.no>
References: <4A099E0C.9000909@student.matnat.uio.no>
Message-ID: 

On Tue, May 12, 2009 at 1:04 PM, Dag Sverre Seljebotn
 wrote:
> A)
>
> Ticket #303 manifests itself like this:
>
> cdef extern:
> ? ? ctypedef float footype # really double!
>
> cdef class A:
> ? ? cdef public footype myproperty
>
> My suggestion for a fix here is to replace all T_INT, T_UINT etc. with
> simply three types: __Pyx_T_SIGNED, __Pyx_T_UNSIGNED, __Pyx_T_FLOAT.
> These would use sizeof to determine the right type to flag the extension
> type property as (like Lisandro did with T_SIZET already does).
>

I'm -1 on this...

For (A), we need to enhance Cython to generate a couple of converter
functions to_py/from_py for externally defined integral or floating
types. For the integral ones, the only requisite is to specify sign
part. I've already have some preliminary work on this, but I'm not
sure how to go on because of the delegation-based implementation in
CTypedefNode.

For (B), I believe that if the type is an external typedef, a
transform should change the "cdef publid footype value" to this:

cdef class A:
   property value:
      def __get__(self): return self.value
      def __set__(self, footype x) : self.value = x

In short, I already have implemented the utility code for (A), but I'm
in doubt about how to reimplement CTypedefNode. Additionally, I
believe my appoach for (B) could be easily implemented (likely Dag can
implement this part far better than me).



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Thu May 14 00:13:15 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 13 May 2009 15:13:15 -0700
Subject: [Cython] #303 + simplifying type system
In-Reply-To: 
References: <4A099E0C.9000909@student.matnat.uio.no>
	<4A099EC7.8040109@student.matnat.uio.no>
	
	<8e8be90c33e92cc1cca8b77d1d0ce407.squirrel@webmail.uio.no>
	
	
	
Message-ID: 

On May 13, 2009, at 2:38 PM, Carl Witty wrote:

> On Wed, May 13, 2009 at 1:49 PM, Dag Sverre Seljebotn
>  wrote:
>> Yes, but these are only used in temporaries which are presumably  
>> read just
>> after. Wouldn't the compiler be able to optimize it? E.g.
>>
>> short a = ..., b = ..., c;
>> long long long long int temp;
>>
>> temp = a + b;
>> c = temp;
>>
>> It seems plausible that the compiler could decide that temp could  
>> be made
>> a short (?). But it definitely needs investigation to see exactly  
>> which
>> sort of situations the knowledge about the size is used in.

No, I don't trust the compiler that much. Also, I'm thinking of type  
inference of the kind

def foo(int a, short b):
     cdef int a
     cdef short b
     i = a+b # in the future, this could be a declaration of a cdef  
i, which should (IMHO) be an int
     return i

> This doesn't work in more complicated situations.
>
> unsigned int a=..., b=..., c=..., d = ..., result;
> unsigned long long temp1, temp2, temp3, temp4, temp5, temp6;
>
> temp1 = a+1;
> temp2 = b+1;
> temp3 = c+1;
> temp4 = d+1;
> temp5 = temp1*temp2;
> temp6 = temp3*temp4;
> result = temp5/temp6;
>
> And assume that int is 32 bits, long long is 64 bits.
>
> Here C semantics guarantee that temp5 and temp6 are computed to full
> 64-bit resolution, and that the division is done in 64 bits (and then
> truncated to 32 bits).  But if the temporaries were unsigned int,
> instead, then temp5 and temp6 would be truncated to 32 bits before
> doing the division.  So the C compiler can't possibly optimize temp5
> and temp6 to unsigned int.  (I believe it could optimize temp1 ..
> temp4 to unsigned int, as long as it still used 32x32->64 multiplies
> to compute temp5 and temp6.)
>
> I certainly have written C code that depends on the compiler exactly
> following the C semantics for overflow on unsigned types.  I don't
> remember if I've written such code in Cython (probably not).

I think this is a valid point too. In particular, with python  
division semantics, it must store the denominator in a temp in order  
to detect whether or not to raise a ZeroDivisionError.

I think we should be using sizeof(T) as often as possible to do the  
right thing, but we should still keep track of the rank. I do think  
the current type system could be simplified without getting rid of  
the rank though.

- Robert


From dagss at student.matnat.uio.no  Thu May 14 06:21:21 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 14 May 2009 06:21:21 +0200 (CEST)
Subject: [Cython] #303 + simplifying type system
In-Reply-To: 
References: <4A099E0C.9000909@student.matnat.uio.no>
	<4A099EC7.8040109@student.matnat.uio.no>
	
	<8e8be90c33e92cc1cca8b77d1d0ce407.squirrel@webmail.uio.no>
	
	
	
	
Message-ID: <83c16f5d9eaf56a700b547bff6950f31.squirrel@webmail.uio.no>

Robert Bradshaw wrote:
> I think we should be using sizeof(T) as often as possible to do the
> right thing, but we should still keep track of the rank. I do think
> the current type system could be simplified without getting rid of
> the rank though.

But currently there is just no way to reliably compare the ranks of
external typedefs; and if we introduced that requirement, there's still no
way we could compare them with the ranks of non-external types.

Dag Sverre


From dagss at student.matnat.uio.no  Thu May 14 06:35:31 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 14 May 2009 06:35:31 +0200 (CEST)
Subject: [Cython] #303 + simplifying type system
In-Reply-To: 
References: <4A099E0C.9000909@student.matnat.uio.no>
	<4A099EC7.8040109@student.matnat.uio.no>
	
	<8e8be90c33e92cc1cca8b77d1d0ce407.squirrel@webmail.uio.no>
	
	
	
	
Message-ID: 

Robert Bradshaw wrote:
> On May 13, 2009, at 2:38 PM, Carl Witty wrote:
>
>> On Wed, May 13, 2009 at 1:49 PM, Dag Sverre Seljebotn
>>  wrote:
>>> Yes, but these are only used in temporaries which are presumably
>>> read just
>>> after. Wouldn't the compiler be able to optimize it? E.g.
>>>
>>> short a = ..., b = ..., c;
>>> long long long long int temp;
>>>
>>> temp = a + b;
>>> c = temp;
>>>
>>> It seems plausible that the compiler could decide that temp could
>>> be made
>>> a short (?). But it definitely needs investigation to see exactly
>>> which
>>> sort of situations the knowledge about the size is used in.
>> This doesn't work in more complicated situations.
>>
>> unsigned int a=..., b=..., c=..., d = ..., result;
>> unsigned long long temp1, temp2, temp3, temp4, temp5, temp6;
>>
>> temp1 = a+1;
>> temp2 = b+1;
>> temp3 = c+1;
>> temp4 = d+1;
>> temp5 = temp1*temp2;
>> temp6 = temp3*temp4;
>> result = temp5/temp6;
>>
>> And assume that int is 32 bits, long long is 64 bits.
>>
>> Here C semantics guarantee that temp5 and temp6 are computed to full
>> 64-bit resolution, and that the division is done in 64 bits (and then
>> truncated to 32 bits).  But if the temporaries were unsigned int,
>> instead, then temp5 and temp6 would be truncated to 32 bits before
>> doing the division.  So the C compiler can't possibly optimize temp5
>> and temp6 to unsigned int.  (I believe it could optimize temp1 ..
>> temp4 to unsigned int, as long as it still used 32x32->64 multiplies
>> to compute temp5 and temp6.)
>>
>> I certainly have written C code that depends on the compiler exactly
>> following the C semantics for overflow on unsigned types.  I don't
>> remember if I've written such code in Cython (probably not).
>
> I think this is a valid point too. In particular, with python
> division semantics, it must store the denominator in a temp in order
> to detect whether or not to raise a ZeroDivisionError.

What one could do is:

short temp_short
int temp_int
long long temp_longlong
switch (sizeof(footype)) {
  case sizeof(short): temp_short = ....
  case sizeof(int): temp_short = ....
  case sizeof(long long): temp_short = ....
}
....
result... = ((sizeof(footype) == sizeof(short) ? temp_short : ...)

I sure hope I think of something better though :-)

Dag Sverre


From dagss at student.matnat.uio.no  Thu May 14 06:43:59 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 14 May 2009 06:43:59 +0200 (CEST)
Subject: [Cython] #303 + simplifying type system
In-Reply-To: 
References: <4A099E0C.9000909@student.matnat.uio.no>
	<4A099EC7.8040109@student.matnat.uio.no>
	
	<8e8be90c33e92cc1cca8b77d1d0ce407.squirrel@webmail.uio.no>
	
	
	
	
	
Message-ID: <98a9b3e6b57076a72224680c140cf168.squirrel@webmail.uio.no>

Dag Sverre wrote:
> What one could do is:
>
> short temp_short
> int temp_int
> long long temp_longlong
> switch (sizeof(footype)) {
>   case sizeof(short): temp_short = ....
>   case sizeof(int): temp_short = ....
>   case sizeof(long long): temp_short = ....
> }
> ....
> result... = ((sizeof(footype) == sizeof(short) ? temp_short : ...)

Consider that proof that it is still 06 in the morning here...

The idea holds, but what one must do is make comparisons between types
(sizeof(footype) > sizeof(bartype)?) to select between two alternative
temps.

The problem is then when that is used in another expression with a third
external typedef, which would lead to three possibilities for the temp,
and so on (in practice I can't see the number of external typedefs used in
any expression to go above four or five so it's not horrible).

Again, I don't like it, but it shows that there is a way.

But I should probably use my Cython time for more worthy goals and just
tell people that "if you don't know the size of an external typedef,
declare it at the widest possible type it can be" for now. People can't
then use overflowing behaviour for external typedefs, which seem OK.

Dag Sverre


From prabhu at aero.iitb.ac.in  Thu May 14 08:02:43 2009
From: prabhu at aero.iitb.ac.in (Prabhu Ramachandran)
Date: Thu, 14 May 2009 11:32:43 +0530
Subject: [Cython] Paper about Cython
In-Reply-To: <4A0B0704.1010205@simula.no>
References: <4A0B0704.1010205@simula.no>
Message-ID: <4A0BB403.9080706@aero.iitb.ac.in>

Hi,

On 05/13/09 23:14, Ilmar Wilbers wrote:
> It must be noted that the time for the arithmetic mean for Cython has 
> decreased since the paper was accepted, as Dag Sverre pointed me to the 
> additional argument 'mode' for the arrays. This reduces the time for 
> Cython from 1.6 to 1.3 compared to the pure Fortran implementation. I 
> apologize for any mistakes concerning Cython in the paper, as I'm sure 
> the developers have some comments about stuff I've missed.

Perhaps you are unaware of this?

  http://www.scipy.org/PerformancePython

It predates your study (using Pyrex) by several years (at least 4-5 
years)!  This was done by the community and now features comparisons 
with Matlab and Octave as well.

cheers,
prabhu

From robertwb at math.washington.edu  Thu May 14 08:06:17 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 13 May 2009 23:06:17 -0700
Subject: [Cython] #303 + simplifying type system
In-Reply-To: <98a9b3e6b57076a72224680c140cf168.squirrel@webmail.uio.no>
References: <4A099E0C.9000909@student.matnat.uio.no>
	<4A099EC7.8040109@student.matnat.uio.no>
	
	<8e8be90c33e92cc1cca8b77d1d0ce407.squirrel@webmail.uio.no>
	
	
	
	
	
	<98a9b3e6b57076a72224680c140cf168.squirrel@webmail.uio.no>
Message-ID: 

On May 13, 2009, at 9:43 PM, Dag Sverre Seljebotn wrote:

> Dag Sverre wrote:
>> What one could do is:
>>
>> short temp_short
>> int temp_int
>> long long temp_longlong
>> switch (sizeof(footype)) {
>>   case sizeof(short): temp_short = ....
>>   case sizeof(int): temp_short = ....
>>   case sizeof(long long): temp_short = ....
>> }
>> ....
>> result... = ((sizeof(footype) == sizeof(short) ? temp_short : ...)
>
> Consider that proof that it is still 06 in the morning here...
>
> The idea holds, but what one must do is make comparisons between types
> (sizeof(footype) > sizeof(bartype)?) to select between two alternative
> temps.
>
> The problem is then when that is used in another expression with a  
> third
> external typedef, which would lead to three possibilities for the  
> temp,
> and so on (in practice I can't see the number of external typedefs  
> used in
> any expression to go above four or five so it's not horrible).
>
> Again, I don't like it, but it shows that there is a way.

Well, considering how infrequently Cython uses temp variables, it  
won't make the code much more verbose, will it? :D

> But I should probably use my Cython time for more worthy goals and  
> just
> tell people that "if you don't know the size of an external typedef,
> declare it at the widest possible type it can be" for now. People  
> can't
> then use overflowing behaviour for external typedefs, which seem OK.


I completely agree. Conversion to/from Cython can be done smarter  
using sizeof(), but for the cases discussed here if one really cares  
about overflow semantics than one has to be careful about defining  
types more precisely, or not mixing external types with builtin int  
types.

You up for testing complex support sometime really soon?

- Robert


From dagss at student.matnat.uio.no  Thu May 14 08:13:29 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 14 May 2009 08:13:29 +0200 (CEST)
Subject: [Cython] #303 + simplifying type system
In-Reply-To: 
References: <4A099E0C.9000909@student.matnat.uio.no>
	<4A099EC7.8040109@student.matnat.uio.no>
	
	<8e8be90c33e92cc1cca8b77d1d0ce407.squirrel@webmail.uio.no>
	
	
	
	
	
	<98a9b3e6b57076a72224680c140cf168.squirrel@webmail.uio.no>
	
Message-ID: <692a1b8c9b804e733c3ce1e61026e3a1.squirrel@webmail.uio.no>

> On May 13, 2009, at 9:43 PM, Dag Sverre Seljebotn wrote:
> You up for testing complex support sometime really soon?

Yay!

Well, I expect my daughter to fall asleep within an hour and sleep for an
hour, but it's a tight timeslot to hit :-) But if not ready then I'll do
it later today.

Dag Sverre


From robertwb at math.washington.edu  Thu May 14 08:17:20 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 13 May 2009 23:17:20 -0700
Subject: [Cython] #303 + simplifying type system
In-Reply-To: <692a1b8c9b804e733c3ce1e61026e3a1.squirrel@webmail.uio.no>
References: <4A099E0C.9000909@student.matnat.uio.no>
	<4A099EC7.8040109@student.matnat.uio.no>
	
	<8e8be90c33e92cc1cca8b77d1d0ce407.squirrel@webmail.uio.no>
	
	
	
	
	
	<98a9b3e6b57076a72224680c140cf168.squirrel@webmail.uio.no>
	
	<692a1b8c9b804e733c3ce1e61026e3a1.squirrel@webmail.uio.no>
Message-ID: <6725B0C1-43BB-4872-9D9B-231351764A3A@math.washington.edu>

On May 13, 2009, at 11:13 PM, Dag Sverre Seljebotn wrote:

>> On May 13, 2009, at 9:43 PM, Dag Sverre Seljebotn wrote:
>> You up for testing complex support sometime really soon?
>
> Yay!
>
> Well, I expect my daughter to fall asleep within an hour and sleep  
> for an
> hour, but it's a tight timeslot to hit :-) But if not ready then  
> I'll do
> it later today.

I'm hoping to get it to you before my daughters wake up to eat in a  
couple of hours, just testing it now.

- Robert


From jelleferinga at gmail.com  Thu May 14 09:27:47 2009
From: jelleferinga at gmail.com (jelle feringa)
Date: Thu, 14 May 2009 09:27:47 +0200
Subject: [Cython] Paper about Cython
In-Reply-To: <4A0BB403.9080706@aero.iitb.ac.in>
References: <4A0B0704.1010205@simula.no> <4A0BB403.9080706@aero.iitb.ac.in>
Message-ID: <37fac3b90905140027y1a5c71ffs23e5bc721faecff1@mail.gmail.com>

interesting! thanks for the heads up.

From robertwb at math.washington.edu  Thu May 14 10:35:22 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 14 May 2009 01:35:22 -0700
Subject: [Cython] Cython complex numbers
Message-ID: <44E38307-17FF-492D-8735-BB8DE5968ADD@math.washington.edu>

I just pushed a series of patches to http://hg.cython.org/cython- 
devel/ for complex number support. See

http://hg.cython.org/cython-devel/file/5c014b865386/tests/run/ 
complex_numbers_T305.pyx .

Despite being a rather large change, and shortly before a release, it  
belongs in -devel because this is one of the motivations to do a  
release so soon. If complex.h is included, it uses c99 complex types,  
otherwise it falls back to its own implementation using structs. If  
this interests you please play around with it.

- Robert


From darkgl0w at yahoo.com  Thu May 14 10:37:41 2009
From: darkgl0w at yahoo.com (Cristi Constantin)
Date: Thu, 14 May 2009 01:37:41 -0700 (PDT)
Subject: [Cython] Faster Cython mappings?
Message-ID: <477042.42563.qm@web52110.mail.re2.yahoo.com>

Good day.
I am trying to obtain MAXIMUM speed for this mappings:

MyString = '\n'.join([ ''.join([j.encode('utf8') for j in i]) for i in NdArrayList ])

"NdArrayList" is a list of numpy ndarrays that contain one Unicode character and i want to transform it into a string, AS FAST AS POSSIBLE.
For example : NdArrayList can be = [ np.array([u'a', u'\u2588'], dtype='

What's the best way of increasing the warning level from a distutils 
setup.py?

(Just wasted an hour over doing

arr.data

when arr was typed as object instead of ndarray. Never again!)

-- 
Dag Sverre

From robertwb at math.washington.edu  Thu May 14 10:45:59 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 14 May 2009 01:45:59 -0700
Subject: [Cython] Faster Cython mappings?
In-Reply-To: <477042.42563.qm@web52110.mail.re2.yahoo.com>
References: <477042.42563.qm@web52110.mail.re2.yahoo.com>
Message-ID: 

On May 14, 2009, at 1:37 AM, Cristi Constantin wrote:

> Good day.
> I am trying to obtain MAXIMUM speed for this mappings:
>
> MyString = '\n'.join([ ''.join([j.encode('utf8') for j in i]) for i  
> in NdArrayList ])
>
> "NdArrayList" is a list of numpy ndarrays that contain one Unicode  
> character and i want to transform it into a string, AS FAST AS  
> POSSIBLE.
> For example : NdArrayList can be = [ np.array([u'a', u'\u2588'],  
> dtype=' dtype='
> I also want to make MyList = [ ''.join([j.encode('utf8') for j in  
> i]) for i in NdArrayList ]. In this case i need a list of united-as- 
> string numpy ndarrays.
>
> When i call the codes from Python i get better performance than  
> compiling into a Cython function like:
>
> cdef str __Ndarray2String( list TempA ):
>     #
>     return '\n'.join([ ''.join([j.encode('utf8') for j in i]) for i  
> in TempA ])
>     #
> def Ndarray2String( v ):
>     return __Ndarray2String( v )
>
> Can anyone please sugest a good method?

To go for maximum speed, I would suggest pulling all the data out of  
the array first (perhaps using the buffer interface) and using  
assignment and/or memcpy to put it into a big string (manually  
inserting the newlines). Then call encode to get the whole thing in  
utf8.

- Robert


From robertwb at math.washington.edu  Thu May 14 10:48:04 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 14 May 2009 01:48:04 -0700
Subject: [Cython] Warnings from setup.py?
In-Reply-To: <4A0BD992.6010802@student.matnat.uio.no>
References: <4A0BD992.6010802@student.matnat.uio.no>
Message-ID: <812EAB57-4D87-48FF-80E0-C598BF729EE1@math.washington.edu>

On May 14, 2009, at 1:42 AM, Dag Sverre Seljebotn wrote:

> What's the best way of increasing the warning level from a distutils
> setup.py?

Not sure, perhaps you can actually set the relevant module-level  
variable.

> (Just wasted an hour over doing
>
> arr.data
>
> when arr was typed as object instead of ndarray. Never again!)

Ouch.

Is there any reason to even allow this (other than to be backwards  
compatible with broken code). I think we should allow object <-->  
void*, or object <--> PyObject*, but no others. (One would then be  
forced to do arr.data if one *really* wanted to.)

- Robert



From dagss at student.matnat.uio.no  Thu May 14 10:53:52 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 14 May 2009 10:53:52 +0200 (CEST)
Subject: [Cython] Warnings from setup.py?
In-Reply-To: <812EAB57-4D87-48FF-80E0-C598BF729EE1@math.washington.edu>
References: <4A0BD992.6010802@student.matnat.uio.no>
	<812EAB57-4D87-48FF-80E0-C598BF729EE1@math.washington.edu>
Message-ID: <2b77a966c4bd38d9a19f6e0baeb83101.squirrel@webmail.uio.no>

Robert wrote:
> On May 14, 2009, at 1:42 AM, Dag Sverre Seljebotn wrote:
>
>> What's the best way of increasing the warning level from a distutils
>> setup.py?
>
> Not sure, perhaps you can actually set the relevant module-level
> variable.

I suppose we should just make a directive for warnings at some point; I'll
leave the issue until then.

>> (Just wasted an hour over doing
>>
>> arr.data
>>
>> when arr was typed as object instead of ndarray. Never again!)
>
> Ouch.
>
> Is there any reason to even allow this (other than to be backwards
> compatible with broken code). I think we should allow object <-->
> void*, or object <--> PyObject*, but no others. (One would then be
> forced to do arr.data if one *really* wanted to.)

Excellent idea!; and as you say should not break code. I'll do this for
0.11.2.

Thanks a lot for the complex support; I'll check it out and add buffer
support later today.

Dag Sverre


From stefan_ml at behnel.de  Thu May 14 11:06:23 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 14 May 2009 11:06:23 +0200 (CEST)
Subject: [Cython] Cython complex numbers
In-Reply-To: <44E38307-17FF-492D-8735-BB8DE5968ADD@math.washington.edu>
References: <44E38307-17FF-492D-8735-BB8DE5968ADD@math.washington.edu>
Message-ID: 

Robert Bradshaw wrote:
> I just pushed a series of patches to http://hg.cython.org/cython-
> devel/ for complex number support. See
>
> http://hg.cython.org/cython-devel/file/5c014b865386/tests/run/
> complex_numbers_T305.pyx .
>
> Despite being a rather large change, and shortly before a release, it
> belongs in -devel because this is one of the motivations to do a
> release so soon.

Does it (potentially) break any existing stuff or is it a plain new feature?

The latter would make it easy to say: "Sure, let's release it; if the
feature is broken in some way, there will be a fix release."

Stefan


From robertwb at math.washington.edu  Thu May 14 11:08:16 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 14 May 2009 02:08:16 -0700
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: 
References: 
	<49EB3943.9090409@behnel.de>
	<4192B3F1-8CF6-41CA-80EE-42B2D32623B0@math.washington.edu>
	<4905F089-AC21-4C61-A54E-F6BAF813F828@math.washington.edu>
	
	
	
	
	
	
	
Message-ID: <3136DF59-DAE3-4279-B3FD-3EB357A3B9E2@math.washington.edu>

On Apr 22, 2009, at 7:18 PM, Mark Lodato wrote:

> The embedded mode is a really good idea, but I think it would make
> more sense as a separate script for several reasons:
> 1. If it is separate, Cython itself doesn't have to change.
> 2. The same Cython C output file be made into a module and also  
> embedded.
> 3. Multiple Cython modules can be embedded into a single executable.
>
> I have included a first draft of such a script.

Any thoughts on shipping this with Cython? (I think it'd be a good  
idea.)

> An additional idea (not implemented) - it would be awesome to have the
> first module listed (that is, the one directly imported) to have
> __name__ == "__main__", so the normal Python idiom works.  I
> researched this for the past few hours, but I could not find a way to
> do this without modifying the module's code.  My best solution is the
> following:
> 1. For each module, mymod.pyx:
> 1a. Add a new externally-visible global variable:
>     int __pyx_mymod_is___main__ = 0;
> 1b. When converting the code for __name__, if __pyx_mymod_is___main__,
> return "__main__", else the module name.
> 2. In the script below, set
>     __pyx_mainmod_is___main__ = 1;
> right before the call to PyImport_ImportModule(), where "mainmod" is
> the main module.

Done. The flag is __pyx_module_is_main_[fullyqualifiedname.replace 
('.', '__')]

- Robert


From robertwb at math.washington.edu  Thu May 14 11:12:55 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 14 May 2009 02:12:55 -0700
Subject: [Cython] Cython complex numbers
In-Reply-To: 
References: <44E38307-17FF-492D-8735-BB8DE5968ADD@math.washington.edu>
	
Message-ID: <7665D0B7-3FA1-4065-8BC7-F057ABB15CF2@math.washington.edu>

On May 14, 2009, at 2:06 AM, Stefan Behnel wrote:

> Robert Bradshaw wrote:
>> I just pushed a series of patches to http://hg.cython.org/cython-
>> devel/ for complex number support. See
>>
>> http://hg.cython.org/cython-devel/file/5c014b865386/tests/run/
>> complex_numbers_T305.pyx .
>>
>> Despite being a rather large change, and shortly before a release, it
>> belongs in -devel because this is one of the motivations to do a
>> release so soon.
>
> Does it (potentially) break any existing stuff or is it a plain new  
> feature?

It's a plain new feature, though I did have to touch a lot of code.

> The latter would make it easy to say: "Sure, let's release it; if the
> feature is broken in some way, there will be a fix release."


- Robert


From dagss at student.matnat.uio.no  Thu May 14 11:25:00 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 14 May 2009 11:25:00 +0200
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: <3136DF59-DAE3-4279-B3FD-3EB357A3B9E2@math.washington.edu>
References: 	<49EB3943.9090409@behnel.de>	<4192B3F1-8CF6-41CA-80EE-42B2D32623B0@math.washington.edu>	<4905F089-AC21-4C61-A54E-F6BAF813F828@math.washington.edu>							
	<3136DF59-DAE3-4279-B3FD-3EB357A3B9E2@math.washington.edu>
Message-ID: <4A0BE36C.9040108@student.matnat.uio.no>

Robert Bradshaw wrote:
> On Apr 22, 2009, at 7:18 PM, Mark Lodato wrote:
> 
>> The embedded mode is a really good idea, but I think it would make
>> more sense as a separate script for several reasons:
>> 1. If it is separate, Cython itself doesn't have to change.
>> 2. The same Cython C output file be made into a module and also  
>> embedded.
>> 3. Multiple Cython modules can be embedded into a single executable.
>>
>> I have included a first draft of such a script.
> 
> Any thoughts on shipping this with Cython? (I think it'd be a good  
> idea.)

+1, though I don't know enough to make the patch for including it.

What's the status of the --embed switch BTW -- is the semantics 
sufficiently set to be in a release (meaning we must consider backwards 
compatability), or are we better off with just disabling the --embed 
switch for now?

-- 
Dag Sverre

From stefan_ml at behnel.de  Thu May 14 11:27:58 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 14 May 2009 11:27:58 +0200 (CEST)
Subject: [Cython] Faster Cython mappings?
In-Reply-To: 
References: <477042.42563.qm@web52110.mail.re2.yahoo.com>
	
Message-ID: <178a1957646172910061be96a5e79750.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Robert Bradshaw wrote:
> On May 14, 2009, at 1:37 AM, Cristi Constantin wrote:
>
>> Good day.
>> I am trying to obtain MAXIMUM speed for this mappings:
>>
>> MyString = '\n'.join([ ''.join([j.encode('utf8') for j in i]) for i
>> in NdArrayList ])

My first try would be to move the .encode() out of the inner loop, to
build one big list instead (including the newlines) and join it at the end
to run the encoding on the complete string result.

If you can afford a dependency on Py2.5+, drop the list comprehensions in
favour of generator expressions (i.e. remove the [] brackets).

No Cython so far, just a faster way to do it in Python space. Both the
encoder run and the method call in the inner most loop above kills
performance here.


>> "NdArrayList" is a list of numpy ndarrays that contain one Unicode
>> character and i want to transform it into a string, AS FAST AS
>> POSSIBLE.
>> For example : NdArrayList can be = [ np.array([u'a', u'\u2588'],
>> dtype='> dtype='>
>> I also want to make MyList = [ ''.join([j.encode('utf8') for j in
>> i]) for i in NdArrayList ]. In this case i need a list of united-as-
>> string numpy ndarrays.
>>
>> When i call the codes from Python i get better performance than
>> compiling into a Cython function like:
>>
>> cdef str __Ndarray2String( list TempA ):
>>     #
>>     return '\n'.join([ ''.join([j.encode('utf8') for j in i]) for i
>> in TempA ])
>>     #
>> def Ndarray2String( v ):
>>     return __Ndarray2String( v )
>>
>> Can anyone please sugest a good method?
>
> To go for maximum speed, I would suggest pulling all the data out of
> the array first (perhaps using the buffer interface) and using
> assignment and/or memcpy to put it into a big string (manually
> inserting the newlines). Then call encode to get the whole thing in
> utf8.

Also, given that the retrieval of each of the substrings has a little
overhead, I'd recommend copying the buffer content directly into a memory
buffer, over-reallocating at need to avoid multiple copying. There are
Python C-API methods that can then encode the resulting PyUNICODE buffers
(assuming that a numpy "
References: 	<49EB3943.9090409@behnel.de>	<4192B3F1-8CF6-41CA-80EE-42B2D32623B0@math.washington.edu>	<4905F089-AC21-4C61-A54E-F6BAF813F828@math.washington.edu>							
	<3136DF59-DAE3-4279-B3FD-3EB357A3B9E2@math.washington.edu>
	<4A0BE36C.9040108@student.matnat.uio.no>
Message-ID: 

On May 14, 2009, at 2:25 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On Apr 22, 2009, at 7:18 PM, Mark Lodato wrote:
>>
>>> The embedded mode is a really good idea, but I think it would make
>>> more sense as a separate script for several reasons:
>>> 1. If it is separate, Cython itself doesn't have to change.
>>> 2. The same Cython C output file be made into a module and also
>>> embedded.
>>> 3. Multiple Cython modules can be embedded into a single executable.
>>>
>>> I have included a first draft of such a script.
>>
>> Any thoughts on shipping this with Cython? (I think it'd be a good
>> idea.)
>
> +1, though I don't know enough to make the patch for including it.
>
> What's the status of the --embed switch BTW -- is the semantics
> sufficiently set to be in a release (meaning we must consider  
> backwards
> compatability), or are we better off with just disabling the --embed
> switch for now?

I think it's good enough--I've disabled it for all but exactly one  
file, so all issues of multi-file compilation are gone, which was the  
only big question.

- Robert



From dagss at student.matnat.uio.no  Thu May 14 14:28:40 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 14 May 2009 14:28:40 +0200
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: <3136DF59-DAE3-4279-B3FD-3EB357A3B9E2@math.washington.edu>
References: 	<49EB3943.9090409@behnel.de>	<4192B3F1-8CF6-41CA-80EE-42B2D32623B0@math.washington.edu>	<4905F089-AC21-4C61-A54E-F6BAF813F828@math.washington.edu>							
	<3136DF59-DAE3-4279-B3FD-3EB357A3B9E2@math.washington.edu>
Message-ID: <4A0C0E78.7090405@student.matnat.uio.no>

Robert Bradshaw wrote:
> 
>> An additional idea (not implemented) - it would be awesome to have the
>> first module listed (that is, the one directly imported) to have
>> __name__ == "__main__", so the normal Python idiom works.  I
>> researched this for the past few hours, but I could not find a way to
>> do this without modifying the module's code.  My best solution is the
>> following:
>> 1. For each module, mymod.pyx:
>> 1a. Add a new externally-visible global variable:
>>     int __pyx_mymod_is___main__ = 0;
>> 1b. When converting the code for __name__, if __pyx_mymod_is___main__,
>> return "__main__", else the module name.
>> 2. In the script below, set
>>     __pyx_mainmod_is___main__ = 1;
>> right before the call to PyImport_ImportModule(), where "mainmod" is
>> the main module.
> 
> Done. The flag is __pyx_module_is_main_[fullyqualifiedname.replace 
> ('.', '__')]
> 

This apparently broke stuff when embed is not turned on. A quick 
hackaround at

http://hg.cython.org/cython-devel/rev/efe5cac433da

but you will have to look.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Thu May 14 15:06:54 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 14 May 2009 15:06:54 +0200
Subject: [Cython] Cython complex numbers
In-Reply-To: <7665D0B7-3FA1-4065-8BC7-F057ABB15CF2@math.washington.edu>
References: <44E38307-17FF-492D-8735-BB8DE5968ADD@math.washington.edu>	
	<7665D0B7-3FA1-4065-8BC7-F057ABB15CF2@math.washington.edu>
Message-ID: <4A0C176E.9000206@student.matnat.uio.no>

Robert Bradshaw wrote:
> On May 14, 2009, at 2:06 AM, Stefan Behnel wrote:
> 
>> Robert Bradshaw wrote:
>>> I just pushed a series of patches to http://hg.cython.org/cython-
>>> devel/ for complex number support. See
>>>
>>> http://hg.cython.org/cython-devel/file/5c014b865386/tests/run/
>>> complex_numbers_T305.pyx .
>>>
>>> Despite being a rather large change, and shortly before a release, it
>>> belongs in -devel because this is one of the motivations to do a
>>> release so soon.
>> Does it (potentially) break any existing stuff or is it a plain new  
>> feature?
> 
> It's a plain new feature, though I did have to touch a lot of code.
> 
>> The latter would make it easy to say: "Sure, let's release it; if the
>> feature is broken in some way, there will be a fix release."

OK I've looked at it and it looks sweet! Buffer support added, it was a 
three line change :-) [1]

I haven't been giving it thorough testing, but as good as it looks I 
suggest that we just release it with an "experimental" note (and 
obviously check whether it breaks existing code in any way).

One thing I miss is the ability to extract real and imaginary parts. I 
assume we want to follow Python here?

cdef double complex r = 1 + 1j
print r.real, r.imag

Tell me if you start on that because it's now on my TODO-list (though I 
might very well not get time before the release).

[1]

I'm making the assumption that C99 complex will be exchangeable with

   struct { T real, imag}

both when it comes to alignment and ordering. Perhaps a runtime check 
for those assumptions should be added at some point.

Without C99 support those assumptions obviously holds anyway.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Thu May 14 15:31:44 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 14 May 2009 15:31:44 +0200
Subject: [Cython] Warnings from setup.py?
In-Reply-To: <2b77a966c4bd38d9a19f6e0baeb83101.squirrel@webmail.uio.no>
References: <4A0BD992.6010802@student.matnat.uio.no>	<812EAB57-4D87-48FF-80E0-C598BF729EE1@math.washington.edu>
	<2b77a966c4bd38d9a19f6e0baeb83101.squirrel@webmail.uio.no>
Message-ID: <4A0C1D40.9070005@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Robert wrote:
>> Is there any reason to even allow this (other than to be backwards
>> compatible with broken code). I think we should allow object <-->
>> void*, or object <--> PyObject*, but no others. (One would then be
>> forced to do arr.data if one *really* wanted to.)
> 
> Excellent idea!; and as you say should not break code. I'll do this for
> 0.11.2.

Done, however there's a slight issue: There's no canonical PyObject.

If somebody has defined it as an empty struct, say, they now have 
problems (but a very helpful error is raised when compiling so it's OK). 
The ones we ship in Includes/ use "ctypedef void PyObject", and they 
still work (as it ends up as void*).

I also removed the warning that used to be emitted for the specific 
cases of pyobj and voidp.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Thu May 14 16:55:49 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 14 May 2009 16:55:49 +0200
Subject: [Cython] #303 + simplifying type system
In-Reply-To: 
References: <4A099E0C.9000909@student.matnat.uio.no>
	
Message-ID: <4A0C30F5.6090101@student.matnat.uio.no>

Lisandro Dalcin wrote:
> On Tue, May 12, 2009 at 1:04 PM, Dag Sverre Seljebotn
>  wrote:
>> A)
>>
>> Ticket #303 manifests itself like this:
>>
>> cdef extern:
>>     ctypedef float footype # really double!
>>
>> cdef class A:
>>     cdef public footype myproperty
>>
>> My suggestion for a fix here is to replace all T_INT, T_UINT etc. with
>> simply three types: __Pyx_T_SIGNED, __Pyx_T_UNSIGNED, __Pyx_T_FLOAT.
>> These would use sizeof to determine the right type to flag the extension
>> type property as (like Lisandro did with T_SIZET already does).
>>
> 
> I'm -1 on this...
> 
> For (A), we need to enhance Cython to generate a couple of converter
> functions to_py/from_py for externally defined integral or floating
> types. For the integral ones, the only requisite is to specify sign
> part. I've already have some preliminary work on this, but I'm not
> sure how to go on because of the delegation-based implementation in
> CTypedefNode.

Sounds like a plan.

I just added a "typedef_is_external" to CTypedefType which could be useful.

If one renamed this to simply "is_external" (or "lacks_size"?) and 
defaulted it to 0 on all types, then the result would be that one could 
simply always query on a conversion whether the type "lacked size", and 
if so use custom converter functions.

Does that solve it?

> For (B), I believe that if the type is an external typedef, a
> transform should change the "cdef publid footype value" to this:
> 
> cdef class A:
>    property value:
>       def __get__(self): return self.value
>       def __set__(self, footype x) : self.value = x
> 
> In short, I already have implemented the utility code for (A), but I'm
> in doubt about how to reimplement CTypedefNode. Additionally, I
> believe my appoach for (B) could be easily implemented (likely Dag can
> implement this part far better than me).

Splendid! This was basically already implemented (by Robert I think) to 
support "cdef public MyCdefClass foo", so the patch was really easy.

http://hg.cython.org/cython-devel/rev/dcc78ab6a498

-- 
Dag Sverre

From tenninosko at msn.com  Thu May 14 17:05:34 2009
From: tenninosko at msn.com (OUARDA MEHDI)
Date: Thu, 14 May 2009 17:05:34 +0200
Subject: [Cython] help...cython code optimisation
Message-ID: 


hello people,
i have just started  my first cython coding (cython version 0.11.1) and i 'd like to optimize at the maximum a python code of matrix calculation based on numpy. cython compiles normally my code when i declare my function as a python one with 'def'.but when i define it as 'cdef'
cython won't compile.
here is my pure python function from 'homogenousmatrix.py' module
------------------------------
# coding=utf-8
"""
Functions for working with homogeneous matrices.

toto
"""
import numpy as np
def rotzyx(angles):
    """homogeneous transformation matrix from pitch-roll-yaw angles)
    
    In short:  R = Rz * Ry * Rx

    example:
    >>> rotzyx((3.14/6, 3.14/4, 3.14/3))
    array([[ 0.61271008,  0.27992274,  0.73907349,  0.        ],
           [ 0.35353151,  0.73930695, -0.57309746,  0.        ],
           [-0.70682518,  0.61242835,  0.35401931,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  1.        ]])
    """
    
    sz = np.sin(angles[0])
    cz = np.cos(angles[0])
    sy = np.sin(angles[1])
    cy = np.cos(angles[1])
    sx = np.sin(angles[2])
    cx = np.cos(angles[2])
    return np.array(
        [[ cz*cy, cz*sy*sx-sz*cx, cz*sy*cx+sz*sx, 0.],
         [ sz*cy, sz*sy*sx+cz*cx, sz*sy*cx-cz*sx, 0.],
         [-sy   , cy*sx         , cy*cx         , 0.],
         [ 0.   , 0.            , 0.            , 1.]])

and here my optimised function from 'homogeneousmatrix_c.pyx' module:

# coding=utf-8
"""
Functions for working with homogeneous matrices.

toto
"""
import numpy as np
#mes modif...python-->cython
# "cimport" is used to import special compile-time information
# about the numpy module (this is stored in a file numpy.pxd which is
# currently part of the Cython distribution).

cimport numpy as np

# We now need to fix a datatype for our arrays. I've used the variable
# DTYPE for this, which is assigned to the usual NumPy runtime
# type info object.

DTYPE = np.float 

# "ctypedef" assigns a corresponding compile-time type to DTYPE_t. For
# every type in the numpy module there's a corresponding compile-time
# type with a _t-suffix.

ctypedef np.float_t DTYPE_t

def np.ndarray [DTYPE_t,ndim=1] rotzyx(np.ndarray [DTYPE_t,ndim=1] angles) :
    """homogeneous transformation matrix from pitch-roll-yaw angles)
    
    In short:  R = Rz * Ry * Rx

    example:
    >>> rotzyx((3.14/6, 3.14/4, 3.14/3))
    array([[ 0.61271008,  0.27992274,  0.73907349,  0.        ],
           [ 0.35353151,  0.73930695, -0.57309746,  0.        ],
           [-0.70682518,  0.61242835,  0.35401931,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  1.        ]])
    """
    
    cdef float sz = np.sin(angles[0])
    cdef float cz = np.cos(angles[0])
    cdef float sy = np.sin(angles[1])
    cdef float cy = np.cos(angles[1])
    cdef float sx = np.sin(angles[2])
    cdef float cx = np.cos(angles[2])

    cdef np.ndarray[DTYPE_t, ndim=2] value = np.array (
        ([ cz*cy, cz*sy*sx-sz*cx, cz*sy*cx+sz*sx, 0.],
         [ sz*cy, sz*sy*sx+cz*cx, sz*sy*cx-cz*sx, 0.],
         [-sy   , cy*sx         , cy*cx         , 0.],
         [ 0.   ,    0.         , 0.            , 1.]),        dtype=DTYPE)

    return value

as i said when i define my rotzyx(..)  function as cpdef i have this as compilation error:

Error converting Pyrex file to C:
------------------------------------------------------------
...

#TAB = np.ndarray([DTYPE_t, ndim=1])
#ctypedef np.ndarray[DTYPE_t, ndim=1]_t TAB_t


cpdef object fun(object.np.ndarray[[DTYPE_t], ndim=1] vect):
                                                 ^
------------------------------------------------------------

/home/tenninos/stage/arboris-python/src/test.pyx:16:50: Expected ']'
building 'test' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c test.c -o build/temp.linux-i686-2.6/test.o
test.c:1:2: error: #error Do not use this file, it is the result of a failed Cython compilation.
error: command 'gcc' failed with exit status 1


please help as quick as its possible...
thanks....


ps: any othre suggestion about best optimisation of the code are very welcome


_________________________________________________________________
In?dit ! Des Emotic?nes D?jant?es! Installez les dans votre Messenger ! 
http://www.ilovemessenger.fr/Emoticones/EmoticonesDejantees.aspx
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090514/007c2c2c/attachment.htm 

From dagss at student.matnat.uio.no  Thu May 14 17:25:14 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 14 May 2009 17:25:14 +0200
Subject: [Cython] help...cython code optimisation
In-Reply-To: 
References: 
Message-ID: <4A0C37DA.9020600@student.matnat.uio.no>

OUARDA MEHDI wrote:
> 
> and here my optimised function from 'homogeneousmatrix_c.pyx' module:
> 
> # coding=utf-8
> """
> Functions for working with homogeneous matrices.
> 
> toto
> """
> import numpy as np
> #mes modif...python-->cython
> # "cimport" is used to import special compile-time information
> # about the numpy module (this is stored in a file numpy.pxd which is
> # currently part of the Cython distribution).
> 
> cimport numpy as np
> 
> # We now need to fix a datatype for our arrays. I've used the variable
> # DTYPE for this, which is assigned to the usual NumPy runtime
> # type info object.
> 
> DTYPE = np.float
> 
> # "ctypedef" assigns a corresponding compile-time type to DTYPE_t. For
> # every type in the numpy module there's a corresponding compile-time
> # type with a _t-suffix.
> 
> ctypedef np.float_t DTYPE_t
> *
> def np.ndarray [DTYPE_t,ndim=1] rotzyx(np.ndarray [DTYPE_t,ndim=1] 
> angles) :*
>     """homogeneous transformation matrix from pitch-roll-yaw angles)
>    
>     In short:  R = Rz * Ry * Rx
> 
>     example:
>     >>> rotzyx((3.14/6, 3.14/4, 3.14/3))
>     array([[ 0.61271008,  0.27992274,  0.73907349,  0.        ],
>            [ 0.35353151,  0.73930695, -0.57309746,  0.        ],
>            [-0.70682518,  0.61242835,  0.35401931,  0.        ],
>            [ 0.        ,  0.        ,  0.        ,  1.        ]])
>     """
>    
>     cdef float sz = np.sin(angles[0])
>     cdef float cz = np.cos(angles[0])
>     cdef float sy = np.sin(angles[1])
>     cdef float cy = np.cos(angles[1])
>     cdef float sx = np.sin(angles[2])
>     cdef float cx = np.cos(angles[2])
> 
>     cdef np.ndarray[DTYPE_t, ndim=2] value = np.array (
>         ([ cz*cy, cz*sy*sx-sz*cx, cz*sy*cx+sz*sx, 0.],
>          [ sz*cy, sz*sy*sx+cz*cx, sz*sy*cx-cz*sx, 0.],
>          [-sy   , cy*sx         , cy*cx         , 0.],
>          [ 0.   ,    0.         , 0.            , 1.]),        dtype=DTYPE)
> 
>     return value
> 
> as i said when i define my rotzyx(..)  function as cpdef i have this as 
> compilation error: 

This will not be faster than in Python.

a) It's only called for one iteration -- you need to bring your loops 
into Cython!

b) You are constructing Python lists which are then converted to 
objects; you must instead first create an array with np.empty and then 
set element by element as arr[i, j] = value.

c) You should call sin and cos from C instead (see Cython docs for 
calling C functions)

d) You cannot type the return value of the function as ndarray[...], 
just drop the return value.

e) Finally, your compilation error points to a line you don't even 
include in your snippet, and which contain an obvious syntax error.

-- 
Dag Sverre

From sccolbert at gmail.com  Thu May 14 17:59:51 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Thu, 14 May 2009 11:59:51 -0400
Subject: [Cython] help...cython code optimisation
In-Reply-To: <4A0C37DA.9020600@student.matnat.uio.no>
References: 
	<4A0C37DA.9020600@student.matnat.uio.no>
Message-ID: <7f014ea60905140859l3ca25c50wc641f8866eadf1a8@mail.gmail.com>

as Dag said, don't use numpy to calculate the trig functions, and don't pass
a numpy array as the argument function. Pass 3 floats instead.

On Thu, May 14, 2009 at 11:25 AM, Dag Sverre Seljebotn <
dagss at student.matnat.uio.no> wrote:

> OUARDA MEHDI wrote:
> >
> > and here my optimised function from 'homogeneousmatrix_c.pyx' module:
> >
> > # coding=utf-8
> > """
> > Functions for working with homogeneous matrices.
> >
> > toto
> > """
> > import numpy as np
> > #mes modif...python-->cython
> > # "cimport" is used to import special compile-time information
> > # about the numpy module (this is stored in a file numpy.pxd which is
> > # currently part of the Cython distribution).
> >
> > cimport numpy as np
> >
> > # We now need to fix a datatype for our arrays. I've used the variable
> > # DTYPE for this, which is assigned to the usual NumPy runtime
> > # type info object.
> >
> > DTYPE = np.float
> >
> > # "ctypedef" assigns a corresponding compile-time type to DTYPE_t. For
> > # every type in the numpy module there's a corresponding compile-time
> > # type with a _t-suffix.
> >
> > ctypedef np.float_t DTYPE_t
> > *
> > def np.ndarray [DTYPE_t,ndim=1] rotzyx(np.ndarray [DTYPE_t,ndim=1]
> > angles) :*
> >     """homogeneous transformation matrix from pitch-roll-yaw angles)
> >
> >     In short:  R = Rz * Ry * Rx
> >
> >     example:
> >     >>> rotzyx((3.14/6, 3.14/4, 3.14/3))
> >     array([[ 0.61271008,  0.27992274,  0.73907349,  0.        ],
> >            [ 0.35353151,  0.73930695, -0.57309746,  0.        ],
> >            [-0.70682518,  0.61242835,  0.35401931,  0.        ],
> >            [ 0.        ,  0.        ,  0.        ,  1.        ]])
> >     """
> >
> >     cdef float sz = np.sin(angles[0])
> >     cdef float cz = np.cos(angles[0])
> >     cdef float sy = np.sin(angles[1])
> >     cdef float cy = np.cos(angles[1])
> >     cdef float sx = np.sin(angles[2])
> >     cdef float cx = np.cos(angles[2])
> >
> >     cdef np.ndarray[DTYPE_t, ndim=2] value = np.array (
> >         ([ cz*cy, cz*sy*sx-sz*cx, cz*sy*cx+sz*sx, 0.],
> >          [ sz*cy, sz*sy*sx+cz*cx, sz*sy*cx-cz*sx, 0.],
> >          [-sy   , cy*sx         , cy*cx         , 0.],
> >          [ 0.   ,    0.         , 0.            , 1.]),
>  dtype=DTYPE)
> >
> >     return value
> >
> > as i said when i define my rotzyx(..)  function as cpdef i have this as
> > compilation error:
>
> This will not be faster than in Python.
>
> a) It's only called for one iteration -- you need to bring your loops
> into Cython!
>
> b) You are constructing Python lists which are then converted to
> objects; you must instead first create an array with np.empty and then
> set element by element as arr[i, j] = value.
>
> c) You should call sin and cos from C instead (see Cython docs for
> calling C functions)
>
> d) You cannot type the return value of the function as ndarray[...],
> just drop the return value.
>
> e) Finally, your compilation error points to a line you don't even
> include in your snippet, and which contain an obvious syntax error.
>
> --
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090514/36538aa3/attachment.htm 

From kxroberto at googlemail.com  Thu May 14 18:05:43 2009
From: kxroberto at googlemail.com (Robert)
Date: Thu, 14 May 2009 18:05:43 +0200
Subject: [Cython] array.array and PIL directly ?
In-Reply-To: <4A0AE4DF.20209@student.matnat.uio.no>
References:  <4A0AE4DF.20209@student.matnat.uio.no>
Message-ID: 

Dag Sverre Seljebotn wrote:
> Robert wrote:
>> How to deal with Python's array.array directly - as with numpy.pxd 
>> is there a array.pxd ?
> 
> Which Python version?

mainly 2.6; and 2.3

> Under Python 3 this should probably happen automatically, try:
> 
> cdef object[int, ndim=1, mode="c"] arr = yourarray
> 
> Not sure about Python 2.6+ 

2.6.2 didn't:
File "calc_c.pyx", line 32, in calc_c.test1 (calc_c.c:808)
     cdef object[float, ndim=1, mode="c"] b = pyarray
TypeError: 'array.array' does not have the buffer interface

in Python "buffer(myarray)" also behaves strange

> For Python 2.5- an array.pxd must be written. It is not difficult, one 
> simply follows the pattern in numpy.pxd (by implementing __getbuffer__ 
> and filling in the Py_buffer struct).
> 
> If somebody ends up doing this, please submit it for inclusion in 
> Python. 

I've put a array.pxd here:
http://trac.cython.org/cython_trac/ticket/314

--
Cython direct interface to Python's array.array type (builtin module).

     * 1D contiguous data view
     * 2D views - contigious or [x,y] transposed/strided
     * tools for fast array creation, maximum C-speed and handiness
     * suitable as allround light weight auto-array within Cython 
code too

     See also: array_example.pyx and doc strings
--

Tests needed. just used it in 2.6 so far.

Robert


PS: For 2.3 a need arose for conditional compilation within the pxd.
http://docs.cython.org/docs/language_basics.html#conditional-compilation
doesn't seem to allow comparison against the python version so far? )


From dagss at student.matnat.uio.no  Thu May 14 18:19:03 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 14 May 2009 18:19:03 +0200
Subject: [Cython] array.array and PIL directly ?
In-Reply-To: 
References:  <4A0AE4DF.20209@student.matnat.uio.no>
	
Message-ID: <4A0C4477.8020300@student.matnat.uio.no>

Robert wrote:
> I've put a array.pxd here:
> http://trac.cython.org/cython_trac/ticket/314
> 
> --
> Cython direct interface to Python's array.array type (builtin module).
> 
>      * 1D contiguous data view
>      * 2D views - contigious or [x,y] transposed/strided
>      * tools for fast array creation, maximum C-speed and handiness
>      * suitable as allround light weight auto-array within Cython 
> code too
> 
>      See also: array_example.pyx and doc strings
> --
> 
> Tests needed. just used it in 2.6 so far.

Wow, great! That's a lot more sophistication than I imagined in there :-)

Tests would be needed before inclusion though, as you already have demo 
code it should be easy to add. Given that someone writes tests for this 
(perhaps I even do it myself if I need to) I think it's safe to say that 
this can enter Cython (in 0.11.3 or 0.12).

Small nitpick: It would be good if you could use /* */ comments in the 
.h-file as // is not portable.

> PS: For 2.3 a need arose for conditional compilation within the pxd.
> http://docs.cython.org/docs/language_basics.html#conditional-compilation
> doesn't seem to allow comparison against the python version so far? )

The problem is that the Python that runs Cython is not always the Python 
that the C file is compiled against...

I think just dropping Py 2.3 for this feature would be OK.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Thu May 14 18:25:36 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 14 May 2009 18:25:36 +0200
Subject: [Cython] array.array and PIL directly ?
In-Reply-To: <4A0C4477.8020300@student.matnat.uio.no>
References: 
	<4A0AE4DF.20209@student.matnat.uio.no>	
	<4A0C4477.8020300@student.matnat.uio.no>
Message-ID: <4A0C4600.5020004@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Robert wrote:
>> I've put a array.pxd here:
>> http://trac.cython.org/cython_trac/ticket/314
>>
>> --
>> Cython direct interface to Python's array.array type (builtin module).
>>
>>      * 1D contiguous data view
>>      * 2D views - contigious or [x,y] transposed/strided
>>      * tools for fast array creation, maximum C-speed and handiness
>>      * suitable as allround light weight auto-array within Cython 
>> code too
>>
>>      See also: array_example.pyx and doc strings
>> --
>>
>> Tests needed. just used it in 2.6 so far.
> 
> Wow, great! That's a lot more sophistication than I imagined in there :-)

Looking a bit more, I wonder if the view2D-functionality fits within a 
Cython version of the pxd (it is, and needs to be by its nature, a bit 
hackish).

Perhaps it could be reintroduced later as a wrapper class or subclass of 
array.array.

Another possibility long-term would be somehow adding a "view with a 
different shape"-functionality to Cython itself. Perhaps incorporated in

http://wiki.cython.org/enhancements/buffersyntax

somehow -- perhaps

cdef int[:] arr1d = contigarray(10)
cdef int[:,:] arr2d = arr1d # ok since 5*2==10

-- 
Dag Sverre

From dalcinl at gmail.com  Thu May 14 19:06:25 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 14 May 2009 14:06:25 -0300
Subject: [Cython] #303 + simplifying type system
In-Reply-To: <4A0C30F5.6090101@student.matnat.uio.no>
References: <4A099E0C.9000909@student.matnat.uio.no>
	
	<4A0C30F5.6090101@student.matnat.uio.no>
Message-ID: 

On Thu, May 14, 2009 at 11:55 AM, Dag Sverre Seljebotn
 wrote:
>
> I just added a "typedef_is_external" to CTypedefType which could be useful.
>
> If one renamed this to simply "is_external" (or "lacks_size"?) and
> defaulted it to 0 on all types, then the result would be that one could
> simply always query on a conversion whether the type "lacked size", and
> if so use custom converter functions.
>
> Does that solve it?
>

That would help a lot. I'll try to dive further on this.


>> For (B), I believe that if the type is an external typedef, a
>> transform should change the "cdef publid footype value" to this:
>>
>> cdef class A:
>> ? ?property value:
>> ? ? ? def __get__(self): return self.value
>> ? ? ? def __set__(self, footype x) : self.value = x
>>
>> In short, I already have implemented the utility code for (A), but I'm
>> in doubt about how to reimplement CTypedefNode. Additionally, I
>> believe my appoach for (B) could be easily implemented (likely Dag can
>> implement this part far better than me).
>
> Splendid! This was basically already implemented (by Robert I think) to
> support "cdef public MyCdefClass foo", so the patch was really easy.
>
> http://hg.cython.org/cython-devel/rev/dcc78ab6a498
>
> --

Your patch looks good.

BTW, further work on this would require to push my patch for #287.

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Thu May 14 19:13:30 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 14 May 2009 14:13:30 -0300
Subject: [Cython] Warnings from setup.py?
In-Reply-To: <4A0C1D40.9070005@student.matnat.uio.no>
References: <4A0BD992.6010802@student.matnat.uio.no>
	<812EAB57-4D87-48FF-80E0-C598BF729EE1@math.washington.edu>
	<2b77a966c4bd38d9a19f6e0baeb83101.squirrel@webmail.uio.no>
	<4A0C1D40.9070005@student.matnat.uio.no>
Message-ID: 

Note however that in my own code I use this:

cdef extern from "Python.h":
    ctypedef struct PyObject


In general, typedef'ing things to void* is a really bad idea, I always
try hard to avoid it. See the code below, Cython will (correctly) not
complain

cdef extern from "header.h":
   ctypedef void PyObject
   void Py_INCREF(PyObject*)

cdef int value = 0

Py_INCREF(&value)



On Thu, May 14, 2009 at 10:31 AM, Dag Sverre Seljebotn
 wrote:
> Dag Sverre Seljebotn wrote:
>> Robert wrote:
>>> Is there any reason to even allow this (other than to be backwards
>>> compatible with broken code). I think we should allow object <-->
>>> void*, or object <--> PyObject*, but no others. (One would then be
>>> forced to do arr.data if one *really* wanted to.)
>>
>> Excellent idea!; and as you say should not break code. I'll do this for
>> 0.11.2.
>
> Done, however there's a slight issue: There's no canonical PyObject.
>
> If somebody has defined it as an empty struct, say, they now have
> problems (but a very helpful error is raised when compiling so it's OK).
> The ones we ship in Includes/ use "ctypedef void PyObject", and they
> still work (as it ends up as void*).
>
> I also removed the warning that used to be emitted for the specific
> cases of pyobj and voidp.
>
> --
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Thu May 14 19:15:24 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 14 May 2009 19:15:24 +0200
Subject: [Cython] #303 + simplifying type system
In-Reply-To: 
References: <4A099E0C.9000909@student.matnat.uio.no>		<4A0C30F5.6090101@student.matnat.uio.no>
	
Message-ID: <4A0C51AC.7020409@student.matnat.uio.no>

Lisandro Dalcin wrote:
> Your patch looks good.
> 
> BTW, further work on this would require to push my patch for #287.

As you are confident in it I think we should just go ahead, and I'll 
include it in the release candidate testing. We are already way past a 
"bugfix-only" kind of release (that was never the intention with 0.11.2 
either). Will you push?

So, for the release notes, your changes in this release sums up (from 
the end-user perspective as):

- Converting negative Python integers to unsigned C types now results in 
OverflowError (or was that 0.11.1?)

- When converting an object to C integers, __int__ is considered before 
__long__ regardless of the target type.

Correct?

--
Dag Sverre

From dagss at student.matnat.uio.no  Thu May 14 19:18:18 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 14 May 2009 19:18:18 +0200
Subject: [Cython] Warnings from setup.py?
In-Reply-To: 
References: <4A0BD992.6010802@student.matnat.uio.no>	<812EAB57-4D87-48FF-80E0-C598BF729EE1@math.washington.edu>	<2b77a966c4bd38d9a19f6e0baeb83101.squirrel@webmail.uio.no>	<4A0C1D40.9070005@student.matnat.uio.no>
	
Message-ID: <4A0C525A.9020008@student.matnat.uio.no>

Lisandro Dalcin wrote:
> Note however that in my own code I use this:
> 
> cdef extern from "Python.h":
>     ctypedef struct PyObject
> 
> 
> In general, typedef'ing things to void* is a really bad idea, I always
> try hard to avoid it. See the code below, Cython will (correctly) not
> complain

I backtracked on this now (as that declaration was used in testcases as 
well). You can now do obj and obj, but not 
obj or obj.

Slightly better, though long-term I'd prefer a canonical builtin 
PyObject and only allow it for that one.

(In python_ref.pxd the contents of PyObject is available as well, and 
there's not really a way to seperate it from other structs unless we ... 
chech the name *shudder*).

-- 
Dag Sverre

From dalcinl at gmail.com  Thu May 14 19:19:35 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 14 May 2009 14:19:35 -0300
Subject: [Cython] Warnings from setup.py?
In-Reply-To: <4A0C525A.9020008@student.matnat.uio.no>
References: <4A0BD992.6010802@student.matnat.uio.no>
	<812EAB57-4D87-48FF-80E0-C598BF729EE1@math.washington.edu>
	<2b77a966c4bd38d9a19f6e0baeb83101.squirrel@webmail.uio.no>
	<4A0C1D40.9070005@student.matnat.uio.no>
	
	<4A0C525A.9020008@student.matnat.uio.no>
Message-ID: 

On Thu, May 14, 2009 at 2:18 PM, Dag Sverre Seljebotn
 wrote:
>
> Slightly better, though long-term I'd prefer a canonical builtin
> PyObject and only allow it for that one.
>

Me too. Definitely +1 on this.

-

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Thu May 14 19:28:14 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 14 May 2009 14:28:14 -0300
Subject: [Cython] #303 + simplifying type system
In-Reply-To: <4A0C51AC.7020409@student.matnat.uio.no>
References: <4A099E0C.9000909@student.matnat.uio.no>
	
	<4A0C30F5.6090101@student.matnat.uio.no>
	
	<4A0C51AC.7020409@student.matnat.uio.no>
Message-ID: 

On Thu, May 14, 2009 at 2:15 PM, Dag Sverre Seljebotn
 wrote:
> Lisandro Dalcin wrote:
>> Your patch looks good.
>>
>> BTW, further work on this would require to push my patch for #287.
>
> As you are confident in it I think we should just go ahead, and I'll
> include it in the release candidate testing. We are already way past a
> "bugfix-only" kind of release (that was never the intention with 0.11.2
> either). Will you push?
>

Yes, I can push after testing nothing breaks.

> So, for the release notes, your changes in this release sums up (from
> the end-user perspective as):
>
> - Converting negative Python integers to unsigned C types now results in
> OverflowError (or was that 0.11.1?)
>

It seems this was the last patch pushed before 0.11.1

> - When converting an object to C integers, __int__ is considered before
> __long__ regardless of the target type.
>

Yes.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Thu May 14 20:18:53 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 14 May 2009 11:18:53 -0700
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: <4A0C0E78.7090405@student.matnat.uio.no>
References: 	<49EB3943.9090409@behnel.de>	<4192B3F1-8CF6-41CA-80EE-42B2D32623B0@math.washington.edu>	<4905F089-AC21-4C61-A54E-F6BAF813F828@math.washington.edu>							
	<3136DF59-DAE3-4279-B3FD-3EB357A3B9E2@math.washington.edu>
	<4A0C0E78.7090405@student.matnat.uio.no>
Message-ID: 

On May 14, 2009, at 5:28 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>>
>>> An additional idea (not implemented) - it would be awesome to  
>>> have the
>>> first module listed (that is, the one directly imported) to have
>>> __name__ == "__main__", so the normal Python idiom works.  I
>>> researched this for the past few hours, but I could not find a  
>>> way to
>>> do this without modifying the module's code.  My best solution is  
>>> the
>>> following:
>>> 1. For each module, mymod.pyx:
>>> 1a. Add a new externally-visible global variable:
>>>     int __pyx_mymod_is___main__ = 0;
>>> 1b. When converting the code for __name__, if  
>>> __pyx_mymod_is___main__,
>>> return "__main__", else the module name.
>>> 2. In the script below, set
>>>     __pyx_mainmod_is___main__ = 1;
>>> right before the call to PyImport_ImportModule(), where "mainmod" is
>>> the main module.
>>
>> Done. The flag is __pyx_module_is_main_[fullyqualifiedname.replace
>> ('.', '__')]
>>
>
> This apparently broke stuff when embed is not turned on. A quick
> hackaround at
>
> http://hg.cython.org/cython-devel/rev/efe5cac433da
>
> but you will have to look.

Oops. Yes.

- Robert



From dagss at student.matnat.uio.no  Thu May 14 20:22:28 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 14 May 2009 20:22:28 +0200
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: 
References: 	<49EB3943.9090409@behnel.de>	<4192B3F1-8CF6-41CA-80EE-42B2D32623B0@math.washington.edu>	<4905F089-AC21-4C61-A54E-F6BAF813F828@math.washington.edu>								<3136DF59-DAE3-4279-B3FD-3EB357A3B9E2@math.washington.edu>	<4A0C0E78.7090405@student.matnat.uio.no>
	
Message-ID: <4A0C6164.4010203@student.matnat.uio.no>

Robert Bradshaw wrote:
> Oops. Yes.

While you're at it, could you review this one as well? Slightly unsure 
about it.

http://hg.cython.org/cython-devel/rev/f0d9b51d7ec8

-- 
Dag Sverre

From robertwb at math.washington.edu  Thu May 14 20:28:26 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 14 May 2009 11:28:26 -0700
Subject: [Cython] Cython complex numbers
In-Reply-To: <4A0C176E.9000206@student.matnat.uio.no>
References: <44E38307-17FF-492D-8735-BB8DE5968ADD@math.washington.edu>	
	<7665D0B7-3FA1-4065-8BC7-F057ABB15CF2@math.washington.edu>
	<4A0C176E.9000206@student.matnat.uio.no>
Message-ID: <74B8A4D9-FD7C-439D-ACCD-B126FB844453@math.washington.edu>

On May 14, 2009, at 6:06 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On May 14, 2009, at 2:06 AM, Stefan Behnel wrote:
>>
>>> Robert Bradshaw wrote:
>>>> I just pushed a series of patches to http://hg.cython.org/cython-
>>>> devel/ for complex number support. See
>>>>
>>>> http://hg.cython.org/cython-devel/file/5c014b865386/tests/run/
>>>> complex_numbers_T305.pyx .
>>>>
>>>> Despite being a rather large change, and shortly before a  
>>>> release, it
>>>> belongs in -devel because this is one of the motivations to do a
>>>> release so soon.
>>> Does it (potentially) break any existing stuff or is it a plain new
>>> feature?
>>
>> It's a plain new feature, though I did have to touch a lot of code.
>>
>>> The latter would make it easy to say: "Sure, let's release it; if  
>>> the
>>> feature is broken in some way, there will be a fix release."
>
> OK I've looked at it and it looks sweet! Buffer support added, it  
> was a
> three line change :-) [1]

Just saw it, that was fast. Does this still work with the complex.h  
header turned on as well (I don't see you using the __Pyx_REAL_PART  
macros).

> I haven't been giving it thorough testing, but as good as it looks I
> suggest that we just release it with an "experimental" note (and
> obviously check whether it breaks existing code in any way).
>
> One thing I miss is the ability to extract real and imaginary parts. I
> assume we want to follow Python here?
>
> cdef double complex r = 1 + 1j
> print r.real, r.imag
>
> Tell me if you start on that because it's now on my TODO-list  
> (though I
> might very well not get time before the release).

Oh, yeah, I meant to do this.

> [1]
>
> I'm making the assumption that C99 complex will be exchangeable with
>
>    struct { T real, imag}
>
> both when it comes to alignment and ordering.

I believe so:

http://c0x.coding-guidelines.com/6.2.5.html
http://home.datacomm.ch/t_wolf/tw/c/c9x_changes.html

The implementation of the complex types is defined by the standard  
(6.2.5(13)) to use cartesian coordinates (real and imaginary part),  
i.e. forbids an implementation using polar coordinates (distance from  
[0,0] and an angle). Furthermore, the same paragraph also specifies  
that a complex type has the same alignment requirements as an array  
of two elements of the corresponding floating types, the first must  
be the real part and the second the imaginary part.

> Perhaps a runtime check
> for those assumptions should be added at some point.

Yeah, that would make things a log messier...

> Without C99 support those assumptions obviously holds anyway.

Yep.

- Robert


From robertwb at math.washington.edu  Thu May 14 20:37:04 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 14 May 2009 11:37:04 -0700
Subject: [Cython] #303 + simplifying type system
In-Reply-To: <4A0C30F5.6090101@student.matnat.uio.no>
References: <4A099E0C.9000909@student.matnat.uio.no>
	
	<4A0C30F5.6090101@student.matnat.uio.no>
Message-ID: 

On May 14, 2009, at 7:55 AM, Dag Sverre Seljebotn wrote:

> Lisandro Dalcin wrote:
>>
>> For (B), I believe that if the type is an external typedef, a
>> transform should change the "cdef publid footype value" to this:
>>
>> cdef class A:
>>    property value:
>>       def __get__(self): return self.value
>>       def __set__(self, footype x) : self.value = x
>>
>> In short, I already have implemented the utility code for (A), but  
>> I'm
>> in doubt about how to reimplement CTypedefNode. Additionally, I
>> believe my appoach for (B) could be easily implemented (likely Dag  
>> can
>> implement this part far better than me).
>
> Splendid! This was basically already implemented (by Robert I  
> think) to
> support "cdef public MyCdefClass foo",

Yep.

> so the patch was really easy.
>
> http://hg.cython.org/cython-devel/rev/dcc78ab6a498

Does this have a performance impact compared to the other way? It  
does bloat the code some. (Maybe it's not critical anyways.)

- Robert


From robertwb at math.washington.edu  Thu May 14 20:38:57 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 14 May 2009 11:38:57 -0700
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: <4A0C6164.4010203@student.matnat.uio.no>
References: 	<49EB3943.9090409@behnel.de>	<4192B3F1-8CF6-41CA-80EE-42B2D32623B0@math.washington.edu>	<4905F089-AC21-4C61-A54E-F6BAF813F828@math.washington.edu>								<3136DF59-DAE3-4279-B3FD-3EB357A3B9E2@math.washington.edu>	<4A0C0E78.7090405@student.matnat.uio.no>
	
	<4A0C6164.4010203@student.matnat.uio.no>
Message-ID: <7447DB6F-5F79-4172-9E92-1E9FA92367E1@math.washington.edu>

On May 14, 2009, at 11:22 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> Oops. Yes.
>
> While you're at it, could you review this one as well? Slightly unsure
> about it.
>
> http://hg.cython.org/cython-devel/rev/f0d9b51d7ec8

I was trying to avoid calling create_*_utility_code many times,  
because it may be slowish.

- Robert



From dagss at student.matnat.uio.no  Thu May 14 20:54:48 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 14 May 2009 20:54:48 +0200
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: <7447DB6F-5F79-4172-9E92-1E9FA92367E1@math.washington.edu>
References: 	<49EB3943.9090409@behnel.de>	<4192B3F1-8CF6-41CA-80EE-42B2D32623B0@math.washington.edu>	<4905F089-AC21-4C61-A54E-F6BAF813F828@math.washington.edu>								<3136DF59-DAE3-4279-B3FD-3EB357A3B9E2@math.washington.edu>	<4A0C0E78.7090405@student.matnat.uio.no>		<4A0C6164.4010203@student.matnat.uio.no>
	<7447DB6F-5F79-4172-9E92-1E9FA92367E1@math.washington.edu>
Message-ID: <4A0C68F8.3080905@student.matnat.uio.no>

Robert Bradshaw wrote:
> On May 14, 2009, at 11:22 AM, Dag Sverre Seljebotn wrote:
> 
>> Robert Bradshaw wrote:
>>> Oops. Yes.
>> While you're at it, could you review this one as well? Slightly unsure
>> about it.
>>
>> http://hg.cython.org/cython-devel/rev/f0d9b51d7ec8
> 
> I was trying to avoid calling create_*_utility_code many times,  
> because it may be slowish.

Sure, and the bug stumped me for quite a while. If you can do it without 
modifying PyrexType objects (e.g. store the state in env instead) then 
feel free.

If we only were in code generation land, you could simply do

if not token in code.globalstate.utility_codes: ...


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Thu May 14 21:01:33 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 14 May 2009 21:01:33 +0200
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: <4A0C68F8.3080905@student.matnat.uio.no>
References: 	<49EB3943.9090409@behnel.de>	<4192B3F1-8CF6-41CA-80EE-42B2D32623B0@math.washington.edu>	<4905F089-AC21-4C61-A54E-F6BAF813F828@math.washington.edu>								<3136DF59-DAE3-4279-B3FD-3EB357A3B9E2@math.washington.edu>	<4A0C0E78.7090405@student.matnat.uio.no>		<4A0C6164.4010203@student.matnat.uio.no>	<7447DB6F-5F79-4172-9E92-1E9FA92367E1@math.washington.edu>
	<4A0C68F8.3080905@student.matnat.uio.no>
Message-ID: <4A0C6A8D.1090308@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Robert Bradshaw wrote:
>> On May 14, 2009, at 11:22 AM, Dag Sverre Seljebotn wrote:
>>
>>> Robert Bradshaw wrote:
>>>> Oops. Yes.
>>> While you're at it, could you review this one as well? Slightly unsure
>>> about it.
>>>
>>> http://hg.cython.org/cython-devel/rev/f0d9b51d7ec8
>> I was trying to avoid calling create_*_utility_code many times,  
>> because it may be slowish.
> 
> Sure, and the bug stumped me for quite a while. If you can do it without 
> modifying PyrexType objects (e.g. store the state in env instead) then 
> feel free.
> 
> If we only were in code generation land, you could simply do
> 
> if not token in code.globalstate.utility_codes: ...

Whoops, sorry, commented on the wrong changeset!!

OK, I see, that is not where the problem was, I can readd the attribute 
checks. (The problem was create*utility_code always returning True, thus 
making the if impossible to enter.) Thanks!

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Thu May 14 21:04:56 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 14 May 2009 21:04:56 +0200
Subject: [Cython] Cython complex numbers
In-Reply-To: <74B8A4D9-FD7C-439D-ACCD-B126FB844453@math.washington.edu>
References: <44E38307-17FF-492D-8735-BB8DE5968ADD@math.washington.edu>		<7665D0B7-3FA1-4065-8BC7-F057ABB15CF2@math.washington.edu>	<4A0C176E.9000206@student.matnat.uio.no>
	<74B8A4D9-FD7C-439D-ACCD-B126FB844453@math.washington.edu>
Message-ID: <4A0C6B58.6080700@student.matnat.uio.no>

Robert Bradshaw wrote:
> On May 14, 2009, at 6:06 AM, Dag Sverre Seljebotn wrote:
> 
>> Robert Bradshaw wrote:
>>> On May 14, 2009, at 2:06 AM, Stefan Behnel wrote:
>>>
>>>> Robert Bradshaw wrote:
>>>>> I just pushed a series of patches to http://hg.cython.org/cython-
>>>>> devel/ for complex number support. See
>>>>>
>>>>> http://hg.cython.org/cython-devel/file/5c014b865386/tests/run/
>>>>> complex_numbers_T305.pyx .
>>>>>
>>>>> Despite being a rather large change, and shortly before a  
>>>>> release, it
>>>>> belongs in -devel because this is one of the motivations to do a
>>>>> release so soon.
>>>> Does it (potentially) break any existing stuff or is it a plain new
>>>> feature?
>>> It's a plain new feature, though I did have to touch a lot of code.
>>>
>>>> The latter would make it easy to say: "Sure, let's release it; if  
>>>> the
>>>> feature is broken in some way, there will be a fix release."
>> OK I've looked at it and it looks sweet! Buffer support added, it  
>> was a
>> three line change :-) [1]
> 
> Just saw it, that was fast. Does this still work with the complex.h  
> header turned on as well (I don't see you using the __Pyx_REAL_PART  
> macros).

The (marginal) tests work, both with and without complex.h.

Basically all that needs to be done is to accept the format string, and 
a cast of a pointer; the actual access happen through 
(__pyx_..complex*)(bufferptr).

So the links in the C99 spec pretty much takes care of it all, as the 
buffer pointer can just be casted then as well.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Thu May 14 21:08:05 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 14 May 2009 21:08:05 +0200
Subject: [Cython] #303 + simplifying type system
In-Reply-To: 
References: <4A099E0C.9000909@student.matnat.uio.no>		<4A0C30F5.6090101@student.matnat.uio.no>
	
Message-ID: <4A0C6C15.3090400@student.matnat.uio.no>

Robert Bradshaw wrote:
> On May 14, 2009, at 7:55 AM, Dag Sverre Seljebotn wrote:
> 
>> Lisandro Dalcin wrote:
>>> For (B), I believe that if the type is an external typedef, a
>>> transform should change the "cdef publid footype value" to this:
>>>
>>> cdef class A:
>>>    property value:
>>>       def __get__(self): return self.value
>>>       def __set__(self, footype x) : self.value = x
>>>
>>> In short, I already have implemented the utility code for (A), but  
>>> I'm
>>> in doubt about how to reimplement CTypedefNode. Additionally, I
>>> believe my appoach for (B) could be easily implemented (likely Dag  
>>> can
>>> implement this part far better than me).
>> Splendid! This was basically already implemented (by Robert I  
>> think) to
>> support "cdef public MyCdefClass foo",
> 
> Yep.
> 
>> so the patch was really easy.
>>
>> http://hg.cython.org/cython-devel/rev/dcc78ab6a498
> 
> Does this have a performance impact compared to the other way? It  
> does bloat the code some. (Maybe it's not critical anyways.)

Well, it is only used for external typedefs, and considering that the 
bug is quite critical (basically it's very easy to do *(long 
long*)&some_short)...

Haven't benchmarked it, we could consider switching to my original 
proposal later..Lisandro, would you like to argue why you were -1?

-- 
Dag Sverre

From dalcinl at gmail.com  Thu May 14 21:58:27 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 14 May 2009 16:58:27 -0300
Subject: [Cython] #303 + simplifying type system
In-Reply-To: <4A0C6C15.3090400@student.matnat.uio.no>
References: <4A099E0C.9000909@student.matnat.uio.no>
	
	<4A0C30F5.6090101@student.matnat.uio.no>
	
	<4A0C6C15.3090400@student.matnat.uio.no>
Message-ID: 

On Thu, May 14, 2009 at 4:08 PM, Dag Sverre Seljebotn
 wrote:
> Robert Bradshaw wrote:
>>
>> Does this have a performance impact compared to the other way? It
>> does bloat the code some. (Maybe it's not critical anyways.)
>

Give me a bit more of time to give a definitive comment on this, but ...

> Well, it is only used for external typedefs, and considering that the
> bug is quite critical (basically it's very easy to do *(long
> long*)&some_short)...
>
> Haven't benchmarked it, we could consider switching to my original
> proposal later..
>

... as long as the "property" is defined with a C protocol (instead of
the Python-side "property" builtin), I do not expect a impact on
performance.

>
>Lisandro, would you like to argue why you were -1?
>

For the same rationale that othes gave on this thread (basically, the
impact on C semantics), and because I believe a better and backward
compatible solution can be found.

Simplifying  the whole type system seems a mistake to me, in the
temptation to guess, I would just guess ONLY in the case of external
ctypedef's. Moreover, we could extend this to make "ctypedef signed
value" as an unknown-size integral, but if the code does "ctypedef int
value", assume that the ctypedef is actually a "int" and treat it like
that. Of course, we have a problem with floating poing. Then perhaps,
you should enable a special declarations for integrals and floatings
like this:

ctypedef signed integral i
ctypedef unsigned integral j
ctypedef floating x # or "floatingpoint", or "inexact"

and ONLY in such case we treat such declarations as unknown-size types.

Does this make sense?

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Thu May 14 22:05:25 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 14 May 2009 22:05:25 +0200
Subject: [Cython] #303 + simplifying type system
In-Reply-To: 
References: <4A099E0C.9000909@student.matnat.uio.no>		<4A0C30F5.6090101@student.matnat.uio.no>		<4A0C6C15.3090400@student.matnat.uio.no>
	
Message-ID: <4A0C7985.3090203@student.matnat.uio.no>

Lisandro Dalcin wrote:
> On Thu, May 14, 2009 at 4:08 PM, Dag Sverre Seljebotn
>  wrote:
>> Robert Bradshaw wrote:
>>> Does this have a performance impact compared to the other way? It
>>> does bloat the code some. (Maybe it's not critical anyways.)
> 
> Give me a bit more of time to give a definitive comment on this, but ...
> 
>> Well, it is only used for external typedefs, and considering that the
>> bug is quite critical (basically it's very easy to do *(long
>> long*)&some_short)...
>>
>> Haven't benchmarked it, we could consider switching to my original
>> proposal later..
>>
> 
> ... as long as the "property" is defined with a C protocol (instead of
> the Python-side "property" builtin), I do not expect a impact on
> performance.
> 
>> Lisandro, would you like to argue why you were -1?
>>
> 
> For the same rationale that othes gave on this thread (basically, the
> impact on C semantics), and because I believe a better and backward
> compatible solution can be found.
> 
> Simplifying  the whole type system seems a mistake to me, in the
> temptation to guess, I would just guess ONLY in the case of external
> ctypedef's. Moreover, we could extend this to make "ctypedef signed
> value" as an unknown-size integral, but if the code does "ctypedef int
> value", assume that the ctypedef is actually a "int" and treat it like
> that. Of course, we have a problem with floating poing. Then perhaps,
> you should enable a special declarations for integrals and floatings
> like this:
> 
> ctypedef signed integral i
> ctypedef unsigned integral j
> ctypedef floating x # or "floatingpoint", or "inexact"
> 
> and ONLY in such case we treat such declarations as unknown-size types.
> 
> Does this make sense?

Sure :-) But I was mainly referring to

"""
My suggestion for a fix here is to replace all T_INT, T_UINT etc. with
simply three types: __Pyx_T_SIGNED, __Pyx_T_UNSIGNED, __Pyx_T_FLOAT.
These would use sizeof to determine the right type to flag the extension
type property as (like Lisandro did with T_SIZET already does).
"""

This is ONLY used when you do

cdef class A:
     cdef public sometype FOO

I.e. in my point (A) I was only arguing collapsing types for the purpose 
of exporting properties (through the CPython member structures).

If you misunderstood my intentions I suppose that answers the question...

Dag Sverre

From dalcinl at gmail.com  Thu May 14 23:32:59 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 14 May 2009 18:32:59 -0300
Subject: [Cython] #303 + simplifying type system
In-Reply-To: <4A0C7985.3090203@student.matnat.uio.no>
References: <4A099E0C.9000909@student.matnat.uio.no>
	
	<4A0C30F5.6090101@student.matnat.uio.no>
	
	<4A0C6C15.3090400@student.matnat.uio.no>
	
	<4A0C7985.3090203@student.matnat.uio.no>
Message-ID: 

Wow... I'm very sorry. I got it totally wrong ...

If the change you are proposing about simplifying the type system only
affect that, I'm actually +1 on your proposal. The canonical Python
way is a mess in the context of Cython.

I would just ask for the following:

1) When going from C to Python, for Python 2.X, try to avoid returning
a PyLong (basically dispach proper calls using
sizeof(type)<=sizeof(long) )

2) Use __Pyx_T_FLOATING, just to avoid confussion.


Of course, other way would be to just use the transform I proposed
previously. I do not expect it to impact performance.


On Thu, May 14, 2009 at 5:05 PM, Dag Sverre Seljebotn
 wrote:
> Lisandro Dalcin wrote:
>> On Thu, May 14, 2009 at 4:08 PM, Dag Sverre Seljebotn
>>  wrote:
>>> Robert Bradshaw wrote:
>>>> Does this have a performance impact compared to the other way? It
>>>> does bloat the code some. (Maybe it's not critical anyways.)
>>
>> Give me a bit more of time to give a definitive comment on this, but ...
>>
>>> Well, it is only used for external typedefs, and considering that the
>>> bug is quite critical (basically it's very easy to do *(long
>>> long*)&some_short)...
>>>
>>> Haven't benchmarked it, we could consider switching to my original
>>> proposal later..
>>>
>>
>> ... as long as the "property" is defined with a C protocol (instead of
>> the Python-side "property" builtin), I do not expect a impact on
>> performance.
>>
>>> Lisandro, would you like to argue why you were -1?
>>>
>>
>> For the same rationale that othes gave on this thread (basically, the
>> impact on C semantics), and because I believe a better and backward
>> compatible solution can be found.
>>
>> Simplifying ?the whole type system seems a mistake to me, in the
>> temptation to guess, I would just guess ONLY in the case of external
>> ctypedef's. Moreover, we could extend this to make "ctypedef signed
>> value" as an unknown-size integral, but if the code does "ctypedef int
>> value", assume that the ctypedef is actually a "int" and treat it like
>> that. Of course, we have a problem with floating poing. Then perhaps,
>> you should enable a special declarations for integrals and floatings
>> like this:
>>
>> ctypedef signed integral i
>> ctypedef unsigned integral j
>> ctypedef floating x # or "floatingpoint", or "inexact"
>>
>> and ONLY in such case we treat such declarations as unknown-size types.
>>
>> Does this make sense?
>
> Sure :-) But I was mainly referring to
>
> """
> My suggestion for a fix here is to replace all T_INT, T_UINT etc. with
> simply three types: __Pyx_T_SIGNED, __Pyx_T_UNSIGNED, __Pyx_T_FLOAT.
> These would use sizeof to determine the right type to flag the extension
> type property as (like Lisandro did with T_SIZET already does).
> """
>
> This is ONLY used when you do
>
> cdef class A:
> ? ? cdef public sometype FOO
>
> I.e. in my point (A) I was only arguing collapsing types for the purpose
> of exporting properties (through the CPython member structures).
>
> If you misunderstood my intentions I suppose that answers the question...
>
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Thu May 14 23:37:39 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 14 May 2009 18:37:39 -0300
Subject: [Cython] C++ keywords
Message-ID: 

Should we also mangle names (look at Symtab.py) if they are C++
keywords? http://en.wikibooks.org/wiki/C%2B%2B_Programming/Code/Keywords

Dag pushed a testcase using "mutable" as a struct field name ;-) ...
It is very easy to forget about C++ when coding and testing using just
C ...

The C++ keywords that are not C99 keywords and Python keywords are
listed below (BTW, at some point we should support bool as a native
type, even in C):

and_eq
asm
bitand
bitor
bool
catch
compl
const_cast
delete
dynamic_cast
explicit
export
false
friend
mutable
namespace
new
not_eq
operator
or_eq
private
protected
public
reinterpret_cast
static_cast
template
this
throw
true
typeid
typename
using
virtual
wchar_t
xor
xor_eq


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From kxroberto at googlemail.com  Fri May 15 00:17:41 2009
From: kxroberto at googlemail.com (Robert)
Date: Fri, 15 May 2009 00:17:41 +0200
Subject: [Cython] array.array and PIL directly ?
In-Reply-To: <4A0C4600.5020004@student.matnat.uio.no>
References: 	<4A0AE4DF.20209@student.matnat.uio.no>		<4A0C4477.8020300@student.matnat.uio.no>
	<4A0C4600.5020004@student.matnat.uio.no>
Message-ID: 

Dag Sverre Seljebotn wrote:
> Dag Sverre Seljebotn wrote:
>> Robert wrote:
>>> I've put a array.pxd here:
>>> http://trac.cython.org/cython_trac/ticket/314
>>>
>>> --
>>> Cython direct interface to Python's array.array type (builtin module).
>>>
>>>      * 1D contiguous data view
>>>      * 2D views - contigious or [x,y] transposed/strided
>>>      * tools for fast array creation, maximum C-speed and handiness
>>>      * suitable as allround light weight auto-array within Cython 
>>> code too
>>>
>>>      See also: array_example.pyx and doc strings
>>> --
>>>
>>> Tests needed. just used it in 2.6 so far.
>> Wow, great! That's a lot more sophistication than I imagined in there :-)
> 
> Looking a bit more, I wonder if the view2D-functionality fits within a 
> Cython version of the pxd (it is, and needs to be by its nature, a bit 
> hackish).

yes, its just a hack to get the row info into the __getbuffer__ 
for the view.
Its just some 20 lines without speed penalty on the 1D. I added it 
more out of curiosity to check out the strided/shape thing of this 
buffer interface ;) - but the 'view' issue seemed to be somewhat 
reasonable principally.
Besides the niceness/readability of [x,y], a 2D view is brings the 
option of 2D boundscheck during debug, negative indexex, ...


> 
> Perhaps it could be reintroduced later as a wrapper class or subclass of 
> array.array.

Yes a subclass with extra storage for the 2D info would be good 
new thing beyond a local view - passing the object around with 
self-knowledge. But accepted everywhere throughout Python, where 
an array.array does.
In array.pxd its just about a local in-place view (without need 
for copy/instantiation of new objects)

I just tried this kind of scheme ...

cdef class array2(array.array):
     cdef Py_ssize_t columns
     # 0:1dim; >0:[row,column]; <0 [column,row]
     def __new__(self, char* typec, initdata=None,
                 Py_ssize_t columns = 0):
         print "hello __new__", typec
     def __init__(self, ctypec,  initdata=None,
                  Py_ssize_t columns = 0):
         print "hello __init__", typec, type(typec), columns
         self.columns=columns
     def reshape(self, Py_ssize_t columns):
         self.columns=columns
     def __getbuffer__(array2 self, Py_buffer* info, int flags):
	...

.. and it seems already to work principally as expected with all 
1D, 2D, python compatibility... (Just the 3rd init parameter will 
never be accepted upon usage - a reshape call always necessary)

Maybe a array2.pxi - kind of fast light-weight numpy within Cython?

In 90%+ cases I just need such light-weight things as these 2 
classes; actual computations along the axes is rush anyway in 
Cython; and using the little Python memory management (through C 
calls) enables to mostly remove the heavy-handed temporary 
stack/malloc/free array stuff from Cython code.


> Another possibility long-term would be somehow adding a "view with a 
> different shape"-functionality to Cython itself. Perhaps incorporated in
> 
> http://wiki.cython.org/enhancements/buffersyntax
> 
> somehow -- perhaps
> 
> cdef int[:] arr1d = contigarray(10)
> cdef int[:,:] arr2d = arr1d # ok since 5*2==10
> 

that would make a clear 'cast' syntax out the view issue.
Yet would a dynamic column information go (readable) into a cast 
operator like this?:

cdef int[:,:] arr2d = arr1d

the info would have to be passed somehow to the __getbuffer__()


-

note: the cdef extern "memset" etc stuff in array.pxd should 
perhaps move over to stdlib.pxd - as far as within ANSI C


Robert


From kxroberto at googlemail.com  Fri May 15 00:36:51 2009
From: kxroberto at googlemail.com (Robert)
Date: Fri, 15 May 2009 00:36:51 +0200
Subject: [Cython] array.array and PIL directly ?
In-Reply-To: 
References: 	<4A0AE4DF.20209@student.matnat.uio.no>		<4A0C4477.8020300@student.matnat.uio.no>	<4A0C4600.5020004@student.matnat.uio.no>
	
Message-ID: 

Robert wrote:
> In array.pxd its just about a local in-place view (without need 
> for copy/instantiation of new objects)

BTW, why is it disallowed to enrich .pxd/cdef extern classes with 
new cdef member functions? as it is possible just by exception 
with __getbuffer__ ?

For example I wanted to put the copy_array() etc inline functions 
as class members..

even if the member function is declared inline like

cdef extern from "arrayarray.h":
     ctypedef class mod.xy[object xy]:
         ...
         cdef inline int f():
             return 1


there is still the error:

  "function definition in pxd file must be declared 'cdef inline'"


From dalcinl at gmail.com  Fri May 15 00:56:42 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 14 May 2009 19:56:42 -0300
Subject: [Cython] Implementation of complex division for the struct case
Message-ID: 

I used to teach numerical calculus, so I cannot resist :-)

The current implementation of complex division for the struct case is
the dangerous one, it uses:

2 divides
6 multiplies
3 additions

A safer way, far less sensible to roundoff/overflow (looks at core
CPython's complexobject.c, and skip the checks for dividing by zero),
would likely require:

2 comparison
3 divides
3 multiplies
3 additions

So the key question is (having a serious headache right now, hope I
got the number above and below right):

Are 2 comparison and 1 divide SLOWER than 3 multiplies ??

If the answer is NO, we should clearly use the SAFER implementation.
If the answer is YES, should we trade SPEED over roundoff/overflow issues?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Fri May 15 01:07:21 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 14 May 2009 16:07:21 -0700
Subject: [Cython] Implementation of complex division for the struct case
In-Reply-To: 
References: 
Message-ID: 

On May 14, 2009, at 3:56 PM, Lisandro Dalcin wrote:

> I used to teach numerical calculus, so I cannot resist :-)

Thanks for the heads up--I deal mostly with exact arithmetic (e.g.  
when writing that code I had the gaussian integers in my head) so  
it's good to have an expert speak up.

> The current implementation of complex division for the struct case is
> the dangerous one, it uses:
>
> 2 divides
> 6 multiplies
> 3 additions
>
> A safer way, far less sensible to roundoff/overflow (looks at core
> CPython's complexobject.c, and skip the checks for dividing by zero),
> would likely require:
>
> 2 comparison
> 3 divides
> 3 multiplies
> 3 additions
>
> So the key question is (having a serious headache right now, hope I
> got the number above and below right):
>
> Are 2 comparison and 1 divide SLOWER than 3 multiplies ??
>
> If the answer is NO, we should clearly use the SAFER implementation.
> If the answer is YES, should we trade SPEED over roundoff/overflow  
> issues?

No idea--please post some benchmarks. It looks like more than 2  
comparisons, as one has to take absolute values too. Branching can be  
expensive, but it's clearly not obvious enough to tell just by  
talking about it. If they're close, safer is better for sure,  
otherwise we'll have to really have to weight the pros and cons.

- Robert



From carl.witty at gmail.com  Fri May 15 01:31:53 2009
From: carl.witty at gmail.com (Carl Witty)
Date: Thu, 14 May 2009 16:31:53 -0700
Subject: [Cython] Implementation of complex division for the struct case
In-Reply-To: 
References:  
	
Message-ID: 

On Thu, May 14, 2009 at 4:07 PM, Robert Bradshaw
 wrote:
> On May 14, 2009, at 3:56 PM, Lisandro Dalcin wrote:
>
>> I used to teach numerical calculus, so I cannot resist :-)
>
> Thanks for the heads up--I deal mostly with exact arithmetic (e.g.
> when writing that code I had the gaussian integers in my head) so
> it's good to have an expert speak up.
>
>> The current implementation of complex division for the struct case is
>> the dangerous one, it uses:
>>
>> 2 divides
>> 6 multiplies
>> 3 additions
>>
>> A safer way, far less sensible to roundoff/overflow (looks at core
>> CPython's complexobject.c, and skip the checks for dividing by zero),
>> would likely require:
>>
>> 2 comparison
>> 3 divides
>> 3 multiplies
>> 3 additions
>>
>> So the key question is (having a serious headache right now, hope I
>> got the number above and below right):
>>
>> Are 2 comparison and 1 divide SLOWER than 3 multiplies ??
>>
>> If the answer is NO, we should clearly use the SAFER implementation.
>> If the answer is YES, should we trade SPEED over roundoff/overflow
>> issues?
>
> No idea--please post some benchmarks. It looks like more than 2
> comparisons, as one has to take absolute values too. Branching can be
> expensive, but it's clearly not obvious enough to tell just by
> talking about it. If they're close, safer is better for sure,
> otherwise we'll have to really have to weight the pros and cons.

Absolute value need not require a comparison.  On x86 when using the
old 8087-based floating point model, there is an fabs instruction; in
other cases, you can do absolute value with a bitwise AND (and gcc
does so, at least when using SSE2 for the x86).

The cost of comparisons and branches will depend a lot on context.
(Does the branch go the same way almost every time a division is
performed?  Are divisions performed often enough to keep the branches
in the branch prediction cache?)

Divisions are very expensive, but the relative cost of division and
multiplication will obviously depend on the architecture, etc.

In short: Yes, you should do benchmarks :)

But benchmarks should be careful to have a realistic mix of numbers to
divide, to properly test the branch prediction.  For example, compute
a couple of arrays of random complexes, and then benchmark elementwise
division of these arrays (assuming that we think that random complex
numbers are "realistic").

Carl

From dalcinl at gmail.com  Fri May 15 01:48:02 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 14 May 2009 20:48:02 -0300
Subject: [Cython] Implementation of complex division for the struct case
In-Reply-To: 
References: 
	
	
Message-ID: 

On Thu, May 14, 2009 at 8:31 PM, Carl Witty  wrote:
> On Thu, May 14, 2009 at 4:07 PM, Robert Bradshaw
>  wrote:
>> On May 14, 2009, at 3:56 PM, Lisandro Dalcin wrote:
>>
>>> I used to teach numerical calculus, so I cannot resist :-)
>>
>> Thanks for the heads up--I deal mostly with exact arithmetic (e.g.
>> when writing that code I had the gaussian integers in my head) so
>> it's good to have an expert speak up.

I'm far from being an expect on this :-)

>
> Divisions are very expensive, but the relative cost of division and
> multiplication will obviously depend on the architecture, etc.
>
> In short: Yes, you should do benchmarks :)
>

OK. Still, IMHO we should prefer safety on this. After all, all this
is only relevant for the struct case. I would expect any compiler
providing C99 complex support will implement complex division in the
right and optimized way...

However, interestingly enough, my C++  header (from GCC
4.3.2) seems to implements the naive way (although warned in the
comment)... Am I missing something here??

  // 26.2.5/15
  // XXX: This is a grammar school implementation.
  template
    template
    complex<_Tp>&
    complex<_Tp>::operator/=(const complex<_Up>& __z)
    {
      const _Tp __r =  _M_real * __z.real() + _M_imag * __z.imag();
      const _Tp __n = std::norm(__z);
      _M_imag = (_M_imag * __z.real() - _M_real * __z.imag()) / __n;
      _M_real = __r / __n;
      return *this;
    }



> But benchmarks should be careful to have a realistic mix of numbers to
> divide, to properly test the branch prediction. ?For example, compute
> a couple of arrays of random complexes, and then benchmark elementwise
> division of these arrays (assuming that we think that random complex
> numbers are "realistic").
>
> Carl
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Fri May 15 01:51:33 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 14 May 2009 20:51:33 -0300
Subject: [Cython] ctypedef for integrals
Message-ID: 

After Robert changes for supporting complex, things are now far easuer IMHO

See the attached patch, testcase included. Disclaimer: this is a
completely unpolished hack written in 10 minutes before going home.
But now I'm even more confident we will find our way :-)


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
-------------- next part --------------
A non-text attachment was scrubbed...
Name: tpdef-int-types.diff
Type: application/octet-stream
Size: 16635 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090514/1b6c62c0/attachment-0001.obj 

From lrhazi at gmail.com  Fri May 15 02:43:08 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Thu, 14 May 2009 20:43:08 -0400
Subject: [Cython] Help with pointer to array of structs...
Message-ID: 

Hello,

I have a C -lib function requring a params  in this form:  struct
iovec *hdr_names[]
When I declare the above function, as from external .h file, I use the
same syntax, minus the word strurct: iovec *hdr_names[]

Now to create this data, I do this:

cdef iovec *hdr_names=NULL
hdr_names=malloc(total_count * sizeof(iovec))

then go some loops and checks, filling the iovecs like this:

hdr_names[ii].iov_base = hdr_n
hdr_names[ii].iov_len = len(hdr_n)


then dump eveything just to be sure, like this:

for ii in range(total_count):
    print "hdr_names[%d] : "%ii,
PyString_FromStringAndSize(hdr_names[ii].iov_base,
hdr_names[ii].iov_len)
    print "hdr_values[%d]: "%ii,
PyString_FromStringAndSize(hdr_values[ii].iov_base,
hdr_values[ii].iov_len)

then pass it to the C function like this: modify(hdr_names)

That fails to compile: Cannot assign type 'iovec *' to 'iovec **'
So I try: &hdr_names

This compiles, but causes my program to crash with sigfault, right at
that function call.

I tried various other ideas.... but what is the correct way
anyways.... I might have other issues causing the sigfaults, but
wanted to be sure about the very syntax..

Thanks a lot,
Mohamed.

From carl.witty at gmail.com  Fri May 15 03:12:39 2009
From: carl.witty at gmail.com (Carl Witty)
Date: Thu, 14 May 2009 18:12:39 -0700
Subject: [Cython] Implementation of complex division for the struct case
In-Reply-To: 
References:  
	 
	 
	
Message-ID: 

On Thu, May 14, 2009 at 4:48 PM, Lisandro Dalcin  wrote:
> However, interestingly enough, my C++  header (from GCC
> 4.3.2) seems to implements the naive way (although warned in the
> comment)... Am I missing something here??

GSL (the GNU Scientific Library) also uses the naive algorithm.
Perhaps everybody else has decided that speed is more important than
accuracy, or perhaps they haven't thought of the issues at all?

How much difference in accuracy are we talking about?

(BTW, the Sage arbitrary-precision complex and complex interval
arithmetic also use the naive algorithm; if it is significantly bad,
then Sage should be fixed too.)

On the other hand, I think that gcc's "complex double" does not use
the naive algorithm.  (I haven't traced through the assembly language
to see exactly what it does, but it does have conditional branches, if
I'm reading the x86 assembly correctly.)

Carl

From dalcinl at gmail.com  Fri May 15 03:49:55 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 14 May 2009 22:49:55 -0300
Subject: [Cython] Help with pointer to array of structs...
In-Reply-To: 
References: 
Message-ID: 

Could you tell me the EXACT type definition for "iovec" in your C header file?

---

On Thu, May 14, 2009 at 9:43 PM, Mohamed Lrhazi  wrote:
> Hello,
>
> I have a C -lib function requring a params ?in this form: ?struct
> iovec *hdr_names[]
> When I declare the above function, as from external .h file, I use the
> same syntax, minus the word strurct: iovec *hdr_names[]
>
> Now to create this data, I do this:
>
> cdef iovec *hdr_names=NULL
> hdr_names=malloc(total_count * sizeof(iovec))
>
> then go some loops and checks, filling the iovecs like this:
>
> hdr_names[ii].iov_base = hdr_n
> hdr_names[ii].iov_len = len(hdr_n)
>
>
> then dump eveything just to be sure, like this:
>
> for ii in range(total_count):
> ? ?print "hdr_names[%d] : "%ii,
> PyString_FromStringAndSize(hdr_names[ii].iov_base,
> hdr_names[ii].iov_len)
> ? ?print "hdr_values[%d]: "%ii,
> PyString_FromStringAndSize(hdr_values[ii].iov_base,
> hdr_values[ii].iov_len)
>
> then pass it to the C function like this: modify(hdr_names)
>
> That fails to compile: Cannot assign type 'iovec *' to 'iovec **'
> So I try: &hdr_names
>
> This compiles, but causes my program to crash with sigfault, right at
> that function call.
>
> I tried various other ideas.... but what is the correct way
> anyways.... I might have other issues causing the sigfaults, but
> wanted to be sure about the very syntax..
>
> Thanks a lot,
> Mohamed.
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From lrhazi at gmail.com  Fri May 15 03:58:32 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Thu, 14 May 2009 21:58:32 -0400
Subject: [Cython] Help with pointer to array of structs...
In-Reply-To: 
References: 
	
Message-ID: 

On Thu, May 14, 2009 at 9:49 PM, Lisandro Dalcin  wrote:
> Could you tell me the EXACT type definition for "iovec" in your C header file?
>

I defined it like this:

ctypedef struct iovec:
    void *iov_base
    size_t iov_len


> ---
>
> On Thu, May 14, 2009 at 9:43 PM, Mohamed Lrhazi  wrote:
>> Hello,
>>
>> I have a C -lib function requring a params ?in this form: ?struct
>> iovec *hdr_names[]
>> When I declare the above function, as from external .h file, I use the
>> same syntax, minus the word strurct: iovec *hdr_names[]
>>
>> Now to create this data, I do this:
>>
>> cdef iovec *hdr_names=NULL
>> hdr_names=malloc(total_count * sizeof(iovec))
>>
>> then go some loops and checks, filling the iovecs like this:
>>
>> hdr_names[ii].iov_base = hdr_n
>> hdr_names[ii].iov_len = len(hdr_n)
>>
>>
>> then dump eveything just to be sure, like this:
>>
>> for ii in range(total_count):
>> ? ?print "hdr_names[%d] : "%ii,
>> PyString_FromStringAndSize(hdr_names[ii].iov_base,
>> hdr_names[ii].iov_len)
>> ? ?print "hdr_values[%d]: "%ii,
>> PyString_FromStringAndSize(hdr_values[ii].iov_base,
>> hdr_values[ii].iov_len)
>>
>> then pass it to the C function like this: modify(hdr_names)
>>
>> That fails to compile: Cannot assign type 'iovec *' to 'iovec **'
>> So I try: &hdr_names
>>
>> This compiles, but causes my program to crash with sigfault, right at
>> that function call.
>>
>> I tried various other ideas.... but what is the correct way
>> anyways.... I might have other issues causing the sigfaults, but
>> wanted to be sure about the very syntax..
>>
>> Thanks a lot,
>> Mohamed.
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
" Logic merely sanctions the conquests of the intuition."
Jacques Hadamard

From lrhazi at gmail.com  Fri May 15 04:01:52 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Thu, 14 May 2009 22:01:52 -0400
Subject: [Cython] Help with pointer to array of structs...
In-Reply-To: 
References: 
	
	
Message-ID: 

On Thu, May 14, 2009 at 9:58 PM, Mohamed Lrhazi  wrote:
> On Thu, May 14, 2009 at 9:49 PM, Lisandro Dalcin  wrote:
>> Could you tell me the EXACT type definition for "iovec" in your C header file?
>>
>
> I defined it like this:
>
> ctypedef struct iovec:
> ? ?void *iov_base
> ? ?size_t iov_len

and the C header file does not actually define it.. it must be in a
standard system includes?

From lrhazi at gmail.com  Fri May 15 04:04:55 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Thu, 14 May 2009 22:04:55 -0400
Subject: [Cython] Help with pointer to array of structs...
In-Reply-To: 
References: 
	
	
	
Message-ID: 

On Thu, May 14, 2009 at 10:01 PM, Mohamed Lrhazi  wrote:
> On Thu, May 14, 2009 at 9:58 PM, Mohamed Lrhazi  wrote:
>> On Thu, May 14, 2009 at 9:49 PM, Lisandro Dalcin  wrote:
>>> Could you tell me the EXACT type definition for "iovec" in your C header file?


>>>
>>
>> I defined it like this:
>>
>> ctypedef struct iovec:
>> ? ?void *iov_base
>> ? ?size_t iov_len
>
> and the C header file does not actually define it.. it must be in a
> standard system includes?
>


Yes.. it must be:

/* Structure for scatter/gather I/O.  */
struct iovec
  {
    void *iov_base;     /* Pointer to data.  */
    size_t iov_len;     /* Length of data.  */
  };


>From : /usr/include/bits/uio.h

Thanks a lot
Mohamed.

From robertwb at math.washington.edu  Fri May 15 04:25:22 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 14 May 2009 19:25:22 -0700
Subject: [Cython] Implementation of complex division for the struct case
In-Reply-To: 
References: 
	
	
	
	
Message-ID: <914BAB8B-1303-4AD2-83D6-35630CBA7CBD@math.washington.edu>

On May 14, 2009, at 6:12 PM, Carl Witty wrote:

> On Thu, May 14, 2009 at 4:48 PM, Lisandro Dalcin  
>  wrote:
>> However, interestingly enough, my C++  header (from GCC
>> 4.3.2) seems to implements the naive way (although warned in the
>> comment)... Am I missing something here??
>
> GSL (the GNU Scientific Library) also uses the naive algorithm.
> Perhaps everybody else has decided that speed is more important than
> accuracy, or perhaps they haven't thought of the issues at all?
>
> How much difference in accuracy are we talking about?

Not much, but a little:

C = ComplexField(100)
total = 100000

naive_avg = 0
naive_worst = 0
naive_better = 0

other_avg = 0
other_worst = 0
other_better = 0

for i in range(total):
     a = C.random_element()
     b = C.random_element()
     naive_error = abs(C(CDF(a)/CDF(b)) - a/b)/abs(a/b)
     other_error = abs(C(complex(a)/complex(b)) - a/b)/abs(a/b)
     #print naive_error / other_error

     naive_better += naive_error < other_error
     naive_avg += naive_error / total
     naive_worst = max(naive_worst, naive_error)

     other_better += naive_error > other_error
     other_avg += other_error / total
     other_worst = max(other_worst, other_error)

print "naive\n    better", naive_better, "\n    avg", naive_avg,  
"\n    worst", naive_worst
print "other\n    better", other_better, "\n    avg", other_avg,  
"\n    worst", other_worst

naive
     better 26187
     avg 1.4940916064705992895601085724e-16
     worst 5.7659574333851360909896621025e-16
other
     better 63116
     avg 9.5414951065097745299683547276e-17
     worst 3.9584519821557590591785765975e-16

> (BTW, the Sage arbitrary-precision complex and complex interval
> arithmetic also use the naive algorithm; if it is significantly bad,
> then Sage should be fixed too.)

I think the biggest issue is overflow/underflow, which we  
(essentially) don't have to worry about in the arbitrary-precision case.

> On the other hand, I think that gcc's "complex double" does not use
> the naive algorithm.  (I haven't traced through the assembly language
> to see exactly what it does, but it does have conditional branches, if
> I'm reading the x86 assembly correctly.)

Interesting. In terms of branching on abs, we can do a bitwise and  
(but I'm not sure how to write good generic code to do this in both  
the floating point and integer case).

- Robert


From carl.witty at gmail.com  Fri May 15 05:03:37 2009
From: carl.witty at gmail.com (Carl Witty)
Date: Thu, 14 May 2009 20:03:37 -0700
Subject: [Cython] Implementation of complex division for the struct case
In-Reply-To: <914BAB8B-1303-4AD2-83D6-35630CBA7CBD@math.washington.edu>
References:  
	 
	 
	 
	 
	<914BAB8B-1303-4AD2-83D6-35630CBA7CBD@math.washington.edu>
Message-ID: 

On Thu, May 14, 2009 at 7:25 PM, Robert Bradshaw
 wrote:
> Interesting. In terms of branching on abs, we can do a bitwise and
> (but I'm not sure how to write good generic code to do this in both
> the floating point and integer case).

In the floating point case, just calling fabs() should do the right thing.

In the integer case, it's not just a bitwise and.  (Although you can
do abs with bitwise operations.)  And also, in the integer case, you
can't always perform an abs; in a fixed-precision 2's complement
format, the absolute value of the most negative integer cannot be
represented.

Carl

From dalcinl at gmail.com  Fri May 15 05:10:00 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 15 May 2009 00:10:00 -0300
Subject: [Cython] Implementation of complex division for the struct case
In-Reply-To: 
References: 
	
	
	
	
Message-ID: 

On Thu, May 14, 2009 at 10:12 PM, Carl Witty  wrote:
> On Thu, May 14, 2009 at 4:48 PM, Lisandro Dalcin  wrote:
>> However, interestingly enough, my C++  header (from GCC
>> 4.3.2) seems to implements the naive way (although warned in the
>> comment)... Am I missing something here??
>
> GSL (the GNU Scientific Library) also uses the naive algorithm.
> Perhaps everybody else has decided that speed is more important than
> accuracy, or perhaps they haven't thought of the issues at all?
>

I bet that the second one is the case ;-)

>
> How much difference in accuracy are we talking about?
>

Basically, we have two issues: overflow and cancelation.

* Overflow will occur if the real or imaginary part of the denominator
has exponent > 1/2 max_exponent; for example real|imag parts are above
1e20 for float or 1e155 for double

* Cancellation can occur on subtraction when computing the imaginary component.

See yourself the effects of all this in single precision:

import numpy

def cdiv(z1, z2):
    z1 = numpy.array(z1, dtype='F')
    z2 = numpy.array(z2, dtype='F')
    a, b = z1.real, z1.imag
    c, d = z2.real, z2.imag
    nrm = c*c + d*d
    z = numpy.array(0, dtype='F')
    z.real = (a*c + b*d) / nrm
    z.imag = (b*c - a*d) / nrm
    return z

def test(z1, z2):
    print ('z1 = %s' % z1)
    print ('z2 = %s' % z2)
    print ('z1/z2 (naive) = %s' % cdiv(z1,z2))
    print ('z1/z2 (smart) = %s' % (z1/z2))
    print("")

if __name__ == "__main__":
    z1 = 1 + 2j
    z2 = 3 + 4j
    test(z1, z2)


    z1 = 0 + 1e20j
    z2 = 1e20 + 0j
    test(z1, z2)

    z1 = 1 + (1+1e-7)*1j
    z2 = 1 + 1j
    test(z1, z2)

I get this output (Win32, Py2.6, NumPy 1.3.0rc1)

z1 = (1+2j)
z2 = (3+4j)
z1/z2 (naive) = (0.439999997616+0.0799999982119j)
z1/z2 (smart) = (0.44+0.08j)

z1 = 1e+20j
z2 = (1e+20+0j)
z1/z2 (naive) = nan*j
z1/z2 (smart) = 1j

z1 = (1+1.0000001j)
z2 = (1+1j)
z1/z2 (naive) = (1+5.96046447754e-08j)
z1/z2 (smart) = (1.00000005+5.00000000292e-08j)

In the second bit, overflow generated a NaN. In the third bit,
cancellation produced an imaginary component 20% larger.


> (BTW, the Sage arbitrary-precision complex and complex interval
> arithmetic also use the naive algorithm; if it is significantly bad,
> then Sage should be fixed too.)
>
> On the other hand, I think that gcc's "complex double" does not use
> the naive algorithm. ?(I haven't traced through the assembly language
> to see exactly what it does, but it does have conditional branches, if
> I'm reading the x86 assembly correctly.)
>
> Carl
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Fri May 15 05:21:49 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 15 May 2009 00:21:49 -0300
Subject: [Cython] Help with pointer to array of structs...
In-Reply-To: 
References: 
	
	
	
	
Message-ID: 

Then your Cython definition does not match the C definition. Could you
try to fake things in the Cython side like below: ?

cdef extern ... :
    ctypedef struct iovec "struct iovec":
       void *iov_base
       size_t iov_len

If this fails, you will likely have to define like this:

cdef extern ... :
    cdef struct iovec:
       void *iov_base
       size_t iov_len

and always use "struct iovec" as the type, like this:

cdef struct iovec *hdr_names=NULL
hdr_names=malloc(total_count * sizeof(struct iovec))


Hope any of all this work. If it does not, come back.


On Thu, May 14, 2009 at 11:04 PM, Mohamed Lrhazi  wrote:
> On Thu, May 14, 2009 at 10:01 PM, Mohamed Lrhazi  wrote:
>> On Thu, May 14, 2009 at 9:58 PM, Mohamed Lrhazi  wrote:
>>> On Thu, May 14, 2009 at 9:49 PM, Lisandro Dalcin  wrote:
>>>> Could you tell me the EXACT type definition for "iovec" in your C header file?
>
>
>>>>
>>>
>>> I defined it like this:
>>>
>>> ctypedef struct iovec:
>>> ? ?void *iov_base
>>> ? ?size_t iov_len
>>
>> and the C header file does not actually define it.. it must be in a
>> standard system includes?
>>
>
>
> Yes.. it must be:
>
> /* Structure for scatter/gather I/O. ?*/
> struct iovec
> ?{
> ? ?void *iov_base; ? ? /* Pointer to data. ?*/
> ? ?size_t iov_len; ? ? /* Length of data. ?*/
> ?};
>
>
> >From : /usr/include/bits/uio.h
>
> Thanks a lot
> Mohamed.
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Fri May 15 05:30:20 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 15 May 2009 00:30:20 -0300
Subject: [Cython] Implementation of complex division for the struct case
In-Reply-To: <914BAB8B-1303-4AD2-83D6-35630CBA7CBD@math.washington.edu>
References: 
	
	
	
	
	<914BAB8B-1303-4AD2-83D6-35630CBA7CBD@math.washington.edu>
Message-ID: 

On Thu, May 14, 2009 at 11:25 PM, Robert Bradshaw
 wrote:
>
> Not much, but a little:
>
>
> for i in range(total):
> ? ? a = C.random_element()
> ? ? b = C.random_element()

>
> naive
> ? ? better 26187
> ? ? avg 1.4940916064705992895601085724e-16
> ? ? worst 5.7659574333851360909896621025e-16
> other
> ? ? better 63116
> ? ? avg 9.5414951065097745299683547276e-17
> ? ? worst 3.9584519821557590591785765975e-16
>

Any chance that these complex values are all inside a unit-length
square near the origin of the complex plane?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Fri May 15 05:30:37 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 15 May 2009 00:30:37 -0300
Subject: [Cython] Implementation of complex division for the struct case
In-Reply-To: <914BAB8B-1303-4AD2-83D6-35630CBA7CBD@math.washington.edu>
References: 
	
	
	
	
	<914BAB8B-1303-4AD2-83D6-35630CBA7CBD@math.washington.edu>
Message-ID: 

On Thu, May 14, 2009 at 11:25 PM, Robert Bradshaw
 wrote:
>
> Not much, but a little:
>
>
> for i in range(total):
> ? ? a = C.random_element()
> ? ? b = C.random_element()

>
> naive
> ? ? better 26187
> ? ? avg 1.4940916064705992895601085724e-16
> ? ? worst 5.7659574333851360909896621025e-16
> other
> ? ? better 63116
> ? ? avg 9.5414951065097745299683547276e-17
> ? ? worst 3.9584519821557590591785765975e-16
>

Any chance that these complex values are all inside a unit-length
square near the origin of the complex plane?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Fri May 15 05:39:34 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 14 May 2009 20:39:34 -0700
Subject: [Cython] Implementation of complex division for the struct case
In-Reply-To: 
References: 
	
	
	
	
	<914BAB8B-1303-4AD2-83D6-35630CBA7CBD@math.washington.edu>
	
Message-ID: <05A43AFC-E96E-4867-B6C0-D7A0BF485E67@math.washington.edu>

On May 14, 2009, at 8:30 PM, Lisandro Dalcin wrote:

> On Thu, May 14, 2009 at 11:25 PM, Robert Bradshaw
>  wrote:
>>
>> Not much, but a little:
>>
>>
>> for i in range(total):
>>     a = C.random_element()
>>     b = C.random_element()
>
>>
>> naive
>>     better 26187
>>     avg 1.4940916064705992895601085724e-16
>>     worst 5.7659574333851360909896621025e-16
>> other
>>     better 63116
>>     avg 9.5414951065097745299683547276e-17
>>     worst 3.9584519821557590591785765975e-16
>>
>
> Any chance that these complex values are all inside a unit-length
> square near the origin of the complex plane?

Yes. I meant to mention that.

- Robert

From lrhazi at gmail.com  Fri May 15 06:03:07 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Fri, 15 May 2009 00:03:07 -0400
Subject: [Cython] Help with pointer to array of structs...
In-Reply-To: 
References: 
	
	
	
	
	
Message-ID: 

On Thu, May 14, 2009 at 11:21 PM, Lisandro Dalcin  wrote:
> Then your Cython definition does not match the C definition. Could you
> try to fake things in the Cython side like below: ?
>
> cdef extern ... :
> ? ?ctypedef struct iovec "struct iovec":
> ? ? ? void *iov_base
> ? ? ? size_t iov_len
>
> If this fails, you will likely have to define like this:
>
> cdef extern ... :
> ? ?cdef struct iovec:
> ? ? ? void *iov_base
> ? ? ? size_t iov_len
>
> and always use "struct iovec" as the type, like this:
>
> cdef struct iovec *hdr_names=NULL
> hdr_names=malloc(total_count * sizeof(struct iovec))
>
>
> Hope any of all this work. If it does not, come back.
>

I tried first with commented version, then the uncommented:

cdef extern from "sys/uio.h":
    # ctypedef struct iovec "struct iovec":
      # void *iov_base
      # size_t iov_len
    cdef struct iovec:
        void *iov_base
        size_t iov_len


With the first, it fails whenever iovecs are declared,  like this:

uint32_t max_pending_trans
char *traffic_class
struct iovec *req_hdrs
                    ^
------------------------------------------------------------

p_types.pxi:130:21: Syntax error in C variable declaration

The faulty line above is a declaration of field inside another struct.


With the second, it compiles, but then my program sigfaults. I should
add that it never sigfaults if I only have one value in the passed in
array of structs, only if there are more.

Thanks a lot,
Mohamed.

From robertwb at math.washington.edu  Fri May 15 06:13:35 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 14 May 2009 21:13:35 -0700
Subject: [Cython] Implementation of complex division for the struct case
In-Reply-To: 
References: 
	
	
	
	
	
Message-ID: <8BA81C3B-290D-454C-9DFF-AD60E078455B@math.washington.edu>

On May 14, 2009, at 8:10 PM, Lisandro Dalcin wrote:

> On Thu, May 14, 2009 at 10:12 PM, Carl Witty   
> wrote:
>> On Thu, May 14, 2009 at 4:48 PM, Lisandro Dalcin  
>>  wrote:
>>> However, interestingly enough, my C++  header (from GCC
>>> 4.3.2) seems to implements the naive way (although warned in the
>>> comment)... Am I missing something here??
>>
>> GSL (the GNU Scientific Library) also uses the naive algorithm.
>> Perhaps everybody else has decided that speed is more important than
>> accuracy, or perhaps they haven't thought of the issues at all?
>>
>
> I bet that the second one is the case ;-)
>
>>
>> How much difference in accuracy are we talking about?
>>
>
> Basically, we have two issues: overflow and cancelation.
>
> * Overflow will occur if the real or imaginary part of the denominator
> has exponent > 1/2 max_exponent; for example real|imag parts are above
> 1e20 for float or 1e155 for double

Yep.

> * Cancellation can occur on subtraction when computing the  
> imaginary component.

Or, of course, on addition when computing the real part.

The smart algorithm has exactly the same problem. I am at a loss to  
explain why the smart algorithm is better 60% of the time, and the  
naive only 25% of the time. (I tried various other distributions, the  
results are all the same.)

> See yourself the effects of all this in single precision:

[...]

> z1 = (1+2j)
> z2 = (3+4j)
> z1/z2 (naive) = (0.439999997616+0.0799999982119j)
> z1/z2 (smart) = (0.44+0.08j)
>
> z1 = 1e+20j
> z2 = (1e+20+0j)
> z1/z2 (naive) = nan*j
> z1/z2 (smart) = 1j
>
> z1 = (1+1.0000001j)
> z2 = (1+1j)
> z1/z2 (naive) = (1+5.96046447754e-08j)
> z1/z2 (smart) = (1.00000005+5.00000000292e-08j)
>
> In the second bit, overflow generated a NaN. In the third bit,
> cancellation produced an imaginary component 20% larger.

Of course the comparison is a bit unfair here, as your smart method  
is actually using doubles. Re-running this with all doubles and the  
naive algorithm gets all the same answers (up to the precision  
printed at least). Note that the smart algorithm gives an answer  
that's pretty far off too (though not quite as bad):

sage: complex(1+1.000000000000001j)/complex(1+1j)
(1.0000000000000004+5.5511151231257827e-16j)

This isn't to say that the smart algorithm shouldn't be used, but if  
its two or three times slower, I'm dubious that it's worth it for a  
slightly smaller average rounding error.

- Robert


From dalcinl at gmail.com  Fri May 15 06:16:19 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 15 May 2009 01:16:19 -0300
Subject: [Cython] Help with pointer to array of structs...
In-Reply-To: 
References: 
	
	
	
	
	
	
Message-ID: 

On Fri, May 15, 2009 at 1:03 AM, Mohamed Lrhazi  wrote:
>
> I tried first with commented version, then the uncommented:
>
> cdef extern from "sys/uio.h":
> ? ?# ctypedef struct iovec "struct iovec":
> ? ? ?# void *iov_base
> ? ? ?# size_t iov_len
> ? ?cdef struct iovec:
> ? ? ? ?void *iov_base
> ? ? ? ?size_t iov_len
>
>
> With the first, it fails whenever iovecs are declared, ?like this:
>
> uint32_t max_pending_trans
> char *traffic_class
> struct iovec *req_hdrs
> ? ? ? ? ? ? ? ? ? ?^

But if you use the first, commented version, then do not use "struct
iovec *req_hdrs", just "iovec *req_hdrs" !!


>
> With the second, it compiles, but then my program sigfaults. I should
> add that it never sigfaults if I only have one value in the passed in
> array of structs, only if there are more.
>

In your original code, when you fill

hdr_names[ii].iov_base = hdr_n
hdr_names[ii].iov_len = len(hdr_n)

What exactly is "hdr_n"? A Python string? Where "hdr_n" comes from?
The iovec entries in hdr_names are supposed to be readonly? Or are you
trying to receive data on them?



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Fri May 15 06:23:17 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 15 May 2009 06:23:17 +0200 (CEST)
Subject: [Cython] Help with pointer to array of structs...
In-Reply-To: 
References: 
	
	
	
	
	
	
	
Message-ID: <94fd8545ae70b59f269f5249278ec6b3.squirrel@webmail.uio.no>

Lisandro wrote:
> On Fri, May 15, 2009 at 1:03 AM, Mohamed Lrhazi  wrote:
>>
>> I tried first with commented version, then the uncommented:
>>
>> cdef extern from "sys/uio.h":
>> ? ?# ctypedef struct iovec "struct iovec":
>> ? ? ?# void *iov_base
>> ? ? ?# size_t iov_len
>> ? ?cdef struct iovec:
>> ? ? ? ?void *iov_base
>> ? ? ? ?size_t iov_len
>>
>>
>> With the first, it fails whenever iovecs are declared, ?like this:
>>
>> uint32_t max_pending_trans
>> char *traffic_class
>> struct iovec *req_hdrs
>> ? ? ? ? ? ? ? ? ? ?^
>
> But if you use the first, commented version, then do not use "struct
> iovec *req_hdrs", just "iovec *req_hdrs" !!

Actually, I'm fairly certain that in Cython it is

iovec* req_hrds

in both cases. Cython does away with the strange C requirement in the same
way as C++ does.

Dag Sverre


From lrhazi at gmail.com  Fri May 15 06:28:42 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Fri, 15 May 2009 00:28:42 -0400
Subject: [Cython] Help with pointer to array of structs...
In-Reply-To: 
References: 
	
	
	
	
	
	
	
Message-ID: 

On Fri, May 15, 2009 at 12:16 AM, Lisandro Dalcin  wrote:
> On Fri, May 15, 2009 at 1:03 AM, Mohamed Lrhazi  wrote:
>>
>> I tried first with commented version, then the uncommented:
>>
>> cdef extern from "sys/uio.h":
>> ? ?# ctypedef struct iovec "struct iovec":
>> ? ? ?# void *iov_base
>> ? ? ?# size_t iov_len
>> ? ?cdef struct iovec:
>> ? ? ? ?void *iov_base
>> ? ? ? ?size_t iov_len
>>
>>
>> With the first, it fails whenever iovecs are declared, ?like this:
>>
>> uint32_t max_pending_trans
>> char *traffic_class
>> struct iovec *req_hdrs
>> ? ? ? ? ? ? ? ? ? ?^
>
> But if you use the first, commented version, then do not use "struct
> iovec *req_hdrs", just "iovec *req_hdrs" !!
>

I just retired that way, it compiles, but then sigfaults. :(

>
>>
>> With the second, it compiles, but then my program sigfaults. I should
>> add that it never sigfaults if I only have one value in the passed in
>> array of structs, only if there are more.
>>
>
> In your original code, when you fill
>
> hdr_names[ii].iov_base = hdr_n
> hdr_names[ii].iov_len = len(hdr_n)
>
> What exactly is "hdr_n"? A Python string? Where "hdr_n" comes from?
> The iovec entries in hdr_names are supposed to be readonly? Or are you
> trying to receive data on them?
>

hdr_n is a string that came from Python... maybe full code with help
you better see my mistakes :)  http://pastebin.com/dbd60984

that function is called by code in Python land, passing lists or
header names to delete, or dicts of header name/value to add/replace.

Thanks a lot.
Mohamed.

From dagss at student.matnat.uio.no  Fri May 15 06:32:16 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 15 May 2009 06:32:16 +0200 (CEST)
Subject: [Cython] ctypedef for integrals
In-Reply-To: 
References: 
Message-ID: 

Lisandro wrote:
> After Robert changes for supporting complex, things are now far easuer
> IMHO

A sure hallmark of (Robert's) well-written code is that it makes things
easier for cases that wasn't considered originally :-)

> See the attached patch, testcase included. Disclaimer: this is a
> completely unpolished hack written in 10 minutes before going home.
> But now I'm even more confident we will find our way :-)

Wonderful, it looks very good. I'd love to get this in the release if
possible but I'll have to wait until you trust it (I'll update this if I
get time for building Sage with it).

There shouldn't be any behavioural changes *if* the external typedef was
exact, right?

Dag Sverre


From lrhazi at gmail.com  Fri May 15 06:33:11 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Fri, 15 May 2009 00:33:11 -0400
Subject: [Cython] Help with pointer to array of structs...
In-Reply-To: 
References: 
	
	
	
	
	
	
	
	
Message-ID: 

And yes the hdr_n and hdr_v strings are read only, the c-lib is not
expected to modify them.

Thanks.

From robertwb at math.washington.edu  Fri May 15 06:45:18 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 14 May 2009 21:45:18 -0700
Subject: [Cython] Help with pointer to array of structs...
In-Reply-To: 
References: 
	
	
	
	
	
	
	
	
Message-ID: <1915868E-C660-4DF0-9198-881CBAC79056@math.washington.edu>

On May 14, 2009, at 9:28 PM, Mohamed Lrhazi wrote:

> On Fri, May 15, 2009 at 12:16 AM, Lisandro Dalcin  
>  wrote:
>> On Fri, May 15, 2009 at 1:03 AM, Mohamed Lrhazi   
>> wrote:
>>>
>>> I tried first with commented version, then the uncommented:
>>>
>>> cdef extern from "sys/uio.h":
>>>    # ctypedef struct iovec "struct iovec":
>>>      # void *iov_base
>>>      # size_t iov_len
>>>    cdef struct iovec:
>>>        void *iov_base
>>>        size_t iov_len
>>>
>>>
>>> With the first, it fails whenever iovecs are declared,  like this:
>>>
>>> uint32_t max_pending_trans
>>> char *traffic_class
>>> struct iovec *req_hdrs
>>>                    ^
>>
>> But if you use the first, commented version, then do not use "struct
>> iovec *req_hdrs", just "iovec *req_hdrs" !!
>>
>
> I just retired that way, it compiles, but then sigfaults. :(
>
>>
>>>
>>> With the second, it compiles, but then my program sigfaults. I  
>>> should
>>> add that it never sigfaults if I only have one value in the  
>>> passed in
>>> array of structs, only if there are more.
>>>
>>
>> In your original code, when you fill
>>
>> hdr_names[ii].iov_base = hdr_n
>> hdr_names[ii].iov_len = len(hdr_n)
>>
>> What exactly is "hdr_n"? A Python string? Where "hdr_n" comes from?
>> The iovec entries in hdr_names are supposed to be readonly? Or are  
>> you
>> trying to receive data on them?
>>
>
> hdr_n is a string that came from Python... maybe full code with help
> you better see my mistakes :)  http://pastebin.com/dbd60984
>
> that function is called by code in Python land, passing lists or
> header names to delete, or dicts of header name/value to add/replace.

See http://docs.cython.org/docs/language_basics.html#caveats-when- 
using-a-python-string-in-a-c-context

(Perhaps "Caveats" is too weak of a warning to use here...)

- Robert


From lrhazi at gmail.com  Fri May 15 07:22:28 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Fri, 15 May 2009 01:22:28 -0400
Subject: [Cython] Help with pointer to array of structs...
In-Reply-To: <1915868E-C660-4DF0-9198-881CBAC79056@math.washington.edu>
References: 
	
	
	
	
	
	
	
	
	<1915868E-C660-4DF0-9198-881CBAC79056@math.washington.edu>
Message-ID: 

On Fri, May 15, 2009 at 12:45 AM, Robert Bradshaw
 wrote:
>> hdr_n is a string that came from Python... maybe full code with help
>> you better see my mistakes :) ?http://pastebin.com/dbd60984
>>
>> that function is called by code in Python land, passing lists or
>> header names to delete, or dicts of header name/value to add/replace.
>
> See http://docs.cython.org/docs/language_basics.html#caveats-when-
> using-a-python-string-in-a-c-context
>
> (Perhaps "Caveats" is too weak of a warning to use here...)
>

Thanks Robert... did not pay attention to that little paragraph...
Now, I tried this a solution: http://pastebin.com/d4ab1c687

putting my Python strings in a list of tuples, and refering to them in
the  assignment.

My program still get sigfault... isnt not a solution to the said Caveat?

Mohamed.

From robertwb at math.washington.edu  Fri May 15 07:32:08 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 14 May 2009 22:32:08 -0700
Subject: [Cython] Help with pointer to array of structs...
In-Reply-To: 
References: 
	
	
	
	
	
	
	
	
	<1915868E-C660-4DF0-9198-881CBAC79056@math.washington.edu>
	
Message-ID: 

On May 14, 2009, at 10:22 PM, Mohamed Lrhazi wrote:

> On Fri, May 15, 2009 at 12:45 AM, Robert Bradshaw
>  wrote:
>>> hdr_n is a string that came from Python... maybe full code with help
>>> you better see my mistakes :)  http://pastebin.com/dbd60984
>>>
>>> that function is called by code in Python land, passing lists or
>>> header names to delete, or dicts of header name/value to add/ 
>>> replace.
>>
>> See http://docs.cython.org/docs/language_basics.html#caveats-when-
>> using-a-python-string-in-a-c-context
>>
>> (Perhaps "Caveats" is too weak of a warning to use here...)
>>
>
> Thanks Robert... did not pay attention to that little paragraph...
> Now, I tried this a solution: http://pastebin.com/d4ab1c687
>
> putting my Python strings in a list of tuples, and refering to them in
> the  assignment.
>
> My program still get sigfault... isnt not a solution to the said  
> Caveat?

How long do you keep the tuple around?

- Robert



From hoytak at cs.ubc.ca  Fri May 15 07:34:33 2009
From: hoytak at cs.ubc.ca (Hoyt Koepke)
Date: Thu, 14 May 2009 22:34:33 -0700
Subject: [Cython] Help with pointer to array of structs...
In-Reply-To: 
References:  
	 
	 
	 
	 
	 
	 
	 
	<1915868E-C660-4DF0-9198-881CBAC79056@math.washington.edu> 
	
Message-ID: <4db580fd0905142234w907e2d8ta6191f80d0bdae6f@mail.gmail.com>

A couple comments (which may get corrected by the experts):

Please indicate why this post is abusive, and provide any other useful
information.

   1.
      def modify_headers(self,handle_a,ctx_a,add={},delete=[],replace={}):

If you can add types to add, delete, and replace (dict and list), that
will speed some things up.


  21.
          for i,hdr_n in enumerate(delete):
  22.
              ii=i

You should type your indices above as type int or size_t.  Also, it
would be faster to do

for i from 0 <= i < len(delete):
    hdr_n = delete[i]

as this avoids the overhead of creating and unpacking tuples and
converting one to an index.  If this solves your problem, then there's
probably in error somewhere.


  29.
          for i,(hdr_n,hdr_v) in enumerate(add.items()):

In this case, it would be faster, if you aren't changing add and it's
ad dict, to do
i = 0
for hdr_n, hdr_v in add.iteritems():
   # other code
   i += 1

-- 

++++++++++++++++++++++++++++++++++++++++++++++++
+ Hoyt Koepke
+ University of Washington Department of Statistics
+ http://www.stat.washington.edu/~hoytak/
+ hoytak at gmail.com
++++++++++++++++++++++++++++++++++++++++++

From lrhazi at gmail.com  Fri May 15 07:40:55 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Fri, 15 May 2009 01:40:55 -0400
Subject: [Cython] Help with pointer to array of structs...
In-Reply-To: 
References: 
	
	
	
	
	
	
	<1915868E-C660-4DF0-9198-881CBAC79056@math.washington.edu>
	
	
Message-ID: 

On Fri, May 15, 2009 at 1:32 AM, Robert Bradshaw
 wrote:
> How long do you keep the tuple around?
>
till the end of the function where everything is happening, mainly
till after I attempt the C-lib function call, where I sigfault.

Hoyt, thanks for the suggestions. I will try them. I am so desperate.
Thanks.

From hoytak at cs.ubc.ca  Fri May 15 07:50:15 2009
From: hoytak at cs.ubc.ca (Hoyt Koepke)
Date: Thu, 14 May 2009 22:50:15 -0700
Subject: [Cython] Help with pointer to array of structs...
In-Reply-To: 
References:  
	 
	 
	 
	 
	 
	<1915868E-C660-4DF0-9198-881CBAC79056@math.washington.edu> 
	 
	 
	
Message-ID: <4db580fd0905142250m1eeb4ba7me5b2a8a1eab81f4f@mail.gmail.com>

Are the &hrd_names and &hdr_values in this line correct?  If so, do
you know what xz_trans... does with them?  Since it's a double ptr as
written, your original ptr might be modified.

result=xz_transaction_hdrs_modify(handle,ctx,&hdr_names,&hdr_values,del_count,add_count,replace_count)

--Hoyt

On Thu, May 14, 2009 at 10:40 PM, Mohamed Lrhazi  wrote:
> On Fri, May 15, 2009 at 1:32 AM, Robert Bradshaw
>  wrote:
>> How long do you keep the tuple around?
>>
> till the end of the function where everything is happening, mainly
> till after I attempt the C-lib function call, where I sigfault.
>
> Hoyt, thanks for the suggestions. I will try them. I am so desperate.
> Thanks.
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 

++++++++++++++++++++++++++++++++++++++++++++++++
+ Hoyt Koepke
+ University of Washington Department of Statistics
+ http://www.stat.washington.edu/~hoytak/
+ hoytak at gmail.com
++++++++++++++++++++++++++++++++++++++++++

From stefan_ml at behnel.de  Thu May 14 08:20:09 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 14 May 2009 08:20:09 +0200
Subject: [Cython] Notes on code streams refactoring
In-Reply-To: <7AC765CE-531D-4146-B100-E9203FF38CED@math.washington.edu>
References: <4A092A52.2010309@student.matnat.uio.no>
	<7AC765CE-531D-4146-B100-E9203FF38CED@math.washington.edu>
Message-ID: <4A0BB819.3070109@behnel.de>


Robert Bradshaw wrote:
> On May 12, 2009, at 12:50 AM, Dag Sverre Seljebotn wrote:
>> http://hg.cython.org/cython-devel/rev/b86a7b374a95
>> http://hg.cython.org/cython-devel/rev/fd6cc8a5998d
>> http://hg.cython.org/cython-devel/rev/df8dd466bc4a
> 
> Looks fine to me after a cursory glance.

Same from here.

Stefan



From lrhazi at gmail.com  Fri May 15 08:03:23 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Fri, 15 May 2009 02:03:23 -0400
Subject: [Cython] Help with pointer to array of structs...
In-Reply-To: <4db580fd0905142250m1eeb4ba7me5b2a8a1eab81f4f@mail.gmail.com>
References: 
	
	
	
	
	<1915868E-C660-4DF0-9198-881CBAC79056@math.washington.edu>
	
	
	
	<4db580fd0905142250m1eeb4ba7me5b2a8a1eab81f4f@mail.gmail.com>
Message-ID: 

On Fri, May 15, 2009 at 1:50 AM, Hoyt Koepke  wrote:
> Are the &hrd_names and &hdr_values in this line correct? ?If so, do
> you know what xz_trans... does with them? ?Since it's a double ptr as
> written, your original ptr might be modified.
>
> result=xz_transaction_hdrs_modify(handle,ctx,&hdr_names,&hdr_values,del_count,add_count,replace_count)
>

That is the signature of the function in the c lib's header file. What
could it mean that it is expecting a double pointer? a pointer of an
array of structs?

What the function does is allow me to tell it which headers, as in
HTTP speak, I want added/deleted/replaced, from the current HTTP
transaction it is handling for me.

So I would guess it simply needs to read all those strings am passing in.

From hoytak at cs.ubc.ca  Fri May 15 08:17:06 2009
From: hoytak at cs.ubc.ca (Hoyt Koepke)
Date: Thu, 14 May 2009 23:17:06 -0700
Subject: [Cython] Help with pointer to array of structs...
In-Reply-To: 
References:  
	 
	 
	 
	<1915868E-C660-4DF0-9198-881CBAC79056@math.washington.edu> 
	 
	 
	 
	<4db580fd0905142250m1eeb4ba7me5b2a8a1eab81f4f@mail.gmail.com> 
	
Message-ID: <4db580fd0905142317y6bffe5dcm6866f99e65e0eca5@mail.gmail.com>

So the signature of your function has iovec **?  If it does, what that
means is that it can modify the value of hdr_names nad hdr_values in
your code, as you are giving pointers to them.  If so, try checking
whether they change when you call the function.  On the other hand, if
the function signature is iovec*, then you are doing it wrong and need
to remove the &.  That would definitely call a segfault.

BTW, do you get any warnings during compilation?

--Hoyt



On Thu, May 14, 2009 at 11:03 PM, Mohamed Lrhazi  wrote:
> On Fri, May 15, 2009 at 1:50 AM, Hoyt Koepke  wrote:
>> Are the &hrd_names and &hdr_values in this line correct? ?If so, do
>> you know what xz_trans... does with them? ?Since it's a double ptr as
>> written, your original ptr might be modified.
>>
>> result=xz_transaction_hdrs_modify(handle,ctx,&hdr_names,&hdr_values,del_count,add_count,replace_count)
>>
>
> That is the signature of the function in the c lib's header file. What
> could it mean that it is expecting a double pointer? a pointer of an
> array of structs?
>
> What the function does is allow me to tell it which headers, as in
> HTTP speak, I want added/deleted/replaced, from the current HTTP
> transaction it is handling for me.
>
> So I would guess it simply needs to read all those strings am passing in.
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 

++++++++++++++++++++++++++++++++++++++++++++++++
+ Hoyt Koepke
+ University of Washington Department of Statistics
+ http://www.stat.washington.edu/~hoytak/
+ hoytak at gmail.com
++++++++++++++++++++++++++++++++++++++++++

From lrhazi at gmail.com  Fri May 15 08:48:55 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Fri, 15 May 2009 02:48:55 -0400
Subject: [Cython] Help with pointer to array of structs...
In-Reply-To: <4db580fd0905142317y6bffe5dcm6866f99e65e0eca5@mail.gmail.com>
References: 
	
	
	<1915868E-C660-4DF0-9198-881CBAC79056@math.washington.edu>
	
	
	
	<4db580fd0905142250m1eeb4ba7me5b2a8a1eab81f4f@mail.gmail.com>
	
	<4db580fd0905142317y6bffe5dcm6866f99e65e0eca5@mail.gmail.com>
Message-ID: 

On Fri, May 15, 2009 at 2:17 AM, Hoyt Koepke  wrote:
> So the signature of your function has iovec **? ?If it does, what that
> means is that it can modify the value of hdr_names nad hdr_values in
> your code, as you are giving pointers to them. ?If so, try checking
> whether they change when you call the function. ?On the other hand, if
> the function signature is iovec*, then you are doing it wrong and need
> to remove the &. ?That would definitely call a segfault.
>

The sig is: struct iovec *hdr_names[], struct iovec *hdr_values[]

When I call the function with a single header name/value, that is,
arrays of a single iovec each, then it works. printing the addresses
of pointers and the strings they point to does not indicate that the
function changes anything.

When I call with two iovecs in the arrays, that is two header
name/values.... then the function call sigfaults and I can see if the
function did attaempt to change anything i passed.

I tried forcing things by changing the function signature to say
"struct iovec *hdr_values", and then removing the & when I call, it
sigfaults just the same.

> BTW, do you get any warnings during compilation?
>
It does not seem like I have any warning regarding the lines of code
surrounding this function call...


Thanks a lot.
Mohamed.

From robertwb at math.washington.edu  Fri May 15 08:49:26 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 14 May 2009 23:49:26 -0700
Subject: [Cython] ctypedef for integrals
In-Reply-To: 
References: 
	
Message-ID: 

On May 14, 2009, at 9:32 PM, Dag Sverre Seljebotn wrote:

> Lisandro wrote:
>> After Robert changes for supporting complex, things are now far  
>> easuer
>> IMHO
>
> A sure hallmark of (Robert's) well-written code is that it makes  
> things
> easier for cases that wasn't considered originally :-)

Thanks. And here I felt like I was sticking is_complex tests in all  
sorts of odd places... (I did try to minimize that.)

>> See the attached patch, testcase included. Disclaimer: this is a
>> completely unpolished hack written in 10 minutes before going home.
>> But now I'm even more confident we will find our way :-)
>
> Wonderful, it looks very good. I'd love to get this in the release if
> possible but I'll have to wait until you trust it (I'll update this  
> if I
> get time for building Sage with it).

Looks good to me too. The line

type_name = self.qualified_name.split('.')[-1]

in __getattr__ seems extraneous, as does the empty comment in the  
PyInt_from... code. With all this testing for basetype.is_int, it is  
almost testing to make a subclass just for int typedefs. Also, we are  
assuming that the only sizes that can occur are char, short, int,  
long, and long long. (It sounds inconceivable, but the C spec would  
allow 16-bit ints and 64-bit longs, or 64-bit ints as in ILP64.)

> There shouldn't be any behavioural changes *if* the external  
> typedef was
> exact, right?

I shouldn't think so.

- Robert


From kxroberto at googlemail.com  Fri May 15 13:01:48 2009
From: kxroberto at googlemail.com (Robert)
Date: Fri, 15 May 2009 13:01:48 +0200
Subject: [Cython] Faster Cython mappings?
In-Reply-To: <178a1957646172910061be96a5e79750.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
References: <477042.42563.qm@web52110.mail.re2.yahoo.com>	
	<178a1957646172910061be96a5e79750.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
Message-ID: 

Stefan Behnel wrote:
>>> Good day.
>>> I am trying to obtain MAXIMUM speed for this mappings:
>>>
>>> MyString = '\n'.join([ ''.join([j.encode('utf8') for j in i]) for i
>>> in NdArrayList ])
>  
...
> Also, given that the retrieval of each of the substrings has a little
> overhead, I'd recommend copying the buffer content directly into a memory
> buffer, over-reallocating at need to avoid multiple copying. There are
> Python C-API methods that can then encode the resulting PyUNICODE buffers
> (assuming that a numpy " string.
> 

the array.pxd (Python's array.array) of
http://trac.cython.org/cython_trac/ticket/314
http://trac.cython.org/cython_trac/attachment/ticket/314/array.zip

.. which may appear in one of the next Cython releases in some 
fashion, has ready made C-fast inline extend functions, efficient 
auto-growth,memory management etc already implemented.
cdef array.array a=array('u'/'c') or so.
What array_example.pyx shows in these lines in fast_cat()

         array.array_extend(a, b)
         ...
         array.array_extend_buffer(a, s, PyString_Size(s) )
         ...

may be quite near to what you want. taking the ndarr.data stuff 
instead.


Robert


From hoytak at cs.ubc.ca  Fri May 15 17:49:40 2009
From: hoytak at cs.ubc.ca (Hoyt Koepke)
Date: Fri, 15 May 2009 08:49:40 -0700
Subject: [Cython] Help with pointer to array of structs...
In-Reply-To: 
References:  
	 
	<1915868E-C660-4DF0-9198-881CBAC79056@math.washington.edu> 
	 
	 
	 
	<4db580fd0905142250m1eeb4ba7me5b2a8a1eab81f4f@mail.gmail.com> 
	 
	<4db580fd0905142317y6bffe5dcm6866f99e65e0eca5@mail.gmail.com> 
	
Message-ID: <4db580fd0905150849h39324364nc124fc56432bbabb@mail.gmail.com>

> The sig is: struct iovec *hdr_names[], struct iovec *hdr_values[]

Okay, this is probably the problem then.  It is expecting an array of
pointers to individual structs, not a pointer to an array of structs.
When it iterates over the array, it iterates over the itvec**
instance, expecting an iovec* at each iteration.  What you need to do
with your code is create an intermediate array that is what it wants.
If there is only one element, then the two cases are the same, so it
would not segfault.  Something like:

cdef iovec** ptrarray = malloc(n*sizeof(iovec*)

for 0 <= i < n:
    ptr_array[i] = &hrd_array[i]

...

free(ptr_array)

--Hoyt

++++++++++++++++++++++++++++++++++++++++++++++++
+ Hoyt Koepke
+ University of Washington Department of Statistics
+ http://www.stat.washington.edu/~hoytak/
+ hoytak at gmail.com
++++++++++++++++++++++++++++++++++++++++++

From dalcinl at gmail.com  Fri May 15 18:14:11 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 15 May 2009 13:14:11 -0300
Subject: [Cython] ctypedef for integrals
In-Reply-To: 
References: 
	
	
Message-ID: 

On Fri, May 15, 2009 at 3:49 AM, Robert Bradshaw
 wrote:
> On May 14, 2009, at 9:32 PM, Dag Sverre Seljebotn wrote:
>>
>> Wonderful, it looks very good. I'd love to get this in the release if
>> possible but I'll have to wait until you trust it (I'll update this
>> if I
>> get time for building Sage with it).
>
> Looks good to me too. The line
>
> type_name = self.qualified_name.split('.')[-1]
>
> in __getattr__ seems extraneous, as does the empty comment in the
> PyInt_from... code. With all this testing for basetype.is_int, it is
> almost testing to make a subclass just for int typedefs.

As I said, my patch is a very preliminar one... it is far from ready...

> Also, we are
> assuming that the only sizes that can occur are char, short, int,
> long, and long long. (It sounds inconceivable, but the C spec would
> allow 16-bit ints and 64-bit longs, or 64-bit ints as in ILP64.)
>

OK... I could fix this returning a PyLong instance.

>> There shouldn't be any behavioural changes *if* the external
>> typedef was
>> exact, right?
>
> I shouldn't think so.
>

And that's the BIG problem... If we are going to stop making
assumptions about sizes, we cannot use the rank stuff for binary
operations. Moreover, if a user is sure a ctypedef match a particular
type, we should use that information. And we have the bacward
compatibility issues... Then I'm thinking on this syntax:

cdef extern:
    ctypedef signed ? MySignedInt
    ctypedef unsigned ? MyUnsignedInt
    ctypedef float ? MyFloat
    ctypedef complex ? MyComplex

or perhaps

cdef extern:
    ctypedef signed MySignedInt?
    ctypedef unsigned MyUnsignedInt?
    ctypedef float MyFloat?
    ctypedef complex MyComplex?


In short, the user has to inform Cython to not make assumptions about
the size of a particular ctypedef.

What do you think? Does this make sense? Any better syntax?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From lrhazi at gmail.com  Fri May 15 18:23:39 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Fri, 15 May 2009 12:23:39 -0400
Subject: [Cython] Help with pointer to array of structs...
In-Reply-To: <4db580fd0905150849h39324364nc124fc56432bbabb@mail.gmail.com>
References: 
	<1915868E-C660-4DF0-9198-881CBAC79056@math.washington.edu>
	
	
	
	<4db580fd0905142250m1eeb4ba7me5b2a8a1eab81f4f@mail.gmail.com>
	
	<4db580fd0905142317y6bffe5dcm6866f99e65e0eca5@mail.gmail.com>
	
	<4db580fd0905150849h39324364nc124fc56432bbabb@mail.gmail.com>
Message-ID: 

On Fri, May 15, 2009 at 11:49 AM, Hoyt Koepke  wrote:
>> The sig is: struct iovec *hdr_names[], struct iovec *hdr_values[]
>
> Okay, this is probably the problem then. ?It is expecting an array of
> pointers to individual structs, not a pointer to an array of structs.
> When it iterates over the array, it iterates over the itvec**
> instance, expecting an iovec* at each iteration. ?What you need to do
> with your code is create an intermediate array that is what it wants.
> If there is only one element, then the two cases are the same, so it
> would not segfault. ?Something like:
>
> cdef iovec** ptrarray = malloc(n*sizeof(iovec*)
>
> for 0 <= i < n:
> ? ?ptr_array[i] = &hrd_array[i]
>
> ...
>
> free(ptr_array)
>

I don't have a million to give you Hoyt , so just accept my thank you's.

I tried this:

cdef iovec** ptr_array_n = malloc(total_count * sizeof(iovec*))
cdef iovec** ptr_array_v = malloc(total_count * sizeof(iovec*))
...
ptr_array_n[0] = &hdr_names[0]
ptr_array_v[0] = &hdr_values[0]
ptr_array_n[1] = &hdr_names[1]
ptr_array_v[1] = &hdr_values[1]

call_function(...,ptr_array_n,ptr_array_v,...)

I'll return to my earlier code and try that.

Mohamed.

From dagss at student.matnat.uio.no  Fri May 15 19:03:11 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 15 May 2009 19:03:11 +0200 (CEST)
Subject: [Cython] ctypedef for integrals
In-Reply-To: 
References: 
	
	
	
Message-ID: <94b65240f7c9ba5e0c29c2b1ca6c79e2.squirrel@webmail.uio.no>

Lisandro wrote:
> And that's the BIG problem... If we are going to stop making
> assumptions about sizes, we cannot use the rank stuff for binary
> operations. Moreover, if a user is sure a ctypedef match a particular
> type, we should use that information. And we have the bacward
> compatibility issues... Then I'm thinking on this syntax:
>
> cdef extern:
>     ctypedef signed ? MySignedInt
>     ctypedef unsigned ? MyUnsignedInt
>     ctypedef float ? MyFloat
>     ctypedef complex ? MyComplex
>
> or perhaps
>
> cdef extern:
>     ctypedef signed MySignedInt?
>     ctypedef unsigned MyUnsignedInt?
>     ctypedef float MyFloat?
>     ctypedef complex MyComplex?

I tried proposing something similar (unknown size) some time ago and it
was voted down; perhaps the climate has changed now.

I support this anyway for more transparency about what Cython does (though
none of those syntaxes), but I don't think it is really needed:

This only affects situations where the type size is actually used for
something. I can only think of constructing temporaries once this patch
goes in.

And how many situations can you think of which constructs temporaries of
the type of the result of a binary operation?

The only one I can think of is Python-style division (which was only
recently added, and where it is probably possible to rely on the compiler
optimizing out a common expression for external typedefs instead of using
a temporary -- must benchmark first though).

So this isn't a big problem now, IMO.

It WILL become a problem with type inference, but that's a bigger
discussion we can have later.

Dag Sverre


From lrhazi at gmail.com  Fri May 15 19:03:14 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Fri, 15 May 2009 13:03:14 -0400
Subject: [Cython] Help with pointer to array of structs...
In-Reply-To: 
References: 
	
	
	
	<4db580fd0905142250m1eeb4ba7me5b2a8a1eab81f4f@mail.gmail.com>
	
	<4db580fd0905142317y6bffe5dcm6866f99e65e0eca5@mail.gmail.com>
	
	<4db580fd0905150849h39324364nc124fc56432bbabb@mail.gmail.com>
	
Message-ID: 

On Fri, May 15, 2009 at 12:23 PM, Mohamed Lrhazi  wrote:
> On Fri, May 15, 2009 at 11:49 AM, Hoyt Koepke  wrote:
>>> The sig is: struct iovec *hdr_names[], struct iovec *hdr_values[]
>>
>> Okay, this is probably the problem then. ?It is expecting an array of
>> pointers to individual structs, not a pointer to an array of structs.

Just to close up this thread, am not sure I was clear earlier. It
turns out to work the way Hoyt describes it.
Thank you all for your time and patience with me.

Mohamed.

From hoytak at cs.ubc.ca  Fri May 15 19:05:41 2009
From: hoytak at cs.ubc.ca (Hoyt Koepke)
Date: Fri, 15 May 2009 10:05:41 -0700
Subject: [Cython] Help with pointer to array of structs...
In-Reply-To: 
References:  
	 
	 
	<4db580fd0905142250m1eeb4ba7me5b2a8a1eab81f4f@mail.gmail.com> 
	 
	<4db580fd0905142317y6bffe5dcm6866f99e65e0eca5@mail.gmail.com> 
	 
	<4db580fd0905150849h39324364nc124fc56432bbabb@mail.gmail.com> 
	 
	
Message-ID: <4db580fd0905151005u4d9db09dv3b2512c8d4cd6318@mail.gmail.com>

> Just to close up this thread, am not sure I was clear earlier. It
> turns out to work the way Hoyt describes it.
> Thank you all for your time and patience with me.

Great!  Glad to help.  Good luck.

--Hoyt

++++++++++++++++++++++++++++++++++++++++++++++++
+ Hoyt Koepke
+ University of Washington Department of Statistics
+ http://www.stat.washington.edu/~hoytak/
+ hoytak at gmail.com
++++++++++++++++++++++++++++++++++++++++++

From dalcinl at gmail.com  Fri May 15 20:41:36 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 15 May 2009 15:41:36 -0300
Subject: [Cython] ctypedef for integrals
In-Reply-To: <94b65240f7c9ba5e0c29c2b1ca6c79e2.squirrel@webmail.uio.no>
References: 
	
	
	
	<94b65240f7c9ba5e0c29c2b1ca6c79e2.squirrel@webmail.uio.no>
Message-ID: 

On Fri, May 15, 2009 at 2:03 PM, Dag Sverre Seljebotn
 wrote:
> Lisandro wrote:
>
> I tried proposing something similar (unknown size) some time ago and it
> was voted down; perhaps the climate has changed now.
>

I voted it down perhaps because I misunderstood your proposal. I
understood you were proposing this for the whole type system.

If this is done for external ctypedefs, I would no oppose it.

>
> I support this anyway for more transparency about what Cython does (though
> none of those syntaxes), but I don't think it is really needed:
>

Well, I have a bit of fear of some user coding "cdef extern: ctypedef
signed long Something" and then Cython stop treating "Something" as
EXACTLY a "signed long". A user could complain: Hey, I know
"Something" is EXACTLY a "signed long", Why aren't you using such
declaration?

> This only affects situations where the type size is actually used for
> something. I can only think of constructing temporaries once this patch
> goes in.
>
> And how many situations can you think of which constructs temporaries of
> the type of the result of a binary operation?
>

What would happen below?

cdef MyInt a, b
....
cdef object o = a BINOP b



> The only one I can think of is Python-style division (which was only
> recently added, and where it is probably possible to rely on the compiler
> optimizing out a common expression for external typedefs instead of using
> a temporary -- must benchmark first though).
>
> So this isn't a big problem now, IMO.
>
> It WILL become a problem with type inference, but that's a bigger
> discussion we can have later.
>
> Dag Sverre
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan_ml at behnel.de  Fri May 15 20:58:32 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 15 May 2009 20:58:32 +0200
Subject: [Cython] Merge of cython-devel to cython-unstable needed
Message-ID: <4A0DBB58.6070301@behnel.de>

Hi,

the large number of major changes in cython-devel induces a number of merge
conflicts in cython-unstable. I'm very happy to see all those great
improvements all over the place, but could those who implemented them
please try to re-integrate them with the unstable branch?

Thanks,

Stefan

From robertwb at math.washington.edu  Fri May 15 21:03:28 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 15 May 2009 12:03:28 -0700
Subject: [Cython] ctypedef for integrals
In-Reply-To: 
References: 
	
	
	
Message-ID: 

On May 15, 2009, at 9:14 AM, Lisandro Dalcin wrote:

> On Fri, May 15, 2009 at 3:49 AM, Robert Bradshaw
>  wrote:
>> On May 14, 2009, at 9:32 PM, Dag Sverre Seljebotn wrote:
>>>
>>> Wonderful, it looks very good. I'd love to get this in the  
>>> release if
>>> possible but I'll have to wait until you trust it (I'll update this
>>> if I
>>> get time for building Sage with it).
>>
>> Looks good to me too. The line
>>
>> type_name = self.qualified_name.split('.')[-1]
>>
>> in __getattr__ seems extraneous, as does the empty comment in the
>> PyInt_from... code. With all this testing for basetype.is_int, it is
>> almost testing to make a subclass just for int typedefs.
>
> As I said, my patch is a very preliminar one... it is far from  
> ready...

Sure, just trying to make it more ready :)

>> Also, we are
>> assuming that the only sizes that can occur are char, short, int,
>> long, and long long. (It sounds inconceivable, but the C spec would
>> allow 16-bit ints and 64-bit longs, or 64-bit ints as in ILP64.)
>>
>
> OK... I could fix this returning a PyLong instance.

My point was that there are systems with none of char, short, int,  
long being 32 bits, but your extern type could have 32 bits. The if- 
then statement dispatching on type sizes wouldn't work.

>>> There shouldn't be any behavioural changes *if* the external
>>> typedef was
>>> exact, right?
>>
>> I shouldn't think so.
>
> And that's the BIG problem... If we are going to stop making
> assumptions about sizes, we cannot use the rank stuff for binary
> operations. Moreover, if a user is sure a ctypedef match a particular
> type, we should use that information. And we have the bacward
> compatibility issues... Then I'm thinking on this syntax:
>
> cdef extern:
>     ctypedef signed ? MySignedInt
>     ctypedef unsigned ? MyUnsignedInt
>     ctypedef float ? MyFloat
>     ctypedef complex ? MyComplex
>
> or perhaps
>
> cdef extern:
>     ctypedef signed MySignedInt?
>     ctypedef unsigned MyUnsignedInt?
>     ctypedef float MyFloat?
>     ctypedef complex MyComplex?
>
>
> In short, the user has to inform Cython to not make assumptions about
> the size of a particular ctypedef.
>
> What do you think? Does this make sense? Any better syntax?

I'm -1 on both the syntax and the idea, comments in the next email.

- Robert


From dagss at student.matnat.uio.no  Fri May 15 21:04:22 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 15 May 2009 21:04:22 +0200 (CEST)
Subject: [Cython] Merge of cython-devel to cython-unstable needed
In-Reply-To: <4A0DBB58.6070301@behnel.de>
References: <4A0DBB58.6070301@behnel.de>
Message-ID: <127c1a4bfb524662928d1387c85f97b3.squirrel@webmail.uio.no>

> Hi,
>
> the large number of major changes in cython-devel induces a number of
> merge
> conflicts in cython-unstable. I'm very happy to see all those great
> improvements all over the place, but could those who implemented them
> please try to re-integrate them with the unstable branch?

At least for what I should be doing: Because of time constraints I need to
keep focus on cython-devel myself; I'll be happy to merge next week after
the release is done.

I don't think what's happened in -devel is a blocker for anything that
should happen in unstable the next week?

Dag Sverre


From dagss at student.matnat.uio.no  Fri May 15 21:11:56 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 15 May 2009 21:11:56 +0200 (CEST)
Subject: [Cython] ctypedef for integrals
In-Reply-To: 
References: 
	
	
	
	<94b65240f7c9ba5e0c29c2b1ca6c79e2.squirrel@webmail.uio.no>
	
Message-ID: 

Lisandro wrote:
> On Fri, May 15, 2009 at 2:03 PM, Dag Sverre Seljebotn
>  wrote:
>> Lisandro wrote:
>>
>> I tried proposing something similar (unknown size) some time ago and it
>> was voted down; perhaps the climate has changed now.
>>
>
> I voted it down perhaps because I misunderstood your proposal. I
> understood you were proposing this for the whole type system.

I was referring to myself pitching the same idea you just did (syntax for
types one does not know the size of) in here during my GSoC last summer.
It seems the climate hasn't changed :-)

> What would happen below?
>
> cdef MyInt a, b
> ....
> cdef object o = a BINOP b

Well, since both a and b is MyInt, this is simple, just use sizeof(MyInt)
to discriminate.

If you refer to

cdef IntA a
cdef IntB b
cdef object o = a + b

then...you are right, some creative solution is needed (it is not
impossible, any of my tricks proposed earlier this week would do the job).

Waiting to see what Robert will write before writing more about this.

Dag


From dalcinl at gmail.com  Fri May 15 21:26:08 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 15 May 2009 16:26:08 -0300
Subject: [Cython] ctypedef for integrals
In-Reply-To: 
References: 
	
	
	
	
Message-ID: 

On Fri, May 15, 2009 at 4:03 PM, Robert Bradshaw
 wrote:
>
>>> Also, we are
>>> assuming that the only sizes that can occur are char, short, int,
>>> long, and long long. (It sounds inconceivable, but the C spec would
>>> allow 16-bit ints and 64-bit longs, or 64-bit ints as in ILP64.)
>>>
>>
>> OK... I could fix this returning a PyLong instance.
>

> My point was that there are systems with none of char, short, int,
> long being 32 bits, but your extern type could have 32 bits. The if-
> then statement dispatching on type sizes wouldn't work.
>

I understood precisely that, and my point was that is more or less
easy to support, for example

...
else if (sizeof(type) == sizeof(long long))
...
else {
       unsigned char *bytes = (unsigned char *) &VALUE;
       int one = 1; int little = (int)*(unsigned char*)&one;
       return _PyLong_FromByteArray(bytes, sizeof(type), little, SIGNED);
}

>> What do you think? Does this make sense? Any better syntax?
>
> I'm -1 on both the syntax and the idea, comments in the next email.
>

OK, waiting for it...


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Fri May 15 21:28:45 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 15 May 2009 16:28:45 -0300
Subject: [Cython] Merge of cython-devel to cython-unstable needed
In-Reply-To: <127c1a4bfb524662928d1387c85f97b3.squirrel@webmail.uio.no>
References: <4A0DBB58.6070301@behnel.de>
	<127c1a4bfb524662928d1387c85f97b3.squirrel@webmail.uio.no>
Message-ID: 

Well, in PETSc (not petsc4py) we have a rather strict rule. Any change
that goes to release-maint is immediately merged to -dev. But we never
introduce new features in release-maint.

On Fri, May 15, 2009 at 4:04 PM, Dag Sverre Seljebotn
 wrote:
>> Hi,
>>
>> the large number of major changes in cython-devel induces a number of
>> merge
>> conflicts in cython-unstable. I'm very happy to see all those great
>> improvements all over the place, but could those who implemented them
>> please try to re-integrate them with the unstable branch?
>
> At least for what I should be doing: Because of time constraints I need to
> keep focus on cython-devel myself; I'll be happy to merge next week after
> the release is done.
>
> I don't think what's happened in -devel is a blocker for anything that
> should happen in unstable the next week?
>
> Dag Sverre
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Fri May 15 21:29:17 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 15 May 2009 12:29:17 -0700
Subject: [Cython] ctypedef for integrals
In-Reply-To: 
References: 
	
	
	
	<94b65240f7c9ba5e0c29c2b1ca6c79e2.squirrel@webmail.uio.no>
	
Message-ID: 

On May 15, 2009, at 11:41 AM, Lisandro Dalcin wrote:

> On Fri, May 15, 2009 at 2:03 PM, Dag Sverre Seljebotn
>  wrote:
>> Lisandro wrote:
>>
>> I tried proposing something similar (unknown size) some time ago  
>> and it
>> was voted down; perhaps the climate has changed now.
>
> I voted it down perhaps because I misunderstood your proposal. I
> understood you were proposing this for the whole type system.
>
> If this is done for external ctypedefs, I would no oppose it.

I just don't see how such a construct would be useful. For everything  
that can be deferred to using sizeof (e.g. conversion to/from Python)  
we are moving to using just this, not using the ranking information  
at all, so there's no gain. Now for the stuff where we do use ranking  
information (only temps and type inference come to mind) we either  
use the ranking the user gave (and blame them if they gave it wrong  
enough that it messed them up) or disallow, e.g., all arithmetic  
between unranked and ranked types. Thus

cdef extern from *:
     ctypedef signed unranked int MyInt

cdef MyInt x = 100
print x*5

would be disallowed. You could make an exception for literals, but then

cdef MyInt x = 100
cdef int y = sys.maxint
print x*y

has issues. They could be solved (e.g. doing the a whole if-then list  
on sizeof(MyInt)) but this doesn't seem needed, unless you want to be  
really pedantic.

I just don't see when/why someone would really find this useful,  
perhaps someone could justify it with a good usecase?

>> I support this anyway for more transparency about what Cython does  
>> (though
>> none of those syntaxes), but I don't think it is really needed:
>
> Well, I have a bit of fear of some user coding "cdef extern: ctypedef
> signed long Something" and then Cython stop treating "Something" as
> EXACTLY a "signed long". A user could complain: Hey, I know
> "Something" is EXACTLY a "signed long", Why aren't you using such
> declaration?

If Something is exactly a signed long, it will always be treated and  
behave as so. The only question is what to do if it's not exactly a  
signed long.

>> This only affects situations where the type size is actually used for
>> something. I can only think of constructing temporaries once this  
>> patch
>> goes in.
>>
>> And how many situations can you think of which constructs  
>> temporaries of
>> the type of the result of a binary operation?
>
> What would happen below?
>
> cdef MyInt a, b
> ....
> cdef object o = a BINOP b

A new PyInt_from_MyInt method would be created and called (as you're  
doing now). If a and b were different types, it gets messy,  
especially if things are carried through for a while (e.g. imagine  
"a / b / c / d" where all types are distinct). I say the solution is  
to be more clear about what the ranking information is used for so  
the user can make the declarations appropriately.

- Robert


From robertwb at math.washington.edu  Fri May 15 21:32:43 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 15 May 2009 12:32:43 -0700
Subject: [Cython] ctypedef for integrals
In-Reply-To: 
References: 
	
	
	
	
	
Message-ID: <3F344D60-4AC5-4BC7-A1CC-A2943D4D69D7@math.washington.edu>

On May 15, 2009, at 12:26 PM, Lisandro Dalcin wrote:

> On Fri, May 15, 2009 at 4:03 PM, Robert Bradshaw
>  wrote:
>>
>>>> Also, we are
>>>> assuming that the only sizes that can occur are char, short, int,
>>>> long, and long long. (It sounds inconceivable, but the C spec would
>>>> allow 16-bit ints and 64-bit longs, or 64-bit ints as in ILP64.)
>>>>
>>>
>>> OK... I could fix this returning a PyLong instance.
>>
>> My point was that there are systems with none of char, short, int,
>> long being 32 bits, but your extern type could have 32 bits. The if-
>> then statement dispatching on type sizes wouldn't work.
>>
>
> I understood precisely that, and my point was that is more or less
> easy to support, for example

OK, I was the one who didn't get what you were saying then.

Really, we only need three cases

1) fits in long
2) fits in unsigned long (for unsigned types only)
3) fits in (un)signed long long

- Robert


From stefan_ml at behnel.de  Fri May 15 21:44:35 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 15 May 2009 21:44:35 +0200
Subject: [Cython] Merge of cython-devel to cython-unstable needed
In-Reply-To: 
References: <4A0DBB58.6070301@behnel.de>	<127c1a4bfb524662928d1387c85f97b3.squirrel@webmail.uio.no>
	
Message-ID: <4A0DC623.4060401@behnel.de>


Lisandro Dalcin wrote:
> On Fri, May 15, 2009 at 4:04 PM, Dag Sverre Seljebotn wrote:
>> Stefan Behnel wrote:
>>> the large number of major changes in cython-devel induces a number of
>>> merge
>>> conflicts in cython-unstable. I'm very happy to see all those great
>>> improvements all over the place, but could those who implemented them
>>> please try to re-integrate them with the unstable branch?
>> At least for what I should be doing: Because of time constraints I need to
>> keep focus on cython-devel myself; I'll be happy to merge next week after
>> the release is done.
>
> Well, in PETSc (not petsc4py) we have a rather strict rule. Any change
> that goes to release-maint is immediately merged to -dev. But we never
> introduce new features in release-maint.

Yes, it's a lot easier to adapt changes right after applying them to a
branch than to do so a couple weeks later when you happen to try a merge.
Plus, the latter increases the chance that others stumble into merge hell
for you.

>> I don't think what's happened in -devel is a blocker for anything that
>> should happen in unstable the next week?

I agree that there are no blockers for cython-unstable currently, and I was
merely saying that I won't be the one who will do that merge. But I'm also
suggesting that those who applied these changes should feel responsible for
getting them applied to all three active branches.

Stefan


From dagss at student.matnat.uio.no  Fri May 15 21:44:58 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 15 May 2009 21:44:58 +0200 (CEST)
Subject: [Cython] ctypedef for integrals
In-Reply-To: 
References: 
	
	
	
	<94b65240f7c9ba5e0c29c2b1ca6c79e2.squirrel@webmail.uio.no>
	
	
Message-ID: 

Robert wrote:
> On May 15, 2009, at 11:41 AM, Lisandro Dalcin wrote:
>
>> On Fri, May 15, 2009 at 2:03 PM, Dag Sverre Seljebotn
>>  wrote:
>>> Lisandro wrote:
>>>
>>> I tried proposing something similar (unknown size) some time ago
>>> and it
>>> was voted down; perhaps the climate has changed now.
>>
>> I voted it down perhaps because I misunderstood your proposal. I
>> understood you were proposing this for the whole type system.
>>
>> If this is done for external ctypedefs, I would no oppose it.
>
> I just don't see how such a construct would be useful. For everything
> that can be deferred to using sizeof (e.g. conversion to/from Python)
> we are moving to using just this, not using the ranking information
> at all, so there's no gain. Now for the stuff where we do use ranking
> information (only temps and type inference come to mind) we either
> use the ranking the user gave (and blame them if they gave it wrong
> enough that it messed them up) or disallow, e.g., all arithmetic
> between unranked and ranked types. Thus

What you are missing here is mixing external typedefs (for which a
coherent ranking can be assigned as a closed system, sure) with pure
builtin types.

cdef np.int32_t a = ...
cdef long b = ...
b = a / b

NOTE: This is also relevant for Cython itself without any external
typedefs, as it would be nice to support C99 int32_t, int64_t etc., but we
have no idea how to rank them relative to existing integer types.

For the overview, here's three solutions:
 A) Document, document, document; the behaviour is difficult to understand
but at least we are clear about it
 B) Accept that Cython becomes part of the final compilation stage with
#ifdefs and exact types; this will happen anyway if Cython parses .h
files!!
 C) Trick around the temps as well with some ugly kludge to make ranking
information unecessarry (though nice to have if available because nicer
code is generated).

(I'm feeling up one more but it's still very hazy.)

There's lots of options for C). In addition to what has been mentioned; if
one accept a small ./configure-style prestep, one could e.g. write a
program to output a header file containing

#define sizeof_IntA 4

...and so on. That would do wonders, as you could get widest_IntA_IntB as
a typedef.

Note that in C++ this *might* be less of a problem because one could
likely use template techniques to construct the temps one needs...though I
didn't try.

Dag Sverre


From stefan_ml at behnel.de  Fri May 15 22:52:50 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 15 May 2009 22:52:50 +0200
Subject: [Cython] enumerate optimisation in cython-unstable
Message-ID: <4A0DD622.4080308@behnel.de>

Hi,

couldn't resist to announce that this little optimisation is now in
cython-unstable. An example of a corresponding for-in loop is below.

Have fun,

Stefan

/* ...
 * def multi_c_enumerate():
 *     cdef int a,b,c,d
 *     for a,(b,(c,d)) in enumerate(enumerate(enumerate(range(1,5)))):
        # <<<<<<<<<<<<<<
 *         print a,b,c,d
 */
  __pyx_t_1 = 0;
  __pyx_t_2 = 0;
  __pyx_t_3 = 0;
  for (__pyx_t_4 = 1; __pyx_t_4 < 5; __pyx_t_4+=1) {
    __pyx_v_d = __pyx_t_4;
    __pyx_v_c = __pyx_t_3;
    __pyx_t_3 = (__pyx_t_3 + 1);
    __pyx_v_b = __pyx_t_2;
    __pyx_t_2 = (__pyx_t_2 + 1);
    __pyx_v_a = __pyx_t_1;
    __pyx_t_1 = (__pyx_t_1 + 1);


From robertwb at math.washington.edu  Sat May 16 00:32:47 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 15 May 2009 15:32:47 -0700
Subject: [Cython] enumerate optimisation in cython-unstable
In-Reply-To: <4A0DD622.4080308@behnel.de>
References: <4A0DD622.4080308@behnel.de>
Message-ID: <179020E5-16D8-44C9-B3DD-8D7487948FA3@math.washington.edu>

Cool.

On May 15, 2009, at 1:52 PM, Stefan Behnel wrote:

> Hi,
>
> couldn't resist to announce that this little optimisation is now in
> cython-unstable. An example of a corresponding for-in loop is below.
>
> Have fun,
>
> Stefan
>
> /* ...
>  * def multi_c_enumerate():
>  *     cdef int a,b,c,d
>  *     for a,(b,(c,d)) in enumerate(enumerate(enumerate(range(1,5)))):
>         # <<<<<<<<<<<<<<
>  *         print a,b,c,d
>  */
>   __pyx_t_1 = 0;
>   __pyx_t_2 = 0;
>   __pyx_t_3 = 0;
>   for (__pyx_t_4 = 1; __pyx_t_4 < 5; __pyx_t_4+=1) {
>     __pyx_v_d = __pyx_t_4;
>     __pyx_v_c = __pyx_t_3;
>     __pyx_t_3 = (__pyx_t_3 + 1);
>     __pyx_v_b = __pyx_t_2;
>     __pyx_t_2 = (__pyx_t_2 + 1);
>     __pyx_v_a = __pyx_t_1;
>     __pyx_t_1 = (__pyx_t_1 + 1);
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From robertwb at math.washington.edu  Sat May 16 00:33:32 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 15 May 2009 15:33:32 -0700
Subject: [Cython] Merge of cython-devel to cython-unstable needed
In-Reply-To: <4A0DC623.4060401@behnel.de>
References: <4A0DBB58.6070301@behnel.de>	<127c1a4bfb524662928d1387c85f97b3.squirrel@webmail.uio.no>
	
	<4A0DC623.4060401@behnel.de>
Message-ID: 

On May 15, 2009, at 12:44 PM, Stefan Behnel wrote:

> Lisandro Dalcin wrote:
>> On Fri, May 15, 2009 at 4:04 PM, Dag Sverre Seljebotn wrote:
>>> Stefan Behnel wrote:
>>>> the large number of major changes in cython-devel induces a  
>>>> number of
>>>> merge
>>>> conflicts in cython-unstable. I'm very happy to see all those great
>>>> improvements all over the place, but could those who implemented  
>>>> them
>>>> please try to re-integrate them with the unstable branch?
>>> At least for what I should be doing: Because of time constraints  
>>> I need to
>>> keep focus on cython-devel myself; I'll be happy to merge next  
>>> week after
>>> the release is done.
>>
>> Well, in PETSc (not petsc4py) we have a rather strict rule. Any  
>> change
>> that goes to release-maint is immediately merged to -dev. But we  
>> never
>> introduce new features in release-maint.
>
> Yes, it's a lot easier to adapt changes right after applying them to a
> branch than to do so a couple weeks later when you happen to try a  
> merge.
> Plus, the latter increases the chance that others stumble into  
> merge hell
> for you.
>
>>> I don't think what's happened in -devel is a blocker for anything  
>>> that
>>> should happen in unstable the next week?
>
> I agree that there are no blockers for cython-unstable currently,  
> and I was
> merely saying that I won't be the one who will do that merge. But  
> I'm also
> suggesting that those who applied these changes should feel  
> responsible for
> getting them applied to all three active branches.

If I have time, I'll try my hand at it later tonight.

- Robert



From robertwb at math.washington.edu  Sat May 16 01:06:05 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 15 May 2009 16:06:05 -0700
Subject: [Cython] ctypedef for integrals
In-Reply-To: 
References: 
	
	
	
	<94b65240f7c9ba5e0c29c2b1ca6c79e2.squirrel@webmail.uio.no>
	
	
	
Message-ID: 

On May 15, 2009, at 12:44 PM, Dag Sverre Seljebotn wrote:

> Robert wrote:
>> On May 15, 2009, at 11:41 AM, Lisandro Dalcin wrote:
>>
>>> On Fri, May 15, 2009 at 2:03 PM, Dag Sverre Seljebotn
>>>  wrote:
>>>> Lisandro wrote:
>>>>
>>>> I tried proposing something similar (unknown size) some time ago
>>>> and it
>>>> was voted down; perhaps the climate has changed now.
>>>
>>> I voted it down perhaps because I misunderstood your proposal. I
>>> understood you were proposing this for the whole type system.
>>>
>>> If this is done for external ctypedefs, I would no oppose it.
>>
>> I just don't see how such a construct would be useful. For everything
>> that can be deferred to using sizeof (e.g. conversion to/from Python)
>> we are moving to using just this, not using the ranking information
>> at all, so there's no gain. Now for the stuff where we do use ranking
>> information (only temps and type inference come to mind) we either
>> use the ranking the user gave (and blame them if they gave it wrong
>> enough that it messed them up) or disallow, e.g., all arithmetic
>> between unranked and ranked types. Thus
>
> What you are missing here is mixing external typedefs (for which a
> coherent ranking can be assigned as a closed system, sure) with pure
> builtin types.
>
> cdef np.int32_t a = ...
> cdef long b = ...
> b = a / b

I think your intent here is to have a/b being done in the longest  
type of int32_t and long, as determined at C compile time, right?

> NOTE: This is also relevant for Cython itself without any external
> typedefs, as it would be nice to support C99 int32_t, int64_t etc.,  
> but we
> have no idea how to rank them relative to existing integer types.

I would probably put these in a shipped .pxd file, but the point  
remains that the user isn't determining the rankings.

My resistance is that I would really like to avoid a combinatorial  
explosion of types, especially if every statement using the resulting  
variables has to be written many times surrounded by various if  
statements. For now, we only use type inference for temps and  
intermediate expressions, so it's relatively local, but we should  
come up with a solution that works for variable type inference as  
well. If we don't want to explicitly code every possibility we need  
to query the C compiler, which either breaks shipping .c files or  
creating ./configure like scripts.

> For the overview, here's three solutions:

0) Remove any unnecessary dependance on the ranking (this has already  
mostly happened). It will only be used for widest-type style inference.

>  A) Document, document, document; the behaviour is difficult to  
> understand
> but at least we are clear about it

This should be done regardless.

>  B) Accept that Cython becomes part of the final compilation stage  
> with
> #ifdefs and exact types; this will happen anyway if Cython parses .h
> files!!

I'm not sure exactly what you mean here, but it sounds like this  
would break just being able to ship a single set of generated .c  
files for a variety of platforms.

>  C) Trick around the temps as well with some ugly kludge to make  
> ranking
> information unecessarry (though nice to have if available because  
> nicer
> code is generated).
>
> (I'm feeling up one more but it's still very hazy.)
>
> There's lots of options for C). In addition to what has been  
> mentioned; if
> one accept a small ./configure-style prestep, one could e.g. write a
> program to output a header file containing
>
> #define sizeof_IntA 4
>
> ...and so on. That would do wonders, as you could get  
> widest_IntA_IntB as
> a typedef.

The widest_IntA_IntB idea certainly sounds like the cleanest  
solution, but as above breaks being able to ship .c files (or  
requires a configuration script on the other end).

D) (Not really new) Users can declare their types as a long long, and  
all arithmetic will coerce to it no matter what its size (if we break  
ties by choosing the extern type, or have a new modifier that means  
"wider than anything". Summing two such extern type would be  
unspecified (take the left?)

- Robert



From lrhazi at gmail.com  Sat May 16 01:22:45 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Fri, 15 May 2009 19:22:45 -0400
Subject: [Cython] About using Python's logging facility
Message-ID: 

Hello,

I did an "import logging" in my Cython code, then proceeded to replace
all my "print" debugging statements with logger.debug(), and then some
logger.info(), and then stopped to think...

How bad is it for performance to include such frequent calls to a
standard Python module? I don't understand how Cython works well
enough, Please clarify.

Thanks a lot.
Mohamed.

From robertwb at math.washington.edu  Sat May 16 02:21:14 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 15 May 2009 17:21:14 -0700
Subject: [Cython] About using Python's logging facility
In-Reply-To: 
References: 
Message-ID: <6B2238E6-A433-4129-8F86-438273FD594C@math.washington.edu>

On May 15, 2009, at 4:22 PM, Mohamed Lrhazi wrote:

> Hello,
>
> I did an "import logging" in my Cython code, then proceeded to replace
> all my "print" debugging statements with logger.debug(), and then some
> logger.info(), and then stopped to think...
>
> How bad is it for performance to include such frequent calls to a
> standard Python module? I don't understand how Cython works well
> enough, Please clarify.

It is the cost of a normal Python call (plus running the  
corresponding Python code), which may or may not matter relative to  
the other things you're doing. In other words, it'll cost about the  
same as it would doing it from Python (but you might notice more  
because the surrounding code is faster, so it has a higher relative  
cost). I certainly wouldn't put it in my inner loops, but a call to  
it now and then probably won't hurt.

Though not as flexible as the logging module, branching on a local  
int variable will be orders of magnitude faster.

- Robert


From robertwb at math.washington.edu  Sat May 16 02:26:49 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 15 May 2009 17:26:49 -0700
Subject: [Cython] Merge of cython-devel to cython-unstable needed
In-Reply-To: <127c1a4bfb524662928d1387c85f97b3.squirrel@webmail.uio.no>
References: <4A0DBB58.6070301@behnel.de>
	<127c1a4bfb524662928d1387c85f97b3.squirrel@webmail.uio.no>
Message-ID: <5E668DCB-2220-406D-B39D-C85A968185AE@math.washington.edu>

On May 15, 2009, at 12:04 PM, Dag Sverre Seljebotn wrote:

>> Hi,
>>
>> the large number of major changes in cython-devel induces a number of
>> merge
>> conflicts in cython-unstable. I'm very happy to see all those great
>> improvements all over the place, but could those who implemented them
>> please try to re-integrate them with the unstable branch?
>
> At least for what I should be doing: Because of time constraints I  
> need to
> keep focus on cython-devel myself; I'll be happy to merge next week  
> after
> the release is done.
>
> I don't think what's happened in -devel is a blocker for anything that
> should happen in unstable the next week?

I just took a quick look and it seems most of the conflicts are due  
to refactoring of the Code.py and code streams. Given your work in  
the area it seems, Dag, that you're the one who would be best capable  
of doing this, even if it has to wait 'till next week.

- Robert


From lrhazi at gmail.com  Sat May 16 03:31:54 2009
From: lrhazi at gmail.com (Mohamed Lrhazi)
Date: Fri, 15 May 2009 21:31:54 -0400
Subject: [Cython] About using Python's logging facility
In-Reply-To: <6B2238E6-A433-4129-8F86-438273FD594C@math.washington.edu>
References: 
	<6B2238E6-A433-4129-8F86-438273FD594C@math.washington.edu>
Message-ID: 

Thanks a lot Robert.... The bad thing is that logger.debug() would
still incure the costs you mention, even though it would be igonered
by the logger in production time....
I guess I should be using DEF and IF to only compile those calls at dev time....

Mohamed.


On Fri, May 15, 2009 at 8:21 PM, Robert Bradshaw
 wrote:
> On May 15, 2009, at 4:22 PM, Mohamed Lrhazi wrote:
>
>> Hello,
>>
>> I did an "import logging" in my Cython code, then proceeded to replace
>> all my "print" debugging statements with logger.debug(), and then some
>> logger.info(), and then stopped to think...
>>
>> How bad is it for performance to include such frequent calls to a
>> standard Python module? I don't understand how Cython works well
>> enough, Please clarify.
>
> It is the cost of a normal Python call (plus running the
> corresponding Python code), which may or may not matter relative to
> the other things you're doing. In other words, it'll cost about the
> same as it would doing it from Python (but you might notice more
> because the surrounding code is faster, so it has a higher relative
> cost). I certainly wouldn't put it in my inner loops, but a call to
> it now and then probably won't hurt.
>
> Though not as flexible as the logging module, branching on a local
> int variable will be orders of magnitude faster.
>
> - Robert
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
" Logic merely sanctions the conquests of the intuition."
Jacques Hadamard

From robertwb at math.washington.edu  Sat May 16 10:42:57 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 16 May 2009 01:42:57 -0700
Subject: [Cython] About using Python's logging facility
In-Reply-To: 
References: 
	<6B2238E6-A433-4129-8F86-438273FD594C@math.washington.edu>
	
Message-ID: <8BC83C52-8395-4F2F-9F43-1A8C82287076@math.washington.edu>

On May 15, 2009, at 6:31 PM, Mohamed Lrhazi wrote:

> Thanks a lot Robert.... The bad thing is that logger.debug() would
> still incure the costs you mention, even though it would be igonered
> by the logger in production time....
> I guess I should be using DEF and IF to only compile those calls at  
> dev time....

You could do

cdef int verbosity = 0
cdef inline void log(char* msg, int level):
     if level >= verbosity:
         logger.log(level, msg)

Which would be extremely fast (nanoseconds) when turned off and could  
be turned on/adjusted at runtime. Of course DEF and IF statements  
would work as well.

- Robert


>
> On Fri, May 15, 2009 at 8:21 PM, Robert Bradshaw
>  wrote:
>> On May 15, 2009, at 4:22 PM, Mohamed Lrhazi wrote:
>>
>>> Hello,
>>>
>>> I did an "import logging" in my Cython code, then proceeded to  
>>> replace
>>> all my "print" debugging statements with logger.debug(), and then  
>>> some
>>> logger.info(), and then stopped to think...
>>>
>>> How bad is it for performance to include such frequent calls to a
>>> standard Python module? I don't understand how Cython works well
>>> enough, Please clarify.
>>
>> It is the cost of a normal Python call (plus running the
>> corresponding Python code), which may or may not matter relative to
>> the other things you're doing. In other words, it'll cost about the
>> same as it would doing it from Python (but you might notice more
>> because the surrounding code is faster, so it has a higher relative
>> cost). I certainly wouldn't put it in my inner loops, but a call to
>> it now and then probably won't hurt.
>>
>> Though not as flexible as the logging module, branching on a local
>> int variable will be orders of magnitude faster.
>>
>> - Robert
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
>
> -- 
> " Logic merely sanctions the conquests of the intuition."
> Jacques Hadamard
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From robertwb at math.washington.edu  Sat May 16 10:52:56 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 16 May 2009 01:52:56 -0700
Subject: [Cython] Cython 0.11.2 beta
In-Reply-To: <4A09E193.20001@student.matnat.uio.no>
References: <4A09E193.20001@student.matnat.uio.no>
Message-ID: 

On May 12, 2009, at 1:52 PM, Dag Sverre Seljebotn wrote:

> Please try it out and report any problems:
>
> http://sage.math.washington.edu/home/dagss/Cython-0.11.2.beta.tar.gz

I upgraded a fresh sage-3.4.2 and it built without any changes or  
hitches, and is now running tests (which look good so far).

http://sage.math.washington.edu/home/robertwb/sage-3.4.2-cython/log.txt

> (This is the same trunk as was discussed earlier today, I poked  
> around a
> bit more and didn't find more problems.)
>
> NOTE that the semantics for int conversion has changed in this  
> version,
> so that converting a negative Python int to an unsigned C type will
> raise an OverflowException.

As mentioned before, this went in in 0.11.1, so no need to worry if  
you've already upgraded to that point.

- Robert


From dagss at student.matnat.uio.no  Sat May 16 11:33:28 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 16 May 2009 11:33:28 +0200 (CEST)
Subject: [Cython] Cython 0.11.2 beta
In-Reply-To: 
References: <4A09E193.20001@student.matnat.uio.no>
	
Message-ID: 

> On May 12, 2009, at 1:52 PM, Dag Sverre Seljebotn wrote:
>
>> Please try it out and report any problems:
>>
>> http://sage.math.washington.edu/home/dagss/Cython-0.11.2.beta.tar.gz
>
> I upgraded a fresh sage-3.4.2 and it built without any changes or
> hitches, and is now running tests (which look good so far).
>
> http://sage.math.washington.edu/home/robertwb/sage-3.4.2-cython/log.txt

Yes, I did that before releasing the beta. So my problems is with a
changeset after the beta point, in current cython-devel.

Dag Sverre


From robertwb at math.washington.edu  Sat May 16 11:37:36 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 16 May 2009 02:37:36 -0700
Subject: [Cython] Cython 0.11.2 beta
In-Reply-To: 
References: <4A09E193.20001@student.matnat.uio.no>
	
	
Message-ID: 

On May 16, 2009, at 2:33 AM, Dag Sverre Seljebotn wrote:

>> On May 12, 2009, at 1:52 PM, Dag Sverre Seljebotn wrote:
>>
>>> Please try it out and report any problems:
>>>
>>> http://sage.math.washington.edu/home/dagss/Cython-0.11.2.beta.tar.gz
>>
>> I upgraded a fresh sage-3.4.2 and it built without any changes or
>> hitches, and is now running tests (which look good so far).
>>
>> http://sage.math.washington.edu/home/robertwb/sage-3.4.2-cython/ 
>> log.txt
>
> Yes, I did that before releasing the beta. So my problems is with a
> changeset after the beta point, in current cython-devel.

This is actually with the latest from -devel.

- Robert

From dagss at student.matnat.uio.no  Sat May 16 12:06:55 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 16 May 2009 12:06:55 +0200 (CEST)
Subject: [Cython] Cython 0.11.2 beta
In-Reply-To: 
References: <4A09E193.20001@student.matnat.uio.no>
	
	
	
Message-ID: 

> On May 16, 2009, at 2:33 AM, Dag Sverre Seljebotn wrote:
>
>>> On May 12, 2009, at 1:52 PM, Dag Sverre Seljebotn wrote:
>>>
>>>> Please try it out and report any problems:
>>>>
>>>> http://sage.math.washington.edu/home/dagss/Cython-0.11.2.beta.tar.gz
>>>
>>> I upgraded a fresh sage-3.4.2 and it built without any changes or
>>> hitches, and is now running tests (which look good so far).
>>>
>>> http://sage.math.washington.edu/home/robertwb/sage-3.4.2-cython/
>>> log.txt
>>
>> Yes, I did that before releasing the beta. So my problems is with a
>> changeset after the beta point, in current cython-devel.
>
> This is actually with the latest from -devel.

OK. Great! Will you post again when it is done testing?

I'll then blame the fact that I didn't use screen and got disconnected
(but I was running a detached script and the log indicated completion,
which I trusted....anyway, it is shady enough not to look further into
it.)

I'm still a bit worried about compiling multiple files with your utility
code modifying types; I fixed one instance but there's likely more. I have
an idea for pushing that to code generation time which I'll see if I get
time to implement (though if you have time by all means have a look
yourself, I've got a lot on my platter today).

Then I'll try to get in the latest patches to pyximport, and then do an RC
this weekend.

On my tutorial on Monday I'll just point them to the RC, and then
hopefully release the RC without any changes on Wednesday or so.

Dag Sverre


From dagss at student.matnat.uio.no  Sat May 16 12:12:50 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 16 May 2009 12:12:50 +0200 (CEST)
Subject: [Cython] Cython 0.11.2 beta
In-Reply-To: 
References: <4A09E193.20001@student.matnat.uio.no>
	
	
	
	
Message-ID: <48b63165fa8661ab55749d8d2066b456.squirrel@webmail.uio.no>

On May 16, 2009, at 2:33 AM, Dag Sverre Seljebotn wrote:
> I'm still a bit worried about compiling multiple files with your utility
> code modifying types; I fixed one instance but there's likely more. I have
> an idea for pushing that to code generation time which I'll see if I get
> time to implement (though if you have time by all means have a look
> yourself, I've got a lot on my platter today).

I was referring to this:

http://hg.cython.org/cython-devel/rev/894871645727

(see comment in patch)

Dag Sverre


From robertwb at math.washington.edu  Sat May 16 12:26:57 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 16 May 2009 03:26:57 -0700
Subject: [Cython] Cython 0.11.2 beta
In-Reply-To: 
References: <4A09E193.20001@student.matnat.uio.no>
	
	
	
	
Message-ID: <6E38A358-F189-4847-853E-DD8FEC5B21AA@math.washington.edu>

On May 16, 2009, at 3:06 AM, Dag Sverre Seljebotn wrote:

>> On May 16, 2009, at 2:33 AM, Dag Sverre Seljebotn wrote:
>>
>>>> On May 12, 2009, at 1:52 PM, Dag Sverre Seljebotn wrote:
>>>>
>>>>> Please try it out and report any problems:
>>>>>
>>>>> http://sage.math.washington.edu/home/dagss/ 
>>>>> Cython-0.11.2.beta.tar.gz
>>>>
>>>> I upgraded a fresh sage-3.4.2 and it built without any changes or
>>>> hitches, and is now running tests (which look good so far).
>>>>
>>>> http://sage.math.washington.edu/home/robertwb/sage-3.4.2-cython/
>>>> log.txt
>>>
>>> Yes, I did that before releasing the beta. So my problems is with a
>>> changeset after the beta point, in current cython-devel.
>>
>> This is actually with the latest from -devel.
>
> OK. Great! Will you post again when it is done testing?
>
> I'll then blame the fact that I didn't use screen and got disconnected
> (but I was running a detached script and the log indicated completion,
> which I trusted....anyway, it is shady enough not to look further into
> it.)

----------------------------------------------------------------------
The following tests failed:


	sage -t  "devel/sage/sage/rings/polynomial/polynomial_template.pxi"
	sage -t  "devel/sage/sage/interfaces/sage0.py"
Total time for all tests: 4273.7 seconds
Please see /home/robertwb/sage-3.4.2-cython/tmp/test.log for the  
complete log from this test.

Neither of these look like they're due to Cython.

> I'm still a bit worried about compiling multiple files with your  
> utility
> code modifying types; I fixed one instance but there's likely more.

I doubt it, that was an artifact of some refactoring I did.

> I have
> an idea for pushing that to code generation time which I'll see if  
> I get
> time to implement (though if you have time by all means have a look
> yourself, I've got a lot on my platter today).

:) Me too. Better get to bed now.

> Then I'll try to get in the latest patches to pyximport, and then  
> do an RC
> this weekend.
>
> On my tutorial on Monday I'll just point them to the RC, and then
> hopefully release the RC without any changes on Wednesday or so.

Sure.

- Robert


From dagss at student.matnat.uio.no  Sat May 16 15:50:23 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 16 May 2009 15:50:23 +0200 (CEST)
Subject: [Cython] pyximport ignors [build] compiler = mingw32
In-Reply-To: 
References: 
	
	
	
	 
Message-ID: 

Robert wrote:
> Robert wrote:
>>
>> this and more related cleanup, speedup of importer, --inplace
>> --verbose reload() .. here now:
>>
>> http://trac.cython.org/cython_trac/ticket/312
>>
>
> maybe sb likes to test it further:
>
> * support for dynamic "reload()" added: e.g change of
> the Cython code without stopping & relaunching a big app.
> .reloadNN files may arise, as previously loaded files are
> blocked.

This is great; I've always wanted reload to work...

Reading over the current patch, I see some commented-out code which should
be removed etc.

How ready to you consider this yourself? Should I include it in Cython
0.11.2? (I'm certainly tempted to.)

However with comments like "#very dangerous with --inplace ? " and "TODO"
I'd like a comment from you before deciding :-)

Dag Sverre


From dagss at student.matnat.uio.no  Sat May 16 16:03:43 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 16 May 2009 16:03:43 +0200 (CEST)
Subject: [Cython] pyximport ignors [build] compiler = mingw32
In-Reply-To: 
References: 
	
	
	
	 
	
Message-ID: <5fdf9d28730a8af711bfa84ee0256e56.squirrel@webmail.uio.no>

> Robert wrote:
>> Robert wrote:
>>>
>>> this and more related cleanup, speedup of importer, --inplace
>>> --verbose reload() .. here now:
>>>
>>> http://trac.cython.org/cython_trac/ticket/312
>>>
>>
>> maybe sb likes to test it further:
>>
>> * support for dynamic "reload()" added: e.g change of
>> the Cython code without stopping & relaunching a big app.
>> .reloadNN files may arise, as previously loaded files are
>> blocked.
>
> This is great; I've always wanted reload to work...
>
> Reading over the current patch, I see some commented-out code which should
> be removed etc.
>
> How ready to you consider this yourself? Should I include it in Cython
> 0.11.2? (I'm certainly tempted to.)

Hmm. I just had a segfault with reload_support:

In [1]: import pyximport

In [2]: pyximport.install(build_dir='.', reload_support=True)

In [3]: import integrate_sinsq

In [4]: integrate_sinsq.integrate_sinsq(0, 1, 100)
hello
Out[4]: 0.30606995203805437

In [5]: reload(integrate_sinsq)
Out[5]: 

In [6]: integrate_sinsq.integrate_sinsq(0, 1, 100)
Segmentation fault

The contents of integrate_sinsq is trivial, although it does have a call
so "sin" in math.h (I didn't link m, I think Python loads this for the
module -- at least it works before the reload).

Dag Sverre


From dalcinl at gmail.com  Sat May 16 18:41:59 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 16 May 2009 13:41:59 -0300
Subject: [Cython] Windows calling convention for 0.11.2 ?
Message-ID: 

I've already implemented the @cython.callspec("something") and 'cdef
callspec("something") void f()'

The first one is enough to solve my issues in mpi4py, so I would
really like to get at least that part in. That change should be really
safe, no parser modifications involved.

About the second one, I'm still not sure about that syntax. Moreover,
it modifies the parser so this could potentially break things.

What do you think about pushing the decorator form for 0.11.2, and
defer the other for a future release?



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Sat May 16 18:44:09 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 16 May 2009 18:44:09 +0200 (CEST)
Subject: [Cython] Windows calling convention for 0.11.2 ?
In-Reply-To: 
References: 
Message-ID: <5fefd62b3fb601f7567110b82c945c3e.squirrel@webmail.uio.no>

> I've already implemented the @cython.callspec("something") and 'cdef
> callspec("something") void f()'
>
> The first one is enough to solve my issues in mpi4py, so I would
> really like to get at least that part in. That change should be really
> safe, no parser modifications involved.
>
> About the second one, I'm still not sure about that syntax. Moreover,
> it modifies the parser so this could potentially break things.
>
> What do you think about pushing the decorator form for 0.11.2, and
> defer the other for a future release?

+1. Just push it.

As for the second form, just push it right after the release is done
unless somebody raises their voice in opposition.

Dag Sverre


From jusa.sj at gmail.com  Sat May 16 19:05:59 2009
From: jusa.sj at gmail.com (Juha Salo)
Date: Sat, 16 May 2009 20:05:59 +0300
Subject: [Cython] Weirdness with Cython class: "object has no attribute ..."
Message-ID: <20db9a7d0905161005u3268b7d2je5ab5ac6976cce72@mail.gmail.com>

Hi,
I'm totally out of ideas with this one. I have a Cython class defined in
.pyx and .pxd files like this:

#foo.pxd

cdef class Foo:
    cdef float n[3]

#foo.pyx

cdef class Foo:
    def __init__(self, float a=0.0, float b=0.0, float c=0.0):
         self.n[0] = a
         self.n[1] = b
         self.n[2] = c



The problem is that when I try to import and initialize a Foo object, I get
the following error:

>>> from foo import Foo
>>> f = Foo()

Traceback (most recent call last):
  File "", line 1, in 
  File "foo.pyx", line 3, in foo.Foo.__init__ (foo.c:701)
    self.n[0] = a
AttributeError: 'foo.Foo' object has no attribute 'n'



I've tried all kinds of crazy stuff trying to fix this and I found a strange
thing: When I copy foo.pxd and foo.pyx to another folder and build them from
that path, they work and I get no errors at all. What could cause something
like this?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090516/90d71ac6/attachment.htm 

From stefan_ml at behnel.de  Sat May 16 20:33:31 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 16 May 2009 20:33:31 +0200
Subject: [Cython] Weirdness with Cython class: "object has no attribute
 ..."
In-Reply-To: <20db9a7d0905161005u3268b7d2je5ab5ac6976cce72@mail.gmail.com>
References: <20db9a7d0905161005u3268b7d2je5ab5ac6976cce72@mail.gmail.com>
Message-ID: <4A0F06FB.6080903@behnel.de>


Juha Salo wrote:
> I'm totally out of ideas with this one. I have a Cython class defined in
> .pyx and .pxd files like this:
> 
> #foo.pxd
> 
> cdef class Foo:
>     cdef float n[3]
> 
> #foo.pyx
> 
> cdef class Foo:
>     def __init__(self, float a=0.0, float b=0.0, float c=0.0):
>          self.n[0] = a
>          self.n[1] = b
>          self.n[2] = c
> 
> 
> 
> The problem is that when I try to import and initialize a Foo object, I get
> the following error:
> 
> >>> from foo import Foo
> >>> f = Foo()
> 
> Traceback (most recent call last):
>   File "", line 1, in 
>   File "foo.pyx", line 3, in foo.Foo.__init__ (foo.c:701)
>     self.n[0] = a
> AttributeError: 'foo.Foo' object has no attribute 'n'
> 
> 
> 
> I've tried all kinds of crazy stuff trying to fix this and I found a strange
> thing: When I copy foo.pxd and foo.pyx to another folder and build them from
> that path, they work and I get no errors at all. What could cause something
> like this?

You must keep both in a "foo" package directory that has an __init__.py
file (to mark it as package). That's the way Python packages work.

Stefan


From stefan_ml at behnel.de  Sat May 16 20:43:14 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 16 May 2009 20:43:14 +0200
Subject: [Cython] pyximport ignors [build] compiler = mingw32
In-Reply-To: 
References: 				
	
	
Message-ID: <4A0F0942.3060307@behnel.de>


Dag Sverre Seljebotn wrote:
> Robert wrote:
>> Robert wrote:
>>> this and more related cleanup, speedup of importer, --inplace
>>> --verbose reload() .. here now:
>>>
>>> http://trac.cython.org/cython_trac/ticket/312
>>>
>> maybe sb likes to test it further:
>>
>> * support for dynamic "reload()" added: e.g change of
>> the Cython code without stopping & relaunching a big app.
>> .reloadNN files may arise, as previously loaded files are
>> blocked.
> 
> This is great; I've always wanted reload to work...

Reloading a C module is a very unsafe thing. It basically requires
supporting PEP 3121 in Py3 and won't work in Py2 in most cases. You risk
all sorts of memory leaks and unexpected behaviour.

Stefan

From dalcinl at gmail.com  Sat May 16 20:57:46 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 16 May 2009 15:57:46 -0300
Subject: [Cython] pyximport ignors [build] compiler = mingw32
In-Reply-To: <4A0F0942.3060307@behnel.de>
References: 
	
	
	
	 
	
	<4A0F0942.3060307@behnel.de>
Message-ID: 

On Sat, May 16, 2009 at 3:43 PM, Stefan Behnel  wrote:
>
> Dag Sverre Seljebotn wrote:
>> Robert wrote:
>>> Robert wrote:
>>>> this and more related cleanup, speedup of importer, --inplace
>>>> --verbose reload() .. here now:
>>>>
>>>> http://trac.cython.org/cython_trac/ticket/312
>>>>
>>> maybe sb likes to test it further:
>>>
>>> * support for dynamic "reload()" added: e.g change of
>>> the Cython code without stopping & relaunching a big app.
>>> .reloadNN files may arise, as previously loaded files are
>>> blocked.
>>
>> This is great; I've always wanted reload to work...
>
> Reloading a C module is a very unsafe thing. It basically requires
> supporting PEP 3121 in Py3 and won't work in Py2 in most cases. You risk
> all sorts of memory leaks and unexpected behaviour.
>

However, if things are carefully coded and module cleanup is
generated, reloading could work in Py2. Not sure if this would be
equivalent to reloading, but see
http://code.google.com/p/mpi4py/source/browse/trunk/demo/embedding/helloworld.c
. There you have some code using mpi4py between calls to
Py_Initialize()/Py_Finalize().

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Sat May 16 21:21:20 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 16 May 2009 21:21:20 +0200 (CEST)
Subject: [Cython] pyximport ignors [build] compiler = mingw32
In-Reply-To: <4A0F0942.3060307@behnel.de>
References: 
	
	
	
	 
	
	<4A0F0942.3060307@behnel.de>
Message-ID: <311bf9168cc7ea937556a7391b54cdf1.squirrel@webmail.uio.no>

Dag Sverre Seljebotn wrote:
> Dag Sverre Seljebotn wrote:
>> Robert wrote:
>>> Robert wrote:
>>>> this and more related cleanup, speedup of importer, --inplace
>>>> --verbose reload() .. here now:
>>>>
>>>> http://trac.cython.org/cython_trac/ticket/312
>>>>
>>> maybe sb likes to test it further:
>>>
>>> * support for dynamic "reload()" added: e.g change of
>>> the Cython code without stopping & relaunching a big app.
>>> .reloadNN files may arise, as previously loaded files are
>>> blocked.
>>
>> This is great; I've always wanted reload to work...
>
> Reloading a C module is a very unsafe thing. It basically requires
> supporting PEP 3121 in Py3 and won't work in Py2 in most cases. You risk
> all sorts of memory leaks and unexpected behaviour.

Yes, I remember trying to do this myself and reading some tickets about
this not working in Py2...

Robert (not Bradshaw), can you perhaps submit the patch with the reload
stuff taken out for now?

Idea:

Have pyximport "load" the module by generating a Python file and load it:

from pyxbuilds.foo_1 import *

Then when reloading this py-file, hook that, do a new compile and
overwrite it with

from pyxbuilds.foo_2 import *

before Python reloads as usual. Could this work?

Dag Sverre


From dalcinl at gmail.com  Sat May 16 21:41:56 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 16 May 2009 16:41:56 -0300
Subject: [Cython] Windows calling convention for 0.11.2 ?
In-Reply-To: <5fefd62b3fb601f7567110b82c945c3e.squirrel@webmail.uio.no>
References: 
	<5fefd62b3fb601f7567110b82c945c3e.squirrel@webmail.uio.no>
Message-ID: 

On Sat, May 16, 2009 at 1:44 PM, Dag Sverre Seljebotn
 wrote:
>> I've already implemented the @cython.callspec("something") and 'cdef
>> callspec("something") void f()'
>>
>> The first one is enough to solve my issues in mpi4py, so I would
>> really like to get at least that part in. That change should be really
>> safe, no parser modifications involved.
>>
>> About the second one, I'm still not sure about that syntax. Moreover,
>> it modifies the parser so this could potentially break things.
>>
>> What do you think about pushing the decorator form for 0.11.2, and
>> defer the other for a future release?
>
> +1. Just push it.
>

OK.  I'm going to add this:

-calling_convention_words = ("__stdcall", "__cdecl")
+calling_convention_words = ("__stdcall", "__cdecl", "__fastcall")


Buth then, I'm in doubt about what to do with this bit:

     def same_calling_convention_as(self, other):
        sc1 = self.calling_convention == '__stdcall'
        sc2 = other.calling_convention == '__stdcall'
        return sc1 == sc2

This code did not take into account "__fastcall". Moreover, it assumes
that "__cdecl" is compatible with a empty calling convention. But IMHO
that is dangerous, as the actual calling convention for functions with
empty callspec can be tweaked by passing flags to the C compiler.

I'm thinking about doing things like this:

     def same_calling_convention_as(self, other):
        callspec_words = ("__stdcall", "__cdecl", "__fastcall")
        cs1 = self.calling_convention
        cs2 = other.calling_convention
        if (cs1 in callspec_words or
            cs2 in callspec_words):
            return cs1 == cs2
        else:
            return True



> As for the second form, just push it right after the release is done
> unless somebody raises their voice in opposition.
>
> Dag Sverre
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Sat May 16 22:05:45 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 16 May 2009 22:05:45 +0200 (CEST)
Subject: [Cython] Windows calling convention for 0.11.2 ?
In-Reply-To: 
References: 
	<5fefd62b3fb601f7567110b82c945c3e.squirrel@webmail.uio.no>
	
Message-ID: 

> On Sat, May 16, 2009 at 1:44 PM, Dag Sverre Seljebotn
>  wrote:
>>> I've already implemented the @cython.callspec("something") and 'cdef
>>> callspec("something") void f()'
>>>
>>> The first one is enough to solve my issues in mpi4py, so I would
>>> really like to get at least that part in. That change should be really
>>> safe, no parser modifications involved.
>>>
>>> About the second one, I'm still not sure about that syntax. Moreover,
>>> it modifies the parser so this could potentially break things.
>>>
>>> What do you think about pushing the decorator form for 0.11.2, and
>>> defer the other for a future release?
>>
>> +1. Just push it.
>>
>
> OK.  I'm going to add this:
>
> -calling_convention_words = ("__stdcall", "__cdecl")
> +calling_convention_words = ("__stdcall", "__cdecl", "__fastcall")
>
>
> Buth then, I'm in doubt about what to do with this bit:
>
>      def same_calling_convention_as(self, other):
>         sc1 = self.calling_convention == '__stdcall'
>         sc2 = other.calling_convention == '__stdcall'
>         return sc1 == sc2
>
> This code did not take into account "__fastcall". Moreover, it assumes
> that "__cdecl" is compatible with a empty calling convention. But IMHO
> that is dangerous, as the actual calling convention for functions with
> empty callspec can be tweaked by passing flags to the C compiler.
>
> I'm thinking about doing things like this:
>
>      def same_calling_convention_as(self, other):
>         callspec_words = ("__stdcall", "__cdecl", "__fastcall")
>         cs1 =
>         cs2 = other.calling_convention
>         if (cs1 in callspec_words or
>             cs2 in callspec_words):
>             return cs1 == cs2
>         else:
>             return True

Why is this function only considering those, and not user-defined
callspecs as well?

It would seem to me that simply

self.calling_convention == other.calling_convention

would seem natural here, if one cannot trust "no convention" to be the
same as any others.

Dag Sverre


From dalcinl at gmail.com  Sat May 16 22:32:44 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 16 May 2009 17:32:44 -0300
Subject: [Cython] Windows calling convention for 0.11.2 ?
In-Reply-To: 
References: 
	<5fefd62b3fb601f7567110b82c945c3e.squirrel@webmail.uio.no>
	
	
Message-ID: 

On Sat, May 16, 2009 at 5:05 PM, Dag Sverre Seljebotn
 wrote:
>> ? ? ?def same_calling_convention_as(self, other):
>> ? ? ? ? callspec_words = ("__stdcall", "__cdecl", "__fastcall")
>> ? ? ? ? cs1 =
>> ? ? ? ? cs2 = other.calling_convention
>> ? ? ? ? if (cs1 in callspec_words or
>> ? ? ? ? ? ? cs2 in callspec_words):
>> ? ? ? ? ? ? return cs1 == cs2
>> ? ? ? ? else:
>> ? ? ? ? ? ? return True
>
> Why is this function only considering those, and not user-defined
> callspecs as well?
>
> It would seem to me that simply
>
> self.calling_convention == other.calling_convention
>
> would seem natural here, if one cannot trust "no convention" to be the
> same as any others.
>

Two reasons from my side:

1) If we are going to just push the decorator form, there is no way to
annotate external declarations until 'callspec("something")' is
introduced.

2) User-defined (and preprocessor macro controlled) calling
conventions should be seen as a power-user feature for solving corner
cases. Then I would like to be strict only with the "standard" calling
convention names. When a user has to start using things like
@cython.callspec("MY_CALL_SPEC"), I'm pretty sure that uses knows very
well why he has to do that, and also have managed to make sure
MY_CALL_SPEC is defined in the preprocessor...

Let's go back to mpi4py. I have to use
@cython.callspec("PyMPI_API_CALL") in a few callback functions. If I
were forced to add such decoration (or annotation using the second
part of my patch) in the MPI calls declared on my pxd's then I would
likely generate trouble to all mpi4py users, and almost all of them
are not working on Windows...




-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Sat May 16 23:02:03 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 16 May 2009 23:02:03 +0200 (CEST)
Subject: [Cython] Windows calling convention for 0.11.2 ?
In-Reply-To: 
References: 
	<5fefd62b3fb601f7567110b82c945c3e.squirrel@webmail.uio.no>
	
	
	
Message-ID: <25a31e4256e3c2a367755e1f65e68e38.squirrel@webmail.uio.no>

Lisandro wrote:
> On Sat, May 16, 2009 at 5:05 PM, Dag Sverre Seljebotn
>  wrote:
>>> ? ? ?def same_calling_convention_as(self, other):
>>> ? ? ? ? callspec_words = ("__stdcall", "__cdecl", "__fastcall")
>>> ? ? ? ? cs1 =
>>> ? ? ? ? cs2 = other.calling_convention
>>> ? ? ? ? if (cs1 in callspec_words or
>>> ? ? ? ? ? ? cs2 in callspec_words):
>>> ? ? ? ? ? ? return cs1 == cs2
>>> ? ? ? ? else:
>>> ? ? ? ? ? ? return True
>>
>> Why is this function only considering those, and not user-defined
>> callspecs as well?
>>
>> It would seem to me that simply
>>
>> self.calling_convention == other.calling_convention
>>
>> would seem natural here, if one cannot trust "no convention" to be the
>> same as any others.
>>
>
> Two reasons from my side:
>
> 1) If we are going to just push the decorator form, there is no way to
> annotate external declarations until 'callspec("something")' is
> introduced.
>
> 2) User-defined (and preprocessor macro controlled) calling
> conventions should be seen as a power-user feature for solving corner
> cases. Then I would like to be strict only with the "standard" calling
> convention names. When a user has to start using things like
> @cython.callspec("MY_CALL_SPEC"), I'm pretty sure that uses knows very
> well why he has to do that, and also have managed to make sure
> MY_CALL_SPEC is defined in the preprocessor...
>
> Let's go back to mpi4py. I have to use
> @cython.callspec("PyMPI_API_CALL") in a few callback functions. If I
> were forced to add such decoration (or annotation using the second
> part of my patch) in the MPI calls declared on my pxd's then I would
> likely generate trouble to all mpi4py users, and almost all of them
> are not working on Windows...

Sounds reasonable. I'm worried about this breaking backwards compatability
though (for __cdecl).

Could you just push basically what you have written above, but where
__cdecl is still interchangeable with the empty callspec, until we have
more input from others perhaps?

Dag Sverre


From dagss at student.matnat.uio.no  Sat May 16 23:03:23 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 16 May 2009 23:03:23 +0200 (CEST)
Subject: [Cython] Windows calling convention for 0.11.2 ?
In-Reply-To: 
References: 
	<5fefd62b3fb601f7567110b82c945c3e.squirrel@webmail.uio.no>
	
	
	
Message-ID: <0e11b411ad9422997286a8b8a720ffd0.squirrel@webmail.uio.no>

> On Sat, May 16, 2009 at 5:05 PM, Dag Sverre Seljebotn
>  wrote:
>>> ? ? ?def same_calling_convention_as(self, other):
>>> ? ? ? ? callspec_words = ("__stdcall", "__cdecl", "__fastcall")
>>> ? ? ? ? cs1 =
>>> ? ? ? ? cs2 = other.calling_convention
>>> ? ? ? ? if (cs1 in callspec_words or
>>> ? ? ? ? ? ? cs2 in callspec_words):
>>> ? ? ? ? ? ? return cs1 == cs2
>>> ? ? ? ? else:
>>> ? ? ? ? ? ? return True
>>
>> Why is this function only considering those, and not user-defined
>> callspecs as well?
>>
>> It would seem to me that simply
>>
>> self.calling_convention == other.calling_convention
>>
>> would seem natural here, if one cannot trust "no convention" to be the
>> same as any others.
>>
>
> Two reasons from my side:
>
> 1) If we are going to just push the decorator form, there is no way to
> annotate external declarations until 'callspec("something")' is
> introduced.
>
> 2) User-defined (and preprocessor macro controlled) calling
> conventions should be seen as a power-user feature for solving corner
> cases. Then I would like to be strict only with the "standard" calling
> convention names. When a user has to start using things like
> @cython.callspec("MY_CALL_SPEC"), I'm pretty sure that uses knows very
> well why he has to do that, and also have managed to make sure
> MY_CALL_SPEC is defined in the preprocessor...
>
> Let's go back to mpi4py. I have to use
> @cython.callspec("PyMPI_API_CALL") in a few callback functions. If I
> were forced to add such decoration (or annotation using the second
> part of my patch) in the MPI calls declared on my pxd's then I would
> likely generate trouble to all mpi4py users, and almost all of them
> are not working on Windows...
>

Note that I'll make the RC during the weekend, so please push as much as
you can as early as possible. I.e. the @callspec should be able to go in
before this is resolved, seems orthogonal.

Dag Sverre


From kxroberto at googlemail.com  Sat May 16 23:36:05 2009
From: kxroberto at googlemail.com (Robert)
Date: Sat, 16 May 2009 23:36:05 +0200
Subject: [Cython] pyximport ignors [build] compiler = mingw32
In-Reply-To: <311bf9168cc7ea937556a7391b54cdf1.squirrel@webmail.uio.no>
References: 				
			<4A0F0942.3060307@behnel.de>
	<311bf9168cc7ea937556a7391b54cdf1.squirrel@webmail.uio.no>
Message-ID: 

Dag Sverre Seljebotn wrote:
> Dag Sverre Seljebotn wrote:
>> Dag Sverre Seljebotn wrote:
>>> Robert wrote:
>>>> Robert wrote:
>>>>> this and more related cleanup, speedup of importer, --inplace
>>>>> --verbose reload() .. here now:
>>>>>
>>>>> http://trac.cython.org/cython_trac/ticket/312
>>>>>
>>>> maybe sb likes to test it further:
>>>>
>>>> * support for dynamic "reload()" added: e.g change of
>>>> the Cython code without stopping & relaunching a big app.
>>>> .reloadNN files may arise, as previously loaded files are
>>>> blocked.
>>> This is great; I've always wanted reload to work...
>> Reloading a C module is a very unsafe thing. It basically requires
>> supporting PEP 3121 in Py3 and won't work in Py2 in most cases. You risk
>> all sorts of memory leaks and unexpected behaviour.
> 
> Yes, I remember trying to do this myself and reading some tickets about
> this not working in Py2...
> 
> Robert (not Bradshaw), can you perhaps submit the patch with the reload
> stuff taken out for now?

what is the time schedule?
a new one is there: 
http://trac.cython.org/cython_trac/attachment/ticket/312/
http://trac.cython.org/cython_trac/attachment/ticket/312/pyxi3.patch


did you with install(... reload_support=True) ? or without?

I just saw that on Linux you can (even without any 
Cython/pyximport) overwrite the .so,  then reload(themodule) and: 
Bus error or Segfault.

On Windows you cannot overwrite/remove/rename a loaded DLL file - 
and I (implicitly) relied on that - even when 
reload_support=False. The build,linking actually fails automatically.

I locked the reload in normal mode again completely as it was 
before and with reload_support=True it now goes always via fresh 
temporary .reloadXX files, not relying on the blockage.


> Idea:
> 
> Have pyximport "load" the module by generating a Python file and load it:
> 
> from pyxbuilds.foo_1 import *
> 
> Then when reloading this py-file, hook that, do a new compile and
> overwrite it with
> 
> from pyxbuilds.foo_2 import *
> 
> before Python reloads as usual. Could this work?
> 

this is what implicitely happens/happened here with 
reload_support=True , loading from another temporary file - so far 
on Windows. Yet he iterated just upon the write blockage. Yet low 
level via imp.load_dynamic (no restriction on the shape of filename).
( The thing with extra python file is 95%+ not necessary I guess. 
As its a low level DLL/filesystem thing, not about the name in 
sys.modules or so)

On Win there was no problem; used it all the time since I made it 
- with file in package trees too. I typically reload a 
continuously edited .pyx module up to some 20+ times into a 
running GUI app, and wonderful.

The memory usage of course will go up, DLL relocation going on 
etc... but its anyway just for dev/debugging.


I have no Cython setup currently on my Linux, but will go tomorrow 
see myself - but it shall work now.
With reload_support=False (default) at least, there is full stop 
again anyway. And the rest is for debugging anyway.


Robert












From dagss at student.matnat.uio.no  Sat May 16 23:49:50 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 16 May 2009 23:49:50 +0200 (CEST)
Subject: [Cython] pyximport ignors [build] compiler = mingw32
In-Reply-To: 
References: 
	
	
	
	 
	
	<4A0F0942.3060307@behnel.de>
	<311bf9168cc7ea937556a7391b54cdf1.squirrel@webmail.uio.no>
	
Message-ID: 

Robert wrote:
> Dag Sverre Seljebotn wrote:
>> Robert (not Bradshaw), can you perhaps submit the patch with the reload
>> stuff taken out for now?
>
> what is the time schedule?
> a new one is there:
> http://trac.cython.org/cython_trac/attachment/ticket/312/
> http://trac.cython.org/cython_trac/attachment/ticket/312/pyxi3.patch

A release candidate is dropped any time I get time for it, certainly
within the next 24 hours. After then this would have to go in 0.11.3 (it
might have to anyway, not sure yet).

> did you with install(... reload_support=True) ? or without?

With.

> I just saw that on Linux you can (even without any
> Cython/pyximport) overwrite the .so,  then reload(themodule) and:
> Bus error or Segfault.
>
> On Windows you cannot overwrite/remove/rename a loaded DLL file -
> and I (implicitly) relied on that - even when
> reload_support=False. The build,linking actually fails automatically.

I will have to try your new patch, but don't think I'll get time before
the release.

> this is what implicitely happens/happened here with
> reload_support=True , loading from another temporary file - so far
> on Windows. Yet he iterated just upon the write blockage. Yet low
> level via imp.load_dynamic (no restriction on the shape of filename).
> ( The thing with extra python file is 95%+ not necessary I guess.
> As its a low level DLL/filesystem thing, not about the name in
> sys.modules or so)

I find threads on the internet with people struggling with the PEP which
pyximport uses and reload functionality; I'm fairly sure it is more
complicated than this.

I'd rather have a patch with only the configuration-file related changes,
otherwise this will definitely have to wait for the next release (if I was
less time-constrained I would have been kinder :-) ).

(It's also nicer if you can use "hg export" to create the patches.)

Dag Sverre


From kxroberto at googlemail.com  Sat May 16 23:52:34 2009
From: kxroberto at googlemail.com (Robert)
Date: Sat, 16 May 2009 23:52:34 +0200
Subject: [Cython] pyximport ignors [build] compiler = mingw32
In-Reply-To: <311bf9168cc7ea937556a7391b54cdf1.squirrel@webmail.uio.no>
References: 				
			<4A0F0942.3060307@behnel.de>
	<311bf9168cc7ea937556a7391b54cdf1.squirrel@webmail.uio.no>
Message-ID: 

Dag Sverre Seljebotn wrote:

> from pyxbuilds.foo_2 import *

Have to rethink if this intermediate namespace is still necessary, 
because of perhaps the old post-import created module globals, 
which shall perhaps survive, when not 'overwritten' by import.
  guess its hardly a need.  The reload handler could transport the 
globals ... but I have to actually check it out :-)

Thats a special issue - had no importance for my use cases.

I typically have a short "re-work things upon reload code" in 
those modules, which I iterate frequently upon - and that enables 
for a reload and quicker edit-run cycle in 90% of code changes.


Robert


From dalcinl at gmail.com  Sun May 17 01:12:37 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Sat, 16 May 2009 20:12:37 -0300
Subject: [Cython] Windows calling convention for 0.11.2 ?
In-Reply-To: <0e11b411ad9422997286a8b8a720ffd0.squirrel@webmail.uio.no>
References: 
	<5fefd62b3fb601f7567110b82c945c3e.squirrel@webmail.uio.no>
	
	
	
	<0e11b411ad9422997286a8b8a720ffd0.squirrel@webmail.uio.no>
Message-ID: 

On Sat, May 16, 2009 at 6:03 PM, Dag Sverre Seljebotn
 wrote:
>
> Note that I'll make the RC during the weekend, so please push as much as
> you can as early as possible. I.e. the @callspec should be able to go in
> before this is resolved, seems orthogonal.
>

Pushed.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From kxroberto at googlemail.com  Sun May 17 01:17:23 2009
From: kxroberto at googlemail.com (Robert)
Date: Sun, 17 May 2009 01:17:23 +0200
Subject: [Cython] pyximport ignors [build] compiler = mingw32
In-Reply-To: 
References: 				
			<4A0F0942.3060307@behnel.de>	<311bf9168cc7ea937556a7391b54cdf1.squirrel@webmail.uio.no>	
	
Message-ID: 

Dag Sverre Seljebotn wrote:
> Robert wrote:
>> Dag Sverre Seljebotn wrote:
>>> Robert (not Bradshaw), can you perhaps submit the patch with the reload
>>> stuff taken out for now?
>> what is the time schedule?
>> a new one is there:
>> http://trac.cython.org/cython_trac/attachment/ticket/312/
>> http://trac.cython.org/cython_trac/attachment/ticket/312/pyxi3.patch
> 
> A release candidate is dropped any time I get time for it, certainly
> within the next 24 hours. After then this would have to go in 0.11.3 (it
> might have to anyway, not sure yet).
> 
>> did you with install(... reload_support=True) ? or without?
> 
> With.
> 
>> I just saw that on Linux you can (even without any
>> Cython/pyximport) overwrite the .so,  then reload(themodule) and:
>> Bus error or Segfault.
>>
>> On Windows you cannot overwrite/remove/rename a loaded DLL file -
>> and I (implicitly) relied on that - even when
>> reload_support=False. The build,linking actually fails automatically.
> 
> I will have to try your new patch, but don't think I'll get time before
> the release.
> 

yes, its maybe too much a rush

>> this is what implicitely happens/happened here with
>> reload_support=True , loading from another temporary file - so far
>> on Windows. Yet he iterated just upon the write blockage. Yet low
>> level via imp.load_dynamic (no restriction on the shape of filename).
>> ( The thing with extra python file is 95%+ not necessary I guess.
>> As its a low level DLL/filesystem thing, not about the name in
>> sys.modules or so)
> 
> I find threads on the internet with people struggling with the PEP which
> pyximport uses and reload functionality; I'm fairly sure it is more
> complicated than this.
> 

I've set up my Linux too now. That latest pyxi3.patch and it does.

With reload_support=False its exactly like the old 0.11.1 behavior :

 >>> import my	
 >>> reload(my)
Traceback (most recent call last):
   File "", line 1, in ?
ImportError: No module named my

which is in my opinion bogus, because
 >>> reload(time)




and with reload_support=True its now ok

 >>> my.f()
from f
5
 >>> reload(my)

 >>> reload(my)

 >>> reload(my)

 >>> reload(my)    # changed the .pyx code here <<<<<<<<<<<<
hello from pyx

 >>> my.f()
from f
7


tried things in a few cases a dozen times. Think it was only that 
issue, that .so's are overwriteable.
The reload of a different file with this imp-method is simply like 
a new module object (in other folder e.g.) (with same name). Guess 
it has not to do with this PEP.

It crashed here before the patch (same as when I overwrote time.so 
and reload(time) ), and now not anymore.

But more people should try on different installations ...

When ok, I recommend to change the old&again reload_support=False 
behavior to do the same as normal reload(time) etc and not cause 
"ImportError: No module named my" in future.



> I'd rather have a patch with only the configuration-file related changes,
> otherwise this will definitely have to wait for the next release (if I was
> less time-constrained I would have been kinder :-) ).
> 
> (It's also nicer if you can use "hg export" to create the patches.)
> 

have to learn this; never used ..



Robert


From kxroberto at googlemail.com  Sun May 17 01:42:34 2009
From: kxroberto at googlemail.com (Robert)
Date: Sun, 17 May 2009 01:42:34 +0200
Subject: [Cython] pyximport ignors [build] compiler = mingw32
In-Reply-To: 
References: 				
			<4A0F0942.3060307@behnel.de>	<311bf9168cc7ea937556a7391b54cdf1.squirrel@webmail.uio.no>	
	
Message-ID: 

Dag Sverre Seljebotn wrote:
> I'd rather have a patch with only the configuration-file related changes,
> otherwise this will definitely have to wait for the next release (if I was
> less time-constrained I would have been kinder :-) ).

.. forgot on that. I've no separate branches here without breaking 
things up again manually. the changes regarding the config 
files/args are even more/deeper. best lets both mature together.


Robert


From jusa.sj at gmail.com  Sun May 17 08:33:49 2009
From: jusa.sj at gmail.com (Juha Salo)
Date: Sun, 17 May 2009 09:33:49 +0300
Subject: [Cython] Weirdness with Cython class: "object has no attribute
	..."
In-Reply-To: <4A0F06FB.6080903@behnel.de>
References: <20db9a7d0905161005u3268b7d2je5ab5ac6976cce72@mail.gmail.com>
	<4A0F06FB.6080903@behnel.de>
Message-ID: <20db9a7d0905162333v47c522e4gad2939cbd4a5207@mail.gmail.com>

It seems that the class builds right only from a folder *without* an
__init__.py file. Previously I was trying to build it from a package folder
and removing the __init__.py made the class work again. Strange.
I found that I can make the resulting foo.pyd work within a Python package
afterwards by moving the .pyd file to the desired package folder (wich
contains an __init__.py file). Now I can "from foo_pck.foo import Foo" and
the class still works.

2009/5/16 Stefan Behnel 
>
>
> You must keep both in a "foo" package directory that has an __init__.py
> file (to mark it as package). That's the way Python packages work.
>
> Stefan
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090517/0e9d83c1/attachment.htm 

From kxroberto at googlemail.com  Sun May 17 14:11:42 2009
From: kxroberto at googlemail.com (Robert)
Date: Sun, 17 May 2009 14:11:42 +0200
Subject: [Cython] pyximport ignors [build] compiler = mingw32
In-Reply-To: 
References: 							<4A0F0942.3060307@behnel.de>	<311bf9168cc7ea937556a7391b54cdf1.squirrel@webmail.uio.no>		
	
Message-ID: 

Robert wrote:
> 
> I've set up my Linux too now. That latest pyxi3.patch and it does.
> 
> With reload_support=False its exactly like the old 0.11.1 behavior :
> 
>  >>> import my	
>  >>> reload(my)
> Traceback (most recent call last):
>    File "", line 1, in ?
> ImportError: No module named my
> 
> which is in my opinion bogus, because
>  >>> reload(time)
> 
> 
> 
> 
> and with reload_support=True its now ok
> 


I've done a few more tests on Linux and Win, and it does so far on 
my side in all modes - for practical edit-run/reload debugging 
purposes. The reload_support=False is anyway default.

the little 3+.patch in attachment in addition to pyxi3.patch
http://trac.cython.org/cython_trac/ticket/312
(or manually against 0.11.1)
finally would solve the mentioned "reload(my) -> ImportError: No 
module named my" problem of 0.11.1  or of reload_support=False 
default mode

I leave it open so far from my side.


Robert











-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: 3+.patch
Url: http://codespeak.net/pipermail/cython-dev/attachments/20090517/6f8bd12e/attachment.diff 

From dagss at student.matnat.uio.no  Sun May 17 15:49:30 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 17 May 2009 15:49:30 +0200 (CEST)
Subject: [Cython] Cython 0.11.2 release candidate
Message-ID: <8222a0ef1408ac1532b5be32ea838103.squirrel@webmail.uio.no>

A "release candidate" is here:

http://sage.math.washington.edu/home/dagss/Cython-0.11.2.rc1.tar.gz
http://sage.math.washington.edu/home/dagss/Cython-0.11.2.rc1.zip

I already know about two smallish issues [1], but I won't have more time
for this until Wednesday and so I thought it was better to get this out
sooner rather than later.

This marks the final feature freeze. Consider cython-devel open for
general development again; any changes to 0.11.2 should be attached as
patches in Trac and I'll merge them on my local branch.

[1] http://trac.cython.org/cython_trac/report/3

Dag Sverre


From sccolbert at gmail.com  Sun May 17 22:23:28 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Sun, 17 May 2009 16:23:28 -0400
Subject: [Cython] interfacing with OpenCV?
Message-ID: <7f014ea60905171323j5fe490bfm8ba57de2352eea07@mail.gmail.com>

How would I go about interfacing with OpenCV data types in cython?

I've read the wiki on interfacing with external C code but didn't fully
follow.

I'm currently using a ctypes wrapper for OpenCV from python and doing some
manipulation there, but I would like to chain together a bunch of functions
to run outside of the GIL on another core.

>From ctypes, I can pass a function a pointer to my IPLimage (the opencv
data).

So I would imagine it would be something along the lines of:

- write my functions in a pure C source file
- declare extern functions in cython for the functions in the C source
- compile the Cython file
- compil the C source and compiled cython into the same pyd file

The program flow would be like:
- pass the pointer to the image to the cython function
- forward that pointer to another function that executes with nogil which
calls the functions from the C library
- return the pointer to the modified image to calling function
- return that same pointer back to python.


Is that about right? Or am I way off?

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090517/95945b37/attachment.htm 

From jusa.sj at gmail.com  Mon May 18 15:58:26 2009
From: jusa.sj at gmail.com (Juha Salo)
Date: Mon, 18 May 2009 16:58:26 +0300
Subject: [Cython] Modifying module-level Cython variable in a Cython function
Message-ID: <20db9a7d0905180658o6001911au1c205ea3657f12ab@mail.gmail.com>

Hi,
I ran into a rather basic problem where I can't seem to change the value of
a module-level Cython variable through a cpdef'd Cython function. I have the
following:

#foo.pyx

cdef float foovar = 0.001

cpdef float get_foo():
    return foovar

cpdef set_foo(float v=0.001):
    foovar = v



When I call get_foo() at the Python interpreter I get the value 0.001, but
trying to change foovar's value with set_foo() has no effect. I also tried
to cpdef foovar but that didn't help. I could probably (?) define foovar as
a regular Python variable, but I'd rather have it as a Cython variable for
speed. Is there anything special that needs to be done to get this to work?

Thanks,
J.S.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090518/fa339b3e/attachment.htm 

From lasizoillo at gmail.com  Mon May 18 16:07:29 2009
From: lasizoillo at gmail.com (lasizoillo)
Date: Mon, 18 May 2009 16:07:29 +0200
Subject: [Cython] Modifying module-level Cython variable in a Cython
	function
In-Reply-To: <20db9a7d0905180658o6001911au1c205ea3657f12ab@mail.gmail.com>
References: <20db9a7d0905180658o6001911au1c205ea3657f12ab@mail.gmail.com>
Message-ID: <49414f570905180707p8db3c07q7349bf081f61504b@mail.gmail.com>

2009/5/18 Juha Salo :
> Hi,
> I ran into a rather basic problem where I can't seem to change the value of
> a module-level Cython variable through a cpdef'd Cython function. I have the
> following:
> #foo.pyx
> cdef float foovar = 0.001
> cpdef float get_foo():
> ?? ?return foovar
> cpdef set_foo(float v=0.001):
> ?? ?foovar = v

Maybe you are modifing a local variable foovar in set_foo function.
Try to use global scope or encapsulate all in a class.

>
>
> When I call get_foo() at the Python interpreter I get the value 0.001, but
> trying to change foovar's?value with set_foo() has no effect. I also tried
> to cpdef foovar but that didn't help. I could probably (?) define foovar as
> a regular Python variable, but I'd rather have it as a Cython variable for
> speed. Is there anything special that needs to be done to get this to work?
> Thanks,
> J.S.
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>

From dalcinl at gmail.com  Mon May 18 17:59:42 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Mon, 18 May 2009 12:59:42 -0300
Subject: [Cython] Modifying module-level Cython variable in a Cython
	function
In-Reply-To: <20db9a7d0905180658o6001911au1c205ea3657f12ab@mail.gmail.com>
References: <20db9a7d0905180658o6001911au1c205ea3657f12ab@mail.gmail.com>
Message-ID: 

As in normal Python code, use 'global', if not, Cython will treat
'foovar' as a local variable:

cpdef set_foo(float v=0.001):
    global foobar
    foovar = v




On Mon, May 18, 2009 at 10:58 AM, Juha Salo  wrote:
> Hi,
> I ran into a rather basic problem where I can't seem to change the value of
> a module-level Cython variable through a cpdef'd Cython function. I have the
> following:
> #foo.pyx
> cdef float foovar = 0.001
> cpdef float get_foo():
> ?? ?return foovar
> cpdef set_foo(float v=0.001):
> ?? ?foovar = v
>
>
> When I call get_foo() at the Python interpreter I get the value 0.001, but
> trying to change foovar's?value with set_foo() has no effect. I also tried
> to cpdef foovar but that didn't help. I could probably (?) define foovar as
> a regular Python variable, but I'd rather have it as a Cython variable for
> speed. Is there anything special that needs to be done to get this to work?
> Thanks,
> J.S.
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From jusa.sj at gmail.com  Mon May 18 18:56:44 2009
From: jusa.sj at gmail.com (Juha Salo)
Date: Mon, 18 May 2009 19:56:44 +0300
Subject: [Cython] Modifying module-level Cython variable in a Cython
	function
In-Reply-To: 
References: <20db9a7d0905180658o6001911au1c205ea3657f12ab@mail.gmail.com>
	
Message-ID: <20db9a7d0905180956i44b7287fv510682aef79180ae@mail.gmail.com>

That solved it. Thanks!

2009/5/18 Lisandro Dalcin 

> As in normal Python code, use 'global', if not, Cython will treat
> 'foovar' as a local variable:
>
> cpdef set_foo(float v=0.001):
>     global foobar
>    foovar = v
>
>
>
>
> On Mon, May 18, 2009 at 10:58 AM, Juha Salo  wrote:
> > Hi,
> > I ran into a rather basic problem where I can't seem to change the value
> of
> > a module-level Cython variable through a cpdef'd Cython function. I have
> the
> > following:
> > #foo.pyx
> > cdef float foovar = 0.001
> > cpdef float get_foo():
> >     return foovar
> > cpdef set_foo(float v=0.001):
> >     foovar = v
> >
> >
> > When I call get_foo() at the Python interpreter I get the value 0.001,
> but
> > trying to change foovar's value with set_foo() has no effect. I also
> tried
> > to cpdef foovar but that didn't help. I could probably (?) define foovar
> as
> > a regular Python variable, but I'd rather have it as a Cython variable
> for
> > speed. Is there anything special that needs to be done to get this to
> work?
> > Thanks,
> > J.S.
> > _______________________________________________
> > Cython-dev mailing list
> > Cython-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/cython-dev
> >
> >
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090518/047c1135/attachment.htm 

From robertwb at math.washington.edu  Mon May 18 21:22:56 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 18 May 2009 12:22:56 -0700
Subject: [Cython] Cython 0.11.2 release candidate
In-Reply-To: <8222a0ef1408ac1532b5be32ea838103.squirrel@webmail.uio.no>
References: <8222a0ef1408ac1532b5be32ea838103.squirrel@webmail.uio.no>
Message-ID: <2B2B7F5C-390D-4568-AB34-79DB11ED3C15@math.washington.edu>

Thanks. (Haven't tested Sage with it again, but it doesn't look like  
much has changed.)

On May 17, 2009, at 6:49 AM, Dag Sverre Seljebotn wrote:

> A "release candidate" is here:
>
> http://sage.math.washington.edu/home/dagss/Cython-0.11.2.rc1.tar.gz
> http://sage.math.washington.edu/home/dagss/Cython-0.11.2.rc1.zip
>
> I already know about two smallish issues [1], but I won't have more  
> time
> for this until Wednesday and so I thought it was better to get this  
> out
> sooner rather than later.
>
> This marks the final feature freeze. Consider cython-devel open for
> general development again; any changes to 0.11.2 should be attached as
> patches in Trac and I'll merge them on my local branch.
>
> [1] http://trac.cython.org/cython_trac/report/3
>
> Dag Sverre
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From robertwb at math.washington.edu  Mon May 18 23:02:07 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 18 May 2009 14:02:07 -0700
Subject: [Cython] interfacing with OpenCV?
In-Reply-To: <7f014ea60905171323j5fe490bfm8ba57de2352eea07@mail.gmail.com>
References: <7f014ea60905171323j5fe490bfm8ba57de2352eea07@mail.gmail.com>
Message-ID: <2E5D32D9-D400-4434-8A42-9A804172C257@math.washington.edu>

On May 17, 2009, at 1:23 PM, Chris Colbert wrote:

> How would I go about interfacing with OpenCV data types in cython?
>
> I've read the wiki on interfacing with external C code but didn't  
> fully follow.
>
> I'm currently using a ctypes wrapper for OpenCV from python and  
> doing some manipulation there, but I would like to chain together a  
> bunch of functions to run outside of the GIL on another core.
>
> From ctypes, I can pass a function a pointer to my IPLimage (the  
> opencv data).
>
> So I would imagine it would be something along the lines of:
>
> - write my functions in a pure C source file
> - declare extern functions in cython for the functions in the C source
> - compile the Cython file
> - compil the C source and compiled cython into the same pyd file
>
> The program flow would be like:
> - pass the pointer to the image to the cython function
> - forward that pointer to another function that executes with nogil  
> which calls the functions from the C library
> - return the pointer to the modified image to calling function
> - return that same pointer back to python.
>
>
> Is that about right? Or am I way off?

You can probably skip the "write my functions in a pure C source  
file" step and write them directly in Cython.

- Robert


From sccolbert at gmail.com  Tue May 19 00:21:31 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Mon, 18 May 2009 18:21:31 -0400
Subject: [Cython] interfacing with OpenCV?
In-Reply-To: <2E5D32D9-D400-4434-8A42-9A804172C257@math.washington.edu>
References: <7f014ea60905171323j5fe490bfm8ba57de2352eea07@mail.gmail.com>
	<2E5D32D9-D400-4434-8A42-9A804172C257@math.washington.edu>
Message-ID: <7f014ea60905181521m3495a68fle0fa38c5441d4b8c@mail.gmail.com>

"You can probably skip the "write my functions in a pure C source
file" step and write them directly in Cython.

- Robert"



i tried doing that today and was running into boat loads of trouble
resolving the IplImage data type and header dependencies.

This is actually turning out to be quite difficult.

The C function I am testing is this:


#include "highgui.h"

int test(char* file){
    IplImage* img = cvLoadImage(file, 1);
    cvNamedWindow("Test", CV_WINDOW_AUTOSIZE);
    cvShowImage("Test", img);
    cvWaitKey(0);
    cvReleaseImage(&img);
    cvDestroyWindow("Test");
}

my equivalent (and horribly wrong) cython implementation is this:

cdef extern from "cxtypes.h":
    ctypedef struct IplImage:
        pass

cdef extern from "highgui.h":
    cdef IplImage* cvLoadImage(char* file, int iscolor)
    cdef int cvNamedWindow(char* name, int flag)
    cdef void cvShowImage(char* name, IplImage* img)
    cdef int cvWaitKey(int delay)

cdef int test(char* file):
    cdef IplImage* img = cvLoadImage(file, 1)
    cvNamedWindow("Test", 1)
    cvShowImage("Test", img)
    cvWaitKey(0)
    return 0

def myinterface(char* file):
    test(file)


and i'm compiling with this command on mingw:
gcc -shared -fPIC -fwrapv -O2 -fno-strict-aliasing -IC:\Python25\include
-IC:\OpenCV\cxcore\include -IC:\OpenCV\otherlibs\highgui -LC:\OpenCV\lib -o
opencvcython.pyd opencvcython.c -lhighgui -lcxcore


and i get all sorts of undefined _pyobject references.


the main reason i'm doing all of this is so i can chain together a bunch of
calls that can run outside of the gil. I don't mind writing the functions in
C if i have to. In fact, i compiled the intial C code into a dll and called
it from ctypes without trouble.

any ideas to make this not-broke?

Thanks for the help!

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090518/770835c4/attachment.htm 

From robertwb at math.washington.edu  Tue May 19 01:02:57 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 18 May 2009 16:02:57 -0700
Subject: [Cython] interfacing with OpenCV?
In-Reply-To: <7f014ea60905181521m3495a68fle0fa38c5441d4b8c@mail.gmail.com>
References: <7f014ea60905171323j5fe490bfm8ba57de2352eea07@mail.gmail.com>
	<2E5D32D9-D400-4434-8A42-9A804172C257@math.washington.edu>
	<7f014ea60905181521m3495a68fle0fa38c5441d4b8c@mail.gmail.com>
Message-ID: 

On May 18, 2009, at 3:21 PM, Chris Colbert wrote:

> "You can probably skip the "write my functions in a pure C source
> file" step and write them directly in Cython.
>
> - Robert"
>
>
>
> i tried doing that today and was running into boat loads of trouble  
> resolving the IplImage data type and header dependencies.
>
> This is actually turning out to be quite difficult.
>
> The C function I am testing is this:

[...]

>
> and i'm compiling with this command on mingw:
> gcc -shared -fPIC -fwrapv -O2 -fno-strict-aliasing -IC:\Python25 
> \include -IC:\OpenCV\cxcore\include -IC:\OpenCV\otherlibs\highgui - 
> LC:\OpenCV\lib -o opencvcython.pyd opencvcython.c -lhighgui -lcxcore
>
>
> and i get all sorts of undefined _pyobject references.

This error seems like you're not linking to the python library. I  
would recommend using distutils to compile your files. Can you  
compile the Cython file

print "hello"

> the main reason i'm doing all of this is so i can chain together a  
> bunch of calls that can run outside of the gil. I don't mind  
> writing the functions in C if i have to. In fact, i compiled the  
> intial C code into a dll and called it from ctypes without trouble.
>
> any ideas to make this not-broke?

This is totally doable from Cython.

- Robert


From sccolbert at gmail.com  Tue May 19 01:30:45 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Mon, 18 May 2009 19:30:45 -0400
Subject: [Cython] interfacing with OpenCV?
In-Reply-To: 
References: <7f014ea60905171323j5fe490bfm8ba57de2352eea07@mail.gmail.com>
	<2E5D32D9-D400-4434-8A42-9A804172C257@math.washington.edu>
	<7f014ea60905181521m3495a68fle0fa38c5441d4b8c@mail.gmail.com>
	
Message-ID: <7f014ea60905181630i79c85213gfc2225fa6d5fda1@mail.gmail.com>

>
>
> This error seems like you're not linking to the python library. I
> would recommend using distutils to compile your files. Can you
> compile the Cython file
>
> print "hello"
>
>
> This is totally doable from Cython.
>
>
 I can compile and use cython modules that don't depend on external c
libraries with no problem.


i thought it was using the external libraries that was killing me, but you
were right, it was a python linking problem.

using the following distutils script solved my problem, and everything is
working.

Thanks Robert!

Chris

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

sourcefiles = ['opencvcython.pyx']
include_dirs = ['C:\OpenCV\cxcore\include',
                'C:\OpenCV\otherlibs\highgui',
                'C:\OpenCV\cv\include',
                'C:\OpenCV\ml\include']
library_dirs = ['C:\OpenCV\lib']
libraries = ['highgui',
        'cxcore',
        'cv',
        'cvaux',
        'cxts',
        'cvhaartraining',
        'ml']



setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = [Extension("opencvcython", sourcefiles,
                                include_dirs=include_dirs,
                                library_dirs=library_dirs,
                                libraries = libraries)])
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090518/a059fcbf/attachment.htm 

From sccolbert at gmail.com  Tue May 19 01:31:45 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Mon, 18 May 2009 19:31:45 -0400
Subject: [Cython] interfacing with OpenCV?
In-Reply-To: <7f014ea60905181630i79c85213gfc2225fa6d5fda1@mail.gmail.com>
References: <7f014ea60905171323j5fe490bfm8ba57de2352eea07@mail.gmail.com>
	<2E5D32D9-D400-4434-8A42-9A804172C257@math.washington.edu>
	<7f014ea60905181521m3495a68fle0fa38c5441d4b8c@mail.gmail.com>
	
	<7f014ea60905181630i79c85213gfc2225fa6d5fda1@mail.gmail.com>
Message-ID: <7f014ea60905181631j236b8efet550d0234922c7185@mail.gmail.com>

On Mon, May 18, 2009 at 7:30 PM, Chris Colbert  wrote:

>
>
>>
>> This error seems like you're not linking to the python library. I
>> would recommend using distutils to compile your files. Can you
>> compile the Cython file
>>
>> print "hello"
>>
>>
>> This is totally doable from Cython.
>>
>>
>  I can compile and use cython modules that don't depend on external c
> libraries with no problem.
>
>
> i thought it was using the external libraries that was killing me, but you
> were right, it was a python linking problem.
>
> using the following distutils script solved my problem, and everything is
> working.
>
> Thanks Robert!
>
> Chris
>
> from distutils.core import setup
> from distutils.extension import Extension
> from Cython.Distutils import build_ext
>
> sourcefiles = ['opencvcython.pyx']
> include_dirs = ['C:\OpenCV\cxcore\include',
>                 'C:\OpenCV\otherlibs\highgui',
>                 'C:\OpenCV\cv\include',
>                 'C:\OpenCV\ml\include']
> library_dirs = ['C:\OpenCV\lib']
> libraries = ['highgui',
>         'cxcore',
>         'cv',
>         'cvaux',
>         'cxts',
>         'cvhaartraining',
>         'ml']
>
>
>
> setup(
>     cmdclass = {'build_ext': build_ext},
>     ext_modules = [Extension("opencvcython", sourcefiles,
>                                 include_dirs=include_dirs,
>                                 library_dirs=library_dirs,
>                                 libraries = libraries)])



i also also have a distutils.cfg file in my distutils directory with the
following (to use mingw)

[build]
compiler = mingw32
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090518/bb3e3d2a/attachment-0001.htm 

From sccolbert at gmail.com  Tue May 19 01:35:01 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Mon, 18 May 2009 19:35:01 -0400
Subject: [Cython] interfacing with OpenCV?
In-Reply-To: <7f014ea60905181631j236b8efet550d0234922c7185@mail.gmail.com>
References: <7f014ea60905171323j5fe490bfm8ba57de2352eea07@mail.gmail.com>
	<2E5D32D9-D400-4434-8A42-9A804172C257@math.washington.edu>
	<7f014ea60905181521m3495a68fle0fa38c5441d4b8c@mail.gmail.com>
	
	<7f014ea60905181630i79c85213gfc2225fa6d5fda1@mail.gmail.com>
	<7f014ea60905181631j236b8efet550d0234922c7185@mail.gmail.com>
Message-ID: <7f014ea60905181635l6dfc9726q3f9360d9009e135e@mail.gmail.com>

new question:

How do i get this to release the gil:

when i change the cython module to the following Cython compilation fails
claiming I'm calling a gil required function:

cdef extern from "cxtypes.h":
    ctypedef struct IplImage:
        pass

cdef extern from "highgui.h":
    cdef IplImage* cvLoadImage(char* file, int iscolor)
    cdef int cvNamedWindow(char* name, int flag)
    cdef void cvShowImage(char* name, IplImage* img)
    cdef int cvWaitKey(int delay)

cdef int test(char* file) nogil:
    cdef IplImage* img = cvLoadImage(file, 1)
    cvNamedWindow("Test", 1)
    cvShowImage("Test", img)
    cvWaitKey(0)
    return 0

def myinterface(char* file):
    with nogil:
        test(file)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090518/3045cdd5/attachment.htm 

From hoytak at cs.ubc.ca  Tue May 19 01:47:17 2009
From: hoytak at cs.ubc.ca (Hoyt Koepke)
Date: Mon, 18 May 2009 16:47:17 -0700
Subject: [Cython] interfacing with OpenCV?
In-Reply-To: <7f014ea60905181635l6dfc9726q3f9360d9009e135e@mail.gmail.com>
References: <7f014ea60905171323j5fe490bfm8ba57de2352eea07@mail.gmail.com> 
	<2E5D32D9-D400-4434-8A42-9A804172C257@math.washington.edu> 
	<7f014ea60905181521m3495a68fle0fa38c5441d4b8c@mail.gmail.com> 
	 
	<7f014ea60905181630i79c85213gfc2225fa6d5fda1@mail.gmail.com> 
	<7f014ea60905181631j236b8efet550d0234922c7185@mail.gmail.com> 
	<7f014ea60905181635l6dfc9726q3f9360d9009e135e@mail.gmail.com>
Message-ID: <4db580fd0905181647q1195b677s4be96fa2211e2c0c@mail.gmail.com>

Hi,

> How do i get this to release the gil:
>
> when i change the cython module to the following Cython compilation fails
> claiming I'm calling a gil required function:

All functions, even external ones, that can be run without the gil
need the "nogil" decorator on the end or cython assumes it needs the
gil.  E.g.

> cdef extern from "highgui.h":
> ??? cdef IplImage* cvLoadImage(char* file, int iscolor) nogil
> ??? cdef int cvNamedWindow(char* name, int flag) nogil
...

--Hoyt

++++++++++++++++++++++++++++++++++++++++++++++++
+ Hoyt Koepke
+ University of Washington Department of Statistics
+ http://www.stat.washington.edu/~hoytak/
+ hoytak at gmail.com
++++++++++++++++++++++++++++++++++++++++++

From sccolbert at gmail.com  Tue May 19 01:55:48 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Mon, 18 May 2009 19:55:48 -0400
Subject: [Cython] interfacing with OpenCV?
In-Reply-To: <4db580fd0905181647q1195b677s4be96fa2211e2c0c@mail.gmail.com>
References: <7f014ea60905171323j5fe490bfm8ba57de2352eea07@mail.gmail.com>
	<2E5D32D9-D400-4434-8A42-9A804172C257@math.washington.edu>
	<7f014ea60905181521m3495a68fle0fa38c5441d4b8c@mail.gmail.com>
	
	<7f014ea60905181630i79c85213gfc2225fa6d5fda1@mail.gmail.com>
	<7f014ea60905181631j236b8efet550d0234922c7185@mail.gmail.com>
	<7f014ea60905181635l6dfc9726q3f9360d9009e135e@mail.gmail.com>
	<4db580fd0905181647q1195b677s4be96fa2211e2c0c@mail.gmail.com>
Message-ID: <7f014ea60905181655idd84571t4cfb0c4e8ca16931@mail.gmail.com>

Hoyt,

All functions, even external ones, that can be run without the gil
> need the "nogil" decorator on the end or cython assumes it needs the
> gil.  E.g.
>
> > cdef extern from "highgui.h":
> >     cdef IplImage* cvLoadImage(char* file, int iscolor) nogil
> >     cdef int cvNamedWindow(char* name, int flag) nogil
> ...
>

This worked beautifully.

thank you very much!

This list is very patient with the newbies. I appreciate that!

Cheers,

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090518/e50645f9/attachment.htm 

From sccolbert at gmail.com  Tue May 19 03:40:11 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Mon, 18 May 2009 21:40:11 -0400
Subject: [Cython] interfacing with OpenCV?
In-Reply-To: <7f014ea60905181655idd84571t4cfb0c4e8ca16931@mail.gmail.com>
References: <7f014ea60905171323j5fe490bfm8ba57de2352eea07@mail.gmail.com>
	<2E5D32D9-D400-4434-8A42-9A804172C257@math.washington.edu>
	<7f014ea60905181521m3495a68fle0fa38c5441d4b8c@mail.gmail.com>
	
	<7f014ea60905181630i79c85213gfc2225fa6d5fda1@mail.gmail.com>
	<7f014ea60905181631j236b8efet550d0234922c7185@mail.gmail.com>
	<7f014ea60905181635l6dfc9726q3f9360d9009e135e@mail.gmail.com>
	<4db580fd0905181647q1195b677s4be96fa2211e2c0c@mail.gmail.com>
	<7f014ea60905181655idd84571t4cfb0c4e8ca16931@mail.gmail.com>
Message-ID: <7f014ea60905181840h54c56ab3gecf072f1cd393b56@mail.gmail.com>

yet again:

for various reasons, I would like to use both the ctypes wrapper and cython
in same program. Specifically I have a ctypes instance of an OpenCV image. I
pass the pointer of this image to a cython function which returns a pointer
to a new image. I need to get this pointer back into python as a new image.
The following cython code complains that it cant convert the pointer to a
python object. but thats exactly the purpose of the POINTER() call from
ctypes (from what I gather).

Here is the code:

from ctypes import *

cdef extern from "cxtypes.h":
    ctypedef struct IplImage:
        pass

cdef extern from "cxcore.h":
    cdef IplImage* cvCloneImage(IplImage* img) nogil


cdef IplImage* clone(IplImage* img) nogil:
    cdef IplImage* out = cvCloneImage(img)
    return out


def myclone(pyimg):
    cdef IplImage* dst
    with nogil:
        dst = clone(byref(pyimg))

    pydst = POINTER(dst)
    return pydst


any ideas?

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090518/17e13491/attachment.htm 

From dalcinl at gmail.com  Tue May 19 05:20:56 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 19 May 2009 00:20:56 -0300
Subject: [Cython] interfacing with OpenCV?
In-Reply-To: <7f014ea60905181840h54c56ab3gecf072f1cd393b56@mail.gmail.com>
References: <7f014ea60905171323j5fe490bfm8ba57de2352eea07@mail.gmail.com>
	<2E5D32D9-D400-4434-8A42-9A804172C257@math.washington.edu>
	<7f014ea60905181521m3495a68fle0fa38c5441d4b8c@mail.gmail.com>
	
	<7f014ea60905181630i79c85213gfc2225fa6d5fda1@mail.gmail.com>
	<7f014ea60905181631j236b8efet550d0234922c7185@mail.gmail.com>
	<7f014ea60905181635l6dfc9726q3f9360d9009e135e@mail.gmail.com>
	<4db580fd0905181647q1195b677s4be96fa2211e2c0c@mail.gmail.com>
	<7f014ea60905181655idd84571t4cfb0c4e8ca16931@mail.gmail.com>
	<7f014ea60905181840h54c56ab3gecf072f1cd393b56@mail.gmail.com>
Message-ID: 

Likely (never used ctypes) you will need to call "ctypes.addressof",
and a call to PyLong_AsVoidPtr(). See yourself:

http://svn.python.org/view/python/trunk/Modules/_ctypes/callproc.c?view=markup

http://docs.python.org/c-api/long.html#PyLong_AsVoidPtr

cdef extern from "Python.h"
    void *PyLong_AsVoidPtr(object)

cdef object iptr = ctypes.addressoff(pyimg) # assuming pyimg was a
ctypes instance
cdef IplImage *dst = PyLong_AsVoidPtr(iptr)
with nogil:
     dst = clone(dst)

Disclaimer: NEVER used ctypes myself... But the above code should be near...

In general, you will have to be very careful if you want to mix ctypes
and Cython. They are different beast ... Once you get used to code
Cython, and assuming you have some C/C++ background, I bet you will
stop using ctypes ;-) .. well, except perhaps for the purposes of
supporting your own legacy code ...



On Mon, May 18, 2009 at 10:40 PM, Chris Colbert  wrote:
> yet again:
>
> for various reasons, I would like to use both the ctypes wrapper and cython
> in same program. Specifically I have a ctypes instance of an OpenCV image. I
> pass the pointer of this image to a cython function which returns a pointer
> to a new image. I need to get this pointer back into python as a new image.
> The following cython code complains that it cant convert the pointer to a
> python object. but thats exactly the purpose of the POINTER() call from
> ctypes (from what I gather).
>
> Here is the code:
>
> from ctypes import *
>
> cdef extern from "cxtypes.h":
> ??? ctypedef struct IplImage:
> ??????? pass
>
> cdef extern from "cxcore.h":
> ??? cdef IplImage* cvCloneImage(IplImage* img) nogil
>
>
> cdef IplImage* clone(IplImage* img) nogil:
> ??? cdef IplImage* out = cvCloneImage(img)
> ??? return out
>
>
> def myclone(pyimg):
> ??? cdef IplImage* dst
> ??? with nogil:
> ??????? dst = clone(byref(pyimg))
>
> ??? pydst = POINTER(dst)
> ??? return pydst
>
>
> any ideas?
>
> Chris
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Tue May 19 05:40:48 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 19 May 2009 00:40:48 -0300
Subject: [Cython] temp allocation and nogil blocks
Message-ID: 

Do any of you know how to make Cython complain about the code below?
Found some time ago, forgot to report:
http://code.google.com/p/mpi4py/issues/detail?id=2


cdef void foo(int i) nogil:
    pass

def bar(arg):
    with nogil:
        foo(arg)


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Tue May 19 07:39:40 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 19 May 2009 07:39:40 +0200 (CEST)
Subject: [Cython] temp allocation and nogil blocks
In-Reply-To: 
References: 
Message-ID: 

Lisandro wrote:
> Do any of you know how to make Cython complain about the code below?
> Found some time ago, forgot to report:
> http://code.google.com/p/mpi4py/issues/detail?id=2
>
>
> cdef void foo(int i) nogil:
>     pass
>
> def bar(arg):
>     with nogil:
>         foo(arg)
>

It might be that I should reopen

http://trac.cython.org/cython_trac/ticket/273

if so, I know exactly how it should be done.

However the temporary above is an int, so it is probably not the right
thing here...

Perhaps have a look at gil_check in NameNode?

Dag Sverre


From stefan_ml at behnel.de  Tue May 19 07:57:21 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 19 May 2009 07:57:21 +0200
Subject: [Cython] temp allocation and nogil blocks
In-Reply-To: 
References: 
	
Message-ID: <4A124A41.6060909@behnel.de>


Dag Sverre Seljebotn wrote:
> Lisandro wrote:
>> Do any of you know how to make Cython complain about the code below?
>> Found some time ago, forgot to report:
>> http://code.google.com/p/mpi4py/issues/detail?id=2
>>
>>
>> cdef void foo(int i) nogil:
>>     pass
>>
>> def bar(arg):
>>     with nogil:
>>         foo(arg)
>>
> 
> It might be that I should reopen
> 
> http://trac.cython.org/cython_trac/ticket/273
> 
> if so, I know exactly how it should be done.
> 
> However the temporary above is an int, so it is probably not the right
> thing here...
> 
> Perhaps have a look at gil_check in NameNode?

... and if someone does, please do it in cython-unstable, which has
refactorings for both temps and nogil-checks. Fixing cython-devel here is
futile.

Stefan

From dagss at student.matnat.uio.no  Tue May 19 08:14:20 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 19 May 2009 08:14:20 +0200 (CEST)
Subject: [Cython] Tutorial at NOTUR 2009
Message-ID: 

I prepared some slides etc. for a full-day tutorial at NOTUR 2009
(http://www.notur.no/notur2009/) yesterday. You can download it from here:

http://wiki.cython.org/talks/notur2009

Slides only:
http://sage.math.washington.edu/home/dagss/cython-notur09/notur2009.pdf

(Some of that is Robert's material BTW.)

Dag Sverre



From dagss at student.matnat.uio.no  Tue May 19 08:29:00 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 19 May 2009 08:29:00 +0200 (CEST)
Subject: [Cython] pyximport
In-Reply-To: 
References: 
	
	
	
	 
	
	<4A0F0942.3060307@behnel.de>
	<311bf9168cc7ea937556a7391b54cdf1.squirrel@webmail.uio.no>
	
	
	 
Message-ID: <186d467d97205ba211a488d0a1c1b7f0.squirrel@webmail.uio.no>

> Robert wrote:
>
> I've done a few more tests on Linux and Win, and it does so far on
> my side in all modes - for practical edit-run/reload debugging
> purposes. The reload_support=False is anyway default.
>
> the little 3+.patch in attachment in addition to pyxi3.patch
> http://trac.cython.org/cython_trac/ticket/312
> (or manually against 0.11.1)
> finally would solve the mentioned "reload(my) -> ImportError: No
> module named my" problem of 0.11.1  or of reload_support=False
> default mode
>
> I leave it open so far from my side.

Thanks. I wanted to discuss some things a bit more, which is why it didn?t
make it.

The thing is that when a new format is introduced (pyxbld in this case),
we have to make sure it is the right one because once it's there we must
be backwards compatible.

In my opinion, pyximport is for the "simple cases" where you do not really
want a build system. The pyxbld file however is verbose enough that one
might perhaps as well write a setup.py...or have pyximport invoke a
setup.py...

One thing I was thinking about is embedding the information in the pyx
file itself. Something like this perhaps:

from cython import pyxbuild
pyxbuild.include_path = [...]
pyxbuild.libraries = [...]

.. rest of pyx file...

What do you think?

Dag Sverre


From brett.calcott at gmail.com  Tue May 19 10:16:13 2009
From: brett.calcott at gmail.com (Brett Calcott)
Date: Tue, 19 May 2009 18:16:13 +1000
Subject: [Cython] Incorrect error message
Message-ID: 

Hi,

This following code gives an very misleading error message. It says
there is something wrong with the definition of another_array (than
ndarray is not a type identifier). If you comment out the definition
of another_array, then it gives the correct (I assume) error message:
that Buffer types can only be local.

At least, I think that is the case.

Cheers,
Brett

import numpy as np
cimport numpy as np
cimport cython

cdef class Boodle:
    cdef np.ndarray[double, ndim=1] an_array
    def __cinit__(self):
        cdef np.ndarray[double, ndim=1, mode="c"] \
                another_array = np.zeros(self.E, dtype=np.float64)

From kxroberto at googlemail.com  Tue May 19 13:44:00 2009
From: kxroberto at googlemail.com (Robert)
Date: Tue, 19 May 2009 13:44:00 +0200
Subject: [Cython] pyximport
In-Reply-To: <186d467d97205ba211a488d0a1c1b7f0.squirrel@webmail.uio.no>
References: 				
			<4A0F0942.3060307@behnel.de>	<311bf9168cc7ea937556a7391b54cdf1.squirrel@webmail.uio.no>			
	
	<186d467d97205ba211a488d0a1c1b7f0.squirrel@webmail.uio.no>
Message-ID: 

Dag Sverre Seljebotn wrote:
>> Robert wrote:
>>
>> I've done a few more tests on Linux and Win, and it does so far on
>> my side in all modes - for practical edit-run/reload debugging
>> purposes. The reload_support=False is anyway default.
>>
>> the little 3+.patch in attachment in addition to pyxi3.patch
>> http://trac.cython.org/cython_trac/ticket/312
>> (or manually against 0.11.1)
>> finally would solve the mentioned "reload(my) -> ImportError: No
>> module named my" problem of 0.11.1  or of reload_support=False
>> default mode
>>
>> I leave it open so far from my side.
> 
> Thanks. I wanted to discuss some things a bit more, which is why it didn?t
> make it.
> 
> The thing is that when a new format is introduced (pyxbld in this case),
> we have to make sure it is the right one because once it's there we must
> be backwards compatible.

.pyxbld/.pyxdep mechanism was already in 0.11.1 and long before 
probably and allowed already to define a specific Extension class 
(to tell about other .c files, dependencies etc).

I just added the setup_args, to allow crucial things like --debug, 
--inplace, --compiler and all ... which one might have special in 
pyximport mode too: its typically a debug situation.

The mode of pyximport compilation is often anyway different from 
the regular setup.py script for the final project. And pyximport 
uses its own temp and build files/folder by default (correctly).

> 
> In my opinion, pyximport is for the "simple cases" where you do not really

still it must allow to define the basic dependencies, and other .c 
files ...
=>  the info for a setup() build_ext call

> want a build system. The pyxbld file however is verbose enough that one
> might perhaps as well write a setup.py...or have pyximport invoke a
> setup.py...

indeed the .pyxbld is a kind of setup.py, related to the .pyx

Just: its stuff is imported (and (pre)compiled!) and is not run as 
script. fast, no new process/time delay for mere dependency checks 
unless a compiler run...

at the moment of import, just a module name exist, modname.pyx is 
found => associate .pyxbld.

Or info must be parsed from source code ..

> 
> One thing I was thinking about is embedding the information in the pyx
> file itself. Something like this perhaps:
> 
> from cython import pyxbuild
> pyxbuild.include_path = [...]
> pyxbuild.libraries = [...]
> 
> .. rest of pyx file...
> 
> What do you think?


If the pyx meta importer must parse the .pyx source code (before 
he can even check the dependencies), it is very slow.

Also including such special build info into the source file is a 
questionable intermix.
Consider, you give the source file to somebody else with different 
environment ...

so I'd say, a separate, fast, imported and compiled .pyxbld(c) 
next to the .pyx, as it was, is primarily the right natural thing.

Questioning its format:

It could be more *like* a setup.py. But it cannot not just be a 
normal free one.
Because: pyximport must inject/weave in the pyximport.install() 
args, the .pyxdep,  must be able to tell separate default paths 
(to not necessarily affect the regular build system). it must 
direct/know location/take over the produced files; reload_support 
mode, etc.

Thus pre-defining a single setup() call more or less (but not 
executing directly) is perhaps the "simple case", the task of the 
.pyxbld.

so I think a make_extension and make_setup_args inside the .pyxbld 
is principally quite the right thing. sb who knows to write a 
setup.py/setup(), will face quite the same format here - just 
canalized.

Also, because its imported (and makeXX funktions are called), one 
can still let execute *any* python code (before the build as of 
now only). So its very flexible.

The current hierarchy of usage (and quite the evolution history):
1) pyximport.install() args - general behavior
2) .pyxdep - simple, only dependencies
3) .pyxbld/make_extension - other .c files...; can
     extend/override/replace (2) and (1)
4) NEW .pyxbld/make_setup_args - alter higher options; can
     extend/override/replace (3),(2),(1)
5) arbitrary effective python code in .pyxbld globally or
    in the makeXX functions

I do not really have a better principal idea, and the mechanism is 
flexibile and can be further evolved (e.g. a function called back 
at compile/post build time or so). details could be straightened out.


If one is concerned about not doubling build/dependency info:
The .pyxbld file can be used/included right *from* a setup.py 
script because of the compatibility, like:

#setup-cyex.py --------------------------------------


from distutils.core import setup
from Cython.Distutils import build_ext
execfile("cyex.pyxbld")

setup(
     cmdclass = {'build_ext': build_ext},
     ext_modules = [make_ext('cyex','cyex.pyx')],
     ##**make_setup_args()
)


#cyex.pyxbld ----------------------------------------
def make_ext(modname, pyxfilename):
     from distutils.extension import Extension
     return Extension(name = modname,
                      sources=[pyxfilename, 'hello.c'],
                      depends=['hello.h'],
                       )

def make_setup_args():
     # args like for distutils Distribution/setup()
     return dict(
         script_args=[
             ##"--compiler=mingw32",
             ##"--verbose",
             "--inplace",
             "--debug",
     ])
-------------------------------------------------------


thats maybe even natural to have the specific build/dep info for 
each .pyx right next in the .pyxbld .  A (quite constant) setup.py 
for the general distutils project integration...


Robert








From tenninosko at msn.com  Tue May 19 14:01:37 2009
From: tenninosko at msn.com (OUARDA MEHDI)
Date: Tue, 19 May 2009 14:01:37 +0200
Subject: [Cython] help...cython code optimisation
Message-ID: 


hi people,
her is my issue: I'm working on optimization of a simulator  written in python and using numpy a lot.as i'm a new user of cython, I'm asking help about defining functions like cpdef foo(..)  and using at the same time  @proprety decorator.i got this msg when compiling the .pyx file:
Error converting Pyrex file to C:
------------------------------------------------------------
...

    @property
    def pose(self):
        return homogeneousmatrix.rotzyx(self.gpos)

    @property
   ^
------------------------------------------------------------
/home/tenninos/stage/arboris-python/src/joints_c.pyx:109:4: Cdef functions cannot take arbitrary decorators.

and here is both my python and cython code.i'd be really grateful  if you could help me  further in my optimisation :

from numpy import array, zeros, eye, dot, sin, cos, dot
from abc import ABCMeta, abstractmethod
import homogeneousmatrix
from misc import NamedObject
from arboris import Joint
class RzRyRxJoint(LinearConfigurationSpaceJoint):
    """Ball and socket (3-dof) implemented with 3 serial hinges

    the resulting homogeneous matrix is given by H = Rz*Ry*Rx
    """
    def __init__(self, gpos=[0.,0.,0.], gvel=[0.,0.,0.], name=None):
        self.gpos = array(gpos).reshape((3))
        self.gvel = array(gvel).reshape((3))
        Joint.__init__(self, name)

    @property
    def ndof(self):
        return 3

    @property
    def pose(self):
        return homogeneousmatrix.rotzyx(self.gpos)

    @property
    def jacobian(self):
        """
        T_n/r = 
        """   
        sx = sin(self.gpos[0])
        cx = cos(self.gpos[0])
        sy = sin(self.gpos[1])
        cy = cos(self.gpos[1])
        return array(
            [[ 1.   ,  0. , -sy    ],
             [ 0.   , cx  ,  sx*sy ],
             [ 0.   ,-sx  ,  cx*cy ],
             [ 0.   ,  0. ,  0.    ],
             [ 0.   ,  0. ,  0.    ],
             [ 0.   ,  0. ,  0.    ]])

cython:....

from numpy import array, zeros, eye, dot, sin, cos, dot
from abc import ABCMeta, abstractmethod
import homogeneousmatrix
from misc import NamedObject
from arboris import Joint

#mes modifs....
cimport numpy as np
DTYPE = np.float 
ctypedef np.float_t DTYPE_t
cdef extern from "math.h":
    float cosf(float theta)
    float sinf(float theta)

cdef class RzRyRxJoint(LinearConfigurationSpaceJoint):
    """Ball and socket (3-dof) implemented with 3 serial hinges

    the resulting homogeneous matrix is given by H = Rz*Ry*Rx
    """
    def __init__(self, gpos=[0.,0.,0.], gvel=[0.,0.,0.], name=None):
        self.gpos = array(gpos).reshape((3))
        self.gvel = array(gvel).reshape((3))
        Joint.__init__(self, name)

    @property
    def ndof(self):
        return 3

    @property
    def pose(self):
        return homogeneousmatrix.rotzyx(self.gpos)

    @property
    cpdef jacobian(self):           #mes modifs.......
        """
        T_n/r = 
        """   
        cdef float sx = sinf(self.gpos[0])
        cdef float cx = cosf(self.gpos[0])
        cdef float sy = sinf(self.gpos[1])
        cdef float cy = cosf(self.gpos[1])
        cdef np.ndarray[DTYPE_t, ndim=2] value = np.array (
            ([ 1.   ,  0. , -sy    ],
             [ 0.   , cx  ,  sx*sy ],
             [ 0.   ,-sx  ,  cx*cy ],
             [ 0.   ,  0. ,  0.    ],
             [ 0.   ,  0. ,  0.    ],
             [ 0.   ,  0. ,  0.    ]),   dtype=DTYPE)
        return value

_________________________________________________________________
D?couvrez toutes les possibilit?s de communication avec vos proches
http://www.microsoft.com/windows/windowslive/default.aspx
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090519/4dfb86a6/attachment.htm 

From kwmsmith at gmail.com  Tue May 19 15:22:33 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Tue, 19 May 2009 08:22:33 -0500
Subject: [Cython] help...cython code optimisation
In-Reply-To: 
References: 
Message-ID: 

On Tue, May 19, 2009 at 7:01 AM, OUARDA MEHDI  wrote:
> hi people,
> her is my issue: I'm working on optimization of a simulator? written in
> python and using numpy a lot.as i'm a new user of cython, I'm asking help
> about defining functions like cpdef foo(..)? and using at the same time
> @proprety decorator.i got this msg when compiling the .pyx file:

[snip]

>
> cython:....
>
> from numpy import array, zeros, eye, dot, sin, cos, dot
> from abc import ABCMeta, abstractmethod
> import homogeneousmatrix
> from misc import NamedObject
> from arboris import Joint
>
> #mes modifs....
> cimport numpy as np
> DTYPE = np.float
> ctypedef np.float_t DTYPE_t
> cdef extern from "math.h":
> ??? float cosf(float theta)
> ??? float sinf(float theta)
>
> cdef class RzRyRxJoint(LinearConfigurationSpaceJoint):
> ??? """Ball and socket (3-dof) implemented with 3 serial hinges
>
> ??? the resulting homogeneous matrix is given by H = Rz*Ry*Rx
> ??? """
> ??? def __init__(self, gpos=[0.,0.,0.], gvel=[0.,0.,0.], name=None):
> ??????? self.gpos = array(gpos).reshape((3))
> ??????? self.gvel = array(gvel).reshape((3))
> ??????? Joint.__init__(self, name)
>
> ??? @property
> ??? def ndof(self):
> ??????? return 3
>
> ??? @property
> ??? def pose(self):
> ??????? return homogeneousmatrix.rotzyx(self.gpos)
>
> ??? @property
> ??? cpdef jacobian(self):?????????? #mes modifs.......
> ??????? """
> ??????? T_n/r =
> ??????? """
> ??????? cdef float sx = sinf(self.gpos[0])
> ??????? cdef float cx = cosf(self.gpos[0])
> ??????? cdef float sy = sinf(self.gpos[1])
> ??????? cdef float cy = cosf(self.gpos[1])
> ??????? cdef np.ndarray[DTYPE_t, ndim=2] value = np.array (
> ??????????? ([ 1.?? ,? 0. , -sy??? ],
> ???????????? [ 0.?? , cx? ,? sx*sy ],
> ???????????? [ 0.?? ,-sx? ,? cx*cy ],
> ???????????? [ 0.?? ,? 0. ,? 0.??? ],
> ???????????? [ 0.?? ,? 0. ,? 0.??? ],
> ???????????? [ 0.?? ,? 0. ,? 0.??? ]),?? dtype=DTYPE)
> ??????? return value
>
>

>From the docs:

http://docs.cython.org/docs/extension_types.html#properties

There's a special syntax to define properties for extension types.

Instead of

---------------------------------------------------------
@property
def ndof(self):
    return 3
---------------------------------------------------------
You'd do:

---------------------------------------------------------
property ndof:
    def __get__(self):
        return 3
----------------------------------------------------------

Kurt

From stefan_ml at behnel.de  Tue May 19 15:43:46 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 19 May 2009 15:43:46 +0200 (CEST)
Subject: [Cython] help...cython code optimisation
In-Reply-To: 
References: 
	
Message-ID: <6f7693ed86620a72ebc66a9db234c2c4.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

Kurt Smith wrote:
> On Tue, May 19, 2009 at 7:01 AM, OUARDA MEHDI wrote:
> [snip]
>>
>> cdef class RzRyRxJoint(LinearConfigurationSpaceJoint):
>>
>> ??? @property
>> ??? cpdef jacobian(self):?????????? #mes modifs.......
>>       ...
>
> There's a special syntax to define properties for extension types.
>
> Instead of
>
> ---------------------------------------------------------
> @property
> def ndof(self):
>     return 3
> ---------------------------------------------------------
> You'd do:
>
> ---------------------------------------------------------
> property ndof:
>     def __get__(self):
>         return 3
> ----------------------------------------------------------

While this is true, it doesn't really solve the problem that the OP wanted
to use a property for a cpdef method. The problem here is that 'c(p)def'
methods are C functions and you cannot wrap them in a property. What you
can do, however, is to move the property code into a cdef method (not
cpdef) and use that in your own code, and then additionally declare a
property as Kurt described above, which simply calls the cdef method. That
way, you get fast access from your Cython code and property access from
Python code.

I'm not even sure if it's technically possible to 'unroll' properties in
Cython, so that you'd get the equivalent of a cpdef method for properties.
There may be some tricky inheritance issues involved. At least, you can't
just call into the implementing descriptor methods.

Stefan


From sccolbert at gmail.com  Tue May 19 17:29:17 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Tue, 19 May 2009 11:29:17 -0400
Subject: [Cython] fundamentals question
Message-ID: <7f014ea60905190829w12fc8af2la0e646f39115c8e6@mail.gmail.com>

I think i'm starting to wrap my head around all of this, but I have a
question about expose a C data type to python.

I'm interfacing with the OpenCV library whose fundamental data structure is
an IplImage. Most of the functional code passes around pointers to these
types and can do this in cython with no problem. I hit the wall when I try
to get my final result back into python. I have been trying to just pass the
pointer back which obviously wont work. Now, I am thinking the right way to
go about it is to write an extension type that interfacing with the
structure on the C level.

i.e. something like:

cdef class IplImage:
      cdef IplImage* img

      def __init__(
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090519/5acefecf/attachment.htm 

From sccolbert at gmail.com  Tue May 19 17:30:40 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Tue, 19 May 2009 11:30:40 -0400
Subject: [Cython] fundamentals question
In-Reply-To: <7f014ea60905190829w12fc8af2la0e646f39115c8e6@mail.gmail.com>
References: <7f014ea60905190829w12fc8af2la0e646f39115c8e6@mail.gmail.com>
Message-ID: <7f014ea60905190830k296daa2ekb7ded15d93ded6e2@mail.gmail.com>

this message got sent before it was finished.

At any rate, am I on the right track here? I'll need a way, at the very
least to expose the data buffer of the image to python so I can display on
the screen via wxPython.

Chris

On Tue, May 19, 2009 at 11:29 AM, Chris Colbert  wrote:

> I think i'm starting to wrap my head around all of this, but I have a
> question about expose a C data type to python.
>
> I'm interfacing with the OpenCV library whose fundamental data structure is
> an IplImage. Most of the functional code passes around pointers to these
> types and can do this in cython with no problem. I hit the wall when I try
> to get my final result back into python. I have been trying to just pass the
> pointer back which obviously wont work. Now, I am thinking the right way to
> go about it is to write an extension type that interfacing with the
> structure on the C level.
>
> i.e. something like:
>
> cdef class IplImage:
>       cdef IplImage* img
>
>       def __init__(
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090519/ca1def4a/attachment.htm 

From dalcinl at gmail.com  Tue May 19 17:49:32 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 19 May 2009 12:49:32 -0300
Subject: [Cython] fundamentals question
In-Reply-To: <7f014ea60905190830k296daa2ekb7ded15d93ded6e2@mail.gmail.com>
References: <7f014ea60905190829w12fc8af2la0e646f39115c8e6@mail.gmail.com>
	<7f014ea60905190830k296daa2ekb7ded15d93ded6e2@mail.gmail.com>
Message-ID: 

On Tue, May 19, 2009 at 12:30 PM, Chris Colbert  wrote:
> this message got sent before it was finished.
>
> At any rate, am I on the right track here? I'll need a way, at the very
> least to expose the data buffer of the image to python so I can display on
> the screen via wxPython.
>

IMHO, that's definitely the right track. However, take into account
name clashing (between your cdef class and the C handles).


> Chris
>
> On Tue, May 19, 2009 at 11:29 AM, Chris Colbert  wrote:
>>
>> I think i'm starting to wrap my head around all of this, but I have a
>> question about expose a C data type to python.
>>
>> I'm interfacing with the OpenCV library whose fundamental data structure
>> is an IplImage. Most of the functional code passes around pointers to these
>> types and can do this in cython with no problem. I hit the wall when I try
>> to get my final result back into python. I have been trying to just pass the
>> pointer back which obviously wont work. Now, I am thinking the right way to
>> go about it is to write an extension type that interfacing with the
>> structure on the C level.
>>
>> i.e. something like:
>>
>> cdef class IplImage:
>> ????? cdef IplImage* img
>>
>> ????? def __init__(
>>
>>
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From sccolbert at gmail.com  Tue May 19 17:54:16 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Tue, 19 May 2009 11:54:16 -0400
Subject: [Cython] fundamentals question
In-Reply-To: 
References: <7f014ea60905190829w12fc8af2la0e646f39115c8e6@mail.gmail.com>
	<7f014ea60905190830k296daa2ekb7ded15d93ded6e2@mail.gmail.com>
	
Message-ID: <7f014ea60905190854x2ef75b22s12e5cb0d0bebf9e9@mail.gmail.com>

Do I need to worry about python garbage collecting what I reference through
my IplImage*. i.e. when i create my images on the c-level, I alone am
responsible for freeing that space when no longer needed correct?


Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090519/05a55b9d/attachment.htm 

From dalcinl at gmail.com  Tue May 19 18:08:03 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 19 May 2009 13:08:03 -0300
Subject: [Cython] Tutorial at NOTUR 2009
In-Reply-To: 
References: 
Message-ID: 

Dag, could you check permission on files inside
http://sage.math.washington.edu/home/dagss/cython-notur09/mpi/ ? I'm
getting "Forbidden" here...

On Tue, May 19, 2009 at 3:14 AM, Dag Sverre Seljebotn
 wrote:
> I prepared some slides etc. for a full-day tutorial at NOTUR 2009
> (http://www.notur.no/notur2009/) yesterday. You can download it from here:
>
> http://wiki.cython.org/talks/notur2009
>
> Slides only:
> http://sage.math.washington.edu/home/dagss/cython-notur09/notur2009.pdf
>
> (Some of that is Robert's material BTW.)
>
> Dag Sverre
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Tue May 19 18:12:32 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 19 May 2009 13:12:32 -0300
Subject: [Cython] fundamentals question
In-Reply-To: <7f014ea60905190854x2ef75b22s12e5cb0d0bebf9e9@mail.gmail.com>
References: <7f014ea60905190829w12fc8af2la0e646f39115c8e6@mail.gmail.com>
	<7f014ea60905190830k296daa2ekb7ded15d93ded6e2@mail.gmail.com>
	
	<7f014ea60905190854x2ef75b22s12e5cb0d0bebf9e9@mail.gmail.com>
Message-ID: 

On Tue, May 19, 2009 at 12:54 PM, Chris Colbert  wrote:
> Do I need to worry about python garbage collecting what I reference through
> my IplImage*. i.e. when i create my images on the c-level, I alone am
> responsible for freeing that space when no longer needed correct?
>

Well, it depens on what you want to achive... If different Python
objects are going to proxy the same IplImage* handle, then you will
have to manage memory yourself being carefull about making leaks or
double deallocations. Or you could perhaps implement your own
reference count scheme for the C data.

>
>
> Chris
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From sccolbert at gmail.com  Tue May 19 18:20:11 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Tue, 19 May 2009 12:20:11 -0400
Subject: [Cython] fundamentals question
In-Reply-To: 
References: <7f014ea60905190829w12fc8af2la0e646f39115c8e6@mail.gmail.com>
	<7f014ea60905190830k296daa2ekb7ded15d93ded6e2@mail.gmail.com>
	
	<7f014ea60905190854x2ef75b22s12e5cb0d0bebf9e9@mail.gmail.com>
	
Message-ID: <7f014ea60905190920p127b5589j5cd3380c443b4cbc@mail.gmail.com>

I guess what i'm asking, if I have a single python object reference the
IplImage* handle, and I delete the python object, the IplImage still exists,
so I would need to explicitly deallocate the image in the __dealloc__
special method, correct?


Thanks!

Chris

On Tue, May 19, 2009 at 12:12 PM, Lisandro Dalcin  wrote:

> On Tue, May 19, 2009 at 12:54 PM, Chris Colbert 
> wrote:
> > Do I need to worry about python garbage collecting what I reference
> through
> > my IplImage*. i.e. when i create my images on the c-level, I alone am
> > responsible for freeing that space when no longer needed correct?
> >
>
> Well, it depens on what you want to achive... If different Python
> objects are going to proxy the same IplImage* handle, then you will
> have to manage memory yourself being carefull about making leaks or
> double deallocations. Or you could perhaps implement your own
> reference count scheme for the C data.
>
> >
> >
> > Chris
> >
> > _______________________________________________
> > Cython-dev mailing list
> > Cython-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/cython-dev
> >
> >
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090519/197641af/attachment.htm 

From stefan_ml at behnel.de  Tue May 19 19:23:12 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 19 May 2009 19:23:12 +0200
Subject: [Cython] fundamentals question
In-Reply-To: <7f014ea60905190920p127b5589j5cd3380c443b4cbc@mail.gmail.com>
References: <7f014ea60905190829w12fc8af2la0e646f39115c8e6@mail.gmail.com>	<7f014ea60905190830k296daa2ekb7ded15d93ded6e2@mail.gmail.com>		<7f014ea60905190854x2ef75b22s12e5cb0d0bebf9e9@mail.gmail.com>	
	<7f014ea60905190920p127b5589j5cd3380c443b4cbc@mail.gmail.com>
Message-ID: <4A12EB00.2070305@behnel.de>


Chris Colbert wrote:
> I guess what i'm asking, if I have a single python object reference the
> IplImage* handle, and I delete the python object, the IplImage still exists,
> so I would need to explicitly deallocate the image in the __dealloc__
> special method, correct?

Exactly. As long as you can make sure you have a 1:1 mapping between Python
objects and C structs, that's all you need to care about.

Stefan


From sccolbert at gmail.com  Tue May 19 22:45:34 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Tue, 19 May 2009 16:45:34 -0400
Subject: [Cython] fundamentals question
In-Reply-To: <4A12EB00.2070305@behnel.de>
References: <7f014ea60905190829w12fc8af2la0e646f39115c8e6@mail.gmail.com>
	<7f014ea60905190830k296daa2ekb7ded15d93ded6e2@mail.gmail.com>
	
	<7f014ea60905190854x2ef75b22s12e5cb0d0bebf9e9@mail.gmail.com>
	
	<7f014ea60905190920p127b5589j5cd3380c443b4cbc@mail.gmail.com>
	<4A12EB00.2070305@behnel.de>
Message-ID: <7f014ea60905191345q45bf9392l20c12e8545d92c37@mail.gmail.com>

how do a pass a pointer to a C datatype to my python extension type
constructor? I have the following in my code:


 #### in a cdef extern from .h block ######
struct _IplImage:
         
ctypedef _IplImage IplImage


###### extension type declaration #######
cdef class PyIplImage:

    cdef IplImage* img
    cdef readonly int width, height

    def __cinit__(self, IplImage* imgptr):
        self.img = imgptr
        self.width = self.img[0].width
        self.height = self.img[0].height


###### test functions #################

cdef IplImage* load(char* filename):
    cdef IplImage* img = cvLoadImage(filename, 1)

    return img

def testload(filename):
    cdef IplImage* img = load(filename)
    pyimg = PyIplImage(img)
    return pyimg




Cython compilation errs with "Cannot convert IplImage* to python object"


Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090519/645a5a0f/attachment.htm 

From robertwb at math.washington.edu  Tue May 19 22:56:25 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 19 May 2009 13:56:25 -0700
Subject: [Cython] fundamentals question
In-Reply-To: <7f014ea60905191345q45bf9392l20c12e8545d92c37@mail.gmail.com>
References: <7f014ea60905190829w12fc8af2la0e646f39115c8e6@mail.gmail.com>
	<7f014ea60905190830k296daa2ekb7ded15d93ded6e2@mail.gmail.com>
	
	<7f014ea60905190854x2ef75b22s12e5cb0d0bebf9e9@mail.gmail.com>
	
	<7f014ea60905190920p127b5589j5cd3380c443b4cbc@mail.gmail.com>
	<4A12EB00.2070305@behnel.de>
	<7f014ea60905191345q45bf9392l20c12e8545d92c37@mail.gmail.com>
Message-ID: <886D2376-8FEE-4CC0-91CD-2DBC1FACBF23@math.washington.edu>

On May 19, 2009, at 1:45 PM, Chris Colbert wrote:

> how do a pass a pointer to a C datatype to my python extension type  
> constructor? I have the following in my code:
>
>
>  #### in a cdef extern from .h block ######
> struct _IplImage:
>          
> ctypedef _IplImage IplImage
>
>
> ###### extension type declaration #######
> cdef class PyIplImage:
>
>     cdef IplImage* img
>     cdef readonly int width, height
>
>     def __cinit__(self, IplImage* imgptr):
>         self.img = imgptr
>         self.width = self.img[0].width
>         self.height = self.img[0].height
>
>
> ###### test functions #################
>
> cdef IplImage* load(char* filename):
>     cdef IplImage* img = cvLoadImage(filename, 1)
>
>     return img
>
> def testload(filename):
>     cdef IplImage* img = load(filename)
>     pyimg = PyIplImage(img)
>     return pyimg
>
>
> Cython compilation errs with "Cannot convert IplImage* to python  
> object"

Unfortunately, you can't, because the parameters to __cinit__ are the  
same parameters to __init__ which are fixed to be Python objects.

What you can do is allow it to pass in a char* (bytes) object with a  
length, have a cdef set method, use opaque object pointer wrappers  
(there was a previous thread about this). You could also make a function

> def testload(filename):
>     cdef IplImage* img = load(filename)
>     cdef PyIplImage pyimg = PyIplImage
>     pyimg.img = img
>     return pyimg



- Robert



From sccolbert at gmail.com  Tue May 19 23:02:00 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Tue, 19 May 2009 17:02:00 -0400
Subject: [Cython] fundamentals question
In-Reply-To: <886D2376-8FEE-4CC0-91CD-2DBC1FACBF23@math.washington.edu>
References: <7f014ea60905190829w12fc8af2la0e646f39115c8e6@mail.gmail.com>
	<7f014ea60905190830k296daa2ekb7ded15d93ded6e2@mail.gmail.com>
	
	<7f014ea60905190854x2ef75b22s12e5cb0d0bebf9e9@mail.gmail.com>
	
	<7f014ea60905190920p127b5589j5cd3380c443b4cbc@mail.gmail.com>
	<4A12EB00.2070305@behnel.de>
	<7f014ea60905191345q45bf9392l20c12e8545d92c37@mail.gmail.com>
	<886D2376-8FEE-4CC0-91CD-2DBC1FACBF23@math.washington.edu>
Message-ID: <7f014ea60905191402k2211431bs7dbca1b10dca59e@mail.gmail.com>

>
>
> What you can do is allow it to pass in a char* (bytes) object with a
> length, have a cdef set method, use opaque object pointer wrappers
> (there was a previous thread about this).


i'm not sure I understand what your saying with this ^^^


> You could also make a function
>
> > def testload(filename):
> >     cdef IplImage* img = load(filename)
> >     cdef PyIplImage pyimg = PyIplImage
> >     pyimg.img = img
> >     return pyimg


this makes sense. Just instantiate an empty PyIplImage then set the pointer
after instantiation. But that would mean all attributes (width, height, etc)
would have to be accessed as properties right? since I wouldn't be able to
set them on __init___

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090519/fbdc260e/attachment.htm 

From dagss at student.matnat.uio.no  Tue May 19 23:07:27 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 19 May 2009 23:07:27 +0200
Subject: [Cython] Merge of cython-devel to cython-unstable needed
In-Reply-To: <5E668DCB-2220-406D-B39D-C85A968185AE@math.washington.edu>
References: <4A0DBB58.6070301@behnel.de>	<127c1a4bfb524662928d1387c85f97b3.squirrel@webmail.uio.no>
	<5E668DCB-2220-406D-B39D-C85A968185AE@math.washington.edu>
Message-ID: <4A131F8F.50606@student.matnat.uio.no>

Robert Bradshaw wrote:
> On May 15, 2009, at 12:04 PM, Dag Sverre Seljebotn wrote:
> 
>>> Hi,
>>>
>>> the large number of major changes in cython-devel induces a number of
>>> merge
>>> conflicts in cython-unstable. I'm very happy to see all those great
>>> improvements all over the place, but could those who implemented them
>>> please try to re-integrate them with the unstable branch?
>> At least for what I should be doing: Because of time constraints I  
>> need to
>> keep focus on cython-devel myself; I'll be happy to merge next week  
>> after
>> the release is done.
>>
>> I don't think what's happened in -devel is a blocker for anything that
>> should happen in unstable the next week?
> 
> I just took a quick look and it seems most of the conflicts are due  
> to refactoring of the Code.py and code streams. Given your work in  
> the area it seems, Dag, that you're the one who would be best capable  
> of doing this, even if it has to wait 'till next week.

Done.

-- 
Dag Sverre

From robertwb at math.washington.edu  Tue May 19 23:17:51 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 19 May 2009 14:17:51 -0700
Subject: [Cython] fundamentals question
In-Reply-To: <7f014ea60905191402k2211431bs7dbca1b10dca59e@mail.gmail.com>
References: <7f014ea60905190829w12fc8af2la0e646f39115c8e6@mail.gmail.com>
	<7f014ea60905190830k296daa2ekb7ded15d93ded6e2@mail.gmail.com>
	
	<7f014ea60905190854x2ef75b22s12e5cb0d0bebf9e9@mail.gmail.com>
	
	<7f014ea60905190920p127b5589j5cd3380c443b4cbc@mail.gmail.com>
	<4A12EB00.2070305@behnel.de>
	<7f014ea60905191345q45bf9392l20c12e8545d92c37@mail.gmail.com>
	<886D2376-8FEE-4CC0-91CD-2DBC1FACBF23@math.washington.edu>
	<7f014ea60905191402k2211431bs7dbca1b10dca59e@mail.gmail.com>
Message-ID: <4646288C-4C25-489B-A266-9E3C6AB95639@math.washington.edu>

On May 19, 2009, at 2:02 PM, Chris Colbert wrote:

> What you can do is allow it to pass in a char* (bytes) object with a
> length, have a cdef set method, use opaque object pointer wrappers
> (there was a previous thread about this).
>
> i'm not sure I understand what your saying with this ^^^
>
> You could also make a function
>
> > def testload(filename):
> >     cdef IplImage* img = load(filename)
> >     cdef PyIplImage pyimg = PyIplImage
> >     pyimg.img = img
> >     return pyimg
>
> this makes sense. Just instantiate an empty PyIplImage then set the  
> pointer after instantiation. But that would mean all attributes  
> (width, height, etc) would have to be accessed as properties right?  
> since I wouldn't be able to set them on __init___

You can set readonly attributes from C.

- Robert



From robertwb at math.washington.edu  Tue May 19 23:18:20 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 19 May 2009 14:18:20 -0700
Subject: [Cython] Merge of cython-devel to cython-unstable needed
In-Reply-To: <4A131F8F.50606@student.matnat.uio.no>
References: <4A0DBB58.6070301@behnel.de>	<127c1a4bfb524662928d1387c85f97b3.squirrel@webmail.uio.no>
	<5E668DCB-2220-406D-B39D-C85A968185AE@math.washington.edu>
	<4A131F8F.50606@student.matnat.uio.no>
Message-ID: <03045CA8-56A7-4F44-85FA-78054E742C71@math.washington.edu>

On May 19, 2009, at 2:07 PM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On May 15, 2009, at 12:04 PM, Dag Sverre Seljebotn wrote:
>>
>>>> Hi,
>>>>
>>>> the large number of major changes in cython-devel induces a  
>>>> number of
>>>> merge
>>>> conflicts in cython-unstable. I'm very happy to see all those great
>>>> improvements all over the place, but could those who implemented  
>>>> them
>>>> please try to re-integrate them with the unstable branch?
>>> At least for what I should be doing: Because of time constraints I
>>> need to
>>> keep focus on cython-devel myself; I'll be happy to merge next week
>>> after
>>> the release is done.
>>>
>>> I don't think what's happened in -devel is a blocker for anything  
>>> that
>>> should happen in unstable the next week?
>>
>> I just took a quick look and it seems most of the conflicts are due
>> to refactoring of the Code.py and code streams. Given your work in
>> the area it seems, Dag, that you're the one who would be best capable
>> of doing this, even if it has to wait 'till next week.
>
> Done.

Thanks!

- Robert



From sccolbert at gmail.com  Tue May 19 23:22:12 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Tue, 19 May 2009 17:22:12 -0400
Subject: [Cython] fundamentals question
In-Reply-To: <4646288C-4C25-489B-A266-9E3C6AB95639@math.washington.edu>
References: <7f014ea60905190829w12fc8af2la0e646f39115c8e6@mail.gmail.com>
	
	<7f014ea60905190854x2ef75b22s12e5cb0d0bebf9e9@mail.gmail.com>
	
	<7f014ea60905190920p127b5589j5cd3380c443b4cbc@mail.gmail.com>
	<4A12EB00.2070305@behnel.de>
	<7f014ea60905191345q45bf9392l20c12e8545d92c37@mail.gmail.com>
	<886D2376-8FEE-4CC0-91CD-2DBC1FACBF23@math.washington.edu>
	<7f014ea60905191402k2211431bs7dbca1b10dca59e@mail.gmail.com>
	<4646288C-4C25-489B-A266-9E3C6AB95639@math.washington.edu>
Message-ID: <7f014ea60905191422u629321f1ycf0246e7003e26b4@mail.gmail.com>

True, but it seems a little "off" to programatically set every attribute one
by one every time i create an instance of that type.

Is it not recommended to declare a property for every attribute I want
access from in python?

Chris

On Tue, May 19, 2009 at 5:17 PM, Robert Bradshaw <
robertwb at math.washington.edu> wrote:

> On May 19, 2009, at 2:02 PM, Chris Colbert wrote:
>
> > What you can do is allow it to pass in a char* (bytes) object with a
> > length, have a cdef set method, use opaque object pointer wrappers
> > (there was a previous thread about this).
> >
> > i'm not sure I understand what your saying with this ^^^
> >
> > You could also make a function
> >
> > > def testload(filename):
> > >     cdef IplImage* img = load(filename)
> > >     cdef PyIplImage pyimg = PyIplImage
> > >     pyimg.img = img
> > >     return pyimg
> >
> > this makes sense. Just instantiate an empty PyIplImage then set the
> > pointer after instantiation. But that would mean all attributes
> > (width, height, etc) would have to be accessed as properties right?
> > since I wouldn't be able to set them on __init___
>
> You can set readonly attributes from C.
>
> - Robert
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090519/c34fc4dd/attachment-0001.htm 

From sccolbert at gmail.com  Wed May 20 01:22:56 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Tue, 19 May 2009 19:22:56 -0400
Subject: [Cython] interfacing with OpenCV?
In-Reply-To: 
References: <7f014ea60905171323j5fe490bfm8ba57de2352eea07@mail.gmail.com>
	<7f014ea60905181521m3495a68fle0fa38c5441d4b8c@mail.gmail.com>
	
	<7f014ea60905181630i79c85213gfc2225fa6d5fda1@mail.gmail.com>
	<7f014ea60905181631j236b8efet550d0234922c7185@mail.gmail.com>
	<7f014ea60905181635l6dfc9726q3f9360d9009e135e@mail.gmail.com>
	<4db580fd0905181647q1195b677s4be96fa2211e2c0c@mail.gmail.com>
	<7f014ea60905181655idd84571t4cfb0c4e8ca16931@mail.gmail.com>
	<7f014ea60905181840h54c56ab3gecf072f1cd393b56@mail.gmail.com>
	
Message-ID: <7f014ea60905191622o1f5adb5fw8c35b590ecd98d55@mail.gmail.com>

>
>  Once you get used to code
> Cython, and assuming you have some C/C++ background, I bet you will
> stop using ctypes ;-) ..
>

I think you are right :). In an afternoon, I've got a decent amount of what
I need wrapped :)

Cheers!

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090519/dce8e27a/attachment.htm 

From sccolbert at gmail.com  Wed May 20 02:05:48 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Tue, 19 May 2009 20:05:48 -0400
Subject: [Cython] returning a char* as a python string
Message-ID: <7f014ea60905191705u1b24ae7eta37c2c42daead945@mail.gmail.com>

I know that Cython does an implicit conversion from char* to Python string,
but do I need to somehow specify the length of the string?
I ask because I have a data buffer of known size (7854000 bytes uint8) but
when i pass the pointer to that buffer into python, my string length is only
4617.

this is in python 2.5.

am I missing something?

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090519/17ded3e1/attachment.htm 

From dalcinl at gmail.com  Wed May 20 02:08:30 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 19 May 2009 21:08:30 -0300
Subject: [Cython] fundamentals question
In-Reply-To: <7f014ea60905191422u629321f1ycf0246e7003e26b4@mail.gmail.com>
References: <7f014ea60905190829w12fc8af2la0e646f39115c8e6@mail.gmail.com>
	<7f014ea60905190854x2ef75b22s12e5cb0d0bebf9e9@mail.gmail.com>
	
	<7f014ea60905190920p127b5589j5cd3380c443b4cbc@mail.gmail.com>
	<4A12EB00.2070305@behnel.de>
	<7f014ea60905191345q45bf9392l20c12e8545d92c37@mail.gmail.com>
	<886D2376-8FEE-4CC0-91CD-2DBC1FACBF23@math.washington.edu>
	<7f014ea60905191402k2211431bs7dbca1b10dca59e@mail.gmail.com>
	<4646288C-4C25-489B-A266-9E3C6AB95639@math.washington.edu>
	<7f014ea60905191422u629321f1ycf0246e7003e26b4@mail.gmail.com>
Message-ID: 

Not sure is this will help you, but perhaps you could define your
__cinit__() to have an empty image (perhaps with a NULL pointer), and
then define some creational methods, then you could do from Python
side

img1 = Image()
img1.createFromFileData(filename)

img2 = Image()
img2.createFromNumpyData(numpy_array)

img3 = Image()
img4.createFromStringData(python_byte_string)


Other way could be do define static/class factory methods, and then you do

img1 = Image.createFromFileData(filename)

img2 = Image.createFromNumpyData(numpy_array)

img4 = Image.createFromStringData(python_byte_string)

The two alternatives are rather similar, however, the second has some
conveniences.

A BIG problem with the fist option is that if you let a Python
instance to have a NULL pointer, then you will need to check against
that NULL on almost every other method call calling you C lib
routines, if not, you could potentially get nasty segfaults.

By using the second patter, you can restrict Python instances to
always have a non-NULL, valid handle to a C-side, then you can code
your methods with prior knowledge that the handle is a valid one...

In short, all this is in general a matter of personal style and
preferences. I would perhaps follow option 2, I mean, define some
factory @classmethod's you call from Python side to create images from
the many sources...




On Tue, May 19, 2009 at 6:22 PM, Chris Colbert  wrote:
> True, but it seems a little "off" to programatically set every attribute one
> by one every time i create an instance of that type.
>
> Is it not recommended to declare a property for every attribute I want
> access from in python?
>
> Chris
>
> On Tue, May 19, 2009 at 5:17 PM, Robert Bradshaw
>  wrote:
>>
>> On May 19, 2009, at 2:02 PM, Chris Colbert wrote:
>>
>> > What you can do is allow it to pass in a char* (bytes) object with a
>> > length, have a cdef set method, use opaque object pointer wrappers
>> > (there was a previous thread about this).
>> >
>> > i'm not sure I understand what your saying with this ^^^
>> >
>> > You could also make a function
>> >
>> > > def testload(filename):
>> > > ? ? cdef IplImage* img = load(filename)
>> > > ? ? cdef PyIplImage pyimg = PyIplImage
>> > > ? ? pyimg.img = img
>> > > ? ? return pyimg
>> >
>> > this makes sense. Just instantiate an empty PyIplImage then set the
>> > pointer after instantiation. But that would mean all attributes
>> > (width, height, etc) would have to be accessed as properties right?
>> > since I wouldn't be able to set them on __init___
>>
>> You can set readonly attributes from C.
>>
>> - Robert
>>
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From sccolbert at gmail.com  Wed May 20 02:11:55 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Tue, 19 May 2009 20:11:55 -0400
Subject: [Cython] fundamentals question
In-Reply-To: 
References: <7f014ea60905190829w12fc8af2la0e646f39115c8e6@mail.gmail.com>
	
	<7f014ea60905190920p127b5589j5cd3380c443b4cbc@mail.gmail.com>
	<4A12EB00.2070305@behnel.de>
	<7f014ea60905191345q45bf9392l20c12e8545d92c37@mail.gmail.com>
	<886D2376-8FEE-4CC0-91CD-2DBC1FACBF23@math.washington.edu>
	<7f014ea60905191402k2211431bs7dbca1b10dca59e@mail.gmail.com>
	<4646288C-4C25-489B-A266-9E3C6AB95639@math.washington.edu>
	<7f014ea60905191422u629321f1ycf0246e7003e26b4@mail.gmail.com>
	
Message-ID: <7f014ea60905191711w5140d1fdw215b347f106d0de@mail.gmail.com>

Those are good ideas.

thanks Lisandro!

On Tue, May 19, 2009 at 8:08 PM, Lisandro Dalcin  wrote:

> Not sure is this will help you, but perhaps you could define your
> __cinit__() to have an empty image (perhaps with a NULL pointer), and
> then define some creational methods, then you could do from Python
> side
>
> img1 = Image()
> img1.createFromFileData(filename)
>
> img2 = Image()
> img2.createFromNumpyData(numpy_array)
>
> img3 = Image()
> img4.createFromStringData(python_byte_string)
>
>
> Other way could be do define static/class factory methods, and then you do
>
> img1 = Image.createFromFileData(filename)
>
> img2 = Image.createFromNumpyData(numpy_array)
>
> img4 = Image.createFromStringData(python_byte_string)
>
> The two alternatives are rather similar, however, the second has some
> conveniences.
>
> A BIG problem with the fist option is that if you let a Python
> instance to have a NULL pointer, then you will need to check against
> that NULL on almost every other method call calling you C lib
> routines, if not, you could potentially get nasty segfaults.
>
> By using the second patter, you can restrict Python instances to
> always have a non-NULL, valid handle to a C-side, then you can code
> your methods with prior knowledge that the handle is a valid one...
>
> In short, all this is in general a matter of personal style and
> preferences. I would perhaps follow option 2, I mean, define some
> factory @classmethod's you call from Python side to create images from
> the many sources...
>
>
>
>
> On Tue, May 19, 2009 at 6:22 PM, Chris Colbert 
> wrote:
> > True, but it seems a little "off" to programatically set every attribute
> one
> > by one every time i create an instance of that type.
> >
> > Is it not recommended to declare a property for every attribute I want
> > access from in python?
> >
> > Chris
> >
> > On Tue, May 19, 2009 at 5:17 PM, Robert Bradshaw
> >  wrote:
> >>
> >> On May 19, 2009, at 2:02 PM, Chris Colbert wrote:
> >>
> >> > What you can do is allow it to pass in a char* (bytes) object with a
> >> > length, have a cdef set method, use opaque object pointer wrappers
> >> > (there was a previous thread about this).
> >> >
> >> > i'm not sure I understand what your saying with this ^^^
> >> >
> >> > You could also make a function
> >> >
> >> > > def testload(filename):
> >> > >     cdef IplImage* img = load(filename)
> >> > >     cdef PyIplImage pyimg = PyIplImage
> >> > >     pyimg.img = img
> >> > >     return pyimg
> >> >
> >> > this makes sense. Just instantiate an empty PyIplImage then set the
> >> > pointer after instantiation. But that would mean all attributes
> >> > (width, height, etc) would have to be accessed as properties right?
> >> > since I wouldn't be able to set them on __init___
> >>
> >> You can set readonly attributes from C.
> >>
> >> - Robert
> >>
> >>
> >> _______________________________________________
> >> Cython-dev mailing list
> >> Cython-dev at codespeak.net
> >> http://codespeak.net/mailman/listinfo/cython-dev
> >
> >
> > _______________________________________________
> > Cython-dev mailing list
> > Cython-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/cython-dev
> >
> >
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090519/2a2a3315/attachment.htm 

From dalcinl at gmail.com  Wed May 20 02:33:57 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 19 May 2009 21:33:57 -0300
Subject: [Cython] returning a char* as a python string
In-Reply-To: <7f014ea60905191705u1b24ae7eta37c2c42daead945@mail.gmail.com>
References: <7f014ea60905191705u1b24ae7eta37c2c42daead945@mail.gmail.com>
Message-ID: 

On Tue, May 19, 2009 at 9:05 PM, Chris Colbert  wrote:
> I know that Cython does an implicit conversion from char* to Python string,
> but do I need to somehow specify the length of the string?

But then Cython assumes you string is a NULL-terminated one. In other
words, you cannot return this way a char* buffer with embed NULL's.

> I ask because I have a data buffer of known size (7854000 bytes uint8) but
> when i pass the pointer to that buffer into python, my string length is only
> 4617.

Likely your char* buffer has a NULL at index 4617.

>
> this is in python 2.5.
>
> am I missing something?
>

Yes. You should use this core Python C-API call:

http://docs.python.org/c-api/string.html#PyString_FromStringAndSize

This is the way to create Python (byte) strings with embed NULL's.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From sccolbert at gmail.com  Wed May 20 02:48:30 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Tue, 19 May 2009 20:48:30 -0400
Subject: [Cython] returning a char* as a python string
In-Reply-To: 
References: <7f014ea60905191705u1b24ae7eta37c2c42daead945@mail.gmail.com>
	
Message-ID: <7f014ea60905191748ma512b4csda19ec3e0761eb61@mail.gmail.com>

Lisandro,

Once again, thank you!

I think I finally have a tricky question for you now however:

I have the OpenCV IplImage class wrapped as so:

####################################

cdef class IplImage:

    cdef c_cxcore.IplImage* thisptr

    #Convienience Functions

    def show(self):
    # a convienience function which displays the image in a new window
    # useful for debugging. window exits on keypress.
        c_highgui.cvNamedWindow('test', 1)
        c_highgui.cvShowImage('test', self.thisptr)
        c_highgui.cvWaitKey(0)
        c_highgui.cvDestroyWindow('test')

    def __dealloc__(self):
        c_cxcore.cvReleaseImage(&self.thisptr)


     # property access methods
     ....


def cvLoadImage(filename):
    cdef c_cxcore.IplImage* img = c_highgui.cvLoadImage(filename, 1)
    cdef IplImage pyimg = IplImage()
    pyimg.thisptr = img
    return pyimg
#############################




now in python, when i do something like this:

img = cvLoadImage('test1.bmp')
img.show()                                  <----- this works fine
img = cvLoadImage('test2.bmp')    <------ this also works
img.show()                                  <---------- I crash here



i'm thinking that python is calling my __dealloc__ and its somehow deleting
the new pointer out from under me


what do you think?

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090519/461c645d/attachment.htm 

From sccolbert at gmail.com  Wed May 20 02:57:10 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Tue, 19 May 2009 20:57:10 -0400
Subject: [Cython] returning a char* as a python string
In-Reply-To: <7f014ea60905191748ma512b4csda19ec3e0761eb61@mail.gmail.com>
References: <7f014ea60905191705u1b24ae7eta37c2c42daead945@mail.gmail.com>
	
	<7f014ea60905191748ma512b4csda19ec3e0761eb61@mail.gmail.com>
Message-ID: <7f014ea60905191757u1b16e55x26dd613c02d61cf3@mail.gmail.com>

oops, sorry, I wrote the wrong things (it's getting late : )

i'm actually erring on this

######### typdef ############

cdef class CvCapture:

    cdef c_highgui.CvCapture* thisptr

    # convienience functions

    def queryFrame(self):
    # get and return an IplImage from from a capture source
    # equivalent to img = cvQueryFrame(CvCapture* capture)
    # make sure to set capture width and height before calling this
        cdef c_cxcore.IplImage* img
        cdef IplImage pyimg = IplImage()
        img = c_highgui.cvQueryFrame(self.thisptr)
        pyimg.thisptr = img
        return pyimg

    def __dealloc__(self):
        c_highgui.cvReleaseCapture(&self.thisptr)


def cvCreateCameraCapture(device):
    cdef c_highgui.CvCapture* cap = c_highgui.cvCreateCameraCapture(device)
    cdef CvCapture pycap = CvCapture()
    pycap.thisptr = cap
    return pycap

####################

python code

cam = cvCreateCameraCapture(0)
img = cam.queryFrame()
img.show()                                     <----- this works fine
img = cam.queryFrame()                <---------- I crash here


sorry for the confusion


Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090519/8260cb6b/attachment.htm 

From dalcinl at gmail.com  Wed May 20 04:11:41 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 19 May 2009 23:11:41 -0300
Subject: [Cython] returning a char* as a python string
In-Reply-To: <7f014ea60905191757u1b16e55x26dd613c02d61cf3@mail.gmail.com>
References: <7f014ea60905191705u1b24ae7eta37c2c42daead945@mail.gmail.com>
	
	<7f014ea60905191748ma512b4csda19ec3e0761eb61@mail.gmail.com>
	<7f014ea60905191757u1b16e55x26dd613c02d61cf3@mail.gmail.com>
Message-ID: 

Any chance that cvQueryFrame() returns an image that you should not
free?? When you do twice:

img = cam.queryFrame()
img = cam.queryFrame()

the first img is likely garbage-collected. Perhaps your
Image.__dealloc__ is not doing the right thing in this case?

Could you try to do:

cam = cvCreateCameraCapture(0)
img1 = cam.queryFrame()
img1.show()
img2 = cam.queryFrame()

and tell me if this also fails?




On Tue, May 19, 2009 at 9:57 PM, Chris Colbert  wrote:
> oops, sorry, I wrote the wrong things (it's getting late : )
>
> i'm actually erring on this
>
> ######### typdef ############
>
> cdef class CvCapture:
>
> ??? cdef c_highgui.CvCapture* thisptr
>
> ??? # convienience functions
>
> ??? def queryFrame(self):
> ??? # get and return an IplImage from from a capture source
> ??? # equivalent to img = cvQueryFrame(CvCapture* capture)
> ??? # make sure to set capture width and height before calling this
> ??????? cdef c_cxcore.IplImage* img
> ??????? cdef IplImage pyimg = IplImage()
> ??????? img = c_highgui.cvQueryFrame(self.thisptr)
> ??????? pyimg.thisptr = img
> ??????? return pyimg
>
> ??? def __dealloc__(self):
> ??????? c_highgui.cvReleaseCapture(&self.thisptr)
>
>
> def cvCreateCameraCapture(device):
> ??? cdef c_highgui.CvCapture* cap = c_highgui.cvCreateCameraCapture(device)
> ??? cdef CvCapture pycap = CvCapture()
> ??? pycap.thisptr = cap
> ??? return pycap
>
> ####################
>
> python code
>
> cam = cvCreateCameraCapture(0)
> img = cam.queryFrame()
> img.show()???????????????????????????????????? <----- this works fine
> img = cam.queryFrame() ?????????????? <---------- I crash here
>
>
> sorry for the confusion
>
>
> Chris
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From sccolbert at gmail.com  Wed May 20 04:42:20 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Tue, 19 May 2009 22:42:20 -0400
Subject: [Cython] returning a char* as a python string
In-Reply-To: 
References: <7f014ea60905191705u1b24ae7eta37c2c42daead945@mail.gmail.com>
	
	<7f014ea60905191748ma512b4csda19ec3e0761eb61@mail.gmail.com>
	<7f014ea60905191757u1b16e55x26dd613c02d61cf3@mail.gmail.com>
	
Message-ID: <7f014ea60905191942m1ff8413by2300a327f08781d4@mail.gmail.com>

Any chance that cvQueryFrame() returns an image that you should not
> free??
>
> that's exactly it (i feel really dumb for having missed it)


>
> the first img is likely garbage-collected. Perhaps your
> Image.__dealloc__ is not doing the right thing in this case?


would you suggest setting a private C-attribute (flag) that indicates the
image was created by this (or any other) function where i'm no supposed to
explicity release it, then check that flag in __dealloc__ and proceed
accordingly?

This would be my knee jerk reaction around the problem, but maybe you have a
more elegant solution.

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090519/12668c61/attachment.htm 

From dalcinl at gmail.com  Wed May 20 05:27:08 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 20 May 2009 00:27:08 -0300
Subject: [Cython] returning a char* as a python string
In-Reply-To: <7f014ea60905191942m1ff8413by2300a327f08781d4@mail.gmail.com>
References: <7f014ea60905191705u1b24ae7eta37c2c42daead945@mail.gmail.com>
	
	<7f014ea60905191748ma512b4csda19ec3e0761eb61@mail.gmail.com>
	<7f014ea60905191757u1b16e55x26dd613c02d61cf3@mail.gmail.com>
	
	<7f014ea60905191942m1ff8413by2300a327f08781d4@mail.gmail.com>
Message-ID: 

Such a flag would work... For example, IIUC ctypes does this (I'm
remembering a 'needs_free'  attribute or something like that on some
ctypes instances...)

Just in case, I'll ask... Any chance the library you are using has
some sort of reference count mechanism? If it do, you sould rely on
that to implement Image.__dealloc__() ... If not, how do you track
ownership of your C handles in a pure-C code ??


On Tue, May 19, 2009 at 11:42 PM, Chris Colbert  wrote:
>
>
>> Any chance that cvQueryFrame() returns an image that you should not
>> free??
>>
> that's exactly it (i feel really dumb for having missed it)
>
>>
>> the first img is likely garbage-collected. Perhaps your
>> Image.__dealloc__ is not doing the right thing in this case?
>
>
> would you suggest setting a private C-attribute (flag) that indicates the
> image was created by this (or any other) function where i'm no supposed to
> explicity release it, then check that flag in __dealloc__ and proceed
> accordingly?
>
> This would be my knee jerk reaction around the problem, but maybe you have a
> more elegant solution.
>
> Chris
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Wed May 20 08:29:00 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 20 May 2009 08:29:00 +0200 (CEST)
Subject: [Cython] Tutorial at NOTUR 2009
In-Reply-To: 
References: 
	
Message-ID: 

> Dag, could you check permission on files inside
> http://sage.math.washington.edu/home/dagss/cython-notur09/mpi/ ? I'm
> getting "Forbidden" here...

Thanks. It's fixed now. Please tell me if I did anything wrong with MPI
:-) And there's an example in matmul/matmul_mpi.pyx as well.

(BTW what's the LaTeX command for the "i" in your last name?)

Dag Sverre


From tenninosko at msn.com  Wed May 20 11:30:16 2009
From: tenninosko at msn.com (OUARDA MEHDI)
Date: Wed, 20 May 2009 11:30:16 +0200
Subject: [Cython] help....cython code optimisation
Message-ID: 


hi,
thanks gays for your help and patience.but i sill have a problem
even when i follow Stefan recommendations.in fact some of my cython
classes inherits from other pure python classes(no multiple
inheritance) and i wonder if it causes problem when using methods from
it(i was thinking about rewriting *.pxd header for parent classes and
cimport methods from it or something like...).back to property
problem...here is my code:


from numpy import array, zeros, eye, dot, sin, cos          #no more need of sin, cos
from abc import ABCMeta, abstractmethod
import homogeneousmatrix
from misc import NamedObject
from arboris import Joint

#mes modifs....
cimport numpy as np
DTYPE = np.float 
ctypedef np.float_t DTYPE_t
cdef extern from "math.h":
    float cosf(float theta)
    float sinf(float theta)


#must see Joint class to enhance this one...
class LinearConfigurationSpaceJoint(Joint):
    """
    joints whose space is diffeomorph to the real set.
    """

    def integrate(self, dt):
        self.gpos += dt * self.gvel



cdef class FreeJoint(Joint):

    """Free joint (6-dof)
    """
    def __init__(self, gpos=None, gvel=None, name=None):
        """
        example:
        >>> j = FreeJoint()
        >>> j.gpos
        array([[ 1.,  0.,  0.,  0.],
               [ 0.,  1.,  0.,  0.],
               [ 0.,  0.,  1.,  0.],
               [ 0.,  0.,  0.,  1.]])
        >>> j.gvel
        array([ 0.,  0.,  0.,  0.,  0.,  0.])
        """
        if gpos is None:
            gpos = eye(4)
        if gvel is None:
            gvel = zeros((6))
        self.gpos = array(gpos).reshape((4,4))
        self.gvel = array(gvel).reshape((6))
        Joint.__init__(self, name)

    property ndof:
        def __get__(self):   # This is called when the property is read.
            return 6
    
    property pose:
        def __get__(self):
            return self.gpos.copy()

    property twist:
        def __get__(self):
            return self.gvel.copy()

    property jacobian:
        def __get__(self):
            return eye(6)

    property djacobian:
        def __get__(self):
            return zeros((6,6))

    def integrate(self, dt):         #must see twistvector, exp dot to enhance this methode
        from twistvector import exp  #twistvector must be declared 'cdef exter class tewistsvector'
        self.gpos = dot(self.gpos, exp( dt*self.gvel))

#class PivotJoint(Joint):
#
#    """Pivot (2-dof)
#    """
#
#class BallJoint(Joint):
#
#    """Ball and socket (3-dof)
#    """



cdef class RzRyRxJoint(LinearConfigurationSpaceJoint):
    """Ball and socket (3-dof) implemented with 3 serial hinges

    the resulting homogeneous matrix is given by H = Rz*Ry*Rx
    """
    def __init__(self, gpos=[0.,0.,0.], gvel=[0.,0.,0.], name=None):
        self.gpos = array(gpos).reshape((3))
        self.gvel = array(gvel).reshape((3))
        Joint.__init__(self, name)

    property ndof:  #mes modifs.......
        def __get__(self):
            return 3

    property pose:
        def __get__(self):
            return homogeneousmatrix.rotzyx(self.gpos)
    
    cdef _getjacobian(self):
    
        """
        T_n/r = 
        """   
        cdef float sx = sinf(self.gpos[0])
        cdef float cx = cosf(self.gpos[0])
        cdef float sy = sinf(self.gpos[1])
        cdef float cy = cosf(self.gpos[1])
        cdef np.ndarray[DTYPE_t, ndim=2] value = np.array (
            ([ 1.   ,  0. , -sy    ],
             [ 0.   , cx  ,  sx*sy ],
             [ 0.   ,-sx  ,  cx*cy ],
             [ 0.   ,  0. ,  0.    ],
             [ 0.   ,  0. ,  0.    ],
             [ 0.   ,  0. ,  0.    ]),   dtype=DTYPE)
        return value
    #we used this method to preserve the use of proprety:(
    #eventally look further if we could write cpdef jacobian.... and at the same time preserve proprety
    #pay attention about names...
    property jacobian:
        def __get__(self):
            return self._getjacobian()         
    #sam thing for djacobian
    cdef _getdjacobian(self):
        cdef float sx = sinf(self.gpos[0])
        cdef float cx = cosf(self.gpos[0])
        cdef float sy = sinf(self.gpos[1])
        cdef float cy = cosf(self.gpos[1])
        cdef float dx = self.gvel[0]
        cdef float dy = self.gvel[1]
        cdef np.ndarray[DTYPE_t, ndim=2] value = np.array (
            ([ 0.   , 0.    ,-dy*cy             ],
             [ 0.   ,-dx*sx , dx*cx*sy+dy*sx*cy ],
             [ 0.   ,-dx*cx ,-dx*sx*cy-dy*cx*sy ],
             [ 0.   ,  0. ,  0.    ],
             [ 0.   ,  0. ,  0.    ],
             [ 0.   ,  0. ,  0.    ]), dtype=DTYPE)
        return value

    property djacobian:
        def __get__(self):
            return self._getdjacobian()         

and here is the compilation error messages:

tenninos at tenninos-laptop:~/stage/arboris-python/src$ python setup.py build_ext --inplace
running build_ext
cythoning joints_c.pyx to joints_c.c

Error converting Pyrex file to C:
------------------------------------------------------------
...
    def integrate(self, dt):
        self.gpos += dt * self.gvel



cdef class FreeJoint(Joint):
    ^
------------------------------------------------------------

/home/tenninos/stage/arboris-python/src/joints_c.pyx:31:5: 'Joint' is not a type name

Error converting Pyrex file to C:
------------------------------------------------------------
...
#    """Ball and socket (3-dof)
#    """



cdef class RzRyRxJoint(LinearConfigurationSpaceJoint):
    ^
------------------------------------------------------------

/home/tenninos/stage/arboris-python/src/joints_c.pyx:91:5: 'LinearConfigurationSpaceJoint' is not a type name

Error converting Pyrex file to C:
------------------------------------------------------------
...

    property djacobian:
        def __get__(self):
            return self._getdjacobian()         

cdef class RzRyJoint(LinearConfigurationSpaceJoint):
    ^
------------------------------------------------------------

/home/tenninos/stage/arboris-python/src/joints_c.pyx:153:5: 'LinearConfigurationSpaceJoint' is not a type name

Error converting Pyrex file to C:
------------------------------------------------------------
...
        return value
    property djacobian:
        def __get__(self):
            return self._getdjacobian()

cdef class RzRxJoint(LinearConfigurationSpaceJoint):
    ^
------------------------------------------------------------


_________________________________________________________________
T?l?phonez gratuitement ? tous vos proches avec Windows Live Messenger? !? T?l?chargez-le maintenant !?
http://www.windowslive.fr/messenger/1.asp
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090520/e4ef70a7/attachment-0001.htm 

From stefan_ml at behnel.de  Wed May 20 13:00:19 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 20 May 2009 13:00:19 +0200 (CEST)
Subject: [Cython] help....cython code optimisation
In-Reply-To: 
References: 
Message-ID: <1ce5df04e2929935a1b97cc0dbe68e67.squirrel@groupware.dvs.informatik.tu-darmstadt.de>

OUARDA MEHDI wrote:
> thanks gays for your help and patience.but i sill have a problem
> even when i follow Stefan recommendations.

Does this help clearing things up?

http://docs.cython.org/docs/extension_types.html

Stefan


From dagss at student.matnat.uio.no  Wed May 20 14:49:36 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 20 May 2009 14:49:36 +0200
Subject: [Cython] Note on C99 compound literals
Message-ID: <4A13FC60.50001@student.matnat.uio.no>

I did some digging and decided to share what I found, since this only 
occurs on a specific compiler and is thus hard to discover.

Kurt, pay attention, as I just recommended that you do this :-)

Apparently code like this:

  (foo_struct){0, 0, NULL}

is a C99 extension, so we probably shouldn't use it. Furthermore it 
makes things fail in g++ 4.2.4 (but not in earlier or later versions I 
tried -- anyway, 4.2.4 is the one currently on sage.math).

(The reason it fails is largely due to us jumping out of the scope 
everywhere on exceptions)

What I did for complex numbers was replace their use with an inline 
function returning a struct with the given values. Don't know if this 
can hit performance -- if the compiler inlines the function I wouldn't 
think it would make a difference.

http://gcc.gnu.org/onlinedocs/gcc-3.3.1/gcc/Compound-Literals.html

http://hg.cython.org/cython-devel/rev/15ad532e2127

-- 
Dag Sverre

From stefan at sun.ac.za  Wed May 20 17:07:19 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 20 May 2009 17:07:19 +0200
Subject: [Cython] Static methods
Message-ID: <9457e7c80905200807u7642e019j1ef54619fad162ea@mail.gmail.com>

Hi all,

I read in the documentation that static methods can be defined as follows:

cdef int MyClass_my_method(int x):
    return x

cdef class MyClass:
    my_method = classmethod(MyClass_my_method)

Now, I would like to define the corresponding header:

cdef class MyClass:
    cdef my_method(int x)

However, when I try to compile I get:

"Self argument of C method does not match parent type."

What is the correct way to address this problem?

Regards
St?fan

From sccolbert at gmail.com  Wed May 20 17:38:10 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Wed, 20 May 2009 11:38:10 -0400
Subject: [Cython] returning a char* as a python string
In-Reply-To: 
References: <7f014ea60905191705u1b24ae7eta37c2c42daead945@mail.gmail.com>
	
	<7f014ea60905191748ma512b4csda19ec3e0761eb61@mail.gmail.com>
	<7f014ea60905191757u1b16e55x26dd613c02d61cf3@mail.gmail.com>
	
	<7f014ea60905191942m1ff8413by2300a327f08781d4@mail.gmail.com>
	
Message-ID: <7f014ea60905200838l22add4c3pdbb09811c6c48860@mail.gmail.com>

It seems that certain functions implement they're memory management (this
one being one of them) and the rest rely on the programmer to explicitly
free memory.

I guess I will go with a need_free attribute as well.

Thanks for all the help!

Chris

On Tue, May 19, 2009 at 11:27 PM, Lisandro Dalcin  wrote:

> Such a flag would work... For example, IIUC ctypes does this (I'm
> remembering a 'needs_free'  attribute or something like that on some
> ctypes instances...)
>
> Just in case, I'll ask... Any chance the library you are using has
> some sort of reference count mechanism? If it do, you sould rely on
> that to implement Image.__dealloc__() ... If not, how do you track
> ownership of your C handles in a pure-C code ??
>
>
> On Tue, May 19, 2009 at 11:42 PM, Chris Colbert 
> wrote:
> >
> >
> >> Any chance that cvQueryFrame() returns an image that you should not
> >> free??
> >>
> > that's exactly it (i feel really dumb for having missed it)
> >
> >>
> >> the first img is likely garbage-collected. Perhaps your
> >> Image.__dealloc__ is not doing the right thing in this case?
> >
> >
> > would you suggest setting a private C-attribute (flag) that indicates the
> > image was created by this (or any other) function where i'm no supposed
> to
> > explicity release it, then check that flag in __dealloc__ and proceed
> > accordingly?
> >
> > This would be my knee jerk reaction around the problem, but maybe you
> have a
> > more elegant solution.
> >
> > Chris
> >
> > _______________________________________________
> > Cython-dev mailing list
> > Cython-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/cython-dev
> >
> >
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090520/29a0cb53/attachment.htm 

From dagss at student.matnat.uio.no  Wed May 20 17:47:05 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 20 May 2009 17:47:05 +0200
Subject: [Cython] ANN: Cython 0.11.2 released
Message-ID: <4A1425F9.9030100@student.matnat.uio.no>

I'm happy to announce the release of Cython 0.11.2.

http://sage.math.washington.edu/home/dagss/Cython-0.11.2.tar.gz
http://sage.math.washington.edu/home/dagss/Cython-0.11.2.zip

(Will be present on the Cython front page in a few days.)

New features:

  * There's now native complex floating point support! C99 complex will 
be used if complex.h is included, otherwise explicit complex arithmetic 
working on all C compilers is used. [Robert Bradshaw]

    cdef double complex a = 1 + 0.3j
    cdef np.ndarray[np.complex128_t, ndim=2] arr = \
        np.zeros(10, np.complex128)

  * Cython can now generate a main()-method for embedding of the Python 
interpreter into an executable (see #289) [Robert Bradshaw]

  * @wraparound directive (another way to disable arr[idx] for negative 
idx) [Dag Sverre Seljebotn]

  * Correct support for NumPy record dtypes with different alignments, 
and "cdef packed struct" support [Dag Sverre Seljebotn]

  * @callspec directive, allowing custom calling convention macros 
[Lisandro Dalcin]

  * Bug fixes and smaller improvements. For the full list, see [1].

Contributors to this release:
  - Stefan Behnel
  - Robert Bradshaw
  - Lisandro Dalcin
  - Dag Sverre Seljebotn

Thanks also to everybody who's helping us out in our discussions on the 
mailing list.

[1] 


-- 
Dag Sverre

From dalcinl at gmail.com  Wed May 20 17:46:44 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 20 May 2009 12:46:44 -0300
Subject: [Cython] Static methods
In-Reply-To: <9457e7c80905200807u7642e019j1ef54619fad162ea@mail.gmail.com>
References: <9457e7c80905200807u7642e019j1ef54619fad162ea@mail.gmail.com>
Message-ID: 

You can use @staticmethod/@classmethod decorators. But IIRC this only
work for normal 'def' methods, not for 'cdef' ones.

2009/5/20 St?fan van der Walt :
> Hi all,
>
> I read in the documentation that static methods can be defined as follows:
>
> cdef int MyClass_my_method(int x):
> ? ?return x
>
> cdef class MyClass:
> ? ?my_method = classmethod(MyClass_my_method)
>
> Now, I would like to define the corresponding header:
>
> cdef class MyClass:
> ? ?cdef my_method(int x)
>
> However, when I try to compile I get:
>
> "Self argument of C method does not match parent type."
>
> What is the correct way to address this problem?
>
> Regards
> St?fan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From stefan at sun.ac.za  Wed May 20 17:58:33 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Wed, 20 May 2009 17:58:33 +0200
Subject: [Cython] Static methods
In-Reply-To: 
References: <9457e7c80905200807u7642e019j1ef54619fad162ea@mail.gmail.com> 
	
Message-ID: <9457e7c80905200858p489633b7ud42f56532aa44f22@mail.gmail.com>

Hi Lisando

2009/5/20 Lisandro Dalcin :
> You can use @staticmethod/@classmethod decorators. But IIRC this only
> work for normal 'def' methods, not for 'cdef' ones.

Thanks for replying.

Here's my problem: I have to store a large number of data objects (>
250,000).  My current plan is to have a minimal data class (to save
memory), on which I can call several classmethod operators in rapid
succession.  As is, I can simply define all those "classmethods"
outside a class structure, but I thought it would be more elegant to
have them combined under one roof.

Regards
St?fan

From dagss at student.matnat.uio.no  Wed May 20 18:00:31 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 20 May 2009 18:00:31 +0200
Subject: [Cython] -unstable mergeback?
Message-ID: <4A14291F.2090806@student.matnat.uio.no>

It seems that there's now a danger of -unstable becoming the main 
development branch; I also think the dangerous things are completed 
myself. If so, I think it should replace the current -devel. This will 
make a few tickets blockers for the next release though.

Stefan has the best overview of the status I think?

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Wed May 20 18:12:18 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 20 May 2009 18:12:18 +0200
Subject: [Cython] Project: Buffer arithmetic operations
In-Reply-To: <4db580fd0905131341n43a00289lbb36b6c29da0306d@mail.gmail.com>
References: <4db580fd0905121314j142a30c7s915e4eb59d772b71@mail.gmail.com>
	<4A0A73D2.6070105@student.matnat.uio.no>	
	<4db580fd0905131253x4e18b782kc978b6894a5f7c5e@mail.gmail.com>
	<4db580fd0905131256s66327119s981ce9648c97ad63@mail.gmail.com>
	<1703a22623e40eb7bf68365fef4b4c91.squirrel@webmail.uio.no>
	<4db580fd0905131341n43a00289lbb36b6c29da0306d@mail.gmail.com>
Message-ID: <4A142BE2.6090303@student.matnat.uio.no>

Hoyt Koepke wrote:
> I would definitely like to continue trying to develop cython in an
> ongoing way, and my intent was to try to use the sage days coding
> sprint as a way to gain some familiarity with the cython code base.  I
> suggested the previous project in part because there are a lot of
> aspects that I could do on my own so I don't monopolize other people's
> time with questions.  In general, though, I am still balancing
> classes, prelims, and research, which is the only reason I haven't got
> more involved with cython til now, and I don't think things will let
> up until the fall.

I've now got a bit more time.

I don't know how things are going at Sage days; but if you're still up 
for it, and e.g. Robert has an hour for you later on, I could spend some 
time posting an outline for how I'd go about the (2) project and then 
perhaps Robert could mentor a bit from there.

I.e. the goal would be to allow for instance

cdef np.ndarray[int, ndim=3] arr1 = ..., arr2
for idx in cython.iterate(arr1, arr2):
     arr1[idx] = arr1[idx] + arr2[idx]

where idx is some "psuedo-tuple" of ndim length, and have it compile to 
a native loop.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Wed May 20 19:21:15 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 20 May 2009 19:21:15 +0200
Subject: [Cython] pyximport
In-Reply-To: 
References: 							<4A0F0942.3060307@behnel.de>	<311bf9168cc7ea937556a7391b54cdf1.squirrel@webmail.uio.no>					<186d467d97205ba211a488d0a1c1b7f0.squirrel@webmail.uio.no>
	
Message-ID: <4A143C0B.4040508@student.matnat.uio.no>

Robert wrote:
> .pyxbld/.pyxdep mechanism was already in 0.11.1 and long before 
> probably and allowed already to define a specific Extension class 
> (to tell about other .c files, dependencies etc).
> 
> I just added the setup_args, to allow crucial things like --debug, 
> --inplace, --compiler and all ... which one might have special in 
> pyximport mode too: its typically a debug situation.

Ahh ok. Speaks about how little I really know about pyximport.

>> One thing I was thinking about is embedding the information in the pyx
>> file itself. Something like this perhaps:
>>
>> from cython import pyxbuild
>> pyxbuild.include_path = [...]
>> pyxbuild.libraries = [...]
>>
>> .. rest of pyx file...
>>
>> What do you think?
> 
> 
> If the pyx meta importer must parse the .pyx source code (before 
> he can even check the dependencies), it is very slow.

You're likely right. The idea would still hold for something like 
special comments at the top of the pyx file or similar (only allowed on 
the first few lines).

> 
> Also including such special build info into the source file is a 
> questionable intermix.
> Consider, you give the source file to somebody else with different 
> environment ...

Well, in that case I'd say one should use a proper build system. Anyway, 
it seems like .pyxbld is here to stay so that would keep being an 
alternative.

All I'm looking for is a way to make the overhead of getting Cython code 
up and running lower than it is today.

Anyway, this needn't be discussed before somebody is ready to do it, 
which I'm not, now that I see that pyxbld is already there :-)

> Questioning its format:
> 
> It could be more *like* a setup.py. But it cannot not just be a 
> normal free one.
> Because: pyximport must inject/weave in the pyximport.install() 
> args, the .pyxdep,  must be able to tell separate default paths 
> (to not necessarily affect the regular build system). it must 
> direct/know location/take over the produced files; reload_support 
> mode, etc.
> 
> Thus pre-defining a single setup() call more or less (but not 
> executing directly) is perhaps the "simple case", the task of the 
> .pyxbld.
> 
> so I think a make_extension and make_setup_args inside the .pyxbld 
> is principally quite the right thing. sb who knows to write a 
> setup.py/setup(), will face quite the same format here - just 
> canalized.
> 
> Also, because its imported (and makeXX funktions are called), one 
> can still let execute *any* python code (before the build as of 
> now only). So its very flexible.
> 
> The current hierarchy of usage (and quite the evolution history):
> 1) pyximport.install() args - general behavior
> 2) .pyxdep - simple, only dependencies
> 3) .pyxbld/make_extension - other .c files...; can
>      extend/override/replace (2) and (1)
> 4) NEW .pyxbld/make_setup_args - alter higher options; can
>      extend/override/replace (3),(2),(1)
> 5) arbitrary effective python code in .pyxbld globally or
>     in the makeXX functions
> 
> I do not really have a better principal idea, and the mechanism is 
> flexibile and can be further evolved (e.g. a function called back 
> at compile/post build time or so). details could be straightened out.
> 
> 
> If one is concerned about not doubling build/dependency info:
> The .pyxbld file can be used/included right *from* a setup.py 
> script because of the compatibility, like:
> 
> #setup-cyex.py --------------------------------------
> 
> 
> from distutils.core import setup
> from Cython.Distutils import build_ext
> execfile("cyex.pyxbld")
> 
> setup(
>      cmdclass = {'build_ext': build_ext},
>      ext_modules = [make_ext('cyex','cyex.pyx')],
>      ##**make_setup_args()
> )
> 
> 
> #cyex.pyxbld ----------------------------------------
> def make_ext(modname, pyxfilename):
>      from distutils.extension import Extension
>      return Extension(name = modname,
>                       sources=[pyxfilename, 'hello.c'],
>                       depends=['hello.h'],
>                        )
> 
> def make_setup_args():
>      # args like for distutils Distribution/setup()
>      return dict(
>          script_args=[
>              ##"--compiler=mingw32",
>              ##"--verbose",
>              "--inplace",
>              "--debug",
>      ])
> -------------------------------------------------------
> 
> 
> thats maybe even natural to have the specific build/dep info for 
> each .pyx right next in the .pyxbld .  A (quite constant) setup.py 
> for the general distutils project integration...
> 
> 
> Robert
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


-- 
Dag Sverre

From dalcinl at gmail.com  Wed May 20 19:30:01 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 20 May 2009 14:30:01 -0300
Subject: [Cython] Static methods
In-Reply-To: <9457e7c80905200858p489633b7ud42f56532aa44f22@mail.gmail.com>
References: <9457e7c80905200807u7642e019j1ef54619fad162ea@mail.gmail.com>
	
	<9457e7c80905200858p489633b7ud42f56532aa44f22@mail.gmail.com>
Message-ID: 

2009/5/20 St?fan van der Walt :
> Hi Lisando
>
> 2009/5/20 Lisandro Dalcin :
>> You can use @staticmethod/@classmethod decorators. But IIRC this only
>> work for normal 'def' methods, not for 'cdef' ones.
>
> Thanks for replying.
>
> Here's my problem: I have to store a large number of data objects (>
> 250,000). ?My current plan is to have a minimal data class (to save
> memory), on which I can call several classmethod operators in rapid
> succession. ?As is, I can simply define all those "classmethods"
> outside a class structure, but I thought it would be more elegant to
> have them combined under one roof.

I also like your approach and agree on your rationale, but the problem
is that Cython does not currently support cdef
staticmethod/classmethod's :-)

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Wed May 20 20:33:08 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 20 May 2009 11:33:08 -0700
Subject: [Cython] ANN: Cython 0.11.2 released
In-Reply-To: <4A1425F9.9030100@student.matnat.uio.no>
References: <4A1425F9.9030100@student.matnat.uio.no>
Message-ID: <0C42799C-E7E3-4FD9-B2AB-FA86EB1FEAD5@math.washington.edu>

On May 20, 2009, at 8:47 AM, Dag Sverre Seljebotn wrote:

> I'm happy to announce the release of Cython 0.11.2.
>
> http://sage.math.washington.edu/home/dagss/Cython-0.11.2.tar.gz
> http://sage.math.washington.edu/home/dagss/Cython-0.11.2.zip
>
> (Will be present on the Cython front page in a few days.)

Done.

http://cython.org/Cython-0.11.2.tar.gz
http://cython.org/Cython-0.11.2.zip

Also up at http://pypi.python.org/pypi/Cython/0.11.2

>
> New features:
>
>   * There's now native complex floating point support! C99 complex  
> will
> be used if complex.h is included, otherwise explicit complex  
> arithmetic
> working on all C compilers is used. [Robert Bradshaw]
>
>     cdef double complex a = 1 + 0.3j
>     cdef np.ndarray[np.complex128_t, ndim=2] arr = \
>         np.zeros(10, np.complex128)
>
>   * Cython can now generate a main()-method for embedding of the  
> Python
> interpreter into an executable (see #289) [Robert Bradshaw]
>
>   * @wraparound directive (another way to disable arr[idx] for  
> negative
> idx) [Dag Sverre Seljebotn]
>
>   * Correct support for NumPy record dtypes with different alignments,
> and "cdef packed struct" support [Dag Sverre Seljebotn]
>
>   * @callspec directive, allowing custom calling convention macros
> [Lisandro Dalcin]
>
>   * Bug fixes and smaller improvements. For the full list, see [1].
>
> Contributors to this release:
>   - Stefan Behnel
>   - Robert Bradshaw
>   - Lisandro Dalcin
>   - Dag Sverre Seljebotn
>
> Thanks also to everybody who's helping us out in our discussions on  
> the
> mailing list.
>
> [1]
>  group=component&milestone=0.11.2&resolution=fixed>
>
> -- 
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From robertwb at math.washington.edu  Wed May 20 20:38:53 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 20 May 2009 11:38:53 -0700
Subject: [Cython] -unstable mergeback?
In-Reply-To: <4A14291F.2090806@student.matnat.uio.no>
References: <4A14291F.2090806@student.matnat.uio.no>
Message-ID: 

On May 20, 2009, at 9:00 AM, Dag Sverre Seljebotn wrote:

> It seems that there's now a danger of -unstable becoming the main
> development branch; I also think the dangerous things are completed
> myself. If so, I think it should replace the current -devel. This will
> make a few tickets blockers for the next release though.

If -unstable has stabilized, I think this makes sense. I envisioned - 
unstable as being for truly unstable things (e.g. closures, temp  
allocation rewrites) rather than the main development branch.

- Robert


From robertwb at math.washington.edu  Wed May 20 20:41:42 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 20 May 2009 11:41:42 -0700
Subject: [Cython] Note on C99 compound literals
In-Reply-To: <4A13FC60.50001@student.matnat.uio.no>
References: <4A13FC60.50001@student.matnat.uio.no>
Message-ID: <270275DC-E839-4FB8-B6A7-2CB538DACF55@math.washington.edu>

On May 20, 2009, at 5:49 AM, Dag Sverre Seljebotn wrote:

> I did some digging and decided to share what I found, since this only
> occurs on a specific compiler and is thus hard to discover.
>
> Kurt, pay attention, as I just recommended that you do this :-)
>
> Apparently code like this:
>
>   (foo_struct){0, 0, NULL}
>
> is a C99 extension, so we probably shouldn't use it. Furthermore it
> makes things fail in g++ 4.2.4 (but not in earlier or later versions I
> tried -- anyway, 4.2.4 is the one currently on sage.math).

I think it's fine in C (not sure if it's just gcc), but has issues  
with C++. We ran into this issue before with cdef optional arguments.

> (The reason it fails is largely due to us jumping out of the scope
> everywhere on exceptions)
>
> What I did for complex numbers was replace their use with an inline
> function returning a struct with the given values. Don't know if this
> can hit performance -- if the compiler inlines the function I wouldn't
> think it would make a difference.


Yep, this is why I did it this way (well, that and I wanted needed to  
be able to provide a common interface to c99 complexes)--hopefully  
it's exactly the same.

- Robert


From dagss at student.matnat.uio.no  Wed May 20 20:44:48 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 20 May 2009 20:44:48 +0200
Subject: [Cython] Note on C99 compound literals
In-Reply-To: <270275DC-E839-4FB8-B6A7-2CB538DACF55@math.washington.edu>
References: <4A13FC60.50001@student.matnat.uio.no>
	<270275DC-E839-4FB8-B6A7-2CB538DACF55@math.washington.edu>
Message-ID: <4A144FA0.4020704@student.matnat.uio.no>

Robert Bradshaw wrote:
> On May 20, 2009, at 5:49 AM, Dag Sverre Seljebotn wrote:
> 
>> I did some digging and decided to share what I found, since this only
>> occurs on a specific compiler and is thus hard to discover.
>>
>> Kurt, pay attention, as I just recommended that you do this :-)
>>
>> Apparently code like this:
>>
>>   (foo_struct){0, 0, NULL}
>>
>> is a C99 extension, so we probably shouldn't use it. Furthermore it
>> makes things fail in g++ 4.2.4 (but not in earlier or later versions I
>> tried -- anyway, 4.2.4 is the one currently on sage.math).
> 
> I think it's fine in C (not sure if it's just gcc), but has issues  
> with C++. We ran into this issue before with cdef optional arguments.

The link I posted lists it as a C99 extension though:

"""
As an extension, GCC supports compound literals in C89 mode and in C++.
"""

http://gcc.gnu.org/onlinedocs/gcc-3.3.1/gcc/Compound-Literals.html


-- 
Dag Sverre

From sccolbert at gmail.com  Wed May 20 21:46:03 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Wed, 20 May 2009 15:46:03 -0400
Subject: [Cython] trouble splitting extension type into definition and
	implementation files
Message-ID: <7f014ea60905201246m7ddea54uea53ca68cde98122@mail.gmail.com>

Hi,

I have this definition file:

#### cy_cvtypes.pxd #####

cimport c_cxcore
cimport c_highgui

cdef class IplImage:
    cdef c_cxcore.IplImage* thisptr
    cdef int needs_free

###########################


and the corresponding implementation:

#### cy_cvtypes.pxy ######

cimport c_cxcore
cimport c_highgui

cdef class IplImage:

    def __cinit__(self, needs_free=1):
    # sets the flag which determines whether or not to free the image
resource during
    # garbage collection. Make sure to set to zero if your not supposed to
release the
    # the image.
        self.needs_free = needs_free

    def __dealloc__(self):
        if self.needs_free:
            c_cxcore.cvReleaseImage(&self.thisptr)

    #Convienience Functions

    def show(self):
    # a convienience function which displays the image in a new window
    # useful for debugging. window exits on keypress.
        c_highgui.cvNamedWindow('test', 1)
        c_highgui.cvShowImage('test', self.thisptr)
        c_highgui.cvWaitKey(0)
        c_highgui.cvDestroyWindow('test')

#############################################

c_cxcore and c_highgui define the types and funtions from the respective
header files


when compiling, I get complaints about not being able to convert python
objects.

example:

Error converting Pyrex file to C:
------------------------------------------------------------
...

    def show(self):
    # a convienience function which displays the image in a new window
    # useful for debugging. window exits on keypress.
        c_highgui.cvNamedWindow('test', 1)
        c_highgui.cvShowImage('test', self.thisptr)
                                         ^
------------------------------------------------------------

F:\Desktop\cythonopencv\cy_cvtypes.pyx:22:42: Cannot convert Python object
to 'c
_cxcore.CvArr *'




It seems that the cython compiler is not reading the cy_cvtype.pxd defintion
file.



Also, I am attempting to compile more than one .pyx source file into the
same module by adding multiple .pyx files to the sources [] in  setup.py.
Could that be the source of the problem? i.e. to I have to compile a
separate .pyd module for every .pyx file?

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090520/93b65178/attachment-0001.htm 

From sccolbert at gmail.com  Wed May 20 21:49:46 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Wed, 20 May 2009 15:49:46 -0400
Subject: [Cython] trouble splitting extension type into definition and
	implementation files
In-Reply-To: <7f014ea60905201246m7ddea54uea53ca68cde98122@mail.gmail.com>
References: <7f014ea60905201246m7ddea54uea53ca68cde98122@mail.gmail.com>
Message-ID: <7f014ea60905201249g74e0cd6k926da0d126d9a64b@mail.gmail.com>

Ok, so compiling each .pyx file into an individual .pyd worked.

Is there a way to properly compile everything into a single .pyd, or should
I create a dummy module at the end that imports each individual module when
called from python?

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090520/cf3ddadf/attachment.htm 

From robertwb at math.washington.edu  Wed May 20 23:47:46 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 20 May 2009 14:47:46 -0700
Subject: [Cython] Special methods and __hash__
Message-ID: 

According to the Python/C API, returning -1 from __hash__ is the way  
to signal an error. Of course, if one returns -1 manually one gets a  
runtime error that an error was raised without setting the exception.  
We could document this, but it's still kind of obscure.

In Python one can do

sage: class A:
....:     def __hash__(self):
....:         return int(-1)
....:
sage: hash(A())
-2

Should we just check for -1 and return -2 for cdef class __hash__  
functions as well?

- Robert


From dalcinl at gmail.com  Thu May 21 00:04:18 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 20 May 2009 19:04:18 -0300
Subject: [Cython] Special methods and __hash__
In-Reply-To: 
References: 
Message-ID: 

On Wed, May 20, 2009 at 6:47 PM, Robert Bradshaw
 wrote:
> According to the Python/C API, returning -1 from __hash__ is the way
> to signal an error. Of course, if one returns -1 manually one gets a
> runtime error that an error was raised without setting the exception.
> We could document this, but it's still kind of obscure.
>
> In Python one can do
>
> sage: class A:
> ....: ? ? def __hash__(self):
> ....: ? ? ? ? return int(-1)
> ....:
> sage: hash(A())
> -2
>
> Should we just check for -1 and return -2 for cdef class __hash__
> functions as well?
>

IMHO, yes. Cython should do something similar to what core CPyhton
does in typeobject.c... see this

static long
slot_tp_hash(PyObject *self)
{
	PyObject *func;
	static PyObject *hash_str, *eq_str, *cmp_str;
	long h;

	func = lookup_method(self, "__hash__", &hash_str);
<.....>
	if (h == -1 && !PyErr_Occurred())
		h = -2;
	return h;
}


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From greg.ewing at canterbury.ac.nz  Thu May 21 01:08:29 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 21 May 2009 11:08:29 +1200
Subject: [Cython] help....cython code optimisation
In-Reply-To: 
References: 
Message-ID: <4A148D6D.1000509@canterbury.ac.nz>

OUARDA MEHDI wrote:

> class LinearConfigurationSpaceJoint(Joint):
...
> cdef class RzRyRxJoint(LinearConfigurationSpaceJoint):

Unless Cython has enhanced something here, you can't
do that. At least in Pyrex, an extension type can
only inherit from another built-in or extension
type, not a Python class.

Since you're defining the base class in Cython,
there's no reason not to make it an extension type
as well.

-- 
Greg

From hoytak at cs.ubc.ca  Thu May 21 01:22:53 2009
From: hoytak at cs.ubc.ca (Hoyt Koepke)
Date: Wed, 20 May 2009 16:22:53 -0700
Subject: [Cython] Project: Buffer arithmetic operations
In-Reply-To: <4A142BE2.6090303@student.matnat.uio.no>
References: <4db580fd0905121314j142a30c7s915e4eb59d772b71@mail.gmail.com> 
	<4A0A73D2.6070105@student.matnat.uio.no>
	 
	<4db580fd0905131253x4e18b782kc978b6894a5f7c5e@mail.gmail.com> 
	<4db580fd0905131256s66327119s981ce9648c97ad63@mail.gmail.com> 
	<1703a22623e40eb7bf68365fef4b4c91.squirrel@webmail.uio.no> 
	<4db580fd0905131341n43a00289lbb36b6c29da0306d@mail.gmail.com> 
	<4A142BE2.6090303@student.matnat.uio.no>
Message-ID: <4db580fd0905201622u778d45b8u43048c81daf93fb4@mail.gmail.com>

Hi,

> I've now got a bit more time.
>
> I don't know how things are going at Sage days; but if you're still up
> for it, and e.g. Robert has an hour for you later on, I could spend some
> time posting an outline for how I'd go about the (2) project and then
> perhaps Robert could mentor a bit from there.
>
> I.e. the goal would be to allow for instance
>
> cdef np.ndarray[int, ndim=3] arr1 = ..., arr2
> for idx in cython.iterate(arr1, arr2):
> ? ? arr1[idx] = arr1[idx] + arr2[idx]
>
> where idx is some "psuedo-tuple" of ndim length, and have it compile to
> a native loop.

I have been swamped the past week trying to get a research project
back on track, so unfortunately I haven't been able to take advantage
of sage days here :-(.  Bummer.  However, I still want to help out
with this and take up your mentoring offer and work on this.  To be honest,
I probably won't have a significant chunk of time, though, until after June 5.

I'm wondering, though, if it would be better to do this in two stages.
 It seems like having a type, say index_tuple, which the user can
define and then manipulate.  It seems we would be doing this anyway,
and explicitly exposing it to the user could have use cases.  In the C
code, it seems like it'd just be an int array.

e.g.

cdef size_t i
cdef index_tuple[2] idx
cdef ndarray[double, ndim=2] A

idx[0] = 0

for 0 <= i < A.shape[1]:
    idx[1] = i
    A[idx] = 1


Obviously this is a simple example, but there might easily be other
use cases, such as passing indices to a function, etc..

What do you think?

--Hoyt

++++++++++++++++++++++++++++++++++++++++++++++++
+ Hoyt Koepke
+ University of Washington Department of Statistics
+ http://www.stat.washington.edu/~hoytak/
+ hoytak at gmail.com
++++++++++++++++++++++++++++++++++++++++++

From robertwb at math.washington.edu  Thu May 21 01:36:22 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 20 May 2009 16:36:22 -0700
Subject: [Cython] help....cython code optimisation
In-Reply-To: <4A148D6D.1000509@canterbury.ac.nz>
References: 
	<4A148D6D.1000509@canterbury.ac.nz>
Message-ID: 

On May 20, 2009, at 4:08 PM, Greg Ewing wrote:

> OUARDA MEHDI wrote:
>
>> class LinearConfigurationSpaceJoint(Joint):
> ...
>> cdef class RzRyRxJoint(LinearConfigurationSpaceJoint):
>
> Unless Cython has enhanced something here, you can't
> do that.

Nope, that would be quite the feat.

> At least in Pyrex, an extension type can
> only inherit from another built-in or extension
> type, not a Python class.
>
> Since you're defining the base class in Cython,
> there's no reason not to make it an extension type
> as well.

+1

- Robert


From greg.ewing at canterbury.ac.nz  Thu May 21 02:15:22 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 21 May 2009 12:15:22 +1200
Subject: [Cython] Static methods
In-Reply-To: <9457e7c80905200858p489633b7ud42f56532aa44f22@mail.gmail.com>
References: <9457e7c80905200807u7642e019j1ef54619fad162ea@mail.gmail.com>
	
	<9457e7c80905200858p489633b7ud42f56532aa44f22@mail.gmail.com>
Message-ID: <4A149D1A.3090505@canterbury.ac.nz>

St?fan van der Walt wrote:
> As is, I can simply define all those "classmethods"
> outside a class structure, but I thought it would be more elegant to
> have them combined under one roof.

Is there some reason that a module is not a
suitable kind of "roof" to put them under?

-- 
Greg

From greg.ewing at canterbury.ac.nz  Thu May 21 02:35:58 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 21 May 2009 12:35:58 +1200
Subject: [Cython] trouble splitting extension type into definition and
 implementation files
In-Reply-To: <7f014ea60905201249g74e0cd6k926da0d126d9a64b@mail.gmail.com>
References: <7f014ea60905201246m7ddea54uea53ca68cde98122@mail.gmail.com>
	<7f014ea60905201249g74e0cd6k926da0d126d9a64b@mail.gmail.com>
Message-ID: <4A14A1EE.2090700@canterbury.ac.nz>

Chris Colbert wrote:

> Is there a way to properly compile everything into a single .pyd,

You can use include statements to split a .pyx into
multiple files, but I wouldn't recommend that, because
it makes it hard for the reader to tell where things
are defined.

Also, separate modules means there's less to recompile
when you change one of the implementation files, and
it seems that compiling a number of small C files can
sometimes be faster than compiling a single large C
file of the same total size.

-- 
Greg

From dalcinl at gmail.com  Thu May 21 02:57:40 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 20 May 2009 21:57:40 -0300
Subject: [Cython] Static methods
In-Reply-To: <4A149D1A.3090505@canterbury.ac.nz>
References: <9457e7c80905200807u7642e019j1ef54619fad162ea@mail.gmail.com>
	
	<9457e7c80905200858p489633b7ud42f56532aa44f22@mail.gmail.com>
	<4A149D1A.3090505@canterbury.ac.nz>
Message-ID: 

On Wed, May 20, 2009 at 9:15 PM, Greg Ewing  wrote:
> St?fan van der Walt wrote:
>> As is, I can simply define all those "classmethods"
>> outside a class structure, but I thought it would be more elegant to
>> have them combined under one roof.
>
> Is there some reason that a module is not a
> suitable kind of "roof" to put them under?
>

IMHO, factory methods are good candidates for classmethod's. And of
course, if you are trying to emulate a C++ API for using in Python (as
I do in mpi4py), you may be "forced" to use {static|class}method's,
just because you want to mimic the C++ API as much as possible.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From lodatom at gmail.com  Thu May 21 03:43:17 2009
From: lodatom at gmail.com (Mark Lodato)
Date: Wed, 20 May 2009 21:43:17 -0400
Subject: [Cython] Cython just to avoid coding C?
In-Reply-To: <3136DF59-DAE3-4279-B3FD-3EB357A3B9E2@math.washington.edu>
References: 
	<4905F089-AC21-4C61-A54E-F6BAF813F828@math.washington.edu>
	
	
	
	
	
	
	
	<3136DF59-DAE3-4279-B3FD-3EB357A3B9E2@math.washington.edu>
Message-ID: 

On Thu, May 14, 2009 at 5:08 AM, Robert Bradshaw
 wrote:
> On Apr 22, 2009, at 7:18 PM, Mark Lodato wrote:
>> An additional idea (not implemented) - it would be awesome to have the
>> first module listed (that is, the one directly imported) to have
>> __name__ == "__main__", so the normal Python idiom works. ?I
>> researched this for the past few hours, but I could not find a way to
>> do this without modifying the module's code. ?My best solution is the
>> following:
>> 1. For each module, mymod.pyx:
>> 1a. Add a new externally-visible global variable:
>> ? ? int __pyx_mymod_is___main__ = 0;
>> 1b. When converting the code for __name__, if __pyx_mymod_is___main__,
>> return "__main__", else the module name.
>> 2. In the script below, set
>> ? ? __pyx_mainmod_is___main__ = 1;
>> right before the call to PyImport_ImportModule(), where "mainmod" is
>> the main module.
>
> Done. The flag is __pyx_module_is_main_[fullyqualifiedname.replace
> ('.', '__')]


Awesome - thanks!

I updated my cython_freeze.py script and uploaded it to GitHub:
http://github.com/MarkLodato/cython_freeze

It now uses the __pyx_module_is_main_* flag to set the main module, so
it requires Cython 0.11.2.

I also included detailed documentation and a simple example.

If you all have any suggestions, I would be happy to implement them.
It would be nice to include this in some form as part of the Cython
distribution.

~Mark

From kwmsmith at gmail.com  Thu May 21 06:07:46 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Wed, 20 May 2009 23:07:46 -0500
Subject: [Cython] Note on C99 compound literals
In-Reply-To: <4A144FA0.4020704@student.matnat.uio.no>
References: <4A13FC60.50001@student.matnat.uio.no>
	<270275DC-E839-4FB8-B6A7-2CB538DACF55@math.washington.edu>
	<4A144FA0.4020704@student.matnat.uio.no>
Message-ID: 

On Wed, May 20, 2009 at 1:44 PM, Dag Sverre Seljebotn
 wrote:
> Robert Bradshaw wrote:
>> On May 20, 2009, at 5:49 AM, Dag Sverre Seljebotn wrote:
>>
>>> I did some digging and decided to share what I found, since this only
>>> occurs on a specific compiler and is thus hard to discover.
>>>
>>> Kurt, pay attention, as I just recommended that you do this :-)
>>>
>>> Apparently code like this:
>>>
>>> ? (foo_struct){0, 0, NULL}
>>>
>>> is a C99 extension, so we probably shouldn't use it. Furthermore it
>>> makes things fail in g++ 4.2.4 (but not in earlier or later versions I
>>> tried -- anyway, 4.2.4 is the one currently on sage.math).
>>
>> I think it's fine in C (not sure if it's just gcc), but has issues
>> with C++. We ran into this issue before with cdef optional arguments.
>
> The link I posted lists it as a C99 extension though:
>
> """
> As an extension, GCC supports compound literals in C89 mode and in C++.
> """
>
> http://gcc.gnu.org/onlinedocs/gcc-3.3.1/gcc/Compound-Literals.html

So was a verdict reached?  Can compound literals be used?  (They'd be
a mote easier for the coercion I'm working on in #299, but one can
always just use an inline function returning a struct.)

Kurt

From dagss at student.matnat.uio.no  Thu May 21 07:10:09 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 21 May 2009 07:10:09 +0200 (CEST)
Subject: [Cython] Special methods and __hash__
In-Reply-To: 
References: 
Message-ID: <1074aefe36699e13d6cf77c04525dcb8.squirrel@webmail.uio.no>

Robert wrote:
> According to the Python/C API, returning -1 from __hash__ is the way
> to signal an error. Of course, if one returns -1 manually one gets a
> runtime error that an error was raised without setting the exception.
> We could document this, but it's still kind of obscure.
>
> In Python one can do
>
> sage: class A:
> ....:     def __hash__(self):
> ....:         return int(-1)
> ....:
> sage: hash(A())
> -2
>
> Should we just check for -1 and return -2 for cdef class __hash__
> functions as well?

+1, but IMO it should wait until the rest of the operators have been
discussed for alignment with Python as well.

For instance I'd like to propose:
- Introduce "__cadd__" to take over for the current "__add__"
- Create a script which can convert source files (e.g. in Sage) and do
this change automatically (since the change below will slow things down)
- Make __cadd__ by default do something like this:

cdef class MyClass:
    def __cadd__(a, b):
        # body inserted if __add__ is implemented
        if isinstance(a, MyClass):
            return a.__add__(a, b)
        else:
            return b.__add__(b, a)

Dag Sverre


From dagss at student.matnat.uio.no  Thu May 21 07:17:10 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 21 May 2009 07:17:10 +0200 (CEST)
Subject: [Cython] Note on C99 compound literals
In-Reply-To: 
References: <4A13FC60.50001@student.matnat.uio.no>
	<270275DC-E839-4FB8-B6A7-2CB538DACF55@math.washington.edu>
	<4A144FA0.4020704@student.matnat.uio.no>
	
Message-ID: 

Kurt wrote:
> On Wed, May 20, 2009 at 1:44 PM, Dag Sverre Seljebotn
>  wrote:
>> Robert Bradshaw wrote:
>>> On May 20, 2009, at 5:49 AM, Dag Sverre Seljebotn wrote:
>>>
>>>> I did some digging and decided to share what I found, since this only
>>>> occurs on a specific compiler and is thus hard to discover.
>>>>
>>>> Kurt, pay attention, as I just recommended that you do this :-)
>>>>
>>>> Apparently code like this:
>>>>
>>>> ? (foo_struct){0, 0, NULL}
>>>>
>>>> is a C99 extension, so we probably shouldn't use it. Furthermore it
>>>> makes things fail in g++ 4.2.4 (but not in earlier or later versions I
>>>> tried -- anyway, 4.2.4 is the one currently on sage.math).
>>>
>>> I think it's fine in C (not sure if it's just gcc), but has issues
>>> with C++. We ran into this issue before with cdef optional arguments.
>>
>> The link I posted lists it as a C99 extension though:
>>
>> """
>> As an extension, GCC supports compound literals in C89 mode and in C++.
>> """
>>
>> http://gcc.gnu.org/onlinedocs/gcc-3.3.1/gcc/Compound-Literals.html
>
> So was a verdict reached?  Can compound literals be used?  (They'd be
> a mote easier for the coercion I'm working on in #299, but one can
> always just use an inline function returning a struct.)

Note that if you create a constructor function like this:

static INLINE MyStruct create_MyStruct(int a, int b) {
    MyStruct r; r.a = a; r.b = b; return r;
}

then it shouldn't make much difference IMO. (You can create such a
function using utility code.)

(Judging by how my previous attempt to use a feature which was shown to be
C99 only was recieved, I have a feeling this pretty much settles it.)

Dag Sverre


From kxroberto at googlemail.com  Thu May 21 08:19:34 2009
From: kxroberto at googlemail.com (Robert)
Date: Thu, 21 May 2009 08:19:34 +0200
Subject: [Cython] pyximport
In-Reply-To: <4A143C0B.4040508@student.matnat.uio.no>
References: 							<4A0F0942.3060307@behnel.de>	<311bf9168cc7ea937556a7391b54cdf1.squirrel@webmail.uio.no>					<186d467d97205ba211a488d0a1c1b7f0.squirrel@webmail.uio.no>	
	<4A143C0B.4040508@student.matnat.uio.no>
Message-ID: 

Dag Sverre Seljebotn wrote:
> Robert wrote:
>> If the pyx meta importer must parse the .pyx source code (before 
>> he can even check the dependencies), it is very slow.
> 
> You're likely right. The idea would still hold for something like 
> special comments at the top of the pyx file or similar (only allowed on 
> the first few lines).

>> Also including such special build info into the source file is a 
>> questionable intermix.
>> Consider, you give the source file to somebody else with different 
>> environment ...
> 
> Well, in that case I'd say one should use a proper build system. Anyway, 
> it seems like .pyxbld is here to stay so that would keep being an 
> alternative.
> 
> All I'm looking for is a way to make the overhead of getting Cython code 
> up and running lower than it is today.
> 
> Anyway, this needn't be discussed before somebody is ready to do it, 
> which I'm not, now that I see that pyxbld is already there :-)

yes, special comments (a new language) would also require the 
person to lookup the docs for an example. And there he can as well 
read about .pyxbld/def make_ext().." .. Perhaps just a matter of 
convention and a good doc string

> 
>> Questioning its format:
>>
>> It could be more *like* a setup.py. But it cannot not just be a 
>> normal free one.... Thus pre-defining a single setup() call more or less (but not
 >> ...
>> If one is concerned about not doubling build/dependency info:
>> The .pyxbld file can be used/included right *from* a setup.py 
>> script because of the compatibility, like:
>>
>> #setup-cyex.py --------------------------------------
>>
>>
>> from distutils.core import setup
>> from Cython.Distutils import build_ext
>> execfile("cyex.pyxbld")
>>
>> setup(
>>      cmdclass = {'build_ext': build_ext},
>>      ext_modules = [make_ext('cyex','cyex.pyx')],
>>      ##**make_setup_args()
>> )
>>
>>
>> #cyex.pyxbld ----------------------------------------
>> def make_ext(modname, pyxfilename):
>>      from distutils.extension import Extension
>>      return Extension(name = modname,
>>                       sources=[pyxfilename, 'hello.c'],
>>                       depends=['hello.h'],
>>                        )
>>
>> def make_setup_args():
>>      # args like for distutils Distribution/setup()
>>      return dict(
>>          script_args=[
>>              ##"--compiler=mingw32",
>>              ##"--verbose",
>>              "--inplace",
>>              "--debug",
>>      ])
>> -------------------------------------------------------


or: the cyex.pyxbld (with if '__main__'...setup(...) in it) could 
become the setup script by Cython convention:

python cyex.pyxbld build_ext

It would then be at the same time, as imported module, the 
definition file for pyximport.

-------------------------

or: the naked cyex.pyxbld (quite naked: not existing / nothing in 
it / or the make_ext and optionally make_setup_args)
and:

cython cyex.pyx  => translates to .c

cython cyex.pyxbld   =>  build_ext's the cyex.so/.pyd; or even 
auto-creates the .pyxbld, if not existing

cython -b cyex.pyx => translates AND builds the DLL; when no 
cyex.pyxbld, then default settings

maybe thats quite intuitive? and no need for writing full setup 
script infrastructure for any/each module before really 
distributing something?

the thing is, that .pyx is about only ONE DLL anyway.
And as one has typically anyway more .pyx'es in a folder, then
the one and only good old "setup.py" could just integrate all the 
.pyx/.pyxbld   => "the project/package" .

##### setup.py ##################
from distutils.core import setup
from Cython.Distutils import build_ext
extensions = build_ext.collect("cyex.pyx","nextex.pyx",...)
setup(
     ext_modules = extensions,
     )
#########################

and the .pyxbld's with details do not need to exist until extra .c 
files..



Robert







From robertwb at math.washington.edu  Thu May 21 08:49:08 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 20 May 2009 23:49:08 -0700
Subject: [Cython] Special methods and __hash__
In-Reply-To: <1074aefe36699e13d6cf77c04525dcb8.squirrel@webmail.uio.no>
References: 
	<1074aefe36699e13d6cf77c04525dcb8.squirrel@webmail.uio.no>
Message-ID: <629C160E-2834-4F50-B41C-BDD74786A82A@math.washington.edu>

On May 20, 2009, at 10:10 PM, Dag Sverre Seljebotn wrote:

> Robert wrote:
>> According to the Python/C API, returning -1 from __hash__ is the way
>> to signal an error. Of course, if one returns -1 manually one gets a
>> runtime error that an error was raised without setting the exception.
>> We could document this, but it's still kind of obscure.
>>
>> In Python one can do
>>
>> sage: class A:
>> ....:     def __hash__(self):
>> ....:         return int(-1)
>> ....:
>> sage: hash(A())
>> -2
>>
>> Should we just check for -1 and return -2 for cdef class __hash__
>> functions as well?
>
> +1, but IMO it should wait until the rest of the operators have been
> discussed for alignment with Python as well.
>
> For instance I'd like to propose:
> - Introduce "__cadd__" to take over for the current "__add__"
> - Create a script which can convert source files (e.g. in Sage) and do
> this change automatically (since the change below will slow things  
> down)
> - Make __cadd__ by default do something like this:
>
> cdef class MyClass:
>     def __cadd__(a, b):
>         # body inserted if __add__ is implemented
>         if isinstance(a, MyClass):
>             return a.__add__(a, b)
>         else:
>             return b.__add__(b, a)

This is a much more invasive change--I think the hash function is  
safer because it turns what was a certain runtime error into  
something reasonable.

- Robert


From dagss at student.matnat.uio.no  Thu May 21 10:10:53 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 21 May 2009 10:10:53 +0200
Subject: [Cython] Project: Buffer arithmetic operations
In-Reply-To: <4db580fd0905201622u778d45b8u43048c81daf93fb4@mail.gmail.com>
References: <4db580fd0905121314j142a30c7s915e4eb59d772b71@mail.gmail.com>
	<4A0A73D2.6070105@student.matnat.uio.no>	
	<4db580fd0905131253x4e18b782kc978b6894a5f7c5e@mail.gmail.com>
	<4db580fd0905131256s66327119s981ce9648c97ad63@mail.gmail.com>
	<1703a22623e40eb7bf68365fef4b4c91.squirrel@webmail.uio.no>
	<4db580fd0905131341n43a00289lbb36b6c29da0306d@mail.gmail.com>
	<4A142BE2.6090303@student.matnat.uio.no>
	<4db580fd0905201622u778d45b8u43048c81daf93fb4@mail.gmail.com>
Message-ID: <4A150C8D.3090709@student.matnat.uio.no>

Hoyt Koepke wrote:
> Hi,
> 
>> I've now got a bit more time.
>>
>> I don't know how things are going at Sage days; but if you're still up
>> for it, and e.g. Robert has an hour for you later on, I could spend some
>> time posting an outline for how I'd go about the (2) project and then
>> perhaps Robert could mentor a bit from there.
>>
>> I.e. the goal would be to allow for instance
>>
>> cdef np.ndarray[int, ndim=3] arr1 = ..., arr2
>> for idx in cython.iterate(arr1, arr2):
>>     arr1[idx] = arr1[idx] + arr2[idx]
>>
>> where idx is some "psuedo-tuple" of ndim length, and have it compile to
>> a native loop.
> 
> I have been swamped the past week trying to get a research project
> back on track, so unfortunately I haven't been able to take advantage
> of sage days here :-(.  Bummer.  However, I still want to help out
> with this and take up your mentoring offer and work on this.  To be honest,
> I probably won't have a significant chunk of time, though, until after June 5.

Sure, just ask again when you do.

> I'm wondering, though, if it would be better to do this in two stages.
>  It seems like having a type, say index_tuple, which the user can
> define and then manipulate.  It seems we would be doing this anyway,
> and explicitly exposing it to the user could have use cases.  In the C
> code, it seems like it'd just be an int array.

I was actually thinking it would be slightly more magic than this (for 
speed) and that "arr[idx]" would translate to "*head", where head was a 
pointer in the innermost loop:

while (not at end) {
     ...iterate n-1 outermost dimension.., set head and rowend
     while (++head != rowend) { // iterator innermost, perhaps contiguous
         BODY
     }
}

however, if "idx" was used, one would add updating an idx array to this 
scheme (but it would only be used for idx and idx[dim], not 
arr[idx]).


> cdef size_t i
> cdef index_tuple[2] idx
> cdef ndarray[double, ndim=2] A
> 
> idx[0] = 0
> 
> for 0 <= i < A.shape[1]:
>     idx[1] = i
>     A[idx] = 1
> 
> 
> Obviously this is a simple example, but there might easily be other
> use cases, such as passing indices to a function, etc..
> 
> What do you think?

Sounds ok to me, and definitely would be a stepping stone for the above. 
I'd have another look at the syntax and make it more generic though -- 
some proposals:

cdef tuple[Py_ssize_t, len=2] idx

cdef (Py_ssize_t, Py_ssize_t) idx

I'm actually leaning towards the latter, although it would be an 
entirely new construct. The reason is that it could be used as a syntax 
to support multiple return values:

cdef (int, float) hello(): return (3, 3.14)

Perhaps even

cdef (Py_ssize_t,)*4 idx

:-)

-- 
Dag Sverre

From robertwb at math.washington.edu  Thu May 21 10:28:42 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 21 May 2009 01:28:42 -0700
Subject: [Cython] Note on C99 compound literals
In-Reply-To: 
References: <4A13FC60.50001@student.matnat.uio.no>
	<270275DC-E839-4FB8-B6A7-2CB538DACF55@math.washington.edu>
	<4A144FA0.4020704@student.matnat.uio.no>
	
	
Message-ID: <308A5EFB-A23F-46D1-8C10-97435CE58DBB@math.washington.edu>

On May 20, 2009, at 10:17 PM, Dag Sverre Seljebotn wrote:

> Kurt wrote:
>> On Wed, May 20, 2009 at 1:44 PM, Dag Sverre Seljebotn
>>  wrote:
>>> Robert Bradshaw wrote:
>>>> On May 20, 2009, at 5:49 AM, Dag Sverre Seljebotn wrote:
>>>>
>>>>> I did some digging and decided to share what I found, since  
>>>>> this only
>>>>> occurs on a specific compiler and is thus hard to discover.
>>>>>
>>>>> Kurt, pay attention, as I just recommended that you do this :-)
>>>>>
>>>>> Apparently code like this:
>>>>>
>>>>>   (foo_struct){0, 0, NULL}
>>>>>
>>>>> is a C99 extension, so we probably shouldn't use it.  
>>>>> Furthermore it
>>>>> makes things fail in g++ 4.2.4 (but not in earlier or later  
>>>>> versions I
>>>>> tried -- anyway, 4.2.4 is the one currently on sage.math).
>>>>
>>>> I think it's fine in C (not sure if it's just gcc), but has issues
>>>> with C++. We ran into this issue before with cdef optional  
>>>> arguments.
>>>
>>> The link I posted lists it as a C99 extension though:
>>>
>>> """
>>> As an extension, GCC supports compound literals in C89 mode and  
>>> in C++.
>>> """
>>>
>>> http://gcc.gnu.org/onlinedocs/gcc-3.3.1/gcc/Compound-Literals.html
>>
>> So was a verdict reached?  Can compound literals be used?  (They'd be
>> a mote easier for the coercion I'm working on in #299, but one can
>> always just use an inline function returning a struct.)
>
> Note that if you create a constructor function like this:
>
> static INLINE MyStruct create_MyStruct(int a, int b) {
>     MyStruct r; r.a = a; r.b = b; return r;
> }
>
> then it shouldn't make much difference IMO. (You can create such a
> function using utility code.)
>
> (Judging by how my previous attempt to use a feature which was  
> shown to be
> C99 only was recieved, I have a feeling this pretty much settles it.)


I'd like to not require much more than Python itself requires.

- Robert


From kxroberto at googlemail.com  Thu May 21 12:07:26 2009
From: kxroberto at googlemail.com (Robert)
Date: Thu, 21 May 2009 12:07:26 +0200
Subject: [Cython] cdef extern from "" ?
Message-ID: 

failed:
cdef extern from :
cdef extern from "":

Or is it otherwise somehow possible to tell him library /I includes?


Also for the implicit python includes is there a need for #include 
"Python.h" "frameobject.h" .. instead of 

SWIG for example produces #include 

came about the problem because the dependecies from "gcc -MM" 
swelled up. It would be more clear/convenient also maybe?


Robert







From dagss at student.matnat.uio.no  Thu May 21 12:32:33 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 21 May 2009 12:32:33 +0200 (CEST)
Subject: [Cython] cdef extern from "" ?
In-Reply-To: 
References: 
Message-ID: 

Robert wrote:
> failed:
> cdef extern from :
> cdef extern from "":
>
> Or is it otherwise somehow possible to tell him library /I includes?

As long as you don't have a file called times.h in the same directory as
the compiled c file (which I suppose would only happen on --inplace),
"file" and  does the same.

In fact, one could argue that Cython should *always* use  instead
of "file.h", so that one always get the same behaviour regardless of build
method.

Or?

I don't know anything about your other question.

Dag Sverre


From dalcinl at gmail.com  Thu May 21 15:57:59 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 21 May 2009 10:57:59 -0300
Subject: [Cython] cdef extern from "" ?
In-Reply-To: 
References: 
	
Message-ID: 

On Thu, May 21, 2009 at 7:32 AM, Dag Sverre Seljebotn
 wrote:
> Robert wrote:
>> failed:
>> cdef extern from :
>> cdef extern from "":
>>
>> Or is it otherwise somehow possible to tell him library /I includes?
>
> As long as you don't have a file called times.h in the same directory as
> the compiled c file (which I suppose would only happen on --inplace),
> "file" and  does the same.
>
> In fact, one could argue that Cython should *always* use  instead
> of "file.h", so that one always get the same behaviour regardless of build
> method.
>
> Or?
>
> I don't know anything about your other question.
>

What about making Cython behave like this:

For this Cython code:

cdef extern from "":
   pass

generate the C include like htis:

#include 





> Dag Sverre
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From kxroberto at googlemail.com  Thu May 21 16:20:29 2009
From: kxroberto at googlemail.com (Robert)
Date: Thu, 21 May 2009 16:20:29 +0200
Subject: [Cython] cdef extern from "" ?
In-Reply-To: 
References: 
	
Message-ID: 

Dag Sverre Seljebotn wrote:
> Robert wrote:
>> failed:
>> cdef extern from :
>> cdef extern from "":
>>
>> Or is it otherwise somehow possible to tell him library /I includes?
> 
> As long as you don't have a file called times.h in the same directory as
> the compiled c file (which I suppose would only happen on --inplace),
> "file" and  does the same.
> 

yes, its not severe.
just with "gcc -MM modname.c" (create dependencies) it makes 
problems. he errors on missing ""-includes, but not on 
<>-includes, when -I is omitted.
Usually one doesn't want all the constant library stuff in the 
project depend file.

> In fact, one could argue that Cython should *always* use  instead
> of "file.h", so that one always get the same behaviour regardless of build
> method.

hmm, brackets for the automatic python includes, but still "" by 
default for extern from "file.h", because that are often project h 
files.

just, when: extern from "", as Lisandro said.

> 
> Or?
> 
> I don't know anything about your other question.
> 

above "gcc -MM" ?


Robert


From dagss at student.matnat.uio.no  Thu May 21 17:46:48 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 21 May 2009 17:46:48 +0200 (CEST)
Subject: [Cython] cdef extern from "" ?
In-Reply-To: 
References: 
	
	
Message-ID: <36fa15ada30e707eb1802a114d26968e.squirrel@webmail.uio.no>

Lisandro wrote:
> For this Cython code:
>
> cdef extern from "":
>    pass
>
> generate the C include like htis:
>
> #include 

-0

What is the point? It usually makes absolutely no difference for the C
compiler, as "sys/time.h" isn't going to be located in
build/temp-linux..../pyrex/ anyway.

(Unless you rely on the "-iquote" feature of gcc. I suppose we could start
using -iquote it to point to the base of the .pyx file ... personally I
just consider this a misfeature of C preprocessor though.)

http://gcc.gnu.org/onlinedocs/cpp/Include-Syntax.html

Dag Sverre


From dagss at student.matnat.uio.no  Thu May 21 17:59:32 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 21 May 2009 17:59:32 +0200 (CEST)
Subject: [Cython] [Numpy-discussion] ANN: Cython 0.11.2 released
In-Reply-To: 
References: <4A1425F9.9030100@student.matnat.uio.no>
	
Message-ID: <2418c278951e6461a58d1baea226bf95.squirrel@webmail.uio.no>

> Where can I find release notes?  (It would be helpful if I can point to a
> URL as part of the fedora release)
>

OK, I put my email announcement up here:

http://wiki.cython.org/ReleaseNotes-0.11.2

Tell me if you need something else (different format or level of detail --
the list of tickets on trac is always the most accurate thing).

Dag Sverre


From sccolbert at gmail.com  Thu May 21 18:52:24 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Thu, 21 May 2009 12:52:24 -0400
Subject: [Cython] trouble splitting extension type into definition and
	implementation files
In-Reply-To: <4A14A1EE.2090700@canterbury.ac.nz>
References: <7f014ea60905201246m7ddea54uea53ca68cde98122@mail.gmail.com>
	<7f014ea60905201249g74e0cd6k926da0d126d9a64b@mail.gmail.com>
	<4A14A1EE.2090700@canterbury.ac.nz>
Message-ID: <7f014ea60905210952o49124bd9w1990af7bd756dc@mail.gmail.com>

so, what would the setup script look like for something like this?

if i do:

sources = ['file1.pyx', 'file2.pyx', ... ]

setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = [Extension("mymodule", sourcefiles,
                                include_dirs=include_dirs,
                                library_dirs=library_dirs,
                                libraries = libraries)])

it fails.

but if i build each module separately, everything works. i.e.:

sources = ['file1.pyx']

setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = [Extension("mymodule1", sourcefiles,
                                include_dirs=include_dirs,
                                library_dirs=library_dirs,
                                libraries = libraries)])

^^^repeat the above for each module.


What I want to be able to do, is have all of my multiple .pyx and .pxd files
compiled and built into a single  .pyd

Is that possible?

Chris


On Wed, May 20, 2009 at 8:35 PM, Greg Ewing wrote:

> Chris Colbert wrote:
>
> > Is there a way to properly compile everything into a single .pyd,
>
> You can use include statements to split a .pyx into
> multiple files, but I wouldn't recommend that, because
> it makes it hard for the reader to tell where things
> are defined.
>
> Also, separate modules means there's less to recompile
> when you change one of the implementation files, and
> it seems that compiling a number of small C files can
> sometimes be faster than compiling a single large C
> file of the same total size.
>
> --
> Greg
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090521/f61eb2cc/attachment.htm 

From dalcinl at gmail.com  Thu May 21 19:33:55 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 21 May 2009 14:33:55 -0300
Subject: [Cython] Note on C99 compound literals
In-Reply-To: <308A5EFB-A23F-46D1-8C10-97435CE58DBB@math.washington.edu>
References: <4A13FC60.50001@student.matnat.uio.no>
	<270275DC-E839-4FB8-B6A7-2CB538DACF55@math.washington.edu>
	<4A144FA0.4020704@student.matnat.uio.no>
	
	
	<308A5EFB-A23F-46D1-8C10-97435CE58DBB@math.washington.edu>
Message-ID: 

On Thu, May 21, 2009 at 5:28 AM, Robert Bradshaw
 wrote:
> On May 20, 2009, at 10:17 PM, Dag Sverre Seljebotn wrote:
>
> I'd like to not require much more than Python itself requires.
>

Indeed. I bet MSVC does not support compound literals.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Thu May 21 19:46:05 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 21 May 2009 19:46:05 +0200
Subject: [Cython] Special methods and __hash__
In-Reply-To: <629C160E-2834-4F50-B41C-BDD74786A82A@math.washington.edu>
References: 	<1074aefe36699e13d6cf77c04525dcb8.squirrel@webmail.uio.no>
	<629C160E-2834-4F50-B41C-BDD74786A82A@math.washington.edu>
Message-ID: <4A15935D.8020403@student.matnat.uio.no>

Robert Bradshaw wrote:
> On May 20, 2009, at 10:10 PM, Dag Sverre Seljebotn wrote:
> 
>> Robert wrote:
>>> According to the Python/C API, returning -1 from __hash__ is the way
>>> to signal an error. Of course, if one returns -1 manually one gets a
>>> runtime error that an error was raised without setting the exception.
>>> We could document this, but it's still kind of obscure.
>>>
>>> In Python one can do
>>>
>>> sage: class A:
>>> ....:     def __hash__(self):
>>> ....:         return int(-1)
>>> ....:
>>> sage: hash(A())
>>> -2
>>>
>>> Should we just check for -1 and return -2 for cdef class __hash__
>>> functions as well?
>> +1, but IMO it should wait until the rest of the operators have been
>> discussed for alignment with Python as well.
>>
>> For instance I'd like to propose:
>> - Introduce "__cadd__" to take over for the current "__add__"
>> - Create a script which can convert source files (e.g. in Sage) and do
>> this change automatically (since the change below will slow things  
>> down)
>> - Make __cadd__ by default do something like this:
>>
>> cdef class MyClass:
>>     def __cadd__(a, b):
>>         # body inserted if __add__ is implemented
>>         if isinstance(a, MyClass):
>>             return a.__add__(a, b)
>>         else:
>>             return b.__add__(b, a)
> 
> This is a much more invasive change--I think the hash function is  
> safer because it turns what was a certain runtime error into  
> something reasonable.

Good point!

-- 
Dag Sverre

From dalcinl at gmail.com  Thu May 21 20:01:39 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 21 May 2009 15:01:39 -0300
Subject: [Cython] cdef extern from "" ?
In-Reply-To: <36fa15ada30e707eb1802a114d26968e.squirrel@webmail.uio.no>
References: 
	
	
	<36fa15ada30e707eb1802a114d26968e.squirrel@webmail.uio.no>
Message-ID: 

On Thu, May 21, 2009 at 12:46 PM, Dag Sverre Seljebotn
 wrote:
> Lisandro wrote:
>> For this Cython code:
>>
>> cdef extern from "":
>> ? ?pass
>>
>> generate the C include like htis:
>>
>> #include 
>
> -0
>
> What is the point? It usually makes absolutely no difference for the C
> compiler, as "sys/time.h" isn't going to be located in
> build/temp-linux..../pyrex/ anyway.
>

IIUC, you are assuming that generated C sources are ALWAYS at
build/temp-linux..../pyrex/ . For my own projects, that is not the
case...


> (Unless you rely on the "-iquote" feature of gcc. I suppose we could start
> using -iquote it to point to the base of the .pyx file ... personally I
>

Well, if GCC is the only compiler you target for your code, then you
can pass the flag. But Cython should not rely on that IMHO.

>
> just consider this a misfeature of C preprocessor though.)
>

Perhaps, but despite that we have to live with it.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Thu May 21 20:16:54 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 21 May 2009 20:16:54 +0200
Subject: [Cython] The future of the bracket syntax
Message-ID: <4A159A96.9070907@student.matnat.uio.no>

A decision is needed here within a few weeks; it would be great if any 
core developers who don't care could respond to vote 0.

The decision was made to use [] for templates. To reiterate the main 
arguments:

  - It leaves the way open for exporting pre-instantiated templates to 
Python in a /very/ natural fashion.

  - Templates are a bit like a collection of types anyway.


However this creates some syntax issues:

A) Currently

   cdef object[int] arr = ...

can be used for PEP 3118 access on a generic object. While not a 
technical problem (object will never become a template), it looks a bit 
odd when introducing with a template feature.

B) How would a templated buffer type look like? MyType[int][float]?

Proposal:

The object[int] syntax is removed (are anyone using it?). This is 
basically done by making any cdef class with the 
__cythonbufferdefaults__ magic attribute become a "buffer template". 
Such cdef classes cannot be templates. Any other class cannot be used as 
a buffer-supporting object.

This removes functionality (much numeric code could care less about 
whether the underlying object is an ndarray). As a substitute, the main 
idea in http://wiki.cython.org/enhancements/buffersyntax is accepted, 
which is to use

cdef int[:] arr = ...

for generic access, without any object access (this means that for now 
only indexing and getting the shape is allowed at all.)

I've been wondering a lot about whether this is the right way, and it 
still seems so, even if it means that there will be (and likely continue 
to be) two different ways of accessing a buffer; one through an explicit 
"buffer type" and one through an "automatic template".

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Thu May 21 20:26:41 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 21 May 2009 20:26:41 +0200
Subject: [Cython] cdef extern from "" ?
In-Reply-To: 
References: 			<36fa15ada30e707eb1802a114d26968e.squirrel@webmail.uio.no>
	
Message-ID: <4A159CE1.3050202@student.matnat.uio.no>

Lisandro Dalcin wrote:
> On Thu, May 21, 2009 at 12:46 PM, Dag Sverre Seljebotn
>  wrote:
>> What is the point? It usually makes absolutely no difference for the C
>> compiler, as "sys/time.h" isn't going to be located in
>> build/temp-linux..../pyrex/ anyway.
>>
> 
> IIUC, you are assuming that generated C sources are ALWAYS at
> build/temp-linux..../pyrex/ . For my own projects, that is not the
> case...
> 
> 
>> (Unless you rely on the "-iquote" feature of gcc. I suppose we could start
>> using -iquote it to point to the base of the .pyx file ... personally I
>>
> 
> Well, if GCC is the only compiler you target for your code, then you
> can pass the flag. But Cython should not rely on that IMHO.
> 
>> just consider this a misfeature of C preprocessor though.)
>>
> 
> Perhaps, but despite that we have to live with it.

Because we have to support projects which need to include both "./foo.h" 
and "/usr/include/foo.h" as distinct header files in the same pyx file?

-- 
Dag Sverre

From robertwb at math.washington.edu  Thu May 21 20:33:00 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 21 May 2009 11:33:00 -0700
Subject: [Cython] trouble splitting extension type into definition and
	implementation files
In-Reply-To: <7f014ea60905210952o49124bd9w1990af7bd756dc@mail.gmail.com>
References: <7f014ea60905201246m7ddea54uea53ca68cde98122@mail.gmail.com>
	<7f014ea60905201249g74e0cd6k926da0d126d9a64b@mail.gmail.com>
	<4A14A1EE.2090700@canterbury.ac.nz>
	<7f014ea60905210952o49124bd9w1990af7bd756dc@mail.gmail.com>
Message-ID: <2D5B9066-1F2B-4C52-9770-CB1281FB9E63@math.washington.edu>

On May 21, 2009, at 9:52 AM, Chris Colbert wrote:

> so, what would the setup script look like for something like this?
>
> if i do:
>
> sources = ['file1.pyx', 'file2.pyx', ... ]
>
> setup(
>     cmdclass = {'build_ext': build_ext},
>     ext_modules = [Extension("mymodule", sourcefiles,
>                                 include_dirs=include_dirs,
>                                 library_dirs=library_dirs,
>                                 libraries = libraries)])
>
> it fails.
>
> but if i build each module separately, everything works. i.e.:
>
> sources = ['file1.pyx']
>
> setup(
>     cmdclass = {'build_ext': build_ext},
>     ext_modules = [Extension("mymodule1", sourcefiles,
>                                 include_dirs=include_dirs,
>                                 library_dirs=library_dirs,
>                                 libraries = libraries)])
>
> ^^^repeat the above for each module.
>
>
> What I want to be able to do, is have all of my multiple .pyx  
> and .pxd files compiled and built into a single  .pyd
>
> Is that possible?

I don't know how it works on Windows, but on linux/OS X Python does a  
lookup on modulename.so as part of the import mechanism... Perhaps  
some hackery could be done, but it wouldn't work out of the box.

- Robert



From dalcinl at gmail.com  Thu May 21 21:11:24 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 21 May 2009 16:11:24 -0300
Subject: [Cython] cdef extern from "" ?
In-Reply-To: <4A159CE1.3050202@student.matnat.uio.no>
References: 
	
	
	<36fa15ada30e707eb1802a114d26968e.squirrel@webmail.uio.no>
	
	<4A159CE1.3050202@student.matnat.uio.no>
Message-ID: 

On Thu, May 21, 2009 at 3:26 PM, Dag Sverre Seljebotn
 wrote:
> Lisandro Dalcin wrote:
>
> Because we have to support projects which need to include both "./foo.h"
> and "/usr/include/foo.h" as distinct header files in the same pyx file?
>

Yes, we need to support that.  could resolve to a header in
"/usr/include", or even to a header at -I/some/dir, and such inclusion
could potentially arise in a pxd from a third-party Cython-based
project your own Cython code depends on.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Thu May 21 22:36:13 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 21 May 2009 22:36:13 +0200
Subject: [Cython] Utility code changesets to be aware of
Message-ID: <4A15BB3D.3050008@student.matnat.uio.no>

Since both Robert and Lisandro has been using templating utility code a 
lot lately, I think this commit was important to note:

You can now do

class StructUtilityCode(object):
     def __eq__(self, other): ...
     def put_code(self, output):
         code = output['utility_code_def']
         proto = output['utility_code_proto']

         code.putln("%s {" % self.header)
         code.putln("PyObject* res;")
         ...

class CStructOrUnionType(CType):
     def create_to_py_utility_code(self, env):
         ...
         env.use_utility_code(StructUtilityCode(self))


The "old" UtilityCode is still available, merely implementing put_code 
by putting the various stored parts in the various places.

http://hg.cython.org/cython-unstable/rev/5a66cd8359e8

and before it.

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Thu May 21 22:38:15 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 21 May 2009 22:38:15 +0200
Subject: [Cython] Utility code changesets to be aware of
In-Reply-To: <4A15BB3D.3050008@student.matnat.uio.no>
References: <4A15BB3D.3050008@student.matnat.uio.no>
Message-ID: <4A15BBB7.1070604@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Since both Robert and Lisandro has been using templating utility code a 
> lot lately, I think this commit was important to note:
> 
> You can now do
> 
> class StructUtilityCode(object):
>      def __eq__(self, other): ...
>      def put_code(self, output):
>          code = output['utility_code_def']
>          proto = output['utility_code_proto']
> 
>          code.putln("%s {" % self.header)
>          code.putln("PyObject* res;")
>          ...
> 
> class CStructOrUnionType(CType):
>      def create_to_py_utility_code(self, env):
>          ...
>          env.use_utility_code(StructUtilityCode(self))
> 
> 
> The "old" UtilityCode is still available, merely implementing put_code 
> by putting the various stored parts in the various places.

I moved it to Code.py though, it seemed proper after the change.

-- 
Dag Sverre

From kxroberto at googlemail.com  Fri May 22 00:07:18 2009
From: kxroberto at googlemail.com (Robert)
Date: Fri, 22 May 2009 00:07:18 +0200
Subject: [Cython] trouble splitting extension type into definition and
 implementation files
In-Reply-To: <7f014ea60905210952o49124bd9w1990af7bd756dc@mail.gmail.com>
References: <7f014ea60905201246m7ddea54uea53ca68cde98122@mail.gmail.com>	<7f014ea60905201249g74e0cd6k926da0d126d9a64b@mail.gmail.com>	<4A14A1EE.2090700@canterbury.ac.nz>
	<7f014ea60905210952o49124bd9w1990af7bd756dc@mail.gmail.com>
Message-ID: 

Chris Colbert wrote:
> so, what would the setup script look like for something like this?
> 
> if i do:
> 
> sources = ['file1.pyx', 'file2.pyx', ... ]
> 
> setup(
>     cmdclass = {'build_ext': build_ext},
>     ext_modules = [Extension("mymodule", sourcefiles,
>                                 include_dirs=include_dirs,
>                                 library_dirs=library_dirs,
>                                 libraries = libraries)])
> 
> it fails.
> 
> but if i build each module separately, everything works. i.e.:
> 
> sources = ['file1.pyx']
> 
> setup(
>     cmdclass = {'build_ext': build_ext},
>     ext_modules = [Extension("mymodule1", sourcefiles,
>                                 include_dirs=include_dirs,
>                                 library_dirs=library_dirs,
>                                 libraries = libraries)])
> 
> ^^^repeat the above for each module.
> 
> 
> What I want to be able to do, is have all of my multiple .pyx and .pxd 
> files compiled and built into a single  .pyd
> 
> Is that possible?
> 

each .pyx defines a module (name), so how do you want to address 
that stuff consistently?
Why not merge the 2 .pyx (and .pxd) in the first place in one 
module? Or make a package dir with a __init__.py which merges with 
from ... import * ?



But just for fun its possible to squeeze 2 pyx modules it one DLL:
say modules a.pyx/a.c and b.pyx/b.c

#in a.pyx add (Python2.x):

cdef extern from *:
     void initb()
def loadmod_b():
     initb()
     ##exec "from b import *" in globals()
##loadmod_b()


#cython both, :

#in b.c edit (cython forgot a 'static'):

const char *__pyx_modulename = "b";
->
static const char *__pyx_modulename = "b";

#then on e.g. Windows/MingW do like

gcc -mno-cygwin -O -IC:\Python26\include -g a.c b.c -shared 
-mno-cygwin -LC:\Python26\libs -lpython26 -lmsvcr90 -o a.pyd

# then you can import a, and let a.loadmod_b()  initialize your 
module b

then you can 'import b' from everywhere too...



Robert





From sccolbert at gmail.com  Fri May 22 00:09:23 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Thu, 21 May 2009 18:09:23 -0400
Subject: [Cython] nesting a union or struct within a struct?
Message-ID: <7f014ea60905211509q5578b9eve0ddef3e3a31d26d@mail.gmail.com>

Hi,

I'm trying to write a wrapper for a C library.

The C header from which I am making declaration has this:


struct Foo
{
     int blah;

     union
     {
          int Bar;
     } data;
}



can this be wrapped in Cython?

I've tried:


cdef extern from "c.h":

    ctypedef struct Foo:

            int step

        union data:
            int Bar



but that doesnt compile.

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090521/e21ad347/attachment-0001.htm 

From sccolbert at gmail.com  Fri May 22 01:05:02 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Thu, 21 May 2009 19:05:02 -0400
Subject: [Cython] how to pass opaque pointers to functions?
Message-ID: <7f014ea60905211605p130dce59lcae07171c7b30fbf@mail.gmail.com>

I have certain functions that should accept types of IplImage or CvMat

I have the two types declared as inheriting from a generic CvArr as such

cdef class CvArr:
       cdef void* thisptr

cdef class IplImage(CvArr):
      .....

cdef class CvMat(CvArr):
      ....


and a function declared like this:

def cvSmooth(CvArr src, .....):
      c_lib.cvSmooth(src.thisptr, ....)

when compiling and calling that function with instances of IplImage, the
Cython compiler complains that src is not a CvArr.

Is there a way around this?

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090521/74f0c277/attachment.htm 

From greg.ewing at canterbury.ac.nz  Fri May 22 02:13:01 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 22 May 2009 12:13:01 +1200
Subject: [Cython] cdef extern from "" ?
In-Reply-To: 
References: 
	
Message-ID: <4A15EE0D.7040200@canterbury.ac.nz>

Dag Sverre Seljebotn wrote:

> In fact, one could argue that Cython should *always* use  instead
> of "file.h", so that one always get the same behaviour regardless of build
> method.

But what if you *do* want to include a file from the
same directory?

Having a local include file with the same name as a
system one seems like an anti-pattern to me, so I
don't see an issue here.

-- 
Greg

From sccolbert at gmail.com  Fri May 22 02:16:35 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Thu, 21 May 2009 20:16:35 -0400
Subject: [Cython] how to pass opaque pointers to functions?
In-Reply-To: <7f014ea60905211605p130dce59lcae07171c7b30fbf@mail.gmail.com>
References: <7f014ea60905211605p130dce59lcae07171c7b30fbf@mail.gmail.com>
Message-ID: <7f014ea60905211716m3dddc701udaa60e72cef67fdf@mail.gmail.com>

the problem is that the C library has several functions that will accept one
of many types of arrays.

The function parameter type is CvArr whose definition is (in C):

typedef void CvArr;

and the functions are declared as accepting CvArr* types as arguments.

two of the types I am wrapping are IplImage and CvMat both of which can be
passed as CvArr*.

I would like to make python callable functions that accept either instance
of IplImage or CvMat (which I have created as extension types).

Unfortunately, i haven't yet been able to figure it out through various
tinkering with subclassing, casting, or void pointers....


I would greatly appreciate any insight one could give..

Thanks and Cheers!

Chris

On Thu, May 21, 2009 at 7:05 PM, Chris Colbert  wrote:

> I have certain functions that should accept types of IplImage or CvMat
>
> I have the two types declared as inheriting from a generic CvArr as such
>
> cdef class CvArr:
>        cdef void* thisptr
>
> cdef class IplImage(CvArr):
>       .....
>
> cdef class CvMat(CvArr):
>       ....
>
>
> and a function declared like this:
>
> def cvSmooth(CvArr src, .....):
>       c_lib.cvSmooth(src.thisptr, ....)
>
> when compiling and calling that function with instances of IplImage, the
> Cython compiler complains that src is not a CvArr.
>
> Is there a way around this?
>
> Chris
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090521/3f93f4cd/attachment.htm 

From greg.ewing at canterbury.ac.nz  Fri May 22 02:20:24 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 22 May 2009 12:20:24 +1200
Subject: [Cython] cdef extern from "" ?
In-Reply-To: 
References: 
	
	
Message-ID: <4A15EFC8.8080602@canterbury.ac.nz>

Robert wrote:

> just with "gcc -MM modname.c" (create dependencies) it makes 
> problems. he errors on missing ""-includes, but not on 
> <>-includes, when -I is omitted.

The generated C file seems like the wrong level to
be tracking dependencies at, to me.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Fri May 22 02:27:10 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 22 May 2009 12:27:10 +1200
Subject: [Cython] trouble splitting extension type into definition and
 implementation files
In-Reply-To: <7f014ea60905210952o49124bd9w1990af7bd756dc@mail.gmail.com>
References: <7f014ea60905201246m7ddea54uea53ca68cde98122@mail.gmail.com>
	<7f014ea60905201249g74e0cd6k926da0d126d9a64b@mail.gmail.com>
	<4A14A1EE.2090700@canterbury.ac.nz>
	<7f014ea60905210952o49124bd9w1990af7bd756dc@mail.gmail.com>
Message-ID: <4A15F15E.3030004@canterbury.ac.nz>

Chris Colbert wrote:

> What I want to be able to do, is have all of my multiple .pyx and .pxd 
> files compiled and built into a single  .pyd

If you're keeping them as separate modules, you end
up with a .pyd for each module, and the setup.py
has a separate Extension instance for each module.

You can put them in a package with an __init__.py
that imports all the names to be made public, so
that it looks like a single module to Python.

But if you do that, be sure to consult the docs on
how to compile .pyx files that are to be part of a
package.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Fri May 22 02:38:47 2009
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 22 May 2009 12:38:47 +1200
Subject: [Cython] trouble splitting extension type into definition and
 implementation files
In-Reply-To: <2D5B9066-1F2B-4C52-9770-CB1281FB9E63@math.washington.edu>
References: <7f014ea60905201246m7ddea54uea53ca68cde98122@mail.gmail.com>
	<7f014ea60905201249g74e0cd6k926da0d126d9a64b@mail.gmail.com>
	<4A14A1EE.2090700@canterbury.ac.nz>
	<7f014ea60905210952o49124bd9w1990af7bd756dc@mail.gmail.com>
	<2D5B9066-1F2B-4C52-9770-CB1281FB9E63@math.washington.edu>
Message-ID: <4A15F417.6050805@canterbury.ac.nz>

Robert Bradshaw wrote:

> I don't know how it works on Windows, but on linux/OS X Python does a  
> lookup on modulename.so as part of the import mechanism... Perhaps  
> some hackery could be done, but it wouldn't work out of the box.

Possibly it could be made to work by having the init
code of the "main" module explicitly call the init
functions of the other modules. That would cause
the relevant module objects to be created and put in
sys.modules, where subsequent imports would find
them.

Seems like a pointless exercise to me, though, when
a package can give you much the same end result with
no hackery.

-- 
Greg

From dalcinl at gmail.com  Fri May 22 03:18:57 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 21 May 2009 22:18:57 -0300
Subject: [Cython] how to pass opaque pointers to functions?
In-Reply-To: <7f014ea60905211716m3dddc701udaa60e72cef67fdf@mail.gmail.com>
References: <7f014ea60905211605p130dce59lcae07171c7b30fbf@mail.gmail.com>
	<7f014ea60905211716m3dddc701udaa60e72cef67fdf@mail.gmail.com>
Message-ID: 

cdef class CvArr:
   def c_CvArr* handle(self):
       raise NotImplementedError

cdef class IplImage(CvArr):
    cdef c_IplImage* image
    def c_CvArr* handle(self):
       return self.image

cdef class CvMat(CvArr):
    cdef c_CvMat *cvmat
    def c_CvArr* handle(self):
       return self.cvmat


Mmm... just a use case for "cdef properties" managed by calling
get/setters in the vtable?


On Thu, May 21, 2009 at 9:16 PM, Chris Colbert  wrote:
> the problem is that the C library has several functions that will accept one
> of many types of arrays.
>
> The function parameter type is CvArr whose definition is (in C):
>
> typedef void CvArr;
>
> and the functions are declared as accepting CvArr* types as arguments.
>
> two of the types I am wrapping are IplImage and CvMat both of which can be
> passed as CvArr*.
>
> I would like to make python callable functions that accept either instance
> of IplImage or CvMat (which I have created as extension types).
>
> Unfortunately, i haven't yet been able to figure it out through various
> tinkering with subclassing, casting, or void pointers....
>
>
> I would greatly appreciate any insight one could give..
>
> Thanks and Cheers!
>
> Chris
>
> On Thu, May 21, 2009 at 7:05 PM, Chris Colbert  wrote:
>>
>> I have certain functions that should accept types of IplImage or CvMat
>>
>> I have the two types declared as inheriting from a generic CvArr as such
>>
>> cdef class CvArr:
>> ?????? cdef void* thisptr
>>
>> cdef class IplImage(CvArr):
>> ????? .....
>>
>> cdef class CvMat(CvArr):
>> ????? ....
>>
>>
>> and a function declared like this:
>>
>> def cvSmooth(CvArr src, .....):
>> ????? c_lib.cvSmooth(src.thisptr, ....)
>>
>> when compiling and calling that function with instances of IplImage, the
>> Cython compiler complains that src is not a CvArr.
>>
>> Is there a way around this?
>>
>> Chris
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Fri May 22 03:23:57 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 21 May 2009 22:23:57 -0300
Subject: [Cython] nesting a union or struct within a struct?
In-Reply-To: <7f014ea60905211509q5578b9eve0ddef3e3a31d26d@mail.gmail.com>
References: <7f014ea60905211509q5578b9eve0ddef3e3a31d26d@mail.gmail.com>
Message-ID: 

cdef extern ...

   ctypedef union Foo_Inner: # fake name !
        int Bar

   struct Foo:
      int step
      Foo_Inner data

Perhaps it works, never tried ... :-)

On Thu, May 21, 2009 at 7:09 PM, Chris Colbert  wrote:
> Hi,
>
> I'm trying to write a wrapper for a C library.
>
> The C header from which I am making declaration has this:
>
>
> struct Foo
> {
> ???? int blah;
>
> ???? union
> ???? {
> ????????? int Bar;
> ???? } data;
> }
>
>
>
> can this be wrapped in Cython?
>
> I've tried:
>
>
> cdef extern from "c.h":
>
> ??? ctypedef struct Foo:
>
> ??????????? int step
>
> ??????? union data:
> ??????????? int Bar
>
>
>
> but that doesnt compile.
>
> Chris
>
>
>
>
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From sccolbert at gmail.com  Fri May 22 06:49:47 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Fri, 22 May 2009 00:49:47 -0400
Subject: [Cython] how to pass opaque pointers to functions?
In-Reply-To: 
References: <7f014ea60905211605p130dce59lcae07171c7b30fbf@mail.gmail.com>
	<7f014ea60905211716m3dddc701udaa60e72cef67fdf@mail.gmail.com>
	
Message-ID: <7f014ea60905212149ld9bd849r70f74f57ff872667@mail.gmail.com>

is 'handle()' a method that is called internally when passing extension
types as parameters?

I'm exposing a C function to Python as follows:

def cvSomeFunction(CvArr image):  #image should be able to be CvArr IplImage
or CvMat
     c_lib.cvSomeFunction(image.thisptr)    # C func prototype = void
cvSomeFunction(CvArr* image) and image can be CvArr* IplImage* or CvMat*

I'm not sure I understand if the classes you show will do that.

For example, my IplImage class is like this:

cdef class IplImage:
      cdef c_lib.IplImage* thisptr

      <----snip---->

and i work with it like this:

cdef c_lib.IplImage* img = cvFunctionThatReturnsPtr()
cdef py_lib.IplImage pyimg = py_lib.IplImage()
pyimg.thisptr = img

etc....

now, if I rewrite my python function wrapper above like this, it works, but
only when i pass the python type IplImage
 def cvSomeFunction(py_lib.IplImage image):
     c_lib.cvSomeFunction(image.thisptr)  # C prototype is still --- void
cvSomeFunction(CvArr* image)


I understand why its not working like that, I'm just hoping there is some
way around it that doesnt require a bunch of type checking in Python. I'm
trying to keep the wrapper as thin as possible.

I appreciate all your help Lisandro!

Chris




On Thu, May 21, 2009 at 9:18 PM, Lisandro Dalcin  wrote:

> cdef class CvArr:
>   def c_CvArr* handle(self):
>       raise NotImplementedError
>
> cdef class IplImage(CvArr):
>    cdef c_IplImage* image
>    def c_CvArr* handle(self):
>       return self.image
>
> cdef class CvMat(CvArr):
>    cdef c_CvMat *cvmat
>    def c_CvArr* handle(self):
>       return self.cvmat
>
>
> Mmm... just a use case for "cdef properties" managed by calling
> get/setters in the vtable?
>
>
> On Thu, May 21, 2009 at 9:16 PM, Chris Colbert 
> wrote:
> > the problem is that the C library has several functions that will accept
> one
> > of many types of arrays.
> >
> > The function parameter type is CvArr whose definition is (in C):
> >
> > typedef void CvArr;
> >
> > and the functions are declared as accepting CvArr* types as arguments.
> >
> > two of the types I am wrapping are IplImage and CvMat both of which can
> be
> > passed as CvArr*.
> >
> > I would like to make python callable functions that accept either
> instance
> > of IplImage or CvMat (which I have created as extension types).
> >
> > Unfortunately, i haven't yet been able to figure it out through various
> > tinkering with subclassing, casting, or void pointers....
> >
> >
> > I would greatly appreciate any insight one could give..
> >
> > Thanks and Cheers!
> >
> > Chris
> >
> > On Thu, May 21, 2009 at 7:05 PM, Chris Colbert 
> wrote:
> >>
> >> I have certain functions that should accept types of IplImage or CvMat
> >>
> >> I have the two types declared as inheriting from a generic CvArr as such
> >>
> >> cdef class CvArr:
> >>        cdef void* thisptr
> >>
> >> cdef class IplImage(CvArr):
> >>       .....
> >>
> >> cdef class CvMat(CvArr):
> >>       ....
> >>
> >>
> >> and a function declared like this:
> >>
> >> def cvSmooth(CvArr src, .....):
> >>       c_lib.cvSmooth(src.thisptr, ....)
> >>
> >> when compiling and calling that function with instances of IplImage, the
> >> Cython compiler complains that src is not a CvArr.
> >>
> >> Is there a way around this?
> >>
> >> Chris
> >
> >
> > _______________________________________________
> > Cython-dev mailing list
> > Cython-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/cython-dev
> >
> >
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090522/06684119/attachment.htm 

From dagss at student.matnat.uio.no  Fri May 22 06:50:44 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 22 May 2009 06:50:44 +0200 (CEST)
Subject: [Cython] how to pass opaque pointers to functions?
In-Reply-To: <7f014ea60905211716m3dddc701udaa60e72cef67fdf@mail.gmail.com>
References: <7f014ea60905211605p130dce59lcae07171c7b30fbf@mail.gmail.com>
	<7f014ea60905211716m3dddc701udaa60e72cef67fdf@mail.gmail.com>
Message-ID: 

Hi Chris,

> the problem is that the C library has several functions that will accept
> one
> of many types of arrays.
>
> The function parameter type is CvArr whose definition is (in C):
>
> typedef void CvArr;
>
> and the functions are declared as accepting CvArr* types as arguments.
>
> two of the types I am wrapping are IplImage and CvMat both of which can be
> passed as CvArr*.
>
> I would like to make python callable functions that accept either instance
> of IplImage or CvMat (which I have created as extension types).
>
> Unfortunately, i haven't yet been able to figure it out through various
> tinkering with subclassing, casting, or void pointers....

Note that

def f(CvMat foo): ...

is almost exactly the same as

def f(foo):
    if not isinstance(foo, CvMat): raise some error...

So, what you do is

def f(obj):
    if isinstance(foo, CvMat): (a)
    elif isinstance(foo, IplImage): (b)
    else: raise some error

Dag Sverre


From sccolbert at gmail.com  Fri May 22 07:13:19 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Fri, 22 May 2009 01:13:19 -0400
Subject: [Cython] how to pass opaque pointers to functions?
In-Reply-To: 
References: <7f014ea60905211605p130dce59lcae07171c7b30fbf@mail.gmail.com>
	<7f014ea60905211716m3dddc701udaa60e72cef67fdf@mail.gmail.com>
	
Message-ID: <7f014ea60905212213q5da215d5s9a2728bbf929b3a6@mail.gmail.com>

so if I have:

cdef class IplImage:
       cdef c_lib.IplImage* thisptr


cdef class CvMat:
      cdef c_lib.CvMat* thisptr



then I can just do this?

def exposedPythonFunc(image):   where image can be an IplImage or CvMat
      c_lib.cFunction(image.thisptr)      since I know thisptr exists



or must I do?

def exposedPythonFunc(image):
     if isinstance(image, IplImage):
            c_lib.cFunction(image.thisptr)

    elif isinstance(image, CvMat):
           c_lib.cFunction(image.thisptr)

    else:
           raise NotImplementedError


Thanks!

Chris

On Fri, May 22, 2009 at 12:50 AM, Dag Sverre Seljebotn <
dagss at student.matnat.uio.no> wrote:

> Hi Chris,
>
> > the problem is that the C library has several functions that will accept
> > one
> > of many types of arrays.
> >
> > The function parameter type is CvArr whose definition is (in C):
> >
> > typedef void CvArr;
> >
> > and the functions are declared as accepting CvArr* types as arguments.
> >
> > two of the types I am wrapping are IplImage and CvMat both of which can
> be
> > passed as CvArr*.
> >
> > I would like to make python callable functions that accept either
> instance
> > of IplImage or CvMat (which I have created as extension types).
> >
> > Unfortunately, i haven't yet been able to figure it out through various
> > tinkering with subclassing, casting, or void pointers....
>
> Note that
>
> def f(CvMat foo): ...
>
> is almost exactly the same as
>
> def f(foo):
>    if not isinstance(foo, CvMat): raise some error...
>
> So, what you do is
>
> def f(obj):
>    if isinstance(foo, CvMat): (a)
>    elif isinstance(foo, IplImage): (b)
>    else: raise some error
>
> Dag Sverre
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090522/27d14534/attachment.htm 

From dagss at student.matnat.uio.no  Fri May 22 07:40:16 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 22 May 2009 07:40:16 +0200 (CEST)
Subject: [Cython] how to pass opaque pointers to functions?
In-Reply-To: <7f014ea60905212213q5da215d5s9a2728bbf929b3a6@mail.gmail.com>
References: <7f014ea60905211605p130dce59lcae07171c7b30fbf@mail.gmail.com>
	<7f014ea60905211716m3dddc701udaa60e72cef67fdf@mail.gmail.com>
	
	<7f014ea60905212213q5da215d5s9a2728bbf929b3a6@mail.gmail.com>
Message-ID: 

Chris Colbert wrote:
> so if I have:
>
> cdef class IplImage:
>        cdef c_lib.IplImage* thisptr
>
>
> cdef class CvMat:
>       cdef c_lib.CvMat* thisptr
>
> then I can just do this?
>
> def exposedPythonFunc(image):   where image can be an IplImage or CvMat
>       c_lib.cFunction(image.thisptr)      since I know thisptr exists
>
> or must I do?
>
> def exposedPythonFunc(image):
>      if isinstance(image, IplImage):
>             c_lib.cFunction(image.thisptr)
>
>     elif isinstance(image, CvMat):
>            c_lib.cFunction(image.thisptr)
>
>     else:
>            raise NotImplementedError

Worse:

def exposedPythonFunc(image):
     if isinstance(image, IplImage):
            c_lib.cFunction((image).thisptr)
     elif isinstance(image, CvMat):
           c_lib.cFunction((image).thisptr)
...

Dag Sverre


From sccolbert at gmail.com  Fri May 22 07:43:45 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Fri, 22 May 2009 01:43:45 -0400
Subject: [Cython] how to pass opaque pointers to functions?
In-Reply-To: 
References: <7f014ea60905211605p130dce59lcae07171c7b30fbf@mail.gmail.com>
	<7f014ea60905211716m3dddc701udaa60e72cef67fdf@mail.gmail.com>
	
	<7f014ea60905212213q5da215d5s9a2728bbf929b3a6@mail.gmail.com>
	
Message-ID: <7f014ea60905212243j4d713606n453d09830009e055@mail.gmail.com>

lol, OK thanks!

Just for my own education which parts of this will translate to C and which
will remain Python after Cython compilation?

Chris


> Worse:
>
> def exposedPythonFunc(image):
>     if isinstance(image, IplImage):
>            c_lib.cFunction((image).thisptr)
>     elif isinstance(image, CvMat):
>           c_lib.cFunction((image).thisptr)
> ...
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090522/d5586b79/attachment.htm 

From dagss at student.matnat.uio.no  Fri May 22 07:59:41 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 22 May 2009 07:59:41 +0200 (CEST)
Subject: [Cython] how to pass opaque pointers to functions?
In-Reply-To: <7f014ea60905212243j4d713606n453d09830009e055@mail.gmail.com>
References: <7f014ea60905211605p130dce59lcae07171c7b30fbf@mail.gmail.com>
	<7f014ea60905211716m3dddc701udaa60e72cef67fdf@mail.gmail.com>
	
	<7f014ea60905212213q5da215d5s9a2728bbf929b3a6@mail.gmail.com>
	
	<7f014ea60905212243j4d713606n453d09830009e055@mail.gmail.com>
Message-ID: 

Chris Colbert wrote:
> lol, OK thanks!
>
> Just for my own education which parts of this will translate to C and
> which
> will remain Python after Cython compilation?

I'd have to check myself -- isinstance has been subject to various
optimizations and I'm not sure where it stands now. You can just try it
out, perhaps with "cython -a".

Dag Sverre


From kxroberto at googlemail.com  Fri May 22 09:41:36 2009
From: kxroberto at googlemail.com (Robert)
Date: Fri, 22 May 2009 09:41:36 +0200
Subject: [Cython] cdef extern from "" ?
In-Reply-To: <4A15EFC8.8080602@canterbury.ac.nz>
References: 		
	<4A15EFC8.8080602@canterbury.ac.nz>
Message-ID: 

Greg Ewing wrote:
> Robert wrote:
> 
>> just with "gcc -MM modname.c" (create dependencies) it makes 
>> problems. he errors on missing ""-includes, but not on 
>> <>-includes, when -I is omitted.
> 
> The generated C file seems like the wrong level to
> be tracking dependencies at, to me.
> 

it establishes, like in normal c projects, correctly tracking the 
possibly many complex dependencies at the c compilation stage. E.g 
when you mix many normal .c/.h modules with a .pyx as "python 
head", and one of the indirectly included .h files changes ...
And thats an issue for me.

And those dep's are effective after the .pyx -> .c rule. Its a 
separate thing. So for the Cython compilation stage one just has 
to add manually the typically few Cython-only specific dependies - 
not the complex .h file stack etc:

pyxmod.c : pyxmod.pyx xy.pxd abc.pxi ...


maybe even a kind of "cython -MM mymod.pyx >> .depend"  or so 
could in the future auto-produce the additional cython level 
dependencies, and the whole deps chain could become automatic. But 
that may be pedantic, because there are just very few things 
typically on that level - which one can easily put manually..


Robert


From dalcinl at gmail.com  Fri May 22 16:17:45 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Fri, 22 May 2009 11:17:45 -0300
Subject: [Cython] how to pass opaque pointers to functions?
In-Reply-To: <7f014ea60905212149ld9bd849r70f74f57ff872667@mail.gmail.com>
References: <7f014ea60905211605p130dce59lcae07171c7b30fbf@mail.gmail.com>
	<7f014ea60905211716m3dddc701udaa60e72cef67fdf@mail.gmail.com>
	
	<7f014ea60905212149ld9bd849r70f74f57ff872667@mail.gmail.com>
Message-ID: 

On Fri, May 22, 2009 at 1:49 AM, Chris Colbert  wrote:
> is 'handle()' a method that is called internally?when passing extension
> types as parameters?
>

No, it is a "polymorphic" method you have to manually call yourself.
Instead of doing "self.thisptr", you do "self.handle()", just to extra
parents.

> I'm exposing a C function to Python as follows:
>
> def cvSomeFunction(CvArr image):? #image should be able to be CvArr IplImage
> or CvMat
> ???? c_lib.cvSomeFunction(image.thisptr)??? # C func prototype = void
> cvSomeFunction(CvArr* image) and image can be CvArr* IplImage* or CvMat*
>

If you use my trick, the functions should be implemented like this:

def cvSomeFunction(CvArr image):
    c_lib.cvSomeFunction(image.handle())

Did you get the idea ?? Perhaps I'm not being clear enough.

-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From sccolbert at gmail.com  Fri May 22 17:05:20 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Fri, 22 May 2009 11:05:20 -0400
Subject: [Cython] how to pass opaque pointers to functions?
In-Reply-To: 
References: <7f014ea60905211605p130dce59lcae07171c7b30fbf@mail.gmail.com>
	<7f014ea60905211716m3dddc701udaa60e72cef67fdf@mail.gmail.com>
	
	<7f014ea60905212149ld9bd849r70f74f57ff872667@mail.gmail.com>
	
Message-ID: <7f014ea60905220805i3ffd803em13b1c52bd56dd189@mail.gmail.com>

Ok, I get what your saying. that's a good idea and I think it's a little
cleaner than casting during the call.

Thanks!

Chris

On Fri, May 22, 2009 at 10:17 AM, Lisandro Dalcin  wrote:

> On Fri, May 22, 2009 at 1:49 AM, Chris Colbert 
> wrote:
> > is 'handle()' a method that is called internally when passing extension
> > types as parameters?
> >
>
> No, it is a "polymorphic" method you have to manually call yourself.
> Instead of doing "self.thisptr", you do "self.handle()", just to extra
> parents.
>
> > I'm exposing a C function to Python as follows:
> >
> > def cvSomeFunction(CvArr image):  #image should be able to be CvArr
> IplImage
> > or CvMat
> >      c_lib.cvSomeFunction(image.thisptr)    # C func prototype = void
> > cvSomeFunction(CvArr* image) and image can be CvArr* IplImage* or CvMat*
> >
>
> If you use my trick, the functions should be implemented like this:
>
> def cvSomeFunction(CvArr image):
>    c_lib.cvSomeFunction(image.handle())
>
> Did you get the idea ?? Perhaps I'm not being clear enough.
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090522/21ab4263/attachment.htm 

From sccolbert at gmail.com  Fri May 22 17:58:28 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Fri, 22 May 2009 11:58:28 -0400
Subject: [Cython] how to pass opaque pointers to functions?
In-Reply-To: <7f014ea60905220805i3ffd803em13b1c52bd56dd189@mail.gmail.com>
References: <7f014ea60905211605p130dce59lcae07171c7b30fbf@mail.gmail.com>
	<7f014ea60905211716m3dddc701udaa60e72cef67fdf@mail.gmail.com>
	
	<7f014ea60905212149ld9bd849r70f74f57ff872667@mail.gmail.com>
	
	<7f014ea60905220805i3ffd803em13b1c52bd56dd189@mail.gmail.com>
Message-ID: <7f014ea60905220858u500c072fsed05559450911263@mail.gmail.com>

well, i tried your way Lisandro, and it compiled fine, but upon importing
the module, it raised an error along the lines of "it appears IplImage is
not of the right type".

Dag's method works, but I'm still gonna try to find something cleaner.

Chris

On Fri, May 22, 2009 at 11:05 AM, Chris Colbert  wrote:

> Ok, I get what your saying. that's a good idea and I think it's a little
> cleaner than casting during the call.
>
> Thanks!
>
> Chris
>
>
> On Fri, May 22, 2009 at 10:17 AM, Lisandro Dalcin wrote:
>
>> On Fri, May 22, 2009 at 1:49 AM, Chris Colbert 
>> wrote:
>> > is 'handle()' a method that is called internally when passing extension
>> > types as parameters?
>> >
>>
>> No, it is a "polymorphic" method you have to manually call yourself.
>> Instead of doing "self.thisptr", you do "self.handle()", just to extra
>> parents.
>>
>> > I'm exposing a C function to Python as follows:
>> >
>> > def cvSomeFunction(CvArr image):  #image should be able to be CvArr
>> IplImage
>> > or CvMat
>> >      c_lib.cvSomeFunction(image.thisptr)    # C func prototype = void
>> > cvSomeFunction(CvArr* image) and image can be CvArr* IplImage* or CvMat*
>> >
>>
>> If you use my trick, the functions should be implemented like this:
>>
>> def cvSomeFunction(CvArr image):
>>    c_lib.cvSomeFunction(image.handle())
>>
>> Did you get the idea ?? Perhaps I'm not being clear enough.
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090522/48f51bb8/attachment.htm 

From nmd at illinois.edu  Fri May 22 18:14:52 2009
From: nmd at illinois.edu (Nathan Dunfield)
Date: Fri, 22 May 2009 11:14:52 -0500
Subject: [Cython] Problem with 0.11.2
Message-ID: <95C3E04A-6FDC-4F28-B669-73A9E08B6A39@illinois.edu>

Dear Cython folks,

Upgrading to 0.11.2 from 0.11.1 breaks the following code (using  
Python 2.6.2 under OS X.5):

-----------------------------------------
cdef extern from "ntest-def.h":
    ctypedef char Boolean

cdef public Boolean gLongComputationInProgress
------------------------------------------

with resulting error:

----------------------------------------
cythoning ntest.pyx to ntest.c

Error converting Pyrex file to C:
------------------------------------------------------------
...
cdef extern from "ntest-def.h":
    ctypedef char Boolean

cdef public Boolean gLongComputationInProgress
                   ^
------------------------------------------------------------

/Users/dunfield/work/SnapPy/ntest/ntest.pyx:4:20: Compiler crash in  
AnalyseDeclarationsTransform

ModuleNode.body = StatListNode(ntest.pyx:1:0)
StatListNode.stats[1] = CVarDefNode(ntest.pyx:4:5,
    need_properties = [...]/1,
    visibility = u'public')
File 'Nodes.py', line 2778, in analyse_declarations:  
PropertyNode(ntest.pyx:4:20,
    name = u'gLongComputationInProgress')

Compiler crash traceback from this point on:
  File "/Library/Frameworks/Python.framework/Versions/2.6/lib/ 
python2.6/site-packages/Cython-0.11.2-py2.6-macosx-10.3-fat.egg/Cython/ 
Compiler/Nodes.py", line 2779, in analyse_declarations
    entry = env.declare_property(self.name, self.doc, self.pos)
AttributeError: 'ModuleScope' object has no attribute 'declare_property'
building 'ntest' extension
-------------------------------------------------------------

Replacing the initial code with

--------------------------------------------------
ctypedef char Boolean

cdef public Boolean gLongComputationInProgress
--------------------------------------------------

fixes the problem, though I'm not sure why.

While I have not filed a bug report on this, though I'm happy to do so  
if requested.  I've attached a minimal example exhibiting the bug that  
conversely works fine with Cython 0.11.1.

	Best,

	Nathan



-------------- next part --------------
A non-text attachment was scrubbed...
Name: ntest.tgz
Type: application/octet-stream
Size: 427 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20090522/6378aacf/attachment.obj 
-------------- next part --------------


From sccolbert at gmail.com  Fri May 22 18:12:14 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Fri, 22 May 2009 12:12:14 -0400
Subject: [Cython] how to pass opaque pointers to functions?
In-Reply-To: <7f014ea60905220858u500c072fsed05559450911263@mail.gmail.com>
References: <7f014ea60905211605p130dce59lcae07171c7b30fbf@mail.gmail.com>
	<7f014ea60905211716m3dddc701udaa60e72cef67fdf@mail.gmail.com>
	
	<7f014ea60905212149ld9bd849r70f74f57ff872667@mail.gmail.com>
	
	<7f014ea60905220805i3ffd803em13b1c52bd56dd189@mail.gmail.com>
	<7f014ea60905220858u500c072fsed05559450911263@mail.gmail.com>
Message-ID: <7f014ea60905220912p5b05a1ei774cb13f1148f28e@mail.gmail.com>

ok, maybe it wasnt your method. Something weird is going on, I removed the
inheritance from the class and it still compiled fine but drew errors when
importing from python.

Here are the relevant .pxd and .pxy files:


/********* .pxd ***********/
cimport c_cxcore
cimport c_highgui

cdef class CvArr:
    cdef c_cxcore.CvArr* handle(self)

cdef class IplImage:
    cdef c_cxcore.IplImage* thisptr
    cdef int needs_free
    cdef c_cxcore.CvArr* handle(self)


/******** .pxy ***********/

cimport c_cxcore
cimport c_highgui

cdef class CvArr:

    cdef c_cxcore.CvArr* handle(self):
        raise NotImplementedError

cdef class IplImage:

    cdef c_cxcore.CvArr* handle(self):
        return (self.thisptr)

/********** python import error **********/

>>> import cyopencv
Traceback (most recent call last):
  File "", line 1, in 
  File "cyopencv.pyx", line 2, in cyopencv (cyopencv.c:1062)
    from cy_cxcore import *
  File "cy_cvtypes.pxd", line 6, in cy_cxcore (cy_cxcore.c:821)

ValueError: cy_cvtypes.IplImage does not appear to be the correct type
object


/****** line 821 in cy_cxcore.c *********/

 /*--- Type import code ---*/
  __pyx_ptype_10cy_cvtypes_IplImage = __Pyx_ImportType("cy_cvtypes",
"IplImage", sizeof(struct __pyx_obj_10cy_cvtypes_IplImage)); if
(unlikely(!__pyx_ptype_10cy_cvtypes_IplImage)) {__pyx_filename = __pyx_f[1];
__pyx_lineno = 6; __pyx_clineno = __LINE__; goto __pyx_L1_error;}

/******** line 1062 in cyopencv.c *************/

__pyx_1 = __Pyx_Import(__pyx_kp_cy_cxcore, ((PyObject *)__pyx_t_1)); if
(unlikely(!__pyx_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 2;
__pyx_clineno = __LINE__; goto __pyx_L1_error;}





I don't know why its saying the object is unlikely the right type..

Chris




On Fri, May 22, 2009 at 11:58 AM, Chris Colbert  wrote:

> well, i tried your way Lisandro, and it compiled fine, but upon importing
> the module, it raised an error along the lines of "it appears IplImage is
> not of the right type".
>
> Dag's method works, but I'm still gonna try to find something cleaner.
>
> Chris
>
>
> On Fri, May 22, 2009 at 11:05 AM, Chris Colbert wrote:
>
>> Ok, I get what your saying. that's a good idea and I think it's a little
>> cleaner than casting during the call.
>>
>> Thanks!
>>
>> Chris
>>
>>
>> On Fri, May 22, 2009 at 10:17 AM, Lisandro Dalcin wrote:
>>
>>> On Fri, May 22, 2009 at 1:49 AM, Chris Colbert 
>>> wrote:
>>> > is 'handle()' a method that is called internally when passing extension
>>> > types as parameters?
>>> >
>>>
>>> No, it is a "polymorphic" method you have to manually call yourself.
>>> Instead of doing "self.thisptr", you do "self.handle()", just to extra
>>> parents.
>>>
>>> > I'm exposing a C function to Python as follows:
>>> >
>>> > def cvSomeFunction(CvArr image):  #image should be able to be CvArr
>>> IplImage
>>> > or CvMat
>>> >      c_lib.cvSomeFunction(image.thisptr)    # C func prototype = void
>>> > cvSomeFunction(CvArr* image) and image can be CvArr* IplImage* or
>>> CvMat*
>>> >
>>>
>>> If you use my trick, the functions should be implemented like this:
>>>
>>> def cvSomeFunction(CvArr image):
>>>    c_lib.cvSomeFunction(image.handle())
>>>
>>> Did you get the idea ?? Perhaps I'm not being clear enough.
>>>
>>> --
>>> Lisandro Dalc?n
>>> ---------------
>>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>>> Tel/Fax: +54-(0)342-451.1594
>>> _______________________________________________
>>> Cython-dev mailing list
>>> Cython-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/cython-dev
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090522/eee5a148/attachment.htm 

From sccolbert at gmail.com  Fri May 22 18:20:59 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Fri, 22 May 2009 12:20:59 -0400
Subject: [Cython] how to pass opaque pointers to functions?
In-Reply-To: <7f014ea60905220912p5b05a1ei774cb13f1148f28e@mail.gmail.com>
References: <7f014ea60905211605p130dce59lcae07171c7b30fbf@mail.gmail.com>
	<7f014ea60905211716m3dddc701udaa60e72cef67fdf@mail.gmail.com>
	
	<7f014ea60905212149ld9bd849r70f74f57ff872667@mail.gmail.com>
	
	<7f014ea60905220805i3ffd803em13b1c52bd56dd189@mail.gmail.com>
	<7f014ea60905220858u500c072fsed05559450911263@mail.gmail.com>
	<7f014ea60905220912p5b05a1ei774cb13f1148f28e@mail.gmail.com>
Message-ID: <7f014ea60905220920r14eb0a08nda9574853e2c0ab0@mail.gmail.com>

alright I figured it out.

Lisandro, your method worked perfect. the problem was in my function
prototype in the .pxd file.

I had to change it to this (fixed the argument types):

/********* .pxd ***********/
cimport c_cxcore
cimport c_highgui

cdef class CvArr:
    cdef c_cxcore.CvArr* handle(CvArr)

cdef class IplImage(CvArr):
    cdef c_cxcore.IplImage* thisptr
    cdef int needs_free
    cdef c_cxcore.CvArr* handle(IplImage)



thanks for the help guys!

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090522/30d3a779/attachment.htm 

From sccolbert at gmail.com  Fri May 22 18:24:53 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Fri, 22 May 2009 12:24:53 -0400
Subject: [Cython] how to pass opaque pointers to functions?
In-Reply-To: <7f014ea60905220920r14eb0a08nda9574853e2c0ab0@mail.gmail.com>
References: <7f014ea60905211605p130dce59lcae07171c7b30fbf@mail.gmail.com>
	<7f014ea60905211716m3dddc701udaa60e72cef67fdf@mail.gmail.com>
	
	<7f014ea60905212149ld9bd849r70f74f57ff872667@mail.gmail.com>
	
	<7f014ea60905220805i3ffd803em13b1c52bd56dd189@mail.gmail.com>
	<7f014ea60905220858u500c072fsed05559450911263@mail.gmail.com>
	<7f014ea60905220912p5b05a1ei774cb13f1148f28e@mail.gmail.com>
	<7f014ea60905220920r14eb0a08nda9574853e2c0ab0@mail.gmail.com>
Message-ID: <7f014ea60905220924y3490503dtd8148a37495384ef@mail.gmail.com>

I also had to add a dummy pointer to the CvArr class

/********* .pxd ***********/
cimport c_cxcore
cimport c_highgui

cdef class CvArr:
    cdef c_cxcore.CvArr* dummyptr
    cdef c_cxcore.CvArr* handle(CvArr)

cdef class IplImage(CvArr):
    cdef c_cxcore.IplImage* thisptr
    cdef int needs_free
    cdef c_cxcore.CvArr* handle(IplImage)



/********* .pxy *************/

cdef class CvArr:

      cdef c_cxcore.CvArr* handle(self):
             return self.dummyptr


On Fri, May 22, 2009 at 12:20 PM, Chris Colbert  wrote:

> alright I figured it out.
>
> Lisandro, your method worked perfect. the problem was in my function
> prototype in the .pxd file.
>
> I had to change it to this (fixed the argument types):
>
> /********* .pxd ***********/
> cimport c_cxcore
> cimport c_highgui
>
> cdef class CvArr:
>     cdef c_cxcore.CvArr* handle(CvArr)
>
> cdef class IplImage(CvArr):
>     cdef c_cxcore.IplImage* thisptr
>     cdef int needs_free
>     cdef c_cxcore.CvArr* handle(IplImage)
>
>
>
> thanks for the help guys!
>
> Chris
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090522/4b309b41/attachment.htm 

From dagss at student.matnat.uio.no  Fri May 22 19:19:05 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 22 May 2009 19:19:05 +0200 (CEST)
Subject: [Cython] Problem with 0.11.2
In-Reply-To: <95C3E04A-6FDC-4F28-B669-73A9E08B6A39@illinois.edu>
References: <95C3E04A-6FDC-4F28-B669-73A9E08B6A39@illinois.edu>
Message-ID: <5373b668ef3313775f06e0be085a834e.squirrel@webmail.uio.no>

Nathan Dunfield wrote:
> Replacing the initial code with
>
> --------------------------------------------------
> ctypedef char Boolean
>
> cdef public Boolean gLongComputationInProgress
> --------------------------------------------------
>
> fixes the problem, though I'm not sure why.
>
> While I have not filed a bug report on this, though I'm happy to do so
> if requested.  I've attached a minimal example exhibiting the bug that
> conversely works fine with Cython 0.11.1.

This is definitely my fault :-( I'll get a fix up today or tomorrow.

What do people think -- should we do a 0.11.2.1 or 0.11.3 at once with a
fix, or should we wait a week to see if there are more problems?

Dag Sverre


From robertwb at math.washington.edu  Fri May 22 20:53:59 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 22 May 2009 11:53:59 -0700
Subject: [Cython] Problem with 0.11.2
In-Reply-To: <5373b668ef3313775f06e0be085a834e.squirrel@webmail.uio.no>
References: <95C3E04A-6FDC-4F28-B669-73A9E08B6A39@illinois.edu>
	<5373b668ef3313775f06e0be085a834e.squirrel@webmail.uio.no>
Message-ID: 

On May 22, 2009, at 10:19 AM, Dag Sverre Seljebotn wrote:

> Nathan Dunfield wrote:
>> Replacing the initial code with
>>
>> --------------------------------------------------
>> ctypedef char Boolean
>>
>> cdef public Boolean gLongComputationInProgress
>> --------------------------------------------------
>>
>> fixes the problem, though I'm not sure why.
>>
>> While I have not filed a bug report on this, though I'm happy to  
>> do so
>> if requested.  I've attached a minimal example exhibiting the bug  
>> that
>> conversely works fine with Cython 0.11.1.
>
> This is definitely my fault :-( I'll get a fix up today or tomorrow.
>
> What do people think -- should we do a 0.11.2.1 or 0.11.3 at once  
> with a
> fix, or should we wait a week to see if there are more problems?

Let's wait a week. If this is the only fix, it should probably just  
be a 0.11.2.1. We can push the fix to the http://hg.cython.org/cython  
though.

- Robert


From robertwb at math.washington.edu  Sat May 23 00:00:15 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 22 May 2009 15:00:15 -0700
Subject: [Cython] cdef extern from "" ?
In-Reply-To: <4A159CE1.3050202@student.matnat.uio.no>
References: 			<36fa15ada30e707eb1802a114d26968e.squirrel@webmail.uio.no>
	
	<4A159CE1.3050202@student.matnat.uio.no>
Message-ID: 

On May 21, 2009, at 11:26 AM, Dag Sverre Seljebotn wrote:

> Because we have to support projects which need to include both "./ 
> foo.h"
> and "/usr/include/foo.h" as distinct header files in the same pyx  
> file?

I concur that this isn't a good development practice in nearly all  
cases, but I don't think that's a compelling enough argument to not  
support it.

- Robert


From rdingman at u.washington.edu  Sat May 23 01:21:22 2009
From: rdingman at u.washington.edu (Ryan Dingman)
Date: Fri, 22 May 2009 16:21:22 -0700
Subject: [Cython] Python objects as members of C structs
Message-ID: <383e1c5b0905221621o1644b3b5u27a3073b2e05f908@mail.gmail.com>

How can I store a reference to a Python object in a C struct? If I try
to define a C struct as follows:

ctypedef struct my_struct:
   object my_object

Cython tells me "C struct/union member cannot be a Python object". So,
I thought I'd try storing the objects as void *

ctypedef struct my_struct:
   void *my_object

But this is obviously problematic because when I make an assignment:

my_struct.my_object = some_object

The proper ref counting won't occur. Is there a way that I can
manually ref count this object? Is there another approach for storing
Python objects in a struct?

Any help would be appreciated.

Ryan

From robertwb at math.washington.edu  Sat May 23 01:45:54 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 22 May 2009 16:45:54 -0700
Subject: [Cython] Python objects as members of C structs
In-Reply-To: <383e1c5b0905221621o1644b3b5u27a3073b2e05f908@mail.gmail.com>
References: <383e1c5b0905221621o1644b3b5u27a3073b2e05f908@mail.gmail.com>
Message-ID: <7F03327A-0DA1-4E0A-9E5A-8354C9847C62@math.washington.edu>

On May 22, 2009, at 4:21 PM, Ryan Dingman wrote:

> How can I store a reference to a Python object in a C struct? If I try
> to define a C struct as follows:
>
> ctypedef struct my_struct:
>    object my_object
>
> Cython tells me "C struct/union member cannot be a Python object". So,
> I thought I'd try storing the objects as void *
>
> ctypedef struct my_struct:
>    void *my_object
>
> But this is obviously problematic because when I make an assignment:
>
> my_struct.my_object = some_object
>
> The proper ref counting won't occur. Is there a way that I can
> manually ref count this object? Is there another approach for storing
> Python objects in a struct?
>
> Any help would be appreciated.

The reason it is disallowed is because Cython is unable to do the ref- 
counting automatically. Your approach of casting to/from a void* is  
the right one to do here (but the duty is on you to be careful about  
it).

To do things manually, import the relevant functions as

from python_ref cimport Py_DECREF, Py_INCREF

See http://hg.cython.org/cython/file/c6388e2a9739/Cython/Includes/ 
python_ref.pxd

- Robert


From tenninosko at msn.com  Tue May 26 16:35:42 2009
From: tenninosko at msn.com (OUARDA MEHDI)
Date: Tue, 26 May 2009 16:35:42 +0200
Subject: [Cython] cython code optimisation...misc
Message-ID: 


hello,
in fact I'm rewriting (in cython 2.6)  a python code of a simulator in order to optimize the slowest functions.
i've a couple of questions....
is there any equivalent of abstractproprety, abstractmethod in cython ?and how to use it
is there any way to inherit from builtin classes in cython like class Exception?... any trick to do something equivalent will be welcome
here is an exemple:
class DuplicateNameError(Exception):
    pass 
when i define my class as cpdef... an error occurs while compiling...here is the message i get:
/home/tenninos/stage/arboris-python/src/misc.pyx:19:6: 'Exception' is not a type name building 'misc' extension

thanks a lot

Cheers!

_________________________________________________________________
In?dit ! Des Emotic?nes D?jant?es! Installez les dans votre Messenger ! 
http://www.ilovemessenger.fr/Emoticones/EmoticonesDejantees.aspx
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090526/e1a66518/attachment-0001.htm 

From dalcinl at gmail.com  Tue May 26 22:51:53 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 26 May 2009 17:51:53 -0300
Subject: [Cython] cython code optimisation...misc
In-Reply-To: 
References: 
Message-ID: 

On Tue, May 26, 2009 at 11:35 AM, OUARDA MEHDI  wrote:
> hello,
> in fact I'm rewriting (in cython 2.6)? a python code of a simulator in order
> to optimize the slowest functions.

Did you mean Python 2.6?

> i've a couple of questions....
> is there any equivalent of abstractproprety, abstractmethod in cython ?and
> how to use it

Are talking about the new ABC stuff in Py2.6/3.0? Mmm... not sure if
they will work.

> is there any way to inherit from builtin classes in cython like class
> Exception?... any trick to do something equivalent will be welcome
> here is an exemple:
> class DuplicateNameError(Exception):
> ??? pass
> when i define my class as cpdef... an error occurs while compiling...here is
> the message i get:
> /home/tenninos/stage/arboris-python/src/misc.pyx:19:6: 'Exception' is not a
> type name building 'misc' extension
>

Do you really need your exceptions to be cdef classes? That would not
work on Py < 2.5... Anyway (perhaps being compatible with older Python
is not an issue for you), look the first lines of this code, that
should work. Disclaimer: I've not tested this for some time, that code
is there just for reference, I'm not currently using it in my project
just because of the compatibility with Py2.3 and Py2.4:

http://code.google.com/p/mpi4py/source/browse/trunk/src/MPI/ExceptionC.pyx


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Tue May 26 23:00:12 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 26 May 2009 23:00:12 +0200
Subject: [Cython] Problem with 0.11.2
In-Reply-To: <5373b668ef3313775f06e0be085a834e.squirrel@webmail.uio.no>
References: <95C3E04A-6FDC-4F28-B669-73A9E08B6A39@illinois.edu>
	<5373b668ef3313775f06e0be085a834e.squirrel@webmail.uio.no>
Message-ID: <4A1C585C.10209@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Nathan Dunfield wrote:
>> Replacing the initial code with
>>
>> --------------------------------------------------
>> ctypedef char Boolean
>>
>> cdef public Boolean gLongComputationInProgress
>> --------------------------------------------------
>>
>> fixes the problem, though I'm not sure why.
>>
>> While I have not filed a bug report on this, though I'm happy to do so
>> if requested.  I've attached a minimal example exhibiting the bug that
>> conversely works fine with Cython 0.11.1.
> 
> This is definitely my fault :-( I'll get a fix up today or tomorrow.

Well, then nice weather hit and it seemed less attractive to waste my 
time on Cython :-)

A fix is up on hg.cython.org [1]. You can grab this by through the links 
here, direct link:

http://hg.cython.org/cython/archive/tip.tar.gz

It's not as extensively tested but it "should work" (as Robert spoke of 
we may do a minor release in about a week where we've tested this more).

(Note to self: More extensive tests in tests/run/typedfieldbug_T303.pyx 
needed for this before I'll close the #303 ticket.)

[1] http://hg.cython.org/cython/rev/11e826f95cde

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Tue May 26 23:03:57 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 26 May 2009 23:03:57 +0200
Subject: [Cython] #303 + simplifying type system
In-Reply-To: 
References: <4A099E0C.9000909@student.matnat.uio.no>		<4A0C30F5.6090101@student.matnat.uio.no>		<4A0C6C15.3090400@student.matnat.uio.no>		<4A0C7985.3090203@student.matnat.uio.no>
	
Message-ID: <4A1C593D.5050807@student.matnat.uio.no>

Lisandro Dalcin wrote:
> Wow... I'm very sorry. I got it totally wrong ...
> 
> If the change you are proposing about simplifying the type system only
> affect that, I'm actually +1 on your proposal. The canonical Python
> way is a mess in the context of Cython.
> 
> I would just ask for the following:
> 
> 1) When going from C to Python, for Python 2.X, try to avoid returning
> a PyLong (basically dispach proper calls using
> sizeof(type)<=sizeof(long) )
> 
> 2) Use __Pyx_T_FLOATING, just to avoid confussion.
> 
> 
> Of course, other way would be to just use the transform I proposed
> previously. I do not expect it to impact performance.

To follow up here: Since Robert was concerned and I'm too lazy to 
benchmark, I redid it. (This also fixes the recently reported bug of 
course.)

http://hg.cython.org/cython/rev/11e826f95cde


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Tue May 26 23:10:30 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 26 May 2009 23:10:30 +0200
Subject: [Cython] pyximport ignors [build] compiler = mingw32
In-Reply-To: 
References: 							<4A0F0942.3060307@behnel.de>	<311bf9168cc7ea937556a7391b54cdf1.squirrel@webmail.uio.no>			
	
Message-ID: <4A1C5AC6.50803@student.matnat.uio.no>

Robert wrote:
> Robert wrote:
>>
>> I've set up my Linux too now. That latest pyxi3.patch and it does.
>>
>> With reload_support=False its exactly like the old 0.11.1 behavior :
>>
>>  >>> import my   
>>  >>> reload(my)
>> Traceback (most recent call last):
>>    File "", line 1, in ?
>> ImportError: No module named my
>>
>> which is in my opinion bogus, because
>>  >>> reload(time)
>> 
>>
>>
>>
>> and with reload_support=True its now ok
>>
> 
> 
> I've done a few more tests on Linux and Win, and it does so far on my 
> side in all modes - for practical edit-run/reload debugging purposes. 
> The reload_support=False is anyway default.

Just a note that even though I started picking up this thread with all 
intentions on following up, I can't seem to find the time to review this 
for the next two weeks at least. If nobody else picks it up in the 
meantime I'll try to remember to get back to it, feel free to ping me if 
not.

Something to think about: It would be a lot easier to review 
pyximport-related changes if we had a test suite for pyximport. I 
suppose it wouldn't be very difficult to do:

1) Have a directory structure with py and pyx and various configuration 
files in it
2) Have a test script which just copies this whole directory and then 
runs doctest on test_*.py in the directory (each probably in a new 
Python instance, to get pyximport and pyx modules loaded/unloaded).

I made a wishlist ticket for this:

http://trac.cython.org/cython_trac/ticket/321

-- 
Dag Sverre

From dalcinl at gmail.com  Wed May 27 00:44:44 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 26 May 2009 19:44:44 -0300
Subject: [Cython] #303 + simplifying type system
In-Reply-To: <4A1C593D.5050807@student.matnat.uio.no>
References: <4A099E0C.9000909@student.matnat.uio.no>
	
	<4A0C30F5.6090101@student.matnat.uio.no>
	
	<4A0C6C15.3090400@student.matnat.uio.no>
	
	<4A0C7985.3090203@student.matnat.uio.no>
	
	<4A1C593D.5050807@student.matnat.uio.no>
Message-ID: 

On Tue, May 26, 2009 at 6:03 PM, Dag Sverre Seljebotn
 wrote:
> Lisandro Dalcin wrote:
>> Wow... I'm very sorry. I got it totally wrong ...
>>
>> If the change you are proposing about simplifying the type system only
>> affect that, I'm actually +1 on your proposal. The canonical Python
>> way is a mess in the context of Cython.
>>
>> I would just ask for the following:
>>
>> 1) When going from C to Python, for Python 2.X, try to avoid returning
>> a PyLong (basically dispach proper calls using
>> sizeof(type)<=sizeof(long) )
>>
>> 2) Use __Pyx_T_FLOATING, just to avoid confussion.
>>
>>
>> Of course, other way would be to just use the transform I proposed
>> previously. I do not expect it to impact performance.
>
> To follow up here: Since Robert was concerned and I'm too lazy to
> benchmark, I redid it. (This also fixes the recently reported bug of
> course.)
>

But you are really, really lazy ;-) ... Try this:

$ python -m timeit -n 10000000 -s 'import sys; f=sys.stdout' 'f.softspace'
10000000 loops, best of 3: 0.218 usec per loop

$ python -m timeit -n 10000000 -s 'import sys; f=sys.stdout' 'f.closed'
10000000 loops, best of 3: 0.194 usec per loop


So very little difference... Dict lookups likely dominate:

$ python -m timeit -n 10000000 -s 'd=file.__dict__' 'd["closed"]'
10000000 loops, best of 3: 0.177 usec per loop
$ python -m timeit -n 10000000 -s 'd=file.__dict__' 'd["softspace"]'
10000000 loops, best of 3: 0.168 usec per loop


Even if we ignore the overhead of dict lookups, a PyMemberDef-based
attribute (the outcome of 'cdef public something') should have almost
identical overhead to a PyGetSetDef-based attribute (the outcome of
'cdef property'). Moreover, the former (PyMemberDef) requires a
dispatching switch, then being perhaps a bit less efficient? The
numbers above seems to support me, though I should do a better test
(softspace is a 'int' and 'closed' a 'bool').


>
> http://hg.cython.org/cython/rev/11e826f95cde
>

Looks good. Though perhaps in the future we could review all this and
get rid at all of PyMemberDef-based attributes... IMHO, that is just
sugar for human C extension writers.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From sccolbert at gmail.com  Wed May 27 05:57:11 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Tue, 26 May 2009 23:57:11 -0400
Subject: [Cython] having a weird issue with wrapped function failing in only
	certain cases
Message-ID: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>

So I'm wrapping the OpenCV library in Cython and its coming along quite
nicely (thanks to everyone here for getting me up to speed!), but today I've
run across a strange issue where a wrapped function fails for certain values
of integer arguments, but a pure c implementation testing the same thing
with the same arguments works.
Here is the code excerpts im using:

#---------- .pyx ------------------#

def cvResize(cy_cvtypes.CvArr src, cy_cvtypes.CvArr dst, interpolation):
    cdef c_cxcore.CvArr* c_src = src.handle()
    cdef c_cxcore.CvArr* c_dst = dst.handle()
    cdef int c_interpolation = interpolation
    with nogil:
        c_cv.cvResize(c_src, c_dst, c_interpolation)

#-------- c_cv.pxd -----------------#

cdef extern from "cv.h":

    void cvResize(CvArr* src, CvArr* dst, int interpolation) nogil


#--------- test.py -----------------#

from cyopencv import *

if __name__=='__main__':
    img = cvLoadImage('2.bmp')
    img2 = cvCreateImage((640, 480), img.depth, img.nChannels)
    cvResize(img, img2, 3)
    img2.show()



#--------- pureC.c -----------#

#include "highgui.h"
#include "cv.h"
#include "cxcore.h"
#include "cxtypes.h"

int main(void)
{
    IplImage* img, *img2;
    img = cvLoadImage("2.bmp", 1);
    img2 = cvCreateImage(cvSize(640, 480), img->depth, img->nChannels);
    cvResize(img, img2, 3);
    cvNamedWindow("test", 1);
    cvShowImage("test", img2);
    cvWaitKey(0);
    cvDestroyAllWindows();
    cvReleaseImage(&img);
    cvReleaseImage(&img2);
}



valid integers for the argument 'interpolation' are 0, 1, 2, 3, 4 (as tested
with the C function). My Cython wrapper fails with a Python crash for
integer values 1 & 2.  Values of 0,1,4 work properly and the output image
matches the output image of the C test function under the same conditions.

I've looked at the generated Cython code and everything seems in order, I
just can't think of what could be causing this behavior.

Any thoughts?

Cheers!

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090526/b073ff8f/attachment.htm 

From tenninosko at msn.com  Wed May 27 11:15:34 2009
From: tenninosko at msn.com (OUARDA MEHDI)
Date: Wed, 27 May 2009 11:15:34 +0200
Subject: [Cython] cython code optimisation...misc
In-Reply-To: 
References: 
	
Message-ID: 


hi,
thanks for the code... that helped me to compile the *.pyx module.however, i still missing something: 
i thought that cdef class can't inherit python objects(at least for py 2.6).however,if i could avoid defining my functions
as cdef and it still works when using it with cython classes that would be ok for me...now the issue is this one, if i define my class like
this:
cdef extern from "Python.h":
    ctypedef class __builtin__.Exception [object PyBaseExceptionObject]:
        pass

cpdef class DuplicatedNameError(Exception):    
    pass                                                         


when a DuplicatedNameError occurs, did it calls the python build in class Exeption or shall' i impliment my self the class Exception as
in the expemle (http://code.google.com/p/mpi4py/source/browse/trunk/src/MPI/ExceptionC.pyx)


ps: 'cause there is always one
      is there any way to find an equivalent of abstractproprety, abstractmethod for *.pyx modules.that would be really helpful.
      finallly, yes indeed...i meant python 2.6 (newbie mistake ;-) 
      

> Date: Tue, 26 May 2009 17:51:53 -0300
> From: dalcinl at gmail.com
> To: cython-dev at codespeak.net
> Subject: Re: [Cython] cython code optimisation...misc
> 
> On Tue, May 26, 2009 at 11:35 AM, OUARDA MEHDI  wrote:
> > hello,
> > in fact I'm rewriting (in cython 2.6)  a python code of a simulator in order
> > to optimize the slowest functions.
> 
> Did you mean Python 2.6?
> 
> > i've a couple of questions....
> > is there any equivalent of abstractproprety, abstractmethod in cython ?and
> > how to use it
> 
> Are talking about the new ABC stuff in Py2.6/3.0? Mmm... not sure if
> they will work.
> 
> > is there any way to inherit from builtin classes in cython like class
> > Exception?... any trick to do something equivalent will be welcome
> > here is an exemple:
> > class DuplicateNameError(Exception):
> >     pass
> > when i define my class as cpdef... an error occurs while compiling...here is
> > the message i get:
> > /home/tenninos/stage/arboris-python/src/misc.pyx:19:6: 'Exception' is not a
> > type name building 'misc' extension
> >
> 
> Do you really need your exceptions to be cdef classes? That would not
> work on Py < 2.5... Anyway (perhaps being compatible with older Python
> is not an issue for you), look the first lines of this code, that
> should work. Disclaimer: I've not tested this for some time, that code
> is there just for reference, I'm not currently using it in my project
> just because of the compatibility with Py2.3 and Py2.4:
> 
> http://code.google.com/p/mpi4py/source/browse/trunk/src/MPI/ExceptionC.pyx
> 
> 
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev

_________________________________________________________________
Vous voulez savoir ce que vous pouvez faire avec le nouveau Windows Live?? Lancez-vous !
http://www.microsoft.com/windows/windowslive/default.aspx
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090527/3456d261/attachment.htm 

From dagss at student.matnat.uio.no  Wed May 27 11:38:28 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 27 May 2009 11:38:28 +0200
Subject: [Cython] cython code optimisation...misc
In-Reply-To: 
References: 	
	
Message-ID: <4A1D0A14.9090904@student.matnat.uio.no>

OUARDA MEHDI wrote:
> hi,
> thanks for the code... that helped me to compile the *.pyx 
> module.however, i still missing something:
> i thought that cdef class can't inherit python objects(at least for py 
> 2.6).however,if i could avoid defining my functions
> as cdef and it still works when using it with cython classes that 
> would be ok for me...now the issue is this one, if i define my class like
> this:
> cdef extern from "Python.h":
>     ctypedef class __builtin__.Exception [object PyBaseExceptionObject]:
>         pass
>
> cpdef class DuplicatedNameError(Exception):   
>     pass                                                        
>
> when a DuplicatedNameError occurs, did it calls the python build in 
> class Exeption or shall' i impliment my self the class Exception as
> in the expemle 
> (http://code.google.com/p/mpi4py/source/browse/trunk/src/MPI/ExceptionC.pyx)

For exceptions I think it is very, very unlikely that you actually need 
any cdef things at all. Can't you just use

class DuplicatedNameError(Exception):
    def somefunc(self): ...

? There's no need to add types unless you have to :-), and for 
exceptions one usually don't.

Dag Sverre

From dalcinl at gmail.com  Wed May 27 15:01:14 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 27 May 2009 10:01:14 -0300
Subject: [Cython] cython code optimisation...misc
In-Reply-To: 
References: 
	
	
Message-ID: 

On Wed, May 27, 2009 at 6:15 AM, OUARDA MEHDI  wrote:
>
> when a DuplicatedNameError occurs, did it calls the python build in class
> Exeption or shall' i impliment my self the class Exception as
> in the expemle
> (http://code.google.com/p/mpi4py/source/browse/trunk/src/MPI/ExceptionC.pyx)
>
>

No, No, please!!! That 'Exception' class in mpi4py code actually hides
the built-in "Exception". This is a really bad thing, but as I'm
following the MPI C++ bindings very closely, I did things like that.

In short, just do what you already did. Anyway, I agree with Dag's
comments, you should not need to use a cdef class for exceptions.

> ps: 'cause there is always one
> ????? is there any way to find an equivalent of abstractproprety,
> abstractmethod for *.pyx modules.that would be really helpful.
> ????? finallly, yes indeed...i meant python 2.6 (newbie mistake ;-)
>

Py2.6 'abstractproperty' will likely work, though 'abstractmethod' likely not...

If you can post a minimal (please, really minimal) example of what do
you want to do, perhaps I could give you a recipe. After that, I may
ask you to contribute by posting the recipe to the Cython wiki ;-)

>
>> Date: Tue, 26 May 2009 17:51:53 -0300
>> From: dalcinl at gmail.com
>> To: cython-dev at codespeak.net
>> Subject: Re: [Cython] cython code optimisation...misc
>>
>> On Tue, May 26, 2009 at 11:35 AM, OUARDA MEHDI  wrote:
>> > hello,
>> > in fact I'm rewriting (in cython 2.6)? a python code of a simulator in
>> > order
>> > to optimize the slowest functions.
>>
>> Did you mean Python 2.6?
>>
>> > i've a couple of questions....
>> > is there any equivalent of abstractproprety, abstractmethod in cython
>> > ?and
>> > how to use it
>>
>> Are talking about the new ABC stuff in Py2.6/3.0? Mmm... not sure if
>> they will work.
>>
>> > is there any way to inherit from builtin classes in cython like class
>> > Exception?... any trick to do something equivalent will be welcome
>> > here is an exemple:
>> > class DuplicateNameError(Exception):
>> > ??? pass
>> > when i define my class as cpdef... an error occurs while
>> > compiling...here is
>> > the message i get:
>> > /home/tenninos/stage/arboris-python/src/misc.pyx:19:6: 'Exception' is
>> > not a
>> > type name building 'misc' extension
>> >
>>
>> Do you really need your exceptions to be cdef classes? That would not
>> work on Py < 2.5... Anyway (perhaps being compatible with older Python
>> is not an issue for you), look the first lines of this code, that
>> should work. Disclaimer: I've not tested this for some time, that code
>> is there just for reference, I'm not currently using it in my project
>> just because of the compatibility with Py2.3 and Py2.4:
>>
>> http://code.google.com/p/mpi4py/source/browse/trunk/src/MPI/ExceptionC.pyx
>>
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
> ________________________________
> Vous voulez savoir ce que vous pouvez faire avec le nouveau Windows Live??
> Lancez-vous !
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Wed May 27 15:40:29 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 27 May 2009 10:40:29 -0300
Subject: [Cython] having a weird issue with wrapped function failing in
	only certain cases
In-Reply-To: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>
References: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>
Message-ID: 

According to the docs,
http://opencv.willowgarage.com/wiki/CvReference#Sampling.2CInterpolationandGeometricalTransforms,
There are 4 possible values (you mention 5). Is that OpenCV
documentation dated? IMHO, you should use them. To expose these
constants in Cython code (likely in some pxd), you can do this:

cdef extern from ....:

    enum: CV_INTER_NN
    enum: CV_INTER_LINEAR
    enum: CV_INTER_AREA
    enum: CV_INTER_CUBIC



On Wed, May 27, 2009 at 12:57 AM, Chris Colbert  wrote:
> So I'm wrapping the OpenCV library in Cython and its coming along quite
> nicely (thanks to everyone here for getting me up to speed!), but today I've
> run across a strange issue where a wrapped function fails for certain values
> of integer arguments, but a pure c implementation testing the same thing
> with the same arguments works.
> Here is the code?excerpts?im using:
> #---------- .pyx ------------------#
> def cvResize(cy_cvtypes.CvArr src, cy_cvtypes.CvArr dst, interpolation):
> ?? ?cdef c_cxcore.CvArr* c_src = src.handle()
> ?? ?cdef c_cxcore.CvArr* c_dst = dst.handle()
> ?? ?cdef int c_interpolation = interpolation
> ?? ?with nogil:
> ?? ? ? ?c_cv.cvResize(c_src, c_dst, c_interpolation)
> #-------- c_cv.pxd -----------------#
> cdef extern from "cv.h":
> ?? ?void cvResize(CvArr* src, CvArr* dst, int interpolation) nogil
>
> #--------- test.py -----------------#
> from cyopencv import *
> if __name__=='__main__':
> ?? ?img = cvLoadImage('2.bmp')
> ?? ?img2 = cvCreateImage((640, 480), img.depth, img.nChannels)
> ?? ?cvResize(img, img2, 3)
> ?? ?img2.show()
>
>
> #--------- pureC.c -----------#
> #include "highgui.h"
> #include "cv.h"
> #include "cxcore.h"
> #include "cxtypes.h"
> int main(void)
> {
> ?? ?IplImage* img, *img2;
> ?? ?img = cvLoadImage("2.bmp", 1);
> ?? ?img2 = cvCreateImage(cvSize(640, 480), img->depth, img->nChannels);
> ?? ?cvResize(img, img2, 3);
> ?? ?cvNamedWindow("test", 1);
> ?? ?cvShowImage("test", img2);
> ?? ?cvWaitKey(0);
> ?? ?cvDestroyAllWindows();
> ?? ?cvReleaseImage(&img);
> ?? ?cvReleaseImage(&img2);
> }
>
>
> valid integers for the argument 'interpolation' are 0, 1, 2, 3, 4 (as tested
> with the C function). My Cython wrapper fails with a Python crash for
> integer values 1 & 2. ?Values of 0,1,4 work properly and the output image
> matches the output image of the C test function under the same conditions.
> I've looked at the generated Cython code and everything seems in order, I
> just can't think of what could be causing this behavior.
> Any thoughts?
> Cheers!
> Chris
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From sccolbert at gmail.com  Wed May 27 16:36:41 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Wed, 27 May 2009 10:36:41 -0400
Subject: [Cython] having a weird issue with wrapped function failing in
	only certain cases
In-Reply-To: 
References: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>
	
Message-ID: <7f014ea60905270736v682b1257jdeeb8fbd230282ba@mail.gmail.com>

sometimes the docs are a bit dated. But nonetheless, those constants are
just integers defines in the header files and is where I got the values 0-4,
I used the integers in the example just to make it a bit more clear. But 0-4
are valid constants for the call, as they worked in the pure C example.
What you showed for exposing the constants however, is interesting, I have
been just defining the constants as module level python integers so they can
be used from python. If defined as an enum as you show, will they
be accessible from python?

Chris

On Wed, May 27, 2009 at 9:40 AM, Lisandro Dalcin  wrote:

> According to the docs,
>
> http://opencv.willowgarage.com/wiki/CvReference#Sampling.2CInterpolationandGeometricalTransforms
> ,
> There are 4 possible values (you mention 5). Is that OpenCV
> documentation dated? IMHO, you should use them. To expose these
> constants in Cython code (likely in some pxd), you can do this:
>
> cdef extern from ....:
>
>    enum: CV_INTER_NN
>    enum: CV_INTER_LINEAR
>    enum: CV_INTER_AREA
>    enum: CV_INTER_CUBIC
>
>
>
> On Wed, May 27, 2009 at 12:57 AM, Chris Colbert 
> wrote:
> > So I'm wrapping the OpenCV library in Cython and its coming along quite
> > nicely (thanks to everyone here for getting me up to speed!), but today
> I've
> > run across a strange issue where a wrapped function fails for certain
> values
> > of integer arguments, but a pure c implementation testing the same thing
> > with the same arguments works.
> > Here is the code excerpts im using:
> > #---------- .pyx ------------------#
> > def cvResize(cy_cvtypes.CvArr src, cy_cvtypes.CvArr dst, interpolation):
> >     cdef c_cxcore.CvArr* c_src = src.handle()
> >     cdef c_cxcore.CvArr* c_dst = dst.handle()
> >     cdef int c_interpolation = interpolation
> >     with nogil:
> >         c_cv.cvResize(c_src, c_dst, c_interpolation)
> > #-------- c_cv.pxd -----------------#
> > cdef extern from "cv.h":
> >     void cvResize(CvArr* src, CvArr* dst, int interpolation) nogil
> >
> > #--------- test.py -----------------#
> > from cyopencv import *
> > if __name__=='__main__':
> >     img = cvLoadImage('2.bmp')
> >     img2 = cvCreateImage((640, 480), img.depth, img.nChannels)
> >     cvResize(img, img2, 3)
> >     img2.show()
> >
> >
> > #--------- pureC.c -----------#
> > #include "highgui.h"
> > #include "cv.h"
> > #include "cxcore.h"
> > #include "cxtypes.h"
> > int main(void)
> > {
> >     IplImage* img, *img2;
> >     img = cvLoadImage("2.bmp", 1);
> >     img2 = cvCreateImage(cvSize(640, 480), img->depth, img->nChannels);
> >     cvResize(img, img2, 3);
> >     cvNamedWindow("test", 1);
> >     cvShowImage("test", img2);
> >     cvWaitKey(0);
> >     cvDestroyAllWindows();
> >     cvReleaseImage(&img);
> >     cvReleaseImage(&img2);
> > }
> >
> >
> > valid integers for the argument 'interpolation' are 0, 1, 2, 3, 4 (as
> tested
> > with the C function). My Cython wrapper fails with a Python crash for
> > integer values 1 & 2.  Values of 0,1,4 work properly and the output image
> > matches the output image of the C test function under the same
> conditions.
> > I've looked at the generated Cython code and everything seems in order, I
> > just can't think of what could be causing this behavior.
> > Any thoughts?
> > Cheers!
> > Chris
> >
> > _______________________________________________
> > Cython-dev mailing list
> > Cython-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/cython-dev
> >
> >
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090527/7487b7be/attachment-0001.htm 

From dalcinl at gmail.com  Wed May 27 16:49:47 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 27 May 2009 11:49:47 -0300
Subject: [Cython] having a weird issue with wrapped function failing in
	only certain cases
In-Reply-To: <7f014ea60905270736v682b1257jdeeb8fbd230282ba@mail.gmail.com>
References: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>
	
	<7f014ea60905270736v682b1257jdeeb8fbd230282ba@mail.gmail.com>
Message-ID: 

On Wed, May 27, 2009 at 11:36 AM, Chris Colbert  wrote:
> sometimes the docs are a bit dated. But nonetheless, those constants are
> just integers defines in the header files and is where I got the values 0-4,
> I used the integers in the example just to make it a bit more clear. But 0-4
> are valid constants for the call, as they worked in the pure C example.

Mmm... they I have no idea what's going on... Did you try to run under
valgrind? That usually works for me...

> What you showed for exposing the constants however, is interesting, I have
> been just defining the constants as module level python integers so they can
> be used from python. If defined as an enum as you show, will they
> be?accessible?from python?
>

No, you have to explicitly expose them. If you declared them in some
pxd, let say "mycvdefs.pxd", you can write in "mycvmodule.pyx"

CV_SOME_CONSTANT = mycvdefs.CV_SOME_CONSTANT





> Chris
>
> On Wed, May 27, 2009 at 9:40 AM, Lisandro Dalcin  wrote:
>>
>> According to the docs,
>>
>> http://opencv.willowgarage.com/wiki/CvReference#Sampling.2CInterpolationandGeometricalTransforms,
>> There are 4 possible values (you mention 5). Is that OpenCV
>> documentation dated? IMHO, you should use them. To expose these
>> constants in Cython code (likely in some pxd), you can do this:
>>
>> cdef extern from ....:
>>
>> ? ?enum: CV_INTER_NN
>> ? ?enum: CV_INTER_LINEAR
>> ? ?enum: CV_INTER_AREA
>> ? ?enum: CV_INTER_CUBIC
>>
>>
>>
>> On Wed, May 27, 2009 at 12:57 AM, Chris Colbert 
>> wrote:
>> > So I'm wrapping the OpenCV library in Cython and its coming along quite
>> > nicely (thanks to everyone here for getting me up to speed!), but today
>> > I've
>> > run across a strange issue where a wrapped function fails for certain
>> > values
>> > of integer arguments, but a pure c implementation testing the same thing
>> > with the same arguments works.
>> > Here is the code?excerpts?im using:
>> > #---------- .pyx ------------------#
>> > def cvResize(cy_cvtypes.CvArr src, cy_cvtypes.CvArr dst, interpolation):
>> > ?? ?cdef c_cxcore.CvArr* c_src = src.handle()
>> > ?? ?cdef c_cxcore.CvArr* c_dst = dst.handle()
>> > ?? ?cdef int c_interpolation = interpolation
>> > ?? ?with nogil:
>> > ?? ? ? ?c_cv.cvResize(c_src, c_dst, c_interpolation)
>> > #-------- c_cv.pxd -----------------#
>> > cdef extern from "cv.h":
>> > ?? ?void cvResize(CvArr* src, CvArr* dst, int interpolation) nogil
>> >
>> > #--------- test.py -----------------#
>> > from cyopencv import *
>> > if __name__=='__main__':
>> > ?? ?img = cvLoadImage('2.bmp')
>> > ?? ?img2 = cvCreateImage((640, 480), img.depth, img.nChannels)
>> > ?? ?cvResize(img, img2, 3)
>> > ?? ?img2.show()
>> >
>> >
>> > #--------- pureC.c -----------#
>> > #include "highgui.h"
>> > #include "cv.h"
>> > #include "cxcore.h"
>> > #include "cxtypes.h"
>> > int main(void)
>> > {
>> > ?? ?IplImage* img, *img2;
>> > ?? ?img = cvLoadImage("2.bmp", 1);
>> > ?? ?img2 = cvCreateImage(cvSize(640, 480), img->depth, img->nChannels);
>> > ?? ?cvResize(img, img2, 3);
>> > ?? ?cvNamedWindow("test", 1);
>> > ?? ?cvShowImage("test", img2);
>> > ?? ?cvWaitKey(0);
>> > ?? ?cvDestroyAllWindows();
>> > ?? ?cvReleaseImage(&img);
>> > ?? ?cvReleaseImage(&img2);
>> > }
>> >
>> >
>> > valid integers for the argument 'interpolation' are 0, 1, 2, 3, 4 (as
>> > tested
>> > with the C function). My Cython wrapper fails with a Python crash for
>> > integer values 1 & 2. ?Values of 0,1,4 work properly and the output
>> > image
>> > matches the output image of the C test function under the same
>> > conditions.
>> > I've looked at the generated Cython code and everything seems in order,
>> > I
>> > just can't think of what could be causing this behavior.
>> > Any thoughts?
>> > Cheers!
>> > Chris
>> >
>> > _______________________________________________
>> > Cython-dev mailing list
>> > Cython-dev at codespeak.net
>> > http://codespeak.net/mailman/listinfo/cython-dev
>> >
>> >
>>
>>
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From sccolbert at gmail.com  Wed May 27 18:03:41 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Wed, 27 May 2009 12:03:41 -0400
Subject: [Cython] having a weird issue with wrapped function failing in
	only certain cases
In-Reply-To: 
References: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>
	
	<7f014ea60905270736v682b1257jdeeb8fbd230282ba@mail.gmail.com>
	
Message-ID: <7f014ea60905270903y7e90c4f2ic014536b55bdf8b9@mail.gmail.com>

I've never used valgrind before. Any pointers?

I went back to the Cython generated C source and explicitly included the
appropriate header file and explicitly passed CV_INTER_LINEAR (defined as 1)
to the C function call. Same story, it crashes.

I've gone line by line now through the Cython code for that function and I
cant see anything wrong. You have any ideas I could try?

Thanks!

Chris

On Wed, May 27, 2009 at 10:49 AM, Lisandro Dalcin  wrote:

> On Wed, May 27, 2009 at 11:36 AM, Chris Colbert 
> wrote:
> > sometimes the docs are a bit dated. But nonetheless, those constants are
> > just integers defines in the header files and is where I got the values
> 0-4,
> > I used the integers in the example just to make it a bit more clear. But
> 0-4
> > are valid constants for the call, as they worked in the pure C example.
>
> Mmm... they I have no idea what's going on... Did you try to run under
> valgrind? That usually works for me...
>
> > What you showed for exposing the constants however, is interesting, I
> have
> > been just defining the constants as module level python integers so they
> can
> > be used from python. If defined as an enum as you show, will they
> > be accessible from python?
> >
>
> No, you have to explicitly expose them. If you declared them in some
> pxd, let say "mycvdefs.pxd", you can write in "mycvmodule.pyx"
>
> CV_SOME_CONSTANT = mycvdefs.CV_SOME_CONSTANT
>
>
>
>
>
> > Chris
> >
> > On Wed, May 27, 2009 at 9:40 AM, Lisandro Dalcin 
> wrote:
> >>
> >> According to the docs,
> >>
> >>
> http://opencv.willowgarage.com/wiki/CvReference#Sampling.2CInterpolationandGeometricalTransforms
> ,
> >> There are 4 possible values (you mention 5). Is that OpenCV
> >> documentation dated? IMHO, you should use them. To expose these
> >> constants in Cython code (likely in some pxd), you can do this:
> >>
> >> cdef extern from ....:
> >>
> >>    enum: CV_INTER_NN
> >>    enum: CV_INTER_LINEAR
> >>    enum: CV_INTER_AREA
> >>    enum: CV_INTER_CUBIC
> >>
> >>
> >>
> >> On Wed, May 27, 2009 at 12:57 AM, Chris Colbert 
> >> wrote:
> >> > So I'm wrapping the OpenCV library in Cython and its coming along
> quite
> >> > nicely (thanks to everyone here for getting me up to speed!), but
> today
> >> > I've
> >> > run across a strange issue where a wrapped function fails for certain
> >> > values
> >> > of integer arguments, but a pure c implementation testing the same
> thing
> >> > with the same arguments works.
> >> > Here is the code excerpts im using:
> >> > #---------- .pyx ------------------#
> >> > def cvResize(cy_cvtypes.CvArr src, cy_cvtypes.CvArr dst,
> interpolation):
> >> >     cdef c_cxcore.CvArr* c_src = src.handle()
> >> >     cdef c_cxcore.CvArr* c_dst = dst.handle()
> >> >     cdef int c_interpolation = interpolation
> >> >     with nogil:
> >> >         c_cv.cvResize(c_src, c_dst, c_interpolation)
> >> > #-------- c_cv.pxd -----------------#
> >> > cdef extern from "cv.h":
> >> >     void cvResize(CvArr* src, CvArr* dst, int interpolation) nogil
> >> >
> >> > #--------- test.py -----------------#
> >> > from cyopencv import *
> >> > if __name__=='__main__':
> >> >     img = cvLoadImage('2.bmp')
> >> >     img2 = cvCreateImage((640, 480), img.depth, img.nChannels)
> >> >     cvResize(img, img2, 3)
> >> >     img2.show()
> >> >
> >> >
> >> > #--------- pureC.c -----------#
> >> > #include "highgui.h"
> >> > #include "cv.h"
> >> > #include "cxcore.h"
> >> > #include "cxtypes.h"
> >> > int main(void)
> >> > {
> >> >     IplImage* img, *img2;
> >> >     img = cvLoadImage("2.bmp", 1);
> >> >     img2 = cvCreateImage(cvSize(640, 480), img->depth,
> img->nChannels);
> >> >     cvResize(img, img2, 3);
> >> >     cvNamedWindow("test", 1);
> >> >     cvShowImage("test", img2);
> >> >     cvWaitKey(0);
> >> >     cvDestroyAllWindows();
> >> >     cvReleaseImage(&img);
> >> >     cvReleaseImage(&img2);
> >> > }
> >> >
> >> >
> >> > valid integers for the argument 'interpolation' are 0, 1, 2, 3, 4 (as
> >> > tested
> >> > with the C function). My Cython wrapper fails with a Python crash for
> >> > integer values 1 & 2.  Values of 0,1,4 work properly and the output
> >> > image
> >> > matches the output image of the C test function under the same
> >> > conditions.
> >> > I've looked at the generated Cython code and everything seems in
> order,
> >> > I
> >> > just can't think of what could be causing this behavior.
> >> > Any thoughts?
> >> > Cheers!
> >> > Chris
> >> >
> >> > _______________________________________________
> >> > Cython-dev mailing list
> >> > Cython-dev at codespeak.net
> >> > http://codespeak.net/mailman/listinfo/cython-dev
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Lisandro Dalc?n
> >> ---------------
> >> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> >> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> >> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> >> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> >> Tel/Fax: +54-(0)342-451.1594
> >> _______________________________________________
> >> Cython-dev mailing list
> >> Cython-dev at codespeak.net
> >> http://codespeak.net/mailman/listinfo/cython-dev
> >
> >
> > _______________________________________________
> > Cython-dev mailing list
> > Cython-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/cython-dev
> >
> >
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090527/61f2e938/attachment.htm 

From dalcinl at gmail.com  Wed May 27 18:21:44 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 27 May 2009 13:21:44 -0300
Subject: [Cython] having a weird issue with wrapped function failing in
	only certain cases
In-Reply-To: <7f014ea60905270903y7e90c4f2ic014536b55bdf8b9@mail.gmail.com>
References: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>
	
	<7f014ea60905270736v682b1257jdeeb8fbd230282ba@mail.gmail.com>
	
	<7f014ea60905270903y7e90c4f2ic014536b55bdf8b9@mail.gmail.com>
Message-ID: 

On Wed, May 27, 2009 at 1:03 PM, Chris Colbert  wrote:
> I've never used valgrind before. Any pointers?
>

Just run

$ valgrind python somescript.py

You may get a lot of junk warnings because of Python's mem allocator
implementation. There is a way to suppress them,

$ valgrind --suppressions=/path/to/valgrind-python.supp python somescript.py

You may have valgrind-python.supp installed somewhere (from package
python-devel), for example, in my Fedora 10 box, it is at.

/usr/share/doc/python-devel-2.5.2/valgrind-python.supp

Perhaps a good idea (I'm doing that) is to add an executable shell
script in your $HOME/bin (assuming that dir in on $PATH), like this:

$ cat $HOME/bin/valgrind-py
#!/bin/sh
valgrind --suppressions=$HOME/lib/valgrind-python.supp $@

Then I run Python under Valgrind like this:

$ valgrind-py python somescript.py

You could even modify the script to add 'python' before the $@, I do
not do that just because I want to use it against many Python
versions. But perhaps this is not an issue for you.



> I went back to the Cython generated C source and explicitly included the
> appropriate header file and explicitly passed CV_INTER_LINEAR (defined as 1)
> to the C function call. Same story, it crashes.
>
> I've gone line by line now through the Cython code for that function and I
> cant see anything wrong. You have any ideas I could try?
>
> Thanks!
>
> Chris
>
> On Wed, May 27, 2009 at 10:49 AM, Lisandro Dalcin  wrote:
>>
>> On Wed, May 27, 2009 at 11:36 AM, Chris Colbert 
>> wrote:
>> > sometimes the docs are a bit dated. But nonetheless, those constants are
>> > just integers defines in the header files and is where I got the values
>> > 0-4,
>> > I used the integers in the example just to make it a bit more clear. But
>> > 0-4
>> > are valid constants for the call, as they worked in the pure C example.
>>
>> Mmm... they I have no idea what's going on... Did you try to run under
>> valgrind? That usually works for me...
>>
>> > What you showed for exposing the constants however, is interesting, I
>> > have
>> > been just defining the constants as module level python integers so they
>> > can
>> > be used from python. If defined as an enum as you show, will they
>> > be?accessible?from python?
>> >
>>
>> No, you have to explicitly expose them. If you declared them in some
>> pxd, let say "mycvdefs.pxd", you can write in "mycvmodule.pyx"
>>
>> CV_SOME_CONSTANT = mycvdefs.CV_SOME_CONSTANT
>>
>>
>>
>>
>>
>> > Chris
>> >
>> > On Wed, May 27, 2009 at 9:40 AM, Lisandro Dalcin 
>> > wrote:
>> >>
>> >> According to the docs,
>> >>
>> >>
>> >> http://opencv.willowgarage.com/wiki/CvReference#Sampling.2CInterpolationandGeometricalTransforms,
>> >> There are 4 possible values (you mention 5). Is that OpenCV
>> >> documentation dated? IMHO, you should use them. To expose these
>> >> constants in Cython code (likely in some pxd), you can do this:
>> >>
>> >> cdef extern from ....:
>> >>
>> >> ? ?enum: CV_INTER_NN
>> >> ? ?enum: CV_INTER_LINEAR
>> >> ? ?enum: CV_INTER_AREA
>> >> ? ?enum: CV_INTER_CUBIC
>> >>
>> >>
>> >>
>> >> On Wed, May 27, 2009 at 12:57 AM, Chris Colbert 
>> >> wrote:
>> >> > So I'm wrapping the OpenCV library in Cython and its coming along
>> >> > quite
>> >> > nicely (thanks to everyone here for getting me up to speed!), but
>> >> > today
>> >> > I've
>> >> > run across a strange issue where a wrapped function fails for certain
>> >> > values
>> >> > of integer arguments, but a pure c implementation testing the same
>> >> > thing
>> >> > with the same arguments works.
>> >> > Here is the code?excerpts?im using:
>> >> > #---------- .pyx ------------------#
>> >> > def cvResize(cy_cvtypes.CvArr src, cy_cvtypes.CvArr dst,
>> >> > interpolation):
>> >> > ?? ?cdef c_cxcore.CvArr* c_src = src.handle()
>> >> > ?? ?cdef c_cxcore.CvArr* c_dst = dst.handle()
>> >> > ?? ?cdef int c_interpolation = interpolation
>> >> > ?? ?with nogil:
>> >> > ?? ? ? ?c_cv.cvResize(c_src, c_dst, c_interpolation)
>> >> > #-------- c_cv.pxd -----------------#
>> >> > cdef extern from "cv.h":
>> >> > ?? ?void cvResize(CvArr* src, CvArr* dst, int interpolation) nogil
>> >> >
>> >> > #--------- test.py -----------------#
>> >> > from cyopencv import *
>> >> > if __name__=='__main__':
>> >> > ?? ?img = cvLoadImage('2.bmp')
>> >> > ?? ?img2 = cvCreateImage((640, 480), img.depth, img.nChannels)
>> >> > ?? ?cvResize(img, img2, 3)
>> >> > ?? ?img2.show()
>> >> >
>> >> >
>> >> > #--------- pureC.c -----------#
>> >> > #include "highgui.h"
>> >> > #include "cv.h"
>> >> > #include "cxcore.h"
>> >> > #include "cxtypes.h"
>> >> > int main(void)
>> >> > {
>> >> > ?? ?IplImage* img, *img2;
>> >> > ?? ?img = cvLoadImage("2.bmp", 1);
>> >> > ?? ?img2 = cvCreateImage(cvSize(640, 480), img->depth,
>> >> > img->nChannels);
>> >> > ?? ?cvResize(img, img2, 3);
>> >> > ?? ?cvNamedWindow("test", 1);
>> >> > ?? ?cvShowImage("test", img2);
>> >> > ?? ?cvWaitKey(0);
>> >> > ?? ?cvDestroyAllWindows();
>> >> > ?? ?cvReleaseImage(&img);
>> >> > ?? ?cvReleaseImage(&img2);
>> >> > }
>> >> >
>> >> >
>> >> > valid integers for the argument 'interpolation' are 0, 1, 2, 3, 4 (as
>> >> > tested
>> >> > with the C function). My Cython wrapper fails with a Python crash for
>> >> > integer values 1 & 2. ?Values of 0,1,4 work properly and the output
>> >> > image
>> >> > matches the output image of the C test function under the same
>> >> > conditions.
>> >> > I've looked at the generated Cython code and everything seems in
>> >> > order,
>> >> > I
>> >> > just can't think of what could be causing this behavior.
>> >> > Any thoughts?
>> >> > Cheers!
>> >> > Chris
>> >> >
>> >> > _______________________________________________
>> >> > Cython-dev mailing list
>> >> > Cython-dev at codespeak.net
>> >> > http://codespeak.net/mailman/listinfo/cython-dev
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Lisandro Dalc?n
>> >> ---------------
>> >> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> >> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> >> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> >> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> >> Tel/Fax: +54-(0)342-451.1594
>> >> _______________________________________________
>> >> Cython-dev mailing list
>> >> Cython-dev at codespeak.net
>> >> http://codespeak.net/mailman/listinfo/cython-dev
>> >
>> >
>> > _______________________________________________
>> > Cython-dev mailing list
>> > Cython-dev at codespeak.net
>> > http://codespeak.net/mailman/listinfo/cython-dev
>> >
>> >
>>
>>
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From sccolbert at gmail.com  Wed May 27 20:09:30 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Wed, 27 May 2009 14:09:30 -0400
Subject: [Cython] having a weird issue with wrapped function failing in
	only certain cases
In-Reply-To: 
References: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>
	
	<7f014ea60905270736v682b1257jdeeb8fbd230282ba@mail.gmail.com>
	
	<7f014ea60905270903y7e90c4f2ic014536b55bdf8b9@mail.gmail.com>
	
Message-ID: <7f014ea60905271109p463c9544y66cd9c9a09eb937e@mail.gmail.com>

Lisandro,

Unfortunately, I'm on a windows machine and thus can't use valgrind.

I went ahead and exposed all (actually like 97%) of the contants in OpenCV
to python using the method you described.

Now, my wrapper works for the contants:  CV_INTER_NN and CV_INTER_AREA

but fails on CV_INTER_LINEAR and CV_INTER_CUBIC which have the integer
values 1 and 2 respectively, when printed to the console.

I haven't asked on the OpenCV lists yet because this seems to be a problem
in my code or a bug in Cython (given that it all works in pure C).

that will be my next stop though, if we can't figure it out here.

Do you think posting the Cython generated code will help?

Cheers!

Chris

On Wed, May 27, 2009 at 12:21 PM, Lisandro Dalcin  wrote:

> On Wed, May 27, 2009 at 1:03 PM, Chris Colbert 
> wrote:
> > I've never used valgrind before. Any pointers?
> >
>
> Just run
>
> $ valgrind python somescript.py
>
> You may get a lot of junk warnings because of Python's mem allocator
> implementation. There is a way to suppress them,
>
> $ valgrind --suppressions=/path/to/valgrind-python.supp python
> somescript.py
>
> You may have valgrind-python.supp installed somewhere (from package
> python-devel), for example, in my Fedora 10 box, it is at.
>
> /usr/share/doc/python-devel-2.5.2/valgrind-python.supp
>
> Perhaps a good idea (I'm doing that) is to add an executable shell
> script in your $HOME/bin (assuming that dir in on $PATH), like this:
>
> $ cat $HOME/bin/valgrind-py
> #!/bin/sh
> valgrind --suppressions=$HOME/lib/valgrind-python.supp $@
>
> Then I run Python under Valgrind like this:
>
> $ valgrind-py python somescript.py
>
> You could even modify the script to add 'python' before the $@, I do
> not do that just because I want to use it against many Python
> versions. But perhaps this is not an issue for you.
>
>
>
> > I went back to the Cython generated C source and explicitly included the
> > appropriate header file and explicitly passed CV_INTER_LINEAR (defined as
> 1)
> > to the C function call. Same story, it crashes.
> >
> > I've gone line by line now through the Cython code for that function and
> I
> > cant see anything wrong. You have any ideas I could try?
> >
> > Thanks!
> >
> > Chris
> >
> > On Wed, May 27, 2009 at 10:49 AM, Lisandro Dalcin 
> wrote:
> >>
> >> On Wed, May 27, 2009 at 11:36 AM, Chris Colbert 
> >> wrote:
> >> > sometimes the docs are a bit dated. But nonetheless, those constants
> are
> >> > just integers defines in the header files and is where I got the
> values
> >> > 0-4,
> >> > I used the integers in the example just to make it a bit more clear.
> But
> >> > 0-4
> >> > are valid constants for the call, as they worked in the pure C
> example.
> >>
> >> Mmm... they I have no idea what's going on... Did you try to run under
> >> valgrind? That usually works for me...
> >>
> >> > What you showed for exposing the constants however, is interesting, I
> >> > have
> >> > been just defining the constants as module level python integers so
> they
> >> > can
> >> > be used from python. If defined as an enum as you show, will they
> >> > be accessible from python?
> >> >
> >>
> >> No, you have to explicitly expose them. If you declared them in some
> >> pxd, let say "mycvdefs.pxd", you can write in "mycvmodule.pyx"
> >>
> >> CV_SOME_CONSTANT = mycvdefs.CV_SOME_CONSTANT
> >>
> >>
> >>
> >>
> >>
> >> > Chris
> >> >
> >> > On Wed, May 27, 2009 at 9:40 AM, Lisandro Dalcin 
> >> > wrote:
> >> >>
> >> >> According to the docs,
> >> >>
> >> >>
> >> >>
> http://opencv.willowgarage.com/wiki/CvReference#Sampling.2CInterpolationandGeometricalTransforms
> ,
> >> >> There are 4 possible values (you mention 5). Is that OpenCV
> >> >> documentation dated? IMHO, you should use them. To expose these
> >> >> constants in Cython code (likely in some pxd), you can do this:
> >> >>
> >> >> cdef extern from ....:
> >> >>
> >> >>    enum: CV_INTER_NN
> >> >>    enum: CV_INTER_LINEAR
> >> >>    enum: CV_INTER_AREA
> >> >>    enum: CV_INTER_CUBIC
> >> >>
> >> >>
> >> >>
> >> >> On Wed, May 27, 2009 at 12:57 AM, Chris Colbert  >
> >> >> wrote:
> >> >> > So I'm wrapping the OpenCV library in Cython and its coming along
> >> >> > quite
> >> >> > nicely (thanks to everyone here for getting me up to speed!), but
> >> >> > today
> >> >> > I've
> >> >> > run across a strange issue where a wrapped function fails for
> certain
> >> >> > values
> >> >> > of integer arguments, but a pure c implementation testing the same
> >> >> > thing
> >> >> > with the same arguments works.
> >> >> > Here is the code excerpts im using:
> >> >> > #---------- .pyx ------------------#
> >> >> > def cvResize(cy_cvtypes.CvArr src, cy_cvtypes.CvArr dst,
> >> >> > interpolation):
> >> >> >     cdef c_cxcore.CvArr* c_src = src.handle()
> >> >> >     cdef c_cxcore.CvArr* c_dst = dst.handle()
> >> >> >     cdef int c_interpolation = interpolation
> >> >> >     with nogil:
> >> >> >         c_cv.cvResize(c_src, c_dst, c_interpolation)
> >> >> > #-------- c_cv.pxd -----------------#
> >> >> > cdef extern from "cv.h":
> >> >> >     void cvResize(CvArr* src, CvArr* dst, int interpolation) nogil
> >> >> >
> >> >> > #--------- test.py -----------------#
> >> >> > from cyopencv import *
> >> >> > if __name__=='__main__':
> >> >> >     img = cvLoadImage('2.bmp')
> >> >> >     img2 = cvCreateImage((640, 480), img.depth, img.nChannels)
> >> >> >     cvResize(img, img2, 3)
> >> >> >     img2.show()
> >> >> >
> >> >> >
> >> >> > #--------- pureC.c -----------#
> >> >> > #include "highgui.h"
> >> >> > #include "cv.h"
> >> >> > #include "cxcore.h"
> >> >> > #include "cxtypes.h"
> >> >> > int main(void)
> >> >> > {
> >> >> >     IplImage* img, *img2;
> >> >> >     img = cvLoadImage("2.bmp", 1);
> >> >> >     img2 = cvCreateImage(cvSize(640, 480), img->depth,
> >> >> > img->nChannels);
> >> >> >     cvResize(img, img2, 3);
> >> >> >     cvNamedWindow("test", 1);
> >> >> >     cvShowImage("test", img2);
> >> >> >     cvWaitKey(0);
> >> >> >     cvDestroyAllWindows();
> >> >> >     cvReleaseImage(&img);
> >> >> >     cvReleaseImage(&img2);
> >> >> > }
> >> >> >
> >> >> >
> >> >> > valid integers for the argument 'interpolation' are 0, 1, 2, 3, 4
> (as
> >> >> > tested
> >> >> > with the C function). My Cython wrapper fails with a Python crash
> for
> >> >> > integer values 1 & 2.  Values of 0,1,4 work properly and the output
> >> >> > image
> >> >> > matches the output image of the C test function under the same
> >> >> > conditions.
> >> >> > I've looked at the generated Cython code and everything seems in
> >> >> > order,
> >> >> > I
> >> >> > just can't think of what could be causing this behavior.
> >> >> > Any thoughts?
> >> >> > Cheers!
> >> >> > Chris
> >> >> >
> >> >> > _______________________________________________
> >> >> > Cython-dev mailing list
> >> >> > Cython-dev at codespeak.net
> >> >> > http://codespeak.net/mailman/listinfo/cython-dev
> >> >> >
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Lisandro Dalc?n
> >> >> ---------------
> >> >> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> >> >> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> >> >> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> >> >> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> >> >> Tel/Fax: +54-(0)342-451.1594
> >> >> _______________________________________________
> >> >> Cython-dev mailing list
> >> >> Cython-dev at codespeak.net
> >> >> http://codespeak.net/mailman/listinfo/cython-dev
> >> >
> >> >
> >> > _______________________________________________
> >> > Cython-dev mailing list
> >> > Cython-dev at codespeak.net
> >> > http://codespeak.net/mailman/listinfo/cython-dev
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Lisandro Dalc?n
> >> ---------------
> >> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> >> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> >> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> >> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> >> Tel/Fax: +54-(0)342-451.1594
> >> _______________________________________________
> >> Cython-dev mailing list
> >> Cython-dev at codespeak.net
> >> http://codespeak.net/mailman/listinfo/cython-dev
> >
> >
> > _______________________________________________
> > Cython-dev mailing list
> > Cython-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/cython-dev
> >
> >
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090527/9388d999/attachment.htm 

From dagss at student.matnat.uio.no  Wed May 27 21:20:18 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 27 May 2009 21:20:18 +0200 (CEST)
Subject: [Cython] having a weird issue with wrapped function failing in
 only certain cases
In-Reply-To: <7f014ea60905271109p463c9544y66cd9c9a09eb937e@mail.gmail.com>
References: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>
	
	<7f014ea60905270736v682b1257jdeeb8fbd230282ba@mail.gmail.com>
	
	<7f014ea60905270903y7e90c4f2ic014536b55bdf8b9@mail.gmail.com>
	
	<7f014ea60905271109p463c9544y66cd9c9a09eb937e@mail.gmail.com>
Message-ID: 

Chris Colbert wrote:
> Unfortunately, I'm on a windows machine and thus can't use valgrind.

VMWare Player and Ubuntu VMWare image?

> I went ahead and exposed all (actually like 97%) of the contants in OpenCV
> to python using the method you described.
>
> Now, my wrapper works for the contants:  CV_INTER_NN and CV_INTER_AREA
>
> but fails on CV_INTER_LINEAR and CV_INTER_CUBIC which have the integer
> values 1 and 2 respectively, when printed to the console.
>
> I haven't asked on the OpenCV lists yet because this seems to be a problem
> in my code or a bug in Cython (given that it all works in pure C).

If it works in pure C but not in pure Cython, there's a way of approaching
that will be guaranteed to find the issues:

a) Boil down the Cython code as much as possible. Make sure a simple cdef
function with no Python types or operations exhibit the crash.
b) Open the generate C file and replace the body of said cdef function
with the code that works.
c) If that doesn't work, it's likely a linker bug, strange use of macros
in the API or similar. Otherwise, incrementally change back to
Cython-generated code until it no longer works.

Dag Sverre



From sccolbert at gmail.com  Wed May 27 21:44:04 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Wed, 27 May 2009 15:44:04 -0400
Subject: [Cython] having a weird issue with wrapped function failing in
	only certain cases
In-Reply-To: 
References: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>
	
	<7f014ea60905270736v682b1257jdeeb8fbd230282ba@mail.gmail.com>
	
	<7f014ea60905270903y7e90c4f2ic014536b55bdf8b9@mail.gmail.com>
	
	<7f014ea60905271109p463c9544y66cd9c9a09eb937e@mail.gmail.com>
	
Message-ID: <7f014ea60905271244v39747565wd70872778b22d16c@mail.gmail.com>

> a) Boil down the Cython code as much as possible. Make sure a simple cdef
> function with no Python types or operations exhibit the crash.
> b) Open the generate C file and replace the body of said cdef function
> with the code that works.
> c) If that doesn't work, it's likely a linker bug, strange use of macros
> in the API or similar. Otherwise, incrementally change back to
> Cython-generated code until it no longer works.
>


I just started doing something similar. I started inserting printf
statements into the cython code up to the point of failure, just before that
I inserted my known working C-code and the recompiled. Now, the good code
also fails on the same integers, so i'm  thinking it may be a linker bug.

I'll boil down a simple Cython example and see if I can replicate it.


Thanks!

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090527/2ce614a3/attachment.htm 

From dagss at student.matnat.uio.no  Wed May 27 21:47:56 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 27 May 2009 21:47:56 +0200 (CEST)
Subject: [Cython] having a weird issue with wrapped function failing in
 only certain cases
In-Reply-To: <7f014ea60905271244v39747565wd70872778b22d16c@mail.gmail.com>
References: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>
	
	<7f014ea60905270736v682b1257jdeeb8fbd230282ba@mail.gmail.com>
	
	<7f014ea60905270903y7e90c4f2ic014536b55bdf8b9@mail.gmail.com>
	
	<7f014ea60905271109p463c9544y66cd9c9a09eb937e@mail.gmail.com>
	
	<7f014ea60905271244v39747565wd70872778b22d16c@mail.gmail.com>
Message-ID: 

>> a) Boil down the Cython code as much as possible. Make sure a simple
>> cdef
>> function with no Python types or operations exhibit the crash.
>> b) Open the generate C file and replace the body of said cdef function
>> with the code that works.
>> c) If that doesn't work, it's likely a linker bug, strange use of macros
>> in the API or similar. Otherwise, incrementally change back to
>> Cython-generated code until it no longer works.
>>
>
>
> I just started doing something similar. I started inserting printf
> statements into the cython code up to the point of failure, just before
> that
> I inserted my known working C-code and the recompiled. Now, the good code
> also fails on the same integers, so i'm  thinking it may be a linker bug.

Quick note: By linker bug I really meant a bug in your make system, not in
the linker itself :-) But I guess that's how you interpreted it.

Dag Sverre


From dagss at student.matnat.uio.no  Wed May 27 21:54:22 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 27 May 2009 21:54:22 +0200 (CEST)
Subject: [Cython] Users list revisited
Message-ID: <9d5d8cd541112bde2dede154eaf65e82.squirrel@webmail.uio.no>

Something occurred to me:

When we discussed whether there should be a cython-users list, we were
arguing from how the situation is today, and that the current active users
don't think it is needed.

However, one should probably take into account future scenarios:

a) It might be that cython-users would attract more posters, who don't
think about asking their questions on cython-dev in the first place
(dev-lists in general being a RTFM kind of place, even though this one
isn't)

b) It might be that cython-users would in time make it easier for non-devs
to answer questions, creating a users community in the same

As for me personally, I totally don't mind the users questions in here and
would subscribe to both lists (as I'm sure the other devs would as well),
but it is now getting to a point where having the different things sorted
into two folders in my mail program wouldn't hurt either.

So, +1 from me on a new cython-users list.

Dag Sverre


From sccolbert at gmail.com  Wed May 27 21:56:10 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Wed, 27 May 2009 15:56:10 -0400
Subject: [Cython] having a weird issue with wrapped function failing in
	only certain cases
In-Reply-To: 
References: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>
	
	<7f014ea60905270736v682b1257jdeeb8fbd230282ba@mail.gmail.com>
	
	<7f014ea60905270903y7e90c4f2ic014536b55bdf8b9@mail.gmail.com>
	
	<7f014ea60905271109p463c9544y66cd9c9a09eb937e@mail.gmail.com>
	
	<7f014ea60905271244v39747565wd70872778b22d16c@mail.gmail.com>
	
Message-ID: <7f014ea60905271256u6b49d408ta3babe4b8f50809a@mail.gmail.com>

>
>
> Quick note: By linker bug I really meant a bug in your make system, not in
> the linker itself :-) But I guess that's how you interpreted it.
>
> Dag Sverre
>
> _______________________________________________
>


Here is a pure cython example that exhibits the same behavior. The cimports
are simply .pxd files of header prototypes.


#---------- purecython.pyx -----------------#
cimport c_cxcore
cimport c_highgui
cimport c_cv
cimport c_constants

cdef tester():
    cdef c_cxcore.IplImage* img1
    cdef c_cxcore.IplImage* img2
    img1 = c_highgui.cvLoadImage("2.bmp", 1)
    img2 = c_cxcore.cvCreateImage(c_cxcore.cvSize(640, 480), 8, 3)
    c_cv.cvResize(img1, img2, c_constants.CV_INTER_AREA)
    c_highgui.cvNamedWindow("test", 1)
    c_highgui.cvShowImage("test", img2)
    c_highgui.cvWaitKey(0)

def test():
    tester()




What do you mean by problem with my make system? Meaning a problem with my
setup.py or a problem with the compiler?

here is my setup.py script:

#-------------- setup.py --------------------#
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

sourcefiles = ['cy_constants.pyx', 'cy_cvtypes.pyx', 'cy_cxcore.pyx',
'cy_highgui.pyx', 'cy_cv.pyx', 'cyopencv.pyx']

include_dirs = ['C:\\OpenCVsvn\\build\\include\\opencv']


library_dirs = ['C:\\OpenCVsvn\\build\\lib']

libraries = ['highgui110',
        'cxcore110',
        'cv110',
        'cvaux110',
        'cxts110',
        'cvhaartraining',
        'ml110']

for source in sourcefiles:
    setup(
        cmdclass = {'build_ext': build_ext},
        ext_modules = [Extension(source.rstrip('.pyx'), [source],
                                    include_dirs=include_dirs,
                                    library_dirs=library_dirs,
                                    libraries = libraries)]).



Cheers!

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090527/9394efbf/attachment.htm 

From sccolbert at gmail.com  Wed May 27 22:00:47 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Wed, 27 May 2009 16:00:47 -0400
Subject: [Cython] Users list revisited
In-Reply-To: <9d5d8cd541112bde2dede154eaf65e82.squirrel@webmail.uio.no>
References: <9d5d8cd541112bde2dede154eaf65e82.squirrel@webmail.uio.no>
Message-ID: <7f014ea60905271300y3bea0d03tdf010085b0cb50c0@mail.gmail.com>

I agree, I think a Cython-users is a great idea.

You all have a been a great help to me in learning Cython, but I feel
horrible having to ask such newbie questions all the time, and fear I will
one day wear out my welcome.

So +1 for a Cython-users list.

Chris

On Wed, May 27, 2009 at 3:54 PM, Dag Sverre Seljebotn <
dagss at student.matnat.uio.no> wrote:

> Something occurred to me:
>
> When we discussed whether there should be a cython-users list, we were
> arguing from how the situation is today, and that the current active users
> don't think it is needed.
>
> However, one should probably take into account future scenarios:
>
> a) It might be that cython-users would attract more posters, who don't
> think about asking their questions on cython-dev in the first place
> (dev-lists in general being a RTFM kind of place, even though this one
> isn't)
>
> b) It might be that cython-users would in time make it easier for non-devs
> to answer questions, creating a users community in the same
>
> As for me personally, I totally don't mind the users questions in here and
> would subscribe to both lists (as I'm sure the other devs would as well),
> but it is now getting to a point where having the different things sorted
> into two folders in my mail program wouldn't hurt either.
>
> So, +1 from me on a new cython-users list.
>
> Dag Sverre
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090527/2f97f581/attachment.htm 

From dsurviver at gmail.com  Wed May 27 22:05:45 2009
From: dsurviver at gmail.com (Danilo Freitas)
Date: Wed, 27 May 2009 17:05:45 -0300
Subject: [Cython] Users list revisited
In-Reply-To: <7f014ea60905271300y3bea0d03tdf010085b0cb50c0@mail.gmail.com>
References: <9d5d8cd541112bde2dede154eaf65e82.squirrel@webmail.uio.no>
	<7f014ea60905271300y3bea0d03tdf010085b0cb50c0@mail.gmail.com>
Message-ID: <45239150905271305k1efe5d70i887eda735433cf5f@mail.gmail.com>

+1 for me.
There are many questions here about using cython, and it would be
better if these questions were directly to other list, and this one
stay only for developing stuffs.


-- 
- Danilo Freitas

From kwmsmith at gmail.com  Wed May 27 22:11:45 2009
From: kwmsmith at gmail.com (Kurt Smith)
Date: Wed, 27 May 2009 15:11:45 -0500
Subject: [Cython] Users list revisited
In-Reply-To: <9d5d8cd541112bde2dede154eaf65e82.squirrel@webmail.uio.no>
References: <9d5d8cd541112bde2dede154eaf65e82.squirrel@webmail.uio.no>
Message-ID: 

On Wed, May 27, 2009 at 2:54 PM, Dag Sverre Seljebotn
 wrote:
> Something occurred to me:
>
> When we discussed whether there should be a cython-users list, we were
> arguing from how the situation is today, and that the current active users
> don't think it is needed.
>
> However, one should probably take into account future scenarios:
>
> a) It might be that cython-users would attract more posters, who don't
> think about asking their questions on cython-dev in the first place
> (dev-lists in general being a RTFM kind of place, even though this one
> isn't)
>
> b) It might be that cython-users would in time make it easier for non-devs
> to answer questions, creating a users community in the same
>
> As for me personally, I totally don't mind the users questions in here and
> would subscribe to both lists (as I'm sure the other devs would as well),
> but it is now getting to a point where having the different things sorted
> into two folders in my mail program wouldn't hurt either.
>
> So, +1 from me on a new cython-users list.

+1 for me.

From dalcinl at gmail.com  Wed May 27 22:48:35 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 27 May 2009 17:48:35 -0300
Subject: [Cython] having a weird issue with wrapped function failing in
	only certain cases
In-Reply-To: <7f014ea60905271109p463c9544y66cd9c9a09eb937e@mail.gmail.com>
References: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>
	
	<7f014ea60905270736v682b1257jdeeb8fbd230282ba@mail.gmail.com>
	
	<7f014ea60905270903y7e90c4f2ic014536b55bdf8b9@mail.gmail.com>
	
	<7f014ea60905271109p463c9544y66cd9c9a09eb937e@mail.gmail.com>
Message-ID: 

On Wed, May 27, 2009 at 3:09 PM, Chris Colbert  wrote:
> Lisandro,
>
> Unfortunately, I'm on a windows machine and thus can't use valgrind.
>

Wow... Well, not in a good position to help you... Anyway, Did you
build OpenCV with EXACTLY the same compiler your installed Python was
built?

---

> I went ahead and exposed all (actually like 97%) of the contants in OpenCV
> to python using the method you described.
>
> Now, my wrapper works for the contants:? CV_INTER_NN and CV_INTER_AREA
>
> but fails on CV_INTER_LINEAR and CV_INTER_CUBIC which have the integer
> values 1 and 2 respectively, when printed to the console.
>
> I haven't asked on the OpenCV lists yet because this seems to be a problem
> in my code or a bug in Cython (given that it all works in pure C).
>
> that will be my next stop though, if we can't figure it out here.
>
> Do you think posting the Cython generated code will help?
>
> Cheers!
>
> Chris
>
> On Wed, May 27, 2009 at 12:21 PM, Lisandro Dalcin  wrote:
>>
>> On Wed, May 27, 2009 at 1:03 PM, Chris Colbert 
>> wrote:
>> > I've never used valgrind before. Any pointers?
>> >
>>
>> Just run
>>
>> $ valgrind python somescript.py
>>
>> You may get a lot of junk warnings because of Python's mem allocator
>> implementation. There is a way to suppress them,
>>
>> $ valgrind --suppressions=/path/to/valgrind-python.supp python
>> somescript.py
>>
>> You may have valgrind-python.supp installed somewhere (from package
>> python-devel), for example, in my Fedora 10 box, it is at.
>>
>> /usr/share/doc/python-devel-2.5.2/valgrind-python.supp
>>
>> Perhaps a good idea (I'm doing that) is to add an executable shell
>> script in your $HOME/bin (assuming that dir in on $PATH), like this:
>>
>> $ cat $HOME/bin/valgrind-py
>> #!/bin/sh
>> valgrind --suppressions=$HOME/lib/valgrind-python.supp $@
>>
>> Then I run Python under Valgrind like this:
>>
>> $ valgrind-py python somescript.py
>>
>> You could even modify the script to add 'python' before the $@, I do
>> not do that just because I want to use it against many Python
>> versions. But perhaps this is not an issue for you.
>>
>>
>>
>> > I went back to the Cython generated C source and explicitly included the
>> > appropriate header file and explicitly passed CV_INTER_LINEAR (defined
>> > as 1)
>> > to the C function call. Same story, it crashes.
>> >
>> > I've gone line by line now through the Cython code for that function and
>> > I
>> > cant see anything wrong. You have any ideas I could try?
>> >
>> > Thanks!
>> >
>> > Chris
>> >
>> > On Wed, May 27, 2009 at 10:49 AM, Lisandro Dalcin 
>> > wrote:
>> >>
>> >> On Wed, May 27, 2009 at 11:36 AM, Chris Colbert 
>> >> wrote:
>> >> > sometimes the docs are a bit dated. But nonetheless, those constants
>> >> > are
>> >> > just integers defines in the header files and is where I got the
>> >> > values
>> >> > 0-4,
>> >> > I used the integers in the example just to make it a bit more clear.
>> >> > But
>> >> > 0-4
>> >> > are valid constants for the call, as they worked in the pure C
>> >> > example.
>> >>
>> >> Mmm... they I have no idea what's going on... Did you try to run under
>> >> valgrind? That usually works for me...
>> >>
>> >> > What you showed for exposing the constants however, is interesting, I
>> >> > have
>> >> > been just defining the constants as module level python integers so
>> >> > they
>> >> > can
>> >> > be used from python. If defined as an enum as you show, will they
>> >> > be?accessible?from python?
>> >> >
>> >>
>> >> No, you have to explicitly expose them. If you declared them in some
>> >> pxd, let say "mycvdefs.pxd", you can write in "mycvmodule.pyx"
>> >>
>> >> CV_SOME_CONSTANT = mycvdefs.CV_SOME_CONSTANT
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> > Chris
>> >> >
>> >> > On Wed, May 27, 2009 at 9:40 AM, Lisandro Dalcin 
>> >> > wrote:
>> >> >>
>> >> >> According to the docs,
>> >> >>
>> >> >>
>> >> >>
>> >> >> http://opencv.willowgarage.com/wiki/CvReference#Sampling.2CInterpolationandGeometricalTransforms,
>> >> >> There are 4 possible values (you mention 5). Is that OpenCV
>> >> >> documentation dated? IMHO, you should use them. To expose these
>> >> >> constants in Cython code (likely in some pxd), you can do this:
>> >> >>
>> >> >> cdef extern from ....:
>> >> >>
>> >> >> ? ?enum: CV_INTER_NN
>> >> >> ? ?enum: CV_INTER_LINEAR
>> >> >> ? ?enum: CV_INTER_AREA
>> >> >> ? ?enum: CV_INTER_CUBIC
>> >> >>
>> >> >>
>> >> >>
>> >> >> On Wed, May 27, 2009 at 12:57 AM, Chris Colbert
>> >> >> 
>> >> >> wrote:
>> >> >> > So I'm wrapping the OpenCV library in Cython and its coming along
>> >> >> > quite
>> >> >> > nicely (thanks to everyone here for getting me up to speed!), but
>> >> >> > today
>> >> >> > I've
>> >> >> > run across a strange issue where a wrapped function fails for
>> >> >> > certain
>> >> >> > values
>> >> >> > of integer arguments, but a pure c implementation testing the same
>> >> >> > thing
>> >> >> > with the same arguments works.
>> >> >> > Here is the code?excerpts?im using:
>> >> >> > #---------- .pyx ------------------#
>> >> >> > def cvResize(cy_cvtypes.CvArr src, cy_cvtypes.CvArr dst,
>> >> >> > interpolation):
>> >> >> > ?? ?cdef c_cxcore.CvArr* c_src = src.handle()
>> >> >> > ?? ?cdef c_cxcore.CvArr* c_dst = dst.handle()
>> >> >> > ?? ?cdef int c_interpolation = interpolation
>> >> >> > ?? ?with nogil:
>> >> >> > ?? ? ? ?c_cv.cvResize(c_src, c_dst, c_interpolation)
>> >> >> > #-------- c_cv.pxd -----------------#
>> >> >> > cdef extern from "cv.h":
>> >> >> > ?? ?void cvResize(CvArr* src, CvArr* dst, int interpolation) nogil
>> >> >> >
>> >> >> > #--------- test.py -----------------#
>> >> >> > from cyopencv import *
>> >> >> > if __name__=='__main__':
>> >> >> > ?? ?img = cvLoadImage('2.bmp')
>> >> >> > ?? ?img2 = cvCreateImage((640, 480), img.depth, img.nChannels)
>> >> >> > ?? ?cvResize(img, img2, 3)
>> >> >> > ?? ?img2.show()
>> >> >> >
>> >> >> >
>> >> >> > #--------- pureC.c -----------#
>> >> >> > #include "highgui.h"
>> >> >> > #include "cv.h"
>> >> >> > #include "cxcore.h"
>> >> >> > #include "cxtypes.h"
>> >> >> > int main(void)
>> >> >> > {
>> >> >> > ?? ?IplImage* img, *img2;
>> >> >> > ?? ?img = cvLoadImage("2.bmp", 1);
>> >> >> > ?? ?img2 = cvCreateImage(cvSize(640, 480), img->depth,
>> >> >> > img->nChannels);
>> >> >> > ?? ?cvResize(img, img2, 3);
>> >> >> > ?? ?cvNamedWindow("test", 1);
>> >> >> > ?? ?cvShowImage("test", img2);
>> >> >> > ?? ?cvWaitKey(0);
>> >> >> > ?? ?cvDestroyAllWindows();
>> >> >> > ?? ?cvReleaseImage(&img);
>> >> >> > ?? ?cvReleaseImage(&img2);
>> >> >> > }
>> >> >> >
>> >> >> >
>> >> >> > valid integers for the argument 'interpolation' are 0, 1, 2, 3, 4
>> >> >> > (as
>> >> >> > tested
>> >> >> > with the C function). My Cython wrapper fails with a Python crash
>> >> >> > for
>> >> >> > integer values 1 & 2. ?Values of 0,1,4 work properly and the
>> >> >> > output
>> >> >> > image
>> >> >> > matches the output image of the C test function under the same
>> >> >> > conditions.
>> >> >> > I've looked at the generated Cython code and everything seems in
>> >> >> > order,
>> >> >> > I
>> >> >> > just can't think of what could be causing this behavior.
>> >> >> > Any thoughts?
>> >> >> > Cheers!
>> >> >> > Chris
>> >> >> >
>> >> >> > _______________________________________________
>> >> >> > Cython-dev mailing list
>> >> >> > Cython-dev at codespeak.net
>> >> >> > http://codespeak.net/mailman/listinfo/cython-dev
>> >> >> >
>> >> >> >
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Lisandro Dalc?n
>> >> >> ---------------
>> >> >> Centro Internacional de M?todos Computacionales en Ingenier?a
>> >> >> (CIMEC)
>> >> >> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica
>> >> >> (INTEC)
>> >> >> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> >> >> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> >> >> Tel/Fax: +54-(0)342-451.1594
>> >> >> _______________________________________________
>> >> >> Cython-dev mailing list
>> >> >> Cython-dev at codespeak.net
>> >> >> http://codespeak.net/mailman/listinfo/cython-dev
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > Cython-dev mailing list
>> >> > Cython-dev at codespeak.net
>> >> > http://codespeak.net/mailman/listinfo/cython-dev
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Lisandro Dalc?n
>> >> ---------------
>> >> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> >> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> >> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> >> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> >> Tel/Fax: +54-(0)342-451.1594
>> >> _______________________________________________
>> >> Cython-dev mailing list
>> >> Cython-dev at codespeak.net
>> >> http://codespeak.net/mailman/listinfo/cython-dev
>> >
>> >
>> > _______________________________________________
>> > Cython-dev mailing list
>> > Cython-dev at codespeak.net
>> > http://codespeak.net/mailman/listinfo/cython-dev
>> >
>> >
>>
>>
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From sccolbert at gmail.com  Wed May 27 22:51:51 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Wed, 27 May 2009 16:51:51 -0400
Subject: [Cython] having a weird issue with wrapped function failing in
	only certain cases
In-Reply-To: 
References: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>
	
	<7f014ea60905270736v682b1257jdeeb8fbd230282ba@mail.gmail.com>
	
	<7f014ea60905270903y7e90c4f2ic014536b55bdf8b9@mail.gmail.com>
	
	<7f014ea60905271109p463c9544y66cd9c9a09eb937e@mail.gmail.com>
	
Message-ID: <7f014ea60905271351o1da8eb5dj714422c1c6ef2985@mail.gmail.com>

can't do that... Python 2.5 was built with VS 2003 which is no longer
available. So i'm using minGW.

I may throw Ubuntu on a partition and test there if it continues to be an
issue.

Thanks for your help though!

Chris

On Wed, May 27, 2009 at 4:48 PM, Lisandro Dalcin  wrote:

> On Wed, May 27, 2009 at 3:09 PM, Chris Colbert 
> wrote:
> > Lisandro,
> >
> > Unfortunately, I'm on a windows machine and thus can't use valgrind.
> >
>
> Wow... Well, not in a good position to help you... Anyway, Did you
> build OpenCV with EXACTLY the same compiler your installed Python was
> built?
>
> ---
>
> > I went ahead and exposed all (actually like 97%) of the contants in
> OpenCV
> > to python using the method you described.
> >
> > Now, my wrapper works for the contants:  CV_INTER_NN and CV_INTER_AREA
> >
> > but fails on CV_INTER_LINEAR and CV_INTER_CUBIC which have the integer
> > values 1 and 2 respectively, when printed to the console.
> >
> > I haven't asked on the OpenCV lists yet because this seems to be a
> problem
> > in my code or a bug in Cython (given that it all works in pure C).
> >
> > that will be my next stop though, if we can't figure it out here.
> >
> > Do you think posting the Cython generated code will help?
> >
> > Cheers!
> >
> > Chris
> >
> > On Wed, May 27, 2009 at 12:21 PM, Lisandro Dalcin 
> wrote:
> >>
> >> On Wed, May 27, 2009 at 1:03 PM, Chris Colbert 
> >> wrote:
> >> > I've never used valgrind before. Any pointers?
> >> >
> >>
> >> Just run
> >>
> >> $ valgrind python somescript.py
> >>
> >> You may get a lot of junk warnings because of Python's mem allocator
> >> implementation. There is a way to suppress them,
> >>
> >> $ valgrind --suppressions=/path/to/valgrind-python.supp python
> >> somescript.py
> >>
> >> You may have valgrind-python.supp installed somewhere (from package
> >> python-devel), for example, in my Fedora 10 box, it is at.
> >>
> >> /usr/share/doc/python-devel-2.5.2/valgrind-python.supp
> >>
> >> Perhaps a good idea (I'm doing that) is to add an executable shell
> >> script in your $HOME/bin (assuming that dir in on $PATH), like this:
> >>
> >> $ cat $HOME/bin/valgrind-py
> >> #!/bin/sh
> >> valgrind --suppressions=$HOME/lib/valgrind-python.supp $@
> >>
> >> Then I run Python under Valgrind like this:
> >>
> >> $ valgrind-py python somescript.py
> >>
> >> You could even modify the script to add 'python' before the $@, I do
> >> not do that just because I want to use it against many Python
> >> versions. But perhaps this is not an issue for you.
> >>
> >>
> >>
> >> > I went back to the Cython generated C source and explicitly included
> the
> >> > appropriate header file and explicitly passed CV_INTER_LINEAR (defined
> >> > as 1)
> >> > to the C function call. Same story, it crashes.
> >> >
> >> > I've gone line by line now through the Cython code for that function
> and
> >> > I
> >> > cant see anything wrong. You have any ideas I could try?
> >> >
> >> > Thanks!
> >> >
> >> > Chris
> >> >
> >> > On Wed, May 27, 2009 at 10:49 AM, Lisandro Dalcin 
> >> > wrote:
> >> >>
> >> >> On Wed, May 27, 2009 at 11:36 AM, Chris Colbert  >
> >> >> wrote:
> >> >> > sometimes the docs are a bit dated. But nonetheless, those
> constants
> >> >> > are
> >> >> > just integers defines in the header files and is where I got the
> >> >> > values
> >> >> > 0-4,
> >> >> > I used the integers in the example just to make it a bit more
> clear.
> >> >> > But
> >> >> > 0-4
> >> >> > are valid constants for the call, as they worked in the pure C
> >> >> > example.
> >> >>
> >> >> Mmm... they I have no idea what's going on... Did you try to run
> under
> >> >> valgrind? That usually works for me...
> >> >>
> >> >> > What you showed for exposing the constants however, is interesting,
> I
> >> >> > have
> >> >> > been just defining the constants as module level python integers so
> >> >> > they
> >> >> > can
> >> >> > be used from python. If defined as an enum as you show, will they
> >> >> > be accessible from python?
> >> >> >
> >> >>
> >> >> No, you have to explicitly expose them. If you declared them in some
> >> >> pxd, let say "mycvdefs.pxd", you can write in "mycvmodule.pyx"
> >> >>
> >> >> CV_SOME_CONSTANT = mycvdefs.CV_SOME_CONSTANT
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> > Chris
> >> >> >
> >> >> > On Wed, May 27, 2009 at 9:40 AM, Lisandro Dalcin <
> dalcinl at gmail.com>
> >> >> > wrote:
> >> >> >>
> >> >> >> According to the docs,
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> http://opencv.willowgarage.com/wiki/CvReference#Sampling.2CInterpolationandGeometricalTransforms
> ,
> >> >> >> There are 4 possible values (you mention 5). Is that OpenCV
> >> >> >> documentation dated? IMHO, you should use them. To expose these
> >> >> >> constants in Cython code (likely in some pxd), you can do this:
> >> >> >>
> >> >> >> cdef extern from ....:
> >> >> >>
> >> >> >>    enum: CV_INTER_NN
> >> >> >>    enum: CV_INTER_LINEAR
> >> >> >>    enum: CV_INTER_AREA
> >> >> >>    enum: CV_INTER_CUBIC
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> On Wed, May 27, 2009 at 12:57 AM, Chris Colbert
> >> >> >> 
> >> >> >> wrote:
> >> >> >> > So I'm wrapping the OpenCV library in Cython and its coming
> along
> >> >> >> > quite
> >> >> >> > nicely (thanks to everyone here for getting me up to speed!),
> but
> >> >> >> > today
> >> >> >> > I've
> >> >> >> > run across a strange issue where a wrapped function fails for
> >> >> >> > certain
> >> >> >> > values
> >> >> >> > of integer arguments, but a pure c implementation testing the
> same
> >> >> >> > thing
> >> >> >> > with the same arguments works.
> >> >> >> > Here is the code excerpts im using:
> >> >> >> > #---------- .pyx ------------------#
> >> >> >> > def cvResize(cy_cvtypes.CvArr src, cy_cvtypes.CvArr dst,
> >> >> >> > interpolation):
> >> >> >> >     cdef c_cxcore.CvArr* c_src = src.handle()
> >> >> >> >     cdef c_cxcore.CvArr* c_dst = dst.handle()
> >> >> >> >     cdef int c_interpolation = interpolation
> >> >> >> >     with nogil:
> >> >> >> >         c_cv.cvResize(c_src, c_dst, c_interpolation)
> >> >> >> > #-------- c_cv.pxd -----------------#
> >> >> >> > cdef extern from "cv.h":
> >> >> >> >     void cvResize(CvArr* src, CvArr* dst, int interpolation)
> nogil
> >> >> >> >
> >> >> >> > #--------- test.py -----------------#
> >> >> >> > from cyopencv import *
> >> >> >> > if __name__=='__main__':
> >> >> >> >     img = cvLoadImage('2.bmp')
> >> >> >> >     img2 = cvCreateImage((640, 480), img.depth, img.nChannels)
> >> >> >> >     cvResize(img, img2, 3)
> >> >> >> >     img2.show()
> >> >> >> >
> >> >> >> >
> >> >> >> > #--------- pureC.c -----------#
> >> >> >> > #include "highgui.h"
> >> >> >> > #include "cv.h"
> >> >> >> > #include "cxcore.h"
> >> >> >> > #include "cxtypes.h"
> >> >> >> > int main(void)
> >> >> >> > {
> >> >> >> >     IplImage* img, *img2;
> >> >> >> >     img = cvLoadImage("2.bmp", 1);
> >> >> >> >     img2 = cvCreateImage(cvSize(640, 480), img->depth,
> >> >> >> > img->nChannels);
> >> >> >> >     cvResize(img, img2, 3);
> >> >> >> >     cvNamedWindow("test", 1);
> >> >> >> >     cvShowImage("test", img2);
> >> >> >> >     cvWaitKey(0);
> >> >> >> >     cvDestroyAllWindows();
> >> >> >> >     cvReleaseImage(&img);
> >> >> >> >     cvReleaseImage(&img2);
> >> >> >> > }
> >> >> >> >
> >> >> >> >
> >> >> >> > valid integers for the argument 'interpolation' are 0, 1, 2, 3,
> 4
> >> >> >> > (as
> >> >> >> > tested
> >> >> >> > with the C function). My Cython wrapper fails with a Python
> crash
> >> >> >> > for
> >> >> >> > integer values 1 & 2.  Values of 0,1,4 work properly and the
> >> >> >> > output
> >> >> >> > image
> >> >> >> > matches the output image of the C test function under the same
> >> >> >> > conditions.
> >> >> >> > I've looked at the generated Cython code and everything seems in
> >> >> >> > order,
> >> >> >> > I
> >> >> >> > just can't think of what could be causing this behavior.
> >> >> >> > Any thoughts?
> >> >> >> > Cheers!
> >> >> >> > Chris
> >> >> >> >
> >> >> >> > _______________________________________________
> >> >> >> > Cython-dev mailing list
> >> >> >> > Cython-dev at codespeak.net
> >> >> >> > http://codespeak.net/mailman/listinfo/cython-dev
> >> >> >> >
> >> >> >> >
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> Lisandro Dalc?n
> >> >> >> ---------------
> >> >> >> Centro Internacional de M?todos Computacionales en Ingenier?a
> >> >> >> (CIMEC)
> >> >> >> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica
> >> >> >> (INTEC)
> >> >> >> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas
> (CONICET)
> >> >> >> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> >> >> >> Tel/Fax: +54-(0)342-451.1594
> >> >> >> _______________________________________________
> >> >> >> Cython-dev mailing list
> >> >> >> Cython-dev at codespeak.net
> >> >> >> http://codespeak.net/mailman/listinfo/cython-dev
> >> >> >
> >> >> >
> >> >> > _______________________________________________
> >> >> > Cython-dev mailing list
> >> >> > Cython-dev at codespeak.net
> >> >> > http://codespeak.net/mailman/listinfo/cython-dev
> >> >> >
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Lisandro Dalc?n
> >> >> ---------------
> >> >> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> >> >> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> >> >> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> >> >> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> >> >> Tel/Fax: +54-(0)342-451.1594
> >> >> _______________________________________________
> >> >> Cython-dev mailing list
> >> >> Cython-dev at codespeak.net
> >> >> http://codespeak.net/mailman/listinfo/cython-dev
> >> >
> >> >
> >> > _______________________________________________
> >> > Cython-dev mailing list
> >> > Cython-dev at codespeak.net
> >> > http://codespeak.net/mailman/listinfo/cython-dev
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Lisandro Dalc?n
> >> ---------------
> >> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> >> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> >> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> >> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> >> Tel/Fax: +54-(0)342-451.1594
> >> _______________________________________________
> >> Cython-dev mailing list
> >> Cython-dev at codespeak.net
> >> http://codespeak.net/mailman/listinfo/cython-dev
> >
> >
> > _______________________________________________
> > Cython-dev mailing list
> > Cython-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/cython-dev
> >
> >
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090527/963b97bc/attachment-0001.htm 

From dagss at student.matnat.uio.no  Wed May 27 23:06:46 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 27 May 2009 23:06:46 +0200
Subject: [Cython] having a weird issue with wrapped function failing in
 only certain cases
In-Reply-To: <7f014ea60905271256u6b49d408ta3babe4b8f50809a@mail.gmail.com>
References: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>		<7f014ea60905270736v682b1257jdeeb8fbd230282ba@mail.gmail.com>		<7f014ea60905270903y7e90c4f2ic014536b55bdf8b9@mail.gmail.com>		<7f014ea60905271109p463c9544y66cd9c9a09eb937e@mail.gmail.com>		<7f014ea60905271244v39747565wd70872778b22d16c@mail.gmail.com>	
	<7f014ea60905271256u6b49d408ta3babe4b8f50809a@mail.gmail.com>
Message-ID: <4A1DAB66.10901@student.matnat.uio.no>

Chris Colbert wrote:
> 
>     Quick note: By linker bug I really meant a bug in your make system,
>     not in
>     the linker itself :-) But I guess that's how you interpreted it.
> 
>     Dag Sverre
> 
>     _______________________________________________
> 
> 
> 
> Here is a pure cython example that exhibits the same behavior. The 
> cimports are simply .pxd files of header prototypes.
> 
> 
> #---------- purecython.pyx -----------------#
> cimport c_cxcore
> cimport c_highgui
> cimport c_cv
> cimport c_constants
> 
> cdef tester():
>     cdef c_cxcore.IplImage* img1
>     cdef c_cxcore.IplImage* img2
>     img1 = c_highgui.cvLoadImage("2.bmp", 1)
>     img2 = c_cxcore.cvCreateImage(c_cxcore.cvSize(640, 480), 8, 3)
>     c_cv.cvResize(img1, img2, c_constants.CV_INTER_AREA)
>     c_highgui.cvNamedWindow("test", 1)
>     c_highgui.cvShowImage("test", img2)
>     c_highgui.cvWaitKey(0)
>    
> def test():
>     tester()

Well, you never free the memory for img1 and img2, but that shouldn't 
matter.

Anyway, just to be clear, you are saying that this crashes, but the 
exact same C code, that is

IplImage* img1;
IplImage* img2;
img1 = cvLoadImage("2.bmp", 1);
... and so on

does not crash?

(If the pure C code ever crashes you must bring this to the OpenCV 
forums instead.)

In that case, do manual inspection of the Cython-generated C source -- 
is there any difference at all (up to variable names etc.) between 
Cython-generated and manual C code?

-- 
Dag Sverre

From sccolbert at gmail.com  Wed May 27 23:39:16 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Wed, 27 May 2009 17:39:16 -0400
Subject: [Cython] having a weird issue with wrapped function failing in
	only certain cases
In-Reply-To: <4A1DAB66.10901@student.matnat.uio.no>
References: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>
	
	<7f014ea60905270903y7e90c4f2ic014536b55bdf8b9@mail.gmail.com>
	
	<7f014ea60905271109p463c9544y66cd9c9a09eb937e@mail.gmail.com>
	
	<7f014ea60905271244v39747565wd70872778b22d16c@mail.gmail.com>
	
	<7f014ea60905271256u6b49d408ta3babe4b8f50809a@mail.gmail.com>
	<4A1DAB66.10901@student.matnat.uio.no>
Message-ID: <7f014ea60905271439j2827e1b5l33ca3fdea2528e12@mail.gmail.com>

Dag,

I've listed the code below and added the memory cleanup to the Cython file.
Behavior is the same.

#-------the pure C code which doesnt crash-----------#

#include "highgui.h"
#include "cv.h"
#include "cxcore.h"
#include "cxtypes.h"

int main(void)
{
    IplImage* img, *img2;
    img = cvLoadImage("2.bmp", 1);
    img2 = cvCreateImage(cvSize(640, 480), img->depth, img->nChannels);
    cvResize(img, img2, CV_INTER_LINEAR);
    cvNamedWindow("test", 1);
    cvShowImage("test", img2);
    cvWaitKey(0);
    cvDestroyAllWindows();
    cvReleaseImage(&img);
    cvReleaseImage(&img2);
}



#-------the corresponding "identical" Cython file which does crash:-------#

cimport c_cxcore
cimport c_highgui
cimport c_cv
cimport c_constants

cdef tester():
    cdef c_cxcore.IplImage* img
    cdef c_cxcore.IplImage* img2
    img = c_highgui.cvLoadImage("2.bmp", 1)
    img2 = c_cxcore.cvCreateImage(c_cxcore.cvSize(640, 480), img.depth,
img.nChannels)
    c_cv.cvResize(img, img2, c_constants.CV_INTER_LINEAR)
    c_highgui.cvNamedWindow("test", 1)
    c_highgui.cvShowImage("test", img2)
    c_highgui.cvWaitKey(0)
    c_highgui.cvDestroyAllWindows()
    c_cxcore.cvReleaseImage(&img)
    c_cxcore.cvReleaseImage(&img2)

def test():
    tester()


#--------- Cython Generated C - Code (implementation only)---------------#

#include "cxtypes.h"
#include "cxcore.h"
#include "highgui.h"
#include "cv.h"
#include "cvtypes.h"


/* Implementation of purecythontest */

static char __pyx_k___main__[] = "__main__";
static PyObject *__pyx_kp___main__;
static char __pyx_k_1[] = "2.bmp";
static char __pyx_k_2[] = "test";
static char __pyx_k_3[] = "test";


static  PyObject *__pyx_f_14purecythontest_tester(void) {
  IplImage *__pyx_v_img;
  IplImage *__pyx_v_img2;
  PyObject *__pyx_r = NULL;
  __Pyx_SetupRefcountContext("tester");


  __pyx_v_img = cvLoadImage(__pyx_k_1, 1);

  __pyx_v_img2 = cvCreateImage(cvSize(640, 480), __pyx_v_img->depth,
__pyx_v_img->nChannels);

  cvResize(__pyx_v_img, __pyx_v_img2, CV_INTER_LINEAR);

  cvNamedWindow(__pyx_k_2, 1);

  cvShowImage(__pyx_k_3, __pyx_v_img2);

  cvWaitKey(0);

  cvDestroyAllWindows();

  cvReleaseImage((&__pyx_v_img));

    cvReleaseImage((&__pyx_v_img2));

  __pyx_r = Py_None; __Pyx_INCREF(Py_None);
  __Pyx_XGIVEREF(__pyx_r);
  __Pyx_FinishRefcountContext();
  return __pyx_r;
}




To me, the two C files look equivalent. Do you notice anything out of the
ordinary?

Cheers!

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090527/5c0df51d/attachment.htm 

From frostedcheerios at gmail.com  Thu May 28 00:45:12 2009
From: frostedcheerios at gmail.com (David Martin)
Date: Wed, 27 May 2009 17:45:12 -0500
Subject: [Cython] Running Cython 0.11.2 tests using MinGW32
In-Reply-To: <4A1DB976.4060309@gmail.com>
References: <4A1DB976.4060309@gmail.com>
Message-ID: <4A1DC278.9080704@gmail.com>

Is it possible and/or supported to run the Cython tests using MinGW32 on 
Windows? There isn't an option for it in runtests.py, and I do not have 
a set of Visual Studio compilers available to compile with. Thanks.

-David

From dalcinl at gmail.com  Thu May 28 01:33:27 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 27 May 2009 20:33:27 -0300
Subject: [Cython] having a weird issue with wrapped function failing in
	only certain cases
In-Reply-To: <7f014ea60905271439j2827e1b5l33ca3fdea2528e12@mail.gmail.com>
References: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>
	<7f014ea60905270903y7e90c4f2ic014536b55bdf8b9@mail.gmail.com>
	
	<7f014ea60905271109p463c9544y66cd9c9a09eb937e@mail.gmail.com>
	
	<7f014ea60905271244v39747565wd70872778b22d16c@mail.gmail.com>
	
	<7f014ea60905271256u6b49d408ta3babe4b8f50809a@mail.gmail.com>
	<4A1DAB66.10901@student.matnat.uio.no>
	<7f014ea60905271439j2827e1b5l33ca3fdea2528e12@mail.gmail.com>
Message-ID: 

On Wed, May 27, 2009 at 6:39 PM, Chris Colbert  wrote:
>
> To me, the two C files look equivalent. Do you notice anything out of the
> ordinary?
>

A final suggestion... Do you know if it is possible to "force" MinGW
to use the same MS C runtime library your Python 2.5 is using? In such
case, I would try to build OpenCV in such way...



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Thu May 28 01:36:29 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 27 May 2009 20:36:29 -0300
Subject: [Cython] Running Cython 0.11.2 tests using MinGW32
In-Reply-To: <4A1DC278.9080704@gmail.com>
References: <4A1DB976.4060309@gmail.com> <4A1DC278.9080704@gmail.com>
Message-ID: 

On Wed, May 27, 2009 at 7:45 PM, David Martin  wrote:
> Is it possible and/or supported to run the Cython tests using MinGW32 on
> Windows? There isn't an option for it in runtests.py, and I do not have
> a set of Visual Studio compilers available to compile with. Thanks.
>

Yes, I do this at home. Just a couple of lines need to be changed in
runtests.py. Wait me a couple of hours and I'll post the diff (of
contact you at Gmail chat if you are online).

However, be prepared for some failures (mostly because the linker
complains about some "cdef extern" definitions). Perhaps you could
help me to fix all that crap?




-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Thu May 28 01:37:51 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Wed, 27 May 2009 20:37:51 -0300
Subject: [Cython] Running Cython 0.11.2 tests using MinGW32
In-Reply-To: 
References: <4A1DB976.4060309@gmail.com> <4A1DC278.9080704@gmail.com>
	
Message-ID: 

BTW, we could add a '--mingw' option to runtests.py, right?

On Wed, May 27, 2009 at 8:36 PM, Lisandro Dalcin  wrote:
> On Wed, May 27, 2009 at 7:45 PM, David Martin  wrote:
>> Is it possible and/or supported to run the Cython tests using MinGW32 on
>> Windows? There isn't an option for it in runtests.py, and I do not have
>> a set of Visual Studio compilers available to compile with. Thanks.
>>
>
> Yes, I do this at home. Just a couple of lines need to be changed in
> runtests.py. Wait me a couple of hours and I'll post the diff (of
> contact you at Gmail chat if you are online).
>
> However, be prepared for some failures (mostly because the linker
> complains about some "cdef extern" definitions). Perhaps you could
> help me to fix all that crap?
>
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From frostedcheerios at gmail.com  Thu May 28 01:42:55 2009
From: frostedcheerios at gmail.com (David Martin)
Date: Wed, 27 May 2009 18:42:55 -0500
Subject: [Cython] Cython 0.11.2 test errors
Message-ID: <4A1DCFFF.1030906@gmail.com>

I am getting quite a few errors when running the Cython tests through 
runtests.py. This is on a machine with VS .NET 2003 installed, using the 
command 'python runtests.py --no-refnanny". I get the same set of errors 
when I run the tests using the mingw compiler.

-David M

Running tests against Cython 0.11.2
Python 2.5.4 |Master installer 1.0.2| (r254:67916, Mar 25 2009, 
13:58:21) [MSC v.1310 32 bit (Intel)]

   Creating library 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\c_directives.lib 
and object 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\c_directives.exp
callingconvention.c
   Creating library 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\callingconvention.lib 
and object 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\callingconvention.exp
callingconvention.obj : error LNK2019: unresolved external symbol @f4 at 0 
referenced in function _initcallingconvention
callingconvention.obj : error LNK2019: unresolved external symbol _p4 
referenced in function _initcallingconvention
callingconvention.obj : error LNK2019: unresolved external symbol _f3 at 0 
referenced in function _initcallingconvention
callingconvention.obj : error LNK2019: unresolved external symbol _p3 
referenced in function _initcallingconvention
callingconvention.obj : error LNK2019: unresolved external symbol _f2 
referenced in function _initcallingconvention
callingconvention.obj : error LNK2019: unresolved external symbol _p2 
referenced in function _initcallingconvention
callingconvention.obj : error LNK2019: unresolved external symbol _f1 
referenced in function _initcallingconvention
callingconvention.obj : error LNK2019: unresolved external symbol _p1 
referenced in function _initcallingconvention
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\callingconvention.pyd 
: fatal error LNK1120: 8 unresolved externals
callingconvention.cpp
   Creating library 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\callingconvention.lib 
and object 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\callingconvention.exp
callingconvention.obj : error LNK2019: unresolved external symbol @f4 at 0 
referenced in function _initcallingconvention
callingconvention.obj : error LNK2019: unresolved external symbol _p4 
referenced in function _initcallingconvention
callingconvention.obj : error LNK2019: unresolved external symbol _f3 at 0 
referenced in function _initcallingconvention
callingconvention.obj : error LNK2019: unresolved external symbol _p3 
referenced in function _initcallingconvention
callingconvention.obj : error LNK2019: unresolved external symbol _f2 
referenced in function _initcallingconvention
callingconvention.obj : error LNK2019: unresolved external symbol _p2 
referenced in function _initcallingconvention
callingconvention.obj : error LNK2019: unresolved external symbol _f1 
referenced in function _initcallingconvention
callingconvention.obj : error LNK2019: unresolved external symbol _p1 
referenced in function _initcallingconvention
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\callingconvention.pyd 
: fatal error LNK1120: 8 unresolved externals

coercearraytoptr.c
   Creating library 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\coercearraytoptr.lib 
and object 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\coercearraytoptr.exp
coercearraytoptr.obj : error LNK2019: unresolved external symbol _spam 
referenced in function ___pyx_f_16coercearraytoptr_eggs
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\coercearraytoptr.pyd 
: fatal error LNK1120: 1 unresolved externals
coercearraytoptr.cpp
   Creating library 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\coercearraytoptr.lib 
and object 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\coercearraytoptr.exp
coercearraytoptr.obj : error LNK2019: unresolved external symbol _spam 
referenced in function "void __cdecl 
__pyx_f_16coercearraytoptr_eggs(void)" 
(?__pyx_f_16coercearraytoptr_eggs@@YAXXZ)
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\coercearraytoptr.pyd 
: fatal error LNK1120: 1 unresolved externals
coercetovoidptr.c
   Creating library 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\coercetovoidptr.lib 
and object 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\coercetovoidptr.exp
coercetovoidptr.cpp

   Creating library 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\cpdef.lib 
and object 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\cpdef.exp
cpp_exceptions_T265.cpp
   Creating library 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\cpp_exceptions_T265.lib 
and object 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\cpp_exceptions_T265.exp
cpp_exceptions_T265.obj : error LNK2019: unresolved external symbol "int 
__cdecl dynamic_error(void)" (?dynamic_error@@YAHXZ) referenced in 
function "struct _object * __cdecl 
__pyx_pf_19cpp_exceptions_T265_test_it(struct _object *,struct _object 
*)" (?__pyx_pf_19cpp_exceptions_T265_test_it@@YAPAU_object@@PAU1 at 0@Z)
cpp_exceptions_T265.obj : error LNK2019: unresolved external symbol "int 
__cdecl specified_error(void)" (?specified_error@@YAHXZ) referenced in 
function "struct _object * __cdecl 
__pyx_pf_19cpp_exceptions_T265_test_it(struct _object *,struct _object 
*)" (?__pyx_pf_19cpp_exceptions_T265_test_it@@YAPAU_object@@PAU1 at 0@Z)
cpp_exceptions_T265.obj : error LNK2019: unresolved external symbol "int 
__cdecl generic_error(void)" (?generic_error@@YAHXZ) referenced in 
function "struct _object * __cdecl 
__pyx_pf_19cpp_exceptions_T265_test_it(struct _object *,struct _object 
*)" (?__pyx_pf_19cpp_exceptions_T265_test_it@@YAPAU_object@@PAU1 at 0@Z)
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\cpp_exceptions_T265.pyd 
: fatal error LNK1120: 3 unresolved externals
cstructreturn.c
e:\py\build\trunk\recipes\cython-0.11.2\tests\build\compile\c\cstructreturn.c(357) 
: warning C4700: local variable '__pyx_r' used without having been 
initialized
   Creating library 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\cstructreturn.lib 
and object 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\cstructreturn.exp
cstructreturn.cpp

   Creating library 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\declarations.lib 
and object 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\declarations.exp
declarations.obj : error LNK2019: unresolved external symbol _ifnp 
referenced in function ___pyx_f_12declarations_f
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\declarations.pyd 
: fatal error LNK1120: 1 unresolved externals
declarations.cpp
   Creating library 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\declarations.lib 
and object 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\declarations.exp
declarations.obj : error LNK2019: unresolved external symbol _ifnp 
referenced in function "void __cdecl __pyx_f_12declarations_f(void)" 
(?__pyx_f_12declarations_f@@YAXXZ)
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\declarations.pyd 
: fatal error LNK1120: 1 unresolved externals

excvalcheck.c
   Creating library 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\excvalcheck.lib 
and object 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\excvalcheck.exp
excvalcheck.obj : error LNK2019: unresolved external symbol _tomato 
referenced in function ___pyx_f_11excvalcheck_eggs
excvalcheck.obj : error LNK2019: unresolved external symbol _grail 
referenced in function ___pyx_f_11excvalcheck_eggs
excvalcheck.obj : error LNK2019: unresolved external symbol _spam 
referenced in function ___pyx_f_11excvalcheck_eggs
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\excvalcheck.pyd 
: fatal error LNK1120: 3 unresolved externals
excvalcheck.cpp
   Creating library 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\excvalcheck.lib 
and object 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\excvalcheck.exp
excvalcheck.obj : error LNK2019: unresolved external symbol _tomato 
referenced in function "void __cdecl __pyx_f_11excvalcheck_eggs(void)" 
(?__pyx_f_11excvalcheck_eggs@@YAXXZ)
excvalcheck.obj : error LNK2019: unresolved external symbol _grail 
referenced in function "void __cdecl __pyx_f_11excvalcheck_eggs(void)" 
(?__pyx_f_11excvalcheck_eggs@@YAXXZ)
excvalcheck.obj : error LNK2019: unresolved external symbol _spam 
referenced in function "void __cdecl __pyx_f_11excvalcheck_eggs(void)" 
(?__pyx_f_11excvalcheck_eggs@@YAXXZ)
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\excvalcheck.pyd 
: fatal error LNK1120: 3 unresolved externals

nogil.c
   Creating library 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\nogil.lib 
and object 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\nogil.exp
nogil.obj : error LNK2019: unresolved external symbol _e2 referenced in 
function ___pyx_f_5nogil_h
nogil.obj : error LNK2019: unresolved external symbol _e1 referenced in 
function ___pyx_f_5nogil_h
nogil.obj : error LNK2019: unresolved external symbol _g2 referenced in 
function ___pyx_f_5nogil_h
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\c\nogil.pyd 
: fatal error LNK1120: 3 unresolved externals
nogil.cpp
   Creating library 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\nogil.lib 
and object 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\nogil.exp
nogil.obj : error LNK2019: unresolved external symbol "void * __cdecl 
e2(void)" (?e2@@YAPAXXZ) referenced in function "void __cdecl 
__pyx_f_5nogil_h(struct _object *)" (?__pyx_f_5nogil_h@@YAXPAU_object@@@Z)
nogil.obj : error LNK2019: unresolved external symbol "void __cdecl 
e1(void)" (?e1@@YAXXZ) referenced in function "void __cdecl 
__pyx_f_5nogil_h(struct _object *)" (?__pyx_f_5nogil_h@@YAXPAU_object@@@Z)
nogil.obj : error LNK2019: unresolved external symbol _g2 referenced in 
function "void __cdecl __pyx_f_5nogil_h(struct _object *)" 
(?__pyx_f_5nogil_h@@YAXPAU_object@@@Z)
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\compile\cpp\nogil.pyd 
: fatal error LNK1120: 3 unresolved externals

r_pernici1.c
r_pernici1.c(348) : error C2065: 'M_PI' : undeclared identifier
r_pernici1.cpp
r_pernici1.cpp(348) : error C2065: 'M_PI' : undeclared identifier
r_pernici1.cpp(366) : error C3861: 'M_PI': identifier not found, even 
with argument-dependent lookup

callargs.c
callargs.c(398) : error C2026: string too big, trailing characters truncated
callargs.cpp
callargs.cpp(398) : error C2026: string too big, trailing characters 
truncated

cdivision_CEP_516.c
cdivision_CEP_516.c(676) : warning C4244: '=' : conversion from 'double' 
to 'float', possible loss of data
cdivision_CEP_516.c(677) : warning C4244: '=' : conversion from 'double' 
to 'float', possible loss of data
cdivision_CEP_516.c(681) : warning C4244: '=' : conversion from 'double' 
to 'float', possible loss of data
cdivision_CEP_516.c(682) : warning C4244: '=' : conversion from 'double' 
to 'float', possible loss of data
cdivision_CEP_516.c(850) : warning C4244: '=' : conversion from 'double' 
to 'float', possible loss of data
cdivision_CEP_516.c(851) : warning C4244: '=' : conversion from 'double' 
to 'float', possible loss of data
cdivision_CEP_516.c(855) : warning C4244: '=' : conversion from 'double' 
to 'float', possible loss of data
cdivision_CEP_516.c(856) : warning C4244: '=' : conversion from 'double' 
to 'float', possible loss of data
cdivision_CEP_516.c(875) : warning C4013: 'fmodf' undefined; assuming 
extern returning int
cdivision_CEP_516.c(1939) : warning C4244: '=' : conversion from 
'double' to 'float', possible loss of data
cdivision_CEP_516.c(1940) : warning C4244: '=' : conversion from 
'double' to 'float', possible loss of data
cdivision_CEP_516.c(1941) : warning C4244: '=' : conversion from 
'double' to 'float', possible loss of data
cdivision_CEP_516.c(1945) : warning C4244: '=' : conversion from 
'double' to 'float', possible loss of data
cdivision_CEP_516.c(1946) : warning C4244: '=' : conversion from 
'double' to 'float', possible loss of data
cdivision_CEP_516.c(1947) : warning C4244: '=' : conversion from 
'double' to 'float', possible loss of data
cdivision_CEP_516.c(2482) : warning C4244: 'initializing' : conversion 
from 'int' to 'float', possible loss of data
   Creating library 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\run\c\cdivision_CEP_516.lib 
and object 
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\run\c\cdivision_CEP_516.exp
cdivision_CEP_516.obj : error LNK2019: unresolved external symbol _fmodf 
referenced in function ___Pyx_mod_float
E:\py\build\trunk\recipes\Cython-0.11.2\tests\BUILD\run\c\cdivision_CEP_516.pyd 
: fatal error LNK1120: 1 unresolved externals

classkwonlyargs.c
classkwonlyargs.c(313) : error C2026: string too big, trailing 
characters truncated
classkwonlyargs.cpp
classkwonlyargs.cpp(313) : error C2026: string too big, trailing 
characters truncated

embedsignatures.c
embedsignatures.c(382) : error C2026: string too big, trailing 
characters truncated
embedsignatures.c(1532) : warning C4244: '=' : conversion from 'double' 
to 'float', possible loss of data
embedsignatures.c(1897) : warning C4244: '=' : conversion from 'double' 
to 'float', possible loss of data
embedsignatures.c(1905) : warning C4244: '=' : conversion from 'double' 
to 'float', possible loss of data
embedsignatures.c(3642) : warning C4244: '=' : conversion from 'double' 
to 'float', possible loss of data
embedsignatures.cpp
embedsignatures.cpp(382) : error C2026: string too big, trailing 
characters truncated
embedsignatures.cpp(1532) : warning C4244: '=' : conversion from 
'double' to 'float', possible loss of data
embedsignatures.cpp(1897) : warning C4244: '=' : conversion from 
'double' to 'float', possible loss of data
embedsignatures.cpp(1905) : warning C4244: '=' : conversion from 
'double' to 'float', possible loss of data
embedsignatures.cpp(3642) : warning C4244: '=' : conversion from 
'double' to 'float', possible loss of data

extkwonlyargs.c
extkwonlyargs.c(324) : error C2026: string too big, trailing characters 
truncated
extkwonlyargs.cpp
extkwonlyargs.cpp(324) : error C2026: string too big, trailing 
characters truncated

extstarargs.c
extstarargs.c(330) : error C2026: string too big, trailing characters 
truncated
extstarargs.cpp
extstarargs.cpp(330) : error C2026: string too big, trailing characters 
truncated

kwonlyargs.c
kwonlyargs.c(311) : error C2026: string too big, trailing characters 
truncated
kwonlyargs.cpp
kwonlyargs.cpp(311) : error C2026: string too big, trailing characters 
truncated
kwonlyargscall.c
kwonlyargscall.c(317) : error C2026: string too big, trailing characters 
truncated
kwonlyargscall.cpp
kwonlyargscall.cpp(317) : error C2026: string too big, trailing 
characters truncated

starargs.c
starargs.c(313) : error C2026: string too big, trailing characters truncated
starargs.cpp
starargs.cpp(313) : error C2026: string too big, trailing characters 
truncated

----------------------------------------------------------------------

Ran 1460 tests in 316.701s
FAILED (errors=30)

From dalcinl at gmail.com  Thu May 28 05:56:27 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 28 May 2009 00:56:27 -0300
Subject: [Cython] long double and MSVC
Message-ID: 

See this, (in particular the note about MSVC)

http://en.wikipedia.org/wiki/Long_double

In numpy_test.pyx, a doctest line associates "g" (numpy.float96) to
longdouble, then the test fail because dtype.itemsize != sizeof(long
double).

Should we disable such doctest lines if 'win' in sys.platform ??




-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Thu May 28 06:11:27 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 28 May 2009 01:11:27 -0300
Subject: [Cython] long double and MSVC
In-Reply-To: 
References: 
Message-ID: 

BTW, typedfieldbug_T303.py is also affected by this
sizeof(double)==sizeof(long double) in MSVC.


On Thu, May 28, 2009 at 12:56 AM, Lisandro Dalcin  wrote:
> See this, (in particular the note about MSVC)
>
> http://en.wikipedia.org/wiki/Long_double
>
> In numpy_test.pyx, a doctest line associates "g" (numpy.float96) to
> longdouble, then the test fail because dtype.itemsize != sizeof(long
> double).
>
> Should we disable such doctest lines if 'win' in sys.platform ??
>
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dagss at student.matnat.uio.no  Thu May 28 06:18:35 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 28 May 2009 06:18:35 +0200 (CEST)
Subject: [Cython] Cython 0.11.2 test errors
In-Reply-To: <4A1DCFFF.1030906@gmail.com>
References: <4A1DCFFF.1030906@gmail.com>
Message-ID: 

David Martin wrote:
> I am getting quite a few errors when running the Cython tests through
> runtests.py. This is on a machine with VS .NET 2003 installed, using the
> command 'python runtests.py --no-refnanny". I get the same set of errors
> when I run the tests using the mingw compiler.

There's a similar report here:

http://trac.cython.org/cython_trac/ticket/106

The problem is simply that there's a lack of Microsoft products among some
of the Cython developers :-) I think we really need an additional person
with MSVC knowledge willing to maintain this side of Cython.

Another option would be trying to get a http://www.snakebite.org/ account,
which should provide reliable Windows access...

Do you mean you get "roughly the same errors" with mingw, or "the exact
same text" with mingw? If the latter you are likely not compiling with
mingw anyway.

Dag Sverre


From dagss at student.matnat.uio.no  Thu May 28 06:49:02 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 28 May 2009 06:49:02 +0200 (CEST)
Subject: [Cython] long double and MSVC
In-Reply-To: 
References: 
	
Message-ID: <06b2e618434288afaea0ad8725271408.squirrel@webmail.uio.no>

Lisandro wrote:
> BTW, typedfieldbug_T303.py is also affected by this
> sizeof(double)==sizeof(long double) in MSVC.

Yes I figured :-)

It's just a badly written testcase I suppose. I think the testcases can
just check for sizeof(double) == sizeof(long double) directly though
rather than sys.platform; IIRC this applies on gcc on PPC as well.

I'm really starting to crave an account on snakebite here :-)

Dag Sverre


From sccolbert at gmail.com  Thu May 28 07:00:05 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Thu, 28 May 2009 01:00:05 -0400
Subject: [Cython] having a weird issue with wrapped function failing in
	only certain cases
In-Reply-To: 
References: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>
	
	<7f014ea60905271109p463c9544y66cd9c9a09eb937e@mail.gmail.com>
	
	<7f014ea60905271244v39747565wd70872778b22d16c@mail.gmail.com>
	
	<7f014ea60905271256u6b49d408ta3babe4b8f50809a@mail.gmail.com>
	<4A1DAB66.10901@student.matnat.uio.no>
	<7f014ea60905271439j2827e1b5l33ca3fdea2528e12@mail.gmail.com>
	
Message-ID: <7f014ea60905272200q5afd9af4kaa2399158d2c4f63@mail.gmail.com>

I think I'm starting to narrow it down.

I made the following python extension module by hand and it works. I can
pass any value (0, 1, 2, or 3) to the function exttest.test() in python and
they all work properly.

#------------- exttest.c ---------------------#

#include "Python.h"
#include "highgui.h"
#include "cxcore.h"
#include "cv.h"

static PyObject* run_test(PyObject* self, PyObject* args)
{
    int val;
    if (!PyArg_ParseTuple(args, "i", &val)){
        return NULL;
    }
    IplImage* img = cvLoadImage("2.bmp", 1);
    IplImage* img2 = cvCreateImage(cvSize(640, 480), img->depth,
img->nChannels);
    cvResize(img, img2, val);
    cvNamedWindow("test", 1);
    cvShowImage("test", img2);
    cvWaitKey(0);
    cvDestroyAllWindows();
    cvReleaseImage(&img);
    cvReleaseImage(&img2);
    Py_RETURN_NONE;
}

static PyMethodDef exttestMethods[] =
{{"test", run_test, METH_VARARGS, "some test"},{NULL, NULL, 0, NULL}};

PyMODINIT_FUNC initexttest(void)
{
    (void) Py_InitModule("exttest", exttestMethods);
}




#------------------ setup.py build ---------------------#
from distutils.core import setup, Extension

library_dirs=["C:\\OpenCVsvn\\build\\lib"]
include_dirs=["C:\\OpenCVsvn\\build\\include\\opencv"]
libraries = ["highgui110", "cxcore110", "cv110"]

module1 = Extension('exttest', sources = ['exttest.c'],
libraries=libraries,
library_dirs=library_dirs, include_dirs=include_dirs)

setup (ext_modules = [module1])




Now, If I go back to the Cython generated .c file and copy this into its
body (removing all other code), and build the .c file with an equivalent
setup.py  it still fails on integers 1 and 2 (but not 0 and 3)

#------------ code substituted into cython c file -------------- #

static PyObject *__pyx_pf_5cy_cv_cvResize(PyObject *__pyx_self, PyObject
*__pyx_args, PyObject *__pyx_kwds)
{
    int val = 1;
    IplImage* img = cvLoadImage("2.bmp", 1);
    IplImage* img2 = cvCreateImage(cvSize(640, 480), img->depth,
img->nChannels);
    cvResize(img, img2, val);
    cvNamedWindow("test", 1);
    cvShowImage("test", img2);
    cvWaitKey(0);
    cvDestroyAllWindows();
    cvReleaseImage(&img);
    cvReleaseImage(&img2);
    Py_RETURN_NONE;
}


So, I think I've ruled out a compiler issue. Any pointers on where to look
now?

Cheers!

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090528/9c0ddcbb/attachment-0001.htm 

From sccolbert at gmail.com  Thu May 28 07:01:40 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Thu, 28 May 2009 01:01:40 -0400
Subject: [Cython] having a weird issue with wrapped function failing in
	only certain cases
In-Reply-To: <7f014ea60905272200q5afd9af4kaa2399158d2c4f63@mail.gmail.com>
References: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>
	<7f014ea60905271109p463c9544y66cd9c9a09eb937e@mail.gmail.com>
	
	<7f014ea60905271244v39747565wd70872778b22d16c@mail.gmail.com>
	
	<7f014ea60905271256u6b49d408ta3babe4b8f50809a@mail.gmail.com>
	<4A1DAB66.10901@student.matnat.uio.no>
	<7f014ea60905271439j2827e1b5l33ca3fdea2528e12@mail.gmail.com>
	
	<7f014ea60905272200q5afd9af4kaa2399158d2c4f63@mail.gmail.com>
Message-ID: <7f014ea60905272201j2eff6573o1655d3595135140@mail.gmail.com>

I should note in the last code example, I manually change the integer and
the recompile. i.e. the python arguments passed to the function are
irrelevant.
Chris

On Thu, May 28, 2009 at 1:00 AM, Chris Colbert  wrote:

> I think I'm starting to narrow it down.
>
> I made the following python extension module by hand and it works. I can
> pass any value (0, 1, 2, or 3) to the function exttest.test() in python and
> they all work properly.
>
> #------------- exttest.c ---------------------#
>
> #include "Python.h"
> #include "highgui.h"
> #include "cxcore.h"
> #include "cv.h"
>
> static PyObject* run_test(PyObject* self, PyObject* args)
> {
>     int val;
>     if (!PyArg_ParseTuple(args, "i", &val)){
>         return NULL;
>     }
>     IplImage* img = cvLoadImage("2.bmp", 1);
>     IplImage* img2 = cvCreateImage(cvSize(640, 480), img->depth,
> img->nChannels);
>     cvResize(img, img2, val);
>     cvNamedWindow("test", 1);
>     cvShowImage("test", img2);
>     cvWaitKey(0);
>     cvDestroyAllWindows();
>     cvReleaseImage(&img);
>     cvReleaseImage(&img2);
>     Py_RETURN_NONE;
> }
>
> static PyMethodDef exttestMethods[] =
> {{"test", run_test, METH_VARARGS, "some test"},{NULL, NULL, 0, NULL}};
>
> PyMODINIT_FUNC initexttest(void)
> {
>     (void) Py_InitModule("exttest", exttestMethods);
> }
>
>
>
>
> #------------------ setup.py build ---------------------#
> from distutils.core import setup, Extension
>
> library_dirs=["C:\\OpenCVsvn\\build\\lib"]
> include_dirs=["C:\\OpenCVsvn\\build\\include\\opencv"]
> libraries = ["highgui110", "cxcore110", "cv110"]
>
> module1 = Extension('exttest', sources = ['exttest.c'],
> libraries=libraries,
> library_dirs=library_dirs, include_dirs=include_dirs)
>
> setup (ext_modules = [module1])
>
>
>
>
> Now, If I go back to the Cython generated .c file and copy this into its
> body (removing all other code), and build the .c file with an equivalent
> setup.py  it still fails on integers 1 and 2 (but not 0 and 3)
>
> #------------ code substituted into cython c file -------------- #
>
> static PyObject *__pyx_pf_5cy_cv_cvResize(PyObject *__pyx_self, PyObject
> *__pyx_args, PyObject *__pyx_kwds)
> {
>     int val = 1;
>     IplImage* img = cvLoadImage("2.bmp", 1);
>     IplImage* img2 = cvCreateImage(cvSize(640, 480), img->depth,
> img->nChannels);
>     cvResize(img, img2, val);
>     cvNamedWindow("test", 1);
>     cvShowImage("test", img2);
>     cvWaitKey(0);
>     cvDestroyAllWindows();
>     cvReleaseImage(&img);
>     cvReleaseImage(&img2);
>     Py_RETURN_NONE;
> }
>
>
> So, I think I've ruled out a compiler issue. Any pointers on where to look
> now?
>
> Cheers!
>
> Chris
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090528/38cd21ac/attachment.htm 

From dalcinl at gmail.com  Thu May 28 07:14:19 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 28 May 2009 02:14:19 -0300
Subject: [Cython] Cython 0.11.2 test errors
In-Reply-To: 
References: <4A1DCFFF.1030906@gmail.com>
	
Message-ID: 

On Thu, May 28, 2009 at 1:18 AM, Dag Sverre Seljebotn
 wrote:
>
> Do you mean you get "roughly the same errors" with mingw, or "the exact
> same text" with mingw? If the latter you are likely not compiling with
> mingw anyway.
>

Some tests in "tests/compile" use declarations like "cdef extern void
somefunc()". Then the linker complains that "somefunc" is undefined.
This happens on WinDog with MSVC and also with MinGW.


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Thu May 28 07:29:47 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 28 May 2009 02:29:47 -0300
Subject: [Cython] having a weird issue with wrapped function failing in
	only certain cases
In-Reply-To: <7f014ea60905272200q5afd9af4kaa2399158d2c4f63@mail.gmail.com>
References: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>
	<7f014ea60905271109p463c9544y66cd9c9a09eb937e@mail.gmail.com>
	
	<7f014ea60905271244v39747565wd70872778b22d16c@mail.gmail.com>
	
	<7f014ea60905271256u6b49d408ta3babe4b8f50809a@mail.gmail.com>
	<4A1DAB66.10901@student.matnat.uio.no>
	<7f014ea60905271439j2827e1b5l33ca3fdea2528e12@mail.gmail.com>
	
	<7f014ea60905272200q5afd9af4kaa2399158d2c4f63@mail.gmail.com>
Message-ID: 

On Thu, May 28, 2009 at 2:00 AM, Chris Colbert  wrote:
>
> So, I think I've ruled out a compiler issue. Any pointers on where to look
> now?
>

I told you...

1) When building core OpenVC, any chance you or the buildsystem is
passing some special flag to MinGW?

2) Do you know if it is possible  to ask MinGW to use a specific C
runtime lib (the same used for Win Py2.5)?

3) Could you give a try to Python 2.6 the and matching MSVC, and use
that MSVC to build core OpenCV and your Cython extension?



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From dalcinl at gmail.com  Thu May 28 07:34:02 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 28 May 2009 02:34:02 -0300
Subject: [Cython] long double and MSVC
In-Reply-To: <06b2e618434288afaea0ad8725271408.squirrel@webmail.uio.no>
References: 
	
	<06b2e618434288afaea0ad8725271408.squirrel@webmail.uio.no>
Message-ID: 

On Thu, May 28, 2009 at 1:49 AM, Dag Sverre Seljebotn
 wrote:
> Lisandro wrote:
>> BTW, typedfieldbug_T303.py is also affected by this
>> sizeof(double)==sizeof(long double) in MSVC.
>
> Yes I figured :-)
>
> It's just a badly written testcase I suppose. I think the testcases can
> just check for sizeof(double) == sizeof(long double) directly though
> rather than sys.platform;

I agree.

>
> IIRC this applies on gcc on PPC as well.
>

Thanks for the info...

>
> I'm really starting to crave an account on snakebite here :-)
>

Is snakebite actually ready for usage?



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From sccolbert at gmail.com  Thu May 28 07:35:07 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Thu, 28 May 2009 01:35:07 -0400
Subject: [Cython] having a weird issue with wrapped function failing in
	only certain cases
In-Reply-To: 
References: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>
	
	<7f014ea60905271244v39747565wd70872778b22d16c@mail.gmail.com>
	
	<7f014ea60905271256u6b49d408ta3babe4b8f50809a@mail.gmail.com>
	<4A1DAB66.10901@student.matnat.uio.no>
	<7f014ea60905271439j2827e1b5l33ca3fdea2528e12@mail.gmail.com>
	
	<7f014ea60905272200q5afd9af4kaa2399158d2c4f63@mail.gmail.com>
	
Message-ID: <7f014ea60905272235i3859532ah4a129908b03493c2@mail.gmail.com>

1. since i've built the same function as a python extension without using
Cython (manually writing a C extension), but still built it using distutils,
I would think that would rule out a compiler issue.
2. I don't know. But again, based on the results in 1) it shouldnt be the
issue

3. I'll try that next if we can't figure this out.

Cheers,

Chris

On Thu, May 28, 2009 at 1:29 AM, Lisandro Dalcin  wrote:

> On Thu, May 28, 2009 at 2:00 AM, Chris Colbert 
> wrote:
> >
> > So, I think I've ruled out a compiler issue. Any pointers on where to
> look
> > now?
> >
>
> I told you...
>
> 1) When building core OpenVC, any chance you or the buildsystem is
> passing some special flag to MinGW?
>
> 2) Do you know if it is possible  to ask MinGW to use a specific C
> runtime lib (the same used for Win Py2.5)?
>
> 3) Could you give a try to Python 2.6 the and matching MSVC, and use
> that MSVC to build core OpenCV and your Cython extension?
>
>
>
> --
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090528/e2ed3c57/attachment.htm 

From ilmarw at simula.no  Thu May 28 10:06:35 2009
From: ilmarw at simula.no (Ilmar Wilbers)
Date: Thu, 28 May 2009 10:06:35 +0200
Subject: [Cython] Users list revisited
In-Reply-To: <9d5d8cd541112bde2dede154eaf65e82.squirrel@webmail.uio.no>
References: <9d5d8cd541112bde2dede154eaf65e82.squirrel@webmail.uio.no>
Message-ID: <4A1E460B.5080002@simula.no>

Dag Sverre Seljebotn wrote:
> So, +1 from me on a new cython-users list.
>   
+1 here, too.

ilmar

From tenninosko at msn.com  Thu May 28 14:30:04 2009
From: tenninosko at msn.com (OUARDA MEHDI)
Date: Thu, 28 May 2009 14:30:04 +0200
Subject: [Cython] python code optimization...misc
Message-ID: 


hi,
i'm using a lot numpy functions like dot, hstack, vstack, eye... to cumpute matrix in my python programs.i was thinking that numpy uses python iterator and objects so in order to go faster,i wonder if it is worthy to rewrite( in cython )functions like that.
thanks



_________________________________________________________________
T?l?phonez gratuitement ? tous vos proches avec Windows Live Messenger? !? T?l?chargez-le maintenant !?
http://www.windowslive.fr/messenger/1.asp
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090528/c1bf8813/attachment.htm 

From dagss at student.matnat.uio.no  Thu May 28 15:18:31 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 28 May 2009 15:18:31 +0200
Subject: [Cython] python code optimization...misc
In-Reply-To: 
References: 
Message-ID: <4A1E8F27.80109@student.matnat.uio.no>

OUARDA MEHDI wrote:
> hi,
> i'm using a lot numpy functions like dot, hstack, vstack, eye... to 
> cumpute matrix in my python programs.i was thinking that numpy uses 
> python iterator and objects so in order to go faster,i wonder if it is 
> worthy to rewrite( in cython )functions like that.

It is probably better to ask this on the NumPy list.

Dag Sverre

From sccolbert at gmail.com  Thu May 28 17:01:11 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Thu, 28 May 2009 11:01:11 -0400
Subject: [Cython] python code optimization...misc
In-Reply-To: <4A1E8F27.80109@student.matnat.uio.no>
References: 
	<4A1E8F27.80109@student.matnat.uio.no>
Message-ID: <7f014ea60905280801o59287a3egb621d5df7d4d1001@mail.gmail.com>

Many of the numpy functions (dot in particular) already execute as compiled
C extensions. Functions that execute mostly in Python (histogramdd for
example) can see a speedup by using Cython. That said, you would be better
off building numpy from source against lapack and atlas, this will give you
the most bang for you buck. the instructions on how to do that are on the
scipy website.

Chris

On Thu, May 28, 2009 at 9:18 AM, Dag Sverre Seljebotn <
dagss at student.matnat.uio.no> wrote:

> OUARDA MEHDI wrote:
> > hi,
> > i'm using a lot numpy functions like dot, hstack, vstack, eye... to
> > cumpute matrix in my python programs.i was thinking that numpy uses
> > python iterator and objects so in order to go faster,i wonder if it is
> > worthy to rewrite( in cython )functions like that.
>
> It is probably better to ask this on the NumPy list.
>
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090528/ece116eb/attachment.htm 

From sccolbert at gmail.com  Thu May 28 18:16:05 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Thu, 28 May 2009 12:16:05 -0400
Subject: [Cython] having a weird issue with wrapped function failing in
	only certain cases
In-Reply-To: <7f014ea60905272235i3859532ah4a129908b03493c2@mail.gmail.com>
References: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>
	<7f014ea60905271244v39747565wd70872778b22d16c@mail.gmail.com>
	
	<7f014ea60905271256u6b49d408ta3babe4b8f50809a@mail.gmail.com>
	<4A1DAB66.10901@student.matnat.uio.no>
	<7f014ea60905271439j2827e1b5l33ca3fdea2528e12@mail.gmail.com>
	
	<7f014ea60905272200q5afd9af4kaa2399158d2c4f63@mail.gmail.com>
	
	<7f014ea60905272235i3859532ah4a129908b03493c2@mail.gmail.com>
Message-ID: <7f014ea60905280916s1544ebfm432170ee46507490@mail.gmail.com>

Alright, I "fixed" the problem by hacking the Cython generated C file.

I removed everything in the IF clause that determined whether the Python
Arguments were keyword argument or not and left only the PyTuple_GET_ITEM
statements.

I then changed the PyMethodDef line like so:

old entry: {__Pyx_NAMESTR("cvResize"),
(PyCFunction)__pyx_pf_5cy_cv_cvResize, METH_VARARGS|METH_KEYWORDS,
__Pyx_DOCSTR(0)},
new entry: {__Pyx_NAMESTR("cvResize"),
(PyCFunction)__pyx_pf_5cy_cv_cvResize, METH_VARARGS, __Pyx_DOCSTR(0)},


Removing METH_KEYWORDS was essential. Even though the extension compiled
when it was present, the program still crashed.

Without it, it works fine for all integers 0,1,2,3.

I'll test some other compilers at a later date and see what happens.

Cheers!

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090528/f1616020/attachment.htm 

From jtravs at gmail.com  Thu May 28 19:44:43 2009
From: jtravs at gmail.com (John Travers)
Date: Thu, 28 May 2009 18:44:43 +0100
Subject: [Cython] complex numpy buffer support
Message-ID: <3a1077e70905281044q22810706ub33d00d526c29555@mail.gmail.com>

Hi,

First, thanks to everyone for making cython great. I find it very useful!

Second, I notice that recently complex number support has been added
to cython. I can't find any info anywhere about whether this means
complex numpy buffers are supported. Sorry if this is detailed
anaywhere, but has this been done?

Thanks,
John

From dagss at student.matnat.uio.no  Thu May 28 22:27:31 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 28 May 2009 22:27:31 +0200 (CEST)
Subject: [Cython] complex numpy buffer support
In-Reply-To: <3a1077e70905281044q22810706ub33d00d526c29555@mail.gmail.com>
References: <3a1077e70905281044q22810706ub33d00d526c29555@mail.gmail.com>
Message-ID: 

John wrote:
> Hi,
>
> First, thanks to everyone for making cython great. I find it very useful!
>
> Second, I notice that recently complex number support has been added
> to cython. I can't find any info anywhere about whether this means
> complex numpy buffers are supported. Sorry if this is detailed
> anaywhere, but has this been done?

Yes, there was only a small hint in the release notes about it. You can do

cdef np.ndarray[double complex] arr = ...

Dag Sverre


From robertwb at math.washington.edu  Fri May 29 00:29:40 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 28 May 2009 15:29:40 -0700
Subject: [Cython] Running Cython 0.11.2 tests using MinGW32
In-Reply-To: 
References: <4A1DB976.4060309@gmail.com> <4A1DC278.9080704@gmail.com>
	
	
Message-ID: <017A11EC-0E08-4E92-83AF-2275EC9C8CB9@math.washington.edu>

On May 27, 2009, at 4:37 PM, Lisandro Dalcin wrote:

> BTW, we could add a '--mingw' option to runtests.py, right?

Is there any reason this couldn't be auto-detected?

>
> On Wed, May 27, 2009 at 8:36 PM, Lisandro Dalcin  
>  wrote:
>> On Wed, May 27, 2009 at 7:45 PM, David Martin  
>>  wrote:
>>> Is it possible and/or supported to run the Cython tests using  
>>> MinGW32 on
>>> Windows? There isn't an option for it in runtests.py, and I do  
>>> not have
>>> a set of Visual Studio compilers available to compile with. Thanks.
>>>
>>
>> Yes, I do this at home. Just a couple of lines need to be changed in
>> runtests.py. Wait me a couple of hours and I'll post the diff (of
>> contact you at Gmail chat if you are online).
>>
>> However, be prepared for some failures (mostly because the linker
>> complains about some "cdef extern" definitions). Perhaps you could
>> help me to fix all that crap?
>>
>>
>>
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>>
>
>
>
> -- 
> Lisandro Dalc?n
> ---------------
> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
> Tel/Fax: +54-(0)342-451.1594
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From robertwb at math.washington.edu  Fri May 29 00:29:04 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 28 May 2009 15:29:04 -0700
Subject: [Cython] #303 + simplifying type system
In-Reply-To: 
References: <4A099E0C.9000909@student.matnat.uio.no>
	
	<4A0C30F5.6090101@student.matnat.uio.no>
	
	<4A0C6C15.3090400@student.matnat.uio.no>
	
	<4A0C7985.3090203@student.matnat.uio.no>
	
	<4A1C593D.5050807@student.matnat.uio.no>
	
Message-ID: <6440B5A8-2809-4215-BE1A-D96F07310119@math.washington.edu>

On May 26, 2009, at 3:44 PM, Lisandro Dalcin wrote:

> On Tue, May 26, 2009 at 6:03 PM, Dag Sverre Seljebotn
>  wrote:
>> Lisandro Dalcin wrote:
>>> Wow... I'm very sorry. I got it totally wrong ...
>>>
>>> If the change you are proposing about simplifying the type system  
>>> only
>>> affect that, I'm actually +1 on your proposal. The canonical Python
>>> way is a mess in the context of Cython.
>>>
>>> I would just ask for the following:
>>>
>>> 1) When going from C to Python, for Python 2.X, try to avoid  
>>> returning
>>> a PyLong (basically dispach proper calls using
>>> sizeof(type)<=sizeof(long) )
>>>
>>> 2) Use __Pyx_T_FLOATING, just to avoid confussion.
>>>
>>>
>>> Of course, other way would be to just use the transform I proposed
>>> previously. I do not expect it to impact performance.
>>
>> To follow up here: Since Robert was concerned and I'm too lazy to
>> benchmark, I redid it. (This also fixes the recently reported bug of
>> course.)
>>
>
> But you are really, really lazy ;-) ... Try this:
>
> $ python -m timeit -n 10000000 -s 'import sys; f=sys.stdout'  
> 'f.softspace'
> 10000000 loops, best of 3: 0.218 usec per loop
>
> $ python -m timeit -n 10000000 -s 'import sys; f=sys.stdout'  
> 'f.closed'
> 10000000 loops, best of 3: 0.194 usec per loop
>
>
> So very little difference... Dict lookups likely dominate:
>
> $ python -m timeit -n 10000000 -s 'd=file.__dict__' 'd["closed"]'
> 10000000 loops, best of 3: 0.177 usec per loop
> $ python -m timeit -n 10000000 -s 'd=file.__dict__' 'd["softspace"]'
> 10000000 loops, best of 3: 0.168 usec per loop
>
>
> Even if we ignore the overhead of dict lookups, a PyMemberDef-based
> attribute (the outcome of 'cdef public something') should have almost
> identical overhead to a PyGetSetDef-based attribute (the outcome of
> 'cdef property'). Moreover, the former (PyMemberDef) requires a
> dispatching switch, then being perhaps a bit less efficient? The
> numbers above seems to support me, though I should do a better test
> (softspace is a 'int' and 'closed' a 'bool').

Thanks. If someone really cares about speed, I guess they shouldn't  
be accessing this through the Python interface anywayw...

>> http://hg.cython.org/cython/rev/11e826f95cde
>>
>
> Looks good. Though perhaps in the future we could review all this and
> get rid at all of PyMemberDef-based attributes... IMHO, that is just
> sugar for human C extension writers.

It also reduces the .c and shared object library sizes, though that's  
a pretty weak argument for this particular case.

- Robert


From robertwb at math.washington.edu  Fri May 29 00:37:18 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 28 May 2009 15:37:18 -0700
Subject: [Cython] Users list revisited
In-Reply-To: <7f014ea60905271300y3bea0d03tdf010085b0cb50c0@mail.gmail.com>
References: <9d5d8cd541112bde2dede154eaf65e82.squirrel@webmail.uio.no>
	<7f014ea60905271300y3bea0d03tdf010085b0cb50c0@mail.gmail.com>
Message-ID: <33DA9D7A-EB99-4248-AB6F-5BECEC637AB4@math.washington.edu>

On May 27, 2009, at 1:00 PM, Chris Colbert wrote:

> I agree, I think a Cython-users is a great idea.
>
> You all have a been a great help to me in learning Cython, but I  
> feel horrible having to ask such newbie questions all the time, and  
> fear I will one day wear out my welcome.

No worries, you won't be a newbie forever, and then you can pay it  
back by answering other people's questions :).

> So +1 for a Cython-users list.

+1 from me too, for all the reasons listed below. Stefan, do you want  
to set up one at codespeak.net for consistency?

- Robert


>
> On Wed, May 27, 2009 at 3:54 PM, Dag Sverre Seljebotn  
>  wrote:
> Something occurred to me:
>
> When we discussed whether there should be a cython-users list, we were
> arguing from how the situation is today, and that the current  
> active users
> don't think it is needed.
>
> However, one should probably take into account future scenarios:
>
> a) It might be that cython-users would attract more posters, who don't
> think about asking their questions on cython-dev in the first place
> (dev-lists in general being a RTFM kind of place, even though this one
> isn't)
>
> b) It might be that cython-users would in time make it easier for  
> non-devs
> to answer questions, creating a users community in the same
>
> As for me personally, I totally don't mind the users questions in  
> here and
> would subscribe to both lists (as I'm sure the other devs would as  
> well),
> but it is now getting to a point where having the different things  
> sorted
> into two folders in my mail program wouldn't hurt either.
>
> So, +1 from me on a new cython-users list.
>
> Dag Sverre
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev


From robertwb at math.washington.edu  Fri May 29 00:40:29 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 28 May 2009 15:40:29 -0700
Subject: [Cython] having a weird issue with wrapped function failing in
	only certain cases
In-Reply-To: <7f014ea60905280916s1544ebfm432170ee46507490@mail.gmail.com>
References: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>
	<7f014ea60905271244v39747565wd70872778b22d16c@mail.gmail.com>
	
	<7f014ea60905271256u6b49d408ta3babe4b8f50809a@mail.gmail.com>
	<4A1DAB66.10901@student.matnat.uio.no>
	<7f014ea60905271439j2827e1b5l33ca3fdea2528e12@mail.gmail.com>
	
	<7f014ea60905272200q5afd9af4kaa2399158d2c4f63@mail.gmail.com>
	
	<7f014ea60905272235i3859532ah4a129908b03493c2@mail.gmail.com>
	<7f014ea60905280916s1544ebfm432170ee46507490@mail.gmail.com>
Message-ID: <06F6E166-6D3A-4080-BEF9-6FC54F780F05@math.washington.edu>

On May 28, 2009, at 9:16 AM, Chris Colbert wrote:

> Alright, I "fixed" the problem by hacking the Cython generated C file.
>
> I removed everything in the IF clause that determined whether the  
> Python Arguments were keyword argument or not and left only the  
> PyTuple_GET_ITEM statements.
>
> I then changed the PyMethodDef line like so:
>
> old entry: {__Pyx_NAMESTR("cvResize"), (PyCFunction) 
> __pyx_pf_5cy_cv_cvResize, METH_VARARGS|METH_KEYWORDS, __Pyx_DOCSTR 
> (0)},
> new entry: {__Pyx_NAMESTR("cvResize"), (PyCFunction) 
> __pyx_pf_5cy_cv_cvResize, METH_VARARGS, __Pyx_DOCSTR(0)},
>
>
> Removing METH_KEYWORDS was essential. Even though the extension  
> compiled when it was present, the program still crashed.
>
> Without it, it works fine for all integers 0,1,2,3.

Very strange. So to make sure I understand correctly, it was crashing  
with METH_KEYWORDS even if you weren't using keyword arguments?

- Robert


From dalcinl at gmail.com  Fri May 29 00:50:59 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 28 May 2009 19:50:59 -0300
Subject: [Cython] Running Cython 0.11.2 tests using MinGW32
In-Reply-To: <017A11EC-0E08-4E92-83AF-2275EC9C8CB9@math.washington.edu>
References: <4A1DB976.4060309@gmail.com> <4A1DC278.9080704@gmail.com>
	
	
	<017A11EC-0E08-4E92-83AF-2275EC9C8CB9@math.washington.edu>
Message-ID: 

On Thu, May 28, 2009 at 7:29 PM, Robert Bradshaw
 wrote:
> On May 27, 2009, at 4:37 PM, Lisandro Dalcin wrote:
>
>> BTW, we could add a '--mingw' option to runtests.py, right?
>
> Is there any reason this couldn't be auto-detected?
>

Well, perhaps we could autodetect this, though looking how much
contrived is core distutils in this part, not sure.

Anyway, suppose you have installed both MSVC and MinGW, an as you are
a serious Cython developer, you want to run the testsuite with both.
The easiest way to select GCC would be to pass '--mingw' to
runtests.py, right?


>>
>> On Wed, May 27, 2009 at 8:36 PM, Lisandro Dalcin
>>  wrote:
>>> On Wed, May 27, 2009 at 7:45 PM, David Martin
>>>  wrote:
>>>> Is it possible and/or supported to run the Cython tests using
>>>> MinGW32 on
>>>> Windows? There isn't an option for it in runtests.py, and I do
>>>> not have
>>>> a set of Visual Studio compilers available to compile with. Thanks.
>>>>
>>>
>>> Yes, I do this at home. Just a couple of lines need to be changed in
>>> runtests.py. Wait me a couple of hours and I'll post the diff (of
>>> contact you at Gmail chat if you are online).
>>>
>>> However, be prepared for some failures (mostly because the linker
>>> complains about some "cdef extern" definitions). Perhaps you could
>>> help me to fix all that crap?
>>>
>>>
>>>
>>>
>>> --
>>> Lisandro Dalc?n
>>> ---------------
>>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>>> Tel/Fax: +54-(0)342-451.1594
>>>
>>
>>
>>
>> --
>> Lisandro Dalc?n
>> ---------------
>> Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
>> Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
>> Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
>> PTLC - G?emes 3450, (3000) Santa Fe, Argentina
>> Tel/Fax: +54-(0)342-451.1594
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From sccolbert at gmail.com  Fri May 29 04:46:13 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Thu, 28 May 2009 22:46:13 -0400
Subject: [Cython] having a weird issue with wrapped function failing in
	only certain cases
In-Reply-To: <06F6E166-6D3A-4080-BEF9-6FC54F780F05@math.washington.edu>
References: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>
	<7f014ea60905271256u6b49d408ta3babe4b8f50809a@mail.gmail.com>
	<4A1DAB66.10901@student.matnat.uio.no>
	<7f014ea60905271439j2827e1b5l33ca3fdea2528e12@mail.gmail.com>
	
	<7f014ea60905272200q5afd9af4kaa2399158d2c4f63@mail.gmail.com>
	
	<7f014ea60905272235i3859532ah4a129908b03493c2@mail.gmail.com>
	<7f014ea60905280916s1544ebfm432170ee46507490@mail.gmail.com>
	<06F6E166-6D3A-4080-BEF9-6FC54F780F05@math.washington.edu>
Message-ID: <7f014ea60905281946h6f5f3823r5f2cc1e5bd001915@mail.gmail.com>

Robert,
That's correct. Of course I had to remove all the checks related to Keyword
args in the body of the function before it would execute. So essentially,
cleaning the code so that it only accepts positional arguments fixes the
crashing problem.

Chris

On Thu, May 28, 2009 at 6:40 PM, Robert Bradshaw <
robertwb at math.washington.edu> wrote:

> On May 28, 2009, at 9:16 AM, Chris Colbert wrote:
>
> > Alright, I "fixed" the problem by hacking the Cython generated C file.
> >
> > I removed everything in the IF clause that determined whether the
> > Python Arguments were keyword argument or not and left only the
> > PyTuple_GET_ITEM statements.
> >
> > I then changed the PyMethodDef line like so:
> >
> > old entry: {__Pyx_NAMESTR("cvResize"), (PyCFunction)
> > __pyx_pf_5cy_cv_cvResize, METH_VARARGS|METH_KEYWORDS, __Pyx_DOCSTR
> > (0)},
> > new entry: {__Pyx_NAMESTR("cvResize"), (PyCFunction)
> > __pyx_pf_5cy_cv_cvResize, METH_VARARGS, __Pyx_DOCSTR(0)},
> >
> >
> > Removing METH_KEYWORDS was essential. Even though the extension
> > compiled when it was present, the program still crashed.
> >
> > Without it, it works fine for all integers 0,1,2,3.
>
> Very strange. So to make sure I understand correctly, it was crashing
> with METH_KEYWORDS even if you weren't using keyword arguments?
>
> - Robert
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090528/0461268f/attachment.htm 

From sccolbert at gmail.com  Fri May 29 04:50:43 2009
From: sccolbert at gmail.com (Chris Colbert)
Date: Thu, 28 May 2009 22:50:43 -0400
Subject: [Cython] having a weird issue with wrapped function failing in
	only certain cases
In-Reply-To: <7f014ea60905281946h6f5f3823r5f2cc1e5bd001915@mail.gmail.com>
References: <7f014ea60905262057n225fab45g11f0ca6ce0fbb1b4@mail.gmail.com>
	<4A1DAB66.10901@student.matnat.uio.no>
	<7f014ea60905271439j2827e1b5l33ca3fdea2528e12@mail.gmail.com>
	
	<7f014ea60905272200q5afd9af4kaa2399158d2c4f63@mail.gmail.com>
	
	<7f014ea60905272235i3859532ah4a129908b03493c2@mail.gmail.com>
	<7f014ea60905280916s1544ebfm432170ee46507490@mail.gmail.com>
	<06F6E166-6D3A-4080-BEF9-6FC54F780F05@math.washington.edu>
	<7f014ea60905281946h6f5f3823r5f2cc1e5bd001915@mail.gmail.com>
Message-ID: <7f014ea60905281950p5547e67ajf5abdc80d8e01869@mail.gmail.com>

Just to clarify, I've never used keyword arguments with this function.
Chris

On Thu, May 28, 2009 at 10:46 PM, Chris Colbert  wrote:

> Robert,
> That's correct. Of course I had to remove all the checks related to Keyword
> args in the body of the function before it would execute. So essentially,
> cleaning the code so that it only accepts positional arguments fixes the
> crashing problem.
>
> Chris
>
>
> On Thu, May 28, 2009 at 6:40 PM, Robert Bradshaw <
> robertwb at math.washington.edu> wrote:
>
>> On May 28, 2009, at 9:16 AM, Chris Colbert wrote:
>>
>> > Alright, I "fixed" the problem by hacking the Cython generated C file.
>> >
>> > I removed everything in the IF clause that determined whether the
>> > Python Arguments were keyword argument or not and left only the
>> > PyTuple_GET_ITEM statements.
>> >
>> > I then changed the PyMethodDef line like so:
>> >
>> > old entry: {__Pyx_NAMESTR("cvResize"), (PyCFunction)
>> > __pyx_pf_5cy_cv_cvResize, METH_VARARGS|METH_KEYWORDS, __Pyx_DOCSTR
>> > (0)},
>> > new entry: {__Pyx_NAMESTR("cvResize"), (PyCFunction)
>> > __pyx_pf_5cy_cv_cvResize, METH_VARARGS, __Pyx_DOCSTR(0)},
>> >
>> >
>> > Removing METH_KEYWORDS was essential. Even though the extension
>> > compiled when it was present, the program still crashed.
>> >
>> > Without it, it works fine for all integers 0,1,2,3.
>>
>> Very strange. So to make sure I understand correctly, it was crashing
>> with METH_KEYWORDS even if you weren't using keyword arguments?
>>
>> - Robert
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090528/ce13278b/attachment.htm 

From pnorthug at gmail.com  Fri May 29 19:44:41 2009
From: pnorthug at gmail.com (Paul Northug)
Date: Sat, 30 May 2009 01:44:41 +0800
Subject: [Cython] PyArray_SimpleNewFromData bus error
Message-ID: 

I am trying to a wrap a C double array as a numpy array to pass to a
python function. I am using PyArray_SimpleNewFromData but I am getting
a bus error.

A contrived example (at bottom) reproduces the error. What am I doing
wrong? I tried to follow the example given in an earlier post by Dag
and Marcin.

import numpy as np
from numpy cimport ndarray, NPY_DOUBLE, npy_intp

cdef extern from "numpy/arrayobject.h":
     cdef object PyArray_SimpleNewFromData(int nd, npy_intp *dims,
                                           int typenum, void *data)

def test():
    cdef ndarray[double, ndim=1, mode='c'] a =
np.random.rand(10).astype(np.float64)
    cdef npy_intp dims = 10
    # below produces bus error
    cdef ndarray b = PyArray_SimpleNewFromData(1, &dims, NPY_DOUBLE,
a.data)

    return a == b

From dagss at student.matnat.uio.no  Sat May 30 00:10:44 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 30 May 2009 00:10:44 +0200
Subject: [Cython] PyArray_SimpleNewFromData bus error
In-Reply-To: 
References: 
Message-ID: <4A205D64.5080509@student.matnat.uio.no>

Paul Northug wrote:
> I am trying to a wrap a C double array as a numpy array to pass to a
> python function. I am using PyArray_SimpleNewFromData but I am getting
> a bus error.

I can't see why at a glance; you should probably ask this on the NumPy 
mailing list.

Dag Sverre

From robert.kern at gmail.com  Sat May 30 00:29:38 2009
From: robert.kern at gmail.com (Robert Kern)
Date: Fri, 29 May 2009 17:29:38 -0500
Subject: [Cython] PyArray_SimpleNewFromData bus error
In-Reply-To: 
References: 
Message-ID: 

On 2009-05-29 12:44, Paul Northug wrote:
> I am trying to a wrap a C double array as a numpy array to pass to a
> python function. I am using PyArray_SimpleNewFromData but I am getting
> a bus error.
>
> A contrived example (at bottom) reproduces the error. What am I doing
> wrong? I tried to follow the example given in an earlier post by Dag
> and Marcin.

You must call import_array() before using any PyArray_* functions.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From stefan_ml at behnel.de  Sat May 30 09:23:05 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 30 May 2009 09:23:05 +0200
Subject: [Cython] -unstable mergeback?
In-Reply-To: 
References: <4A14291F.2090806@student.matnat.uio.no>
	
Message-ID: <4A20DED9.5010506@behnel.de>

Hi,

sorry, was away for a while.

Robert Bradshaw schrieb:
> On May 20, 2009, at 9:00 AM, Dag Sverre Seljebotn wrote:
> 
>> It seems that there's now a danger of -unstable becoming the main
>> development branch; I also think the dangerous things are completed
>> myself. If so, I think it should replace the current -devel. This will
>> make a few tickets blockers for the next release though.
> 
> If -unstable has stabilized, I think this makes sense. I envisioned - 
> unstable as being for truly unstable things (e.g. closures, temp  
> allocation rewrites) rather than the main development branch.

Same from my side. There are a couple of regressions (failing tests) in the
current -unstable that I couldn't easily fix right away. They need some
more work before merging it back into -devel. BTW, I'd actually go the
opposite way once that's done: merge -devel up into -unstable, drop -devel
and rename -unstable.

Regarding the -closures branch, IMHO it now works well enough to merge it
into cython-unstable. I'd still be happy if others could give it a try
first, maybe just run it on some Python code that uses nested functions.

Stefan

From stefan_ml at behnel.de  Sat May 30 09:44:12 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 30 May 2009 09:44:12 +0200
Subject: [Cython] Users list revisited
In-Reply-To: <33DA9D7A-EB99-4248-AB6F-5BECEC637AB4@math.washington.edu>
References: <9d5d8cd541112bde2dede154eaf65e82.squirrel@webmail.uio.no>	<7f014ea60905271300y3bea0d03tdf010085b0cb50c0@mail.gmail.com>
	<33DA9D7A-EB99-4248-AB6F-5BECEC637AB4@math.washington.edu>
Message-ID: <4A20E3CC.2040508@behnel.de>


Robert Bradshaw wrote:
> On May 27, 2009, at 1:00 PM, Chris Colbert wrote:
> 
>> I agree, I think a Cython-users is a great idea.
>>
>> You all have a been a great help to me in learning Cython, but I  
>> feel horrible having to ask such newbie questions all the time, and  
>> fear I will one day wear out my welcome.
> 
> No worries, you won't be a newbie forever, and then you can pay it  
> back by answering other people's questions :).
> 
>> So +1 for a Cython-users list.
> 
> +1 from me too, for all the reasons listed below. Stefan, do you want  
> to set up one at codespeak.net for consistency?

Asking there right now.

Stefan


From stefan_ml at behnel.de  Sat May 30 15:00:50 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 30 May 2009 15:00:50 +0200
Subject: [Cython] Cython 0.11.2 test errors
In-Reply-To: <4A1DCFFF.1030906@gmail.com>
References: <4A1DCFFF.1030906@gmail.com>
Message-ID: <4A212E02.8040503@behnel.de>


David Martin wrote:
> I am getting quite a few errors when running the Cython tests through 
> runtests.py. This is on a machine with VS .NET 2003 installed, using the 
> command 'python runtests.py --no-refnanny". I get the same set of errors 
> when I run the tests using the mingw compiler.
> [...]
> callargs.c
> callargs.c(398) : error C2026: string too big, trailing characters truncated
> callargs.cpp
> callargs.cpp(398) : error C2026: string too big, trailing characters 
> truncated
> [...]

This is a known limitation of older MS compilers which fail to compile
strings of 2048 bytes or longer (really bytes, not GB or something, just a
tiny 2048 bytes). Usually not too hard to work around for 'normal' string
literals which can be concatenated, but certainly a big problem for doc
strings (which is the case here, Cython's test suite consists of doctests).

I looked into this a while ago, but it would require changing the way
docstrings are set up. Currently, we just write the string literals into a
func/type/etc. struct member somewhere. To fix this, we'd need to build the
literal dynamically during module initialisation, fix the static struct
member to point to it (or rather allocate the struct on the heap and do it
there), and free() the string during module cleanup. Seemed like too much
hassle at the time, especially since the problem has now gone away with
Python 2.6, which requires a newer MS compiler IIRC. MinGW (obviously) does
not suffer from this problem either.

Stefan

From dagss at student.matnat.uio.no  Sat May 30 15:26:44 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sat, 30 May 2009 15:26:44 +0200
Subject: [Cython] Cython 0.11.2 test errors
In-Reply-To: <4A212E02.8040503@behnel.de>
References: <4A1DCFFF.1030906@gmail.com> <4A212E02.8040503@behnel.de>
Message-ID: <4A213414.8010200@student.matnat.uio.no>

Stefan Behnel wrote:
> David Martin wrote:
>> I am getting quite a few errors when running the Cython tests through 
>> runtests.py. This is on a machine with VS .NET 2003 installed, using the 
>> command 'python runtests.py --no-refnanny". I get the same set of errors 
>> when I run the tests using the mingw compiler.
>> [...]
>> callargs.c
>> callargs.c(398) : error C2026: string too big, trailing characters truncated
>> callargs.cpp
>> callargs.cpp(398) : error C2026: string too big, trailing characters 
>> truncated
>> [...]
> 
> This is a known limitation of older MS compilers which fail to compile
> strings of 2048 bytes or longer (really bytes, not GB or something, just a
> tiny 2048 bytes). Usually not too hard to work around for 'normal' string
> literals which can be concatenated, but certainly a big problem for doc
> strings (which is the case here, Cython's test suite consists of doctests).
> 
> I looked into this a while ago, but it would require changing the way
> docstrings are set up. Currently, we just write the string literals into a
> func/type/etc. struct member somewhere. To fix this, we'd need to build the
> literal dynamically during module initialisation, fix the static struct
> member to point to it (or rather allocate the struct on the heap and do it
> there), and free() the string during module cleanup. Seemed like too much
> hassle at the time, especially since the problem has now gone away with
> Python 2.6, which requires a newer MS compiler IIRC. MinGW (obviously) does
> not suffer from this problem either.

I'm all in favor of officially dropping support for older VS compilers 
which newer Python versions do not support. MinGW is always a 
possibility anyway.

It probably should be documented clearly somewhere which compilers we 
aim to support though...

-- 
Dag Sverre

From stefan_ml at behnel.de  Sat May 30 15:52:53 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 30 May 2009 15:52:53 +0200
Subject: [Cython] Cython 0.11.2 test errors
In-Reply-To: <4A213414.8010200@student.matnat.uio.no>
References: <4A1DCFFF.1030906@gmail.com> <4A212E02.8040503@behnel.de>
	<4A213414.8010200@student.matnat.uio.no>
Message-ID: <4A213A35.6000704@behnel.de>


Dag Sverre Seljebotn wrote:
> I'm all in favor of officially dropping support for older VS compilers 
> which newer Python versions do not support.

It's not as easy as that, though. Older Python versions require a specific
MSVC option for building extension modules, so using a newer MS compiler is
not an option here. While MinGW *is* an option, I assume that MSVC is a lot
easier to set up on Windows, so it's not a simple click-and-forget option
but more of a try-to-make-it-work-if-you-really-want-to-use-it option,
especially when also compiling against other external libraries which users
may or may not build from sources.

Stefan

From robertwb at math.washington.edu  Sat May 30 17:01:42 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 30 May 2009 08:01:42 -0700
Subject: [Cython] -unstable mergeback?
In-Reply-To: <4A20DED9.5010506@behnel.de>
References: <4A14291F.2090806@student.matnat.uio.no>
	
	<4A20DED9.5010506@behnel.de>
Message-ID: <204AD889-4406-495C-9BD0-3A262B1D207D@math.washington.edu>

On May 30, 2009, at 12:23 AM, Stefan Behnel wrote:

> Hi,
>
> sorry, was away for a while.
>
> Robert Bradshaw schrieb:
>> On May 20, 2009, at 9:00 AM, Dag Sverre Seljebotn wrote:
>>
>>> It seems that there's now a danger of -unstable becoming the main
>>> development branch; I also think the dangerous things are completed
>>> myself. If so, I think it should replace the current -devel. This  
>>> will
>>> make a few tickets blockers for the next release though.
>>
>> If -unstable has stabilized, I think this makes sense. I envisioned -
>> unstable as being for truly unstable things (e.g. closures, temp
>> allocation rewrites) rather than the main development branch.
>
> Same from my side. There are a couple of regressions (failing  
> tests) in the
> current -unstable that I couldn't easily fix right away. They need  
> some
> more work before merging it back into -devel. BTW, I'd actually go the
> opposite way once that's done: merge -devel up into -unstable, drop  
> -devel
> and rename -unstable.

I'd like to see if there's any major fallout trying to compile sage  
with -unstable before we make the switch though, but assuming things  
aren't catastrophic, let's do that.

> Regarding the -closures branch, IMHO it now works well enough to  
> merge it
> into cython-unstable. I'd still be happy if others could give it a try
> first, maybe just run it on some Python code that uses nested  
> functions.

I found an odd bug while demoing closures, so they're not ready yet,  
but I think they're close--certainly ready for other people to start  
trying them out. This is a feature that I would like people to really  
pound on before release.

- Robert


From robertwb at math.washington.edu  Sat May 30 17:04:46 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 30 May 2009 08:04:46 -0700
Subject: [Cython] Running Cython 0.11.2 tests using MinGW32
In-Reply-To: 
References: <4A1DB976.4060309@gmail.com> <4A1DC278.9080704@gmail.com>
	
	
	<017A11EC-0E08-4E92-83AF-2275EC9C8CB9@math.washington.edu>
	
Message-ID: <99BDB838-B332-427C-8E44-550FBE5336B7@math.washington.edu>

On May 28, 2009, at 3:50 PM, Lisandro Dalcin wrote:

> On Thu, May 28, 2009 at 7:29 PM, Robert Bradshaw
>  wrote:
>> On May 27, 2009, at 4:37 PM, Lisandro Dalcin wrote:
>>
>>> BTW, we could add a '--mingw' option to runtests.py, right?
>>
>> Is there any reason this couldn't be auto-detected?
>>
>
> Well, perhaps we could autodetect this, though looking how much
> contrived is core distutils in this part, not sure.
>
> Anyway, suppose you have installed both MSVC and MinGW, an as you are
> a serious Cython developer, you want to run the testsuite with both.
> The easiest way to select GCC would be to pass '--mingw' to
> runtests.py, right?


I hadn't thought of that. Are you saying that you would have a single  
Python install that works both with MSVC and MinGW?

- Robert


From robertwb at math.washington.edu  Sat May 30 17:10:55 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 30 May 2009 08:10:55 -0700
Subject: [Cython] Cython 0.11.2 test errors
In-Reply-To: 
References: <4A1DCFFF.1030906@gmail.com>
	
Message-ID: <23FDF0FE-439C-4FD0-A9FA-8C029CA71A46@math.washington.edu>

On May 27, 2009, at 9:18 PM, Dag Sverre Seljebotn wrote:

> David Martin wrote:
>> I am getting quite a few errors when running the Cython tests through
>> runtests.py. This is on a machine with VS .NET 2003 installed,  
>> using the
>> command 'python runtests.py --no-refnanny". I get the same set of  
>> errors
>> when I run the tests using the mingw compiler.
>
> There's a similar report here:
>
> http://trac.cython.org/cython_trac/ticket/106
>
> The problem is simply that there's a lack of Microsoft products  
> among some
> of the Cython developers :-)

That is exactly the case for me.

> I think we really need an additional person
> with MSVC knowledge willing to maintain this side of Cython.

+1 To make it clear, Cython is fully intended to be supported on  
Windows, I simply don't have the equipment or expertise to do much  
myself. Also, kudos to Lisandro for doing much of what has been done  
in this area.

> Another option would be trying to get a http://www.snakebite.org/  
> account,
> which should provide reliable Windows access...
>
> Do you mean you get "roughly the same errors" with mingw, or "the  
> exact
> same text" with mingw? If the latter you are likely not compiling with
> mingw anyway.

If it comes down to re-writing tests perhaps we could do that (making  
sure we're still really testing the right thing).

- Robert



From robertwb at math.washington.edu  Sat May 30 17:19:42 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 30 May 2009 08:19:42 -0700
Subject: [Cython] Tutorial at NOTUR 2009
In-Reply-To: 
References: 
	
	
Message-ID: <159E65B9-CFE8-4FF5-89D0-99EC233686E1@math.washington.edu>

On May 19, 2009, at 11:29 PM, Dag Sverre Seljebotn wrote:

>> Dag, could you check permission on files inside
>> http://sage.math.washington.edu/home/dagss/cython-notur09/mpi/ ? I'm
>> getting "Forbidden" here...
>
> Thanks. It's fixed now. Please tell me if I did anything wrong with  
> MPI
> :-) And there's an example in matmul/matmul_mpi.pyx as well.

They look nice. Anything to say about the audience's reaction?

- Robert


From stefan_ml at behnel.de  Sun May 31 08:38:46 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 31 May 2009 08:38:46 +0200
Subject: [Cython] fundamentals question
In-Reply-To: <7f014ea60905191422u629321f1ycf0246e7003e26b4@mail.gmail.com>
References: <7f014ea60905190829w12fc8af2la0e646f39115c8e6@mail.gmail.com>		<7f014ea60905190854x2ef75b22s12e5cb0d0bebf9e9@mail.gmail.com>		<7f014ea60905190920p127b5589j5cd3380c443b4cbc@mail.gmail.com>	<4A12EB00.2070305@behnel.de>	<7f014ea60905191345q45bf9392l20c12e8545d92c37@mail.gmail.com>	<886D2376-8FEE-4CC0-91CD-2DBC1FACBF23@math.washington.edu>	<7f014ea60905191402k2211431bs7dbca1b10dca59e@mail.gmail.com>	<4646288C-4C25-489B-A266-9E3C6AB95639@math.washington.edu>
	<7f014ea60905191422u629321f1ycf0246e7003e26b4@mail.gmail.com>
Message-ID: <4A2225F6.9080603@behnel.de>


Chris Colbert wrote:
> On Tue, May 19, 2009 at 5:17 PM, Robert Bradshaw wrote:
>>>> def testload(filename):
>>>>     cdef IplImage* img = load(filename)
>>>>     cdef PyIplImage pyimg = PyIplImage
>>>>     pyimg.img = img
>>>>     return pyimg
>
> it seems a little "off" to programatically set every attribute one
> by one every time i create an instance of that type.

Just write a factory function to do it for you. That's actually a common
(and recommended) pattern.

> Is it not recommended to declare a property for every attribute I want
> access from in python?

If all you need is plain value access, that's what they are made for.

Stefan

From stefan at sun.ac.za  Sun May 31 17:15:29 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sun, 31 May 2009 17:15:29 +0200
Subject: [Cython] cdef with ndarray parameters
Message-ID: <9457e7c80905310815s227cfbadk68e6a6893c0f9a4d@mail.gmail.com>

Hi all,

I'd like to know if it is currently possible to create a cdef method
that takes a numpy array as a parameter?

The following works in place of cpdef, but feels like a bit of a kludge:

cdef bool _my_func(MyObject s, int* arr, int max_rows, int max_cols)
...
def my_func(MyObject s, np.ndarray[np.int_t, ndim=2] arr):
    return _my_func(s, arr.data, arr.shape[0], arr.shape[1])

When trying to define

cdef bool my_func(MyObject s, np.ndarray[np.int_c, ndim=2] arr)

Cython complains that it

"Expected ']'".

Thanks for any advice,

Regards
St?fan

From dagss at student.matnat.uio.no  Sun May 31 17:40:44 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Sun, 31 May 2009 17:40:44 +0200
Subject: [Cython] cdef with ndarray parameters
In-Reply-To: <9457e7c80905310815s227cfbadk68e6a6893c0f9a4d@mail.gmail.com>
References: <9457e7c80905310815s227cfbadk68e6a6893c0f9a4d@mail.gmail.com>
Message-ID: <4A22A4FC.8080103@student.matnat.uio.no>

St?fan van der Walt wrote:
> Hi all,
> 
> I'd like to know if it is currently possible to create a cdef method
> that takes a numpy array as a parameter?
> 
> The following works in place of cpdef, but feels like a bit of a kludge:
> 
> cdef bool _my_func(MyObject s, int* arr, int max_rows, int max_cols)
> ...
> def my_func(MyObject s, np.ndarray[np.int_t, ndim=2] arr):
>     return _my_func(s, arr.data, arr.shape[0], arr.shape[1])
> 
> When trying to define
> 
> cdef bool my_func(MyObject s, np.ndarray[np.int_c, ndim=2] arr)
> 
> Cython complains that it
> 
> "Expected ']'".
> 
> Thanks for any advice,

http://trac.cython.org/cython_trac/ticket/177

Hope it will be added sometimes over summer.


-- 
Dag Sverre

From stefan at sun.ac.za  Sun May 31 17:45:23 2009
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Sun, 31 May 2009 17:45:23 +0200
Subject: [Cython] cdef with ndarray parameters
In-Reply-To: <4A22A4FC.8080103@student.matnat.uio.no>
References: <9457e7c80905310815s227cfbadk68e6a6893c0f9a4d@mail.gmail.com> 
	<4A22A4FC.8080103@student.matnat.uio.no>
Message-ID: <9457e7c80905310845y500ec9c8q4b4f1cbb45f0d7a9@mail.gmail.com>

2009/5/31 Dag Sverre Seljebotn :
> St?fan van der Walt wrote:
>> I'd like to know if it is currently possible to create a cdef method
>> that takes a numpy array as a parameter?
>>
> http://trac.cython.org/cython_trac/ticket/177
>
> Hope it will be added sometimes over summer.

That's great, thanks for the pointer!

Cheers
St?fan

From paustin at eos.ubc.ca  Sun May 31 19:12:34 2009
From: paustin at eos.ubc.ca (Philip Austin)
Date: Sun, 31 May 2009 10:12:34 -0700
Subject: [Cython] [xpyx]  Re:  Tutorial at NOTUR 2009
In-Reply-To: <159E65B9-CFE8-4FF5-89D0-99EC233686E1@math.washington.edu>
References: 
	
	
	<159E65B9-CFE8-4FF5-89D0-99EC233686E1@math.washington.edu>
Message-ID: <4A22BA82.4080201@eos.ubc.ca>


Thanks for this, very nicely presented.

Looking at the example code at
http://sage.math.washington.edu/home/dagss/cython-notur09.tar.gz

1)  pxd files (lapack.pxd at least) are missing?

2) My 64 bit linux (Centos 5.1) was unhappy with the repack
in readdat.pyx

I get

> >> import readdata
Traceback (most recent call last):
  File "", line 1, in 
  File "readdata.pyx", line 23, in readdata (readdata.c:2850)
    print linearsearch(data, 931)
  File "readdata.pyx", line 11, in readdata.linearsearch (readdata.c:915)
    def linearsearch(np.ndarray[DataLine] data, np.int32_t id):
ValueError: Buffer dtype mismatch; next field is at offset 8 but 2 expected

unless I comment out:

#data = data.astype(np.dtype(DataLine_desc, align=True))

cheers, Phil Austin