From michael.abshoff at googlemail.com Fri May 1 00:53:11 2009 From: michael.abshoff at googlemail.com (Michael Abshoff) Date: Thu, 30 Apr 2009 15:53:11 -0700 Subject: [Cython] C -- Variable-sized array on stack? In-Reply-To: <49F9FE8C.60707@student.matnat.uio.no> References: <49F9F714.70907@student.matnat.uio.no> <49F9FDAE.20301@canonware.com> <49F9FE8C.60707@student.matnat.uio.no> Message-ID: <49FA2BD7.2050503@gmail.com> Dag Sverre Seljebotn wrote: > Jason Evans wrote: >> Dag Sverre Seljebotn wrote: >>> Will I get away with code like this in Cython? >>> >>> void foo(int n) { >>> int stack[n]; >>> ... >>> } >>> >>> I.e. what C language level are we targeting exactly? Also, does the >>> above cause a proper stack allocation? >> I think this is a C99 feature that happens to be part of gnu89 as well. >> I seem to recall that MSVC++ does not support the feature, which is an >> important consideration here. And MSVC did that not in order to spite everyone, but because it opens a giant can of worms including security issues. At 23C3 I saw a demo on how this "feature" can be used to do many bad, bad things with gcc since it did not do any checking at all if the stack was large enough, etc. > Sure, I'm convinced not to use it now. Good :) Cheers, Michael From robertwb at math.washington.edu Fri May 1 07:12:06 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 30 Apr 2009 22:12:06 -0700 Subject: [Cython] Path towards Python completeness In-Reply-To: <49F9D371.3060209@behnel.de> References: <49ED5CE8.6090708@behnel.de> <80500050-599D-4F3A-9FA1-0291E43B3B99@math.washington.edu> <380c38b6623e9a1a6d12a649dee43ec4.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <2038f7c55d17516c79c8b3b2a9385fb1.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <49F76991.5010303@behnel.de> <49F9D371.3060209@behnel.de> Message-ID: <140BC717-E6BC-4188-A5B6-42AC9D529B29@math.washington.edu> On Apr 30, 2009, at 9:36 AM, Stefan Behnel wrote: > Stefan Behnel wrote: >> Stefan Behnel wrote: >>> Robert Bradshaw wrote: >>>> http://hg.cython.org/cython-closures/file/c4ee199f08a2/tests/run/ >>>> closures_T82.pyx >>>> >>>> Before you get your hopes up too much, I want to add a big >>>> disclaimer that though I tried to be careful, this is very first- >>>> draft code and there are bound to be issues (e.g. I think I've run >>>> into some with respect to Python arguments). >>> I'll give it a try during the next days. >> >> I exercised it a bit and added more test cases. That revealed a >> couple of >> problems, although I must say that it works pretty well in general. >> >> I added comments to the tests as far as I got with analysing them. >> One >> segfaults pretty badly, which makes it harder to figure out. > > Ok, that was due to ref-counting problems. I was hoping that's all it was (not that tracking down ref-counting issues is always easy...) > All of these work for me now: > > http://hg.cython.org/cython-closures/file/626e754738b6/tests/run/ > closures_T82.pyx > > I find that pretty impressive. Very good work, Robert! Thanks. > This looks like it could go right into 0.12 rather than waiting for > 0.13. Yep, I think that's very feasible too. - Robert From robertwb at math.washington.edu Fri May 1 07:19:32 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 30 Apr 2009 22:19:32 -0700 Subject: [Cython] hg repo doesn't seem to work? In-Reply-To: References: <85b5c3130904291146v1395cd1en81ec1415da1b9c58@mail.gmail.com> Message-ID: <8F3A5997-2642-4789-A80E-7C2D2D1F0E34@math.washington.edu> On Apr 29, 2009, at 12:33 PM, Robert Bradshaw wrote: > On Apr 29, 2009, at 11:46 AM, Ondrej Certik wrote: > >> Hi, >> >> I did: >> >> $ hg clone http://hg.cython.org/cython/ >> destination directory: cython >> requesting all changes >> adding changesets >> adding manifests >> adding file changes >> transaction abort! >> rollback completed >> abort: 00changelog.i at 696e20737973: no node! >> >> am I doing something wrong? > > Hmm... I'm getting that too. I can't look into this now, but will try > to do so later. You can clone from the others right now. OK, it was missing a file in the repo (?). I think something got lost when we packaged it with distuitils. Resolved. - Robert From dagss at student.matnat.uio.no Fri May 1 14:28:25 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 01 May 2009 14:28:25 +0200 Subject: [Cython] Please vote: wraparound directive Message-ID: <49FAEAE9.7000908@student.matnat.uio.no> A new directive which allows Cython to assume that integers passed to [] are not negative. Currently buffers have the negative_indices option: object[int, negative_indices=False] arr = ... but this can be cumbersome and less flexible. The rationale for not doing this in the first place was that it changes semantics during normal running rather than under exceptional circumstances, but now @cdivision has crossed that line anyway. Also note that this could potentially apply to typed lists and tuples as well, if there's a benefit. -- Dag Sverre From dagss at student.matnat.uio.no Fri May 1 14:53:32 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 01 May 2009 14:53:32 +0200 Subject: [Cython] 0.11.2 release schedule? Message-ID: <49FAF0CC.5070108@student.matnat.uio.no> Like I've been saying on some occasions, I'm going show off Cython 18th of May, and have had a tendency to scratch some itches rather than planning how to explain them... so I'd really like a release to happen before then. (Though I also have some fixes/rewrites which are not done yet which will go in next week.) If it helps I can play release manager this time and do the groundwork for making it build Sage etc. (especially if that gets us complex floats in time :-)). Would it be possible to e.g. do a full stop feature freeze next Friday 8th, and then a release the week after? -- Dag Sverre From kwmsmith at gmail.com Fri May 1 15:50:01 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Fri, 1 May 2009 08:50:01 -0500 Subject: [Cython] Please vote: wraparound directive In-Reply-To: <49FAEAE9.7000908@student.matnat.uio.no> References: <49FAEAE9.7000908@student.matnat.uio.no> Message-ID: On Fri, May 1, 2009 at 7:28 AM, Dag Sverre Seljebotn wrote: > A new directive which allows Cython to assume that integers passed to [] > are not negative. > > Currently buffers have the negative_indices option: > > ? object[int, negative_indices=False] arr = ... > > but this can be cumbersome and less flexible. The rationale for not > doing this in the first place was that it changes semantics during > normal running rather than under exceptional circumstances, but now > @cdivision has crossed that line anyway. +1 for me. Since directives can be module, function or block scoped they're pretty flexible, and it makes it clear & explicit that different semantics are going on. The current state allows the use of two different negative_indices type buffers in the same expression: arr_neg[a,b] = arr_neg[a,b] + arr_no_neg[abs(a), abs(b)] While that would be impossible with the proposed directive. I can't easily think of cases when I'd like to mix negative & non_negative indexing in the same expression, or when I couldn't just revert to regular indexing for these cases when required -- presumably I'm giving the arr_no_neg 'negative_indices=False' since I want to speed things up, and I should do the same for the other arrays. I like that it lifts the semantics out of the individual arrays and puts it in a directive. Kurt > > Also note that this could potentially apply to typed lists and tuples as > well, if there's a benefit. > > -- > Dag Sverre > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev From stefan_ml at behnel.de Fri May 1 18:18:00 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 01 May 2009 18:18:00 +0200 Subject: [Cython] 0.11.2 release schedule? In-Reply-To: <49FAF0CC.5070108@student.matnat.uio.no> References: <49FAF0CC.5070108@student.matnat.uio.no> Message-ID: <49FB20B8.8090506@behnel.de> Dag Sverre Seljebotn wrote: > Like I've been saying on some occasions, I'm going show off Cython 18th > of May, and have had a tendency to scratch some itches rather than > planning how to explain them... so I'd really like a release to happen > before then. (Though I also have some fixes/rewrites which are not done > yet which will go in next week.) > > If it helps I can play release manager this time and do the groundwork > for making it build Sage etc. (especially if that gets us complex floats > in time :-)). > > Would it be possible to e.g. do a full stop feature freeze next Friday > 8th, and then a release the week after? I'm not currently working on cython-devel anyway, so I'm glad to hear that you volunteer as QA manager. :) There are tons of open issues assigned to 0.11.2, though. I don't expect many more releases for 0.11.x (maybe a .3, but not necessarily a .4), so they may have to get moved to 0.12. At least, there are no really critical issues. Some even seem to be so low priority that we should remove their milestone all together. Even some 'wishlist' bugs sound more important than some of the 0.11.2 tickets. A minor bug in 0.11 doesn't mean it needs fixing in a 0.11.x release, especially if the fix is not trivial. Stefan From simon.king at uni-jena.de Fri May 1 19:15:59 2009 From: simon.king at uni-jena.de (Simon King) Date: Fri, 1 May 2009 19:15:59 +0200 Subject: [Cython] Problems with creating a package of extension modules Message-ID: <1241198159.49fb2e4f76162@webmail.uni-jena.de> Dear Cython-Dev, currently I try to create a Cython package comprising couple of extension modules for usage in Sage. It somehow works, but apparently the extension classes in the modules don't know where they belong to. In my setup.py, I have setup( name = "pGroupCohomology", version = "1.0", author = "Simon A. King", author_email = "simon.king at uni-jena.de", url = "http://users.minet.uni-jena.de/~king/cohomology/", description = "Modular Cohomology Rings of Groups of Prime Power Order", packages=["pGroupCohomology"], package_dir={"pGroupCohomology":"CohoSrc"}, ext_package="pGroupCohomology", ext_modules=[ Extension("mtx", sources = ["CohoSrc/mtx.pyx", ...], ...), Extension(...)] cmdclass = {'build_ext': build_ext} ) I learned from some documentation that providing the optional argument 'ext_package' should make the Extensions modules in that package, so, it should be Extension("mtx",...), not Extension("pGroupCohomology.mtx",...). Running it, I get modules that I can import into sage: sage: from pGroupCohomology.mtx import MTX sage: M=MTX('some_data_file') This works, and I can do some computations. But apparently, the class MTX does not know where it lives: sage: M.__class__ Shouldn't it say ? The callable class responsible for unpickling MTX has the same identity crisis. sage: save(M,'Test') --------------------------------------------------------------------------- PicklingError Traceback (most recent call last) ... PicklingError: Can't pickle mtx.MTX_unpickle_class: it's not the same object as mtx.MTX_unpickle_class So, MTX_unpickle_class is not identical with itself! Removing the ext_package parameter and instead writing Extension("pGroupCohomology.mtx",...) did not help. What goes wrong here? Is there a way to tell a module that it belongs to a package? Best regards, Simon ---------------------------------------------------------------- This mail was sent through http://webmail.uni-jena.de From robertwb at math.washington.edu Fri May 1 20:43:20 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 1 May 2009 11:43:20 -0700 Subject: [Cython] Please vote: wraparound directive In-Reply-To: <49FAEAE9.7000908@student.matnat.uio.no> References: <49FAEAE9.7000908@student.matnat.uio.no> Message-ID: On May 1, 2009, at 5:28 AM, Dag Sverre Seljebotn wrote: > A new directive which allows Cython to assume that integers passed > to [] > are not negative. > > Currently buffers have the negative_indices option: > > object[int, negative_indices=False] arr = ... > > but this can be cumbersome and less flexible. The rationale for not > doing this in the first place was that it changes semantics during > normal running rather than under exceptional circumstances, but now > @cdivision has crossed that line anyway. Yes, I like this. I think directives are more explicit and easier to follow than extra data attached to types. > Also note that this could potentially apply to typed lists and > tuples as > well, if there's a benefit. Yes, there is a potential benefit here. - Robert From robertwb at math.washington.edu Fri May 1 20:46:19 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 1 May 2009 11:46:19 -0700 Subject: [Cython] Problems with creating a package of extension modules In-Reply-To: <1241198159.49fb2e4f76162@webmail.uni-jena.de> References: <1241198159.49fb2e4f76162@webmail.uni-jena.de> Message-ID: <922BECD5-7D73-46BC-A541-C76081EAC703@math.washington.edu> On May 1, 2009, at 10:15 AM, Simon King wrote: > Dear Cython-Dev, > > currently I try to create a Cython package comprising couple of > extension > modules for usage in Sage. It somehow works, but apparently the > extension > classes in the modules don't know where they belong to. > > In my setup.py, I have > setup( > name = "pGroupCohomology", > version = "1.0", > author = "Simon A. King", > author_email = "simon.king at uni-jena.de", > url = "http://users.minet.uni-jena.de/~king/cohomology/", > description = "Modular Cohomology Rings of Groups of Prime Power > Order", > packages=["pGroupCohomology"], > package_dir={"pGroupCohomology":"CohoSrc"}, > ext_package="pGroupCohomology", > ext_modules=[ > Extension("mtx", > sources = ["CohoSrc/mtx.pyx", > ...], > ...), > Extension(...)] > cmdclass = {'build_ext': build_ext} > ) > > I learned from some documentation that providing the optional argument > 'ext_package' should make the Extensions modules in that package, > so, it should > be Extension("mtx",...), not Extension("pGroupCohomology.mtx",...). Hmm... could you point me to that bit of documentation? I think it's wrong (or at least not implemented as explained), we do Extension ("package.module",...) all the time in sage and that seems to mix well with pickling. - Robert From robertwb at math.washington.edu Fri May 1 21:07:36 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 1 May 2009 12:07:36 -0700 Subject: [Cython] 0.11.2 release schedule? In-Reply-To: <49FB20B8.8090506@behnel.de> References: <49FAF0CC.5070108@student.matnat.uio.no> <49FB20B8.8090506@behnel.de> Message-ID: On May 1, 2009, at 9:18 AM, Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> Like I've been saying on some occasions, I'm going show off Cython >> 18th >> of May, and have had a tendency to scratch some itches rather than >> planning how to explain them... so I'd really like a release to >> happen >> before then. (Though I also have some fixes/rewrites which are not >> done >> yet which will go in next week.) >> >> If it helps I can play release manager this time and do the >> groundwork >> for making it build Sage etc. (especially if that gets us complex >> floats >> in time :-)). >> >> Would it be possible to e.g. do a full stop feature freeze next >> Friday >> 8th, and then a release the week after? > > I'm not currently working on cython-devel anyway, so I'm glad to > hear that > you volunteer as QA manager. :) Yes, that would be great if you would do the release managing-- definitely would make a mid-May release more feasible. > There are tons of open issues assigned to 0.11.2, though. I don't > expect > many more releases for 0.11.x (maybe a .3, but not necessarily a . > 4), so > they may have to get moved to 0.12. At least, there are no really > critical > issues. Some even seem to be so low priority that we should remove > their > milestone all together. Even some 'wishlist' bugs sound more > important than > some of the 0.11.2 tickets. A minor bug in 0.11 doesn't mean it needs > fixing in a 0.11.x release, especially if the fix is not trivial. In my view, non-critical tickets attached to unreleased milestones (especially 0.x.y ones) are fair game for moving around. Please go through the list and move everything you don't think we need by the release in two weeks. - Robert From simon.king at uni-jena.de Fri May 1 21:30:25 2009 From: simon.king at uni-jena.de (Simon King) Date: Fri, 1 May 2009 21:30:25 +0200 Subject: [Cython] Problems with creating a package of extension modules In-Reply-To: <922BECD5-7D73-46BC-A541-C76081EAC703@math.washington.edu> References: <1241198159.49fb2e4f76162@webmail.uni-jena.de> <922BECD5-7D73-46BC-A541-C76081EAC703@math.washington.edu> Message-ID: <1241206225.49fb4dd161f7c@webmail.uni-jena.de> Dear Robert, Zitat von Robert Bradshaw : > Hmm... could you point me to that bit of documentation? I think it's > wrong (or at least not implemented as explained), we do Extension > ("package.module",...) all the time in sage and that seems to mix > well with pickling. Admittedly it is not Cython-documentation, but Python: At http://docs.python.org/distutils/setupscript.html, they say: --------------------- If you have a number of extensions all in the same package (or all under the same base package), use the ext_package keyword argument to setup(). For example, setup(..., ext_package='pkg', ext_modules=[Extension('foo', ['foo.c']), Extension('subpkg.bar', ['bar.c'])], ) will compile foo.c to the extension pkg.foo, and bar.c to pkg.subpkg.bar. --------------------- Anyway. As I mentioned in my original post, I also tried it *without* ext_package but with explicit package names. Five minutes ago I found that (as a leftover of a previous built) I had mtx.so not only in the pGroupCohomology folder, but also one level up. Perhaps this was confusing for Sage. A propos confusing: It is still not clear to me what the role of the setup-options "name", "packages", and "ext_package". In order to learn (one of the main advantages of open source, imho), I was looking at the setup.py of the optional pil-package for sage. They have name="PIL", package_dir={"": "PIL"}, packages=[""], ext_modules = [Extension("_imaging", ["_imaging.c"])], # dummy and apparently it results in _imaging.so in the folder PIL. If I do, analogously, name='pGroupCohomology', packages=[""], package_dir={"":"CohoSrc"}, Extension("mtx", then my (dummy) __init__.py is copied at the top level (folder SAGE_LOCAL/lib/python2.5/site-packages/), and in the same folder mtx.so shows up -- I hope this did not destroy something important. Anyway, I wonder why the PIL package works. If I do name='pGroupCohomology' packages=[""], package_dir={"":"CohoSrc"}, Extension("pGroupCohomology.mtx", then still my __init__.py runs amok. mtx.so seems in the correct place, but of course I can't import mtx if there is no __init__.py in the package folder. If I do name='pGroupCohomology' packages=["pGroupCohomology"], package_dir={"pGroupCohomology":"CohoSrc"}, ext_package="pGroupCohomology", Extension("mtx", then everything seems in the right place, in accordance with the above-mentioned documentation. I can import mtx sage: from pGroupCohomology.mtx import MTX and construct some instance M, but again sage: M.__class__ and sage: save(M,'Test') --------------------------------------------------------------------------- PicklingError Traceback (most recent call last) ... PicklingError: Can't pickle mtx.MTX_unpickle_class: import of module mtx failed Note that the error is slightly different from my original post. Apparently, at that time, I had a copy of mtx.so in the site-packages top folder, so that the import of mtx worked, but was of course not the same as pGroupCohomology/mtx.so. Conclusion: - I have a folder pGroupCohomology in the site-packages, - it contains __init__.py, hence, gives rise to a package - it contains the module mtx.so - but mtx.so believes that it is not contained in a package. Why? Cheers, Simon ---------------------------------------------------------------- This mail was sent through http://webmail.uni-jena.de From dagss at student.matnat.uio.no Fri May 1 22:00:06 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 01 May 2009 22:00:06 +0200 Subject: [Cython] 0.11.2 release schedule? In-Reply-To: References: <49FAF0CC.5070108@student.matnat.uio.no> <49FB20B8.8090506@behnel.de> Message-ID: <49FB54C6.4020009@student.matnat.uio.no> Robert Bradshaw wrote: > On May 1, 2009, at 9:18 AM, Stefan Behnel wrote: > >> Dag Sverre Seljebotn wrote: >>> Like I've been saying on some occasions, I'm going show off Cython >>> 18th >>> of May, and have had a tendency to scratch some itches rather than >>> planning how to explain them... so I'd really like a release to >>> happen >>> before then. (Though I also have some fixes/rewrites which are not >>> done >>> yet which will go in next week.) >>> >>> If it helps I can play release manager this time and do the >>> groundwork >>> for making it build Sage etc. (especially if that gets us complex >>> floats >>> in time :-)). >>> >>> Would it be possible to e.g. do a full stop feature freeze next >>> Friday >>> 8th, and then a release the week after? >> I'm not currently working on cython-devel anyway, so I'm glad to >> hear that >> you volunteer as QA manager. :) > > Yes, that would be great if you would do the release managing-- > definitely would make a mid-May release more feasible. I'll give it a try. >> There are tons of open issues assigned to 0.11.2, though. I don't >> expect >> many more releases for 0.11.x (maybe a .3, but not necessarily a . >> 4), so >> they may have to get moved to 0.12. At least, there are no really >> critical >> issues. Some even seem to be so low priority that we should remove >> their >> milestone all together. Even some 'wishlist' bugs sound more >> important than >> some of the 0.11.2 tickets. A minor bug in 0.11 doesn't mean it needs >> fixing in a 0.11.x release, especially if the fix is not trivial. > > > In my view, non-critical tickets attached to unreleased milestones > (especially 0.x.y ones) are fair game for moving around. Please go > through the list and move everything you don't think we need by the > release in two weeks. Yes, I share this view -- I tend to think of it in the direction of time-based release cycles and no so much what it says in trac. However I like the split between 0.11.x and 0.12 because it helps get some kind of sense of what needs to go in which repository -- i.e. does it make things unstable, or require something which already went into unstable? I like filing new trivial bugs to the "-devel"-release for this reason. About "wishlist", I think that just a lot of those bugs should be moved out of "wishlist"; feature requests should go there IMO. Anyway, this is not very important, but I think I'll create a 0.11.3 rather than mix things into 0.12. -- Dag Sverre From robertwb at math.washington.edu Fri May 1 23:35:35 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 1 May 2009 14:35:35 -0700 Subject: [Cython] 0.11.2 release schedule? In-Reply-To: <49FB54C6.4020009@student.matnat.uio.no> References: <49FAF0CC.5070108@student.matnat.uio.no> <49FB20B8.8090506@behnel.de> <49FB54C6.4020009@student.matnat.uio.no> Message-ID: <6D3AEDC6-8BC7-4775-83C6-16ACA1E468B2@math.washington.edu> On May 1, 2009, at 1:00 PM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On May 1, 2009, at 9:18 AM, Stefan Behnel wrote: >> >> Yes, that would be great if you would do the release managing-- >> definitely would make a mid-May release more feasible. > > I'll give it a try. Great. Thanks. >>> There are tons of open issues assigned to 0.11.2, though. I don't >>> expect >>> many more releases for 0.11.x (maybe a .3, but not necessarily a . >>> 4), so >>> they may have to get moved to 0.12. At least, there are no really >>> critical >>> issues. Some even seem to be so low priority that we should remove >>> their >>> milestone all together. Even some 'wishlist' bugs sound more >>> important than >>> some of the 0.11.2 tickets. A minor bug in 0.11 doesn't mean it >>> needs >>> fixing in a 0.11.x release, especially if the fix is not trivial. >> >> >> In my view, non-critical tickets attached to unreleased milestones >> (especially 0.x.y ones) are fair game for moving around. Please go >> through the list and move everything you don't think we need by the >> release in two weeks. > > Yes, I share this view -- I tend to think of it in the direction of > time-based release cycles and no so much what it says in trac. > > However I like the split between 0.11.x and 0.12 because it helps get > some kind of sense of what needs to go in which repository -- i.e. > does > it make things unstable, or require something which already went into > unstable? > > I like filing new trivial bugs to the "-devel"-release for this > reason. Yep, I think this is the right place for them. Also, as part of a release, I often see if there's any trivial bugs to knock off. > About "wishlist", I think that just a lot of those bugs should be > moved > out of "wishlist"; feature requests should go there IMO. I've been using this as a "it's would be nice to have, but I don't see anyone working on it in the near term." > Anyway, this is not very important, but I think I'll create a 0.11.3 > rather than mix things into 0.12. Sound good. - Robert From jusa.sj at gmail.com Sat May 2 14:19:25 2009 From: jusa.sj at gmail.com (Juha Salo) Date: Sat, 2 May 2009 15:19:25 +0300 Subject: [Cython] How to produce Python 3.0.1 compatible DLLs? Message-ID: <20db9a7d0905020519r3daba354l5bb044b1996a3718@mail.gmail.com> Hi, I've been struggling with this for a day now. I'm trying to get the Primes example from the Cython tutorial to work with Python 3.0.1 on Windows Vista. So far I've managed to produce a primes.pyd file by executing the setup.py file with Python 2.6. The primes module works when I import it in Python 2.6 but importing the module in Python 3.0.1 gives me the following: "ImportError: Module use of python26.dll conflicts with this version of Python." I haven't found instructions for using cython with Python 3. The only things I've found are the brief mention of Python 3 support in the cython docs and a forum post explaining that cython modules should be build with Python 3 headers before they can be used with Python 3 but I have no idea how to do it. Could anyone offer me some help with this? If there's a tutorial somewhere on the net, I'd love to hear about it. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090502/b27ecd5d/attachment.htm From dagss at student.matnat.uio.no Sat May 2 15:34:46 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 02 May 2009 15:34:46 +0200 Subject: [Cython] How to produce Python 3.0.1 compatible DLLs? In-Reply-To: <20db9a7d0905020519r3daba354l5bb044b1996a3718@mail.gmail.com> References: <20db9a7d0905020519r3daba354l5bb044b1996a3718@mail.gmail.com> Message-ID: <49FC4BF6.50907@student.matnat.uio.no> Juha Salo wrote: > Hi, > > I've been struggling with this for a day now. I'm trying to get the > Primes example from the Cython tutorial to work with Python 3.0.1 on > Windows Vista. So far I've managed to produce a primes.pyd file by > executing the setup.py file with Python 2.6. The primes module works > when I import it in Python 2.6 but importing the module in Python 3.0.1 > gives me the following: > > "ImportError: Module use of python26.dll conflicts with this version of > Python." > > I haven't found instructions for using cython with Python 3. The only > things I've found are the brief mention of Python 3 support in the > cython docs and a forum post explaining that cython modules should be > build with Python 3 headers before they can be used with Python 3 but I > have no idea how to do it. Could anyone offer me some help with this? If > there's a tutorial somewhere on the net, I'd love to hear about it. You need to execute setup.py using Python 3.0 to build for Python 3. Details: The C file generated by Cython is the same for all Python versions, however it must be compiled with a different include path for the Python headers depending on the Python version. Executing setup.py with the right Python version takes care of this automatically. You can see this by Py3> from distutils import sysconfig Py3> sysconfig.get_python_inc() '/home/dagss/opt/python3.0/include/python3.0' which should be passed as an include path to the C compiler. Feel free to add this to the wiki in a place you would have found it if you get it to work. -- Dag Sverre From dagss at student.matnat.uio.no Sat May 2 15:41:43 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 02 May 2009 15:41:43 +0200 Subject: [Cython] How to produce Python 3.0.1 compatible DLLs? In-Reply-To: <49FC4BF6.50907@student.matnat.uio.no> References: <20db9a7d0905020519r3daba354l5bb044b1996a3718@mail.gmail.com> <49FC4BF6.50907@student.matnat.uio.no> Message-ID: <49FC4D97.5070500@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Juha Salo wrote: >> Hi, >> >> I've been struggling with this for a day now. I'm trying to get the >> Primes example from the Cython tutorial to work with Python 3.0.1 on >> Windows Vista. So far I've managed to produce a primes.pyd file by >> executing the setup.py file with Python 2.6. The primes module works >> when I import it in Python 2.6 but importing the module in Python 3.0.1 >> gives me the following: >> >> "ImportError: Module use of python26.dll conflicts with this version of >> Python." >> >> I haven't found instructions for using cython with Python 3. The only >> things I've found are the brief mention of Python 3 support in the >> cython docs and a forum post explaining that cython modules should be >> build with Python 3 headers before they can be used with Python 3 but I >> have no idea how to do it. Could anyone offer me some help with this? If >> there's a tutorial somewhere on the net, I'd love to hear about it. > > You need to execute setup.py using Python 3.0 to build for Python 3. Heh. Actually, that is likely not possible as the current release of Cython doesn't run on Python 3 (it will not too far off though). Anyway, what *should* work is a) executing Cython manually to get a C file, and b) executing the C compiler manually with the appropriate flags. You can figure out the flags to pass to the C compiler like this (I'm not an expert on this and there might be other options to retrieve which would be slightly more appropriate...) In [1]: from distutils import sysconfig In [7]: sysconfig.get_config_var("CFLAGS_FOR_SHARED") In [8]: sysconfig.get_config_var("CFLAGSFORSHARED") Out[8]: '-fPIC' In [9]: sysconfig.get_config_var("CFLAGS") Out[9]: '-fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes' In [10]: sysconfig.get_python_inc() Out[10]: '/usr/include/python2.5' Your output will be different on Windows. -- Dag Sverre From jusa.sj at gmail.com Sat May 2 15:59:52 2009 From: jusa.sj at gmail.com (Juha Salo) Date: Sat, 2 May 2009 16:59:52 +0300 Subject: [Cython] How to produce Python 3.0.1 compatible DLLs? In-Reply-To: <49FC4D97.5070500@student.matnat.uio.no> References: <20db9a7d0905020519r3daba354l5bb044b1996a3718@mail.gmail.com> <49FC4BF6.50907@student.matnat.uio.no> <49FC4D97.5070500@student.matnat.uio.no> Message-ID: <20db9a7d0905020659s1e5a3f0aibc4ad491b1dd560e@mail.gmail.com> Ok, thanks. I'll try that. 2009/5/2 Dag Sverre Seljebotn > Dag Sverre Seljebotn wrote: > > Juha Salo wrote: > >> Hi, > >> > >> I've been struggling with this for a day now. I'm trying to get the > >> Primes example from the Cython tutorial to work with Python 3.0.1 on > >> Windows Vista. So far I've managed to produce a primes.pyd file by > >> executing the setup.py file with Python 2.6. The primes module works > >> when I import it in Python 2.6 but importing the module in Python 3.0.1 > >> gives me the following: > >> > >> "ImportError: Module use of python26.dll conflicts with this version of > >> Python." > >> > >> I haven't found instructions for using cython with Python 3. The only > >> things I've found are the brief mention of Python 3 support in the > >> cython docs and a forum post explaining that cython modules should be > >> build with Python 3 headers before they can be used with Python 3 but I > >> have no idea how to do it. Could anyone offer me some help with this? If > >> there's a tutorial somewhere on the net, I'd love to hear about it. > > > > You need to execute setup.py using Python 3.0 to build for Python 3. > > Heh. Actually, that is likely not possible as the current release of > Cython doesn't run on Python 3 (it will not too far off though). > > Anyway, what *should* work is a) executing Cython manually to get a C > file, and b) executing the C compiler manually with the appropriate flags. > > You can figure out the flags to pass to the C compiler like this (I'm > not an expert on this and there might be other options to retrieve which > would be slightly more appropriate...) > > In [1]: from distutils import sysconfig > > In [7]: sysconfig.get_config_var("CFLAGS_FOR_SHARED") > > In [8]: sysconfig.get_config_var("CFLAGSFORSHARED") > Out[8]: '-fPIC' > > In [9]: sysconfig.get_config_var("CFLAGS") > Out[9]: '-fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall > -Wstrict-prototypes' > > In [10]: sysconfig.get_python_inc() > Out[10]: '/usr/include/python2.5' > > Your output will be different on Windows. > > -- > Dag Sverre > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090502/fc65d77d/attachment.htm From stefan at sun.ac.za Mon May 4 20:33:21 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 4 May 2009 20:33:21 +0200 Subject: [Cython] List type, link on Wiki front-page Message-ID: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> Hi all, Is there a way to tell Cython if you are using a homogeneous list? E.g. cdef list[int] my_list my_list.append(1) my_list.append(2) I don't have access to edit the Wiki, but the link to WritingFastPyrexCode should be http://wiki.sagemath.org/WritingFastPyrexCode Thanks, St?fan From robertwb at math.washington.edu Mon May 4 21:37:33 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 4 May 2009 12:37:33 -0700 Subject: [Cython] List type, link on Wiki front-page In-Reply-To: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> Message-ID: On May 4, 2009, at 11:33 AM, St?fan van der Walt wrote: > Hi all, > > Is there a way to tell Cython if you are using a homogeneous list? > E.g. > > cdef list[int] my_list > > my_list.append(1) > my_list.append(2) > No, there's not. It would be nice, but it would be hard to control what people stick in the list (e.g. if you pass it off to another routine). You might want to look at using NumPy arrays, which can store lists of actual c ints, etc. - Robert From kwmsmith at gmail.com Mon May 4 21:50:08 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Mon, 4 May 2009 14:50:08 -0500 Subject: [Cython] List type, link on Wiki front-page In-Reply-To: References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> Message-ID: On Mon, May 4, 2009 at 2:37 PM, Robert Bradshaw wrote: > On May 4, 2009, at 11:33 AM, St?fan van der Walt wrote: > >> Hi all, >> >> Is there a way to tell Cython if you are using a homogeneous list? >> E.g. >> >> cdef list[int] my_list >> >> my_list.append(1) >> my_list.append(2) >> > > No, there's not. It would be nice, but it would be hard to control > what people stick in the list (e.g. if you pass it off to another > routine). You might want to look at using NumPy arrays, which can > store lists of actual c ints, etc. In addition to NumPy arrays, you might consider using array.array which support 'append' and other list methods. They're basically homogeneous lists within the Python standard lib. A bit more flexible but not as powerful as NumPy arrays. Kurt From dalcinl at gmail.com Mon May 4 21:51:05 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 4 May 2009 16:51:05 -0300 Subject: [Cython] possible regression on cython-devel Message-ID: After hg update'd cython-devel, and now the code below generates a compiler error: cdef extern from "compat.h": pass ^ ------------------------------------------------------------ /u/dalcinl/Devel/petsc4py-dev/src/PETSc/PETSc.pyx:67:4: Expected an identifier, found 'pass' IIRC, that is a documented way to force inclusion of C headers... What this intentional? BTW... "tests/broken/cdefexternempty.pyx" has a tescase for this... For some reason, it never got promoted as a working test... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Mon May 4 22:22:00 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 04 May 2009 22:22:00 +0200 Subject: [Cython] possible regression on cython-devel In-Reply-To: References: Message-ID: <49FF4E68.3050100@behnel.de> Lisandro Dalcin wrote: > After hg update'd cython-devel, and now the code below generates a > compiler error: > > cdef extern from "compat.h": > pass > ^ > ------------------------------------------------------------ > > /u/dalcinl/Devel/petsc4py-dev/src/PETSc/PETSc.pyx:67:4: Expected an > identifier, found 'pass' Hmm, looks like I've broken that. http://hg.cython.org/cython-devel/rev/6515c3757571 The grammar of the cdef statement is pretty bizarre. It previously allowed you to write "cdef pass" for a "pass" statement, for example. It looks like the change in p_c_simple_base_type() is responsible here, although I suspect the real problem is in p_statement(). It parses a statement differently when the cdef_flag is set, also externally, i.e. when the statement itself is not preceded by a "cdef". This makes cdef extern from "compat.h": pass and cdef extern from "compat.h": int a possible, but also allows interpreting "pass" as type name. That shouldn't be allowed. > IIRC, that is a documented way to force inclusion of C headers... What > this intentional? BTW... "tests/broken/cdefexternempty.pyx" has a > tescase for this... For some reason, it never got promoted as a > working test... Yes, some of those are just there because they didn't make sense in the test runner, e.g. because the resulting C code wouldn't compile, or the module wouldn't load into CPython. Greg's test suite doesn't actually C-compile stuff, but only checks that the C code comes out as expected. That might work for a one-person-project, but definitely not for Cython. Could you try to come up with a compilable test case for this? I pushed an untested fix. http://hg.cython.org/cython-devel/rev/e6f5fc96fb42 Stefan From dagss at student.matnat.uio.no Mon May 4 22:26:16 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 04 May 2009 22:26:16 +0200 Subject: [Cython] List type, link on Wiki front-page In-Reply-To: References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> Message-ID: <49FF4F68.2000506@student.matnat.uio.no> Kurt Smith wrote: > On Mon, May 4, 2009 at 2:37 PM, Robert Bradshaw > wrote: >> On May 4, 2009, at 11:33 AM, St?fan van der Walt wrote: >> >>> Hi all, >>> >>> Is there a way to tell Cython if you are using a homogeneous list? >>> E.g. >>> >>> cdef list[int] my_list >>> >>> my_list.append(1) >>> my_list.append(2) >>> >> No, there's not. It would be nice, but it would be hard to control >> what people stick in the list (e.g. if you pass it off to another >> routine). You might want to look at using NumPy arrays, which can >> store lists of actual c ints, etc. > > In addition to NumPy arrays, you might consider using array.array > which support 'append' and other list methods. They're basically > homogeneous lists within the Python standard lib. A bit more flexible > but not as powerful as NumPy arrays. The question then becomes: Do we support "array.array[int]"? And I think the answer is that we actually might in Python 2.6+ due to PEP 3118 if you make a proper pxd for the array module, but as far as I know noone has tested it. We could have basic support for this in older Python versions as well, somebody just has to write the pxd. Using append at the same time would cause a lot of trouble though (but again, they might have changed append to raise an exception if a buffer is acquired in Py2.6+ -- I don't know). -- Dag Sverre From dalcinl at gmail.com Mon May 4 22:41:56 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 4 May 2009 17:41:56 -0300 Subject: [Cython] possible regression on cython-devel In-Reply-To: <49FF4E68.3050100@behnel.de> References: <49FF4E68.3050100@behnel.de> Message-ID: On Mon, May 4, 2009 at 5:22 PM, Stefan Behnel wrote: > > > Could you try to come up with a compilable test case for this? I pushed an > untested fix. > > http://hg.cython.org/cython-devel/rev/e6f5fc96fb42 > Moving the testcase from test/broken to test/compile and adding a empty header now works for me (as well as the code in my own projects)... $ hg status A tests/compile/cdefexternempty.pyx A tests/compile/cheese.h R tests/broken/cdefexternempty.pyx IIUC, these test are C-compiled... so this should be enough, right? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Mon May 4 22:42:13 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 04 May 2009 22:42:13 +0200 Subject: [Cython] main-embedding problem Message-ID: <49FF5325.7020705@behnel.de> Hi, I just tried to use the new main() embedding feature to compile a multi-module Python program. This doesn't seem to work as expected. What I would like to see is that when I say $ cython --embed somemain.py other1.py other2.py ... Cython should generate a .c file for each .py file and add a main() function only to the first module. This main() function should then register all other modules that were compiled at the same run, so that the resulting main program can become self-contained by simply compiling all .c files into a single executable. Since this is not how it currently works (instead, all .c files get their own main() function), we might want to at least disable the --embed option for multiple compilation in 0.11.2, so that people do not start relying on this. Stefan From stefan_ml at behnel.de Mon May 4 22:47:27 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 04 May 2009 22:47:27 +0200 Subject: [Cython] possible regression on cython-devel In-Reply-To: References: <49FF4E68.3050100@behnel.de> Message-ID: <49FF545F.4010006@behnel.de> Lisandro Dalcin wrote: > On Mon, May 4, 2009 at 5:22 PM, Stefan Behnel wrote: >> >> Could you try to come up with a compilable test case for this? I pushed an >> untested fix. >> >> http://hg.cython.org/cython-devel/rev/e6f5fc96fb42 > > Moving the testcase from test/broken to test/compile and adding a > empty header now works for me (as well as the code in my own > projects)... > > $ hg status > A tests/compile/cdefexternempty.pyx > A tests/compile/cheese.h > R tests/broken/cdefexternempty.pyx > > IIUC, these test are C-compiled... so this should be enough, right? Yes, please push this to cython-devel, nothing more to test here. Stefan From dalcinl at gmail.com Mon May 4 22:53:48 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 4 May 2009 17:53:48 -0300 Subject: [Cython] main-embedding problem In-Reply-To: <49FF5325.7020705@behnel.de> References: <49FF5325.7020705@behnel.de> Message-ID: On Mon, May 4, 2009 at 5:42 PM, Stefan Behnel wrote: > Hi, > > I just tried to use the new main() embedding feature to compile a > multi-module Python program. This doesn't seem to work as expected. What I > would like to see is that when I say > > ? ?$ cython --embed somemain.py other1.py other2.py ... > > Cython should generate a .c file for each .py file and add a main() > function only to the first module. This main() function should then > register all other modules that were compiled at the same run, so that the > resulting main program can become self-contained by simply compiling all .c > files into a single executable. Since this is not how it currently works > (instead, all .c files get their own main() function), we might want to at > least disable the --embed option for multiple compilation in 0.11.2, so > that people do not start relying on this. > Unfortunatelly, I have to agree, for different reasons... As an starter, the current code does not handle Py3K on Windows (because of the new wchar_t based API) -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Mon May 4 22:59:26 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 4 May 2009 17:59:26 -0300 Subject: [Cython] possible regression on cython-devel In-Reply-To: <49FF545F.4010006@behnel.de> References: <49FF4E68.3050100@behnel.de> <49FF545F.4010006@behnel.de> Message-ID: On Mon, May 4, 2009 at 5:47 PM, Stefan Behnel wrote: >> >> $ hg status >> A tests/compile/cdefexternempty.pyx >> A tests/compile/cheese.h >> R tests/broken/cdefexternempty.pyx >> >> IIUC, these test are C-compiled... so this should be enough, right? > > Yes, please push this to cython-devel, nothing more to test here. > Pushed. BTW, could I use the same trick (I mean, add header files as appropriate) to make some testcases pass the linking step on Windows? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From lodatom at gmail.com Mon May 4 23:18:26 2009 From: lodatom at gmail.com (Mark Lodato) Date: Mon, 4 May 2009 17:18:26 -0400 Subject: [Cython] main-embedding problem In-Reply-To: <49FF5325.7020705@behnel.de> References: <49FF5325.7020705@behnel.de> Message-ID: What about using something like my script? http://www.mail-archive.com/cython-dev at codespeak.net/msg04989.html I think this solution would be much cleaner - to create a *new* C file in addition to everything on the command line. That way you can compile all your modules to object files, and then either make them into shared objects or compile them with the extra C file into a standalone. Mark On Mon, May 4, 2009 at 4:42 PM, Stefan Behnel wrote: > Hi, > > I just tried to use the new main() embedding feature to compile a > multi-module Python program. This doesn't seem to work as expected. What I > would like to see is that when I say > > ? ?$ cython --embed somemain.py other1.py other2.py ... > > Cython should generate a .c file for each .py file and add a main() > function only to the first module. This main() function should then > register all other modules that were compiled at the same run, so that the > resulting main program can become self-contained by simply compiling all .c > files into a single executable. Since this is not how it currently works > (instead, all .c files get their own main() function), we might want to at > least disable the --embed option for multiple compilation in 0.11.2, so > that people do not start relying on this. > > Stefan > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From stefan at sun.ac.za Mon May 4 23:33:56 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 4 May 2009 23:33:56 +0200 Subject: [Cython] List type, link on Wiki front-page In-Reply-To: <49FF4F68.2000506@student.matnat.uio.no> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> Message-ID: <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> 2009/5/4 Dag Sverre Seljebotn : > Using append at the same time would cause a lot of trouble though (but > again, they might have changed append to raise an exception if a buffer > is acquired in Py2.6+ -- I don't know). This is my main use-case: I need a homogenous container that automatically resizes when it is full. Maybe the performance impact of Python lists isn't even so bad -- I was only worried because I saw lots of yellow in my annotated pyx :) Cheers St?fan From robertwb at math.washington.edu Mon May 4 23:41:18 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 4 May 2009 14:41:18 -0700 Subject: [Cython] List type, link on Wiki front-page In-Reply-To: <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> Message-ID: <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> On May 4, 2009, at 2:33 PM, St?fan van der Walt wrote: > 2009/5/4 Dag Sverre Seljebotn : >> Using append at the same time would cause a lot of trouble though >> (but >> again, they might have changed append to raise an exception if a >> buffer >> is acquired in Py2.6+ -- I don't know). > > This is my main use-case: I need a homogenous container that > automatically resizes when it is full. Maybe the performance impact > of Python lists isn't even so bad -- I was only worried because I saw > lots of yellow in my annotated pyx :) Lists are actually rather fast. As a rule of thumb, yellow != slow, it just means that you're using the Python/C API (which is still often a good place to look for slowness). - Robert From stefan at sun.ac.za Mon May 4 23:43:00 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Mon, 4 May 2009 23:43:00 +0200 Subject: [Cython] Storing many, many instances Message-ID: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> Hi all, I have code that generates about 40000 instances of a certain cdef'd class. I would like to know: 1) If I define my cdef class in my_class.pyx, and then, in main.pyx do: from my_class import MyClass [.. instantiate many, many MyClass objects ..] Does this happen in C, or am I making Python calls along the way? 2) Is there a standard pattern for lowering the cost of instance creation? I think I may have to switch to structs instead of classes, but then I no longer have convenient methods or array members. Maybe I should rewrite my class to have many static methods that operate on a struct, and then collect the structs as data instead of the full instances. Any input much appreciated, Regards St?fan From dalcinl at gmail.com Mon May 4 23:45:32 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 4 May 2009 18:45:32 -0300 Subject: [Cython] main-embedding problem In-Reply-To: References: <49FF5325.7020705@behnel.de> Message-ID: On Mon, May 4, 2009 at 6:18 PM, Mark Lodato wrote: > What about using something like my script? > http://www.mail-archive.com/cython-dev at codespeak.net/msg04989.html > > I think this solution would be much cleaner - to create a *new* C file > in addition to everything on the command line. ?That way you can > compile all your modules to object files, and then either make them > into shared objects or compile them with the extra C file into a > standalone. > Looks good, however... * Not sure if the usage of PyMODINIT_FUNC is fine in the Windows case... (because of the __dclspec stuff) * In Py3K, "main()"" should be "wmain()" using wchar_t API. * Instead of using C stdlib calls for reporting failures and exiting, I would use Py_FatalError() Up to now, I'm still -1, just because: >>> import this ... Now is better than never. Although never is often better than *right* now. ... Anyway, the inittab extending stuff seems the way to go... > Mark > > On Mon, May 4, 2009 at 4:42 PM, Stefan Behnel wrote: >> Hi, >> >> I just tried to use the new main() embedding feature to compile a >> multi-module Python program. This doesn't seem to work as expected. What I >> would like to see is that when I say >> >> ? ?$ cython --embed somemain.py other1.py other2.py ... >> >> Cython should generate a .c file for each .py file and add a main() >> function only to the first module. This main() function should then >> register all other modules that were compiled at the same run, so that the >> resulting main program can become self-contained by simply compiling all .c >> files into a single executable. Since this is not how it currently works >> (instead, all .c files get their own main() function), we might want to at >> least disable the --embed option for multiple compilation in 0.11.2, so >> that people do not start relying on this. >> >> Stefan >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Mon May 4 23:49:15 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 4 May 2009 14:49:15 -0700 Subject: [Cython] main-embedding problem In-Reply-To: References: <49FF5325.7020705@behnel.de> Message-ID: On May 4, 2009, at 2:18 PM, Mark Lodato wrote: > What about using something like my script? > http://www.mail-archive.com/cython-dev at codespeak.net/msg04989.html > > I think this solution would be much cleaner - to create a *new* C file > in addition to everything on the command line. That way you can > compile all your modules to object files, and then either make them > into shared objects or compile them with the extra C file into a > standalone. If you want to include many files, this is probably a more obvious way to go. It's also nice because one can run the Cython compiler ahead of time and use the same output for .so files. > > On Mon, May 4, 2009 at 4:42 PM, Stefan Behnel > wrote: >> Hi, >> >> I just tried to use the new main() embedding feature to compile a >> multi-module Python program. This doesn't seem to work as >> expected. What I >> would like to see is that when I say >> >> $ cython --embed somemain.py other1.py other2.py ... >> >> Cython should generate a .c file for each .py file and add a main() >> function only to the first module. This main() function should then >> register all other modules that were compiled at the same run, so >> that the >> resulting main program can become self-contained by simply >> compiling all .c >> files into a single executable. Since this is not how it currently >> works >> (instead, all .c files get their own main() function), we might >> want to at >> least disable the --embed option for multiple compilation in >> 0.11.2, so >> that people do not start relying on this. Yes, lets disable it for multiple compilation for now at least. - Robert From robertwb at math.washington.edu Tue May 5 00:10:09 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 4 May 2009 15:10:09 -0700 Subject: [Cython] main-embedding problem In-Reply-To: References: <49FF5325.7020705@behnel.de> Message-ID: <7B8637C0-61E3-40D6-8BE4-04E635F78682@math.washington.edu> On May 4, 2009, at 1:53 PM, Lisandro Dalcin wrote: > On Mon, May 4, 2009 at 5:42 PM, Stefan Behnel > wrote: >> Hi, >> >> I just tried to use the new main() embedding feature to compile a >> multi-module Python program. This doesn't seem to work as >> expected. What I >> would like to see is that when I say >> >> $ cython --embed somemain.py other1.py other2.py ... >> >> Cython should generate a .c file for each .py file and add a main() >> function only to the first module. This main() function should then >> register all other modules that were compiled at the same run, so >> that the >> resulting main program can become self-contained by simply >> compiling all .c >> files into a single executable. Since this is not how it currently >> works >> (instead, all .c files get their own main() function), we might >> want to at >> least disable the --embed option for multiple compilation in >> 0.11.2, so >> that people do not start relying on this. >> > > Unfortunatelly, I have to agree, for different reasons... As an > starter, the current code does not handle Py3K on Windows (because of > the new wchar_t based API) This should be sufficient, right? http://hg.cython.org/cython-devel/rev/2e35d5caac86 - Robert From robertwb at math.washington.edu Tue May 5 00:15:10 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 4 May 2009 15:15:10 -0700 Subject: [Cython] Storing many, many instances In-Reply-To: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> Message-ID: <644465C5-4A7B-4FED-BED6-26D97C7C3CDF@math.washington.edu> On May 4, 2009, at 2:43 PM, St?fan van der Walt wrote: > Hi all, > > I have code that generates about 40000 instances of a certain > cdef'd class. > > I would like to know: > > 1) If I define my cdef class in my_class.pyx, and then, in main.pyx > do: > > from my_class import MyClass > > [.. instantiate many, many MyClass objects ..] > > Does this happen in C, or am I making Python calls along the way? You have to cimport to use C calls from another module. However, it will still call __init__ with Python semantics. The more arguments you pass (especially keyword arguments), the slower it'll get. > 2) Is there a standard pattern for lowering the cost of instance > creation? > > I think I may have to switch to structs instead of classes, but then I > no longer have convenient methods or array members. Maybe I should > rewrite my class to have many static methods that operate on a struct, > and then collect the structs as data instead of the full instances. > > Any input much appreciated, You could look into PY_NEW that we use in Sage. There was a thread on this earlier. There is inherently overhead in using Python objects, but it's usually worth the convenience. Another option would be to write a single class that provides access to a large set your objects. It all depends on your use case. - Robert From stefan at sun.ac.za Tue May 5 00:36:51 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 5 May 2009 00:36:51 +0200 Subject: [Cython] Storing many, many instances In-Reply-To: <644465C5-4A7B-4FED-BED6-26D97C7C3CDF@math.washington.edu> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <644465C5-4A7B-4FED-BED6-26D97C7C3CDF@math.washington.edu> Message-ID: <9457e7c80905041536r19777c3el8caf576b48a60845@mail.gmail.com> 2009/5/5 Robert Bradshaw : >> Does this happen in C, or am I making Python calls along the way? > > You have to cimport to use C calls from another module. However, it > will still call __init__ with Python semantics. The more arguments > you pass (especially keyword arguments), the slower it'll get. I feel really silly that I can't get this working, but Cython complains that MyClass.pxd was not found, and then says: Name 'MyClass' not declared in module 'mymodule.myclass' (but it is defined in myclass.pxd and importable in Python as myclass.MyClass). It doesn't look like sage.rings.Integer has a .pxd file, so I'm not sure why mine is failing. > You could look into PY_NEW that we use in Sage. There was a thread on > this earlier. There is inherently overhead in using Python objects, > but it's usually worth the convenience. Another option would be to > write a single class that provides access to a large set your > objects. It all depends on your use case. Thank you, I'll have a look! Cheers St?fan From greg.ewing at canterbury.ac.nz Tue May 5 02:18:51 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 05 May 2009 12:18:51 +1200 Subject: [Cython] possible regression on cython-devel In-Reply-To: <49FF4E68.3050100@behnel.de> References: <49FF4E68.3050100@behnel.de> Message-ID: <49FF85EB.6010401@canterbury.ac.nz> Stefan Behnel wrote: > The grammar of the cdef statement is pretty bizarre. Talking about "the cdef statement" is a bit misleading. The way I think of it in Pyrex, there isn't a single cdef statement. Rather, there is a cdef prefix that can be applied to declarations or suites of declarations to give them a different meaning. Since 'pass' is a valid declaration statement (it needs to be so that you can have empty suites of declarations), 'cdef pass' becomes a slightly fancier way of writing an empty declaration. It might look a bit odd, but it doesn't do any harm to allow it, IMO. Another thing you need to be careful not to break is cdef: where all the statements in the suite are treated as though they were prefixed with cdef (including pass statements!). -- Greg From greg.ewing at canterbury.ac.nz Tue May 5 02:48:00 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 05 May 2009 12:48:00 +1200 Subject: [Cython] List type, link on Wiki front-page In-Reply-To: <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> Message-ID: <49FF8CC0.7020907@canterbury.ac.nz> Robert Bradshaw wrote: > Lists are actually rather fast. As a rule of thumb, yellow != slow, > it just means that you're using the Python/C API (which is still > often a good place to look for slowness). What's likely to be more of a problem is converting between C and Python ints a lot, rather than the list appends themselves. -- Greg From brett.calcott at gmail.com Tue May 5 05:08:21 2009 From: brett.calcott at gmail.com (Brett Calcott) Date: Tue, 5 May 2009 13:08:21 +1000 Subject: [Cython] Include paths for pyximport Message-ID: Hi all, I'm using pyimport and writing some extensions to work numpy. To get things to happen magically the compiler needs to be able to find the header files. I've got it working on my mac, by adding a line to my .bashrc export C_INCLUDE_PATH=/Library/Python/2.5/site-packages/numpy/core/include I'd like to get it working on windows. I've added the equivalent path in my windows env (set INCLUDE=...), but I'm getting an error telling me it cannot find the "include/arrayobject.h". I've tried setting the right directory in the GUI too, but to no avail. Can anyone tell me the easy/right way to add include paths for pyximport on windows. Is there a better and portable way to do this? Thanks, Brett ps. pyximport rocks! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090505/318288a9/attachment.htm From hoytak at cs.ubc.ca Tue May 5 05:26:26 2009 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Mon, 4 May 2009 20:26:26 -0700 Subject: [Cython] List type, link on Wiki front-page In-Reply-To: <49FF8CC0.7020907@canterbury.ac.nz> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> Message-ID: <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> This is a common use case for me as well. What I've done (for the case of ints, doubles, etc.) is have a separate struct that emulates the vector<> in C++; there are associated functions that resize it, create it, etc. Like vector<>, it allocates more space than needed for new members, and uses malloc and realloc as appropriate. I then have a .d structure member, holding a pointer to the data, so in my code I just use v.d[i] to access the i'th element. However, this approach is far from as elegant as I'd like it to be. Out of curiosity, how difficult would it be to implement a new container type that emulates python's list but only holds one time and would work with both PyObject* and c types? Say vector[type] to distinguish it from list. Here would be my wishlist, given my current use cases, in order of desirability. 1. Very fast c level element access using []. Specifically, as fast as C array access if it holds a C data type and as fast as list if holding a python type. 2. All memory management issues handled automatically. 3. List-like methods -- append, pop, del, etc. -- that avoid any python overhead except possible ref counting stuff. 4. Ability to convert to and from python list in interfacing with python code (doesn't need to be the fastest, but would have in-bulk type checking / C--PyObject* conversions where needed). 5. Slicing that works like list's slicing. I suspect there are tons of hidden issues behind each of these, and I also understand it would be good to do get other issues out of the way in this. I am quite busy currently due to classes, research, and an upcoming prelim exam, but would be happy to put some time into this if someone can mentor me. I've been meaning to get more into cython development for a long time but am always too busy, so I might as well start sometime. --Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From dagss at student.matnat.uio.no Tue May 5 06:35:59 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 5 May 2009 06:35:59 +0200 (CEST) Subject: [Cython] Storing many, many instances In-Reply-To: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> Message-ID: <8f5db410ca5009dd2aec60dce437cd23.squirrel@webmail.uio.no> St?fan van der Walt wrote: > I think I may have to switch to structs instead of classes, but then I > no longer have convenient methods or array members. Maybe I should > rewrite my class to have many static methods that operate on a struct, > and then collect the structs as data instead of the full instances. Technically it would not be too difficult to allow cdef struct MyStruct: void foo(self): # self is pure MyStruct, perhaps stack-allocated in Cython, and transform mystructinstance.foo() into __mangled_MyStruct_foo(mystructinstance) The question is whether we want it or not, perhaps it would just be confusing to provide OO-like syntax on something that's neither reference counted nor polymorphic. Also the line has to be drawn somewhere before we rewrite C++ :-) So I don't know what I think about it myself, but it's not too difficult technically. Dag Sverre From dagss at student.matnat.uio.no Tue May 5 06:39:26 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 5 May 2009 06:39:26 +0200 (CEST) Subject: [Cython] Storing many, many instances In-Reply-To: <8f5db410ca5009dd2aec60dce437cd23.squirrel@webmail.uio.no> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <8f5db410ca5009dd2aec60dce437cd23.squirrel@webmail.uio.no> Message-ID: <7c940a5d130e97a4d739efc6f3343bff.squirrel@webmail.uio.no> > St?fan van der Walt wrote: >> I think I may have to switch to structs instead of classes, but then I >> no longer have convenient methods or array members. Maybe I should >> rewrite my class to have many static methods that operate on a struct, >> and then collect the structs as data instead of the full instances. > > Technically it would not be too difficult to allow > > cdef struct MyStruct: > void foo(self): > # self is pure MyStruct, perhaps stack-allocated > > in Cython, and transform > > mystructinstance.foo() > > into > > __mangled_MyStruct_foo(mystructinstance) Sorry, it should be MyStruct* as self, and one would transform a call into __mangled_MyStruct_foo(&mystructinstance) Dag Sverre From dagss at student.matnat.uio.no Tue May 5 06:52:36 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 5 May 2009 06:52:36 +0200 (CEST) Subject: [Cython] List type, link on Wiki front-page In-Reply-To: <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> Message-ID: Hoyt Koepke wrote: > This is a common use case for me as well. What I've done (for the > case of ints, doubles, etc.) is have a separate struct that emulates > the vector<> in C++; there are associated functions that resize it, > create it, etc. Like vector<>, it allocates more space than needed > for new members, and uses malloc and realloc as appropriate. I then > have a .d structure member, holding a pointer to the data, so in my > code I just use v.d[i] to access the i'th element. However, this > approach is far from as elegant as I'd like it to be. > > Out of curiosity, how difficult would it be to implement a new > container type that emulates python's list but only holds one time and > would work with both PyObject* and c types? Say vector[type] to > distinguish it from list. Here would be my wishlist, given my current > use cases, in order of desirability. I don't think such a type should be built into Cython itself, but should be written as just another class in pyx and pxd files (which can be shipped with Cython perhaps). However, it seems to require templates/generics in Cython. So that would be the place to start working. Unfortunately that would probably be some work (on the order of a GSoC I think?), but I think that would be a much better approach than hard-coding in a brand new vector type in Cython. (This is when I usually state that I think it is better if we can properly wrap STL though, which will hopefully be done something about in summer, since the list of data structures which it would be nice to have doesn't at all stop with vectors.) Dag Sverre From hoytak at cs.ubc.ca Tue May 5 07:17:57 2009 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Mon, 4 May 2009 22:17:57 -0700 Subject: [Cython] List type, link on Wiki front-page In-Reply-To: References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> Message-ID: <4db580fd0905042217v30ce34dal615cf55bdc586b73@mail.gmail.com> >> Out of curiosity, how difficult would it be to implement a new >> container type that emulates python's list but only holds one time and >> would work with both PyObject* and c types? ?Say vector[type] to >> distinguish it from list. ?Here would be my wishlist, given my current >> use cases, in order of desirability. > > I don't think such a type should be built into Cython itself, but should > be written as just another class in pyx and pxd files (which can be > shipped with Cython perhaps). Yes, I agree, that makes sense. Being able to do these operations for general stuff, beyond just something like vector[int], would be well worth implementing, at least in my book. I would use them frequently. > However, it seems to require templates/generics in Cython. So that would > be the place to start working. Unfortunately that would probably be some > work (on the order of a GSoC I think?), but I think that would be a much > better approach than hard-coding in a brand new vector type in Cython. Yes, I see; well I would love to help, but grad school has a way of sucking up one's free time, and there are probably better ways for me to learn the cython code base. > (This is when I usually state that I think it is better if we can properly > wrap STL though, which will hopefully be done something about in summer, > since the list of data structures which it would be nice to have doesn't > at all stop with vectors.) I totally agree; STL-type maps of int to double, etc. would be awesome and used often. --Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From stefan_ml at behnel.de Tue May 5 09:50:22 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 May 2009 09:50:22 +0200 Subject: [Cython] template types In-Reply-To: References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> Message-ID: <49FFEFBE.6010800@behnel.de> Dag Sverre Seljebotn wrote: > Hoyt Koepke wrote: >> Out of curiosity, how difficult would it be to implement a new >> container type that emulates python's list but only holds one time and >> would work with both PyObject* and c types? Say vector[type] to >> distinguish it from list. Here would be my wishlist, given my current >> use cases, in order of desirability. > > I don't think such a type should be built into Cython itself, but should > be written as just another class in pyx and pxd files (which can be > shipped with Cython perhaps). Agreed. > However, it seems to require templates/generics in Cython. So that would > be the place to start working. Unfortunately that would probably be some > work (on the order of a GSoC I think?), but I think that would be a much > better approach than hard-coding in a brand new vector type in Cython. > > (This is when I usually state that I think it is better if we can properly > wrap STL though, which will hopefully be done something about in summer, > since the list of data structures which it would be nice to have doesn't > at all stop with vectors.) Wrapping the STL sounds like a different feature. I don't think we should *require* C++ here. Writing type templates into plain C would be a lot more useful. It would be more work, sure, but it shouldn't be too hard. It would mainly require smarter type checks (i.e. MyType != MyType) and code duplication in the generated C code. Maybe it's not even that hard to hack that into the current code. Imagine you could write cdef class MyType(object): # type T gets defined in the class scope to make the parser happy cdef T* my_attribute def __init__(self, T value): self.my_attribute = malloc(sizeof(T)*1000) self.my_attribute[100] = value def T do_work(self, T input): return input + self.my_attribute[100] o = MyType(20) # first occurrence creates/registers the type o.do_work(18) ctypedef MyType MyIntType # reuses the type and provides a name cdef MyIntType ot = MyIntType(25) print ot.do_work(19) ctypedef MyType MyDType # creates and defines a new type I'm not sure if a common base class would make sense here, something that would just implement everything that doesn't depend on T. But maybe that's overkill already. And most stuff *will* We might be able to implement this completely textually, i.e. the parser would declare the specialised types on the fly (as it does now for a ctypedef or extension type, i.e. new C type names). The resulting types would really be named "MyType" internally, and we'd just do some name mangling on the way out. It would just be about as 'powerful' as Java's compile-time generics (minus being limited to types), and not more. That doesn't sound like a GSoC to me. Even template functions would be out of scope at the beginning (as they would require knowing the type of a value/name that you pass, which isn't available at parse time, i.e. doesn't fit the current type declaration scheme). Stefan From dagss at student.matnat.uio.no Tue May 5 10:22:54 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 5 May 2009 10:22:54 +0200 (CEST) Subject: [Cython] template types In-Reply-To: <49FFEFBE.6010800@behnel.de> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> Message-ID: <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> However, it seems to require templates/generics in Cython. So that would >> be the place to start working. Unfortunately that would probably be some >> work (on the order of a GSoC I think?), but I think that would be a much >> better approach than hard-coding in a brand new vector type in Cython. >> >> (This is when I usually state that I think it is better if we can >> properly >> wrap STL though, which will hopefully be done something about in summer, >> since the list of data structures which it would be nice to have doesn't >> at all stop with vectors.) > > Wrapping the STL sounds like a different feature. I don't think we should > *require* C++ here. Writing type templates into plain C would be a lot > more > useful. It would be more work, sure, but it shouldn't be too hard. It > would > mainly require smarter type checks (i.e. MyType != MyType) > and > code duplication in the generated C code. Maybe it's not even that hard to > hack that into the current code. > > Imagine you could write > > cdef class MyType(object): > # type T gets defined in the class scope to make the parser happy > cdef T* my_attribute > def __init__(self, T value): > self.my_attribute = malloc(sizeof(T)*1000) > self.my_attribute[100] = value > def T do_work(self, T input): > return input + self.my_attribute[100] > > o = MyType(20) # first occurrence creates/registers the type > o.do_work(18) > > ctypedef MyType MyIntType # reuses the type and provides a name > > cdef MyIntType ot = MyIntType(25) > print ot.do_work(19) > > ctypedef MyType MyDType # creates and defines a new type > > I'm not sure if a common base class would make sense here, something that > would just implement everything that doesn't depend on T. But maybe that's > overkill already. And most stuff *will* Hmm. Yes, you are right, there's lots of stuff which doesn't need to be supported at first. Somebody still has to do it though :-) Before the C++ syntax sticks though I'd like highlight some benefits of the [] syntax: - It is legal Python to do "vector[cython.int]()". (Which leaves the way open for providing a meta-class doing run-time type-checking of arguments and return values, which would be the Python run-time compatability layer in the shadow module. That's far off and perhaps too little useful, but I like leaving the way open.). - Guido once used the [] syntax for something similar (which never became anything) [1] - Using < and > makes the parser more complicated, while [] must already be parsed. (in C++ there's the mess of having to write "vector >", note the space, because there would be some problem with the >> operator and the parser otherwise). [1] http://www.artima.com/weblogs/viewpost.jsp?thread=86641 I even think one should use [] for wrapping STL personally. Dag Sverre From stefan_ml at behnel.de Tue May 5 13:07:35 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 5 May 2009 13:07:35 +0200 (CEST) Subject: [Cython] template types In-Reply-To: <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> Message-ID: <75369f0f2977f46a3f1c2a8be7952925.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > Before the C++ syntax sticks I actually think it makes sense. In declarations, the parameter can appear only behind type names, so parsing them would be trivial - except for cases where the type name appears in an expression: isinstance(x, MyType) type_ref = MyType vs. type_ref = SomeValue < something > 5 Not sure if type_ref = MyType[int] makes this any simpler - it may at least simplify the parser, although there might be further ambiguities with buffers, array sizes and indexing. I also considered keyword arguments to the type (like Py3's "metaclass"), but didn't find an obvious way to keep that information in the type name when referring to it later on. cdef class MyType(SomeParent, templates=[T,V]): ... cdef MyType(T=int, V=list) my_instance my_instance = MyType(1,2,3,4, T=int, V=list) ... does that look readable? > I'd like highlight some benefits of the [] syntax: > > - It is legal Python to do "vector[cython.int]()". (Which leaves the way > open for providing a meta-class doing run-time type-checking of arguments > and return values, which would be the Python run-time compatability layer > in the shadow module. That's far off and perhaps too little useful, but I > like leaving the way open.). IMHO not a major goal, though. Plus, the type definition itself would still not match Python's syntax in that case, so the gain is rather marginal. > - Guido once used the [] syntax for something similar (which never became > anything) [1] > [1] http://www.artima.com/weblogs/viewpost.jsp?thread=86641 Yes, although I wouldn't count on that. If you read the link at the top of that post (i.e. what became of it), you'll see that that syntax is basically gone before being examined. > - Using < and > makes the parser more complicated, while [] must already > be parsed. (in C++ there's the mess of having to write "vector >>", note the space, because there would be some problem with the >> > operator and the parser otherwise). That's sick. Looks like a scanner problem, though, not a parser problem. The parser would know that '>>' doesn't make any sense at all at that position (ok, it's C++, but let's assume a sane kind of 'sense' here). Stefan From hoytak at cs.ubc.ca Tue May 5 17:40:40 2009 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Tue, 5 May 2009 08:40:40 -0700 Subject: [Cython] template types In-Reply-To: <75369f0f2977f46a3f1c2a8be7952925.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <75369f0f2977f46a3f1c2a8be7952925.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <4db580fd0905050840q56f1a25cx5f9496fbb0d1475d@mail.gmail.com> On Tue, May 5, 2009 at 4:07 AM, Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> Before the C++ syntax sticks > > I actually think it makes sense. In declarations, the parameter can appear > only behind type names, so parsing them would be trivial - except for > cases where the type name appears in an expression: > > ? ?isinstance(x, MyType) > ? ?type_ref = MyType ?vs. ?type_ref = SomeValue < something > 5 > > Not sure if > > ? ?type_ref = MyType[int] > > makes this any simpler - it may at least simplify the parser, although > there might be further ambiguities with buffers, array sizes and indexing. One other thing that you all might want to consider is D's syntax for templates (http://www.digitalmars.com/d/2.0/templates-revisited.html). As much as I am familiar with C++ template stuff, I think it is much better thought out than C++. I know it's completely tangential to python/C, but might be worth considering. --Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From dagss at student.matnat.uio.no Tue May 5 17:47:59 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 05 May 2009 17:47:59 +0200 Subject: [Cython] template types In-Reply-To: <75369f0f2977f46a3f1c2a8be7952925.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <75369f0f2977f46a3f1c2a8be7952925.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <4A005FAF.4080107@student.matnat.uio.no> Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> Before the C++ syntax sticks > > I actually think it makes sense. In declarations, the parameter can appear Well it certainly has the advantage that C++ and Java knows what it means right away. > only behind type names, so parsing them would be trivial - except for > cases where the type name appears in an expression: > > isinstance(x, MyType) > type_ref = MyType vs. type_ref = SomeValue < something > 5 In general, foo(baz) can mean two things, and you don't know until you know what kind of expression foo is. This makes it virtually guaranteed that if Python gets generic interfaces somehow [1] this will not be the syntax of choice, as Guido hates these kinds of things even if it is technically possible to seperate it, possible to require that you write a<(b>a) or similar. Also in Python (and in Cython with metaprogramming/macro support!), one could envisage doing def f(): if wierd: return FooTemplate else: return BarTemplate f()(baz) # operators or template instantiation? won't know 'till runtime [1] Well, I suppose 3rd party libraries can already add interfaces, including templated ones, using metaclasses > Not sure if > > type_ref = MyType[int] > > makes this any simpler - it may at least simplify the parser, although > there might be further ambiguities with buffers, array sizes and indexing. Well, there's already ambiguities here (and still a regression compared to before I mixed things up with buffers) making the current parser approach suboptimal. One day when I get time I think I'll just introduce BracketOperatorNode, and leave deciding what it does to later transforms. One thing I really dislike in current Cython code is that the parser has to know whether things are types or not in a given context. (Although I think Robert fixed at least some of this.) BTW one can look at a template as a collection of types, and template instantiation as fetching the right type in the collection. > I also considered keyword arguments to the type (like Py3's "metaclass"), > but didn't find an obvious way to keep that information in the type name > when referring to it later on. > > cdef class MyType(SomeParent, templates=[T,V]): > ... > > cdef MyType(T=int, V=list) my_instance > my_instance = MyType(1,2,3,4, T=int, V=list) > > ... does that look readable? Hmm. It's an alternative to consider although I must admit my guts prefer MyType to this. > That's sick. Looks like a scanner problem, though, not a parser problem. > The parser would know that '>>' doesn't make any sense at all at that > position (ok, it's C++, but let's assume a sane kind of 'sense' here). Well it is the same kind of problem as above; basically you don't know whether a> d; is doing arithmetic or instantiating a variable until you can fully resolve what kind of identifiers a, b, c and d are. Just makes everything more complicated, and it is harder to parse for humans as well. The difference with a[b](c) is that you at least know how to structure the parse tree and the operator precedence just by looking at it without knowing anything about a, b and c. -- Dag Sverre From seb.binet at gmail.com Tue May 5 17:48:11 2009 From: seb.binet at gmail.com (Sebastien Binet) Date: Tue, 5 May 2009 17:48:11 +0200 Subject: [Cython] template types In-Reply-To: <4db580fd0905050840q56f1a25cx5f9496fbb0d1475d@mail.gmail.com> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <75369f0f2977f46a3f1c2a8be7952925.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4db580fd0905050840q56f1a25cx5f9496fbb0d1475d@mail.gmail.com> Message-ID: <200905051748.11645.binet@cern.ch> hi, lurking_mode=off On Tuesday 05 May 2009 17:40:40 Hoyt Koepke wrote: > On Tue, May 5, 2009 at 4:07 AM, Stefan Behnel wrote: > > Dag Sverre Seljebotn wrote: > >> Before the C++ syntax sticks > > > > I actually think it makes sense. In declarations, the parameter can > > appear only behind type names, so parsing them would be trivial - except > > for cases where the type name appears in an expression: > > > > isinstance(x, MyType) > > type_ref = MyType vs. type_ref = SomeValue < something > 5 > > > > Not sure if > > > > type_ref = MyType[int] > > > > makes this any simpler - it may at least simplify the parser, although > > there might be further ambiguities with buffers, array sizes and > > indexing. > > One other thing that you all might want to consider is D's syntax for > templates (http://www.digitalmars.com/d/2.0/templates-revisited.html). > As much as I am familiar with C++ template stuff, I think it is much > better thought out than C++. I know it's completely tangential to > python/C, but might be worth considering. it is a bit disturbing though: auto foo = Bar !(3, Baz, 42); auto bar = Foo !(z); it is hard to not think of these statements as negating something (probably especially the second one ?) I wish D had chosen @ or $ instead of !. cheers, sebastien. -- ######################################### # Dr. Sebastien Binet # Laboratoire de l'Accelerateur Lineaire # Universite Paris-Sud XI # Batiment 200 # 91898 Orsay ######################################### From stefan at sun.ac.za Tue May 5 18:03:46 2009 From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=) Date: Tue, 5 May 2009 18:03:46 +0200 Subject: [Cython] Storing many, many instances In-Reply-To: <9457e7c80905041536r19777c3el8caf576b48a60845@mail.gmail.com> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <644465C5-4A7B-4FED-BED6-26D97C7C3CDF@math.washington.edu> <9457e7c80905041536r19777c3el8caf576b48a60845@mail.gmail.com> Message-ID: <9457e7c80905050903u63e65f69j2c0ec0cbe3f9f98f@mail.gmail.com> 2009/5/5 St?fan van der Walt : >> You have to cimport to use C calls from another module. However, it >> will still call __init__ with Python semantics. The more arguments >> you pass (especially keyword arguments), the slower it'll get. > > I feel really silly that I can't get this working, but Cython > complains that MyClass.pxd was not found, and then says: > > Name 'MyClass' not declared in module 'mymodule.myclass' (but it is > defined in myclass.pxd and importable in Python as myclass.MyClass). > > It doesn't look like sage.rings.Integer has a .pxd file, so I'm not > sure why mine is failing. In case someone comes across this thread: Sage *does* have .pxd files to the integer class, otherwise this wouldn't work. The procedure is: 1. Define your class in myclass.pyx: cdef class MyClass: cdef test(self): print "hello!" 2. Define a pxd "header" file: cdef class MyClass: cdef test(self) 3. Use *both* cimport and normal import: from myclass cimport MyClass as MyClassT from myclass import MyClass cdef MyClassT m = MyClass() m.test() Hope that saves someone some time along the way. Cheers St?fan From stefan_ml at behnel.de Tue May 5 18:57:39 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 May 2009 18:57:39 +0200 Subject: [Cython] Storing many, many instances In-Reply-To: <9457e7c80905050903u63e65f69j2c0ec0cbe3f9f98f@mail.gmail.com> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <644465C5-4A7B-4FED-BED6-26D97C7C3CDF@math.washington.edu> <9457e7c80905041536r19777c3el8caf576b48a60845@mail.gmail.com> <9457e7c80905050903u63e65f69j2c0ec0cbe3f9f98f@mail.gmail.com> Message-ID: <4A007003.7090708@behnel.de> St?fan van der Walt wrote: > 1. Define your class in myclass.pyx: > > cdef class MyClass: > cdef test(self): > print "hello!" > > 2. Define a pxd "header" file: > > cdef class MyClass: > cdef test(self) > > 3. Use *both* cimport and normal import: > > from myclass cimport MyClass as MyClassT > from myclass import MyClass > > cdef MyClassT m = MyClass() Note that this will generate unnecessary type checks in the code as Cython cannot know that both types are identical. There should be no reason to import both, just stick with the cimport. Stefan From hoytak at cs.ubc.ca Tue May 5 20:42:06 2009 From: hoytak at cs.ubc.ca (Hoyt Koepke) Date: Tue, 5 May 2009 11:42:06 -0700 Subject: [Cython] template types In-Reply-To: <200905051748.11645.binet@cern.ch> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <75369f0f2977f46a3f1c2a8be7952925.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4db580fd0905050840q56f1a25cx5f9496fbb0d1475d@mail.gmail.com> <200905051748.11645.binet@cern.ch> Message-ID: <4db580fd0905051142r69076d82re4f9301c54bb51d@mail.gmail.com> > it is a bit disturbing though: > ?auto foo = Bar !(3, Baz, 42); > ?auto bar = Foo !(z); > > it is hard to not think of these statements as negating something (probably > especially the second one ?) > ?I wish D had chosen @ or $ instead of !. Perhaps, but isn't it true that every language has slightly confusing stuff when first considered? Whitespace in python, < > for template stuff in C++, [] notation for buffers in cython (I've gotten used to it fine, and don't mind it now, but it did make me to do a few double-takes at first). What I like about D's syntax is that it's easy to pick out !( ) as type arguments to a function, class, method, etc. and ( ) for the regular arguments. What I have found to be nice (and granted, I haven't done much D programming recently) is that type parameters and regular parameters follow similar syntax, by intention. I've found that templating stuff in D is an amazingly nice experience (C++ just stinks in comparison) as once I've got my head around what can and can't be done, the syntax is very intuitive. Just my 2 cents, but I'm far from being experienced enough to deside what would work best here. -- Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From stefan_ml at behnel.de Tue May 5 20:48:26 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 May 2009 20:48:26 +0200 Subject: [Cython] template types In-Reply-To: <4db580fd0905050840q56f1a25cx5f9496fbb0d1475d@mail.gmail.com> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <75369f0f2977f46a3f1c2a8be7952925.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4db580fd0905050840q56f1a25cx5f9496fbb0d1475d@mail.gmail.com> Message-ID: <4A0089FA.6060403@behnel.de> Hoyt Koepke wrote: > On Tue, May 5, 2009 at 4:07 AM, Stefan Behnel wrote: >> Dag Sverre Seljebotn wrote: >>> Before the C++ syntax sticks >> I actually think it makes sense. In declarations, the parameter can appear >> only behind type names, so parsing them would be trivial - except for >> cases where the type name appears in an expression: >> >> isinstance(x, MyType) >> type_ref = MyType vs. type_ref = SomeValue < something > 5 >> >> Not sure if >> >> type_ref = MyType[int] >> >> makes this any simpler - it may at least simplify the parser, although >> there might be further ambiguities with buffers, array sizes and indexing. > > One other thing that you all might want to consider is D's syntax for > templates (http://www.digitalmars.com/d/2.0/templates-revisited.html). > As much as I am familiar with C++ template stuff, I think it is much > better thought out than C++. I know it's completely tangential to > python/C, but might be worth considering. Hmm, the '!' might make sense in the D (and maybe C) corner, but for Python and Cython? Stefan From stefan_ml at behnel.de Tue May 5 10:08:10 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 May 2009 10:08:10 +0200 Subject: [Cython] possible regression on cython-devel In-Reply-To: <49FF85EB.6010401@canterbury.ac.nz> References: <49FF4E68.3050100@behnel.de> <49FF85EB.6010401@canterbury.ac.nz> Message-ID: <49FFF3EA.6050606@behnel.de> Greg Ewing wrote: > Talking about "the cdef statement" is a bit misleading. > The way I think of it in Pyrex, there isn't a single > cdef statement. Rather, there is a cdef prefix that > can be applied to declarations or suites of declarations > to give them a different meaning. That makes it clearer, although the parser function is still called "p_cdef_statement". > Since 'pass' is a valid declaration statement (it needs > to be so that you can have empty suites of declarations), > 'cdef pass' becomes a slightly fancier way of writing > an empty declaration. It might look a bit odd, but it > doesn't do any harm to allow it, IMO. It's perfectly sufficient to allow "pass" in a cdef block, rather than in any cdef context. That's what I did in my last fix. > Another thing you need to be careful not to break is > > cdef: > > > where all the statements in the suite are treated as > though they were prefixed with cdef (including pass > statements!). Thanks, I just checked, and there is a test case for that (tests/compile/ia_cdefblock.pyx). I still think it makes a lot more sense to support cdef: pass than to allow cdef pass explicitly in the grammar. Stefan From stefan_ml at behnel.de Tue May 5 13:53:34 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 May 2009 13:53:34 +0200 Subject: [Cython] Storing many, many instances In-Reply-To: <7c940a5d130e97a4d739efc6f3343bff.squirrel@webmail.uio.no> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <8f5db410ca5009dd2aec60dce437cd23.squirrel@webmail.uio.no> <7c940a5d130e97a4d739efc6f3343bff.squirrel@webmail.uio.no> Message-ID: <4A0028BE.9030903@behnel.de> Dag Sverre Seljebotn wrote: >> St?fan van der Walt wrote: >>> I think I may have to switch to structs instead of classes, but then I >>> no longer have convenient methods or array members. Maybe I should >>> rewrite my class to have many static methods that operate on a struct, >>> and then collect the structs as data instead of the full instances. >> Technically it would not be too difficult to allow >> >> cdef struct MyStruct: >> void foo(self): >> # self is pure MyStruct, perhaps stack-allocated >> >> in Cython, and transform >> >> mystructinstance.foo() >> >> into >> >> __mangled_MyStruct_foo(mystructinstance) > > Sorry, it should be MyStruct* as self, and one would transform a call into > > __mangled_MyStruct_foo(&mystructinstance) I wouldn't mind. We have a lot of convenience stuff in structs already (e.g. dict coercion and keyword arguments), so adding methods with the above semantics wouldn't hurt. It's not rewriting C++ either, as long as it makes sense in Cython as a language. Stefan From seb.binet at gmail.com Tue May 5 20:59:23 2009 From: seb.binet at gmail.com (Sebastien Binet) Date: Tue, 5 May 2009 20:59:23 +0200 Subject: [Cython] template types In-Reply-To: <4db580fd0905051142r69076d82re4f9301c54bb51d@mail.gmail.com> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <200905051748.11645.binet@cern.ch> <4db580fd0905051142r69076d82re4f9301c54bb51d@mail.gmail.com> Message-ID: <200905052059.24065.binet@cern.ch> On Tuesday 05 May 2009 20:42:06 Hoyt Koepke wrote: > > it is a bit disturbing though: > > auto foo = Bar !(3, Baz, 42); > > auto bar = Foo !(z); > > > > it is hard to not think of these statements as negating something > > (probably especially the second one ?) > > I wish D had chosen @ or $ instead of !. > > Perhaps, but isn't it true that every language has slightly confusing > stuff when first considered? Whitespace in python, < > for template > stuff in C++, [] notation for buffers in cython (I've gotten used to > it fine, and don't mind it now, but it did make me to do a few > double-takes at first). What I like about D's syntax is that it's > easy to pick out !( ) as type arguments to a function, class, method, > etc. and ( ) for the regular arguments. What I have found to be nice > (and granted, I haven't done much D programming recently) is that type > parameters and regular parameters follow similar syntax, by intention. > I've found that templating stuff in D is an amazingly nice experience > (C++ just stinks in comparison) as once I've got my head around what > can and can't be done, the syntax is very intuitive. in fact, I am not really sure I am buying that this: auto foo = Foo !(T1, T2, T3)(arg1, arg2); is really cleaner/clearer than: auto foo = Foo (T1, T2, T3)(arg1, arg2); that is what we do in python/PyROOT (the main python-to-C/C++ module we use in my field (High Energy Physics)): import ROOT cls =ROOT.std.vector(double) # instantiates std::vector data = cls() assert hasattr(data, 'size') my_data = ROOT.std.vector('MyNS::MyClass')() my_data.push_back(...) assert isinstance(my_data[0], ROOT.MyNS.MyClass) ROOT.std.vector being just a function returning classes, by type or by (C++) class name... cheers, sebastien. -- ######################################### # Dr. Sebastien Binet # Laboratoire de l'Accelerateur Lineaire # Universite Paris-Sud XI # Batiment 200 # 91898 Orsay ######################################### From stefan_ml at behnel.de Tue May 5 21:10:18 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 May 2009 21:10:18 +0200 Subject: [Cython] template types In-Reply-To: <200905052059.24065.binet@cern.ch> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <200905051748.11645.binet@cern.ch> <4db580fd0905051142r69076d82re4f9301c54bb51d@mail.gmail.com> <200905052059.24065.binet@cern.ch> Message-ID: <4A008F1A.2070400@behnel.de> Sebastien Binet wrote: > On Tuesday 05 May 2009 20:42:06 Hoyt Koepke wrote: >>> it is a bit disturbing though: >>> auto foo = Bar !(3, Baz, 42); >>> auto bar = Foo !(z); >>> >>> it is hard to not think of these statements as negating something >>> (probably especially the second one ?) >>> I wish D had chosen @ or $ instead of !. >> Perhaps, but isn't it true that every language has slightly confusing >> stuff when first considered? Whitespace in python, < > for template >> stuff in C++, [] notation for buffers in cython (I've gotten used to >> it fine, and don't mind it now, but it did make me to do a few >> double-takes at first). What I like about D's syntax is that it's >> easy to pick out !( ) as type arguments to a function, class, method, >> etc. and ( ) for the regular arguments. What I have found to be nice >> (and granted, I haven't done much D programming recently) is that type >> parameters and regular parameters follow similar syntax, by intention. >> I've found that templating stuff in D is an amazingly nice experience >> (C++ just stinks in comparison) as once I've got my head around what >> can and can't be done, the syntax is very intuitive. > > in fact, I am not really sure I am buying that this: > auto foo = Foo !(T1, T2, T3)(arg1, arg2); > > is really cleaner/clearer than: > auto foo = Foo (T1, T2, T3)(arg1, arg2); > > that is what we do in python/PyROOT (the main python-to-C/C++ module we use in > my field (High Energy Physics)): > import ROOT > cls =ROOT.std.vector(double) # instantiates std::vector > data = cls() > assert hasattr(data, 'size') > > my_data = ROOT.std.vector('MyNS::MyClass')() > my_data.push_back(...) > assert isinstance(my_data[0], ROOT.MyNS.MyClass) > > ROOT.std.vector being just a function returning classes, by type or by (C++) > class name... That looks a bit like partial(), and similar to what I 'proposed' earlier with the keyword arguments. It might make sense to stay in Python-syntax completely here. Maybe there are more ideas on how to achieve this nicely. Stefan From stefan_ml at behnel.de Tue May 5 21:23:57 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 May 2009 21:23:57 +0200 Subject: [Cython] template types In-Reply-To: <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> Message-ID: <4A00924D.9070202@behnel.de> Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Imagine you could write >> >> cdef class MyType(object): >> # type T gets defined in the class scope to make the parser happy >> cdef T* my_attribute >> def __init__(self, T value): >> self.my_attribute = malloc(sizeof(T)*1000) >> self.my_attribute[100] = value >> def T do_work(self, T input): >> return input + self.my_attribute[100] >> >> o = MyType(20) # first occurrence creates/registers the type >> o.do_work(18) >> >> ctypedef MyType MyIntType # reuses the type and provides a name >> >> cdef MyIntType ot = MyIntType(25) >> print ot.do_work(19) >> >> ctypedef MyType MyDType # creates and defines a new type >> >> I'm not sure if a common base class would make sense here, something that >> would just implement everything that doesn't depend on T. But maybe that's >> overkill already. And most stuff *will* > > - Using < and > makes the parser more complicated, while [] must already > be parsed. (in C++ there's the mess of having to write "vector >> ", note the space, because there would be some problem with the >> > operator and the parser otherwise). What about avoiding this problem all-together? We could require a ctypedef in the beginning, which would alleviate the need for a syntax that works well in expressions. Stefan From dagss at student.matnat.uio.no Tue May 5 21:59:19 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 05 May 2009 21:59:19 +0200 Subject: [Cython] template types In-Reply-To: <4A00924D.9070202@behnel.de> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> Message-ID: <4A009A97.50608@student.matnat.uio.no> Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> Stefan Behnel wrote: >>> Imagine you could write >>> >>> cdef class MyType(object): >>> # type T gets defined in the class scope to make the parser happy >>> cdef T* my_attribute >>> def __init__(self, T value): >>> self.my_attribute = malloc(sizeof(T)*1000) >>> self.my_attribute[100] = value >>> def T do_work(self, T input): >>> return input + self.my_attribute[100] >>> >>> o = MyType(20) # first occurrence creates/registers the type >>> o.do_work(18) >>> >>> ctypedef MyType MyIntType # reuses the type and provides a name >>> >>> cdef MyIntType ot = MyIntType(25) >>> print ot.do_work(19) >>> >>> ctypedef MyType MyDType # creates and defines a new type >>> >>> I'm not sure if a common base class would make sense here, something that >>> would just implement everything that doesn't depend on T. But maybe that's >>> overkill already. And most stuff *will* >> - Using < and > makes the parser more complicated, while [] must already >> be parsed. (in C++ there's the mess of having to write "vector >>> ", note the space, because there would be some problem with the >> >> operator and the parser otherwise). > > What about avoiding this problem all-together? We could require a ctypedef > in the beginning, which would alleviate the need for a syntax that works > well in expressions. I would feel it was inconvenient to use a seperate line to define IntVector, then remember what name one gave it etc., rather than just express what one wants. Templates quickly becomes second-nature in C++ and Java and this seems to work against that. OTOH, this brings up another interesting question: What happens with exporting templates to Python-land? If there was indeed something very like typedef, but which also made the templated type available Python-side... I guess IntVector = Vector would currently achieve that without any further magic though. With [] or () or somesuch one could export some wrapper type for "Vector" and look up the Vector[cython.int] type from Python-space, however one must know compile-time which templates to instantiate for Python access anyway. The template types to always instantiate and export to Python could perhaps be embedded in the template declaration syntax. (I'd say that's a valid usecase eventually, but of course not something to support at first. But it /does/ impact the chosen syntax quite heavily as <> is ruled straight out.) -- Dag Sverre From dalcinl at gmail.com Tue May 5 22:41:55 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 5 May 2009 17:41:55 -0300 Subject: [Cython] template types In-Reply-To: <4A009A97.50608@student.matnat.uio.no> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> Message-ID: On Tue, May 5, 2009 at 4:59 PM, Dag Sverre Seljebotn wrote: > > OTOH, this brings up another interesting question: What happens with > exporting templates to Python-land? > SWIG supports templates and exposing templates to Python side... but having to choose a name for the "instantiation" always bothered me... At some point, I've started to use some hackery via a registry and a custom class abusing of __getitem__() to support on Python side: myinst = MyClass[sometype](args) At first, I was tempted to use "MyClass(sometype)(args)", but almost immediately I ruled it out because that was too much confusing. So, I'm +1 on "[]" for templates... > > (I'd say that's a valid usecase eventually, but of course not something > to support at first. > Why not? IMHO, that would be easiest part to implement from the whole beast (assuming that a registry and a companion type implementing dispatching via __getitem__() would be enough). > But it /does/ impact the chosen syntax quite > heavily as <> is ruled straight out.) > Indeed. That's the reason I support "[]" for templates over the C++/D way. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Tue May 5 22:54:02 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 5 May 2009 17:54:02 -0300 Subject: [Cython] Storing many, many instances In-Reply-To: <4A0028BE.9030903@behnel.de> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <8f5db410ca5009dd2aec60dce437cd23.squirrel@webmail.uio.no> <7c940a5d130e97a4d739efc6f3343bff.squirrel@webmail.uio.no> <4A0028BE.9030903@behnel.de> Message-ID: On Tue, May 5, 2009 at 8:53 AM, Stefan Behnel wrote: > > Dag Sverre Seljebotn wrote: >>> St?fan van der Walt wrote: >>>> I think I may have to switch to structs instead of classes, but then I >>>> no longer have convenient methods or array members. ?Maybe I should >>>> rewrite my class to have many static methods that operate on a struct, >>>> and then collect the structs as data instead of the full instances. >>> Technically it would not be too difficult to allow >>> >>> cdef struct MyStruct: >>> ? ? void foo(self): >>> ? ? ? ? # self is pure MyStruct, perhaps stack-allocated >>> > > I wouldn't mind. We have a lot of convenience stuff in structs already > (e.g. dict coercion and keyword arguments), so adding methods with the > above semantics wouldn't hurt. > I agree... > It's not rewriting C++ either, as long as it makes sense in Cython as a > language. > But them someone will ask for polimorphism, and then the struct instances could have a pointer to a struct-specific vtable, and then you are more or less rewriting C++. And that would not bother me, currently cdef methods do work as a sort C++, right?. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Tue May 5 23:11:10 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 May 2009 23:11:10 +0200 Subject: [Cython] template types In-Reply-To: <4A009A97.50608@student.matnat.uio.no> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF4F68.2000506@student.matnat.uio.no> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> Message-ID: <4A00AB6E.90909@behnel.de> Dag Sverre Seljebotn wrote: > OTOH, this brings up another interesting question: What happens with > exporting templates to Python-land? > > If there was indeed something very like typedef, but which also made the > templated type available Python-side... > > I guess > > IntVector = Vector > > would currently achieve that without any further magic though. Sure. > With [] or () or somesuch one could export some wrapper type for > "Vector" and look up the Vector[cython.int] type from Python-space, > however one must know compile-time which templates to instantiate for > Python access anyway. The template types to always instantiate and > export to Python could perhaps be embedded in the template declaration > syntax. Why? If you want to provide a type to Python space, you can just instantiate it somewhere in your program, e.g. by adding it to a name->type mapping. No magic whatsoever involved. > (I'd say that's a valid usecase eventually, but of course not something > to support at first. But it /does/ impact the chosen syntax quite > heavily as <> is ruled straight out.) The syntax would be Cython-only, no need to move it into Python. Stefan From stefan_ml at behnel.de Tue May 5 23:18:59 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 May 2009 23:18:59 +0200 Subject: [Cython] template types In-Reply-To: References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> Message-ID: <4A00AD43.1050709@behnel.de> Lisandro Dalcin wrote: > At some point, I've started to use some hackery via a registry and a > custom class abusing of __getitem__() to support on Python side: > > myinst = MyClass[sometype](args) Actually, now that I see this - it actually makes sense to think of a parametrised type as a meta-type that maps types to types, so a mapping syntax makes sense here. If you have more than one type parameter, it'd look like this: myinst = MyClass[(sometype, someothertype)](*args) and the concrete type would basically be 'looked up' at compile time. It doesn't directly map to an 'obvious' declaration syntax, but I guess cdef MyType(object) [(sometype, someothertype)]: ... works well enough. So I'm +1 on this, too. Stefan From stefan_ml at behnel.de Tue May 5 23:28:17 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 May 2009 23:28:17 +0200 Subject: [Cython] template types In-Reply-To: <4A00AD43.1050709@behnel.de> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> Message-ID: <4A00AF71.7010103@behnel.de> Stefan Behnel schrieb: > Lisandro Dalcin wrote: >> At some point, I've started to use some hackery via a registry and a >> custom class abusing of __getitem__() to support on Python side: >> >> myinst = MyClass[sometype](args) > > Actually, now that I see this - it actually makes sense to think of a > parametrised type as a meta-type that maps types to types, so a mapping > syntax makes sense here. If you have more than one type parameter, it'd > look like this: > > myinst = MyClass[(sometype, someothertype)](*args) > > and the concrete type would basically be 'looked up' at compile time. > > It doesn't directly map to an 'obvious' declaration syntax, but I guess > > cdef MyType(object) [(sometype, someothertype)]: > ... > > works well enough. I meant cdef MyType(object) [(T,V)]: here, although I now noticed that there is already the "private type" syntax: cdef public class _Document [ type LxmlDocumentType, object LxmlDocument ]: ... so this won't work straight away either... Stefan From dalcinl at gmail.com Tue May 5 23:28:30 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 5 May 2009 18:28:30 -0300 Subject: [Cython] template types In-Reply-To: <4A00AD43.1050709@behnel.de> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> Message-ID: On Tue, May 5, 2009 at 6:18 PM, Stefan Behnel wrote: > > Lisandro Dalcin wrote: >> At some point, I've started to use some hackery via a registry and a >> custom class abusing of __getitem__() to support on Python side: >> >> myinst = MyClass[sometype](args) > > Actually, now that I see this - it actually makes sense to think of a > parametrised type as a meta-type that maps types to types, so a mapping > syntax makes sense here. Nice! You just added another point for me to like "[]", the mapping concept fits well IMHO... > If you have more than one type parameter, it'd > look like this: > > ? ?myinst = MyClass[(sometype, someothertype)](*args) > Why the tuple? This should work just fine: myinst = MyClass[sometype, someothertype](*args) thoug the MyClass.__getitem__() will receive a tuple actually... (sugar added to Python slice syntax long ago following request of Numeric) > and the concrete type would basically be 'looked up' at compile time. > > It doesn't directly map to an 'obvious' declaration syntax, but I guess > > ? ?cdef MyType(object) [(sometype, someothertype)]: > ? ? ?... I would prefer: cdef MyType[sometype, someothertype](object): -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Tue May 5 23:33:16 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 May 2009 23:33:16 +0200 Subject: [Cython] template types In-Reply-To: References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> Message-ID: <4A00B09C.4030606@behnel.de> Lisandro Dalcin wrote: > On Tue, May 5, 2009 at 6:18 PM, Stefan Behnel wrote: >> Lisandro Dalcin wrote: >>> At some point, I've started to use some hackery via a registry and a >>> custom class abusing of __getitem__() to support on Python side: >>> >>> myinst = MyClass[sometype](args) > >> Actually, now that I see this - it actually makes sense to think of a >> parametrised type as a meta-type that maps types to types, so a mapping >> syntax makes sense here. > > Nice! You just added another point for me to like "[]", the mapping > concept fits well IMHO... > >> If you have more than one type parameter, it'd >> look like this: >> >> myinst = MyClass[(sometype, someothertype)](*args) >> > > Why the tuple? This should work just fine: > > myinst = MyClass[sometype, someothertype](*args) > > thoug the MyClass.__getitem__() will receive a tuple actually... > (sugar added to Python slice syntax long ago following request of > Numeric) > > >> and the concrete type would basically be 'looked up' at compile time. >> >> It doesn't directly map to an 'obvious' declaration syntax, but I guess >> >> cdef MyType(object) [(sometype, someothertype)]: >> ... > > I would prefer: > > cdef MyType[sometype, someothertype](object): ... which also wouldn't be ambiguous with public types. Looks CEP-able to me. Stefan From dalcinl at gmail.com Tue May 5 23:33:56 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 5 May 2009 18:33:56 -0300 Subject: [Cython] template types In-Reply-To: <4A00AF71.7010103@behnel.de> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> <4A00AF71.7010103@behnel.de> Message-ID: On Tue, May 5, 2009 at 6:28 PM, Stefan Behnel wrote: > I meant > > ? ?cdef MyType(object) [(T,V)]: > > here, although I now noticed that there is already the "private type" syntax: > > cdef public class _Document [ type LxmlDocumentType, object LxmlDocument ]: > ? ?... > > so this won't work straight away either... > But this would work anyway, right? Perhaps you could even support {} for the [type ..., object ...] part... cdef public class MyBaseClass[T,V] [type MyBaseClass_Type, object MyBaseClassObject ]: cdef public class MyClass[T,V](MyBaseClass)[ type MyClass_Type, object MyClassObject ]: -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Wed May 6 00:04:51 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 5 May 2009 19:04:51 -0300 Subject: [Cython] main-embedding problem In-Reply-To: <7B8637C0-61E3-40D6-8BE4-04E635F78682@math.washington.edu> References: <49FF5325.7020705@behnel.de> <7B8637C0-61E3-40D6-8BE4-04E635F78682@math.washington.edu> Message-ID: On Mon, May 4, 2009 at 7:10 PM, Robert Bradshaw wrote: >> Unfortunatelly, I have to agree, for different reasons... As an >> starter, the current code does not handle Py3K on Windows (because of >> the new wchar_t based API) > > This should be sufficient, right? > > http://hg.cython.org/cython-devel/rev/2e35d5caac86 > No, unfortunately it is not enough... I didn't tell you the whole story... Py3K has a wchar_t-based API for all platforms. On Windows, the fix is easy, you just use wmain(). On POSIX, not so easy... you have to use char-based, traditional main(), and use mbrtowc() to convent all the argument one by one... IMHO, core CPython is doing it wrong here... See the nightmare in py3k/Modules/python.c ... IMHO, Python should provide a both char-based and wchar_t-based API's, and internally implement the char-based ones as wrappers, managing the conversion char->wchat_t and then call the wchar_t-based API's. If this is not done, any one trying to embed python have to start converting arguments (and managing the tmp mem blocks), and have to cope with the apparent brokenness of mbrtowc() on some platforms... I've already had to implement all that crap in mpi4py, see http://code.google.com/p/mpi4py/source/browse/trunk/src/python.c (borrow code from Py3_Main_GetArgs() if you like it). Fortunately, that file is not frequently required for end-users, but I'm not even sure if it works on the many Unix-like systems out-there (current supercomputer systems hardly have python3 available for end-users) I do not have the energy to go to Python-Dev to discuss all this... Do any of you have it :-)? Stefan? Greg? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Wed May 6 07:15:42 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 06 May 2009 07:15:42 +0200 Subject: [Cython] template types In-Reply-To: References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> <4A00AF71.7010103@behnel.de> Message-ID: <4A011CFE.7070006@behnel.de> Lisandro Dalcin wrote: > On Tue, May 5, 2009 at 6:28 PM, Stefan Behnel wrote: >> I meant >> >> cdef MyType(object) [(T,V)]: >> >> here, although I now noticed that there is already the "private type" syntax: >> >> cdef public class _Document [ type LxmlDocumentType, object LxmlDocument ]: >> ... >> >> so this won't work straight away either... >> > > But this would work anyway, right? Not that easily. You could have both [] parts next to each other, or just any one of them. So the parser would fail if you decided to call your template variable "type", which isn't that a bad name in this context. > Perhaps you could even support {} > for the [type ..., object ...] part... > > cdef public class MyBaseClass[T,V] [type MyBaseClass_Type, object > MyBaseClassObject ]: > > cdef public class MyClass[T,V](MyBaseClass)[ type MyClass_Type, object > MyClassObject ]: Well, if we get to change *that* syntax (which I'd do straight away if it wasn't for compatibility - it really hurt my eyes the first time I saw it), then this seems a lot better: cdef public class MyClass[T,V](MyBaseClass, public_ctype="MyClass_Type", public_cobject="MyClassObject"): The parser can easily know and check all valid keywords here, so there'd be no change to the rest of the compiler. And using strings as arguments makes it a lot clearer (at least to me) that this is only configuration that is supposed to end up in the generated C code, and not meant for real use in the program. Stefan From dagss at student.matnat.uio.no Wed May 6 07:17:54 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 6 May 2009 07:17:54 +0200 (CEST) Subject: [Cython] template types In-Reply-To: <4A00AD43.1050709@behnel.de> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> Message-ID: <899fcc678c0606bb152339924fa2c459.squirrel@webmail.uio.no> Stefan Behnel wrote: > Lisandro Dalcin wrote: >> At some point, I've started to use some hackery via a registry and a >> custom class abusing of __getitem__() to support on Python side: >> >> myinst = MyClass[sometype](args) > > Actually, now that I see this - it actually makes sense to think of a > parametrised type as a meta-type that maps types to types, so a mapping > syntax makes sense here. If you have more than one type parameter, it'd > look like this: > > myinst = MyClass[(sometype, someothertype)](*args) Well, in Python there's no way to seperate that from myinst = MyClass[sometype, someothertype](*args) which is syntax candy for the same. Of course we can in Cython, but why? Dag Sverre From stefan_ml at behnel.de Wed May 6 07:21:21 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 06 May 2009 07:21:21 +0200 Subject: [Cython] template types In-Reply-To: <899fcc678c0606bb152339924fa2c459.squirrel@webmail.uio.no> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <9457e7c80905041433v2b15b615lecb9fd27a2004ef8@mail.gmail.com> <46099E5F-C136-478D-8C00-C33784F3DB54@math.washington.edu> <49FF8CC0.7020907@canterbury.ac.nz> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> <899fcc678c0606bb152339924fa2c459.squirrel@webmail.uio.no> Message-ID: <4A011E51.5080609@behnel.de> Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Lisandro Dalcin wrote: >>> At some point, I've started to use some hackery via a registry and a >>> custom class abusing of __getitem__() to support on Python side: >>> >>> myinst = MyClass[sometype](args) >> Actually, now that I see this - it actually makes sense to think of a >> parametrised type as a meta-type that maps types to types, so a mapping >> syntax makes sense here. If you have more than one type parameter, it'd >> look like this: >> >> myinst = MyClass[(sometype, someothertype)](*args) > > Well, in Python there's no way to seperate that from > > myinst = MyClass[sometype, someothertype](*args) > > which is syntax candy for the same. Fine with me. Stefan From dagss at student.matnat.uio.no Wed May 6 07:23:39 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 6 May 2009 07:23:39 +0200 (CEST) Subject: [Cython] template types In-Reply-To: <4A011CFE.7070006@behnel.de> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> <4A00AF71.7010103@behnel.de> <4A011CFE.7070006@behnel.de> Message-ID: <3439af420a85daac11ba6238c254c18f.squirrel@webmail.uio.no> Stefan Behnel wrote: > Lisandro Dalcin wrote: >> On Tue, May 5, 2009 at 6:28 PM, Stefan Behnel wrote: >>> I meant >>> >>> cdef MyType(object) [(T,V)]: >>> >>> here, although I now noticed that there is already the "private type" >>> syntax: >>> >>> cdef public class _Document [ type LxmlDocumentType, object >>> LxmlDocument ]: >>> ... >>> >>> so this won't work straight away either... >>> >> >> But this would work anyway, right? > > Not that easily. You could have both [] parts next to each other, or just > any one of them. So the parser would fail if you decided to call your > template variable "type", which isn't that a bad name in this context. How about cdef class A: T, V = cython.template_args(type, type) ? (The reason for the arguments: In C++ you can have stuff like ints in template arguments as well, might as well leave the possibility open even if only passing type is allowed for now). This even puts T and V in the proper scope for @cython.locals declarations, e.g. this passes through (and could even be made to do the checks) in pure Python: @cython.cdef class A: T, V = template_args(type, type) @cython.locals(a=T, b=v) def foo(self, a, b): ... Dag Sverre From robertwb at math.washington.edu Wed May 6 09:02:52 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 00:02:52 -0700 Subject: [Cython] template types In-Reply-To: <3439af420a85daac11ba6238c254c18f.squirrel@webmail.uio.no> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> <4A00AF71.7010103@behnel.de> <4A011CFE.7070006@behnel.de> <3439af420a85daac11ba6238c254c18f.squirrel@webmail.uio.no> Message-ID: <8B75D598-FF6D-4986-8E3D-0167BA5CE778@math.washington.edu> On May 5, 2009, at 10:23 PM, Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Lisandro Dalcin wrote: >>> On Tue, May 5, 2009 at 6:28 PM, Stefan Behnel wrote: >>>> I meant >>>> >>>> cdef MyType(object) [(T,V)]: >>>> >>>> here, although I now noticed that there is already the "private >>>> type" >>>> syntax: >>>> >>>> cdef public class _Document [ type LxmlDocumentType, object >>>> LxmlDocument ]: >>>> ... >>>> >>>> so this won't work straight away either... >>>> >>> >>> But this would work anyway, right? >> >> Not that easily. You could have both [] parts next to each other, >> or just >> any one of them. So the parser would fail if you decided to call your >> template variable "type", which isn't that a bad name in this >> context. > > How about > > cdef class A: > T, V = cython.template_args(type, type) > > ? I prefer the MyType[T,V] syntax. It encapsulates the "meta-type" idea, and also the same syntax can be used both in declaring the type and instantiating the type. > (The reason for the arguments: In C++ you can have stuff like ints in > template arguments as well, might as well leave the possibility > open even > if only passing type is allowed for now). > > This even puts T and V in the proper scope for @cython.locals > declarations, e.g. this passes through (and could even be made to > do the > checks) in pure Python: > > @cython.cdef > class A: > T, V = template_args(type, type) > @cython.locals(a=T, b=v) > def foo(self, a, b): ... I'd rather see the pure Python mode decorate the class than have a magic method, but I don't think decorators get applied soon enough to inject variables into the scope. - Robert From stefan_ml at behnel.de Wed May 6 09:40:51 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 6 May 2009 09:40:51 +0200 (CEST) Subject: [Cython] template types In-Reply-To: <8B75D598-FF6D-4986-8E3D-0167BA5CE778@math.washington.edu> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> <4A00AF71.7010103@behnel.de> <4A011CFE.7070006@behnel.de> <3439af420a85daac11ba6238c254c18f.squirrel@webmail.uio.no> <8B75D598-FF6D-4986-8E3D-0167BA5CE778@math.washington.edu> Message-ID: <1409138872f9c0172f9467e4d3380dfe.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Robert Bradshaw wrote: > On May 5, 2009, at 10:23 PM, Dag Sverre Seljebotn wrote: >> How about >> >> cdef class A: >> T, V = cython.template_args(type, type) > > I prefer the MyType[T,V] syntax. It encapsulates the "meta-type" > idea, and also the same syntax can be used both in declaring the type > and instantiating the type. +1 >> (The reason for the arguments: In C++ you can have stuff like ints in >> template arguments as well, might as well leave the possibility >> open even >> if only passing type is allowed for now). >> >> This even puts T and V in the proper scope for @cython.locals >> declarations, e.g. this passes through (and could even be made to >> do the checks) in pure Python: >> >> @cython.cdef >> class A: >> T, V = template_args(type, type) >> @cython.locals(a=T, b=v) >> def foo(self, a, b): ... > > I'd rather see the pure Python mode decorate the class than have a > magic method, but I don't think decorators get applied soon enough to > inject variables into the scope. I wouldn't care so much about that for now. Whatever declaration we choose, it's easily added when the feature itself is there, and I think parametrised types are such an advanced feature that users can well wait for it to become available in pure Python mode. The priorities should be to make the feature simple and well readable in Cython code, and then 'somehow' available in Python code. Stefan From dagss at student.matnat.uio.no Wed May 6 09:42:53 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 May 2009 09:42:53 +0200 Subject: [Cython] template types In-Reply-To: <8B75D598-FF6D-4986-8E3D-0167BA5CE778@math.washington.edu> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> <4A00AF71.7010103@behnel.de> <4A011CFE.7070006@behnel.de> <3439af420a85daac11ba6238c254c18f.squirrel@webmail.uio.no> <8B75D598-FF6D-4986-8E3D-0167BA5CE778@math.washington.edu> Message-ID: <4A013F7D.9070101@student.matnat.uio.no> Robert Bradshaw wrote: > On May 5, 2009, at 10:23 PM, Dag Sverre Seljebotn wrote: > >> Stefan Behnel wrote: >>> Lisandro Dalcin wrote: >>>> On Tue, May 5, 2009 at 6:28 PM, Stefan Behnel wrote: >>>>> I meant >>>>> >>>>> cdef MyType(object) [(T,V)]: >>>>> >>>>> here, although I now noticed that there is already the "private >>>>> type" >>>>> syntax: >>>>> >>>>> cdef public class _Document [ type LxmlDocumentType, object >>>>> LxmlDocument ]: >>>>> ... >>>>> >>>>> so this won't work straight away either... >>>>> >>>> But this would work anyway, right? >>> Not that easily. You could have both [] parts next to each other, >>> or just >>> any one of them. So the parser would fail if you decided to call your >>> template variable "type", which isn't that a bad name in this >>> context. >> How about >> >> cdef class A: >> T, V = cython.template_args(type, type) >> >> ? > > I prefer the MyType[T,V] syntax. It encapsulates the "meta-type" > idea, and also the same syntax can be used both in declaring the type > and instantiating the type. > >> (The reason for the arguments: In C++ you can have stuff like ints in >> template arguments as well, might as well leave the possibility >> open even >> if only passing type is allowed for now). >> >> This even puts T and V in the proper scope for @cython.locals >> declarations, e.g. this passes through (and could even be made to >> do the >> checks) in pure Python: >> >> @cython.cdef >> class A: >> T, V = template_args(type, type) >> @cython.locals(a=T, b=v) >> def foo(self, a, b): ... > > I'd rather see the pure Python mode decorate the class than have a > magic method, but I don't think decorators get applied soon enough to > inject variables into the scope. That is correct. The whole advantage was that T and V gets injected in the scope in the right place. A better example is once we start to support argument decorators: class A: T, V = template_args(type, type) def foo(self, a: T, b: V): ... This currently passes just fine in Python 3. We could even allow referring to T and V as self.T and self.V within functions (in addition or as the only way -- it is more Pythonic as it doesn't invent a template-only scope). -- Dag Sverre From dagss at student.matnat.uio.no Wed May 6 09:47:15 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 May 2009 09:47:15 +0200 Subject: [Cython] template types In-Reply-To: <1409138872f9c0172f9467e4d3380dfe.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> <4A00AF71.7010103@behnel.de> <4A011CFE.7070006@behnel.de> <3439af420a85daac11ba6238c254c18f.squirrel@webmail.uio.no> <8B75D598-FF6D-4986-8E3D-0167BA5CE778@math.washington.edu> <1409138872f9c0172f9467e4d3380dfe.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <4A014083.1030500@student.matnat.uio.no> Stefan Behnel wrote: > I wouldn't care so much about that for now. Whatever declaration we > choose, it's easily added when the feature itself is there, and I think > parametrised types are such an advanced feature that users can well wait > for it to become available in pure Python mode. > > The priorities should be to make the feature simple and well readable in > Cython code, and then 'somehow' available in Python code. I disagree with this principle. (I guess it might be a matter of taste in the end, so I'll just state my opinion and try not to drag this on for too long.) In perhaps about a year, with Python 3 argument decorators (def foo(x: int) -> float) and a bit of type inference, "pure Python mode" could actually become as convenient as Cython syntax and the preferred syntax we'd teach new users in my opinion. And anyway, I'm in favor of avoiding to have to invent two kinds of syntax for new features. Especially when the benefits are so small! (One uses a template a lot more than writing them.) It's just a mess to have to tell users "this one is the Cython syntax but you can also use this one for Python compatability" etc.; that problem is there to stay but I don't see the sense of keeping adding to it. -- Dag Sverre From robertwb at math.washington.edu Wed May 6 09:57:08 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 00:57:08 -0700 Subject: [Cython] template types In-Reply-To: <4A014083.1030500@student.matnat.uio.no> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> <4A00AF71.7010103@behnel.de> <4A011CFE.7070006@behnel.de> <3439af420a85daac11ba6238c254c18f.squirrel@webmail.uio.no> <8B75D598-FF6D-4986-8E3D-0167BA5CE778@math.washington.edu> <1409138872f9c0172f9467e4d3380dfe.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A014083.1030500@student.matnat.uio.no> Message-ID: <4D8338A7-3164-423A-89F4-A52FDCCDDF22@math.washington.edu> On May 6, 2009, at 12:47 AM, Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> I wouldn't care so much about that for now. Whatever declaration we >> choose, it's easily added when the feature itself is there, and I >> think >> parametrised types are such an advanced feature that users can >> well wait >> for it to become available in pure Python mode. >> >> The priorities should be to make the feature simple and well >> readable in >> Cython code, and then 'somehow' available in Python code. > > I disagree with this principle. (I guess it might be a matter of taste > in the end, so I'll just state my opinion and try not to drag this on > for too long.) > > In perhaps about a year, with Python 3 argument decorators (def foo(x: > int) -> float) and a bit of type inference, "pure Python mode" could > actually become as convenient as Cython syntax and the preferred > syntax > we'd teach new users in my opinion. I think "cdef int i,j,k" is in always going to be easier than i = cython.declare(int) j = cython.declare(int) k = cython.declare(int) but in general I agree with this philosophy. > And anyway, I'm in favor of avoiding to have to invent two kinds of > syntax for new features. Especially when the benefits are so small! > (One > uses a template a lot more than writing them.) I'm guessing you mean you're *not* in favor? > It's just a mess to have to tell users "this one is the Cython syntax > but you can also use this one for Python compatability" etc.; that > problem is there to stay but I don't see the sense of keeping > adding to it. Only if there's a non-pure Python syntax that's a lot cleaner (which I think the class Foo[T] is). - Robert From brett.calcott at gmail.com Wed May 6 10:09:44 2009 From: brett.calcott at gmail.com (Brett Calcott) Date: Wed, 6 May 2009 18:09:44 +1000 Subject: [Cython] Passing numpy buffers into a cdef function Message-ID: Hi, I am very happy with the cython based extensions I have written to operate on numpy arrays. I'd like to refactor some code that is in these functions, putting it into a common cdef function that they can each call. Something like this: cdef common_stuff(np.ndarray[double, ndim=2] elements): # Tight loop here ... def fun1(np.ndarray[double, ndim=2] elements): # Do something ... common_stuff(elements) def fun2(np.ndarray[double, ndim=2] elements): # Do something different ... common_stuff(elements) But I get an error on compilation, that the parser "Expected ']'" where ndim is defined. What is the right way to do this? Thanks Brett From dagss at student.matnat.uio.no Wed May 6 10:11:55 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 May 2009 10:11:55 +0200 Subject: [Cython] template types In-Reply-To: <4D8338A7-3164-423A-89F4-A52FDCCDDF22@math.washington.edu> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> <4A00AF71.7010103@behnel.de> <4A011CFE.7070006@behnel.de> <3439af420a85daac11ba6238c254c18f.squirrel@webmail.uio.no> <8B75D598-FF6D-4986-8E3D-0167BA5CE778@math.washington.edu> <1409138872f9c0172f9467e4d3380dfe.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A014083.1030500@student.matnat.uio.no> <4D8338A7-3164-423A-89F4-A52FDCCDDF22@math.washington.edu> Message-ID: <4A01464B.7060506@student.matnat.uio.no> Robert Bradshaw wrote: > On May 6, 2009, at 12:47 AM, Dag Sverre Seljebotn wrote: > >> Stefan Behnel wrote: >>> I wouldn't care so much about that for now. Whatever declaration we >>> choose, it's easily added when the feature itself is there, and I >>> think >>> parametrised types are such an advanced feature that users can >>> well wait >>> for it to become available in pure Python mode. >>> >>> The priorities should be to make the feature simple and well >>> readable in >>> Cython code, and then 'somehow' available in Python code. >> I disagree with this principle. (I guess it might be a matter of taste >> in the end, so I'll just state my opinion and try not to drag this on >> for too long.) >> >> In perhaps about a year, with Python 3 argument decorators (def foo(x: >> int) -> float) and a bit of type inference, "pure Python mode" could >> actually become as convenient as Cython syntax and the preferred >> syntax >> we'd teach new users in my opinion. > > I think "cdef int i,j,k" is in always going to be easier than > > i = cython.declare(int) > j = cython.declare(int) > k = cython.declare(int) > > but in general I agree with this philosophy. Yes, like I said, type inference would be needed. So perhaps more than a year. But type inference wouldn't need to be that great, just at a minimum pick up manual hints like i = cython.int(func_returning_any_type()) and if i is never assigned to anything than known cython.int instances, then type it accordingly. I've also been thinking about doing things like def foo(x: int, y: int, _, localvar1: float, localvar2: float): ... but it is too ugly :-( In Py3 we could also abuse the new def f(x, *, kwonly): ... syntax in the same way, but it already has a use so probvably not...but I'm still playing with variations on the theme. > Only if there's a non-pure Python syntax that's a lot cleaner (which > I think the class Foo[T] is). Yes, I was basing what I said on writing a template to be much rarer (and done by a lot fewer users) and so not "needing" a clean syntax; I agree that it is cleaner. (I'm still concerned about adding Yet Another Scope Rule (I haven't even learned the current ones fully yet!) for this, but mostly on the philosophical level, in practice it would work as the user expects.) -- Dag Sverre From dagss at student.matnat.uio.no Wed May 6 10:13:24 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 May 2009 10:13:24 +0200 Subject: [Cython] Passing numpy buffers into a cdef function In-Reply-To: References: Message-ID: <4A0146A4.106@student.matnat.uio.no> Brett Calcott wrote: > Hi, > > I am very happy with the cython based extensions I have written to > operate on numpy arrays. I'd like to refactor some code that is in these > functions, putting it into a common cdef function that they can each > call. Something like this: > > cdef common_stuff(np.ndarray[double, ndim=2] elements): > # Tight loop here > ... > > def fun1(np.ndarray[double, ndim=2] elements): > # Do something > ... > common_stuff(elements) > > def fun2(np.ndarray[double, ndim=2] elements): > # Do something different > ... > common_stuff(elements) > > > But I get an error on compilation, that the parser "Expected ']'" where > ndim is defined. > > What is the right way to do this? You can't. You have to do cdef func(elements_) cdef np.ndarray[double, ndim=2] elements = elements_ and there's a certain overhead. This will likely be fixed during summer. http://trac.cython.org/cython_trac/ticket/177 -- Dag Sverre From brett.calcott at gmail.com Wed May 6 10:21:11 2009 From: brett.calcott at gmail.com (Brett Calcott) Date: Wed, 6 May 2009 18:21:11 +1000 Subject: [Cython] Passing numpy buffers into a cdef function In-Reply-To: <4A0146A4.106@student.matnat.uio.no> References: <4A0146A4.106@student.matnat.uio.no> Message-ID: 2009/5/6 Dag Sverre Seljebotn : > > You can't. You have to do > > cdef func(elements_) > ? ?cdef np.ndarray[double, ndim=2] elements = elements_ > > and there's a certain overhead. > > This will likely be fixed during summer. > > http://trac.cython.org/cython_trac/ticket/177 > Hi Dag, thanks for the info. I'll just stick with some ugly code for now then :) Cheers, Brett From dagss at student.matnat.uio.no Wed May 6 10:31:43 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 May 2009 10:31:43 +0200 Subject: [Cython] template types In-Reply-To: <8B75D598-FF6D-4986-8E3D-0167BA5CE778@math.washington.edu> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> <4A00AF71.7010103@behnel.de> <4A011CFE.7070006@behnel.de> <3439af420a85daac11ba6238c254c18f.squirrel@webmail.uio.no> <8B75D598-FF6D-4986-8E3D-0167BA5CE778@math.washington.edu> Message-ID: <4A014AEF.7030901@student.matnat.uio.no> Robert Bradshaw wrote: > On May 5, 2009, at 10:23 PM, Dag Sverre Seljebotn wrote: > >> Stefan Behnel wrote: >>> Lisandro Dalcin wrote: >>>> On Tue, May 5, 2009 at 6:28 PM, Stefan Behnel wrote: >>>>> I meant >>>>> >>>>> cdef MyType(object) [(T,V)]: >>>>> >>>>> here, although I now noticed that there is already the "private >>>>> type" >>>>> syntax: >>>>> >>>>> cdef public class _Document [ type LxmlDocumentType, object >>>>> LxmlDocument ]: >>>>> ... >>>>> >>>>> so this won't work straight away either... >>>>> >>>> But this would work anyway, right? >>> Not that easily. You could have both [] parts next to each other, >>> or just >>> any one of them. So the parser would fail if you decided to call your >>> template variable "type", which isn't that a bad name in this >>> context. >> How about >> >> cdef class A: >> T, V = cython.template_args(type, type) >> >> ? > > I prefer the MyType[T,V] syntax. It encapsulates the "meta-type" > idea, and also the same syntax can be used both in declaring the type > and instantiating the type. BTW will this discussion also be used for C++ templates, so that the consensus seem to be some_cpp_class[int] for template instantiation? -- Dag Sverre From robertwb at math.washington.edu Wed May 6 10:37:13 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 01:37:13 -0700 Subject: [Cython] template types In-Reply-To: <4A014AEF.7030901@student.matnat.uio.no> References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <4db580fd0905042026l7daf70f4u51feb2a8b7796e4f@mail.gmail.com> <49FFEFBE.6010800@behnel.de> <03bf16bb66619b7d44833a3ac76ea6d6.squirrel@webmail.uio.no> <4A00924D.9070202@behnel.de> <4A009A97.50608@student.matnat.uio.no> <4A00AD43.1050709@behnel.de> <4A00AF71.7010103@behnel.de> <4A011CFE.7070006@behnel.de> <3439af420a85daac11ba6238c254c18f.squirrel@webmail.uio.no> <8B75D598-FF6D-4986-8E3D-0167BA5CE778@math.washington.edu> <4A014AEF.7030901@student.matnat.uio.no> Message-ID: On May 6, 2009, at 1:31 AM, Dag Sverre Seljebotn wrote: > BTW will this discussion also be used for C++ templates, so that the > consensus seem to be > > some_cpp_class[int] > > for template instantiation? I think so. Speak now or forever hold your peace... :) - Robert From robertwb at math.washington.edu Wed May 6 11:02:15 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 02:02:15 -0700 Subject: [Cython] main-embedding problem In-Reply-To: References: <49FF5325.7020705@behnel.de> <7B8637C0-61E3-40D6-8BE4-04E635F78682@math.washington.edu> Message-ID: On May 5, 2009, at 3:04 PM, Lisandro Dalcin wrote: > On Mon, May 4, 2009 at 7:10 PM, Robert Bradshaw > wrote: >>> Unfortunatelly, I have to agree, for different reasons... As an >>> starter, the current code does not handle Py3K on Windows >>> (because of >>> the new wchar_t based API) >> >> This should be sufficient, right? >> >> http://hg.cython.org/cython-devel/rev/2e35d5caac86 >> > > No, unfortunately it is not enough... I didn't tell you the whole > story... Py3K has a wchar_t-based API for all platforms. On Windows, > the fix is easy, you just use wmain(). On POSIX, not so easy... you > have to use char-based, traditional main(), and use mbrtowc() to > convent all the argument one by one... > > IMHO, core CPython is doing it wrong here... See the nightmare in > py3k/Modules/python.c ... IMHO, Python should provide a both > char-based and wchar_t-based API's, and internally implement the > char-based ones as wrappers, managing the conversion char->wchat_t and > then call the wchar_t-based API's. > > If this is not done, any one trying to embed python have to start > converting arguments (and managing the tmp mem blocks), and have to > cope with the apparent brokenness of mbrtowc() on some platforms... > I've already had to implement all that crap in mpi4py, see > http://code.google.com/p/mpi4py/source/browse/trunk/src/python.c > (borrow code from Py3_Main_GetArgs() if you like it). Fortunately, > that file is not frequently required for end-users, but I'm not even > sure if it works on the many Unix-like systems out-there (current > supercomputer systems hardly have python3 available for end-users) > > I do not have the energy to go to Python-Dev to discuss all this... Do > any of you have it :-)? Stefan? Greg? :( That is just ugly. I am with you that CPython should be fixed, but don't have the time to look into hacking around this myself. - Robert From robertwb at math.washington.edu Wed May 6 11:08:50 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 02:08:50 -0700 Subject: [Cython] Storing many, many instances In-Reply-To: References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <8f5db410ca5009dd2aec60dce437cd23.squirrel@webmail.uio.no> <7c940a5d130e97a4d739efc6f3343bff.squirrel@webmail.uio.no> <4A0028BE.9030903@behnel.de> Message-ID: On May 5, 2009, at 1:54 PM, Lisandro Dalcin wrote: > On Tue, May 5, 2009 at 8:53 AM, Stefan Behnel > wrote: >> >> Dag Sverre Seljebotn wrote: >>>> St?fan van der Walt wrote: >>>>> I think I may have to switch to structs instead of classes, but >>>>> then I >>>>> no longer have convenient methods or array members. Maybe I >>>>> should >>>>> rewrite my class to have many static methods that operate on a >>>>> struct, >>>>> and then collect the structs as data instead of the full >>>>> instances. >>>> Technically it would not be too difficult to allow >>>> >>>> cdef struct MyStruct: >>>> void foo(self): >>>> # self is pure MyStruct, perhaps stack-allocated >>>> >> >> I wouldn't mind. We have a lot of convenience stuff in structs >> already >> (e.g. dict coercion and keyword arguments), so adding methods with >> the >> above semantics wouldn't hurt. > > I agree... I wouldn't be opposed... > >> It's not rewriting C++ either, as long as it makes sense in Cython >> as a >> language. >> > > But them someone will ask for polimorphism, and then the struct > instances could have a pointer to a struct-specific vtable, and then > you are more or less rewriting C++. And that would not bother me, > currently cdef methods do work as a sort C++, right?. Yeah, I'm not sure how far one wants to take this though. We should look into making instantiating (cdef) classes faster by default as well. - Robert From robertwb at math.washington.edu Wed May 6 11:10:28 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 02:10:28 -0700 Subject: [Cython] Problems with creating a package of extension modules In-Reply-To: <1241206225.49fb4dd161f7c@webmail.uni-jena.de> References: <1241198159.49fb2e4f76162@webmail.uni-jena.de> <922BECD5-7D73-46BC-A541-C76081EAC703@math.washington.edu> <1241206225.49fb4dd161f7c@webmail.uni-jena.de> Message-ID: <09C1EF75-876B-4775-A8AB-552ABCF9882D@math.washington.edu> On May 1, 2009, at 12:30 PM, Simon King wrote: > Dear Robert, > > Zitat von Robert Bradshaw : >> Hmm... could you point me to that bit of documentation? I think it's >> wrong (or at least not implemented as explained), we do Extension >> ("package.module",...) all the time in sage and that seems to mix >> well with pickling. > > Admittedly it is not Cython-documentation, but Python: > At http://docs.python.org/distutils/setupscript.html, they say: > > --------------------- > If you have a number of extensions all in the same package (or all > under the > same base package), use the ext_package keyword argument to setup > (). For > example, > > setup(..., > ext_package='pkg', > ext_modules=[Extension('foo', ['foo.c']), > Extension('subpkg.bar', ['bar.c'])], > ) > > will compile foo.c to the extension pkg.foo, and bar.c to > pkg.subpkg.bar. To wrap this thread up, it turns out that the ext_package option is incompatible with using .pxd files. Without the ext_package it works as expected (One complication--when trying stuff like this out, one has to make sure to delete the .c file as well as the build directory.) - Robert From robertwb at math.washington.edu Wed May 6 11:39:06 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 02:39:06 -0700 Subject: [Cython] New idea for C++ stack allocation In-Reply-To: <49F6A8E5.2040006@student.matnat.uio.no> References: <49F6A8E5.2040006@student.matnat.uio.no> Message-ID: <759D34EC-9F78-4E80-8818-987F7551AA07@math.washington.edu> On Apr 27, 2009, at 11:57 PM, Dag Sverre Seljebotn wrote: > I got another idea for how using C++ objects without no-arg > constructors > could work. It means enforcing C++ variable scoping through compile- > time > errors, but keeping Python syntax. I think for our first pass at least, we should allocate everything on the heap. We could then try to (transparently?) move things to the stack when we can (this would have implications for passing things around though). > The transform necesarry to enforce these rules should not be too > complicated. I don't think we need full analysis, just track > assignments > and deletions and usages on a per-name basis. > > Finally, the C++ output is made by adding a block, i.e. > > cdef CppObject obj > print 1 > obj = bar > obj.func() > del obj > print 2 > > Results in > > print 1 > { > CppObject obj = bar; > obj.func() > } # del > print 2 What about cdef CppObject a, b a = bar b = bar del a del b ? - Robert From stefan_ml at behnel.de Wed May 6 12:07:35 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 6 May 2009 12:07:35 +0200 (CEST) Subject: [Cython] Storing many, many instances In-Reply-To: References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <8f5db410ca5009dd2aec60dce437cd23.squirrel@webmail.uio.no> <7c940a5d130e97a4d739efc6f3343bff.squirrel@webmail.uio.no> <4A0028BE.9030903@behnel.de> Message-ID: <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Robert Bradshaw wrote: > On May 5, 2009, at 1:54 PM, Lisandro Dalcin wrote: >> But them someone will ask for polimorphism, and then the struct >> instances could have a pointer to a struct-specific vtable, and then >> you are more or less rewriting C++. And that would not bother me, >> currently cdef methods do work as a sort C++, right?. > > Yeah, I'm not sure how far one wants to take this though. I think a good point to stop is where we have to change the struct itself. Adding a vtable to it would be rather surprising for users as it could break stuff. We can do with cdef classes what we want, but C structs should stay the way they are defined in user code. Adding 'methods' to them is a rather straight transform in Cython that does not impact the generated C code, so that's just fine. > We should > look into making instantiating (cdef) classes faster by default as well. Different topic, but surely worth it (and worth a ticket). It might work to always use PY_NEW() for instantiation and then generate a separate (and direct) call to __init__() only if the type defines it. Stefan From dagss at student.matnat.uio.no Wed May 6 14:10:55 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 May 2009 14:10:55 +0200 Subject: [Cython] Storing many, many instances In-Reply-To: <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <8f5db410ca5009dd2aec60dce437cd23.squirrel@webmail.uio.no> <7c940a5d130e97a4d739efc6f3343bff.squirrel@webmail.uio.no> <4A0028BE.9030903@behnel.de> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <4A017E4F.5050800@student.matnat.uio.no> Stefan Behnel wrote: > Robert Bradshaw wrote: >> We should >> look into making instantiating (cdef) classes faster by default as well. > > Different topic, but surely worth it (and worth a ticket). It might work > to always use PY_NEW() for instantiation and then generate a separate (and > direct) call to __init__() only if the type defines it. Should one consider allocating arrays of cdef class objects as one big memory block? I would think that allocating first and then run a constructor N times would be faster than N allocations. -- Dag Sverre From dagss at student.matnat.uio.no Wed May 6 14:23:03 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 May 2009 14:23:03 +0200 Subject: [Cython] New idea for C++ stack allocation In-Reply-To: <759D34EC-9F78-4E80-8818-987F7551AA07@math.washington.edu> References: <49F6A8E5.2040006@student.matnat.uio.no> <759D34EC-9F78-4E80-8818-987F7551AA07@math.washington.edu> Message-ID: <4A018127.4090208@student.matnat.uio.no> Robert Bradshaw wrote: > On Apr 27, 2009, at 11:57 PM, Dag Sverre Seljebotn wrote: > >> I got another idea for how using C++ objects without no-arg >> constructors >> could work. It means enforcing C++ variable scoping through compile- >> time >> errors, but keeping Python syntax. > > I think for our first pass at least, we should allocate everything on > the heap. We could then try to (transparently?) move things to the > stack when we can (this would have implications for passing things > around though). I agree to this. If I understand you correctly it means that C++ classes would work very much like cdef classes (like in the wiki proposal). Some further notes: Passing things around shouldn't be such a problem; I don't think it would be much more complicated to do the heap allocation through a wrapper Python object (it could be opaque at first, only feature would be type-checking on coercion to a typed C++ class and perhaps store the C++ class name for its repr). Conversion of "CppClass" (refcounted wrapper) to "CppClass*" could be allowed; but probably not the other way (at least without using a function saying whether ownership of the pointer should be taken over or not). All of this *could* have been done for C structs as well, and in C++ there's no real distinction between a struct and a classes. I suppose a distinction must be done in Cython though because of code like cdef MyStruct a, b a.field = ... b = a # copies value, not reference (Note that the /only/ difference between the struct keyword and class keyword in C++ is whether private or public is the default visibility. A vtable is added if and only if a function is declared as virtual, whether it is declared as struct or not.) See also the other discussion on methods in structs. > What about > > cdef CppObject a, b > a = bar > b = bar > del a > del b > > ? Yes, I thought of that after sending it. -- Dag Sverre From dagss at student.matnat.uio.no Wed May 6 14:33:59 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 May 2009 14:33:59 +0200 Subject: [Cython] New idea for C++ stack allocation In-Reply-To: <759D34EC-9F78-4E80-8818-987F7551AA07@math.washington.edu> References: <49F6A8E5.2040006@student.matnat.uio.no> <759D34EC-9F78-4E80-8818-987F7551AA07@math.washington.edu> Message-ID: <4A0183B7.6080508@student.matnat.uio.no> Robert Bradshaw wrote: > What about > > cdef CppObject a, b > a = bar > b = bar > del a > del b > > ? Actually this *can* be supported by overloading operator new. I'll change the example to cdef CppObject a, b a = CppObject(3) b = CppObject(4) del a del b to avoid an unrelated and orthogonal issue with operator=. Then you can do: struct {} CythonOperators; void* operator new(size_t bytes, char* arg, CythonOperators foo) { return arg; } void operator delete(void* arg, CythonOperators foo) {/*noop*/} void f() { char a[sizeof(CppObject)]; char b[sizeof(CppObject)]; new(a, CythonOperators) CppObject(3); new(b, CythonOperators) CppObject(4); delete(CythonOperators) a; delete(CythonOperators) b; } And /yes/, this is real C++ code... *sigh* :-) (though I'm not sure if I got the details 100% right). I wouldn't really recommend going this route though. -- Dag Sverre From stefan_ml at behnel.de Wed May 6 14:34:52 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 6 May 2009 14:34:52 +0200 (CEST) Subject: [Cython] Storing many, many instances In-Reply-To: <4A017E4F.5050800@student.matnat.uio.no> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <8f5db410ca5009dd2aec60dce437cd23.squirrel@webmail.uio.no> <7c940a5d130e97a4d739efc6f3343bff.squirrel@webmail.uio.no> <4A0028BE.9030903@behnel.de> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A017E4F.5050800@student.matnat.uio.no> Message-ID: <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Robert Bradshaw wrote: >>> We should >>> look into making instantiating (cdef) classes faster by default as >>> well. >> >> Different topic, but surely worth it (and worth a ticket). It might work >> to always use PY_NEW() for instantiation and then generate a separate >> (and direct) call to __init__() only if the type defines it. > > Should one consider allocating arrays of cdef class objects as one big > memory block? We do not control the allocation, CPython handles that. > I would think that allocating first and then run a > constructor N times would be faster than N allocations. How would you instantiate arrays of classes in a single step? Stefan From dagss at student.matnat.uio.no Wed May 6 14:55:48 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 May 2009 14:55:48 +0200 Subject: [Cython] Storing many, many instances In-Reply-To: <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <8f5db410ca5009dd2aec60dce437cd23.squirrel@webmail.uio.no> <7c940a5d130e97a4d739efc6f3343bff.squirrel@webmail.uio.no> <4A0028BE.9030903@behnel.de> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A017E4F.5050800@student.matnat.uio.no> <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <4A0188D4.1050208@student.matnat.uio.no> Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> Stefan Behnel wrote: >>> Robert Bradshaw wrote: >>>> We should >>>> look into making instantiating (cdef) classes faster by default as >>>> well. >>> Different topic, but surely worth it (and worth a ticket). It might work >>> to always use PY_NEW() for instantiation and then generate a separate >>> (and direct) call to __init__() only if the type defines it. >> Should one consider allocating arrays of cdef class objects as one big >> memory block? > > We do not control the allocation, CPython handles that. OK. I didn't know exactly how fine-grained the CPython API was and whether it was possible to run a constructor on already allocated memory (like you can do in e.g. C++). It just seemed like that could be one of the major bottlenecks. -- Dag Sverre From stefan_ml at behnel.de Wed May 6 15:23:59 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 6 May 2009 15:23:59 +0200 (CEST) Subject: [Cython] Storing many, many instances In-Reply-To: <4A0188D4.1050208@student.matnat.uio.no> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <8f5db410ca5009dd2aec60dce437cd23.squirrel@webmail.uio.no> <7c940a5d130e97a4d739efc6f3343bff.squirrel@webmail.uio.no> <4A0028BE.9030903@behnel.de> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A017E4F.5050800@student.matnat.uio.no> <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A0188D4.1050208@student.matnat.uio.no> Message-ID: Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Dag Sverre Seljebotn wrote: >>> Stefan Behnel wrote: >>>> It might work >>>> to always use PY_NEW() for instantiation and then generate a separate >>>> (and direct) call to __init__() only if the type defines it. >>>> >>> Should one consider allocating arrays of cdef class objects as one big >>> memory block? >> >> We do not control the allocation, CPython handles that. > > OK. I didn't know exactly how fine-grained the CPython API was and > whether it was possible to run a constructor on already allocated memory > (like you can do in e.g. C++). It just seemed like that could be one of > the major bottlenecks. What we tend to call PY_NEW here is a direct call to a type's tp_new(), which handles both the allocation and the basic initialisation of an instance. Cython could do a tiny bit better here, since we know exactly what function a cdef class has in its tp_new slot. However, that function will always recursively run into the base type's tp_new and at the end of the chain into the one of Python's 'type' type, and there is not much we can do to improve this situation in general. Stefan From dalcinl at gmail.com Wed May 6 15:57:15 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 6 May 2009 10:57:15 -0300 Subject: [Cython] Include paths for pyximport In-Reply-To: References: Message-ID: On Tue, May 5, 2009 at 12:08 AM, Brett Calcott wrote: > Hi all, > I'm using pyimport and writing some extensions to work numpy. To get things > to happen magically the compiler needs to be able to find the header files. That's a very valid use case... > I've got it working on my mac, by adding a line to my .bashrc > export C_INCLUDE_PATH=/Library/Python/2.5/site-packages/numpy/core/include > I'd like to get it working on windows. Though it works on your Mac, it is not clear if this works on other POSIX systems... > I've added the equivalent path in my > windows env (set INCLUDE=...), but I'm getting an error telling me it cannot > find the "include/arrayobject.h". I've tried setting the right directory in > the GUI too, but to no avail. On Windows, distutils hardly inspect the environment, so playing with it is not a general solution. > Can anyone tell me the easy/right way to add include paths for pyximport on > windows. Is there a better and portable way to do this? AFAIK, pyximport does not support this at all... However, it would be really easy to enhance it. I would add a 'options' or 'config' (or whatever name) at the end of "pyximport.install()" (or perhaps provide a new call for this task) accepting a dictionary. That dictionary should contain stuff to pass as kwargs to the distutils Extension, the more important ones would be include_dirs define_macros undef_macros library_dirs libraries runtime_library_dirs That stuff should be (after checking and perhaps sanitizing the dict contents) passed to Extension() in "pyximport.get_distutils_extension()". Unfortunately, I'm really busy to implement something like that for review. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dalcinl at gmail.com Wed May 6 16:21:23 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 6 May 2009 11:21:23 -0300 Subject: [Cython] asking for review of my patch for #287 Message-ID: I want to raise your attention to ticket #287, as my patch could be controversial. I even changed my mind, and fixed my previous work on all this to prefer 'nb_int' over 'nb_long' in __Pyx_PyNumber_Int(). This change was not a quick hack to get things solved, but rather a hard to take decision after diving core CPython sources from 2.3 to 2.7. IMHO, the 'nb_long' slot should have been deprecated long, long ago in core CPython. I would like this to be discussed a bit more before next release. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From brett.calcott at gmail.com Wed May 6 16:43:36 2009 From: brett.calcott at gmail.com (Brett Calcott) Date: Thu, 7 May 2009 00:43:36 +1000 Subject: [Cython] Include paths for pyximport In-Reply-To: References: Message-ID: 2009/5/6 Lisandro Dalcin : > > --- Snip --- > >> Can anyone tell me the easy/right way to add include paths for pyximport on >> windows. Is there a better and portable way to do this? > > > AFAIK, pyximport does not support this at all... However, it would be > really easy to enhance it. I would add a 'options' or 'config' (or > whatever name) at the end of "pyximport.install()" (or perhaps provide > a new call for this task) accepting a dictionary. That dictionary > should contain stuff to pass as kwargs to the distutils Extension, the > more important ones would be > > include_dirs > define_macros > undef_macros > library_dirs > libraries > runtime_library_dirs > > That stuff should be (after checking and perhaps sanitizing the dict > contents) passed to Extension() in > "pyximport.get_distutils_extension()". > > Unfortunately, I'm really busy to implement something like that for review. > Hi Lisandro. I've just had a look at the pyximport files, and what you are suggesting makes sense to me. I'll give it a go myself, and then get back to the list with any progress. Thanks, Brett -- Brett Calcott Philosophy Program, Research School of Social Sciences, AND Centre for Macroevolution & Macroecology, School of Botany & Zoology, AUSTRALIAN NATIONAL UNIVERSITY, Canberra, ACT, 0200, AUSTRALIA From carl.witty at gmail.com Wed May 6 17:56:27 2009 From: carl.witty at gmail.com (Carl Witty) Date: Wed, 6 May 2009 08:56:27 -0700 Subject: [Cython] Storing many, many instances In-Reply-To: References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <7c940a5d130e97a4d739efc6f3343bff.squirrel@webmail.uio.no> <4A0028BE.9030903@behnel.de> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A017E4F.5050800@student.matnat.uio.no> <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A0188D4.1050208@student.matnat.uio.no> Message-ID: On Wed, May 6, 2009 at 6:23 AM, Stefan Behnel wrote: > What we tend to call PY_NEW here is a direct call to a type's tp_new(), > which handles both the allocation and the basic initialisation of an > instance. Cython could do a tiny bit better here, since we know exactly > what function a cdef class has in its tp_new slot. However, that function > will always recursively run into the base type's tp_new and at the end of > the chain into the one of Python's 'type' type, and there is not much we > can do to improve this situation in general. Well, you could optimize tp_new. You could have it call the base type's tp_new directly, instead of through a vtable (at least if the base type is a Cython type); or better yet, inline the base type's tp_new (for Cython base types). (Gary Furnish complained that calling through the tp_new chain was a major expense if you had a deep inheritance hierarchy; I haven't confirmed this myself, but it sounds plausible.) Carl From ellisonbg.net at gmail.com Wed May 6 18:08:37 2009 From: ellisonbg.net at gmail.com (Brian Granger) Date: Wed, 6 May 2009 09:08:37 -0700 Subject: [Cython] Include paths for pyximport In-Reply-To: References: Message-ID: <6ce0ac130905060908wf3542d1m8148afda3bec9a9@mail.gmail.com> > AFAIK, pyximport does not support this at all... However, it would be > really easy to enhance it. I would add a 'options' or 'config' (or > whatever name) at the end of "pyximport.install()" (or perhaps provide > a new call for this task) accepting a dictionary. That dictionary > should contain stuff to pass as kwargs to the distutils Extension, the > more important ones would be > > include_dirs > define_macros > undef_macros > library_dirs > libraries > runtime_library_dirs > > That stuff should be (after checking and perhaps sanitizing the dict > contents) passed to Extension() in > "pyximport.get_distutils_extension()". +1, great idea From dagss at student.matnat.uio.no Wed May 6 18:31:06 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 May 2009 18:31:06 +0200 Subject: [Cython] Buffer passing implementation details Message-ID: <4A01BB4A.1060003@student.matnat.uio.no> (Kurt is the major recipient of this.) I've been thinking some more on buffer passing. In the end (and perhaps even in summer; I could perhaps have a go at it working alongside you) the scenario we are looking at is something like external_set_foo_array(foo_handle, self.arr[2::2]) with fast operations all the way. That is a) Efficient slicing without Python overhead (#178) b) Storing acquired buffers in cdef class fields (#301) c) External functions can keep a reference to the buffer (not really necesarry, but it is necesarry for internal Cython "cdef" functions, and it would be nice to treat them the same) d) The base pointer may have to be moved (again slicing can do this) This seems to make it clear that a) The Py_buffer is not suitable as the primary vessel of our buffer data, e.g. to the external function. It can't work that well with slicing as we must maintain the original Py_buffer data when releasing it. b) We need a reference count. E.g. doing a slice, or assigning to self.arr, would increase this count. This must go on the heap; and so we might as well put the entire Py_buffer on the heap. I have made a new ticket outlining the thoughts in detail in #311 and added a comment in #299. #311 is not really in your direct interest for GSoC but it is very tightly coupled with #299 so it would be good to keep in the back of your mind anyway. Thoughts? It sheems a shame to let go of a neatly PEP-defined Py_buffer for passing to external functions, but I think it won't be too bad if we with each Cython version ship nice C header and Fortran include files containing the appropriate structs and access macros. -- Dag Sverre From dagss at student.matnat.uio.no Wed May 6 19:21:23 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 May 2009 19:21:23 +0200 Subject: [Cython] endianness bug fix in numpy_test.pyx In-Reply-To: References: <49E991D7.5000809@student.matnat.uio.no> <49E9CE21.8040105@student.matnat.uio.no> Message-ID: <4A01C713.7080302@student.matnat.uio.no> Kurt Smith wrote: > On Sat, Apr 18, 2009 at 7:57 AM, Dag Sverre Seljebotn > wrote: >> Dag Sverre Seljebotn wrote: >>> Kurt Smith wrote: >>>> The current numpy_test.pyx file fails for PowerPC macs due to >>>> endianness issues in the dtype. This is a small fix to make it work >>>> (and make all tests pass on my machine). It also adds an explicit >>>> big-endian test to the doctest. >>>> >>>> diff -r fc73225aaea1 tests/run/numpy_test.pyx >>>> --- a/tests/run/numpy_test.pyx Fri Apr 17 09:11:16 2009 +0200 >>>> +++ b/tests/run/numpy_test.pyx Fri Apr 17 15:43:11 2009 -0500 >>>> @@ -132,13 +132,20 @@ >>>> >>> test_recordarray() >>>> >>>> >>> test_nested_dtypes(np.zeros((3,), dtype=np.dtype([\ >>>> - ('a', np.dtype('i,i')),\ >>>> - ('b', np.dtype('i,i'))\ >>>> + ('a', np.dtype('>>> + ('b', np.dtype('>>> ]))) >>>> array([((0, 0), (0, 0)), ((1, 2), (1, 4)), ((1, 2), (1, 4))], >>>> dtype=[('a', [('f0', '>>> '>> Any ideas on why this particular case actually passes on your machine? >>> The Cython code alwayws works with big-endian on your machine, so I >>> wouldn't expect to see ((1, 2), (1, 4)) there, but the byteswapped >>> versions of those numbers... >>> >> Ahh right; the only value read from the input array (in the test >> function) is zero, which is the same in both endians, and you added >> explicit byteswap on the output. > > Sorry -- can't devote the time to address these questions right now > but I'll get to them as soon as I can (Monday?). Could you try again with current cython-devel at some point? I believe this should be fixed now but obviously it should be tested on a PPC. -- Dag Sverre From robertwb at math.washington.edu Wed May 6 19:30:46 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 10:30:46 -0700 Subject: [Cython] New idea for C++ stack allocation In-Reply-To: <4A0183B7.6080508@student.matnat.uio.no> References: <49F6A8E5.2040006@student.matnat.uio.no> <759D34EC-9F78-4E80-8818-987F7551AA07@math.washington.edu> <4A0183B7.6080508@student.matnat.uio.no> Message-ID: On May 6, 2009, at 5:33 AM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> What about >> >> cdef CppObject a, b >> a = bar >> b = bar >> del a >> del b >> >> ? > > Actually this *can* be supported by overloading operator new. > > I'll change the example to > > cdef CppObject a, b > a = CppObject(3) > b = CppObject(4) > del a > del b > > to avoid an unrelated and orthogonal issue with operator=. Then you > can do: > > struct {} CythonOperators; > > void* operator new(size_t bytes, char* arg, CythonOperators foo) { > return arg; } > void operator delete(void* arg, CythonOperators foo) {/*noop*/} > > void f() { > char a[sizeof(CppObject)]; > char b[sizeof(CppObject)]; > > new(a, CythonOperators) CppObject(3); > new(b, CythonOperators) CppObject(4); > > delete(CythonOperators) a; > delete(CythonOperators) b; > } > > And /yes/, this is real C++ code... *sigh* :-) (though I'm not sure > if I > got the details 100% right). Actually, the new operator already supports passing in an address to be used for memory, but I wasn't sure enough of how it worked to bring it up. There's an issue of alignment as well. > I wouldn't really recommend going this route though. I like it better than trying to use blocks to accomplish stack-based allocation though. - Robert From robertwb at math.washington.edu Wed May 6 19:38:53 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 10:38:53 -0700 Subject: [Cython] Storing many, many instances In-Reply-To: References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <7c940a5d130e97a4d739efc6f3343bff.squirrel@webmail.uio.no> <4A0028BE.9030903@behnel.de> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A017E4F.5050800@student.matnat.uio.no> <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A0188D4.1050208@student.matnat.uio.no> Message-ID: <11E61C9F-3175-4473-8B39-20277284E9D1@math.washington.edu> On May 6, 2009, at 8:56 AM, Carl Witty wrote: > On Wed, May 6, 2009 at 6:23 AM, Stefan Behnel > wrote: >> What we tend to call PY_NEW here is a direct call to a type's >> tp_new(), >> which handles both the allocation and the basic initialisation of an >> instance. Cython could do a tiny bit better here, since we know >> exactly >> what function a cdef class has in its tp_new slot. However, that >> function >> will always recursively run into the base type's tp_new and at the >> end of >> the chain into the one of Python's 'type' type, and there is not >> much we >> can do to improve this situation in general. > > Well, you could optimize tp_new. You could have it call the base > type's tp_new directly, instead of through a vtable (at least if the > base type is a Cython type); I already implemented this optimization, though only for cdef classes within the same module. (If they're small enough, they may get even inlined by the C compiler.) I'm not sure how possible this would be to do across modules, but one difficulty is that the contents of tp_new can change based on the .pyx file, not just the .pxd file. That being said, one does have control over allocation via the tp_alloc method. Allocating an array of objects as a big chuck is messy though--how would one handle deallocating them one by one? Or would one try to reference-count them as a group? I think it could be done, but it would be a non-trivial interface. - Robert From dagss at student.matnat.uio.no Wed May 6 19:56:02 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 May 2009 19:56:02 +0200 Subject: [Cython] New idea for C++ stack allocation In-Reply-To: References: <49F6A8E5.2040006@student.matnat.uio.no> <759D34EC-9F78-4E80-8818-987F7551AA07@math.washington.edu> <4A0183B7.6080508@student.matnat.uio.no> Message-ID: <4A01CF32.8050207@student.matnat.uio.no> Robert Bradshaw wrote: > On May 6, 2009, at 5:33 AM, Dag Sverre Seljebotn wrote: > >> Robert Bradshaw wrote: >>> What about >>> >>> cdef CppObject a, b >>> a = bar >>> b = bar >>> del a >>> del b >>> >>> ? >> Actually this *can* be supported by overloading operator new. >> >> I'll change the example to >> >> cdef CppObject a, b >> a = CppObject(3) >> b = CppObject(4) >> del a >> del b >> >> to avoid an unrelated and orthogonal issue with operator=. Then you >> can do: >> >> struct {} CythonOperators; >> >> void* operator new(size_t bytes, char* arg, CythonOperators foo) { >> return arg; } >> void operator delete(void* arg, CythonOperators foo) {/*noop*/} >> >> void f() { >> char a[sizeof(CppObject)]; >> char b[sizeof(CppObject)]; >> >> new(a, CythonOperators) CppObject(3); >> new(b, CythonOperators) CppObject(4); >> >> delete(CythonOperators) a; >> delete(CythonOperators) b; >> } >> >> And /yes/, this is real C++ code... *sigh* :-) (though I'm not sure >> if I >> got the details 100% right). > > Actually, the new operator already supports passing in an address to > be used for memory, but I wasn't sure enough of how it worked to > bring it up. There's an issue of alignment as well. Hmm. Interesting. Looked it up properly and tried it out, attaching for any future reference (though I suppose stack allocation is far down the line and after heap then). Some memory is wasted for the alignment issue. #include #include using namespace std; class CppObject { int data; public: CppObject(int d) : data(d) { cout << data << " construct" << endl; } ~CppObject() { cout << data << " destruct" << endl; } }; int main () { void* __pyx_mem_a[(sizeof(CppObject) / sizeof(void*))+1]; CppObject& __pyx_v_a = *(CppObject*)buf; new(buf) CppObject(4); // use __pyx_v_a as any stack allocated object co.~CppObject(); cout << "done" << endl; } -- Dag Sverre From dagss at student.matnat.uio.no Wed May 6 20:12:10 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 May 2009 20:12:10 +0200 Subject: [Cython] New idea for C++ stack allocation In-Reply-To: <4A01CF32.8050207@student.matnat.uio.no> References: <49F6A8E5.2040006@student.matnat.uio.no> <759D34EC-9F78-4E80-8818-987F7551AA07@math.washington.edu> <4A0183B7.6080508@student.matnat.uio.no> <4A01CF32.8050207@student.matnat.uio.no> Message-ID: <4A01D2FA.400@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On May 6, 2009, at 5:33 AM, Dag Sverre Seljebotn wrote: >> >>> Robert Bradshaw wrote: >>>> What about >>>> >>>> cdef CppObject a, b >>>> a = bar >>>> b = bar >>>> del a >>>> del b >>>> >>>> ? >>> Actually this *can* be supported by overloading operator new. >>> >>> I'll change the example to >>> >>> cdef CppObject a, b >>> a = CppObject(3) >>> b = CppObject(4) >>> del a >>> del b >>> >>> to avoid an unrelated and orthogonal issue with operator=. Then you >>> can do: >>> >>> struct {} CythonOperators; >>> >>> void* operator new(size_t bytes, char* arg, CythonOperators foo) { >>> return arg; } >>> void operator delete(void* arg, CythonOperators foo) {/*noop*/} >>> >>> void f() { >>> char a[sizeof(CppObject)]; >>> char b[sizeof(CppObject)]; >>> >>> new(a, CythonOperators) CppObject(3); >>> new(b, CythonOperators) CppObject(4); >>> >>> delete(CythonOperators) a; >>> delete(CythonOperators) b; >>> } >>> >>> And /yes/, this is real C++ code... *sigh* :-) (though I'm not sure >>> if I >>> got the details 100% right). >> Actually, the new operator already supports passing in an address to >> be used for memory, but I wasn't sure enough of how it worked to >> bring it up. There's an issue of alignment as well. > > Hmm. Interesting. Looked it up properly and tried it out, attaching for > any future reference (though I suppose stack allocation is far down the > line and after heap then). /Honestly/... This time without changing any variables names while in email mode... #include #include using namespace std; class CppObject { int data; public: CppObject(int d) : data(d) { cout << data << " construct" << endl; } ~CppObject() { cout << data << " destruct" << endl; } }; int main () { void* __pyx_mem_myvar[(sizeof(CppObject) / sizeof(void*))+1]; CppObject& __pyx_v_myvar = *(CppObject*)__pyx_mem_myvar; new(__pyx_mem_myvar) CppObject(4); // use __pyx_v_myvar as normal CppObject on stack __pyx_v_myvar.~CppObject(); cout << "done" << endl; } -- Dag Sverre From carl.witty at gmail.com Wed May 6 20:18:59 2009 From: carl.witty at gmail.com (Carl Witty) Date: Wed, 6 May 2009 11:18:59 -0700 Subject: [Cython] Storing many, many instances In-Reply-To: <11E61C9F-3175-4473-8B39-20277284E9D1@math.washington.edu> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A017E4F.5050800@student.matnat.uio.no> <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A0188D4.1050208@student.matnat.uio.no> <11E61C9F-3175-4473-8B39-20277284E9D1@math.washington.edu> Message-ID: On Wed, May 6, 2009 at 10:38 AM, Robert Bradshaw wrote: > I already implemented this optimization, though only for cdef classes > within the same module. (If they're small enough, they may get even > inlined by the C compiler.) I'm not sure how possible this would be > to do across modules, but one difficulty is that the contents of > tp_new can change based on the .pyx file, not just the .pxd file. Really? What in the .pyx file can change tp_new? (I'm mostly just curious.) Across modules, you should at least be able to call the tp_new function directly instead of going through the vtable. (I assume from the .pxd file you can at least determine the mangled name of the base's tp_new function?) Carl From kwmsmith at gmail.com Wed May 6 20:50:27 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Wed, 6 May 2009 13:50:27 -0500 Subject: [Cython] endianness bug fix in numpy_test.pyx In-Reply-To: <4A01C713.7080302@student.matnat.uio.no> References: <49E991D7.5000809@student.matnat.uio.no> <49E9CE21.8040105@student.matnat.uio.no> <4A01C713.7080302@student.matnat.uio.no> Message-ID: On Wed, May 6, 2009 at 12:21 PM, Dag Sverre Seljebotn wrote: > Could you try again with current cython-devel at some point? I believe > this should be fixed now but obviously it should be tested on a PPC. Using changeset 2028:1377b8aff3a1 from cython-devel. $ uname -a Darwin ksmith.physics.wisc.edu 8.11.0 Darwin Kernel Version 8.11.0: Wed Oct 10 18:26:00 PDT 2007; root:xnu-792.24.17~1/RELEASE_PPC Power Macintosh powerpc $ python -V Python 2.5.2 $ python runtests.py -vv Yields 3 failures and 7 errors. None seem related to the endianness issues; the 7 errors are related to recent fixes to Buffer.py. I'm attaching the full output -- copied here are just the failure summaries: ====================================================================== FAIL: Doctest: cdef_setitem_T284 ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/doctest.py", line 2128, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for cdef_setitem_T284 File "/Users/ksmith/Devel/cython/cython-devel/BUILD/run/c/cdef_setitem_T284.so", line unknown line number, in cdef_setitem_T284 ---------------------------------------------------------------------- File "/Users/ksmith/Devel/cython/cython-devel/BUILD/run/c/cdef_setitem_T284.so", line ?, in cdef_setitem_T284 Failed example: test_list(range(11), "invalid index", None) Expected: Traceback (most recent call last): ... TypeError: list indices must be integers, not str Got: Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/doctest.py", line 1228, in __run compileflags, 1) in test.globs File "", line 1, in test_list(range(11), "invalid index", None) File "cdef_setitem_T284.pyx", line 26, in cdef_setitem_T284.test_list (cdef_setitem_T284.c:606) TypeError: list indices must be integers ====================================================================== FAIL: Doctest: cdef_setitem_T284 ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/doctest.py", line 2128, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for cdef_setitem_T284 File "/Users/ksmith/Devel/cython/cython-devel/BUILD/run/cpp/cdef_setitem_T284.so", line unknown line number, in cdef_setitem_T284 ---------------------------------------------------------------------- File "/Users/ksmith/Devel/cython/cython-devel/BUILD/run/cpp/cdef_setitem_T284.so", line ?, in cdef_setitem_T284 Failed example: test_list(range(11), "invalid index", None) Expected: Traceback (most recent call last): ... TypeError: list indices must be integers, not str Got: Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/doctest.py", line 1228, in __run compileflags, 1) in test.globs File "", line 1, in test_list(range(11), "invalid index", None) File "cdef_setitem_T284.pyx", line 26, in cdef_setitem_T284.test_list (cdef_setitem_T284.cpp:606) TypeError: list indices must be integers ====================================================================== FAIL: Doctest: numpy_test ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/doctest.py", line 2128, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for numpy_test File "/Users/ksmith/Devel/cython/cython-devel/BUILD/run/c/numpy_test.so", line 159, in numpy_test ---------------------------------------------------------------------- File "/Users/ksmith/Devel/cython/cython-devel/BUILD/run/c/numpy_test.so", line 326, in numpy_test Failed example: test_bad_cast() Expected: Traceback (most recent call last): ... ValueError: Item size of buffer (1 byte) does not match size of 'long' (8 bytes) Got: Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/doctest.py", line 1228, in __run compileflags, 1) in test.globs File "", line 1, in test_bad_cast() File "numpy_test.pyx", line 353, in numpy_test.test_bad_cast (numpy_test.c:5123) ValueError: Item size of buffer (1 byte) does not match size of 'long' (4 bytes) ---------------------------------------------------------------------- -------------- next part -------------- A non-text attachment was scrubbed... Name: runtests.tgz Type: application/x-gzip Size: 10564 bytes Desc: not available Url : http://codespeak.net/pipermail/cython-dev/attachments/20090506/c2601bcd/attachment.bin From dagss at student.matnat.uio.no Wed May 6 21:16:31 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 06 May 2009 21:16:31 +0200 Subject: [Cython] endianness bug fix in numpy_test.pyx In-Reply-To: References: <49E991D7.5000809@student.matnat.uio.no> <49E9CE21.8040105@student.matnat.uio.no> <4A01C713.7080302@student.matnat.uio.no> Message-ID: <4A01E20F.2080506@student.matnat.uio.no> Kurt Smith wrote: > On Wed, May 6, 2009 at 12:21 PM, Dag Sverre Seljebotn > wrote: > >> Could you try again with current cython-devel at some point? I believe >> this should be fixed now but obviously it should be tested on a PPC. > > Using changeset 2028:1377b8aff3a1 from cython-devel. > > $ uname -a > Darwin ksmith.physics.wisc.edu 8.11.0 Darwin Kernel Version 8.11.0: > Wed Oct 10 18:26:00 PDT 2007; root:xnu-792.24.17~1/RELEASE_PPC Power > Macintosh powerpc > > $ python -V > Python 2.5.2 > > $ python runtests.py -vv > > Yields 3 failures and 7 errors. None seem related to the endianness > issues; the 7 errors are related to recent fixes to Buffer.py. Doh; I forgot to test buffers on C++! Anyway your log was helpful in other areas too, thanks! -- Dag Sverre From robertwb at math.washington.edu Wed May 6 21:24:13 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 12:24:13 -0700 Subject: [Cython] Storing many, many instances In-Reply-To: References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A017E4F.5050800@student.matnat.uio.no> <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A0188D4.1050208@student.matnat.uio.no> <11E61C9F-3175-4473-8B39-20277284E9D1@math.washington.edu> Message-ID: On May 6, 2009, at 11:18 AM, Carl Witty wrote: > On Wed, May 6, 2009 at 10:38 AM, Robert Bradshaw > wrote: >> I already implemented this optimization, though only for cdef classes >> within the same module. (If they're small enough, they may get even >> inlined by the C compiler.) I'm not sure how possible this would be >> to do across modules, but one difficulty is that the contents of >> tp_new can change based on the .pyx file, not just the .pxd file. > > Really? What in the .pyx file can change tp_new? (I'm mostly just > curious.) The __cinit__ (aka __new__) function, if any, get's invoked. > Across modules, you should at least be able to call the tp_new > function directly instead of going through the vtable. (I assume from > the .pxd file you can at least determine the mangled name of the > base's tp_new function?) This is the optimization I was referring too--look at the generated sources for, e.g., sage.structure.element.pyx - Robert From robertwb at math.washington.edu Wed May 6 21:30:17 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 12:30:17 -0700 Subject: [Cython] Syntax to declare a c++ class? Message-ID: Just trying to get a syntax, how about cdef extern from "header.h": cdef cclass MyClass[TemplateParamers](BaseCPPClass): int member int method(int) The only concern I have is whether or not "cdef cclass" is to similar to "cdef class" (probably not). Any other ideas? - Robert From dalcinl at gmail.com Wed May 6 21:46:49 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 6 May 2009 16:46:49 -0300 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: Message-ID: On Wed, May 6, 2009 at 4:30 PM, Robert Bradshaw wrote: > Just trying to get a syntax, how about > > cdef extern from "header.h": > ? ? cdef cclass MyClass[TemplateParamers](BaseCPPClass): > ? ? ? ? int member > ? ? ? ? int method(int) > > The only concern I have is whether or not "cdef cclass" is to similar > to "cdef class" (probably not). Any other ideas? > Any chance you do not need 'cclass' and 'struct' would be enough (perhaps with ctypedef instead of cdef)? Cython will not be able to access private stuff from C++ classes... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Wed May 6 21:48:42 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 6 May 2009 21:48:42 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: Message-ID: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> Robert Bradshaw wrote: > Just trying to get a syntax, how about > > cdef extern from "header.h": > cdef cclass MyClass[TemplateParamers](BaseCPPClass): > int member > int method(int) > > The only concern I have is whether or not "cdef cclass" is to similar > to "cdef class" (probably not). Any other ideas? cdef cppclass? Yes, I think cclass is to similar to class. Anyway, when I proposed a Fortran syntax earlier you and Stefan made a good point about the language specification belonging in the "cdef extern" part, as, after all, the "header.h" file contains only C++ code. Also consider that C++ needs namespace specifications, for instance cdef extern ...: namespace std: cdef cppclass vector[T]: ... or cdef cclass std::vector[T]: ... Dag Sverre From carl.witty at gmail.com Wed May 6 22:08:46 2009 From: carl.witty at gmail.com (Carl Witty) Date: Wed, 6 May 2009 13:08:46 -0700 Subject: [Cython] Storing many, many instances In-Reply-To: References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A017E4F.5050800@student.matnat.uio.no> <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A0188D4.1050208@student.matnat.uio.no> <11E61C9F-3175-4473-8B39-20277284E9D1@math.washington.edu> Message-ID: On Wed, May 6, 2009 at 12:24 PM, Robert Bradshaw wrote: > On May 6, 2009, at 11:18 AM, Carl Witty wrote: > >> On Wed, May 6, 2009 at 10:38 AM, Robert Bradshaw >> wrote: >>> I already implemented this optimization, though only for cdef classes >>> within the same module. (If they're small enough, they may get even >>> inlined by the C compiler.) I'm not sure how possible this would be >>> to do across modules, but one difficulty is that the contents of >>> tp_new can change based on the .pyx file, not just the .pxd file. >> >> Really? ?What in the .pyx file can change tp_new? ?(I'm mostly just >> curious.) > > The __cinit__ (aka __new__) function, if any, get's invoked. So if we decided that the optimization was important enough, we could fix this by requiring that __cinit__ functions be declared in the .pxd file. >> Across modules, you should at least be able to call the tp_new >> function directly instead of going through the vtable. ?(I assume from >> the .pxd file you can at least determine the mangled name of the >> base's tp_new function?) > > This is the optimization I was referring too--look at the generated > sources for, e.g., sage.structure.element.pyx Yes, but you said that you only implement it within a single module (which is what I see in element.c); I'm saying that it should also be possible to implement across modules (just make tp_new functions non-static, add an extern declaration in the second module, and call the function directly). Carl From robertwb at math.washington.edu Wed May 6 22:12:26 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 13:12:26 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> Message-ID: On May 6, 2009, at 12:48 PM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> Just trying to get a syntax, how about >> >> cdef extern from "header.h": >> cdef cclass MyClass[TemplateParamers](BaseCPPClass): >> int member >> int method(int) >> >> The only concern I have is whether or not "cdef cclass" is to similar >> to "cdef class" (probably not). Any other ideas? > > cdef cppclass? I don't like this one, but I think just for aesthetic reasons. Nor cpp_class. > Yes, I think cclass is to similar to class. > > Anyway, when I proposed a Fortran syntax earlier you and Stefan made a > good point about the language specification belonging in the "cdef > extern" > part, as, after all, the "header.h" file contains only C++ code. Will the compiler need to know that header.h is C++ code? If not, I see no reason to declare it. > Also consider that C++ needs namespace specifications, for instance > > cdef extern ...: > namespace std: > cdef cppclass vector[T]: ... > > or > > cdef cclass std::vector[T]: ... Yep. That too. I'm a fan of both declarations. (Or even rolling the "namespace" into the cdef extern statement.) - Robert From robertwb at math.washington.edu Wed May 6 22:17:07 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 13:17:07 -0700 Subject: [Cython] Storing many, many instances In-Reply-To: References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A017E4F.5050800@student.matnat.uio.no> <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A0188D4.1050208@student.matnat.uio.no> <11E61C9F-3175-4473-8B39-20277284E9D1@math.washington.edu> Message-ID: <4B519262-77AC-4BC4-A62B-026B4E254FAA@math.washington.edu> On May 6, 2009, at 1:08 PM, Carl Witty wrote: > On Wed, May 6, 2009 at 12:24 PM, Robert Bradshaw > wrote: >> On May 6, 2009, at 11:18 AM, Carl Witty wrote: >> >>> On Wed, May 6, 2009 at 10:38 AM, Robert Bradshaw >>> wrote: >>>> I already implemented this optimization, though only for cdef >>>> classes >>>> within the same module. (If they're small enough, they may get even >>>> inlined by the C compiler.) I'm not sure how possible this would be >>>> to do across modules, but one difficulty is that the contents of >>>> tp_new can change based on the .pyx file, not just the .pxd file. >>> >>> Really? What in the .pyx file can change tp_new? (I'm mostly just >>> curious.) >> >> The __cinit__ (aka __new__) function, if any, get's invoked. > > So if we decided that the optimization was important enough, we could > fix this by requiring that __cinit__ functions be declared in the .pxd > file. > >>> Across modules, you should at least be able to call the tp_new >>> function directly instead of going through the vtable. (I assume >>> from >>> the .pxd file you can at least determine the mangled name of the >>> base's tp_new function?) >> >> This is the optimization I was referring too--look at the generated >> sources for, e.g., sage.structure.element.pyx > > Yes, but you said that you only implement it within a single module > (which is what I see in element.c); I'm saying that it should also be > possible to implement across modules (just make tp_new functions > non-static, add an extern declaration in the second module, and call > the function directly). Yes, I we could do this. We could also support exported cdef functions this way, instead of the current function import mechanism. (Or was there some reason it wasn't done this way in the first place? Certainly it results in "kinder" errors.) - Robert From dagss at student.matnat.uio.no Wed May 6 22:33:42 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Wed, 6 May 2009 22:33:42 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> Message-ID: <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> Robert Bradshaw wrote: > On May 6, 2009, at 12:48 PM, Dag Sverre Seljebotn wrote: > >> Robert Bradshaw wrote: >>> Just trying to get a syntax, how about >>> >>> cdef extern from "header.h": >>> cdef cclass MyClass[TemplateParamers](BaseCPPClass): >>> int member >>> int method(int) >>> >>> The only concern I have is whether or not "cdef cclass" is to similar >>> to "cdef class" (probably not). Any other ideas? >> >> cdef cppclass? > > I don't like this one, but I think just for aesthetic reasons. Nor > cpp_class. Myself I'm used to a .cpp suffix on C++ files; I suppose people who are used to the competing .cc will have stronger associations with your proposal. "cclass" gives me no associations at all initially. >> Yes, I think cclass is to similar to class. >> >> Anyway, when I proposed a Fortran syntax earlier you and Stefan made a >> good point about the language specification belonging in the "cdef >> extern" >> part, as, after all, the "header.h" file contains only C++ code. > > Will the compiler need to know that header.h is C++ code? If not, I > see no reason to declare it. No -- it's just the fact that usually all classes within the extern block will be a C++ class. I was thinking just plain "cdef class" could then change meaning, which would have a happy association with the heap-allocation thing (which C++ classes would share with existing cdef classes). cdef extern "C++" from "myheader.h": cdef class A: ... OTOH, if heap-allocation-and-refcounted semantics isn't selected, one should use "cdef struct" instead, like Lisandro suggests. In the unlikely event that you have both a C++ class and a Python extension class in the same header you can always repeat the extern statement. After all, C++ uses extern "C" { } to interface with C. > >> Also consider that C++ needs namespace specifications, for instance >> >> cdef extern ...: >> namespace std: >> cdef cppclass vector[T]: ... >> >> or >> >> cdef cclass std::vector[T]: ... > > Yep. That too. I'm a fan of both declarations. (Or even rolling the > "namespace" into the cdef extern statement.) FWIW I don't like the second one. I like rolling it into the extern statement. A single header file can declare things for many namespaces, but it is not common and you can always use two cdef extern blocks for the same header in such cases. Dag Sverre From kwmsmith at gmail.com Wed May 6 23:41:12 2009 From: kwmsmith at gmail.com (Kurt Smith) Date: Wed, 6 May 2009 16:41:12 -0500 Subject: [Cython] Buffer passing implementation details In-Reply-To: <4A01BB4A.1060003@student.matnat.uio.no> References: <4A01BB4A.1060003@student.matnat.uio.no> Message-ID: On Wed, May 6, 2009 at 11:31 AM, Dag Sverre Seljebotn wrote: > (Kurt is the major recipient of this.) > > I've been thinking some more on buffer passing. In the end (and perhaps > even in summer; I could perhaps have a go at it working alongside you) > the scenario we are looking at is something like > > external_set_foo_array(foo_handle, self.arr[2::2]) > > with fast operations all the way. > > That is > a) Efficient slicing without Python overhead (#178) > b) Storing acquired buffers in cdef class fields (#301) > c) External functions can keep a reference to the buffer (not really > necesarry, but it is necesarry for internal Cython "cdef" functions, and > it would be nice to treat them the same) > d) The base pointer may have to be moved (again slicing can do this) > > This seems to make it clear that > a) The Py_buffer is not suitable as the primary vessel of our buffer > data, e.g. to the external function. It can't work that well with > slicing as we must maintain the original Py_buffer data when releasing it. > b) We need a reference count. E.g. doing a slice, or assigning to > self.arr, would increase this count. This must go on the heap; and so we > might as well put the entire Py_buffer on the heap. Doing the slicing manually on the buffer data and having a refcounted heap-allocated Py_buffer would address the object-Py_buffer data synchronization problem, right? The above modifications would have that, once the buffer is acquired, all modifications to it are done at the buffer level. Although, what about when we require a contiguous copy (or any mode, for that matter) of a buffer? Would we set the top-level object reference to None, as suggested in a previous thread? That would prevent slicing at the Python API level, though, right? > > I have made a new ticket outlining the thoughts in detail in #311 and > added a comment in #299. #311 is not really in your direct interest for > GSoC but it is very tightly coupled with #299 so it would be good to > keep in the back of your mind anyway. > > Thoughts? General question: how difficult would it be for the Cython-side programmer to get the underlying char *data inside the Py_buffer, in case it needs to be accessed? It seems there are more than a few 'layers' -- are they all necessary? The justifications I see for each are the following: typedef struct { void *buf; PyObject *obj; Py_ssize_t len; Py_ssize_t itemsize; int readonly; int ndim; char *format; Py_ssize_t *shape; Py_ssize_t *strides; Py_ssize_t *suboffsets; void *internal; } Py_buffer; The "base layer" -- interface structure for tp_getbuffer from objects, etc. typedef struct { size_t refcount; Py_buffer bufinfo; } __Pyx_Buffer; The Cython wrapper around Py_buffer -- includes cython-managed refcounting upon assignment/slicing (other cases?) typedef struct { __Pyx_buffer* bufinfo; char* data; Py_ssize_t shape0, stride0; } __Pyx_StridedBuf_1D; The "int[:] buf" buffer -- exists to make the new buffer syntax more transparent (not part of GSoC, but putting in framework for it). char *data pointer to allow fast slicing w/o Python API (e.g. arr[10:]), shape0 & stride0 copied from buffer for optimizations. typedef struct { __Pyx_StridedBuf_1D buffer; PyObject* object; } __Pyx_StridedBuf_1D_Obj; The "object[int, mode='strided'] buf" object buffer. This is the top level and holds a reference to the original PyObject from which the buffer was acquired. Object reference required when a python-level access/assignment is required. So, initially, if one passes a 'buffer' to a function like so: external_func(arr[2::2]) What is the slicing modifying/updating? The top-level through the Python API layer via the PyObject? Or is it being done to the __Pyx_StridedBuf_1D members -- offsetting the char *data pointer by 2, and changing the shape0 and stride0 members? (Would it be the top layer object at first & for the GSoC, then when faster, low-level slicing is implemented, directly on the lower-level?) Besides Python API-level slicing, why is the top-level object reference required? Is it for buffer reacquisition through tp_getbuffer? That will be handled at the next lower level through a struct copy by value and increase of the refcount. It isn't for buffer release, either, since that doesn't require the original object reference explicitly. (There is an object reference at the Py_buffer level). Basically, nothing Cython-side would require the PyObject from which the original buffer is acquired once all these are implemented, right? (Again, not all part of GSoC, but at least putting in the framework.) This is something of me thinking out loud, so let me know confusing parts, etc. > It sheems a shame to let go of a neatly PEP-defined Py_buffer for > passing to external functions, but I think it won't be too bad if we > with each Cython version ship nice C header and Fortran include files > containing the appropriate structs and access macros. I guess the access macros would make it easy for the external functions to get to the important stuff, so my previous concern isn't so essential (removing the 'top level' layer from the buffer struct heirarchy). Kurt From dsurviver at gmail.com Wed May 6 23:53:33 2009 From: dsurviver at gmail.com (Danilo Freitas) Date: Wed, 6 May 2009 18:53:33 -0300 Subject: [Cython] template types In-Reply-To: References: <9457e7c80905041133j2d2abb88na8fabaaf57fa6023@mail.gmail.com> <4A00AD43.1050709@behnel.de> <4A00AF71.7010103@behnel.de> <4A011CFE.7070006@behnel.de> <3439af420a85daac11ba6238c254c18f.squirrel@webmail.uio.no> <8B75D598-FF6D-4986-8E3D-0167BA5CE778@math.washington.edu> <4A014AEF.7030901@student.matnat.uio.no> Message-ID: <45239150905061453s62c8dc63p45fa1d483c731256@mail.gmail.com> Hey. I got a few late here. I think the template syntax needs to be simple, "readable" and easy to use and implement (I think this is obvious). I saw here some options really unreadable. I vote for the [] syntax. It's easy to understand and the __getitem__ could be used here. I think it will be easier to choose one syntax when somebody try to implement it (is anybody doing it?). This way, we get more concrete solutions (and problems) for it. Whatever, it's just my opinion :) From robertwb at math.washington.edu Thu May 7 00:52:47 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 15:52:47 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> Message-ID: <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> On May 6, 2009, at 1:33 PM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On May 6, 2009, at 12:48 PM, Dag Sverre Seljebotn wrote: >> >>> Robert Bradshaw wrote: >>>> Just trying to get a syntax, how about >>>> >>>> cdef extern from "header.h": >>>> cdef cclass MyClass[TemplateParamers](BaseCPPClass): >>>> int member >>>> int method(int) >>>> >>>> The only concern I have is whether or not "cdef cclass" is to >>>> similar >>>> to "cdef class" (probably not). Any other ideas? >>> >>> cdef cppclass? >> >> I don't like this one, but I think just for aesthetic reasons. Nor >> cpp_class. > > Myself I'm used to a .cpp suffix on C++ files; I suppose people who > are > used to the competing .cc will have stronger associations with your > proposal. "cclass" gives me no associations at all initially. I could get used to cppclass, but I'm hoping someone comes up with a name that's obviously the "right" one :). >>> Yes, I think cclass is to similar to class. >>> >>> Anyway, when I proposed a Fortran syntax earlier you and Stefan >>> made a >>> good point about the language specification belonging in the "cdef >>> extern" >>> part, as, after all, the "header.h" file contains only C++ code. >> >> Will the compiler need to know that header.h is C++ code? If not, I >> see no reason to declare it. > > No -- it's just the fact that usually all classes within the extern > block > will be a C++ class. I was thinking just plain "cdef class" could then > change meaning, which would have a happy association with the > heap-allocation thing (which C++ classes would share with existing > cdef > classes). > > cdef extern "C++" from "myheader.h": > cdef class A: ... I'm -1 on having "cdef class" mean two very different things. > OTOH, if heap-allocation-and-refcounted semantics isn't selected, one > should use "cdef struct" instead, like Lisandro suggests. Structs are also something different (well, there's a lot of similarity, but the idea of inheritance is a big one), especially if we're looking at heap allocated classes for the first iteration. (I don't think we want to try to do our own refcounting, let the user use new and delete directly if they want to use C++.) - Robert > From greg.ewing at canterbury.ac.nz Thu May 7 03:55:46 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 07 May 2009 13:55:46 +1200 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> Message-ID: <4A023FA2.3040001@canterbury.ac.nz> Robert Bradshaw wrote: > On May 6, 2009, at 1:33 PM, Dag Sverre Seljebotn wrote: >>cdef extern "C++" from "myheader.h": >> cdef class A: ... For Pyrex I'm thinking about using "cdef+" as a general prefix for C++-related declarations, e.g. cdef+ extern from "myheader.hpp": ... cdef+ struct Foo: ... -- Greg From dsurviver at gmail.com Thu May 7 04:26:13 2009 From: dsurviver at gmail.com (Danilo Freitas) Date: Wed, 6 May 2009 23:26:13 -0300 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <4A023FA2.3040001@canterbury.ac.nz> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <4A023FA2.3040001@canterbury.ac.nz> Message-ID: <45239150905061926g58c5cb97hd770669552e7b48a@mail.gmail.com> 2009/5/6 Greg Ewing : > Robert Bradshaw wrote: >> On May 6, 2009, at 1:33 PM, Dag Sverre Seljebotn wrote: >>>cdef extern "C++" from "myheader.h": >>> ? ?cdef class A: ... > > For Pyrex I'm thinking about using "cdef+" as a > general prefix for C++-related declarations, e.g. > > ? cdef+ extern from "myheader.hpp": > ? ? ... > > ? cdef+ struct Foo: > ? ? ... I think this is weird. But it's a solution. I don't see many problems with cppdef. From robertwb at math.washington.edu Thu May 7 04:37:12 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 19:37:12 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <45239150905061926g58c5cb97hd770669552e7b48a@mail.gmail.com> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <4A023FA2.3040001@canterbury.ac.nz> <45239150905061926g58c5cb97hd770669552e7b48a@mail.gmail.com> Message-ID: <40EB49CE-6598-46BA-912D-231D05295E06@math.washington.edu> On May 6, 2009, at 7:26 PM, Danilo Freitas wrote: > 2009/5/6 Greg Ewing : >> Robert Bradshaw wrote: >>> On May 6, 2009, at 1:33 PM, Dag Sverre Seljebotn wrote: >>>> cdef extern "C++" from "myheader.h": >>>> cdef class A: ... >> >> For Pyrex I'm thinking about using "cdef+" as a >> general prefix for C++-related declarations, e.g. >> >> cdef+ extern from "myheader.hpp": >> ... >> >> cdef+ struct Foo: >> ... > I think this is weird. But it's a solution. I don't see many problems > with cppdef. It's too close to cpdef I think. I asked about this during our sage dev meeting today and people seemed to like cdef cppclass Foo: .... - Robert From dsurviver at gmail.com Thu May 7 04:52:35 2009 From: dsurviver at gmail.com (Danilo Freitas) Date: Wed, 6 May 2009 23:52:35 -0300 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <40EB49CE-6598-46BA-912D-231D05295E06@math.washington.edu> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <4A023FA2.3040001@canterbury.ac.nz> <45239150905061926g58c5cb97hd770669552e7b48a@mail.gmail.com> <40EB49CE-6598-46BA-912D-231D05295E06@math.washington.edu> Message-ID: <45239150905061952r56d831d3md0e980a43af2909d@mail.gmail.com> > > It's too close to cpdef I think. I I agree. It may confuse a little. From dagss at student.matnat.uio.no Thu May 7 07:06:15 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 7 May 2009 07:06:15 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> Message-ID: <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> Robert Bradshaw wrote: > On May 6, 2009, at 1:33 PM, Dag Sverre Seljebotn wrote: >> OTOH, if heap-allocation-and-refcounted semantics isn't selected, one >> should use "cdef struct" instead, like Lisandro suggests. > > Structs are also something different (well, there's a lot of > similarity, but the idea of inheritance is a big one), especially if > we're looking at heap allocated classes for the first iteration. (I > don't think we want to try to do our own refcounting, let the user > use new and delete directly if they want to use C++.) Well, they are not something different in C++ :-) If you don't want to try your own refcounting, please, please tell me that the only allowed form of usage of the class will be cdef CppClass* obj Allowing cdef CppClass obj in any form and requiring a manual delete would be very confusing and difficult to remember. (That's what I mean by refcounting -- I imagined the difference between these two syntaxes would be that the latter of these would be refcounted; and subject for automatic stack-allocation-optimization.) Dag Sverre From stefan_ml at behnel.de Thu May 7 08:05:18 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 07 May 2009 08:05:18 +0200 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <4A023FA2.3040001@canterbury.ac.nz> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <4A023FA2.3040001@canterbury.ac.nz> Message-ID: <4A027A1E.7020604@behnel.de> Greg Ewing wrote: > Robert Bradshaw wrote: >> On May 6, 2009, at 1:33 PM, Dag Sverre Seljebotn wrote: >>> cdef extern "C++" from "myheader.h": >>> cdef class A: ... > > For Pyrex I'm thinking about using "cdef+" as a > general prefix for C++-related declarations, e.g. > > cdef+ extern from "myheader.hpp": > ... > > cdef+ struct Foo: > ... ... and if we ever support the D language, we'd need a 'ddef' or 'cdefd'? What about using some kind of qualifier as above, but more like cdef[c++] extern ... or cdef(c++) extern ... or even cdef("c++") extern ... to come a bit closer to pure Python code. That would be easily extensible to cdef[fortran] ... cdef[D] and would simply default to cdef[C] if no qualifier is provided in a plain "cdef". Stefan From stefan_ml at behnel.de Thu May 7 08:23:14 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 07 May 2009 08:23:14 +0200 Subject: [Cython] Storing many, many instances In-Reply-To: <4B519262-77AC-4BC4-A62B-026B4E254FAA@math.washington.edu> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A017E4F.5050800@student.matnat.uio.no> <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A0188D4.1050208@student.matnat.uio.no> <11E61C9F-3175-4473-8B39-20277284E9D1@math.washington.edu> <4B519262-77AC-4BC4-A62B-026B4E254FAA@math.washington.edu> Message-ID: <4A027E52.9070003@behnel.de> Robert Bradshaw wrote: > On May 6, 2009, at 1:08 PM, Carl Witty wrote: >> Yes, but you said that you only implement it within a single module >> (which is what I see in element.c); I'm saying that it should also be >> possible to implement across modules (just make tp_new functions >> non-static, add an extern declaration in the second module, and call >> the function directly). > > Yes, I we could do this. We could also support exported cdef > functions this way, instead of the current function import mechanism. > (Or was there some reason it wasn't done this way in the first place? There are two cdef function export mechanisms: "public" and "api". The second is targeted towards calling functions between separate extension modules (through "__pyx_capi"), while "public" (and declaration in the module .pxd file IIRC) simply removes the name mangling and makes functions non-static for use in other C files linked into the same extension module. So this would mimic the "public" mechanism in a way, although it'd be better to do the normal name mangling for tp_new() even if we export it. Stefan From robertwb at math.washington.edu Thu May 7 08:36:06 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 23:36:06 -0700 Subject: [Cython] Storing many, many instances In-Reply-To: <4A027E52.9070003@behnel.de> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A017E4F.5050800@student.matnat.uio.no> <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A0188D4.1050208@student.matnat.uio.no> <11E61C9F-3175-4473-8B39-20277284E9D1@math.washington.edu> <4B519262-77AC-4BC4-A62B-026B4E254FAA@math.washington.edu> <4A027E52.9070003@behnel.de> Message-ID: <8C4356C0-2011-4519-A5EB-4476E63EA971@math.washington.edu> On May 6, 2009, at 11:23 PM, Stefan Behnel wrote: > Robert Bradshaw wrote: >> On May 6, 2009, at 1:08 PM, Carl Witty wrote: >>> Yes, but you said that you only implement it within a single module >>> (which is what I see in element.c); I'm saying that it should >>> also be >>> possible to implement across modules (just make tp_new functions >>> non-static, add an extern declaration in the second module, and call >>> the function directly). >> >> Yes, I we could do this. We could also support exported cdef >> functions this way, instead of the current function import mechanism. >> (Or was there some reason it wasn't done this way in the first place? > > There are two cdef function export mechanisms: "public" and "api". The > second is targeted towards calling functions between separate > extension > modules (through "__pyx_capi"), while "public" (and declaration in the > module .pxd file IIRC) simply removes the name mangling and makes > functions > non-static for use in other C files linked into the same extension > module. > > So this would mimic the "public" mechanism in a way, although it'd be > better to do the normal name mangling for tp_new() even if we > export it. Yes, I'm thinking of "api" functions, mangled and all, but without the __pyx_capi object. Actually, I don't know if this would work, as the .so files may not be loaded in the right order. - Robert From robertwb at math.washington.edu Thu May 7 08:50:49 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 May 2009 23:50:49 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <4A027A1E.7020604@behnel.de> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <4A023FA2.3040001@canterbury.ac.nz> <4A027A1E.7020604@behnel.de> Message-ID: On May 6, 2009, at 11:05 PM, Stefan Behnel wrote: > > Greg Ewing wrote: >> Robert Bradshaw wrote: >>> On May 6, 2009, at 1:33 PM, Dag Sverre Seljebotn wrote: >>>> cdef extern "C++" from "myheader.h": >>>> cdef class A: ... >> >> For Pyrex I'm thinking about using "cdef+" as a >> general prefix for C++-related declarations, e.g. >> >> cdef+ extern from "myheader.hpp": >> ... >> >> cdef+ struct Foo: >> ... > > ... and if we ever support the D language, we'd need a 'ddef' or > 'cdefd'? > > What about using some kind of qualifier as above, but more like > > cdef[c++] extern ... > > or > > cdef(c++) extern ... > > or even > > cdef("c++") extern ... > > to come a bit closer to pure Python code. > > That would be easily extensible to > > cdef[fortran] ... > cdef[D] > > and would simply default to > > cdef[C] > > if no qualifier is provided in a plain "cdef". Parameterized keywords? Sorry, but that hurts my eyes... :( The only time we'd need to declare a language is for external declarations. I'd rather have cdef extern from "foo.h" language "cpp": ... or cdef extern from cpp "foo.h": ... Actually, the whole cdef keyword here is redundant, maybe one could just write "extern from ..." (though I do like the fact that nearly all non-Python syntax/blocks begin with a cdef). I think this is somewhat tangential to the question of how to declare classes, unless one wants "class" or "struct" to have new meanings in different contexts (which I don't like). - Robert From robertwb at math.washington.edu Thu May 7 09:03:02 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 May 2009 00:03:02 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> Message-ID: <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> On May 6, 2009, at 10:06 PM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On May 6, 2009, at 1:33 PM, Dag Sverre Seljebotn wrote: >>> OTOH, if heap-allocation-and-refcounted semantics isn't selected, >>> one >>> should use "cdef struct" instead, like Lisandro suggests. >> >> Structs are also something different (well, there's a lot of >> similarity, but the idea of inheritance is a big one), especially if >> we're looking at heap allocated classes for the first iteration. (I >> don't think we want to try to do our own refcounting, let the user >> use new and delete directly if they want to use C++.) > > Well, they are not something different in C++ :-) A C++ class is a glorified struct, but one still has plain old structs too. > If you don't want to try your own refcounting, please, please tell > me that > the only allowed form of usage of the class will be > > cdef CppClass* obj This is what I imagined (for now at least). > Allowing > > cdef CppClass obj > > in any form and requiring a manual delete would be very confusing and > difficult to remember. > > (That's what I mean by refcounting -- I imagined the difference > between > these two syntaxes would be that the latter of these would be > refcounted; > and subject for automatic stack-allocation-optimization.) By "refcounting", are you implying global garbage collection of some sort, just like Python objects, or something that emulates being on the stack (e.g. objects are deleted when the scope exits, and all assignments are done by value)? - Robert From dagss at student.matnat.uio.no Thu May 7 09:12:26 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 07 May 2009 09:12:26 +0200 Subject: [Cython] Storing many, many instances In-Reply-To: <8C4356C0-2011-4519-A5EB-4476E63EA971@math.washington.edu> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A017E4F.5050800@student.matnat.uio.no> <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A0188D4.1050208@student.matnat.uio.no> <11E61C9F-3175-4473-8B39-20277284E9D1@math.washington.edu> <4B519262-77AC-4BC4-A62B-026B4E254FAA@math.washington.edu> <4A027E52.9070003@behnel.de> <8C4356C0-2011-4519-A5EB-4476E63EA971@math.washington.edu> Message-ID: <4A0289DA.4000009@student.matnat.uio.no> Robert Bradshaw wrote: > On May 6, 2009, at 11:23 PM, Stefan Behnel wrote: > >> Robert Bradshaw wrote: >>> On May 6, 2009, at 1:08 PM, Carl Witty wrote: >>>> Yes, but you said that you only implement it within a single module >>>> (which is what I see in element.c); I'm saying that it should >>>> also be >>>> possible to implement across modules (just make tp_new functions >>>> non-static, add an extern declaration in the second module, and call >>>> the function directly). >>> Yes, I we could do this. We could also support exported cdef >>> functions this way, instead of the current function import mechanism. >>> (Or was there some reason it wasn't done this way in the first place? >> There are two cdef function export mechanisms: "public" and "api". The >> second is targeted towards calling functions between separate >> extension >> modules (through "__pyx_capi"), while "public" (and declaration in the >> module .pxd file IIRC) simply removes the name mangling and makes >> functions >> non-static for use in other C files linked into the same extension >> module. >> >> So this would mimic the "public" mechanism in a way, although it'd be >> better to do the normal name mangling for tp_new() even if we >> export it. > > Yes, I'm thinking of "api" functions, mangled and all, but without > the __pyx_capi object. Actually, I don't know if this would work, as > the .so files may not be loaded in the right order. And by default, the .so files are loaded with RTLD_PRIVATE, not RTLD_GLOBAL, so that symbols in one isn't available in another. I think you'd have a hard time getting it to work reliably; but if you do it would be cool as it would speed up cross-Cython-module code (?). One could do a combination approach where __pyx_capi is available but a custom Cython classloader (like pyximport) would load the .so as RTLD_GLOBAL. A bit convoluted though, and I wonder whether it should be a priority... In the end, there are "inlineable" functions which are so small that the overhead matters, and "non-inlineable" which do so much anyway that it doesn't matter. The former can be put in inline __init__ code in pxd files, while the latter should be able to live with __pyx_capi. -- Dag Sverre From dagss at student.matnat.uio.no Thu May 7 09:24:52 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 07 May 2009 09:24:52 +0200 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> Message-ID: <4A028CC4.6040903@student.matnat.uio.no> Robert Bradshaw wrote: > On May 6, 2009, at 10:06 PM, Dag Sverre Seljebotn wrote: > >> Robert Bradshaw wrote: >>> On May 6, 2009, at 1:33 PM, Dag Sverre Seljebotn wrote: >>>> OTOH, if heap-allocation-and-refcounted semantics isn't selected, >>>> one >>>> should use "cdef struct" instead, like Lisandro suggests. >>> Structs are also something different (well, there's a lot of >>> similarity, but the idea of inheritance is a big one), especially if >>> we're looking at heap allocated classes for the first iteration. (I >>> don't think we want to try to do our own refcounting, let the user >>> use new and delete directly if they want to use C++.) >> Well, they are not something different in C++ :-) > > A C++ class is a glorified struct, but one still has plain old > structs too. ...which is the same as a class with only public fields in them :-) What I'm getting here at is that we are creating a distinction in Cython which actually does not exist in C++ (rather C++ adds member methods, constructors, private members, and vtables all as orthogonal and independent extensions). But I think that is how it should be. >> If you don't want to try your own refcounting, please, please tell >> me that >> the only allowed form of usage of the class will be >> >> cdef CppClass* obj > > This is what I imagined (for now at least). > >> Allowing >> >> cdef CppClass obj >> >> in any form and requiring a manual delete would be very confusing and >> difficult to remember. >> >> (That's what I mean by refcounting -- I imagined the difference >> between >> these two syntaxes would be that the latter of these would be >> refcounted; >> and subject for automatic stack-allocation-optimization.) > > By "refcounting", are you implying global garbage collection of some > sort, just like Python objects, or something that emulates being on > the stack (e.g. objects are deleted when the scope exits, and all > assignments are done by value)? I was referring to something which have the full semantics of Python objects (after the result of the "New idea for C++ stack allocation" thread, which expored the opposite idea). So cdef CppClass obj = None obj.do_something() # Illegal, it is None obj = CppClass(3) obj.do_something() # OK ...and so on, just like cdef classes. Then stack allocation would be a pure transparent optimization (an important one; you don't really want to be heap-allocating STL iterators for each iteration!) where semantics can be preserved. When interfacing with external code; like with my favourite example: cdef extern from "vector" namespace std: cdef cppclass vector[T]: cdef cppclass iterator: # next/prev operator overloading somehow, # we still disagree here I guess iterator begin() then when one does cdef vector[int].iterator it it = myvec.begin() it would mean getting hold of new std::vector::iterator(myvec.begin()) and refcounting it -- however, since begin() returns by value, it would be legal to instead allocate on stack, as long as semantics are preserved. (Perhaps just changing this to using placement new and skip refcounting.) The alternative to this seems to be the ideas you already rejected. And I must say I like emulating cdef classes better. -- Dag Sverre From stefan_ml at behnel.de Thu May 7 09:39:09 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 7 May 2009 09:39:09 +0200 (CEST) Subject: [Cython] Storing many, many instances In-Reply-To: <8C4356C0-2011-4519-A5EB-4476E63EA971@math.washington.edu> References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A017E4F.5050800@student.matnat.uio.no> <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A0188D4.1050208@student.matnat.uio.no> <11E61C9F-3175-4473-8B39-20277284E9D1@math.washington.edu> <4B519262-77AC-4BC4-A62B-026B4E254FAA@math.washington.edu> <4A027E52.9070003@behnel.de> <8C4356C0-2011-4519-A5EB-4476E63EA971@math.washington.edu> Message-ID: <6c13a7cafc2a4a8deff53300fbffb1a9.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Robert Bradshaw wrote: > Yes, I'm thinking of "api" functions, mangled and all, but without > the __pyx_capi object. Actually, I don't know if this would work, as > the .so files may not be loaded in the right order. Well, it wouldn't work due to symbol export restrictions (as Dag noted). Plus, you'd loose a lot for basically no gain. The current mechanism is arbitrarily portable, almost as fast as a direct call, and provides a simple form of module encapsulation to support future API extensions without breaking existing modules. If you want inlined functions and/or direct calls, use .pxd inline functions and/or include or link your file into the module, instead of using separate modules. Stefan From stefan_ml at behnel.de Thu May 7 09:42:00 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 7 May 2009 09:42:00 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <4A023FA2.3040001@canterbury.ac.nz> <4A027A1E.7020604@behnel.de> Message-ID: Robert Bradshaw wrote: > On May 6, 2009, at 11:05 PM, Stefan Behnel wrote: >> cdef[c++] extern ... > > Parameterized keywords? Sorry, but that hurts my eyes... :( Ok, not on keywords then. What about cdef extern[c++] ... ? Stefan From robertwb at math.washington.edu Thu May 7 09:42:45 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 May 2009 00:42:45 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <4A028CC4.6040903@student.matnat.uio.no> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> Message-ID: On May 7, 2009, at 12:24 AM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On May 6, 2009, at 10:06 PM, Dag Sverre Seljebotn wrote: >> >>> Robert Bradshaw wrote: >>>> On May 6, 2009, at 1:33 PM, Dag Sverre Seljebotn wrote: >>>>> OTOH, if heap-allocation-and-refcounted semantics isn't selected, >>>>> one >>>>> should use "cdef struct" instead, like Lisandro suggests. >>>> Structs are also something different (well, there's a lot of >>>> similarity, but the idea of inheritance is a big one), >>>> especially if >>>> we're looking at heap allocated classes for the first iteration. (I >>>> don't think we want to try to do our own refcounting, let the user >>>> use new and delete directly if they want to use C++.) >>> Well, they are not something different in C++ :-) >> >> A C++ class is a glorified struct, but one still has plain old >> structs too. > > ...which is the same as a class with only public fields in them :-) > > What I'm getting here at is that we are creating a distinction in > Cython > which actually does not exist in C++ (rather C++ adds member methods, > constructors, private members, and vtables all as orthogonal and > independent extensions). > > But I think that is how it should be. Yep, if C++ considers them different enough to allocate two separate keywords for them, than so should we. (I think it'll also be less confusing). >>> If you don't want to try your own refcounting, please, please tell >>> me that >>> the only allowed form of usage of the class will be >>> >>> cdef CppClass* obj >> >> This is what I imagined (for now at least). >> >>> Allowing >>> >>> cdef CppClass obj >>> >>> in any form and requiring a manual delete would be very confusing >>> and >>> difficult to remember. >>> >>> (That's what I mean by refcounting -- I imagined the difference >>> between >>> these two syntaxes would be that the latter of these would be >>> refcounted; >>> and subject for automatic stack-allocation-optimization.) >> >> By "refcounting", are you implying global garbage collection of some >> sort, just like Python objects, or something that emulates being on >> the stack (e.g. objects are deleted when the scope exits, and all >> assignments are done by value)? > > I was referring to something which have the full semantics of Python > objects (after the result of the "New idea for C++ stack allocation" > thread, which expored the opposite idea). > > So > > cdef CppClass obj = None > obj.do_something() # Illegal, it is None > obj = CppClass(3) > obj.do_something() # OK > > ...and so on, just like cdef classes. > > Then stack allocation would be a pure transparent optimization (an > important one; you don't really want to be heap-allocating STL > iterators > for each iteration!) where semantics can be preserved. I agree it makes stuff like STL iterators nice to work with, but can one do cdef CppClass x(): obj = CppClass(3) return obj Or will it be freed on exit unlike a Python object? Also, what about def y(): cdef CppClass a = CppClass(5) cdef CppClass b = a return # don't free a twice... Or perhaps you're thinking about wrapping C++ class instances inside actual objects? - Robert From robertwb at math.washington.edu Thu May 7 09:46:20 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 May 2009 00:46:20 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <4A023FA2.3040001@canterbury.ac.nz> <4A027A1E.7020604@behnel.de> Message-ID: <47F3AE68-2D7B-44DD-90E7-14587EEB0D33@math.washington.edu> On May 7, 2009, at 12:42 AM, Stefan Behnel wrote: > Robert Bradshaw wrote: >> On May 6, 2009, at 11:05 PM, Stefan Behnel wrote: >>> cdef[c++] extern ... >> >> Parameterized keywords? Sorry, but that hurts my eyes... :( > > Ok, not on keywords then. What about > > cdef extern[c++] ... "extern" feels a lot like a keyword too. - Robert From dagss at student.matnat.uio.no Thu May 7 09:58:11 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 7 May 2009 09:58:11 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> Message-ID: Robert Bradshaw wrote: > On May 7, 2009, at 12:24 AM, Dag Sverre Seljebotn wrote: >> What I'm getting here at is that we are creating a distinction in >> Cython >> which actually does not exist in C++ (rather C++ adds member methods, >> constructors, private members, and vtables all as orthogonal and >> independent extensions). >> >> But I think that is how it should be. > > Yep, if C++ considers them different enough to allocate two separate > keywords for them, than so should we. (I think it'll also be less > confusing). But C++ doesn't!! class Foo { int x; public: int y; void z(); }; is EXACTLY the same as struct Foo { int y; void z() private: int x; }; Whatever the distinction is supposed to be in Cython, it cannot be the default private/public, as only the public stuff is available to us anyway! I've been using the distinction to mean "these structs should be reference-counted, not have stack-allocated semantics like we use for C structs". If one doesn't plan to ever go for a reference counted scheme, we should just use plain struct. >>>> If you don't want to try your own refcounting, please, please tell >>>> me that >>>> the only allowed form of usage of the class will be >>>> >>>> cdef CppClass* obj >>> >>> This is what I imagined (for now at least). >>> >>>> Allowing >>>> >>>> cdef CppClass obj >>>> >>>> in any form and requiring a manual delete would be very confusing >>>> and >>>> difficult to remember. >>>> >>>> (That's what I mean by refcounting -- I imagined the difference >>>> between >>>> these two syntaxes would be that the latter of these would be >>>> refcounted; >>>> and subject for automatic stack-allocation-optimization.) >>> >>> By "refcounting", are you implying global garbage collection of some >>> sort, just like Python objects, or something that emulates being on >>> the stack (e.g. objects are deleted when the scope exits, and all >>> assignments are done by value)? >> >> I was referring to something which have the full semantics of Python >> objects (after the result of the "New idea for C++ stack allocation" >> thread, which expored the opposite idea). >> >> So >> >> cdef CppClass obj = None >> obj.do_something() # Illegal, it is None >> obj = CppClass(3) >> obj.do_something() # OK >> >> ...and so on, just like cdef classes. >> >> Then stack allocation would be a pure transparent optimization (an >> important one; you don't really want to be heap-allocating STL >> iterators >> for each iteration!) where semantics can be preserved. > > I agree it makes stuff like STL iterators nice to work with, but can > one do > > cdef CppClass x(): > obj = CppClass(3) > return obj > > Or will it be freed on exit unlike a Python object? Also, what about > > def y(): > cdef CppClass a = CppClass(5) > cdef CppClass b = a > return # don't free a twice... > > Or perhaps you're thinking about wrapping C++ class instances inside > actual objects? Yes! Allocation would happen through an actual (as light-weight as possible, and probably mostly opaque, perhaps doing as much as printing the C++ name of the class in its repr) Python object; and assigning it to "cdef CppClass a" would basically make Cython automatically do all accesses (except passing around references) on the contained pointer. Again -- this is a larger debate with huge consequences for how one should do operator overloading etc. For instance, += would have to be implemented like it += 1 would mean it = please_refcount_iteratorclassname(new std::vector::iterator(it + 1)) However, this kind of semantics ensures that we avoid that cdef vector[int].iterator a, b a = vec.begin() b = a a += 1 # also changes b which seems to be the alternative (where a and b are just kept on the stack and we somehow circumvent the constructor business). This is, BTW, a good reason why I proposed calling operators by "rewrapping" them, because then a = vec.begin() b = a a.next() # also changes b, through calling a++ makes perfect sense! I'm happy that only "CppClass*" will be allowed at first, which takes much of this worry away. However note that this might have consequences for how operators should be defined (I haven't seen a concrete proposal for what you are thinking there so I don't know how it would fare). Dag Sverre From stefan_ml at behnel.de Thu May 7 10:10:34 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 7 May 2009 10:10:34 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <47F3AE68-2D7B-44DD-90E7-14587EEB0D33@math.washington.edu> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <4A023FA2.3040001@canterbury.ac.nz> <4A027A1E.7020604@behnel.de> <47F3AE68-2D7B-44DD-90E7-14587EEB0D33@math.washington.edu> Message-ID: Robert Bradshaw wrote: > On May 7, 2009, at 12:42 AM, Stefan Behnel wrote: > >> Robert Bradshaw wrote: >>> On May 6, 2009, at 11:05 PM, Stefan Behnel wrote: >>>> cdef[c++] extern ... >>> >>> Parameterized keywords? Sorry, but that hurts my eyes... :( >> >> Ok, not on keywords then. What about >> >> cdef extern[c++] ... > > "extern" feels a lot like a keyword too. Fine, but you get the idea, right? In a way, the declarations *are* parametrised in that they come from different languages. I think it makes sense to add that to the "extern" 'keyword' to say: "this external declaration obeys C++ semantics". Stefan From stefan_ml at behnel.de Thu May 7 10:24:02 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 7 May 2009 10:24:02 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> Message-ID: <472918d332225688ad7bf5e86e1c9d2a.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On May 7, 2009, at 12:24 AM, Dag Sverre Seljebotn wrote: >>> What I'm getting here at is that we are creating a distinction in >>> Cython >>> which actually does not exist in C++ (rather C++ adds member methods, >>> constructors, private members, and vtables all as orthogonal and >>> independent extensions). >>> >>> But I think that is how it should be. >> >> Yep, if C++ considers them different enough to allocate two separate >> keywords for them, than so should we. (I think it'll also be less >> confusing). > > But C++ doesn't!! > > class Foo { > int x; > public: > int y; > void z(); > }; > > is EXACTLY the same as > > struct Foo { > int y; > void z() > private: > int x; > }; > > Whatever the distinction is supposed to be in Cython, it cannot be the > default private/public, as only the public stuff is available to us > anyway! > > I've been using the distinction to mean "these structs should be > reference-counted, not have stack-allocated semantics like we use for C > structs". If one doesn't plan to ever go for a reference counted scheme, > we should just use plain struct. Ok, so these are orthogonal goals, IIUC. In that case, I'd vote for plain "struct" to avoid the ambiguity with Python's "class". Then, struct methods would behave different for C and C++ semantics in that the C++ code would call the class method, while the C code would use the transformed plain function call. C struct methods can only be defined in Cython (.pyx) code, whereas C++ struct methods can only be defined in external C++ declarations. Note that we can always switch to "class" or whatever other declaration later on if we want to enable ref-counting and consider it similar enough to Python's class behaviour. That would be an additional feature. Stefan From dagss at student.matnat.uio.no Thu May 7 10:35:38 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 7 May 2009 10:35:38 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <472918d332225688ad7bf5e86e1c9d2a.squirrel@groupware.dvs.informatik.tu -darmstadt.de> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <472918d332225688ad7bf5e86e1c9d2a.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <3292c0f02874a62289df377e877c024b.squirrel@webmail.uio.no> Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >> Robert Bradshaw wrote: >>> On May 7, 2009, at 12:24 AM, Dag Sverre Seljebotn wrote: >>>> What I'm getting here at is that we are creating a distinction in >>>> Cython >>>> which actually does not exist in C++ (rather C++ adds member methods, >>>> constructors, private members, and vtables all as orthogonal and >>>> independent extensions). >>>> >>>> But I think that is how it should be. >>> >>> Yep, if C++ considers them different enough to allocate two separate >>> keywords for them, than so should we. (I think it'll also be less >>> confusing). >> >> But C++ doesn't!! >> >> class Foo { >> int x; >> public: >> int y; >> void z(); >> }; >> >> is EXACTLY the same as >> >> struct Foo { >> int y; >> void z() >> private: >> int x; >> }; >> >> Whatever the distinction is supposed to be in Cython, it cannot be the >> default private/public, as only the public stuff is available to us >> anyway! >> >> I've been using the distinction to mean "these structs should be >> reference-counted, not have stack-allocated semantics like we use for C >> structs". If one doesn't plan to ever go for a reference counted scheme, >> we should just use plain struct. > > Ok, so these are orthogonal goals, IIUC. In that case, I'd vote for plain > "struct" to avoid the ambiguity with Python's "class". > > Then, struct methods would behave different for C and C++ semantics in > that the C++ code would call the class method, while the C code would use > the transformed plain function call. C struct methods can only be defined > in Cython (.pyx) code, whereas C++ struct methods can only be defined in > external C++ declarations. That's a very constructive idea. But couldn't we just use cdef extern from "myheader.h": cdef struct MyStruct: # External declaration int cppfunc(self, float) # Struct convenience wrapper int cythonwrappingfunc(self): print self.a (Here cppfunc has "self" just for visual consistency, it could be dropped.) > Note that we can always switch to "class" or whatever other declaration > later on if we want to enable ref-counting and consider it similar enough > to Python's class behaviour. That would be an additional feature. I'm a bit wary of encouraging people to start writing cdef CppClass a which would then be possible (if and only if the constructor has no arguments -- which is rather rare anyway in my experience); because it would start calling constructors and destructors, interfering with our knowledge of control flow. A feature of "cppclass" could be that it disallowed code like cdef CppClass a with "Error: Cannot allocate C++ object on stack". Dag Sverre From dagss at student.matnat.uio.no Thu May 7 10:37:30 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 7 May 2009 10:37:30 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> Message-ID: <5305e3422c242075b014430f42fc75ca.squirrel@webmail.uio.no> > Robert Bradshaw wrote: >> On May 7, 2009, at 12:24 AM, Dag Sverre Seljebotn wrote: >>> What I'm getting here at is that we are creating a distinction in >>> Cython >>> which actually does not exist in C++ (rather C++ adds member methods, >>> constructors, private members, and vtables all as orthogonal and >>> independent extensions). >>> >>> But I think that is how it should be. >> >> Yep, if C++ considers them different enough to allocate two separate >> keywords for them, than so should we. (I think it'll also be less >> confusing). > > But C++ doesn't!! > > class Foo { > int x; > public: > int y; > void z(); > }; > > is EXACTLY the same as > > struct Foo { > int y; > void z() > private: > int x; > }; > > Whatever the distinction is supposed to be in Cython, it cannot be the > default private/public, as only the public stuff is available to us > anyway! > > I've been using the distinction to mean "these structs should be > reference-counted, not have stack-allocated semantics like we use for C > structs". If one doesn't plan to ever go for a reference counted scheme, > we should just use plain struct. > >>>>> If you don't want to try your own refcounting, please, please tell >>>>> me that >>>>> the only allowed form of usage of the class will be >>>>> >>>>> cdef CppClass* obj >>>> >>>> This is what I imagined (for now at least). >>>> >>>>> Allowing >>>>> >>>>> cdef CppClass obj >>>>> >>>>> in any form and requiring a manual delete would be very confusing >>>>> and >>>>> difficult to remember. >>>>> >>>>> (That's what I mean by refcounting -- I imagined the difference >>>>> between >>>>> these two syntaxes would be that the latter of these would be >>>>> refcounted; >>>>> and subject for automatic stack-allocation-optimization.) >>>> >>>> By "refcounting", are you implying global garbage collection of some >>>> sort, just like Python objects, or something that emulates being on >>>> the stack (e.g. objects are deleted when the scope exits, and all >>>> assignments are done by value)? >>> >>> I was referring to something which have the full semantics of Python >>> objects (after the result of the "New idea for C++ stack allocation" >>> thread, which expored the opposite idea). >>> >>> So >>> >>> cdef CppClass obj = None >>> obj.do_something() # Illegal, it is None >>> obj = CppClass(3) >>> obj.do_something() # OK >>> >>> ...and so on, just like cdef classes. >>> >>> Then stack allocation would be a pure transparent optimization (an >>> important one; you don't really want to be heap-allocating STL >>> iterators >>> for each iteration!) where semantics can be preserved. >> >> I agree it makes stuff like STL iterators nice to work with, but can >> one do >> >> cdef CppClass x(): >> obj = CppClass(3) >> return obj >> >> Or will it be freed on exit unlike a Python object? Also, what about >> >> def y(): >> cdef CppClass a = CppClass(5) >> cdef CppClass b = a >> return # don't free a twice... >> >> Or perhaps you're thinking about wrapping C++ class instances inside >> actual objects? > > Yes! Allocation would happen through an actual (as light-weight as > possible, and probably mostly opaque, perhaps doing as much as printing > the C++ name of the class in its repr) Python object; and assigning it to > "cdef CppClass a" would basically make Cython automatically do all > accesses (except passing around references) on the contained pointer. > > Again -- this is a larger debate with huge consequences for how one should > do operator overloading etc. For instance, += would have to be implemented > like > > it += 1 > > would mean > > it = please_refcount_iteratorclassname(new std::vector::iterator(it + > 1)) > > However, this kind of semantics ensures that we avoid that > > cdef vector[int].iterator a, b > a = vec.begin() > b = a > a += 1 # also changes b > > which seems to be the alternative (where a and b are just kept on the > stack and we somehow circumvent the constructor business). Sorry, it's too early in the morning :-) please disregard this last bit, it doesn't make sense .. "b = a" make a copy if stack-allocation, which makes sure that "a += 1" only modifies a. Dag Sverre From stefan_ml at behnel.de Thu May 7 10:39:02 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 7 May 2009 10:39:02 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> Message-ID: <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: > Allocation would happen through an actual (as light-weight as > possible, and probably mostly opaque, perhaps doing as much as printing > the C++ name of the class in its repr) Python object That wouldn't work, though. Imagine you'd return the C++ object pointer from a function. What would happen to the Python object that holds it? How would you keep the ref-counting context for it? If we enable ref-counting for C++ objects, we should do it ourselves, rather than relying on Python object creation. Not sure if that is doable at all, but Python objects simply don't work here. Stefan From robertwb at math.washington.edu Thu May 7 10:44:07 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 May 2009 01:44:07 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> Message-ID: <16C56E2D-7A3A-405D-986C-377D043E7D8F@math.washington.edu> On May 7, 2009, at 12:58 AM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On May 7, 2009, at 12:24 AM, Dag Sverre Seljebotn wrote: >>> What I'm getting here at is that we are creating a distinction in >>> Cython >>> which actually does not exist in C++ (rather C++ adds member >>> methods, >>> constructors, private members, and vtables all as orthogonal and >>> independent extensions). >>> >>> But I think that is how it should be. >> >> Yep, if C++ considers them different enough to allocate two separate >> keywords for them, than so should we. (I think it'll also be less >> confusing). > > But C++ doesn't!! > > class Foo { > int x; > public: > int y; > void z(); > }; > > is EXACTLY the same as > > struct Foo { > int y; > void z() > private: > int x; > }; I agree it's a tiny distinction, but it *was* enough to convince the authors of C++ that a new keyword was warranted. > Whatever the distinction is supposed to be in Cython, it cannot be the > default private/public, as only the public stuff is available to us > anyway! > > I've been using the distinction to mean "these structs should be > reference-counted, not have stack-allocated semantics like we use > for C > structs". If one doesn't plan to ever go for a reference counted > scheme, > we should just use plain struct. Hmm... I've been thinking of structs as plain old dumb C structs, as they always have been in Cython, and classes as the C++ version with all the goodies like inheritance, polymorphism, operator overloading, etc. BTW, Not ruling out stack/automanaged memory management down the road. >> Or perhaps you're thinking about wrapping C++ class instances inside >> actual objects? > > Yes! OK, that changes everything. It still has its issues though. I'd assume you'd try to delete the C++ object on deallocation? But what if two distinct Python objects refer to the same C++ object? Or the C+ + object (pointer) gets passed and stored in some library somewhere? I don't see a way to ensure a one-to-one relationship between C++ objects and Python objects--short of having stack allocated semantics (i.e. it goes out of scope and is deconstructed when the function exits) I don't see how one can automatically refcount C++ classes. > Allocation would happen through an actual (as light-weight as > possible, and probably mostly opaque, perhaps doing as much as > printing > the C++ name of the class in its repr) Python object; and assigning > it to > "cdef CppClass a" would basically make Cython automatically do all > accesses (except passing around references) on the contained pointer. You know people are then going to want to have access to all it's methods and members, so that just declaring the C++ class will be creating a full-fledged wrapper around it. I can just see the expectations now :). [...] > I'm happy that only "CppClass*" will be allowed at first, which > takes much > of this worry away. One drawback is that it makes not having a good referencing operator a pain. One will have to write a lot of code like c[0] = a[0] + b[0] * c[0] Cython does automatically promote a.b to a->b when a is a pointer type, and it'd be really nice to do something similar, but, e.g., CppClass* + int already has meaning. > However note that this might have consequences for how > operators should be defined (I haven't seen a concrete proposal for > what > you are thinking there so I don't know how it would fare). http://www.mail-archive.com/cython-dev at codespeak.net/msg05140.html - Robert From dagss at student.matnat.uio.no Thu May 7 10:50:28 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 7 May 2009 10:50:28 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu -darmstadt.de> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> > Dag Sverre Seljebotn wrote: >> Allocation would happen through an actual (as light-weight as >> possible, and probably mostly opaque, perhaps doing as much as printing >> the C++ name of the class in its repr) Python object > > That wouldn't work, though. Imagine you'd return the C++ object pointer > from a function. What would happen to the Python object that holds it? How > would you keep the ref-counting context for it? There would be no coercion between pointers and refcounted variables. I.e. if you have cdef extern CppObject* foo() you can NOT do cdef CppObject obj = foo() It would only work when C++ returned an object *by value*: cdef extern CppObject foo() cdef CppObject obj = foo() This would then heap-allocate a CppObject, passing in the object that is returned to the copy constructor. We *could* perhaps add manual conversion from pointers, with a "managed" boolean keyword etc., but that is something else entirely. Robert, this answers your problems with it as well I believe. Dag Sverre From dagss at student.matnat.uio.no Thu May 7 11:03:34 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 7 May 2009 11:03:34 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <16C56E2D-7A3A-405D-986C-377D043E7D8F@math.washington.edu> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <16C56E2D-7A3A-405D-986C-377D043E7D8F@math.washington.edu> Message-ID: <05801ec60573b09b3f0bc28c72764050.squirrel@webmail.uio.no> > On May 7, 2009, at 12:58 AM, Dag Sverre Seljebotn wrote: > >> Robert Bradshaw wrote: >>> On May 7, 2009, at 12:24 AM, Dag Sverre Seljebotn wrote: >>>> What I'm getting here at is that we are creating a distinction in >>>> Cython >>>> which actually does not exist in C++ (rather C++ adds member >>>> methods, >>>> constructors, private members, and vtables all as orthogonal and >>>> independent extensions). >>>> >>>> But I think that is how it should be. >>> >>> Yep, if C++ considers them different enough to allocate two separate >>> keywords for them, than so should we. (I think it'll also be less >>> confusing). >> >> But C++ doesn't!! >> >> class Foo { >> int x; >> public: >> int y; >> void z(); >> }; >> >> is EXACTLY the same as >> >> struct Foo { >> int y; >> void z() >> private: >> int x; >> }; > > I agree it's a tiny distinction, but it *was* enough to convince the > authors of C++ that a new keyword was warranted. > >> Whatever the distinction is supposed to be in Cython, it cannot be the >> default private/public, as only the public stuff is available to us >> anyway! >> >> I've been using the distinction to mean "these structs should be >> reference-counted, not have stack-allocated semantics like we use >> for C >> structs". If one doesn't plan to ever go for a reference counted >> scheme, >> we should just use plain struct. > > Hmm... I've been thinking of structs as plain old dumb C structs, as > they always have been in Cython, and classes as the C++ version with > all the goodies like inheritance, polymorphism, operator overloading, > etc. > > BTW, Not ruling out stack/automanaged memory management down the road. > >>> Or perhaps you're thinking about wrapping C++ class instances inside >>> actual objects? >> >> Yes! > > OK, that changes everything. It still has its issues though. I'd > assume you'd try to delete the C++ object on deallocation? But what > if two distinct Python objects refer to the same C++ object? Or the C+ > + object (pointer) gets passed and stored in some library somewhere? All crossings between Cython and C++ would pass the object by value. You'd still need to use & to get a pointer, which would be similar to str -> char*. > I don't see a way to ensure a one-to-one relationship between C++ > objects and Python objects--short of having stack allocated semantics > (i.e. it goes out of scope and is deconstructed when the function > exits) I don't see how one can automatically refcount C++ classes. I don't see the problem, sorry. > You know people are then going to want to have access to all it's > methods and members, so that just declaring the C++ class will be > creating a full-fledged wrapper around it. I can just see the > expectations now :). Well, obviously that is down the pipeline, but no need to go there now. I just want *some* way to avoid having to write it[0][0] to dereference an STL iterator. Everything else is secondary really :-) > [...] > >> I'm happy that only "CppClass*" will be allowed at first, which >> takes much >> of this worry away. > > One drawback is that it makes not having a good referencing operator > a pain. One will have to write a lot of code like > > c[0] = a[0] + b[0] * c[0] > > Cython does automatically promote a.b to a->b when a is a pointer > type, and it'd be really nice to do something similar, but, e.g., > CppClass* + int already has meaning. Yes, that's the problem my Python-object-solution was designed to take away. > >> However note that this might have consequences for how >> operators should be defined (I haven't seen a concrete proposal for >> what >> you are thinking there so I don't know how it would fare). > > http://www.mail-archive.com/cython-dev at codespeak.net/msg05140.html Hmm. So how would e.g. += be treated? Dag Sverre From dagss at student.matnat.uio.no Thu May 7 11:12:23 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 7 May 2009 11:12:23 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> Message-ID: >> Dag Sverre Seljebotn wrote: >>> Allocation would happen through an actual (as light-weight as >>> possible, and probably mostly opaque, perhaps doing as much as printing >>> the C++ name of the class in its repr) Python object >> >> That wouldn't work, though. Imagine you'd return the C++ object pointer >> from a function. What would happen to the Python object that holds it? >> How >> would you keep the ref-counting context for it? > > There would be no coercion between pointers and refcounted variables. I.e. > if you have > > cdef extern CppObject* foo() > > you can NOT do > > cdef CppObject obj = foo() > > It would only work when C++ returned an object *by value*: The deeper point here is that Cython has no concept of objects by value, which C++ has, and I'm trying to avoid that from creeping in. For instance you have CppObject a, b; b = a; a.set_foo(4); // does not set foo of b which would look strange in Cython. So I have two goals: - convenient STL iterator syntax - avoiding the above semantics in Cython and everything else, though it might seem complicated, kind of flows from that (and it's not hard to implement). Dag Sverre From stefan_ml at behnel.de Thu May 7 11:14:26 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 7 May 2009 11:14:26 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> Message-ID: <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Dag Sverre Seljebotn wrote: >> Dag Sverre Seljebotn wrote: >>> Allocation would happen through an actual (as light-weight as >>> possible, and probably mostly opaque, perhaps doing as much as printing >>> the C++ name of the class in its repr) Python object >> >> That wouldn't work, though. Imagine you'd return the C++ object pointer >> from a function. What would happen to the Python object that holds it? >> How would you keep the ref-counting context for it? > > There would be no coercion between pointers and refcounted variables. I.e. > if you have > > cdef extern CppObject* foo() > > you can NOT do > > cdef CppObject obj = foo() > > It would only work when C++ returned an object *by value*: > > cdef extern CppObject foo() > cdef CppObject obj = foo() > > This would then heap-allocate a CppObject, passing in the object that is > returned to the copy constructor. Ah, ok, got it. So they'd basically be distinct types and you cannot (implicitly) coerce a C++ object to a pointer. That should work then. Can C++ objects override their own allocation, or could you use a PyVarObject to allocate both the object and the Cython wrapper at once? (targeting heap allocation here, I assume that stack allocated objects do not need ref-counting anyway?) Stefan From robertwb at math.washington.edu Thu May 7 11:27:12 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 May 2009 02:27:12 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <05801ec60573b09b3f0bc28c72764050.squirrel@webmail.uio.no> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <16C56E2D-7A3A-405D-986C-377D043E7D8F@math.washington.edu> <05801ec60573b09b3f0bc28c72764050.squirrel@webmail.uio.no> Message-ID: <38519F5C-179F-4292-8DD0-32AAF2DBB404@math.washington.edu> On May 7, 2009, at 2:03 AM, Dag Sverre Seljebotn wrote: >> OK, that changes everything. It still has its issues though. I'd >> assume you'd try to delete the C++ object on deallocation? But what >> if two distinct Python objects refer to the same C++ object? Or >> the C+ >> + object (pointer) gets passed and stored in some library somewhere? > > All crossings between Cython and C++ would pass the object by > value. You'd > still need to use & to get a pointer, which would be similar to str -> > char*. > >> I don't see a way to ensure a one-to-one relationship between C++ >> objects and Python objects--short of having stack allocated semantics >> (i.e. it goes out of scope and is deconstructed when the function >> exits) I don't see how one can automatically refcount C++ classes. > > I don't see the problem, sorry. I hadn't seen your retraction of "also changes b" so I thought you were trying to pass things around by reference. Then what does b = a actually do? Construct a new Python object, then copy by value into it, and assign to be? So a is b is now False? >> You know people are then going to want to have access to all it's >> methods and members, so that just declaring the C++ class will be >> creating a full-fledged wrapper around it. I can just see the >> expectations now :). > > Well, obviously that is down the pipeline, but no need to go there > now. > > I just want *some* way to avoid having to write it[0][0] to > dereference an > STL iterator. Everything else is secondary really :-) Yeah, me too :-) > >> >>> However note that this might have consequences for how >>> operators should be defined (I haven't seen a concrete proposal for >>> what >>> you are thinking there so I don't know how it would fare). >> >> http://www.mail-archive.com/cython-dev at codespeak.net/msg05140.html > > Hmm. So how would e.g. += be treated? Perhaps we could declare it as having an __iadd__ operator, but realistically it might make sense to disallow all assignment operators for the first pass. - Robert From robertwb at math.washington.edu Thu May 7 11:39:53 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 May 2009 02:39:53 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: On May 7, 2009, at 2:14 AM, Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: >>> Dag Sverre Seljebotn wrote: >>>> Allocation would happen through an actual (as light-weight as >>>> possible, and probably mostly opaque, perhaps doing as much as >>>> printing >>>> the C++ name of the class in its repr) Python object >>> >>> That wouldn't work, though. Imagine you'd return the C++ object >>> pointer >>> from a function. What would happen to the Python object that >>> holds it? >>> How would you keep the ref-counting context for it? >> >> There would be no coercion between pointers and refcounted >> variables. I.e. >> if you have >> >> cdef extern CppObject* foo() >> >> you can NOT do >> >> cdef CppObject obj = foo() >> >> It would only work when C++ returned an object *by value*: >> >> cdef extern CppObject foo() >> cdef CppObject obj = foo() >> >> This would then heap-allocate a CppObject, passing in the object >> that is >> returned to the copy constructor. > > Ah, ok, got it. So they'd basically be distinct types and you cannot > (implicitly) coerce a C++ object to a pointer. That should work then. Will dereferencing a CppObject pointer always create a Python object then? You could take the address of a CppObject and get a volitile pointer (just as with bytes -> char*), right? > Can C++ objects override their own allocation, or could you use a > PyVarObject to allocate both the object and the Cython wrapper at > once? Yes, you can, we do that in Sage. I don't think one can declare a struct member to be an object without a no-args constructor without extra work though, so one would have to do the same hackery that was proposed for stack-allocated objects. > (targeting heap allocation here, I assume that stack allocated > objects do > not need ref-counting anyway?) Yep, stack-allocated objects disappear with the stack. - Robert From dagss at student.matnat.uio.no Thu May 7 11:47:05 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 7 May 2009 11:47:05 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <38519F5C-179F-4292-8DD0-32AAF2DBB404@math.washington.edu> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <16C56E2D-7A3A-405D-986C-377D043E7D8F@math.washington.edu> <05801ec60573b09b3f0bc28c72764050.squirrel@webmail.uio.no> <38519F5C-179F-4292-8DD0-32AAF2DBB404@math.washington.edu> Message-ID: <30eeafae0e895c6e1aca708240db3cc3.squirrel@webmail.uio.no> Robert wrote: > On May 7, 2009, at 2:03 AM, Dag Sverre Seljebotn wrote: > >>> OK, that changes everything. It still has its issues though. I'd >>> assume you'd try to delete the C++ object on deallocation? But what >>> if two distinct Python objects refer to the same C++ object? Or >>> the C+ >>> + object (pointer) gets passed and stored in some library somewhere? >> >> All crossings between Cython and C++ would pass the object by >> value. You'd >> still need to use & to get a pointer, which would be similar to str -> >> char*. >> >>> I don't see a way to ensure a one-to-one relationship between C++ >>> objects and Python objects--short of having stack allocated semantics >>> (i.e. it goes out of scope and is deconstructed when the function >>> exits) I don't see how one can automatically refcount C++ classes. >> >> I don't see the problem, sorry. > > I hadn't seen your retraction of "also changes b" so I thought you > were trying to pass things around by reference. Then what does That whole explanation was a mess, as my daughter waked up from her sleep at the time before I could improve it :-) > b = a > > actually do? Construct a new Python object, then copy by value into > it, and assign to be? So > > a is b > > is now False? I'll just forget about earlier explanations. But what I am advocating is to just have a regular Python reference transfer, no C++ operators involved. With that approach, cdef CppObject a += 2 would mean the same as a = CppObject(a + 2) i.e. a new Python wrapper passing in a+2 as the argument. (C++ classes almost always has this "copy constructor", or else they can't be passed by value in the first place anyway). I am agreeing with Stefan that this is probably orthogonal. It is something you might want to support for C structs as well, so that cdef CWidget a, b b = a a.show_caption = True # also changes b would hold, i.e. an ability to give C structs Python semantics. This would especially be useful given the recent method-on-struct idea, as b = a a.set_value(3) # does not modify b is a "new" wierd exception from Python semantics. I still believe it should be discussed now because a) it heavily impacts how one thinks about C++ operators b) it is a way to solve the construction/destruction problem with C++ classes but as a feature, it could be mostly orthogonal. Dag Sverre From dagss at student.matnat.uio.no Thu May 7 11:56:33 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 7 May 2009 11:56:33 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> Message-ID: <716739d959ac4aa061bfe5a55f1b64fe.squirrel@webmail.uio.no> Robert wrote: > On May 7, 2009, at 2:14 AM, Stefan Behnel wrote: > >> Dag Sverre Seljebotn wrote: >>>> Dag Sverre Seljebotn wrote: >>>>> Allocation would happen through an actual (as light-weight as >>>>> possible, and probably mostly opaque, perhaps doing as much as >>>>> printing >>>>> the C++ name of the class in its repr) Python object >>>> >>>> That wouldn't work, though. Imagine you'd return the C++ object >>>> pointer >>>> from a function. What would happen to the Python object that >>>> holds it? >>>> How would you keep the ref-counting context for it? >>> >>> There would be no coercion between pointers and refcounted >>> variables. I.e. >>> if you have >>> >>> cdef extern CppObject* foo() >>> >>> you can NOT do >>> >>> cdef CppObject obj = foo() >>> >>> It would only work when C++ returned an object *by value*: >>> >>> cdef extern CppObject foo() >>> cdef CppObject obj = foo() >>> >>> This would then heap-allocate a CppObject, passing in the object >>> that is >>> returned to the copy constructor. >> >> Ah, ok, got it. So they'd basically be distinct types and you cannot >> (implicitly) coerce a C++ object to a pointer. That should work then. > > Will dereferencing a CppObject pointer always create a Python object > then? You could take the address of a CppObject and get a volitile > pointer (just as with bytes -> char*), right? &obj would be like bytes -> char*, same caveats ptr[0] would (in my proposal anyway) return a Python object; BUT if that is immedeately sent off to a C++ function by value it is just a middle-person that can be left out safely, as an optimization. > Yes, you can, we do that in Sage. I don't think one can declare a > struct member to be an object without a no-args constructor without > extra work though, so one would have to do the same hackery that was > proposed for stack-allocated objects. I don't know enough CPython, but another (cleaner) option could be to allocate the whole PyObject using C++ new (which would pass on the constructor arguments to the field using the "MyClass(int arg) : myfield(arg) {}" syntax), and override tp_alloc to return that object? If it is possible to tell tp_alloc about that somehow. Another idea BTW is that we can use C++ templates to generate these Python wrappers, so we only need to output one (manually written) template to our code, and can then use CppPyWrapper and it could instantiate the right type structure through more template programming etc. Dag Sverre From dagss at student.matnat.uio.no Thu May 7 12:00:57 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 7 May 2009 12:00:57 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <30eeafae0e895c6e1aca708240db3cc3.squirrel@webmail.uio.no> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <16C56E2D-7A3A-405D-986C-377D043E7D8F@math.washington.edu> <05801ec60573b09b3f0bc28c72764050.squirrel@webmail.uio.no> <38519F5C-179F-4292-8DD0-32AAF2DBB404@math.washington.edu> <30eeafae0e895c6e1aca708240db3cc3.squirrel@webmail.uio.no> Message-ID: > Robert wrote: >> On May 7, 2009, at 2:03 AM, Dag Sverre Seljebotn wrote: >> >>>> OK, that changes everything. It still has its issues though. I'd >>>> assume you'd try to delete the C++ object on deallocation? But what >>>> if two distinct Python objects refer to the same C++ object? Or >>>> the C+ >>>> + object (pointer) gets passed and stored in some library somewhere? >>> >>> All crossings between Cython and C++ would pass the object by >>> value. You'd >>> still need to use & to get a pointer, which would be similar to str -> >>> char*. >>> >>>> I don't see a way to ensure a one-to-one relationship between C++ >>>> objects and Python objects--short of having stack allocated semantics >>>> (i.e. it goes out of scope and is deconstructed when the function >>>> exits) I don't see how one can automatically refcount C++ classes. >>> >>> I don't see the problem, sorry. >> >> I hadn't seen your retraction of "also changes b" so I thought you >> were trying to pass things around by reference. Then what does > > That whole explanation was a mess, as my daughter waked up from her sleep > at the time before I could improve it :-) > >> b = a >> >> actually do? Construct a new Python object, then copy by value into >> it, and assign to be? So >> >> a is b >> >> is now False? > > I'll just forget about earlier explanations. > > But what I am advocating is to just have a regular Python reference > transfer, no C++ operators involved. > > With that approach, > > cdef CppObject > a += 2 > > would mean the same as > > a = CppObject(a + 2) > > i.e. a new Python wrapper passing in a+2 as the argument. (C++ classes > almost always has this "copy constructor", or else they can't be passed by > value in the first place anyway). > > I am agreeing with Stefan that this is probably orthogonal. It is > something you might want to support for C structs as well, so that > > cdef CWidget a, b > b = a > a.show_caption = True # also changes b > > would hold, i.e. an ability to give C structs Python semantics. I keep making mistakes. cdef CWidget a, b # a and b are None a = CWidget() b = a a.show_caption = True # also changes b (Obviously you'd change the syntax for declaring the CWidget struct heavily, probably call it something with class). Dag Sverre From dagss at student.matnat.uio.no Thu May 7 12:10:09 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 7 May 2009 12:10:09 +0200 (CEST) Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <716739d959ac4aa061bfe5a55f1b64fe.squirrel@webmail.uio.no> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <716739d959ac4aa061bfe5a55f1b64fe.squirrel@webmail.uio.no> Message-ID: I wrote: > Robert wrote: >> Will dereferencing a CppObject pointer always create a Python object >> then? You could take the address of a CppObject and get a volitile >> pointer (just as with bytes -> char*), right? > > &obj would be like bytes -> char*, same caveats > > ptr[0] would (in my proposal anyway) return a Python object; BUT if that > is immedeately sent off to a C++ function by value it is just a > middle-person that can be left out safely, as an optimization. I think the right way to think about this is ptr[0] returning a C++ CppObject by value, but what then happens to it? If it is coerced to object then obviously one is constructed, likewise if it assigned a a Cython refcounted "cdef CppObject". But passing to C++ code would happen by value. Dag Sverre From robertwb at math.washington.edu Thu May 7 12:22:58 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 May 2009 03:22:58 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <30eeafae0e895c6e1aca708240db3cc3.squirrel@webmail.uio.no> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <16C56E2D-7A3A-405D-986C-377D043E7D8F@math.washington.edu> <05801ec60573b09b3f0bc28c72764050.squirrel@webmail.uio.no> <38519F5C-179F-4292-8DD0-32AAF2DBB404@math.washington.edu> <30eeafae0e895c6e1aca708240db3cc3.squirrel@webmail.uio.no> Message-ID: On May 7, 2009, at 2:47 AM, Dag Sverre Seljebotn wrote: > That whole explanation was a mess, as my daughter waked up from her > sleep > at the time before I could improve it :-) Don't worry, I *totally* understand. :-) >> b = a >> >> actually do? Construct a new Python object, then copy by value into >> it, and assign to be? So >> >> a is b >> >> is now False? > > I'll just forget about earlier explanations. > > But what I am advocating is to just have a regular Python reference > transfer, no C++ operators involved. [...] > I am agreeing with Stefan that this is probably orthogonal. It is > something you might want to support for C structs as well, so that > > cdef CWidget a, b > b = a > a.show_caption = True # also changes b > > would hold, i.e. an ability to give C structs Python semantics. > This would > especially be useful given the recent method-on-struct idea, as > > b = a > a.set_value(3) # does not modify b > > is a "new" wierd exception from Python semantics. This has very little advantage over just creating a simple cdef class... if one doesn't mind the overhead (which is usually the reason to go with structs in the first place). This is almost sounding like a proposal to do cdef managed CWidget which would be an object wrapping a CWidget. > I still believe it should be discussed now because > a) it heavily impacts how one thinks about C++ operators Yes. > b) it is a way to solve the construction/destruction problem with C+ > + classes I'm not sure it does. - Robert From dagss at student.matnat.uio.no Thu May 7 12:39:07 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 07 May 2009 12:39:07 +0200 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <16C56E2D-7A3A-405D-986C-377D043E7D8F@math.washington.edu> <05801ec60573b09b3f0bc28c72764050.squirrel@webmail.uio.no> <38519F5C-179F-4292-8DD0-32AAF2DBB404@math.washington.edu> <30eeafae0e895c6e1aca708240db3cc3.squirrel@webmail.uio.no> Message-ID: <4A02BA4B.5080302@student.matnat.uio.no> Robert Bradshaw wrote: > On May 7, 2009, at 2:47 AM, Dag Sverre Seljebotn wrote: > reason to go with structs in the first place). This is almost > sounding like a proposal to do > > cdef managed CWidget > > which would be an object wrapping a CWidget. Perhaps that is a better way to think about it. Still, introducing that as the direct way of wrapping C++ classes sidesteps the issue of figuring out how C++ semantics should work in Cython (just in order to be wrapped). >> I still believe it should be discussed now because >> a) it heavily impacts how one thinks about C++ operators > > Yes. > >> b) it is a way to solve the construction/destruction problem with C+ >> + classes > > I'm not sure it does. I am referring to having to do it[0][0], which is needed because one cannot do cdef vector[int].iterator it currently. With my proposal you can do cdef vector[int].iterator it print it is None # True it = vec.begin() while it != vec.end(): # != would go through to wrapped objects print deref(it) # it.get() with my operator proposal inc(it) # it.next() with my operator proposal which is a "solved" in my book. (I've let inc operate in-place here, so that "b = a; inc(a)" would modify b as well. As if inc is calling a method on a. If you did while it != vec.end(): ... it += 1 then you would start craving stack-allocation optimizations as this would have an malloc per iteration; but that's not too hard, just notice that "it" is never referenced directly anywhere (only used in operations) and so it can go on the stack instead). -- Dag Sverre From robertwb at math.washington.edu Thu May 7 12:41:06 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 May 2009 03:41:06 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <716739d959ac4aa061bfe5a55f1b64fe.squirrel@webmail.uio.no> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <716739d959ac4aa061bfe5a55f1b64fe.squirrel@webmail.uio.no> Message-ID: <0AE19092-B472-47D8-B0BA-E590768F9E5E@math.washington.edu> On May 7, 2009, at 2:56 AM, Dag Sverre Seljebotn wrote: >> Yes, you can, we do that in Sage. I don't think one can declare a >> struct member to be an object without a no-args constructor without >> extra work though, so one would have to do the same hackery that was >> proposed for stack-allocated objects. > > I don't know enough CPython, but another (cleaner) option could be to > allocate the whole PyObject using C++ new (which would pass on the > constructor arguments to the field using the "MyClass(int arg) : > myfield(arg) {}" syntax), and override tp_alloc to return that > object? If > it is possible to tell tp_alloc about that somehow. PyObject *tp_alloc(PyTypeObject *self, int nitems) I wouldn't call it cleaner, but one could have tp_alloc/tp_new raise an error, and then always construct the objects manually. Feels like we're abusing something there. > Another idea BTW is that we can use C++ templates to generate these > Python > wrappers, so we only need to output one (manually written) template > to our > code, and can then use > > CppPyWrapper > > and it could instantiate the right type structure through more > template > programming etc. One would have to have struct CppPyWrapper { PyObject_HEAD MyCppClass; CppPyWrapper() : MyCppClass(default_args) { }; }; which is no better than simply trying to put MyCppClass on the stack, so sticking the memory chunk in a PyObject* doesn't really save you anything. (Well, it lets you pass it out of the scope, and gives reference assignment semantics, but I'm not sure either of these are desirable for something that looks like its allocated on the stack, and one could provide this in an orthogonal way (e.g. with the "managed" keyword idea). Also, giving it reference semantics means that we could never go and optimize it by placing things actually on the stack. - Robert From robertwb at math.washington.edu Thu May 7 12:52:58 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 May 2009 03:52:58 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <4A02BA4B.5080302@student.matnat.uio.no> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <16C56E2D-7A3A-405D-986C-377D043E7D8F@math.washington.edu> <05801ec60573b09b3f0bc28c72764050.squirrel@webmail.uio.no> <38519F5C-179F-4292-8DD0-32AAF2DBB404@math.washington.edu> <30eeafae0e895c6e1aca708240db3cc3.squirrel@webmail.uio.no> <4A02BA4B.5080302@student.matnat.uio.no> Message-ID: <14F264DD-8789-4D99-9F94-63CAAE9FF157@math.washington.edu> On May 7, 2009, at 3:39 AM, Dag Sverre Seljebotn wrote: >>> b) it is a way to solve the construction/destruction problem with C+ >>> + classes >> >> I'm not sure it does. > > I am referring to having to do it[0][0], which is needed because one > cannot do > > cdef vector[int].iterator it > > currently. > > With my proposal you can do > > cdef vector[int].iterator it > print it is None # True > it = vec.begin() > while it != vec.end(): # != would go through to wrapped objects > print deref(it) # it.get() with my operator proposal > inc(it) # it.next() with my operator proposal To clarify, your operator proposal is the inline C++ code chunk strings? > which is a "solved" in my book. No, it's pushed the problem of allocating vector[int].iterator on the stack to allocating vector[int].iterator as a member of a struct. - Robert From dagss at student.matnat.uio.no Thu May 7 12:56:57 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 07 May 2009 12:56:57 +0200 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <14F264DD-8789-4D99-9F94-63CAAE9FF157@math.washington.edu> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <16C56E2D-7A3A-405D-986C-377D043E7D8F@math.washington.edu> <05801ec60573b09b3f0bc28c72764050.squirrel@webmail.uio.no> <38519F5C-179F-4292-8DD0-32AAF2DBB404@math.washington.edu> <30eeafae0e895c6e1aca708240db3cc3.squirrel@webmail.uio.no> <4A02BA4B.5080302@student.matnat.uio.no> <14F264DD-8789-4D99-9F94-63CAAE9FF157@math.washington.edu> Message-ID: <4A02BE79.2050304@student.matnat.uio.no> Robert Bradshaw wrote: > On May 7, 2009, at 3:39 AM, Dag Sverre Seljebotn wrote: > >>>> b) it is a way to solve the construction/destruction problem with C+ >>>> + classes >>> I'm not sure it does. >> I am referring to having to do it[0][0], which is needed because one >> cannot do >> >> cdef vector[int].iterator it >> >> currently. >> >> With my proposal you can do >> >> cdef vector[int].iterator it >> print it is None # True >> it = vec.begin() >> while it != vec.end(): # != would go through to wrapped objects >> print deref(it) # it.get() with my operator proposal >> inc(it) # it.next() with my operator proposal > > To clarify, your operator proposal is the inline C++ code chunk strings? > >> which is a "solved" in my book. > > No, it's pushed the problem of allocating vector[int].iterator on the > stack to allocating vector[int].iterator as a member of a struct. My problem isn't at all with the technical difficulties. I'm fine with using placement new for stack allocation if that is the problem. My problem is how the user semantics should look like. Python semantics has a concept of constructors/destructors which I'm borrowing here to fix the semantics part. The alternative is somehow invent a syntax to call constructors on explicitly stack-allocated variables, and you seemed to reject that in the thread I started about it. -- Dag Sverre From dagss at student.matnat.uio.no Thu May 7 12:57:29 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 07 May 2009 12:57:29 +0200 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <0AE19092-B472-47D8-B0BA-E590768F9E5E@math.washington.edu> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <716739d959ac4aa061bfe5a55f1b64fe.squirrel@webmail.uio.no> <0AE19092-B472-47D8-B0BA-E590768F9E5E@math.washington.edu> Message-ID: <4A02BE99.3060608@student.matnat.uio.no> Robert Bradshaw wrote: > which is no better than simply trying to put MyCppClass on the stack, > so sticking the memory chunk in a PyObject* doesn't really save you > anything. (Well, it lets you pass it out of the scope, and gives > reference assignment semantics, but I'm not sure either of these are > desirable for something that looks like its allocated on the stack, > and one could provide this in an orthogonal way (e.g. with the > "managed" keyword idea). Also, giving it reference semantics means > that we could never go and optimize it by placing things actually on > the stack. We're hitting the core of the matter here! I can see why one would like to stay "close to the metal", so to speak. I was mainly (mis)interpreting your lack of support in the "C++ stack allocation" thread as being in favour of this other proposal... Still, with direct stack allocation one needs a way to do cdef CppObject a, b and then run their constructors before they are used. With structs you can do cdef CObject a a.x = 3 which just doesn't fly with C++ with most classes. My thoughts were in the other thread, but I guess it boils down to making sure that "a" is assigned to before it is used. You are not rejecting that then? That would bring the whole C++ semantics into Cython, so that b = a would invoke operator=, b = a a.set_foo(3) # does not modify b and so on. I guess I don't have to like it to support it, I can see why you'd do it. Keeping "managed" strictly orthogonal does make a lot of sense. -- Dag Sverre From dagss at student.matnat.uio.no Thu May 7 13:00:02 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 07 May 2009 13:00:02 +0200 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <14F264DD-8789-4D99-9F94-63CAAE9FF157@math.washington.edu> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <16C56E2D-7A3A-405D-986C-377D043E7D8F@math.washington.edu> <05801ec60573b09b3f0bc28c72764050.squirrel@webmail.uio.no> <38519F5C-179F-4292-8DD0-32AAF2DBB404@math.washington.edu> <30eeafae0e895c6e1aca708240db3cc3.squirrel@webmail.uio.no> <4A02BA4B.5080302@student.matnat.uio.no> <14F264DD-8789-4D99-9F94-63CAAE9FF157@math.washington.edu> Message-ID: <4A02BF32.7050908@student.matnat.uio.no> Robert Bradshaw wrote: > On May 7, 2009, at 3:39 AM, Dag Sverre Seljebotn wrote: > >>>> b) it is a way to solve the construction/destruction problem with C+ >>>> + classes >>> I'm not sure it does. >> I am referring to having to do it[0][0], which is needed because one >> cannot do >> >> cdef vector[int].iterator it >> >> currently. >> >> With my proposal you can do >> >> cdef vector[int].iterator it >> print it is None # True >> it = vec.begin() >> while it != vec.end(): # != would go through to wrapped objects >> print deref(it) # it.get() with my operator proposal >> inc(it) # it.next() with my operator proposal > > To clarify, your operator proposal is the inline C++ code chunk strings? Yes. -- Dag Sverre From dagss at student.matnat.uio.no Thu May 7 13:28:05 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 07 May 2009 13:28:05 +0200 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <4A02BE99.3060608@student.matnat.uio.no> References: <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <716739d959ac4aa061bfe5a55f1b64fe.squirrel@webmail.uio.no> <0AE19092-B472-47D8-B0BA-E590768F9E5E@math.washington.edu> <4A02BE99.3060608@student.matnat.uio.no> Message-ID: <4A02C5C5.4000509@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> which is no better than simply trying to put MyCppClass on the stack, >> so sticking the memory chunk in a PyObject* doesn't really save you >> anything. (Well, it lets you pass it out of the scope, and gives >> reference assignment semantics, but I'm not sure either of these are >> desirable for something that looks like its allocated on the stack, >> and one could provide this in an orthogonal way (e.g. with the >> "managed" keyword idea). Also, giving it reference semantics means >> that we could never go and optimize it by placing things actually on >> the stack. > > We're hitting the core of the matter here! > > I can see why one would like to stay "close to the metal", so to speak. > I was mainly (mis)interpreting your lack of support in the "C++ stack > allocation" thread as being in favour of this other proposal... OK I'm sorry, my C++ stack allocation thread was way more far-reaching. Just scratch that reference. -- Dag Sverre From dagss at student.matnat.uio.no Thu May 7 13:32:03 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 07 May 2009 13:32:03 +0200 Subject: [Cython] [Numpy-discussion] efficient 3d histogram creation In-Reply-To: <9457e7c80905070411x6d51c768g20f4dd0990b843a8@mail.gmail.com> References: <7f014ea60905031715h635a69faof6a06e10c3621ba7@mail.gmail.com> <91cf711d0905041218p70bb44ct35a601844c8c262b@mail.gmail.com> <7f014ea60905041300y49b48055i6df8d5e598d0fe80@mail.gmail.com> <1cd32cbb0905041318g11dea0b9oa61f08bcc380144d@mail.gmail.com> <91cf711d0905050646q7652eepc55d1aed17b1d1ed@mail.gmail.com> <7f014ea60905061506o7253a315p940d6cbd2cc2b420@mail.gmail.com> <1cd32cbb0905061630t1e73e8a5i4bd454f789e22714@mail.gmail.com> <7f014ea60905061639k21dd12b5j4ee6c2738758284@mail.gmail.com> <1cd32cbb0905061721y233aa4c7uf580ad1d99539b1c@mail.gmail.com> <7f014ea60905061734y3dfcb299w14681ff0020a4de1@mail.gmail.com> <9457e7c80905070411x6d51c768g20f4dd0990b843a8@mail.gmail.com> Message-ID: <4A02C6B3.3030404@student.matnat.uio.no> St?fan van der Walt wrote: > 2009/5/7 Chris Colbert : >> This was really my first attempt at doing anything constructive with Cython. >> It was actually unbelievably easy to work with. I think i spent less time >> working on this, than I did trying to find an optimized solution using pure >> numpy and python. > > One aspect we often overlook is how easy it is to write a for-loop in > comparison to vectorisation. Besides, for-loops are sometimes easier > to read as well! > > I think the Cython guys are planning some sort of templating, but I'll > CC Dag so that he can tell us more. We were discussing how it would/should look like, but noone's committed to implementing it so it's pretty much up in the blue I think -- someone might jump in and do it next week, or it might go another year, I can't tell. While I'm here, also note in that code Chris wrote that you want to pay attention to the change of default division semantics on Cython 0.12 (especially for speed). http://wiki.cython.org/enhancements/division -- Dag Sverre From robertwb at math.washington.edu Thu May 7 13:37:05 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 May 2009 04:37:05 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <4A02BE99.3060608@student.matnat.uio.no> References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <716739d959ac4aa061bfe5a55f1b64fe.squirrel@webmail.uio.no> <0AE19092-B472-47D8-B0BA-E590768F9E5E@math.washington.edu>! <4A02BE99.3060608@student.matnat.uio.no> Message-ID: On May 7, 2009, at 3:57 AM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> which is no better than simply trying to put MyCppClass on the stack, >> so sticking the memory chunk in a PyObject* doesn't really save you >> anything. (Well, it lets you pass it out of the scope, and gives >> reference assignment semantics, but I'm not sure either of these are >> desirable for something that looks like its allocated on the stack, >> and one could provide this in an orthogonal way (e.g. with the >> "managed" keyword idea). Also, giving it reference semantics means >> that we could never go and optimize it by placing things actually on >> the stack. > > We're hitting the core of the matter here! > > I can see why one would like to stay "close to the metal", so to > speak. > I was mainly (mis)interpreting your lack of support in the "C++ stack > allocation" thread as being in favour of this other proposal... > > Still, with direct stack allocation one needs a way to do > > cdef CppObject a, b > > and then run their constructors before they are used. With structs you > can do > > cdef CObject a > a.x = 3 > > which just doesn't fly with C++ with most classes. > > My thoughts were in the other thread, but I guess it boils down to > making sure that "a" is assigned to before it is used. > > You are not rejecting that then? No. Have I been critical of it? Yes, but I have been of every potential solution so far (including my own). Honestly, I'm still feeling my way around here...and learning a lot about C++ as I go :) > That would bring the whole C++ semantics into Cython, so that > > b = a > > would invoke operator=, > > b = a > a.set_foo(3) # does not modify b > > and so on. Yep. > I guess I don't have to like it to support it, I can see why you'd do > it. Keeping "managed" strictly orthogonal does make a lot of sense. Originally, I thought we could keep things simple by only supporting CppClass*, but the pain of having to manually dereference everything, plus cumbersome dealings with objects that return objects by value, started to convince me otherwise. With very few exceptions (the new division semantics being one of them), I think it's valuable that code manipulating all C types translates nearly directly to C with very little extra magic or layers. One is then free to write as nice or raw of an interface on top of that as one wishes. I think applying the same principles to C+ + is a good idea. - Robert From dagss at student.matnat.uio.no Thu May 7 14:03:46 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 07 May 2009 14:03:46 +0200 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <716739d959ac4aa061bfe5a55f1b64fe.squirrel@webmail.uio.no> <0AE19092-B472-47D8-B0BA-E590768F9E5E@math.washington.edu>! <4A02BE99.3060608@student.matnat.uio.no> Message-ID: <4A02CE22.8050005@student.matnat.uio.no> Robert Bradshaw wrote: > With very few exceptions (the new division semantics being one of > them), I think it's valuable that code manipulating all C types > translates nearly directly to C with very little extra magic or > layers. One is then free to write as nice or raw of an interface on > top of that as one wishes. I think applying the same principles to C+ > + is a good idea. OK. Variable scoping is still a problem. I'll just dump my current, much clearer thoughts for the record. 1) The "with" clause has C variable scope semantics but would be kind of a pain to use. with vector[int].iterator(vec.begin()) as it: # cdefs the "it" in block while it != vec.end(): ... 2) Explicit construct/destruct is a possibility: cdef vector[int].iterator it ... construct(it, vec.begin()) destruct(it) ... But, ugly. If you construct twice in a function or forget to destruct you get to keep the pieces (often nothing bad will happen though). 3) Just tracking the first assignment, and destruct at end of function, would kind of work, but there's a problem: Basically the first assignment must invoke placement new, while subsequent assignments must use "=". This could be dealt with with a seperate flag though! So you have a flag "is_constructed", and if it is not, use placement new for assignment, otherwise use "=". As for eliminating such a flag, it would be easy to support cdef iterator it it = vec1.begin() # placement new if not foo: it = vec2.begin() # "=" but much more difficult to support if foo: it = vec1.begin() # placement new else: it = vec2.begin() # placement new and impossible to do if foo: it = vec1.begin() it = vec2.begin() # placement new or "="? need a flag Still, using such a flag in all situations before we have control flow analysis is a lot better than refcounting until we have control flow analysis :-) Finally, we could still have an explicit destruct(it) which would also toggle the flag back, making the next assignment reconstruct the object. This would actually work :-) (assuming void*[] is all the alignment we need). -- Dag Sverre From dagss at student.matnat.uio.no Thu May 7 14:10:02 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 07 May 2009 14:10:02 +0200 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <4A02CE22.8050005@student.matnat.uio.no> References: <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <716739d959ac4aa061bfe5a55f1b64fe.squirrel@webmail.uio.no> <0AE19092-B472-47D8-B0BA-E590768F9E5E@math.washington.edu>! <4A02BE99.3060608@student.matnat.uio.no> <4A02CE22.8050005@student.matnat.uio.no> Message-ID: <4A02CF9A.8090502@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > 3) Just tracking the first assignment, and destruct at end of function, > would kind of work, but there's a problem: Basically the first > assignment must invoke placement new, while subsequent assignments must > use "=". If this was unintuitive, consider the fact that when doing a = b what really happens is a.~ClassName(); // run destructor a.ClassName(b); // run copy constructor Thus we simply need to "achieve parity" with this, so to speak. -- Dag Sverre From dagss at student.matnat.uio.no Thu May 7 14:16:22 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 07 May 2009 14:16:22 +0200 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <4A02CF9A.8090502@student.matnat.uio.no> References: <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <716739d959ac4aa061bfe5a55f1b64fe.squirrel@webmail.uio.no> <0AE19092-B472-47D8-B0BA-E590768F9E5E@math.washington.edu>! <4A02BE99.3060608@student.matnat.uio.no> <4A02CE22.8050005@student.matnat.uio.no> <4A02CF9A.8090502@student.matnat.uio.no> Message-ID: <4A02D116.4010308@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Dag Sverre Seljebotn wrote: >> 3) Just tracking the first assignment, and destruct at end of function, >> would kind of work, but there's a problem: Basically the first >> assignment must invoke placement new, while subsequent assignments must >> use "=". > > If this was unintuitive, consider the fact that when doing > > a = b > > what really happens is > > a.~ClassName(); // run destructor > a.ClassName(b); // run copy constructor Wrong, wrong, wrong!! operator= is invoked! I'll just let this be thread be for now :-) -- Dag Sverre From kxroberto at googlemail.com Thu May 7 16:10:41 2009 From: kxroberto at googlemail.com (Robert) Date: Thu, 07 May 2009 16:10:41 +0200 Subject: [Cython] pyximport ignors [build] compiler = mingw32 Message-ID: On Windows in distutils/distutils.cfg MingW is set instead of VC default [build] compiler = mingw32 works well with extensions, including setup.py-method for building Cython modules. But it fails with pyximport method. Is this a bug - or how to tell using the MingW compiler? Robert >>> import pyximport >>> pyximport.install() >>> import cyex Traceback (most recent call last): File "", line 1, in File "C:\Python26\lib\site-packages\pyximport\pyximport.py", line 288, in load_module self.pyxbuild_dir) File "C:\Python26\lib\site-packages\pyximport\pyximport.py", line 154, in load_module raise ImportError("Building module failed: %s" % e) ImportError: Building module failed: DistutilsPlatformError('Unable to find vcvarsall.bat',) - File "C:\Python26\lib\distutils\dist.py", line 975, in run_commands self.run_command(cmd) File "C:\Python26\lib\distutils\dist.py", line 995, in run_command cmd_obj.run() File "C:\Python26\lib\distutils\command\build_ext.py", line 345, in run self.build_extensions() File "C:\Python26\lib\site-packages\Cython\Distutils\build_ext.py", line 83, in build_extensions self.build_extension(ext) File "C:\Python26\lib\distutils\command\build_ext.py", line 536, in build_extension depends=ext.depends) File "C:\Python26\lib\distutils\msvc9compiler.py", line 448, in compile self.initialize() File "C:\Python26\lib\distutils\msvc9compiler.py", line 358, in initialize vc_env = query_vcvarsall(VERSION, plat_spec) File "C:\Python26\lib\distutils\msvc9compiler.py", line 250, in query_vcvarsall raise DistutilsPlatformError("Unable to find vcvarsall.bat") DistutilsPlatformError: Unable to find vcvarsall.bat From dalcinl at gmail.com Thu May 7 18:44:24 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 7 May 2009 13:44:24 -0300 Subject: [Cython] pyximport ignors [build] compiler = mingw32 In-Reply-To: References: Message-ID: On Thu, May 7, 2009 at 11:10 AM, Robert wrote: > On Windows in distutils/distutils.cfg MingW is set instead of VC > default > > [build] > compiler = mingw32 > > > works well with extensions, including setup.py-method for building > Cython modules. But it fails with pyximport method. > Is this a bug - or how to tell using the MingW compiler? > I would not say it is a bug, but rather a consequence of the implementation of pyximport. Likely a call to "dist.parse_config_files()" should be added just before "dist.parse_command_line()" in pyximport/pyxbuild.py However, if you do this, a "setup.cfg" file will be parse if present in $PWD... This could be seen as a gotcha, or as a feature... > Robert > > > ?>>> import pyximport > ?>>> pyximport.install() > ?>>> import cyex > Traceback (most recent call last): > ? File "", line 1, in > ? File "C:\Python26\lib\site-packages\pyximport\pyximport.py", > line 288, in load_module > ? ? self.pyxbuild_dir) > ? File "C:\Python26\lib\site-packages\pyximport\pyximport.py", > line 154, in load_module > ? ? raise ImportError("Building module failed: %s" % e) > ImportError: Building module failed: > DistutilsPlatformError('Unable to find vcvarsall.bat',) > > > - > > ? File "C:\Python26\lib\distutils\dist.py", line 975, in run_commands > ? ? self.run_command(cmd) > ? File "C:\Python26\lib\distutils\dist.py", line 995, in run_command > ? ? cmd_obj.run() > ? File "C:\Python26\lib\distutils\command\build_ext.py", line > 345, in run > ? ? self.build_extensions() > ? File > "C:\Python26\lib\site-packages\Cython\Distutils\build_ext.py", > line 83, in build_extensions > ? ? self.build_extension(ext) > ? File "C:\Python26\lib\distutils\command\build_ext.py", line > 536, in build_extension > ? ? depends=ext.depends) > ? File "C:\Python26\lib\distutils\msvc9compiler.py", line 448, in > compile > ? ? self.initialize() > ? File "C:\Python26\lib\distutils\msvc9compiler.py", line 358, in > initialize > ? ? vc_env = query_vcvarsall(VERSION, plat_spec) > ? File "C:\Python26\lib\distutils\msvc9compiler.py", line 250, in > query_vcvarsall > ? ? raise DistutilsPlatformError("Unable to find vcvarsall.bat") > DistutilsPlatformError: Unable to find vcvarsall.bat > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Thu May 7 21:17:55 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 May 2009 12:17:55 -0700 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: <4A02CE22.8050005@student.matnat.uio.no> References: <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <716739d959ac4aa061bfe5a55f1b64fe.squirrel@webmail.uio.no> <0AE19092-B472-47D8-B0BA-E590768F9E5E@math.washington.edu>! <4A02BE99.3060608@student.matnat.uio.no> <4A02CE22.805000! 5@student.matnat.uio.no> Message-ID: On May 7, 2009, at 5:03 AM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> With very few exceptions (the new division semantics being one of >> them), I think it's valuable that code manipulating all C types >> translates nearly directly to C with very little extra magic or >> layers. One is then free to write as nice or raw of an interface on >> top of that as one wishes. I think applying the same principles to C+ >> + is a good idea. > > OK. Variable scoping is still a problem. I'll just dump my current, > much > clearer thoughts for the record. > > 1) The "with" clause has C variable scope semantics but would be > kind of > a pain to use. > > with vector[int].iterator(vec.begin()) as it: # cdefs the "it" in > block > while it != vec.end(): > ... > > 2) Explicit construct/destruct is a possibility: > > cdef vector[int].iterator it > ... > construct(it, vec.begin()) > > destruct(it) > ... > > But, ugly. If you construct twice in a function or forget to destruct > you get to keep the pieces (often nothing bad will happen though). > > 3) Just tracking the first assignment, and destruct at end of > function, > would kind of work, but there's a problem: Basically the first > assignment must invoke placement new, while subsequent assignments > must > use "=". > > This could be dealt with with a seperate flag though! So you have a > flag > "is_constructed", and if it is not, use placement new for assignment, > otherwise use "=". > > As for eliminating such a flag, it would be easy to support > > cdef iterator it > it = vec1.begin() # placement new > if not foo: > it = vec2.begin() # "=" > > but much more difficult to support > > if foo: > it = vec1.begin() # placement new > else: > it = vec2.begin() # placement new > > and impossible to do > > if foo: > it = vec1.begin() > it = vec2.begin() # placement new or "="? need a flag > > Still, using such a flag in all situations before we have control flow > analysis is a lot better than refcounting until we have control flow > analysis :-) > > Finally, we could still have an explicit > > destruct(it) > > which would also toggle the flag back, making the next assignment > reconstruct the object. > > This would actually work :-) (assuming void*[] is all the alignment we > need). I much prefer option (3), that seems both feasible and obvious to use. - Robert From dagss at student.matnat.uio.no Thu May 7 21:32:41 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 07 May 2009 21:32:41 +0200 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <716739d959ac4aa061bfe5a55f1b64fe.squirrel@webmail.uio.no> <0AE19092-B472-47D8-B0BA-E590768F9E5E@math.washington.edu>! <4A02BE99.3060608@student.matnat.uio.no> <4A02CE22.805000! 5@student.matnat.uio.no> Message-ID: <4A033759.7060007@student.matnat.uio.no> Robert Bradshaw wrote: > On May 7, 2009, at 5:03 AM, Dag Sverre Seljebotn wrote: > >> Robert Bradshaw wrote: >>> With very few exceptions (the new division semantics being one of >>> them), I think it's valuable that code manipulating all C types >>> translates nearly directly to C with very little extra magic or >>> layers. One is then free to write as nice or raw of an interface on >>> top of that as one wishes. I think applying the same principles to C+ >>> + is a good idea. >> OK. Variable scoping is still a problem. I'll just dump my current, >> much >> clearer thoughts for the record. >> >> 1) The "with" clause has C variable scope semantics but would be >> kind of >> a pain to use. >> >> with vector[int].iterator(vec.begin()) as it: # cdefs the "it" in >> block >> while it != vec.end(): >> ... >> >> 2) Explicit construct/destruct is a possibility: >> >> cdef vector[int].iterator it >> ... >> construct(it, vec.begin()) >> >> destruct(it) >> ... >> >> But, ugly. If you construct twice in a function or forget to destruct >> you get to keep the pieces (often nothing bad will happen though). >> >> 3) Just tracking the first assignment, and destruct at end of >> function, >> would kind of work, but there's a problem: Basically the first >> assignment must invoke placement new, while subsequent assignments >> must >> use "=". >> >> This could be dealt with with a seperate flag though! So you have a >> flag >> "is_constructed", and if it is not, use placement new for assignment, >> otherwise use "=". >> >> As for eliminating such a flag, it would be easy to support >> >> cdef iterator it >> it = vec1.begin() # placement new >> if not foo: >> it = vec2.begin() # "=" >> >> but much more difficult to support >> >> if foo: >> it = vec1.begin() # placement new >> else: >> it = vec2.begin() # placement new >> >> and impossible to do >> >> if foo: >> it = vec1.begin() >> it = vec2.begin() # placement new or "="? need a flag >> >> Still, using such a flag in all situations before we have control flow >> analysis is a lot better than refcounting until we have control flow >> analysis :-) >> >> Finally, we could still have an explicit >> >> destruct(it) >> >> which would also toggle the flag back, making the next assignment >> reconstruct the object. >> >> This would actually work :-) (assuming void*[] is all the alignment we >> need). > > I much prefer option (3), that seems both feasible and obvious to use. Sounds like a plan. (I guess a @notassignedtoyetcheck directive is in order too then). I'd feel it would be natural to allow f() = 3 and f(a) # a is modified as well then -- why stop at 90%... I'd like to eventually have (1) available as well a more obvious alternative in case you *need* explicit control over when the destructor is called, or the object is a bit big. with DbConnection("http://...") as db: ... seems very natural, and this is a very common idiom in C++ (they basically use stack allocation for both the "normal" situations, *and* for situations where one would use "with" in Python). It would then be natural tohave this in general, so that you could do with cython.int(3) as a: # locally scoped "cdef int a = 3" -- Dag Sverre From stefan_ml at behnel.de Thu May 7 21:46:26 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 07 May 2009 21:46:26 +0200 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: <5a57400a2d4156d4be6768682da7b637.squirrel@webmail.uio.no> <3123AB27-179B-4A6C-825B-AE908DA8912D@math.washington.edu> <4A028CC4.6040903@student.matnat.uio.no> <68d82818b156325a1804496c0c648926.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <0f1fc8d2fd9e3c82a7a0f702c0ae46cb.squirrel@webmail.uio.no> <81455c7931400ec3a50841de4c657a22.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <716739d959ac4aa061bfe5a55f1b64fe.squirrel@webmail.uio.no> <0AE19092-B472-47D8-B0BA-E590768F9E5E@math.washington.edu>! <4A02BE99.3060608@student.matnat.uio.no> <4A02CE22.805000! 5@student.matnat.uio.no> Message-ID: <4A033A92.3000100@behnel.de> Robert Bradshaw wrote: > On May 7, 2009, at 5:03 AM, Dag Sverre Seljebotn wrote: >> 3) Just tracking the first assignment, and destruct at end of >> function, >> would kind of work, but there's a problem: Basically the first >> assignment must invoke placement new, while subsequent assignments >> must >> use "=". >> >> This could be dealt with with a seperate flag though! So you have a >> flag >> "is_constructed", and if it is not, use placement new for assignment, >> otherwise use "=". >> >> As for eliminating such a flag, it would be easy to support >> >> cdef iterator it >> it = vec1.begin() # placement new >> if not foo: >> it = vec2.begin() # "=" >> >> but much more difficult to support >> >> if foo: >> it = vec1.begin() # placement new >> else: >> it = vec2.begin() # placement new >> >> and impossible to do >> >> if foo: >> it = vec1.begin() >> it = vec2.begin() # placement new or "="? need a flag >> >> Still, using such a flag in all situations before we have control flow >> analysis is a lot better than refcounting until we have control flow >> analysis :-) >> >> Finally, we could still have an explicit >> >> destruct(it) >> >> which would also toggle the flag back, making the next assignment >> reconstruct the object. >> >> This would actually work :-) (assuming void*[] is all the alignment we >> need). > > I much prefer option (3), that seems both feasible and obvious to use. +1, makes sense to me. Stefan From dagss at student.matnat.uio.no Thu May 7 22:09:13 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 07 May 2009 22:09:13 +0200 Subject: [Cython] Buffer passing implementation details In-Reply-To: References: <4A01BB4A.1060003@student.matnat.uio.no> Message-ID: <4A033FE9.9060001@student.matnat.uio.no> Seems I got too caught up in C++ stuff :-) Here's an answer. Kurt Smith wrote: > On Wed, May 6, 2009 at 11:31 AM, Dag Sverre Seljebotn > wrote: >> (Kurt is the major recipient of this.) >> >> I've been thinking some more on buffer passing. In the end (and perhaps >> even in summer; I could perhaps have a go at it working alongside you) >> the scenario we are looking at is something like >> >> external_set_foo_array(foo_handle, self.arr[2::2]) >> >> with fast operations all the way. >> >> That is >> a) Efficient slicing without Python overhead (#178) >> b) Storing acquired buffers in cdef class fields (#301) >> c) External functions can keep a reference to the buffer (not really >> necesarry, but it is necesarry for internal Cython "cdef" functions, and >> it would be nice to treat them the same) >> d) The base pointer may have to be moved (again slicing can do this) >> >> This seems to make it clear that >> a) The Py_buffer is not suitable as the primary vessel of our buffer >> data, e.g. to the external function. It can't work that well with >> slicing as we must maintain the original Py_buffer data when releasing it. >> b) We need a reference count. E.g. doing a slice, or assigning to >> self.arr, would increase this count. This must go on the heap; and so we >> might as well put the entire Py_buffer on the heap. > > Doing the slicing manually on the buffer data and having a refcounted > heap-allocated Py_buffer would address the object-Py_buffer data > synchronization problem, right? The above modifications would have > that, once the buffer is acquired, all modifications to it are done at > the buffer level. Not sure what you mean, but: Once the buffer is required, all new slices etc. is done in the local structs and the Py_buffer just sits there for querying information that doesn't change and for use release time. > Although, what about when we require a contiguous copy (or any mode, > for that matter) of a buffer? Would we set the top-level object > reference to None, as suggested in a previous thread? That would > prevent slicing at the Python API level, though, right? Hmm. I'm still thinking about contiguous copies. Let's deal with that in the next iteration. > General question: how difficult would it be for the Cython-side > programmer to get the underlying char *data inside the Py_buffer, in > case it needs to be accessed? It seems there are more than a few > 'layers' -- are they all necessary? The justifications I see for each > are the following: > > typedef struct { > void *buf; > PyObject *obj; > Py_ssize_t len; > Py_ssize_t itemsize; > int readonly; > int ndim; > char *format; > Py_ssize_t *shape; > Py_ssize_t *strides; > Py_ssize_t *suboffsets; > void *internal; > } Py_buffer; I think the presence of the obj field here is a bit unsatisfactory. It seems that after months (not to say years) of buffer PEP discussion, someone just added it in a bugfix *sigh* Anyway it is not to my knowledge documented anywhere but there's a long discussion about it here: http://bugs.python.org/issue3139 (You don't really need to read it though.) Things seem a bit unclear due to the lack of official documentation, but at least for now I don't want to assume that the PyObject filled in here is the same as the object you acquire the buffer from. The obj field is filled in by the tp_getbuffer implementation which could do things like make a dedicated copy of the buffer into a new object and then pass that new object into the obj field. Anyway, the idea (I gather from reading CPython sources) is that the obj is who should recieve the release call, which may or may not be the object you acquire on. So really, obj here is just present as the recipient of a release. It might even be set to NULL if there's no need to release the buffer. > The "base layer" -- interface structure for tp_getbuffer from objects, etc. > > typedef struct { > size_t refcount; > Py_buffer bufinfo; > } __Pyx_Buffer; > > The Cython wrapper around Py_buffer -- includes cython-managed > refcounting upon assignment/slicing (other cases?) Basically whenever the __Pyx_buffer* is copied. > typedef struct { > __Pyx_buffer* bufinfo; > char* data; > Py_ssize_t shape0, stride0; > } __Pyx_StridedBuf_1D; > > The "int[:] buf" buffer -- exists to make the new buffer syntax more > transparent (not part of GSoC, but putting in framework for it). char > *data pointer to allow fast slicing w/o Python API (e.g. arr[10:]), > shape0 & stride0 copied from buffer for optimizations. Yep. (And shape0 and stride0 also to allow fast slicing; we don't want to be modifying the shape/stride arrays passed in the Py_buffer) To be pedantic (and make sure things are understood correctly) there's not a Python API for slicing, only a NumPy one. > > typedef struct { > __Pyx_StridedBuf_1D buffer; > PyObject* object; > } __Pyx_StridedBuf_1D_Obj; > > The "object[int, mode='strided'] buf" object buffer. This is the top > level and holds a reference to the original PyObject from which the > buffer was acquired. Object reference required when a python-level > access/assignment is required. Yes. a) Since this doesn't have to be the same as the on in Py_buffer, a seperate reference seems necesarry, b) storing a function local variable inside a pointer will make accesses to the object part of the variable slightly slower. > So, initially, if one passes a 'buffer' to a function like so: > > external_func(arr[2::2]) > > What is the slicing modifying/updating? The top-level through the > Python API layer via the PyObject? Or is it being done to the > __Pyx_StridedBuf_1D members -- offsetting the char *data pointer by 2, > and changing the shape0 and stride0 members? (Would it be the top > layer object at first & for the GSoC, then when faster, low-level > slicing is implemented, directly on the lower-level?) Yes. First & for GSoC, we wouldn't know anything about "arr[2::2]", it would just be "some Python expression". It would (if arr is a NumPy object) return a new NumPy object from which we can acquire a correct Py_buffer. When slicing is optimized, we don't have that luxury, and must store the info in shape0/stride0. > > Besides Python API-level slicing, why is the top-level object > reference required? Is it for buffer reacquisition through > tp_getbuffer? That will be handled at the next lower level through a > struct copy by value and increase of the refcount. It isn't for > buffer release, either, since that doesn't require the original object > reference explicitly. (There is an object reference at the Py_buffer > level). It is needed because when you do cdef np.ndarray[int] obj = ... obj.fooblargh() on which object do you execute "fooblargh()"? (And we can't rely on the Py_buffer level one.) > Basically, nothing Cython-side would require the PyObject from which > the original buffer is acquired once all these are implemented, right? > (Again, not all part of GSoC, but at least putting in the framework.) Yes, that would happen at the same rate that the cdef object[int] a syntax is deprecated in favor of cdef int[:] a This is really just linked to syntax. >> It sheems a shame to let go of a neatly PEP-defined Py_buffer for >> passing to external functions, but I think it won't be too bad if we >> with each Cython version ship nice C header and Fortran include files >> containing the appropriate structs and access macros. > > I guess the access macros would make it easy for the external > functions to get to the important stuff, so my previous concern isn't > so essential (removing the 'top level' layer from the buffer struct > heirarchy). Well, if you think it is easier we can just introduce the new syntax only in a few limited spots for your GSoC and thus drop the top level. Your pick. I don't think it makes much of a difference. -- Dag Sverre From bblais at gmail.com Fri May 8 01:10:23 2009 From: bblais at gmail.com (Brian Blais) Date: Thu, 07 May 2009 19:10:23 -0400 Subject: [Cython] class versus struct...optimization Message-ID: Hello, I am trying to write a neuron simulator, and I have the need for speed. :) Unfortunately, I seem to be taking a serious performance hit by writing things as classes, as opposed to c-structs. I am including the code below (I hope it's not too much) of the smallest meaningful example I could write (although it still doesn't do anything real. :) ). The first is an all-python version, which is easy to follow but slow. In [1]:%timeit py1.runit() 10 loops, best of 3: 606 ms per loop The second is a cython version, which makes extension classes to replace the python ones, and I get a nice speed up of about 30x: In [2]:%timeit cpy2.runit() 10 loops, best of 3: 20.9 ms per loop The final one is ugly code, making an array of structures instead of lists of class instances. Where I would have more classes in the second version, I'd pile those variables all into this one big struct so I can make an array of them easily. It's ugly, but much faster: In [3]:%timeit cpy3.runit() 1000 loops, best of 3: 1.29 ms per loop My question is, can I have the best of both worlds (i.e. am I doing something obviously stupid), or is this the price for cleaner object- oriented code? Are the the method look-ups killing me in the object- oriented way? In the larger code I am working on, the difference between the class versus struct versions is about a factor of 5, not the factor of 20 above, but there is more going on. Still, a factor of 5 makes a big difference if the simulation may take a day to run! thanks, Brian Blais -- Brian Blais bblais at gmail.com http://web.bryant.edu/~bblais #================================================ #==================PYTHON VERSION================ #================================================ from numpy import ones,zeros,prod class Neuron(object): """docstring for Neuron""" def __init__(self, N): super(Neuron, self).__init__() self.N = N self.type=1 self.shape=(N,) self.spikes=zeros(N,int) self.old_spikes=zeros(N,int) self.V=zeros(N) self.connections_to=[] def update(self,t): for c in self.connections_to: V=self.__getattribute__('V') w=c.weights V+=w[:,c.incell.spikes.nonzero()[0]].sum(axis=1) V=self.V self.V-=self.V/100.0 self.spikes[:]=1.0 class Connection(object): """docstring for Connection""" def __init__(self, incell,outcell): super(Connection, self).__init__() self.incell=incell self.outcell=outcell self.shape=(outcell.N,incell.N) self.outcell.connections_to.append(self) self.weights=ones((outcell.N,incell.N),float) def update(self,t): pass def run_sim( duration,neurons,connections): t=0.0 while t<=duration: for n in neurons: n.update(t) # for c in connections: # c.update(t) t+=1.0 def runit(): n1=Neuron(5) n2=Neuron(3) c=Connection(n1,n2) run_sim(10000,[n1,n2],[c]) #================================================ #==================CYTHON VERSION WITH CLASSES================ #================================================ import numpy as np cimport numpy as np DTYPE=np.float64 ctypedef np.float64_t DTYPE_t DTYPE_I=np.int ctypedef np.int_t DTYPE_I_t from numpy import ones,zeros,prod cdef class Neuron: """docstring for Neuron""" cdef readonly int N cdef public np.ndarray spikes cdef public np.ndarray V cdef public object connections_to def __getitem__(self,key): return self.__getattribute__(key) def __init__(self, N): self.N = N self.spikes=zeros(N,int) self.V=zeros(N) self.connections_to=[] cdef void _update(self,double t): cdef int __i,__j cdef np.ndarray[DTYPE_I_t,ndim=1] spikes cdef np.ndarray[DTYPE_t,ndim=1] V cdef np.ndarray[DTYPE_t,ndim=2] _W cdef int k for c in self.connections_to: V=self.__getattribute__('V') _W=c.weights spikes=c.incell.spikes k=c.weights.shape[1] for __i in range(self.N): for __j in range(k): if spikes[__j]: V[__i]+=_W[__i,__j] V=self.V spikes=self.spikes for __i in range(self.N): V[__i]-=V[__i]/100.0 spikes[__i]=1 def update(self,double t): self._update(t) cdef class Connection: """docstring for Connection""" cdef public object incell,outcell cdef public object shape cdef public np.ndarray weights def __init__(self, incell,outcell): self.incell=incell self.outcell=outcell self.shape=(outcell.N,incell.N) self.outcell.connections_to.append(self) self.weights=ones((outcell.N,incell.N),float) def update(self,t): pass def run_sim(duration,neurons,connections): t=0.0 while t<=duration: for n in neurons: n.update(t) # for c in connections: # c.update(t) t+=1.0 #================================================ #==================CYTHON VERSION WITH STRUCTS================ #================================================ import numpy as np cimport numpy as np DTYPE=np.float64 ctypedef np.float64_t DTYPE_t cdef double* DoubleData(np.ndarray M): return M.data cdef char* CharData(np.ndarray M): return M.data cdef int* IntData(np.ndarray M): return M.data cdef struct Neuron_Group_struct: int N int *spikes double *V int type int number_of_connections_to # hard-coded maximum numbers...yuck! int *c_spikes[31] int c_num_incell[31],c_num_outcell[31] double *c_weights[31] # function to initialize the struct cdef Neuron_Group_struct init_Neuron_Group(object self): cdef Neuron_Group_struct s s.type=self.type s.N=self.N s.spikes=IntData(self.spikes) s.V=DoubleData(self.V) s.number_of_connections_to=len(self.connections_to) cdef int cg for cg from 0<=cg References: <9457e7c80905041443h658f3953kbc70f8af93bdc76f@mail.gmail.com> <822e653d4d2f3aaff7f2face512298eb.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A017E4F.5050800@student.matnat.uio.no> <3ddb93d123a6542c56cad2d44d257ccc.squirrel@groupware.dvs.informatik.tu-darmstadt.de> <4A0188D4.1050208@student.matnat.uio.no> <11E61C9F-3175-4473-8B39-20277284E9D1@math.washington.edu> <4B519262-77AC-4BC4-A62B-026B4E254FAA@math.washington.edu> <4A027E52.9070003@behnel.de> Message-ID: <4A037454.4010805@canterbury.ac.nz> Stefan Behnel wrote: > So this would mimic the "public" mechanism in a way, although it'd be > better to do the normal name mangling for tp_new() even if we export it. Yes, the name should definitely still be mangled, since that's the only way of distinguishing between classes having the same name in different modules. And there's no reason not to mangle, because the programmer won't be calling it directly. -- Greg From greg.ewing at canterbury.ac.nz Fri May 8 02:18:51 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 08 May 2009 12:18:51 +1200 Subject: [Cython] Syntax to declare a c++ class? In-Reply-To: References: <171eeee8daf5f4b4c45764c623a0cdc3.squirrel@webmail.uio.no> <3b99bf375590190ac87fccc498e35fd2.squirrel@webmail.uio.no> <0AB84EC4-422E-4E6D-AAED-C72EC1C47BD9@math.washington.edu> <4A023FA2.3040001@canterbury.ac.nz> <4A027A1E.7020604@behnel.de> <47F3AE68-2D7B-44DD-90E7-14587EEB0D33@math.washington.edu> Message-ID: <4A037A6B.9000002@canterbury.ac.nz> Stefan Behnel wrote: > I think it makes > sense to add that to the "extern" 'keyword' to say: "this external > declaration obeys C++ semantics". External declarations aren't necessarily the only place you might want to declare something as having C++ semantics, though. The advantage of cdef+ (or any variation that modifies the cdef keyword) is that it's easily applied to any existing C declaration, and there's no need to invent a new syntax for each case. -- Greg From dalcinl at gmail.com Fri May 8 02:42:50 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 7 May 2009 21:42:50 -0300 Subject: [Cython] class versus struct...optimization In-Reply-To: References: Message-ID: Brian, in the Cython version, class Neuron, do this: instead of "def update(...)", please write "cpdef update(...)" and time your code again. If this make it faster (it should be near the C-struct version), please consider first removing your original "def update", and replace your "cdef _update()" by "cpdef update()". On Thu, May 7, 2009 at 8:10 PM, Brian Blais wrote: > Hello, > I am trying to write a neuron simulator, and I have the need for speed. ?:) > ?Unfortunately, I seem to be taking a serious performance hit by writing > things as classes, as opposed to c-structs. ?I am including the code below > (I hope it's not too much) of the smallest meaningful example I could write > (although it still doesn't do anything real. :) ?). ?The first is an > all-python version, which is easy to follow but slow. > In [1]:%timeit py1.runit() > 10 loops, best of 3: 606 ms per loop > > The second is a cython version, which makes extension classes to replace the > python ones, and I get a nice speed up of about 30x: > In [2]:%timeit cpy2.runit() > 10 loops, best of 3: 20.9 ms per loop > > The final one is ugly code, making an array of structures instead of lists > of class instances. ?Where I would have more classes in the second version, > I'd pile those variables all into this one big struct so I can make an array > of them easily. ?It's ugly, but much faster: > In [3]:%timeit cpy3.runit() > 1000 loops, best of 3: 1.29 ms per loop > > My question is, can I have the best of both worlds (i.e. am I doing > something obviously stupid), or is this the price for cleaner > object-oriented code? ?Are the the method look-ups killing me in the > object-oriented way? ?In the larger code I am working on, the difference > between the class versus struct versions is about a factor of 5, not the > factor of 20 above, but there is more going on. ?Still, a factor of 5 makes > a big difference if the simulation may take a day to run! > thanks, > Brian Blais > > > -- > Brian Blais > bblais at gmail.com > http://web.bryant.edu/~bblais > > #================================================ > #==================PYTHON VERSION================ > #================================================ > from numpy import ones,zeros,prod > class Neuron(object): > ?? ?"""docstring for Neuron""" > ?? ?def __init__(self, N): > ?? ? ? ?super(Neuron, self).__init__() > ?? ? ? ?self.N = N > ?? ? ? ?self.type=1 > ?? ? ? ?self.shape=(N,) > ?? ? ? ?self.spikes=zeros(N,int) > ?? ? ? ?self.old_spikes=zeros(N,int) > ?? ? ? ?self.V=zeros(N) > ?? ? ? ?self.connections_to=[] > > ?? ?def update(self,t): > > ?? ? ? ?for c in self.connections_to: > ?? ? ? ? ? ?V=self.__getattribute__('V') > ?? ? ? ? ? ?w=c.weights > ?? ? ? ? ? ?V+=w[:,c.incell.spikes.nonzero()[0]].sum(axis=1) > ?? ? ? ?V=self.V > ?? ? ? ?self.V-=self.V/100.0 > ?? ? ? ?self.spikes[:]=1.0 > > > class Connection(object): > ?? ?"""docstring for Connection""" > ?? ?def __init__(self, incell,outcell): > ?? ? ? ?super(Connection, self).__init__() > ?? ? ? ?self.incell=incell > ?? ? ? ?self.outcell=outcell > ?? ? ? ?self.shape=(outcell.N,incell.N) > ?? ? ? ?self.outcell.connections_to.append(self) > ?? ? ? ?self.weights=ones((outcell.N,incell.N),float) > > > ?? ?def update(self,t): > ?? ? ? ?pass > > def run_sim( duration,neurons,connections): > > ?? ?t=0.0 > ?? ?while t<=duration: > ?? ? ? ?for n in neurons: > ?? ? ? ? ? ?n.update(t) > > ?? ? ? ?# for c in connections: > ?? ? ? ?# ? ? c.update(t) > ?? ? ? ?t+=1.0 > def runit(): > ?? ?n1=Neuron(5) > ?? ?n2=Neuron(3) > > ?? ?c=Connection(n1,n2) > > ?? ?run_sim(10000,[n1,n2],[c]) > > > > #================================================ > #==================CYTHON VERSION WITH CLASSES================ > #================================================ > import numpy as np > cimport numpy as np > > DTYPE=np.float64 > ctypedef np.float64_t DTYPE_t > DTYPE_I=np.int > ctypedef np.int_t DTYPE_I_t > from numpy import ones,zeros,prod > cdef class Neuron: > ?? ?"""docstring for Neuron""" > > ?? ?cdef readonly int N > ?? ?cdef public np.ndarray spikes > ?? ?cdef public np.ndarray V > ?? ?cdef public object connections_to > ?? ?def __getitem__(self,key): > ?? ? ? ?return self.__getattribute__(key) > > > ?? ?def __init__(self, N): > ?? ? ? ?self.N = N > ?? ? ? ?self.spikes=zeros(N,int) > ?? ? ? ?self.V=zeros(N) > ?? ? ? ?self.connections_to=[] > > ?? ?cdef void _update(self,double t): > ?? ? ? ?cdef int __i,__j > > ?? ? ? ?cdef np.ndarray[DTYPE_I_t,ndim=1] spikes > ?? ? ? ?cdef np.ndarray[DTYPE_t,ndim=1] V > ?? ? ? ?cdef np.ndarray[DTYPE_t,ndim=2] _W > ?? ? ? ?cdef int k > > > ?? ? ? ?for c in self.connections_to: > ?? ? ? ? ? ?V=self.__getattribute__('V') > ?? ? ? ? ? ?_W=c.weights > ?? ? ? ? ? ?spikes=c.incell.spikes > ?? ? ? ? ? ?k=c.weights.shape[1] > ?? ? ? ? ? ?for __i in range(self.N): > ?? ? ? ? ? ? ? ?for __j in range(k): > ?? ? ? ? ? ? ? ? ? ?if spikes[__j]: > ?? ? ? ? ? ? ? ? ? ? ? ?V[__i]+=_W[__i,__j] > > ?? ? ? ?V=self.V > ?? ? ? ?spikes=self.spikes > ?? ? ? ?for __i in range(self.N): > ?? ? ? ? ? ?V[__i]-=V[__i]/100.0 > ?? ? ? ? ? ?spikes[__i]=1 > > > ?? ?def update(self,double t): > ?? ? ? ?self._update(t) > > > cdef class Connection: > ?? ?"""docstring for Connection""" > ?? ?cdef public object incell,outcell > ?? ?cdef public object shape > ?? ?cdef public np.ndarray weights > > ?? ?def __init__(self, incell,outcell): > ?? ? ? ?self.incell=incell > ?? ? ? ?self.outcell=outcell > ?? ? ? ?self.shape=(outcell.N,incell.N) > ?? ? ? ?self.outcell.connections_to.append(self) > ?? ? ? ?self.weights=ones((outcell.N,incell.N),float) > > > ?? ?def update(self,t): > ?? ? ? ?pass > > def run_sim(duration,neurons,connections): > > ?? ?t=0.0 > ?? ?while t<=duration: > ?? ? ? ?for n in neurons: > ?? ? ? ? ? ?n.update(t) > > ?? ? ? ?# for c in connections: > ?? ? ? ?# ? ? c.update(t) > ?? ? ? ?t+=1.0 > > #================================================ > #==================CYTHON VERSION WITH STRUCTS================ > #================================================ > import numpy as np > cimport numpy as np > > DTYPE=np.float64 > ctypedef np.float64_t DTYPE_t > > cdef double* DoubleData(np.ndarray M): > ?? ?return M.data > cdef char* CharData(np.ndarray M): > ?? ?return M.data > > cdef int* IntData(np.ndarray M): > ?? ?return M.data > cdef struct Neuron_Group_struct: > ?? ?int N > ?? ?int *spikes > ?? ?double *V > ?? ?int type > ?? ?int number_of_connections_to > > ?? ?# hard-coded maximum numbers...yuck! > ?? ?int *c_spikes[31] > ?? ?int c_num_incell[31],c_num_outcell[31] > ?? ?double *c_weights[31] > > > # function to initialize the struct > cdef Neuron_Group_struct init_Neuron_Group(object self): > ?? ?cdef Neuron_Group_struct s > ?? ?s.type=self.type > ?? ?s.N=self.N > ?? ?s.spikes=IntData(self.spikes) > ?? ?s.V=DoubleData(self.V) > ?? ?s.number_of_connections_to=len(self.connections_to) > > ?? ?cdef int cg > > ?? ?for cg from 0<=cg ?? ? ? ?s.c_spikes[cg]=IntData(self.connections_to[cg].incell.spikes) > ?? ? ? ?s.c_num_incell[cg]=self.connections_to[cg].incell.N > ?? ? ? ?s.c_num_outcell[cg]=self.connections_to[cg].outcell.N > > ?? ? ? ?s.c_weights[cg]=DoubleData(self.connections_to[cg].weights) > > cdef void Test_update(Neuron_Group_struct *s,object self,double t): > ?? ?cdef int __i,__j > ?? ?cdef int number_of_connections_to > ?? ?number_of_connections_to=s.number_of_connections_to > ?? ?cdef int *spikes > ?? ?cdef int num_incell,num_outcell > ?? ?cdef int cg,ni,no > ?? ?cdef double *weights > ?? ?cdef double *V > > ?? ?V=s.V > > ?? ?for cg from 0<=cg ?? ? ? ?spikes=s.c_spikes[cg] > ?? ? ? ?num_incell=s.c_num_incell[cg] > ?? ? ? ?num_outcell=s.c_num_outcell[cg] > > ?? ? ? ?weights=s.c_weights[cg] > ?? ? ? ?for ni from 0<=ni ?? ? ? ? ? ?if spikes[ni]: > ?? ? ? ? ? ? ? ?for no from 0<=no ?? ? ? ? ? ? ? ? ? V[ni]+=weights[no+ni*num_outcell] > ?? ?V=s.V > ?? ?spikes=s.spikes > ?? ?for no from 0<=no ?? ? ? ?V[no]-=V[no]/100.0 > ?? ? ? ?spikes[no]=1 > > cdef nupdate(Neuron_Group_struct *s,object self,double t): > ?? ?if s.type==-1: > ?? ? ? ?self.update(t) > ?? ? ? ?return > ?? ?if s.type==0: # Silent Neuron > ?? ? ? ?pass > ?? ?elif s.type==1: # Constant Fixed > ?? ? ? ?Test_update(s,self,t) > > def run_sim(duration,neurons,connections): > ?? ?cdef long long t > ?? ?cdef int i > ?? ?cdef int ln,lc > ?? ?ln=len(neurons) > ?? ?lc=len(connections) > ?? ?cdef long long num_iter > ?? ?num_iter=duration > > ?? ?cdef Neuron_Group_struct Sn[31] > ?? ?for i from 0<=i ?? ? ? ?Sn[i]=init_Neuron_Group(neurons[i]) > ?? ?for t from 0<=t ?? ? ? ?for i from 0<=i ?? ? ? ? ? ?nupdate(&Sn[i],neurons[i],t) > > > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From josef.pktd at gmail.com Fri May 8 02:37:58 2009 From: josef.pktd at gmail.com (josef) Date: Fri, 8 May 2009 00:37:58 +0000 (UTC) Subject: [Cython] pyximport ignors [build] compiler = mingw32 References: Message-ID: Robert writes: > > On Windows in distutils/distutils.cfg MingW is set instead of VC > default > > [build] > compiler = mingw32 > > works well with extensions, including setup.py-method for building > Cython modules. But it fails with pyximport method. > Is this a bug - or how to tell using the MingW compiler? > I just hard coded the required include path and compiler in my installed pyxbuild.py, as a quick solution, since I didn't find any options to do it otherwise. args = [quiet, "build_ext", "--compiler=mingw32", r"-Ic:\programs\python25\lib\site-packages\numpy\core\include"] Josef From bblais at gmail.com Fri May 8 12:08:08 2009 From: bblais at gmail.com (Brian Blais) Date: Fri, 08 May 2009 06:08:08 -0400 Subject: [Cython] class versus struct...optimization In-Reply-To: References: Message-ID: On May 7, 2009, at 20:42 , Lisandro Dalcin wrote: > Brian, in the Cython version, class Neuron, do this: instead of "def > update(...)", please write "cpdef update(...)" and time your code > again. If this make it faster (it should be near the C-struct > version), please consider first removing your original "def update", > and replace your "cdef _update()" by "cpdef update()". > thanks, but this didn't change the time much at all (something like . 5 ms out of 20ms). I was following the example in the cython docs Early Binding for Speed section, but now I see it got updated to cpdef! I'll keep hacking away at it. Perhaps I'll try a smaller example, to see the difference between lookups in classes and in structs, if that really is the issue. Is there a better way to find speed bottlenecks than the "comment out and re-compile" method? :) thanks, bb -- Brian Blais bblais at gmail.com http://web.bryant.edu/~bblais -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090508/9889e5e3/attachment.htm From robertwb at math.washington.edu Fri May 8 12:22:17 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 8 May 2009 03:22:17 -0700 Subject: [Cython] class versus struct...optimization In-Reply-To: References: Message-ID: On May 8, 2009, at 3:08 AM, Brian Blais wrote: > On May 7, 2009, at 20:42 , Lisandro Dalcin wrote: > >> Brian, in the Cython version, class Neuron, do this: instead of "def >> update(...)", please write "cpdef update(...)" and time your code >> again. If this make it faster (it should be near the C-struct >> version), please consider first removing your original "def update", >> and replace your "cdef _update()" by "cpdef update()". >> > > thanks, but this didn't change the time much at all (something > like .5 ms out of 20ms). I was following the example in the cython > docs Early Binding for Speed section, but now I see it got updated > to cpdef! > > I'll keep hacking away at it. Perhaps I'll try a smaller example, > to see the difference between lookups in classes and in structs, if > that really is the issue. > > Is there a better way to find speed bottlenecks than the "comment > out and re-compile" method? :) That's of course the only sure way to do it... Try compiling with "cython -a" and then opening up the resulting .html file. Yellow means Python/C API calls, which are often an indication of unintended inefficiencies. - Robert From kxroberto at googlemail.com Fri May 8 12:25:46 2009 From: kxroberto at googlemail.com (Robert) Date: Fri, 08 May 2009 12:25:46 +0200 Subject: [Cython] pyximport ignors [build] compiler = mingw32 In-Reply-To: References: Message-ID: Lisandro Dalcin wrote: > On Thu, May 7, 2009 at 11:10 AM, Robert wrote: >> On Windows in distutils/distutils.cfg MingW is set instead of VC >> default >> >> [build] >> compiler = mingw32 >> >> >> works well with extensions, including setup.py-method for building >> Cython modules. But it fails with pyximport method. >> Is this a bug - or how to tell using the MingW compiler? >> > > I would not say it is a bug, but rather a consequence of the > implementation of pyximport. > > Likely a call to "dist.parse_config_files()" should be added just > before "dist.parse_command_line()" in pyximport/pyxbuild.py > > However, if you do this, a "setup.cfg" file will be parse if present > in $PWD... This could be seen as a gotcha, or as a feature... > Thanks, in attachment my patch so far to: * only load the global distutils.cfg * extend the .pyxbld scheme for enabling custom setup() args * speed up import pyximport;pyximport.install() for use in auto startup files (by late imports). It was quite slowing down python startup crucially. Maybe its interesting for sb else too, or for general. Didn't find into http://trac.cython.org/cython_trac/ so far. Robert -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: pyxi.patch Url: http://codespeak.net/pipermail/cython-dev/attachments/20090508/6645f137/attachment.diff From robertwb at math.washington.edu Fri May 8 12:29:22 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 8 May 2009 03:29:22 -0700 Subject: [Cython] pyximport ignors [build] compiler = mingw32 In-Reply-To: References: Message-ID: <1F3DF633-6EF5-4AE9-B109-F3D481E55F68@math.washington.edu> On May 8, 2009, at 3:25 AM, Robert wrote: > Lisandro Dalcin wrote: >> On Thu, May 7, 2009 at 11:10 AM, Robert >> wrote: >>> On Windows in distutils/distutils.cfg MingW is set instead of VC >>> default >>> >>> [build] >>> compiler = mingw32 >>> >>> >>> works well with extensions, including setup.py-method for building >>> Cython modules. But it fails with pyximport method. >>> Is this a bug - or how to tell using the MingW compiler? >>> >> I would not say it is a bug, but rather a consequence of the >> implementation of pyximport. >> Likely a call to "dist.parse_config_files()" should be added just >> before "dist.parse_command_line()" in pyximport/pyxbuild.py >> However, if you do this, a "setup.cfg" file will be parse if present >> in $PWD... This could be seen as a gotcha, or as a feature... > > Thanks, > > in attachment my patch so far to: > > * only load the global distutils.cfg > * extend the .pyxbld scheme for enabling custom setup() args > * speed up import pyximport;pyximport.install() for use in auto > startup files (by late imports). It was quite slowing down python > startup crucially. > > Maybe its interesting for sb else too, or for general. > Didn't find into http://trac.cython.org/cython_trac/ so far. I don't use Windows, and nor do I use pyximport much, but I think these changes are valuable. Please add a ticket (mail me offlist if you don't yet have an account). - Robert From bblais at gmail.com Fri May 8 15:51:02 2009 From: bblais at gmail.com (Brian Blais) Date: Fri, 08 May 2009 09:51:02 -0400 Subject: [Cython] c-array of numpy ndarrays? Message-ID: Hello, In my question for optimization of my routines, I am trying to pre- fill a static c-array from a list of numpy ndarrays. Previously, I did something like: cdef double *w[30] cdef int r[30],c[30] and then I'd run through my list, and get the shape info and the data pointer, and fill in these arrays. This works, but then I have to do all my 2D -> 1D indexing, like: w[k][i+c[k]*j] which is ugly, and prone to typos. I'd like to do: cdef np.ndarray w[30] so I can do the nicer: w[k][i,j] I tried making an array of buffer objects, but I can only make local versions of that. c-arrays of python objects are forbidden. Is there a c-version of the ndarray? thanks, Brian Blais -- Brian Blais bblais at gmail.com http://web.bryant.edu/~bblais -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090508/2e27801a/attachment.htm From dagss at student.matnat.uio.no Fri May 8 16:27:11 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 08 May 2009 16:27:11 +0200 Subject: [Cython] c-array of numpy ndarrays? In-Reply-To: References: Message-ID: <4A04413F.6050903@student.matnat.uio.no> Brian Blais wrote: > Hello, > > In my question for optimization of my routines, I am trying to pre-fill > a static c-array from a list of numpy ndarrays. Previously, I did > something like: > > cdef double *w[30] > cdef int r[30],c[30] > > and then I'd run through my list, and get the shape info and the data > pointer, and fill in these arrays. This works, but then I have to do > all my 2D -> 1D indexing, like: > > w[k][i+c[k]*j] > > which is ugly, and prone to typos. I'd like to do: So the underlying problem here is that each array has a (widely different) size? > cdef np.ndarray w[30] No, it will be a long while before you can do this. Perhaps in a year one /might/ be able to use e.g. a C++ vector of buffers, but no promises. You can always wrap the solution above into e.g. a cdef class, so that you can write yourspecialarrayobject.get(k, i, j) -- Dag Sverre From dalcinl at gmail.com Fri May 8 17:02:19 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 8 May 2009 12:02:19 -0300 Subject: [Cython] pyximport ignors [build] compiler = mingw32 In-Reply-To: References: Message-ID: Robert, IMHO, you should load pydistutils.cfg, only remove 'setup.cfg'. pydistutils.cfg could be the only way to configure things for users without administrative rights... This should do the trick: cfgfiles = dist.find_config_files() try: cfgfiles.remove('setup.cfg') except ValueError: pass dist.parse_config_files(cfgfiles) On Fri, May 8, 2009 at 7:25 AM, Robert wrote: > Lisandro Dalcin wrote: >> >> On Thu, May 7, 2009 at 11:10 AM, Robert wrote: >>> >>> On Windows in distutils/distutils.cfg MingW is set instead of VC >>> default >>> >>> [build] >>> compiler = mingw32 >>> >>> >>> works well with extensions, including setup.py-method for building >>> Cython modules. But it fails with pyximport method. >>> Is this a bug - or how to tell using the MingW compiler? >>> >> >> I would not say it is a bug, but rather a consequence of the >> implementation of pyximport. >> >> Likely a call to ?"dist.parse_config_files()" should be added just >> before "dist.parse_command_line()" in pyximport/pyxbuild.py >> >> However, if you do this, a "setup.cfg" file will be parse if present >> in $PWD... This could be seen as a gotcha, or as a feature... >> > > Thanks, > > in attachment my patch so far to: > > * only load the global distutils.cfg > * extend the .pyxbld scheme for enabling custom setup() args > * speed up import pyximport;pyximport.install() for use in auto startup > files (by late imports). It was quite slowing down python startup crucially. > > Maybe its interesting for sb else too, or for general. > Didn't find into http://trac.cython.org/cython_trac/ so far. > > Robert > > Only in .: __init__.pyc > diff -u ../pyximport.orig\pyxbuild.py .\pyxbuild.py > --- ../pyximport.orig\pyxbuild.py ? ? ? Fri Apr 03 01:52:00 2009 > +++ .\pyxbuild.py ? ? ? Fri May 08 11:39:52 2009 > @@ -6,7 +6,6 @@ > ?import os > ?import sys > > -import distutils > ?from distutils.dist import Distribution > ?from distutils.errors import DistutilsArgError, DistutilsError, > CCompilerError > ?from distutils.extension import Extension > @@ -16,11 +15,10 @@ > ? ? HAS_CYTHON = True > ?except ImportError: > ? ? HAS_CYTHON = False > -import shutil > > ?DEBUG = 0 > ?def pyx_to_dll(filename, ext = None, force_rebuild = 0, > - ? ? ? ? ? ? ? build_in_temp=False, pyxbuild_dir=None): > + ? ? ? ? ? ? ? build_in_temp=False, pyxbuild_dir=None, setup_args={}): > ? ? """Compile a PYX file to a DLL and return the name of the generated .so > ? ? ? ?or .dll .""" > ? ? assert os.path.exists(filename), "Could not find %s" % > os.path.abspath(filename) > @@ -46,7 +44,11 @@ > ? ? ? ? args.append("--force") > ? ? if HAS_CYTHON and build_in_temp: > ? ? ? ? args.append("--pyrex-c-in-temp") > - ? ?dist = Distribution({"script_name": None, "script_args": args}) > + ? ?sargs=setup_args.copy() > + ? ?sargs.update( > + ? ? ? ?{"script_name": None, > + ? ? ? ? "script_args": args + setup_args.get("script_args",[])} ) > + ? ?dist = Distribution(sargs) > ? ? if not dist.ext_modules: > ? ? ? ? dist.ext_modules = [] > ? ? dist.ext_modules.append(ext) > @@ -55,6 +57,13 @@ > ? ? build = dist.get_command_obj('build') > ? ? build.build_base = pyxbuild_dir > > + ? ?# use global distutils.cfg only > + ? ?cfgfiles = [] > + ? ?sys_dir = os.path.dirname(sys.modules['distutils'].__file__) > + ? ?sys_file = os.path.join(sys_dir, "distutils.cfg") > + ? ?if os.path.isfile(sys_file): > + ? ? ? ?cfgfiles.append(sys_file) > + ? ?dist.parse_config_files(cfgfiles) ? # distutils/distutils.cfg; not > ./pydistutils.cfg; ./setup.cfg; > ? ? try: > ? ? ? ? ok = dist.parse_command_line() > ? ? except DistutilsArgError: > Only in .: pyxbuild.py.opt > Only in .: pyxbuild.pyc > Only in .: pyxi.patch > diff -u ../pyximport.orig\pyximport.py .\pyximport.py > --- ../pyximport.orig\pyximport.py ? ? ?Fri Apr 03 01:52:00 2009 > +++ .\pyximport.py ? ? ?Fri May 08 12:03:42 2009 > @@ -14,6 +14,22 @@ > ?sitecustomize.py with only those two lines at > ?/usr/local/lib/python2.3/site-packages/sitecustomize.py . > > +A custom distutils.core.Extension instance and setup() args > +(Distribution) for for the build can be defined by a .pyxbld > +file like: > + > +# examplemod.pyxbdl > +def make_ext(modname, pyxfilename): > + ? ?from distutils.extension import Extension > + ? ?return Extension(name = modname, > + ? ? ? ? ? ? ? ? ? ? sources=[pyxfilename, 'hello.c'], > + ? ? ? ? ? ? ? ? ? ? include_dirs=['/myinclude'] ) > +def make_setup_args(): > + ? ?return dict(script_args=["--compiler=mingw32"]) > + > +Extra dependencies can be defined by a .pyxdep . > +See README. > + > ?Since Cython 0.11, the :mod:`pyximport` module also has experimental > ?compilation support for normal Python modules. ?This allows you to > ?automatically run Cython on every .pyx and .py module that Python > @@ -34,14 +50,9 @@ > ?import os > ?import glob > ?import imp > -import pyxbuild > -from distutils.dep_util import newer > -from distutils.extension import Extension > - > -try: > - ? ?import hashlib > -except ImportError: > - ? ?import md5 as hashlib > +##import pyxbuild ?# changed: late import for speed > +##from distutils.dep_util import newer > +##from distutils.extension import Extension > > ?mod_name = "pyximport" > > @@ -64,30 +75,43 @@ > ? ? "Load a pyrex file given a name and filename." > > ?def get_distutils_extension(modname, pyxfilename): > - ? ?extra = "_" + hashlib.md5(open(pyxfilename).read()).hexdigest() > +# ? ?try: > +# ? ? ? ?import hashlib > +# ? ?except ImportError: > +# ? ? ? ?import md5 as hashlib > +# ? ?extra = "_" + hashlib.md5(open(pyxfilename).read()).hexdigest() > ?# ? ?modname = modname + extra > - ? ?extension_mod = handle_special_build(modname, pyxfilename) > + ? ?extension_mod,setup_args = handle_special_build(modname, pyxfilename) > ? ? if not extension_mod: > + ? ? ? ?from distutils.extension import Extension > ? ? ? ? extension_mod = Extension(name = modname, sources=[pyxfilename]) > - ? ?return extension_mod > + ? ?return extension_mod,setup_args > > ?def handle_special_build(modname, pyxfilename): > ? ? special_build = os.path.splitext(pyxfilename)[0] + PYXBLD_EXT > - > - ? ?if not os.path.exists(special_build): > - ? ? ? ?ext = None > - ? ?else: > - ? ? ? ?globls = {} > - ? ? ? ?locs = {} > + ? ?ext = None > + ? ?setup_args={} > + ? ?if os.path.exists(special_build): > + ? ? ? ?# globls = {} > + ? ? ? ?# locs = {} > ? ? ? ? # execfile(special_build, globls, locs) > ? ? ? ? # ext = locs["make_ext"](modname, pyxfilename) > ? ? ? ? mod = imp.load_source("XXXX", special_build, open(special_build)) > - ? ? ? ?ext = mod.make_ext(modname, pyxfilename) > - ? ? ? ?assert ext and ext.sources, ("make_ext in %s did not return > Extension" > - ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? % special_build) > + ? ? ? ?make_ext = getattr(mod,'make_ext') > + ? ? ? ?if make_ext: > + ? ? ? ? ? ?ext = make_ext(modname, pyxfilename) > + ? ? ? ? ? ?assert ext and ext.sources, ("make_ext in %s did not return > Extension" > + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? % special_build) > + ? ? ? ?make_setup_args = getattr(mod,'make_setup_args') > + ? ? ? ?if make_setup_args: > + ? ? ? ? ? ?setup_args = make_setup_args() > + ? ? ? ? ? ?assert isinstance(setup_args,dict), ("make_setup_args in %s did > not return a dict" > + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? % special_build) > + ? ? ? ?assert set or setup_args, ("neither make_ext nor make_setup_args > %s" > + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? % special_build) > ? ? ? ? ext.sources = [os.path.join(os.path.dirname(special_build), source) > ? ? ? ? ? ? ? ? ? ? ? ?for source in ext.sources] > - ? ?return ext > + ? ?return ext, setup_args > > ?def handle_dependencies(pyxfilename): > ? ? dependfile = os.path.splitext(pyxfilename)[0] + PYXDEP_EXT > @@ -116,6 +140,7 @@ > ? ? ? ? # the pyx file, 'touch' the pyx file so that distutils will > ? ? ? ? # be tricked into rebuilding it. > ? ? ? ? for file in files: > + ? ? ? ? ? ?from distutils.dep_util import newer > ? ? ? ? ? ? if newer(file, pyxfilename): > ? ? ? ? ? ? ? ? print "Rebuilding because of ", file > ? ? ? ? ? ? ? ? filetime = os.path.getmtime(file) > @@ -127,11 +152,13 @@ > ? ? ? ? "Path does not exist: %s" % pyxfilename) > ? ? handle_dependencies(pyxfilename) > > - ? ?extension_mod = get_distutils_extension(name, pyxfilename) > + ? ?extension_mod,setup_args = get_distutils_extension(name, pyxfilename) > > + ? ?import pyxbuild > ? ? so_path = pyxbuild.pyx_to_dll(pyxfilename, extension_mod, > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? build_in_temp=True, > - ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?pyxbuild_dir=pyxbuild_dir) > + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?pyxbuild_dir=pyxbuild_dir, > + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?setup_args=setup_args ) > ? ? assert os.path.exists(so_path), "Cannot find: %s" % so_path > > ? ? junkpath = os.path.join(os.path.dirname(so_path), name+"_*") > @@ -171,10 +198,10 @@ > ? ? ? ? ? ? print "SEARCHING", fullname, package_path > ? ? ? ? if '.' in fullname: > ? ? ? ? ? ? mod_parts = fullname.split('.') > - ? ? ? ? ? ?package = '.'.join(mod_parts[:-1]) > + ? ? ? ? ? ?#package = '.'.join(mod_parts[:-1]) > ? ? ? ? ? ? module_name = mod_parts[-1] > ? ? ? ? else: > - ? ? ? ? ? ?package = None > + ? ? ? ? ? ?#package = None > ? ? ? ? ? ? module_name = fullname > ? ? ? ? pyx_module_name = module_name + self.extension > ? ? ? ? # this may work, but it returns the file content, not its path > Only in .: pyximport.py.opt > Only in .: pyximport.pyc > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From bblais at gmail.com Fri May 8 17:26:31 2009 From: bblais at gmail.com (Brian Blais) Date: Fri, 08 May 2009 11:26:31 -0400 Subject: [Cython] c-array of numpy ndarrays? In-Reply-To: <4A04413F.6050903@student.matnat.uio.no> References: <4A04413F.6050903@student.matnat.uio.no> Message-ID: On May 8, 2009, at 10:27 , Dag Sverre Seljebotn wrote: > Brian Blais wrote: >> Hello, >> >> In my question for optimization of my routines, I am trying to pre- >> fill >> a static c-array from a list of numpy ndarrays. Previously, I did >> something like: >> >> cdef double *w[30] >> cdef int r[30],c[30] >> >> and then I'd run through my list, and get the shape info and the data >> pointer, and fill in these arrays. This works, but then I have to do >> all my 2D -> 1D indexing, like: >> >> w[k][i+c[k]*j] >> >> which is ugly, and prone to typos. I'd like to do: > > So the underlying problem here is that each array has a (widely > different) size? the underlying problem is that the difference between the following two pieces is about a factor of 20: # using a list of objects, getting an array (called weights) from each one # and doing some calculations cdef np.ndarray[DTYPE_t,ndim=2] weights for c in self.connections_to: num_incell=c.incell.N num_outcell=c.outcell.N weights=c.weights for __i in range(num_outcell): for __j in range(num_incell): a+=weights[__i,__j] and # using an array of structures, prefilled from a list of objects, # getting an pointer to the data (called weights) from each one # and doing some calculations cdef double *weights for cg from 0<=cg > You can always wrap the solution above into e.g. a cdef class, so that > you can write > > yourspecialarrayobject.get(k, i, j) that's a nice idea... thanks. bb -- Brian Blais bblais at gmail.com http://web.bryant.edu/~bblais -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090508/0077772d/attachment.htm From dalcinl at gmail.com Fri May 8 21:20:22 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 8 May 2009 16:20:22 -0300 Subject: [Cython] class versus struct...optimization In-Reply-To: References: Message-ID: On Fri, May 8, 2009 at 7:08 AM, Brian Blais wrote: > On May 7, 2009, at 20:42 , Lisandro Dalcin wrote: > > Brian, in the Cython version, class Neuron, do this: instead of "def > update(...)", please write "cpdef update(...)" and time your code > again. If this make it faster (it should be near the C-struct > version), please consider first removing your original "def update", > and replace your "cdef _update()" by "cpdef update()". > > thanks, but this didn't change the time much at all (something like .5 ms > out of 20ms). ?I was following the example in the cython docs Early Binding > for Speed section, but now I see it got updated to cpdef! > I'll keep hacking away at it. ?Perhaps I'll try a smaller example, to see > the difference between lookups in classes and in structs, if that really is > the issue. I'm almos sure that's the issue :-) Looking closer to your code, it seems the big offender is your Connection class, as it attributes are bare "object" instances... > Is there a better way to find speed bottlenecks than the "comment out and > re-compile" method? ?:) In general, any time to cdef'ine something as "object", you are going to pay the cost of Python attribute lookup via a dict... Try do get rid of these definitions in favor of explicit types... In short, write Cython code as if you were writing C++, where you have to explicitly type everything. > thanks, > > bb > > -- > Brian Blais > bblais at gmail.com > http://web.bryant.edu/~bblais > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Fri May 8 21:27:45 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 8 May 2009 12:27:45 -0700 Subject: [Cython] class versus struct...optimization In-Reply-To: References: Message-ID: <7E1F3DFD-4BDC-46D8-8BE3-A922602A3850@math.washington.edu> On May 8, 2009, at 12:20 PM, Lisandro Dalcin wrote: > On Fri, May 8, 2009 at 7:08 AM, Brian Blais wrote: >> On May 7, 2009, at 20:42 , Lisandro Dalcin wrote: >> >> Brian, in the Cython version, class Neuron, do this: instead of "def >> update(...)", please write "cpdef update(...)" and time your code >> again. If this make it faster (it should be near the C-struct >> version), please consider first removing your original "def update", >> and replace your "cdef _update()" by "cpdef update()". >> >> thanks, but this didn't change the time much at all (something >> like .5 ms >> out of 20ms). I was following the example in the cython docs >> Early Binding >> for Speed section, but now I see it got updated to cpdef! > > >> I'll keep hacking away at it. Perhaps I'll try a smaller example, >> to see >> the difference between lookups in classes and in structs, if that >> really is >> the issue. > > I'm almos sure that's the issue :-) > > Looking closer to your code, it seems the big offender is your > Connection class, as it attributes are bare "object" instances... > >> Is there a better way to find speed bottlenecks than the "comment >> out and >> re-compile" method? :) > > In general, any time to cdef'ine something as "object", you are going > to pay the cost of Python attribute lookup via a dict... Try do get > rid of these definitions in favor of explicit types... In short, write > Cython code as if you were writing C++, where you have to explicitly > type everything. Well, the good thing is that one you only have to type the critical parts of the code, and can use the convienience of Python in not caring for the (usually) bulk of the code linewise that only takes 1% of the runtime. - Robert From dalcinl at gmail.com Fri May 8 21:40:50 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 8 May 2009 16:40:50 -0300 Subject: [Cython] Calling conventions on Windows Message-ID: I'm having a hard time trying to make mpi4py compatible with Microsoft MPI. All the MPI calls are annotated with __stdcall, and callbacks functions (in my case written in Cython) should also be. But other MPI implementations (MPICH2 and DeinoMPI) does different, they require __cdecl. I cannot figure out how to solve that without the help of Cython.... I'm thinking about ADDITIONALLY supporting the following sintax: cde int "CALL_SPEC" somefunc(): then I would be able to pass -DCALL_SPEC=__xxx or even conditionally "#define CALL_SPEC __xxx" in some header... This could have other uses as well, as the string literal could provide other annotations (like __attribute__ stuff in GCC). I've already modified the parser and all seems to work. The only thing left is to make Cython not so strict about the signatures I mean, if I use a bare string literal for annotating the function, then the function types int "ABC" foo() int "XYZ" bar() would be treated as equivalent to the unannotated: inf fun() What do you think? BTW, we should add support for __fastcall calling conventions, like this int __fastcall func() - -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From sccolbert at gmail.com Fri May 8 22:07:41 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Fri, 8 May 2009 16:07:41 -0400 Subject: [Cython] newbie question concerning scope Message-ID: <7f014ea60905081307w1676bb38u5105c274b691a469@mail.gmail.com> Hi all, Just recently got into working with cython for speeding up some calculation with numpy. this question, however, has nothing to do with numpy. The following code is valid python: var = 0 def count(n): for i in range(n): var += i return var Translating to cython results in run-time errors: cdef double var = 0.0 def count(int n): cdef int i for i in range(n): var += i return var However this works (as it should, it's the better way to do it): def count(int n): cdef int i cdef double var = 0.0 for i in range(n): var += i return var now, i'm not debating that anyone would ever want to use the second method, but regardless, in that version, why isn't var found in the module scope? The runtime error i get is 'can't assign nonetype to int' Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090508/40486362/attachment.htm From dagss at student.matnat.uio.no Fri May 8 22:28:55 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 08 May 2009 22:28:55 +0200 Subject: [Cython] Calling conventions on Windows In-Reply-To: References: Message-ID: <4A049607.5040805@student.matnat.uio.no> Lisandro Dalcin wrote: > I'm having a hard time trying to make mpi4py compatible with Microsoft > MPI. All the MPI calls are annotated with __stdcall, and callbacks > functions (in my case written in Cython) should also be. But other MPI > implementations (MPICH2 and DeinoMPI) does different, they require > __cdecl. I cannot figure out how to solve that without the help of > Cython.... > > I'm thinking about ADDITIONALLY supporting the following sintax: > > cde int "CALL_SPEC" somefunc(): > > then I would be able to pass -DCALL_SPEC=__xxx or even conditionally > "#define CALL_SPEC __xxx" in some header... > > This could have other uses as well, as the string literal could > provide other annotations (like __attribute__ stuff in GCC). > > I've already modified the parser and all seems to work. The only thing > left is to make Cython not so strict about the signatures Could we have a decorator instead, perhaps? Or some way of declaring a new callspec? ccallspec __stdcall ... I'm just thinking about the cdef extern unsigned int public inline __stdcall foo "realfoo" () expect+: ... > > I mean, if I use a bare string literal for annotating the function, > then the function types > > int "ABC" foo() > > int "XYZ" bar() > > would be treated as equivalent to the unannotated: > > inf fun() > > What do you think? > > BTW, we should add support for __fastcall calling conventions, like this > > int __fastcall func() > > - > -- Dag Sverre From dagss at student.matnat.uio.no Fri May 8 22:35:33 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 08 May 2009 22:35:33 +0200 Subject: [Cython] Calling conventions on Windows In-Reply-To: <4A049607.5040805@student.matnat.uio.no> References: <4A049607.5040805@student.matnat.uio.no> Message-ID: <4A049795.2090506@student.matnat.uio.no> Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: >> I'm having a hard time trying to make mpi4py compatible with Microsoft >> MPI. All the MPI calls are annotated with __stdcall, and callbacks >> functions (in my case written in Cython) should also be. But other MPI >> implementations (MPICH2 and DeinoMPI) does different, they require >> __cdecl. I cannot figure out how to solve that without the help of >> Cython.... >> >> I'm thinking about ADDITIONALLY supporting the following sintax: >> >> cde int "CALL_SPEC" somefunc(): >> >> then I would be able to pass -DCALL_SPEC=__xxx or even conditionally >> "#define CALL_SPEC __xxx" in some header... >> >> This could have other uses as well, as the string literal could >> provide other annotations (like __attribute__ stuff in GCC). >> >> I've already modified the parser and all seems to work. The only thing >> left is to make Cython not so strict about the signatures > > Could we have a decorator instead, perhaps? Or some way of declaring a > new callspec? > > ccallspec __stdcall > > ... > > I'm just thinking about the > > cdef extern unsigned int public inline __stdcall foo "realfoo" () > expect+: ... Sorry, accidentally hit send. The order is wrong but you get my point (and I forgot to add "api"). >> I mean, if I use a bare string literal for annotating the function, >> then the function types >> >> int "ABC" foo() Well consider cdef extern: int "__fastcall" foo "realfoo" () It's not a very readable syntax (did the real name go before or after? With this change, the compiler won't complain if you try the wrong one first) If one instead had a flexible mechanism for declaring "__fastcall" and friends so that the quotes can disappear, it looks much more readable: cdef extern: int __fastcall foo "realfoo" () -- Dag Sverre From stefan_ml at behnel.de Sat May 9 05:50:29 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 09 May 2009 05:50:29 +0200 Subject: [Cython] newbie question concerning scope In-Reply-To: <7f014ea60905081307w1676bb38u5105c274b691a469@mail.gmail.com> References: <7f014ea60905081307w1676bb38u5105c274b691a469@mail.gmail.com> Message-ID: <4A04FD85.3050908@behnel.de> Chris Colbert wrote: > Just recently got into working with cython for speeding up some calculation > with numpy. > > this question, however, has nothing to do with numpy. > > The following code is valid python: > > var = 0 > > def count(n): > for i in range(n): > var += i > > return var Valid only in the sense that it passes the parser. >>> count(5) Traceback (most recent call last): File "", line 1, in File "", line 3, in count UnboundLocalError: local variable 'var' referenced before assignment > Translating to cython results in run-time errors: > > cdef double var = 0.0 > > def count(int n): > cdef int i > for i in range(n): > var += i > > return var Just like in Python. Cython currently isn't smart enough to figure out at compile time that the local variable var is never initialised in your example. It therefore assigns None to it on initialisation to make sure things don't break at runtime. As in Python, you need to use the "global" statement to make the above work. > The runtime error i get is 'can't assign nonetype to int' Didn't try, but I doubt that that's the exact error message. Stefan From dsurviver at gmail.com Sat May 9 09:43:10 2009 From: dsurviver at gmail.com (Danilo Freitas) Date: Sat, 9 May 2009 04:43:10 -0300 Subject: [Cython] newbie question concerning scope In-Reply-To: <7f014ea60905081307w1676bb38u5105c274b691a469@mail.gmail.com> References: <7f014ea60905081307w1676bb38u5105c274b691a469@mail.gmail.com> Message-ID: <45239150905090043o1326a8b1oa1a4195a64cabe9f@mail.gmail.com> As Stefan said... try: > cdef double var = 0.0 > > def count(int n): > ??? cdef int i global var > ??? for i in range(n): > ??????? var += i > > ??? return var I don't know if this is valid in Cython... but you may try it. I don't how Cython parser works with the "global" keyworld :/ If it isn't valid, somebody may try to fix it (if possible :) ). I don't have enought knowledge about it. The only way to discover it is trying. When you do it, please post here. (I'm speaking only to myself... I don't know if this is a already known problem, and if there is a solution). From kxroberto at googlemail.com Sat May 9 17:19:32 2009 From: kxroberto at googlemail.com (Robert) Date: Sat, 09 May 2009 17:19:32 +0200 Subject: [Cython] pyximport ignors [build] compiler = mingw32 In-Reply-To: References: Message-ID: Lisandro Dalcin wrote: > Robert, IMHO, you should load pydistutils.cfg, only remove > 'setup.cfg'. pydistutils.cfg could be the only way to configure things > for users without administrative rights... > > This should do the trick: > > cfgfiles = dist.find_config_files() > try: cfgfiles.remove('setup.cfg') > except ValueError: pass > dist.parse_config_files(cfgfiles) > > yes, just ./setup.cfg seems to be questionable this and more related cleanup, speedup of importer, --inplace --verbose reload() .. here now: http://trac.cython.org/cython_trac/ticket/312 From sccolbert at gmail.com Sun May 10 00:19:34 2009 From: sccolbert at gmail.com (Chris Colbert) Date: Sat, 9 May 2009 18:19:34 -0400 Subject: [Cython] newbie question concerning scope In-Reply-To: <45239150905090043o1326a8b1oa1a4195a64cabe9f@mail.gmail.com> References: <7f014ea60905081307w1676bb38u5105c274b691a469@mail.gmail.com> <45239150905090043o1326a8b1oa1a4195a64cabe9f@mail.gmail.com> Message-ID: <7f014ea60905091519o5d763a63gfadd3b1a6c8f137@mail.gmail.com> I tested that and it works, thanks. Chris On Sat, May 9, 2009 at 3:43 AM, Danilo Freitas wrote: > As Stefan said... > try: > > > cdef double var = 0.0 > > > > def count(int n): > > cdef int i > global var > > for i in range(n): > > var += i > > > > return var > > I don't know if this is valid in Cython... > but you may try it. > I don't how Cython parser works with the "global" keyworld :/ > > If it isn't valid, somebody may try to fix it (if possible :) ). I > don't have enought knowledge about it. The only way to discover it is > trying. > When you do it, please post here. (I'm speaking only to myself... I > don't know if this is a already known problem, and if there is a > solution). > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090509/e0895f61/attachment.htm From dsurviver at gmail.com Sun May 10 02:05:39 2009 From: dsurviver at gmail.com (Danilo Freitas) Date: Sat, 9 May 2009 21:05:39 -0300 Subject: [Cython] newbie question concerning scope In-Reply-To: <7f014ea60905091519o5d763a63gfadd3b1a6c8f137@mail.gmail.com> References: <7f014ea60905081307w1676bb38u5105c274b691a469@mail.gmail.com> <45239150905090043o1326a8b1oa1a4195a64cabe9f@mail.gmail.com> <7f014ea60905091519o5d763a63gfadd3b1a6c8f137@mail.gmail.com> Message-ID: <45239150905091705u3f07dbf9p46df9968e5538aad@mail.gmail.com> 2009/5/9 Chris Colbert : > I tested that and it works, thanks. > > Chris > It was nothing :D Thanks you too. From greg.ewing at canterbury.ac.nz Sun May 10 03:00:41 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 10 May 2009 13:00:41 +1200 Subject: [Cython] Calling conventions on Windows In-Reply-To: <4A049795.2090506@student.matnat.uio.no> References: <4A049607.5040805@student.matnat.uio.no> <4A049795.2090506@student.matnat.uio.no> Message-ID: <4A062739.2010606@canterbury.ac.nz> Dag Sverre Seljebotn wrote: > cdef extern: > int "__fastcall" foo "realfoo" () Why aren't you getting this from a header file? -- Greg From bblais at gmail.com Mon May 11 12:01:32 2009 From: bblais at gmail.com (Brian Blais) Date: Mon, 11 May 2009 06:01:32 -0400 Subject: [Cython] #define, or equivalent? Message-ID: Hello, I have a number of arrays that need to be some maximum size, and I was wondering if there is some way to define that size at the top of the code, the equivalent of: #define MAXSIZE 10 I haven't seen any examples of this. So far I have just put the number in manually, like: cdef a[10],b[10],c[10] but I'd rather have: cdef a[MAXSIZE],b[MAXSIZE],c[MAXSIZE] is there a way of doing that? thanks, Brian Blais -- Brian Blais bblais at gmail.com http://web.bryant.edu/~bblais -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/cython-dev/attachments/20090511/3a02dd3f/attachment.htm From dalcinl at gmail.com Mon May 11 15:33:15 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 11 May 2009 10:33:15 -0300 Subject: [Cython] Calling conventions on Windows In-Reply-To: <4A062739.2010606@canterbury.ac.nz> References: <4A049607.5040805@student.matnat.uio.no> <4A049795.2090506@student.matnat.uio.no> <4A062739.2010606@canterbury.ac.nz> Message-ID: As Greg pointed out, this has nothing to do with external declarations. The actual issue is when you need to implement in Cython code a callback function. In my particular issue, these callbacks have to respect a standardized API. And the problem is that different implementations of that standardized API use different calling conventions... Then I want my Cython code to work with any of these implementation, but I only see two solutions: 1) Implement all my callbacks in C and include them with 'cdef extern'. But then I've just lost the power of Cython for implementing C functions, particularly when Python objects are involved and the GIL has to be properly managed. 2) Modify Cython to accept any string literal instead of just __stdcall/__cdecl/__fastcall. Then the contents of that literal could be a macro and I would be able to control its actual contents via the C preprocessor. Can any of you figure out an alternative to (2) to save me (or any other with similar issues) from (1) ?? -- On Sat, May 9, 2009 at 10:00 PM, Greg Ewing wrote: > Dag Sverre Seljebotn wrote: > >> cdef extern: >> ? ? ?int "__fastcall" foo "realfoo" () > > Why aren't you getting this from a header file? > > -- > Greg > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Mon May 11 16:45:09 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 11 May 2009 16:45:09 +0200 (CEST) Subject: [Cython] Calling conventions on Windows In-Reply-To: References: <4A049607.5040805@student.matnat.uio.no> <4A049795.2090506@student.matnat.uio.no> <4A062739.2010606@canterbury.ac.nz> Message-ID: <0ade16b730dabdc7433ffa8453e8f0ef.squirrel@webmail.uio.no> Lisandro wrote: > As Greg pointed out, this has nothing to do with external > declarations. The actual issue is when you need to implement in Cython > code a callback function. Sure, you will typically not need cdef extern "declspec" func "realname" () because you can then use a header file. However, it is about creating a syntax in Cython which is easy to learn and remember! And that means making each feature working in any situation where one would "guess" it would work and not put in arbitrary limits (I should know, the number of complaints I get on the buffer-in-function-only restriction :-) ). And making lots of special cases (this syntax is not allowed in this case and this syntax is not allowed in that case) complicates the language a lot. > In my particular issue, these callbacks have to respect a standardized > API. And the problem is that different implementations of that > standardized API use different calling conventions... Then I want my > Cython code to work with any of these implementation, but I only see > two solutions: > > 1) Implement all my callbacks in C and include them with 'cdef > extern'. But then I've just lost the power of Cython for implementing > C functions, particularly when Python objects are involved and the GIL > has to be properly managed. > > 2) Modify Cython to accept any string literal instead of just > __stdcall/__cdecl/__fastcall. Then the contents of that literal could > be a macro and I would be able to control its actual contents via the > C preprocessor. > > Can any of you figure out an alternative to (2) to save me (or any > other with similar issues) from (1) ?? Yes, as I proposed, some kind of syntax to declare your own calling convention; e.g. cdef callspec __mycall "__mycall" cdef __mycall int foo(): return 3 This will avoid cdef extern: int "realfunc" func() from silently compiling but producing unexpected code, without having to arbitrarily disable parts of the language in a perfectly valid corner case. Dag Sverre From dagss at student.matnat.uio.no Mon May 11 16:52:33 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 11 May 2009 16:52:33 +0200 (CEST) Subject: [Cython] Calling conventions on Windows In-Reply-To: <0ade16b730dabdc7433ffa8453e8f0ef.squirrel@webmail.uio.no> References: <4A049607.5040805@student.matnat.uio.no> <4A049795.2090506@student.matnat.uio.no> <4A062739.2010606@canterbury.ac.nz> <0ade16b730dabdc7433ffa8453e8f0ef.squirrel@webmail.uio.no> Message-ID: <7945881b0549cbd48f825565d33060bb.squirrel@webmail.uio.no> Dag wrote: > Lisandro wrote: >> API. And the problem is that different implementations of that >> standardized API use different calling conventions... Then I want my >> Cython code to work with any of these implementation, but I only see >> two solutions: >> >> 1) Implement all my callbacks in C and include them with 'cdef >> extern'. But then I've just lost the power of Cython for implementing >> C functions, particularly when Python objects are involved and the GIL >> has to be properly managed. >> >> 2) Modify Cython to accept any string literal instead of just >> __stdcall/__cdecl/__fastcall. Then the contents of that literal could >> be a macro and I would be able to control its actual contents via the >> C preprocessor. >> >> Can any of you figure out an alternative to (2) to save me (or any >> other with similar issues) from (1) ?? > > Yes, as I proposed, some kind of syntax to declare your own calling > convention; e.g. > > cdef callspec __mycall "__mycall" > > cdef __mycall int foo(): return 3 Alternatively: from cython import callconvention @callconvention("__mycall") cdef int foo(): ... though it might be too long? (Shorter name possible on decorator too.) Similarily one could have @gccattribute("...") which I'd prefer to hacking this into a callconvention mechanism. Dag Sverre From dalcinl at gmail.com Mon May 11 17:54:59 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 11 May 2009 12:54:59 -0300 Subject: [Cython] Calling conventions on Windows In-Reply-To: <7945881b0549cbd48f825565d33060bb.squirrel@webmail.uio.no> References: <4A049607.5040805@student.matnat.uio.no> <4A049795.2090506@student.matnat.uio.no> <4A062739.2010606@canterbury.ac.nz> <0ade16b730dabdc7433ffa8453e8f0ef.squirrel@webmail.uio.no> <7945881b0549cbd48f825565d33060bb.squirrel@webmail.uio.no> Message-ID: On Mon, May 11, 2009 at 11:52 AM, Dag Sverre Seljebotn wrote: > Dag wrote: >> >> Yes, as I proposed, some kind of syntax to declare your own calling >> convention; e.g. >> >> cdef callspec __mycall "__mycall" >> >> cdef __mycall int foo(): return 3 > I really like this, but I also understand that it is going to be a burden to implement. > Alternatively: > > from cython import callconvention > > @callconvention("__mycall") > cdef int foo(): ... > > though it might be too long? (Shorter name possible on decorator too.) @callspec("__mycall") seems short and explicit enough for me. > Similarily one could have > > @gccattribute("...") > > which I'd prefer to hacking this into a callconvention mechanism. > > Dag Sverre OK, even better, though I would call it "@attribute(...)" or something like that, as the "gcc" bit seems not appropriate (because the the syntax could target special features of other compilers, not just GCC) -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From dagss at student.matnat.uio.no Mon May 11 20:18:24 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 11 May 2009 20:18:24 +0200 Subject: [Cython] Calling conventions on Windows In-Reply-To: References: <4A049607.5040805@student.matnat.uio.no> <4A049795.2090506@student.matnat.uio.no> <4A062739.2010606@canterbury.ac.nz> <0ade16b730dabdc7433ffa8453e8f0ef.squirrel@webmail.uio.no> <7945881b0549cbd48f825565d33060bb.squirrel@webmail.uio.no> Message-ID: <4A086BF0.5070007@student.matnat.uio.no> Lisandro Dalcin wrote: > On Mon, May 11, 2009 at 11:52 AM, Dag Sverre Seljebotn > wrote: >> Dag wrote: >>> Yes, as I proposed, some kind of syntax to declare your own calling >>> convention; e.g. >>> >>> cdef callspec __mycall "__mycall" >>> >>> cdef __mycall int foo(): return 3 > > I really like this, but I also understand that it is going to be a > burden to implement. > >> Alternatively: >> >> from cython import callconvention >> >> @callconvention("__mycall") >> cdef int foo(): ... >> >> though it might be too long? (Shorter name possible on decorator too.) > > @callspec("__mycall") seems short and explicit enough for me. +1. Especially since no change in the parser is required :-) (Should we have a +1 from Robert or Stefan as well you think?) If you end up doing this, here's one (of many) strategies: Compiler directives (Options.py and somewhere in ParseTreeTransforms.py) must be extended to accept string arguments and possibly extended with a mechanism to make directives only valid in certain contexts (no "with callspec(...):" :-) ). (Technically this is perhaps not a compiler directive, but it's an easy way of implementing it and not *too* far off...) -- Dag Sverre From robertwb at math.washington.edu Mon May 11 20:37:06 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 11 May 2009 11:37:06 -0700 Subject: [Cython] Calling conventions on Windows In-Reply-To: <4A086BF0.5070007@student.matnat.uio.no> References: <4A049607.5040805@student.matnat.uio.no> <4A049795.2090506@student.matnat.uio.no> <4A062739.2010606@canterbury.ac.nz> <0ade16b730dabdc7433ffa8453e8f0ef.squirrel@webmail.uio.no> <7945881b0549cbd48f825565d33060bb.squirrel@webmail.uio.no> <4A086BF0.5070007@student.matnat.uio.no> Message-ID: On May 11, 2009, at 11:18 AM, Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: >> On Mon, May 11, 2009 at 11:52 AM, Dag Sverre Seljebotn >> wrote: >>> Dag wrote: >>>> Yes, as I proposed, some kind of syntax to declare your own calling >>>> convention; e.g. >>>> >>>> cdef callspec __mycall "__mycall" >>>> >>>> cdef __mycall int foo(): return 3 >> >> I really like this, but I also understand that it is going to be a >> burden to implement. >> >>> Alternatively: >>> >>> from cython import callconvention >>> >>> @callconvention("__mycall") >>> cdef int foo(): ... >>> >>> though it might be too long? (Shorter name possible on decorator >>> too.) >> >> @callspec("__mycall") seems short and explicit enough for me. > > +1. Especially since no change in the parser is required :-) I like @callspec("__mycall"") too, and the preferred way to do things (rather than the ones we support so far). An @attribute decorators would be useful too, but orthogonal (I'd rather have it just be for __attribute__.). - Robert From dagss at student.matnat.uio.no Mon May 11 20:49:52 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 11 May 2009 20:49:52 +0200 Subject: [Cython] Calling conventions on Windows In-Reply-To: References: <4A049607.5040805@student.matnat.uio.no> <4A049795.2090506@student.matnat.uio.no> <4A062739.2010606@canterbury.ac.nz> <0ade16b730dabdc7433ffa8453e8f0ef.squirrel@webmail.uio.no> <7945881b0549cbd48f825565d33060bb.squirrel@webmail.uio.no> Message-ID: <4A087350.1000907@student.matnat.uio.no> Lisandro Dalcin wrote: > On Mon, May 11, 2009 at 11:52 AM, Dag Sverre Seljebotn >> Similarily one could have >> >> @gccattribute("...") >> >> which I'd prefer to hacking this into a callconvention mechanism. >> >> Dag Sverre > > OK, even better, though I would call it "@attribute(...)" or something > like that, as the "gcc" bit seems not appropriate (because the the > syntax could target special features of other compilers, not just GCC) > The reason I proposed to call it "gccattribute" is that AFAIK GCC is the compiler which introduces these and that is how they are referred to (although I think icc supports a lot of them as well). I think "attribute" is far to generic; "cattribute" is slightly better but still these are not standard C. (In most cases we are better off implementing higher-level support anyway, like I did with packed structs, where differenct extensions from more than one compiler is used through #ifdefs. I'm +0 on adding gccattribute, I just responded to Lisandro's ideas on it.) -- Dag Sverre From robertwb at math.washington.edu Mon May 11 21:00:58 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 11 May 2009 12:00:58 -0700 Subject: [Cython] Calling conventions on Windows In-Reply-To: <4A087350.1000907@student.matnat.uio.no> References: <4A049607.5040805@student.matnat.uio.no> <4A049795.2090506@student.matnat.uio.no> <4A062739.2010606@canterbury.ac.nz> <0ade16b730dabdc7433ffa8453e8f0ef.squirrel@webmail.uio.no> <7945881b0549cbd48f825565d33060bb.squirrel@webmail.uio.no> <4A087350.1000907@student.matnat.uio.no> Message-ID: <6AFC4F3E-C3B3-45C0-A1F8-517C74672CB8@math.washington.edu> On May 11, 2009, at 11:49 AM, Dag Sverre Seljebotn wrote: > Lisandro Dalcin wrote: >> On Mon, May 11, 2009 at 11:52 AM, Dag Sverre Seljebotn >>> Similarily one could have >>> >>> @gccattribute("...") >>> >>> which I'd prefer to hacking this into a callconvention mechanism. >>> >>> Dag Sverre >> >> OK, even better, though I would call it "@attribute(...)" or >> something >> like that, as the "gcc" bit seems not appropriate (because the the >> syntax could target special features of other compilers, not just >> GCC) >> > > The reason I proposed to call it "gccattribute" is that AFAIK GCC > is the > compiler which introduces these and that is how they are referred to > (although I think icc supports a lot of them as well). > > I think "attribute" is far to generic; "cattribute" is slightly better > but still these are not standard C. Yeah, gccattribute or cattribute are more clear. > (In most cases we are better off implementing higher-level support > anyway, like I did with packed structs, where differenct extensions > from > more than one compiler is used through #ifdefs. I'm +0 on adding > gccattribute, I just responded to Lisandro's ideas on it.) This opens the option for every obscure attribute one might want to use, whether/before we support it at a higher level. I don't think it's high priority though. - Robert From lrhazi at gmail.com Tue May 12 03:51:13 2009 From: lrhazi at gmail.com (Mohamed Lrhazi) Date: Mon, 11 May 2009 21:51:13 -0400 Subject: [Cython] How do I pass opaque pointers back and forth? Message-ID: Hello, Is it possible in Cython to pass a void pointer to a python function, then have the latter send it back as a parameter to a function defined in Cython, then be able to reuse it? Basically, I am using a C lib that sends back these pointers, pointing to structures whose definitions in the header file are undefined/incomplete. I would like to send them as is to Python, but then be able to recover them back and resend them to lib. I can keep them in Cython as global variables, but was wondering if I could transparently pass them back and forth instead. I thought maybe I could pass the actual address, as in: &pointer[0] but could not figure out how to make a new pointer in Cython assigning the recovered address to it. Thanks a lot, Mohamed. From lrhazi at gmail.com Tue May 12 04:13:18 2009 From: lrhazi at gmail.com (Mohamed Lrhazi) Date: Mon, 11 May 2009 22:13:18 -0400 Subject: [Cython] How do I pass opaque pointers back and forth? In-Reply-To: References: Message-ID: It seems I can make it work passing the address as &pointer[0], then recovering t as: cdef void *new_pointer = recovered_value Is this correct? or is it just luck that my program is not crashing yet :) Thanks, Mohamed. On Mon, May 11, 2009 at 9:51 PM, Mohamed Lrhazi wrote: > Hello, > > Is it possible in Cython to pass a void pointer to a python function, > then have the latter send it back as a parameter to a ?function > defined in Cython, then be able to reuse it? > > Basically, I am using a C lib that sends back these pointers, pointing > to structures whose definitions in the header file are > undefined/incomplete. I would like to send them as is to Python, but > then be able to recover them back and resend them to lib. > > I can keep them in Cython as global variables, but was wondering if I > could transparently pass them back and forth instead. > > I thought maybe I could pass the actual address, as in: &pointer[0] > > but could not figure out how to make a new pointer in Cython assigning > the recovered address to it. > > Thanks a lot, > Mohamed. > -- " Logic merely sanctions the conquests of the intuition." Jacques Hadamard From dalcinl at gmail.com Tue May 12 04:25:30 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 11 May 2009 23:25:30 -0300 Subject: [Cython] How do I pass opaque pointers back and forth? In-Reply-To: References: Message-ID: On Mon, May 11, 2009 at 11:13 PM, Mohamed Lrhazi wrote: > It seems I can make it work passing the address as > &pointer[0], then recovering t as: > > cdef void *new_pointer = recovered_value > > Is this correct? or is it just luck that my program is not crashing yet :) > It is correct as long as you pass back the "correct" integer value... If you ever make a mistake, you will likely have a segfault... Depending on your original C API, you could consider designing an more object oriented, pythonic API where your pointers are (cdef) attributes of a (cdef) class, and their methods provide access to the many function calls of the lib you are wrapping... > Thanks, > Mohamed. > > On Mon, May 11, 2009 at 9:51 PM, Mohamed Lrhazi wrote: >> Hello, >> >> Is it possible in Cython to pass a void pointer to a python function, >> then have the latter send it back as a parameter to a ?function >> defined in Cython, then be able to reuse it? >> >> Basically, I am using a C lib that sends back these pointers, pointing >> to structures whose definitions in the header file are >> undefined/incomplete. I would like to send them as is to Python, but >> then be able to recover them back and resend them to lib. >> >> I can keep them in Cython as global variables, but was wondering if I >> could transparently pass them back and forth instead. >> >> I thought maybe I could pass the actual address, as in: &pointer[0] >> >> but could not figure out how to make a new pointer in Cython assigning >> the recovered address to it. >> >> Thanks a lot, >> Mohamed. >> > > > > -- > " Logic merely sanctions the conquests of the intuition." > Jacques Hadamard > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From robertwb at math.washington.edu Tue May 12 06:21:15 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 11 May 2009 21:21:15 -0700 Subject: [Cython] How do I pass opaque pointers back and forth? In-Reply-To: References: Message-ID: <852A1B6B-7694-42C1-A1FD-AB3D3B499B0C@math.washington.edu> On May 11, 2009, at 7:25 PM, Lisandro Dalcin wrote: > On Mon, May 11, 2009 at 11:13 PM, Mohamed Lrhazi > wrote: >> It seems I can make it work passing the address as >> &pointer[0], then recovering t as: >> >> cdef void *new_pointer = recovered_value >> >> Is this correct? or is it just luck that my program is not >> crashing yet :) >> > > It is correct as long as you pass back the "correct" integer value... > If you ever make a mistake, you will likely have a segfault... > > Depending on your original C API, you could consider designing an more > object oriented, pythonic API where your pointers are (cdef) > attributes of a (cdef) class, and their methods provide access to the > many function calls of the lib you are wrapping... See also http://docs.python.org/c-api/cobject.html which is made for this kind of thing. - Robert From dagss at student.matnat.uio.no Tue May 12 09:50:42 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Tue, 12 May 2009 09:50:42 +0200 Subject: [Cython] Notes on code streams refactoring Message-ID: <4A092A52.2010309@student.matnat.uio.no> I found myself in a corner where the utility code mechanism was not fine-grained enough. Rather than adding yet-another-special-case I decided to refactor some stuff. Reviews welcome; I can of course change it back (though hopefully not until after 0.11.2). a) In Code.py: GlobalState, there's now a list code_layout: code_layout = [ 'h_code', 'utility_code_proto', 'type_declarations', 'module_declarations', 'typeinfo', 'before_global_var', 'global_var', 'all_the_rest', 'utility_code_def' ] This identifies code stream blocks that are set up initially (in that order). b) At any point, your code instance represents the default location, but you can also output to other places: otherplace = code.globalstate['typeinfo'] otherplace.putln(...) If just the right location isn't available, just split up an existing stream in (a). c) An alternative to the classic utility code is now this: if your_id not in code.globalstate.utility_codes: code.globalstate.utility_codes.add(your_id) somestream = code.globalstate['somestream'] somestream.putln(...) I.e. the presence of "some utility code" is now fully orthogonal to which code stream/location in the output the code ends up at. (The UtilityCode class is a convenience for this with a couple of hard-coded streams.) http://hg.cython.org/cython-devel/rev/b86a7b374a95 http://hg.cython.org/cython-devel/rev/fd6cc8a5998d http://hg.cython.org/cython-devel/rev/df8dd466bc4a etc. -- Dag Sverre From Dave.Cross at cdl.co.uk Tue May 12 14:47:11 2009 From: Dave.Cross at cdl.co.uk (Dave Cross) Date: Tue, 12 May 2009 13:47:11 +0100 Subject: [Cython] Problem importing Cython built module from within embedded interpreter. Message-ID: <3C1CBB29A3D34C479B570500C7A73F2907BCD36B@cdlms2.cheshdatasys.co.uk> Hi, We have embedded python 2.6 within our C++ app and it runs quite substantial Python scripts operating on the C++ data perfectly OK. I have just compiled a small module (foo) using Cython and it works fine standalone and within the main script under normal Python. >From within the embedded interpreter, however, the main script reports ImportError: No module named foo despite me specifically appending the source folder to sys.path just prior to the import. Not sure if this is related but I need to do from foo import * in order to get the imported python class name into my local namespace. Any ideas anybody? Dave.

**********************************************************************
Please consider the environment - do you really need to print this email?

This email is intended only for the person(s) named above and may contain private and confidential information. If it has come to you in error, please destroy and permanently delete any copy in your possession and contact us on +44 (0) 161 480 4420. The information in this email is copyright © CDL Group Holdings Limited. We cannot accept any liability for any loss or damage sustained as a result of software viruses. It is your responsibility to carry out such virus checking as is necessary before opening any attachment.
Cheshire Datasystems Limited uses software which automatically screens incoming emails for inappropriate content and attachments. If the software identifies such content or attachment, the email will be forwarded to our Technology Department for checking. You should be aware that any email which you send to Cheshire Datasystems Limited is subject to this procedure.
Cheshire Datasystems Limited, Strata House, Kings Reach Road, Stockport SK4 2HD
Registered in England and Wales with Company Number 3991057
VAT registration: 727 1188 33

 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20090512/c5c155ef/attachment.htm 

From dagss at student.matnat.uio.no  Tue May 12 17:58:05 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 12 May 2009 17:58:05 +0200
Subject: [Cython] asking for review of my patch for #287
In-Reply-To: 
References: 
Message-ID: <4A099C8D.2000407@student.matnat.uio.no>

Lisandro Dalcin wrote:
> I want to raise your attention to ticket #287, as my patch could be
> controversial. I even changed my mind, and fixed my previous work on
> all this to prefer 'nb_int' over 'nb_long' in __Pyx_PyNumber_Int().
> This change was not a quick hack to get things solved, but rather a
> hard to take decision after diving core CPython sources from 2.3 to
> 2.7. IMHO, the 'nb_long' slot should have been deprecated long, long
> ago in core CPython. I would like this to be discussed a bit more
> before next release.

I don't think I'm qualified to comment on this. Do you consider it safe 
enough yourself for inclusion in Cython 0.11.2, or is it blocked pending 
review by Stefan or Robert?

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Tue May 12 18:04:28 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 12 May 2009 18:04:28 +0200
Subject: [Cython] #303 + simplifying type system
Message-ID: <4A099E0C.9000909@student.matnat.uio.no>

A)

Ticket #303 manifests itself like this:

cdef extern:
     ctypedef float footype # really double!

cdef class A:
     cdef public footype myproperty

My suggestion for a fix here is to replace all T_INT, T_UINT etc. with 
simply three types: __Pyx_T_SIGNED, __Pyx_T_UNSIGNED, __Pyx_T_FLOAT. 
These would use sizeof to determine the right type to flag the extension 
type property as (like Lisandro did with T_SIZET already does).

I can then change T_SIZET to __Pyx_T_UNSIGNED, right? (Basically that 
would be a rename...) Or it T_SIZET is defined in any Python version, 
I'll just throw it in on the list of possible "outputs" from the macro 
if it is defined.

B)

Given this change (which could happen in -devel), is there any obstacles 
to getting rid of the detailed type system in PyrexTypes? Basically 
Cython would only deal with "signed", "unsigned" and "float" (and in 
time "complex").

Doing

cdef short i

would basically flag i as "signed"; the "short-ness" would only be 
present as the cname of the type.


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Tue May 12 18:07:35 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 12 May 2009 18:07:35 +0200
Subject: [Cython] #303 + simplifying type system
In-Reply-To: <4A099E0C.9000909@student.matnat.uio.no>
References: <4A099E0C.9000909@student.matnat.uio.no>
Message-ID: <4A099EC7.8040109@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> A)
> 
> Ticket #303 manifests itself like this:
> 
> cdef extern:
>      ctypedef float footype # really double!
> 
> cdef class A:
>      cdef public footype myproperty
> 
> My suggestion for a fix here is to replace all T_INT, T_UINT etc. with 
> simply three types: __Pyx_T_SIGNED, __Pyx_T_UNSIGNED, __Pyx_T_FLOAT. 
> These would use sizeof to determine the right type to flag the extension 
> type property as (like Lisandro did with T_SIZET already does).
> 
> I can then change T_SIZET to __Pyx_T_UNSIGNED, right? (Basically that 
> would be a rename...) Or it T_SIZET is defined in any Python version, 
> I'll just throw it in on the list of possible "outputs" from the macro 
> if it is defined.
> 
> B)
> 
> Given this change (which could happen in -devel), is there any obstacles 
> to getting rid of the detailed type system in PyrexTypes? Basically 
> Cython would only deal with "signed", "unsigned" and "float" (and in 
> time "complex").
> 
> Doing
> 
> cdef short i
> 
> would basically flag i as "signed"; the "short-ness" would only be 
> present as the cname of the type.
> 

Note: I see B) happening in -unstable. (A) in -devel).


-- 
Dag Sverre

From dagss at student.matnat.uio.no  Tue May 12 18:19:17 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 12 May 2009 18:19:17 +0200
Subject: [Cython] Buffer passing implementation details
In-Reply-To: <4A033FE9.9060001@student.matnat.uio.no>
References: <4A01BB4A.1060003@student.matnat.uio.no>	
	<4A033FE9.9060001@student.matnat.uio.no>
Message-ID: <4A09A185.2020008@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Kurt Smith wrote:
>> Although, what about when we require a contiguous copy (or any mode,
>> for that matter) of a buffer?  Would we set the top-level object
>> reference to None, as suggested in a previous thread?  That would
>> prevent slicing at the Python API level, though, right?
> 
> Hmm. I'm still thinking about contiguous copies. Let's deal with that in 
> the next iteration.

OK I've been thinking about copy-making:

This really seems to be linked to in/out/inout specifiers as part of the 
function signature, more than anything else. There's basically two kind 
of functions:

a) The ones that pass in and out data. These can use in/out/inout 
specifiers and copying can happen automatically.

b) The ones that need to hold on to the buffer after returning 
("obj.set_data_store"). Copying here will cause trouble.

So I propose:

cdef void dataonly(inout int[::1] arr) # requires contiguous
cdef void keepsref(int[::1] arr)       # requires contiguous
cdef void keepsref_strided(int[:] arr)


int[:] discontiguous_buf = ...

dataonly(discontiguous_buf) # allowed, copying happen
keepsref(discontiguous_buf) # disallowed, compile-time error
keepsref_strided(discontiguous_buf) # allowed

keepsref(discontiguous_buf.copy()) # allowed

where I envision the copy method as being a way of manually creating a 
contiguous copy.

As for pure Python syntax for in/out/inout modifiers, I'm thinking 
something like this:

@cython.locals(arr=cython.inout & cython.int[:])
def foo(arr):
     ...

or in time

def foo(arr: cython.inout & cython.int[:]):
     ...

(The point here is: Relying on modifiers is not scary, we can use "&" in 
pure Python mode to get it across.)

-- 
Dag Sverre

From dagss at student.matnat.uio.no  Tue May 12 18:21:58 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 12 May 2009 18:21:58 +0200
Subject: [Cython] Buffer passing implementation details
In-Reply-To: <4A09A185.2020008@student.matnat.uio.no>
References: <4A01BB4A.1060003@student.matnat.uio.no>		<4A033FE9.9060001@student.matnat.uio.no>
	<4A09A185.2020008@student.matnat.uio.no>
Message-ID: <4A09A226.9040105@student.matnat.uio.no>

Dag Sverre Seljebotn wrote:
> Dag Sverre Seljebotn wrote:
>> Kurt Smith wrote:
>>> Although, what about when we require a contiguous copy (or any mode,
>>> for that matter) of a buffer?  Would we set the top-level object
>>> reference to None, as suggested in a previous thread?  That would
>>> prevent slicing at the Python API level, though, right?
>> Hmm. I'm still thinking about contiguous copies. Let's deal with that in 
>> the next iteration.
> 
> OK I've been thinking about copy-making:
> 
> This really seems to be linked to in/out/inout specifiers as part of the 
> function signature, more than anything else. There's basically two kind 
> of functions:
> 
> a) The ones that pass in and out data. These can use in/out/inout 
> specifiers and copying can happen automatically.
> 
> b) The ones that need to hold on to the buffer after returning 
> ("obj.set_data_store"). Copying here will cause trouble.
> 
> So I propose:
> 
> cdef void dataonly(inout int[::1] arr) # requires contiguous
> cdef void keepsref(int[::1] arr)       # requires contiguous
> cdef void keepsref_strided(int[:] arr)
> 
> 
> int[:] discontiguous_buf = ...
> 
> dataonly(discontiguous_buf) # allowed, copying happen
> keepsref(discontiguous_buf) # disallowed, compile-time error

Actually, let's keep this a runtime error -- as strided buffers can be 
contiguous (it needs to be runtime anyway for passing Python objects). 
It could hint to "use the .copy() method".

-- 
Dag Sverre

From stefan_ml at behnel.de  Tue May 12 08:45:19 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 12 May 2009 08:45:19 +0200
Subject: [Cython] Cython question
In-Reply-To: <89b937320905110100h1c068eb8ke31a01674ba2f6de@mail.gmail.com>
References: <89b937320905110100h1c068eb8ke31a01674ba2f6de@mail.gmail.com>
Message-ID: <4A091AFF.8070006@behnel.de>

Hi,

I forwarded your question to the Cython mailing list, which is the right
place to ask these things. Please subscribe to it if you are interested in
further discussion.

Jan Svec wrote:
> I have one question regarding the Cython project. Can I create a generic
> dynamically linked library in Cython? In other words I want to export my
> Python functions as DLL.

I'm not sure what you mean. Python's extension modules *are* shared
libraries, so a DLL is always what you get.

If with "generic" you mean: not only callable from Python, but exporting C
functions that other programs can call, then the answer is: I never tried.
It might work to link statically against libpython to create a DLL that
does not depend on the Python library. However, there is some
initialisation code involved that is called from the Python runtime. My
guess is that you'd at least have to add some hand-written code that
imports your module into an embedded Python interpreter so that it gets
properly set up. Some example code for embedding the Python interpreter is
here:

http://www.freenet.org.nz/python/embeddingpyrex/

Recent developer versions of Cython accept an "--embed" flag that will
generate an appropriate main() function (i.e. not exactly what you need,
but the code should be similar).

So I think the answer to your question is: probably yes, but not without a
bit of work on your side.

Stefan


From stefan_ml at behnel.de  Tue May 12 18:36:06 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 12 May 2009 18:36:06 +0200
Subject: [Cython] #define, or equivalent?
In-Reply-To: 
References: 
Message-ID: <4A09A576.4040604@behnel.de>


Brian Blais wrote:
> I have a number of arrays that need to be some maximum size, and I was
> wondering if there is some way to define that size at the top of the
> code, the equivalent of:
> 
> #define MAXSIZE 10
> 
> I haven't seen any examples of this.  So far I have just put the number
> in manually, like:
> 
> cdef a[10],b[10],c[10]
> 
> but I'd rather have:
> 
> cdef a[MAXSIZE],b[MAXSIZE],c[MAXSIZE]
> 
> is there a way of doing that?

You might be happy with the "DEF" statement (capital letters).

Stefan


From dalcinl at gmail.com  Tue May 12 19:58:13 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 12 May 2009 14:58:13 -0300
Subject: [Cython] asking for review of my patch for #287
In-Reply-To: <4A099C8D.2000407@student.matnat.uio.no>
References: 
	<4A099C8D.2000407@student.matnat.uio.no>
Message-ID: 

On Tue, May 12, 2009 at 12:58 PM, Dag Sverre Seljebotn
 wrote:
> Lisandro Dalcin wrote:
>> I want to raise your attention to ticket #287, as my patch could be
>> controversial. I even changed my mind, and fixed my previous work on
>> all this to prefer 'nb_int' over 'nb_long' in __Pyx_PyNumber_Int().
>> This change was not a quick hack to get things solved, but rather a
>> hard to take decision after diving core CPython sources from 2.3 to
>> 2.7. IMHO, the 'nb_long' slot should have been deprecated long, long
>> ago in core CPython. I would like this to be discussed a bit more
>> before next release.
>
> I don't think I'm qualified to comment on this. Do you consider it safe
> enough yourself for inclusion in Cython 0.11.2, or is it blocked pending
> review by Stefan or Robert?
>

IMHO, it is ready for 0.11.2, but I would like to know other opinions,
just in case... As I said, the patch could be controversial, specially
the change for Py2 in Pyx_PyNumber_Int(), where the "nb_int" slot is
preferred over "nb_long" slot.

If Robert/Stefan does not comment on this on a couple of days (and
provide an example showing the brokenness of my approach ;-) ), I
would just push the patch.



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From robertwb at math.washington.edu  Tue May 12 20:12:24 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 12 May 2009 11:12:24 -0700
Subject: [Cython] 0.11.2 release schedule?
In-Reply-To: <49FB54C6.4020009@student.matnat.uio.no>
References: <49FAF0CC.5070108@student.matnat.uio.no>	<49FB20B8.8090506@behnel.de>
	
	<49FB54C6.4020009@student.matnat.uio.no>
Message-ID: <1A7D9422-4220-4C7B-852B-42BE15F93F6F@math.washington.edu>

On May 1, 2009, at 1:00 PM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On May 1, 2009, at 9:18 AM, Stefan Behnel wrote:
>>
>>> Dag Sverre Seljebotn wrote:
>>>> Like I've been saying on some occasions, I'm going show off Cython
>>>> 18th
>>>> of May, and have had a tendency to scratch some itches rather than
>>>> planning how to explain them... so I'd really like a release to
>>>> happen
>>>> before then. (Though I also have some fixes/rewrites which are not
>>>> done
>>>> yet which will go in next week.)
>>>>
>>>> If it helps I can play release manager this time and do the
>>>> groundwork
>>>> for making it build Sage etc. (especially if that gets us complex
>>>> floats
>>>> in time :-)).
>>>>
>>>> Would it be possible to e.g. do a full stop feature freeze next
>>>> Friday
>>>> 8th, and then a release the week after?
>>> I'm not currently working on cython-devel anyway, so I'm glad to
>>> hear that
>>> you volunteer as QA manager. :)
>>
>> Yes, that would be great if you would do the release managing--
>> definitely would make a mid-May release more feasible.
>
> I'll give it a try.

Dag, any chance on a beta in the near future? Finishing up complex  
support is #2 on my todo list, so it'll get pushed today.

- Robert


From robertwb at math.washington.edu  Tue May 12 20:17:15 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 12 May 2009 11:17:15 -0700
Subject: [Cython] asking for review of my patch for #287
In-Reply-To: 
References: 
	<4A099C8D.2000407@student.matnat.uio.no>
	
Message-ID: <3620E7AB-10E1-4A90-A1FB-85E65C8E1758@math.washington.edu>

On May 12, 2009, at 10:58 AM, Lisandro Dalcin wrote:

> On Tue, May 12, 2009 at 12:58 PM, Dag Sverre Seljebotn
>  wrote:
>> Lisandro Dalcin wrote:
>>> I want to raise your attention to ticket #287, as my patch could be
>>> controversial. I even changed my mind, and fixed my previous work on
>>> all this to prefer 'nb_int' over 'nb_long' in __Pyx_PyNumber_Int().
>>> This change was not a quick hack to get things solved, but rather a
>>> hard to take decision after diving core CPython sources from 2.3 to
>>> 2.7. IMHO, the 'nb_long' slot should have been deprecated long, long
>>> ago in core CPython. I would like this to be discussed a bit more
>>> before next release.
>>
>> I don't think I'm qualified to comment on this. Do you consider it  
>> safe
>> enough yourself for inclusion in Cython 0.11.2, or is it blocked  
>> pending
>> review by Stefan or Robert?
>>
>
> IMHO, it is ready for 0.11.2, but I would like to know other opinions,
> just in case... As I said, the patch could be controversial, specially
> the change for Py2 in Pyx_PyNumber_Int(), where the "nb_int" slot is
> preferred over "nb_long" slot.
>
> If Robert/Stefan does not comment on this on a couple of days (and
> provide an example showing the brokenness of my approach ;-) ), I
> would just push the patch.

Sorry, most of my "dev" time has been focused on the immanent Sage  
4.0 release, which is why I haven't had time to comment on many of  
the things that have shown up here.

- Robert


From dagss at student.matnat.uio.no  Tue May 12 20:20:41 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 12 May 2009 20:20:41 +0200
Subject: [Cython] 0.11.2 release schedule?
In-Reply-To: <1A7D9422-4220-4C7B-852B-42BE15F93F6F@math.washington.edu>
References: <49FAF0CC.5070108@student.matnat.uio.no>	<49FB20B8.8090506@behnel.de>		<49FB54C6.4020009@student.matnat.uio.no>
	<1A7D9422-4220-4C7B-852B-42BE15F93F6F@math.washington.edu>
Message-ID: <4A09BDF9.8070109@student.matnat.uio.no>

Robert Bradshaw wrote:
> On May 1, 2009, at 1:00 PM, Dag Sverre Seljebotn wrote:
> 
>> Robert Bradshaw wrote:
>>> On May 1, 2009, at 9:18 AM, Stefan Behnel wrote:
>>>
>>>> Dag Sverre Seljebotn wrote:
>>>>> Like I've been saying on some occasions, I'm going show off Cython
>>>>> 18th
>>>>> of May, and have had a tendency to scratch some itches rather than
>>>>> planning how to explain them... so I'd really like a release to
>>>>> happen
>>>>> before then. (Though I also have some fixes/rewrites which are not
>>>>> done
>>>>> yet which will go in next week.)
>>>>>
>>>>> If it helps I can play release manager this time and do the
>>>>> groundwork
>>>>> for making it build Sage etc. (especially if that gets us complex
>>>>> floats
>>>>> in time :-)).
>>>>>
>>>>> Would it be possible to e.g. do a full stop feature freeze next
>>>>> Friday
>>>>> 8th, and then a release the week after?
>>>> I'm not currently working on cython-devel anyway, so I'm glad to
>>>> hear that
>>>> you volunteer as QA manager. :)
>>> Yes, that would be great if you would do the release managing--
>>> definitely would make a mid-May release more feasible.
>> I'll give it a try.
> 
> Dag, any chance on a beta in the near future? Finishing up complex  
> support is #2 on my todo list, so it'll get pushed today.

Yes, sorry, I got sidetracked for a few days (and left the tree in a 
defect state over the weekend as a bonus...). But I'm doing "make check" 
on Sage as I write this and things are looking good.

I was planning on doing it in a few hours, though if you're pushing 
complex support soon I might wait until after then.

(Any news on server access? I can find another location for it, no problem.)

-- 
Dag Sverre

From robertwb at math.washington.edu  Tue May 12 20:34:40 2009
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 12 May 2009 11:34:40 -0700
Subject: [Cython] 0.11.2 release schedule?
In-Reply-To: <4A09BDF9.8070109@student.matnat.uio.no>
References: <49FAF0CC.5070108@student.matnat.uio.no>	<49FB20B8.8090506@behnel.de>		<49FB54C6.4020009@student.matnat.uio.no>
	<1A7D9422-4220-4C7B-852B-42BE15F93F6F@math.washington.edu>
	<4A09BDF9.8070109@student.matnat.uio.no>
Message-ID: 

On May 12, 2009, at 11:20 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> On May 1, 2009, at 1:00 PM, Dag Sverre Seljebotn wrote:
>>
>>> Robert Bradshaw wrote:
>>>> On May 1, 2009, at 9:18 AM, Stefan Behnel wrote:
>>>>
>>>>> Dag Sverre Seljebotn wrote:
>>>>>> Like I've been saying on some occasions, I'm going show off  
>>>>>> Cython
>>>>>> 18th
>>>>>> of May, and have had a tendency to scratch some itches rather  
>>>>>> than
>>>>>> planning how to explain them... so I'd really like a release to
>>>>>> happen
>>>>>> before then. (Though I also have some fixes/rewrites which are  
>>>>>> not
>>>>>> done
>>>>>> yet which will go in next week.)
>>>>>>
>>>>>> If it helps I can play release manager this time and do the
>>>>>> groundwork
>>>>>> for making it build Sage etc. (especially if that gets us complex
>>>>>> floats
>>>>>> in time :-)).
>>>>>>
>>>>>> Would it be possible to e.g. do a full stop feature freeze next
>>>>>> Friday
>>>>>> 8th, and then a release the week after?
>>>>> I'm not currently working on cython-devel anyway, so I'm glad to
>>>>> hear that
>>>>> you volunteer as QA manager. :)
>>>> Yes, that would be great if you would do the release managing--
>>>> definitely would make a mid-May release more feasible.
>>> I'll give it a try.
>>
>> Dag, any chance on a beta in the near future? Finishing up complex
>> support is #2 on my todo list, so it'll get pushed today.
>
> Yes, sorry, I got sidetracked for a few days (and left the tree in a
> defect state over the weekend as a bonus...). But I'm doing "make  
> check"
> on Sage as I write this and things are looking good.

No problem. As much as we like Cython, it's not our full time job for  
any of us :).

> I was planning on doing it in a few hours, though if you're pushing
> complex support soon I might wait until after then.

Don't hold up for it--often little stuff falls out so we'll need  
another round for release anyways.

> (Any news on server access? I can find another location for it, no  
> problem.)


My plan is to make a "cython" account on the server and move  
everything over to there (giving both of us access)--just haven't  
done it yet. I'll be happy to push to there myself until then, you  
can just stick it in http://sage.math.washington.edu/home/dagss/.

- Robert


From stefan_ml at behnel.de  Tue May 12 20:52:07 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 12 May 2009 20:52:07 +0200
Subject: [Cython] 0.11.2 release schedule?
In-Reply-To: <4A09BDF9.8070109@student.matnat.uio.no>
References: <49FAF0CC.5070108@student.matnat.uio.no>	<49FB20B8.8090506@behnel.de>		<49FB54C6.4020009@student.matnat.uio.no>	<1A7D9422-4220-4C7B-852B-42BE15F93F6F@math.washington.edu>
	<4A09BDF9.8070109@student.matnat.uio.no>
Message-ID: <4A09C557.6050103@behnel.de>


Dag Sverre Seljebotn wrote:
> Robert Bradshaw wrote:
>> Dag, any chance on a beta in the near future? Finishing up complex  
>> support is #2 on my todo list, so it'll get pushed today.
> 
> Yes, sorry, I got sidetracked for a few days (and left the tree in a 
> defect state over the weekend as a bonus...). But I'm doing "make check" 
> on Sage as I write this and things are looking good.
> 
> I was planning on doing it in a few hours, though if you're pushing 
> complex support soon I might wait until after then.

Just a quick note that lxml builds and tests nicely with cython-devel on
Py2.5 and 3.1. So a beta is ok over here.

Stefan


From dagss at student.matnat.uio.no  Tue May 12 21:53:23 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 12 May 2009 21:53:23 +0200
Subject: [Cython] 0.11.2 release schedule?
In-Reply-To: 
References: <49FAF0CC.5070108@student.matnat.uio.no>	<49FB20B8.8090506@behnel.de>		<49FB54C6.4020009@student.matnat.uio.no>	<1A7D9422-4220-4C7B-852B-42BE15F93F6F@math.washington.edu>	<4A09BDF9.8070109@student.matnat.uio.no>
	
Message-ID: <4A09D3B3.5090902@student.matnat.uio.no>

Robert Bradshaw wrote:
> On May 12, 2009, at 11:20 AM, Dag Sverre Seljebotn wrote:
> 
>> Robert Bradshaw wrote:
>>> On May 1, 2009, at 1:00 PM, Dag Sverre Seljebotn wrote:
>>>
>>>> Robert Bradshaw wrote:
>>>>> On May 1, 2009, at 9:18 AM, Stefan Behnel wrote:
>>>>>
>>>>>> Dag Sverre Seljebotn wrote:
>>>>>>> Like I've been saying on some occasions, I'm going show off  
>>>>>>> Cython
>>>>>>> 18th
>>>>>>> of May, and have had a tendency to scratch some itches rather  
>>>>>>> than
>>>>>>> planning how to explain them... so I'd really like a release to
>>>>>>> happen
>>>>>>> before then. (Though I also have some fixes/rewrites which are  
>>>>>>> not
>>>>>>> done
>>>>>>> yet which will go in next week.)
>>>>>>>
>>>>>>> If it helps I can play release manager this time and do the
>>>>>>> groundwork
>>>>>>> for making it build Sage etc. (especially if that gets us complex
>>>>>>> floats
>>>>>>> in time :-)).
>>>>>>>
>>>>>>> Would it be possible to e.g. do a full stop feature freeze next
>>>>>>> Friday
>>>>>>> 8th, and then a release the week after?
>>>>>> I'm not currently working on cython-devel anyway, so I'm glad to
>>>>>> hear that
>>>>>> you volunteer as QA manager. :)
>>>>> Yes, that would be great if you would do the release managing--
>>>>> definitely would make a mid-May release more feasible.
>>>> I'll give it a try.
>>> Dag, any chance on a beta in the near future? Finishing up complex
>>> support is #2 on my todo list, so it'll get pushed today.
>> Yes, sorry, I got sidetracked for a few days (and left the tree in a
>> defect state over the weekend as a bonus...). But I'm doing "make  
>> check"
>> on Sage as I write this and things are looking good.

Status on Sage is that these fail:

         sage -t  "devel/sage/sage/libs/ntl/ntl_ZZ_pX.pyx"
         sage -t  "devel/sage/sage/rings/integer.pyx"
         sage -t  "devel/sage/sage/plot/plot.py"
         sage -t  "devel/sage/sage/interfaces/maxima.py"

The last two seem to be because of fragile doctests. The first two are 
because of changed Cython semantics:

     OverflowError: can't convert negative value to unsigned int

In the cases it is about going from "long" via "object" to "unsigned 
long" (with, confirmed, a negative integer), so I'd say the mistake is 
in Sage here.

Question is again whether this is too much semantics change for a minor 
release, but I'm +1.

-- 
Dag Sverre

From michael.abshoff at googlemail.com  Tue May 12 22:04:33 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Tue, 12 May 2009 13:04:33 -0700
Subject: [Cython] 0.11.2 release schedule?
In-Reply-To: <4A09D3B3.5090902@student.matnat.uio.no>
References: <49FAF0CC.5070108@student.matnat.uio.no>	<49FB20B8.8090506@behnel.de>		<49FB54C6.4020009@student.matnat.uio.no>	<1A7D9422-4220-4C7B-852B-42BE15F93F6F@math.washington.edu>	<4A09BDF9.8070109@student.matnat.uio.no>	
	<4A09D3B3.5090902@student.matnat.uio.no>
Message-ID: <4A09D651.7090309@gmail.com>

Dag Sverre Seljebotn wrote:
> Robert Bradshaw wrote:



Hi Dag,

> Status on Sage is that these fail:
> 
>          sage -t  "devel/sage/sage/libs/ntl/ntl_ZZ_pX.pyx"
>          sage -t  "devel/sage/sage/rings/integer.pyx"
>          sage -t  "devel/sage/sage/plot/plot.py"
>          sage -t  "devel/sage/sage/interfaces/maxima.py"
> 
> The last two seem to be because of fragile doctests. 

Can you post the first doctest failure from each failing file?

Cheers,

Michael

From dagss at student.matnat.uio.no  Tue May 12 22:21:06 2009
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 12 May 2009 22:21:06 +0200
Subject: [Cython] 0.11.2 release schedule?
In-Reply-To: <4A09D651.7090309@gmail.com>
References: <49FAF0CC.5070108@student.matnat.uio.no>	<49FB20B8.8090506@behnel.de>		<49FB54C6.4020009@student.matnat.uio.no>	<1A7D9422-4220-4C7B-852B-42BE15F93F6F@math.washington.edu>	<4A09BDF9.8070109@student.matnat.uio.no>		<4A09D3B3.5090902@student.matnat.uio.no>
	<4A09D651.7090309@gmail.com>
Message-ID: <4A09DA32.2040607@student.matnat.uio.no>

Michael Abshoff wrote:
> Dag Sverre Seljebotn wrote:
>> Robert Bradshaw wrote:
> 
> 
> 
> Hi Dag,
> 
>> Status on Sage is that these fail:
>>
>>          sage -t  "devel/sage/sage/libs/ntl/ntl_ZZ_pX.pyx"
>>          sage -t  "devel/sage/sage/rings/integer.pyx"
>>          sage -t  "devel/sage/sage/plot/plot.py"
>>          sage -t  "devel/sage/sage/interfaces/maxima.py"
>>
>> The last two seem to be because of fragile doctests. 
> 
> Can you post the first doctest failure from each failing file?

This is on Sage 3.4.1 (my own build).

sage -t  "devel/sage/sage/libs/ntl/ntl_ZZ_pX.pyx"
**********************************************************************
File 
"/home/dagss/sage/sage-3.4.1/devel/sage/sage/libs/ntl/ntl_ZZ_pX.pyx", 
line 751:
     sage: f.leading_coefficient()
Exception raised:
     Traceback (most recent call last):
       File "/home/dagss/sage/sage-3.4.1/local/bin/ncadoctest.py", line 
1231, in run_one_test
         self.run_one_example(test, example, filename, compileflags)
       File "/home/dagss/sage/sage-3.4.1/local/bin/sagedoctest.py", line 
38, in run_one_example
         OrigDocTestRunner.run_one_example(self, test, example, 
filename, compileflags)
       File "/home/dagss/sage/sage-3.4.1/local/bin/ncadoctest.py", line 
1172, in run_one_example
         compileflags, 1) in test.globs
       File "", line 1, in 
         f.leading_coefficient()###line 751:
     sage: f.leading_coefficient()
       File "ntl_ZZ_pX.pyx", line 758, in 
sage.libs.ntl.ntl_ZZ_pX.ntl_ZZ_pX.leading_coefficient 
(sage/libs/ntl/ntl_ZZ_pX.cpp:6208)
       File "ntl_ZZ_pX.pyx", line 255, in 
sage.libs.ntl.ntl_ZZ_pX.ntl_ZZ_pX.__getitem__ 
(sage/libs/ntl/ntl_ZZ_pX.cpp:3853)
     OverflowError: can't convert negative value to unsigned int
**********************************************************************


sage -t  "devel/sage/sage/rings/integer.pyx"
**********************************************************************
File "/home/dagss/sage/sage-3.4.1/devel/sage/sage/rings/integer.pyx", 
line 4388:
     sage: w.test_bit(-1)
Exception raised:
     Traceback (most recent call last):
       File "/home/dagss/sage/sage-3.4.1/local/bin/ncadoctest.py", line 
1231, in run_one_test
         self.run_one_example(test, example, filename, compileflags)
       File "/home/dagss/sage/sage-3.4.1/local/bin/sagedoctest.py", line 
38, in run_one_example
         OrigDocTestRunner.run_one_example(self, test, example, 
filename, compileflags)
       File "/home/dagss/sage/sage-3.4.1/local/bin/ncadoctest.py", line 
1172, in run_one_example
         compileflags, 1) in test.globs
       File "", line 1, in 
         w.test_bit(-Integer(1))###line 4388:
     sage: w.test_bit(-1)
       File "integer.pyx", line 4392, in 
sage.rings.integer.Integer.test_bit (sage/rings/integer.c:25745)
     OverflowError: can't convert negative value to unsigned long

sage -t  "devel/sage/sage/plot/plot.py"
**********************************************************************
File "/home/dagss/sage/sage-3.4.1/devel/sage/sage/plot/plot.py", line 206:
     sage: savefig('sage.png')
Exception raised:
     Traceback (most recent call last):
       File "/home/dagss/sage/sage-3.4.1/local/bin/ncadoctest.py", line 
1231, in run_one_test
         self.run_one_example(test, example, filename, compileflags)
       File "/home/dagss/sage/sage-3.4.1/local/bin/sagedoctest.py", line 
38, in run_one_example
         OrigDocTestRunner.run_one_example(self, test, example, 
filename, compileflags)
       File "/home/dagss/sage/sage-3.4.1/local/bin/ncadoctest.py", line 
1172, in run_one_example
         compileflags, 1) in test.globs
       File "", line 1, in 
         savefig('sage.png')###line 206:
     sage: savefig('sage.png')
       File 
"/home/dagss/sage/sage-3.4.1/local/lib/python2.5/site-packages/matplotlib/pyplot.py", 
line 346, in savefig
         return fig.savefig(*args, **kwargs)
       File 
"/home/dagss/sage/sage-3.4.1/local/lib/python2.5/site-packages/matplotlib/figure.py", 
line 1001, in savefig
         self.canvas.print_figure(*args, **kwargs)
       File 
"/home/dagss/sage/sage-3.4.1/local/lib/python2.5/site-packages/matplotlib/backend_bases.py", 
line 1411, in print_figure
         '%s.' % (format, ', '.join(formats)))
     ValueError: Format "png" is not supported.
     Supported formats: eps, ps.
**********************************************************************

sage -t  "devel/sage/sage/interfaces/maxima.py"
**********************************************************************
File "/home/dagss/sage/sage-3.4.1/devel/sage/sage/interfaces/maxima.py", 
line 822:
     sage: maxima._command_runner('describe', 'gcd')
Expected:
     -- Function: gcd (, , , ...)
     ...
Got:
     ;; Loading file /home/dagss/.clisprc ...
     ;; Loaded file /home/dagss/.clisprc
     
      -- Function: gcd (, , , ...)
          Returns the greatest common divisor of  and .  The flag
          `gcd' determines which algorithm is employed.  Setting `gcd' to
          `ez', `subres', `red', or `spmod' selects the `ezgcd',
          subresultant `prs', reduced, or modular algorithm, respectively.
          If `gcd' `false' then `gcd (, , )' always returns 1
          for all .  Many functions (e.g.  `ratsimp', `factor', etc.)
          cause gcd's to be taken implicitly.  For homogeneous polynomials
          it is recommended that `gcd' equal to `subres' be used.  To take
          the gcd when an algebraic is present, e.g., `gcd (^2 -
          2*sqrt(2)* + 2,  - sqrt(2))', `algebraic' must be `true' and
          `gcd' must not be `ez'.
     
          The `gcd' flag, default: `spmod', if `false' will also prevent the
          greatest common divisor from being taken when expressions are
          converted to canonical rational expression (CRE) form.  This will
          sometimes speed the calculation if gcds are not required.
     
     
       There are also some inexact matches for `gcd'.
       Try `?? gcd' to see them.
     
                                          true
     
**********************************************************************


-- 
Dag Sverre

From cswiercz at gmail.com  Tue May 12 22:19:53 2009
From: cswiercz at gmail.com (Chris Swierczewski)
Date: Tue, 12 May 2009 13:19:53 -0700
Subject: [Cython] Converting Bound Methods to Unbound Methods
Message-ID: 

Hello,

I have a cdef'ed class with a cdef'ed method that looks something like this:

cdef class MyClass:
   cdef void myfunc(self, double *x):
       ... (function contents) ...

   def execute():
       call_func( &(self.myfunc) )

I'm linking to a function in an external library that accepts a
function pointer of similar signature

cdef extern from 'somelibrary.h':
   cdef void call_func(void (*func)(double *x))

However, I receive an error at compile time that a function of type

void (*)(MyClass, double *)

cannot be converted to a function of type

void (*)(double *)

I was wondering there is a way to unbind this myfunc class method so
that it can be properly sent to this external function. I know that I
can simply create a function pointer attribute as part of the class
but it would seem cleaner for a user to subclass and override myfunc
instead of subclassing and setting the value of function pointer
attribute from within an overridden __cdef__ method.

Thank you in advance for any hints or suggestions!

-- 
Chris Swierczewski
cswiercz at gmail.com
blog: cswiercz.wordpress.com

From stefan_ml at behnel.de  Tue May 12 22:24:41 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 12 May 2009 22:24:41 +0200
Subject: [Cython] Converting Bound Methods to Unbound Methods
In-Reply-To: 
References: 
Message-ID: <4A09DB09.1020400@behnel.de>


Chris Swierczewski wrote:
> I have a cdef'ed class with a cdef'ed method that looks something like this:
> 
> cdef class MyClass:
>    cdef void myfunc(self, double *x):
>        ... (function contents) ...
> 
>    def execute():
>        call_func( &(self.myfunc) )
> 
> I'm linking to a function in an external library that accepts a
> function pointer of similar signature
> 
> cdef extern from 'somelibrary.h':
>    cdef void call_func(void (*func)(double *x))
> 
> However, I receive an error at compile time that a function of type
> 
> void (*)(MyClass, double *)
> 
> cannot be converted to a function of type
> 
> void (*)(double *)
> 
> I was wondering there is a way to unbind this myfunc class method so
> that it can be properly sent to this external function.

No you can't, the signatures are incompatible at the C level.


> I know that I
> can simply create a function pointer attribute as part of the class
> but it would seem cleaner for a user to subclass and override myfunc
> instead of subclassing and setting the value of function pointer
> attribute from within an overridden __cdef__ method.

The question is: if the method makes sense without the class, why is it a
method in the first place? Maybe you should make it a function and just
call it from the class method?

Stefan

From dalcinl at gmail.com  Tue May 12 22:29:05 2009
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 12 May 2009 17:29:05 -0300
Subject: [Cython] about _USE_MATH_DEFINES
Message-ID: 

In this changeset

changeset:   1024:ce19a1f80694
user:        JimKleckner
date:        Fri Aug 15 01:56:03 2008 -0700
summary:     Windos __stdcall and _USE_MATH_DEFINES

Jim added this line (and other #define's for compatibility of calling
conventions on non-Win32)

code.putln("  #define _USE_MATH_DEFINES")

but it seems (looking at the full change) that this #define will not
be active for Windows 64.

Is this corrent? Should this be fixed for the Win64 case?


-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From michael.abshoff at googlemail.com  Tue May 12 22:30:53 2009
From: michael.abshoff at googlemail.com (Michael Abshoff)
Date: Tue, 12 May 2009 13:30:53 -0700
Subject: [Cython] 0.11.2 release schedule?
In-Reply-To: <4A09DA32.2040607@student.matnat.uio.no>
References: <49FAF0CC.5070108@student.matnat.uio.no>	<49FB20B8.8090506@behnel.de>		<49FB54C6.4020009@student.matnat.uio.no>	<1A7D9422-4220-4C7B-852B-42BE15F93F6F@math.washington.edu>	<4A09BDF9.8070109@student.matnat.uio.no>		<4A09D3B3.5090902@student.matnat.uio.no>	<4A09D651.7090309@gmail.com>
	<4A09DA32.2040607@student.matnat.uio.no>
Message-ID: <4A09DC7D.9010307@gmail.com>

Dag Sverre Seljebotn wrote:
> Michael Abshoff wrote:
>> Dag Sverre Seljebotn wrote:
>>> Robert Bradshaw wrote:
>> 
>>
>> Hi Dag,
>>
>>> Status on Sage is that these fail:
>>>
>>>          sage -t  "devel/sage/sage/libs/ntl/ntl_ZZ_pX.pyx"
>>>          sage -t  "devel/sage/sage/rings/integer.pyx"
>>>          sage -t  "devel/sage/s