From vel.accel at gmail.com Thu Oct 1 01:42:13 2009 From: vel.accel at gmail.com (Peter Alexander) Date: Wed, 30 Sep 2009 19:42:13 -0400 Subject: [Cython] Updating Documentation In-Reply-To: <4AC39AF8.9010906@student.matnat.uio.no> References: <1e52e0880909300141l146ec811k40a90a439ee73a3a@mail.gmail.com> <4AC32A6D.7080000@student.matnat.uio.no> <4AC39271.3080300@noaa.gov> <4AC39AF8.9010906@student.matnat.uio.no> Message-ID: <1e52e0880909301642l5d398060w5ef391141ccc1fda@mail.gmail.com> On Wed, Sep 30, 2009 at 1:52 PM, Dag Sverre Seljebotn wrote: > Christopher Barker wrote: >> Dag Sverre Seljebotn wrote: >>> ?- Numerical computations in Cython (a copy of the corresponding paper) >> >>> 4) I think the current Cython/NumPy tutorial should be removed in favour >>> of the above generic tutorial + numerical Cython paper; opinions on that? >> >> Well, one good intro to numpy+cython is better than two that both need >> to be maintained. > > Thanks for your feedback. My thoughts: > > Yes, but the existing intro on numpy+cython: > a) Lacks quite a bit in several areas > b) Is outdated > > While the tutorial + paper is already written now. > >> However, maybe it's my personal use-case bias, but I think that numpy is >> almost a standard part of Cython use -- I can't think of what I'd like >> to optimize that I wouldn't want to use numpy for! And numpy is why I >> skipped past pyrex and straight to Cython. > > Well, take Stefan Behnel, who doesn't use NumPy at all, but rather works > with XML documents. > > Even within science, the whole Sage project (a very important Cython > user) don't use that much NumPy either, and in particular not the > Cython/NumPy features, as they need exact math which isn't covered by > NumPy's dtypes (and also tend to operate on thousands rather than > billions of numbers I think?). > > Then you have any kind of wrappers around native libraries. > > There's enough usecases for Cython without NumPy to make our primary > Cython tutorial generic and not targeted for numerical users IMO. > > And then you have the question of whether one should have *two* > tutorials. Perhaps a very small intro document for numerical users which > links heavily into the other documents... > > > -- > Dag Sverre > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > 1) Minh, I have a now repo[1] at bitbucket where I will push to often. 2) I agree that Numpy should be a seperate TOC entry, because it deserves special, detailed attention. 3) After some thought and review, the current abstract structure of the docs are good. 4) We need to: a) incorporate all the possible new information, b) add lots of (cross)-references c) and lots of integrated (flowing) examples. 4) Refactor and minimize verbosity. 4) Consider the possiblity of a "Quick Ref" section which are essentially an index of "HowTos" for fast look-up (Its nice to immediately put your fingers on something your looking for, without having to have to read half a page of paragraphs). These could simply be an index of links in to the main doc sections.. maybe.. 5) Consider how to incorporate what Python syntax Cython currently optimizes and what it does not. We need to elliminate this frustrating enigma for un-informed users. Also, whats on the near term agenda for new, further optimizations of syntax magic, etc.. [1] http://bitbucket.org/travlr/cython-docs From vel.accel at gmail.com Thu Oct 1 01:57:45 2009 From: vel.accel at gmail.com (Peter Alexander) Date: Wed, 30 Sep 2009 19:57:45 -0400 Subject: [Cython] Updating Documentation In-Reply-To: <1e52e0880909301642l5d398060w5ef391141ccc1fda@mail.gmail.com> References: <1e52e0880909300141l146ec811k40a90a439ee73a3a@mail.gmail.com> <4AC32A6D.7080000@student.matnat.uio.no> <4AC39271.3080300@noaa.gov> <4AC39AF8.9010906@student.matnat.uio.no> <1e52e0880909301642l5d398060w5ef391141ccc1fda@mail.gmail.com> Message-ID: <1e52e0880909301657xd1632c9rd12771d9adf65f0f@mail.gmail.com> On Wed, Sep 30, 2009 at 7:42 PM, Peter Alexander wrote: > On Wed, Sep 30, 2009 at 1:52 PM, Dag Sverre Seljebotn > wrote: >> Christopher Barker wrote: >>> Dag Sverre Seljebotn wrote: >>>> ?- Numerical computations in Cython (a copy of the corresponding paper) >>> >>>> 4) I think the current Cython/NumPy tutorial should be removed in favour >>>> of the above generic tutorial + numerical Cython paper; opinions on that? >>> >>> Well, one good intro to numpy+cython is better than two that both need >>> to be maintained. >> >> Thanks for your feedback. My thoughts: >> >> Yes, but the existing intro on numpy+cython: >> a) Lacks quite a bit in several areas >> b) Is outdated >> >> While the tutorial + paper is already written now. >> >>> However, maybe it's my personal use-case bias, but I think that numpy is >>> almost a standard part of Cython use -- I can't think of what I'd like >>> to optimize that I wouldn't want to use numpy for! And numpy is why I >>> skipped past pyrex and straight to Cython. >> >> Well, take Stefan Behnel, who doesn't use NumPy at all, but rather works >> with XML documents. >> >> Even within science, the whole Sage project (a very important Cython >> user) don't use that much NumPy either, and in particular not the >> Cython/NumPy features, as they need exact math which isn't covered by >> NumPy's dtypes (and also tend to operate on thousands rather than >> billions of numbers I think?). >> >> Then you have any kind of wrappers around native libraries. >> >> There's enough usecases for Cython without NumPy to make our primary >> Cython tutorial generic and not targeted for numerical users IMO. >> >> And then you have the question of whether one should have *two* >> tutorials. Perhaps a very small intro document for numerical users which >> links heavily into the other documents... >> >> >> -- >> Dag Sverre >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > 1) Minh, I have a now repo[1] at bitbucket where I will push to often. > 2) I agree that Numpy should be a seperate TOC entry, because it > deserves special, detailed attention. > 3) After some thought and review, the current abstract structure of > the docs are good. > 4) We need to: a) incorporate all the possible new information, b) add > lots of (cross)-references c) and lots of integrated (flowing) > examples. 4) Refactor and minimize verbosity. 4) Consider the > possiblity of a "Quick Ref" section which are essentially an index of > "HowTos" for fast look-up (Its nice to immediately put your fingers on > something your looking for, without having to have to read half a page > of paragraphs). These could simply be an index of links in to the main > doc sections.. maybe.. > 5) Consider how to incorporate what Python syntax Cython currently > optimizes and what it does not. We need to elliminate this frustrating > enigma for un-informed users. Also, whats on the near term agenda for > new, further optimizations of syntax magic, etc.. > > [1] http://bitbucket.org/travlr/cython-docs > [...] 6) Provide simple (up-front) information to users about bug reporting/querying, enhancement proposals, utilizing the wiki as documentation appendage for users.. From Chris.Barker at noaa.gov Thu Oct 1 02:07:36 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 30 Sep 2009 17:07:36 -0700 Subject: [Cython] Updating Documentation In-Reply-To: <4AC39AF8.9010906@student.matnat.uio.no> References: <1e52e0880909300141l146ec811k40a90a439ee73a3a@mail.gmail.com> <4AC32A6D.7080000@student.matnat.uio.no> <4AC39271.3080300@noaa.gov> <4AC39AF8.9010906@student.matnat.uio.no> Message-ID: <4AC3F2C8.70006@noaa.gov> Dag Sverre Seljebotn wrote: > Christopher Barker wrote: > Yes, but the existing intro on numpy+cython: > a) Lacks quite a bit in several areas > b) Is outdated > > While the tutorial + paper is already written now. right -- my pint was that we really should have only one -- if the tutorial+paper is the better/more up-to-date one, then by all means let's make that the one people find. >> However, maybe it's my personal use-case bias, but I think that numpy is >> almost a standard part of Cython use -- I can't think of what I'd like >> to optimize that I wouldn't want to use numpy for! And numpy is why I >> skipped past pyrex and straight to Cython. > > Well, take Stefan Behnel, who doesn't use NumPy at all, but rather works > with XML documents. > > Even within science, the whole Sage project (a very important Cython > user) don't use that much NumPy either, and in particular not the > Cython/NumPy features, as they need exact math which isn't covered by > NumPy's dtypes OK -- lot's of non-numpy use. > Then you have any kind of wrappers around native libraries. yes, but if those native libraries are working with arrays of data, numpy may well be a good idea there, too. > There's enough usecases for Cython without NumPy to make our primary > Cython tutorial generic and not targeted for numerical users IMO. agreed, though I may never use the stuff used to process XML docs, but it gets first class treatment... > And then you have the question of whether one should have *two* > tutorials. Perhaps a very small intro document for numerical users which > links heavily into the other documents... Sure -- in fact, it often makes lots of sense for there to be different tutorial focused on different user groups -- either because they have different use-cases, or because they have different backgrounds. I can tell you that I went straight to the numpy+Cython tutorial, because I knew I'd want to work primarily with numpy arrays. We may some day want tutorials for: - how to use Cython to wrap C libs - how to use Cython to wrap C++ libs (maybe targeted for folks that are used to using SWIG, ctypes, etc.) Anyway. I love to see this documentation effort! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From bpederse at gmail.com Thu Oct 1 02:07:58 2009 From: bpederse at gmail.com (Brent Pedersen) Date: Wed, 30 Sep 2009 17:07:58 -0700 Subject: [Cython] Appending... In-Reply-To: <4AC3C575.5010607@noaa.gov> References: <4AC38C4B.5040903@student.matnat.uio.no> <4AC3B001.3060508@behnel.de> <4AC3C575.5010607@noaa.gov> Message-ID: On Wed, Sep 30, 2009 at 1:54 PM, Christopher Barker wrote: > > > Brent Pedersen wrote: >>>> timing on my machine, that gives: >>>> >>>> vector append 1.19197511673 >>>> python append 6.57795596123 >>>> vector get 0.080246925354 >>>> python get 2.01052689552 > > While you're at it, why not give an array.array implementation a try -- > it might be closer to the C++ version -- it should be! > > -Chris looks like it could actually be faster. http://gist.github.com/198576 gives: method: append/get python: 3.5361 / 1.8621 vector: 1.8887 / 0.5865 array : 0.3319 / 0.1467 think that's a reasonably fair test except i cheated for array.array on the inserts by doing a resize() and then inserting, not appending. otherwise, it's _very_ slow to do the array.append() -- over 30 seconds instead of .33 for array.array, i used the stuff attached to this ticket: http://trac.cython.org/cython_trac/ticket/314 > > > > > > -- > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R ? ? ? ? ? ?(206) 526-6959 ? voice > 7600 Sand Point Way NE ? (206) 526-6329 ? fax > Seattle, WA ?98115 ? ? ? (206) 526-6317 ? main reception > > Chris.Barker at noaa.gov > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > From Chris.Barker at noaa.gov Thu Oct 1 03:12:36 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Wed, 30 Sep 2009 18:12:36 -0700 Subject: [Cython] Appending... In-Reply-To: References: <4AC38C4B.5040903@student.matnat.uio.no> <4AC3B001.3060508@behnel.de> <4AC3C575.5010607@noaa.gov> Message-ID: <4AC40204.9030808@noaa.gov> Brent Pedersen wrote: > http://gist.github.com/198576 > > gives: > > method: append/get > python: 3.5361 / 1.8621 > vector: 1.8887 / 0.5865 > array : 0.3319 / 0.1467 > > think that's a reasonably fair test except I cheated for array.array > on the inserts by doing a resize() and then inserting, not appending. Actually, that is a total cheat, and not testing the right thing at all. (IIUC what teh OP was looking for). You've pre-allocated, so of course accumulating is going to be blazingly fast. I think the point of this was for a situation where you don't know how big it's going to get, so you can't pre-allocate. > otherwise, it's _very_ slow to do the array.append() -- over 30 > seconds instead of .33 wow! that is really, really bad. Something has got to be wrong. I've been poking into the array.array over-allocations scheme (thanks Robert), and it is more conservative than lists, so it may be slower, but that seems like too much. > for array.array, i used the stuff attached to this ticket: > http://trac.cython.org/cython_trac/ticket/314 That should make it speedy indeed. Have you looked at the generated C? cdef array.array array_append(N): cdef int i pyaa = array.array('i') array.array_resize(pyaa, N) cdef array.array aa = pyaa for i in range(N): aa._i[i] = i return aa I haven't poked into the code in that ticket, but something is odd. Maybe you need to tell Cython that aa is not only an array.array, but also what type it is? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From robert.kern at gmail.com Thu Oct 1 04:57:26 2009 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 30 Sep 2009 21:57:26 -0500 Subject: [Cython] Updating Documentation In-Reply-To: <4AC3F2C8.70006@noaa.gov> References: <1e52e0880909300141l146ec811k40a90a439ee73a3a@mail.gmail.com> <4AC32A6D.7080000@student.matnat.uio.no> <4AC39271.3080300@noaa.gov> <4AC39AF8.9010906@student.matnat.uio.no> <4AC3F2C8.70006@noaa.gov> Message-ID: Christopher Barker wrote: > Dag Sverre Seljebotn wrote: >> There's enough usecases for Cython without NumPy to make our primary >> Cython tutorial generic and not targeted for numerical users IMO. > > agreed, though I may never use the stuff used to process XML docs, but > it gets first class treatment... There really isn't any special "stuff used to process XML docs" in Cython. There is special stuff in Cython to interact with numpy arrays. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From honnibal at gmail.com Thu Oct 1 07:40:59 2009 From: honnibal at gmail.com (Matthew Honnibal) Date: Thu, 1 Oct 2009 15:40:59 +1000 Subject: [Cython] Question about buffer interface for Python2.6 Message-ID: Hi, I've only just started using Cython today, and I'm having trouble with the buffer interface indexing described here: http://wiki.cython.org/enhancements/buffer . I want to iterate over a unicode string getting contiguous subsequences. My Cython function has the following declaration: def parseDoc(object[object] string): I've tried various Python types, but I always get a runtime error like: TypeError: 'unicode' does not have the buffer interface TypeError: 'array.array' does not have the buffer interface TypeError: 'buffer' does not have the buffer interface The last one was obtained by calling the built-in buffer() function on the string. I can get a step further by doing cimport numpy and using a numpy array, but then unicode is not one of the supported types, and that doesn't seem like it was the intention --- the docs suggest at Python 2.6+, I shouldn't need the numpy work-around. What am I doing wrong? Matthew. P.S. I tried to post this yesterday but it didn't seem to come up. Apologies if this double-posts. From robertwb at math.washington.edu Thu Oct 1 08:53:19 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 30 Sep 2009 23:53:19 -0700 Subject: [Cython] Updating Documentation In-Reply-To: <4AC338AE.1010404@student.matnat.uio.no> References: <1e52e0880909300141l146ec811k40a90a439ee73a3a@mail.gmail.com> <1e52e0880909300253s44949a0al5d987c03c2ba503d@mail.gmail.com> <4AC32BB5.7050604@student.matnat.uio.no> <1e52e0880909300306u1940255ara37e321bc8634b88@mail.gmail.com> <1e52e0880909300327p63bc1fccjf78b58f643454085@mail.gmail.com> <4AC338AE.1010404@student.matnat.uio.no> Message-ID: On Sep 30, 2009, at 3:53 AM, Dag Sverre Seljebotn wrote: > Peter Alexander wrote: >> One thing I'd like to have an idea of atm is if you'd like the entire >> site to be more homogenous. A nice template/theme could run through >> out the website. Maybe using Sphinx's index.html as the portal page >> and then linking to tutorial, manual, howto's, wiki, trac, etc.. from >> there. I not much of a web-site developer but I'm sure I could handle >> a template and css well enough. >> > As far as *system* goes, I personally think the current system is > great. > It's more about *content*. > > Brain dump: > > I like the current system of: > > - cython.org: A *short*, single, static HTML page. It should be as > short > as possible, only providing the essential links ("what is Cython", > docs, > download, mailing list, authors & financial contributions). Everything > else is either relevant to users and should go into docs, or community > stuff/dynamic lists which should go into wiki. > > - docs.cython.org: Served by Sphinx (which can then also build static > Windows help files, documentation we can ship etc. etc). Sphinx can > contain tutorials, howtos etc. in addition to reference manual. > http://docs.python.org is great; many parts, all of which are > searchable > together immedeately. (I like that better than docs.scipy.org). > > - wiki.cython.org: We need a wiki for day-to-day work, but it > should be > used as a place for documentation to a lesser degree than today. > > - trac.cython.org: Perhaps get the logo back up but I don't see any > need > to integrate it nicely into anything else myself. > > Most users will only see docs.cython.org and so layout *will* be > uniform > without us thinking about it. The documentation on the wiki should IMO > be removed from there and replaced with links to docs.cython.org (once > the contents is moved). The Cython Enhancement Proposals should > have the > relevant link to docs.cython.org prominently placed on top of them, > but > stay in the wiki. > > The way I see it, the wiki and trac are ideally (but not today) mostly > for developers and active mailing list participants, and so a nice > experience is less important. Everything that we want people to read > should find their way into http://docs.cython.org. +1 to all of the above. We should be more aggressive about making tickets to moving content on the wiki to actual documentation when things get stable. > Part of the problem is that it was never documented how to change > docs.cython.org (I can change the hg repo but how to publish the > changes?), and so to get documentation for e.g. NumPy stuff out I > needed > to put it on the wiki (Gabriel did a single pull of the tutorial but > it's outdated now). > > BTW the SciPy project can edit their Sphinx documentation through a > web > interface (but I don't think that's the end to start for us; a simple > automatic generation when a push is made to hg.cython.org/cython-docs > will get us far). Currently, there's an .hg repo in docs.cython.org that I pull from and build right there. I'm not sure how to set up the hooks to fire on push (it can probably be done) or even if it's a good idea, but a single script one can run to refresh all docs would be very useful. http://trac.cython.org/cython_trac/ticket/383 - Robert From robertwb at math.washington.edu Thu Oct 1 09:07:02 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 1 Oct 2009 00:07:02 -0700 Subject: [Cython] Updating Documentation In-Reply-To: <4AC3F2C8.70006@noaa.gov> References: <1e52e0880909300141l146ec811k40a90a439ee73a3a@mail.gmail.com> <4AC32A6D.7080000@student.matnat.uio.no> <4AC39271.3080300@noaa.gov> <4AC39AF8.9010906@student.matnat.uio.no> <4AC3F2C8.70006@noaa.gov> Message-ID: <71236E99-6956-4143-BA53-678A8D66B21E@math.washington.edu> On Sep 30, 2009, at 5:07 PM, Christopher Barker wrote: > Dag Sverre Seljebotn wrote: >> Christopher Barker wrote: >> Yes, but the existing intro on numpy+cython: >> a) Lacks quite a bit in several areas >> b) Is outdated >> >> While the tutorial + paper is already written now. > > right -- my pint was that we really should have only one -- if the > tutorial+paper is the better/more up-to-date one, then by all means > let's make that the one people find. > > >>> However, maybe it's my personal use-case bias, but I think that >>> numpy is >>> almost a standard part of Cython use -- I can't think of what I'd >>> like >>> to optimize that I wouldn't want to use numpy for! And numpy is >>> why I >>> skipped past pyrex and straight to Cython. >> >> Well, take Stefan Behnel, who doesn't use NumPy at all, but rather >> works >> with XML documents. >> >> Even within science, the whole Sage project (a very important Cython >> user) don't use that much NumPy either, and in particular not the >> Cython/NumPy features, as they need exact math which isn't covered by >> NumPy's dtypes > > OK -- lot's of non-numpy use. Yep. Sage ships SciPy, and we defer lots of the numerical stuff to that. We could probably use NumPy more than we do, but for most stuff fixed-size fixed-precision arrays SIMD operations just doesn't describe what we do. >> Then you have any kind of wrappers around native libraries. > > yes, but if those native libraries are working with arrays of data, > numpy may well be a good idea there, too. I would say that most (non-scientific) libraries don't work with arrays of data. >> There's enough usecases for Cython without NumPy to make our primary >> Cython tutorial generic and not targeted for numerical users IMO. > > agreed, though I may never use the stuff used to process XML docs, but > it gets first class treatment... I think the main point here is string processing, which is much more broadly applicable than parsing XML. >> And then you have the question of whether one should have *two* >> tutorials. Perhaps a very small intro document for numerical users >> which >> links heavily into the other documents... > > Sure -- in fact, it often makes lots of sense for there to be > different > tutorial focused on different user groups -- either because they have > different use-cases, or because they have different backgrounds. > > I can tell you that I went straight to the numpy+Cython tutorial, > because I knew I'd want to work primarily with numpy arrays. > > We may some day want tutorials for: > - how to use Cython to wrap C libs > - how to use Cython to wrap C++ libs > (maybe targeted for folks that are used to using SWIG, ctypes, > etc.) Maybe something like Intro - Brief overview of what Cython is (and is not) - Building (very simple example, setup.py) -- note that this can be skipped if using the sage notebook. (Eventually it might be good to have a link right to the sage notebook for people to try all the examples out--the setup can be a hurdle, and this gives people a chance to see how easy and powerful it is to have motivation to actually set it up on their machines). - Basics cdef variables, methods(?), and types. cython -a All of the above is essential to any use of Cython. We could then dive into topic-specific tutorials, specifically: - NumPy -- this would be the SciPy paper, minus intro stuff - Calling and wrapping external libraries - Creating extension classes - String processing - Advanced topics (for...from, include, .pxd files) - Common pitfalls - Pure mode - ??? All of the above would assume only the background of the (shortish) intro chapter(s), and be relatively self-contained. There should also be a reference. This would contain things like specific optimizations, directives, language details, and would be heavily linked to from the above. - Robert From robertwb at math.washington.edu Thu Oct 1 09:09:07 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 1 Oct 2009 00:09:07 -0700 Subject: [Cython] Updating Documentation In-Reply-To: References: <1e52e0880909300141l146ec811k40a90a439ee73a3a@mail.gmail.com> <1e52e0880909300253s44949a0al5d987c03c2ba503d@mail.gmail.com> <4AC32BB5.7050604@student.matnat.uio.no> <1e52e0880909300306u1940255ara37e321bc8634b88@mail.gmail.com> <1e52e0880909300327p63bc1fccjf78b58f643454085@mail.gmail.com> <4AC338AE.1010404@student.matnat.uio.no> Message-ID: BTW, who wants/needs push access to hg.cython.org/cython-docs? (Though you can set up your own repos as well.) - Robert From robertwb at math.washington.edu Thu Oct 1 09:18:13 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 1 Oct 2009 00:18:13 -0700 Subject: [Cython] Appending... In-Reply-To: <4AC39696.2070605@noaa.gov> References: <4AC38C4B.5040903@student.matnat.uio.no> <4AC39696.2070605@noaa.gov> Message-ID: <0763C7DF-8AFE-4E7E-B494-6CFE5DA6E0EA@math.washington.edu> On Sep 30, 2009, at 10:34 AM, Christopher Barker wrote: > Magnus Lie Hetland wrote: >> Yeah, one-dimensional data -- simply a list of ints. >> >> I'll just go for a Python list for now, then, and look into it if >> performance becomes a problem. (It's not going to be a long list, but >> it may be traversed hundreds of thousands of times, so who >> knows... :->) > > Funny you should write this. I was just composing a note in my head > about a proposal for a numpy "accumulating array". This idea was > inspired by a discussion at SciPY, in which we spent a LOT of time > during the numpy tutorial talking about how to accumulate values in an > array when you don't know how big the array needs to be when you > start. > > The "standard practice" is to accumulate in a python list, then > convert > the final result into an array. This is a good idea because Python > lists > are standard, well tested, efficient, etc. > > However, as was pointed out in that lengthy discussion, if what you > are > doing is accumulating is a whole bunch of numbers (ints, floats, > whatever), or particularly if you need to accumulate a data type that > plain python doesn't support, there is a lot of overhead involved: a > python float type is pretty heavyweight. If performance or memory > use is > important, it might create issues. > > What I have in mind is very simple. It would be: > - Only 1-d > - Support append() and extend() methods > - Support any valid numpy dtype > - which could even get you pseudo n-d arrays... > - maybe it would act like an array in other ways, I'm not so sure.. > > > It could be written in pure python, using np.resize()(or concatenate, > or,...), and certainly written efficiently in Cython or even (no!) C. Sounds useful. One thing that would be missing from a pure Python implementation is the ability to append and get raw c types, rather than just their wrapped counterparts, which would probably be the bulk of the overhead. (In Cython one would have to write a separate class for each type, oh if only we have templates already...) - Robert From dagss at student.matnat.uio.no Thu Oct 1 09:21:28 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 01 Oct 2009 09:21:28 +0200 Subject: [Cython] Updating Documentation In-Reply-To: <71236E99-6956-4143-BA53-678A8D66B21E@math.washington.edu> References: <1e52e0880909300141l146ec811k40a90a439ee73a3a@mail.gmail.com> <4AC32A6D.7080000@student.matnat.uio.no> <4AC39271.3080300@noaa.gov> <4AC39AF8.9010906@student.matnat.uio.no> <4AC3F2C8.70006@noaa.gov> <71236E99-6956-4143-BA53-678A8D66B21E@math.washington.edu> Message-ID: <4AC45878.6020107@student.matnat.uio.no> Robert Bradshaw wrote: > On Sep 30, 2009, at 5:07 PM, Christopher Barker wrote: > > >> Dag Sverre Seljebotn wrote: >> >>> Christopher Barker wrote: >>> Yes, but the existing intro on numpy+cython: >>> a) Lacks quite a bit in several areas >>> b) Is outdated >>> >>> While the tutorial + paper is already written now. >>> >> right -- my pint was that we really should have only one -- if the >> tutorial+paper is the better/more up-to-date one, then by all means >> let's make that the one people find. >> >> >> >>>> However, maybe it's my personal use-case bias, but I think that >>>> numpy is >>>> almost a standard part of Cython use -- I can't think of what I'd >>>> like >>>> to optimize that I wouldn't want to use numpy for! And numpy is >>>> why I >>>> skipped past pyrex and straight to Cython. >>>> >>> Well, take Stefan Behnel, who doesn't use NumPy at all, but rather >>> works >>> with XML documents. >>> >>> Even within science, the whole Sage project (a very important Cython >>> user) don't use that much NumPy either, and in particular not the >>> Cython/NumPy features, as they need exact math which isn't covered by >>> NumPy's dtypes >>> >> OK -- lot's of non-numpy use. >> > > Yep. Sage ships SciPy, and we defer lots of the numerical stuff to > that. We could probably use NumPy more than we do, but for most stuff > fixed-size fixed-precision arrays SIMD operations just doesn't > describe what we do. > > >>> Then you have any kind of wrappers around native libraries. >>> >> yes, but if those native libraries are working with arrays of data, >> numpy may well be a good idea there, too. >> > > I would say that most (non-scientific) libraries don't work with > arrays of data. > > >>> There's enough usecases for Cython without NumPy to make our primary >>> Cython tutorial generic and not targeted for numerical users IMO. >>> >> agreed, though I may never use the stuff used to process XML docs, but >> it gets first class treatment... >> > > I think the main point here is string processing, which is much more > broadly applicable than parsing XML. > > >>> And then you have the question of whether one should have *two* >>> tutorials. Perhaps a very small intro document for numerical users >>> which >>> links heavily into the other documents... >>> >> Sure -- in fact, it often makes lots of sense for there to be >> different >> tutorial focused on different user groups -- either because they have >> different use-cases, or because they have different backgrounds. >> >> I can tell you that I went straight to the numpy+Cython tutorial, >> because I knew I'd want to work primarily with numpy arrays. >> >> We may some day want tutorials for: >> - how to use Cython to wrap C libs >> - how to use Cython to wrap C++ libs >> (maybe targeted for folks that are used to using SWIG, ctypes, >> etc.) >> > > Maybe something like > > Intro > - Brief overview of what Cython is (and is not) > - Building (very simple example, setup.py) -- note that this can be > skipped if using the sage notebook. (Eventually it might be good to > have a link right to the sage notebook for people to try all the > examples out--the setup can be a hurdle, and this gives people a > chance to see how easy and powerful it is to have motivation to > actually set it up on their machines). > - Basics cdef variables, methods(?), and types. cython -a > > All of the above is essential to any use of Cython. We could then > dive into topic-specific tutorials, specifically: > > - NumPy -- this would be the SciPy paper, minus intro stuff > - Calling and wrapping external libraries > - Creating extension classes > - String processing > - Advanced topics (for...from, include, .pxd files) > - Common pitfalls > - Pure mode > - ??? > That's a great idea. Basically just putting the last half of our SciPy tutorial into seperate chapters and link to them (so that they don't have to be read in a particular order) gets us very far. I was thinking about this when writing the tutorial -- which section to put first of: a) NumPy b) string processing c) wrapping external lib? As they don't build on one another. Dag Sverre From vel.accel at gmail.com Thu Oct 1 09:21:47 2009 From: vel.accel at gmail.com (Peter Alexander) Date: Thu, 1 Oct 2009 03:21:47 -0400 Subject: [Cython] Updating Documentation In-Reply-To: References: <1e52e0880909300141l146ec811k40a90a439ee73a3a@mail.gmail.com> <1e52e0880909300253s44949a0al5d987c03c2ba503d@mail.gmail.com> <4AC32BB5.7050604@student.matnat.uio.no> <1e52e0880909300306u1940255ara37e321bc8634b88@mail.gmail.com> <1e52e0880909300327p63bc1fccjf78b58f643454085@mail.gmail.com> <4AC338AE.1010404@student.matnat.uio.no>

Message-ID: <1e52e0880910010021w1295bf4gdfba2ff0555768b0@mail.gmail.com> On Thu, Oct 1, 2009 at 3:09 AM, Robert Bradshaw wrote: > BTW, who wants/needs push access to hg.cython.org/cython-docs? > (Though you can set up your own repos as well.) > > - Robert > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > Hi Robert. I have a branch repo at http://bitbucket.org/travlr/cython-docs . That should suffice for now, I'd think. Unless you all want it differently. At this point I've redone the super-structure and am iterating inward. Conciseness is my goal. Let me know, at any time, your thoughts or vision do not coincide with what I do.. From robertwb at math.washington.edu Thu Oct 1 09:28:23 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 1 Oct 2009 00:28:23 -0700 Subject: [Cython] Buffer interface in Python2.6 In-Reply-To: References: Message-ID: On Sep 30, 2009, at 10:08 AM, Matthew Honnibal wrote: > Hi, > I've only just started using Cython today, and I'm having trouble with > the buffer interface indexing described here: > http://wiki.cython.org/enhancements/buffer . I want to iterate over a > unicode string getting contiguous subsequences. I'm not sure what the intended behavior is, given that unicode objects can be stored with various encodings, and, in particular, the default (UTF-8) is a variable-length encoding. > My Cython function has the following declaration: > > def parseDoc(object[object] string): If you really want a 1D list of objects, I'd just use a list. > I've tried various Python types, but I always get a runtime error > like: > > TypeError: 'unicode' does not have the buffer interface > TypeError: 'array.array' does not have the buffer interface > TypeError: 'buffer' does not have the buffer interface > > The last one was obtained by calling the built-in buffer() function on > the string. > > I can get a step further by doing cimport numpy and using a numpy > array, but then unicode is not one of the supported types, and that > doesn't seem like it was the intention --- the docs suggest at Python > 2.6+, I shouldn't need the numpy work-around. > > What am I doing wrong? I'm not sure. I can't even get def foo(object[double] a): return a[0] foo(numpy.ndarrray(5)) to work. Dag, any ideas? Is this supposed to work for Py < 3.0? - Robert From dagss at student.matnat.uio.no Thu Oct 1 09:32:08 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 01 Oct 2009 09:32:08 +0200 Subject: [Cython] Buffer interface in Python2.6 In-Reply-To: References:

Message-ID: <4AC45AF8.8050807@student.matnat.uio.no> Robert Bradshaw wrote: > On Sep 30, 2009, at 10:08 AM, Matthew Honnibal wrote: > > >> Hi, >> I've only just started using Cython today, and I'm having trouble with >> the buffer interface indexing described here: >> http://wiki.cython.org/enhancements/buffer . I want to iterate over a >> unicode string getting contiguous subsequences. >> > > I'm not sure what the intended behavior is, given that unicode > objects can be stored with various encodings, and, in particular, the > default (UTF-8) is a variable-length encoding. > > >> My Cython function has the following declaration: >> >> def parseDoc(object[object] string): >> > > If you really want a 1D list of objects, I'd just use a list. > > >> I've tried various Python types, but I always get a runtime error >> like: >> >> TypeError: 'unicode' does not have the buffer interface >> TypeError: 'array.array' does not have the buffer interface >> TypeError: 'buffer' does not have the buffer interface >> >> The last one was obtained by calling the built-in buffer() function on >> the string. >> >> I can get a step further by doing cimport numpy and using a numpy >> array, but then unicode is not one of the supported types, and that >> doesn't seem like it was the intention --- the docs suggest at Python >> 2.6+, I shouldn't need the numpy work-around. >> >> What am I doing wrong? >> > > I'm not sure. > > I can't even get > > def foo(object[double] a): > return a[0] > > foo(numpy.ndarrray(5)) > > to work. Dag, any ideas? Is this supposed to work for Py < 3.0? > Try foo(numpy.array([5]).astype(np.double)) ... I'll answer the rest seperately. Dag Sverre From dagss at student.matnat.uio.no Thu Oct 1 09:45:06 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 01 Oct 2009 09:45:06 +0200 Subject: [Cython] Question about buffer interface for Python2.6 In-Reply-To: References: Message-ID: <4AC45E02.6070408@student.matnat.uio.no> Matthew Honnibal wrote: > Hi, > I've only just started using Cython today, and I'm having trouble with > the buffer interface indexing described here: > http://wiki.cython.org/enhancements/buffer . I want to iterate over a > unicode string getting contiguous subsequences. > The buffer PEP is *available* in Python 2.6, however I don't think objects in the Python standard library exports its buffers using it. Unfortunately. What you can try to do is use the backwards-compatability mechanisms of implementing __getbuffer__ in Cython, something like (untested): from python_unicode cimport Py_UNICODE cdef extern from "Python.h": # Or Python's unicodeobject.h ctypedef class unicode [object PyUnicodeObject]: Py_ssize_t length Py_UNICODE *str def __getbuffer__(self, Py_buffer* buf, int flags): ... fill in buf struct with PEP 3118 information to export self.str/self.length ... Notes: a) If you only want to deal with unicodes, you can probably just as well drop __getbuffer__. With the declaration above, you can still do cdef unicode u = myunicode cdef Py_UNICODE *buf = u.str print buf[3] # gets 4th unicode character without any buffer support. b) If you do write up a decent unicode declaration, make sure to contribute it to Cython's Cython/Includes/python_unicode. c) If you go the __getbuffer__ route for more convenient syntax, be aware that unicode types are not supported; you need to export it as "=I", "=H", "=B" (int, short, byte) depending on sizeof(Py_UNICODE), see struct module, and then acquire a buffer through cdef unicode[Py_UNICODE] u = myunicode cdef Py_UNICODE onechar = u[3] Dag Sverre From stefan_ml at behnel.de Thu Oct 1 10:27:40 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 01 Oct 2009 10:27:40 +0200 Subject: [Cython] Buffer interface in Python2.6 In-Reply-To: References:

Message-ID: <4AC467FC.5000002@behnel.de> Robert Bradshaw wrote: > On Sep 30, 2009, at 10:08 AM, Matthew Honnibal wrote: >> I've only just started using Cython today, and I'm having trouble with >> the buffer interface indexing described here: >> http://wiki.cython.org/enhancements/buffer . I want to iterate over a >> unicode string getting contiguous subsequences. > > I'm not sure what the intended behavior is, given that unicode > objects can be stored with various encodings, and, in particular, the > default (UTF-8) is a variable-length encoding. UTF-8 is not used here. However, the real encoding of the underlying byte buffer is platform specific and one of UCS-4 or UCS-2 as little endian or big endian, i.e. four possible encodings. I never tried, but I'd expect the buffer view of unicode objects to point to that internal buffer. Anyway, the fastest way to get to the data content of a unicode object are the C-API functions PyUnicode_AS_DATA() and PyUnicode_GET_DATA_SIZE(), or if you prefer character chunked data instead of byte data, PyUnicode_AS_UNICODE() and PyUnicode_GET_SIZE(). Also see the definition of the Py_UNICODE character type, which can be a 2-byte or 4-byte integer type, or a wchar_t type, depending on the platform. http://docs.python.org/c-api/unicode.html#Py_UNICODE Stefan From robertwb at math.washington.edu Thu Oct 1 11:22:43 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 1 Oct 2009 02:22:43 -0700 Subject: [Cython] Cython in language shootout? In-Reply-To: <85e81ba30909301439t9aa5e5ex2d426077d8fd61@mail.gmail.com> References: <5B3508FD-AE97-4904-91B1-C74EB449E276@hetland.org> <85e81ba30909301143t7d746dc7o577376dca9490205@mail.gmail.com> <85e81ba30909301439t9aa5e5ex2d426077d8fd61@mail.gmail.com> Message-ID: <6F871507-E4FA-49D3-A146-E3991269BEB8@math.washington.edu> On Sep 30, 2009, at 2:39 PM, William Stein wrote: > On Wed, Sep 30, 2009 at 2:04 PM, Lisandro Dalcin > wrote: >> On Wed, Sep 30, 2009 at 3:43 PM, William Stein >> wrote: >>> On Wed, Sep 30, 2009 at 5:05 AM, Magnus Lie Hetland >>> wrote: >>>> One of my PhD students suggested adding Cython to the Computer >>>> Language Benchmark Game: >>>> >>>> http://shootout.alioth.debian.org/ >>>> >>>> Seems like it would be fun -- after all, there are already Python >>>> implementations there (for 11 problems), so as a start, one could >>>> simply wrap up a proper compilation script for Cython, and >>>> submit it. >>>> Fiddling with declarations etc. would be a natural next step, of >>>> course. >>>> >>>> I'm thinking about doing something here myself, but I'm a bit busy >>>> with another project at the moment, so I thought I'd mention it >>>> here, >>>> in case someone else felt inspired. >>>> >>>> I think this is a shootout many people refer to when discussing >>>> relative performance of languages; getting Cython in there might >>>> peak >>>> the interest of people who might actually need it, but who don't >>>> know >>>> it exists... >>>> >>>> (Sorry for cross-posting -- I guess this might be relevant for both >>>> lists.) >>>> >>> >>> I've always wanted somebody to add Cython to that page, since I >>> think >>> Cython should do as well as C in every benchmark... which would >>> be fun >>> because C wins every benchmark. Yes, we should do that for sure. The --embed mode makes things much easier. I also think we should have two implementations--a Pythonic one (perhaps with types, but still natural), and a maximum performance (essentially C). >> Though IMHO Cython should be tested in two "modes": plain compilation >> of py sources, and cdef-typed implementations... >> > > I just did the pidigits one in about 5-10 minutes... and Robert > Bradshaw just walked into my office and said he's already done some of > them (they are sitting on his laptop somewhere). Now they're up at http://hg.cython.org/shootout/file/tip Please feel free to expand/improve. > http://sage.math.washington.edu/home/wstein/tmp/pidigits.pyx > > The timing for that one is basically the same as C. Robert remarks > that in some cases Python's "print/rawinput" is a serious bottleneck > compared to C if we want to stick with idiomatic Cython. Yep, and several of them are IO intensive. We can use printf and friends if we need. - Robert From dagss at student.matnat.uio.no Thu Oct 1 13:54:25 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 01 Oct 2009 13:54:25 +0200 Subject: [Cython] doctesthack directive Message-ID: <4AC49871.9070400@student.matnat.uio.no> I now have code sitting on my drive for a doctesthack directive. What it does is work around the limitations of the doctest module by automatically creating a global __test__ dictionary mapping descriptive names to function and method docstrings. 1) Can I push it? 2) Can I make it the default? It never does anything if a __test__ already exists, so it should be fully backwards compatible. The only downside I can see is a little bit of extra memory used. Line numbers are embedded into the test names (the keys in __dict__). Note: This does not solve the problem of doctest being unable to print line numbers for class docstrings. Also the test procedure ends up flowing a little bit different from the Python module (as each function is a seperate testcase, and not simply part of the module test). As the name says, it is a hack. Dag Sverre From stefan_ml at behnel.de Thu Oct 1 13:58:58 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 01 Oct 2009 13:58:58 +0200 Subject: [Cython] doctesthack directive In-Reply-To: <4AC49871.9070400@student.matnat.uio.no> References: <4AC49871.9070400@student.matnat.uio.no> Message-ID: <4AC49982.1080806@behnel.de> Dag Sverre Seljebotn wrote: > I now have code sitting on my drive for a doctesthack directive. What it > does is work around the limitations of the doctest module by > automatically creating a global __test__ dictionary mapping descriptive > names to function and method docstrings. > > 1) Can I push it? > 2) Can I make it the default? It never does anything if a __test__ > already exists, so it should be fully backwards compatible. The only > downside I can see is a little bit of extra memory used. > > Line numbers are embedded into the test names (the keys in __dict__). > > Note: This does not solve the problem of doctest being unable to print > line numbers for class docstrings. Also the test procedure ends up > flowing a little bit different from the Python module (as each function > is a seperate testcase, and not simply part of the module test). As the > name says, it is a hack. Now that I know what kind of problems this does *not* fix and what new problems it introduces, what about some more detail what kind of problem this actually fixes? Stefan From dagss at student.matnat.uio.no Thu Oct 1 14:06:14 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 01 Oct 2009 14:06:14 +0200 Subject: [Cython] doctesthack directive In-Reply-To: <4AC49982.1080806@behnel.de> References: <4AC49871.9070400@student.matnat.uio.no> <4AC49982.1080806@behnel.de> Message-ID: <4AC49B36.4070607@student.matnat.uio.no> Stefan Behnel wrote: > Dag Sverre Seljebotn wrote: > >> I now have code sitting on my drive for a doctesthack directive. What it >> does is work around the limitations of the doctest module by >> automatically creating a global __test__ dictionary mapping descriptive >> names to function and method docstrings. >> >> 1) Can I push it? >> 2) Can I make it the default? It never does anything if a __test__ >> already exists, so it should be fully backwards compatible. The only >> downside I can see is a little bit of extra memory used. >> >> Line numbers are embedded into the test names (the keys in __dict__). >> >> Note: This does not solve the problem of doctest being unable to print >> line numbers for class docstrings. Also the test procedure ends up >> flowing a little bit different from the Python module (as each function >> is a seperate testcase, and not simply part of the module test). As the >> name says, it is a hack. >> > > Now that I know what kind of problems this does *not* fix and what new > problems it introduces, what about some more detail what kind of problem > this actually fixes? > What it fixes is that I can take a simple pyx file like this: def func(x): """ >>> func(3) """ ... and then test it like I would a Python module: python -c 'import doctest; import mymod; doctest.testmod(mymod)' doctest does not pick up the function docstring automatically as it reports negative to inspect.isfunction, however putting those functions in the __test__ dict works around it. Arguments in favor: - Lowers the barrier. It is the kind of thing which took me by surprise when I first met it; letting this be the default could save half an hour for every new Cython user. - This also allows very easily writing function-wise doctests in the Cython test suite An approach which requires a custom script for doctesting Cython files is here: http://wiki.cython.org/FAQ#HowcanIrundoctestsinCythoncode.28pyxfiles.29.3F But I happen to like this approach better as it lower the barriers and makes peoples test scripts simpler without Cython-specific logic. Dag Sverre From ndbecker2 at gmail.com Thu Oct 1 16:17:27 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Thu, 01 Oct 2009 10:17:27 -0400 Subject: [Cython] generic types? Message-ID: I need to code some cython functions which will be instantiated for several different data types. Very much like what c++ templates are used for. What approaches can be used to do this? (I'm guessing some kind of macro text processing. Does cython use cpp?) Now suppose in general I need a little more intelligence. For example, suppose I have a function that is templated on a single datatype. Suppose some types internal to that function depend in a (possibly nontrivial way) on that datatype. Same approach? How about a function templated on multiple datatypes? From Chris.Barker at noaa.gov Thu Oct 1 18:22:29 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 01 Oct 2009 09:22:29 -0700 Subject: [Cython] Updating Documentation In-Reply-To: References: <1e52e0880909300141l146ec811k40a90a439ee73a3a@mail.gmail.com> <4AC32A6D.7080000@student.matnat.uio.no> <4AC39271.3080300@noaa.gov> <4AC39AF8.9010906@student.matnat.uio.no> <4AC3F2C8.70006@noaa.gov> Message-ID: <4AC4D745.4020301@noaa.gov> Robert Kern wrote: > Christopher Barker wrote: >> agreed, though I may never use the stuff used to process XML docs, but >> it gets first class treatment... > > There really isn't any special "stuff used to process XML docs" in Cython. There > is special stuff in Cython to interact with numpy arrays. I never used the term "special". My point was that for many users, working with numpy arrays may be just as or more important that python strings (or even lists, etc.) As I see it, the only thing "special" about numpy arrays is that they are not part of the standard library (which I think they should be) -- but they are part of the standard Cython distro. Robert Bradshaw wrote: >>> Then you have any kind of wrappers around native libraries. >> yes, but if those native libraries are working with arrays of data, >> numpy may well be a good idea there, too. > > I would say that most (non-scientific) libraries don't work with > arrays of data. It might be interesting to do a survey, but I can, off the top of my head, think of a couple broad categories that I've worked with: - text-oriented : not numpy - image processing: numpy could/should play a role here - scientific/numerical: duh! - GUI: numpy could/should play a role here, too. mainly for graphics, etc. - GIS: numpy does/could/should play a role here, too. As an example, if pyOpenGL didn't work with numpy arrays, it would be crippled -- or it would have to provide its own data types for storing/manipulating big blocks of data. Any C library is going to work with one or both of two things: text and numbers -- what fraction of libs only work with text? Of course, what I'm really doing here is advocating for an nd-array in the standard lib... > Maybe something like > > Intro > - Brief overview of what Cython is (and is not) > - Building (very simple example, setup.py) -- note that this can be > skipped if using the sage notebook. (Eventually it might be good to > have a link right to the sage notebook for people to try all the > examples out--the setup can be a hurdle, and this gives people a > chance to see how easy and powerful it is to have motivation to > actually set it up on their machines). Sage is wonderful, but I'd not like to see it get too tied into Cython... > - Basics cdef variables, methods(?), and types. cython -a > > All of the above is essential to any use of Cython. We could then > dive into topic-specific tutorials, specifically: > > - NumPy -- this would be the SciPy paper, minus intro stuff > - Calling and wrapping external libraries > - Creating extension classes > - String processing > - Advanced topics (for...from, include, .pxd files) > - Common pitfalls > - Pure mode > - ??? > > All of the above would assume only the background of the (shortish) > intro chapter(s), and be relatively self-contained. Yes, this is a great outline! I think it accomplished what I was getting at -- a short path from newbie to working with numpy arrays, but also an equally short path to working with strings, etc. A new user is likely to come to Cython with a particular problem to solve, and this looks like a good structure to get them there quickly. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From Chris.Barker at noaa.gov Thu Oct 1 18:31:00 2009 From: Chris.Barker at noaa.gov (Christopher Barker) Date: Thu, 01 Oct 2009 09:31:00 -0700 Subject: [Cython] Updating Documentation In-Reply-To: <4AC45878.6020107@student.matnat.uio.no> References: <1e52e0880909300141l146ec811k40a90a439ee73a3a@mail.gmail.com> <4AC32A6D.7080000@student.matnat.uio.no> <4AC39271.3080300@noaa.gov> <4AC39AF8.9010906@student.matnat.uio.no> <4AC3F2C8.70006@noaa.gov> <71236E99-6956-4143-BA53-678A8D66B21E@math.washington.edu> <4AC45878.6020107@student.matnat.uio.no> Message-ID: <4AC4D944.7040805@noaa.gov> Dag Sverre Seljebotn wrote: > which section to put first of: > > a) NumPy > b) string processing > c) wrapping external lib? > > As they don't build on one another. Then it doesn't much matter, but I can't imagine how you could wrap an external lib without using numpy and/or string processing, so maybe that should go last. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov From robertwb at math.washington.edu Thu Oct 1 20:00:08 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 1 Oct 2009 11:00:08 -0700 Subject: [Cython] Updating Documentation In-Reply-To: <4AC4D745.4020301@noaa.gov> References: <1e52e0880909300141l146ec811k40a90a439ee73a3a@mail.gmail.com> <4AC32A6D.7080000@student.matnat.uio.no> <4AC39271.3080300@noaa.gov> <4AC39AF8.9010906@student.matnat.uio.no> <4AC3F2C8.70006@noaa.gov> <4AC4D745.4020301@noaa.gov> Message-ID: <0741A875-6A6E-452D-8B9D-B54FF6BE207E@math.washington.edu> On Oct 1, 2009, at 9:22 AM, Christopher Barker wrote: > Robert Bradshaw wrote: >>>> Then you have any kind of wrappers around native libraries. >>> yes, but if those native libraries are working with arrays of data, >>> numpy may well be a good idea there, too. >> >> I would say that most (non-scientific) libraries don't work with >> arrays of data. > > It might be interesting to do a survey, but I can, off the top of my > head, think of a couple broad categories that I've worked with: > - text-oriented : not numpy > - image processing: numpy could/should play a role here > - scientific/numerical: duh! > - GUI: numpy could/should play a role here, too. mainly for > graphics, > etc. > - GIS: numpy does/could/should play a role here, too. > > As an example, if pyOpenGL didn't work with numpy arrays, it would be > crippled -- or it would have to provide its own data types for > storing/manipulating big blocks of data. > > Any C library is going to work with one or both of two things: text > and > numbers -- what fraction of libs only work with text? Just for fun I did a quick survey of the libraries we access from Cython in Sage (which is probably not typical, but at least is a large set I can say something about): - Neither string processing nor obvious application of NumPy arrays: cliquer, cvxopt, ecl (soon?), ecm, elliptic_curves, flint, gaplib (soon?), givaro, gmp/mpir, iml, lcalc, fplll, libgcrypt, linbox, m4ri, mpfi, mpfr, ntl, palp, pari, polybori, polytopes, pynac (ginac), quaddouble, qs, ratpoints, singular, symmetrica, sympow, zn_poly. - Could potentially use NumPy arrays: ghmm, gd, gsl, lapak. - Non-trivial string processing: Nothing we wrap with Cython (that I know of off the top of my head). Of course there's stdlib, stdio, math, etc. as well. Often we wrap things as (semi-)opaque objects, call methods on them, etc. That being said, I think the scientific/numerical audience is (currently at least) one of the largest audiences for Cython, and it's important to cater to them. > Of course, what I'm really doing here is advocating for an nd-array in > the standard lib... Yeah. The buffer interface is a nice step in that direction though. >> Maybe something like >> >> Intro >> - Brief overview of what Cython is (and is not) >> - Building (very simple example, setup.py) -- note that this can be >> skipped if using the sage notebook. (Eventually it might be good to >> have a link right to the sage notebook for people to try all the >> examples out--the setup can be a hurdle, and this gives people a >> chance to see how easy and powerful it is to have motivation to >> actually set it up on their machines). > > Sage is wonderful, but I'd not like to see it get too tied into > Cython... Well, at least we don't call it SageX anymore ;). I'm thinking more of a "try it now" kind of link. - Robert > From robertwb at math.washington.edu Thu Oct 1 20:09:08 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 1 Oct 2009 11:09:08 -0700 Subject: [Cython] generic types? In-Reply-To: References: Message-ID: <5E1EE29A-FBD0-435B-9D86-A0D7A7846684@math.washington.edu> On Oct 1, 2009, at 7:17 AM, Neal Becker wrote: > I need to code some cython functions which will be instantiated for > several > different data types. Very much like what c++ templates are used for. > > What approaches can be used to do this? > > (I'm guessing some kind of macro text processing. Does cython use > cpp?) > > Now suppose in general I need a little more intelligence. For > example, > suppose I have a function that is templated on a single datatype. > Suppose > some types internal to that function depend in a (possibly > nontrivial way) > on that datatype. Same approach? > > How about a function templated on multiple datatypes? Somewhat hackish is to do this with a pxi. Define a pxi using a dummy type, e.g. cdef class Wrapper: cdef TYPE value cdef set(self, TYPE value): self.value = value cdef TYPE get(self): return self.value ... Then make several pyx/pxd files that do ctypedef TYPE double include "wrapper.pxi" And voila, "templates." Of course, eventually we'll want to have a much nicer way of doing this, e.g. being able to template cdef classes like you can in C++ (though I don't think we want to go with the full template metaprograming model they have). - Robert From robertwb at math.washington.edu Thu Oct 1 20:19:31 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 1 Oct 2009 11:19:31 -0700 Subject: [Cython] doctesthack directive In-Reply-To: <4AC49B36.4070607@student.matnat.uio.no> References: <4AC49871.9070400@student.matnat.uio.no> <4AC49982.1080806@behnel.de> <4AC49B36.4070607@student.matnat.uio.no> Message-ID: <78D599DD-8F8F-478C-B325-0B352A6C2452@math.washington.edu> On Oct 1, 2009, at 5:06 AM, Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >> Dag Sverre Seljebotn wrote: >> >>> I now have code sitting on my drive for a doctesthack directive. >>> What it >>> does is work around the limitations of the doctest module by >>> automatically creating a global __test__ dictionary mapping >>> descriptive >>> names to function and method docstrings. >>> >>> 1) Can I push it? >>> 2) Can I make it the default? It never does anything if a __test__ >>> already exists, so it should be fully backwards compatible. The only >>> downside I can see is a little bit of extra memory used. >>> >>> Line numbers are embedded into the test names (the keys in >>> __dict__). >>> >>> Note: This does not solve the problem of doctest being unable to >>> print >>> line numbers for class docstrings. Also the test procedure ends up >>> flowing a little bit different from the Python module (as each >>> function >>> is a seperate testcase, and not simply part of the module test). >>> As the >>> name says, it is a hack. >>> >> >> Now that I know what kind of problems this does *not* fix and what >> new >> problems it introduces, what about some more detail what kind of >> problem >> this actually fixes? >> > What it fixes is that I can take a simple pyx file like this: > > def func(x): > """ >>>> func(3) > """ > ... > > and then test it like I would a Python module: > > python -c 'import doctest; import mymod; doctest.testmod(mymod)' > > doctest does not pick up the function docstring automatically as it > reports negative to inspect.isfunction, however putting those > functions > in the __test__ dict works around it. > > Arguments in favor: > - Lowers the barrier. It is the kind of thing which took me by > surprise > when I first met it; letting this be the default could save half an > hour > for every new Cython user. > - This also allows very easily writing function-wise doctests in the > Cython test suite > > An approach which requires a custom script for doctesting Cython files > is here: > > http://wiki.cython.org/FAQ#HowcanIrundoctestsinCythoncode. > 28pyxfiles.29.3F > > But I happen to like this approach better as it lower the barriers and > makes peoples test scripts simpler without Cython-specific logic. I actually wasn't aware that this was such an issue, as Sage has its own doctesting framework (e.g. it need to take care of preprocessing, optional doctests, etc.) I think the right thing to do is convince them to fix the doctest module. Of course, that won't help past and current version of Python. That being said, this looks like a sound solution, and I'm in favor of it going in, and probably even being on by default. - Robert From dagss at student.matnat.uio.no Thu Oct 1 20:44:46 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 01 Oct 2009 20:44:46 +0200 Subject: [Cython] doctesthack directive In-Reply-To: <78D599DD-8F8F-478C-B325-0B352A6C2452@math.washington.edu> References: <4AC49871.9070400@student.matnat.uio.no> <4AC49982.1080806@behnel.de> <4AC49B36.4070607@student.matnat.uio.no> <78D599DD-8F8F-478C-B325-0B352A6C2452@math.washington.edu> Message-ID: <4AC4F89E.7050308@student.matnat.uio.no> Robert Bradshaw wrote: > On Oct 1, 2009, at 5:06 AM, Dag Sverre Seljebotn wrote: > >> Stefan Behnel wrote: >>> Dag Sverre Seljebotn wrote: >>> >>>> I now have code sitting on my drive for a doctesthack directive. >>>> What it >>>> does is work around the limitations of the doctest module by >>>> automatically creating a global __test__ dictionary mapping >>>> descriptive >>>> names to function and method docstrings. >>>> >>>> 1) Can I push it? >>>> 2) Can I make it the default? It never does anything if a __test__ >>>> already exists, so it should be fully backwards compatible. The only >>>> downside I can see is a little bit of extra memory used. >>>> >>>> Line numbers are embedded into the test names (the keys in >>>> __dict__). >>>> >>>> Note: This does not solve the problem of doctest being unable to >>>> print >>>> line numbers for class docstrings. Also the test procedure ends up >>>> flowing a little bit different from the Python module (as each >>>> function >>>> is a seperate testcase, and not simply part of the module test). >>>> As the >>>> name says, it is a hack. >>>> >>> Now that I know what kind of problems this does *not* fix and what >>> new >>> problems it introduces, what about some more detail what kind of >>> problem >>> this actually fixes? >>> >> What it fixes is that I can take a simple pyx file like this: >> >> def func(x): >> """ >>>>> func(3) >> """ >> ... >> >> and then test it like I would a Python module: >> >> python -c 'import doctest; import mymod; doctest.testmod(mymod)' >> >> doctest does not pick up the function docstring automatically as it >> reports negative to inspect.isfunction, however putting those >> functions >> in the __test__ dict works around it. >> >> Arguments in favor: >> - Lowers the barrier. It is the kind of thing which took me by >> surprise >> when I first met it; letting this be the default could save half an >> hour >> for every new Cython user. >> - This also allows very easily writing function-wise doctests in the >> Cython test suite >> >> An approach which requires a custom script for doctesting Cython files >> is here: >> >> http://wiki.cython.org/FAQ#HowcanIrundoctestsinCythoncode. >> 28pyxfiles.29.3F >> >> But I happen to like this approach better as it lower the barriers and >> makes peoples test scripts simpler without Cython-specific logic. > > I actually wasn't aware that this was such an issue, as Sage has its > own doctesting framework (e.g. it need to take care of preprocessing, > optional doctests, etc.) I think the right thing to do is convince > them to fix the doctest module. Of course, that won't help past and > current version of Python. That being said, this looks like a sound > solution, and I'm in favor of it going in, and probably even being on > by default. It's possible to rather easily fix doctest to pick up the functions, but to my knowledge there's no way for doctest to retrieve the line numbers -- which has made me reluctant to submit anything. My hack works around it by embedding that information in the test name, so it says something like: Doctest failed at line ?: somemodule.__tests__.somefunction (line 35) I guess we could try to agree with them on a new convention though -- something like: Any module can optionally provide the dict __testlinenumbers__, mapping functions and classes to integers. This dict, if present, is checked before any of the conventional methods for retrieving line numbers. -- Dag Sverre From dagss at student.matnat.uio.no Thu Oct 1 21:02:16 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 01 Oct 2009 21:02:16 +0200 Subject: [Cython] Can't push to repos Message-ID: <4AC4FCB8.9040402@student.matnat.uio.no> I get 500 Internal server error on pushes to hg.cython.org. I tried running "hg verify" in the cython-unstable folder (by SSHing into the server) but it didn't show anything. Any other ideas? -- Dag Sverre From stefan_ml at behnel.de Thu Oct 1 21:07:59 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 01 Oct 2009 21:07:59 +0200 Subject: [Cython] PyBench results compared to Py2.6.2 Message-ID: <4AC4FE0F.1060704@behnel.de> Hi all, for a talk I'm about to give, I ran the pybench benchmark with cython-unstable. I had to fix up the sources a tiny bit (nothing that impacts the results) and select the benchmarks that actually compile (mostly due to inner classes being used). It's run like this: PYTHONPATH=. python2.6 -c 'import pybench; pybench.PyBenchCmdline()' \ --debug -w 1 -n 5 -f results-cython.pybench -t \ 'Arithmetic|IfThen|Loops|Dict|Try.*Except|Import|List|Compare|String|Tuple|Unicode' FWIW, both the Python 2.6.2 interpreter and the Cython generated C modules were built on the same machine with the same CFLAGS. The sources I used are here: http://codespeak.net/lxml/pybench.tar.gz I attached the results. I left out the CompareIntegers/Floats/etc. benchmarks as they only compare numeric literals, which ends up in the C code as, well, a comparison of literals, so that the C compiler finds a lot of dead code to discard. The resulting 100% speedup isn't really that informative. Note that the numbers are for plain Python code without any type annotations, so a lot of possible optimisations are not even used. However, the remaining total improvement of almost 33%, and the huge speedups for control structures (especially loops) and builtin types is clearly worth a look. Have fun, Stefan -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: results-compared.txt Url: http://codespeak.net/pipermail/cython-dev/attachments/20091001/444742ad/attachment-0001.txt From ndbecker2 at gmail.com Thu Oct 1 21:45:24 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Thu, 01 Oct 2009 15:45:24 -0400 Subject: [Cython] generic types? References: <5E1EE29A-FBD0-435B-9D86-A0D7A7846684@math.washington.edu> Message-ID: Robert Bradshaw wrote: > On Oct 1, 2009, at 7:17 AM, Neal Becker wrote: > >> I need to code some cython functions which will be instantiated for >> several >> different data types. Very much like what c++ templates are used for. >> >> What approaches can be used to do this? >> >> (I'm guessing some kind of macro text processing. Does cython use >> cpp?) >> >> Now suppose in general I need a little more intelligence. For >> example, >> suppose I have a function that is templated on a single datatype. >> Suppose >> some types internal to that function depend in a (possibly >> nontrivial way) >> on that datatype. Same approach? >> >> How about a function templated on multiple datatypes? > > Somewhat hackish is to do this with a pxi. > > Define a pxi using a dummy type, e.g. > > cdef class Wrapper: > cdef TYPE value > cdef set(self, TYPE value): > self.value = value > cdef TYPE get(self): > return self.value > ... > > Then make several pyx/pxd files that do > > ctypedef TYPE double > include "wrapper.pxi" > > And voila, "templates." > > Of course, eventually we'll want to have a much nicer way of doing > this, e.g. being able to template cdef classes like you can in C++ > (though I don't think we want to go with the full template > metaprograming model they have). > > - Robert Would that work where a type used within a class or function is 'computed'? Could ctypedef TYPE double be made conditional, for example? From dagss at student.matnat.uio.no Thu Oct 1 21:52:27 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Thu, 01 Oct 2009 21:52:27 +0200 Subject: [Cython] generic types? In-Reply-To: References: <5E1EE29A-FBD0-435B-9D86-A0D7A7846684@math.washington.edu> Message-ID: <4AC5087B.8050901@student.matnat.uio.no> Neal Becker wrote: > Robert Bradshaw wrote: > >> On Oct 1, 2009, at 7:17 AM, Neal Becker wrote: >> >>> I need to code some cython functions which will be instantiated for >>> several >>> different data types. Very much like what c++ templates are used for. >>> >>> What approaches can be used to do this? >>> >>> (I'm guessing some kind of macro text processing. Does cython use >>> cpp?) >>> >>> Now suppose in general I need a little more intelligence. For >>> example, >>> suppose I have a function that is templated on a single datatype. >>> Suppose >>> some types internal to that function depend in a (possibly >>> nontrivial way) >>> on that datatype. Same approach? >>> >>> How about a function templated on multiple datatypes? >> Somewhat hackish is to do this with a pxi. >> >> Define a pxi using a dummy type, e.g. >> >> cdef class Wrapper: >> cdef TYPE value >> cdef set(self, TYPE value): >> self.value = value >> cdef TYPE get(self): >> return self.value >> ... >> >> Then make several pyx/pxd files that do >> >> ctypedef TYPE double >> include "wrapper.pxi" >> >> And voila, "templates." >> >> Of course, eventually we'll want to have a much nicer way of doing >> this, e.g. being able to template cdef classes like you can in C++ >> (though I don't think we want to go with the full template >> metaprograming model they have). >> >> - Robert > > Would that work where a type used within a class or function is 'computed'? > Could > ctypedef TYPE double > be made conditional, for example? Personally I'd save myself the trouble and use a good templating engine to generate my Cython sources. I know people use http://www.cheetahtemplate.org/ to generate GPU code (with PyCUDA) for instance, so it is probably usable for your purpose. -- Dag Sverre From robert.kern at gmail.com Thu Oct 1 22:50:35 2009 From: robert.kern at gmail.com (Robert Kern) Date: Thu, 01 Oct 2009 15:50:35 -0500 Subject: [Cython] Updating Documentation In-Reply-To: <4AC4D745.4020301@noaa.gov> References: <1e52e0880909300141l146ec811k40a90a439ee73a3a@mail.gmail.com> <4AC32A6D.7080000@student.matnat.uio.no> <4AC39271.3080300@noaa.gov> <4AC39AF8.9010906@student.matnat.uio.no> <4AC3F2C8.70006@noaa.gov> <4AC4D745.4020301@noaa.gov> Message-ID: Christopher Barker wrote: > Robert Kern wrote: >> Christopher Barker wrote: > >>> agreed, though I may never use the stuff used to process XML docs, but >>> it gets first class treatment... >> There really isn't any special "stuff used to process XML docs" in Cython. There >> is special stuff in Cython to interact with numpy arrays. > > I never used the term "special". My point was that for many users, > working with numpy arrays may be just as or more important that python > strings (or even lists, etc.) As I see it, the only thing "special" > about numpy arrays is that they are not part of the standard library > (which I think they should be) -- but they are part of the standard > Cython distro. My point in adding the "special" is to note that you probably *are* using the Cython features that are used in lxml whereas most people not using numpy will never touch the buffer syntax. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From robertwb at math.washington.edu Fri Oct 2 03:39:16 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 1 Oct 2009 18:39:16 -0700 Subject: [Cython] Updating Documentation In-Reply-To: <1e52e0880910010021w1295bf4gdfba2ff0555768b0@mail.gmail.com> References: <1e52e0880909300141l146ec811k40a90a439ee73a3a@mail.gmail.com> <1e52e0880909300253s44949a0al5d987c03c2ba503d@mail.gmail.com> <4AC32BB5.7050604@student.matnat.uio.no> <1e52e0880909300306u1940255ara37e321bc8634b88@mail.gmail.com> <1e52e0880909300327p63bc1fccjf78b58f643454085@mail.gmail.com> <4AC338AE.1010404@student.matnat.uio.no>

<1e52e0880910010021w1295bf4gdfba2ff0555768b0@mail.gmail.com> Message-ID: <880EDD51-6ED7-410F-BFD3-4C290D006D65@math.washington.edu> On Oct 1, 2009, at 12:21 AM, Peter Alexander wrote: > Hi Robert. I have a branch repo at > http://bitbucket.org/travlr/cython-docs . That should suffice for now, > I'd think. Unless you all want it differently. Yes, that's totally fine. Distributed revision control is nice :) > At this point I've redone the super-structure and am iterating inward. > Conciseness is my goal. Let me know, at any time, your thoughts or > vision do not coincide with what I do.. Looks good. I think most of the language basics could be put later, with the exception of typing. So the first three sections could be overview, data typing, and compilation. After those three, we could move on to the topic-specific tutorials. - Robert From vel.accel at gmail.com Fri Oct 2 04:13:44 2009 From: vel.accel at gmail.com (Peter Alexander) Date: Thu, 1 Oct 2009 22:13:44 -0400 Subject: [Cython] Updating Documentation In-Reply-To: <880EDD51-6ED7-410F-BFD3-4C290D006D65@math.washington.edu> References: <1e52e0880909300141l146ec811k40a90a439ee73a3a@mail.gmail.com> <1e52e0880909300253s44949a0al5d987c03c2ba503d@mail.gmail.com> <4AC32BB5.7050604@student.matnat.uio.no> <1e52e0880909300306u1940255ara37e321bc8634b88@mail.gmail.com> <1e52e0880909300327p63bc1fccjf78b58f643454085@mail.gmail.com> <4AC338AE.1010404@student.matnat.uio.no>

<1e52e0880910010021w1295bf4gdfba2ff0555768b0@mail.gmail.com> <880EDD51-6ED7-410F-BFD3-4C290D006D65@math.washington.edu> Message-ID: <1e52e0880910011913w4e6c6b04q69b02f3ad61c7d85@mail.gmail.com> On Thu, Oct 1, 2009 at 9:39 PM, Robert Bradshaw wrote: > On Oct 1, 2009, at 12:21 AM, Peter Alexander wrote: > >> Hi Robert. I have a branch repo at >> http://bitbucket.org/travlr/cython-docs . That should suffice for now, >> I'd think. Unless you all want it differently. > > Yes, that's totally fine. Distributed revision control is nice :) > >> At this point I've redone the super-structure and am iterating inward. >> Conciseness is my goal. Let me know, at any time, your thoughts or >> vision do not coincide with what I do.. > > Looks good. I think most of the language basics could be put later, > with the exception of typing. So the first three sections could be > overview, data typing, and compilation. After those three, we could > move on to the topic-specific tutorials. > > - Robert > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > I look at what I'm working on now as a "reference guide" should be solely considered as a "get your fingers on the answer, quickly", guide. Basically, what I'm thinking of for this, is a very complete outline stating the language rules, particulars, etc., with cross-references galore, indexing, and lots of examples, etc., I believe that a multiplicity of documentation sources is a good thing.. Tutorial, reference, and "papers".. etc, etc. Tutorials as introductory "dive-in, and get you hair wet" type, and acomplete, very well thought out, no-nonsense reference material for those engaged in their project that "just want the answer that second" (like me) kind a guy/gal. So.. personally I am invested in the reference type, but I totally think that domain specific tutorials would be great for Cython to support. And a necessity, you might say. But, for now I'd like to concentrate on making the ref guide succinct, accessible, and most of all... complete. I also very much appreciate input from others. So please speak up when you want to... so we can get this right. ~Peter From vel.accel at gmail.com Fri Oct 2 05:27:34 2009 From: vel.accel at gmail.com (Peter Alexander) Date: Thu, 1 Oct 2009 23:27:34 -0400 Subject: [Cython] Updating Documentation In-Reply-To: <1e52e0880910011913w4e6c6b04q69b02f3ad61c7d85@mail.gmail.com> References: <1e52e0880909300141l146ec811k40a90a439ee73a3a@mail.gmail.com> <4AC32BB5.7050604@student.matnat.uio.no> <1e52e0880909300306u1940255ara37e321bc8634b88@mail.gmail.com> <1e52e0880909300327p63bc1fccjf78b58f643454085@mail.gmail.com> <4AC338AE.1010404@student.matnat.uio.no>

<1e52e0880910010021w1295bf4gdfba2ff0555768b0@mail.gmail.com> <880EDD51-6ED7-410F-BFD3-4C290D006D65@math.washington.edu> <1e52e0880910011913w4e6c6b04q69b02f3ad61c7d85@mail.gmail.com> <1e52e0880910012027u2580dc9co79e48fd84920c40f@mail.gmail.com> <1e52e0880910012146r7ed37d78r9dfdbc43d37400f7@mail.gmail.com> <6BA4BC4F-66D0-4167-B65E-31FE32504376@math.washington.edu> <4AC5B2D0.7070705@student.matnat.uio.no> <3CA75457-B053-4657-A282-B760F62B921F@math.washington.edu> Message-ID: <4AC5C2B0.5090005@student.matnat.uio.no> Robert Bradshaw wrote: > On Oct 2, 2009, at 12:59 AM, Dag Sverre Seljebotn wrote: > > > > >> I must say I'm really keen on having code snippets embedded into ReST >> text (and rather automatically exported as files), rather than the >> other >> way around. >> > > > Me too. I like your example of using :hidden and constructing whole > files--I didn't know you could do that. Sounds perfect (for the > tutorials at least). > Basically I think you are just fed everything (arguments + body) of a ReST tag, and are expected to supply a replacement ReST string. So you can do anything. Worst-case sceneario, two passes through the docs would be needed, one to create the whole file...I don't know, but that's not too horrible. So I just invented something. Come to think of it, I'd like #SHOW, #HIDE and #SNIP instructions in Cython source instead so that the number of ReST tags can be reduced. Dag Sverre From robertwb at math.washington.edu Fri Oct 2 11:09:27 2009 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 2 Oct 2009 02:09:27 -0700 Subject: [Cython] Updating Documentation In-Reply-To: References: <1e52e0880909300141l146ec811k40a90a439ee73a3a@mail.gmail.com> <1e52e0880910010021w1295bf4gdfba2ff0555768b0@mail.gmail.com> <880EDD51-6ED7-410F-BFD3-4C290D006D65@math.washington.edu> <1e52e0880910011913w4e6c6b04q69b02f3ad61c7d85@mail.gmail.com> <1e52e0880910012027u2580dc9co79e48fd84920c40f@mail.gmail.com> <1e52e0880910012146r7ed37d78r9dfdbc43d37400f7@mail.gmail.com> <6BA4BC4F-66D0-4167-B65E-31FE32504376@math.washington.edu> <4AC5B2D0.7070705@student.matnat.uio.no> <4AC5B6C6.9050706@student.matnat.uio.no> <1e52e0880910020132y70c075fet7e93dd62f63931a9@mail.gmail.com> <4AC5BDD2.90704@student.matnat.uio.no> Message-ID: <978905F9-B728-4E2B-824F-2FEED13DB33B@math.washington.edu> > > > We do that in Sage and it works very well--the reference manual is > thousands of pages long, nearly all from function docstrings. > >> That doesn't *quite* all the time with >> Cython, but I think it could work great in some types of situations. >> >> Experiment: >> >> class WraparoundDirective(CompilerDirective): # in Options.py >> """ >> wraparound >> >> information about the wraparound compiler directive >> """ >> >> class ForInStatNode(Node): # In Nodes.py >> """ >> Python-style for-loop >> >> more info >> >> TAGS: pythoncompatible, loops >> """ >> >> >> Some transforms would fit for documenting language features as well: >> >> class ForloopOptimizations(CythonTransform): >> """ >> Optimizations done to for-loops: >> >> - for in range is ... >> """ > > No offense intended (as this was clearly marked as an experiment) but > I have to admit this feels rather contrived, especially if we end up > making classes/structuring code just to hold the docstrings of user- > level abstractions. I think it would lead to a sparse scattering of > documentation throughout the codebase (most of the code does *not* > need to be exposed to the user documentation, e.g. all the Nodes that > represent Python nodes, and most transforms). The comments/doctests > in the code should probably be about the code itself (useful to a > cython developer, but probably not to a cython user). [I hit send too soon] It feels a bit as if one put the documentation for the Python language into the source of the CPython interpreter. - Robert From dagss at student.matnat.uio.no Fri Oct 2 11:13:41 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 02 Oct 2009 11:13:41 +0200 Subject: [Cython] Updating Documentation In-Reply-To: References: <1e52e0880909300141l146ec811k40a90a439ee73a3a@mail.gmail.com> <1e52e0880910010021w1295bf4gdfba2ff0555768b0@mail.gmail.com> <880EDD51-6ED7-410F-BFD3-4C290D006D65@math.washington.edu> <1e52e0880910011913w4e6c6b04q69b02f3ad61c7d85@mail.gmail.com> <1e52e0880910012027u2580dc9co79e48fd84920c40f@mail.gmail.com> <1e52e0880910012146r7ed37d78r9dfdbc43d37400f7@mail.gmail.com> <6BA4BC4F-66D0-4167-B65E-31FE32504376@math.washington.edu> <4AC5B2D0.7070705@student.matnat.uio.no> <4AC5B6C6.9050706@student.matnat.uio.no> <1e52e0880910020132y70c075fet7e93dd62f63931a9@mail.gmail.com> <4AC5BDD2.90704@student.matnat.uio.no> Message-ID: <4AC5C445.8050706@student.matnat.uio.no> Robert Bradshaw wrote: > On Oct 2, 2009, at 1:46 AM, Dag Sverre Seljebotn wrote: > > >> Peter Alexander wrote: >> >>> Putting my creative hat on... >>> >>> Would be nice to just be able to toss "chunks" of information into a >>> directory pool, that can be automatically structured to a view, by a >>> "documentation structure model"? >>> >>> That way upkeep could be made more simple because all you'd have to >>> worry about is tossing in the "chunk". >>> >>> That's kind of what I mean by an MVC model. >>> > > It almost sounds like you want a big glorified index. (I'm not saying > that's a bad thing, and it certainly would fit your bill of being > able to look stuff up fast.) > > >> OK, I see where you are coming from. As usual my suggestions are >> mostly >> with tutorials in mind. >> >> Other projects can do this in a great way by making the docstrings in >> the codebase their chunks. >> > > We do that in Sage and it works very well--the reference manual is > thousands of pages long, nearly all from function docstrings. > > >> That doesn't *quite* all the time with >> Cython, but I think it could work great in some types of situations. >> >> Experiment: >> >> class WraparoundDirective(CompilerDirective): # in Options.py >> """ >> wraparound >> >> information about the wraparound compiler directive >> """ >> >> class ForInStatNode(Node): # In Nodes.py >> """ >> Python-style for-loop >> >> more info >> >> TAGS: pythoncompatible, loops >> """ >> >> >> Some transforms would fit for documenting language features as well: >> >> class ForloopOptimizations(CythonTransform): >> """ >> Optimizations done to for-loops: >> >> - for in range is ... >> """ >> > > No offense intended (as this was clearly marked as an experiment) but > I have to admit this feels rather contrived, especially if we end up > making classes/structuring code just to hold the docstrings of user- > level abstractions. I think it would lead to a sparse scattering of > documentation throughout the codebase (most of the code does *not* > need to be exposed to the user documentation, e.g. all the Nodes that > represent Python nodes, and most transforms). The comments/doctests > in the code should probably be about the code itself (useful to a > cython developer, but probably not to a cython user). > > You may be right. (The one advantage is that it makes it easy to have documentation for different versions of Cython.). Thing is I need to have some "way of entry" into writing documentation -- I must either think about the code that implements the feature, or how I'd "tell people about it" directly in the docs. A pure MVC doesn't give me that I think -- it would tend to just be another hierarchy to remember just for the sake of it, when the hierarchy present in the docs (the view) is the one I'm likely to know by heart. So I'm not sure I'd like to just dump a chunk somewhere if it is not in the code, as I wouldn't get into the right "mood" by just dumping a fragment somewhere (how do I find the fragment again later? At least with the codebase I'd know where to look.) So I think I'm not a big supporter of MVC with chunks as seperate files in a seperate directory. However, if all I have to do is submit trac tickets containing documentation for a feature I just wrote, then I don't care much either way. Dag Sverre From vel.accel at gmail.com Fri Oct 2 11:25:27 2009 From: vel.accel at gmail.com (Peter Alexander) Date: Fri, 2 Oct 2009 05:25:27 -0400 Subject: [Cython] Updating Documentation In-Reply-To: <4AC5C445.8050706@student.matnat.uio.no> References: <1e52e0880909300141l146ec811k40a90a439ee73a3a@mail.gmail.com> <1e52e0880910012027u2580dc9co79e48fd84920c40f@mail.gmail.com> <1e52e0880910012146r7ed37d78r9dfdbc43d37400f7@mail.gmail.com> <6BA4BC4F-66D0-4167-B65E-31FE32504376@math.washington.edu> <4AC5B2D0.7070705@student.matnat.uio.no> <4AC5B6C6.9050706@student.matnat.uio.no> <1e52e0880910020132y70c075fet7e93dd62f63931a9@mail.gmail.com> <4AC5BDD2.90704@student.matnat.uio.no> <4AC5C445.8050706@student.matnat.uio.no> Message-ID: <1e52e0880910020225p5d2eb9d4vb82bd9f0a11ed036@mail.gmail.com> On Fri, Oct 2, 2009 at 5:13 AM, Dag Sverre Seljebotn wrote: > Robert Bradshaw wrote: >> On Oct 2, 2009, at 1:46 AM, Dag Sverre Seljebotn wrote: >> >> >>> Peter Alexander wrote: >>> >>>> Putting my creative hat on... >>>> >>>> Would be nice to just be able to toss "chunks" of information into a >>>> directory pool, that can be automatically structured to a view, by a >>>> "documentation structure model"? >>>> >>>> That way upkeep could be made more simple because all you'd have to >>>> worry about is tossing in the "chunk". >>>> >>>> That's kind of what I mean by an MVC model. >>>> >> >> It almost sounds like you want a big glorified index. (I'm not saying >> that's a bad thing, and it certainly would fit your bill of being >> able to look stuff up fast.) >> >> >>> OK, I see where you are coming from. As usual my suggestions are >>> mostly >>> with tutorials in mind. >>> >>> Other projects can do this in a great way by making the docstrings in >>> the codebase their chunks. >>> >> >> We do that in Sage and it works very well--the reference manual is >> thousands of pages long, nearly all from function docstrings. >> >> >>> That doesn't *quite* all the time with >>> Cython, but I think it could work great in some types of situations. >>> >>> Experiment: >>> >>> class WraparoundDirective(CompilerDirective): # in Options.py >>> ? ? """ >>> ? ? wraparound >>> >>> ? ? information about the wraparound compiler directive >>> ? ? """ >>> >>> class ForInStatNode(Node): # In Nodes.py >>> ? ?""" >>> ? ?Python-style for-loop >>> >>> ? ?more info >>> >>> ? ?TAGS: pythoncompatible, loops >>> ? ?""" >>> >>> >>> Some transforms would fit for documenting language features as well: >>> >>> class ForloopOptimizations(CythonTransform): >>> ? ? """ >>> ? ? Optimizations done to for-loops: >>> >>> ? ? ? ?- for in range is ... >>> ? ? """ >>> >> >> No offense intended (as this was clearly marked as an experiment) but >> I have to admit this feels rather contrived, especially if we end up >> making classes/structuring code just to hold the docstrings of user- >> level abstractions. I think it would lead to a sparse scattering of >> documentation throughout the codebase (most of the code does *not* >> need to be exposed to the user documentation, e.g. all the Nodes that >> represent Python nodes, and most transforms). The comments/doctests >> in the code should probably be about the code itself (useful to a >> cython developer, but probably not to a cython user). >> >> > You may be right. (The one advantage is that it makes it easy to have > documentation for different versions of Cython.). > > Thing is I need to have some "way of entry" into writing documentation > -- I must either think about the code that implements the feature, or > how I'd "tell people about it" directly in the docs. A pure MVC doesn't > give me that I think -- it would tend to just be another hierarchy to > remember just for the sake of it, when the hierarchy present in the docs > (the view) is the one I'm likely to know by heart. > > So I'm not sure I'd like to just dump a chunk somewhere if it is not in > the code, as I wouldn't get into the right "mood" by just dumping a > fragment somewhere (how do I find the fragment again later? At least > with the codebase I'd know where to look.) So I think I'm not a big > supporter of MVC with chunks as seperate files in a seperate directory. > > However, if all I have to do is submit trac tickets containing > documentation for a feature I just wrote, then I don't care much either way. > > Dag Sverre > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > The thing that's different about Cython from the others, is that it's not a library which is API centric.. where extracting doc strings from the code source itself makes perfect sense. But, I don't feel that it's necessarily impractical in Cython's case either. It all depends on preference, I think. From vel.accel at gmail.com Fri Oct 2 11:29:19 2009 From: vel.accel at gmail.com (Peter Alexander) Date: Fri, 2 Oct 2009 05:29:19 -0400 Subject: [Cython] Updating Documentation In-Reply-To: <1e52e0880910020225p5d2eb9d4vb82bd9f0a11ed036@mail.gmail.com> References: <1e52e0880909300141l146ec811k40a90a439ee73a3a@mail.gmail.com> <1e52e0880910012146r7ed37d78r9dfdbc43d37400f7@mail.gmail.com> <6BA4BC4F-66D0-4167-B65E-31FE32504376@math.washington.edu> <4AC5B2D0.7070705@student.matnat.uio.no> <4AC5B6C6.9050706@student.matnat.uio.no> <1e52e0880910020132y70c075fet7e93dd62f63931a9@mail.gmail.com> <4AC5BDD2.90704@student.matnat.uio.no> <4AC5C445.8050706@student.matnat.uio.no> <1e52e0880910020225p5d2eb9d4vb82bd9f0a11ed036@mail.gmail.com> Message-ID: <1e52e0880910020229s254fc667gd6cf6e40ba7bdca@mail.gmail.com> On Fri, Oct 2, 2009 at 5:25 AM, Peter Alexander wrote: > On Fri, Oct 2, 2009 at 5:13 AM, Dag Sverre Seljebotn > wrote: >> Robert Bradshaw wrote: >>> On Oct 2, 2009, at 1:46 AM, Dag Sverre Seljebotn wrote: >>> >>> >>>> Peter Alexander wrote: >>>> >>>>> Putting my creative hat on... >>>>> >>>>> Would be nice to just be able to toss "chunks" of information into a >>>>> directory pool, that can be automatically structured to a view, by a >>>>> "documentation structure model"? >>>>> >>>>> That way upkeep could be made more simple because all you'd have to >>>>> worry about is tossing in the "chunk". >>>>> >>>>> That's kind of what I mean by an MVC model. >>>>> >>> >>> It almost sounds like you want a big glorified index. (I'm not saying >>> that's a bad thing, and it certainly would fit your bill of being >>> able to look stuff up fast.) >>> >>> >>>> OK, I see where you are coming from. As usual my suggestions are >>>> mostly >>>> with tutorials in mind. >>>> >>>> Other projects can do this in a great way by making the docstrings in >>>> the codebase their chunks. >>>> >>> >>> We do that in Sage and it works very well--the reference manual is >>> thousands of pages long, nearly all from function docstrings. >>> >>> >>>> That doesn't *quite* all the time with >>>> Cython, but I think it could work great in some types of situations. >>>> >>>> Experiment: >>>> >>>> class WraparoundDirective(CompilerDirective): # in Options.py >>>> ? ? """ >>>> ? ? wraparound >>>> >>>> ? ? information about the wraparound compiler directive >>>> ? ? """ >>>> >>>> class ForInStatNode(Node): # In Nodes.py >>>> ? ?""" >>>> ? ?Python-style for-loop >>>> >>>> ? ?more info >>>> >>>> ? ?TAGS: pythoncompatible, loops >>>> ? ?""" >>>> >>>> >>>> Some transforms would fit for documenting language features as well: >>>> >>>> class ForloopOptimizations(CythonTransform): >>>> ? ? """ >>>> ? ? Optimizations done to for-loops: >>>> >>>> ? ? ? ?- for in range is ... >>>> ? ? """ >>>> >>> >>> No offense intended (as this was clearly marked as an experiment) but >>> I have to admit this feels rather contrived, especially if we end up >>> making classes/structuring code just to hold the docstrings of user- >>> level abstractions. I think it would lead to a sparse scattering of >>> documentation throughout the codebase (most of the code does *not* >>> need to be exposed to the user documentation, e.g. all the Nodes that >>> represent Python nodes, and most transforms). The comments/doctests >>> in the code should probably be about the code itself (useful to a >>> cython developer, but probably not to a cython user). >>> >>> >> You may be right. (The one advantage is that it makes it easy to have >> documentation for different versions of Cython.). >> >> Thing is I need to have some "way of entry" into writing documentation >> -- I must either think about the code that implements the feature, or >> how I'd "tell people about it" directly in the docs. A pure MVC doesn't >> give me that I think -- it would tend to just be another hierarchy to >> remember just for the sake of it, when the hierarchy present in the docs >> (the view) is the one I'm likely to know by heart. >> >> So I'm not sure I'd like to just dump a chunk somewhere if it is not in >> the code, as I wouldn't get into the right "mood" by just dumping a >> fragment somewhere (how do I find the fragment again later? At least >> with the codebase I'd know where to look.) So I think I'm not a big >> supporter of MVC with chunks as seperate files in a seperate directory. >> >> However, if all I have to do is submit trac tickets containing >> documentation for a feature I just wrote, then I don't care much either way. >> >> Dag Sverre >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> > > The thing that's different about Cython from the others, is that it's > not a library which is API centric.. where extracting doc strings from > the code source itself makes perfect sense. > > But, I don't feel that it's necessarily impractical in Cython's case > either. It all depends on preference, I think. > More... come to think of it.. all of Qt's documenation (including technology elaborations, exaples and demos) reside within the source code itself. They use "qdoc" to extract it to a view. This is what DOxygen is modelled after which all though is primarily for API extraction, is not solely for that purpose either. From stefan_ml at behnel.de Fri Oct 2 12:06:28 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 02 Oct 2009 12:06:28 +0200 Subject: [Cython] Updating Documentation In-Reply-To: <1e52e0880910020229s254fc667gd6cf6e40ba7bdca@mail.gmail.com> References: <1e52e0880909300141l146ec811k40a90a439ee73a3a@mail.gmail.com> <1e52e0880910012146r7ed37d78r9dfdbc43d37400f7@mail.gmail.com> <6BA4BC4F-66D0-4167-B65E-31FE32504376@math.washington.edu> <4AC5B2D0.7070705@student.matnat.uio.no> <4AC5B6C6.9050706@student.matnat.uio.no> <1e52e0880910020132y70c075fet7e93dd62f63931a9@mail.gmail.com> <4AC5BDD2.90704@student.matnat.uio.no> <4AC5C445.8050706@student.matnat.uio.no> <1e52e0880910020225p5d2eb9d4vb82bd9f0a11ed036@mail.gmail.com> <1e52e0880910020229s254fc667gd6cf6e40ba7bdca@mail.gmail.com> Message-ID: <4AC5D0A4.9060300@behnel.de> Peter Alexander wrote: > On Fri, Oct 2, 2009 at 5:25 AM, Peter Alexander wrote: >> On Fri, Oct 2, 2009 at 5:13 AM, Dag Sverre Seljebotn wrote: >>> Robert Bradshaw wrote: >>>> No offense intended (as this was clearly marked as an experiment) but >>>> I have to admit this feels rather contrived, especially if we end up >>>> making classes/structuring code just to hold the docstrings of user- >>>> level abstractions. +1 >>>> I think it would lead to a sparse scattering of >>>> documentation throughout the codebase (most of the code does *not* >>>> need to be exposed to the user documentation, e.g. all the Nodes that >>>> represent Python nodes, and most transforms). The comments/doctests >>>> in the code should probably be about the code itself (useful to a >>>> cython developer, but probably not to a cython user). >>>> >>> You may be right. (The one advantage is that it makes it easy to have >>> documentation for different versions of Cython.). That's an orthogonal problem and I find it unlikely that we will ever have to manage so many different Cython versions that the documentation becomes a bottleneck. >>> Thing is I need to have some "way of entry" into writing documentation >>> -- I must either think about the code that implements the feature, or >>> how I'd "tell people about it" directly in the docs. A pure MVC doesn't >>> give me that I think -- it would tend to just be another hierarchy to >>> remember just for the sake of it, when the hierarchy present in the docs >>> (the view) is the one I'm likely to know by heart. >>> >>> So I'm not sure I'd like to just dump a chunk somewhere if it is not in >>> the code, as I wouldn't get into the right "mood" by just dumping a >>> fragment somewhere (how do I find the fragment again later? At least >>> with the codebase I'd know where to look.) So I think I'm not a big >>> supporter of MVC with chunks as seperate files in a seperate directory. +1 >>> However, if all I have to do is submit trac tickets containing >>> documentation for a feature I just wrote, then I don't care much either way. I don't think that can be sufficient. If it's only about a set of examples, there's the Demos directory. If it's anything that needs more commentary and presentation, you'll have to make it fit into the docs somewhere and make sure people a) find it at the right place(s) when they look for it and/or read along, and b) understand what you want to bring across with the example in this specific context. The problem with MVC here is that the model isn't complete. The context (i.e. the view) is just as important. The same line of code can tell very different things depending on the text section you put it into. It'll be hard to write examples that are reused in multiple documents without impacting clarity and conciseness in any of them. The only niche I can think of where a simplified approach might make sense is optimisations. Something like "Since version x.y.z, Cython will recognise /this/ code pattern and translate it into efficient C code like /this/" might work well when put into an index. Everything else is best expressed in flowing text IMO. >> The thing that's different about Cython from the others, is that it's >> not a library which is API centric.. where extracting doc strings from >> the code source itself makes perfect sense. >> >> But, I don't feel that it's necessarily impractical in Cython's case >> either. It all depends on preference, I think. > > More... come to think of it.. all of Qt's documenation (including > technology elaborations, exaples and demos) reside within the source > code itself. They use "qdoc" to extract it to a view. This is what > DOxygen is modelled after which all though is primarily for API > extraction, is not solely for that purpose either. That certainly doesn't apply to Cython. A language needs a very different way of presentation than a simple mapping from an API implementation/spec to a doc index. Stefan From wstein at gmail.com Fri Oct 2 17:59:23 2009 From: wstein at gmail.com (William Stein) Date: Fri, 2 Oct 2009 08:59:23 -0700 Subject: [Cython] sagemath.org, etc. Message-ID: <85e81ba30910020859o71309ccate0c970dad5424a14@mail.gmail.com> Hi, After being rock solid for 9 months, the VMware server crashed badly on boxen.math.washington.edu, and we absolutely could not get it to work despite repeated clean reinstalls, reboots, etc. Repeatedly tempted by things appearing to work, I tried to migrate the virtual infrastructure to VirtualBox, but after many hours of work major performance issues made this impossible for now (though I hold out hope) -- something about my hardware setup or VirtualBox makes it completely unusable. Fortunately, I was able to get VMware server installed on mod.math.washington.edu, and get the virtual servers back online. I had to change the DNS entries for everything to point to mod.math.washington.edu (128.208.160.198) instead of boxen.math.washington.edu (128.208.160.197). As the DNS propagates, you should see all relevant web pages come back and work exactly as before the crash (no data was lost). If you are one of the people that works on sagemath (e.g., to maintain the cython, sagemath, mpir, etc.) webpage, the main change for now is that you login to mod.math.washington.edu instead of boxen.math.washington.edu before connecting to sagemath. (This is temporary.) William -- William Stein Associate Professor of Mathematics University of Washington http://wstein.org From dagss at student.matnat.uio.no Fri Oct 2 18:33:05 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Fri, 02 Oct 2009 18:33:05 +0200 Subject: [Cython] Can't push to repos In-Reply-To: <95BD00DE-0C2B-437A-968A-B884404ED251@math.washington.edu> References: <4AC4FCB8.9040402@student.matnat.uio.no> <95BD00DE-0C2B-437A-968A-B884404ED251@math.washington.edu> Message-ID: <4AC62B41.6030401@student.matnat.uio.no> Robert Bradshaw wrote: > On Oct 1, 2009, at 12:02 PM, Dag Sverre Seljebotn wrote: > >> I get 500 Internal server error on pushes to hg.cython.org. >> >> I tried running "hg verify" in the cython-unstable folder (by SSHing >> into the server) but it didn't show anything. Any other ideas? > > Looks like everything hosted on sagemath is down right now. Looking > into it... This happened before VMWare went down, and now that the server is up again the issue is still there. 500 server error on push, otherwise seems to work OK... -- Dag Sverre From dalcinl at gmail.com Fri Oct 2 19:34:40 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 2 Oct 2009 14:34:40 -0300 Subject: [Cython] cimport behaviour: "from package cimport module" ... bug o feature? Message-ID: See this: I install mpi4py C-API headers, SWIG typemaps, and of course Cython pxd's here: $ tree $HOME/lib/python/mpi4py/include/ /u/dalcinl/lib/python/mpi4py/include/ `-- mpi4py |-- MPI.pxd |-- __init__.pyx |-- mpi.pxi |-- mpi4py.MPI.h |-- mpi4py.MPI_api.h |-- mpi4py.h |-- mpi4py.i `-- mpi_c.pxd As you can see, I have a (empty) __init__.pyx alonside MPI.pxd Now suppose I have Cython code wanting to cimport MPI.pxd like this: $ cat tmp.pyx from mpi4py cimport MPI Now I try to cythonize the source, passing -I (AFAIK) appropriately : $ cython -I$HOME/lib/python/mpi4py/include tmp.pyx Error converting Pyrex file to C: ------------------------------------------------------------ ... from mpi4py cimport MPI ^ ------------------------------------------------------------ /u/dalcinl/Devel/mpi4py-dev/demo/wrap-cython/tmp.pyx:1:0: 'mpi4py.pxd' not found Now if I add an empty mpi4py.pxd $ touch $HOME/lib/python/mpi4py/include/mpi4py.pxd And try to cythonize again... SUCCESS!!. Why should I add a "mpi4py.pxd" alongside the "mpi4py/" directory to make Cython understand that "mpi4py" is a package, provided that "mpi4py/__init__.pyx" file is there? Am I missing something? Is this a bug or a feature? Please note that if I code the cimport like this: cimport mpi4py.MPI as MPI ... then I DO NOT need to add the mpi4py.pxd stub file... This makes me think that the behaviour I'm describing is actually a bug... -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stefan_ml at behnel.de Fri Oct 2 22:23:50 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 02 Oct 2009 22:23:50 +0200 Subject: [Cython] benchmarking input In-Reply-To: <200909211527.n8LFR4xQ028739@theraft.openend.se> References: <200909211527.n8LFR4xQ028739@theraft.openend.se> Message-ID: <4AC66156.1050400@behnel.de> Hi, I just sent this mail to the PyPy list. Given that there is some interest in getting Cython into the Alioth shootout, and the move towards an nightly benchmarking run for PyPy, I thought it would be nice to have not only CPython as comparison. Stefan Stefan Behnel wrote: > Anders Hammarquist wrote: >> Anyway, what I was thinking about, and need input on, is how to get >> at the interpreters to run the benchmark. > [...] > > I wonder, would it be possible to add Cython to the benchmark loop? I would > love to see it compared to PyPy, simply because both projects aim to > compile Python code to C code (amongst other things, obviously). > > I know that Cython can't currently compete with PyPy in terms of feature > completeness - it clearly lacks some very important features of the Python > language, so it won't be able to run all benchmarks for a while, and the > comparison would easily show where the black spots are that need fixing. > > Just out of curiosity (and to wet your appetite :), I ran PyPy's richards > benchmark unmodified in Cython (latest cython-unstable) and got this: > > python2.6 -c 'import richards; richards.main()' > Richards benchmark (Python) starting... [...] > finished. > Total time for 10 iterations: 3.98 secs > Average time per iteration: 398.44 ms > > compared to CPython 2.6.2: > > python2.6 -c 'import richards; richards.main()' > Richards benchmark (Python) starting... [...] > finished. > Total time for 10 iterations: 4.86 secs > Average time per iteration: 485.97 ms > > That's almost 20% faster IMO and not bad at all, given that Cython's main > performance feature (C typing) wasn't used. When I use an external .pxd > file (attached) to redeclare the classes as extension types and to add a C > nature to their methods (still without any benchmark code modifications), I > get this: > > python2.6 -c 'import richards; richards.main(iterations=10)' > Richards benchmark (Python) starting... [...] > finished. > Total time for 10 iterations: 0.99 secs > Average time per iteration: 99.28 ms > > That's almost a factor of five compared to CPython. > > If possible, I would like to add both a normal Cython compiler run and a > pxd enabled run to the benchmark comparison with PyPy and CPython. Any > chance this could be integrated? > > I'm asking now, because I imagine that the benchmarking framework will have > to integrate the Cython compiler somehow, maybe using distutils or > on-the-fly compilation with pyximport. > > Stefan From ndbecker2 at gmail.com Fri Oct 2 23:41:30 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 02 Oct 2009 17:41:30 -0400 Subject: [Cython] Cannot assign type ... Message-ID: How do I fix this? cython doesn't seem to like my use of functions as parameters. /home/nbecker/raysat/mod/test1.pyx:27:110: Cannot assign type 'numpy.int64_t (numpy.int64_t, int, numpy.int64_t, int __pyx_skip_dispatch)' to 'test1.rnd_policy_t' -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: test1.pyx Url: http://codespeak.net/pipermail/cython-dev/attachments/20091002/2f4d2aa8/attachment.diff From ndbecker2 at gmail.com Sat Oct 3 01:51:44 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 02 Oct 2009 19:51:44 -0400 Subject: [Cython] numpy generic iteration Message-ID: I need to write functions that apply element-wise numpy arrays, and should be independent of the number of dimensions. numpy has a mechanism for generic (dimension-independent) iteration over elements of the array, but is there any way to use this (or implement something similar) in cython? From vel.accel at gmail.com Sat Oct 3 04:20:43 2009 From: vel.accel at gmail.com (Peter Alexander) Date: Fri, 2 Oct 2009 22:20:43 -0400 Subject: [Cython] numpy generic iteration In-Reply-To: References: Message-ID: <1e52e0880910021920g75eb5675rc8bc85db40108650@mail.gmail.com> On Fri, Oct 2, 2009 at 7:51 PM, Neal Becker wrote: > I need to write functions that apply element-wise numpy arrays, and should > be independent of the number of dimensions. > > numpy has a mechanism for generic (dimension-independent) iteration over > elements of the array, but is there any way to use this (or implement > something similar) in cython? > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > Hi Neal. I don't know exactly where Dag is right now on his great work with "numpy under cython's covers" but you can simply use pointer arithmetic.. right. Numpy has a set of macros it uses and if I remember correctly there is documentation of a their simple algorithm somewhere... maybe it was in Travis's numpy book.. I don't exactly remember the doc right now. If I come across it I'll come back and post it. From dagss at student.matnat.uio.no Sat Oct 3 09:14:29 2009 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Sat, 03 Oct 2009 09:14:29 +0200 Subject: [Cython] numpy generic iteration In-Reply-To: References: Message-ID: <4AC6F9D5.7030502@student.matnat.uio.no> Neal Becker wrote: > I need to write functions that apply element-wise numpy arrays, and should > be independent of the number of dimensions. > > numpy has a mechanism for generic (dimension-independent) iteration over > elements of the array, but is there any way to use this (or implement > something similar) in cython? There's two methods: ufuncs and multi-iterators. ufuncs is the most common one -- search for ufunc on the Cython wiki, there should be an example somewhere. I just did this using multi-iterators (which I had to do because I needed some context -- a temporary buffer -- during the calculation). Attaching my code for calculating a Wigner 3j symbol elementwise with full broadcasting. (It is not as fast as it could be -- it is possible to extract out the inner loop and have it as a C loop rather than rely on the iterator, which is faster. I didn't bother yet though, mainly because each Wigner symbol calculation is rather heavy.) The NumPy docs is rather good in this area, check there next! (search for the multiiterator functions I use + ufuncs) Also you need a newer numpy.pxd than the one shipped with Cython for both ufuncs and multiiterators. Just replace the one in Cython/Includes with the attachment. I will include it in the next Cython release. [Edit: The attachments were too large, but they're up at sage.math.washington.edu/home/dagss as numpy.pxd and wigner.pyx. The main idea is: cdef np.broadcast it = np.broadcast(l1, l2, l3, m1, m2, m3) shape = (