From nhaldimann at gmx.ch Sat Apr 1 23:10:28 2006 From: nhaldimann at gmx.ch (Niklaus Haldimann) Date: Sat Apr 1 23:10:57 2006 Subject: [pypy-dev] [Fwd: Dynamic Languages Symposium 2006 - Technical Papers (CfP)] Message-ID: <442EEC44.5030505@gmx.ch> Hi there If I remember correctly some people were under the impression that the deadline for the Dynamic Languages Symposium at OOPSLA has already passed. Obviously this isn't so (it's June 1st, see below). If PyPy is going to submit a technical paper to just one conference this year, it should probably be this one. ;) Considering who's on the program committee it's also likely that it will actually be accepted. Cheers Nik -------- Original Message -------- Subject: Dynamic Languages Symposium 2006 - Technical Papers (CfP) Date: Fri, 31 Mar 2006 16:20:47 +0200 From: Robert Hirschfeld Reply-To: hirschfeld@acm.org, The general-purpose Squeak developers list To: squeak-dev@lists.squeakfoundation.org, croquet@lists.wisc.edu Dynamic Languages Symposium 2006 - Technical Papers Call for papers Portland, Oregon, United States, October 22, 2006 http://www.dcl.hpi.uni-potsdam.de/dls2006/ The Dynamic Languages Symposium (DLS) at OOPSLA 2006 is a forum for discussion of dynamic languages, their implementation and application. While mature dynamic languages including Smalltalk, Lisp, Scheme, and Prolog continue to grow and inspire new converts, a new generation of dynamic scripting languages such as Python, Ruby, PHP, and JavaScript are successful in a wide range of applications. DLS provides a place for researchers and practitioners to come together and share their knowledge, experience, and ideas for future research and development. The Technical Papers track of DLS 2006 invites high quality papers reporting original research, innovative contributions or experience related to dynamic languages, their implementation and application. Accepted Papers will be published in the OOPSLA conference companion and the ACM Digital Library. Areas of interest include but are not limited to * Reflection and meta-programming * Very late binding, dynamic composition, and runtime adaptation * Actors and active objects * Innovative language features and implementation techniques * Development and platform support, tools * Language symbiosis and multi-paradigm languages * Experience reports and case studies * Interesting applications * Educational approaches and perspectives * Domain-oriented programming * Object-oriented, aspect-oriented, and context-oriented programming Submissions and proceedings We invite original contributions that neither have been published previously nor are under review by other refereed events or publications. Research papers should describe work that advances the current state of the art. Experience papers should be of broad interest and should describe insights gained from substantive practical applications. The program committee will evaluate each contributed paper based on its relevance, significance, clarity, and originality. Papers are to be submitted electronically at http://www.dcl.hpi.uni-potsdam.de/node2006/ in PDF format. Submissions need to use the ACM format, templates for which can be found at http://www.acm.org/sigs/pubs/proceed/template.html. Important dates Submission of papers (hard deadline): June 1, 2006 (Thursday) Author notification: July 1, 2006 (Saturday) Final version due: July 11, 2006 (Tuesday) DLS Technical Papers Day: October 22, 2006 (Sunday) DLS Invited Talks Day: October 23, 2006 (Monday) Chair Robert Hirschfeld Hasso-Plattner-Institut, University of Potsdam, Germany hirschfeld@acm.org Program committee David Asher, ActiveState, United States Gilad Bracha, Sun Microsystems, United States Pascal Costanza, Vrije Universiteit Brussel, Belgium Richard P. Gabriel, Sun Microsystems Laboratories, United States Robert Hirschfeld, HPI, University of Potsdam, Germany (chair) David Leibs, Advanced Micro Devices, United States Wolfgang De Meuter, Vrije Universiteit Brussel, Belgium Stephane Ducasse, Universite de Savoie, France Oscar Nierstrasz, University of Berne, Switzerland Ian Piumarta, Viewpoints Research Institute, United States David Simmons, Microsoft, United States Michael Sperber, University of Tuebingen, Germany Dave Thomas, Bedarra Research Labs, Canada Martin von Loewis, HPI, University of Potsdam, Germany Jon L White, United States Allen Wirfs-Brock, Microsoft, United States Roel Wuyts, Unversite Libre de Bruxelles, Belgium From lac at strakt.com Sun Apr 2 00:40:01 2006 From: lac at strakt.com (Laura Creighton) Date: Sun Apr 2 00:51:16 2006 Subject: [pypy-dev] [Fwd: Dynamic Languages Symposium 2006 - Technical Papers (CfP)] In-Reply-To: Message from Niklaus Haldimann of "Sat, 01 Apr 2006 23:10:28 +0200." <442EEC44.5030505@gmx.ch> References: <442EEC44.5030505@gmx.ch> Message-ID: <200604012240.k31Me1nk009980@theraft.strakt.com> Thank you Nik for this important information, we did indeed believe that we missed the deadline. David Ascher is Canadian, as is ActiveState. I have cc'd this to him so that he can go about the business of correcting the pure slander that they are American, because Nik has clearly taken this from some other site. Laura Creighton In a message of Sat, 01 Apr 2006 23:10:28 +0200, Niklaus Haldimann writes: >Hi there > >If I remember correctly some people were under the impression that the >deadline for the Dynamic Languages Symposium at OOPSLA has already >passed. Obviously this isn't so (it's June 1st, see below). If PyPy is >going to submit a technical paper to just one conference this year, it >should probably be this one. ;) Considering who's on the program >committee it's also likely that it will actually be accepted. > >Cheers >Nik > >-------- Original Message -------- >Subject: Dynamic Languages Symposium 2006 - Technical Papers (CfP) >Date: Fri, 31 Mar 2006 16:20:47 +0200 >From: Robert Hirschfeld >Reply-To: hirschfeld@acm.org, The general-purpose Squeak developers list > >To: squeak-dev@lists.squeakfoundation.org, croquet@lists.wisc.edu > >Dynamic Languages Symposium 2006 - Technical Papers > >Call for papers > >Portland, Oregon, United States, October 22, 2006 > > >http://www.dcl.hpi.uni-potsdam.de/dls2006/ > > >The Dynamic Languages Symposium (DLS) at OOPSLA 2006 is a forum for >discussion of dynamic languages, their implementation and application. >While mature dynamic languages including Smalltalk, Lisp, Scheme, and >Prolog continue to grow and inspire new converts, a new generation of >dynamic scripting languages such as Python, Ruby, PHP, and JavaScript >are successful in a wide range of applications. DLS provides a place for >researchers and practitioners to come together and share their >knowledge, experience, and ideas for future research and development. > >The Technical Papers track of DLS 2006 invites high quality papers >reporting original research, innovative contributions or experience >related to dynamic languages, their implementation and application. >Accepted Papers will be published in the OOPSLA conference companion and >the ACM Digital Library. > > >Areas of interest include but are not limited to > >* Reflection and meta-programming >* Very late binding, dynamic composition, and runtime adaptation >* Actors and active objects >* Innovative language features and implementation techniques >* Development and platform support, tools >* Language symbiosis and multi-paradigm languages >* Experience reports and case studies >* Interesting applications >* Educational approaches and perspectives >* Domain-oriented programming >* Object-oriented, aspect-oriented, and context-oriented programming > > >Submissions and proceedings > >We invite original contributions that neither have been published >previously nor are under review by other refereed events or >publications. Research papers should describe work that advances the >current state of the art. Experience papers should be of broad interest >and should describe insights gained from substantive practical >applications. The program committee will evaluate each contributed paper >based on its relevance, significance, clarity, and originality. > >Papers are to be submitted electronically at >http://www.dcl.hpi.uni-potsdam.de/node2006/ in PDF format. Submissions >need to use the ACM format, templates for which can be found at >http://www.acm.org/sigs/pubs/proceed/template.html. > > >Important dates > >Submission of papers (hard deadline): June 1, 2006 (Thursday) >Author notification: July 1, 2006 (Saturday) >Final version due: July 11, 2006 (Tuesday) >DLS Technical Papers Day: October 22, 2006 (Sunday) >DLS Invited Talks Day: October 23, 2006 (Monday) > > >Chair > >Robert Hirschfeld >Hasso-Plattner-Institut, University of Potsdam, Germany >hirschfeld@acm.org > > >Program committee > >David Asher, ActiveState, United States >Gilad Bracha, Sun Microsystems, United States >Pascal Costanza, Vrije Universiteit Brussel, Belgium >Richard P. Gabriel, Sun Microsystems Laboratories, United States >Robert Hirschfeld, HPI, University of Potsdam, Germany (chair) >David Leibs, Advanced Micro Devices, United States >Wolfgang De Meuter, Vrije Universiteit Brussel, Belgium >Stephane Ducasse, Universite de Savoie, France >Oscar Nierstrasz, University of Berne, Switzerland >Ian Piumarta, Viewpoints Research Institute, United States >David Simmons, Microsoft, United States >Michael Sperber, University of Tuebingen, Germany >Dave Thomas, Bedarra Research Labs, Canada >Martin von Loewis, HPI, University of Potsdam, Germany >Jon L White, United States >Allen Wirfs-Brock, Microsoft, United States >Roel Wuyts, Unversite Libre de Bruxelles, Belgium > > >_______________________________________________ >pypy-dev@codespeak.net >http://codespeak.net/mailman/listinfo/pypy-dev From DavidA at ActiveState.com Sun Apr 2 00:53:02 2006 From: DavidA at ActiveState.com (David Ascher) Date: Sun Apr 2 00:53:36 2006 Subject: [pypy-dev] [Fwd: Dynamic Languages Symposium 2006 - Technical Papers (CfP)] In-Reply-To: <200604012240.k31Me1nk009980@theraft.strakt.com> References: <442EEC44.5030505@gmx.ch> <200604012240.k31Me1nk009980@theraft.strakt.com> Message-ID: <442F044E.8050001@ActiveState.com> Laura Creighton wrote: >Thank you Nik for this important information, we did indeed >believe that we missed the deadline. > >David Ascher is Canadian, as is ActiveState. I have cc'd this to him >so that he can go about the business of correcting the pure >slander that they are American, because Nik has clearly taken >this from some other site. > Thanks, I'd already gotten the CFP fixed, but after it started spreading. For the record, I'm not Canadian, although I am in Canada. I'm french and american =). From lac at strakt.com Sun Apr 2 01:07:34 2006 From: lac at strakt.com (Laura Creighton) Date: Sun Apr 2 01:18:19 2006 Subject: [pypy-dev] [Fwd: Dynamic Languages Symposium 2006 - Technical Papers (CfP)] In-Reply-To: Message from David Ascher of "Sat, 01 Apr 2006 14:53:02 -0800." <442F044E.8050001@ActiveState.com> References: <442EEC44.5030505@gmx.ch> <200604012240.k31Me1nk009980@theraft.strakt.com> <442F044E.8050001@ActiveState.com> Message-ID: <200604012307.k31N7YSP011668@theraft.strakt.com> In a message of Sat, 01 Apr 2006 14:53:02 -0800, David Ascher writes: >Laura Creighton wrote: > >>Thank you Nik for this important information, we did indeed >>believe that we missed the deadline. >> >>David Ascher is Canadian, as is ActiveState. I have cc'd this to him >>so that he can go about the business of correcting the pure >>slander that they are American, because Nik has clearly taken >>this from some other site. >> >Thanks, I'd already gotten the CFP fixed, but after it started spreading. > >For the record, I'm not Canadian, although I am in Canada. I'm french >and american =). Interesting mix, apologies for getting it wrong. Laura Creighton - in Sweden, but from NOVA SCOTIA or ACADIE for those of us for whom such things matter :-) (we don't feel particularly 'Canadian' and leave that for the people of Ontario, but we sure know where home is. Since Nik is from Switzerland, he will understand this sort of feeling perfectly well. :-) ) Laura From jp at hapra.at Sun Apr 2 20:54:59 2006 From: jp at hapra.at (Jakob Praher) Date: Sun Apr 2 21:06:53 2006 Subject: [pypy-dev] inquiry: phd positions Message-ID: hi all, my name is Jakob Praher. I am located in Linz, Austria. Currently I am finishing my master thesis, which is about developing an aspect oriented dynamic bytecode instrumentation framework for program analysis on top of LLVM. I am very interested to work in that area as a PHd student. From your website I know you are also working on an LLVM backend. PyPy seems to be a European effort, so my naive question is, whether you have any PhD projects in this area or could point me to some further information. I would be very interested to hearing from you. Thank you Jakob From mwh at python.net Mon Apr 3 01:59:29 2006 From: mwh at python.net (Michael Hudson) Date: Mon Apr 3 02:00:18 2006 Subject: [pypy-dev] Re: inquiry: phd positions References: Message-ID: <2mslovttpa.fsf@starship.python.net> Jakob Praher writes: > hi all, > > my name is Jakob Praher. I am located in Linz, Austria. Currently I am > finishing my master thesis, which is about developing an aspect > oriented dynamic bytecode instrumentation framework for program > analysis on top of LLVM. I am very interested to work in that area as > a PHd student. From your website I know you are also working on an > LLVM backend. I'm not _entirely_ sure what you are saying here. We have interests in AOP and interests in LLVM but they are not particularly related, whereas your interests seem to be intertwined. But it's late and I am suffering from a stressful trip, so maybe the confusion is just at my end :) > PyPy seems to be a European effort, Only by accident, mostly... > so my naive question is, whether you have any PhD projects in this > area or could point me to some further information. I would be very > interested to hearing from you. Well, we certainly hope that there are interesting lines of investigation in PyPy. To be more specific, we probably need you to be more specific about your interests too. If you are asking "do we have pots of cash lying around to fund PhDs", then I'm pretty sure the answer is "no". PyPy's funding from the EU was not structured in this sort of way (AIUI). Cheers, mwh -- (FREE|OPEN) BSD: Shire horse. Solid, reliable, only occasionally prone to crushing you against a wall and then only because you've told it to without knowing. -- Jim's pedigree of operating systems, asr From cfbolz at gmx.de Mon Apr 3 02:34:32 2006 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon Apr 3 02:35:06 2006 Subject: [pypy-dev] Re: inquiry: phd positions In-Reply-To: <2mslovttpa.fsf@starship.python.net> References: <2mslovttpa.fsf@starship.python.net> Message-ID: <44306D98.9090708@gmx.de> Hi Jakob! Jakob Praher writes: > my name is Jakob Praher. I am located in Linz, Austria. Currently I am > finishing my master thesis, which is about developing an aspect > oriented dynamic bytecode instrumentation framework for program > analysis on top of LLVM. I am very interested to work in that area as > a PHd student. From your website I know you are also working on an > LLVM backend. Well, yes. But an LLVM backend in PyPy lingo is different from an LLVM backend in LLVM lingo :-). The latter would be a backend to LLVM that targets a certain CPU architecture. But it is really the other way round: we have a backend that produces _LLVM_ code out of our own IL. No clue about the Ph.D., sorry. Cheers, Carl Friedrich From jp at hapra.at Mon Apr 3 09:55:46 2006 From: jp at hapra.at (Jakob Praher) Date: Mon Apr 3 09:56:39 2006 Subject: [pypy-dev] Re: inquiry: phd positions In-Reply-To: <2mslovttpa.fsf@starship.python.net> References: <2mslovttpa.fsf@starship.python.net> Message-ID: hi Michael, Michael Hudson schrieb: > Jakob Praher writes: > >> hi all, >> >> my name is Jakob Praher. I am located in Linz, Austria. Currently I am >> finishing my master thesis, which is about developing an aspect >> oriented dynamic bytecode instrumentation framework for program >> analysis on top of LLVM. I am very interested to work in that area as >> a PHd student. From your website I know you are also working on an >> LLVM backend. > > I'm not _entirely_ sure what you are saying here. We have interests > in AOP and interests in LLVM but they are not particularly related, > whereas your interests seem to be intertwined. I only wanted to give you some information, what I am doing right now. I am also interested in dynamic runtime systems. I have some experience with self and prototype based oo systems. I also had some experiences with Jalpanero (now JikesRVM), which is also selfhosting like PyPy. What I mean is that I am flexible. For me it is important to do open source research and that I could gain some experience for implementing large runtime systems. > > But it's late and I am suffering from a stressful trip, so maybe the > confusion is just at my end :) > >> PyPy seems to be a European effort, > > Only by accident, mostly... That is interesting :-) > >> so my naive question is, whether you have any PhD projects in this >> area or could point me to some further information. I would be very >> interested to hearing from you. > > Well, we certainly hope that there are interesting lines of > investigation in PyPy. To be more specific, we probably need you to > be more specific about your interests too. See above. My biggest concern is whether there could be some real colaboration with a university. It is quite hard to do this kind of research in Linz. Mostly because of a lack of support from the university side. > > If you are asking "do we have pots of cash lying around to fund PhDs", > then I'm pretty sure the answer is "no". PyPy's funding from the EU > was not structured in this sort of way (AIUI). No I just wanted to say that it would be very interesting to work in an area like dynamic object oriented runtimes. But there are many things why I am asking this out loud: * Is there any university involved in this project? * What area of PyPy needs some research in the size of a PhD project? Granted it is somewhat ignorant not to come up with an answer to the second question before writing to this list. But actually I wanted to hear some opinions from you. Anyways Linz is not the best place for hard core open source research. Before working on anything myself again, you have to understand that I have to do some questioning. -- Jakob From jp at hapra.at Mon Apr 3 10:20:35 2006 From: jp at hapra.at (Jakob Praher) Date: Mon Apr 3 10:22:11 2006 Subject: [pypy-dev] Re: inquiry: phd positions In-Reply-To: <44306D98.9090708@gmx.de> References: <2mslovttpa.fsf@starship.python.net> <44306D98.9090708@gmx.de> Message-ID: Carl Friedrich Bolz schrieb: > Hi Jakob! > > Jakob Praher writes: >> my name is Jakob Praher. I am located in Linz, Austria. Currently I am >> finishing my master thesis, which is about developing an aspect >> oriented dynamic bytecode instrumentation framework for program >> analysis on top of LLVM. I am very interested to work in that area as >> a PHd student. From your website I know you are also working on an >> LLVM backend. > > Well, yes. But an LLVM backend in PyPy lingo is different from an LLVM > backend in LLVM lingo :-). The latter would be a backend to LLVM that > targets a certain CPU architecture. But it is really the other way > round: we have a backend that produces _LLVM_ code out of our own IL. Yes :-) I just wanted to point out that I have some knowledge of the LLVM infrastructure and that I was using the JIT for my own project quite substantially. While there are many areas where still need a lot of work to understand, I am very interested in projects like yours and so I just wanted to ask. LLVM lacks symbolic information (hence lowlevel), which makes it impossible to be used as a first class IL for a project like pypy. I will have a look at your IL. > > No clue about the Ph.D., sorry. Never mind. Thanks for your reply. > > Cheers, > > Carl Friedrich > _______________________________________________ > pypy-dev@codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From jacob at strakt.com Mon Apr 3 10:45:16 2006 From: jacob at strakt.com (Jacob Hallen) Date: Mon Apr 3 10:45:49 2006 Subject: [pypy-dev] Re: inquiry: phd positions In-Reply-To: References: <2mslovttpa.fsf@starship.python.net> Message-ID: <200604031045.16156.jacob@strakt.com> Hi Jakob, I think we can safely say that the Pypy project contains several areas that are suitable for PhD level research. Just to mention a few: - Modularisation of the source language, allowing expansion and contraction of the available syntax and semantics. - Adaption of the platform to handle multiple source languages. -Use of the Pypy architecture for distributed computing, multiprocessing and handling of security issues at interpreter level. - Various aspects of code generation including translation, JIT specialisation, garbage collection and backend production. - Pypy integration with existing platforms, like Java and CLI. m?ndag 03 april 2006 09.55 skrev Jakob Praher: > >> so my naive question is, whether you have any PhD projects in this > >> area or could point me to some further information. I would be very > >> interested to hearing from you. > > > > Well, we certainly hope that there are interesting lines of > > investigation in PyPy. To be more specific, we probably need you to > > be more specific about your interests too. > > See above. My biggest concern is whether there could be some real > colaboration with a university. It is quite hard to do this kind of > research in Linz. Mostly because of a lack of support from the > university side. > > > If you are asking "do we have pots of cash lying around to fund PhDs", > > then I'm pretty sure the answer is "no". PyPy's funding from the EU > > was not structured in this sort of way (AIUI). > > No I just wanted to say that it would be very interesting to work in an > area like dynamic object oriented runtimes. But there are many things > why I am asking this out loud: > > * Is there any university involved in this project? > * What area of PyPy needs some research in the size of a PhD project? Heinrich Heine University of Duesseldorf is a member of the project, with professor Leuchel being formally responsible. In practice, it is Armin Rigo and Michael Hudson who are financed to be working on Pypy. As of this week Carl Friedrich Boltz will be doing his Bachelor of Science there, focusing on Pypy. While professor Leuchel is a very open minded person, who seems to enjoy giving people the liberty to pursue the kind of research they are interested in, I have no clue as to what the policy of the CS department or the university are concerning the acceptance of PhD students. I hope what I have said will help you get started on your own investigation. In the meantime, I hope you will have time to get acquainted with the code base. You are also welcome to join one of our sprints, where you will get a chance to collaborate with the developers on specific issues. The most convenient place for you in the reasonably near future will probably be the sprint right after Europython at CERN in Switzerland. We do have an open sprint in Japan before that, but it is a long way to travel and we will probably have as many newcomers as we can cope with. Best of luck Jacob Hall?n From Gerald.Klix at klix.ch Mon Apr 3 14:58:22 2006 From: Gerald.Klix at klix.ch (Gerald Klix) Date: Mon Apr 3 19:41:44 2006 Subject: [pypy-dev] rctypes design document Message-ID: <44311BEE.7000109@klix.ch> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, yesterday I checked in the final version of the rcytpes design document. I am convinced I finally got everything right. Especially the management of keep alive pointers. Have fun, Gerald - -- GPG-Key: http://keyserver.veridis.com:11371/search?q=0xA140D634 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFEMRvuEDg9cqFA1jQRAtOUAJ9hj26mRbeXULn3cxZ9Rhp3bBdOsACfe5P4 ssR79dN5e8brVM0y3JIZDNw= =pLSI -----END PGP SIGNATURE----- From jp at hapra.at Mon Apr 3 20:53:04 2006 From: jp at hapra.at (Jakob Praher) Date: Mon Apr 3 20:53:39 2006 Subject: [pypy-dev] Re: inquiry: phd positions In-Reply-To: <200604031045.16156.jacob@strakt.com> References: <2mslovttpa.fsf@starship.python.net> <200604031045.16156.jacob@strakt.com> Message-ID: <44316F10.80303@hapra.at> Hi Jacob, thank you for your support. Looks very interesting to hear that. What are you currently working on? AFAICT you are employed by strakt.com. Are you working on pypy as a research project? Jacob Hallen schrieb: > - Modularisation of the source language, allowing expansion and contraction of > the available syntax and semantics. > > - Adaption of the platform to handle multiple source languages. > > That is insteresting, especially when supporting other dynamic languages. > -Use of the Pypy architecture for distributed computing, multiprocessing and > handling of security issues at interpreter level. > Ah yes. I recently attended a guest lecture by Michael Franz. He is a professor at University Irvine in California. They are implementing MAC based security labels for virtual machines. He managed to bring the pefromance loss of labeling down to 6 % of the whole execution time. Given the fact that SeLinux for instance already supports MAC, MAC can be an interesting concept to bring to virtual machines as well. > - Various aspects of code generation including translation, JIT > specialisation, garbage collection and backend production. > Interesting. > - Pypy integration with existing platforms, like Java and CLI. > Could be interesting too. Especially Java plans on integrating an invokedynamic instruction. This makes it possible to design more efficient code generators. But I am not really 100 % sure, how this will look like. > > Heinrich Heine University of Duesseldorf is a member of the project, with > professor Leuchel being formally responsible. In practice, it is Armin Rigo > and Michael Hudson who are financed to be working on Pypy. As of this week > Carl Friedrich Boltz will be doing his Bachelor of Science there, focusing on > Pypy. > Thats interesting to hear. > While professor Leuchel is a very open minded person, who seems to enjoy > giving people the liberty to pursue the kind of research they are interested > in, I have no clue as to what the policy of the CS department or the > university are concerning the acceptance of PhD students. I hope what I have > said will help you get started on your own investigation. > Oh great. Thanks for the pointers. The problem I am facing is that PhD slots are pretty scarce and so there are often many people already waiting for one :-) > In the meantime, I hope you will have time to get acquainted with the code > base. You are also welcome to join one of our sprints, where you will get a > chance to collaborate with the developers on specific issues. The most > convenient place for you in the reasonably near future will probably be the > sprint right after Europython at CERN in Switzerland. We do have an open > sprint in Japan before that, but it is a long way to travel and we will > probably have as many newcomers as we can cope with. > > I will surely look at the project code base, no matter how this turns out. Thank you. > Best of luck > > Jacob Hall?n > > From nhaldimann at gmx.ch Mon Apr 3 22:54:47 2006 From: nhaldimann at gmx.ch (Niklaus Haldimann) Date: Mon Apr 3 22:55:36 2006 Subject: [pypy-dev] List and string in ootypesystem In-Reply-To: <442C2FB3.5070800@gmail.com> References: <442BC9CC.6010503@gmail.com> <442BF15B.9070102@gmx.ch> <442C2FB3.5070800@gmail.com> Message-ID: <44318B97.3050408@gmx.ch> Hi Antonio (and others) As you have probably seen, we started work on list support for ootypesystem here at the ongoing Leysin sprint. This cleared things up a bit for me on the conceptual level. Here's how OO lists work: - There is a new type constructor "List" in the ootype module. The low-level type of RPython list in the ootypesystem is a List instance with the item type of the list provided as an argument. E.g., an RPython list of integers has the low-level type List(Signed). - The interface of an OO list is defined by List._METHODS (see List.__init__). This interface is still incomplete, but it should be kept as small as possible, but large enough to support all operations permitted in RPython on lists. - When it comes to low-level operations, Lists are treated similarly to Instances. "new" is used to instantiate an (empty) list. Methods of the interface are rtyped as "oosend" operations with the name of the method. - Backends must special-case "new" with Lists and "oosends" to Lists. Basically, "new" can be mapped to the construction of a native list in the backend. The oosends can be mapped to native methods of the native list. All of this works for some basic list operations at this point (length, append, getitem, setitem), but there's still quite a bit of work to do for full list support in ootypesystem. After all this, the roadmap for bringing all other data structures to ootypesystem is now pretty clear to me: - dicts: Can be implemented analoguous to lists (type constructor, minimal interface etc.) - strings: Analoguous to lists, but no type constructor necessary - ranges, slices: Can be implemented as a special Instance (analoguous to tuples, in a way) I have two more days here at the Leysin sprint to work on lists. After that I again won't have much time to work on PyPy, so Antonio (or anyone else), you're very welcome to continue from here. I think the list code can serve as a good starting point for the other data structures. If there are any questions, I'm happy to answer them (if I can ;)). Cheers Nik From anto.cuni at gmail.com Tue Apr 4 01:15:04 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue Apr 4 01:16:42 2006 Subject: [pypy-dev] List and string in ootypesystem In-Reply-To: <44318B97.3050408@gmx.ch> References: <442BC9CC.6010503@gmail.com> <442BF15B.9070102@gmx.ch> <442C2FB3.5070800@gmail.com> <44318B97.3050408@gmx.ch> Message-ID: <4431AC78.9010705@gmail.com> Hi Niklaus, you have just preceded me by about 2 hours... I would have sent a mail about rlist anyway. I've spent last days working on this topic: I tried to refactor the code for making rlist type-system specific as rtuple, rclass and others already was. It has been a bit difficult because it was my first hacking in the rpython directory: I had to read the sources carefully for understanding in deep how things works, and I'm not sure if I have completely understood the whole logic. I've just committed my work in the http://codespeak.net/svn/user/antocuni/pypy-antocuni directory. I'm not satisfied of it because it is a bit messy and I'm happy to know that now it's no longer needed because yours is surely better :-). By the way I think it has not been a waste of time because it let me to gain some knowledge of pypy's internals that will be useful in future. Niklaus Haldimann wrote: > I have two more days here at the Leysin sprint to work on lists. After > that I again won't have much time to work on PyPy, so Antonio (or anyone > else), you're very welcome to continue from here. I think the list code > can serve as a good starting point for the other data structures. If > there are any questions, I'm happy to answer them (if I can ;)). Sure, I'll be happy to continue from here: it is much better than starting from scratch as I supposed to do! :-) I think you've saved me from a lot of headaches, thanks! Remind me to offer you a beer the first time we meet ;-). ciao Anto From nhaldimann at gmx.ch Tue Apr 4 09:07:30 2006 From: nhaldimann at gmx.ch (Niklaus Haldimann) Date: Tue Apr 4 09:07:58 2006 Subject: [pypy-dev] List and string in ootypesystem In-Reply-To: <4431AC78.9010705@gmail.com> References: <442BC9CC.6010503@gmail.com> <442BF15B.9070102@gmx.ch> <442C2FB3.5070800@gmail.com> <44318B97.3050408@gmx.ch> <4431AC78.9010705@gmail.com> Message-ID: <44321B32.8000504@gmx.ch> Hi Antonio Antonio Cuni wrote: > I've spent last days working on this topic: I tried to refactor the code > for making rlist type-system specific as rtuple, rclass and others > already was. > It has been a bit difficult because it was my first hacking in the > rpython directory: I had to read the sources carefully for understanding > in deep how things works, and I'm not sure if I have completely > understood the whole logic. > > I've just committed my work in the > http://codespeak.net/svn/user/antocuni/pypy-antocuni directory. I'm not > satisfied of it because it is a bit messy and I'm happy to know that now > it's no longer needed because yours is surely better :-). By the way I > think it has not been a waste of time because it let me to gain some > knowledge of pypy's internals that will be useful in future. Oops, I didn't intend to invalidate your work. ;) I actually checked your user directory yesterday, because you said in an earlier mail that you would work on the branch there. But since I didn't see any changes related to rlist I assumed you decided to postpone this ... What you did doesn't look so bad, I just looked at it. In general, I'm impressed that you found your way around the RTyper so easily. ;) The main difference to our work is that you created many new low-level operations for the list interface. Since there will also have to be a dict and string interface this would lead to an explosion of low-level operations. Our approach also makes testing of these data structures at a lower level easier (see test_oolist.py and test_oortype.py). > Sure, I'll be happy to continue from here: it is much better than > starting from scratch as I supposed to do! :-) > I think you've saved me from a lot of headaches, thanks! Remind me to > offer you a beer the first time we meet ;-). Hehe, looking forward to that. ;) Cheers Nik From anto.cuni at gmail.com Tue Apr 4 10:16:32 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue Apr 4 10:20:23 2006 Subject: [pypy-dev] List and string in ootypesystem In-Reply-To: <44321B32.8000504@gmx.ch> References: <442BC9CC.6010503@gmail.com> <442BF15B.9070102@gmx.ch> <442C2FB3.5070800@gmail.com> <44318B97.3050408@gmx.ch> <4431AC78.9010705@gmail.com> <44321B32.8000504@gmx.ch> Message-ID: <44322B60.3050302@gmail.com> Niklaus Haldimann wrote: > Oops, I didn't intend to invalidate your work. ;) I actually checked > your user directory yesterday, because you said in an earlier mail that > you would work on the branch there. But since I didn't see any changes > related to rlist I assumed you decided to postpone this ... No, I was working on that but I didn't commit because I wanted to get all things working before doing that... I've had some problem fixing all various modules that accessed rpython.rlist directly. > What you did doesn't look so bad, I just looked at it. In general, I'm > impressed that you found your way around the RTyper so easily. ;) Don't worry, it has not been so easy! :-) > The > main difference to our work is that you created many new low-level > operations for the list interface. Since there will also have to be a > dict and string interface this would lead to an explosion of low-level > operations. Our approach also makes testing of these data structures at > a lower level easier (see test_oolist.py and test_oortype.py). The reason beyond that is that I've found no other way to do this: I really wanted to create as few low-level ops as possibile, but I coudn't figure out how to do. I try to explain better what I did: my first attempt was to provide a low-level operation 'list_lenght' and then implement rtype_is_true in terms of 'list_lenght', so I wrote a thing like this: class ListRepr(AbstractListRepr): def rtype_len(self, hop): v_lst, = hop.inputargs(self) return hop.genop('list_len', [v_lst], resulttype=Signed) def rtype_is_true(self, hop): v_lst, = hop.inputargs(self) return hop.gendirectcall(ll_list_is_true, v_lst) def ll_list_is_true(lst): return lst is not None and len(lst) != 0 I hoped that the rtyper was smart enough to convert 'len(lst)' into my low-level op 'list_len', but it wasn't: indeed, it generated code that called len function for a generic PyObject* and that was not what I wanted. I tried to copy the implementation of lltypesystem.rlist.ll_list_is_true, but I couldn't because that can call the ll_lenght() method that I didn't have. Now it has no longer importance, but how could I do to get thing working? ciao Anto From anto.cuni at gmail.com Tue Apr 4 20:52:46 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue Apr 4 20:56:24 2006 Subject: [pypy-dev] Rtyping classes definitions? Message-ID: <4432C07E.6090406@gmail.com> Hi, I'm writing the code that generates CLI classes starting from ootypesystem classes definitions, but as always I have a problem :-). At the moment the only way to access class definitions is by calling TranslationContext.annotator.getuserclassdefinitions(), which returns a list of ClassDef instances: I've noticed that class attributes are not typed but only annotated. Obviously for generating CLI classes I need to rtype class definitions, so that I can know the exact low-level type of the fields: I wondered if the absence of the rtyping step is intended or if it is a thing to do. In the latter case I could try to do it (hoping to do better than with rlist :-). ciao Anto From nhaldimann at gmx.ch Tue Apr 4 23:52:43 2006 From: nhaldimann at gmx.ch (Niklaus Haldimann) Date: Tue Apr 4 23:53:10 2006 Subject: [pypy-dev] Rtyping classes definitions? In-Reply-To: <4432C07E.6090406@gmail.com> References: <4432C07E.6090406@gmail.com> Message-ID: <4432EAAB.90805@gmx.ch> Hi Antonio Antonio Cuni wrote: > At the moment the only way to access class definitions is by calling > TranslationContext.annotator.getuserclassdefinitions(), which returns a > list of ClassDef instances: I've noticed that class attributes are not > typed but only annotated. > > Obviously for generating CLI classes I need to rtype class definitions, > so that I can know the exact low-level type of the fields: I wondered if > the absence of the rtyping step is intended or if it is a thing to do. > In the latter case I could try to do it (hoping to do better than with > rlist :-). There must be a misunderstanding here, the rtyping step is not at all missing. ;) If you look at rtyped graphs you'll see that instances (and classes) have low-level types of the ootype.Instance kind. These Instance types have a _field attribute that is a dict mapping field names to their low-level types. You should be operating with these Instance types not with ClassDefs, the latter are mostly an implementation detail of the annotation phase. Cheers Nik From mwh at python.net Tue Apr 4 23:52:46 2006 From: mwh at python.net (Michael Hudson) Date: Tue Apr 4 23:53:50 2006 Subject: [pypy-dev] Re: Rtyping classes definitions? References: <4432C07E.6090406@gmail.com> Message-ID: <2mbqvhrosx.fsf@starship.python.net> Antonio Cuni writes: > Hi, > I'm writing the code that generates CLI classes starting from > ootypesystem classes definitions, but as always I have a problem :-). > > At the moment the only way to access class definitions is by calling > TranslationContext.annotator.getuserclassdefinitions(), which returns > a list of ClassDef instances: I've noticed that class attributes are > not typed but only annotated. Er. Well. I am almost entirely sure that your current approach of (simplified): for graph in self.translator.graphs: f = Function(graph) f.render(self.ilasm) is not really going to work. Have you looked at how (say) genc works? It first computes names for everything (the 'LowLevelDatabase'), and then spits out the code. 'everything' includes all the types referenced by the graphs, and because the graphs have been rtyped, the types you find are things like instances of ootype.Class. > Obviously for generating CLI classes I need to rtype class > definitions, so that I can know the exact low-level type of the > fields: I wondered if the absence of the rtyping step is intended or > if it is a thing to do. In the latter case I could try to do it > (hoping to do better than with rlist :-). I'm not sure this question, as phrased, makes sense :) Cheers, mwh -- A witty saying proves nothing. -- Voltaire From anto.cuni at gmail.com Wed Apr 5 00:04:55 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed Apr 5 00:06:43 2006 Subject: [pypy-dev] Rtyping classes definitions? In-Reply-To: <4432EAAB.90805@gmx.ch> References: <4432C07E.6090406@gmail.com> <4432EAAB.90805@gmx.ch> Message-ID: <4432ED87.8080608@gmail.com> Hi Niklaus, Niklaus Haldimann wrote: > There must be a misunderstanding here, the rtyping step is not at all > missing. ;) If you look at rtyped graphs you'll see that instances (and > classes) have low-level types of the ootype.Instance kind. These > Instance types have a _field attribute that is a dict mapping field > names to their low-level types. You should be operating with these > Instance types not with ClassDefs, the latter are mostly an > implementation detail of the annotation phase. Yes, there was a misunderstanding :-). I was searching for something containing the list of classes to be rendered, but now I've understood that I should collect informations about classes as long as I generate the code, that is the way gensqueak works (I was looking at your sources just now). Thanks a lot for your help, I'm sorry if I make you bored with my many questions, I hope they will decrease as soon as I will get acquainted with pypy's logic. ciao Anto From anto.cuni at gmail.com Wed Apr 5 00:09:10 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed Apr 5 00:10:56 2006 Subject: [pypy-dev] Re: Rtyping classes definitions? In-Reply-To: <2mbqvhrosx.fsf@starship.python.net> References: <4432C07E.6090406@gmail.com> <2mbqvhrosx.fsf@starship.python.net> Message-ID: <4432EE86.3040106@gmail.com> Michael Hudson wrote: > Er. Well. I am almost entirely sure that your current approach of > (simplified): > > for graph in self.translator.graphs: > f = Function(graph) > f.render(self.ilasm) > > is not really going to work. Have you looked at how (say) genc works? > It first computes names for everything (the 'LowLevelDatabase'), and > then spits out the code. 'everything' includes all the types > referenced by the graphs, and because the graphs have been rtyped, the > types you find are things like instances of ootype.Class. Yes, I think you are probably right. It worked until now for simple experiments, but now I have to follow another approach, as I've just wrote to Niklaus. I hope to commit something usable tomorrow... ciao Anto From niko at alum.mit.edu Wed Apr 5 13:51:13 2006 From: niko at alum.mit.edu (Niko Matsakis) Date: Wed Apr 5 13:51:51 2006 Subject: [pypy-dev] how to get into the codebase? Open bugs? Message-ID: <33D3D202-84BC-44B6-83D4-B72615BDB231@alum.mit.edu> Hello, I am yet-another-compiler-PhD-student interested in exploring PyPy. I have been lurking on the list for some time, and a Python enthusiast for longer, but I have yet to really dig into PyPy to get a better understanding of how it works (beyond what is published on the web site). In previous jobs and lives, I've always found that fixing a few simple bugs or implementing some low priority features can be a great way to delve into a code base. As such, I am wondering if there is a list somewhere of such "open bugs" or pending tasks that I might try to tackle. I understand that attending a Code Sprint would also be a great way to get into the code, but I'm not sure how/when I'll be able to fit that into my schedule. Perhaps if there is one close to Switzerland, or somewhere my wife wouldn't going on vacation. :) Eventually, of course, I'd love to be conversant enough with the PyPy code base to try experimenting more boldly with it, but I guess one must take baby steps before one can run. Niko From hpk at trillke.net Wed Apr 5 14:41:36 2006 From: hpk at trillke.net (holger krekel) Date: Wed Apr 5 14:45:57 2006 Subject: [pypy-dev] how to get into the codebase? Open bugs? In-Reply-To: <33D3D202-84BC-44B6-83D4-B72615BDB231@alum.mit.edu> References: <33D3D202-84BC-44B6-83D4-B72615BDB231@alum.mit.edu> Message-ID: <20060405124136.GK21936@solar.trillke> Hi Niko! On Wed, Apr 05, 2006 at 13:51 +0200, Niko Matsakis wrote: > I am yet-another-compiler-PhD-student interested in exploring PyPy. > I have been lurking on the list for some time, and a Python > enthusiast for longer, but I have yet to really dig into PyPy to get > a better understanding of how it works (beyond what is published on > the web site). he, welcome! > In previous jobs and lives, I've always found that fixing a few > simple bugs or implementing some low priority features can be a great > way to delve into a code base. As such, I am wondering if there is a > list somewhere of such "open bugs" or pending tasks that I might try > to tackle. There is https://codespeak.net/issue/pypy-dev/ which i recently updated and lists a number of problems. We are not really keeping very strictly to the classications of "easy,medium,hard" for the issues but it should be helpful a bit, still. > I understand that attending a Code Sprint would also be a great way > to get into the code, but I'm not sure how/when I'll be able to fit > that into my schedule. Perhaps if there is one close to Switzerland, > or somewhere my wife wouldn't going on vacation. :) many of us are at the moment in Leysin (Switzerland) ... > Eventually, of course, I'd love to be conversant enough with the PyPy > code base to try experimenting more boldly with it, but I guess one > must take baby steps before one can run. sounds like a good strategy. feel free to ask questions if you get stuck or wonder about things. best, holger From mwh at python.net Wed Apr 5 15:13:50 2006 From: mwh at python.net (Michael Hudson) Date: Wed Apr 5 15:15:23 2006 Subject: [pypy-dev] Re: how to get into the codebase? Open bugs? References: <33D3D202-84BC-44B6-83D4-B72615BDB231@alum.mit.edu> Message-ID: <2mk6a4qi5t.fsf@starship.python.net> Niko Matsakis writes: > I understand that attending a Code Sprint would also be a great way > to get into the code, but I'm not sure how/when I'll be able to fit > that into my schedule. Perhaps if there is one close to Switzerland, > or somewhere my wife wouldn't going on vacation. :) Well, as well as being in Switzerland right now, as Holger mentioned, there will be sprints around EuroPython in early July, which is being held at CERN this year. > Eventually, of course, I'd love to be conversant enough with the PyPy > code base to try experimenting more boldly with it, but I guess one > must take baby steps before one can run. True, but it definitely helps to have something in mind too! Cheers, mwh -- FORD: Just put the fish in your ear, come on, it's only a little one. ARTHUR: Uuuuuuuuggh! -- The Hitch-Hikers Guide to the Galaxy, Episode 1 From tismer at math.fu-berlin.de Thu Apr 6 04:38:13 2006 From: tismer at math.fu-berlin.de (Christian Tismer) Date: Thu Apr 6 08:08:52 2006 Subject: [pypy-dev] Re: email problem, please use this address In-Reply-To: <43FB7420.6080709@math.fu-berlin.de> References: <43FB7420.6080709@math.fu-berlin.de> Message-ID: <44347F15.6060006@math.fu-berlin.de> Christian Tismer wrote: > Hi, > > I have trouble with tismer.com, which happened to crash right now when > I'm > in the US. I hope to get in contact with service staff late nonight. > > Until then, please use this email address if you want to reach me. > > cheers - chris > This happened again, today! so, pls see above :-) From niko at alum.mit.edu Thu Apr 6 10:21:29 2006 From: niko at alum.mit.edu (Niko Matsakis) Date: Thu Apr 6 10:30:33 2006 Subject: [pypy-dev] Re: how to get into the codebase? Open bugs? In-Reply-To: <2mk6a4qi5t.fsf@starship.python.net> References: <33D3D202-84BC-44B6-83D4-B72615BDB231@alum.mit.edu> <2mk6a4qi5t.fsf@starship.python.net> Message-ID: > Well, as well as being in Switzerland right now, as Holger mentioned, > there will be sprints around EuroPython in early July, which is being > held at CERN this year. Thanks everyone; I may be able to swing by CERN, depending on how my schedule works out. In any case, I will look into some of those smaller issues and see if I can't contribute something. Niko From tismer at stackless.com Thu Apr 6 12:28:00 2006 From: tismer at stackless.com (Christian Tismer) Date: Thu Apr 6 12:28:25 2006 Subject: [pypy-dev] Re: how to get into the codebase? Open bugs? In-Reply-To: References: <33D3D202-84BC-44B6-83D4-B72615BDB231@alum.mit.edu> <2mk6a4qi5t.fsf@starship.python.net> Message-ID: <4434ED30.3090207@stackless.com> Niko Matsakis wrote: Dear Niko, > Thanks everyone; I may be able to swing by CERN, depending on how my > schedule works out. In any case, I will look into some of those smaller > issues and see if I can't contribute something. I'd like to encourage you.! At the same time, I'd like to point out that you are not envisioning a simple task, whqatevery you are trying with PyPy. But feel invited to ask me for assistance, any tine. ciao -- chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From anto.cuni at gmail.com Thu Apr 6 13:03:21 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu Apr 6 13:05:01 2006 Subject: [pypy-dev] A problem with unbound methods Message-ID: <4434F579.2030106@gmail.com> Hi, I have some problems for translating calls to unbound methods. Let's show with an example: class MyClass: def __init__(self, x): self.x = x class MyDerivedClass(MyClass): def __init__(self, x): MyClass.__init__(self, x) During rtyping the field and method names are mangled, so the __init__ method became something like 'o__init____variant0': as long as I call bound methods this is not a problem because the low-level op oosend contains the right mangled name, but difficult arises when I try to call an unbound method such as the one showed above; the low-level op that I obtain is this: v9 = direct_call((), self_1, x_2) Let 'x = op.args[0].value': (Pdb) x._name 'Base.__init__' (Pdb) x.graph.name 'Base.__init__' As you can see I can read only the original unmangled name, but it is useless because I've to deal with the mangled one. I've tried to search around for a place where to read that but I couldn't find it. How can I do? thanks for the help ciao Anto From pedronis at strakt.com Thu Apr 6 13:33:58 2006 From: pedronis at strakt.com (Samuele Pedroni) Date: Thu Apr 6 13:36:30 2006 Subject: [pypy-dev] A problem with unbound methods In-Reply-To: <4434F579.2030106@gmail.com> References: <4434F579.2030106@gmail.com> Message-ID: <4434FCA6.4010901@strakt.com> Antonio Cuni wrote: > Hi, > I have some problems for translating calls to unbound methods. > Let's show with an example: > > class MyClass: > def __init__(self, x): > self.x = x > > class MyDerivedClass(MyClass): > def __init__(self, x): > MyClass.__init__(self, x) > > During rtyping the field and method names are mangled, so the __init__ > method became something like 'o__init____variant0': as long as I call > bound methods this is not a problem because the low-level op oosend > contains the right mangled name, but difficult arises when I try to call > an unbound method such as the one showed above; the low-level op that I > obtain is this: > > v9 = direct_call(( at 0xb78e51ac>), self_1, x_2) > > Let 'x = op.args[0].value': > > (Pdb) x._name > 'Base.__init__' > (Pdb) x.graph.name > 'Base.__init__' > > As you can see I can read only the original unmangled name, but it is > useless because I've to deal with the mangled one. > I've tried to search around for a place where to read that but I > couldn't find it. How can I do? > you should not trust or use graph names in the backend, apart for givin names to things. If a function is reused in more than one class the information would not be useful (this can happen in Python/RPython). The graph would get the name based on the first class under which it was found, this may be unrelated for example for the class for self to the method name under which the graph is attached. Because there are too many variations about what is allowed in terms of supporting functions vs. just methods, calling superclass implementations of methods even when the method is overriden in a subclass etc in the targets, right now it is up to the backend to traverse and consider all classes and direct_calls and if the same graph appears both attached to a method (or methods) in a class (or classes) and in static method(s) in a direct call(s) decide what to do. This is also true in general for graphs that appear as more than on method in one place. Strategies here may vary from using a single static method (e.g. in Java) and having methods delegate to it, or if possible having just the method if the types of the first arguments in the direct call allow that approach ... regards From arigo at tunes.org Thu Apr 6 18:08:10 2006 From: arigo at tunes.org (Armin Rigo) Date: Thu Apr 6 18:08:12 2006 Subject: [pypy-dev] List and string in ootypesystem In-Reply-To: <44322B60.3050302@gmail.com> References: <442BC9CC.6010503@gmail.com> <442BF15B.9070102@gmx.ch> <442C2FB3.5070800@gmail.com> <44318B97.3050408@gmx.ch> <4431AC78.9010705@gmail.com> <44321B32.8000504@gmx.ch> <44322B60.3050302@gmail.com> Message-ID: <20060406160810.GA22957@code0.codespeak.net> Hi Antonio, On Tue, Apr 04, 2006 at 10:16:32AM +0200, Antonio Cuni wrote: > def ll_list_is_true(lst): > return lst is not None and len(lst) != 0 > > I hoped that the rtyper was smart enough to convert 'len(lst)' into my > low-level op 'list_len', but it wasn't: indeed, it generated code that > called len function for a generic PyObject* and that was not what I wanted. > I tried to copy the implementation of > lltypesystem.rlist.ll_list_is_true, but I couldn't because that can call > the ll_lenght() method that I didn't have. Now it has no longer > importance, but how could I do to get thing working? It is probably less confusing to call a method like .length() instead of using len() at this level again. But in both cases, you need to add support for this in the annotator and the rtyper again -- the ll_list_is_true() function is itself passed through both of them. I see you added SomeOOList to annoation.model.lltype_to_annotation(). There is already a generic 'def len()' in the base class SomeObject, so that's how the annotator is happy with your ll function's 'len(lst)'. Fine here. If you wanted a .length() method instead, you would need a 'def method_length()' in unaryop.py. On the rtyper side, you need something similar to rpython/rptr.py that maps SomeOOList back to its low-level type, with yet another Repr. It's this repr that must implement the operations you want to be able to use in low-level helpers; e.g. rtype_len() if you want len(lst) to work; or if instead you use .length() in low-level helpers, then you would need an rtype_method_length() in the repr corresponding to SomeOOList (by opposition to the rtype_len() in the repr corresponding to SomeList). Nik's approach is to map lists to reguar OO classes and instances, which are already supported in the annotator/rtyper with a similarly indirect approach: SomeOOClass/SomeOOInstance in the annotator, which rpython/ootypesystem/rootype.py maps back to the low-level OO types. Just like rptr.py, this rootype.py is only needed to support low-level helpers. A bientot, Armin From arigo at tunes.org Thu Apr 6 18:42:42 2006 From: arigo at tunes.org (Armin Rigo) Date: Thu Apr 6 18:42:42 2006 Subject: [pypy-dev] rctypes design document In-Reply-To: <44311BEE.7000109@klix.ch> References: <44311BEE.7000109@klix.ch> Message-ID: <20060406164242.GB22957@code0.codespeak.net> Hi Gerald, On Mon, Apr 03, 2006 at 02:58:22PM +0200, Gerald Klix wrote: > yesterday I checked in the final version of the rcytpes design document. > I am convinced I finally got everything right. Especially the management > of keep alive pointers. Thanks! We progressed on rctypes quite a bit, and indeed we are implementing a keepalive scheme very much inspired by the one you are describing. One difference (which we did not implemented so far) is that structures and arrays that contain several pointers will likely need one keepalive per pointer field or array item, to be able to keep all stored pointers' target memory alive. A bientot, Armin. From anto.cuni at gmail.com Thu Apr 6 21:31:04 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu Apr 6 21:32:52 2006 Subject: [pypy-dev] A problem with unbound methods In-Reply-To: <4434FCA6.4010901@strakt.com> References: <4434F579.2030106@gmail.com> <4434FCA6.4010901@strakt.com> Message-ID: <44356C78.7030409@gmail.com> Hi Samuele, Samuele Pedroni wrote: > you should not trust or use graph names in the backend, apart > for givin names to things. If a function is reused in more > than one class the information would not be useful (this can > happen in Python/RPython). > The graph would get the name based on the first class > under which it was found, this may be unrelated for example > for the class for self to the method name under which the graph > is attached. nice to know this, I didn't know. I think I have to rethink to my approach for code generation... > Because there are too many variations about what is allowed in terms > of supporting functions vs. just methods, calling superclass > implementations of methods even when the method is overriden > in a subclass etc in the targets, right now it is up to the backend > to traverse and consider all classes and direct_calls and if the same > graph appears both attached to a method (or methods) in a class (or > classes) and in static method(s) in a direct call(s) decide what to do. > This is also true in general for graphs that appear as more than on > method in one place. Ok, now it's clearer, thanks. So, to respond to my original question, I should create a sort of "graph database" to lookup when I need to know where have I put the code for that graph, right? Well, let's begin refactoring! :-) ciao Anto From anto.cuni at gmail.com Thu Apr 6 21:11:21 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu Apr 6 21:33:19 2006 Subject: [pypy-dev] List and string in ootypesystem In-Reply-To: <20060406160810.GA22957@code0.codespeak.net> References: <442BC9CC.6010503@gmail.com> <442BF15B.9070102@gmx.ch> <442C2FB3.5070800@gmail.com> <44318B97.3050408@gmx.ch> <4431AC78.9010705@gmail.com> <44321B32.8000504@gmx.ch> <44322B60.3050302@gmail.com> <20060406160810.GA22957@code0.codespeak.net> Message-ID: <443567D9.3090902@gmail.com> Hi Armin, hi Niklaus Armin Rigo wrote: > I see you added SomeOOList to annoation.model.lltype_to_annotation(). > There is already a generic 'def len()' in the base class SomeObject, so > that's how the annotator is happy with your ll function's 'len(lst)'. > Fine here. If you wanted a .length() method instead, you would need a > 'def method_length()' in unaryop.py. > > On the rtyper side, you need something similar to rpython/rptr.py that > maps SomeOOList back to its low-level type, with yet another Repr. It's > this repr that must implement the operations you want to be able to use > in low-level helpers; e.g. rtype_len() if you want len(lst) to work; or > if instead you use .length() in low-level helpers, then you would need > an rtype_method_length() in the repr corresponding to SomeOOList (by > opposition to the rtype_len() in the repr corresponding to SomeList). > > Nik's approach is to map lists to reguar OO classes and instances, which > are already supported in the annotator/rtyper with a similarly indirect > approach: SomeOOClass/SomeOOInstance in the annotator, which > rpython/ootypesystem/rootype.py maps back to the low-level OO types. > Just like rptr.py, this rootype.py is only needed to support low-level > helpers. Thank you for the great explanation: now I see things much more clearly, especially the role played by rptr.py and rootype.py, which I didn't understand very well before now. It seems that question by question I'm really getting into PyPy's logic... I hope I will be able to finish my apprenticeship soon :-) ciao Anto From mwh at python.net Thu Apr 6 23:21:43 2006 From: mwh at python.net (Michael Hudson) Date: Thu Apr 6 23:22:31 2006 Subject: [pypy-dev] Re: List and string in ootypesystem References: <442BC9CC.6010503@gmail.com> <442BF15B.9070102@gmx.ch> <442C2FB3.5070800@gmail.com> <44318B97.3050408@gmx.ch> <4431AC78.9010705@gmail.com> <44321B32.8000504@gmx.ch> <44322B60.3050302@gmail.com> <20060406160810.GA22957@code0.codespeak.net> <443567D9.3090902@gmail.com> Message-ID: <2mwte2pfh4.fsf@starship.python.net> Antonio Cuni writes: > It seems that question by question I'm really getting into PyPy's > logic... I hope I will be able to finish my apprenticeship soon :-) Unfortunately, I don't think you are Swiss, so you are probably doomed to being an apprentice forever like this rest of us :) More seriously, you seem to be doing very well! Hope you are having fun too :) Cheers, mwh -- M-x psych[TAB][RETURN] -- try it From stephan.diehl at gmx.net Fri Apr 7 18:16:39 2006 From: stephan.diehl at gmx.net (Stephan Diehl) Date: Fri Apr 7 18:17:09 2006 Subject: [pypy-dev] std objectspace interface Message-ID: <200604071816.39500.stephan.diehl@gmx.net> Hallo all, I have a question about the standard objectspace interface. It's about how to best return an iterator. While writing the 'set' implementation, it occured to me, that I'm using a W_SeqIterObject directly, which is bad, since this breaks the intended separation of types / objects. Instead, I'd like to use something like space.newseqiter. But 'newseqiter' takes a wrapped object, while I'd rather use a way to 'translate' some interp level iterator of wrapped objects directly to an application lever iterator. This would be similar to newlist which creates an application level list of application level objects out of an interp level list of application level objects. Or is the dictobject way the preferred one? There we have special iterator object 'W_DictIter_Keys', for example. I hope I make a bit of sense. Cheers Stephan From arigo at tunes.org Fri Apr 7 18:43:27 2006 From: arigo at tunes.org (Armin Rigo) Date: Fri Apr 7 18:43:30 2006 Subject: [pypy-dev] std objectspace interface In-Reply-To: <200604071816.39500.stephan.diehl@gmx.net> References: <200604071816.39500.stephan.diehl@gmx.net> Message-ID: <20060407164327.GA6853@code0.codespeak.net> Hi Stephan, On Fri, Apr 07, 2006 at 06:16:39PM +0200, Stephan Diehl wrote: > space.newseqiter. This is only for iterating over sequences, just like W_SeqIterObject. > Or is the dictobject way the preferred one? There we have special iterator > object 'W_DictIter_Keys', for example. Yes, you need a special iterator class like W_DictIter_Keys. BTW, W_SetObject.data, which is an r_dict, should ideally have 'None' as keys instead of space.w_True. This would make the low-level translated code more memory-efficient, by using 'void' values instead of pointers to space.w_True. A bientot, Armin. From stephan.diehl at gmx.net Fri Apr 7 18:50:58 2006 From: stephan.diehl at gmx.net (Stephan Diehl) Date: Fri Apr 7 18:51:28 2006 Subject: [pypy-dev] std objectspace interface In-Reply-To: <20060407164327.GA6853@code0.codespeak.net> References: <200604071816.39500.stephan.diehl@gmx.net> <20060407164327.GA6853@code0.codespeak.net> Message-ID: <200604071850.58566.stephan.diehl@gmx.net> On Friday 07 April 2006 18:43, Armin Rigo wrote: > Hi Stephan, > > On Fri, Apr 07, 2006 at 06:16:39PM +0200, Stephan Diehl wrote: > > space.newseqiter. > > This is only for iterating over sequences, just like W_SeqIterObject. > > > Or is the dictobject way the preferred one? There we have special > > iterator object 'W_DictIter_Keys', for example. > > Yes, you need a special iterator class like W_DictIter_Keys. o.k. > > BTW, W_SetObject.data, which is an r_dict, should ideally have 'None' as > keys instead of space.w_True. This would make the low-level translated > code more memory-efficient, by using 'void' values instead of pointers > to space.w_True. ahh, good point, I'll change that then (the other stuff, we talked about last week, is already changed, but not checked in) Thanks Stephan > > > A bientot, > > Armin. From bclum at cs.ucsd.edu Mon Apr 10 02:58:45 2006 From: bclum at cs.ucsd.edu (Brian C. Lum) Date: Mon Apr 10 02:59:38 2006 Subject: [pypy-dev] Printing a Flowgraph Message-ID: Dear Pypy developers: I was reading how you generate a flowgraph with Pypy from the getting-started page. I was just wondering if anyone knew how to print that graph once it displays. I don't really have much experience with Graphviz. Is there a way to save the graphviz file or print it to a .ps? http://codespeak.net/pypy/dist/pypy/doc/getting-started.html#trying-out-the-translator Thanks! Brian Just an FYI, if you want to try to follow the directions, they're a little bit outdated. To run it (after installing Pygame), start the interactive translator: cd pypy/bin python translator.py and to view a sample code: >>> t = Translation(test.is_perfect_number) >>> t.view() From mwh at python.net Mon Apr 10 09:57:31 2006 From: mwh at python.net (Michael Hudson) Date: Mon Apr 10 09:58:30 2006 Subject: [pypy-dev] Re: Printing a Flowgraph References: Message-ID: <2m7j5xq2vo.fsf@starship.python.net> "Brian C. Lum" writes: > Dear Pypy developers: > > I was reading how you generate a flowgraph with Pypy from the > getting-started page. I was just wondering if anyone knew how to > print that graph once it displays. > > I don't really have much experience with Graphviz. Is there a way to > save the graphviz file or print it to a .ps? Well, the dot file will probably be lurking in /tmp/usession-*/ somewhere. I don't know if there's an "official" way of getting hold of it, though. One you have the dot file, "dot -Tps flow.dot > flow.ps && lp flow.ps" or whatever should work fine. > http://codespeak.net/pypy/dist/pypy/doc/getting-started.html#trying-out-the-translator > > Thanks! > Brian > > Just an FYI, if you want to try to follow the directions, they're a > little bit outdated. To run it (after installing Pygame), start the > interactive translator: > > cd pypy/bin > python translator.py > > and to view a sample code: > >>>> t = Translation(test.is_perfect_number) >>>> t.view() I'm not sure what you're referring to. The docs you linked to look correct to me (i.e. there is no bin/translator.py any more in svn). Cheers, mwh -- we're already scrubbing the face of intuition with steel wool, setting it on fire, then putting it out with an axe . -- Tim Peters, on comparing recursive structures From bclum at cs.ucsd.edu Tue Apr 11 05:16:13 2006 From: bclum at cs.ucsd.edu (Brian C. Lum) Date: Tue Apr 11 05:16:57 2006 Subject: [pypy-dev] Re: Printing a Flowgraph In-Reply-To: <2m7j5xq2vo.fsf@starship.python.net> References: <2m7j5xq2vo.fsf@starship.python.net> Message-ID: Thanks a lot. Your way worked perfectly. I just downloaded version 0.8.0 of pypy. In that version, they use translation.py. I was not sure whether the document was referring to a newer or older version, so I just included the extra information in case. Brian On Mon, 10 Apr 2006, Michael Hudson wrote: > "Brian C. Lum" writes: > >> Dear Pypy developers: >> >> I was reading how you generate a flowgraph with Pypy from the >> getting-started page. I was just wondering if anyone knew how to >> print that graph once it displays. >> >> I don't really have much experience with Graphviz. Is there a way to >> save the graphviz file or print it to a .ps? > > Well, the dot file will probably be lurking in /tmp/usession-*/ > somewhere. I don't know if there's an "official" way of getting hold > of it, though. > > One you have the dot file, "dot -Tps flow.dot > flow.ps && lp flow.ps" > or whatever should work fine. > >> http://codespeak.net/pypy/dist/pypy/doc/getting-started.html#trying-out-the-translator >> >> Thanks! >> Brian >> >> Just an FYI, if you want to try to follow the directions, they're a >> little bit outdated. To run it (after installing Pygame), start the >> interactive translator: >> >> cd pypy/bin >> python translator.py >> >> and to view a sample code: >> >>>>> t = Translation(test.is_perfect_number) >>>>> t.view() > > I'm not sure what you're referring to. The docs you linked to look > correct to me (i.e. there is no bin/translator.py any more in svn). > > Cheers, > mwh > > From tismer at stackless.com Tue Apr 11 05:54:25 2006 From: tismer at stackless.com (Christian Tismer) Date: Tue Apr 11 05:56:38 2006 Subject: [pypy-dev] picxenk gallery :: logo :: 1 Message-ID: <443B2871.8050301@stackless.com> http://image.xenbio.net/logo/pypy_proto Heh, not bad :-) what do you think? From lac at strakt.com Tue Apr 11 12:14:31 2006 From: lac at strakt.com (Laura Creighton) Date: Tue Apr 11 12:15:12 2006 Subject: [pypy-dev] picxenk gallery :: logo :: 1 In-Reply-To: Message from Christian Tismer of "Mon, 10 Apr 2006 20:54:25 PDT." <443B2871.8050301@stackless.com> References: <443B2871.8050301@stackless.com> Message-ID: <200604111014.k3BAEV6p008124@theraft.strakt.com> In a message of Mon, 10 Apr 2006 20:54:25 PDT, Christian Tismer writes: >http://image.xenbio.net/logo/pypy_proto > >Heh, not bad :-) > >what do you think? >_______________________________________________ >pypy-dev@codespeak.net >http://codespeak.net/mailman/listinfo/pypy-dev I was always very fond of one of the rejected python.org logos Tim Parkin came up with. I include it as an attatchment. Tim also sent me some urls of other rejected python.org logos, and confirmed that if we want it, it is ours. so first the urls, but the one I like best is the attatchment ... http://www.artfiles.org/python.org/pub/beta.python.org/resources/design/history/ logoandtypeface.gif http://www.artfiles.org/python.org/pub/beta.python.org/resources/design/history/ master.png Laura -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 4217 bytes Desc: not available Url : http://codespeak.net/pipermail/pypy-dev/attachments/20060411/e3ce8c2d/attachment.gif From cfbolz at googlemail.com Tue Apr 11 16:18:39 2006 From: cfbolz at googlemail.com (Carl Friedrich Bolz) Date: Tue Apr 11 16:27:15 2006 Subject: [pypy-dev] Re: picxenk gallery :: logo :: 1 In-Reply-To: <200604111014.k3BAEV6p008124@theraft.strakt.com> References: <443B2871.8050301@stackless.com> <200604111014.k3BAEV6p008124@theraft.strakt.com> Message-ID: <348899050604110718j648491adw89355fc874a8dccc@mail.gmail.com> 2006/4/11, Laura Creighton : > In a message of Mon, 10 Apr 2006 20:54:25 PDT, Christian Tismer writes: > >http://image.xenbio.net/logo/pypy_proto > > > >Heh, not bad :-) > > > >what do you think? It has a very cool recursive aspect that I like. Not sure about colors and the placement of the letters. > I was always very fond of one of the rejected python.org logos Tim Parkin > came > up with. I include it as an attatchment. Tim also sent me some urls > of other rejected python.org logos, and confirmed that if we want it, it > is ours. > > so first the urls, but the one I like best is the attatchment ... This one is nice too, but kind of too generic. It has this typical swirly feeling that seems to me quite common. In my opinion none of those two comes close the very cool logo that we have already :-) Cheers, Carl Friedrich From tismer at stackless.com Tue Apr 11 20:41:01 2006 From: tismer at stackless.com (Christian Tismer) Date: Tue Apr 11 20:41:57 2006 Subject: [pypy-dev] Re: picxenk gallery :: logo :: 1 In-Reply-To: <348899050604110718j648491adw89355fc874a8dccc@mail.gmail.com> References: <443B2871.8050301@stackless.com> <200604111014.k3BAEV6p008124@theraft.strakt.com> <348899050604110718j648491adw89355fc874a8dccc@mail.gmail.com> Message-ID: <443BF83D.2020001@stackless.com> Carl Friedrich Bolz wrote: > It has a very cool recursive aspect that I like. Not sure about colors > and the placement of the letters. The recursivenes was what attracted me, too. > In my opinion none of those two comes close the very cool logo that we > have already :-) Maybe one could use the recursive idea a bit, but I agree. We should produce T-shirts, again. Maybe with a large PY on the back. ciao - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From michaelschneider at fuse.net Wed Apr 12 05:46:15 2006 From: michaelschneider at fuse.net (Michael Schneider) Date: Wed Apr 12 08:35:56 2006 Subject: [pypy-dev] pypy embedded / new atmel 32 Message-ID: <443C7807.5080309@fuse.net> Hello All, I have been very interested in pypy and embedded systems. I use atmel's line (cheap, easy architecture, gcc). It is still a bit cramped for pypy. Atmel just introduced their 32 bit line. It is a whopping $16 per unit. Paralles Propeller chip uses sping (some python linage). Would the new AVR be a candidate for pypy? Article http://www.linuxdevices.com/news/NS6110120875.html Data Sheets http://www.atmel.com/dyn/products/datasheets.asp?family_id=682 It would be great to have python on this chip. This could be used in so many domains... Thanks Mike From arigo at tunes.org Wed Apr 12 12:29:48 2006 From: arigo at tunes.org (Armin Rigo) Date: Wed Apr 12 12:29:49 2006 Subject: [pypy-dev] Mixed modules for both PyPy and CPython Message-ID: <20060412102948.GA9512@code0.codespeak.net> Hi all, This is a follow-up on last night's IRC discussion. It's come to a point where e-mail is better. The topic is "WP3", i.e. how to write in Python built-in modules that work for both CPython and PyPy ("built-in" == "extension" == "compileable" for the purpose of this mail). There are two levels of issues involved in this: * In which style the modules should be written in the first place? * Which style is easier to support in both CPython and PyPy given what we have now? In PyPy, we are using "mixed modules" with an interp-level and an app-level part, both optional, with explicit ways to interact between them. The interp-level part is what gets compiled to C. It contains code like 'space.add(w_1, w_2)' to perform an app-level addition. Christian is working on an alternative, which is a single-source approach where the annotator's failures to derive types are taken as a hint that the corresponding variables should contain wrapped objects. This is a convenient but hard-to-control extension of a historical hack, which works mostly because we needed it while we were still developing the annotator and the rtyper. I am clearly biased towards "solution 1", which is to reuse the mixed modules style that we have evolved over several years. Here is some implementation effort comparison between the two styles (1=mixed module, 2=single source). It is easy to develop and test in 2, as it runs on CPython with no further efforts. For 1, the module developer needs the whole of PyPy to test his complete module. We could ease this by writing a dummy interface-compatible gateway and object space, performing sanity-checks and giving useful error messages. Then the developer only needs to check his code against this. Annotation problems in the module are easier to spot early in the model 1; indeed, the fact that with 2 we cannot gracefully crash on SomeObjects, and moreover the need for many fragile-looking small extensions to the flow object space and the annotator, is what makes me most uneasy about 2. The mixed modules are designed for PyPy, so they work there already now. For the single-source approach to work on PyPy, however, there would be many interesting and delicate problems -- both for py.py and for the translated pypy-c. I'd rather avoid to think about it too much right now :-) For translation, for 2 we already have the basic machinery implemented as a leftover from PyPy "early ages". But I want to convince you that the basic support for 1 is extremely easy, or will soon be. We need a new object space to compile with the mixed module; but this "CPyObjSpace" is completely trivial. It could be based on rctypes, in which case it would look like this: class CPyObjSpace: def newint(self, intval): return ctypes.pydll.PyInt_FromLong(intval) def add(self, w_1, w_2): return ctypes.pydll.PyNumber_Add(w_1, w_2) ... Note that this even works on top of CPython! (Not out-of-the-box on Linux, however, where for obscure reasons CPython is by default not compiled as a separate .so file...). The gateway code can be written in the same style, by using ctypes.pydll.PyObject_Call() to call the app-level parts, and ctypes callbacks for the reverse direction. The calls like 'ctypes.pydll.PyInt_FromLong()' return an instance of 'ctypes.py_object', which naturally plays the role of a wrapped object for the CPyObjSpace. The more difficult bits of translation are e.g. support for creating at interp-level types that are exposed to app-level, in particular about the special methods __add__, __getitem__ etc. There is an example of that in pypy.module.Numeric.interp_numeric, where __getitem__ is added to the TypeDef of "array". This creates some difficulty that is common to approaches 1 and 2, and which I think Christian is also working on. In 1 we would probably make the TypeDef declaration turn, for CPython, into static PyTypeObject structures. The special methods can be recognized and put into the proper slots, with a bit of wrapping. The whole PyTypeObject structure with its pointers to functions could be expressed with ctypes too: # use pypy.rpython.rctypes.ctypes_platform to get at the layout PyTypeObject = ctypes_platform.getstruct("PyTypeObject *", ...) def TypeDef(name, **rawdict): mytype = PyTypeObject() mytype.tp_name = name if '__getitem__' in rawdict: # build a ctypes callback mp_subscript = callback_wrapper(rawdict['__dict__']) mytype.tp_as_mapping.mp_subscript = mp_subscript return mytype This gives us prebuilt PyTypeObject structures in ctypes, which get compiled into the C code by rctypes. At the same time, running on top of CPython is still possible; ctypes will let you issue the following kind of calls dynamically when running on CPython: myinstance = ctypes.pydll.PyObject_Call(mytype) The same line translated by rctypes becomes in the C file: PyObject* myinstance = PyObject_Call(mytype); Of course this latter part is dependent on rctypes being completed first. I understand and respect Christian's needs for something that works right now; nevertheless, at this point I think the mixed modules approach is the best medium-term solution. This is all open to discussion, of course. ...but do keep in mind that the people who completed the annotator as it is now are not likely to appreciate the addition of Yet More Ad-Hoc Heuristics all over the place :-/ A bientot, Armin From arigo at tunes.org Wed Apr 12 13:48:19 2006 From: arigo at tunes.org (Armin Rigo) Date: Wed Apr 12 13:48:21 2006 Subject: [pypy-dev] Mixed modules for both PyPy and CPython In-Reply-To: <20060412102948.GA9512@code0.codespeak.net> References: <20060412102948.GA9512@code0.codespeak.net> Message-ID: <20060412114819.GA18540@code0.codespeak.net> Re-hi all, A clarification on a point that caused confusion... The purpose of the whole discussion was about how to have a "write-once, run-everywhere" approach of developing an extension module a single time and then compiling it for either CPython or PyPy. This source code should not need to know if it will be compiled for PyPy or CPython (or even just run on top of the CPython interpreter for testing). The two approaches I opposed are two ways to do that. Some confusion came from the names of the two approaches, which I took from IRC. One approach is "single-source": this just means "source-in-a-single-file", with wrapped objects and native values mixed in the same source code. (It's not in the sense "write a single time the source"; both approaches are about this.) By opposition, "mixed module" means "source-in-two-files", one running at interp-level and one at app-level. This is what we're doing in pypy/module/*/{app,interp}_*.py. Holger suggests to call "single-source" the whole approach of writing once and running everywhere, which makes sense. If we do so, then we need better names for the two approaches. What about "explicit" versus "implicit" levels? Mixed modules provide explicit level separation, whereas the alternative is to rely on the annotator to separate the levels. I'd also like to point out that the latter -- implicit levels -- has a kind of elegance to it that I like. What I dislike, though, beyond the fact that it would require yet another refactoring of PyPy's module approach (and in a way that is quite unclear), is that it requires many additional fragile rules and heuristics all over the flow objspace and the annotator to get the "expected" separation of levels effects. I'd be happy to experiment more in that direction, but I believe that the other direction's work is needed for now and won't be lost anyway. Basically, we can still come back to this and think later about ways to allow more implicit-level manipulations of objects. A bientot, Armin. From arigo at tunes.org Wed Apr 12 15:37:01 2006 From: arigo at tunes.org (Armin Rigo) Date: Wed Apr 12 15:37:02 2006 Subject: [pypy-dev] A problem with unbound methods In-Reply-To: <44356C78.7030409@gmail.com> References: <4434F579.2030106@gmail.com> <4434FCA6.4010901@strakt.com> <44356C78.7030409@gmail.com> Message-ID: <20060412133701.GA25572@code0.codespeak.net> Hi Antonio, On Thu, Apr 06, 2006 at 09:31:04PM +0200, Antonio Cuni wrote: > Ok, now it's clearer, thanks. So, to respond to my original question, I > should create a sort of "graph database" to lookup when I need to know > where have I put the code for that graph, right? We could try to do this in a way that is reusable between multiple back-ends, because many OO back-ends will have a similar problem. What needs to be done is to collect all graphs and see how they are used. For each graph, it means: which regular method(s) are implemented with that graph, and which direct_calls are done to staticmethods implemented with that graph. Then we need some cases. We can be more sutble, but as a minimum we need logic like: if a graph is used only as one method, leave it alone. If it is used only as a static method, leave it alone too (and declare it as a static method somewhere). For more complicated cases, find all methods implemented with this graph, and replace each of them with a new stub graph that just contains a direct_call to the original graph as a static method. Such a transformation would "normalize" the methods in the OOClass objects (so that the approach you're currently using in CLI would continue to work for methods) and also generate a list of remaining graphs that the back-end needs to declare as static methods. A bientot, Armin. From anto.cuni at gmail.com Wed Apr 12 17:03:44 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed Apr 12 17:06:07 2006 Subject: [pypy-dev] A problem with unbound methods In-Reply-To: <20060412133701.GA25572@code0.codespeak.net> References: <4434F579.2030106@gmail.com> <4434FCA6.4010901@strakt.com> <44356C78.7030409@gmail.com> <20060412133701.GA25572@code0.codespeak.net> Message-ID: <443D16D0.3030600@gmail.com> Hi Armin, Armin Rigo wrote: > We could try to do this in a way that is reusable between multiple > back-ends, because many OO back-ends will have a similar problem. What > needs to be done is to collect all graphs and see how they are used. > For each graph, it means: which regular method(s) are implemented with > that graph, and which direct_calls are done to staticmethods implemented > with that graph. > > Then we need some cases. We can be more sutble, but as a minimum we > need logic like: if a graph is used only as one method, leave it alone. > If it is used only as a static method, leave it alone too (and declare > it as a static method somewhere). For more complicated cases, find all > methods implemented with this graph, and replace each of them with a new > stub graph that just contains a direct_call to the original graph as a > static method. > > Such a transformation would "normalize" the methods in the OOClass > objects (so that the approach you're currently using in CLI would > continue to work for methods) and also generate a list of remaining > graphs that the back-end needs to declare as static methods. It sounds reasonable. At the moment by backend suffers just this problem: is a graph is used both as bound and unbound method it will be rendered twice. I though to resolve it in some CLI specific way, but a more general approach is better. By now the few logic I do is localized in translator/cli/database.py, but it is too CLI specific to be reusable: maybe I could try to factor out the backend-independent logic for later reuse. Another good thing to factor out could be the "pending nodes" logic: at the moment both gencli and gensqueak implements that logic independently, but it would be nice to have a reusable framework shared by gensqueak, gencli and possibly other backends. ciao Anto From "news-8a9e0fd91190ca" at northportal.net Wed Apr 12 20:43:57 2006 From: "news-8a9e0fd91190ca" at northportal.net (VanL) Date: Wed Apr 12 20:57:10 2006 Subject: [pypy-dev] Re: Mixed modules for both PyPy and CPython In-Reply-To: <20060412102948.GA9512@code0.codespeak.net> References: <20060412102948.GA9512@code0.codespeak.net> Message-ID: Hello, There is one other issue which may be relevant: restricted python. Restricted python implementations recently showed up on the py3k list as a wishlist item - and they are wanted even for python 2.X. I thought that pypy might be the answer to these restricted python wishes. Implementation would be as follows: For all allowed functionality, create a compilable pypy extension to handle it. For restricted functionality, delegate to a CPyObjectSpace which could allow/disallow or modify the operation. What makes this work where CPy couldn't is the ability to use two different cooperating object spaces to evaluate an expression. CPy doesn't have the object space abstraction which would allow this sort of partitioning. Another benefit is that different restricted interpreters could be created, each with a different set of allowed and disallowed operations. In summary: (compile a restricted PyPy - rpy - as a CPy extension here) >>> import rpy >>> def filehandler(fname, mode): ... if fname in ['allowed1.txt', 'allowed2.txt']: ... return open(fname, mode) ... else: ... raise rpy.Restricted('File access not permitted') ... >>> interp = rpy.new() # restrictions were defined at compile time >>> # Add a callback for some restricted functionality >>> interp.add_handler('file', filehandler) >>> interp.interact() >>>> >>>> # Now in rpy >>>> 2 + 3 # Allowed operation, didn't hit any restrictions 5 >>>> >>>> open('allowed1.txt', 'r') # allowed by handler >>>> >>>> open('disallowed.txt', 'r') # disallowed by handler Traceback (most recent call last): File "", line 1, in ? __main__.__interp__.Restricted: File access not permitted Would either of these approaches lend itself better to this sort of restricted execution idea? VanL From cfbolz at gmx.de Wed Apr 12 21:13:46 2006 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Wed Apr 12 21:14:34 2006 Subject: [pypy-dev] Re: Mixed modules for both PyPy and CPython In-Reply-To: References: <20060412102948.GA9512@code0.codespeak.net> Message-ID: <348899050604121213w1f01c58pb65b4e70394f6deb@mail.gmail.com> Hi! 2006/4/12, VanL : > There is one other issue which may be relevant: restricted python. > Restricted python implementations recently showed up on the py3k list as > a wishlist item - and they are wanted even for python 2.X. Careful. "Restricted Python" (or RPython) means something very specific in PyPy lingo. Restricted Python means that you stick to some restrictions in your coding style to make type inference possible. See http://codespeak.net/pypy/dist/pypy/doc/coding-guide.html#restricted-python for more details. To your mail: security features are part of what we promised to do to the EU, so there will be something happening in that direction (even before November). I personally don't have any clue what :-) [snip] Cheers, Carl Friedrich From "news-8a9e0fd91190ca" at northportal.net Wed Apr 12 21:45:51 2006 From: "news-8a9e0fd91190ca" at northportal.net (VanL) Date: Wed Apr 12 21:47:04 2006 Subject: [pypy-dev] Re: Mixed modules for both PyPy and CPython In-Reply-To: <348899050604121213w1f01c58pb65b4e70394f6deb@mail.gmail.com> References: <20060412102948.GA9512@code0.codespeak.net> <348899050604121213w1f01c58pb65b4e70394f6deb@mail.gmail.com> Message-ID: Hello, Carl Friedrich Bolz wrote: > Careful. "Restricted Python" (or RPython) means something very > specific in PyPy lingo. Restricted Python means that you stick to some > restrictions in your coding style to make type inference possible. See > > http://codespeak.net/pypy/dist/pypy/doc/coding-guide.html#restricted-python > > Sorry - I knew that. I just didn't make the connection from one side of my brain to another. Maybe "secure python" (spy)? What made me think this would work are a couple of comments made here and on the py3k list. First, Armin (about a year ago, I think)made some comments about the objectspace abstraction allowing a "Remote Object Space" and allowing different object spaces to participate in the evaluation of code. Second, comments on py3k list indicated that secure python is difficult because of a) introspection, b) type inference, and c) GIL acquisition. However, a) a partitioned object space would allow introspection to be limited - go too far and you run into an opaque objectspace wall; b) Pypy already has functionality to do some type inference - probably as much as necessary; and c) Pypy doesn't have a GIL. The only relevant GIL would be in the CPy host python, inaccessible to code in the Spy. Of course, I also may be completely wrong :) VanL From vlindberg at novell.com Wed Apr 12 21:33:10 2006 From: vlindberg at novell.com (Van Lindberg) Date: Wed Apr 12 23:41:09 2006 Subject: [pypy-dev] Re: Mixed modules for both PyPy and CPython In-Reply-To: <348899050604121213w1f01c58pb65b4e70394f6deb@mail.gmail.com> References: <20060412102948.GA9512@code0.codespeak.net> <348899050604121213w1f01c58pb65b4e70394f6deb@mail.gmail.com> Message-ID: <443D55F6.10501@novell.com> Hello, Carl Friedrich Bolz wrote: >Careful. "Restricted Python" (or RPython) means something very >specific in PyPy lingo. Restricted Python means that you stick to some >restrictions in your coding style to make type inference possible. See > >http://codespeak.net/pypy/dist/pypy/doc/coding-guide.html#restricted-python > > Sorry - I knew that. I just didn't make the connection from one side of my brain to another. Maybe "secure python" (spy)? What made me think this would work are a couple of comments made here and on the py3k list. First, Armin (about a year ago, I think)made some comments about the objectspace abstraction allowing a "Remote Object Space" and allowing different object spaces to participate in the evaluation of code. Second, comments on py3k list indicated that secure python is difficult because of a) introspection, b) type inference, and c) GIL acquisition. However, a) a partitioned object space would allow introspection to be limited - go too far and you run into an opaque objectspace wall; b) Pypy already has functionality to do some type inference - probably as much as necessary; and c) Pypy doesn't have a GIL. The only relevant GIL would be in the CPy host python, inaccessible to code in the Spy. Of course, I also may be completely wrong :) VanL From bclum at cs.ucsd.edu Thu Apr 13 03:33:13 2006 From: bclum at cs.ucsd.edu (Brian C. Lum) Date: Thu Apr 13 03:34:16 2006 Subject: [pypy-dev] Information from the FlowObjSpace Message-ID: Dear Pypy developers: I have gone through the source code for the FlowObjSpace in pypy.objspace.flow.objspace, but I am confused how to traverse through the blocks or obtain the information for each blocks in the graph. From my understanding: space = FlowObjSpace() graph = space.build_flow(func) Once you have the graph, however, how do you know what instructions are in each block? I can iterate through the graph with iterblocks, but how do I get information from each block? I want to analyze the information in each block to do code analysis for python. Can anyone help me with this? With thanks, Brian From tismer at stackless.com Thu Apr 13 04:39:41 2006 From: tismer at stackless.com (Christian Tismer) Date: Thu Apr 13 04:40:49 2006 Subject: [pypy-dev] Information from the FlowObjSpace In-Reply-To: References: Message-ID: <443DB9ED.8020406@stackless.com> Brian C. Lum wrote: > Dear Pypy developers: > > I have gone through the source code for the FlowObjSpace in > pypy.objspace.flow.objspace, but I am confused how to traverse through > the blocks or obtain the information for each blocks in the graph. From > my understanding: > > space = FlowObjSpace() > graph = space.build_flow(func) > > Once you have the graph, however, how do you know what instructions are > in each block? I can iterate through the graph with iterblocks, but how > do I get information from each block? > > I want to analyze the information in each block to do code analysis for > python. Can anyone help me with this? First of all, flowing doesn't work for full CPython. You need to use the RPython subset (see http://codespeak.net/pypy/dist/pypy/doc/coding-guide.html#restricted-python ) Then, if you have a block, you can iterate over block.operations which is a list, and so on. See pypy/objspace/flow/model.py ciao - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From bclum at cs.ucsd.edu Thu Apr 13 08:14:50 2006 From: bclum at cs.ucsd.edu (Brian C. Lum) Date: Thu Apr 13 08:15:33 2006 Subject: [pypy-dev] Information from the FlowObjSpace In-Reply-To: <443DB9ED.8020406@stackless.com> References: <443DB9ED.8020406@stackless.com> Message-ID: Hi, Thank you for the reply. I have looked at model.py and I can make a FunctionGraph. However, I am still not sure how to use it. space = FlowObjSpace() graph = space.build_flow(test.is_perfect_number) fun = FunctionGraph(test.is_perfect_number, graph.startblock) That is how I create an instance of FunctionGraph, and I can still make the blocks into an iterator, but I still cannot see what is inside the blocks, i.e. what would be the code in the function graph. Basically, all I want is a flowgraph of the code so that I can try to perform some static analysis on it (I want to bound the length of lists at compile time). I cannot figure out how to access the instructions in each basic block from the flowgraph. Calling translate_as_module from pypy.translator.geninterplevel as described in http://codespeak.net/pypy/dist/pypy/doc/translation.html#example actually produces the SSA. I was hoping that I would be able to get the instructions of each block from the flowgraph like in the example. Brian On Wed, 12 Apr 2006, Christian Tismer wrote: > Brian C. Lum wrote: >> Dear Pypy developers: >> >> I have gone through the source code for the FlowObjSpace in >> pypy.objspace.flow.objspace, but I am confused how to traverse through the >> blocks or obtain the information for each blocks in the graph. From my >> understanding: >> >> space = FlowObjSpace() >> graph = space.build_flow(func) >> >> Once you have the graph, however, how do you know what instructions are in >> each block? I can iterate through the graph with iterblocks, but how do I >> get information from each block? >> >> I want to analyze the information in each block to do code analysis for >> python. Can anyone help me with this? > > First of all, flowing doesn't work for full CPython. You need > to use the RPython subset (see > http://codespeak.net/pypy/dist/pypy/doc/coding-guide.html#restricted-python > ) > > Then, if you have a block, you can iterate over block.operations > which is a list, and so on. See pypy/objspace/flow/model.py > > ciao - chris > From mwh at python.net Thu Apr 13 11:17:56 2006 From: mwh at python.net (Michael Hudson) Date: Thu Apr 13 11:18:50 2006 Subject: [pypy-dev] Re: Information from the FlowObjSpace References: <443DB9ED.8020406@stackless.com> Message-ID: <2m3bghn8aj.fsf@starship.python.net> "Brian C. Lum" writes: > Hi, > > Thank you for the reply. I have looked at model.py and I can make a > FunctionGraph. However, I am still not sure how to use it. > > space = FlowObjSpace() > graph = space.build_flow(test.is_perfect_number) > fun = FunctionGraph(test.is_perfect_number, graph.startblock) > > That is how I create an instance of FunctionGraph, and I can still > make the blocks into an iterator, but I still cannot see what is > inside the blocks, i.e. what would be the code in the function graph. > > Basically, all I want is a flowgraph of the code so that I can try to > perform some static analysis on it (I want to bound the length of > lists at compile time). I cannot figure out how to access the > instructions in each basic block from the flowgraph. > for block in graph.iterblocks(): for op in block.operations: print op Cheers, mwh -- In the 1950s and 60s there was a regular brain drain of young Australians from the cities to London, but it was because of money, culture and opportunity, not spiders. -- Al Grant, ucam.chat, from Owen Dunn's review of the year From hpk at trillke.net Thu Apr 13 15:38:39 2006 From: hpk at trillke.net (holger krekel) Date: Thu Apr 13 15:44:03 2006 Subject: [pypy-dev] security prototype & workshop plans/ideas Message-ID: <20060413133839.GF10825@solar.trillke> Hi folks, from the EU side of things there is the plan to organize a security workshop and implement security features within PyPy. I now talked to my IBM research lab contact in Zuerich, and we discussed a few possibilities for such a workshop and the prototype, in particular two possibilities: - data tagging or "label control", or more generally attaching (security) metainformations to a python object and having those propagate through the program automatically. See e.g. http://www.pmg.lcs.mit.edu/papers/myers-popl99.pdf for some more information in this direction. This is especially interesting for PyPy as - i think - we can probably do better by e.g. also allowing to tag primitive types (JFlow doesn't do that, i think) and allowing things to happen more dynamically at runtime. - sandboxing: allowing to execute (user-supplied/induced) code and preventing it from accessing critical resources, e.g. system IO. Label control could be used for tagging e.g. user-level input with the "untrusted" label and then protecting certain functions to require trusted input (e.g. database/file modifications). Then, there could be explicit untrusted_to_trusted() conversions, turning an untrusted input into a trusted output. This would allow to concisely localise how user-supplied/untrusted input is parsed and checked. I also guess we could also go for a more general metadata-tagging approach which might be useful in more than just the security context. This are just initial ideas - there are arbitrary other possibilities (somewhere care to discuss the "E" lang here?), or are there not? :) The challenge is to find an interesting mechanism that elegantly enables such approaches - which should be the topic of our upcoming security prototype and workshop. Sandboxing and/or e.g. limiting RAM/CPU usage are also interesting to the Python world but probably are a rather orthogonal matter. Regarding the timing of the workshop it seems that something like October this year fits best. It depends on our actual plan and availability/selection of workshop participants if we rather should try to meet in Zuerich or in New York. We can also try to first arrange a discussion with them and then a meeting for presenting the results but that may be a bit much for us (we are not really lacking challenges currently :). I am posting here on pypy-dev (rather than just to selected pypy developers) because others may be interested, have comments, suggestions or might think about contributing. Security is certainly not the central topic of PyPy but our design should make it considerably easier to implement strong security features. Hum, and i guess that it's not impossible that the project might for contributors come up with funding for travels at least. best, holger From lac at strakt.com Thu Apr 13 20:07:59 2006 From: lac at strakt.com (Laura Creighton) Date: Thu Apr 13 20:08:46 2006 Subject: [pypy-dev] Re: [Edu-sig] Visual Programming in Python? In-Reply-To: Message from "Douglas S. Blank" of "Thu, 13 Apr 2006 12:56:12 EDT." <1144947372.13112.114.camel@mightymouse.brynmawr.edu> References: <1144947372.13112.114.camel@mightymouse.brynmawr.edu> Message-ID: <200604131807.k3DI7xU3017413@theraft.strakt.com> This just showed up on edu-sig. Is the translator and viewer separate enough that we could just send it to this person? Laura In a message of Thu, 13 Apr 2006 12:56:12 EDT, "Douglas S. Blank" writes: >Python edu-sig, > >I've been thinking about a visual programming/flowchart interface for >Python and was wondering if anyone knows of such a project. I am imaging >a Tkinter canvas (initially) with which one can add blocks that >represent statements, branches, loops... everything. Further, I imaging >that this would save (and load) real Python code, so that you could suck >in raw code, it would get parsed, and shown as a flowchart. Maybe some >additional data would be stored in comments (zoom amount, >positions/colors/properties of particular boxes). Also, one could step >through the chart, block-by-block. > >I've seen some commercial (and open source/non-Python) products, but >they seem heavy and sluggish, as if a whole lot of processing is going >on behind the scenes. Is/would it really be that hard? > >Any pointers or comments appreciated, > >-Doug > >-- >Douglas S. Blank Computer Science >Assistant Professor Bryn Mawr College >(610)526-6501 http://cs.brynmawr.edu/~dblank > > >_______________________________________________ >Edu-sig mailing list >Edu-sig@python.org >http://mail.python.org/mailman/listinfo/edu-sig From anto.cuni at gmail.com Fri Apr 14 16:53:28 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Fri Apr 14 16:56:04 2006 Subject: [pypy-dev] Avoiding code duplication Message-ID: <443FB768.7060604@gmail.com> Hi all, in these days I'm completing the implementation of ootypesystem/rlist.py, but I've found that often I have to write code very similar to those in lltypesystem/rlist.py. Let's explain by an example; consider the ll_listindex function in both files: # lltypesystem def ll_listindex(lst, obj, eqfn): items = lst.ll_items() lng = lst.ll_length() j = 0 while j < lng: if eqfn is None: if items[j] == obj: return j else: if eqfn(items[j], obj): return j j += 1 raise ValueError # can't say 'list.index(x): x not in list' # ootypesystem def ll_listindex(lst, obj, eqfn): lng = lst.length() j = 0 while j < lng: if eqfn is None: if lst.getitem_nonneg(j) == obj: return j else: if eqfn(lst.getitem_nonneg(j), obj): return j j += 1 raise ValueError # can't say 'list.index(x): x not in list' As you can see they are quite similar, but I can't find an elegant way to merge them. There are two problems: 1) the names of some methods don't match (e.g., ll_length vs. length 2) the setitem/getitem interface is very different The first problem is easy to solve; it should be sufficient to rename the methods in ootype.List._GENERIC_METHODS. The same isn't true for the second problem; one possibility could be to pass some dummy placeholder as argument in the same style of dum_checkidx/dum_nocheck, but I guess the code will became a nightmare. Another solution could be to add an extra level of indirection but I guess this could bring to some efficiency penalty. Any idea? ciao Anto From arigo at tunes.org Fri Apr 14 17:22:27 2006 From: arigo at tunes.org (Armin Rigo) Date: Fri Apr 14 17:22:28 2006 Subject: [pypy-dev] Avoiding code duplication In-Reply-To: <443FB768.7060604@gmail.com> References: <443FB768.7060604@gmail.com> Message-ID: <20060414152227.GA29635@code0.codespeak.net> Hi Antonio, On Fri, Apr 14, 2006 at 04:53:28PM +0200, Antonio Cuni wrote: > # lltypesystem > def ll_listindex(lst, obj, eqfn): > items = lst.ll_items() > (...) > if items[j] == obj: > > # ootypesystem > def ll_listindex(lst, obj, eqfn): > (...) > if lst.getitem_nonneg(j) == obj: > Another solution could be to add an extra level of indirection but I > guess this could bring to some efficiency penalty. I think this is kind-of-reasonable. The ADT method approach of the lltypesystem was introduced late during the development of the rtyper; by now, it would be reasonable to define common method names between the ADT methods of the lltypesystem and the GENERIC_METHODS of the ootypesystem. I am unsure about the performance penalty. The current version of many ll helpers, for example, read the 'items' pointer only once and reuse it; if this gets replaced by ADT methods like 'getitem_nonneg()', it means that althought the call is probably inlined there is still the overhead of reading 'items' through each iteration in the list. Who knows, maybe C compilers will notice and move the read out of the loop. Just give it a try on a small example like ll_listindex(), I guess... A different comment: as you mentioned on IRC it would be nice if the back-end could choose which methods it implements natively. At one point there was the idea that maybe the 'oopspec' attributes that started to show up in lltypesystem/rlist.py (used by the JIT only) could be useful in this respect. If I remember correctly, the idea didn't work out because of the different 'lowleveltype' needed, and the difference in the interface. Merging the ADT method names of lltyped lists and the GENERIC_METHODS of ootyped lists could be a step in this direction again. The interesting point is that each oo back-end could then choose to special-case the ll_xxx() functions with the oopspecs that they recognize, and just translate the other ones normally. (The ll back-ends always translate them all.) A bientot, Armin. From pedronis at strakt.com Fri Apr 14 17:52:34 2006 From: pedronis at strakt.com (Samuele Pedroni) Date: Fri Apr 14 17:53:27 2006 Subject: [pypy-dev] Avoiding code duplication In-Reply-To: <20060414152227.GA29635@code0.codespeak.net> References: <443FB768.7060604@gmail.com> <20060414152227.GA29635@code0.codespeak.net> Message-ID: <443FC542.3090707@strakt.com> Armin Rigo wrote: >Hi Antonio, > >On Fri, Apr 14, 2006 at 04:53:28PM +0200, Antonio Cuni wrote: > > >># lltypesystem >>def ll_listindex(lst, obj, eqfn): >> items = lst.ll_items() >>(...) >> if items[j] == obj: >> >># ootypesystem >>def ll_listindex(lst, obj, eqfn): >>(...) >> if lst.getitem_nonneg(j) == obj: >> >> > > > >>Another solution could be to add an extra level of indirection but I >>guess this could bring to some efficiency penalty. >> >> > >I think this is kind-of-reasonable. The ADT method approach of the >lltypesystem was introduced late during the development of the rtyper; >by now, it would be reasonable to define common method names between the >ADT methods of the lltypesystem and the GENERIC_METHODS of the >ootypesystem. > >I am unsure about the performance penalty. The current version of many >ll helpers, for example, read the 'items' pointer only once and reuse >it; if this gets replaced by ADT methods like 'getitem_nonneg()', it >means that althought the call is probably inlined there is still the >overhead of reading 'items' through each iteration in the list. Who >knows, maybe C compilers will notice and move the read out of the loop. >Just give it a try on a small example like ll_listindex(), I guess... > >A different comment: as you mentioned on IRC it would be nice if the >back-end could choose which methods it implements natively. At one >point there was the idea that maybe the 'oopspec' attributes that >started to show up in lltypesystem/rlist.py (used by the JIT only) could >be useful in this respect. If I remember correctly, the idea didn't >work out because of the different 'lowleveltype' needed, > yes, the problem was not re-using code as such, indeed it should not be hard to write generic code that can be shared using adt methods etc. The problem was trying to share the same type between ootype and lltype, Instances are not Ptrs and don't even behave similarly enough, this is where trying to reuse completely the lltypesystem rlist and rtuple code broke. > and the >difference in the interface. Merging the ADT method names of lltyped >lists and the GENERIC_METHODS of ootyped lists could be a step in this >direction again. The interesting point is that each oo back-end could >then choose to special-case the ll_xxx() functions with the oopspecs >that they recognize, and just translate the other ones normally. (The >ll back-ends always translate them all.) > > >A bientot, > >Armin. >_______________________________________________ >pypy-dev@codespeak.net >http://codespeak.net/mailman/listinfo/pypy-dev > > From hpk at trillke.net Fri Apr 14 19:44:12 2006 From: hpk at trillke.net (holger krekel) Date: Fri Apr 14 19:49:44 2006 Subject: [pypy-dev] Mixed modules for both PyPy and CPython In-Reply-To: <20060412114819.GA18540@code0.codespeak.net> References: <20060412102948.GA9512@code0.codespeak.net> <20060412114819.GA18540@code0.codespeak.net> Message-ID: <20060414174412.GL10825@solar.trillke> Hi Armin, hi all, On Wed, Apr 12, 2006 at 13:48 +0200, Armin Rigo wrote: > The purpose of the whole discussion was about how to have a "write-once, > run-everywhere" approach of developing an extension module a single time > and then compiling it for either CPython or PyPy. This source code > should not need to know if it will be compiled for PyPy or CPython (or > even just run on top of the CPython interpreter for testing). > > ... > Mixed modules provide explicit level separation, > whereas the alternative is to rely on the annotator to separate the > levels. Hum, i wonder how strongly opposed these explicit versus implicit level separation models need to be. Is it not possible to support a programming model that can mostly avoid knowing about interpreter versus application level distinction without extending/refactoring the annotator? Maybe we could identify some interesting interaction use cases and try to support them explicitely like e.g. publishing an rpython level class in CPython providing glue code between the rpython class and its CPython representation? However, having an underlying interp/app separation with 'space', wrapped objects and exceptions, still seems like a very good (proven) foundation on which to build such glue code. (Btw, i wouldn't mind if such glue code would not allow all possible interactions - our primary goal is not to provide seemless integration with CPython here). And for the planned June PyPy release we likely want to have the "explicit" approach nicely working before experimenting with where we can go from there, right? best, holger From nhaldimann at gmx.ch Fri Apr 14 21:58:25 2006 From: nhaldimann at gmx.ch (Niklaus Haldimann) Date: Fri Apr 14 21:59:02 2006 Subject: [pypy-dev] Summer of Code 2006 Message-ID: <443FFEE1.5060007@gmx.ch> Hi there Google is doing Summer of Code again this year: http://code.google.com/soc/ It would be possible to enter PyPy directly as a mentoring organization this time, instead of going through the PSF. Last year, student slots were given to mentoring organizations proportional to the number of applications. If there are enough applications for PyPy proper that might bring more students on board than taking slots from the PSF pool as last year (but maybe PyPy's influence in the PSF is big enough, and this doesn't really matter). What is the status of the Summer of PyPy idea, btw? Nik From hpk at trillke.net Fri Apr 14 21:57:47 2006 From: hpk at trillke.net (holger krekel) Date: Fri Apr 14 22:03:19 2006 Subject: [pypy-dev] Summer of Code 2006 In-Reply-To: <443FFEE1.5060007@gmx.ch> References: <443FFEE1.5060007@gmx.ch> Message-ID: <20060414195747.GM10825@solar.trillke> Hi Niklaus, On Fri, Apr 14, 2006 at 21:58 +0200, Niklaus Haldimann wrote: > Google is doing Summer of Code again this year: http://code.google.com/soc/ > > It would be possible to enter PyPy directly as a mentoring organization > this time, instead of going through the PSF. Last year, student slots > were given to mentoring organizations proportional to the number of > applications. If there are enough applications for PyPy proper that > might bring more students on board than taking slots from the PSF pool > as last year (but maybe PyPy's influence in the PSF is big enough, and > this doesn't really matter). sounds good. > What is the status of the Summer of PyPy idea, btw? it hasn't been pursued much lately because it ran concurrent to many other issues we (and i) needed to sort out. There was considerable interest, though, something like 10-15 people mailed me and the list here. best, holger From hpk at trillke.net Sat Apr 15 09:29:55 2006 From: hpk at trillke.net (holger krekel) Date: Sat Apr 15 09:35:32 2006 Subject: [pypy-dev] Re: Mixed modules for both PyPy and CPython In-Reply-To: References: <20060412102948.GA9512@code0.codespeak.net> <348899050604121213w1f01c58pb65b4e70394f6deb@mail.gmail.com> Message-ID: <20060415072955.GO10825@solar.trillke> Hi VanL, On Wed, Apr 12, 2006 at 13:45 -0600, VanL wrote: > Carl Friedrich Bolz wrote: > > > Careful. "Restricted Python" (or RPython) means something very > > specific in PyPy lingo. Restricted Python means that you stick to some > > restrictions in your coding style to make type inference possible. See > > > > > http://codespeak.net/pypy/dist/pypy/doc/coding-guide.html#restricted-python > > > > > > Sorry - I knew that. I just didn't make the connection from one side of > my brain to another. Maybe "secure python" (spy)? just as a side note: "secure" is a very vague term, technically (and politically). The professor i learned with, warned against talking about security without specifying against which kind of attacks and about which scenarios one is talking about it. Did you see my security related posting a few days ago, btw? > What made me think this would work are a couple of comments made here > and on the py3k list. > First, Armin (about a year ago, I think)made some comments about the > objectspace abstraction allowing a "Remote Object Space" and allowing > different object spaces to participate in the evaluation of code. Right, that is still an open topic, a bit touched by the parallel discussion on this list of integrating PyPy (and RPython) with CPython. > Second, comments on py3k list indicated that secure python is difficult > because of a) introspection, b) type inference, and c) GIL acquisition. Hum, this list looks a bit weird to me. Could you state what the actual attacks are for which security measures are discussed? Or which use cases are people on py3k having in mind? cheers & thanks, holger From anto.cuni at gmail.com Sat Apr 15 09:46:16 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Sat Apr 15 09:48:49 2006 Subject: [pypy-dev] Summer of Code 2006 In-Reply-To: <443FFEE1.5060007@gmx.ch> References: <443FFEE1.5060007@gmx.ch> Message-ID: <4440A4C8.7030207@gmail.com> Niklaus Haldimann wrote: > Hi there > > Google is doing Summer of Code again this year: http://code.google.com/soc/ > > It would be possible to enter PyPy directly as a mentoring organization > this time, instead of going through the PSF. Last year, student slots > were given to mentoring organizations proportional to the number of > applications. If there are enough applications for PyPy proper that > might bring more students on board than taking slots from the PSF pool > as last year (but maybe PyPy's influence in the PSF is big enough, and > this doesn't really matter). That's a good news! I'd like to participate to soc for doing pypy's stuffs, I hope I'll able to apply (and to win :-)). ciao Anto From anto.cuni at gmail.com Sat Apr 15 12:59:07 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Sat Apr 15 13:01:49 2006 Subject: [pypy-dev] Avoiding code duplication In-Reply-To: <20060414152227.GA29635@code0.codespeak.net> References: <443FB768.7060604@gmail.com> <20060414152227.GA29635@code0.codespeak.net> Message-ID: <4440D1FB.8000800@gmail.com> Armin Rigo wrote: > > I think this is kind-of-reasonable. The ADT method approach of the > lltypesystem was introduced late during the development of the rtyper; > by now, it would be reasonable to define common method names between the > ADT methods of the lltypesystem and the GENERIC_METHODS of the > ootypesystem. > > I am unsure about the performance penalty. The current version of many > ll helpers, for example, read the 'items' pointer only once and reuse > it; if this gets replaced by ADT methods like 'getitem_nonneg()', it > means that althought the call is probably inlined there is still the > overhead of reading 'items' through each iteration in the list. Who > knows, maybe C compilers will notice and move the read out of the loop. > Just give it a try on a small example like ll_listindex(), I guess... Well, as we decided on #pypy I've changed the ADT interface. As I wrote in the commit log: """ The interface of ListRepr and FixedSizeListRepr has changed: two accessor methods has been added: ll_getitem_fast and ll_setitem_fast. They should be used instead of the ll_items()[index] idiom: that way when ootypesystem's list will support that interface we will able to write function useable with both typesystem with no modification. The various ll_* helper function has been adapted to use the new interface. Moreover function that accessed directly to the "l.length" field has been changed to call the "ll_length()" method instead, for the same reasons as above. """ The next step is to rename ootypesystem's list _GENERIC_METHODS to match the ADT methods in lltypesystem's list, then we could try to share most of ll_* function that currently belongs only to lltypesystem/rlist.py. I hope I will do it tomorrow. > A different comment: as you mentioned on IRC it would be nice if the > back-end could choose which methods it implements natively. At one > point there was the idea that maybe the 'oopspec' attributes that > started to show up in lltypesystem/rlist.py (used by the JIT only) could > be useful in this respect. If I remember correctly, the idea didn't > work out because of the different 'lowleveltype' needed, and the > difference in the interface. Merging the ADT method names of lltyped > lists and the GENERIC_METHODS of ootyped lists could be a step in this > direction again. The interesting point is that each oo back-end could > then choose to special-case the ll_xxx() functions with the oopspecs > that they recognize, and just translate the other ones normally. (The > ll back-ends always translate them all.) I saw that 'oopspec' attributes, but I didn't understand the exact semantic; your proposal sounds reasonable to me: if I can figure out correctly this way the typesystem specific code would be reduced to the minimum and will help to port other Repr such as rdict to ootypesystem, too. I'll investigate a bit in this direction as soon as I can. good Easter to all, ciao Anto From arigo at tunes.org Sat Apr 15 14:59:39 2006 From: arigo at tunes.org (Armin Rigo) Date: Sat Apr 15 14:59:41 2006 Subject: [pypy-dev] Avoiding code duplication In-Reply-To: <4440D1FB.8000800@gmail.com> References: <443FB768.7060604@gmail.com> <20060414152227.GA29635@code0.codespeak.net> <4440D1FB.8000800@gmail.com> Message-ID: <20060415125939.GA29013@code0.codespeak.net> Hi Antonio, On Sat, Apr 15, 2006 at 12:59:07PM +0200, Antonio Cuni wrote: > I saw that 'oopspec' attributes, but I didn't understand the exact > semantic; The oopspec string tells what is the "abstract" list operation that this particular ll_*() function implement. For example: def ll_prepend(l, newitem): ... ll_prepend.oopspec = 'list.insert(l, 0, newitem)' means that ll_prepend() is equivalent to an insert with the index set to zero. In the stirng, the pseudo-arguments between the ( ) are either real argument names of the ll_ function, or constants. So for example, if a backend has got its own way to implement the insert() calls in general, it could figure out from the oopspec that the ll_prepend() helper can be replaced by a custom stub invoking the backend's own version of insertion with an index of 0. That's essentially what the JIT does -- see handle_highlevel_operation() in jit/hintannotator/model.py. A bientot, Armin. From arigo at tunes.org Sat Apr 15 15:09:09 2006 From: arigo at tunes.org (Armin Rigo) Date: Sat Apr 15 15:09:10 2006 Subject: [pypy-dev] Mixed modules for both PyPy and CPython In-Reply-To: <20060414174412.GL10825@solar.trillke> References: <20060412102948.GA9512@code0.codespeak.net> <20060412114819.GA18540@code0.codespeak.net> <20060414174412.GL10825@solar.trillke> Message-ID: <20060415130909.GB29013@code0.codespeak.net> Hi Holger, On Fri, Apr 14, 2006 at 07:44:12PM +0200, holger krekel wrote: > Hum, i wonder how strongly opposed these explicit versus implicit > level separation models need to be. Yes, you're right here. It's mostly about what we need to do next: we must choose one of the two models and develop it enough, until it becomes useful for PyPy and CPython alike. We could possibly do both models in parallel, but I'm not sure it's the best way forward at this point. > Is it not possible to support a > programming model that can mostly avoid knowing about interpreter versus > application level distinction without extending/refactoring the annotator? Sure, no-one thinks about refactoring the annotator. The implicit model already works, by using the existing support for SomeObjects, completed by Christian over the time. It's a bit hackish, though, and we'll definitely need ways to control where SomeObjects are expected or not. At the moment, what makes me reluctant to continue with the implicit model are two other issues: on the one hand, it's unclear how it would work for PyPy (it works for CPython only); and there are many language design issues ahead that I'd rather avoid for the time being. > "explicit" approach nicely working before experimenting with where > we can go from there, right? Yes, exactly my opinion. > (Btw, i wouldn't > mind if such glue code would not allow all possible interactions - > our primary goal is not to provide seemless integration with CPython here). I'm not too worried about this. Our mixed-module model already supports mostly any kind of interaction, including defining new types with properties and overridden operations. The path to support the same for CPython extension modules is more or less clear, and very incremental. A bientot, Armin. From hpk at trillke.net Sat Apr 15 15:10:05 2006 From: hpk at trillke.net (holger krekel) Date: Sat Apr 15 15:15:42 2006 Subject: [pypy-dev] ext module testing modes In-Reply-To: <20060415110215.DF0CF100AC@code0.codespeak.net> References: <20060415110215.DF0CF100AC@code0.codespeak.net> Message-ID: <20060415131005.GA2505@solar.trillke> Hi Armin, On Sat, Apr 15, 2006 at 13:02 +0200, arigo@codespeak.net wrote: > Author: arigo > Date: Sat Apr 15 13:02:14 2006 > New Revision: 25852 > > Added: > pypy/dist/pypy/rpython/rctypes/socketmodule/test_addr.py (contents, props changed) > Modified: > pypy/dist/pypy/rpython/rctypes/socketmodule/_socket.py > pypy/dist/pypy/rpython/rctypes/socketmodule/ctypes_socket.py > Log: > Very incomplete implementation of getaddrinfo(), with a test > (only works on on-line machines so far). The idea is that rctypes > should now support all ctypes constructions that were necessary. > I will start a regular mixed-module _socket based on this, but > first we need to figure out how to best test mixed-modules based > on ctypes -- ideally, they should be testable and compilable > without the rest of the PyPy interpreter... regarding py.test support: I think eventually we may have the following testing distinctions regarding ext modules: - test mixed module with std objspace (running on top of cpython) (current default) - test mixed module with cpy-objspace connected to CPython runtime via rctypes - test mixed module on top of pypy-c I guess the second testing mode could be specified by "--objspace=cpy" and for the third we may simply allow to specify "--appdirect" which would trigger application level tests to run directly on the executable instead through pypy interpreter indirection. (with "--exec=/path/to/executable" you can already point to pypy-c but PyPy does not support enough for py.test to run this way). makes sense? holger From hpk at trillke.net Sat Apr 15 15:58:09 2006 From: hpk at trillke.net (holger krekel) Date: Sat Apr 15 16:03:47 2006 Subject: [pypy-dev] Mixed modules for both PyPy and CPython In-Reply-To: <20060415130909.GB29013@code0.codespeak.net> References: <20060412102948.GA9512@code0.codespeak.net> <20060412114819.GA18540@code0.codespeak.net> <20060414174412.GL10825@solar.trillke> <20060415130909.GB29013@code0.codespeak.net> Message-ID: <20060415135809.GT10825@solar.trillke> Hi Armin, On Sat, Apr 15, 2006 at 15:09 +0200, Armin Rigo wrote: > On Fri, Apr 14, 2006 at 07:44:12PM +0200, holger krekel wrote: > > Hum, i wonder how strongly opposed these explicit versus implicit > > level separation models need to be. > > Yes, you're right here. It's mostly about what we need to do next: we > must choose one of the two models and develop it enough, until it > becomes useful for PyPy and CPython alike. We could possibly do both > models in parallel, but I'm not sure it's the best way forward at this > point. > > > Is it not possible to support a > > programming model that can mostly avoid knowing about interpreter versus > > application level distinction without extending/refactoring the annotator? > > Sure, no-one thinks about refactoring the annotator. The implicit model > already works, by using the existing support for SomeObjects, completed > by Christian over the time. It's a bit hackish, though, and we'll > definitely need ways to control where SomeObjects are expected or not. > At the moment, what makes me reluctant to continue with the implicit > model are two other issues: on the one hand, it's unclear how it would > work for PyPy (it works for CPython only); and there are many language > design issues ahead that I'd rather avoid for the time being. I agree but may have a somewhat different idea in mind when talking about a more implicit model: namely assuming that all objects live within the current RPython model (no SomeObject's whatsoever) and providing explicit interactions (like gateway.interp2app), exposing of type definitions etc. > > "explicit" approach nicely working before experimenting with where > > we can go from there, right? > > Yes, exactly my opinion. > > > (Btw, i wouldn't > > mind if such glue code would not allow all possible interactions - > > our primary goal is not to provide seemless integration with CPython here). > > I'm not too worried about this. Our mixed-module model already supports > mostly any kind of interaction, including defining new types with > properties and overridden operations. The path to support the same for > CPython extension modules is more or less clear, and very incremental. Yes, the mixed modules (and interpreter/gateway's, typedef's) support interaction but by rather explicitely programming the machinery. With "glue code" i mean code where the user does not need to know about such machinery so much. IOW, the question is which implicit models (as seen from the ext module programmer) are possible without having SomeObjects around? holger From arigo at tunes.org Sat Apr 15 18:23:46 2006 From: arigo at tunes.org (Armin Rigo) Date: Sat Apr 15 18:23:49 2006 Subject: [pypy-dev] Mixed modules for both PyPy and CPython In-Reply-To: <20060415135809.GT10825@solar.trillke> References: <20060412102948.GA9512@code0.codespeak.net> <20060412114819.GA18540@code0.codespeak.net> <20060414174412.GL10825@solar.trillke> <20060415130909.GB29013@code0.codespeak.net> <20060415135809.GT10825@solar.trillke> Message-ID: <20060415162346.GA6978@code0.codespeak.net> Hi Holger, On Sat, Apr 15, 2006 at 03:58:09PM +0200, holger krekel wrote: > I agree but may have a somewhat different idea in mind when > talking about a more implicit model: (...) Ok, then we agree everywhere -- short of a confusing terminology: we gave the names "implicit levels" and "explicit levels" to very precise things and now you're calling "an implicit model" something that is inbetween :-) A bientot, Armin From arigo at tunes.org Sat Apr 15 18:34:29 2006 From: arigo at tunes.org (Armin Rigo) Date: Sat Apr 15 18:34:30 2006 Subject: [pypy-dev] Mixed modules for both PyPy and CPython In-Reply-To: <20060412102948.GA9512@code0.codespeak.net> References: <20060412102948.GA9512@code0.codespeak.net> Message-ID: <20060415163429.GA7927@code0.codespeak.net> Re-hi, On Wed, Apr 12, 2006 at 12:29:48PM +0200, Armin Rigo wrote: > class CPyObjSpace: > > def newint(self, intval): > return ctypes.pydll.PyInt_FromLong(intval) > > def add(self, w_1, w_2): > return ctypes.pydll.PyNumber_Add(w_1, w_2) > ... Correction here. It's not 'ctypes.pydll' -- this one is an object used to load other DLLs following the CPython calling convensions. It's 'ctypes.pythonapi' instead, and it works on standard Linux installations as well. Just try: >>> from ctypes import * >>> pythonapi.PyNumber_Add.restype = py_object >>> pythonapi.PyNumber_Add(py_object(5), py_object(6)) 11 (The result is 11 instead of py_object(11) because ctypes does automatic unwrapping of some return types.) A bientot, Armin From arigo at tunes.org Sat Apr 15 18:39:12 2006 From: arigo at tunes.org (Armin Rigo) Date: Sat Apr 15 18:39:13 2006 Subject: [pypy-dev] ext module testing modes In-Reply-To: <20060415131005.GA2505@solar.trillke> References: <20060415110215.DF0CF100AC@code0.codespeak.net> <20060415131005.GA2505@solar.trillke> Message-ID: <20060415163912.GB7927@code0.codespeak.net> Hi Holger, On Sat, Apr 15, 2006 at 03:10:05PM +0200, holger krekel wrote: > - test mixed module with std objspace (running on top of cpython) > (current default) > > - test mixed module with cpy-objspace connected to CPython > runtime via rctypes > > - test mixed module on top of pypy-c > > makes sense? Sure. Note that for testing we could also reintroduce a trivial object space -- ouch! wait! don't hit! I think this would be far simpler than the previous "trivial" space because it wouldn't try to be nice with the PyPy interpreter at all; it would always use the underlying interpreter. For the same reason I expect the CPyObjSpace to look like our initial trivial object space as well, without growing all the complexities. Hopefully. Armin From hpk at trillke.net Sat Apr 15 19:03:31 2006 From: hpk at trillke.net (holger krekel) Date: Sat Apr 15 19:09:08 2006 Subject: [pypy-dev] Mixed modules for both PyPy and CPython In-Reply-To: <20060415162346.GA6978@code0.codespeak.net> References: <20060412102948.GA9512@code0.codespeak.net> <20060412114819.GA18540@code0.codespeak.net> <20060414174412.GL10825@solar.trillke> <20060415130909.GB29013@code0.codespeak.net> <20060415135809.GT10825@solar.trillke> <20060415162346.GA6978@code0.codespeak.net> Message-ID: <20060415170331.GV10825@solar.trillke> On Sat, Apr 15, 2006 at 18:23 +0200, Armin Rigo wrote: > On Sat, Apr 15, 2006 at 03:58:09PM +0200, holger krekel wrote: > > I agree but may have a somewhat different idea in mind when > > talking about a more implicit model: (...) > > Ok, then we agree everywhere -- short of a confusing terminology: we > gave the names "implicit levels" and "explicit levels" to very precise > things and now you're calling "an implicit model" something that is > inbetween :-) indeed, i wasn't quite explicit enough, i guess :) IMO "implicit" and "explicit" do not denote a binary property but there rather can be quantities of "implicit" or "explicit", therefore the "more" in "more implicit model". holger From bokr at oz.net Sun Apr 16 00:03:45 2006 From: bokr at oz.net (Bengt Richter) Date: Sun Apr 16 00:04:41 2006 Subject: [pypy-dev] type inference info derived from test suites for the tested? Message-ID: <5.0.2.1.1.20060415131620.00a7dd90@mail.oz.net> Please excuse if this is OT, but I was wondering about the subject topic, i.e., using a test code suite (run with special-command-line-option?) as a kind of substitute for pragmas within a language e.g. defining a function, and winding up with some intermediate representation of the constrained function that can be used in the compilation and linking of an application that does not include the function test suite per se, yet makes use of the implicit info from the test code (per command line option that caused the compiler's info transfer to the test-info-constrained-function's intermediate representation). Maybe this could also be viewed as a kind of global optimization using combined test and application code, but factoring out and pre-caching some results of separated function+test_function inferences, and editing out test code from the final linking. I guess this is trying to use test code as a general source of auxiliary information using inference as a way to be independent of particular languages' ability to express explicit type declarations or contracts etc. TIA for any pointers/urls where I might read about this kind of thing, and whether it's a dead end or a trail being explored, either in pypy or elsewhere. Regards, Bengt Richter From "news-8a9e0fd91190ca" at northportal.net Sun Apr 16 01:38:30 2006 From: "news-8a9e0fd91190ca" at northportal.net (VanL) Date: Sun Apr 16 01:39:25 2006 Subject: [pypy-dev] Re: Mixed modules for both PyPy and CPython In-Reply-To: <20060415072955.GO10825@solar.trillke> References: <20060412102948.GA9512@code0.codespeak.net> <348899050604121213w1f01c58pb65b4e70394f6deb@mail.gmail.com> <20060415072955.GO10825@solar.trillke> Message-ID: Hello, holger krekel wrote: >> Second, comments on py3k list indicated that secure python is difficult >> because of a) introspection, b) type inference, and c) GIL acquisition. > > Hum, this list looks a bit weird to me. Could you state what > the actual attacks are for which security measures are discussed? > Or which use cases are people on py3k having in mind? This is an amalgam of several different posts (and maybe different threads) but here goes: In the thread "Will we have a true restricted exec environment for python 3000," Vineet Jain asked for a restricted mode which would "1. Limit the memory consumed by the script 2. Limit access to file system and other system resources 3. Limit cpu time that the script will take 4. Be able to specify which modules are available for import." In responses to that request, various people commented on the difficulties of implementing such a restricted mode. On that thread, several people had the same idea I had, to try to use PyPy for this purpose - however, it didn't look like many people were up-to-date reading both lists (and thus familiar-ish with PyPy's execution model). A) Introspection Nick Coghlan stated that: "I'm interested, but I'm also aware of how much work it would be. I'm disinclined to trust any mechanism which allows the untrusted code to run in the same process, as the implications of being able to do: self.__class__.__mro__[-1].__subtypes__() are somewhat staggering, and designing an in-process sandbox to cope with that is a big ask (and demonstrating that the sandbox actually *achieves* that goal is even tougher)." Vineet volunteered with a proposal to start a "light" python subinterpreter, which would be controlled by the main interpreter. Nick countered, "But will it allow you to use numbers or strings? If yes, then you can get to object(), and hence to pretty much whatever C builtins you want. So its not enough to try to hide dangerous builtins like file(), you want to remove them from the light version entirely (routing all file system and network access requests through the main application). But if the file objects are gone, what happens to the Python machinery that relies on them (like import)? Python's powerful introspection is a severe drawback from a security POV - it is *really* hard to make a user stay in a box you put them in without crippling some part of the language as a side effect." Thus, in CPy, allowing someone to access a C type effectively opens up all the C types. In PyPy, however, each type is effectively in its own box. Further, PyPy already has a structure that can deal with these sorts of accesses: the flowgraph. Operations in PyPy come about because of traversals of the graph - certain branches of the graph could be restricted or proxied out to a trusted interpreter. B) GIL Acquisition Another person suggested leveraging the multiple subinterpreter code which already exists in CPython to create a restricted-exec interpreter. MvL noted that GIL acquisition made that difficult: "Part of the problem is that it doesn't really work. Some objects *are* shared across interpreters, such as global objects in extension modules (extension modules are initialized only once). I believe that the GIL management code (for acquiring the GIL out of nowhere) breaks if there are multiple interpreters." C) Type inference I tried to find the thread for this one - its not from the Py3K list - but I recall a couple years ago someone attempting to make an rexec version of python. One of the comments that I recall from that discussion had to do with understanding what types were being manipulated. I believe there was an example somewhat like operator.add is trusted class A: def __add__(self, other): ... something evil here ... a, b = A(), 1 a + b [something evil happens] However, this is a foggy memory that I have so far been unable to substantiate. Thanks, VanL From hpk at trillke.net Sun Apr 16 22:44:43 2006 From: hpk at trillke.net (holger krekel) Date: Sun Apr 16 22:50:29 2006 Subject: [pypy-dev] Re: Mixed modules for both PyPy and CPython In-Reply-To: References: <20060412102948.GA9512@code0.codespeak.net> <348899050604121213w1f01c58pb65b4e70394f6deb@mail.gmail.com> <20060415072955.GO10825@solar.trillke> Message-ID: <20060416204443.GA10825@solar.trillke> Hi VanL! On Sat, Apr 15, 2006 at 17:38 -0600, VanL wrote: > holger krekel wrote: > > >>Second, comments on py3k list indicated that secure python is difficult > >>because of a) introspection, b) type inference, and c) GIL acquisition. > > > >Hum, this list looks a bit weird to me. Could you state what > >the actual attacks are for which security measures are discussed? > >Or which use cases are people on py3k having in mind? > > This is an amalgam of several different posts (and maybe different > threads) but here goes: hey, thanks for the effort! > In the thread "Will we have a true restricted exec environment for > python 3000," Vineet Jain asked for a restricted mode which would > > "1. Limit the memory consumed by the script > 2. Limit access to file system and other system resources > 3. Limit cpu time that the script will take > 4. Be able to specify which modules are available for import." all more or less relates to the "sandbox" idea. > In responses to that request, various people commented on the > difficulties of implementing such a restricted mode. On that thread, > several people had the same idea I had, to try to use PyPy for this > purpose - however, it didn't look like many people were up-to-date > reading both lists (and thus familiar-ish with PyPy's execution model). Yes, using PyPy's metaprogramming facilities for implementing sandbox models should make the tasks relatively easy. "Metaprogramming" because PyPy is about writing a program that generates programs which happen to be a full Python interpreter. But it's also true that we haven't had concise discussions (or rather never documented the results :) about how to implement the above. But the fact that we can instrument our GC, resource handling code, or the main interpreter loop and accordingly produce a full python interpreter gives us various ways to go about the sandbox problem. Funny little thesis topic, i'd say :) > A) Introspection > ... introspection funnyness ... > Python's powerful introspection is a severe drawback from a security POV > - it is *really* hard to make a user stay in a box you put them in > without crippling some part of the language as a side effect." All the possible ways to introspect your way around the python object model makes one wonder if protecting the resources isn't a more viable approach than protecting navigation. > Thus, in CPy, allowing someone to access a C type effectively opens up > all the C types. In PyPy, however, each type is effectively in its own > box. Further, PyPy already has a structure that can deal with these > sorts of accesses: the flowgraph. Operations in PyPy come about because > of traversals of the graph - certain branches of the graph could be > restricted or proxied out to a trusted interpreter. Applying systematic transformations to families of graphs (relating to IO resources, say) is one possibility. Lately we seem to want to express almost everything as graph transformations, btw. > B) GIL Acquisition > > Another person suggested leveraging the multiple subinterpreter code > which already exists in CPython to create a restricted-exec interpreter. > MvL noted that GIL acquisition made that difficult: > > "Part of the problem is that it doesn't really work. Some objects *are* > shared across interpreters, such as global objects in extension modules > (extension modules are initialized only once). I believe that the GIL > management code (for acquiring the GIL out of nowhere) breaks if there > are multiple interpreters." For PyPy, it need not be true that the GIL makes protecting resources harder. Then again, it doesn't matter so much because i don't think that we'll require a sub-interpreter for implementing resource protection (but who knows :) I guess we should discuss sometime on which path to follow (also see my other mail). Any opinions on that, btw? best and thanks! holger From mwh at python.net Mon Apr 17 12:44:01 2006 From: mwh at python.net (Michael Hudson) Date: Mon Apr 17 12:44:57 2006 Subject: [pypy-dev] Re: Summer of Code 2006 References: <443FFEE1.5060007@gmx.ch> Message-ID: <2mmzeklbwu.fsf@starship.python.net> Niklaus Haldimann writes: > Hi there > > Google is doing Summer of Code again this year: http://code.google.com/soc/ > > It would be possible to enter PyPy directly as a mentoring organization > this time, instead of going through the PSF. Last year, student slots > were given to mentoring organizations proportional to the number of > applications. If there are enough applications for PyPy proper that > might bring more students on board than taking slots from the PSF pool > as last year (but maybe PyPy's influence in the PSF is big enough, and > this doesn't really matter). If things are this way again, I suspect that PyPy would get more SoC slots by being part of the PSF (it's also less work :). In either case, I suspect the bottleneck would be finding sufficient mentors. Cheers, mwh -- This is not to say C++ = bad, Lisp = good. It's to say C++ = bad irrespective of everything else. -- Alain Picard, comp.lang.lisp From hpk at trillke.net Mon Apr 17 13:13:08 2006 From: hpk at trillke.net (holger krekel) Date: Mon Apr 17 13:18:58 2006 Subject: [pypy-dev] type inference info derived from test suites for the tested? In-Reply-To: <5.0.2.1.1.20060415131620.00a7dd90@mail.oz.net> References: <5.0.2.1.1.20060415131620.00a7dd90@mail.oz.net> Message-ID: <20060417111308.GD10825@solar.trillke> Hi Bengt, On Sat, Apr 15, 2006 at 15:03 -0700, Bengt Richter wrote: > Please excuse if this is OT, but I was wondering about the subject topic, > i.e., using a test code suite (run with special-command-line-option?) > as a kind of substitute for pragmas within a language e.g. defining a function, > and winding up with some intermediate representation of the constrained function > that can be used in the compilation and linking of an application that does > not include the function test suite per se, yet makes use of the implicit > info from the test code (per command line option that caused the compiler's info > transfer to the test-info-constrained-function's intermediate representation). As far as PyPy's type inference is concerned: it starts from one entry point and follows all possible code paths, annotating function arguments etc. with possible type(s). If you are thinking about a set of disparate API functions (and not a coherent program) then tests could maybe be used to provide entry points. > Maybe this could also be viewed as a kind of global optimization using > combined test and application code, but factoring out and pre-caching > some results of separated function+test_function inferences, and editing > out test code from the final linking. > > I guess this is trying to use test code as a general source of auxiliary > information using inference as a way to be independent of particular > languages' ability to express explicit type declarations or contracts etc. Hum, well PyPy neccessarily does this already for transforming (R)Python programs to lower level representations (and does not make use of tests). Otherwise i doubt that the usual tests are exhaustive enough to implicitly allow defining constraints on function arguments. > TIA for any pointers/urls where I might read about this kind of thing, and > whether it's a dead end or a trail being explored, either in pypy or elsewhere. i take it you have read PyPy's documentation ... nothing else i can currently point to. cheers, holger From anto.cuni at gmail.com Mon Apr 17 14:45:15 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Mon Apr 17 14:48:04 2006 Subject: [pypy-dev] Avoiding code duplication In-Reply-To: <20060414152227.GA29635@code0.codespeak.net> References: <443FB768.7060604@gmail.com> <20060414152227.GA29635@code0.codespeak.net> Message-ID: <44438DDB.6010100@gmail.com> Hi Armin, hi all Armin Rigo wrote: > A different comment: as you mentioned on IRC it would be nice if the > back-end could choose which methods it implements natively. At one > point there was the idea that maybe the 'oopspec' attributes that > started to show up in lltypesystem/rlist.py (used by the JIT only) could > be useful in this respect. If I remember correctly, the idea didn't > work out because of the different 'lowleveltype' needed, and the > difference in the interface. Merging the ADT method names of lltyped > lists and the GENERIC_METHODS of ootyped lists could be a step in this > direction again. The interesting point is that each oo back-end could > then choose to special-case the ll_xxx() functions with the oopspecs > that they recognize, and just translate the other ones normally. (The > ll back-ends always translate them all.) I have successfully moved almost all ll_* helpers from rpython/lltypesystem/rlist.py to rpython/rlist.py so that now they are shared between the two typesystems. For doing that I had to extend the ADT interface of ListRepr to include _ll_resize_ge and _ll_resize_le, as well as to add the same function as _GENERIC_METHODS in ootypesystem. Now ootypesystem should support the full list's semantic but in a far-from-perfect way: the main issue is that most operations are done by the ll_* helpers, even when there could be a more efficient native equivalent. I see three main ways of fixing that: 1) further extend the ADT/_GENERIC_METHODS interface to include common operations that could be natively available in OO targets and modify ll_* helpers to use that methods; e.g., we could insert a "remove_range" and let ll_listdelslice & co. using it. This way OO backends could easily forward call to remove_range to the runtime; moreover if the inliner works well there should be no performance penalties for lltypesystem; 2) follow Armin's suggestion to use oopspec for letting backends forwarding to the RTE only what they want; 3) a mixture of 1 and 2. ciao Anto From tjreedy at udel.edu Mon Apr 17 20:37:07 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Mon Apr 17 20:38:31 2006 Subject: [pypy-dev] Re: Summer of Code 2006 References: <443FFEE1.5060007@gmx.ch> <2mmzeklbwu.fsf@starship.python.net> Message-ID: "Michael Hudson" wrote in message news:2mmzeklbwu.fsf@starship.python.net... > If things are this way again, I suspect that PyPy would get more SoC > slots by being part of the PSF (it's also less work :). > > In either case, I suspect the bottleneck would be finding sufficient > mentors. Neal Norwitz just posted this on py-dev: --------------- We've only got a short time to get setup for Google's Summer of Code. We need to start identifying mentors and collecting ideas for students to implement. We have the SimpleTodo list (http://wiki.python.org/moin/SimpleTodo), but nothing on the SoC page yet (http://wiki.python.org/moin/SummerOfCode). I can help manage the process from inside Google, but I need help gathering mentors and ideas. I'm not certain of the process, but if you are interested in being a mentor, send me an email. I will try to find all the necessary info and post here again tomorrow. Pass the word! I hope all mentors from last year will return again this year. Can someone take ownership of drumming up mentors and ideas? We also need to spread the word to c.l.p and beyond. --------- tjr From jacob at strakt.com Mon Apr 17 22:50:00 2006 From: jacob at strakt.com (Jacob =?iso-8859-1?q?Hall=E9n?=) Date: Mon Apr 17 22:50:47 2006 Subject: [pypy-dev] Re: Summer of Code 2006 In-Reply-To: <2mmzeklbwu.fsf@starship.python.net> References: <443FFEE1.5060007@gmx.ch> <2mmzeklbwu.fsf@starship.python.net> Message-ID: <200604172250.00561.jacob@strakt.com> m?ndagen den 17 april 2006 12.44 skrev Michael Hudson: > Niklaus Haldimann writes: > > Hi there > > > > Google is doing Summer of Code again this year: > > http://code.google.com/soc/ > > > > It would be possible to enter PyPy directly as a mentoring organization > > this time, instead of going through the PSF. Last year, student slots > > were given to mentoring organizations proportional to the number of > > applications. If there are enough applications for PyPy proper that > > might bring more students on board than taking slots from the PSF pool > > as last year (but maybe PyPy's influence in the PSF is big enough, and > > this doesn't really matter). > > If things are this way again, I suspect that PyPy would get more SoC > slots by being part of the PSF (it's also less work :). > > In either case, I suspect the bottleneck would be finding sufficient > mentors. This is a list of plausible candidates: Armin Samuele Carl Michael Arre Anders L Aurelien Ludovic Adrien Holger Eric I think the only other ones who are familiar enough with the codebase to be mentors are Christian and Richard, who are both too involved in other stuff to be reliable mentors. I'd suggest extending this weeks sync-meeting to give room for finding out who is actually available and not too unwilling to be mentor. ;-) Making a list of suggested topics would be on the agenda as well. Jacob From hpk at trillke.net Tue Apr 18 10:37:31 2006 From: hpk at trillke.net (holger krekel) Date: Tue Apr 18 10:43:25 2006 Subject: [pypy-dev] pypy-sync thursday 5:30 GMT+2 Message-ID: <20060418083731.GI10825@solar.trillke> Hi there, the next pypy-sync meeting of active developers is THURSDAY, 20th April, 5:30 PM GMT+2 (30-45 minutes) at #pypy-sync on freenode with the following topics (plus the ones that get mailed ahead of the meeting ): - activity reports + blockers - summer of X, how do we go about it? there are three possibilities: - aim for Summer of PyPy through the EU (becomes unlikely) - aim for becoming an "Summer of Code" organisation - provide mentors and participate at Googles Summer-of-Code through the PSF and try to get a bit of external funding for having participants attend sprints? decision time :) - what needs to be done until Iceland (21st May) for 0.9? Samuele and Holger went through the issue tracker and identified 0.9 release issues in Leysin. But we are missing assignments ... could everyone have a look and take responsibility for resolving particular issues? https://codespeak.net/issue/pypy-dev/ (and hit "Show 0.9"). Of course it is great if you assign yourself already before the sync meeting so we can see what is missing. - tackling and assigning the major 0.9 tasks to be worked on until Iceland: - integrating/refining the new stackless CFG transformations instead of the genc-support - adding missing support (if any) for app level stackless module (e.g. yield_current_frame_to_caller ?) - implementing tasklet pickling - advancing rctypes to support interfacing with CPython runtime through a CPyObjSpace - tackling the actual extension module compiler tool (including testing improvements and writing a tutorial) - overhauling and updating translation documentation (also in preparation for EU reports pending in June) best, holger From arigo at tunes.org Tue Apr 18 10:44:30 2006 From: arigo at tunes.org (Armin Rigo) Date: Tue Apr 18 10:44:31 2006 Subject: [pypy-dev] Upcoming Iceland sprint: 21-28 May Message-ID: <20060418084430.GA29917@code0.codespeak.net> Hi folks, We're pleased to announce the next PyPy sprint in Iceland from 21st (arrival) to 28th (depature) of May 2006. This sprint is kindly hosted and sponsored by EWT and CCPgames and thus we should be able to arrange funding for travels and accomodation for interested people. There are also non-PyPy activities going on -- see the full `EWT announcement`_ and people such as Tim Peters and other python-dev core developers plan to attend. If you'd like to help and join PyPy hacking around some geysers (actually we are meeting in Reykjavik, but well :), please mail us, preferrably on the `pypy-sprint mailing list`_. Please state your interest *until end this week* (friday 21st) to help the organizers plan for the sprint. The sprint goals for PyPy are, as usual, kept open to the interest of the participants, especially more so that there will be many non-PyPy people at the sprint to talk to. However, it is also likely that we will have to keep some focus on the goals of the upcoming June 0.9 release: * The extension module compiler. The goal is to be able to use a single RPython source code as an extension module for both PyPy and CPython. The means to get there is -- most likely -- by compiling ctypes-based modules into either pypy-c or a CPython dll/so. * Write some more modules in this style with ctypes. Sockets come to mind :-) * Stackless: the big missing feature is pickling running tasklets. There is also some smaller work that needs to be done, like exposing all the existing RPython-level interfaces to app-level (e.g. greenlets). * Write more documentation. * Misc topics, depending on interests: back-ends (CLI, Squeak), testing framework, modified semantics (security/sandboxing...), etc. A bientot, Armin + Holger .. _`EWT announcement`: http://mail.python.org/pipermail/python-announce-list/2006-April/004849.html .. _`pypy-sprint mailing list`: http://codespeak.net/mailman/listinfo/pypy-sprint From tismer at stackless.com Tue Apr 18 22:11:07 2006 From: tismer at stackless.com (Christian Tismer) Date: Tue Apr 18 22:11:50 2006 Subject: [pypy-dev] Re: Summer of Code 2006 In-Reply-To: <200604172250.00561.jacob@strakt.com> References: <443FFEE1.5060007@gmx.ch> <2mmzeklbwu.fsf@starship.python.net> <200604172250.00561.jacob@strakt.com> Message-ID: <444547DB.8010407@stackless.com> Jacob Hall?n wrote: > I think the only other ones who are familiar enough with the codebase to be > mentors are Christian and Richard, who are both too involved in other stuff > to be reliable mentors. Preferredly speaking for myself, I feel absolutely able to do mentoring, if the topic falls into my areas. I just asked Richard, and he might be interested, too. > I'd suggest extending this weeks sync-meeting to give room for finding out who > is actually available and not too unwilling to be mentor. ;-) Making a list > of suggested topics would be on the agenda as well. Good idea, I will be present. regards - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer at stackless.com Wed Apr 19 07:05:54 2006 From: tismer at stackless.com (Christian Tismer) Date: Wed Apr 19 07:06:35 2006 Subject: [pypy-dev] funny slowdown Message-ID: <4445C532.4090600@stackless.com> Hi friends, at some time over easter, my compilation time on Windows for extension modules increased remarkably, from a few seconds to a minute or more. Does anybody have an idea what might have caused this? thzanks - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From bclum at cs.ucsd.edu Wed Apr 19 07:38:51 2006 From: bclum at cs.ucsd.edu (Brian C. Lum) Date: Wed Apr 19 07:39:39 2006 Subject: [pypy-dev] Statically type-def-ing Python Message-ID: Dear Pypy group, Let me start by thanking you all for being so helpful in the past while I was learning to work with Pypy. I am trying to do research on how to type-def variables in Python at compile time. I have been reading different papers on bounds checking and static detection with scripting languages (for security), but I have not been able to find a feasible solution with Python. I was wondering if anyone knew some good references. I honestly cannot think of any way to type-def Python, except using heuristics to make guesses at what a variable might possibly be. Thank you in advance for your help, Brian From arigo at tunes.org Wed Apr 19 14:08:01 2006 From: arigo at tunes.org (Armin Rigo) Date: Wed Apr 19 14:08:03 2006 Subject: [pypy-dev] Statically type-def-ing Python In-Reply-To: References: Message-ID: <20060419120801.GA2713@code0.codespeak.net> Hi Brian, On Tue, Apr 18, 2006 at 10:38:51PM -0700, Brian C. Lum wrote: > I was wondering if anyone knew some good references. I honestly cannot > think of any way to type-def Python, except using heuristics to make > guesses at what a variable might possibly be. This is a well-known fact. The standard reference is Brett Cannon's thesis "Localized Type Inference of Atomic Types in Python". About guessing types, this is what RPython is about: http://codespeak.net/pypy/dist/pypy/doc/dynamic-language-translation.html Armin From nnorwitz at gmail.com Thu Apr 20 09:32:56 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Thu Apr 20 09:33:36 2006 Subject: [pypy-dev] Python Software Foundation seeks mentors and students for Google Summer of Code Message-ID: This spring and summer, Google will again provide stipends for students (18+, undergraduate thru PhD programs) to write new open-source code. The Python Software Foundation (PSF) http://www.python.org/psf/ will again act as a sponsoring organization in Google's Summer of Code, matching mentors and projects benefiting Python and Python users. Projects can include work on the core Python language, programmer utilities, libraries, packages, frameworks related to Python, or other Python implementations like Jython, PyPy, or IronPython. Please add your project ideas to the existing set at http://wiki.python.org/moin/SummerOfCode Mentoring instructions are also on this page. The deadline is soon, so please sign up as a mentor early. If you are a student considering a project, you should start deciding now. Feel free to ask questions on python-dev@python.org The main page for the Summer of Code is http://code.google.com/summerofcode.html At the bottom are links to StudentFAQ, MentorFAQ, and TermsOfService. The first two have the timeline. Note that student applications are due between May 1, 17:00 PST and May 8, 17:00 PST. People interested in mentoring a student though PSF are encouraged to contact me, Neal Norwitz at nnorwitz@gmail.com. People unknown to Guido or myself should find a couple of people known within the Python community who are willing to act as references. Feel free to contact me if you have any questions. I look forward to meeting many new mentors and students. Let's improve Python! Cheers, n From arigo at tunes.org Thu Apr 20 13:24:57 2006 From: arigo at tunes.org (Armin Rigo) Date: Thu Apr 20 13:24:59 2006 Subject: [pypy-dev] pypy-sync thursday 5:30 GMT+2 In-Reply-To: <20060418083731.GI10825@solar.trillke> References: <20060418083731.GI10825@solar.trillke> Message-ID: <20060420112457.GA16248@code0.codespeak.net> Hi all, On Tue, Apr 18, 2006 at 10:37:31AM +0200, holger krekel wrote: > THURSDAY, 20th April, 5:30 PM GMT+2 (30-45 minutes) Oups, I'm going to miss this pypy-sync... > - activity reports + blockers LAST: rctypes, which are roughly done by now. NEXT: polish edges, try compiling pypy/module/_demo as a CPython ext mod BLOCKERS: - > - summer of X, how do we go about it? > there are three possibilities: > > - aim for Summer of PyPy through the EU (becomes unlikely) > - aim for becoming an "Summer of Code" organisation > - provide mentors and participate at Googles Summer-of-Code > through the PSF and try to get a bit of external > funding for having participants attend sprints? > > decision time :) I can't judge on the likeliness of various funding models, but at that point I believe (without convincing arguments, though) that just going for the same model as last year is what makes the most sense. Nothing was done on the Summer of PyPy front as far as I can tell. I don't fully see the point of becoming our own Summer of Code organization at this point. I'm also ready to mentor a student, whatever the model. > - what needs to be done until Iceland (21st May) for 0.9? A possible missing clasification on the issue tracker is the separation between the two goals of that release: one is community-oriented, and the other is to fullfill EU requirements. For the latter, we are mostly done apart from tasklet pickling. This is the one that needs to find a leader *now*. It's a contractual thing. I agree that all other points are very nice and some are rather needed, but we are still allowed to miss one of them, like weakrefs, __subclasses__, or moving gc's... My point is that I would personally like some more focus on new experimental features instead of spending too much time on polishing, making performance more stable, even more compliance, etc. -- I think that's what we really promized to both the community and the EU. I'm not trying to minimize the importance of all that work, which will have to be done sooner or later, but basically we promized many great things so the sooner we show at least experimentally that they are possible, the more interested people will become IMHO. I hope this doesn't sound too negative! It was not my intention. My intention is just to reiterate what I see as the mid-term guidelines that I need to keep in mind. > (...) > - adding missing support (if any) for app level stackless module > (e.g. yield_current_frame_to_caller ?) No, this was about the higher-level interfaces -- we cannot expose yield_current_frame_to_caller directly. It's about exposing channels to make tasklets usable, and greenlets. A bientot, Armin From hpk at trillke.net Thu Apr 20 14:59:03 2006 From: hpk at trillke.net (holger krekel) Date: Thu Apr 20 15:05:13 2006 Subject: [pypy-dev] pypy-sync thursday 5:30 GMT+2 In-Reply-To: <20060420112457.GA16248@code0.codespeak.net> References: <20060418083731.GI10825@solar.trillke> <20060420112457.GA16248@code0.codespeak.net> Message-ID: <20060420125903.GC10825@solar.trillke> Hi Armin, On Thu, Apr 20, 2006 at 13:24 +0200, Armin Rigo wrote: > On Tue, Apr 18, 2006 at 10:37:31AM +0200, holger krekel wrote: > > - what needs to be done until Iceland (21st May) for 0.9? > > A possible missing clasification on the issue tracker is the separation > between the two goals of that release: one is community-oriented, and > the other is to fullfill EU requirements. For the latter, we are mostly > done apart from tasklet pickling. I disagree. We cannot just ship the current svn/pypy/dist and i actually think it's very misleading to give this impression. Getting to 0.9 is more than just a few days work or even (as you somewhat imply apart from tasklet pickling) no work at all. And don't forget the reports for that matter. > This is the one that needs to find a > leader *now*. It's a contractual thing. I agree that all other points > are very nice and some are rather needed, but we are still allowed to > miss one of them, like weakrefs, __subclasses__, or moving gc's... sure, but it's not that we don't have tons of other stuff waiting for us after 0.9. Generally i think we should be careful if we further postpone issues into the last 7 months of the EU project (where people might get the funny idea to even take holidays) that got already postponed for 5-9 months. but i am fine with missing _some_ of the issues. > My point is that I would personally like some more focus on new > experimental features instead of spending too much time on polishing, > making performance more stable, even more compliance, etc. -- I think > that's what we really promized to both the community and the EU. I'm > not trying to minimize the importance of all that work, which will have > to be done sooner or later, but basically we promized many great things > so the sooner we show at least experimentally that they are possible, > the more interested people will become IMHO. Hum, but especially users of the release ("the community" resp.) will want to get some reasonably stable and entry points for using PyPy (especially the ext-compiler) which requires some polishing and documentation work, doesn't it? > I hope this doesn't sound too negative! It was not my intention. My > intention is just to reiterate what I see as the mid-term guidelines > that I need to keep in mind. > > > (...) > > - adding missing support (if any) for app level stackless module > > (e.g. yield_current_frame_to_caller ?) > > No, this was about the higher-level interfaces -- we cannot expose > yield_current_frame_to_caller directly. It's about exposing channels to > make tasklets usable, and greenlets. sure, i didn't mean to expose yield_current_frame_to_caller at app-level but to support it for the stackless (interp-level) module which itself supports tasklets etc. holger From lac at strakt.com Fri Apr 21 13:38:19 2006 From: lac at strakt.com (Laura Creighton) Date: Fri Apr 21 13:39:01 2006 Subject: [pypy-dev] From edu-sig, Guido on connecting squeak and python Message-ID: <200604211138.k3LBcKT0021661@theraft.strakt.com> ------- Forwarded Message Return-Path: edu-sig-bounces@python.org Delivery-Date: Fri Apr 21 10:34:25 2006 Return-Path: Message-ID: Date: Fri, 21 Apr 2006 09:29:14 +0100 From: "Guido van Rossum" To: "Paul D. Fernhout" There's a lot to read in your post and the links you post -- more than I have time ofr right now. Let me try to prune some of the ideas. I'm not interested in switching to Jython for this purpose; nor am I interested in directly linking to code that's part of Squeak -- unless, perhaps, there's some low-level code that is independent of the rest of the Squeak environment while providing some functionality we need. I'm also not interested in making Python an entirely self-contained system such as Squeak is -- much of Python's strengths come from its capabilities as a glue language, seamlessly integrating with other software on many different platforms. But, after encouragement from Alan Kay, I *am* interested in producing a Squeak-like environment *on top* of Python. Alan suggested using a slightly different starting point than Squeak; modern graphics cards have a wealth of functionality that can be accessed directly. I'm no graphics expert, but I believe OpenGL and perhaps SVG could be the right basis to get started. The approach that seems to make the most sense to me (but I'm open for alternatives) is to start out by producing a solid low-level graphics package like this that can work across platforms (Linux, Windows and OSX preferably); once that is settled, we could build an application resembling Squeak's UI. There's probably more to it; but typing this email at a busy conference my thoughts are a bit distracted. - --Guido On 4/21/06, Paul D. Fernhout wrote: > I've long been interested in making Python development more Squeak > Smalltalk like. See for example a recent post of mine to the Jython user > mailing list with some code (would be also useful for CPython with a few > changes): > [jython-users] ReloaderWindow 0.2 (improvements to selective reloading) > http://sourceforge.net/mailarchive/message.php?msg_id=14482359 > > On the topic of an integrated 2D/3D crossplatform solution for Python > (like Squeak has), I'd like to point to my related comments on this list > from six (!) years back: > > [Edu-sig] Common Graphical Framework for Python Tutorials? > Fri, 04 Feb 2000 11:16:39 -0500 > http://mail.python.org/pipermail/edu-sig/2000-February/000032.html > > [Edu-sig] a modest proposal II > Fri, 04 Feb 2000 18:03:01 -0500 > http://mail.python.org/pipermail/edu-sig/2000-February/000063.html > > [Edu-sig] IDLE/TK limitations for learning environments > Fri, 04 Feb 2000 18:32:54 -0500 > http://mail.python.org/pipermail/edu-sig/2000-February/000065.html > > [Edu-sig] Not well supported on the Mac? > Sun, 28 May 2000 15:24:21 -0400 > http://mail.python.org/pipermail/edu-sig/2000-May/000495.html > > Glad to see some interest in such ideas is perking up here. :-) > > It's all quite doable with enough effort. Though I'd watch out for Squeak > licensing and some Squeak unfinished complexity management issues. I think > it might be better to just use the Squeak base cross-platform ideas or > base code (or perhaps base a new work on wxWidgets) and build a larger > common framework using Python technology and the Python license (and yet > also of interest to Squeakers, like by adding in support for a Smalltalk > parser). > > Alternatively, one could build on top of Jython -- see my post on this > list from last year on this topic: > > [Edu-sig] On Jython for education > Wed Oct 19 15:26:02 CEST 2005 > http://mail.python.org/pipermail/edu-sig/2005-October/005410.html > > I think the Jython-based approach might be easiest, though one then has to > wrestle with other Java community and licensing issues. [I personally > think the Squeak approach would be more stable and maintainable though, > just 2000 lines of core C to port per platform, with widgets built on > that, and a dynamic loading facility for other native code.] A > cross-platform system supporting both Python and Smalltalk (and perhaps > Java) on a JVM with a complete Smalltalk-like development environment > (including cross-image debugging and development) and with 3D plus some > sort of PythonCard/HyperCard framework out of the box, which had the > option of running as a browser plugin, would be really neat. Probably at > least few person months (for me :-) to get that going to the point where > it reached a critical mass and was something people wanted to use or build > on top of though. I've worked on bits and pieces of all these ideas in a > variety of contexts, but never had a chance to put them all together. > > --Paul Fernhout > > Andre Roberge wrote: > > And, if I may quote from Kirby's follow-up post > > http://controlroom.blogspot.com/2006/04/shuttleworth-summit-day-two.html > > --- > > [...] > > Loosely coupled tools, with a bottom-up, open source curriculum > > writing process, will leave the question of tools somewhat open-ended. > > The lesson plans will specify the software needed, with multiple paths > > possible. > > --- > > +1. I couldn't agree more :-) > > ============ > > [...] > > Momentum seems to be building for a stronger graphics engine, either > > 2D or 3D, with Python bindings, that'll run interactively from within > > a browser. The Squeak folks may be willing to contribute to this > > effort. Guido feels we'll need to recruit new talent for this, as the > > Python community is currently pretty maxed out on projects. Should > > such an engine be developed, turtle stuff would be incorporated > > therein. > > ======= > > > > I would love to see this happening and would definitely be willing to > > contribute to such an effort. Of course, I would use this to port > > rur-ple to the web (as a first step). Anybody else is as excited > > about this possibility as I am? > > > > Andr? > > _______________________________________________ > Edu-sig mailing list > Edu-sig@python.org > http://mail.python.org/mailman/listinfo/edu-sig > - -- - --Guido van Rossum (home page: http://www.python.org/~guido/) _______________________________________________ Edu-sig mailing list Edu-sig@python.org http://mail.python.org/mailman/listinfo/edu-sig ------- End of Forwarded Message From l.oluyede at gmail.com Fri Apr 21 19:59:48 2006 From: l.oluyede at gmail.com (Lawrence Oluyede) Date: Fri Apr 21 20:00:26 2006 Subject: [pypy-dev] Re: [pypy-sprint] Upcoming Iceland sprint: 21-28 May In-Reply-To: <20060418084430.GA29917@code0.codespeak.net> References: <20060418084430.GA29917@code0.codespeak.net> Message-ID: <9eebf5740604211059m32235406xbc353e6f2e6ebfdc@mail.gmail.com> On 4/18/06, Armin Rigo wrote: > Hi folks, > > We're pleased to announce the next PyPy sprint in Iceland from 21st > (arrival) to 28th (depature) of May 2006. This sprint is kindly hosted > and sponsored by EWT and CCPgames and thus we should be able to arrange > funding for travels and accomodation for interested people. There are > also non-PyPy activities going on -- see the full `EWT announcement`_ > and people such as Tim Peters and other python-dev core developers plan > to attend. If you'd like to help and join PyPy hacking around some > geysers (actually we are meeting in Reykjavik, but well :), please mail > us, preferrably on the `pypy-sprint mailing list`_. Please state your > interest *until end this week* (friday 21st) to help the organizers > plan for the sprint. > > The sprint goals for PyPy are, as usual, kept open to the interest of > the participants, especially more so that there will be many non-PyPy > people at the sprint to talk to. However, it is also likely that we > will have to keep some focus on the goals of the upcoming June 0.9 > release: > > * The extension module compiler. The goal is to be able to use a single > RPython source code as an extension module for both PyPy and CPython. > The means to get there is -- most likely -- by compiling ctypes-based > modules into either pypy-c or a CPython dll/so. > > * Write some more modules in this style with ctypes. Sockets come to > mind :-) > > * Stackless: the big missing feature is pickling running tasklets. > There is also some smaller work that needs to be done, like exposing > all the existing RPython-level interfaces to app-level > (e.g. greenlets). > > * Write more documentation. > > * Misc topics, depending on interests: back-ends (CLI, Squeak), testing > framework, modified semantics (security/sandboxing...), etc. Like Antonio I'd really like taking part of this sprint. Unlike him I've never contributed to pypy project. I'm a CS student here in Italy. I'd like to bring further the development of extension modules of PyPy (if my friend Valentino doesn't complete the task in Japan :) ) and I find the Antonio's work very interesting. Unfortunately I can't afford the trip so there are any chances about being funded? ciao :) -- Lawrence http://www.oluyede.org/blog From arigo at tunes.org Fri Apr 21 21:16:59 2006 From: arigo at tunes.org (Armin Rigo) Date: Fri Apr 21 21:16:59 2006 Subject: [pypy-dev] extregistry refactoring Message-ID: <20060421191658.GA22094@code0.codespeak.net> Hi all, Warning: if you're using the extregistry, I just refactored its interface. I think the result is nicer -- you don't have to define a bunch of functions and link them together in a register_xxx() call; instead, all relevant functions are now methods on a class that is linked via a class method call or automatically with a metaclass trick. The motivation for this was a subtle annotation crash in rctypes: indeed, in the previous model, families of built-in functions (e.g. all ctypes function objects) would get annotated in a way that kept creating new closures. The annotator was then complaining about the resulting SomeBuiltin annotation: as it stores the closure in question, it was always different whenever a block was reflown through the annotator. This refactoring was pending for some time, because previously some functions were missing some information and had to hack around to get at it. A bientot, Armin. From lac at strakt.com Sat Apr 22 12:33:11 2006 From: lac at strakt.com (Laura Creighton) Date: Sat Apr 22 12:33:50 2006 Subject: [pypy-dev] educational python environment squeak/python (from edu-sig) Message-ID: <200604221033.k3MAXBke024077@theraft.strakt.com> I'll let Guido's message speak for itself. Laura ------- Forwarded Message Return-Path: edu-sig-bounces@python.org Delivery-Date: Sat Apr 22 12:20:54 2006 Date: Sat, 22 Apr 2006 11:19:02 +0100 From: "Guido van Rossum" To: edu-sig@python.org I'm looking for someone to "own" the development of a Squeak-like environment for Python. I can help by getting you in touch with Alan Kay and other Squeak folks. But I just can't be managing this project myself -- I need to focus on Python 3000, which has quite a different set of goals (not incompatible, just different, and enough to keep me very busy in the next two years). To provide some background, here is (with permission, and slightly edited) a forwarded message from Mark Shuttleworth in response to something I sent him this morning. - --Guido - ---------- Forwarded message ---------- From: Mark Shuttleworth Date: Apr 22, 2006 8:56 AM Subject: Re: Moving forward the educational Python code development To: Guido van Rossum Cc: Kirby Urner , [...] Guido van Rossum wrote: > After Kirby's posts and mine on the Python mailing list for the >education Special Interest Group (edu-sig@python.org), several threads >have spun up showing that there is a lot of interest in the topic of >making Python more "Squeak-like", however you might want to define >this. This is great news. My sense is that it will take three or more years to get a Squeak-like environment built up in the Python world. It will take a lot of work, more raw development work than TSF is willing to fund. Our project deliverable is the curriculum and the training program for teachers to teach it, we don't want to be responsible for delivering the unified development environment through we recognise that having a unified environment is beneficial to the project. The only way the unified development environment will actually happen is if the SqueakLand community, with Alan's leadership, makes this their goal, and figures out how to work with the Python community. It's the SqueakLand folks who understand what that environment needs to "feel" like. They have the strong vision as to what the tools should do. At the same time, the Python community will need some people to step up and welcome that effort, contributing to it over time, because the SqueakLand folks are used to Squeak and they will need to learn how best to execute their vision in Python. TSF can provide some limited funding, but we're not setup to lead development efforts (we learned the hard way with SchoolTool which is now in good shape). Most of the TSF funding will go towards actual curriculum development that USES the tools available. We will initially use Logo, Squeak and Python as-is, because they are there right now, and will develop the curriculum using what's currently available with a view to consolidating it all around Python as Python gains the ability to deliver Logo-like and Squeak-like environments. > I believe now is the time to delegate to someone other than me the >task of researching the course of action; what exactly is needed, >which toolkit to use (e.g. Mozilla, pygame, OpenGL, or something >else?), and to manage the development. Unfortunately I don't have the >time to do this myself. Does the Shuttleworth Foundation have >resources to pick up this project, now that there's momentum building >up? >This could take the form of an existing Shuttleworth employee jumping >in, or some Foundation money to get one of the interested contributors >(perhaps ) to put in their time (part-time). I don't see this >happening (at the time scale envisioned) on a purely volunteer basis. >I'm sure we can get volunteers to do coding for the project; but >managing it, researching the possible options, and making decisions >probably requires someone dedicated to this task. Yes, agreed, and we can contribute some funding towards the leadership of the effort. I'm not keen to fund "part time work with no clear deliverables" because its hard to get predictable results that way. I guess what we need are you and Alan to figure out between you who will make the most effective leader, bringing together the SqueakLand vision in a Python execution, and we would step up to fund portion of that person's time. [...] I think the leadership of this effort needs to be decided after discussion with Alan too, so that whoever is the funded driving force has buy in from both communities. If you'd like to forward this message to other folks in the community as a statement of the parts TSF is able to contribute to, please do. Mark - ---------- End forwarded message ---------- - -- - --Guido van Rossum (home page: http://www.python.org/~guido/) _______________________________________________ Edu-sig mailing list Edu-sig@python.org http://mail.python.org/mailman/listinfo/edu-sig ------- End of Forwarded Message From hpk at trillke.net Sun Apr 23 10:59:39 2006 From: hpk at trillke.net (holger krekel) Date: Sun Apr 23 11:06:07 2006 Subject: [pypy-dev] minutes pypy-sync last thursday Message-ID: <20060423085939.GH10825@solar.trillke> Hi folks, find below the minutes of the pypy-sync meeting last thursday. best, holger pypy-sync 20th April 2006, 05:30 PM (UTC+2) ===================================================== attendants: michael, anto, carl, samuele, christian, richard, eric, aurelien, gromit, arre, anders l. - activity reports + blockers everybody posted (see at the end of these minutes) - summer of X, how do we go about it? we quickly agreed to a) go for being mentors through PSF at Google's Summe of Code campaign b) determined mentors c) to create a directory in extradoc/soc-2006 listing all info (mentors, topics) d) next week Michael will communicate all the info to Neal / PSF / SoC - what needs to be done until Iceland (21st May) for 0.9? There was some initial and involved discussion on this topic, which quickly moved to sorting out EU tasks and responsibilities for some involved people. Also related to the brief pypy-dev discussion between Armin and Holger, we determined a need to settle and agree on a distribution of tasks until and after Iceland. This we agreed to do at a specific meeting for EU developers on TUESDAY, 23rd April, 5pm (UTC+2) where Christian, Michael, Samuele, Holger, Carl, Aurelien, Anders L., and Gromit agreed to attend already. IRC log ------------------------------- Apr 20 17:28:55 stakkars: hi! Apr 20 17:29:25 hi friends Apr 20 17:29:59 I'll give rxe a call Apr 20 17:30:25 hi all! Apr 20 17:30:33 stakkars stedi67 Apr 20 17:30:37 stakkars: want to wake him up? :) Apr 20 17:30:45 hi Apr 20 17:30:53 T-0 according to me Apr 20 17:30:58 yip Apr 20 17:31:01 who is moderating? hpk ? Apr 20 17:31:06 yes, i guess so Apr 20 17:31:17 topics: Apr 20 17:31:17 * activity reports Apr 20 17:31:17 * summer of X Apr 20 17:31:17 * work until iceland for 0.9 topics Apr 20 17:31:17 * assigning major 0.9 topics Apr 20 17:31:23 he's early, probably in the company, swamped with work Apr 20 17:31:43 so let's start with activity reports (and we can see who is actually really here on the channel active :) Apr 20 17:31:59 PREV: vacation, configuring new laptop Apr 20 17:31:59 NEXT: Tokyo sprint Apr 20 17:31:59 BLOCKERS: - Apr 20 17:32:05 PREV: Vacation Apr 20 17:32:06 LAST: Leysin Sprint, iceland organisation, some testing and playing around Apr 20 17:32:06 NEXT: work on testing and build environment Apr 20 17:32:06 BLOCKERS: - Apr 20 17:32:06 LAST: slept off leysin, ACCU Apr 20 17:32:07 NEXT: stackless transform? Apr 20 17:32:07 BLOCKERS: insane busyness Apr 20 17:32:09 LAST: investigating translatability of constraint stuff Apr 20 17:32:10 NEXT: playing with choose() Apr 20 17:32:10 BLOCKERS: working, clonable coroutines Apr 20 17:32:13 LAST: travelling Apr 20 17:32:13 NEXT: Tokyo sprint Apr 20 17:32:13 BLOCKERS: - Apr 20 17:32:16 NEXT: Sprint Apr 20 17:32:18 BLOCKERS: - Apr 20 17:32:23 LAST: set implementation, finding bugs Apr 20 17:32:23 NEXT: finding and documenting more bugs... Apr 20 17:32:23 BLOCKERS: - Apr 20 17:32:26 LAST: completed rlist support for ootypesystem Apr 20 17:32:27 NEXT: more works on ootypesystem (probably rdict) Apr 20 17:32:29 BLOCKER: my still poor knowledge of pypy internals :-) Apr 20 17:32:34 LAST: GC work, optimizations Apr 20 17:32:35 NEXT: uni stuff, trying to implement __del__ with the framework Apr 20 17:32:35 BLOCKERS: new environment, time constraints Apr 20 17:32:43 LAST: {}[[]], tweaks and fixing various transformation problems, rested during easter Apr 20 17:32:45 NEXT: tokyo sprint Apr 20 17:32:46 BLOCKERS: - Apr 20 17:33:03 * auc is away for 5 minutes Apr 20 17:33:07 auc: how badly do you depend on "clonable coroutines" and in which way? Apr 20 17:33:11 oh, never mind then Apr 20 17:33:37 ok, let's head on then Apr 20 17:33:43 * summer of X Apr 20 17:34:05 from in-between discussions and opinions i gather that we are all leaning towards participating in SoC as mentors through PSF Apr 20 17:34:11 yes Apr 20 17:34:12 DONE: wrapping Apr 20 17:34:12 NEXT: finalizing, prepare next PyPy work Apr 20 17:34:12 BLOCK: complexity Apr 20 17:34:12 is that correct or are there other opinions? Apr 20 17:34:35 sounds good to me Apr 20 17:34:51 ditto Apr 20 17:34:56 +1 Apr 20 17:35:00 +1 Apr 20 17:35:04 fine, then let's see how we go about it Apr 20 17:35:05 a) mentors Apr 20 17:35:06 b) topics Apr 20 17:35:15 hpk: just removed last years's mentorship entry from the wiki Apr 20 17:35:21 i suggest that we open a soc-2006 directory in extradoc with such information Apr 20 17:35:32 everyone willing to mentor should email neal norwitz Apr 20 17:35:35 (Aurelien already committed something today to pypy/doc) Apr 20 17:35:48 mwh: i think we should gather on our side and then someone sends all the info Apr 20 17:36:02 should gather info Apr 20 17:36:04 if you need "personal references" as on the python.org wiki, feel free to mention me :) Apr 20 17:36:05 +1 Apr 20 17:36:11 hpk: if you like Apr 20 17:36:24 i'm already signed up as a mentor, cause i was one last year Apr 20 17:36:44 mwh: ok, carl is as well i think Apr 20 17:37:03 or will soon be Apr 20 17:37:06 ok, then let's put all info into soc-2006 and send info off on the weekend Apr 20 17:37:15 just to get a quick impression: Apr 20 17:37:16 is there a deadline for mentoring? Apr 20 17:37:20 who would mentor? Apr 20 17:37:25 mwh: yes, there is one Apr 20 17:37:32 1st may i think Apr 20 17:37:38 cfbolz: me, you, armin for certain Apr 20 17:37:40 (but i may confuse the various deadlines) Apr 20 17:37:56 hpk: i thought 1 may was the deadline for being a mentoring *organization* Apr 20 17:37:57 i would also mentor but preferably to py.test, build-tool stuff i guess Apr 20 17:38:06 hpk: neal writes "soon" :-) Apr 20 17:38:17 mwh: ok, might well be, whatever, let's just get this sorted and communicate to them Apr 20 17:38:25 hpk: sure Apr 20 17:38:28 it shouldn't be hard Apr 20 17:38:48 anybody else? Apr 20 17:39:07 i am sure that aurelien or so would be interested Apr 20 17:39:10 Add me as well Apr 20 17:39:16 I would prefer not to Apr 20 17:39:31 eric also said that he would not like to Apr 20 17:39:41 ok, then that's it for now Apr 20 17:39:44 I would, of course Apr 20 17:39:46 I would like to b a mentor but I think it would not be a good idea (indeed) Apr 20 17:39:49 pedronis, i guess Apr 20 17:40:03 let's just open the soc-2006 directory and put info there Apr 20 17:40:08 hpk: are you going to be the one who checks this in? Apr 20 17:40:17 mwh: ye Apr 20 17:40:17 cfbolz: can you make this happen and make sure we communicate next week to them? Apr 20 17:40:34 hpk: not before sunday, no Apr 20 17:41:00 ok, then i start but would like you or someone else to communicate to the SoC people (Neal) Apr 20 17:41:08 eventually Apr 20 17:41:12 as i haven't been involved there Apr 20 17:41:23 mwh: do you know neal? Apr 20 17:41:34 cfbolz: i talked to him a bit at pycon Apr 20 17:41:45 i can do the communicating Apr 20 17:41:52 cool, thanks Apr 20 17:41:57 i am away 29 apr -> 8 may (ish) Apr 20 17:42:02 so i'll do it before then :) Apr 20 17:42:06 ok, let's try to do it before then Apr 20 17:42:07 yes Apr 20 17:42:10 next topic Apr 20 17:42:24 * 0.9 related work until iceland Apr 20 17:42:44 --> rxe (n=rxe@66.151.59.5) has joined #pypy-sync Apr 20 17:42:47 hi richard :) Apr 20 17:42:57 Hi everyone! :-) Apr 20 17:42:59 :-) Apr 20 17:43:01 hey richard Apr 20 17:43:02 sorry it has been a long time Apr 20 17:43:03 hey rxe Apr 20 17:43:48 so there has been a bit of discussion on pypy-dev Apr 20 17:44:05 what do the others think? Apr 20 17:45:31 the truth is somewhere in the middle. there is definitively more to do than just stackless pickling Apr 20 17:45:51 My feeling is that a 0.9 version should be almost a 1.0 version (which I consider a "this is it" version) does that make sense? Apr 20 17:46:07 which I think we are not quiet yet Apr 20 17:46:08 like finishing the stackless-applevel exposure Apr 20 17:46:33 the major topics of the 0.9 release are the ext-module compiler and stackless features Apr 20 17:46:37 well, there's also other stuff in the WP7 tasks which is explcitly listed Apr 20 17:46:48 yes, __del__, weakref Apr 20 17:47:09 that's just work. pickling is hard since I still don't know how Apr 20 17:47:10 I don't think WP7 says anything about __dell__ or weakrefr Apr 20 17:47:26 well, even just work takes time Apr 20 17:47:31 pedronis: then not :-) Apr 20 17:47:33 pedronis: indeed Apr 20 17:47:53 yes but I can distribute if I know how Apr 20 17:48:06 stakkars: what are you working planning on working on? Apr 20 17:48:21 damn timezones mean i don't get to chat much... Apr 20 17:49:02 well pickling first thing, although it might depend on the new transform Apr 20 17:49:05 cfbolz: the thing regarding "middle ground": we really only have 7 months left for tons of stuff we want to do - just postponing things from a "we don't neccessarily need to do it" will backfire IMO Apr 20 17:49:21 so we should strike a good balance Apr 20 17:49:26 stakkars stedi67 Apr 20 17:49:33 cfbolz: the fact is that some unfinished WP7ish stuff is really WP5 leftovers Apr 20 17:49:34 stakkars: note that the new transform is not fully integrated Apr 20 17:49:45 stakkars: have you looked at the transforrm code? Apr 20 17:49:45 it just mostly works but certainly requires work Apr 20 17:50:10 that's what I'm saying, might be blocker Apr 20 17:50:25 pedronis: what is a wp5 leftover? Apr 20 17:50:35 cfbolz: well part of the GC stuff Apr 20 17:50:53 stakkars: michael and me worked on stackless to support you, not to block you :) Apr 20 17:50:58 i mean, maybe in the short term it makes sense for stakkars to work on channels and greenlets, and for me to try to polish the stackless transform Apr 20 17:51:00 stakkars: you have 9 mm in WP7 Apr 20 17:51:19 * auc is back (ouch) Apr 20 17:51:28 gosh. I appreciate of course. Apr 20 17:51:30 by "short term" i mean "for a week" Apr 20 17:51:59 I just didn't realize that it's necessary,, before i saw pedronis message Apr 20 17:52:21 the idea of the current topic is that we identify the issues and assign responsible persons Apr 20 17:52:30 i am not sure we can achieve it in some minutes Apr 20 17:52:40 but we should aim for getting that clarified latest next week Apr 20 17:53:23 yes Apr 20 17:53:35 because not discussing it will not help either :) Apr 20 17:53:47 indeed :) Apr 20 17:53:54 mwh: that makes a lot of sense Apr 20 17:53:56 stakkars: is really your workpackage. Helping you is shifting things around. It may even become a serious issue after a point. Apr 20 17:54:25 but is not the right forum for that discussion Apr 20 17:54:29 yes, i agree Apr 20 17:55:10 pedronis:I considered the grapg transform as an add-on, nice to have. as said, didn't know Apr 20 17:55:27 stakkars: well, it will be necessary for graph pickling Apr 20 17:55:35 stakkars: so it is not an addon Apr 20 17:55:36 so we need a discussion next week about this specific "how to tackle 0.9 tasks" topic and to define the scope Apr 20 17:55:46 cfbolz: necessary is a strong word, but i more or less agree Apr 20 17:56:03 another point that isn't entirely irrelevant is that i'm enjoying working on 'stackless style' stuff Apr 20 17:56:12 :) Apr 20 17:56:12 * hpk too actually Apr 20 17:56:14 anyway task pickling plus the other missing stuff is not a small task Apr 20 17:56:19 yes Apr 20 17:56:24 to finish in less than a month Apr 20 17:56:50 that we are so late was not expected Apr 20 17:56:58 ok, as we are discussing things mostly from an EU perspective i'd like to invite to a specific meeting for EU developers early next week Apr 20 17:57:15 yes, this is not the right forum Apr 20 17:57:21 tuesday? Apr 20 17:57:28 big thing is to find the right time for everyone :) Apr 20 17:57:35 tuesday would be fine Apr 20 17:57:38 5pm seems likes the best compromise Apr 20 17:57:44 midnight in japan, 8am in CA Apr 20 17:57:45 (i will be in d??sseldorf) Apr 20 17:57:58 stakkars stedi67 Apr 20 17:58:20 mwh, stakkars, cfbolz, pedronis, auc, aleale, arre, ...: fine for you? Apr 20 17:58:23 for me, the other way goes as well (midnight +) Apr 20 17:58:35 ok Apr 20 17:58:37 yes Apr 20 17:59:00 stakkars stedi67 Apr 20 17:59:05 ok Apr 20 17:59:14 ok Apr 20 17:59:28 ok Apr 20 17:59:45 great, than we close this topic (and the next one, which relates to it) Apr 20 17:59:46 ok Apr 20 17:59:58 see you all, i will mail to pypy-dev to not forget anyone Apr 20 18:00:31 auc: you are available for mentoring as well, right? Apr 20 18:00:36 auc: mentoring SoC Apr 20 18:01:12 --> Gromit_ (n=bear@does-d9b90ad6.pool.mediaWays.net) has joined #pypy-sync Apr 20 18:01:49 hpk: i guesss so ... Apr 20 18:02:12 but not necessarily about all of what i posted in pypy/doc From nicolas.chauvat at logilab.fr Sun Apr 23 11:21:28 2006 From: nicolas.chauvat at logilab.fr (Nicolas Chauvat) Date: Sun Apr 23 11:22:01 2006 Subject: [pypy-dev] Re: [pypy-eu-svn] r26155 - pypy/extradoc/sprintinfo/tokyo In-Reply-To: <20060423031638.57E981007C@code0.codespeak.net> References: <20060423031638.57E981007C@code0.codespeak.net> Message-ID: <20060423092128.GC24564@crater.logilab.fr> Hi Sprinters, On Sun, Apr 23, 2006 at 05:16:38AM +0200, nik@codespeak.net wrote: > + - PyPy debian packaging (Yutaka, Holger?) Alexandre has an ITP on this package and he has spent some time thinking about it. You should probably ask him how far he went. What about tasklet serialisation, any chance this could get done during the sprint ? It's becoming a blocker for WP09. Or should we tackle that on our side as soon as we get the chance ? -- Nicolas Chauvat logilab.fr - services en informatique avanc?e et gestion de connaissances From bea at changemaker.nu Sun Apr 23 12:38:14 2006 From: bea at changemaker.nu (Beatrice During) Date: Sun Apr 23 12:38:58 2006 Subject: [pypy-dev] Re: [pypy-eu-svn] r26155 - pypy/extradoc/sprintinfo/tokyo In-Reply-To: <20060423092128.GC24564@crater.logilab.fr> References: <20060423031638.57E981007C@code0.codespeak.net> <20060423092128.GC24564@crater.logilab.fr> Message-ID: <444B5916.4020901@changemaker.nu> Hi there Nicolas Chauvat skrev: > Hi Sprinters, > > On Sun, Apr 23, 2006 at 05:16:38AM +0200, nik@codespeak.net wrote: >> + - PyPy debian packaging (Yutaka, Holger?) > > Alexandre has an ITP on this package and he has spent some time > thinking about it. You should probably ask him how far he went. Yes - I talked with Niibe-san about this and we agreed to check with Alexandre first. I think Niibe-san will send an email..... I am also discussing this with Holger since it concerns packaging, tools and such. Apparently Niibe-san is also interested in the py-lib as well which needs some kind connected packaging..... > What about tasklet serialisation, any chance this could get done > during the sprint ? It's becoming a blocker for WP09. Or should we > tackle that on our side as soon as we get the chance ? > We are discussing this right now on various levels - there is a dev-meeting on Tuesday 17:00 that will discuss the content of the 0.9 release - it will be part of that. So it will be done before the Iceland sprint, and finalized there. Please make sure you and/or your people attend and explain your concerns. So - it is not a topic on this sprint. People here seem more interested in Lisp and Javascript backends and sockets and other weird things ;-) Greetings from the 11th floor of the Dai-Biru of central Akihabara - we are coding with an amazing view of neonsigns surrounding us....;-) Cheers Bea From arigo at tunes.org Sun Apr 23 13:29:00 2006 From: arigo at tunes.org (Armin Rigo) Date: Sun Apr 23 13:29:02 2006 Subject: [pypy-dev] minutes pypy-sync last thursday In-Reply-To: <20060423085939.GH10825@solar.trillke> References: <20060423085939.GH10825@solar.trillke> Message-ID: <20060423112900.GA21389@code0.codespeak.net> Hi, On Sun, Apr 23, 2006 at 10:59:39AM +0200, holger krekel wrote: > agreed to do at a specific meeting for EU developers on > > TUESDAY, 23rd April, 5pm (UTC+2) It will be the 25th, not the 23rd, unless I'm mistaken. Armin From hpk at trillke.net Sun Apr 23 13:30:27 2006 From: hpk at trillke.net (holger krekel) Date: Sun Apr 23 13:36:54 2006 Subject: [pypy-dev] minutes pypy-sync last thursday In-Reply-To: <20060423112900.GA21389@code0.codespeak.net> References: <20060423085939.GH10825@solar.trillke> <20060423112900.GA21389@code0.codespeak.net> Message-ID: <20060423113027.GM10825@solar.trillke> On Sun, Apr 23, 2006 at 13:29 +0200, Armin Rigo wrote: > On Sun, Apr 23, 2006 at 10:59:39AM +0200, holger krekel wrote: > > agreed to do at a specific meeting for EU developers on > > > > TUESDAY, 23rd April, 5pm (UTC+2) > > It will be the 25th, not the 23rd, unless I'm mistaken. indeed, it's stated correctly in the minutes themselves. thanks, holger From arigo at tunes.org Sun Apr 23 13:41:55 2006 From: arigo at tunes.org (Armin Rigo) Date: Sun Apr 23 13:41:58 2006 Subject: [pypy-dev] minutes pypy-sync last thursday In-Reply-To: <20060423113027.GM10825@solar.trillke> References: <20060423085939.GH10825@solar.trillke> <20060423112900.GA21389@code0.codespeak.net> <20060423113027.GM10825@solar.trillke> Message-ID: <20060423114155.GB23077@code0.codespeak.net> Hi Holger, On Sun, Apr 23, 2006 at 01:30:27PM +0200, holger krekel wrote: > > > TUESDAY, 23rd April, 5pm (UTC+2) > > > > It will be the 25th, not the 23rd, unless I'm mistaken. > > indeed, it's stated correctly in the minutes themselves. In the IRC log, yes, but not in the minutes -- both as e-mailed and as checked in. /me fixes... A bientot, Armin From stephan.diehl at gmx.net Sun Apr 23 13:57:45 2006 From: stephan.diehl at gmx.net (Stephan Diehl) Date: Sun Apr 23 13:58:25 2006 Subject: [pypy-dev] difference between py.py and pypy-c Message-ID: <200604231357.45202.stephan.diehl@gmx.net> Hallo, I found another difference in bahaviour between py.py and pypy-c with my set implementation enabled. The 'ior' operator in pypy-c does not behave as expected. My first thought was, that somehow the rdict.update method does not work as expected, but translator/goal/targetrdicttest2.py behaves, as it should, so there goes that theory. I have no idea whatsoever, where to look next. Stephan --------------------- py.py ----------------------------------------- [stephan@bach pypy]$ bin/py.py Loading grammar /home/stephan/projects/pypy-dist/pypy/interpreter/pyparser/data/Grammar2.5a faking faking fake-wrapping interp file ', mode 'w' at 0xb7f82068> fake-wrapping interp file ', mode 'w' at 0xb7f820b0> fake-wrapping interp file ', mode 'r' at 0xb7f82020> PyPy 0.8.0 in StdObjSpace on top of Python 2.4.1 (startuptime: 2.31 secs) >>>> w1 = 'test' >>>> w2 = 'stat' >>>> s = set(w1) >>>> t = set(w2) >>>> s set(['s', 'e', 't']) >>>> t set(['a', 's', 't']) >>>> s |= t >>>> s set(['a', 's', 'e', 't']) >>>> s = set(w1) >>>> s &= t >>>> s set(['s', 't']) >>>> --------------------- pypy-c --------------------------------------- [stephan@bach stephan]$ pypy-c debug: entry point starting debug: argv -> pypy-c debug: importing code debug: calling code.interact() Python 2.4.1 (pypy 0.8.0 build 26169) on linux2 Type "help", "copyright", "credits" or "license" for more information. (InteractiveConsole) >>>> w1 = 'test' >>>> w2 = 'stat' >>>> s = set(w1) >>>> t = set(w2) >>>> s set(['s', 'e', 't']) >>>> t set(['a', 's', 't']) >>>> s |= t >>>> s set(['s', 'e', 't']) >>>> s = set(w1) >>>> s &= t >>>> s set(['s', 't']) >>>> From nicolas.chauvat at logilab.fr Sun Apr 23 14:26:07 2006 From: nicolas.chauvat at logilab.fr (Nicolas Chauvat) Date: Sun Apr 23 14:26:39 2006 Subject: [pypy-dev] Re: [pypy-eu-svn] r26155 - pypy/extradoc/sprintinfo/tokyo In-Reply-To: <444B5916.4020901@changemaker.nu> References: <20060423031638.57E981007C@code0.codespeak.net> <20060423092128.GC24564@crater.logilab.fr> <444B5916.4020901@changemaker.nu> Message-ID: <20060423122607.GI24564@crater.logilab.fr> On Sun, Apr 23, 2006 at 12:38:14PM +0200, Beatrice During wrote: > I am also discussing this with Holger since it concerns packaging, tools > and such. Apparently Niibe-san is also interested in the py-lib as well > which needs some kind connected packaging..... There is a codespeak-pylib package in debian already, althought last time I checked it needed to be updated to the last version of pylib. -- Nicolas Chauvat logilab.fr - services en informatique avanc?e et gestion de connaissances From aurelien.campeas at free.fr Sun Apr 23 19:33:13 2006 From: aurelien.campeas at free.fr (=?ISO-8859-1?Q?Aur=E9lien_Camp=E9as?=) Date: Sun Apr 23 19:33:56 2006 Subject: [pypy-dev] Re: [pypy-eu-svn] r26155 - pypy/extradoc/sprintinfo/tokyo In-Reply-To: <444B5916.4020901@changemaker.nu> References: <20060423031638.57E981007C@code0.codespeak.net> <20060423092128.GC24564@crater.logilab.fr> <444B5916.4020901@changemaker.nu> Message-ID: <444BBA59.90707@free.fr> Beatrice During wrote: > Hi there > > Nicolas Chauvat skrev: >> Hi Sprinters, >> >> On Sun, Apr 23, 2006 at 05:16:38AM +0200, nik@codespeak.net wrote: >>> + - PyPy debian packaging (Yutaka, Holger?) >> >> Alexandre has an ITP on this package and he has spent some time >> thinking about it. You should probably ask him how far he went. > > Yes - I talked with Niibe-san about this and we agreed to check with > Alexandre first. I think Niibe-san will send an email..... > > I am also discussing this with Holger since it concerns packaging, tools > and such. Apparently Niibe-san is also interested in the py-lib as well > which needs some kind connected packaging..... > >> What about tasklet serialisation, any chance this could get done >> during the sprint ? It's becoming a blocker for WP09. Or should we >> tackle that on our side as soon as we get the chance ? >> > We are discussing this right now on various levels - there is a > dev-meeting on Tuesday 17:00 that will discuss the content of the 0.9 > release - it will be part of that. So it will be done before the Iceland > sprint, and finalized there. Please make sure you and/or your people > attend and explain your concerns. > > So - it is not a topic on this sprint. People here seem more interested > in Lisp and Javascript backends and sockets and other weird things ;-) Lisp backend ? I'd be interested in seeing that. Who did express such an idea btw ? - I don't remember having seen this talked about on pypy-dev or other places. > > Greetings from the 11th floor of the Dai-Biru of central Akihabara - we > are coding with an amazing view of neonsigns surrounding us....;-) > > Cheers > > Bea > _______________________________________________ > pypy-dev@codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > > From lac at strakt.com Sun Apr 23 19:37:51 2006 From: lac at strakt.com (Laura Creighton) Date: Sun Apr 23 19:38:33 2006 Subject: [pypy-dev] /pypy/irc-logs/pypy Message-ID: <200604231737.k3NHbphA013571@theraft.strakt.com> Can we rename them from DayMonthYear format to YearMonthDay format so they will list and sort in the correct order on the webpage? Laura From lac at strakt.com Sun Apr 23 19:40:01 2006 From: lac at strakt.com (Laura Creighton) Date: Sun Apr 23 19:40:39 2006 Subject: [pypy-dev] Re: /pypy/irc-logs/pypy In-Reply-To: Message from Laura Creighton of "Sun, 23 Apr 2006 19:37:51 +0200." <200604231737.k3NHbphA013571@theraft.strakt.com> References: <200604231737.k3NHbphA013571@theraft.strakt.com> Message-ID: <200604231740.k3NHe10l013663@theraft.strakt.com> In a message of Sun, 23 Apr 2006 19:37:51 +0200, Laura Creighton writes: >Can we rename them from DayMonthYear format to YearMonthDay format so >they will list and sort in the correct order on the webpage? > >Laura Upon looking further, I see we have started to do this. So all that is needed is clean-up. I'd do it if I had an account on the correct machine ... Laura From lac at strakt.com Sun Apr 23 19:42:37 2006 From: lac at strakt.com (Laura Creighton) Date: Sun Apr 23 19:43:19 2006 Subject: [pypy-dev] Re: [pypy-eu-svn] r26155 - pypy/extradoc/sprintinfo/tokyo In-Reply-To: Message from =?ISO-8859-1?Q?Aur=E9lien_Camp=E9as?= of "Sun, 23 Apr 2006 19:33:13 +0200." <444BBA59.90707@free.fr> References: <20060423031638.57E981007C@code0.codespeak.net> <20060423092128.GC24564@crater.logilab.fr> <444B5916.4020901@changemaker.nu> <444BBA59.90707@free.fr> Message-ID: <200604231742.k3NHgbFQ013775@theraft.strakt.com> In a message of Sun, 23 Apr 2006 19:33:13 +0200, Aur?lien Camp?as writes: >> So - it is not a topic on this sprint. People here seem more interested >> in Lisp and Javascript backends and sockets and other weird things ;-) > >Lisp backend ? I'd be interested in seeing that. Who did express such an >idea btw ? - I don't remember having seen this talked about on pypy-dev >or other places. sanxiyn has been working on this, but had to take time off for compulsory military service. He was able to make it to the Sprint in Japan . Laura From sanxiyn at gmail.com Tue Apr 25 01:48:05 2006 From: sanxiyn at gmail.com (Sanghyeon Seo) Date: Tue Apr 25 01:48:43 2006 Subject: [pypy-dev] Tokyo sprint status Message-ID: <5b0248170604241648h4a5afc7fn74d09316f3fe43af@mail.gmail.com> Hi, everybody! Greeting from Tokyo, You can read about past 2 days here: http://codespeak.net/pypy/extradoc/sprintinfo/tokyo/tokyo-planning.html Despite the file name, it's really records of planning session we have at the beginning of the day, so after that day, it's no more planning. :-) And the document is updated to match the reality then. I think "proper" sprint report will be here soon too. Yours, Sanghyeon From ac at strakt.com Tue Apr 25 06:21:56 2006 From: ac at strakt.com (=?ISO-8859-1?Q?Anders_Chrigstr=F6m?=) Date: Tue Apr 25 06:22:49 2006 Subject: [pypy-dev] Tokyo sprint status In-Reply-To: <5b0248170604241648h4a5afc7fn74d09316f3fe43af@mail.gmail.com> References: <5b0248170604241648h4a5afc7fn74d09316f3fe43af@mail.gmail.com> Message-ID: <444DA3E4.8040205@strakt.com> Sanghyeon Seo wrote: > Hi, everybody! Greeting from Tokyo, > > You can read about past 2 days here: > http://codespeak.net/pypy/extradoc/sprintinfo/tokyo/tokyo-planning.html We allso have some photos of the sprint at http://www.flickr.com/photos/19046555@N00/sets/72057594116388174/ /Arre From alexandre.fayolle at logilab.fr Tue Apr 25 10:44:21 2006 From: alexandre.fayolle at logilab.fr (Alexandre Fayolle) Date: Tue Apr 25 10:44:46 2006 Subject: [pypy-dev] [pypy-eu-svn] r26155 - pypy/extradoc/sprintinfo/tokyo In-Reply-To: <444B5916.4020901@changemaker.nu> References: <20060423031638.57E981007C@code0.codespeak.net> <20060423092128.GC24564@crater.logilab.fr> <444B5916.4020901@changemaker.nu> Message-ID: <20060425084421.GB13531@crater.logilab.fr> On Sun, Apr 23, 2006 at 12:38:14PM +0200, Beatrice During wrote: > Hi there > > Nicolas Chauvat skrev: > >Hi Sprinters, > > > >On Sun, Apr 23, 2006 at 05:16:38AM +0200, nik@codespeak.net wrote: > >>+ - PyPy debian packaging (Yutaka, Holger?) > > > >Alexandre has an ITP on this package and he has spent some time > >thinking about it. You should probably ask him how far he went. > > Yes - I talked with Niibe-san about this and we agreed to check with > Alexandre first. I think Niibe-san will send an email..... > > I am also discussing this with Holger since it concerns packaging, tools > and such. Apparently Niibe-san is also interested in the py-lib as well > which needs some kind connected packaging..... > > >What about tasklet serialisation, any chance this could get done > >during the sprint ? It's becoming a blocker for WP09. Or should we > >tackle that on our side as soon as we get the chance ? > > > We are discussing this right now on various levels - there is a > dev-meeting on Tuesday 17:00 that will discuss the content of the 0.9 > release - it will be part of that. So it will be done before the Iceland > sprint, and finalized there. Please make sure you and/or your people > attend and explain your concerns. I'll try to attend this meeting. Is the 17:00 time in EST/UTC or some other TZ ? The IRC channel is #pypy ? -- Alexandre Fayolle LOGILAB, Paris (France). http://www.logilab.com http://www.logilab.fr http://www.logilab.org -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 481 bytes Desc: Digital signature Url : http://codespeak.net/pipermail/pypy-dev/attachments/20060425/feefb4f2/attachment-0001.pgp From sanxiyn at gmail.com Tue Apr 25 12:37:38 2006 From: sanxiyn at gmail.com (Sanghyeon Seo) Date: Tue Apr 25 12:38:18 2006 Subject: [pypy-dev] Question on snake server test runs Message-ID: <5b0248170604250337m292aaffub6a027df9ce9eaff@mail.gmail.com> http://snake.cs.uni-duesseldorf.de/pypytest/summary.html I added a py.test option to pretty-print Common Lisp source files, and tests on snake server started to fail. Samuele told me that it might depend on the working directory from which py.test is run, but no matter how I try, I couldn't reproduce the failure in my machine. So my question is: how are tests run on snake server? How can I fix this failure? E if conftest.option.prettyprint: > AttributeError: Values instance has no attribute 'prettyprint' Seo Sanghyeon From hpk at trillke.net Tue Apr 25 15:06:42 2006 From: hpk at trillke.net (holger krekel) Date: Tue Apr 25 15:13:24 2006 Subject: [pypy-dev] Question on snake server test runs In-Reply-To: <5b0248170604250337m292aaffub6a027df9ce9eaff@mail.gmail.com> References: <5b0248170604250337m292aaffub6a027df9ce9eaff@mail.gmail.com> Message-ID: <20060425130642.GV10825@solar.trillke> Hi Seo! On Tue, Apr 25, 2006 at 19:37 +0900, Sanghyeon Seo wrote: > http://snake.cs.uni-duesseldorf.de/pypytest/summary.html > > I added a py.test option to pretty-print Common Lisp source files, and > tests on snake server started to fail. you are definining the new option in pypy/translator/cl/conftest.py but this file is not considered for global py.test runs. Considering all conftest.py files recursively in a tree would mean that even for a "py.test --help" you would have to wait too long. The alternative is to allow a conftest.py file to specify a list of relative "dependent" conftest.py files so that we can get rid of this problem. For now, you could just move your extra option to the more global pypy/conftest.py. best, holger From anto.cuni at gmail.com Tue Apr 25 17:06:09 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue Apr 25 17:09:31 2006 Subject: [pypy-dev] Question on snake server test runs In-Reply-To: <5b0248170604250337m292aaffub6a027df9ce9eaff@mail.gmail.com> References: <5b0248170604250337m292aaffub6a027df9ce9eaff@mail.gmail.com> Message-ID: <444E3AE1.9000801@gmail.com> Hi Sanghyeon, Sanghyeon Seo wrote: > http://snake.cs.uni-duesseldorf.de/pypytest/summary.html > > I added a py.test option to pretty-print Common Lisp source files, and > tests on snake server started to fail. Samuele told me that it might > depend on the working directory from which py.test is run, but no > matter how I try, I couldn't reproduce the failure in my machine. > > So my question is: how are tests run on snake server? How can I fix > this failure? > > E if conftest.option.prettyprint: >> AttributeError: Values instance has no attribute 'prettyprint' I had the same problem with gencli: I solved using an helper function that catches AttributeError and returns a default value instead, even if it seems more a hack than a solution. See translator/cli/option.py. ciao Anto From rodsenra at gpr.com.br Tue Apr 25 17:53:16 2006 From: rodsenra at gpr.com.br (Rodrigo Dias Arruda Senra) Date: Tue Apr 25 17:54:02 2006 Subject: [pypy-dev] security prototype & workshop plans/ideas In-Reply-To: <20060413133839.GF10825@solar.trillke> References: <20060413133839.GF10825@solar.trillke> Message-ID: <20060425125316.18502c7e@Fenix.gpr.com.br> Some *very* late shameless comments ;o) [ hpk@trillke.net (holger krekel) ]: ---------------------------------------- | | Hi folks, | | from the EU side of things there is the plan to organize a | security workshop and implement security features within PyPy. #cut | | - data tagging or "label control", or more generally attaching | (security) metainformations to a python object and having those | propagate through the program automatically. See e.g. | #cut | Label control could be used for tagging e.g. user-level input | with the "untrusted" label and then protecting certain | functions to require trusted input (e.g. database/file | modifications). Then, there could be explicit untrusted_to_trusted() | conversions, turning an untrusted input into a trusted | output. This would allow to concisely localise how | user-supplied/untrusted input is parsed and checked. | # cut | | The challenge is to find an interesting mechanism that | elegantly enables such approaches - which should be the | topic of our upcoming security prototype and workshop. # cut | I am posting here on pypy-dev (rather than just to selected pypy | developers) because others may be interested, have comments, | suggestions or might think about contributing. Security is | certainly not the central topic of PyPy but our design should | make it considerably easier to implement strong security features. | Hum, and i guess that it's not impossible that the project | might for contributors come up with funding for travels at | least. | The Guarana MOP [1] might provide some inspiration, since Guarana was a reflective meta-object protocol meant to be secure. The core idea was to provide a VM-level hook where every access to an object would test for the presence of a meta-object, using a pointer in the underlying object representation structure. If the meta-object was present, access would be intercepted and delivered to the meta-object instead. An image [2, 3] worth a thousand words would show it faster. The key points were: - changing the meta-object bound to some object was a negotiation process, where the consent of the installed meta-object was required - the meta-object controlled all access to the underlying object - the meta-object could be a composer delegating decisions to a hierarchy of other meta-objects. I do not know if any of these ideas could/should be used in Pypy for the challenge proposed by Holger. Nevertheless, it is harmless to suggest ;o) [1] http://citeseer.ist.psu.edu/oliva98reflexive.html [2] http://www.students.ic.unicamp.br/~921234/dissert/images/basic_interaction.jpg [3] http://www.students.ic.unicamp.br/~921234/dissert/images/reflective_hook.jpg best regards, Rod Senra http://rodrigo.senra.nom.br From arigo at tunes.org Wed Apr 26 19:50:21 2006 From: arigo at tunes.org (Armin Rigo) Date: Wed Apr 26 19:50:23 2006 Subject: [pypy-dev] Question on snake server test runs In-Reply-To: <5b0248170604250337m292aaffub6a027df9ce9eaff@mail.gmail.com> References: <5b0248170604250337m292aaffub6a027df9ce9eaff@mail.gmail.com> Message-ID: <20060426175021.GA12761@code0.codespeak.net> Hi Seo, On Tue, Apr 25, 2006 at 07:37:38PM +0900, Sanghyeon Seo wrote: > E if conftest.option.prettyprint: > > AttributeError: Values instance has no attribute 'prettyprint' It's hard to reproduce because it doesn't depend on the current working directory, but on the directory specified as the starting point to look for tests. One way to reproduce it is: in pypy/translator: py.test -k test_cl (The -k causes all tests files or names not starting with test_cl to be skipped to make it a bit faster.) The reason for the problem is that you're importing the wrong conftest: the one at pypy level instead of your own. Your option shows up on the object pypy.translator.cl.conftest.option, which I think is not the same as pypy.conftest.option -- at least not always (I'm not sure I understand the precise logic here :-) (I had to try things a bit before figuring this out, so now I've just checked in the result.) A bientot, Armin. From arigo at tunes.org Wed Apr 26 21:33:35 2006 From: arigo at tunes.org (Armin Rigo) Date: Wed Apr 26 21:33:36 2006 Subject: [pypy-dev] difference between py.py and pypy-c In-Reply-To: <200604231357.45202.stephan.diehl@gmx.net> References: <200604231357.45202.stephan.diehl@gmx.net> Message-ID: <20060426193335.GA20579@code0.codespeak.net> Hi Stephan, On Sun, Apr 23, 2006 at 01:57:45PM +0200, Stephan Diehl wrote: > I found another difference in bahaviour between py.py and pypy-c with my set > implementation enabled. Sorry for taking so long to investigate. It turned out to be a bug indirectly related to inlining, which caused wrong code to be generated. In short the calls to _union_dict() with isupdate=True were inlined, and the inlined version was incorrectly following the isupdate=False path... It should be fixed now. A bientot, Armin. From jiwon at stanford.edu Wed Apr 26 23:46:40 2006 From: jiwon at stanford.edu (Jiwon Seo) Date: Wed Apr 26 23:47:25 2006 Subject: [pypy-dev] Error While Translating Interpreter Message-ID: Hi, I'm getting error while translating pypy interpreter (in pypy-0.8.x version) on ppc machine. The machine is Macintosh G5, and OS is Fedora Core 4; and the error code looks like following: [translation:ERROR] Error: [translation:ERROR] Traceback (most recent call last): [translation:ERROR] File "./translate_pypy.py", line 278, in main [translation:ERROR] drv.proceed(goals) [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/translator/goal/drive [translation:ERROR] self._execute(goals, task_skip = self.maybe_skip) [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/translator/tool/taske [translation:ERROR] self._do(goal, taskcallable, *args, **kwds) [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/translator/goal/drive [translation:ERROR] func() [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/translator/goal/drive [translation:ERROR] annotator = translator.annotate(self.inputtypes, policy= [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/translator/translator [translation:ERROR] self.annotator.build_types(graph, input_args_types, func [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/translator/annrpython [translation:ERROR] self.complete() [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/translator/annrpython [translation:ERROR] self.processblock(fn, block) [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/translator/annrpython [translation:ERROR] self.flowin(fn, block) [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/translator/annrpython [translation:ERROR] self.consider_op(block.operations[i]) [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/translator/annrpython [translation:ERROR] resultcell = consider_meth(*argcells) [translation:ERROR] File "", line 3, in consider_op_getattr [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/annotation/unaryop.py [translation:ERROR] attrdef = ins.classdef.find_attribute(attr) [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/annotation/classdef.p [translation:ERROR] return self.locate_attribute(attr).attrs[attr] [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/annotation/classdef.p [translation:ERROR] self.generalize_attr(attr) [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/annotation/classdef.p [translation:ERROR] self._generalize_attr(attr, s_value) [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/annotation/classdef.p [translation:ERROR] newattr.add_constant_source(source, classdef) [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/annotation/classdef.p [translation:ERROR] s_value = self.bookkeeper.immutablevalue( [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/annotation/bookkeeper [translation:ERROR] clsdef.add_source_for_attribute(attr, x) # can trigger r [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/annotation/classdef.p ribute [translation:ERROR] attrdef.add_constant_source(source, clsdef) [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/annotation/classdef.p [translation:ERROR] s_value = self.bookkeeper.immutablevalue( [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/annotation/bookkeeper [translation:ERROR] result.dictdef.generalize_value(self.immutablevalue(ev)) [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/annotation/bookkeeper [translation:ERROR] clsdef.add_source_for_attribute(attr, x) # can trigger r [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/annotation/classdef.p ribute [translation:ERROR] attrdef.add_constant_source(source, clsdef) [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/annotation/classdef.p [translation:ERROR] s_value = self.bookkeeper.immutablevalue( [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/annotation/bookkeeper [translation:ERROR] result.dictdef.generalize_value(self.immutablevalue(ev)) [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/annotation/bookkeeper [translation:ERROR] frozen = hasattr(x, '_freeze_') and x._freeze_() [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/interpreter/mixedmodu [translation:ERROR] self.getdict() [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/interpreter/mixedmodu [translation:ERROR] w_value = self.get(name) [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/interpreter/mixedmodu [translation:ERROR] w_value = self.getdictvalue(space, name) [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/interpreter/mixedmodu [translation:ERROR] w_value = loader(space) [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/interpreter/mixedmodu [translation:ERROR] return app.wget(space, attrname) [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/interpreter/gateway.p [translation:ERROR] w_globals = self.getwdict(space) [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/interpreter/gateway.p [translation:ERROR] return space.fromcache(ApplevelCache).getorbuild(self) [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/tool/cache.py", line [translation:ERROR] result = self._build(key) [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/interpreter/baseobjsp [translation:ERROR] return self.build(key) [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/interpreter/gateway.p [translation:ERROR] return PyPyCacheDir.build_applevelinterp_dict(app, self. [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/interpreter/gateway.p rp_dict [translation:ERROR] w_glob = initfunc(space) [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/_cache/app_array_f5e7 ine 4006, in init__builtin__ [translation:ERROR] glong_minus_2147483648 = long_helper('x80000000') [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/_cache/app_array_f5e7 ine 4005, in long_helper [translation:ERROR] return space.eval("long(%r, 16)" % value, dic, dic) [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/interpreter/baseobjsp [translation:ERROR] return expression.exec_code(self, w_globals, w_locals) [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/interpreter/eval.py", [translation:ERROR] return frame.run() [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/interpreter/eval.py", [translation:ERROR] result = self.eval(executioncontext) [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/interpreter/pyframe.p [translation:ERROR] ctlflowexc.action(self) [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/interpreter/pyframe.p [translation:ERROR] ControlFlowException.action(self, frame) [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/interpreter/pyframe.p [translation:ERROR] self.emptystack(frame) [translation:ERROR] File "/u/jiwon/proj/pypy-0.8.x/pypy/interpreter/pyframe.p [translation:ERROR] raise self.operr [translation:ERROR] OperationError: [: W_StringObject 0000')] [translation:ERROR] Processing block: [translation:ERROR] block@228 is a SpamBlock in the graph of Module.pick_builti [translation:ERROR] at pypy.module.__builtin__:118 [translation:ERROR] containing the following operations: [translation:ERROR] v2 = simple_call((type Module), space_0, (None)) [translation:ERROR] v3 = getattr(space_0, ('setitem')) [translation:ERROR] v4 = getattr(v2, ('w_dict')) [translation:ERROR] v5 = getattr(space_0, ('wrap')) [translation:ERROR] v6 = simple_call(v5, ('None')) [translation:ERROR] v7 = getattr(space_0, ('w_None')) [translation:ERROR] v8 = simple_call(v3, v4, v6, v7) [translation:ERROR] --end-- Any idea what's going on? or does translator run ok on ppc linux? or do you advise me to run it with latest version of pypy? Thanks, -Jiwon From arigo at tunes.org Thu Apr 27 09:59:09 2006 From: arigo at tunes.org (Armin Rigo) Date: Thu Apr 27 09:59:12 2006 Subject: [pypy-dev] Re: [pypy-svn] r26303 - pypy/dist/pypy/doc/discussion In-Reply-To: <20060425072552.2A60110078@code0.codespeak.net> References: <20060425072552.2A60110078@code0.codespeak.net> Message-ID: <20060427075909.GA30800@code0.codespeak.net> Hi all, On Tue, Apr 25, 2006 at 09:25:52AM +0200, tismer@codespeak.net wrote: > Added: > pypy/dist/pypy/doc/discussion/howtoimplementpickling.txt (contents, props changed) > Log: > some mockup about thread pickling, not really complete, yet. Last Tuesday we decided to discuss this more on an IRC meeting, with Christian, Michael, Samuele and myself, plus of course any other interested people. Christian, can you check in the extra details you promized? I wanted to make sure we would discuss this on the basis of previously-written details. I'll anyway propose that we meet tomorrow Friday, again at 5pm CET. A bientot, Armin From anto.cuni at gmail.com Thu Apr 27 11:05:27 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu Apr 27 11:08:57 2006 Subject: [pypy-dev] PyPy is flooding Genova :-) Message-ID: <44508957.2040705@gmail.com> Hi all, as the subject suggests, people seems to get interested in PyPy here at Genova's university, due to my thesis :-). As a consequence next week I'll probably have a talk for introducing some interested people to PyPy. Since there is a number of introductory presentations I was thinking of using one of those instead of writing yet another one; the more updated seems to be the accu2006 one, right? I've noticed that it has been written with Keynote: can someone send it me in a format I can open with OpenOffice or PowerPoint, so that I can add some slides specific to the CLI backend, please? Moreover my supervising professor is considering the possibility to establish a more actual collaboration between my university and the PyPy project, so he would be glad to speak with someone of the core team about this. ciao Anto From mwh at python.net Thu Apr 27 12:03:37 2006 From: mwh at python.net (Michael Hudson) Date: Thu Apr 27 12:04:29 2006 Subject: [pypy-dev] Re: PyPy is flooding Genova :-) References: <44508957.2040705@gmail.com> Message-ID: <2mpsj3fi86.fsf@starship.python.net> Antonio Cuni writes: > Hi all, > as the subject suggests, people seems to get interested in PyPy here > at Genova's university, due to my thesis :-). Cool! > As a consequence next week I'll probably have a talk for introducing > some interested people to PyPy. > Since there is a number of introductory presentations I was thinking > of using one of those instead of writing yet another one; the more > updated seems to be the accu2006 one, right? Probably yes. If you have a more getting started with coding focus, my sprint introduction from the PyCon sprint may be better. > I've noticed that it has been written with Keynote: can someone send > it me in a format I can open with OpenOffice or PowerPoint, so that I > can add some slides specific to the CLI backend, please? Oh right, yes, I'll check in ppts of both of these. When I tried before exporting ppts and importing into OpenOffice had some issues with things like arrow heads, but I don't know if it was the import or the export that was bad :) (I don't have PowerPoint). They basically worked, anyway. > Moreover my supervising professor is considering the possibility to > establish a more actual collaboration between my university and the > PyPy project, so he would be glad to speak with someone of the core > team about this. Very cool! Cheers, mwh -- INEFFICIENT CAPITALIST YOUR OPULENT TOILET WILL BE YOUR UNDOING -- from Twisted.Quotes From l.oluyede at gmail.com Thu Apr 27 14:09:57 2006 From: l.oluyede at gmail.com (Lawrence Oluyede) Date: Thu Apr 27 14:10:36 2006 Subject: [pypy-dev] Re: [pypy-svn] r26432 - pypy/dist/pypy/doc In-Reply-To: <20060427102546.BEC55100B6@code0.codespeak.net> References: <20060427102546.BEC55100B6@code0.codespeak.net> Message-ID: <9eebf5740604270509m7f1f5348x7a8904646410cf4c@mail.gmail.com> > +* Rewrite one or several CPython extension modules to be based on **ctypes** > + (newly integrated in Python 2.5): this is generally useful for Python > + developpers, and it is now the best path to write extension modules that are > + compatible with both CPython and PyPy. See for example > + http://wiki.python.org/moin/CodingProjectIdeas/PygameOnCtypes . A related > + idea is to provide efficient numeric arrays (as in numeric/numpy/numarray) > + in this way. How a SoC (2 months longer) task can be? I'd like to apply for this kind of project but I'd like to know more about it too. We can discuss about 5, or 10 or so modules that PyPy needs? Or it's up to the student to choose them? Thanks in advance. -- Lawrence http://www.oluyede.org/blog From anto.cuni at gmail.com Thu Apr 27 16:05:49 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu Apr 27 16:09:26 2006 Subject: [pypy-dev] Summer of code Message-ID: <4450CFBD.4040204@gmail.com> Hi all, I'm considering the possibility of applying to Google Summer of Code 2006: obviously the topic of my application would be pypy :-). As you can guess I'd like to continue working on the CLI backend, also considering that I probably won't be able to finish it before I graduate. The point is that by now I can't know how mature gencli will be when soc starts, so it's difficult to write a good proposal: how can I say where I'll arrive to if I don't know where I have to start from? I could submit a vague proposal, but I guess that something like "working on the CLI backend" is a bit too elusive for being accepted. Any suggestions? ciao Anto From anto.cuni at gmail.com Thu Apr 27 20:27:32 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu Apr 27 20:31:06 2006 Subject: [pypy-dev] Re: PyPy is flooding Genova :-) In-Reply-To: <2mpsj3fi86.fsf@starship.python.net> References: <44508957.2040705@gmail.com> <2mpsj3fi86.fsf@starship.python.net> Message-ID: <44510D14.2000004@gmail.com> Hi Michael, Michael Hudson wrote: >> I've noticed that it has been written with Keynote: can someone send >> it me in a format I can open with OpenOffice or PowerPoint, so that I >> can add some slides specific to the CLI backend, please? > > Oh right, yes, I'll check in ppts of both of these. When I tried > before exporting ppts and importing into OpenOffice had some issues > with things like arrow heads, but I don't know if it was the import or > the export that was bad :) (I don't have PowerPoint). They basically > worked, anyway. I just tried to open them with Openoffice and it seems to work fine, thanks. >> Moreover my supervising professor is considering the possibility to >> establish a more actual collaboration between my university and the >> PyPy project, so he would be glad to speak with someone of the core >> team about this. > > Very cool! Are there any volunteers for this? :-) ciao Anto From tjreedy at udel.edu Thu Apr 27 21:52:17 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Thu Apr 27 21:53:33 2006 Subject: [pypy-dev] Re: Summer of code References: <4450CFBD.4040204@gmail.com> Message-ID: "Antonio Cuni" wrote in message news:4450CFBD.4040204@gmail.com... > Hi all, > I'm considering the possibility of applying to Google Summer of Code > 2006: obviously the topic of my application would be pypy :-). > As you can guess I'd like to continue working on the CLI backend, also > considering that I probably won't be able to finish it before I graduate. I thought it was your thesis project, which you would need to finish. In any case, assuming you do not already have a summer stipend for the same work, I would encourage you to apply -- after reading the FAQ carefully. > The point is that by now I can't know how mature gencli will be when soc > starts, so it's difficult to write a good proposal: how can I say where > I'll arrive to if I don't know where I have to start from? > > I could submit a vague proposal, but I guess that something like "working > on the CLI backend" is a bit too elusive for being accepted. Agreed > Any suggestions? In a couple of sentences, describe PyPy in relation to Python and link to site. Describe your CLI (what is that?) backend project and how it fits into PyPy and why it is a useful thing (to other people) to do. List what you have done (and when you began) up to application date. Then list your next several steps. Indicate what you anticipate doing before the project starts and what you anticipate doing during the project. (I think the FAQ addresses the question of starting 'early' -- after approval but before the official start date -- but forget the answer. I recommend you find it.) If you think needed, add a caveat about minor adjustments of schedule. Mention where code is being deposited and if publicly accessible. If your CLI backend is already approved in principle (when sufficiently well done) as one of the PyPy backends, say that too. And make sure your proposed mentor(s) have contacted Neal to get URL to signup with Google. Good luck and good coding. Terry Jan Reedy From anto.cuni at gmail.com Thu Apr 27 22:55:05 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu Apr 27 22:59:01 2006 Subject: [pypy-dev] Re: Summer of code In-Reply-To: References: <4450CFBD.4040204@gmail.com> Message-ID: <44512FA9.3040202@gmail.com> Hi Terry, Terry Reedy wrote: > I thought it was your thesis project, which you would need to finish. In > any case, assuming you do not already have a summer stipend for the same > work, I would encourage you to apply -- after reading the FAQ carefully. It is my thesis project, but I don't need to finish: my supervising professor is happy for my work and told me to code as much as I can, but fortunately I have no mandatory goal to reach. This doesn't mean that I'll abandon gencli as soon as I graduate: I'd like to finish my work at best, and if I can get payed is much better! :-) > In a couple of sentences, describe PyPy in relation to Python and link to > site. Describe your CLI (what is that?) backend project and how it fits > into PyPy and why it is a useful thing (to other people) to do. List what > you have done (and when you began) up to application date. Then list your > next several steps. Indicate what you anticipate doing before the project > starts and what you anticipate doing during the project. (I think the FAQ > addresses the question of starting 'early' -- after approval but before the > official start date -- but forget the answer. I recommend you find it.) > If you think needed, add a caveat about minor adjustments of schedule. > > Mention where code is being deposited and if publicly accessible. If your > CLI backend is already approved in principle (when sufficiently well done) > as one of the PyPy backends, say that too. And make sure your proposed > mentor(s) have contacted Neal to get URL to signup with Google. Thanks for the suggestions, they will be useful. I read the student FAQ but I missed the one of starting early: it seems that it's fine, so there should be no problem for this. ciao Anto From stephan.diehl at gmx.net Fri Apr 28 10:12:16 2006 From: stephan.diehl at gmx.net (Stephan Diehl) Date: Fri Apr 28 10:12:54 2006 Subject: [pypy-dev] difference between py.py and pypy-c In-Reply-To: <20060426193335.GA20579@code0.codespeak.net> References: <200604231357.45202.stephan.diehl@gmx.net> <20060426193335.GA20579@code0.codespeak.net> Message-ID: <200604281012.16045.stephan.diehl@gmx.net> Hi Armin, On Wednesday 26 April 2006 21:33, Armin Rigo wrote: > Hi Stephan, > > On Sun, Apr 23, 2006 at 01:57:45PM +0200, Stephan Diehl wrote: > > I found another difference in bahaviour between py.py and pypy-c with my > > set implementation enabled. > > Sorry for taking so long to investigate. Please don't worry. I'm actually amazed how fast you always come up with a solution (and do tons of other stuff as well). > > It turned out to be a bug > indirectly related to inlining, which caused wrong code to be generated. > In short the calls to _union_dict() with isupdate=True were inlined, and > the inlined version was incorrectly following the isupdate=False path... > It should be fixed now. It is fixed. Thanks a lot. Cheers Stephan > > > A bientot, > > Armin. From arigo at tunes.org Fri Apr 28 12:20:26 2006 From: arigo at tunes.org (Armin Rigo) Date: Fri Apr 28 12:20:28 2006 Subject: [pypy-dev] Error While Translating Interpreter In-Reply-To: References: Message-ID: <20060428102026.GA32352@code0.codespeak.net> Hi Jiwon, On Wed, Apr 26, 2006 at 02:46:40PM -0700, Jiwon Seo wrote: > [translation:ERROR] glong_minus_2147483648 = long_helper('x80000000') This bug was fixed in the meantime, so yes, you should use the current PyPy version. Actually the bug only showed up on CPython 2.5. It's caused by the new AST compiler (a CPython "bug" or change that wasn't fixed so far). Are you using 2.5 already? There are other problems with 2.5 in PyPy... A bientot, Armin. From arigo at tunes.org Fri Apr 28 13:29:19 2006 From: arigo at tunes.org (Armin Rigo) Date: Fri Apr 28 13:29:20 2006 Subject: [pypy-dev] Re: PyPy is flooding Genova :-) In-Reply-To: <44510D14.2000004@gmail.com> References: <44508957.2040705@gmail.com> <2mpsj3fi86.fsf@starship.python.net> <44510D14.2000004@gmail.com> Message-ID: <20060428112919.GB32352@code0.codespeak.net> Hi Antonio, On Thu, Apr 27, 2006 at 08:27:32PM +0200, Antonio Cuni wrote: > >>Moreover my supervising professor is considering the possibility to > >>establish a more actual collaboration between my university and the > >>PyPy project, so he would be glad to speak with someone of the core > >>team about this. > > > >Very cool! Very cool indeed! > Are there any volunteers for this? :-) Although Samuele may be the closest geographically when he's home, I guess I'd be interested too. Of course, convincing him to come to EuroPython might also be a good course of action :-) A bientot, Armin. From arigo at tunes.org Fri Apr 28 13:45:12 2006 From: arigo at tunes.org (Armin Rigo) Date: Fri Apr 28 13:45:15 2006 Subject: [pypy-dev] Re: [pypy-svn] r26432 - pypy/dist/pypy/doc In-Reply-To: <9eebf5740604270509m7f1f5348x7a8904646410cf4c@mail.gmail.com> References: <20060427102546.BEC55100B6@code0.codespeak.net> <9eebf5740604270509m7f1f5348x7a8904646410cf4c@mail.gmail.com> Message-ID: <20060428114512.GC32352@code0.codespeak.net> Hi Lawrence, On Thu, Apr 27, 2006 at 02:09:57PM +0200, Lawrence Oluyede wrote: > > +* Rewrite one or several CPython extension modules to be based on **ctypes** > > + (newly integrated in Python 2.5): this is generally useful for Python > > + developpers, and it is now the best path to write extension modules that are > > + compatible with both CPython and PyPy. See for example > > + http://wiki.python.org/moin/CodingProjectIdeas/PygameOnCtypes . A related > > + idea is to provide efficient numeric arrays (as in numeric/numpy/numarray) > > + in this way. > > How a SoC (2 months longer) task can be? I'd like to apply for this > kind of project but I'd like to know more about it too. We can discuss > about 5, or 10 or so modules that PyPy needs? Or it's up to the > student to choose them? I would expect the Pygame-on-ctypes proposal to be a full two-months work by itself, but tackling smaller modules might indeed allow 5 or more of them to be done. About the modules choice, yes, it's mostly up to the student. There are many built-in CPython modules that we are missing, and thus having them would be nice, but on the other hand some non-built-in modules are just cooler and more widely used (I'm a Pygame fan :-). For reference, the built-in modules that we have (or at least started to work on) are in http://codespeak.net/svn/pypy/dist/pypy/module or in http://codespeak.net/svn/pypy/dist/pypy/lib if they've been reimplemented purely at app-level. Looking at the list of CPython 2.4 modules, I find a lot of small or medium useful built-in modules that PyPy is missing -- a personal selection in alphabetical order: * _curses * _ssl * bz2 * fcntl * mmap * os (i.e. posix/nt, partially done) * readline (a topic by itself: write or find existing readline replacements in pure Python, also useful for CPython) * select * signal * termios * time (partially done) * zipimport * zlib Feel free to propose any choice, from this list or not, here or on #pypy. A bientot, Armin. From seojiwon at gmail.com Fri Apr 28 17:05:26 2006 From: seojiwon at gmail.com (Jiwon Seo) Date: Fri Apr 28 17:06:07 2006 Subject: [pypy-dev] Error While Translating Interpreter In-Reply-To: <20060428102026.GA32352@code0.codespeak.net> References: <20060428102026.GA32352@code0.codespeak.net> Message-ID: Hi, > This bug was fixed in the meantime, so yes, you should use the current > PyPy version. > > Actually the bug only showed up on CPython 2.5. It's caused by the new > AST compiler (a CPython "bug" or change that wasn't fixed so far). Are > you using 2.5 already? There are other problems with 2.5 in PyPy... No. Actually, it works fine now. Now I have to look into translated code since the simulator that I'm linking pypy against and running on is very unstable and gives me segfault. Any ideas? :) -Jiwon From l.oluyede at gmail.com Fri Apr 28 17:48:56 2006 From: l.oluyede at gmail.com (Lawrence Oluyede) Date: Fri Apr 28 17:49:33 2006 Subject: [pypy-dev] Re: [pypy-svn] r26432 - pypy/dist/pypy/doc In-Reply-To: <20060428114512.GC32352@code0.codespeak.net> References: <20060427102546.BEC55100B6@code0.codespeak.net> <9eebf5740604270509m7f1f5348x7a8904646410cf4c@mail.gmail.com> <20060428114512.GC32352@code0.codespeak.net> Message-ID: <9eebf5740604280848x58dc5ccaq1e87e9184c23a9f4@mail.gmail.com> > I would expect the Pygame-on-ctypes proposal to be a full two-months > work by itself, I read the page about the pygame-on-ctypes project (I like the Pygame idea too) but what the pygame crew think about it? I tried to ask on #pygame some days ago but they didn't seem really interested (it's preferable doing something *really* useful...) > but tackling smaller modules might indeed allow 5 or > more of them to be done. That's great. > About the modules choice, yes, it's mostly up > to the student. There are many built-in CPython modules that we are > missing, and thus having them would be nice, but on the other hand some > non-built-in modules are just cooler and more widely used (I'm a Pygame > fan :-). I know (I remember something from your psyco-python-pygame presentation) > * _curses > * _ssl > * bz2 > * fcntl > * mmap > * os (i.e. posix/nt, partially done) > * readline (a topic by itself: write or find existing readline > replacements in pure Python, also useful for CPython) > * select > * signal > * termios > * time (partially done) > * zipimport > * zlib I'll take a look at them and try to decide which and how many. If you (or other PyPy dev) have some preferences (or priorities) let me know! > Feel free to propose any choice, from this list or not, here or on > #pypy. I'm often there. rhymes is my nick :) -- Lawrence http://www.oluyede.org/blog From arigo at tunes.org Fri Apr 28 19:04:20 2006 From: arigo at tunes.org (Armin Rigo) Date: Fri Apr 28 19:04:20 2006 Subject: [pypy-dev] Error While Translating Interpreter In-Reply-To: References: <20060428102026.GA32352@code0.codespeak.net> Message-ID: <20060428170420.GA23736@code0.codespeak.net> Hi Jiwon, On Fri, Apr 28, 2006 at 08:05:26AM -0700, Jiwon Seo wrote: > No. Actually, it works fine now. Now I have to look into translated > code since the simulator that I'm linking pypy against and running on > is very unstable and gives me segfault. Any ideas? :) It's hard to separate a priori between the five things that are special in your case :-) It could be some incompatibility with the simulator, or the fact that PyPy cannot cross-translate itself -- the result only works on the same host as the one used for translation, but a simulator is a new virtual host. Or it could simply be your platform, or the fact that we never tried to link PyPy against external stuff, or just plainly bugs that we fixed since the 0.8 release... A bientot, Armin. From nnorwitz at gmail.com Fri Apr 28 19:37:11 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Fri Apr 28 19:37:53 2006 Subject: [pypy-dev] Summer of Code mailing list Message-ID: There's a new SoC mailing list. soc2006@python.org You can sign up here: http://mail.python.org/mailman/listinfo/soc2006 This list is for any SoC discussion: mentors, students, idea, etc. Student can submit applications starting May 1, so now is the time to get students interested in your ideas! Please pass this information along. Cheers, n From tismer at stackless.com Fri Apr 28 20:10:48 2006 From: tismer at stackless.com (Christian Tismer) Date: Fri Apr 28 20:11:19 2006 Subject: [pypy-dev] missed the meeting Message-ID: <44525AA8.5020306@stackless.com> Hi PyPyers, for some reason I was sure that we cancelled this week's sync meeting. This was absolutely unintentional, very sorry about that! cheers - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From arigo at tunes.org Sat Apr 29 21:15:27 2006 From: arigo at tunes.org (Armin Rigo) Date: Sat Apr 29 21:15:28 2006 Subject: [pypy-dev] missed the meeting In-Reply-To: <44525AA8.5020306@stackless.com> References: <44525AA8.5020306@stackless.com> Message-ID: <20060429191527.GA15749@code0.codespeak.net> Hi Christian, On Fri, Apr 28, 2006 at 11:10:48AM -0700, Christian Tismer wrote: > for some reason I was sure that we cancelled this week's > sync meeting. This was absolutely unintentional, > very sorry about that! It was not a normal sync meeting. I called for a stackless meeting on Friday, as we decided on Tuesday. I wouldn't hide the fact that we were a bit disappointed not to see you there, and also not to see any docs checked in, countrary to what you promized... We talked a bit about ways to go forward but without deciding anything yet, given that we are not sure what exactly you have in mind about the hardest problems we foresee. Now Michael is on holidays, and people are travelling back from the sprint, so we'll try again maybe on Monday... Armin From arigo at tunes.org Sat Apr 29 22:50:47 2006 From: arigo at tunes.org (Armin Rigo) Date: Sat Apr 29 22:50:49 2006 Subject: [pypy-dev] llinterp trace Message-ID: <20060429205047.GA21312@code0.codespeak.net> Hi all, As a quick hack I made llinterp produce traces of all operations it executes -- no longer to stdout or py.log, but to an html file written to /tmp/usession-*. It's a hierarchical-looking page showing the nested calls. With Javascript browsers we can hide and show sub-calls by clicking around; all calls are initially hidden. It's certainly a more useful presentation than the terminal-based one. Of course I'm open to further suggestions :-) A bientot, Armin. From tismer at stackless.com Mon May 1 01:54:05 2006 From: tismer at stackless.com (Christian Tismer) Date: Mon May 1 01:54:40 2006 Subject: [pypy-dev] missed the meeting In-Reply-To: <20060429191527.GA15749@code0.codespeak.net> References: <44525AA8.5020306@stackless.com> <20060429191527.GA15749@code0.codespeak.net> Message-ID: <44554E1D.4010007@stackless.com> Armin Rigo wrote: > Hi Christian, > > On Fri, Apr 28, 2006 at 11:10:48AM -0700, Christian Tismer wrote: >> for some reason I was sure that we cancelled this week's >> sync meeting. This was absolutely unintentional, >> very sorry about that! > > It was not a normal sync meeting. I called for a stackless meeting on > Friday, as we decided on Tuesday. I wouldn't hide the fact that we were > a bit disappointed not to see you there, and also not to see any docs > checked in, countrary to what you promized... I can imagine. Although I do admit that I tend to be a sloppy and lazy guy, this time I was really hit by the ongoings with my son, which happened right after the Tuesday meeting. It drove me crazy, I could neither sleep nor work nor even think for the rest of the week, and it made me a very unhappy man. I think I grew a lot of gray hair. My son has left my house, stopped his school six weeks before the final exams, which he would have made (I asked the teachers), in order to live near to his girlfriend and work for some company, without Abitur and a very limited future. This is such a slap into my face that I have a hard time recovering from "loosing" my son. > We talked a bit about > ways to go forward but without deciding anything yet, given that we are > not sure what exactly you have in mind about the hardest problems we > foresee. Now Michael is on holidays, and people are travelling back > from the sprint, so we'll try again maybe on Monday... Monday isn't perfect, since I'm traveling back in the afternoon and will be very busy, but I will do my best to be on pypy-sync? at 5pm your time. The thing we need to discuss is about the "essence of Stackless", and it is crucial to get this well understood and done right. I will force myself right now to finish the design document and give a clear picture about how I think the Stackless part of PyPy should be finished with the minimum impact on the current implementation. all the best - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer at stackless.com Mon May 1 10:21:43 2006 From: tismer at stackless.com (Christian Tismer) Date: Mon May 1 10:21:38 2006 Subject: [pypy-dev] The Essentials of Stackless Message-ID: <4455C517.3080004@stackless.com> Dear friends, I've finally come up with a rough document, which for my vision of things says almost everything what needs to be said about Stackless Python and how to implement it for PyPy, especially regarding how to implement the thread pickling. It may be incomplete and not well-designed, it is more like a brain-dump after a week of more or less thinking, pretending to be a structured document. In the end, as an addition, I come to an interesting conclusion about life-ness analysis and GC transform, that I really did not expect beforehand. Eventually, I'm beginning to understand what I did when I wrote Stackless Python. please have a look and comment about http://codespeak.net:/svn/pypy/dist/pypy/doc/discussion/howtoimplementpickling.txt all the best -- chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From seojiwon at gmail.com Tue May 2 07:33:47 2006 From: seojiwon at gmail.com (Jiwon Seo) Date: Tue May 2 07:34:26 2006 Subject: [pypy-dev] Error on translation? Message-ID: Hi all, I am trying to run translated versions of targetnopstandalone.py, targetrpystonedalone.py (and eventually pypy-c) on a simulator. However, even before that, the translated versions on real machine (so, not on a simulator) crash. (Machine spec and options for translate.py are at the end) For instance, translated version of targetrpystonedalone.py produces segmentation fault when I ran it. The stack when segfault occurs looks like following: _RPyListOfString_SetItem(list, i, s) in main.h pypy_g__RPyListOfString_SetItem__listPtr_Signed_rpy_str(struct pypy_list0 *l_l_1, long l_index_0, struct pypy_rpy_string0 *l_newstring_0) in implement.c pypy_g_ll_setitem_nonneg__dum_nocheckConst_listPtr_Sign(struct pypy_list0 *l_l_2, long l_index_1, struct pypy_rpy_string0 *l_newitem_2) in implement.c and inside the last function, segmentation fault occurs at l_v10 = l_l_2->l_items; because l_l_2 is not allocated. (actually, I don't see any place that l_l_2 is allocated before that line) I'm runnint the translated executable on ppc linux: (uname -a shows following: Linux rundmc.stanford.edu 2.6.15-1.2054_FC5 #1 SMP Tue Mar 14 15:57:06 EST 2006 ppc64 ppc64 ppc64 GNU/Linux) and I ran translate.py with --text --no-compile --source --gc=ref options, and then mv /tmp/usession-x/testing_1/ to a new place, then compiled it. Ah, and I'm using current version of pypy. Any idea? -Jiwon From sanxiyn at gmail.com Tue May 2 08:49:00 2006 From: sanxiyn at gmail.com (Sanghyeon Seo) Date: Tue May 2 08:49:37 2006 Subject: [pypy-dev] oopspec Message-ID: <5b0248170605012349x18fed4e0j681e70218a5fe25c@mail.gmail.com> Is there some documentation on oopspec attribute? How one may use it in the backend? Seo Sanghyeon From sanxiyn at gmail.com Tue May 2 09:02:46 2006 From: sanxiyn at gmail.com (Sanghyeon Seo) Date: Tue May 2 09:03:24 2006 Subject: [pypy-dev] Signature PEP Message-ID: <5b0248170605020002o154555chc48deb1377970ce3@mail.gmail.com> Hi, PyPy people, What do you think about this Signature PEP? http://mail.python.org/pipermail/python-3000/2006-April/001249.html PyPy has something similar as pypy.interpreter.pycode.cpython_code_signature, and I think it's definitely a good idea to have some better interface than code.co_* and CO_FLAGS. http://codespeak.net/svn/pypy/dist/pypy/interpreter/pycode.py We also have more extensive interface in pypy.annotation.description: http://codespeak.net/svn/pypy/dist/pypy/annotation/description.py There's FunctionDesc, ClassDesc, and MethodDesc there. Though they are quite PyPy-specific. Seo Sanghyeon From pedronis at strakt.com Tue May 2 10:09:52 2006 From: pedronis at strakt.com (Samuele Pedroni) Date: Tue May 2 10:10:31 2006 Subject: [pypy-dev] Google summer of code is open student for submissions Message-ID: <445713D0.7060906@strakt.com> Deadline is the 8th of May. PyPy related projects may be submitted under the PSF umbrella. A list of possible PyPy project ideas can be found at: http://codespeak.net/svn/pypy/dist/pypy/doc/independent-project-ideas.txt regards. From fabien-ml at x-phuture.com Tue May 2 11:57:10 2006 From: fabien-ml at x-phuture.com (Fabien Schwob) Date: Tue May 2 11:57:49 2006 Subject: [pypy-dev] Google summer of code is open student for submissions In-Reply-To: <445713D0.7060906@strakt.com> Message-ID: <20060502095710.45DA67EC9B@postix.sdv.fr> > A list of possible PyPy project ideas can be found at: > > http://codespeak.net/svn/pypy/dist/pypy/doc/independent-project-ideas > .txt I'm really interested in the following project : * Write an interpreter for **another dynamic language** in the PyPy framework. For example, a Javascript interpreter would be suitable. Ruby too (though the latter is probably more than two months of work). Or Scheme, or... etc. And I would like to have more information about it if possible. I would like to check if I understand it. So the goal would be to parse the Javascript instructions and to create an AST tree from it ? So, it would be possible to write Javascript code that can be after executed on the PyPy backends (CLI, C, LLVM, etc...). Thanks in advance. From anto.cuni at gmail.com Tue May 2 12:01:27 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue May 2 12:02:33 2006 Subject: [pypy-dev] oopspec In-Reply-To: <5b0248170605012349x18fed4e0j681e70218a5fe25c@mail.gmail.com> References: <5b0248170605012349x18fed4e0j681e70218a5fe25c@mail.gmail.com> Message-ID: <44572DF7.90900@gmail.com> Hi Seo, Sanghyeon Seo wrote: > Is there some documentation on oopspec attribute? How one may use it > in the backend? I think there are no docs about the oopspec attribute. Armin wrote these lines some time ago to respond to the same question: """ The oopspec string tells what is the "abstract" list operation that this particular ll_*() function implement. For example: def ll_prepend(l, newitem): ... ll_prepend.oopspec = 'list.insert(l, 0, newitem)' means that ll_prepend() is equivalent to an insert with the index set to zero. In the stirng, the pseudo-arguments between the ( ) are either real argument names of the ll_ function, or constants. So for example, if a backend has got its own way to implement the insert() calls in general, it could figure out from the oopspec that the ll_prepend() helper can be replaced by a custom stub invoking the backend's own version of insertion with an index of 0. That's essentially what the JIT does -- see handle_highlevel_operation() in jit/hintannotator/model.py. """ The CLI backend uses the oopspec attribute for replacing calls to selected low-level helpers with native builtin methods; the code is still very experimental since it doesn't parse the argument line: it simply forwards the call using the first parameter as the target object and subsequent parameters as method's arguments. By now the only recognized oopspec is 'list.append' (i.e., ll_append) that is translated to the 'Add' method. If you want to look at my code see translator/cli/oopspec.py and the _Call.render method in translator/cli/metavm.py; at the moment I put these files in the cli/ directory, but they are general enough to be shared among multiple backends (metavm would be useful only for backends emitting bytecode, so not for gencl nor gensqueak, I guess). The rationale behind this is that this way a backend can quickly gain full list support by simply supply basic operations such as ll_getitem_fast & Co; then each backend can choose what operation to optimize based on their knowledge of the target system. Btw, I have a doubt about oopspec, too: Armin told that the 'oopspec' specifies the "abstract" operation that each ll_* helper implements; does this abstract operation have to be one of standard list methods or can I add new operations? I was thinking to add a 'll_remove_range' or similar to be used by other helpers such as delslice and company; this way backends can replace ll_remove_range with their almost-surely-present equivalent without having to care about python-specific logic such as slices or so. ciao Anto From seojiwon at gmail.com Tue May 2 19:33:19 2006 From: seojiwon at gmail.com (Jiwon Seo) Date: Tue May 2 19:33:57 2006 Subject: [pypy-dev] Re: Error on translation? In-Reply-To: References: Message-ID: Um, I was using wrong main.h. Thank Armin for the help. -Jiwon On 5/1/06, Jiwon Seo wrote: > Hi all, > > I am trying to run translated versions of targetnopstandalone.py, > targetrpystonedalone.py (and eventually pypy-c) on a simulator. > However, even before that, the translated versions on real machine > (so, not on a simulator) crash. (Machine spec and options for > translate.py are at the end) > > For instance, translated version of targetrpystonedalone.py produces > segmentation fault when I ran it. The stack when segfault occurs > looks like following: > > _RPyListOfString_SetItem(list, i, s) in main.h > > pypy_g__RPyListOfString_SetItem__listPtr_Signed_rpy_str(struct > pypy_list0 *l_l_1, long l_index_0, struct pypy_rpy_string0 > *l_newstring_0) in implement.c > > pypy_g_ll_setitem_nonneg__dum_nocheckConst_listPtr_Sign(struct > pypy_list0 *l_l_2, long l_index_1, struct pypy_rpy_string0 > *l_newitem_2) in implement.c > > and inside the last function, segmentation fault occurs at > l_v10 = l_l_2->l_items; > > because l_l_2 is not allocated. (actually, I don't see any place that > l_l_2 is allocated before that line) > > I'm runnint the translated executable on ppc linux: > (uname -a shows following: Linux rundmc.stanford.edu 2.6.15-1.2054_FC5 > #1 SMP Tue Mar 14 15:57:06 EST 2006 ppc64 ppc64 ppc64 GNU/Linux) > > and I ran translate.py with --text --no-compile --source --gc=ref > options, and then mv /tmp/usession-x/testing_1/ to a new place, then > compiled it. Ah, and I'm using current version of pypy. > > Any idea? > > -Jiwon > From arigo at tunes.org Wed May 3 00:06:24 2006 From: arigo at tunes.org (Armin Rigo) Date: Wed May 3 00:06:25 2006 Subject: [pypy-dev] oopspec In-Reply-To: <44572DF7.90900@gmail.com> References: <5b0248170605012349x18fed4e0j681e70218a5fe25c@mail.gmail.com> <44572DF7.90900@gmail.com> Message-ID: <20060502220624.GA7698@code0.codespeak.net> Hi Antonio, On Tue, May 02, 2006 at 12:01:27PM +0200, Antonio Cuni wrote: > Btw, I have a doubt about oopspec, too: Armin told that the 'oopspec' > specifies the "abstract" operation that each ll_* helper implements; > does this abstract operation have to be one of standard list methods or > can I add new operations? I was thinking to add a 'll_remove_range' or > similar to be used by other helpers such as delslice and company; this > way backends can replace ll_remove_range with their > almost-surely-present equivalent without having to care about > python-specific logic such as slices or so. It's quite open; every piece of code using oopspec should be prepared to see names that it doesn't know about, and ignore them. So feel free to add new names. As a guideline, let's stick as far as possible to the Python name for the method or for the __xxx__ special method name (with __ removed): 'remove_range' looks like it could be called 'delslice' in oopspec. A bientot, Armin From arigo at tunes.org Wed May 3 00:11:47 2006 From: arigo at tunes.org (Armin Rigo) Date: Wed May 3 00:12:00 2006 Subject: [pypy-dev] Signature PEP In-Reply-To: <5b0248170605020002o154555chc48deb1377970ce3@mail.gmail.com> References: <5b0248170605020002o154555chc48deb1377970ce3@mail.gmail.com> Message-ID: <20060502221146.GB7698@code0.codespeak.net> Hi Seo, On Tue, May 02, 2006 at 04:02:46PM +0900, Sanghyeon Seo wrote: > What do you think about this Signature PEP? > http://mail.python.org/pipermail/python-3000/2006-April/001249.html It's a language design issue so it's off topic :-) More seriously, sure, it would be nice to have some kind of standard. Here, and in many similar code-manipulation places in PyPy, we had to hack together our own minimal solution. A harder problem along these lines would be a better way to build pieces of code, in order to create functions from scratch. \n-joining together source lines full of %s, avoiding name clashes, and getting the indentation right, is not fun. A bientot, Armin From arigo at tunes.org Wed May 3 00:40:19 2006 From: arigo at tunes.org (Armin Rigo) Date: Wed May 3 00:40:21 2006 Subject: [pypy-dev] Google summer of code is open student for submissions In-Reply-To: <20060502095710.45DA67EC9B@postix.sdv.fr> References: <445713D0.7060906@strakt.com> <20060502095710.45DA67EC9B@postix.sdv.fr> Message-ID: <20060502224019.GC7698@code0.codespeak.net> Hi Fabien, On Tue, May 02, 2006 at 11:57:10AM +0200, Fabien Schwob wrote: > And I would like to have more information about it if possible. I would > like to check if I understand it. So the goal would be to parse the > Javascript instructions and to create an AST tree from it ? So, it would > be possible to write Javascript code that can be after executed on the > PyPy backends (CLI, C, LLVM, etc...). The goal is to write an *interpreter*. At this point it's completely unrelated to the translation backends. Let's consider Javascript (the same argument applies to any interpreted language): to write an interpreter, some people focus on the task of compiling, say, Javascript source code to get an AST; this is only part of the job, and it should be the shortest part -- ideally, finding an existing parser would be best, otherwise plugging one together with existing tools shouldn't take too long for a regular language like Javascript. It could be an AST or bytecode in any format. The core of the work is actually to build an interpreter for this AST or bytecode, because in addition to the AST or bytecode interpreter you also need to put together the object model, and build the standard types. This means basically rewriting a Standard Interpreter that interprets Javascript instead of full Python -- for what Standard Interpreter exactly means here, see http://codespeak.net/pypy/dist/pypy/doc/architecture.html . (Getting the basics working to interpret Javascript is rather easier than to interpret Python, as the JS language and object model are far simpler; nevertheless, a two-months job will probably leave many details unpolished, which is ok.) Per se, this goal is not related to the back-ends of the Translation Process of PyPy (see again the same link for "Translation Process"). So it's not about providing either a front-end nor a back-end for Javascript. The reason it's still interesting to write a JS interpreter is that the JS interpreter itself can then be turned into C code, or .NET code, or any of the platforms that the Translation Process supports or will support. In this way, writing a JS interpreter onces gives you Javascript.NET, Javascript-on-Java, etc. all at once, using the unmodified translation toolchain. This works for any program written in the PyPy framework (i.e. written in RPython); but the reason PyPy is well suited to write - more specifically - *interpreters* instead of something else, is about what interesting things we can do with the RPython program if it's an interpreter. This means e.g. turn it into a Just-In-Time compiler. Our goal (not achieved yet) is that with relatively little efforts, each of the interpreters written in the PyPy framework will be turned into JIT compilers. So we might later obtain a Javascript JIT out of the Summer of Code-sized Javascript interpreter :-) Clearly, that's outside the scope of the project, that's just motivation. A bientot, Armin. From anto.cuni at gmail.com Wed May 3 01:00:28 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed May 3 01:01:35 2006 Subject: [pypy-dev] oopspec In-Reply-To: <20060502220624.GA7698@code0.codespeak.net> References: <5b0248170605012349x18fed4e0j681e70218a5fe25c@mail.gmail.com> <44572DF7.90900@gmail.com> <20060502220624.GA7698@code0.codespeak.net> Message-ID: <4457E48C.4070408@gmail.com> Hi Armin Armin Rigo wrote: > It's quite open; every piece of code using oopspec should be prepared to > see names that it doesn't know about, and ignore them. So feel free to > add new names. As a guideline, let's stick as far as possible to the > Python name for the method or for the __xxx__ special method name (with > __ removed): 'remove_range' looks like it could be called 'delslice' in > oopspec. I'm unsure about using 'delslice': it make me think that such an operation behave exactly like the corresponding python statement, but that was not my intent: in my mind the difference between 'remove_range' and 'delslice' is that the first doesn't handle negative indexes and so it is likely that the target has some built-in method that implement it natively. Btw I have not thought to it too much, so I don't know if such a refactoring is worth the pain: maybe it is simply more effective to write a delslice method in C# (or whatever language fit other backends) and forwarding the hypothetic 'delslice' to it, even it would mean that each backend has to write its own implementation if they don't get satisfaction with the default one. ciao Anto From sanxiyn at gmail.com Wed May 3 02:32:21 2006 From: sanxiyn at gmail.com (Sanghyeon Seo) Date: Wed May 3 02:32:59 2006 Subject: [pypy-dev] Google summer of code is open student for submissions In-Reply-To: <20060502224019.GC7698@code0.codespeak.net> References: <445713D0.7060906@strakt.com> <20060502095710.45DA67EC9B@postix.sdv.fr> <20060502224019.GC7698@code0.codespeak.net> Message-ID: <5b0248170605021732j383ba149tffdc0b7d1dde9ef1@mail.gmail.com> 2006/5/3, Armin Rigo : > Let's consider Javascript (the same argument applies to any interpreted > language): to write an interpreter, some people focus on the task of > compiling, say, Javascript source code to get an AST; this is only part > of the job, and it should be the shortest part -- ideally, finding an > existing parser would be best, otherwise plugging one together with > existing tools shouldn't take too long for a regular language like > Javascript. Sadly, parsing Javascript is unnecessarily difficult because of optional semicolon insertion. Best to look at existing parsers. Seo Sanghyeon From nnorwitz at gmail.com Wed May 3 09:15:33 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Wed May 3 09:16:15 2006 Subject: [pypy-dev] Seeking students for the Summer of Code Message-ID: There is less than a week left before students must submit a final application. There are a bunch of ideas up on the wiki: http://wiki.python.org/moin/SummerOfCode/ The wiki has instructions for how to submit a proposal. There are many different areas including: core language features, libraries, and applications. This is a great opportunity to get real coding experience. Not to mention the chance to work with a nice and fun group of people. The earlier you submit an application, the more feedback you can get to improve it and increase your liklihood of getting accepted. Feel free to contact me if you have any questions. Cheers, n From arigo at tunes.org Wed May 3 09:22:20 2006 From: arigo at tunes.org (Armin Rigo) Date: Wed May 3 09:22:21 2006 Subject: [pypy-dev] oopspec In-Reply-To: <4457E48C.4070408@gmail.com> References: <5b0248170605012349x18fed4e0j681e70218a5fe25c@mail.gmail.com> <44572DF7.90900@gmail.com> <20060502220624.GA7698@code0.codespeak.net> <4457E48C.4070408@gmail.com> Message-ID: <20060503072220.GB6154@code0.codespeak.net> Hi Antonio, On Wed, May 03, 2006 at 01:00:28AM +0200, Antonio Cuni wrote: > I'm unsure about using 'delslice': it make me think that such an > operation behave exactly like the corresponding python statement, but > that was not my intent: in my mind the difference between 'remove_range' > and 'delslice' is that the first doesn't handle negative indexes and so > it is likely that the target has some built-in method that implement it > natively. I remember thinking about this that the hint "not negative" could be specified somewhere in addition to the basic oopspec: def ll_getitem_nonneg(func, l, index): ... ll_getitem_nonneg.oopspec = 'list.getitem(l, index)' ll_getitem_nonneg.oopspechint = {'index': '>=0'} # <==== def ll_getitem(func, l, index): ... ll_getitem.oopspec = 'list.getitem(l, index)' The point is that a backend can ignore the hint and implement a general getitem operation that accepts negatives and raises IndexError, or select a more optimized version if it's got one when it sees the hint. Also, note that all our slice operations at RPython level assume that indices are non-negative, with a special exception for exactly x[:-1]. (see rslice.py) A bientot, Armin From arigo at tunes.org Wed May 3 09:27:13 2006 From: arigo at tunes.org (Armin Rigo) Date: Wed May 3 09:27:17 2006 Subject: [pypy-dev] Seeking students for the Summer of Code In-Reply-To: References: Message-ID: <20060503072712.GA7587@code0.codespeak.net> Hi all, On Wed, May 03, 2006 at 12:15:33AM -0700, Neal Norwitz wrote: > The earlier you submit an application, the more feedback you can get > to improve it and increase your liklihood of getting accepted. I'd like to emphasis this point too :-) The proposals need to somehow give the impression that they are yours, and that your are interested in them and have some idea about what is involved. Feel free to discuss ideas here, and submit proposals early -- you'll get early feedback from us too, and you can update your proposal any number of times until the deadline. Basically, just copy-and-pasting one of the paragraphs from http://codespeak.net/svn/pypy/dist/pypy/doc/independent-project-ideas.txt is *not* enough for a proposal! A bientot, Armin. From davidf at sjsoft.com Wed May 3 16:28:18 2006 From: davidf at sjsoft.com (David Fraser) Date: Wed May 3 16:29:24 2006 Subject: [pypy-dev] Signature PEP In-Reply-To: <20060502221146.GB7698@code0.codespeak.net> References: <5b0248170605020002o154555chc48deb1377970ce3@mail.gmail.com> <20060502221146.GB7698@code0.codespeak.net> Message-ID: <4458BE02.2010308@sjsoft.com> Armin Rigo wrote: > Hi Seo, > > On Tue, May 02, 2006 at 04:02:46PM +0900, Sanghyeon Seo wrote: > >> What do you think about this Signature PEP? >> http://mail.python.org/pipermail/python-3000/2006-April/001249.html >> > > It's a language design issue so it's off topic :-) > > More seriously, sure, it would be nice to have some kind of standard. > Here, and in many similar code-manipulation places in PyPy, we had to > hack together our own minimal solution. A harder problem along these > lines would be a better way to build pieces of code, in order to create > functions from scratch. \n-joining together source lines full of %s, > avoiding name clashes, and getting the indentation right, is not fun. > > I had a look a while ago at trying to build Python code using ASTs and then compile those, for this very reason. In the end constructing the source was easier, but in theory this is a code-based alternative Cheers David From hpk at trillke.net Wed May 3 16:40:22 2006 From: hpk at trillke.net (holger krekel) Date: Wed May 3 16:47:58 2006 Subject: [pypy-dev] Google summer of code is open student for submissions In-Reply-To: <5b0248170605021732j383ba149tffdc0b7d1dde9ef1@mail.gmail.com> References: <445713D0.7060906@strakt.com> <20060502095710.45DA67EC9B@postix.sdv.fr> <20060502224019.GC7698@code0.codespeak.net> <5b0248170605021732j383ba149tffdc0b7d1dde9ef1@mail.gmail.com> Message-ID: <20060503144022.GL24602@solar.trillke> Hi Seo, On Wed, May 03, 2006 at 09:32 +0900, Sanghyeon Seo wrote: > 2006/5/3, Armin Rigo : > >Let's consider Javascript (the same argument applies to any interpreted > >language): to write an interpreter, some people focus on the task of > >compiling, say, Javascript source code to get an AST; this is only part > >of the job, and it should be the shortest part -- ideally, finding an > >existing parser would be best, otherwise plugging one together with > >existing tools shouldn't take too long for a regular language like > >Javascript. > > Sadly, parsing Javascript is unnecessarily difficult because of > optional semicolon insertion. Best to look at existing parsers. IMO for a SoC project it would be ok to have a stricter parser. Having javascript programs running on top of PyPy (JsPy or whatever :) being able to run unmodified on other javascript interpreters would be interesting enough. And such an approach would still have the advantages that Armin points out. best, holger From anto.cuni at gmail.com Thu May 4 00:26:12 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu May 4 00:27:24 2006 Subject: [pypy-dev] Flow graph documentation Message-ID: <44592E04.2010403@gmail.com> Hi all, at last I've begin writing my thesis, though it's still very incomplete and some parts are more similar to a thought flow than to an academic text :-). Since I've not found any doc about flow graphs I've written a bit about them hoping that it could be useful even outside of my thesis. The point is that all I know about flow graphs is deduced from the experience, thus there are possibilities that I've written absurdities. If someone feels like to do quick review of that part my rough draft is here: http://codespeak.net/svn/user/antocuni/tesi/overview.txt thanks ciao Anto From mojobojo at gmail.com Thu May 4 08:29:35 2006 From: mojobojo at gmail.com (Brian Jones) Date: Thu May 4 08:30:57 2006 Subject: [pypy-dev] Seeking students for the Summer of Code In-Reply-To: <20060503072712.GA7587@code0.codespeak.net> References: <20060503072712.GA7587@code0.codespeak.net> Message-ID: <61f323ea0605032329v27be5335o2a34d68b20cca046@mail.gmail.com> On 5/3/06, Armin Rigo wrote: > Hi all, > > On Wed, May 03, 2006 at 12:15:33AM -0700, Neal Norwitz wrote: > > The earlier you submit an application, the more feedback you can get > > to improve it and increase your liklihood of getting accepted. > > I'd like to emphasis this point too :-) The proposals need to somehow > give the impression that they are yours, and that your are interested in > them and have some idea about what is involved. Feel free to discuss > ideas here, and submit proposals early -- you'll get early feedback from > us too, and you can update your proposal any number of times until the > deadline. Basically, just copy-and-pasting one of the paragraphs from > http://codespeak.net/svn/pypy/dist/pypy/doc/independent-project-ideas.txt > is *not* enough for a proposal! Hello everyone, I have some questions/problems I'd like to pose in regards to SoC and applying to do a PyPy project. For starters, I am on my second semester of studying abroad in Japan right now, and the semester ends around the beginning-middle of August. I feel that I have plenty of time to devote to a PyPy project, however, many projects I have looked at seem to have a rather strict "This is your #1 priority" policy. This makes sense of course, most of the students applying are probably going into summer break right now. Does this stack against me? Next, most of my programming experience to date is focused around higher level languages (Python, Ruby), and my C knowledge dates back to earlier low level CS courses. I feel comfortable writing something in any language, but would of course be much slower writing C code until I kicked some rust off. I would also want to note that my degree isn't computer related at all. Are either of these another hinderance? Last, is of course a project plan (or lack thereof). I was pointed at PyPy by Chris when I inquired about Stackless several months ago, and have had kind of a backburnered interest in both for awhile now. I would love to contribute to either project, but I really have no idea where my 'niche' would be. Pickling process states so that they can be handed across multiple machines is interesting to me, and even more interesting would be machines with a different target language. I've always liked the idea of platform/language independent objects, although I've never had any idea for practical use (beyond something like wildly dreaming game development). I suppose that even if I cannot participate in SoC, this email is probably a nice start in regards to getting to know the PyPy community. Thanks in advance, Brian Jones From arigo at tunes.org Fri May 5 21:16:21 2006 From: arigo at tunes.org (Armin Rigo) Date: Fri May 5 21:16:23 2006 Subject: [pypy-dev] Seeking students for the Summer of Code In-Reply-To: <61f323ea0605032329v27be5335o2a34d68b20cca046@mail.gmail.com> References: <20060503072712.GA7587@code0.codespeak.net> <61f323ea0605032329v27be5335o2a34d68b20cca046@mail.gmail.com> Message-ID: <20060505191621.GA23019@code0.codespeak.net> Hi Brian, On Thu, May 04, 2006 at 03:29:35PM +0900, Brian Jones wrote: > I have some questions/problems I'd like to pose in regards to SoC and > applying to do a PyPy project. Welcome! > For starters, I am on my second semester of studying abroad in Japan > right now, and the semester ends around the beginning-middle of > August. This is potentially a problem, but maybe not a too serious one if you are usually working on side stuff a lot during semesters. > Next, most of my programming experience to date is focused around > higher level languages (Python, Ruby), and my C knowledge dates back > to earlier low level CS courses. Not a problem for PyPy - we have about 7000 lines of C in a corner but we're working on removing that :-) > Last, is of course a project plan (or lack thereof). I was pointed at > PyPy by Chris when I inquired about Stackless several months ago, and > have had kind of a backburnered interest in both for awhile now. Of course, you need a more concrete project. At first, going through the PyPy documentation should give you some ideas to start with, and then come back to pypy-dev or to the #pypy channel on irc.freenode.net to discuss them. In the area of pickling process or thread states, our current situation is as follows: we are starting to work on it these days (E.U. funding requirements). By the start of the Summer of Code we won't be finished with the basics, though. A bit later -- say by the half of the SoC project duration -- we will hopefully be done with the basics of pickling, and then we will like to move to other topics (we have just too many things to do!). At this point your constribution would be welcome to complete and polish the area (which is a long and never-finished topic, so I don't fear lack of work here :-) So maybe there is a way to submit a SoC proposal that would start with another small PyPy topic to get started, and move to pickling in a second part. At least it would fit. A bientot, Armin. From arigo at tunes.org Fri May 5 21:19:32 2006 From: arigo at tunes.org (Armin Rigo) Date: Fri May 5 21:19:34 2006 Subject: [pypy-dev] Flow graph documentation In-Reply-To: <44592E04.2010403@gmail.com> References: <44592E04.2010403@gmail.com> Message-ID: <20060505191932.GB23019@code0.codespeak.net> Hi Antonio, On Thu, May 04, 2006 at 12:26:12AM +0200, Antonio Cuni wrote: > Since I've not found any doc about flow graphs I've written a bit about > them hoping that it could be useful even outside of my thesis. Ouch! At first sight, your text looks nice, but now we have a merging issue: there is already documentation about flow graphs, it's even a kind-of-well-documented part of PyPy in comparison. It's at http://codespeak.net/pypy/dist/pypy/doc/objspace.html#the-flow-model (I haven't read your docs so far, will come back...) A bientot, Armin. From tismer at stackless.com Sat May 6 11:06:01 2006 From: tismer at stackless.com (Christian Tismer) Date: Sat May 6 11:06:42 2006 Subject: [pypy-dev] Flow graph documentation In-Reply-To: <20060505191932.GB23019@code0.codespeak.net> References: <44592E04.2010403@gmail.com> <20060505191932.GB23019@code0.codespeak.net> Message-ID: <445C66F9.2070301@stackless.com> Armin Rigo wrote: > Hi Antonio, > > On Thu, May 04, 2006 at 12:26:12AM +0200, Antonio Cuni wrote: >> Since I've not found any doc about flow graphs I've written a bit about >> them hoping that it could be useful even outside of my thesis. > > Ouch! At first sight, your text looks nice, but now we have a merging > issue: there is already documentation about flow graphs, it's even a > kind-of-well-documented part of PyPy in comparison. It's at > http://codespeak.net/pypy/dist/pypy/doc/objspace.html#the-flow-model Maybe we need some glossary or other reverse indexes. Certain things are not obvious where to find. -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From elmo13 at jippii.fi Sun May 7 14:02:59 2006 From: elmo13 at jippii.fi (=?UTF-8?B?RWxtbyBNw6RudHluZW4=?=) Date: Sun May 7 14:03:43 2006 Subject: [pypy-dev] An error in doc/getting-started.txt Message-ID: <445DE1F3.1040005@jippii.fi> Here is a diff -u (attached) of a corrected version. The diff is self-explanatory. Elmo -------------- next part -------------- A non-text attachment was scrubbed... Name: getting-started.diff Type: text/x-patch Size: 926 bytes Desc: not available Url : http://codespeak.net/pipermail/pypy-dev/attachments/20060507/d707f787/getting-started.bin From elmo13 at jippii.fi Sun May 7 20:20:53 2006 From: elmo13 at jippii.fi (=?UTF-8?B?RWxtbyBNw6RudHluZW4=?=) Date: Sun May 7 20:21:38 2006 Subject: [pypy-dev] "compilemodule.py _demo" fails Message-ID: <445E3A85.1070503@jippii.fi> Attached is the output of "py.test ". Apart from that, I'm interested if the few warnings are just irrelevant and will go away eventually, or are they related. Thanks for your great effort, I can't wait till you get the app-level side of this working. Elmo -------------- next part -------------- ============================= test process starts ============================= testing-mode: inprocess executable: /usr/bin/python (2.3.5-final-0) using py lib: /home/drayko/Workfolder/Security/Python/cvs/pypy-dist/py pypy/rpython/rctypes/tool/test/test_compilemodule.py[1] F pypy/rpython/rctypes/tool/test/test_ctypes_platform.py[7] ....... _______________________________________________________________________________ - - - - - - - - - - - - test_demo: recorded stdout - - - - - - - - - - - - - (pypy.rpython.rctypes.tool.compilemodule:27)__init__: SomeObject -> const SomePBC (?:1)trampoline: SomeObject, SomeObject -> SomeObject (?:1)trampoline: SomeObject, SomeObject -> SomeObject - - - - - - - - - - - - test_demo: recorded stderr - - - - - - - - - - - - - [cbuild:execute] cc -O2 -pthread -I/usr/include/python2.3 -c ctypesplatcheck_0.c -o ctypesplatcheck_0.o [cbuild:execute] cc -pthread /tmp/usession-6/ctypesplatcheck_0.o -lm -lpthread -o /tmp/usession-6/ctypesplatcheck_0 [cbuild:execute] cc -O2 -pthread -c ctypesplatcheck_1.c -o ctypesplatcheck_1.o [cbuild:execute] cc -pthread /tmp/usession-6/ctypesplatcheck_1.o -lm -lpthread -o /tmp/usession-6/ctypesplatcheck_1 [translation:info] Annotating&simplifying... [translation:info] with policy: pypy.objspace.cpy.ann_policy.CPyAnnotatorPolicy [flowgraph:start] (pypy.rpython.rctypes.tool.compilemodule:27)__init__ [flowgraph:done] __init__ [flowgraph:start] (pypy.objspace.cpy.objspace:33)wrap /usr/lib/python2.3/site-packages/ctypes/__init__.py:305: FutureWarning: %u/%o/%x/%X of negative int will return a signed string in Python 2.4 and up return "<%s '%s', handle %x at %x>" % \ [flowgraph:done] wrap [flowgraph:start] (pypy.objspace.cpy.wrappable:53)reraise [flowgraph:done] reraise [annrpython] -> SomeCTypesObject(knowntype=W_Object, memorystate='OWNSMEMORY') [flowgraph:start] (pypy.interpreter.baseobjspace:551)call_method [flowgraph:done] call_method [flowgraph:start] (pypy.objspace.cpy.objspace:79)call_function [flowgraph:done] call_function [annrpython] -> SomeCTypesObject(knowntype=W_Object, memorystate='OWNSMEMORY') [annrpython] -> SomeCTypesObject(knowntype=W_Object, memorystate='OWNSMEMORY') [flowgraph:start] (pypy.interpreter.baseobjspace:551)call_method [flowgraph:done] call_method [flowgraph:start] (pypy.objspace.cpy.objspace:79)call_function [flowgraph:done] call_function [annrpython] -> SomeCTypesObject(knowntype=W_Object, memorystate='OWNSMEMORY') [annrpython] -> SomeCTypesObject(knowntype=W_Object, memorystate='OWNSMEMORY') [annrpython] -> SomePBC(can_be_None=True, const=None) [flowgraph:start] (?:1)trampoline [flowgraph:done] trampoline [flowgraph:start] (?:1)trampoline [flowgraph:done] trampoline [flowgraph:start] (pypy.objspace.cpy.ctypes_base:29)rctypes_pyerrchecker [flowgraph:done] rctypes_pyerrchecker [annrpython] -> SomePBC(can_be_None=True, const=None) [flowgraph:start] (pypy.module._demo.demo:39)measuretime [flowgraph:done] measuretime [flowgraph:start] (pypy.interpreter.error:18)OperationError.__init__ [flowgraph:done] __init__ [flowgraph:start] (pypy.module._demo.demo:34)get [flowgraph:done] get [annrpython] -> SomePBC(can_be_None=True, const=None) [flowgraph:start] (pypy.objspace.cpy.objspace:30)getbuiltinmodule [flowgraph:done] getbuiltinmodule [annrpython] -> SomeCTypesObject(knowntype=W_Object, memorystate='OWNSMEMORY') [flowgraph:start] (pypy.objspace.cpy.objspace:33)wrap [flowgraph:done] wrap [flowgraph:start] (pypy.objspace.cpy.refcount:27)Py_XIncref [flowgraph:done] Py_XIncref [annrpython] -> SomeCTypesObject(knowntype=W_Object, memorystate='OWNSMEMORY') [annrpython] -> SomePBC(can_be_None=True, const=None) [annrpython] -> SomePBC(can_be_None=True, const=None) [annrpython] -> SomeCTypesObject(knowntype=W_Object, memorystate='OWNSMEMORY') [annrpython] -> SomeObject() [annrpython:WARNING] / result degenerated to SomeObject [annrpython] -> SomeObject() [annrpython:WARNING] / result degenerated to SomeObject [annrpython] -> SomeCTypesObject(knowntype=W_Object, memorystate='OWNSMEMORY') [translation:info] All exceptblocks seem sane [translation:info] No lost method defs [translation:WARNING] -- someobjectness 18% (3 of 16 functions polluted by SomeObjects) [translation:info] RTyping... [flowgraph:start] (pypy.rpython.lltypesystem.rclass:803)ll_runtime_type_info [flowgraph:done] ll_runtime_type_info [annrpython] -> SomePtr(ll_ptrtype=<* RuntimeTypeInfo (opaque)>) [flowgraph:start] (pypy.rpython.lltypesystem.rrange:42)ll_newrange [flowgraph:done] ll_newrange [annrpython] -> SomePtr(ll_ptrtype=<* GcStruct range { start: Signed, stop: Signed }>) [flowgraph:start] (pypy.rpython.lltypesystem.rrange:64)ll_rangeiter [flowgraph:done] ll_rangeiter [annrpython] -> SomePtr(ll_ptrtype=<* GcStruct range { next: Signed, stop: Signed }>) [flowgraph:start] (pypy.rpython.lltypesystem.rlist:327)ll_fixed_newlist [flowgraph:done] ll_fixed_newlist [annrpython] -> SomePtr(ll_ptrtype=<* GcArray of * GcStruct rpy_string { hash: Signed, chars: Array of Char } >) [flowgraph:start] (pypy.rpython.rlist:518)ll_setitem_nonneg [flowgraph:done] ll_setitem_nonneg [flowgraph:start] (pypy.rpython.lltypesystem.rlist:342)ll_fixed_setitem_fast [flowgraph:done] ll_fixed_setitem_fast [annrpython] -> SomePBC(can_be_None=True, const=None) [annrpython] -> SomePBC(can_be_None=True, const=None) [flowgraph:start] (pypy.rpython.lltypesystem.rlist:370)ll_listiter [flowgraph:done] ll_listiter [annrpython] -> SomePtr(ll_ptrtype=<* GcStruct listiter { list: * GcArray of * GcStruct rpy_string { hash: Signed, chars: Array of Char } , index: Signed }>) [flowgraph:start] (pypy.rpython.lltypesystem.rlist:327)ll_fixed_newlist [flowgraph:done] ll_fixed_newlist [annrpython] -> SomePtr(ll_ptrtype=<* GcArray of Void >) [flowgraph:start] (pypy.rpython.rlist:518)ll_setitem_nonneg [flowgraph:done] ll_setitem_nonneg [flowgraph:start] (pypy.rpython.lltypesystem.rlist:342)ll_fixed_setitem_fast [flowgraph:done] ll_fixed_setitem_fast [annrpython] -> SomePBC(can_be_None=True, const=None) [annrpython] -> SomePBC(can_be_None=True, const=None) [flowgraph:start] (pypy.rpython.rrange:158)ll_rangenext_up [flowgraph:done] ll_rangenext_up [annrpython] -> SomeInteger(knowntype=int, nonneg=False, unsigned=False) [flowgraph:start] (pypy.rpython.rstr:635)ll_strlen [flowgraph:done] ll_strlen [annrpython] -> SomeInteger(knowntype=int, nonneg=True, unsigned=False) [flowgraph:start] (pypy.rpython.rctypes.rchar_p:142)ll_setstring [flowgraph:done] ll_setstring [annrpython] -> SomePBC(can_be_None=True, const=None) [flowgraph:start] (pypy.rpython.rctypes.rchar_p:114)ll_str2charp [flowgraph:done] ll_str2charp [annrpython] -> SomePtr(ll_ptrtype=<* FixedSizeArray of 1 Char >) [flowgraph:start] (pypy.rpython.lltypesystem.rlist:376)ll_listnext [flowgraph:done] ll_listnext [flowgraph:start] (pypy.rpython.lltypesystem.rlist:333)ll_fixed_length [flowgraph:done] ll_fixed_length [annrpython] -> SomeInteger(knowntype=int, nonneg=True, unsigned=False) [flowgraph:start] (pypy.rpython.lltypesystem.rlist:339)ll_fixed_getitem_fast [flowgraph:done] ll_fixed_getitem_fast [annrpython] -> SomePtr(ll_ptrtype=<* GcStruct rpy_string { hash: Signed, chars: Array of Char }>) [annrpython] -> SomePtr(ll_ptrtype=<* GcStruct rpy_string { hash: Signed, chars: Array of Char }>) [flowgraph:start] (pypy.rpython.rctypes.rpyobject:42)ll_pyobjbox_is_true [flowgraph:done] ll_pyobjbox_is_true [annrpython] -> SomeBool() [rtyper] specializing: 100 / 130 blocks (76%) [flowgraph:start] (pypy.rpython.lltypesystem.rclass:779)ll_issubclass [flowgraph:done] ll_issubclass [annrpython] -> SomeBool() [flowgraph:start] (pypy.rpython.lltypesystem.rclass:776)ll_type [flowgraph:done] ll_type [annrpython] -> SomePtr(ll_ptrtype=<* Struct object_vtable { parenttypeptr: * Struct object_vtable { ... }, subclassrange_min: Signed, subclassrange_max: Signed, rtti: * RuntimeTypeInfo (opaque), name: * Array of Char , instantiate: * Func ( ) -> * GcStruct object { typeptr: * Struct object_vtable { ... } } }>) [flowgraph:start] (pypy.rpython.lltypesystem.exceptiondata:108)ll_pyexcclass2exc [flowgraph:done] ll_pyexcclass2exc [annrpython] -> SomePtr(ll_ptrtype=<* GcStruct object { typeptr: * Struct object_vtable { parenttypeptr: * Struct object_vtable { ... }, subclassrange_min: Signed, subclassrange_max: Signed, rtti: * RuntimeTypeInfo (opaque), name: * Array of Char , instantiate: * Func ( ) -> * GcStruct object { ... } } }>) [flowgraph:start] (pypy.rpython.lltypesystem.exceptiondata:31)ll_raise_OSError [flowgraph:done] ll_raise_OSError [annrpython] -> SomeImpossibleValue() [translation:info] Back-end optimisations... [backendopt:removecasts] removed 2 cast_pointers in rctypes_pyerrchecker [backendopt:removecasts] removed 2 cast_pointers in measuretime [backendopt:removecasts] removed 4 cast_pointers in OperationError.__init__ [backendopt:removecasts] removed 2 cast_pointers in ll_rangenext_up__rangePtr_Signed [backendopt:removecasts] removed 2 cast_pointers in ll_listnext__listiterPtr [backendopt:removecasts] removed 2 cast_pointers in ll_raise_OSError__Signed [backendopt:inlining] 2.00 ll_fixed_setitem_fast__arrayPtr_Signed_rpy_stringPtr [backendopt:inlining] 2.00 ll_fixed_newlist__GcArray_VoidLlT_Signed [backendopt:inlining] 2.00 ll_fixed_setitem_fast__arrayPtr_Signed_NoneConst [backendopt:inlining] 2.00 ll_fixed_length__arrayPtr [backendopt:inlining] 2.00 ll_fixed_getitem_fast__arrayPtr_Signed [backendopt:inlining] 2.00 ll_fixed_newlist__GcArray_Ptr_GcStruct_rpy_strin_Signed [backendopt:inlining] 4.00 ll_strlen__rpy_stringPtr [backendopt:inlining] 4.00 ll_str2charp__rpy_stringPtr [backendopt:inlining] 8.00 ll_listiter__Ptr_GcStruct_listiterLlT_arrayPtr [backendopt:inlining] 8.00 ll_newrange__Signed_Signed [backendopt:inlining] 8.50 ll_pyobjbox_is_true__CtypesBox_W_ObjectPtr [backendopt:inlining] 2.00 ll_setitem_nonneg__dum_nocheckConst_arrayPtr_Signed_rpy_stringPtr [backendopt:inlining] 12.00 ll_rangeiter__Ptr_GcStruct_rangeLlT_rangePtr [backendopt:inlining] 17.00 ll_rangenext_up__rangePtr_Signed [backendopt:inlining] 15.00 ll_setstring__CtypesBox_c_char_pPtr_rpy_stringPtr [backendopt:inlining] 24.00 getbuiltinmodule [backendopt:inlining] 24.10 Py_XIncref [backendopt:inlining] 28.00 call_function_star0 [backendopt:inlining] 24.50 OperationError.__init__ [backendopt:inlining] 20.50 ll_listnext__listiterPtr [backendopt:malloc] 12 simple mallocs removed in '__init__' [backendopt:malloc] 2 simple mallocs removed in '__init__' [backendopt:malloc] 2 simple mallocs removed in 'wrap__str' [backendopt:malloc] 2 simple mallocs removed in 'wrap__str' [backendopt:malloc] 1 simple mallocs removed in 'wrap__str' [backendopt:malloc] 6 simple mallocs removed in 'reraise' [backendopt:malloc] 3 simple mallocs removed in 'call_method_star0' [backendopt:malloc] 2 simple mallocs removed in 'call_method_star1' [backendopt:malloc] 1 simple mallocs removed in 'trampoline' [backendopt:malloc] 1 simple mallocs removed in 'trampoline' [backendopt:malloc] 1 simple mallocs removed in 'trampoline' [backendopt:malloc] 1 simple mallocs removed in 'trampoline' [backendopt:malloc] 6 simple mallocs removed in 'rctypes_pyerrchecker' [backendopt:malloc] 3 simple mallocs removed in 'rctypes_pyerrchecker' [backendopt:malloc] 3 simple mallocs removed in 'rctypes_pyerrchecker' [backendopt:malloc] 4 simple mallocs removed in 'measuretime' [backendopt:malloc] 2 simple mallocs removed in 'get' [backendopt:malloc] 1 simple mallocs removed in 'get' [backendopt:malloc] 1 simple mallocs removed in 'wrap__int' [backendopt:malloc] 1 simple mallocs removed in 'wrap__int' [backendopt:malloc] 1 simple mallocs removed in 'Py_XIncref' [backendopt:malloc] removed 56 simple mallocs in total [translation:info] inserting stack checks... [flowgraph:start] (pypy.rpython.module.ll_stack:11)ll_stack_check [flowgraph:done] ll_stack_check [flowgraph:start] (pypy.rpython.module.ll_stack:3)ll_stack_too_big [flowgraph:done] ll_stack_too_big [annrpython] -> SomeBool() [annrpython] -> SomePBC(can_be_None=True, const=None) [flowgraph:start] (pypy.rpython.module.ll_stack:7)ll_stack_unwind [flowgraph:done] ll_stack_unwind [annrpython] -> SomePBC(can_be_None=True, const=None) [translation:info] Creating database for generating c source... [flowgraph:start] (pypy.translator.c.exceptiontransform:52)rpyexc_occured [flowgraph:done] rpyexc_occured [flowgraph:start] (pypy.translator.c.exceptiontransform:55)rpyexc_fetch_type [flowgraph:done] rpyexc_fetch_type [flowgraph:start] (pypy.translator.c.exceptiontransform:58)rpyexc_fetch_value [flowgraph:done] rpyexc_fetch_value [flowgraph:start] (pypy.translator.c.exceptiontransform:61)rpyexc_clear [flowgraph:done] rpyexc_clear [flowgraph:start] (pypy.translator.c.exceptiontransform:65)rpyexc_raise [flowgraph:done] rpyexc_raise [rtyper:WARNING] prebuilt instance has no attribute 'exc_type' [rtyper:WARNING] prebuilt instance has no attribute 'exc_value' [flowgraph:start] (pypy.rpython.memory.gctransform:386)ll_incref [flowgraph:done] ll_incref [flowgraph:start] (pypy.rpython.memory.gctransform:390)ll_decref [flowgraph:done] ll_decref [flowgraph:start] (pypy.rpython.memory.gctransform:397)ll_no_pointer_dealloc [flowgraph:done] ll_no_pointer_dealloc [flowgraph:start] (pypy.rpython.memory.gctransform:569)ll_dealloc [flowgraph:done] ll_dealloc [flowgraph:start] (pypy.translator.c.extfunc:94)RPyString_New [flowgraph:done] RPyString_New [annrpython] -> SomePtr(ll_ptrtype=<* GcStruct rpy_string { hash: Signed, chars: Array of Char }>) [flowgraph:start] (?:1)ll_deallocator [flowgraph:done] ll_deallocator [flowgraph:start] (?:1)ll_deallocator [flowgraph:done] ll_deallocator [flowgraph:start] (?:1)ll_deallocator [flowgraph:done] ll_deallocator [flowgraph:start] (?:1)ll_deallocator [flowgraph:done] ll_deallocator ____________________________ entrypoint: test_demo ____________________________ def test_demo(): > mod = compilemodule('_demo') [/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/rpython/rctypes/tool/test/test_compilemodule.py:5] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def compilemodule(modname): "Compile a PyPy module for CPython." import pypy.rpython.rctypes.implementation from pypy.objspace.cpy.objspace import CPyObjSpace from pypy.objspace.cpy.wrappable import reraise from pypy.objspace.cpy.ann_policy import CPyAnnotatorPolicy from pypy.translator.driver import TranslationDriver from pypy.interpreter.error import OperationError space = CPyObjSpace() ModuleClass = __import__('pypy.module.%s' % modname, None, None, ['Module']).Module module = ModuleClass(space, space.wrap(modname)) w_moduledict = module.getdict() def __init__(mod): w_mod = CPyObjSpace.W_Object(mod) try: ## space.appexec([w_mod, w_moduledict], ## '''(mod, newdict): ## old = mod.__dict__.copy() ## for key in ['__name__', '__doc__', 'RPythonError']: ## newdict[key] = old[key] ## newdict['__rpython__'] = old ## mod.__dict__.clear() ## mod.__dict__.update(newdict) ## ''') # the same at interp-level: w_moddict = space.getattr(w_mod, space.wrap('__dict__')) w_old = space.call_method(w_moddict, 'copy') space.call_method(w_moddict, 'clear') space.setitem(w_moddict, space.wrap('__rpython__'), w_old) for key in ['__name__', '__doc__', 'RPythonError']: w_key = space.wrap(key) try: w1 = space.getitem(w_old, w_key) except OperationError: pass else: space.setitem(w_moddict, w_key, w1) space.call_method(w_moddict, 'update', w_moduledict) except OperationError, e: reraise(e) __init__.allow_someobjects = True driver = TranslationDriver(extmod_name=modname) driver.setup(__init__, [object], policy=CPyAnnotatorPolicy(space)) > driver.proceed(['compile_c']) [/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/rpython/rctypes/tool/compilemodule.py:61] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def proceed(self, goals): if not goals: if self.default_goal: goals = [self.default_goal] else: self.log.info("nothing to do") return elif isinstance(goals, str): goals = [goals] goals = self.backend_select_goals(goals) > return self._execute(goals, task_skip = self._maybe_skip()) [/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/driver.py:417] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def _execute(self, goals, *args, **kwds): task_skip = kwds.get('task_skip', []) res = None for goal in self._plan(goals, skip=task_skip): taskcallable, _ = self.tasks[goal] self._event('pre', goal, taskcallable) try: > res = self._do(goal, taskcallable, *args, **kwds) [/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/tool/taskengine.py:108] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def _do(self, goal, func, *args, **kwds): title = func.task_title if goal in self.done: self.log.info("already done: %s" % title) return else: self.log.info("%s..." % title) > res = func() [/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/driver.py:143] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def task_database_c(self): translator = self.translator opt = self.options if translator.annotator is not None: translator.frozen = True standalone = self.standalone gcpolicy = None if opt.gc =='boehm': from pypy.translator.c import gc gcpolicy = gc.BoehmGcPolicy if opt.gc =='exact_boehm': from pypy.translator.c import gc gcpolicy = gc.MoreExactBoehmGcPolicy if opt.gc == 'none': from pypy.translator.c import gc gcpolicy = gc.NoneGcPolicy if opt.gc == 'framework': from pypy.translator.c import gc gcpolicy = gc.FrameworkGcPolicy if standalone: from pypy.translator.c.genc import CStandaloneBuilder as CBuilder else: from pypy.translator.c.genc import CExtModuleBuilder as CBuilder cbuilder = CBuilder(self.translator, self.entry_point, gcpolicy = gcpolicy, thread_enabled = getattr(opt, 'thread', False)) cbuilder.stackless = opt.stackless if not standalone: # xxx more messy cbuilder.modulename = self.extmod_name > database = cbuilder.build_database() [/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/driver.py:255] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def build_database(self, exports=[], pyobj_options=None): translator = self.translator db = LowLevelDatabase(translator, standalone=self.standalone, gcpolicy=self.gcpolicy, thread_enabled=self.thread_enabled) if self.stackless: from pypy.translator.c.stackless import StacklessData db.stacklessdata = StacklessData(db) db.use_stackless_transformation = self.use_stackless_transformation # pass extra options into pyobjmaker if pyobj_options: for key, value in pyobj_options.items(): setattr(db.pyobjmaker, key, value) # we need a concrete gcpolicy to do this self.libraries += db.gcpolicy.gc_libraries() # give the gc a chance to register interest in the start-up functions it # need (we call this for its side-effects of db.get()) list(db.gcpolicy.gc_startup_code()) # build entrypoint and eventually other things to expose pf = self.getentrypointptr() pfname = db.get(pf) self.exports[self.entrypoint.func_name] = pf for obj in exports: if type(obj) is tuple: objname, obj = obj elif hasattr(obj, '__name__'): objname = obj.__name__ else: objname = None po = self.getentrypointptr(obj) poname = db.get(po) objname = objname or poname if objname in self.exports: raise NameError, 'duplicate name in export: %s is %s and %s' % ( objname, db.get(self.exports[objname]), poname) self.exports[objname] = po > db.complete() [/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c/genc.py:77] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def complete(self, show_progress=True): assert not self.completed if self.translator and self.translator.rtyper: do_the_getting(self, self.translator.rtyper) def dump(): lst = ['%s: %d' % keyvalue for keyvalue in self.containerstats.items()] lst.sort() log.event('%8d nodes [ %s ]' % (i, ' '.join(lst))) i = self.completedcontainers if show_progress: show_i = (i//1000 + 1) * 1000 else: show_i = -1 work_to_do = True is_later_yet = False while work_to_do: while True: if hasattr(self, 'pyobjmaker'): > self.pyobjmaker.collect_initcode() [/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c/database.py:204] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def collect_initcode(self): while self.latercode: gen, self.debugstack = self.latercode.pop() #self.initcode.extend(gen) -- eats TypeError! bad CPython! > for line in gen: [/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c/pyobj.py:517] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def initdict(): for k in dic: if type(k) is str: > yield '%s[%r] = %s' % (name, k, self.nameof(dic[k])) [/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c/pyobj.py:474] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def nameof(self, obj, debug=None): if debug: stackentry = debug, obj else: stackentry = obj self.debugstack = (self.debugstack, stackentry) try: > return self.getvalue(pyobjectptr(obj)) [/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c/pyobj.py:52] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def get(self, obj): if isinstance(obj, ErrorValue): T = obj.TYPE if isinstance(T, Primitive): return PrimitiveErrorValue[T] elif isinstance(T, Ptr): return 'NULL' else: raise Exception("don't know about %r" % (T,)) else: T = typeOf(obj) if isinstance(T, Primitive): return PrimitiveName[T](obj, self) elif isinstance(T, Ptr): if obj: # test if the ptr is non-NULL if isinstance(obj._obj, int): # special case for tagged odd-valued pointers return '((%s) %d)' % (cdecl(self.gettype(T), ''), obj._obj) > node = self.getcontainernode(obj._obj) [/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c/database.py:178] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def getcontainernode(self, container): try: node = self.containernodes[container] except KeyError: assert not self.completed T = typeOf(container) if isinstance(T, (lltype.Array, lltype.Struct)): if hasattr(self.gctransformer, 'consider_constant'): self.gctransformer.consider_constant(T, container) nodefactory = ContainerNodeFactory[T.__class__] > node = nodefactory(self, T, container) [/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c/database.py:149] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def __init__(self, db, T, obj): # obj is a _pyobject here; obj.value is the underlying CPython object self.db = db self.T = T self.obj = obj > self.name = db.pyobjmaker.computenameof(obj.value) [/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c/node.py:711] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def computenameof(self, obj): obj_builtin_base = builtin_base(obj) if obj_builtin_base in (object, int, long) and type(obj) is not obj_builtin_base: if isinstance(obj, FunctionGraph): return self.nameof_graph(obj) # assume it's a user defined thingy return self.nameof_instance(obj) else: for cls in type(obj).__mro__: meth = getattr(self, 'nameof_' + cls.__name__.replace(' ', ''), None) if meth: break else: raise Exception, "nameof(%r)" % (obj,) > return meth(obj) [/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c/pyobj.py:73] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def nameof_tuple(self, tup): name = self.uniquename('g%dtuple' % len(tup)) > args = [self.nameof(x) for x in tup] [/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c/pyobj.py:449] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def nameof(self, obj, debug=None): if debug: stackentry = debug, obj else: stackentry = obj self.debugstack = (self.debugstack, stackentry) try: > return self.getvalue(pyobjectptr(obj)) [/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c/pyobj.py:52] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def get(self, obj): if isinstance(obj, ErrorValue): T = obj.TYPE if isinstance(T, Primitive): return PrimitiveErrorValue[T] elif isinstance(T, Ptr): return 'NULL' else: raise Exception("don't know about %r" % (T,)) else: T = typeOf(obj) if isinstance(T, Primitive): return PrimitiveName[T](obj, self) elif isinstance(T, Ptr): if obj: # test if the ptr is non-NULL if isinstance(obj._obj, int): # special case for tagged odd-valued pointers return '((%s) %d)' % (cdecl(self.gettype(T), ''), obj._obj) > node = self.getcontainernode(obj._obj) [/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c/database.py:178] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def getcontainernode(self, container): try: node = self.containernodes[container] except KeyError: assert not self.completed T = typeOf(container) if isinstance(T, (lltype.Array, lltype.Struct)): if hasattr(self.gctransformer, 'consider_constant'): self.gctransformer.consider_constant(T, container) nodefactory = ContainerNodeFactory[T.__class__] > node = nodefactory(self, T, container) [/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c/database.py:149] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def __init__(self, db, T, obj): # obj is a _pyobject here; obj.value is the underlying CPython object self.db = db self.T = T self.obj = obj > self.name = db.pyobjmaker.computenameof(obj.value) [/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c/node.py:711] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def computenameof(self, obj): obj_builtin_base = builtin_base(obj) if obj_builtin_base in (object, int, long) and type(obj) is not obj_builtin_base: if isinstance(obj, FunctionGraph): return self.nameof_graph(obj) # assume it's a user defined thingy return self.nameof_instance(obj) else: for cls in type(obj).__mro__: meth = getattr(self, 'nameof_' + cls.__name__.replace(' ', ''), None) if meth: break else: raise Exception, "nameof(%r)" % (obj,) > return meth(obj) [/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c/pyobj.py:73] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def nameof_object(self, value): if isinstance(object, property): return self.nameof_property(value) if type(value) is not object: E raise Exception, "nameof(%r)" % (value,) > Exception: nameof() [/home/drayko/Workfolder/Security/Python/cvs/pypy-dist/pypy/translator/c/pyobj.py:87] ============= tests finished: 7 passed, 1 failed in 12.09 seconds ============= inserting into sys.path: /home/drayko/Workfolder/Security/Python/cvs/pypy-dist From fijal at genesilico.pl Sun May 7 23:57:40 2006 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Mon May 8 00:00:35 2006 Subject: [pypy-dev] SoC proposal Message-ID: <445E6D54.8050002@genesilico.pl> I've written down SoC proposal, I has been refactored a little bit since IRC discussion, so I send you all application links. HTML version: http://students.mimuw.edu.pl/~mf197855/application.html Source reStructuredText: http://students.mimuw.edu.pl/~mf197855/application Any comment is very appreciated, also about language. I would like to thank You all with help around that proposal. Regards, Maciek Fijalkowski From janzert at janzert.com Tue May 9 05:25:23 2006 From: janzert at janzert.com (Janzert) Date: Tue May 9 05:28:07 2006 Subject: [pypy-dev] test_all.py results from windows Message-ID: I was asked on #pypy if I would post these here, so here they are. :) http://python.janzert.com/pypy/test_all_26973.txt Sorry if this double posts first time doesn't seem to have taken. Janzert From hpk at trillke.net Tue May 9 12:26:45 2006 From: hpk at trillke.net (holger krekel) Date: Tue May 9 12:34:59 2006 Subject: [pypy-dev] iceland sprint canceled Message-ID: <20060509102645.GQ19139@solar.trillke> Hi folks, as some of you have heard our PyPy Iceland sprint is canceled. There is a chain of events that lead up to this cancelation which is not easy to comprehend or to describe. What it probably boils down to: expectation mismatches and communication problems. A month ago we were asked by a sponsor to freely and quickly select 8 people and work on what we find suitable for PyPy. We planned on this basis and matched it with our 0.9 release goals and explicitely communicated both our goals and participant list to the sprint organizer. We got an OK for it and proceeded with our plannings. Last week it turned out that this OK was not reflecting what the CEO of the sponsoring company thought for the whole sprint event and he was very upset at the situation. This put us into a very unfortunate situation although we had clearly communicated our intentions and had gotten confirmation. We then decided last friday to adapt to the new situation, shifted our sprint goals, planned for explaining the EU some delays and told the sponsor that we were ready to withdraw 4 of the originally intended 8 participants of the sprint. But the CEO somehow perceived our moves as asking "for more" and canceled his funding offer. He had apparently expected us to silently drop the people. Bummer. Of course, we talked a lot yesterday about what went wrong and i (holger) asked myself that even more because of being involved in presenting the modification and "retraction offer" towards the sponsor and probably not hitting the right tone given the situation on the sponsors side. We also think there were misconceptions at play which are based on the difficulties of bringing commercial interests and open source projects together. And certainly some cultural differences. It's interesting to discuss this further, also with respect to possible further endeavours. But maybe not right now. We now are bound to decide soon about an alternative venue for our next PyPy sprint. We'd be happy if the iceland participants and many more can show up. best and a bientot, Holger and Armin From simon at arrowtheory.com Wed May 10 04:32:01 2006 From: simon at arrowtheory.com (Simon Burton) Date: Wed May 10 04:32:41 2006 Subject: [pypy-dev] sprint at europython ? Message-ID: <20060510123201.444d8576.simon@arrowtheory.com> Hi, I'm hoping to come to europython this year; one of the main attractions will be the pypy sprint. Have the dates for this been decided ? cheers, Simon. -- Simon Burton, B.Sc. Licensed PO Box 8066 ANU Canberra 2601 Australia Ph. 61 02 6249 6940 http://arrowtheory.com From mwh at python.net Wed May 10 10:37:02 2006 From: mwh at python.net (Michael Hudson) Date: Wed May 10 13:05:58 2006 Subject: [pypy-dev] Re: sprint at europython ? References: <20060510123201.444d8576.simon@arrowtheory.com> Message-ID: <2mmzdqe0nl.fsf@starship.python.net> Simon Burton writes: > Hi, > > I'm hoping to come to europython this year; one of > the main attractions will be the pypy sprint. Have the > dates for this been decided ? It will be after the conference, probably running for three or four days. > Simon Burton, B.Sc. > Licensed PO Box 8066 > ANU Canberra 2601 > Australia Another aussie at EuroPython? Wow. Cheers, mwh -- okay, tell me if i am crazy you are damn -- from Twisted.Quotes From hpk at trillke.net Thu May 11 08:05:06 2006 From: hpk at trillke.net (holger krekel) Date: Thu May 11 08:05:41 2006 Subject: [pypy-dev] #pypy-sync TODAY 5pm Message-ID: <20060511060506.GX19139@solar.trillke> Hi folks, let's have a developer pypy-sync meeting today (THURSDAY 11TH May) as was mentioned here and there already. Time: 5pm UTC+2 Location: #pypy-sync on freenode Here are the topics, please mail here before if you like to change/amend them: - activity reports (lines of LAST, NEXT and BLOCKERS) - thread cloning approaches As discussed earlier it is urgent to have a working thread cloning approach. Armin and Christian will try to present the approache(s) and time frames for this feature in some written form. - Next PyPy sprint Preliminary discussions resulted in the suggestion to do the next PyPy sprint from 2nd-9th June 2006 and very likely in Duesseldorf (Germany) because that seems to suit most of us best. Topics will be the 0.9 release, ext-compiler, stackless and more or less all topics that have been floating lately. Carl will be trying to fix a place. From cfbolz at gmx.de Thu May 11 15:42:31 2006 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Thu May 11 15:42:16 2006 Subject: [pypy-dev] #pypy-sync TODAY 5pm In-Reply-To: <20060511060506.GX19139@solar.trillke> References: <20060511060506.GX19139@solar.trillke> Message-ID: <44633F47.7080506@gmx.de> holger krekel wrote: > let's have a developer pypy-sync meeting today (THURSDAY 11TH > May) as was mentioned here and there already. > > Time: 5pm UTC+2 > Location: #pypy-sync on freenode > > Here are the topics, please mail here before if you > like to change/amend them: sorry, I won't make it, lost too much time with snake :-( my lines: LAST: weakref, tiny stuff, lots of university work NEXT: finally weakref on boehm, starting with __del__ hopefully BLOCKERS: too much at once Cheers, Carl Friedrich From arigo at tunes.org Thu May 11 16:24:13 2006 From: arigo at tunes.org (Armin Rigo) Date: Thu May 11 16:24:15 2006 Subject: [pypy-dev] Thread cloning (coroutine cloning, really) In-Reply-To: <20060511060506.GX19139@solar.trillke> References: <20060511060506.GX19139@solar.trillke> Message-ID: <20060511142413.GA2986@code0.codespeak.net> Hi all, On Thu, May 11, 2006 at 08:05:06AM +0200, holger krekel wrote: > - thread cloning approaches > > As discussed earlier it is urgent to have a working > thread cloning approach. Armin and Christian will > try to present the approache(s) and time frames for > this feature in some written form. A word of introduction: we are dividing the work in two directions; one is thread pickling, which is lead by Christian. The other is thread cloning, lead by me. Thread cloning is in principle a subset of what can be done with pickling, as we could pickle a thread, unpickle it, et voila: we have cloned the thread. But pickling is harder and comes with a different set of problems than "just" cloning. It is also likely that pickling threads in random interpreter states will be very difficult, a restriction that does not apply to thread cloning. So we divided the work; the most important reason is that cloning is what the constraint solver developments require urgently. So here is my current point of view on cloning. The goal is to give to RPython code some way to duplicate a "chain of frames", i.e. an RPython coroutine. There are three levels of issues: 1. the interface that RPython code needs to use to do that 2. the selection of what GcStructures must be duplicated or shared 3. the automatic generation of the necessary walkers from the stackless transformer In the following days we should focus on 3. This will need quite some coding efforts by itself, maybe ~ 1 week of dedicated work from where we stand now. Then we need to experiment with various ways to select what needs to be duplicated or not. The problem is that there are some RPython-level and app-level objects that need "obviously" to be shared, like app-level modules, and like interp-level singleton state objects; and others that need obviously to be duplicated so that the newly cloned coroutine has its own copy, like local lists. It's unclear which option we will choose, but Christian has proposed some good ideas. Only experimentation will tell. A possibly good solution that he proposed (but hard to implement efficiently) is to duplicate exactly those objects that have been allocated by the coroutine that we are cloning. We could try to do that *inefficiently*, e.g. by adding an "allocated-by" field to every GcStructs, and see how the result works in practice. Trying that shouldn't be extremely hard; at most 1 more week of dedicated efforts. An advantage of this approach over alternatives is that it's conceptually simpler, and also that the required RPython-level interface is very easy: something along the lines of "newstate = state.fork()". In light of this, it seems that we are at some 2 weeks of hard work from a prototype, in the line of thread cloning. I have intentionally left out the other stackless subject, which is thread pickling; I'll let Christian present it when he's ready to. A bientot, Armin. From elmo13 at jippii.fi Thu May 11 19:17:31 2006 From: elmo13 at jippii.fi (=?UTF-8?B?RWxtbyBNw6RudHluZW4=?=) Date: Thu May 11 19:18:17 2006 Subject: [pypy-dev] Translation segfaults with "compilemodule.py _demo" Message-ID: <446371AB.2060405@jippii.fi> python2.4 pypy/rpython/rctypes/tool/compilemodule.py _demo [cbuild:execute] cc -O2 -pthread -I/usr/include/python2.4 -c ctypesplatcheck_0.c -o ctypesplatcheck_0.o [cbuild:execute] cc -pthread /tmp/usession-27/ctypesplatcheck_0.o -lm -lpthread -o /tmp/usession-27/ctypesplatcheck_0 [cbuild:execute] cc -O2 -pthread -c ctypesplatcheck_1.c -o ctypesplatcheck_1.o [cbuild:execute] cc -pthread /tmp/usession-27/ctypesplatcheck_1.o -lm -lpthread -o /tmp/usession-27/ctypesplatcheck_1 [translation:info] Annotating&simplifying... [translation:info] with policy: pypy.objspace.cpy.ann_policy.CPyAnnotatorPolicy Segmentation fault Other related minor errors: Propably because of the segfault, test.py gives EOFError, which isn't nice =). Something you might know is that using pyhon2.3 I get an "OverflowError: 0 does not fit in signed 64-bit integer" (happens with" from pypy.rpython.lltypesystem import lltype"), which has probably something to do with the resent changes in rarithmetic.py. Should I send the whole tracebacks for the these errors? Elmo From elmo13 at jippii.fi Thu May 11 21:30:37 2006 From: elmo13 at jippii.fi (=?UTF-8?B?RWxtbyBNw6RudHluZW4=?=) Date: Thu May 11 21:31:15 2006 Subject: [pypy-dev] Translation segfaults with "compilemodule.py _demo" In-Reply-To: <446371AB.2060405@jippii.fi> References: <446371AB.2060405@jippii.fi> Message-ID: <446390DD.7090803@jippii.fi> Elmo M?ntynen wrote: >python2.4 pypy/rpython/rctypes/tool/compilemodule.py _demo > >[cbuild:execute] cc -O2 -pthread -I/usr/include/python2.4 -c >ctypesplatcheck_0.c -o ctypesplatcheck_0.o >[cbuild:execute] cc -pthread /tmp/usession-27/ctypesplatcheck_0.o -lm >-lpthread -o /tmp/usession-27/ctypesplatcheck_0 >[cbuild:execute] cc -O2 -pthread -c ctypesplatcheck_1.c -o >ctypesplatcheck_1.o >[cbuild:execute] cc -pthread /tmp/usession-27/ctypesplatcheck_1.o -lm >-lpthread -o /tmp/usession-27/ctypesplatcheck_1 >[translation:info] Annotating&simplifying... >[translation:info] with policy: >pypy.objspace.cpy.ann_policy.CPyAnnotatorPolicy >Segmentation fault > > Segfaults with 2.4 but not with 2.3. The exception in .../pyobj.py:87, which I reported earlier, is still thrown though. >Other related minor errors: >Propably because of the segfault, test.py gives EOFError, which isn't >nice =). > This ---^ is propably py-devel stuff. >Something you might know is that using pyhon2.3 I get an >"OverflowError: 0 does not fit in signed 64-bit integer" (happens with" >from pypy.rpython.lltypesystem import lltype"), which has probably >something to do with the resent changes in rarithmetic.py. > Works now ---^ >Should I send >the whole tracebacks for the these errors? > >Elmo >_______________________________________________ >pypy-dev@codespeak.net >http://codespeak.net/mailman/listinfo/pypy-dev > > From aurelien.campeas at logilab.fr Fri May 12 10:24:42 2006 From: aurelien.campeas at logilab.fr (=?iso-8859-1?Q?Aur=E9lien_Camp=E9as?=) Date: Fri May 12 10:25:16 2006 Subject: [pypy-dev] Thread cloning (coroutine cloning, really) In-Reply-To: <20060511142413.GA2986@code0.codespeak.net> References: <20060511060506.GX19139@solar.trillke> <20060511142413.GA2986@code0.codespeak.net> Message-ID: <20060512082442.GA28391@crater.logilab.fr> On Thu, May 11, 2006 at 04:24:13PM +0200, Armin Rigo wrote: > Hi all, > > On Thu, May 11, 2006 at 08:05:06AM +0200, holger krekel wrote: > > - thread cloning approaches > > > > As discussed earlier it is urgent to have a working > > thread cloning approach. Armin and Christian will > > try to present the approache(s) and time frames for > > this feature in some written form. > > A word of introduction: we are dividing the work in two directions; one > is thread pickling, which is lead by Christian. The other is thread > cloning, lead by me. Thread cloning is in principle a subset of what > can be done with pickling, as we could pickle a thread, unpickle it, et > voila: we have cloned the thread. But pickling is harder and comes with > a different set of problems than "just" cloning. It is also likely that > pickling threads in random interpreter states will be very difficult, a > restriction that does not apply to thread cloning. So we divided the > work; the most important reason is that cloning is what the constraint > solver developments require urgently. Just a note there : the constraint solver doesn't really need thread cloning. It's the framework that makes possible modular integration of constraint solving and logic programming that needs it. Even more off-topic : I've been playing with Mercurial, one in the million new distributed SCM that appears these days (it's Python all over the place of course), and I am pleasantly surprised to discover how the basic concepts of computation spaces seem to match these of an DSCM. Some of the primitives are just the same : clone, merge (with an SCM you have the added ability to arbitrate merge conflicts, with comp. spaces conflicts just mean only one can win), commit (seemingly same semantics, but different usage patterns). This convergence is reassuring, at least on the front of the generality of the aforementioned primitives > > So here is my current point of view on cloning. The goal is to give to > RPython code some way to duplicate a "chain of frames", i.e. an RPython > coroutine. There are three levels of issues: > > 1. the interface that RPython code needs to use to do that > 2. the selection of what GcStructures must be duplicated or shared > 3. the automatic generation of the necessary walkers from the > stackless transformer > > In the following days we should focus on 3. This will need quite some > coding efforts by itself, maybe ~ 1 week of dedicated work from where we > stand now. > > Then we need to experiment with various ways to select what needs to be > duplicated or not. The problem is that there are some RPython-level and > app-level objects that need "obviously" to be shared, like app-level > modules, and like interp-level singleton state objects; and others that > need obviously to be duplicated so that the newly cloned coroutine has > its own copy, like local lists. Applevel modules are the responsibility of their implementors : if they contain shared global mutable state, then sharing them makes them thread-unsafe and unsuitable for use in comp. spaces. But that's life. well, I cannot but ask : would it be possible to be able to clone these anyway ? Now I wonder if built-in modules will need a make-them-thread-safe pass (this question is motivated by these interp-level singleton state objects you mention). > > It's unclear which option we will choose, but Christian has proposed > some good ideas. Only experimentation will tell. A possibly good > solution that he proposed (but hard to implement efficiently) is to > duplicate exactly those objects that have been allocated by the > coroutine that we are cloning. We could try to do that *inefficiently*, > e.g. by adding an "allocated-by" field to every GcStructs, and see how > the result works in practice. Trying that shouldn't be extremely hard; > at most 1 more week of dedicated efforts. An advantage of this approach > over alternatives is that it's conceptually simpler, and also that the > required RPython-level interface is very easy: something along the lines > of "newstate = state.fork()". > > In light of this, it seems that we are at some 2 weeks of hard work from > a prototype, in the line of thread cloning. Happy to hear this :) Thanks for all of this. Aur?lien. From arigo at tunes.org Fri May 12 11:05:37 2006 From: arigo at tunes.org (Armin Rigo) Date: Fri May 12 11:05:37 2006 Subject: [pypy-dev] Thread cloning (coroutine cloning, really) In-Reply-To: <20060512082442.GA28391@crater.logilab.fr> References: <20060511060506.GX19139@solar.trillke> <20060511142413.GA2986@code0.codespeak.net> <20060512082442.GA28391@crater.logilab.fr> Message-ID: <20060512090537.GA26683@code0.codespeak.net> Hi Aurelien, On Fri, May 12, 2006 at 10:24:42AM +0200, Aur?lien Camp?as wrote: > Just a note there : the constraint solver doesn't really need thread > cloning. It's the framework that makes possible modular integration of > constraint solving and logic programming that needs it. I was about to ask precisely if the operation I described here makes sense from your point of view. That's basically what you need, isn't it? > Applevel modules are the responsibility of their implementors : if > they contain shared global mutable state, then sharing them makes them > thread-unsafe and unsuitable for use in comp. spaces. But that's > life. > > well, I cannot but ask : would it be possible to be able to clone > these anyway ? It's part of the experimentation that we'll need. They would be cloned if we go for the approach that all objects *created* by a thread are cloned together with the thread. This includes app-level objects. This interpretation follows from a nice -- if vague -- high-level description from Christian: if you start a thread and it computes something up to some point, and clone it as this point, then you get a new thread that "looks like" it has been started from scratch and then ran up to the same point, repeating the same computations. Of course, this is vague because we need to explain that input-output and other side effects that the thread might have had are not duplicated. A bientot, Armin. From aurelien.campeas at logilab.fr Fri May 12 12:42:17 2006 From: aurelien.campeas at logilab.fr (=?iso-8859-1?Q?Aur=E9lien_Camp=E9as?=) Date: Fri May 12 12:41:50 2006 Subject: [pypy-dev] Thread cloning (coroutine cloning, really) In-Reply-To: <20060512090537.GA26683@code0.codespeak.net> References: <20060511060506.GX19139@solar.trillke> <20060511142413.GA2986@code0.codespeak.net> <20060512082442.GA28391@crater.logilab.fr> <20060512090537.GA26683@code0.codespeak.net> Message-ID: <20060512104217.GB28391@crater.logilab.fr> On Fri, May 12, 2006 at 11:05:37AM +0200, Armin Rigo wrote: Hi Armin, > Hi Aurelien, > > On Fri, May 12, 2006 at 10:24:42AM +0200, Aur?lien Camp?as wrote: > > Just a note there : the constraint solver doesn't really need thread > > cloning. It's the framework that makes possible modular integration of > > constraint solving and logic programming that needs it. > > I was about to ask precisely if the operation I described here makes > sense from your point of view. That's basically what you need, isn't > it? Roughly I think so. Some funny details may be discovered later of course. Well, clone != fork in my mind. We really want a *copy*. Clone or copy are good words for this. Absolutely avoid to share stuff (unless we know for sure sharing is safe but that would be an implementation detail). (fork is confusingly related to some unixish syscalls whose precise semantics are no more clear to me at this points -- it's been a long time since I've directly used it). Especially, cloning a space is different from 'newspace', which is much more akin to the fork operator. I don't have the time right now to expand on this & will try to make things more clear next week. I'm still unsure about the semantics of newspace... > > > Applevel modules are the responsibility of their implementors : if > > they contain shared global mutable state, then sharing them makes them > > thread-unsafe and unsuitable for use in comp. spaces. But that's > > life. > > > > well, I cannot but ask : would it be possible to be able to clone > > these anyway ? > > It's part of the experimentation that we'll need. They would be cloned > if we go for the approach that all objects *created* by a thread are > cloned together with the thread. This includes app-level objects. I'm not sure I understand what another approach would look like... We could restrict the programming style for code to be running inside comp. spaces but if it is possible to effectively clone everything (for some ill-defined notion of everything) then it'l be fine. I'm not 100% sure. Here are some of the constraints : In principle, side-effects on the parent space and the outer world should be forbidden from within a running comp. space ('cause the jury is still out on its outcome). The problem is not that we want to clone app-level modules (most Python code tend to be decomposed into many modules, I don't believe we want to forbid that) but that we will, by doing so, allow the computation running inside a space to do unwanted stuff, like sneakily doing I/O for instance, or indirectly calling some fancy C code that does God-knows-what. Something like a "restricted execution" environment would be interesting to have. In Mozart/Oz, for what it's worth, they 'just' filter any attempt to mutate the upper/outside world. I suspect it comes with a price, at least in terms of implementation complexity (I've not actually seen the code though). They do this also because comp. spaces in Oz are more, hmmm, expressive that what we'll have in PyPy : the 'newspace' is not scheduled to go into PyPy (well, at this point). Its usefulness is unclear to us. So, yes, we want to clone app-level modules, and generally, that everything created by a thread be cloned (first). We'll see later what can be made to prevent insanity to happen from within spaces. > > This interpretation follows from a nice -- if vague -- high-level > description from Christian: if you start a thread and it computes > something up to some point, and clone it as this point, then you get a > new thread that "looks like" it has been started from scratch and then > ran up to the same point, repeating the same computations. Of course, > this is vague because we need to explain that input-output and other > side effects that the thread might have had are not duplicated. The Oz way is to leave IO to the 'top-level' space. What really counts is the internal state of the threads. Whether we would like a 'logic/relational program' to perform IO is a topic for another day (I suspect currently we expect such a program to act on local state, where local means 'belongs 100% to the Python world' (but what does/doesn't exactly ? I don't know for sure). Oh, one last point : in my view, cloning one thread means cloning a whole tree of threads. Confusedly & hastily yours Hope this helps a bit. Aur?lien. From picxenk at gmail.com Sun May 14 10:42:56 2006 From: picxenk at gmail.com (SeungBum Kim) Date: Sun May 14 10:43:33 2006 Subject: [pypy-dev] New logo for pypy Message-ID: Hi, all I created a prototype for new pypy logo before. Some of you like a recursive aspect from my design. Old one was.. http://image.xenbio.net/logo/pypy_proto I made another one. It also has a recursive concept, but a little bit abstract image. http://image.xenbio.net/logo/neopypy1 Have fun. From mwh at python.net Mon May 15 11:38:49 2006 From: mwh at python.net (Michael Hudson) Date: Mon May 15 11:39:45 2006 Subject: [pypy-dev] Duesseldorf PyPy sprint 2-9 June 2006 Message-ID: <2mzmhj8w5y.fsf@starship.python.net> The next PyPy sprint will be held in the Computer Science department of Heinrich-Heine Universitaet Duesseldorf from the 2nd to the 9th of June. The main focus of the sprint will be on the goals of the upcoming June 0.9 release: * The extension module compiler. The goal is to be able to use a single RPython source code as an extension module for both PyPy and CPython. The means to get there is -- most likely -- by compiling ctypes-based modules into either pypy-c or a CPython dll/so. * Write some more modules in this style with ctypes. * Stackless: the big missing feature is pickling running tasklets. There is also some smaller work that needs to be done, like exposing all the existing RPython-level interfaces to app-level (e.g. greenlets). * Write more documentation. * Misc topics, depending on interests: back-ends (CLI, Squeak), testing framework, modified semantics (security/sandboxing...), etc. If you'd like to come, please subscribe to the `pypy-sprint mailing list`_ and drop a note about your interests and post any questions. More organisational information will be send to that list. We'll keep a list of `people`_ which we'll update (which you can do so yourself if you have codespeak commit rights). .. _`pypy-sprint mailing list`: http://codespeak.net/mailman/listinfo/pypy-sprint .. _`people`: http://codespeak.net/pypy/extradoc/sprintinfo/ddorf2006/people.html Cheers, mwh -- Slim Shady is fed up with your shit, and he's going to kill you. -- Eminem, "Public Service Announcement 2000" From mwh at python.net Thu May 18 01:11:09 2006 From: mwh at python.net (Michael Hudson) Date: Thu, 18 May 2006 00:11:09 +0100 Subject: [pypy-dev] pypy-sync tomorrow! Message-ID: <2mfyj86ycy.fsf@starship.python.net> It's Thursday and that means it's pypy-sync time. I'd like to invite all active pypy developers to the #pypy-sync channel on freenode at 5:30 PM GMT+2 tomorrow (well, today really by now, 2006-05-18). The known-at-this-point topics are: 1) activity reports as usual 2) oopsla paper We should submit a paper to the Dynamic Languages Symposium 2006 which is attached to this year's OOPSLA: http://www.dcl.hpi.uni-potsdam.de/dls2006/openconf.php What should be in it, who should write it, etc. The deadline is June 1. Some background reading would be overview papers on self, jikes, ... 3) state of the world There has been quite a lot of development over the past few weeks, and it's been fairly hard to keep up with it all. It would be good to have a quick chat about where things are, esp the things targetted for the 0.9 release. If you have any other topics, please email them here beforehand. Cheers, mwh -- People think I'm a nice guy, and the fact is that I'm a scheming, conniving bastard who doesn't care for any hurt feelings or lost hours of work if it just results in what I consider to be a better system. -- Linus Torvalds From tismer at stackless.com Thu May 18 16:16:49 2006 From: tismer at stackless.com (Christian Tismer) Date: Thu, 18 May 2006 16:16:49 +0200 Subject: [pypy-dev] pypy-sync: in case that I'm not there at 5pm Message-ID: <446C81D1.8090306@stackless.com> Hi friends, we are invited to an open-air concert, today. If it does not start raining, again, then I will miss the pypy-sync meeting. DONE: coached Stephan on app-level Stackless, helped Eric with the pickling support, analyzed many flow-graphs NEXT: support for costate for app-level, more coaching, implement a few demo picklers, investigate more on feasibility of the pickling design doc, prepare for Iceland sprint BLOCK: I cannot estimate what will come out of it. I'm not sure if I can find a way to avoid pickling of many records. cheers - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From anto.cuni at gmail.com Sun May 21 12:37:42 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Sun, 21 May 2006 12:37:42 +0200 Subject: [pypy-dev] gencli checkin In-Reply-To: <446E2E0C.2040707@genesilico.pl> References: <446E2E0C.2040707@genesilico.pl> Message-ID: <447042F6.90708@gmail.com> Hi Maciek, hi all Maciek Fijalkowski wrote: > I've just checked in my changes to gencli which are essential for genjs > to use gencli. It does not brake any more tests (7 failed as was > before). I hope I don't get in conflict with your actual edits. sorry for the very late response; I'm cc:ing to pypy-dev so that others can read. I've read your changes, good work: I think that now the best thing to do is to collect the modules used by both backends in a proper place, e.g. a genoo package or so (better names are welcome :-). Here are some comments on the modules that might be moved to genoo: class_.py, function.py, record.py: they contain both generic code and gencli specific code (such as Class.get_base_class() and and Class._ctor() methods). We could put generic 'Class', 'Function' and 'Record' class in genoo and subclass it in the backends. cts.py: this is CLI specific, but each backend need a similar module: we should at least put the interface specification in genoo, and possibly some shared code. database.py: there are both generic and specific parts, too. We should provide a more flexible way to handle constants, and maybe we can integrate it with the code from gensqueak that implements a sort of priority queue for emitting things in the right order. IMHO each backends should be able to decide whether to emit things in sequential order (as gencli does, should be faster) or in "declaration order" (as gensqueak and genjs2 need). ilgenerator.py: same as cts.py. oopspec.py: same as class_.py, function.py, record.py: some code can be shared, some need to be backend-specific. rte.py: its goal is to automatically compile a DLL every time the source if modified: this could be useful for all backends that need to compile some runtime environment "offline". sdk.py: very cli-specific. metavm.py: the code here is of three types: - generic code that can be used by all backends: the Generator/MicroInstrucion/InstructionList architecture is designed to be as much generic as possible, so it's not a problem to reuse it (note that the Generator interface is not very up-to-date, see Function.py). - some classes are not cli-specific but they have been designed for emitting code on stack-based machines; I guess they are not very useful for genjs2, but they could be useful for a hypotetic genjava, for example. - some code is very cli-specific, such as the _RuntimeNew class. I wrote some ideas on how you can use and extend metavm for your goal, but I decided to move them to a fresh e-mail so we don't mix the two discussions. At last we need a way to tell genoo where to find the code specific to each backend. I've seen that at the moment you decided to separately pass things such as CTS, Function, opcodes etc as arguments to GenCli constructor, but perhaps it would be better to collect them in a sort of 'container' class and pass this all around; something like this: class CliFunction: ... class CliClass: ... class CTS: ... cli_opcodes = { ... } class CliBackend: Function = CliFunction Class = CliClass TypeSystem = CTS opcodes = cli_opcodes ... Better names other than CliBackend are welcome, too :-). What do you think about this design? Valentino, Nikh, Seo and other that are working on high level backends: do you think that such a genoo could be useful for gensqueak and gencl, too? ciao Anto From anto.cuni at gmail.com Sun May 21 12:38:40 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Sun, 21 May 2006 12:38:40 +0200 Subject: [pypy-dev] MetaVM design Message-ID: <44704330.1020609@gmail.com> Hi Maciek, I all as I announced in my previous e-mail here are some ideas on how to use metavm for genjs2. I think the best way to use metavm for your purposes is to write a bunch of MicroInstruction classes designed for emitting source code. For example I guess that some rpython low level instruction need to be mapped to infix operator (such as int_add & co.) while others need to be mapped to some support function (such as int_abs). You could write something like this: class InfixOperator(MicroInstruction): def __init__(self, operator): self.operator = operator def render(self, generator, op): generator.emit('(%s %s %s)' %\ (op.args[0], self.operator, op.args[1]) class CallHelper(MicroInstruction): def __init__(self, helper): self.helper = helper def render(self, generator, op): arglist = ... # compute arglist from op.args generator.emit('%s(%s)' % (self.helper, arglist) Then, in your opcodes.py: opcodes = { 'int_add': InfixOperator('+'), 'int_abs': CallHelper('abs') } The difficult thing here is to design the Generator interface in a way that is suitable for emitting both asm code and source code. Maybe we try to separate the two worlds, in this way: genoo/metavm.py --------------- class Generator: def function_signature(self, graph): pass # put here all methods shared by both AsmGenerator and SourceGenerator class StackBasedGenerator(Generator): def emit(self, instr, *args): pass def call(self, func_name): pass def load(self, v): pass def store(self, v): pass ... class SourceCodeGenerator(Generator): def emit(self, expression): pass # and so on genoo/function.py ----------------- class Function(Node, Generator): # some shared code ... class StackBasedFunction(Node, StackBasedGenerator): ... class SourceCodeFunction(Node, SourceCodeGenerator): ... gencli/function.py ------------------ class CliFunction(StackBasedFucntion): ... genjs2/function.py ------------------ class JsFunction(SourceCodeFunction): ... Once we have this design, each MicroInstruction subclass can decide whether to target the generic Generator interface or one of its subinterfaces. ciao Anto From simon at arrowtheory.com Wed May 24 06:02:07 2006 From: simon at arrowtheory.com (Simon Burton) Date: Wed, 24 May 2006 14:02:07 +1000 Subject: [pypy-dev] llvm backend Message-ID: <20060524140207.4bc81933.simon@arrowtheory.com> Hi, I'm exploring the llvm backend of pypy. Does it work without llvm-gcc ? I tried the example at doc/getting-started.html and it seems to first compile to c code then invoke llvm-gcc. >>> f = t.compile_llvm() (calls to llvm-gcc) But I'm also looking at the source for the llvm translator and there is code there for generating llva directly. Now I'm trying to run the demos in pypy/translator/llvm/demo. I assume I can use my regular python to run these.. somehow.. $ python run.py Traceback (most recent call last): ... File "/mnt/hda9/simonb/local/pypy-svn/pypy/annotation/bookkeeper.py", line 16, in ? from pypy.annotation.listdef import ListDef, MOST_GENERAL_LISTDEF ImportError: cannot import name ListDef ? Simon. -- Simon Burton, B.Sc. Licensed PO Box 8066 ANU Canberra 2601 Australia Ph. 61 02 6249 6940 http://arrowtheory.com From simon at arrowtheory.com Wed May 24 06:40:21 2006 From: simon at arrowtheory.com (Simon Burton) Date: Wed, 24 May 2006 14:40:21 +1000 Subject: [pypy-dev] llvm backend In-Reply-To: <20060524140207.4bc81933.simon@arrowtheory.com> References: <20060524140207.4bc81933.simon@arrowtheory.com> Message-ID: <20060524144021.086eea94.simon@arrowtheory.com> On Wed, 24 May 2006 14:02:07 +1000 Simon Burton wrote: > > But I'm also looking at the source for the llvm translator and there is > code there for generating llva directly. I'm gradually finding more pieces of the puzzle: 1) there is a wrapper to LLVM's ExecutionEngine.ParseAssemblyString 2) there are calls to ee.parse in pyllvm/test/test_ee.py, but only with test snippets but then: 3) there is code for generating llva in codewriter.py 4) calls to codewriter come from *node.py modules, these stitch together the pypy flow and type objects into llva code 5) genllvm.py coordinates all of this and then calls to buildllvm It seems that steps 1 and 2 are yet to be used as buildllvm uses the commandline llvm-as at the moment. Simon. -- Simon Burton, B.Sc. Licensed PO Box 8066 ANU Canberra 2601 Australia Ph. 61 02 6249 6940 http://arrowtheory.com From dooms at info.ucl.ac.be Wed May 24 09:06:20 2006 From: dooms at info.ucl.ac.be (=?ISO-8859-1?Q?Gr=E9goire_Dooms?=) Date: Wed, 24 May 2006 09:06:20 +0200 Subject: [pypy-dev] pypy-dev Digest, Vol 310, Issue 5 In-Reply-To: References: Message-ID: <447405EC.7000007@info.ucl.ac.be> > Date: Fri, 12 May 2006 12:42:17 +0200 > From: Aur?lien Camp?as > Subject: Re: [pypy-dev] Thread cloning (coroutine cloning, really) > To: Armin Rigo > Cc: pypy-dev at codespeak.net > Message-ID: <20060512104217.GB28391 at crater.logilab.fr> > Content-Type: text/plain; charset=iso-8859-1 > > We could restrict the programming style for code to be running inside > comp. spaces but if it is possible to effectively clone everything > (for some ill-defined notion of everything) then it'l be fine. I'm not > 100% sure. Here are some of the constraints : > > In principle, side-effects on the parent space and the outer world > should be forbidden from within a running comp. space ('cause the jury > is still out on its outcome). > If I can throw in my two cents, you should really try to remove this limitation of Mozart. This is a real pain in the ass when you want to know what happens in a space (I used to need that in Mozart for debugging or statistics). In Mozart it is possible to use a hack in order to have side effects out of a space (ie modifiy state outside the space) by using ports. I think a design which would allow it would be much more Pythonic (we are between consenting adults :-) Best, -- Gr?goire From aurelien.campeas at logilab.fr Wed May 24 09:54:36 2006 From: aurelien.campeas at logilab.fr (=?iso-8859-1?Q?Aur=E9lien_Camp=E9as?=) Date: Wed, 24 May 2006 09:54:36 +0200 Subject: [pypy-dev] pypy-dev Digest, Vol 310, Issue 5 In-Reply-To: <447405EC.7000007@info.ucl.ac.be> References: <447405EC.7000007@info.ucl.ac.be> Message-ID: <20060524075436.GB7772@crater.logilab.fr> On Wed, May 24, 2006 at 09:06:20AM +0200, Gr?goire Dooms wrote: > > >Date: Fri, 12 May 2006 12:42:17 +0200 > >From: Aur?lien Camp?as > >Subject: Re: [pypy-dev] Thread cloning (coroutine cloning, really) > >To: Armin Rigo > >Cc: pypy-dev at codespeak.net > >Message-ID: <20060512104217.GB28391 at crater.logilab.fr> > >Content-Type: text/plain; charset=iso-8859-1 > > > > >We could restrict the programming style for code to be running inside > >comp. spaces but if it is possible to effectively clone everything > >(for some ill-defined notion of everything) then it'l be fine. I'm not > >100% sure. Here are some of the constraints : > > > >In principle, side-effects on the parent space and the outer world > >should be forbidden from within a running comp. space ('cause the jury > >is still out on its outcome). > > > > If I can throw in my two cents, you should really try to remove this > limitation of Mozart. This is a real pain in the ass when you want to > know what happens in a space (I used to need that in Mozart for > debugging or statistics). In Mozart it is possible to use a hack in > order to have side effects out of a space (ie modifiy state outside the > space) by using ports. > I think a design which would allow it would be much more Pythonic (we > are between consenting adults :-) I had been thinking about that indeed. Glad you tell us what you think. Anyway, by default/as a starting point PyPy's comp spaces will be "open" ; as seen in Mozart with ports or whatever it must be hard to restrict operations enough to ensure a proper sealing of a space and in the case of Python, worse than hard ... > > Best, > -- > Gr?goire > From eric at vanrietpaap.nl Wed May 24 10:02:30 2006 From: eric at vanrietpaap.nl (Eric van Riet Paap) Date: Wed, 24 May 2006 10:02:30 +0200 Subject: [pypy-dev] llvm backend In-Reply-To: <20060524140207.4bc81933.simon@arrowtheory.com> References: <20060524140207.4bc81933.simon@arrowtheory.com> Message-ID: <68A555EA-B3F4-4971-BFD5-2C9569B71957@vanrietpaap.nl> Hi Simon, llvm-gcc is used to compile the external function C code (that genllvm shares with genc) to a .ll file. The code in llvm/demo does not seem to work at the moment, sorry for that :( It looks like a circular dependency issue. You assumption about running the files in ..../demo with regular python is correct, however the run.py is respronsible for the invokation of genc/genllvm in this case so you would have to do something like: "python bpnn.py". About your other mail about genllvm: We indeed explored the usage of llvm's jit feature with ExecutionEngine.ParseAssemblyString but the use of this is currently limited to pypy/jit/codegen/llvm , where genllvm generates .ll code that gets added to the llvm jit with the execution engine. The steps the genllvm follows are: - (when necessary) create a .ll file out of genc's external function c code (one day to be replaced by (r)ctypes code) - generate and attach our source file - run llvm-as to convert the .ll to a .bc file - run opt to optimize with llvm's transformation - generate (a) assembly with llvm's native (x86) backend or (b) c code again with llvm's c backend - in case of b we run this code through gcc to further (profile based) optimize and create the final executable. I can image this information is a bit to dense so if you have any more questions please ask them here. It's good for us to know what people require for getting a better overview. cheers Eric On May 24, 2006, at 6:02 AM, Simon Burton wrote: > > Hi, > > I'm exploring the llvm backend of pypy. Does it work without llvm- > gcc ? > > I tried the example at doc/getting-started.html and it seems to first > compile to c code then invoke llvm-gcc. > >>>> f = t.compile_llvm() > (calls to llvm-gcc) > > But I'm also looking at the source for the llvm translator and > there is > code there for generating llva directly. > > Now I'm trying to run the demos in pypy/translator/llvm/demo. > I assume I can use my regular python to run these.. somehow.. > > $ python run.py > Traceback (most recent call last): > ... > File "/mnt/hda9/simonb/local/pypy-svn/pypy/annotation/ > bookkeeper.py", line 16, in ? > from pypy.annotation.listdef import ListDef, MOST_GENERAL_LISTDEF > ImportError: cannot import name ListDef > > ? > > Simon. > > > -- > Simon Burton, B.Sc. > Licensed PO Box 8066 > ANU Canberra 2601 > Australia > Ph. 61 02 6249 6940 > http://arrowtheory.com > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev From arigo at tunes.org Wed May 24 10:36:32 2006 From: arigo at tunes.org (Armin Rigo) Date: Wed, 24 May 2006 10:36:32 +0200 Subject: [pypy-dev] llvm backend In-Reply-To: <20060524140207.4bc81933.simon@arrowtheory.com> References: <20060524140207.4bc81933.simon@arrowtheory.com> Message-ID: <20060524083632.GA27954@code0.codespeak.net> Hi Simon, On Wed, May 24, 2006 at 02:02:07PM +1000, Simon Burton wrote: > $ python run.py > Traceback (most recent call last): > ... > File "/mnt/hda9/simonb/local/pypy-svn/pypy/annotation/bookkeeper.py", line 16, in ? > from pypy.annotation.listdef import ListDef, MOST_GENERAL_LISTDEF > ImportError: cannot import name ListDef Oups, mea culpa. I added an import at the top of run.py without checking that it was still working. Importing pypy.annotation.listdef before importing the other modules of that package creates a circular import bug. As a workaround to this particular cycle, I added an import line in the __init__ of the package. A bientot, Armin From eric at vanrietpaap.nl Wed May 24 10:43:13 2006 From: eric at vanrietpaap.nl (Eric van Riet Paap) Date: Wed, 24 May 2006 10:43:13 +0200 Subject: [pypy-dev] llvm backend In-Reply-To: <20060524140207.4bc81933.simon@arrowtheory.com> References: <20060524140207.4bc81933.simon@arrowtheory.com> Message-ID: <3DEDC750-E75A-440E-BC9E-50122FF003D1@vanrietpaap.nl> Simon, hi! Something I have to add to my last email... The profile based optimization stuff is only the the nightly benchmark cronjob (pypy/translator/goal/bench-cronjob.py) but could in principle (with proper support from genc and genllvm) be added as an option to ./translate.py . cheers Eric From arigo at tunes.org Wed May 24 10:50:04 2006 From: arigo at tunes.org (Armin Rigo) Date: Wed, 24 May 2006 10:50:04 +0200 Subject: [pypy-dev] llvm backend In-Reply-To: <20060524083632.GA27954@code0.codespeak.net> References: <20060524140207.4bc81933.simon@arrowtheory.com> <20060524083632.GA27954@code0.codespeak.net> Message-ID: <20060524085004.GB27954@code0.codespeak.net> ... On Wed, May 24, 2006 at 10:36:32AM +0200, Armin Rigo wrote: > As a workaround to this particular cycle, I added an import line in the > __init__ of the package. Now that I think about it, there are a few places (including this cycle) where modules import other modules only for the side-effects of having something registered; e.g. pypy.annotation.model imports pypy.annotation.unaryop and pypy.annotation.binaryop, and pypy.rpython.rtyper imports many of the pypy.rpython.r* modules. There is the same problem with every place that uses the extregistry (currently only rctypes and some ootypesystem functions and types). Maybe we could reorganize that by moving the necessary import statements to the __init__ of the appropriate packages. It certainly looks clearer and easier to control than these import loops. (It would also require us to finally do the long-pending clean-up of dividing the 'rpython' package in two: one for the rtyper and one for the modules of stuff that RPython programs are allowed to call, e.g. rarithmetic and objectmodel.) A bientot, Armin From arigo at tunes.org Wed May 24 13:47:28 2006 From: arigo at tunes.org (Armin Rigo) Date: Wed, 24 May 2006 13:47:28 +0200 Subject: [pypy-dev] Security ideas Message-ID: <20060524114728.GA6374@code0.codespeak.net> Hi all, On Monday I was at an inspiring seminar about (a specific form of) language-level security. I've collected the PyPy-ification of these ideas there: http://codespeak.net/svn/pypy/dist/pypy/doc/discussion/security-ideas.txt Although the focus is different, it makes me think that we could also use similar ideas to implement a form of 'rexec' (restricted execution), with functions compiled by secure() as in the draft above, but running at a priviledge level which is lower than the default ambiant level instead of higher. A bientot, Armin From jacob at strakt.com Wed May 24 14:21:54 2006 From: jacob at strakt.com (Jacob =?iso-8859-1?q?Hall=E9n?=) Date: Wed, 24 May 2006 14:21:54 +0200 Subject: [pypy-dev] Security ideas In-Reply-To: <20060524114728.GA6374@code0.codespeak.net> References: <20060524114728.GA6374@code0.codespeak.net> Message-ID: <200605241421.55683.jacob@strakt.com> On Wednesday 24 May 2006 13:47, Armin Rigo wrote: > Hi all, > > On Monday I was at an inspiring seminar about (a specific form of) > language-level security. I've collected the PyPy-ification of these > ideas there: > > http://codespeak.net/svn/pypy/dist/pypy/doc/discussion/security-ideas.txt > > Although the focus is different, it makes me think that we could also > use similar ideas to implement a form of 'rexec' (restricted execution), > with functions compiled by secure() as in the draft above, but running > at a priviledge level which is lower than the default ambiant level > instead of higher. This is quite interesting, but I have some concerns over the scheme presented. It seems to only take into consideration who gets to see the contents of an object. However, real information security is just as often concerned with who gets to set or modify the contents of an object. This produces security classifications that can't be represented as a linear scale, leading to a much more complex infrastructure for determining what classification to give to an object that receives it from multiple parents. Jacob From arigo at tunes.org Wed May 24 19:12:07 2006 From: arigo at tunes.org (Armin Rigo) Date: Wed, 24 May 2006 19:12:07 +0200 Subject: [pypy-dev] Security ideas In-Reply-To: <200605241421.55683.jacob@strakt.com> References: <20060524114728.GA6374@code0.codespeak.net> <200605241421.55683.jacob@strakt.com> Message-ID: <20060524171206.GA22544@code0.codespeak.net> Hi Jacob, On Wed, May 24, 2006 at 02:21:54PM +0200, Jacob Hall?n wrote: > This is quite interesting, but I have some concerns over the scheme presented. > It seems to only take into consideration who gets to see the contents of an > object. However, real information security is just as often concerned with > who gets to set or modify the contents of an object. This produces security > classifications that can't be represented as a linear scale, leading to a > much more complex infrastructure for determining what classification to give > to an object that receives it from multiple parents. Definitely. The scheme presented doesn't propose any classification, and it allows us to experiment with variants. As presented, the levels don't have to form a linear order but only have to be arranged in a semi-lattice (i.e. where "public" is the most permissive, and for any two levels L1 and L2 we can form a higher level "L1 union L2" that represents someone with both L1 and L2 credential levels). Moreover nothing is specified about what the level means and where exactly they are enforced, so we can definitely have object attributes that require different levels to be read or modified. There are many theories available; I've heard of one where each level is actually a set of tuples, where each tuple gives an "owner" name and a set of names of persons that can read the information. Also, it's easy to come up with anything custom in this approach -- as opposed to other approaches where changing the theory requires a whole compiler to be rewritten, syntax to be redesigned, etc. A bientot, Armin. From mwh at python.net Wed May 24 23:21:08 2006 From: mwh at python.net (Michael Hudson) Date: Wed, 24 May 2006 22:21:08 +0100 Subject: [pypy-dev] [pypy-svn] r27660 - in pypy/branch/njriley-trans/pypy: rpython/lltypesystem translator/c In-Reply-To: <20060524184814.8B3941006E@code0.codespeak.net> (njriley@codespeak.net's message of "Wed, 24 May 2006 20:48:14 +0200 (CEST)") References: <20060524184814.8B3941006E@code0.codespeak.net> Message-ID: <2mac973yrf.fsf@starship.python.net> njriley at codespeak.net writes: > Author: njriley > Date: Wed May 24 20:48:13 2006 > New Revision: 27660 > > Modified: > pypy/branch/njriley-trans/pypy/rpython/lltypesystem/rclass.py > pypy/branch/njriley-trans/pypy/translator/c/exceptiontransform.py > pypy/branch/njriley-trans/pypy/translator/c/node.py > Log: > Make exceptions thread-local Would this change make sense on the trunk too? Cheers, mwh -- Hmmm... its Sunday afternoon: I could do my work, or I could do a Fourier analysis of my computer's fan noise. -- Amit Muthu, ucam.chat (from Owen Dunn's summary of the year) From njriley at uiuc.edu Wed May 24 23:39:38 2006 From: njriley at uiuc.edu (Nicholas Riley) Date: Wed, 24 May 2006 16:39:38 -0500 Subject: [pypy-dev] [pypy-svn] r27660 - in pypy/branch/njriley-trans/pypy: rpython/lltypesystem translator/c In-Reply-To: <2mac973yrf.fsf@starship.python.net> References: <20060524184814.8B3941006E@code0.codespeak.net> <2mac973yrf.fsf@starship.python.net> Message-ID: <20060524213938.GA7462@uiuc.edu> On Wed, May 24, 2006 at 10:21:08PM +0100, Michael Hudson wrote: > njriley at codespeak.net writes: > > > Author: njriley > > Date: Wed May 24 20:48:13 2006 > > New Revision: 27660 > > > > Modified: > > pypy/branch/njriley-trans/pypy/rpython/lltypesystem/rclass.py > > pypy/branch/njriley-trans/pypy/translator/c/exceptiontransform.py > > pypy/branch/njriley-trans/pypy/translator/c/node.py > > Log: > > Make exceptions thread-local > > Would this change make sense on the trunk too? Assuming the GIL model, I don't see anywhere that the exception state gets swapped on a thread switch now; so, if this is the case, then yes, they'd make sense. With actual concurrent execution as I'm doing, that wouldn't help, of course. My changes would be fine if we could expect __thread (or some equivalent compiler-managed TLS) to be available everywhere, but PyPy doesn't make that assumption (see Samuele's work, e.g. pypy.translator.tool.cbuild.check_under_under_thread). The work I did for the old, pre-exceptiontransform model used pthread thread-specific variables, which would be possible as an alternative, but it'd take some work to automatically generate, and it'd impose considerably higher overhead on exception accesses. -- Nicholas Riley | From hpk at trillke.net Thu May 25 16:10:26 2006 From: hpk at trillke.net (holger krekel) Date: Thu, 25 May 2006 16:10:26 +0200 Subject: [pypy-dev] pypy-sync Monday 29th 5pm UTC+2 Message-ID: <20060525141026.GL12245@solar.trillke> Hi folks, the DDorf sprint, EuroPython talk deadlines and the 0.9 release are all jointly and quickly approaching. Let's have a brief #pypy-sync meeting Monday afternoon 5pm UTC+2, 29th May with these topics and try to co-ordinate last preps and efforts. And please have your "last/next/blockers" lines ready. have a nice weekend! best, holger From faassen at infrae.com Fri May 26 16:26:19 2006 From: faassen at infrae.com (Martijn Faassen) Date: Fri, 26 May 2006 16:26:19 +0200 Subject: [pypy-dev] Security ideas In-Reply-To: <20060524114728.GA6374@code0.codespeak.net> References: <20060524114728.GA6374@code0.codespeak.net> Message-ID: <4477100B.8050602@infrae.com> Armin Rigo wrote: > Hi all, > > On Monday I was at an inspiring seminar about (a specific form of) > language-level security. I've collected the PyPy-ification of these > ideas there: > > http://codespeak.net/svn/pypy/dist/pypy/doc/discussion/security-ideas.txt > > Although the focus is different, it makes me think that we could also > use similar ideas to implement a form of 'rexec' (restricted execution), > with functions compiled by secure() as in the draft above, but running > at a priviledge level which is lower than the default ambiant level > instead of higher. As a general note it might be useful to talk to Jim Fulton for real-world experience concerning language-level security in Python. I'll cc him so he at least is aware of your security ideas document. In Zope 2, there is a precompiler for untrusted Python code, offering, as far as I understand, true language-level security. In Zope 3 this approach has been dropped as hard to maintain and replaced with object level security (attribute access is controlled with a permission system). Regards, Martijn From pedronis at strakt.com Sat May 27 17:59:22 2006 From: pedronis at strakt.com (Samuele Pedroni) Date: Sat, 27 May 2006 17:59:22 +0200 Subject: [pypy-dev] pypy-sync Monday 29th 5pm UTC+2 In-Reply-To: <20060525141026.GL12245@solar.trillke> References: <20060525141026.GL12245@solar.trillke> Message-ID: <4478775A.4090409@strakt.com> holger krekel wrote: > Hi folks, > > the DDorf sprint, EuroPython talk deadlines and > the 0.9 release are all jointly and quickly approaching. > > Let's have a brief > > #pypy-sync meeting Monday afternoon 5pm UTC+2, 29th May > > with these topics and try to co-ordinate last preps > and efforts. And please have your "last/next/blockers" > lines ready. > I'm going to be traveling back to Gothenburg at that time on monday. Before leaving for the vacation I helped on various gc/stackless stuff and experimented with an idea that may be useufl for tasklet unpickling. Samuele. From nicolas.chauvat at logilab.fr Mon May 29 17:45:16 2006 From: nicolas.chauvat at logilab.fr (Nicolas Chauvat) Date: Mon, 29 May 2006 17:45:16 +0200 Subject: [pypy-dev] Security ideas In-Reply-To: <20060524114728.GA6374@code0.codespeak.net> References: <20060524114728.GA6374@code0.codespeak.net> Message-ID: <20060529154516.GE28271@crater.logilab.fr> On Wed, May 24, 2006 at 01:47:28PM +0200, Armin Rigo wrote: > On Monday I was at an inspiring seminar about (a specific form of) > language-level security. I've collected the PyPy-ification of these > ideas there: > > http://codespeak.net/svn/pypy/dist/pypy/doc/discussion/security-ideas.txt Let me add a reference, just in case it could be useful to future readers of the archives: http://en.wikipedia.org/wiki/E_programming_language -- Nicolas Chauvat logilab.fr - services en informatique avanc?e et gestion de connaissances From fijal at genesilico.pl Mon May 29 19:44:47 2006 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Mon, 29 May 2006 19:44:47 +0200 Subject: [pypy-dev] Ootypesystem based genjs (genjs2) Message-ID: <447B330F.4080804@genesilico.pl> I've added what I've done in last week with genjs, altough I'm not very happy with some issues. - LoopFinder does not find nested whiles (I think it's also gensquek problem) - Inheritance is not implemented - Dict support is not implemented - There is some code redundancy, because of copying from gencli. I hope to refactor it with antonio at DDorf sprint. - I don't like builtin method and function mapping solution, especially copying whole misc.js at the beginning of source file. Some function should be copied on higher, some on lower level. From fijal at genesilico.pl Mon May 29 20:04:24 2006 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Mon, 29 May 2006 20:04:24 +0200 Subject: [pypy-dev] SoC Message-ID: <447B37A8.8020908@genesilico.pl> Maybe that's a little bit late, but I've got broadband finally. I would like to thank all of pypy team for helping me with my SoC. From anto.cuni at gmail.com Mon May 29 20:22:40 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Mon, 29 May 2006 20:22:40 +0200 Subject: [pypy-dev] SoC In-Reply-To: <447B37A8.8020908@genesilico.pl> References: <447B37A8.8020908@genesilico.pl> Message-ID: <447B3BF0.6050502@gmail.com> Maciek Fijalkowski wrote: > Maybe that's a little bit late, but I've got broadband finally. > > I would like to thank all of pypy team for helping me with my SoC. A big 'thank you all' from me, too. I hope we'll be able to repay your trust by completing our works nicely :-). ciao Anto From l.oluyede at gmail.com Mon May 29 21:13:02 2006 From: l.oluyede at gmail.com (Lawrence Oluyede) Date: Mon, 29 May 2006 21:13:02 +0200 Subject: [pypy-dev] SoC In-Reply-To: <447B3BF0.6050502@gmail.com> References: <447B37A8.8020908@genesilico.pl> <447B3BF0.6050502@gmail.com> Message-ID: <9eebf5740605291213q9933a49rc9f9e935fd36b8a6@mail.gmail.com> On 5/29/06, Antonio Cuni wrote: > Maciek Fijalkowski wrote: > > Maybe that's a little bit late, but I've got broadband finally. > > > > I would like to thank all of pypy team for helping me with my SoC. > > A big 'thank you all' from me, too. I hope we'll be able to repay your > trust by completing our works nicely :-). I'm really lucky being part of the SoC project and have the opportunity to do something useful for the community of the language I use and love :-) Thanks guys. -- Lawrence http://www.oluyede.org/blog From anto.cuni at gmail.com Tue May 30 11:06:42 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 30 May 2006 11:06:42 +0200 Subject: [pypy-dev] Missed pypy-sync Message-ID: <447C0B22.2090200@gmail.com> Hi all, I've realized just now that I've missed the pypy-sync yesterday... sorry! Btw, here is my activity report: LAST: work on unifying rpython/ and cli/ tests; basic string support in gencli NEXT: DDorf sprint BLOCKERS: uni stuffs ciao Anto From paul at prescod.net Mon Jun 5 01:31:33 2006 From: paul at prescod.net (Paul Prescod) Date: Sun, 4 Jun 2006 19:31:33 -0400 Subject: [pypy-dev] Vancouver Python Workshop Message-ID: <1cb725390606041631r263b38f5r904ad3b0b6a59b0d@mail.gmail.com> We would be very appreciative if one or more of you PyPy experts would come to the Vancouver Python Workshop and inform of us of your progress on this exciting project. For invited talks like this, we offer a substantially reduced discount on our already minimal registration fees. In addition, we already have some really good talks organized on non-PyPy topics, including Python 3000 and IronPython, so the conference should be fun for whoever decides to come. http://www.vanpyz.org/conference Paul Prescod Vancouver Python Workshop -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20060604/af22eb4c/attachment.htm From bea at changemaker.nu Tue Jun 6 13:36:51 2006 From: bea at changemaker.nu (Beatrice During) Date: Tue, 06 Jun 2006 13:36:51 +0200 Subject: [pypy-dev] Vancouver Python Workshop In-Reply-To: <1cb725390606041631r263b38f5r904ad3b0b6a59b0d@mail.gmail.com> References: <1cb725390606041631r263b38f5r904ad3b0b6a59b0d@mail.gmail.com> Message-ID: <448568D3.8060806@changemaker.nu> Hi there Paul Prescod skrev: > We would be very appreciative if one or more of you PyPy experts would > come to the Vancouver Python Workshop and inform of us of your progress > on this exciting project. Thanks for inviting us! > For invited talks like this, we offer a substantially reduced discount > on our already minimal registration fees. In addition, we already have > some really good talks organized on non-PyPy topics, including Python > 3000 and IronPython, so the conference should be fun for whoever decides > to come. > > http://www.vanpyz.org/conference > > Paul Prescod > Vancouver Python Workshop > We are checking whether some one of us can participate - we will get back to you as soon as possible. Cheers Beatrice D?ring > ------------------------------------------------------------------------ > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev From mwh at python.net Wed Jun 7 12:49:57 2006 From: mwh at python.net (Michael Hudson) Date: Wed, 07 Jun 2006 11:49:57 +0100 Subject: [pypy-dev] Duesseldorf Sprint Report Days 1-4 Message-ID: <2mverdqlyi.fsf@starship.python.net> Hi pypy-dev! So we're sitting in the geography^Wcomputer science department of Heinrich-Heine-Universitaet Duesseldorf, the alleged employers of Armin and Michael, and (also alleged) educator of Carl. We've been doing the usual sprinty things of drinking beer, getting up late, going to bed even later (and so still not really getting much sleep) and hacking. An early task was finishing off the mostly complete weakref support. After drawing an extremely confusing diagram: http://codespeak.net/~mwh/blackboard.jpg Carl and Arre realized that there were a few strange cases that our model cannot support (in the case of a object and a weakref-with-callback to the same object both being in the same trash cycle, it is unclear whether the callback will be called). We plan to deal with this by calling anyone who claims to care insane. They then looked at getting CPython's test_weakref to run (something that's been on the "that would be nice" list for a looong time). This uncovered an amusing problem, a result of three facts: 1. the conservative Boehm collector cannot distinguish between a pointer and an integer that happens to have the same bit pattern as the pointer (this is what is meant by "conservative"). 2. the hash of a weakref object is the hash of the referenced object. 3. the default hash of a Python instance is it's id, which is usually the memory address cast to an int. This made weakref.WeakKeyDictionary rather poorly named. This was fixed by changing the id when using Boehm; the pointer is cast to an integer and then bitwise complemented. Accidentally, this means that that this: bool(id(object()) & 1) is now a check for pypy-c linked with the Boehm GC (use this fact with care). After solving the weak-keys-are-strong problem and adding a few calls to gc.collect() to CPython's test_weakref (to give Boehm a hurry up), test_weakref now passes. This leads on to the work that Anders and Holger did during the first day of the sprint: improving PyPy's "compliance score": http://codespeak.net/~hpk/pypy-testresult/ After a couple of months of relative neglect, this measure of how accurately we implement CPython's behaviour had slipped to around 80%. After fixing a few problems with the new interp-level implementation of the complex type and the above weakref work, we are back well over 95%. Michael helped Samuele remember exactly what he was thinking of in a branch that previously had only been the subject of hacking in the small hours of the night before his vacation. The goal of the nocturnal head bashing was to introduce the concept of an "explicit resume point" in an RPython program: a named, arbitrary location in the code and a way of transferring control to that point together with values for all live local variables. For example: def f(x, y, z): r = x + y rstack.resume_point("rp_f", r, z) return r + z defines a resume point called "rp_f" and records that the values of r and z must be supplied when the state is jumped to, like so: state = rstack.resume_state_create(None, "rp_f", 3, 4) result = rstack.resume_state_invoke(state) Something like this is a necessary piece in the puzzle that is implementing tasklet pickling, or more properly tasklet unpickling. This was a nice illustration of the "build one to throw away" concept: we were able to see that what Samuele had implemented as one operation really needed to be implemented as two (resume_state_create and resume_state_invoke in the above examples), and the rest of the coding only required moderate amounts of head bashing. By this morning, they had written a test that manually copied a (interp-level) coroutine, which validates the approach to at least some extent. You can also use explicit resume points to write some extremely confusing programs (as we discovered by mistake when debugging). The other major pieces of the tasklet pickling puzzle are being able to pickle and unpickle the core interpreter types such as frames, generators, tracebacks, ..., placing enough explicit resume points in the source of the Standard Interpreter to be able to resume a pickled tasklet and integration of the various pieces. Christian and Eric (working remotely; he arrives in Duesseldorf today) worked on the pickling and unpickling of interpreter types and have now implemented this for enough types that we now have no real option but to face the scary task of making something useful. For the last couple of days, the attention of Holger, Armin (still recovering from the 'flu, that's why we haven't mentioned him yet), Arre and a bit of Carl have been focussed on the elusive "ext-compiler": a tool designed to take an RPython module in a form suitable for making an "extension" for PyPy and translating it into an efficient extension module for CPython. This would clearly already be useful for purely algorithmic code, but the new-ish rctypes package of PyPy allows it to be used for modules that wrap an external library. While most of the basic technology is present for this, the tool is still far too rough to be unleashed even on unsuspecting Summer of Code students, never mind mere humans. Holger and Arre fleshed out ideas for how to build a more usable interface and wrote some documentation explaining this interface (who said open source projects are rubbish at documentation?). Armin began implementing support for typedefs, i.e. exposing custom types in the module interface, but ran into a small army of confusing levels of abstraction, which shift around as soon as you aren't staring at them. We're sure they're no match for Armin's planet-sized [#]_ brain though :) .. [#] http://mail.python.org/pipermail/python-dev/2003-January/032600.html While some of the above work involved much discussion and occasional voluminous cursing, two or three people have been sitting in the corner muttering about strange things like "ootypes" and "dot-net" and "javaskript", as well as producing a stream of high quality checkins (most of them have the log message "More ootypesystem tests", though). PyPy semi-regular Nik and newcomers and Summer of Code participants Antonio and Maciej worked on the ootypesystem backends, fixing many bugs and implementing new features, to the point that the richards and rpystone benchmarks can be run using the llinterpreter (now increasingly badly named; rinterpreter would prooooobably be better, but would have other unhelpful connotations). They couldn't run these benchmarks without a small modification: removing the print at the end that tells you how fast they ran (running things on the llinterpreter is so horrendously slow that this doesn't defeat the point of the excercise as you might at first think). The problem with printing is that it involves calling an external function, something that was not at all supported in the ootypesystem's world. So they decided to fix that :) And they did. Maciej also worked on the newer, ootype-using version of Javascript backend, and can now write python programs which bounce bub-n-bros characters around a browser window. This was a good way of getting Armin's attention :) Pozdrawiam, mwh & Carl Friedrich -- "Well, the old ones go Mmmmmbbbbzzzzttteeeeeep as they start up and the new ones go whupwhupwhupwhooopwhooooopwhooooooommmmmmmmmm." -- Graham Reed explains subway engines on asr From holger at merlinux.de Wed Jun 7 13:33:19 2006 From: holger at merlinux.de (holger krekel) Date: Wed, 7 Jun 2006 13:33:19 +0200 Subject: [pypy-dev] PyPy post-EP sprint (6-9th july) Message-ID: <20060607113319.GP25313@solar.trillke> Geneva/CERN Post-EuroPython PyPy sprint 6-9th July ==================================================== The next PyPy sprint will be held at CERN/Geneva right after the EuroPython conference. We specifically invite newcomers and people interested to get to know and get into PyPy development. We are going to work on a variety of topics and will give introductions to various aspects of PyPy (on the 6th morning). Some foreseen focus will be on the following topics: * The extension module compiler. Using it for implementing extension modules both for PyPy and CPython from the same source code. (for example if you have a Need for Speed :) * work and experiment with our to-be-relased 0.9 stackless features * work on high level backends: .NET, Javascript etc. * optimization of core Python data types, making full use of PyPy's flexible architecture and python-implemented (and then translated) type system. * You may even dare to dive into ongoing work on the JIT compiler * experimenting with novel security systems for Python, enabled by PyPy * and so on :) Registration ---------------------- If you'd like to come, please subscribe to the `pypy-sprint mailing list`_ and drop a note about your interests and post any questions. More organisational information will be send to that list. We'll keep a list of `people`_ which we'll update (which you can do so yourself if you have codespeak commit rights). Special note to Students! ---------------------------- As you might know, there are three students in Google's Summer of Code program who are working on projects related to PyPy. In addition we are planning our own "Summer of PyPy": we can cover the expenses of attending some of our sprints and provide mentorship. In this vein, if you have an interesting idea for a project relating to PyPy you should send us ("pypy-tb at codespeak.net") a proposal. We will consider it get back to you on whether we can fund your travel and accomodation. We expect to able to fund between 4 and 6 students in this way. As we still need to figure out budget/money issues, for now we can only promise for now that we can fund this Post-EP2006 sprint for such students. But don't hesitate to come to our irc channel, ask around and produce proposals! .. _`pypy-sprint mailing list`: http://codespeak.net/mailman/listinfo/pypy-sprint .. _`people`: people.html From tismer at stackless.com Thu Jun 8 11:34:07 2006 From: tismer at stackless.com (Christian Tismer) Date: Thu, 08 Jun 2006 11:34:07 +0200 Subject: [pypy-dev] The sound of a sprint :-) Message-ID: <4487EF0F.8040509@stackless.com> http://tismerysoft.de/~tismer/movies/sprint-ddorf/CIMG0648.AVI recorded in the backyard of building 25.12 of HHU Duesseldorf. You go through the cafeteria and pass two hallways, then there is an open air space with all this sound Python food. From bea at changemaker.nu Thu Jun 8 11:52:26 2006 From: bea at changemaker.nu (Beatrice During) Date: Thu, 08 Jun 2006 11:52:26 +0200 Subject: [pypy-dev] The sound of a sprint :-) In-Reply-To: <4487EF0F.8040509@stackless.com> References: <4487EF0F.8040509@stackless.com> Message-ID: <4487F35A.1090005@changemaker.nu> Hi there Christian Tismer skrev: > http://tismerysoft.de/~tismer/movies/sprint-ddorf/CIMG0648.AVI > > recorded in the backyard of building 25.12 of HHU Duesseldorf. > You go through the cafeteria and pass two hallways, then > there is an open air space with all this sound Python food. So if you kiss the frogs they turn into Armin and Christian and.... ;-) Good to hear that the sprint is productive - although I must say I did not understand everything that was said ;-) Cheers Bea From tismer at stackless.com Thu Jun 8 15:09:15 2006 From: tismer at stackless.com (Christian Tismer) Date: Thu, 08 Jun 2006 15:09:15 +0200 Subject: [pypy-dev] The sound of a sprint :-) In-Reply-To: <4487F35A.1090005@changemaker.nu> References: <4487EF0F.8040509@stackless.com> <4487F35A.1090005@changemaker.nu> Message-ID: <4488217B.8050609@stackless.com> Beatrice During wrote: > So if you kiss the frogs they turn into Armin and Christian and.... Aah. This was the missing spot. Of course, this is the picture. A bewitched sprint in the magic wood. Thinking to write a small poem :-> But I think *you* need to do the kissing. If I did it you would probably get a shaving machine or a fridge... ciao - quaaa quaaa bregegegex -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From arigo at tunes.org Fri Jun 9 10:24:50 2006 From: arigo at tunes.org (Armin Rigo) Date: Fri, 9 Jun 2006 10:24:50 +0200 Subject: [pypy-dev] Core sprint before EuroPython Message-ID: <20060609082450.GA10125@code0.codespeak.net> Hi all, ------------------------- Leysin Summer 2006 sprint ------------------------- As mentionned previously, there will be another sprint in Switzerland just before EuroPython. This one is meant for more "core" people in the broad sense, i.e. anyone that has already some knowledge about PyPy. For newcomers we recommend the post-EuroPython sprint at the conference venue :-) The main sprint focus will be on the JIT, but we can also work on other areas, as usual. This pre-EP sprint will take place from the 28th of June (included), to the 2nd of July (probably excluded, to have a break and travel to Geneva). The location is Leysin, about 1h45 away from Geneva by train. The sprint place is the same as during the previous Leysin sprints: Chalet Ermina, http://www.ermina.ch/ . I am arranging a group booking as usual. List of participants (add yourself there): http://codespeak.net/svn/pypy/extradoc/sprintinfo/leysin-summer-2006/people.txt A bientot, Armin From tismer at stackless.com Sat Jun 10 16:53:18 2006 From: tismer at stackless.com (Christian Tismer) Date: Sat, 10 Jun 2006 16:53:18 +0200 Subject: [pypy-dev] [pypy-svn] r28615 - in pypy/dist/pypy/module/stackless: . test In-Reply-To: <20060610130802.DAA3510053@code0.codespeak.net> References: <20060610130802.DAA3510053@code0.codespeak.net> Message-ID: <448ADCDE.2000702@stackless.com> arigo at codespeak.net wrote: > Author: arigo > Date: Sat Jun 10 15:07:59 2006 > New Revision: 28615 > > Modified: > pypy/dist/pypy/module/stackless/interp_coroutine.py > pypy/dist/pypy/module/stackless/test/test_interp_coroutine.py > Log: > Long test and one-liner fix for coroutines and app-coroutines, Ah, well. This is a typical one. It is always hard to tell "who am I" in the coroutine stuff. By default, it is someone else :-) -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From stephan.diehl at gmx.net Mon Jun 12 19:25:39 2006 From: stephan.diehl at gmx.net (Stephan Diehl) Date: Mon, 12 Jun 2006 19:25:39 +0200 Subject: [pypy-dev] christian is ill Message-ID: <448DA393.1030207@gmx.net> Hi all, I've just talked with Christian. Unfortunatelly, he's ill and at a hospital. He won't be able to work for a couple of days. I guess that it's a good bet that he won't be available before friday. Cheers Stephan From mwh at python.net Tue Jun 13 00:11:46 2006 From: mwh at python.net (Michael Hudson) Date: Mon, 12 Jun 2006 23:11:46 +0100 Subject: [pypy-dev] pypy-c vs cpython tests Message-ID: <2md5deowgt.fsf@starship.python.net> I've spent an aggravating little while running CPython's regrtest with a recent pypy-c (default options, so boehm and no fancy stuff). After a bit of trial and error, this was the result: 69 tests failed: GC Warning: Finalization cycle involving d66c120 GC Warning: Finalization cycle involving a8e06c0 GC Warning: Finalization cycle involving d286d08 GC Warning: Finalization cycle involving d180780 GC Warning: Finalization cycle involving d12a108 test___all__ test_anydbm test_calendar test_cfgparser test_class test_codecencodings_cn test_codecencodings_hk test_codecencodings_jp test_codecencodings_kr test_codecencodings_tw test_coercion test_commands test_compare test_compiler test_cookielib test_cpickle test_decimal test_descr test_descrtut test_dis test_distutils test_doctest2 test_dumbdbm test_extcall test_file test_filecmp test_fileinput test_frozen test_gc test_getargs test_gettext test_glob test_inspect test_iter test_mailbox test_marshal test_mhlib test_mimetools test_minidom test_multibytecodec test_new test_os test_peepholer test_pep292 test_pickle test_pickletools test_posixpath test_profile test_pyclbr test_random test_repr test_scope test_sort test_strftime test_stringprep test_strptime test_structseq test_tarfile test_tempfile test_time test_traceback test_transformer test_ucn test_unicodedata test_weakref test_whichdb test_xmlrpc test_xpickle test_zipfile 97 tests skipped: test__locale test_aepack test_al test_applesingle test_asynchat test_audioop test_bsddb test_bsddb185 test_bsddb3 test_bz2 test_capi test_cd test_cgi test_cl test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_crypt test_csv test_curses test_dbm test_dl test_doctest test_email test_email_codecs test_fcntl test_fork1 test_gdbm test_getargs2 test_gl test_grp test_gzip test_hotshot test_httplib test_imageop test_imaplib test_imgfile test_imp test_ioctl test_largefile test_linuxaudiodev test_locale test_logging test_macfs test_macostools test_mimetypes test_mmap test_nis test_normalization test_openpty test_ossaudiodev test_pep277 test_plistlib test_poll test_popen test_popen2 test_pty test_pwd test_pyexpat test_queue test_regex test_resource test_rgbimg test_robotparser test_sax test_scriptpackages test_select test_signal test_site test_socket test_socket_ssl test_socketserver test_strop test_subprocess test_sunaudiodev test_sundry test_symtable test_tcl test_thread test_threaded_import test_threadedtempfile test_threading test_threading_local test_threadsignals test_timeout test_timing test_unicode_file test_urllib test_urllib2 test_urllib2net test_urllibnet test_winreg test_winsound test_zipimport test_zlib Segmentation fault I had to run with '-x test_datetime test_format test_shutil test_shelve test_str test_string test_unicode test_userstring' in the end. test_datetime, test_shutil and test_format kill the build. the string tests fail in ways that suggest that the boehm gc really didn't expect the usage patterns they use. I don't remember why I skipped test_shelve now :) I count around 250 test_* files in CPython's test suite, so we actually pass around 40% of them, and a lot of the skips and plenty of the fails are missing (parts of) extension modules, so we're not really in *that* bad a shape. But fixing the crashers would be nice. We need to get some automated version of this set up. Cheers, mwh -- ARTHUR: Why should he want to know where his towel is? FORD: Everybody should know where his towel is. ARTHUR: I think your head's come undone. -- The Hitch-Hikers Guide to the Galaxy, Episode 7 From mwh at python.net Tue Jun 13 09:52:24 2006 From: mwh at python.net (Michael Hudson) Date: Tue, 13 Jun 2006 08:52:24 +0100 Subject: [pypy-dev] pypy-c vs cpython tests References: <2md5deowgt.fsf@starship.python.net> Message-ID: <2m3be9pk5j.fsf@starship.python.net> Michael Hudson writes: > I've spent an aggravating little while running CPython's regrtest with > a recent pypy-c (default options, so boehm and no fancy stuff). > > After a bit of trial and error, this was the result: > > 69 tests failed: > 97 tests skipped: In some good news, the stacklessgc build passes exactly one fewer test (test_builtin, known issue) and doesn't even do this: > Segmentation fault and the end :) Though it does exit with a rather strange ImportError: No module named _socket Haven't dug yet. On this evidence, I think I'm inclined to say that framework GC builds are at least as solid as the Boehm builds. Cheers, mwh -- C is not clean -- the language has _many_ gotchas and traps, and although its semantics are _simple_ in some sense, it is not any cleaner than the assembly-language design it is based on. -- Erik Naggum, comp.lang.lisp From hpk at trillke.net Thu Jun 15 07:56:49 2006 From: hpk at trillke.net (holger krekel) Date: Thu, 15 Jun 2006 07:56:49 +0200 Subject: [pypy-dev] 0.9 release meeting 11AM (GMT+2) Message-ID: <20060615055649.GV25313@solar.trillke> Hi folks, today at 11AM (GMT+2) we'll meet on #pypy-release for a meeting regarding the upcoming 0.9 release Topics will be: * extension compiler * stackless features * (high level backends / snapshot of work) * logic/constraint works * feature freeze (branching off?) * documentation about all this and whatever we decide to be relevant for it. best, holger From stephan.diehl at gmx.net Thu Jun 15 16:05:34 2006 From: stephan.diehl at gmx.net (Stephan Diehl) Date: Thu, 15 Jun 2006 16:05:34 +0200 Subject: [pypy-dev] stackless (the module) exception handling Message-ID: <4491692E.60306@gmx.net> this is all very confusing, but I'll do my best. The fun stuff is happening in interp_coroutine.py which looks at the moment like this: -------------------------------------------------------------- def _execute(self, incoming_frame): syncstate.switched(incoming_frame) state = self.costate try: try: self.thunk.call() finally: self.finished() self.thunk = None resume_point("coroutine__bind", self, state) except CoroutineExit: # ignore a shutdown exception pass except Exception, e: # redirect all unhandled exceptions to the parent syncstate.things_to_do = True syncstate.temp_exc = e while self.parent is not None and self.parent.frame is None: # greenlet behavior is fine self.parent = self.parent.parent return state.update(self.parent) ------------------------------------------- I've inserted the 'self.finished()' hook yesterday, to give applevel coroutines (in this case tasklets) the chance to do their last rites. Specifically, I needed that so that the stackless scheduler has a chance to stay in sync. Now, the tasklet.finished method contains a 'schedule'. What happens is that the Exception handling in _execute above is ignored. Bad. As a next test, I've moved the 'self.finished()' call right before the return. This certainly gives the exception handling a chance to do it's thing and the stackless code still does it things. But now I get the following error message from py.py: Traceback (most recent call last): File "/usr/local/bin/py.py", line 207, in ? sys.exit(main_(sys.argv)) File "/usr/local/bin/py.py", line 118, in main_ if not main.run_toplevel(space, doit, verbose=Options.verbose): File "/home/stephan/projects/pypy-dist/pypy/interpreter/main.py", line 80, in run_toplevel f() File "/usr/local/bin/py.py", line 105, in doit main.run_file(args[0], space=space) File "/home/stephan/projects/pypy-dist/pypy/interpreter/main.py", line 67, in run_file run_string(istring, filename, space) File "/home/stephan/projects/pypy-dist/pypy/interpreter/main.py", line 58, in run_string _run_eval_string(source, filename, space, False) File "/home/stephan/projects/pypy-dist/pypy/interpreter/main.py", line 47, in _run_eval_string retval = pycode.exec_code(space, w_globals, w_globals) File "/home/stephan/projects/pypy-dist/pypy/interpreter/eval.py", line 26, in exec_code return frame.run() File "/home/stephan/projects/pypy-dist/pypy/interpreter/eval.py", line 162, in resume executioncontext.leave(self) File "/home/stephan/projects/pypy-dist/pypy/interpreter/executioncontext.py", line 38, in leave self.framestack.pop() File "/home/stephan/projects/pypy-dist/pypy/interpreter/miscutils.py", line 34, in pop return self.items.pop() IndexError: pop from empty list ---------------------------------------------------- So, somehow I either have nice stackless scheduler clean up code, or I have proper exception handling. If somebody has a nice idea what to do, I'd be very happy to hear about that. Cheers Stephan From mwh at python.net Thu Jun 15 16:36:20 2006 From: mwh at python.net (Michael Hudson) Date: Thu, 15 Jun 2006 15:36:20 +0100 Subject: [pypy-dev] 0.9 release meeting 11AM (GMT+2) References: <20060615055649.GV25313@solar.trillke> Message-ID: <2mslm6mqor.fsf@starship.python.net> hpk at trillke.net (holger krekel) writes: > Hi folks, > > today at 11AM (GMT+2) we'll meet on #pypy-release > for a meeting regarding the upcoming 0.9 release Here's an attempt at a summary of the current status and current plans for work before the release. Our original plan called for a release branch to be made tomorrow night. This now seems rather too early, and we're aiming for makint branch on Monday evening instead. The release is still planned for Thursday. If anyone between now and the 0.9 release commits code that breaks translation, it gets reverted. Please don't :) > * extension compiler The status is that it works, apart from four things: * it doesn't support modules that have an app-level component properly * you can't define "special methods" (__add__, etc) on exported types * you can't test the mixed module on top of cpython * documenation. We decided that the second of these is not that important at this stage, but the others need work. Armin is going to work on it today, at least. Armin is going to take overall responsibility for this item. > * stackless features The main problem with this item is that the status is that we don't know what the status is :( We need a way of running tests with a pypy-c and more tests to run. There are three subtopics: * tasklet cloning * tasklet pickling * exposing features to applications. A while ago I promised to play with exposed tasklet cloning to app level; various stupid mistakes on my part, a lot of distractions and the fact that stacklessgc builds take ages mean I'm not done yet, but I'm getting there. We still think tasklet pickling is working, but given the lack of tests, it's hard to be sure (see above). The exposing of app level features to applications is roughly speaking done, but needs more tests (see above). Hogler had hoped to work on running tests with pypy-c but may not be able to. Michael and him will discuss approaches and one of them will implement something by tomorrow, allowing Eric, Stephan, me, Carl and possibly some others to write tests over the weekend or next week. Michael and Samuele will coordinate this issue. > * (high level backends / snapshot of work) We'd like to have some kind of preview and documentation of a high level backend in the 0.9 release, which basically means the CLI backend. Unfornuately "Mr. CLI Backend, Antonio, has exams this and next week and so he doesn't have much time to work on this. Nevertheless, him and Armin will confer and decide on a practical approach -- some integration of the CLI backend into translatorshell.py and driver.py and a section in getting-started seem like a resonable target. Armin will coordinate this issue. > * logic/constraint works It would be nice to have a translatable logic object space in 0.9, but this is a lot of work and isn't going to happen in time. It more or less works on top of py.py though, so we'll settle for making it work properly on py.py (Carl) and writing some documentation (Anders). The documentation was promised by Tuesday night. Anders and Carl will coordinate this issue. > * feature freeze (branching off?) As discussed above, aiming for Monday night. > * documentation about all this A certain amount of discussion about documentation took place while talking about the above items, and in general the person responsible for coordinating an item is responsible for the docs for that item too. In addition, I promised to review translation.txt (which is now rather out of date) and Arre, with Armin's help, promised to document rctypes. There was an additional item: * general stability that got mentioned at the end of the meeting. We've now fixed _most_ of the problems that running CPython's test suite with pypy-c revealed, but there are issues around the area of very large (i.e. close to LONG_MAX) allocations. It's not clear what we can get done about this before 0.9, but hopefully we can clear most of them up. Cheers, mwh -- To summarise the summary of the summary:- people are a problem. -- The Hitch-Hikers Guide to the Galaxy, Episode 12 From anto.cuni at gmail.com Fri Jun 16 21:14:00 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Fri, 16 Jun 2006 21:14:00 +0200 Subject: [pypy-dev] Ladies and gentlemen... Message-ID: <449302F8.8050509@gmail.com> ... I'm proud to announce that gencli is the first high level backend that can compile and run rpystone and richards :-). Some early benchmarks (I post them here so I'll know where to find them when I need it later :-)): pystone ------- gencli: 177429.668653 rpystones/second genc: 4926108.374384 rpystones/second genc w/o backendopt: 1592356.687898 rpystoned/second richards -------- gencli: 28.652590 ms/iteration genc: 7.431851 ms/iteration genc w/o backendopt: 16.203486 ms/iteration ciao Anto From arigo at tunes.org Sat Jun 17 12:15:59 2006 From: arigo at tunes.org (Armin Rigo) Date: Sat, 17 Jun 2006 12:15:59 +0200 Subject: [pypy-dev] 0.9 release meeting 11AM (GMT+2) In-Reply-To: <2mslm6mqor.fsf@starship.python.net> References: <20060615055649.GV25313@solar.trillke> <2mslm6mqor.fsf@starship.python.net> Message-ID: <20060617101558.GA17735@code0.codespeak.net> Hi, Here is an update: On Thu, Jun 15, 2006 at 03:36:20PM +0100, Michael Hudson wrote: > > * extension compiler > > * it doesn't support modules that have an app-level component properly > * you can't define "special methods" (__add__, etc) on exported types > * you can't test the mixed module on top of cpython > * documenation. I finished this, apart from the "special methods" item. > Arre, with Armin's help, promised to document > rctypes. I collected the existing documentation, and removed bits that are out-of-date design documents. It's now all in pypy/doc/rctypes.txt. It could probably do with some completing or polishing. A bientot, Armin From arigo at tunes.org Sat Jun 17 12:17:45 2006 From: arigo at tunes.org (Armin Rigo) Date: Sat, 17 Jun 2006 12:17:45 +0200 Subject: [pypy-dev] Ladies and gentlemen... In-Reply-To: <449302F8.8050509@gmail.com> References: <449302F8.8050509@gmail.com> Message-ID: <20060617101745.GB17735@code0.codespeak.net> Hi Anto, On Fri, Jun 16, 2006 at 09:14:00PM +0200, Antonio Cuni wrote: > ... I'm proud to announce that gencli is the first high level backend > that can compile and run rpystone and richards :-). (Applause) Great! Now translating the whole of PyPy can't be that far away any more :-) Armin From arigo at tunes.org Sun Jun 18 23:44:49 2006 From: arigo at tunes.org (Armin Rigo) Date: Sun, 18 Jun 2006 23:44:49 +0200 Subject: [pypy-dev] Random updates Message-ID: <20060618214449.GA15178@code0.codespeak.net> Hi all, I did a bit of detective work this evening, and found the following. The reason Eric's cron job is no longer showing results for stackless builds is that we broke old-stackless builds of PyPy by adding resume point operations (for unpickling tasklets). They end up as OP_RESUME_STATE_CREATE() macro calls in genc! Argh. Either with fix by somehow detecting in the _stackless module if we are being compiled with old- or new-stackless. Or we don't care any more about the old-stackless and declare it dead for the purpose of compiling PyPy. I also found out that the test summary page at http://snake.cs.uni-duesseldorf.de/pypytest/summary.html was doing a great job at hiding the fact that the tests no longer ran to the end, and that, for more than two weeks now. Either test_ee in pyllvm was getting a SIGIOT signal (people who ever heard about this signal, raise your hand), or test_wrapping was segfaulting (unreproductably - fun). I still have no clue why test_wrapping occasionally segfaults. In any case I fixed the output by considering abnormal termination as a failure. It shows up now at the beginning of the failures. I also modified the misleading '.', which previously only meant 'this test did not fail in that revision' -- e.g. because they were not run at all :-/ Now, a '.' really means 'known to have passed in that revision'. Otherwise you can get an 's', or a ' ' if the test wasn't run at all. Note that before the 12th of June, the necessary information was not collected by the run, so all non-failures there show up as ' ' even if the test was actually run. A bientot, Armin. From elmo13 at jippii.fi Mon Jun 19 00:36:54 2006 From: elmo13 at jippii.fi (=?ISO-8859-1?Q?Elmo_M=E4ntynen?=) Date: Mon, 19 Jun 2006 01:36:54 +0300 Subject: [pypy-dev] Random updates In-Reply-To: <20060618214449.GA15178@code0.codespeak.net> References: <20060618214449.GA15178@code0.codespeak.net> Message-ID: <4495D586.4090907@jippii.fi> Armin Rigo wrote: > ... > > I also found out that the test summary page at > http://snake.cs.uni-duesseldorf.de/pypytest/summary.html was doing a > great job at hiding the fact that the tests no longer ran to the end, > and that, for more than two weeks now. Either test_ee in pyllvm was > getting a SIGIOT signal (people who ever heard about this signal, raise > your hand), or test_wrapping was segfaulting (unreproductably - fun). > I still have no clue why test_wrapping occasionally segfaults. > ... > I reported a segfault some time ago (python2.4 specific), of which you can read in the archives, but no one seemed to be interested and I didn't want to push it. My post was about compilemodule.py, but this might very well be related since they use the same code quite the same way I think. The segfaulting stopped at some point, but if you try a revision around the time that I sent my post (11.5), you might be able to reproduce it. The segfaulting itself seemed to me as a bug in python, but since it didn't happen with 2.5 (some other error did though =), I didn't see it as so important. Maybe I should've pushed it harder =). Just say if I'm totally off the course, Elmo From mistobaan at gmail.com Mon Jun 19 18:28:48 2006 From: mistobaan at gmail.com (Fabrizio Milo) Date: Mon, 19 Jun 2006 18:28:48 +0200 Subject: [pypy-dev] Ideas for Student Proposal. Message-ID: Hello to everybody, I am an Italian student who will participate in the euro-pypy-sprint. Looking for a good idea for a proposal, I saw https://codespeak.net/issue/pypy-dev/ and many issues are interesting. like: - https://codespeak.net/issue/pypy-dev/issue87 - https://codespeak.net/issue/pypy-dev/issue111 ( or/ and ) - https://codespeak.net/issue/pypy-dev/issue205 - https://codespeak.net/issue/pypy-dev/issue224 The common subject here is "Improving test machinery". What do you think, can issue87 + issue111 be a good base for a proposal? Is it required to produce some proof-of-concept? Are there other ideas in the community not listed in that page? I am a good programmer ( I am a the end of a Computer Science Master course in Rome ) so don't let me escape from helping this wonderful project :) Fabrizio Milo From hpk at trillke.net Mon Jun 19 18:57:19 2006 From: hpk at trillke.net (holger krekel) Date: Mon, 19 Jun 2006 18:57:19 +0200 Subject: [pypy-dev] Ideas for Student Proposal. In-Reply-To: References: Message-ID: <20060619165719.GS25313@solar.trillke> Hi Fabrizio! On Mon, Jun 19, 2006 at 18:28 +0200, Fabrizio Milo wrote: > I am an Italian student who will participate in the euro-pypy-sprint. cool. btw, did you already send a note to pypy-sprint at codespeak.net? please also state your timeframe and interests there (redundantly). > Looking for a good idea for a proposal, I saw > https://codespeak.net/issue/pypy-dev/ and many issues are interesting. > > like: > - https://codespeak.net/issue/pypy-dev/issue87 > - https://codespeak.net/issue/pypy-dev/issue111 > > ( or/ and ) > - https://codespeak.net/issue/pypy-dev/issue205 > - https://codespeak.net/issue/pypy-dev/issue224 > > The common subject here is "Improving test machinery". Yes, however, issue224 is more about adding tests for pypy stackless functionality (needed _this_ week :) while the other three are about actually improving the underlying test mechanisms mostly and thus belong to py.test/py library and not primarily to PyPy (although both are involved). > What do you think, can issue87 + issue111 be a good base for a proposal? > Is it required to produce some proof-of-concept? I'd suggest that you draft at least the contents of issue87, issue111 (and bits of/integration with issue205) into a proposal under your proposed headline working title. You may bounce it to me or to py-dev at codespeak.net (the py lib / py.test developer mailing list) if you have further questions. I guess the already known sub tasks would be "adding doctest support" and "refined error reporting" for py.test ensuring suitability for PyPy testing purposes (including integrating/helping with Issue205 - the pypy-c test runs). PyPy adds quite some custom functionality on top of py.test and it is sometimes a challenge to design things flexibly enough for it. > Are there other ideas in the community not listed in that page? Related to testing there is the issue of distributing tests across many machines (using py.execnet as an ad-hoc networking mechanism if you ask me). I see this as a rather separate topic from the other three, though. But there is nothing keeping you from adding another proposal after you finished your first :) > I am a good programmer ( I am a the end of a Computer Science Master > course in Rome ) > so don't let me escape from helping this wonderful project :) We will do our best :) Summary: it's best if you look around some more, ask questions and draft a proposal and let me or better py-dev review it. cheers, holger From arigo at tunes.org Wed Jun 21 11:10:29 2006 From: arigo at tunes.org (Armin Rigo) Date: Wed, 21 Jun 2006 11:10:29 +0200 Subject: [pypy-dev] Random updates In-Reply-To: <4495D586.4090907@jippii.fi> References: <20060618214449.GA15178@code0.codespeak.net> <4495D586.4090907@jippii.fi> Message-ID: <20060621091029.GA659@code0.codespeak.net> Hi Elmo, On Mon, Jun 19, 2006 at 01:36:54AM +0300, Elmo M?ntynen wrote: > > I also found out that the test summary page at > > http://snake.cs.uni-duesseldorf.de/pypytest/summary.html was doing a > > great job at hiding the fact that the tests no longer ran to the end, > > and that, for more than two weeks now. Either test_ee in pyllvm was > > getting a SIGIOT signal (people who ever heard about this signal, raise > > your hand), or test_wrapping was segfaulting (unreproductably - fun). > > I still have no clue why test_wrapping occasionally segfaults. > > ... > > > I reported a segfault some time ago (python2.4 specific), of which you > can read in the archives, but no one seemed to be interested and I > didn't want to push it. My post was about > compilemodule.py, but this might very well be related since they use > the same code quite the same way I think. No, test_wrapping is about something different than what compilemodule.py uses. The kind of wrapping tested here is only used by external tools developed by Christian Tismer, which he meanwhile checked in as pypy/translator/rool/raymond.py. As I have no full clue about what test_wrapping does (and it uses names like "test_asd", which makes it crystal clear what the intent of the test is), I think I can live with this crash for now. Sorry for not having come back to you about the segfault you reported, I could never reproduce it myself... > The segfaulting itself seemed to me as a bug in python, but since it > didn't happen with 2.5 (some other error did though =), I didn't see it > as so important. Maybe I should've pushed it harder =). So what is the current status: does compilemodule.py _demo work for you? On several Python versions? A bientot, Armin From elmo13 at jippii.fi Thu Jun 22 01:20:54 2006 From: elmo13 at jippii.fi (=?ISO-8859-1?Q?Elmo_M=E4ntynen?=) Date: Thu, 22 Jun 2006 02:20:54 +0300 Subject: [pypy-dev] Random updates In-Reply-To: <20060621091029.GA659@code0.codespeak.net> References: <20060618214449.GA15178@code0.codespeak.net> <4495D586.4090907@jippii.fi> <20060621091029.GA659@code0.codespeak.net> Message-ID: <4499D456.2030706@jippii.fi> Armin Rigo wrote: > Hi Elmo, > > On Mon, Jun 19, 2006 at 01:36:54AM +0300, Elmo M?ntynen wrote: > >>> I also found out that the test summary page at >>> http://snake.cs.uni-duesseldorf.de/pypytest/summary.html was doing a >>> great job at hiding the fact that the tests no longer ran to the end, >>> and that, for more than two weeks now. Either test_ee in pyllvm was >>> getting a SIGIOT signal (people who ever heard about this signal, raise >>> your hand), or test_wrapping was segfaulting (unreproductably - fun). >>> I still have no clue why test_wrapping occasionally segfaults. >>> ... >>> >>> >> I reported a segfault some time ago (python2.4 specific), of which you >> can read in the archives, but no one seemed to be interested and I >> didn't want to push it. My post was about >> compilemodule.py, but this might very well be related since they use >> the same code quite the same way I think. >> > > No, test_wrapping is about something different than what > compilemodule.py uses. The kind of wrapping tested here is only used by > external tools developed by Christian Tismer, which he meanwhile checked > in as pypy/translator/rool/raymond.py. As I have no full clue about > what test_wrapping does (and it uses names like "test_asd", which makes > it crystal clear what the intent of the test is), I think I can live > with this crash for now. > > Sorry for not having come back to you about the segfault you reported, I > could never reproduce it myself... > > >> The segfaulting itself seemed to me as a bug in python, but since it >> didn't happen with 2.5 (some other error did though =), I didn't see it >> as so important. Maybe I should've pushed it harder =). >> > > So what is the current status: does compilemodule.py _demo work for you? > On several Python versions? > 2.3 seems to work nicely, 2.4 segfaults (?! I thought it worked again), which I'll try to reproduce with different machine (I'll report soon): ... [translation:info] with policy: pypy.objspace.cpy.ann_policy.CPyAnnotatorPolicy ... segfault and latest 2.5 works. This might just be something on my end, but any pointers on what might be wrong would be useful. A small complaint about py.test: a segfaulting test crashes py.test with EOFError and with an older rev of 2.5 compilemodule.py _demo errored and I got a pdb prompt, but py.test test_compilemodule.py reported a success nonetheless!? From hpk at trillke.net Thu Jun 22 07:49:13 2006 From: hpk at trillke.net (holger krekel) Date: Thu, 22 Jun 2006 07:49:13 +0200 Subject: [pypy-dev] Random updates In-Reply-To: <4499D456.2030706@jippii.fi> References: <20060618214449.GA15178@code0.codespeak.net> <4495D586.4090907@jippii.fi> <20060621091029.GA659@code0.codespeak.net> <4499D456.2030706@jippii.fi> Message-ID: <20060622054913.GS25313@solar.trillke> Hi Elmo, On Thu, Jun 22, 2006 at 02:20 +0300, Elmo M?ntynen wrote: > A small complaint about py.test: a segfaulting test crashes py.test with > EOFError and with an older rev of 2.5 > compilemodule.py _demo errored and I got a pdb prompt, but py.test > test_compilemodule.py reported a success nonetheless!? Maybe the test doesn't do the same test as the direct invocation of the compilemodule.py does? Either way, i added a feature issue (https://codespeak.net/issue/pypy-dev/issue233) so that py.test allows more robust test modes. Thanks for the comment! It has been an issue in the past but its severity grew with more and more tests being able to segfault. best, holger From elmo13 at jippii.fi Thu Jun 22 11:25:50 2006 From: elmo13 at jippii.fi (=?ISO-8859-1?Q?Elmo_M=E4ntynen?=) Date: Thu, 22 Jun 2006 12:25:50 +0300 Subject: [pypy-dev] Random updates In-Reply-To: <20060622054913.GS25313@solar.trillke> References: <20060618214449.GA15178@code0.codespeak.net> <4495D586.4090907@jippii.fi> <20060621091029.GA659@code0.codespeak.net> <4499D456.2030706@jippii.fi> <20060622054913.GS25313@solar.trillke> Message-ID: <449A621E.5060206@jippii.fi> holger krekel wrote: > Hi Elmo, > > On Thu, Jun 22, 2006 at 02:20 +0300, Elmo M?ntynen wrote: > >> A small complaint about py.test: a segfaulting test crashes py.test with >> EOFError and with an older rev of 2.5 >> compilemodule.py _demo errored and I got a pdb prompt, but py.test >> test_compilemodule.py reported a success nonetheless!? >> > > Maybe the test doesn't do the same test as the direct invocation > of the compilemodule.py does? > Actually, they seem to do the very same thing: they call compilemodule('_demo'). If they weren't doing the same thing, would the test be testing the right thing? Anyway, I happened to update my python2.5 and don't have any record of what happened, so I can't be sure about it. > Either way, i added a feature issue (https://codespeak.net/issue/pypy-dev/issue233) > so that py.test allows more robust test modes. Thanks for the comment! > It has been an issue in the past but its severity grew with more and > more tests being able to segfault. > > best, > > holger > From arigo at tunes.org Thu Jun 22 11:38:13 2006 From: arigo at tunes.org (Armin Rigo) Date: Thu, 22 Jun 2006 11:38:13 +0200 Subject: [pypy-dev] Random updates In-Reply-To: <4499D456.2030706@jippii.fi> References: <20060618214449.GA15178@code0.codespeak.net> <4495D586.4090907@jippii.fi> <20060621091029.GA659@code0.codespeak.net> <4499D456.2030706@jippii.fi> Message-ID: <20060622093813.GA26980@code0.codespeak.net> Hi Elmo, On Thu, Jun 22, 2006 at 02:20:54AM +0300, Elmo M?ntynen wrote: > 2.3 seems to work nicely, 2.4 segfaults (?! I thought it worked again), > which I'll try to reproduce with different machine (I'll report soon): > ... > [translation:info] with policy: > pypy.objspace.cpy.ann_policy.CPyAnnotatorPolicy > ... > segfault > and latest 2.5 works. We'd need some much more precise info about this to be able to do anything. Which version of CPython is that precisely, which PyPy revision, which OS? Can you also please paste the whole output before the segfault? If you manage to get a error and a pdb prompt with another Python version, can you also please paste the whole output here, together with the exact CPython version number? Thanks! Armin From elmo13 at jippii.fi Thu Jun 22 11:44:56 2006 From: elmo13 at jippii.fi (=?ISO-8859-1?Q?Elmo_M=E4ntynen?=) Date: Thu, 22 Jun 2006 12:44:56 +0300 Subject: [pypy-dev] Random updates In-Reply-To: <449A621E.5060206@jippii.fi> References: <20060618214449.GA15178@code0.codespeak.net> <4495D586.4090907@jippii.fi> <20060621091029.GA659@code0.codespeak.net> <4499D456.2030706@jippii.fi> <20060622054913.GS25313@solar.trillke> <449A621E.5060206@jippii.fi> Message-ID: <449A6698.6030109@jippii.fi> Elmo M?ntynen wrote: > holger krekel wrote: > >> Hi Elmo, >> >> On Thu, Jun 22, 2006 at 02:20 +0300, Elmo M?ntynen wrote: >> >> >>> A small complaint about py.test: a segfaulting test crashes py.test with >>> EOFError and with an older rev of 2.5 >>> compilemodule.py _demo errored and I got a pdb prompt, but py.test >>> test_compilemodule.py reported a success nonetheless!? >>> >>> >> Maybe the test doesn't do the same test as the direct invocation >> of the compilemodule.py does? >> >> > Actually, they seem to do the very same thing: they call > compilemodule('_demo'). If they weren't doing the same thing, would the > test be testing the right thing? Anyway, I happened to update my > python2.5 and don't have any record of what happened, so I can't be sure > about it. > Actually, there's one difference: try: driver.proceed(['compile_c']) except SystemExit: raise except: if not interactive: raise debug(driver) raise SystemExit(1) return driver.cbuilder.c_ext_module where interactive is True for the direct invocation. Could this just be the fault of a broken CPython interpreter? From elmo13 at jippii.fi Thu Jun 22 12:38:26 2006 From: elmo13 at jippii.fi (=?ISO-8859-1?Q?Elmo_M=E4ntynen?=) Date: Thu, 22 Jun 2006 13:38:26 +0300 Subject: [pypy-dev] Random updates In-Reply-To: <20060622093813.GA26980@code0.codespeak.net> References: <20060618214449.GA15178@code0.codespeak.net> <4495D586.4090907@jippii.fi> <20060621091029.GA659@code0.codespeak.net> <4499D456.2030706@jippii.fi> <20060622093813.GA26980@code0.codespeak.net> Message-ID: <449A7322.2080700@jippii.fi> Armin Rigo wrote: > Hi Elmo, > On Thu, Jun 22, 2006 at 02:20:54AM +0300, Elmo M?ntynen wrote: > >> 2.3 seems to work nicely, 2.4 segfaults (?! I thought it worked again), >> which I'll try to reproduce with different machine (I'll report soon): >> [cbuild:execute] cc -O2 -pthread -I/usr/include/python2.4 -c ctypesplatcheck_0.c -o ctypesplatcheck_0.o [cbuild:execute] cc -pthread /tmp/usession-20/ctypesplatcheck_0.o -lm -lpthread -o /tmp/usession-20/ctypesplatcheck_0 [cbuild:execute] cc -O2 -pthread -c ctypesplatcheck_1.c -o ctypesplatcheck_1.o [cbuild:execute] cc -pthread /tmp/usession-20/ctypesplatcheck_1.o -lm -lpthread -o /tmp/usession-20/ctypesplatcheck_1 [translation:info] Annotating&simplifying... [translation:info] with policy: pypy.objspace.cpy.ann_policy.CPyAnnotatorPolicy Segmentation fault > > We'd need some much more precise info about this to be able to do > anything. Which version of CPython is that precisely, which PyPy > revision, which OS? Can you also please paste the whole output before > the segfault? > CPython: Python 2.4.4c0 (#2, Jun 14 2006, 22:35:41) [GCC 4.1.2 20060613 (prerelease) (Debian 4.1.1-4)] on linux2 OS: Debian Etch, linux 2.6.15 Pypy: latest revision to be continued... > If you manage to get a error and a pdb prompt with another Python > version, can you also please paste the whole output here, together with > the exact CPython version number? > When I tried debugging this, I had hunted down the particular place where the segfault happens, and there was nothing, meaning that there wasn't anything particular that seemed like something that CPython wasn't made to handle, just some normal looking code. The gist of all this is that it seemed to me that the segfault is a plain bug in CPython, just a more obscure one that shows up with the wuite massive pypy machinery. From mwh at python.net Sun Jun 25 14:05:44 2006 From: mwh at python.net (Michael Hudson) Date: Sun, 25 Jun 2006 13:05:44 +0100 Subject: [pypy-dev] pypy-0.9.0: stackless, new extension compiler Message-ID: <2m4py9jv8n.fsf@starship.python.net> The PyPy development team has been busy working and we've now packaged our latest improvements, completed work and new experiments as version 0.9.0, our fourth public release. The highlights of this fourth release of PyPy are: **implementation of "stackless" features** We now support the larger part of the interface of the original Stackless Python -- see http://www.stackless.com for more. A significant part of this is the pickling and unpickling of a running tasklet. These features, especially the pickling, can be considered to be a "technology preview" -- they work, but for example the error handling is a little patchy in places. **ext-compiler** The "extension compiler" is a new way of writing a C extension for CPython and PyPy at the same time. For more information, see its documentation: http://codespeak.net/pypy/dist/pypy/doc/extcompiler.html **rctypes** Most useful in combination with the ext-compiler is the fact that our translation framework can translate code that uses the standard-in-Python-2.5 ctypes module. See its documentation for more: http://codespeak.net/pypy/dist/pypy/doc/rctypes.html **framework GCs** PyPy's interpreter can now be compiled to use a garbage collector written in RPython. This added control over PyPy's execution makes the implementation of new and interesting features possible, apart from being a significant achievement in its own right. **__del__/weakref/__subclasses__** The PyPy interpreter's compatibility with CPython continues improves: now we support __del__ methods, the __subclasses__ method on types and weak references. We now pass around 95% of CPython's core tests. **logic space preview** This release contains the first version of the logic object space, which will add logical variables to Python. See its docs for more: http://codespeak.net/pypy/dist/pypy/doc/howto-logicobjspace-0.9.html **high level backends preview** This release contains the first versions of new backends targeting high level languages such as Squeak and .NET/CLI and updated versions of the JavaScript and Common Lisp backends. They can't compile the PyPy interpreter yet, but they're getting there... **bugfixes, better performance** As you would expect, performance continues to improve and bugs continue to be fixed. The performance of the translated PyPy interpreter is 2.5-3x times faster than 0.8 (on richards and pystone), and is now stable enough to be able to run CPython's test suite to the end. **testing refinements** py.test, our testing tool, now has preliminary support for doctests. We now run all our tests every night, and you can see the summary at: http://snake.cs.uni-duesseldorf.de/pypytest/summary.html What is PyPy (about)? ------------------------------------------------ PyPy is a MIT-licensed research-oriented reimplementation of Python written in Python itself, flexible and easy to experiment with. It translates itself to lower level languages. Our goals are to target a large variety of platforms, small and large, by providing a compilation toolsuite that can produce custom Python versions. Platform, memory and threading models are to become aspects of the translation process - as opposed to encoding low level details into the language implementation itself. Eventually, dynamic optimization techniques - implemented as another translation aspect - should become robust against language changes. Note that PyPy is mainly a research and development project and does not by itself focus on getting a production-ready Python implementation although we do hope and expect it to become a viable contender in that area sometime next year. PyPy is partially funded as a research project under the European Union's IST programme. Where to start? ----------------------------- Getting started: http://codespeak.net/pypy/dist/pypy/doc/getting-started.html PyPy Documentation: http://codespeak.net/pypy/dist/pypy/doc/ PyPy Homepage: http://codespeak.net/pypy/ The interpreter and object model implementations shipped with the 0.9 version can run on their own and implement the core language features of Python as of CPython 2.4. However, we still do not recommend using PyPy for anything else than for education, playing or research purposes. Ongoing work and near term goals --------------------------------- The Just-in-Time compiler and other performance improvements will be one of the main topics of the next few months' work, along with finishing the logic object space. Project Details --------------- PyPy has been developed during approximately 20 coding sprints across Europe and the US. It continues to be a very dynamically and incrementally evolving project with many of these one-week workshops to follow. PyPy has been a community effort from the start and it would not have got that far without the coding and feedback support from numerous people. Please feel free to give feedback and raise questions. contact points: http://codespeak.net/pypy/dist/pypy/doc/contact.html have fun, the pypy team, (Armin Rigo, Samuele Pedroni, Holger Krekel, Christian Tismer, Carl Friedrich Bolz, Michael Hudson, and many others: http://codespeak.net/pypy/dist/pypy/doc/contributor.html) PyPy development and activities happen as an open source project and with the support of a consortium partially funded by a two year European Union IST research grant. The full partners of that consortium are: Heinrich-Heine University (Germany), AB Strakt (Sweden) merlinux GmbH (Germany), tismerysoft GmbH (Germany) Logilab Paris (France), DFKI GmbH (Germany) ChangeMaker (Sweden), Impara (Germany) -- Hiro dicks about with his computer, naturally. Being stranded on a life raft in the Pacific is a perfect venue for a hacker. -- Snow Crash, Neal Stephenson From ms at cerenity.org Sun Jun 25 14:16:18 2006 From: ms at cerenity.org (Michael) Date: Sun, 25 Jun 2006 13:16:18 +0100 Subject: [pypy-dev] pypy-0.9.0: stackless, new extension compiler In-Reply-To: <2m4py9jv8n.fsf@starship.python.net> References: <2m4py9jv8n.fsf@starship.python.net> Message-ID: <200606251316.18134.ms@cerenity.org> Woooooo! Michael From Ben.Young at risk.sungard.com Mon Jun 26 11:34:08 2006 From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com) Date: Mon, 26 Jun 2006 10:34:08 +0100 Subject: [pypy-dev] [Python-Dev] pypy-0.9.0: stackless, new extension compiler In-Reply-To: <2mslltigm2.fsf@starship.python.net> Message-ID: Congratulations! python-dev-bounces+python=theyoungfamily.co.uk at python.org wrote on 25/06/2006 13:07:01: > The PyPy development team has been busy working and we've now packaged > our latest improvements, completed work and new experiments as > version 0.9.0, our fourth public release. > > The highlights of this fourth release of PyPy are: > > **implementation of "stackless" features** > We now support the larger part of the interface of the original > Stackless Python -- see http://www.stackless.com for more. A > significant part of this is the pickling and unpickling of a running > tasklet. > > These features, especially the pickling, can be considered to be a > "technology preview" -- they work, but for example the error handling > is a little patchy in places. > > **ext-compiler** > The "extension compiler" is a new way of writing a C extension for > CPython and PyPy at the same time. For more information, see its > documentation: http://codespeak.net/pypy/dist/pypy/doc/extcompiler.html > > **rctypes** > Most useful in combination with the ext-compiler is the fact that our > translation framework can translate code that uses the > standard-in-Python-2.5 ctypes module. See its documentation for more: > http://codespeak.net/pypy/dist/pypy/doc/rctypes.html > > **framework GCs** > PyPy's interpreter can now be compiled to use a garbage collector > written in RPython. This added control over PyPy's execution makes the > implementation of new and interesting features possible, apart from > being a significant achievement in its own right. > > **__del__/weakref/__subclasses__** > The PyPy interpreter's compatibility with CPython continues improves: > now we support __del__ methods, the __subclasses__ method on types and > weak references. We now pass around 95% of CPython's core tests. > > **logic space preview** > This release contains the first version of the logic object space, > which will add logical variables to Python. See its docs for more: > http://codespeak.net/pypy/dist/pypy/doc/howto-logicobjspace-0.9.html > > **high level backends preview** > This release contains the first versions of new backends targeting high > level languages such as Squeak and .NET/CLI and updated versions of the > JavaScript and Common Lisp backends. They can't compile the PyPy > interpreter yet, but they're getting there... > > **bugfixes, better performance** > As you would expect, performance continues to improve and bugs continue > to be fixed. The performance of the translated PyPy interpreter is > 2.5-3x times faster than 0.8 (on richards and pystone), and is now > stable enough to be able to run CPython's test suite to the end. > > **testing refinements** > py.test, our testing tool, now has preliminary support for doctests. > We now run all our tests every night, and you can see the summary at: > http://snake.cs.uni-duesseldorf.de/pypytest/summary.html > > What is PyPy (about)? > ------------------------------------------------ > > PyPy is a MIT-licensed research-oriented reimplementation of Python > written in Python itself, flexible and easy to experiment with. It > translates itself to lower level languages. Our goals are to target a > large variety of platforms, small and large, by providing a > compilation toolsuite that can produce custom Python versions. > Platform, memory and threading models are to become aspects of the > translation process - as opposed to encoding low level details into > the language implementation itself. Eventually, dynamic optimization > techniques - implemented as another translation aspect - should become > robust against language changes. > > Note that PyPy is mainly a research and development project and does > not by itself focus on getting a production-ready Python > implementation although we do hope and expect it to become a viable > contender in that area sometime next year. > > PyPy is partially funded as a research project under the European > Union's IST programme. > > Where to start? > ----------------------------- > > Getting started: http://codespeak.net/pypy/dist/pypy/doc/getting- > started.html > > PyPy Documentation: http://codespeak.net/pypy/dist/pypy/doc/ > > PyPy Homepage: http://codespeak.net/pypy/ > > The interpreter and object model implementations shipped with the 0.9 > version can run on their own and implement the core language features > of Python as of CPython 2.4. However, we still do not recommend using > PyPy for anything else than for education, playing or research > purposes. > > Ongoing work and near term goals > --------------------------------- > > The Just-in-Time compiler and other performance improvements will be one of > the main topics of the next few months' work, along with finishing the > logic object space. > > Project Details > --------------- > > PyPy has been developed during approximately 20 coding sprints across > Europe and the US. It continues to be a very dynamically and > incrementally evolving project with many of these one-week workshops > to follow. > > PyPy has been a community effort from the start and it would > not have got that far without the coding and feedback support > from numerous people. Please feel free to give feedback and > raise questions. > > contact points: http://codespeak.net/pypy/dist/pypy/doc/contact.html > > have fun, > > the pypy team, (Armin Rigo, Samuele Pedroni, > Holger Krekel, Christian Tismer, > Carl Friedrich Bolz, Michael Hudson, > and many others: http://codespeak.net/pypy/dist/pypy/doc/contributor.html) > > PyPy development and activities happen as an open source project > and with the support of a consortium partially funded by a two > year European Union IST research grant. The full partners of that > consortium are: > > Heinrich-Heine University (Germany), AB Strakt (Sweden) > merlinux GmbH (Germany), tismerysoft GmbH (Germany) > Logilab Paris (France), DFKI GmbH (Germany) > ChangeMaker (Sweden), Impara (Germany) > > -- > And not only in the sense that they imagine heretics where these > do not exist, but also that inquistors repress the heretical > putrefaction so vehemently that many are driven to share in it, > in their hatred of the judges. -- The Name Of The Rose, Umberto Eco > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python- > dev/python%40theyoungfamily.co.uk > From arigo at tunes.org Tue Jun 27 10:34:41 2006 From: arigo at tunes.org (Armin Rigo) Date: Tue, 27 Jun 2006 10:34:41 +0200 Subject: [pypy-dev] pypy-0.9.0: stackless, new extension compiler Message-ID: <20060627083441.GA485@code0.codespeak.net> Hi all, (message by cfbolz which I'm forwarding here) _______________________________________________ Unfortunately the download links for the release tarballs did not work until very recently. They are now working though. You can download the 0.9 release of PyPy under: http://codespeak.net/download/pypy/pypy-0.9.0.tar.bz2 http://codespeak.net/download/pypy/pypy-0.9.0.tar.gz http://codespeak.net/download/pypy/pypy-0.9.0.zip For detailed notes about how to get started into the world of PyPy see here: http://codespeak.net/pypy/dist/pypy/doc/getting-started.html Sorry for the fuss and cheers, Carl Friedrich Bolz _______________________________________________ From cfbolz at gmx.de Tue Jun 27 10:50:54 2006 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Tue, 27 Jun 2006 10:50:54 +0200 Subject: [pypy-dev] pypy-0.9.0: stackless, new extension compiler In-Reply-To: <20060627083441.GA485@code0.codespeak.net> References: <20060627083441.GA485@code0.codespeak.net> Message-ID: <348899050606270150s24e15821vb8c88e69b1a7556f@mail.gmail.com> Hi Armin! 2006/6/27, Armin Rigo : > Hi all, (message by cfbolz which I'm forwarding here) wuaa, thanks. I sent the mail only to Michael :-) Cheers, Carl Friedrich From wanja at merlinux.de Thu Jun 29 11:58:00 2006 From: wanja at merlinux.de (Wanja Saatkamp) Date: Thu, 29 Jun 2006 11:58:00 +0200 Subject: [pypy-dev] Release of PyPy videos Message-ID: <44A3A428.60408@merlinux.de> The PyPy team is happy to announce that the first bunch of PyPy videos can now be downloaded from: http://codespeak.net/pypy/dist/pypy/doc/video-index.html The videos introduce involved people and contain different talks, tutorials and interviews such as: 1. Technical talk on the PyPy architecture, standard interpreter, translation toolchain and Just-in-time compiler at the University of Palma de Mallorca (72 min) 2. Coding discussion between Armin Rigo and Samuele Pedroni during the PyPy sprint at the University of Palma de Mallorca (40 min) 3. Core developer Holger Krekel and project manager Beatrice During explain the agile open source methods used in the PyPy project at Pycon, Dallas (26 min) 4. Core developers Michael Hudson and Christian Tismer are giving an introductory talk at Pycon, Dallas (28 min) 5. A PyPy architecture session given by core developers Holger Krekel and Armin Rigo at Pycon, Dallas (48 min) 6. A Sprint tutorial by core developer Michael Hudson who provides a detailed and hands-on overview about the architecture of PyPy, especially the translation toolchain (44 min) 7. What do you think about PyPy? Interview with American software developer Bob Ippolito at this year's PyCon, Dallas (8 min) 8. Interview with CPython core developer Tim Peters at this year's PyCon, Dallas (23 min) Some technical details: All films are in European PAL system, encoded with DivX and packed in .avi file format. The files can be downloaded using a bittorrent client. Further details regarding downloading and viewing can be found at the URL above. Have fun and don't hesitate to contact us at pypy-dev at codespeak.net if you encounter any problems or have any other comments, Best regards, Wanja Saatkamp & the PyPy team From pedronis at strakt.com Fri Jun 30 19:44:34 2006 From: pedronis at strakt.com (Samuele Pedroni) Date: Fri, 30 Jun 2006 19:44:34 +0200 Subject: [pypy-dev] [pypy-svn] r29534 - pypy/dist/pypy/rpython/ootypesystem/test In-Reply-To: <20060630140821.989F310081@code0.codespeak.net> References: <20060630140821.989F310081@code0.codespeak.net> Message-ID: <44A56302.2090808@strakt.com> antocuni at codespeak.net wrote: > Author: antocuni > Date: Fri Jun 30 16:08:16 2006 > New Revision: 29534 > > Modified: > pypy/dist/pypy/rpython/ootypesystem/test/test_oorecord.py > Log: > Added a failing test. > > Basically, if we modify the Record after its hash has already been > computed, we might obtain two objects that compare equal but hash > differently. > > Disabling the caching in LowLevelType.__hash__ fix the test, but I > think it's not what we want. > > Adding the following __hash__ to ootype.Record also fix the test, but > I'm not sure if it's correct, especially with recursive structures: > > def __hash__(self): > return hash(self._fields) > Records are really like lltypesystem Structs, although because some of their code was copied from Instance, they may give the impression that you can add fields after the fact, but that should not be done, it breaks the fact that they are supposed to compare by structure. If you are needing that you need some other approach or introduce some kind of forward definition. _add_fields should really disappear from Record. > > > > Modified: pypy/dist/pypy/rpython/ootypesystem/test/test_oorecord.py > ============================================================================== > --- pypy/dist/pypy/rpython/ootypesystem/test/test_oorecord.py (original) > +++ pypy/dist/pypy/rpython/ootypesystem/test/test_oorecord.py Fri Jun 30 16:08:16 2006 > @@ -38,3 +38,14 @@ > t.a = 1 > t.b = 2 > assert ooidentityhash(t) != ooidentityhash(t2) > + > +import py > +def test_hash(): > + #py.test.skip("LowLevelType.__hash__ bug waiting to be fixed") > + T1 = Record({"item0": Signed, "item1": Signed}) > + T2 = Record({"item0": Signed}) > + > + hash(T2) # compute the hash, it will stored in __cached_hash > + T2._add_fields({"item1": Signed}) # modify the object > + assert T1 == T2 > + assert hash(T1) == hash(T2) > _______________________________________________ > pypy-svn mailing list > pypy-svn at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-svn From anto.cuni at gmail.com Fri Jun 30 20:28:22 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Fri, 30 Jun 2006 20:28:22 +0200 Subject: [pypy-dev] [pypy-svn] r29534 - pypy/dist/pypy/rpython/ootypesystem/test In-Reply-To: <44A56302.2090808@strakt.com> References: <20060630140821.989F310081@code0.codespeak.net> <44A56302.2090808@strakt.com> Message-ID: <44A56D46.7040605@gmail.com> Samuele Pedroni wrote: > Records are really like lltypesystem Structs, although because some of > their code was copied from Instance, they may give the impression > that you can add fields after the fact, but that should not be done, > it breaks the fact that they are supposed to compare by structure. > > If you are needing that you need some other approach or introduce > some kind of forward definition. > > _add_fields should really disappear from Record. ok, so it's my fault, but it don't solve my original problem. The problem is that TestCliTuple.test_inst_tuple_add_getitem in test_rpython.py used to fail because the IL code contained two copies of the same Record. After a bit of investigation I discovered that the reason was because I got two Records that compare equal but have different hashes; again, the problem was that the __cached_hash was different from the real one. Then I tried to reproduce the bug, and so I wrote the failing test, thinking that using _add_fields was fine. But after your comment I've understood that this is not the point, because Record._add_fields is called only in its __init__. For now the problem is worked-around in database.py (the lines marked with "XXX: temporary hack"), but the bug is still here: any idea of why I get two equal Records with different hashes? ciao Anto From anto.cuni at gmail.com Fri Jun 30 20:54:18 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Fri, 30 Jun 2006 20:54:18 +0200 Subject: [pypy-dev] [pypy-svn] r29534 - pypy/dist/pypy/rpython/ootypesystem/test In-Reply-To: <44A56302.2090808@strakt.com> References: <20060630140821.989F310081@code0.codespeak.net> <44A56302.2090808@strakt.com> Message-ID: <44A5735A.6090908@gmail.com> Ok, I've found the bug, it's entirely a my fault, because in translator/cli/record.py I attach a new attribute _name to the Record, so the hash is no longer valid. Probably it's safer to store the name in some dictionary. ciao Anto