From nhaldimann at gmx.ch Sat Apr 1 23:10:28 2006 From: nhaldimann at gmx.ch (Niklaus Haldimann) Date: Sat Apr 1 23:10:57 2006 Subject: [pypy-dev] [Fwd: Dynamic Languages Symposium 2006 - Technical Papers (CfP)] Message-ID: <442EEC44.5030505@gmx.ch> Hi there If I remember correctly some people were under the impression that the deadline for the Dynamic Languages Symposium at OOPSLA has already passed. Obviously this isn't so (it's June 1st, see below). If PyPy is going to submit a technical paper to just one conference this year, it should probably be this one. ;) Considering who's on the program committee it's also likely that it will actually be accepted. Cheers Nik -------- Original Message -------- Subject: Dynamic Languages Symposium 2006 - Technical Papers (CfP) Date: Fri, 31 Mar 2006 16:20:47 +0200 From: Robert Hirschfeld Reply-To: hirschfeld@acm.org, The general-purpose Squeak developers list To: squeak-dev@lists.squeakfoundation.org, croquet@lists.wisc.edu Dynamic Languages Symposium 2006 - Technical Papers Call for papers Portland, Oregon, United States, October 22, 2006 http://www.dcl.hpi.uni-potsdam.de/dls2006/ The Dynamic Languages Symposium (DLS) at OOPSLA 2006 is a forum for discussion of dynamic languages, their implementation and application. While mature dynamic languages including Smalltalk, Lisp, Scheme, and Prolog continue to grow and inspire new converts, a new generation of dynamic scripting languages such as Python, Ruby, PHP, and JavaScript are successful in a wide range of applications. DLS provides a place for researchers and practitioners to come together and share their knowledge, experience, and ideas for future research and development. The Technical Papers track of DLS 2006 invites high quality papers reporting original research, innovative contributions or experience related to dynamic languages, their implementation and application. Accepted Papers will be published in the OOPSLA conference companion and the ACM Digital Library. Areas of interest include but are not limited to * Reflection and meta-programming * Very late binding, dynamic composition, and runtime adaptation * Actors and active objects * Innovative language features and implementation techniques * Development and platform support, tools * Language symbiosis and multi-paradigm languages * Experience reports and case studies * Interesting applications * Educational approaches and perspectives * Domain-oriented programming * Object-oriented, aspect-oriented, and context-oriented programming Submissions and proceedings We invite original contributions that neither have been published previously nor are under review by other refereed events or publications. Research papers should describe work that advances the current state of the art. Experience papers should be of broad interest and should describe insights gained from substantive practical applications. The program committee will evaluate each contributed paper based on its relevance, significance, clarity, and originality. Papers are to be submitted electronically at http://www.dcl.hpi.uni-potsdam.de/node2006/ in PDF format. Submissions need to use the ACM format, templates for which can be found at http://www.acm.org/sigs/pubs/proceed/template.html. Important dates Submission of papers (hard deadline): June 1, 2006 (Thursday) Author notification: July 1, 2006 (Saturday) Final version due: July 11, 2006 (Tuesday) DLS Technical Papers Day: October 22, 2006 (Sunday) DLS Invited Talks Day: October 23, 2006 (Monday) Chair Robert Hirschfeld Hasso-Plattner-Institut, University of Potsdam, Germany hirschfeld@acm.org Program committee David Asher, ActiveState, United States Gilad Bracha, Sun Microsystems, United States Pascal Costanza, Vrije Universiteit Brussel, Belgium Richard P. Gabriel, Sun Microsystems Laboratories, United States Robert Hirschfeld, HPI, University of Potsdam, Germany (chair) David Leibs, Advanced Micro Devices, United States Wolfgang De Meuter, Vrije Universiteit Brussel, Belgium Stephane Ducasse, Universite de Savoie, France Oscar Nierstrasz, University of Berne, Switzerland Ian Piumarta, Viewpoints Research Institute, United States David Simmons, Microsoft, United States Michael Sperber, University of Tuebingen, Germany Dave Thomas, Bedarra Research Labs, Canada Martin von Loewis, HPI, University of Potsdam, Germany Jon L White, United States Allen Wirfs-Brock, Microsoft, United States Roel Wuyts, Unversite Libre de Bruxelles, Belgium From lac at strakt.com Sun Apr 2 00:40:01 2006 From: lac at strakt.com (Laura Creighton) Date: Sun Apr 2 00:51:16 2006 Subject: [pypy-dev] [Fwd: Dynamic Languages Symposium 2006 - Technical Papers (CfP)] In-Reply-To: Message from Niklaus Haldimann of "Sat, 01 Apr 2006 23:10:28 +0200." <442EEC44.5030505@gmx.ch> References: <442EEC44.5030505@gmx.ch> Message-ID: <200604012240.k31Me1nk009980@theraft.strakt.com> Thank you Nik for this important information, we did indeed believe that we missed the deadline. David Ascher is Canadian, as is ActiveState. I have cc'd this to him so that he can go about the business of correcting the pure slander that they are American, because Nik has clearly taken this from some other site. Laura Creighton In a message of Sat, 01 Apr 2006 23:10:28 +0200, Niklaus Haldimann writes: >Hi there > >If I remember correctly some people were under the impression that the >deadline for the Dynamic Languages Symposium at OOPSLA has already >passed. Obviously this isn't so (it's June 1st, see below). If PyPy is >going to submit a technical paper to just one conference this year, it >should probably be this one. ;) Considering who's on the program >committee it's also likely that it will actually be accepted. > >Cheers >Nik > >-------- Original Message -------- >Subject: Dynamic Languages Symposium 2006 - Technical Papers (CfP) >Date: Fri, 31 Mar 2006 16:20:47 +0200 >From: Robert Hirschfeld >Reply-To: hirschfeld@acm.org, The general-purpose Squeak developers list > >To: squeak-dev@lists.squeakfoundation.org, croquet@lists.wisc.edu > >Dynamic Languages Symposium 2006 - Technical Papers > >Call for papers > >Portland, Oregon, United States, October 22, 2006 > > >http://www.dcl.hpi.uni-potsdam.de/dls2006/ > > >The Dynamic Languages Symposium (DLS) at OOPSLA 2006 is a forum for >discussion of dynamic languages, their implementation and application. >While mature dynamic languages including Smalltalk, Lisp, Scheme, and >Prolog continue to grow and inspire new converts, a new generation of >dynamic scripting languages such as Python, Ruby, PHP, and JavaScript >are successful in a wide range of applications. DLS provides a place for >researchers and practitioners to come together and share their >knowledge, experience, and ideas for future research and development. > >The Technical Papers track of DLS 2006 invites high quality papers >reporting original research, innovative contributions or experience >related to dynamic languages, their implementation and application. >Accepted Papers will be published in the OOPSLA conference companion and >the ACM Digital Library. > > >Areas of interest include but are not limited to > >* Reflection and meta-programming >* Very late binding, dynamic composition, and runtime adaptation >* Actors and active objects >* Innovative language features and implementation techniques >* Development and platform support, tools >* Language symbiosis and multi-paradigm languages >* Experience reports and case studies >* Interesting applications >* Educational approaches and perspectives >* Domain-oriented programming >* Object-oriented, aspect-oriented, and context-oriented programming > > >Submissions and proceedings > >We invite original contributions that neither have been published >previously nor are under review by other refereed events or >publications. Research papers should describe work that advances the >current state of the art. Experience papers should be of broad interest >and should describe insights gained from substantive practical >applications. The program committee will evaluate each contributed paper >based on its relevance, significance, clarity, and originality. > >Papers are to be submitted electronically at >http://www.dcl.hpi.uni-potsdam.de/node2006/ in PDF format. Submissions >need to use the ACM format, templates for which can be found at >http://www.acm.org/sigs/pubs/proceed/template.html. > > >Important dates > >Submission of papers (hard deadline): June 1, 2006 (Thursday) >Author notification: July 1, 2006 (Saturday) >Final version due: July 11, 2006 (Tuesday) >DLS Technical Papers Day: October 22, 2006 (Sunday) >DLS Invited Talks Day: October 23, 2006 (Monday) > > >Chair > >Robert Hirschfeld >Hasso-Plattner-Institut, University of Potsdam, Germany >hirschfeld@acm.org > > >Program committee > >David Asher, ActiveState, United States >Gilad Bracha, Sun Microsystems, United States >Pascal Costanza, Vrije Universiteit Brussel, Belgium >Richard P. Gabriel, Sun Microsystems Laboratories, United States >Robert Hirschfeld, HPI, University of Potsdam, Germany (chair) >David Leibs, Advanced Micro Devices, United States >Wolfgang De Meuter, Vrije Universiteit Brussel, Belgium >Stephane Ducasse, Universite de Savoie, France >Oscar Nierstrasz, University of Berne, Switzerland >Ian Piumarta, Viewpoints Research Institute, United States >David Simmons, Microsoft, United States >Michael Sperber, University of Tuebingen, Germany >Dave Thomas, Bedarra Research Labs, Canada >Martin von Loewis, HPI, University of Potsdam, Germany >Jon L White, United States >Allen Wirfs-Brock, Microsoft, United States >Roel Wuyts, Unversite Libre de Bruxelles, Belgium > > >_______________________________________________ >pypy-dev@codespeak.net >http://codespeak.net/mailman/listinfo/pypy-dev From DavidA at ActiveState.com Sun Apr 2 00:53:02 2006 From: DavidA at ActiveState.com (David Ascher) Date: Sun Apr 2 00:53:36 2006 Subject: [pypy-dev] [Fwd: Dynamic Languages Symposium 2006 - Technical Papers (CfP)] In-Reply-To: <200604012240.k31Me1nk009980@theraft.strakt.com> References: <442EEC44.5030505@gmx.ch> <200604012240.k31Me1nk009980@theraft.strakt.com> Message-ID: <442F044E.8050001@ActiveState.com> Laura Creighton wrote: >Thank you Nik for this important information, we did indeed >believe that we missed the deadline. > >David Ascher is Canadian, as is ActiveState. I have cc'd this to him >so that he can go about the business of correcting the pure >slander that they are American, because Nik has clearly taken >this from some other site. > Thanks, I'd already gotten the CFP fixed, but after it started spreading. For the record, I'm not Canadian, although I am in Canada. I'm french and american =). From lac at strakt.com Sun Apr 2 01:07:34 2006 From: lac at strakt.com (Laura Creighton) Date: Sun Apr 2 01:18:19 2006 Subject: [pypy-dev] [Fwd: Dynamic Languages Symposium 2006 - Technical Papers (CfP)] In-Reply-To: Message from David Ascher of "Sat, 01 Apr 2006 14:53:02 -0800." <442F044E.8050001@ActiveState.com> References: <442EEC44.5030505@gmx.ch> <200604012240.k31Me1nk009980@theraft.strakt.com> <442F044E.8050001@ActiveState.com> Message-ID: <200604012307.k31N7YSP011668@theraft.strakt.com> In a message of Sat, 01 Apr 2006 14:53:02 -0800, David Ascher writes: >Laura Creighton wrote: > >>Thank you Nik for this important information, we did indeed >>believe that we missed the deadline. >> >>David Ascher is Canadian, as is ActiveState. I have cc'd this to him >>so that he can go about the business of correcting the pure >>slander that they are American, because Nik has clearly taken >>this from some other site. >> >Thanks, I'd already gotten the CFP fixed, but after it started spreading. > >For the record, I'm not Canadian, although I am in Canada. I'm french >and american =). Interesting mix, apologies for getting it wrong. Laura Creighton - in Sweden, but from NOVA SCOTIA or ACADIE for those of us for whom such things matter :-) (we don't feel particularly 'Canadian' and leave that for the people of Ontario, but we sure know where home is. Since Nik is from Switzerland, he will understand this sort of feeling perfectly well. :-) ) Laura From jp at hapra.at Sun Apr 2 20:54:59 2006 From: jp at hapra.at (Jakob Praher) Date: Sun Apr 2 21:06:53 2006 Subject: [pypy-dev] inquiry: phd positions Message-ID: hi all, my name is Jakob Praher. I am located in Linz, Austria. Currently I am finishing my master thesis, which is about developing an aspect oriented dynamic bytecode instrumentation framework for program analysis on top of LLVM. I am very interested to work in that area as a PHd student. From your website I know you are also working on an LLVM backend. PyPy seems to be a European effort, so my naive question is, whether you have any PhD projects in this area or could point me to some further information. I would be very interested to hearing from you. Thank you Jakob From mwh at python.net Mon Apr 3 01:59:29 2006 From: mwh at python.net (Michael Hudson) Date: Mon Apr 3 02:00:18 2006 Subject: [pypy-dev] Re: inquiry: phd positions References: Message-ID: <2mslovttpa.fsf@starship.python.net> Jakob Praher writes: > hi all, > > my name is Jakob Praher. I am located in Linz, Austria. Currently I am > finishing my master thesis, which is about developing an aspect > oriented dynamic bytecode instrumentation framework for program > analysis on top of LLVM. I am very interested to work in that area as > a PHd student. From your website I know you are also working on an > LLVM backend. I'm not _entirely_ sure what you are saying here. We have interests in AOP and interests in LLVM but they are not particularly related, whereas your interests seem to be intertwined. But it's late and I am suffering from a stressful trip, so maybe the confusion is just at my end :) > PyPy seems to be a European effort, Only by accident, mostly... > so my naive question is, whether you have any PhD projects in this > area or could point me to some further information. I would be very > interested to hearing from you. Well, we certainly hope that there are interesting lines of investigation in PyPy. To be more specific, we probably need you to be more specific about your interests too. If you are asking "do we have pots of cash lying around to fund PhDs", then I'm pretty sure the answer is "no". PyPy's funding from the EU was not structured in this sort of way (AIUI). Cheers, mwh -- (FREE|OPEN) BSD: Shire horse. Solid, reliable, only occasionally prone to crushing you against a wall and then only because you've told it to without knowing. -- Jim's pedigree of operating systems, asr From cfbolz at gmx.de Mon Apr 3 02:34:32 2006 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon Apr 3 02:35:06 2006 Subject: [pypy-dev] Re: inquiry: phd positions In-Reply-To: <2mslovttpa.fsf@starship.python.net> References: <2mslovttpa.fsf@starship.python.net> Message-ID: <44306D98.9090708@gmx.de> Hi Jakob! Jakob Praher writes: > my name is Jakob Praher. I am located in Linz, Austria. Currently I am > finishing my master thesis, which is about developing an aspect > oriented dynamic bytecode instrumentation framework for program > analysis on top of LLVM. I am very interested to work in that area as > a PHd student. From your website I know you are also working on an > LLVM backend. Well, yes. But an LLVM backend in PyPy lingo is different from an LLVM backend in LLVM lingo :-). The latter would be a backend to LLVM that targets a certain CPU architecture. But it is really the other way round: we have a backend that produces _LLVM_ code out of our own IL. No clue about the Ph.D., sorry. Cheers, Carl Friedrich From jp at hapra.at Mon Apr 3 09:55:46 2006 From: jp at hapra.at (Jakob Praher) Date: Mon Apr 3 09:56:39 2006 Subject: [pypy-dev] Re: inquiry: phd positions In-Reply-To: <2mslovttpa.fsf@starship.python.net> References: <2mslovttpa.fsf@starship.python.net> Message-ID: hi Michael, Michael Hudson schrieb: > Jakob Praher writes: > >> hi all, >> >> my name is Jakob Praher. I am located in Linz, Austria. Currently I am >> finishing my master thesis, which is about developing an aspect >> oriented dynamic bytecode instrumentation framework for program >> analysis on top of LLVM. I am very interested to work in that area as >> a PHd student. From your website I know you are also working on an >> LLVM backend. > > I'm not _entirely_ sure what you are saying here. We have interests > in AOP and interests in LLVM but they are not particularly related, > whereas your interests seem to be intertwined. I only wanted to give you some information, what I am doing right now. I am also interested in dynamic runtime systems. I have some experience with self and prototype based oo systems. I also had some experiences with Jalpanero (now JikesRVM), which is also selfhosting like PyPy. What I mean is that I am flexible. For me it is important to do open source research and that I could gain some experience for implementing large runtime systems. > > But it's late and I am suffering from a stressful trip, so maybe the > confusion is just at my end :) > >> PyPy seems to be a European effort, > > Only by accident, mostly... That is interesting :-) > >> so my naive question is, whether you have any PhD projects in this >> area or could point me to some further information. I would be very >> interested to hearing from you. > > Well, we certainly hope that there are interesting lines of > investigation in PyPy. To be more specific, we probably need you to > be more specific about your interests too. See above. My biggest concern is whether there could be some real colaboration with a university. It is quite hard to do this kind of research in Linz. Mostly because of a lack of support from the university side. > > If you are asking "do we have pots of cash lying around to fund PhDs", > then I'm pretty sure the answer is "no". PyPy's funding from the EU > was not structured in this sort of way (AIUI). No I just wanted to say that it would be very interesting to work in an area like dynamic object oriented runtimes. But there are many things why I am asking this out loud: * Is there any university involved in this project? * What area of PyPy needs some research in the size of a PhD project? Granted it is somewhat ignorant not to come up with an answer to the second question before writing to this list. But actually I wanted to hear some opinions from you. Anyways Linz is not the best place for hard core open source research. Before working on anything myself again, you have to understand that I have to do some questioning. -- Jakob From jp at hapra.at Mon Apr 3 10:20:35 2006 From: jp at hapra.at (Jakob Praher) Date: Mon Apr 3 10:22:11 2006 Subject: [pypy-dev] Re: inquiry: phd positions In-Reply-To: <44306D98.9090708@gmx.de> References: <2mslovttpa.fsf@starship.python.net> <44306D98.9090708@gmx.de> Message-ID: Carl Friedrich Bolz schrieb: > Hi Jakob! > > Jakob Praher writes: >> my name is Jakob Praher. I am located in Linz, Austria. Currently I am >> finishing my master thesis, which is about developing an aspect >> oriented dynamic bytecode instrumentation framework for program >> analysis on top of LLVM. I am very interested to work in that area as >> a PHd student. From your website I know you are also working on an >> LLVM backend. > > Well, yes. But an LLVM backend in PyPy lingo is different from an LLVM > backend in LLVM lingo :-). The latter would be a backend to LLVM that > targets a certain CPU architecture. But it is really the other way > round: we have a backend that produces _LLVM_ code out of our own IL. Yes :-) I just wanted to point out that I have some knowledge of the LLVM infrastructure and that I was using the JIT for my own project quite substantially. While there are many areas where still need a lot of work to understand, I am very interested in projects like yours and so I just wanted to ask. LLVM lacks symbolic information (hence lowlevel), which makes it impossible to be used as a first class IL for a project like pypy. I will have a look at your IL. > > No clue about the Ph.D., sorry. Never mind. Thanks for your reply. > > Cheers, > > Carl Friedrich > _______________________________________________ > pypy-dev@codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From jacob at strakt.com Mon Apr 3 10:45:16 2006 From: jacob at strakt.com (Jacob Hallen) Date: Mon Apr 3 10:45:49 2006 Subject: [pypy-dev] Re: inquiry: phd positions In-Reply-To: References: <2mslovttpa.fsf@starship.python.net> Message-ID: <200604031045.16156.jacob@strakt.com> Hi Jakob, I think we can safely say that the Pypy project contains several areas that are suitable for PhD level research. Just to mention a few: - Modularisation of the source language, allowing expansion and contraction of the available syntax and semantics. - Adaption of the platform to handle multiple source languages. -Use of the Pypy architecture for distributed computing, multiprocessing and handling of security issues at interpreter level. - Various aspects of code generation including translation, JIT specialisation, garbage collection and backend production. - Pypy integration with existing platforms, like Java and CLI. m?ndag 03 april 2006 09.55 skrev Jakob Praher: > >> so my naive question is, whether you have any PhD projects in this > >> area or could point me to some further information. I would be very > >> interested to hearing from you. > > > > Well, we certainly hope that there are interesting lines of > > investigation in PyPy. To be more specific, we probably need you to > > be more specific about your interests too. > > See above. My biggest concern is whether there could be some real > colaboration with a university. It is quite hard to do this kind of > research in Linz. Mostly because of a lack of support from the > university side. > > > If you are asking "do we have pots of cash lying around to fund PhDs", > > then I'm pretty sure the answer is "no". PyPy's funding from the EU > > was not structured in this sort of way (AIUI). > > No I just wanted to say that it would be very interesting to work in an > area like dynamic object oriented runtimes. But there are many things > why I am asking this out loud: > > * Is there any university involved in this project? > * What area of PyPy needs some research in the size of a PhD project? Heinrich Heine University of Duesseldorf is a member of the project, with professor Leuchel being formally responsible. In practice, it is Armin Rigo and Michael Hudson who are financed to be working on Pypy. As of this week Carl Friedrich Boltz will be doing his Bachelor of Science there, focusing on Pypy. While professor Leuchel is a very open minded person, who seems to enjoy giving people the liberty to pursue the kind of research they are interested in, I have no clue as to what the policy of the CS department or the university are concerning the acceptance of PhD students. I hope what I have said will help you get started on your own investigation. In the meantime, I hope you will have time to get acquainted with the code base. You are also welcome to join one of our sprints, where you will get a chance to collaborate with the developers on specific issues. The most convenient place for you in the reasonably near future will probably be the sprint right after Europython at CERN in Switzerland. We do have an open sprint in Japan before that, but it is a long way to travel and we will probably have as many newcomers as we can cope with. Best of luck Jacob Hall?n From Gerald.Klix at klix.ch Mon Apr 3 14:58:22 2006 From: Gerald.Klix at klix.ch (Gerald Klix) Date: Mon Apr 3 19:41:44 2006 Subject: [pypy-dev] rctypes design document Message-ID: <44311BEE.7000109@klix.ch> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, yesterday I checked in the final version of the rcytpes design document. I am convinced I finally got everything right. Especially the management of keep alive pointers. Have fun, Gerald - -- GPG-Key: http://keyserver.veridis.com:11371/search?q=0xA140D634 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFEMRvuEDg9cqFA1jQRAtOUAJ9hj26mRbeXULn3cxZ9Rhp3bBdOsACfe5P4 ssR79dN5e8brVM0y3JIZDNw= =pLSI -----END PGP SIGNATURE----- From jp at hapra.at Mon Apr 3 20:53:04 2006 From: jp at hapra.at (Jakob Praher) Date: Mon Apr 3 20:53:39 2006 Subject: [pypy-dev] Re: inquiry: phd positions In-Reply-To: <200604031045.16156.jacob@strakt.com> References: <2mslovttpa.fsf@starship.python.net> <200604031045.16156.jacob@strakt.com> Message-ID: <44316F10.80303@hapra.at> Hi Jacob, thank you for your support. Looks very interesting to hear that. What are you currently working on? AFAICT you are employed by strakt.com. Are you working on pypy as a research project? Jacob Hallen schrieb: > - Modularisation of the source language, allowing expansion and contraction of > the available syntax and semantics. > > - Adaption of the platform to handle multiple source languages. > > That is insteresting, especially when supporting other dynamic languages. > -Use of the Pypy architecture for distributed computing, multiprocessing and > handling of security issues at interpreter level. > Ah yes. I recently attended a guest lecture by Michael Franz. He is a professor at University Irvine in California. They are implementing MAC based security labels for virtual machines. He managed to bring the pefromance loss of labeling down to 6 % of the whole execution time. Given the fact that SeLinux for instance already supports MAC, MAC can be an interesting concept to bring to virtual machines as well. > - Various aspects of code generation including translation, JIT > specialisation, garbage collection and backend production. > Interesting. > - Pypy integration with existing platforms, like Java and CLI. > Could be interesting too. Especially Java plans on integrating an invokedynamic instruction. This makes it possible to design more efficient code generators. But I am not really 100 % sure, how this will look like. > > Heinrich Heine University of Duesseldorf is a member of the project, with > professor Leuchel being formally responsible. In practice, it is Armin Rigo > and Michael Hudson who are financed to be working on Pypy. As of this week > Carl Friedrich Boltz will be doing his Bachelor of Science there, focusing on > Pypy. > Thats interesting to hear. > While professor Leuchel is a very open minded person, who seems to enjoy > giving people the liberty to pursue the kind of research they are interested > in, I have no clue as to what the policy of the CS department or the > university are concerning the acceptance of PhD students. I hope what I have > said will help you get started on your own investigation. > Oh great. Thanks for the pointers. The problem I am facing is that PhD slots are pretty scarce and so there are often many people already waiting for one :-) > In the meantime, I hope you will have time to get acquainted with the code > base. You are also welcome to join one of our sprints, where you will get a > chance to collaborate with the developers on specific issues. The most > convenient place for you in the reasonably near future will probably be the > sprint right after Europython at CERN in Switzerland. We do have an open > sprint in Japan before that, but it is a long way to travel and we will > probably have as many newcomers as we can cope with. > > I will surely look at the project code base, no matter how this turns out. Thank you. > Best of luck > > Jacob Hall?n > > From nhaldimann at gmx.ch Mon Apr 3 22:54:47 2006 From: nhaldimann at gmx.ch (Niklaus Haldimann) Date: Mon Apr 3 22:55:36 2006 Subject: [pypy-dev] List and string in ootypesystem In-Reply-To: <442C2FB3.5070800@gmail.com> References: <442BC9CC.6010503@gmail.com> <442BF15B.9070102@gmx.ch> <442C2FB3.5070800@gmail.com> Message-ID: <44318B97.3050408@gmx.ch> Hi Antonio (and others) As you have probably seen, we started work on list support for ootypesystem here at the ongoing Leysin sprint. This cleared things up a bit for me on the conceptual level. Here's how OO lists work: - There is a new type constructor "List" in the ootype module. The low-level type of RPython list in the ootypesystem is a List instance with the item type of the list provided as an argument. E.g., an RPython list of integers has the low-level type List(Signed). - The interface of an OO list is defined by List._METHODS (see List.__init__). This interface is still incomplete, but it should be kept as small as possible, but large enough to support all operations permitted in RPython on lists. - When it comes to low-level operations, Lists are treated similarly to Instances. "new" is used to instantiate an (empty) list. Methods of the interface are rtyped as "oosend" operations with the name of the method. - Backends must special-case "new" with Lists and "oosends" to Lists. Basically, "new" can be mapped to the construction of a native list in the backend. The oosends can be mapped to native methods of the native list. All of this works for some basic list operations at this point (length, append, getitem, setitem), but there's still quite a bit of work to do for full list support in ootypesystem. After all this, the roadmap for bringing all other data structures to ootypesystem is now pretty clear to me: - dicts: Can be implemented analoguous to lists (type constructor, minimal interface etc.) - strings: Analoguous to lists, but no type constructor necessary - ranges, slices: Can be implemented as a special Instance (analoguous to tuples, in a way) I have two more days here at the Leysin sprint to work on lists. After that I again won't have much time to work on PyPy, so Antonio (or anyone else), you're very welcome to continue from here. I think the list code can serve as a good starting point for the other data structures. If there are any questions, I'm happy to answer them (if I can ;)). Cheers Nik From anto.cuni at gmail.com Tue Apr 4 01:15:04 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue Apr 4 01:16:42 2006 Subject: [pypy-dev] List and string in ootypesystem In-Reply-To: <44318B97.3050408@gmx.ch> References: <442BC9CC.6010503@gmail.com> <442BF15B.9070102@gmx.ch> <442C2FB3.5070800@gmail.com> <44318B97.3050408@gmx.ch> Message-ID: <4431AC78.9010705@gmail.com> Hi Niklaus, you have just preceded me by about 2 hours... I would have sent a mail about rlist anyway. I've spent last days working on this topic: I tried to refactor the code for making rlist type-system specific as rtuple, rclass and others already was. It has been a bit difficult because it was my first hacking in the rpython directory: I had to read the sources carefully for understanding in deep how things works, and I'm not sure if I have completely understood the whole logic. I've just committed my work in the http://codespeak.net/svn/user/antocuni/pypy-antocuni directory. I'm not satisfied of it because it is a bit messy and I'm happy to know that now it's no longer needed because yours is surely better :-). By the way I think it has not been a waste of time because it let me to gain some knowledge of pypy's internals that will be useful in future. Niklaus Haldimann wrote: > I have two more days here at the Leysin sprint to work on lists. After > that I again won't have much time to work on PyPy, so Antonio (or anyone > else), you're very welcome to continue from here. I think the list code > can serve as a good starting point for the other data structures. If > there are any questions, I'm happy to answer them (if I can ;)). Sure, I'll be happy to continue from here: it is much better than starting from scratch as I supposed to do! :-) I think you've saved me from a lot of headaches, thanks! Remind me to offer you a beer the first time we meet ;-). ciao Anto From nhaldimann at gmx.ch Tue Apr 4 09:07:30 2006 From: nhaldimann at gmx.ch (Niklaus Haldimann) Date: Tue Apr 4 09:07:58 2006 Subject: [pypy-dev] List and string in ootypesystem In-Reply-To: <4431AC78.9010705@gmail.com> References: <442BC9CC.6010503@gmail.com> <442BF15B.9070102@gmx.ch> <442C2FB3.5070800@gmail.com> <44318B97.3050408@gmx.ch> <4431AC78.9010705@gmail.com> Message-ID: <44321B32.8000504@gmx.ch> Hi Antonio Antonio Cuni wrote: > I've spent last days working on this topic: I tried to refactor the code > for making rlist type-system specific as rtuple, rclass and others > already was. > It has been a bit difficult because it was my first hacking in the > rpython directory: I had to read the sources carefully for understanding > in deep how things works, and I'm not sure if I have completely > understood the whole logic. > > I've just committed my work in the > http://codespeak.net/svn/user/antocuni/pypy-antocuni directory. I'm not > satisfied of it because it is a bit messy and I'm happy to know that now > it's no longer needed because yours is surely better :-). By the way I > think it has not been a waste of time because it let me to gain some > knowledge of pypy's internals that will be useful in future. Oops, I didn't intend to invalidate your work. ;) I actually checked your user directory yesterday, because you said in an earlier mail that you would work on the branch there. But since I didn't see any changes related to rlist I assumed you decided to postpone this ... What you did doesn't look so bad, I just looked at it. In general, I'm impressed that you found your way around the RTyper so easily. ;) The main difference to our work is that you created many new low-level operations for the list interface. Since there will also have to be a dict and string interface this would lead to an explosion of low-level operations. Our approach also makes testing of these data structures at a lower level easier (see test_oolist.py and test_oortype.py). > Sure, I'll be happy to continue from here: it is much better than > starting from scratch as I supposed to do! :-) > I think you've saved me from a lot of headaches, thanks! Remind me to > offer you a beer the first time we meet ;-). Hehe, looking forward to that. ;) Cheers Nik From anto.cuni at gmail.com Tue Apr 4 10:16:32 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue Apr 4 10:20:23 2006 Subject: [pypy-dev] List and string in ootypesystem In-Reply-To: <44321B32.8000504@gmx.ch> References: <442BC9CC.6010503@gmail.com> <442BF15B.9070102@gmx.ch> <442C2FB3.5070800@gmail.com> <44318B97.3050408@gmx.ch> <4431AC78.9010705@gmail.com> <44321B32.8000504@gmx.ch> Message-ID: <44322B60.3050302@gmail.com> Niklaus Haldimann wrote: > Oops, I didn't intend to invalidate your work. ;) I actually checked > your user directory yesterday, because you said in an earlier mail that > you would work on the branch there. But since I didn't see any changes > related to rlist I assumed you decided to postpone this ... No, I was working on that but I didn't commit because I wanted to get all things working before doing that... I've had some problem fixing all various modules that accessed rpython.rlist directly. > What you did doesn't look so bad, I just looked at it. In general, I'm > impressed that you found your way around the RTyper so easily. ;) Don't worry, it has not been so easy! :-) > The > main difference to our work is that you created many new low-level > operations for the list interface. Since there will also have to be a > dict and string interface this would lead to an explosion of low-level > operations. Our approach also makes testing of these data structures at > a lower level easier (see test_oolist.py and test_oortype.py). The reason beyond that is that I've found no other way to do this: I really wanted to create as few low-level ops as possibile, but I coudn't figure out how to do. I try to explain better what I did: my first attempt was to provide a low-level operation 'list_lenght' and then implement rtype_is_true in terms of 'list_lenght', so I wrote a thing like this: class ListRepr(AbstractListRepr): def rtype_len(self, hop): v_lst, = hop.inputargs(self) return hop.genop('list_len', [v_lst], resulttype=Signed) def rtype_is_true(self, hop): v_lst, = hop.inputargs(self) return hop.gendirectcall(ll_list_is_true, v_lst) def ll_list_is_true(lst): return lst is not None and len(lst) != 0 I hoped that the rtyper was smart enough to convert 'len(lst)' into my low-level op 'list_len', but it wasn't: indeed, it generated code that called len function for a generic PyObject* and that was not what I wanted. I tried to copy the implementation of lltypesystem.rlist.ll_list_is_true, but I couldn't because that can call the ll_lenght() method that I didn't have. Now it has no longer importance, but how could I do to get thing working? ciao Anto From anto.cuni at gmail.com Tue Apr 4 20:52:46 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue Apr 4 20:56:24 2006 Subject: [pypy-dev] Rtyping classes definitions? Message-ID: <4432C07E.6090406@gmail.com> Hi, I'm writing the code that generates CLI classes starting from ootypesystem classes definitions, but as always I have a problem :-). At the moment the only way to access class definitions is by calling TranslationContext.annotator.getuserclassdefinitions(), which returns a list of ClassDef instances: I've noticed that class attributes are not typed but only annotated. Obviously for generating CLI classes I need to rtype class definitions, so that I can know the exact low-level type of the fields: I wondered if the absence of the rtyping step is intended or if it is a thing to do. In the latter case I could try to do it (hoping to do better than with rlist :-). ciao Anto From nhaldimann at gmx.ch Tue Apr 4 23:52:43 2006 From: nhaldimann at gmx.ch (Niklaus Haldimann) Date: Tue Apr 4 23:53:10 2006 Subject: [pypy-dev] Rtyping classes definitions? In-Reply-To: <4432C07E.6090406@gmail.com> References: <4432C07E.6090406@gmail.com> Message-ID: <4432EAAB.90805@gmx.ch> Hi Antonio Antonio Cuni wrote: > At the moment the only way to access class definitions is by calling > TranslationContext.annotator.getuserclassdefinitions(), which returns a > list of ClassDef instances: I've noticed that class attributes are not > typed but only annotated. > > Obviously for generating CLI classes I need to rtype class definitions, > so that I can know the exact low-level type of the fields: I wondered if > the absence of the rtyping step is intended or if it is a thing to do. > In the latter case I could try to do it (hoping to do better than with > rlist :-). There must be a misunderstanding here, the rtyping step is not at all missing. ;) If you look at rtyped graphs you'll see that instances (and classes) have low-level types of the ootype.Instance kind. These Instance types have a _field attribute that is a dict mapping field names to their low-level types. You should be operating with these Instance types not with ClassDefs, the latter are mostly an implementation detail of the annotation phase. Cheers Nik From mwh at python.net Tue Apr 4 23:52:46 2006 From: mwh at python.net (Michael Hudson) Date: Tue Apr 4 23:53:50 2006 Subject: [pypy-dev] Re: Rtyping classes definitions? References: <4432C07E.6090406@gmail.com> Message-ID: <2mbqvhrosx.fsf@starship.python.net> Antonio Cuni writes: > Hi, > I'm writing the code that generates CLI classes starting from > ootypesystem classes definitions, but as always I have a problem :-). > > At the moment the only way to access class definitions is by calling > TranslationContext.annotator.getuserclassdefinitions(), which returns > a list of ClassDef instances: I've noticed that class attributes are > not typed but only annotated. Er. Well. I am almost entirely sure that your current approach of (simplified): for graph in self.translator.graphs: f = Function(graph) f.render(self.ilasm) is not really going to work. Have you looked at how (say) genc works? It first computes names for everything (the 'LowLevelDatabase'), and then spits out the code. 'everything' includes all the types referenced by the graphs, and because the graphs have been rtyped, the types you find are things like instances of ootype.Class. > Obviously for generating CLI classes I need to rtype class > definitions, so that I can know the exact low-level type of the > fields: I wondered if the absence of the rtyping step is intended or > if it is a thing to do. In the latter case I could try to do it > (hoping to do better than with rlist :-). I'm not sure this question, as phrased, makes sense :) Cheers, mwh -- A witty saying proves nothing. -- Voltaire From anto.cuni at gmail.com Wed Apr 5 00:04:55 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed Apr 5 00:06:43 2006 Subject: [pypy-dev] Rtyping classes definitions? In-Reply-To: <4432EAAB.90805@gmx.ch> References: <4432C07E.6090406@gmail.com> <4432EAAB.90805@gmx.ch> Message-ID: <4432ED87.8080608@gmail.com> Hi Niklaus, Niklaus Haldimann wrote: > There must be a misunderstanding here, the rtyping step is not at all > missing. ;) If you look at rtyped graphs you'll see that instances (and > classes) have low-level types of the ootype.Instance kind. These > Instance types have a _field attribute that is a dict mapping field > names to their low-level types. You should be operating with these > Instance types not with ClassDefs, the latter are mostly an > implementation detail of the annotation phase. Yes, there was a misunderstanding :-). I was searching for something containing the list of classes to be rendered, but now I've understood that I should collect informations about classes as long as I generate the code, that is the way gensqueak works (I was looking at your sources just now). Thanks a lot for your help, I'm sorry if I make you bored with my many questions, I hope they will decrease as soon as I will get acquainted with pypy's logic. ciao Anto From anto.cuni at gmail.com Wed Apr 5 00:09:10 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed Apr 5 00:10:56 2006 Subject: [pypy-dev] Re: Rtyping classes definitions? In-Reply-To: <2mbqvhrosx.fsf@starship.python.net> References: <4432C07E.6090406@gmail.com> <2mbqvhrosx.fsf@starship.python.net> Message-ID: <4432EE86.3040106@gmail.com> Michael Hudson wrote: > Er. Well. I am almost entirely sure that your current approach of > (simplified): > > for graph in self.translator.graphs: > f = Function(graph) > f.render(self.ilasm) > > is not really going to work. Have you looked at how (say) genc works? > It first computes names for everything (the 'LowLevelDatabase'), and > then spits out the code. 'everything' includes all the types > referenced by the graphs, and because the graphs have been rtyped, the > types you find are things like instances of ootype.Class. Yes, I think you are probably right. It worked until now for simple experiments, but now I have to follow another approach, as I've just wrote to Niklaus. I hope to commit something usable tomorrow... ciao Anto From niko at alum.mit.edu Wed Apr 5 13:51:13 2006 From: niko at alum.mit.edu (Niko Matsakis) Date: Wed Apr 5 13:51:51 2006 Subject: [pypy-dev] how to get into the codebase? Open bugs? Message-ID: <33D3D202-84BC-44B6-83D4-B72615BDB231@alum.mit.edu> Hello, I am yet-another-compiler-PhD-student interested in exploring PyPy. I have been lurking on the list for some time, and a Python enthusiast for longer, but I have yet to really dig into PyPy to get a better understanding of how it works (beyond what is published on the web site). In previous jobs and lives, I've always found that fixing a few simple bugs or implementing some low priority features can be a great way to delve into a code base. As such, I am wondering if there is a list somewhere of such "open bugs" or pending tasks that I might try to tackle. I understand that attending a Code Sprint would also be a great way to get into the code, but I'm not sure how/when I'll be able to fit that into my schedule. Perhaps if there is one close to Switzerland, or somewhere my wife wouldn't going on vacation. :) Eventually, of course, I'd love to be conversant enough with the PyPy code base to try experimenting more boldly with it, but I guess one must take baby steps before one can run. Niko From hpk at trillke.net Wed Apr 5 14:41:36 2006 From: hpk at trillke.net (holger krekel) Date: Wed Apr 5 14:45:57 2006 Subject: [pypy-dev] how to get into the codebase? Open bugs? In-Reply-To: <33D3D202-84BC-44B6-83D4-B72615BDB231@alum.mit.edu> References: <33D3D202-84BC-44B6-83D4-B72615BDB231@alum.mit.edu> Message-ID: <20060405124136.GK21936@solar.trillke> Hi Niko! On Wed, Apr 05, 2006 at 13:51 +0200, Niko Matsakis wrote: > I am yet-another-compiler-PhD-student interested in exploring PyPy. > I have been lurking on the list for some time, and a Python > enthusiast for longer, but I have yet to really dig into PyPy to get > a better understanding of how it works (beyond what is published on > the web site). he, welcome! > In previous jobs and lives, I've always found that fixing a few > simple bugs or implementing some low priority features can be a great > way to delve into a code base. As such, I am wondering if there is a > list somewhere of such "open bugs" or pending tasks that I might try > to tackle. There is https://codespeak.net/issue/pypy-dev/ which i recently updated and lists a number of problems. We are not really keeping very strictly to the classications of "easy,medium,hard" for the issues but it should be helpful a bit, still. > I understand that attending a Code Sprint would also be a great way > to get into the code, but I'm not sure how/when I'll be able to fit > that into my schedule. Perhaps if there is one close to Switzerland, > or somewhere my wife wouldn't going on vacation. :) many of us are at the moment in Leysin (Switzerland) ... > Eventually, of course, I'd love to be conversant enough with the PyPy > code base to try experimenting more boldly with it, but I guess one > must take baby steps before one can run. sounds like a good strategy. feel free to ask questions if you get stuck or wonder about things. best, holger From mwh at python.net Wed Apr 5 15:13:50 2006 From: mwh at python.net (Michael Hudson) Date: Wed Apr 5 15:15:23 2006 Subject: [pypy-dev] Re: how to get into the codebase? Open bugs? References: <33D3D202-84BC-44B6-83D4-B72615BDB231@alum.mit.edu> Message-ID: <2mk6a4qi5t.fsf@starship.python.net> Niko Matsakis writes: > I understand that attending a Code Sprint would also be a great way > to get into the code, but I'm not sure how/when I'll be able to fit > that into my schedule. Perhaps if there is one close to Switzerland, > or somewhere my wife wouldn't going on vacation. :) Well, as well as being in Switzerland right now, as Holger mentioned, there will be sprints around EuroPython in early July, which is being held at CERN this year. > Eventually, of course, I'd love to be conversant enough with the PyPy > code base to try experimenting more boldly with it, but I guess one > must take baby steps before one can run. True, but it definitely helps to have something in mind too! Cheers, mwh -- FORD: Just put the fish in your ear, come on, it's only a little one. ARTHUR: Uuuuuuuuggh! -- The Hitch-Hikers Guide to the Galaxy, Episode 1 From tismer at math.fu-berlin.de Thu Apr 6 04:38:13 2006 From: tismer at math.fu-berlin.de (Christian Tismer) Date: Thu Apr 6 08:08:52 2006 Subject: [pypy-dev] Re: email problem, please use this address In-Reply-To: <43FB7420.6080709@math.fu-berlin.de> References: <43FB7420.6080709@math.fu-berlin.de> Message-ID: <44347F15.6060006@math.fu-berlin.de> Christian Tismer wrote: > Hi, > > I have trouble with tismer.com, which happened to crash right now when > I'm > in the US. I hope to get in contact with service staff late nonight. > > Until then, please use this email address if you want to reach me. > > cheers - chris > This happened again, today! so, pls see above :-) From niko at alum.mit.edu Thu Apr 6 10:21:29 2006 From: niko at alum.mit.edu (Niko Matsakis) Date: Thu Apr 6 10:30:33 2006 Subject: [pypy-dev] Re: how to get into the codebase? Open bugs? In-Reply-To: <2mk6a4qi5t.fsf@starship.python.net> References: <33D3D202-84BC-44B6-83D4-B72615BDB231@alum.mit.edu> <2mk6a4qi5t.fsf@starship.python.net> Message-ID: > Well, as well as being in Switzerland right now, as Holger mentioned, > there will be sprints around EuroPython in early July, which is being > held at CERN this year. Thanks everyone; I may be able to swing by CERN, depending on how my schedule works out. In any case, I will look into some of those smaller issues and see if I can't contribute something. Niko From tismer at stackless.com Thu Apr 6 12:28:00 2006 From: tismer at stackless.com (Christian Tismer) Date: Thu Apr 6 12:28:25 2006 Subject: [pypy-dev] Re: how to get into the codebase? Open bugs? In-Reply-To: References: <33D3D202-84BC-44B6-83D4-B72615BDB231@alum.mit.edu> <2mk6a4qi5t.fsf@starship.python.net> Message-ID: <4434ED30.3090207@stackless.com> Niko Matsakis wrote: Dear Niko, > Thanks everyone; I may be able to swing by CERN, depending on how my > schedule works out. In any case, I will look into some of those smaller > issues and see if I can't contribute something. I'd like to encourage you.! At the same time, I'd like to point out that you are not envisioning a simple task, whqatevery you are trying with PyPy. But feel invited to ask me for assistance, any tine. ciao -- chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From anto.cuni at gmail.com Thu Apr 6 13:03:21 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu Apr 6 13:05:01 2006 Subject: [pypy-dev] A problem with unbound methods Message-ID: <4434F579.2030106@gmail.com> Hi, I have some problems for translating calls to unbound methods. Let's show with an example: class MyClass: def __init__(self, x): self.x = x class MyDerivedClass(MyClass): def __init__(self, x): MyClass.__init__(self, x) During rtyping the field and method names are mangled, so the __init__ method became something like 'o__init____variant0': as long as I call bound methods this is not a problem because the low-level op oosend contains the right mangled name, but difficult arises when I try to call an unbound method such as the one showed above; the low-level op that I obtain is this: v9 = direct_call((), self_1, x_2) Let 'x = op.args[0].value': (Pdb) x._name 'Base.__init__' (Pdb) x.graph.name 'Base.__init__' As you can see I can read only the original unmangled name, but it is useless because I've to deal with the mangled one. I've tried to search around for a place where to read that but I couldn't find it. How can I do? thanks for the help ciao Anto From pedronis at strakt.com Thu Apr 6 13:33:58 2006 From: pedronis at strakt.com (Samuele Pedroni) Date: Thu Apr 6 13:36:30 2006 Subject: [pypy-dev] A problem with unbound methods In-Reply-To: <4434F579.2030106@gmail.com> References: <4434F579.2030106@gmail.com> Message-ID: <4434FCA6.4010901@strakt.com> Antonio Cuni wrote: > Hi, > I have some problems for translating calls to unbound methods. > Let's show with an example: > > class MyClass: > def __init__(self, x): > self.x = x > > class MyDerivedClass(MyClass): > def __init__(self, x): > MyClass.__init__(self, x) > > During rtyping the field and method names are mangled, so the __init__ > method became something like 'o__init____variant0': as long as I call > bound methods this is not a problem because the low-level op oosend > contains the right mangled name, but difficult arises when I try to call > an unbound method such as the one showed above; the low-level op that I > obtain is this: > > v9 = direct_call(( at 0xb78e51ac>), self_1, x_2) > > Let 'x = op.args[0].value': > > (Pdb) x._name > 'Base.__init__' > (Pdb) x.graph.name > 'Base.__init__' > > As you can see I can read only the original unmangled name, but it is > useless because I've to deal with the mangled one. > I've tried to search around for a place where to read that but I > couldn't find it. How can I do? > you should not trust or use graph names in the backend, apart for givin names to things. If a function is reused in more than one class the information would not be useful (this can happen in Python/RPython). The graph would get the name based on the first class under which it was found, this may be unrelated for example for the class for self to the method name under which the graph is attached. Because there are too many variations about what is allowed in terms of supporting functions vs. just methods, calling superclass implementations of methods even when the method is overriden in a subclass etc in the targets, right now it is up to the backend to traverse and consider all classes and direct_calls and if the same graph appears both attached to a method (or methods) in a class (or classes) and in static method(s) in a direct call(s) decide what to do. This is also true in general for graphs that appear as more than on method in one place. Strategies here may vary from using a single static method (e.g. in Java) and having methods delegate to it, or if possible having just the method if the types of the first arguments in the direct call allow that approach ... regards From arigo at tunes.org Thu Apr 6 18:08:10 2006 From: arigo at tunes.org (Armin Rigo) Date: Thu Apr 6 18:08:12 2006 Subject: [pypy-dev] List and string in ootypesystem In-Reply-To: <44322B60.3050302@gmail.com> References: <442BC9CC.6010503@gmail.com> <442BF15B.9070102@gmx.ch> <442C2FB3.5070800@gmail.com> <44318B97.3050408@gmx.ch> <4431AC78.9010705@gmail.com> <44321B32.8000504@gmx.ch> <44322B60.3050302@gmail.com> Message-ID: <20060406160810.GA22957@code0.codespeak.net> Hi Antonio, On Tue, Apr 04, 2006 at 10:16:32AM +0200, Antonio Cuni wrote: > def ll_list_is_true(lst): > return lst is not None and len(lst) != 0 > > I hoped that the rtyper was smart enough to convert 'len(lst)' into my > low-level op 'list_len', but it wasn't: indeed, it generated code that > called len function for a generic PyObject* and that was not what I wanted. > I tried to copy the implementation of > lltypesystem.rlist.ll_list_is_true, but I couldn't because that can call > the ll_lenght() method that I didn't have. Now it has no longer > importance, but how could I do to get thing working? It is probably less confusing to call a method like .length() instead of using len() at this level again. But in both cases, you need to add support for this in the annotator and the rtyper again -- the ll_list_is_true() function is itself passed through both of them. I see you added SomeOOList to annoation.model.lltype_to_annotation(). There is already a generic 'def len()' in the base class SomeObject, so that's how the annotator is happy with your ll function's 'len(lst)'. Fine here. If you wanted a .length() method instead, you would need a 'def method_length()' in unaryop.py. On the rtyper side, you need something similar to rpython/rptr.py that maps SomeOOList back to its low-level type, with yet another Repr. It's this repr that must implement the operations you want to be able to use in low-level helpers; e.g. rtype_len() if you want len(lst) to work; or if instead you use .length() in low-level helpers, then you would need an rtype_method_length() in the repr corresponding to SomeOOList (by opposition to the rtype_len() in the repr corresponding to SomeList). Nik's approach is to map lists to reguar OO classes and instances, which are already supported in the annotator/rtyper with a similarly indirect approach: SomeOOClass/SomeOOInstance in the annotator, which rpython/ootypesystem/rootype.py maps back to the low-level OO types. Just like rptr.py, this rootype.py is only needed to support low-level helpers. A bientot, Armin From arigo at tunes.org Thu Apr 6 18:42:42 2006 From: arigo at tunes.org (Armin Rigo) Date: Thu Apr 6 18:42:42 2006 Subject: [pypy-dev] rctypes design document In-Reply-To: <44311BEE.7000109@klix.ch> References: <44311BEE.7000109@klix.ch> Message-ID: <20060406164242.GB22957@code0.codespeak.net> Hi Gerald, On Mon, Apr 03, 2006 at 02:58:22PM +0200, Gerald Klix wrote: > yesterday I checked in the final version of the rcytpes design document. > I am convinced I finally got everything right. Especially the management > of keep alive pointers. Thanks! We progressed on rctypes quite a bit, and indeed we are implementing a keepalive scheme very much inspired by the one you are describing. One difference (which we did not implemented so far) is that structures and arrays that contain several pointers will likely need one keepalive per pointer field or array item, to be able to keep all stored pointers' target memory alive. A bientot, Armin. From anto.cuni at gmail.com Thu Apr 6 21:31:04 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu Apr 6 21:32:52 2006 Subject: [pypy-dev] A problem with unbound methods In-Reply-To: <4434FCA6.4010901@strakt.com> References: <4434F579.2030106@gmail.com> <4434FCA6.4010901@strakt.com> Message-ID: <44356C78.7030409@gmail.com> Hi Samuele, Samuele Pedroni wrote: > you should not trust or use graph names in the backend, apart > for givin names to things. If a function is reused in more > than one class the information would not be useful (this can > happen in Python/RPython). > The graph would get the name based on the first class > under which it was found, this may be unrelated for example > for the class for self to the method name under which the graph > is attached. nice to know this, I didn't know. I think I have to rethink to my approach for code generation... > Because there are too many variations about what is allowed in terms > of supporting functions vs. just methods, calling superclass > implementations of methods even when the method is overriden > in a subclass etc in the targets, right now it is up to the backend > to traverse and consider all classes and direct_calls and if the same > graph appears both attached to a method (or methods) in a class (or > classes) and in static method(s) in a direct call(s) decide what to do. > This is also true in general for graphs that appear as more than on > method in one place. Ok, now it's clearer, thanks. So, to respond to my original question, I should create a sort of "graph database" to lookup when I need to know where have I put the code for that graph, right? Well, let's begin refactoring! :-) ciao Anto From anto.cuni at gmail.com Thu Apr 6 21:11:21 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu Apr 6 21:33:19 2006 Subject: [pypy-dev] List and string in ootypesystem In-Reply-To: <20060406160810.GA22957@code0.codespeak.net> References: <442BC9CC.6010503@gmail.com> <442BF15B.9070102@gmx.ch> <442C2FB3.5070800@gmail.com> <44318B97.3050408@gmx.ch> <4431AC78.9010705@gmail.com> <44321B32.8000504@gmx.ch> <44322B60.3050302@gmail.com> <20060406160810.GA22957@code0.codespeak.net> Message-ID: <443567D9.3090902@gmail.com> Hi Armin, hi Niklaus Armin Rigo wrote: > I see you added SomeOOList to annoation.model.lltype_to_annotation(). > There is already a generic 'def len()' in the base class SomeObject, so > that's how the annotator is happy with your ll function's 'len(lst)'. > Fine here. If you wanted a .length() method instead, you would need a > 'def method_length()' in unaryop.py. > > On the rtyper side, you need something similar to rpython/rptr.py that > maps SomeOOList back to its low-level type, with yet another Repr. It's > this repr that must implement the operations you want to be able to use > in low-level helpers; e.g. rtype_len() if you want len(lst) to work; or > if instead you use .length() in low-level helpers, then you would need > an rtype_method_length() in the repr corresponding to SomeOOList (by > opposition to the rtype_len() in the repr corresponding to SomeList). > > Nik's approach is to map lists to reguar OO classes and instances, which > are already supported in the annotator/rtyper with a similarly indirect > approach: SomeOOClass/SomeOOInstance in the annotator, which > rpython/ootypesystem/rootype.py maps back to the low-level OO types. > Just like rptr.py, this rootype.py is only needed to support low-level > helpers. Thank you for the great explanation: now I see things much more clearly, especially the role played by rptr.py and rootype.py, which I didn't understand very well before now. It seems that question by question I'm really getting into PyPy's logic... I hope I will be able to finish my apprenticeship soon :-) ciao Anto From mwh at python.net Thu Apr 6 23:21:43 2006 From: mwh at python.net (Michael Hudson) Date: Thu Apr 6 23:22:31 2006 Subject: [pypy-dev] Re: List and string in ootypesystem References: <442BC9CC.6010503@gmail.com> <442BF15B.9070102@gmx.ch> <442C2FB3.5070800@gmail.com> <44318B97.3050408@gmx.ch> <4431AC78.9010705@gmail.com> <44321B32.8000504@gmx.ch> <44322B60.3050302@gmail.com> <20060406160810.GA22957@code0.codespeak.net> <443567D9.3090902@gmail.com> Message-ID: <2mwte2pfh4.fsf@starship.python.net> Antonio Cuni writes: > It seems that question by question I'm really getting into PyPy's > logic... I hope I will be able to finish my apprenticeship soon :-) Unfortunately, I don't think you are Swiss, so you are probably doomed to being an apprentice forever like this rest of us :) More seriously, you seem to be doing very well! Hope you are having fun too :) Cheers, mwh -- M-x psych[TAB][RETURN] -- try it From stephan.diehl at gmx.net Fri Apr 7 18:16:39 2006 From: stephan.diehl at gmx.net (Stephan Diehl) Date: Fri Apr 7 18:17:09 2006 Subject: [pypy-dev] std objectspace interface Message-ID: <200604071816.39500.stephan.diehl@gmx.net> Hallo all, I have a question about the standard objectspace interface. It's about how to best return an iterator. While writing the 'set' implementation, it occured to me, that I'm using a W_SeqIterObject directly, which is bad, since this breaks the intended separation of types / objects. Instead, I'd like to use something like space.newseqiter. But 'newseqiter' takes a wrapped object, while I'd rather use a way to 'translate' some interp level iterator of wrapped objects directly to an application lever iterator. This would be similar to newlist which creates an application level list of application level objects out of an interp level list of application level objects. Or is the dictobject way the preferred one? There we have special iterator object 'W_DictIter_Keys', for example. I hope I make a bit of sense. Cheers Stephan From arigo at tunes.org Fri Apr 7 18:43:27 2006 From: arigo at tunes.org (Armin Rigo) Date: Fri Apr 7 18:43:30 2006 Subject: [pypy-dev] std objectspace interface In-Reply-To: <200604071816.39500.stephan.diehl@gmx.net> References: <200604071816.39500.stephan.diehl@gmx.net> Message-ID: <20060407164327.GA6853@code0.codespeak.net> Hi Stephan, On Fri, Apr 07, 2006 at 06:16:39PM +0200, Stephan Diehl wrote: > space.newseqiter. This is only for iterating over sequences, just like W_SeqIterObject. > Or is the dictobject way the preferred one? There we have special iterator > object 'W_DictIter_Keys', for example. Yes, you need a special iterator class like W_DictIter_Keys. BTW, W_SetObject.data, which is an r_dict, should ideally have 'None' as keys instead of space.w_True. This would make the low-level translated code more memory-efficient, by using 'void' values instead of pointers to space.w_True. A bientot, Armin. From stephan.diehl at gmx.net Fri Apr 7 18:50:58 2006 From: stephan.diehl at gmx.net (Stephan Diehl) Date: Fri Apr 7 18:51:28 2006 Subject: [pypy-dev] std objectspace interface In-Reply-To: <20060407164327.GA6853@code0.codespeak.net> References: <200604071816.39500.stephan.diehl@gmx.net> <20060407164327.GA6853@code0.codespeak.net> Message-ID: <200604071850.58566.stephan.diehl@gmx.net> On Friday 07 April 2006 18:43, Armin Rigo wrote: > Hi Stephan, > > On Fri, Apr 07, 2006 at 06:16:39PM +0200, Stephan Diehl wrote: > > space.newseqiter. > > This is only for iterating over sequences, just like W_SeqIterObject. > > > Or is the dictobject way the preferred one? There we have special > > iterator object 'W_DictIter_Keys', for example. > > Yes, you need a special iterator class like W_DictIter_Keys. o.k. > > BTW, W_SetObject.data, which is an r_dict, should ideally have 'None' as > keys instead of space.w_True. This would make the low-level translated > code more memory-efficient, by using 'void' values instead of pointers > to space.w_True. ahh, good point, I'll change that then (the other stuff, we talked about last week, is already changed, but not checked in) Thanks Stephan > > > A bientot, > > Armin. From bclum at cs.ucsd.edu Mon Apr 10 02:58:45 2006 From: bclum at cs.ucsd.edu (Brian C. Lum) Date: Mon Apr 10 02:59:38 2006 Subject: [pypy-dev] Printing a Flowgraph Message-ID: Dear Pypy developers: I was reading how you generate a flowgraph with Pypy from the getting-started page. I was just wondering if anyone knew how to print that graph once it displays. I don't really have much experience with Graphviz. Is there a way to save the graphviz file or print it to a .ps? http://codespeak.net/pypy/dist/pypy/doc/getting-started.html#trying-out-the-translator Thanks! Brian Just an FYI, if you want to try to follow the directions, they're a little bit outdated. To run it (after installing Pygame), start the interactive translator: cd pypy/bin python translator.py and to view a sample code: >>> t = Translation(test.is_perfect_number) >>> t.view() From mwh at python.net Mon Apr 10 09:57:31 2006 From: mwh at python.net (Michael Hudson) Date: Mon Apr 10 09:58:30 2006 Subject: [pypy-dev] Re: Printing a Flowgraph References: Message-ID: <2m7j5xq2vo.fsf@starship.python.net> "Brian C. Lum" writes: > Dear Pypy developers: > > I was reading how you generate a flowgraph with Pypy from the > getting-started page. I was just wondering if anyone knew how to > print that graph once it displays. > > I don't really have much experience with Graphviz. Is there a way to > save the graphviz file or print it to a .ps? Well, the dot file will probably be lurking in /tmp/usession-*/ somewhere. I don't know if there's an "official" way of getting hold of it, though. One you have the dot file, "dot -Tps flow.dot > flow.ps && lp flow.ps" or whatever should work fine. > http://codespeak.net/pypy/dist/pypy/doc/getting-started.html#trying-out-the-translator > > Thanks! > Brian > > Just an FYI, if you want to try to follow the directions, they're a > little bit outdated. To run it (after installing Pygame), start the > interactive translator: > > cd pypy/bin > python translator.py > > and to view a sample code: > >>>> t = Translation(test.is_perfect_number) >>>> t.view() I'm not sure what you're referring to. The docs you linked to look correct to me (i.e. there is no bin/translator.py any more in svn). Cheers, mwh -- we're already scrubbing the face of intuition with steel wool, setting it on fire, then putting it out with an axe . -- Tim Peters, on comparing recursive structures From bclum at cs.ucsd.edu Tue Apr 11 05:16:13 2006 From: bclum at cs.ucsd.edu (Brian C. Lum) Date: Tue Apr 11 05:16:57 2006 Subject: [pypy-dev] Re: Printing a Flowgraph In-Reply-To: <2m7j5xq2vo.fsf@starship.python.net> References: <2m7j5xq2vo.fsf@starship.python.net> Message-ID: Thanks a lot. Your way worked perfectly. I just downloaded version 0.8.0 of pypy. In that version, they use translation.py. I was not sure whether the document was referring to a newer or older version, so I just included the extra information in case. Brian On Mon, 10 Apr 2006, Michael Hudson wrote: > "Brian C. Lum" writes: > >> Dear Pypy developers: >> >> I was reading how you generate a flowgraph with Pypy from the >> getting-started page. I was just wondering if anyone knew how to >> print that graph once it displays. >> >> I don't really have much experience with Graphviz. Is there a way to >> save the graphviz file or print it to a .ps? > > Well, the dot file will probably be lurking in /tmp/usession-*/ > somewhere. I don't know if there's an "official" way of getting hold > of it, though. > > One you have the dot file, "dot -Tps flow.dot > flow.ps && lp flow.ps" > or whatever should work fine. > >> http://codespeak.net/pypy/dist/pypy/doc/getting-started.html#trying-out-the-translator >> >> Thanks! >> Brian >> >> Just an FYI, if you want to try to follow the directions, they're a >> little bit outdated. To run it (after installing Pygame), start the >> interactive translator: >> >> cd pypy/bin >> python translator.py >> >> and to view a sample code: >> >>>>> t = Translation(test.is_perfect_number) >>>>> t.view() > > I'm not sure what you're referring to. The docs you linked to look > correct to me (i.e. there is no bin/translator.py any more in svn). > > Cheers, > mwh > > From tismer at stackless.com Tue Apr 11 05:54:25 2006 From: tismer at stackless.com (Christian Tismer) Date: Tue Apr 11 05:56:38 2006 Subject: [pypy-dev] picxenk gallery :: logo :: 1 Message-ID: <443B2871.8050301@stackless.com> http://image.xenbio.net/logo/pypy_proto Heh, not bad :-) what do you think? From lac at strakt.com Tue Apr 11 12:14:31 2006 From: lac at strakt.com (Laura Creighton) Date: Tue Apr 11 12:15:12 2006 Subject: [pypy-dev] picxenk gallery :: logo :: 1 In-Reply-To: Message from Christian Tismer of "Mon, 10 Apr 2006 20:54:25 PDT." <443B2871.8050301@stackless.com> References: <443B2871.8050301@stackless.com> Message-ID: <200604111014.k3BAEV6p008124@theraft.strakt.com> In a message of Mon, 10 Apr 2006 20:54:25 PDT, Christian Tismer writes: >http://image.xenbio.net/logo/pypy_proto > >Heh, not bad :-) > >what do you think? >_______________________________________________ >pypy-dev@codespeak.net >http://codespeak.net/mailman/listinfo/pypy-dev I was always very fond of one of the rejected python.org logos Tim Parkin came up with. I include it as an attatchment. Tim also sent me some urls of other rejected python.org logos, and confirmed that if we want it, it is ours. so first the urls, but the one I like best is the attatchment ... http://www.artfiles.org/python.org/pub/beta.python.org/resources/design/history/ logoandtypeface.gif http://www.artfiles.org/python.org/pub/beta.python.org/resources/design/history/ master.png Laura -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 4217 bytes Desc: not available Url : http://codespeak.net/pipermail/pypy-dev/attachments/20060411/e3ce8c2d/attachment.gif From cfbolz at googlemail.com Tue Apr 11 16:18:39 2006 From: cfbolz at googlemail.com (Carl Friedrich Bolz) Date: Tue Apr 11 16:27:15 2006 Subject: [pypy-dev] Re: picxenk gallery :: logo :: 1 In-Reply-To: <200604111014.k3BAEV6p008124@theraft.strakt.com> References: <443B2871.8050301@stackless.com> <200604111014.k3BAEV6p008124@theraft.strakt.com> Message-ID: <348899050604110718j648491adw89355fc874a8dccc@mail.gmail.com> 2006/4/11, Laura Creighton : > In a message of Mon, 10 Apr 2006 20:54:25 PDT, Christian Tismer writes: > >http://image.xenbio.net/logo/pypy_proto > > > >Heh, not bad :-) > > > >what do you think? It has a very cool recursive aspect that I like. Not sure about colors and the placement of the letters. > I was always very fond of one of the rejected python.org logos Tim Parkin > came > up with. I include it as an attatchment. Tim also sent me some urls > of other rejected python.org logos, and confirmed that if we want it, it > is ours. > > so first the urls, but the one I like best is the attatchment ... This one is nice too, but kind of too generic. It has this typical swirly feeling that seems to me quite common. In my opinion none of those two comes close the very cool logo that we have already :-) Cheers, Carl Friedrich From tismer at stackless.com Tue Apr 11 20:41:01 2006 From: tismer at stackless.com (Christian Tismer) Date: Tue Apr 11 20:41:57 2006 Subject: [pypy-dev] Re: picxenk gallery :: logo :: 1 In-Reply-To: <348899050604110718j648491adw89355fc874a8dccc@mail.gmail.com> References: <443B2871.8050301@stackless.com> <200604111014.k3BAEV6p008124@theraft.strakt.com> <348899050604110718j648491adw89355fc874a8dccc@mail.gmail.com> Message-ID: <443BF83D.2020001@stackless.com> Carl Friedrich Bolz wrote: > It has a very cool recursive aspect that I like. Not sure about colors > and the placement of the letters. The recursivenes was what attracted me, too. > In my opinion none of those two comes close the very cool logo that we > have already :-) Maybe one could use the recursive idea a bit, but I agree. We should produce T-shirts, again. Maybe with a large PY on the back. ciao - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From michaelschneider at fuse.net Wed Apr 12 05:46:15 2006 From: michaelschneider at fuse.net (Michael Schneider) Date: Wed Apr 12 08:35:56 2006 Subject: [pypy-dev] pypy embedded / new atmel 32 Message-ID: <443C7807.5080309@fuse.net> Hello All, I have been very interested in pypy and embedded systems. I use atmel's line (cheap, easy architecture, gcc). It is still a bit cramped for pypy. Atmel just introduced their 32 bit line. It is a whopping $16 per unit. Paralles Propeller chip uses sping (some python linage). Would the new AVR be a candidate for pypy? Article http://www.linuxdevices.com/news/NS6110120875.html Data Sheets http://www.atmel.com/dyn/products/datasheets.asp?family_id=682 It would be great to have python on this chip. This could be used in so many domains... Thanks Mike From arigo at tunes.org Wed Apr 12 12:29:48 2006 From: arigo at tunes.org (Armin Rigo) Date: Wed Apr 12 12:29:49 2006 Subject: [pypy-dev] Mixed modules for both PyPy and CPython Message-ID: <20060412102948.GA9512@code0.codespeak.net> Hi all, This is a follow-up on last night's IRC discussion. It's come to a point where e-mail is better. The topic is "WP3", i.e. how to write in Python built-in modules that work for both CPython and PyPy ("built-in" == "extension" == "compileable" for the purpose of this mail). There are two levels of issues involved in this: * In which style the modules should be written in the first place? * Which style is easier to support in both CPython and PyPy given what we have now? In PyPy, we are using "mixed modules" with an interp-level and an app-level part, both optional, with explicit ways to interact between them. The interp-level part is what gets compiled to C. It contains code like 'space.add(w_1, w_2)' to perform an app-level addition. Christian is working on an alternative, which is a single-source approach where the annotator's failures to derive types are taken as a hint that the corresponding variables should contain wrapped objects. This is a convenient but hard-to-control extension of a historical hack, which works mostly because we needed it while we were still developing the annotator and the rtyper. I am clearly biased towards "solution 1", which is to reuse the mixed modules style that we have evolved over several years. Here is some implementation effort comparison between the two styles (1=mixed module, 2=single source). It is easy to develop and test in 2, as it runs on CPython with no further efforts. For 1, the module developer needs the whole of PyPy to test his complete module. We could ease this by writing a dummy interface-compatible gateway and object space, performing sanity-checks and giving useful error messages. Then the developer only needs to check his code against this. Annotation problems in the module are easier to spot early in the model 1; indeed, the fact that with 2 we cannot gracefully crash on SomeObjects, and moreover the need for many fragile-looking small extensions to the flow object space and the annotator, is what makes me most uneasy about 2. The mixed modules are designed for PyPy, so they work there already now. For the single-source approach to work on PyPy, however, there would be many interesting and delicate problems -- both for py.py and for the translated pypy-c. I'd rather avoid to think about it too much right now :-) For translation, for 2 we already have the basic machinery implemented as a leftover from PyPy "early ages". But I want to convince you that the basic support for 1 is extremely easy, or will soon be. We need a new object space to compile with the mixed module; but this "CPyObjSpace" is completely trivial. It could be based on rctypes, in which case it would look like this: class CPyObjSpace: def newint(self, intval): return ctypes.pydll.PyInt_FromLong(intval) def add(self, w_1, w_2): return ctypes.pydll.PyNumber_Add(w_1, w_2) ... Note that this even works on top of CPython! (Not out-of-the-box on Linux, however, where for obscure reasons CPython is by default not compiled as a separate .so file...). The gateway code can be written in the same style, by using ctypes.pydll.PyObject_Call() to call the app-level parts, and ctypes callbacks for the reverse direction. The calls like 'ctypes.pydll.PyInt_FromLong()' return an instance of 'ctypes.py_object', which naturally plays the role of a wrapped object for the CPyObjSpace. The more difficult bits of translation are e.g. support for creating at interp-level types that are exposed to app-level, in particular about the special methods __add__, __getitem__ etc. There is an example of that in pypy.module.Numeric.interp_numeric, where __getitem__ is added to the TypeDef of "array". This creates some difficulty that is common to approaches 1 and 2, and which I think Christian is also working on. In 1 we would probably make the TypeDef declaration turn, for CPython, into static PyTypeObject structures. The special methods can be recognized and put into the proper slots, with a bit of wrapping. The whole PyTypeObject structure with its pointers to functions could be expressed with ctypes too: # use pypy.rpython.rctypes.ctypes_platform to get at the layout PyTypeObject = ctypes_platform.getstruct("PyTypeObject *", ...) def TypeDef(name, **rawdict): mytype = PyTypeObject() mytype.tp_name = name if '__getitem__' in rawdict: # build a ctypes callback mp_subscript = callback_wrapper(rawdict['__dict__']) mytype.tp_as_mapping.mp_subscript = mp_subscript return mytype This gives us prebuilt PyTypeObject structures in ctypes, which get compiled into the C code by rctypes. At the same time, running on top of CPython is still possible; ctypes will let you issue the following kind of calls dynamically when running on CPython: myinstance = ctypes.pydll.PyObject_Call(mytype) The same line translated by rctypes becomes in the C file: PyObject* myinstance = PyObject_Call(mytype); Of course this latter part is dependent on rctypes being completed first. I understand and respect Christian's needs for something that works right now; nevertheless, at this point I think the mixed modules approach is the best medium-term solution. This is all open to discussion, of course. ...but do keep in mind that the people who completed the annotator as it is now are not likely to appreciate the addition of Yet More Ad-Hoc Heuristics all over the place :-/ A bientot, Armin From arigo at tunes.org Wed Apr 12 13:48:19 2006 From: arigo at tunes.org (Armin Rigo) Date: Wed Apr 12 13:48:21 2006 Subject: [pypy-dev] Mixed modules for both PyPy and CPython In-Reply-To: <20060412102948.GA9512@code0.codespeak.net> References: <20060412102948.GA9512@code0.codespeak.net> Message-ID: <20060412114819.GA18540@code0.codespeak.net> Re-hi all, A clarification on a point that caused confusion... The purpose of the whole discussion was about how to have a "write-once, run-everywhere" approach of developing an extension module a single time and then compiling it for either CPython or PyPy. This source code should not need to know if it will be compiled for PyPy or CPython (or even just run on top of the CPython interpreter for testing). The two approaches I opposed are two ways to do that. Some confusion came from the names of the two approaches, which I took from IRC. One approach is "single-source": this just means "source-in-a-single-file", with wrapped objects and native values mixed in the same source code. (It's not in the sense "write a single time the source"; both approaches are about this.) By opposition, "mixed module" means "source-in-two-files", one running at interp-level and one at app-level. This is what we're doing in pypy/module/*/{app,interp}_*.py. Holger suggests to call "single-source" the whole approach of writing once and running everywhere, which makes sense. If we do so, then we need better names for the two approaches. What about "explicit" versus "implicit" levels? Mixed modules provide explicit level separation, whereas the alternative is to rely on the annotator to separate the levels. I'd also like to point out that the latter -- implicit levels -- has a kind of elegance to it that I like. What I dislike, though, beyond the fact that it would require yet another refactoring of PyPy's module approach (and in a way that is quite unclear), is that it requires many additional fragile rules and heuristics all over the flow objspace and the annotator to get the "expected" separation of levels effects. I'd be happy to experiment more in that direction, but I believe that the other direction's work is needed for now and won't be lost anyway. Basically, we can still come back to this and think later about ways to allow more implicit-level manipulations of objects. A bientot, Armin. From arigo at tunes.org Wed Apr 12 15:37:01 2006 From: arigo at tunes.org (Armin Rigo) Date: Wed Apr 12 15:37:02 2006 Subject: [pypy-dev] A problem with unbound methods In-Reply-To: <44356C78.7030409@gmail.com> References: <4434F579.2030106@gmail.com> <4434FCA6.4010901@strakt.com> <44356C78.7030409@gmail.com> Message-ID: <20060412133701.GA25572@code0.codespeak.net> Hi Antonio, On Thu, Apr 06, 2006 at 09:31:04PM +0200, Antonio Cuni wrote: > Ok, now it's clearer, thanks. So, to respond to my original question, I > should create a sort of "graph database" to lookup when I need to know > where have I put the code for that graph, right? We could try to do this in a way that is reusable between multiple back-ends, because many OO back-ends will have a similar problem. What needs to be done is to collect all graphs and see how they are used. For each graph, it means: which regular method(s) are implemented with that graph, and which direct_calls are done to staticmethods implemented with that graph. Then we need some cases. We can be more sutble, but as a minimum we need logic like: if a graph is used only as one method, leave it alone. If it is used only as a static method, leave it alone too (and declare it as a static method somewhere). For more complicated cases, find all methods implemented with this graph, and replace each of them with a new stub graph that just contains a direct_call to the original graph as a static method. Such a transformation would "normalize" the methods in the OOClass objects (so that the approach you're currently using in CLI would continue to work for methods) and also generate a list of remaining graphs that the back-end needs to declare as static methods. A bientot, Armin. From anto.cuni at gmail.com Wed Apr 12 17:03:44 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed Apr 12 17:06:07 2006 Subject: [pypy-dev] A problem with unbound methods In-Reply-To: <20060412133701.GA25572@code0.codespeak.net> References: <4434F579.2030106@gmail.com> <4434FCA6.4010901@strakt.com> <44356C78.7030409@gmail.com> <20060412133701.GA25572@code0.codespeak.net> Message-ID: <443D16D0.3030600@gmail.com> Hi Armin, Armin Rigo wrote: > We could try to do this in a way that is reusable between multiple > back-ends, because many OO back-ends will have a similar problem. What > needs to be done is to collect all graphs and see how they are used. > For each graph, it means: which regular method(s) are implemented with > that graph, and which direct_calls are done to staticmethods implemented > with that graph. > > Then we need some cases. We can be more sutble, but as a minimum we > need logic like: if a graph is used only as one method, leave it alone. > If it is used only as a static method, leave it alone too (and declare > it as a static method somewhere). For more complicated cases, find all > methods implemented with this graph, and replace each of them with a new > stub graph that just contains a direct_call to the original graph as a > static method. > > Such a transformation would "normalize" the methods in the OOClass > objects (so that the approach you're currently using in CLI would > continue to work for methods) and also generate a list of remaining > graphs that the back-end needs to declare as static methods. It sounds reasonable. At the moment by backend suffers just this problem: is a graph is used both as bound and unbound method it will be rendered twice. I though to resolve it in some CLI specific way, but a more general approach is better. By now the few logic I do is localized in translator/cli/database.py, but it is too CLI specific to be reusable: maybe I could try to factor out the backend-independent logic for later reuse. Another good thing to factor out could be the "pending nodes" logic: at the moment both gencli and gensqueak implements that logic independently, but it would be nice to have a reusable framework shared by gensqueak, gencli and possibly other backends. ciao Anto From "news-8a9e0fd91190ca" at northportal.net Wed Apr 12 20:43:57 2006 From: "news-8a9e0fd91190ca" at northportal.net (VanL) Date: Wed Apr 12 20:57:10 2006 Subject: [pypy-dev] Re: Mixed modules for both PyPy and CPython In-Reply-To: <20060412102948.GA9512@code0.codespeak.net> References: <20060412102948.GA9512@code0.codespeak.net> Message-ID: Hello, There is one other issue which may be relevant: restricted python. Restricted python implementations recently showed up on the py3k list as a wishlist item - and they are wanted even for python 2.X. I thought that pypy might be the answer to these restricted python wishes. Implementation would be as follows: For all allowed functionality, create a compilable pypy extension to handle it. For restricted functionality, delegate to a CPyObjectSpace which could allow/disallow or modify the operation. What makes this work where CPy couldn't is the ability to use two different cooperating object spaces to evaluate an expression. CPy doesn't have the object space abstraction which would allow this sort of partitioning. Another benefit is that different restricted interpreters could be created, each with a different set of allowed and disallowed operations. In summary: (compile a restricted PyPy - rpy - as a CPy extension here) >>> import rpy >>> def filehandler(fname, mode): ... if fname in ['allowed1.txt', 'allowed2.txt']: ... return open(fname, mode) ... else: ... raise rpy.Restricted('File access not permitted') ... >>> interp = rpy.new() # restrictions were defined at compile time >>> # Add a callback for some restricted functionality >>> interp.add_handler('file', filehandler) >>> interp.interact() >>>> >>>> # Now in rpy >>>> 2 + 3 # Allowed operation, didn't hit any restrictions 5 >>>> >>>> open('allowed1.txt', 'r') # allowed by handler >>>> >>>> open('disallowed.txt', 'r') # disallowed by handler Traceback (most recent call last): File "", line 1, in ? __main__.__interp__.Restricted: File access not permitted Would either of these approaches lend itself better to this sort of restricted execution idea? VanL From cfbolz at gmx.de Wed Apr 12 21:13:46 2006 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Wed Apr 12 21:14:34 2006 Subject: [pypy-dev] Re: Mixed modules for both PyPy and CPython In-Reply-To: References: <20060412102948.GA9512@code0.codespeak.net> Message-ID: <348899050604121213w1f01c58pb65b4e70394f6deb@mail.gmail.com> Hi! 2006/4/12, VanL : > There is one other issue which may be relevant: restricted python. > Restricted python implementations recently showed up on the py3k list as > a wishlist item - and they are wanted even for python 2.X. Careful. "Restricted Python" (or RPython) means something very specific in PyPy lingo. Restricted Python means that you stick to some restrictions in your coding style to make type inference possible. See http://codespeak.net/pypy/dist/pypy/doc/coding-guide.html#restricted-python for more details. To your mail: security features are part of what we promised to do to the EU, so there will be something happening in that direction (even before November). I personally don't have any clue what :-) [snip] Cheers, Carl Friedrich From "news-8a9e0fd91190ca" at northportal.net Wed Apr 12 21:45:51 2006 From: "news-8a9e0fd91190ca" at northportal.net (VanL) Date: Wed Apr 12 21:47:04 2006 Subject: [pypy-dev] Re: Mixed modules for both PyPy and CPython In-Reply-To: <348899050604121213w1f01c58pb65b4e70394f6deb@mail.gmail.com> References: <20060412102948.GA9512@code0.codespeak.net> <348899050604121213w1f01c58pb65b4e70394f6deb@mail.gmail.com> Message-ID: Hello, Carl Friedrich Bolz wrote: > Careful. "Restricted Python" (or RPython) means something very > specific in PyPy lingo. Restricted Python means that you stick to some > restrictions in your coding style to make type inference possible. See > > http://codespeak.net/pypy/dist/pypy/doc/coding-guide.html#restricted-python > > Sorry - I knew that. I just didn't make the connection from one side of my brain to another. Maybe "secure python" (spy)? What made me think this would work are a couple of comments made here and on the py3k list. First, Armin (about a year ago, I think)made some comments about the objectspace abstraction allowing a "Remote Object Space" and allowing different object spaces to participate in the evaluation of code. Second, comments on py3k list indicated that secure python is difficult because of a) introspection, b) type inference, and c) GIL acquisition. However, a) a partitioned object space would allow introspection to be limited - go too far and you run into an opaque objectspace wall; b) Pypy already has functionality to do some type inference - probably as much as necessary; and c) Pypy doesn't have a GIL. The only relevant GIL would be in the CPy host python, inaccessible to code in the Spy. Of course, I also may be completely wrong :) VanL From vlindberg at novell.com Wed Apr 12 21:33:10 2006 From: vlindberg at novell.com (Van Lindberg) Date: Wed Apr 12 23:41:09 2006 Subject: [pypy-dev] Re: Mixed modules for both PyPy and CPython In-Reply-To: <348899050604121213w1f01c58pb65b4e70394f6deb@mail.gmail.com> References: <20060412102948.GA9512@code0.codespeak.net> <348899050604121213w1f01c58pb65b4e70394f6deb@mail.gmail.com> Message-ID: <443D55F6.10501@novell.com> Hello, Carl Friedrich Bolz wrote: >Careful. "Restricted Python" (or RPython) means something very >specific in PyPy lingo. Restricted Python means that you stick to some >restrictions in your coding style to make type inference possible. See > >http://codespeak.net/pypy/dist/pypy/doc/coding-guide.html#restricted-python > > Sorry - I knew that. I just didn't make the connection from one side of my brain to another. Maybe "secure python" (spy)? What made me think this would work are a couple of comments made here and on the py3k list. First, Armin (about a year ago, I think)made some comments about the objectspace abstraction allowing a "Remote Object Space" and allowing different object spaces to participate in the evaluation of code. Second, comments on py3k list indicated that secure python is difficult because of a) introspection, b) type inference, and c) GIL acquisition. However, a) a partitioned object space would allow introspection to be limited - go too far and you run into an opaque objectspace wall; b) Pypy already has functionality to do some type inference - probably as much as necessary; and c) Pypy doesn't have a GIL. The only relevant GIL would be in the CPy host python, inaccessible to code in the Spy. Of course, I also may be completely wrong :) VanL From bclum at cs.ucsd.edu Thu Apr 13 03:33:13 2006 From: bclum at cs.ucsd.edu (Brian C. Lum) Date: Thu Apr 13 03:34:16 2006 Subject: [pypy-dev] Information from the FlowObjSpace Message-ID: Dear Pypy developers: I have gone through the source code for the FlowObjSpace in pypy.objspace.flow.objspace, but I am confused how to traverse through the blocks or obtain the information for each blocks in the graph. From my understanding: space = FlowObjSpace() graph = space.build_flow(func) Once you have the graph, however, how do you know what instructions are in each block? I can iterate through the graph with iterblocks, but how do I get information from each block? I want to analyze the information in each block to do code analysis for python. Can anyone help me with this? With thanks, Brian From tismer at stackless.com Thu Apr 13 04:39:41 2006 From: tismer at stackless.com (Christian Tismer) Date: Thu Apr 13 04:40:49 2006 Subject: [pypy-dev] Information from the FlowObjSpace In-Reply-To: References: Message-ID: <443DB9ED.8020406@stackless.com> Brian C. Lum wrote: > Dear Pypy developers: > > I have gone through the source code for the FlowObjSpace in > pypy.objspace.flow.objspace, but I am confused how to traverse through > the blocks or obtain the information for each blocks in the graph. From > my understanding: > > space = FlowObjSpace() > graph = space.build_flow(func) > > Once you have the graph, however, how do you know what instructions are > in each block? I can iterate through the graph with iterblocks, but how > do I get information from each block? > > I want to analyze the information in each block to do code analysis for > python. Can anyone help me with this? First of all, flowing doesn't work for full CPython. You need to use the RPython subset (see http://codespeak.net/pypy/dist/pypy/doc/coding-guide.html#restricted-python ) Then, if you have a block, you can iterate over block.operations which is a list, and so on. See pypy/objspace/flow/model.py ciao - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From bclum at cs.ucsd.edu Thu Apr 13 08:14:50 2006 From: bclum at cs.ucsd.edu (Brian C. Lum) Date: Thu Apr 13 08:15:33 2006 Subject: [pypy-dev] Information from the FlowObjSpace In-Reply-To: <443DB9ED.8020406@stackless.com> References: <443DB9ED.8020406@stackless.com> Message-ID: Hi, Thank you for the reply. I have looked at model.py and I can make a FunctionGraph. However, I am still not sure how to use it. space = FlowObjSpace() graph = space.build_flow(test.is_perfect_number) fun = FunctionGraph(test.is_perfect_number, graph.startblock) That is how I create an instance of FunctionGraph, and I can still make the blocks into an iterator, but I still cannot see what is inside the blocks, i.e. what would be the code in the function graph. Basically, all I want is a flowgraph of the code so that I can try to perform some static analysis on it (I want to bound the length of lists at compile time). I cannot figure out how to access the instructions in each basic block from the flowgraph. Calling translate_as_module from pypy.translator.geninterplevel as described in http://codespeak.net/pypy/dist/pypy/doc/translation.html#example actually produces the SSA. I was hoping that I would be able to get the instructions of each block from the flowgraph like in the example. Brian On Wed, 12 Apr 2006, Christian Tismer wrote: > Brian C. Lum wrote: >> Dear Pypy developers: >> >> I have gone through the source code for the FlowObjSpace in >> pypy.objspace.flow.objspace, but I am confused how to traverse through the >> blocks or obtain the information for each blocks in the graph. From my >> understanding: >> >> space = FlowObjSpace() >> graph = space.build_flow(func) >> >> Once you have the graph, however, how do you know what instructions are in >> each block? I can iterate through the graph with iterblocks, but how do I >> get information from each block? >> >> I want to analyze the information in each block to do code analysis for >> python. Can anyone help me with this? > > First of all, flowing doesn't work for full CPython. You need > to use the RPython subset (see > http://codespeak.net/pypy/dist/pypy/doc/coding-guide.html#restricted-python > ) > > Then, if you have a block, you can iterate over block.operations > which is a list, and so on. See pypy/objspace/flow/model.py > > ciao - chris > From mwh at python.net Thu Apr 13 11:17:56 2006 From: mwh at python.net (Michael Hudson) Date: Thu Apr 13 11:18:50 2006 Subject: [pypy-dev] Re: Information from the FlowObjSpace References: <443DB9ED.8020406@stackless.com> Message-ID: <2m3bghn8aj.fsf@starship.python.net> "Brian C. Lum" writes: > Hi, > > Thank you for the reply. I have looked at model.py and I can make a > FunctionGraph. However, I am still not sure how to use it. > > space = FlowObjSpace() > graph = space.build_flow(test.is_perfect_number) > fun = FunctionGraph(test.is_perfect_number, graph.startblock) > > That is how I create an instance of FunctionGraph, and I can still > make the blocks into an iterator, but I still cannot see what is > inside the blocks, i.e. what would be the code in the function graph. > > Basically, all I want is a flowgraph of the code so that I can try to > perform some static analysis on it (I want to bound the length of > lists at compile time). I cannot figure out how to access the > instructions in each basic block from the flowgraph. > for block in graph.iterblocks(): for op in block.operations: print op Cheers, mwh -- In the 1950s and 60s there was a regular brain drain of young Australians from the cities to London, but it was because of money, culture and opportunity, not spiders. -- Al Grant, ucam.chat, from Owen Dunn's review of the year From hpk at trillke.net Thu Apr 13 15:38:39 2006 From: hpk at trillke.net (holger krekel) Date: Thu Apr 13 15:44:03 2006 Subject: [pypy-dev] security prototype & workshop plans/ideas Message-ID: <20060413133839.GF10825@solar.trillke> Hi folks, from the EU side of things there is the plan to organize a security workshop and implement security features within PyPy. I now talked to my IBM research lab contact in Zuerich, and we discussed a few possibilities for such a workshop and the prototype, in particular two possibilities: - data tagging or "label control", or more generally attaching (security) metainformations to a python object and having those propagate through the program automatically. See e.g. http://www.pmg.lcs.mit.edu/papers/myers-popl99.pdf for some more information in this direction. This is especially interesting for PyPy as - i think - we can probably do better by e.g. also allowing to tag primitive types (JFlow doesn't do that, i think) and allowing things to happen more dynamically at runtime. - sandboxing: allowing to execute (user-supplied/induced) code and preventing it from accessing critical resources, e.g. system IO. Label control could be used for tagging e.g. user-level input with the "untrusted" label and then protecting certain functions to require trusted input (e.g. database/file modifications). Then, there could be explicit untrusted_to_trusted() conversions, turning an untrusted input into a trusted output. This would allow to concisely localise how user-supplied/untrusted input is parsed and checked. I also guess we could also go for a more general metadata-tagging approach which might be useful in more than just the security context. This are just initial ideas - there are arbitrary other possibilities (somewhere care to discuss the "E" lang here?), or are there not? :) The challenge is to find an interesting mechanism that elegantly enables such approaches - which should be the topic of our upcoming security prototype and workshop. Sandboxing and/or e.g. limiting RAM/CPU usage are also interesting to the Python world but probably are a rather orthogonal matter. Regarding the timing of the workshop it seems that something like October this year fits best. It depends on our actual plan and availability/selection of workshop participants if we rather should try to meet in Zuerich or in New York. We can also try to first arrange a discussion with them and then a meeting for presenting the results but that may be a bit much for us (we are not really lacking challenges currently :). I am posting here on pypy-dev (rather than just to selected pypy developers) because others may be interested, have comments, suggestions or might think about contributing. Security is certainly not the central topic of PyPy but our design should make it considerably easier to implement strong security features. Hum, and i guess that it's not impossible that the project might for contributors come up with funding for travels at least. best, holger From lac at strakt.com Thu Apr 13 20:07:59 2006 From: lac at strakt.com (Laura Creighton) Date: Thu Apr 13 20:08:46 2006 Subject: [pypy-dev] Re: [Edu-sig] Visual Programming in Python? In-Reply-To: Message from "Douglas S. Blank" of "Thu, 13 Apr 2006 12:56:12 EDT." <1144947372.13112.114.camel@mightymouse.brynmawr.edu> References: <1144947372.13112.114.camel@mightymouse.brynmawr.edu> Message-ID: <200604131807.k3DI7xU3017413@theraft.strakt.com> This just showed up on edu-sig. Is the translator and viewer separate enough that we could just send it to this person? Laura In a message of Thu, 13 Apr 2006 12:56:12 EDT, "Douglas S. Blank" writes: >Python edu-sig, > >I've been thinking about a visual programming/flowchart interface for >Python and was wondering if anyone knows of such a project. I am imaging >a Tkinter canvas (initially) with which one can add blocks that >represent statements, branches, loops... everything. Further, I imaging >that this would save (and load) real Python code, so that you could suck >in raw code, it would get parsed, and shown as a flowchart. Maybe some >additional data would be stored in comments (zoom amount, >positions/colors/properties of particular boxes). Also, one could step >through the chart, block-by-block. > >I've seen some commercial (and open source/non-Python) products, but >they seem heavy and sluggish, as if a whole lot of processing is going >on behind the scenes. Is/would it really be that hard? > >Any pointers or comments appreciated, > >-Doug > >-- >Douglas S. Blank Computer Science >Assistant Professor Bryn Mawr College >(610)526-6501 http://cs.brynmawr.edu/~dblank > > >_______________________________________________ >Edu-sig mailing list >Edu-sig@python.org >http://mail.python.org/mailman/listinfo/edu-sig From anto.cuni at gmail.com Fri Apr 14 16:53:28 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Fri Apr 14 16:56:04 2006 Subject: [pypy-dev] Avoiding code duplication Message-ID: <443FB768.7060604@gmail.com> Hi all, in these days I'm completing the implementation of ootypesystem/rlist.py, but I've found that often I have to write code very similar to those in lltypesystem/rlist.py. Let's explain by an example; consider the ll_listindex function in both files: # lltypesystem def ll_listindex(lst, obj, eqfn): items = lst.ll_items() lng = lst.ll_length() j = 0 while j < lng: if eqfn is None: if items[j] == obj: return j else: if eqfn(items[j], obj): return j j += 1 raise ValueError # can't say 'list.index(x): x not in list' # ootypesystem def ll_listindex(lst, obj, eqfn): lng = lst.length() j = 0 while j < lng: if eqfn is None: if lst.getitem_nonneg(j) == obj: return j else: if eqfn(lst.getitem_nonneg(j), obj): return j j += 1 raise ValueError # can't say 'list.index(x): x not in list' As you can see they are quite similar, but I can't find an elegant way to merge them. There are two problems: 1) the names of some methods don't match (e.g., ll_length vs. length 2) the setitem/getitem interface is very different The first problem is easy to solve; it should be sufficient to rename the methods in ootype.List._GENERIC_METHODS. The same isn't true for the second problem; one possibility could be to pass some dummy placeholder as argument in the same style of dum_checkidx/dum_nocheck, but I guess the code will became a nightmare. Another solution could be to add an extra level of indirection but I guess this could bring to some efficiency penalty. Any idea? ciao Anto From arigo at tunes.org Fri Apr 14 17:22:27 2006 From: arigo at tunes.org (Armin Rigo) Date: Fri Apr 14 17:22:28 2006 Subject: [pypy-dev] Avoiding code duplication In-Reply-To: <443FB768.7060604@gmail.com> References: <443FB768.7060604@gmail.com> Message-ID: <20060414152227.GA29635@code0.codespeak.net> Hi Antonio, On Fri, Apr 14, 2006 at 04:53:28PM +0200, Antonio Cuni wrote: > # lltypesystem > def ll_listindex(lst, obj, eqfn): > items = lst.ll_items() > (...) > if items[j] == obj: > > # ootypesystem > def ll_listindex(lst, obj, eqfn): > (...) > if lst.getitem_nonneg(j) == obj: > Another solution could be to add an extra level of indirection but I > guess this could bring to some efficiency penalty. I think this is kind-of-reasonable. The ADT method approach of the lltypesystem was introduced late during the development of the rtyper; by now, it would be reasonable to define common method names between the ADT methods of the lltypesystem and the GENERIC_METHODS of the ootypesystem. I am unsure about the performance penalty. The current version of many ll helpers, for example, read the 'items' pointer only once and reuse it; if this gets replaced by ADT methods like 'getitem_nonneg()', it means that althought the call is probably inlined there is still the overhead of reading 'items' through each iteration in the list. Who knows, maybe C compilers will notice and move the read out of the loop. Just give it a try on a small example like ll_listindex(), I guess... A different comment: as you mentioned on IRC it would be nice if the back-end could choose which methods it implements natively. At one point there was the idea that maybe the 'oopspec' attributes that started to show up in lltypesystem/rlist.py (used by the JIT only) could be useful in this respect. If I remember correctly, the idea didn't work out because of the different 'lowleveltype' needed, and the difference in the interface. Merging the ADT method names of lltyped lists and the GENERIC_METHODS of ootyped lists could be a step in this direction again. The interesting point is that each oo back-end could then choose to special-case the ll_xxx() functions with the oopspecs that they recognize, and just translate the other ones normally. (The ll back-ends always translate them all.) A bientot, Armin. From pedronis at strakt.com Fri Apr 14 17:52:34 2006 From: pedronis at strakt.com (Samuele Pedroni) Date: Fri Apr 14 17:53:27 2006 Subject: [pypy-dev] Avoiding code duplication In-Reply-To: <20060414152227.GA29635@code0.codespeak.net> References: <443FB768.7060604@gmail.com> <20060414152227.GA29635@code0.codespeak.net> Message-ID: <443FC542.3090707@strakt.com> Armin Rigo wrote: >Hi Antonio, > >On Fri, Apr 14, 2006 at 04:53:28PM +0200, Antonio Cuni wrote: > > >># lltypesystem >>def ll_listindex(lst, obj, eqfn): >> items = lst.ll_items() >>(...) >> if items[j] == obj: >> >># ootypesystem >>def ll_listindex(lst, obj, eqfn): >>(...) >> if lst.getitem_nonneg(j) == obj: >> >> > > > >>Another solution could be to add an extra level of indirection but I >>guess this could bring to some efficiency penalty. >> >> > >I think this is kind-of-reasonable. The ADT method approach of the >lltypesystem was introduced late during the development of the rtyper; >by now, it would be reasonable to define common method names between the >ADT methods of the lltypesystem and the GENERIC_METHODS of the >ootypesystem. > >I am unsure about the performance penalty. The current version of many >ll helpers, for example, read the 'items' pointer only once and reuse >it; if this gets replaced by ADT methods like 'getitem_nonneg()', it >means that althought the call is probably inlined there is still the >overhead of reading 'items' through each iteration in the list. Who >knows, maybe C compilers will notice and move the read out of the loop. >Just give it a try on a small example like ll_listindex(), I guess... > >A different comment: as you mentioned on IRC it would be nice if the >back-end could choose which methods it implements natively. At one >point there was the idea that maybe the 'oopspec' attributes that >started to show up in lltypesystem/rlist.py (used by the JIT only) could >be useful in this respect. If I remember correctly, the idea didn't >work out because of the different 'lowleveltype' needed, > yes, the problem was not re-using code as such, indeed it should not be hard to write generic code that can be shared using adt methods etc. The problem was trying to share the same type between ootype and lltype, Instances are not Ptrs and don't even behave similarly enough, this is where trying to reuse completely the lltypesystem rlist and rtuple code broke. > and the >difference in the interface. Merging the ADT method names of lltyped >lists and the GENERIC_METHODS of ootyped lists could be a step in this >direction again. The interesting point is that each oo back-end could >then choose to special-case the ll_xxx() functions with the oopspecs >that they recognize, and just translate the other ones normally. (The >ll back-ends always translate them all.) > > >A bientot, > >Armin. >_______________________________________________ >pypy-dev@codespeak.net >http://codespeak.net/mailman/listinfo/pypy-dev > > From hpk at trillke.net Fri Apr 14 19:44:12 2006 From: hpk at trillke.net (holger krekel) Date: Fri Apr 14 19:49:44 2006 Subject: [pypy-dev] Mixed modules for both PyPy and CPython In-Reply-To: <20060412114819.GA18540@code0.codespeak.net> References: <20060412102948.GA9512@code0.codespeak.net> <20060412114819.GA18540@code0.codespeak.net> Message-ID: <20060414174412.GL10825@solar.trillke> Hi Armin, hi all, On Wed, Apr 12, 2006 at 13:48 +0200, Armin Rigo wrote: > The purpose of the whole discussion was about how to have a "write-once, > run-everywhere" approach of developing an extension module a single time > and then compiling it for either CPython or PyPy. This source code > should not need to know if it will be compiled for PyPy or CPython (or > even just run on top of the CPython interpreter for testing). > > ... > Mixed modules provide explicit level separation, > whereas the alternative is to rely on the annotator to separate the > levels. Hum, i wonder how strongly opposed these explicit versus implicit level separation models need to be. Is it not possible to support a programming model that can mostly avoid knowing about interpreter versus application level distinction without extending/refactoring the annotator? Maybe we could identify some interesting interaction use cases and try to support them explicitely like e.g. publishing an rpython level class in CPython providing glue code between the rpython class and its CPython representation? However, having an underlying interp/app separation with 'space', wrapped objects and exceptions, still seems like a very good (proven) foundation on which to build such glue code. (Btw, i wouldn't mind if such glue code would not allow all possible interactions - our primary goal is not to provide seemless integration with CPython here). And for the planned June PyPy release we likely want to have the "explicit" approach nicely working before experimenting with where we can go from there, right? best, holger From nhaldimann at gmx.ch Fri Apr 14 21:58:25 2006 From: nhaldimann at gmx.ch (Niklaus Haldimann) Date: Fri Apr 14 21:59:02 2006 Subject: [pypy-dev] Summer of Code 2006 Message-ID: <443FFEE1.5060007@gmx.ch> Hi there Google is doing Summer of Code again this year: http://code.google.com/soc/ It would be possible to enter PyPy directly as a mentoring organization this time, instead of going through the PSF. Last year, student slots were given to mentoring organizations proportional to the number of applications. If there are enough applications for PyPy proper that might bring more students on board than taking slots from the PSF pool as last year (but maybe PyPy's influence in the PSF is big enough, and this doesn't really matter). What is the status of the Summer of PyPy idea, btw? Nik From hpk at trillke.net Fri Apr 14 21:57:47 2006 From: hpk at trillke.net (holger krekel) Date: Fri Apr 14 22:03:19 2006 Subject: [pypy-dev] Summer of Code 2006 In-Reply-To: <443FFEE1.5060007@gmx.ch> References: <443FFEE1.5060007@gmx.ch> Message-ID: <20060414195747.GM10825@solar.trillke> Hi Niklaus, On Fri, Apr 14, 2006 at 21:58 +0200, Niklaus Haldimann wrote: > Google is doing Summer of Code again this year: http://code.google.com/soc/ > > It would be possible to enter PyPy directly as a mentoring organization > this time, instead of going through the PSF. Last year, student slots > were given to mentoring organizations proportional to the number of > applications. If there are enough applications for PyPy proper that > might bring more students on board than taking slots from the PSF pool > as last year (but maybe PyPy's influence in the PSF is big enough, and > this doesn't really matter). sounds good. > What is the status of the Summer of PyPy idea, btw? it hasn't been pursued much lately because it ran concurrent to many other issues we (and i) needed to sort out. There was considerable interest, though, something like 10-15 people mailed me and the list here. best, holger From hpk at trillke.net Sat Apr 15 09:29:55 2006 From: hpk at trillke.net (holger krekel) Date: Sat Apr 15 09:35:32 2006 Subject: [pypy-dev] Re: Mixed modules for both PyPy and CPython In-Reply-To: References: <20060412102948.GA9512@code0.codespeak.net> <348899050604121213w1f01c58pb65b4e70394f6deb@mail.gmail.com> Message-ID: <20060415072955.GO10825@solar.trillke> Hi VanL, On Wed, Apr 12, 2006 at 13:45 -0600, VanL wrote: > Carl Friedrich Bolz wrote: > > > Careful. "Restricted Python" (or RPython) means something very > > specific in PyPy lingo. Restricted Python means that you stick to some > > restrictions in your coding style to make type inference possible. See > > > > > http://codespeak.net/pypy/dist/pypy/doc/coding-guide.html#restricted-python > > > > > > Sorry - I knew that. I just didn't make the connection from one side of > my brain to another. Maybe "secure python" (spy)? just as a side note: "secure" is a very vague term, technically (and politically). The professor i learned with, warned against talking about security without specifying against which kind of attacks and about which scenarios one is talking about it. Did you see my security related posting a few days ago, btw? > What made me think this would work are a couple of comments made here > and on the py3k list. > First, Armin (about a year ago, I think)made some comments about the > objectspace abstraction allowing a "Remote Object Space" and allowing > different object spaces to participate in the evaluation of code. Right, that is still an open topic, a bit touched by the parallel discussion on this list of integrating PyPy (and RPython) with CPython. > Second, comments on py3k list indicated that secure python is difficult > because of a) introspection, b) type inference, and c) GIL acquisition. Hum, this list looks a bit weird to me. Could you state what the actual attacks are for which security measures are discussed? Or which use cases are people on py3k having in mind? cheers & thanks, holger From anto.cuni at gmail.com Sat Apr 15 09:46:16 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Sat Apr 15 09:48:49 2006 Subject: [pypy-dev] Summer of Code 2006 In-Reply-To: <443FFEE1.5060007@gmx.ch> References: <443FFEE1.5060007@gmx.ch> Message-ID: <4440A4C8.7030207@gmail.com> Niklaus Haldimann wrote: > Hi there > > Google is doing Summer of Code again this year: http://code.google.com/soc/ > > It would be possible to enter PyPy directly as a mentoring organization > this time, instead of going through the PSF. Last year, student slots > were given to mentoring organizations proportional to the number of > applications. If there are enough applications for PyPy proper that > might bring more students on board than taking slots from the PSF pool > as last year (but maybe PyPy's influence in the PSF is big enough, and > this doesn't really matter). That's a good news! I'd like to participate to soc for doing pypy's stuffs, I hope I'll able to apply (and to win :-)). ciao Anto From anto.cuni at gmail.com Sat Apr 15 12:59:07 2006 From: anto.cuni at gmail.com (Antonio Cuni) Date: Sat Apr 15 13:01:49 2006 Subject: [pypy-dev] Avoiding code duplication In-Reply-To: <20060414152227.GA29635@code0.codespeak.net> References: <443FB768.7060604@gmail.com> <20060414152227.GA29635@code0.codespeak.net> Message-ID: <4440D1FB.8000800@gmail.com> Armin Rigo wrote: > > I think this is kind-of-reasonable. The ADT method approach of the > lltypesystem was introduced late during the development of the rtyper; > by now, it would be reasonable to define common method names between the > ADT methods of the lltypesystem and the GENERIC_METHODS of the > ootypesystem. > > I am unsure about the performance penalty. The current version of many > ll helpers, for example, read the 'items' pointer only once and reuse > it; if this gets replaced by ADT methods like 'getitem_nonneg()', it > means that althought the call is probably inlined there is still the > overhead of reading 'items' through each iteration in the list. Who > knows, maybe C compilers will notice and move the read out of the loop. > Just give it a try on a small example like ll_listindex(), I guess... Well, as we decided on #pypy I've changed the ADT interface. As I wrote in the commit log: """ The interface of ListRepr and FixedSizeListRepr has changed: two accessor methods has been added: ll_getitem_fast and ll_setitem_fast. They should be used instead of the ll_items()[index] idiom: that way when ootypesystem's list will support that interface we will able to write function useable with both typesystem with no modification. The various ll_* helper function has been adapted to use the new interface. Moreover function that accessed directly to the "l.length" field has been changed to call the "ll_length()" method instead, for the same reasons as above. """ The next step is to rename ootypesystem's list _GENERIC_METHODS to match the ADT methods in lltypesystem's list, then we could try to share most of ll_* function that currently belongs only to lltypesystem/rlist.py. I hope I will do it tomorrow. > A different comment: as you mentioned on IRC it would be nice if the > back-end could choose which methods it implements natively. At one > point there was the idea that maybe the 'oopspec' attributes that > started to show up in lltypesystem/rlist.py (used by the JIT only) could > be useful in this respect. If I remember correctly, the idea didn't > work out because of the different 'lowleveltype' needed, and the > difference in the interface. Merging the ADT method names of lltyped > lists and the GENERIC_METHODS of ootyped lists could be a step in this > direction again. The interesting point is that each oo back-end could > then choose to special-case the ll_xxx() functions with the oopspecs > that they recognize, and just translate the other ones normally. (The > ll back-ends always translate them all.) I saw that 'oopspec' attributes, but I didn't understand the exact semantic; your proposal sounds reasonable to me: if I can figure out correctly this way the typesystem specific code would be reduced to the minimum and will help to port other Repr such as rdict to ootypesystem, too. I'll investigate a bit in this direction as soon as I can. good Easter to all, ciao Anto From arigo at tunes.org Sat Apr 15 14:59:39 2006 From: arigo at tunes.org (Armin Rigo) Date: Sat Apr 15 14:59:41 2006 Subject: [pypy-dev] Avoiding code duplication In-Reply-To: <4440D1FB.8000800@gmail.com> References: <443FB768.7060604@gmail.com> <20060414152227.GA29635@code0.codespeak.net> <4440D1FB.8000800@gmail.com> Message-ID: <20060415125939.GA29013@code0.codespeak.net> Hi Antonio, On Sat, Apr 15, 2006 at 12:59:07PM +0200, Antonio Cuni wrote: > I saw that 'oopspec' attributes, but I didn't understand the exact > semantic; The oopspec string tells what is the "abstract" list operation that this particular ll_*() function implement. For example: def ll_prepend(l, newitem): ... ll_prepend.oopspec = 'list.insert(l, 0, newitem)' means that ll_prepend() is equivalent to an insert with the index set to zero. In the stirng, the pseudo-arguments between the ( ) are either real argument names of the ll_ function, or constants. So for example, if a backend has got its own way to implement the insert() calls in general, it could figure out from the oopspec that the ll_prepend() helper can be replaced by a custom stub invoking the backend's own version of insertion with an index of 0. That's essentially what the JIT does -- see handle_highlevel_operation() in jit/hintannotator/model.py. A bientot, Armin. From arigo at tunes.org Sat Apr 15 15:09:09 2006 From: arigo at tunes.org (Armin Rigo) Date: Sat Apr 15 15:09:10 2006 Subject: [pypy-dev] Mixed modules for both PyPy and CPython In-Reply-To: <20060414174412.GL10825@solar.trillke> References: <20060412102948.GA9512@code0.codespeak.net> <20060412114819.GA18540@code0.codespeak.net> <20060414174412.GL10825@solar.trillke> Message-ID: <20060415130909.GB29013@code0.codespeak.net> Hi Holger, On Fri, Apr 14, 2006 at 07:44:12PM +0200, holger krekel wrote: > Hum, i wonder how strongly opposed these explicit versus implicit > level separation models need to be. Yes, you're right here. It's mostly about what we need to do next: we must choose one of the two models and develop it enough, until it becomes useful for PyPy and CPython alike. We could possibly do both models in parallel, but I'm not sure it's the best way forward at this point. > Is it not possible to support a > programming model that can mostly avoid knowing about interpreter versus > application level distinction without extending/refactoring the annotator? Sure, no-one thinks about refactoring the annotator. The implicit model already works, by using the existing support for SomeObjects, completed by Christian over the time. It's a bit hackish, though, and we'll definitely need ways to control where SomeObjects are expected or not. At the moment, what makes me reluctant to continue with the implicit model are two other issues: on the one hand, it's unclear how it would work for PyPy (it works for CPython only); and there are many language design issues ahead that I'd rather avoid for the time being. > "explicit" approach nicely working before experimenting with where > we can go from there, right? Yes, exactly my opinion. > (Btw, i wouldn't > mind if such glue code would not allow all possible interactions - > our primary goal is not to provide seemless integration with CPython here). I'm not too worried about this. Our mixed-module model already supports mostly any kind of interaction, including defining new types with properties and overridden operations. The path to support the same for CPython extension modules is more or less clear, and very incremental. A bientot, Armin. From hpk at trillke.net Sat Apr 15 15:10:05 2006 From: hpk at trillke.net (holger krekel) Date: Sat Apr 15 15:15:42 2006 Subject: [pypy-dev] ext module testing modes In-Reply-To: <20060415110215.DF0CF100AC@code0.codespeak.net> References: <20060415110215.DF0CF100AC@code0.codespeak.net> Message-ID: <20060415131005.GA2505@solar.trillke> Hi Armin, On Sat, Apr 15, 2006 at 13:02 +0200, arigo@codespeak.net wrote: > Author: arigo > Date: Sat Apr 15 13:02:14 2006 > New Revision: 25852 > > Added: > pypy/dist/pypy/rpython/rctypes/socketmodule/test_addr.py (contents, props changed) > Modified: > pypy/dist/pypy/rpython/rctypes/socketmodule/_socket.py > pypy/dist/pypy/rpython/rctypes/socketmodule/ctypes_socket.py > Log: > Very incomplete implementation of getaddrinfo(), with a test > (only works on on-line machines so far). The idea is that rctypes > should now support all ctypes constructions that were necessary. > I will start a regular mixed-module _socket based on this, but > first we need to figure out how to best test mixed-modules based > on ctypes -- ideally, they should be testable and compilable > without the rest of the PyPy interpreter... regarding py.test support: I think eventually we may have the following testing distinctions regarding ext modules: - test mixed module with std objspace (running on top of cpython) (current default) - test mixed module with cpy-objspace connected to CPython runtime via rctypes - test mixed module on top of pypy-c I guess the second testing mode could be specified by "--objspace=cpy" and for the third we may simply allow to specify "--appdirect" which would trigger application level tests to run dir