Appendix Z1: History and background =================================== Why Python? What is Python? --------------------------- Python is a portable, interpreted, object-oriented Very-High Level Language (VHLL). Python is believed to have nearly two hundred thosand active developers, which makes it one of the top ten most popular programming languages in the world. Some other languages, specifically C, C++, Java, Perl, and Visual Basic, have even larger user bases but they are either proprietary or rather low-level languages. On the other hand, the languages which most excite the Computer Science Research community -- Self, Lisp, Haskell, Limbo, ML, and so on -- are nowhere on this list, yet they are the targets of most European academic research and innovation. Thus European economic competitiveness suffers. The innovative research lives in academia, trapped in languages that are rarely used for commercial development. Instead, commercial companies have turned to offerings from closed-source American companies. This is risky. American companies must act in their own business interests, not in their customers interests, whenever they conflict. Perhaps even more important, proprietary licensing policies make it expensive and sometimes impossible to adapt these languages to specialized hardware. Python is a successful attempt to built a commercially robust language (not an academic toy) that is an Open Source alternative. European SMEs are moving to Free/Open Source platforms ------------------------------------------------------ In the face of these threats and flexibility problems, European SMEs are moving to Free/Open Source languages such as Python. This gives the Python language developers a larger user-base, with commercial concerns not seen in academia. In order to properly present this concerns, and to leverage business acumen and cooporate on shared goals, a group of SME's who rely on the Python programming language came together to form the Python Business Forum (www.python-in-business.org), at EuroPython, the European Python Community Conference (www.europython.org). The PBF is modelled after Lantbrukarnas Riksförbund - the non-profit Federation of Swedish Farmers, and is registered as a Non-Profit in Sweden. Advancing the Python platform ----------------------------- While each SME member of the Python Business Forum has sufficient faith in the Python programming language to use it for the development of its own projects, it was agreed that there are defects in the current implementation of the language. The two most often cited ones are that the current implementation is too large for embedded applications and applications designed for handhelds, and that the interpreter itself ran too slowly in some situations. The developers of embedded systems would like a highly productive language with a smaller memory footprint. They would like to strip out everything which they do not need and put in special extensions in an easy way. Today, this is hard to do in any language, and Python was not implemented with this goal in mind. Another goal which was not paramount in Python's design and implementation was execution speed. Python is a dynamically-typed, late-binding, interpreted language. While this proved to provide extremely productive development environments, execution speed sometimes is not fast enough. Today, optimisation of high-level languages must be done at run-time, and is notoriously more difficult to optimise than statically typed, early-binding compiled languages such as C, C++ or Java. Now a number of people and factors played together to start what is now one of the most promising very-high-level-language projects. Some high-profile research -------------------------- Some researchers who worked with Python were pondering writing an implementation of Python in Python itself. This group included Armin Rigo, author of Psyco, a specialising Just-In-Time compiler for Python. He is intimately familiar with both Python internals and advanced research in compilers and runtime systems, and saw a Python implementation in Python itself as a chance to put the two fields together. It is useful to quote the goals of Psyco_ in full:: My goal in programming Psyco is to contribute to reduce the following wide gap between academic computer science and industrial programming tools. While the former develops a number of programming languages with very cool semantics and features, the latter stick with low-level languages principally for performance reasons, on the ground that the higher the level of a language, the slower it is. Although clearly justified in practise, this belief is theoretically false, and even completely inverted --- for large, evolving systems like a whole operating system and its applications, high-level programming can deliver much higher performances. The new class of languages called 'dynamic scripting languages', of which Python is an example, is semantically close to long-studied languages like Lisp. The constraints behind their designs are however different: some high-level languages can be relatively well statically compiled, we can do some type inference, and so on, whereas with Python it is much harder --- the design goal was different. We now have powerful enough machines to stick with interpretation for a number of applications. This, of course, contributes to the common belief that high-level languages are terribly slow. Psyco is both an academic and an industrial project. It is an academic experiment testing some new techniques in the field of on-line specialization. It develops an industrially useful performance benefit for Python. And first of all it is a modest step towards: High-level languages are faster than low-level ones! Although Armin Rigo proved with his project that higher level languages can sometimes be optimized to be as fast or faster than C, he was very limited by the fact that Python is itself implemented in C like almost all other languages in industrial use today. There was no real-life project who tried to go all the 'optimize high-level down to machine-code' way. Christian Tismer with his 'Stackless' project had already come to a similar conclusion from a more industrial viewpoint in that **it is difficult to advance language technology while relying on a large C-code base**. Some mailing list discussions ----------------------------- On the German Python mailing list an independent discussion evolved where developers and academics were pondering about the possibility of developing a 'minimal' Python implemented in Python itself. Most noticeably Christian Tismer author of an industrial-use extension ('Stackless') of the Python language noted in a postscript to one of his mails that having a minimized version of Python could provide a new base to advance the language. Nevertheless, his extensions to the language already proved to be useful for companies who needed a way to have millions of active objects and he had a branch of CPython to make this possible. Some organizational experience ------------------------------ Meanwhile Holger Krekel had joined the Python community in 2001. For some years he had designed the architecture for platforms and consulted for CEO's of some large banking centers in Europe. While participating in two 'coding Sprints' of the Zope3 web-platform (the to-be successor of the successful Zope web-platform) he realized that agile 'Sprints' in combination with the rapid development language Python provide an extremely productive way of communicating about and coding complex projects where traditional, slow-moving methods often fail. At a Sprint a group of people assemble to write code and practice Agile software methodological techniques, such as Pair-Programming and writing unit-tests before implementing the actual functionality. Not only is this a lot of fun, but it is a way to transmit knowledge and understanding throughout the group. Holger Krekel, seeing the opportunity to initiate the PyPy project with Armin Rigo and Christian Tismer organized the first one-week meeting, the 'Sprint towards a minimal Python'. Soon many interested developers joined and intense academical and practical planning ensued. The Sprint took place at 'Trillke-Gut', a castle-like building in Hildesheim, Germany, bringing together some key developers, among them Michael Hudson, the release manager of version 2.2.1 of Python. Here is his german mail_ (a reply to Christian Tismer) that started this now rapidly evolving project:: Gerson Kurz wrote: > [interessante analysen, die ich zum grossteil nachvollziehen kann] > [christian tismer] > > p.s.: Gerson, ganz vom aktuellen Python abgesehen fände > > ich es durchaus interessant, ein "Ptn" zu machen: Ein > > total abgespecktes Minimal-Python mit weniger Features > > als 1.5.2, sozusagen spartanisch aber aus heutiger Sicht. > > Hey, genau das würde ich mir auch wünschen! Ich wäre auch sofort > bereit, > dabei mitzuhelfen, so es mir den Zeitlich und von meinen Möglichkeiten > gesehen her möglich ist. (Ich fahre am Samstag erstmal für zwei Wochen > in > den Urlaub, aber dann...) > > Hast du schon nähere Pläne dazu? Genauer, hast du schonmal überlegt, > welche > Python-Features Ptn-würdig wären, und welche nicht? bin auch interessiert. wenn noch Armin Rigo (PSYCO) ins boot kaeme, dann waere es noch besser. Ich habe auf jeden Fall auch interesse, zumal ich zwischen weinachten endlich mal in die python-internals (compile.c & ceval.c) eingestiegen bin. Christian (auf python-dev und c.l.py) scheint ja auch die Philosophie zu verfolgen, python moeglichst viel in python zu implementieren und sich von C-Ballast zu befreien. Armin will gar den interpreter neu in python schreiben. Um es mal zu konkretisieren, biete ich an, einen "Sprint" auszurichten. Das waeren drei tage, an denen wir die fundamentals eines MiniPython entwickeln und ins Netz stellen. Abgesehen von der Vorbereitung kann ich anbieten, dass wir mit netzconnection und voller verpflegung einen 120qm saal mit drei bis vier beamern zur verfuegung haben. und zwar in diesem gebauede: http://www.trillke.net/images/trillke_schnee.png das liegt in hildesheim, also 30 km von hannover. falls euch das zu schnell bzw. zu konkret ist, dann koennen wir auch abstraker weiterdiskutieren :-) liebe gruesse und ebenfalls ein schoenes neues jahr, holger Leveraging open-source development tools ---------------------------------------- From the beginning, the PyPy developers were committed using and integrating the most promising open-source technologies. Jens-Uwe Mager, the retired CTO from Helios (http://www.helios.de) attended the Sprint and helped set up a state-of-the-art open-source development environment. With his 12-year experience of setting up and leading a SME-company which is one of the worldwide leaders in print-preprocessing technology he helped organise the development and net-connectivity for the various web services needed by the PyPy developers. Research, pragmatism and industry experience combined ----------------------------------------------------- This combination of research, applicability and industrial experience was precisely what the Python Business Forum members had been looking for. Academics, developers and practitioners conversant in the latest in language and platform architecture who were interested in producing the Python interpreter which they needed. Laura Creighton and Jacob Hallén from the PBF attended the sprint and began participating in the project. As Board Memebers of AB Strakt, a Swedish company developing a novel Workflow application, entirely in Python, they were well aware that their main product would soon be pushing up against the limitations of the current version of the language. Their question was - could this team actually deliver on a flexible, high-performance version of the language which would not have these limitations? The approach taken was original. Original solutions need to be tested. The group decided soon to test their proof of concept by developing a working prototype and to test their ability to work together. They would meet for Sprints in their own private time, and work on the prototype. After the first Sprint at Trillke-Gut in Hildesheim, Jacob Hallén and Laura Creighton organized the next Sprint at AB Strakt in Gothenburg, Sweden. At this Sprint, Samuele Pedroni, the lead developer of Jython (a tight industrial-use integration of Java and Python), joined the project not only because such a project was relevant to his own computer science research interests, but also to see if PyPy would help with his own java-integration project. Prototyping went quickly, with the work of all these experienced people, and by the end of the week you could already run simple Python programs within PyPy. PyPy had gone from being 'a nice idea' to 'something we knew we could do'. The third sprint was again organized by interested developers at the University in Louvain-La-Neuve, Belgium and held June 20-24. We invited Guido van Rossum, the inventor of Python to attend. He not only attended the Belgium sprint but announced a few days later at the EuroPython conference that PyPy had a high priority on his list of 'dreams he hoped would come true' and he enjoyed Sprinting with us a lot. By the end of the third one-week sprint at the University in Louvain-La-Neuve, the PyPy project had already produced a fully working prototype. There existed a working interpreter and standard implementation of 90% of the Python core types, as well as advanced language features such as nested scopes, generators and metaclasses. Armin Rigo and Guido van Rossum had started work on a type-inference engine as the foundation for the final missing step: generating a native machine-level version of Python from its high-level PyPy-implementation. Perhaps most importantly, we had the enthusiastic support of the Python community at the EuroPython conference. The fourth sprint in Berlin prototyped translation of Python functions to low-level code with the help of Pyrex, an intermediate layer. The architecture was refactored to implement abstract interpretation and the actual translation process separatly. With a combination of novel techniques, Object Spaces and Abstract Interpretation, it was clear that the general direction was viable and productive. No work had been done on actually optimizing the complete prototype into low-level static code so it still runs around 30,000 times slower than the existing CPython-implementation but this was expected from the start. Nevertheless, for a proof of concept, approximately six weeks work total for a group of about a dozen people, it was clearly a success. It was time to look for funding to bring the project to its full potential and to intensify research on several interesting areas. On June 17th, the 2nd Call of the Information Society Technologies [IST] Priority went out. Included in it was IST-2002-2.3.2.3 - Open development platforms for software and services. We believe that what we intend to do is a perfect fit for the goals of this call. .. _mail: http://starship.python.net/pipermail/python-de/2003q1/002925.html .. _Psyco: http://psyco.sourceforge.net/introduction.html