From anto.cuni at gmail.com Tue Jul 3 15:56:51 2007 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 03 Jul 2007 15:56:51 +0200 Subject: [pypy-dev] Alternative algorithm for attribute lookup Message-ID: <468A55A3.4080009@gmail.com> Hi all, I passed the last few days slowly experimenting with alternative/faster ways to do lookup of attributes. For simplicity, I didn't touch the interpreter but I simply wrote few rpython programs doing only the last part of the attribute lookup, i.e. accessing the underlying RPython dictionary. Moreover, I played only with dict of strings, since this is how objects are implemented when multidict is enabled. My first experiment was with precomputed hashes for keys; I introduced a new method get_fast on dictionaries, which accepts both the key and its hash value, so it does not need to compute it. Of course this is just a quick hack because programs using get_fast works only when translated and can't run on top of CPython, but it was the simplest way to experiment. You can find both the get_fast patch and the benchmark here: http://codespeak.net/svn/user/antocuni/lookup-hack/get_fast.patch http://codespeak.net/svn/user/antocuni/lookup-hack/targethashcache.py The interesting thing is the result of the benchmark: since RPython strings already cache their hash I didn't expect get_fast to be much faster than get, but the benchmark says the opposite; on my machine: antocuni at anto lookup-hack $ ./targethashcache-c 50000000 iterations get: 3.820000 seconds get_fast: 2.520000 seconds get_fast: 1.515873 times faster It is 50% faster. I really don't know why, maybe there is something wrong in the benchmark or maybe something is wrong with the hash cache of rpython strings. The second and more insteresting experiment is a bit more involved. The idea is that for most of Python objects, we can guess a set of "likely attributes" at compile time: - for modules, by inspecting its global namespace; - for classes, by inspecting the namespace of a class definition; - for instances, by looking for LOAD_FAST 0, STORE_ATTR x inside the methods of the class. Note that we don't pretend to catch all the possible attributes, just the most likely. The assumption is that it's very uncommon to have a "non-likely attribute" and that for most of the objects the set of the attributes at runtime is a subset of the likely attributes. The idea is to speed up consistently all the accesses to likely attributes, at the cost of a slighly slow down for accessing the non-likely ones. Once we have collected the set of likely attributes, we compute a perfect hash function for this set; for my experiment, I used an hacked version of this algorithm: http://www.amk.ca/files/python/perfect_hash.txt The perfect hash function looks like this, where N and G vary from set to set of likely attributes:: def perfect_hash(N, G, key): h = hash(key) # normal hash i = h & N-1 j = i+1 res = G[i] + G[j] res = res & N-1 return res Moreover, since h depends only on key we could also precomputed it at compile time, as we did for get_fast:: def perfect_hash(N, G, key, h): ... Finally, we insert a fast path for the common case in the lookup code:: def lookup(d, key, h): # h is the precomputed hash index = perfect_hash(d.N, d.G, key, h) entry = d.entries[index] if entry.is_valid() and entry.key is key: return entry.value else: # do a full lookup Note that in the fast path we use 'is' instead of == to compare the keys; since the W_Strings storing the names of the attributes are interned, this should be enough. Note also that G and N are stored on the dict (thus we can have different perfect hashes for different set of likely attributes, as we need). If my assumption about likely attributes is true, most of LOAD_ATTR will follow the fast path, resulting in a considerable speedup: if the object only uses likely attributes, there will never be a collision. Most important, this does not break anything, because in case the fast path is not followed (for example when we do {get,set}attr(obj, 'attr')), the full lookup will work as well. You can find the benchmark here (this doesn't need get_fast.patch): http://codespeak.net/svn/user/antocuni/lookup-hack/targetperfectdict.py For the test, I implemented a perfect_dict as an RPython class, thus it might be slower than what we can get if we implement it as a real rpython type on the style of r_dict; in particular, RPython does index-checking when calcultaing G[i] and G[j], but it's not necessary. Although perfect_dict is not as fast as possible, the result of the benchmark is very interesting: antocuni at anto lookup-hack $ ./targetperfectdict-c 50000000 iterations get: 3.740000 seconds pdict: 1.450000 seconds pdict: 2.579310 times faster What do you think about this idea? Is there any obvious bug/wrong assumption I can't catch? Do you think it might be worth of implementing it? (it would be a nice sprint topic). ciao Anto From scott+pypy-dev at scottdial.com Wed Jul 4 22:02:21 2007 From: scott+pypy-dev at scottdial.com (Scott Dial) Date: Wed, 04 Jul 2007 16:02:21 -0400 Subject: [pypy-dev] select/test_readable on Win32 blocks forever Message-ID: <468BFCCD.6040201@scottdial.com> Greetings, I haven't figured this one out. As the subject says, test_readable for the select module gets stuck forever. If I trust the history of the bot, then it appeared somewhere after r44639 and before r44646. I've looked through the revisions and nothing jumped out at me.. happy hunting to someone more aware of the changes. -Scott -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From wchunming at gmail.com Thu Jul 5 07:47:25 2007 From: wchunming at gmail.com (=?GB2312?B?zfW0usP3?=) Date: Thu, 5 Jul 2007 13:47:25 +0800 Subject: [pypy-dev] debug(read source code and test) pypy in pydev Message-ID: <64e4dfb0707042247n7818ecdfvd051a827abc29e3@mail.gmail.com> Hi, I am a newbie to python and pypy. I choose to read pypy source code to learn about python and python library. But when I try to run pypy in debug mode in pydev, I got the following problem: /// error message begin pydev debugger Traceback (most recent call last): File "F:\TOOLS\eclipse\plugins\org.python.pydev.debug_1.3.4\pysrc \pydevd.py", line 754, in debugger.run(setup['file'], None, None) File "F:\TOOLS\eclipse\plugins\org.python.pydev.debug_1.3.4\pysrc \pydevd.py", line 597, in run execfile(file, globals, locals) #execute the script File "E:\Java\pySamples\pypy\bin\py.py", line 14, in from pypy.tool import option File "E:\Java\pySamples\pypy\tool\option.py", line 4, in from pypy.config.pypyoption import get_pypy_config File "E:\Java\pySamples\pypy\config\pypyoption.py", line 2, in import py, os File "E:\Java\pySamples\pypy\bin\py.py", line 14, in from pypy.tool import option ImportError: cannot import name option Exception in thread pydevd.Writer (most likely raised during interpreter shutdown): Traceback (most recent call last): File "C:\Python25\lib\threading.py", line 460, in __bootstrap File "F:\TOOLS\eclipse\plugins\org.python.pydev.debug_1.3.4\pysrc \pydevd_comm.py", line 258, in run /// error message end I try to figure out why this exception or error come out. I found that py.py import option.py, and option.py import pypyoption.py, and pypyoption.py import py.py again. Is this the problem of pypy(I don't think so), or the problem of pydev debugger, or something else? why? Any comments or assistance that can light up the way are much appreciated.Thanks. Wang Chunming -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20070705/7ce79964/attachment.htm From amauryfa at gmail.com Thu Jul 5 14:41:16 2007 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Thu, 5 Jul 2007 14:41:16 +0200 Subject: [pypy-dev] debug(read source code and test) pypy in pydev In-Reply-To: References: <64e4dfb0707042247n7818ecdfvd051a827abc29e3@mail.gmail.com> Message-ID: Hello, Wang Chunming wrote: > Hi, > I am a newbie to python and pypy. I choose to read pypy source code > to learn about python and python library. But when I try to run pypy > in debug mode in pydev, I got the following problem: [Traceback] > > I try to figure out why this exception or error come out. I found > that py.py import option.py, and option.py import pypyoption.py, and > pypyoption.py import py.py again. Is this the problem of pypy(I don't > think so), or the problem of pydev debugger, or something else? why? > Any comments or assistance that can light up the way are much > appreciated.Thanks. There are two "py" involved here: - you are running the py.py script - the other is the py library: http://codespeak.net/py/dist/ Normally, this does not cause any problem, but Pydev seems to tweak the pythonpath and adds several entries... Try to set the current directory to E:\Java\pySamples\pypy\bin\ (In eclipse, open the "Run/Debug..." window, and select the "Arguments tab") Or you could just rename py.py to something else (runpy.py for example). Note that because of a python bug, you cannot use python 2.5 to debug pypy. (A strange interaction between generator destructors, the sys.settrace function and the current thread state. www.python.org/sf/1733973 shows a simple case, filed by the pydev developer) Python 2.4 seems to work correctly. Hope this helps, -- Amaury Forgeot d'Arc From amauryfa at gmail.com Thu Jul 5 15:28:21 2007 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Thu, 5 Jul 2007 15:28:21 +0200 Subject: [pypy-dev] select/test_readable on Win32 blocks forever In-Reply-To: <468BFCCD.6040201@scottdial.com> References: <468BFCCD.6040201@scottdial.com> Message-ID: Hello Scott, Scott Dial wrote: > Greetings, > > I haven't figured this one out. As the subject says, test_readable for > the select module gets stuck forever. If I trust the history of the bot, > then it appeared somewhere after r44639 and before r44646. I've looked > through the revisions and nothing jumped out at me.. happy hunting to > someone more aware of the changes. When I wrote these tests a few months ago, they passed on the two windows machines where I ran them (Win2k + python2.5 behind a firewall, and XP + python2.4 at home). They still pass without problem (except for some network operations forbidden by the firewall). Is it possible that the bot runs in a restricted environment? What is the configuration of this machine? I suggest to run the following test function, either alone, or somewhere in the test suite Could you put it near the top of pypy/module/select/test/test_select.py ? It has nothing to do with pypy, but repeats the same test at the python level. # Sanity check of the standard Python interpreter def test_CPython_select(): import thread, socket, select sock = socket.socket() try_ports = [1023] + range(20000, 30000, 437) for port in try_ports: print 'binding to port %d:' % (port,), sockaddress = ('127.0.0.1', port) try: sock.bind(sockaddress) print 'works' break except socket.error, e: print e else: raise e else: assert False, "Could not create server" sock.listen(1) writeend = socket.socket() thread.start_new_thread(writeend.connect, (sockaddress,)) readend, addr2 = sock.accept() try: iwtd, owtd, ewtd = select.select([readend], [], [], 0) assert iwtd == owtd == ewtd == [] writeend.send('X') iwtd, owtd, ewtd = select.select([readend], [], []) assert iwtd == [readend] assert owtd == ewtd == [] print "Test OK" finally: writeend.close() readend.close() -- Amaury Forgeot d'Arc From jgustak at gmail.com Fri Jul 6 08:55:50 2007 From: jgustak at gmail.com (Jakub Gustak) Date: Fri, 6 Jul 2007 08:55:50 +0200 Subject: [pypy-dev] scheme interpreter [status report] In-Reply-To: References: Message-ID: Hello. What we have done on scheme interpreter: Working lambdas, with closures almost as described in r5rs. What we miss here is proper tail-recursion and issues, when lambda is called with wrong number of arguments (though should be simple to add). Some rethinking on ExecutionContext. Now we have global and local scope (looks like python). When copying ExecutionContext only local scope is copied, global one is shared. What more, copied ExecutionContext cannot bound new variables in global scope, though it can set new values to existing ones. Done quotations both as (quote ) and as '. After all it looks like distinction between W_Identifier and W_Symbol is useless. So probably only one of them will live in the future. Also implemented parsing for dotted lists (needed for lambdas definition). cons, car, cdr were easy ones. Probably not much hacking before the sprint. So the plan for EP sprint is RPython. I would like also to discuss some other features, if time will let us. See you soon on #pypy or on EP. Cheers, Jakub From cfbolz at gmx.de Mon Jul 9 19:33:30 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 9 Jul 2007 19:33:30 +0200 Subject: [pypy-dev] paper on runtime specialization Message-ID: <348899050707091033o67c92e9by818a25f6877cae2@mail.gmail.com> Hi all! Today I found this rather interesting paper on runtime specialization of C: http://www.cs.washington.edu/research/dyncomp/Papers/tr-97-03-03-abstract.html I have only skimmed it so far, but it seems there are many shared ideas between PyPy and this approach. Cheers, Carl Friedrich From cfbolz at gmx.de Mon Jul 9 20:58:33 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 9 Jul 2007 20:58:33 +0200 Subject: [pypy-dev] paper on runtime specialization In-Reply-To: <348899050707091033o67c92e9by818a25f6877cae2@mail.gmail.com> References: <348899050707091033o67c92e9by818a25f6877cae2@mail.gmail.com> Message-ID: <348899050707091158x1f36d4fdxacc990f1e9d46ca7@mail.gmail.com> 2007/7/9, Carl Friedrich Bolz : > Today I found this rather interesting paper on runtime specialization of C: There is even more interesting stuff (including papers) on their web-page: http://www.cs.washington.edu/research/dyncomp/ Cheers, Carl Friedrich From fijal at genesilico.pl Wed Jul 11 13:13:40 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Wed, 11 Jul 2007 13:13:40 +0200 Subject: [pypy-dev] Quick personal view regarding js interpreter Message-ID: <4694BB64.5050700@genesilico.pl> It's completely unusable. Usually you end up with Fatal RPythonError in case you try to run stuff. This is not how we go about stuff! All error reporting should work well not only on top of js_interactive, but actually on top of js-c as well. Leonardo - have you even run js_interactive once before you commit? Last time it was unbound local or so. Cheers, fijal :. From scott+pypy-dev at scottdial.com Wed Jul 11 16:56:20 2007 From: scott+pypy-dev at scottdial.com (Scott Dial) Date: Wed, 11 Jul 2007 10:56:20 -0400 Subject: [pypy-dev] Win32 tester Message-ID: <4694EF94.2010908@scottdial.com> Greetings Everyone, I am going to have to stop running the win32 test suite. I am not sure there is anyone actively correcting problems on win32, so it may be of no consequence. If someone else is interested in running this, then I would be more than glad to share my knowledge with them. Otherwise, sorry to leave you without. -Scott -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From bea at changemaker.nu Thu Jul 12 15:29:20 2007 From: bea at changemaker.nu (=?ISO-8859-1?Q?Beatrice_D=FCring?=) Date: Thu, 12 Jul 2007 15:29:20 +0200 Subject: [pypy-dev] Win32 tester In-Reply-To: <4694EF94.2010908@scottdial.com> References: <4694EF94.2010908@scottdial.com> Message-ID: <46962CB0.8030409@changemaker.nu> Hi there Scott Dial skrev: > Greetings Everyone, > > I am going to have to stop running the win32 test suite. I am not sure > there is anyone actively correcting problems on win32, so it may be of > no consequence. If someone else is interested in running this, then I > would be more than glad to share my knowledge with them. Otherwise, > sorry to leave you without. > > -Scott > > Thanks for helping out with this Scott! And the door is always open - as you know ;-) Cheers Bea From fijal at genesilico.pl Fri Jul 13 13:14:00 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Fri, 13 Jul 2007 13:14:00 +0200 Subject: [pypy-dev] Win32 tester In-Reply-To: <4694EF94.2010908@scottdial.com> References: <4694EF94.2010908@scottdial.com> Message-ID: <46975E78.7030007@genesilico.pl> Scott Dial wrote: > Greetings Everyone, > > I am going to have to stop running the win32 test suite. I am not sure > there is anyone actively correcting problems on win32, so it may be of > no consequence. If someone else is interested in running this, then I > would be more than glad to share my knowledge with them. Otherwise, > sorry to leave you without. > > -Scott > > I would blame a bit (from my side) lack of a single web page showing all failures on all platforms. Unless we have one, it's probably a bit useless to have such a status page, just because. I would like to set up one at some point, but that's another issue. Thanks for cooperating and I (personally) used it from time to time (although never said a word). Cheers, fijal :. From gbowyer at fastmail.co.uk Fri Jul 13 15:37:57 2007 From: gbowyer at fastmail.co.uk (Greg Bowyer) Date: Fri, 13 Jul 2007 14:37:57 +0100 Subject: [pypy-dev] Taking pypy for a test drive Message-ID: Sorry if this is an exceptional newbie post ... Now that pypy has hit v1 I have tried using it for a few test examples, most of this stuff so far is a little bit simple, however one test is to run a django application that I am developing for someone with pypy rather that cpython However when I fire up pypy i get the following from the django application, what am I missing here greg at gregslaptop ~/projects/minesite/minesite $ ./pypy-c manage.py runserver debug: WARNING: library path not found, using compiled-in sys.path Traceback (most recent call last): File "?", line 32, in run_toplevel File "manage.py", line 11, in execute_manager(settings) File "/usr/lib/python2.4/site-packages/django/core/management.py", line 1736, in execute_manager execute_from_command_line(action_mapping, argv) File "/usr/lib/python2.4/site-packages/django/core/management.py", line 1626, in execute_from_command_line translation.activate('en-us') File "/usr/lib/python2.4/site-packages/django/utils/translation/__init__.py", line 66, in activate return real_activate(language) File "/usr/lib/python2.4/site-packages/django/utils/translation/__init__.py", line 29, in delayed_loader from django.conf import settings File "/usr/lib/python2.4/site-packages/django/conf/__init__.py", line 149, in __builtins__['_'] = first_time_gettext TypeError: cannot set items on object From exarkun at divmod.com Fri Jul 13 15:50:59 2007 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Fri, 13 Jul 2007 09:50:59 -0400 Subject: [pypy-dev] Taking pypy for a test drive In-Reply-To: Message-ID: <20070713135059.4947.464118518.divmod.quotient.11057@ohm> On Fri, 13 Jul 2007 14:37:57 +0100, Greg Bowyer wrote: >Sorry if this is an exceptional newbie post ... > >Now that pypy has hit v1 I have tried using it for a few test examples, >most of this stuff so far is a little bit simple, however one test is to >run a django application that I am developing for someone with pypy >rather that cpython > >However when I fire up pypy i get the following from the django >application, what am I missing here > >greg at gregslaptop ~/projects/minesite/minesite $ ./pypy-c manage.py runserver >debug: WARNING: library path not found, using compiled-in sys.path >Traceback (most recent call last): > File "?", line 32, in run_toplevel > File "manage.py", line 11, in > execute_manager(settings) > File "/usr/lib/python2.4/site-packages/django/core/management.py", >line 1736, in execute_manager > execute_from_command_line(action_mapping, argv) > File "/usr/lib/python2.4/site-packages/django/core/management.py", >line 1626, in execute_from_command_line > translation.activate('en-us') > File >"/usr/lib/python2.4/site-packages/django/utils/translation/__init__.py", >line 66, in activate > return real_activate(language) > File >"/usr/lib/python2.4/site-packages/django/utils/translation/__init__.py", >line 29, in delayed_loader > from django.conf import settings > File "/usr/lib/python2.4/site-packages/django/conf/__init__.py", line >149, in > __builtins__['_'] = first_time_gettext >TypeError: cannot set items on object It seems likely that what django ought to be doing is: import __builtin__ __builtin__._ = first_time_gettext And a similar change in first_time_gettext. __builtin__/__builtins__ is a bit subtle and I might be forgetting something, but I think the above is correct. I remember that __builtins__ is somethings a dict instead of the __builtin__ module, but I forget when (and I couldn't find the case with a few simple tests). I think python-dev has pointed out that this is an implementation detail of CPython on several occassions, though, and that explicitly importing the __builtin__ module is the right thing to do. Jean-Paul From njriley at uiuc.edu Fri Jul 13 16:07:10 2007 From: njriley at uiuc.edu (Nicholas Riley) Date: Fri, 13 Jul 2007 09:07:10 -0500 Subject: [pypy-dev] Taking pypy for a test drive In-Reply-To: <20070713135059.4947.464118518.divmod.quotient.11057@ohm> References: <20070713135059.4947.464118518.divmod.quotient.11057@ohm> Message-ID: <20070713140710.GA14844@uiuc.edu> On Fri, Jul 13, 2007 at 09:50:59AM -0400, Jean-Paul Calderone wrote: > It seems likely that what django ought to be doing is: > > import __builtin__ > __builtin__._ = first_time_gettext > > And a similar change in first_time_gettext. __builtin__/__builtins__ is > a bit subtle and I might be forgetting something, but I think the above > is correct. I remember that __builtins__ is somethings a dict instead of > the __builtin__ module, but I forget when (and I couldn't find the case > with a few simple tests). I think python-dev has pointed out that this > is an implementation detail of CPython on several occassions, though, and > that explicitly importing the __builtin__ module is the right thing to > do. It's documented in the Python reference manual. -- Nicholas Riley | From simon at arrowtheory.com Tue Jul 17 03:22:03 2007 From: simon at arrowtheory.com (Simon Burton) Date: Mon, 16 Jul 2007 18:22:03 -0700 Subject: [pypy-dev] revision 45089 Message-ID: <20070716182203.0d1953e1.simon@arrowtheory.com> I found that revision 45089 makes compilation much slower. No idea why. Simon. ------------------------------------------------------------------------ r45089 | justas | 2007-07-14 08:35:59 -0700 (Sat, 14 Jul 2007) | 8 lines (Arlo, Justas) Introduce --pyversion option to select the parser/compiler grammar Defaults to 2.4, also accepts 2.3 and 2.5a. Changed a bunch of imports to use the selected grammar. TODO: pypy/interpreter/pyparser/symbol.py uses a fixed list of symbols, and it shouldn't From arigo at tunes.org Tue Jul 17 11:57:09 2007 From: arigo at tunes.org (Armin Rigo) Date: Tue, 17 Jul 2007 11:57:09 +0200 Subject: [pypy-dev] revision 45089 In-Reply-To: <20070716182203.0d1953e1.simon@arrowtheory.com> References: <20070716182203.0d1953e1.simon@arrowtheory.com> Message-ID: <20070717095709.GA11010@code0.codespeak.net> Hi Simon, On Mon, Jul 16, 2007 at 06:22:03PM -0700, Simon Burton wrote: > I found that revision 45089 makes compilation much slower. No idea why. Thanks for the note. I fixed it in r45153 after a cProfile run of some tests - the problem was obscure: the flow object space was taking ages (about 0.2 seconds for simple functions). This is because the parser and compiler instantiation code, which is not needed at all for the flow space, was actually called many times per function. And r45089 made that code take much longer. It's not a problem in general because it's not supposed to happen frequently - indeed, it does not need to be called at all when we use the flow space. A bientot, Armin. From cfbolz at gmx.de Tue Jul 17 14:26:08 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Tue, 17 Jul 2007 14:26:08 +0200 Subject: [pypy-dev] sprint report Message-ID: <469CB560.9030108@gmx.de> Hi all! Since I didn't manage to come to Europython and the sprint afterwards, I would really appreciate it if somebody wrote a sprint report. Since I guess that is kind of unlikely to happen now, could at least everybody write a paragraph about what he worked on? Cheers, Carl Friedrich From micahel at gmail.com Tue Jul 17 15:20:36 2007 From: micahel at gmail.com (Michael Hudson) Date: Tue, 17 Jul 2007 14:20:36 +0100 Subject: [pypy-dev] sprint report In-Reply-To: <469CB560.9030108@gmx.de> References: <469CB560.9030108@gmx.de> Message-ID: On 17/07/07, Carl Friedrich Bolz wrote: > Hi all! > > Since I didn't manage to come to Europython and the sprint afterwards, I > would really appreciate it if somebody wrote a sprint report. Since I > guess that is kind of unlikely to happen now, could at least everybody > write a paragraph about what he worked on? Samuele and I hacked on the twisted mess of memory management implementations. Basically we managed to much simplify the interface the low-level backends need to provide -- there is no longer any need to support any 'malloc' operation that deals with types. There is still confusion between various malloc flavours and operations, but it's getting better. Backends that support Boehm need to implement three new operations: boehm_malloc, boehm_malloc_atomic and boehm_register_finalizer. These could/should one day be removed and implemented with rffi (they're just function calls to named function^Wmacros, after all), but that seemed like too much wizardry for the time we had. Cheers, mwh From simon at arrowtheory.com Tue Jul 17 17:02:53 2007 From: simon at arrowtheory.com (Simon Burton) Date: Tue, 17 Jul 2007 08:02:53 -0700 Subject: [pypy-dev] expected_extra_mallocs Message-ID: <20070717080253.a1387d0e.simon@arrowtheory.com> Here is a little test: def func(): x = 1. print x def test_1(): from pypy.translator.c.test.test_genc import compile _func = compile(func, []) _func() I get this error: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def checking_fn(*args, **kwds): if 'expected_extra_mallocs' in kwds: expected_extra_mallocs = kwds.pop('expected_extra_mallocs') else: expected_extra_mallocs = 0 res = compiled_fn(*args, **kwds) mallocs, frees = module.malloc_counters() E assert mallocs - frees == expected_extra_mallocs > assert (6 - 5) == 0 [/home/simon/local/pypy-latest/pypy/translator/c/test/test_genc.py:54] So, do I call this with expected_extra_mallocs=1 ? In general, how many extra mallocs should I expect ? Simon. From jacob at openend.se Tue Jul 17 19:33:32 2007 From: jacob at openend.se (Jacob =?utf-8?q?Hall=C3=A9n?=) Date: Tue, 17 Jul 2007 19:33:32 +0200 Subject: [pypy-dev] More sprint reporting Message-ID: <200707171933.32223.jacob@openend.se> Here is what I know happened in other groups: Armin and Laura worked on rffi. After 2 days of trying different approaches and getting stumped each time, they made a breakthrough on Saturday, and managed to produce a fairly complete slution in a matter of a few hours. Making the solution more complete would require a rewrite of ctypes (Ick!). Maciek worked on getting PyPy to be self hosting under pypy-c. This succeeded before he left the sprint. Jakub worked on more scheme features. With the help of Antonio, he managed to get quite a bit done. Exactly what he did, I don't know. Christian worked on fixing pickling of tasklets. He got as far as understanding what the real problem was before I left. Michael and Samuele worked on malloc cleanup. I think they were fairly satisfied with the result. Samuele found and fixed a very nasty bug somewhere in the translator toolchain. Exception transformation was dependent on the order the graphs were evaluated. The result was that exceptions could be lost without a trace. There were 3 people who's names I have forgotten who worked on better documentation on how to leverage the javascript backend. There were some results checked in, but I don't know how complete it is. I think Holger did some work on improving py.test. That was at least his plan, but I never got a chance to ask what he actually did. I think that is about it. Jacob From jacob at openend.se Tue Jul 17 19:38:39 2007 From: jacob at openend.se (Jacob =?utf-8?q?Hall=C3=A9n?=) Date: Tue, 17 Jul 2007 19:38:39 +0200 Subject: [pypy-dev] Sprint report Message-ID: <200707171938.39921.jacob@openend.se> I think this got stuck in moderation, as I sent it with a from address that didn't match my mailinglist subscription. Justas, Arlo Belshee and I worked on Python 2.5 compliance. We started with the "with" statement and fairly quickly found that it seemed to work as intended, except for the fact that the necessary "from __future__ import with_statement" had no effect. To make this work properly, we decided to make a clean implementation - CPython does nasty things in order to avoid modifying the AST on the fly. This required modifying the grammar, which in turn led to the discovery that building a parser from the grammar lacked unit tests. We then created the neccessary test setup and made tests for the modifications we were making for the grammar. Then we went on to being able to insert a rule for the "with" statement into the builder on the fly. This required more instrumentation for unit testing and some modifications to the builder. Finally, added a config option to allow selecting different versions from the command line. This was essentially finished, though there may still be places which have the Pthon version hard coded. Jacob From jgustak at gmail.com Tue Jul 17 21:44:24 2007 From: jgustak at gmail.com (Jakub Gustak) Date: Tue, 17 Jul 2007 21:44:24 +0200 Subject: [pypy-dev] More sprint reporting In-Reply-To: <200707171933.32223.jacob@openend.se> References: <200707171933.32223.jacob@openend.se> Message-ID: Here is what happened with scheme interpreter: Antonio and I were working on scheme interpreter to translate. - (antocuni, jlg) The biggest issue was with number handling, it resulted with numbers hierarchy rethinking and arithmetical operations refactoring most of them are dynamically generated. - (fijal, jlg) Some effort was put into syntax checking, especially in macros where lot of w_pair.car/cdr had to be checked if are still instance of W_Pair. - (antocuni, jlg) Then there was work on delayed evaluation. During this Armin found bug in lambda definition identical to one I made with W_Promise. - (arigo, jlg) Finally we made proper tail-recursion work for calls in tail context. - (arigo, jlg) Thats it! btw. I made some dummy benchmark on tail-recursive loop and some arithmetical operations. Our ss-c is about 5 times slower than guile. Cheers, Jakub From simon at arrowtheory.com Tue Jul 17 22:42:38 2007 From: simon at arrowtheory.com (Simon Burton) Date: Tue, 17 Jul 2007 13:42:38 -0700 Subject: [pypy-dev] sprint report In-Reply-To: <469CB560.9030108@gmx.de> References: <469CB560.9030108@gmx.de> Message-ID: <20070717134238.a1c0d9ab.simon@arrowtheory.com> I worked with Maciek on generating rffi wrappers from ctypes objects. The first method we tried used code generation. Later on I got it to generate live rffi objects from the ctypes objects, without code generation. I also made some progress on generating rffi wrappers for cairo and SDL, and also testing these on ll2ctypes. Simon. From fijal at genesilico.pl Wed Jul 18 11:54:49 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Wed, 18 Jul 2007 11:54:49 +0200 Subject: [pypy-dev] Sprint report In-Reply-To: <200707171938.39921.jacob@openend.se> References: <200707171938.39921.jacob@openend.se> Message-ID: <469DE369.7010905@genesilico.pl> Three guys who's names Jacob doesn't remember were working on flex backend (reusing JavaScript one to be able to produce flex code and some libraries). I've got no idea what was their progress (hopefully they're reading pypy-dev and are able to reply :-) Cheers, fijal :. From lac at openend.se Wed Jul 18 12:05:03 2007 From: lac at openend.se (Laura Creighton) Date: Wed, 18 Jul 2007 12:05:03 +0200 Subject: [pypy-dev] Sprint report In-Reply-To: Message from Maciek Fijalkowski of "Wed, 18 Jul 2007 11:54:49 +0200." <469DE369.7010905@genesilico.pl> References: <200707171938.39921.jacob@openend.se> <469DE369.7010905@genesilico.pl> Message-ID: <200707181005.l6IA53ih016094@theraft.openend.se> In a message of Wed, 18 Jul 2007 11:54:49 +0200, Maciek Fijalkowski writes: >Three guys who's names Jacob doesn't remember were working on flex >backend (reusing JavaScript one to be able to produce flex code and some >libraries). I've got no idea what was their progress (hopefully they're >reading pypy-dev and are able to reply :-) > >Cheers, >fijal I think they were Ren? Dudfield, Alejandro Curia and Lucio Torre, from Australia, Argentina and Argentina, who may not be home yet. Laura From anto.cuni at gmail.com Wed Jul 18 18:13:20 2007 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed, 18 Jul 2007 18:13:20 +0200 Subject: [pypy-dev] sprint report In-Reply-To: <469CB560.9030108@gmx.de> References: <469CB560.9030108@gmx.de> Message-ID: <469E3C20.7010803@gmail.com> Carl Friedrich Bolz wrote: > Hi all! > > Since I didn't manage to come to Europython and the sprint afterwards, I > would really appreciate it if somebody wrote a sprint report. Since I > guess that is kind of unlikely to happen now, could at least everybody > write a paragraph about what he worked on? On the first day I worked with Jakub to make the scheme interpreter translatable. On the second day, I paired with Maciek trying to make pypy-c self-hosted: we fixed few bugs and now it is self-hosted, as long as pypy-c is being translated with the same opcodes as the hosting pypy-c. Finally, I spent the third day by working again with Jakub on the scheme interpreter and by experimenting with method lookup in the interpreter, without concluding anything interesting :-). ciao Anto From simon at arrowtheory.com Fri Jul 20 03:31:17 2007 From: simon at arrowtheory.com (Simon Burton) Date: Thu, 19 Jul 2007 18:31:17 -0700 Subject: [pypy-dev] rffi strict typing Message-ID: <20070719183117.97ad1ce3.simon@arrowtheory.com> The lltype's are very strict about types. If I declare an llexternal that takes a Float arg, I cannot call it with an int arg. Well maybe that's not so bad. But in rlib.rsdl there are functions that take Unsigned, and these don't work even if called with a non-negative constant int... Can we loosen up these restrictions ? Maybe put an "is_compatable(self, other)" method on LowLevelType's ? Simon. using py lib: /home/simon/local/pypy-latest/py test/test_SDL.py[2] FF __________________________________________________________________________________________________________________________________________________________________________________________________________ entrypoint: test_run ____________________________________________________________ def test_run(): > demo() [/home/simon/local/pypy-latest/pypy/rlib/rsdl/test/test_SDL.py:48] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def demo(): > if SDL.Init(SDL.INIT_VIDEO) < 0: [/home/simon/local/pypy-latest/pypy/rlib/rsdl/test/test_SDL.py:13] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def __call__(self, *args): if isinstance(self._T, FuncType): if len(args) != len(self._T.ARGS): raise TypeError,"calling %r with wrong argument number: %r" % (self._T, args) for a, ARG in zip(args, self._T.ARGS): if typeOf(a) != ARG: E raise TypeError,"calling %r with wrong argument types: %r" % (self._T, args) > TypeError: calling Signed> with wrong argument types: (32,) ... From jgustak at gmail.com Fri Jul 20 11:39:28 2007 From: jgustak at gmail.com (Jakub Gustak) Date: Fri, 20 Jul 2007 11:39:28 +0200 Subject: [pypy-dev] scheme interpreter [status report] In-Reply-To: References: Message-ID: What's on with scheme interpreter: Nice progress during post EuroPython sprint, see: http://codespeak.net/pipermail/pypy-dev/2007q3/003919.html And after the sprint: (define (fun ) ) syntactic sugar added to lambda definitions. (begin ...), (letrec ...) added. This leads to small changes in ExecutionContext (set-car! ), (set-cdr! ) added. Careful, it is possible to create circural list, and crash interpreter when it will try to display such list. Added quasi-quotations. Code is not so straightforward. Maybe it can be done differently. W_Identifier and W_Symbol are now W_Symbol. All symbols are created by symbol(name) function, which stores all symbols in global W_Symbol.obarray dictionary or returns existing one. It makes comparing symbols trivial. Next step is to implement macros. There was also plan to move some some of the syntax checking to parser: e.g. (if test then else) to get parsed directly to MacroIf(test, then, else). The problem is, that this syntax can change in scheme on runtime: e.g. (define if #t) so it doesn't make much sense. And later adding main primitive procedures will make scheme interpreter quite usable. Cheers, Jakub From arigo at tunes.org Fri Jul 20 14:48:17 2007 From: arigo at tunes.org (Armin Rigo) Date: Fri, 20 Jul 2007 14:48:17 +0200 Subject: [pypy-dev] sprint report In-Reply-To: <469CB560.9030108@gmx.de> References: <469CB560.9030108@gmx.de> Message-ID: <20070720124817.GA19336@code0.codespeak.net> Hi all, As Laura and Jacob already hinted at, Laura and me managed to write three times and trash twice code that allows ll external functions to be really called on top of CPython. This means that code can now be written using rffi, including calls to external functions, and this can be tested on top of CPython. We have thus a ctypes replacement, from the point of the view of the CPython user; and we no longer need rctypes at all from the point of the view of the RPython user. Our code (in ll2ctypes.py) is able to transparently convert structures created by lltype.malloc(TYPE, flavor='raw') so that their storage becomes a ctypes object internally. This is done lazily, when the structure "escapes" to a call to an external function. The call itself is also performed through ctypes. As usual we had to fight a bit with ctypes to have it do what we really need (instead of what it thinks you'd prefer today) but nothing deep (Jacob's mail seems to say that we were blocked unless we rewrote ctypes - I'm not sure what he refers to). This is a no-API piece of code invoked automatically when an lltype pointer to an external function is called on top of CPython. A few new pieces of API have been added to rffi.py for general rffi usage, notably rffi.cast(TARGETTYPE, value) which tries to have the same semantics as the C-level cast operator. A bientot, Armin. From arigo at tunes.org Fri Jul 20 14:50:13 2007 From: arigo at tunes.org (Armin Rigo) Date: Fri, 20 Jul 2007 14:50:13 +0200 Subject: [pypy-dev] expected_extra_mallocs In-Reply-To: <20070717080253.a1387d0e.simon@arrowtheory.com> References: <20070717080253.a1387d0e.simon@arrowtheory.com> Message-ID: <20070720125013.GB19336@code0.codespeak.net> Hi Simon, On Tue, Jul 17, 2007 at 08:02:53AM -0700, Simon Burton wrote: > def func(): > x = 1. > print x It's a known non-issue: any RPython program calling the "print" statement "leaks" exactly one malloced piece of memory. It's not an issue because in truth the memory is simply kept around to be reused by the next "print". I guess you need to expect one extra malloc in this case. A bientot, Armin. From arigo at tunes.org Fri Jul 20 14:52:54 2007 From: arigo at tunes.org (Armin Rigo) Date: Fri, 20 Jul 2007 14:52:54 +0200 Subject: [pypy-dev] rffi strict typing In-Reply-To: <20070719183117.97ad1ce3.simon@arrowtheory.com> References: <20070719183117.97ad1ce3.simon@arrowtheory.com> Message-ID: <20070720125254.GC19336@code0.codespeak.net> Hi Simon, On Thu, Jul 19, 2007 at 06:31:17PM -0700, Simon Burton wrote: > The lltype's are very strict about types. Indeed, it seems worthwhile to loosen this restriction at least for the purpose of calling external functions... Not sure how, but there are possible hacks at least. Armin From fijal at genesilico.pl Sun Jul 22 22:42:57 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Sun, 22 Jul 2007 22:42:57 +0200 Subject: [pypy-dev] Strange annotation problem. Message-ID: <46A3C151.5030409@genesilico.pl> Discovered by exarkun. following diff breaks translation :. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: out.diff Url: http://codespeak.net/pipermail/pypy-dev/attachments/20070722/54bd2e47/attachment.diff From amauryfa at gmail.com Mon Jul 23 11:28:52 2007 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Mon, 23 Jul 2007 11:28:52 +0200 Subject: [pypy-dev] Strange annotation problem. In-Reply-To: <46A3C151.5030409@genesilico.pl> References: <46A3C151.5030409@genesilico.pl> Message-ID: Hello, Maciek Fijalkowski wrote: > Discovered by exarkun. following diff breaks translation > > :. > > > Index: ../../module/posix/__init__.py > =================================================================== > --- ../../module/posix/__init__.py (revision 45246) > +++ ../../module/posix/__init__.py (working copy) > @@ -83,6 +83,9 @@ > # interpleveldefs['uname'] = 'interp_posix.uname' > if hasattr(os, 'ttyname'): > interpleveldefs['ttyname'] = 'interp_posix.ttyname' > + if hasattr(os, 'setsid'): > + interpleveldefs['xxx'] = 'interp_posix.xxx' > + > for name in w_star: > if hasattr(os, name): > interpleveldefs[name] = 'interp_posix.' + name > Index: ../../module/posix/interp_posix.py > =================================================================== > --- ../../module/posix/interp_posix.py (revision 45246) > +++ ../../module/posix/interp_posix.py (working copy) > @@ -499,6 +499,22 @@ > raise OperationError(space.w_TypeError, space.wrap(msg)) > utime.unwrap_spec = [ObjSpace, str, W_Root] > > +def xxx(space): > + """xxx() -> pid > + > + Stuff > + """ > + return 3 > +xxx.unwrap_spec = [ObjSpace] > + The function seems to return an unwrapped object. does it behave better if you replace the return value with something like space.wrap(3) ? -- Amaury Forgeot d'Arc From fijal at genesilico.pl Mon Jul 23 12:02:29 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Mon, 23 Jul 2007 12:02:29 +0200 Subject: [pypy-dev] Strange annotation problem. In-Reply-To: References: <46A3C151.5030409@genesilico.pl> Message-ID: <46A47CB5.2070607@genesilico.pl> Amaury Forgeot d'Arc wrote: > Hello, > > Maciek Fijalkowski wrote: >> Discovered by exarkun. following diff breaks translation >> >> :. >> >> >> Index: ../../module/posix/__init__.py >> =================================================================== >> --- ../../module/posix/__init__.py (revision 45246) >> +++ ../../module/posix/__init__.py (working copy) >> @@ -83,6 +83,9 @@ >> # interpleveldefs['uname'] = 'interp_posix.uname' >> if hasattr(os, 'ttyname'): >> interpleveldefs['ttyname'] = 'interp_posix.ttyname' >> + if hasattr(os, 'setsid'): >> + interpleveldefs['xxx'] = 'interp_posix.xxx' >> + >> for name in w_star: >> if hasattr(os, name): >> interpleveldefs[name] = 'interp_posix.' + name >> Index: ../../module/posix/interp_posix.py >> =================================================================== >> --- ../../module/posix/interp_posix.py (revision 45246) >> +++ ../../module/posix/interp_posix.py (working copy) >> @@ -499,6 +499,22 @@ >> raise OperationError(space.w_TypeError, space.wrap(msg)) >> utime.unwrap_spec = [ObjSpace, str, W_Root] >> >> +def xxx(space): >> + """xxx() -> pid >> + >> + Stuff >> + """ >> + return 3 >> +xxx.unwrap_spec = [ObjSpace] >> + > > The function seems to return an unwrapped object. > does it behave better if you replace the return value with > something like space.wrap(3) ? > Hum. Good point :-) I must be sleeping again... Cheers, fijal :. From anto.cuni at gmail.com Mon Jul 23 13:07:55 2007 From: anto.cuni at gmail.com (Antonio Cuni) Date: Mon, 23 Jul 2007 13:07:55 +0200 Subject: [pypy-dev] Proposal of timeline In-Reply-To: <9c0bb8a00707211508k3697f0e3oe8a5a2276c2f5d83@mail.gmail.com> References: <469BE246.20107@gmail.com> <9c0bb8a00707161429v3d8761ag681353aacb7c87d7@mail.gmail.com> <9c0bb8a00707211508k3697f0e3oe8a5a2276c2f5d83@mail.gmail.com> Message-ID: <46A48C0B.20705@gmail.com> Hi Paul, I'm cc-ing pypy-dev. Really, don't be shy and don't worry to bother the list with your questions. They are not spam. I'm sure that the others are happy to hear your questions and also to answer in case I can't do it immediately (as happened in the last weekend). Paul deGrandis wrote: > Antonio, > > I've been working on the JVM backend. When I trace the float tests that > look at uint handling, weird NEGATIVE values pop up. I originally > associated these with overflow, but it's a different issue. Any > insight? The way to do unsigned values in Java is to use the next > primary type larger, for example, a uint is stored in a double... but > currently no checks exist for negative values. as Niko pointed out unsigned types are stored in the corresponding signed types in the JVM: this is ok becasue they occupy the same amount of bits, but you have to been careful when dealing with them because the JVM thinks you want to represent a signed value. In particular, if you try to print one to stdout it will be printed as a signed, and so this might be the reason why you see negative values. You have to write a custom procedure to format unsigned values into the right human readable implementation. Please note that this is necessary only during testing or when doing I/O: as long as your unsigned values are kept inside the JVM, they works just fine (apart from comparisons). The simplest way I can think to format an unsigned into an human readable form is to convert it into a double, as I already wrote in a previous email: double from_uint(int x) { double res = x; if (x < 0) // i.e., the leftmost but is 1 res += 4294967294; return res; } But again, you should use this *only* for I/O and testing, not for doing any real operation. > I looked at the test_overflow but these tests don't make sense for the > jvm. You have overflow built in to CLI so you want to make sure that > ovf_check from the rpython definition works, but the JVM defines these > operations. What exactly are these tests aiming to illustrate and test? RPython is a language with a (more or less :-)) well defined semantics, so every backend must take care of implementing this semantics correctly, even if it's not natively supported by the underlying platform. In this particular case, RPython programs expect overflow to be detected correctly. If the JVM does not do this for you, you have to handle those cases by yourself. GenC has the same problem: the C platform does not detect any overflow, so it must explicitly check for that; you can easily reuse the same logic; for example, this macro is taken from translator/c/src/int.h: #define OP_INT_ADD_OVF(x,y,r) \ OP_INT_ADD(x,y,r); \ if ((r^(x)) >= 0 || (r^(y)) >= 0); \ else FAIL_OVF("integer addition") for genjvm, you could e.g. write a java function doing the same (and raising the appropriate exception in case of overflow). > I haven't tackled overriding default values (as skipped in test_class) > yet, but that's really all there is needed left to pass the test_class > requirements I think you could skip this for now. IIRC it's only needed to translate pypy with the thunk objspace, it's not a priority. > I'm still working on the others, but I don't think I'll have trouble > having these first bugs fixed by the end of the work day on the 27th. ok, that's cool! :-) > Lastly, what is the easy approach to I/O and external functions? the easy approach is to forward the external functions to the backend, which will take care of rendering them as appropriate (usually by calling an helper function written in java). This approach is already used to implement time.time, time.clock and os.write: you can find their implementation in jvm/src/pypy/PyPy.java (ll_time_time, ll_time_clock, ll_os_write). The code that calls these functions is in jvm/generator.py, method 'call_primitive'. ciao Anto From fijal at genesilico.pl Mon Jul 23 13:21:41 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Mon, 23 Jul 2007 13:21:41 +0200 Subject: [pypy-dev] Proposal of timeline In-Reply-To: <46A48C0B.20705@gmail.com> References: <469BE246.20107@gmail.com> <9c0bb8a00707161429v3d8761ag681353aacb7c87d7@mail.gmail.com> <9c0bb8a00707211508k3697f0e3oe8a5a2276c2f5d83@mail.gmail.com> <46A48C0B.20705@gmail.com> Message-ID: <46A48F45.8090505@genesilico.pl> > >> Lastly, what is the easy approach to I/O and external functions? >> > > the easy approach is to forward the external functions to the backend, > which will take care of rendering them as appropriate (usually by > calling an helper function written in java). This approach is already > used to implement time.time, time.clock and os.write: you can find their > implementation in jvm/src/pypy/PyPy.java (ll_time_time, ll_time_clock, > ll_os_write). The code that calls these functions is in > jvm/generator.py, method 'call_primitive'. > > ciao Anto > Small addition from my side. I've been playing recently with a very naive reuse of BasicExternal infrastructure which is supporting external objects in JS backend. Looks nice so far and I'm able to hard code some RPython-accessible objects (in a very ad hoc manner, mind you). I don't know Java that well, but I would be happy to help with this issue. Cheers, fijal :. From amauryfa at gmail.com Mon Jul 23 14:32:46 2007 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Mon, 23 Jul 2007 14:32:46 +0200 Subject: [pypy-dev] [pypy-svn] r45269 - in pypy/dist/pypy/rpython/module: . tes Message-ID: Hello again, > Author: fijal > Date: Mon Jul 23 13:52:26 2007 > New Revision: 45269 > > Modified: > pypy/dist/pypy/rpython/module/ll_os.py > pypy/dist/pypy/rpython/module/test/test_posix.py > Log: > Add os.uname. RPython level. segfaults ll2ctypes in some strange way, > I'm not sure I want to know > > > Modified: pypy/dist/pypy/rpython/module/ll_os.py > ============================================================================== > --- pypy/dist/pypy/rpython/module/ll_os.py (original) > +++ pypy/dist/pypy/rpython/module/ll_os.py Mon Jul 23 13:52:26 2007 > @@ -110,6 +110,34 @@ > register_external(os.setsid, [], int, export_name="ll_os.ll_os_setsid", > llimpl=setsid_lltypeimpl) > > +# ------------------------------- os.uname ------------------------------ > + > +if hasattr(os, 'uname'): > + UTSNAMEP = rffi.CStruct('utsname', ('sysname', rffi.CCHARP), > + ('nodename', rffi.CCHARP), > + ('release', rffi.CCHARP), > + ('version', rffi.CCHARP), > + ('machine', rffi.CCHARP), > + ('stuff', rffi.CCHARP)) I think the error comes from the ctypes interpretation of the structure. utsname is not made of char* pointers, but (at least on the debian machine I have access to) the members are actually fixed arrays of chars, the lengths of which are not public, but well specified in sys/utsname.h. It should make no difference when translated, because the code looks like v->sysname etc. ctypes on the other hand needs to know the precise sizeof and offsets of each member. I tried to modify the _socket module to also use rffi, and had the same problem. I think that we will need a tool similar to pypy.rpython.rctypes.tool.ctypes_platform, which generates and compiles C code to get the different sizes and offsets. (We will need support for C defines as well) Hope this helps, -- Amaury Forgeot d'Arc From fijal at genesilico.pl Mon Jul 23 14:36:37 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Mon, 23 Jul 2007 14:36:37 +0200 Subject: [pypy-dev] [pypy-svn] r45269 - in pypy/dist/pypy/rpython/module: . tes In-Reply-To: References: Message-ID: <46A4A0D5.9020405@genesilico.pl> Amaury Forgeot d'Arc wrote: > Hello again, > >> Author: fijal >> Date: Mon Jul 23 13:52:26 2007 >> New Revision: 45269 >> >> Modified: >> pypy/dist/pypy/rpython/module/ll_os.py >> pypy/dist/pypy/rpython/module/test/test_posix.py >> Log: >> Add os.uname. RPython level. segfaults ll2ctypes in some strange way, >> I'm not sure I want to know >> >> >> Modified: pypy/dist/pypy/rpython/module/ll_os.py >> ============================================================================== >> >> --- pypy/dist/pypy/rpython/module/ll_os.py (original) >> +++ pypy/dist/pypy/rpython/module/ll_os.py Mon Jul 23 13:52:26 2007 >> @@ -110,6 +110,34 @@ >> register_external(os.setsid, [], int, >> export_name="ll_os.ll_os_setsid", >> llimpl=setsid_lltypeimpl) >> >> +# ------------------------------- os.uname >> ------------------------------ >> + >> +if hasattr(os, 'uname'): >> + UTSNAMEP = rffi.CStruct('utsname', ('sysname', rffi.CCHARP), >> + ('nodename', rffi.CCHARP), >> + ('release', rffi.CCHARP), >> + ('version', rffi.CCHARP), >> + ('machine', rffi.CCHARP), >> + ('stuff', rffi.CCHARP)) > > I think the error comes from the ctypes interpretation of the > structure. utsname is not made of char* pointers, but (at least on the > debian machine I have access to) the members are actually fixed arrays > of chars, the lengths of which are not public, but well specified in > sys/utsname.h. > > It should make no difference when translated, because the code looks > like v->sysname etc. > ctypes on the other hand needs to know the precise sizeof and offsets > of each member. > > I tried to modify the _socket module to also use rffi, and had the > same problem. > > I think that we will need a tool similar to > pypy.rpython.rctypes.tool.ctypes_platform, which generates and > compiles C code to get the different sizes and offsets. > (We will need support for C defines as well) > > Hope this helps, > I think armin worked a bit on it. We need some more support for that I'm afraid. Let's dig... :. From fijal at genesilico.pl Mon Jul 23 14:38:32 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Mon, 23 Jul 2007 14:38:32 +0200 Subject: [pypy-dev] [pypy-svn] r45269 - in pypy/dist/pypy/rpython/module: . tes In-Reply-To: References: Message-ID: <46A4A148.2090802@genesilico.pl> From the manpage: The length of the arrays in a struct utsname is unspecified; the fields are terminated by a null byte (?? ?). ??? I'm completely confused. :. From arigo at tunes.org Tue Jul 24 18:08:22 2007 From: arigo at tunes.org (Armin Rigo) Date: Tue, 24 Jul 2007 18:08:22 +0200 Subject: [pypy-dev] [pypy-svn] r45269 - in pypy/dist/pypy/rpython/module: . tes In-Reply-To: <46A4A148.2090802@genesilico.pl> References: <46A4A148.2090802@genesilico.pl> Message-ID: <20070724160822.GA25719@code0.codespeak.net> Hi Fijal, On Mon, Jul 23, 2007 at 02:38:32PM +0200, Maciek Fijalkowski wrote: > The length of the arrays in a struct utsname is unspecified; the > fields > are terminated by a null byte (?????? ???). > > I'm completely confused. It means that the structure looks like this in lltype notation: Struct('utsname', ('text1', FixedSizeArray(Char, 10)), ('text2', FixedSizeArray(Char, 20)), ...) where, annoyingly, the numbers '10' and '20' are platform-dependent and must be obtained by trying to compile snippets of C code using sizeof() and offsetof(). Or are you confused about the bit saying that the fields are zero-terminated? It just means that in the array, characters 0 to N-1 are non-null, charater N is null, and the rest of the characters are undefined. A bientot, Armin. From tismer at stackless.com Tue Jul 24 21:18:48 2007 From: tismer at stackless.com (Christian Tismer) Date: Tue, 24 Jul 2007 21:18:48 +0200 Subject: [pypy-dev] [pypy-svn] r45269 - in pypy/dist/pypy/rpython/module: . tes In-Reply-To: <20070724160822.GA25719@code0.codespeak.net> References: <46A4A148.2090802@genesilico.pl> <20070724160822.GA25719@code0.codespeak.net> Message-ID: <01F2E744-7AB2-4A88-8049-52F9FDD9DA06@stackless.com> On 24.07.2007, at 18:08, Armin Rigo wrote: > Hi Fijal, > > On Mon, Jul 23, 2007 at 02:38:32PM +0200, Maciek Fijalkowski wrote: >> The length of the arrays in a struct utsname is >> unspecified; the >> fields >> are terminated by a null byte (?????? ???). >> >> I'm completely confused. > > It means that the structure looks like this in lltype notation: > > Struct('utsname', ('text1', FixedSizeArray(Char, 10)), > ('text2', FixedSizeArray(Char, 20)), > ...) > > where, annoyingly, the numbers '10' and '20' are platform-dependent > and > must be obtained by trying to compile snippets of C code using > sizeof() > and offsetof(). > > Or are you confused about the bit saying that the fields are > zero-terminated? It just means that in the array, characters 0 to N-1 > are non-null, charater N is null, and the rest of the characters are > undefined. Which reminds us to implement this carefully and not to rely on the terminating byte, or we will open the door to buffer overflow exploits :-) ciao - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20070724/42748819/attachment.htm From tismer at stackless.com Tue Jul 24 21:33:09 2007 From: tismer at stackless.com (Christian Tismer) Date: Tue, 24 Jul 2007 21:33:09 +0200 Subject: [pypy-dev] sprint report In-Reply-To: <20070717134238.a1c0d9ab.simon@arrowtheory.com> References: <469CB560.9030108@gmx.de> <20070717134238.a1c0d9ab.simon@arrowtheory.com> Message-ID: <1D942D29-1FEB-4228-9EA2-625B295EE659@stackless.com> As Jacob mentioned, I worked on fixing Stackless Pickling. I thought this would be a quick thing for the first morning, but it blocked me very much until the end of the sprint, where a discussion with Samuele clarified to us (we both didn't understand it for quite long). The problem holds for any app-level implementation of any switching concept that should be pickled: Unless you implement this very ugly with global variables, only, you will always have a reference to the tasklet/whatever on the python stack, that tries to do the pickling. I.E. you always end up pickling the pickler itself, in principal. This does not happen from the main tasklet, because it has special handing for that case. This needs to be re-thought. Some special code, somewhat similar to resume points in RPython needs to be implemented, to shortcut the frames on the Python stack which should not be visible (and therefore being pickled) at all. Hoping to be more productive and communicative next time - this was a huge blocker. ciao - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20070724/5fcd0e4d/attachment-0001.htm From fijal at genesilico.pl Wed Jul 25 09:17:03 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Wed, 25 Jul 2007 09:17:03 +0200 Subject: [pypy-dev] [pypy-svn] r45269 - in pypy/dist/pypy/rpython/module: . tes In-Reply-To: <01F2E744-7AB2-4A88-8049-52F9FDD9DA06@stackless.com> References: <46A4A148.2090802@genesilico.pl> <20070724160822.GA25719@code0.codespeak.net> <01F2E744-7AB2-4A88-8049-52F9FDD9DA06@stackless.com> Message-ID: <46A6F8EF.9030202@genesilico.pl> Christian Tismer wrote: > > On 24.07.2007, at 18:08, Armin Rigo wrote: > >> Hi Fijal, >> >> On Mon, Jul 23, 2007 at 02:38:32PM +0200, Maciek Fijalkowski wrote: >>> The length of the arrays in a struct utsname is unspecified; the >>> fields >>> are terminated by a null byte (?????? ???). >>> >>> I'm completely confused. >> >> It means that the structure looks like this in lltype notation: >> >> Struct('utsname', ('text1', FixedSizeArray(Char, 10)), >> ('text2', FixedSizeArray(Char, 20)), >> ...) >> >> where, annoyingly, the numbers '10' and '20' are platform-dependent and >> must be obtained by trying to compile snippets of C code using sizeof() >> and offsetof(). >> >> Or are you confused about the bit saying that the fields are >> zero-terminated? It just means that in the array, characters 0 to N-1 >> are non-null, charater N is null, and the rest of the characters are >> undefined. > > Which reminds us to implement this carefully and not to rely on the > terminating byte, or we will open the door to buffer overflow exploits :-) > > ciao - chris > I personally don't think that implementation of uname() might be a security hazard - if kernel really wants to hack us it can for sure :] Cheers, fijal :. From fijal at genesilico.pl Thu Jul 26 14:10:32 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Thu, 26 Jul 2007 14:10:32 +0200 Subject: [pypy-dev] Design question Message-ID: <46A88F38.4040009@genesilico.pl> I'm approaching "how to make cache failures not crash on you every time you import anything". So in short, file like ll_os.py looks right now: if cache.defined('XXX'): TYPE = cache.inttype(...) register_external(...) Idea is that if any of the functions involved in creation of external (cache functions...), show a warning and than create a register_external stub which will explode if referenced (by ie a specific extregistry entry). The question is how it should look like than. My idea is: class Os(registeringclass): def register_os_uname(...): ... def register_os_other(...): ... and so on. This will create huge indentation. Any other ideas? Cheers, fijal :. From jgustak at gmail.com Thu Jul 26 17:47:16 2007 From: jgustak at gmail.com (Jakub Gustak) Date: Thu, 26 Jul 2007 17:47:16 +0200 Subject: [pypy-dev] scheme interpreter [status report] In-Reply-To: References: Message-ID: > Next step is to implement macros. We have working simple macros. They are hygienic and referentially transparent. There is no support so you can't use (expr ...) syntax yet. There is also no support for recursive expanding also. There was again some ExecutionContext refactoring. Still only (let ...) and (let* ...) are fully compatible with macro expanding. (letrec ...) is not. In the future: more macros! (ellipsis, recursive expanding and CPS style macros) Cheers, Jakub Gustak From jgustak at gmail.com Thu Jul 26 19:37:59 2007 From: jgustak at gmail.com (Jakub Gustak) Date: Thu, 26 Jul 2007 19:37:59 +0200 Subject: [pypy-dev] scheme interpreter [status report] In-Reply-To: References: Message-ID: Some more info. > We have working simple macros. They are hygienic and referentially transparent. r5rs states that if macro introduces new binding, it should be renamed to avoid conflicts with other identifiers. In our case we create syntactic closure like described in: http://people.csail.mit.edu/people/jhbrown/scheme/macroslides04.pdf ExecutionContext.sput() (creating new binding) checks if it deals with syntactic closure or not, and depending on that binds location in proper context. W_Transformer can be called with quoted expression as argument, then appropriate template will expand and then expanded expression will be evaluated. This way of evaluation will surely cause problems with recursive macros, so it is not recommended. __ Jakub From fijal at genesilico.pl Fri Jul 27 23:54:30 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Fri, 27 Jul 2007 23:54:30 +0200 Subject: [pypy-dev] partial evaluation links Message-ID: <46AA6996.3070101@genesilico.pl> not sure, if this is of any interest: http://partial-eval.org/pe-impl.html list of partial evaluation implementations (mostly scheme, but also C). Cheers, fijal :. From fijal at genesilico.pl Sun Jul 29 20:30:26 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Sun, 29 Jul 2007 20:30:26 +0200 Subject: [pypy-dev] A quick question Message-ID: <46ACDCC2.3080104@genesilico.pl> Can we move jit tests to the end of nightly test run? It'll help quick-checking whether there are no breakages spread all around. Cheers, fijal :. From fijal at genesilico.pl Wed Aug 1 12:28:42 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Wed, 01 Aug 2007 12:28:42 +0200 Subject: [pypy-dev] Stackless vs Erlang benchmarks Message-ID: <46B0605A.6010305@genesilico.pl> http://muharem.wordpress.com/2007/07/31/erlang-vs-stackless-python-a-first-benchmark/ Christian: with a dedication for you :) We should try pypy on this btw. :. From jgustak at gmail.com Thu Aug 2 18:08:01 2007 From: jgustak at gmail.com (Jakub Gustak) Date: Thu, 2 Aug 2007 18:08:01 +0200 Subject: [pypy-dev] scheme interpreter [status report] In-Reply-To: References: Message-ID: A first, a joke: (syntax-rules () ((i-have-spare day ...) (hack 'macros (day ....)))) So we have recursive macros expanding recursively, as they should. I was working on macros with ellipsis almost whole week i. I was changing approach several times. But it looks like flat ellipses (not nested) work well. All most important parts of macros are SyntaxRule.match method, or rather matchr, and W_Transformer.substitute. They raise exception when discover ellipsis to handle it at higher level. matchr is kinda handling nested ellipses, but substitute not yet. That's pretty it. I would like to get nested ellipses working and then start playing with continuations. Wish me luck. Cheers, Jakub Gustak From simon at arrowtheory.com Fri Aug 3 21:14:13 2007 From: simon at arrowtheory.com (Simon Burton) Date: Fri, 3 Aug 2007 12:14:13 -0700 Subject: [pypy-dev] rffi feature request Message-ID: <20070803121413.bd2d992a.simon@arrowtheory.com> I would like to expose some functions as external symbols when i build a .so def foo(i, j): return i+j foo._expose_ = [rffi.INT, rffi.INT] This is basically so I can write cpython extension modules in rpython. (and manually doing ref counting (etc.) on the cpython objects.) Simon. From simon at arrowtheory.com Sun Aug 5 03:14:03 2007 From: simon at arrowtheory.com (Simon Burton) Date: Sat, 4 Aug 2007 18:14:03 -0700 Subject: [pypy-dev] rffi feature request In-Reply-To: <20070803121413.bd2d992a.simon@arrowtheory.com> References: <20070803121413.bd2d992a.simon@arrowtheory.com> Message-ID: <20070804181403.f15cdbb6.simon@arrowtheory.com> On Fri, 3 Aug 2007 12:14:13 -0700 Simon Burton wrote: > > I would like to expose some functions as external > symbols when i build a .so > > def foo(i, j): > return i+j > > foo._expose_ = [rffi.INT, rffi.INT] It seems like this could also enable a plugin system for rpython, and for example, (c or rpython) extension modules for the PyPy interpreter. Simon. From cfbolz at gmx.de Sun Aug 5 22:00:43 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Sun, 05 Aug 2007 22:00:43 +0200 Subject: [pypy-dev] Stackless vs Erlang benchmarks In-Reply-To: <46B0605A.6010305@genesilico.pl> References: <46B0605A.6010305@genesilico.pl> Message-ID: <46B62C6B.5040306@gmx.de> Hi Maciek Maciek Fijalkowski wrote: > http://muharem.wordpress.com/2007/07/31/erlang-vs-stackless-python-a-first-benchmark/ > > Christian: with a dedication for you :) > > We should try pypy on this btw. seems a bit meaningless, given that one of erlang's most important strengths is the possibility of using it to transparently across multiple processes and especially multiple machines. Cheers, Carl Friedrich From paul.degrandis at gmail.com Tue Aug 7 03:46:28 2007 From: paul.degrandis at gmail.com (Paul deGrandis) Date: Mon, 6 Aug 2007 21:46:28 -0400 Subject: [pypy-dev] Trouble with ooparse_int/ooparse_float Message-ID: <9c0bb8a00708061846m265d5f30q7f47eac4442242e1@mail.gmail.com> Anto, Niko, and all, I've been trying to get some of the string and float tests to pass. The trouble I'm running into is that ooparse_float doesn't know how to parse ll_str_0, but for the life of me can't find where I need to be looking or where I should override the ooparse_float method. I feel like it should go in my opcodes.py file (translator/jvm). Also, one question I had about long conversions, in translator/jvm/test/test_float.py:22, there is a test for long conversion. Tracing the results shows I get the correct answer for the conversion, but my result is not an r_longlong, the tests returns a Java Long. Any tips or hints? Regards, Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20070806/d565ae99/attachment.htm From anto.cuni at gmail.com Tue Aug 7 14:09:26 2007 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 07 Aug 2007 14:09:26 +0200 Subject: [pypy-dev] Trouble with ooparse_int/ooparse_float In-Reply-To: <9c0bb8a00708061846m265d5f30q7f47eac4442242e1@mail.gmail.com> References: <9c0bb8a00708061846m265d5f30q7f47eac4442242e1@mail.gmail.com> Message-ID: <46B860F6.9030306@gmail.com> Hi Paul! Paul deGrandis wrote: > Anto, Niko, and all, > > I've been trying to get some of the string and float tests to pass. The > trouble I'm running into is that ooparse_float doesn't know how to parse > ll_str_0, but for the life of me can't find where I need to be looking > or where I should override the ooparse_float method. I feel like it > should go in my opcodes.py file (translator/jvm). I'm not sure to understand the question. ooparse_float is an opcode which converts strings to floats. To implement it, you must add the corresponding definition to translator/jvm/opcodes.py; the simplest thing to do is probably to call java.lang.Double.parseDouble(); I don't know the exact syntax to use, but I guess Niko could help here :-). > Also, one question I had about long conversions, in > translator/jvm/test/test_float.py:22, there is a test for long > conversion. Tracing the results shows I get the correct answer for the > conversion, but my result is not an r_longlong, the tests returns a Java > Long. Any tips or hints? This happens because the same tests are used to test both the llinterpreter and the backends: when run in the llinterpreter, the return type is effectively r_longlong and the test runs fine, but when translated by a backend such as cli or jvm, the result is simply printed to stdout, without any information about the type: thus, there is no way to test if the result was effectively a r_longlong. Look at the source of the test, in rpython/test/test_rfloat.py:69: you see that the type-checking is done by calling the is_of_type method; now, look how the CLI test framerwork implements this method, in cli/test/runtest.py:299: as you can see, it simply returns always true, because it has no chance to test it. What you need to do is simply to do the same in jvm/test/runtest.py. ciao Anto From fijal at genesilico.pl Tue Aug 7 14:15:48 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Tue, 07 Aug 2007 14:15:48 +0200 Subject: [pypy-dev] rffi feature request In-Reply-To: <20070804181403.f15cdbb6.simon@arrowtheory.com> References: <20070803121413.bd2d992a.simon@arrowtheory.com> <20070804181403.f15cdbb6.simon@arrowtheory.com> Message-ID: <46B86274.2040401@genesilico.pl> Simon Burton wrote: > On Fri, 3 Aug 2007 12:14:13 -0700 > Simon Burton wrote: > > >> I would like to expose some functions as external >> symbols when i build a .so >> >> def foo(i, j): >> return i+j >> >> foo._expose_ = [rffi.INT, rffi.INT] >> > > It seems like this could also enable a plugin system for rpython, > and for example, (c or rpython) extension modules for the PyPy interpreter. > > Simon. > I kind of don't understand what are you trying to do. Could you explain in a bit more detail? (Ie are you trying to have rffi-rffi bridge between interpreter and compiled module?) Why not fix extcompiler instead? :. From simon at arrowtheory.com Wed Aug 8 18:14:40 2007 From: simon at arrowtheory.com (Simon Burton) Date: Wed, 8 Aug 2007 09:14:40 -0700 Subject: [pypy-dev] rffi feature request In-Reply-To: <46B86274.2040401@genesilico.pl> References: <20070803121413.bd2d992a.simon@arrowtheory.com> <20070804181403.f15cdbb6.simon@arrowtheory.com> <46B86274.2040401@genesilico.pl> Message-ID: <20070808091440.f8e06de1.simon@arrowtheory.com> On Tue, 07 Aug 2007 14:15:48 +0200 Maciek Fijalkowski wrote: > Simon Burton wrote: > > On Fri, 3 Aug 2007 12:14:13 -0700 > > Simon Burton wrote: > > > > > >> I would like to expose some functions as external > >> symbols when i build a .so > >> > >> def foo(i, j): > >> return i+j > >> > >> foo._expose_ = [rffi.INT, rffi.INT] > >> > > > > It seems like this could also enable a plugin system for rpython, > > and for example, (c or rpython) extension modules for the PyPy interpreter. > > > > Simon. > > > I kind of don't understand what are you trying to do. Could you explain > in a bit more detail? (Ie are you trying to have rffi-rffi bridge > between interpreter and compiled module?) Why not fix extcompiler instead? > > :. > well, the above code would produce: extern int foo(int i, int j) { return i+j; } (and perhaps an accompanying .h file) thereby providing an interface for other C programs. This is rffi producing rather than consuming a C interface. Simon. From jgustak at gmail.com Thu Aug 9 21:30:41 2007 From: jgustak at gmail.com (Jakub Gustak) Date: Thu, 9 Aug 2007 21:30:41 +0200 Subject: [pypy-dev] scheme interpreter [status report] In-Reply-To: References: Message-ID: Macros looks like working. One thing to be added: they are acting like first-class objects, but should not. Right now I am working on continuations. First a puzzle: What will return evaluation of s-expression like this: (call/cc call/cc) Or what will be bound to k after this s-expression: (define k (call/cc call/cc)) After some asking on #pypy I decided to create own implementation of continuations, and not try to use stackless. The reason is that coroutines are one-shot. So I created execution stack simulation, or rather stack for continuation frames. Right now not every macro uses it, so not every one is continuation-safe. If macro has defined method continue_tr it should get along with continuations well. Another issue: this implementations is stack-only so it introduce lot of overhead both on capturing and throwing continuations. And one more: every W_Procedure is continuations-safe but VERY slow, even when no capture occurs. I'd rather would like to think about how to throw continuations more generally, and not have to implement continue_tr for every W_Callable, than implementing capture differently. Or maybe try totally different approach e.g write every call in CPS, but it probably would require way too much changes right now. Cheers, Jakub Gustak From overminddl1 at gmail.com Fri Aug 10 03:21:21 2007 From: overminddl1 at gmail.com (OvermindDL1) Date: Thu, 9 Aug 2007 19:21:21 -0600 Subject: [pypy-dev] Stackless vs Erlang benchmarks In-Reply-To: <46B62C6B.5040306@gmx.de> References: <46B0605A.6010305@genesilico.pl> <46B62C6B.5040306@gmx.de> Message-ID: <3f49a9f40708091821u16f1055esb0777c2df4e1d4ac@mail.gmail.com> I made a library not long ago for stackless cpython meant to partially emulate that aspect of Erlang (called the library Pylang aptly enough, no I have no website, mostly personal use, I have never seen anyone in the community actively interested in such things from my Python experience, although if anyone voices opinions here then I could be convinced to release what I have). But you write your actors as if they are mini programs with their own even loops and so forth. They should not communicate by direct interaction of objects (unless you know beyond any doubt they will be in the same process, still better to just purely use message passing as I do with it) but rather strictly by message passing. Here is one of my tests that show the event loop and so forth (this test tests the message loop, does not show PID usage, network interaction, or anything else): _______________________________________________________ from ..core import getMessage # Yes, yes, I know, castrate me now # for using relative imports, this file is pylang/tests/test0.py and I do # need to do some file renaming and class shuffling if I release this from ..nameserver import NameServer # If I release, I need to rename # this as well since it does far more then the original design called for # This vars and this class are just to make ease of message passing # in this first test more simple, it is not necessary though TEST0_CONFIRMATION=0 TEST0_TESTSUBROUTINE=1 TEST0_ONLYSUBROUTINE=2 TEST0_EXIT=-1 class test0Message(object): def __init__(self, type): self.type=type def __str__(self): return "Message type: %i" % (self.type) def test0SubRoutine(): print "\t 0:Subroutine entered, waiting for signal only for me" e=getMessage(lambda e: isinstance(e, test0Message) and e.type==TEST0_ONLYSUBROUTINE) print "\t 0:Subroutine message receieved (should be %i):"%(TEST0_ONLYSUBROUTINE), e print "\t 0:Exiting subroutine" def test0Main(): while True: e=getMessage(timeout=1) if e is None: print "\t0:Timeout receieved" elif not isinstance(e, test0Message): print "\t0:Unknown message receieved, not of this test type" elif e.type==TEST0_CONFIRMATION: print "\t0:Confirmation received" elif e.type==TEST0_TESTSUBROUTINE: print "\t0:Testing subroutine" test0SubRoutine() elif e.type==TEST0_ONLYSUBROUTINE: print "0:ERROR: Only Subroutine message received when not in a callback" elif e.type==TEST0_EXIT: print "\t0:Exit request received, exiting process" return else: print "\t0:Unknown message type received, either need to set a condition to only get what you want, or dump this:", e def test0Generator(p): global exitScheduler print "Generator started, sending confirmation" p.sendMessage(test0Message(TEST0_CONFIRMATION)) Sleep(0.01) print "Generator sleeping for 1.5 seconds" Sleep(1.5) print "Sending test subroutine" p.sendMessage(test0Message(TEST0_TESTSUBROUTINE)) Sleep(0.01) print "Sending confirmation" p.sendMessage(test0Message(TEST0_CONFIRMATION)) Sleep(0.01) print "Sending only subroutine" p.sendMessage(test0Message(TEST0_ONLYSUBROUTINE)) Sleep(0.01) print "Sending confirmation" p.sendMessage(test0Message(TEST0_CONFIRMATION)) Sleep(0.01) print "Sending untyped test0Message" p.sendMessage(test0Message(10)) Sleep(0.01) print "Sending exit message" p.sendMessage(test0Message(TEST0_EXIT)) Sleep(0.01) print "Generator exiting" NameServer.__nameserver__.shutdown() # This sends a close message to all open # Processes and continues for a bit, if they have not all died by the timeout then they # are sent the taskletkill exception, which will kill them regardless. Also shuts down # the server and other such things def runTest0(): print "\t0:Testing local process and local message communication" NameServer(serverNode=NullServer()) # NullServer does not host squat. You can # create servernodes to handle tcp, udp, some other communication type, equiv to # a driver in Erlang, everything on the mesh network needs to use the same thing # without using a gateway inside the mesh of some sort to connect different ones p=Process(test0Main)() Process(test0Generator)(p) # Not the proper way to spawn processes now, # should call spawn(), but this was an early testcase that is still useful as-is NameServer.__nameserver__.run() _______________________________________________________ And it works as expected. The testcase that just tests network setup and destruction (no message passing or anything, just creation, verification, and destruction) is this: _______________________________________________________ from ..nameserver import NameServer from ..node import TCPServer from ..core import Process, getMessage def tempKillAll(waitTime=50, ns=None): if not ns: ns = NameServer.__nameserver__ Sleep(waitTime%10) for i in xrange(int(waitTime)-(waitTime%10), 0, -10): print "%i seconds until death" %(i) Sleep(10) print "dieing" ns.shutdown() def testServer(timeout=30, cookie='nocookie', localname='myhostA'): NameServer(serverNode=TCPServer(localname=localname, cookie=cookie, host='', port=42586, backlog=5)) Process(tempKillAll)(timeout) NameServer.__nameserver__.run() def testClient(timeout=3, cookie='nocookie', localname='myhostB', remoteURL='pylang://myhostA:42586/'): # no the port in the url is not # necessary as it is the default port used by this anyway NameServer(serverNode=TCPServer(localname=localname, cookie=cookie, host='', port=42587, backlog=5)) Process(tempKillAll)(timeout) Sleep(0.1) NameServer.__nameserver__.createNewNode(remoteURL) NameServer.__nameserver__.run() _______________________________________________________ The URL in full would actually be "pylang://myhostB:nocookie at myhostA:42586/". But if any are omitted then it uses the local hostname, the cookie, the remote host, the default port, etc... To connect to a remote named process (not an anonymous process) then you could get its PID with: PID("pylang://myhostB:nocookie at myhostA:42586/someNamedProcess") and if the connection to the remote node is not made then it will be made. If you have a Node pointer to an already connected node then you can just call a method on it to get a PID to a remote named process. If you send a message through a PID (the main way of sending messages) then they may or may not arrive because, although you can get a PID to a named process for example, the process may not actually exist. You can query a process to see if it exists which involves calling a ping method on the PID which will actually perform a test on the remote node (which may be on the same process/node, or a networked computer or what-not) and send back a specially crafted message saying pong or pang for success or fail with the PID as the param. A PID can always be converted to a pylang URL and vice-versa (as even anonymous processes have a guid generated for them). There can also be a type param of the url (example: "pylang://myhostB:nocookie at myhostA:42586;tcp/", then 'tcp' would be the type, standard url syntax) and if so then it will try to create a connection to the remote node using that connection (tcp, udp, ssl, pipe, raknet, whateverisMade) instead of the default registered to the nameserver by the servernode param, useful for connecting to a mesh that uses a different server type.. Processes can choose to save themselves for transmission across a network, save to a db or file, etc..., and they will retain their GUID. When they are serialized they are not serialized as a whole but they send a structure with the method call which is returned as an initial message to a process when it is restarted so it can reconstruct itself from it, the guid is always included and handled transparently. Thus, a process can be 'pickled' quite easilly, but it must be built for it (not hard to do, just needs to be done), reason being is that although stackless allows full tasklets to be pickled, some process may have some rather... adverse effects to being pickled (like one that handled any form of resources, db, file, etc...) hence why I have it do it this other, more 'erlangish' method. The message loop function (I do apologize for jumping around in thoughts...) is not just a standard pull the first off the queue like in a normal OS message loop, but has a signature of: "message getMessage(conditionFn=lambda e:True, timeout=-1)". When getMessage is called then it starts testing the messages in the queue for the process it was called from in order they were received, testing each with the conditionFn (allows for nice complex testing for something like "lambda e: isinstance(e, test0Message) and e.type==TEST0_ONLYSUBROUTINE" or "lambda e: isinstance(e, tuple) and len(e)==2 and e[0]==14" or whatever to simple things like just returning true to get the next top thing as is the default to passing in another function. If it good to get and empty out the queue on occasion to make sure it does not fill up with messages you do not want, slowing it down slightly. The other param is timeout (in seconds). If timeout equals -1 then it will wait forever for a matching message. If timeout equals 0 then it just test and returns a message if one matches that it already has, or it returns None (due to zero timeout). If greater then zero then it tests for a matching message, if none, then it waits up to the timeout for a message, if one arrives by that time then it returns it immediately, if not then it returns None when it reaches the timeout. It is not a spin-wait nor does it delay receiving the message or anything of so forth. The nameserver runs a pure tasklet (no message queue, no nothing like that) that just checks a list, if the top-most item on it is past the time delay then it sends a message to it and removes it then tests the next and repeats, else it inserts itself back into the stackless queue and waits for the next cycle to come by. A Process, when it has a timeout but no messages match, then it adds itself to the afore-mentioned list as a tuple of (timeTheTimeoutOccurs, processWaitChannel) then sorts it then blocks on its wait channel. If a message arrives at the process it tests the wait channel to see if it is blocked, if so it returns the message on that channel so it can then be tested, if it does not match then the process re-blocks, if it does match then it removes itself from the wait list and returns the message. If an exception occurs (taskletkill for example), it still properly removes itself from the list before it re-raises. Many other things done, but those are the major parts. All listed above is either working (near all of it) but may still need to be refined in its interface and so forth, or is partially done (pickling is more... hands-on currently, the interface described above is the new one I am partially moved to). The purpose of this was just to describe that erlangs strength can also be a strength for python as well. I really would like to move this library to PyPy when PyPy becomes usable. I have written that linked test above as well (not in this library, but pure stackless, a vastly different way then the article did though) and erlang still beat stackless cpython, hopefully pypy will fare better. On 8/5/07, Carl Friedrich Bolz wrote: > Hi Maciek > > Maciek Fijalkowski wrote: > > http://muharem.wordpress.com/2007/07/31/erlang-vs-stackless-python-a-first-benchmark/ > > > > Christian: with a dedication for you :) > > > > We should try pypy on this btw. > > seems a bit meaningless, given that one of erlang's most important > strengths is the possibility of using it to transparently across > multiple processes and especially multiple machines. > > Cheers, > > Carl Friedrich > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From fijal at genesilico.pl Fri Aug 10 11:53:25 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Fri, 10 Aug 2007 11:53:25 +0200 Subject: [pypy-dev] rffi feature request In-Reply-To: <20070808091440.f8e06de1.simon@arrowtheory.com> References: <20070803121413.bd2d992a.simon@arrowtheory.com> <20070804181403.f15cdbb6.simon@arrowtheory.com> <46B86274.2040401@genesilico.pl> <20070808091440.f8e06de1.simon@arrowtheory.com> Message-ID: <46BC3595.10606@genesilico.pl> > well, the above code would produce: > > extern int foo(int i, int j) > { > return i+j; > } > > (and perhaps an accompanying .h file) > > thereby providing an interface for other C programs. > This is rffi producing rather than consuming a C interface. > > Simon. > Hey Simon. It's doable, and not even hard, but it requires work. I kind of don't see the use-case for me right now (but I perfectly see the use-case in general), so it's a bit unlikely that I'll try to implement it, although I might help you a bit. Cheers, fijal :. From anto.cuni at gmail.com Sat Aug 11 12:02:42 2007 From: anto.cuni at gmail.com (Antonio Cuni) Date: Sat, 11 Aug 2007 12:02:42 +0200 Subject: [pypy-dev] rffi feature request In-Reply-To: <20070808091440.f8e06de1.simon@arrowtheory.com> References: <20070803121413.bd2d992a.simon@arrowtheory.com> <20070804181403.f15cdbb6.simon@arrowtheory.com> <46B86274.2040401@genesilico.pl> <20070808091440.f8e06de1.simon@arrowtheory.com> Message-ID: <46BD8942.2030309@gmail.com> Simon Burton wrote: >>>> I would like to expose some functions as external >>>> symbols when i build a .so >>>> >>>> def foo(i, j): >>>> return i+j >>>> >>>> foo._expose_ = [rffi.INT, rffi.INT] >>>> > well, the above code would produce: > > extern int foo(int i, int j) > { > return i+j; > } > > (and perhaps an accompanying .h file) > > thereby providing an interface for other C programs. > This is rffi producing rather than consuming a C interface. I think that what you need is similar to what carbonpython does: basically, carbonpython is a frontend for the translation toolchain that takes all the exposed functions/classes in an input file and produce a .NET libraries to be reused by other programs. Functions are exposed with the @export decorator: @export(int, int) def foo(a, b): return a+b The frontend creates a TranslationDriver objects, but instead of calling driver.setup() it calls driver.setup_library(), which allows to pass more than one entry point. I think all this could be reused for your needs. Then, the next step is to teach the backend how to deal with libraries; for genc, this would probably mean not to mangle functions names, to create a companying .h and to modify the Makefile to produce a .so instead of an executable. About name mangling: one possible solution could be to mangle the names as now and put some #define in the .h to allow the programmer to use non-mangled names. Probably this is the less invasive solution. ciao Anto From jason at jasondavies.com Mon Aug 13 00:29:29 2007 From: jason at jasondavies.com (Jason Davies) Date: Sun, 12 Aug 2007 23:29:29 +0100 Subject: [pypy-dev] Lua Frontend Message-ID: <46BF89C9.8000309@jasondavies.com> Hello, A couple of years ago I wrote a JIT compiler for Lua [1] for my final-year dissertation. Looking through the project-ideas page [2] I noticed there might be interest in writing Lua interpreter using PyPy. I'd be interested in doing this as a personal project. How would I go about getting started? Jason [1] http://www.lua.org/ [2] http://codespeak.net/pypy/dist/pypy/doc/project-ideas.html#write-a-new-front-end From gbowyer at fastmail.co.uk Mon Aug 13 16:49:41 2007 From: gbowyer at fastmail.co.uk (Greg Bowyer) Date: Mon, 13 Aug 2007 15:49:41 +0100 Subject: [pypy-dev] Silly idea - Pypy on Xen Message-ID: Taking a look at this http://www.informationweek.com/news/showArticle.jhtml?articleID=201311330&pgno=1&queryText= I wonder if the same could be done for pypy (this is just high level thinking, nothing serious) From fijal at genesilico.pl Mon Aug 13 17:21:26 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Mon, 13 Aug 2007 17:21:26 +0200 Subject: [pypy-dev] Silly idea - Pypy on Xen In-Reply-To: References: Message-ID: <46C076F6.1090707@genesilico.pl> Greg Bowyer wrote: > Taking a look at this > http://www.informationweek.com/news/showArticle.jhtml?articleID=201311330&pgno=1&queryText= > > I wonder if the same could be done for pypy (this is just high level > thinking, nothing serious) > > Look at unununium project (dead for quite a while), they tried to have a python shell as a basic thing for an operating system and had quite a lot of nice concepts. Right now obvious approach would be to target some virtualised environment instead of hardware itself, so it's not something very new. There are questions what abstraction do you provide for hard drive, etc. etc. What pypy has unique here is RPython, which can easily map low level details onto high level. Just few thoughts Cheers, fijal :. From cfbolz at gmx.de Mon Aug 13 17:23:36 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 13 Aug 2007 17:23:36 +0200 Subject: [pypy-dev] Silly idea - Pypy on Xen In-Reply-To: <46C076F6.1090707@genesilico.pl> References: <46C076F6.1090707@genesilico.pl> Message-ID: <46C07778.6050009@gmx.de> Maciek Fijalkowski wrote: > Greg Bowyer wrote: >> Taking a look at this >> http://www.informationweek.com/news/showArticle.jhtml?articleID=201311330&pgno=1&queryText= >> >> I wonder if the same could be done for pypy (this is just high level >> thinking, nothing serious) >> >> > Look at unununium project (dead for quite a while), they tried to have a > python shell as a basic thing for an operating system and had quite a > lot of nice concepts. Right now obvious approach would be to target some > virtualised environment instead of hardware itself, so it's not > something very new. There are questions what abstraction do you provide > for hard drive, etc. etc. > > What pypy has unique here is RPython, which can easily map low level > details onto high level. Yes, I kind of agree. IF you want to write an OS in Python then it probably makes most sense to target some virtualized environment and you might want to use RPython. Of course it depends very much what you actually want :-). Cheers, Carl Friedrich From fijal at genesilico.pl Mon Aug 13 17:28:46 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Mon, 13 Aug 2007 17:28:46 +0200 Subject: [pypy-dev] Silly idea - Pypy on Xen In-Reply-To: <46C07778.6050009@gmx.de> References: <46C076F6.1090707@genesilico.pl> <46C07778.6050009@gmx.de> Message-ID: <46C078AE.50508@genesilico.pl> Carl Friedrich Bolz wrote: > Maciek Fijalkowski wrote: >> Greg Bowyer wrote: >>> Taking a look at this >>> http://www.informationweek.com/news/showArticle.jhtml?articleID=201311330&pgno=1&queryText= >>> >>> >>> I wonder if the same could be done for pypy (this is just high level >>> thinking, nothing serious) >>> >>> >> Look at unununium project (dead for quite a while), they tried to >> have a python shell as a basic thing for an operating system and had >> quite a lot of nice concepts. Right now obvious approach would be to >> target some virtualised environment instead of hardware itself, so >> it's not something very new. There are questions what abstraction do >> you provide for hard drive, etc. etc. >> >> What pypy has unique here is RPython, which can easily map low level >> details onto high level. > > Yes, I kind of agree. IF you want to write an OS in Python then it > probably makes most sense to target some virtualized environment and > you might want to use RPython. Of course it depends very much what you > actually want :-). > > Cheers, > > Carl Friedrich > > :. > Come on, OS is an outdated concept. Cheers, fijal :. From gbowyer at fastmail.co.uk Mon Aug 13 17:54:18 2007 From: gbowyer at fastmail.co.uk (Greg Bowyer) Date: Mon, 13 Aug 2007 16:54:18 +0100 Subject: [pypy-dev] Silly idea - Pypy on Xen In-Reply-To: <46C078AE.50508@genesilico.pl> References: <46C076F6.1090707@genesilico.pl> <46C07778.6050009@gmx.de> <46C078AE.50508@genesilico.pl> Message-ID: Maciek Fijalkowski wrote: > Carl Friedrich Bolz wrote: >> Maciek Fijalkowski wrote: >>> Greg Bowyer wrote: >>>> Taking a look at this >>>> http://www.informationweek.com/news/showArticle.jhtml?articleID=201311330&pgno=1&queryText= >>>> >>>> >>>> I wonder if the same could be done for pypy (this is just high level >>>> thinking, nothing serious) >>>> >>>> >>> Look at unununium project (dead for quite a while), they tried to >>> have a python shell as a basic thing for an operating system and had >>> quite a lot of nice concepts. Right now obvious approach would be to >>> target some virtualised environment instead of hardware itself, so >>> it's not something very new. There are questions what abstraction do >>> you provide for hard drive, etc. etc. >>> >>> What pypy has unique here is RPython, which can easily map low level >>> details onto high level. >> Yes, I kind of agree. IF you want to write an OS in Python then it >> probably makes most sense to target some virtualized environment and >> you might want to use RPython. Of course it depends very much what you >> actually want :-). >> >> Cheers, >> >> Carl Friedrich >> >> :. >> > Come on, OS is an outdated concept. > > Cheers, > fijal > > > :. > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > I was thinking the selling point might be RPython From cfbolz at gmx.de Mon Aug 13 18:06:50 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 13 Aug 2007 18:06:50 +0200 Subject: [pypy-dev] Silly idea - Pypy on Xen In-Reply-To: <46C078AE.50508@genesilico.pl> References: <46C076F6.1090707@genesilico.pl> <46C07778.6050009@gmx.de> <46C078AE.50508@genesilico.pl> Message-ID: <46C0819A.6000302@gmx.de> Maciek Fijalkowski wrote: >> Yes, I kind of agree. IF you want to write an OS in Python then it >> probably makes most sense to target some virtualized environment and >> you might want to use RPython. Of course it depends very much what you >> actually want :-). >> >> Cheers, >> >> Carl Friedrich >> >> :. >> > Come on, OS is an outdated concept. Come on, you need a name to call the "thing-that-cares-for-your-resources-and-does-stuff" :-). Cheers, Carl Friedrich From cfbolz at gmx.de Mon Aug 13 18:17:21 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 13 Aug 2007 18:17:21 +0200 Subject: [pypy-dev] Lua Frontend In-Reply-To: <46BF89C9.8000309@jasondavies.com> References: <46BF89C9.8000309@jasondavies.com> Message-ID: <46C08411.1080902@gmx.de> Hi Jason, Jason Davies wrote: > A couple of years ago I wrote a JIT compiler for Lua [1] for my > final-year dissertation. Is this on the web somewhere? How did it work? > Looking through the project-ideas page [2] I > noticed there might be interest in writing Lua interpreter using PyPy. I think it would be interesting, yes. On the other hand, to me it is unclear whether this Lua interpreter would get any users, since the typical Lua users embed the Lua interpreter into a bigger project, which wouldn't be easily possible with a PyPy-based Lua-VM. > I'd be interested in doing this as a personal project. How would I go > about getting started? I would try to write an interpreter mirroring Lua's bytecode. This way you don't have to care for the tedious work of parsing and bytecode-compiling at first. I guess it is best to start with implementing some of the data structures of Lua, most notably tables (other data structures, like strings and floats are probably simple to implement, they just need to box rpython-level strings and floats). I guess you can have a naive implementation of tables in the beginning, though. Then you can probably start by implementing a simple interpreter for Lua's bytecode (probably by first writing some code that reads in Lua's bytecode and stores it nicely accessible somehow). If you are serious about this project and want to develop it under PyPy's umbrella you should ask for commit rights (e.g. by sending Michael Hudson a mail, micahel at gmail.com ). Cheers, Carl Friedrich From fijal at genesilico.pl Tue Aug 14 18:16:16 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Tue, 14 Aug 2007 18:16:16 +0200 Subject: [pypy-dev] [pypy-svn] r45656 - in pypy/branch/pypy-more-rtti-inprogress: . module/posix rlib rpython rpython/lltypesystem rpython/lltypesystem/module rpython/module rpython/module/test rpython/ootypesystem/module rpython/test translator translator/c translator/c/src translator/c/test translator/sandbox translator/sandbox/test In-Reply-To: <20070814161046.E284280E8@code0.codespeak.net> References: <20070814161046.E284280E8@code0.codespeak.net> Message-ID: <46C1D550.8020508@genesilico.pl> arigo at codespeak.net wrote: > For backup purposes, in-progress: convert more of the os module to rtti > and try to get rid of the rllib.ros module by making the native > interfaces RPythonic. This looks quite good in my opinion - seems that > we've finally learned a reasonable way to do external functions. > > Hooray! I would suggest to be consistent in names. Either rename it to rtti or keep using rffi (I know that fonts basically suck when it comes to this :) Seriously - I would like to move a bit rsocket, shall I do this on the branch or on trunk? Cheers, fijal :. From cfbolz at gmx.de Tue Aug 14 21:42:12 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Tue, 14 Aug 2007 21:42:12 +0200 Subject: [pypy-dev] Paper about accurate GC in C Message-ID: <46C20594.6050506@gmx.de> Hi all, the following paper describes a variant of the root-finding approach we use for the framework GC: http://citeseer.ist.psu.edu/henderson02accurate.html They use a "shadow stack" as well, but keep all their GCed locals always in it: The tradeoffs are slightly different than ours. They reach performance quite similar to that of the Boehm GC with a simple semispace collector. Chris Lattner told me that this is how LLVM lowers GC operations currently (since no backend directly supports the GC primitives in a better way). Cheers, Carl Friedrich From cfbolz at gmx.de Tue Aug 14 21:53:12 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Tue, 14 Aug 2007 21:53:12 +0200 Subject: [pypy-dev] [pypy-svn] r45656 - in pypy/branch/pypy-more-rtti-inprogress: . module/posix rlib rpython rpython/lltypesystem rpython/lltypesystem/module rpython/module rpython/module/test rpython/ootypesystem/module rpython/test translator translator/c translator/c/src translator/c/test translator/sandbox translator/sandbox/test In-Reply-To: <46C1D550.8020508@genesilico.pl> References: <20070814161046.E284280E8@code0.codespeak.net> <46C1D550.8020508@genesilico.pl> Message-ID: <46C20828.3010905@gmx.de> Maciek Fijalkowski wrote: > arigo at codespeak.net wrote: >> For backup purposes, in-progress: convert more of the os module to rtti >> and try to get rid of the rllib.ros module by making the native >> interfaces RPythonic. This looks quite good in my opinion - seems that >> we've finally learned a reasonable way to do external functions. >> >> > Hooray! Seconded: very cool! Thanks to all the people that are working on this (I am looking at you, Maciek :-) ). Cheers, Carl Friedrich From jgustak at gmail.com Fri Aug 17 14:35:20 2007 From: jgustak at gmail.com (Jakub Gustak) Date: Fri, 17 Aug 2007 14:35:20 +0200 Subject: [pypy-dev] scheme interpreter [status report] In-Reply-To: References: Message-ID: This is last scheme status report during SoC. So I will sum things up in the end. > Macros looks like working. > One thing to be added: they are acting like first-class objects, but should not. Found a bug, when ellipsis matched zero objects, fized now. Scheme48 acts differently in some cases. On the other hand in this cases mzScheme works just as expected. In this meaning macros are evil. There is no explicit specification how they should behave in r5rs. They look like added to the specification in last minute. Continuations: > Another issue: this implementations is stack-only so it introduce lot > of overhead both on capturing and throwing continuations. > > And one more: every W_Procedure is continuations-safe but VERY slow, > even when no capture occurs. I made them a little faster by decreasing number of operations, but still for every non-tail-call a new instance of ContinuationFrame is created and is stored on stack. On every return it is popped form stack and forgotten. When no capture will happen, it is time and memory waste. > I'd rather would like to think about how to throw continuations more > generally, and not have to implement continue_tr for every W_Callable, > than implementing capture differently. No progress in this direction. Still every W_Callable to be continuations friendly has to use eval_cf instead of eval and must implement continue_tr method. > Or maybe try totally different approach e.g write every call in CPS, > but it probably would require way too much changes right now. Nor here. Summarization: * Interactive interpreter (not RPythonic) lang/scheme/interactive.py * Can be translated using: translation/goal/targetscheme.py PyPy Scheme Interpreter features: * uses rpythonic packrat parser * every "syntax" definitions are implemented * quotations and quasi-quotations * delayed evaluation * proper tail-recursion * hygienic macros * partly working continuations (some macros are not continuations friendly yet). * no dynamic-wind procedure Known issues and "to be added": * lambda does not check if it is called with correct arguments number * macros are acting like first-class objects, though they shouldn't * missing input/output procedures * no support for chars, vectors * strings are supported, but no string-operations procedures * quite a few procedures are missing, but they can be easily added * no call-with-values and values * missing some "library syntax", can be defined using macros What's next? I will probably mysteriously disappear and became grave-digger or hermit. Never touch computer again and live happily ever after ;-). Cheers, Jakub Gustak From fijal at genesilico.pl Fri Aug 17 15:58:43 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Fri, 17 Aug 2007 15:58:43 +0200 Subject: [pypy-dev] scheme interpreter [status report] In-Reply-To: References: Message-ID: <46C5A993.4040901@genesilico.pl> Jakub Gustak wrote: > This is last scheme status report during SoC. So I will sum things up > in the end. > > Summarization: > * Interactive interpreter (not RPythonic) lang/scheme/interactive.py > * Can be translated using: translation/goal/targetscheme.py > > PyPy Scheme Interpreter features: > * uses rpythonic packrat parser > * every "syntax" definitions are implemented > * quotations and quasi-quotations > * delayed evaluation > * proper tail-recursion > * hygienic macros > * partly working continuations (some macros are not continuations friendly yet). > * no dynamic-wind procedure > > > I would say congratulations! (Although I don't know scheme that much to be able to evaluate this, even positively) Cheers, fijal :. From arigo at tunes.org Fri Aug 17 20:03:30 2007 From: arigo at tunes.org (Armin Rigo) Date: Fri, 17 Aug 2007 20:03:30 +0200 Subject: [pypy-dev] scheme interpreter [status report] In-Reply-To: References: Message-ID: <20070817180330.GA5301@code0.codespeak.net> Hi Jakub, On Fri, Aug 17, 2007 at 02:35:20PM +0200, Jakub Gustak wrote: > This is last scheme status report during SoC. So I will sum things up > in the end. Thanks for the good work you did during the summer! The result looks good to me, even with the missing features, but we'd really need a Scheme guy to guide us from now on, if we want to make the interpreter more usable. > > I'd rather would like to think about how to throw continuations more > > generally, and not have to implement continue_tr for every W_Callable, > > than implementing capture differently. > > No progress in this direction. > Still every W_Callable to be continuations friendly has to use eval_cf > instead of eval and must implement continue_tr method. > > > Or maybe try totally different approach e.g write every call in CPS, > > but it probably would require way too much changes right now. > > Nor here. Indeed, the current approach allows impressive-looking tests to pass, but the code is quite verbose and inefficient. We can start thinking at some point about using the stackless and/or GC-based cloning operations of RPython as a more orthogonal alternative, but that's an involved topic. > What's next? > I will probably mysteriously disappear and became grave-digger or > hermit. Never touch computer again and live happily ever after ;-). If you do, then farewell! In the event that you're interested to continue, though (:-), here are in summary some possible future directions: * get some Scheme people interested in the project, try to run real Scheme projects * implement more of the missing features * think how to do continuations with stackless or GC-based cloning * also, we could try to integrate pypy/lang/scheme and pypy/interpreter in one of the many possible ways, to offer a combined Python+Scheme environment... A bientot, Armin. From fijal at genesilico.pl Sat Aug 18 13:26:55 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Sat, 18 Aug 2007 13:26:55 +0200 Subject: [pypy-dev] PyPy work plan Message-ID: <46C6D77F.3020807@genesilico.pl> As you probably have observed already, for the last few months pypy is slowly approaching "being usable" state instead of "having more ultra-cool features", as we've got enough of them to be willing to use them. As we're moving towards maintaining pypy as an open source project I would suggest to look around parts of pypy and have list of people who are willing to maintain certain parts. Maintain not in a sense of developing those, but rather accepting/rejecting patches, keeping it up to date with other parts, etc. I would also suggest to move parts with no obvious maintainer somewhere else in svn (branch? tag?) not to confuse people who checkout whole svn repository with parts never used by anyone. This option would make it easier to: * have people entering project, since it would be obvious who to contact for different parts. * maintain whole codebase, since some cleanup changes are pervasive enough to break different parts. * this would also hopefully reduce whole code size. I've also checked in pypy parts (more or less) into http://codespeak.net/svn/pypy/dist/pypy/doc/maintainers.txt feel free to add yourself wherever you like and to modify this list as well What do you think? Cheers, fijal :. From pedronis at openendsystems.com Sat Aug 18 17:08:34 2007 From: pedronis at openendsystems.com (Samuele Pedroni) Date: Sat, 18 Aug 2007 17:08:34 +0200 Subject: [pypy-dev] PyPy work plan In-Reply-To: <46C6D77F.3020807@genesilico.pl> References: <46C6D77F.3020807@genesilico.pl> Message-ID: <46C70B72.4050704@openendsystems.com> Maciek Fijalkowski wrote: > > I've also checked in pypy parts (more or less) into > http://codespeak.net/svn/pypy/dist/pypy/doc/maintainers.txt > feel free to add yourself wherever you like and to modify this list as well > I looked at it, didn't really know how to fill it in without being confused or confusing. I removed it. > What do you think? > I think identifying orphaned areas, agree on them and act accordingly is useful. I suppose there are indeed specific areas with specific maintainers, I suppose listing those may be useful. Most of PyPy is too interconnected and demanding for anything but global ownership to make sense tough, at least at this point in time. My check-ins on the 4-5 of August are a case in point. From lac at openend.se Sat Aug 18 17:24:27 2007 From: lac at openend.se (Laura Creighton) Date: Sat, 18 Aug 2007 17:24:27 +0200 Subject: [pypy-dev] PyPy work plan In-Reply-To: Message from Maciek Fijalkowski of "Sat, 18 Aug 2007 13:26:55 +0200." <46C6D77F.3020807@genesilico.pl> References: <46C6D77F.3020807@genesilico.pl> Message-ID: <200708181524.l7IFORYH028243@theraft.openend.se> Hi Simon. I never like to quote somebody's email without informing them. I am about to use ICCARUS (why 2 Cs) as an example of what I am looking for. The rest is the middle of a discusison that you are not expected to understand. Thank you for ICCARUS. -- Laura In a message of Sat, 18 Aug 2007 13:26:55 +0200, Maciek Fijalkowski writes: > >As you probably have observed already, for the last few months pypy >is slowly approaching "being usable" state instead of "having more >ultra-cool >features", as we've got enough of them to be willing to use them. >As we're moving towards maintaining pypy as an open source project I woul >d >suggest to look around parts of pypy and have list >of people who are willing to maintain certain parts. Maintain not in a se >nse >of developing those, but rather accepting/rejecting patches, keeping it u >p >to date with other parts, etc. I would also suggest >to move parts with no obvious maintainer somewhere else in svn >(branch? tag?) not to confuse people who checkout whole svn repository >with parts never used by anyone. This option would make it easier to: > >* have people entering project, since it would be obvious who to > contact for different parts. > >* maintain whole codebase, since some cleanup changes are pervasive > enough to break different parts. > >* this would also hopefully reduce whole code size. > >I've also checked in pypy parts (more or less) into >http://codespeak.net/svn/pypy/dist/pypy/doc/maintainers.txt >feel free to add yourself wherever you like and to modify this list as we >ll > >What do you think? > >Cheers, >fijal As mentioned just in irc, I am unhappy with the dividing up of pypy into pieces with maintainers. That way the people who envision the small can dominate those who envision the large, and one of the strengths of this project is the sheer visionary power of some of its members. on the other hand, some tidying would be nice. and we sure as hell could be friendlier for newbies. I am now looking at code visualisation tools, and time visualisation tools. I get the idea that if we could visualise things better, we could undersgtand who knows what about what, and what else, then we would need less in the way of 'task management'. for static things I am thinking of: http://simile.mit.edu/timeline/ or everything at the top level simile.mit.edu domain. But then yesterday I read this on the pygame mailing list. ..... From: "Simon Wittber" To: pygame-users at seul.org Subject: [pygame] 3D Social Network Visualisation A few evening ago we demonstrated ICCARUS at a web party. It is a piece of software which provides a visualisation of social networks in three dimensions :-) It uses pygame and a Pyrexed OpenGL lib called GFX. You can see screencast here: http://scouta.com/faves/8raO17jT8Y3/ and a vid of the presentation here: http://blog.viddler.com/cdevroe/webjam-iccarus/ Thanks to python and pygame, this project was conceived and implemented over 3 days. ...... I thought this was way cool. Especially the 3 days part. I want our code base to work like this. Who knows what? stuff that everybody knows, stuff that carl freidrich and armin and maciek and samuele know, stuff that nobody knows but samuele, and even -- 'stuff that nobody knows' somebody did at one time but we all have forgotten by now. This is a half-baked proposal for code base management, normally I would not mention this at all, but some of us are getting grumpy, and I thought that other ideas, however weird might be welcome. Laura From lac at openend.se Sat Aug 18 17:30:01 2007 From: lac at openend.se (Laura Creighton) Date: Sat, 18 Aug 2007 17:30:01 +0200 Subject: [pypy-dev] scheme interpreter [status report] In-Reply-To: Message from Armin Rigo of "Fri, 17 Aug 2007 20:03:30 +0200." <20070817180330.GA5301@code0.codespeak.net> References: <20070817180330.GA5301@code0.codespeak.net> Message-ID: <200708181530.l7IFU1k8029132@theraft.openend.se> >> What's next? >> I will probably mysteriously disappear and became grave-digger or >> hermit. Never touch computer again and live happily ever after ;-). This sort of peace is good for 2 weeks, a month, even a year in severe burn-out cases. UNless you plan on being struck by lightning on your vacation in the hermitage or graveyard, I would expect the technical world to grab its claw around you once again. Know that you are welcome here whenever this happens. :-) Laura From santagada at gmail.com Sun Aug 19 00:22:22 2007 From: santagada at gmail.com (Leonardo Santagada) Date: Sat, 18 Aug 2007 19:22:22 -0300 Subject: [pypy-dev] Javascript Frontend (and mybe some unrelated future plans) Message-ID: I've been on a cave for some time, but I want to get back to work more on pypy but I don't know what to do with the javascript interpreter. While I was working with it it was getting better but almost all the architecture work i've done alone and I'm not happy with the results. The code looks kinda ugly and I don't really know how to get it into a good shape. Maybe I need someone to take a look at it. Also there is some stuff that I don't know what to do about like automatic semicolon insertion and regexes. And them there is the stuff that I really think is essential but that is missing wich is a simple RPython API to make native objects to javascript (that we could use to implement DOM, Regexes and XMLHttpRequests using rsockets). All those are internal architectural problems, them there are the external ones like, who would like to use our javascript interpreter? Web javascript guys for testing and to make their tools using js, or people wanting to know how to make an interpreter on top of pypy (people always says that a scheme interpreter would be easier for that). So those are my troubles with the js interpreter, as it is now I don't know what it adds to the pypy project. But I'm always a mail away and I still track the development of pypy and can maintain it. I still want to work more with pypy, just don't know what to do with the js frontend now. Maybe I should take another part of pypy to work with, I want to know what you guys have to say about all that i've written. As always I want to express my gratitude to everyone as you all really helped me a lot. I learned so much working on pypy, I don't know what to say but thanks. best wishes, -- Leonardo Santagada From arigo at tunes.org Sun Aug 19 12:33:09 2007 From: arigo at tunes.org (Armin Rigo) Date: Sun, 19 Aug 2007 12:33:09 +0200 Subject: [pypy-dev] pypy-dev@codespeak.net Message-ID: <20070819103308.GA28257@code0.codespeak.net> Hi all, Those that follow IRC already know it, but it's worth being announced a bit more widely: I've been working on a form of sandboxing for RPython programs, which now seems to work for the whole of PyPy. It's "sandboxing" as in "full virtualization", but done in normal C with no OS support at all. It's a two-processes model: we can translate PyPy to a special "pypy-c-sandbox" executable, which is safe in the sense that it doesn't do any library or system call - instead, whenever it would like to perform such an operation, it marshals the operation name and the arguments to its stdout and it waits for the marshalled result on its stdin. This pypy-c-sandbox process is meant to be run by an outer "controller" program that answers to these operation requests. The pypy-c-sandbox program is obtained by adding a transformation during translation, which turns all RPython-level external function calls into stubs that do the marshalling/waiting/unmarshalling. An attacker that tries to escape the sandbox is stuck within a C program that contains no external function call at all except to write to stdout and read from stdin. (It's still attackable, e.g. by exploiting segfault-like situations, but as far as I can tell - unlike CPython - any RPython program is really robust against this kind of attack, at least if we enable the extra checks that all RPython list and string indexing are in range. Alternatively, on Linux there is a lightweight OS-level sandboxing technique available by default - google for 'seccomp'.) The outer controller is a plain Python program that can run in CPython or a regular PyPy. It can perform any virtualization it likes, by giving the subprocess any custom view on its world. For example, while the subprocess thinks it's using file handles, in reality the numbers are created by the controller process and so they need not be (and probably should not be) real OS-level file handles at all. In the demo controller I've implemented there is simply a mapping from numbers to file-like objects. The controller answers to the "os_open" operation by translating the requested path to some file or file-like object in some virtual and completely custom directory hierarchy. The file-like object is put in the mapping with any unused number >= 3 as a key, and the latter is returned to the subprocess. The "os_read" operation works by mapping the pseudo file handle given by the subprocess back to a file-like object in the controller, and reading from the file-like object. Enough explanations, here's a how-to: For now, this lives in a branch at the 'pypy' level: cd ..../pypy-dist/pypy svn switch http://codespeak.net/svn/pypy/branch/pypy-more-rtti-inprogress/ In pypy/translator/goal: ./translate.py --sandbox --source targetpypystandalone.py --withoutmod-gc --withoutmod-_weakref (the gc and _weakref modules are disabled because they are a bit too unsafe, in the sense that they could allow bogus memory accesses) Then in the directory where the sources were created, compile with the extra RPython-level assertions enabled: make CFLAGS="-O2 -DRPY_ASSERT" mv testing_1 /some/path/pypy-c-sandbox Run it with the tools in the pypy/translator/sandbox directory: ./pypy_interact.py /some/path/pypy-c-sandbox [args...] Just like pypy-c, if you pass no argument you get the interactive prompt. In theory it's impossible to do anything bad or read a random file on the machine from this prompt. (There is no protection against using all the RAM or CPU yet.) To pass a script as an argument you need to put it in a directory along with all its dependencies, and ask pypy_interact to export this directory (read-only) to the subprocess' virtual /tmp directory with the --tmp=DIR option. Not all operations are supported; e.g. if you type os.readlink('...'), the controller crashes with an exception and the subprocess is killed. Other operations make the subprocess die directly with a "Fatal RPython error". None of this is a security hole; it just means that if you try to run some random program, it risks getting killed depending on the Python built-in functions it tries to call. By the way, as you should have realized, it's really independent from the fact that it's PyPy that we are translating. Any RPython program should do. I've successfully tried it on the JS interpreter. The controller is only called "pypy_interact" because it emulates a file hierarchy that makes pypy-c-sandbox happy - it contains (read-only) virtual directories like /bin/lib-python and /bin/pypy/lib and it pretends that the executable is /bin/pypy-c. A bientot, Armin. From grobinson at goombah.com Sun Aug 19 19:40:52 2007 From: grobinson at goombah.com (Gary Robinson) Date: Sun, 19 Aug 2007 13:40:52 -0400 Subject: [pypy-dev] How's the JIT coming along? Message-ID: <20070819134052388521.b4a47875@goombah.com> The docs for PyPy 1.0 it say: > A foreword of warning about the JIT of PyPy as of March 2007: single > functions doing integer arithmetic get great speed-ups; about > anything else will be a bit slower with the JIT than without. We are > working on this - you can even expect quick progress, because it is > mostly a matter of adding a few careful hints in the source code of > the Python interpreter of PyPy. Given the mention of the possibility of "quick progress", I'm wondering where the JIT is 5 months later... does anyone care to give a status report and/or any estimates of when it will be more broadly useful? Thanks, Gary -- Gary Robinson CTO Emergent Music, LLC grobinson at goombah.com 207-942-3463 Company: http://www.goombah.com Blog: http://www.garyrobinson.net From arigo at tunes.org Mon Aug 20 17:30:51 2007 From: arigo at tunes.org (Armin Rigo) Date: Mon, 20 Aug 2007 17:30:51 +0200 Subject: [pypy-dev] pypy-dev@codespeak.net In-Reply-To: <20070819103308.GA28257@code0.codespeak.net> References: <20070819103308.GA28257@code0.codespeak.net> Message-ID: <20070820153051.GA19399@code0.codespeak.net> Hi, On Sun, Aug 19, 2007 at 12:33:09PM +0200, Armin Rigo wrote: > Then in the directory where the sources were created, compile with the > extra RPython-level assertions enabled: > > make CFLAGS="-O2 -DRPY_ASSERT" > mv testing_1 /some/path/pypy-c-sandbox You can now just say 'make llsafer' instead. This enables a new flag, -DRPY_LL_ASSERT, which differs from RPY_ASSERT in some ways explained in translator/c/src/support.h and which is better suited for this situation. I would say that the resulting sandboxed PyPy is quite safe then - at most, it will abort() itself if you play too strange tricks with 'exec new.code(...)'. For paranoia bonus points you can enable both RPY_ASSERT and RPY_LL_ASSERT. For what it's worth, the -DRPY_LL_ASSERT inserts tons of checks everywhere, for an acceptable performance hit (~10%?). A bientot, Armin. From arigo at tunes.org Mon Aug 20 17:41:42 2007 From: arigo at tunes.org (Armin Rigo) Date: Mon, 20 Aug 2007 17:41:42 +0200 Subject: [pypy-dev] pypy-dev@codespeak.net In-Reply-To: <20070820153051.GA19399@code0.codespeak.net> References: <20070819103308.GA28257@code0.codespeak.net> <20070820153051.GA19399@code0.codespeak.net> Message-ID: <20070820154142.GA20520@code0.codespeak.net> Re-hi, On Mon, Aug 20, 2007 at 05:30:51PM +0200, Armin Rigo wrote: > > make CFLAGS="-O2 -DRPY_ASSERT" > > mv testing_1 /some/path/pypy-c-sandbox > > You can now just say 'make llsafer' instead. While I was at it I enabled RPY_LL_ASSERT automatically in all programs translated with --sandbox, assuming that better safety by default is a good idea in this case. So now you can say ./translate.py --sandbox and get a correctly compiled result for free. A bientot, Armin. From bea at changemaker.nu Tue Aug 21 15:54:28 2007 From: bea at changemaker.nu (=?ISO-8859-1?Q?Beatrice_D=FCring?=) Date: Tue, 21 Aug 2007 15:54:28 +0200 Subject: [pypy-dev] How's the JIT coming along? In-Reply-To: <20070819134052388521.b4a47875@goombah.com> References: <20070819134052388521.b4a47875@goombah.com> Message-ID: <46CAEE94.1060004@changemaker.nu> Hi there Gary Robinson skrev: > The docs for PyPy 1.0 it say: > > >> A foreword of warning about the JIT of PyPy as of March 2007: single >> functions doing integer arithmetic get great speed-ups; about >> anything else will be a bit slower with the JIT than without. We are >> working on this - you can even expect quick progress, because it is >> mostly a matter of adding a few careful hints in the source code of >> the Python interpreter of PyPy. >> > > Given the mention of the possibility of "quick progress", I'm wondering where the JIT is 5 months later... does anyone care to give a status report and/or any estimates of when it will be more broadly useful? > > Thanks, > Gary > > > Sorry for not having responded on this issue even prior to your email. We have had discussions some of us to go for a sprint in end week of October, probably in Germany, where we would focus on clean-up work/refactoring and also to discuss plans for a 1.1 release (scope, content, time). The sprint plan need to be discussed in the upcoming week with a sprint announcement going out. _My guess_ is that the JIT of course would also be involved in the scope of that sprint (clean up/refactoring) and that the 1.1 release would also contain results from such a clean up. Some of the reasons for the "silence" in the JIT area is partly because of people having vacation and post-eu sabbaticals, we have been going to conferences such as (Europython, Dyla) to do talks about the JIT to get input as well as focusing on getting external functions, RPython and PyPy to work more smoothly (which was feedback from users and would be pypy-users at Europython in Vilnius). But yes - we might have been better at expressing this. If i might ask - what are the use cases you are thinking about when talking about the JIT (and/or the JIT generator?). Cheers Bea D?ring -- Beatrice D?ring, PMP Change Maker J?rntorget 3 413 04 Gothenburg www.changemaker.nu email: bea at changemaker.nu Phone: +46 31 7750940 Cellphone: +46 734 22 89 06 PyPy: www.pypy.org From simon at arrowtheory.com Tue Aug 21 18:06:37 2007 From: simon at arrowtheory.com (Simon Burton) Date: Tue, 21 Aug 2007 09:06:37 -0700 Subject: [pypy-dev] buffer class and ll2ctypes problem Message-ID: <20070821090637.f57b0323.simon@arrowtheory.com> I've been working with Richard on a new (rpython) buffer class. See attached for two approaches. The richbuf seems much clearer, but i'm not sure how to get other types out of it (like long long or short), or do byte swapping. I don't understand this much, can someone point the right direction ? Ie. which buf is the right direction to go. (I would also like to re-visit the array type in pypy/rpython/numpy, and perhaps think more generally about __getitem__ in rpython.) The rffi buffer (buffers.py) fails on CPython: $ python buffers.py Traceback (most recent call last): File "buffers.py", line 55, in test_run() File "buffers.py", line 51, in test_run assert demo(1234) == 1234 File "buffers.py", line 39, in demo buf.put_int32(value) File "buffers.py", line 31, in put_int32 intp = rffi.cast(INTP, buf) File "/home/simon/site-packages/pypy/rpython/lltypesystem/ll2ctypes.py", line 491, in force_cast cvalue = lltype2ctypes(value) File "/home/simon/site-packages/pypy/rpython/lltypesystem/ll2ctypes.py", line 336, in lltype2ctypes convert_struct(container) File "/home/simon/site-packages/pypy/rpython/lltypesystem/ll2ctypes.py", line 161, in convert_struct convert_struct(parent) File "/home/simon/site-packages/pypy/rpython/lltypesystem/ll2ctypes.py", line 165, in convert_struct cstruct = cls._malloc() File "/home/simon/site-packages/pypy/rpython/lltypesystem/ll2ctypes.py", line 89, in _malloc raise TypeError, "array length must be an int" TypeError: array length must be an int Simon. -------------- next part -------------- A non-text attachment was scrubbed... Name: buffers.py Type: text/x-python Size: 1328 bytes Desc: not available Url : http://codespeak.net/pipermail/pypy-dev/attachments/20070821/3aa80795/attachment.py -------------- next part -------------- A non-text attachment was scrubbed... Name: richbuf.py Type: text/x-python Size: 1370 bytes Desc: not available Url : http://codespeak.net/pipermail/pypy-dev/attachments/20070821/3aa80795/attachment-0001.py From david at ar.media.kyoto-u.ac.jp Wed Aug 22 04:22:41 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 22 Aug 2007 11:22:41 +0900 Subject: [pypy-dev] How's the JIT coming along? In-Reply-To: <46CAEE94.1060004@changemaker.nu> References: <20070819134052388521.b4a47875@goombah.com> <46CAEE94.1060004@changemaker.nu> Message-ID: <46CB9DF1.8060103@ar.media.kyoto-u.ac.jp> Beatrice D?ring wrote: > Sorry for not having responded on this issue even prior to your email. > > We have had discussions some of us to go for a sprint in end week of > October, > probably in Germany, where we would focus on clean-up work/refactoring and > also to discuss plans for a 1.1 release (scope, content, time). The > sprint plan need to > be discussed in the upcoming week with a sprint announcement going out. > > _My guess_ is that the JIT of course would also be involved in the scope > of that sprint > (clean up/refactoring) and that the 1.1 release would also contain > results from such a clean up. > > Some of the reasons for the "silence" in the JIT area is partly because > of people having vacation > and post-eu sabbaticals, we have been going to conferences such as > (Europython, Dyla) to do > talks about the JIT to get input as well as focusing on getting external > functions, RPython and > PyPy to work more smoothly (which was feedback from users and would be > pypy-users > at Europython in Vilnius). But yes - we might have been better at > expressing this. > > If i might ask - what are the use cases you are thinking about when > talking about the JIT > (and/or the JIT generator?). I cannot speak for Garry, but I myself would be interested in pypy for numerical computing in python. Basically, there are cases where numpy is not enough and require coding in C. The two cases I am thinking are: - recursive algorithms: this means many functions calls, which are too expensive in python (eg: imagine you have a two buffers of many float x, and you want to compute f(x[i], [y[i], nu[i]) = x[i+1] = x[i] + nu[i] * (x[i] - y[i]); even using ctypes for the trivial computation in C kills performances because of the many calls) - algorithms which require many temporaries to be efficient in numpy. Both of them, if my understanding is right, would be perfect exemples of easy to optimize using JIT. David From santagada at gmail.com Wed Aug 22 06:43:01 2007 From: santagada at gmail.com (Leonardo Santagada) Date: Wed, 22 Aug 2007 01:43:01 -0300 Subject: [pypy-dev] How's the JIT coming along? In-Reply-To: <46CB9DF1.8060103@ar.media.kyoto-u.ac.jp> References: <20070819134052388521.b4a47875@goombah.com> <46CAEE94.1060004@changemaker.nu> <46CB9DF1.8060103@ar.media.kyoto-u.ac.jp> Message-ID: <4B0F48B0-D9AD-416E-81F6-3A839C002620@gmail.com> Em Aug 21, 2007, ?s 11:22 PM, David Cournapeau escreveu: > I cannot speak for Garry, but I myself would be interested in pypy for > numerical computing in python. Basically, there are cases where > numpy is > not enough and require coding in C. The two cases I am thinking are: > - recursive algorithms: this means many functions calls, which are > too expensive in python (eg: imagine you have a two buffers of many > float x, and you want to compute f(x[i], [y[i], nu[i]) = x[i+1] = x > [i] + > nu[i] * (x[i] - y[i]); even using ctypes for the trivial > computation in > C kills performances because of the many calls) > - algorithms which require many temporaries to be efficient in > numpy. > > Both of them, if my understanding is right, would be perfect > exemples of > easy to optimize using JIT. I'm guessing here, but to my knoledge to make python functions more lightweight using jit would only work by making tail call optimization, but for that you can also do the "same" optimization by using a explicit stack and not use function calls at all. -- Leonardo Santagada From david at ar.media.kyoto-u.ac.jp Wed Aug 22 07:56:15 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 22 Aug 2007 14:56:15 +0900 Subject: [pypy-dev] How's the JIT coming along? In-Reply-To: <4B0F48B0-D9AD-416E-81F6-3A839C002620@gmail.com> References: <20070819134052388521.b4a47875@goombah.com> <46CAEE94.1060004@changemaker.nu> <46CB9DF1.8060103@ar.media.kyoto-u.ac.jp> <4B0F48B0-D9AD-416E-81F6-3A839C002620@gmail.com> Message-ID: <46CBCFFF.2060708@ar.media.kyoto-u.ac.jp> Leonardo Santagada wrote: > > Em Aug 21, 2007, ?s 11:22 PM, David Cournapeau escreveu: >> I cannot speak for Garry, but I myself would be interested in pypy for >> numerical computing in python. Basically, there are cases where numpy is >> not enough and require coding in C. The two cases I am thinking are: >> - recursive algorithms: this means many functions calls, which are >> too expensive in python (eg: imagine you have a two buffers of many >> float x, and you want to compute f(x[i], [y[i], nu[i]) = x[i+1] = x[i] + >> nu[i] * (x[i] - y[i]); even using ctypes for the trivial computation in >> C kills performances because of the many calls) >> - algorithms which require many temporaries to be efficient in >> numpy. >> >> Both of them, if my understanding is right, would be perfect exemples of >> easy to optimize using JIT. > > I'm guessing here, but to my knoledge to make python functions more > lightweight using jit would only work by making tail call > optimization, but for that you can also do the "same" optimization by > using a explicit stack and not use function calls at all. Mmh, tail call optimization is useful for recursion, right ? I should not have used recursive in my post, actually, because the recursive I am talking about is not really what is meant in programming. The probleme in my case is not so much the stack, but the function call by itself. For example, the following python code: class A: def _foo(self): return None def foo(self): return self._foo() Is extremely slow compared to a compiled language like C. On my computer, I can only execute the above function around 1 millon times a second (it takes around 3500 cycles by using %timeit from ipython). This forces me to avoid functions in some performance intensive code, which is ugly. Basically, I am interested in the kind of things described there: http://www.avibryant.com/2006/09/index.html. David From cfbolz at gmx.de Wed Aug 22 08:24:12 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Wed, 22 Aug 2007 08:24:12 +0200 Subject: [pypy-dev] How's the JIT coming along? In-Reply-To: <4B0F48B0-D9AD-416E-81F6-3A839C002620@gmail.com> References: <20070819134052388521.b4a47875@goombah.com> <46CAEE94.1060004@changemaker.nu> <46CB9DF1.8060103@ar.media.kyoto-u.ac.jp> <4B0F48B0-D9AD-416E-81F6-3A839C002620@gmail.com> Message-ID: <46CBD68C.5010604@gmx.de> Leonardo Santagada wrote: > Em Aug 21, 2007, ?s 11:22 PM, David Cournapeau escreveu: >> I cannot speak for Garry, but I myself would be interested in pypy for >> numerical computing in python. Basically, there are cases where >> numpy is >> not enough and require coding in C. The two cases I am thinking are: >> - recursive algorithms: this means many functions calls, which are >> too expensive in python (eg: imagine you have a two buffers of many >> float x, and you want to compute f(x[i], [y[i], nu[i]) = x[i+1] = x >> [i] + >> nu[i] * (x[i] - y[i]); even using ctypes for the trivial >> computation in >> C kills performances because of the many calls) >> - algorithms which require many temporaries to be efficient in >> numpy. >> >> Both of them, if my understanding is right, would be perfect >> exemples of >> easy to optimize using JIT. > > I'm guessing here, but to my knoledge to make python functions more > lightweight using jit would only work by making tail call > optimization, but for that you can also do the "same" optimization by > using a explicit stack and not use function calls at all. I don't have time to go into this right now, but you are guessing wrong. With some sophistication you can also use the JIT to optimize normal calls (even inline them, with enough effort). It would require some more work, though. Cheers, Carl Friedrich From erik at medallia.com Wed Aug 22 14:01:27 2007 From: erik at medallia.com (Erik Gorset) Date: Wed, 22 Aug 2007 14:01:27 +0200 Subject: [pypy-dev] How's the JIT coming along? In-Reply-To: <46CBCFFF.2060708@ar.media.kyoto-u.ac.jp> References: <20070819134052388521.b4a47875@goombah.com> <46CAEE94.1060004@changemaker.nu> <46CB9DF1.8060103@ar.media.kyoto-u.ac.jp> <4B0F48B0-D9AD-416E-81F6-3A839C002620@gmail.com> <46CBCFFF.2060708@ar.media.kyoto-u.ac.jp> Message-ID: <9E4AD833-8A8C-47AE-8CF4-FB0F961A3FC8@medallia.com> On Aug 22, 2007, at 7:56 AM, David Cournapeau wrote: > For example, the following python code: > > class A: > def _foo(self): > return None > def foo(self): > return self._foo() > > Is extremely slow compared to a compiled language like C. On my > computer, I can only execute the above function around 1 millon > times a > second (it takes around 3500 cycles by using %timeit from ipython). > This > forces me to avoid functions in some performance intensive code, which > is ugly. Basically, I am interested in the kind of things described > there: http://www.avibryant.com/2006/09/index.html. The self technology is extremely interesting. For more information, you can read Urs thesis [0], which is the bases for the cool optimization tricks in the strongtalk vm [1]. The important trick here is that the vm is able to do optimistic inlining based on type feedback during runtime, and deoptimize the compiled code when it guesses wrong or when you enter the debugger. I guess you can do the same thing with partial evaluation if you gather enough runtime statistics to figure out which functions to specialize/inline. More specific for your example, it will result in code looking something like this: class A_compiled: def foo(self): if type(self) == A: return None # _foo has been inlined else: return self._foo() # uncommon case Of course, the higher up you can start inlining, the less calls is needed and better performance is achieved: a = obj.foo() turns into: if type(obj) == A: a = None else: a = obj.foo() [0] http://www.cs.ucsb.edu/~urs/oocsb/self/papers/urs-thesis.html [1] http://strongtalk.org/ - Erik Gorset -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 1962 bytes Desc: not available Url : http://codespeak.net/pipermail/pypy-dev/attachments/20070822/e3373d89/attachment-0001.bin From grobinson at goombah.com Wed Aug 22 14:21:23 2007 From: grobinson at goombah.com (Gary Robinson) Date: Wed, 22 Aug 2007 08:21:23 -0400 Subject: [pypy-dev] How's the JIT coming along? Message-ID: <20070822082123472317.d675fd87@goombah.com> > If i might ask - what are the use cases you are thinking about when > talking about the JIT > (and/or the JIT generator?). Numerical computing. I have a lot of floating point arithmetic. I was also going to mention the function calling overhead that I see was already discussed (yesterday) in this thread! -- Gary Robinson CTO Emergent Music, LLC grobinson at goombah.com 207-942-3463 Company: http://www.goombah.com Blog: http://www.garyrobinson.net From cfbolz at gmx.de Wed Aug 22 16:03:48 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Wed, 22 Aug 2007 16:03:48 +0200 Subject: [pypy-dev] How's the JIT coming along? In-Reply-To: <20070822082123472317.d675fd87@goombah.com> References: <20070822082123472317.d675fd87@goombah.com> Message-ID: <348899050708220703i1ad5a8bey1e863f4a288779eb@mail.gmail.com> Hi Gary, 2007/8/22, Gary Robinson : > > If i might ask - what are the use cases you are thinking about when > > talking about the JIT > > (and/or the JIT generator?). > > Numerical computing. I have a lot of floating point arithmetic. I was also going to mention the > function calling overhead that I see was already discussed (yesterday) in this thread! We are quite far from supporting floats, I fear. Our i386 assembly backend doesn't have support for floats so far. That doesn't mean that you cannot JIT functions that do float operations, it just means that you won't get any great speedups. Cheers, Carl Friedrich From cfbolz at gmx.de Wed Aug 22 16:09:21 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Wed, 22 Aug 2007 16:09:21 +0200 Subject: [pypy-dev] How's the JIT coming along? In-Reply-To: <46CBCFFF.2060708@ar.media.kyoto-u.ac.jp> References: <20070819134052388521.b4a47875@goombah.com> <46CAEE94.1060004@changemaker.nu> <46CB9DF1.8060103@ar.media.kyoto-u.ac.jp> <4B0F48B0-D9AD-416E-81F6-3A839C002620@gmail.com> <46CBCFFF.2060708@ar.media.kyoto-u.ac.jp> Message-ID: <348899050708220709t4ab7eeadt46417b5087099048@mail.gmail.com> Hi David, 2007/8/22, David Cournapeau : > For example, the following python code: > > class A: > def _foo(self): > return None > def foo(self): > return self._foo() > > Is extremely slow compared to a compiled language like C. On my > computer, I can only execute the above function around 1 millon times a > second (it takes around 3500 cycles by using %timeit from ipython). This > forces me to avoid functions in some performance intensive code, which > is ugly. Basically, I am interested in the kind of things described > there: http://www.avibryant.com/2006/09/index.html. This page is talking about polymorphic inline caches. One of the more important building blocks of our JIT, "promotion", can be considered to be a generalization of polymorphic inline caches. So yes, PyPy's Jit will eventually be able to do things like this. Cheers, Carl Friedrich From cfbolz at gmx.de Wed Aug 22 16:13:23 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Wed, 22 Aug 2007 16:13:23 +0200 Subject: [pypy-dev] How's the JIT coming along? In-Reply-To: <9E4AD833-8A8C-47AE-8CF4-FB0F961A3FC8@medallia.com> References: <20070819134052388521.b4a47875@goombah.com> <46CAEE94.1060004@changemaker.nu> <46CB9DF1.8060103@ar.media.kyoto-u.ac.jp> <4B0F48B0-D9AD-416E-81F6-3A839C002620@gmail.com> <46CBCFFF.2060708@ar.media.kyoto-u.ac.jp> <9E4AD833-8A8C-47AE-8CF4-FB0F961A3FC8@medallia.com> Message-ID: <348899050708220713l32c9acc7s57bd36569af61af4@mail.gmail.com> Hi Erik, 2007/8/22, Erik Gorset : > On Aug 22, 2007, at 7:56 AM, David Cournapeau wrote: > > For example, the following python code: > > > > class A: > > def _foo(self): > > return None > > def foo(self): > > return self._foo() > > [snip] > More specific for your example, it will result in code looking > something like this: > > class A_compiled: > def foo(self): > if type(self) == A: > return None # _foo has been inlined > else: > return self._foo() # uncommon case > > Of course, the higher up you can start inlining, the less calls is > needed and > better performance is achieved: > > a = obj.foo() > > turns into: > > if type(obj) == A: > a = None > else: > a = obj.foo() those inline checks are not enough in the Python case. You also need to check that the method A.foo was not changed in the meantime. But yes, in principle this is the idea. It works even better in the case of integer-handling functions, because the inlined operations can in many cases be implemented by processor opcodes. Cheers, Carl Friedrich From erik at medallia.com Wed Aug 22 17:47:20 2007 From: erik at medallia.com (Erik Gorset) Date: Wed, 22 Aug 2007 17:47:20 +0200 Subject: [pypy-dev] How's the JIT coming along? In-Reply-To: <348899050708220713l32c9acc7s57bd36569af61af4@mail.gmail.com> References: <20070819134052388521.b4a47875@goombah.com> <46CAEE94.1060004@changemaker.nu> <46CB9DF1.8060103@ar.media.kyoto-u.ac.jp> <4B0F48B0-D9AD-416E-81F6-3A839C002620@gmail.com> <46CBCFFF.2060708@ar.media.kyoto-u.ac.jp> <9E4AD833-8A8C-47AE-8CF4-FB0F961A3FC8@medallia.com> <348899050708220713l32c9acc7s57bd36569af61af4@mail.gmail.com> Message-ID: <8BD45F0D-F754-4B09-B206-1925E53A2653@medallia.com> On Aug 22, 2007, at 4:13 PM, Carl Friedrich Bolz wrote: > those inline checks are not enough in the Python case. You also need > to check that the method A.foo was not changed in the meantime. But > yes, in principle this is the idea. It works even better in the case > of integer-handling functions, because the inlined operations can in > many cases be implemented by processor opcodes. Yes, and one way to do this is to keep a dependency graph for the compiled code, and invalidate when a dependency changes. You can get away with checks at relatively few points, and you can also insert traps for assignments to objects which will (possibly) shadow compiled dependencies. -- Erik Gorset -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 1962 bytes Desc: not available Url : http://codespeak.net/pipermail/pypy-dev/attachments/20070822/394ad402/attachment.bin From len-l at telus.net Wed Aug 22 19:42:12 2007 From: len-l at telus.net (Lenard Lindstrom) Date: Wed, 22 Aug 2007 10:42:12 -0700 Subject: [pypy-dev] How's the JIT coming along? In-Reply-To: <20070822082123472317.d675fd87@goombah.com> References: <20070822082123472317.d675fd87@goombah.com> Message-ID: <46CC7574.5070500@telus.net> Gary Robinson wrote: >> If i might ask - what are the use cases you are thinking about when >> talking about the JIT >> (and/or the JIT generator?). >> > > Numerical computing. I have a lot of floating point arithmetic. I was also going to mention the function calling overhead that I see was already discussed (yesterday) in this thread! > > > I would say that Python function calls are not the only calls to consider. At some point the ctypes module will need porting to PyPy. A JIT could remove the overhead of libffi. -- Lenard Lindstrom From fijall at gmail.com Wed Aug 22 20:49:12 2007 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 22 Aug 2007 20:49:12 +0200 Subject: [pypy-dev] PyPy cleanups Message-ID: <693bc9ab0708221149i4819f01l3a4b7fd71574064b@mail.gmail.com> Hello, This is the list of possibly orphaned parts of pypy. We should consider each item here and think in detail what to do with them. They're mostly broken or not actively maintained. If nobody shows an interest to maintain them, deleting would be the best solution to avoid clutter. Also they pose a maintenance burden when we proceed with the needed large refactorings. Maybe it is also a solution to lazily delete them as they are broken by refactoring if nobody steps up. Of course deleted things can easily be brought back from svn history, if there is renewed interest. So the above may sound scarier than it is. * llvm backend - we need a maintainer for that * pyrex backend (llvm depends on it tough) * js interpreter - needs a lot of work, before we might have usecases for that. * common lisp backend * squeak backend * ext compiler - would need a large rewrite from rctypes to rffi * rctypes (both) - delete as soon as they would not be needed * logic objspace, including constraint libraries Cheers, fijal, Samuele, Simon & Carl Friedrich -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20070822/84d755dc/attachment.htm From cfbolz at gmx.de Wed Aug 22 23:02:28 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Wed, 22 Aug 2007 23:02:28 +0200 Subject: [pypy-dev] How's the JIT coming along? In-Reply-To: <46CC7574.5070500@telus.net> References: <20070822082123472317.d675fd87@goombah.com> <46CC7574.5070500@telus.net> Message-ID: <46CCA464.7010301@gmx.de> Lenard Lindstrom wrote: > Gary Robinson wrote: >>> If i might ask - what are the use cases you are thinking about when >>> talking about the JIT >>> (and/or the JIT generator?). >>> >> Numerical computing. I have a lot of floating point arithmetic. I was >> also going to mention the function calling overhead that I see was >> already discussed (yesterday) in this thread! > > I would say that Python function calls are not the only calls to > consider. At some point the ctypes module will need porting to PyPy. A > JIT could remove the overhead of libffi. Absolutely. This is part of the more longer-term plans for the JIT (especially since it involves re-implementing ctypes, which will be its own kind of pain). In theory the JIT can not only remove some of the overhead of libffi but also be faster than statically compiled extension modules, since in some cases it can know in advance that some runtime type checks that a static extension does are always true and skip them. Cheers, Carl Friedrich From simon at arrowtheory.com Thu Aug 23 00:23:40 2007 From: simon at arrowtheory.com (Simon Burton) Date: Wed, 22 Aug 2007 15:23:40 -0700 Subject: [pypy-dev] rffi strict typing In-Reply-To: <20070720125254.GC19336@code0.codespeak.net> References: <20070719183117.97ad1ce3.simon@arrowtheory.com> <20070720125254.GC19336@code0.codespeak.net> Message-ID: <20070822152340.8075e11c.simon@arrowtheory.com> On Fri, 20 Jul 2007 14:52:54 +0200 Armin Rigo wrote: > > Hi Simon, > > On Thu, Jul 19, 2007 at 06:31:17PM -0700, Simon Burton wrote: > > The lltype's are very strict about types. > > Indeed, it seems worthwhile to loosen this restriction at least for the > purpose of calling external functions... Not sure how, but there are > possible hacks at least. Following some hints from Samuele, I am trying to wrap such functions in another function that does some casting. Here is my first attempt. It does no casting, but i already can't get it to work. def softwrapper(funcptr, arg_tps): unrolling_arg_tps = unrolling_iterable(enumerate(arg_tps)) def softfunc(*args): real_args = tuple([args[i] for i, tp in unrolling_arg_tps]) result = funcptr(*real_args) return softfunc where funcptr comes from a call to llexternal. It seems the tuple construction should not work (each args[i] is a different type), but instead i get: CallPatternTooComplex': '*' argument must be SomeTuple. Um.. My only guess now is to malloc a TUPLE_TYPE.. ?!? no idea.. (I am really sick of code generation and hope it doesn't come to that). Simon. From alexandre.fayolle at logilab.fr Thu Aug 23 11:18:50 2007 From: alexandre.fayolle at logilab.fr (Alexandre Fayolle) Date: Thu, 23 Aug 2007 11:18:50 +0200 Subject: [pypy-dev] PyPy cleanups In-Reply-To: <693bc9ab0708221149i4819f01l3a4b7fd71574064b@mail.gmail.com> References: <693bc9ab0708221149i4819f01l3a4b7fd71574064b@mail.gmail.com> Message-ID: <20070823091850.GB4812@logilab.fr> On Wed, Aug 22, 2007 at 08:49:12PM +0200, Maciej Fijalkowski wrote: > Hello, > > This is the list of possibly orphaned parts of pypy. > We should consider each item here and think in detail > what to do with them. They're mostly broken or not > actively maintained. If nobody shows an interest > to maintain them, deleting would be the best solution > to avoid clutter. Also they pose a maintenance burden > when we proceed with the needed large refactorings. > Maybe it is also a solution to lazily delete them as > they are broken by refactoring if nobody steps up. > > Of course deleted things can easily be brought back from > svn history, if there is renewed interest. So the above > may sound scarier than it is. > > * llvm backend - we need a maintainer for that > * pyrex backend (llvm depends on it tough) > * js interpreter - needs a lot of work, before > we might have usecases for that. > * common lisp backend > * squeak backend > * ext compiler - would need a large rewrite from rctypes to rffi > * rctypes (both) - delete as soon as they would not be needed > * logic objspace, including constraint libraries Please add aop.py and dbc.py (from pypy/lib) to that list. -- Alexandre Fayolle LOGILAB, Paris (France) Formations Python, Zope, Plone, Debian: http://www.logilab.fr/formations D?veloppement logiciel sur mesure: http://www.logilab.fr/services Informatique scientifique: http://www.logilab.fr/science Reprise et maintenance de sites CPS: http://www.migration-cms.com/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 481 bytes Desc: Digital signature Url : http://codespeak.net/pipermail/pypy-dev/attachments/20070823/14713677/attachment-0001.pgp From simon at arrowtheory.com Thu Aug 23 19:07:37 2007 From: simon at arrowtheory.com (Simon Burton) Date: Thu, 23 Aug 2007 10:07:37 -0700 Subject: [pypy-dev] rffi strict typing In-Reply-To: <20070822152340.8075e11c.simon@arrowtheory.com> References: <20070719183117.97ad1ce3.simon@arrowtheory.com> <20070720125254.GC19336@code0.codespeak.net> <20070822152340.8075e11c.simon@arrowtheory.com> Message-ID: <20070823100737.1f3f56f9.simon@arrowtheory.com> On Wed, 22 Aug 2007 15:23:40 -0700 Simon Burton wrote: > > Following some hints from Samuele, I am trying to wrap such functions in > another function that does some casting. Here is the latest: def softwrapper(funcptr, arg_tps): unrolling_arg_tps = unrolling_iterable(enumerate(arg_tps)) def softfunc(*args): real_args = () for i, tp in unrolling_arg_tps: real_args = real_args + (args[i],) result = funcptr(*real_args) return result return softfunc When applied to llexternal's that have pointer-to-struct args the generated c code breaks; it decides to declare&use anonymous structs: long pypy_g_softfunc_star2_1(struct pypy__cairo_surface0 *l_stararg0_7, struct pypy_array3 *l_stararg1_7) { long l_v492; block0: l_v492 = cairo_surface_write_to_png(l_stararg0_7, l_stararg1_7); goto block1; block1: RPY_DEBUG_RETURN(); return l_v492; } where cairo_surface_write_to_png is declared: cairo_surface_write_to_png (cairo_surface_t *surface, const char *filename); I wonder if the annotator is getting confused by the real_args tuple growing... From fijal at genesilico.pl Thu Aug 23 19:13:26 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Thu, 23 Aug 2007 19:13:26 +0200 Subject: [pypy-dev] rffi strict typing In-Reply-To: <20070823100737.1f3f56f9.simon@arrowtheory.com> References: <20070719183117.97ad1ce3.simon@arrowtheory.com> <20070720125254.GC19336@code0.codespeak.net> <20070822152340.8075e11c.simon@arrowtheory.com> <20070823100737.1f3f56f9.simon@arrowtheory.com> Message-ID: <46CDC036.20506@genesilico.pl> Simon Burton wrote: > On Wed, 22 Aug 2007 15:23:40 -0700 > Simon Burton wrote: > > >> Following some hints from Samuele, I am trying to wrap such functions in >> another function that does some casting. >> > > Here is the latest: > > > def softwrapper(funcptr, arg_tps): > unrolling_arg_tps = unrolling_iterable(enumerate(arg_tps)) > def softfunc(*args): > real_args = () > for i, tp in unrolling_arg_tps: > real_args = real_args + (args[i],) > result = funcptr(*real_args) > return result > return softfunc > > When applied to llexternal's that have pointer-to-struct args > the generated c code breaks; it decides to declare&use anonymous > structs: > > long pypy_g_softfunc_star2_1(struct pypy__cairo_surface0 *l_stararg0_7, struct pypy_array3 *l_stararg1_7) { > long l_v492; > > block0: > l_v492 = cairo_surface_write_to_png(l_stararg0_7, l_stararg1_7); > goto block1; > > block1: > RPY_DEBUG_RETURN(); > return l_v492; > } > > where cairo_surface_write_to_png is declared: > > cairo_surface_write_to_png (cairo_surface_t *surface, > const char *filename); > > > I wonder if the annotator is getting confused by the real_args tuple growing... > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > > :. > > You need to create the structure which you push by hand. As well as array argument. :. From simon at arrowtheory.com Thu Aug 23 21:42:08 2007 From: simon at arrowtheory.com (Simon Burton) Date: Thu, 23 Aug 2007 12:42:08 -0700 Subject: [pypy-dev] rffi strict typing In-Reply-To: <46CDC036.20506@genesilico.pl> References: <20070719183117.97ad1ce3.simon@arrowtheory.com> <20070720125254.GC19336@code0.codespeak.net> <20070822152340.8075e11c.simon@arrowtheory.com> <20070823100737.1f3f56f9.simon@arrowtheory.com> <46CDC036.20506@genesilico.pl> Message-ID: <20070823124208.dfbcea83.simon@arrowtheory.com> On Thu, 23 Aug 2007 19:13:26 +0200 Maciek Fijalkowski wrote: > > You need to create the structure which you push by hand. As well as > array argument. No, structure comes from a call to another llexternal. It makes no sense to clone it. It should be an opaque struct anyway. Simon. From simon at arrowtheory.com Thu Aug 23 22:05:52 2007 From: simon at arrowtheory.com (Simon Burton) Date: Thu, 23 Aug 2007 13:05:52 -0700 Subject: [pypy-dev] rffi strict typing In-Reply-To: <46CDC036.20506@genesilico.pl> References: <20070719183117.97ad1ce3.simon@arrowtheory.com> <20070720125254.GC19336@code0.codespeak.net> <20070822152340.8075e11c.simon@arrowtheory.com> <20070823100737.1f3f56f9.simon@arrowtheory.com> <46CDC036.20506@genesilico.pl> Message-ID: <20070823130552.ce5e77bd.simon@arrowtheory.com> On Thu, 23 Aug 2007 19:13:26 +0200 Maciek Fijalkowski wrote: > > You need to create the structure which you push by hand. As well as > array argument. I stuck a test in test_rffi but didn't want to commit it. It works fine. (attached) Simon. -------------- next part -------------- A non-text attachment was scrubbed... Name: test_rffi.py Type: text/x-python Size: 6958 bytes Desc: not available Url : http://codespeak.net/pipermail/pypy-dev/attachments/20070823/bd5e26a2/attachment.py From fijal at genesilico.pl Fri Aug 24 10:06:01 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Fri, 24 Aug 2007 10:06:01 +0200 Subject: [pypy-dev] rffi strict typing In-Reply-To: <20070823124208.dfbcea83.simon@arrowtheory.com> References: <20070719183117.97ad1ce3.simon@arrowtheory.com> <20070720125254.GC19336@code0.codespeak.net> <20070822152340.8075e11c.simon@arrowtheory.com> <20070823100737.1f3f56f9.simon@arrowtheory.com> <46CDC036.20506@genesilico.pl> <20070823124208.dfbcea83.simon@arrowtheory.com> Message-ID: <46CE9169.2070307@genesilico.pl> Simon Burton wrote: > On Thu, 23 Aug 2007 19:13:26 +0200 > Maciek Fijalkowski wrote: > > >> You need to create the structure which you push by hand. As well as >> array argument. >> > > No, structure comes from a call to another llexternal. It makes no sense to clone it. > It should be an opaque struct anyway. > > Simon. > There is a COpaque and COpaquePtr for opaque structures. cheers, fijal :. From cfbolz at gmx.de Sat Aug 25 00:08:01 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Sat, 25 Aug 2007 00:08:01 +0200 Subject: [pypy-dev] away for two weeks Message-ID: <46CF56C1.7020304@gmx.de> Hi all, I will be away from home (and probably from Internet) for about two weeks, starting from Sunday morning. Cheers, Carl Friedrich From exarkun at divmod.com Sat Aug 25 01:18:08 2007 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Fri, 24 Aug 2007 19:18:08 -0400 Subject: [pypy-dev] translation broken in pypy-more-rtti-inprogress branch In-Reply-To: 0 Message-ID: <20070824231808.8162.305325374.divmod.quotient.1425@ohm> Just a heads up :) Here's the traceback I get: [translation:ERROR] Error: [translation:ERROR] Traceback (most recent call last): [translation:ERROR] File "./translate.py", line 272, in main [translation:ERROR] drv.proceed(goals) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/translator/driver.py", line 748, in proceed [translation:ERROR] return self._execute(goals, task_skip = self._maybe_skip()) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/translator/tool/taskengine.py", line 112, in _execute [translation:ERROR] res = self._do(goal, taskcallable, *args, **kwds) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/translator/driver.py", line 265, in _do [translation:ERROR] res = func() [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/translator/driver.py", line 334, in task_rtype_lltype [translation:ERROR] crash_on_first_typeerror=insist) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/rpython/rtyper.py", line 180, in specialize [translation:ERROR] self.specialize_more_blocks() [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/rpython/rtyper.py", line 226, in specialize_more_blocks [translation:ERROR] self.specialize_block(block) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/rpython/rtyper.py", line 345, in specialize_block [translation:ERROR] self.translate_hl_to_ll(hop, varmapping) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/rpython/rtyper.py", line 474, in translate_hl_to_ll [translation:ERROR] resultvar = hop.dispatch() [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/rpython/rtyper.py", line 712, in dispatch [translation:ERROR] return translate_meth(self) [translation:ERROR] File "None", line 5, in translate_op_delitem [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/rpython/lltypesystem/rdict.py", line 319, in rtype_delitem [translation:ERROR] return hop.gendirectcall(ll_dict_delitem, v_dict, v_key) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/rpython/rtyper.py", line 750, in gendirectcall [translation:ERROR] return self.llops.gendirectcall(ll_function, *args_v) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/rpython/rtyper.py", line 908, in gendirectcall [translation:ERROR] rtyper.lowlevel_ann_policy) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/rpython/annlowlevel.py", line 103, in annotate_lowlevel_helper [translation:ERROR] return annotator.annotate_helper(ll_function, args_s, policy) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/annotation/annrpython.py", line 129, in annotate_helper [translation:ERROR] self.complete_helpers(policy) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/annotation/annrpython.py", line 175, in complete_helpers [translation:ERROR] self.complete() [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/annotation/annrpython.py", line 249, in complete [translation:ERROR] self.processblock(graph, block) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/annotation/annrpython.py", line 475, in processblock [translation:ERROR] self.flowin(graph, block) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/annotation/annrpython.py", line 535, in flowin [translation:ERROR] self.consider_op(block.operations[i]) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/annotation/annrpython.py", line 737, in consider_op [translation:ERROR] raise_nicer_exception(op, str(graph)) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/annotation/annrpython.py", line 734, in consider_op [translation:ERROR] resultcell = consider_meth(*argcells) [translation:ERROR] File "", line 3, in consider_op_call_args [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/annotation/unaryop.py", line 169, in call_args [translation:ERROR] return obj.call(getbookkeeper().build_args("call_args", args_s)) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/annotation/unaryop.py", line 582, in call [translation:ERROR] return bltn.analyser(*args_s, **kwds_s) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/annotation/builtin.py", line 418, in malloc [translation:ERROR] assert (s_n is None or s_n.knowntype == int [translation:ERROR] AssertionError': [translation:ERROR] .. v1 = call_args((function malloc), ((2, ('zero',), False, False)), v0, new_size_0, (True)) [translation:ERROR] .. '(pypy.rpython.lltypesystem.rdict:475)ll_dict_resize__dicttablePtr' [translation:ERROR] Processing block: [translation:ERROR] block at 97 is a [translation:ERROR] in (pypy.rpython.lltypesystem.rdict:475)ll_dict_resize__dicttablePtr [translation:ERROR] containing the following operations: [translation:ERROR] v2 = simple_call((function typeOf), old_entries_0) [translation:ERROR] v0 = getattr(v2, ('TO')) [translation:ERROR] v1 = call_args((function malloc), ((2, ('zero',), False, False)), v0, new_size_0, (True)) [translation:ERROR] v3 = setattr(d_0, ('entries'), v1) [translation:ERROR] v4 = setattr(d_0, ('num_items'), (0)) [translation:ERROR] v5 = setattr(d_0, ('num_pristine_entries'), new_size_0) [translation:ERROR] --end-- [translation] start debugger... > /home/exarkun/Projects/pypy/trunk/pypy/annotation/builtin.py(418)malloc() -> assert (s_n is None or s_n.knowntype == int (Pdb+) This is from r45970 Jean-Paul From simon at arrowtheory.com Wed Aug 29 02:35:57 2007 From: simon at arrowtheory.com (Simon Burton) Date: Tue, 28 Aug 2007 17:35:57 -0700 Subject: [pypy-dev] buffer class and ll2ctypes problem In-Reply-To: <20070821090637.f57b0323.simon@arrowtheory.com> References: <20070821090637.f57b0323.simon@arrowtheory.com> Message-ID: <20070828173557.1f161eec.simon@arrowtheory.com> On Tue, 21 Aug 2007 09:06:37 -0700 Simon Burton wrote: > > I don't understand this much, can someone point the right direction ? > Ie. which buf is the right direction to go. failing that, does anyone have a random opinion ? Simon. From elmo13 at jippii.fi Thu Aug 30 21:10:57 2007 From: elmo13 at jippii.fi (=?ISO-8859-1?Q?Elmo_M=E4ntynen?=) Date: Thu, 30 Aug 2007 22:10:57 +0300 Subject: [pypy-dev] About adding a blist implementation... In-Reply-To: <20070830112103.d8da6bdc.simon@arrowtheory.com> References: <466FE95A.6020805@jippii.fi> <46779173.3050902@jippii.fi> <20070621181001.d9322aef.simon@arrowtheory.com> <467BD864.8040505@jippii.fi> <20070830112103.d8da6bdc.simon@arrowtheory.com> Message-ID: <46D71641.2050000@jippii.fi> Sorry, I got distracted, and now I have commitments (schoolwork) which prevent me from doing anything moderately involved. And no, I have only started porting the implementation and fixed some general multilist bugs (nothing grave). I will return to all this soon, probably after a month or two... If you need a btree, you'd better do it yourself anyway, and we'll see later if a refactoring is possible. Elmo Simon Burton wrote: > Elmo! > did you get anywhere with this ?? > > Simon. > > On Fri, 22 Jun 2007 17:10:44 +0300 > Elmo M?ntynen wrote: > > >> Leonardo Santagada wrote: >> >>> Em 21/06/2007, ?s 22:10, Simon Burton escreveu: >>> >>> >>> >>>> On Tue, 19 Jun 2007 11:18:59 +0300 >>>> Elmo M?ntynen wrote: >>>> >>>> >>>> >>>>> Since I'm adding a list implementation, I'd like to know what >>>>> benchmarks >>>>> have you used to test the other ones. Also, if you know apps with >>>>> heavy >>>>> usage of lists and some that use very long lists, I'd be >>>>> interested very >>>>> much. >>>>> >>>>> >>>> Can you explain what this blist is ? Is it a btree ? I think I need >>>> an rpython btree. >>>> >>>> Simon. >>>> >>>> >>> I dunno but this guy is probably talking about something like this >>> http://mail.python.org/pipermail/python-3000/2007-May/thread.html#7127 >>> >>> If it is not related I would also like to understand the diferences. >>> >>> -- >>> Leonardo Santagada >>> >>> Sent from my iPhone >>> >>> >>> >> Hi. >> It is an list implementation that has much better performance with very >> big lists, available as an extension to CPython, and soon included in >> pypy by me. >> This cheeseshop entry is a good place to start: >> http://www.python.org/pypi/blist/ >> >> The thread might be about integrading it into CPython or something, >> haven't read. >> >> It is based on btree, but the implementation I'm working on is based on >> the python prototype included in the tar distrib, which won't probably >> be useful if you specifically need a btree. In the future, maybe some >> refactoring would be useful, but for now I'm happy with the way I started. >> >> I'd still be interested in ways to test the different list >> implementations =) >> >> Elmo >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev >> From arigo at tunes.org Fri Aug 31 10:05:55 2007 From: arigo at tunes.org (Armin Rigo) Date: Fri, 31 Aug 2007 10:05:55 +0200 Subject: [pypy-dev] Talked at ESUG Message-ID: <20070831080555.GA19496@code0.codespeak.net> Hi all, As some of you know I've just given a talk at the ESUG (European Smalltalk User Group) conference, in Lugano. I mostly showed up for my talk only, staying only one night. The talk yesterday late afternoon went well; it was the demo part of the talk we gave at Dyla. Thanks to Roel for pushing us to give a presentation here - it seems quite unusual to have presentations that are given by "outsiders" of Smalltalk, and the organizers suggested to the audiance that I should get an extra applause for daring come and talk :-) Some people had hard about PyPy already, even though it's clear that in the Smalltalk situation the whole approach we took in PyPy is a bit overkill: it's a very stable language that existed and had good VMs for a very long time, with a relatively small core and a lot of things merely written in Smalltalk on top of it. This was the argument I received when I tried to discuss with someone working on XTC, a JIT compiler for Squeak (yet another) - the language is too small to bother with more general techniques, and he might be right for now. (I cannot seem to find his work by googling.) Where I failed to convince him was that PyPy was a possibly better alternative to "plans" like: all Smalltalk-like languages (Self, dialects, etc.) should be implemented in a single VM in a way that allows a single JIT for them all. Non-performance-related benefits of PyPy are also not completely clear for Smalltalk given that the language is more flexible than Python to start with. However, people generally see why we'd like such things for Python-like languages - I think it's a very positive thing. I also believe that presenting the PyPy approach to different communities can contribute just a little bit in the long run to push forward some ideas; for example, people may think twice about Slang or equivalents and see if it could be practical to make these languages just a bit higher level, e.g. move them above the GC. A final note is that talking about PyPy in live seems to be very important. Various people had spend some time looking around our web site because they had heard about PyPy, but mostly failed to understand what we were trying to do. For most talks we have given about PyPy I remember getting some feedback of people that now, at least, had got an overview about what we were doing. A bientot, Armin. From tismer at stackless.com Fri Aug 31 14:04:14 2007 From: tismer at stackless.com (Christian Tismer) Date: Fri, 31 Aug 2007 14:04:14 +0200 Subject: [pypy-dev] PyPy cleanups In-Reply-To: <693bc9ab0708221149i4819f01l3a4b7fd71574064b@mail.gmail.com> References: <693bc9ab0708221149i4819f01l3a4b7fd71574064b@mail.gmail.com> Message-ID: <46D803BE.50001@stackless.com> Maciej Fijalkowski wrote: > Hello, > ... > Of course deleted things can easily be brought back from > svn history, if there is renewed interest. So the above > may sound scarier than it is. No, it is scary for me. Things that I can't see are gone. The svn history argument does not help. I would prefer disabling the tests and moving unmaintained stuff into maybe a "unmaintained" folder or renaming ist's path to include a syllable for unmaintained. Deleting is final, in a sense. scared - ly y'rs -- chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From paul.degrandis at gmail.com Fri Aug 31 14:43:29 2007 From: paul.degrandis at gmail.com (Paul deGrandis) Date: Fri, 31 Aug 2007 08:43:29 -0400 Subject: [pypy-dev] PyPy cleanups In-Reply-To: <46D803BE.50001@stackless.com> References: <693bc9ab0708221149i4819f01l3a4b7fd71574064b@mail.gmail.com> <46D803BE.50001@stackless.com> Message-ID: <9c0bb8a00708310543s3efb3092ye03a56a49af5edd@mail.gmail.com> I like Christian's ideas. let's not delete them but move them to "pasture" or "unmaintained". Some people may want it, and someone (unfamilar to the project) may bring one of them back to life. Paul On 8/31/07, Christian Tismer wrote: > > Maciej Fijalkowski wrote: > > Hello, > > > ... > > > Of course deleted things can easily be brought back from > > svn history, if there is renewed interest. So the above > > may sound scarier than it is. > > No, it is scary for me. Things that I can't see are gone. > The svn history argument does not help. > > I would prefer disabling the tests and moving > unmaintained stuff into maybe a "unmaintained" folder or > renaming ist's path to include a syllable for unmaintained. > > Deleting is final, in a sense. > > scared - ly y'rs -- chris > > -- > Christian Tismer :^) > tismerysoft GmbH : Have a break! Take a ride on Python's > Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ > 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ > work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 > PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 > whom do you want to sponsor today? http://www.stackless.com/ > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20070831/1e112474/attachment.htm From tismer at stackless.com Fri Aug 31 16:55:28 2007 From: tismer at stackless.com (Christian Tismer) Date: Fri, 31 Aug 2007 16:55:28 +0200 Subject: [pypy-dev] PyPy cleanups In-Reply-To: <9c0bb8a00708310543s3efb3092ye03a56a49af5edd@mail.gmail.com> References: <693bc9ab0708221149i4819f01l3a4b7fd71574064b@mail.gmail.com> <46D803BE.50001@stackless.com> <9c0bb8a00708310543s3efb3092ye03a56a49af5edd@mail.gmail.com> Message-ID: <46D82BE0.2050503@stackless.com> Paul deGrandis wrote: > I like Christian's ideas. > > let's not delete them but move them to "pasture" or "unmaintained". > Some people may want it, and someone (unfamilar to the project) may bring > one of them back to life. And I'd rather go for this earlier than later. Not removing but renaming is not destructive, and may shake up people who thought "at some time, I will work on this" to consider a reaction. Things that are out of sight are forgotten very fast. Things that get a nasty name in their path are crying for maintenance, also expressing that people are sad to loose them. cheers - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From fijal at genesilico.pl Fri Aug 31 20:49:15 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Fri, 31 Aug 2007 20:49:15 +0200 Subject: [pypy-dev] I'm off for 4 weeks Message-ID: <46D862AB.40500@genesilico.pl> I'll be probably completely offline for like next 4 weeks, starting from monday. Cheers, fijal :. From arigo at tunes.org Wed Sep 5 13:37:02 2007 From: arigo at tunes.org (Armin Rigo) Date: Wed, 5 Sep 2007 13:37:02 +0200 Subject: [pypy-dev] Branch Merged! Message-ID: <20070905113702.GA29505@code0.codespeak.net> Hi all, We've just merged the pypy-more-rtti-inprogress branch. For those of you still working in there - I'm thining about Simon in particular - you need to switch back to the regular trunk. In principle, you can issue the following command in your working copy even if there are local changes, which should stay untouched as long as they don't conflict (but make a backup, just in case!): In a checkout of pypy/dist/pypy, in the 'pypy' directory: svn switch http://codespeak.net/svn/pypy/dist/pypy This branch was a port of all low-level functions of the os module to use the new RFFI interface. The branch also contains progress made on RFFI itself. I've tested and fixed the branch yesterday for Windows XP, so it should compile there as well. A bientot, Armin. From gotcha at bubblenet.be Thu Sep 6 10:43:05 2007 From: gotcha at bubblenet.be (Godefroid Chapelle) Date: Thu, 06 Sep 2007 10:43:05 +0200 Subject: [pypy-dev] Help raise Open Source visibility in the government area Message-ID: <46DFBD99.8050506@bubblenet.be> Hi, ( If you have already received this email and have voted, thank you ! ) ( If you have not voted, and wanted to, this is a nice reminder. ) ( If you do not wish to vote, sorry for the noise. ) Please take 10 minutes to help raise open source profile in the government area by voting for the PloneGov project at the european egov awards. See below what PloneGov is and why you could vote to support open source even if you do not use Plone. Anyone from any country can vote, you don't have to live in the EU. And please vote *now*, the deadline is almost upon us **SEPTEMBER 7** ? you are likely to forget if you don't set aside 10 minutes now. This process is unfortunately a bit involved ? since we're dealing with governments ? but shouldn't take longer than 10 minutes to complete. Step-by-step: 1. Go to http://www.epractice.eu/register 2. Wait for the email confirmation and log in. 3. Fill out your profile: http://www.epractice.eu/myprofile/personal http://www.epractice.eu/myprofile/professional http://www.epractice.eu/myprofile/contact (If you don't want to provide professional info, we suggest you add "I just want to vote" in the required fields) 4. Vote for PloneGov by clicking here: http://www.epractice.eu/do/voteinspiring/1026 Thanks! More information below, if you're interested in what PloneGov is. _____________________________________ What is PloneGov? PloneGov aims to create a collaborative software ecosystem where governmental organizations work together, in close relationship with SMEs, to share the cost of software development and enhanced capabilities. This new initiative started by small towns now reaches all levels of public administration and spans over 13 countries on 3 continents. http://www.plonegov.org Your vote for the "Most innovative good practice" award is a unique opportunity to reach politicians and decision makers. Winning this award will highlight the maturity achieved by the Open Source collaborative model as a pragmatic way to tackle IT challenges in society. You may vote until Friday, September 7th. The Public Sector has an important influence on many sectors of society. A wider Open Source acceptance in this field is strategic to all Open Source users and advocates. Users in sectors such as Education, NGOs and businesses would directly benefit from broader adoption. It would result in a growing pool of available software, tools, and developers. Beyond the practical example of PloneGov, your vote contributes to support Open Source values that are based on collaboration, openness and fair trade. It is an opportunity to support Open Source as an innovative model of IT development in which the Public Sector, NGOs and Education can work together and all of society will benefit. -- Godefroid Chapelle (aka __gotcha)- BubbleNet http://bubblenet.be From jwal at jwal.me.uk Sun Sep 9 17:38:14 2007 From: jwal at jwal.me.uk (James Ascroft-Leigh) Date: Sun, 9 Sep 2007 16:38:14 +0100 Subject: [pypy-dev] Testing generated JavaScript code Message-ID: <8b3cc5810709090838t45b3b63av48e40041fa407d7e@mail.gmail.com> Hi all, I am interested in the potential for PyPy to run python code within the web browser using, at least as a fallback, the JavaScript interpreter. I have my own project (very immature and not yet usable) to try an achieve just this goal and, like PyPy, my tests exercise the generated JavaScript code using an external interpreter (currently rhino). I am just writing to let you know about this great project I discovered the other day called jpype (http://jpype.sf.net/) that lets you interact with an in-process JVM. Now that the JVM includes the Rhino JS interpreter I am going to switch my tests over to using this to run my generated JS code. This allows a higher level interaction between the languages because you no longer have to serialise the output of the script to stdout and de-serialise it to compare to the expected output. Maybe PyPy could do the same? I might get time to post a patch for this but, as the attached snippet shows, it is not too hard to get started so somebody else might want to do it sooner. What do people think about adding a testing dependency on the JVM and this jpype package? Regards, James -------------- next part -------------- A non-text attachment was scrubbed... Name: rhino.py Type: text/x-python Size: 464 bytes Desc: not available Url : http://codespeak.net/pipermail/pypy-dev/attachments/20070909/377f0033/attachment-0001.py From gholling at vni.com Tue Sep 11 00:27:41 2007 From: gholling at vni.com (Greg Holling) Date: Mon, 10 Sep 2007 16:27:41 -0600 Subject: [pypy-dev] NumPy in PyPy Message-ID: <84625E3AC73C7943A07A8712F940DB91015857DD@peabody.vni.com> Sorry if this has been posted before, but I didn't see anything in the archives. I see that there's been some recent activity in subversion related to numpy in pypy (specifically in pypy/dist/pypy/rpython/numpy). Can anybody on this list give me some more information about this development work (goals, status, etc.)? I have an upcoming project where it might be useful. Thanks! - Greg - -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20070910/9d1c32f9/attachment.htm From simon at arrowtheory.com Tue Sep 11 07:16:16 2007 From: simon at arrowtheory.com (Simon Burton) Date: Mon, 10 Sep 2007 22:16:16 -0700 Subject: [pypy-dev] NumPy in PyPy In-Reply-To: <84625E3AC73C7943A07A8712F940DB91015857DD@peabody.vni.com> References: <84625E3AC73C7943A07A8712F940DB91015857DD@peabody.vni.com> Message-ID: <20070910221616.67dd69db.simon@arrowtheory.com> On Mon, 10 Sep 2007 16:27:41 -0600 "Greg Holling" wrote: > > I see that there's been some recent activity in subversion related to > numpy in pypy (specifically in pypy/dist/pypy/rpython/numpy). Can > anybody on this list give me some more information about this > development work (goals, status, etc.)? I have an upcoming project > where it might be useful. Yes, i've been working on this recently. I really have been aiming to have this functionality at the rpython level; it would take some re-thinking to get this to work in the pypy interpreter. And then, i think it only makes sense to do if we can exploit the JIT somehow. You can see the status from looking at the test code. I will be writing a more comprehensive email to pypy-dev in the next few days. One idea i had was to be able to use this in conjunction with cpython&numpy. But that's a few weeks away from working, and would still involve programming in rpython. Simon. From cfbolz at gmx.de Tue Sep 11 21:12:01 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Tue, 11 Sep 2007 21:12:01 +0200 Subject: [pypy-dev] cleanup sprint planning Message-ID: <46E6E881.50803@gmx.de> Hi PyPy-Dev, there has been the vague plan for a cleanup sprint since the end of the EU project. It is becoming more concrete now: There is the vague idea of doing a sprint in October/November (probably November) at Laura's place in Gothenburg. To coordinate the possible times a bit, I have set up a doodle: http://doodle.ch/5awzskkt5mian94a Please enter your available times if you want to attend. Note that this sprint is going to be likely newbie-unfriendly because of its refactoring and cleanup focus. About the work for the sprint: this is a list of possible tasks (they may be too much and some may happen before the sprint), other suggestions are also possible as long as they involve "cleanups". Meta-Task --------- - Tracker-gardening - Start slowly thinking about 1.1 plans Translation toolchain --------------------- - finish rctypes removal - implement correct implicit GIL release - move away from implicit keepalives - raw_malloc should be split, rffi one should not use obmalloc (it's not thread-safe) - kill simulator, run the semi space tests on llinterp (may happen before) - have a moving gc correct version of the gc framework transformation - delegate finding type stuff like vtables etc to GC, cleaner interface for rtti, simplify translator/c/gc.py - think about approaches to id, especially concerning boehm, where the id will keep the object alive and concerning a moving GC - clean up the tangle of including headers in the C backend - review pdbplus, especially the graph commands, also in the light of https://codespeak.net/issue/pypy-dev/issue303 and the fact that we can have more than one translator/annotator around (with the timeshifter) - remember and fix the inlining exception bug Interpreter ----------- - there's a shadow tracking bug it seems - fix taint space - review the things implemented at applevel whether they are performance-critical JIT ------- - start writing real unit tests for the merging logic Please post all comments you might have about the planning or the tasks above. Looking forward to a productive sprint. Carl Friedrich & Samuele From fijall at gmail.com Wed Sep 12 17:16:31 2007 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 12 Sep 2007 23:16:31 +0800 Subject: [pypy-dev] pypy cleanup sprint Message-ID: <693bc9ab0709120816w6f944e02w660541ac924b7ada@mail.gmail.com> Few points from me: * Kill half concrete wrapper (kill kill kill) * Kill opaque hacks on the C backend * decide how to implement constants in rffi * think about/remove orphaned parts * finish ll2ctypes, especially rffi_platform The last one is not very much cleanup, but is needed to finish transition from rctypes (like rsocket). Also, why not make sprint in Poland? (hint, hint, cheap place). PS. Greetings from china Cheers, fijal -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20070912/e371f509/attachment.htm From cfbolz at gmx.de Thu Sep 13 13:46:20 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Thu, 13 Sep 2007 13:46:20 +0200 Subject: [pypy-dev] pypy cleanup sprint In-Reply-To: <693bc9ab0709120816w6f944e02w660541ac924b7ada@mail.gmail.com> References: <693bc9ab0709120816w6f944e02w660541ac924b7ada@mail.gmail.com> Message-ID: <46E9230C.2060905@gmx.de> Hi Maciek! Maciej Fijalkowski wrote: > Few points from me: > > * Kill half concrete wrapper (kill kill kill) > > * Kill opaque hacks on the C backend What exactly do you mean there? The other points make sense to me. > * decide how to implement constants in rffi > > * think about/remove orphaned parts > > * finish ll2ctypes, especially rffi_platform > > The last one is not very much cleanup, but is needed to finish > transition from rctypes (like rsocket). Also, why not make sprint in > Poland? (hint, hint, cheap place). I would actually prefer to make this rather important sprint at a place where many of us have been and where we had sprints already. Besides, we will be in need for sprint venues after next one, so it would be great if you could organize one in Poland anyway. Cheers, Carl Friedrich From paul.degrandis at gmail.com Thu Sep 13 15:11:52 2007 From: paul.degrandis at gmail.com (Paul deGrandis) Date: Thu, 13 Sep 2007 09:11:52 -0400 Subject: [pypy-dev] pypy cleanup sprint In-Reply-To: <46E9230C.2060905@gmx.de> References: <693bc9ab0709120816w6f944e02w660541ac924b7ada@mail.gmail.com> <46E9230C.2060905@gmx.de> Message-ID: <9c0bb8a00709130611h777c9cbl289f0168c948d137@mail.gmail.com> If at all possibly, can we also consider a place that has cheaper air fare deals? I'd really like to come to this sprint. Paul On 9/13/07, Carl Friedrich Bolz wrote: > > Hi Maciek! > > Maciej Fijalkowski wrote: > > Few points from me: > > > > * Kill half concrete wrapper (kill kill kill) > > > > * Kill opaque hacks on the C backend > > What exactly do you mean there? The other points make sense to me. > > > * decide how to implement constants in rffi > > > > * think about/remove orphaned parts > > > > * finish ll2ctypes, especially rffi_platform > > > > The last one is not very much cleanup, but is needed to finish > > transition from rctypes (like rsocket). Also, why not make sprint in > > Poland? (hint, hint, cheap place). > > I would actually prefer to make this rather important sprint at a place > where many of us have been and where we had sprints already. Besides, we > will be in need for sprint venues after next one, so it would be great > if you could organize one in Poland anyway. > > Cheers, > > Carl Friedrich > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20070913/63b67849/attachment.htm From cfbolz at gmx.de Thu Sep 13 15:21:18 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Thu, 13 Sep 2007 15:21:18 +0200 Subject: [pypy-dev] pypy cleanup sprint In-Reply-To: <9c0bb8a00709130611h777c9cbl289f0168c948d137@mail.gmail.com> References: <693bc9ab0709120816w6f944e02w660541ac924b7ada@mail.gmail.com> <46E9230C.2060905@gmx.de> <9c0bb8a00709130611h777c9cbl289f0168c948d137@mail.gmail.com> Message-ID: <46E9394E.2060705@gmx.de> Hi Paul, Paul deGrandis wrote: > If at all possibly, can we also consider a place that has cheaper air > fare deals? > I'd really like to come to this sprint. The sprint venue (as discussed on IRC a while ago) is really quite convenient for many, since Samuele, Laura, Jacob, Bea are living there, many of us know the sprint venue and the city, etc. So I don't think changing is a good idea. Maybe you can fly to Stockholm (or even Kopenhagen) and take the train? That takes a few hours but is actually very nice. Cheers, Carl Friedrich From paul.degrandis at gmail.com Thu Sep 13 15:25:06 2007 From: paul.degrandis at gmail.com (Paul deGrandis) Date: Thu, 13 Sep 2007 09:25:06 -0400 Subject: [pypy-dev] pypy cleanup sprint In-Reply-To: <46E9394E.2060705@gmx.de> References: <693bc9ab0709120816w6f944e02w660541ac924b7ada@mail.gmail.com> <46E9230C.2060905@gmx.de> <9c0bb8a00709130611h777c9cbl289f0168c948d137@mail.gmail.com> <46E9394E.2060705@gmx.de> Message-ID: <9c0bb8a00709130625q370f8a47g17d3a6bcc939e4c8@mail.gmail.com> Sounds like an awesome idea to me. Thanks Carl. Paul On 9/13/07, Carl Friedrich Bolz wrote: > > Hi Paul, > > Paul deGrandis wrote: > > If at all possibly, can we also consider a place that has cheaper air > > fare deals? > > I'd really like to come to this sprint. > > The sprint venue (as discussed on IRC a while ago) is really quite > convenient for many, since Samuele, Laura, Jacob, Bea are living there, > many of us know the sprint venue and the city, etc. So I don't think > changing is a good idea. Maybe you can fly to Stockholm (or even > Kopenhagen) and take the train? That takes a few hours but is actually > very nice. > > Cheers, > > Carl Friedrich > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20070913/8648c4c9/attachment.htm From lac at openend.se Thu Sep 13 15:29:21 2007 From: lac at openend.se (Laura Creighton) Date: Thu, 13 Sep 2007 15:29:21 +0200 Subject: [pypy-dev] pypy cleanup sprint In-Reply-To: Message from Carl Friedrich Bolz of "Thu, 13 Sep 2007 15:21:18 +0200." <46E9394E.2060705@gmx.de> References: <693bc9ab0709120816w6f944e02w660541ac924b7ada@mail.gmail.com> <46E9230C.2060905@gmx.de> <9c0bb8a00709130611h777c9cbl289f0168c948d137@mail.gmail.com> <46E9394E.2060705@gmx.de> Message-ID: <200709131329.l8DDTLtB008526@theraft.openend.se> In a message of Thu, 13 Sep 2007 15:21:18 +0200, Carl Friedrich Bolz writes: >Hi Paul, > >Paul deGrandis wrote: >> If at all possibly, can we also consider a place that has cheaper air >> fare deals? >> I'd really like to come to this sprint. > >The sprint venue (as discussed on IRC a while ago) is really quite >convenient for many, since Samuele, Laura, Jacob, Bea are living there, >many of us know the sprint venue and the city, etc. So I don't think >changing is a good idea. Maybe you can fly to Stockholm (or even >Kopenhagen) and take the train? That takes a few hours but is actually >very nice. > >Cheers, > >Carl Friedrich Planes may actually be cheaper. Where are you Paul? Laura From cfbolz at gmx.de Thu Sep 13 15:29:23 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Thu, 13 Sep 2007 15:29:23 +0200 Subject: [pypy-dev] pypy cleanup sprint In-Reply-To: <200709131329.l8DDTLtB008526@theraft.openend.se> References: <693bc9ab0709120816w6f944e02w660541ac924b7ada@mail.gmail.com> <46E9230C.2060905@gmx.de> <9c0bb8a00709130611h777c9cbl289f0168c948d137@mail.gmail.com> <46E9394E.2060705@gmx.de> <200709131329.l8DDTLtB008526@theraft.openend.se> Message-ID: <46E93B33.4060305@gmx.de> Laura Creighton wrote: > Planes may actually be cheaper. Where are you Paul? but it is more stressful and you get to see less :-) Cheers, Carl Friedrich From paul.degrandis at gmail.com Thu Sep 13 15:33:34 2007 From: paul.degrandis at gmail.com (Paul deGrandis) Date: Thu, 13 Sep 2007 09:33:34 -0400 Subject: [pypy-dev] pypy cleanup sprint In-Reply-To: <46E93B33.4060305@gmx.de> References: <693bc9ab0709120816w6f944e02w660541ac924b7ada@mail.gmail.com> <46E9230C.2060905@gmx.de> <9c0bb8a00709130611h777c9cbl289f0168c948d137@mail.gmail.com> <46E9394E.2060705@gmx.de> <200709131329.l8DDTLtB008526@theraft.openend.se> <46E93B33.4060305@gmx.de> Message-ID: <9c0bb8a00709130633i5814253di322e840c3b1ac4c0@mail.gmail.com> Philadelphia, PA int the US. Paul On 9/13/07, Carl Friedrich Bolz wrote: > > Laura Creighton wrote: > > Planes may actually be cheaper. Where are you Paul? > > but it is more stressful and you get to see less :-) > > Cheers, > > Carl Friedrich > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20070913/c612939c/attachment.htm From lac at openend.se Thu Sep 13 15:36:08 2007 From: lac at openend.se (Laura Creighton) Date: Thu, 13 Sep 2007 15:36:08 +0200 Subject: [pypy-dev] pypy cleanup sprint In-Reply-To: Message from Carl Friedrich Bolz of "Thu, 13 Sep 2007 15:29:23 +0200." <46E93B33.4060305@gmx.de> References: <693bc9ab0709120816w6f944e02w660541ac924b7ada@mail.gmail.com> <46E9230C.2060905@gmx.de> <9c0bb8a00709130611h777c9cbl289f0168c948d137@mail.gmail.com> <46E9394E.2060705@gmx.de> <200709131329.l8DDTLtB008526@theraft.openend.se> <46E93B33.4060305@gmx.de> Message-ID: <200709131336.l8DDa8lf010277@theraft.openend.se> In a message of Thu, 13 Sep 2007 15:29:23 +0200, Carl Friedrich Bolz writes: >Laura Creighton wrote: >> Planes may actually be cheaper. Where are you Paul? > >but it is more stressful and you get to see less :-) > >Cheers, > >Carl Friedrich Absolutely. By all means take the train for maximum fun. But given that the request was for cheap airfair, the fact that the plane is usually the cheaper option to get from Stockholm to G?teborg may be relevant. Laura From lac at openend.se Thu Sep 13 16:55:01 2007 From: lac at openend.se (Laura Creighton) Date: Thu, 13 Sep 2007 16:55:01 +0200 Subject: [pypy-dev] pypy cleanup sprint In-Reply-To: Message from "Paul deGrandis" of "Thu, 13 Sep 2007 09:33:34 EDT." <9c0bb8a00709130633i5814253di322e840c3b1ac4c0@mail.gmail.com> References: <693bc9ab0709120816w6f944e02w660541ac924b7ada@mail.gmail.com> <46E9230C.2060905@gmx.de> <9c0bb8a00709130611h777c9cbl289f0168c948d137@mail.gmail.com> <46E9394E.2060705@gmx.de> <200709131329.l8DDTLtB008526@theraft.openend.se> <46E93B33.4060305@gmx.de> <9c0bb8a00709130633i5814253di322e840c3b1ac4c0@mail.gmail.com> Message-ID: <200709131455.l8DEt1gF024383@theraft.openend.se> In a message of Thu, 13 Sep 2007 09:33:34 EDT, "Paul deGrandis" writes: >------=_Part_27723_30036397.1189690414424 >Content-Type: text/plain; charset=ISO-8859-1 >Content-Transfer-Encoding: 7bit >Content-Disposition: inline > >Philadelphia, PA int the US. > >Paul > >On 9/13/07, Carl Friedrich Bolz wrote: >> >> Laura Creighton wrote: >> > Planes may actually be cheaper. Where are you Paul? >> >> but it is more stressful and you get to see less :-) >> >> Cheers, >> >> Carl Friedrich Well, taking the Train from Oslo, Copenhagen or Stockholm will definitely be pleasant. You can also take the overnight ferry from Kiel in Germany or Fredrikshavn in Denmark. For you, though, the bulk of the cost will be on the transatlantic flight. Last time I checked, flying KLM to Amsterdam and then Amsterdam to G?teborg was the cheapest, despite the fact that KLM is not a discount airline. http://www.lfv.se/templates/LFV_InfoSida_Bred____1582.aspx lists the airlines that run to Landvetter Airport. Discount airlines include: Malm? Aviation (sometimes), G?teborg City Airline (sometimes) Sterling, Welcome Air, Blue1, Wider?e Then there is G?teborg Ciry airport. They have nothing but discount airlines flying there. http://www.goteborgcityairport.se/info/resenaerer/destinationer.asp Let me know if you need help with anything. Laura From tismer at stackless.com Fri Sep 14 04:31:34 2007 From: tismer at stackless.com (Christian Tismer) Date: Fri, 14 Sep 2007 04:31:34 +0200 Subject: [pypy-dev] pypy cleanup sprint In-Reply-To: <200709131455.l8DEt1gF024383@theraft.openend.se> References: <693bc9ab0709120816w6f944e02w660541ac924b7ada@mail.gmail.com> <46E9230C.2060905@gmx.de> <9c0bb8a00709130611h777c9cbl289f0168c948d137@mail.gmail.com> <46E9394E.2060705@gmx.de> <200709131329.l8DDTLtB008526@theraft.openend.se> <46E93B33.4060305@gmx.de> <9c0bb8a00709130633i5814253di322e840c3b1ac4c0@mail.gmail.com> <200709131455.l8DEt1gF024383@theraft.openend.se> Message-ID: <46E9F286.7030007@stackless.com> Laura Creighton wrote: ... Well, I'd like to come if I'm wellcomed. Has there been a concrete timing proposal, already? -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From lac at openend.se Fri Sep 14 06:02:25 2007 From: lac at openend.se (Laura Creighton) Date: Fri, 14 Sep 2007 06:02:25 +0200 Subject: [pypy-dev] pypy cleanup sprint In-Reply-To: Message from Christian Tismer of "Fri, 14 Sep 2007 04:31:34 +0200." <46E9F286.7030007@stackless.com> References: <693bc9ab0709120816w6f944e02w660541ac924b7ada@mail.gmail.com> <46E9230C.2060905@gmx.de> <9c0bb8a00709130611h777c9cbl289f0168c948d137@mail.gmail.com> <46E9394E.2060705@gmx.de> <200709131329.l8DDTLtB008526@theraft.openend.se> <46E93B33.4060305@gmx.de> <9c0bb8a00709130633i5814253di322e840c3b1ac4c0@mail.gmail.com> <200709131455.l8DEt1gF024383@theraft.openend.se> <46E9F286.7030007@stackless.com> Message-ID: <200709140402.l8E42PnJ028861@theraft.openend.se> In a message of Fri, 14 Sep 2007 04:31:34 +0200, Christian Tismer writes: >Laura Creighton wrote: > >... > >Well, I'd like to come if I'm wellcomed. >Has there been a concrete timing proposal, already? >-- >Christian Tismer :^) >tismerysoft GmbH : Have a break! Take a ride on Python's >Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ >14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ >work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 >PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 > whom do you want to sponsor today? http://www.stackless.com/ Of course you are most welcome. People are trying to find a time now. see: http://doodle.ch/5awzskkt5mian94a There is now a direct flight from Berlin to Gbg. http://www.goteborgcityairport.se/info/resenaerer/destinationer.asp from (Tegel) -- so not sure exactly where that is, via Air Berlin. Laura From tismer at stackless.com Fri Sep 14 20:07:59 2007 From: tismer at stackless.com (Christian Tismer) Date: Fri, 14 Sep 2007 20:07:59 +0200 Subject: [pypy-dev] Stackless PyPy mini sprint 2007 Message-ID: <46EACDFF.4020301@stackless.com> Hi Stackless PyPy people. We have been sprinting once, quite a long time, ago. Time to start over, again. I'm thinking of doing a mini sprint, before the PyPy cleanup-sprint. It should be a last sprint of clarifying issues which are not resolved, yet, concerning completeness of app-level basic Stackless implementations, and it should lead to an almost final application of the scalability goals stated in PyPy. Maybe these statements need to be adjusted as well. Effectively, I'd like to target this as the final sprint before transforming results into RPython. As a wish, RPython implementations might me tackled as well. But my primary intent is to have a very complete, well-tested code-base for app-level stackless, including composability. I will give a very intense introductory talk about this in the very beginning. Everybody will conceive composability as a simple concept which is not really new, but becomes new by introducing coroutines as the underlying structure for everything. This is non-intrusive, it just needs to be understood. The sprint can be done in my house, with a limit of about five external people, free food and accommodation included. I think a period of three days would be appropriate. Please give me timing proposals and different opinions as you see fit. Sincerely -- chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From ondrej at certik.cz Sat Sep 15 02:58:39 2007 From: ondrej at certik.cz (Ondrej Certik) Date: Sat, 15 Sep 2007 02:58:39 +0200 Subject: [pypy-dev] pypy with sympy Message-ID: <85b5c3130709141758s2c832322uc4b986144b87a14e@mail.gmail.com> Hi, since pypy is in Debian for some time, I sometimes test it with our project SymPy[1], written in pure Python 2.4 and newer. Generally, it works out of the box and it's very nice and quite fast. You did an awesome job! However, sometimes, it fails, because of the bug[2], that we reported two months ago. Do you think you could please apply the patch provided in there, or just fix it somehow? Also another question - what is your plan with your RPython to javascript converter - are you planning to create something like the Google Web Toolkit, or just leave it as is, if someone becomes interested? I think it would be a very cool feature to have something like this in Python, just a little more documented and officially released. Thanks a lot, Ondrej Certik [1] http://code.google.com/p/sympy/ [2] https://codespeak.net/issue/pypy-dev/issue316 From cfbolz at gmx.de Sat Sep 15 14:48:54 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Sat, 15 Sep 2007 14:48:54 +0200 Subject: [pypy-dev] pypy with sympy In-Reply-To: <85b5c3130709141758s2c832322uc4b986144b87a14e@mail.gmail.com> References: <85b5c3130709141758s2c832322uc4b986144b87a14e@mail.gmail.com> Message-ID: <46EBD4B6.2050500@gmx.de> Hi Ondrej, Ondrej Certik wrote: > since pypy is in Debian for some time, I sometimes test it with our > project SymPy[1], written in pure Python 2.4 and newer. Generally, it > works out of the box and it's very nice and quite fast. You did an > awesome job! Thanks for doing that! It's nice to hear that it's mostly working. Indeed at one point we could probably consider to use SymPy as some sort of benchmark for PyPy ? it's not easy to find nice pure-Python programs that are computationally bound. > However, sometimes, it fails, because of the bug[2], that we reported > two months ago. Do you think you could please apply the patch provided > in there, or just fix it somehow? Thanks for reporting it. Our problem with it is that we don't quite know how to fix it properly. Potentially, int_w itself should be calling __int__, but we are not sure whether this is good for all use-cases of int_w. > Also another question - what is your plan with your RPython to > javascript converter - are you planning to create something like the > Google Web Toolkit, or just leave it as is, if someone becomes > interested? I think it would be a very cool feature to have something > like this in Python, just a little more documented and officially > released. I think the PyPy team cannot put much more effort into this than what we did already. So we would need somebody interested who would work mostly exclusively on the JavaScript backend to make it live up to its full potential. Cheers, Carl Friedrich From ondrej at certik.cz Sat Sep 15 15:17:34 2007 From: ondrej at certik.cz (Ondrej Certik) Date: Sat, 15 Sep 2007 15:17:34 +0200 Subject: [pypy-dev] pypy with sympy In-Reply-To: <46EBD4B6.2050500@gmx.de> References: <85b5c3130709141758s2c832322uc4b986144b87a14e@mail.gmail.com> <46EBD4B6.2050500@gmx.de> Message-ID: <85b5c3130709150617j37b76299o306b713bdc6bbb40@mail.gmail.com> Hi Carl, > Ondrej Certik wrote: > > since pypy is in Debian for some time, I sometimes test it with our > > project SymPy[1], written in pure Python 2.4 and newer. Generally, it > > works out of the box and it's very nice and quite fast. You did an > > awesome job! > > Thanks for doing that! It's nice to hear that it's mostly working. > Indeed at one point we could probably consider to use SymPy as some sort > of benchmark for PyPy ? it's not easy to find nice pure-Python programs > that are computationally bound. We are actually optimizing sympy to by fast on top of a normal CPython, for example instantiating python classes is very expensive. But I guess it should be very similar in pypy as well. There is another problem actually: $ py.test --exec=pypy * opening PopenGateway: /usr/bin/pypy MASTER: initiated slave terminal session -> MASTER: send start info, topdir=/home/ondra/sympy Traceback (most recent call last): File "?", line 32, in run_toplevel File "?", line 265, in run_it File "", line 1, in File "", line 40, in ImportError: No module named thread Traceback (most recent call last): File "/usr/bin/py.test", line 4, in ? py.test.cmdline.main() File "/usr/lib/python2.4/site-packages/py/test/cmdline.py", line 15, in main failures = session.main() File "/usr/lib/python2.4/site-packages/py/test/terminal/remote.py", line 82, in main failures = self.run_remote_session(failures) File "/usr/lib/python2.4/site-packages/py/test/terminal/remote.py", line 111, in run_remote_session return channel.receive() File "/usr/lib/python2.4/site-packages/py/execnet/channel.py", line 175, in receive raise self._getremoteerror() or EOFError() EOFError There is not module thread in there. So I cannot test the whole sympy using pypy, because we have all our tests made using py.test. Feel free to put this into the issues. > Thanks for reporting it. Our problem with it is that we don't quite know > how to fix it properly. Potentially, int_w itself should be calling > __int__, but we are not sure whether this is good for all use-cases of > int_w. Right, but do you agree, that this should be working in python 2.4? So there must be some way to do it. > I think the PyPy team cannot put much more effort into this than what we > did already. So we would need somebody interested who would work mostly > exclusively on the JavaScript backend to make it live up to its full > potential. So what is your priority now with pypy? I think it has such an enormous potential, but currently for me it's just a playground. Do you plan to apply pypy in some area, where people will use it as the best technology available in there? Or do you plan more to have pypy just as a means of testing new JIT ideas? Ondrej From ondrej at certik.cz Sat Sep 15 15:23:47 2007 From: ondrej at certik.cz (Ondrej Certik) Date: Sat, 15 Sep 2007 15:23:47 +0200 Subject: [pypy-dev] pypy with sympy In-Reply-To: <85b5c3130709150617j37b76299o306b713bdc6bbb40@mail.gmail.com> References: <85b5c3130709141758s2c832322uc4b986144b87a14e@mail.gmail.com> <46EBD4B6.2050500@gmx.de> <85b5c3130709150617j37b76299o306b713bdc6bbb40@mail.gmail.com> Message-ID: <85b5c3130709150623p5de74662j107cccc39c65db8c@mail.gmail.com> > There is another problem actually: > > $ py.test --exec=pypy > * opening PopenGateway: /usr/bin/pypy > MASTER: initiated slave terminal session -> > MASTER: send start info, topdir=/home/ondra/sympy > Traceback (most recent call last): > File "?", line 32, in run_toplevel > File "?", line 265, in run_it > File "", line 1, in > File "", line 40, in > ImportError: No module named thread > Traceback (most recent call last): > File "/usr/bin/py.test", line 4, in ? > py.test.cmdline.main() > File "/usr/lib/python2.4/site-packages/py/test/cmdline.py", line 15, in main > failures = session.main() > File "/usr/lib/python2.4/site-packages/py/test/terminal/remote.py", > line 82, in main > failures = self.run_remote_session(failures) > File "/usr/lib/python2.4/site-packages/py/test/terminal/remote.py", > line 111, in run_remote_session > return channel.receive() > File "/usr/lib/python2.4/site-packages/py/execnet/channel.py", line > 175, in receive > raise self._getremoteerror() or EOFError() > EOFError I forgot to add - I am using the versions from Debian unstable. I don't have time to test everything from svn, however, if you want, I can make new versions of pypy/py.test available in Debian when you release them, but I think the current maintainer will do also do it - Alexandre Fayolle is the maintainer of those packages - should I report bugs in them to Debian BTS, or here? I think the way to go is just users report problems against the packages in their distributions, like Debian, it is forwarded here, you fix it in svn, release a new version, it gets to Debian, etc. Ondrej From lac at openend.se Sat Sep 15 16:05:41 2007 From: lac at openend.se (Laura Creighton) Date: Sat, 15 Sep 2007 16:05:41 +0200 Subject: [pypy-dev] pypy with sympy In-Reply-To: Message from "Ondrej Certik" of "Sat, 15 Sep 2007 15:17:34 +0200." <85b5c3130709150617j37b76299o306b713bdc6bbb40@mail.gmail.com> References: <85b5c3130709141758s2c832322uc4b986144b87a14e@mail.gmail.com> <46EBD4B6.2050500@gmx.de> <85b5c3130709150617j37b76299o306b713bdc6bbb40@mail.gmail.com> Message-ID: <200709151405.l8FE5fk8029135@theraft.openend.se> In a message of Sat, 15 Sep 2007 15:17:34 +0200, "Ondrej Certik" writes: >So what is your priority now with pypy? I think it has such an >enormous potential, but currently for me it's just a playground. Do >you plan to apply pypy in some area, where people will use it as the >best technology available in there? Or do you plan more to have pypy >just as a means of testing new JIT ideas? > >Ondrej We really want to push this as a real language that people should use to solve real problems they have. Currently, however, things are in a bad need of a cleanup. While it was terrific to have the EU funding which let many of us work on pypy full time, the deadlines for deliverables were such that many things were hacked together in ways that are ok for a prototype but not really robust enough for production use. We are planning our next sprint to work on the roughest of these. Then, of course, the problem is that there are so many things that could be the 'next step' for PyPy. Some of us (me for instance) are actively looking for paying customers who have a business case for wanting PyPy to do X. Tim Parkin, who designed the current python.org site is also interested in working on finding out what should be the next step for PyPy. He is collecting opinions, and people who are willing to put money where their mouth is here: http://wiki.python.org/moin/PyPyDonations It would be nice to get PyPy to such a state that it again had many tasks for those people who don't feel up to hacking the Just in Time Specialiser. A better garbage collector is necessary, but I think that people find that intimidating as well. We're much further along with the new scheme for working with C extensions, and that too is necessary. It would be nice to find out exactly who wants what next from PyPy, and perhaps more importantly, who would be willing to help on PyPy, but cannot now, and what they would need in the way of changes before they could contribute. Besides fixes for the bugs you have reported, is there anything in particular you wish we were doing next? Laura From ondrej at certik.cz Sat Sep 15 17:13:02 2007 From: ondrej at certik.cz (Ondrej Certik) Date: Sat, 15 Sep 2007 17:13:02 +0200 Subject: [pypy-dev] pypy with sympy In-Reply-To: <200709151405.l8FE5fk8029135@theraft.openend.se> References: <85b5c3130709141758s2c832322uc4b986144b87a14e@mail.gmail.com> <46EBD4B6.2050500@gmx.de> <85b5c3130709150617j37b76299o306b713bdc6bbb40@mail.gmail.com> <200709151405.l8FE5fk8029135@theraft.openend.se> Message-ID: <85b5c3130709150813s20ba3e4exd5a516953400cf4a@mail.gmail.com> > Then, of course, the problem is that there are so many things that > could be the 'next step' for PyPy. Some of us (me for instance) are > actively looking for paying customers who have a business case for > wanting PyPy to do X. Tim Parkin, who designed the current python.org > site is also interested in working on finding out what should be the > next step for PyPy. He is collecting opinions, and people who are > willing to put money where their mouth is here: > http://wiki.python.org/moin/PyPyDonations I'd suggest to put that link on the front page of pypy. And also to put that question where to go now on the front page. Compared to your donations, I hope you won't take mine as an offence, but I am a student. > It would be nice to get PyPy to such a state that it again had many > tasks for those people who don't feel up to hacking the Just in Time > Specialiser. A better garbage collector is necessary, but I think that > people find that intimidating as well. We're much further along with the > new scheme for working with C extensions, and that too is necessary. > > It would be nice to find out exactly who wants what next from PyPy, > and perhaps more importantly, who would be willing to help on PyPy, > but cannot now, and what they would need in the way of changes before > they could contribute. Besides fixes for the bugs you have reported, > is there anything in particular you wish we were doing next? I have many ideas: 1) Make predictable releases, get pypy into distributions, fix bugs and just make it another Python interpreter. 2) Provide a standard interpreter to: java, .NET and other platforms. In Jython, they need to code it in Java, by hand I guess. In PyPy, if I understand it correctly, you just update the interpeter in rpython and you get all those interpreters and all platforms for free. I guess this is one of the goals of pypy. So that for example SymPy can be run on all those platforms. 3) Allow efficient writing of modules in C/Fortran. There is SWIG, ctypes, f2py, then there are many projects like pyrex, Cython, etc. It's a mess. I think pypy has also something to say in this area and maybe make things more systematic, robust and multiplatform. 4) Myabe allow to speed my code up (JIT), or make Python as fast as possible. But this is rather a long term goal. Generally, I don't see why my code written in C and the same code written in Python, not using much dynamic features, couldn't be the same fast. (Of course I understand why now it is slower, Python is interpreted, etc. But I mean in principle. Python can have as much information about my code as the C compiler.) I use Python for devising new numerical algorithms, and than I need to rewrite them by hand to fortran for speed. It's a pain. 5) Google Web Toolkit, but in Python using PyPy. It's a lot of work, nevertheless it's already possible and that would be a boom to pypy and python in general, I am pretty sure about that. 6) Allow efficient threading? CPython has the GIL (for good technological reasons). Generally pypy can improve how to do things in parallel in python. Currently, one needs to use pympi, pypar, or something. 7) Using RPython as a compiled languge. Generally, many of my algorithms are just RPython. If they could be tranlated to C and all the other backends, the same efficient as I would do by hand, that'd be just awesome. This is connected to 4). 8) This is very long term thing - the whole pypy machinery should in principle allow to run Python on basically everything. So knowing Python could be enough to contribute to a project written in almost any language. Anyway, I think pypy either can already do the things above, or has a very big potential to do so, like a proof of concept. Ondrej From cfbolz at gmx.de Sun Sep 16 01:30:42 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Sun, 16 Sep 2007 01:30:42 +0200 Subject: [pypy-dev] pypy with sympy In-Reply-To: <85b5c3130709150813s20ba3e4exd5a516953400cf4a@mail.gmail.com> References: <85b5c3130709141758s2c832322uc4b986144b87a14e@mail.gmail.com> <46EBD4B6.2050500@gmx.de> <85b5c3130709150617j37b76299o306b713bdc6bbb40@mail.gmail.com> <200709151405.l8FE5fk8029135@theraft.openend.se> <85b5c3130709150813s20ba3e4exd5a516953400cf4a@mail.gmail.com> Message-ID: <46EC6B22.1060600@gmx.de> Hi Ondrej, > I have many ideas: > > 1) Make predictable releases, get pypy into distributions, fix bugs > and just make it another Python interpreter. That one is a bit boring, don't you think? :-) No seriously, we can't really compete with CPython on its home turf, we need additional features that set us apart. > 2) Provide a standard interpreter to: java, .NET and other platforms. > In Jython, they need to code it in Java, by hand I guess. In PyPy, if > I understand it correctly, you just update the interpeter in rpython > and you get all those interpreters and all platforms for free. I guess > this is one of the goals of pypy. So that for example SymPy can be run > on all those platforms. Yes, that's correct. We are working on this, actually. Especially Java is interesting, I guess, since Jython is still behind CPython. Competing with IronPython won't be so easy. > 3) Allow efficient writing of modules in C/Fortran. There is SWIG, > ctypes, f2py, then there are many projects like pyrex, Cython, etc. > It's a mess. I think pypy has also something to say in this area and > maybe make things more systematic, robust and multiplatform. > > 4) Myabe allow to speed my code up (JIT), or make Python as fast as > possible. But this is rather a long term goal. Generally, I don't see > why my code written in C and the same code written in Python, not > using much dynamic features, couldn't be the same fast. (Of course I > understand why now it is slower, Python is interpreted, etc. But I > mean in principle. Python can have as much information about my code > as the C compiler.) I use Python for devising new numerical > algorithms, and than I need to rewrite them by hand to fortran for > speed. It's a pain. This, too, is definitely part of PyPy's agenda. We plan to work on this not even too far into the future, I would think. > 5) Google Web Toolkit, but in Python using PyPy. It's a lot of work, > nevertheless it's already possible and that would be a boom to pypy > and python in general, I am pretty sure about that. In my opinion this is way outside what the current team of people actively working on PyPy can manage. We would need outside people that are ready to work on this. > 6) Allow efficient threading? CPython has the GIL (for good > technological reasons). Generally pypy can improve how to do things in > parallel in python. Currently, one needs to use pympi, pypar, or > something. Right now we have a GIL as well, since it was easiest to implement. It's too early to think about removing the GIL, since we would need a better thread-aware garbage collector first ? Boehm is just not up to the task. > 7) Using RPython as a compiled languge. Generally, many of my > algorithms are just RPython. If they could be tranlated to C and all > the other backends, the same efficient as I would do by hand, that'd > be just awesome. Not sure whether it is as efficient as by hand, but such a translation is obviously possible. > This is connected to 4). I disagree with this. The JIT techniques that PyPy uses are very different from its RPython translators. > 8) This is very long term thing - the whole pypy machinery should in > principle allow to run Python on basically everything. So knowing > Python could be enough to contribute to a project written in almost > any language. I guess so, yes. Although right now our Python interpreter is not exactly lightweight, so running on very small machines is not going to be reasonable without quite some additional work. Cheers, Carl Friedrich From tismer at stackless.com Sun Sep 16 01:58:36 2007 From: tismer at stackless.com (Christian Tismer) Date: Sun, 16 Sep 2007 01:58:36 +0200 Subject: [pypy-dev] 46638 made Stackless to crash. Message-ID: <46EC71AC.5070404@stackless.com> Hi Armin, today you pointed me to stackless failures. They were caused by your revision 46638. both MacGyver:~/pypy-dist/pypy/lib/app_test tismer$ ../../../py/bin/py.test -A test_stackless.py -v -s and MacGyver:~/pypy-dist/pypy/lib/app_test tismer$ ~/pypy-builds/pypy-c ../../../py/bin/py.test -A test_stackless.py -v -s worked until your check-in. It is a bit annoying to get blamed on pypy-dev by you, after *you* obviously did not run these tests. Anyway, I would really like to understand what you did, since this bug does not make much sense to me. ciao -- chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer at stackless.com Sun Sep 16 02:48:05 2007 From: tismer at stackless.com (Christian Tismer) Date: Sun, 16 Sep 2007 02:48:05 +0200 Subject: [pypy-dev] tephan. In-Reply-To: <20070915133824.B7E2B80A6@code0.codespeak.net> References: <20070915133824.B7E2B80A6@code0.codespeak.net> Message-ID: <46EC7D45.1030200@stackless.com> Hi friends, Something in this check-in has a huge impact on all of the stackless tests in pypy/lib/app-tests. I think that this was unforeseen by Armin, although it was not appropriate to blame me for this without any reflection. I felt exposed to something which I was unable to react on. Especially since I did not write these modules but Stephan. Thanks to cfbolz who reminded me how to track such things down. I would like to know if the resulting errors come from a mis-use on our side, or if we did all-right so far. cheers -- chris arigo at codespeak.net wrote: > Author: arigo > Date: Sat Sep 15 15:38:24 2007 > New Revision: 46638 > > Modified: > pypy/dist/pypy/tool/cache.py > Log: > Protect cache building in a multithreaded py.py. > > > Modified: pypy/dist/pypy/tool/cache.py > ============================================================================== > --- pypy/dist/pypy/tool/cache.py (original) > +++ pypy/dist/pypy/tool/cache.py Sat Sep 15 15:38:24 2007 > @@ -25,6 +25,9 @@ > # Be sure to call the parent __init__() if you override it. > # > > +from threading import RLock > +lock = RLock() # multithreading protection > + > > class Cache(object): > def __init__(self): > @@ -32,20 +35,24 @@ > self._building = {} > > def getorbuild(self, key): > + lock.acquire() > try: > - return self.content[key] > - except KeyError: > - if key in self._building: > - raise Exception, "%s recursive building of %r" % ( > - self, key) > - self._building[key] = True > try: > - result = self._build(key) > - self.content[key] = result > - finally: > - del self._building[key] > - self._ready(result) > - return result > + return self.content[key] > + except KeyError: > + if key in self._building: > + raise Exception, "%s recursive building of %r" % ( > + self, key) > + self._building[key] = True > + try: > + result = self._build(key) > + self.content[key] = result > + finally: > + del self._building[key] > + self._ready(result) > + return result > + finally: > + lock.release() > getorbuild._annspecialcase_ = "specialize:memo" > > def _ready(self, result): > _______________________________________________ > pypy-svn mailing list > pypy-svn at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-svn -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From arigo at tunes.org Sun Sep 16 11:16:42 2007 From: arigo at tunes.org (Armin Rigo) Date: Sun, 16 Sep 2007 11:16:42 +0200 Subject: [pypy-dev] 46638 made Stackless to crash. In-Reply-To: <46EC71AC.5070404@stackless.com> References: <46EC71AC.5070404@stackless.com> Message-ID: <20070916091642.GA22349@code0.codespeak.net> Hi Christian, On Sun, Sep 16, 2007 at 01:58:36AM +0200, Christian Tismer wrote: > today you pointed me to stackless failures. > > They were caused by your revision 46638. I wouldn't be surprized if that was my fault - I recently caused JIT tests to break in a completely unexpected manner by changing apparently unrelated code. However it seems that you are mistaken in this case: the failures I'm referring to (e.g. [1]) have been on the wyvern report page since a few days longer than 46638. On my laptop I found that they are there since revision 46507 - this is the revision that fixes the IndentationError in stackless.py. Maybe a caching problem? Try removing the pypy/_cache directory (and all .pyc files as usual...). [1] http://wyvern.cs.uni-duesseldorf.de/pypytest/46667/failed/lib.app_test.test_stackless.py.Test_Stacklesstest_balance_recv.html Armin From tismer at stackless.com Sun Sep 16 16:37:50 2007 From: tismer at stackless.com (Christian Tismer) Date: Sun, 16 Sep 2007 16:37:50 +0200 Subject: [pypy-dev] 46638 made Stackless to crash. In-Reply-To: <20070916091642.GA22349@code0.codespeak.net> References: <46EC71AC.5070404@stackless.com> <20070916091642.GA22349@code0.codespeak.net> Message-ID: <46ED3FBE.8070807@stackless.com> Hello Armin, > On Sun, Sep 16, 2007 at 01:58:36AM +0200, Christian Tismer wrote: >> today you pointed me to stackless failures. >> >> They were caused by your revision 46638. > > I wouldn't be surprized if that was my fault - I recently caused JIT > tests to break in a completely unexpected manner by changing apparently > unrelated code. However it seems that you are mistaken in this case: > the failures I'm referring to (e.g. [1]) have been on the wyvern report > page since a few days longer than 46638. On my laptop I found that they > are there since revision 46507 - this is the revision that fixes the > IndentationError in stackless.py. > > Maybe a caching problem? Try removing the pypy/_cache directory (and > all .pyc files as usual...). > > [1] http://wyvern.cs.uni-duesseldorf.de/pypytest/46667/failed/lib.app_test.test_stackless.py.Test_Stacklesstest_balance_recv.html I'm not convinced. What I did yesterday was a completely new check-out, and I could binary search to 46638 in a repeatable way. I will try again now with cache clearing - no, ATM I even don't have a _cache. Ok, the result is the same. Exactly this change to pypy/tool/cache.py causes the errors. After the change, the threading module can no longer import thread, as like the stackless module no longer can import greenlet. cheers - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From erik at medallia.com Sun Sep 16 19:23:58 2007 From: erik at medallia.com (Erik Gorset) Date: Sun, 16 Sep 2007 19:23:58 +0200 Subject: [pypy-dev] Stackless PyPy mini sprint 2007 In-Reply-To: <46EACDFF.4020301@stackless.com> References: <46EACDFF.4020301@stackless.com> Message-ID: Hi, On Sep 14, 2007, at 8:07 PM, Christian Tismer wrote: > As a wish, RPython implementations might me tackled as well. > But my primary intent is to have a very complete, well-tested > code-base for app-level stackless, including composability. > I will give a very intense introductory talk about this in the > very beginning. Everybody will conceive composability as a simple > concept which is not really new, but becomes new by introducing > coroutines as the underlying structure for everything. > This is non-intrusive, it just needs to be understood. I'm wondering if there is any plans for pypy/stackless to support true continuations as first class citizen in the future? It would be easy to implement coroutines and exceptions in normal python code if call/cc was supported. After all, ruby, scheme and smalltalk can support this :-) -- Erik Gorset -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 1962 bytes Desc: not available Url : http://codespeak.net/pipermail/pypy-dev/attachments/20070916/d08702d0/attachment.bin From tismer at stackless.com Sun Sep 16 19:57:27 2007 From: tismer at stackless.com (Christian Tismer) Date: Sun, 16 Sep 2007 19:57:27 +0200 Subject: [pypy-dev] clarification of "46638 made Stackless to crash." In-Reply-To: <46ED3FBE.8070807@stackless.com> References: <46EC71AC.5070404@stackless.com> <20070916091642.GA22349@code0.codespeak.net> <46ED3FBE.8070807@stackless.com> Message-ID: <46ED6E87.9050007@stackless.com> Christian Tismer wrote: ... > I'm not convinced. What I did yesterday was a completely new > check-out, and I could binary search to 46638 in a repeatable way. > > I will try again now with cache clearing - no, ATM I even don't > have a _cache. > > Ok, the result is the same. > Exactly this change to pypy/tool/cache.py causes the errors. > After the change, the threading module can no longer import thread, > as like the stackless module no longer can import greenlet. This was actually a bad combination of errors and misunderstandings. The import problem was really caused by rev. 46638. The rest of the mess was my fault: My check-in from short before PyCon UK had a small glitch, and although it fixed pickling (modulo a bus error which is still sought), it made the tests in app_test/test_stackless fail completely. I was blind to this error, because things worked in compiled pypy-c and with my local cpython. I was not aware that I had changed the default interpreter during the conference to real stackless. This way, the buggy code that I checked in before was never executed, so I did not expect anything to be wrong. I felt innocent but I was guilty. Armin, please take by apologies. mixing too many stacklesses is probably a bad idea. sincerely -- chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer at stackless.com Sun Sep 16 20:13:20 2007 From: tismer at stackless.com (Christian Tismer) Date: Sun, 16 Sep 2007 20:13:20 +0200 Subject: [pypy-dev] Stackless PyPy mini sprint 2007 In-Reply-To: References: <46EACDFF.4020301@stackless.com> Message-ID: <46ED7240.3060903@stackless.com> Hi Erik, Erik Gorset wrote: > Hi, > > On Sep 14, 2007, at 8:07 PM, Christian Tismer wrote: >> As a wish, RPython implementations might me tackled as well. >> But my primary intent is to have a very complete, well-tested >> code-base for app-level stackless, including composability. >> I will give a very intense introductory talk about this in the >> very beginning. Everybody will conceive composability as a simple >> concept which is not really new, but becomes new by introducing >> coroutines as the underlying structure for everything. >> This is non-intrusive, it just needs to be understood. > > I'm wondering if there is any plans for pypy/stackless to support > true continuations as first class citizen in the future? It would be easy > to implement coroutines and exceptions in normal python code if > call/cc was supported. After all, ruby, scheme and smalltalk can > support this :-) Maybe this is possible, but that's not a reason to do it. I have spent a little too much of my lifetime with taming continuations. And since Python has all the control flow it needs, it does not make sense to add continuations, which impose extra complications for no gain. I think to stick with coroutines as the building block, unless there is a real need for continuations. cheers - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From rhamph at gmail.com Sun Sep 16 22:03:03 2007 From: rhamph at gmail.com (Adam Olsen) Date: Sun, 16 Sep 2007 14:03:03 -0600 Subject: [pypy-dev] pypy with sympy Message-ID: > 4) Myabe allow to speed my code up (JIT), or make Python as fast as > possible. But this is rather a long term goal. Generally, I don't see > why my code written in C and the same code written in Python, not > using much dynamic features, couldn't be the same fast. (Of course I > understand why now it is slower, Python is interpreted, etc. But I > mean in principle. Python can have as much information about my code > as the C compiler.) I use Python for devising new numerical > algorithms, and than I need to rewrite them by hand to fortran for > speed. It's a pain. > 6) Allow efficient threading? CPython has the GIL (for good > technological reasons). Generally pypy can improve how to do things in > parallel in python. Currently, one needs to use pympi, pypar, or > something. These two together are, in my mind, where PyPy can really shine. Performance on par with CPython is critical to be taken seriously, but CPython's difficulties with the GIL mean I can only make it scalable at about 60% to 65% of the traditional efficiency (on pystones at least). That gap gives PyPy enough room to surpass CPython on scalable performance. -- Adam Olsen, aka Rhamphoryncus From erik at medallia.com Mon Sep 17 03:00:51 2007 From: erik at medallia.com (Erik Gorset) Date: Mon, 17 Sep 2007 03:00:51 +0200 Subject: [pypy-dev] Stackless PyPy mini sprint 2007 In-Reply-To: <46ED7240.3060903@stackless.com> References: <46EACDFF.4020301@stackless.com> <46ED7240.3060903@stackless.com> Message-ID: <7577F030-2075-4009-B05F-FCB2B12CF519@medallia.com> On Sep 16, 2007, at 8:13 PM, Christian Tismer wrote: > Erik Gorset wrote: >> I'm wondering if there is any plans for pypy/stackless to support >> true continuations as first class citizen in the future? It would >> be easy >> to implement coroutines and exceptions in normal python code if >> call/cc was supported. After all, ruby, scheme and smalltalk can >> support this :-) > > Maybe this is possible, but that's not a reason to do it. > I have spent a little too much of my lifetime with taming > continuations. And since Python has all the control flow > it needs, it does not make sense to add continuations, which > impose extra complications for no gain. > > I think to stick with coroutines as the building block, > unless there is a real need for continuations. Actually, having continuations as a fundamental building block is a big simplification as all other control flows can be expressed by it. The only "flaw" is that you can't use the c stack in a normal way since the executing contexts will form a tree instead of a linear structure. Of course, we can argue that this isn't pythonic anymore, so I believe I see your point. I'm already using greenlet extension for cpython heavily in my code, which is just GREAT :-) So I can live happily in python without continuations as long as I have coroutines. Btw, it's possible to simulate some of call/cc's behavior using _stackless.fork() (I have a play-implementation lying around here somewhere). But it has several issues. The biggest one is that fork() actually copies some of the objects found on the stack. It's been a while since I've played around with cloneable and fork, so maybe this has been fixed. -- Erik Gorset -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 1962 bytes Desc: not available Url : http://codespeak.net/pipermail/pypy-dev/attachments/20070917/477d2f02/attachment.bin From tismer at stackless.com Mon Sep 17 03:42:10 2007 From: tismer at stackless.com (Christian Tismer) Date: Mon, 17 Sep 2007 03:42:10 +0200 Subject: [pypy-dev] Stackless PyPy mini sprint 2007 In-Reply-To: <7577F030-2075-4009-B05F-FCB2B12CF519@medallia.com> References: <46EACDFF.4020301@stackless.com> <46ED7240.3060903@stackless.com> <7577F030-2075-4009-B05F-FCB2B12CF519@medallia.com> Message-ID: <46EDDB72.8050603@stackless.com> Erik Gorset wrote: > On Sep 16, 2007, at 8:13 PM, Christian Tismer wrote: >> Erik Gorset wrote: >>> I'm wondering if there is any plans for pypy/stackless to support >>> true continuations as first class citizen in the future? It would be >>> easy >>> to implement coroutines and exceptions in normal python code if >>> call/cc was supported. After all, ruby, scheme and smalltalk can >>> support this :-) >> >> Maybe this is possible, but that's not a reason to do it. >> I have spent a little too much of my lifetime with taming >> continuations. And since Python has all the control flow >> it needs, it does not make sense to add continuations, which >> impose extra complications for no gain. >> >> I think to stick with coroutines as the building block, >> unless there is a real need for continuations. > > Actually, having continuations as a fundamental building block is a > big simplification as all other control flows can be expressed by it. It is a complication, since continuation support is not trivial. The demands are much harder than for coroutines. > The only "flaw" is that you can't use the c stack in a normal way > since the executing contexts will form a tree instead of a linear > structure. Please don't continue this discussion on this list. This is a stackless issue in the first place, not PyPy's. I was just inviting people here to a sprint. Stackless matters go to the stackless list. See http://www.stackless.com/mailman/listinfo/stackless Also the issue of continuations has been discussed at length there. Please use this list for question about PyPy. cheers - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From cfbolz at gmx.de Mon Sep 17 10:34:11 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 17 Sep 2007 10:34:11 +0200 Subject: [pypy-dev] Stackless PyPy mini sprint 2007 In-Reply-To: <46EDDB72.8050603@stackless.com> References: <46EACDFF.4020301@stackless.com> <46ED7240.3060903@stackless.com> <7577F030-2075-4009-B05F-FCB2B12CF519@medallia.com> <46EDDB72.8050603@stackless.com> Message-ID: <46EE3C03.7000507@gmx.de> Christian Tismer wrote: [snip discussions about continuations] > Please don't continue this discussion on this list. > This is a stackless issue in the first place, not PyPy's. > I was just inviting people here to a sprint. > Stackless matters go to the stackless list. > > See http://www.stackless.com/mailman/listinfo/stackless > > Also the issue of continuations has been discussed at length there. > Please use this list for question about PyPy. While I agree with Christian's opinions that true continuations are probably a can of worms that you don't want to open, I don't see this discussion to be wrongly placed on this list. PyPy-dev is quite low-traffic and many of the things discussed here are "off-topic". Cheers, Carl Friedrich From arigo at tunes.org Mon Sep 17 12:31:23 2007 From: arigo at tunes.org (Armin Rigo) Date: Mon, 17 Sep 2007 12:31:23 +0200 Subject: [pypy-dev] Stackless PyPy mini sprint 2007 In-Reply-To: <7577F030-2075-4009-B05F-FCB2B12CF519@medallia.com> References: <46EACDFF.4020301@stackless.com> <46ED7240.3060903@stackless.com> <7577F030-2075-4009-B05F-FCB2B12CF519@medallia.com> Message-ID: <20070917103122.GA30798@code0.codespeak.net> Hi Erik, On Mon, Sep 17, 2007 at 03:00:51AM +0200, Erik Gorset wrote: > Btw, it's possible to simulate some of call/cc's behavior using > _stackless.fork() (I have a play-implementation lying around here > somewhere). But it has several issues. The biggest one is that fork() > actually copies some of the objects found on the stack. It's been > a while since I've played around with cloneable and fork, so maybe > this has been fixed. A deep issue with continuations in a language like Python is that there is no single best definition of what it should capture and what it should not. If you think about what you would like to occur when you capture a continuation or fork a coroutine, then you'll notice that you actually want some objects on the stack to be duplicated, and others not. For example, in: def f(n): return [x*x for x in range(n)] which, remember, is equivalent to: def f(n): lst = [] for x in range(n): lst.append(x*x) return lst if we pause this function in the middle of the loop and fork it, then it feels "right" to duplicate the list -- otherwise both the original and the copy will append the remaining items into the same list, resulting in a single list with more than n elements. In essence, it seems to me that both forking and full continuations should not be part of the core of a language, for the same reasons as pickling -- in my own experience at least, more than half of the time "pickle" doesn't capture enough or capture too much. There is no single best solution. A bientot, Armin. From arigo at tunes.org Mon Sep 17 12:33:10 2007 From: arigo at tunes.org (Armin Rigo) Date: Mon, 17 Sep 2007 12:33:10 +0200 Subject: [pypy-dev] tephan. In-Reply-To: <46EC7D45.1030200@stackless.com> References: <20070915133824.B7E2B80A6@code0.codespeak.net> <46EC7D45.1030200@stackless.com> Message-ID: <20070917103310.GB30798@code0.codespeak.net> Hi, On Sun, Sep 16, 2007 at 02:48:05AM +0200, Christian Tismer wrote: > Something in this check-in has a huge impact on all of the > stackless tests in pypy/lib/app-tests. Fixed in r46676. Armin From tismer at stackless.com Mon Sep 17 15:25:30 2007 From: tismer at stackless.com (Christian Tismer) Date: Mon, 17 Sep 2007 15:25:30 +0200 Subject: [pypy-dev] continuations (was: Stackless PyPy mini sprint 2007) In-Reply-To: <46EE3C03.7000507@gmx.de> References: <46EACDFF.4020301@stackless.com> <46ED7240.3060903@stackless.com> <7577F030-2075-4009-B05F-FCB2B12CF519@medallia.com> <46EDDB72.8050603@stackless.com> <46EE3C03.7000507@gmx.de> Message-ID: <46EE804A.9010307@stackless.com> Carl Friedrich Bolz wrote: > Christian Tismer wrote: > [snip discussions about continuations] >> Please don't continue this discussion on this list. >> This is a stackless issue in the first place, not PyPy's. >> I was just inviting people here to a sprint. >> Stackless matters go to the stackless list. >> >> See http://www.stackless.com/mailman/listinfo/stackless >> >> Also the issue of continuations has been discussed at length there. >> Please use this list for question about PyPy. > > While I agree with Christian's opinions that true continuations are > probably a can of worms that you don't want to open, I don't see this > discussion to be wrongly placed on this list. PyPy-dev is quite > low-traffic and many of the things discussed here are "off-topic". Ok, no problem, PyPy is the platform with the most chances to do this right, so why not discuss it here. I'd just prefer to change the topic then, since this is about a stackless sprint to finish and clean up things which are waiting. These things are going to be folded back into CPython stackless, where the re-incarnation of continuations is pretty unlikely. cheers - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From cfbolz at gmx.de Mon Sep 17 16:02:18 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 17 Sep 2007 16:02:18 +0200 Subject: [pypy-dev] continuations In-Reply-To: <46EE804A.9010307@stackless.com> References: <46EACDFF.4020301@stackless.com> <46ED7240.3060903@stackless.com> <7577F030-2075-4009-B05F-FCB2B12CF519@medallia.com> <46EDDB72.8050603@stackless.com> <46EE3C03.7000507@gmx.de> <46EE804A.9010307@stackless.com> Message-ID: <46EE88EA.9020108@gmx.de> Christian Tismer wrote: > Ok, no problem, PyPy is the platform with the most chances > to do this right, so why not discuss it here. > > I'd just prefer to change the topic then, since this is about a > stackless sprint to finish and clean up things which are waiting. > These things are going to be folded back into CPython stackless, > where the re-incarnation of continuations is pretty unlikely. Oh, absolutely. A change of topic (or even a new thread) would have been good. Cheers, Carl Friedrich From cfbolz at gmx.de Mon Sep 17 16:19:38 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 17 Sep 2007 16:19:38 +0200 Subject: [pypy-dev] pypy with sympy In-Reply-To: <85b5c3130709150623p5de74662j107cccc39c65db8c@mail.gmail.com> References: <85b5c3130709141758s2c832322uc4b986144b87a14e@mail.gmail.com> <46EBD4B6.2050500@gmx.de> <85b5c3130709150617j37b76299o306b713bdc6bbb40@mail.gmail.com> <85b5c3130709150623p5de74662j107cccc39c65db8c@mail.gmail.com> Message-ID: <46EE8CFA.2080209@gmx.de> Ondrej Certik wrote: >> There is another problem actually: >> >> $ py.test --exec=pypy >> * opening PopenGateway: /usr/bin/pypy >> MASTER: initiated slave terminal session -> >> MASTER: send start info, topdir=/home/ondra/sympy >> Traceback (most recent call last): >> File "?", line 32, in run_toplevel >> File "?", line 265, in run_it >> File "", line 1, in >> File "", line 40, in >> ImportError: No module named thread >> Traceback (most recent call last): >> File "/usr/bin/py.test", line 4, in ? >> py.test.cmdline.main() >> File "/usr/lib/python2.4/site-packages/py/test/cmdline.py", line 15, in main >> failures = session.main() >> File "/usr/lib/python2.4/site-packages/py/test/terminal/remote.py", >> line 82, in main >> failures = self.run_remote_session(failures) >> File "/usr/lib/python2.4/site-packages/py/test/terminal/remote.py", >> line 111, in run_remote_session >> return channel.receive() >> File "/usr/lib/python2.4/site-packages/py/execnet/channel.py", line >> 175, in receive >> raise self._getremoteerror() or EOFError() >> EOFError > > I forgot to add - I am using the versions from Debian unstable. I > don't have time to test everything from svn, however, if you want, I > can make new versions of pypy/py.test available in Debian when you > release them, but I think the current maintainer will do also do it - > Alexandre Fayolle is the maintainer of those packages - should I > report bugs in them to Debian BTS, or here? > > I think the way to go is just users report problems against the > packages in their distributions, like Debian, it is forwarded here, > you fix it in svn, release a new version, it gets to Debian, etc. Ah, I that can be circumvented by doing pypy `which py.test` ... instead of py.test --exec=pypy ... Cheers, Carl Friedrich From ondrej at certik.cz Mon Sep 17 16:32:13 2007 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 17 Sep 2007 16:32:13 +0200 Subject: [pypy-dev] pypy with sympy In-Reply-To: <46EE8CFA.2080209@gmx.de> References: <85b5c3130709141758s2c832322uc4b986144b87a14e@mail.gmail.com> <46EBD4B6.2050500@gmx.de> <85b5c3130709150617j37b76299o306b713bdc6bbb40@mail.gmail.com> <85b5c3130709150623p5de74662j107cccc39c65db8c@mail.gmail.com> <46EE8CFA.2080209@gmx.de> Message-ID: <85b5c3130709170732m3c32dd1co7aa7351c7762aa81@mail.gmail.com> On 9/17/07, Carl Friedrich Bolz wrote: > Ondrej Certik wrote: > >> There is another problem actually: > >> > >> $ py.test --exec=pypy > >> * opening PopenGateway: /usr/bin/pypy > >> MASTER: initiated slave terminal session -> > >> MASTER: send start info, topdir=/home/ondra/sympy > >> Traceback (most recent call last): > >> File "?", line 32, in run_toplevel > >> File "?", line 265, in run_it > >> File "", line 1, in > >> File "", line 40, in > >> ImportError: No module named thread > >> Traceback (most recent call last): > >> File "/usr/bin/py.test", line 4, in ? > >> py.test.cmdline.main() > >> File "/usr/lib/python2.4/site-packages/py/test/cmdline.py", line 15, in main > >> failures = session.main() > >> File "/usr/lib/python2.4/site-packages/py/test/terminal/remote.py", > >> line 82, in main > >> failures = self.run_remote_session(failures) > >> File "/usr/lib/python2.4/site-packages/py/test/terminal/remote.py", > >> line 111, in run_remote_session > >> return channel.receive() > >> File "/usr/lib/python2.4/site-packages/py/execnet/channel.py", line > >> 175, in receive > >> raise self._getremoteerror() or EOFError() > >> EOFError > > > > I forgot to add - I am using the versions from Debian unstable. I > > don't have time to test everything from svn, however, if you want, I > > can make new versions of pypy/py.test available in Debian when you > > release them, but I think the current maintainer will do also do it - > > Alexandre Fayolle is the maintainer of those packages - should I > > report bugs in them to Debian BTS, or here? > > > > I think the way to go is just users report problems against the > > packages in their distributions, like Debian, it is forwarded here, > > you fix it in svn, release a new version, it gets to Debian, etc. > > Ah, I that can be circumvented by doing > > pypy `which py.test` ... > > instead of > > py.test --exec=pypy ... I tried that one too: $ pypy `which py.test` Traceback (most recent call last): File "?", line 32, in run_toplevel File "/usr/bin/py.test", line 3, in import py ImportError: No module named py Ondrej From cfbolz at gmx.de Mon Sep 17 17:39:09 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 17 Sep 2007 17:39:09 +0200 Subject: [pypy-dev] pypy with sympy In-Reply-To: References: Message-ID: <46EE9F9D.8000809@gmx.de> Adam Olsen wrote: >> 4) Myabe allow to speed my code up (JIT), or make Python as fast as >> possible. But this is rather a long term goal. Generally, I don't see >> why my code written in C and the same code written in Python, not >> using much dynamic features, couldn't be the same fast. (Of course I >> understand why now it is slower, Python is interpreted, etc. But I >> mean in principle. Python can have as much information about my code >> as the C compiler.) I use Python for devising new numerical >> algorithms, and than I need to rewrite them by hand to fortran for >> speed. It's a pain. > >> 6) Allow efficient threading? CPython has the GIL (for good >> technological reasons). Generally pypy can improve how to do things in >> parallel in python. Currently, one needs to use pympi, pypar, or >> something. > > These two together are, in my mind, where PyPy can really shine. > Performance on par with CPython is critical to be taken seriously, but > CPython's difficulties with the GIL mean I can only make it scalable > at about 60% to 65% of the traditional efficiency (on pystones at > least). That gap gives PyPy enough room to surpass CPython on > scalable performance. > We are confident that we can surpass CPython's speed with the help of the JIT. As for free threading, the big prerequisite for that is a GC that plays well with threads. Boehm is not very good in this respect. After this is done we can think about how to insert locking operations in an automatic way to reduce the likelyhood of errors (this is currently already done for releasing the GIL around external function calls). Then there will probably be a huge amount of painful debugging :-) Cheers, Carl Friedrich From ondrej at certik.cz Mon Sep 17 17:51:54 2007 From: ondrej at certik.cz (Ondrej Certik) Date: Mon, 17 Sep 2007 17:51:54 +0200 Subject: [pypy-dev] pypy with sympy In-Reply-To: <46EE9F9D.8000809@gmx.de> References: <46EE9F9D.8000809@gmx.de> Message-ID: <85b5c3130709170851m58ddfa39we707dc04dd5557e9@mail.gmail.com> > We are confident that we can surpass CPython's speed with the help of > the JIT. As for free threading, the big prerequisite for that is a GC > that plays well with threads. Boehm is not very good in this respect. > After this is done we can think about how to insert locking operations > in an automatic way to reduce the likelihood of errors (this is > currently already done for releasing the GIL around external function > calls). Then there will probably be a huge amount of painful debugging :-) I remembered other few remarks to point out: 1) Current pypy in Debian doesn't have modules: thread, ctypes. Thread is needed for py.test, ctypes for plotting facilities in sympy. Ondrej From tismer at stackless.com Mon Sep 17 18:23:30 2007 From: tismer at stackless.com (Christian Tismer) Date: Mon, 17 Sep 2007 18:23:30 +0200 Subject: [pypy-dev] pypy with sympy In-Reply-To: <46EE9F9D.8000809@gmx.de> References: <46EE9F9D.8000809@gmx.de> Message-ID: <46EEAA02.7080408@stackless.com> Carl Friedrich Bolz wrote: ... > We are confident that we can surpass CPython's speed with the help of > the JIT. As for free threading, the big prerequisite for that is a GC > that plays well with threads. Boehm is not very good in this respect. > After this is done we can think about how to insert locking operations > in an automatic way to reduce the likelyhood of errors (this is > currently already done for releasing the GIL around external function > calls). Then there will probably be a huge amount of painful debugging :-) I'm actually no longer convinced that it makes soo much sense to get rid of the GIL. Python's data model is not made for free threading. Have you had a look into http://www.parallelpython.com/ ? Might be a starting point into threads with disjoint interpreters and some locking just for data exchange. cheers - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From rhamph at gmail.com Mon Sep 17 19:39:23 2007 From: rhamph at gmail.com (Adam Olsen) Date: Mon, 17 Sep 2007 11:39:23 -0600 Subject: [pypy-dev] pypy with sympy In-Reply-To: <46EE9F9D.8000809@gmx.de> References: <46EE9F9D.8000809@gmx.de> Message-ID: On 9/17/07, Carl Friedrich Bolz wrote: > Adam Olsen wrote: > >> 4) Myabe allow to speed my code up (JIT), or make Python as fast as > >> possible. But this is rather a long term goal. Generally, I don't see > >> why my code written in C and the same code written in Python, not > >> using much dynamic features, couldn't be the same fast. (Of course I > >> understand why now it is slower, Python is interpreted, etc. But I > >> mean in principle. Python can have as much information about my code > >> as the C compiler.) I use Python for devising new numerical > >> algorithms, and than I need to rewrite them by hand to fortran for > >> speed. It's a pain. > > > >> 6) Allow efficient threading? CPython has the GIL (for good > >> technological reasons). Generally pypy can improve how to do things in > >> parallel in python. Currently, one needs to use pympi, pypar, or > >> something. > > > > These two together are, in my mind, where PyPy can really shine. > > Performance on par with CPython is critical to be taken seriously, but > > CPython's difficulties with the GIL mean I can only make it scalable > > at about 60% to 65% of the traditional efficiency (on pystones at > > least). That gap gives PyPy enough room to surpass CPython on > > scalable performance. > > > > We are confident that we can surpass CPython's speed with the help of > the JIT. As for free threading, the big prerequisite for that is a GC > that plays well with threads. Boehm is not very good in this respect. > After this is done we can think about how to insert locking operations > in an automatic way to reduce the likelyhood of errors (this is > currently already done for releasing the GIL around external function > calls). Then there will probably be a huge amount of painful debugging :-) I'm confident CPython will provide a language-level solution to locking. This does mean waiting over a year from this point though. -- Adam Olsen, aka Rhamphoryncus From erik at medallia.com Tue Sep 18 02:57:25 2007 From: erik at medallia.com (Erik Gorset) Date: Tue, 18 Sep 2007 02:57:25 +0200 Subject: [pypy-dev] continuations In-Reply-To: <46EE88EA.9020108@gmx.de> References: <46EACDFF.4020301@stackless.com> <46ED7240.3060903@stackless.com> <7577F030-2075-4009-B05F-FCB2B12CF519@medallia.com> <46EDDB72.8050603@stackless.com> <46EE3C03.7000507@gmx.de> <46EE804A.9010307@stackless.com> <46EE88EA.9020108@gmx.de> Message-ID: <3DA3B58D-07A3-4300-924A-DDE2F029CFFC@medallia.com> On Sep 17, 2007, at 4:02 PM, Carl Friedrich Bolz wrote: > Christian Tismer wrote: >> Ok, no problem, PyPy is the platform with the most chances >> to do this right, so why not discuss it here. >> >> I'd just prefer to change the topic then, since this is about a >> stackless sprint to finish and clean up things which are waiting. >> These things are going to be folded back into CPython stackless, >> where the re-incarnation of continuations is pretty unlikely. > > Oh, absolutely. A change of topic (or even a new thread) would have > been > good. Ok, thanks. I hope I'm not stepping on anyones toes by discussing continuations here. I'm a big fan of pypy project and find the work you're doing very interesting. I was just curious if there was any plans for supporting my favorite language feature :-) I'll insert my answer to Armin here: On Sep 17, 2007, at 12:31 PM, Armin Rigo wrote: > On Mon, Sep 17, 2007 at 03:00:51AM +0200, Erik Gorset wrote: >> Btw, it's possible to simulate some of call/cc's behavior using >> _stackless.fork() (I have a play-implementation lying around here >> somewhere). But it has several issues. The biggest one is that fork() >> actually copies some of the objects found on the stack. It's been >> a while since I've played around with cloneable and fork, so maybe >> this has been fixed. > > A deep issue with continuations in a language like Python is that > there > is no single best definition of what it should capture and what it > should not. If you think about what you would like to occur when you > capture a continuation or fork a coroutine, then you'll notice that > you > actually want some objects on the stack to be duplicated, and others > not. Actually, continuation should never copy objects. It's only about the non-local returns. For example the return statement makes it possible to jump back to the caller. If return instead was a function which you could save and call again, you would have the equivalent of call/cc. The big point here is that you don't have a stack anymore, but a tree of continuations. > For example, in: > > def f(n): > return [x*x for x in range(n)] > > which, remember, is equivalent to: > > def f(n): > lst = [] > for x in range(n): > lst.append(x*x) > return lst > > if we pause this function in the middle of the loop and fork it, > then it > feels "right" to duplicate the list -- otherwise both the original and > the copy will append the remaining items into the same list, resulting > in a single list with more than n elements. This is similar to problems with some implementation of map in scheme, not all are safe to use with call/cc. Some care must be taken to make sure different control flows are compatible with each other. With great power comes great responsibility ;-) The thing is, with call/cc it's possible to embed different paradigms in python like constraint, logic and dataflow programming. I find this very interesting :-) > In essence, it seems to me that both forking and full continuations > should not be part of the core of a language, for the same reasons as > pickling -- in my own experience at least, more than half of the time > "pickle" doesn't capture enough or capture too much. There is no > single > best solution. I think I will have to disagree here. Several uses of continuations already exists in the language like generators/coroutines, the return statement, loops and exceptions. True continuations unifies this into one concept with simple semantics. The option for a virtual machine implementation is then to either implement all the normal flow controls as special cases, or use true continuations as the fundamental building block for the rest. -- Erik Gorset -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 1962 bytes Desc: not available Url : http://codespeak.net/pipermail/pypy-dev/attachments/20070918/85561e7f/attachment-0001.bin From arigo at tunes.org Wed Sep 19 13:58:14 2007 From: arigo at tunes.org (Armin Rigo) Date: Wed, 19 Sep 2007 13:58:14 +0200 Subject: [pypy-dev] pypy with sympy In-Reply-To: <85b5c3130709170732m3c32dd1co7aa7351c7762aa81@mail.gmail.com> References: <85b5c3130709141758s2c832322uc4b986144b87a14e@mail.gmail.com> <46EBD4B6.2050500@gmx.de> <85b5c3130709150617j37b76299o306b713bdc6bbb40@mail.gmail.com> <85b5c3130709150623p5de74662j107cccc39c65db8c@mail.gmail.com> <46EE8CFA.2080209@gmx.de> <85b5c3130709170732m3c32dd1co7aa7351c7762aa81@mail.gmail.com> Message-ID: <20070919115814.GA8402@code0.codespeak.net> Hi Ondrej, On Mon, Sep 17, 2007 at 04:32:13PM +0200, Ondrej Certik wrote: > $ pypy `which py.test` > ImportError: No module named py That's obviously because the python path of pypy doesn't allow it to find the 'py' module. You don't have this problem if you don't use the globally-installed py.test, but the one from a PyPy checkout instead: pypy .../pypy-dist/py/bin/py.test args... or equivalently pypy .../pypy-dist/pypy/test_all.py args... A bientot, Armin. From cfbolz at gmx.de Wed Sep 19 14:19:51 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Wed, 19 Sep 2007 14:19:51 +0200 Subject: [pypy-dev] pypy with sympy In-Reply-To: <20070919115814.GA8402@code0.codespeak.net> References: <85b5c3130709141758s2c832322uc4b986144b87a14e@mail.gmail.com> <46EBD4B6.2050500@gmx.de> <85b5c3130709150617j37b76299o306b713bdc6bbb40@mail.gmail.com> <85b5c3130709150623p5de74662j107cccc39c65db8c@mail.gmail.com> <46EE8CFA.2080209@gmx.de> <85b5c3130709170732m3c32dd1co7aa7351c7762aa81@mail.gmail.com> <20070919115814.GA8402@code0.codespeak.net> Message-ID: <348899050709190519k764d7973n602e95c77b101945@mail.gmail.com> Hi Armin, 2007/9/19, Armin Rigo : > On Mon, Sep 17, 2007 at 04:32:13PM +0200, Ondrej Certik wrote: > > $ pypy `which py.test` > > ImportError: No module named py > > That's obviously because the python path of pypy doesn't allow it to > find the 'py' module. You don't have this problem if you don't use the > globally-installed py.test, but the one from a PyPy checkout instead: > > pypy .../pypy-dist/py/bin/py.test args... > > or equivalently > > pypy .../pypy-dist/pypy/test_all.py args... that doesn't really help if you don't have a pypy checkout at all, but only use the pypy binary installed by Debian. I guess some rethinking there is in order. Cheers, Carl Friedrich From ondrej at certik.cz Wed Sep 19 14:22:38 2007 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 19 Sep 2007 14:22:38 +0200 Subject: [pypy-dev] pypy with sympy In-Reply-To: <20070919115814.GA8402@code0.codespeak.net> References: <85b5c3130709141758s2c832322uc4b986144b87a14e@mail.gmail.com> <46EBD4B6.2050500@gmx.de> <85b5c3130709150617j37b76299o306b713bdc6bbb40@mail.gmail.com> <85b5c3130709150623p5de74662j107cccc39c65db8c@mail.gmail.com> <46EE8CFA.2080209@gmx.de> <85b5c3130709170732m3c32dd1co7aa7351c7762aa81@mail.gmail.com> <20070919115814.GA8402@code0.codespeak.net> Message-ID: <85b5c3130709190522u6cd98510o83ea4adcd0a0d2f4@mail.gmail.com> Hi Armin, > That's obviously because the python path of pypy doesn't allow it to > find the 'py' module. You don't have this problem if you don't use the > globally-installed py.test, but the one from a PyPy checkout instead: > > pypy .../pypy-dist/py/bin/py.test args... > > or equivalently > > pypy .../pypy-dist/pypy/test_all.py args... I tried: ondra at fuji:~/sympy$ PYTHONPATH="/usr/lib/python2.4/site-packages/" pypy `which py.test` and got many errors, but I think all are the same: def __init__(self, targetfd, tmpfile=None): self.targetfd = targetfd if tmpfile is None: > tmpfile = self.maketmpfile() [/usr/lib/python2.4/site-packages/py/io/fdcapture.py:12] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def maketmpfile(self): """ create a temporary file """ E f = os.tmpfile() > AttributeError: 'module' object has no attribute 'tmpfile' [/usr/lib/python2.4/site-packages/py/io/fdcapture.py:44] _______________________________________________________________________________ ================= tests finished: 21 failed in 11.93 seconds ================== So, I think it's the problem in py lib in debian, that it doesn't work with pypy? Then I tried as you suggested now: in /tmp: $ svn co http://codespeak.net/svn/pypy/dist pypy-dist in ~/sympy: $ pypy /tmp/pypy-dist/py/bin/py.test But the exact same problem as above: def maketmpfile(self): """ create a temporary file """ E f = os.tmpfile() > AttributeError: 'module' object has no attribute 'tmpfile' [/tmp/pypy-dist/py/io/fdcapture.py:44] _______________________________________________________________________________ ================== tests finished: 21 failed in 1.80 seconds ================== then I tried: $ pypy /tmp/pypy-dist/pypy/test_all.py and again the same problem. Maybe just the pypy in Debian is old? In this case maybe it's time for a new release. :) If you tell me what to fix, I'll send a patch to the debian maintainer to fix it in Debian. Ondrej From faassen at startifact.com Wed Sep 19 17:24:17 2007 From: faassen at startifact.com (Martijn Faassen) Date: Wed, 19 Sep 2007 17:24:17 +0200 Subject: [pypy-dev] pypy with sympy In-Reply-To: <200709151405.l8FE5fk8029135@theraft.openend.se> References: <85b5c3130709141758s2c832322uc4b986144b87a14e@mail.gmail.com> <46EBD4B6.2050500@gmx.de> <85b5c3130709150617j37b76299o306b713bdc6bbb40@mail.gmail.com> <200709151405.l8FE5fk8029135@theraft.openend.se> Message-ID: Laura Creighton wrote: [snip] > It would be nice to find out exactly who wants what next from PyPy, > and perhaps more importantly, who would be willing to help on PyPy, > but cannot now, and what they would need in the way of changes before > they could contribute. Besides fixes for the bugs you have reported, > is there anything in particular you wish we were doing next? I'm glad to see this important discussion is taking place again. For my input, see my mails from a few months ago. :) Regards, Martijn From faassen at startifact.com Wed Sep 19 17:27:14 2007 From: faassen at startifact.com (Martijn Faassen) Date: Wed, 19 Sep 2007 17:27:14 +0200 Subject: [pypy-dev] pypy with sympy In-Reply-To: <46EC6B22.1060600@gmx.de> References: <85b5c3130709141758s2c832322uc4b986144b87a14e@mail.gmail.com> <46EBD4B6.2050500@gmx.de> <85b5c3130709150617j37b76299o306b713bdc6bbb40@mail.gmail.com> <200709151405.l8FE5fk8029135@theraft.openend.se> <85b5c3130709150813s20ba3e4exd5a516953400cf4a@mail.gmail.com> <46EC6B22.1060600@gmx.de> Message-ID: Hey, Carl Friedrich Bolz wrote: > > 1) Make predictable releases, get pypy into distributions, fix bugs > > and just make it another Python interpreter. > > That one is a bit boring, don't you think? :-) No seriously, we can't > really compete with CPython on its home turf, we need additional > features that set us apart. Heh, it's definitely boring, but it's essential if you want PyPy to be adopted more widely. You already have plenty of features. I want PyPy to be in a state so I can use it in production. :) Regards, Martijn From cfbolz at gmx.de Wed Sep 19 17:39:18 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Wed, 19 Sep 2007 17:39:18 +0200 Subject: [pypy-dev] pypy with sympy In-Reply-To: References: <85b5c3130709141758s2c832322uc4b986144b87a14e@mail.gmail.com> <46EBD4B6.2050500@gmx.de> <85b5c3130709150617j37b76299o306b713bdc6bbb40@mail.gmail.com> <200709151405.l8FE5fk8029135@theraft.openend.se> <85b5c3130709150813s20ba3e4exd5a516953400cf4a@mail.gmail.com> <46EC6B22.1060600@gmx.de> Message-ID: <348899050709190839v20623b04wa8a84c2ae9000025@mail.gmail.com> Hi Martijn, 2007/9/19, Martijn Faassen : > Carl Friedrich Bolz wrote: > > > 1) Make predictable releases, get pypy into distributions, fix bugs > > > and just make it another Python interpreter. > > > > That one is a bit boring, don't you think? :-) No seriously, we can't > > really compete with CPython on its home turf, we need additional > > features that set us apart. > > Heh, it's definitely boring, but it's essential if you want PyPy to be > adopted more widely. You already have plenty of features. I want PyPy to > be in a state so I can use it in production. :) I was referring to the "make it another Python interpreter", not to the first part. Probably wasn't all too clear. Cheers, Carl Friedrich From ondrej at certik.cz Wed Sep 19 17:43:22 2007 From: ondrej at certik.cz (Ondrej Certik) Date: Wed, 19 Sep 2007 17:43:22 +0200 Subject: [pypy-dev] pypy with sympy In-Reply-To: <348899050709190839v20623b04wa8a84c2ae9000025@mail.gmail.com> References: <85b5c3130709141758s2c832322uc4b986144b87a14e@mail.gmail.com> <46EBD4B6.2050500@gmx.de> <85b5c3130709150617j37b76299o306b713bdc6bbb40@mail.gmail.com> <200709151405.l8FE5fk8029135@theraft.openend.se> <85b5c3130709150813s20ba3e4exd5a516953400cf4a@mail.gmail.com> <46EC6B22.1060600@gmx.de> <348899050709190839v20623b04wa8a84c2ae9000025@mail.gmail.com> Message-ID: <85b5c3130709190843w5d9f2cc6ydd217dc06ba7caaa@mail.gmail.com> > I was referring to the "make it another Python interpreter", not to > the first part. Probably wasn't all too clear. Well, but I think that "make it another Python interpreter" is maybe boring but essential. When substituting pypy for python, it shoud just work. Ondrej From simon at arrowtheory.com Wed Sep 19 22:31:19 2007 From: simon at arrowtheory.com (Simon Burton) Date: Wed, 19 Sep 2007 13:31:19 -0700 Subject: [pypy-dev] about breaking RPython, RCTypes In-Reply-To: References: Message-ID: <20070919133119.41621464.simon@arrowtheory.com> On Wed, 20 Jun 2007 12:23:06 +0200 Martijn Faassen wrote: ... > > I happen to be interested in use case a) though. It is quite exciting to > me that we could have a faster template language interpreter What is this interpreter you refer to here ? Simon. From simon at arrowtheory.com Wed Sep 19 22:36:43 2007 From: simon at arrowtheory.com (Simon Burton) Date: Wed, 19 Sep 2007 13:36:43 -0700 Subject: [pypy-dev] pypy-dev@codespeak.net In-Reply-To: <20070819103308.GA28257@code0.codespeak.net> References: <20070819103308.GA28257@code0.codespeak.net> Message-ID: <20070919133643.3ee8394a.simon@arrowtheory.com> On Sun, 19 Aug 2007 12:33:09 +0200 Armin Rigo wrote: > > Hi all, > > Those that follow IRC already know it, but it's worth being announced a > bit more widely: I've been working on a form of sandboxing for RPython > programs, which now seems to work for the whole of PyPy. > > It's "sandboxing" as in "full virtualization", but done in normal C with > no OS support at all. It's a two-processes model: we can translate PyPy > to a special "pypy-c-sandbox" executable, which is safe in the sense > that it doesn't do any library or system call - instead, whenever it > would like to perform such an operation, it marshals the operation name > and the arguments to its stdout and it waits for the marshalled result > on its stdin. This pypy-c-sandbox process is meant to be run by an > outer "controller" program that answers to these operation requests. How is this different to just linking against a libc wrapper (that does whatever marshal magic is required) ? Simon. From simon at arrowtheory.com Thu Sep 20 00:37:47 2007 From: simon at arrowtheory.com (Simon Burton) Date: Wed, 19 Sep 2007 15:37:47 -0700 Subject: [pypy-dev] transactional memory Message-ID: <20070919153747.ee8f0749.simon@arrowtheory.com> Someone on pypy irc posted a link to a talk by Simon Peyton-Jones on transactional memory: http://oscon.blip.tv/file/317758/ it's worth checking out. Also, i noticed on the llvm list someone has added support for TM: http://lists.cs.uiuc.edu/pipermail/llvmdev/2007-August/010365.html Simon. From rahulgarg44 at gmail.com Thu Sep 20 12:15:30 2007 From: rahulgarg44 at gmail.com (Rahul Garg) Date: Thu, 20 Sep 2007 04:15:30 -0600 Subject: [pypy-dev] llvm backend status? Message-ID: Hi. I recently discovered PyPy and its one cool project in which I would love to be involved in some capacity :) So I have a bunch of questions about the direction and status of the project a) How is the LLVM backend coming along? b) What really are the focus areas of the project currently? c) Any wishlists ? thanks, rahul -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20070920/8931102d/attachment.htm From santagada at gmail.com Thu Sep 20 17:25:52 2007 From: santagada at gmail.com (Leonardo Santagada) Date: Thu, 20 Sep 2007 12:25:52 -0300 Subject: [pypy-dev] llvm backend status? In-Reply-To: References: Message-ID: <25D6F8C5-5FED-46B0-89BD-6427C94D7F0E@gmail.com> Em Sep 20, 2007, ?s 7:15 AM, Rahul Garg escreveu: > Hi. > > I recently discovered PyPy and its one cool project in which I > would love to > be involved in some capacity :) GReat just great > So I have a bunch of questions about the direction and status of > the project > > > a) How is the LLVM backend coming along? The LLVM backend is unmaintened for some time, so to ressurect it you would need to work a little on it. Recently someone commited some changes that made most of the tests passing again using LLVM 1.9 so what you would need to do is: * update LLVM to use the LLVM 2.0 sintax (specially accepting LLVM 2.0 generated code from llvm-gcc) * make threads work (I think this was one of the things the last commit removed) * make rffi work on it (this is a new way to call C code, created after the LLVM got unmaintened). * Lot's of other little things... * profit :) > b) What really are the focus areas of the project currently? After the eu project ended people took some time off the project to recharge the batteries... and now people have started working again, and the focus is on cleanups, you can find more on pypy/doc/cleanup- todo.txt. > c) Any wishlists ? I think that everyone has its own besides cleanup-todo, mine is suport for unicode at the RPython level, a new garbage collector, and the revival of the LLVM backend, and of course the completion of the JIT. This is all from someone following the development of pypy from the outside, but I think I got most things right :) []'s -- Leonardo Santagada From rahulgarg44 at gmail.com Fri Sep 21 00:21:03 2007 From: rahulgarg44 at gmail.com (Rahul Garg) Date: Thu, 20 Sep 2007 16:21:03 -0600 Subject: [pypy-dev] llvm backend status? In-Reply-To: <25D6F8C5-5FED-46B0-89BD-6427C94D7F0E@gmail.com> References: <25D6F8C5-5FED-46B0-89BD-6427C94D7F0E@gmail.com> Message-ID: Hi. Thanks for the info :) My plan is to first use pypy for a few weeks as a user (perhaps writing a few benchmarks, testing out stuff like AOP, reading the docs on pypy and psyco etc). I hope to be involved more directly from sometime in october. thanks, rahul On 9/20/07, Leonardo Santagada wrote: > > > Em Sep 20, 2007, ?s 7:15 AM, Rahul Garg escreveu: > > > Hi. > > > > I recently discovered PyPy and its one cool project in which I > > would love to > > be involved in some capacity :) > > GReat just great > > > So I have a bunch of questions about the direction and status of > > the project > > > > > > a) How is the LLVM backend coming along? > > The LLVM backend is unmaintened for some time, so to ressurect it you > would need to work a little on it. Recently someone commited some > changes that made most of the tests passing again using LLVM 1.9 so > what you would need to do is: > * update LLVM to use the LLVM 2.0 sintax (specially accepting LLVM > 2.0 generated code from llvm-gcc) > * make threads work (I think this was one of the things the last > commit removed) > * make rffi work on it (this is a new way to call C code, created > after the LLVM got unmaintened). > * Lot's of other little things... > * profit :) > > > > b) What really are the focus areas of the project currently? > > After the eu project ended people took some time off the project to > recharge the batteries... and now people have started working again, > and the focus is on cleanups, you can find more on pypy/doc/cleanup- > todo.txt. > > > c) Any wishlists ? > > I think that everyone has its own besides cleanup-todo, mine is > suport for unicode at the RPython level, a new garbage collector, and > the revival of the LLVM backend, and of course the completion of the > JIT. > > This is all from someone following the development of pypy from the > outside, but I think I got most things right :) > > []'s > -- > Leonardo Santagada > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20070920/2e7def0f/attachment.htm From cfbolz at gmx.de Sat Sep 22 09:18:50 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Sat, 22 Sep 2007 09:18:50 +0200 Subject: [pypy-dev] PyPy/Squeak Sprint Announcement Message-ID: <46F4C1DA.7080004@gmx.de> Bern Squeak/PyPy sprint October 22 - October 26 2007 ===================================================== We are proud to announce the first Squeak-PyPy-collaboration-sprint. The plan is to bring people from the Squeak/Smalltalk and the PyPy/Python communities together to learn about each others projects and to think about collaboration possibilies. The sprint will be hosted by the Software Composition Group of the University of Bern from the 22nd to the 26th of October 2007. Topics and goals ---------------- The general technical goal of the sprint is to explore implementing the bytecode interpreter as well as the primitives of Squeak in RPython. This should make it eventually possible to auto-generate a JIT for Squeak with the help of PyPy technologies. The other important goal of the sprint is to teach the first-time-sprinters the sprint-driven-development methodology. The first day of the sprint will be a tutorial day, were talks about PyPy, Squeak and sprint-driven-development will be given. The further days will be sprinting days. Of course the most important goal of the sprint is to get to know each other and have some fun :-). Topics of the sprint: - learning about: - PyPy - the internals of Squeaks VM - sprint-driven-development - exploring ways to implement Squeaks bytecode-interpreter and the primitives in Python - or (maybe alternatively) implement a Squeak-backend for PyPy - think about how to load and store of images in PyPy Location -------- The sprint will take place in a seminar room at the University of Bern. Further details of how to get there will be announced before the sprint. Registration ------------ If you'd like to come, please subscribe to the `pypy-sprint mailing list`_ and drop a note about your interests and post any questions. More organisational information will be send to that list. We'll keep a list of `people`_ which we'll update (which you can do so yourself if you have codespeak commit rights). If you have no clue where to stay in Bern ask around on the list, maybe we can help you find something or find some other solution. .. _`pypy-sprint mailing list`: http://codespeak.net/mailman/listinfo/pypy-sprint .. _`people`: http://codespeak.net/pypy/extradoc/sprintinfo/bern2007/people.html From arigo at tunes.org Tue Sep 25 11:03:11 2007 From: arigo at tunes.org (Armin Rigo) Date: Tue, 25 Sep 2007 11:03:11 +0200 Subject: [pypy-dev] pypy-dev@codespeak.net In-Reply-To: <20070919133643.3ee8394a.simon@arrowtheory.com> References: <20070819103308.GA28257@code0.codespeak.net> <20070919133643.3ee8394a.simon@arrowtheory.com> Message-ID: <20070925090311.GA28896@code0.codespeak.net> Hi Simon, On Wed, Sep 19, 2007 at 01:36:43PM -0700, Simon Burton wrote: > > It's "sandboxing" as in "full virtualization", but done in normal C with > > no OS support at all. (...) > > How is this different to just linking against a libc wrapper (that does > whatever marshal magic is required) ? The result is similar; what differs is how we arrive there, and the level of confidence I'd have in the security of the result. In the case of PyPy the wrapping is done automatically and in a platform-independent way; contrast this with the need for the designers of the libc wrapper to carefully close all possible ways the C program could invoke the system and carefully review the result, which is error-prone and platform-specific. More importantly for the user, in the PyPy approach the C code is not random C code, but was generated from RPython. This (together with extra run-time assertions that the translation toolchain can insert for the paranoid) means that buffer overflow or memory management attacks should not be possible. This means that there is no need to review the source code of the whole PyPy interpreter for security issues. By contrast, if you take say CPython and put it inside a libc wrapper, the result is not safe because CPython itself is open to attacks (e.g. memory management issues where carefully crafted app-level Python code could force CPython to execute arbitrary machine code - including system calls bypassing the libc wrapper). A bientot, Armin. From arigo at tunes.org Tue Sep 25 11:05:43 2007 From: arigo at tunes.org (Armin Rigo) Date: Tue, 25 Sep 2007 11:05:43 +0200 Subject: [pypy-dev] pypy with sympy In-Reply-To: <85b5c3130709190522u6cd98510o83ea4adcd0a0d2f4@mail.gmail.com> References: <85b5c3130709141758s2c832322uc4b986144b87a14e@mail.gmail.com> <46EBD4B6.2050500@gmx.de> <85b5c3130709150617j37b76299o306b713bdc6bbb40@mail.gmail.com> <85b5c3130709150623p5de74662j107cccc39c65db8c@mail.gmail.com> <46EE8CFA.2080209@gmx.de> <85b5c3130709170732m3c32dd1co7aa7351c7762aa81@mail.gmail.com> <20070919115814.GA8402@code0.codespeak.net> <85b5c3130709190522u6cd98510o83ea4adcd0a0d2f4@mail.gmail.com> Message-ID: <20070925090542.GB28896@code0.codespeak.net> Hi Ondrej, On Wed, Sep 19, 2007 at 02:22:38PM +0200, Ondrej Certik wrote: > E f = os.tmpfile() > > AttributeError: 'module' object has no attribute 'tmpfile' > > Maybe just the pypy in Debian is old? In this case maybe it's time for > a new release. :) If you tell me what to fix, I'll send a patch to the > debian maintainer to fix it in Debian. Yes, os.tmpfile() was added in the meantime. I personally don't see much the point of making Debian packages of PyPy at the moment, given that we're in a state where you are almost always better off if you start from our Subversion repository instead of from a release a few months old. We're not "user-ready" and this situation will not change before some more months, maybe 1 year. I'd really recommend starting from our 'getting-started' document on the web site instead of the Debian packages. A bientot, Armin. From simon at arrowtheory.com Tue Sep 25 18:38:00 2007 From: simon at arrowtheory.com (Simon Burton) Date: Tue, 25 Sep 2007 09:38:00 -0700 Subject: [pypy-dev] rpython numpy Message-ID: <20070925093800.02d7ad4a.simon@arrowtheory.com> I've been having some fun working on a (tiny) rpython version of numpy. At the heart of numpy is a multi-dimensional array type, ndarray. For example it (the cpython version) can handle processing image arrays like so: >>> import numpy >>> from PIL import Image >>> im=Image.open('jit.png') >>> data=numpy.array(im) now data is a 3 dimensional array, data.ndim==3 and data.shape==(1158, 1173, 3). The last dimension is the color dimension. The cool part of numpy is all the indexing tricks. This is a single pixel: >>> data[0,0,:] array([224, 255, 224], dtype=uint8) this is the red channel: >>> data[:,:,0] array([[224, 224, 224, ..., 224, 224, 224], ... now we mix red and blue: >>> data[:,:,0]+data[:,:,2] (this actually didn't work so well because it's an array of uint8 and we get overflow, but that can be fixed with a cast) The trick with numpy is, every time we use an indexing operation it creates a "view" onto the original data, ie. the memory is aliased. Each ndarray stores a list of "strides" (integers) which specify how many bytes to skip when incrementing an index in a dimension: >>> data.strides (3519, 3, 1) >>> data[:,:,0].strides (3519, 3) Then numpy (at the c level) uses iterators to perform getattr, setattr, and other (pointwise) operations. "broadcasting" allows a smaller array to be used as a larger array, as long as the shape of the smaller array is a suffix of the shape of the larger. >>> green = numpy.array([0,255,0], dtype=numpy.uint8) >>> data[:] = green This sets every pixel to green. I made a design choice with rnumpy to specialize on the number of dimensions (ndim) of an array. This means that we can generate efficient code (more about this below); which was just too tempting to ignore. But also, * it would be impossible to access arbitrary ndim arrays from a pypy wrapped rnumpy. * the shape attribute is read-only. To change the shape of an array, use the reshape method (this does not copy the data): >>> numpy.array(range(12)).reshape((3,4)) array([[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]]) The above mentioned "efficient code" is perhaps a non-issue. Numpy code tends to become memory-bound, not cpu-bound. However it is an interesting use of the rpython optimiser. In general, to iterate over an n-dimensional array requires n-nested for loops, or the use of a "coordinate" array. In this implementation (as in the c implementation) there is an iterator object to hold the coordinates (and the corresponding strides). With malloc removal, all this iterator data become local variables. It's nifty. I don't really know where this is all going; an obvious goal would be to extend all this with a variable dimension array type (and write an applevel numpy). One of the original ideas was to somehow produce cache friendly code by doing lazy operations. Another issue is that it has been mind-boggleingly difficult to directly extend the compiler. I would like to rethink this whole approach; perhaps special methods can help, or some other "low-level to high-level" interaction. Simon. From david at ar.media.kyoto-u.ac.jp Wed Sep 26 04:22:33 2007 From: david at ar.media.kyoto-u.ac.jp (David Cournapeau) Date: Wed, 26 Sep 2007 11:22:33 +0900 Subject: [pypy-dev] rpython numpy In-Reply-To: <20070925093800.02d7ad4a.simon@arrowtheory.com> References: <20070925093800.02d7ad4a.simon@arrowtheory.com> Message-ID: <46F9C269.4090605@ar.media.kyoto-u.ac.jp> Simon Burton wrote: > I've been having some fun working on a (tiny) rpython version of numpy. That's great, this is really one of the most interesting part of pypy for me. > > > The above mentioned "efficient code" is perhaps a non-issue. Numpy > code tends to become memory-bound, not cpu-bound. I tend to agree, at least from my limited observation and hacking on numpy: some (quite fundamental) operations are not cache friendly at all. But I thought this was what rpython + pypy could improve quite a lot, actually. For example, one problem with numpy, specially with broadcasting and the ufunc machinery, is to decide on which dimension to compute for maximum performances. See for example http://www.mail-archive.com/numpy-discussion at scipy.org/msg03759.html. David From brandon_lewis at berkeley.edu Wed Sep 26 06:07:59 2007 From: brandon_lewis at berkeley.edu (Brandon Lewis) Date: Tue, 25 Sep 2007 21:07:59 -0700 Subject: [pypy-dev] New on list Message-ID: <46F9DB1F.8010901@berkeley.edu> Hi, my name is Brandon Lewis. I'm an undergrad Cognitive Science major at U.C. Berkeley. I'm currently taking the upper-division compilers class. Our final project will be writing a JavaScript interpreter. Interestingly enough, we are using Python as the implementation language; so, when I discovered this project while looking through some python documentation, I was intrigued. Mostly just wanted to introduce myself. --Brandon Lewis From cfbolz at gmx.de Wed Sep 26 10:15:40 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Wed, 26 Sep 2007 10:15:40 +0200 Subject: [pypy-dev] New on list In-Reply-To: <46F9DB1F.8010901@berkeley.edu> References: <46F9DB1F.8010901@berkeley.edu> Message-ID: <348899050709260115p67e8d101v815d8892c6e649a6@mail.gmail.com> Hi Brandon! 2007/9/26, Brandon Lewis : > Hi, my name is Brandon Lewis. I'm an undergrad Cognitive Science major > at U.C. Berkeley. I'm currently taking the upper-division compilers class. Welcome! > Our final project will be writing a JavaScript interpreter. > Interestingly enough, we are using Python as the implementation > language; so, when I discovered this project while looking through some > python documentation, I was intrigued. I guess stealing the PyPy JavaScript interpreter andhanding it in is not an option, right? :-) Cheers, Carl Friedrich From santagada at gmail.com Wed Sep 26 12:42:57 2007 From: santagada at gmail.com (Leonardo Santagada) Date: Wed, 26 Sep 2007 07:42:57 -0300 Subject: [pypy-dev] New on list In-Reply-To: <46F9DB1F.8010901@berkeley.edu> References: <46F9DB1F.8010901@berkeley.edu> Message-ID: <8D369274-3FEC-4412-AB23-53DB51F71A93@gmail.com> Em Sep 26, 2007, ?s 1:07 AM, Brandon Lewis escreveu: > Hi, my name is Brandon Lewis. I'm an undergrad Cognitive Science major > at U.C. Berkeley. I'm currently taking the upper-division compilers > class. > > Our final project will be writing a JavaScript interpreter. > Interestingly enough, we are using Python as the implementation > language; so, when I discovered this project while looking through > some > python documentation, I was intrigued. > > Mostly just wanted to introduce myself. Great, if you want to use anything from the PyPy JS interpreter be welcome, and if you want to contribute back to that it would be perfect. But if you need to start from scratch (because your teacher told you to, or for any other reason) we would be happy to help you still. If you have any questions just mail it to the list and/or to me that I would be happy to help you get used to the code base. -- Leonardo Santagada From paul.degrandis at gmail.com Wed Sep 26 13:59:44 2007 From: paul.degrandis at gmail.com (Paul deGrandis) Date: Wed, 26 Sep 2007 07:59:44 -0400 Subject: [pypy-dev] New on list In-Reply-To: <46F9DB1F.8010901@berkeley.edu> References: <46F9DB1F.8010901@berkeley.edu> Message-ID: <9c0bb8a00709260459h162da238ga05f3f08ac4423e9@mail.gmail.com> On 9/26/07, Brandon Lewis wrote: > > Hi, my name is Brandon Lewis. I'm an undergrad Cognitive Science major > at U.C. Berkeley. I'm currently taking the upper-division compilers class. Finally I'm not the only person from the United States! :) Hi Brandon and welcome! I'm a BS/MS student at Drexel University. My BS is in Software Engineering. My MS is in Computer Science with my thesis on Dependable Systems and Autonomic Computing. Our final project will be writing a JavaScript interpreter. > Interestingly enough, we are using Python as the implementation > language; so, when I discovered this project while looking through some > python documentation, I was intrigued. It sounds like you have a great professor! Enjoy the class. Paul Mostly just wanted to introduce myself. > > --Brandon Lewis > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20070926/30f2220a/attachment.htm From adam.jtm30 at gmail.com Wed Sep 26 16:36:18 2007 From: adam.jtm30 at gmail.com (Adam Bark) Date: Wed, 26 Sep 2007 15:36:18 +0100 Subject: [pypy-dev] New on list In-Reply-To: <9c0bb8a00709260459h162da238ga05f3f08ac4423e9@mail.gmail.com> References: <46F9DB1F.8010901@berkeley.edu> <9c0bb8a00709260459h162da238ga05f3f08ac4423e9@mail.gmail.com> Message-ID: On 26/09/2007, Paul deGrandis wrote: > > > > On 9/26/07, Brandon Lewis wrote: > > > > Hi, my name is Brandon Lewis. I'm an undergrad Cognitive Science major > > at U.C. Berkeley. I'm currently taking the upper-division compilers > > class. > > > > Finally I'm not the only person from the United States! :) > Hi Brandon and welcome! I'm a BS/MS student at Drexel University. My BS is in Software Engineering. > My MS is in Computer Science with my thesis on Dependable Systems and > Autonomic Computing. > > Our final project will be writing a JavaScript interpreter. > > Interestingly enough, we are using Python as the implementation > > language; so, when I discovered this project while looking through some > > python documentation, I was intrigued. > > > It sounds like you have a great professor! Enjoy the class. > > Paul > > Mostly just wanted to introduce myself. > > > > --Brandon Lewis > > Hi, I thought while people were introducing themselves I'd jump in. I'm a BSc Physics student at the University of Leicester in England I program almost exclusively for fun, I've been looking at getting into an established project but it's not as easy as it sounds so any help with getting me into pypy development would be cool. Thanks, sorry about the hijack Brandon :-p Adam Bark. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20070926/71eb4f4f/attachment.htm From richardemslie at gmail.com Wed Sep 26 20:07:11 2007 From: richardemslie at gmail.com (Richard Emslie) Date: Wed, 26 Sep 2007 19:07:11 +0100 Subject: [pypy-dev] PyPy cleanups In-Reply-To: <44A9D542-1F27-4E64-9EBA-123407BFB220@gmail.com> References: <44A9D542-1F27-4E64-9EBA-123407BFB220@gmail.com> Message-ID: Hi! Sorry it has taken forever to reply... > > This is the list of possibly orphaned parts of pypy. > We should consider each item here and think in detail > what to do with them. They're mostly broken or not > actively maintained. If nobody shows an interest > to maintain them, deleting would be the best solution > to avoid clutter. Also they pose a maintenance burden > when we proceed with the needed large refactorings. > Maybe it is also a solution to lazily delete them as > they are broken by refactoring if nobody steps up. > > Of course deleted things can easily be brought back from > svn history, if there is renewed interest. So the above > may sound scarier than it is. > > * llvm backend - we need a maintainer for that As much I as I agree that deleting code is better than leaving it to rot beyond all recognition, I'm still not sure why we need a "maintainer" for this given that no other backend has an explicit maintainer. But anyways I will take that role if it needs be. Most work to be done is in factoring code out of genc and making rffi work (which I am guessing making changes to rffi, but not sure since haven't been following things for last however many months). > * pyrex backend (llvm depends on it tough) There is no longer this dependency. Cheers, Richard From fijall at gmail.com Thu Sep 27 09:14:10 2007 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 27 Sep 2007 15:14:10 +0800 Subject: [pypy-dev] PyPy cleanups In-Reply-To: References: <44A9D542-1F27-4E64-9EBA-123407BFB220@gmail.com> Message-ID: <693bc9ab0709270014k6e1b788ej6bf5fc7bc24db859@mail.gmail.com> Good to hear. There is no reason for maintainer, but we need llvm backend to be refactored so it would be easier to maintain for whoever makes changes. Also right now llvm version dependency is so bizzare that I cannot really test it on my machine without additional hassle. I can help with that, if possible. Cheers, fijal On 9/27/07, Richard Emslie wrote: > > Hi! > > Sorry it has taken forever to reply... > > > > > This is the list of possibly orphaned parts of pypy. > > We should consider each item here and think in detail > > what to do with them. They're mostly broken or not > > actively maintained. If nobody shows an interest > > to maintain them, deleting would be the best solution > > to avoid clutter. Also they pose a maintenance burden > > when we proceed with the needed large refactorings. > > Maybe it is also a solution to lazily delete them as > > they are broken by refactoring if nobody steps up. > > > > Of course deleted things can easily be brought back from > > svn history, if there is renewed interest. So the above > > may sound scarier than it is. > > > > * llvm backend - we need a maintainer for that > > As much I as I agree that deleting code is better than leaving it to > rot beyond all recognition, I'm still not sure why we need a > "maintainer" for this given that no other backend has an explicit > maintainer. But anyways I will take that role if it needs be. Most > work to be done is in factoring code out of genc and making rffi work > (which I am guessing making changes to rffi, but not sure since > haven't been following things for last however many months). > > > * pyrex backend (llvm depends on it tough) > > There is no longer this dependency. > > Cheers, > Richard > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20070927/fbb2f283/attachment.htm From fijall at gmail.com Thu Sep 27 09:15:41 2007 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 27 Sep 2007 15:15:41 +0800 Subject: [pypy-dev] New on list In-Reply-To: <348899050709260115p67e8d101v815d8892c6e649a6@mail.gmail.com> References: <46F9DB1F.8010901@berkeley.edu> <348899050709260115p67e8d101v815d8892c6e649a6@mail.gmail.com> Message-ID: <693bc9ab0709270015h4d63cdfepe27cec9fa1c7fa30@mail.gmail.com> Also I would be more than happy if we can at least share testing code :-) On 9/26/07, Carl Friedrich Bolz wrote: > > Hi Brandon! > > 2007/9/26, Brandon Lewis : > > Hi, my name is Brandon Lewis. I'm an undergrad Cognitive Science major > > at U.C. Berkeley. I'm currently taking the upper-division compilers > class. > > Welcome! > > > Our final project will be writing a JavaScript interpreter. > > Interestingly enough, we are using Python as the implementation > > language; so, when I discovered this project while looking through some > > python documentation, I was intrigued. > > I guess stealing the PyPy JavaScript interpreter andhanding it in is > not an option, right? :-) > > Cheers, > > Carl Friedrich > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20070927/78fce7d7/attachment.htm From cfbolz at gmx.de Thu Sep 27 10:41:18 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Thu, 27 Sep 2007 10:41:18 +0200 Subject: [pypy-dev] New on list In-Reply-To: References: <46F9DB1F.8010901@berkeley.edu> <9c0bb8a00709260459h162da238ga05f3f08ac4423e9@mail.gmail.com> Message-ID: <348899050709270141j21854c2bm640bc247b3ef6e18@mail.gmail.com> Hi Adam, 2007/9/26, Adam Bark : > Hi, I thought while people were introducing themselves I'd jump in. I'm a > BSc Physics student at the University of Leicester in England I program > almost exclusively for fun, I've been looking at getting into an established > project but it's not as easy as it sounds so any help with getting me into > pypy development would be cool. Finally a physicist, welcome! (I did physics for a while but abandoned it later in favour of computer science). Cheers, Carl Friedrich From brandon_lewis at berkeley.edu Thu Sep 27 19:06:22 2007 From: brandon_lewis at berkeley.edu (Brandon Lewis) Date: Thu, 27 Sep 2007 10:06:22 -0700 Subject: [pypy-dev] Replies to list? Message-ID: <46FBE30E.3060506@berkeley.edu> Just wondering about how this list handles replies. If I get a reply off-list should I reply to it on-list? --Brandon From brandon_lewis at berkeley.edu Thu Sep 27 19:09:02 2007 From: brandon_lewis at berkeley.edu (Brandon Lewis) Date: Thu, 27 Sep 2007 10:09:02 -0700 Subject: [pypy-dev] New on list In-Reply-To: References: <46F9DB1F.8010901@berkeley.edu> <9c0bb8a00709260459h162da238ga05f3f08ac4423e9@mail.gmail.com> Message-ID: <46FBE3AE.8030003@berkeley.edu> > > Hi, I thought while people were introducing themselves I'd jump in. I'm > a BSc Physics student at the University of Leicester in England I > program almost exclusively for fun, I've been looking at getting into an > established project but it's not as easy as it sounds so any help with > getting me into pypy development would be cool. > Thanks, sorry about the hijack Brandon :-p > > Adam Bark. > > No, worries. To be honest, a language interpreter/compiler is not the easiest thing to get started on. From brandon_lewis at berkeley.edu Thu Sep 27 19:13:34 2007 From: brandon_lewis at berkeley.edu (Brandon Lewis) Date: Thu, 27 Sep 2007 10:13:34 -0700 Subject: [pypy-dev] New on list In-Reply-To: References: <46F9DB1F.8010901@berkeley.edu> <9c0bb8a00709260459h162da238ga05f3f08ac4423e9@mail.gmail.com> Message-ID: <46FBE4BE.2030009@berkeley.edu> > Hi, I thought while people were introducing themselves I'd jump in. I'm > a BSc Physics student at the University of Leicester in England I > program almost exclusively for fun, I've been looking at getting into an > established project but it's not as easy as it sounds so any help with > getting me into pypy development would be cool. I was in england last summer for G.U.A.D.E.C.! Nice to meet you. To be honest, a language interpreter/compiler is not an easy project starter project. I remember sweating through the introductory scheme course at Berkeley, where we had to write a scheme interpreter in scheme for our final project. But if you can wrap your head around it, it will be worth the effort. > Thanks, sorry about the hijack Brandon :-p > > Adam Bark. > No worries =) --Brandon Lewis From paul.degrandis at gmail.com Thu Sep 27 20:22:15 2007 From: paul.degrandis at gmail.com (Paul deGrandis) Date: Thu, 27 Sep 2007 14:22:15 -0400 Subject: [pypy-dev] Replies to list? In-Reply-To: <46FBE30E.3060506@berkeley.edu> References: <46FBE30E.3060506@berkeley.edu> Message-ID: <9c0bb8a00709271122w418d8750u46a3b41bfc85b52@mail.gmail.com> My word is not law by anymeans but if you feel it's something the whole communiety should know or be able to go back and search on, reply on the list. I know sometimes I click "reply" instead of "reply to all" and it seems that in all of those cases, the reply back to me has been sent to the list. Paul On 9/27/07, Brandon Lewis wrote: > > Just wondering about how this list handles replies. If I get a reply > off-list should I reply to it on-list? > > --Brandon > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20070927/cc769e71/attachment.htm From cfbolz at gmx.de Thu Sep 27 20:35:29 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Thu, 27 Sep 2007 20:35:29 +0200 Subject: [pypy-dev] Replies to list? In-Reply-To: <9c0bb8a00709271122w418d8750u46a3b41bfc85b52@mail.gmail.com> References: <46FBE30E.3060506@berkeley.edu> <9c0bb8a00709271122w418d8750u46a3b41bfc85b52@mail.gmail.com> Message-ID: <348899050709271135q5652077cr5cad3c2cfc04ffd@mail.gmail.com> 2007/9/27, Paul deGrandis : > My word is not law by anymeans but if you feel it's something the whole > communiety should know or be able to go back and search on, reply on the > list. Sounds good. If you want to answer the whole list, choose "reply to all", if you want to answer a person personally use "reply". If somebody wrote you personally and you are sure it was by mistake you can CC the list again (don't forget to not snip the other persons post then when replying). Cheers, Carl Friedrich From overminddl1 at gmail.com Thu Sep 27 21:03:15 2007 From: overminddl1 at gmail.com (OvermindDL1) Date: Thu, 27 Sep 2007 13:03:15 -0600 Subject: [pypy-dev] Fwd: New on list In-Reply-To: <3f49a9f40709271202y6f71d336kbc8fdd76b391919e@mail.gmail.com> References: <46F9DB1F.8010901@berkeley.edu> <9c0bb8a00709260459h162da238ga05f3f08ac4423e9@mail.gmail.com> <3f49a9f40709271202y6f71d336kbc8fdd76b391919e@mail.gmail.com> Message-ID: <3f49a9f40709271203p5ec35444gaf3f301820bb66cf@mail.gmail.com> I'm from the USA as well, although I tend to be extremely quiet on here... I'm hoping for PyPy to be a faster Python mostly (would love the exporting to C++), especially with stackless functionality. I know the CPython internals quite well, so if there is anywhere I can use that knowledge in here, feel free to ask, short on time but of what little I have I can give up to this. EDIT: Helps if I send to the right list... Too many too similar lists... On 9/26/07, Paul deGrandis wrote: > > > On 9/26/07, Brandon Lewis wrote: > > Hi, my name is Brandon Lewis. I'm an undergrad Cognitive Science major > > at U.C. Berkeley. I'm currently taking the upper-division compilers class. > > > Finally I'm not the only person from the United > States! :) > Hi Brandon and welcome! I'm a BS/MS student at > Drexel University. My BS is in Software > Engineering. My MS is in Computer Science with my thesis on Dependable > Systems and Autonomic Computing. > > Our final project will be writing a JavaScript interpreter. > > Interestingly enough, we are using Python as the implementation > > language; so, when I discovered this project while looking through some > > python documentation, I was intrigued. > > It sounds like you have a great professor! Enjoy the class. > > Paul > > > Mostly just wanted to introduce myself. > > > > --Brandon Lewis > > _______________________________________________ > > pypy-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/pypy-dev > > > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From bokr at oz.net Thu Sep 27 22:18:45 2007 From: bokr at oz.net (Bengt Richter) Date: Thu, 27 Sep 2007 13:18:45 -0700 Subject: [pypy-dev] PyPy/Squeak Sprint Announcement In-Reply-To: <46F4C1DA.7080004@gmx.de> Message-ID: <5.0.2.1.1.20070927115740.00aef730@mail.oz.net> At 09:18 2007-09-22 +0200, Carl Friedrich Bolz wrote: >Bern Squeak/PyPy sprint October 22 - October 26 2007 >===================================================== > >We are proud to announce the first Squeak-PyPy-collaboration-sprint. >The plan is to bring people from the Squeak/Smalltalk and the >PyPy/Python communities together to learn about each others projects and >to think about collaboration possibilies. The sprint will be hosted by >the Software Composition Group of the University of Bern from the 22nd >to the 26th of October 2007. > > >Topics and goals >---------------- > >The general technical goal of the sprint is to explore implementing the >bytecode interpreter as well as the primitives of Squeak in RPython. >This should make it eventually possible to auto-generate a JIT for >Squeak with the help of PyPy technologies. > >The other important goal of the sprint is to teach the >first-time-sprinters the sprint-driven-development methodology. > >The first day of the sprint will be a tutorial day, were talks about >PyPy, Squeak and sprint-driven-development will be given. The further >days will be sprinting days. > >Of course the most important goal of the sprint is to get to know each >other and have some fun :-). > >Topics of the sprint: > >- learning about: > > - PyPy > - the internals of Squeaks VM > - sprint-driven-development > >- exploring ways to implement Squeaks bytecode-interpreter and the > primitives in Python > >- or (maybe alternatively) implement a Squeak-backend for PyPy > >- think about how to load and store of images in PyPy > > >Location >-------- > >The sprint will take place in a seminar room at the University of Bern. >Further details of how to get there will be announced before the sprint. > > >Registration >------------ > >If you'd like to come, please subscribe to the `pypy-sprint mailing list`_ >and drop a note about your interests and post any questions. More >organisational information will be send to that list. We'll keep a list >of `people`_ which we'll update (which you can do so yourself if you >have codespeak commit rights). If you have no clue where to stay in >Bern ask around on the list, maybe we can help you find something or >find some other solution. > >.. _`pypy-sprint mailing list`: >http://codespeak.net/mailman/listinfo/pypy-sprint >.. _`people`: >http://codespeak.net/pypy/extradoc/sprintinfo/bern2007/people.html >_______________________________________________ >pypy-dev at codespeak.net >http://codespeak.net/mailman/listinfo/pypy-dev Hello. This sounds interesting! If a video or sound track plus slides wound up on video.google.com or youtube.com I would certainly watch it. Hint, hint ;-) Perhaps others would agree, and have the influence and/or ability/interest/energy to make it happen? (Perhaps the University of Bern has an audio/visual dept with someone who would be willing to come in and set up good sound recording etc for the event? Or maybe someone is in a music group that has a good sound-recordist? I emphasize sound because I think good/bad sound makes/breaks a video more than visual quality, especially when the speaker gets past reading bullet points (also helpful) and is interacting and explaining and musing.) Regards, Bengt Richter Still lurking now and then ;-) From cfbolz at gmx.de Sat Sep 29 15:50:33 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Sat, 29 Sep 2007 15:50:33 +0200 Subject: [pypy-dev] Gothenburg Sprint Announcement Message-ID: <348899050709290650qba97961k91b6c073a7e0ea34@mail.gmail.com> PyPy G?teborg Cleanup Sprint November 19 - November 25 2007 =========================================================== The next post-EU-project PyPy sprint will be in Gothenburg, Sweden. It will focus on cleaning up the PyPy codebase and making it ready for the next round of improvements. It is a "public" sprint but it will probably be more suitable for people already somewhat acquainted with PyPy. Topics and goals ---------------- Meta-Tasks ++++++++++ - Tracker-gardening - Start slowly thinking about 1.1 plans Translation toolchain +++++++++++++++++++++ - finish rctypes removal - move away from implicit keepalives - raw_malloc should be split, rffi one should not use obmalloc (it's not thread-safe) - kill simulator, run the semi space tests on llinterp (may happen before) - have a moving gc correct version of the gc framework transformation - delegate finding type stuff like vtables etc to GC, cleaner interface for rtti, simplify translator/c/gc.py - think about approaches to id, especially concerning boehm, where the id will keep the object alive and concerning a moving GC - clean up the tangle of including headers in the C backend - review pdbplus, especially the graph commands, also in the light of https://codespeak.net/issue/pypy-dev/issue303 and the fact that we can have more than one translator/annotator around (with the timeshifter) - Kill half concrete wrapper - Kill opaque hacks on the C backend - decide how to implement constants in rffi - think about/remove orphaned parts - finish ll2ctypes, especially rffi_platform Interpreter +++++++++++ - there's a shadow tracking bug it seems - fix the taint space - review the things implemented at applevel whether they are performance-critical JIT +++ - start writing real unit tests for the merging logic Location -------- The sprint will be held in the apartment of Laura Creighton and Jacob Hall?n which is at G?tabergsgatan 22 in Gothenburg, Sweden. Here is a map_. This is in central Gothenburg. It is between the tram_ stops of Vasaplatsen and Valand, where many lines call. .. _tram: http://www.vasttrafik.se .. _map: http://gulasidorna.eniro.se/query?stq=0&streetname=G%F6tabergsgatan+22%2C+G%F6teborg&what=map&asearch=1 Probably cheapest and not too far away is to book accomodation at `SGS Veckobostader`_. There are also hotels to suit every price range and desire for luxury. Just ask in the mailing list. .. _`SGS Veckobostader`: http://www.sgsveckobostader.com Registration ------------ If you'd like to come, please subscribe to the `pypy-sprint mailing list`_ and drop a note about your interests and post any questions. More organisational information will be send to that list. We'll keep a list of `people`_ which we'll update (which you can do so yourself if you have codespeak commit rights). .. _`pypy-sprint mailing list`: http://codespeak.net/mailman/listinfo/pypy-sprint .. _`people`: http://codespeak.net/pypy/extradoc/sprintinfo/gothenburg-2007/people.html