From anto.cuni at gmail.com Tue Jul 3 15:56:51 2007 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 03 Jul 2007 15:56:51 +0200 Subject: [pypy-dev] Alternative algorithm for attribute lookup Message-ID: <468A55A3.4080009@gmail.com> Hi all, I passed the last few days slowly experimenting with alternative/faster ways to do lookup of attributes. For simplicity, I didn't touch the interpreter but I simply wrote few rpython programs doing only the last part of the attribute lookup, i.e. accessing the underlying RPython dictionary. Moreover, I played only with dict of strings, since this is how objects are implemented when multidict is enabled. My first experiment was with precomputed hashes for keys; I introduced a new method get_fast on dictionaries, which accepts both the key and its hash value, so it does not need to compute it. Of course this is just a quick hack because programs using get_fast works only when translated and can't run on top of CPython, but it was the simplest way to experiment. You can find both the get_fast patch and the benchmark here: http://codespeak.net/svn/user/antocuni/lookup-hack/get_fast.patch http://codespeak.net/svn/user/antocuni/lookup-hack/targethashcache.py The interesting thing is the result of the benchmark: since RPython strings already cache their hash I didn't expect get_fast to be much faster than get, but the benchmark says the opposite; on my machine: antocuni at anto lookup-hack $ ./targethashcache-c 50000000 iterations get: 3.820000 seconds get_fast: 2.520000 seconds get_fast: 1.515873 times faster It is 50% faster. I really don't know why, maybe there is something wrong in the benchmark or maybe something is wrong with the hash cache of rpython strings. The second and more insteresting experiment is a bit more involved. The idea is that for most of Python objects, we can guess a set of "likely attributes" at compile time: - for modules, by inspecting its global namespace; - for classes, by inspecting the namespace of a class definition; - for instances, by looking for LOAD_FAST 0, STORE_ATTR x inside the methods of the class. Note that we don't pretend to catch all the possible attributes, just the most likely. The assumption is that it's very uncommon to have a "non-likely attribute" and that for most of the objects the set of the attributes at runtime is a subset of the likely attributes. The idea is to speed up consistently all the accesses to likely attributes, at the cost of a slighly slow down for accessing the non-likely ones. Once we have collected the set of likely attributes, we compute a perfect hash function for this set; for my experiment, I used an hacked version of this algorithm: http://www.amk.ca/files/python/perfect_hash.txt The perfect hash function looks like this, where N and G vary from set to set of likely attributes:: def perfect_hash(N, G, key): h = hash(key) # normal hash i = h & N-1 j = i+1 res = G[i] + G[j] res = res & N-1 return res Moreover, since h depends only on key we could also precomputed it at compile time, as we did for get_fast:: def perfect_hash(N, G, key, h): ... Finally, we insert a fast path for the common case in the lookup code:: def lookup(d, key, h): # h is the precomputed hash index = perfect_hash(d.N, d.G, key, h) entry = d.entries[index] if entry.is_valid() and entry.key is key: return entry.value else: # do a full lookup Note that in the fast path we use 'is' instead of == to compare the keys; since the W_Strings storing the names of the attributes are interned, this should be enough. Note also that G and N are stored on the dict (thus we can have different perfect hashes for different set of likely attributes, as we need). If my assumption about likely attributes is true, most of LOAD_ATTR will follow the fast path, resulting in a considerable speedup: if the object only uses likely attributes, there will never be a collision. Most important, this does not break anything, because in case the fast path is not followed (for example when we do {get,set}attr(obj, 'attr')), the full lookup will work as well. You can find the benchmark here (this doesn't need get_fast.patch): http://codespeak.net/svn/user/antocuni/lookup-hack/targetperfectdict.py For the test, I implemented a perfect_dict as an RPython class, thus it might be slower than what we can get if we implement it as a real rpython type on the style of r_dict; in particular, RPython does index-checking when calcultaing G[i] and G[j], but it's not necessary. Although perfect_dict is not as fast as possible, the result of the benchmark is very interesting: antocuni at anto lookup-hack $ ./targetperfectdict-c 50000000 iterations get: 3.740000 seconds pdict: 1.450000 seconds pdict: 2.579310 times faster What do you think about this idea? Is there any obvious bug/wrong assumption I can't catch? Do you think it might be worth of implementing it? (it would be a nice sprint topic). ciao Anto From scott+pypy-dev at scottdial.com Wed Jul 4 22:02:21 2007 From: scott+pypy-dev at scottdial.com (Scott Dial) Date: Wed, 04 Jul 2007 16:02:21 -0400 Subject: [pypy-dev] select/test_readable on Win32 blocks forever Message-ID: <468BFCCD.6040201@scottdial.com> Greetings, I haven't figured this one out. As the subject says, test_readable for the select module gets stuck forever. If I trust the history of the bot, then it appeared somewhere after r44639 and before r44646. I've looked through the revisions and nothing jumped out at me.. happy hunting to someone more aware of the changes. -Scott -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From wchunming at gmail.com Thu Jul 5 07:47:25 2007 From: wchunming at gmail.com (=?GB2312?B?zfW0usP3?=) Date: Thu, 5 Jul 2007 13:47:25 +0800 Subject: [pypy-dev] debug(read source code and test) pypy in pydev Message-ID: <64e4dfb0707042247n7818ecdfvd051a827abc29e3@mail.gmail.com> Hi, I am a newbie to python and pypy. I choose to read pypy source code to learn about python and python library. But when I try to run pypy in debug mode in pydev, I got the following problem: /// error message begin pydev debugger Traceback (most recent call last): File "F:\TOOLS\eclipse\plugins\org.python.pydev.debug_1.3.4\pysrc \pydevd.py", line 754, in debugger.run(setup['file'], None, None) File "F:\TOOLS\eclipse\plugins\org.python.pydev.debug_1.3.4\pysrc \pydevd.py", line 597, in run execfile(file, globals, locals) #execute the script File "E:\Java\pySamples\pypy\bin\py.py", line 14, in from pypy.tool import option File "E:\Java\pySamples\pypy\tool\option.py", line 4, in from pypy.config.pypyoption import get_pypy_config File "E:\Java\pySamples\pypy\config\pypyoption.py", line 2, in import py, os File "E:\Java\pySamples\pypy\bin\py.py", line 14, in from pypy.tool import option ImportError: cannot import name option Exception in thread pydevd.Writer (most likely raised during interpreter shutdown): Traceback (most recent call last): File "C:\Python25\lib\threading.py", line 460, in __bootstrap File "F:\TOOLS\eclipse\plugins\org.python.pydev.debug_1.3.4\pysrc \pydevd_comm.py", line 258, in run /// error message end I try to figure out why this exception or error come out. I found that py.py import option.py, and option.py import pypyoption.py, and pypyoption.py import py.py again. Is this the problem of pypy(I don't think so), or the problem of pydev debugger, or something else? why? Any comments or assistance that can light up the way are much appreciated.Thanks. Wang Chunming -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20070705/7ce79964/attachment.htm From amauryfa at gmail.com Thu Jul 5 14:41:16 2007 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Thu, 5 Jul 2007 14:41:16 +0200 Subject: [pypy-dev] debug(read source code and test) pypy in pydev In-Reply-To: References: <64e4dfb0707042247n7818ecdfvd051a827abc29e3@mail.gmail.com> Message-ID: Hello, Wang Chunming wrote: > Hi, > I am a newbie to python and pypy. I choose to read pypy source code > to learn about python and python library. But when I try to run pypy > in debug mode in pydev, I got the following problem: [Traceback] > > I try to figure out why this exception or error come out. I found > that py.py import option.py, and option.py import pypyoption.py, and > pypyoption.py import py.py again. Is this the problem of pypy(I don't > think so), or the problem of pydev debugger, or something else? why? > Any comments or assistance that can light up the way are much > appreciated.Thanks. There are two "py" involved here: - you are running the py.py script - the other is the py library: http://codespeak.net/py/dist/ Normally, this does not cause any problem, but Pydev seems to tweak the pythonpath and adds several entries... Try to set the current directory to E:\Java\pySamples\pypy\bin\ (In eclipse, open the "Run/Debug..." window, and select the "Arguments tab") Or you could just rename py.py to something else (runpy.py for example). Note that because of a python bug, you cannot use python 2.5 to debug pypy. (A strange interaction between generator destructors, the sys.settrace function and the current thread state. www.python.org/sf/1733973 shows a simple case, filed by the pydev developer) Python 2.4 seems to work correctly. Hope this helps, -- Amaury Forgeot d'Arc From amauryfa at gmail.com Thu Jul 5 15:28:21 2007 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Thu, 5 Jul 2007 15:28:21 +0200 Subject: [pypy-dev] select/test_readable on Win32 blocks forever In-Reply-To: <468BFCCD.6040201@scottdial.com> References: <468BFCCD.6040201@scottdial.com> Message-ID: Hello Scott, Scott Dial wrote: > Greetings, > > I haven't figured this one out. As the subject says, test_readable for > the select module gets stuck forever. If I trust the history of the bot, > then it appeared somewhere after r44639 and before r44646. I've looked > through the revisions and nothing jumped out at me.. happy hunting to > someone more aware of the changes. When I wrote these tests a few months ago, they passed on the two windows machines where I ran them (Win2k + python2.5 behind a firewall, and XP + python2.4 at home). They still pass without problem (except for some network operations forbidden by the firewall). Is it possible that the bot runs in a restricted environment? What is the configuration of this machine? I suggest to run the following test function, either alone, or somewhere in the test suite Could you put it near the top of pypy/module/select/test/test_select.py ? It has nothing to do with pypy, but repeats the same test at the python level. # Sanity check of the standard Python interpreter def test_CPython_select(): import thread, socket, select sock = socket.socket() try_ports = [1023] + range(20000, 30000, 437) for port in try_ports: print 'binding to port %d:' % (port,), sockaddress = ('127.0.0.1', port) try: sock.bind(sockaddress) print 'works' break except socket.error, e: print e else: raise e else: assert False, "Could not create server" sock.listen(1) writeend = socket.socket() thread.start_new_thread(writeend.connect, (sockaddress,)) readend, addr2 = sock.accept() try: iwtd, owtd, ewtd = select.select([readend], [], [], 0) assert iwtd == owtd == ewtd == [] writeend.send('X') iwtd, owtd, ewtd = select.select([readend], [], []) assert iwtd == [readend] assert owtd == ewtd == [] print "Test OK" finally: writeend.close() readend.close() -- Amaury Forgeot d'Arc From jgustak at gmail.com Fri Jul 6 08:55:50 2007 From: jgustak at gmail.com (Jakub Gustak) Date: Fri, 6 Jul 2007 08:55:50 +0200 Subject: [pypy-dev] scheme interpreter [status report] In-Reply-To: References: Message-ID: Hello. What we have done on scheme interpreter: Working lambdas, with closures almost as described in r5rs. What we miss here is proper tail-recursion and issues, when lambda is called with wrong number of arguments (though should be simple to add). Some rethinking on ExecutionContext. Now we have global and local scope (looks like python). When copying ExecutionContext only local scope is copied, global one is shared. What more, copied ExecutionContext cannot bound new variables in global scope, though it can set new values to existing ones. Done quotations both as (quote ) and as '. After all it looks like distinction between W_Identifier and W_Symbol is useless. So probably only one of them will live in the future. Also implemented parsing for dotted lists (needed for lambdas definition). cons, car, cdr were easy ones. Probably not much hacking before the sprint. So the plan for EP sprint is RPython. I would like also to discuss some other features, if time will let us. See you soon on #pypy or on EP. Cheers, Jakub From cfbolz at gmx.de Mon Jul 9 19:33:30 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 9 Jul 2007 19:33:30 +0200 Subject: [pypy-dev] paper on runtime specialization Message-ID: <348899050707091033o67c92e9by818a25f6877cae2@mail.gmail.com> Hi all! Today I found this rather interesting paper on runtime specialization of C: http://www.cs.washington.edu/research/dyncomp/Papers/tr-97-03-03-abstract.html I have only skimmed it so far, but it seems there are many shared ideas between PyPy and this approach. Cheers, Carl Friedrich From cfbolz at gmx.de Mon Jul 9 20:58:33 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 9 Jul 2007 20:58:33 +0200 Subject: [pypy-dev] paper on runtime specialization In-Reply-To: <348899050707091033o67c92e9by818a25f6877cae2@mail.gmail.com> References: <348899050707091033o67c92e9by818a25f6877cae2@mail.gmail.com> Message-ID: <348899050707091158x1f36d4fdxacc990f1e9d46ca7@mail.gmail.com> 2007/7/9, Carl Friedrich Bolz : > Today I found this rather interesting paper on runtime specialization of C: There is even more interesting stuff (including papers) on their web-page: http://www.cs.washington.edu/research/dyncomp/ Cheers, Carl Friedrich From fijal at genesilico.pl Wed Jul 11 13:13:40 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Wed, 11 Jul 2007 13:13:40 +0200 Subject: [pypy-dev] Quick personal view regarding js interpreter Message-ID: <4694BB64.5050700@genesilico.pl> It's completely unusable. Usually you end up with Fatal RPythonError in case you try to run stuff. This is not how we go about stuff! All error reporting should work well not only on top of js_interactive, but actually on top of js-c as well. Leonardo - have you even run js_interactive once before you commit? Last time it was unbound local or so. Cheers, fijal :. From scott+pypy-dev at scottdial.com Wed Jul 11 16:56:20 2007 From: scott+pypy-dev at scottdial.com (Scott Dial) Date: Wed, 11 Jul 2007 10:56:20 -0400 Subject: [pypy-dev] Win32 tester Message-ID: <4694EF94.2010908@scottdial.com> Greetings Everyone, I am going to have to stop running the win32 test suite. I am not sure there is anyone actively correcting problems on win32, so it may be of no consequence. If someone else is interested in running this, then I would be more than glad to share my knowledge with them. Otherwise, sorry to leave you without. -Scott -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From bea at changemaker.nu Thu Jul 12 15:29:20 2007 From: bea at changemaker.nu (=?ISO-8859-1?Q?Beatrice_D=FCring?=) Date: Thu, 12 Jul 2007 15:29:20 +0200 Subject: [pypy-dev] Win32 tester In-Reply-To: <4694EF94.2010908@scottdial.com> References: <4694EF94.2010908@scottdial.com> Message-ID: <46962CB0.8030409@changemaker.nu> Hi there Scott Dial skrev: > Greetings Everyone, > > I am going to have to stop running the win32 test suite. I am not sure > there is anyone actively correcting problems on win32, so it may be of > no consequence. If someone else is interested in running this, then I > would be more than glad to share my knowledge with them. Otherwise, > sorry to leave you without. > > -Scott > > Thanks for helping out with this Scott! And the door is always open - as you know ;-) Cheers Bea From fijal at genesilico.pl Fri Jul 13 13:14:00 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Fri, 13 Jul 2007 13:14:00 +0200 Subject: [pypy-dev] Win32 tester In-Reply-To: <4694EF94.2010908@scottdial.com> References: <4694EF94.2010908@scottdial.com> Message-ID: <46975E78.7030007@genesilico.pl> Scott Dial wrote: > Greetings Everyone, > > I am going to have to stop running the win32 test suite. I am not sure > there is anyone actively correcting problems on win32, so it may be of > no consequence. If someone else is interested in running this, then I > would be more than glad to share my knowledge with them. Otherwise, > sorry to leave you without. > > -Scott > > I would blame a bit (from my side) lack of a single web page showing all failures on all platforms. Unless we have one, it's probably a bit useless to have such a status page, just because. I would like to set up one at some point, but that's another issue. Thanks for cooperating and I (personally) used it from time to time (although never said a word). Cheers, fijal :. From gbowyer at fastmail.co.uk Fri Jul 13 15:37:57 2007 From: gbowyer at fastmail.co.uk (Greg Bowyer) Date: Fri, 13 Jul 2007 14:37:57 +0100 Subject: [pypy-dev] Taking pypy for a test drive Message-ID: Sorry if this is an exceptional newbie post ... Now that pypy has hit v1 I have tried using it for a few test examples, most of this stuff so far is a little bit simple, however one test is to run a django application that I am developing for someone with pypy rather that cpython However when I fire up pypy i get the following from the django application, what am I missing here greg at gregslaptop ~/projects/minesite/minesite $ ./pypy-c manage.py runserver debug: WARNING: library path not found, using compiled-in sys.path Traceback (most recent call last): File "?", line 32, in run_toplevel File "manage.py", line 11, in execute_manager(settings) File "/usr/lib/python2.4/site-packages/django/core/management.py", line 1736, in execute_manager execute_from_command_line(action_mapping, argv) File "/usr/lib/python2.4/site-packages/django/core/management.py", line 1626, in execute_from_command_line translation.activate('en-us') File "/usr/lib/python2.4/site-packages/django/utils/translation/__init__.py", line 66, in activate return real_activate(language) File "/usr/lib/python2.4/site-packages/django/utils/translation/__init__.py", line 29, in delayed_loader from django.conf import settings File "/usr/lib/python2.4/site-packages/django/conf/__init__.py", line 149, in __builtins__['_'] = first_time_gettext TypeError: cannot set items on object From exarkun at divmod.com Fri Jul 13 15:50:59 2007 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Fri, 13 Jul 2007 09:50:59 -0400 Subject: [pypy-dev] Taking pypy for a test drive In-Reply-To: Message-ID: <20070713135059.4947.464118518.divmod.quotient.11057@ohm> On Fri, 13 Jul 2007 14:37:57 +0100, Greg Bowyer wrote: >Sorry if this is an exceptional newbie post ... > >Now that pypy has hit v1 I have tried using it for a few test examples, >most of this stuff so far is a little bit simple, however one test is to >run a django application that I am developing for someone with pypy >rather that cpython > >However when I fire up pypy i get the following from the django >application, what am I missing here > >greg at gregslaptop ~/projects/minesite/minesite $ ./pypy-c manage.py runserver >debug: WARNING: library path not found, using compiled-in sys.path >Traceback (most recent call last): > File "?", line 32, in run_toplevel > File "manage.py", line 11, in > execute_manager(settings) > File "/usr/lib/python2.4/site-packages/django/core/management.py", >line 1736, in execute_manager > execute_from_command_line(action_mapping, argv) > File "/usr/lib/python2.4/site-packages/django/core/management.py", >line 1626, in execute_from_command_line > translation.activate('en-us') > File >"/usr/lib/python2.4/site-packages/django/utils/translation/__init__.py", >line 66, in activate > return real_activate(language) > File >"/usr/lib/python2.4/site-packages/django/utils/translation/__init__.py", >line 29, in delayed_loader > from django.conf import settings > File "/usr/lib/python2.4/site-packages/django/conf/__init__.py", line >149, in > __builtins__['_'] = first_time_gettext >TypeError: cannot set items on object It seems likely that what django ought to be doing is: import __builtin__ __builtin__._ = first_time_gettext And a similar change in first_time_gettext. __builtin__/__builtins__ is a bit subtle and I might be forgetting something, but I think the above is correct. I remember that __builtins__ is somethings a dict instead of the __builtin__ module, but I forget when (and I couldn't find the case with a few simple tests). I think python-dev has pointed out that this is an implementation detail of CPython on several occassions, though, and that explicitly importing the __builtin__ module is the right thing to do. Jean-Paul From njriley at uiuc.edu Fri Jul 13 16:07:10 2007 From: njriley at uiuc.edu (Nicholas Riley) Date: Fri, 13 Jul 2007 09:07:10 -0500 Subject: [pypy-dev] Taking pypy for a test drive In-Reply-To: <20070713135059.4947.464118518.divmod.quotient.11057@ohm> References: <20070713135059.4947.464118518.divmod.quotient.11057@ohm> Message-ID: <20070713140710.GA14844@uiuc.edu> On Fri, Jul 13, 2007 at 09:50:59AM -0400, Jean-Paul Calderone wrote: > It seems likely that what django ought to be doing is: > > import __builtin__ > __builtin__._ = first_time_gettext > > And a similar change in first_time_gettext. __builtin__/__builtins__ is > a bit subtle and I might be forgetting something, but I think the above > is correct. I remember that __builtins__ is somethings a dict instead of > the __builtin__ module, but I forget when (and I couldn't find the case > with a few simple tests). I think python-dev has pointed out that this > is an implementation detail of CPython on several occassions, though, and > that explicitly importing the __builtin__ module is the right thing to > do. It's documented in the Python reference manual. -- Nicholas Riley | From simon at arrowtheory.com Tue Jul 17 03:22:03 2007 From: simon at arrowtheory.com (Simon Burton) Date: Mon, 16 Jul 2007 18:22:03 -0700 Subject: [pypy-dev] revision 45089 Message-ID: <20070716182203.0d1953e1.simon@arrowtheory.com> I found that revision 45089 makes compilation much slower. No idea why. Simon. ------------------------------------------------------------------------ r45089 | justas | 2007-07-14 08:35:59 -0700 (Sat, 14 Jul 2007) | 8 lines (Arlo, Justas) Introduce --pyversion option to select the parser/compiler grammar Defaults to 2.4, also accepts 2.3 and 2.5a. Changed a bunch of imports to use the selected grammar. TODO: pypy/interpreter/pyparser/symbol.py uses a fixed list of symbols, and it shouldn't From arigo at tunes.org Tue Jul 17 11:57:09 2007 From: arigo at tunes.org (Armin Rigo) Date: Tue, 17 Jul 2007 11:57:09 +0200 Subject: [pypy-dev] revision 45089 In-Reply-To: <20070716182203.0d1953e1.simon@arrowtheory.com> References: <20070716182203.0d1953e1.simon@arrowtheory.com> Message-ID: <20070717095709.GA11010@code0.codespeak.net> Hi Simon, On Mon, Jul 16, 2007 at 06:22:03PM -0700, Simon Burton wrote: > I found that revision 45089 makes compilation much slower. No idea why. Thanks for the note. I fixed it in r45153 after a cProfile run of some tests - the problem was obscure: the flow object space was taking ages (about 0.2 seconds for simple functions). This is because the parser and compiler instantiation code, which is not needed at all for the flow space, was actually called many times per function. And r45089 made that code take much longer. It's not a problem in general because it's not supposed to happen frequently - indeed, it does not need to be called at all when we use the flow space. A bientot, Armin. From cfbolz at gmx.de Tue Jul 17 14:26:08 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Tue, 17 Jul 2007 14:26:08 +0200 Subject: [pypy-dev] sprint report Message-ID: <469CB560.9030108@gmx.de> Hi all! Since I didn't manage to come to Europython and the sprint afterwards, I would really appreciate it if somebody wrote a sprint report. Since I guess that is kind of unlikely to happen now, could at least everybody write a paragraph about what he worked on? Cheers, Carl Friedrich From micahel at gmail.com Tue Jul 17 15:20:36 2007 From: micahel at gmail.com (Michael Hudson) Date: Tue, 17 Jul 2007 14:20:36 +0100 Subject: [pypy-dev] sprint report In-Reply-To: <469CB560.9030108@gmx.de> References: <469CB560.9030108@gmx.de> Message-ID: On 17/07/07, Carl Friedrich Bolz wrote: > Hi all! > > Since I didn't manage to come to Europython and the sprint afterwards, I > would really appreciate it if somebody wrote a sprint report. Since I > guess that is kind of unlikely to happen now, could at least everybody > write a paragraph about what he worked on? Samuele and I hacked on the twisted mess of memory management implementations. Basically we managed to much simplify the interface the low-level backends need to provide -- there is no longer any need to support any 'malloc' operation that deals with types. There is still confusion between various malloc flavours and operations, but it's getting better. Backends that support Boehm need to implement three new operations: boehm_malloc, boehm_malloc_atomic and boehm_register_finalizer. These could/should one day be removed and implemented with rffi (they're just function calls to named function^Wmacros, after all), but that seemed like too much wizardry for the time we had. Cheers, mwh From simon at arrowtheory.com Tue Jul 17 17:02:53 2007 From: simon at arrowtheory.com (Simon Burton) Date: Tue, 17 Jul 2007 08:02:53 -0700 Subject: [pypy-dev] expected_extra_mallocs Message-ID: <20070717080253.a1387d0e.simon@arrowtheory.com> Here is a little test: def func(): x = 1. print x def test_1(): from pypy.translator.c.test.test_genc import compile _func = compile(func, []) _func() I get this error: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def checking_fn(*args, **kwds): if 'expected_extra_mallocs' in kwds: expected_extra_mallocs = kwds.pop('expected_extra_mallocs') else: expected_extra_mallocs = 0 res = compiled_fn(*args, **kwds) mallocs, frees = module.malloc_counters() E assert mallocs - frees == expected_extra_mallocs > assert (6 - 5) == 0 [/home/simon/local/pypy-latest/pypy/translator/c/test/test_genc.py:54] So, do I call this with expected_extra_mallocs=1 ? In general, how many extra mallocs should I expect ? Simon. From jacob at openend.se Tue Jul 17 19:33:32 2007 From: jacob at openend.se (Jacob =?utf-8?q?Hall=C3=A9n?=) Date: Tue, 17 Jul 2007 19:33:32 +0200 Subject: [pypy-dev] More sprint reporting Message-ID: <200707171933.32223.jacob@openend.se> Here is what I know happened in other groups: Armin and Laura worked on rffi. After 2 days of trying different approaches and getting stumped each time, they made a breakthrough on Saturday, and managed to produce a fairly complete slution in a matter of a few hours. Making the solution more complete would require a rewrite of ctypes (Ick!). Maciek worked on getting PyPy to be self hosting under pypy-c. This succeeded before he left the sprint. Jakub worked on more scheme features. With the help of Antonio, he managed to get quite a bit done. Exactly what he did, I don't know. Christian worked on fixing pickling of tasklets. He got as far as understanding what the real problem was before I left. Michael and Samuele worked on malloc cleanup. I think they were fairly satisfied with the result. Samuele found and fixed a very nasty bug somewhere in the translator toolchain. Exception transformation was dependent on the order the graphs were evaluated. The result was that exceptions could be lost without a trace. There were 3 people who's names I have forgotten who worked on better documentation on how to leverage the javascript backend. There were some results checked in, but I don't know how complete it is. I think Holger did some work on improving py.test. That was at least his plan, but I never got a chance to ask what he actually did. I think that is about it. Jacob From jacob at openend.se Tue Jul 17 19:38:39 2007 From: jacob at openend.se (Jacob =?utf-8?q?Hall=C3=A9n?=) Date: Tue, 17 Jul 2007 19:38:39 +0200 Subject: [pypy-dev] Sprint report Message-ID: <200707171938.39921.jacob@openend.se> I think this got stuck in moderation, as I sent it with a from address that didn't match my mailinglist subscription. Justas, Arlo Belshee and I worked on Python 2.5 compliance. We started with the "with" statement and fairly quickly found that it seemed to work as intended, except for the fact that the necessary "from __future__ import with_statement" had no effect. To make this work properly, we decided to make a clean implementation - CPython does nasty things in order to avoid modifying the AST on the fly. This required modifying the grammar, which in turn led to the discovery that building a parser from the grammar lacked unit tests. We then created the neccessary test setup and made tests for the modifications we were making for the grammar. Then we went on to being able to insert a rule for the "with" statement into the builder on the fly. This required more instrumentation for unit testing and some modifications to the builder. Finally, added a config option to allow selecting different versions from the command line. This was essentially finished, though there may still be places which have the Pthon version hard coded. Jacob From jgustak at gmail.com Tue Jul 17 21:44:24 2007 From: jgustak at gmail.com (Jakub Gustak) Date: Tue, 17 Jul 2007 21:44:24 +0200 Subject: [pypy-dev] More sprint reporting In-Reply-To: <200707171933.32223.jacob@openend.se> References: <200707171933.32223.jacob@openend.se> Message-ID: Here is what happened with scheme interpreter: Antonio and I were working on scheme interpreter to translate. - (antocuni, jlg) The biggest issue was with number handling, it resulted with numbers hierarchy rethinking and arithmetical operations refactoring most of them are dynamically generated. - (fijal, jlg) Some effort was put into syntax checking, especially in macros where lot of w_pair.car/cdr had to be checked if are still instance of W_Pair. - (antocuni, jlg) Then there was work on delayed evaluation. During this Armin found bug in lambda definition identical to one I made with W_Promise. - (arigo, jlg) Finally we made proper tail-recursion work for calls in tail context. - (arigo, jlg) Thats it! btw. I made some dummy benchmark on tail-recursive loop and some arithmetical operations. Our ss-c is about 5 times slower than guile. Cheers, Jakub From simon at arrowtheory.com Tue Jul 17 22:42:38 2007 From: simon at arrowtheory.com (Simon Burton) Date: Tue, 17 Jul 2007 13:42:38 -0700 Subject: [pypy-dev] sprint report In-Reply-To: <469CB560.9030108@gmx.de> References: <469CB560.9030108@gmx.de> Message-ID: <20070717134238.a1c0d9ab.simon@arrowtheory.com> I worked with Maciek on generating rffi wrappers from ctypes objects. The first method we tried used code generation. Later on I got it to generate live rffi objects from the ctypes objects, without code generation. I also made some progress on generating rffi wrappers for cairo and SDL, and also testing these on ll2ctypes. Simon. From fijal at genesilico.pl Wed Jul 18 11:54:49 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Wed, 18 Jul 2007 11:54:49 +0200 Subject: [pypy-dev] Sprint report In-Reply-To: <200707171938.39921.jacob@openend.se> References: <200707171938.39921.jacob@openend.se> Message-ID: <469DE369.7010905@genesilico.pl> Three guys who's names Jacob doesn't remember were working on flex backend (reusing JavaScript one to be able to produce flex code and some libraries). I've got no idea what was their progress (hopefully they're reading pypy-dev and are able to reply :-) Cheers, fijal :. From lac at openend.se Wed Jul 18 12:05:03 2007 From: lac at openend.se (Laura Creighton) Date: Wed, 18 Jul 2007 12:05:03 +0200 Subject: [pypy-dev] Sprint report In-Reply-To: Message from Maciek Fijalkowski of "Wed, 18 Jul 2007 11:54:49 +0200." <469DE369.7010905@genesilico.pl> References: <200707171938.39921.jacob@openend.se> <469DE369.7010905@genesilico.pl> Message-ID: <200707181005.l6IA53ih016094@theraft.openend.se> In a message of Wed, 18 Jul 2007 11:54:49 +0200, Maciek Fijalkowski writes: >Three guys who's names Jacob doesn't remember were working on flex >backend (reusing JavaScript one to be able to produce flex code and some >libraries). I've got no idea what was their progress (hopefully they're >reading pypy-dev and are able to reply :-) > >Cheers, >fijal I think they were Ren? Dudfield, Alejandro Curia and Lucio Torre, from Australia, Argentina and Argentina, who may not be home yet. Laura From anto.cuni at gmail.com Wed Jul 18 18:13:20 2007 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed, 18 Jul 2007 18:13:20 +0200 Subject: [pypy-dev] sprint report In-Reply-To: <469CB560.9030108@gmx.de> References: <469CB560.9030108@gmx.de> Message-ID: <469E3C20.7010803@gmail.com> Carl Friedrich Bolz wrote: > Hi all! > > Since I didn't manage to come to Europython and the sprint afterwards, I > would really appreciate it if somebody wrote a sprint report. Since I > guess that is kind of unlikely to happen now, could at least everybody > write a paragraph about what he worked on? On the first day I worked with Jakub to make the scheme interpreter translatable. On the second day, I paired with Maciek trying to make pypy-c self-hosted: we fixed few bugs and now it is self-hosted, as long as pypy-c is being translated with the same opcodes as the hosting pypy-c. Finally, I spent the third day by working again with Jakub on the scheme interpreter and by experimenting with method lookup in the interpreter, without concluding anything interesting :-). ciao Anto From simon at arrowtheory.com Fri Jul 20 03:31:17 2007 From: simon at arrowtheory.com (Simon Burton) Date: Thu, 19 Jul 2007 18:31:17 -0700 Subject: [pypy-dev] rffi strict typing Message-ID: <20070719183117.97ad1ce3.simon@arrowtheory.com> The lltype's are very strict about types. If I declare an llexternal that takes a Float arg, I cannot call it with an int arg. Well maybe that's not so bad. But in rlib.rsdl there are functions that take Unsigned, and these don't work even if called with a non-negative constant int... Can we loosen up these restrictions ? Maybe put an "is_compatable(self, other)" method on LowLevelType's ? Simon. using py lib: /home/simon/local/pypy-latest/py test/test_SDL.py[2] FF __________________________________________________________________________________________________________________________________________________________________________________________________________ entrypoint: test_run ____________________________________________________________ def test_run(): > demo() [/home/simon/local/pypy-latest/pypy/rlib/rsdl/test/test_SDL.py:48] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def demo(): > if SDL.Init(SDL.INIT_VIDEO) < 0: [/home/simon/local/pypy-latest/pypy/rlib/rsdl/test/test_SDL.py:13] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def __call__(self, *args): if isinstance(self._T, FuncType): if len(args) != len(self._T.ARGS): raise TypeError,"calling %r with wrong argument number: %r" % (self._T, args) for a, ARG in zip(args, self._T.ARGS): if typeOf(a) != ARG: E raise TypeError,"calling %r with wrong argument types: %r" % (self._T, args) > TypeError: calling Signed> with wrong argument types: (32,) ... From jgustak at gmail.com Fri Jul 20 11:39:28 2007 From: jgustak at gmail.com (Jakub Gustak) Date: Fri, 20 Jul 2007 11:39:28 +0200 Subject: [pypy-dev] scheme interpreter [status report] In-Reply-To: References: Message-ID: What's on with scheme interpreter: Nice progress during post EuroPython sprint, see: http://codespeak.net/pipermail/pypy-dev/2007q3/003919.html And after the sprint: (define (fun ) ) syntactic sugar added to lambda definitions. (begin ...), (letrec ...) added. This leads to small changes in ExecutionContext (set-car! ), (set-cdr! ) added. Careful, it is possible to create circural list, and crash interpreter when it will try to display such list. Added quasi-quotations. Code is not so straightforward. Maybe it can be done differently. W_Identifier and W_Symbol are now W_Symbol. All symbols are created by symbol(name) function, which stores all symbols in global W_Symbol.obarray dictionary or returns existing one. It makes comparing symbols trivial. Next step is to implement macros. There was also plan to move some some of the syntax checking to parser: e.g. (if test then else) to get parsed directly to MacroIf(test, then, else). The problem is, that this syntax can change in scheme on runtime: e.g. (define if #t) so it doesn't make much sense. And later adding main primitive procedures will make scheme interpreter quite usable. Cheers, Jakub From arigo at tunes.org Fri Jul 20 14:48:17 2007 From: arigo at tunes.org (Armin Rigo) Date: Fri, 20 Jul 2007 14:48:17 +0200 Subject: [pypy-dev] sprint report In-Reply-To: <469CB560.9030108@gmx.de> References: <469CB560.9030108@gmx.de> Message-ID: <20070720124817.GA19336@code0.codespeak.net> Hi all, As Laura and Jacob already hinted at, Laura and me managed to write three times and trash twice code that allows ll external functions to be really called on top of CPython. This means that code can now be written using rffi, including calls to external functions, and this can be tested on top of CPython. We have thus a ctypes replacement, from the point of the view of the CPython user; and we no longer need rctypes at all from the point of the view of the RPython user. Our code (in ll2ctypes.py) is able to transparently convert structures created by lltype.malloc(TYPE, flavor='raw') so that their storage becomes a ctypes object internally. This is done lazily, when the structure "escapes" to a call to an external function. The call itself is also performed through ctypes. As usual we had to fight a bit with ctypes to have it do what we really need (instead of what it thinks you'd prefer today) but nothing deep (Jacob's mail seems to say that we were blocked unless we rewrote ctypes - I'm not sure what he refers to). This is a no-API piece of code invoked automatically when an lltype pointer to an external function is called on top of CPython. A few new pieces of API have been added to rffi.py for general rffi usage, notably rffi.cast(TARGETTYPE, value) which tries to have the same semantics as the C-level cast operator. A bientot, Armin. From arigo at tunes.org Fri Jul 20 14:50:13 2007 From: arigo at tunes.org (Armin Rigo) Date: Fri, 20 Jul 2007 14:50:13 +0200 Subject: [pypy-dev] expected_extra_mallocs In-Reply-To: <20070717080253.a1387d0e.simon@arrowtheory.com> References: <20070717080253.a1387d0e.simon@arrowtheory.com> Message-ID: <20070720125013.GB19336@code0.codespeak.net> Hi Simon, On Tue, Jul 17, 2007 at 08:02:53AM -0700, Simon Burton wrote: > def func(): > x = 1. > print x It's a known non-issue: any RPython program calling the "print" statement "leaks" exactly one malloced piece of memory. It's not an issue because in truth the memory is simply kept around to be reused by the next "print". I guess you need to expect one extra malloc in this case. A bientot, Armin. From arigo at tunes.org Fri Jul 20 14:52:54 2007 From: arigo at tunes.org (Armin Rigo) Date: Fri, 20 Jul 2007 14:52:54 +0200 Subject: [pypy-dev] rffi strict typing In-Reply-To: <20070719183117.97ad1ce3.simon@arrowtheory.com> References: <20070719183117.97ad1ce3.simon@arrowtheory.com> Message-ID: <20070720125254.GC19336@code0.codespeak.net> Hi Simon, On Thu, Jul 19, 2007 at 06:31:17PM -0700, Simon Burton wrote: > The lltype's are very strict about types. Indeed, it seems worthwhile to loosen this restriction at least for the purpose of calling external functions... Not sure how, but there are possible hacks at least. Armin From fijal at genesilico.pl Sun Jul 22 22:42:57 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Sun, 22 Jul 2007 22:42:57 +0200 Subject: [pypy-dev] Strange annotation problem. Message-ID: <46A3C151.5030409@genesilico.pl> Discovered by exarkun. following diff breaks translation :. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: out.diff Url: http://codespeak.net/pipermail/pypy-dev/attachments/20070722/54bd2e47/attachment.diff From amauryfa at gmail.com Mon Jul 23 11:28:52 2007 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Mon, 23 Jul 2007 11:28:52 +0200 Subject: [pypy-dev] Strange annotation problem. In-Reply-To: <46A3C151.5030409@genesilico.pl> References: <46A3C151.5030409@genesilico.pl> Message-ID: Hello, Maciek Fijalkowski wrote: > Discovered by exarkun. following diff breaks translation > > :. > > > Index: ../../module/posix/__init__.py > =================================================================== > --- ../../module/posix/__init__.py (revision 45246) > +++ ../../module/posix/__init__.py (working copy) > @@ -83,6 +83,9 @@ > # interpleveldefs['uname'] = 'interp_posix.uname' > if hasattr(os, 'ttyname'): > interpleveldefs['ttyname'] = 'interp_posix.ttyname' > + if hasattr(os, 'setsid'): > + interpleveldefs['xxx'] = 'interp_posix.xxx' > + > for name in w_star: > if hasattr(os, name): > interpleveldefs[name] = 'interp_posix.' + name > Index: ../../module/posix/interp_posix.py > =================================================================== > --- ../../module/posix/interp_posix.py (revision 45246) > +++ ../../module/posix/interp_posix.py (working copy) > @@ -499,6 +499,22 @@ > raise OperationError(space.w_TypeError, space.wrap(msg)) > utime.unwrap_spec = [ObjSpace, str, W_Root] > > +def xxx(space): > + """xxx() -> pid > + > + Stuff > + """ > + return 3 > +xxx.unwrap_spec = [ObjSpace] > + The function seems to return an unwrapped object. does it behave better if you replace the return value with something like space.wrap(3) ? -- Amaury Forgeot d'Arc From fijal at genesilico.pl Mon Jul 23 12:02:29 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Mon, 23 Jul 2007 12:02:29 +0200 Subject: [pypy-dev] Strange annotation problem. In-Reply-To: References: <46A3C151.5030409@genesilico.pl> Message-ID: <46A47CB5.2070607@genesilico.pl> Amaury Forgeot d'Arc wrote: > Hello, > > Maciek Fijalkowski wrote: >> Discovered by exarkun. following diff breaks translation >> >> :. >> >> >> Index: ../../module/posix/__init__.py >> =================================================================== >> --- ../../module/posix/__init__.py (revision 45246) >> +++ ../../module/posix/__init__.py (working copy) >> @@ -83,6 +83,9 @@ >> # interpleveldefs['uname'] = 'interp_posix.uname' >> if hasattr(os, 'ttyname'): >> interpleveldefs['ttyname'] = 'interp_posix.ttyname' >> + if hasattr(os, 'setsid'): >> + interpleveldefs['xxx'] = 'interp_posix.xxx' >> + >> for name in w_star: >> if hasattr(os, name): >> interpleveldefs[name] = 'interp_posix.' + name >> Index: ../../module/posix/interp_posix.py >> =================================================================== >> --- ../../module/posix/interp_posix.py (revision 45246) >> +++ ../../module/posix/interp_posix.py (working copy) >> @@ -499,6 +499,22 @@ >> raise OperationError(space.w_TypeError, space.wrap(msg)) >> utime.unwrap_spec = [ObjSpace, str, W_Root] >> >> +def xxx(space): >> + """xxx() -> pid >> + >> + Stuff >> + """ >> + return 3 >> +xxx.unwrap_spec = [ObjSpace] >> + > > The function seems to return an unwrapped object. > does it behave better if you replace the return value with > something like space.wrap(3) ? > Hum. Good point :-) I must be sleeping again... Cheers, fijal :. From anto.cuni at gmail.com Mon Jul 23 13:07:55 2007 From: anto.cuni at gmail.com (Antonio Cuni) Date: Mon, 23 Jul 2007 13:07:55 +0200 Subject: [pypy-dev] Proposal of timeline In-Reply-To: <9c0bb8a00707211508k3697f0e3oe8a5a2276c2f5d83@mail.gmail.com> References: <469BE246.20107@gmail.com> <9c0bb8a00707161429v3d8761ag681353aacb7c87d7@mail.gmail.com> <9c0bb8a00707211508k3697f0e3oe8a5a2276c2f5d83@mail.gmail.com> Message-ID: <46A48C0B.20705@gmail.com> Hi Paul, I'm cc-ing pypy-dev. Really, don't be shy and don't worry to bother the list with your questions. They are not spam. I'm sure that the others are happy to hear your questions and also to answer in case I can't do it immediately (as happened in the last weekend). Paul deGrandis wrote: > Antonio, > > I've been working on the JVM backend. When I trace the float tests that > look at uint handling, weird NEGATIVE values pop up. I originally > associated these with overflow, but it's a different issue. Any > insight? The way to do unsigned values in Java is to use the next > primary type larger, for example, a uint is stored in a double... but > currently no checks exist for negative values. as Niko pointed out unsigned types are stored in the corresponding signed types in the JVM: this is ok becasue they occupy the same amount of bits, but you have to been careful when dealing with them because the JVM thinks you want to represent a signed value. In particular, if you try to print one to stdout it will be printed as a signed, and so this might be the reason why you see negative values. You have to write a custom procedure to format unsigned values into the right human readable implementation. Please note that this is necessary only during testing or when doing I/O: as long as your unsigned values are kept inside the JVM, they works just fine (apart from comparisons). The simplest way I can think to format an unsigned into an human readable form is to convert it into a double, as I already wrote in a previous email: double from_uint(int x) { double res = x; if (x < 0) // i.e., the leftmost but is 1 res += 4294967294; return res; } But again, you should use this *only* for I/O and testing, not for doing any real operation. > I looked at the test_overflow but these tests don't make sense for the > jvm. You have overflow built in to CLI so you want to make sure that > ovf_check from the rpython definition works, but the JVM defines these > operations. What exactly are these tests aiming to illustrate and test? RPython is a language with a (more or less :-)) well defined semantics, so every backend must take care of implementing this semantics correctly, even if it's not natively supported by the underlying platform. In this particular case, RPython programs expect overflow to be detected correctly. If the JVM does not do this for you, you have to handle those cases by yourself. GenC has the same problem: the C platform does not detect any overflow, so it must explicitly check for that; you can easily reuse the same logic; for example, this macro is taken from translator/c/src/int.h: #define OP_INT_ADD_OVF(x,y,r) \ OP_INT_ADD(x,y,r); \ if ((r^(x)) >= 0 || (r^(y)) >= 0); \ else FAIL_OVF("integer addition") for genjvm, you could e.g. write a java function doing the same (and raising the appropriate exception in case of overflow). > I haven't tackled overriding default values (as skipped in test_class) > yet, but that's really all there is needed left to pass the test_class > requirements I think you could skip this for now. IIRC it's only needed to translate pypy with the thunk objspace, it's not a priority. > I'm still working on the others, but I don't think I'll have trouble > having these first bugs fixed by the end of the work day on the 27th. ok, that's cool! :-) > Lastly, what is the easy approach to I/O and external functions? the easy approach is to forward the external functions to the backend, which will take care of rendering them as appropriate (usually by calling an helper function written in java). This approach is already used to implement time.time, time.clock and os.write: you can find their implementation in jvm/src/pypy/PyPy.java (ll_time_time, ll_time_clock, ll_os_write). The code that calls these functions is in jvm/generator.py, method 'call_primitive'. ciao Anto From fijal at genesilico.pl Mon Jul 23 13:21:41 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Mon, 23 Jul 2007 13:21:41 +0200 Subject: [pypy-dev] Proposal of timeline In-Reply-To: <46A48C0B.20705@gmail.com> References: <469BE246.20107@gmail.com> <9c0bb8a00707161429v3d8761ag681353aacb7c87d7@mail.gmail.com> <9c0bb8a00707211508k3697f0e3oe8a5a2276c2f5d83@mail.gmail.com> <46A48C0B.20705@gmail.com> Message-ID: <46A48F45.8090505@genesilico.pl> > >> Lastly, what is the easy approach to I/O and external functions? >> > > the easy approach is to forward the external functions to the backend, > which will take care of rendering them as appropriate (usually by > calling an helper function written in java). This approach is already > used to implement time.time, time.clock and os.write: you can find their > implementation in jvm/src/pypy/PyPy.java (ll_time_time, ll_time_clock, > ll_os_write). The code that calls these functions is in > jvm/generator.py, method 'call_primitive'. > > ciao Anto > Small addition from my side. I've been playing recently with a very naive reuse of BasicExternal infrastructure which is supporting external objects in JS backend. Looks nice so far and I'm able to hard code some RPython-accessible objects (in a very ad hoc manner, mind you). I don't know Java that well, but I would be happy to help with this issue. Cheers, fijal :. From amauryfa at gmail.com Mon Jul 23 14:32:46 2007 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Mon, 23 Jul 2007 14:32:46 +0200 Subject: [pypy-dev] [pypy-svn] r45269 - in pypy/dist/pypy/rpython/module: . tes Message-ID: Hello again, > Author: fijal > Date: Mon Jul 23 13:52:26 2007 > New Revision: 45269 > > Modified: > pypy/dist/pypy/rpython/module/ll_os.py > pypy/dist/pypy/rpython/module/test/test_posix.py > Log: > Add os.uname. RPython level. segfaults ll2ctypes in some strange way, > I'm not sure I want to know > > > Modified: pypy/dist/pypy/rpython/module/ll_os.py > ============================================================================== > --- pypy/dist/pypy/rpython/module/ll_os.py (original) > +++ pypy/dist/pypy/rpython/module/ll_os.py Mon Jul 23 13:52:26 2007 > @@ -110,6 +110,34 @@ > register_external(os.setsid, [], int, export_name="ll_os.ll_os_setsid", > llimpl=setsid_lltypeimpl) > > +# ------------------------------- os.uname ------------------------------ > + > +if hasattr(os, 'uname'): > + UTSNAMEP = rffi.CStruct('utsname', ('sysname', rffi.CCHARP), > + ('nodename', rffi.CCHARP), > + ('release', rffi.CCHARP), > + ('version', rffi.CCHARP), > + ('machine', rffi.CCHARP), > + ('stuff', rffi.CCHARP)) I think the error comes from the ctypes interpretation of the structure. utsname is not made of char* pointers, but (at least on the debian machine I have access to) the members are actually fixed arrays of chars, the lengths of which are not public, but well specified in sys/utsname.h. It should make no difference when translated, because the code looks like v->sysname etc. ctypes on the other hand needs to know the precise sizeof and offsets of each member. I tried to modify the _socket module to also use rffi, and had the same problem. I think that we will need a tool similar to pypy.rpython.rctypes.tool.ctypes_platform, which generates and compiles C code to get the different sizes and offsets. (We will need support for C defines as well) Hope this helps, -- Amaury Forgeot d'Arc From fijal at genesilico.pl Mon Jul 23 14:36:37 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Mon, 23 Jul 2007 14:36:37 +0200 Subject: [pypy-dev] [pypy-svn] r45269 - in pypy/dist/pypy/rpython/module: . tes In-Reply-To: References: Message-ID: <46A4A0D5.9020405@genesilico.pl> Amaury Forgeot d'Arc wrote: > Hello again, > >> Author: fijal >> Date: Mon Jul 23 13:52:26 2007 >> New Revision: 45269 >> >> Modified: >> pypy/dist/pypy/rpython/module/ll_os.py >> pypy/dist/pypy/rpython/module/test/test_posix.py >> Log: >> Add os.uname. RPython level. segfaults ll2ctypes in some strange way, >> I'm not sure I want to know >> >> >> Modified: pypy/dist/pypy/rpython/module/ll_os.py >> ============================================================================== >> >> --- pypy/dist/pypy/rpython/module/ll_os.py (original) >> +++ pypy/dist/pypy/rpython/module/ll_os.py Mon Jul 23 13:52:26 2007 >> @@ -110,6 +110,34 @@ >> register_external(os.setsid, [], int, >> export_name="ll_os.ll_os_setsid", >> llimpl=setsid_lltypeimpl) >> >> +# ------------------------------- os.uname >> ------------------------------ >> + >> +if hasattr(os, 'uname'): >> + UTSNAMEP = rffi.CStruct('utsname', ('sysname', rffi.CCHARP), >> + ('nodename', rffi.CCHARP), >> + ('release', rffi.CCHARP), >> + ('version', rffi.CCHARP), >> + ('machine', rffi.CCHARP), >> + ('stuff', rffi.CCHARP)) > > I think the error comes from the ctypes interpretation of the > structure. utsname is not made of char* pointers, but (at least on the > debian machine I have access to) the members are actually fixed arrays > of chars, the lengths of which are not public, but well specified in > sys/utsname.h. > > It should make no difference when translated, because the code looks > like v->sysname etc. > ctypes on the other hand needs to know the precise sizeof and offsets > of each member. > > I tried to modify the _socket module to also use rffi, and had the > same problem. > > I think that we will need a tool similar to > pypy.rpython.rctypes.tool.ctypes_platform, which generates and > compiles C code to get the different sizes and offsets. > (We will need support for C defines as well) > > Hope this helps, > I think armin worked a bit on it. We need some more support for that I'm afraid. Let's dig... :. From fijal at genesilico.pl Mon Jul 23 14:38:32 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Mon, 23 Jul 2007 14:38:32 +0200 Subject: [pypy-dev] [pypy-svn] r45269 - in pypy/dist/pypy/rpython/module: . tes In-Reply-To: References: Message-ID: <46A4A148.2090802@genesilico.pl> From the manpage: The length of the arrays in a struct utsname is unspecified; the fields are terminated by a null byte (?? ?). ??? I'm completely confused. :. From arigo at tunes.org Tue Jul 24 18:08:22 2007 From: arigo at tunes.org (Armin Rigo) Date: Tue, 24 Jul 2007 18:08:22 +0200 Subject: [pypy-dev] [pypy-svn] r45269 - in pypy/dist/pypy/rpython/module: . tes In-Reply-To: <46A4A148.2090802@genesilico.pl> References: <46A4A148.2090802@genesilico.pl> Message-ID: <20070724160822.GA25719@code0.codespeak.net> Hi Fijal, On Mon, Jul 23, 2007 at 02:38:32PM +0200, Maciek Fijalkowski wrote: > The length of the arrays in a struct utsname is unspecified; the > fields > are terminated by a null byte (?????? ???). > > I'm completely confused. It means that the structure looks like this in lltype notation: Struct('utsname', ('text1', FixedSizeArray(Char, 10)), ('text2', FixedSizeArray(Char, 20)), ...) where, annoyingly, the numbers '10' and '20' are platform-dependent and must be obtained by trying to compile snippets of C code using sizeof() and offsetof(). Or are you confused about the bit saying that the fields are zero-terminated? It just means that in the array, characters 0 to N-1 are non-null, charater N is null, and the rest of the characters are undefined. A bientot, Armin. From tismer at stackless.com Tue Jul 24 21:18:48 2007 From: tismer at stackless.com (Christian Tismer) Date: Tue, 24 Jul 2007 21:18:48 +0200 Subject: [pypy-dev] [pypy-svn] r45269 - in pypy/dist/pypy/rpython/module: . tes In-Reply-To: <20070724160822.GA25719@code0.codespeak.net> References: <46A4A148.2090802@genesilico.pl> <20070724160822.GA25719@code0.codespeak.net> Message-ID: <01F2E744-7AB2-4A88-8049-52F9FDD9DA06@stackless.com> On 24.07.2007, at 18:08, Armin Rigo wrote: > Hi Fijal, > > On Mon, Jul 23, 2007 at 02:38:32PM +0200, Maciek Fijalkowski wrote: >> The length of the arrays in a struct utsname is >> unspecified; the >> fields >> are terminated by a null byte (?????? ???). >> >> I'm completely confused. > > It means that the structure looks like this in lltype notation: > > Struct('utsname', ('text1', FixedSizeArray(Char, 10)), > ('text2', FixedSizeArray(Char, 20)), > ...) > > where, annoyingly, the numbers '10' and '20' are platform-dependent > and > must be obtained by trying to compile snippets of C code using > sizeof() > and offsetof(). > > Or are you confused about the bit saying that the fields are > zero-terminated? It just means that in the array, characters 0 to N-1 > are non-null, charater N is null, and the rest of the characters are > undefined. Which reminds us to implement this carefully and not to rely on the terminating byte, or we will open the door to buffer overflow exploits :-) ciao - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20070724/42748819/attachment.htm From tismer at stackless.com Tue Jul 24 21:33:09 2007 From: tismer at stackless.com (Christian Tismer) Date: Tue, 24 Jul 2007 21:33:09 +0200 Subject: [pypy-dev] sprint report In-Reply-To: <20070717134238.a1c0d9ab.simon@arrowtheory.com> References: <469CB560.9030108@gmx.de> <20070717134238.a1c0d9ab.simon@arrowtheory.com> Message-ID: <1D942D29-1FEB-4228-9EA2-625B295EE659@stackless.com> As Jacob mentioned, I worked on fixing Stackless Pickling. I thought this would be a quick thing for the first morning, but it blocked me very much until the end of the sprint, where a discussion with Samuele clarified to us (we both didn't understand it for quite long). The problem holds for any app-level implementation of any switching concept that should be pickled: Unless you implement this very ugly with global variables, only, you will always have a reference to the tasklet/whatever on the python stack, that tries to do the pickling. I.E. you always end up pickling the pickler itself, in principal. This does not happen from the main tasklet, because it has special handing for that case. This needs to be re-thought. Some special code, somewhat similar to resume points in RPython needs to be implemented, to shortcut the frames on the Python stack which should not be visible (and therefore being pickled) at all. Hoping to be more productive and communicative next time - this was a huge blocker. ciao - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20070724/5fcd0e4d/attachment-0001.htm From fijal at genesilico.pl Wed Jul 25 09:17:03 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Wed, 25 Jul 2007 09:17:03 +0200 Subject: [pypy-dev] [pypy-svn] r45269 - in pypy/dist/pypy/rpython/module: . tes In-Reply-To: <01F2E744-7AB2-4A88-8049-52F9FDD9DA06@stackless.com> References: <46A4A148.2090802@genesilico.pl> <20070724160822.GA25719@code0.codespeak.net> <01F2E744-7AB2-4A88-8049-52F9FDD9DA06@stackless.com> Message-ID: <46A6F8EF.9030202@genesilico.pl> Christian Tismer wrote: > > On 24.07.2007, at 18:08, Armin Rigo wrote: > >> Hi Fijal, >> >> On Mon, Jul 23, 2007 at 02:38:32PM +0200, Maciek Fijalkowski wrote: >>> The length of the arrays in a struct utsname is unspecified; the >>> fields >>> are terminated by a null byte (?????? ???). >>> >>> I'm completely confused. >> >> It means that the structure looks like this in lltype notation: >> >> Struct('utsname', ('text1', FixedSizeArray(Char, 10)), >> ('text2', FixedSizeArray(Char, 20)), >> ...) >> >> where, annoyingly, the numbers '10' and '20' are platform-dependent and >> must be obtained by trying to compile snippets of C code using sizeof() >> and offsetof(). >> >> Or are you confused about the bit saying that the fields are >> zero-terminated? It just means that in the array, characters 0 to N-1 >> are non-null, charater N is null, and the rest of the characters are >> undefined. > > Which reminds us to implement this carefully and not to rely on the > terminating byte, or we will open the door to buffer overflow exploits :-) > > ciao - chris > I personally don't think that implementation of uname() might be a security hazard - if kernel really wants to hack us it can for sure :] Cheers, fijal :. From fijal at genesilico.pl Thu Jul 26 14:10:32 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Thu, 26 Jul 2007 14:10:32 +0200 Subject: [pypy-dev] Design question Message-ID: <46A88F38.4040009@genesilico.pl> I'm approaching "how to make cache failures not crash on you every time you import anything". So in short, file like ll_os.py looks right now: if cache.defined('XXX'): TYPE = cache.inttype(...) register_external(...) Idea is that if any of the functions involved in creation of external (cache functions...), show a warning and than create a register_external stub which will explode if referenced (by ie a specific extregistry entry). The question is how it should look like than. My idea is: class Os(registeringclass): def register_os_uname(...): ... def register_os_other(...): ... and so on. This will create huge indentation. Any other ideas? Cheers, fijal :. From jgustak at gmail.com Thu Jul 26 17:47:16 2007 From: jgustak at gmail.com (Jakub Gustak) Date: Thu, 26 Jul 2007 17:47:16 +0200 Subject: [pypy-dev] scheme interpreter [status report] In-Reply-To: References: Message-ID: > Next step is to implement macros. We have working simple macros. They are hygienic and referentially transparent. There is no support so you can't use (expr ...) syntax yet. There is also no support for recursive expanding also. There was again some ExecutionContext refactoring. Still only (let ...) and (let* ...) are fully compatible with macro expanding. (letrec ...) is not. In the future: more macros! (ellipsis, recursive expanding and CPS style macros) Cheers, Jakub Gustak From jgustak at gmail.com Thu Jul 26 19:37:59 2007 From: jgustak at gmail.com (Jakub Gustak) Date: Thu, 26 Jul 2007 19:37:59 +0200 Subject: [pypy-dev] scheme interpreter [status report] In-Reply-To: References: Message-ID: Some more info. > We have working simple macros. They are hygienic and referentially transparent. r5rs states that if macro introduces new binding, it should be renamed to avoid conflicts with other identifiers. In our case we create syntactic closure like described in: http://people.csail.mit.edu/people/jhbrown/scheme/macroslides04.pdf ExecutionContext.sput() (creating new binding) checks if it deals with syntactic closure or not, and depending on that binds location in proper context. W_Transformer can be called with quoted expression as argument, then appropriate template will expand and then expanded expression will be evaluated. This way of evaluation will surely cause problems with recursive macros, so it is not recommended. __ Jakub From fijal at genesilico.pl Fri Jul 27 23:54:30 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Fri, 27 Jul 2007 23:54:30 +0200 Subject: [pypy-dev] partial evaluation links Message-ID: <46AA6996.3070101@genesilico.pl> not sure, if this is of any interest: http://partial-eval.org/pe-impl.html list of partial evaluation implementations (mostly scheme, but also C). Cheers, fijal :. From fijal at genesilico.pl Sun Jul 29 20:30:26 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Sun, 29 Jul 2007 20:30:26 +0200 Subject: [pypy-dev] A quick question Message-ID: <46ACDCC2.3080104@genesilico.pl> Can we move jit tests to the end of nightly test run? It'll help quick-checking whether there are no breakages spread all around. Cheers, fijal :. From fijal at genesilico.pl Wed Aug 1 12:28:42 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Wed, 01 Aug 2007 12:28:42 +0200 Subject: [pypy-dev] Stackless vs Erlang benchmarks Message-ID: <46B0605A.6010305@genesilico.pl> http://muharem.wordpress.com/2007/07/31/erlang-vs-stackless-python-a-first-benchmark/ Christian: with a dedication for you :) We should try pypy on this btw. :. From jgustak at gmail.com Thu Aug 2 18:08:01 2007 From: jgustak at gmail.com (Jakub Gustak) Date: Thu, 2 Aug 2007 18:08:01 +0200 Subject: [pypy-dev] scheme interpreter [status report] In-Reply-To: References: Message-ID: A first, a joke: (syntax-rules () ((i-have-spare day ...) (hack 'macros (day ....)))) So we have recursive macros expanding recursively, as they should. I was working on macros with ellipsis almost whole week i. I was changing approach several times. But it looks like flat ellipses (not nested) work well. All most important parts of macros are SyntaxRule.match method, or rather matchr, and W_Transformer.substitute. They raise exception when discover ellipsis to handle it at higher level. matchr is kinda handling nested ellipses, but substitute not yet. That's pretty it. I would like to get nested ellipses working and then start playing with continuations. Wish me luck. Cheers, Jakub Gustak From simon at arrowtheory.com Fri Aug 3 21:14:13 2007 From: simon at arrowtheory.com (Simon Burton) Date: Fri, 3 Aug 2007 12:14:13 -0700 Subject: [pypy-dev] rffi feature request Message-ID: <20070803121413.bd2d992a.simon@arrowtheory.com> I would like to expose some functions as external symbols when i build a .so def foo(i, j): return i+j foo._expose_ = [rffi.INT, rffi.INT] This is basically so I can write cpython extension modules in rpython. (and manually doing ref counting (etc.) on the cpython objects.) Simon. From simon at arrowtheory.com Sun Aug 5 03:14:03 2007 From: simon at arrowtheory.com (Simon Burton) Date: Sat, 4 Aug 2007 18:14:03 -0700 Subject: [pypy-dev] rffi feature request In-Reply-To: <20070803121413.bd2d992a.simon@arrowtheory.com> References: <20070803121413.bd2d992a.simon@arrowtheory.com> Message-ID: <20070804181403.f15cdbb6.simon@arrowtheory.com> On Fri, 3 Aug 2007 12:14:13 -0700 Simon Burton wrote: > > I would like to expose some functions as external > symbols when i build a .so > > def foo(i, j): > return i+j > > foo._expose_ = [rffi.INT, rffi.INT] It seems like this could also enable a plugin system for rpython, and for example, (c or rpython) extension modules for the PyPy interpreter. Simon. From cfbolz at gmx.de Sun Aug 5 22:00:43 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Sun, 05 Aug 2007 22:00:43 +0200 Subject: [pypy-dev] Stackless vs Erlang benchmarks In-Reply-To: <46B0605A.6010305@genesilico.pl> References: <46B0605A.6010305@genesilico.pl> Message-ID: <46B62C6B.5040306@gmx.de> Hi Maciek Maciek Fijalkowski wrote: > http://muharem.wordpress.com/2007/07/31/erlang-vs-stackless-python-a-first-benchmark/ > > Christian: with a dedication for you :) > > We should try pypy on this btw. seems a bit meaningless, given that one of erlang's most important strengths is the possibility of using it to transparently across multiple processes and especially multiple machines. Cheers, Carl Friedrich From paul.degrandis at gmail.com Tue Aug 7 03:46:28 2007 From: paul.degrandis at gmail.com (Paul deGrandis) Date: Mon, 6 Aug 2007 21:46:28 -0400 Subject: [pypy-dev] Trouble with ooparse_int/ooparse_float Message-ID: <9c0bb8a00708061846m265d5f30q7f47eac4442242e1@mail.gmail.com> Anto, Niko, and all, I've been trying to get some of the string and float tests to pass. The trouble I'm running into is that ooparse_float doesn't know how to parse ll_str_0, but for the life of me can't find where I need to be looking or where I should override the ooparse_float method. I feel like it should go in my opcodes.py file (translator/jvm). Also, one question I had about long conversions, in translator/jvm/test/test_float.py:22, there is a test for long conversion. Tracing the results shows I get the correct answer for the conversion, but my result is not an r_longlong, the tests returns a Java Long. Any tips or hints? Regards, Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20070806/d565ae99/attachment.htm From anto.cuni at gmail.com Tue Aug 7 14:09:26 2007 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 07 Aug 2007 14:09:26 +0200 Subject: [pypy-dev] Trouble with ooparse_int/ooparse_float In-Reply-To: <9c0bb8a00708061846m265d5f30q7f47eac4442242e1@mail.gmail.com> References: <9c0bb8a00708061846m265d5f30q7f47eac4442242e1@mail.gmail.com> Message-ID: <46B860F6.9030306@gmail.com> Hi Paul! Paul deGrandis wrote: > Anto, Niko, and all, > > I've been trying to get some of the string and float tests to pass. The > trouble I'm running into is that ooparse_float doesn't know how to parse > ll_str_0, but for the life of me can't find where I need to be looking > or where I should override the ooparse_float method. I feel like it > should go in my opcodes.py file (translator/jvm). I'm not sure to understand the question. ooparse_float is an opcode which converts strings to floats. To implement it, you must add the corresponding definition to translator/jvm/opcodes.py; the simplest thing to do is probably to call java.lang.Double.parseDouble(); I don't know the exact syntax to use, but I guess Niko could help here :-). > Also, one question I had about long conversions, in > translator/jvm/test/test_float.py:22, there is a test for long > conversion. Tracing the results shows I get the correct answer for the > conversion, but my result is not an r_longlong, the tests returns a Java > Long. Any tips or hints? This happens because the same tests are used to test both the llinterpreter and the backends: when run in the llinterpreter, the return type is effectively r_longlong and the test runs fine, but when translated by a backend such as cli or jvm, the result is simply printed to stdout, without any information about the type: thus, there is no way to test if the result was effectively a r_longlong. Look at the source of the test, in rpython/test/test_rfloat.py:69: you see that the type-checking is done by calling the is_of_type method; now, look how the CLI test framerwork implements this method, in cli/test/runtest.py:299: as you can see, it simply returns always true, because it has no chance to test it. What you need to do is simply to do the same in jvm/test/runtest.py. ciao Anto From fijal at genesilico.pl Tue Aug 7 14:15:48 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Tue, 07 Aug 2007 14:15:48 +0200 Subject: [pypy-dev] rffi feature request In-Reply-To: <20070804181403.f15cdbb6.simon@arrowtheory.com> References: <20070803121413.bd2d992a.simon@arrowtheory.com> <20070804181403.f15cdbb6.simon@arrowtheory.com> Message-ID: <46B86274.2040401@genesilico.pl> Simon Burton wrote: > On Fri, 3 Aug 2007 12:14:13 -0700 > Simon Burton wrote: > > >> I would like to expose some functions as external >> symbols when i build a .so >> >> def foo(i, j): >> return i+j >> >> foo._expose_ = [rffi.INT, rffi.INT] >> > > It seems like this could also enable a plugin system for rpython, > and for example, (c or rpython) extension modules for the PyPy interpreter. > > Simon. > I kind of don't understand what are you trying to do. Could you explain in a bit more detail? (Ie are you trying to have rffi-rffi bridge between interpreter and compiled module?) Why not fix extcompiler instead? :. From simon at arrowtheory.com Wed Aug 8 18:14:40 2007 From: simon at arrowtheory.com (Simon Burton) Date: Wed, 8 Aug 2007 09:14:40 -0700 Subject: [pypy-dev] rffi feature request In-Reply-To: <46B86274.2040401@genesilico.pl> References: <20070803121413.bd2d992a.simon@arrowtheory.com> <20070804181403.f15cdbb6.simon@arrowtheory.com> <46B86274.2040401@genesilico.pl> Message-ID: <20070808091440.f8e06de1.simon@arrowtheory.com> On Tue, 07 Aug 2007 14:15:48 +0200 Maciek Fijalkowski wrote: > Simon Burton wrote: > > On Fri, 3 Aug 2007 12:14:13 -0700 > > Simon Burton wrote: > > > > > >> I would like to expose some functions as external > >> symbols when i build a .so > >> > >> def foo(i, j): > >> return i+j > >> > >> foo._expose_ = [rffi.INT, rffi.INT] > >> > > > > It seems like this could also enable a plugin system for rpython, > > and for example, (c or rpython) extension modules for the PyPy interpreter. > > > > Simon. > > > I kind of don't understand what are you trying to do. Could you explain > in a bit more detail? (Ie are you trying to have rffi-rffi bridge > between interpreter and compiled module?) Why not fix extcompiler instead? > > :. > well, the above code would produce: extern int foo(int i, int j) { return i+j; } (and perhaps an accompanying .h file) thereby providing an interface for other C programs. This is rffi producing rather than consuming a C interface. Simon. From jgustak at gmail.com Thu Aug 9 21:30:41 2007 From: jgustak at gmail.com (Jakub Gustak) Date: Thu, 9 Aug 2007 21:30:41 +0200 Subject: [pypy-dev] scheme interpreter [status report] In-Reply-To: References: Message-ID: Macros looks like working. One thing to be added: they are acting like first-class objects, but should not. Right now I am working on continuations. First a puzzle: What will return evaluation of s-expression like this: (call/cc call/cc) Or what will be bound to k after this s-expression: (define k (call/cc call/cc)) After some asking on #pypy I decided to create own implementation of continuations, and not try to use stackless. The reason is that coroutines are one-shot. So I created execution stack simulation, or rather stack for continuation frames. Right now not every macro uses it, so not every one is continuation-safe. If macro has defined method continue_tr it should get along with continuations well. Another issue: this implementations is stack-only so it introduce lot of overhead both on capturing and throwing continuations. And one more: every W_Procedure is continuations-safe but VERY slow, even when no capture occurs. I'd rather would like to think about how to throw continuations more generally, and not have to implement continue_tr for every W_Callable, than implementing capture differently. Or maybe try totally different approach e.g write every call in CPS, but it probably would require way too much changes right now. Cheers, Jakub Gustak From overminddl1 at gmail.com Fri Aug 10 03:21:21 2007 From: overminddl1 at gmail.com (OvermindDL1) Date: Thu, 9 Aug 2007 19:21:21 -0600 Subject: [pypy-dev] Stackless vs Erlang benchmarks In-Reply-To: <46B62C6B.5040306@gmx.de> References: <46B0605A.6010305@genesilico.pl> <46B62C6B.5040306@gmx.de> Message-ID: <3f49a9f40708091821u16f1055esb0777c2df4e1d4ac@mail.gmail.com> I made a library not long ago for stackless cpython meant to partially emulate that aspect of Erlang (called the library Pylang aptly enough, no I have no website, mostly personal use, I have never seen anyone in the community actively interested in such things from my Python experience, although if anyone voices opinions here then I could be convinced to release what I have). But you write your actors as if they are mini programs with their own even loops and so forth. They should not communicate by direct interaction of objects (unless you know beyond any doubt they will be in the same process, still better to just purely use message passing as I do with it) but rather strictly by message passing. Here is one of my tests that show the event loop and so forth (this test tests the message loop, does not show PID usage, network interaction, or anything else): _______________________________________________________ from ..core import getMessage # Yes, yes, I know, castrate me now # for using relative imports, this file is pylang/tests/test0.py and I do # need to do some file renaming and class shuffling if I release this from ..nameserver import NameServer # If I release, I need to rename # this as well since it does far more then the original design called for # This vars and this class are just to make ease of message passing # in this first test more simple, it is not necessary though TEST0_CONFIRMATION=0 TEST0_TESTSUBROUTINE=1 TEST0_ONLYSUBROUTINE=2 TEST0_EXIT=-1 class test0Message(object): def __init__(self, type): self.type=type def __str__(self): return "Message type: %i" % (self.type) def test0SubRoutine(): print "\t 0:Subroutine entered, waiting for signal only for me" e=getMessage(lambda e: isinstance(e, test0Message) and e.type==TEST0_ONLYSUBROUTINE) print "\t 0:Subroutine message receieved (should be %i):"%(TEST0_ONLYSUBROUTINE), e print "\t 0:Exiting subroutine" def test0Main(): while True: e=getMessage(timeout=1) if e is None: print "\t0:Timeout receieved" elif not isinstance(e, test0Message): print "\t0:Unknown message receieved, not of this test type" elif e.type==TEST0_CONFIRMATION: print "\t0:Confirmation received" elif e.type==TEST0_TESTSUBROUTINE: print "\t0:Testing subroutine" test0SubRoutine() elif e.type==TEST0_ONLYSUBROUTINE: print "0:ERROR: Only Subroutine message received when not in a callback" elif e.type==TEST0_EXIT: print "\t0:Exit request received, exiting process" return else: print "\t0:Unknown message type received, either need to set a condition to only get what you want, or dump this:", e def test0Generator(p): global exitScheduler print "Generator started, sending confirmation" p.sendMessage(test0Message(TEST0_CONFIRMATION)) Sleep(0.01) print "Generator sleeping for 1.5 seconds" Sleep(1.5) print "Sending test subroutine" p.sendMessage(test0Message(TEST0_TESTSUBROUTINE)) Sleep(0.01) print "Sending confirmation" p.sendMessage(test0Message(TEST0_CONFIRMATION)) Sleep(0.01) print "Sending only subroutine" p.sendMessage(test0Message(TEST0_ONLYSUBROUTINE)) Sleep(0.01) print "Sending confirmation" p.sendMessage(test0Message(TEST0_CONFIRMATION)) Sleep(0.01) print "Sending untyped test0Message" p.sendMessage(test0Message(10)) Sleep(0.01) print "Sending exit message" p.sendMessage(test0Message(TEST0_EXIT)) Sleep(0.01) print "Generator exiting" NameServer.__nameserver__.shutdown() # This sends a close message to all open # Processes and continues for a bit, if they have not all died by the timeout then they # are sent the taskletkill exception, which will kill them regardless. Also shuts down # the server and other such things def runTest0(): print "\t0:Testing local process and local message communication" NameServer(serverNode=NullServer()) # NullServer does not host squat. You can # create servernodes to handle tcp, udp, some other communication type, equiv to # a driver in Erlang, everything on the mesh network needs to use the same thing # without using a gateway inside the mesh of some sort to connect different ones p=Process(test0Main)() Process(test0Generator)(p) # Not the proper way to spawn processes now, # should call spawn(), but this was an early testcase that is still useful as-is NameServer.__nameserver__.run() _______________________________________________________ And it works as expected. The testcase that just tests network setup and destruction (no message passing or anything, just creation, verification, and destruction) is this: _______________________________________________________ from ..nameserver import NameServer from ..node import TCPServer from ..core import Process, getMessage def tempKillAll(waitTime=50, ns=None): if not ns: ns = NameServer.__nameserver__ Sleep(waitTime%10) for i in xrange(int(waitTime)-(waitTime%10), 0, -10): print "%i seconds until death" %(i) Sleep(10) print "dieing" ns.shutdown() def testServer(timeout=30, cookie='nocookie', localname='myhostA'): NameServer(serverNode=TCPServer(localname=localname, cookie=cookie, host='', port=42586, backlog=5)) Process(tempKillAll)(timeout) NameServer.__nameserver__.run() def testClient(timeout=3, cookie='nocookie', localname='myhostB', remoteURL='pylang://myhostA:42586/'): # no the port in the url is not # necessary as it is the default port used by this anyway NameServer(serverNode=TCPServer(localname=localname, cookie=cookie, host='', port=42587, backlog=5)) Process(tempKillAll)(timeout) Sleep(0.1) NameServer.__nameserver__.createNewNode(remoteURL) NameServer.__nameserver__.run() _______________________________________________________ The URL in full would actually be "pylang://myhostB:nocookie at myhostA:42586/". But if any are omitted then it uses the local hostname, the cookie, the remote host, the default port, etc... To connect to a remote named process (not an anonymous process) then you could get its PID with: PID("pylang://myhostB:nocookie at myhostA:42586/someNamedProcess") and if the connection to the remote node is not made then it will be made. If you have a Node pointer to an already connected node then you can just call a method on it to get a PID to a remote named process. If you send a message through a PID (the main way of sending messages) then they may or may not arrive because, although you can get a PID to a named process for example, the process may not actually exist. You can query a process to see if it exists which involves calling a ping method on the PID which will actually perform a test on the remote node (which may be on the same process/node, or a networked computer or what-not) and send back a specially crafted message saying pong or pang for success or fail with the PID as the param. A PID can always be converted to a pylang URL and vice-versa (as even anonymous processes have a guid generated for them). There can also be a type param of the url (example: "pylang://myhostB:nocookie at myhostA:42586;tcp/", then 'tcp' would be the type, standard url syntax) and if so then it will try to create a connection to the remote node using that connection (tcp, udp, ssl, pipe, raknet, whateverisMade) instead of the default registered to the nameserver by the servernode param, useful for connecting to a mesh that uses a different server type.. Processes can choose to save themselves for transmission across a network, save to a db or file, etc..., and they will retain their GUID. When they are serialized they are not serialized as a whole but they send a structure with the method call which is returned as an initial message to a process when it is restarted so it can reconstruct itself from it, the guid is always included and handled transparently. Thus, a process can be 'pickled' quite easilly, but it must be built for it (not hard to do, just needs to be done), reason being is that although stackless allows full tasklets to be pickled, some process may have some rather... adverse effects to being pickled (like one that handled any form of resources, db, file, etc...) hence why I have it do it this other, more 'erlangish' method. The message loop function (I do apologize for jumping around in thoughts...) is not just a standard pull the first off the queue like in a normal OS message loop, but has a signature of: "message getMessage(conditionFn=lambda e:True, timeout=-1)". When getMessage is called then it starts testing the messages in the queue for the process it was called from in order they were received, testing each with the conditionFn (allows for nice complex testing for something like "lambda e: isinstance(e, test0Message) and e.type==TEST0_ONLYSUBROUTINE" or "lambda e: isinstance(e, tuple) and len(e)==2 and e[0]==14" or whatever to simple things like just returning true to get the next top thing as is the default to passing in another function. If it good to get and empty out the queue on occasion to make sure it does not fill up with messages you do not want, slowing it down slightly. The other param is timeout (in seconds). If timeout equals -1 then it will wait forever for a matching message. If timeout equals 0 then it just test and returns a message if one matches that it already has, or it returns None (due to zero timeout). If greater then zero then it tests for a matching message, if none, then it waits up to the timeout for a message, if one arrives by that time then it returns it immediately, if not then it returns None when it reaches the timeout. It is not a spin-wait nor does it delay receiving the message or anything of so forth. The nameserver runs a pure tasklet (no message queue, no nothing like that) that just checks a list, if the top-most item on it is past the time delay then it sends a message to it and removes it then tests the next and repeats, else it inserts itself back into the stackless queue and waits for the next cycle to come by. A Process, when it has a timeout but no messages match, then it adds itself to the afore-mentioned list as a tuple of (timeTheTimeoutOccurs, processWaitChannel) then sorts it then blocks on its wait channel. If a message arrives at the process it tests the wait channel to see if it is blocked, if so it returns the message on that channel so it can then be tested, if it does not match then the process re-blocks, if it does match then it removes itself from the wait list and returns the message. If an exception occurs (taskletkill for example), it still properly removes itself from the list before it re-raises. Many other things done, but those are the major parts. All listed above is either working (near all of it) but may still need to be refined in its interface and so forth, or is partially done (pickling is more... hands-on currently, the interface described above is the new one I am partially moved to). The purpose of this was just to describe that erlangs strength can also be a strength for python as well. I really would like to move this library to PyPy when PyPy becomes usable. I have written that linked test above as well (not in this library, but pure stackless, a vastly different way then the article did though) and erlang still beat stackless cpython, hopefully pypy will fare better. On 8/5/07, Carl Friedrich Bolz wrote: > Hi Maciek > > Maciek Fijalkowski wrote: > > http://muharem.wordpress.com/2007/07/31/erlang-vs-stackless-python-a-first-benchmark/ > > > > Christian: with a dedication for you :) > > > > We should try pypy on this btw. > > seems a bit meaningless, given that one of erlang's most important > strengths is the possibility of using it to transparently across > multiple processes and especially multiple machines. > > Cheers, > > Carl Friedrich > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From fijal at genesilico.pl Fri Aug 10 11:53:25 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Fri, 10 Aug 2007 11:53:25 +0200 Subject: [pypy-dev] rffi feature request In-Reply-To: <20070808091440.f8e06de1.simon@arrowtheory.com> References: <20070803121413.bd2d992a.simon@arrowtheory.com> <20070804181403.f15cdbb6.simon@arrowtheory.com> <46B86274.2040401@genesilico.pl> <20070808091440.f8e06de1.simon@arrowtheory.com> Message-ID: <46BC3595.10606@genesilico.pl> > well, the above code would produce: > > extern int foo(int i, int j) > { > return i+j; > } > > (and perhaps an accompanying .h file) > > thereby providing an interface for other C programs. > This is rffi producing rather than consuming a C interface. > > Simon. > Hey Simon. It's doable, and not even hard, but it requires work. I kind of don't see the use-case for me right now (but I perfectly see the use-case in general), so it's a bit unlikely that I'll try to implement it, although I might help you a bit. Cheers, fijal :. From anto.cuni at gmail.com Sat Aug 11 12:02:42 2007 From: anto.cuni at gmail.com (Antonio Cuni) Date: Sat, 11 Aug 2007 12:02:42 +0200 Subject: [pypy-dev] rffi feature request In-Reply-To: <20070808091440.f8e06de1.simon@arrowtheory.com> References: <20070803121413.bd2d992a.simon@arrowtheory.com> <20070804181403.f15cdbb6.simon@arrowtheory.com> <46B86274.2040401@genesilico.pl> <20070808091440.f8e06de1.simon@arrowtheory.com> Message-ID: <46BD8942.2030309@gmail.com> Simon Burton wrote: >>>> I would like to expose some functions as external >>>> symbols when i build a .so >>>> >>>> def foo(i, j): >>>> return i+j >>>> >>>> foo._expose_ = [rffi.INT, rffi.INT] >>>> > well, the above code would produce: > > extern int foo(int i, int j) > { > return i+j; > } > > (and perhaps an accompanying .h file) > > thereby providing an interface for other C programs. > This is rffi producing rather than consuming a C interface. I think that what you need is similar to what carbonpython does: basically, carbonpython is a frontend for the translation toolchain that takes all the exposed functions/classes in an input file and produce a .NET libraries to be reused by other programs. Functions are exposed with the @export decorator: @export(int, int) def foo(a, b): return a+b The frontend creates a TranslationDriver objects, but instead of calling driver.setup() it calls driver.setup_library(), which allows to pass more than one entry point. I think all this could be reused for your needs. Then, the next step is to teach the backend how to deal with libraries; for genc, this would probably mean not to mangle functions names, to create a companying .h and to modify the Makefile to produce a .so instead of an executable. About name mangling: one possible solution could be to mangle the names as now and put some #define in the .h to allow the programmer to use non-mangled names. Probably this is the less invasive solution. ciao Anto From jason at jasondavies.com Mon Aug 13 00:29:29 2007 From: jason at jasondavies.com (Jason Davies) Date: Sun, 12 Aug 2007 23:29:29 +0100 Subject: [pypy-dev] Lua Frontend Message-ID: <46BF89C9.8000309@jasondavies.com> Hello, A couple of years ago I wrote a JIT compiler for Lua [1] for my final-year dissertation. Looking through the project-ideas page [2] I noticed there might be interest in writing Lua interpreter using PyPy. I'd be interested in doing this as a personal project. How would I go about getting started? Jason [1] http://www.lua.org/ [2] http://codespeak.net/pypy/dist/pypy/doc/project-ideas.html#write-a-new-front-end From gbowyer at fastmail.co.uk Mon Aug 13 16:49:41 2007 From: gbowyer at fastmail.co.uk (Greg Bowyer) Date: Mon, 13 Aug 2007 15:49:41 +0100 Subject: [pypy-dev] Silly idea - Pypy on Xen Message-ID: Taking a look at this http://www.informationweek.com/news/showArticle.jhtml?articleID=201311330&pgno=1&queryText= I wonder if the same could be done for pypy (this is just high level thinking, nothing serious) From fijal at genesilico.pl Mon Aug 13 17:21:26 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Mon, 13 Aug 2007 17:21:26 +0200 Subject: [pypy-dev] Silly idea - Pypy on Xen In-Reply-To: References: Message-ID: <46C076F6.1090707@genesilico.pl> Greg Bowyer wrote: > Taking a look at this > http://www.informationweek.com/news/showArticle.jhtml?articleID=201311330&pgno=1&queryText= > > I wonder if the same could be done for pypy (this is just high level > thinking, nothing serious) > > Look at unununium project (dead for quite a while), they tried to have a python shell as a basic thing for an operating system and had quite a lot of nice concepts. Right now obvious approach would be to target some virtualised environment instead of hardware itself, so it's not something very new. There are questions what abstraction do you provide for hard drive, etc. etc. What pypy has unique here is RPython, which can easily map low level details onto high level. Just few thoughts Cheers, fijal :. From cfbolz at gmx.de Mon Aug 13 17:23:36 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 13 Aug 2007 17:23:36 +0200 Subject: [pypy-dev] Silly idea - Pypy on Xen In-Reply-To: <46C076F6.1090707@genesilico.pl> References: <46C076F6.1090707@genesilico.pl> Message-ID: <46C07778.6050009@gmx.de> Maciek Fijalkowski wrote: > Greg Bowyer wrote: >> Taking a look at this >> http://www.informationweek.com/news/showArticle.jhtml?articleID=201311330&pgno=1&queryText= >> >> I wonder if the same could be done for pypy (this is just high level >> thinking, nothing serious) >> >> > Look at unununium project (dead for quite a while), they tried to have a > python shell as a basic thing for an operating system and had quite a > lot of nice concepts. Right now obvious approach would be to target some > virtualised environment instead of hardware itself, so it's not > something very new. There are questions what abstraction do you provide > for hard drive, etc. etc. > > What pypy has unique here is RPython, which can easily map low level > details onto high level. Yes, I kind of agree. IF you want to write an OS in Python then it probably makes most sense to target some virtualized environment and you might want to use RPython. Of course it depends very much what you actually want :-). Cheers, Carl Friedrich From fijal at genesilico.pl Mon Aug 13 17:28:46 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Mon, 13 Aug 2007 17:28:46 +0200 Subject: [pypy-dev] Silly idea - Pypy on Xen In-Reply-To: <46C07778.6050009@gmx.de> References: <46C076F6.1090707@genesilico.pl> <46C07778.6050009@gmx.de> Message-ID: <46C078AE.50508@genesilico.pl> Carl Friedrich Bolz wrote: > Maciek Fijalkowski wrote: >> Greg Bowyer wrote: >>> Taking a look at this >>> http://www.informationweek.com/news/showArticle.jhtml?articleID=201311330&pgno=1&queryText= >>> >>> >>> I wonder if the same could be done for pypy (this is just high level >>> thinking, nothing serious) >>> >>> >> Look at unununium project (dead for quite a while), they tried to >> have a python shell as a basic thing for an operating system and had >> quite a lot of nice concepts. Right now obvious approach would be to >> target some virtualised environment instead of hardware itself, so >> it's not something very new. There are questions what abstraction do >> you provide for hard drive, etc. etc. >> >> What pypy has unique here is RPython, which can easily map low level >> details onto high level. > > Yes, I kind of agree. IF you want to write an OS in Python then it > probably makes most sense to target some virtualized environment and > you might want to use RPython. Of course it depends very much what you > actually want :-). > > Cheers, > > Carl Friedrich > > :. > Come on, OS is an outdated concept. Cheers, fijal :. From gbowyer at fastmail.co.uk Mon Aug 13 17:54:18 2007 From: gbowyer at fastmail.co.uk (Greg Bowyer) Date: Mon, 13 Aug 2007 16:54:18 +0100 Subject: [pypy-dev] Silly idea - Pypy on Xen In-Reply-To: <46C078AE.50508@genesilico.pl> References: <46C076F6.1090707@genesilico.pl> <46C07778.6050009@gmx.de> <46C078AE.50508@genesilico.pl> Message-ID: Maciek Fijalkowski wrote: > Carl Friedrich Bolz wrote: >> Maciek Fijalkowski wrote: >>> Greg Bowyer wrote: >>>> Taking a look at this >>>> http://www.informationweek.com/news/showArticle.jhtml?articleID=201311330&pgno=1&queryText= >>>> >>>> >>>> I wonder if the same could be done for pypy (this is just high level >>>> thinking, nothing serious) >>>> >>>> >>> Look at unununium project (dead for quite a while), they tried to >>> have a python shell as a basic thing for an operating system and had >>> quite a lot of nice concepts. Right now obvious approach would be to >>> target some virtualised environment instead of hardware itself, so >>> it's not something very new. There are questions what abstraction do >>> you provide for hard drive, etc. etc. >>> >>> What pypy has unique here is RPython, which can easily map low level >>> details onto high level. >> Yes, I kind of agree. IF you want to write an OS in Python then it >> probably makes most sense to target some virtualized environment and >> you might want to use RPython. Of course it depends very much what you >> actually want :-). >> >> Cheers, >> >> Carl Friedrich >> >> :. >> > Come on, OS is an outdated concept. > > Cheers, > fijal > > > :. > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > I was thinking the selling point might be RPython From cfbolz at gmx.de Mon Aug 13 18:06:50 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 13 Aug 2007 18:06:50 +0200 Subject: [pypy-dev] Silly idea - Pypy on Xen In-Reply-To: <46C078AE.50508@genesilico.pl> References: <46C076F6.1090707@genesilico.pl> <46C07778.6050009@gmx.de> <46C078AE.50508@genesilico.pl> Message-ID: <46C0819A.6000302@gmx.de> Maciek Fijalkowski wrote: >> Yes, I kind of agree. IF you want to write an OS in Python then it >> probably makes most sense to target some virtualized environment and >> you might want to use RPython. Of course it depends very much what you >> actually want :-). >> >> Cheers, >> >> Carl Friedrich >> >> :. >> > Come on, OS is an outdated concept. Come on, you need a name to call the "thing-that-cares-for-your-resources-and-does-stuff" :-). Cheers, Carl Friedrich From cfbolz at gmx.de Mon Aug 13 18:17:21 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 13 Aug 2007 18:17:21 +0200 Subject: [pypy-dev] Lua Frontend In-Reply-To: <46BF89C9.8000309@jasondavies.com> References: <46BF89C9.8000309@jasondavies.com> Message-ID: <46C08411.1080902@gmx.de> Hi Jason, Jason Davies wrote: > A couple of years ago I wrote a JIT compiler for Lua [1] for my > final-year dissertation. Is this on the web somewhere? How did it work? > Looking through the project-ideas page [2] I > noticed there might be interest in writing Lua interpreter using PyPy. I think it would be interesting, yes. On the other hand, to me it is unclear whether this Lua interpreter would get any users, since the typical Lua users embed the Lua interpreter into a bigger project, which wouldn't be easily possible with a PyPy-based Lua-VM. > I'd be interested in doing this as a personal project. How would I go > about getting started? I would try to write an interpreter mirroring Lua's bytecode. This way you don't have to care for the tedious work of parsing and bytecode-compiling at first. I guess it is best to start with implementing some of the data structures of Lua, most notably tables (other data structures, like strings and floats are probably simple to implement, they just need to box rpython-level strings and floats). I guess you can have a naive implementation of tables in the beginning, though. Then you can probably start by implementing a simple interpreter for Lua's bytecode (probably by first writing some code that reads in Lua's bytecode and stores it nicely accessible somehow). If you are serious about this project and want to develop it under PyPy's umbrella you should ask for commit rights (e.g. by sending Michael Hudson a mail, micahel at gmail.com ). Cheers, Carl Friedrich From fijal at genesilico.pl Tue Aug 14 18:16:16 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Tue, 14 Aug 2007 18:16:16 +0200 Subject: [pypy-dev] [pypy-svn] r45656 - in pypy/branch/pypy-more-rtti-inprogress: . module/posix rlib rpython rpython/lltypesystem rpython/lltypesystem/module rpython/module rpython/module/test rpython/ootypesystem/module rpython/test translator translator/c translator/c/src translator/c/test translator/sandbox translator/sandbox/test In-Reply-To: <20070814161046.E284280E8@code0.codespeak.net> References: <20070814161046.E284280E8@code0.codespeak.net> Message-ID: <46C1D550.8020508@genesilico.pl> arigo at codespeak.net wrote: > For backup purposes, in-progress: convert more of the os module to rtti > and try to get rid of the rllib.ros module by making the native > interfaces RPythonic. This looks quite good in my opinion - seems that > we've finally learned a reasonable way to do external functions. > > Hooray! I would suggest to be consistent in names. Either rename it to rtti or keep using rffi (I know that fonts basically suck when it comes to this :) Seriously - I would like to move a bit rsocket, shall I do this on the branch or on trunk? Cheers, fijal :. From cfbolz at gmx.de Tue Aug 14 21:42:12 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Tue, 14 Aug 2007 21:42:12 +0200 Subject: [pypy-dev] Paper about accurate GC in C Message-ID: <46C20594.6050506@gmx.de> Hi all, the following paper describes a variant of the root-finding approach we use for the framework GC: http://citeseer.ist.psu.edu/henderson02accurate.html They use a "shadow stack" as well, but keep all their GCed locals always in it: The tradeoffs are slightly different than ours. They reach performance quite similar to that of the Boehm GC with a simple semispace collector. Chris Lattner told me that this is how LLVM lowers GC operations currently (since no backend directly supports the GC primitives in a better way). Cheers, Carl Friedrich From cfbolz at gmx.de Tue Aug 14 21:53:12 2007 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Tue, 14 Aug 2007 21:53:12 +0200 Subject: [pypy-dev] [pypy-svn] r45656 - in pypy/branch/pypy-more-rtti-inprogress: . module/posix rlib rpython rpython/lltypesystem rpython/lltypesystem/module rpython/module rpython/module/test rpython/ootypesystem/module rpython/test translator translator/c translator/c/src translator/c/test translator/sandbox translator/sandbox/test In-Reply-To: <46C1D550.8020508@genesilico.pl> References: <20070814161046.E284280E8@code0.codespeak.net> <46C1D550.8020508@genesilico.pl> Message-ID: <46C20828.3010905@gmx.de> Maciek Fijalkowski wrote: > arigo at codespeak.net wrote: >> For backup purposes, in-progress: convert more of the os module to rtti >> and try to get rid of the rllib.ros module by making the native >> interfaces RPythonic. This looks quite good in my opinion - seems that >> we've finally learned a reasonable way to do external functions. >> >> > Hooray! Seconded: very cool! Thanks to all the people that are working on this (I am looking at you, Maciek :-) ). Cheers, Carl Friedrich From jgustak at gmail.com Fri Aug 17 14:35:20 2007 From: jgustak at gmail.com (Jakub Gustak) Date: Fri, 17 Aug 2007 14:35:20 +0200 Subject: [pypy-dev] scheme interpreter [status report] In-Reply-To: References: Message-ID: This is last scheme status report during SoC. So I will sum things up in the end. > Macros looks like working. > One thing to be added: they are acting like first-class objects, but should not. Found a bug, when ellipsis matched zero objects, fized now. Scheme48 acts differently in some cases. On the other hand in this cases mzScheme works just as expected. In this meaning macros are evil. There is no explicit specification how they should behave in r5rs. They look like added to the specification in last minute. Continuations: > Another issue: this implementations is stack-only so it introduce lot > of overhead both on capturing and throwing continuations. > > And one more: every W_Procedure is continuations-safe but VERY slow, > even when no capture occurs. I made them a little faster by decreasing number of operations, but still for every non-tail-call a new instance of ContinuationFrame is created and is stored on stack. On every return it is popped form stack and forgotten. When no capture will happen, it is time and memory waste. > I'd rather would like to think about how to throw continuations more > generally, and not have to implement continue_tr for every W_Callable, > than implementing capture differently. No progress in this direction. Still every W_Callable to be continuations friendly has to use eval_cf instead of eval and must implement continue_tr method. > Or maybe try totally different approach e.g write every call in CPS, > but it probably would require way too much changes right now. Nor here. Summarization: * Interactive interpreter (not RPythonic) lang/scheme/interactive.py * Can be translated using: translation/goal/targetscheme.py PyPy Scheme Interpreter features: * uses rpythonic packrat parser * every "syntax" definitions are implemented * quotations and quasi-quotations * delayed evaluation * proper tail-recursion * hygienic macros * partly working continuations (some macros are not continuations friendly yet). * no dynamic-wind procedure Known issues and "to be added": * lambda does not check if it is called with correct arguments number * macros are acting like first-class objects, though they shouldn't * missing input/output procedures * no support for chars, vectors * strings are supported, but no string-operations procedures * quite a few procedures are missing, but they can be easily added * no call-with-values and values * missing some "library syntax", can be defined using macros What's next? I will probably mysteriously disappear and became grave-digger or hermit. Never touch computer again and live happily ever after ;-). Cheers, Jakub Gustak From fijal at genesilico.pl Fri Aug 17 15:58:43 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Fri, 17 Aug 2007 15:58:43 +0200 Subject: [pypy-dev] scheme interpreter [status report] In-Reply-To: References: Message-ID: <46C5A993.4040901@genesilico.pl> Jakub Gustak wrote: > This is last scheme status report during SoC. So I will sum things up > in the end. > > Summarization: > * Interactive interpreter (not RPythonic) lang/scheme/interactive.py > * Can be translated using: translation/goal/targetscheme.py > > PyPy Scheme Interpreter features: > * uses rpythonic packrat parser > * every "syntax" definitions are implemented > * quotations and quasi-quotations > * delayed evaluation > * proper tail-recursion > * hygienic macros > * partly working continuations (some macros are not continuations friendly yet). > * no dynamic-wind procedure > > > I would say congratulations! (Although I don't know scheme that much to be able to evaluate this, even positively) Cheers, fijal :. From arigo at tunes.org Fri Aug 17 20:03:30 2007 From: arigo at tunes.org (Armin Rigo) Date: Fri, 17 Aug 2007 20:03:30 +0200 Subject: [pypy-dev] scheme interpreter [status report] In-Reply-To: References: Message-ID: <20070817180330.GA5301@code0.codespeak.net> Hi Jakub, On Fri, Aug 17, 2007 at 02:35:20PM +0200, Jakub Gustak wrote: > This is last scheme status report during SoC. So I will sum things up > in the end. Thanks for the good work you did during the summer! The result looks good to me, even with the missing features, but we'd really need a Scheme guy to guide us from now on, if we want to make the interpreter more usable. > > I'd rather would like to think about how to throw continuations more > > generally, and not have to implement continue_tr for every W_Callable, > > than implementing capture differently. > > No progress in this direction. > Still every W_Callable to be continuations friendly has to use eval_cf > instead of eval and must implement continue_tr method. > > > Or maybe try totally different approach e.g write every call in CPS, > > but it probably would require way too much changes right now. > > Nor here. Indeed, the current approach allows impressive-looking tests to pass, but the code is quite verbose and inefficient. We can start thinking at some point about using the stackless and/or GC-based cloning operations of RPython as a more orthogonal alternative, but that's an involved topic. > What's next? > I will probably mysteriously disappear and became grave-digger or > hermit. Never touch computer again and live happily ever after ;-). If you do, then farewell! In the event that you're interested to continue, though (:-), here are in summary some possible future directions: * get some Scheme people interested in the project, try to run real Scheme projects * implement more of the missing features * think how to do continuations with stackless or GC-based cloning * also, we could try to integrate pypy/lang/scheme and pypy/interpreter in one of the many possible ways, to offer a combined Python+Scheme environment... A bientot, Armin. From fijal at genesilico.pl Sat Aug 18 13:26:55 2007 From: fijal at genesilico.pl (Maciek Fijalkowski) Date: Sat, 18 Aug 2007 13:26:55 +0200 Subject: [pypy-dev] PyPy work plan Message-ID: <46C6D77F.3020807@genesilico.pl> As you probably have observed already, for the last few months pypy is slowly approaching "being usable" state instead of "having more ultra-cool features", as we've got enough of them to be willing to use them. As we're moving towards maintaining pypy as an open source project I would suggest to look around parts of pypy and have list of people who are willing to maintain certain parts. Maintain not in a sense of developing those, but rather accepting/rejecting patches, keeping it up to date with other parts, etc. I would also suggest to move parts with no obvious maintainer somewhere else in svn (branch? tag?) not to confuse people who checkout whole svn repository with parts never used by anyone. This option would make it easier to: * have people entering project, since it would be obvious who to contact for different parts. * maintain whole codebase, since some cleanup changes are pervasive enough to break different parts. * this would also hopefully reduce whole code size. I've also checked in pypy parts (more or less) into http://codespeak.net/svn/pypy/dist/pypy/doc/maintainers.txt feel free to add yourself wherever you like and to modify this list as well What do you think? Cheers, fijal :. From pedronis at openendsystems.com Sat Aug 18 17:08:34 2007 From: pedronis at openendsystems.com (Samuele Pedroni) Date: Sat, 18 Aug 2007 17:08:34 +0200 Subject: [pypy-dev] PyPy work plan In-Reply-To: <46C6D77F.3020807@genesilico.pl> References: <46C6D77F.3020807@genesilico.pl> Message-ID: <46C70B72.4050704@openendsystems.com> Maciek Fijalkowski wrote: > > I've also checked in pypy parts (more or less) into > http://codespeak.net/svn/pypy/dist/pypy/doc/maintainers.txt > feel free to add yourself wherever you like and to modify this list as well > I looked at it, didn't really know how to fill it in without being confused or confusing. I removed it. > What do you think? > I think identifying orphaned areas, agree on them and act accordingly is useful. I suppose there are indeed specific areas with specific maintainers, I suppose listing those may be useful. Most of PyPy is too interconnected and demanding for anything but global ownership to make sense tough, at least at this point in time. My check-ins on the 4-5 of August are a case in point. From lac at openend.se Sat Aug 18 17:24:27 2007 From: lac at openend.se (Laura Creighton) Date: Sat, 18 Aug 2007 17:24:27 +0200 Subject: [pypy-dev] PyPy work plan In-Reply-To: Message from Maciek Fijalkowski of "Sat, 18 Aug 2007 13:26:55 +0200." <46C6D77F.3020807@genesilico.pl> References: <46C6D77F.3020807@genesilico.pl> Message-ID: <200708181524.l7IFORYH028243@theraft.openend.se> Hi Simon. I never like to quote somebody's email without informing them. I am about to use ICCARUS (why 2 Cs) as an example of what I am looking for. The rest is the middle of a discusison that you are not expected to understand. Thank you for ICCARUS. -- Laura In a message of Sat, 18 Aug 2007 13:26:55 +0200, Maciek Fijalkowski writes: > >As you probably have observed already, for the last few months pypy >is slowly approaching "being usable" state instead of "having more >ultra-cool >features", as we've got enough of them to be willing to use them. >As we're moving towards maintaining pypy as an open source project I woul >d >suggest to look around parts of pypy and have list >of people who are willing to maintain certain parts. Maintain not in a se >nse >of developing those, but rather accepting/rejecting patches, keeping it u >p >to date with other parts, etc. I would also suggest >to move parts with no obvious maintainer somewhere else in svn >(branch? tag?) not to confuse people who checkout whole svn repository >with parts never used by anyone. This option would make it easier to: > >* have people entering project, since it would be obvious who to > contact for different parts. > >* maintain whole codebase, since some cleanup changes are pervasive > enough to break different parts. > >* this would also hopefully reduce whole code size. > >I've also checked in pypy parts (more or less) into >http://codespeak.net/svn/pypy/dist/pypy/doc/maintainers.txt >feel free to add yourself wherever you like and to modify this list as we >ll > >What do you think? > >Cheers, >fijal As mentioned just in irc, I am unhappy with the dividing up of pypy into pieces with maintainers. That way the people who envision the small can dominate those who envision the large, and one of the strengths of this project is the sheer visionary power of some of its members. on the other hand, some tidying would be nice. and we sure as hell could be friendlier for newbies. I am now looking at code visualisation tools, and time visualisation tools. I get the idea that if we could visualise things better, we could undersgtand who knows what about what, and what else, then we would need less in the way of 'task management'. for static things I am thinking of: http://simile.mit.edu/timeline/ or everything at the top level simile.mit.edu domain. But then yesterday I read this on the pygame mailing list. ..... From: "Simon Wittber" To: pygame-users at seul.org Subject: [pygame] 3D Social Network Visualisation A few evening ago we demonstrated ICCARUS at a web party. It is a piece of software which provides a visualisation of social networks in three dimensions :-) It uses pygame and a Pyrexed OpenGL lib called GFX. You can see screencast here: http://scouta.com/faves/8raO17jT8Y3/ and a vid of the presentation here: http://blog.viddler.com/cdevroe/webjam-iccarus/ Thanks to python and pygame, this project was conceived and implemented over 3 days. ...... I thought this was way cool. Especially the 3 days part. I want our code base to work like this. Who knows what? stuff that everybody knows, stuff that carl freidrich and armin and maciek and samuele know, stuff that nobody knows but samuele, and even -- 'stuff that nobody knows' somebody did at one time but we all have forgotten by now. This is a half-baked proposal for code base management, normally I would not mention this at all, but some of us are getting grumpy, and I thought that other ideas, however weird might be welcome. Laura From lac at openend.se Sat Aug 18 17:30:01 2007 From: lac at openend.se (Laura Creighton) Date: Sat, 18 Aug 2007 17:30:01 +0200 Subject: [pypy-dev] scheme interpreter [status report] In-Reply-To: Message from Armin Rigo of "Fri, 17 Aug 2007 20:03:30 +0200." <20070817180330.GA5301@code0.codespeak.net> References: <20070817180330.GA5301@code0.codespeak.net> Message-ID: <200708181530.l7IFU1k8029132@theraft.openend.se> >> What's next? >> I will probably mysteriously disappear and became grave-digger or >> hermit. Never touch computer again and live happily ever after ;-). This sort of peace is good for 2 weeks, a month, even a year in severe burn-out cases. UNless you plan on being struck by lightning on your vacation in the hermitage or graveyard, I would expect the technical world to grab its claw around you once again. Know that you are welcome here whenever this happens. :-) Laura From santagada at gmail.com Sun Aug 19 00:22:22 2007 From: santagada at gmail.com (Leonardo Santagada) Date: Sat, 18 Aug 2007 19:22:22 -0300 Subject: [pypy-dev] Javascript Frontend (and mybe some unrelated future plans) Message-ID: I've been on a cave for some time, but I want to get back to work more on pypy but I don't know what to do with the javascript interpreter. While I was working with it it was getting better but almost all the architecture work i've done alone and I'm not happy with the results. The code looks kinda ugly and I don't really know how to get it into a good shape. Maybe I need someone to take a look at it. Also there is some stuff that I don't know what to do about like automatic semicolon insertion and regexes. And them there is the stuff that I really think is essential but that is missing wich is a simple RPython API to make native objects to javascript (that we could use to implement DOM, Regexes and XMLHttpRequests using rsockets). All those are internal architectural problems, them there are the external ones like, who would like to use our javascript interpreter? Web javascript guys for testing and to make their tools using js, or people wanting to know how to make an interpreter on top of pypy (people always says that a scheme interpreter would be easier for that). So those are my troubles with the js interpreter, as it is now I don't know what it adds to the pypy project. But I'm always a mail away and I still track the development of pypy and can maintain it. I still want to work more with pypy, just don't know what to do with the js frontend now. Maybe I should take another part of pypy to work with, I want to know what you guys have to say about all that i've written. As always I want to express my gratitude to everyone as you all really helped me a lot. I learned so much working on pypy, I don't know what to say but thanks. best wishes, -- Leonardo Santagada From arigo at tunes.org Sun Aug 19 12:33:09 2007 From: arigo at tunes.org (Armin Rigo) Date: Sun, 19 Aug 2007 12:33:09 +0200 Subject: [pypy-dev] pypy-dev@codespeak.net Message-ID: <20070819103308.GA28257@code0.codespeak.net> Hi all, Those that follow IRC already know it, but it's worth being announced a bit more widely: I've been working on a form of sandboxing for RPython programs, which now seems to work for the whole of PyPy. It's "sandboxing" as in "full virtualization", but done in normal C with no OS support at all. It's a two-processes model: we can translate PyPy to a special "pypy-c-sandbox" executable, which is safe in the sense that it doesn't do any library or system call - instead, whenever it would like to perform such an operation, it marshals the operation name and the arguments to its stdout and it waits for the marshalled result on its stdin. This pypy-c-sandbox process is meant to be run by an outer "controller" program that answers to these operation requests. The pypy-c-sandbox program is obtained by adding a transformation during translation, which turns all RPython-level external function calls into stubs that do the marshalling/waiting/unmarshalling. An attacker that tries to escape the sandbox is stuck within a C program that contains no external function call at all except to write to stdout and read from stdin. (It's still attackable, e.g. by exploiting segfault-like situations, but as far as I can tell - unlike CPython - any RPython program is really robust against this kind of attack, at least if we enable the extra checks that all RPython list and string indexing are in range. Alternatively, on Linux there is a lightweight OS-level sandboxing technique available by default - google for 'seccomp'.) The outer controller is a plain Python program that can run in CPython or a regular PyPy. It can perform any virtualization it likes, by giving the subprocess any custom view on its world. For example, while the subprocess thinks it's using file handles, in reality the numbers are created by the controller process and so they need not be (and probably should not be) real OS-level file handles at all. In the demo controller I've implemented there is simply a mapping from numbers to file-like objects. The controller answers to the "os_open" operation by translating the requested path to some file or file-like object in some virtual and completely custom directory hierarchy. The file-like object is put in the mapping with any unused number >= 3 as a key, and the latter is returned to the subprocess. The "os_read" operation works by mapping the pseudo file handle given by the subprocess back to a file-like object in the controller, and reading from the file-like object. Enough explanations, here's a how-to: For now, this lives in a branch at the 'pypy' level: cd ..../pypy-dist/pypy svn switch http://codespeak.net/svn/pypy/branch/pypy-more-rtti-inprogress/ In pypy/translator/goal: ./translate.py --sandbox --source targetpypystandalone.py --withoutmod-gc --withoutmod-_weakref (the gc and _weakref modules are disabled because they are a bit too unsafe, in the sense that they could allow bogus memory accesses) Then in the directory whe