From arigo at tunes.org Sat Jan 1 18:24:26 2011 From: arigo at tunes.org (Armin Rigo) Date: Sat, 1 Jan 2011 18:24:26 +0100 Subject: [pypy-dev] 1.4.1 threading In-Reply-To: References: Message-ID: Hi Paolo, Although I'm sure that your post is very interesting with the right mindset, I have a problem understanding the connection with Python. I know that it's exactly what you are complaining about, i.e. that Python developers appear not to care much about this deep Java/C++ issue. However, if you mean to do anything about it, you need to first understand yourself why it is so -- and the following reason is by far not enough: > As a matter of fact, this work is unsurprisingly often totally ignored > in many discussions about Python's memory model. I'm not surprised > because it's complex stuff, and understanding it doesn't matter for > today's Python programming. The most obvious misunderstanding, in my opinion, is that in Java or C++ it's about fields reads and writes, whereas in Python it's about any operation on any built-in type -- which can be for example reading or writing attributes, or insert()ing in a list, or doing setdefault() on a dictionary, or... any complex operation. This means that you cannot *at all* reduce the problem to field reads and field writes. As long as the discussion at the level of Java or C++ is only about fields, it's going to be ignored as "mostly uninteresting" by the Python folks. On the other hand, the links you posted about nonblockinghashmap are indeed interesting in this context. From a Python point of view, what is left to discuss is whether these hash maps offer enough "consistency" behavior to be usable on Python's default memory model. Or maybe it's not that interesting any more now that PyPy tends not to use a lot of dictionaries any more (the attributes of objects are not in a hash map, internally). I am myself just expressing vague, uninformed opinions again. But the final point is that this really needs someone motivated to experiment with PyPy (or CPython), and as long as no-one shows up, it will mostly be just "moving air around", i.e. wasting time to discuss this too much in depth. A bient?t, Armin. From tobami at googlemail.com Sun Jan 2 11:24:03 2011 From: tobami at googlemail.com (Miquel Torres) Date: Sun, 2 Jan 2011 11:24:03 +0100 Subject: [pypy-dev] funding/popularity? In-Reply-To: References: Message-ID: Hi all, I had a look at the issue and could finally pin it (more or less). Codespeed does destroy and recreate the whole element containing the canvas, so it is NOT a but in jqPlot (nor in Codespeed). It is a tricky GC issue where memory taken by big canvases is not properly freed. It probably affects several bowsers with modern JS JITs. Actually, in Firefox 3.6 it is so bad that not even closing the tab frees up the memory for a while! Chrome exhibits similar behaviour, but it does free old canvases after 20-40s. I am sure you will be interested in the fact that Konqueror 4.4 (with KHTML and kjs), which probably doesn't have such an aggressive JIT (or no JIT at all?), does NOT suffer this problem. Every refresh properly frees up the memory of the old canvas. For anyone further interested, Dmtrii did an excellent job filling a new bug for it with simple example code that triggers the problem (I added my observations there): https://bugzilla.mozilla.org/show_bug.cgi?id=621416 So what about speed.pypy.org? It being a multi-browser GC bug, I did the only thing I could do: limit the size of the sometimes ridiculously big horizontal plots ;-) It now has a 2000px limit. It does not completely fix the issue, but it is much more bearable. In the first 2 refreshes (selecting or -deselecting benchmarks), FF even manages to free up the memory. After that it keeps on eating more, but its much better. Thanks Dmitrii for caring! :-D (Please tell me whether it got better for you.) Cheers, Miquel 2010/12/23 Armin Rigo : > Hi Paolo, > > On Thu, Dec 23, 2010 at 3:09 PM, Paolo Giarrusso wrote: >> From your description, I guess that what you describe as "raw memory" >> is not accounted in the stats triggering GC. That would explain the >> behavior you describe, and suggest an easy fix. Indeed, one could wrap >> the raw-memory allocator to create and update such stats; then the >> heap-overflow check could consider them to decide whether to trigger >> GC. The details of the modified heap-overflow check would probably not >> be entirely trivial, but still doable. > > Absolutely. ?I will write it down in extradoc/planning/ for now. > > > Armin > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From no-reply at dropboxmail.com Mon Jan 3 23:05:05 2011 From: no-reply at dropboxmail.com (Dropbox) Date: Mon, 03 Jan 2011 22:05:05 +0000 Subject: [pypy-dev] Wing Sit invited you to Dropbox Message-ID: <20110103220505.DB30E3F550D@mailman-2.dropboxmail.com> Wing Sit wants you to use Dropbox to sync and share files online and across computers. Get started here: http://www.dropbox.com/link/20.eHuq1oJQCM/NjU2NTA1ODEwNw?src=referrals_bulk2 - The Dropbox Team ____________________________________________________ To stop receiving invites from Dropbox, please go to http://www.dropbox.com/bl/4b1a4ffd25d4/pypy-dev%40codespeak.net -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110103/7d421d6a/attachment.htm From dmalcolm at redhat.com Mon Jan 3 23:13:38 2011 From: dmalcolm at redhat.com (David Malcolm) Date: Mon, 03 Jan 2011 17:13:38 -0500 Subject: [pypy-dev] PyPy is now available in Fedora Message-ID: <1294092818.26496.64.camel@radiator.bos.redhat.com> I've packaged pypy in RPM form for the Fedora distribution [1] - RPM packages are now built in the development branch targeting the next major release (Fedora 15). So it should now be possible for Fedora users to type # yum install pypy and obtain a precompiled /usr/bin/pypy executable via an rpm package, consisting of the JIT-enabled pypy, with all standard settings, without needing to do the full build themselves. I'll do my best to keep these "downstream" packages promptly updated as further "upstream" releases of PyPy occur. Many thanks to everyone who helped with this, especially fijal, for all his great feedback on the packaging review [2] (Caveat: actually, the build won't be available yet via "yum" until a cronjob runs tonight; for now, the build can be downloaded from: http://koji.fedoraproject.org/koji/buildinfo?buildID=212473 ) Some other links, for the curious, can be seen here: https://admin.fedoraproject.org/pkgdb/acls/name/pypy e.g. to the rpm packaging sources. Hope this is helpful Dave [1] http://fedoraproject.org/ [2] https://bugzilla.redhat.com/show_bug.cgi?id=588941 From fijall at gmail.com Tue Jan 4 08:48:09 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 4 Jan 2011 09:48:09 +0200 Subject: [pypy-dev] PyPy is now available in Fedora In-Reply-To: <1294092818.26496.64.camel@radiator.bos.redhat.com> References: <1294092818.26496.64.camel@radiator.bos.redhat.com> Message-ID: Good job! On Tue, Jan 4, 2011 at 12:13 AM, David Malcolm wrote: > I've packaged pypy in RPM form for the Fedora distribution [1] - RPM > packages are now built in the development branch targeting the next > major release (Fedora 15). > > So it should now be possible for Fedora users to type > ?# yum install pypy > and obtain a precompiled > ?/usr/bin/pypy > executable via an rpm package, consisting of the JIT-enabled pypy, with > all standard settings, without needing to do the full build themselves. > > I'll do my best to keep these "downstream" packages promptly updated as > further "upstream" releases of PyPy occur. > > Many thanks to everyone who helped with this, especially fijal, for all > his great feedback on the packaging review [2] > > > (Caveat: actually, the build won't be available yet via "yum" until a > cronjob runs tonight; for now, the build can be downloaded from: > ?http://koji.fedoraproject.org/koji/buildinfo?buildID=212473 ) > > Some other links, for the curious, can be seen here: > ?https://admin.fedoraproject.org/pkgdb/acls/name/pypy > e.g. to the rpm packaging sources. > > Hope this is helpful > Dave > [1] http://fedoraproject.org/ > [2] https://bugzilla.redhat.com/show_bug.cgi?id=588941 > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From Ben.Young at sungard.com Tue Jan 4 09:40:31 2011 From: Ben.Young at sungard.com (Ben.Young at sungard.com) Date: Tue, 4 Jan 2011 08:40:31 +0000 Subject: [pypy-dev] pypy GC on large objects Re: funding/popularity? In-Reply-To: References: Message-ID: <01781CA2CC22B145B230504679ECF48C02AA1FF7@EMEA-EXCHANGE03.internal.sungard.corp> > -----Original Message----- > From: pypy-dev-bounces at codespeak.net [mailto:pypy-dev- > bounces at codespeak.net] On Behalf Of Paolo Giarrusso > Sent: 24 December 2010 11:39 > To: Dima Tisnek > Cc: PyPy Dev; Armin Rigo > Subject: Re: [pypy-dev] pypy GC on large objects Re: > funding/popularity? > > On Thu, Dec 23, 2010 at 20:30, Dima Tisnek wrote: > > Basically collecting this is hard: > > > > dict(a=range(9**9)) > > > > large list is referenced, the object that holds the only reference is > > small no matter how you look at it. > First, usually (in most GC-ed languages) you can collect the list > before the dict. In PyPy, if finalizers are involved (is this the case > here? That'd be surprising), this is no more true. > > However, object size is not the point. For standard algorithms, the > size of an object does not matter at all in deciding when it's > collected - I already discussed this in my other email in this thread, > and I noted what actually could happen in the examples described by > Armin, and your examples show that it is a good property. A large > object in the same heap can fill it up and trigger an earlier garbage > collection. > > In general, if GC ran in the background (but it usually doesn't, and > not in PyPy) it could make sense to free objects sooner or later, > depending not on object size, but on "how much memory would be > 'indirectly freed' by freeing this object". However, because of > sharing, answering this question is too complex (it requires > collecting data from the whole heap). Moreover, the whole thing makes > no sense at all with usual, stop-the-world collectors: the app is > stopped, then the whole young generation, or the whole heap, is > collected, then the app is resumed. > > When separate heaps are involved (such as with ctypes, or with Large > Object Spaces, which avoid using a copy collector for large objects), > it is more complicated to ensure that the same property holds: you > need to consider stats of all heaps to decide whether to trigger GC. > > > I guess it gets harder still if there are many small live objects, as > > getting to this dict takes a while > > (easier in this simple case with generataional collector, O(n) in > general case) > > Not sure what you mean; I can make sense of it (not fully) only with > an incremental collector, and they are still used seldom (especially, > not in PyPy). > > Best regards > > > On 23 December 2010 06:38, Armin Rigo wrote: > >> Hi Ren?, > >> > >> On Thu, Dec 23, 2010 at 2:33 PM, Ren? Dudfield > wrote: > >>> I think this is a case where the object returned by > >>> ctypes.create_string_buffer() could use a correct __sizeof__ method > >>> return value. ?If pypy supported that, then the GC's could support > >>> extensions, and 'opaque' data structures in C too a little more > >>> nicely. > >> > >> I think you are confusing levels. ?There is no way the GC can call > >> some app-level Python method to get information about the objects it > >> frees (and when would it even call it?). ?Remember that our GC is > >> written at a level where it works for any interpreter for any > >> language, not just Python. > >> .NET supports calls to GC.AddMemoryPressure and GC.RemoveMemoryPressure to inform the GC you are allocating things outside of its knowledge. Maybe something similar would help? Cheers, Ben > >> > >> A bient?t, > >> > >> Armin. > >> _______________________________________________ > >> pypy-dev at codespeak.net > >> http://codespeak.net/mailman/listinfo/pypy-dev > >> > > _______________________________________________ > > pypy-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/pypy-dev > > > > -- > Paolo Giarrusso - Ph.D. Student > http://www.informatik.uni-marburg.de/~pgiarrusso/ > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev From p.giarrusso at gmail.com Tue Jan 4 15:50:50 2011 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Tue, 4 Jan 2011 15:50:50 +0100 Subject: [pypy-dev] pypy GC on large objects Re: funding/popularity? In-Reply-To: <01781CA2CC22B145B230504679ECF48C02AA1FF7@EMEA-EXCHANGE03.internal.sungard.corp> References: <01781CA2CC22B145B230504679ECF48C02AA1FF7@EMEA-EXCHANGE03.internal.sungard.corp> Message-ID: On Tue, Jan 4, 2011 at 09:40, wrote: > > >> -----Original Message----- >> From: pypy-dev-bounces at codespeak.net [mailto:pypy-dev- >> bounces at codespeak.net] On Behalf Of Paolo Giarrusso >> Sent: 24 December 2010 11:39 >> To: Dima Tisnek >> Cc: PyPy Dev; Armin Rigo >> Subject: Re: [pypy-dev] pypy GC on large objects Re: >> funding/popularity? >> >> On Thu, Dec 23, 2010 at 20:30, Dima Tisnek wrote: >> > Basically collecting this is hard: >> > >> > dict(a=range(9**9)) >> > >> > large list is referenced, the object that holds the only reference is >> > small no matter how you look at it. >> First, usually (in most GC-ed languages) you can collect the list >> before the dict. In PyPy, if finalizers are involved (is this the case >> here? That'd be surprising), this is no more true. >> >> However, object size is not the point. For standard algorithms, the >> size of an object does not matter at all in deciding when it's >> collected - I already discussed this in my other email in this thread, >> and I noted what actually could happen in the examples described by >> Armin, and your examples show that it is a good property. A large >> object in the same heap can fill it up and trigger an earlier garbage >> collection. >> >> In general, if GC ran in the background (but it usually doesn't, and >> not in PyPy) it could make sense to free objects sooner or later, >> depending not on object size, but on "how much memory would be >> 'indirectly freed' by freeing this object". However, because of >> sharing, answering this question is too complex (it requires >> collecting data from the whole heap). Moreover, the whole thing makes >> no sense at all with usual, stop-the-world collectors: the app is >> stopped, then the whole young generation, or the whole heap, is >> collected, then the app is resumed. >> >> When separate heaps are involved (such as with ctypes, or with Large >> Object Spaces, which avoid using a copy collector for large objects), >> it is more complicated to ensure that the same property holds: you >> need to consider stats of all heaps to decide whether to trigger GC. >> >> > I guess it gets harder still if there are many small live objects, as >> > getting to this dict takes a while >> > (easier in this simple case with generataional collector, O(n) in >> general case) >> >> Not sure what you mean; I can make sense of it (not fully) only with >> an incremental collector, and they are still used seldom (especially, >> not in PyPy). >> >> Best regards >> >> > On 23 December 2010 06:38, Armin Rigo wrote: >> >> Hi Ren?, >> >> >> >> On Thu, Dec 23, 2010 at 2:33 PM, Ren? Dudfield >> wrote: >> >>> I think this is a case where the object returned by >> >>> ctypes.create_string_buffer() could use a correct __sizeof__ method >> >>> return value. ?If pypy supported that, then the GC's could support >> >>> extensions, and 'opaque' data structures in C too a little more >> >>> nicely. >> >> >> >> I think you are confusing levels. ?There is no way the GC can call >> >> some app-level Python method to get information about the objects it >> >> frees (and when would it even call it?). ?Remember that our GC is >> >> written at a level where it works for any interpreter for any >> >> language, not just Python. >> >> > > > .NET supports calls to GC.AddMemoryPressure and GC.RemoveMemoryPressure to inform the GC you are allocating things outside of its knowledge. Maybe something similar would help? That's interesting as well. I and Armin discussed something similar in another branch of this thread, and he included that among planned ideas: http://codespeak.net/pipermail/pypy-dev/2010q4/006648.html http://codespeak.net/pipermail/pypy-dev/2010q4/006649.html The difference is that in my proposal one would hook the memory allocator for Python extensions, the .NET requires adding explicit calls to the source code. However, the key idea is that you might need to GC sooner if there is lots of unmanaged memory. Unfortunately, MSDN docs about those methods do not give pointers to the heuristics used: http://msdn.microsoft.com/en-us/library/system.gc.addmemorypressure.aspx http://msdn.microsoft.com/en-us/library/system.gc.removememorypressure.aspx Best regards -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From duncan at grisby.org Tue Jan 4 18:25:36 2011 From: duncan at grisby.org (Duncan Grisby) Date: Tue, 04 Jan 2011 17:25:36 +0000 Subject: [pypy-dev] Calling into PyPy from multi-threaded C++ Message-ID: <1294161936.1893.20.camel@localhost> Hi, I've been experimenting with using PyPy in a big application, and the initial results are very encouraging. One part of the application is a fairly complex compiler, and that goes 2.5 times faster with PyPy than with Python 2.6, which is a very positive result. The compiler is the one CPU-intensive bit of the system that I could easily isolate to test with PyPy. All the rest of it consists of a number of servers connected together using omniORB (which I maintain). omniORBpy integrates the C++ core of omniORB with Python, via the C API. omniORB is multi-threaded, and needs to call into Python from threads created in the C++ world. With CPython that's possible, although it takes some care to behave properly with thread states and the interpreter lock and so on. As far as I can see, PyPy's C API implementation doesn't support the thread state functions like PyGILState_Ensure and PyThreadState_Get, which isn't a huge surprise given how different PyPy is. Are these functions likely to be implemented at any point? Alternatively, is there any other way for C/C++ code to call into PyPy from a thread created outside PyPy? If there isn't a mechanism to do it at the moment, is it possible at all? How much effort would it be to create one? Thanks, Duncan. -- -- Duncan Grisby -- -- duncan at grisby.org -- -- http://www.grisby.org -- From alex.gaynor at gmail.com Wed Jan 5 21:51:18 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Wed, 5 Jan 2011 14:51:18 -0600 Subject: [pypy-dev] psycopg2 on PyPy Message-ID: Hi all, For the past week and a half I've been working on an RPython implementation of psycopg2 (the most popular database adapter for PostgreSQL). I'm happy to announce that at this point it is passing all the DB-API2.0 tests, and all the psycopg2 tests I converted into AppTests (most of them I think), it also passes the entire DJango test suite (which makes extensive use of the database). The downside is that running the full Django test suite on top of pypy with it is still about 20% slower than CPython. So we have some work to do. The code is all at: http://bitbucket.org/alex_gaynor/pypy-postgresql, I don't plan on merging this back into the main dev tree, as it doesn't seem appropriate for pypy-core to have a PostgreSQL adapter, rather I'd like to spend some time looking at seperate compilation, so we can have external RPython modules. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero "Code can always be simpler than you think, but never as simple as you want" -- Me -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110105/00ec839e/attachment.htm From santagada at gmail.com Thu Jan 6 03:29:12 2011 From: santagada at gmail.com (Leonardo Santagada) Date: Thu, 6 Jan 2011 00:29:12 -0200 Subject: [pypy-dev] psycopg2 on PyPy In-Reply-To: References: Message-ID: On Wed, Jan 5, 2011 at 6:51 PM, Alex Gaynor wrote: > Hi all, > For the past week and a half I've been working on an RPython implementation > of psycopg2 (the most popular database adapter for PostgreSQL). ?I'm happy > to announce that at this point it is passing all the DB-API2.0 tests, and > all the psycopg2 tests I converted into AppTests (most of them I think), it > also passes the entire DJango test suite (which makes?extensive?use of the > database). ?The downside is that running the full Django test suite on top > of pypy with it is still about 20% slower than CPython. ?So we have some > work to do. ?The code is all > at:?http://bitbucket.org/alex_gaynor/pypy-postgresql, I don't plan on > merging this back into the main dev tree, as it doesn't seem appropriate for > pypy-core to have a PostgreSQL adapter, rather I'd like to spend some time > looking at seperate compilation, so we can have external RPython modules. > What do you think are the main performance problem? Is it just the nature of unittests that are too small for the jit to work, is it that you have maybe too much rpython code that is not being jitted as well as it could or is it just a performance problem of pypy? -- Leonardo Santagada From amauryfa at gmail.com Thu Jan 6 03:59:43 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Thu, 6 Jan 2011 03:59:43 +0100 Subject: [pypy-dev] psycopg2 on PyPy In-Reply-To: References: Message-ID: Hi, 2011/1/5 Alex Gaynor : > The code is all at:?http://bitbucket.org/alex_gaynor/pypy-postgresql, I > don't plan on merging this back into the main dev tree, as it doesn't seem > appropriate for pypy-core to have a PostgreSQL adapter, There is already a module/oracle directory in the pypy tree, so why not... but you should be careful with the license: If your work is a translation of psycopg2, it probably needs to be licensed under the GPL. so this would be incompatible with PyPy which is licensed under a MIT-like license. (the oracle module in pypy is based on cx_Oracle, which also uses the MIT license) > rather I'd like to > spend some time looking at seperate compilation, so we can have external > RPython modules. If your goal is to build RPython modules incrementally, after the pypy interpreter has been compiled, it it a very difficult task, and at least three attempts to do it have failed. (The fourth one eventually produced the cpyext module, which is not a complete failure, but has nothing to do with RPython) On the other hand, it would be handy to be able to translate modules which source is outside pypy tree; something like:: ./translate.py -Ojit targetpypystandalone.py --with-external-module=/path/to/my/psycopg2 -- Amaury Forgeot d'Arc From renesd at gmail.com Thu Jan 6 14:57:13 2011 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Thu, 6 Jan 2011 13:57:13 +0000 Subject: [pypy-dev] Calling into PyPy from multi-threaded C++ In-Reply-To: <1294161936.1893.20.camel@localhost> References: <1294161936.1893.20.camel@localhost> Message-ID: Hi, are the thread GIL releasing functions going to be supported in pypy with CPyExt? To allow this usage: PyThreadState *_save; _save = PyEval_SaveThread(); // ...Do some blocking I/O operation, or CPU intensive operation that does not use the python api... PyEval_RestoreThread(_save); Or using the macros... Py_BEGIN_ALLOW_THREADS; // ...Do some blocking I/O operation, or CPU intensive operation that does not use the python api... Py_END_ALLOW_THREADS; Is it not possible or too hard, or is it just not implemented yet? On Tue, Jan 4, 2011 at 5:25 PM, Duncan Grisby wrote: > Hi, > > I've been experimenting with using PyPy in a big application, and the > initial results are very encouraging. One part of the application is a > fairly complex compiler, and that goes 2.5 times faster with PyPy than > with Python 2.6, which is a very positive result. > > The compiler is the one CPU-intensive bit of the system that I could > easily isolate to test with PyPy. All the rest of it consists of a > number of servers connected together using omniORB (which I maintain). > omniORBpy integrates the C++ core of omniORB with Python, via the C API. > omniORB is multi-threaded, and needs to call into Python from threads > created in the C++ world. With CPython that's possible, although it > takes some care to behave properly with thread states and the > interpreter lock and so on. > > As far as I can see, PyPy's C API implementation doesn't support the > thread state functions like PyGILState_Ensure and PyThreadState_Get, > which isn't a huge surprise given how different PyPy is. Are these > functions likely to be implemented at any point? ?Alternatively, is > there any other way for C/C++ code to call into PyPy from a thread > created outside PyPy? > > If there isn't a mechanism to do it at the moment, is it possible at > all? ?How much effort would it be to create one? > > Thanks, > > Duncan. > > -- > ?-- Duncan Grisby ? ? ? ? -- > ?-- duncan at grisby.org ? ? -- > ? -- http://www.grisby.org -- > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From amauryfa at gmail.com Thu Jan 6 15:13:39 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Thu, 6 Jan 2011 15:13:39 +0100 Subject: [pypy-dev] Calling into PyPy from multi-threaded C++ In-Reply-To: References: <1294161936.1893.20.camel@localhost> Message-ID: Hi, 2011/1/6 Ren? Dudfield : > Hi, > > are the thread GIL releasing functions going to be supported in pypy > with CPyExt? > > To allow this usage: > > PyThreadState *_save; > _save = PyEval_SaveThread(); > // ...Do some blocking I/O operation, or CPU intensive operation that > does not use the python api... > PyEval_RestoreThread(_save); > > Or using the macros... > > Py_BEGIN_ALLOW_THREADS; > // ...Do some blocking I/O operation, or CPU intensive operation that > does not use the python api... > Py_END_ALLOW_THREADS; > > Is it not possible or too hard, or is it just not implemented yet? Frankly, I even don't know. At least, the Py_BEGIN_ALLOW_THREADS macros are there and will compile. OTOH it seems that every call from the pypy interpreter to C releases the GIL, and that conversely, every call from to a Python API function will grab the GIL for the duration of the call (yes, even Py_INCREF) cpyext is maybe already thread-safe, only awfully slow... But this is just a guess, from looking at the source code. I'd be grateful if you could write a custom module and do some tests in this area, with and without the Py_BEGIN_ALLOW_THREADS calls. Cheers, -- Amaury Forgeot d'Arc From renesd at gmail.com Thu Jan 6 15:35:46 2011 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Thu, 6 Jan 2011 14:35:46 +0000 Subject: [pypy-dev] Calling into PyPy from multi-threaded C++ In-Reply-To: References: <1294161936.1893.20.camel@localhost> Message-ID: Hi, here is a basic function... without the whole module stuff around it. I'd need to do the rest of the module another time. PyObject* do_long_run (PyObject* self) { int i, a, b, c; a = 2; Py_BEGIN_ALLOW_THREADS; for(i = 0; i < 2000000000; i++) { b = i + a; c = a + i; } a = c - b; Py_END_ALLOW_THREADS; Py_RETURN_NONE; } // ... { "do_long_run", (PyCFunction) do_long_run , METH_NOARGS, "runs some cpu intensive code whilst releasing the GIL"}, // ... You could test it by calling that function in separate threads. With another thread printing something from python... like in the python code below. from threading import Thread # this assumes the _long_runner is the C module with the do_long_run function in it. import _long_runner class PythonPrinter(Thread): def run(self): for x in range(1000000): print (x) class CLongRunner(Thread): def run(self): _long_runner.do_long_run() threads = [PythonPrinter(), CLongRunner()] for t in threads: t.start() for t in threads: t.join() That is untested, and typed without compiling/running... but it should work, and should give you some insight on how the threading works. Maybe you can throw the c function into an existing module for easier testing... otherwise maybe I can write it in a few days. cya. On Thu, Jan 6, 2011 at 2:13 PM, Amaury Forgeot d'Arc wrote: > Hi, > > 2011/1/6 Ren? Dudfield : >> Hi, >> >> are the thread GIL releasing functions going to be supported in pypy >> with CPyExt? >> >> To allow this usage: >> >> PyThreadState *_save; >> _save = PyEval_SaveThread(); >> // ...Do some blocking I/O operation, or CPU intensive operation that >> does not use the python api... >> PyEval_RestoreThread(_save); >> >> Or using the macros... >> >> Py_BEGIN_ALLOW_THREADS; >> // ...Do some blocking I/O operation, or CPU intensive operation that >> does not use the python api... >> Py_END_ALLOW_THREADS; >> >> Is it not possible or too hard, or is it just not implemented yet? > > Frankly, I even don't know. > At least, the Py_BEGIN_ALLOW_THREADS macros are there and will compile. > > OTOH it seems that every call from the pypy interpreter to C releases the GIL, > and that conversely, every call from to a Python API function will > grab the GIL for the duration of the call (yes, even Py_INCREF) > cpyext is maybe already thread-safe, only awfully slow... > But this is just a guess, from looking at the source code. > > I'd be grateful if you could write a custom module and do some tests > in this area, > with and without the Py_BEGIN_ALLOW_THREADS calls. > > Cheers, > > -- > Amaury Forgeot d'Arc > From nathanael.jones at gmail.com Mon Jan 10 05:24:09 2011 From: nathanael.jones at gmail.com (Nathanael D. Jones) Date: Sun, 9 Jan 2011 23:24:09 -0500 Subject: [pypy-dev] Continuations and sandboxing Message-ID: Hi folks, I've got a challenging set of requirements that I think PyPy may be able to meet, but I'd like to get some feedback before I jump in. I'm developing a collaboratively edited game. It's not a MUD, but similar in concept, and also text based in nature. HogwartsLive.com and lotgd.ne t are examples of the genre I'm going for. I believe if a game of that genre makes user modification simple and rewarding enough, it could be self-sustaining and grow quickly enough to keep users interested perpetually, like Wikipedia. It's also a fascinating computer science challenge, because it requires a LOT from a computer language. 1) Sandboxing. To allow users to make changes to the game without being able to "cheat" at it or escalate privileges. 2) Serializeable continuations. With gameplay being based on good plot and story flow, continuations are critical to allow straightforward implementation of 'workflows' that depend on user choice at every turn. 3) Tail-call elimination. By nature, players will accumulate a very large call stack. While this isn't terribly bad a first glance, the following issue combines with it to cause a very big problem: When code changes underneath a continuation, we need to determine how to resume flow. One option is annotating a checkpoint method in each code 'file'. However, if a user's call stack includes every file in the system, each change will cause them to restart. Tail-call elimination would help eliminate unneeded stack frames and minimize re-spawning. 3) Dynamic, strong typing and metaprogramming are key for keeping the API simple. 4) Dynamic code loading. Users will be able to 'branch' their own version of the world and share it with others. There may be thousands of versions of a class, and they need to be able to execute in separate sandboxes at the same time. Source code will be pulled from a Git repository or some kind of versioning database. I'm interested in knowing which of these PyPy already does, and which of them I can make it do. I appreciate your help! Nathanael Jones http://nathanaeljones.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110109/55ef7d24/attachment.htm From dimaqq at gmail.com Mon Jan 10 06:24:22 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Sun, 9 Jan 2011 22:24:22 -0700 Subject: [pypy-dev] Continuations and sandboxing In-Reply-To: References: Message-ID: First of all, congratulations on getting THE idea, I think user content is essential too, but I see why commercial games shy away from it An open source game could do really well! 1, yes, but not in a way you want (you want many boxes that can't touch each other, right?) 2, yes, stackless is the name/misnomer 3a, I doubt is relevant, you have to show that your workload has a substantial number of tail calls; stack depth is not that important with stackless. 3b, I imagine this is impossible in general case, how do you see this done in any language? 3 again, yes of course, and I bet you knew this already. 4 indeed, unloading old code is a bit harder as you have to make sure all references to module or function or class are lost in timely fashion. p.s. I'm not an expert on pypy yet On 9 January 2011 21:24, Nathanael D. Jones wrote: > Hi folks, > I've got a challenging set of requirements that I think PyPy may be able to > meet, but I'd like to get some feedback before I jump in. > I'm developing a collaboratively?edited?game. It's not a MUD, but?similar?in > concept, and also text based in nature. > HogwartsLive.com and?lotgd.net?are examples of the genre I'm going for. > I believe if a game of that genre makes user modification simple and > rewarding enough, it could be self-sustaining?and grow quickly enough to > keep users interested perpetually, like Wikipedia. > It's also a fascinating computer science challenge, because it requires a > LOT from a computer language. > 1) Sandboxing. To allow users to make changes to the game without being able > to "cheat" at it or escalate?privileges. > 2) Serializeable continuations. With gameplay being based on good plot and > story flow, continuations are critical to allow straightforward > implementation of 'workflows' that depend on user choice at every turn. > 3) Tail-call elimination. ?By nature, players will accumulate a very large > call stack. While this isn't terribly bad a first glance, the following > issue combines with it to cause a very big problem: > When code changes underneath a continuation, we need to determine how to > resume flow. One option is annotating a checkpoint method in each code > 'file'. However, if a user's call stack includes every file in the system, > each change will cause them to restart. > Tail-call elimination would help eliminate unneeded stack frames and > minimize re-spawning. > 3) Dynamic, strong typing and metaprogramming are key for keeping the API > simple. > 4) Dynamic code loading. Users will be able to 'branch' their own version of > the world and share it with others. There may be thousands of versions of a > class, and they need to be able to execute in separate sandboxes at the same > time. Source code will be pulled from a Git repository or some kind of > versioning database. > I'm interested in knowing which of these PyPy already does, and which of > them I can make it do. > I appreciate your help! > Nathanael Jones > http://nathanaeljones.com/ > > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From william.leslie.ttg at gmail.com Mon Jan 10 09:22:27 2011 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Mon, 10 Jan 2011 19:22:27 +1100 Subject: [pypy-dev] Continuations and sandboxing In-Reply-To: References: Message-ID: On 10 January 2011 15:24, Nathanael D. Jones wrote: > Hi folks, Hi! > 2) Serializeable continuations. With gameplay being based on good plot and > story flow, continuations are critical to allow straightforward > implementation of 'workflows' that depend on user choice at every turn. > 3) Tail-call elimination. ?By nature, players will accumulate a very large > call stack. While this isn't terribly bad a first glance, the following > issue combines with it to cause a very big problem: > When code changes underneath a continuation, we need to determine how to > resume flow. One option is annotating a checkpoint method in each code > 'file'. However, if a user's call stack includes every file in the system, > each change will cause them to restart. > Tail-call elimination would help eliminate unneeded stack frames and > minimize re-spawning. I wonder if you don't want instead to manage the (user) dynamic context yourself - if players already need to remember to write all actions in a tail-recursive manner, it is probably worth making that API explicit, and if you do that, the underlying implementation doesn't need to do TCE at all. But I guess the existing stackless support should be good enough if you just want one-shot continuations. > 4) Dynamic code loading. Users will be able to 'branch' their own version of > the world and share it with others. There may be thousands of versions of a > class, and they need to be able to execute in separate sandboxes at the same > time. Source code will be pulled from a Git repository or some kind of > versioning database. Quite like this idea. You do have to deal with a bunch of (fairly well known) problems, which any specific implementation of dynamic code loading is going to need to solve (or not). Pypy doesn't currently implement any hot-schema-change magic, and reloading has always been error prone in the presence of state. First-class mutable types make it particularly difficult (there is no sensible answer to what it means to reload a python class). The one issue that interests me is where you implement the persistence boundary - do you go with orthogonal persistence and act as if everything is preserved, or assume all user code is run within some sort of (fast and loose) transaction that can be re-entered at will, providing an API for persistent data access? The second case makes the reloading question a bit more reasonable, because you can always throw away the current delta and replay the external effects, assuming the interface for the external events hasn't changed significantly. -- William Leslie From fijall at gmail.com Mon Jan 10 12:23:12 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 10 Jan 2011 13:23:12 +0200 Subject: [pypy-dev] [pypy-svn] pypy jit-longlong: Intermediate check-in, breaks everything by exposing cases of In-Reply-To: <20110109223334.6A6D9282B9E@codespeak.net> References: <20110109223334.6A6D9282B9E@codespeak.net> Message-ID: For what is worth I'm happy with making "rething regalloc API" a sprint topic. So far I failed to come up with something more reasonable. Cheers, fijal On Mon, Jan 10, 2011 at 12:33 AM, arigo wrote: > Author: Armin Rigo > Branch: jit-longlong > Changeset: r40545:9a472ddf9893 > Date: 2011-01-09 23:33 +0100 > http://bitbucket.org/pypy/pypy/changeset/9a472ddf9893/ > > Log: ? ?Intermediate check-in, breaks everything by exposing cases of > ? ? ? ?constant arguments that are not correctly handled so far. /me is > ? ? ? ?again annoyed by the API of regalloc.py, which seems to be very > ? ? ? ?reasonable, until we hit these cases. > > diff --git a/pypy/jit/backend/x86/assembler.py b/pypy/jit/backend/x86/assembler.py > --- a/pypy/jit/backend/x86/assembler.py > +++ b/pypy/jit/backend/x86/assembler.py > @@ -1174,6 +1174,13 @@ > ? ? ? ? self.mc.SBB_rr(resloc.value, resloc.value) > ? ? ? ? self.mc.NEG_r(resloc.value) > > + ? ?def genop_llong_lt(self, op, arglocs, resloc): > + ? ? ? ?# XXX just a special case for now: "x < 0" > + ? ? ? ?loc1, = arglocs > + ? ? ? ?self.mc.PMOVMSKB_rx(resloc.value, loc1.value) > + ? ? ? ?self.mc.SHR_ri(resloc.value, 7) > + ? ? ? ?self.mc.AND_ri(resloc.value, 1) > + > ? ? def genop_new_with_vtable(self, op, arglocs, result_loc): > ? ? ? ? assert result_loc is eax > ? ? ? ? loc_vtable = arglocs[-1] > @@ -1784,6 +1791,7 @@ > ? ? ? ? ? ? if isinstance(op.getdescr(), LongLongCallDescr): > ? ? ? ? ? ? ? ? self.mc.MOV_br(resloc.value, eax.value) ? ? ?# long long > ? ? ? ? ? ? ? ? self.mc.MOV_br(resloc.value + 4, edx.value) > + ? ? ? ? ? ? ? ?# XXX should ideally not move the result on the stack > ? ? ? ? ? ? else: > ? ? ? ? ? ? ? ? self.mc.FSTP_b(resloc.value) ? # float return > ? ? ? ? elif size == WORD: > > diff --git a/pypy/jit/codewriter/jtransform.py b/pypy/jit/codewriter/jtransform.py > --- a/pypy/jit/codewriter/jtransform.py > +++ b/pypy/jit/codewriter/jtransform.py > @@ -894,7 +894,8 @@ > ? ? ? ? ? ? ? ? args = op.args > ? ? ? ? ? ? ? ? op1 = self.prepare_builtin_call(op, "llong_%s", args) > ? ? ? ? ? ? ? ? op2 = self._handle_oopspec_call(op1, args, > - ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?EffectInfo.OS_LLONG_%s) > + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?EffectInfo.OS_LLONG_%s, > + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?EffectInfo.EF_PURE) > ? ? ? ? ? ? ? ? return op2 > ? ? ? ? ''' % (_op, _oopspec.lower(), _oopspec)).compile() > > @@ -1273,7 +1274,7 @@ > > ? ? def _handle_oopspec_call(self, op, args, oopspecindex, extraeffect=None): > ? ? ? ? calldescr = self.callcontrol.getcalldescr(op, oopspecindex) > - ? ? ? ?if extraeffect: > + ? ? ? ?if extraeffect is not None: > ? ? ? ? ? ? calldescr.get_extra_info().extraeffect = extraeffect > ? ? ? ? if isinstance(op.args[0].value, str): > ? ? ? ? ? ? pass ?# for tests only > > diff --git a/pypy/jit/backend/x86/regalloc.py b/pypy/jit/backend/x86/regalloc.py > --- a/pypy/jit/backend/x86/regalloc.py > +++ b/pypy/jit/backend/x86/regalloc.py > @@ -653,6 +653,22 @@ > ? ? ? ? self.PerformLLong(op, [loc1, loc2, loc3], loc0) > ? ? ? ? self.xrm.possibly_free_vars(args) > > + ? ?def _maybe_consider_llong_lt(self, op): > + ? ? ? ?# XXX just a special case for now > + ? ? ? ?from pypy.rlib.longlong2float import longlong2float > + ? ? ? ?box = op.getarg(2) > + ? ? ? ?if not isinstance(box, ConstFloat): > + ? ? ? ? ? ?return False > + ? ? ? ?if not (box.value == longlong2float(r_longlong(0))): > + ? ? ? ? ? ?return False > + ? ? ? ?# "x < 0" > + ? ? ? ?box = op.getarg(1) > + ? ? ? ?loc1 = self.xrm.make_sure_var_in_reg(box, imm_fine=False) > + ? ? ? ?loc0 = self.rm.force_allocate_reg(op.result) > + ? ? ? ?self.PerformLLong(op, [loc1], loc0) > + ? ? ? ?self.xrm.possibly_free_var(box) > + ? ? ? ?return True > + > ? ? def _consider_llong_to_int(self, op): > ? ? ? ? # accept an argument in a xmm register or in the stack > ? ? ? ? loc1 = self.xrm.loc(op.getarg(1)) > @@ -769,6 +785,9 @@ > ? ? ? ? ? ? ? ? if (oopspecindex == EffectInfo.OS_LLONG_EQ or > ? ? ? ? ? ? ? ? ? ? oopspecindex == EffectInfo.OS_LLONG_NE): > ? ? ? ? ? ? ? ? ? ? return self._consider_llong_cmp_xx(op) > + ? ? ? ? ? ? ? ?if oopspecindex == EffectInfo.OS_LLONG_LT: > + ? ? ? ? ? ? ? ? ? ?if self._maybe_consider_llong_lt(op): > + ? ? ? ? ? ? ? ? ? ? ? ?return > ? ? ? ? # > ? ? ? ? self._consider_call(op) > > _______________________________________________ > pypy-svn mailing list > pypy-svn at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-svn > From p.giarrusso at gmail.com Mon Jan 10 21:18:05 2011 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Mon, 10 Jan 2011 21:18:05 +0100 Subject: [pypy-dev] Continuations and sandboxing In-Reply-To: References: Message-ID: Hi all, On Mon, Jan 10, 2011 at 09:22, William ML Leslie wrote: > On 10 January 2011 15:24, Nathanael D. Jones wrote: >> Hi folks, >> 2) Serializeable continuations. With gameplay being based on good plot and >> story flow, continuations are critical to allow straightforward >> implementation of 'workflows' that depend on user choice at every turn. >> 3) Tail-call elimination. ?By nature, players will accumulate a very large >> call stack. While this isn't terribly bad a first glance, the following >> issue combines with it to cause a very big problem: >> When code changes underneath a continuation, we need to determine how to >> resume flow. One option is annotating a checkpoint method in each code >> 'file'. [I didn't get this part. Reading what's below, I assume that for each call frame, you remember somehow the file defining the called function.] > > However, if a user's call stack includes every file in the system, >> each change will cause them to restart. >> Tail-call elimination would help eliminate unneeded stack frames and >> minimize re-spawning. That optimization looks maybe invalid. Suppose file A contains a function f() which tail-calls g(1, 2, 3), and then file A is modified, so that f() does another tail-call. Now it's not clear why do you do restart: if you do restart when executed code is modified, then this optimization is invalid. If you reload only when code yet to execute is modified, then the optimization is valid, but you could perform also more advanced optimizations to avoid restart (you could compare the generated bytecode up to the end of the outermost loop containing the point where the call frame was saved and another procedure was invoked). It is also not clear which semantic guarantee you want to achieve by this restart. Would you use transactions to avoid performing side-effects again? >> 4) Dynamic code loading. Users will be able to 'branch' their own version of >> the world and share it with others. There may be thousands of versions of a >> class, and they need to be able to execute in separate sandboxes at the same >> time. Source code will be pulled from a Git repository or some kind of >> versioning database. > Quite like this idea. > You do have to deal with a bunch of (fairly well known) problems, > which any specific implementation of dynamic code loading is going to > need to solve (or not). ?Pypy doesn't currently implement any > hot-schema-change magic, and reloading has always been error prone in > the presence of state. ?First-class mutable types make it particularly > difficult (there is no sensible answer to what it means to reload a > python class). You might want to reuse the solutions to those issues used in the Java (and maybe .NET) world. Java allows reloading a class in a different classloader, and that has been used inside OSGi (see http://www.osgi.org/About/Technology). Not sure about the solution in OSGi, but Java serialization allows to serialize an instance of version 1 of a class, and to de-serialize it with version 2 of that class, if v2 takes extra care for that; one could use this to convert existing instances. > The one issue that interests me is where you implement the persistence > boundary - do you go with orthogonal persistence and act as if > everything is preserved, or assume all user code is run within some > sort of (fast and loose) transaction that can be re-entered at will, > providing an API for persistent data access? ?The second case makes > the reloading question a bit more reasonable, because you can always > throw away the current delta and replay the external effects, assuming > the interface for the external events hasn't changed significantly. The key question is: when would you start and commit such transactions? Apart from that, your idea looks very similar to Software Transactional Memory (STM). STM restarts explicitly-marked transactions in a thread when some other thread modifies the affected data (which would be a data race) and commits its transaction. In your case, a transaction is restarted when some other thread modifies the involved code. Cheers, -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From william.leslie.ttg at gmail.com Mon Jan 10 23:05:36 2011 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Tue, 11 Jan 2011 09:05:36 +1100 Subject: [pypy-dev] Continuations and sandboxing In-Reply-To: References: Message-ID: On 11 January 2011 07:18, Paolo Giarrusso wrote: > Hi all, > > On Mon, Jan 10, 2011 at 09:22, William ML Leslie > wrote: >> On 10 January 2011 15:24, Nathanael D. Jones wrote: >>> 4) Dynamic code loading. Users will be able to 'branch' their own version of >>> the world and share it with others. There may be thousands of versions of a >>> class, and they need to be able to execute in separate sandboxes at the same >>> time. Source code will be pulled from a Git repository or some kind of >>> versioning database. > >> Quite like this idea. > >> You do have to deal with a bunch of (fairly well known) problems, >> which any specific implementation of dynamic code loading is going to >> need to solve (or not). ?Pypy doesn't currently implement any >> hot-schema-change magic, and reloading has always been error prone in >> the presence of state. ?First-class mutable types make it particularly >> difficult (there is no sensible answer to what it means to reload a >> python class). > > You might want to reuse the solutions to those issues used in the Java > (and maybe .NET) world. Java allows reloading a class in a different > classloader, and that has been used inside OSGi (see > http://www.osgi.org/About/Technology). > Not sure about the solution in OSGi, but Java serialization allows to > serialize an instance of version 1 of a class, and to de-serialize it > with version 2 of that class, if v2 takes extra care for that; one > could use this to convert existing instances. Sure, and there is also Dynamic Class Evolution for Hotspot, and many other VMs support some form of reloading and redefinition (goops is an exceptional and accessible example, see http://wingolog.org/archives/2009/11/09/class-redefinition-in-guile for a post with lots of diagrams). What I am trying to say is that the *ability* to do reloading does not mean your code will work in the presence of interface changes. You have to decide whether updating a closure in a way that captures more variables or changes the signature updates existing instances of that closure: if it does, there will be entries missing in the closure slot, if it doesn't, older closures won't be callable in the same way as the new closures. No matter what, you here need to write code to be explicitly reloadable, or provide some way to instruct the runtime what to do for each schema change. > >> The one issue that interests me is where you implement the persistence >> boundary - do you go with orthogonal persistence and act as if >> everything is preserved, or assume all user code is run within some >> sort of (fast and loose) transaction that can be re-entered at will, >> providing an API for persistent data access? ?The second case makes >> the reloading question a bit more reasonable, because you can always >> throw away the current delta and replay the external effects, assuming >> the interface for the external events hasn't changed significantly. > > The key question is: when would you start and commit such transactions? > > Apart from that, your idea looks very similar to Software > Transactional Memory (STM). STM restarts explicitly-marked > transactions in a thread when some other thread modifies the affected > data (which would be a data race) and commits its transaction. In your > case, a transaction is restarted when some other thread modifies the > involved code. There is a bit of ambiguity in the "some other thread modifies" here. I don't know what synchronisation and communication is going on in your game, but I suspect that it only rarely interacts with reloading code in an interesting way. I'll reply to this properly in another email, I'd better get back to work :) -- William Leslie From nathanael.jones at gmail.com Tue Jan 11 05:25:04 2011 From: nathanael.jones at gmail.com (Nathanael D. Jones) Date: Mon, 10 Jan 2011 23:25:04 -0500 Subject: [pypy-dev] Continuations and sandboxing In-Reply-To: References: Message-ID: Regardng #1: Sandboxing is a major concern for me. Different code will need different sandboxing levels depending upon who created/approved the code. I can't have everything in one sandbox - I need isolated boxes on a per-request level. I think I see a way to sidestep the need for #3 and also help with hot-swapping. If I try to empty the stack as often as possible, it should provide frequent opportunities for 'free' reloading. I.E. (psuedocode) def garden(): if btnHouse.clicked: return house def house(): if btnGarden.clicked: return garden def loop(): func = house; while(true): #TODO: Update func to the latest version #TODO: Build an execution sandbox appropriate to the security clearance func has func = sandbox.call(func) print "Your are now in the " + func The idea is that if functions return the next function to call instead of calling them, we have explicit tail-call elimination, and we have an explicit point at which we can rebuild the sandbox and upgrade the code. Regarding the persistence boundary, I've seen some very good points come up. A certain amount of orthogonal persistence is needed in the form of a continuation, but that would only be function-scoped data. I don't intent to allow users to use global variables or suchlike to persist data, I want a tailored API. Much data will be user scoped, and therefore lockable. However, some data will be shared across all users. I'm not sure what the best way to handle this is. With sandboxing and full control over the persistence API I could theoretically implement STM. I want to make sure the architecture is very scalable, so I've been considering something like BigTable or SimpleDB as the persistence store. Here transaction and locking options are more limited. Thoughts? Nathanael On Mon, Jan 10, 2011 at 5:05 PM, William ML Leslie < william.leslie.ttg at gmail.com> wrote: > On 11 January 2011 07:18, Paolo Giarrusso wrote: > > Hi all, > > > > On Mon, Jan 10, 2011 at 09:22, William ML Leslie > > wrote: > >> On 10 January 2011 15:24, Nathanael D. Jones > wrote: > >>> 4) Dynamic code loading. Users will be able to 'branch' their own > version of > >>> the world and share it with others. There may be thousands of versions > of a > >>> class, and they need to be able to execute in separate sandboxes at the > same > >>> time. Source code will be pulled from a Git repository or some kind of > >>> versioning database. > > > >> Quite like this idea. > > > >> You do have to deal with a bunch of (fairly well known) problems, > >> which any specific implementation of dynamic code loading is going to > >> need to solve (or not). Pypy doesn't currently implement any > >> hot-schema-change magic, and reloading has always been error prone in > >> the presence of state. First-class mutable types make it particularly > >> difficult (there is no sensible answer to what it means to reload a > >> python class). > > > > You might want to reuse the solutions to those issues used in the Java > > (and maybe .NET) world. Java allows reloading a class in a different > > classloader, and that has been used inside OSGi (see > > http://www.osgi.org/About/Technology). > > Not sure about the solution in OSGi, but Java serialization allows to > > serialize an instance of version 1 of a class, and to de-serialize it > > with version 2 of that class, if v2 takes extra care for that; one > > could use this to convert existing instances. > > Sure, and there is also Dynamic Class Evolution for Hotspot, and many > other VMs support some form of reloading and redefinition (goops is an > exceptional and accessible example, see > http://wingolog.org/archives/2009/11/09/class-redefinition-in-guile > for a post with lots of diagrams). > > What I am trying to say is that the *ability* to do reloading does not > mean your code will work in the presence of interface changes. You > have to decide whether updating a closure in a way that captures more > variables or changes the signature updates existing instances of that > closure: if it does, there will be entries missing in the closure > slot, if it doesn't, older closures won't be callable in the same way > as the new closures. No matter what, you here need to write code to > be explicitly reloadable, or provide some way to instruct the runtime > what to do for each schema change. > > > > >> The one issue that interests me is where you implement the persistence > >> boundary - do you go with orthogonal persistence and act as if > >> everything is preserved, or assume all user code is run within some > >> sort of (fast and loose) transaction that can be re-entered at will, > >> providing an API for persistent data access? The second case makes > >> the reloading question a bit more reasonable, because you can always > >> throw away the current delta and replay the external effects, assuming > >> the interface for the external events hasn't changed significantly. > > > > The key question is: when would you start and commit such transactions? > > > > Apart from that, your idea looks very similar to Software > > Transactional Memory (STM). STM restarts explicitly-marked > > transactions in a thread when some other thread modifies the affected > > data (which would be a data race) and commits its transaction. In your > > case, a transaction is restarted when some other thread modifies the > > involved code. > > There is a bit of ambiguity in the "some other thread modifies" here. > I don't know what synchronisation and communication is going on in > your game, but I suspect that it only rarely interacts with reloading > code in an interesting way. I'll reply to this properly in another > email, I'd better get back to work :) > > -- > William Leslie > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110110/2eac54b5/attachment.htm From poelzi at poelzi.org Tue Jan 11 06:49:14 2011 From: poelzi at poelzi.org (Daniel Poelzleithner) Date: Tue, 11 Jan 2011 06:49:14 +0100 Subject: [pypy-dev] good by swap of death Message-ID: <4D2BEF5A.4000107@poelzi.org> hi, as I'm surly not the only one having sometimes memory problems when translating pypy. Slowly getting bitten by the swap of death, not registering anything until the desktop stops and rescuing is a very slow process (yes, i have a special root login with /bin/sh shell and memlockd...). I was just able to fix the problem with ulatencyd [1] and as I think translating pypy is a real usecase I would be glad having some feedback. I was quite surprised that after a simple memleak emulator got isolated successfully, pypy translation works like a charm, even when i put additional pressure on the system :-) [1] https://github.com/poelzi/ulatencyd kind regards poelzi -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 900 bytes Desc: OpenPGP digital signature Url : http://codespeak.net/pipermail/pypy-dev/attachments/20110111/fcc693e4/attachment-0001.pgp From fijall at gmail.com Tue Jan 11 07:45:07 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 11 Jan 2011 08:45:07 +0200 Subject: [pypy-dev] good by swap of death In-Reply-To: <4D2BEF5A.4000107@poelzi.org> References: <4D2BEF5A.4000107@poelzi.org> Message-ID: Hey. In general it's next to impossible to translate pypy if swapping. You need at least 2G of RAM for 32bit and 4G for 64bit build. On Tue, Jan 11, 2011 at 7:49 AM, Daniel Poelzleithner wrote: > hi, > > as I'm surly not the only one having sometimes memory problems when > translating pypy. Slowly getting bitten by the swap of death, not > registering anything until the desktop stops and rescuing is a very slow > process (yes, i have a special root login with /bin/sh shell and > memlockd...). > > I was just able to fix the problem with ulatencyd [1] and as I think > translating pypy is a real usecase I would be glad having some feedback. > I was quite surprised that after a simple memleak emulator got isolated > successfully, pypy translation works like a charm, even when i put > additional pressure on the system :-) > > [1] https://github.com/poelzi/ulatencyd > > kind regards > ?poelzi > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From william.leslie.ttg at gmail.com Tue Jan 11 08:43:50 2011 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Tue, 11 Jan 2011 18:43:50 +1100 Subject: [pypy-dev] Continuations and sandboxing In-Reply-To: References: Message-ID: On 11 January 2011 15:25, Nathanael D. Jones wrote: > Regardng #1: Sandboxing is a major concern for me. Different code will need > different sandboxing levels depending upon who created/approved the code. > I can't have everything in one sandbox - I need isolated boxes on a > per-request level. You will probably want to implement a capability architecture, where the authority to perform some action is bundled together with the method that invokes it. The language E ( http://erights.org ) was designed for this sort of thing (I think it was written for a MUD originally), and would be worth looking at even if you didn't use the language itself. Capability theory is sadly not well known, considering just how well it solves ever more common problems. > The idea is that if functions return the next function to call instead of > calling them, we have explicit tail-call elimination, and we have an > explicit point at which we can rebuild the sandbox and upgrade the code. Right, this is the "driver loop" implementation of TCE. There are several implementations of TCE kicking around, (including one I did for cpython some time ago, a decorator that would patch load ... call* ... return bytecodes with a call to a class that collects the function and arguments, and wraps the function in a driver loop that is bypassed when the target is marked as recursive, too). Unfortunately, using only implicit continuations, once you enter somebody's function, you have no way to get out of it - that person can except: and trap you there if they like. E has a way to protect against this, but you can't protect against the function busy-looping without something more (a way to kill threads, specifically, which probably means taking down the process). > Much data will be user scoped, and therefore lockable. What does lockable mean in this case? Under what conditions does reloading happen while a user continuation is in play? How do you want continuations to interact with the locking? > On Mon, Jan 10, 2011 at 5:05 PM, William ML Leslie > wrote: >> >> On 11 January 2011 07:18, Paolo Giarrusso wrote: >> >> The one issue that interests me is where you implement the persistence >> >> boundary - do you go with orthogonal persistence and act as if >> >> everything is preserved, or assume all user code is run within some >> >> sort of (fast and loose) transaction that can be re-entered at will, >> >> providing an API for persistent data access? ?The second case makes >> >> the reloading question a bit more reasonable, because you can always >> >> throw away the current delta and replay the external effects, assuming >> >> the interface for the external events hasn't changed significantly. >> > >> > The key question is: when would you start and commit such transactions? >> > >> > Apart from that, your idea looks very similar to Software >> > Transactional Memory (STM). STM restarts explicitly-marked >> > transactions in a thread when some other thread modifies the affected >> > data (which would be a data race) and commits its transaction. In your >> > case, a transaction is restarted when some other thread modifies the >> > involved code. I don't think it would be useful at all to have regular STM transactions delimited at the room boundary - there are surely going to be other people interacting with the room at the same time as you, and sending everyone else back to the start of the room every time someone modifies the room is going to get old very quickly. The boundary introduced by modifying code is probably significantly more coarse. If you are storing (as the transaction log) only the external effects (user input, random.* output), however, you can build effect commutators automatically. For example, given a code change to object X, you can initialise it with the previous persistent state, and replay the effects within the current transaction. That said, I think having a transaction per room is too coarse - per input or set of inputs makes even more sense. The idea there is that there is much less code to replay when two effects interfere. -- William Leslie From fijall at gmail.com Tue Jan 11 08:48:41 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 11 Jan 2011 09:48:41 +0200 Subject: [pypy-dev] Continuations and sandboxing In-Reply-To: References: Message-ID: Hello. This discussion have drifted considerably. In general this is the pypy development mailing list, so personally I would like to keep discussions only to things that are either very interesting for people actively working on PyPy or have remote possibility that someone will start working on them. Discussing STM without someone being interested in actually looking is out of topic. PS. Feel free to correct me if anyone working actively on pypy is finding this discussion useful for PyPy development. Cheers, fijal On Tue, Jan 11, 2011 at 9:43 AM, William ML Leslie wrote: > On 11 January 2011 15:25, Nathanael D. Jones wrote: >> Regardng #1: Sandboxing is a major concern for me. Different code will need >> different sandboxing levels depending upon who created/approved the code. >> I can't have everything in one sandbox - I need isolated boxes on a >> per-request level. > > You will probably want to implement a capability architecture, where > the authority to perform some action is bundled together with the > method that invokes it. ?The language E ( http://erights.org ) was > designed for this sort of thing (I think it was written for a MUD > originally), and would be worth looking at even if you didn't use the > language itself. ?Capability theory is sadly not well known, > considering just how well it solves ever more common problems. > >> The idea is that if functions return the next function to call instead of >> calling them, we have explicit tail-call elimination, and we have an >> explicit point at which we can rebuild the sandbox and upgrade the code. > > Right, this is the "driver loop" implementation of TCE. ?There are > several implementations of TCE kicking around, (including one I did > for cpython some time ago, a decorator that would patch load ... call* > ... return bytecodes with a call to a class that collects the function > and arguments, and wraps the function in a driver loop that is > bypassed when the target is marked as recursive, too). > > Unfortunately, using only implicit continuations, once you enter > somebody's function, you have no way to get out of it - that person > can except: and trap you there if they like. ?E has a way to protect > against this, but you can't protect against the function busy-looping > without something more (a way to kill threads, specifically, which > probably means taking down the process). > >> Much data will be user scoped, and therefore lockable. > > What does lockable mean in this case? > > Under what conditions does reloading happen while a user continuation > is in play? ?How do you want continuations to interact with the > locking? > >> On Mon, Jan 10, 2011 at 5:05 PM, William ML Leslie >> wrote: >>> >>> On 11 January 2011 07:18, Paolo Giarrusso wrote: >>> >> The one issue that interests me is where you implement the persistence >>> >> boundary - do you go with orthogonal persistence and act as if >>> >> everything is preserved, or assume all user code is run within some >>> >> sort of (fast and loose) transaction that can be re-entered at will, >>> >> providing an API for persistent data access? ?The second case makes >>> >> the reloading question a bit more reasonable, because you can always >>> >> throw away the current delta and replay the external effects, assuming >>> >> the interface for the external events hasn't changed significantly. >>> > >>> > The key question is: when would you start and commit such transactions? >>> > >>> > Apart from that, your idea looks very similar to Software >>> > Transactional Memory (STM). STM restarts explicitly-marked >>> > transactions in a thread when some other thread modifies the affected >>> > data (which would be a data race) and commits its transaction. In your >>> > case, a transaction is restarted when some other thread modifies the >>> > involved code. > > I don't think it would be useful at all to have regular STM > transactions delimited at the room boundary - there are surely going > to be other people interacting with the room at the same time as you, > and sending everyone else back to the start of the room every time > someone modifies the room is going to get old very quickly. ?The > boundary introduced by modifying code is probably significantly more > coarse. > > If you are storing (as the transaction log) only the external effects > (user input, random.* output), however, you can build effect > commutators automatically. ?For example, given a code change to object > X, you can initialise it with the previous persistent state, and > replay the effects within the current transaction. > > That said, I think having a transaction per room is too coarse - per > input or set of inputs makes even more sense. ?The idea there is that > there is much less code to replay when two effects interfere. > > -- > William Leslie > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From renesd at gmail.com Tue Jan 11 11:41:58 2011 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Tue, 11 Jan 2011 10:41:58 +0000 Subject: [pypy-dev] Calling into PyPy from multi-threaded C++ In-Reply-To: References: <1294161936.1893.20.camel@localhost> Message-ID: Hi, did you find this useful? Or would you like me to complete the rest of the module code around it? cheers, On Thu, Jan 6, 2011 at 2:35 PM, Ren? Dudfield wrote: > Hi, > > here is a basic function... without the whole module stuff around it. > I'd need to do the rest of the module another time. > > PyObject* do_long_run (PyObject* self) { > ? ?int i, a, b, c; > ? ?a = 2; > > ? ?Py_BEGIN_ALLOW_THREADS; > ? ?for(i = 0; i < 2000000000; i++) { > ? ? ? ?b = i + a; > ? ? ? ?c = a + i; > ? ?} > ? ?a = c - b; > ? ?Py_END_ALLOW_THREADS; > > ? ?Py_RETURN_NONE; > } > > > // ... > ? ?{ "do_long_run", (PyCFunction) do_long_run , METH_NOARGS, "runs > some cpu intensive code whilst releasing the GIL"}, > // ... > > > > You could test it by calling that function in separate threads. ?With > another thread printing something from python... like in the python > code below. > > > > from threading import Thread > # this assumes the _long_runner is the C module with the do_long_run > function in it. > import _long_runner > > class PythonPrinter(Thread): > ? ?def run(self): > ? ? ? ?for x in range(1000000): > ? ? ? ? ? ?print (x) > > class CLongRunner(Thread): > ? ?def run(self): > ? ? ? ?_long_runner.do_long_run() > > > threads = [PythonPrinter(), CLongRunner()] > for t in threads: > ? ?t.start() > for t in threads: > ? ?t.join() > > > That is untested, and typed without compiling/running... but it should > work, and should give you some insight on how the threading works. > > Maybe you can throw the c function into an existing module for easier > testing... otherwise maybe I can write it in a few days. > > cya. > > > > On Thu, Jan 6, 2011 at 2:13 PM, Amaury Forgeot d'Arc wrote: >> Hi, >> >> 2011/1/6 Ren? Dudfield : >>> Hi, >>> >>> are the thread GIL releasing functions going to be supported in pypy >>> with CPyExt? >>> >>> To allow this usage: >>> >>> PyThreadState *_save; >>> _save = PyEval_SaveThread(); >>> // ...Do some blocking I/O operation, or CPU intensive operation that >>> does not use the python api... >>> PyEval_RestoreThread(_save); >>> >>> Or using the macros... >>> >>> Py_BEGIN_ALLOW_THREADS; >>> // ...Do some blocking I/O operation, or CPU intensive operation that >>> does not use the python api... >>> Py_END_ALLOW_THREADS; >>> >>> Is it not possible or too hard, or is it just not implemented yet? >> >> Frankly, I even don't know. >> At least, the Py_BEGIN_ALLOW_THREADS macros are there and will compile. >> >> OTOH it seems that every call from the pypy interpreter to C releases the GIL, >> and that conversely, every call from to a Python API function will >> grab the GIL for the duration of the call (yes, even Py_INCREF) >> cpyext is maybe already thread-safe, only awfully slow... >> But this is just a guess, from looking at the source code. >> >> I'd be grateful if you could write a custom module and do some tests >> in this area, >> with and without the Py_BEGIN_ALLOW_THREADS calls. >> >> Cheers, >> >> -- >> Amaury Forgeot d'Arc >> > From lac at openend.se Tue Jan 11 12:03:45 2011 From: lac at openend.se (Laura Creighton) Date: Tue, 11 Jan 2011 12:03:45 +0100 Subject: [pypy-dev] Continuations and sandboxing In-Reply-To: Message from William ML Leslie of "Tue, 11 Jan 2011 18:43:50 +1100." References: Message-ID: <201101111103.p0BB3j9q006294@theraft.openend.se> When I was last in the USA, I met Alessandro Warth, who now works for the Viewpoints Research Institute and whose Phd thesis which you can get on this page: http://www.tinlizzie.org/~awarth/ is about experiments in computer languages to do the sorts of things that you are interested in. He had a neat implementation of something he called 'worlds' to show off, and gave a talk about it. It's chapter 4 of his thesis. It was neat, but I got the distinct impression that the price he was paying for having unlimited cheap undo was that it was going to be quite difficult when users in different worlds, all of whom 'sprouted' from a common, but very distant ancestor and who didn't sprout from each other wanted to share information. Many other people in the room had pressing needs for such things now, and some of them were interested in PyPy. They concluded that what they would really like is a programming language which multiple heaps -- in the sense of chunks of memory that you get when you malloc -- guaranteed to be separate from each other. I couldn't tell them how hard this would be to implement in PyPy. Aside from needing to write a custom garbage collector, what else would be needed? Laura From william.leslie.ttg at gmail.com Tue Jan 11 12:51:55 2011 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Tue, 11 Jan 2011 22:51:55 +1100 Subject: [pypy-dev] Continuations and sandboxing In-Reply-To: <201101111103.p0BB3j9q006294@theraft.openend.se> References: <201101111103.p0BB3j9q006294@theraft.openend.se> Message-ID: Fijal is right, fwiw. There are lots of wonderful things that *could* be implemented, but what we *have* is the stackless transformation and a low-level sandbox system. Discussions of python architecture are always enjoyable, but they don't have a proper place, as far as I know; I often abuse #python for that. On 11 January 2011 22:03, Laura Creighton wrote: > When I was last in the USA, I met Alessandro Warth, who now works for > the Viewpoints Research Institute and whose Phd thesis which you can get > on this page: http://www.tinlizzie.org/~awarth/ is about experiments in > computer languages to do the sorts of things that you are interested in. > He had a neat implementation of something he called 'worlds' to show off, > and gave a talk about it. ?It's chapter 4 of his thesis. I first read this paper as background for pymeta - it's one of those that you just can't put down. There is one thing I felt was missing from the worlds thesis though, namely, handling arrays. To treat them like any other ecmascript object results in strange semantics - if two worlds push to the one array and both are merged, it is as if only one of the pushes occurred. Python lists have different usage patterns than javascript objects, so handling their diffs correctly and efficiently might require some experiments. > They concluded that what they would really like is a programming language > which multiple heaps -- in the sense of chunks of memory that you get > when you malloc -- guaranteed to be separate from each other. ?I couldn't > tell them how hard this would be to implement in PyPy. ?Aside from needing > to write a custom garbage collector, what else would be needed? How would that be different to separate pypy processes? Or is that more like regions in cyclone, or like subinterpreters in TCL? -- William Leslie PS: worlds are an ideal use case for first-class algebraic effects! From amauryfa at gmail.com Tue Jan 11 13:06:57 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 11 Jan 2011 13:06:57 +0100 Subject: [pypy-dev] Calling into PyPy from multi-threaded C++ In-Reply-To: References: <1294161936.1893.20.camel@localhost> Message-ID: Hi, 2011/1/11 Ren? Dudfield : > On Thu, Jan 6, 2011 at 2:35 PM, Ren? Dudfield wrote: >> Hi, >> >> here is a basic function... without the whole module stuff around it. >> I'd need to do the rest of the module another time. >> ... I know how to write extension modules, but I don't have the time nor the interest to do it currently. Would you like to do these experiments yourself, and report us the real status of threads in pypy extensions? -- Amaury Forgeot d'Arc From renesd at gmail.com Tue Jan 11 13:21:16 2011 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Tue, 11 Jan 2011 12:21:16 +0000 Subject: [pypy-dev] Calling into PyPy from multi-threaded C++ In-Reply-To: References: <1294161936.1893.20.camel@localhost> Message-ID: cool, ok. On Tue, Jan 11, 2011 at 12:06 PM, Amaury Forgeot d'Arc wrote: > Hi, > > 2011/1/11 Ren? Dudfield : >> On Thu, Jan 6, 2011 at 2:35 PM, Ren? Dudfield wrote: >>> Hi, >>> >>> here is a basic function... without the whole module stuff around it. >>> I'd need to do the rest of the module another time. >>> ... > > I know how to write extension modules, but I don't have the time nor > the interest > to do it currently. > Would you like to do these experiments yourself, and report us the real > status of threads in pypy extensions? > > -- > Amaury Forgeot d'Arc > From holger at merlinux.eu Tue Jan 11 13:58:19 2011 From: holger at merlinux.eu (holger krekel) Date: Tue, 11 Jan 2011 13:58:19 +0100 Subject: [pypy-dev] Continuations and sandboxing In-Reply-To: References: Message-ID: <20110111125819.GF2145@trillke.net> Hi Maciej, it's not clear to me what posts your are refering to. Certainly not the original post from Nathan? If you think it's a general issue then do a top-level posting and please suggest how "non-active" (defined how?) pypy developers are supposed to know if some topic is interesting without posting about it. And include some concrete rules about when side discussions are ok and when not. Probably all harder to do than improving your ignoring techniques? cheers, holger On Tue, Jan 11, 2011 at 09:48 +0200, Maciej Fijalkowski wrote: > Hello. > > This discussion have drifted considerably. In general this is the pypy > development mailing list, so personally I would like to keep > discussions only to things that are either very interesting for people > actively working on PyPy or have remote possibility that someone will > start working on them. Discussing STM without someone being interested > in actually looking is out of topic. > > PS. Feel free to correct me if anyone working actively on pypy is > finding this discussion useful for PyPy development. > > Cheers, > fijal > > On Tue, Jan 11, 2011 at 9:43 AM, William ML Leslie > wrote: > > On 11 January 2011 15:25, Nathanael D. Jones wrote: > >> Regardng #1: Sandboxing is a major concern for me. Different code will need > >> different sandboxing levels depending upon who created/approved the code. > >> I can't have everything in one sandbox - I need isolated boxes on a > >> per-request level. > > > > You will probably want to implement a capability architecture, where > > the authority to perform some action is bundled together with the > > method that invokes it. ?The language E ( http://erights.org ) was > > designed for this sort of thing (I think it was written for a MUD > > originally), and would be worth looking at even if you didn't use the > > language itself. ?Capability theory is sadly not well known, > > considering just how well it solves ever more common problems. > > > >> The idea is that if functions return the next function to call instead of > >> calling them, we have explicit tail-call elimination, and we have an > >> explicit point at which we can rebuild the sandbox and upgrade the code. > > > > Right, this is the "driver loop" implementation of TCE. ?There are > > several implementations of TCE kicking around, (including one I did > > for cpython some time ago, a decorator that would patch load ... call* > > ... return bytecodes with a call to a class that collects the function > > and arguments, and wraps the function in a driver loop that is > > bypassed when the target is marked as recursive, too). > > > > Unfortunately, using only implicit continuations, once you enter > > somebody's function, you have no way to get out of it - that person > > can except: and trap you there if they like. ?E has a way to protect > > against this, but you can't protect against the function busy-looping > > without something more (a way to kill threads, specifically, which > > probably means taking down the process). > > > >> Much data will be user scoped, and therefore lockable. > > > > What does lockable mean in this case? > > > > Under what conditions does reloading happen while a user continuation > > is in play? ?How do you want continuations to interact with the > > locking? > > > >> On Mon, Jan 10, 2011 at 5:05 PM, William ML Leslie > >> wrote: > >>> > >>> On 11 January 2011 07:18, Paolo Giarrusso wrote: > >>> >> The one issue that interests me is where you implement the persistence > >>> >> boundary - do you go with orthogonal persistence and act as if > >>> >> everything is preserved, or assume all user code is run within some > >>> >> sort of (fast and loose) transaction that can be re-entered at will, > >>> >> providing an API for persistent data access? ?The second case makes > >>> >> the reloading question a bit more reasonable, because you can always > >>> >> throw away the current delta and replay the external effects, assuming > >>> >> the interface for the external events hasn't changed significantly. > >>> > > >>> > The key question is: when would you start and commit such transactions? > >>> > > >>> > Apart from that, your idea looks very similar to Software > >>> > Transactional Memory (STM). STM restarts explicitly-marked > >>> > transactions in a thread when some other thread modifies the affected > >>> > data (which would be a data race) and commits its transaction. In your > >>> > case, a transaction is restarted when some other thread modifies the > >>> > involved code. > > > > I don't think it would be useful at all to have regular STM > > transactions delimited at the room boundary - there are surely going > > to be other people interacting with the room at the same time as you, > > and sending everyone else back to the start of the room every time > > someone modifies the room is going to get old very quickly. ?The > > boundary introduced by modifying code is probably significantly more > > coarse. > > > > If you are storing (as the transaction log) only the external effects > > (user input, random.* output), however, you can build effect > > commutators automatically. ?For example, given a code change to object > > X, you can initialise it with the previous persistent state, and > > replay the effects within the current transaction. > > > > That said, I think having a transaction per room is too coarse - per > > input or set of inputs makes even more sense. ?The idea there is that > > there is much less code to replay when two effects interfere. > > > > -- > > William Leslie > > _______________________________________________ > > pypy-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/pypy-dev > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev -- From fijall at gmail.com Tue Jan 11 14:10:58 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 11 Jan 2011 15:10:58 +0200 Subject: [pypy-dev] Continuations and sandboxing In-Reply-To: <20110111125819.GF2145@trillke.net> References: <20110111125819.GF2145@trillke.net> Message-ID: On Tue, Jan 11, 2011 at 2:58 PM, holger krekel wrote: > Hi Maciej, > > it's not clear to me what posts your are refering to. Certainly > not the original post from Nathan? > > If you think it's a general issue then do a top-level posting and please > suggest how "non-active" (defined how?) pypy developers are supposed to > know if some topic is interesting without posting about it. ?And include > some concrete rules about when side discussions are ok and when not. > Probably all harder to do than improving your ignoring techniques? Well, I can easily ignore things that are not interesting to me. My point was that discussions about topics (like capabilities) only make sense if there is someone even remotely interested in implementing that in PyPy. Otherwise discussions tend to drift into "but that's also a cool idea", which is off-topic for PyPy, since noone actually wants to implement any of those. For example I would say all GIL-removal discussions are offtopic unless someone has a will to experiment with implementing it. That's, as I said above, my personal opinion, backed by a fact that this is the list for discussing developing PyPy (so some people, like me, feel obliged to read whole discussions here). Feel free to propose some other guidelines for what is on-topic and what is off-topic for that list. As active I mean anyone doing any work on PyPy. Be it a typo in documentation :) Cheers, fijal > > cheers, > holger > > On Tue, Jan 11, 2011 at 09:48 +0200, Maciej Fijalkowski wrote: >> Hello. >> >> This discussion have drifted considerably. In general this is the pypy >> development mailing list, so personally I would like to keep >> discussions only to things that are either very interesting for people >> actively working on PyPy or have remote possibility that someone will >> start working on them. Discussing STM without someone being interested >> in actually looking is out of topic. >> >> PS. Feel free to correct me if anyone working actively on pypy is >> finding this discussion useful for PyPy development. >> >> Cheers, >> fijal >> >> On Tue, Jan 11, 2011 at 9:43 AM, William ML Leslie >> wrote: >> > On 11 January 2011 15:25, Nathanael D. Jones wrote: >> >> Regardng #1: Sandboxing is a major concern for me. Different code will need >> >> different sandboxing levels depending upon who created/approved the code. >> >> I can't have everything in one sandbox - I need isolated boxes on a >> >> per-request level. >> > >> > You will probably want to implement a capability architecture, where >> > the authority to perform some action is bundled together with the >> > method that invokes it. ?The language E ( http://erights.org ) was >> > designed for this sort of thing (I think it was written for a MUD >> > originally), and would be worth looking at even if you didn't use the >> > language itself. ?Capability theory is sadly not well known, >> > considering just how well it solves ever more common problems. >> > >> >> The idea is that if functions return the next function to call instead of >> >> calling them, we have explicit tail-call elimination, and we have an >> >> explicit point at which we can rebuild the sandbox and upgrade the code. >> > >> > Right, this is the "driver loop" implementation of TCE. ?There are >> > several implementations of TCE kicking around, (including one I did >> > for cpython some time ago, a decorator that would patch load ... call* >> > ... return bytecodes with a call to a class that collects the function >> > and arguments, and wraps the function in a driver loop that is >> > bypassed when the target is marked as recursive, too). >> > >> > Unfortunately, using only implicit continuations, once you enter >> > somebody's function, you have no way to get out of it - that person >> > can except: and trap you there if they like. ?E has a way to protect >> > against this, but you can't protect against the function busy-looping >> > without something more (a way to kill threads, specifically, which >> > probably means taking down the process). >> > >> >> Much data will be user scoped, and therefore lockable. >> > >> > What does lockable mean in this case? >> > >> > Under what conditions does reloading happen while a user continuation >> > is in play? ?How do you want continuations to interact with the >> > locking? >> > >> >> On Mon, Jan 10, 2011 at 5:05 PM, William ML Leslie >> >> wrote: >> >>> >> >>> On 11 January 2011 07:18, Paolo Giarrusso wrote: >> >>> >> The one issue that interests me is where you implement the persistence >> >>> >> boundary - do you go with orthogonal persistence and act as if >> >>> >> everything is preserved, or assume all user code is run within some >> >>> >> sort of (fast and loose) transaction that can be re-entered at will, >> >>> >> providing an API for persistent data access? ?The second case makes >> >>> >> the reloading question a bit more reasonable, because you can always >> >>> >> throw away the current delta and replay the external effects, assuming >> >>> >> the interface for the external events hasn't changed significantly. >> >>> > >> >>> > The key question is: when would you start and commit such transactions? >> >>> > >> >>> > Apart from that, your idea looks very similar to Software >> >>> > Transactional Memory (STM). STM restarts explicitly-marked >> >>> > transactions in a thread when some other thread modifies the affected >> >>> > data (which would be a data race) and commits its transaction. In your >> >>> > case, a transaction is restarted when some other thread modifies the >> >>> > involved code. >> > >> > I don't think it would be useful at all to have regular STM >> > transactions delimited at the room boundary - there are surely going >> > to be other people interacting with the room at the same time as you, >> > and sending everyone else back to the start of the room every time >> > someone modifies the room is going to get old very quickly. ?The >> > boundary introduced by modifying code is probably significantly more >> > coarse. >> > >> > If you are storing (as the transaction log) only the external effects >> > (user input, random.* output), however, you can build effect >> > commutators automatically. ?For example, given a code change to object >> > X, you can initialise it with the previous persistent state, and >> > replay the effects within the current transaction. >> > >> > That said, I think having a transaction per room is too coarse - per >> > input or set of inputs makes even more sense. ?The idea there is that >> > there is much less code to replay when two effects interfere. >> > >> > -- >> > William Leslie >> > _______________________________________________ >> > pypy-dev at codespeak.net >> > http://codespeak.net/mailman/listinfo/pypy-dev >> > >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev > -- > From Arnd.Rechenburg at tomtom.com Tue Jan 11 15:02:03 2011 From: Arnd.Rechenburg at tomtom.com (Arnd Rechenburg) Date: Tue, 11 Jan 2011 15:02:03 +0100 Subject: [pypy-dev] "OSError: [Errno 10] No child processes" Message-ID: Hi, Could someone help me to avoid the following exception? Exception "OSError: [Errno 10] No child processes" in method __del__ of ', mode 'r' at 0x00002b58059f05f8> ignored Thanks in advance, Arnd -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110111/33ff03e6/attachment.htm From amauryfa at gmail.com Tue Jan 11 15:33:46 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 11 Jan 2011 15:33:46 +0100 Subject: [pypy-dev] "OSError: [Errno 10] No child processes" In-Reply-To: References: Message-ID: Hi, 2011/1/11 Arnd Rechenburg : > Could someone help me to avoid the following exception? > > Exception "OSError: [Errno 10] No child processes" in method __del__ of > ', mode 'r' at 0x00002b58059f05f8> ignored More context would be useful: which version of pypy are you using, what are you trying to do, are there some kind of subprocesses, etc etc. -- Amaury Forgeot d'Arc From Arnd.Rechenburg at tomtom.com Tue Jan 11 16:04:10 2011 From: Arnd.Rechenburg at tomtom.com (Arnd Rechenburg) Date: Tue, 11 Jan 2011 16:04:10 +0100 Subject: [pypy-dev] "OSError: [Errno 10] No child processes" In-Reply-To: References: Message-ID: Hi Amaury, I use the following environment: Python 2.5.2 (e503e483e9ac, Dec 21 2010, 12:02:48) [PyPy 1.4.1] on linux2 For the moment I use os.execv and os.waitpid(-1, os.WNOHANG) to start and handle parallel jobs by myself. I tried already a 'try except OSError' around os.waitpid but without success. Greetings, Arnd -----Original Message----- From: Amaury Forgeot d'Arc [mailto:amauryfa at gmail.com] Sent: Dienstag, 11. Januar 2011 15:34 To: Arnd Rechenburg Cc: pypy-dev at codespeak.net Subject: Re: [pypy-dev] "OSError: [Errno 10] No child processes" Hi, 2011/1/11 Arnd Rechenburg : > Could someone help me to avoid the following exception? > > Exception "OSError: [Errno 10] No child processes" in method __del__ of > ', mode 'r' at 0x00002b58059f05f8> ignored More context would be useful: which version of pypy are you using, what are you trying to do, are there some kind of subprocesses, etc etc. -- Amaury Forgeot d'Arc From santagada at gmail.com Tue Jan 11 16:25:27 2011 From: santagada at gmail.com (Leonardo Santagada) Date: Tue, 11 Jan 2011 13:25:27 -0200 Subject: [pypy-dev] Continuations and sandboxing In-Reply-To: References: <20110111125819.GF2145@trillke.net> Message-ID: On Tue, Jan 11, 2011 at 11:10 AM, Maciej Fijalkowski wrote: > On Tue, Jan 11, 2011 at 2:58 PM, holger krekel wrote: >> Hi Maciej, >> >> it's not clear to me what posts your are refering to. Certainly >> not the original post from Nathan? >> >> If you think it's a general issue then do a top-level posting and please >> suggest how "non-active" (defined how?) pypy developers are supposed to >> know if some topic is interesting without posting about it. ?And include >> some concrete rules about when side discussions are ok and when not. >> Probably all harder to do than improving your ignoring techniques? > > Well, I can easily ignore things that are not interesting to me. > > My point was that discussions about topics (like capabilities) only > make sense if there is someone even remotely interested in > implementing that in PyPy. Otherwise discussions tend to drift into > "but that's also a cool idea", which is off-topic for PyPy, since > noone actually wants to implement any of those. Well we could tell him how to implement capabilities in pypy and how easy/hard this seems. Without looking much at it I think you could implement it by creating a new object space, like those: http://codespeak.net/pypy/dist/pypy/doc/objspace-proxies.html > For example I would say all GIL-removal discussions are offtopic > unless someone has a will to experiment with implementing it. This is also something I see as a problem in communication, there should be at least a FAQ question about this or even a good description of how pypy does Threads and why it does the same thing as CPython. > That's, as I said above, my personal opinion, backed by a fact that > this is the list for discussing developing PyPy (so some people, like > me, feel obliged to read whole discussions here). Feel free to propose > some other guidelines for what is on-topic and what is off-topic for > that list. Maybe pypy developers could steers discussions into being on topic. > As active I mean anyone doing any work on PyPy. Be it a typo in documentation :) People feel that they are helping by discussing on the list ideas and approaches that maybe the pypy team might not know. > Cheers, > fijal > -- Leonardo Santagada From amauryfa at gmail.com Tue Jan 11 16:41:47 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 11 Jan 2011 16:41:47 +0100 Subject: [pypy-dev] "OSError: [Errno 10] No child processes" In-Reply-To: References: Message-ID: 2011/1/11 Arnd Rechenburg : > Hi Amaury, > > I use the following environment: > > Python 2.5.2 (e503e483e9ac, Dec 21 2010, 12:02:48) > [PyPy 1.4.1] on linux2 > > For the moment I use os.execv and os.waitpid(-1, os.WNOHANG) to start > and handle parallel jobs by myself. > > I tried already a 'try except OSError' around os.waitpid but without > success. It's probably related to our file.__del__ which does not silence errors raised by close(). Could you file a ticket about this on the issue tracker? with a small test case if possible. https://codespeak.net/issue/pypy-dev/ -- Amaury Forgeot d'Arc From nathanael.jones at gmail.com Tue Jan 11 16:48:53 2011 From: nathanael.jones at gmail.com (Nathanael D. Jones) Date: Tue, 11 Jan 2011 10:48:53 -0500 Subject: [pypy-dev] Continuations and sandboxing In-Reply-To: References: <201101111103.p0BB3j9q006294@theraft.openend.se> Message-ID: @fijal: I agree that parts of this topic have drifted off-topic for the py-py dev mailing list, so I've created a new mailing list specifically for discussions of collaboratively-edited software. I'm inviting everyone to join the new mailing list at collaborative-games at googlegroups.com As far as the discussion regarding PyPy's capabilities with multiple-sandboxing, reloading, and continuation serialization go, I think that is a relevant discussion for this forum. I'm still very interested in understanding (a) what PyPy can already do, and (b) what is involved in adding the features I need. I'm itching to work on PyPy :) @Laura + William: Alessandro's paper concisely describes what I was planning on doing, with the exception that the sprout() method was taking place in Java. After reading the paper I see the utility of also allowing sprouting within Javascript. I am a bit concerned about data size, however. Loading all world data is an expensive op, even if it creates a simple way to perform psuedo-transactions. Nathanael http://nathanaeljones.com On Tue, Jan 11, 2011 at 6:51 AM, William ML Leslie < william.leslie.ttg at gmail.com> wrote: > Fijal is right, fwiw. There are lots of wonderful things that *could* > be implemented, but what we *have* is the stackless transformation and > a low-level sandbox system. Discussions of python architecture are > always enjoyable, but they don't have a proper place, as far as I > know; I often abuse #python for that. > > On 11 January 2011 22:03, Laura Creighton wrote: > > When I was last in the USA, I met Alessandro Warth, who now works for > > the Viewpoints Research Institute and whose Phd thesis which you can get > > on this page: http://www.tinlizzie.org/~awarth/ is about experiments in > > computer languages to do the sorts of things that you are interested in. > > He had a neat implementation of something he called 'worlds' to show off, > > and gave a talk about it. It's chapter 4 of his thesis. > > I first read this paper as background for pymeta - it's one of those > that you just can't put down. There is one thing I felt was missing > from the worlds thesis though, namely, handling arrays. To treat them > like any other ecmascript object results in strange semantics - if two > worlds push to the one array and both are merged, it is as if only one > of the pushes occurred. Python lists have different usage patterns > than javascript objects, so handling their diffs correctly and > efficiently might require some experiments. > > > They concluded that what they would really like is a programming language > > which multiple heaps -- in the sense of chunks of memory that you get > > when you malloc -- guaranteed to be separate from each other. I couldn't > > tell them how hard this would be to implement in PyPy. Aside from > needing > > to write a custom garbage collector, what else would be needed? > > How would that be different to separate pypy processes? Or is that > more like regions in cyclone, or like subinterpreters in TCL? > > -- > William Leslie > > PS: worlds are an ideal use case for first-class algebraic effects! > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110111/6175be76/attachment.htm From cfbolz at gmx.de Tue Jan 11 17:23:17 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Tue, 11 Jan 2011 17:23:17 +0100 Subject: [pypy-dev] Continuations and sandboxing In-Reply-To: References: <201101111103.p0BB3j9q006294@theraft.openend.se> Message-ID: <4D2C83F5.8010202@gmx.de> On 01/11/2011 04:48 PM, Nathanael D. Jones wrote: > As far as the discussion regarding PyPy's capabilities with > multiple-sandboxing, reloading, and continuation serialization go, I > think that is a relevant discussion for this forum. > > I'm still very interested in understanding (a) what PyPy can already do, > and (b) what is involved in adding the features I need. I'm itching to > work on PyPy :) PyPy implements two things that are related somehow: 1) process sandboxing, which is just a way to control what one instance of an interpreter is allowed to do in terms of system calls, CPU and memory used. This cannot be controlled in a fine-grained way, every sandbox needs to be its own process, and communication is not implemented yet. 2) "stackless", which is a way for interpreters to get some control over the C stack, including being able to artificially create one. This does not mean that you get serialization for free, it's just that we implemented it carefully for Python. To go further, I really think you need careful language design to get the capability model and the versioning/updating right. IMO you won't be able to extend standard Python, the semantics of it are too complex to shoehorn something with (so far) vague goals on top of it. This language design is partially orthogonal to what PyPy provides you with. After thinking about what your language should look like, you can consider to use PyPy as an implementation platform for it, or not. Cheers, Carl Friedrich From arigo at tunes.org Tue Jan 11 21:56:53 2011 From: arigo at tunes.org (Armin Rigo) Date: Tue, 11 Jan 2011 21:56:53 +0100 Subject: [pypy-dev] "OSError: [Errno 10] No child processes" In-Reply-To: References: Message-ID: Hi Amaury, On Tue, Jan 11, 2011 at 4:41 PM, Amaury Forgeot d'Arc wrote: > It's probably related to our file.__del__ which does not silence > errors raised by close(). Indeed, it seems to be the case. Arnd, to answer the original question: there is a file that you are not explicitly closing, and when its __del__ method is (later) invoked, it raises an OSError which is printed and ignored. The real problem you have there is that you should close your files (in this case, an file, so I guess it's actually a pipe returned by os.popen*()). The secondary problem that we have is what to do when file.__del__ gets an exception from calling close(). Is it ok to just ignore it? FWIW in CPython, file.__del__ also prints an error message if close() raises an exception. So from that point of view there is nothing to fix in PyPy, apart maybe making the error message a bit more explicit (CPython prints "close failed in file object destructor:..."). A bient?t, Armin. From amauryfa at gmail.com Tue Jan 11 22:09:21 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 11 Jan 2011 22:09:21 +0100 Subject: [pypy-dev] "OSError: [Errno 10] No child processes" In-Reply-To: References: Message-ID: 2011/1/11 Armin Rigo : > The secondary problem that we have is what to do when file.__del__ > gets an exception from calling close(). ?Is it ok to just ignore it? > FWIW in CPython, file.__del__ also prints an error message if close() > raises an exception. ?So from that point of view there is nothing to > fix in PyPy, apart maybe making the error message a bit more explicit > (CPython prints "close failed in file object destructor:..."). The io module chose to silence the error; I suggest to do the same, even if the reasoning is different from the comment below:: class IOBase: def __del__(self): """Destructor. Calls close().""" # The try/except block is in case this is called at program # exit time, when it's possible that globals have already been # deleted, and then the close() call might fail. Since # there's nothing we can do about such failures and they annoy # the end users, we suppress the traceback. try: self.close() except: pass -- Amaury Forgeot d'Arc From p.giarrusso at gmail.com Wed Jan 12 02:49:10 2011 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Wed, 12 Jan 2011 02:49:10 +0100 Subject: [pypy-dev] "OSError: [Errno 10] No child processes" In-Reply-To: References: Message-ID: On Tue, Jan 11, 2011 at 22:09, Amaury Forgeot d'Arc wrote: > 2011/1/11 Armin Rigo : >> The secondary problem that we have is what to do when file.__del__ >> gets an exception from calling close(). ?Is it ok to just ignore it? >> FWIW in CPython, file.__del__ also prints an error message if close() >> raises an exception. ?So from that point of view there is nothing to >> fix in PyPy, apart maybe making the error message a bit more explicit >> (CPython prints "close failed in file object destructor:..."). > > The io module chose to silence the error; > I suggest to do the same, even if the reasoning is different > from the comment below:: > > class IOBase: > ? ?def __del__(self): > ? ? ? ?"""Destructor. ?Calls close().""" > ? ? ? ?# The try/except block is in case this is called at program > ? ? ? ?# exit time, when it's possible that globals have already been > ? ? ? ?# deleted, and then the close() call might fail. ?Since > ? ? ? ?# there's nothing we can do about such failures and they annoy > ? ? ? ?# the end users, we suppress the traceback. I propose that PyPy keeps reporting the error for files opened in any write mode, because then an error on close might mean that an I/O error happened (it is possible in some scenarios on Linux, see below [1]); the programmer thus need to explicitly close the file and check for exceptions. Having writable files closed by __del__ also sounds like a bad idea if any buffering is involved, because close() time becomes unpredictable. Those are mostly arguments against __del__ _silently_ calling close(); but when the potential bug is triggered, i.e. you hit an I/O error, it's even more important to report it to the user. I also propose that any error is ignored for read-mode files - because then why should we care for it? The original error message talked about a read-mode file handle. [1] From the close(2) man page: "Not checking the return value of close() is a common but nevertheless serious programming error. It is quite possible that errors on a previous write(2) operation are first reported at the final close(). Not checking the return value when closing the file may lead to silent loss of data. This can especially be observed with NFS and with disk quota." Cheers, -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From arigo at tunes.org Thu Jan 13 13:12:42 2011 From: arigo at tunes.org (Armin Rigo) Date: Thu, 13 Jan 2011 13:12:42 +0100 Subject: [pypy-dev] "OSError: [Errno 10] No child processes" In-Reply-To: References: Message-ID: Hi Amaury, On Wed, Jan 12, 2011 at 2:49 AM, Paolo Giarrusso wrote: > I propose that PyPy keeps reporting the error for files opened in any > write mode I would also think that it's better to keep reporting errors (and for all files instead of just write mode files). In my opinion, it would, if nothing else, give users the message: "we got an error when close()ing this file automatically, but you should really close it explicitly yourself in the first place". Maybe it can even be written in that sense in the error message. A bient?t, Armin. From fijall at gmail.com Thu Jan 13 13:25:49 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 13 Jan 2011 14:25:49 +0200 Subject: [pypy-dev] "OSError: [Errno 10] No child processes" In-Reply-To: References: Message-ID: On Thu, Jan 13, 2011 at 2:12 PM, Armin Rigo wrote: > Hi Amaury, > > On Wed, Jan 12, 2011 at 2:49 AM, Paolo Giarrusso wrote: >> I propose that PyPy keeps reporting the error for files opened in any >> write mode > > I would also think that it's better to keep reporting errors (and for > all files instead of just write mode files). ?In my opinion, it would, > if nothing else, give users the message: "we got an error when > close()ing this file automatically, but you should really close it > explicitly yourself in the first place". ?Maybe it can even be written > in that sense in the error message. > How about a link to differencies between pypy and cpython, especially about closing files? Cheers, fijal > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From arigo at tunes.org Thu Jan 13 13:43:40 2011 From: arigo at tunes.org (Armin Rigo) Date: Thu, 13 Jan 2011 13:43:40 +0100 Subject: [pypy-dev] "OSError: [Errno 10] No child processes" In-Reply-To: References: Message-ID: Hi, On Thu, Jan 13, 2011 at 1:25 PM, Maciej Fijalkowski wrote: >> close()ing this file automatically, but you should really close it >> explicitly yourself in the first place". ?Maybe it can even be written >> in that sense in the error message. > > How about a link to differencies between pypy and cpython, especially > about closing files? Yes please :-) Armin From p.giarrusso at gmail.com Thu Jan 13 13:49:50 2011 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Thu, 13 Jan 2011 13:49:50 +0100 Subject: [pypy-dev] "OSError: [Errno 10] No child processes" In-Reply-To: References: Message-ID: On Thu, Jan 13, 2011 at 13:43, Armin Rigo wrote: > Hi, > > On Thu, Jan 13, 2011 at 1:25 PM, Maciej Fijalkowski wrote: >>> close()ing this file automatically, but you should really close it >>> explicitly yourself in the first place". ?Maybe it can even be written >>> in that sense in the error message. >> How about a link to differencies between pypy and cpython, especially >> about closing files? And I guess you can also mention Jython and IronPython as being PyPy-like (if the behavior just depends on having a GC instead of refcounting). Disclaimer: I've never used IronPython. > Yes please :-) -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From bhartsho at yahoo.com Sat Jan 15 03:09:24 2011 From: bhartsho at yahoo.com (Hart's Antler) Date: Fri, 14 Jan 2011 18:09:24 -0800 (PST) Subject: [pypy-dev] PyPy on Android Message-ID: <493797.19850.qm@web114002.mail.gq1.yahoo.com> Hi all PyPy-devs, I'm not sure if my error is simply a 64bit issue, and it would work from 32bit linux. ?Can i simply -DMAX_LONG not to be 64bit and then using the Android NDK gcc (which i think is 32bits) is ok? ?Or should i recompile Android's NDK gcc so that its 64bits? Almost working example:http://pastebin.com/bjwEh8E7 Error:/home/brett/android-ndk-r5/toolchains/arm-linux-androideabi-4.4.3/prebuilt/linux-x86/bin/arm-linux-androideabi-gcc -MMD -MP -MF /tmp/usession-default-6/obj/local/armeabi/objs/testing_1/testing_1.o.d -fpic -ffunction-sections -funwind-tables -fstack-protector -D__ARM_ARCH_5__ -D__ARM_ARCH_5T__ -D__ARM_ARCH_5E__ -D__ARM_ARCH_5TE__ ?-Wno-psabi -march=armv5te -mtune=xscale -msoft-float -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing -finline-limit=64 -I/tmp/usession-default-6/jni -DANDROID -I/home/brett/RPythonic/pypy/pypy/translator/c -DPYPY_STANDALONE -Wa,--noexecstack -O2 -DNDEBUG -g -I/home/brett/android-ndk-r5/platforms/android-3/arch-arm/usr/include -c ?/tmp/usession-default-6/jni/testing_1.c -o /tmp/usession-default-6/obj/local/armeabi/objs/testing_1/testing_1.oIn file included from /home/brett/RPythonic/pypy/pypy/translator/c/src/g_prerequisite.h:7,?? ? ? ? ? ? ? ? from /tmp/usession-default-6/jni/common_header.h:44,?? ? ? ? ? ? ? ? from /tmp/usession-default-6/jni/testing_1.c:1:/home/brett/RPythonic/pypy/pypy/translator/c/src/commondefs.h:58:6: error: #error "error in LONG_MAX (64-bit sources but a 32-bit compiler?)"/home/brett/RPythonic/pypy/pypy/translator/c/src/commondefs.h:61:6: error: #error "unsupported value for LONG_MIN"/tmp/usession-default-6/jni/testing_1.c:85: error: expected specifier-qualifier-list before 'PyObject'/tmp/usession-default-6/jni/testing_1.c:404: error: expected '=', ',', ';', 'asm' or '__attribute__' before '*' token/tmp/usession-default-6/jni/testing_1.c:409: error: expected ')' before '*' tokenIn file included from /home/brett/RPythonic/pypy/pypy/translator/c/src/g_include.h:58,?? ? ? ? ? ? ? ? from /tmp/usession-default-6/jni/testing_1.c:564: -brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110114/019bfeab/attachment.htm From p.giarrusso at gmail.com Sat Jan 15 12:42:19 2011 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Sat, 15 Jan 2011 12:42:19 +0100 Subject: [pypy-dev] PyPy on Android In-Reply-To: <493797.19850.qm@web114002.mail.gq1.yahoo.com> References: <493797.19850.qm@web114002.mail.gq1.yahoo.com> Message-ID: On Sat, Jan 15, 2011 at 03:09, Hart's Antler wrote: > > Hi all PyPy-devs, > I'm not sure if my error is simply a 64bit issue, and it would work from 32bit linux. ?Can i simply -DMAX_LONG not to be 64bit and then using the Android NDK gcc (which i think is 32bits) is ok? I guess MAX_LONG is calculated internally. You probably need to use a different Python interpreter, a 32bit one, to build PyPy (instructions on the website). If none is available, you need to build one from scratch. Given that you're cross-compiling PyPy, I wonder whether Python on ARM (the one you'd use in a non-cross-platform build) is any different from Python on x86, and if this difference is important; surely it is different from Python on x86_64, as you noticed. > Or should i recompile Android's NDK gcc so that its 64bits? Maybe I missed something, but is that even possible? Your toolchain seems to target ARM, and according to common sense and Wikipedia [1] there are no 64bit ARM processors. It seems that Android supports other target architectures, even x86 it seems, still I doubt the existence of 64bit phones :-D. [1] http://en.wikipedia.org/wiki/ARM_architecture > Almost working example: > http://pastebin.com/bjwEh8E7 > Error: > /home/brett/android-ndk-r5/toolchains/arm-linux-androideabi-4.4.3/prebuilt/linux-x86/bin/arm-linux-androideabi-gcc -MMD -MP -MF /tmp/usession-default-6/obj/local/armeabi/objs/testing_1/testing_1.o.d -fpic -ffunction-sections -funwind-tables -fstack-protector -D__ARM_ARCH_5__ -D__ARM_ARCH_5T__ -D__ARM_ARCH_5E__ -D__ARM_ARCH_5TE__ ?-Wno-psabi -march=armv5te -mtune=xscale -msoft-float -mthumb -Os -fomit-frame-pointer -fno-strict-aliasing -finline-limit=64 -I/tmp/usession-default-6/jni -DANDROID -I/home/brett/RPythonic/pypy/pypy/translator/c -DPYPY_STANDALONE -Wa,--noexecstack -O2 -DNDEBUG -g -I/home/brett/android-ndk-r5/platforms/android-3/arch-arm/usr/include -c ?/tmp/usession-default-6/jni/testing_1.c -o /tmp/usession-default-6/obj/local/armeabi/objs/testing_1/testing_1.o > In file included from /home/brett/RPythonic/pypy/pypy/translator/c/src/g_prerequisite.h:7, > ?? ? ? ? ? ? ? ? from /tmp/usession-default-6/jni/common_header.h:44, > ?? ? ? ? ? ? ? ? from /tmp/usession-default-6/jni/testing_1.c:1: > /home/brett/RPythonic/pypy/pypy/translator/c/src/commondefs.h:58:6: error: #error "error in LONG_MAX (64-bit sources but a 32-bit compiler?)" > /home/brett/RPythonic/pypy/pypy/translator/c/src/commondefs.h:61:6: error: #error "unsupported value for LONG_MIN" > /tmp/usession-default-6/jni/testing_1.c:85: error: expected specifier-qualifier-list before 'PyObject' > /tmp/usession-default-6/jni/testing_1.c:404: error: expected '=', ',', ';', 'asm' or '__attribute__' before '*' token > /tmp/usession-default-6/jni/testing_1.c:409: error: expected ')' before '*' token > In file included from /home/brett/RPythonic/pypy/pypy/translator/c/src/g_include.h:58, > ?? ? ? ? ? ? ? ? from /tmp/usession-default-6/jni/testing_1.c:564: > > -brett > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From konryd at gmail.com Mon Jan 17 12:30:29 2011 From: konryd at gmail.com (Konrad Delong) Date: Mon, 17 Jan 2011 12:30:29 +0100 Subject: [pypy-dev] In search of references Message-ID: Hi all, I am currently finishing up my master thesis. In it, I'd like to mention PyPy, but I think I need references to a couple of statements in order... well... not to lie :) So here it is: 1. PyPy is based on an approach first used in implementing Smalltalk-80 (the idea of implementing a dynamic language in its own subset, then statically analyzing and compiling the interpreter) 2. The main reason PyPy decided to implement GIL was for the c extensions to run unchanged. I obviously don't want you to do my job for me, but I searched a lot on this topic already, and any hint where to look further would be appreciated. cheers, Konrad From cfbolz at gmx.de Mon Jan 17 13:13:29 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 17 Jan 2011 13:13:29 +0100 Subject: [pypy-dev] In search of references In-Reply-To: References: Message-ID: <4D343269.1060905@gmx.de> On 01/17/2011 12:30 PM, Konrad Delong wrote: > Hi all, > > I am currently finishing up my master thesis. In it, I'd like to > mention PyPy, but I think I need references to a couple of statements > in order... well... not to lie :) > > So here it is: > > 1. PyPy is based on an approach first used in implementing > Smalltalk-80 (the idea of implementing a dynamic language in its own > subset, then statically analyzing and compiling the interpreter) Read this: http://portal.acm.org/citation.cfm?id=1176753 It has a discussion about the relation to Squeak at the end. > 2. The main reason PyPy decided to implement GIL was for the c > extensions to run unchanged. There is no "official" reference for this. The reason why we have a GIL is because doing anything else is significantly harder, and would require a GC that works well with threads. This is a non-trivial task. In general, if you are looking for more "scientific" reference, look here: http://codespeak.net/pypy/dist/pypy/doc/extradoc.html There is also an article about to be published about the main JIT optimization here: http://codespeak.net/svn/pypy/extradoc/talk/pepm2011/bolz-allocation-removal.pdf Cheers, Carl Friedrich From p.giarrusso at gmail.com Mon Jan 17 13:40:22 2011 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Mon, 17 Jan 2011 13:40:22 +0100 Subject: [pypy-dev] In search of references In-Reply-To: References: Message-ID: On Mon, Jan 17, 2011 at 12:30, Konrad Delong wrote: > Hi all, > > I am currently finishing up my master thesis. In it, I'd like to > mention PyPy, but I think I need references to a couple of statements > in order... well... not to lie :) > > So here it is: > > 1. PyPy is based on an approach first used in implementing > Smalltalk-80 (the idea of implementing a dynamic language in its own > subset, then statically analyzing and compiling the interpreter) > 2. The main reason PyPy decided to implement GIL was for the c > extensions to run unchanged. Beyond what Carl wrote, no C extension ran unchanged until the introduction of cpyext in 2010: http://morepypy.blogspot.com/2010/04/using-cpython-extension-modules-with.html Bye -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From jacob at openend.se Thu Jan 20 16:16:44 2011 From: jacob at openend.se (Jacob =?iso-8859-1?q?Hall=E9n?=) Date: Thu, 20 Jan 2011 16:16:44 +0100 Subject: [pypy-dev] Branches in the mercurial repository Message-ID: <201101201616.46698.jacob@openend.se> I have made a cleanup of branches in the mercurial repository, closing about 40 of them. We currently have 34 open branches and I think that some of them may no longer be useful. Please check the branches you are responsible for and close them if they are no longer needed. By keeping the repository free from cruft we make it a lot easier for newcomers to navigate. The procedure for closing a branch is simple: hg up -C hg commit --close-branch -m 'Closing disused branch.' hg up -C default # switch back to "good" branch Here is a list of currently open branches: Alex mixed-submodules Mon Jan 03 12:50:00 2011 Alexander sepcomp Wed Dec 09 12:15:29 2009 Amaury build-external Tue Jun 10 10:48:46 2008 Amaury cpyext-init-cleanup Wed Jun 09 15:56:49 2010 Amaury separate-compilation Fri Jan 22 17:32:50 2010 Amaury unicode_filename Fri Apr 24 16:17:08 2009 Amaury unicode_filename-2 Fri Jul 16 20:39:39 2010 Antonio cli-jit Wed Jan 20 09:09:23 2010 Antonio jitypes2 Thu Jan 20 10:16:27 2011 Armin 32ptr-on-64bit Fri Oct 15 13:13:44 2010 Armin inline-shadowstack Mon Dec 06 10:46:52 2010 Armin jit-longlong Fri Jan 14 16:17:13 2011 Armin jit-lsprofile Wed Jan 05 18:26:14 2011 Armin jit-tagged Mon Dec 20 10:36:09 2010 Bartosz fast-ctypes Thu Aug 19 11:35:15 2010 Carl bridges-experimental Thu Jan 21 15:33:18 2010 Carl reflex-support Tue Jan 11 14:32:08 2011 Carl unroll-safe-if-const-arg Tue Jan 26 15:36:20 2010 Dan gdbm Thu Nov 25 00:00:30 2010 Daniel micronumpy Sun Sep 12 08:59:28 2010 Daniel micronumpy-resync Tue Oct 19 05:19:25 2010 Daniel ootype-virtualrefs Sun Nov 07 19:39:49 2010 Daniel psycopg2compatibility Thu Jan 13 16:05:44 2011 David arm-backed-float Mon Jan 17 17:31:39 2011 David arm-backend-2 Fri Oct 08 01:35:03 2010 Hakan jit-short-preamble Mon Jan 17 07:33:06 2011 Jean-Philippe avm Mon Jul 26 22:47:11 2010 Maciej guard-improvements Tue Jan 11 15:49:26 2011 Maciej out-of-line-guards Mon Jan 17 19:00:12 2011 Michael bytearray Thu Jan 20 13:44:58 2011 Samuele ctypes-stable Fri Apr 11 15:44:57 2008 Samuele run-django Sat Jul 12 09:46:07 2008 Travis jit-stackless Mon Apr 05 16:48:39 2010 holger pytest2 Wed Jan 19 19:24:33 2011 Jacob Hall?n From samuele.pedroni at gmail.com Thu Jan 20 17:33:39 2011 From: samuele.pedroni at gmail.com (Samuele Pedroni) Date: Thu, 20 Jan 2011 17:33:39 +0100 Subject: [pypy-dev] Branches in the mercurial repository In-Reply-To: <201101201616.46698.jacob@openend.se> References: <201101201616.46698.jacob@openend.se> Message-ID: > Samuele ? ? ? ? ? ? ?ctypes-stable ? ? ? ? ? ?Fri Apr 11 15:44:57 2008 > Samuele ? ? ? ? ? ? ?run-django ? ? ? ? ? ? ? Sat Jul 12 09:46:07 2008 these corresponded to results of the Google contracts, whether the svn repo is staying around read-only or going away, they were referred through urls to the svn repo of course. I suppose they are past their material usefulness, I suppose they could be closed, don't know if people have a different opinion on this. Samuele From alex.gaynor at gmail.com Thu Jan 20 17:42:51 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Thu, 20 Jan 2011 10:42:51 -0600 Subject: [pypy-dev] Branches in the mercurial repository In-Reply-To: References: <201101201616.46698.jacob@openend.se> Message-ID: On Thu, Jan 20, 2011 at 10:33 AM, Samuele Pedroni wrote: > > Samuele ctypes-stable Fri Apr 11 15:44:57 2008 > > Samuele run-django Sat Jul 12 09:46:07 2008 > > these corresponded to results of the Google contracts, whether the svn > repo is staying around read-only or going away, they were referred > through urls to the svn repo of course. I suppose they are past their > material usefulness, I suppose they could be closed, don't know if > people have a different opinion on this. > > Samuele > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > closed just means that they are no longer being used (and won't be shown in the default list of branches), I tihnk those are safe to close. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero "Code can always be simpler than you think, but never as simple as you want" -- Me -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110120/08810f5e/attachment.htm From tobami at googlemail.com Thu Jan 20 22:00:41 2011 From: tobami at googlemail.com (Miquel Torres) Date: Thu, 20 Jan 2011 22:00:41 +0100 Subject: [pypy-dev] New version of Codespeed (0.7) for speed.pypy.org Message-ID: Hi all, I want to announce the release of Codespeed 0.7, the version which is now powering speed.pypy.org The announcement has been made on http://groups.google.com/group/codespeed/browse_thread/thread/3dd2d9491b26d044, but I will paste it here: -------------------------------------------------- I will briefly describe the most significant improvements. Result Reports -------------- While the previous Codespeed version allowed a project to analyse and visualize performance changes and compare different versions or even "competitors", it didn't address well a use-case: a core developer that is interested in the daily performance variations. You had to go to the Changes view, click on the desired revision, maybe another executable, and again for every revision you wanted to check. That use case is now addressed by Reports. They are a summary of the most significant changes on every run. It's a ultra-condensed Changes table. It is shown on the main page, and you can even subscribe to a RSS feed. That way you only visit the page when you are interested in some particular revision or change. After a few weeks of use, you may find out that a different "algorithm" is better (for example dropping trends or trends triggering the red state...), or that the thresholds need to be tweaked. And there are surely other interesting run statistics that could be gathered and shown. I eagerly await the feedback and ensuing discussion ;-) Support for small screen sizes ------------------------------ Using CSS media queries, the site layout adapts to smaller screens. It is optimized for 3 sizes: - Desktop: normal layout for big screens - Netbook, tablets (~1000px): Similar layout but smaller margins, smaller plots. Coincidentally, it is about the same width of Maciej's laptop screen ;-) - Smartphones: iPhones and Androids in landscape mode should be now able to visit the site without problems. Though it can be tested just by resizing the browser window, it would be great to get some feedback of anyone actually using the above mentioned devices! Performance improvements ------------------------ Exarkun made the timeline view faster by pre-fetching all related data. The timeline grid took 1.46 seconds to load, and a single plot 0.45 seconds. They now need 1.26 and 0.26 seconds respectively. A big improvement! There is yet another improvement waiting in the wings, but we still need to figure out some regression it introduces. The Changes view is also much faster (from 480ms to 210ms), because it now uses the cached table stored in the corresponding report (only when viewing with the default trend=10 ). A last detail --------------- Another change you may notice is that the default checked executables in the Comparison view are now configurable (I didn't forget Maciej! :). So they are not all selected per default any more, so the plot shouldn't look any longer like the Flying Spaghetti Monster ;-) I'll leave it there. I hope you enjoy the changes! Miquel From fijall at gmail.com Thu Jan 20 23:06:15 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 21 Jan 2011 00:06:15 +0200 Subject: [pypy-dev] New version of Codespeed (0.7) for speed.pypy.org In-Reply-To: References: Message-ID: Great! On Thu, Jan 20, 2011 at 11:00 PM, Miquel Torres wrote: > Hi all, > > I want to announce the release of Codespeed 0.7, the version which is > now powering speed.pypy.org > > The announcement has been made on > http://groups.google.com/group/codespeed/browse_thread/thread/3dd2d9491b26d044, > but I will paste it here: > > -------------------------------------------------- > ?I will briefly describe the most significant improvements. > > > Result Reports > -------------- > While the previous Codespeed version allowed a project to analyse and > visualize performance changes and compare different versions or even > "competitors", it didn't address well a use-case: a core developer > that is interested in the daily performance variations. > > You had to go to the Changes view, click on the desired revision, > maybe another executable, and again for every revision you wanted to > check. > > That use case is now addressed by Reports. They are a summary of the > most significant changes on every run. It's a ultra-condensed Changes > table. It is shown on the main page, and you can even subscribe to a > RSS feed. That way you only visit the page when you are interested in > some particular revision or change. > > After a few weeks of use, you may find out that a different > "algorithm" is better (for example dropping trends or trends > triggering the red state...), or that the thresholds need to be > tweaked. And there are surely other interesting run statistics that > could be gathered and shown. I eagerly await the feedback and ensuing > discussion ;-) > > > Support for small screen sizes > ------------------------------ > > Using CSS media queries, the site layout adapts to smaller screens. It > is optimized for 3 sizes: > - Desktop: normal layout for big screens > - Netbook, tablets (~1000px): Similar layout but smaller margins, > smaller plots. Coincidentally, it is about the same width of Maciej's > laptop screen ;-) > - Smartphones: iPhones and Androids in landscape mode should be now > able to visit the site without problems. > > Though it can be tested just by resizing the browser window, it would > be great to get some feedback of anyone actually using the above > mentioned devices! > > > Performance improvements > ------------------------ > > Exarkun made the timeline view faster by pre-fetching all related > data. The timeline grid took 1.46 seconds to load, and a single plot > 0.45 seconds. They now need 1.26 and 0.26 seconds respectively. A big > improvement! > > There is yet another improvement waiting in the wings, but we still > need to figure out some regression it introduces. > > The Changes view is also much faster (from 480ms to 210ms), because it > now uses the cached table stored in the corresponding report (only > when viewing with the default trend=10 ). > > > A last detail > --------------- > > Another change you may notice is that the default checked executables > in the Comparison view are now configurable (I didn't forget Maciej! > :). So they are not all selected per default any more, so the plot > shouldn't look any longer like the Flying Spaghetti Monster ;-) > > I'll leave it there. I hope you enjoy the changes! > > Miquel > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From anto.cuni at gmail.com Thu Jan 20 23:11:28 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu, 20 Jan 2011 23:11:28 +0100 Subject: [pypy-dev] New version of Codespeed (0.7) for speed.pypy.org In-Reply-To: References: Message-ID: <4D38B310.304@gmail.com> Hi Miquel On 20/01/11 22:00, Miquel Torres wrote: > Hi all, > > I want to announce the release of Codespeed 0.7, the version which is > now powering speed.pypy.org wow, that's very nice, thank you once more :-). From my side, there is only one feature that I miss a lot, which is the possibility to benchmark a branch. IIRC, at the moment if you just run benchmarks on a branch, they are appended to the list of results, which can be a bit confusing especially if you look at them weeks later. I think that maybe it's possible to do it by (ab)using an additional environment, but probably a more builtin solution is better. Do you have any idea/opinion/suggestion on this? ciao, Anto From tobami at googlemail.com Fri Jan 21 08:49:50 2011 From: tobami at googlemail.com (Miquel Torres) Date: Fri, 21 Jan 2011 08:49:50 +0100 Subject: [pypy-dev] New version of Codespeed (0.7) for speed.pypy.org In-Reply-To: <4D38B310.304@gmail.com> References: <4D38B310.304@gmail.com> Message-ID: @Anto Yes, branches are a pending item that has been requested a couple of times now. The current solution is actually not to abuse an environment like you say, but to create a new project for a branch. That way it gets cleanly separated, and in a way branches are like different projects. But it is of course not optimal. Technically it is very easy to come up with several solutions (add a branch dimension, for example), but interface-wise it is not easy to find something that doesn't clutter things. So one idea would be: - Every revision has a branch associated to it (a problem though is that subversion has the same revision number for all branches, while mercurial and git not) - Branch would be a selectable parameter on the left menu box, just like executables or environments - You would be able to compare a branch using the comparison view, but another possibility is that the timeline view doesn't have branch as a radio button, but as a checkbox. That means that a plot is associated only to an executable and environment, and that you can choose to display either all branches in the same plot (trunk or master + any branch) or only the ones you select. - There are problems with this approach. In the Changes view, when you select a revision that doesn't have results for the currently selected branch, nothing would be displayed. You would need to blindly search for your results. An option is to always display all branches for a revision, but sometimes a revision will contain results for a branch, sometimes for others. Another idea would be to associate executables to a branch. This avoids the problem in the Changes view. But it is not really much different from the current solution of creating a different project for each branch. I would gladly hear other ideas. Cheers, Miquel 2011/1/20 Antonio Cuni : > Hi Miquel > > On 20/01/11 22:00, Miquel Torres wrote: >> >> Hi all, >> >> I want to announce the release of Codespeed 0.7, the version which is >> now powering speed.pypy.org > > wow, that's very nice, thank you once more :-). > > From my side, there is only one feature that I miss a lot, which is the > possibility to benchmark a branch. > > IIRC, at the moment if you just run benchmarks on a branch, they are > appended to the list of results, which can be a bit confusing especially if > you look at them weeks later. > > I think that maybe it's possible to do it by (ab)using an additional > environment, but probably a more builtin solution is better. Do you have any > idea/opinion/suggestion on this? > > ciao, > Anto > From p.giarrusso at gmail.com Fri Jan 21 11:24:17 2011 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Fri, 21 Jan 2011 11:24:17 +0100 Subject: [pypy-dev] New version of Codespeed (0.7) for speed.pypy.org In-Reply-To: References: <4D38B310.304@gmail.com> Message-ID: Hi Miquel! > I would gladly hear other ideas. Here are my two cents! On Fri, Jan 21, 2011 at 08:49, Miquel Torres wrote: > So one idea would be: > - Every revision has a branch associated to it (a problem though is > that subversion has the same revision number for all branches, while > mercurial and git not) This suggests (to me) that a revision would be associated to a specific branch, but then I don't get expressions like "all the branches for a revision". Maybe you mean that you would replace revisions with pairs (Revision, Branch)? [...] > - There are problems with this approach. In the Changes view, when you > select a revision that doesn't have results for the currently selected > branch, nothing would be displayed. You're referring to an SVN context, right? Otherwise I can't make sense of it. I guess you can use, on other branches, the "closest" revision available, after defining a linear order on revisions across branches. You have that order in SVN; since you don't have that in Git, why don't you just use the _date_ of a revision as the ordering key? [For the data structures to implement this, I've seen even seen interfaces to tree-based dictionaries allowing navigating to the key-predecessor and key-successor, which might be especially handy here (such as NavigableMap in Java, dunno about Python).] Of course, that's suitable just as default, so you need to be able to navigate between revisions independently for each branch. Please note that with complex merge histories, using the date does not necessarily make sense - so that at least Git tools provide many linearization algorithms. The problem is that a commit might be made on a certain date in some repository, but be merged on mainline much later, especially when many developers are not committers themselves; one solution is to consider only revisions which were committed directly on the branch or merged shortly after. I also would believe this use case does not appear for PyPy. See help about git log --topo-order and --date-order (such docs used to be more complete, though). > You would need to blindly search > for your results. An option is to always display all branches for a > revision, but sometimes a revision will contain results for a branch, > sometimes for others. Outside of the SVN context, I'm not sure I get what you mean here, because revisions in different branches are not necessarily comparable. -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From eva_maia at sapo.pt Fri Jan 21 16:03:55 2011 From: eva_maia at sapo.pt (Eva Maia) Date: Fri, 21 Jan 2011 15:03:55 +0000 Subject: [pypy-dev] rPython Message-ID: <4D39A05B.60209@sapo.pt> Hi, It is possible through the PyPy toolchain check if a program is a rPython program? I know the tool Pylint (RPylint) gives an idea, but it seems to me that this tool gives some errors that result from being a little outdated. Thanks, Eva Maia. From fijall at gmail.com Fri Jan 21 16:15:40 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 21 Jan 2011 17:15:40 +0200 Subject: [pypy-dev] rPython In-Reply-To: <4D39A05B.60209@sapo.pt> References: <4D39A05B.60209@sapo.pt> Message-ID: On Fri, Jan 21, 2011 at 5:03 PM, Eva Maia wrote: > Hi, > > It is possible through the PyPy toolchain check if a program is a > rPython program? > > I know the tool Pylint (RPylint) gives an idea, but it seems to me that > this tool gives some errors that result from being a little outdated. > > Thanks, > > Eva Maia. > _______________________________________________ Hi. This tool is completely broken. Primary reason is that this tool operates on the level of source code, while RPython operates on live python objects and that's a huge difference. That said, I don't think there is any other way of finding what's RPython without compiling it. In fact, RPython is defined as a thing that can be accepted by our translation toolchain :-) PS. Why you want to write RPython in the first place? Cheers, fijal From tobami at googlemail.com Fri Jan 21 16:24:33 2011 From: tobami at googlemail.com (Miquel Torres) Date: Fri, 21 Jan 2011 16:24:33 +0100 Subject: [pypy-dev] New version of Codespeed (0.7) for speed.pypy.org In-Reply-To: References: <4D38B310.304@gmail.com> Message-ID: Hi Paolo, >> - There are problems with this approach. In the Changes view, when you >> select a revision that doesn't have results for the currently selected >> branch, nothing would be displayed. > > You're referring to an SVN context, right? Otherwise I can't make sense of it. The opposite. For subversion, revisions mark the whole repository, so a revision could have results for both trunk and a branch. That is not the case in mercurial. Regarding my explanation, I understand it is a bit difficult to visualize this UI issues without images. >> You would need to blindly search >> for your results. An option is to always display all branches for a >> revision, but sometimes a revision will contain results for a branch, >> sometimes for others. > Outside of the SVN context, I'm not sure I get what you mean here, > because revisions in different branches are not necessarily > comparable. Same as above. So the differences between version control systems really suggest that you cannot add a branch dimension to a revision. More like a result is not only associated to a project, revision, and executable, but also to a branch... 2011/1/21 Paolo Giarrusso : > Hi Miquel! > >> I would gladly hear other ideas. > Here are my two cents! > > On Fri, Jan 21, 2011 at 08:49, Miquel Torres wrote: >> So one idea would be: >> - Every revision has a branch associated to it (a problem though is >> that subversion has the same revision number for all branches, while >> mercurial and git not) > This suggests (to me) that a revision would be associated to a > specific branch, but then I don't get expressions like "all the > branches for a revision". Maybe you mean that you would replace > revisions with pairs (Revision, Branch)? > [...] >> - There are problems with this approach. In the Changes view, when you >> select a revision that doesn't have results for the currently selected >> branch, nothing would be displayed. > > You're referring to an SVN context, right? Otherwise I can't make sense of it. > > I guess you can use, on other branches, the "closest" revision > available, after defining a linear order on revisions across branches. > You have that order in SVN; since you don't have that in Git, why > don't you just use the _date_ of a revision as the ordering key? > [For the data structures to implement this, I've seen even seen > interfaces to tree-based dictionaries allowing navigating to the > key-predecessor and key-successor, which might be especially handy > here (such as NavigableMap in Java, dunno about Python).] > Of course, that's suitable just as default, so you need to be able to > navigate between revisions independently for each branch. > > Please note that with complex merge histories, using the date does not > necessarily make sense - so that at least Git tools provide many > linearization algorithms. The problem is that a commit might be made > on a certain date in some repository, but be merged on mainline much > later, especially when many developers are not committers themselves; > one solution is to consider only revisions which were committed > directly on the branch or merged shortly after. > I also would believe this use case does not appear for PyPy. > See help about git log --topo-order and --date-order (such docs used > to be more complete, though). > >> You would need to blindly search >> for your results. An option is to always display all branches for a >> revision, but sometimes a revision will contain results for a branch, >> sometimes for others. > Outside of the SVN context, I'm not sure I get what you mean here, > because revisions in different branches are not necessarily > comparable. > -- > Paolo Giarrusso - Ph.D. Student > http://www.informatik.uni-marburg.de/~pgiarrusso/ > From dimaqq at gmail.com Fri Jan 21 19:07:39 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Fri, 21 Jan 2011 11:07:39 -0700 Subject: [pypy-dev] rPython In-Reply-To: <4D39A05B.60209@sapo.pt> References: <4D39A05B.60209@sapo.pt> Message-ID: Hi, Eva I assume you are asking if there is a program is a valid rpython program, right? Since rpython is a subset of python, you can check if it is a valid python program and then verify the restrictions imposed by rpython. "RPython: a Step Towards Reconciling Dynamically and Statically Typed OO Languages" lists these. PyPy impementation of rpython appears to check these constraints at "initialization" time, which is in a sense, python runtime. The result is, I think, to try and see if it compiles. Come to think of it, vaildity and correctness are blurred in dynamic languages. Syntax is checked, but then Imports and asserts can affect if all of the program is even considered. Alternatively if you want to quickly distinguish practical rpython code from python, I guess you could look for clues such as what types are used, what modules imported, etc. Dima On 21 January 2011 08:03, Eva Maia wrote: > Hi, > > It is possible through the PyPy toolchain check if a program is a > rPython program? > > I know the tool Pylint (RPylint) gives an idea, but it seems to me that > this tool gives some errors that result from being a little outdated. > > Thanks, > > Eva Maia. > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From anto.cuni at gmail.com Fri Jan 21 19:14:28 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Fri, 21 Jan 2011 19:14:28 +0100 Subject: [pypy-dev] New version of Codespeed (0.7) for speed.pypy.org In-Reply-To: References: <4D38B310.304@gmail.com> Message-ID: <4D39CD04.3050104@gmail.com> On 21/01/11 08:49, Miquel Torres wrote: > @Anto > > Yes, branches are a pending item that has been requested a couple of times now. yes, I think most of the requests has been by me :) > The current solution is actually not to abuse an environment like you > say, but to create a new project for a branch. That way it gets > cleanly separated, and in a way branches are like different projects. > But it is of course not optimal. Technically it is very easy to come > up with several solutions (add a branch dimension, for example), but > interface-wise it is not easy to find something that doesn't clutter > things. Uhm, I don't think that using a different project is a good idea. For branches, we are usually not much interested in the branch history, but in the comparison with trunk (ideally, with trunk at the point we created the branch, or at the point of the last merge from trunk). As for visualize changes, I think that we don't need anything fancy, for example it would be already immensely useful to have the possibility of displaying the Changes page in a way that it compares the performances of the branch against trunk. ciao, Anto From exarkun at twistedmatrix.com Fri Jan 21 19:34:11 2011 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Fri, 21 Jan 2011 18:34:11 -0000 Subject: [pypy-dev] New version of Codespeed (0.7) for speed.pypy.org In-Reply-To: <4D39CD04.3050104@gmail.com> References: <4D38B310.304@gmail.com> <4D39CD04.3050104@gmail.com> Message-ID: <20110121183411.1851.1980573527.divmod.xquotient.20@localhost.localdomain> On 06:14 pm, anto.cuni at gmail.com wrote: >On 21/01/11 08:49, Miquel Torres wrote: >>@Anto >> >>Yes, branches are a pending item that has been requested a couple of >>times now. > >yes, I think most of the requests has been by me :) >>The current solution is actually not to abuse an environment like you >>say, but to create a new project for a branch. That way it gets >>cleanly separated, and in a way branches are like different projects. >>But it is of course not optimal. Technically it is very easy to come >>up with several solutions (add a branch dimension, for example), but >>interface-wise it is not easy to find something that doesn't clutter >>things. > >Uhm, I don't think that using a different project is a good idea. For >branches, we are usually not much interested in the branch history, but >in the >comparison with trunk (ideally, with trunk at the point we created the >branch, >or at the point of the last merge from trunk). > >As for visualize changes, I think that we don't need anything fancy, >for >example it would be already immensely useful to have the possibility of >displaying the Changes page in a way that it compares the performances >of the >branch against trunk. How about a "Branches" checkbox (per project? per executable? per graph? One of those maybe). When it's checked, branch results within the revision horizon (last 10, 50, 200, etc) get plotted on the same graph as the trunk data is plotted. Each branch could be a different color, perhaps (but would at least have hover info telling you what it is). This implies adding a branch column to the results table (or is it the revisions table?). Maybe that's just the obvious way to do it and everyone else already thought of and discarded it already, though. Actually, in general I'd like a way to plot more things on one graph. So maybe this is just a special case of that. Jean-Paul From samuele.pedroni at gmail.com Fri Jan 21 21:29:19 2011 From: samuele.pedroni at gmail.com (Samuele Pedroni) Date: Fri, 21 Jan 2011 21:29:19 +0100 Subject: [pypy-dev] Branches in the mercurial repository In-Reply-To: <4D394D85.2010106@openend.se> References: <201101201616.46698.jacob@openend.se> <4D394D85.2010106@openend.se> Message-ID: On Fri, Jan 21, 2011 at 10:10 AM, Bea During wrote: >> On Thu, Jan 20, 2011 at 10:33 AM, Samuele Pedroni >> > wrote: >> >> ? ?> Samuele ? ? ? ? ? ? ?ctypes-stable ? ? ? ? ? ?Fri Apr 11 >> ? ?15:44:57 2008 >> ? ?> Samuele ? ? ? ? ? ? ?run-django ? ? ? ? ? ? ? Sat Jul 12 >> ? ?09:46:07 2008 ... >> closed just means that they are no longer being used (and won't be shown >> in the default list of branches), I tihnk those are safe to close. >> > +1. > closed Samuele From tobami at googlemail.com Sun Jan 23 20:37:39 2011 From: tobami at googlemail.com (Miquel Torres) Date: Sun, 23 Jan 2011 20:37:39 +0100 Subject: [pypy-dev] New version of Codespeed (0.7) for speed.pypy.org In-Reply-To: <20110121183411.1851.1980573527.divmod.xquotient.20@localhost.localdomain> References: <4D38B310.304@gmail.com> <4D39CD04.3050104@gmail.com> <20110121183411.1851.1980573527.divmod.xquotient.20@localhost.localdomain> Message-ID: It doesn't sound like a bad idea. But how would you save the branch data? 2011/1/21 : > On 06:14 pm, anto.cuni at gmail.com wrote: >>On 21/01/11 08:49, Miquel Torres wrote: >>>@Anto >>> >>>Yes, branches are a pending item that has been requested a couple of >>>times now. >> >>yes, I think most of the requests has been by me :) >>>The current solution is actually not to abuse an environment like you >>>say, but to create a new project for a branch. That way it gets >>>cleanly separated, and in a way branches are like different projects. >>>But it is of course not optimal. Technically it is very easy to come >>>up with several solutions (add a branch dimension, for example), but >>>interface-wise it is not easy to find something that doesn't clutter >>>things. >> >>Uhm, I don't think that using a different project is a good idea. For >>branches, we are usually not much interested in the branch history, but >>in the >>comparison with trunk (ideally, with trunk at the point we created the >>branch, >>or at the point of the last merge from trunk). >> >>As for visualize changes, I think that we don't need anything fancy, >>for >>example it would be already immensely useful to have the possibility of >>displaying the Changes page in a way that it compares the performances >>of the >>branch against trunk. > > How about a "Branches" checkbox (per project? ?per executable? ?per > graph? ?One of those maybe). ?When it's checked, branch results within > the revision horizon (last 10, 50, 200, etc) get plotted on the same > graph as the trunk data is plotted. ?Each branch could be a different > color, perhaps (but would at least have hover info telling you what it > is). > > This implies adding a branch column to the results table (or is it the > revisions table?). > > Maybe that's just the obvious way to do it and everyone else already > thought of and discarded it already, though. > > Actually, in general I'd like a way to plot more things on one graph. > So maybe this is just a special case of that. > > Jean-Paul > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From exarkun at twistedmatrix.com Mon Jan 24 00:37:27 2011 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Sun, 23 Jan 2011 23:37:27 -0000 Subject: [pypy-dev] New version of Codespeed (0.7) for speed.pypy.org In-Reply-To: References: <4D38B310.304@gmail.com> <4D39CD04.3050104@gmail.com> <20110121183411.1851.1980573527.divmod.xquotient.20@localhost.localdomain> Message-ID: <20110123233727.1699.1400508058.divmod.xquotient.222@localhost.localdomain> On 07:37 pm, tobami at googlemail.com wrote: >It doesn't sound like a bad idea. But how would you save the branch >data? What do you need beyond an extra column in the revisions table? Everything else would be the same, except when you upload data, include the branch it is for, and when you query for data, limit yourself to trunk/default/whatever unless you know you want more. Jean-Paul > >2011/1/21 : >>On 06:14 pm, anto.cuni at gmail.com wrote: >>>On 21/01/11 08:49, Miquel Torres wrote: >>>>@Anto >>>> >>>>Yes, branches are a pending item that has been requested a couple of >>>>times now. >>> >>>yes, I think most of the requests has been by me :) >>>>The current solution is actually not to abuse an environment like >>>>you >>>>say, but to create a new project for a branch. That way it gets >>>>cleanly separated, and in a way branches are like different >>>>projects. >>>>But it is of course not optimal. Technically it is very easy to come >>>>up with several solutions (add a branch dimension, for example), but >>>>interface-wise it is not easy to find something that doesn't clutter >>>>things. >>> >>>Uhm, I don't think that using a different project is a good idea. For >>>branches, we are usually not much interested in the branch history, >>>but >>>in the >>>comparison with trunk (ideally, with trunk at the point we created >>>the >>>branch, >>>or at the point of the last merge from trunk). >>> >>>As for visualize changes, I think that we don't need anything fancy, >>>for >>>example it would be already immensely useful to have the possibility >>>of >>>displaying the Changes page in a way that it compares the >>>performances >>>of the >>>branch against trunk. >> >>How about a "Branches" checkbox (per project? ?per executable? ?per >>graph? ?One of those maybe). ?When it's checked, branch results within >>the revision horizon (last 10, 50, 200, etc) get plotted on the same >>graph as the trunk data is plotted. ?Each branch could be a different >>color, perhaps (but would at least have hover info telling you what it >>is). >> >>This implies adding a branch column to the results table (or is it the >>revisions table?). >> >>Maybe that's just the obvious way to do it and everyone else already >>thought of and discarded it already, though. >> >>Actually, in general I'd like a way to plot more things on one graph. >>So maybe this is just a special case of that. >> >>Jean-Paul >>_______________________________________________ >>pypy-dev at codespeak.net >>http://codespeak.net/mailman/listinfo/pypy-dev > From Arnd.Rechenburg at tomtom.com Mon Jan 24 09:38:52 2011 From: Arnd.Rechenburg at tomtom.com (Arnd Rechenburg) Date: Mon, 24 Jan 2011 09:38:52 +0100 Subject: [pypy-dev] PyObject_AsCharBuffer: compiler warning Message-ID: Hi, When I try to use the function PyObject_AsCharBuffer I get the following compiler warning: warning: passing argument 2 of 'PyObject_AsCharBuffer' from incompatible pointer type By using Python it works without problems. In Python: int PyObject_AsCharBuffer(PyObject *obj, const char **buffer, Py_ssize_t *buffer_len) Is there something different in pypy? Regards, Arnd -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110124/b40f17b4/attachment.htm From fijall at gmail.com Mon Jan 24 09:41:49 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 24 Jan 2011 10:41:49 +0200 Subject: [pypy-dev] PyObject_AsCharBuffer: compiler warning In-Reply-To: References: Message-ID: On Mon, Jan 24, 2011 at 10:38 AM, Arnd Rechenburg wrote: > Hi, > > > > When I try to use the function PyObject_AsCharBuffer I get the following > compiler warning: > > warning: passing argument 2 of 'PyObject_AsCharBuffer' from incompatible > pointer type > > > > By using Python it works without problems. > > In Python: > > int PyObject_AsCharBuffer(PyObject *obj, const char **buffer, Py_ssize_t > *buffer_len) > > > > Is there something different in pypy? > > We don't have const char**, we use char** instead. It's a small deficiency of how we do stuff now, probably fixable in the future, but don't worry too much, it works the same way. Cheers, fijal From fijall at gmail.com Mon Jan 24 20:01:25 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 24 Jan 2011 21:01:25 +0200 Subject: [pypy-dev] Fwd: [issue10994] implementation details in sys module In-Reply-To: <1295878409.3679.12.camel@localhost.localdomain> References: <1295877929.5.0.527131313453.issue10994@psf.upfronthosting.co.za> <1295878409.3679.12.camel@localhost.localdomain> Message-ID: I think this explanation what sys.getsizeof does is interesting. We might want to have it. ---------- Forwarded message ---------- From: Antoine Pitrou Date: Mon, Jan 24, 2011 at 4:13 PM Subject: [issue10994] implementation details in sys module To: fijall at gmail.com Antoine Pitrou added the comment: > I suppose wrt getsizeof it's more of "if you provide us with a > reasonable expectations, we can implement this" other than anything > else. The expectation is that it returns the memory footprint of the given object, and only it (not taking into account sharing, caching, dependencies or anything else). For example, an instance will not count its attribute __dict__. But a str object will count its object header plus the string payload, if the payload is private. Of course, you are free to tweak these semantics for the PyPy implementation. ---------- _______________________________________ Python tracker _______________________________________ From tvoglou at gmail.com Mon Jan 31 22:27:53 2011 From: tvoglou at gmail.com (Tasos Vogiatzoglou) Date: Mon, 31 Jan 2011 23:27:53 +0200 Subject: [pypy-dev] Issue with SSL module while translating with VS2010 and 64bit Message-ID: Hello All, I am trying to translate and compile pypy using Visual Studio 2010 to Win64a and I am getting the following issue with ssl. [translation:WARNING] The module '_ssl' is disabled [translation:WARNING] because importing pypy.module._ssl.interp_ssl raised CompilationError [translation:WARNING] After that the translation operation stops and the debugger kicks in. The strange part is that when I execute cl/link from a command prompt the file is compiled and linked without issues. I've compiled OpenSSL with 64bit support removing bufferoverflowu.lib which is not needed in vs2010. Regards Tasos From amauryfa at gmail.com Mon Jan 31 22:36:30 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Mon, 31 Jan 2011 22:36:30 +0100 Subject: [pypy-dev] Issue with SSL module while translating with VS2010 and 64bit In-Reply-To: References: Message-ID: Hi, 2011/1/31 Tasos Vogiatzoglou : > Hello All, > > I am trying to translate and compile pypy using Visual Studio 2010 to > Win64a Wow, you are the first one! But I don't think it will work: sadly in PyPy there is the implicit assumption that a pointer can be stored in a C "long", which is wrong on win64. > and I am getting the following issue with ssl. > [translation:WARNING] The module '_ssl' is disabled > [translation:WARNING] because importing pypy.module._ssl.interp_ssl > raised CompilationError > [translation:WARNING] > > After that the translation operation stops and the debugger kicks in. But these are *warnings*. Which error do you get exactly? do you have a (red) traceback? -- Amaury Forgeot d'Arc From tvoglou at gmail.com Mon Jan 31 22:39:25 2011 From: tvoglou at gmail.com (Tasos Vogiatzoglou) Date: Mon, 31 Jan 2011 23:39:25 +0200 Subject: [pypy-dev] Issue with SSL module while translating with VS2010 and 64bit In-Reply-To: References: Message-ID: Hello Amaury, Yes. It's the same compiler error : [platform:ERROR] platcheck_49.obj : error LNK2019: unresolved external symbol _SSLeay_version referenced in function _dump_section_SSLEAY_VERSION [platform:ERROR] c:\users\flatline\appdata\local\temp\usession-default-56\platcheck_49.exe : fatal error LNK1120: 1 unresolved externals [translation:ERROR] Error: [translation:ERROR] Traceback (most recent call last): [translation:ERROR] File "translate.py", line 268, in main [translation:ERROR] default_goal='compile') [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\driver.py", line 831, in from_targetspec [translation:ERROR] spec = target(driver, args) [translation:ERROR] File "targetpypystandalone.py", line 229, in target [translation:ERROR] return self.get_entry_point(config) [translation:ERROR] File "targetpypystandalone.py", line 240, in get_entry_point [translation:ERROR] space = make_objspace(config) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\tool\option.py", line 48, in make_objspace [translation:ERROR] space = Space(config) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\interpreter\baseobjspace.py", line 280, in __init__ [translation:ERROR] self.initialize() [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\objspace\std\objspace.py", line 78, in initialize [translation:ERROR] self.make_builtins() [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\interpreter\baseobjspace.py", line 490, in make_builtins [translation:ERROR] self.install_mixedmodule(mixedname, installed_builtin_modules) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\interpreter\baseobjspace.py", line 529, in install_mixedmodule [translation:ERROR] modname = self.setbuiltinmodule(mixedname) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\interpreter\baseobjspace.py", line 370, in setbuiltinmodule [translation:ERROR] None, None, ["Module"]).Module [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\module\_hashlib\__init__.py", line 2, in [translation:ERROR] from pypy.module._hashlib.interp_hashlib import algorithms [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\module\_hashlib\interp_hashlib.py", line 9, in [translation:ERROR] from pypy.rlib import ropenssl [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\rlib\ropenssl.py", line 76, in [translation:ERROR] for k, v in rffi_platform.configure(CConfig).items(): [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\rpython\tool\rffi_platform.py", line 201, in configure [translation:ERROR] infolist = list(run_example_code(writer.path, eci)) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\rpython\tool\rffi_platform.py", line 685, in run_example_code [translation:ERROR] output = build_executable_cache(files, eci) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\tool\gcc_cache.py", line 27, in build_executable_cache [translation:ERROR] result = platform.execute(platform.compile(c_files, eci) ) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\platform\__init__.py", line 52, in compile [translation:ERROR] return self._finish_linking(ofiles, eci, outputfilename, standalone) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\platform\__init__.py", line 194, in _finish_linking [translation:ERROR] return self._link(cc_link, ofiles, largs, standalone, exe_name) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\platform\windows.py", line 176, in _link [translation:ERROR] self._execute_c_compiler(self.link, args, exe_name) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\platform\__init__.py", line 118, in _execute_c_compiler [translation:ERROR] self._handle_error(returncode, stderr, stdout, outname) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\platform\windows.py", line 199, in _handle_error [translation:ERROR] raise CompilationError(stdout, stderr) [translation:ERROR] CompilationError: Regards Tasos Vogiatzoglou On Mon, Jan 31, 2011 at 11:36 PM, Amaury Forgeot d'Arc wrote: > Hi, > > 2011/1/31 Tasos Vogiatzoglou : >> Hello All, >> >> I am trying to translate and compile pypy using Visual Studio 2010 to >> Win64a > > Wow, you are the first one! But I don't think it will work: sadly in > PyPy there is the implicit assumption that a pointer can be stored in > a C "long", which is wrong on win64. > >> ?and I am getting the following issue with ssl. >> [translation:WARNING] The module '_ssl' is disabled >> [translation:WARNING] because importing pypy.module._ssl.interp_ssl >> raised CompilationError >> [translation:WARNING] >> >> After that the translation operation stops and the debugger kicks in. > > But these are *warnings*. Which error do you get exactly? do you have > a (red) traceback? > > -- > Amaury Forgeot d'Arc > From amauryfa at gmail.com Mon Jan 31 22:44:31 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Mon, 31 Jan 2011 22:44:31 +0100 Subject: [pypy-dev] Issue with SSL module while translating with VS2010 and 64bit In-Reply-To: References: Message-ID: 2011/1/31 Tasos Vogiatzoglou : > Hello Amaury, > > Yes. It's the same compiler error : > [translation:ERROR] ? ?File "c:\Users\flatline\development\pypy\src\pypy\module\_hashlib\interp_hashlib.py", Ah, indeed the _hashlib module should be disabled when openssl cannot be processed. I'll try to fix it, in the meantime I suggest you to skip this module: python translate.py targetpypystandalone --withoutmod-_hashlib -- Amaury Forgeot d'Arc From tvoglou at gmail.com Mon Jan 31 22:48:57 2011 From: tvoglou at gmail.com (Tasos Vogiatzoglou) Date: Mon, 31 Jan 2011 23:48:57 +0200 Subject: [pypy-dev] Issue with SSL module while translating with VS2010 and 64bit In-Reply-To: References: Message-ID: Amaury, Isn't there a way to fix the SSL issue? I was looking checking for name decoration issues or call-conv issues but there seems none. Is there any guess why it fails from the translation procedure but works when I execute the commands in cmd prompt ? (I also tried disabling the env processing that is happening, leaving POpen to get the default one but nothing :( ) Thanks, Tasos On Mon, Jan 31, 2011 at 11:44 PM, Amaury Forgeot d'Arc wrote: > 2011/1/31 Tasos Vogiatzoglou : >> Hello Amaury, >> >> Yes. It's the same compiler error : > >> [translation:ERROR] ? ?File "c:\Users\flatline\development\pypy\src\pypy\module\_hashlib\interp_hashlib.py", > > Ah, indeed the _hashlib module should be disabled when openssl cannot > be processed. > I'll try to fix it, in the meantime I suggest you to skip this module: > > python translate.py targetpypystandalone --withoutmod-_hashlib > > > -- > Amaury Forgeot d'Arc > From tvoglou at gmail.com Tue Feb 1 09:01:26 2011 From: tvoglou at gmail.com (Tasos Vogiatzoglou) Date: Tue, 1 Feb 2011 10:01:26 +0200 Subject: [pypy-dev] Link errors while translating with VS2010 and 64bit Message-ID: Amaury, It seems that there is a general issue with the compiler/link . I did the translations without the _hashlib and ssl and after a while I got the following errors. [platform:ERROR] Creating library c:\users\flatline\appdata\local\temp\usessi on-default-57\testing_1\libpypy-c.lib and object c:\users\flatline\appdata\local\temp\usession-default-57\testing_1\libpypy-c.exp [platform:ERROR] implement_52.obj : error LNK2019: unresolved external symbol _inflateEnd referenced in function _pypy_g_ccall_inflateEnd__z_streamPtr [platform:ERROR] implement_64.obj : error LNK2019: unresolved external symbol _inflateInit2_ referenced in function _pypy_g_ccall_inflateInit2___z_streamPtr_Sig ned_arrayPtr [platform:ERROR] implement_64.obj : error LNK2019: unresolved external symbol _inflate referenced in function _pypy_g_ccall_inflate__z_streamPtr_Signed [platform:ERROR] implement_67.obj : error LNK2019: unresolved external symbol _deflateEnd referenced in function _pypy_g_ccall_deflateEnd__z_streamPtr [platform:ERROR] implement_69.obj : error LNK2019: unresolved external symbol _deflate referenced in function _pypy_g_ccall_deflate__z_streamPtr_Signed [platform:ERROR] implement_70.obj : error LNK2019: unresolved external symbol _adler32 referenced in function _pypy_g_ccall_adler32__Unsigned_arrayPtr_Unsigned [platform:ERROR] implement_71.obj : error LNK2019: unresolved external symbol _deflateInit2_ referenced in function _pypy_g_ccall_deflateInit2___z_streamPtr_Sig ned_Signed_S [platform:ERROR] implement_71.obj : error LNK2019: unresolved external symbol _crc32 referenced in function _pypy_g_ccall_crc32__Unsigned_arrayPtr_Unsigned [platform:ERROR] c:\users\flatline\appdata\local\temp\usession-default-57\testing_1\libpypy-c.dll : fatal error LNK1120: 8 unresolved externals [Timer] Timings: [Timer] annotate --- 1447.8 s [Timer] rtype_lltype --- 768.2 s [Timer] backendopt_lltype --- 415.9 s [Timer] stackcheckinsertion_lltype --- 60.1 s [Timer] database_c --- 498.3 s [Timer] source_c --- 1190.5 s [Timer] compile_c --- 216.4 s [Timer] =========================================== [Timer] Total: --- 4597.3 s [translation:ERROR] Error: [translation:ERROR] Traceback (most recent call last): [translation:ERROR] File "translate.py", line 290, in main [translation:ERROR] drv.proceed(goals) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\driver.py", line 812, in proceed [translation:ERROR] return self._execute(goals, task_skip = self._maybe_skip()) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\tool\taskengine.py", line 116, in _execute [translation:ERROR] res = self._do(goal, taskcallable, *args, **kwds) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\driver.py", line 288, in _do [translation:ERROR] res = func() [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\driver.py", line 575, in task_compile_c [translation:ERROR] cbuilder.compile(**kwds) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\c\genc.py", line 527, in compile [translation:ERROR] self.executable_name = compiler.build(shared=shared) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\c\genc.py", line 95, in build [translation:ERROR] return self._build(shared=shared) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\c\genc.py", line 90, in _build [translation:ERROR] standalone=not shared) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\platform\__init__.py", line 52, in compile [translation:ERROR] return self._finish_linking(ofiles, eci, outputfilename, standalone) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\platform\__init__.py", line 194, in _finish_linking [translation:ERROR] return self._link(cc_link, ofiles, largs, standalone, exe_name) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\platform\windows.py", line 176, in _link [translation:ERROR] self._execute_c_compiler(self.link, args, exe_name) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\platform\__init__.py", line 118, in _execute_c_compiler [translation:ERROR] self._handle_error(returncode, stderr, stdout, outname) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\platform\windows.py", line 199, in _handle_error [translation:ERROR] raise CompilationError(stdout, stderr) [translation:ERROR] CompilationError: Thanks, Tasos On Mon, Jan 31, 2011 at 11:48 PM, Tasos Vogiatzoglou wrote: > Amaury, > > Isn't there a way to fix the SSL issue? I was looking checking for > name decoration issues or call-conv issues but there seems none. > > Is there any guess why it fails from the translation procedure but > works when I execute the commands in cmd prompt ? > > (I also tried disabling the env processing that is happening, leaving > POpen to get the default one but nothing :( ) > > Thanks, > Tasos > > > > On Mon, Jan 31, 2011 at 11:44 PM, Amaury Forgeot d'Arc > wrote: >> 2011/1/31 Tasos Vogiatzoglou : >>> Hello Amaury, >>> >>> Yes. It's the same compiler error : >> >>> [translation:ERROR] ? ?File "c:\Users\flatline\development\pypy\src\pypy\module\_hashlib\interp_hashlib.py", >> >> Ah, indeed the _hashlib module should be disabled when openssl cannot >> be processed. >> I'll try to fix it, in the meantime I suggest you to skip this module: >> >> python translate.py targetpypystandalone --withoutmod-_hashlib >> >> >> -- >> Amaury Forgeot d'Arc >> > From amauryfa at gmail.com Tue Feb 1 09:11:04 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 1 Feb 2011 09:11:04 +0100 Subject: [pypy-dev] Link errors while translating with VS2010 and 64bit In-Reply-To: References: Message-ID: Hi, 2011/2/1 Tasos Vogiatzoglou : > Amaury, > > It seems that there is a general issue with the compiler/link . > > I did the translations without the _hashlib and ssl and after a while > I got the following errors. [...] > [platform:ERROR] implement_52.obj : error LNK2019: unresolved external > symbol _inflateEnd referenced in function _pypy_g_ccall_inflateEnd__z_streamPtr We have never tried the win64 platform, and I don't have access to a Windows 64bit machine. But I suspect that even if you removed all external dependencies, the result would not work; pypy's compilation tools implicitly assume that sizeof(long)==sizeof(void*) Before running a translation, could you run the tests in pypy/translator/c ? in the pypy directory, run: python test_all.py translator/c I'd like at least the files "test_genc" and "test_newgc" to pass without errors. -- Amaury Forgeot d'Arc From fijall at gmail.com Tue Feb 1 09:14:19 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 1 Feb 2011 10:14:19 +0200 Subject: [pypy-dev] Link errors while translating with VS2010 and 64bit In-Reply-To: References: Message-ID: On Tue, Feb 1, 2011 at 10:11 AM, Amaury Forgeot d'Arc wrote: > Hi, > > 2011/2/1 Tasos Vogiatzoglou : >> Amaury, >> >> It seems that there is a general issue with the compiler/link . >> >> I did the translations without the _hashlib and ssl and after a while >> I got the following errors. > [...] >> [platform:ERROR] implement_52.obj : error LNK2019: unresolved external >> symbol _inflateEnd referenced in function _pypy_g_ccall_inflateEnd__z_streamPtr > > We have never tried the win64 platform, and I don't have access to a > Windows 64bit machine. > But I suspect that even if you removed all external dependencies, the > result would not work; > pypy's compilation tools implicitly assume that sizeof(long)==sizeof(void*) > > Before running a translation, could you run the tests in pypy/translator/c ? > in the pypy directory, run: > ? ?python test_all.py translator/c > I'd like at least the files "test_genc" and "test_newgc" to pass without errors. Surprisingly enough, I have > > -- > Amaury Forgeot d'Arc > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From tvoglou at gmail.com Tue Feb 1 09:20:55 2011 From: tvoglou at gmail.com (Tasos Vogiatzoglou) Date: Tue, 1 Feb 2011 10:20:55 +0200 Subject: [pypy-dev] Link errors while translating with VS2010 and 64bit In-Reply-To: References: Message-ID: <-3229585076398878208@unknownmsgid> Amaury, I'll try and I'll let you know. Thanks, Tasos On 1 ??? 2011, at 10:11, Amaury Forgeot d'Arc wrote: > Hi, > > 2011/2/1 Tasos Vogiatzoglou : >> Amaury, >> >> It seems that there is a general issue with the compiler/link . >> >> I did the translations without the _hashlib and ssl and after a while >> I got the following errors. > [...] >> [platform:ERROR] implement_52.obj : error LNK2019: unresolved external >> symbol _inflateEnd referenced in function _pypy_g_ccall_inflateEnd__z_streamPtr > > We have never tried the win64 platform, and I don't have access to a > Windows 64bit machine. > But I suspect that even if you removed all external dependencies, the > result would not work; > pypy's compilation tools implicitly assume that sizeof(long)==sizeof(void*) > > Before running a translation, could you run the tests in pypy/translator/c ? > in the pypy directory, run: > python test_all.py translator/c > I'd like at least the files "test_genc" and "test_newgc" to pass without errors. > > -- > Amaury Forgeot d'Arc From fijall at gmail.com Tue Feb 1 10:47:24 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 1 Feb 2011 11:47:24 +0200 Subject: [pypy-dev] Worrying set of checkins Message-ID: Anyone feeling like looking into this: http://speed.pypy.org/changes/?tre=10&rev=41510:77f94f44989a&exe=3&env=tannit ? From amauryfa at gmail.com Tue Feb 1 11:17:14 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 1 Feb 2011 11:17:14 +0100 Subject: [pypy-dev] Worrying set of checkins In-Reply-To: References: Message-ID: 2011/2/1 Maciej Fijalkowski : > Anyone feeling like looking into this: > > http://speed.pypy.org/changes/?tre=10&rev=41510:77f94f44989a&exe=3&env=tannit This show benchmarks for pypy *without* JIT. the pypy-c-jit figures are more positive. -- Amaury Forgeot d'Arc From fijall at gmail.com Tue Feb 1 11:19:57 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 1 Feb 2011 12:19:57 +0200 Subject: [pypy-dev] Worrying set of checkins In-Reply-To: References: Message-ID: On Tue, Feb 1, 2011 at 12:17 PM, Amaury Forgeot d'Arc wrote: > 2011/2/1 Maciej Fijalkowski : >> Anyone feeling like looking into this: >> >> http://speed.pypy.org/changes/?tre=10&rev=41510:77f94f44989a&exe=3&env=tannit > > This show benchmarks for pypy *without* JIT. > the pypy-c-jit figures are more positive. > > -- > Amaury Forgeot d'Arc > Yes. Still, worth looking into. From fijall at gmail.com Tue Feb 1 11:24:15 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 1 Feb 2011 12:24:15 +0200 Subject: [pypy-dev] Worrying set of checkins In-Reply-To: References: Message-ID: On Tue, Feb 1, 2011 at 12:19 PM, Maciej Fijalkowski wrote: > On Tue, Feb 1, 2011 at 12:17 PM, Amaury Forgeot d'Arc > wrote: >> 2011/2/1 Maciej Fijalkowski : >>> Anyone feeling like looking into this: >>> >>> http://speed.pypy.org/changes/?tre=10&rev=41510:77f94f44989a&exe=3&env=tannit >> >> This show benchmarks for pypy *without* JIT. >> the pypy-c-jit figures are more positive. >> >> -- >> Amaury Forgeot d'Arc >> > > Yes. Still, worth looking into. > Well, false alarms. Next run nulled out those changes. From tvoglou at gmail.com Tue Feb 1 18:10:42 2011 From: tvoglou at gmail.com (Tasos Vogiatzoglou) Date: Tue, 1 Feb 2011 19:10:42 +0200 Subject: [pypy-dev] Link errors while translating with VS2010 and 64bit In-Reply-To: <-3229585076398878208@unknownmsgid> References: <-3229585076398878208@unknownmsgid> Message-ID: Amaury, You were right, there are some issues with type sizes. test_genc and test_newgc are failing. I'll try to see what I can do to fix them. Are there any directions I should move towards ? Thanks, Tasos On Tue, Feb 1, 2011 at 10:20 AM, Tasos Vogiatzoglou wrote: > Amaury, > > I'll try and I'll let you know. > > Thanks, > Tasos > > On 1 ??? 2011, at 10:11, Amaury Forgeot d'Arc wrote: > >> Hi, >> >> 2011/2/1 Tasos Vogiatzoglou : >>> Amaury, >>> >>> It seems that there is a general issue with the compiler/link . >>> >>> I did the translations without the _hashlib and ssl and after a while >>> I got the following errors. >> [...] >>> [platform:ERROR] implement_52.obj : error LNK2019: unresolved external >>> symbol _inflateEnd referenced in function _pypy_g_ccall_inflateEnd__z_streamPtr >> >> We have never tried the win64 platform, and I don't have access to a >> Windows 64bit machine. >> But I suspect that even if you removed all external dependencies, the >> result would not work; >> pypy's compilation tools implicitly assume that sizeof(long)==sizeof(void*) >> >> Before running a translation, could you run the tests in pypy/translator/c ? >> in the pypy directory, run: >> ? ?python test_all.py translator/c >> I'd like at least the files "test_genc" and "test_newgc" to pass without errors. >> >> -- >> Amaury Forgeot d'Arc > From renesd at gmail.com Tue Feb 1 18:32:36 2011 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Tue, 1 Feb 2011 17:32:36 +0000 Subject: [pypy-dev] Link errors while translating with VS2010 and 64bit In-Reply-To: References: <-3229585076398878208@unknownmsgid> Message-ID: Hi, A naive guess would be the different sizes between 64bit windows, and 64 bit linux? (long int is 32bit on windows and 64bit on linux) >From http://en.wikipedia.org/wiki/64-bit 64-bit data models Data model short (integer) int long (integer) long long pointers/size_t Sample operating systems LLP64/ IL32P64 16 32 32 64 64 Microsoft Windows (X64/IA-64) LP64/ I32LP64 16 32 64 64 64 Most Unix and Unix-like systems, e.g. Solaris, Linux, and Mac OS X ILP64 16 64 64 64 64 HAL Computer Systems port of Solaris to SPARC64 SILP64 64 64 64 64 64 Unicos On Tue, Feb 1, 2011 at 5:10 PM, Tasos Vogiatzoglou wrote: > Amaury, > > You were right, there are some issues with type sizes. test_genc and > test_newgc are failing. > > I'll try to see what I can do to fix them. Are there any directions I > should move towards ? > > Thanks, > Tasos > > > > On Tue, Feb 1, 2011 at 10:20 AM, Tasos Vogiatzoglou wrote: >> Amaury, >> >> I'll try and I'll let you know. >> >> Thanks, >> Tasos >> >> On 1 ??? 2011, at 10:11, Amaury Forgeot d'Arc wrote: >> >>> Hi, >>> >>> 2011/2/1 Tasos Vogiatzoglou : >>>> Amaury, >>>> >>>> It seems that there is a general issue with the compiler/link . >>>> >>>> I did the translations without the _hashlib and ssl and after a while >>>> I got the following errors. >>> [...] >>>> [platform:ERROR] implement_52.obj : error LNK2019: unresolved external >>>> symbol _inflateEnd referenced in function _pypy_g_ccall_inflateEnd__z_streamPtr >>> >>> We have never tried the win64 platform, and I don't have access to a >>> Windows 64bit machine. >>> But I suspect that even if you removed all external dependencies, the >>> result would not work; >>> pypy's compilation tools implicitly assume that sizeof(long)==sizeof(void*) >>> >>> Before running a translation, could you run the tests in pypy/translator/c ? >>> in the pypy directory, run: >>> ? ?python test_all.py translator/c >>> I'd like at least the files "test_genc" and "test_newgc" to pass without errors. >>> >>> -- >>> Amaury Forgeot d'Arc >> > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev From pgiarrusso at mathematik.uni-marburg.de Wed Feb 2 00:35:54 2011 From: pgiarrusso at mathematik.uni-marburg.de (Paolo Giarrusso) Date: Wed, 2 Feb 2011 00:35:54 +0100 Subject: [pypy-dev] Link errors while translating with VS2010 and 64bit In-Reply-To: References: <-3229585076398878208@unknownmsgid> Message-ID: 2011/2/1 Ren? Dudfield : > Hi, > > A naive guess would be the different sizes between 64bit windows, and > 64 bit linux? ?(long int is 32bit on windows and 64bit on linux) > > From http://en.wikipedia.org/wiki/64-bit > > 64-bit data models > Data model ? ? ?short (integer) int ? ? long (integer) ?long > long ? ?pointers/size_t Sample operating systems > LLP64/ > IL32P64 16 ? ? ?32 ? ? ?32 ? ? ?64 ? ? ?64 ? ? ?Microsoft Windows (X64/IA-64) > LP64/ > I32LP64 16 ? ? ?32 ? ? ?64 ? ? ?64 ? ? ?64 ? ? ?Most Unix and Unix-like systems, e.g. Solaris, > Linux, and Mac OS X > ILP64 ? 16 ? ? ?64 ? ? ?64 ? ? ?64 ? ? ?64 ? ? ?HAL Computer Systems port of Solaris to SPARC64 > SILP64 ?64 ? ? ?64 ? ? ?64 ? ? ?64 ? ? ?64 ? ? ?Unicos >From what Amaury said, you need to identify places where long is used and it is assumed that it has 64bits. A first guess, in general, is that _every_ use of long assumes it to be 64bit on 64bits systems. What about using intptr_t (or uintptr_t) from stdint.h? IOW, you'd simply replace long by this type everywhere, including in functions which generate C code through some library. I don't know the details, could somebody help here? IIRC this header was introduced in C99, and for sure this type is optional - i.e., it must be declared if it is available at all. Of course, there might be systems where this type is not available, so you need to provide a fallback (in a configure-like fashion). If this header is available on (modern) Windows environments (the only system where long is 32 bits), using "long" as fallback is safe. Cheers, -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From lac at openend.se Wed Feb 2 00:51:36 2011 From: lac at openend.se (Laura Creighton) Date: Wed, 02 Feb 2011 00:51:36 +0100 Subject: [pypy-dev] testing floating point Message-ID: <201102012351.p11NpaJk011803@theraft.openend.se> running the test suite of this package might be useful. Laura ------- Forwarded Message Return-Path: python-announce-list-bounces+lac=openend.se at python.org Delivery-Date: Wed Feb 2 00:29:19 2011 Return-Path: Received: from mail.python.org (mail.python.org [82.94.164.166]) by theraft.openend.se (8.14.3/8.14.3/Debian-5+lenny1) with ESMTP id p11NTCgZ009584 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK) for ; Wed, 2 Feb 2011 00:29:14 +0100 Received: from albatross.python.org (localhost [127.0.0.1]) by mail.python.org (Postfix) with ESMTP id 1089FEE9C0 for ; Wed, 2 Feb 2011 00:29:12 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=python.org; s=200901; t=1296602952; bh=SjeURYFJ0JANVg28+rd+7SNHYu2uW4+khr521TTR24c=; h=MIME-Version:Date:Message-ID:Subject:From:To:Reply-To:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: Content-Type:Content-Transfer-Encoding:Sender; b=u3Vd1rser4Yn6sV8mw8GsXc9yRHbfI8ct22qpxaml8r0lxcbKfbHnjjLVdST1kZY3 0MKS5mryz3ElA42r6P3S60hM+tWW0uz4xFk2AVUCrIeWt3+tpHWgYx3oXKHdTtdKRN ZgYWWK3rX3RoCiBOhAKDddtq25GWXs4jg2p75OVc= X-Original-To: python-announce-list at python.org Delivered-To: python-announce-list at mail.python.org Received: from albatross.python.org (localhost [127.0.0.1]) by mail.python.org (Postfix) with ESMTP id 51C0CEE9B8 for ; Tue, 1 Feb 2011 23:11:46 +0100 (CET) X-Spam-Status: OK 0.000 X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'bug': 0.02; 'subject:ANN': 0.02; 'subject:Python': 0.05; '2.x': 0.05; 'implements': 0.05; 'library': 0.06; '3.x': 0.07; 'derivatives': 0.07; 'tracker': 0.07; 'url:googlecode': 0.07; 'python': 0.08; 'url:pypi': 0.08; 'arithmetic': 0.09; 'url:code': 0.15; '0.17': 0.16; 'codebase,': 0.16; 'subject:0.17': 0.16; 'url:changes': 0.16; 'url:issues': 0.16; 'url:svn': 0.16; 'url:trunk': 0.16; 'downloaded': 0.17; 'url:doc': 0.19; '2.4': 0.23; 'version': 0.24; 'url:group': 0.25; 'function': 0.27; 'functions.': 0.28; 'project': 0.29; 'message- id:@mail.gmail.com': 0.29; 'url:)': 0.29; 'thanks': 0.30; 'dropped': 0.30; 'standalone': 0.30; 'all,': 0.30; 'contributed': 0.31; 'requires': 0.33; 'version.': 0.34; 'url:google': 0.34; '2.5': 0.36; 'mathematical': 0.36; 'to:addr:python-announce-list': 0.36; 'used': 0.36; 'set': 0.37; 'list:': 0.37; 'works': 0.38; 'received:209.85': 0.38; 'received:google.com': 0.38; 'zero': 0.38; 'url:org': 0.38; 'subject:: ': 0.39; 'url:python': 0.39; 'to:addr:python.org': 0.40; 'comments': 0.40; 'url:p': 0.60; 'to:2**2': 0.64; 'url:0': 0.64; 'details,': 0.65; 'website:': 0.71; 'to:no real name:2**2': 0.75; 'url:17': 0.76; 'news': 0.78; 'fredrik': 0.91 Received: from localhost (HELO mail.python.org) (127.0.0.1) by albatross.python.org with SMTP; 01 Feb 2011 23:11:46 +0100 Received: from mail-qw0-f46.google.com (mail-qw0-f46.google.com [209.85.216.46]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) by mail.python.org (Postfix) with ESMTPS for ; Tue, 1 Feb 2011 23:11:45 +0100 (CET) Received: by qwa26 with SMTP id 26so7212791qwa.19 for ; Tue, 01 Feb 2011 14:11:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:date:message-id:subject:from:to :content-type; bh=6yZ49a5xiXwaVsySNvnihUrvwjNRAycqDeQTj98mFN4=; b=TFkhQdC5AeC7EecRVy8bO+ZVjyDOniTJB9u5S4BVnuXSAIAa8U58xnIPqCNAEojT5M 2bA2NizGpC/zHNyQLa/yu92nhY85wb2Wv5FW72QkL9kklUpzkYFZp4/vi4nidCwFoA2R 7qa0fa3fksMnYLpIyfOE5Vv0NeyWrZKmxt/Jc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=qdEow0h50/OGUU89qp7muEE5FACELAhcuIpeNUeeV8srLOc4roz/kZHOS7NooIktaG 8lMo4dIEgm6aPAqeAxssYBAiQ5YDk/k0YxSsZovPJFV1incJEWpc9HIuuS9922zQUHeW J9J1GUwfUr/2QjPYed8ZCEQJkzSyAaSvGNFpI= MIME-Version: 1.0 Received: by 10.229.236.134 with SMTP id kk6mr7177714qcb.93.1296598304793; Tue, 01 Feb 2011 14:11:44 -0800 (PST) Received: by 10.229.245.209 with HTTP; Tue, 1 Feb 2011 14:11:44 -0800 (PST) Date: Tue, 1 Feb 2011 23:11:44 +0100 Message-ID: Subject: ANN: mpmath 0.17 (Python 3 support and more) From: Fredrik Johansson To: mpmath at googlegroups.com, sympy at googlegroups.com, sage-devel at googlegroups.com, python-announce-list at python.org X-Mailman-Approved-At: Wed, 02 Feb 2011 00:22:14 +0100 X-BeenThere: python-announce-list at python.org X-Mailman-Version: 2.1.12 Precedence: list Reply-To: python-list at python.org List-Id: Announcement-only list for the Python programming language List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: python-announce-list-bounces+lac=openend.se at python.org Errors-To: python-announce-list-bounces+lac=openend.se at python.org X-Greylist: Sender succeeded STARTTLS authentication, not delayed by milter-greylist-3.0 (theraft.openend.se [83.140.78.130]); Wed, 02 Feb 2011 00:29:14 +0100 (CET) X-DSPAM-Result: Innocent X-DSPAM-Processed: Wed Feb 2 00:29:19 2011 X-DSPAM-Confidence: 0.9994 X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 1004,4d48974f95865701517258 Hi all, Version 0.17 of mpmath is now available on the project website: http://code.google.com/p/mpmath/ It can also be downloaded from the Python Package Index: http://pypi.python.org/pypi/mpmath/0.17 Mpmath is a pure-Python library for arbitrary-precision floating-point arithmetic that implements an extensive set of mathematical functions. It can be used as a standalone library or via SymPy (http://code.google.com/p/sympy/), and is also available as a standard component of Sage (http://sagemath.org/). The major news in 0.17 is that mpmath now works with Python 3. To support both Python 2.x and 3.x with the same codebase, compatibility with Python 2.4 has been dropped (mpmath now requires 2.5 or higher). New functionality in mpmath 0.17 includes an implementation of the Lerch transcendent, Riemann zeta zero counting, and improved support for evaluating derivatives of the Hurwitz zeta function and related functions. Many thanks to Juan Arias de Reyna and Case Vanhorsen who contributed to this version. For more details, see the changelog: http://mpmath.googlecode.com/svn/tags/0.17/CHANGES Extensive documentation is available at: http://mpmath.googlecode.com/svn/trunk/doc/build/index.html or http://mpmath.googlecode.com/svn/tags/0.17/doc/build/index.html Bug reports and other comments are welcome on the issue tracker at http://code.google.com/p/mpmath/issues/list or the mpmath mailing list: http://groups.google.com/group/mpmath Enjoy, Fredrik Johansson - -- http://mail.python.org/mailman/listinfo/python-announce-list Support the Python Software Foundation: http://www.python.org/psf/donations/ ------- End of Forwarded Message From tvoglou at gmail.com Wed Feb 2 00:56:26 2011 From: tvoglou at gmail.com (Tasos Vogiatzoglou) Date: Wed, 2 Feb 2011 01:56:26 +0200 Subject: [pypy-dev] Link errors while translating with VS2010 and 64bit In-Reply-To: References: <-3229585076398878208@unknownmsgid> Message-ID: Hello Paolo, Dunno if I am in the correct direction but till now I found two things 1) test_genc was throwing an error on fixup_c_type (complaining about conflicting field type between HLOCAL and pointer to char array) so I changed it to an opaque pointer (and the HANDLE type) 2) test_genc passed the tests, so the next step was to change the env to use the 64bit libs changing from vsvars32.bat to vcvarsall.bat with amd64 param in translator/platform/windows.py # _get_msvc_env(vsver): After that change I had lot's of linking issues like [platform:ERROR] testing_25.obj : error LNK2019: unresolved external symbol __imp_Py_BuildValue referenced in function malloc_counters [platform:ERROR] testing_25.obj : error LNK2019: unresolved external symbol __imp_PyMethod_New referenced in function gencfunc_descr_get [platform:ERROR] testing_25.obj : error LNK2019: unresolved external symbol __imp__Py_NoneStruct referenced in function gencfunc_descr_get Any idea ? Thanks, Tasos On Wed, Feb 2, 2011 at 1:35 AM, Paolo Giarrusso wrote: > 2011/2/1 Ren? Dudfield : >> Hi, >> >> A naive guess would be the different sizes between 64bit windows, and >> 64 bit linux? ?(long int is 32bit on windows and 64bit on linux) >> >> From http://en.wikipedia.org/wiki/64-bit >> >> 64-bit data models >> Data model ? ? ?short (integer) int ? ? long (integer) ?long >> long ? ?pointers/size_t Sample operating systems >> LLP64/ >> IL32P64 16 ? ? ?32 ? ? ?32 ? ? ?64 ? ? ?64 ? ? ?Microsoft Windows (X64/IA-64) >> LP64/ >> I32LP64 16 ? ? ?32 ? ? ?64 ? ? ?64 ? ? ?64 ? ? ?Most Unix and Unix-like systems, e.g. Solaris, >> Linux, and Mac OS X >> ILP64 ? 16 ? ? ?64 ? ? ?64 ? ? ?64 ? ? ?64 ? ? ?HAL Computer Systems port of Solaris to SPARC64 >> SILP64 ?64 ? ? ?64 ? ? ?64 ? ? ?64 ? ? ?64 ? ? ?Unicos > > From what Amaury said, you need to identify places where long is used > and it is assumed that it has 64bits. A first guess, in general, is > that _every_ use of long assumes it to be 64bit on 64bits systems. > > What about using intptr_t (or uintptr_t) from stdint.h? IOW, you'd > simply replace long by this type everywhere, including in functions > which generate C code through some library. I don't know the details, > could somebody help here? > IIRC this header was introduced in C99, and for sure this type is > optional - i.e., it must be declared if it is available at all. > > Of course, there might be systems where this type is not available, so > you need to provide a fallback (in a configure-like fashion). If this > header is available on (modern) Windows environments (the only system > where long is 32 bits), using "long" as fallback is safe. > > Cheers, > -- > Paolo Giarrusso - Ph.D. Student > http://www.informatik.uni-marburg.de/~pgiarrusso/ > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev From amauryfa at gmail.com Wed Feb 2 01:06:56 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 2 Feb 2011 01:06:56 +0100 Subject: [pypy-dev] Link errors while translating with VS2010 and 64bit In-Reply-To: References: <-3229585076398878208@unknownmsgid> Message-ID: 2011/2/2 Tasos Vogiatzoglou : > After that change I had lot's of linking issues like > > [platform:ERROR] testing_25.obj : error LNK2019: unresolved external > symbol __imp_Py_BuildValue referenced in function malloc_counters > [platform:ERROR] testing_25.obj : error LNK2019: unresolved external > symbol __imp_PyMethod_New referenced in function gencfunc_descr_get > [platform:ERROR] testing_25.obj : error LNK2019: unresolved external > symbol __imp__Py_NoneStruct referenced in function gencfunc_descr_get > > Any idea ? You are certainly building an extension module for the host Python (the one which runs the tests). You should check that the link command finds the correct Python27.lib, corresponding to your installation of Python. btw, are you really running a 64bit Python? -- Amaury Forgeot d'Arc From amauryfa at gmail.com Wed Feb 2 01:02:22 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 2 Feb 2011 01:02:22 +0100 Subject: [pypy-dev] testing floating point In-Reply-To: <201102012351.p11NpaJk011803@theraft.openend.se> References: <201102012351.p11NpaJk011803@theraft.openend.se> Message-ID: Hi, 2011/2/2 Laura Creighton : > running the test suite of this package might be useful. Some user already tried it one year ago, this even triggered two bugs in the early version of the JIT: http://codespeak.net/pipermail/pypy-dev/2010q1/005722.html I don't remember how fast is was, though. -- Amaury Forgeot d'Arc From tvoglou at gmail.com Wed Feb 2 01:15:10 2011 From: tvoglou at gmail.com (Tasos Vogiatzoglou) Date: Wed, 2 Feb 2011 02:15:10 +0200 Subject: [pypy-dev] Link errors while translating with VS2010 and 64bit In-Reply-To: References: <-3229585076398878208@unknownmsgid> Message-ID: Amaury, You are right. My mistake. It was picking the 2.6 32bit python. Now is working and tests are passing Thanks, Tasos On Wed, Feb 2, 2011 at 2:06 AM, Amaury Forgeot d'Arc wrote: > 2011/2/2 Tasos Vogiatzoglou : >> After that change I had lot's of linking issues like >> >> [platform:ERROR] testing_25.obj : error LNK2019: unresolved external >> symbol __imp_Py_BuildValue referenced in function malloc_counters >> [platform:ERROR] testing_25.obj : error LNK2019: unresolved external >> symbol __imp_PyMethod_New referenced in function gencfunc_descr_get >> [platform:ERROR] testing_25.obj : error LNK2019: unresolved external >> symbol __imp__Py_NoneStruct referenced in function gencfunc_descr_get >> >> Any idea ? > > You are certainly building an extension module for the host Python > (the one which runs the tests). > You should check that the link command finds the correct Python27.lib, > corresponding to your > installation of Python. > btw, are you really running a 64bit Python? > > -- > Amaury Forgeot d'Arc > From orangewarrior at gmail.com Wed Feb 2 03:09:09 2011 From: orangewarrior at gmail.com (=?iso-8859-2?q?=A3ukasz_Ligowski?=) Date: Wed, 2 Feb 2011 03:09:09 +0100 Subject: [pypy-dev] pypy jit Message-ID: <201102020309.09640.orangewarrior@gmail.com> Hi, First of all I'd like to thank all developers for good work on PyPy ;) I'd like to ask are there any rules of thumb to write code that PyPy JIT can easily optimize? Or maybe which constructs are hardly optimizable by JIT so it's better to avoid them? Best regards, Lukasz Ligowski From benjamin at python.org Wed Feb 2 03:18:31 2011 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 1 Feb 2011 20:18:31 -0600 Subject: [pypy-dev] pypy jit In-Reply-To: <201102020309.09640.orangewarrior@gmail.com> References: <201102020309.09640.orangewarrior@gmail.com> Message-ID: 2011/2/1 ?ukasz Ligowski : > Hi, > > First of all I'd like to thank all developers for good work on PyPy ;) > > I'd like to ask are there any rules of thumb to write code that PyPy JIT can > easily optimize? Or maybe which constructs are hardly optimizable by JIT so > it's better to avoid them? https://codespeak.net/viewvc/pypy/extradoc/talk/pycon2010/crossinterp/talk.pdf?view=log Mostly things you'd normally avoid anyway like sys._getframe(). -- Regards, Benjamin From jacob at openend.se Wed Feb 2 15:07:54 2011 From: jacob at openend.se (Jacob =?utf-8?q?Hall=C3=A9n?=) Date: Wed, 2 Feb 2011 15:07:54 +0100 Subject: [pypy-dev] pypy jit In-Reply-To: <201102020309.09640.orangewarrior@gmail.com> References: <201102020309.09640.orangewarrior@gmail.com> Message-ID: <201102021507.55292.jacob@openend.se> onsdag 02 februari 2011 03.09.09 skrev ?ukasz Ligowski: > > I'd like to ask are there any rules of thumb to write code that PyPy JIT > can easily optimize? Or maybe which constructs are hardly optimizable by > JIT so it's better to avoid them? If you want absolutely best performance, write your code in a simple, straight forward way, trying to keep your most used loops free of branches. if a: for x in range(big number): calculate something else: for x in range(big number): calculate something else generates better code than for x in range(big number): if a: calculate something else: caculate something else In the second case, there will be a guard inside the loop that has to be evaluated every time through. Jacob Hall?n From pgiarrusso at mathematik.uni-marburg.de Wed Feb 2 15:08:07 2011 From: pgiarrusso at mathematik.uni-marburg.de (Paolo Giarrusso) Date: Wed, 2 Feb 2011 15:08:07 +0100 Subject: [pypy-dev] pypy jit In-Reply-To: <201102021507.55292.jacob@openend.se> References: <201102020309.09640.orangewarrior@gmail.com> <201102021507.55292.jacob@openend.se> Message-ID: On Wed, Feb 2, 2011 at 15:07, Jacob Hall?n wrote: > onsdag 02 februari 2011 03.09.09 skrev ??ukasz Ligowski: >> >> I'd like to ask are there any rules of thumb to write code that PyPy JIT >> can easily optimize? Or maybe which constructs are hardly optimizable by >> JIT so it's better to avoid them? > > If you want absolutely best performance, write your code in a simple, straight > forward way, trying to keep your most used loops free of branches. > > if a: > ? ?for x in range(big number): > ? ? ? ?calculate something > else: > ? ?for x in range(big number): > ? ? ? ?calculate something else > > generates better code than > > for x in range(big number): > ? ?if a: > ? ? ? ?calculate something > ? ?else: > ? ? ? ?caculate something else > > In the second case, there will be a guard inside the loop that has to be > evaluated every time through. Is there a plan to automate this optimization, or are there problems to extend it to tracing JIT compilation? I think the standard name is code hoisting and other compilers do it. -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From alex.gaynor at gmail.com Wed Feb 2 15:06:32 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Wed, 2 Feb 2011 09:06:32 -0500 Subject: [pypy-dev] pypy jit In-Reply-To: <201102021507.55292.jacob@openend.se> References: <201102020309.09640.orangewarrior@gmail.com> <201102021507.55292.jacob@openend.se> Message-ID: On Wed, Feb 2, 2011 at 9:07 AM, Jacob Hall?n wrote: > onsdag 02 februari 2011 03.09.09 skrev ?ukasz Ligowski: > > > > I'd like to ask are there any rules of thumb to write code that PyPy JIT > > can easily optimize? Or maybe which constructs are hardly optimizable by > > JIT so it's better to avoid them? > > If you want absolutely best performance, write your code in a simple, > straight > forward way, trying to keep your most used loops free of branches. > > if a: > for x in range(big number): > calculate something > else: > for x in range(big number): > calculate something else > > generates better code than > > for x in range(big number): > if a: > calculate something > else: > caculate something else > > In the second case, there will be a guard inside the loop that has to be > evaluated every time through. > > Jacob Hall?n > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > If you happen to be doing numerical calculations using the array module instead of lists will help. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110202/50f48f7e/attachment-0001.htm From fijall at gmail.com Wed Feb 2 16:47:53 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 2 Feb 2011 17:47:53 +0200 Subject: [pypy-dev] pypy jit In-Reply-To: <201102021507.55292.jacob@openend.se> References: <201102020309.09640.orangewarrior@gmail.com> <201102021507.55292.jacob@openend.se> Message-ID: On Wed, Feb 2, 2011 at 4:07 PM, Jacob Hall?n wrote: > onsdag 02 februari 2011 03.09.09 skrev ??ukasz Ligowski: >> >> I'd like to ask are there any rules of thumb to write code that PyPy JIT >> can easily optimize? Or maybe which constructs are hardly optimizable by >> JIT so it's better to avoid them? > > If you want absolutely best performance, write your code in a simple, straight > forward way, trying to keep your most used loops free of branches. > > if a: > ? ?for x in range(big number): > ? ? ? ?calculate something > else: > ? ?for x in range(big number): > ? ? ? ?calculate something else > > generates better code than > > for x in range(big number): > ? ?if a: > ? ? ? ?calculate something > ? ?else: > ? ? ? ?caculate something else > > In the second case, there will be a guard inside the loop that has to be > evaluated every time through. You have outdated information Jacob, not any more :) > > Jacob Hall?n > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev From fijall at gmail.com Wed Feb 2 16:48:50 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 2 Feb 2011 17:48:50 +0200 Subject: [pypy-dev] pypy jit In-Reply-To: <201102020309.09640.orangewarrior@gmail.com> References: <201102020309.09640.orangewarrior@gmail.com> Message-ID: 2011/2/2 ?ukasz Ligowski : > Hi, > > First of all I'd like to thank all developers for good work on PyPy ;) > > I'd like to ask are there any rules of thumb to write code that PyPy JIT can > easily optimize? Or maybe which constructs are hardly optimizable by JIT so > it's better to avoid them? I should probably right it down somewhere. As of now there is no such guide. However, the rule of thumb is simple is better than complex. > > Best regards, > Lukasz Ligowski > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From fredrik.johansson at gmail.com Thu Feb 3 00:02:15 2011 From: fredrik.johansson at gmail.com (Fredrik Johansson) Date: Thu, 3 Feb 2011 00:02:15 +0100 Subject: [pypy-dev] testing floating point In-Reply-To: References: <201102012351.p11NpaJk011803@theraft.openend.se> Message-ID: Hello, On Wed, Feb 2, 2011 at 1:02 AM, Amaury Forgeot d'Arc wrote: > Hi, > > 2011/2/2 Laura Creighton : >> running the test suite of this package might be useful. > > Some user already tried it one year ago, this even triggered two bugs > in the early version of the JIT: > http://codespeak.net/pipermail/pypy-dev/2010q1/005722.html Indeed (that user was me, and it was nice to see the problem fixed so quickly!). There is one incompatibility left between CPython and PyPy that breaks some functions in mpmath. In CPython, round() and functions in the math module accept (and convert) inputs of any custom type with a __float__ method, whereas they trigger a TypeError in PyPy (1.4). > I don't remember how fast is was, though. On win32, PyPy 1.4 is about 40% slower than CPython 2.6 for running the entire set of tests. This is probably mostly due to the slowness of longs in PyPy. However, much of the regular floating point code seems to be slower in PyPy as well. Fredrik From amauryfa at gmail.com Thu Feb 3 00:10:46 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Thu, 3 Feb 2011 00:10:46 +0100 Subject: [pypy-dev] testing floating point In-Reply-To: References: <201102012351.p11NpaJk011803@theraft.openend.se> Message-ID: 2011/2/3 Fredrik Johansson : > Indeed (that user was me, and it was nice to see the problem fixed so > quickly!). There is one incompatibility left between CPython and PyPy > that breaks some functions in mpmath. In CPython, round() and > functions in the math module accept (and convert) inputs of any custom > type with a __float__ method, whereas they trigger a TypeError in PyPy > (1.4). It seems corrected today; the script below indeed fails with pypy-1.4. >>>> class C: .... def __float__(self): return 42.0 .... >>>> x = C() >>>> import math >>>> math.sqrt(x) 6.48074069840786 >> I don't remember how fast is was, though. > > On win32, PyPy 1.4 is about 40% slower than CPython 2.6 for running > the entire set of tests. This is probably mostly due to the slowness > of longs in PyPy. However, much of the regular floating point code > seems to be slower in PyPy as well. The standard operations, or the math module? It seems that every function in the math module releases the GIL. This may explain some slowness. -- Amaury Forgeot d'Arc From pgiarrusso at mathematik.uni-marburg.de Thu Feb 3 00:28:41 2011 From: pgiarrusso at mathematik.uni-marburg.de (Paolo Giarrusso) Date: Thu, 3 Feb 2011 00:28:41 +0100 Subject: [pypy-dev] testing floating point In-Reply-To: References: <201102012351.p11NpaJk011803@theraft.openend.se> Message-ID: On Thu, Feb 3, 2011 at 00:10, Amaury Forgeot d'Arc wrote: > 2011/2/3 Fredrik Johansson : >> Indeed (that user was me, and it was nice to see the problem fixed so >> quickly!). There is one incompatibility left between CPython and PyPy >> that breaks some functions in mpmath. In CPython, round() and >> functions in the math module accept (and convert) inputs of any custom >> type with a __float__ method, whereas they trigger a TypeError in PyPy >> (1.4). > > It seems corrected today; the script below indeed fails with pypy-1.4. According to Fredik, the script is supposed to pass. By "corrected" you mean it's fixed on the trunk? >>>>> class C: > .... ? def __float__(self): return 42.0 > .... >>>>> x = C() >>>>> import math >>>>> math.sqrt(x) > 6.48074069840786 >>> I don't remember how fast is was, though. >> >> On win32, PyPy 1.4 is about 40% slower than CPython 2.6 for running >> the entire set of tests. This is probably mostly due to the slowness >> of longs in PyPy. However, much of the regular floating point code >> seems to be slower in PyPy as well. > > The standard operations, or the math module? > It seems that every function in the math module releases the GIL. Does that module live in pypy/module/math/? Reading the source doesn't show obvious interactions with the GIL, so I guess that for many other modules the GIL is released automatically, potentially even for other fast functions. > This may explain some slowness. In other words, a simple call to sin()/cos()/ceil() triggers a cache flush (costing around ~100 cycles) because of releasing a lock, if not a system call (costing thousands of cycles), say if the lock was contended? If those functions were lifted to operate on lists they should release the lock for long lists, otherwise it seems a bad idea. In other words, I'd propose to remove the unlocking/relocking if possible. Can that affect correctness for some weird reason? I'd guess not, but I'm not familiar with the code enough to get fully what happens. Best regards -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From fredrik.johansson at gmail.com Thu Feb 3 00:29:30 2011 From: fredrik.johansson at gmail.com (Fredrik Johansson) Date: Thu, 3 Feb 2011 00:29:30 +0100 Subject: [pypy-dev] testing floating point In-Reply-To: References: <201102012351.p11NpaJk011803@theraft.openend.se> Message-ID: On Thu, Feb 3, 2011 at 12:10 AM, Amaury Forgeot d'Arc wrote: > 2011/2/3 Fredrik Johansson : >> Indeed (that user was me, and it was nice to see the problem fixed so >> quickly!). There is one incompatibility left between CPython and PyPy >> that breaks some functions in mpmath. In CPython, round() and >> functions in the math module accept (and convert) inputs of any custom >> type with a __float__ method, whereas they trigger a TypeError in PyPy >> (1.4). > > It seems corrected today; the script below indeed fails with pypy-1.4. Ah, that's nice. Does round() work as well? >> On win32, PyPy 1.4 is about 40% slower than CPython 2.6 for running >> the entire set of tests. This is probably mostly due to the slowness >> of longs in PyPy. However, much of the regular floating point code >> seems to be slower in PyPy as well. > > The standard operations, or the math module? > It seems that every function in the math module releases the GIL. > This may explain some slowness. I have only looked at some quick macro level timings. Generally a lot of overhead for function calls and type checking is involved (although I would hope that PyPy can kill precisely some of that eventually). Fredrik From alex.gaynor at gmail.com Thu Feb 3 00:32:12 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Wed, 2 Feb 2011 18:32:12 -0500 Subject: [pypy-dev] testing floating point In-Reply-To: References: <201102012351.p11NpaJk011803@theraft.openend.se> Message-ID: On Wed, Feb 2, 2011 at 6:29 PM, Fredrik Johansson < fredrik.johansson at gmail.com> wrote: > On Thu, Feb 3, 2011 at 12:10 AM, Amaury Forgeot d'Arc > wrote: > > 2011/2/3 Fredrik Johansson : > >> Indeed (that user was me, and it was nice to see the problem fixed so > >> quickly!). There is one incompatibility left between CPython and PyPy > >> that breaks some functions in mpmath. In CPython, round() and > >> functions in the math module accept (and convert) inputs of any custom > >> type with a __float__ method, whereas they trigger a TypeError in PyPy > >> (1.4). > > > > It seems corrected today; the script below indeed fails with pypy-1.4. > > Ah, that's nice. Does round() work as well? > > >> On win32, PyPy 1.4 is about 40% slower than CPython 2.6 for running > >> the entire set of tests. This is probably mostly due to the slowness > >> of longs in PyPy. However, much of the regular floating point code > >> seems to be slower in PyPy as well. > > > > The standard operations, or the math module? > > It seems that every function in the math module releases the GIL. > > This may explain some slowness. > > I have only looked at some quick macro level timings. Generally a lot > of overhead for function calls and type checking is involved (although > I would hope that PyPy can kill precisely some of that eventually). > > Fredrik > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > If you have any benchmarks where we are slower that don't involve longs that'd be great, since for float operations we really should be able to beat up on CPython. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110202/573693c5/attachment.htm From amauryfa at gmail.com Thu Feb 3 00:42:09 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Thu, 3 Feb 2011 00:42:09 +0100 Subject: [pypy-dev] testing floating point In-Reply-To: References: <201102012351.p11NpaJk011803@theraft.openend.se> Message-ID: 2011/2/3 Paolo Giarrusso : >> It seems that every function in the math module releases the GIL. > Does that module live in pypy/module/math/? Reading the source doesn't > show obvious interactions with the GIL, so I guess that for many other > modules the GIL is released automatically, potentially even for other > fast functions. Fortunately, I was wrong, sorry. The functions of the math module don't release the GIL. The low-level operations are in pypy/rpython/lltypesystem/module/ll_math.py, all functions are marked with "sandboxsafe=True" which, according to a comment in rffi.py, ensures that the GIL won't be released. -- Amaury Forgeot d'Arc From fredrik.johansson at gmail.com Thu Feb 3 00:43:57 2011 From: fredrik.johansson at gmail.com (Fredrik Johansson) Date: Thu, 3 Feb 2011 00:43:57 +0100 Subject: [pypy-dev] testing floating point In-Reply-To: References: <201102012351.p11NpaJk011803@theraft.openend.se> Message-ID: On Thu, Feb 3, 2011 at 12:32 AM, Alex Gaynor wrote: > If you have any benchmarks where we are slower that don't involve longs > that'd be great, since for float operations we really should be able to beat > up on CPython. A simple example: mpmath.fp.fsum(mpmath.fp.sqrt(k) for k in xrange(1000000)) PyPy 1.4 runs this about 2.4 times slower than CPython. Fredrik From fijall at gmail.com Thu Feb 3 08:01:08 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 3 Feb 2011 09:01:08 +0200 Subject: [pypy-dev] testing floating point In-Reply-To: References: <201102012351.p11NpaJk011803@theraft.openend.se> Message-ID: On Thu, Feb 3, 2011 at 1:43 AM, Fredrik Johansson wrote: > On Thu, Feb 3, 2011 at 12:32 AM, Alex Gaynor wrote: >> If you have any benchmarks where we are slower that don't involve longs >> that'd be great, since for float operations we really should be able to beat >> up on CPython. > > A simple example: mpmath.fp.fsum(mpmath.fp.sqrt(k) for k in xrange(1000000)) A brief follow up. * PyPy trunk is faster (by quite a bit). * I noticed that you happily use mixture of old and new style classes. As of now this is a really bad case for PyPy. Example: >>>> [isinstance(i, type) for i in mpmath.fp.__class__.__mro__] [True, True, True, True, False, False, True, True, False, True, True, True, True, True, True] when I replace it with newstyle classes it runs much faster Other things that speed up both CPython and PyPy: * Put this things into function instead of at global scope * Use list comprehension instead of generator expression. That all should make PyPy 3x faster than CPython. > > PyPy 1.4 runs this about 2.4 times slower than CPython. > > Fredrik > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From fredrik.johansson at gmail.com Thu Feb 3 10:56:13 2011 From: fredrik.johansson at gmail.com (Fredrik Johansson) Date: Thu, 3 Feb 2011 10:56:13 +0100 Subject: [pypy-dev] testing floating point In-Reply-To: References: <201102012351.p11NpaJk011803@theraft.openend.se> Message-ID: On Thu, Feb 3, 2011 at 8:01 AM, Maciej Fijalkowski wrote: > On Thu, Feb 3, 2011 at 1:43 AM, Fredrik Johansson > wrote: >> On Thu, Feb 3, 2011 at 12:32 AM, Alex Gaynor wrote: >>> If you have any benchmarks where we are slower that don't involve longs >>> that'd be great, since for float operations we really should be able to beat >>> up on CPython. >> >> A simple example: mpmath.fp.fsum(mpmath.fp.sqrt(k) for k in xrange(1000000)) > > A brief follow up. > > * PyPy trunk is faster (by quite a bit). > > * I noticed that you happily use mixture of old and new style classes. > As of now this is a really bad case for PyPy. Example: > >>>>> [isinstance(i, type) for i in mpmath.fp.__class__.__mro__] > [True, True, True, True, False, False, True, True, False, True, True, > True, True, True, True] > > when I replace it with newstyle classes it runs much faster Interesting. The mixture of old and new style classes is a mistake, of course. I'm going to add a test to make sure this doesn't happen. Thanks for pointing it out. In fact this speeds up another benchmark I did -- [fp.lambertw(k) for k in xrange(50000)]-- by 10x, which is quite a ridiculous ratio! > Other things that speed up both CPython and PyPy: > > * Put this things into function instead of at global scope Do you mean in the benchmark or did you have some other code you saw in mind? > * Use list comprehension instead of generator expression. I hope PyPy can do more in the future to speed up generator expressions. Fredrik From fijall at gmail.com Thu Feb 3 11:14:53 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 3 Feb 2011 12:14:53 +0200 Subject: [pypy-dev] testing floating point In-Reply-To: References: <201102012351.p11NpaJk011803@theraft.openend.se> Message-ID: On Thu, Feb 3, 2011 at 11:56 AM, Fredrik Johansson wrote: > On Thu, Feb 3, 2011 at 8:01 AM, Maciej Fijalkowski wrote: >> On Thu, Feb 3, 2011 at 1:43 AM, Fredrik Johansson >> wrote: >>> On Thu, Feb 3, 2011 at 12:32 AM, Alex Gaynor wrote: >>>> If you have any benchmarks where we are slower that don't involve longs >>>> that'd be great, since for float operations we really should be able to beat >>>> up on CPython. >>> >>> A simple example: mpmath.fp.fsum(mpmath.fp.sqrt(k) for k in xrange(1000000)) >> >> A brief follow up. >> >> * PyPy trunk is faster (by quite a bit). >> >> * I noticed that you happily use mixture of old and new style classes. >> As of now this is a really bad case for PyPy. Example: >> >>>>>> [isinstance(i, type) for i in mpmath.fp.__class__.__mro__] >> [True, True, True, True, False, False, True, True, False, True, True, >> True, True, True, True] >> >> when I replace it with newstyle classes it runs much faster > > Interesting. The mixture of old and new style classes is a mistake, of > course. I'm going to add a test to make sure this doesn't happen. > Thanks for pointing it out. > > In fact this speeds up another benchmark I did -- [fp.lambertw(k) for > k in xrange(50000)]-- by 10x, which is quite a ridiculous ratio! Mixture of old and new style classes is not only preventing us from doing optimizations but also hits a bad case of tradeoffs. However, we decided we don't care that much. You should use new style classes anyway :) > >> Other things that speed up both CPython and PyPy: >> >> * Put this things into function instead of at global scope > > Do you mean in the benchmark or did you have some other code you saw in mind? The benchmark. > >> * Use list comprehension instead of generator expression. > > I hope PyPy can do more in the future to speed up generator expressions. It doesn't speed up things by much. Yeah, I can imagine we can improve on this, but it's also a bit hard. > > Fredrik > From fredrik.johansson at gmail.com Thu Feb 3 11:23:02 2011 From: fredrik.johansson at gmail.com (Fredrik Johansson) Date: Thu, 3 Feb 2011 11:23:02 +0100 Subject: [pypy-dev] testing floating point In-Reply-To: References: <201102012351.p11NpaJk011803@theraft.openend.se> Message-ID: On Thu, Feb 3, 2011 at 11:14 AM, Maciej Fijalkowski wrote: > Mixture of old and new style classes is not only preventing us from > doing optimizations but also hits a bad case of tradeoffs. However, we > decided we don't care that much. You should use new style classes > anyway :) Yes, this is perfectly reasonable. You should make PyPy print warnings when it encounters mixed-type classes :) Fredrik From fijall at gmail.com Thu Feb 3 11:26:46 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 3 Feb 2011 12:26:46 +0200 Subject: [pypy-dev] testing floating point In-Reply-To: References: <201102012351.p11NpaJk011803@theraft.openend.se> Message-ID: On Thu, Feb 3, 2011 at 12:23 PM, Fredrik Johansson wrote: > On Thu, Feb 3, 2011 at 11:14 AM, Maciej Fijalkowski wrote: >> Mixture of old and new style classes is not only preventing us from >> doing optimizations but also hits a bad case of tradeoffs. However, we >> decided we don't care that much. You should use new style classes >> anyway :) > > Yes, this is perfectly reasonable. You should make PyPy print warnings > when it encounters mixed-type classes :) > > Fredrik > That's not a bad idea actually. Maybe with something like some -X option? From amauryfa at gmail.com Thu Feb 3 13:10:43 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Thu, 3 Feb 2011 13:10:43 +0100 Subject: [pypy-dev] testing floating point In-Reply-To: References: <201102012351.p11NpaJk011803@theraft.openend.se> Message-ID: 2011/2/3 Maciej Fijalkowski : > You should make PyPy print warnings >> when it encounters mixed-type classes :) >> > That's not a bad idea actually. Maybe with something like some -X option? If we use the warnings module to emit JitWarnings, the option could be -Wd::JitWarning -- Amaury Forgeot d'Arc From fijall at gmail.com Thu Feb 3 13:13:55 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 3 Feb 2011 14:13:55 +0200 Subject: [pypy-dev] testing floating point In-Reply-To: References: <201102012351.p11NpaJk011803@theraft.openend.se> Message-ID: On Thu, Feb 3, 2011 at 2:10 PM, Amaury Forgeot d'Arc wrote: > 2011/2/3 Maciej Fijalkowski : >> You should make PyPy print warnings >>> when it encounters mixed-type classes :) >>> >> That's not a bad idea actually. Maybe with something like some -X option? > > If we use the warnings module to emit JitWarnings, the option could be > -Wd::JitWarning I'm not sure about warnings module. How about --jit warnings=1 ? That would fit with other jit options. I was more thinking about this being optional > > -- > Amaury Forgeot d'Arc > From anto.cuni at gmail.com Thu Feb 3 13:21:57 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu, 03 Feb 2011 13:21:57 +0100 Subject: [pypy-dev] testing floating point In-Reply-To: References: <201102012351.p11NpaJk011803@theraft.openend.se> Message-ID: <4D4A9DE5.1070601@gmail.com> On 03/02/11 13:13, Maciej Fijalkowski wrote: > I'm not sure about warnings module. How about > --jit warnings=1 > ? > > That would fit with other jit options. not really. The other jit options really belongs to the JIT engine, while this one is dependent on the Python interpreter ciao, Anto From fijall at gmail.com Thu Feb 3 13:22:57 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 3 Feb 2011 14:22:57 +0200 Subject: [pypy-dev] testing floating point In-Reply-To: <4D4A9DE5.1070601@gmail.com> References: <201102012351.p11NpaJk011803@theraft.openend.se> <4D4A9DE5.1070601@gmail.com> Message-ID: On Thu, Feb 3, 2011 at 2:21 PM, Antonio Cuni wrote: > On 03/02/11 13:13, Maciej Fijalkowski wrote: > >> I'm not sure about warnings module. How about >> --jit warnings=1 >> ? >> >> That would fit with other jit options. > > not really. The other jit options really belongs to the JIT engine, while this > one is dependent on the Python interpreter true. > > ciao, > Anto > From arigo at tunes.org Thu Feb 3 13:30:16 2011 From: arigo at tunes.org (Armin Rigo) Date: Thu, 3 Feb 2011 13:30:16 +0100 Subject: [pypy-dev] Link errors while translating with VS2010 and 64bit In-Reply-To: References: <-3229585076398878208@unknownmsgid> Message-ID: Hi Tasos, On Wed, Feb 2, 2011 at 1:15 AM, Tasos Vogiatzoglou wrote: > You are right. My mistake. It was picking the 2.6 32bit python. Now is > working and tests are passing A few extra notes. What is needed to make this work is three-fold: * make sure that in the C code, "intptr_t" is generated instead of "long" for lltype.Signed, and similarly "uintptr_t" instead of "unsigned long" for lltype.Unsigned. This involves mainly fixing translator/c/primitive.py to declare types using "intptr_t" instead of "long", and probably fixing a number of other uses of "long" in the .py files in that directory. * make sure that in the hand-written headers translator/c/src/*.h, we use "intptr_t" instead of "long" whenever it's needed (mostly everywhere, I suppose, but not absolutely everywhere, e.g. not in calls to external C functions that are declared to take or return a long). The changes above should fix the generation of code. Now the third part is the longest, probably. You need to distinguish the two types used in RPython, which are lltype.Signed and rffi.LONG, and which are equal so far. Of course rffi.LONG should remain equivalent to the C long. This probably needs looking around in all modules and in pypy/rlib/, to make sure that we use the correct one, either lltype.Signed or rffi.LONG. Fortunately, errors resulting from this confusion will probably just give translation-time errors, so we can fix the places one after the other. But you need to carefully check the declaration of external functions that are implemented in translator/c/src/*.h, like pypy/module/signal/interp_signal.py (this one happens to use mostly rffi.INT, so it should work right now, with the exception of LONG_STRUCT that uses a lltype.Signed). A bient?t, Armin. From tvoglou at gmail.com Thu Feb 3 15:09:00 2011 From: tvoglou at gmail.com (Tasos Vogiatzoglou) Date: Thu, 3 Feb 2011 16:09:00 +0200 Subject: [pypy-dev] Link errors while translating with VS2010 and 64bit In-Reply-To: References: <-3229585076398878208@unknownmsgid> Message-ID: <-4189686393236333789@unknownmsgid> Armin, thanks for the directions. I'll start working on them and I'll report back. Regards, Tasos On 3 ??? 2011, at 14:30, Armin Rigo wrote: > Hi Tasos, > > On Wed, Feb 2, 2011 at 1:15 AM, Tasos Vogiatzoglou wrote: >> You are right. My mistake. It was picking the 2.6 32bit python. Now is >> working and tests are passing > > A few extra notes. What is needed to make this work is three-fold: > > * make sure that in the C code, "intptr_t" is generated instead of > "long" for lltype.Signed, and similarly "uintptr_t" instead of > "unsigned long" for lltype.Unsigned. This involves mainly fixing > translator/c/primitive.py to declare types using "intptr_t" instead of > "long", and probably fixing a number of other uses of "long" in the > .py files in that directory. > > * make sure that in the hand-written headers translator/c/src/*.h, we > use "intptr_t" instead of "long" whenever it's needed (mostly > everywhere, I suppose, but not absolutely everywhere, e.g. not in > calls to external C functions that are declared to take or return a > long). > > The changes above should fix the generation of code. Now the third > part is the longest, probably. You need to distinguish the two types > used in RPython, which are lltype.Signed and rffi.LONG, and which are > equal so far. Of course rffi.LONG should remain equivalent to the C > long. This probably needs looking around in all modules and in > pypy/rlib/, to make sure that we use the correct one, either > lltype.Signed or rffi.LONG. Fortunately, errors resulting from this > confusion will probably just give translation-time errors, so we can > fix the places one after the other. But you need to carefully check > the declaration of external functions that are implemented in > translator/c/src/*.h, like pypy/module/signal/interp_signal.py (this > one happens to use mostly rffi.INT, so it should work right now, with > the exception of LONG_STRUCT that uses a lltype.Signed). > > > A bient?t, > > Armin. From arigo at tunes.org Thu Feb 3 16:03:13 2011 From: arigo at tunes.org (Armin Rigo) Date: Thu, 3 Feb 2011 16:03:13 +0100 Subject: [pypy-dev] pypy jit In-Reply-To: References: <201102020309.09640.orangewarrior@gmail.com> Message-ID: Hi, On Wed, Feb 2, 2011 at 4:48 PM, Maciej Fijalkowski wrote: > As of now there is no such guide. However, the rule of thumb is simple > is better than complex. To some extend that is true; however, note that the second rule is "but even on messily complicated code the JIT can do something" :-) For example, "translate.py" was not written with the JIT in mind at all, but turns out to be twice as fast on PyPy. To get a better summary, I think that we can say that, to some extent, there is little point in spending ages tweaking your Python code to make it more JIT-friendly. If there is any Python code that would get seriously faster by some trivial tweaking, then it's somehow a bug, and we need to fix it. Of course, if you use introspection of the stack frames or even pdb (the Python debugger) all over the place, you need to expect the code to be hard to optimize for us. But I guess that it should somehow be clear. For the rest, any decision that can be resolved "locally" can nicely be optimized by the JIT, whatever the answer is (for example, "is this '+' going to just add integers, or does it invoke some __add__() method?"). A bient?t, Armin. From stefan_ml at behnel.de Thu Feb 3 17:15:53 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 03 Feb 2011 17:15:53 +0100 Subject: [pypy-dev] testing floating point In-Reply-To: References: <201102012351.p11NpaJk011803@theraft.openend.se> Message-ID: Maciej Fijalkowski, 03.02.2011 11:14: > On Thu, Feb 3, 2011 at 11:56 AM, Fredrik Johansson wrote: >> On Thu, Feb 3, 2011 at 8:01 AM, Maciej Fijalkowski wrote: >>> * Use list comprehension instead of generator expression. >> >> I hope PyPy can do more in the future to speed up generator expressions. > > It doesn't speed up things by much. Yeah, I can imagine we can improve > on this, but it's also a bit hard. Does PyPy generate inlined looping code for them if applicable? (e.g. for any(), all(), sum(), sorted(), ...) Stefan From alex.gaynor at gmail.com Thu Feb 3 17:25:41 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Thu, 3 Feb 2011 11:25:41 -0500 Subject: [pypy-dev] testing floating point In-Reply-To: References: <201102012351.p11NpaJk011803@theraft.openend.se> Message-ID: On Thu, Feb 3, 2011 at 11:15 AM, Stefan Behnel wrote: > Maciej Fijalkowski, 03.02.2011 11:14: > > On Thu, Feb 3, 2011 at 11:56 AM, Fredrik Johansson wrote: > >> On Thu, Feb 3, 2011 at 8:01 AM, Maciej Fijalkowski wrote: > >>> * Use list comprehension instead of generator expression. > >> > >> I hope PyPy can do more in the future to speed up generator expressions. > > > > It doesn't speed up things by much. Yeah, I can imagine we can improve > > on this, but it's also a bit hard. > > Does PyPy generate inlined looping code for them if applicable? > > (e.g. for any(), all(), sum(), sorted(), ...) > > Stefan > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > No, it does not (with the exception of map() which has its own JIT, but that code is still not inlined into the caller). Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110203/1cd239a4/attachment.htm From fijall at gmail.com Fri Feb 4 07:41:07 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 4 Feb 2011 08:41:07 +0200 Subject: [pypy-dev] testing floating point In-Reply-To: References: <201102012351.p11NpaJk011803@theraft.openend.se> Message-ID: On Thu, Feb 3, 2011 at 6:15 PM, Stefan Behnel wrote: > Maciej Fijalkowski, 03.02.2011 11:14: >> On Thu, Feb 3, 2011 at 11:56 AM, Fredrik Johansson wrote: >>> On Thu, Feb 3, 2011 at 8:01 AM, Maciej Fijalkowski wrote: >>>> * Use list comprehension instead of generator expression. >>> >>> I hope PyPy can do more in the future to speed up generator expressions. >> >> It doesn't speed up things by much. Yeah, I can imagine we can improve >> on this, but it's also a bit hard. > > Does PyPy generate inlined looping code for them if applicable? > > (e.g. for any(), all(), sum(), sorted(), ...) We don't inline generator expr. The reason is that it's hard to judge when generator frame escapes. We can probably do something with it, but it's not done yet. > > Stefan > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From fijall at gmail.com Fri Feb 4 07:44:07 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 4 Feb 2011 08:44:07 +0200 Subject: [pypy-dev] pypy jit In-Reply-To: References: <201102020309.09640.orangewarrior@gmail.com> Message-ID: On Thu, Feb 3, 2011 at 5:03 PM, Armin Rigo wrote: > Hi, > > On Wed, Feb 2, 2011 at 4:48 PM, Maciej Fijalkowski wrote: >> As of now there is no such guide. However, the rule of thumb is simple >> is better than complex. > > To some extend that is true; however, note that the second rule is > "but even on messily complicated code the JIT can do something" :-) > For example, "translate.py" was not written with the JIT in mind at > all, but turns out to be twice as fast on PyPy. > > To get a better summary, I think that we can say that, to some extent, > there is little point in spending ages tweaking your Python code to > make it more JIT-friendly. ?If there is any Python code that would get > seriously faster by some trivial tweaking, then it's somehow a bug, > and we need to fix it. ?Of course, if you use introspection of the > stack frames or even pdb (the Python debugger) all over the place, you > need to expect the code to be hard to optimize for us. ?But I guess > that it should somehow be clear. ?For the rest, any decision that can > be resolved "locally" can nicely be optimized by the JIT, whatever the > answer is (for example, "is this '+' going to just add integers, or > does it invoke some __add__() method?"). > I disagree a bit. The example would be newstylization of classes. This is making your code more JIT friendly and it's not trivial to optimize this from the JIT standpoint. > > A bient?t, > > Armin. > From arigo at tunes.org Fri Feb 4 17:12:55 2011 From: arigo at tunes.org (Armin Rigo) Date: Fri, 4 Feb 2011 17:12:55 +0100 Subject: [pypy-dev] pypy jit In-Reply-To: References: <201102020309.09640.orangewarrior@gmail.com> Message-ID: Hi, On Fri, Feb 4, 2011 at 7:44 AM, Maciej Fijalkowski wrote: > I disagree a bit. The example would be newstylization of classes. This > is making your code more JIT friendly and it's not trivial to optimize > this from the JIT standpoint. "We know there is an issue there but we don't want to invest the large efforts to improve it because it's really the Python program's fault for using a mixture of deprecated and modern features"... Sure, you have a point. I was taking a rather theoretical point of view in my previous e-mail. A bient?t, Armin. From fijall at gmail.com Fri Feb 4 18:55:40 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 4 Feb 2011 19:55:40 +0200 Subject: [pypy-dev] pypy jit In-Reply-To: References: <201102020309.09640.orangewarrior@gmail.com> Message-ID: On Fri, Feb 4, 2011 at 6:12 PM, Armin Rigo wrote: > Hi, > > On Fri, Feb 4, 2011 at 7:44 AM, Maciej Fijalkowski wrote: >> I disagree a bit. The example would be newstylization of classes. This >> is making your code more JIT friendly and it's not trivial to optimize >> this from the JIT standpoint. > > "We know there is an issue there but we don't want to invest the large > efforts to improve it because it's really the Python program's fault > for using a mixture of deprecated and modern features"... ?Sure, you > have a point. ?I was taking a rather theoretical point of view in my > previous e-mail. > Thank you for summarizing :-) I started a wiki btw: https://bitbucket.org/pypy/pypy/wiki/JitFriendliness > > A bient?t, > > Armin. > From tismer at stackless.com Sat Feb 5 23:09:19 2011 From: tismer at stackless.com (Christian Tismer) Date: Sat, 05 Feb 2011 23:09:19 +0100 Subject: [pypy-dev] Compat. in 1.4.1 __del__ Message-ID: <4D4DCA8F.7070508@stackless.com> Howdy, studying the differences of PyPy vs. CPython, most seem to be fine; one thing where I an unsure is the __del__ behavior. I am not addressing its delayed call or the number it is called, this is similar to Jython and IronPython. But assigning to __del__ after a class is created, is that so hard to implement? It is clear to me that the existence of a __del__ heavily influences the whole code generation of a class. So as a naive thought, can we just abandon the whole compilation and start over (maybe even less)? The problem that I see is re-generation of already created instances, triggering some re-compilation. I would think of all compiled code always going through some "fuse code" that would trigger re-compilation if an assignment to __del__ happens. For me, this looks like a guard, that exists in other places of the JIT, where existing assumptions must check if they are still true, and otherwise start over. I have not read the current implementation, yet, because my reading is still quite impaired. But from the top of my head: Can it be that such a re-compile as I mentioned is not yet considered because that is not involving the JIT, but an already fixed part of the static compilation chain? I think not, since we are not in RPython, but at the translation of an applevel class, which is completely run-time compiled. So, where is my lack of seeing the problem? Or is it simply a lot of effort? If it is an efficiency concern, then my POV is: Better to be slow in a few cases of re-compilation, than to have to hunt for such delicate existing code. I would instead issue an optional warning to dynamic __del__ assignment instead, in order to let people avoid it in the future. regards, and thanks for correcting me - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From benjamin at python.org Sat Feb 5 23:47:50 2011 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 5 Feb 2011 16:47:50 -0600 Subject: [pypy-dev] Compat. in 1.4.1 __del__ In-Reply-To: <4D4DCA8F.7070508@stackless.com> References: <4D4DCA8F.7070508@stackless.com> Message-ID: 2011/2/5 Christian Tismer : > Howdy, > > studying the differences of PyPy vs. CPython, most seem to be fine; > one thing where I an unsure is the __del__ behavior. > > I am not addressing its delayed call or the number it is called, this > is similar to Jython and IronPython. > > But assigning to __del__ after a class is created, is that so hard > to implement? It's not a JIT problem rather a RPython/gc one. All the RPython classes with finalizers must be known at translation time. __del__ is expensive in the for gc. To implement user level __del__, a different underlying interp class is used which has its own __del__ which the gc will call. -- Regards, Benjamin From tismer at stackless.com Sun Feb 6 00:10:48 2011 From: tismer at stackless.com (Christian Tismer) Date: Sun, 06 Feb 2011 00:10:48 +0100 Subject: [pypy-dev] Compat. in 1.4.1 __del__ In-Reply-To: References: <4D4DCA8F.7070508@stackless.com> Message-ID: <4D4DD8F8.7080308@stackless.com> On 2/5/11 11:47 PM, Benjamin Peterson wrote: > 2011/2/5 Christian Tismer: >> Howdy, >> >> studying the differences of PyPy vs. CPython, most seem to be fine; >> one thing where I an unsure is the __del__ behavior. >> >> I am not addressing its delayed call or the number it is called, this >> is similar to Jython and IronPython. >> >> But assigning to __del__ after a class is created, is that so hard >> to implement? > It's not a JIT problem rather a RPython/gc one. All the RPython > classes with finalizers must be known at translation time. __del__ is > expensive in the for gc. To implement user level __del__, a different > underlying interp class is used which has its own __del__ which the gc > will call. I understand. Then having a __del__ is always expensive, I guess? I mean, does it involve overhead to have a __del__ at all, runtime or compile time? How feasible would it be to generate always two versions of the RPython class with a stub __del__ method, which calls a yet non-existing function? sorry if that is non-sense, but maybe something can be done to isolate these bad spots, without simply silently not calling them. cheers - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From benjamin at python.org Sun Feb 6 00:25:26 2011 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 5 Feb 2011 17:25:26 -0600 Subject: [pypy-dev] Compat. in 1.4.1 __del__ In-Reply-To: <4D4DD8F8.7080308@stackless.com> References: <4D4DCA8F.7070508@stackless.com> <4D4DD8F8.7080308@stackless.com> Message-ID: 2011/2/5 Christian Tismer : > On 2/5/11 11:47 PM, Benjamin Peterson wrote: >> >> 2011/2/5 Christian Tismer: >>> >>> Howdy, >>> >>> studying the differences of PyPy vs. CPython, most seem to be fine; >>> one thing where I an unsure is the __del__ behavior. >>> >>> I am not addressing its delayed call or the number it is called, this >>> is similar to Jython and IronPython. >>> >>> But assigning to __del__ after a class is created, is that so hard >>> to implement? >> >> It's not a JIT problem rather a RPython/gc one. All the RPython >> classes with finalizers must be known at translation time. __del__ is >> expensive in the for gc. To implement user level __del__, a different >> underlying interp class is used which has its own __del__ which the gc >> will call. > > I understand. Then having a __del__ is always expensive, I guess? > I mean, does it involve overhead to have a __del__ at all, runtime > or compile time? It just means there's a lot of runtime bookkeeping in the GC for objects with __del__. > > How feasible would it be to generate always two versions of > the RPython class with a stub __del__ method, which calls a > yet non-existing function? > > sorry if that is non-sense, but maybe something can be done > to isolate these bad spots, without simply silently not calling them. Well, you do at least get a nice warning. I don't think this is too hard to work around in apps. There's also weakrefs. > > cheers - chris -- Regards, Benjamin From tismer at stackless.com Sun Feb 6 00:49:00 2011 From: tismer at stackless.com (Christian Tismer) Date: Sun, 06 Feb 2011 00:49:00 +0100 Subject: [pypy-dev] Compat. in 1.4.1 __del__ In-Reply-To: References: <4D4DCA8F.7070508@stackless.com> <4D4DD8F8.7080308@stackless.com> Message-ID: <4D4DE1EC.6020107@stackless.com> On 2/6/11 12:25 AM, Benjamin Peterson wrote: > 2011/2/5 Christian Tismer: >> On 2/5/11 11:47 PM, Benjamin Peterson wrote: >>> 2011/2/5 Christian Tismer: >>>> Howdy, >>>> >>>> studying the differences of PyPy vs. CPython, most seem to be fine; >>>> one thing where I an unsure is the __del__ behavior. >>>> >>>> I am not addressing its delayed call or the number it is called, this >>>> is similar to Jython and IronPython. >>>> >>>> But assigning to __del__ after a class is created, is that so hard >>>> to implement? >>> It's not a JIT problem rather a RPython/gc one. All the RPython >>> classes with finalizers must be known at translation time. __del__ is >>> expensive in the for gc. To implement user level __del__, a different >>> underlying interp class is used which has its own __del__ which the gc >>> will call. >> I understand. Then having a __del__ is always expensive, I guess? >> I mean, does it involve overhead to have a __del__ at all, runtime >> or compile time? > It just means there's a lot of runtime bookkeeping in the GC for > objects with __del__. > >> How feasible would it be to generate always two versions of >> the RPython class with a stub __del__ method, which calls a >> yet non-existing function? >> >> sorry if that is non-sense, but maybe something can be done >> to isolate these bad spots, without simply silently not calling them. > Well, you do at least get a nice warning. I don't think this is too > hard to work around in apps. There's also weakrefs. > Ah, we do get a warning, that is good, did not know. Anyway, the concept of RPython is pretty old, since we needed something that is treatable. And RPython is a good compromise for most things. Meanwhile, a lot of evolution has happened, most of it in the JIT area, and maybe some things can be generated more lazily and taken over by the JIT, since dynamic changes like the sudden appearance of a method could probably be handled differently than it is right now. I will shut up, this is becoming a more general topic. ciao - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From cfbolz at gmx.de Sun Feb 6 05:48:32 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Sun, 06 Feb 2011 05:48:32 +0100 Subject: [pypy-dev] Compat. in 1.4.1 __del__ In-Reply-To: <4D4DE1EC.6020107@stackless.com> References: <4D4DCA8F.7070508@stackless.com> <4D4DD8F8.7080308@stackless.com> <4D4DE1EC.6020107@stackless.com> Message-ID: <4D4E2820.6080605@gmx.de> On Saturday, February 5, 2011, Christian Tismer wrote: > Ah, we do get a warning, that is good, did not know. > > Anyway, the concept of RPython is pretty old, since we needed > something that is treatable. And RPython is a good compromise > for most things. > > Meanwhile, a lot of evolution has happened, most of it in the JIT > area, and maybe some things can be generated more lazily > and taken over by the JIT, since dynamic changes like the sudden > appearance of a method could probably be handled differently > than it is right now. > This is not really a JIT thing at all. The GC needs to do a lot of work for every object with a del at every collection. So giving a del to all objects would need a very different approach in the GC. Cheers, Carl Friedrich From dinov at microsoft.com Sun Feb 6 06:04:37 2011 From: dinov at microsoft.com (Dino Viehland) Date: Sun, 6 Feb 2011 05:04:37 +0000 Subject: [pypy-dev] Compat. in 1.4.1 __del__ In-Reply-To: <4D4DCA8F.7070508@stackless.com> References: <4D4DCA8F.7070508@stackless.com> Message-ID: <6C7ABA8B4E309440B857D74348836F2E0148B204@TK5EX14MBXC141.redmond.corp.microsoft.com> Christianwrote: > Howdy, > > studying the differences of PyPy vs. CPython, most seem to be fine; one > thing where I an unsure is the __del__ behavior. > > I am not addressing its delayed call or the number it is called, this is similar to > Jython and IronPython. > > But assigning to __del__ after a class is created, is that so hard to implement? IronPython also doesn't handle assigning to __del__ after the class is created, and I'd be surprised if Jython did as well. To make this work we'd need to maintain a weak reference for every object of a user defined type and I think most users would rather not pay that expense for such a corner case. I've also never actually heard of this breaking compatibility anywhere. I'd say if this was really important to you then start off w/ a nop __del__. Then you can change __del_ to whatever you want later. I'm not certain that would work w/ PyPy but I'd be surprised if it didn't - it will work w/ IronPython. From alex.gaynor at gmail.com Sun Feb 6 06:26:19 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Sun, 6 Feb 2011 00:26:19 -0500 Subject: [pypy-dev] Compat. in 1.4.1 __del__ In-Reply-To: <6C7ABA8B4E309440B857D74348836F2E0148B204@TK5EX14MBXC141.redmond.corp.microsoft.com> References: <4D4DCA8F.7070508@stackless.com> <6C7ABA8B4E309440B857D74348836F2E0148B204@TK5EX14MBXC141.redmond.corp.microsoft.com> Message-ID: On Sun, Feb 6, 2011 at 12:04 AM, Dino Viehland wrote: > > Christianwrote: > > Howdy, > > > > studying the differences of PyPy vs. CPython, most seem to be fine; one > > thing where I an unsure is the __del__ behavior. > > > > I am not addressing its delayed call or the number it is called, this is > similar to > > Jython and IronPython. > > > > But assigning to __del__ after a class is created, is that so hard to > implement? > > IronPython also doesn't handle assigning to __del__ after the class is > created, and > I'd be surprised if Jython did as well. To make this work we'd need to > maintain a > weak reference for every object of a user defined type and I think most > users > would rather not pay that expense for such a corner case. I've also never > actually > heard of this breaking compatibility anywhere. > > I'd say if this was really important to you then start off w/ a nop > __del__. Then you can > change __del_ to whatever you want later. I'm not certain that would work > w/ > PyPy but I'd be surprised if it didn't - it will work w/ IronPython. > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > I think it'll work with pypy, *and* still trigger the warning, but that's from memory. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110206/8260e103/attachment.htm From arigo at tunes.org Sun Feb 6 12:57:06 2011 From: arigo at tunes.org (Armin Rigo) Date: Sun, 6 Feb 2011 12:57:06 +0100 Subject: [pypy-dev] Compat. in 1.4.1 __del__ In-Reply-To: References: <4D4DCA8F.7070508@stackless.com> <6C7ABA8B4E309440B857D74348836F2E0148B204@TK5EX14MBXC141.redmond.corp.microsoft.com> Message-ID: Hi Alex, On Sun, Feb 6, 2011 at 6:26 AM, Alex Gaynor wrote: >> Then you can change __del_ to whatever you want later. > > I think it'll work with pypy, *and* still trigger the warning, but that's > from memory. Just try it out then :-) The answer is no. You get a warning only if a '__del__' entry is put in a class which didn't have one so far. Armin From hakan at debian.org Sun Feb 6 17:15:16 2011 From: hakan at debian.org (Hakan Ardo) Date: Sun, 6 Feb 2011 17:15:16 +0100 Subject: [pypy-dev] Virtuals discoverd by bridge Message-ID: Hi, at the sprint we talked about how to better handle boxes that become virtual in a bridge. In the jit-virtual_state branch there is now a first step towards that. Try: py.test test/test_virtual.py -k test_dual_counter --viewloops in pypy/jit/metainterp. The idea is compare the state of all the OptValues of the jump_args at the end of the bridge white all the OptValues of the jump_args at the end of the preamble. We will need this check anyway to be sure that it is safe to call the loop (just being able to inline the short preamble turned out not to be enough). When the states differ we have 3 options: - Force some of the virtuals to make the states match - Emitt guards to make the states match - Retrace the loop to form a new specialized version We will need some heuristic to choose between these. Currently I emit GUARD_CLASS if the boxes was of correct class during the current trace (box.value). In all other cases I retrace (which is probably too often). For the test above there are 2 virtuals, node1 and node2. There will be 3 specialized version of the loop generated: - node1 virtual, node2 allocated - node2 virtual, node1 allocated - both node1 and node2 virtuals The first 2 of these are currently generated. The last is not. For the last to be generated the retraced loop will need to inherit the virtuals from the bridge that causes it to be retraced (in the same manner as the bridges inherit the virtuals of their parent loops). -- H?kan Ard? From tismer at stackless.com Tue Feb 8 23:00:53 2011 From: tismer at stackless.com (Christian Tismer) Date: Tue, 08 Feb 2011 23:00:53 +0100 Subject: [pypy-dev] Compat. in 1.4.1 __del__ In-Reply-To: <4F9D7C9F-D3CF-445E-9594-4ABD8B69E92C@underboss.org> References: <4D4DCA8F.7070508@stackless.com> <6C7ABA8B4E309440B857D74348836F2E0148B204@TK5EX14MBXC141.redmond.corp.microsoft.com> <4F9D7C9F-D3CF-445E-9594-4ABD8B69E92C@underboss.org> Message-ID: <4D51BD15.5010607@stackless.com> On 2/8/11 9:25 PM, Philip Jenvey wrote: > On Feb 5, 2011, at 9:04 PM, Dino Viehland wrote: > >> Christianwrote: >>> Howdy, >>> >>> studying the differences of PyPy vs. CPython, most seem to be fine; one >>> thing where I an unsure is the __del__ behavior. >>> >>> I am not addressing its delayed call or the number it is called, this is similar to >>> Jython and IronPython. >>> >>> But assigning to __del__ after a class is created, is that so hard to implement? >> IronPython also doesn't handle assigning to __del__ after the class is created, and >> I'd be surprised if Jython did as well. To make this work we'd need to maintain a >> weak reference for every object of a user defined type and I think most users >> would rather not pay that expense for such a corner case. I've also never actually >> heard of this breaking compatibility anywhere. > Jython doesn't either. We've been meaning to add the same warning PyPy does when you create a __del__ after the fact. > > The feature is simply not worth the effort. > > -- > Philip Jenvey Fine with me! I just wanted to know the reasons why this is more problematic than I thought. Thanks to all who replied. cheers - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From fijall at gmail.com Wed Feb 9 07:57:32 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 9 Feb 2011 08:57:32 +0200 Subject: [pypy-dev] [pypy-svn] pypy jit-virtual_state: never retrace a loop more than 5 times In-Reply-To: <20110208184238.EDF99282BF6@codespeak.net> References: <20110208184238.EDF99282BF6@codespeak.net> Message-ID: On Tue, Feb 8, 2011 at 8:42 PM, hakanardo wrote: > Author: Hakan Ardo > Branch: jit-virtual_state > Changeset: r41714:91586ef4b9c5 > Date: 2011-02-08 19:42 +0100 > http://bitbucket.org/pypy/pypy/changeset/91586ef4b9c5/ > > Log: ? ?never retrace a loop more than 5 times Is this always a good idea? Maybe we should have a configurable parameter for that (like threshold). > > diff --git a/pypy/jit/metainterp/optimizeopt/unroll.py b/pypy/jit/metainterp/optimizeopt/unroll.py > --- a/pypy/jit/metainterp/optimizeopt/unroll.py > +++ b/pypy/jit/metainterp/optimizeopt/unroll.py > @@ -671,7 +671,10 @@ > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? "jumping to preamble instead") > ? ? ? ? ? ? ? ? ? ? ? ? ? ? self.emit_operation(op) > ? ? ? ? ? ? ? ? ? ? ? ? return > - ? ? ? ? ? ? ? ?if not self.retraced: > + ? ? ? ? ? ? ? ?retraced_count = len(short) > + ? ? ? ? ? ? ? ?if descr.failed_states: > + ? ? ? ? ? ? ? ? ? ?retraced_count += len(descr.failed_states) > + ? ? ? ? ? ? ? ?if not self.retraced and retraced_count<5: > ? ? ? ? ? ? ? ? ? ? if not descr.failed_states: > ? ? ? ? ? ? ? ? ? ? ? ? raise RetraceLoop > ? ? ? ? ? ? ? ? ? ? for failed in descr.failed_states: > _______________________________________________ > pypy-svn mailing list > pypy-svn at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-svn > From arigo at tunes.org Thu Feb 10 12:23:51 2011 From: arigo at tunes.org (Armin Rigo) Date: Thu, 10 Feb 2011 12:23:51 +0100 Subject: [pypy-dev] Fwd: [codespeak-ann] termination of codespeak.net services end FEB 2011 In-Reply-To: <20110210084509.GE14054@trillke.net> References: <20110210084509.GE14054@trillke.net> Message-ID: Hi all, It seems that Codespeak is going down sooner rather than later. Here is a copy of the start of holger's announce: ---------- Forwarded message ---------- From: holger krekel Date: Thu, Feb 10, 2011 at 9:45 AM Subject: [codespeak-ann] termination of codespeak.net services end FEB 2011 To: codespeak-ann at codespeak.net hi codespeak.net users, (sorry if you get mail twice, i wanted to make sure ...) after 8 years of operation codespeak.net services are bound to terminate, starting ? ?END OF FEBRUARY 2011 ------------------------------------------------ As far as PyPy is concerned, we need to move somewhere else: * the mailing lists pypy-dev, pypy-svn, pypy-z (also the history, please :-) * the remainings parts that are in Subversion: at least svn/pypy/extradoc, svn/pypy/benchmarks, svn/pypy/lang, which should all be converted to Mercurial I suppose (help needed here) * the developers web site (from pypy/doc/) * the users web site (http://pypy.org/) * the buildmaster * the issue tracker * the svn/user/xxx directory for people like me that use it a lot * something else? It's not like we weren't warned plenty of times. It still looks like a lot of work. The timing is a bit bad, because I wanted to finish fixing the 2.7 test failures in time to do a PyPy 1.5 release for PyCon, but it looks a bit more unlikely right now :-( (Not too important, we can do a pre-release any time, and should do one soon anyway.) A bient?t, Armin. From lac at openend.se Thu Feb 10 12:40:38 2011 From: lac at openend.se (Laura Creighton) Date: Thu, 10 Feb 2011 12:40:38 +0100 Subject: [pypy-dev] Fwd: [codespeak-ann] termination of codespeak.net services end FEB 2011 In-Reply-To: Message from Armin Rigo of "Thu, 10 Feb 2011 12:23:51 +0100." References: <20110210084509.GE14054@trillke.net> Message-ID: <201102101140.p1ABec9U026243@theraft.openend.se> We can move things to Open End if people are interested. But there is a lot to be said for moving to a place where the whole hosting matter is somebody else's problem. Right now we seem awfully dependent on home-grown tools. Are there any that people cannot live without? Note: I am playing with Sphinx and liking it. It takes ReST as input. Georg Brandl is about to come out with a release that lets readers post notes on documentation pages, which I think would be a very neat feature to have. So'd I'd sort of like to be able to run sphinx wherever we go, and, at some point convert all of our documentation to use sphinx. Laura From stefan_ml at behnel.de Thu Feb 10 13:22:34 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 10 Feb 2011 13:22:34 +0100 Subject: [pypy-dev] Fwd: [codespeak-ann] termination of codespeak.net services end FEB 2011 In-Reply-To: <201102101140.p1ABec9U026243@theraft.openend.se> References: <20110210084509.GE14054@trillke.net> <201102101140.p1ABec9U026243@theraft.openend.se> Message-ID: Laura Creighton, 10.02.2011 12:40: > We can move things to Open End if people are interested. Hmm, a web site that's full of "Alternate HTML content should be placed here." boxes doesn't sound very inviting to me as a hosting solution. Stefan From arigo at tunes.org Thu Feb 10 13:38:23 2011 From: arigo at tunes.org (Armin Rigo) Date: Thu, 10 Feb 2011 13:38:23 +0100 Subject: [pypy-dev] Fwd: [codespeak-ann] termination of codespeak.net services end FEB 2011 In-Reply-To: <201102101140.p1ABec9U026243@theraft.openend.se> References: <20110210084509.GE14054@trillke.net> <201102101140.p1ABec9U026243@theraft.openend.se> Message-ID: Hi Laura, On Thu, Feb 10, 2011 at 12:40 PM, Laura Creighton wrote: > We can move things to Open End if people are interested. ?But there is a > lot to be said for moving to a place where the whole hosting matter is > somebody else's problem. I agree, but I still think that we will take your offer for *some* of the services I described. At least the buildmaster should be moved to Open End (I doubt there is an existing provider that supports customized buildbot masters like we need). Armin From hakan at debian.org Thu Feb 10 17:12:20 2011 From: hakan at debian.org (Hakan Ardo) Date: Thu, 10 Feb 2011 17:12:20 +0100 Subject: [pypy-dev] Resume Message-ID: Hi, the jit-virtual_state branch is getting ready to be merged. It fixes the issue in test_new_virtual_member_in_bridge. However I've added a fieldstate member to the VirtualInfo classes, which might have some memory usage impacts? Would it be possible/a good idea to integrate the new VirtualStateAdder in unroll.py more closely with the resume stuff to maybe get rid of this member to use less memory? Also, at some (later) point I would like to experiment with including NotVirtualInfo objects in the resumedata to allow a bridge to inherit information about non-virtuals from it's parent loop. -- H?kan Ard? From holger at merlinux.eu Thu Feb 10 18:01:30 2011 From: holger at merlinux.eu (holger krekel) Date: Thu, 10 Feb 2011 18:01:30 +0100 Subject: [pypy-dev] Fwd: [codespeak-ann] termination of codespeak.net services end FEB 2011 In-Reply-To: References: <20110210084509.GE14054@trillke.net> Message-ID: <20110210170130.GF14054@trillke.net> Hi Armin, for pypy things are and will remain special ... sorry for not having that this morning when i sent the general announce. I was a bit in a hurry. Anyway, there is no need to fear that pypy services suddenly vanish end february. Let's discuss options sometime on IRC. I also have some suggestions ... cheers, holger > Hi all, > > It seems that Codespeak is going down sooner rather than later. Here > is a copy of the start of holger's announce: > > ---------- Forwarded message ---------- > From: holger krekel > Date: Thu, Feb 10, 2011 at 9:45 AM > Subject: [codespeak-ann] termination of codespeak.net services end FEB 2011 > To: codespeak-ann at codespeak.net > > > hi codespeak.net users, (sorry if you get mail twice, i wanted to make sure ...) > > after 8 years of operation codespeak.net services are bound to > terminate, starting > > ? ?END OF FEBRUARY 2011 > > ------------------------------------------------ > > As far as PyPy is concerned, we need to move somewhere else: > > * the mailing lists pypy-dev, pypy-svn, pypy-z (also the history, please :-) > > * the remainings parts that are in Subversion: at least > svn/pypy/extradoc, svn/pypy/benchmarks, svn/pypy/lang, which should > all be converted to Mercurial I suppose (help needed here) > > * the developers web site (from pypy/doc/) > > * the users web site (http://pypy.org/) > > * the buildmaster > > * the issue tracker > > * the svn/user/xxx directory for people like me that use it a lot > > * something else? > > It's not like we weren't warned plenty of times. It still looks like > a lot of work. The timing is a bit bad, because I wanted to finish > fixing the 2.7 test failures in time to do a PyPy 1.5 release for > PyCon, but it looks a bit more unlikely right now :-( (Not too > important, we can do a pre-release any time, and should do one soon > anyway.) > > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -- From dcolish at gmail.com Sun Feb 13 02:53:16 2011 From: dcolish at gmail.com (Dan Colish) Date: Sat, 12 Feb 2011 17:53:16 -0800 Subject: [pypy-dev] PSF Sponsored Sprint in Portland, OR Message-ID: Hello Developers! I'm one of the organizers of the Portland PSF Sprint. We've been working hard on our proposal to come up with a strong program for a full day of sprinting in Portland. The day will mostly focus on porting libraries to Python 3, hacking on PyPY 2.7 compatibility and testing. It'll be a great time so if you're in the area stop on by! The current project list includes:- Fabric - Django - Flatland - PyPy - Alfajor Want to suggest your own? Use the signup sheet here. Breakfast is being sponsored by Idealist.org, lunch is being sponsored by Emma and beverages will be made available by Urban Airship. More information: http://goo.gl/XTMWv. Signup sheet: http://goo.gl/a5Gqk. -- Dan From vsapre80 at gmail.com Sun Feb 13 18:55:43 2011 From: vsapre80 at gmail.com (Vishal) Date: Sun, 13 Feb 2011 23:25:43 +0530 Subject: [pypy-dev] Hello...and a small question Message-ID: Hello, This is my first email to the PyPy Dev mailing list. I would like to ask if there is a place where a potential user of PyPy would know, what C extensions support PyPy? Could this be a way of helping PyPy? -- Thanks and best regards, Vishal Sapre --- "So say...Day by day, in every way, I am getting better, better and better !!!" "A Strong and Positive attitude creates more miracles than anything else. Because...Life is 10% how you make it, and 90% how you take it" "Diamond is another piece of coal that did well under pressure? "May we do good and not evil. May we find forgiveness for ourself and forgive others. May we share freely, never taking more than we give." -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110213/cdf3c843/attachment-0001.htm From fuzzyman at gmail.com Mon Feb 14 01:42:46 2011 From: fuzzyman at gmail.com (Michael Foord) Date: Mon, 14 Feb 2011 00:42:46 +0000 Subject: [pypy-dev] Winter Sprint report: first draft Message-ID: Hey guys, I finally found time to write a first draft of a blog entry reporting on the winter sprint. Comments, corrections, insults and suggestions solicited: PyPy Winter Sprint A few weeks ago I had the great fortune to attend the winter sprint in Leysin Switzerland. I've wanted to contribute to PyPy for a long time and I thought diving into a sprint might be a good way to get familiar with some of the code. What I wasn't expecting was to be implementing new built-in methods in RPython on the first day. The main thing I took away from the sprint was just how easy it is to get involved in developing PyPy (well, some bits of it at least and being surrounded by core developers helps). I wrote up a very short description of how to get started here ( https://bitbucket.org/pypy/pypy/wiki/How%20to%20run%20lib-python%20tests ), but I'll do a longer blog post with examples on my own blog ( http://www.voidspace.org.uk/python/weblog/ ) soon(ish). The sprint was kicked off by Armin merging the "fast-forward" branch of PyPy onto trunk. "fast-forward" brings PyPy from Python 2.5 compatibility to Python 2.7. Along with this it brought a large number of test failures, as the sterling work done by Benjamin Peterson and Amaury Forgeot d'Arc was not complete. This immediately set the primary sprint goal to reduce the number of test failures. We made a great deal of progress on this front, and you can see how close PyPy is now from the buildbot: http://codespeak.net:8099/summary?branch=%3Ctrunk%3E Jacob Hallen and I started working through the list alphabetically. We made short work of test_asyncore and moved onto test_bytes where I was stuck for the rest of the sprint. I spent much of the remaining days working with Laura Creighton on the pypy bytearray implementation to make it more compatible with the Python 2.7 implementation. This meant adding new methods, changing some of the Python protocol method implementations and even changing the way that bytearray is constructed. All in all great fun and a great introduction to working with RPython. If you add on top of this the great people, the beautiful scenery, the Swiss cheese fondues, managing to not kill myself with a days skiing and traditional pypy card games, I can heartily recommend pypy sprints as a close approximation of geek nirvana. Working on 2.7 compatibility wasn't the only work that happened during the sprint. Other activities included: * Antonio Cuni worked on the "jittypes" branch. This is a reimplementation of the core of the PyPy ctypes code to make it jittable. The goal is that for common cases the jit should be able to turn ctypes calls into direct jumps. This work was not completed but very close and is great for the future of integrating C libraries with PyPy. As ctypes is also available in CPython and IronPython, and hopefully will be available in Jython soon, integrating C code with Python through ctypes is the most "implementation portable" technique. * David Schneider continued his work on the JIT backend for ARM. PyPy has been cross-compilable to ARM for a long time, but bringing the JIT to ARM will provide a *fast* PyPy for ARM, which includes platforms like Android. Again David didn't complete this work but did complete the float support. * Hakan Ardo was present for two days and continued his crazy-clever work on JIT optimisations, some of which are described in the "Loop invariant code motion" ( http://morepypy.blogspot.com/2011/01/loop-invariant-code-motion.html ) blog entry. * Holger Krekel worked on updating the PyPy test suite to the latest version of py.test and also worked with me on the interminable bytearray changes for part of the sprint. * No one was sure what Maciej Fijalkowsk worked on but he seemed to be quite busy. I think that was most of the work done during the actual sprint. There was also a great deal of healthy discussion about the future of PyPy. Expect lots more interesting and exciting developments over the coming year. Michael Foord -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110214/c71cdbde/attachment.htm From alex.gaynor at gmail.com Mon Feb 14 06:04:52 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Mon, 14 Feb 2011 00:04:52 -0500 Subject: [pypy-dev] Hello...and a small question In-Reply-To: References: Message-ID: On Sun, Feb 13, 2011 at 12:55 PM, Vishal wrote: > Hello, > > This is my first email to the PyPy Dev mailing list. > > I would like to ask if there is a place where a potential user of PyPy > would know, what C extensions support PyPy? > > Could this be a way of helping PyPy? > > -- > Thanks and best regards, > Vishal Sapre > > --- > "So say...Day by day, in every way, I am getting better, better and better > !!!" > "A Strong and Positive attitude creates more miracles than anything else. > Because...Life is 10% how you make it, and 90% how you take it" > "Diamond is another piece of coal that did well under pressure? > "May we do good and not evil. May we find forgiveness for ourself and > forgive others. May we share freely, never taking more than we give." > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > Someone started a wikipage on this subject: https://bitbucket.org/pypy/compatibility/wiki/Home, help filling it out is always welcome. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110214/d5140d88/attachment.htm From fijall at gmail.com Mon Feb 14 06:11:44 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 13 Feb 2011 22:11:44 -0700 Subject: [pypy-dev] Winter Sprint report: first draft In-Reply-To: References: Message-ID: Hey. Few comments: * jittypes - it's technically called C-level calls and not jumps. So ctypes calls become direct calls on the C level. Jumps are without calling mechanism. * My name is missing an "i" at the end. Also unicode friendly name is Fija?kowski (although it's fine to keep it that way to remain british and keep cultural heritage of horrible misspelling). * I was working on investigating why twisted_tcp got slower and shaving yaks in between, but what you said pretty much describes what I did, even in a better detail. On Sun, Feb 13, 2011 at 5:42 PM, Michael Foord wrote: > Hey guys, > I finally found time to write a first draft of a blog entry reporting on the > winter sprint. Comments, corrections, insults and suggestions solicited: > > PyPy Winter Sprint > A few weeks ago I had the great fortune to attend the winter sprint in > Leysin Switzerland. I've wanted to contribute to PyPy for a long time and I > thought diving into a sprint might be a good way to get familiar with some > of the code. What I wasn't expecting was to be implementing new built-in > methods in RPython on the first day. The main thing I took away from the > sprint was just how easy it is to get involved in developing PyPy (well, > some bits of it at least and being surrounded by core developers helps). I > wrote up a very short description of how to get started here ( > https://bitbucket.org/pypy/pypy/wiki/How%20to%20run%20lib-python%20tests ), > but I'll do a longer blog post with examples on my own blog ( > http://www.voidspace.org.uk/python/weblog/ ) soon(ish). > The sprint was kicked off by Armin merging the "fast-forward" branch of PyPy > onto trunk. "fast-forward" brings PyPy from Python 2.5 compatibility to > Python 2.7. Along with this it brought a large number of test failures, as > the sterling work done by Benjamin Peterson and Amaury Forgeot d'Arc was not > complete. This immediately set the primary sprint goal to reduce the number > of test failures. > We made a great deal of progress on this front, and you can see how close > PyPy is now from the buildbot: > http://codespeak.net:8099/summary?branch=%3Ctrunk%3E > Jacob Hallen and I started working through the list alphabetically. We made > short work of test_asyncore and moved onto test_bytes where I was stuck for > the rest of the sprint. I spent much of the remaining days working with > Laura Creighton on the pypy bytearray implementation to make it more > compatible with the Python 2.7 implementation. This meant adding new > methods, changing some of the Python protocol method implementations and > even changing the way that bytearray is constructed. All in all great fun > and a great introduction to working with RPython. > If you add on top of this the great people, the beautiful scenery, the Swiss > cheese fondues, managing to not kill myself with a days skiing and > traditional pypy card games, I can heartily recommend pypy sprints as a > close approximation of geek nirvana. > Working on 2.7 compatibility wasn't the only work that happened during the > sprint. Other activities included: > * Antonio Cuni worked on the "jittypes" branch. This is a reimplementation > of the core of the PyPy ctypes code to make it jittable. The goal is that > for common cases the jit should be able to turn ctypes calls into direct > jumps. This work was not completed but very close and is great for the > future of integrating C libraries with PyPy. As ctypes is also available in > CPython and IronPython, and hopefully will be available in Jython soon, > integrating C code with Python through ctypes is the most "implementation > portable" technique. > * David Schneider continued his work on the JIT backend for ARM. PyPy has > been cross-compilable to ARM for a long time, but bringing the JIT to ARM > will provide a *fast* PyPy for ARM, which includes platforms like Android. > Again David didn't complete this work but did complete the float support. > * Hakan Ardo was present for two days and continued his crazy-clever work on > JIT optimisations, some of which are described in the "Loop invariant code > motion" ( > http://morepypy.blogspot.com/2011/01/loop-invariant-code-motion.html ) blog > entry. > * Holger Krekel worked on updating the PyPy test suite to the latest version > of py.test and also worked with me on the interminable bytearray changes for > part of the sprint. > * No one was sure what ?Maciej Fijalkowsk worked on but he seemed to be > quite busy. > I think that was most of the work done during the actual sprint. There was > also a great deal of healthy discussion about the future of PyPy. Expect > lots more interesting and exciting developments over the coming year. > Michael Foord > -- > > http://www.voidspace.org.uk/ > > May you do good and not evil > May you find forgiveness for yourself and forgive others > > May you share freely, never taking more than you give. > -- the sqlite blessing http://www.sqlite.org/different.html > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From dimaqq at gmail.com Mon Feb 14 06:20:30 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Sun, 13 Feb 2011 22:20:30 -0700 Subject: [pypy-dev] Winter Sprint report: first draft In-Reply-To: References: Message-ID: On 13 February 2011 22:11, Maciej Fijalkowski wrote: > * My name is missing an "i" at the end. Also unicode friendly name is > Fija?kowski (although it's fine to keep it that way to remain british > and keep cultural heritage of horrible misspelling). In the interest of the Commonwealth you are from here on Matthew Violet ;-) d. From arigo at tunes.org Mon Feb 14 11:36:22 2011 From: arigo at tunes.org (Armin Rigo) Date: Mon, 14 Feb 2011 11:36:22 +0100 Subject: [pypy-dev] Winter Sprint report: first draft In-Reply-To: References: Message-ID: Hi Michael, Thanks for the upcoming blog entry! :-) On Mon, Feb 14, 2011 at 1:42 AM, Michael Foord wrote: > http://codespeak.net:8099/summary?branch=%3Ctrunk%3E It's best to point to the future-proofer url: http://buildbot.pypy.org/summary?branch=%3Ctrunk%3E . Armin From fuzzyman at gmail.com Mon Feb 14 11:40:12 2011 From: fuzzyman at gmail.com (Michael Foord) Date: Mon, 14 Feb 2011 10:40:12 +0000 Subject: [pypy-dev] Winter Sprint report: first draft In-Reply-To: References: Message-ID: Thanks for the feedback and corrections folks. Either someone else will have to post the slightly edited version, or someone will need to give me permissions to add blog entries. (For blogger *this* emil address is my blogger id I'm pretty sure.) Either way, maybe worth waiting a little bit in case any of the other folk from the sprint want to correct anything else. All the best, Michael On 14 February 2011 10:36, Armin Rigo wrote: > Hi Michael, > > Thanks for the upcoming blog entry! :-) > > On Mon, Feb 14, 2011 at 1:42 AM, Michael Foord wrote: > > http://codespeak.net:8099/summary?branch=%3Ctrunk%3E > > It's best to point to the future-proofer url: > http://buildbot.pypy.org/summary?branch=%3Ctrunk%3E . > > > Armin > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110214/a4f4034f/attachment-0001.htm From fuzzyman at gmail.com Mon Feb 14 13:06:30 2011 From: fuzzyman at gmail.com (Michael Foord) Date: Mon, 14 Feb 2011 12:06:30 +0000 Subject: [pypy-dev] Winter Sprint report: first draft In-Reply-To: References: Message-ID: It's now live, but I can still edit it and correct mistakes of course: http://morepypy.blogspot.com/2011/02/pypy-winter-sprint-report.html Thanks for a great sprint everyone! Michael On 14 February 2011 10:40, Michael Foord wrote: > Thanks for the feedback and corrections folks. Either someone else will > have to post the slightly edited version, or someone will need to give me > permissions to add blog entries. (For blogger *this* emil address is my > blogger id I'm pretty sure.) > > Either way, maybe worth waiting a little bit in case any of the other folk > from the sprint want to correct anything else. > > All the best, > > Michael > > > On 14 February 2011 10:36, Armin Rigo wrote: > >> Hi Michael, >> >> Thanks for the upcoming blog entry! :-) >> >> On Mon, Feb 14, 2011 at 1:42 AM, Michael Foord >> wrote: >> > http://codespeak.net:8099/summary?branch=%3Ctrunk%3E >> >> It's best to point to the future-proofer url: >> http://buildbot.pypy.org/summary?branch=%3Ctrunk%3E . >> >> >> Armin >> > > > > -- > > http://www.voidspace.org.uk/ > > May you do good and not evil > May you find forgiveness for yourself and forgive others > > May you share freely, never taking more than you give. > -- the sqlite blessing http://www.sqlite.org/different.html > > > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110214/ae63ad97/attachment.htm From victorgarcianet at gmail.com Mon Feb 14 23:41:09 2011 From: victorgarcianet at gmail.com (Victor Garcia) Date: Mon, 14 Feb 2011 20:41:09 -0200 Subject: [pypy-dev] PyPy Compatibility Wiki Message-ID: Hi Vishal (and list), Thanks for wanting to help PyPy :) I'm working on the compatibility wiki and I like your idea of having a page like http://getpython3.net/ to summarize it and make it prettier. We could have it as a totally separated site (hosting needed, any implementation we want), a separated site hosted on bitbucket (like getpypy.bitbucket.org, static HTML + JS needed, any design we want) or as the front page of the wiki. The whole design of the wiki is open to changes. The information from the wiki is available in YAML format (there's a script to convert YAML<->creole and do other checks), so it would be very easy to generate a different format for a site like that. Right now I'd like to invest what time I have to improve coverage of compatibility-tested packages and implement some items listed in https://bitbucket.org/pypy/compatibility/wiki/Discussion#!to-do, but I'd sure help with building a site base on wiki info if people agree it's a good idea. Since we haven't discussed the wiki much at all, I'd also like to invite everyone to chime in about what they'd like to see done in the wiki. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110214/99fc90c7/attachment.htm From fijall at gmail.com Tue Feb 15 02:13:54 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 14 Feb 2011 18:13:54 -0700 Subject: [pypy-dev] Separate compilation and friends Message-ID: Hi. There is growing interest about PyPy and especially about extension modules. Apparently there are some people (like Alex) that are willing to write modules in RPython that should not go to the main tree. Since separate compilation is considered hard, how hard would it be to provide separate loading? This would mean you still compile the whole interpreter, but you can load the module from a compiled PyPy that didn't have this option on. Cheers, fijal From dimaqq at gmail.com Tue Feb 15 03:41:11 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Mon, 14 Feb 2011 19:41:11 -0700 Subject: [pypy-dev] Separate compilation and friends In-Reply-To: References: Message-ID: On a related note, how hard is it to "freeze" the translator/compiler state of a given pypy version just before it begins to read extension modules and distribute that, it would speed up module development a lot. It would be a quick equivalent of distributing headers for a C library. d. On 14 February 2011 18:13, Maciej Fijalkowski wrote: > Hi. > > There is growing interest about PyPy and especially about extension > modules. Apparently there are some people (like Alex) that are willing > to write modules in RPython that should not go to the main tree. Since > separate compilation is considered hard, how hard would it be to > provide separate loading? This would mean you still compile the whole > interpreter, but you can load the module from a compiled PyPy that > didn't have this option on. > > Cheers, > fijal > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From william.leslie.ttg at gmail.com Tue Feb 15 04:34:44 2011 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Tue, 15 Feb 2011 14:34:44 +1100 Subject: [pypy-dev] Separate compilation and friends In-Reply-To: References: Message-ID: On 15 February 2011 12:13, Maciej Fijalkowski wrote: > Hi. > > There is growing interest about PyPy and especially about extension > modules. Apparently there are some people (like Alex) that are willing > to write modules in RPython that should not go to the main tree. Since > separate compilation is considered hard, how hard would it be to > provide separate loading? This would mean you still compile the whole > interpreter, but you can load the module from a compiled PyPy that > didn't have this option on. By "this option" you mean --withmod-xxx ? -- William Leslie From anto.cuni at gmail.com Tue Feb 15 09:10:08 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 15 Feb 2011 09:10:08 +0100 Subject: [pypy-dev] Separate compilation and friends In-Reply-To: References: Message-ID: <4D5A34E0.3060602@gmail.com> On 15/02/11 03:41, Dima Tisnek wrote: > On a related note, how hard is it to "freeze" the translator/compiler > state of a given pypy version just before it begins to read extension > modules and distribute that, it would speed up module development a > lot. > It would be a quick equivalent of distributing headers for a C library. this is hard, because the compilation of the modules is intermixed with the compilation of the rest of the interpreter for each phase: we have (roughly) something like: - annotation of the interpreter - annotation of the modules - rtyping of the interpreter - rtyping of the modules - etc. etc. ciao, Anto From dimaqq at gmail.com Tue Feb 15 09:17:58 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Tue, 15 Feb 2011 01:17:58 -0700 Subject: [pypy-dev] Separate compilation and friends In-Reply-To: <4D5A34E0.3060602@gmail.com> References: <4D5A34E0.3060602@gmail.com> Message-ID: On 15 February 2011 01:10, Antonio Cuni wrote: > On 15/02/11 03:41, Dima Tisnek wrote: >> On a related note, how hard is it to "freeze" the translator/compiler >> state of a given pypy version just before it begins to read extension >> modules and distribute that, it would speed up module development a >> lot. >> It would be a quick equivalent of distributing headers for a C library. > > this is hard, because the compilation of the modules is intermixed with the > compilation of the rest of the interpreter for each phase: we have (roughly) > something like: > > - annotation of the interpreter > - annotation of the modules > - rtyping of the interpreter > - rtyping of the modules > - etc. etc. > > ciao, > Anto > Yeah I figured as much, I was wondering if it could be changed like this: - annotation of the interpreter, save state (1) - rtyping of the interpreter, shouldn't depend on modules here, save state (2) - etc. - annotation of the modules, using state from 1 - rtyping of the modules, using state from 1,2 - etc. I assume here that modules don't introduce dependencies into the iterpreter. I guess in the long run this ought to be the case, right? If this is possible, it would be a useful quick hack to separate module build from main build. If it's still very hard, then some else is in order. I'd love to play with this myself, but I don't have enough ram for a full build ;-( d. From pgiarrusso at mathematik.uni-marburg.de Tue Feb 15 09:21:05 2011 From: pgiarrusso at mathematik.uni-marburg.de (Paolo Giarrusso) Date: Tue, 15 Feb 2011 09:21:05 +0100 Subject: [pypy-dev] Separate compilation and friends In-Reply-To: <4D5A34E0.3060602@gmail.com> References: <4D5A34E0.3060602@gmail.com> Message-ID: On Tue, Feb 15, 2011 at 09:10, Antonio Cuni wrote: > On 15/02/11 03:41, Dima Tisnek wrote: >> On a related note, how hard is it to "freeze" the translator/compiler >> state of a given pypy version just before it begins to read extension >> modules and distribute that, it would speed up module development a >> lot. >> It would be a quick equivalent of distributing headers for a C library. > > this is hard, because the compilation of the modules is intermixed with the > compilation of the rest of the interpreter for each phase: we have (roughly) > something like: > > - annotation of the interpreter > - annotation of the modules > - rtyping of the interpreter > - rtyping of the modules > - etc. etc. Thanks for explaining the problem. IIRC, rtyping is a global analysis because for performance you perform type inference globally, so that for instance if a module allows inferring that a field is a number, that information is propagated to the original definition. Depending on what Maciej meant, his proposal might be as hard: > [Separate loading] This would mean you still compile the whole interpreter, but you can load the module from a compiled PyPy that didn't have this option on. At least as far as I understand, if "interpreter" means just the interpreter "core", as in Antonio's mail, this seems just as hard. Otherwise, if "interpreter" includes modules, this extension would not accomodate the needs of authors of external modules: one would have download all external modules and compile them together, but at runtime one could save the RAM footprint of external modules. Is that the relevant problem? Cheers, -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From pgiarrusso at mathematik.uni-marburg.de Tue Feb 15 09:32:50 2011 From: pgiarrusso at mathematik.uni-marburg.de (Paolo Giarrusso) Date: Tue, 15 Feb 2011 09:32:50 +0100 Subject: [pypy-dev] Separate compilation and friends In-Reply-To: References: <4D5A34E0.3060602@gmail.com> Message-ID: On Tue, Feb 15, 2011 at 09:17, Dima Tisnek wrote: > On 15 February 2011 01:10, Antonio Cuni wrote: >> On 15/02/11 03:41, Dima Tisnek wrote: >>> On a related note, how hard is it to "freeze" the translator/compiler >>> state of a given pypy version just before it begins to read extension >>> modules and distribute that, it would speed up module development a >>> lot. >>> It would be a quick equivalent of distributing headers for a C library. >> >> this is hard, because the compilation of the modules is intermixed with the >> compilation of the rest of the interpreter for each phase: we have (roughly) >> something like: >> >> - annotation of the interpreter >> - annotation of the modules How much time would be saved by saving the annotation results? >> - rtyping of the interpreter >> - rtyping of the modules >> - etc. etc. >> >> ciao, >> Anto >> > > Yeah I figured as much, I was wondering if it could be changed like this: > > - annotation of the interpreter, save state (1) > - rtyping of the interpreter, shouldn't depend on modules here, save state (2) > - etc. > - annotation of the modules, using state from 1 > - rtyping of the modules, using state from 1,2 > - etc. > > I assume here that modules don't introduce dependencies into the iterpreter. > I guess in the long run this ought to be the case, right? I don't think you can guarantee this. Type inference is global, and you might need a user for each API to better infer its type. Maybe uses of an API in testcases allow fully inferring their types, but I'd guess not. However, what is true in general is that if less specific types are inferred, that affects just performance, not correctness (I don't know if that's true of PyPy, but you ought to be able to pass "object"s around). Maybe the slowdown is insignificant, maybe it is a huge problem, maybe few annotations can save the day. However, it is still not clear (to me) where previous efforts stopped. Is it hard to: 1) devise an algorithm like Dima proposed or to 2) implement it (because of too much code to change and limited manpower) or to 3) or to have a small performance loss? Per-file separate compilation would likely fall into 3), because too little type inference would happen, isn't it? > If this is possible, it would be a useful quick hack to separate > module build from main build. > If it's still very hard, then some else is in order. > I'd love to play with this myself, but I don't have enough ram for a > full build ;-( -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From dimaqq at gmail.com Tue Feb 15 10:01:21 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Tue, 15 Feb 2011 02:01:21 -0700 Subject: [pypy-dev] Separate compilation and friends In-Reply-To: References: <4D5A34E0.3060602@gmail.com> Message-ID: On 15 February 2011 01:32, Paolo Giarrusso wrote: > On Tue, Feb 15, 2011 at 09:17, Dima Tisnek wrote: >> On 15 February 2011 01:10, Antonio Cuni wrote: >>> On 15/02/11 03:41, Dima Tisnek wrote: >>>> On a related note, how hard is it to "freeze" the translator/compiler >>>> state of a given pypy version just before it begins to read extension >>>> modules and distribute that, it would speed up module development a >>>> lot. >>>> It would be a quick equivalent of distributing headers for a C library. >>> >>> this is hard, because the compilation of the modules is intermixed with the >>> compilation of the rest of the interpreter for each phase: we have (roughly) >>> something like: >>> >>> - annotation of the interpreter >>> - annotation of the modules > > How much time would be saved by saving the annotation results? > >>> - rtyping of the interpreter >>> - rtyping of the modules >>> - etc. etc. >>> >>> ciao, >>> Anto >>> >> >> Yeah I figured as much, I was wondering if it could be changed like this: >> >> - annotation of the interpreter, save state (1) >> - rtyping of the interpreter, shouldn't depend on modules here, save state (2) >> - etc. >> - annotation of the modules, using state from 1 >> - rtyping of the modules, using state from 1,2 >> - etc. >> >> I assume here that modules don't introduce dependencies into the iterpreter. >> I guess in the long run this ought to be the case, right? > > I don't think you can guarantee this. Type inference is global, and > you might need a user for each API to better infer its type. Maybe > uses of an API in testcases allow fully inferring their types, but I'd > guess not. > > However, what is true in general is that if less specific types are > inferred, that affects just performance, not correctness (I don't know > if that's true of PyPy, but you ought to be able to pass "object"s > around). Maybe the slowdown is insignificant, maybe it is a huge > problem, maybe few annotations can save the day. Correct me if I'm wong, but I assume that rpython and python type inferences are quite different. After all we don't have user code when we translate pypy itself, do we? Coming back to rpython-only discussion: I suppose one option woud be include type speciation in the modules themselves, or is that an overkill? Say if I write an ubermodule with uberdatum class in rpython and then use standard library bisect in that module, currenty as all modules and core are compiled together, an uberdatum-specific bisect loop can be made. If we move ubermodule out, it cannot be done. At the same time, if modules carried own versions of everything including all builtins, those modules would be just too big. Something in between would be a fixed "ABI" where core and stdlib are one blob and each 3rd party module is another blob, and everything that crosses the boundary is treated as a "pyobject". > > However, it is still not clear (to me) where previous efforts stopped. > Is it hard to: > 1) devise an algorithm like Dima proposed > or to > 2) implement it (because of too much code to change and limited manpower) > or to > 3) or to have a small performance loss? > > Per-file separate compilation would likely fall into 3), because too > little type inference would happen, isn't it? > >> If this is possible, it would be a useful quick hack to separate >> module build from main build. >> If it's still very hard, then some else is in order. >> I'd love to play with this myself, but I don't have enough ram for a >> full build ;-( > > -- > Paolo Giarrusso - Ph.D. Student > http://www.informatik.uni-marburg.de/~pgiarrusso/ > From mail at stevesimmons.com Tue Feb 15 09:57:21 2011 From: mail at stevesimmons.com (Stephen Simmons) Date: Tue, 15 Feb 2011 02:57:21 -0600 (CST) Subject: [pypy-dev] European sprints? Message-ID: <58722.170.148.215.157.1297760241.squirrel@mail5.webfaction.com> Hi... I have just started following the PyPy mailing list and am interested in getting involved in some small way. Are there any European sprints planned in the coming few months? Thanks Stephen From holger at merlinux.eu Tue Feb 15 11:25:06 2011 From: holger at merlinux.eu (holger krekel) Date: Tue, 15 Feb 2011 11:25:06 +0100 Subject: [pypy-dev] google should build a pypy phone or someone Message-ID: <20110215102506.GM30557@trillke.net> Hi all, what do you think or know why Google choose Java over Python for its Android system? Surely, Python doesn't have a big track record in terms of targetting mobile devices. Also CPython is limited in how it can change and adapt i guess. I imagine PyPy could do much better. Wouldn't it be cool to have a totally python-based state-of-the-art phone? :) cheers, holger From anto.cuni at gmail.com Tue Feb 15 11:36:43 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 15 Feb 2011 11:36:43 +0100 Subject: [pypy-dev] google should build a pypy phone or someone In-Reply-To: <20110215102506.GM30557@trillke.net> References: <20110215102506.GM30557@trillke.net> Message-ID: <4D5A573B.40403@gmail.com> On 15/02/11 11:25, holger krekel wrote: > Hi all, > > what do you think or know why Google choose Java over Python for > its Android system? Surely, Python doesn't have a big track record > in terms of targetting mobile devices. Also CPython is limited > in how it can change and adapt i guess. I imagine PyPy could do much > better. Wouldn't it be cool to have a totally python-based > state-of-the-art phone? :) yes: this way, when my friends ask me what the hell I'm working on, I could just show them the phone :-) More seriously, I think that nowadays one of the most important points for a mobile platform is to have a huge selection of 3rd party apps: in this sense, the popularity of Java and the big number of developers (especially mediocre and cheap developers, which is what you need to develop most of the apps out there) is a big win for android. On the other hand, iphone shows that you can do that even if your primary language is Objective-C, but it AFAIK objc was already used quite a lot in the apple ecosystem. Anyway, back to the original topic: what would python offer more than java for android? ciao, Anto From chef at ghum.de Tue Feb 15 11:52:23 2011 From: chef at ghum.de (Massa, Harald Armin) Date: Tue, 15 Feb 2011 11:52:23 +0100 Subject: [pypy-dev] google should build a pypy phone or someone In-Reply-To: <4D5A573B.40403@gmail.com> References: <20110215102506.GM30557@trillke.net> <4D5A573B.40403@gmail.com> Message-ID: Antonio, > > Anyway, back to the original topic: what would python offer more than java > for > android? > > a much less challenging licence-situation. Harald -- GHUM GmbH Harald Armin Massa Spielberger Stra?e 49 70435 Stuttgart 0173/9409607 Amtsgericht Stuttgart, HRB 734971 - persuadere. et programmare -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110215/bb150a03/attachment.htm From lac at openend.se Tue Feb 15 11:54:35 2011 From: lac at openend.se (Laura Creighton) Date: Tue, 15 Feb 2011 11:54:35 +0100 Subject: [pypy-dev] European sprints? In-Reply-To: Message from "Stephen Simmons" of "Tue, 15 Feb 2011 02:57:21 CST." <58722.170.148.215.157.1297760241.squirrel@mail5.webfaction.com> References: <58722.170.148.215.157.1297760241.squirrel@mail5.webfaction.com> Message-ID: <201102151054.p1FAsZWb018955@theraft.openend.se> In a message of Tue, 15 Feb 2011 02:57:21 CST, "Stephen Simmons" writes: >Hi... > >I have just started following the PyPy mailing list and am interested in >getting involved in some small way. > >Are there any European sprints planned in the coming few months? > >Thanks >Stephen I want to have one in G?teborg some time after PyCON. That's about as far as this discussion has got so far. Laura From holger at merlinux.eu Tue Feb 15 11:55:42 2011 From: holger at merlinux.eu (holger krekel) Date: Tue, 15 Feb 2011 11:55:42 +0100 Subject: [pypy-dev] google should build a pypy phone or someone In-Reply-To: References: <20110215102506.GM30557@trillke.net> <4D5A573B.40403@gmail.com> Message-ID: <20110215105542.GN30557@trillke.net> On Tue, Feb 15, 2011 at 11:52 +0100, Massa, Harald Armin wrote: > Antonio, > > > > > Anyway, back to the original topic: what would python offer more than java > > for > > android? > > > > a much less challenging licence-situation. > > Harald for those not in the know: Oracle filed a patents lawsuit against Googles Android Dalvik VM which is maybe mostly but not completely without merits. Holger > > -- > GHUM GmbH > Harald Armin Massa > Spielberger Stra?e 49 > 70435 Stuttgart > 0173/9409607 > > Amtsgericht Stuttgart, HRB 734971 > - > persuadere. > et programmare > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev -- From chef at ghum.de Tue Feb 15 12:00:31 2011 From: chef at ghum.de (Massa, Harald Armin) Date: Tue, 15 Feb 2011 12:00:31 +0100 Subject: [pypy-dev] google should build a pypy phone or someone In-Reply-To: <20110215105542.GN30557@trillke.net> References: <20110215102506.GM30557@trillke.net> <4D5A573B.40403@gmail.com> <20110215105542.GN30557@trillke.net> Message-ID: > > > > > a much less challenging licence-situation. > > for those not in the know: Oracle filed a patents lawsuit against Googles > Android > Dalvik VM which is maybe mostly but not completely without merits. > and, independend of the merits that may or may not be present, the legal expenses and collateral damage of that lawsuit will most surely be higher than the money put into PyPy the last 10years, European funding included. Harald -- GHUM GmbH Harald Armin Massa Spielberger Stra?e 49 70435 Stuttgart 0173/9409607 Amtsgericht Stuttgart, HRB 734971 - persuadere. et programmare -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110215/1e8d619c/attachment.htm From anto.cuni at gmail.com Tue Feb 15 12:14:29 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 15 Feb 2011 12:14:29 +0100 Subject: [pypy-dev] google should build a pypy phone or someone In-Reply-To: References: <20110215102506.GM30557@trillke.net> <4D5A573B.40403@gmail.com> Message-ID: <4D5A6015.9010509@gmail.com> On 15/02/11 11:52, Massa, Harald Armin wrote: > Antonio, > > > Anyway, back to the original topic: what would python offer more than java for > android? > > a much less challenging licence-situation. I'm not completely sure. AFAIK, the Oracle-Google lawsuit is about patents, not licences: in particular, they are complaining about how Dalvik is implemented, not about the fact that it implements Java. >From what I could read at least a couple of those patents apply to PyPy as well (and to whatever language with JIT, fwiw), e.g. IIRC there is one about "compiling an optimized version of code that executes often" or something similar (whether this patent is valid is another topic, of course). ciao, Anto From holger at merlinux.eu Tue Feb 15 12:26:56 2011 From: holger at merlinux.eu (holger krekel) Date: Tue, 15 Feb 2011 12:26:56 +0100 Subject: [pypy-dev] google should build a pypy phone or someone In-Reply-To: <4D5A6015.9010509@gmail.com> References: <20110215102506.GM30557@trillke.net> <4D5A573B.40403@gmail.com> <4D5A6015.9010509@gmail.com> Message-ID: <20110215112656.GO30557@trillke.net> On Tue, Feb 15, 2011 at 12:14 +0100, Antonio Cuni wrote: > On 15/02/11 11:52, Massa, Harald Armin wrote: > > Antonio, > > > > > > Anyway, back to the original topic: what would python offer more than java for > > android? > > > > a much less challenging licence-situation. > > I'm not completely sure. > AFAIK, the Oracle-Google lawsuit is about patents, not licences: in > particular, they are complaining about how Dalvik is implemented, not about > the fact that it implements Java. > > >From what I could read at least a couple of those patents apply to PyPy as > well (and to whatever language with JIT, fwiw), e.g. IIRC there is one about > "compiling an optimized version of code that executes often" or something > similar (whether this patent is valid is another topic, of course). FWIW, I haven't read that patent and i doubt Armin who invented Psyco ages ago and many parts of PyPy, has. Or other people inviting JIT-compilers since 1970 for that matter :) Being safe from patents is probably impossible, at least in the US. Still the situation with PyPy should be better as we didn't start with code that had patents already mentioned left and right. But let's try to not discuss patents and licensing too much. I am fine with it being a potentially pro-pypy aspect :) best, holger From lac at openend.se Tue Feb 15 12:51:49 2011 From: lac at openend.se (Laura Creighton) Date: Tue, 15 Feb 2011 12:51:49 +0100 Subject: [pypy-dev] google should build a pypy phone or someone In-Reply-To: Message from Antonio Cuni of "Tue, 15 Feb 2011 11:36:43 +0100." <4D5A573B.40403@gmail.com> References: <20110215102506.GM30557@trillke.net> <4D5A573B.40403@gmail.com> Message-ID: <201102151151.p1FBpnsk024534@theraft.openend.se> In a message of Tue, 15 Feb 2011 11:36:43 +0100, Antonio Cuni writes: > >Anyway, back to the original topic: what would python offer more than jav >a for >android? > >ciao, >Anto As some of you saw in Leysin, I own one of these. http://www.freedominput.com/freedom-accessories/freedom-pro-keyboard which works with Android. which makes the whole idea of being able to sit down and program whenever the mood strikes you, even when your laptop is far away or out of power more attractive, since you can touchtype. But that doesn't make me want to start programming in java. Especially for the sort of quick and dirty calculations I use python -c for. Laura From cfbolz at gmx.de Tue Feb 15 15:02:28 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Tue, 15 Feb 2011 15:02:28 +0100 Subject: [pypy-dev] European sprints? In-Reply-To: <201102151054.p1FAsZWb018955@theraft.openend.se> References: <58722.170.148.215.157.1297760241.squirrel@mail5.webfaction.com> <201102151054.p1FAsZWb018955@theraft.openend.se> Message-ID: <4D5A8774.3050700@gmx.de> On 02/15/2011 11:54 AM, Laura Creighton wrote: > In a message of Tue, 15 Feb 2011 02:57:21 CST, "Stephen Simmons" writes: >> Hi... >> >> I have just started following the PyPy mailing list and am interested in >> getting involved in some small way. >> >> Are there any European sprints planned in the coming few months? >> >> Thanks >> Stephen > > I want to have one in G?teborg some time after PyCON. That's about > as far as this discussion has got so far. Also I guess it is likely that we will have another one in D?sseldorf later in the year. Carl Friedrich From cfbolz at gmx.de Tue Feb 15 15:06:07 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Tue, 15 Feb 2011 15:06:07 +0100 Subject: [pypy-dev] google should build a pypy phone or someone In-Reply-To: <201102151151.p1FBpnsk024534@theraft.openend.se> References: <20110215102506.GM30557@trillke.net> <4D5A573B.40403@gmail.com> <201102151151.p1FBpnsk024534@theraft.openend.se> Message-ID: <4D5A884F.9020405@gmx.de> On 02/15/2011 12:51 PM, Laura Creighton wrote: > In a message of Tue, 15 Feb 2011 11:36:43 +0100, Antonio Cuni writes: > >> >> Anyway, back to the original topic: what would python offer more than jav >> a for >> android? >> >> ciao, >> Anto > > As some of you saw in Leysin, I own one of these. > http://www.freedominput.com/freedom-accessories/freedom-pro-keyboard which > works with Android. > > which makes the whole idea of being able to sit down and program whenever > the mood strikes you, even when your laptop is far away or out of power > more attractive, since you can touchtype. > > But that doesn't make me want to start programming in java. > Especially for the sort of quick and dirty calculations I use python > -c for. I am sure you know of this then: http://code.google.com/p/android-scripting/ It's an app for Android that lets you script the phone with various dynamic languages (including Python). Actually I think that PyPy could/should be made to work with this thing. It would not quite make PyPy a first class VM on the phone, but would already give you access to a number of phone-specific features. Carl Friedrich From anto.cuni at gmail.com Tue Feb 15 15:06:50 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 15 Feb 2011 15:06:50 +0100 Subject: [pypy-dev] European sprints? In-Reply-To: <4D5A8774.3050700@gmx.de> References: <58722.170.148.215.157.1297760241.squirrel@mail5.webfaction.com> <201102151054.p1FAsZWb018955@theraft.openend.se> <4D5A8774.3050700@gmx.de> Message-ID: <4D5A887A.2040206@gmail.com> On 15/02/11 15:02, Carl Friedrich Bolz wrote: > Also I guess it is likely that we will have another one in D?sseldorf > later in the year. and probably one after europython, as usual? ciao, Anto From renesd at gmail.com Tue Feb 15 15:16:25 2011 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Tue, 15 Feb 2011 14:16:25 +0000 Subject: [pypy-dev] google should build a pypy phone or someone In-Reply-To: <20110215102506.GM30557@trillke.net> References: <20110215102506.GM30557@trillke.net> Message-ID: Hi, a pypy phone would be lovely :) Especially if it could take advantage of the graphics processor (OpenGL ES2 shader language), and the programmable DSP chips (eg OMAP3). btw, python runs on nokia phones (with nokia supplied python, s60 and mamaeo/meego), windows CE phones/pocket pcs, android (see recent pygame android release), and even apple phones (it used to be technically banned from app store, but has run on i* for a long time. I guess the main "python platform" is OLPC, which is still quite interesting. ps. ) On Tue, Feb 15, 2011 at 10:25 AM, holger krekel wrote: > Hi all, > > what do you think or know why Google choose Java over Python for > its Android system? ?Surely, Python doesn't have a big track record > in terms of targetting mobile devices. ?Also CPython is limited > in how it can change and adapt i guess. ?I imagine PyPy could do much > better. ?Wouldn't it be cool to have a totally python-based > state-of-the-art phone? :) > > cheers, > holger > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From pypy at pocketnix.org Tue Feb 15 15:29:54 2011 From: pypy at pocketnix.org (pypy at pocketnix.org) Date: Tue, 15 Feb 2011 14:29:54 +0000 Subject: [pypy-dev] google should build a pypy phone or someone In-Reply-To: <4D5A884F.9020405@gmx.de> References: <20110215102506.GM30557@trillke.net> <4D5A573B.40403@gmail.com> <201102151151.p1FBpnsk024534@theraft.openend.se> <4D5A884F.9020405@gmx.de> Message-ID: <20110215142954.GB10549@pocketnix.org> On Tue, Feb 15, 2011 at 03:06:07PM +0100, Carl Friedrich Bolz wrote: > On 02/15/2011 12:51 PM, Laura Creighton wrote: > > http://www.freedominput.com/freedom-accessories/freedom-pro-keyboard which > > works with Android. > > > > which makes the whole idea of being able to sit down and program whenever > > the mood strikes you, even when your laptop is far away or out of power > > more attractive, since you can touchtype. > > > > But that doesn't make me want to start programming in java. > > Especially for the sort of quick and dirty calculations I use python > > -c for. > > I am sure you know of this then: > > http://code.google.com/p/android-scripting/ > > It's an app for Android that lets you script the phone with various > dynamic languages (including Python). Actually I think that PyPy > could/should be made to work with this thing. It would not quite make > PyPy a first class VM on the phone, but would already give you access to > a number of phone-specific features. the Scripting eniroment uses an enviroment varible to indicate a socket to connect to, once connected you make JSON calls over the socket to access the platform API's that are done via a java proxy. porting pypy to run in the enviroment would be as simple as compiling an arm port and importing the (all python) proxy libary i have a couple of scripts running on my phone for utility functions (checking my site is up and such) the API is a bit limited and i think that pypy running on the native VM would be more flexible for creating "full" programs (cant acceses many GUI componenets from this restricted enviroment) From dimaqq at gmail.com Tue Feb 15 18:22:04 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Tue, 15 Feb 2011 10:22:04 -0700 Subject: [pypy-dev] google should build a pypy phone or someone In-Reply-To: <20110215102506.GM30557@trillke.net> References: <20110215102506.GM30557@trillke.net> Message-ID: The real answers: 1. Google bought Android (company) 2. Java allows execution of untrusted 3rd party code 3, maybe, Java was "mature" around the time of acquisition, python 2.4 was released and there was psyco, but that's x86 only, when Android Inc started I guess they didn't have a serious python option at all. tools like IDEs and debuggers and hoards of underpaid Java developers in the wild surely contributed too. This topic is really not for pypy. My 2c. On 15 February 2011 03:25, holger krekel wrote: > Hi all, > > what do you think or know why Google choose Java over Python for > its Android system? ?Surely, Python doesn't have a big track record > in terms of targetting mobile devices. ?Also CPython is limited > in how it can change and adapt i guess. ?I imagine PyPy could do much > better. ?Wouldn't it be cool to have a totally python-based > state-of-the-art phone? :) > > cheers, > holger > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From drsalists at gmail.com Tue Feb 15 19:40:37 2011 From: drsalists at gmail.com (Dan Stromberg) Date: Tue, 15 Feb 2011 10:40:37 -0800 Subject: [pypy-dev] google should build a pypy phone or someone In-Reply-To: <20110215142954.GB10549@pocketnix.org> References: <20110215102506.GM30557@trillke.net> <4D5A573B.40403@gmail.com> <201102151151.p1FBpnsk024534@theraft.openend.se> <4D5A884F.9020405@gmx.de> <20110215142954.GB10549@pocketnix.org> Message-ID: On Tue, Feb 15, 2011 at 6:29 AM, wrote: > On Tue, Feb 15, 2011 at 03:06:07PM +0100, Carl Friedrich Bolz wrote: >> On 02/15/2011 12:51 PM, Laura Creighton wrote: > >> I am sure you know of this then: >> >> http://code.google.com/p/android-scripting/ >> >> It's an app for Android that lets you script the phone with various >> dynamic languages (including Python). Actually I think that PyPy >> could/should be made to work with this thing. It would not quite make >> PyPy a first class VM on the phone, but would already give you access to >> a number of phone-specific features. > > the Scripting eniroment uses an enviroment varible to indicate a > socket to connect to, once connected you make JSON calls over the > socket to access the platform API's that are done via a java proxy. > porting pypy to run in the enviroment would be as simple as compiling > an arm port and importing the (all python) proxy libary A guy at my local Python User Group uses an Android phone, and has explored both Python and Ruby in the Android Scripting Environment. He tells me that because the Ruby implementation is running on the Dalvik virtual machine, that its API access is more complete than that of the CPython port. Presumably you need a wrapper function for everything you want to chain into over the socket (in CPython), and people create those wrappers mostly on an as-needed basis. In light of this, it seems it might be better to put a JVM (which is almost the same as Dalvik) version of Pypy, or Jython, on Android rather than Pypy built for ARM. From dimaqq at gmail.com Tue Feb 15 20:09:49 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Tue, 15 Feb 2011 12:09:49 -0700 Subject: [pypy-dev] google should build a pypy phone or someone In-Reply-To: References: <20110215102506.GM30557@trillke.net> <4D5A573B.40403@gmail.com> <201102151151.p1FBpnsk024534@theraft.openend.se> <4D5A884F.9020405@gmx.de> <20110215142954.GB10549@pocketnix.org> Message-ID: There's always NDK which includes JNI bindings, so you can comile (or jit-compile) pypy and python code natively and still access android framework. If it really is a good idea, I cannot tell. I suppose if you want simple scripting, json-rpc would suit you fine, if you wanted more scripting, reflection, shared object space and good integration perhaps dalvik jvm is a way to go, if cpython or pypy becomes standard in android and apps are witten in python entirely, jni bindings are probably better, whether static or runtime reflection. I'm sure there would be a few issues to resolve though, object lifetime, garbage collection and callbacks for one. 2c from someone who went dumbphone 2 years ago :P On 15 February 2011 11:40, Dan Stromberg wrote: > On Tue, Feb 15, 2011 at 6:29 AM, ? wrote: >> On Tue, Feb 15, 2011 at 03:06:07PM +0100, Carl Friedrich Bolz wrote: >>> On 02/15/2011 12:51 PM, Laura Creighton wrote: >> > >>> I am sure you know of this then: >>> >>> http://code.google.com/p/android-scripting/ >>> >>> It's an app for Android that lets you script the phone with various >>> dynamic languages (including Python). Actually I think that PyPy >>> could/should be made to work with this thing. It would not quite make >>> PyPy a first class VM on the phone, but would already give you access to >>> a number of phone-specific features. >> >> the Scripting eniroment uses an enviroment varible to indicate a >> socket to connect to, once connected you make JSON calls over the >> socket to access the platform API's that are done via a java proxy. >> porting pypy to run in the enviroment would be as simple as compiling >> an arm port and importing the (all python) proxy libary > > A guy at my local Python User Group uses an Android phone, and has > explored both Python and Ruby in the Android Scripting Environment. > > He tells me that because the Ruby implementation is running on the > Dalvik virtual machine, that its API access is more complete than that > of the CPython port. ?Presumably you need a wrapper function for > everything you want to chain into over the socket (in CPython), and > people create those wrappers mostly on an as-needed basis. > > In light of this, it seems it might be better to put a JVM (which is > almost the same as Dalvik) version of Pypy, or Jython, on Android > rather than Pypy built for ARM. > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From stefan_ml at behnel.de Wed Feb 16 07:32:36 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 16 Feb 2011 07:32:36 +0100 Subject: [pypy-dev] google should build a pypy phone or someone In-Reply-To: References: <20110215102506.GM30557@trillke.net> <4D5A573B.40403@gmail.com> <201102151151.p1FBpnsk024534@theraft.openend.se> <4D5A884F.9020405@gmx.de> <20110215142954.GB10549@pocketnix.org> Message-ID: Dima Tisnek, 15.02.2011 20:09: > 2c from someone who went dumbphone 2 years ago :P Yay to phones that allow you to call other people without trying to distract you from what you were just thinking to do. Stefan From bea at changemaker.nu Wed Feb 16 12:01:38 2011 From: bea at changemaker.nu (Bea During) Date: Wed, 16 Feb 2011 12:01:38 +0100 Subject: [pypy-dev] European sprints? In-Reply-To: <4D5A887A.2040206@gmail.com> References: <58722.170.148.215.157.1297760241.squirrel@mail5.webfaction.com> <201102151054.p1FAsZWb018955@theraft.openend.se> <4D5A8774.3050700@gmx.de> <4D5A887A.2040206@gmail.com> Message-ID: <4D5BAE92.2080306@changemaker.nu> Hi there Regarding PyPy sprints in Europe during 2011: Antonio Cuni skrev: > On 15/02/11 15:02, Carl Friedrich Bolz wrote: > > >> Also I guess it is likely that we will have another one in D?sseldorf >> later in the year. >> > > and probably one after europython, as usual? > > ciao, > Anto > Here is a suggestion of places and dates based on Lauras, Carl Friedrich and Antos input: - Gothenburg sprint: 25th of April to 1st of May - Europython sprint/Florence: 25th of June to 26th of June (EP2011 official sprint dates) - D?sseldorf sprint: 22nd of August to 28th of August (fits the plan of the funded PyJIT project which ends end August) What do you think about these dates - would they work? Cheers Bea From lac at openend.se Wed Feb 16 13:08:07 2011 From: lac at openend.se (Laura Creighton) Date: Wed, 16 Feb 2011 13:08:07 +0100 Subject: [pypy-dev] European sprints? In-Reply-To: Message from Bea During of "Wed, 16 Feb 2011 12:01:38 +0100." <4D5BAE92.2080306@changemaker.nu> References: <58722.170.148.215.157.1297760241.squirrel@mail5.webfaction.com> <201102151054.p1FAsZWb018955@theraft.openend.se> <4D5A8774.3050700@gmx.de> <4D5A887A.2040206@gmail.com> <4D5BAE92.2080306@changemaker.nu> Message-ID: <201102161208.p1GC875O027115@theraft.openend.se> In a message of Wed, 16 Feb 2011 12:01:38 +0100, Bea During writes: >Here is a suggestion of places and dates based on Lauras, Carl Friedrich >and Antos >input: > >- Gothenburg sprint: 25th of April to 1st of May >- Europython sprint/Florence: 25th of June to 26th of June (EP2011 >official sprint dates) >- D?sseldorf sprint: 22nd of August to 28th of August (fits the plan of >the funded PyJIT project which ends end August) > >What do you think about these dates - would they work? > >Cheers > >Bea For me: The G?teborg dates work. The Florence dates also work, but we might want to consider staying around longer, because 2 days of sprint is awfully short. And I have no way to know if any dates in July or August will work until the members of N?set's Paddlarklub decide which weeks will be spend on sea kayaking tours. I don't know when that will be decided -- maybe Jacob does -- but not in the next week at any rate. Laura From william.leslie.ttg at gmail.com Wed Feb 16 13:24:01 2011 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Wed, 16 Feb 2011 23:24:01 +1100 Subject: [pypy-dev] Separate compilation and friends In-Reply-To: References: Message-ID: On 15 February 2011 12:13, Maciej Fijalkowski wrote: > Hi. > > There is growing interest about PyPy and especially about extension > modules. Apparently there are some people (like Alex) that are willing > to write modules in RPython that should not go to the main tree. Since > separate compilation is considered hard, how hard would it be to > provide separate loading? This would mean you still compile the whole > interpreter, but you can load the module from a compiled PyPy that > didn't have this option on. 0. What do you do about linking? Are rpython class and function names mangled consistently enough that a module compiled against one patch-level runtime with one set of options will link against another patch version with some different options? 1. Is it reasonable to ensure that *all* symbols that may be visible to an extension module are exported by the runtime? f may be inlined into g, but if f may still be callable by an extension module, it must be available by being exported static, too. 2. I always assumed this ("separate loading", ie, common translation) was the intended way to do separate compilation in rpython *anyway*; but there is one nagging thing. In order to define the boundary of the compilation, you already need to declare the interface of the module. Maybe not the annotations of the arguments, but at least which functions, when the annotator sees them, should be placed in your ".so". For the specific case of modules in the pypy python runtime this can be the interpleveldefs attribute of Module objects, but such functions all seem to have known signature annotation; they take a space and a number of wrapped arguments and return a wrapped object. Armed with this and the signature annotations of the functions in the core runtime, it is reasonable to expect that we can determine which functions belong in the ".so". Exactly how those core annotations are obtained doesn't need to be set in stone - obtained within the same translation or annotation performed earlier are both reasonable and not incompatible places to start. In both cases, the annotation is *computed* in the usual way. The alternative is defining the interface exported by the runtime explicitly, which is a mammoth task, and I don't think anyone has suggested it: is that what you are arguing against, fijal? There are still some details, such as how you export JitCodes from functions in your module, and what it means to do so - but nothing prohibitive that I can see. On 15 February 2011 19:32, Paolo Giarrusso wrote: > On Tue, Feb 15, 2011 at 09:17, Dima Tisnek wrote: >> I assume here that modules don't introduce dependencies into the iterpreter. >> I guess in the long run this ought to be the case, right? > > I don't think you can guarantee this. Type inference is global, and > you might need a user for each API to better infer its type. Maybe > uses of an API in testcases allow fully inferring their types, but I'd > guess not. If this does happen, it makes the callee part of a public interface, which should probably be explicitly annotated. I think checking for this case (where annotation in the extension would generalise the type signature of a dependency) is not too difficult. > However, what is true in general is that if less specific types are > inferred, that affects just performance, not correctness (I don't know > if that's true of PyPy, but you ought to be able to pass "object"s > around). Maybe the slowdown is insignificant, maybe it is a huge > problem, maybe few annotations can save the day. Annotation widens, rather than narrows: overly specific types are more likely to be inferred. > However, it is still not clear (to me) where previous efforts stopped. > Is it hard to: > 1) devise an algorithm like Dima proposed > or to > 2) implement it (because of too much code to change and limited manpower) > or to > 3) or to have a small performance loss? I understood that there was a lot of design to be done and there were other priorities. Devising an algorithm is not difficult, specifying it in terms of our existing annotation & flow model is slightly more so. > Per-file separate compilation would likely fall into 3), because too > little type inference would happen, isn't it? I suspect you are thinking of accidentally boxing interp-level integers or something, but that is an impossible condition (the dreaded SomeObject annotation). If not, where do you think a loss would come from? -- William Leslie From fuzzyman at voidspace.org.uk Wed Feb 16 13:30:22 2011 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 16 Feb 2011 12:30:22 +0000 Subject: [pypy-dev] Separate compilation and friends In-Reply-To: References: Message-ID: On 15 February 2011 01:13, Maciej Fijalkowski wrote: > Hi. > > There is growing interest about PyPy and especially about extension > modules. Apparently there are some people (like Alex) that are willing > to write modules in RPython that should not go to the main tree. As a side note I think Alex's code *should* go in the main tree. It is really *needed* to work with django and pypy (at least for those using postgres which is a significant proportion of django users) and it would be much better [for pypy] if that "just worked" out of the box. Michael > Since > separate compilation is considered hard, how hard would it be to > provide separate loading? This would mean you still compile the whole > interpreter, but you can load the module from a compiled PyPy that > didn't have this option on. > > Cheers, > fijal > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110216/13971646/attachment.htm From anto.cuni at gmail.com Wed Feb 16 13:33:12 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed, 16 Feb 2011 13:33:12 +0100 Subject: [pypy-dev] European sprints? In-Reply-To: <4D5BAE92.2080306@changemaker.nu> References: <58722.170.148.215.157.1297760241.squirrel@mail5.webfaction.com> <201102151054.p1FAsZWb018955@theraft.openend.se> <4D5A8774.3050700@gmx.de> <4D5A887A.2040206@gmail.com> <4D5BAE92.2080306@changemaker.nu> Message-ID: <4D5BC408.4060900@gmail.com> On 16/02/11 12:01, Bea During wrote: > Here is a suggestion of places and dates based on Lauras, Carl Friedrich and > Antos > input: > > - Gothenburg sprint: 25th of April to 1st of May > - Europython sprint/Florence: 25th of June to 26th of June (EP2011 official > sprint dates) > - D?sseldorf sprint: 22nd of August to 28th of August (fits the plan of the > funded PyJIT project which ends end August) > > What do you think about these dates - would they work? for me they "mostly" work, although I might arrive one day later or depart one day earlier for the Gothenburg and Duesseldorf ones. ciao, Anto From bea at changemaker.nu Wed Feb 16 14:31:17 2011 From: bea at changemaker.nu (Bea During) Date: Wed, 16 Feb 2011 14:31:17 +0100 Subject: [pypy-dev] European sprints? In-Reply-To: <4D5BC408.4060900@gmail.com> References: <58722.170.148.215.157.1297760241.squirrel@mail5.webfaction.com> <201102151054.p1FAsZWb018955@theraft.openend.se> <4D5A8774.3050700@gmx.de> <4D5A887A.2040206@gmail.com> <4D5BAE92.2080306@changemaker.nu> <4D5BC408.4060900@gmail.com> Message-ID: <4D5BD1A5.4050102@changemaker.nu> Hi there Antonio Cuni skrev: > On 16/02/11 12:01, Bea During wrote: > > >> Here is a suggestion of places and dates based on Lauras, Carl Friedrich and >> Antos >> input: >> >> - Gothenburg sprint: 25th of April to 1st of May >> - Europython sprint/Florence: 25th of June to 26th of June (EP2011 official >> sprint dates) >> - D?sseldorf sprint: 22nd of August to 28th of August (fits the plan of the >> funded PyJIT project which ends end August) >> >> What do you think about these dates - would they work? >> > > for me they "mostly" work, although I might arrive one day later or depart one > day earlier for the Gothenburg and Duesseldorf ones. > > ciao, > Anto > > My main idea with these suggested dates is to be able to state that these are our known sprint plans so far on the PyPy blog. We need to detail them more when we get closer and announce them through the usual channels. So if this would fit for D?sseldorf we should post on the blog about it. I can also see the need to sprint longer than 2 days at EP 2011 but we need to talk to the organizers to see if they have facilities available (this has been an issue during previous conferences). Cheers Bea From holger at merlinux.eu Thu Feb 17 15:23:06 2011 From: holger at merlinux.eu (holger krekel) Date: Thu, 17 Feb 2011 15:23:06 +0100 Subject: [pypy-dev] further pypy repo migrations Message-ID: <20110217142305.GC30557@trillke.net> hi all, i'd like to continue consolidation of PyPy repositories (and more generally pypy services but that in a separate mail). Thanks to Ronny Pfannschmidt, sponsorship from merlinux (my company), and work from Armin, Anto and others the main PyPy development repository has been migrated to mercurial on Bitbucket. Btw, we just asked and Bitbucket nicely upgraded the PyPy project account to become an unlimited one, thanks! Now we still have some more repositories on codespeak svn. One is "extradoc" which contains a lot of talk, sprint, misc infos and the "pypy.org" website content. We can either move this wholesale to an "extradoc" one on bitbucket or split it maybe into "pypy.org" and "extradoc". Note that the old codespeak.net subversion URLs will remain valid for quite some time so any cleanup/removal of dirs should better happen in the new mercurial repo. Another repo is "pypy-z", maybe not known to most. It is a repository where core project members keep information about more internal bits, like our previous contracts with Google, the European Union and some un-named entities (ask your favourite PSU representative for more info). I suggest we move this to become a "private" mercurial repo on bitbucket. Any more pypy related repositories that need migration or any notes/comments? best holger -- From anto.cuni at gmail.com Thu Feb 17 15:48:04 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu, 17 Feb 2011 15:48:04 +0100 Subject: [pypy-dev] further pypy repo migrations In-Reply-To: <20110217142305.GC30557@trillke.net> References: <20110217142305.GC30557@trillke.net> Message-ID: <4D5D3524.3040509@gmail.com> On 17/02/11 15:23, holger krekel wrote: > Thanks to Ronny Pfannschmidt, sponsorship from merlinux (my company), and work > from Armin, Anto and others the main PyPy development repository has been > migrated to mercurial on Bitbucket. Btw, we just asked and Bitbucket nicely > upgraded the PyPy project account to become an unlimited one, thanks! just to clarify: "unlimited account" means that we can have private repositories with as many users as we want (which is what we needed for pypy-z). Thanks to Bitbucket also from my side :-) > Any more pypy related repositories that need migration > or any notes/comments? currently, pypy has three svn subrepos which live on codespeak: greenlet = [svn]http://codespeak.net/svn/greenlet/trunk/c testrunner = [svn]http://codespeak.net/svn/pypy/build/testrunner lib_pypy/pyrepl = [svn]http://codespeak.net/svn/pyrepl/trunk/pyrepl/pyrepl we will need to migrate those as well at some point, and probably make them mercurial subrepositories instead of svn ones. ciao, Anto From lac at openend.se Thu Feb 17 17:43:20 2011 From: lac at openend.se (Laura Creighton) Date: Thu, 17 Feb 2011 17:43:20 +0100 Subject: [pypy-dev] further pypy repo migrations In-Reply-To: Message from holger krekel of "Thu, 17 Feb 2011 15:23:06 +0100." <20110217142305.GC30557@trillke.net> References: <20110217142305.GC30557@trillke.net> Message-ID: <201102171643.p1HGhKDX019756@theraft.openend.se> In a message of Thu, 17 Feb 2011 15:23:06 +0100, holger krekel writes: >Any more pypy related repositories that need migration >or any notes/comments? > >best >holger I just found a surprising number (at least it surprised me) of references in my own stuff to /svn/user/arigo/hack/ Laura From tismer at stackless.com Thu Feb 17 18:15:57 2011 From: tismer at stackless.com (Christian Tismer) Date: Thu, 17 Feb 2011 18:15:57 +0100 Subject: [pypy-dev] further pypy repo migrations In-Reply-To: <4D5D3524.3040509@gmail.com> References: <20110217142305.GC30557@trillke.net> <4D5D3524.3040509@gmail.com> Message-ID: <4D5D57CD.90100@stackless.com> On 2/17/11 3:48 PM, Antonio Cuni wrote: > On 17/02/11 15:23, holger krekel wrote: > >> Thanks to Ronny Pfannschmidt, sponsorship from merlinux (my company), and work >> from Armin, Anto and others the main PyPy development repository has been >> migrated to mercurial on Bitbucket. Btw, we just asked and Bitbucket nicely >> upgraded the PyPy project account to become an unlimited one, thanks! > just to clarify: "unlimited account" means that we can have private > repositories with as many users as we want (which is what we needed for > pypy-z). Thanks to Bitbucket also from my side :-) > > >> Any more pypy related repositories that need migration >> or any notes/comments? > currently, pypy has three svn subrepos which live on codespeak: > > greenlet = [svn]http://codespeak.net/svn/greenlet/trunk/c > testrunner = [svn]http://codespeak.net/svn/pypy/build/testrunner > lib_pypy/pyrepl = [svn]http://codespeak.net/svn/pyrepl/trunk/pyrepl/pyrepl > > we will need to migrate those as well at some point, and probably make them > mercurial subrepositories instead of svn ones. > > ciao, > Anto > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev Hi folks, if it helps, then I can offer to build a repos or website on my machine. I have a big machine with plenty of resources, which are only little used. Maybe for something that doesn't fit bitbucket very well, or as a mirror site, whatever makes sense. Right now I'm hosting starship.python.net and stackless.com as virtual machines, plus some experimental hidden servers. I'd be happy to donate an OpenVZ Debian server of reasonable size. Just let me know. cheers - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From arigo at tunes.org Fri Feb 18 16:18:15 2011 From: arigo at tunes.org (Armin Rigo) Date: Fri, 18 Feb 2011 16:18:15 +0100 Subject: [pypy-dev] further pypy repo migrations In-Reply-To: <201102171643.p1HGhKDX019756@theraft.openend.se> References: <20110217142305.GC30557@trillke.net> <201102171643.p1HGhKDX019756@theraft.openend.se> Message-ID: Hi Laura, On Thu, Feb 17, 2011 at 5:43 PM, Laura Creighton wrote: > I just found a surprising number (at least it surprised me) of references > in my own stuff to /svn/user/arigo/hack/ I moved this part of the repository to http://bitbucket.org/arigo/arigo . Armin From cfbolz at gmx.de Fri Feb 18 12:14:47 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Fri, 18 Feb 2011 12:14:47 +0100 Subject: [pypy-dev] CfP: ICOOOLPS 2011 Message-ID: <4D5E54A7.5000908@gmx.de> Hi all, I think we should consider sending a paper to ICOOOLPS this year again, see call for papers below. Carl Friedrich -- Call For Papers for the sixth annual workshop on Implementation, Compilation, Optimization of Object-Oriented Languages, Programs and Systems (ICOOOLPS) to be held at ECOOP 2011 in Lancaster, UK on July 26th 2011 http://www.icooolps.info/ Overview: Computer programming languages, especially Object-Oriented languages, are pervasive and play a significant role in computer science and engineering life and sometime appear as ubiquitous and completely mature. However, despite a large number of works, there still is a clear need for solutions for efficient implementation and compilation of OO languages in various application domains ranging from embedded and real-time systems to desktop systems. The ICOOOLPS workshop thus aims to address this crucial issue of optimization, mainly but not only in OO languages, programs and systems. It intends to do so by bringing together researchers and practitioners working in the field of implementation and optimization of programming languages. Its main goals are identifying fundamental bases and key current issues pertaining to the efficient implementation, compilation and optimization of OO languages, and outlining future challenges and research directions. An expected output of this workshop is a synthesis identifying fundamental bases and key current issues pertaining to the efficient implementation and compilation of OO languages, in order to spread them further amongst the various computing systems. It is also intended to extend this synthesis to encompass future challenges and research directions in the field of OO languages implementation and optimization, as well as non-OO languages. Important Dates: Workshop paper submission: April 15, 2011 Workshop paper acceptance: May 20, 2011 Workshop: July 26, 2011 Program Committee: Mark van den Brand TU Eindhoven, The Netherlands Peter Dickman Google, Switzerland Roland Ducournau LIRMM, France M. Anton Ertl Technical University of Vienna (TU Wien), Austria Remi Forax University of Marne-la-Vall?e, France Bj?rn Franke University of Edinburgh, UK Andreas Gal Mozilla Corporation, USA Kevin Hammond University of St Andrews, UK Tim Harris Microsoft Research, UK Michael Haupt Hasso-Plattner-Institut, Germany Eric Jul (co-chair) DIKU, The University of Copenhagen, Denmark Tomas Kalibera The University of Kent, UK Stein Krogdahl University of Oslo, Norway Francis Chi Moon Lau The University of Hong Kong, Hong Kong Ian Rogers (chair) Azul Systems, USA Jennifer Sartor Ecole Polytechnique Federale de Lausanne, Switzerland Manuel Serrano INRIA, France Mario Wolczko Oracle Labs, USA Olivier Zendra (co-chair) INRIA/LORIA, France From alex.gaynor at gmail.com Fri Feb 18 17:55:27 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Fri, 18 Feb 2011 11:55:27 -0500 Subject: [pypy-dev] CfP: ICOOOLPS 2011 In-Reply-To: <4D5E54A7.5000908@gmx.de> References: <4D5E54A7.5000908@gmx.de> Message-ID: On Fri, Feb 18, 2011 at 6:14 AM, Carl Friedrich Bolz wrote: > Hi all, > > I think we should consider sending a paper to ICOOOLPS this year again, > see call for papers below. > > Carl Friedrich > > > -- > > Call For Papers for the sixth annual workshop on Implementation, > Compilation, Optimization of Object-Oriented Languages, Programs and > Systems (ICOOOLPS) to be held at ECOOP 2011 in Lancaster, UK on July > 26th 2011 > > http://www.icooolps.info/ > > Overview: > > Computer programming languages, especially Object-Oriented languages, > are pervasive and play a significant role in computer science and > engineering life and sometime appear as ubiquitous and completely > mature. However, despite a large number of works, there still is a clear > need for solutions for efficient implementation and compilation of OO > languages in various application domains ranging from embedded and > real-time systems to desktop systems. > > The ICOOOLPS workshop thus aims to address this crucial issue of > optimization, mainly but not only in OO languages, programs and systems. > It intends to do so by bringing together researchers and practitioners > working in the field of implementation and optimization of programming > languages. Its main goals are identifying fundamental bases and key > current issues pertaining to the efficient implementation, compilation > and optimization of OO languages, and outlining future challenges and > research directions. > > An expected output of this workshop is a synthesis identifying > fundamental bases and key current issues pertaining to the efficient > implementation and compilation of OO languages, in order to spread them > further amongst the various computing systems. It is also intended to > extend this synthesis to encompass future challenges and research > directions in the field of OO languages implementation and optimization, > as well as non-OO languages. > > Important Dates: > > Workshop paper submission: April 15, 2011 > Workshop paper acceptance: May 20, 2011 > Workshop: July 26, 2011 > > Program Committee: > > Mark van den Brand TU Eindhoven, The Netherlands > Peter Dickman Google, Switzerland > Roland Ducournau LIRMM, France > M. Anton Ertl Technical University of Vienna (TU Wien), Austria > Remi Forax University of Marne-la-Vall?e, France > Bj?rn Franke University of Edinburgh, UK > Andreas Gal Mozilla Corporation, USA > Kevin Hammond University of St Andrews, UK > Tim Harris Microsoft Research, UK > Michael Haupt Hasso-Plattner-Institut, Germany > Eric Jul (co-chair) DIKU, The University of Copenhagen, Denmark > Tomas Kalibera The University of Kent, UK > Stein Krogdahl University of Oslo, Norway > Francis Chi Moon Lau The University of Hong Kong, Hong Kong > Ian Rogers (chair) Azul Systems, USA > Jennifer Sartor Ecole Polytechnique Federale de Lausanne, > Switzerland > Manuel Serrano INRIA, France > Mario Wolczko Oracle Labs, USA > Olivier Zendra (co-chair) INRIA/LORIA, France > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev Do you have any topics you think we should consider submitting? Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110218/b481d822/attachment.htm From cfbolz at gmx.de Mon Feb 21 18:12:22 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 21 Feb 2011 18:12:22 +0100 Subject: [pypy-dev] further pypy repo migrations In-Reply-To: <20110217142305.GC30557@trillke.net> References: <20110217142305.GC30557@trillke.net> Message-ID: <4D629CF6.4010007@gmx.de> On 02/17/2011 03:23 PM, holger krekel wrote: > i'd like to continue consolidation of PyPy repositories (and more generally > pypy services but that in a separate mail). > > Thanks to Ronny Pfannschmidt, sponsorship from merlinux (my company), and work > from Armin, Anto and others the main PyPy development repository has been > migrated to mercurial on Bitbucket. Btw, we just asked and Bitbucket nicely > upgraded the PyPy project account to become an unlimited one, thanks! > > Now we still have some more repositories on codespeak svn. One is "extradoc" > which contains a lot of talk, sprint, misc infos and the "pypy.org" website > content. We can either move this wholesale to an "extradoc" one on bitbucket or > split it maybe into "pypy.org" and "extradoc". Note that the old codespeak.net > subversion URLs will remain valid for quite some time so any cleanup/removal of > dirs should better happen in the new mercurial repo. > > Another repo is "pypy-z", maybe not known to most. It is a repository where > core project members keep information about more internal bits, like our > previous contracts with Google, the European Union and some un-named entities > (ask your favourite PSU representative for more info). I suggest we move this > to become a "private" mercurial repo on bitbucket. > > Any more pypy related repositories that need migration > or any notes/comments? There are also the repos in svn/pypy/lang that would need conversion. At least the Smalltalk one has branches in its history, maybe it's not important to keep the precise branch history though. Another consideration I have is that we have been handing out external pointers into our repositories from various places, e.g. from our blog posts and also in some papers. I don't see an easy solution to keep them valid, which is a bit of a pity. Cheers, Carl Friedrich From dcolish at gmail.com Tue Feb 22 03:39:47 2011 From: dcolish at gmail.com (Dan Colish) Date: Mon, 21 Feb 2011 18:39:47 -0800 Subject: [pypy-dev] further pypy repo migrations In-Reply-To: <4D629CF6.4010007@gmx.de> References: <20110217142305.GC30557@trillke.net> <4D629CF6.4010007@gmx.de> Message-ID: <3368745349BF49BD9866112698CBAF3B@gmail.com> On Monday, February 21, 2011 at 9:12 AM, Carl Friedrich Bolz wrote: On 02/17/2011 03:23 PM, holger krekel wrote: > > i'd like to continue consolidation of PyPy repositories (and more generally > > pypy services but that in a separate mail). > > > > Thanks to Ronny Pfannschmidt, sponsorship from merlinux (my company), and work > > from Armin, Anto and others the main PyPy development repository has been > > migrated to mercurial on Bitbucket. Btw, we just asked and Bitbucket nicely > > upgraded the PyPy project account to become an unlimited one, thanks! > > > > Now we still have some more repositories on codespeak svn. One is "extradoc" > > which contains a lot of talk, sprint, misc infos and the "pypy.org" website > > content. We can either move this wholesale to an "extradoc" one on bitbucket or > > split it maybe into "pypy.org" and "extradoc". Note that the old codespeak.net > > subversion URLs will remain valid for quite some time so any cleanup/removal of > > dirs should better happen in the new mercurial repo. > > > > Another repo is "pypy-z", maybe not known to most. It is a repository where > > core project members keep information about more internal bits, like our > > previous contracts with Google, the European Union and some un-named entities > > (ask your favourite PSU representative for more info). I suggest we move this > > to become a "private" mercurial repo on bitbucket. > > > > Any more pypy related repositories that need migration > > or any notes/comments? > > There are also the repos in svn/pypy/lang that would need conversion. At > least the Smalltalk one has branches in its history, maybe it's not > important to keep the precise branch history though. > > Another consideration I have is that we have been handing out external > pointers into our repositories from various places, e.g. from our blog > posts and also in some papers. I don't see an easy solution to keep them > valid, which is a bit of a pity. I'd be happy to write a forwarding application that can redirect any requests to the codespeak svn. This would allow the group to move these additional repositories to bitbucket, but not break existing links. With a good url mapping, this application could serve to forward most codespeak projects beyond pypy, which might be enticing for the codespeak admins. The application is fairly simple, so let me know if there is interest and I'll finish up the preliminary code that I've got done now. --Dan From dcolish at gmail.com Tue Feb 22 04:52:10 2011 From: dcolish at gmail.com (Dan Colish) Date: Mon, 21 Feb 2011 19:52:10 -0800 Subject: [pypy-dev] further pypy repo migrations In-Reply-To: <4D629CF6.4010007@gmx.de> References: <20110217142305.GC30557@trillke.net> <4D629CF6.4010007@gmx.de> Message-ID: Just a quick update, I've sketched out a working bouncer for just the pypy svn trunk. Its living here for now: https://gist.github.com/838189 -- Dan From dcolish at gmail.com Tue Feb 22 04:57:33 2011 From: dcolish at gmail.com (Dan Colish) Date: Mon, 21 Feb 2011 19:57:33 -0800 Subject: [pypy-dev] further pypy repo migrations In-Reply-To: <4D629CF6.4010007@gmx.de> References: <20110217142305.GC30557@trillke.net> <4D629CF6.4010007@gmx.de> Message-ID: <69510A9460C44204AC8F89603528560C@gmail.com> Errr, sorry I lied. The code's here https://bitbucket.org/dcolish/pypy-bouncer. There were a few more files that made it worth getting into a proper vcs repo. -- Dan From luke at freeculture.co.uk Tue Feb 22 15:11:22 2011 From: luke at freeculture.co.uk (Luke Taylor) Date: Tue, 22 Feb 2011 14:11:22 +0000 Subject: [pypy-dev] Couple of questions... Message-ID: Hi everybody, I'm not a developer but I'm a keen fan of pypy and I have been watching the project for about 6 months now. I'm interested to know.. if there are any plans to make PyPy implement Python 3.x ? Why don't extension modules for Python with bits written in C (eg. SciPy) work with PyPy? Thanks for your time, Luke -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110222/8e89f79a/attachment.htm From dimaqq at gmail.com Tue Feb 22 18:40:01 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Tue, 22 Feb 2011 10:40:01 -0700 Subject: [pypy-dev] Couple of questions... In-Reply-To: References: Message-ID: IIUC 2.7 support branch was being merged recently but not finished (?) and that's the natural way towards 3.x On 22 February 2011 07:11, Luke Taylor wrote: > Hi everybody, > > I'm not a developer but I'm a keen fan of pypy and I have been watching the > project for about 6 months now. > > I'm interested to know.. > > if there are any plans to make PyPy implement Python 3.x ? > > Why don't extension modules for Python with bits written in C (eg. SciPy) > work with PyPy? > > > Thanks for your time, > > > Luke > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From cfbolz at gmx.de Tue Feb 22 18:48:08 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Tue, 22 Feb 2011 18:48:08 +0100 Subject: [pypy-dev] further pypy repo migrations In-Reply-To: <3368745349BF49BD9866112698CBAF3B@gmail.com> References: <20110217142305.GC30557@trillke.net> <4D629CF6.4010007@gmx.de> <3368745349BF49BD9866112698CBAF3B@gmail.com> Message-ID: <4D63F6D8.2010702@gmx.de> Hi Dan, On 02/22/2011 03:39 AM, Dan Colish wrote: > I'd be happy to write a forwarding application that can redirect any > requests to the codespeak svn. This would allow the group to move > these additional repositories to bitbucket, but not break existing > links. With a good url mapping, this application could serve to > forward most codespeak projects beyond pypy, which might be enticing > for the codespeak admins. The application is fairly simple, so let me > know if there is interest and I'll finish up the preliminary code > that I've got done now. That sounds very cool. I have no clue how that would be integrated in whatever codespeak is going to end up being. Carl Friedrich From dcolish at gmail.com Tue Feb 22 19:21:00 2011 From: dcolish at gmail.com (Dan Colish) Date: Tue, 22 Feb 2011 10:21:00 -0800 Subject: [pypy-dev] further pypy repo migrations In-Reply-To: <4D63F6D8.2010702@gmx.de> References: <20110217142305.GC30557@trillke.net> <4D629CF6.4010007@gmx.de> <3368745349BF49BD9866112698CBAF3B@gmail.com> <4D63F6D8.2010702@gmx.de> Message-ID: <68C0B5EF6E354467AD4DE1B2A75E13FE@gmail.com> On Tuesday, February 22, 2011 at 9:48 AM, Carl Friedrich Bolz wrote: Hi Dan, > > On 02/22/2011 03:39 AM, Dan Colish wrote: > > I'd be happy to write a forwarding application that can redirect any > > requests to the codespeak svn. This would allow the group to move > > these additional repositories to bitbucket, but not break existing > > links. With a good url mapping, this application could serve to > > forward most codespeak projects beyond pypy, which might be enticing > > for the codespeak admins. The application is fairly simple, so let me > > know if there is interest and I'll finish up the preliminary code > > that I've got done now. > > That sounds very cool. I have no clue how that would be integrated in > whatever codespeak is going to end up being. > Who can I contact at codespeak to begin a discussion on integrating this bouncer application? I've got a working draft that seems to do the trick up on bitbucket now, so I think I can grind this bit of the migration out pretty quickly. --Dan From lac at openend.se Tue Feb 22 20:00:30 2011 From: lac at openend.se (Laura Creighton) Date: Tue, 22 Feb 2011 20:00:30 +0100 Subject: [pypy-dev] further pypy repo migrations In-Reply-To: Message from Dan Colish of "Tue, 22 Feb 2011 10:21:00 PST." <68C0B5EF6E354467AD4DE1B2A75E13FE@gmail.com> References: <20110217142305.GC30557@trillke.net> <4D629CF6.4010007@gmx.de> <3368745349BF49BD9866112698CBAF3B@gmail.com> <4D63F6D8.2010702@gmx.de> <68C0B5EF6E354467AD4DE1B2A75E13FE@gmail.com> Message-ID: <201102221900.p1MJ0UG0030841@theraft.openend.se> Holger Krekel is the person to talk with. But he hasn't been around for a few days. I think he is on vacation. Laura From arigo at tunes.org Wed Feb 23 16:03:02 2011 From: arigo at tunes.org (Armin Rigo) Date: Wed, 23 Feb 2011 16:03:02 +0100 Subject: [pypy-dev] Couple of questions... In-Reply-To: References: Message-ID: Hi Luke, On Tue, Feb 22, 2011 at 3:11 PM, Luke Taylor wrote: > if there are any plans to make PyPy implement Python 3.x ? There are no plans at the moment, though I imagine that such plans will emerge at one point. Indeed, we are almost done with supporting 2.7, which is "on the way there". > Why don't extension modules for Python with bits written in C (eg. SciPy) > work with PyPy? You can't generalize like that; some of them do, thanks to the cpyext module of PyPy. But not SciPy, which is using a lot of fonctionality of CPython that is "semi-internal". Moreover, cpyext is slow, so it's rather meant to support C extensions like wxWindow that are not there for performance but for functionality. We can improve performance a bit, but it's unlikely to ever get at a level where it's appropriate for NumPy or SciPy. Instead, what needs to be done there is rewrite some parts in a more PyPy-friendly way. (For NumPy there is a project at http://projects.scipy.org/numpy/wiki/NumPyRefactoring that is likely to be useful for us too.) A bient?t, Armin. From dcolish at gmail.com Wed Feb 23 17:29:53 2011 From: dcolish at gmail.com (Dan Colish) Date: Wed, 23 Feb 2011 08:29:53 -0800 Subject: [pypy-dev] Writing Mixed Modules for MacOS Message-ID: I've been working on porting the OSX specific libraries in CPython to PyPy and I'm running into issues. The shape of many fundamental data structures used in the CoreFoundation libraries are not available to me. I was hoping to just fudge it by using a ptr to that structure, but it is not working out as I had hoped. Here is a terribly incorrect sample of what I've been hacking on: http://paste.pocoo.org/show/yENK4gE11yRM9i6p46Ra and here is the corresponding osx documentation: http://goo.gl/i6mYD. I was looking at the approach taken for windows, but it is not clear to me where the fundamental structures for windows lie outside of the win32 specific bits in pypy.rpython.lltypesystem.module.ll_os. For the most part they seem to be defined in the modules where they will be used, so it is difficult for me to understand how it all fits together. In general, a higher level description of how to handle data types which you cannot openly declare would be very helpful. Thanks in advance. -- Dan From benjamin at python.org Wed Feb 23 18:49:06 2011 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 23 Feb 2011 11:49:06 -0600 Subject: [pypy-dev] Writing Mixed Modules for MacOS In-Reply-To: References: Message-ID: 2011/2/23 Dan Colish : > I've been working on porting the OSX specific libraries in CPython to PyPy and I'm running into issues. The shape of many fundamental data structures used in the CoreFoundation libraries are not available to me. I was hoping to just fudge it by using a ptr to that structure, but it is not working out as I had hoped. Here is a terribly incorrect sample of what I've been hacking on: http://paste.pocoo.org/show/yENK4gE11yRM9i6p46Ra and here is the corresponding osx documentation: http://goo.gl/i6mYD. To get an INT by reference you allocate an int by lltype.malloc. See some of the functions in ll_os for examples. > > I was looking at the approach taken for windows, but it is not clear to me where the fundamental structures for windows lie outside of the win32 specific bits in pypy.rpython.lltypesystem.module.ll_os. For the most part they seem to be defined in the modules where they will be used, so it is difficult for me to understand how it all fits together. In general, a higher level description of how to handle data types which you cannot openly declare would be very helpful. ://codespeak.net/mailman/listinfo/pypy-dev > -- Regards, Benjamin From thefridgeowl at gmail.com Wed Feb 23 19:04:22 2011 From: thefridgeowl at gmail.com (Henry Mason) Date: Wed, 23 Feb 2011 10:04:22 -0800 Subject: [pypy-dev] Writing Mixed Modules for MacOS In-Reply-To: References: Message-ID: <2BCB5410-64DA-437A-9146-9E3F49DB9BF9@gmail.com> On Feb 23, 2011, at 8:29 AM, Dan Colish wrote: > I've been working on porting the OSX specific libraries in CPython to PyPy and I'm running into issues. The shape of many fundamental data structures used in the CoreFoundation libraries are not available to me. I was hoping to just fudge it by using a ptr to that structure, but it is not working out as I had hoped. Here is a terribly incorrect sample of what I've been hacking on: http://paste.pocoo.org/show/yENK4gE11yRM9i6p46Ra and here is the corresponding osx documentation: http://goo.gl/i6mYD. Are you saying you don't know the shape of the opaque CFTypes, like CFNumber? Pointers to forward-declared structures is a pretty pervasive pattern in C API. For example, I see the FILE* returned by fopen() declared like this: FILEP = rffi.COpaquePtr('FILE') fopen = rffi.llexternal('fopen', [rffi.CCHARP, rffi.CCHARP], FILEP) fclose = rffi.llexternal('fclose', [FILEP], rffi.INT) in pypy/rpython/lltypesystem/test/test_ll2ctypes.py, so I imagine you'd be able to do the same for CFTypes: CFAllocatorRef = rffi.COpaquePtr('__ CFAllocator') CFNumberRef = rffi.COpaquePtr('__CFNumber') CFNumberCreateInt = rffi.llexternal('CFNumberCreate', [CFAllocatorRef, rffi.INT, rffi.VOIDP], CFNumberRef) > > I was looking at the approach taken for windows, but it is not clear to me where the fundamental structures for windows lie outside of the win32 specific bits in pypy.rpython.lltypesystem.module.ll_os. For the most part they seem to be defined in the modules where they will be used, so it is difficult for me to understand how it all fits together. In general, a higher level description of how to handle data types which you cannot openly declare would be very helpful. > > Thanks in advance. > > -- Dan > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev From dcolish at gmail.com Wed Feb 23 19:48:46 2011 From: dcolish at gmail.com (Dan Colish) Date: Wed, 23 Feb 2011 10:48:46 -0800 Subject: [pypy-dev] Writing Mixed Modules for MacOS In-Reply-To: <2BCB5410-64DA-437A-9146-9E3F49DB9BF9@gmail.com> References: <2BCB5410-64DA-437A-9146-9E3F49DB9BF9@gmail.com> Message-ID: On Wednesday, February 23, 2011 at 10:04 AM, Henry Mason wrote: > On Feb 23, 2011, at 8:29 AM, Dan Colish wrote: > > > I've been working on porting the OSX specific libraries in CPython to PyPy and I'm running into issues. The shape of many fundamental data structures used in the CoreFoundation libraries are not available to me. I was hoping to just fudge it by using a ptr to that structure, but it is not working out as I had hoped. Here is a terribly incorrect sample of what I've been hacking on: http://paste.pocoo.org/show/yENK4gE11yRM9i6p46Ra and here is the corresponding osx documentation: http://goo.gl/i6mYD. > > Are you saying you don't know the shape of the opaque CFTypes, like CFNumber? Pointers to forward-declared structures is a pretty pervasive pattern in C API. For example, I see the FILE* returned by fopen() declared like this: > > FILEP = rffi.COpaquePtr('FILE') > fopen = rffi.llexternal('fopen', [rffi.CCHARP, rffi.CCHARP], FILEP) > fclose = rffi.llexternal('fclose', [FILEP], rffi.INT) > > in pypy/rpython/lltypesystem/test/test_ll2ctypes.py, so I imagine you'd be able to do the same for CFTypes: > > CFAllocatorRef = rffi.COpaquePtr('__ CFAllocator') > CFNumberRef = rffi.COpaquePtr('__CFNumber') > CFNumberCreateInt = rffi.llexternal('CFNumberCreate', [CFAllocatorRef, rffi.INT, rffi.VOIDP], CFNumberRef) > > > This is exactly what I was looking for, thank you! --Dan From holger at merlinux.eu Thu Feb 24 21:31:15 2011 From: holger at merlinux.eu (holger krekel) Date: Thu, 24 Feb 2011 21:31:15 +0100 Subject: [pypy-dev] further pypy repo migrations In-Reply-To: <4D629CF6.4010007@gmx.de> References: <20110217142305.GC30557@trillke.net> <4D629CF6.4010007@gmx.de> Message-ID: <20110224203115.GA14130@trillke.net> On Mon, Feb 21, 2011 at 18:12 +0100, Carl Friedrich Bolz wrote: > On 02/17/2011 03:23 PM, holger krekel wrote: > > There are also the repos in svn/pypy/lang that would need conversion. At > least the Smalltalk one has branches in its history, maybe it's not > important to keep the precise branch history though. OK, thanks for pointing out. > Another consideration I have is that we have been handing out external > pointers into our repositories from various places, e.g. from our blog > posts and also in some papers. I don't see an easy solution to keep them > valid, which is a bit of a pity. As posted earlier i plan to keep a read-only version of the svn repo around. We could degrade this to statically serving files or maybe use Dan's forwarder. best, holger From Ronny.Pfannschmidt at gmx.de Fri Feb 25 22:36:16 2011 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Fri, 25 Feb 2011 22:36:16 +0100 Subject: [pypy-dev] implementing the additional repo migrations Message-ID: <1298669776.21626.20.camel@Klappe2> hi, this night i started taking a look at the extra repos that need to be migrated. many of them contain reconstructible, but large pdf files, that i'd like to kill off for saving space. this shouldn't be a problem as long as the svn that many people link to stays available read-only, but it would need a bit of a plan if that goes as well. im currently investigating what parts of extradoc are reconstructible it seems the most costly things are * the split + full variants of the eu report (no tex aviliable) * papers (also no tex available) * api_html.tar.gz * the talks directory the talks directory is the largest size offender as well as the tree that?s the hardest to figure out (in terms of what?s reconstructible and what not). (more details later) The language repo's seem to be mostly fine, except for the prolog one, which contains a few reconstructible pdf's regards Ronny -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part Url : http://codespeak.net/pipermail/pypy-dev/attachments/20110225/1b43f93e/attachment.pgp From lac at openend.se Sat Feb 26 03:31:31 2011 From: lac at openend.se (Laura Creighton) Date: Sat, 26 Feb 2011 03:31:31 +0100 Subject: [pypy-dev] implementing the additional repo migrations In-Reply-To: Message from Ronny Pfannschmidt of "Fri, 25 Feb 2011 22:36:16 +0100." <1298669776.21626.20.camel@Klappe2> References: <1298669776.21626.20.camel@Klappe2> Message-ID: <201102260231.p1Q2VVOU018324@theraft.openend.se> In a message of Fri, 25 Feb 2011 22:36:16 +0100, Ronny Pfannschmidt writes: >hi, > >this night i started taking a look at the extra repos that need to be >migrated. > >many of them contain reconstructible, but large pdf files, that i'd like >to kill off for saving space. Why should we ever care about space? >this shouldn't be a problem as long as the svn that many people link to >stays available read-only, but it would need a bit of a plan if that >goes as well. > >im currently investigating what parts of extradoc are reconstructible I don't think that much (or maybe any) reconstruction is necessary. The things in extradoc really are the tex files, pdfs and what-have-you that you would expect from their filename extensions. The problem is that codespeak improperly serves them up as binary files. So no reconstruction needed. Just serve them properly. Laura >regards Ronny From holger at merlinux.eu Sat Feb 26 09:09:50 2011 From: holger at merlinux.eu (holger krekel) Date: Sat, 26 Feb 2011 09:09:50 +0100 Subject: [pypy-dev] implementing the additional repo migrations In-Reply-To: <201102260231.p1Q2VVOU018324@theraft.openend.se> References: <1298669776.21626.20.camel@Klappe2> <201102260231.p1Q2VVOU018324@theraft.openend.se> Message-ID: <20110226080950.GN12505@trillke.net> Hi Laura, On Sat, Feb 26, 2011 at 03:31 +0100, Laura Creighton wrote: > In a message of Fri, 25 Feb 2011 22:36:16 +0100, Ronny Pfannschmidt writes: > >hi, > > > >this night i started taking a look at the extra repos that need to be > >migrated. > > > >many of them contain reconstructible, but large pdf files, that i'd like > >to kill off for saving space. > > Why should we ever care about space? Small repositories are faster to clone and work with. I am fine to copy everything related to extradoc, though. > >this shouldn't be a problem as long as the svn that many people link to > >stays available read-only, but it would need a bit of a plan if that > >goes as well. > > > >im currently investigating what parts of extradoc are reconstructible > > I don't think that much (or maybe any) reconstruction is necessary. > The things in extradoc really are the tex files, pdfs and what-have-you > that you would expect from their filename extensions. The problem > is that codespeak improperly serves them up as binary files. So no > reconstruction needed. Just serve them properly. this has nothing to do with codespeak but which svn:mime-type files have in the svn repository. If you find a file in the repo that should have a certain mime-type you can e. g. svn ps svn:mime-type application/pdf path/to/file.pdf it. I am actually not sure how mercurial or bitbucket handles serving such files. But I think we should anyway go for serving talks via http://pypy.org/talks/ just through apache and avoid giving out links to version control repositories. We might eventually migrate away from bitbucket and we would then again have the problem of stale links. cheers, holger From jacob at openend.se Sat Feb 26 10:25:45 2011 From: jacob at openend.se (Jacob =?iso-8859-1?q?Hall=E9n?=) Date: Sat, 26 Feb 2011 10:25:45 +0100 Subject: [pypy-dev] implementing the additional repo migrations In-Reply-To: <201102260231.p1Q2VVOU018324@theraft.openend.se> References: <1298669776.21626.20.camel@Klappe2> <201102260231.p1Q2VVOU018324@theraft.openend.se> Message-ID: <201102261025.46282.jacob@openend.se> Saturday 26 February 2011 you wrote: > In a message of Fri, 25 Feb 2011 22:36:16 +0100, Ronny Pfannschmidt writes: > >hi, > > > >this night i started taking a look at the extra repos that need to be > >migrated. > > > >many of them contain reconstructible, but large pdf files, that i'd like > >to kill off for saving space. > > Why should we ever care about space? A mercurial repository is cloned as a whole, including all revision history. This does take up a lot of space with text files, but produces very large repositories with binary formats, resulting in bad behaviour on slow links and problems on devices with limited space. While I am fine with dropping older revisions of just about everything in extradoc, I wonder if it wouldn't be better better for the future to keep this repository in svn format. That way you will only get one copy of everything when cloning the repository. Jacob -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. Url : http://codespeak.net/pipermail/pypy-dev/attachments/20110226/547e5268/attachment.pgp From lac at openend.se Sat Feb 26 10:29:47 2011 From: lac at openend.se (Laura Creighton) Date: Sat, 26 Feb 2011 10:29:47 +0100 Subject: [pypy-dev] implementing the additional repo migrations In-Reply-To: Message from holger krekel of "Sat, 26 Feb 2011 09:09:50 +0100." <20110226080950.GN12505@trillke.net> References: <1298669776.21626.20.camel@Klappe2> <201102260231.p1Q2VVOU018324@theraft.openend.se> <20110226080950.GN12505@trillke.net> Message-ID: <201102260929.p1Q9TlRf022670@theraft.openend.se> In a message of Sat, 26 Feb 2011 09:09:50 +0100, holger krekel writes: >Hi Laura, > >> Why should we ever care about space? > >Small repositories are faster to clone and work with. >I am fine to copy everything related to extradoc, though. I thought we had unlimited repos in bitbucket. If so, then this is an argument for putting the extra documentation in a different repo than the pypy one, not for getting rid of things in extradoc. >> I don't think that much (or maybe any) reconstruction is necessary. >> The things in extradoc really are the tex files, pdfs and what-have-you >> that you would expect from their filename extensions. The problem >> is that codespeak improperly serves them up as binary files. So no >> reconstruction needed. Just serve them properly. > >this has nothing to do with codespeak but which svn:mime-type >files have in the svn repository. If you find a file in the repo >that should have a certain mime-type you can e. g. > > svn ps svn:mime-type application/pdf path/to/file.pdf > >it. > >I am actually not sure how mercurial or bitbucket handles serving such fi >les. Sorry, I knew that, I didn't mean to imply that it was some flaw in codespeak. Though it is very far from clear to me how svn got in the business of giving our files mimetypes, which I think happened secretly, under-the-hood. I don't think any of us explicitly went around typing svn commands to set a mime-type of binary to those files, it just happened silently. And, for me, at any rate, svn was an unexpected culprit -- by this I mean that when I went investigating the problem of 'why is this talk being served up as a binary file' svn was not one of the places I thought to look. The mercurial authors (like me) don't like the concept of binrary files, see http://mercurial.selenic.com/wiki/BinaryFiles , so I guess all we need to find out is does bitbucket itself make assumptions. I wouldn't think so, but then I thought that about svn as well. >But I think we should anyway go for serving talks via http://pypy.org/tal >ks/ >just through apache and avoid giving out links to version control >repositories. We might eventually migrate away from bitbucket >and we would then again have the problem of stale links. I'm not fond of a solution which leaves the extradoc files in two versions, the one in the repo and the one on http://pypy.org/talks , and where you have to keep remembering to change the pypy.org one because you have made a change to the one in the repo. >cheers, >holger I think we may have a more general problem, in that the flexibility of pypy should lead to massive experimentation, not only by ourselves but by third parties. What then should we do with the experiments? I'd like to find a way to keep the interesting ones around without making branching horribly slow. I thought that having many separate repositories would go a long way to getting this, was this also a mistaken assumption? Laura From alex.gaynor at gmail.com Sat Feb 26 10:29:20 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Sat, 26 Feb 2011 04:29:20 -0500 Subject: [pypy-dev] implementing the additional repo migrations In-Reply-To: <201102261025.46282.jacob@openend.se> References: <1298669776.21626.20.camel@Klappe2> <201102260231.p1Q2VVOU018324@theraft.openend.se> <201102261025.46282.jacob@openend.se> Message-ID: On Sat, Feb 26, 2011 at 4:25 AM, Jacob Hall?n wrote: > Saturday 26 February 2011 you wrote: > > In a message of Fri, 25 Feb 2011 22:36:16 +0100, Ronny Pfannschmidt > writes: > > >hi, > > > > > >this night i started taking a look at the extra repos that need to be > > >migrated. > > > > > >many of them contain reconstructible, but large pdf files, that i'd like > > >to kill off for saving space. > > > > Why should we ever care about space? > > A mercurial repository is cloned as a whole, including all revision > history. > This does take up a lot of space with text files, but produces very large > repositories with binary formats, resulting in bad behaviour on slow links > and > problems on devices with limited space. > > While I am fine with dropping older revisions of just about everything in > extradoc, I wonder if it wouldn't be better better for the future to keep > this > repository in svn format. That way you will only get one copy of everything > when cloning the repository. > > Jacob > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > There's also http://mercurial.selenic.com/wiki/BigfilesExtension, which I know nothing about. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110226/36f577ba/attachment.htm From lac at openend.se Sat Feb 26 10:51:34 2011 From: lac at openend.se (Laura Creighton) Date: Sat, 26 Feb 2011 10:51:34 +0100 Subject: [pypy-dev] implementing the additional repo migrations In-Reply-To: Message from Jacob =?iso-8859-1?q?Hall=E9n?= of "Sat, 26 Feb 2011 10:25:45 +0100." <201102261025.46282.jacob@openend.se> References: <1298669776.21626.20.camel@Klappe2> <201102260231.p1Q2VVOU018324@theraft.openend.se> <201102261025.46282.jacob@openend.se> Message-ID: <201102260951.p1Q9pYQ9024873@theraft.openend.se> In a message of Sat, 26 Feb 2011 10:25:45 +0100, Jacob Hall?n writes: >While I am fine with dropping older revisions of just about everything in >extradoc, I wonder if it wouldn't be better better for the future to keep >this repository in svn format. That way you will only get one copy of >everything when cloning the repository. > >Jacob I'm not fine with the dropping of older revisions. One of the chief benefits for me of moving to a version control system was that I could feel comfortable ruthlessly condensing my writing, knowing that if I ever wanted this stuff later -- say to use in a different document, I could always go back and get the old revision that contained the wonderful words or diagrams I now propose to cut. And this has happened in the past, where early versions of things I wrote ended up raised out of the grave of the repository to live on as part of completely different documents. I'm not going to be comfortable deleting stuff this if I think that the grim reaper is out there, just waiting to purge all my earlier attempts once some document is deemed to be 'final'. So I either won't delete stuff, or I will go back to my old practice of having dozens of versions around 'just in case'. I'm fine with continuing to have the extradoc managed by svn, though I really want a script that runs nightly looking for things in extradoc that have a mimetype of binary and complains about this. Laura From fijall at gmail.com Sat Feb 26 10:54:39 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 26 Feb 2011 11:54:39 +0200 Subject: [pypy-dev] implementing the additional repo migrations In-Reply-To: <201102260951.p1Q9pYQ9024873@theraft.openend.se> References: <1298669776.21626.20.camel@Klappe2> <201102260231.p1Q2VVOU018324@theraft.openend.se> <201102261025.46282.jacob@openend.se> <201102260951.p1Q9pYQ9024873@theraft.openend.se> Message-ID: On Sat, Feb 26, 2011 at 11:51 AM, Laura Creighton wrote: > In a message of Sat, 26 Feb 2011 10:25:45 +0100, Jacob Hall?n writes: > > > >>While I am fine with dropping older revisions of just about everything in >>extradoc, I wonder if it wouldn't be better better for the future to keep >>this repository in svn format. That way you will only get one copy of >>everything when cloning the repository. >> >>Jacob > > I'm not fine with the dropping of older revisions. ?One of the chief > benefits for me of moving to a version control system was that I could > feel comfortable ruthlessly condensing my writing, knowing that if I > ever wanted this stuff later -- say to use in a different document, > I could always go back and get the old revision that contained the > wonderful words or diagrams I now propose to cut. ?And this has > happened in the past, where early versions of things I wrote ended > up raised out of the grave of the repository to live on as part of > completely different documents. > > I'm not going to be comfortable deleting stuff this if I think that > the grim reaper is out there, just waiting to purge all my earlier > attempts once some document is deemed to be 'final'. ?So I either > won't delete stuff, or I will go back to my old practice of having > dozens of versions around 'just in case'. > > I'm fine with continuing to have the extradoc managed by svn, though > I really want a script that runs nightly looking for things in > extradoc that have a mimetype of binary and complains about this. > > Laura > Isn't the debate mostly about older revisions of binary files, since the rest is fine? Or you want to keep those as well? (say .doc) From lac at openend.se Sat Feb 26 10:59:09 2011 From: lac at openend.se (Laura Creighton) Date: Sat, 26 Feb 2011 10:59:09 +0100 Subject: [pypy-dev] implementing the additional repo migrations In-Reply-To: Message from Maciej Fijalkowski of "Sat, 26 Feb 2011 11:54:39 +0200." References: <1298669776.21626.20.camel@Klappe2> <201102260231.p1Q2VVOU018324@theraft.openend.se> <201102261025.46282.jacob@openend.se> <201102260951.p1Q9pYQ9024873@theraft.openend.se> Message-ID: <201102260959.p1Q9x9jT025385@theraft.openend.se> In a message of Sat, 26 Feb 2011 11:54:39 +0200, Maciej Fijalkowski writes: >Isn't the debate mostly about older revisions of binary files, since >the rest is fine? Or you want to keep those as well? (say .doc) I don't care about the old versions of binary files. Laura From pgiarrusso at mathematik.uni-marburg.de Sat Feb 26 13:01:41 2011 From: pgiarrusso at mathematik.uni-marburg.de (Paolo Giarrusso) Date: Sat, 26 Feb 2011 13:01:41 +0100 Subject: [pypy-dev] implementing the additional repo migrations In-Reply-To: <201102260929.p1Q9TlRf022670@theraft.openend.se> References: <1298669776.21626.20.camel@Klappe2> <201102260231.p1Q2VVOU018324@theraft.openend.se> <20110226080950.GN12505@trillke.net> <201102260929.p1Q9TlRf022670@theraft.openend.se> Message-ID: On Sat, Feb 26, 2011 at 10:29, Laura Creighton wrote: > I think we may have a more general problem, in that the flexibility of > pypy should lead to massive experimentation, not only by ourselves but > by third parties. ?What then should we do with the experiments? ?I'd like > to find a way to keep the interesting ones around without making > branching horribly slow. ?I thought that having many separate repositories > would go a long way to getting this, was this also a mistaken assumption? DVCS allow you to give a different meaning to "many separate repositories", which was not possible with SVN and which answers your question. Namely, there is little reason why experiment branches should be part of the same repo - you can just fork the project through bitbucket tools and make changes on the fork. If and when the fork is completed and useful, it can be pulled back, otherwise it just stays separate. With Hg the only problem is in storing N repositories on a computer without storing N copies of the common history, which is a different but related problem. To make branching fast, hardlinks allow to share the history between related repos, especially in Git but it seems also in Hg - search "hardlink" in the hg(1) man page, and this extension: http://mercurial.selenic.com/wiki/RelinkExtension While I'm a fan of Hg, the Git repository layout seems intrinsically better from this point of view: with Hg, as soon as you commit a change to a file in a branch, the existing history of that file can't be shared any more between the two repos (at least, it can't possibly be shared through a hardlink; it doesn't seem they implement other solutions, and it would be hard to do that in a way coherent with their design). I've got no experience with Hg on huge projects, while I have 8 branches of the Linux kernel laying around: the history of each additional branches takes ~5M (compared to 450M for the history of the main repo). If the branch stays close enough, though, this might not be a problem, but YMMV. Cheers, -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From arigo at tunes.org Sat Feb 26 13:03:15 2011 From: arigo at tunes.org (Armin Rigo) Date: Sat, 26 Feb 2011 13:03:15 +0100 Subject: [pypy-dev] implementing the additional repo migrations In-Reply-To: <201102260959.p1Q9x9jT025385@theraft.openend.se> References: <1298669776.21626.20.camel@Klappe2> <201102260231.p1Q2VVOU018324@theraft.openend.se> <201102261025.46282.jacob@openend.se> <201102260951.p1Q9pYQ9024873@theraft.openend.se> <201102260959.p1Q9x9jT025385@theraft.openend.se> Message-ID: Hi Laura, On Sat, Feb 26, 2011 at 10:59 AM, Laura Creighton wrote: > I don't care about the old versions of binary files. That was the only thing we talked about -- as far as I understood, it was never suggested that we should stop tracking revisions of .txt or .tex files. I don't know the BigfilesExtension either, but it looks to me like we can achieve some more precise result manually. Something along the lines of: the .pdf's built from .tex's are not checked in, but they are in some standardized place on http://pypy.org, where we can fetch them, update them (via ssh), or point people to (via their url). This can be easily done with a script independent from Mercurial. (The point is of course that tracking revisions is a bit useless, because we can always go back in time and re-run latex2pdf.) Well, this was my 2 cents to this discussion, but maybe Ronny is just worrying too much. I don't think we care that much about saving space or transfer time. It's anyway not like everybody on the planet should download our extradoc repository. A bient?t, Armin. From cfbolz at gmx.de Sat Feb 26 22:17:21 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Sat, 26 Feb 2011 22:17:21 +0100 Subject: [pypy-dev] implementing the additional repo migrations In-Reply-To: <1298669776.21626.20.camel@Klappe2> References: <1298669776.21626.20.camel@Klappe2> Message-ID: <4D696DE1.7020906@gmx.de> On 02/25/2011 10:36 PM, Ronny Pfannschmidt wrote: > this night i started taking a look at the extra repos that need to be > migrated. > > many of them contain reconstructible, but large pdf files, that i'd like > to kill off for saving space. > > this shouldn't be a problem as long as the svn that many people link to > stays available read-only, but it would need a bit of a plan if that > goes as well. > > im currently investigating what parts of extradoc are reconstructible > > > it seems the most costly things are > > * the split + full variants of the eu report (no tex aviliable) > * papers (also no tex available) > * api_html.tar.gz > * the talks directory > > the talks directory is the largest size offender as well as the tree > that?s the hardest to figure out (in terms of what?s reconstructible and > what not). > > (more details later) > > > The language repo's seem to be mostly fine, > except for the prolog one, which contains a few reconstructible pdf's Don't worry about Prolog, that has been manually migrated already by me a while ago. Carl Friedrich From cfbolz at gmx.de Sat Feb 26 22:25:05 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Sat, 26 Feb 2011 22:25:05 +0100 Subject: [pypy-dev] implementing the additional repo migrations In-Reply-To: References: <1298669776.21626.20.camel@Klappe2> <201102260231.p1Q2VVOU018324@theraft.openend.se> <201102261025.46282.jacob@openend.se> <201102260951.p1Q9pYQ9024873@theraft.openend.se> <201102260959.p1Q9x9jT025385@theraft.openend.se> Message-ID: <4D696FB1.5020100@gmx.de> On 02/26/2011 01:03 PM, Armin Rigo wrote: > Hi Laura, > > On Sat, Feb 26, 2011 at 10:59 AM, Laura Creighton wrote: >> I don't care about the old versions of binary files. > > That was the only thing we talked about -- as far as I understood, it > was never suggested that we should stop tracking revisions of .txt or > .tex files. I don't know the BigfilesExtension either, but it looks > to me like we can achieve some more precise result manually. > Something along the lines of: the .pdf's built from .tex's are not > checked in, but they are in some standardized place on > http://pypy.org, where we can fetch them, update them (via ssh), or > point people to (via their url). This can be easily done with a > script independent from Mercurial. (The point is of course that > tracking revisions is a bit useless, because we can always go back in > time and re-run latex2pdf.) Not necessarily, it's always possible that whatever latex packages were needed to compile the pdf are no longer around or a big hassle to install. This can make regeneration impractical. So I am in favor of keeping the PDFs in the repo. Carl Friedrich From lac at openend.se Sun Feb 27 16:38:03 2011 From: lac at openend.se (Laura Creighton) Date: Sun, 27 Feb 2011 16:38:03 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: Message from Laura Creighton of "Sun, 27 Feb 2011 14:59:46 +0100." <201102271359.p1RDxkpX012065@theraft.openend.se> References: <201102271359.p1RDxkpX012065@theraft.openend.se> Message-ID: <201102271538.p1RFc3rG020721@theraft.openend.se> I'd like to change what is displayed on the speed.python.org frontpage. Right now, as I look at http://speed.pypy.org/ I see, under a section called 'latest results' a list of all the recent times when we became slower. It's thus a 'recent problems' page -- we have actually improved in recent times in many areas, and nowhere is that shown. As we go off to PyCON, which is March 9-17, I intend to mention how great PyPy is, and that you can see it for yourself at speed.pypy.org. Thus, without lying, I would like it if the first impression of PyPy's speed that people got when looking at the site was 'we're getting faster'. Do you think you could change the front page so that what was displayed was more balanced with respect to good news and bad news? I realise that there is nothing you can do if we make a recent build that slows everything down, but for instance in build 42312:392b (Feb 26) we have improvments which are not shown on the main page. I actually think that the _trend_ is a more useful thing to display on the front page, though that might be because it is so green right now. :-) The other thing I want is for the graphs you get, for instance with http://speed.pypy.org/changes/?rev=42312:392bbf936179&exe=%203&env=tannit to have, in addition to the selection button beside: 'result for revision' an actual label that says 'build 42312:392b' or something that you can select with your mouse and use to paste into things like this mail article. It would also be useful to label the run with something more meaningful than 'tannit' for outsiders -- 64 bit ubuntu linux for instance. Thanks very much, Laura Creighton From anto.cuni at gmail.com Sun Feb 27 18:52:44 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Sun, 27 Feb 2011 18:52:44 +0100 Subject: [pypy-dev] implementing the additional repo migrations In-Reply-To: <4D696FB1.5020100@gmx.de> References: <1298669776.21626.20.camel@Klappe2> <201102260231.p1Q2VVOU018324@theraft.openend.se> <201102261025.46282.jacob@openend.se> <201102260951.p1Q9pYQ9024873@theraft.openend.se> <201102260959.p1Q9x9jT025385@theraft.openend.se> <4D696FB1.5020100@gmx.de> Message-ID: +1 on keeping the pdf files. It happens quite often not to have all the needed latex packages On 2/26/11, Carl Friedrich Bolz wrote: > On 02/26/2011 01:03 PM, Armin Rigo wrote: >> Hi Laura, >> >> On Sat, Feb 26, 2011 at 10:59 AM, Laura Creighton wrote: >>> I don't care about the old versions of binary files. >> >> That was the only thing we talked about -- as far as I understood, it >> was never suggested that we should stop tracking revisions of .txt or >> .tex files. I don't know the BigfilesExtension either, but it looks >> to me like we can achieve some more precise result manually. >> Something along the lines of: the .pdf's built from .tex's are not >> checked in, but they are in some standardized place on >> http://pypy.org, where we can fetch them, update them (via ssh), or >> point people to (via their url). This can be easily done with a >> script independent from Mercurial. (The point is of course that >> tracking revisions is a bit useless, because we can always go back in >> time and re-run latex2pdf.) > > Not necessarily, it's always possible that whatever latex packages were > needed to compile the pdf are no longer around or a big hassle to > install. This can make regeneration impractical. So I am in favor of > keeping the PDFs in the repo. > > Carl Friedrich > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From tobami at googlemail.com Sun Feb 27 19:36:52 2011 From: tobami at googlemail.com (Miquel Torres) Date: Sun, 27 Feb 2011 19:36:52 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: <201102271538.p1RFc3rG020721@theraft.openend.se> References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> Message-ID: Hi Laura, you bring up good points, however, it is not as straight forward as it seems. >I see, under a section > called 'latest results' a list of all the recent times when we became > slower. ?It's thus a 'recent problems' page -- we have actually improved > in recent times in many areas, and nowhere is that shown. Well, it really is a list of the latest results. The problem is that speed.pypy.org is foremost a tool to help in development. As such, the logic behind the "latest results" list is regression oriented, or let us say pessimistic. For example, this revision: http://speed.pypy.org/changes/?rev=42312:392bbf936179 The average change is actually -0.91%, which is actually an improvement, though not an statistical significant one. However, There was a sizeable regression in spitfire_cstringio, +5.21. The "summary" for that revision is then "regression for an individual benchmark". Which is actually what developers need to know: they should check whether that revision really introduced a real regression in performance. Now, it is true that since that new main page was introduced the impression it has given is one of regressions, mostly. But if you look at all the graphs together 8http://speed.pypy.org/timeline), you see that in the last weeks there has been a slight upwards (worse performance) trend in many benchmarks. So I think Codespeed has done the right thing! Note that a week ago I did up the threshold from 3% to 4% changes That said, I do understand where you are coming from. I would point outsiders though directly to http://speed.pypy.org/comparison/ So what can some body think about what could be changed or added so that the main page doesn't give a negative impression to the non-developer? Something I could think of is to add, above the results list, a plot showing the overall trend over the last 2 or 3 months. What do you think? > The other thing I want is for the graphs you get, for instance with > http://speed.pypy.org/changes/?rev=42312:392bbf936179&exe=%203&env=tannit > to have, in addition to the selection button beside: 'result for revision' > an actual label that says 'build 42312:392b' or something that you > can select with your mouse and use to paste into things like this mail > article. I think to the right of the changes table there is a box with info for the revision, with a text field you can select and copy. Isn't that what you want? > article. ?It would also be useful to label the run with something more > meaningful than 'tannit' for outsiders -- 64 bit ubuntu linux ?for instance. Agreed. I would also rather call the machines/environments like that. 2011/2/27 Laura Creighton : > > ?sorry Miguel.> > > I'd like to change what is displayed on the speed.python.org frontpage. > Right now, as I look at http://speed.pypy.org/ ?I see, under a section > called 'latest results' a list of all the recent times when we became > slower. ?It's thus a 'recent problems' page -- we have actually improved > in recent times in many areas, and nowhere is that shown. ?As we go off > to PyCON, which is March 9-17, I intend to mention how great PyPy is, > and that you can see it for yourself at speed.pypy.org. ?Thus, without > lying, I would like it if the first impression of PyPy's speed that > people got when looking at the site was 'we're getting faster'. ?Do you > think you could change the front page so that what was displayed was > more balanced with respect to good news and bad news? > > I realise that there is nothing you can do if we make a recent build > that slows everything down, but for instance in build 42312:392b (Feb 26) > we have improvments which are not shown on the main page. ?I actually > think that the _trend_ ?is a more useful thing to display on the front page, > though that might be because it is so green right now. :-) > > The other thing I want is for the graphs you get, for instance with > http://speed.pypy.org/changes/?rev=42312:392bbf936179&exe=%203&env=tannit > to have, in addition to the selection button beside: 'result for revision' > an actual label that says 'build 42312:392b' or something that you > can select with your mouse and use to paste into things like this mail > article. ?It would also be useful to label the run with something more > meaningful than 'tannit' for outsiders -- 64 bit ubuntu linux ?for instance. > > Thanks very much, > Laura Creighton > From lac at openend.se Sun Feb 27 20:11:39 2011 From: lac at openend.se (Laura Creighton) Date: Sun, 27 Feb 2011 20:11:39 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: Message from Miquel Torres of "Sun, 27 Feb 2011 19:36:52 +0100." References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> Message-ID: <201102271911.p1RJBdOg007318@theraft.openend.se> In a message of Sun, 27 Feb 2011 19:36:52 +0100, Miquel Torres writes: >Hi Laura, > >you bring up good points, however, it is not as straight forward as it se >em= >Well, it really is a list of the latest results. The problem is that >speed.pypy.org is foremost a tool to help in development. As such, the >logic behind the "latest results" list is regression oriented, or let >us say pessimistic. > >For example, this revision: >http://speed.pypy.org/changes/?rev=3D42312:392bbf936179 > >The average change is actually -0.91%, which is actually an >improvement, though not an statistical significant one. However, There >was a sizeable regression in spitfire_cstringio, +5.21. The "summary" >for that revision is then "regression for an individual benchmark". >Which is actually what developers need to know: they should check >whether that revision really introduced a real regression in >performance. I understand this. I just don't think that this should be on our front page. Developers can get used to looking anywhere, and indeed I never use the front page for looking at anything important -- I am always looking at the complete stats for any runs. I think that speed.pypy.org -- the front page -- should primarily be of use to people who want to find out if pypy is good for them, people who want to convince their bosses that they should switch, and people who just want to cheer us on and add to the general warm feeling about pypy. I see it as the prime tool in the world domination project. >That said, I do understand where you are coming from. I would point >outsiders though directly to http://speed.pypy.org/comparison/ This is the wrong way to do things from a usability point of view. What the casual person wants is not a way to dig down and get information, but something already packaged for them which already tells them the main story. Then they can dig down if they actually care. This is what Steve Krug calls the 'Don't Make Me Think' principle. >So what can some body think about what could be changed or added so >that the main page doesn't give a negative impression to the >non-developer? > >Something I could think of is to add, above the results list, a plot >showing the overall trend over the last 2 or 3 months. What do you >think? I think this would be a very nice thing to have as the home page for speed.pypy.org Then the page that we have now could be called something like http://speed.pypy.org/regressions or something. >> The other thing I want is for the graphs you get, for instance with >> http://speed.pypy.org/changes/?rev=3D42312:392bbf936179&exe=3D%203&env= >3D= >tannit >> to have, in addition to the selection button beside: 'result for revisi >on= >' >> an actual label that says 'build 42312:392b' or something that you >> can select with your mouse and use to paste into things like this mail >> article. > >I think to the right of the changes table there is a box with info for >the revision, with a text field you can select and copy. Isn't that >what you want? Looking at http://speed.pypy.org/changes/?rev=42312:392bbf936179&exe=%203&env=tannit There is nothing to the right of the table. To the right of the label that says: Results for revision is a box that you can use to select different builds to look at. The text in this box is not selectable; you cannot paste it anywhere. To the right of that is a link that says 'Permalink'. As far as I can tell clicking it causes the page to refresh and nothing more. This is with iceweasel 3.5.16 (which is debian's repackage of firefox 3.5.16) Laura From tobami at googlemail.com Sun Feb 27 20:22:57 2011 From: tobami at googlemail.com (Miquel Torres) Date: Sun, 27 Feb 2011 20:22:57 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: <201102271911.p1RJBdOg007318@theraft.openend.se> References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102271911.p1RJBdOg007318@theraft.openend.se> Message-ID: Ok, I'll try that then. > There is nothing to the right of the table. What is your screen width? The Revision info box should be below the table if your width is small. 2011/2/27 Laura Creighton : > In a message of Sun, 27 Feb 2011 19:36:52 +0100, Miquel Torres writes: >>Hi Laura, >> >>you bring up good points, however, it is not as straight forward as it se >>em= > >>Well, it really is a list of the latest results. The problem is that >>speed.pypy.org is foremost a tool to help in development. As such, the >>logic behind the "latest results" list is regression oriented, or let >>us say pessimistic. >> >>For example, this revision: >>http://speed.pypy.org/changes/?rev=3D42312:392bbf936179 >> >>The average change is actually -0.91%, which is actually an >>improvement, though not an statistical significant one. However, There >>was a sizeable regression in spitfire_cstringio, +5.21. The "summary" >>for that revision is then "regression for an individual benchmark". >>Which is actually what developers need to know: they should check >>whether that revision really introduced a real regression in >>performance. > > I understand this. ?I just don't think that this should be on our > front page. ? Developers can get used to looking anywhere, and indeed > I never use the front page for looking at anything important -- I am > always looking at the complete stats for any runs. > > I think that speed.pypy.org -- the front page -- should primarily be > of use to people who want to find out if pypy is good for them, people > who want to convince their bosses that they should switch, and people > who just want to cheer us on and add to the general warm feeling > about pypy. ?I see it as the prime tool in the world domination > project. > >>That said, I do understand where you are coming from. I would point >>outsiders though directly to http://speed.pypy.org/comparison/ > > This is the wrong way to do things from a usability point of view. > What the casual person wants is not a way to dig down and get information, > but something already packaged for them which already tells them > the main story. ?Then they can dig down if they actually care. ?This is > what Steve Krug calls the 'Don't Make Me Think' principle. > >>So what can some body think about what could be changed or added so >>that the main page doesn't give a negative impression to the >>non-developer? >> >>Something I could think of is to add, above the results list, a plot >>showing the overall trend over the last 2 or 3 months. What do you >>think? > > I think this would be a very nice thing to have as the home page for > speed.pypy.org ?Then the page that we have now could be called something > like http://speed.pypy.org/regressions or something. > >>> The other thing I want is for the graphs you get, for instance with >>> http://speed.pypy.org/changes/?rev=3D42312:392bbf936179&exe=3D%203&env= >>3D= >>tannit >>> to have, in addition to the selection button beside: 'result for revisi >>on= >>' >>> an actual label that says 'build 42312:392b' or something that you >>> can select with your mouse and use to paste into things like this mail >>> article. >> >>I think to the right of the changes table there is a box with info for >>the revision, with a text field you can select and copy. Isn't that >>what you want? > > Looking at http://speed.pypy.org/changes/?rev=42312:392bbf936179&exe=%203&env=tannit > > There is nothing to the right of the table. ?To the right of the label > that says: Results for revision ?is a box that you can use to select different > builds to look at. ?The text in this box is not selectable; you cannot > paste it anywhere. ?To the right of that is a link that says 'Permalink'. > As far as I can tell clicking it causes the page to refresh and nothing > more. > > This is with iceweasel 3.5.16 (which is debian's repackage of firefox > 3.5.16) > > Laura > > From fijall at gmail.com Sun Feb 27 20:57:33 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 27 Feb 2011 21:57:33 +0200 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102271911.p1RJBdOg007318@theraft.openend.se> Message-ID: Hey. Just my 5 cents. It would be cool if default view has a down-scaled version of comparison against CPython. I can look anywhere for recent changes. Also the recent changes as they're now are not very informative and I don't use them at all. They stick around, so I don't know if they're new or old. I'm also as interested in good as in bad changes. Simply this: http://speed.pypy.org/changes/ is way more informative. Can we either just remove the red recent changes for now or simply put a vs cpython, scaled down graph there? At least for pycon this seems like a better way to go. Cheers, fijal PS. Miquel, don't get me wrong, I think you're doing an awesome job, the speed website itself was a huge step forward for us. From lac at openend.se Sun Feb 27 21:04:36 2011 From: lac at openend.se (Laura Creighton) Date: Sun, 27 Feb 2011 21:04:36 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: Message from Miquel Torres of "Sun, 27 Feb 2011 20:22:57 +0100." References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102271911.p1RJBdOg007318@theraft.openend.se> Message-ID: <201102272004.p1RK4aeB011714@theraft.openend.se> In a message of Sun, 27 Feb 2011 20:22:57 +0100, Miquel Torres writes: >Ok, I'll try that then. > >> There is nothing to the right of the table. >What is your screen width? >The Revision info box should be below the table if your width is small. xpdyinfo says dimensions: 1024x768 pixels (270x203 millimeters) Laura From chef at ghum.de Sun Feb 27 21:06:07 2011 From: chef at ghum.de (Massa, Harald Armin) Date: Sun, 27 Feb 2011 21:06:07 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102271911.p1RJBdOg007318@theraft.openend.se> Message-ID: Adding to Maciej's 5cents: > It would be cool if default view has a down-scaled version of > comparison against CPython. I for one second this suggestion. Comparison against cPython is the TLDR of speed.pypy.org; the abstract, the executive-level-information. Harald -- GHUM GmbH Harald Armin Massa Spielberger Stra?e 49 70435 Stuttgart 0173/9409607 Amtsgericht Stuttgart, HRB 734971 - persuadere. et programmare From lac at openend.se Sun Feb 27 21:08:37 2011 From: lac at openend.se (Laura Creighton) Date: Sun, 27 Feb 2011 21:08:37 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: Message from Maciej Fijalkowski of "Sun, 27 Feb 2011 21:57:33 +0200." References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102271911.p1RJBdOg007318@theraft.openend.se> Message-ID: <201102272008.p1RK8bh7012202@theraft.openend.se> In a message of Sun, 27 Feb 2011 21:57:33 +0200, Maciej Fijalkowski writes: >Hey. > >Just my 5 cents. > >It would be cool if default view has a down-scaled version of >comparison against CPython. I can look anywhere for recent changes. >Also the recent changes as they're now are not very informative and I >don't use them at all. They stick around, so I don't know if they're >new or old. I'm also as interested in good as in bad changes. Simply >this: http://speed.pypy.org/changes/ is way more informative. > >Can we either just remove the red recent changes for now or simply put >a vs cpython, scaled down graph there? > >At least for pycon this seems like a better way to go. > >Cheers, >fijal > >PS. Miquel, don't get me wrong, I think you're doing an awesome job, >the speed website itself was a huge step forward for us. This sounds good to me as well, and I too don't want Miquel to think that I am ungrateful for all his hard work. The site is really good for us, and thank you. Laura From tobami at googlemail.com Sun Feb 27 23:00:38 2011 From: tobami at googlemail.com (Miquel Torres) Date: Sun, 27 Feb 2011 23:00:38 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: <201102272008.p1RK8bh7012202@theraft.openend.se> References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102271911.p1RJBdOg007318@theraft.openend.se> <201102272008.p1RK8bh7012202@theraft.openend.se> Message-ID: Hi all, hey, it's not like I crumble at the sound of the slightest criticism ;-) Open Source is about this. Ideas, feedback, etc. Only then can we get better! I think I can draw two conclusions: - For Pycon, and maybe permanently, a general comparison against CPython or an overall timeline would be better. I see I have about 1 week to come up with something. - If Maciej, who obviously is a developer, doesn't think that the report, or "recent results" feature is useful, that means that at the very least it needs tweaking. I will think and experiment with different possibilities. For one I will increase the threshold again, to 5 o 6% so that there is a higher signal to noise ratio. I can also imagine showing two numbers: one for the biggest improvement and one for the biggest regression. And maybe the average. We'll see. If anyone has a clear idea of how that could become more useful please share. Cheers, Miquel 2011/2/27 Laura Creighton : > In a message of Sun, 27 Feb 2011 21:57:33 +0200, Maciej Fijalkowski writes: >>Hey. >> >>Just my 5 cents. >> >>It would be cool if default view has a down-scaled version of >>comparison against CPython. I can look anywhere for recent changes. >>Also the recent changes as they're now are not very informative and I >>don't use them at all. They stick around, so I don't know if they're >>new or old. I'm also as interested in good as in bad changes. Simply >>this: http://speed.pypy.org/changes/ is way more informative. >> >>Can we either just remove the red recent changes for now or simply put >>a vs cpython, scaled down graph there? >> >>At least for pycon this seems like a better way to go. >> >>Cheers, >>fijal >> >>PS. Miquel, don't get me wrong, I think you're doing an awesome job, >>the speed website itself was a huge step forward for us. > > This sounds good to me as well, and I too don't want Miquel to think > that I am ungrateful for all his hard work. ?The site is really good > for us, and thank you. > > Laura > From fijall at gmail.com Mon Feb 28 07:04:47 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 28 Feb 2011 08:04:47 +0200 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102271911.p1RJBdOg007318@theraft.openend.se> <201102272008.p1RK8bh7012202@theraft.openend.se> Message-ID: On Mon, Feb 28, 2011 at 12:00 AM, Miquel Torres wrote: > Hi all, > > hey, it's not like I crumble at the sound of the slightest criticism ;-) > Open Source is about this. Ideas, feedback, etc. Only then can we get better! > > I think I can draw two conclusions:"I > - For Pycon, and maybe permanently, a general comparison against > CPython or an overall timeline would be better. I see I have about 1 > week to come up with something. > - If Maciej, who obviously is a developer, doesn't think that the > report, or "recent results" feature is useful, that means that at the > very least it needs tweaking. I will think and experiment with > different possibilities. For one I will increase the threshold again, > to 5 o 6% so that there is a higher signal to noise ratio. I can also > imagine showing two numbers: one for the biggest improvement and one > for the biggest regression. And maybe the average. We'll see. If > anyone has a clear idea of how that could become more useful please > share. My biggest issue is with the fact that they show up and just stay. This means there is no way to distinguish between "I've seen this" or "I didn't see this" which make it useless. > > Cheers, > Miquel > > > 2011/2/27 Laura Creighton : >> In a message of Sun, 27 Feb 2011 21:57:33 +0200, Maciej Fijalkowski writes: >>>Hey. >>> >>>Just my 5 cents. >>> >>>It would be cool if default view has a down-scaled version of >>>comparison against CPython. I can look anywhere for recent changes. >>>Also the recent changes as they're now are not very informative and I >>>don't use them at all. They stick around, so I don't know if they're >>>new or old. I'm also as interested in good as in bad changes. Simply >>>this: http://speed.pypy.org/changes/ is way more informative. >>> >>>Can we either just remove the red recent changes for now or simply put >>>a vs cpython, scaled down graph there? >>> >>>At least for pycon this seems like a better way to go. >>> >>>Cheers, >>>fijal >>> >>>PS. Miquel, don't get me wrong, I think you're doing an awesome job, >>>the speed website itself was a huge step forward for us. >> >> This sounds good to me as well, and I too don't want Miquel to think >> that I am ungrateful for all his hard work. ?The site is really good >> for us, and thank you. >> >> Laura >> > From tobami at googlemail.com Mon Feb 28 08:33:16 2011 From: tobami at googlemail.com (Miquel Torres) Date: Mon, 28 Feb 2011 08:33:16 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102271911.p1RJBdOg007318@theraft.openend.se> <201102272008.p1RK8bh7012202@theraft.openend.se> Message-ID: Well, that use-case is covered by the RSS feed ;-) 2011/2/28 Maciej Fijalkowski : > On Mon, Feb 28, 2011 at 12:00 AM, Miquel Torres wrote: >> Hi all, >> >> hey, it's not like I crumble at the sound of the slightest criticism ;-) >> Open Source is about this. Ideas, feedback, etc. Only then can we get better! >> >> I think I can draw two conclusions:"I >> - For Pycon, and maybe permanently, a general comparison against >> CPython or an overall timeline would be better. I see I have about 1 >> week to come up with something. >> - If Maciej, who obviously is a developer, doesn't think that the >> report, or "recent results" feature is useful, that means that at the >> very least it needs tweaking. I will think and experiment with >> different possibilities. For one I will increase the threshold again, >> to 5 o 6% so that there is a higher signal to noise ratio. I can also >> imagine showing two numbers: one for the biggest improvement and one >> for the biggest regression. And maybe the average. We'll see. If >> anyone has a clear idea of how that could become more useful please >> share. > > My biggest issue is with the fact that they show up and just stay. > This means there is no way to distinguish between "I've seen this" or > "I didn't see this" which make it useless. > > >> >> Cheers, >> Miquel >> >> >> 2011/2/27 Laura Creighton : >>> In a message of Sun, 27 Feb 2011 21:57:33 +0200, Maciej Fijalkowski writes: >>>>Hey. >>>> >>>>Just my 5 cents. >>>> >>>>It would be cool if default view has a down-scaled version of >>>>comparison against CPython. I can look anywhere for recent changes. >>>>Also the recent changes as they're now are not very informative and I >>>>don't use them at all. They stick around, so I don't know if they're >>>>new or old. I'm also as interested in good as in bad changes. Simply >>>>this: http://speed.pypy.org/changes/ is way more informative. >>>> >>>>Can we either just remove the red recent changes for now or simply put >>>>a vs cpython, scaled down graph there? >>>> >>>>At least for pycon this seems like a better way to go. >>>> >>>>Cheers, >>>>fijal >>>> >>>>PS. Miquel, don't get me wrong, I think you're doing an awesome job, >>>>the speed website itself was a huge step forward for us. >>> >>> This sounds good to me as well, and I too don't want Miquel to think >>> that I am ungrateful for all his hard work. ?The site is really good >>> for us, and thank you. >>> >>> Laura >>> >> > From lac at openend.se Mon Feb 28 08:44:15 2011 From: lac at openend.se (Laura Creighton) Date: Mon, 28 Feb 2011 08:44:15 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: Message from Miquel Torres of "Sun, 27 Feb 2011 19:36:52 +0100." References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> Message-ID: <201102280744.p1S7iFh7009804@theraft.openend.se> Also, can you change every usage of the word 'less' on the axis of the graphs to 'smaller'. As far as I can tell, every time it is an incorrect and ungramatically usage of English. Sometimes the legend is 'seconds - less is better'. But seconds are inherantly countable, so it is incorrect English to say 'there are less seconds' -- rather that there are *fewer* seconds. Or you could change that legend to read 'time in seconds' -- because time is considered uncountable, so you have more or less time, not more or fewer time. Smaller would also work here. Ratios, however, such as on http://speed.pypy.org/comparison/ cannot be 'more or less' or 'more or fewer'. They have to be greater (or larger) or smaller. Thus we need 'smaller' here. Laura From dcolish at gmail.com Mon Feb 28 18:51:22 2011 From: dcolish at gmail.com (Dan Colish) Date: Mon, 28 Feb 2011 09:51:22 -0800 Subject: [pypy-dev] _scproxy module and tweaks for osx Message-ID: <237B76A5746A4108B33E98626E152D15@gmail.com> I've been playing with a few ideas on OSX to get the build to be more stable. Attached is a patch that does this, but it is far from complete. There are still bugs in the way that frameworks are used when the final Makefile is created. In addition, there are problems with libintl that I'd like to be a little more pragmatic about. The _scproxy module itself works with the framework hardcoded fixes. Anyway, I'd appreciate some feedback and also so thoughts on how best to deal with various build environments on OSX. -- Dan -------------- next part -------------- A non-text attachment was scrubbed... Name: stable-scproxy.diff Type: application/octet-stream Size: 16448 bytes Desc: not available Url : http://codespeak.net/pipermail/pypy-dev/attachments/20110228/7909ca99/attachment-0001.obj From holger at merlinux.eu Tue Mar 1 19:42:57 2011 From: holger at merlinux.eu (holger krekel) Date: Tue, 1 Mar 2011 18:42:57 +0000 Subject: [pypy-dev] extradoc moved / pypy-z moved / new pypy groups Message-ID: <20110301184257.GD16231@merlinux.eu> Hi all, I just turned http://codespeak.net/svn/pypy/extradoc as well as the internal pypy-z repository READ-ONLY. Thanks to Ronny's conversion work we now have these repositories at bitbucket.org: http://bitbucket.org/pypy/extradoc talks, sprintinfo, the source of pypy.org website and http://bitbucket.org/pypy/z (PRIVATE) internal contracts/docs, available to long-time contributors I created a new pypy group on bitbucket, the "pypy:committers" group which has general read/write access to all public pypy repositories. It contains the people formerly listed individually for the pypy repo. New committers should be added to that pypy:committers group so they get access to all pypy repos. There also is a pypy:z group which also gets write/admin acccess to all repos but additionally can read/write the pypy/z one. Next in line for conversion are: pyrepl -> pypy/pyrepl (discussed with Michael Hudson, the primary author already) testrunner -> to be flatly inlined into pypy repo and then the remaining parts of pypy/lang After that there should be nothing left from PyPy that would require codespeak's svn. best, holger From fijall at gmail.com Wed Mar 2 07:45:12 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 2 Mar 2011 08:45:12 +0200 Subject: [pypy-dev] extradoc moved / pypy-z moved / new pypy groups In-Reply-To: <20110301184257.GD16231@merlinux.eu> References: <20110301184257.GD16231@merlinux.eu> Message-ID: On Tue, Mar 1, 2011 at 8:42 PM, holger krekel wrote: > Hi all, > > I just turned > > ? ?http://codespeak.net/svn/pypy/extradoc > > as well as the internal pypy-z repository READ-ONLY. > > Thanks to Ronny's conversion work we now have these repositories at bitbucket.org: > > ? ?http://bitbucket.org/pypy/extradoc > ? ? ? ?talks, sprintinfo, the source of pypy.org website > > and > > ? ?http://bitbucket.org/pypy/z (PRIVATE) > ? ? ? ?internal contracts/docs, available to long-time contributors > > I created a new pypy group on bitbucket, the "pypy:committers" group > which has general read/write access to all public pypy repositories. It > contains the people formerly listed individually for the pypy repo. ?New > committers should be added to that pypy:committers group so they get > access to all pypy repos. ?There also is a pypy:z group which also gets > write/admin acccess to all repos but additionally can read/write the > pypy/z one. > > Next in line for conversion are: > > ? ?pyrepl -> pypy/pyrepl ?(discussed with Michael Hudson, the primary author already) > ? ?testrunner -> to be flatly inlined into pypy repo > ? ?and then the remaining parts of pypy/lang What about sqlite3? > > After that there should be nothing left from PyPy that would > require codespeak's svn. > > best, > holger > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From alex.gaynor at gmail.com Wed Mar 2 07:47:51 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Wed, 2 Mar 2011 01:47:51 -0500 Subject: [pypy-dev] extradoc moved / pypy-z moved / new pypy groups In-Reply-To: References: <20110301184257.GD16231@merlinux.eu> Message-ID: On Wed, Mar 2, 2011 at 1:45 AM, Maciej Fijalkowski wrote: > On Tue, Mar 1, 2011 at 8:42 PM, holger krekel wrote: > > Hi all, > > > > I just turned > > > > http://codespeak.net/svn/pypy/extradoc > > > > as well as the internal pypy-z repository READ-ONLY. > > > > Thanks to Ronny's conversion work we now have these repositories at > bitbucket.org: > > > > http://bitbucket.org/pypy/extradoc > > talks, sprintinfo, the source of pypy.org website > > > > and > > > > http://bitbucket.org/pypy/z (PRIVATE) > > internal contracts/docs, available to long-time contributors > > > > I created a new pypy group on bitbucket, the "pypy:committers" group > > which has general read/write access to all public pypy repositories. It > > contains the people formerly listed individually for the pypy repo. New > > committers should be added to that pypy:committers group so they get > > access to all pypy repos. There also is a pypy:z group which also gets > > write/admin acccess to all repos but additionally can read/write the > > pypy/z one. > > > > Next in line for conversion are: > > > > pyrepl -> pypy/pyrepl (discussed with Michael Hudson, the primary > author already) > > testrunner -> to be flatly inlined into pypy repo > > and then the remaining parts of pypy/lang > > What about sqlite3? > > > > > After that there should be nothing left from PyPy that would > > require codespeak's svn. > > > > best, > > holger > > > > _______________________________________________ > > pypy-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/pypy-dev > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > Amaury already moved it into the main PyPy repo I think. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110302/6597762c/attachment.htm From cfbolz at gmx.de Wed Mar 2 11:25:52 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Wed, 02 Mar 2011 11:25:52 +0100 Subject: [pypy-dev] extradoc moved / pypy-z moved / new pypy groups In-Reply-To: <20110301184257.GD16231@merlinux.eu> References: <20110301184257.GD16231@merlinux.eu> Message-ID: <4D6E1B30.6060504@gmx.de> Hi Holger, On 03/01/2011 07:42 PM, holger krekel wrote: > I just turned > > http://codespeak.net/svn/pypy/extradoc > > as well as the internal pypy-z repository READ-ONLY. > > Thanks to Ronny's conversion work we now have these repositories at bitbucket.org: > > http://bitbucket.org/pypy/extradoc > talks, sprintinfo, the source of pypy.org website > > and > > http://bitbucket.org/pypy/z (PRIVATE) > internal contracts/docs, available to long-time contributors > > I created a new pypy group on bitbucket, the "pypy:committers" group > which has general read/write access to all public pypy repositories. It > contains the people formerly listed individually for the pypy repo. New > committers should be added to that pypy:committers group so they get > access to all pypy repos. There also is a pypy:z group which also gets > write/admin acccess to all repos but additionally can read/write the > pypy/z one. Makes perfect sense to me. > Next in line for conversion are: > > pyrepl -> pypy/pyrepl (discussed with Michael Hudson, the primary author already) > testrunner -> to be flatly inlined into pypy repo > and then the remaining parts of pypy/lang > > After that there should be nothing left from PyPy that would > require codespeak's svn. There is also this: http://codespeak.net/svn/pypy/benchmarks/ sorry for not thinking of it earlier. Thanks for the good work to Ronny and you, Carl Friedrich From lac at openend.se Thu Mar 3 17:34:31 2011 From: lac at openend.se (Laura Creighton) Date: Thu, 3 Mar 2011 17:34:31 +0100 Subject: [pypy-dev] we need to fix the links on our blog posts. Message-ID: <201103031634.p23GYVV6000421@theraft.openend.se> http://morepypy.blogspot.com/2008/06/hi-all-some-news-from-jit-front.html the link to the pyrolog paper is broken. This works today, but needs to have its mimetype set http://codespeak.net/svn/user/cfbolz/jitpl/paper/iclp08/jit_iclp08.pdf But I don't know where Carl Friedrich's svn dir is migrating to from codespeak, so this isn't a permanent fix. This paper really should be in our papers, in extradoc, no? I cannot find it there. Laura From lac at openend.se Thu Mar 3 17:37:01 2011 From: lac at openend.se (Laura Creighton) Date: Thu, 03 Mar 2011 17:37:01 +0100 Subject: [pypy-dev] we need to fix the links on our blog posts. In-Reply-To: Message from Laura Creighton of "Thu, 03 Mar 2011 17:34:31 +0100." <201103031634.p23GYVV6000421@theraft.openend.se> References: <201103031634.p23GYVV6000421@theraft.openend.se> Message-ID: <201103031637.p23Gb1Fp000896@theraft.openend.se> Do we need dcolish's bouncer to work for what used to be people's svn directories on codespeak? Laura From arigo at tunes.org Thu Mar 3 18:01:05 2011 From: arigo at tunes.org (Armin Rigo) Date: Thu, 3 Mar 2011 09:01:05 -0800 Subject: [pypy-dev] we need to fix the links on our blog posts. In-Reply-To: <201103031637.p23Gb1Fp000896@theraft.openend.se> References: <201103031634.p23GYVV6000421@theraft.openend.se> <201103031637.p23Gb1Fp000896@theraft.openend.se> Message-ID: Hi Laura, On Thu, Mar 3, 2011 at 8:37 AM, Laura Creighton wrote: > Do we need dcolish's bouncer to work for what used to be people's svn > directories on codespeak? Actually, does bitbucket serve files straight from repositories, like http://codespeak.net/svn/some/path did? I tried a bit, but seem to only get links to specific revisions of files, not "whatever the latest revision is". A bient?t, Armin. From anto.cuni at gmail.com Thu Mar 3 18:11:08 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu, 3 Mar 2011 18:11:08 +0100 Subject: [pypy-dev] we need to fix the links on our blog posts. In-Reply-To: References: <201103031634.p23GYVV6000421@theraft.openend.se> <201103031637.p23Gb1Fp000896@theraft.openend.se> Message-ID: you can get a link to "whatever the latest revision is" on bitbucket by manually substituting the mercurial id with "default" inside the link. On 3/3/11, Armin Rigo wrote: > Hi Laura, > > On Thu, Mar 3, 2011 at 8:37 AM, Laura Creighton wrote: >> Do we need dcolish's bouncer to work for what used to be people's svn >> directories on codespeak? > > Actually, does bitbucket serve files straight from repositories, like > http://codespeak.net/svn/some/path did? I tried a bit, but seem to > only get links to specific revisions of files, not "whatever the > latest revision is". > > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From scott+pypy-dev at scottdial.com Thu Mar 3 18:19:15 2011 From: scott+pypy-dev at scottdial.com (Scott Dial) Date: Thu, 03 Mar 2011 12:19:15 -0500 Subject: [pypy-dev] we need to fix the links on our blog posts. In-Reply-To: References: <201103031634.p23GYVV6000421@theraft.openend.se> <201103031637.p23Gb1Fp000896@theraft.openend.se> Message-ID: <4D6FCD93.5050203@scottdial.com> On 3/3/2011 12:01 PM, Armin Rigo wrote: > On Thu, Mar 3, 2011 at 8:37 AM, Laura Creighton wrote: >> Do we need dcolish's bouncer to work for what used to be people's svn >> directories on codespeak? > > Actually, does bitbucket serve files straight from repositories, like > http://codespeak.net/svn/some/path did? I tried a bit, but seem to > only get links to specific revisions of files, not "whatever the > latest revision is". AFAICT, you can substitute "tip" for the revision id in the URL and that will track the latest. For example, you can browse to a file and copy the raw link: https://bitbucket.org/pypy/pypy/raw/b9887cd8aab0/README And turn that into: https://bitbucket.org/pypy/pypy/raw/tip/README -Scott -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From lac at openend.se Fri Mar 4 19:45:30 2011 From: lac at openend.se (Laura Creighton) Date: Fri, 04 Mar 2011 19:45:30 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: Message from Miquel Torres of "Fri, 04 Mar 2011 19:34:15 +0100." References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102280744.p1S7iFh7009804@theraft.openend.se> Message-ID: <201103041845.p24IjUDR015867@theraft.openend.se> In a message of Fri, 04 Mar 2011 19:34:15 +0100, Miquel Torres writes: >Hi, > >just a quick note to Laura and the rest to let you know that I am >working on the issues you raised, and expect to have everything nailed >down this weekend, just in time for PyCon. > >Miquel Thank you very much. Laura From ademan555 at gmail.com Fri Mar 4 03:02:20 2011 From: ademan555 at gmail.com (Dan Roberts) Date: Thu, 3 Mar 2011 18:02:20 -0800 Subject: [pypy-dev] Yelp RSVP In-Reply-To: References: Message-ID: Apparently I missed the memo about RSVPing to the Yelp talk, and we're having trouble getting in the door, assuming the situation can be rectified, can someone help out? Cheers, Dan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110303/7cf2fb8b/attachment.htm From tbaldridge at gmail.com Fri Mar 4 14:14:43 2011 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Fri, 4 Mar 2011 07:14:43 -0600 Subject: [pypy-dev] EBNF translation error Message-ID: I hope this is the right place for this question. I'm trying to write a simple JIT-enabled interpreter. However, I'm hitting a odd error when it comes to translation. I've copied the JavaScript example parser almost verbatim, but here's my issues: First of all, here's parser (almost 100% like the JS example): from pypy.rlib.parsing.ebnfparse import parse_ebnf, make_parse_function from pypy.rlib.parsing.parsing import ParseError, Rule import py import sys GFILE = py.magic.autopath().dirpath().join("grammar.txt") try: t = GFILE.read(mode="U") regexs, rules, ToAST = parse_ebnf(t) except ParseError, e: print e.nice_error_message(filename=str(GFILE), source=t) raise parsef = make_parse_function(regexs, rules, eof=True) def parse(code): t = parsef(code) return ToAST().transform(t) and my grammar: STRING: "\\"[^\\\\"]*\\""; SYMBOL: "[A-Za-z+-_*<>]+"; KEYWORD: ":[A-Za-z+-_*<>]+"; INTEGER: "\-?([0-9]+)"; DECIMAL: "\-?(0|[1-9][0-9]*)(\.[0-9]+)?([eE][\+\-]?[0-9]+)?"; IGNORE: " |\n|\t|,"; value: | | | | | | | ; hash: ["{"] (entry [","])* entry ["}"]; vector: ["["] value* ["]"]; entry: STRING [":"] value; sexps: ["("] value+ [")"]; I'm doing the following to compile the code to c: import parse t = Translation(parse.parse) t.annotate([str]) t.rtype() t.compile_c() >>> <---snip----> File "/home/tbaldridge/pypy/pypy/translator/c/genc.py", line 339, in getentrypointptr self._wrapper = new_wrapper(self.entrypoint, self.translator) File "/home/tbaldridge/pypy/pypy/translator/llsupport/wrapper.py", line 57, in new_wrapper r_to = pyobj_repr) File "/home/tbaldridge/pypy/pypy/rpython/rtyper.py", line 931, in convertvar (r_from, r_to)) pypy.rpython.error.TyperError: don't know how to convert from to What am I missing? This seemed so straight-forward.... Thanks, Timothy -- ?One of the main causes of the fall of the Roman Empire was that?lacking zero?they had no way to indicate successful termination of their C programs.? (Robert Firth) From arigo at tunes.org Sat Mar 5 16:42:36 2011 From: arigo at tunes.org (Armin Rigo) Date: Sat, 5 Mar 2011 07:42:36 -0800 Subject: [pypy-dev] Yelp RSVP In-Reply-To: References: Message-ID: Hi Dan, On Thu, Mar 3, 2011 at 6:02 PM, Dan Roberts wrote: > Apparently I missed the memo about RSVPing to the Yelp talk, and we're > having trouble getting in the door, assuming the situation can be rectified, > can someone help out? For (un?)interested readers of this mailing list, the situation was eventually solved. Armin From arigo at tunes.org Sat Mar 5 16:51:44 2011 From: arigo at tunes.org (Armin Rigo) Date: Sat, 5 Mar 2011 07:51:44 -0800 Subject: [pypy-dev] EBNF translation error In-Reply-To: References: Message-ID: Hi Timothy, On Fri, Mar 4, 2011 at 5:14 AM, Timothy Baldridge wrote: > I hope this is the right place for this question. > <---snip----> > pypy.rpython.error.TyperError: don't know how to convert from > to PyObject> Basically, it worked all right, except that you are trying to return from the main translated function an instance of Node, which the translator doesn't know how to convert back to Python level. You can only return simple types, like ints, strings, and so on. In "real usage", i.e. translating a stand-alone program with pypy/translator/goal/translate.py, the main translated function needs anyway to have a specific signature, similar to the main() function in C code: takes a list of strings (argv), and returns an integer (exit code). A bient?t, Armin. From tobami at googlemail.com Fri Mar 4 19:34:15 2011 From: tobami at googlemail.com (Miquel Torres) Date: Fri, 4 Mar 2011 19:34:15 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: <201102280744.p1S7iFh7009804@theraft.openend.se> References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102280744.p1S7iFh7009804@theraft.openend.se> Message-ID: Hi, just a quick note to Laura and the rest to let you know that I am working on the issues you raised, and expect to have everything nailed down this weekend, just in time for PyCon. Miquel 2011/2/28 Laura Creighton : > Also, can you change every usage of the word 'less' on the axis of the > graphs to 'smaller'. ?As far as I can tell, every time it is an > incorrect and ungramatically usage of English. ?Sometimes the legend > is 'seconds - less is better'. ?But seconds are inherantly countable, > so it is incorrect English to say 'there are less seconds' -- rather > that there are *fewer* seconds. ?Or you could change that legend to > read 'time in seconds' -- because time is considered uncountable, so > you have more or less time, not more or fewer time. ?Smaller would also > work here. > > Ratios, however, such as on http://speed.pypy.org/comparison/ > cannot be 'more or less' or 'more or fewer'. ?They have to be greater > (or larger) ?or smaller. ?Thus we need 'smaller' here. > > Laura > > From holger at merlinux.eu Sun Mar 6 20:00:04 2011 From: holger at merlinux.eu (holger krekel) Date: Sun, 6 Mar 2011 19:00:04 +0000 Subject: [pypy-dev] pytest2 branch ready Message-ID: <20110306190004.GU16231@merlinux.eu> Hi all, the pytest2 branch is ready to be merged. The branch updates the inlined pytest version to the pytest-2.0.2 dev5 release candidate. I did some pypy test runs on buildbot and it looks good to me. Anything that speaks against merging now? After merging one can install the "pytest" package indepedently and type "py.test" or use the version that comes inlined with the PyPy source tree and type "python pypy/test_all.py". On a sidenote, i'd actually like to avoid shipping pytest and py lib within the PyPy source tree especially now that there are three extra items "pytest.py", "py" and "_pytest" cluttering up the pypy root. (There are good reasons for this which have not much to do with PyPy though). FWIW I don't plan to do many wild releases of pytest in the foreseeable future :) best, holger From lac at openend.se Mon Mar 7 00:43:38 2011 From: lac at openend.se (Laura Creighton) Date: Mon, 07 Mar 2011 00:43:38 +0100 Subject: [pypy-dev] pytest2 branch ready In-Reply-To: Message from holger krekel of "Sun, 06 Mar 2011 19:00:04 GMT." <20110306190004.GU16231@merlinux.eu> References: <20110306190004.GU16231@merlinux.eu> Message-ID: <201103062343.p26Nhc0m006701@theraft.openend.se> In a message of Sun, 06 Mar 2011 19:00:04 GMT, holger krekel writes: >On a sidenote, i'd actually like to avoid shipping pytest and >py lib within the PyPy source tree especially now that there are >three extra items "pytest.py", "py" and "_pytest" cluttering >up the pypy root. (There are good reasons for this which >have not much to do with PyPy though). FWIW I don't plan to >do many wild releases of pytest in the foreseeable future :) > >best, >holger I think I am confused. If we begin shipping pypy without a py.test that is known to work with it, then some year in the future some poor soul (maybe me) who happens to want to run an old verion of pypy -- say a hypothetical pypy 1.5 that comes with no included py.test -- may find that your latest and greatest py.test does not run the tests. And then it will be a matter of archeology to find out what version of py.test does work with this release. Is there something wrong with this understanding? Laura From benjamin at python.org Mon Mar 7 01:00:00 2011 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 6 Mar 2011 18:00:00 -0600 Subject: [pypy-dev] pytest2 branch ready In-Reply-To: <201103062343.p26Nhc0m006701@theraft.openend.se> References: <20110306190004.GU16231@merlinux.eu> <201103062343.p26Nhc0m006701@theraft.openend.se> Message-ID: 2011/3/6 Laura Creighton : > In a message of Sun, 06 Mar 2011 19:00:04 GMT, holger krekel writes: > > >>On a sidenote, i'd actually like to avoid shipping pytest and >>py lib within the PyPy source tree especially now that there are >>three extra items "pytest.py", "py" and "_pytest" cluttering >>up the pypy root. (There are good reasons for this which >>have not much to do with PyPy though). ?FWIW I don't plan to >>do many wild releases of pytest in the foreseeable future :) >> >>best, >>holger > > I think I am confused. ?If we begin shipping pypy without a py.test > that is known to work with it, then some year in the future some poor > soul (maybe me) who happens to want to run an old verion of pypy -- > say a hypothetical pypy 1.5 that comes with no included py.test -- > may find that your latest and greatest py.test does not run the tests. > And then it will be a matter of archeology to find out what version of > py.test does work with this release. ?Is there something wrong with > this understanding? If you're using an old version of PyPy, there's probably going to be a lot of archeology anyway. I don't see how py.test is any different than another dep like gcc or boehm. -- Regards, Benjamin From arigo at tunes.org Mon Mar 7 01:36:17 2011 From: arigo at tunes.org (Armin Rigo) Date: Sun, 6 Mar 2011 16:36:17 -0800 Subject: [pypy-dev] pytest2 branch ready In-Reply-To: References: <20110306190004.GU16231@merlinux.eu> <201103062343.p26Nhc0m006701@theraft.openend.se> Message-ID: Hi Benjamin, On Sun, Mar 6, 2011 at 4:00 PM, Benjamin Peterson wrote: > If you're using an old version of PyPy, there's probably going to be a > lot of archeology anyway. I don't see how py.test is any different > than another dep like gcc or boehm. No, that's wrong. Any version of PyPy supports any version of gcc within reason, and any version of Boehm as long as it's not a too buggy version. But PyPy from, say, the month of January only works with the version of the py lib that was the one included at that time. Right now I can say "hg up -r xyz" for some xyz from a few months ago and just run py.test in it with no further installation, which is essential from time to time (e.g. to run hg bisect). A bient?t, Armin. From tbaldridge at gmail.com Mon Mar 7 17:48:38 2011 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Mon, 7 Mar 2011 10:48:38 -0600 Subject: [pypy-dev] GIL in non-Python languages Message-ID: I'm in the process of writing a PyPy based Clojure interpreter/JIT. One of the man tenants of Clojure is immutability, and one of the other tenants is extreme co-currency. Starting soon I'd like to start experimenting with co-currency in my interpreter. For those who aren't familiar with Clojure, let me give an example: (def my-agent (agent "Hello")) (send my-agent #(str %1 " World")) Send then takes the current state of my-agent and then calls the given lambda function, passing it the current state of the agent (%1 points to the first argument of the lambda). The lambda simply then concats the two strings together. So here's the deal. My entire interpreter is 100% thread safe. All my variables are either immutable, or local to the given function. But from what I'm hearing on IRC, it sounds like that a GIL will be automatically generated in my interpreter after I run it through PyPy. I know ripping the GIL out of Python is a major deal. But how about my project? If I start with the concept of a GIL-less interpreter, will PyPy get in my way? If so, how can I get it to "back-off" a bit? Thanks, Timothy -- ?One of the main causes of the fall of the Roman Empire was that?lacking zero?they had no way to indicate successful termination of their C programs.? (Robert Firth) From benjamin at python.org Mon Mar 7 17:50:50 2011 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 7 Mar 2011 10:50:50 -0600 Subject: [pypy-dev] GIL in non-Python languages In-Reply-To: References: Message-ID: 2011/3/7 Timothy Baldridge : > I'm in the process of writing a PyPy based Clojure interpreter/JIT. > One of the man tenants of Clojure is immutability, and one of the > other tenants is extreme co-currency. Starting soon I'd like to start > experimenting with co-currency in my interpreter. For those who aren't > familiar with Clojure, let me give an example: > > > (def my-agent (agent "Hello")) > > (send my-agent #(str %1 " World")) > > Send then takes the current state of my-agent and then calls the given > lambda function, passing it the current state of the agent (%1 points > to the first argument of the lambda). The lambda simply then concats > the two strings together. > > So here's the deal. My entire interpreter is 100% thread safe. All my > variables are either immutable, or local to the given function. But > from what I'm hearing on IRC, it sounds like that a GIL will be > automatically generated in my interpreter after I run it through PyPy. > I know ripping the GIL out of Python is a major deal. But how about my > project? If I start with the concept of a GIL-less interpreter, will > PyPy get in my way? If so, how can I get it to "back-off" a bit? One major problem is that RPython its self is not really thread-safe. For example, the gc is very non-concurrent. So, that would have to be fixed. -- Regards, Benjamin From tbaldridge at gmail.com Mon Mar 7 18:08:20 2011 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Mon, 7 Mar 2011 11:08:20 -0600 Subject: [pypy-dev] GIL in non-Python languages In-Reply-To: References: Message-ID: Right, that would be an issue. In this case, I could probably start with a reference counting GC....as I'm not exactly sure that it's possible to get cyclic references with immutable objects. At least not without allot of work...yeah, I'm pretty sure they are impossible to code in Clojure. But someone may correct me there. Anyway, reference counting would get rid of most of the parallel issues with the GC, I could simply use a CAS or atomic instructions to increment/decrement reference counts. Timothy > One major problem is that RPython its self is not really thread-safe. > For example, the gc is very non-concurrent. So, that would have to be > fixed. > > > > -- > Regards, > Benjamin > -- ?One of the main causes of the fall of the Roman Empire was that?lacking zero?they had no way to indicate successful termination of their C programs.? (Robert Firth) From fijall at gmail.com Mon Mar 7 18:39:43 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 7 Mar 2011 09:39:43 -0800 Subject: [pypy-dev] GIL in non-Python languages In-Reply-To: References: Message-ID: On Mon, Mar 7, 2011 at 9:08 AM, Timothy Baldridge wrote: > Right, that would be an issue. In this case, I could probably start > with a reference counting GC....as I'm not exactly sure that it's > possible to get cyclic references with immutable objects. At least not > without allot of work...yeah, I'm pretty sure they are impossible to > code in Clojure. But someone may correct me there. Hey. Refcounting GC would be generated for you, have you chosen to use it, but it'll be very inefficient and probably not thread safe. What you can do *right now* is to use boehm GC which is thread safe, but not very fast (much faster and much better than refcounting though). I don't think there are any other issues that are not thread safe in RPython than GC. Obviously it's like C - if you do something wrong you'll segfault. However, the true answer would be to work on a real GC that thread-friendly and thread-safe. This is work, it's however fun :-) > > Anyway, reference counting would get rid of most of the parallel > issues with the GC, I could simply use a CAS or atomic instructions to > increment/decrement reference counts. > > Timothy > > >> One major problem is that RPython its self is not really thread-safe. >> For example, the gc is very non-concurrent. So, that would have to be >> fixed. >> >> >> >> -- >> Regards, >> Benjamin >> > > > > -- > ?One of the main causes of the fall of the Roman Empire was > that?lacking zero?they had no way to indicate successful termination > of their C programs.? > (Robert Firth) > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From tobami at googlemail.com Tue Mar 8 09:10:32 2011 From: tobami at googlemail.com (Miquel Torres) Date: Tue, 8 Mar 2011 09:10:32 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: <201103041845.p24IjUDR015867@theraft.openend.se> References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102280744.p1S7iFh7009804@theraft.openend.se> <201103041845.p24IjUDR015867@theraft.openend.se> Message-ID: Hi, I finished the changes to the speed.pypy.org home page last night, but alas!, I didn't have time to deploy. I will do it later today and will then ping you back. The extra info provided is really nice as an overview, you will see ;-) 2011/3/4 Laura Creighton : > In a message of Fri, 04 Mar 2011 19:34:15 +0100, Miquel Torres writes: >>Hi, >> >>just a quick note to Laura and the rest to let you know that I am >>working on the issues you raised, and expect to have everything nailed >>down this weekend, just in time for PyCon. >> >>Miquel > > Thank you very much. > > Laura > > From tbaldridge at gmail.com Tue Mar 8 17:12:51 2011 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Tue, 8 Mar 2011 10:12:51 -0600 Subject: [pypy-dev] Basic math ops Message-ID: In my interpreter, I have several primitive wrapper objects: IntObject, FloatObject, etc. Currently I'm planning to create functions for every combination: def add_int_float(i, f): return FloatObject(i.int_value() + f.float_value()) then I'd need some sort of dispatch code that would say if isinstance(arg1, IntObject) and isinstance(arg2, FloatObject): return add_int_float(arg1, arg2) It seems like pypy should have something to do this already. I saw some information on this but that seemed specific to the Python interpreter. What's the recommended procedure on this? Is there a way to tell pypy "this object wraps a single value, so keep it unwrapped if you can", or does pypy figure that out automagically? Thanks, Timothy -- ?One of the main causes of the fall of the Roman Empire was that?lacking zero?they had no way to indicate successful termination of their C programs.? (Robert Firth) From lac at openend.se Tue Mar 8 17:14:52 2011 From: lac at openend.se (Laura Creighton) Date: Tue, 08 Mar 2011 17:14:52 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: Message from Miquel Torres of "Tue, 08 Mar 2011 09:10:32 +0100." References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102280744.p1S7iFh7009804@theraft.openend.se> <201103041845.p24IjUDR015867@theraft.openend.se> Message-ID: <201103081614.p28GEqKT013585@theraft.openend.se> In a message of Tue, 08 Mar 2011 09:10:32 +0100, Miquel Torres writes: >Hi, > >I finished the changes to the speed.pypy.org home page last night, but >alas!, I didn't have time to deploy. I will do it later today and will >then ping you back. > >The extra info provided is really nice as an overview, you will see ;-) > > Ah good. Thank you very much. We spent yesterday afternoon with the Mozilla engineers, and I got to talk to the person who maintains the benchmarks for tracemonkey. He had timelines very much like ours. There is one feature he has that I would like to have. Take a look at the timeline for spectral.norm. There are two spikes there. Mozilla has lines like that too, though mostly it is because their jit decides that the whole benchmark is bogus and optimises out all the code. So it takes 0 time. oops. At any rate, aside from knowing that something went horribly wrong with that rev, you don't really need to know how wrong. And by making the graph display up to that point means that the dots where things really do matter get crammed closer together than would otherwise be the case. So he had a mode where things wehre displayed with an arbitrary value at the bottom (in our coase it would be the top) which he could specify. Then the graph would be replotted, with the outliers off the graph, but making it easier to read the dots for the more normal cases. Any chance we could do that too? Laura From fijall at gmail.com Tue Mar 8 17:29:02 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 8 Mar 2011 08:29:02 -0800 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: <201103081614.p28GEqKT013585@theraft.openend.se> References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102280744.p1S7iFh7009804@theraft.openend.se> <201103041845.p24IjUDR015867@theraft.openend.se> <201103081614.p28GEqKT013585@theraft.openend.se> Message-ID: On Tue, Mar 8, 2011 at 8:14 AM, Laura Creighton wrote: > In a message of Tue, 08 Mar 2011 09:10:32 +0100, Miquel Torres writes: >>Hi, >> >>I finished the changes to the speed.pypy.org home page last night, but >>alas!, I didn't have time to deploy. I will do it later today and will >>then ping you back. >> >>The extra info provided is really nice as an overview, you will see ;-) >> >> > > Ah good. ?Thank you very much. ?We spent yesterday afternoon with > the Mozilla engineers, and I got to talk to the person who maintains > the benchmarks for tracemonkey. ?He had timelines very much like ours. > There is one feature he has that I would like to have. ?Take a look > at ? ?the timeline for spectral.norm. ?There are two spikes there. > Mozilla has lines like that too, though mostly it is because their > jit decides that the whole benchmark is bogus and optimises out all the > code. ?So it takes 0 time. ?oops. > > At any rate, aside from knowing that something went horribly wrong with > that rev, you don't really need to know how wrong. ?And by making the > graph display up to that point means that the dots where things really > do matter get crammed closer together than would otherwise be the case. > So he had a mode where things wehre displayed with an arbitrary value > at the bottom (in our coase it would be the top) which he could specify. > Then the graph would be replotted, with the outliers off the graph, but > making it easier to read the dots for the more normal cases. > > Any chance we could do that too? Link maybe? > > Laura > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From benjamin at python.org Tue Mar 8 18:14:27 2011 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 8 Mar 2011 11:14:27 -0600 Subject: [pypy-dev] Basic math ops In-Reply-To: References: Message-ID: 2011/3/8 Timothy Baldridge : > In my interpreter, I have several primitive wrapper objects: > IntObject, FloatObject, etc. > > Currently I'm planning to create functions for every combination: > > def add_int_float(i, f): > ? ? return FloatObject(i.int_value() + f.float_value()) > > then I'd need some sort of dispatch code that would say > > if isinstance(arg1, IntObject) and isinstance(arg2, FloatObject): > ? ?return add_int_float(arg1, arg2) > > It seems like pypy should have something to do this already. I saw > some information on this but that seemed specific to the Python > interpreter. You could use multimethods. Look at the stuff in pypy/objspace/std/ for examples. (It's a bit messy.) > > What's the recommended procedure on this? Is there a way to tell pypy > "this object wraps a single value, so keep it unwrapped if you can", > or does pypy figure that out automagically? The JIT will unwrap things automatically. -- Regards, Benjamin From tobami at googlemail.com Tue Mar 8 18:17:17 2011 From: tobami at googlemail.com (Miquel Torres) Date: Tue, 8 Mar 2011 18:17:17 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102280744.p1S7iFh7009804@theraft.openend.se> <201103041845.p24IjUDR015867@theraft.openend.se> <201103081614.p28GEqKT013585@theraft.openend.se> Message-ID: you mean this timeline, right?: http://speed.pypy.org/timeline/?ben=spectral-norm Because the December 22 result is so high, the yaxis maximum goes up to 2.5, thus having less space for the more interesting < 1 range, right? Regarding mozilla, do you mean this site?: http://arewefastyet.com/ I can see their timelines have some holes, probably failed runs... I see a problem with the approach you suggest. Entering an arbitrary maximum yaxis number is not a good thing. I think the onus is there on the benchmark infrastructure to not send results that aren't statistically significant. See Javastats (http://www.elis.ugent.be/en/JavaStats), or ReBench (https://github.com/smarr/ReBench). Something that can be done on the Codespeed side is to treat differently points that have a too high stddev. In the aforementioned spectral-norm timeline, the stddev "floor" is around 0.0050, while the spike has a 0.30 stddev, much higher. A "strict" mode could be implemented that invalidates or hides statistically unsound data. Btw., I had written to the arewefastyet guys about the possibility of configuring a Codespeed instance for them. We may yet see collaboration there ;-) Miquel 2011/3/8 Maciej Fijalkowski : > On Tue, Mar 8, 2011 at 8:14 AM, Laura Creighton wrote: >> In a message of Tue, 08 Mar 2011 09:10:32 +0100, Miquel Torres writes: >>>Hi, >>> >>>I finished the changes to the speed.pypy.org home page last night, but >>>alas!, I didn't have time to deploy. I will do it later today and will >>>then ping you back. >>> >>>The extra info provided is really nice as an overview, you will see ;-) >>> >>> >> >> Ah good. ?Thank you very much. ?We spent yesterday afternoon with >> the Mozilla engineers, and I got to talk to the person who maintains >> the benchmarks for tracemonkey. ?He had timelines very much like ours. >> There is one feature he has that I would like to have. ?Take a look >> at ? ?the timeline for spectral.norm. ?There are two spikes there. >> Mozilla has lines like that too, though mostly it is because their >> jit decides that the whole benchmark is bogus and optimises out all the >> code. ?So it takes 0 time. ?oops. >> >> At any rate, aside from knowing that something went horribly wrong with >> that rev, you don't really need to know how wrong. ?And by making the >> graph display up to that point means that the dots where things really >> do matter get crammed closer together than would otherwise be the case. >> So he had a mode where things wehre displayed with an arbitrary value >> at the bottom (in our coase it would be the top) which he could specify. >> Then the graph would be replotted, with the outliers off the graph, but >> making it easier to read the dots for the more normal cases. >> >> Any chance we could do that too? > > Link maybe? > >> >> Laura >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev >> > From tobami at googlemail.com Tue Mar 8 20:20:06 2011 From: tobami at googlemail.com (Miquel Torres) Date: Tue, 8 Mar 2011 20:20:06 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102280744.p1S7iFh7009804@theraft.openend.se> <201103041845.p24IjUDR015867@theraft.openend.se> <201103081614.p28GEqKT013585@theraft.openend.se> Message-ID: Ok, I just committed the changes. They address two general cases: - You want to know how fast PyPy is *now* compared to CPython in different benchmark scenarios, or tasks. - You want to know how PyPy has been *improving* overall over the last releases That is now answered on the front page, and the reports are now much less prominent (I didn't change the logic because it is something I want to do properly, not just as a hack for speed.pypy). - I have not yet addressed the "smaller is better" point. I am aware that the wording of the "faster on average" needs to be improved (I am discussing it with Holger even now ;). Please chime in so that we can have a good paragraph that is informative and short enough while at the same time not being misleading. Miquel 2011/3/8 Miquel Torres : > you mean this timeline, right?: > http://speed.pypy.org/timeline/?ben=spectral-norm > > Because the December 22 result is so high, the yaxis maximum goes up > to 2.5, thus having less space for the more interesting < 1 range, > right? > > Regarding mozilla, do you mean this site?: http://arewefastyet.com/ > I can see their timelines have some holes, probably failed runs... > > I see a problem with the approach you suggest. Entering an arbitrary > maximum yaxis number is not a good thing. I think the onus is there on > the benchmark infrastructure to not send results that aren't > statistically significant. See Javastats > (http://www.elis.ugent.be/en/JavaStats), or ReBench > (https://github.com/smarr/ReBench). > > Something that can be done on the Codespeed side is to treat > differently points that have a too high stddev. In the aforementioned > spectral-norm timeline, the stddev "floor" is around 0.0050, while the > spike has a 0.30 stddev, much higher. A "strict" mode could be > implemented that invalidates or hides statistically unsound data. > > Btw., I had written to the arewefastyet guys about the possibility of > configuring a Codespeed instance for them. We may yet see > collaboration there ;-) > > Miquel > > > 2011/3/8 Maciej Fijalkowski : >> On Tue, Mar 8, 2011 at 8:14 AM, Laura Creighton wrote: >>> In a message of Tue, 08 Mar 2011 09:10:32 +0100, Miquel Torres writes: >>>>Hi, >>>> >>>>I finished the changes to the speed.pypy.org home page last night, but >>>>alas!, I didn't have time to deploy. I will do it later today and will >>>>then ping you back. >>>> >>>>The extra info provided is really nice as an overview, you will see ;-) >>>> >>>> >>> >>> Ah good. ?Thank you very much. ?We spent yesterday afternoon with >>> the Mozilla engineers, and I got to talk to the person who maintains >>> the benchmarks for tracemonkey. ?He had timelines very much like ours. >>> There is one feature he has that I would like to have. ?Take a look >>> at ? ?the timeline for spectral.norm. ?There are two spikes there. >>> Mozilla has lines like that too, though mostly it is because their >>> jit decides that the whole benchmark is bogus and optimises out all the >>> code. ?So it takes 0 time. ?oops. >>> >>> At any rate, aside from knowing that something went horribly wrong with >>> that rev, you don't really need to know how wrong. ?And by making the >>> graph display up to that point means that the dots where things really >>> do matter get crammed closer together than would otherwise be the case. >>> So he had a mode where things wehre displayed with an arbitrary value >>> at the bottom (in our coase it would be the top) which he could specify. >>> Then the graph would be replotted, with the outliers off the graph, but >>> making it easier to read the dots for the more normal cases. >>> >>> Any chance we could do that too? >> >> Link maybe? >> >>> >>> Laura >>> _______________________________________________ >>> pypy-dev at codespeak.net >>> http://codespeak.net/mailman/listinfo/pypy-dev >>> >> > From holger at merlinux.eu Tue Mar 8 20:27:31 2011 From: holger at merlinux.eu (holger krekel) Date: Tue, 8 Mar 2011 19:27:31 +0000 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: References: <201102280744.p1S7iFh7009804@theraft.openend.se> <201103041845.p24IjUDR015867@theraft.openend.se> <201103081614.p28GEqKT013585@theraft.openend.se> Message-ID: <20110308192731.GD16231@merlinux.eu> On Tue, Mar 08, 2011 at 20:20 +0100, Miquel Torres wrote: > Ok, I just committed the changes. > > They address two general cases: > - You want to know how fast PyPy is *now* compared to CPython in > different benchmark scenarios, or tasks. > - You want to know how PyPy has been *improving* overall over the last releases > > That is now answered on the front page, and the reports are now much > less prominent (I didn't change the logic because it is something I > want to do properly, not just as a hack for speed.pypy). > - I have not yet addressed the "smaller is better" point. Great work, Miquel. I like it! > I am aware that the wording of the "faster on average" needs to be > improved (I am discussing it with Holger even now ;). Please chime in > so that we can have a good paragraph that is informative and short > enough while at the same time not being misleading. Maybe something that reads like this:: On average, PyPy trunk runs the benchmarks 3.3 times faster than CPython. The average is computed as the geometric mean of all benchmark timings. Idea is to avoid the notion that "pypy is 3.3. times faster than cpython" at this point because the benchmarks are not yet a balanced selection of real life use cases. best, holger > Miquel > > > 2011/3/8 Miquel Torres : > > you mean this timeline, right?: > > http://speed.pypy.org/timeline/?ben=spectral-norm > > > > Because the December 22 result is so high, the yaxis maximum goes up > > to 2.5, thus having less space for the more interesting < 1 range, > > right? > > > > Regarding mozilla, do you mean this site?: http://arewefastyet.com/ > > I can see their timelines have some holes, probably failed runs... > > > > I see a problem with the approach you suggest. Entering an arbitrary > > maximum yaxis number is not a good thing. I think the onus is there on > > the benchmark infrastructure to not send results that aren't > > statistically significant. See Javastats > > (http://www.elis.ugent.be/en/JavaStats), or ReBench > > (https://github.com/smarr/ReBench). > > > > Something that can be done on the Codespeed side is to treat > > differently points that have a too high stddev. In the aforementioned > > spectral-norm timeline, the stddev "floor" is around 0.0050, while the > > spike has a 0.30 stddev, much higher. A "strict" mode could be > > implemented that invalidates or hides statistically unsound data. > > > > Btw., I had written to the arewefastyet guys about the possibility of > > configuring a Codespeed instance for them. We may yet see > > collaboration there ;-) > > > > Miquel > > > > > > 2011/3/8 Maciej Fijalkowski : > >> On Tue, Mar 8, 2011 at 8:14 AM, Laura Creighton wrote: > >>> In a message of Tue, 08 Mar 2011 09:10:32 +0100, Miquel Torres writes: > >>>>Hi, > >>>> > >>>>I finished the changes to the speed.pypy.org home page last night, but > >>>>alas!, I didn't have time to deploy. I will do it later today and will > >>>>then ping you back. > >>>> > >>>>The extra info provided is really nice as an overview, you will see ;-) > >>>> > >>>> > >>> > >>> Ah good. ?Thank you very much. ?We spent yesterday afternoon with > >>> the Mozilla engineers, and I got to talk to the person who maintains > >>> the benchmarks for tracemonkey. ?He had timelines very much like ours. > >>> There is one feature he has that I would like to have. ?Take a look > >>> at ? ?the timeline for spectral.norm. ?There are two spikes there. > >>> Mozilla has lines like that too, though mostly it is because their > >>> jit decides that the whole benchmark is bogus and optimises out all the > >>> code. ?So it takes 0 time. ?oops. > >>> > >>> At any rate, aside from knowing that something went horribly wrong with > >>> that rev, you don't really need to know how wrong. ?And by making the > >>> graph display up to that point means that the dots where things really > >>> do matter get crammed closer together than would otherwise be the case. > >>> So he had a mode where things wehre displayed with an arbitrary value > >>> at the bottom (in our coase it would be the top) which he could specify. > >>> Then the graph would be replotted, with the outliers off the graph, but > >>> making it easier to read the dots for the more normal cases. > >>> > >>> Any chance we could do that too? > >> > >> Link maybe? > >> > >>> > >>> Laura > >>> _______________________________________________ > >>> pypy-dev at codespeak.net > >>> http://codespeak.net/mailman/listinfo/pypy-dev > >>> > >> > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From chef at ghum.de Tue Mar 8 20:28:47 2011 From: chef at ghum.de (Massa, Harald Armin) Date: Tue, 8 Mar 2011 20:28:47 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102280744.p1S7iFh7009804@theraft.openend.se> <201103041845.p24IjUDR015867@theraft.openend.se> <201103081614.p28GEqKT013585@theraft.openend.se> Message-ID: Miquel, that looks great! Could you please add an additional PHB-Explanation of "orange = cPython (xxx), blue = PyPy" ? best wishes, Harald -- GHUM GmbH Harald Armin Massa Spielberger Stra?e 49 70435 Stuttgart 0173/9409607 Amtsgericht Stuttgart, HRB 734971 - persuadere. et programmare From drsalists at gmail.com Tue Mar 8 20:39:41 2011 From: drsalists at gmail.com (Dan Stromberg) Date: Tue, 8 Mar 2011 11:39:41 -0800 Subject: [pypy-dev] Amusing graph Message-ID: I thought there might be some people here who'd be interested in this graph: http://stromberg.dnsalias.org/~dstromberg/backshift/ It shows pypy performing very well when compared to a number of other Python runtime systems, including cpython+cython (2 and 3) and cpython+psyco (just 2 of course). It's just one part (the critical section) of one application, but still... -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110308/3c5b0937/attachment.htm From janzert at janzert.com Tue Mar 8 23:11:17 2011 From: janzert at janzert.com (Janzert) Date: Tue, 08 Mar 2011 17:11:17 -0500 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102280744.p1S7iFh7009804@theraft.openend.se> <201103041845.p24IjUDR015867@theraft.openend.se> <201103081614.p28GEqKT013585@theraft.openend.se> Message-ID: On 3/8/2011 12:17 PM, Miquel Torres wrote: > > I see a problem with the approach you suggest. Entering an arbitrary > maximum yaxis number is not a good thing. I think the onus is there on > the benchmark infrastructure to not send results that aren't > statistically significant. See Javastats > (http://www.elis.ugent.be/en/JavaStats), or ReBench > (https://github.com/smarr/ReBench). > I think the problem is that even a statistically significant result can be arbitrarily bad (e.g. if a major regression is checked in). It does seem like it would be really nice to have the option of viewing with a scale such that the maximum displayed is something like say 2 standard deviations above the mean of displayed results. Janzert From lac at openend.se Wed Mar 9 05:12:56 2011 From: lac at openend.se (Laura Creighton) Date: Wed, 09 Mar 2011 05:12:56 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: Message from Miquel Torres of "Tue, 08 Mar 2011 18:17:17 +0100." References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102280744.p1S7iFh7009804@theraft.openend.se> <201103041845.p24IjUDR015867@theraft.openend.se> <201103081614.p28GEqKT013585@theraft.openend.se> Message-ID: <201103090412.p294CuMS011453@theraft.openend.se> In a message of Tue, 08 Mar 2011 18:17:17 +0100, Miquel Torres writes: >you mean this timeline, right?: >http://speed.pypy.org/timeline/?ben=3Dspectral-norm > >Because the December 22 result is so high, the yaxis maximum goes up >to 2.5, thus having less space for the more interesting < 1 range, >right? yes > >Regarding mozilla, do you mean this site?: http://arewefastyet.com/ >I can see their timelines have some holes, probably failed runs... I was seeing something else, and I don't have a url. I think that what I was seeing is what they use to make the arewefastyet.com site. >I see a problem with the approach you suggest. Entering an arbitrary >maximum yaxis number is not a good thing. I think the onus is there on >the benchmark infrastructure to not send results that aren't >statistically significant. See Javastats >(http://www.elis.ugent.be/en/JavaStats), or ReBench >(https://github.com/smarr/ReBench). I don't think you understand what I want. Sorry if I was unclear. I am fine with the way that the benchmarks are displayed right now, but I want a way to dynamically do there and say, I want to throw away all data that is higher than a certain figure, or lower than a certain one, because right now I am onoy interested in results in a certain range. I'm not looking to change what the benchmark says for everybody who looks at it, or change how it is presented in general. I just want a way to zoom in and only see results in the range that interests me. You and anybody else might have a different range that interests you, and you should be free to get this as well. >Something that can be done on the Codespeed side is to treat >differently points that have a too high stddev. In the aforementioned >spectral-norm timeline, the stddev "floor" is around 0.0050, while the >spike has a 0.30 stddev, much higher. A "strict" mode could be >implemented that invalidates or hides statistically unsound data. The problem is that I want to throw away arbitrary amounts of data regardless of whether they are statistically significant or not, on the basis of I know what I want to see, and this other stuff is getting in the way or being distracting?. >Btw., I had written to the arewefastyet guys about the possibility of >configuring a Codespeed instance for them. We may yet see >collaboration there ;-) > >Miquel Laura From lac at openend.se Wed Mar 9 06:17:41 2011 From: lac at openend.se (Laura Creighton) Date: Wed, 09 Mar 2011 06:17:41 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: Message from Miquel Torres of "Tue, 08 Mar 2011 20:20:06 +0100." References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102280744.p1S7iFh7009804@theraft.openend.se> <201103041845.p24IjUDR015867@theraft.openend.se> <201103081614.p28GEqKT013585@theraft.openend.se> Message-ID: <201103090517.p295HfDa017396@theraft.openend.se> In a message of Tue, 08 Mar 2011 20:20:06 +0100, Miquel Torres writes: >Ok, I just committed the changes. > >They address two general cases: >- You want to know how fast PyPy is *now* compared to CPython in >different benchmark scenarios, or tasks. >- You want to know how PyPy has been *improving* overall over the last re >le= >ases > >That is now answered on the front page, and the reports are now much >less prominent (I didn't change the logic because it is something I >want to do properly, not just as a hack for speed.pypy). >- I have not yet addressed the "smaller is better" point. > >I am aware that the wording of the "faster on average" needs to be >improved (I am discussing it with Holger even now ;). Please chime in >so that we can have a good paragraph that is informative and short >enough while at the same time not being misleading. > >Miquel The graphic is lovely. you have a s?pelling error s/taks/task/. Many of us are at PyCon now, so working on wording may not be something we have time for now. I am not sure that the geometric mean of all benchmarks give you anything meaningful, so I would have avoided saying anything like that. More specifically, I think that there is a division between some highly mathematical programs, where you might get a speedup of 20 to 30 times CPython, and the benchmarks whcih I find much more meaningful, those that represent actualy python programs -- where I think we are typically only between 2 and 3 times faster. The only reason to have some of the benchmarks is because they are well known. So people expect them. But doing very well on them is not actually all that significant -- it would be easy to write something that is great and running these contrived, synthetic benchmarks, but really lousy at running real python code. And I don't think that you can use the geometric mean to prove a thing with this. So I think talking about it makes us look bad -- we are making claims based on either bad science, or pseudo-science. And I want the 'lestest results' stuff gone from the front page. It's as misleading as ever. And I have done a poll of developers. it's not just me. Nobody finds it valuable. Because things stay red forever, we all ignore it all the time and go directly to the raw results of runs, which is what we are interested in. This also tells us of improvements, which we are also interested in, because unexpected improvements often mean something is very wrong. The whole thing has the same problem as those popup windows 'do you really want to delete that file? confirm y/n'. You get used to typing y. Then you do it when you meant not to save the file. The red pages get ignored for precisely the same reason. We're all used to all the red, which generally refer to problems which were already fixed, and thus are things that we _should_ ignore. So we end up ignoring it all, all of the time. So it therefore doesn't serve its purpose of reminding developers of anything. The general public, however, reads the thing and thinks -- latest results, pypy has just gone to hell, look at all those slowdowns. Explaining that they aren't current, or latest results, but instead 'sometime in the past when we were bad once' is getting irritating. Can you please move it someplace else so I don't have to have this conversation with pycon attendees any more? Later, when pycon is over, we can discuss and work out a better design for informing developers that a given build may have broken things. This way is not working. Laura From tobami at googlemail.com Wed Mar 9 09:14:53 2011 From: tobami at googlemail.com (Miquel Torres) Date: Wed, 9 Mar 2011 09:14:53 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102280744.p1S7iFh7009804@theraft.openend.se> <201103041845.p24IjUDR015867@theraft.openend.se> <201103081614.p28GEqKT013585@theraft.openend.se> Message-ID: I added a legend to the first plot 2011/3/8 Massa, Harald Armin : > Miquel, > > that looks great! Could you please add an additional PHB-Explanation > of "orange = cPython (xxx), blue = PyPy" ? > > best wishes, > > Harald > > > -- > GHUM GmbH > Harald Armin Massa > Spielberger Stra?e 49 > 70435 Stuttgart > 0173/9409607 > > Amtsgericht Stuttgart, HRB 734971 > - > persuadere. > et programmare > From tobami at googlemail.com Wed Mar 9 09:26:29 2011 From: tobami at googlemail.com (Miquel Torres) Date: Wed, 9 Mar 2011 09:26:29 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: <201103090412.p294CuMS011453@theraft.openend.se> References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102280744.p1S7iFh7009804@theraft.openend.se> <201103041845.p24IjUDR015867@theraft.openend.se> <201103081614.p28GEqKT013585@theraft.openend.se> <201103090412.p294CuMS011453@theraft.openend.se> Message-ID: I see. There is an easy solution for that, at least for the moment: enabling zooming. I just did that, and you can now use zooming in a timeline plot to select a narrower yaxis range or just view a particular area in detail. A single click resets the zoom level. If that is not enough, we can discuss a better solution when you have more time. 2011/3/9 Laura Creighton : > In a message of Tue, 08 Mar 2011 18:17:17 +0100, Miquel Torres writes: >>you mean this timeline, right?: >>http://speed.pypy.org/timeline/?ben=3Dspectral-norm >> >>Because the December 22 result is so high, the yaxis maximum goes up >>to 2.5, thus having less space for the more interesting < 1 range, >>right? > > yes > >> >>Regarding mozilla, do you mean this site?: http://arewefastyet.com/ >>I can see their timelines have some holes, probably failed runs... > > I was seeing something else, and I don't have a url. I think that what > I was seeing is what they use to make the arewefastyet.com site. > >>I see a problem with the approach you suggest. Entering an arbitrary >>maximum yaxis number is not a good thing. I think the onus is there on >>the benchmark infrastructure to not send results that aren't >>statistically significant. See Javastats >>(http://www.elis.ugent.be/en/JavaStats), or ReBench >>(https://github.com/smarr/ReBench). > > I don't think you understand what I want. Sorry if I was unclear. > I am fine with the way that the benchmarks are displayed right now, > but I want a way to dynamically do there and say, I want to throw > away all data that is higher than a certain figure, or lower than > a certain one, because right now I am onoy interested in results > in a certain range. > > I'm not looking to change what the benchmark says for everybody > who looks at it, or change how it is presented in general. ?I just > want a way to zoom in and only see results in the range that > interests me. ?You and anybody else might have a different > range that interests you, and you should be free to get this as well. > >>Something that can be done on the Codespeed side is to treat >>differently points that have a too high stddev. In the aforementioned >>spectral-norm timeline, the stddev "floor" is around 0.0050, while the >>spike has a 0.30 stddev, much higher. A "strict" mode could be >>implemented that invalidates or hides statistically unsound data. > > The problem is that I want to throw away arbitrary amounts of data > regardless of whether they are statistically significant or not, > on the basis of I know what I want to see, and this other stuff > is getting in the way or being distracting?. > >>Btw., I had written to the arewefastyet guys about the possibility of >>configuring a Codespeed instance for them. We may yet see >>collaboration there ;-) >> >>Miquel > > Laura > From tobami at googlemail.com Wed Mar 9 09:46:14 2011 From: tobami at googlemail.com (Miquel Torres) Date: Wed, 9 Mar 2011 09:46:14 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: <201103090517.p295HfDa017396@theraft.openend.se> References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102280744.p1S7iFh7009804@theraft.openend.se> <201103041845.p24IjUDR015867@theraft.openend.se> <201103081614.p28GEqKT013585@theraft.openend.se> <201103090517.p295HfDa017396@theraft.openend.se> Message-ID: > And I want the 'lestest results' stuff gone from the front page. > It's as misleading as ever. And I have done a poll of developers. it's > not just me. Nobody finds it valuable. Because things stay red forever, > we all ignore it all the time and go directly to the raw results of > runs, which is what we are interested in. This also tells us of > improvements, which we are also interested in, because unexpected > improvements often mean something is very wrong. Ok, that is pretty clear. > Explaining that they aren't current, or latest results, but instead > 'sometime in the past when we were bad once' is getting irritating. > Can you please move it someplace else so I don't have to have this > conversation with pycon attendees any more? Sorry about that. I have removed it from the site. > Later, when pycon is over, we can discuss and work out a better design > for informing developers that a given build may have broken things. This > way is not working. Yes, we can figure out a better approach after PyCon. > And I don't think that you can use the geometric mean to prove a thing > with this. So I think talking about it makes us look bad -- we are > making claims based on either bad science, or pseudo-science. I agree that it wasn't the best explanation. I have removed most text, so that it doesn't explicitly say that it means PyPy *is* X times faster. It now just states that the geometric mean is X times faster. If it is still too much, we can remove all text. But then some people won't understand the numbers. We can also remove the second plot. The mean of a set of benchmarks that does not represent a balanced real-world task mix is of course not very good. But then again the world is complicated. And, as a normal Python developer, I find the second plot extremely interesting, because it gives me a ballpark idea of where the PyPy project is heading to. Before I decide to use PyPy in production instead of CPython, I will do my own testing for *my* application, but I assure you that not having a ballpark average won't be a plus in considering to invest the time to test and adopt PyPy. But that is my opinion of course ;-) Have fun at PyCon! Miquel 2011/3/9 Laura Creighton : > In a message of Tue, 08 Mar 2011 20:20:06 +0100, Miquel Torres writes: >>Ok, I just committed the changes. >> >>They address two general cases: >>- You want to know how fast PyPy is *now* compared to CPython in >>different benchmark scenarios, or tasks. >>- You want to know how PyPy has been *improving* overall over the last re >>le= >>ases >> >>That is now answered on the front page, and the reports are now much >>less prominent (I didn't change the logic because it is something I >>want to do properly, not just as a hack for speed.pypy). >>- I have not yet addressed the "smaller is better" point. >> >>I am aware that the wording of the "faster on average" needs to be >>improved (I am discussing it with Holger even now ;). Please chime in >>so that we can have a good paragraph that is informative and short >>enough while at the same time not being misleading. >> >>Miquel > > The graphic is lovely. > > you have a s?pelling error s/taks/task/. > > Many of us are at PyCon now, so working on wording may not be > something we have time for now. ?I am not sure that the geometric > mean of all benchmarks give you anything meaningful, so I would > have avoided saying anything like that. ?More specifically, I think > that there is a division between some highly mathematical programs, > where you might get a speedup of 20 to 30 times CPython, and the > benchmarks whcih I find much more meaningful, those that represent > actualy python programs -- where I think we are typically only between > 2 and 3 times faster. > > The only reason to have some of the benchmarks is because they are > well known. ?So people expect them. ? But doing very well on them is > not actually all that significant -- it would be easy to write something > that is great and running these contrived, synthetic benchmarks, but > really lousy at running real python code. > > And I don't think that you can use the geometric mean to prove a thing > with this. ?So I think talking about it makes us look bad -- we are > making claims based on either bad science, or pseudo-science. > > And I want the 'lestest results' stuff gone from the front page. > It's as misleading as ever. ?And I have done a poll of developers. ?it's > not just me. ?Nobody finds it valuable. ?Because things stay red forever, > we all ignore it all the time and go directly to the raw results of > runs, which is what we are interested in. ?This also tells us of > improvements, which we are also interested in, because unexpected > improvements often mean something is very wrong. > > The whole thing has the same problem as those popup windows 'do you > really want to delete that file? confirm y/n'. ?You get used to typing > y. ?Then you do it when you meant not to save the file. ?The red pages > get ignored for precisely the same reason. ?We're all used to all the > red, which generally refer to problems which were already fixed, and > thus are things that we _should_ ignore. ?So we end up ignoring it all, > all of the time. ?So it therefore doesn't serve its purpose of reminding > developers of anything. > > The general public, however, reads the thing and thinks -- latest > results, pypy has just gone to hell, look at all those slowdowns. > Explaining that they aren't current, or latest results, but instead > 'sometime in the past when we were bad once' is getting irritating. > Can you please move it someplace else so I don't have to have this > conversation with pycon attendees any more? > > Later, when pycon is over, we can discuss and work out a better design > for informing developers that a given build may have broken things. ?This > way is not working. > > Laura > From greg at quora.com Wed Mar 9 13:29:15 2011 From: greg at quora.com (Greg Price) Date: Wed, 9 Mar 2011 04:29:15 -0800 Subject: [pypy-dev] [PATCH] Fix segmentation fault on parsing empty list-assignments Message-ID: This same patch is on bitbucket at https://bitbucket.org/price/pypy, where I've sent a pull request. Holger Krekel suggested on IRC that I send mail here. If others have different preferences for how to submit a patch, let me know. Before this patch, "[] = []" would abort the interpreter, with a segmentation fault if in pypy-c. A segmentation fault is always bad, but in this case further the code is valid Python, if not very useful. (In my commit message on bitbucket, I incorrectly said it only affects invalid Python, like "[] += []".) Greg diff -r eb44d135f334 -r 0db4ac049ea2 pypy/interpreter/astcompiler/asthelpers.py --- a/pypy/interpreter/astcompiler/asthelpers.py Tue Mar 08 11:14:36 2011 -0800 +++ b/pypy/interpreter/astcompiler/asthelpers.py Wed Mar 09 03:26:54 2011 -0800 @@ -40,9 +40,10 @@ ?? ? ? ? return self.elts ?? ? def set_context(self, ctx): - ? ? ? ?for elt in self.elts: - ? ? ? ? ? ?elt.set_context(ctx) - ? ? ? ?self.ctx = ctx + ? ? ? ?if self.elts: + ? ? ? ? ? ?for elt in self.elts: + ? ? ? ? ? ? ? ?elt.set_context(ctx) + ? ? ? ? ? ?self.ctx = ctx ?class __extend__(ast.Attribute): diff -r eb44d135f334 -r 0db4ac049ea2 pypy/interpreter/astcompiler/test/test_compiler.py --- a/pypy/interpreter/astcompiler/test/test_compiler.py Tue Mar 08 11:14:36 2011 -0800 +++ b/pypy/interpreter/astcompiler/test/test_compiler.py Wed Mar 09 03:26:54 2011 -0800 @@ -70,6 +70,9 @@ ?? ? st = simple_test + ? ?def error_test(self, source, exc_type): + ? ? ? ?py.test.raises(exc_type, self.simple_test, source, None, None) + ?? ? def test_long_jump(self): ?? ? ? ? func = """def f(x): ?? ? y = 0 @@ -98,11 +101,13 @@ ?? ? ? ? self.simple_test(stmt, "type(x)", int) ?? ? def test_tuple_assign(self): + ? ? ? ?yield self.error_test, "() = 1", SyntaxError ?? ? ? ? yield self.simple_test, "x,= 1,", "x", 1 ?? ? ? ? yield self.simple_test, "x,y = 1,2", "x,y", (1, 2) ?? ? ? ? yield self.simple_test, "x,y,z = 1,2,3", "x,y,z", (1, 2, 3) ?? ? ? ? yield self.simple_test, "x,y,z,t = 1,2,3,4", "x,y,z,t", (1, 2, 3, 4) ?? ? ? ? yield self.simple_test, "x,y,x,t = 1,2,3,4", "x,y,t", (3, 2, 4) + ? ? ? ?yield self.simple_test, "[] = []", "1", 1 ?? ? ? ? yield self.simple_test, "[x]= 1,", "x", 1 ?? ? ? ? yield self.simple_test, "[x,y] = [1,2]", "x,y", (1, 2) ?? ? ? ? yield self.simple_test, "[x,y,z] = 1,2,3", "x,y,z", (1, 2, 3) From fijall at gmail.com Wed Mar 9 13:36:26 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 9 Mar 2011 04:36:26 -0800 Subject: [pypy-dev] Basic math ops In-Reply-To: References: Message-ID: On Tue, Mar 8, 2011 at 9:14 AM, Benjamin Peterson wrote: > 2011/3/8 Timothy Baldridge : >> In my interpreter, I have several primitive wrapper objects: >> IntObject, FloatObject, etc. >> >> Currently I'm planning to create functions for every combination: >> >> def add_int_float(i, f): >> ? ? return FloatObject(i.int_value() + f.float_value()) >> >> then I'd need some sort of dispatch code that would say >> >> if isinstance(arg1, IntObject) and isinstance(arg2, FloatObject): >> ? ?return add_int_float(arg1, arg2) >> >> It seems like pypy should have something to do this already. I saw >> some information on this but that seemed specific to the Python >> interpreter. > > You could use multimethods. Look at the stuff in pypy/objspace/std/ > for examples. (It's a bit messy.) > >> >> What's the recommended procedure on this? Is there a way to tell pypy >> "this object wraps a single value, so keep it unwrapped if you can", >> or does pypy figure that out automagically? > > The JIT will unwrap things automatically. > For JIT to be happier you can however say: class A(object): _immutable_fields_ = ['value'] so it'll know that fields don't change once the object is constructed > > > -- > Regards, > Benjamin > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev From fijall at gmail.com Wed Mar 9 13:39:59 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 9 Mar 2011 04:39:59 -0800 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102280744.p1S7iFh7009804@theraft.openend.se> <201103041845.p24IjUDR015867@theraft.openend.se> <201103081614.p28GEqKT013585@theraft.openend.se> <201103090517.p295HfDa017396@theraft.openend.se> Message-ID: Hey. Awesome job Miquel, I love it. This is exactly what we wanted From tobami at googlemail.com Wed Mar 9 14:10:29 2011 From: tobami at googlemail.com (Miquel Torres) Date: Wed, 9 Mar 2011 14:10:29 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102280744.p1S7iFh7009804@theraft.openend.se> <201103041845.p24IjUDR015867@theraft.openend.se> <201103081614.p28GEqKT013585@theraft.openend.se> <201103090517.p295HfDa017396@theraft.openend.se> Message-ID: Cool! ;-) PS: Currently, only 20 benchmarks are shown (and averaged!) because older pypy revisions (mostly PyPy 1.3) don't have numbers for the newest benchmarks. If Maciej reruns those revisions the rest will be displayed ;-) 2011/3/9 Maciej Fijalkowski : > Hey. > > Awesome job Miquel, I love it. This is exactly what we wanted > From chef at ghum.de Wed Mar 9 14:19:22 2011 From: chef at ghum.de (Massa, Harald Armin) Date: Wed, 9 Mar 2011 14:19:22 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102280744.p1S7iFh7009804@theraft.openend.se> <201103041845.p24IjUDR015867@theraft.openend.se> <201103081614.p28GEqKT013585@theraft.openend.se> <201103090517.p295HfDa017396@theraft.openend.se> Message-ID: I really, really like the new display! And it motivated me to dig into the data ... which is a great result on its own. The first question for myself was "hey, why is it slow on slowspitfire, and, btw, what is slowspitfire? Could that be something that my application does, too?" But I was unable to find out what slowspitfire is doing ... I found spitfire, which does some HTML templating stuff, and deducted, that slowspitfire will do some slow HTML templating stuff. Where did I click wrong? Is there a path down to the slowspitfire.py file or an explanation what slowspitfire is doing? Harald -- GHUM GmbH Harald Armin Massa Spielberger Stra?e 49 70435 Stuttgart 0173/9409607 Amtsgericht Stuttgart, HRB 734971 - persuadere. et programmare From fijall at gmail.com Wed Mar 9 14:21:50 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 9 Mar 2011 08:21:50 -0500 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102280744.p1S7iFh7009804@theraft.openend.se> <201103041845.p24IjUDR015867@theraft.openend.se> <201103081614.p28GEqKT013585@theraft.openend.se> <201103090517.p295HfDa017396@theraft.openend.se> Message-ID: On Wed, Mar 9, 2011 at 8:19 AM, Massa, Harald Armin wrote: > I really, really like the new display! > > And it motivated me to dig into the data ... which is a great result on its own. > > The first question for myself was "hey, why is it slow on > slowspitfire, and, btw, what is slowspitfire? Could that be something > that my application does, too?" > > But I was unable to find out what slowspitfire is doing ... I found > spitfire, which does some HTML templating stuff, and deducted, that > slowspitfire will do some slow HTML templating stuff. Where did I > click wrong? Is there a path down to the slowspitfire.py file or an > explanation what slowspitfire is doing? > > Harald > https://bitbucket.org/pypy/benchmarks/src/b93caae762a0/unladen_swallow/performance/bm_spitfire.py It's creating a very large template table (1000x1000 elements I think) The explanation "why it's slow" is a bit longish. It's a combination of factors, including very long lists with GC objects in it, using ''.join(list) instead of cStringIO (the latter is faster and yes, it is a bug) and a bit of other factors. > > -- > GHUM GmbH > Harald Armin Massa > Spielberger Stra?e 49 > 70435 Stuttgart > 0173/9409607 > > Amtsgericht Stuttgart, HRB 734971 > - > persuadere. > et programmare > From fijall at gmail.com Wed Mar 9 14:53:47 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 9 Mar 2011 08:53:47 -0500 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102280744.p1S7iFh7009804@theraft.openend.se> <201103041845.p24IjUDR015867@theraft.openend.se> <201103081614.p28GEqKT013585@theraft.openend.se> <201103090517.p295HfDa017396@theraft.openend.se> Message-ID: Hey Miquel. A small feature request ;-) Can we get favicon? Cheers, fijal From lac at openend.se Wed Mar 9 14:58:16 2011 From: lac at openend.se (Laura Creighton) Date: Wed, 09 Mar 2011 14:58:16 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: Message from Miquel Torres of "Wed, 09 Mar 2011 09:26:29 +0100." References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102280744.p1S7iFh7009804@theraft.openend.se> <201103041845.p24IjUDR015867@theraft.openend.se> <201103081614.p28GEqKT013585@theraft.openend.se> <201103090412.p294CuMS011453@theraft.openend.se> Message-ID: <201103091358.p29DwGYw032370@theraft.openend.se> In a message of Wed, 09 Mar 2011 09:26:29 +0100, Miquel Torres writes: >I see. > >There is an easy solution for that, at least for the moment: enabling >zooming. I just did that, and you can now use zooming in a timeline >plot to select a narrower yaxis range or just view a particular area >in detail. A single click resets the zoom level. > >If that is not enough, we can discuss a better solution when you have mor >e time. This is terrific. Thank you very much. Laura From anto.cuni at gmail.com Wed Mar 9 14:59:19 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed, 09 Mar 2011 14:59:19 +0100 Subject: [pypy-dev] [PATCH] Fix segmentation fault on parsing empty list-assignments In-Reply-To: References: Message-ID: <4D7787B7.70509@gmail.com> On 09/03/11 13:29, Greg Price wrote: > This same patch is on bitbucket at https://bitbucket.org/price/pypy, > where I've sent a pull request. Holger Krekel suggested on IRC that I > send mail here. If others have different preferences for how to submit > a patch, let me know. Hi Greg, I have pushed your commit upstream, thanks for help! ciao, anto From pgiarrusso at mathematik.uni-marburg.de Wed Mar 9 15:15:37 2011 From: pgiarrusso at mathematik.uni-marburg.de (Paolo Giarrusso) Date: Wed, 9 Mar 2011 15:15:37 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102280744.p1S7iFh7009804@theraft.openend.se> <201103041845.p24IjUDR015867@theraft.openend.se> <201103081614.p28GEqKT013585@theraft.openend.se> <201103090517.p295HfDa017396@theraft.openend.se> Message-ID: Hi all, On Wed, Mar 9, 2011 at 14:21, Maciej Fijalkowski wrote: > On Wed, Mar 9, 2011 at 8:19 AM, Massa, Harald Armin wrote: > > I really, really like the new display! Congratulations to Miquel for the great work! A minor comment about the homepage: the answer to "How fast is PyPy?" would better stay close to the question, i.e. above Plot 1 (at least with the current wording). > > But I was unable to find out what slowspitfire is doing ... I found > > spitfire, which does some HTML templating stuff, and deducted, that > > slowspitfire will do some slow HTML templating stuff. Where did I > > click wrong? > > Is there a path down to the slowspitfire.py file or an > > explanation what slowspitfire is doing? Is there a place the answer this information to the website? I propose a link to the source in each benchmark page. Additionally, on the frontpage the individual benchmark names could be links to the benchmark page, like in the grid view. The specific benchmark has no description: http://speed.pypy.org/timeline/?exe=1%2C3&base=2%2B35&ben=slowspitfire&env=tannit&revs=200 Following the spitfire_cstringio template, I propose the following rewording of the below answer (I'm not entirely happy but I guess Maciej can fix it easily if it's too bad): slowspitfire: Uses the Spitfire template system to build a 1000x1000-cell HTML table; it differs from spitfire which is slower on PyPy: it uses .join(list) instead of cStringIO module, has very long lists with GC objects in it, and some other smaller problems. > https://bitbucket.org/pypy/benchmarks/src/b93caae762a0/unladen_swallow/performance/bm_spitfire.py > It's creating a very large template table (1000x1000 elements I think) > > The explanation "why it's slow" is a bit longish. It's a combination > of factors, including very long lists with GC objects in it, using > ''.join(list) instead of cStringIO (the latter is faster and yes, it > is a bug) and a bit of other factors. Another small problem I had with zooming (which is really cool, BTW): >There is an easy solution for that, at least for the moment: enabling >zooming. I just did that, and you can now use zooming in a timeline >plot to select a narrower yaxis range or just view a particular area >in detail. A single click resets the zoom level. While trying this I clicked on a revision; I immediately clicked on "back", but I was brought too much backwards, to the grid of all benchmarks, which loads slow enough for one to notice. If you instead click "Back" from a specific benchmark page, you are brought back to the home. Fixing this without loading a separate page for each plot seems hard; however, it seems that e.g. Facebook handles this by modifying the URL part after #, so that the page is not reloaded from scratch, but I'm no web developer, so you probably know better than me. Cheers, -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From lac at openend.se Wed Mar 9 15:56:26 2011 From: lac at openend.se (Laura Creighton) Date: Wed, 09 Mar 2011 15:56:26 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: Message from Miquel Torres of "Wed, 09 Mar 2011 09:46:14 +0100." References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102280744.p1S7iFh7009804@theraft.openend.se> <201103041845.p24IjUDR015867@theraft.openend.se> <201103081614.p28GEqKT013585@theraft.openend.se> <201103090517.p295HfDa017396@theraft.openend.se> Message-ID: <201103091456.p29EuQqA006242@theraft.openend.se> Thank you very, very much. I am sorry if I was short with you last night, I was very tired. This is great. I am fine with saying that the geometric mean is X times faster. I'd like to come up with a good way to say that we are faster on real programs, not just contrived examples, but this will take work and thought. Maybe some of the people who are on this list but not at pycon can come up with something. Thank you very much. I'm very happy now. Laura From tobami at googlemail.com Wed Mar 9 21:00:33 2011 From: tobami at googlemail.com (Miquel Torres) Date: Wed, 9 Mar 2011 21:00:33 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102280744.p1S7iFh7009804@theraft.openend.se> <201103041845.p24IjUDR015867@theraft.openend.se> <201103081614.p28GEqKT013585@theraft.openend.se> <201103090517.p295HfDa017396@theraft.openend.se> Message-ID: > But I was unable to find out what slowspitfire is doing ... I found > spitfire, which does some HTML templating stuff, and deducted, that > slowspitfire will do some slow HTML templating stuff. Where did I > click wrong? Is there a path down to the slowspitfire.py file or an > explanation what slowspitfire is doing? You definitely have a point, and there are features planned that should make benchmark-discovery more user-friendly. 2011/3/9 Massa, Harald Armin : > I really, really like the new display! > > And it motivated me to dig into the data ... which is a great result on its own. > > The first question for myself was "hey, why is it slow on > slowspitfire, and, btw, what is slowspitfire? Could that be something > that my application does, too?" > > But I was unable to find out what slowspitfire is doing ... I found > spitfire, which does some HTML templating stuff, and deducted, that > slowspitfire will do some slow HTML templating stuff. Where did I > click wrong? Is there a path down to the slowspitfire.py file or an > explanation what slowspitfire is doing? > > Harald > > > -- > GHUM GmbH > Harald Armin Massa > Spielberger Stra?e 49 > 70435 Stuttgart > 0173/9409607 > > Amtsgericht Stuttgart, HRB 734971 > - > persuadere. > et programmare > From tobami at googlemail.com Wed Mar 9 21:11:50 2011 From: tobami at googlemail.com (Miquel Torres) Date: Wed, 9 Mar 2011 21:11:50 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102280744.p1S7iFh7009804@theraft.openend.se> <201103041845.p24IjUDR015867@theraft.openend.se> <201103081614.p28GEqKT013585@theraft.openend.se> <201103090517.p295HfDa017396@theraft.openend.se> Message-ID: > The specific benchmark has no description: > http://speed.pypy.org/timeline/?exe=1%2C3&base=2%2B35&ben=slowspitfire&env=tannit&revs=200 Thanks, I have added your description. > Is there a place the answer this information to the website? I propose > a link to the source in each benchmark page. Yes, I will try to find a way to put links to the code in each benchmark page. > Additionally, on the frontpage the individual benchmark names could be > links to the benchmark page, like in the grid view. I first thought that it would be difficult because the benchmark names themselves are tick labels for the axes, but I think I can actually make the bars clickable. I will see. > While trying this I clicked on a revision; I immediately clicked on > "back", but I was brought too much backwards, to the grid of all > benchmarks, which loads slow enough for one to notice. If you instead > click "Back" from a specific benchmark page, you are brought back to > the home. > Fixing this without loading a separate page for each plot seems hard; > however, it seems that e.g. Facebook handles this by modifying the URL > part after #, so that the page is not reloaded from scratch, but I'm > no web developer, so you probably know better than me. Yes, we are looking at a solution there. See issue #17 : https://github.com/tobami/codespeed/issues#issue/17 Stefan Marr has already implemented it. There were some issues at first, and he uses a development un-minified version of a plugin, but it works. So we should be able to integrate the feature soon. Miquel 2011/3/9 Paolo Giarrusso : > Hi all, > > On Wed, Mar 9, 2011 at 14:21, Maciej Fijalkowski wrote: >> On Wed, Mar 9, 2011 at 8:19 AM, Massa, Harald Armin wrote: >> > I really, really like the new display! > Congratulations to Miquel for the great work! > > A minor comment about the homepage: the answer to "How fast is PyPy?" > would better stay close to the question, i.e. above Plot 1 (at least > with the current wording). > >> > But I was unable to find out what slowspitfire is doing ... I found >> > spitfire, which does some HTML templating stuff, and deducted, that >> > slowspitfire will do some slow HTML templating stuff. Where did I >> > click wrong? > >> > Is there a path down to the slowspitfire.py file or an >> > explanation what slowspitfire is doing? > > Is there a place the answer this information to the website? I propose > a link to the source in each benchmark page. > Additionally, on the frontpage the individual benchmark names could be > links to the benchmark page, like in the grid view. > > The specific benchmark has no description: > http://speed.pypy.org/timeline/?exe=1%2C3&base=2%2B35&ben=slowspitfire&env=tannit&revs=200 > > Following the spitfire_cstringio template, I propose the following > rewording of the below answer (I'm not entirely happy but I guess > Maciej can fix it easily if it's too bad): > slowspitfire: Uses the Spitfire template system to build a > 1000x1000-cell HTML table; it differs from spitfire which is slower on > PyPy: it uses .join(list) instead of cStringIO module, has very long > lists with GC objects in it, and some other smaller problems. > >> https://bitbucket.org/pypy/benchmarks/src/b93caae762a0/unladen_swallow/performance/bm_spitfire.py > >> It's creating a very large template table (1000x1000 elements I think) >> >> The explanation "why it's slow" is a bit longish. It's a combination >> of factors, including very long lists with GC objects in it, using >> ''.join(list) instead of cStringIO (the latter is faster and yes, it >> is a bug) and a bit of other factors. > > Another small problem I had with zooming (which is really cool, BTW): > >>There is an easy solution for that, at least for the moment: enabling >>zooming. I just did that, and you can now use zooming in a timeline >>plot to select a narrower yaxis range or just view a particular area >>in detail. A single click resets the zoom level. > > While trying this I clicked on a revision; I immediately clicked on > "back", but I was brought too much backwards, to the grid of all > benchmarks, which loads slow enough for one to notice. If you instead > click "Back" from a specific benchmark page, you are brought back to > the home. > Fixing this without loading a separate page for each plot seems hard; > however, it seems that e.g. Facebook handles this by modifying the URL > part after #, so that the page is not reloaded from scratch, but I'm > no web developer, so you probably know better than me. > > Cheers, > -- > Paolo Giarrusso - Ph.D. Student > http://www.informatik.uni-marburg.de/~pgiarrusso/ > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From tobami at googlemail.com Wed Mar 9 21:21:13 2011 From: tobami at googlemail.com (Miquel Torres) Date: Wed, 9 Mar 2011 21:21:13 +0100 Subject: [pypy-dev] Change to the frontpage of speed.pypy.org In-Reply-To: <201103091456.p29EuQqA006242@theraft.openend.se> References: <201102271359.p1RDxkpX012065@theraft.openend.se> <201102271538.p1RFc3rG020721@theraft.openend.se> <201102280744.p1S7iFh7009804@theraft.openend.se> <201103041845.p24IjUDR015867@theraft.openend.se> <201103081614.p28GEqKT013585@theraft.openend.se> <201103090517.p295HfDa017396@theraft.openend.se> <201103091456.p29EuQqA006242@theraft.openend.se> Message-ID: > I am sorry if I was short with you last night, I was very tired. No problem Laura, that only shows that you care ;-) It would have also been much easier to discuss things if I had finished the changes a couple of days earlier instead of in the middle of your arrival in PyCon. But I didn't have the time. Life's like that. Criticism is sometimes hard to swallow, but very necessary to improve, as a person and as a project. As long as people use reasoned arguments and are respectful, it is not only OK to do so: It is your duty! ;-) So thanks to you too. Cheers, Miquel 2011/3/9 Laura Creighton : > Thank you very, very much. ?I am sorry if I was short with you last > night, I was very tired. ?This is great. ?I am fine with > saying that the geometric mean is X times faster. ?I'd ?like to come up > with a good way to say that we are faster on real programs, not just > contrived examples, but this will take work and thought. ?Maybe some > of the people who are on this list but not at pycon can come up with > something. > > Thank you very much. ?I'm very happy now. > > Laura > From tbaldridge at gmail.com Wed Mar 9 23:30:16 2011 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Wed, 9 Mar 2011 16:30:16 -0600 Subject: [pypy-dev] Regex for RPython Message-ID: I know rlib has a regex module, but it seems to be limited to only recognizing characters and not returning groups. Is there a full featured regex parser somewhere that is supported by rpython? Thanks, Timothy -- ?One of the main causes of the fall of the Roman Empire was that?lacking zero?they had no way to indicate successful termination of their C programs.? (Robert Firth) From greg at quora.com Thu Mar 10 01:13:46 2011 From: greg at quora.com (Greg Price) Date: Wed, 9 Mar 2011 16:13:46 -0800 Subject: [pypy-dev] wrong precedence of __radd__ vs list __iadd__ Message-ID: The following program works in CPython, but fails in PyPy: class C(object): def __radd__(self, other): other.append(1) return other z = [] z += C() print z # should be [1] In PyPy, this fails with "TypeError: 'C' object is not iterable". The issue is that PyPy is allowing the list's inplace_add behavior to take precedence over the C object's __radd__ method, while CPython does the reverse. A similar issue occurs in the following program: class C(object): def __rmul__(self, other): other *= 2 return other def __index__(self): return 3 print [1] * C() # should be [1,1] where PyPy instead prints [1,1,1]. In Python, if the LHS of a foo-augmented assignment has an in-place foo method defined, then that method is given the first opportunity to perform the operation, followed by the foo method and the RHS's reverse-foo methods. http://docs.python.org/reference/datamodel.html#object.__iadd__ But: CPython's 'list' type does not have any numeric methods defined at the C level, in-place or otherwise. See PyList_Type in listobject.c, where tp_as_number is null. So an in-place addition falls through to the RHS's nb_add, if present, and for a class with metaclass 'type' and an __radd__() method this is slot_nb_add() from typeobject.c, which calls __radd__(). So when "z += [1,2]" runs in CPython, it works via a further wrinkle. The meat of the implementation of INPLACE_ADD is PyNumber_InPlaceAdd() in abstract.c. When all numeric methods fail to handle the addition, this function falls back to sequence methods, looking for sq_inplace_concat or sq_concat methods on the LHS. 'list' has these methods, so its sq_inplace_concat method handles the operation. Similarly, PyNumber_InPlaceMultiply() tries all numeric methods first, before falling back to sq_inplace_repeat or sq_repeat. In PyPy, by contrast, it doesn't look like there's any logic for falling back to a concatenate method if numeric methods fail. Instead, if I'm reading correctly, the inplace_add__List_ANY() method in pypy.objspace.std.listobject runs *before* any numeric methods on the RHS are tried. For the narrow case at hand, a sufficient hack for e.g. addition would be to teach inplace_add__List_ANY() to look for an __radd__() first. At a quick grep through CPython, full compatibility by that approach would require similar hacks for bytes, array.array, collections.deque, str, unicode, buffer, and bytearray; plus the users of structseq.c, including type(sys.float_info), pwd.struct_passwd, grp.struct_group, posix.stat_result, time.struct_time, and several others. Those types in CPython all have sq_concat and not nb_add or nb_inplace_add, so they will all permit an RHS's __radd__ to take precedence, but then fall back on concatenation if no such method exists. A more comprehensive approach would teach the generic dispatch code about sequence methods falling after numeric methods in the dispatch sequence. Then each sequence type would need to identify its sequence methods for that code. Perhaps a hybrid approach is best. Assume that no type with a sq_concat also has a nb_add, and no type with a sq_repeat also has a nb_multiply. (I believe this is true for all built-in, stdlib, and pure-Python types.) Then whenever we define a method {inplace_,}{add,mul}__Foo_ANY, for a sequence type Foo, it's enough for that method to check for __r{add,mul}__ on the RHS. So we can write a generic helper function and use that in each such method. Does that last approach sound reasonable? I'm happy to go and implement it, but I'm open to other suggestions. I've posted tests (which fail) at https://bitbucket.org/price/pypy-queue/changeset/9dd9c2a5116a Greg From alex.gaynor at gmail.com Thu Mar 10 01:32:26 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Wed, 9 Mar 2011 19:32:26 -0500 Subject: [pypy-dev] Regex for RPython In-Reply-To: References: Message-ID: On Wed, Mar 9, 2011 at 5:30 PM, Timothy Baldridge wrote: > I know rlib has a regex module, but it seems to be limited to only > recognizing characters and not returning groups. Is there a full > featured regex parser somewhere that is supported by rpython? > > Thanks, > > Timothy > > -- > ?One of the main causes of the fall of the Roman Empire was > that?lacking zero?they had no way to indicate successful termination > of their C programs.? > (Robert Firth) > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > The PyPy python interpret just uses the stdlib regex parser, so no I don't think there is a regex parser. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110309/57c55060/attachment.htm From cfbolz at gmx.de Thu Mar 10 10:01:24 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Thu, 10 Mar 2011 10:01:24 +0100 Subject: [pypy-dev] Regex for RPython In-Reply-To: References: Message-ID: <4D789364.5030206@gmx.de> On 03/09/2011 11:30 PM, Timothy Baldridge wrote: > I know rlib has a regex module, but it seems to be limited to only > recognizing characters and not returning groups. Is there a full > featured regex parser somewhere that is supported by rpython? Are you just interested in the parser or also in a way to do regex matching? For the parser we indeed use the stdlib version, for the matching there is an RPython implementation taking as input the bytecodes that the stdlib regex parser produces. Cheers, Carl Friedrich From alex.gaynor at gmail.com Thu Mar 10 14:42:05 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Thu, 10 Mar 2011 08:42:05 -0500 Subject: [pypy-dev] wrong precedence of __radd__ vs list __iadd__ In-Reply-To: References: Message-ID: On Wed, Mar 9, 2011 at 7:13 PM, Greg Price wrote: > The following program works in CPython, but fails in PyPy: > > class C(object): > def __radd__(self, other): > other.append(1) > return other > > z = [] > z += C() > print z # should be [1] > > In PyPy, this fails with "TypeError: 'C' object is not iterable". > > The issue is that PyPy is allowing the list's inplace_add behavior to > take precedence over the C object's __radd__ method, while CPython > does the reverse. > > A similar issue occurs in the following program: > > class C(object): > def __rmul__(self, other): > other *= 2 > return other > def __index__(self): > return 3 > > print [1] * C() # should be [1,1] > > where PyPy instead prints [1,1,1]. > > > In Python, if the LHS of a foo-augmented assignment has an in-place > foo method defined, then that method is given the first opportunity to > perform the operation, followed by the foo method and the RHS's > reverse-foo methods. > http://docs.python.org/reference/datamodel.html#object.__iadd__ > > But: CPython's 'list' type does not have any numeric methods defined > at the C level, in-place or otherwise. See PyList_Type in > listobject.c, where tp_as_number is null. So an in-place addition > falls through to the RHS's nb_add, if present, and for a class with > metaclass 'type' and an __radd__() method this is slot_nb_add() from > typeobject.c, which calls __radd__(). > > So when "z += [1,2]" runs in CPython, it works via a further wrinkle. > The meat of the implementation of INPLACE_ADD is PyNumber_InPlaceAdd() > in abstract.c. When all numeric methods fail to handle the addition, > this function falls back to sequence methods, looking for > sq_inplace_concat or sq_concat methods on the LHS. 'list' has these > methods, so its sq_inplace_concat method handles the operation. > > Similarly, PyNumber_InPlaceMultiply() tries all numeric methods first, > before falling back to sq_inplace_repeat or sq_repeat. > > In PyPy, by contrast, it doesn't look like there's any logic for > falling back to a concatenate method if numeric methods fail. Instead, > if I'm reading correctly, the inplace_add__List_ANY() method in > pypy.objspace.std.listobject runs *before* any numeric methods on the > RHS are tried. > > > For the narrow case at hand, a sufficient hack for e.g. addition would > be to teach inplace_add__List_ANY() to look for an __radd__() first. > At a quick grep through CPython, full compatibility by that approach > would require similar hacks for bytes, array.array, collections.deque, > str, unicode, buffer, and bytearray; plus the users of structseq.c, > including type(sys.float_info), pwd.struct_passwd, grp.struct_group, > posix.stat_result, time.struct_time, and several others. Those types > in CPython all have sq_concat and not nb_add or nb_inplace_add, so > they will all permit an RHS's __radd__ to take precedence, but then > fall back on concatenation if no such method exists. > > A more comprehensive approach would teach the generic dispatch code > about sequence methods falling after numeric methods in the dispatch > sequence. Then each sequence type would need to identify its sequence > methods for that code. > > Perhaps a hybrid approach is best. Assume that no type with a > sq_concat also has a nb_add, and no type with a sq_repeat also has a > nb_multiply. (I believe this is true for all built-in, stdlib, and > pure-Python types.) Then whenever we define a method > {inplace_,}{add,mul}__Foo_ANY, for a sequence type Foo, it's enough > for that method to check for __r{add,mul}__ on the RHS. So we can > write a generic helper function and use that in each such method. > > Does that last approach sound reasonable? I'm happy to go and > implement it, but I'm open to other suggestions. > > > I've posted tests (which fail) at > https://bitbucket.org/price/pypy-queue/changeset/9dd9c2a5116a > > Greg > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > Greg, Is this affecting some real world code? I ask because right now the semantics we implement basically treat everything with the same semantics as a pure Python class on CPython would, and I think we'd like to keep it that way, as it makes the source a good bit simpler. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110310/80430efd/attachment.htm From pgiarrusso at mathematik.uni-marburg.de Thu Mar 10 15:08:07 2011 From: pgiarrusso at mathematik.uni-marburg.de (Paolo Giarrusso) Date: Thu, 10 Mar 2011 15:08:07 +0100 Subject: [pypy-dev] wrong precedence of __radd__ vs list __iadd__ In-Reply-To: References: Message-ID: On Thu, Mar 10, 2011 at 14:42, Alex Gaynor wrote: > Greg, > > Is this affecting some real world code?? I ask because right now the > semantics we implement basically treat everything with the same semantics as > a pure Python class on CPython would, and I think we'd like to keep it that > way, as it makes the source a good bit simpler. Hi, wasn't very good compatibility with CPython one of PyPy's points, without restrictions to real-world code? For instance Armin (IIRC) made significant effort to support Python's semantics for destructors (in particular, to call destructors in the "right" order), unlike done in Jython and IronPython. Cheers, -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ From anto.cuni at gmail.com Thu Mar 10 16:21:06 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu, 10 Mar 2011 16:21:06 +0100 Subject: [pypy-dev] problem after merging of jit-virtual_state Message-ID: <4D78EC62.8030101@gmail.com> Hi Hakan, hi all, as you might have noticed, there is test_pypy_c_new.test_f1 which is currently failing: http://buildbot.pypy.org/summary/longrepr?testname=TestPyPyCNew.%28%29.test_f1&builder=pypy-c-jit-linux-x86-32&build=699&mod=pypy.module.pypyjit.test_pypy_c.test_pypy_c_new buildbot did not show it earlier because before yesterday it did not run the test at all. As you can see from the traceback, the problem is that running f1 makes three loops instead of one as expected. I did a bit of manual binary search and discovered that the test started failing between r42273:b8bb5d085684 and r42305:3fe93dd500bd. Looking at the commit, it seems that the only one relevant to the problem is r42305:3fe93dd500bd, i.e. the merging of jit-virtual_state. Do you have any idea why jit-virtual_state causes this problem? (if it's a problem at all, but I think so because the three loops contain the very same operations, so I don't see why we need to compile the more than once). ciao, Anto From anto.cuni at gmail.com Thu Mar 10 18:22:37 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu, 10 Mar 2011 18:22:37 +0100 Subject: [pypy-dev] problem after merging of jit-virtual_state In-Reply-To: <4D78EC62.8030101@gmail.com> References: <4D78EC62.8030101@gmail.com> Message-ID: <4D7908DD.40503@gmail.com> On 10/03/11 16:21, Antonio Cuni wrote: > I did a bit of manual binary search and discovered that the test started > failing between r42273:b8bb5d085684 and r42305:3fe93dd500bd. Looking at the > commit, it seems that the only one relevant to the problem is > r42305:3fe93dd500bd, i.e. the merging of jit-virtual_state. I confirm that the problem is the merging of jit-virtual_state: I just tried the revision just before the merge, and the test passes. ciao, Anto From arigo at tunes.org Thu Mar 10 21:53:49 2011 From: arigo at tunes.org (Armin Rigo) Date: Thu, 10 Mar 2011 15:53:49 -0500 Subject: [pypy-dev] wrong precedence of __radd__ vs list __iadd__ In-Reply-To: References: Message-ID: Hi Greg, On Wed, Mar 9, 2011 at 7:13 PM, Greg Price wrote: > The following program works in CPython, but fails in PyPy: This is (if we are positive) an internal implementation detail and (if we are negative) a bug in CPython. There is no way to define in pure Python a list-like type that would have the exact same behavior. We already have one such special case for __add__/__radd__ on subclasses of strings and unicodes, which appears to be the use case that people rely on most often, but I don't see the point in duplicating this strange behavior for lists, tuples, and so on, unless there are really several programs out there that rely on it. A bient?t, Armin. From greg at quora.com Thu Mar 10 23:38:35 2011 From: greg at quora.com (Greg Price) Date: Thu, 10 Mar 2011 14:38:35 -0800 Subject: [pypy-dev] wrong precedence of __radd__ vs list __iadd__ In-Reply-To: References: Message-ID: Hi Armin and Alex, > This is (if we are positive) an internal implementation detail and (if > we are negative) a bug in CPython. ?There is no way to define in pure > Python a list-like type that would have the exact same behavior. I don't disagree with this -- indeed, I think a strict reading of the language reference would clearly identify this as wrong behavior by CPython. Nevertheless, real-world programs do exist that rely on this behavior. I don't have a broad survey of real programs (I don't think anyone does), but this issue causes Quora's codebase to fail a ton of unit tests and basically not work at all under PyPy without modification. The reason is that we have this idiom in thousands of places: def f(): z = [] z += Foo() z += Bar() for x in something(): z += Baz(x) return z When the classes Foo, Bar, Baz, etc were implemented, the author gave them __radd__ methods to implement the above behavior -- after all, addition is precisely what we're doing here, right? So they look like class Foo(object): def __radd__(self, other): other.append("something") return other as in my original mail. Now, knowing how this all works, it's not too hard to fix my company's code so that it works here -- I just add an __iter__ method like so: class Foo(object): [...] def __iter__(self): yield "something" And if the original author had been developing under PyPy, no doubt he would have seen the TypeError, accepted the behavior as is, and quickly figured out this approach. Programmers adapt to all kinds of details in their development environments, whether those details are justified well or poorly. But for someone considering PyPy and bringing over an existing codebase, it's awfully disconcerting to see a bunch of tests fail and have to go debugging. Especially so when it's pure Python and doesn't muck around with GC, or in sys, or use anything that a non-expert would recognize as an implementation detail. I know I'm an early, early adopter and so I don't mind, but for the broader class of users that I hope PyPy acquires in the next few years, an issue like this would be a pretty negative signal. As I outlined at the end of my first email, I don't think this would require a great deal of code. I appreciate that there's a cost to making things more complex, but I think it's worth it to make the way smooth for people moving from CPython. Let me know what you think. I'm happy to write up a patch. Greg On Thu, Mar 10, 2011 at 12:53 PM, Armin Rigo wrote: > Hi Greg, > > On Wed, Mar 9, 2011 at 7:13 PM, Greg Price wrote: >> The following program works in CPython, but fails in PyPy: > > This is (if we are positive) an internal implementation detail and (if > we are negative) a bug in CPython. ?There is no way to define in pure > Python a list-like type that would have the exact same behavior. ?We > already have one such special case for __add__/__radd__ on subclasses > of strings and unicodes, which appears to be the use case that people > rely on most often, but I don't see the point in duplicating this > strange behavior for lists, tuples, and so on, unless there are really > several programs out there that rely on it. > > > A bient?t, > > Armin. > From holger at merlinux.eu Fri Mar 11 19:46:54 2011 From: holger at merlinux.eu (holger krekel) Date: Fri, 11 Mar 2011 18:46:54 +0000 Subject: [pypy-dev] tooling/sandboxing (remarks from Ikai Lan) Message-ID: <20110311184654.GR16231@merlinux.eu> Hi all, especially devs at Pycon, FYI i just saw saw some tweets from Ikan Lan (http://twitter.com/#!/ikai ) who works with the Google App Engine time, oldest to newest: At the Python VM panel. I want to believe alternate VMs are about more than just benchmarks. Question do not agree with me #pycon For instance, I want to see the tooling that can be built around alternative VMs. Profilers, heap analyzers, better debuggers #pycon Python VM implementors think sandboxing is not something that should be implemented at VM layer. Oh, well. #pycon To me, this does not seem to be fitting for PyPy, does it? Maybe somebody can discuss a bit with him. I still believe that Google and GAE are a potentially very good application of PyPy tech. best, holger From alex.gaynor at gmail.com Fri Mar 11 19:48:48 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Fri, 11 Mar 2011 13:48:48 -0500 Subject: [pypy-dev] tooling/sandboxing (remarks from Ikai Lan) In-Reply-To: <20110311184654.GR16231@merlinux.eu> References: <20110311184654.GR16231@merlinux.eu> Message-ID: On Fri, Mar 11, 2011 at 1:46 PM, holger krekel wrote: > Hi all, especially devs at Pycon, > > FYI i just saw saw some tweets from Ikan Lan (http://twitter.com/#!/ikai ) > who works with the Google App Engine time, oldest to newest: > > At the Python VM panel. I want to believe alternate VMs are about > more than just benchmarks. Question do not agree with me #pycon > > For instance, I want to see the tooling that can be built around > alternative VMs. Profilers, heap analyzers, better debuggers #pycon > > Python VM implementors think sandboxing is not something that should > be implemented at VM layer. Oh, well. #pycon > > To me, this does not seem to be fitting for PyPy, does it? > Maybe somebody can discuss a bit with him. I still believe that Google > and GAE are a potentially very good application of PyPy tech. > > best, > holger > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > That's a bit confusing IMO. Maciej, Frank, and Dino all said that their VM supports sandboxing, but it shouldn't be implemente din a python specific way because Python is such a big language. Not sure why someone would want Python specific sandboxing. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110311/dc16d957/attachment.htm From holger at merlinux.eu Fri Mar 11 20:07:43 2011 From: holger at merlinux.eu (holger krekel) Date: Fri, 11 Mar 2011 19:07:43 +0000 Subject: [pypy-dev] tooling/sandboxing (remarks from Ikai Lan) In-Reply-To: References: <20110311184654.GR16231@merlinux.eu> Message-ID: <20110311190743.GS16231@merlinux.eu> On Fri, Mar 11, 2011 at 13:48 -0500, Alex Gaynor wrote: > On Fri, Mar 11, 2011 at 1:46 PM, holger krekel wrote: > > > Hi all, especially devs at Pycon, > > > > FYI i just saw saw some tweets from Ikan Lan (http://twitter.com/#!/ikai ) > > who works with the Google App Engine time, oldest to newest: > > > > At the Python VM panel. I want to believe alternate VMs are about > > more than just benchmarks. Question do not agree with me #pycon > > > > For instance, I want to see the tooling that can be built around > > alternative VMs. Profilers, heap analyzers, better debuggers #pycon > > > > Python VM implementors think sandboxing is not something that should > > be implemented at VM layer. Oh, well. #pycon > > > > To me, this does not seem to be fitting for PyPy, does it? > > Maybe somebody can discuss a bit with him. I still believe that Google > > and GAE are a potentially very good application of PyPy tech. > > > > That's a bit confusing IMO. Maciej, Frank, and Dino all said that their VM > supports sandboxing, but it shouldn't be implemente din a python specific > way because Python is such a big language. Not sure why someone would want > Python specific sandboxing. Ah, indeed confusing. I would describe PyPy's sandobxing to be at VM level as opposed to approaches that rather try to limit access to attributes or recursively control by proxies on top of a VM. best, holger From arigo at tunes.org Fri Mar 11 20:16:45 2011 From: arigo at tunes.org (Armin Rigo) Date: Fri, 11 Mar 2011 14:16:45 -0500 Subject: [pypy-dev] tooling/sandboxing (remarks from Ikai Lan) In-Reply-To: <20110311190743.GS16231@merlinux.eu> References: <20110311184654.GR16231@merlinux.eu> <20110311190743.GS16231@merlinux.eu> Message-ID: Hi Holger, Yes, that's what all three representatives of alternate VMs said at the panel: sandboxing at the VM level is cool and supported, but we don't want sandboxing at the language level. As far as I can tell, Ikan Lan just misunderstood it and got the opposite message. Armin From greg at quora.com Sat Mar 12 02:55:33 2011 From: greg at quora.com (Greg Price) Date: Fri, 11 Mar 2011 17:55:33 -0800 Subject: [pypy-dev] [PATCH] Rework borrowed-ref bookkeeping, to fix a crash Message-ID: (I've sent this also as a Bitbucket pull request. Sending it here too because I'm not sure whose attention that reaches, and also for the sake of developing in the open.) We were increfing an object on the first time we borrow a reference to it, then decrefing it on the first time we destroy a borrowed-from container. So if we borrow from two containers, then destroy one, we'd take the refcount back where it started -- possibly zero -- even though the C code still has a legitimate reference. This is demonstrated by test_borrow_destroy in test_borrow.py, added last night. Change the logic to incref the object the first time we borrow it from each container, and decref when any borrowed-from container is destroyed. Also fix a NameError in some code that runs when DEBUG_REFCOUNT is set. pypy/module/cpyext/pyobject.py | 32 +++++++++----------------------- pypy/module/cpyext/test/test_borrow.py | 17 ++++++++++++++++- 2 files changed, 25 insertions(+), 24 deletions(-) diff --git a/pypy/module/cpyext/pyobject.py b/pypy/module/cpyext/pyobject.py index 99aabfb..9e42153 100644 --- a/pypy/module/cpyext/pyobject.py +++ b/pypy/module/cpyext/pyobject.py @@ -144,14 +144,11 @@ class RefcountState: # { w_container -> { w_containee -> None } } # the None entry manages references borrowed during a call to # generic_cpy_call() - self.borrowed_objects = {} - # { addr of containee -> None } # For tests self.non_heaptypes_w = [] def _freeze_(self): - assert not self.borrowed_objects assert self.borrow_mapping == {None: {}} self.py_objects_r2w.clear() # is not valid anymore after translation return False @@ -187,22 +184,19 @@ class RefcountState: """ ref = make_ref(self.space, w_borrowed) obj_ptr = rffi.cast(ADDR, ref) - if obj_ptr not in self.borrowed_objects: - # borrowed_objects owns the reference - self.borrowed_objects[obj_ptr] = None - else: - Py_DecRef(self.space, ref) # already in borrowed list borrowees = self.borrow_mapping.setdefault(w_container, {}) - borrowees[w_borrowed] = None + if w_borrowed in borrowees: + Py_DecRef(self.space, w_borrowed) # cancel incref from make_ref() + else: + borrowees[w_borrowed] = None + return ref def reset_borrowed_references(self): "Used in tests" - while self.borrowed_objects: - addr, _ = self.borrowed_objects.popitem() - w_obj = self.py_objects_r2w[addr] - Py_DecRef(self.space, w_obj) + for w_container, w_borrowed in self.borrow_mapping.items(): + Py_DecRef(self.space, w_borrowed) self.borrow_mapping = {None: {}} def delete_borrower(self, w_obj): @@ -232,17 +226,10 @@ class RefcountState: ref = self.py_objects_w2r.get(w_obj, lltype.nullptr(PyObject.TO)) if not ref: if DEBUG_REFCOUNT: - print >>sys.stderr, "Borrowed object is already gone:", \ - hex(containee) + print >>sys.stderr, "Borrowed object is already gone!" return - containee_ptr = rffi.cast(ADDR, ref) - try: - del self.borrowed_objects[containee_ptr] - except KeyError: - pass - else: - Py_DecRef(self.space, ref) + Py_DecRef(self.space, ref) class InvalidPointerException(Exception): pass @@ -290,7 +277,6 @@ def track_reference(space, py_obj, w_obj, replace=False): if not replace: assert w_obj not in state.py_objects_w2r assert ptr not in state.py_objects_r2w - assert ptr not in state.borrowed_objects state.py_objects_w2r[w_obj] = py_obj if ptr: # init_typeobject() bootstraps with NULL references state.py_objects_r2w[ptr] = w_obj diff --git a/pypy/module/cpyext/test/test_borrow.py b/pypy/module/cpyext/test/test_borrow.py index 25f5101..c903271 100644 --- a/pypy/module/cpyext/test/test_borrow.py +++ b/pypy/module/cpyext/test/test_borrow.py @@ -39,7 +39,6 @@ class AppTestBorrow(AppTestCpythonExtensionBase): assert module.test_borrowing() # the test should not leak def test_borrow_destroy(self): - skip("FIXME") module = self.import_extension('foo', [ ("test_borrow_destroy", "METH_NOARGS", """ @@ -59,3 +58,19 @@ class AppTestBorrow(AppTestCpythonExtensionBase): """), ]) assert module.test_borrow_destroy() == 42 + + def test_double_borrow(self): + module = self.import_extension('foo', [ + ("run", "METH_NOARGS", + """ + PyObject *t = PyTuple_New(1); + PyObject *i = PyInt_FromLong(42); + PyTuple_SetItem(t, 0, i); + i = PyTuple_GetItem(t, 0); + PyTuple_GetItem(t, 0); + Py_DECREF(t); + return PyInt_FromLong(PyInt_AsLong(i)); + """), + ]) + # i.e., there was an InvalidPointerException + assert module.run() == 0 From amauryfa at gmail.com Sat Mar 12 15:40:55 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Sat, 12 Mar 2011 15:40:55 +0100 Subject: [pypy-dev] [PATCH] Rework borrowed-ref bookkeeping, to fix a crash In-Reply-To: References: Message-ID: Hi, 2011/3/12 Greg Price : > + ? ? ? ? ? ? ? ? """ > + ? ? ? ? ? ? ? ? ? ?PyObject *t = PyTuple_New(1); > + ? ? ? ? ? ? ? ? ? ?PyObject *i = PyInt_FromLong(42); > + ? ? ? ? ? ? ? ? ? ?PyTuple_SetItem(t, 0, i); > + ? ? ? ? ? ? ? ? ? ?i = PyTuple_GetItem(t, 0); > + ? ? ? ? ? ? ? ? ? ?PyTuple_GetItem(t, 0); > + ? ? ? ? ? ? ? ? ? ?Py_DECREF(t); > + ? ? ? ? ? ? ? ? ? ?return PyInt_FromLong(PyInt_AsLong(i)); > + ? ? ? ? ? ? ? ? """), This example is wrong: you don't own the reference to int(42); after the tuple is released, PyInt_AsLong(i) is illegal. -- Amaury Forgeot d'Arc From hakan at debian.org Sat Mar 12 19:25:53 2011 From: hakan at debian.org (Hakan Ardo) Date: Sat, 12 Mar 2011 19:25:53 +0100 Subject: [pypy-dev] problem after merging of jit-virtual_state In-Reply-To: <4D7908DD.40503@gmail.com> References: <4D78EC62.8030101@gmail.com> <4D7908DD.40503@gmail.com> Message-ID: Yes, this is probably the VirtualState checking. It will retrace a loop whenever the VirtualState at the end of a bridge differs from the VirtualState at the beginning of the compiled trace (any of the compiled traces). This might indeed produce an identical trace if we are unlucky, but the idea is that this should only happen rarely. This is because the VirtualState at the beginning of a trace is the state of all the OptValue of the inputargs produced during the optimization of the trace. This does not have to be the most general state for which the trace is usable (which would be hard to calculate I'm afraid). A few cases that would (most likely) result in identical traces are salvaged in NotVirtualInfo._generate_guards by producing some extra gurads at the end of a bridge to make the VirtualState there match the VirtualState of a compiled trace. This is however only done if the guards would (most likely) not fail for the traced iteration. I'll look into what's happening in this particular test... On Thu, Mar 10, 2011 at 6:22 PM, Antonio Cuni wrote: > On 10/03/11 16:21, Antonio Cuni wrote: > >> I did a bit of manual binary search and discovered that the test started >> failing between r42273:b8bb5d085684 and r42305:3fe93dd500bd. Looking at the >> commit, it seems that the only one relevant to the problem is >> r42305:3fe93dd500bd, i.e. the merging of jit-virtual_state. > > I confirm that the problem is the merging of jit-virtual_state: I just tried > the revision just before the merge, and the test passes. > > ciao, > Anto > -- H?kan Ard? From anto.cuni at gmail.com Sat Mar 12 20:34:11 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Sat, 12 Mar 2011 20:34:11 +0100 Subject: [pypy-dev] problem after merging of jit-virtual_state In-Reply-To: References: <4D78EC62.8030101@gmail.com> <4D7908DD.40503@gmail.com> Message-ID: <4D7BCAB3.7020906@gmail.com> Hi Hakan, On 12/03/11 19:25, Hakan Ardo wrote: > Yes, this is probably the VirtualState checking. It will retrace a > loop whenever the VirtualState at the end of a bridge differs from the > VirtualState at the beginning of the compiled trace (any of the > compiled traces). This might indeed produce an identical trace if we > are unlucky, but the idea is that this should only happen rarely. ok, that's clear. So, hopefully this particular example looks a bit bad, but in general it should not be an issue. It'd be nice to have a way to check this thesis, but I agree that it's a bit hard. > This is because the VirtualState at the beginning of a trace is the > state of all the OptValue of the inputargs produced during the > optimization of the trace. This does not have to be the most general > state for which the trace is usable (which would be hard to calculate > I'm afraid). so, if I understand correctly, this is what happens: 1. we trace, optimize and compile loop A 2. after a while, we trace, optimize a compile a bridge B which then jumps back to A; by chance, the bridge looks the same as the loop Am I right? > A few cases that would (most likely) result in identical traces are > salvaged in NotVirtualInfo._generate_guards by producing some extra > gurads at the end of a bridge to make the VirtualState there match the > VirtualState of a compiled trace. This is however only done if the > guards would (most likely) not fail for the traced iteration. > > I'll look into what's happening in this particular test... I just did a quick check because I'm in a hurry, but from what I see we get three actual *loops*, not bridges. ciao, Anto From hakan at debian.org Sat Mar 12 22:59:11 2011 From: hakan at debian.org (Hakan Ardo) Date: Sat, 12 Mar 2011 22:59:11 +0100 Subject: [pypy-dev] problem after merging of jit-virtual_state In-Reply-To: <4D7BCAB3.7020906@gmail.com> References: <4D78EC62.8030101@gmail.com> <4D7908DD.40503@gmail.com> <4D7BCAB3.7020906@gmail.com> Message-ID: On Sat, Mar 12, 2011 at 8:34 PM, Antonio Cuni wrote: > Hi Hakan, > > On 12/03/11 19:25, Hakan Ardo wrote: >> Yes, this is probably the VirtualState checking. It will retrace a >> loop whenever the VirtualState at the end of a bridge differs from the >> VirtualState at the beginning of the compiled trace (any of the >> compiled traces). This might indeed produce an identical trace if we >> are unlucky, but the idea is that this should only happen rarely. > > ok, that's clear. So, hopefully this particular example looks a bit bad, but > in general it should not be an issue. It'd be nice to have a way to check this > thesis, but I agree that it's a bit hard. We should probably log the VirtualState together with the produced loops and bridges. That would allow us to see how they differ when a new version of a loop is traced. There are __repr__ methods I've been using for that while debugging. They might need some rpythonizing to translate though--- > >> This is because the VirtualState ?at the beginning of a trace is the >> state of all the OptValue of the inputargs produced during the >> optimization of the trace. This does not have to be the most general >> state for which the trace is usable (which would be hard to calculate >> I'm afraid). > > so, if I understand correctly, this is what happens: > > 1. we trace, optimize and compile loop A > > 2. after a while, we trace, optimize a compile a bridge B which then jumps > back to A; by chance, the bridge looks the same as the loop > > Am I right? Maybe, I've not had the chance to look into any details yet. I'll do that tomorrow... > >> A few cases that would (most likely) result in identical traces are >> salvaged in NotVirtualInfo._generate_guards by producing some extra >> gurads at the end of a bridge to make the VirtualState there match the >> VirtualState of a compiled trace. This is however only done if the >> guards would (most likely) not fail for the traced iteration. >> >> I'll look into what's happening in this particular test... > > I just did a quick check because I'm in a hurry, but from what I see we get > three actual *loops*, not bridges. So if it's the same loop traced several times they should all have the same preamble, and the preamble would have two bridges leading to the two second versions of the loop. The preamble and it's two bridges should end with different VirtualState's. The loops should be specialized to the different VirtualState's, but if the VirtualState's are similar enough (but not equal) they might consist of the exact same operations. If there are 3 preambles for the same loop, there is a bug somewhere... -- H?kan Ard? From pjenvey at underboss.org Sat Mar 12 23:21:01 2011 From: pjenvey at underboss.org (Philip Jenvey) Date: Sat, 12 Mar 2011 17:21:01 -0500 Subject: [pypy-dev] wrong precedence of __radd__ vs list __iadd__ In-Reply-To: References: Message-ID: On Mar 10, 2011, at 3:53 PM, Armin Rigo wrote: > Hi Greg, > > On Wed, Mar 9, 2011 at 7:13 PM, Greg Price wrote: >> The following program works in CPython, but fails in PyPy: > > This is (if we are positive) an internal implementation detail and (if > we are negative) a bug in CPython. Jython passes the first example. Shouldn't pypy's inplace_add__List_ANY be returning NotImplemented (or whatever the pypy equiv. would be, FailedToImplement?) instead of raising the exception? To allow the binop rules to continue. We fail the 2nd example like PyPy does but that's a different problem. -- Philip Jenvey From greg at quora.com Sat Mar 12 23:34:30 2011 From: greg at quora.com (Greg Price) Date: Sat, 12 Mar 2011 14:34:30 -0800 Subject: [pypy-dev] [PATCH] Rework borrowed-ref bookkeeping, to fix a crash In-Reply-To: References: Message-ID: On Sat, Mar 12, 2011 at 6:40 AM, Amaury Forgeot d'Arc wrote: > 2011/3/12 Greg Price : >> + ? ? ? ? ? ? ? ? """ >> + ? ? ? ? ? ? ? ? ? ?PyObject *t = PyTuple_New(1); >> + ? ? ? ? ? ? ? ? ? ?PyObject *i = PyInt_FromLong(42); >> + ? ? ? ? ? ? ? ? ? ?PyTuple_SetItem(t, 0, i); >> + ? ? ? ? ? ? ? ? ? ?i = PyTuple_GetItem(t, 0); >> + ? ? ? ? ? ? ? ? ? ?PyTuple_GetItem(t, 0); >> + ? ? ? ? ? ? ? ? ? ?Py_DECREF(t); >> + ? ? ? ? ? ? ? ? ? ?return PyInt_FromLong(PyInt_AsLong(i)); >> + ? ? ? ? ? ? ? ? """), > > This example is wrong: you don't own the reference to int(42); > after the tuple is released, PyInt_AsLong(i) is illegal. Yes, that's the intent of the test; see the comment just below. An earlier version of my patch would have left 'i' alive, so I wanted to check that in a test. This C code returns 42 if the int stays alive, 0 if it is destroyed when it should be. This way of testing is a little ugly, though, so if you can suggest a better way to check for the InvalidPointerException I'd be glad to hear it. Greg From herve.coatanhay at gmail.com Sun Mar 13 03:11:40 2011 From: herve.coatanhay at gmail.com (=?utf-8?Q?Herv=C3=A9_Coatanhay?=) Date: Sat, 12 Mar 2011 21:11:40 -0500 Subject: [pypy-dev] translation with --stackless option Message-ID: <790BCE1130C7404C94FDFFB77C7BEFA6@gmail.com> After a clone of the repository, whereas i can do the following translation with just some warnings: python translate.py --opt=jit targetpypystandalone.py I got an exception with that one: python translate.py --stackless targetpypystandalone.py As the --stackless option is not detailed in documentation maybe I should had something along with it ? I tried this on both Mac OS X and a CentOS Linux box, the attach file contains the traceback of the translation. Any idea on what i am doing wrong ? -- Herv? Coatanhay -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110312/893747be/attachment.htm -------------- next part -------------- A non-text attachment was scrubbed... Name: pypy_traceback.txt Type: application/octet-stream Size: 3544 bytes Desc: not available Url : http://codespeak.net/pipermail/pypy-dev/attachments/20110312/893747be/attachment.obj From fijall at gmail.com Sun Mar 13 03:26:21 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 12 Mar 2011 21:26:21 -0500 Subject: [pypy-dev] translation with --stackless option In-Reply-To: <790BCE1130C7404C94FDFFB77C7BEFA6@gmail.com> References: <790BCE1130C7404C94FDFFB77C7BEFA6@gmail.com> Message-ID: On Sat, Mar 12, 2011 at 9:11 PM, Herv? Coatanhay wrote: > After a clone of the repository,?whereas i can do the following translation > with just some warnings: > python translate.py --opt=jit targetpypystandalone.py > I got an exception with that one: > python translate.py --stackless targetpypystandalone.py > As the --stackless option is not detailed in documentation maybe I should > had something along with it ? > I tried this on both Mac OS X and a CentOS Linux box, the attach file > contains the traceback of the translation. > Any idea on what i am doing wrong ? It's broken. We should fix it some time, preferably soon. That said, there is a feature missing which is stackless working with JIT. We're not actively working on it but this is something that would be a useful contribution to the project and not too much work. But yes, we should fix the failure first,. > -- > Herv? Coatanhay > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From herve.coatanhay at gmail.com Sun Mar 13 04:30:34 2011 From: herve.coatanhay at gmail.com (=?UTF-8?Q?Herv=C3=A9_Coatanhay?=) Date: Sat, 12 Mar 2011 22:30:34 -0500 Subject: [pypy-dev] translation with --stackless option In-Reply-To: References: <790BCE1130C7404C94FDFFB77C7BEFA6@gmail.com> Message-ID: On Sat, Mar 12, 2011 at 9:26 PM, Maciej Fijalkowski wrote: > On Sat, Mar 12, 2011 at 9:11 PM, Herv? Coatanhay > wrote: >> After a clone of the repository,?whereas i can do the following translation >> with just some warnings: >> python translate.py --opt=jit targetpypystandalone.py >> I got an exception with that one: >> python translate.py --stackless targetpypystandalone.py >> As the --stackless option is not detailed in documentation maybe I should >> had something along with it ? >> I tried this on both Mac OS X and a CentOS Linux box, the attach file >> contains the traceback of the translation. >> Any idea on what i am doing wrong ? > > It's broken. > > We should fix it some time, preferably soon. That said, there is a > feature missing which is stackless working with JIT. We're not > actively working on it but this is something that would be a useful > contribution to the project and not too much work. But yes, we should > fix the failure first,. Thanks for the answer. I guess I'll have to wait a bit before giving a try at pypy as a stackless implementation. Maybe while waiting, I should look at documentation about JIT anyway. -- Herv? Coatanhay From lac at openend.se Sun Mar 13 05:08:00 2011 From: lac at openend.se (Laura Creighton) Date: Sun, 13 Mar 2011 05:08:00 +0100 Subject: [pypy-dev] wrong precedence of __radd__ vs list __iadd__ In-Reply-To: Message from Philip Jenvey of "Sat, 12 Mar 2011 17:21:01 EST." References: Message-ID: <201103130408.p2D480Zd013368@theraft.openend.se> In a message of Sat, 12 Mar 2011 17:21:01 EST, Philip Jenvey writes: > >On Mar 10, 2011, at 3:53 PM, Armin Rigo wrote: > >> Hi Greg, >> >> On Wed, Mar 9, 2011 at 7:13 PM, Greg Price wrote: >>> The following program works in CPython, but fails in PyPy: >> >> This is (if we are positive) an internal implementation detail and (if >> we are negative) a bug in CPython. > >Jython passes the first example. Shouldn't pypy's inplace_add__List_ANY b >e returning NotImplemented (or whatever the pypy equiv. would be, FailedT >oImplement?) instead of raising the exception? To allow the binop rules t >o continue. > >We fail the 2nd example like PyPy does but that's a different problem. > >-- >Philip Jenvey I posted a note about this to python-dev http://mail.python.org/pipermail/python-dev/2011-March/109130.html and the reaction on python-dev seems to be unanimous that Cpython is broken, and that pypy is doing the correct thing. Current discussion is about whether to just fix it, as a bug in Cpython or whether to deprecate it and then stop it, as a kindness to those who relied on it, or to backport the fix to 2.7 as well. Laura From hakan at debian.org Sun Mar 13 11:12:56 2011 From: hakan at debian.org (Hakan Ardo) Date: Sun, 13 Mar 2011 11:12:56 +0100 Subject: [pypy-dev] problem after merging of jit-virtual_state In-Reply-To: References: <4D78EC62.8030101@gmail.com> <4D7908DD.40503@gmail.com> <4D7BCAB3.7020906@gmail.com> Message-ID: Hi, this is what happens here: 1. The inner loop is traced and Loop0 is produced with preamble Loop1 2. A bridge from Guard3 (the test in the while) back to Loop0 is traced (i.e the remaining parts of the outer loop) 3. At the end of this bridge the VirtualState does not match the VirtualState of Loop0, so the loop is retraced 4. The VirtualState of the newly traced version of the loop does not match the VirtualState at the end of the bridge so the bridge has to jump to the preamble instead of jumping to the new specialized version of the loop. 5. A bridge from Guard6 (signal/thread counter) is traced and the same thing happens for this bridge. This means that the additional two versions of the loop will never be used and should hopefully be reomved by the gc... So there are two issues: A. The additional specialized versions created does not become usable. This is the issue I'm working on in the jit-usable_retrace branch. The idea there is to have the retrace inherit the OptValue's of the jumpargs at the end of the bridge. This will become a fairly large change functionality wise... B. The VirtualStates' s differs in the first place forcing a retrace. This is probably fixable by introducing some more cases in NotVirtualInfo._generate_guards(). The jit-usable_retrace branch contains more cases than trunk, don't know if those are enough for this test though... Note however that jit/metainterp/test/test_nested_loops_discovered_by_bridge in test_loop_unroll.py, which conatins the same loop for a simple interpreter, does work nicely, wihtout the issues above. On Sat, Mar 12, 2011 at 10:59 PM, Hakan Ardo wrote: > On Sat, Mar 12, 2011 at 8:34 PM, Antonio Cuni wrote: >> Hi Hakan, >> >> On 12/03/11 19:25, Hakan Ardo wrote: >>> Yes, this is probably the VirtualState checking. It will retrace a >>> loop whenever the VirtualState at the end of a bridge differs from the >>> VirtualState at the beginning of the compiled trace (any of the >>> compiled traces). This might indeed produce an identical trace if we >>> are unlucky, but the idea is that this should only happen rarely. >> >> ok, that's clear. So, hopefully this particular example looks a bit bad, but >> in general it should not be an issue. It'd be nice to have a way to check this >> thesis, but I agree that it's a bit hard. > > We should probably log the VirtualState together with the produced > loops and bridges. That would allow us to see how they differ when a > new version of a loop is traced. There are __repr__ methods I've been > using for that while debugging. They might need some rpythonizing to > translate though--- > >> >>> This is because the VirtualState ?at the beginning of a trace is the >>> state of all the OptValue of the inputargs produced during the >>> optimization of the trace. This does not have to be the most general >>> state for which the trace is usable (which would be hard to calculate >>> I'm afraid). >> >> so, if I understand correctly, this is what happens: >> >> 1. we trace, optimize and compile loop A >> >> 2. after a while, we trace, optimize a compile a bridge B which then jumps >> back to A; by chance, the bridge looks the same as the loop >> >> Am I right? > > Maybe, I've not had the chance to look into any details yet. I'll do > that tomorrow... > >> >>> A few cases that would (most likely) result in identical traces are >>> salvaged in NotVirtualInfo._generate_guards by producing some extra >>> gurads at the end of a bridge to make the VirtualState there match the >>> VirtualState of a compiled trace. This is however only done if the >>> guards would (most likely) not fail for the traced iteration. >>> >>> I'll look into what's happening in this particular test... >> >> I just did a quick check because I'm in a hurry, but from what I see we get >> three actual *loops*, not bridges. > > So if it's the same loop traced several times they should all have the > same preamble, and the preamble would have two bridges leading to the > two second versions of the loop. The preamble and it's two bridges > should end with different VirtualState's. The loops should ?be > specialized to the different VirtualState's, but if the VirtualState's > are similar enough (but not equal) they might consist of the exact > same operations. > > If there are 3 preambles for the same loop, there is a bug somewhere... > > -- > H?kan Ard? > -- H?kan Ard? From arigo at tunes.org Sun Mar 13 11:19:30 2011 From: arigo at tunes.org (Armin Rigo) Date: Sun, 13 Mar 2011 06:19:30 -0400 Subject: [pypy-dev] wrong precedence of __radd__ vs list __iadd__ In-Reply-To: References: Message-ID: Hi Philip, On Sat, Mar 12, 2011 at 5:21 PM, Philip Jenvey wrote: > Jython passes the first example. Shouldn't pypy's inplace_add__List_ANY be returning NotImplemented (or whatever the pypy equiv. would be, FailedToImplement?) instead of raising the exception? To allow the binop rules to continue. Good idea. Right now, both on CPython and on PyPy, [].__iadd__(5) raises TypeError (instead of returning NotImplemented), but [].__add__(5) already behaves differently: it raises TypeError on CPython and returns NotImplemented on PyPy, with the result that []+Bar() does call the __radd__() method on Bar(). So having the same difference in the __iadd__() method looks like a useful compromise. This is particularly true given that it already works on all sequence types different from lists, because these don't have an __iadd__() method at all. Armin From arigo at tunes.org Sun Mar 13 12:05:42 2011 From: arigo at tunes.org (Armin Rigo) Date: Sun, 13 Mar 2011 07:05:42 -0400 Subject: [pypy-dev] wrong precedence of __radd__ vs list __iadd__ In-Reply-To: References: Message-ID: Hi, Checked in 72ce40f4803c. Greg, about the second example you reported: class C(object): def __rmul__(self, other): other *= 2 return other def __index__(self): return 3 print [1] * C() # -> CPython: [1,1] PyPy: [1,1,1] This is obscure. It works that way because in that particular case, CPython first tries __mul__/__rmul__ and then tries the internal sq_repeat slot of the list (which calls C.__index__()). I think there is just no way to reproduce this behavior while remaining sane :-/ Armin From lac at openend.se Sun Mar 13 13:24:27 2011 From: lac at openend.se (Laura Creighton) Date: Sun, 13 Mar 2011 13:24:27 +0100 Subject: [pypy-dev] wrong precedence of __radd__ vs list __iadd__ In-Reply-To: Message from Armin Rigo of "Sun, 13 Mar 2011 07:05:42 -0400." References: Message-ID: <201103131224.p2DCORkq025186@theraft.openend.se> Note: I told python-dev about this behaviour http://mail.python.org/pipermail/python-dev/2011-March/109130.html thinking to get a 'NotPortableWarning'. Now python-dev is hot for making this a bug in Cpython as well. Laura From hakan at debian.org Sun Mar 13 13:35:27 2011 From: hakan at debian.org (Hakan Ardo) Date: Sun, 13 Mar 2011 13:35:27 +0100 Subject: [pypy-dev] problem after merging of jit-virtual_state In-Reply-To: References: <4D78EC62.8030101@gmail.com> <4D7908DD.40503@gmail.com> <4D7BCAB3.7020906@gmail.com> Message-ID: Hi agian, I digged a bit more, and it turns out that the issue B below is actually a special case of issue A :) The first version of the loop becomes specialized for the variables j and x being virtual integers. After the bridge passing through the the outer loop once, the variable i also becomes a vritual integer. At this point a new specialized version of the loop is generated now for the case of i, j and x being virtual integers. The new version will contain the exact same operations since the trace does not touch the variable i. This is actually the intended behavior. Even though i is not touched in the loop it might be used in some bridge from the loop and this would allow it to stay virtual within that bridge. The third version of the loop I think is generated because the information that n is nonnull is not inherited by the bridge from the loop. This is worked around in the jit-usable_retrace branch by producing an extra guard_nonnull() at the end of the bridge instead of retracing. That fix could be easily cheery-picked into default as well. On Sun, Mar 13, 2011 at 11:12 AM, Hakan Ardo wrote: > Hi, > this is what happens here: > > 1. The inner loop is traced and Loop0 is produced with preamble Loop1 > > 2. A bridge from Guard3 (the test in the while) back to Loop0 is > traced (i.e the remaining parts of the outer loop) > > 3. At the end of this bridge the VirtualState does not match the > VirtualState of Loop0, so the loop is retraced > > 4. The VirtualState of the newly traced version of the loop does not > match the VirtualState at the end of the bridge so the bridge has to > jump to the preamble instead of jumping to the new specialized version > of the loop. > > 5. A bridge from Guard6 (signal/thread counter) is traced and the same > thing happens for this bridge. > > This means that the additional two versions of the loop will never be > used and should hopefully be reomved by the gc... > > So there are two issues: > > A. The additional specialized versions created does not become usable. > This is the issue I'm working on in the jit-usable_retrace branch. The > idea there is to have the retrace inherit the OptValue's of the > jumpargs at the end of the bridge. This will become a fairly large > change functionality wise... > > B. The VirtualStates' s differs in the first place forcing a retrace. > This is probably fixable by introducing some more cases in > NotVirtualInfo._generate_guards(). The jit-usable_retrace branch > contains more cases than trunk, don't know if those are enough for > this test though... > > Note however that > jit/metainterp/test/test_nested_loops_discovered_by_bridge in > test_loop_unroll.py, which conatins the same loop for a simple > interpreter, does work nicely, wihtout the issues above. > > On Sat, Mar 12, 2011 at 10:59 PM, Hakan Ardo wrote: >> On Sat, Mar 12, 2011 at 8:34 PM, Antonio Cuni wrote: >>> Hi Hakan, >>> >>> On 12/03/11 19:25, Hakan Ardo wrote: >>>> Yes, this is probably the VirtualState checking. It will retrace a >>>> loop whenever the VirtualState at the end of a bridge differs from the >>>> VirtualState at the beginning of the compiled trace (any of the >>>> compiled traces). This might indeed produce an identical trace if we >>>> are unlucky, but the idea is that this should only happen rarely. >>> >>> ok, that's clear. So, hopefully this particular example looks a bit bad, but >>> in general it should not be an issue. It'd be nice to have a way to check this >>> thesis, but I agree that it's a bit hard. >> >> We should probably log the VirtualState together with the produced >> loops and bridges. That would allow us to see how they differ when a >> new version of a loop is traced. There are __repr__ methods I've been >> using for that while debugging. They might need some rpythonizing to >> translate though--- >> >>> >>>> This is because the VirtualState ?at the beginning of a trace is the >>>> state of all the OptValue of the inputargs produced during the >>>> optimization of the trace. This does not have to be the most general >>>> state for which the trace is usable (which would be hard to calculate >>>> I'm afraid). >>> >>> so, if I understand correctly, this is what happens: >>> >>> 1. we trace, optimize and compile loop A >>> >>> 2. after a while, we trace, optimize a compile a bridge B which then jumps >>> back to A; by chance, the bridge looks the same as the loop >>> >>> Am I right? >> >> Maybe, I've not had the chance to look into any details yet. I'll do >> that tomorrow... >> >>> >>>> A few cases that would (most likely) result in identical traces are >>>> salvaged in NotVirtualInfo._generate_guards by producing some extra >>>> gurads at the end of a bridge to make the VirtualState there match the >>>> VirtualState of a compiled trace. This is however only done if the >>>> guards would (most likely) not fail for the traced iteration. >>>> >>>> I'll look into what's happening in this particular test... >>> >>> I just did a quick check because I'm in a hurry, but from what I see we get >>> three actual *loops*, not bridges. >> >> So if it's the same loop traced several times they should all have the >> same preamble, and the preamble would have two bridges leading to the >> two second versions of the loop. The preamble and it's two bridges >> should end with different VirtualState's. The loops should ?be >> specialized to the different VirtualState's, but if the VirtualState's >> are similar enough (but not equal) they might consist of the exact >> same operations. >> >> If there are 3 preambles for the same loop, there is a bug somewhere... >> >> -- >> H?kan Ard? >> > > > > -- > H?kan Ard? > -- H?kan Ard? From arigo at tunes.org Sun Mar 13 14:59:42 2011 From: arigo at tunes.org (Armin Rigo) Date: Sun, 13 Mar 2011 09:59:42 -0400 Subject: [pypy-dev] wrong precedence of __radd__ vs list __iadd__ In-Reply-To: <201103131224.p2DCORkq025186@theraft.openend.se> References: <201103131224.p2DCORkq025186@theraft.openend.se> Message-ID: Hi Laura, On Sun, Mar 13, 2011 at 8:24 AM, Laura Creighton wrote: > Note: I told python-dev about this behaviour Yes, I replied on the CPython bug tracker. The fix I checked into PyPy is for one particular case, but there are other cases that remain as differences of behavior where CPython's is arguably wrong and PyPy's cannot easily match it. Armin. From lac at openend.se Sun Mar 13 20:04:21 2011 From: lac at openend.se (Laura Creighton) Date: Sun, 13 Mar 2011 20:04:21 +0100 Subject: [pypy-dev] possibly of use for our documentation Message-ID: <201103131904.p2DJ4LTi028359@theraft.openend.se> I went and saw the pybookbuilder project's poster. https://launchpad.net/pybookbuilder This is Jeff Elkner and his extremely smart high school students. They have a neat plugin for doing Latex inside sphinx, and to combine it with ReST to make, well, books of documentation. I don't want to lose this idea. Laura From anto.cuni at gmail.com Mon Mar 14 11:49:25 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Mon, 14 Mar 2011 11:49:25 +0100 Subject: [pypy-dev] problem after merging of jit-virtual_state In-Reply-To: References: <4D78EC62.8030101@gmail.com> <4D7908DD.40503@gmail.com> <4D7BCAB3.7020906@gmail.com> Message-ID: <4D7DF2B5.50506@gmail.com> Hi Hakan, thank you for the deep explanation. Now I understand what's going on :-) So, I changed test_pypy_c_new to add a sys.setcheckinterval(some-huge-number), so that the bridge from the signal/thread counter is never created and we can forget about it. Now, if I understand correctly, the two remaining loops are one for the case "i non virtual" and the other for the case "i virtual", although both lead to the same operations. I think this is the expected behavior in this case, so are you ok if I just fix test_f1 to expect two loops? ciao, Anto On 13/03/11 11:12, Hakan Ardo wrote: > Hi, > this is what happens here: > > 1. The inner loop is traced and Loop0 is produced with preamble Loop1 > > 2. A bridge from Guard3 (the test in the while) back to Loop0 is > traced (i.e the remaining parts of the outer loop) > > 3. At the end of this bridge the VirtualState does not match the > VirtualState of Loop0, so the loop is retraced > > 4. The VirtualState of the newly traced version of the loop does not > match the VirtualState at the end of the bridge so the bridge has to > jump to the preamble instead of jumping to the new specialized version > of the loop. > > 5. A bridge from Guard6 (signal/thread counter) is traced and the same > thing happens for this bridge. > > This means that the additional two versions of the loop will never be > used and should hopefully be reomved by the gc... > > So there are two issues: > > A. The additional specialized versions created does not become usable. > This is the issue I'm working on in the jit-usable_retrace branch. The > idea there is to have the retrace inherit the OptValue's of the > jumpargs at the end of the bridge. This will become a fairly large > change functionality wise... > > B. The VirtualStates' s differs in the first place forcing a retrace. > This is probably fixable by introducing some more cases in > NotVirtualInfo._generate_guards(). The jit-usable_retrace branch > contains more cases than trunk, don't know if those are enough for > this test though... > > Note however that > jit/metainterp/test/test_nested_loops_discovered_by_bridge in > test_loop_unroll.py, which conatins the same loop for a simple > interpreter, does work nicely, wihtout the issues above. > > On Sat, Mar 12, 2011 at 10:59 PM, Hakan Ardo wrote: >> On Sat, Mar 12, 2011 at 8:34 PM, Antonio Cuni wrote: >>> Hi Hakan, >>> >>> On 12/03/11 19:25, Hakan Ardo wrote: >>>> Yes, this is probably the VirtualState checking. It will retrace a >>>> loop whenever the VirtualState at the end of a bridge differs from the >>>> VirtualState at the beginning of the compiled trace (any of the >>>> compiled traces). This might indeed produce an identical trace if we >>>> are unlucky, but the idea is that this should only happen rarely. >>> >>> ok, that's clear. So, hopefully this particular example looks a bit bad, but >>> in general it should not be an issue. It'd be nice to have a way to check this >>> thesis, but I agree that it's a bit hard. >> >> We should probably log the VirtualState together with the produced >> loops and bridges. That would allow us to see how they differ when a >> new version of a loop is traced. There are __repr__ methods I've been >> using for that while debugging. They might need some rpythonizing to >> translate though--- >> >>> >>>> This is because the VirtualState at the beginning of a trace is the >>>> state of all the OptValue of the inputargs produced during the >>>> optimization of the trace. This does not have to be the most general >>>> state for which the trace is usable (which would be hard to calculate >>>> I'm afraid). >>> >>> so, if I understand correctly, this is what happens: >>> >>> 1. we trace, optimize and compile loop A >>> >>> 2. after a while, we trace, optimize a compile a bridge B which then jumps >>> back to A; by chance, the bridge looks the same as the loop >>> >>> Am I right? >> >> Maybe, I've not had the chance to look into any details yet. I'll do >> that tomorrow... >> >>> >>>> A few cases that would (most likely) result in identical traces are >>>> salvaged in NotVirtualInfo._generate_guards by producing some extra >>>> gurads at the end of a bridge to make the VirtualState there match the >>>> VirtualState of a compiled trace. This is however only done if the >>>> guards would (most likely) not fail for the traced iteration. >>>> >>>> I'll look into what's happening in this particular test... >>> >>> I just did a quick check because I'm in a hurry, but from what I see we get >>> three actual *loops*, not bridges. >> >> So if it's the same loop traced several times they should all have the >> same preamble, and the preamble would have two bridges leading to the >> two second versions of the loop. The preamble and it's two bridges >> should end with different VirtualState's. The loops should be >> specialized to the different VirtualState's, but if the VirtualState's >> are similar enough (but not equal) they might consist of the exact >> same operations. >> >> If there are 3 preambles for the same loop, there is a bug somewhere... >> >> -- >> H?kan Ard? >> > > > From hakan at debian.org Mon Mar 14 11:52:55 2011 From: hakan at debian.org (Hakan Ardo) Date: Mon, 14 Mar 2011 11:52:55 +0100 Subject: [pypy-dev] problem after merging of jit-virtual_state In-Reply-To: <4D7DF2B5.50506@gmail.com> References: <4D78EC62.8030101@gmail.com> <4D7908DD.40503@gmail.com> <4D7BCAB3.7020906@gmail.com> <4D7DF2B5.50506@gmail.com> Message-ID: On Mon, Mar 14, 2011 at 11:49 AM, Antonio Cuni wrote: > Hi Hakan, > thank you for the deep explanation. Now I understand what's going on :-) > > So, I changed test_pypy_c_new to add a sys.setcheckinterval(some-huge-number), > so that the bridge from the signal/thread counter is never created and we can > forget about it. > > Now, if I understand correctly, the two remaining loops are one for the case > "i non virtual" and the other for the case "i virtual", although both lead to > the same operations. I think this is the expected behavior in this case, so > are you ok if I just fix test_f1 to expect two loops? Yes > > ciao, > Anto > > On 13/03/11 11:12, Hakan Ardo wrote: >> Hi, >> this is what happens here: >> >> 1. The inner loop is traced and Loop0 is produced with preamble Loop1 >> >> 2. A bridge from Guard3 (the test in the while) back to Loop0 is >> traced (i.e the remaining parts of the outer loop) >> >> 3. At the end of this bridge the VirtualState does not match the >> VirtualState of Loop0, so the loop is retraced >> >> 4. The VirtualState of the newly traced version of the loop does not >> match the VirtualState at the end of the bridge so the bridge has to >> jump to the preamble instead of jumping to the new specialized version >> of the loop. >> >> 5. A bridge from Guard6 (signal/thread counter) is traced and the same >> thing happens for this bridge. >> >> This means that the additional two versions of the loop will never be >> used and should hopefully be reomved by the gc... >> >> So there are two issues: >> >> A. The additional specialized versions created does not become usable. >> This is the issue I'm working on in the jit-usable_retrace branch. The >> idea there is to have the retrace inherit the OptValue's of the >> jumpargs at the end of the bridge. This will become a fairly large >> change functionality wise... >> >> B. The VirtualStates' s differs in the first place forcing a retrace. >> This is probably fixable by introducing some more cases in >> NotVirtualInfo._generate_guards(). The jit-usable_retrace branch >> contains more cases than trunk, don't know if those are enough for >> this test though... >> >> Note however that >> jit/metainterp/test/test_nested_loops_discovered_by_bridge in >> test_loop_unroll.py, which conatins the same loop for a simple >> interpreter, does work nicely, wihtout the issues above. >> >> On Sat, Mar 12, 2011 at 10:59 PM, Hakan Ardo wrote: >>> On Sat, Mar 12, 2011 at 8:34 PM, Antonio Cuni wrote: >>>> Hi Hakan, >>>> >>>> On 12/03/11 19:25, Hakan Ardo wrote: >>>>> Yes, this is probably the VirtualState checking. It will retrace a >>>>> loop whenever the VirtualState at the end of a bridge differs from the >>>>> VirtualState at the beginning of the compiled trace (any of the >>>>> compiled traces). This might indeed produce an identical trace if we >>>>> are unlucky, but the idea is that this should only happen rarely. >>>> >>>> ok, that's clear. So, hopefully this particular example looks a bit bad, but >>>> in general it should not be an issue. It'd be nice to have a way to check this >>>> thesis, but I agree that it's a bit hard. >>> >>> We should probably log the VirtualState together with the produced >>> loops and bridges. That would allow us to see how they differ when a >>> new version of a loop is traced. There are __repr__ methods I've been >>> using for that while debugging. They might need some rpythonizing to >>> translate though--- >>> >>>> >>>>> This is because the VirtualState ?at the beginning of a trace is the >>>>> state of all the OptValue of the inputargs produced during the >>>>> optimization of the trace. This does not have to be the most general >>>>> state for which the trace is usable (which would be hard to calculate >>>>> I'm afraid). >>>> >>>> so, if I understand correctly, this is what happens: >>>> >>>> 1. we trace, optimize and compile loop A >>>> >>>> 2. after a while, we trace, optimize a compile a bridge B which then jumps >>>> back to A; by chance, the bridge looks the same as the loop >>>> >>>> Am I right? >>> >>> Maybe, I've not had the chance to look into any details yet. I'll do >>> that tomorrow... >>> >>>> >>>>> A few cases that would (most likely) result in identical traces are >>>>> salvaged in NotVirtualInfo._generate_guards by producing some extra >>>>> gurads at the end of a bridge to make the VirtualState there match the >>>>> VirtualState of a compiled trace. This is however only done if the >>>>> guards would (most likely) not fail for the traced iteration. >>>>> >>>>> I'll look into what's happening in this particular test... >>>> >>>> I just did a quick check because I'm in a hurry, but from what I see we get >>>> three actual *loops*, not bridges. >>> >>> So if it's the same loop traced several times they should all have the >>> same preamble, and the preamble would have two bridges leading to the >>> two second versions of the loop. The preamble and it's two bridges >>> should end with different VirtualState's. The loops should ?be >>> specialized to the different VirtualState's, but if the VirtualState's >>> are similar enough (but not equal) they might consist of the exact >>> same operations. >>> >>> If there are 3 preambles for the same loop, there is a bug somewhere... >>> >>> -- >>> H?kan Ard? >>> >> >> >> > > -- H?kan Ard? From lac at openend.se Mon Mar 14 19:13:34 2011 From: lac at openend.se (Laura Creighton) Date: Mon, 14 Mar 2011 19:13:34 +0100 Subject: [pypy-dev] Thinking about the GIL Message-ID: <201103141813.p2EIDY9C027808@theraft.openend.se> Robert Hancock hosted a bof at pycon about concurrency and multiprocessing. I went there looking to find out how other people were doing things, especially looking for information about how other languages handled things. It would be nice to kill the GIL, if only we knew of a brilliant way to do this. Unfortunately, I was one year to late for this discussion. This is what Robert Hancock David Beazley, Peter Portante and others discussed at Pycon _last year_. So I asked Robert Hancock for the notes he took then. (I continue after this forwarded message) ------- Forwarded Message Return-Path: hancock.robert at gmail.com Delivery-Date: Mon Mar 14 17:09:51 2011 Return-Path: Subject: Re: please send me the notes you took last year To: Laura Creighton These are the books that I mentioned: Machine Learning: An Algorithmic Perspective http://www.amazon.com/gp/product/1420067184 I found this more approachable than the Bishop and a number of examples are in Python. Introduction to Data Mining http://www.amazon.com/gp/product/0321321367 I've only started this, but it is nice with David Mease's Google Tech Talk series. http://www.youtube.com/watch?v=zRsMEl6PHhM 1. Make all IO non-blocking and mediate the processes like greenlets. This does not allow you take advantage of the OS level thread scheduler which is far more sophisticated than greenlets. See the Linux kernel specifications for the details of the multi-level feedback queue. 2. Construct a multi-level feedback queue within Python. This is extraordinarily complicated and complex to implement. Why duplicate what already exists? 3. Do we need to maintain compatibility with being able to call out to C functions? The primary complaint about the GIL is that it does not efficiently handle CPU bound processes and multi-cores. Running sequential processes in threads on multi-cores can actually slow down the processes. 4. Who has already solved this problem as part of the language? - Erlang (No one knew the nitty gritty details.) - Go - based on Tony Hoare's CSP and the work done on Plan 9 at Bell Labs. Uses the system scheduler and creates its own mini-threads (4k). Need to investiagate the source code on line. Goroutines do not have OS thread affinity; they can multiplex over multiple threads. - Java - Early on Java used several versions of Greenlets, but now uses system threads. The JVM punts to the OS. Conclusions -------------- 1. Do not reinvent the wheel! Many people have worked decades on this problem. Leverage thier expertise. 2. Coroutines are frequently better than threads, but do scale and each coroutine must me restarted in the thread where it was spawned. See greenlet.c. Greenlets are also chained and have mutual dependencies. The order of execution is arbitrary with not method for priorities. 3. Investigate if there is an alternative to the current method of calling external C objects. 4. Dave did a POC on priorities: http://dabeaz.blogspot.com/2010/02/revisiting-thread-priorities-and-new.html 5. Everyone agreed that some type of priority mechanism is a good idea, but wanted to see what Unladen Swallow does. (As of March 2011 Google is no longer actively developing this project.) References - ----------- Dave Beazley - GIL Wars Dave Beazley - Yieldable Threads http://www.dabeaz.com/blog.html Linux Kernel http://goo.gl/RkxVs Erlang Go - golang.org CSP - Tony Hoare http://www.usingcsp.com/cspbook.pdf I spoke with Peter Portante yesterday, and he would be very interested in participating even though he has very little free time. Peter works at HP and worked on their OS threading model. Also, see his Pycon 2010 talk on non-blocking IO and the 2011 talk on co-routines. Let me know if you have any questions. Bob Hancock Blog - www.bobhancock.org Twitter - bob_hancock and nycgtug ------- End of Forwarded Message And, indeed, Peter Portante is very interested in thinking about doing without the GIL. He's already sent me this: Date: Sun, 13 Mar 2011 16:42:00 -0400 To: Laura Creighton From: Peter Portante Subject: Re: [pypy-dev] possibly of use for our documentation Return-Path: peter.a.portante at gmail.com Delivery-Date: Sun Mar 13 21:42:21 2011 Return-Path: Hi Laura, Just left pycon and heard about talks of pypy removing the gil. I work on tru64 unix's thread library for 8 years. If there is any thing I can do to help with this effort, please let me know. Thanks, -peter ------------------------- Note: I have never promised anybody anything. This was a 'please educate me appeal'. But Bob Hancock is coming back this afternoon to talk with us. Anybody got any questions they want to make sure I ask him? Laura From tbaldridge at gmail.com Mon Mar 14 20:08:20 2011 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Mon, 14 Mar 2011 14:08:20 -0500 Subject: [pypy-dev] Thinking about the GIL In-Reply-To: <201103141813.p2EIDY9C027808@theraft.openend.se> References: <201103141813.p2EIDY9C027808@theraft.openend.se> Message-ID: I guess I'm missing something, but what's wrong with simply ripping out the GIL? In C# we have threads, C FFI (via PInvoke), and never have any major issues with threading. It seems to me that the GIL is only needed because assumptions were made when writing the interpreter. I don't know if it's full of global variables or something, but can anyone explain why a GIL is needed at all? I've done quite a bit of multi-threading programming, and I fail to see the need for a GIL. >Why duplicate what > already exists? Timothy From benjamin at python.org Mon Mar 14 20:12:35 2011 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 14 Mar 2011 14:12:35 -0500 Subject: [pypy-dev] Thinking about the GIL In-Reply-To: References: <201103141813.p2EIDY9C027808@theraft.openend.se> Message-ID: 2011/3/14 Timothy Baldridge : > I guess I'm missing something, but what's wrong with simply ripping > out the GIL? In C# we have threads, C FFI (via PInvoke), and never > have any major issues with threading. It seems to me that the GIL is > only needed because assumptions were made when writing the > interpreter. I don't know if it's full of global variables or > something, but can anyone explain why a GIL is needed at all? I've > done quite a bit of multi-threading programming, and I fail to see the > need for a GIL. Many reasons. The biggest being python data structures aren't thread-safe. -- Regards, Benjamin From amauryfa at gmail.com Mon Mar 14 20:19:53 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Mon, 14 Mar 2011 20:19:53 +0100 Subject: [pypy-dev] Thinking about the GIL In-Reply-To: References: <201103141813.p2EIDY9C027808@theraft.openend.se> Message-ID: Hi, 2011/3/14 Timothy Baldridge : > I guess I'm missing something, but what's wrong with simply ripping > out the GIL? In C# we have threads, C FFI (via PInvoke), and never > have any major issues with threading. It seems to me that the GIL is > only needed because assumptions were made when writing the > interpreter. I don't know if it's full of global variables or > something, but can anyone explain why a GIL is needed at all? I've > done quite a bit of multi-threading programming, and I fail to see the > need for a GIL. The GIL greatly simplifies multi-threaded programming in Python. For example, you are guaranteed that someList.append(x) won't run into some race condition and crash the program. In CPython, simply incrementing the reference count of an object needs some synchronization between threads. And in PyPy, garbage collectors are not (yet) thread-safe. See better explanations about the GIL in CPython: http://docs.python.org/c-api/init.html#thread-state-and-the-global-interpreter-lock http://effbot.org/pyfaq/can-t-we-get-rid-of-the-global-interpreter-lock.htm -- Amaury Forgeot d'Arc From benjamin at python.org Mon Mar 14 20:28:39 2011 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 14 Mar 2011 14:28:39 -0500 Subject: [pypy-dev] Thinking about the GIL In-Reply-To: References: <201103141813.p2EIDY9C027808@theraft.openend.se> Message-ID: 2011/3/14 Timothy Baldridge : > They may not be thread-safe, but as far as a program goes, do we > really care? If I have two threads adding items to the same list, > should I really be expecting the interpreter to keep things straight? > What's wrong with forcing the user to lock structures before editing > them? This is something that Java, and C#, and C++ all require. Yes, the Python interpreter should never crash because of user mistakes. -- Regards, Benjamin From Ben.Young at sungard.com Tue Mar 15 09:20:11 2011 From: Ben.Young at sungard.com (Ben.Young at sungard.com) Date: Tue, 15 Mar 2011 08:20:11 +0000 Subject: [pypy-dev] Thinking about the GIL In-Reply-To: References: <201103141813.p2EIDY9C027808@theraft.openend.se> Message-ID: <01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp> > -----Original Message----- > From: pypy-dev-bounces at codespeak.net [mailto:pypy-dev- > bounces at codespeak.net] On Behalf Of Benjamin Peterson > Sent: 14 March 2011 19:29 > To: Timothy Baldridge > Cc: PyPy Dev > Subject: Re: [pypy-dev] Thinking about the GIL > > 2011/3/14 Timothy Baldridge : > > They may not be thread-safe, but as far as a program goes, do we > > really care? If I have two threads adding items to the same list, > > should I really be expecting the interpreter to keep things straight? > > What's wrong with forcing the user to lock structures before editing > > them? This is something that Java, and C#, and C++ all require. > > Yes, the Python interpreter should never crash because of user > mistakes. > C# structures aren't thread-safe, but you can't crash the CLR by using them in a multithreaded manner Ben > > > -- > Regards, > Benjamin > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev From benjamin at python.org Tue Mar 15 15:17:58 2011 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 15 Mar 2011 09:17:58 -0500 Subject: [pypy-dev] Thinking about the GIL In-Reply-To: <01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp> References: <201103141813.p2EIDY9C027808@theraft.openend.se> <01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp> Message-ID: 2011/3/15 : > > >> -----Original Message----- >> From: pypy-dev-bounces at codespeak.net [mailto:pypy-dev- >> bounces at codespeak.net] On Behalf Of Benjamin Peterson >> Sent: 14 March 2011 19:29 >> To: Timothy Baldridge >> Cc: PyPy Dev >> Subject: Re: [pypy-dev] Thinking about the GIL >> >> 2011/3/14 Timothy Baldridge : >> > They may not be thread-safe, but as far as a program goes, do we >> > really care? If I have two threads adding items to the same list, >> > should I really be expecting the interpreter to keep things > straight? >> > What's wrong with forcing the user to lock structures before editing >> > them? This is something that Java, and C#, and C++ all require. >> >> Yes, the Python interpreter should never crash because of user >> mistakes. >> > > C# structures aren't thread-safe, but you can't crash the CLR by using > them in a multithreaded manner The primitive data structures must be thread-safe, though. -- Regards, Benjamin From Ben.Young at sungard.com Tue Mar 15 15:36:50 2011 From: Ben.Young at sungard.com (Ben.Young at sungard.com) Date: Tue, 15 Mar 2011 14:36:50 +0000 Subject: [pypy-dev] Thinking about the GIL In-Reply-To: References: <201103141813.p2EIDY9C027808@theraft.openend.se><01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp> Message-ID: <01781CA2CC22B145B230504679ECF48C030D2E28@EMEA-EXCHANGE03.internal.sungard.corp> > -----Original Message----- > From: pypy-dev-bounces at codespeak.net [mailto:pypy-dev- > bounces at codespeak.net] On Behalf Of Benjamin Peterson > Sent: 15 March 2011 14:18 > To: Young, Ben > Cc: pypy-dev at codespeak.net > Subject: Re: [pypy-dev] Thinking about the GIL > > 2011/3/15 : > > > > > >> -----Original Message----- > >> From: pypy-dev-bounces at codespeak.net [mailto:pypy-dev- > >> bounces at codespeak.net] On Behalf Of Benjamin Peterson > >> Sent: 14 March 2011 19:29 > >> To: Timothy Baldridge > >> Cc: PyPy Dev > >> Subject: Re: [pypy-dev] Thinking about the GIL > >> > >> 2011/3/14 Timothy Baldridge : > >> > They may not be thread-safe, but as far as a program goes, do we > >> > really care? If I have two threads adding items to the same list, > >> > should I really be expecting the interpreter to keep things > > straight? > >> > What's wrong with forcing the user to lock structures before > editing > >> > them? This is something that Java, and C#, and C++ all require. > >> > >> Yes, the Python interpreter should never crash because of user > >> mistakes. > >> > > > > C# structures aren't thread-safe, but you can't crash the CLR by > using > > them in a multithreaded manner > > The primitive data structures must be thread-safe, though. > No, just cleverly designed with the memory model in mind I believe (at least there's no locking or even atomic synchronisation), and you can get errors, it just wont bring down the VM Cheers, Ben > > > -- > Regards, > Benjamin > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev From tbaldridge at gmail.com Tue Mar 15 16:04:04 2011 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Tue, 15 Mar 2011 10:04:04 -0500 Subject: [pypy-dev] Any Multithreading Primitives? Message-ID: Does PyPy have any sort of multithreading primitives that are available in rpython programs? I'm specifically looking for CAS. I can probably simulate a mutex with CAS. Or do I have to drop down to some sort of assembly generation to atomically update a object reference? Thanks, Timothy -- ?One of the main causes of the fall of the Roman Empire was that?lacking zero?they had no way to indicate successful termination of their C programs.? (Robert Firth) From lac at openend.se Tue Mar 15 16:56:23 2011 From: lac at openend.se (Laura Creighton) Date: Tue, 15 Mar 2011 16:56:23 +0100 Subject: [pypy-dev] Any Multithreading Primitives? In-Reply-To: Message from Timothy Baldridge of "Tue, 15 Mar 2011 10:04:04 EST." References: Message-ID: <201103151556.p2FFuNqE017090@theraft.openend.se> In a message of Tue, 15 Mar 2011 10:04:04 EST, Timothy Baldridge writes: >Does PyPy have any sort of multithreading primitives that are >available in rpython programs? I'm specifically looking for CAS. I can >probably simulate a mutex with CAS. Or do I have to drop down to some >sort of assembly generation to atomically update a object reference? > >Thanks, > >Timothy Will anything written here: http://codespeak.net/pypy/dist/pypy/doc/stackless.html be of any use to you? Note, this documentation isquite old and needs attention, and right now the stackless transform does not work with the JIT. changing things so that it would work doesn't seem to be that much work, though, we just haven't gotten around to it. Laura From tbaldridge at gmail.com Tue Mar 15 17:04:30 2011 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Tue, 15 Mar 2011 11:04:30 -0500 Subject: [pypy-dev] Any Multithreading Primitives? In-Reply-To: <201103151556.p2FFuNqE017090@theraft.openend.se> References: <201103151556.p2FFuNqE017090@theraft.openend.se> Message-ID: > Will anything written here: http://codespeak.net/pypy/dist/pypy/doc/stackless.html > be of any use to you? Note, this documentation isquite old and needs > attention, and right now the stackless transform does not work with the JIT. > changing things so that it would work doesn't seem to be that much work, > though, we just haven't gotten around to it. Not really. What I need are CAS operations so I can do the following: while(True): oldval = var newval = oldval + 1 if cas(oldval, newval, var): break Basically cas takes the value you think var currently has, the value you want var to be, and the var. Cas then updates var if and only if var's current value == oldvar. But cas is atomic. So basically it returns false if the cas failed, and true if it worked. Using cas you can build mutexes, shared objects, etc. http://en.wikipedia.org/wiki/Compare-and-swap Timothy From amauryfa at gmail.com Tue Mar 15 17:15:41 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 15 Mar 2011 17:15:41 +0100 Subject: [pypy-dev] Any Multithreading Primitives? In-Reply-To: References: <201103151556.p2FFuNqE017090@theraft.openend.se> Message-ID: Hi, 2011/3/15 Timothy Baldridge : > Basically cas takes the value you think var currently has, the value > you want var to be, and the var. Cas then updates var if and only if > var's current value == oldvar. But cas is atomic. So basically it > returns false if the cas failed, and true if it worked. Using cas you > can build mutexes, shared objects, etc. In pypy/module/thread/ll_thread.py, allocate_lock() creates a Lock object, similar to the Python's thread.Lock. Careful though with threads: most of PyPy's garbage collectors are not thread-safe. -- Amaury Forgeot d'Arc From santagada at gmail.com Tue Mar 15 18:25:23 2011 From: santagada at gmail.com (Leonardo Santagada) Date: Tue, 15 Mar 2011 14:25:23 -0300 Subject: [pypy-dev] Thinking about the GIL In-Reply-To: <01781CA2CC22B145B230504679ECF48C030D2E28@EMEA-EXCHANGE03.internal.sungard.corp> References: <201103141813.p2EIDY9C027808@theraft.openend.se> <01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp> <01781CA2CC22B145B230504679ECF48C030D2E28@EMEA-EXCHANGE03.internal.sungard.corp> Message-ID: On Tue, Mar 15, 2011 at 11:36 AM, wrote: > > >> -----Original Message----- >> From: pypy-dev-bounces at codespeak.net [mailto:pypy-dev- >> bounces at codespeak.net] On Behalf Of Benjamin Peterson >> Sent: 15 March 2011 14:18 >> To: Young, Ben >> Cc: pypy-dev at codespeak.net >> Subject: Re: [pypy-dev] Thinking about the GIL >> >> 2011/3/15 ?: >> > >> > >> >> -----Original Message----- >> >> From: pypy-dev-bounces at codespeak.net [mailto:pypy-dev- >> >> bounces at codespeak.net] On Behalf Of Benjamin Peterson >> >> Sent: 14 March 2011 19:29 >> >> To: Timothy Baldridge >> >> Cc: PyPy Dev >> >> Subject: Re: [pypy-dev] Thinking about the GIL >> >> >> >> 2011/3/14 Timothy Baldridge : >> >> > They may not be thread-safe, but as far as a program goes, do we >> >> > really care? If I have two threads adding items to the same list, >> >> > should I really be expecting the interpreter to keep things >> > straight? >> >> > What's wrong with forcing the user to lock structures before >> editing >> >> > them? This is something that Java, and C#, and C++ all require. >> >> >> >> Yes, the Python interpreter should never crash because of user >> >> mistakes. >> >> >> > >> > C# structures aren't thread-safe, but you can't crash the CLR by >> using >> > them in a multithreaded manner >> >> The primitive data structures must be thread-safe, though. >> > > No, just cleverly designed with the memory model in mind I believe (at > least there's no locking or even atomic synchronisation), and you can > get errors, it just wont bring down the VM That is what I think he meant. The only way that I know for a vm to be thread-safe is its internal structures to be thread-safe. The exposed C# structures might not be, but the VM ones have to. -- Leonardo Santagada From Ben.Young at sungard.com Wed Mar 16 09:24:17 2011 From: Ben.Young at sungard.com (Ben.Young at sungard.com) Date: Wed, 16 Mar 2011 08:24:17 +0000 Subject: [pypy-dev] Thinking about the GIL In-Reply-To: References: <201103141813.p2EIDY9C027808@theraft.openend.se> <01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp> <01781CA2CC22B145B230504679ECF48C030D2E28@EMEA-EXCHANGE03.internal.sungard.corp> Message-ID: <01781CA2CC22B145B230504679ECF48C030D2FDC@EMEA-EXCHANGE03.internal.sungard.corp> > -----Original Message----- > From: Leonardo Santagada [mailto:santagada at gmail.com] > Sent: 15 March 2011 17:25 > To: Young, Ben > Cc: benjamin at python.org; pypy-dev at codespeak.net > Subject: Re: [pypy-dev] Thinking about the GIL > > On Tue, Mar 15, 2011 at 11:36 AM, wrote: > > > > > >> -----Original Message----- > >> From: pypy-dev-bounces at codespeak.net [mailto:pypy-dev- > >> bounces at codespeak.net] On Behalf Of Benjamin Peterson > >> Sent: 15 March 2011 14:18 > >> To: Young, Ben > >> Cc: pypy-dev at codespeak.net > >> Subject: Re: [pypy-dev] Thinking about the GIL > >> > >> 2011/3/15 ?: > >> > > >> > > >> >> -----Original Message----- > >> >> From: pypy-dev-bounces at codespeak.net [mailto:pypy-dev- > >> >> bounces at codespeak.net] On Behalf Of Benjamin Peterson > >> >> Sent: 14 March 2011 19:29 > >> >> To: Timothy Baldridge > >> >> Cc: PyPy Dev > >> >> Subject: Re: [pypy-dev] Thinking about the GIL > >> >> > >> >> 2011/3/14 Timothy Baldridge : > >> >> > They may not be thread-safe, but as far as a program goes, do > we > >> >> > really care? If I have two threads adding items to the same > list, > >> >> > should I really be expecting the interpreter to keep things > >> > straight? > >> >> > What's wrong with forcing the user to lock structures before > >> editing > >> >> > them? This is something that Java, and C#, and C++ all require. > >> >> > >> >> Yes, the Python interpreter should never crash because of user > >> >> mistakes. > >> >> > >> > > >> > C# structures aren't thread-safe, but you can't crash the CLR by > >> using > >> > them in a multithreaded manner > >> > >> The primitive data structures must be thread-safe, though. > >> > > > > No, just cleverly designed with the memory model in mind I believe > (at > > least there's no locking or even atomic synchronisation), and you can > > get errors, it just wont bring down the VM > > That is what I think he meant. The only way that I know for a vm to be > thread-safe is its internal structures to be thread-safe. The exposed > C# structures might not be, but the VM ones have to. > Yes, I guess so. I just meant that you won't run into scaling issues in C# because the core VM is locking all over the place or anything like that. In Python (and PyPy) the problem is worse because the VM is implemented using many of the same structures as are exposed at app level, whereas the CLR is much lower level and so is probably easier to write in a thread safe manner. Having said that, if the PyPy GCs were thread-safe, how many other structures would need to be changed to allow a free threaded python that's "use at own risk" e.g. one that's allowed to crash if you do silly things as I guess most people would be happy with that. The import machinery would need to be thread safe but how much more? Cheers, Ben > -- > Leonardo Santagada From amauryfa at gmail.com Wed Mar 16 09:59:35 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 16 Mar 2011 09:59:35 +0100 Subject: [pypy-dev] Thinking about the GIL In-Reply-To: <01781CA2CC22B145B230504679ECF48C030D2FDC@EMEA-EXCHANGE03.internal.sungard.corp> References: <201103141813.p2EIDY9C027808@theraft.openend.se> <01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp> <01781CA2CC22B145B230504679ECF48C030D2E28@EMEA-EXCHANGE03.internal.sungard.corp> <01781CA2CC22B145B230504679ECF48C030D2FDC@EMEA-EXCHANGE03.internal.sungard.corp> Message-ID: Hi 2011/3/16 : > Having said that, if the PyPy GCs were thread-safe, how many other > structures would need to be changed to allow a free threaded python that's > "use at own risk" e.g. one that's allowed to crash if you do silly things as > I guess most people would be happy with that. The import machinery would > need to be thread safe but how much more? All low-level structures: rlist, rdict &co. For example these functions would need some kind of synchronization: https://bitbucket.org/pypy/pypy/src/6dabfe362323/pypy/rpython/lltypesystem/rdict.py#cl-712 And probably many more... -- Amaury Forgeot d'Arc From renesd at gmail.com Wed Mar 16 10:52:26 2011 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Wed, 16 Mar 2011 09:52:26 +0000 Subject: [pypy-dev] Thinking about the GIL In-Reply-To: References: <201103141813.p2EIDY9C027808@theraft.openend.se> <01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp> Message-ID: Hi, one alternative approach is to have a separate VM in each thread. Then pass messages between them. Works well, and no GIL in each VM. You have to have clean code that allows you to have a separate VMs in a process. However, it's easier to make your code be able to run in separate VMs, than to recode it to allow concurrent thread access to all data structures. This way is easier to implement than removing a GIL. eg, >>> vms = [VM() for i in range(8)] >>> vms[0].send(["going in"]) >>> vms[0].get() ["coming out!"] I did this with tinypy vms in a cpython host, and it worked kind of ok. It still used the GIL when I wanted to communicate with the VMs. Only one vm could be talked to at a time. I used it with the SDL "fastevent" event queue, so each vm posted into the queue could be read from cpython. The other communication that worked well was using mmap'd data structures. This means you do not need to serialise the data when sharing messages between threads. (As a side note... It turned out to be less code just using CPython separate processes communicating via shared mmap buffers for this particular task. Since the data was all numpy/pygame.Surface they could be shared easily. If it was pure python code, it probably would have been a different matter.) Apart from easier implementation on the VM level, this approach also has other advantages. One is that it makes message sharing more explicit between threads.The other is that GC pressure is made smaller. Each VM has its own GC heap, rather than all of the objects in one big heap shared by all the threads. (another aside, CPython can also use multiple vms in one process, and that's how some webservers have embedded python... one python vm per thread). cya. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110316/671ff567/attachment.htm From Ben.Young at sungard.com Wed Mar 16 11:06:36 2011 From: Ben.Young at sungard.com (Ben.Young at sungard.com) Date: Wed, 16 Mar 2011 10:06:36 +0000 Subject: [pypy-dev] Thinking about the GIL In-Reply-To: References: <201103141813.p2EIDY9C027808@theraft.openend.se><01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp> Message-ID: <01781CA2CC22B145B230504679ECF48C030D3081@EMEA-EXCHANGE03.internal.sungard.corp> It's a nice solution for lots of problems, but unfortunately not the areas I'm interested in, where there's a shared constant memory data set of 20-100GB and 32-64 threads all performing calculations on it. As soon as you do any data copying you blow the memory requirements and kill the performance. Of course, I can't say I'm working in a "standard" environment, but for some things nothing but true threading will do Note this is in C# which does scale quite well to 64 threads, as long as you don't do any allocation! Cheers, Ben From: Ren? Dudfield [mailto:renesd at gmail.com] Sent: 16 March 2011 09:52 To: Benjamin Peterson Cc: Young, Ben; pypy-dev at codespeak.net Subject: Re: [pypy-dev] Thinking about the GIL Hi, one alternative approach is to have a separate VM in each thread. Then pass messages between them. Works well, and no GIL in each VM. You have to have clean code that allows you to have a separate VMs in a process. However, it's easier to make your code be able to run in separate VMs, than to recode it to allow concurrent thread access to all data structures. This way is easier to implement than removing a GIL. eg, >>> vms = [VM() for i in range(8)] >>> vms[0].send(["going in"]) >>> vms[0].get() ["coming out!"] I did this with tinypy vms in a cpython host, and it worked kind of ok. It still used the GIL when I wanted to communicate with the VMs. Only one vm could be talked to at a time. I used it with the SDL "fastevent" event queue, so each vm posted into the queue could be read from cpython. The other communication that worked well was using mmap'd data structures. This means you do not need to serialise the data when sharing messages between threads. (As a side note... It turned out to be less code just using CPython separate processes communicating via shared mmap buffers for this particular task. Since the data was all numpy/pygame.Surface they could be shared easily. If it was pure python code, it probably would have been a different matter.) Apart from easier implementation on the VM level, this approach also has other advantages. One is that it makes message sharing more explicit between threads.The other is that GC pressure is made smaller. Each VM has its own GC heap, rather than all of the objects in one big heap shared by all the threads. (another aside, CPython can also use multiple vms in one process, and that's how some webservers have embedded python... one python vm per thread). cya. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110316/70ce1c57/attachment-0001.htm From tbaldridge at gmail.com Wed Mar 16 14:13:06 2011 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Wed, 16 Mar 2011 08:13:06 -0500 Subject: [pypy-dev] Thinking about the GIL In-Reply-To: References: <201103141813.p2EIDY9C027808@theraft.openend.se> <01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp> Message-ID: > one alternative approach is to have a separate VM in each thread.? Then pass > messages between them.? Works well, and no GIL in each VM.? You have to have > clean code that allows you to have a separate VMs in a process.? However, > it's easier to make your code be able to run in separate VMs, than to recode > it to allow concurrent thread access to all data structures.? This way is > easier to implement than removing a GIL. > This is what Erlang does, but unfortunately it scales very poorly when you have large datasets (as mentioned above). If you want 4 threads pounding on 1GB of data, either they need to ask a 5th thread for each data item, or each thread needs its own copy of that 1GB dataset. Even embarrassingly parallel things such as raytracing are very hard in this model. Timothy From bob at redivi.com Wed Mar 16 18:30:29 2011 From: bob at redivi.com (Bob Ippolito) Date: Wed, 16 Mar 2011 13:30:29 -0400 Subject: [pypy-dev] Thinking about the GIL In-Reply-To: References: <201103141813.p2EIDY9C027808@theraft.openend.se> <01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp> Message-ID: On Wed, Mar 16, 2011 at 9:13 AM, Timothy Baldridge wrote: >> one alternative approach is to have a separate VM in each thread.? Then pass >> messages between them.? Works well, and no GIL in each VM.? You have to have >> clean code that allows you to have a separate VMs in a process.? However, >> it's easier to make your code be able to run in separate VMs, than to recode >> it to allow concurrent thread access to all data structures.? This way is >> easier to implement than removing a GIL. >> > > This is what Erlang does, but unfortunately it scales very poorly when > ?you have large datasets (as mentioned above). If you want 4 threads > pounding on 1GB of data, either they need to ask a 5th thread for each > data item, or each thread needs its own copy of that 1GB dataset. Even > embarrassingly parallel things such as raytracing are very hard in > this model. If the data is constant you can share it in Erlang by generating a module for it and putting it in the code server (a hack, but it works) or simply using a binary. These are zero-copy in Erlang. -bob From tbaldridge at gmail.com Wed Mar 16 18:51:37 2011 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Wed, 16 Mar 2011 12:51:37 -0500 Subject: [pypy-dev] Thinking about the GIL In-Reply-To: References: <201103141813.p2EIDY9C027808@theraft.openend.se> <01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp> Message-ID: > If the data is constant you can share it in Erlang by generating a > module for it and putting it in the code server (a hack, but it works) > or simply using a binary. These are zero-copy in Erlang. > > -bob True, but that doesn't really solve the problem of writes; you can't have 5 threads hammering the same data structure. In essence this model doesn't really solve any more of the same basic issues that multiprocessing does. In the erlang model you can have thousands of "threads", and currently with multiprocessing you're limited to < 100-ish. But they still both require you to pass messages. We should probably mention a somewhat new method of co-currency. This is my favorite, and is used by Clojure. First of all, I must preface this with the statement that almost all structures in Clojure are immutable: d = {} d = d.assoc(key, value) l = [] l = l.append("foo") So basically every time you add or remove something from a collection it returns a new collection. Behind the scenes the collections carefully manipulate the structures as to a) not modify the old structure and b) maintain reasonable performance (adding a item to a hashset does not copy the entire hashset). That alone gets rid of 90% of your co-currency issues. From there you have refs for Software Transactional Memory. Here's an example of how it would be used in a Python-esque syntax: val = 0 account1 = Ref(100) account2 = Ref(0) transaction: account1.value = account1.value - 50 account2.value = account2.value + 50 The transaction block insures that either the entire block succeeds, or fails as an atomic unit. Now the nice thing is, most of this can be done without any special syntax. It's possible to use the Clojure-CLR STM libs within C#, and I have, and it works like a dream. It's a tad ugly, but it works. Adding a bit of syntax sugar to python (like the transaction block) should make this quite elegant. At any rate, here's a video from Rich Hickey the creator of Clojure on how modern lock based systems and even perhaps the Erlang model is quite flawed: http://www.infoq.com/presentations/Are-We-There-Yet-Rich-Hickey Just some more points to think over, Timothy From tbaldridge at gmail.com Wed Mar 16 18:53:44 2011 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Wed, 16 Mar 2011 12:53:44 -0500 Subject: [pypy-dev] *Rant* No Reply-To? Message-ID: Why is there no "Reply-To" in the headers on this list? Whenever, in gmail, I hit "reply" it automatically sets the to address to the "from" field in the mailing list e-mail. So I end up e-mailing the person directly instead of the list. This is highly frustrating. Timothy -- ?One of the main causes of the fall of the Roman Empire was that?lacking zero?they had no way to indicate successful termination of their C programs.? (Robert Firth) From lac at openend.se Wed Mar 16 19:34:18 2011 From: lac at openend.se (Laura Creighton) Date: Wed, 16 Mar 2011 19:34:18 +0100 Subject: [pypy-dev] *Rant* No Reply-To? In-Reply-To: Message from Timothy Baldridge of "Wed, 16 Mar 2011 12:53:44 EST." References: Message-ID: <201103161834.p2GIYI52006709@theraft.openend.se> In a message of Wed, 16 Mar 2011 12:53:44 EST, Timothy Baldridge writes: >Why is there no "Reply-To" in the headers on this list? Whenever, in >gmail, I hit "reply" it automatically sets the to address to the >"from" field in the mailing list e-mail. So I end up e-mailing the >person directly instead of the list. This is highly frustrating. > >Timothy > http://woozle.org/~neale/papers/reply-to-still-harmful.html Laura From victorgarcianet at gmail.com Wed Mar 16 19:33:11 2011 From: victorgarcianet at gmail.com (Victor Garcia) Date: Wed, 16 Mar 2011 15:33:11 -0300 Subject: [pypy-dev] *Rant* No Reply-To? In-Reply-To: References: Message-ID: Timothy Baldridge wrote: > Why is there no "Reply-To" in the headers on this list? Whenever, in > gmail, I hit "reply" it automatically sets the to address to the > "from" field in the mailing list e-mail. So I end up e-mailing the > person directly instead of the list. This is highly frustrating. A quick workaround is to enable "Default 'Reply to all'" in GMail's labs. Victor From gorla.patricia at gmail.com Wed Mar 16 21:02:31 2011 From: gorla.patricia at gmail.com (gorla.patricia at gmail.com) Date: Wed, 16 Mar 2011 20:02:31 +0000 Subject: [pypy-dev] *Rant* No Reply-To? Message-ID: <834631399-1300305654-cardhu_decombobulator_blackberry.rim.net-1634046335-@bda216.bisx.prod.on.blackberry> I'd imagine a default 'reply-to-all' would be much more harmful, unless you can apply these settings based on a filter. New weekend project? ------Original Message------ From: Victor Garcia Sender: pypy-dev-bounces at codespeak.net To: Timothy Baldridge Cc: pypy-dev at codespeak.net Subject: Re: [pypy-dev] *Rant* No Reply-To? Sent: Mar 16, 2011 14:33 Timothy Baldridge wrote: > Why is there no "Reply-To" in the headers on this list? Whenever, in > gmail, I hit "reply" it automatically sets the to address to the > "from" field in the mailing list e-mail. So I end up e-mailing the > person directly instead of the list. This is highly frustrating. A quick workaround is to enable "Default 'Reply to all'" in GMail's labs. Victor _______________________________________________ pypy-dev at codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev Sent on the Sprint? Now Network from my BlackBerry? From adam.jtm30 at gmail.com Wed Mar 16 21:06:00 2011 From: adam.jtm30 at gmail.com (Adam Bark) Date: Wed, 16 Mar 2011 20:06:00 +0000 Subject: [pypy-dev] *Rant* No Reply-To? In-Reply-To: <834631399-1300305654-cardhu_decombobulator_blackberry.rim.net-1634046335-@bda216.bisx.prod.on.blackberry> References: <834631399-1300305654-cardhu_decombobulator_blackberry.rim.net-1634046335-@bda216.bisx.prod.on.blackberry> Message-ID: <4D811828.90109@gmail.com> I believe you can just set it for certain email addresses, ie. mailing lists. On 16/03/11 20:02, gorla.patricia at gmail.com wrote: > I'd imagine a default 'reply-to-all' would be much more harmful, unless you can apply these settings based on a filter. > > New weekend project? > ------Original Message------ > From: Victor Garcia > Sender: pypy-dev-bounces at codespeak.net > To: Timothy Baldridge > Cc: pypy-dev at codespeak.net > Subject: Re: [pypy-dev] *Rant* No Reply-To? > Sent: Mar 16, 2011 14:33 > > Timothy Baldridge wrote: >> Why is there no "Reply-To" in the headers on this list? Whenever, in >> gmail, I hit "reply" it automatically sets the to address to the >> "from" field in the mailing list e-mail. So I end up e-mailing the >> person directly instead of the list. This is highly frustrating. > A quick workaround is to enable "Default 'Reply to all'" in GMail's labs. > > Victor > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > > > Sent on the Sprint? Now Network from my BlackBerry? > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev From william.leslie.ttg at gmail.com Thu Mar 17 03:18:07 2011 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Thu, 17 Mar 2011 13:18:07 +1100 Subject: [pypy-dev] Thinking about the GIL In-Reply-To: References: <201103141813.p2EIDY9C027808@theraft.openend.se> <01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp> Message-ID: Where did you want this discussion to go, Laura? It looks like you wanted to talk about the specific problems that need to be dealt with while removing the GIL, but it seems to have disintegrated into the same "concurrency model X is better than concurrency model Y" free for all that regularly seems to happen on this list. Regardless of the API that runtimes written with the translation toolkit may provide, getting rid of the GIL is a precursor to the implementations of most of these models. -- William Leslie From dimaqq at gmail.com Thu Mar 17 04:51:59 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Wed, 16 Mar 2011 20:51:59 -0700 Subject: [pypy-dev] Any Multithreading Primitives? In-Reply-To: References: <201103151556.p2FFuNqE017090@theraft.openend.se> Message-ID: how about dict.setdefault or dict.popitem as a quick hack? unless you are trying to get rid of gil, in which case, sorry I missed your point On 15 March 2011 09:04, Timothy Baldridge wrote: >> Will anything written here: http://codespeak.net/pypy/dist/pypy/doc/stackless.html >> be of any use to you? ?Note, this documentation isquite old and needs >> attention, and right now the stackless transform does not work with the JIT. >> changing things so that it would work doesn't seem to be that much work, >> though, we just haven't gotten around to it. > > > Not really. What I need are CAS operations so I can do the following: > > while(True): > ? oldval = var > ? newval = oldval + 1 > ? if cas(oldval, newval, var): > ? ? ? break > > Basically cas takes the value you think var currently has, the value > you want var to be, and the var. Cas then updates var if and only if > var's current value == oldvar. But cas is atomic. So basically it > returns false if the cas failed, and true if it worked. Using cas you > can build mutexes, shared objects, etc. > > http://en.wikipedia.org/wiki/Compare-and-swap > > Timothy > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From dimaqq at gmail.com Thu Mar 17 05:04:01 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Wed, 16 Mar 2011 21:04:01 -0700 Subject: [pypy-dev] *Rant* No Reply-To? In-Reply-To: <4D811828.90109@gmail.com> References: <834631399-1300305654-cardhu_decombobulator_blackberry.rim.net-1634046335-@bda216.bisx.prod.on.blackberry> <4D811828.90109@gmail.com> Message-ID: +1 for reply-to from me. the munging flame is from last century! it's a different world now. alternatively convince google to use list-post header, see when that happens On 16 March 2011 13:06, Adam Bark wrote: > I believe you can just set it for certain email addresses, ie. mailing > lists. > > On 16/03/11 20:02, gorla.patricia at gmail.com wrote: >> I'd imagine a default 'reply-to-all' would be much more harmful, unless you can apply these settings based on a filter. >> >> New weekend project? >> ------Original Message------ >> From: Victor Garcia >> Sender: pypy-dev-bounces at codespeak.net >> To: Timothy Baldridge >> Cc: pypy-dev at codespeak.net >> Subject: Re: [pypy-dev] *Rant* No Reply-To? >> Sent: Mar 16, 2011 14:33 >> >> Timothy Baldridge wrote: >>> Why is there no "Reply-To" in the headers on this list? Whenever, in >>> gmail, I hit "reply" it automatically sets the to address to the >>> "from" field in the mailing list e-mail. So I end up e-mailing the >>> person directly instead of the list. This is highly frustrating. >> A quick workaround is to enable "Default 'Reply to all'" in GMail's labs. >> >> Victor >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev >> >> >> Sent on the Sprint? Now Network from my BlackBerry? >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From lac at openend.se Thu Mar 17 11:30:26 2011 From: lac at openend.se (Laura Creighton) Date: Thu, 17 Mar 2011 11:30:26 +0100 Subject: [pypy-dev] Thinking about the GIL In-Reply-To: Message from William ML Leslie of "Thu, 17 Mar 2011 13:18:07 +1100." References: <201103141813.p2EIDY9C027808@theraft.openend.se> <01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp> Message-ID: <201103171030.p2HAUQCT028822@theraft.openend.se> In a message of Thu, 17 Mar 2011 13:18:07 +1100, William ML Leslie writes: >Where did you want this discussion to go, Laura? It looks like you >wanted to talk about the specific problems that need to be dealt with >while removing the GIL, but it seems to have disintegrated into the >same "concurrency model X is better than concurrency model Y" free for >all that regularly seems to happen on this list. Regardless of the >API that runtimes written with the translation toolkit may provide, >getting rid of the GIL is a precursor to the implementations of most >of these models. > >-- >William Leslie I'm at a Sprint at PyCON, as are many of the people I think would be best at answering these specific questions. So it is not surprising that they are not answering them now. I, myself, am personally interested in finding out how languages I have never looked at do these things, because I expect it to influence how one gets rid of the GIL. I was hoping to have an insight as to how one could avoid going the route of reimplementing fine grained locks everywhere, pervasively, all through the codebase. But all I am seeing now is more evidence that this is impossible. Laura From santagada at gmail.com Thu Mar 17 12:50:20 2011 From: santagada at gmail.com (Leonardo Santagada) Date: Thu, 17 Mar 2011 08:50:20 -0300 Subject: [pypy-dev] Thinking about the GIL In-Reply-To: <201103171030.p2HAUQCT028822@theraft.openend.se> References: <201103141813.p2EIDY9C027808@theraft.openend.se> <01781CA2CC22B145B230504679ECF48C030D2B7F@EMEA-EXCHANGE03.internal.sungard.corp> <201103171030.p2HAUQCT028822@theraft.openend.se> Message-ID: On Thu, Mar 17, 2011 at 7:30 AM, Laura Creighton wrote: > In a message of Thu, 17 Mar 2011 13:18:07 +1100, William ML Leslie writes: >>Where did you want this discussion to go, Laura? ?It looks like you >>wanted to talk about the specific problems that need to be dealt with >>while removing the GIL, but it seems to have disintegrated into the >>same "concurrency model X is better than concurrency model Y" free for >>all that regularly seems to happen on this list. ?Regardless of the >>API that runtimes written with the translation toolkit may provide, >>getting rid of the GIL is a precursor to the implementations of most >>of these models. >> >>-- >>William Leslie > > I'm at a Sprint at PyCON, as are many of the people I think would be > best at answering these specific questions. ? So it is not surprising > that they are not answering them now. ?I, myself, am personally interested > in finding out how languages I have never looked at do these things, > because I expect it to influence how one gets rid of the GIL. ?I was > hoping to have an insight as to how one could avoid going the route > of reimplementing fine grained locks everywhere, pervasively, all > through the codebase. ?But all I am seeing now is more evidence that > this is impossible. > > Laura I think it would be cool to have something to point people to in the docs, a page describing why pypy has a gil and what would it take to remove it. I would like to see a clear separation on what steps is needed to make RPython threadsafe (ie. fixing gc choosing and implementing a concurrency model) and what would it take to make the pypy-python interpreter not need a gil (choosing a maybe even different concurrency model and memory semantics, etc). -- Leonardo Santagada From anto.cuni at gmail.com Thu Mar 17 17:09:19 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu, 17 Mar 2011 17:09:19 +0100 Subject: [pypy-dev] ideas for Google Summer of code Message-ID: <4D82322F.7070700@gmail.com> Hi all, the deadlines for GSoc are approaching, and at some point we should probably make a blog post about that. But first, we need to 1) collect ideas for possible tasks and 2) find potential mentors. Two ideas that just came to my mind: - "general work" on speed.pypy.org (we need to define better what we want, of course) - improving the jitviewer, maybe integrating it with the profiler (when we'll have one :-)) - :-) ciao, Anto From lac at openend.se Thu Mar 17 17:34:20 2011 From: lac at openend.se (Laura Creighton) Date: Thu, 17 Mar 2011 17:34:20 +0100 Subject: [pypy-dev] ideas for Google Summer of code In-Reply-To: Message from Antonio Cuni of "Thu, 17 Mar 2011 17:09:19 +0100." <4D82322F.7070700@gmail.com> References: <4D82322F.7070700@gmail.com> Message-ID: <201103171634.p2HGYKVf029380@theraft.openend.se> In a message of Thu, 17 Mar 2011 17:09:19 +0100, Antonio Cuni writes: >Hi all, > >the deadlines for GSoc are approaching, and at some point we should proba >bly >make a blog post about that. > >But first, we need to 1) collect ideas for possible tasks and 2) find >potential mentors. > >Two ideas that just came to my mind: > >- "general work" on speed.pypy.org (we need to define better what we want >, of >course) > >- improving the jitviewer, maybe integrating it with the profiler (when w >e'll >have one :-)) > >- :-) > >ciao, >Anto 3.x conversions -- a) write an interpreter b) do the fiddly bits needed to integrate the new interpreter with our codebase c) get the 3.whatever tests to pass I think this is too much work for one SoC student, but maybe not if it was set up as 2 projects, one of which stared after the other did. I am not sure how SoC is being handles for people who live in the Southern hemisphere and who go to classes in June, July, etc. >_______________________________________________ >pypy-dev at codespeak.net >http://codespeak.net/mailman/listinfo/pypy-dev From chef at ghum.de Thu Mar 17 17:48:43 2011 From: chef at ghum.de (Massa, Harald Armin) Date: Thu, 17 Mar 2011 17:48:43 +0100 Subject: [pypy-dev] phb impression from speed.pypy.org Message-ID: Hello, I really love the improvements to the clearity of speed.pypy.org In the last days I have dropped some times on that page, aaaand... although I read the description and understand it, what my mind FIRST processes from the How has PyPy performance evolved over time? picture is that "performance got worse" Yes; right after that I read the small print of "smaller is better". But still... my eyes tell me "it got worse". Would it be possible to think of a presentation that allows to GROW those elements? Having a measurement comparable to transaction per second; or whatever-stones. Or just project the 1/x of every measurment ... just the subconcious thinks "something shrinking is not sexy". There is allways the experience with "wow, you have grown so much" ... nobody tells their (shrinking) elders "wow, you have shrunken so much"... Another small point: The headline is "...over time", but the x-axis shows "over versions". If I call correctly, the delta-t between "release of cpython 2.6.2" to "pypy1.3" is different form dt(1.3,1.4) is different from dt(1.4,trunk) As much as I remember, the speedup is speeding up ... Would such changes be possible? Does someone else think they may be positive? harald -- GHUM GmbH Harald Armin Massa Spielberger Stra?e 49 70435 Stuttgart 0173/9409607 Amtsgericht Stuttgart, HRB 734971 - persuadere. et programmare From bokr at oz.net Thu Mar 17 07:04:09 2011 From: bokr at oz.net (Bengt Richter) Date: Wed, 16 Mar 2011 23:04:09 -0700 Subject: [pypy-dev] *Rant* No Reply-To? In-Reply-To: References: Message-ID: <4D81A459.6050007@oz.net> On 03/16/2011 10:53 AM Timothy Baldridge wrote: > Why is there no "Reply-To" in the headers on this list? Whenever, in > gmail, I hit "reply" it automatically sets the to address to the > "from" field in the mailing list e-mail. So I end up e-mailing the > person directly instead of the list. This is highly frustrating. > > Timothy > I am using Thunderbird on Linux, and this is a reply-all to your post received as a plain email (as opposed to using the newsgroup interface of Thunderbird, which is also available, e.g. via gmane). There is also a plain reply and a reply-list option. I will save this and the result of the other options as drafts, and extract the headers, so you can see what it did. First this email: _______________________________________________________________________ From - Wed Mar 16 22:42:08 2011 X-Mozilla-Status: 0000 X-Mozilla-Status2: 00000000 X-Mozilla-Keys: FCC: mailbox://bokr at mail.oz.net/Sent X-Identity-Key: id1 Message-ID:<4D819F30.1090203 at oz.net> Date: Wed, 16 Mar 2011 22:42:08 -0700 From: Bengt Richter X-Mozilla-Draft-Info: internal/draft; vcard=0; receipt=0; DSN=0; uuencode=0 User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.11) Gecko/20101013 Thunderbird/3.1.5 MIME-Version: 1.0 To: Timothy Baldridge CC: pypy-dev at codespeak.net Subject: Re: [pypy-dev] *Rant* No Reply-To? References: In-Reply-To: Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit _____________________________________________________________________________ Next, headers for plain plain reply: From - Wed Mar 16 22:46:45 2011 X-Mozilla-Status: 0000 X-Mozilla-Status2: 00000000 X-Mozilla-Keys: FCC: mailbox://bokr at mail.oz.net/Sent X-Identity-Key: id1 Message-ID:<4D81A045.9080306 at oz.net> Date: Wed, 16 Mar 2011 22:46:45 -0700 From: Bengt Richter X-Mozilla-Draft-Info: internal/draft; vcard=0; receipt=0; DSN=0; uuencode=0 User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.11) Gecko/20101013 Thunderbird/3.1.5 MIME-Version: 1.0 To: Timothy Baldridge Subject: Re: [pypy-dev] *Rant* No Reply-To? References: In-Reply-To: Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit ___________________________________________________________________________ Next, headers resulting from reply-list: From - Wed Mar 16 22:49:10 2011 X-Mozilla-Status: 0000 X-Mozilla-Status2: 00000000 X-Mozilla-Keys: FCC: mailbox://bokr at mail.oz.net/Sent X-Identity-Key: id1 Message-ID:<4D81A0D6.7090102 at oz.net> Date: Wed, 16 Mar 2011 22:49:10 -0700 From: Bengt Richter X-Mozilla-Draft-Info: internal/draft; vcard=0; receipt=0; DSN=0; uuencode=0 User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.11) Gecko/20101013 Thunderbird/3.1.5 MIME-Version: 1.0 To: pypy-dev at codespeak.net Subject: Re: [pypy-dev] *Rant* No Reply-To? References: In-Reply-To: Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit ___________________________________________________________________________ Does gmail not have similar reply options? PS. The source of your incoming post as email, so you know what Thunderbird used to generate the reply options: ########################################################################### From - Wed Mar 16 10:58:14 2011 X-Account-Key: account2 X-UIDL: 0C777EF600002BB7 X-Mozilla-Status: 0001 X-Mozilla-Status2: 00000000 X-Mozilla-Keys: X-Eon-Dm: dm34 Return-Path: Received: from codespeak.net (88.198.193.90 [88.198.193.90]) by dm34.mta.everyone.net (EON-INBOUND) with ESMTP id dm34.4d751061.1bddd2b for; Wed, 16 Mar 2011 10:53:51 -0700 Received: from codespeak.net (localhost [127.0.0.1]) by codespeak.net (Postfix) with ESMTP id 198A7282BD6; Wed, 16 Mar 2011 18:53:48 +0100 (CET) X-Original-To: pypy-dev at codespeak.net Delivered-To: pypy-dev at codespeak.net Received: from mail-iw0-f182.google.com (mail-iw0-f182.google.com [209.85.214.182]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by codespeak.net (Postfix) with ESMTP id 81F28282B9D for; Wed, 16 Mar 2011 18:53:45 +0100 (CET) Received: by iwn33 with SMTP id 33so3148560iwn.27 for; Wed, 16 Mar 2011 10:53:44 -0700 (PDT) MIME-Version: 1.0 Received: by 10.43.51.65 with SMTP id vh1mr374043icb.435.1300298024183; Wed, 16 Mar 2011 10:53:44 -0700 (PDT) Received: by 10.42.223.9 with HTTP; Wed, 16 Mar 2011 10:53:44 -0700 (PDT) Date: Wed, 16 Mar 2011 12:53:44 -0500 Message-ID: From: Timothy Baldridge To: pypy-dev at codespeak.net Subject: [pypy-dev] *Rant* No Reply-To? X-BeenThere: pypy-dev at codespeak.net X-Mailman-Version: 2.1.9 Precedence: list List-Id: development of a Python Implementation mostly developed within Python List-Unsubscribe:, List-Archive: List-Post: List-Help: List-Subscribe:, Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: quoted-printable Sender: pypy-dev-bounces at codespeak.net Errors-To: pypy-dev-bounces at codespeak.net Why is there no "Reply-To" in the headers on this list? Whenever, in gmail, I hit "reply" it automatically sets the to address to the "from" field in the mailing list e-mail. So I end up e-mailing the person directly instead of the list. This is highly frustrating. Timothy -- = =93One of the main causes of the fall of the Roman Empire was that=96lacking zero=96they had no way to indicate successful termination of their C programs.=94 (Robert Firth) ########################################################################### Regards, Bengt Richter Munging does not seem necessary? (Thanks for the link, Laura) From santagada at gmail.com Thu Mar 17 18:11:32 2011 From: santagada at gmail.com (Leonardo Santagada) Date: Thu, 17 Mar 2011 14:11:32 -0300 Subject: [pypy-dev] *Rant* No Reply-To? In-Reply-To: <4D81A459.6050007@oz.net> References: <4D81A459.6050007@oz.net> Message-ID: On Thu, Mar 17, 2011 at 3:04 AM, Bengt Richter wrote: > Does gmail not have similar reply options? Nope, also Mail.app doesn't have reply-list also. > PS. The source of your incoming post as email, so you know what > Thunderbird used to generate the reply options: Which doesn't contain the List-to header that wooz talked about. -- Leonardo Santagada From lac at openend.se Thu Mar 17 18:22:39 2011 From: lac at openend.se (Laura Creighton) Date: Thu, 17 Mar 2011 18:22:39 +0100 Subject: [pypy-dev] phb impression from speed.pypy.org In-Reply-To: Message from "Massa, Harald Armin" of "Thu, 17 Mar 2011 17:48:43 +0100." References: Message-ID: <201103171722.p2HHMdIs001438@theraft.openend.se> Smaller is better when you are dieting. Or when you are racing. Given that there is talk that we will measure memory size as well, and turn into performance.pypy.org then I think that the 'smaller is better' idea will be well understood. As a practical matter, making 'faster' be 'bigger' doesn't make sense in terms of benchmarks, in which you want to be the first to finish. Laura From santagada at gmail.com Thu Mar 17 18:23:59 2011 From: santagada at gmail.com (Leonardo Santagada) Date: Thu, 17 Mar 2011 14:23:59 -0300 Subject: [pypy-dev] ideas for Google Summer of code In-Reply-To: <201103171634.p2HGYKVf029380@theraft.openend.se> References: <4D82322F.7070700@gmail.com> <201103171634.p2HGYKVf029380@theraft.openend.se> Message-ID: My ideas, take the ones you guys like and don't bother if some are too hard or the pypy team is not interested: 1 - Some pypy compatibility site, like the one brett made for python 3 2 - Rework pypy website to make it pretty and more organized, maybe migrate or integrate bitbucket wiki with the rest of the docs 3 - make a killer app for pypy: either make mercurial/bzr or django faster on pypy to get more users 4 - Finish faster ctypes, make a cython/swig backend 5 - stackless on pypy with jit 6 - make an embeding api (for mod_wsgi and uWSGI) 7 - make pypy on .net jit work (or on java) 8 - better mac os or windows support (or strange unix like aix and [net|open|free]bsd) 9 - Port ZODB or other less complex c extension to pypy/ctypes (pycripto) 10 - ultra fast pickle (nice for everyone specially to incentive zodb to be ported) 11 - make the 64bit jit better (IIRC it was not as great as it could be) 12 - powerpc jit 13 - better GC On Thu, Mar 17, 2011 at 1:34 PM, Laura Creighton wrote: > In a message of Thu, 17 Mar 2011 17:09:19 +0100, Antonio Cuni writes: >>Hi all, >> >>the deadlines for GSoc are approaching, and at some point we should proba >>bly >>make a blog post about that. >> >>But first, we need to 1) collect ideas for possible tasks and 2) find >>potential mentors. >> >>Two ideas that just came to my mind: >> >>- "general work" on speed.pypy.org (we need to define better what we want >>, of >>course) >> >>- improving the jitviewer, maybe integrating it with the profiler (when w >>e'll >>have one :-)) >> >>- :-) >> >>ciao, >>Anto > > 3.x conversions -- a) write an interpreter > ? ? ? ? ? ? ? ? ? b) do the fiddly bits needed to integrate the new > ? ? ? ? ? ? ? ? ? ? ?interpreter with our codebase > ? ? ? ? ? ? ? ? ? c) get the 3.whatever tests to pass > > I think this is too much work for one SoC student, but maybe not if it > was set up as 2 projects, one of which stared after the other did. ?I > am not sure how SoC is being handles for people who live in the > Southern hemisphere and who go to classes in June, July, etc. > >>_______________________________________________ >>pypy-dev at codespeak.net >>http://codespeak.net/mailman/listinfo/pypy-dev > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -- Leonardo Santagada From baiju.m.mail at gmail.com Thu Mar 17 18:37:51 2011 From: baiju.m.mail at gmail.com (Baiju M) Date: Thu, 17 Mar 2011 13:37:51 -0400 Subject: [pypy-dev] ideas for Google Summer of code In-Reply-To: References: <4D82322F.7070700@gmail.com> <201103171634.p2HGYKVf029380@theraft.openend.se> Message-ID: On Thu, Mar 17, 2011 at 1:23 PM, Leonardo Santagada wrote: > My ideas, take the ones you guys like and don't bother if some are too > hard or the pypy team is not interested: > > 1 - Some pypy compatibility site, like the one brett made for python 3 I am interested to set up a site for PyPY. I have already created a similar site for Python 3: http://getpython3.net/ If this idea sound good, you can add one DNS "A record" pointing to this IP: 184.106.69.139 A sub-domain like http://compatibility.pypy.org/ would be fine. Otherwise you can provide me a hosting place also. BTW, the code is here: https://github.com/baijum/getpython3 (Flask app) Regards, Baiju M From greg at quora.com Thu Mar 17 18:38:56 2011 From: greg at quora.com (Greg Price) Date: Thu, 17 Mar 2011 10:38:56 -0700 Subject: [pypy-dev] *Rant* No Reply-To? In-Reply-To: References: <4D81A459.6050007@oz.net> Message-ID: On Thu, Mar 17, 2011 at 10:11 AM, Leonardo Santagada wrote: > On Thu, Mar 17, 2011 at 3:04 AM, Bengt Richter wrote: >> Does gmail not have similar reply options? > > Nope, also Mail.app doesn't have reply-list also. Gmail has "Reply to all". Use that. I do every day. Last I looked over someone's shoulder using Mail.app, it did too. I'd be surprised to hear of a mail client that did not. As Bengt demonstrated by his choice to use it, reply-all is most often what you want instead of reply-list in any case. Greg From dsurviver at gmail.com Thu Mar 17 19:00:58 2011 From: dsurviver at gmail.com (Danilo Freitas) Date: Thu, 17 Mar 2011 15:00:58 -0300 Subject: [pypy-dev] ideas for Google Summer of code In-Reply-To: References: <4D82322F.7070700@gmail.com> <201103171634.p2HGYKVf029380@theraft.openend.se> Message-ID: Hi, all. I'm interested in applying for GSoC this year. I'm talking to Miquel Torres about some stuff in Codespeed, but I don't know if it could be considered as PyPy project for GSoC. We're trying to allow Codespeed branch comparing, to check if a feature branch is getting faster than trunk. So, we'll see if a feature is really evolving. This would also affect speed.pypy.org. After that, we shall work on more stuff. So, could Codespeed improvements be considered as PyPy GSoC projects? Laura, I'm from Brazil and was a GSoC student in 2009 for Cython. I had about only 1 month free of college (June~July), but I completed what I promised without problems caused by college. So, I guess people from south hemisphere can handle with it, if they dedicate themselves :) 2011/3/17 Baiju M : > On Thu, Mar 17, 2011 at 1:23 PM, Leonardo Santagada wrote: >> My ideas, take the ones you guys like and don't bother if some are too >> hard or the pypy team is not interested: >> >> 1 - Some pypy compatibility site, like the one brett made for python 3 > > I am interested to set up a site for PyPY. ?I have already created a similar > site for Python 3: ?http://getpython3.net/ > > If this idea sound good, you can add one DNS "A record" pointing to this IP: > 184.106.69.139 ?A sub-domain like http://compatibility.pypy.org/ would be fine. > Otherwise you can provide me a hosting place also. ?BTW, the code is here: > https://github.com/baijum/getpython3 ?(Flask app) > > Regards, > Baiju M > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -- - Danilo Freitas From garyrob at me.com Thu Mar 17 18:38:36 2011 From: garyrob at me.com (Gary Robinson) Date: Thu, 17 Mar 2011 13:38:36 -0400 Subject: [pypy-dev] ideas for Google Summer of code Message-ID: <6393F570-E4D1-44CE-8AC6-CA55234DF287@me.com> Work on numpy/scipy integration. That will really help pypy be more useful to people in the scientific and A.I. communities.. -- Gary Robinson CTO Emergent Discovery, LLC personal email: garyrob at me.com work email: grobinson at emergentdiscovery.com Company: http://www.emergentdiscovery.com Blog: http://www.garyrobinson.net From wlavrijsen at lbl.gov Thu Mar 17 19:33:56 2011 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Thu, 17 Mar 2011 11:33:56 -0700 (PDT) Subject: [pypy-dev] ideas for Google Summer of code In-Reply-To: <4D82322F.7070700@gmail.com> References: <4D82322F.7070700@gmail.com> Message-ID: Hi Anto, > - :-) the Reflex work has at one time before been proposed as a GSoC. Now that a prototype is in place, there are several minor tasks that can be done, which can lead to bigger/more research tasks as desired/appropriate. How does this work anyway, do you need to come with your own student? Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From dmalcolm at redhat.com Thu Mar 17 19:20:20 2011 From: dmalcolm at redhat.com (David Malcolm) Date: Thu, 17 Mar 2011 14:20:20 -0400 Subject: [pypy-dev] Documentation sprint at PyCon Message-ID: <1300386020.16889.16.camel@surprise> Laura, me, and others sprinted on documentation cleanups at PyCon, using https://bitbucket.org/dmalcolm/pypy-dmalcolm as a development branch. Laura and Armin just merged the changes from that repo Significant changes are: - the Sphinxification of the docs - the renaming of the sources from .txt to .rst, to play better with text editors with smarts for rst - organizing the documentation, to try to stress the high-important up-to-date material, whilst moving the more out-of-date materials to a clearly-marked annex (see cleanup.rst). Running: make html within pypy/docs should now generate the documentation part of the site. You can also use: make linkcheck to check links; we believe we've fixed all internal links, but some external links are still broken. There's a nasty hack for pointing at sources which you can see in: https://bitbucket.org/pypy/pypy/changeset/c6f7ecf2dc01 but fixing this properly looks like we would need to write a new sphinx plugin; as it is, it works, but is ugly at the rst level. Going forward, our idea is that http://docs.pypy.org ought to point at the sphinx-generated html, and the "development" link on pypy.org should point to docs.pypy.org (moving another thing off of codespeak). The folks at readthedocs.org have offered hosting space, and can keep http://pypy.readthedocs.org up-to-date with a nightly build of the documentation. They have some docs on setting up a CNAME for this: http://read-the-docs.readthedocs.org/en/latest/alternate_domains.html#cname-support so that docs.pypy.org can be set up to point there. Currently that site has an out-of-date build of my bitbucket branch I mentioned at the top of this mail (and there've been a lot of changes since then). Someone will need to sync up with the readthedocs folks to make their site points at the correct bitbucket repo, and to ensure that it's getting regularly rebuilt Hope this makes sense and is helpful. Dave From drsalists at gmail.com Thu Mar 17 21:25:38 2011 From: drsalists at gmail.com (Dan Stromberg) Date: Thu, 17 Mar 2011 13:25:38 -0700 Subject: [pypy-dev] ideas for Google Summer of code In-Reply-To: References: <4D82322F.7070700@gmail.com> <201103171634.p2HGYKVf029380@theraft.openend.se> Message-ID: On Thu, Mar 17, 2011 at 10:23 AM, Leonardo Santagada wrote: > 7 - make pypy on .net jit work (or on java) > This reminds me: it might be good to make the JVM PyPy be able to call native Java code - on a typical JRE, and on Android. Last I heard, on Android people were using a CPython port, which reportedly requires a stub for the various Android library calls that're written in what-is-essentially-Java. What I was told is that the Ruby port to Android gives much better API access, because they started with a Ruby that runs on a JVM. Also, on the matter of performance testing, coming up with a bunch of tests that run unmodified on a large number of Python interpreters might be included - though perhaps that goes without saying. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110317/4bf01b11/attachment.htm From santagada at gmail.com Thu Mar 17 21:32:48 2011 From: santagada at gmail.com (Leonardo Santagada) Date: Thu, 17 Mar 2011 17:32:48 -0300 Subject: [pypy-dev] ideas for Google Summer of code In-Reply-To: References: <4D82322F.7070700@gmail.com> <201103171634.p2HGYKVf029380@theraft.openend.se> Message-ID: On Thu, Mar 17, 2011 at 5:25 PM, Dan Stromberg wrote: > > On Thu, Mar 17, 2011 at 10:23 AM, Leonardo Santagada > wrote: >> >> 7 - make pypy on .net jit work (or on java) > > This reminds me: it might be good to make the JVM PyPy be able to call > native Java code - on a typical JRE, and on Android.? Last I heard, on > Android people were using a CPython port, which reportedly requires a stub > for the various Android library calls that're written in > what-is-essentially-Java.? What I was told is that the Ruby port to Android > gives much better API access, because they started with a Ruby that runs on > a JVM. I think dalvik and jme don't support compiling during runs, so no jit, then I think making jython (which also needs compilation at runtime) work on dalvik seems like a better idea. > Also, on the matter of performance testing, coming up with a bunch of tests > that run unmodified on a large number of Python interpreters might be > included - though perhaps that goes without saying. This is part of the gsoc to implement a speed.python.org (so it is a great idea, but it is not a pypy gsoc). -- Leonardo Santagada From chris.mulligan at gmail.com Thu Mar 17 21:34:56 2011 From: chris.mulligan at gmail.com (Chris Mulligan) Date: Thu, 17 Mar 2011 16:34:56 -0400 Subject: [pypy-dev] GC Tuning script values Message-ID: <6E22742A-65E3-42F7-BBFA-7CBFAD7B5198@gmail.com> Just installed PyPy on my Macbook Pro per Bob Ippolito's instructions. Very easy! Here's a link to my tuning values http://a.libpa.st/hMGxQ. In short with a dual cure i5 w/ 3MB L3 the following were fastest: 1 CPU: 2M 2 CPU: 768K 3 CPU: 768K 4 CPU: 512K Congrats on the hard work, everyone. I'm very excited to start testing some of our own code with pypy. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110317/980795ee/attachment.htm From anto.cuni at gmail.com Thu Mar 17 22:26:55 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu, 17 Mar 2011 22:26:55 +0100 Subject: [pypy-dev] ideas for Google Summer of code In-Reply-To: References: <4D82322F.7070700@gmail.com> <201103171634.p2HGYKVf029380@theraft.openend.se> Message-ID: <4D827C9F.9090506@gmail.com> On 17/03/11 21:25, Dan Stromberg wrote: > > On Thu, Mar 17, 2011 at 10:23 AM, Leonardo Santagada > wrote: > > 7 - make pypy on .net jit work (or on java) this is probably a too large task for GSoc. For one, before working on the JIT it is necessary to make normal translation working, because right now ootype backends are broken. IIRC, it's not too hard because it consists of porting virtualrefs to ootype, which should be simple. But it is possible that there are other issues after that, since ootype translations have not run since a long time (more than 1 year, I think). About the JIT: JIT on .NET is not going to be any fast. I did it for my thesis when the JIT was more .NET friendly (no virtualrefs) and results were interesting as a research project, but not good enough to be used in production. The JIT for pypy-jvm is an open topic: the JVM has the potential to do a much better job than the CLI, but we cannot know until we really try. Having a working pypy-jvm-jit is a lot of work, though. It consists of: 1) make pypy-jvm (without jit) working again 2) design and implement a way to use Java objects at RPython level: this is needed to write the backend 3) port the JIT to ootype again. Should not be too hard, but there are going to be issues, because the codewriter has been heavily refactored since last year, when the JIT on ootype worked well 4) write the backend All together, it's probably huge for GSoc. But e.g 1+2 could fit it; we already had a GSoc on this topic few years ago, but it didn't work well. > > This reminds me: it might be good to make the JVM PyPy be able to call native > Java code - on a typical JRE, and on Android. Last yes, that's another interesting topic. It requires both points 1 and 2 above, though. Once we have that, it should not be too hard, as it has already been implemented for .NET and could use the same techniques (and probably reuse also most of the code). ciao, Anto From anto.cuni at gmail.com Thu Mar 17 22:30:05 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu, 17 Mar 2011 22:30:05 +0100 Subject: [pypy-dev] ideas for Google Summer of code In-Reply-To: References: <4D82322F.7070700@gmail.com> Message-ID: <4D827D5D.9040703@gmail.com> Hi Wim, On 17/03/11 19:33, wlavrijsen at lbl.gov wrote: > Hi Anto, > >> - :-) > > the Reflex work has at one time before been proposed as a GSoC. Now that > a prototype is in place, there are several minor tasks that can be done, > which can lead to bigger/more research tasks as desired/appropriate. good idea! > How does this work anyway, do you need to come with your own student? not necessarily. At this stage, the goal is to collect ideas which are reasonable, then publish it and see which students are interested in them. However, nothing stops you to suggest a student of course (especially if you already know him and you are confident that can do the job well). ciao, Anto From fijall at gmail.com Thu Mar 17 22:32:56 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 17 Mar 2011 15:32:56 -0600 Subject: [pypy-dev] ideas for Google Summer of code In-Reply-To: References: <4D82322F.7070700@gmail.com> <201103171634.p2HGYKVf029380@theraft.openend.se> Message-ID: On Thu, Mar 17, 2011 at 12:00 PM, Danilo Freitas wrote: > Hi, all. > > I'm interested in applying for GSoC this year. > I'm talking to Miquel Torres about some stuff in Codespeed, but I > don't know if it could be considered as PyPy project for GSoC. > We're trying to allow Codespeed branch comparing, to check if a > feature branch is getting faster than trunk. So, we'll see if a > feature is really evolving. > This would also affect speed.pypy.org. After that, we shall work on more stuff. > So, could Codespeed improvements be considered as PyPy GSoC projects? > > Laura, I'm from Brazil and was a GSoC student in 2009 for Cython. I > had about only 1 month free of college (June~July), but I completed > what I promised without problems caused by college. So, I guess people > from south hemisphere can handle with it, if they dedicate themselves > :) Hi. That would definitely be considered PyPy project. One ideas that I have in mind is to create speed.python.org - a place where a whole lot of different implementations can be run. This requires improvements to both our benchmark infrastructure and codespeed itself. > > 2011/3/17 Baiju M : >> On Thu, Mar 17, 2011 at 1:23 PM, Leonardo Santagada wrote: >>> My ideas, take the ones you guys like and don't bother if some are too >>> hard or the pypy team is not interested: >>> >>> 1 - Some pypy compatibility site, like the one brett made for python 3 >> >> I am interested to set up a site for PyPY. ?I have already created a similar >> site for Python 3: ?http://getpython3.net/ >> >> If this idea sound good, you can add one DNS "A record" pointing to this IP: >> 184.106.69.139 ?A sub-domain like http://compatibility.pypy.org/ would be fine. >> Otherwise you can provide me a hosting place also. ?BTW, the code is here: >> https://github.com/baijum/getpython3 ?(Flask app) >> >> Regards, >> Baiju M >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev >> > > > > -- > - Danilo Freitas > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From jacob at openend.se Thu Mar 17 23:30:15 2011 From: jacob at openend.se (Jacob =?iso-8859-1?q?Hall=E9n?=) Date: Thu, 17 Mar 2011 23:30:15 +0100 Subject: [pypy-dev] ideas for Google Summer of code In-Reply-To: <4D827D5D.9040703@gmail.com> References: <4D82322F.7070700@gmail.com> <4D827D5D.9040703@gmail.com> Message-ID: <201103172330.15939.jacob@openend.se> torsdag 17 mars 2011 22.30.05 skrev Antonio Cuni: > Hi Wim, > > On 17/03/11 19:33, wlavrijsen at lbl.gov wrote: > > Hi Anto, > > > >> - :-) > > > > the Reflex work has at one time before been proposed as a GSoC. Now that > > a prototype is in place, there are several minor tasks that can be done, > > which can lead to bigger/more research tasks as desired/appropriate. > > good idea! > > > How does this work anyway, do you need to come with your own student? > > not necessarily. At this stage, the goal is to collect ideas which are > reasonable, then publish it and see which students are interested in them. > However, nothing stops you to suggest a student of course (especially if > you already know him and you are confident that can do the job well). We should also note that mentors are the really scarce resource. If yoiu are willing to mentor a student, the chances of this being a GSoC project increases dramatically. Jacob Hall?n From wlavrijsen at lbl.gov Thu Mar 17 23:28:11 2011 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Thu, 17 Mar 2011 15:28:11 -0700 (PDT) Subject: [pypy-dev] ideas for Google Summer of code In-Reply-To: <201103172330.15939.jacob@openend.se> References: <4D82322F.7070700@gmail.com> <4D827D5D.9040703@gmail.com> <201103172330.15939.jacob@openend.se> Message-ID: Hi Jacob, > If yoiu are willing to mentor a student most definitely. > the chances of this being a GSoC project increases dramatically. Great! Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From lac at openend.se Fri Mar 18 01:10:30 2011 From: lac at openend.se (Laura Creighton) Date: Fri, 18 Mar 2011 01:10:30 +0100 Subject: [pypy-dev] ideas for Google Summer of code In-Reply-To: Message from Maciej Fijalkowski of "Thu, 17 Mar 2011 15:32:56 CST." References: <4D82322F.7070700@gmail.com> <201103171634.p2HGYKVf029380@theraft.openend.se> Message-ID: <201103180010.p2I0AVLY006544@theraft.openend.se> I'd like speed.python.org to become performance.python.org so we can measure memoryt consumption too. Laura From hyarion at iinet.net.au Fri Mar 18 01:44:41 2011 From: hyarion at iinet.net.au (hyarion at iinet.net.au) Date: Fri, 18 Mar 2011 08:44:41 +0800 Subject: [pypy-dev] Thinking about the GIL Message-ID: <41662.1300409081@iinet.net.au> On Thu Mar 17 21:30 , Laura Creighton sent: >In a message of Thu, 17 Mar 2011 13:18:07 +1100, William ML Leslie writes: >>Where did you want this discussion to go, Laura? It looks like you >>wanted to talk about the specific problems that need to be dealt with >>while removing the GIL, but it seems to have disintegrated into the >>same "concurrency model X is better than concurrency model Y" free for >>all that regularly seems to happen on this list. Regardless of the >>API that runtimes written with the translation toolkit may provide, >>getting rid of the GIL is a precursor to the implementations of most >>of these models. >> >>-- >>William Leslie > >I'm at a Sprint at PyCON, as are many of the people I think would be >best at answering these specific questions. So it is not surprising >that they are not answering them now. I, myself, am personally interested >in finding out how languages I have never looked at do these things, >because I expect it to influence how one gets rid of the GIL. I was >hoping to have an insight as to how one could avoid going the route >of reimplementing fine grained locks everywhere, pervasively, all >through the codebase. But all I am seeing now is more evidence that >this is impossible. This may be a dumb question, but has anyone considered software transactional memory for this sort of thing? My experience comes from the implementation side of STM in a very different language to (R)Python, but it seems like this might be a reasonable fir for STM's strengths. The GIL gives you a serial execution order of bytecodes, with no particular guarantees about which order (because if it matters there should be application level concurrency control). Wrapping each bytecode in an STM transaction would give you an as-if-serial execution order, again with no guarantees about which order. You get transaction overheads instead of lock/unlock overheads, but some STM systems can be quite efficient for short transactions that rarely conflict. There'd be a lot of problems to solve, of course. But has anyone already considered this and figured out that it's impossible or impractical? -- Ben From william.leslie.ttg at gmail.com Fri Mar 18 02:01:00 2011 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Fri, 18 Mar 2011 12:01:00 +1100 Subject: [pypy-dev] Thinking about the GIL In-Reply-To: <41662.1300409081@iinet.net.au> References: <41662.1300409081@iinet.net.au> Message-ID: On 18 March 2011 11:44, hyarion at iinet.net.au wrote: > There'd be a lot of problems to solve, of course. But has anyone already considered > this and figured out that it's impossible or impractical? My own fork of pypy will eventually use STM as an implementation technique, more or less, so we will get to find out first hand if it is practical, at least within an environment with a fine-grain effect system. -- William Leslie From asenchi at gmail.com Fri Mar 18 04:35:25 2011 From: asenchi at gmail.com (Curt Micol) Date: Thu, 17 Mar 2011 23:35:25 -0400 Subject: [pypy-dev] gcbench run Message-ID: Hello, In his post today, Bob Ippolito said the devs were interested in results from the gcbench run. Here's a gist with all of the data. Hope this is helpful. https://gist.github.com/874910 Thanks for the great work on PyPy, -- # Curt Micol From arigo at tunes.org Fri Mar 18 04:43:34 2011 From: arigo at tunes.org (Armin Rigo) Date: Thu, 17 Mar 2011 23:43:34 -0400 Subject: [pypy-dev] GC Tuning script values In-Reply-To: <6E22742A-65E3-42F7-BBFA-7CBFAD7B5198@gmail.com> References: <6E22742A-65E3-42F7-BBFA-7CBFAD7B5198@gmail.com> Message-ID: Hi Chris, On Thu, Mar 17, 2011 at 4:34 PM, Chris Mulligan wrote: > Just installed PyPy on my Macbook Pro per Bob Ippolito's instructions. Very > easy! Thanks for the feedback. The results make it clear that we should somehow tune the number according to the load of the machine -- picking up the right number for the load can easily make a 20% speed difference (at least on Mac OS X, but I strongly suspect the same is true on other platforms). Ideally, it should dynamically adapt its nursery size in order to minimize the cache misses. If anyone has a suggestion on how to implement that, preferably in a non-OS-specific way (e.g. by reading some x86 CPU counters), I'd welcome it :-) A bient?t, Armin. From bob at redivi.com Fri Mar 18 05:52:27 2011 From: bob at redivi.com (Bob Ippolito) Date: Fri, 18 Mar 2011 00:52:27 -0400 Subject: [pypy-dev] GC Tuning script values In-Reply-To: References: <6E22742A-65E3-42F7-BBFA-7CBFAD7B5198@gmail.com> Message-ID: It also seems that about L3//4 is a pretty good number on both his machine and mine. Not optimal in the single core case but works well as load increases. Of course, single or four core machines could be wildly different. On Thursday, March 17, 2011, Armin Rigo wrote: > Hi Chris, > > On Thu, Mar 17, 2011 at 4:34 PM, Chris Mulligan > wrote: >> Just installed PyPy on my Macbook Pro per Bob Ippolito's instructions. Very >> easy! > > Thanks for the feedback. ?The results make it clear that we should > somehow tune the number according to the load of the machine -- > picking up the right number for the load can easily make a 20% speed > difference (at least on Mac OS X, but I strongly suspect the same is > true on other platforms). > > Ideally, it should dynamically adapt its nursery size in order to > minimize the cache misses. ?If anyone has a suggestion on how to > implement that, preferably in a non-OS-specific way (e.g. by reading > some x86 CPU counters), I'd welcome it :-) > > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From greg at quora.com Fri Mar 18 05:53:28 2011 From: greg at quora.com (Greg Price) Date: Thu, 17 Mar 2011 21:53:28 -0700 Subject: [pypy-dev] ideas for Google Summer of code In-Reply-To: <201103180010.p2I0AVLY006544@theraft.openend.se> References: <4D82322F.7070700@gmail.com> <201103171634.p2HGYKVf029380@theraft.openend.se> <201103180010.p2I0AVLY006544@theraft.openend.se> Message-ID: Hi Laura, On Mar 17, 2011 5:10 PM, "Laura Creighton" wrote: > > I'd like speed.python.org to become performance.python.org so we can > measure memoryt consumption too. We should definitely measure memory consumption too, but "speed.python.org" has a nice rhythm to it - few syllables, quick to say - such that I think it might make sense to keep the name even so. To me, "speed" also suggests that it *is* fast, more so than "performance" suggests that the performance actually is good, as opposed to neutrally measuring it, whatever it is. I always thought "speed.pypy.org" quite a bold name, saying "we are fast" right there in the domain so nobody could miss the point. Maybe that last is just me, though. Greg -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110317/16c2bf53/attachment.htm From tobami at googlemail.com Fri Mar 18 08:13:15 2011 From: tobami at googlemail.com (Miquel Torres) Date: Fri, 18 Mar 2011 08:13:15 +0100 Subject: [pypy-dev] phb impression from speed.pypy.org In-Reply-To: <201103171722.p2HHMdIs001438@theraft.openend.se> References: <201103171722.p2HHMdIs001438@theraft.openend.se> Message-ID: I understand what you say, and it is certainly possible to turn benchmarks into "bigger is better". As Laura wrote though, it is only natural to measure time, because that is what you actually what to do: reduce the time it takes to complete some task. Other projects do the same. See for example jQuery announcing a newer, faster release: http://blog.jquery.com/2010/10/16/jquery-143-released/ (scroll down for lots of performance plots) *However*, it seems that for the last release they have done what you propose: http://blog.jquery.com/2011/01/31/jquery-15-released/ They have changed the units from "seconds" to number of "iterations per second". In any case that is something for the PyPy developers to decide. Cheers, Miquel 2011/3/17 Laura Creighton : > > Smaller is better when you are dieting. ?Or when you are racing. > Given that there is talk that we will measure memory size as well, and > turn into performance.pypy.org then I think that the 'smaller is > better' idea will be well understood. ?As a practical matter, making > 'faster' be 'bigger' doesn't make sense in terms of benchmarks, in which > you want to be the first to finish. > > Laura > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From tobami at googlemail.com Fri Mar 18 08:23:51 2011 From: tobami at googlemail.com (Miquel Torres) Date: Fri, 18 Mar 2011 08:23:51 +0100 Subject: [pypy-dev] phb impression from speed.pypy.org In-Reply-To: References: <201103171722.p2HHMdIs001438@theraft.openend.se> Message-ID: Oh! it just downed on me that there is a very easy way to do a "bigger is better" plot without changing any of the current views of plots. The "How has PyPy performance evolved over time?" plot, with represents the geometric average of all normalized benchmarks. You may have noticed the number inside parenthesis, which it is precisely the inverse. That one plot can thus be easily inverted, to be able to show "times faster than", without changing the rest of the site. 2011/3/18 Miquel Torres : > I understand what you say, and it is certainly possible to turn > benchmarks into "bigger is better". As Laura wrote though, it is only > natural to measure time, because that is what you actually what to do: > reduce the time it takes to complete some task. > > Other projects do the same. See for example jQuery announcing a newer, > faster release: http://blog.jquery.com/2010/10/16/jquery-143-released/ > > (scroll down for lots of performance plots) > > *However*, it seems that for the last release they have done what you > propose: http://blog.jquery.com/2011/01/31/jquery-15-released/ > > They have changed the units from "seconds" to number of "iterations per second". > > In any case that is something for the PyPy developers to decide. > > Cheers, > Miquel > > > 2011/3/17 Laura Creighton : >> >> Smaller is better when you are dieting. ?Or when you are racing. >> Given that there is talk that we will measure memory size as well, and >> turn into performance.pypy.org then I think that the 'smaller is >> better' idea will be well understood. ?As a practical matter, making >> 'faster' be 'bigger' doesn't make sense in terms of benchmarks, in which >> you want to be the first to finish. >> >> Laura >> >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev >> > From holger at merlinux.eu Fri Mar 18 10:25:19 2011 From: holger at merlinux.eu (holger krekel) Date: Fri, 18 Mar 2011 09:25:19 +0000 Subject: [pypy-dev] ideas for Google Summer of code In-Reply-To: References: <4D82322F.7070700@gmail.com> <201103171634.p2HGYKVf029380@theraft.openend.se> <201103180010.p2I0AVLY006544@theraft.openend.se> Message-ID: <20110318092519.GA16231@merlinux.eu> Hi Greg, all, On Thu, Mar 17, 2011 at 21:53 -0700, Greg Price wrote: > Hi Laura, > > On Mar 17, 2011 5:10 PM, "Laura Creighton" wrote: > > > > I'd like speed.python.org to become performance.python.org so we can > > measure memoryt consumption too. > > We should definitely measure memory consumption too, but "speed.python.org" > has a nice rhythm to it - few syllables, quick to say - such that I think it > might make sense to keep the name even so. To me, "speed" also suggests that > it *is* fast, more so than "performance" suggests that the performance > actually is good, as opposed to neutrally measuring it, whatever it is. I > always thought "speed.pypy.org" quite a bold name, saying "we are fast" > right there in the domain so nobody could miss the point. Maybe that last is > just me, though. Nope, that was actually the idea when we came up with the domain name and i think it makes sense to keep it such :) But maybe we can postpone further discussion until when we have nice memory measurement. cheers, holger From holger at merlinux.eu Fri Mar 18 14:32:19 2011 From: holger at merlinux.eu (holger krekel) Date: Fri, 18 Mar 2011 13:32:19 +0000 Subject: [pypy-dev] Documentation sprint at PyCon In-Reply-To: <1300386020.16889.16.camel@surprise> References: <1300386020.16889.16.camel@surprise> Message-ID: <20110318133219.GB16231@merlinux.eu> Hey David, Laura, Armin, nice work! I just removed the documentation tests now that sphinx performs link-checking. I noticed that the :config:`OPTNAME` was not ported to sphinx. It generated a link to the appropriate config option. Maybe some one can port it or we can just manually replace the few places where it is used. The docutils role-registering code is in pypy/config/makerestdoc.py and is a bit involved - maybe Carl Friedrich can tell if this is still neccessary. If someone takes care for setting up readthedocs or some other place to generate the docs i can point the doc.pypy.org domain name to it. cheers, holger On Thu, Mar 17, 2011 at 14:20 -0400, David Malcolm wrote: > Laura, me, and others sprinted on documentation cleanups at PyCon, using > https://bitbucket.org/dmalcolm/pypy-dmalcolm > as a development branch. > > Laura and Armin just merged the changes from that repo > > Significant changes are: > - the Sphinxification of the docs > - the renaming of the sources from .txt to .rst, to play better with > text editors with smarts for rst > - organizing the documentation, to try to stress the high-important > up-to-date material, whilst moving the more out-of-date materials to a > clearly-marked annex (see cleanup.rst). > > Running: > make html > within pypy/docs should now generate the documentation part of the site. > > You can also use: > make linkcheck > to check links; we believe we've fixed all internal links, but some > external links are still broken. > > There's a nasty hack for pointing at sources which you can see in: > https://bitbucket.org/pypy/pypy/changeset/c6f7ecf2dc01 > but fixing this properly looks like we would need to write a new sphinx > plugin; as it is, it works, but is ugly at the rst level. > > Going forward, our idea is that http://docs.pypy.org ought to point at > the sphinx-generated html, and the "development" link on pypy.org should > point to docs.pypy.org (moving another thing off of codespeak). > > The folks at readthedocs.org have offered hosting space, and can keep > http://pypy.readthedocs.org > up-to-date with a nightly build of the documentation. > > They have some docs on setting up a CNAME for this: > http://read-the-docs.readthedocs.org/en/latest/alternate_domains.html#cname-support > so that docs.pypy.org can be set up to point there. > > Currently that site has an out-of-date build of my bitbucket branch I > mentioned at the top of this mail (and there've been a lot of changes > since then). > > Someone will need to sync up with the readthedocs folks to make their > site points at the correct bitbucket repo, and to ensure that it's > getting regularly rebuilt > > Hope this makes sense and is helpful. > Dave > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From lac at openend.se Fri Mar 18 17:48:46 2011 From: lac at openend.se (Laura Creighton) Date: Fri, 18 Mar 2011 17:48:46 +0100 Subject: [pypy-dev] European sprints? In-Reply-To: Message from Laura Creighton of "Wed, 16 Feb 2011 13:08:07 +0100." <201102161208.p1GC875O027115@theraft.openend.se> References: <58722.170.148.215.157.1297760241.squirrel@mail5.webfaction.com> <201102151054.p1FAsZWb018955@theraft.openend.se> <4D5A8774.3050700@gmx.de> <4D5A887A.2040206@gmail.com> <4D5BAE92.2080306@changemaker.nu> <201102161208.p1GC875O027115@theraft.openend.se> Message-ID: <201103181648.p2IGmktc030478@theraft.openend.se> In a message of Wed, 16 Feb 2011 13:08:07 +0100, Laura Creighton writes: >In a message of Wed, 16 Feb 2011 12:01:38 +0100, Bea During writes: >>Here is a suggestion of places and dates based on Lauras, Carl Friedrich > >>and Antos >>input: >> >>- Gothenburg sprint: 25th of April to 1st of May >>- Europython sprint/Florence: 25th of June to 26th of June (EP2011 >>official sprint dates) >>- D?sseldorf sprint: 22nd of August to 28th of August (fits the plan of >>the funded PyJIT project which ends end August) >> >>What do you think about these dates - would they work? >> >>Cheers >> >>Bea There is now a reason to like the week before Florence EuroPython June 20-26 -- so June 12 - 20 or so -- EuroDjangoCon is June 6-10 in Amsterdam http://djangocon.eu/ So people fron out of Europe could plan a Python summer, if they were coming to EuroDjangocon anyway. Also, I have just got mail from the original author of this thread. He'd like to book tickets to come to G?teborg and sprint with us. Can we confirm the dates? Laura From anto.cuni at gmail.com Sat Mar 19 10:07:21 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Sat, 19 Mar 2011 10:07:21 +0100 Subject: [pypy-dev] [pypy-svn] pypy fold_intadd: Changed test to reflect my optimizeopt's decision to emit int_sub(iX, -x) when x < 0 In-Reply-To: <20110319034511.8510A282BAA@codespeak.net> References: <20110319034511.8510A282BAA@codespeak.net> Message-ID: <4D847249.8000401@gmail.com> Hi Daniel, On 19/03/11 04:45, ademan wrote: > Author: Daniel Roberts > Branch: fold_intadd > Changeset: r42802:386510d0fb45 > Date: 2011-03-18 20:41 -0700 > http://bitbucket.org/pypy/pypy/changeset/386510d0fb45/ > > Log: Changed test to reflect my optimizeopt's decision to emit > int_sub(iX, -x) when x < 0 what happens when you are on 32 bit and see int_add(i0, -2147483648)? ciao, Anto From dimaqq at gmail.com Sat Mar 19 17:19:52 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Sat, 19 Mar 2011 09:19:52 -0700 Subject: [pypy-dev] phb impression from speed.pypy.org In-Reply-To: References: <201103171722.p2HHMdIs001438@theraft.openend.se> Message-ID: I'm all for smaller is better in the evolution plot Seems more rational except you can't call it "performance" in the header above the plot then. On 18 March 2011 00:23, Miquel Torres wrote: > Oh! it just downed on me that there is a very easy way to do a "bigger > is better" plot without changing any of the current views of plots. > > The "How has PyPy performance evolved over time?" plot, with > represents the geometric average of all normalized benchmarks. You may > have noticed the number inside parenthesis, which it is precisely the > inverse. > > That one plot can thus be easily inverted, to be able to show "times > faster than", without changing the rest of the site. > > > > 2011/3/18 Miquel Torres : >> I understand what you say, and it is certainly possible to turn >> benchmarks into "bigger is better". As Laura wrote though, it is only >> natural to measure time, because that is what you actually what to do: >> reduce the time it takes to complete some task. >> >> Other projects do the same. See for example jQuery announcing a newer, >> faster release: http://blog.jquery.com/2010/10/16/jquery-143-released/ >> >> (scroll down for lots of performance plots) >> >> *However*, it seems that for the last release they have done what you >> propose: http://blog.jquery.com/2011/01/31/jquery-15-released/ >> >> They have changed the units from "seconds" to number of "iterations per second". >> >> In any case that is something for the PyPy developers to decide. >> >> Cheers, >> Miquel >> >> >> 2011/3/17 Laura Creighton : >>> >>> Smaller is better when you are dieting. ?Or when you are racing. >>> Given that there is talk that we will measure memory size as well, and >>> turn into performance.pypy.org then I think that the 'smaller is >>> better' idea will be well understood. ?As a practical matter, making >>> 'faster' be 'bigger' doesn't make sense in terms of benchmarks, in which >>> you want to be the first to finish. >>> >>> Laura >>> >>> _______________________________________________ >>> pypy-dev at codespeak.net >>> http://codespeak.net/mailman/listinfo/pypy-dev >>> >> > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From arigo at tunes.org Sun Mar 20 14:14:46 2011 From: arigo at tunes.org (Armin Rigo) Date: Sun, 20 Mar 2011 14:14:46 +0100 Subject: [pypy-dev] [pypy-svn] pypy fold_intadd: Changed test to reflect my optimizeopt's decision to emit int_sub(iX, -x) when x < 0 In-Reply-To: <4D847249.8000401@gmail.com> References: <20110319034511.8510A282BAA@codespeak.net> <4D847249.8000401@gmail.com> Message-ID: Hi Anto, hi Ademan, On Sat, Mar 19, 2011 at 10:07 AM, Antonio Cuni wrote: > what happens when you are on 32 bit and see int_add(i0, -2147483648)? It probably works, because -(-2147483648) == -2147483648, and int_add(i0, -2147483648) and int_sub(i0, -2147483648) do the same thing. However I have no clue why this replacement is giving you anything. If anything at all I would say that replacing int_add(i0, -128) with int_sub(i0, 128) is a (very marginally) bad idea on x86 because the constant -128 fits in 8 bits but the constant 128 doesn't. Well, I suppose that "it makes it easier for me, the human, to understand it" is a valid reason after all. A bient?t, Armin. From brownan at gmail.com Tue Mar 22 20:44:36 2011 From: brownan at gmail.com (Andrew Brown) Date: Tue, 22 Mar 2011 15:44:36 -0400 Subject: [pypy-dev] Pypy custom interpreter JIT question Message-ID: Hello PyPy team, So I became interested in the translation toolchain plus JIT compiler generator, and in attempt to learn more about it, I set out to write a very simple interpreter for the language BF (brainf***). Simple enough, 8 opcodes, each with no arguments, and most have a one line implementation. Also plenty of examples out there to run, like a mandelbrot generator =) So I wrote up an interpreter for it in RPython. This worked great until I tried to enable the JIT option, at which point it would produce incorrect results. Strange, I thought I may have been using the hints incorrectly, I couldn't find too many details on exactly what the red and green vars should be. Here's the strange part though: I finally fixed it by changing an implementation detail that shouldn't have changed semantics at all. My implementation creates an instance of a Tape object which has two attributes: a list of integers representing the state of the machine, and a single integer identifying the current active cell on the tape. The implementation of each opcode was a method of this class, and the state of the program (what I passed as the "red" variable) was the instance of this class. After I manually factored the functionality of the class directly into the main dispatch loop and got rid of the class entirely, the JIT compiler started producing correct results. Can anyone help me figure out why my first attempt didn't work? Do red variables that are in class instances need to be handled different somehow? Here's the initial version that runs incorrectly when translated with JIT: https://bitbucket.org/brownan/bf-interpreter/src/c4679b354313/targetbf.py Here's the modified version that seems to work just fine: https://bitbucket.org/brownan/bf-interpreter/src/8095853278e9/targetbf.py In particular, note the elimination of the Tape object in the second version, and the differences in the mainloop function as well as the differences in the "red" variables. I've also included a few example BF programs if someone wants to try it out. The hanoi example crashes almost immediately with the first version translated with JIT. By the way, I've been translating it with the latest version of PyPy off of bitbucket. (latest as of a few weeks ago, that is) Thanks and great work on this project! -Andrew Brown -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110322/11394033/attachment.htm From tbaldridge at gmail.com Tue Mar 22 21:41:19 2011 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Tue, 22 Mar 2011 15:41:19 -0500 Subject: [pypy-dev] Implementing CAS Message-ID: After my last discussion about getting some multi-threading primitives, I took a look at how locks are implemented in PyPy. PyPy currently uses OS level mutexes...which is okay for most uses, but in my case, I need super fast spinlocks, and for that I need a CAS operation. What I'm looking for is a bit of direction on how to go about implementing this function (shown in C syntax): int cas(*expected, *new, **location) This function looks at the contents of the data pointed at by **location. If the contents == *expected, the contents are replaced with *new and the function returns true. Otherwise the function returns false. There are several issues I see with implementing this in python: 1) **location must be pointer to a object pointer, is this even possible in PyPy? 2) I'd rather not call an FFI C function for this, when this entire function could be inlined with just a few lines of assembly (CAS is a single instruction on x86). So could this be created as a C function that is simply inserted into the generated C code, so that GCC could inline it at will? I'm just brainstorming here. No matter how you dice it, CAS is pretty much a prerequisite to any sort of multi-threaded programming these days. That is unless you want to spend thousands of clock cycles in context switches. Timothy -- ?One of the main causes of the fall of the Roman Empire was that?lacking zero?they had no way to indicate successful termination of their C programs.? (Robert Firth) From benjamin at python.org Tue Mar 22 22:14:09 2011 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 22 Mar 2011 16:14:09 -0500 Subject: [pypy-dev] Implementing CAS In-Reply-To: References: Message-ID: 2011/3/22 Timothy Baldridge : > After my last discussion about getting some multi-threading > primitives, I took a look at how locks are implemented in PyPy. PyPy > currently uses OS level mutexes...which is okay for most uses, but in > my case, I need super fast spinlocks, and for that I need a CAS > operation. What I'm looking for is a bit of direction on how to go > about implementing this function (shown in C syntax): > > int cas(*expected, *new, **location) I suggest you create a new low-level operation. Then the C backend can translate it into asm. -- Regards, Benjamin From fijall at gmail.com Tue Mar 22 22:18:48 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 22 Mar 2011 15:18:48 -0600 Subject: [pypy-dev] Implementing CAS In-Reply-To: References: Message-ID: On Tue, Mar 22, 2011 at 3:14 PM, Benjamin Peterson wrote: > 2011/3/22 Timothy Baldridge : >> After my last discussion about getting some multi-threading >> primitives, I took a look at how locks are implemented in PyPy. PyPy >> currently uses OS level mutexes...which is okay for most uses, but in >> my case, I need super fast spinlocks, and for that I need a CAS >> operation. What I'm looking for is a bit of direction on how to go >> about implementing this function (shown in C syntax): >> >> int cas(*expected, *new, **location) > > I suggest you create a new low-level operation. Then the C backend can > translate it into asm. > Which requires touching multiple places, but it's relatively easy. Start with writing a test and then add it to pypy/rpython/lltypesystem/lloperation.py > > -- > Regards, > Benjamin > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From arigo at tunes.org Tue Mar 22 22:54:33 2011 From: arigo at tunes.org (Armin Rigo) Date: Tue, 22 Mar 2011 22:54:33 +0100 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: References: Message-ID: Hi Andrew, On Tue, Mar 22, 2011 at 8:44 PM, Andrew Brown wrote: > https://bitbucket.org/brownan/bf-interpreter/src/c4679b354313/targetbf.py can_enter_jit() is not correct. For it to work, it must be called just before jit_merge_point(). It's wrong that there are two intermediate instructions here: "pc+=1" and the "pc < len(program)" condition. As a first attempt, you should just not call can_enter_jit() at all. Nowadays, if can_enter_jit is never called, it's done automatically for you; moreover, a misplaced can_enter_jit can give nonsensical results, as opposed to many other hints, which cannot give a result worst than terribly bad performance (like the greens/reds variable separation --- which seems correct in your example). A bient?t, Armin. From lac at openend.se Wed Mar 23 02:17:21 2011 From: lac at openend.se (Laura Creighton) Date: Wed, 23 Mar 2011 02:17:21 +0100 Subject: [pypy-dev] status of the graphviz server on codespeak? Message-ID: <201103230117.p2N1HLfS016404@theraft.openend.se> getting-started-dev says: Download and install `Dot Graphviz`_ (optional if you have an internet connection: the flowgraph viewer then connects to codespeak.net and lets it convert the flowgraph by a graphviz server). What's going to happen to that when codespeak goes away? Laura From fijall at gmail.com Wed Mar 23 02:36:02 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 22 Mar 2011 19:36:02 -0600 Subject: [pypy-dev] status of the graphviz server on codespeak? In-Reply-To: <201103230117.p2N1HLfS016404@theraft.openend.se> References: <201103230117.p2N1HLfS016404@theraft.openend.se> Message-ID: On Tue, Mar 22, 2011 at 7:17 PM, Laura Creighton wrote: > getting-started-dev says: > > Download and install `Dot Graphviz`_ (optional if you have an internet > ? ?connection: the flowgraph viewer then connects to > ? ?codespeak.net and lets it convert the flowgraph by a graphviz server). > > What's going to happen to that when codespeak goes away? This one is defunct for ages now I think. > > Laura > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From pypy at pocketnix.org Wed Mar 23 05:34:53 2011 From: pypy at pocketnix.org (pypy at pocketnix.org) Date: Wed, 23 Mar 2011 04:34:53 +0000 Subject: [pypy-dev] Implementing CAS In-Reply-To: References: Message-ID: <20110323043453.GA14527@pocketnix.org> On Tue, Mar 22, 2011 at 03:41:19PM -0500, Timothy Baldridge wrote: > After my last discussion about getting some multi-threading > primitives, I took a look at how locks are implemented in PyPy. PyPy > currently uses OS level mutexes...which is okay for most uses, but in > my case, I need super fast spinlocks, and for that I need a CAS > operation. What I'm looking for is a bit of direction on how to go > about implementing this function (shown in C syntax): > > int cas(*expected, *new, **location) > > This function looks at the contents of the data pointed at by > **location. If the contents == *expected, the contents are replaced > with *new and the function returns true. Otherwise the function > returns false. > > There are several issues I see with implementing this in python: > > 1) **location must be pointer to a object pointer, is this even > possible in PyPy? > 2) I'd rather not call an FFI C function for this, when this entire > function could be inlined with just a few lines of assembly (CAS is a > single instruction on x86). So could this be created as a C function > that is simply inserted into the generated C code, so that GCC could > inline it at will? > > I'm just brainstorming here. No matter how you dice it, CAS is pretty > much a prerequisite to any sort of multi-threaded programming these > days. That is unless you want to spend thousands of clock cycles in > context switches. > > Timothy > > -- > ?One of the main causes of the fall of the Roman Empire was > that?lacking zero?they had no way to indicate successful termination > of their C programs.? > (Robert Firth) > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev Dont know if this would be helpful at all but i have been wondering if RCU's would be useful at all http://en.wikipedia.org/wiki/Read-copy-update (userspace libary at http://lttng.org/urcu ) from what i ahve read up on them i thought they would be a nice match to python and pypy From anto.cuni at gmail.com Wed Mar 23 09:02:25 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed, 23 Mar 2011 09:02:25 +0100 Subject: [pypy-dev] status of the graphviz server on codespeak? In-Reply-To: References: <201103230117.p2N1HLfS016404@theraft.openend.se> Message-ID: <4D89A911.3050004@gmail.com> On 23/03/11 02:36, Maciej Fijalkowski wrote: > On Tue, Mar 22, 2011 at 7:17 PM, Laura Creighton wrote: >> getting-started-dev says: >> >> Download and install `Dot Graphviz`_ (optional if you have an internet >> connection: the flowgraph viewer then connects to >> codespeak.net and lets it convert the flowgraph by a graphviz server). >> >> What's going to happen to that when codespeak goes away? > > This one is defunct for ages now I think. no, I think it's still up: http://codespeak.net/pypy/convertdot.cgi we can either migrate it to some other machine (tannit?) or discontinue the service. Nowadays, it's only useful for us developers, and (I think) we all have graphviz installed locally. ciao, Anto From lac at openend.se Wed Mar 23 09:36:40 2011 From: lac at openend.se (Laura Creighton) Date: Wed, 23 Mar 2011 09:36:40 +0100 Subject: [pypy-dev] status of the graphviz server on codespeak? In-Reply-To: Message from Antonio Cuni of "Wed, 23 Mar 2011 09:02:25 +0100." <4D89A911.3050004@gmail.com> References: <201103230117.p2N1HLfS016404@theraft.openend.se> <4D89A911.3050004@gmail.com> Message-ID: <201103230836.p2N8aeg6023861@theraft.openend.se> In a message of Wed, 23 Mar 2011 09:02:25 +0100, Antonio Cuni writes: >On 23/03/11 02:36, Maciej Fijalkowski wrote: >> On Tue, Mar 22, 2011 at 7:17 PM, Laura Creighton wrote >: >>> getting-started-dev says: >>> >>> Download and install `Dot Graphviz`_ (optional if you have an internet >>> connection: the flowgraph viewer then connects to >>> codespeak.net and lets it convert the flowgraph by a graphviz serve >r). >>> >>> What's going to happen to that when codespeak goes away? >> >> This one is defunct for ages now I think. > >no, I think it's still up: >http://codespeak.net/pypy/convertdot.cgi > >we can either migrate it to some other machine (tannit?) or discontinue t >he >service. Nowadays, it's only useful for us developers, and (I think) we a >ll >have graphviz installed locally. I think discontinuation is the way to go. Running on tannit is a bad idea, since it would impact our benchmarks, but running someplace else is a possibility. I don't think it's worth it. Laura > >ciao, >Anto From fijall at gmail.com Wed Mar 23 15:16:40 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 23 Mar 2011 08:16:40 -0600 Subject: [pypy-dev] status of the graphviz server on codespeak? In-Reply-To: <201103230836.p2N8aeg6023861@theraft.openend.se> References: <201103230117.p2N1HLfS016404@theraft.openend.se> <4D89A911.3050004@gmail.com> <201103230836.p2N8aeg6023861@theraft.openend.se> Message-ID: On Wed, Mar 23, 2011 at 2:36 AM, Laura Creighton wrote: > In a message of Wed, 23 Mar 2011 09:02:25 +0100, Antonio Cuni writes: >>On 23/03/11 02:36, Maciej Fijalkowski wrote: >>> On Tue, Mar 22, 2011 at 7:17 PM, Laura Creighton wrote >>: >>>> getting-started-dev says: >>>> >>>> Download and install `Dot Graphviz`_ (optional if you have an internet >>>> ? ?connection: the flowgraph viewer then connects to >>>> ? ?codespeak.net and lets it convert the flowgraph by a graphviz serve >>r). >>>> >>>> What's going to happen to that when codespeak goes away? >>> >>> This one is defunct for ages now I think. >> >>no, I think it's still up: >>http://codespeak.net/pypy/convertdot.cgi >> >>we can either migrate it to some other machine (tannit?) or discontinue t >>he >>service. Nowadays, it's only useful for us developers, and (I think) we a >>ll >>have graphviz installed locally. > > I think discontinuation is the way to go. ?Running on tannit is a bad > idea, since it would impact our benchmarks, but running someplace else > is a possibility. ?I don't think it's worth it. I assumed it doesn't work because it never worked for me :) Crash on dot also crashed later on trying to run on codespeak. I would say just remove > > Laura > >> >>ciao, >>Anto > From akg2136 at columbia.edu Wed Mar 23 16:48:40 2011 From: akg2136 at columbia.edu (Alexander Golec) Date: Wed, 23 Mar 2011 11:48:40 -0400 Subject: [pypy-dev] When translating pypy, how do I notify it of library locations? Message-ID: I'm building pypy with the default JIT configuration, and I'm on a machine where I do not have root access. I've installed the libraries that I need in my ~/lib folder, but I can't get pypy to see them. I can neither see the library files, nor the headers. I've tried setting the following options : --cflags=COMPILERFLAGS --ldflags=LINKERFLAGS but to no avail... Alex From anto.cuni at gmail.com Wed Mar 23 17:09:32 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed, 23 Mar 2011 17:09:32 +0100 Subject: [pypy-dev] When translating pypy, how do I notify it of library locations? In-Reply-To: References: Message-ID: <4D8A1B3C.1040008@gmail.com> Hi Alexander, On 23/03/11 16:48, Alexander Golec wrote: > I'm building pypy with the default JIT configuration, and I'm on a machine where I do not have root access. I've installed the libraries that I need in my ~/lib folder, but I can't get pypy to see them. I can neither see the library files, nor the headers. which libraries are you talking about? Pure python modules/packages (i.e., the ones contained in lib-python and lib_pypy) or shared libraries? ciao, Anto From lac at openend.se Wed Mar 23 19:01:09 2011 From: lac at openend.se (Laura Creighton) Date: Wed, 23 Mar 2011 19:01:09 +0100 Subject: [pypy-dev] way to refer to our online doc Message-ID: <201103231801.p2NI19Y4012616@theraft.openend.se> over on bitbucket I am looking at https://bitbucket.org/pypy/pypy/src/04d276c92744/pypy/doc/ and I assume that the 04*44 is just some revision number. How do we want to refer to the docs in general now that we have moved? Laura From amauryfa at gmail.com Wed Mar 23 19:12:57 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 23 Mar 2011 19:12:57 +0100 Subject: [pypy-dev] way to refer to our online doc In-Reply-To: <201103231801.p2NI19Y4012616@theraft.openend.se> References: <201103231801.p2NI19Y4012616@theraft.openend.se> Message-ID: Hi, 2011/3/23 Laura Creighton : > over on bitbucket I am looking at > > https://bitbucket.org/pypy/pypy/src/04d276c92744/pypy/doc/ > > and I assume that the 04*44 is just some revision number. ?How do > we want to refer to the docs in general now that we have moved? I just replaced the rev number with "default": https://bitbucket.org/pypy/pypy/src/default/pypy/doc/ Another branch name should work equally. -- Amaury Forgeot d'Arc From lac at openend.se Wed Mar 23 19:18:58 2011 From: lac at openend.se (Laura Creighton) Date: Wed, 23 Mar 2011 19:18:58 +0100 Subject: [pypy-dev] way to refer to our online doc In-Reply-To: Message from "Amaury Forgeot d'Arc" of "Wed, 23 Mar 2011 19:12:57 +0100." References: <201103231801.p2NI19Y4012616@theraft.openend.se> Message-ID: <201103231818.p2NIIwC7014407@theraft.openend.se> In a message of Wed, 23 Mar 2011 19:12:57 +0100, "Amaury Forgeot d'Arc" writes: >Hi, >2011/3/23 Laura Creighton : >> over on bitbucket I am looking at >> >> https://bitbucket.org/pypy/pypy/src/04d276c92744/pypy/doc/ >> >> and I assume that the 04*44 is just some revision number. =A0How do >> we want to refer to the docs in general now that we have moved? > >I just replaced the rev number with "default": >https://bitbucket.org/pypy/pypy/src/default/pypy/doc/ > >Another branch name should work equally. > >Amaury Forgeot d'Arc I guess the question is, do we want to refer to 'default' or to 'tip' ? Laura From amauryfa at gmail.com Wed Mar 23 19:23:24 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 23 Mar 2011 19:23:24 +0100 Subject: [pypy-dev] way to refer to our online doc In-Reply-To: <201103231818.p2NIIwC7014407@theraft.openend.se> References: <201103231801.p2NI19Y4012616@theraft.openend.se> <201103231818.p2NIIwC7014407@theraft.openend.se> Message-ID: 2011/3/23 Laura Creighton : > I guess the question is, do we want to refer to 'default' or to 'tip' ? "tip" refers to the last committed change, and may belongs to any development branch -- Amaury Forgeot d'Arc From brownan at gmail.com Wed Mar 23 20:27:40 2011 From: brownan at gmail.com (Andrew Brown) Date: Wed, 23 Mar 2011 15:27:40 -0400 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: References: Message-ID: On Tue, Mar 22, 2011 at 5:54 PM, Armin Rigo wrote: > can_enter_jit() is not correct. For it to work, it must be called > just before jit_merge_point(). It's wrong that there are two > intermediate instructions here: "pc+=1" and the "pc < len(program)" > condition. > Okay, I think I understand. I'm still learning how all this stuff works. Regardless... > > As a first attempt, you should just not call can_enter_jit() at all. > Nowadays, if can_enter_jit is never called, it's done automatically > for you; I did not know this. Good to know! I've removed the can_enter_jit() call from the two versions of my interpreter. However, the version that didn't work before still does not run correctly. It seems like I'm still left with the same problem as before. This works (version without Tape class and with can_enter_jit call removed) https://bitbucket.org/brownan/bf-interpreter/src/6c6c80397554/targetbf.py Incidentally, I think it may run slightly slower now, but I'm not sure. This version still does not (version *with* Tape class, and with can_enter_jit removed) https://bitbucket.org/brownan/bf-interpreter/src/1d16c3eed7e2/targetbf.py Any other ideas? I'm still at a loss. Thanks for taking a look! -Andrew -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110323/74980c11/attachment.htm From fijall at gmail.com Wed Mar 23 20:43:33 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 23 Mar 2011 13:43:33 -0600 Subject: [pypy-dev] When translating pypy, how do I notify it of library locations? In-Reply-To: References: Message-ID: On Wed, Mar 23, 2011 at 9:48 AM, Alexander Golec wrote: > I'm building pypy with the default JIT configuration, and I'm on a machine where I do not have root access. I've installed the libraries that I need in my ~/lib folder, but I can't get pypy to see them. I can neither see the library files, nor the headers. > > I've tried setting the following options : > > --cflags=COMPILERFLAGS > --ldflags=LINKERFLAGS > > but to no avail... Can you explain exactly what didn't work? I think we honor those options. > > Alex > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From alex.perry at pamurray.com Thu Mar 24 01:50:01 2011 From: alex.perry at pamurray.com (Alex Perry) Date: Wed, 23 Mar 2011 17:50:01 -0700 Subject: [pypy-dev] Is the built-in re.py supposed to translate? In-Reply-To: References: Message-ID: $ ./pypy-c translate.py target.py [...] ?AnnotatorError: annotation of degenerated to SomeObject() ?Simple call of incompatible family: ? ? ? (KeyError getting at the binding!) ?Happened at file /usr/src/pypy/lib-python/2.7.0/re.py line 232 ? ? ? ? cachekey = (type(key[0]),) + key ?==> ? ? p = _cache.get(cachekey) ? ? ? ? if p is not None: ?Previous annotation: ? (none) ?Processing block: ?block at 9 is a ?in (re:229)_compile__star_2 ?containing the following operations: ? ? ? ?v0 = getitem(key_0, (0)) ? ? ? ?v1 = type(v0) ? ? ? ?v2 = newtuple(v1) ? ? ? ?v3 = add(v2, key_0) ? ? ? ?v4 = simple_call((method get), v3) ? ? ? ?v5 = is_(v4, (None)) ? ? ? ?v6 = is_true(v5) ?--end-- $ cat target.py #! /usr/bin/python2.6 def entry_point(argv): ?import re ?s = re.sub(r'[^a-fA-F\d]', '', str(argv[1])) ?print s.decode('hex') def target(*args): ?return entry_point, None if __name__ == '__main__': ?import sys ?entry_point(sys.argv) From alex.gaynor at gmail.com Thu Mar 24 01:51:24 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Wed, 23 Mar 2011 20:51:24 -0400 Subject: [pypy-dev] Is the built-in re.py supposed to translate? In-Reply-To: References: Message-ID: On Wed, Mar 23, 2011 at 8:50 PM, Alex Perry wrote: > $ ./pypy-c translate.py target.py > [...] > AnnotatorError: annotation of object at 0x0000000004cd7a98> degenerated to SomeObject() > Simple call of incompatible family: > (KeyError getting at the binding!) > > Happened at file /usr/src/pypy/lib-python/2.7.0/re.py line 232 > > cachekey = (type(key[0]),) + key > ==> p = _cache.get(cachekey) > if p is not None: > > Previous annotation: > (none) > Processing block: > block at 9 is a > in (re:229)_compile__star_2 > containing the following operations: > v0 = getitem(key_0, (0)) > v1 = type(v0) > v2 = newtuple(v1) > v3 = add(v2, key_0) > v4 = simple_call((method get), v3) > v5 = is_(v4, (None)) > v6 = is_true(v5) > --end-- > > $ cat target.py > #! /usr/bin/python2.6 > > def entry_point(argv): > import re > s = re.sub(r'[^a-fA-F\d]', '', str(argv[1])) > print s.decode('hex') > > def target(*args): > return entry_point, None > > if __name__ == '__main__': > import sys > entry_point(sys.argv) > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > Nope, basically nothing in the stdlib is RPython. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110323/5c77f1ee/attachment.htm From alex.gaynor at gmail.com Thu Mar 24 02:06:59 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Wed, 23 Mar 2011 21:06:59 -0400 Subject: [pypy-dev] Is the built-in re.py supposed to translate? In-Reply-To: References: Message-ID: On Wed, Mar 23, 2011 at 9:01 PM, Alex Perry wrote: > I'm confused then. I thought all imports were supposed to go through > the type emulation before they hit rpython and the nitty gritty of > translation. Do I have to get my target.py to load the type emulator > and then import re.py on top of it? > > On Wed, Mar 23, 2011 at 5:51 PM, Alex Gaynor > wrote: > > > > > > On Wed, Mar 23, 2011 at 8:50 PM, Alex Perry > wrote: > >> > >> $ ./pypy-c translate.py target.py > >> [...] > >> AnnotatorError: annotation of >> object at 0x0000000004cd7a98> degenerated to SomeObject() > >> Simple call of incompatible family: > >> (KeyError getting at the binding!) > >> > >> Happened at file /usr/src/pypy/lib-python/2.7.0/re.py line 232 > >> > >> cachekey = (type(key[0]),) + key > >> ==> p = _cache.get(cachekey) > >> if p is not None: > >> > >> Previous annotation: > >> (none) > >> Processing block: > >> block at 9 is a > >> in (re:229)_compile__star_2 > >> containing the following operations: > >> v0 = getitem(key_0, (0)) > >> v1 = type(v0) > >> v2 = newtuple(v1) > >> v3 = add(v2, key_0) > >> v4 = simple_call((method get), v3) > >> v5 = is_(v4, (None)) > >> v6 = is_true(v5) > >> --end-- > >> > >> $ cat target.py > >> #! /usr/bin/python2.6 > >> > >> def entry_point(argv): > >> import re > >> s = re.sub(r'[^a-fA-F\d]', '', str(argv[1])) > >> print s.decode('hex') > >> > >> def target(*args): > >> return entry_point, None > >> > >> if __name__ == '__main__': > >> import sys > >> entry_point(sys.argv) > >> _______________________________________________ > >> pypy-dev at codespeak.net > >> http://codespeak.net/mailman/listinfo/pypy-dev > > > > Nope, basically nothing in the stdlib is RPython. > > > > Alex > > > > -- > > "I disapprove of what you say, but I will defend to the death your right > to > > say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > > "The people's good is the highest law." -- Cicero > > > > > I'm not sure I follow, what is the type emulator? Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110323/a624fc22/attachment.htm From arigo at tunes.org Thu Mar 24 17:11:18 2011 From: arigo at tunes.org (Armin Rigo) Date: Thu, 24 Mar 2011 17:11:18 +0100 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: References: Message-ID: Hi Andrew, On Wed, Mar 23, 2011 at 8:27 PM, Andrew Brown wrote: > However, the version that didn't work before still does not run > correctly. It seems like I'm still left with the same problem as before. Ah. Looking more closely, it turns out to be a bug in the optimization step of the JIT which just never showed up so far :-/ Working on it, by cleaning up optimizeopt/heap.py... Armin From arigo at tunes.org Thu Mar 24 17:56:29 2011 From: arigo at tunes.org (Armin Rigo) Date: Thu, 24 Mar 2011 17:56:29 +0100 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: References: Message-ID: Re-hi, On Thu, Mar 24, 2011 at 5:11 PM, Armin Rigo wrote: > Working on it, by cleaning up optimizeopt/heap.py... Done, at least as far as fixing the bug is concerned. Now your original version (with can_enter_jit removed) works. Armin From alex.perry at pamurray.com Thu Mar 24 22:14:56 2011 From: alex.perry at pamurray.com (Alex Perry) Date: Thu, 24 Mar 2011 14:14:56 -0700 Subject: [pypy-dev] Ctypes module written in RPython Message-ID: I can't find it in the docs, but it has been alluded to in the past. How far is the project from being able to compile a rpython module? I'd expect that to emit a wrapper that imports ctypes and declares calls into a shared library, and a directory of C code which implements the internals and can compile into the expected shared library. From fijall at gmail.com Fri Mar 25 01:25:40 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 24 Mar 2011 18:25:40 -0600 Subject: [pypy-dev] Ctypes module written in RPython In-Reply-To: References: Message-ID: On Thu, Mar 24, 2011 at 3:14 PM, Alex Perry wrote: > I can't find it in the docs, but it has been alluded to in the past. > How far is the project from being able to compile a rpython module? > I'd expect that to emit a wrapper that imports ctypes and declares > calls into a shared library, and a directory of C code which > implements the internals and can compile into the expected shared > library. Hi. I'm a bit confused what you're trying to achieve. re compiler is not RPython, however the sre module (which runs the regular expression is). Regular expressions are kind of fast and they can be improved in places where JIT don't run, but in general re module is faster than the equivalent written in RPython, because it's jitted. I completely don't follow the second part of your mail. Again: what are you trying to achieve? > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From cfbolz at gmx.de Fri Mar 25 10:39:37 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Fri, 25 Mar 2011 10:39:37 +0100 Subject: [pypy-dev] Ctypes module written in RPython In-Reply-To: References: Message-ID: <4D8C62D9.2030407@gmx.de> On 03/24/2011 10:14 PM, Alex Perry wrote: > I can't find it in the docs, but it has been alluded to in the past. > How far is the project from being able to compile a rpython module? > I'd expect that to emit a wrapper that imports ctypes and declares > calls into a shared library, and a directory of C code which > implements the internals and can compile into the expected shared > library. I think the crucial part of your question that you don't state is that you want to be able to use these RPython modules compiled to C from CPython and Jython or really anything that supports ctypes, right? Carl Friedrich From arigo at tunes.org Fri Mar 25 11:41:16 2011 From: arigo at tunes.org (Armin Rigo) Date: Fri, 25 Mar 2011 11:41:16 +0100 Subject: [pypy-dev] Thinking about the GIL In-Reply-To: <41662.1300409081@iinet.net.au> References: <41662.1300409081@iinet.net.au> Message-ID: Hi, On Fri, Mar 18, 2011 at 1:44 AM, hyarion at iinet.net.au wrote: > (...) Wrapping each bytecode in an STM > transaction would give you an as-if-serial execution order, again with no guarantees > about which order. You get transaction overheads instead of lock/unlock overheads, > but some STM systems can be quite efficient for short transactions that rarely > conflict. Yes, I also thought about this as one of the solutions that would "fit the model" of PyPy by not needing changes all over the place. However, I am unsure that the performance of STM is good enough for that application so far. Maybe I'm wrong, but I fear (a priori, with no precise experience) that it would be really too slow to wrap *all* memory reads and writes with the STM machinery. I would be interested in learning if I'm wrong, or if there are hardware solutions around the corner ready to be tried. A bient?t, Armin. From tbaldridge at gmail.com Fri Mar 25 16:00:22 2011 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Fri, 25 Mar 2011 10:00:22 -0500 Subject: [pypy-dev] Runtime module loading Message-ID: I have a program written in RPython. Is there a built-in way to import other RPython modules at runtime? Or is that some sort of upcoming feature? Basically I'm writing a language using RPython and I'm trying to figure out how to allow the user to load modules without making them create an FFI. Timothy -- ?One of the main causes of the fall of the Roman Empire was that?lacking zero?they had no way to indicate successful termination of their C programs.? (Robert Firth) From brownan at gmail.com Fri Mar 25 17:47:20 2011 From: brownan at gmail.com (Andrew Brown) Date: Fri, 25 Mar 2011 12:47:20 -0400 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: References: Message-ID: Thanks, it does indeed work now! -Andrew On Thu, Mar 24, 2011 at 12:56 PM, Armin Rigo wrote: > Re-hi, > > On Thu, Mar 24, 2011 at 5:11 PM, Armin Rigo wrote: > > Working on it, by cleaning up optimizeopt/heap.py... > > Done, at least as far as fixing the bug is concerned. Now your > original version (with can_enter_jit removed) works. > > Armin > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110325/b629dbc2/attachment.htm From arigo at tunes.org Fri Mar 25 19:18:10 2011 From: arigo at tunes.org (Armin Rigo) Date: Fri, 25 Mar 2011 19:18:10 +0100 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: References: Message-ID: Hi Andrew, On Fri, Mar 25, 2011 at 5:47 PM, Andrew Brown wrote: > Thanks, it does indeed work now! The next step is to have a look at the traces produced (run with PYPYLOG=jit-log-opt:logfile), and spot the obvious missing optimizations. The biggest issue seems to be the fact that the dictionary 'bracket_map' is green, but it is not enough to ensure that it is a constant dict (it could be mutated behind the JIT's back); so in the end, every trace contains reads from it. You could fix it by moving the line newpc = bracket_map[pc] to a new function to which you apply the decorator @pypy.rlib.jit.pure_function. A bient?t, Armin. From benjamin at python.org Fri Mar 25 19:41:46 2011 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 25 Mar 2011 13:41:46 -0500 Subject: [pypy-dev] Runtime module loading In-Reply-To: References: Message-ID: 2011/3/25 Timothy Baldridge : > I have a program written in RPython. Is there a built-in way to import > other RPython modules at runtime? Or is that some sort of upcoming > feature? Basically I'm writing a language using RPython and I'm trying > to figure out how to allow the user to load modules without making > them create an FFI. No. -- Regards, Benjamin From dimaqq at gmail.com Sat Mar 26 05:53:30 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Fri, 25 Mar 2011 21:53:30 -0700 Subject: [pypy-dev] cpyext reference counting and other gc's Message-ID: Hey, I had a look at cpyext recently and saw that reference counting is emulated with, err, reference counting, seeminlgy in the referenced object itself. Does this mean that cpyext would not work with other gc's or is there some wrapping going on behind the scenes? From amauryfa at gmail.com Sat Mar 26 09:33:45 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Sat, 26 Mar 2011 09:33:45 +0100 Subject: [pypy-dev] cpyext reference counting and other gc's In-Reply-To: References: Message-ID: Hi, 2011/3/26 Dima Tisnek : > Hey, I had a look at cpyext recently and saw that reference counting > is emulated with, err, reference counting, seeminlgy in the referenced > object itself. > > Does this mean that cpyext would not work with other gc's or is there > some wrapping going on behind the scenes? Cpyext works with all pypy gc's. The PyObject* exposed to C code is actually a proxy to the "real" interpreter object; a dict lookup is necessary each time a reference crosses the C/pypy boundary. Yes, this is slow. This is implemented in pypy/module/cpyext/pyobject.py; the main functions are create_ref() and from_ref(). -- Amaury Forgeot d'Arc From arigo at tunes.org Sat Mar 26 11:50:36 2011 From: arigo at tunes.org (Armin Rigo) Date: Sat, 26 Mar 2011 11:50:36 +0100 Subject: [pypy-dev] Runtime module loading In-Reply-To: References: Message-ID: Hi Timothy, On Fri, Mar 25, 2011 at 4:00 PM, Timothy Baldridge wrote: > I have a program written in RPython. Is there a built-in way to import > other RPython modules at runtime? No, and that's not obviously fixed; for example, the GC relies on having a table of all the RPython types in the program. In order to load dynamically new RPython extension modules, we would need to do something about that. Nothing unsolvable, but there are quite a few such places around. A bient?t, Armin. From dimaqq at gmail.com Sat Mar 26 18:03:47 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Sat, 26 Mar 2011 10:03:47 -0700 Subject: [pypy-dev] cpyext reference counting and other gc's In-Reply-To: References: Message-ID: I have an alternative idea in mind I'll write up a doc, stick it on github and share with you guys in a couple of days thanks for a clear answer, I just couldn't figure that out form code easily :P d. On 26 March 2011 01:33, Amaury Forgeot d'Arc wrote: > Hi, > > 2011/3/26 Dima Tisnek : >> Hey, I had a look at cpyext recently and saw that reference counting >> is emulated with, err, reference counting, seeminlgy in the referenced >> object itself. >> >> Does this mean that cpyext would not work with other gc's or is there >> some wrapping going on behind the scenes? > > Cpyext works with all pypy gc's. The PyObject* exposed to C code is actually > a proxy to the "real" interpreter object; a dict lookup is necessary each time a > reference crosses the C/pypy boundary. Yes, this is slow. > > This is implemented in pypy/module/cpyext/pyobject.py; the main functions are > create_ref() and from_ref(). > > -- > Amaury Forgeot d'Arc > From arigo at tunes.org Sun Mar 27 19:49:00 2011 From: arigo at tunes.org (Armin Rigo) Date: Sun, 27 Mar 2011 19:49:00 +0200 Subject: [pypy-dev] Next sprint Message-ID: Hi all, The next sprint will most probably occur in Gothenburg, April 25th to May 1st. A proper sprint announcement should follow shortly. A bient?t, Armin. From romain.py at gmail.com Sun Mar 27 22:07:04 2011 From: romain.py at gmail.com (Romain Guillebert) Date: Sun, 27 Mar 2011 21:07:04 +0100 Subject: [pypy-dev] Google Summer of Code Idea Proposal Message-ID: <20110327200703.GA12448@ubuntu> Hi I'm interested in doing the Summer of code on PyPy, from what I saw on the mailing list and on irc, I would like to work on 2 things which might interest you (there is no order of preference). * Python backend for Cython (to interface with C code, this could use use ctypes or an API exposed by PyPy), the backend would either produce python code or python bytecode and will allow PyPy to JIT the Cython extension (I talked about it with Alex Gaynor yesterday). * Improve the JVM backend, making it translate on the trunk and interface (R)Python code with Java. (As proposed by Antonio Cuni). I'm available from the end of May (around the 25th) to the beginning of September so this shouldn't be a problem. If this is OK, which one would you prefer ? Regards Romain From ademan555 at gmail.com Sun Mar 27 22:27:48 2011 From: ademan555 at gmail.com (Dan Roberts) Date: Sun, 27 Mar 2011 13:27:48 -0700 Subject: [pypy-dev] Google Summer of Code Idea Proposal Message-ID: Hi Romain, I just wanted to chime in that I think a Python backend for Cython would be awesome. You wouldn't want to generate bytecode for compatibility reasons: PyPy doesn't use exactly CPython's bytecode, and afaik IronPython and Jython both just use their host VM bytecode directly. Generating ctypes+pure python sources would be the most generally useful, and I think that'd be the best GSoC project. Some great follow-up work would be to skip ctypes on PyPy and generate different calls, as Antonio has found that even with his awesome optimizations, ctypes is significantly (I think it was 5x) slower than the alternative, rawffi I think, but don't quote me on that or the number. As for reviving the jvm backend, I think that's a very *interesting* project, I actually have looked at doing that myself, there's a tiny bit of work in the ootype-virtualrefs that I've done. You'll need to learn a ton about how RPython works, the translator, and so on. I'm not sure that one would get approval though, especially pitted against awesome projects like your ctypes project. But don't let me discourage you from applying for *both*, I just wouldn't put all my eggs in the jvm-backend basket. Cheers, Dan On Mar 27, 2011 1:07 PM, "Romain Guillebert" wrote: -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110327/6fdd2eb1/attachment.htm From hyarion at iinet.net.au Mon Mar 28 02:53:55 2011 From: hyarion at iinet.net.au (hyarion at iinet.net.au) Date: Mon, 28 Mar 2011 00:53:55 +0000 Subject: [pypy-dev] Thinking about the GIL Message-ID: <33870.1301273635@iinet.net.au> On Fri Mar 25 21:41 , Armin Rigo sent: >Hi, > >On Fri, Mar 18, 2011 at 1:44 AM, hyarion at iinet.net.au >hyarion at iinet.net.au> wrote: >> (...) Wrapping each bytecode in an STM >> transaction would give you an as-if-serial execution order, again with no guarantees >> about which order. You get transaction overheads instead of lock/unlock overheads, >> but some STM systems can be quite efficient for short transactions that rarely >> conflict. > >Yes, I also thought about this as one of the solutions that would "fit >the model" of PyPy by not needing changes all over the place. >However, I am unsure that the performance of STM is good enough for >that application so far. Maybe I'm wrong, but I fear (a priori, with >no precise experience) that it would be really too slow to wrap *all* >memory reads and writes with the STM machinery. Yeah, I suppose to get the performance I'm thinking of you probably need to know what you're sharing (and for it not to be everything). You wouldn't need all memory reads and writes to be transactional, just ones to interpreter-level values that are mutable and shared between threads, and ones to things that represent app-level shareable values. The STM system I worked on was for a declarative language with immutable data. In that context it turns out the system only had to be careful when retrieving references from STM; what the reference points to could still be used as normal non- shared data. In a language like Python I guess almost any bit of memory could potentially be (or later become) a shared value that another thread could be using. What's the "success-rate" of the JIT's malloc-removal like? You could omit the transactional overhead for those values. Would that get close enough for the important 20% of code? You could probably do some other jiggery-pokery under the hood to minimise the amount of data protected by STM overhead too. If I had more time I'd love to look at this sort of thing. -- Ben From alex.gaynor at gmail.com Mon Mar 28 03:01:28 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Sun, 27 Mar 2011 21:01:28 -0400 Subject: [pypy-dev] Thinking about the GIL In-Reply-To: <33870.1301273635@iinet.net.au> References: <33870.1301273635@iinet.net.au> Message-ID: On Sun, Mar 27, 2011 at 8:53 PM, hyarion at iinet.net.au wrote: > > On Fri Mar 25 21:41 , Armin Rigo sent: > > >Hi, > > > >On Fri, Mar 18, 2011 at 1:44 AM, hyarion at iinet.net.au > >hyarion at iinet.net.au> wrote: > >> (...) Wrapping each bytecode in an STM > >> transaction would give you an as-if-serial execution order, again with > no > guarantees > >> about which order. You get transaction overheads instead of lock/unlock > overheads, > >> but some STM systems can be quite efficient for short transactions that > rarely > >> conflict. > > > >Yes, I also thought about this as one of the solutions that would "fit > >the model" of PyPy by not needing changes all over the place. > >However, I am unsure that the performance of STM is good enough for > >that application so far. Maybe I'm wrong, but I fear (a priori, with > >no precise experience) that it would be really too slow to wrap *all* > >memory reads and writes with the STM machinery. > > Yeah, I suppose to get the performance I'm thinking of you probably need to > know > what you're sharing (and for it not to be everything). You wouldn't need > all memory > reads and writes to be transactional, just ones to interpreter-level values > that are > mutable and shared between threads, and ones to things that represent > app-level > shareable values. > > The STM system I worked on was for a declarative language with immutable > data. In > that context it turns out the system only had to be careful when retrieving > references from STM; what the reference points to could still be used as > normal non- > shared data. In a language like Python I guess almost any bit of memory > could > potentially be (or later become) a shared value that another thread could > be using. > > What's the "success-rate" of the JIT's malloc-removal like? You could omit > the > transactional overhead for those values. Would that get close enough for > the > important 20% of code? You could probably do some other jiggery-pokery > under the > hood to minimise the amount of data protected by STM overhead too. > > If I had more time I'd love to look at this sort of thing. > > -- Ben > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > https://bitbucket.org/pypy/extradoc/raw/63e4617062b2/talk/pepm2011/bolz-allocation-removal.pdfhas info on the success rate of allocation removal. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110327/bea74927/attachment.htm From brownan at gmail.com Mon Mar 28 19:21:01 2011 From: brownan at gmail.com (Andrew Brown) Date: Mon, 28 Mar 2011 13:21:01 -0400 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: References: Message-ID: I tried that logging option once, but I didn't know how to read the logs. They're not exactly self explanatory. Is there a resource somewhere that explains how to read those logs? Regardless, I've implemented your suggestion and moved reads from that dictionary to a function decorated with @purefunction. Indeed, performance is greatly improved! Thanks! Current version: https://bitbucket.org/brownan/bf-interpreter/src/d3394345272e/targetbf.py A few questions: When the optimizer encounters a "pure" function, it must compare the objects passed in to previous invocations... does it consider the contents of container or other mutatible objects? or just the object identity, to be part of the function's input? It looks like, from logs of my new version, it's not reading from the dictionary at all during the trace, so I would guess it's not considering the actual contents of the dictionary as part of the function's input. This isn't surprising, but I just want to know for sure. Second, I noticed in jit.py the function hint() which has a parameter: "promote - promote the argument from a variable into a constant". Could this be an appropriate alternate to the @purefunction solution? Or, I'm guessing, does it just mean the name bracket_map won't change bindings, but does not impose a restriction on mutating the dictionary? -Andrew On Fri, Mar 25, 2011 at 2:18 PM, Armin Rigo wrote: > Hi Andrew, > > On Fri, Mar 25, 2011 at 5:47 PM, Andrew Brown wrote: > > Thanks, it does indeed work now! > > The next step is to have a look at the traces produced (run with > PYPYLOG=jit-log-opt:logfile), and spot the obvious missing > optimizations. The biggest issue seems to be the fact that the > dictionary 'bracket_map' is green, but it is not enough to ensure that > it is a constant dict (it could be mutated behind the JIT's back); so > in the end, every trace contains reads from it. You could fix it by > moving the line > > newpc = bracket_map[pc] > > to a new function to which you apply the decorator > @pypy.rlib.jit.pure_function. > > > A bient?t, > > Armin. > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110328/e58c14de/attachment.htm From cfbolz at gmx.de Mon Mar 28 19:25:22 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 28 Mar 2011 19:25:22 +0200 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: References: Message-ID: <4D90C482.7070608@gmx.de> On 03/28/2011 07:21 PM, Andrew Brown wrote: > I tried that logging option once, but I didn't know how to read the > logs. They're not exactly self explanatory. Is there a resource > somewhere that explains how to read those logs? Not really, no :-( > Regardless, I've implemented your suggestion and moved reads from that > dictionary to a function decorated with @purefunction. Indeed, > performance is greatly improved! Thanks! > > Current version: > https://bitbucket.org/brownan/bf-interpreter/src/d3394345272e/targetbf.py > > A few questions: > > When the optimizer encounters a "pure" function, it must compare the > objects passed in to previous invocations... does it consider the > contents of container or other mutatible objects? or just the object > identity, to be part of the function's input? Just the object's identity. > It looks like, from logs of my new version, it's not reading from the > dictionary at all during the trace, so I would guess it's not > considering the actual contents of the dictionary as part of the > function's input. This isn't surprising, but I just want to know for sure. > > Second, I noticed in jit.py the function hint() which has a parameter: > "promote - promote the argument from a variable into a constant". Could > this be an appropriate alternate to the @purefunction solution? Or, I'm > guessing, does it just mean the name bracket_map won't change bindings, > but does not impose a restriction on mutating the dictionary? > If you are interested, this blog series explains the usage of hints: http://bit.ly/bundles/cfbolz/1 The logs there are a bit niceified though. Carl Friedrich From fijall at gmail.com Tue Mar 29 00:24:33 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 28 Mar 2011 16:24:33 -0600 Subject: [pypy-dev] Google Summer of Code Idea Proposal In-Reply-To: <20110327200703.GA12448@ubuntu> References: <20110327200703.GA12448@ubuntu> Message-ID: On Sun, Mar 27, 2011 at 2:07 PM, Romain Guillebert wrote: > Hi > > I'm interested in doing the Summer of code on PyPy, from what I saw on > the mailing list and on irc, I would like to work on 2 things which > might interest you (there is no order of preference). > > * Python backend for Cython (to interface with C code, this could use > ?use ctypes or an API exposed by PyPy), the backend would either > ?produce python code or python bytecode and will allow PyPy to JIT the > ?Cython extension (I talked about it with Alex Gaynor yesterday). > > * Improve the JVM backend, making it translate on the trunk and > ?interface (R)Python code with Java. (As proposed by Antonio Cuni). > > I'm available from the end of May (around the 25th) to the beginning of > September so this shouldn't be a problem. > > If this is OK, which one would you prefer ? Hey. Personally I would vastly prefer the first one. It's more immediately usable. We do require people submitting proposals to contribute some code (like fixing a small issue) first before being accepted. Anywhere you would like to contribute something? Feel free to pop up on IRC and ask people around what's interesting. Cheers, fijal > > Regards > Romain > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From anto.cuni at gmail.com Tue Mar 29 09:31:19 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 29 Mar 2011 09:31:19 +0200 Subject: [pypy-dev] Hacking dropbox Message-ID: <4D918AC7.7070604@gmail.com> Hi, I don't have any news from the dropbox side, but as an aside note, I found a project which is open source and implements the very same technique I used for dropbox, and additionally it also decompiles back to the source code: http://code.google.com/p/pyretic/ ciao, Anto From arigo at tunes.org Tue Mar 29 11:33:52 2011 From: arigo at tunes.org (Armin Rigo) Date: Tue, 29 Mar 2011 11:33:52 +0200 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: References: Message-ID: Hi Andrew, On Mon, Mar 28, 2011 at 7:21 PM, Andrew Brown wrote: > When the optimizer encounters a "pure" function, it must compare the objects > "promote - promote the argument from a variable into a constant". Could this > be an appropriate alternate to the @purefunction solution? Or, I'm guessing, > does it just mean the name bracket_map won't change bindings, but does not > impose a restriction on mutating the dictionary? One point of view on 'promote' is to mean "this variable was red, but now turn it green (i.e. make it constant)". It has no effect on a variable that is already green (= a constant). We have no support for considering that a dict is immutable, so it needs to be done with @purefunction. But to be effective, @purefunction must receive constant arguments; so in one or two places in the source code of PyPy you will find a construction like: x = hint(x, promote=True) # turn x into a constant some_pure_function(x) # call this pure function on x Indeed, Carl Friedrich's blog series explains it nicely, but it should also mention that when the hints described in the blog are applied not to integer but to pointers, they apply only to the pointers themselves, not on the fields of the objects they point to. A bient?t, Armin. From fwierzbicki at gmail.com Wed Mar 30 04:40:10 2011 From: fwierzbicki at gmail.com (fwierzbicki at gmail.com) Date: Tue, 29 Mar 2011 19:40:10 -0700 Subject: [pypy-dev] The JVM backend and Jython Message-ID: Hi all, It was nice meeting up with many of you at PyCon! I've been thinking about the first steps towards collaboration between the Jython project and the PyPy project. It looks like it isn't going to be too long before we are all (CPython, PyPy, IronPython, Jython, etc) working on a single shared repository for all of our standard library .py code. In my ideal world there would come a day when there is also no standalone Java code in the Jython project: that is the shared standard library would contain all of Jython's .py files, and all of the Java would be generated from PyPy and Jython as a standalone project would disappear. It is possible that this is too ambitious, but big goals are more fun, right? In reality even if this where to get going, I imagine it would be a 10+ year plan :) So to my question - just how broken is the JVM backend? Are there workarounds that would allow the Java code to get generated? I ask because I would like to evaluate the generated Java parser as a potential replacement for our current ANTLR based parser. It seems like a nice baby step towards real collaboration since it seems like a relatively easy place to start. Clearly it would need adjustments to actually work for Jython - but I'd be able to look into that part. I don't think I have the time to try to unbreak the translation though... -Frank From anto.cuni at gmail.com Wed Mar 30 09:18:57 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed, 30 Mar 2011 09:18:57 +0200 Subject: [pypy-dev] The JVM backend and Jython In-Reply-To: References: Message-ID: <4D92D961.3070309@gmail.com> Hi Frank, On 30/03/11 04:40, fwierzbicki at gmail.com wrote: > Hi all, > > It was nice meeting up with many of you at PyCon! > > I've been thinking about the first steps towards collaboration between > the Jython project and the PyPy project. It looks like it isn't going > to be too long before we are all (CPython, PyPy, IronPython, Jython, > etc) working on a single shared repository for all of our standard > library .py code. In my ideal world there would come a day when there > is also no standalone Java code in the Jython project: that is the > shared standard library would contain all of Jython's .py files, and > all of the Java would be generated from PyPy and Jython as a > standalone project would disappear. It is possible that this is too > ambitious, but big goals are more fun, right? In reality even if this > where to get going, I imagine it would be a 10+ year plan :) wow, that's definitely a nice (and big) plan :-) > So to my question - just how broken is the JVM backend? Are there > workarounds that would allow the Java code to get generated? "not much broken". Last time I tried, the only broken thing was "virtualrefs", which is something needed for the jit, but that at the moment is not supported at all by ootype and thus it blocks the translation. However, I think that fixing it is probably very easy: there is a branch for this which was started by Ademan: https://bitbucket.org/pypy/pypy/src/ootype-virtualrefs Dan, do you plan to finish the work on it? Else, I can just do it probably. > I ask > because I would like to evaluate the generated Java parser as a > potential replacement for our current ANTLR based parser. It seems > like a nice baby step towards real collaboration since it seems like a > relatively easy place to start. Clearly it would need adjustments to > actually work for Jython - but I'd be able to look into that part. I > don't think I have the time to try to unbreak the translation > though... I have to warn you that at the moment, you cannot invoke any Java code from RPython. Implementing it has been on my todo list for years now :-(, but I never managed to find the time and the motivation to do it. However, for using the PyPy parser inside Jython it should be enough to do the other way around, i.e. call RPython code from Java, which should be possible. ciao, Anto From fwierzbicki at gmail.com Wed Mar 30 19:37:02 2011 From: fwierzbicki at gmail.com (fwierzbicki at gmail.com) Date: Wed, 30 Mar 2011 10:37:02 -0700 Subject: [pypy-dev] The JVM backend and Jython In-Reply-To: <4D92D961.3070309@gmail.com> References: <4D92D961.3070309@gmail.com> Message-ID: On Wed, Mar 30, 2011 at 12:18 AM, Antonio Cuni wrote: > I have to warn you that at the moment, you cannot invoke any Java code from > RPython. ?Implementing it has been on my todo list for years now :-(, but I > never managed to find the time and the motivation to do it. ?However, for > using the PyPy parser inside Jython it should be enough to do the other way > around, i.e. call RPython code from Java, which should be possible. My thoughts here are taking a very primitive step - that is run the JVM translation and look at the generated Java - then see what needs to be modified so that I could use the generated Java parser from Jython. At this stage I would be using PyPy exactly the way I use ANTLR now - as a parser generator. There wouldn't be any need at all for calling into Java code (as far as I can think of). I think if we Jython developers get some experience with PyPY - we might be able to help with the task of calling into Java from PyPy - since we know a bit about that :) -Frank From santagada at gmail.com Wed Mar 30 19:57:30 2011 From: santagada at gmail.com (Leonardo Santagada) Date: Wed, 30 Mar 2011 14:57:30 -0300 Subject: [pypy-dev] The JVM backend and Jython In-Reply-To: References: <4D92D961.3070309@gmail.com> Message-ID: On Wed, Mar 30, 2011 at 2:37 PM, fwierzbicki at gmail.com wrote: > On Wed, Mar 30, 2011 at 12:18 AM, Antonio Cuni wrote: >> I have to warn you that at the moment, you cannot invoke any Java code from >> RPython. ?Implementing it has been on my todo list for years now :-(, but I >> never managed to find the time and the motivation to do it. ?However, for >> using the PyPy parser inside Jython it should be enough to do the other way >> around, i.e. call RPython code from Java, which should be possible. > My thoughts here are taking a very primitive step - that is run the > JVM translation and look at the generated Java - then see what needs > to be modified so that I could use the generated Java parser from > Jython. At this stage I would be using PyPy exactly the way I use > ANTLR now - as a parser generator. There wouldn't be any need at all > for calling into Java code (as far as I can think of). I think if we > Jython developers get some experience with PyPY - we might be able to > help with the task of calling into Java from PyPy - since we know a > bit about that :) IIRC the jvm backend generates java bytecode directly in text form for a java assembler (I forgot the name of it), maybe a step would be to see if there is any way to import the .class back in a java program. -- Leonardo Santagada From anto.cuni at gmail.com Wed Mar 30 20:11:36 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed, 30 Mar 2011 20:11:36 +0200 Subject: [pypy-dev] The JVM backend and Jython In-Reply-To: References: <4D92D961.3070309@gmail.com> Message-ID: <4D937258.6090500@gmail.com> On 30/03/11 19:37, fwierzbicki at gmail.com wrote: > My thoughts here are taking a very primitive step - that is run the > JVM translation and look at the generated Java - then see what needs > to be modified so that I could use the generated Java parser from > Jython. At this stage I would be using PyPy exactly the way I use > ANTLR now - as a parser generator. There wouldn't be any need at all > for calling into Java code (as far as I can think of). yes, I think it makes sense. Actually, as Leonardo says we don't generate java code but assembler which is converted to .class by jasmin. However, it should not change anything. > I think if we > Jython developers get some experience with PyPY - we might be able to > help with the task of calling into Java from PyPy - since we know a > bit about that :) that would be extremely cool :-) Ok, so if Ademan tells me that he's not going to work on the ootype-virtualref branch, I'll try to finish the work so you can start playing with it. ciao, Anto From pjenvey at underboss.org Wed Mar 30 21:59:39 2011 From: pjenvey at underboss.org (Philip Jenvey) Date: Wed, 30 Mar 2011 12:59:39 -0700 Subject: [pypy-dev] The JVM backend and Jython In-Reply-To: <4D92D961.3070309@gmail.com> References: <4D92D961.3070309@gmail.com> Message-ID: On Mar 30, 2011, at 12:18 AM, Antonio Cuni wrote: > >> I ask >> because I would like to evaluate the generated Java parser as a >> potential replacement for our current ANTLR based parser. It seems >> like a nice baby step towards real collaboration since it seems like a >> relatively easy place to start. Clearly it would need adjustments to >> actually work for Jython - but I'd be able to look into that part. I >> don't think I have the time to try to unbreak the translation >> though... > > I have to warn you that at the moment, you cannot invoke any Java code from > RPython. Implementing it has been on my todo list for years now :-(, but I > never managed to find the time and the motivation to do it. However, for > using the PyPy parser inside Jython it should be enough to do the other way > around, i.e. call RPython code from Java, which should be possible. FYI there's an interesting solution on how to call into arbitrary Java code from an invokedynamic enabled language via Atilla Szegedi's somewhat experimental Meta Object Protocol: https://github.com/szegedi/dynalink Basically on Java 7 invokedynamic a dynamic language invocation instruction is something along the lines of: obj.someattr = someobj -> invokedynamic "__setattr__"(Ljava/lang/String;Ljava/lang/Object;I)V; Which might dispatch to a PyObject.__setattr__(String name, PyObject value) method (or easily something completely different in invokedynamic land). The MOP ads another layer in between the invocation and the call site. So as the language implementor you'd use the MOP library to 'relink' your __setattr__ call site to the meta object protocol's more generic version (it only supports a few features but one of them is generic property access). Then you can do the invocation via the MOP, something like: invokedynamic "dyn:setProp:someattr"(Ljava/lang/Object;I)V; The point being that other JVM languages will eventually support the MOP protocol and then you'd get property access, invocation, etc. to those languages for free. More importantly, out of the box the MOP lib implements the protocol for plane old Java objects. If you're already using invokedynamic the library seems simple to hook into and there's basically no call overhead added. I'm not sure this would even be applicable to RPython as it's more static in nature. But it will certainly help in calling Java from regular Python. -- Philip Jenvey From fwierzbicki at gmail.com Thu Mar 31 00:35:42 2011 From: fwierzbicki at gmail.com (fwierzbicki at gmail.com) Date: Wed, 30 Mar 2011 15:35:42 -0700 Subject: [pypy-dev] The JVM backend and Jython In-Reply-To: <4D937258.6090500@gmail.com> References: <4D92D961.3070309@gmail.com> <4D937258.6090500@gmail.com> Message-ID: On Wed, Mar 30, 2011 at 11:11 AM, Antonio Cuni wrote: > On 30/03/11 19:37, fwierzbicki at gmail.com wrote: > >> My thoughts here are taking a very primitive step - that is run the >> JVM translation and look at the generated Java - then see what needs >> to be modified so that I could use the generated Java parser from >> Jython. At this stage I would be using PyPy exactly the way I use >> ANTLR now - as a parser generator. There wouldn't be any need at all >> for calling into Java code (as far as I can think of). > > yes, I think it makes sense. > Actually, as Leonardo says we don't generate java code but assembler which is > converted to .class by jasmin. However, it should not change anything. Assember/.class files shouldn't be a problem. >> I think if we >> Jython developers get some experience with PyPY - we might be able to >> help with the task of calling into Java from PyPy - since we know a >> bit about that :) > > that would be extremely cool :-) > > Ok, so if Ademan tells me that he's not going to work on the ootype-virtualref > branch, I'll try to finish the work so you can start playing with it. Great thanks! -Frank From brownan at gmail.com Thu Mar 31 14:28:48 2011 From: brownan at gmail.com (Andrew Brown) Date: Thu, 31 Mar 2011 08:28:48 -0400 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: References: Message-ID: Thanks for the info! That's all the questions I have, for now at least. Feel free to reply with any more tips if you think of any. I read over your posts you linked, Carl. They were certainly informative and helpful, thanks. I'll keep thinking of ways to improve the performance of my test interpreter, but it's so simple, I don't think there's much more that can be done. The shared attribute maps described by that link don't really apply here. In any case, I'm satisfied with the speed. It's still beaten by a BF to C translator combined with gcc -O2 though, that'd be a tough case to beat. =) -Andrew On Tue, Mar 29, 2011 at 5:33 AM, Armin Rigo wrote: > Hi Andrew, > > On Mon, Mar 28, 2011 at 7:21 PM, Andrew Brown wrote: > > When the optimizer encounters a "pure" function, it must compare the > objects > > "promote - promote the argument from a variable into a constant". Could > this > > be an appropriate alternate to the @purefunction solution? Or, I'm > guessing, > > does it just mean the name bracket_map won't change bindings, but does > not > > impose a restriction on mutating the dictionary? > > One point of view on 'promote' is to mean "this variable was red, but > now turn it green (i.e. make it constant)". It has no effect on a > variable that is already green (= a constant). > > We have no support for considering that a dict is immutable, so it > needs to be done with @purefunction. But to be effective, > @purefunction must receive constant arguments; so in one or two places > in the source code of PyPy you will find a construction like: > > x = hint(x, promote=True) # turn x into a constant > some_pure_function(x) # call this pure function on x > > Indeed, Carl Friedrich's blog series explains it nicely, but it should > also mention that when the hints described in the blog are applied not > to integer but to pointers, they apply only to the pointers > themselves, not on the fields of the objects they point to. > > > A bient?t, > > Armin. > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110331/8c564280/attachment.htm From anto.cuni at gmail.com Thu Mar 31 14:33:40 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu, 31 Mar 2011 14:33:40 +0200 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: References: Message-ID: <4D9474A4.3020909@gmail.com> On 31/03/11 14:28, Andrew Brown wrote: > In any case, I'm satisfied with the speed. It's still beaten by a BF to C > translator combined with gcc -O2 though, that'd be a tough case to beat. =) what happens if you combine the BF to C with gcc -O0 or -O1? Anyway, I think that if you feel like writing a post explaining your experience with using pypy and its jit for writing an interpreter, we could publish it on our blog. I suppose it would be useful/interesting for other people as well. What do the others think? From lac at openend.se Thu Mar 31 14:38:24 2011 From: lac at openend.se (Laura Creighton) Date: Thu, 31 Mar 2011 14:38:24 +0200 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: Message from Antonio Cuni of "Thu, 31 Mar 2011 14:33:40 +0200." <4D9474A4.3020909@gmail.com> References: <4D9474A4.3020909@gmail.com> Message-ID: <201103311238.p2VCcOuk024603@theraft.openend.se> In a message of Thu, 31 Mar 2011 14:33:40 +0200, Antonio Cuni writes: >On 31/03/11 14:28, Andrew Brown wrote: >> In any case, I'm satisfied with the speed. It's still beaten by a BF to > C >> translator combined with gcc -O2 though, that'd be a tough case to beat >. =) > >what happens if you combine the BF to C with gcc -O0 or -O1? > >Anyway, I think that if you feel like writing a post explaining your >experience with using pypy and its jit for writing an interpreter, we cou >ld >publish it on our blog. I suppose it would be useful/interesting for oth >er >people as well. > >What do the others think? I'd look forward to reading it. Laura From brownan at gmail.com Thu Mar 31 14:40:09 2011 From: brownan at gmail.com (Andrew Brown) Date: Thu, 31 Mar 2011 08:40:09 -0400 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: <4D9474A4.3020909@gmail.com> References: <4D9474A4.3020909@gmail.com> Message-ID: Compiling with -O0 is really quick, but the runtime is fairly slow. I haven't tried with -O1. -O2 takes a few seconds to compile, but that plus runtime is still faster than the pypy version with jit, but not by too much (I'm recalling the tests I did with the mandelbrot program specifically). I can get some actual numbers later today. Sure I'll write up a post. This was a lot of fun, and I think it's a great way to teach people how pypy works. -Andrew On Thu, Mar 31, 2011 at 8:33 AM, Antonio Cuni wrote: > On 31/03/11 14:28, Andrew Brown wrote: > > In any case, I'm satisfied with the speed. It's still beaten by a BF to C > > translator combined with gcc -O2 though, that'd be a tough case to beat. > =) > > what happens if you combine the BF to C with gcc -O0 or -O1? > > Anyway, I think that if you feel like writing a post explaining your > experience with using pypy and its jit for writing an interpreter, we could > publish it on our blog. I suppose it would be useful/interesting for other > people as well. > > What do the others think? > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110331/eaa0ee18/attachment.htm From tbaldridge at gmail.com Thu Mar 31 16:19:53 2011 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Thu, 31 Mar 2011 09:19:53 -0500 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: References: <4D9474A4.3020909@gmail.com> Message-ID: > Sure I'll write up a post. This was a lot of fun, and I think it's a great > way to teach people how pypy works. I'd love to read a post on this. Perhaps I'll get a few pointers that I can use in my Clojure-pypy port. Timothy -- ?One of the main causes of the fall of the Roman Empire was that?lacking zero?they had no way to indicate successful termination of their C programs.? (Robert Firth) From brownan at gmail.com Thu Mar 31 17:44:32 2011 From: brownan at gmail.com (Andrew Brown) Date: Thu, 31 Mar 2011 11:44:32 -0400 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: References: Message-ID: Here's three emails that accidentally didn't get sent to the list. On Thu, Mar 31, 2011 at 10:41 AM, Dan Roberts wrote: > Hi Andrew, > Did you ever try your interpreter with the 99 bottles program? I got my > interpreter down faster than the beef interpreter ~4s vs ~15s on mandelbrot, > however even with that speed, both 'bf' and 'beef' trounced my interpreter > by an absurd amount. It seems like it probably was a problem with my code > base, when I first saw you were working on this I meant to ask you to try > 99bottles.bf and see if you had similar problems. I haven't had a chance > to examine where the problem is coming from though. > > Cheers, > Dan > On Thu, Mar 31, 2011 at 11:02 AM, Andrew Brown wrote: > Hi Dan, > Did you mean to send this to the list as well? I only ask because it's easy > to hit "Reply" instead of "Reply All". > > Regardless, I have a 99 bottles program, in the comments it says it's > written by Andrew Paczkowski. I haven't mentioned it just because it runs > absurdly fast: 0.02 seconds or so (compared with 0.2s for running the py > code on cpython), so I didn't consider it a good test. I wanted something > that took a bit longer. > > I just searched for another and found one by Raphael Bois, but that runs in > 0.04 seconds. > > Perhaps you're using a different version of this program that's less > efficient and runs faster? (or maybe this really is just that fast?) > > Also, the mandelbrot program that I included in my repo takes 8.4 seconds > to run on my computer. Not quite the 4 second time you're getting (have you > published your interpreter anywhere? I'd like to look at it) I have a > feeling I've taken this interpreter as far as it will go without doing any > more intelligent inspection of the bf code directly. > > -Andrew > On Thu, Mar 31, 2011 at 11:20 AM, Dan Roberts wrote: > Hey, > Yeah, that's the second or third time I didn't reply to all lately :-/ > And my interpreter is on paste.pocoo.org somewhere, I can paste it again > when I go home today. I suspect there's something wrong with it though, > considering you're getting proper performance on 99bottles, it takes about 3 > minutes here! (On the same system where it wins on mandelbrot by >66% > against 'beef' or bf whichever one is faster) One immediately obvious > difference was your use of the bracket map, which I think is an awesome > idea. I may adopt it, currently I calculate how far backwards/forwards to > travel at "runtime" instead of preprocessing it. I made it pure so that it > would be constant folded by the JIT, but I suppose the 1000 iterations > before the JIT kicks in (per loop) could explain a large performance > difference. I could probably combine both techniques and cache the results > at runtime, by the second run, it'll be a dict lookup, so it'll be jitted > essentially the same, and I won't have to think about parsing to find > matching braces :-) > > If you can think of a good way to bring this discussion back on the mailing > list that'd be fine. > > Cheers, > Dan > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110331/7a51cb94/attachment.htm From dimaqq at gmail.com Thu Mar 31 20:09:36 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Thu, 31 Mar 2011 11:09:36 -0700 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: <4D9474A4.3020909@gmail.com> References: <4D9474A4.3020909@gmail.com> Message-ID: On 31 March 2011 05:33, Antonio Cuni wrote: > On 31/03/11 14:28, Andrew Brown wrote: >> In any case, I'm satisfied with the speed. It's still beaten by a BF to C >> translator combined with gcc -O2 though, that'd be a tough case to beat. =) What if bf code was really really large? bf to c then gcc could take a hit as it might thrash cpu cache, as single pass gcc doesn't know what a given program would actually do at runtime. jit'd rpy would only have 1 hotspot, always in cache, and might be a little smarter too. I suppose it's hard to beat 2-pass (profile driven optimized) compiled c though. > > what happens if you combine the BF to C with gcc -O0 or -O1? > > Anyway, I think that if you feel like writing a post explaining your > experience with using pypy and its jit for writing an interpreter, we could > publish it on our blog. ?I suppose it would be useful/interesting for other > people as well. > > What do the others think? I think it can be a great example. It's very educational ;-) It could go into official docs/howto too. From fijall at gmail.com Thu Mar 31 21:57:20 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 31 Mar 2011 13:57:20 -0600 Subject: [pypy-dev] The JVM backend and Jython In-Reply-To: <4D937258.6090500@gmail.com> References: <4D92D961.3070309@gmail.com> <4D937258.6090500@gmail.com> Message-ID: On Wed, Mar 30, 2011 at 12:11 PM, Antonio Cuni wrote: > On 30/03/11 19:37, fwierzbicki at gmail.com wrote: > >> My thoughts here are taking a very primitive step - that is run the >> JVM translation and look at the generated Java - then see what needs >> to be modified so that I could use the generated Java parser from >> Jython. At this stage I would be using PyPy exactly the way I use >> ANTLR now - as a parser generator. There wouldn't be any need at all >> for calling into Java code (as far as I can think of). > > yes, I think it makes sense. > Actually, as Leonardo says we don't generate java code but assembler which is > converted to .class by jasmin. However, it should not change anything. > >> I think if we >> Jython developers get some experience with PyPY - we might be able to >> help with the task of calling into Java from PyPy - since we know a >> bit about that :) > > that would be extremely cool :-) > > Ok, so if Ademan tells me that he's not going to work on the ootype-virtualref > branch, I'll try to finish the work so you can start playing with it. Note to frank: this is kind of cool but only needed for the JIT, otherwise it's a normal reference. > > ciao, > Anto > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From brownan at gmail.com Thu Mar 31 22:05:49 2011 From: brownan at gmail.com (Andrew Brown) Date: Thu, 31 Mar 2011 16:05:49 -0400 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: References: <4D9474A4.3020909@gmail.com> Message-ID: On Thu, Mar 31, 2011 at 2:09 PM, Dima Tisnek wrote: > What if bf code was really really large? > I've only been testing with the examples at hand included in my repo. The mandelbrot and towers of hanoi examples are pretty big though. If you can find some larger examples, I'd like to try them. I think it can be a great example. It's very educational ;-) > It could go into official docs/howto too. > Awesome! I'm working on writing up everything, it's turning out to be pretty long. I'm assuming no prior PyPy knowledge in the readers though =) Here are a few numbers from tests I just did. python double-interpreted: > 78m (did not finish) pypy-c (with jit) double-interpreted: 41m 34.528s translated interpreter no jit: 45s translated interpreter jit: 7.5s translated direct to C, gcc -O0 translate: 0.2s compile: 0.4s run: 18.5s translated direct to C, gcc -O1 translate: 0.2s compile: 0.85s run: 1.28s translated direct to C, gcc -O2 translate: 0.2s compile: 2.0s run: 1.34s These were all running the mandelbrot program. -Andrew -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110331/66012db5/attachment-0001.htm