From anto.cuni at gmail.com Fri Apr 1 00:00:53 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Fri, 01 Apr 2011 00:00:53 +0200 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: References: <4D9474A4.3020909@gmail.com> Message-ID: <4D94F995.5020509@gmail.com> On 31/03/11 22:05, Andrew Brown wrote: > python double-interpreted: > 78m (did not finish) > pypy-c (with jit) double-interpreted: 41m 34.528s this is interesting. We are beating cpython by more than 2x even in a "worst case" scenario, because interpreters in theory are not a very good target for tracing JITs. However, it's not the first time that we experience this, so it might be that this interpreter/tracing JIT thing is just a legend :-) > translated interpreter no jit: 45s > translated interpreter jit: 7.5s > translated direct to C, gcc -O0 > translate: 0.2s > compile: 0.4s > run: 18.5s > translated direct to C, gcc -O1 > translate: 0.2s > compile: 0.85s > run: 1.28s > translated direct to C, gcc -O2 > translate: 0.2s > compile: 2.0s > run: 1.34s these are cool as well. We are 3x faster than gcc -O0 and ~3x slower than -O1 and -O2. Pretty good, I'd say :-) ciao, anto From anto.cuni at gmail.com Fri Apr 1 00:02:52 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Fri, 01 Apr 2011 00:02:52 +0200 Subject: [pypy-dev] The JVM backend and Jython In-Reply-To: References: <4D92D961.3070309@gmail.com> <4D937258.6090500@gmail.com> Message-ID: <4D94FA0C.6080206@gmail.com> On 31/03/11 21:57, Maciej Fijalkowski wrote: >> Ok, so if Ademan tells me that he's not going to work on the ootype-virtualref >> branch, I'll try to finish the work so you can start playing with it. > > Note to frank: this is kind of cool but only needed for the JIT, > otherwise it's a normal reference. well, no. Virtualrefs were introduced for the JIT, but they also need to be supported by normal backends. This is why translation is broken at the moment. It is true that the implementation is straightforward, though (I suppose this is what you meant originally :-)) ciao, Anto From alex.gaynor at gmail.com Fri Apr 1 00:29:57 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Thu, 31 Mar 2011 18:29:57 -0400 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: <4D94F995.5020509@gmail.com> References: <4D9474A4.3020909@gmail.com> <4D94F995.5020509@gmail.com> Message-ID: On Thu, Mar 31, 2011 at 6:00 PM, Antonio Cuni wrote: > On 31/03/11 22:05, Andrew Brown wrote: > > > python double-interpreted: > 78m (did not finish) > > pypy-c (with jit) double-interpreted: 41m 34.528s > > this is interesting. We are beating cpython by more than 2x even in a > "worst > case" scenario, because interpreters in theory are not a very good target > for > tracing JITs. > However, it's not the first time that we experience this, so it might be > that > this interpreter/tracing JIT thing is just a legend :-) > > Well the issue with tracing an interpreter is the large number of paths, a brainfuck interpreter has relatively few paths compared to something like a Python VM. > > translated interpreter no jit: 45s > > translated interpreter jit: 7.5s > > translated direct to C, gcc -O0 > > translate: 0.2s > > compile: 0.4s > > run: 18.5s > > translated direct to C, gcc -O1 > > translate: 0.2s > > compile: 0.85s > > run: 1.28s > > translated direct to C, gcc -O2 > > translate: 0.2s > > compile: 2.0s > > run: 1.34s > > these are cool as well. We are 3x faster than gcc -O0 and ~3x slower than > -O1 > and -O2. Pretty good, I'd say :-) > > ciao, > anto > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110331/b499f26e/attachment.htm From szoth at ubertechnique.com Fri Apr 1 02:27:14 2011 From: szoth at ubertechnique.com (Seth de l'Isle) Date: Thu, 31 Mar 2011 16:27:14 -0800 Subject: [pypy-dev] [PATCH] fix executing a file under sandbox by implementing fstat Message-ID: I ran into a problem trying to follow the instructions to run a python file under the sandbox version of pypy-c. See the console log: http://paste.pocoo.org/show/363348 I was following the instructions here: http://codespeak.net/pypy/dist/pypy/doc/sandbox.html I got some help from arigato, ronny and antocuni on IRC (see log below) and they gave me the courage to hack on sandlib.py a little to see if I could get things working. http://www.tismer.com/pypy/irc-logs/pypy/%23pypy.log.20110331 The following patch changes the code so that it tracks the virtual file system nodes that correspond to each virtual file descriptor so that the node.stat() function can be used for fstat the same way it is used for stat. Thanks! diff -r 601862ed288e pypy/translator/sandbox/sandlib.py --- a/pypy/translator/sandbox/sandlib.py Mon Mar 28 13:12:49 2011 +0200 +++ b/pypy/translator/sandbox/sandlib.py Thu Mar 31 16:13:41 2011 -0800 @@ -391,6 +391,7 @@ super(VirtualizedSandboxedProc, self).__init__(*args, **kwds) self.virtual_root = self.build_virtual_root() self.open_fds = {} # {virtual_fd: real_file_object} + self.fd_to_node = {} def build_virtual_root(self): raise NotImplementedError("must be overridden") @@ -425,10 +426,17 @@ def do_ll_os__ll_os_stat(self, vpathname): node = self.get_node(vpathname) return node.stat() + do_ll_os__ll_os_stat.resulttype = s_StatResult do_ll_os__ll_os_lstat = do_ll_os__ll_os_stat + def do_ll_os__ll_os_fstat(self, fd): + node = self.fd_to_node[fd] + return node.stat() + + do_ll_os__ll_os_fstat.resulttype = s_StatResult + def do_ll_os__ll_os_isatty(self, fd): return self.virtual_console_isatty and fd in (0, 1, 2) @@ -452,11 +460,14 @@ raise OSError(errno.EPERM, "write access denied") # all other flags are ignored f = node.open() - return self.allocate_fd(f) + fd = self.allocate_fd(f) + self.fd_to_node[fd] = node + return fd def do_ll_os__ll_os_close(self, fd): f = self.get_file(fd) del self.open_fds[fd] + del self.fd_to_node[fd] f.close() def do_ll_os__ll_os_read(self, fd, size): From fijall at gmail.com Fri Apr 1 06:26:46 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 31 Mar 2011 22:26:46 -0600 Subject: [pypy-dev] The JVM backend and Jython In-Reply-To: <4D94FA0C.6080206@gmail.com> References: <4D92D961.3070309@gmail.com> <4D937258.6090500@gmail.com> <4D94FA0C.6080206@gmail.com> Message-ID: On Thu, Mar 31, 2011 at 4:02 PM, Antonio Cuni wrote: > On 31/03/11 21:57, Maciej Fijalkowski wrote: > >>> Ok, so if Ademan tells me that he's not going to work on the ootype-virtualref >>> branch, I'll try to finish the work so you can start playing with it. >> >> Note to frank: this is kind of cool but only needed for the JIT, >> otherwise it's a normal reference. > > well, no. Virtualrefs were introduced for the JIT, but they also need to be > supported by normal backends. ?This is why translation is broken at the moment. > > It is true that the implementation is straightforward, though (I suppose this > is what you meant originally :-)) Sure. I was mostly saying "the complex part of the implementation for ootype can be ommited if we skip the JIT part". > > ciao, > Anto > From stefan_ml at behnel.de Fri Apr 1 07:59:36 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 01 Apr 2011 07:59:36 +0200 Subject: [pypy-dev] The JVM backend and Jython In-Reply-To: References: Message-ID: fwierzbicki at gmail.com, 30.03.2011 04:40: > I've been thinking about the first steps towards collaboration between > the Jython project and the PyPy project. It looks like it isn't going > to be too long before we are all (CPython, PyPy, IronPython, Jython, > etc) working on a single shared repository for all of our standard > library .py code. On a somewhat related note, the Cython project is pushing towards reimplementing parts of CPython's stdlib C modules in Cython. That would make it easier for other projects to use the implementation in one way or another, rather than having to reimplement and maintain it separately by following C code. http://thread.gmane.org/gmane.comp.python.devel/122273/focus=122716 The advantage for other-than-CPython-Pythons obviously depends on the module. If it's just implemented in C for performance reasons (e.g. itertools etc.), it would likely end up as a Python module with additional static typing, which would make it easy to adapt. If it's using lots of stuff from libc and C I/O, or even from external libraries, the code itself would obviously be less useful, although it would likely still be easier to port changes/fixes. Stefan From anto.cuni at gmail.com Fri Apr 1 11:23:52 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Fri, 01 Apr 2011 11:23:52 +0200 Subject: [pypy-dev] [PATCH] fix executing a file under sandbox by implementing fstat In-Reply-To: References: Message-ID: <4D9599A8.8080106@gmail.com> Hello Seth, thank you for your patch! On 01/04/11 02:27, Seth de l'Isle wrote: [cut] > The following patch changes the code so that it tracks the virtual > file system nodes that correspond to each virtual file descriptor so > that the node.stat() function can > be used for fstat the same way it is used for stat. could you please write the corresponding test in test_sandlib.py please? > + def do_ll_os__ll_os_fstat(self, fd): > + node = self.fd_to_node[fd] > + return node.stat() Also, you probably need to handle the case in which we call fstat with a fd which doesn't exist (and write the corresponding test :-)) ciao, Anto From lac at openend.se Sat Apr 2 21:57:15 2011 From: lac at openend.se (Laura Creighton) Date: Sat, 2 Apr 2011 21:57:15 +0200 Subject: [pypy-dev] Europython reservations Message-ID: <201104021957.p32JvFe0007507@theraft.openend.se> Jacob and I just arranged to be staying at Residence Michelangiolo June 19-26 . see: http://www.tripadvisor.com/Hotel_Review-g187895-d581477-Reviews-Residence_Michelangiolo-Florence_Tuscany.html among others for review. At 980 Euros for the whole time, it seems to be substantially cheaper that the conference hotel, and not that far a walk. Other differences -- no included breakfast, but pantry -- refrigerator, counter, and clean up space -- in your room. For people who wanted to make their own breakfast anyway, this is a win. There may be a cook space too. There is internet by wire in each room but no wifi. On the 5th, we should be on a ferry to get us to kayaking vacation in Corsica. see: http://ucpa.se/Aktivitetsprogram-havskajaksafari-Korsika (in Swedish, but maybe it will help you to find the same program from ucpa described in whatever language you prefer.) This leaves the 26-5th to figure out. Vacation in Tuscany is one idea. Or to extend the sprint because 2 days is not enough is another. If so, maybe we could hold it at this place? Seems spacious enough. I thought it would be fun to fill Residence Michelangiolo with friends, at least for the 19-26 so posted this here. Laura From lac at openend.se Mon Apr 4 14:34:44 2011 From: lac at openend.se (Laura Creighton) Date: Mon, 4 Apr 2011 14:34:44 +0200 Subject: Post Easter PyPy Sprint, Göteborg Sweden April 25 -- May 1. Message-ID: <201104041234.p34CYia5004393@theraft.openend.se> he next PyPy sprint will be in Gothenburg, Sweden. It is a public sprint, very suitable for newcomers. We'll focus on making the 1.5 release (if it hasn't already happened) and whatever interests the Sprint attendees. Topics and goals ---------------- The main goal is to polish and release PyPy 1.5, supporting Python 2.7 as well as the last few months' improvements in the JIT (provided that it hasn't already happened). Other topics: - Going over our documentation, and classifying our docs in terms of mouldiness. Deciding what needs writing, and maybe writing it. - Helping people get their code running with PyPy - maybe work on EuroPython Training, and talks - Summer of Code preparation - speed.pypy.org - any other programming task is welcome too -- e.g. tweaking the Python or JavaScript interpreter, Stackless support, and so on. Location -------- The sprint will be held in the apartment of Laura Creighton and Jacob Hall?n which is at G?tabergsgatan 22 in Gothenburg, Sweden. Here is a map_. This is in central Gothenburg. It is between the tram_ stops of Vasaplatsen and Valand, (a distance of 4 blocks) where many lines call -- the 2, 3, 4, 5, 7, 10 and 13. .. _tram: http://www.vasttrafik.se/en/ .. _map: http://bit.ly/grRuQe Probably cheapest and not too far away is to book accomodation at `SGS Veckobostader`_. The `Elite Park Avenyn Hotel`_ is a luxury hotel just a few blocks away. There are scores of hotels a short walk away from the sprint location, suitable for every budget, desire for luxury, and desire for the unusual. You could, for instance, stay on a `boat`_. Options are oo numerous to go into here. Just ask in the mailing list or on the blog. .. _`SGS Veckobostader`: http://www.sgsveckobostader.se/en .. _`Elite Park Avenyn Hotel`: http://www.elite.se/hotell/goteborg/park/ .. _`boat`: http://www.liseberg.se/en/home/Accommodation/Hotel/Hotel-Barken-Viki ng/ Hours will be from 10:00 until people have had enough. It's a good idea to arrive a day before the sprint starts and leave a day later. In the middle of the sprint there usually is a break day and it's usually ok to take half-days off if you feel like it. Good to Know ------------ Sweden is not part of the Euro zone. One SEK (krona in singular, kronor in plural) is roughly 1/10th of a Euro (9.36 SEK to 1 Euro). The venue is central in Gothenburg. There is a large selection of places to get food nearby, from edible-and-cheap to outstanding. We often cook meals together, so let us know if you have any food allergies, dislikes, or special requirements. Sweden uses the same kind of plugs as Germany. 230V AC. The Sprint will be held the week following Easter. This means, as always, that Gothcon_ will be taking place the weekend before (Easter weekend). Gothcon, now in its 35 year, is the largest European game players conference. Some of you may be interested in arriving early for the board games. The conference site is only in Swedish, alas. You don't need to register in advance unless you are planning to host a tournament, (and it's too late for that anyway). .. _Gothcon: http://www.gothcon.se/ Getting Here ------------ If are coming train, you will arrive at the `Central Station`_. It is about 12 blocks to the site from there, or you can take a tram_. There are two airports which are local to G?teborg, `Landvetter`_ (the main one) and `Gothenburg City Airport`_ (where some budget airlines fly). If you arrive at `Landvetter`_ the airport bus stops right downtown at `Elite Park Avenyn Hotel`_ which is the second stop, 4 blocks from the Sprint site, as well as the end of the line, which is the `Central Station`_. If you arrive at `Gothenburg City Airport`_ take the bus to the end of the line. You will be at the `Central Station`_. You can also arrive by ferry_, from either Kiel in Germany or Frederikshavn in Denmark. .. _`Central Station`: http://bit.ly/fON43p .. _`Landvetter`: http://swedavia.se/en/Goteborg/Traveller-information/Traffic-i nformation/ .. _`Gothenburg City Airport`: http://www.goteborgairport.se/eng.asp .. _ferry: http://www.stenaline.nl/en/ferry/ Who's Coming? -------------- If you'd like to come, please let us know when you will be arriving and leaving, as well as letting us know your interests We'll keep a list of `people`_ which we'll update (which you can do so yourself if you have bitbucket pypy commit rights). .. _`people`: https://bitbucket.org/pypy/extradoc/src/tip/sprintinfo/gothenburg- 2011/people.txt From brownan at gmail.com Mon Apr 4 16:12:33 2011 From: brownan at gmail.com (Andrew Brown) Date: Mon, 4 Apr 2011 10:12:33 -0400 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: References: <4D9474A4.3020909@gmail.com> <4D94F995.5020509@gmail.com> Message-ID: Submitted for everyone's approval, I've written a draft of a pypy tutorial going over everything I learned in writing this example interpreter. https://bitbucket.org/brownan/pypy-tutorial/src See the main document at tutorial.rst I'd love some feedback on it. I've made an effort to keep things accurate yet simple, but if there are any inaccuracies, let me know. Or fork the repo and make the correction yourself =) -Andrew On Thu, Mar 31, 2011 at 6:29 PM, Alex Gaynor wrote: > > > On Thu, Mar 31, 2011 at 6:00 PM, Antonio Cuni wrote: > >> On 31/03/11 22:05, Andrew Brown wrote: >> >> > python double-interpreted: > 78m (did not finish) >> > pypy-c (with jit) double-interpreted: 41m 34.528s >> >> this is interesting. We are beating cpython by more than 2x even in a >> "worst >> case" scenario, because interpreters in theory are not a very good target >> for >> tracing JITs. >> However, it's not the first time that we experience this, so it might be >> that >> this interpreter/tracing JIT thing is just a legend :-) >> >> > Well the issue with tracing an interpreter is the large number of paths, a > brainfuck interpreter has relatively few paths compared to something like a > Python VM. > > > > >> > translated interpreter no jit: 45s >> > translated interpreter jit: 7.5s >> > translated direct to C, gcc -O0 >> > translate: 0.2s >> > compile: 0.4s >> > run: 18.5s >> > translated direct to C, gcc -O1 >> > translate: 0.2s >> > compile: 0.85s >> > run: 1.28s >> > translated direct to C, gcc -O2 >> > translate: 0.2s >> > compile: 2.0s >> > run: 1.34s >> >> these are cool as well. We are 3x faster than gcc -O0 and ~3x slower than >> -O1 >> and -O2. Pretty good, I'd say :-) >> >> ciao, >> anto >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev >> > > Alex > > -- > "I disapprove of what you say, but I will defend to the death your right to > say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110404/c139e75e/attachment.htm From cfbolz at gmx.de Mon Apr 4 17:22:01 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 04 Apr 2011 17:22:01 +0200 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: References: <4D9474A4.3020909@gmail.com> <4D94F995.5020509@gmail.com> Message-ID: <4D99E219.3050308@gmx.de> On 04/04/2011 04:12 PM, Andrew Brown wrote: > Submitted for everyone's approval, I've written a draft of a pypy > tutorial going over everything I learned in writing this example > interpreter. > > https://bitbucket.org/brownan/pypy-tutorial/src > > See the main document at tutorial.rst > > I'd > love some feedback on it. I've made an effort to keep things accurate > yet simple, but if there are any inaccuracies, let me know. Or fork the > repo and make the correction yourself =) Looks very nice! Would you be up to making a guest post out of this on the PyPy blog? Carl Friedrich From arigo at tunes.org Mon Apr 4 17:16:55 2011 From: arigo at tunes.org (Armin Rigo) Date: Mon, 4 Apr 2011 17:16:55 +0200 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: References: <4D9474A4.3020909@gmail.com> <4D94F995.5020509@gmail.com> Message-ID: Hi Andrew, On Mon, Apr 4, 2011 at 4:12 PM, Andrew Brown wrote: > Submitted for everyone's approval, I've written a draft of a pypy tutorial > going over everything I learned in writing this example interpreter. > https://bitbucket.org/brownan/pypy-tutorial/src Excellent and, as far as I can tell, very clear too! A bient?t, Armin. From brownan at gmail.com Mon Apr 4 17:43:07 2011 From: brownan at gmail.com (Andrew Brown) Date: Mon, 4 Apr 2011 11:43:07 -0400 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: <4D99E219.3050308@gmx.de> References: <4D9474A4.3020909@gmail.com> <4D94F995.5020509@gmail.com> <4D99E219.3050308@gmx.de> Message-ID: On Mon, Apr 4, 2011 at 11:22 AM, Carl Friedrich Bolz wrote: > > Looks very nice! Would you be up to making a guest post out of this on > the PyPy blog? > > Sure! What needs to be done to turn it into a blog post and get it posted? I assume there are format considerations, but I'm also open to any content suggestions and feedback before it "goes live". -Andrew -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110404/2fc9b38a/attachment.htm From cfbolz at gmx.de Mon Apr 4 18:17:17 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 04 Apr 2011 18:17:17 +0200 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: References: <4D9474A4.3020909@gmail.com> <4D94F995.5020509@gmail.com> <4D99E219.3050308@gmx.de> Message-ID: <4D99EF0D.1000507@gmx.de> On 04/04/2011 05:43 PM, Andrew Brown wrote: > On Mon, Apr 4, 2011 at 11:22 AM, Carl Friedrich Bolz > wrote: > > > Looks very nice! Would you be up to making a guest post out of this on > the PyPy blog? > > Sure! What needs to be done to turn it into a blog post and get it > posted? I assume there are format considerations, but I'm also open to > any content suggestions and feedback before it "goes live". I looked again, added two places that could use small fixes. And I updated two links, see my merge request. Apart from that, the blog post would not need many changes. It would need an introductionary line like: "This is a guest post by Andrew Brown. It's a tutorial for how to write an interpreter with PyPy, generating a JIT. It is suitable for beginners and assumes very little knowledge of PyPy." Then we should link to the repo, and replace all file links with links to bitbucket. I can do all that, and post it (tomorrow), if you are fine with that. Carl Friedrich From anto.cuni at gmail.com Mon Apr 4 18:34:45 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Mon, 04 Apr 2011 18:34:45 +0200 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: References: <4D9474A4.3020909@gmail.com> <4D94F995.5020509@gmail.com> <4D99E219.3050308@gmx.de> Message-ID: <4D99F325.5050605@gmail.com> On 04/04/11 17:43, Andrew Brown wrote: > Sure! What needs to be done to turn it into a blog post and get it posted? I > assume there are format considerations, but I'm also open to any content > suggestions and feedback before it "goes live". Hello Andrew, thanks for the tutorial, it's really well written and easy to read. Two notes: 1) do you know about the existence of rlib.streamio? It's is part of the "RPython standard library" and it allows you to read/write files in a higher level way than file descriptors 2) Maybe the tutorial is a bit too long to fit in just one post; what about splitting it into two parts? (e.g., one until "Adding JIT" and one after). ciao, Anto From brownan at gmail.com Mon Apr 4 18:30:37 2011 From: brownan at gmail.com (Andrew Brown) Date: Mon, 4 Apr 2011 12:30:37 -0400 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: <4D99EF0D.1000507@gmx.de> References: <4D9474A4.3020909@gmail.com> <4D94F995.5020509@gmail.com> <4D99E219.3050308@gmx.de> <4D99EF0D.1000507@gmx.de> Message-ID: Thanks for the feedback. I'll clarify those parts, and I have a few touch-ups of my own. Also, I think I forgot to add my name =) I'm fine with you posting it as you described. An into line like that was just what I had in mind. I'd wait until tomorrow though to see if any other feedback surfaces. On Mon, Apr 4, 2011 at 12:17 PM, Carl Friedrich Bolz wrote: > On 04/04/2011 05:43 PM, Andrew Brown wrote: > >> On Mon, Apr 4, 2011 at 11:22 AM, Carl Friedrich Bolz > > wrote: >> >> >> Looks very nice! Would you be up to making a guest post out of this on >> the PyPy blog? >> >> Sure! What needs to be done to turn it into a blog post and get it >> posted? I assume there are format considerations, but I'm also open to >> any content suggestions and feedback before it "goes live". >> > > I looked again, added two places that could use small fixes. And I updated > two links, see my merge request. Apart from that, the blog post would not > need many changes. It would need an introductionary line like: > > "This is a guest post by Andrew Brown. It's a tutorial for how to write an > interpreter with PyPy, generating a JIT. It is suitable for beginners and > assumes very little knowledge of PyPy." > > Then we should link to the repo, and replace all file links with links to > bitbucket. I can do all that, and post it (tomorrow), if you are fine with > that. > > Carl Friedrich > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110404/7e47f90e/attachment-0001.htm From brownan at gmail.com Mon Apr 4 19:46:13 2011 From: brownan at gmail.com (Andrew Brown) Date: Mon, 4 Apr 2011 13:46:13 -0400 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: <4D99F325.5050605@gmail.com> References: <4D9474A4.3020909@gmail.com> <4D94F995.5020509@gmail.com> <4D99E219.3050308@gmx.de> <4D99F325.5050605@gmail.com> Message-ID: On Mon, Apr 4, 2011 at 12:34 PM, Antonio Cuni wrote: > 1) do you know about the existence of rlib.streamio? It's is part of the > "RPython standard library" and it allows you to read/write files in a > higher > level way than file descriptors > > No, I didn't. That's good to know. I don't think it's worth updating the examples though, so unless you disagree, I'll just add a note about this module's existence. > 2) Maybe the tutorial is a bit too long to fit in just one post; what about > splitting it into two parts? (e.g., one until "Adding JIT" and one after). > > Yes, it is quite long. Carl, feel free to break it up as necessary when you post it. Breaking it up at the "Adding JIT" section seems ideal, since both parts are useful on their own. -Andrew -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110404/0f5ba250/attachment.htm From anto.cuni at gmail.com Mon Apr 4 20:12:57 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Mon, 04 Apr 2011 20:12:57 +0200 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: References: <4D9474A4.3020909@gmail.com> <4D94F995.5020509@gmail.com> <4D99E219.3050308@gmx.de> <4D99F325.5050605@gmail.com> Message-ID: <4D9A0A29.1000603@gmail.com> On 04/04/11 19:46, Andrew Brown wrote: > 1) do you know about the existence of rlib.streamio? It's is part of the > "RPython standard library" and it allows you to read/write files in a higher > level way than file descriptors > > No, I didn't. That's good to know. I don't think it's worth updating the > examples though, so unless you disagree, I'll just add a note about this > module's existence. sure, I think that for this example, using fd is fine. Btw, in case you want to do more with pypy, having a look to rlib might be a good idea, there is useful stuff there :) ciao, Anto From brownan at gmail.com Mon Apr 4 22:28:51 2011 From: brownan at gmail.com (Andrew Brown) Date: Mon, 4 Apr 2011 16:28:51 -0400 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: <4D9A0A29.1000603@gmail.com> References: <4D9474A4.3020909@gmail.com> <4D94F995.5020509@gmail.com> <4D99E219.3050308@gmx.de> <4D99F325.5050605@gmail.com> <4D9A0A29.1000603@gmail.com> Message-ID: On Mon, Apr 4, 2011 at 2:12 PM, Antonio Cuni wrote: > sure, I think that for this example, using fd is fine. > > Btw, in case you want to do more with pypy, having a look to rlib might be > a > good idea, there is useful stuff there :) > > Definitely. In any case, I've made some changes, re-worded some things. Carl, I've addressed your suggestions, let me know what you think. I also re-worded a few things in the "Adding JIT" section to make it flow a bit better assuming it will be split up. It may still need some editing though. -Andrew -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110404/a728166d/attachment.htm From fuzzyman at gmail.com Tue Apr 5 02:22:26 2011 From: fuzzyman at gmail.com (Michael Foord) Date: Tue, 5 Apr 2011 01:22:26 +0100 Subject: [pypy-dev] A Little Bit of Python Episode 17 Message-ID: An interview with Armin, recorded at the Leysin sprint: http://advocacy.python.org/podcasts/littlebit/2011-04-04.mp3 It turned out pretty well I think. All the best, Michael Foord -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110405/9a9bab0e/attachment.htm From igouy2 at yahoo.com Tue Apr 5 02:23:36 2011 From: igouy2 at yahoo.com (Isaac Gouy) Date: Mon, 4 Apr 2011 17:23:36 -0700 (PDT) Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? Message-ID: <602617.13125.qm@web65613.mail.ac4.yahoo.com> Hi A simple yes / no question. Do you want PyPy to be shown in the benchmarks game or not? Please consider the question amongst yourselves and then let me know. Of course, I'll make up my own mind but at least I'll be able to take your wishes into account. best wishes, Isaac From dimaqq at gmail.com Tue Apr 5 02:46:29 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Mon, 4 Apr 2011 17:46:29 -0700 Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: <602617.13125.qm@web65613.mail.ac4.yahoo.com> References: <602617.13125.qm@web65613.mail.ac4.yahoo.com> Message-ID: On 4 April 2011 17:23, Isaac Gouy wrote: > Hi > > A simple yes / no question. > > Do you want PyPy to be shown in the benchmarks game or not? > > Please consider the question amongst yourselves and then let me know. > > Of course, I'll make up my own mind but at least I'll be able to take your wishes into account. > > best wishes, Isaac > > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > I think it ought to be included, although I have big reservation about some of the benchmarks. Some work and/or discussion on benchmarks would be in order. my 2c. d. From lstask at gmail.com Tue Apr 5 07:25:47 2011 From: lstask at gmail.com (Liam Staskawicz) Date: Mon, 4 Apr 2011 22:25:47 -0700 Subject: [pypy-dev] minimum system requirements, build configuration Message-ID: <3B1535B65AE041F0B53FF5F13BF8178D@gmail.com> Hi - I've been looking at pypy as a possible application runtime in an ARM9 Linux based system. I understand that the in-progress ARM jit backend targets only ARMv7, but I'm still interested in characterizing a translated pypy-c interpreter on this system with regard to CPU and memory usage. Of course I look forward to possible ARMv5 support for the jit in the future, but if I've understood correctly, fully interpreted mode should be supported - please let me know if this is not correct! To that end, are there minimum recommended system requirements for pypy, specifically in terms of memory? As a reference, something like the Mono 'Small Footprint' wiki page applied to pypy would be what I'm looking for: http://www.mono-project.com/Small_footprint On a related note, most of the build config info I've come across seems to revolve around a scratchbox2 build environment targeting maemo - indeed it seems to be the main other 'platform' option in the pypy build script, other than 'host'. Is there any particularly tight coupling between maemo and pypy, or should I hope to be able to set up a generic sbox2 environment for my target and come away with a working build? Are there any other docs or config snippets that detail how to set up a generic cross compile environment for pypy? Thanks for any tips or relevant info! Liam -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110404/66233408/attachment.htm From william.leslie.ttg at gmail.com Tue Apr 5 07:45:21 2011 From: william.leslie.ttg at gmail.com (William ML Leslie) Date: Tue, 5 Apr 2011 15:45:21 +1000 Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: <602617.13125.qm@web65613.mail.ac4.yahoo.com> References: <602617.13125.qm@web65613.mail.ac4.yahoo.com> Message-ID: On 5 April 2011 10:23, Isaac Gouy wrote: > Hi > > A simple yes / no question. > > Do you want PyPy to be shown in the benchmarks game or not? > > Please consider the question amongst yourselves and then let me know. There seems to have been general confusion here about what the implementations of these benchmarks are supposed to represent. Are they to be representative of idiomatic code, or optimised for a particular implementation? Or something else entirely? Since pypy have generally tried to optimise for and encourage idiomatic python usage, those benchmark implementations that go to great length to use confusing and non-standard performance hacks represent neither the performance of real-world code, nor what one can do with a specific implementation. If the answer is that they are going to be tuned to a particular implementation and that implementation is not going to be ours, we probably *could* live with that: realistically, the sort of code people are applying pypy to occasionally contains performance hacks that are no longer relevant and possibly detrimental. But it does seem to change the meaning of the benchmark, and it would be useful to get some authoritative clarification on this before we consider it. -- William Leslie From arigo at tunes.org Tue Apr 5 09:59:29 2011 From: arigo at tunes.org (Armin Rigo) Date: Tue, 5 Apr 2011 09:59:29 +0200 Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: <602617.13125.qm@web65613.mail.ac4.yahoo.com> References: <602617.13125.qm@web65613.mail.ac4.yahoo.com> Message-ID: Hi, On Tue, Apr 5, 2011 at 2:23 AM, Isaac Gouy wrote: > A simple yes / no question. > Do you want PyPy to be shown in the benchmarks game or not? Sorry for missing general knowledge of what you mean by "benchmarks game". According to Google, it is probably what I know as the language shootout (shootout.alioth.debian.org). Is it the case? Assuming it is, then my position is pretty much the same as William Leslie: I have little interest if the benchmarks that PyPy runs with are written in completely non-idiomatic ways, super hand-optimized for CPython, for the reasons explained in detail by William. It would be interesting, however, if you accept versions rewritten in "just plain Python". A bient?t, Armin. From cfbolz at gmx.de Tue Apr 5 14:49:38 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Tue, 05 Apr 2011 14:49:38 +0200 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: References: <4D9474A4.3020909@gmail.com> <4D94F995.5020509@gmail.com> <4D99E219.3050308@gmx.de> <4D99F325.5050605@gmail.com> <4D9A0A29.1000603@gmail.com> Message-ID: <4D9B0FE2.9020003@gmx.de> On 04/04/2011 10:28 PM, Andrew Brown wrote: > On Mon, Apr 4, 2011 at 2:12 PM, Antonio Cuni > wrote: > > sure, I think that for this example, using fd is fine. > > Btw, in case you want to do more with pypy, having a look to rlib > might be a > good idea, there is useful stuff there :) > > Definitely. > > In any case, I've made some changes, re-worded some things. Carl, I've > addressed your suggestions, let me know what you think. > > I also re-worded a few things in the "Adding JIT" section to make it > flow a bit better assuming it will be split up. It may still need some > editing though. Looked good, I just went ahead and posted the first part: http://morepypy.blogspot.com/2011/04/tutorial-writing-interpreter-with-pypy.html Will do the second part tomorrow. Thanks a lot for the tutorial, I think it's really great. Carl Friedrich From renesd at gmail.com Tue Apr 5 14:54:32 2011 From: renesd at gmail.com (=?ISO-8859-1?Q?Ren=E9_Dudfield?=) Date: Tue, 5 Apr 2011 13:54:32 +0100 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: <4D9B0FE2.9020003@gmx.de> References: <4D9474A4.3020909@gmail.com> <4D94F995.5020509@gmail.com> <4D99E219.3050308@gmx.de> <4D99F325.5050605@gmail.com> <4D9A0A29.1000603@gmail.com> <4D9B0FE2.9020003@gmx.de> Message-ID: Nice one! hehe, I like how you managed to avoid the first letter of the second word of the language ;) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110405/b9c9a055/attachment.htm From brownan at gmail.com Tue Apr 5 15:40:17 2011 From: brownan at gmail.com (Andrew Brown) Date: Tue, 5 Apr 2011 09:40:17 -0400 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: References: <4D9474A4.3020909@gmail.com> <4D94F995.5020509@gmail.com> <4D99E219.3050308@gmx.de> <4D99F325.5050605@gmail.com> <4D9A0A29.1000603@gmail.com> <4D9B0FE2.9020003@gmx.de> Message-ID: On Tue, Apr 5, 2011 at 8:54 AM, Ren? Dudfield wrote: > hehe, I like how you managed to avoid the first letter of the second word > of the language ;) > =) yeah, I had to think about that a bit. Looked good, I just went ahead and posted the first part: > > > http://morepypy.blogspot.com/2011/04/tutorial-writing-interpreter-with-pypy.html > > Will do the second part tomorrow. Thanks a lot for the tutorial, I think > it's really great. > Thanks! It looks great up there. I corrected a typo and changed the wording in 2 places. They're not any huge deals, but if you want to edit the post, see my changes here: https://bitbucket.org/brownan/pypy-tutorial/changeset/8cfb3cd72515 (summary: re-inventing -> re-implementing, toochain -> toolchain, and clarified that the mandelbrot program is written in BF) -Andrew -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110405/2cc78831/attachment.htm From arigo at tunes.org Tue Apr 5 15:54:08 2011 From: arigo at tunes.org (Armin Rigo) Date: Tue, 5 Apr 2011 15:54:08 +0200 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: References: <4D9474A4.3020909@gmail.com> <4D94F995.5020509@gmail.com> <4D99E219.3050308@gmx.de> <4D99F325.5050605@gmail.com> <4D9A0A29.1000603@gmail.com> <4D9B0FE2.9020003@gmx.de> Message-ID: Hi, On Tue, Apr 5, 2011 at 3:40 PM, Andrew Brown wrote: > I corrected a typo and changed the wording in 2 places. They're not any huge > deals, but if you want to edit the post, see my changes here: > https://bitbucket.org/brownan/pypy-tutorial/changeset/8cfb3cd72515 Thanks, applied. I also removed your explicit e-mail address, and replaced it with a link to one of your previous posts on this mailing list, from where people can still get your e-mail if they want --- but at least it's partially filtered against spammers. A bient?t, Armin. From dimaqq at gmail.com Tue Apr 5 16:10:12 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Tue, 5 Apr 2011 07:10:12 -0700 Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: References: <602617.13125.qm@web65613.mail.ac4.yahoo.com> Message-ID: you guessed right. I had to guess too btw :) here's where dicsussions about particular tests are and should go, as there's no single point of contact for shootout: http://alioth.debian.org/forum/forum.php?forum_id=999 I'll try to find my posts about particular benchmarks I had problems with. On 5 April 2011 00:59, Armin Rigo wrote: > Hi, > > On Tue, Apr 5, 2011 at 2:23 AM, Isaac Gouy wrote: >> A simple yes / no question. >> Do you want PyPy to be shown in the benchmarks game or not? > > Sorry for missing general knowledge of what you mean by "benchmarks > game". ?According to Google, it is probably what I know as the > language shootout (shootout.alioth.debian.org). ?Is it the case? > > Assuming it is, then my position is pretty much the same as William > Leslie: I have little interest if the benchmarks that PyPy runs with > are written in completely non-idiomatic ways, super hand-optimized for > CPython, for the reasons explained in detail by William. ?It would be > interesting, however, if you accept versions rewritten in "just plain > Python". > > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From lstask at gmail.com Tue Apr 5 17:08:57 2011 From: lstask at gmail.com (Liam Staskawicz) Date: Tue, 5 Apr 2011 08:08:57 -0700 Subject: [pypy-dev] minimum system requirements, build configuration In-Reply-To: References: <3B1535B65AE041F0B53FF5F13BF8178D@gmail.com> Message-ID: Hi David, thanks for the quick response. On Tuesday, April 5, 2011 at 7:18 AM, David Schneider wrote: > Hi Liam, > > On Tue, Apr 5, 2011 at 07:25, Liam Staskawicz wrote: > > Hi - I've been looking at pypy as a possible application runtime in an ARM9 > > Linux based system. I understand that the in-progress ARM jit backend > > targets only ARMv7, but I'm still interested in characterizing a translated > > pypy-c interpreter on this system with regard to CPU and memory usage. Of > > course I look forward to possible ARMv5 support for the jit in the future, > > but if I've understood correctly, fully interpreted mode should be supported > > - please let me know if this is not correct! > > This is correct, the non-jitted version can be cross-translated to > ARM. Although it still only supports the Boehm GC so running PyPy > without the JIT will be slower than CPython. The JIT currently targets > the ARMv7 architecture, we have not tested on an ARMv5 processor yet, > but it would be interesting to see what is missing for compatibility > with ARMv5. I see - non-jitted is perhaps not such an interesting option at the moment then, I suppose. Are the other GC options not available because they rely on the jit, or is it another dependency? I'll try to find some time to look into what's required for ARMv5 support. If you have any thoughts on particularly ARMv7-centric functionality in the current port that would need to be addressed, that would be interesting to know. > > > To that end, are there minimum recommended system requirements for pypy, > > specifically in terms of memory? As a reference, something like the Mono > > 'Small Footprint' wiki page applied to pypy would be what I'm looking > > for: http://www.mono-project.com/Small_footprint > At this point in time we have no official minimum requirements to run > pypy on ARM, but (quoting Armin) probably anything starting from 64MB > should be fine, depending on the application. The BeagleBoard which is > used to develop the ARM backend has 512 MB and pypy works fine on it > so far. OK, good to know - thanks! > > > On a related note, most of the build config info I've come across seems to > > revolve around a scratchbox2 build environment targeting maemo - indeed it > > seems to be the main other 'platform' option in the pypy build script, other > > than 'host'. Is there any particularly tight coupling between maemo and > > pypy, or should I hope to be able to set up a generic sbox2 environment for > > my target and come away with a working build? Are there any other docs or > > config snippets that detail how to set up a generic cross compile > > environment for pypy? > > We used the scratchbox2 because it allows to create an environment > which can cross-compile programs, but also run small programs in a > context similar to the target environment. This is used to gather > information about the target, required for the translation. > For the ARM backend we are using an Ubuntu rootfs with the build-tools > installed and the gcc-arm cross compiler installed on the host. The > translation toolchain only needs to now the name of the sb2 > environment in which to compile and execute the programs. One thing to > note is that the Python interpreter on the host must match the target > system in the sense of 32/64 bit. Besides these points any sb2 > environment could be used to try the translation for a given target. Sounds good - I'll give it a shot. Thanks again. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110405/145e16f7/attachment-0001.htm From arigo at tunes.org Tue Apr 5 17:14:50 2011 From: arigo at tunes.org (Armin Rigo) Date: Tue, 5 Apr 2011 17:14:50 +0200 Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: References: <602617.13125.qm@web65613.mail.ac4.yahoo.com> Message-ID: Hi Isaac, If you're not interested in this discussion and just want a Yes or No answer to precisely the following question: "should PyPy be added to the benchmarks game's site, with no changes whatsoever to any of the Python benchmarks that are there so far?" then the answer is No. We are not interested in the results of PyPy on such skewed benchmarks, and publishing them would only add to the general confusion. Armin From arigo at tunes.org Tue Apr 5 17:54:31 2011 From: arigo at tunes.org (Armin Rigo) Date: Tue, 5 Apr 2011 17:54:31 +0200 Subject: [pypy-dev] minimum system requirements, build configuration In-Reply-To: References: <3B1535B65AE041F0B53FF5F13BF8178D@gmail.com> Message-ID: Hi, On Tue, Apr 5, 2011 at 5:08 PM, Liam Staskawicz wrote: > This is correct, the non-jitted version can be cross-translated to > ARM. Although it still only supports the Boehm GC I think that you are saying something wrong: "no jit + our own GC + the shadowstack gc root finder" should get you pure ANSI C, and basically run anywhere. Still, even this version is 1.5x to 2x slower than CPython. You should also mention that some work I did recently adds support for the shadowstack gc root finder to the JIT, so that even on ARM having the JIT with a good GC is not far away (even if it's indeed not done yet). > At this point in time we have no official minimum requirements to run > pypy on ARM, but (quoting Armin) probably anything starting from 64MB > should be fine, depending on the application. I'm rather confident than 64MB is enough, but I also guess that 32MB has a chance to work. With only 16MB I bet nothing much works right now (given that e.g. our GC, by default, does not collect before the heap grew to at least 16MB). It may be possible for anyone with enough motivation to improve the situation on 16MB. A bient?t, Armin. From dimaqq at gmail.com Tue Apr 5 21:26:26 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Tue, 5 Apr 2011 12:26:26 -0700 Subject: [pypy-dev] minimum system requirements, build configuration In-Reply-To: References: <3B1535B65AE041F0B53FF5F13BF8178D@gmail.com> Message-ID: On 5 April 2011 08:54, Armin Rigo wrote: [snip] > I'm rather confident than 64MB is enough, but I also guess that 32MB > has a chance to work. ?With only 16MB I bet nothing much works right > now (given that e.g. our GC, by default, does not collect before the > heap grew to at least 16MB). ?It may be possible for anyone with > enough motivation to improve the situation on 16MB. Ouch! That's something to do. I'm currently running cpython 2.6 with a bunch of extension in pre-production on a system that has 32MB in total; and another case, a slim long-running process that runs on a big server, cpython footprint only several MB. In the first case, your treshold would create high memory pressure (bad for os caching things), in the 2nd a saw-like memory use and perhaps significant latency spikes. What I'm trying to say, is gc should adapt to run-time behaviour of a particular script, in some cases 16MB heap threshold would impact both user expectation and performance significantly. Put it up on TODO list! d. From dimaqq at gmail.com Tue Apr 5 21:30:39 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Tue, 5 Apr 2011 12:30:39 -0700 Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: References: <602617.13125.qm@web65613.mail.ac4.yahoo.com> Message-ID: It seems most pypy benchmarks are moved from cpython and as a result they fail because a particular module is unavalable. A more pragmatic approach is to go through the list of benchmarks and either accept current python code for pypy or fail on purpose if python code is too un-pypy-like. Later on, hopefully, someone volunteers to rewrite problem benchmarks. This, btw, is almost status quo. how about that? both visibility for pypy and avoidance of worst problems... d. On 5 April 2011 08:14, Armin Rigo wrote: > Hi Isaac, > > If you're not interested in this discussion and just want a Yes or No > answer to precisely the following question: > > "should PyPy be added to the benchmarks game's site, with no changes > whatsoever to any of the Python benchmarks that are there so far?" > > then the answer is No. ?We are not interested in the results of PyPy > on such skewed benchmarks, and publishing them would only add to the > general confusion. > > > Armin > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From arigo at tunes.org Tue Apr 5 21:44:37 2011 From: arigo at tunes.org (Armin Rigo) Date: Tue, 5 Apr 2011 21:44:37 +0200 Subject: [pypy-dev] minimum system requirements, build configuration In-Reply-To: References: <3B1535B65AE041F0B53FF5F13BF8178D@gmail.com> Message-ID: Hi Dima, On Tue, Apr 5, 2011 at 9:26 PM, Dima Tisnek wrote: > What I'm trying to say, is gc should adapt to run-time behaviour of a > particular script, in some cases 16MB heap threshold would impact both > user expectation and performance significantly. It may impact user expectation and performance, but: User expectation: so far we've considered the "desktop" users, which do not care about 10MB versus 20MB but start to care when it's about 300MB versus 600MB. It's true that other categories of users exist. Sorry for not being able to care for all possible use cases at once :-) Performance: actually this setting of ours -- not collecting before we have at least 16MB of data -- was done to avoid degradation of performance in cases where the Python script has really low memory usage, so the idea "it's consuming 30MB instead of 5MB so it must have a terrible performance" sounds pretty abstract. But again this is assuming a system where 30MB is a small fraction of the total amount of RAM. If you want to care about the use case of, say, systems with 16MB of RAM in total, then feel free to tweak PyPy. I suppose that you'd get the best results by tweaking one of our GCs, or writing a new one. (For example it probably doesn't make much sense to have a nursery at all.) I hope this helps to clarify the issue, Armin From dimaqq at gmail.com Tue Apr 5 21:52:24 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Tue, 5 Apr 2011 12:52:24 -0700 Subject: [pypy-dev] minimum system requirements, build configuration In-Reply-To: References: <3B1535B65AE041F0B53FF5F13BF8178D@gmail.com> Message-ID: Oh, I guessed your reasons. Lanugages like python make a lot of garbage, so 16MB will fill up pretty fast as long as the program does something at all. What I mean to say is that there's gotta be a more clever way where gc thresholds depend on e.g. size of working set or rate of new allocations or something yet smarter. d. On 5 April 2011 12:44, Armin Rigo wrote: > Hi Dima, > > On Tue, Apr 5, 2011 at 9:26 PM, Dima Tisnek wrote: >> What I'm trying to say, is gc should adapt to run-time behaviour of a >> particular script, in some cases 16MB heap threshold would impact both >> user expectation and performance significantly. > > It may impact user expectation and performance, but: > > User expectation: so far we've considered the "desktop" users, which > do not care about 10MB versus 20MB but start to care when it's about > 300MB versus 600MB. ?It's true that other categories of users exist. > Sorry for not being able to care for all possible use cases at once > :-) > > Performance: actually this setting of ours -- not collecting before we > have at least 16MB of data -- was done to avoid degradation of > performance in cases where the Python script has really low memory > usage, so the idea "it's consuming 30MB instead of 5MB so it must have > a terrible performance" sounds pretty abstract. ?But again this is > assuming a system where 30MB is a small fraction of the total amount > of RAM. > > If you want to care about the use case of, say, systems with 16MB of > RAM in total, then feel free to tweak PyPy. ?I suppose that you'd get > the best results by tweaking one of our GCs, or writing a new one. > (For example it probably doesn't make much sense to have a nursery at > all.) > > > I hope this helps to clarify the issue, > > Armin > From arigo at tunes.org Tue Apr 5 21:53:18 2011 From: arigo at tunes.org (Armin Rigo) Date: Tue, 5 Apr 2011 21:53:18 +0200 Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: References: <602617.13125.qm@web65613.mail.ac4.yahoo.com> Message-ID: Hi Dima, On Tue, Apr 5, 2011 at 9:30 PM, Dima Tisnek wrote: > A more pragmatic approach is to go through the list of benchmarks and > either accept current python code for pypy or fail on purpose if > python code is too un-pypy-like. > > Later on, hopefully, someone volunteers to rewrite problem benchmarks. Several of us already went down that road. However, as I said, the unresolved issue right now is if the maintainer (Isaac?) is willing to accept or not any new, simpler, more portable version of the benchmarks programs. If not, then that's the end of this discussion, I fear. Armin From arigo at tunes.org Tue Apr 5 22:00:23 2011 From: arigo at tunes.org (Armin Rigo) Date: Tue, 5 Apr 2011 22:00:23 +0200 Subject: [pypy-dev] minimum system requirements, build configuration In-Reply-To: References: <3B1535B65AE041F0B53FF5F13BF8178D@gmail.com> Message-ID: Hi Dima, On Tue, Apr 5, 2011 at 9:52 PM, Dima Tisnek wrote: > What I mean > to say is that there's gotta be a more clever way where gc thresholds > depend on e.g. size of working set or rate of new allocations or > something yet smarter. Yes, it does in PyPy; we do a full collection when the total amount of data reaches 1.82 times (by default) the amount of live data at the end of the previous collection, with additional tweaks to improve performance in various cases -- and one such tweak is to set the minimum threshold to 16MB (actually, now I think it is not fixed to 16MB but it depends on the amount of L2 cache). It was all reached by various benchmarks on various desktop machines, including the minimum threshold. You can see the various thresholds and their defaults at the start of pypy/rpython/memory/gc/minimark.py (and that's only if you are using minimark, our default GC). Of course all the numbers -- and even half of the algorithms -- are going to be bogus if you start to think about very different machines. That's what I meant when I said that there is open work to do, and you or anyone with an interest in the area (and corresponding hardware) is welcome to attack it. A bient?t, Armin From igouy2 at yahoo.com Tue Apr 5 22:38:48 2011 From: igouy2 at yahoo.com (Isaac Gouy) Date: Tue, 5 Apr 2011 13:38:48 -0700 (PDT) Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: Message-ID: <151772.71454.qm@web65605.mail.ac4.yahoo.com> --- On Tue, 4/5/11, Armin Rigo wrote: > what I know as the language shootout (shootout.alioth.debian.org).? > Is it the case? The project was renamed back on 20th April 2007 http://c2.com/cgi/wiki?GreatComputerLanguageShootout > Assuming it is, then my position is pretty much the same as > William Leslie: I have little interest if the benchmarks that > PyPy runs with are written in completely non-idiomatic ways, > super hand-optimized for CPython, for the reasons explained in > detail by William.? It would be interesting, however, if you > accept versions rewritten in "just plain Python". I have no objection to programs written in "just plain Python" - just as I have no objection to programs written "in completely non-idiomatic ways, super hand-optimized for CPython". However, unless there was something interesting about them - such as PyPy made them fast - they might be weeded out in the future. In most cases I do have an objection to numpy and to calling C using ctypes - programs have to be more "plain Python" than that. From dimaqq at gmail.com Tue Apr 5 22:51:42 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Tue, 5 Apr 2011 13:51:42 -0700 Subject: [pypy-dev] minimum system requirements, build configuration In-Reply-To: References: <3B1535B65AE041F0B53FF5F13BF8178D@gmail.com> Message-ID: Hi Armin, thanks for pointing me in the right direction. If minimark.py docstring is up to date, looks like it will do close to the right thing on arm5/9. Some of the older arms don't have L2 cache, I'm not sure what /proc/cpuinfo reports in this case, perhaps L1 cache size or nothing. It seems if the cache line is missing in cpuinfo, get_L2cache_linux2 would return maxint (ouch memory) or, if kernel prints 0 there, then 0 (ouch cpu), or if -1 gets in the way, default nursery size, which is 256 bytes (ouch cpu again). If anyone tries to run pypy under QEMU in system mode, results might be very very odd! Modern arm processors have L2 cache from 256KB to 1MB, thus the expected minimum threshold is 1MB to 4MB, seems reasonable enough, as importing a decent set of python standard library is around that. I'm sure Liam is pleased :P d. On 5 April 2011 13:00, Armin Rigo wrote: > Hi Dima, > > On Tue, Apr 5, 2011 at 9:52 PM, Dima Tisnek wrote: >> What I mean >> to say is that there's gotta be a more clever way where gc thresholds >> depend on e.g. size of working set or rate of new allocations or >> something yet smarter. > > Yes, it does in PyPy; we do a full collection when the total amount of > data reaches 1.82 times (by default) the amount of live data at the > end of the previous collection, with additional tweaks to improve > performance in various cases -- and one such tweak is to set the > minimum threshold to 16MB (actually, now I think it is not fixed to > 16MB but it depends on the amount of L2 cache). ?It was all reached by > various benchmarks on various desktop machines, including the minimum > threshold. ?You can see the various thresholds and their defaults at > the start of pypy/rpython/memory/gc/minimark.py (and that's only if > you are using minimark, our default GC). > > Of course all the numbers -- and even half of the algorithms -- are > going to be bogus if you start to think about very different machines. > ?That's what I meant when I said that there is open work to do, and > you or anyone with an interest in the area (and corresponding > hardware) is welcome to attack it. > > > A bient?t, > > Armin > From dimaqq at gmail.com Tue Apr 5 22:54:29 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Tue, 5 Apr 2011 13:54:29 -0700 Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: <151772.71454.qm@web65605.mail.ac4.yahoo.com> References: <151772.71454.qm@web65605.mail.ac4.yahoo.com> Message-ID: On 5 April 2011 13:38, Isaac Gouy wrote: > > > --- On Tue, 4/5/11, Armin Rigo wrote: > >> what I know as the language shootout (shootout.alioth.debian.org). >> Is it the case? > > > The project was renamed back on 20th April 2007 > > http://c2.com/cgi/wiki?GreatComputerLanguageShootout > > > >> Assuming it is, then my position is pretty much the same as >> William Leslie: I have little interest if the benchmarks that >> PyPy runs with are written in completely non-idiomatic ways, >> super hand-optimized for CPython, for the reasons explained in >> detail by William.? It would be interesting, however, if you >> accept versions rewritten in "just plain Python". > > > I have no objection to programs written in "just plain Python" - just as I have no objection to programs written "in completely non-idiomatic ways, super hand-optimized for CPython". > > However, unless there was something interesting about them - such as PyPy made them fast - they might be weeded out in the future. > > In most cases I do have an objection to numpy and to calling C using ctypes - programs have to be more "plain Python" than that. Perhaps there should be separate categories for "python" and "numpy" for benchmarks where it makes sense? From lstask at gmail.com Tue Apr 5 23:09:55 2011 From: lstask at gmail.com (Liam Staskawicz) Date: Tue, 5 Apr 2011 14:09:55 -0700 Subject: [pypy-dev] minimum system requirements, build configuration In-Reply-To: References: <3B1535B65AE041F0B53FF5F13BF8178D@gmail.com> Message-ID: Indeed - great to see some of the thought going into this area of the project :) If I make any progress on the actual HW I'll be sure to let you know. Liam On Tue, Apr 5, 2011 at 1:51 PM, Dima Tisnek wrote: > Hi Armin, thanks for pointing me in the right direction. > > If minimark.py docstring is up to date, looks like it will do close to > the right thing on arm5/9. > > Some of the older arms don't have L2 cache, I'm not sure what > /proc/cpuinfo reports in this case, perhaps L1 cache size or nothing. > It seems if the cache line is missing in cpuinfo, get_L2cache_linux2 > would return maxint (ouch memory) or, if kernel prints 0 there, then 0 > (ouch cpu), or if -1 gets in the way, default nursery size, which is > 256 bytes (ouch cpu again). If anyone tries to run pypy under QEMU in > system mode, results might be very very odd! > > Modern arm processors have L2 cache from 256KB to 1MB, thus the > expected minimum threshold is 1MB to 4MB, seems reasonable enough, as > importing a decent set of python standard library is around that. > > I'm sure Liam is pleased :P > > d. > > On 5 April 2011 13:00, Armin Rigo wrote: > > Hi Dima, > > > > On Tue, Apr 5, 2011 at 9:52 PM, Dima Tisnek wrote: > >> What I mean > >> to say is that there's gotta be a more clever way where gc thresholds > >> depend on e.g. size of working set or rate of new allocations or > >> something yet smarter. > > > > Yes, it does in PyPy; we do a full collection when the total amount of > > data reaches 1.82 times (by default) the amount of live data at the > > end of the previous collection, with additional tweaks to improve > > performance in various cases -- and one such tweak is to set the > > minimum threshold to 16MB (actually, now I think it is not fixed to > > 16MB but it depends on the amount of L2 cache). It was all reached by > > various benchmarks on various desktop machines, including the minimum > > threshold. You can see the various thresholds and their defaults at > > the start of pypy/rpython/memory/gc/minimark.py (and that's only if > > you are using minimark, our default GC). > > > > Of course all the numbers -- and even half of the algorithms -- are > > going to be bogus if you start to think about very different machines. > > That's what I meant when I said that there is open work to do, and > > you or anyone with an interest in the area (and corresponding > > hardware) is welcome to attack it. > > > > > > A bient?t, > > > > Armin > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110405/a9529403/attachment.htm From qbproger at gmail.com Tue Apr 5 23:49:15 2011 From: qbproger at gmail.com (Joe) Date: Tue, 5 Apr 2011 17:49:15 -0400 Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: <602617.13125.qm@web65613.mail.ac4.yahoo.com> References: <602617.13125.qm@web65613.mail.ac4.yahoo.com> Message-ID: While I spent my Saturday trying to make PyPy look better in the language shootout, I'm leaning towards taking it out. While a comparison between languages may be interesting, maybe having 1 implementation per language in the shootout would work better. Then there is one target to optimize for. PyPy and CPython have very different performance characteristics. I feel as though speed.python.org may be a better venue for comparing python implementations. Since the pypy developers have closer ties to the Python Core developers and it's been stated they'll have influence. It can be made to be fair for all parties involved. Since all parties will likely be python implementations they can all agree on one implementation and use that. Joe On Mon, Apr 4, 2011 at 8:23 PM, Isaac Gouy wrote: > Hi > > A simple yes / no question. > > Do you want PyPy to be shown in the benchmarks game or not? > > Please consider the question amongst yourselves and then let me know. > > Of course, I'll make up my own mind but at least I'll be able to take your wishes into account. > > best wishes, Isaac > > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From barren.yak at gmail.com Tue Apr 5 23:27:36 2011 From: barren.yak at gmail.com (Ryan Baker) Date: Tue, 5 Apr 2011 14:27:36 -0700 Subject: [pypy-dev] A Short Introduction Message-ID: Greetings PyPy-dev! I'm new here on the mailing list (and to PyPy) and thought I would give a short introduction. My name is Ryan and I'm a student at Oregon State University in Computer Science. I'm currently interning at Intel until September. A little technical background: I love Python (surprise!), and have been programming in various languages since I was about 12. I have a large C and Java background while dabbling in others such as LISP, Ruby, Obj-C, and Lua. Most of my coding has been small personal projects or academic work and decided to finally get involved in a community. I was drawn to the PyPy project after I played with if over a weekend and found it to be extremely relevant to my personal interests. I look forward to meeting all of you both on the mailing list and on IRC and I hope that I can put some of my time to good use and help PyPy! On a side note, I usually am very quick to respond to emails so if you ever have a question, idea, or anything you want to bounce off of me, please feel free to shoot me a message. Best Regards, Ryan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110405/7ec7f452/attachment.htm From barren.yak at gmail.com Wed Apr 6 00:27:55 2011 From: barren.yak at gmail.com (Ryan Baker) Date: Tue, 5 Apr 2011 15:27:55 -0700 Subject: [pypy-dev] =?utf-8?q?A_Short_Introducti=E2=80=8Bon?= Message-ID: I apologize for the additional message, but it seems I forgot to put mention IRC nick (thanks Laura!). I'll be using the nick barren_yak. See you all there. Best Regards, Ryan -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110405/12af3d9e/attachment.htm From aaron.devore at gmail.com Wed Apr 6 01:22:21 2011 From: aaron.devore at gmail.com (Aaron DeVore) Date: Tue, 5 Apr 2011 16:22:21 -0700 Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: References: <602617.13125.qm@web65613.mail.ac4.yahoo.com> Message-ID: On Tue, Apr 5, 2011 at 2:49 PM, Joe wrote: > While I spent my Saturday trying to make PyPy look better in the > language shootout, I'm leaning towards taking it out. ?While a > comparison between languages may be interesting, maybe having 1 > implementation per language in the shootout would work better. ?Then > there is one target to optimize for. ?PyPy and CPython have very > different performance characteristics. > > I feel as though speed.python.org may be a better venue for comparing > python implementations. ?Since the pypy developers have closer ties to > the Python Core developers and it's been stated they'll have > influence. ?It can be made to be fair for all parties involved. ?Since > all parties will likely be python implementations they can all agree > on one implementation and use that. > > Joe I heavily recommend keeping PyPy in the Shootout in some form. Even with the Shootout's flaws, it is nice to have a general idea of how PyPy compares to both CPython and other languages. -Aaron DeVore From fijall at gmail.com Wed Apr 6 12:14:22 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 6 Apr 2011 12:14:22 +0200 Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: References: <602617.13125.qm@web65613.mail.ac4.yahoo.com> Message-ID: On Wed, Apr 6, 2011 at 1:22 AM, Aaron DeVore wrote: > On Tue, Apr 5, 2011 at 2:49 PM, Joe wrote: >> While I spent my Saturday trying to make PyPy look better in the >> language shootout, I'm leaning towards taking it out. ?While a >> comparison between languages may be interesting, maybe having 1 >> implementation per language in the shootout would work better. ?Then >> there is one target to optimize for. ?PyPy and CPython have very >> different performance characteristics. >> >> I feel as though speed.python.org may be a better venue for comparing >> python implementations. ?Since the pypy developers have closer ties to >> the Python Core developers and it's been stated they'll have >> influence. ?It can be made to be fair for all parties involved. ?Since >> all parties will likely be python implementations they can all agree >> on one implementation and use that. >> >> Joe > > I heavily recommend keeping PyPy in the Shootout in stome form. Even > with the Shootout's flaws, it is nice to have a general idea of how > PyPy compares to both CPython and other languages. We don't get that information now at least, since those benchmarks are badly skewed towards CPython. I know how hard is to find out a reasonable set of benchmarks and how to keep them balanced. I have another issue with ctypes & numpy. This is that C implementations are allowed to use gcc-specific hacks and non-standard libraries (apache malloc for gcbench). Why wouldn't we be allowed to do the same then? I would like to have PyPy included, but I would also like the benchmark game to be "fair" or as close to "fair" as we can get. Having benchmarks with different rules for different languages doesn't seem to be quite fair in my opinion. Note that this is up to discussions - I'm fine with saying "numpy & ctypes are disallowed" or having a separate category "python + numpy" with both CPython and PyPy (once we get numpy). > > -Aaron DeVore > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From igouy2 at yahoo.com Wed Apr 6 15:08:10 2011 From: igouy2 at yahoo.com (Isaac Gouy) Date: Wed, 6 Apr 2011 06:08:10 -0700 (PDT) Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: Message-ID: <564429.74157.qm@web65602.mail.ac4.yahoo.com> --- On Wed, 4/6/11, Maciej Fijalkowski wrote: -snip- > We don't get that information now at least, since those > benchmarks are badly skewed towards CPython. I know how > hard is to find out a reasonable set of benchmarks and how to > keep them balanced. Do you mean the program you contributed is badly skewed towards CPython? http://shootout.alioth.debian.org/u32/program.php?test=nbody&lang=pypy&id=1 Do you mean that the n-body problem is badly skewed towards CPython? Your PyPy program is shown as so much faster - how is that "badly skewed towards CPython"? From cfbolz at gmx.de Wed Apr 6 15:13:13 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Wed, 06 Apr 2011 15:13:13 +0200 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: References: <4D9474A4.3020909@gmail.com> <4D94F995.5020509@gmail.com> <4D99E219.3050308@gmx.de> <4D99F325.5050605@gmail.com> <4D9A0A29.1000603@gmail.com> <4D9B0FE2.9020003@gmx.de> Message-ID: <4D9C66E9.8020003@gmx.de> On 04/05/2011 03:54 PM, Armin Rigo wrote: > Hi, > > On Tue, Apr 5, 2011 at 3:40 PM, Andrew Brown wrote: >> I corrected a typo and changed the wording in 2 places. They're not any huge >> deals, but if you want to edit the post, see my changes here: >> https://bitbucket.org/brownan/pypy-tutorial/changeset/8cfb3cd72515 > > Thanks, applied. I also removed your explicit e-mail address, and > replaced it with a link to one of your previous posts on this mailing > list, from where people can still get your e-mail if they want --- but > at least it's partially filtered against spammers. Second post is up too: http://bit.ly/fLjGHs Thanks again, Andrew! FWIW, the first post is already on place four of all PyPy blog posts in the ranking of page impressions. Carl Friedrich From igouy2 at yahoo.com Wed Apr 6 15:15:36 2011 From: igouy2 at yahoo.com (Isaac Gouy) Date: Wed, 6 Apr 2011 06:15:36 -0700 (PDT) Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: Message-ID: <405881.94798.qm@web65614.mail.ac4.yahoo.com> --- On Wed, 4/6/11, Maciej Fijalkowski wrote: -snip- > I have another issue with ctypes & numpy. This is that > C implementations are allowed to use gcc-specific hacks and > non-standard libraries (apache malloc for gcbench). Why wouldn't > we be allowed to do the same then? Would you describe C as "batteries included" ? From fijall at gmail.com Wed Apr 6 15:31:12 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 6 Apr 2011 15:31:12 +0200 Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: <405881.94798.qm@web65614.mail.ac4.yahoo.com> References: <405881.94798.qm@web65614.mail.ac4.yahoo.com> Message-ID: On Wed, Apr 6, 2011 at 3:15 PM, Isaac Gouy wrote: > > > --- On Wed, 4/6/11, Maciej Fijalkowski wrote: > > -snip- >> I have another issue with ctypes & numpy. This is that >> C implementations are allowed to use gcc-specific hacks and >> non-standard libraries (apache malloc for gcbench). Why wouldn't >> we be allowed to do the same then? > > Would you describe C as "batteries included" ? > nope, I would not. However, ctypes come included in standard python distribution (unlike numpy or gmpy) so this is not really a valid argument. > > > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From fijall at gmail.com Wed Apr 6 15:34:56 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 6 Apr 2011 15:34:56 +0200 Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: <564429.74157.qm@web65602.mail.ac4.yahoo.com> References: <564429.74157.qm@web65602.mail.ac4.yahoo.com> Message-ID: On Wed, Apr 6, 2011 at 3:08 PM, Isaac Gouy wrote: > > > --- On Wed, 4/6/11, Maciej Fijalkowski wrote: > > -snip- >> We don't get that information now at least, since those >> benchmarks are badly skewed towards CPython. I know how >> hard is to find out a reasonable set of benchmarks and how to >> keep them balanced. > > > Do you mean the program you contributed is badly skewed towards CPython? > > http://shootout.alioth.debian.org/u32/program.php?test=nbody&lang=pypy&id=1 > No > > Do you mean that the n-body problem is badly skewed towards CPython? No, that would be nonsense. I would never discuss whether those benchmarks does represent typical workflow in language X because it's impossible to find such a set that's true for every X. I never did discuss the choice of problems. > > Your PyPy program is shown as so much faster - how is that "badly skewed towards CPython"? > That's true, but that's one that got through. For example reverse complement (the current version) is skewed towards CPython. I'm fine with saying that ctypes (or numpy) are not allowed, with a good explanation (and maybe an explanation why custom malloc library is allowed for C and gcbench). Another question which was raised - are programs that only work on PyPy allowed? (Due to pypy's extensions or cpython bugs). Since programs that only compile on GCC clearly are. Cheers, fijal From brownan at gmail.com Wed Apr 6 16:03:44 2011 From: brownan at gmail.com (Andrew Brown) Date: Wed, 6 Apr 2011 10:03:44 -0400 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: <4D9C66E9.8020003@gmx.de> References: <4D9474A4.3020909@gmail.com> <4D94F995.5020509@gmail.com> <4D99E219.3050308@gmx.de> <4D99F325.5050605@gmail.com> <4D9A0A29.1000603@gmail.com> <4D9B0FE2.9020003@gmx.de> <4D9C66E9.8020003@gmx.de> Message-ID: Hmm, looks like the line numbers for the JIT trace output are mis-aligned, although it may just be my browser (Chrome beta). Looks fine in Firefox. Oh well. But anyways... On Wed, Apr 6, 2011 at 9:13 AM, Carl Friedrich Bolz wrote: > Thanks again, Andrew! FWIW, the first post is already on place four of > all PyPy blog posts in the ranking of page impressions. > You're welcome! That's awesome to hear, I'm glad I could contribute. Also, Dan, if you wanted to post your version I'm curious to see your approach. -Andrew -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110406/51ab76a2/attachment.htm From cfbolz at gmx.de Wed Apr 6 16:05:02 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Wed, 06 Apr 2011 16:05:02 +0200 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: References: <4D9474A4.3020909@gmail.com> <4D94F995.5020509@gmail.com> <4D99E219.3050308@gmx.de> <4D99F325.5050605@gmail.com> <4D9A0A29.1000603@gmail.com> <4D9B0FE2.9020003@gmx.de> <4D9C66E9.8020003@gmx.de> Message-ID: <4D9C730E.2070904@gmx.de> On 04/06/2011 04:03 PM, Andrew Brown wrote: > Hmm, looks like the line numbers for the JIT trace output are > mis-aligned, although it may just be my browser (Chrome beta). Looks > fine in Firefox. Oh well. Can you re-load? I tried to fix it. Carl Friedrich From brownan at gmail.com Wed Apr 6 16:12:13 2011 From: brownan at gmail.com (Andrew Brown) Date: Wed, 6 Apr 2011 10:12:13 -0400 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: <4D9C730E.2070904@gmx.de> References: <4D9474A4.3020909@gmail.com> <4D94F995.5020509@gmail.com> <4D99E219.3050308@gmx.de> <4D99F325.5050605@gmail.com> <4D9A0A29.1000603@gmail.com> <4D9B0FE2.9020003@gmx.de> <4D9C66E9.8020003@gmx.de> <4D9C730E.2070904@gmx.de> Message-ID: No good, it still looks like this: http://i.imgur.com/nuLIf.png Chrome 12.0.712.0 dev on Ubuntu. -A On Wed, Apr 6, 2011 at 10:05 AM, Carl Friedrich Bolz wrote: > On 04/06/2011 04:03 PM, Andrew Brown wrote: > >> Hmm, looks like the line numbers for the JIT trace output are >> mis-aligned, although it may just be my browser (Chrome beta). Looks >> fine in Firefox. Oh well. >> > > Can you re-load? I tried to fix it. > > Carl Friedrich > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110406/6d45fc29/attachment.htm From dimaqq at gmail.com Wed Apr 6 16:55:36 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Wed, 6 Apr 2011 07:55:36 -0700 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: References: <4D9474A4.3020909@gmail.com> <4D94F995.5020509@gmail.com> <4D99E219.3050308@gmx.de> <4D99F325.5050605@gmail.com> <4D9A0A29.1000603@gmail.com> <4D9B0FE2.9020003@gmx.de> <4D9C66E9.8020003@gmx.de> <4D9C730E.2070904@gmx.de> Message-ID: line numbers are totally broken in opera - looks like double-digit number are split and spilt to the next line safari looks better, but still as if line numbers are offset half a line, so as if numbers point between the source code lines. I used SyntaxHighlighter in blogs before, that works, highlights well, gives you line numbers and doesn't interfere with selection. It's presumably tested with all the browsers out there and it's a simple drop-in. d. On 6 April 2011 07:12, Andrew Brown wrote: > No good, it still looks like this:?http://i.imgur.com/nuLIf.png > Chrome?12.0.712.0 dev on Ubuntu. > -A > > On Wed, Apr 6, 2011 at 10:05 AM, Carl Friedrich Bolz wrote: >> >> On 04/06/2011 04:03 PM, Andrew Brown wrote: >>> >>> Hmm, looks like the line numbers for the JIT trace output are >>> mis-aligned, although it may just be my browser (Chrome beta). Looks >>> fine in Firefox. Oh well. >> >> Can you re-load? I tried to fix it. >> >> Carl Friedrich > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From igouy2 at yahoo.com Wed Apr 6 17:18:03 2011 From: igouy2 at yahoo.com (Isaac Gouy) Date: Wed, 6 Apr 2011 08:18:03 -0700 (PDT) Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: Message-ID: <646173.36708.qm@web65603.mail.ac4.yahoo.com> --- On Wed, 4/6/11, Maciej Fijalkowski wrote: -snip- > > Do you mean the program you contributed is badly > skewed towards CPython? > > > > http://shootout.alioth.debian.org/u32/program.php?test=nbody?=pypy&id=1 > > > > No > > > > > Do you mean that the n-body problem is badly skewed > towards CPython? > > No, that would be nonsense. I would never discuss whether > those > benchmarks does represent typical workflow in language X > because it's > impossible to find such a set that's true for every X. I > never did > discuss the choice of problems. > > > > > Your PyPy program is shown as so much faster - how is > that "badly skewed towards CPython"? > > > > That's true, but that's one that got through. I don't see how the program you contributed could be described as "one that got through". Here's what happened - I noticed the n-body program failed with PyPy, I asked you guys about the problem and was told "we have nbody_modified in our benchmarks" and then I asked you guys to contribute your modified program. http://morepypy.blogspot.com/2010/03/introducing-pypy-12-release.html Once you contributed the program modified for PyPy it was displayed on the website within 3 hours. > For example reverse complement (the current version) is > skewed towards CPython. If only someone could manage to write a reverse complement skewed towards PyPy without using libc.write ;-) > I'm fine with saying that ctypes (or numpy) are not allowed, > with a good explanation (and maybe an explanation why custom malloc > library is allowed for C and gcbench). > > Another question which was raised - are programs that only > work on PyPy allowed? (Due to pypy's extensions or cpython bugs). PyPy extensions - No. CPython bugs - How strange that the CPython bug was never mentioned! - maybe. > Since programs that only compile on GCC clearly are. How many C language implementations are shown? How many Python language implementations are shown? If only one Python language implementation was shown do you think it would be PyPy ? From fijall at gmail.com Wed Apr 6 17:32:08 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 6 Apr 2011 17:32:08 +0200 Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: <646173.36708.qm@web65603.mail.ac4.yahoo.com> References: <646173.36708.qm@web65603.mail.ac4.yahoo.com> Message-ID: On Wed, Apr 6, 2011 at 5:18 PM, Isaac Gouy wrote: > > > --- On Wed, 4/6/11, Maciej Fijalkowski wrote: > > -snip- >> > Do you mean the program you contributed is badly >> skewed towards CPython? >> > >> > http://shootout.alioth.debian.org/u32/program.php?test=nbody?=pypy&id=1 >> > >> >> No >> >> > >> > Do you mean that the n-body problem is badly skewed >> towards CPython? >> >> No, that would be nonsense. I would never discuss whether >> those >> benchmarks does represent typical workflow in language X >> because it's >> impossible to find such a set that's true for every X. I >> never did >> discuss the choice of problems. >> >> > >> > Your PyPy program is shown as so much faster - how is >> that "badly skewed towards CPython"? >> > >> >> That's true, but that's one that got through. > > > I don't see how the program you contributed could be described as "one that got through". > > Here's what happened - I noticed the n-body program failed with PyPy, I asked you guys about the problem and was told "we have nbody_modified in our benchmarks" and then I asked you guys to contribute your modified program. > > http://morepypy.blogspot.com/2010/03/introducing-pypy-12-release.html > > Once you contributed the program modified for PyPy it was displayed on the website within 3 hours. > Cool, thank you. > > >> For example reverse complement (the current version) is >> skewed towards CPython. > > If only someone could manage to write a reverse complement skewed towards PyPy without using libc.write ;-) > Deal, we'll do that. > > >> I'm fine with saying that ctypes (or numpy) are not allowed, >> with a good explanation (and maybe an explanation why custom malloc >> library is allowed for C and gcbench). >> >> Another question which was raised - are programs that only >> work on PyPy allowed? (Due to pypy's extensions or cpython bugs). > > PyPy extensions - No. > > CPython bugs - How strange that the CPython bug was never mentioned! - maybe. > Ok. The bug was not mentioned because it takes time to decide "it's a bug". > >> Since programs that only compile on GCC clearly are. > > How many C language implementations are shown? > > How many Python language implementations are shown? > > If only one Python language implementation was shown do you think it would be PyPy ? > I can't really read your mind, but from my opinion if the question is "how fast Python programs can be run" then the answer is Yes. So the position is that GCC is allowed to use extensions because it's the only C implementation shown and PyPy is not, because all Python programs should run on each runtime, is that correct? I'm not stating an opinion about it, I just want to know what the rules are. > > > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev From dasdasich at googlemail.com Wed Apr 6 18:52:24 2011 From: dasdasich at googlemail.com (DasIch) Date: Wed, 6 Apr 2011 18:52:24 +0200 Subject: [pypy-dev] [GSoC] Developing a benchmark suite (for Python 3.x) Message-ID: Hello Guys, I would like to present my proposal for the Google Summer of Code, concerning the idea of porting the benchmarks to Python 3.x for speed.pypy.org. I think I have successfully integrated the feedback I got from prior discussions on the topic and I would like to hear your opinion. Abstract ======= As of now there are several benchmark suites used by Python implementations, PyPy[1] uses the benchmarks developed for the Unladen Swallow[2] project as well as several other benchmarks they implemented on their own, CPython[3] uses the Unladen Swallow benchmarks and several "crap benchmarks used for historical reasons"[4]. This makes comparisons unnecessarily hard and causes confusion. As a solution to this problem I propose merging the existing benchmarks - at least those considered worth having - into a single benchmark suite which can be shared by all implementations and ported to Python 3.x. Milestones The project can be divided into several milestones: 1. Definition of the benchmark suite. This will entail contacting developers of Python implementations (CPython, PyPy, IronPython and Jython), via discussion on the appropriate mailing lists. This might be achievable as part of this proposal. 2. Implementing the benchmark suite. Based on the prior agreed upon definition, the suite will be implemented, which means that the benchmarks will be merged into a single mercurial repository on Bitbucket[5]. 3. Porting the suite to Python 3.x. The suite will be ported to 3.x using 2to3[6], as far as possible. The usage of 2to3 will make it easier make changes to the repository especially for those still focusing on 2.x. It is to be expected that some benchmarks cannot be ported due to dependencies which are not available on Python 3.x. Those will be ignored by this project to be ported at a later time, when the necessary requirements are met. Start of Program (May 24) ====================== Before the coding, milestones 2 and 3, can begin it is necessary to agree upon a set of benchmarks, everyone is happy with, as described. Midterm Evaluation (July 12) ======================= During the midterm I want to finish the second milestone and before the evaluation I want to start in the third milestone. Final Evaluation (Aug 16) ===================== In this period the benchmark suite will be ported. If everything works out perfectly I will even have some time left, if there are problems I have a buffer here. Probably Asked Questions ====================== Why not use one of the existing benchmark suites for porting? The effort will be wasted if there is no good base to build upon, creating a new benchmark suite based upon the existing ones ensures that. Why not use Git/Bazaar/...? Mercurial is used by CPython, PyPy and is fairly well known and used in the Python community. This ensures easy accessibility for everyone. What will happen with the Repository after GSoC/How will access to the repository be handled? I propose to give administrative rights to one or two representatives of each project. Those will provide other developers with write access. Communication ============= Communication of the progress will be done via Twitter[7] and my blog[8], if desired I can also send an email with the contents of the blog post to the mailing lists of the implementations. Furthermore I am usually quick to answer via IRC (DasIch on freenode), Twitter or E-Mail(dasdasich at gmail.com) if anyone has any questions. Contact to the mentor can be established via the means mentioned above or via Skype. About Me ======== My name is Daniel Neuh?user, I am 19 years old and currently a student at the Bergstadt-Gymnasium L?denscheid[9]. I started programming (with Python) about 4 years ago and became a member of the Pocoo Team[10] after successfully participating in the Google Summer of Code last year, during which I ported Sphinx[11] to Python 3.x and implemented an algorithm to diff abstract syntax trees to preserve comments and translated strings which has been used by the other GSoC projects targeting Sphinx. .. [1]: https://bitbucket.org/pypy/benchmarks/src .. [2]: http://code.google.com/p/unladen-swallow/ .. [3]: http://hg.python.org/benchmarks/file/tip/performance .. [4]: http://hg.python.org/benchmarks/file/62e754c57a7f/performance/README .. [5]: http://bitbucket.org/ .. [6]: http://docs.python.org/library/2to3.html .. [7]: http://twitter.com/#!/DasIch .. [8]: http://dasdasich.blogspot.com/ .. [9]: http://bergstadt-gymnasium.de/ .. [10]: http://www.pocoo.org/team/#daniel-neuhauser .. [11]: http://sphinx.pocoo.org/ P.S.: I would like to get in touch with the IronPython developers as well, unfortunately I was not able to find a mailing list or IRC channel is there anybody how can send me in the right direction? From igouy2 at yahoo.com Wed Apr 6 19:33:20 2011 From: igouy2 at yahoo.com (Isaac Gouy) Date: Wed, 6 Apr 2011 10:33:20 -0700 (PDT) Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: Message-ID: <795302.82949.qm@web65602.mail.ac4.yahoo.com> --- On Wed, 4/6/11, Maciej Fijalkowski wrote: -snip- > > CPython bugs - How strange that the CPython bug was > never mentioned! - maybe. > > > > Ok. The bug was not mentioned because it takes time to > decide "it's a bug". I know someone decided "it's a bug" because someone said so in a blog post they pushed out across the blogosphere and proggit and Hacker News and ... How strange that CPython bug was never mentioned to me! > Since programs that only compile on GCC clearly > are. > > > > How many C language implementations are shown? > > > > How many Python language implementations are shown? > > > > If only one Python language implementation was shown > do you think it would be PyPy ? > > > > I can't really read your mind, but from my opinion if the > question is "how fast Python programs can be run" then the > answer is Yes. The Help page says something about showing working programs written in less familiar programming languages. > So the position is that GCC is allowed to use extensions > because it's the only C implementation shown and PyPy is not, > because all Python programs should run on each runtime, is that > correct? I don't see a way to compare CPython and PyPy unless the comparison programs do at least run on both CPython and PyPy (and x86 and x64). From fijall at gmail.com Wed Apr 6 19:40:50 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 6 Apr 2011 19:40:50 +0200 Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: <795302.82949.qm@web65602.mail.ac4.yahoo.com> References: <795302.82949.qm@web65602.mail.ac4.yahoo.com> Message-ID: On Wed, Apr 6, 2011 at 7:33 PM, Isaac Gouy wrote: > > --- On Wed, 4/6/11, Maciej Fijalkowski wrote: > > -snip- > >> > CPython bugs - How strange that the CPython bug was >> never mentioned! - maybe. >> > >> >> Ok. The bug was not mentioned because it takes time to >> decide "it's a bug". > > > I know someone decided "it's a bug" because someone said so in a blog post they pushed out across the blogosphere and proggit and Hacker News and ... > > How strange that CPython bug was never mentioned to me! > > >> Since programs that only compile on GCC clearly >> are. >> > >> > How many C language implementations are shown? >> > >> > How many Python language implementations are shown? >> > >> > If only one Python language implementation was shown >> do you think it would be PyPy ? >> > >> >> I can't really read your mind, but from my opinion if the >> question is "how fast Python programs can be run" then the >> answer is Yes. > > The Help page says something about showing working programs written in less familiar programming languages. > Ok, will look it up later. > >> So the position is that GCC is allowed to use extensions >> because it's the only C implementation shown and PyPy is not, >> because all Python programs should run on each runtime, is that >> correct? > > I don't see a way to compare CPython and PyPy unless the comparison programs do at least run on both CPython and PyPy (and x86 and x64). > Well I can always write a program that runs on both (and uses more efficient data structure for pypy for example). The question is a bit academic, because I don't have any particular implementation now in mind. But would be cool to be able to do that. From ademan555 at gmail.com Wed Apr 6 21:49:56 2011 From: ademan555 at gmail.com (Dan Roberts) Date: Wed, 6 Apr 2011 12:49:56 -0700 Subject: [pypy-dev] Pypy custom interpreter JIT question In-Reply-To: References: <4D9474A4.3020909@gmail.com> <4D94F995.5020509@gmail.com> <4D99E219.3050308@gmx.de> <4D99F325.5050605@gmail.com> <4D9A0A29.1000603@gmail.com> <4D9B0FE2.9020003@gmx.de> <4D9C66E9.8020003@gmx.de> Message-ID: Hey sorry about that, I'm working full time now, and balancing life, school and full time work is new for me... http://paste.pocoo.org/show/366731/ Is it in its current state. I actually haven't tried it on "stock" PyPy. I wrote an extra JIT optimization in the fold_intadd branch, which gave me around 50% on mandelbrot, it may yield more for you, you might try it out (I hope to have it merged once I get another set of eyes on it, but like I said, life's been hectic, so I haven't gotten the ball rolling on code review yet). My 99bottles.bf performance is still abysmal. I'm going to go and implement caching of matching brackets right now (lunch break woo!) and see what that does for my performance. Cheers, Dan On Wed, Apr 6, 2011 at 7:03 AM, Andrew Brown wrote: > Hmm, looks like the line numbers for the JIT trace output are mis-aligned, > although it may just be my browser (Chrome beta). Looks fine in Firefox. Oh > well. > But anyways... > > On Wed, Apr 6, 2011 at 9:13 AM, Carl Friedrich Bolz wrote: >> >> Thanks again, Andrew! FWIW, the first post is already on place four of >> all PyPy blog posts in the ranking of page impressions. > > You're welcome! That's awesome to hear, I'm glad I could contribute. > Also, Dan, if you wanted to post your version I'm curious to see your > approach. > -Andrew From santagada at gmail.com Thu Apr 7 04:31:09 2011 From: santagada at gmail.com (Leonardo Santagada) Date: Wed, 6 Apr 2011 23:31:09 -0300 Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: <795302.82949.qm@web65602.mail.ac4.yahoo.com> References: <795302.82949.qm@web65602.mail.ac4.yahoo.com> Message-ID: On Wed, Apr 6, 2011 at 2:33 PM, Isaac Gouy wrote: > >> So the position is that GCC is allowed to use extensions >> because it's the only C implementation shown and PyPy is not, >> because all Python programs should run on each runtime, is that >> correct? > > I don't see a way to compare CPython and PyPy unless the comparison programs do at least run on both CPython and PyPy (and x86 and x64). I don't see why this is, it is the same as comparing python to ruby, I want to see how fast can you make a program in said vm that does the same task. If the description of the problem doesn't limit what you can use I really don't see why can't you use a PyPy (or cpython) extension for it. For me a shootout without extensions (at least without numpy) is just comparing how fast a language can do without anything, which is not interesting at all. One where c programs can use libraries but python cannot is even more meaningless. -- Leonardo Santagada From fijall at gmail.com Thu Apr 7 05:52:31 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 7 Apr 2011 05:52:31 +0200 Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: References: <795302.82949.qm@web65602.mail.ac4.yahoo.com> Message-ID: On Thu, Apr 7, 2011 at 4:31 AM, Leonardo Santagada wrote: > On Wed, Apr 6, 2011 at 2:33 PM, Isaac Gouy wrote: >> >>> So the position is that GCC is allowed to use extensions >>> because it's the only C implementation shown and PyPy is not, >>> because all Python programs should run on each runtime, is that >>> correct? >> >> I don't see a way to compare CPython and PyPy unless the comparison programs do at least run on both CPython and PyPy (and x86 and x64). > > I don't see why this is, it is the same as comparing python to ruby, I > want to see how fast can you make a program in said vm that does the > same task. If the description of the problem doesn't limit what you > can use I really don't see why can't you use a PyPy (or cpython) > extension for it. > > For me a shootout without extensions (at least without numpy) is just > comparing how fast a language can do without anything, which is not > interesting at all. One where c programs can use libraries but python > cannot is even more meaningless. Leonardo please calm down a bit. I can see reasons and why I might not agree with that, they do make sense and can be justified. It does make sense to compare CPython and PyPy on the same set of benchmarks (actually that's what we do with speed.pypy.org, we deliberately tried to avoid modifying benchmarks). > > -- > Leonardo Santagada > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From vincent.legoll at gmail.com Thu Apr 7 09:51:55 2011 From: vincent.legoll at gmail.com (Vincent Legoll) Date: Thu, 7 Apr 2011 09:51:55 +0200 Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: References: <795302.82949.qm@web65602.mail.ac4.yahoo.com> Message-ID: Hello, On Thu, Apr 7, 2011 at 5:52 AM, Maciej Fijalkowski wrote: > It does make sense to compare CPython and > PyPy on the same set of benchmarks (actually that's what we do with > speed.pypy.org, we deliberately tried to avoid modifying benchmarks). But speed.pypy.org is about comparing cpython vs pypy, whereas the benchmark game compares a lot of quite different laguages, that is not exactly the same thing. So, that may warrant different rules... -- Vincent Legoll From damonmc at gmail.com Thu Apr 7 19:15:35 2011 From: damonmc at gmail.com (Damon McCormick) Date: Thu, 7 Apr 2011 10:15:35 -0700 Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: References: <795302.82949.qm@web65602.mail.ac4.yahoo.com> Message-ID: The benchmark game also compares on code size. So if PyPy provides better performance with smaller code size (i.e. if allows you to write something in the most concise, Pythonic way and get great performance), this may not show up unless PyPy can run a different version of a benchmark that actually uses less code. (Disclaimer: it's been a few years since I looked at the benchmark game Python programs. Maybe they're already written very concisely, in which case this point is moot) -Damon On Thu, Apr 7, 2011 at 12:51 AM, Vincent Legoll wrote: > Hello, > > On Thu, Apr 7, 2011 at 5:52 AM, Maciej Fijalkowski > wrote: > > It does make sense to compare CPython and > > PyPy on the same set of benchmarks (actually that's what we do with > > speed.pypy.org, we deliberately tried to avoid modifying benchmarks). > > But speed.pypy.org is about comparing cpython vs pypy, whereas > the benchmark game compares a lot of quite different laguages, that > is not exactly the same thing. So, that may warrant different rules... > > -- > Vincent Legoll > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110407/5a0389b8/attachment.htm From dimaqq at gmail.com Thu Apr 7 20:16:18 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Thu, 7 Apr 2011 11:16:18 -0700 Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: References: <795302.82949.qm@web65602.mail.ac4.yahoo.com> Message-ID: btw, benchmark game also tracks memory footprint, pypy is a little liberal here for small benchmarks, which there are many. On 7 April 2011 10:15, Damon McCormick wrote: > The benchmark game also compares on code size. ?So if PyPy provides better > performance with smaller code size (i.e. if allows you to write something in > the most concise, Pythonic way and get great performance), this may not show > up unless PyPy can run a different version of a benchmark that actually uses > less code. > (Disclaimer: it's been a few years since I looked at the benchmark game > Python programs. ?Maybe they're already written very concisely, in which > case this point is moot) > -Damon > > On Thu, Apr 7, 2011 at 12:51 AM, Vincent Legoll > wrote: >> >> Hello, >> >> On Thu, Apr 7, 2011 at 5:52 AM, Maciej Fijalkowski >> wrote: >> > It does make sense to compare CPython and >> > PyPy on the same set of benchmarks (actually that's what we do with >> > speed.pypy.org, we deliberately tried to avoid modifying benchmarks). >> >> But speed.pypy.org is about comparing cpython vs pypy, whereas >> the benchmark game compares a lot of quite different laguages, that >> is not exactly the same thing. So, that may warrant different rules... >> >> -- >> Vincent Legoll >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From romain.py at gmail.com Fri Apr 8 19:34:51 2011 From: romain.py at gmail.com (Romain Guillebert) Date: Fri, 8 Apr 2011 18:34:51 +0100 Subject: [pypy-dev] Gothenburg sprint and sprint fund Message-ID: <20110408173451.GA6615@ubuntu> Hi I would like to attends to the sprint at the end of this month but I can afford the plane, the train (I come from Ireland and there is no direct plane between Dublin and Gothenburg) and the accomodation so I would like to know if I can get help from the PyPy sprint fund. >From what I previewed it would cost : - 17? * 2 for the bus between my town and Dublin's airport - 397.75? for the plane between Dublin and Copenhagen (It's really expensive in my opinion but I can't find something cheaper) - 394 SEK * 2 for the train between Copenhagen and Gothenburg I plan to stay from the 23rd to the 1st, I will sleep at SGS Veckobostader, I don't know if I can share a room with someone but I would stay from the 23rd to the 1st. I can handle some of it but I won't be able to make it without help. Thanks Romain From igouy2 at yahoo.com Fri Apr 8 19:43:15 2011 From: igouy2 at yahoo.com (Isaac Gouy) Date: Fri, 8 Apr 2011 10:43:15 -0700 (PDT) Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: Message-ID: <996929.61539.qm@web65602.mail.ac4.yahoo.com> The benchmarks game web pages now only show one language implementation for each programming language. Java -Xint, Tracemonkey JavaScript, LuaJIT, CPython, Iron Python, PyPy, Ruby 1.8.7 and JRuby 1.6 are no longer shown. From fijall at gmail.com Fri Apr 8 20:11:55 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 8 Apr 2011 20:11:55 +0200 Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: <996929.61539.qm@web65602.mail.ac4.yahoo.com> References: <996929.61539.qm@web65602.mail.ac4.yahoo.com> Message-ID: On Fri, Apr 8, 2011 at 7:43 PM, Isaac Gouy wrote: > The benchmarks game web pages now only show one language implementation for each programming language. > > Java -Xint, Tracemonkey JavaScript, LuaJIT, CPython, Iron Python, PyPy, Ruby 1.8.7 and JRuby 1.6 are no longer shown. > As I understand this is CPython 3.2 for Python right? anyway, what's the point of the above discussion then? > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From dasdasich at googlemail.com Fri Apr 8 20:21:12 2011 From: dasdasich at googlemail.com (DasIch) Date: Fri, 8 Apr 2011 20:21:12 +0200 Subject: [pypy-dev] [GSoC] Developing a benchmark suite (for Python 3.x) In-Reply-To: References: Message-ID: I talked to Fijal about my project last night, the result is that basically the project as is, is not that interesting because the means to execute the benchmarks on multiple interpreters are currently missing. Another point we talked about was that porting the benchmarks would not be very useful as the interesting ones all have dependencies which have not (yet) been ported to Python 3.x. The first point, execution on multiple interpreters, has to be solved or this project is pretty much pointless, therefore I've changed my proposal to include just that. However the proposal still includes porting the benchmarks although this is planned to happen after the development of an application able to run the benchmarks on multiple interpreters. The reason for this is that even though the portable benchmarks might not prove to be that interesting the basic stuff for porting using 2to3 would be there, making it easier to port benchmarks in the future, as the dependencies become available under Python 3.x. However I plan to do that after implementing the prior mentioned application putting the application at higher priority. This way, should I not be able to complete all my goals, it is unlikely that anything but the porting will suffer and the project would still produce useful results during the GSoC. Anyway here is the current, updated, proposal: Abstract ======= As of now there are several benchmark suites used by Python implementations, PyPy uses the benchmarks[1] developed for the Unladen Swallow[2] project as well as several other benchmarks they implemented on their own, CPython[3] uses the Unladen Swallow benchmarks and several "crap benchmarks used for historical reasons"[4]. This makes comparisons unnecessarily hard and causes confusion. As a solution to this problem I propose merging the existing benchmarks - at least those considered worth having - into a single benchmark suite which can be shared by all implementations and ported to Python 3.x. Another problem reported by Maciej Fijalkowski is that currenly the way benchmarks are executed by PyPy is more or less a hack. Work will have to be done to allow execution of the benchmarks on different interpreters and their most recent versions (from their respective repositories). The application for this should also be able to upload the results to a codespeed instance such as http://speed.pypy.org. Milestones ========= The project can be divided into several milestones: 1. Definition of the benchmark suite. This will entail contacting developers of Python implementations (CPython, PyPy, IronPython and Jython), via discussion on the appropriate mailing lists. This might be achievable as part of this proposal. 2. Merging the benchmarks. Based on the prior agreed upon definition, the benchmarks will be merged into a single suite. 3. Implementing a system to run the benchmarks. In order to execute the benchmarks it will be necessary to have a configurable application which downloads the interpreters from their repositories, builds them and executes the benchmarks with them. 4. Porting the suite to Python 3.x. The suite will be ported to 3.x using 2to3[5], as far as possible. The usage of 2to3 will make it easier make changes to the repository especially for those still focusing on 2.x. It is to be expected that some benchmarks cannot be ported due to dependencies which are not available on Python 3.x. Those will be ignored by this project to be ported at a later time, when the necessary requirements are met. Start of Program (May 24) ====================== Before the coding, milestones 2 and 3, can begin it is necessary to agree upon a set of benchmarks, everyone is happy with, as described. Midterm Evaluation (July 12) ======================= During the midterm I want to merge the benchmarks and implement a way to execute them. Final Evaluation (Aug 16) ===================== In this period the benchmark suite will be ported. If everything works out perfectly I will even have some time left, if there are problems I have a buffer here. Implementation of the Benchmark Runner ================================== In order to run the benchmarks I propose a simple application which can be configured to download multiple interpreters, to build them and execute the benchmarks. The configuration could be similar to tox[6], downloads of the interpreters could be handled using anyvc[7]. For a site such as http://speed.pypy.org a cronjob, buildbot or whatelse is preferred, could be setup which executes the application regularly. Repository Handling ================ The code for the project will be developed in a Mercurial[8] repository hosted on Bitbucket[9], both PyPy and CPython use Mercurial and most people in the Python community should be able to use it. Probably Asked Questions ====================== Why not use one of the existing benchmark suites for porting? The effort will be wasted if there is no good base to build upon, creating a new benchmark suite based upon the existing ones ensures that. Why not use Git/Bazaar/...? Mercurial is used by CPython, PyPy and is fairly well known and used in the Python community. This ensures easy accessibility for everyone. What will happen with the Repository after GSoC/How will access to the repository be handled? I propose to give administrative rights to one or two representatives of each project. Those will provide other developers with write access. Communication ============= Communication of the progress will be done via Twitter[10] and my blog[11], if desired I can also send an email with the contents of the blog post to the mailing lists of the implementations. Furthermore I am usually quick to answer via IRC(DasIch on freenode), Twitter or E-Mail(dasdasich at gmail.com) if anyone has any questions. Contact to the mentor can be established via the means mentioned above or via Skype. About Me ======== My name is Daniel Neuh?user, I am 19 years old and currently a student at the Bergstadt-Gymnasium L?denscheid[12]. I started programming (with Python) about 4 years ago and became a member of the Pocoo Team[13] after successfully participating in the Google Summer of Code last year, during which I ported Sphinx[14] to Python 3.x and implemented an algorithm to diff abstract syntax trees to preserve comments and translated strings which has been used by the other GSoC projects targeting Sphinx. .. [1]: https://bitbucket.org/pypy/benchmarks/src .. [2]: http://code.google.com/p/unladen-swallow/ .. [3]: http://hg.python.org/benchmarks/file/tip/performance .. [4]: http://hg.python.org/benchmarks/file/62e754c57a7f/performance/README .. [5]: http://docs.python.org/library/2to3.html .. [6]: http://codespeak.net/tox/ .. [7]: http://anyvc.readthedocs.org/en/latest/?redir .. [8]: http://mercurial.selenic.com/ .. [9]: https://bitbucket.org/ .. [10]: http://twitter.com/#!/DasIch .. [11]: http://dasdasich.blogspot.com/ .. [12]: http://bergstadt-gymnasium.de/ .. [13]: http://www.pocoo.org/team/#daniel-neuhauser .. [14]: http://sphinx.pocoo.org/ From igouy2 at yahoo.com Fri Apr 8 20:30:06 2011 From: igouy2 at yahoo.com (Isaac Gouy) Date: Fri, 8 Apr 2011 11:30:06 -0700 (PDT) Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: Message-ID: <946445.91725.qm@web65615.mail.ac4.yahoo.com> --- On Fri, 4/8/11, Maciej Fijalkowski wrote: > > The benchmarks game web pages now only show one > language implementation for each programming language. > > > > Java -Xint, Tracemonkey JavaScript, LuaJIT, CPython, > Iron Python, PyPy, Ruby 1.8.7 and JRuby 1.6 are no longer > shown. > > > > As I understand this is CPython 3.2 for Python right? CPython 2.7 is no longer shown - 3.2 is still shown. > anyway, what's the point of the above discussion then? The point of the discussion was to hear the views of pypy-dev, and I have. From fijall at gmail.com Fri Apr 8 20:39:33 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 8 Apr 2011 20:39:33 +0200 Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: <946445.91725.qm@web65615.mail.ac4.yahoo.com> References: <946445.91725.qm@web65615.mail.ac4.yahoo.com> Message-ID: On Fri, Apr 8, 2011 at 8:30 PM, Isaac Gouy wrote: > > > --- On Fri, 4/8/11, Maciej Fijalkowski wrote: > >> > The benchmarks game web pages now only show one >> language implementation for each programming language. >> > >> > Java -Xint, Tracemonkey JavaScript, LuaJIT, CPython, >> Iron Python, PyPy, Ruby 1.8.7 and JRuby 1.6 are no longer >> shown. >> > >> >> As I understand this is CPython 3.2 for Python right? > > CPython 2.7 is no longer shown - 3.2 is still shown. > > >> anyway, what's the point of the above discussion then? > > The point of the discussion was to hear the views of pypy-dev, and I have. > I guess a lot of discussions are about getting some sort of consensus. I see this one is so you can know what we think and that's it. Well, that comes as a bit of surprise. I think it's super stupid to remove Tracemonkey, LuaJIT and PyPy from it, but that's as you pointed out *your* website. On the other hand it's good, because people won't cite the computer language shootout anymore and those benchmarks are more silly than they have to be. Farewell. > > > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From igouy2 at yahoo.com Fri Apr 8 21:03:06 2011 From: igouy2 at yahoo.com (Isaac Gouy) Date: Fri, 8 Apr 2011 12:03:06 -0700 (PDT) Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: Message-ID: <734328.13811.qm@web65602.mail.ac4.yahoo.com> --- On Fri, 4/8/11, Maciej Fijalkowski wrote: -snip- > I guess a lot of discussions are about getting some sort of > consensus. I see this one is so you can know what we think > and that's it. Well, that comes as a bit of surprise. I don't know why that would come as any surprise - "Of course, I'll make up my own mind but at least I'll be able to take your wishes into account." http://permalink.gmane.org/gmane.comp.python.pypy/7303 > I think it's super stupid to remove Tracemonkey, LuaJIT and > PyPy from it, but that's as you pointed out *your* website. > On the other hand it's good, because people won't cite the > computer language shootout anymore and those benchmarks are > more silly than they have to be. You express both of the contrary wishes that I've heard here this week - you seem to want yes and no :-) fwiw someone did write - "While a comparison between languages may be interesting, maybe having 1 implementation per language in the shootout would work better." - let's hope at least they are happy now. From fijall at gmail.com Fri Apr 8 21:12:06 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 8 Apr 2011 21:12:06 +0200 Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: <734328.13811.qm@web65602.mail.ac4.yahoo.com> References: <734328.13811.qm@web65602.mail.ac4.yahoo.com> Message-ID: On Fri, Apr 8, 2011 at 9:03 PM, Isaac Gouy wrote: > > > --- On Fri, 4/8/11, Maciej Fijalkowski wrote: > > -snip- >> I guess a lot of discussions are about getting some sort of >> consensus. I see this one is so you can know what we think >> and that's it. Well, that comes as a bit of surprise. > > I don't know why that would come as any surprise - "Of course, I'll make up my own mind but at least I'll be able to take your wishes into account." > > http://permalink.gmane.org/gmane.comp.python.pypy/7303 > > > >> I think it's super stupid to remove Tracemonkey, LuaJIT and >> PyPy from it, but that's as you pointed out *your* website. >> On the other hand it's good, because people won't cite the >> computer language shootout anymore and those benchmarks are >> more silly than they have to be. > > You express both of the contrary wishes that I've heard here this week - you seem to want yes and no :-) No. I think it's bad for you good for us. > > fwiw someone did write - "While a comparison between languages may be interesting, maybe having 1 implementation per language in the shootout would work better." - let's hope at least they are happy now. > I sure hope so. > > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From piotr.skamruk at gmail.com Fri Apr 8 21:19:15 2011 From: piotr.skamruk at gmail.com (Piotr Skamruk) Date: Fri, 8 Apr 2011 21:19:15 +0200 Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: <734328.13811.qm@web65602.mail.ac4.yahoo.com> References: <734328.13811.qm@web65602.mail.ac4.yahoo.com> Message-ID: 2011/4/8 Isaac Gouy : > [...] >> I think it's super stupid to remove Tracemonkey, LuaJIT and >> PyPy from it, but that's as you pointed out *your* website. >> On the other hand it's good, because people won't cite the >> computer language shootout anymore and those benchmarks are >> more silly than they have to be. > > You express both of the contrary wishes that I've heard here this week - you seem to want yes and no :-) > > fwiw someone did write - "While a comparison between languages may be interesting, maybe having 1 implementation per language in the shootout would work better." - let's hope at least they are happy now. sorry, but now it's not a comparsion of languages, but comparison of several randomly selected programs. remember that You are not comparing languages, but their implementations (still that looks like randomly choosen - cpython 3.2 have much, much less usage than 2.6/2.7). From dimaqq at gmail.com Fri Apr 8 21:48:54 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Fri, 8 Apr 2011 12:48:54 -0700 Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: References: <734328.13811.qm@web65602.mail.ac4.yahoo.com> Message-ID: Overall one language implementation per language is a good idea for shootout. Consequently, using the latest version of most widely used implementation (cpython 3.2) is indeed the way to go, even in the face of all the 2.6 recalcitrants, myself included :-) As for pypy, I think it is in the pypy's best interest to be represented as widely as possible, I am surprised to see negative attitude here... Though I understand pypy implementation of these benchmarks requires work; btw that would be a great project for, say, summer of code. Coming back to Isaac's question, perhaps you want to a tiered representation for languages, e.g. compiled vs dynamic, then inside dynamic perl vs python, then inside that cpython 2 vs cpython 3 vs pypy. This is really a matter of presentation. Alternative is to list "supported" implementations separate from that untested/contributed/questionable/etc. Coming back to presentation, shootout is pretty old by now, compare it to speed.pypy.org which is outright sexy! Also you need serious QA of the benchmarks, rules, implementations and even the test environment itself. Dima Tisnek On 8 April 2011 12:19, Piotr Skamruk wrote: > 2011/4/8 Isaac Gouy : >> [...] >>> I think it's super stupid to remove Tracemonkey, LuaJIT and >>> PyPy from it, but that's as you pointed out *your* website. >>> On the other hand it's good, because people won't cite the >>> computer language shootout anymore and those benchmarks are >>> more silly than they have to be. >> >> You express both of the contrary wishes that I've heard here this week - you seem to want yes and no :-) >> >> fwiw someone did write - "While a comparison between languages may be interesting, maybe having 1 implementation per language in the shootout would work better." - let's hope at least they are happy now. > sorry, but now it's not a comparsion of languages, but comparison of > several randomly selected programs. > remember that You are not comparing languages, but their > implementations (still that looks like randomly choosen - cpython 3.2 > have much, much less usage than 2.6/2.7). > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From piotr.skamruk at gmail.com Fri Apr 8 22:22:26 2011 From: piotr.skamruk at gmail.com (Piotr Skamruk) Date: Fri, 8 Apr 2011 22:22:26 +0200 Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: References: <734328.13811.qm@web65602.mail.ac4.yahoo.com> Message-ID: 2011/4/8 Dima Tisnek : > Overall one language implementation per language is a good idea for > shootout. Consequently, using the latest version of most widely used > implementation (cpython 3.2) is indeed the way to go, even in the face > of all the 2.6 recalcitrants, myself included :-) So last (3.2) or most used (2.6)? Sorry, but 3.x is not widely used for now... > As for pypy, I think it is in the pypy's best interest to be > represented as widely as possible, I am surprised to see negative > attitude here... Though I understand pypy implementation of these > benchmarks requires work; btw that would be a great project for, say, > summer of code. pypy is probably fastest implementation for now, but also it could not be described as "widely used" > Coming back to Isaac's question, perhaps you want to a tiered > representation for languages, e.g. compiled vs dynamic, then inside > dynamic perl vs python, then inside that cpython 2 vs cpython 3 vs > pypy. This is really a matter of presentation. Alternative is to list > "supported" implementations ?separate from that > untested/contributed/questionable/etc. Coming back to presentation, > shootout is pretty old by now, compare it to speed.pypy.org which is > outright sexy! Also you need serious QA of the benchmarks, rules, > implementations and even the test environment itself. If someone is comparing some implemetations of languages, than that person should probably choose most common used version of compiler/interpreter - not "most recent release/revision". shootout.alioth.debian.org is runned under debian domain, so it's for me hard to understand why py3.2 was choosen as python representation when it has minor usage even in debian... (still in debian 2.6 is more mainline, even 2.7 is not so supported outside of ubuntu) speed.pypy.org have resonable value probably mostly for developers, with strong focus on python developers. shootout.alioth.debian.org, as more general site, is focused on informing users less with less knowledge about programming. in current state comparsion in shootout.alioth.debian.org introduces more confusion than usefull information, what probably had in mind Maciej. From igouy2 at yahoo.com Fri Apr 8 22:38:51 2011 From: igouy2 at yahoo.com (Isaac Gouy) Date: Fri, 8 Apr 2011 13:38:51 -0700 (PDT) Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: Message-ID: <518937.62474.qm@web65614.mail.ac4.yahoo.com> --- On Fri, 4/8/11, Piotr Skamruk wrote: -snip- > remember that You are not comparing languages, but their > implementations That's correct! "Which programming languages are fastest? No. Which programming language implementations have the fastest benchmark programs?" http://shootout.alioth.debian.org/u32/which-programming-languages-are-fastest.php#chart From igouy2 at yahoo.com Fri Apr 8 23:25:28 2011 From: igouy2 at yahoo.com (Isaac Gouy) Date: Fri, 8 Apr 2011 14:25:28 -0700 (PDT) Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: Message-ID: <651381.86449.qm@web65607.mail.ac4.yahoo.com> --- On Fri, 4/8/11, Maciej Fijalkowski wrote: -snip- > No. I think it's bad for you good for us. Then you have your wish - be happy ;-) From jacob at openend.se Sat Apr 9 00:43:43 2011 From: jacob at openend.se (Jacob =?iso-8859-1?q?Hall=E9n?=) Date: Sat, 9 Apr 2011 00:43:43 +0200 Subject: [pypy-dev] PyPy in the benchmarks game - yes or no? In-Reply-To: <734328.13811.qm@web65602.mail.ac4.yahoo.com> References: <734328.13811.qm@web65602.mail.ac4.yahoo.com> Message-ID: <201104090043.50198.jacob@openend.se> Friday 08 April 2011 you wrote: > --- On Fri, 4/8/11, Maciej Fijalkowski wrote: > > -snip- > > > I guess a lot of discussions are about getting some sort of > > consensus. I see this one is so you can know what we think > > and that's it. Well, that comes as a bit of surprise. > > I don't know why that would come as any surprise - "Of course, I'll make up > my own mind but at least I'll be able to take your wishes into account." > > http://permalink.gmane.org/gmane.comp.python.pypy/7303 > > > I think it's super stupid to remove Tracemonkey, LuaJIT and > > PyPy from it, but that's as you pointed out *your* website. > > On the other hand it's good, because people won't cite the > > computer language shootout anymore and those benchmarks are > > more silly than they have to be. > > You express both of the contrary wishes that I've heard here this week - > you seem to want yes and no :-) > > fwiw someone did write - "While a comparison between languages may be > interesting, maybe having 1 implementation per language in the shootout > would work better." - let's hope at least they are happy now. I think you should consider your own wishes a little more. If you want the language shootout to be relevant to people, you can't ignore multiple implementations. Especially it seems to b excessively excentric to ignore the fastest implementations of some languages while not doing so for others. I assume you are not measuring C speed by the old AT&T reference implementation. I think you have to make up your mind. Do you want to provide a service to other people, in which case you should work on making fair and reasonable comparisons, mislead people, in which case you should continue on the current course. Perhaps you don't care either way; perhaps you are just in it for the ego boost. Then we want to find out, so we can stop wasting our time trying to get a reasonable outcome. I think you need to stop and think about what you really want, instead of emmotionally reacting to evolving events. When you have given it a good thought, you tell people what you have come up with and go off implementing it. Tose who like it will gather around, those who dislike it will go away. If you don't know what you want out of the language shootout, you might ask people in different camps what they would like. You will be surprised by the number of sane and reasonable answers you would get. Jacob Hall?n -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. Url : http://codespeak.net/pipermail/pypy-dev/attachments/20110409/22054499/attachment.pgp From getxsick at gmail.com Sat Apr 9 13:12:04 2011 From: getxsick at gmail.com (Bartosz SKOWRON) Date: Sat, 9 Apr 2011 13:12:04 +0200 Subject: [pypy-dev] Gothenburg sprint and sprint fund In-Reply-To: <20110408173451.GA6615@ubuntu> References: <20110408173451.GA6615@ubuntu> Message-ID: On Fri, Apr 8, 2011 at 7:34 PM, Romain Guillebert wrote: > - 397.75? for the plane between Dublin and Copenhagen (It's really > ?expensive in my opinion but I can't find something cheaper) > - 394 SEK * 2 for the train between Copenhagen and Gothenburg Very quick research showed me a connection between Dublin and Gothenburg for 160?. From lac at openend.se Sat Apr 9 13:17:16 2011 From: lac at openend.se (Laura Creighton) Date: Sat, 09 Apr 2011 13:17:16 +0200 Subject: [pypy-dev] Gothenburg sprint and sprint fund In-Reply-To: Message from Bartosz SKOWRON of "Sat, 09 Apr 2011 13:12:04 +0200." References: <20110408173451.GA6615@ubuntu> Message-ID: <201104091117.p39BHGsY006157@theraft.openend.se> What did you find? Because both of the Gothenburg airports claim to have no connections to Ireland whatsoever. Laura, who would be very pleasantly surprised if this is wrong. From getxsick at gmail.com Sat Apr 9 13:19:32 2011 From: getxsick at gmail.com (Bartosz SKOWRON) Date: Sat, 9 Apr 2011 13:19:32 +0200 Subject: [pypy-dev] Gothenburg sprint and sprint fund In-Reply-To: <201104091117.p39BHGsY006157@theraft.openend.se> References: <20110408173451.GA6615@ubuntu> <201104091117.p39BHGsY006157@theraft.openend.se> Message-ID: On Sat, Apr 9, 2011 at 1:17 PM, Laura Creighton wrote: > What did you find? ?Because both of the Gothenburg airports claim to > have no connections to Ireland whatsoever. This is with a stop in London Stansted. However now I found a direct connection from Dublin to Stockholm Skavsta for 95? by RyanAir. It's really cheap. From lac at openend.se Sat Apr 9 13:52:13 2011 From: lac at openend.se (Laura Creighton) Date: Sat, 09 Apr 2011 13:52:13 +0200 Subject: [pypy-dev] Gothenburg sprint and sprint fund In-Reply-To: Message from Bartosz SKOWRON of "Sat, 09 Apr 2011 13:19:32 +0200." References: <20110408173451.GA6615@ubuntu> <201104091117.p39BHGsY006157@theraft.openend.se> Message-ID: <201104091152.p39BqDw1009245@theraft.openend.se> In a message of Sat, 09 Apr 2011 13:19:32 +0200, Bartosz SKOWRON writes: >On Sat, Apr 9, 2011 at 1:17 PM, Laura Creighton wrote: > >> What did you find? =C2=A0Because both of the Gothenburg airports claim >to >> have no connections to Ireland whatsoever. > >This is with a stop in London Stansted. However now I found a direct >connection from Dublin to Stockholm Skavsta for by RyanAir. It's >really cheap. The Skavsta Airport really isn't in Stockholm -- it's 90 minutes away by bus. What is is close to is Nyk?ping, which is 7 km away. And you can catch a train to G?teborg from Nyk?ping. So this looks like a lot cheaper. Thank you. http://www.skavsta.se/sv/content/2/38/t%C3%A5g.html says that it costs 200 SEK to get to the Nyk?ping train station from Skavska airport by taxi, or 20 SEK if you take the local bus 515 or 715 Laura From lac at openend.se Sat Apr 9 14:05:15 2011 From: lac at openend.se (Laura Creighton) Date: Sat, 09 Apr 2011 14:05:15 +0200 Subject: [pypy-dev] Gothenburg sprint and sprint fund In-Reply-To: Message from Laura Creighton of "Sat, 09 Apr 2011 13:52:13 +0200." <201104091152.p39BqDw1009245@theraft.openend.se> References: <20110408173451.GA6615@ubuntu> <201104091117.p39BHGsY006157@theraft.openend.se> <201104091152.p39BqDw1009245@theraft.openend.se> Message-ID: <201104091205.p39C5FPP010642@theraft.openend.se> In a message of Sat, 09 Apr 2011 13:52:13 +0200, Laura Creighton writes: re: >>This is with a stop in London Stansted. However now I found a direct >>connection from Dublin to Stockholm Skavsta for mailer didn't like-- laura > by RyanAir. It's >>really cheap. And I am getting a price of 39,99 Euros out (on the 23rd) and 9,98 if you leave on the 3rd. 59,99 if you leave on the 2nd. So yes, much cheaper. Thank you. Laura From getxsick at gmail.com Sat Apr 9 14:27:49 2011 From: getxsick at gmail.com (Bartosz SKOWRON) Date: Sat, 9 Apr 2011 14:27:49 +0200 Subject: [pypy-dev] Gothenburg sprint and sprint fund In-Reply-To: <201104091205.p39C5FPP010642@theraft.openend.se> References: <20110408173451.GA6615@ubuntu> <201104091117.p39BHGsY006157@theraft.openend.se> <201104091152.p39BqDw1009245@theraft.openend.se> <201104091205.p39C5FPP010642@theraft.openend.se> Message-ID: On Sat, Apr 9, 2011 at 2:05 PM, Laura Creighton wrote: > And I am getting a price of 39,99 Euros out (on the 23rd) > and 9,98 if you leave on the 3rd. ?59,99 if you leave on the 2nd. > > So yes, much cheaper. ?Thank you. :) I'm glad I could help. Wish to come to the sprint either but can't make it for this date :( From lac at openend.se Sat Apr 9 14:38:53 2011 From: lac at openend.se (Laura Creighton) Date: Sat, 09 Apr 2011 14:38:53 +0200 Subject: [pypy-dev] Gothenburg sprint and sprint fund In-Reply-To: Message from Bartosz SKOWRON of "Sat, 09 Apr 2011 14:27:49 +0200." References: <20110408173451.GA6615@ubuntu> <201104091117.p39BHGsY006157@theraft.openend.se> <201104091152.p39BqDw1009245@theraft.openend.se> <201104091205.p39C5FPP010642@theraft.openend.se> Message-ID: <201104091238.p39Ccr2J013284@theraft.openend.se> In a message of Sat, 09 Apr 2011 14:27:49 +0200, Bartosz SKOWRON writes: >On Sat, Apr 9, 2011 at 2:05 PM, Laura Creighton wrote: > >> And I am getting a price of 39,99 Euros out (on the 23rd) >> and 9,98 if you leave on the 3rd. =C2=A059,99 if you leave on the 2nd. >> >> So yes, much cheaper. =C2=A0Thank you. > >:) I'm glad I could help. Wish to come to the sprint either but can't >make it for this date :( That's really too bad. Are you going to Europython? Laura From getxsick at gmail.com Sat Apr 9 14:42:27 2011 From: getxsick at gmail.com (Bartosz SKOWRON) Date: Sat, 9 Apr 2011 14:42:27 +0200 Subject: [pypy-dev] Gothenburg sprint and sprint fund In-Reply-To: <201104091238.p39Ccr2J013284@theraft.openend.se> References: <20110408173451.GA6615@ubuntu> <201104091117.p39BHGsY006157@theraft.openend.se> <201104091152.p39BqDw1009245@theraft.openend.se> <201104091205.p39C5FPP010642@theraft.openend.se> <201104091238.p39Ccr2J013284@theraft.openend.se> Message-ID: On Sat, Apr 9, 2011 at 2:38 PM, Laura Creighton wrote: > That's really too bad. ?Are you going to Europython? Yes, I'm going to come for the conference and the PyPy sprint. At least I will try to come ;-) From jacob at openend.se Sat Apr 9 22:50:07 2011 From: jacob at openend.se (Jacob =?utf-8?q?Hall=C3=A9n?=) Date: Sat, 9 Apr 2011 22:50:07 +0200 Subject: [pypy-dev] Thoughts on multithreading in PyPy Message-ID: <201104092250.14600.jacob@openend.se> I have been thinking about the multithreading problem in PyPy for a while and I have come up with an idea. I'd like to have feedback from people who know the codebase well. The first and hardest step is to change the PyPy runtime so that it can run multiple threads at the same time. To simplify matters, we allocate all external resources to one thread to start with. We assume that other threads don't use them. Neither do they call into extension modules that do messy things. Whenever we spawn a new thread, we give it its own object space and its own instance of the memory manager/garbage collector. Having gotten this far, we would have N threads that could run in parallell. Since they have no interaction with each other and no contetion for resources, they would require no locking mechanism. The thread with the external resources would still be dependent on the GIL, but the other ones wouldn't even see it. This setup would of course be utterly useless, because all but one of the threads would have no means of comunicationg their results to the world. So, in a second step, we provide for special data types that can be shared between threads. These would typically be allocated in non-movable memory, to avoid the complexity of garbage collection of memory with shared use. You can make simple fifo structures for communication between the threads and complex structures with advanced algorithms for dealing with shared access. In a third step, you may relax the requirement that the first thread owns all resources. You should be able to hand them out in a controlled manner. For instance, you may want to spawn a thread for each socket connection and have that thread deal with all the communication with the socket. Now I wonder about the feasability of the first step. How much global state would have to be wrapped in per-tread objects and how hard would that be? What other obstacles would there be to doing this change? I guess there is a complication with requesting memory from the kernel and returning memory, but I think that could be solved in more or less elegant ways. Jacob Hall?n -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. Url : http://codespeak.net/pipermail/pypy-dev/attachments/20110409/82afa4a4/attachment.pgp From arigo at tunes.org Sat Apr 9 23:28:34 2011 From: arigo at tunes.org (Armin Rigo) Date: Sat, 9 Apr 2011 23:28:34 +0200 Subject: [pypy-dev] Thoughts on multithreading in PyPy In-Reply-To: <201104092250.14600.jacob@openend.se> References: <201104092250.14600.jacob@openend.se> Message-ID: Hi Jacob, On Sat, Apr 9, 2011 at 10:50 PM, Jacob Hall?n wrote: > So, in a second step, we provide for special data types that can be shared > between threads. These would typically be allocated in non-movable memory, to > avoid the complexity of garbage collection of memory with shared use. You can > make simple fifo structures for communication between the threads and complex > structures with advanced algorithms for dealing with shared access. That's where the real issue is. You can come up with some reasonable API to communicate with other threads, but precisely, they will be some API, which means that they will only work in programs written specifically to use them. Designing a new API (at the level of the Python language) is something we carefully avoided so far in PyPy; but it's possible that this issue is important enough for us to break that rule :-) What you are describing sounds similar to the multiprocessing module in CPython, which achieves the same goal using separated processes (and tons and tons of hacks), and requires the program to use a custom API. The advantage of doing it in PyPy rather than CPython is probably limited to the fact that it would be easier in PyPy (but still some work) to make sure that the multiple threads have no shared state. You still have to design some custom API. There is also another possible goal with more "pypy-like" goals and results, which would be to use some technique to "weave" a solution in the interpreter transparently for the Python programmer (so, a solution that works without requiring the Python programmer to learn another system than threads). I can imagine a Software Transactional Memory solution that would in theory work very nicely, but in practice have completely dreadful performance, because it would do large amounts of checked memory access for each bytecode. As far as I know it means that that approach does not work, but it may one day, if Hardware Transactional Memory really shows up and supports that scale. A bient?t, Armin. From jacob at openend.se Sun Apr 10 00:15:50 2011 From: jacob at openend.se (Jacob =?iso-8859-1?q?Hall=E9n?=) Date: Sun, 10 Apr 2011 00:15:50 +0200 Subject: [pypy-dev] Thoughts on multithreading in PyPy In-Reply-To: References: <201104092250.14600.jacob@openend.se> Message-ID: <201104100015.57221.jacob@openend.se> Saturday 09 April 2011 you wrote: > Hi Jacob, > > On Sat, Apr 9, 2011 at 10:50 PM, Jacob Hall?n wrote: > > So, in a second step, we provide for special data types that can be > > shared between threads. These would typically be allocated in > > non-movable memory, to avoid the complexity of garbage collection of > > memory with shared use. You can make simple fifo structures for > > communication between the threads and complex structures with advanced > > algorithms for dealing with shared access. > > That's where the real issue is. You can come up with some reasonable > API to communicate with other threads, but precisely, they will be > some API, which means that they will only work in programs written > specifically to use them. Designing a new API (at the level of the > Python language) is something we carefully avoided so far in PyPy; but > it's possible that this issue is important enough for us to break that > rule :-) > > What you are describing sounds similar to the multiprocessing module > in CPython, which achieves the same goal using separated processes > (and tons and tons of hacks), and requires the program to use a custom > API. The advantage of doing it in PyPy rather than CPython is > probably limited to the fact that it would be easier in PyPy (but > still some work) to make sure that the multiple threads have no shared > state. You still have to design some custom API. The multiprocessing API contains some classic primitives that could be kept. Lots of the rest seem way too complicated. This is because they are dealing with separate processes. I think you are downplaying the advantages of using PyPy. Apart from being easier and cleaner to implement, it would be using threads instead of processes, providing for much quicker communication and context switches between threads. Then it would be able to use the JIT, providing much better performance. You could also provide proxy object spaces to transparently spread load over multiple physical machines. Now, I don't think we should go ahead and start work on this now. I just like exploring the idea. If people come along wanting to do GIL removal, we can present them with a plan and set them off working. > There is also another possible goal with more "pypy-like" goals and > results, which would be to use some technique to "weave" a solution in > the interpreter transparently for the Python programmer (so, a > solution that works without requiring the Python programmer to learn > another system than threads). I can imagine a Software Transactional > Memory solution that would in theory work very nicely, but in practice > have completely dreadful performance, because it would do large > amounts of checked memory access for each bytecode. As far as I know > it means that that approach does not work, but it may one day, if > Hardware Transactional Memory really shows up and supports that scale. While being a very neat idea, it is still pie in the sky. My idea could be pie-on-plate, though it hasn't been baked yet. Jacob -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. Url : http://codespeak.net/pipermail/pypy-dev/attachments/20110410/8b5e9466/attachment-0001.pgp From dimaqq at gmail.com Sun Apr 10 05:02:01 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Sat, 9 Apr 2011 20:02:01 -0700 Subject: [pypy-dev] Thoughts on multithreading in PyPy In-Reply-To: <201104100015.57221.jacob@openend.se> References: <201104092250.14600.jacob@openend.se> <201104100015.57221.jacob@openend.se> Message-ID: IMO real zest and usefulness of multithreading are complex shared data structures. That said, most multithreaded programs share a very small, albeit critical subset of their data. What you propose is something akin to cpython multiprocessing or stackless with the addition of parallel execution of independent tasklets. I think stackless users would like that actually. Now when you get to the point of introducing some communication between processes, do you mean to pass only byte streams? primitive types? complex data structures? The last you cannot do as you have separate gc's, so you are limited to copyable data structures only, which is as useful as multiprocessing (concept, not module) and json/pickle messages. In short, a clone of multiprocessing would be useful, perhaps when numpy evolves to support user defined pypy functions well. It's a niche, not general approach though. d. On 9 April 2011 15:15, Jacob Hall?n wrote: > Saturday 09 April 2011 you wrote: >> Hi Jacob, >> >> On Sat, Apr 9, 2011 at 10:50 PM, Jacob Hall?n wrote: >> > So, in a second step, we provide for special data types that can be >> > shared between threads. These would typically be allocated in >> > non-movable memory, to avoid the complexity of garbage collection of >> > memory with shared use. You can make simple fifo structures for >> > communication between the threads and complex structures with advanced >> > algorithms for dealing with shared access. >> >> That's where the real issue is. ?You can come up with some reasonable >> API to communicate with other threads, but precisely, they will be >> some API, which means that they will only work in programs written >> specifically to use them. ?Designing a new API (at the level of the >> Python language) is something we carefully avoided so far in PyPy; but >> it's possible that this issue is important enough for us to break that >> rule :-) >> >> What you are describing sounds similar to the multiprocessing module >> in CPython, which achieves the same goal using separated processes >> (and tons and tons of hacks), and requires the program to use a custom >> API. ?The advantage of doing it in PyPy rather than CPython is >> probably limited to the fact that it would be easier in PyPy (but >> still some work) to make sure that the multiple threads have no shared >> state. ?You still have to design some custom API. > > The multiprocessing API contains some classic primitives that could be kept. > Lots of the rest seem way too complicated. This is because they are dealing > with separate processes. > > I think you are downplaying the advantages of using PyPy. Apart from being > easier and cleaner to implement, it would be using threads instead of > processes, providing for much quicker communication and context switches > between threads. Then it would be able to use the JIT, providing much better > performance. You could also provide proxy object spaces to transparently > spread load over multiple physical machines. > > Now, I don't think we should go ahead and start work on this now. I just like > exploring the idea. If people come along wanting to do GIL removal, we can > present them with a plan and set them off working. > >> There is also another possible goal with more "pypy-like" goals and >> results, which would be to use some technique to "weave" a solution in >> the interpreter transparently for the Python programmer (so, a >> solution that works without requiring the Python programmer to learn >> another system than threads). ?I can imagine a Software Transactional >> Memory solution that would in theory work very nicely, but in practice >> have completely dreadful performance, because it would do large >> amounts of checked memory access for each bytecode. ?As far as I know >> it means that that approach does not work, but it may one day, if >> Hardware Transactional Memory really shows up and supports that scale. > > While being a very neat idea, it is still pie in the sky. My idea could be > pie-on-plate, though it hasn't been baked yet. > > Jacob > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From pypy at pocketnix.org Sun Apr 10 13:33:40 2011 From: pypy at pocketnix.org (pypy at pocketnix.org) Date: Sun, 10 Apr 2011 11:33:40 +0000 Subject: [pypy-dev] Thoughts on multithreading in PyPy In-Reply-To: <201104100015.57221.jacob@openend.se> References: <201104092250.14600.jacob@openend.se> <201104100015.57221.jacob@openend.se> Message-ID: <20110410113340.GB7395@pocketnix.org> What i am wondering about is if some of the base services provided by pypy can be moved into another thread, eg GC and JIT compilation and how much of a benfiit there would be to doing so at least with moving the gc to another thread i would think that doing so would provide some insight into concurrent access to python objects From matthew at woodcraft.me.uk Sun Apr 10 14:46:05 2011 From: matthew at woodcraft.me.uk (Matthew Woodcraft) Date: Sun, 10 Apr 2011 13:46:05 +0100 Subject: [pypy-dev] jit.backend.test.test_random Message-ID: <20110410124605.GB21912@golux.woodcraft.me.uk> I found that pypy.jit.backend.test.test_random is failing if I pass --backend=x86 The following patch fixes it for me: --- a/pypy/jit/backend/test/test_random.py +++ b/pypy/jit/backend/test/test_random.py @@ -717,6 +717,7 @@ def test_random_function(BuilderClass=OperationBuilder): r = Random() cpu = get_cpu() + cpu.setup_once() if pytest.config.option.repeat == -1: while 1: check_random_function(cpu, BuilderClass, r) -M- From exarkun at twistedmatrix.com Sun Apr 10 17:06:24 2011 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Sun, 10 Apr 2011 15:06:24 -0000 Subject: [pypy-dev] Thoughts on multithreading in PyPy In-Reply-To: References: <201104092250.14600.jacob@openend.se> <201104100015.57221.jacob@openend.se> Message-ID: <20110410150624.1992.1458428484.divmod.xquotient.365@localhost.localdomain> On 03:02 am, dimaqq at gmail.com wrote: >IMO real zest and usefulness of multithreading are complex shared data >structures. You forgot to say "immutable" ;) Jean-Paul From arigo at tunes.org Sun Apr 10 21:44:40 2011 From: arigo at tunes.org (Armin Rigo) Date: Sun, 10 Apr 2011 21:44:40 +0200 Subject: [pypy-dev] Thoughts on multithreading in PyPy In-Reply-To: <20110410113340.GB7395@pocketnix.org> References: <201104092250.14600.jacob@openend.se> <201104100015.57221.jacob@openend.se> <20110410113340.GB7395@pocketnix.org> Message-ID: Hi, On Sun, Apr 10, 2011 at 1:33 PM, wrote: > What i am wondering about is if some of the base services provided by > pypy can be moved into another thread, eg GC and JIT compilation and > how much of a benfiit there would be to doing so It would be possible to move the GC or the JIT to another thread, introducing a lot of complexity but keeping it hidden to the Python programmer. You would end up with a program that can use maybe 1.2 core instead of just 1. That sounds like a lot of work for a minimal gain. A bient?t, Armin. From arigo at tunes.org Sun Apr 10 21:50:44 2011 From: arigo at tunes.org (Armin Rigo) Date: Sun, 10 Apr 2011 21:50:44 +0200 Subject: [pypy-dev] jit.backend.test.test_random In-Reply-To: <20110410124605.GB21912@golux.woodcraft.me.uk> References: <20110410124605.GB21912@golux.woodcraft.me.uk> Message-ID: Hi Matthew, On Sun, Apr 10, 2011 at 2:46 PM, Matthew Woodcraft wrote: > + ? ?cpu.setup_once() Thanks, fixed. We did not notice this failure because pypy/jit/backend/x86/test/test_zll_random.py provides its own interface over calling test_random.py directly with --backend=x86. A bient?t, Armin. From bea at changemaker.nu Sun Apr 10 22:59:26 2011 From: bea at changemaker.nu (Bea During) Date: Sun, 10 Apr 2011 22:59:26 +0200 Subject: [pypy-dev] Gothenburg sprint and sprint fund In-Reply-To: References: <20110408173451.GA6615@ubuntu> <201104091117.p39BHGsY006157@theraft.openend.se> <201104091152.p39BqDw1009245@theraft.openend.se> <201104091205.p39C5FPP010642@theraft.openend.se> <201104091238.p39Ccr2J013284@theraft.openend.se> Message-ID: <4DA21A2E.5090506@changemaker.nu> Hi there Bartosz SKOWRON skrev 2011-04-09 14:42: > On Sat, Apr 9, 2011 at 2:38 PM, Laura Creighton wrote: > >> That's really too bad. Are you going to Europython? > Yes, I'm going to come for the conference and the PyPy sprint. > At least I will try to come ;-) > _______________________________________________ Just a reminder that this sprint related thread of discussion fits better on the mailinglist we have specifically for these purposes: pypy-sprint at codespeak.net. Cheers Bea From fijall at gmail.com Mon Apr 11 02:01:42 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 11 Apr 2011 02:01:42 +0200 Subject: [pypy-dev] An interesting article on chip design Message-ID: For those interested in hardware/assembler. http://www.lighterra.com/papers/modernmicroprocessors/ It's a good read and fills some of our gaps :) Cheers, fijal From fijall at gmail.com Mon Apr 11 11:53:58 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 11 Apr 2011 11:53:58 +0200 Subject: [pypy-dev] Waf benchmark Message-ID: Hello. I propose the waf benchmark removal. Originally, the idea was that we're slower than CPython for no good reason. Now that this benchmark measures some obscure piece of stdlib time (subprocesses) I don't think it's that necessary. Besides: * the variation between runs is too big, so we don't care * noone was ever remotely interested in speeding this up any opinions? Cheers, fijal From pypy at pocketnix.org Mon Apr 11 13:44:21 2011 From: pypy at pocketnix.org (pypy at pocketnix.org) Date: Mon, 11 Apr 2011 11:44:21 +0000 Subject: [pypy-dev] translationmodules option failing Message-ID: <20110411114421.GC7395@pocketnix.org> Hi just tried to bootstrap myself in the quickest way possible via the --translationmodules option (cmdline below) and encountered an issue with the md5 module which appears to be renamed _md5. the patch below corrects this. while there was an md5 directory present in my tree it only contained untracked pyc files and as such i have also removed this ------------------------------------- ./translate.py -O jit --thread --make-jobs=9 ./targetpypystandalone.py --translationmodules ------------------------------------- --- a/pypy/config/pypyoption.py +++ b/pypy/config/pypyoption.py @@ -39,7 +39,7 @@ translation_modules = default_modules.copy() translation_modules.update(dict.fromkeys( ["fcntl", "rctime", "select", "signal", "_rawffi", "zlib", - "struct", "md5", "cStringIO", "array"])) + "struct", "_md5", "cStringIO", "array"])) ------------------------------------- From pypy at pocketnix.org Mon Apr 11 14:17:39 2011 From: pypy at pocketnix.org (pypy at pocketnix.org) Date: Mon, 11 Apr 2011 12:17:39 +0000 Subject: [pypy-dev] benchunix.py fixes Message-ID: <20110411121739.GD7395@pocketnix.org> Hi bench-unix.py did not work so i did q quick repair job and enhancement on a small bit of the code, included as 3 patches below. i seem to recall someone mentioning that this was depreciated however as a new user it was tempting to use it as it was the first thing i saw and is a nice quick test for getting a rough guide of the relative performance of several versions of pypy i have generated currently contemplating a rewrite of the code so let me know if there is something better or if someone else is working on something ---------------------------------------------------- Add newer python implementations and benchmark system default interpreter --- a/pypy/translator/goal/bench-unix.py +++ b/pypy/translator/goal/bench-unix.py @@ -102,7 +102,7 @@ ref_rich, ref_stone = None, None # for exe in '/usr/local/bin/python2.5 python2.4 python2.3'.split(): - for exe in 'python2.4 python2.3'.split(): + for exe in 'python2.7 python2.6 python2.4 python2.3 python'.split(): v = os.popen(exe + ' -c "import sys;print sys.version.split()[0]"').rea if not v: continue ------------------------------------------------------------------------------ Discard missing interpreters --- a/pypy/translator/goal/bench-unix.py +++ b/pypy/translator/goal/bench-unix.py @@ -102,7 +102,13 @@ ref_rich, ref_stone = None, None # for exe in '/usr/local/bin/python2.5 python2.4 python2.3'.split(): - for exe in 'python2.4 python2.3'.split(): + for exe in 'python2.7 python2.6 python2.4 python2.3 python'.split(): + path = os.environ.get("PATH", "") + path = [x + os.sep + exe for x in path.split(os.pathsep) if exe in os.l + if len(path) > 0: + exe = path[0] + else: + continue v = os.popen(exe + ' -c "import sys;print sys.version.split()[0]"').rea if not v: continue ------------------------------------------------------------------------------ Fix off by one --- a/pypy/translator/goal/bench-unix.py +++ b/pypy/translator/goal/bench-unix.py @@ -85,7 +85,7 @@ if os.path.isdir(exe) or exe.endswith('.jar'): continue try: - exes.append( (exe.split('-')[2], exe) ) + exes.append( (exe.split('-')[1], exe) ) except: pass #skip filenames without version number exes.sort() ------------------------------------------------------------------------------ From pypy at pocketnix.org Mon Apr 11 14:45:41 2011 From: pypy at pocketnix.org (pypy at pocketnix.org) Date: Mon, 11 Apr 2011 12:45:41 +0000 Subject: [pypy-dev] File overwriting (--output flag to translate.py) Message-ID: <20110411124541.GE7395@pocketnix.org> Hi again i have been compiling a bunch of different pypy instances with different levels of optimizations and features and found that if i run pypy-c from the current directory and dont specify a new output filename it will attempt to and fail to overwrite pypy-c due to the file being "in use". unfortunately the exception generated is in the shutil lib from mem and the error message/exception does not give away immediately what the cause is which can lead to some frustration on some of the longer compiles ;) its quick and dirty and i don't mind if it gets changed at all. not 100% sure what the correct way to report an error and abandon is other than what everything else does (uncaught exception to a pdb shell) --- a/pypy/translator/goal/translate.py +++ b/pypy/translator/goal/translate.py @@ -285,6 +285,10 @@ elif drv.exe_name is None and '__name__' in targetspec_dic: drv.exe_name = targetspec_dic['__name__'] + '-%(backend)s' + # Ensure the file does not exisit else we fail at end of translation + if os.path.isfile(drv.exe_name): + raise ValueError('File "' + drv.exe_name+ '" already exisits (--output)') + goals = translateconfig.goals try: drv.proceed(goals) From me at samuelreis.de Mon Apr 11 14:38:44 2011 From: me at samuelreis.de (Samuel Reis) Date: Mon, 11 Apr 2011 14:38:44 +0200 Subject: [pypy-dev] Licensing of PyPy Speed Logo Artwork Message-ID: <5840850C-21C9-4356-A3DD-9992124AD0CC@samuelreis.de> Hello everybody. As fijal asked me to do, I will now clarify the licensing of the PyPy Logo Artwork that was provided by me and is currently in use at speed.pypy.org. Herewith I declare this artwork to be subject to the terms of the Creative Commons Attribution-Share Alike license (http://creativecommons.org/licenses/by-sa/3.0/). I view the aspect of attribution to be fulfilled by the mention of my name in the LICENSE file of the PyPy source distribution. I wish you guys good progress and success, so that we all can see a compliant Python that is gaining more and more speed. Greetings, Samuel -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110411/1592a0f1/attachment.htm From bea at changemaker.nu Mon Apr 11 14:47:28 2011 From: bea at changemaker.nu (Bea During) Date: Mon, 11 Apr 2011 14:47:28 +0200 Subject: [pypy-dev] Pre sprint beer session Sunday 24th of April Message-ID: <4DA2F860.6030606@changemaker.nu> ( emailing this list since I was not allowed to post to pypy-sprint and people don?t seem to use it?) Hi there Since I am travelling to Japan on the 25th of April I will miss the Gothenburg sprint (that was not the original plan but my family had to move our trip because of what happened there). I still would like to meet up so I invite you to a pre sprint beer session at Bishops Arms pub in central Gothenburg, Sunday 24th of April, starting from 18:00. The Bishop Arms pub is a very nice, quiet pub (library inspired but cosy) and they have a good selection of beers and whiskey. Here is a map: http://www.bishopsarms.com/Goteborg_Park/Presentation This way I get to meet most of you - and drink beer! Talk about eating the cake and having it ;-) Cheers Bea p.s. Anto - will miss you there d.s From lac at openend.se Mon Apr 11 15:40:11 2011 From: lac at openend.se (Laura Creighton) Date: Mon, 11 Apr 2011 15:40:11 +0200 Subject: [pypy-dev] thinking about the EuroPython sprint Message-ID: <201104111340.p3BDeB7t019802@theraft.openend.se> 2 days after the conference is not a lot of time. Do we want to rent some space to have a longer sprint? Or is it too late, people will already have booked their plane tickets, or ... Laura From anto.cuni at gmail.com Mon Apr 11 20:27:37 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Mon, 11 Apr 2011 20:27:37 +0200 Subject: [pypy-dev] Pre sprint beer session Sunday 24th of April In-Reply-To: <4DA2F860.6030606@changemaker.nu> References: <4DA2F860.6030606@changemaker.nu> Message-ID: <4DA34819.1080007@gmail.com> On 11/04/11 14:47, Bea During wrote: > I still would like to meet up so I invite you to a pre sprint beer > session at Bishops Arms pub in central Gothenburg, Sunday 24th of April, > starting from 18:00. [cut] > p.s. Anto - will miss you there d.s yeah, unfortunately I won't be able to make it :-( Have fun! ciao, Anto From anto.cuni at gmail.com Mon Apr 11 20:34:42 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Mon, 11 Apr 2011 20:34:42 +0200 Subject: [pypy-dev] The JVM backend and Jython In-Reply-To: References: Message-ID: <4DA349C2.3080403@gmail.com> Hi Frank, On 30/03/11 04:40, fwierzbicki at gmail.com wrote: [cut] > So to my question - just how broken is the JVM backend? Are there > workarounds that would allow the Java code to get generated? so, now the jvm (and cli) translation works again. You can just type ./translate.py -b jvm, and the fish the relevant .class/.j files from /tmp/usession-default-*/pypy. ciao, Anto From matthew at woodcraft.me.uk Mon Apr 11 20:52:28 2011 From: matthew at woodcraft.me.uk (Matthew Woodcraft) Date: Mon, 11 Apr 2011 19:52:28 +0100 Subject: [pypy-dev] An interesting article on chip design In-Reply-To: References: Message-ID: On 2011-04-11 01:01, Maciej Fijalkowski wrote: > For those interested in hardware/assembler. > > http://www.lighterra.com/papers/modernmicroprocessors/ For specifics of x86 processors, http://www.agner.org/optimize/ is a good source of information (particularly microarchitecture.pdf and optimizing_assembly.pdf). -M- From arigo at tunes.org Mon Apr 11 21:16:30 2011 From: arigo at tunes.org (Armin Rigo) Date: Mon, 11 Apr 2011 21:16:30 +0200 Subject: [pypy-dev] Waf benchmark In-Reply-To: References: Message-ID: Hi Maciej, On Mon, Apr 11, 2011 at 11:53 AM, Maciej Fijalkowski wrote: > I propose the waf benchmark removal. Given that its speed is at "1 for CPython, 1 for PyPy without JIT, 1 for PyPy with JIT", it seems rather pointless indeed, at least for us. Armin From arigo at tunes.org Mon Apr 11 21:19:22 2011 From: arigo at tunes.org (Armin Rigo) Date: Mon, 11 Apr 2011 21:19:22 +0200 Subject: [pypy-dev] translationmodules option failing In-Reply-To: <20110411114421.GC7395@pocketnix.org> References: <20110411114421.GC7395@pocketnix.org> Message-ID: Hi, On Mon, Apr 11, 2011 at 1:44 PM, wrote: > - ? ? "struct", "md5", "cStringIO", "array"])) > + ? ? "struct", "_md5", "cStringIO", "array"])) Thanks! Applied. Armin From arigo at tunes.org Mon Apr 11 21:27:09 2011 From: arigo at tunes.org (Armin Rigo) Date: Mon, 11 Apr 2011 21:27:09 +0200 Subject: [pypy-dev] File overwriting (--output flag to translate.py) In-Reply-To: <20110411124541.GE7395@pocketnix.org> References: <20110411124541.GE7395@pocketnix.org> Message-ID: Re-Hi, On Mon, Apr 11, 2011 at 2:45 PM, wrote: > + ? ? ? ?# Ensure the file does not exisit else we fail at end of translation > + ? ? ? ?if os.path.isfile(drv.exe_name): > + ? ? ? ? ? ?raise ValueError('File "' + drv.exe_name+ '" already exisits (--output)') Sadly everyone so far has his own additional hacks to categorize multiple translated versions. Mine is to ignore the pypy-c entirely and copy the executable from the /tmp, after it has been produced there. I also copy the C sources (but not the other files produced by gcc). Anyway, my point is that the particular change you are proposing would actually harm me, because I always have a pypy-c and I'm fine if it gets overwritten by every translation :-) We need to think of some better solution... Armin From arigo at tunes.org Mon Apr 11 21:44:07 2011 From: arigo at tunes.org (Armin Rigo) Date: Mon, 11 Apr 2011 21:44:07 +0200 Subject: [pypy-dev] Pre sprint beer session Sunday 24th of April In-Reply-To: <4DA34819.1080007@gmail.com> References: <4DA2F860.6030606@changemaker.nu> <4DA34819.1080007@gmail.com> Message-ID: Hi Bea, On Mon, Apr 11, 2011 at 8:27 PM, Antonio Cuni wrote: >> session at Bishops Arms pub in central Gothenburg, Sunday 24th of April, Good, it will be nice to see you again Bea! Armin From arigo at tunes.org Mon Apr 11 21:46:25 2011 From: arigo at tunes.org (Armin Rigo) Date: Mon, 11 Apr 2011 21:46:25 +0200 Subject: [pypy-dev] thinking about the EuroPython sprint In-Reply-To: <201104111340.p3BDeB7t019802@theraft.openend.se> References: <201104111340.p3BDeB7t019802@theraft.openend.se> Message-ID: Hi Laura, On Mon, Apr 11, 2011 at 3:40 PM, Laura Creighton wrote: > 2 days after the conference is not a lot of time. ?Do we want to rent some > space to have a longer sprint? ?Or is it too late, people will already > have booked their plane tickets, or ... As far as I'm concerned, I think a sprint should be a bit longer to be useful. Wasn't there also talk about having the sprint (or *a* sprint) done before the conference? Armin From stefan_ml at behnel.de Mon Apr 11 23:02:32 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 11 Apr 2011 23:02:32 +0200 Subject: [pypy-dev] Waf benchmark In-Reply-To: References: Message-ID: Maciej Fijalkowski, 11.04.2011 11:53: > I propose the waf benchmark removal. > > Originally, the idea was that we're slower than CPython for no good > reason. Now that this benchmark measures some obscure piece of stdlib > time (subprocesses) I don't think it's that necessary. > > Besides: > > * the variation between runs is too big, so we don't care > * noone was ever remotely interested in speeding this up > > any opinions? Despite the relatively large variations, Cython runs this benchmark persistently ~1/3 faster than CPython 2.7 for me - minus the currently missing support for "__file__", which is used at build time here. So my vote would be to leave it in, maybe someone has an incentive to speed this up once you have bars up for Cython. :) Stefan From alex.gaynor at gmail.com Mon Apr 11 23:04:56 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Mon, 11 Apr 2011 17:04:56 -0400 Subject: [pypy-dev] Waf benchmark In-Reply-To: References: Message-ID: On Mon, Apr 11, 2011 at 5:02 PM, Stefan Behnel wrote: > Maciej Fijalkowski, 11.04.2011 11:53: > > I propose the waf benchmark removal. > > > > Originally, the idea was that we're slower than CPython for no good > > reason. Now that this benchmark measures some obscure piece of stdlib > > time (subprocesses) I don't think it's that necessary. > > > > Besides: > > > > * the variation between runs is too big, so we don't care > > * noone was ever remotely interested in speeding this up > > > > any opinions? > > Despite the relatively large variations, Cython runs this benchmark > persistently ~1/3 faster than CPython 2.7 for me - minus the currently > missing support for "__file__", which is used at build time here. So my > vote would be to leave it in, maybe someone has an incentive to speed this > up once you have bars up for Cython. :) > > Stefan > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > Personally I'd be happier if it was a bit more of a microbenchmark, it's apparently a macro-benchmark, of subprocess ATM, which makes no sense really :) Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110411/237a6e91/attachment.htm From lac at openend.se Tue Apr 12 00:13:13 2011 From: lac at openend.se (Laura Creighton) Date: Tue, 12 Apr 2011 00:13:13 +0200 Subject: [pypy-dev] thinking about the EuroPython sprint In-Reply-To: Message from Armin Rigo of "Mon, 11 Apr 2011 21:46:25 +0200." References: <201104111340.p3BDeB7t019802@theraft.openend.se> Message-ID: <201104112213.p3BMDDEJ002041@theraft.openend.se> In a message of Mon, 11 Apr 2011 21:46:25 +0200, Armin Rigo writes: >Hi Laura, > >On Mon, Apr 11, 2011 at 3:40 PM, Laura Creighton wrote: >> 2 days after the conference is not a lot of time. =A0Do we want to rent > s= >ome >> space to have a longer sprint? =A0Or is it too late, people will alread >y >> have booked their plane tickets, or ... > >As far as I'm concerned, I think a sprint should be a bit longer to be >useful. Wasn't there also talk about having the sprint (or *a* >sprint) done before the conference? yes, I was hoping to catch Alex Gaynor for that, as he is attending EuroDjango earlier the same month. My point is that we need to decide what we are doing very, very soon. else everybody will have their plane tickets already, since we will have to announce what we are doing differently. Laura > > >Armin From arigo at tunes.org Tue Apr 12 08:52:13 2011 From: arigo at tunes.org (Armin Rigo) Date: Tue, 12 Apr 2011 08:52:13 +0200 Subject: [pypy-dev] thinking about the EuroPython sprint In-Reply-To: <201104112213.p3BMDDEJ002041@theraft.openend.se> References: <201104111340.p3BDeB7t019802@theraft.openend.se> <201104112213.p3BMDDEJ002041@theraft.openend.se> Message-ID: Hi Laura, On Tue, Apr 12, 2011 at 12:13 AM, Laura Creighton wrote: > yes, I was hoping to catch Alex Gaynor for that, as he is attending > EuroDjango earlier the same month. ? My point is that we need to decide > what we are doing very, very soon. ?else everybody will have their > plane tickets already, since we will have to announce what we are > doing differently. Then what about a short week's sprint before the conference, plus the two days after? Armin From david at silveregg.co.jp Tue Apr 12 08:52:49 2011 From: david at silveregg.co.jp (David) Date: Tue, 12 Apr 2011 15:52:49 +0900 Subject: [pypy-dev] Waf benchmark In-Reply-To: References: Message-ID: <4DA3F6C1.8070601@silveregg.co.jp> On 04/12/2011 06:02 AM, Stefan Behnel wrote: > Maciej Fijalkowski, 11.04.2011 11:53: >> I propose the waf benchmark removal. >> >> Originally, the idea was that we're slower than CPython for no good >> reason. Now that this benchmark measures some obscure piece of stdlib >> time (subprocesses) I don't think it's that necessary. >> >> Besides: >> >> * the variation between runs is too big, so we don't care >> * noone was ever remotely interested in speeding this up >> >> any opinions? > > Despite the relatively large variations, Cython runs this benchmark > persistently ~1/3 faster than CPython 2.7 for me - minus the currently > missing support for "__file__", which is used at build time here. So my > vote would be to leave it in, maybe someone has an incentive to speed this > up once you have bars up for Cython. :) I could look at the waf script used for the benchmark so that it depends more on python performances. One thing which is annoying in all the python build tools (waf, scons, etc...) is that when you use multiple builds with python-based build functions, it is slower than with a single job. That's a bottleneck for the waf-based build of numpy, at least. cheers, David From anto.cuni at gmail.com Tue Apr 12 09:27:07 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 12 Apr 2011 09:27:07 +0200 Subject: [pypy-dev] thinking about the EuroPython sprint In-Reply-To: References: <201104111340.p3BDeB7t019802@theraft.openend.se> <201104112213.p3BMDDEJ002041@theraft.openend.se> Message-ID: <4DA3FECB.3080702@gmail.com> On 12/04/11 08:52, Armin Rigo wrote: > Then what about a short week's sprint before the conference, plus the > two days after? if people are interested, I might try to organize a sprint in Genova or surrounding (which is ~3h away from florence by train). However, I cannot assure that I'll be able to find a suitable place, because I see only two options: - ask the uni (but there are not really many rooms suitable for a sprint -- just one or two, and I think it'll be hard to reserve them for a whole week) - find a place "a la leysin", i.e. a hotel which gives also us a room for the sprint. However, middle of june is already vacation time here, so it is possible that hotels are full anyway and prefer to have "normal" visitors instead of nerds which stay there all the time :-) ciao, Anto From pypy at pocketnix.org Tue Apr 12 10:57:52 2011 From: pypy at pocketnix.org (pypy at pocketnix.org) Date: Tue, 12 Apr 2011 08:57:52 +0000 Subject: [pypy-dev] File overwriting (--output flag to translate.py) In-Reply-To: References: <20110411124541.GE7395@pocketnix.org> Message-ID: <20110412085751.GF7395@pocketnix.org> On Mon, Apr 11, 2011 at 09:27:09PM +0200, Armin Rigo wrote: > > Sadly everyone so far has his own additional hacks to categorize > multiple translated versions. Mine is to ignore the pypy-c entirely > and copy the executable from the /tmp, after it has been produced > there. I also copy the C sources (but not the other files produced by > gcc). Anyway, my point is that the particular change you are > proposing would actually harm me, because I always have a pypy-c and > I'm fine if it gets overwritten by every translation :-) > > We need to think of some better solution... > > Armin here is an updated version that specifically checks if the destination specified by the --output flag is the same as the running interpreter, this matches the error i was having more closely and should hopefully not interfere with other users work flows of course i am assuming here that you are not relying on the pdb shell as a notification to copy the file over, if you do feel free to ignore this updated patch - Da_Blitz ------------------------------------------------------------- Double check to ensure we are not overwriting the current interpreter --- a/pypy/translator/goal/translate.py +++ b/pypy/translator/goal/translate.py @@ -285,6 +285,10 @@ elif drv.exe_name is None and '__name__' in targetspec_dic: drv.exe_name = targetspec_dic['__name__'] + '-%(backend)s' + # Double check to ensure we are not overwriting the current interpreter + if os.path.realpath(drv.exe_name) == sys.executable: + raise ValueError('File "' + drv.exe_name+ '" already exisits (--output)') + goals = translateconfig.goals try: drv.proceed(goals) From cfbolz at gmx.de Tue Apr 12 11:36:08 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Tue, 12 Apr 2011 11:36:08 +0200 Subject: [pypy-dev] File overwriting (--output flag to translate.py) In-Reply-To: <20110411124541.GE7395@pocketnix.org> References: <20110411124541.GE7395@pocketnix.org> Message-ID: <4DA41D08.80803@gmx.de> On 04/11/2011 02:45 PM, pypy at pocketnix.org wrote: > Hi again > > i have been compiling a bunch of different pypy instances with different > levels of optimizations and features and found that if i run pypy-c > from the current directory and dont specify a new output filename it > will attempt to and fail to overwrite pypy-c due to the file being "in > use". unfortunately the exception generated is in the shutil lib from > mem and the error message/exception does not give away immediately what > the cause is which can lead to some frustration on some of the longer > compiles ;) Even if the copy fails, you can always fish the executable from the temp directory, under the name $TEMPDIR/usession-*/testing_1/pypy-c so you don't lose the previously translated executable. Carl Friedrich From pypy at pocketnix.org Tue Apr 12 12:34:50 2011 From: pypy at pocketnix.org (pypy at pocketnix.org) Date: Tue, 12 Apr 2011 10:34:50 +0000 Subject: [pypy-dev] translationmodules option failing In-Reply-To: References: <20110411114421.GC7395@pocketnix.org> Message-ID: <20110412103450.GG7395@pocketnix.org> On Mon, Apr 11, 2011 at 09:19:22PM +0200, Armin Rigo wrote: > Hi, > > On Mon, Apr 11, 2011 at 1:44 PM, wrote: > > - ? ? "struct", "md5", "cStringIO", "array"])) > > + ? ? "struct", "_md5", "cStringIO", "array"])) > > Thanks! Applied. > > Armin seems i missed the ctypes dependence on the threading module (i always passed --thread to translation.py) while the first pypy-c would compile correctly, attempting to translate a second pypy interpreter would fail with an exception related to ctypes with the patch below executing "./pypy-c ./translate.py -O jit ./targetpypystandalone.py --translationmodules" will work without passing any more arguments in case anyone is interested i included the last few lines of the traceback to the end of the email from the second run that fails - ? ? "struct", "_md5", "cStringIO", "array"])) + ? ? "struct", "_md5", "cStringIO", "array", "thread"])) ------------------------------ File "/home/dablitz/code/pypy/pypy/rpython/lltypesystem/ll2ctypes.py", line 247, in get_ctypes_type cls = build_new_ctypes_type(T, delayed_builders) File "/home/dablitz/code/pypy/pypy/rpython/lltypesystem/ll2ctypes.py", line 284, in build_new_ctypes_type _setup_ctypes_cache() File "/home/dablitz/code/pypy/pypy/rpython/lltypesystem/ll2ctypes.py", line 92, in _setup_ctypes_cache lltype.Signed: ctypes.c_long, AttributeError: 'NoneType' object has no attribute 'c_long' From arigo at tunes.org Tue Apr 12 12:40:11 2011 From: arigo at tunes.org (Armin Rigo) Date: Tue, 12 Apr 2011 12:40:11 +0200 Subject: [pypy-dev] translationmodules option failing In-Reply-To: <20110412103450.GG7395@pocketnix.org> References: <20110411114421.GC7395@pocketnix.org> <20110412103450.GG7395@pocketnix.org> Message-ID: Re-hi, On Tue, Apr 12, 2011 at 12:34 PM, wrote: > - ? ? "struct", "_md5", "cStringIO", "array"])) > + ? ? "struct", "_md5", "cStringIO", "array", "thread"])) Uh, we can't import ctypes if we have no thread module? That looks obscure and should be fixed... Armin From arigo at tunes.org Tue Apr 12 21:07:36 2011 From: arigo at tunes.org (Armin Rigo) Date: Tue, 12 Apr 2011 21:07:36 +0200 Subject: [pypy-dev] translationmodules option failing In-Reply-To: References: <20110411114421.GC7395@pocketnix.org> <20110412103450.GG7395@pocketnix.org> Message-ID: Hi, On Tue, Apr 12, 2011 at 12:40 PM, Armin Rigo wrote: > Uh, we can't import ctypes if we have no thread module? ?That looks > obscure and should be fixed... Fixed. Now a "--translationmodules" translation should, as originally planned, have no threads but ctypes should import on it anyway -- as well hopefully as all other modules needed by translate.py; feel free to report if some are still missing. (Note that the fix is purely app-level code, which means that you don't need to retranslate to see the benefit.) A bient?t, Armin. From anto.cuni at gmail.com Wed Apr 13 12:20:26 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed, 13 Apr 2011 12:20:26 +0200 Subject: [pypy-dev] Possible sprint in Genova before/after Europython Message-ID: <4DA578EA.2080108@gmail.com> Hi all, as we were discussing yesterday in another thread, the post-europython sprint will be only two days long, and so we might want to have a longer one either before or after europython. I am considering organizing one in my place. It could be either in Genova or more preferably in some other town in the nearby of the italian riviera. The place is ~3hr away from Florence by train. Googling images for "pegli" or "arenzano" (i.e., two of the towns we could go to) shows pictures which (I think) should encourage people to come :-) http://tinyurl.com/655p2at http://tinyurl.com/67q67r3 My idea is to find a hotel that gives us a sprinting room and internet in exchange of N people sleeping there, as we do in leysin. But before asking them, I need to have a rough idea about which value of "N" we are talking about (of course the higher the more interesting for them it is). Thus, I ask everybody who is potentially/likely interested in coming to let me know. Also, would you prefer to do it before or after europython? ciao, Anto From fijall at gmail.com Wed Apr 13 12:27:18 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 13 Apr 2011 12:27:18 +0200 Subject: [pypy-dev] Possible sprint in Genova before/after Europython In-Reply-To: <4DA578EA.2080108@gmail.com> References: <4DA578EA.2080108@gmail.com> Message-ID: On Wed, Apr 13, 2011 at 12:20 PM, Antonio Cuni wrote: > Hi all, > > as we were discussing yesterday in another thread, the post-europython sprint > will be only two days long, and so we might want to have a longer one either > before or after europython. > > I am considering organizing one in my place. ?It could be either in Genova or > more preferably in some other town in the nearby of the italian riviera. > The place is ~3hr away from Florence by train. > > Googling images for "pegli" or "arenzano" (i.e., two of the towns we could go > to) shows pictures which (I think) should encourage people to come :-) > http://tinyurl.com/655p2at > http://tinyurl.com/67q67r3 > > My idea is to find a hotel that gives us a sprinting room and internet in > exchange of N people sleeping there, as we do in leysin. > > But before asking them, I need to have a rough idea about which value of "N" > we are talking about (of course the higher the more interesting for them it is). > > Thus, I ask everybody who is potentially/likely interested in coming to let me > know. > > Also, would you prefer to do it before or after europython? > > ciao, > Anto > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > Not that I can't google myself, but those links are broken From anto.cuni at gmail.com Wed Apr 13 12:29:40 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed, 13 Apr 2011 12:29:40 +0200 Subject: [pypy-dev] Possible sprint in Genova before/after Europython In-Reply-To: References: <4DA578EA.2080108@gmail.com> Message-ID: <4DA57B14.6030806@gmail.com> On 13/04/11 12:27, Maciej Fijalkowski wrote: > Not that I can't google myself, but those links are broken uhm, indeed. These seems to work :-) http://bit.ly/e6vHkh http://bit.ly/fkHwMu From lac at openend.se Wed Apr 13 14:21:50 2011 From: lac at openend.se (Laura Creighton) Date: Wed, 13 Apr 2011 14:21:50 +0200 Subject: [pypy-dev] Possible sprint in Genova before/after Europython In-Reply-To: Message from Antonio Cuni of "Wed, 13 Apr 2011 12:20:26 +0200." <4DA578EA.2080108@gmail.com> References: <4DA578EA.2080108@gmail.com> Message-ID: <201104131221.p3DCLpg4021192@theraft.openend.se> In a message of Wed, 13 Apr 2011 12:20:26 +0200, Antonio Cuni writes: >Also, would you prefer to do it before or after europython? I am coming, and both times are fine for me. I am going to be spending the week after Europython in Italy somewhere anyway, prior to going kayaking in Sicily. Laura (who is speaking for Jacob here too) >ciao, >Anto From jacob at openend.se Wed Apr 13 14:53:26 2011 From: jacob at openend.se (Jacob =?iso-8859-1?q?Hall=E9n?=) Date: Wed, 13 Apr 2011 14:53:26 +0200 Subject: [pypy-dev] Possible sprint in Genova before/after Europython In-Reply-To: <201104131221.p3DCLpg4021192@theraft.openend.se> References: <4DA578EA.2080108@gmail.com> <201104131221.p3DCLpg4021192@theraft.openend.se> Message-ID: <201104131453.26813.jacob@openend.se> Wednesday 13 April 2011 you wrote: > In a message of Wed, 13 Apr 2011 12:20:26 +0200, Antonio Cuni writes: > >Also, would you prefer to do it before or after europython? > > I am coming, and both times are fine for me. I am going to be > spending the week after Europython in Italy somewhere anyway, prior > to going kayaking in Sicily. Corsica, not Sicily. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. Url : http://codespeak.net/pipermail/pypy-dev/attachments/20110413/85399ad9/attachment.pgp From anto.cuni at gmail.com Wed Apr 13 15:14:20 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed, 13 Apr 2011 15:14:20 +0200 Subject: [pypy-dev] Possible sprint in Genova before/after Europython In-Reply-To: <201104131453.26813.jacob@openend.se> References: <4DA578EA.2080108@gmail.com> <201104131221.p3DCLpg4021192@theraft.openend.se> <201104131453.26813.jacob@openend.se> Message-ID: <4DA5A1AC.6050305@gmail.com> On 13/04/11 14:53, Jacob Hall?n wrote: > Wednesday 13 April 2011 you wrote: >> In a message of Wed, 13 Apr 2011 12:20:26 +0200, Antonio Cuni writes: >>> Also, would you prefer to do it before or after europython? >> >> I am coming, and both times are fine for me. I am going to be >> spending the week after Europython in Italy somewhere anyway, prior >> to going kayaking in Sicily. > > Corsica, not Sicily. uhm, how do you go there? If by boat, it's very likely that you'll start by genova or savona (which is also close, ~50 km) From jacob at openend.se Wed Apr 13 16:42:04 2011 From: jacob at openend.se (Jacob =?iso-8859-1?q?Hall=E9n?=) Date: Wed, 13 Apr 2011 16:42:04 +0200 Subject: [pypy-dev] Possible sprint in Genova before/after Europython In-Reply-To: <4DA5A1AC.6050305@gmail.com> References: <4DA578EA.2080108@gmail.com> <201104131453.26813.jacob@openend.se> <4DA5A1AC.6050305@gmail.com> Message-ID: <201104131642.04792.jacob@openend.se> Wednesday 13 April 2011 you wrote: > On 13/04/11 14:53, Jacob Hall?n wrote: > > Wednesday 13 April 2011 you wrote: > >> In a message of Wed, 13 Apr 2011 12:20:26 +0200, Antonio Cuni writes: > >>> Also, would you prefer to do it before or after europython? > >> > >> I am coming, and both times are fine for me. I am going to be > >> spending the week after Europython in Italy somewhere anyway, prior > >> to going kayaking in Sicily. > > > > Corsica, not Sicily. > > uhm, how do you go there? If by boat, it's very likely that you'll start by > genova or savona (which is also close, ~50 km) We have looked at leaving from Livorno by boat, but there are other alternatives. Flying seems to out of the question, shortest way seems to be to go to Geneva to change planes. Jacob -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part. Url : http://codespeak.net/pipermail/pypy-dev/attachments/20110413/f0020d17/attachment.pgp From lac at openend.se Wed Apr 13 19:32:03 2011 From: lac at openend.se (Laura Creighton) Date: Wed, 13 Apr 2011 19:32:03 +0200 Subject: [pypy-dev] This in from Jesse Noller Message-ID: <201104131732.p3DHW3X1017075@theraft.openend.se> ------- Forwarded Message Return-Path: jnoller at gmail.com Delivery-Date: Wed Apr 13 18:54:12 2011 Subject: Re: Mentor for Py3 benchmarking To: the-fellowship-of-the-packaging at googlegroups.com, Maciej Fijalkowski , Laura Creighton Cc: Arc Riley Content-Type: text/plain; charset=ISO-8859-1 Also, reach out to the PyPy team. They know more about speed.pypy.org than anyone else, and would be best suited to mentor. Python-Dev and PyPy-Dev are going to be the best avenues. On Tue, Apr 12, 2011 at 11:33 AM, Arc Riley wrote: > We have a number of students who proposed to port PyPy's benchmarking sui= te > to Python3 to run on speed.python.org, we don't have a mentor for these a= t > the moment. > > Would anyone here (pref previous GSoC mentor/student) like to take one of > these on? > > We have until Monday (4/18) to evaluate students, get patches/blogs/etc > taken care of, and all mentors assigned. If there are people here who want > to mentor talk to either Tarek (for packaging) or Martin v. L?wis (for > python-core).If you're an existing python-dev contributor we could > especially use your help. > - ------- End of Forwarded Message From hakan at debian.org Thu Apr 14 14:53:19 2011 From: hakan at debian.org (Hakan Ardo) Date: Thu, 14 Apr 2011 14:53:19 +0200 Subject: [pypy-dev] [pypy-svn] pypy default: port test_intbound_addsub_ge to test_pypy_c_new In-Reply-To: <20110414124402.CD7472A204D@codespeak.net> References: <20110414124402.CD7472A204D@codespeak.net> Message-ID: On Thu, Apr 14, 2011 at 2:44 PM, antocuni wrote: > Author: Antonio Cuni > Branch: > Changeset: r43345:fd3f23ae8324 > Date: 2011-04-14 14:42 +0200 > http://bitbucket.org/pypy/pypy/changeset/fd3f23ae8324/ > > Log: ? ?port test_intbound_addsub_ge to test_pypy_c_new > > diff --git a/pypy/module/pypyjit/test_pypy_c/test_pypy_c_new.py b/pypy/module/pypyjit/test_pypy_c/test_pypy_c_new.py > --- a/pypy/module/pypyjit/test_pypy_c/test_pypy_c_new.py > +++ b/pypy/module/pypyjit/test_pypy_c/test_pypy_c_new.py > @@ -1194,3 +1194,33 @@ > ? ? ? ? ? ? --TICK-- > ? ? ? ? ? ? jump(p0, p1, p2, p3, i11, i13, descr=) > ? ? ? ? """) > + > + ? ?def test_intbound_addsub_ge(self): > + ? ? ? ?def main(n): > + ? ? ? ? ? ?i, a, b = 0, 0, 0 > + ? ? ? ? ? ?while i < n: > + ? ? ? ? ? ? ? ?if i + 5 >= 5: > + ? ? ? ? ? ? ? ? ? ?a += 1 > + ? ? ? ? ? ? ? ?if i - 1 >= -1: > + ? ? ? ? ? ? ? ? ? ?b += 1 > + ? ? ? ? ? ? ? ?i += 1 > + ? ? ? ? ? ?return (a, b) > + ? ? ? ?# > + ? ? ? ?log = self.run(main, [300], threshold=200) > + ? ? ? ?assert log.result == (300, 300) > + ? ? ? ?loop, = log.loops_by_filename(self.filepath) > + ? ? ? ?assert loop.match(""" > + ? ? ? ? ? ?i10 = int_lt(i8, i9) > + ? ? ? ? ? ?guard_true(i10, descr=...) > + ? ? ? ?# XXX: why do we need ovf check here? If we put a literal "300" > + ? ? ? ?# instead of "n", it disappears With n==sys.maxint, the operation i+5 would be the one overflowing. -- H?kan Ard? From anto.cuni at gmail.com Thu Apr 14 15:01:17 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu, 14 Apr 2011 15:01:17 +0200 Subject: [pypy-dev] [pypy-svn] pypy default: port test_intbound_addsub_ge to test_pypy_c_new In-Reply-To: References: <20110414124402.CD7472A204D@codespeak.net> Message-ID: <4DA6F01D.8020505@gmail.com> Hi Hakan, On 14/04/11 14:53, Hakan Ardo wrote: >> + def test_intbound_addsub_ge(self): >> + def main(n): >> + i, a, b = 0, 0, 0 >> + while i < n: >> + if i + 5 >= 5: >> + a += 1 >> + if i - 1 >= -1: >> + b += 1 >> + i += 1 >> + return (a, b) >> + # >> + log = self.run(main, [300], threshold=200) >> + assert log.result == (300, 300) >> + loop, = log.loops_by_filename(self.filepath) >> + assert loop.match(""" >> + i10 = int_lt(i8, i9) >> + guard_true(i10, descr=...) >> + # XXX: why do we need ovf check here? If we put a literal "300" >> + # instead of "n", it disappears > > With n==sys.maxint, the operation i+5 would be the one overflowing. yes, you are right of course. I was just thinking nonsense, but I realized only after I asked the question :-). Of course the ovf check needs to be there because we don't specialize the loop on the value of n. Although it might be cool to be able to do promotion at applevel, for those who really want :-) ciao, Anto From hakan at debian.org Thu Apr 14 16:01:09 2011 From: hakan at debian.org (Hakan Ardo) Date: Thu, 14 Apr 2011 16:01:09 +0200 Subject: [pypy-dev] [pypy-svn] pypy default: port test_intbound_addsub_ge to test_pypy_c_new In-Reply-To: <4DA6F01D.8020505@gmail.com> References: <20110414124402.CD7472A204D@codespeak.net> <4DA6F01D.8020505@gmail.com> Message-ID: On Thu, Apr 14, 2011 at 3:01 PM, Antonio Cuni wrote: > > Of course the ovf check needs to be there because we don't specialize the loop > on the value of n. Although it might be cool to be able to do promotion at > applevel, for those who really want :-) Well, you can actually (sort of): def main(n): i, a = 0, 0 exec """def promote(n): assert n==%d""" % n while i < n: promote(n) a += i+5 i += 1 return a With this I get two extra operations in the loop: i11 = ptr_eq(ConstPtr(ptr10), p7) guard_false(i11, descr=) [p1, p0, p2, p3, p4, i5, i6] but p7 is loop-invariant so they should be easy to get rid of. I don't know why they are not already... -- H?kan Ard? From hakan at debian.org Thu Apr 14 16:44:27 2011 From: hakan at debian.org (Hakan Ardo) Date: Thu, 14 Apr 2011 16:44:27 +0200 Subject: [pypy-dev] [pypy-svn] pypy default: port test_intbound_addsub_ge to test_pypy_c_new In-Reply-To: References: <20110414124402.CD7472A204D@codespeak.net> <4DA6F01D.8020505@gmail.com> Message-ID: Second though, that will recompile on every call, but if we cache the promote functions: def main(n, promoters={}): i, a = 0, 0 if n not in promoters: exec """def promote(n): assert n==%d""" % n promoters[n] = promote else: promote = promoters[n] while i < n: promote(n) a += i+5 i += 1 return a we will actualy get rid of the extra ptr_eq and guard_false too... On Thu, Apr 14, 2011 at 4:01 PM, Hakan Ardo wrote: > On Thu, Apr 14, 2011 at 3:01 PM, Antonio Cuni wrote: >> >> Of course the ovf check needs to be there because we don't specialize the loop >> on the value of n. Although it might be cool to be able to do promotion at >> applevel, for those who really want :-) > > Well, you can actually (sort of): > > ? ?def main(n): > ? ? ? ?i, a = 0, 0 > ? ? ? ?exec """def promote(n): > ? ? ? ? ? ? ? ? ? ?assert n==%d""" % n > ? ? ? ?while i < n: > ? ? ? ? ? ?promote(n) > ? ? ? ? ? ?a += i+5 > ? ? ? ? ? ?i += 1 > ? ? ? ?return a > > With this I get two extra operations in the loop: > > ? ?i11 = ptr_eq(ConstPtr(ptr10), p7) > ? ?guard_false(i11, descr=) [p1, p0, p2, p3, p4, i5, i6] > > but p7 is loop-invariant so they should be easy to get rid of. I don't > know why they are not already... > > -- > H?kan Ard? > -- H?kan Ard? From anto.cuni at gmail.com Thu Apr 14 16:56:03 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Thu, 14 Apr 2011 16:56:03 +0200 Subject: [pypy-dev] [pypy-svn] pypy default: port test_intbound_addsub_ge to test_pypy_c_new In-Reply-To: References: <20110414124402.CD7472A204D@codespeak.net> <4DA6F01D.8020505@gmail.com> Message-ID: <4DA70B03.7090106@gmail.com> On 14/04/11 16:44, Hakan Ardo wrote: > Second though, that will recompile on every call, but if we cache the > promote functions: > > def main(n, promoters={}): > i, a = 0, 0 > if n not in promoters: > exec """def promote(n): > assert n==%d""" % n > promoters[n] = promote > else: > promote = promoters[n] > while i < n: > promote(n) > a += i+5 > i += 1 > return a > > we will actualy get rid of the extra ptr_eq and guard_false too... wow, that's advanced... and without knowing the internals of the JIT, it really looks like black magic. While we are at it and if you have time/feel like, could you please have a look to test_zeropadded and test_circular in test_pypy_c_new? It's not clear to me what they are supposed to check (note that it's fine if you say "they just check that the program works correctly" :-)). From alex.gaynor at gmail.com Thu Apr 14 17:15:13 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Thu, 14 Apr 2011 11:15:13 -0400 Subject: [pypy-dev] [pypy-svn] pypy default: port test_intbound_addsub_ge to test_pypy_c_new In-Reply-To: <4DA70B03.7090106@gmail.com> References: <20110414124402.CD7472A204D@codespeak.net> <4DA6F01D.8020505@gmail.com> <4DA70B03.7090106@gmail.com> Message-ID: On Thu, Apr 14, 2011 at 10:56 AM, Antonio Cuni wrote: > On 14/04/11 16:44, Hakan Ardo wrote: > > Second though, that will recompile on every call, but if we cache the > > promote functions: > > > > def main(n, promoters={}): > > i, a = 0, 0 > > if n not in promoters: > > exec """def promote(n): > > assert n==%d""" % n > > promoters[n] = promote > > else: > > promote = promoters[n] > > while i < n: > > promote(n) > > a += i+5 > > i += 1 > > return a > > > > we will actualy get rid of the extra ptr_eq and guard_false too... > > wow, that's advanced... and without knowing the internals of the JIT, it > really looks like black magic. > > While we are at it and if you have time/feel like, could you please have a > look to test_zeropadded and test_circular in test_pypy_c_new? It's not > clear > to me what they are supposed to check (note that it's fine if you say "they > just check that the program works correctly" :-)). > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > The idea of a builtin app level promote is cool, I guess it should be smart and unbox primitives though. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110414/63f7b87c/attachment.htm From holger at merlinux.eu Thu Apr 14 19:47:39 2011 From: holger at merlinux.eu (holger krekel) Date: Thu, 14 Apr 2011 17:47:39 +0000 Subject: [pypy-dev] project infrastructure issues Message-ID: <20110414174739.GX16231@merlinux.eu> Hey all, now that pypy's codespeak subversion usage is basically gone i'd like to push for remaining issues related to the pypy infrastructure: - apache/website - buildbot/master - roundup/issue tracker - mailman/mailing lists pypy-dev/commits/z Which of them shall we try to move elsewhere? My preliminary suggestion: - website -> readthedocs? or other site - buildbot -> python.org? or other site - issue tracker -> bitbucket issue tracker - mailing lists -> google groups or python.org or other site The "other site" could be a host that i anyway need to have for remaining codespeak and merlinux stuff and which thus is somewhat guaranteed to work mail- and otherwise. Other people could get admin access as well, of course. any suggestions or comments? best, holger From fwierzbicki at gmail.com Thu Apr 14 19:20:27 2011 From: fwierzbicki at gmail.com (fwierzbicki at gmail.com) Date: Thu, 14 Apr 2011 10:20:27 -0700 Subject: [pypy-dev] The JVM backend and Jython In-Reply-To: <4DA349C2.3080403@gmail.com> References: <4DA349C2.3080403@gmail.com> Message-ID: On Mon, Apr 11, 2011 at 11:34 AM, Antonio Cuni wrote: > Hi Frank, > > On 30/03/11 04:40, fwierzbicki at gmail.com wrote: > [cut] >> So to my question - just how broken is the JVM backend? Are there >> workarounds that would allow the Java code to get generated? > > so, now the jvm (and cli) translation works again. You can just type > ./translate.py -b jvm, and the fish the relevant .class/.j files from > /tmp/usession-default-*/pypy. Nice - thanks! -Frank From alex.gaynor at gmail.com Fri Apr 15 03:01:21 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Thu, 14 Apr 2011 21:01:21 -0400 Subject: [pypy-dev] project infrastructure issues In-Reply-To: <20110414174739.GX16231@merlinux.eu> References: <20110414174739.GX16231@merlinux.eu> Message-ID: On Thu, Apr 14, 2011 at 1:47 PM, holger krekel wrote: > Hey all, > > now that pypy's codespeak subversion usage is basically gone i'd like to > push for remaining issues related to the pypy infrastructure: > > - apache/website > - buildbot/master > - roundup/issue tracker > - mailman/mailing lists pypy-dev/commits/z > > Which of them shall we try to move elsewhere? > > My preliminary suggestion: > > - website -> readthedocs? or other site > - buildbot -> python.org? or other site > - issue tracker -> bitbucket issue tracker > - mailing lists -> google groups or python.org or other site > > The "other site" could be a host that i anyway > need to have for remaining codespeak and merlinux stuff > and which thus is somewhat guaranteed to work > mail- and otherwise. Other people could get admin > access as well, of course. > > any suggestions or comments? > > best, > holger > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > readthedocs seems like the right solution for docs, should just be a matter of setting up the post-commit hook and adding a cname for docs.pypy.org Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110414/fbe33cda/attachment.htm From anto.cuni at gmail.com Fri Apr 15 09:58:01 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Fri, 15 Apr 2011 09:58:01 +0200 Subject: [pypy-dev] [pypy-svn] pypy default: fixed test_circular In-Reply-To: <20110414171414.3B30C2A2043@codespeak.net> References: <20110414171414.3B30C2A2043@codespeak.net> Message-ID: <4DA7FA89.2000908@gmail.com> Hi Hakan, thanks for the commits > + # We want to check that the array bound checks are removed, > + # so it's this part of the trace. However we dont care about > + # the force_token()'s. Can they be ignored? yes, I think they can be just ignored, because AFAIK operations without side effects and whose result is unused, are removed by the backend regalloc. ciao, Anto From anto.cuni at gmail.com Fri Apr 15 10:38:30 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Fri, 15 Apr 2011 10:38:30 +0200 Subject: [pypy-dev] [pypy-svn] pypy default: fixed test_circular In-Reply-To: References: <20110414171414.3B30C2A2043@codespeak.net> <4DA7FA89.2000908@gmail.com> Message-ID: > Right. My point was that since we dont care if they are there or not > the test should not test that they are there and fail if they are not. > So if there is an easy way to ignore them in this new test_pypy_c > framework (which is very cool by the way), we should. If it's not easy > I'm fine with keeping the test as it is. My main motivation here is to > learn about the new framework :) ah, I understand now. No, ignoring all force_tokens at once is not possible at the moment, but I agree that it would be a nice feature, I think I'll implement it later. Btw, I fear I need more of your help with test_silly_max and test_iter_max (see 2e5bd737be0c): what do we want to check there? From fijall at gmail.com Fri Apr 15 10:45:08 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 15 Apr 2011 10:45:08 +0200 Subject: [pypy-dev] [pypy-svn] pypy default: fixed test_circular In-Reply-To: References: <20110414171414.3B30C2A2043@codespeak.net> <4DA7FA89.2000908@gmail.com> Message-ID: On Fri, Apr 15, 2011 at 10:38 AM, Antonio Cuni wrote: >> Right. My point was that since we dont care if they are there or not >> the test should not test that they are there and fail if they are not. >> So if there is an easy way to ignore them in this new test_pypy_c >> framework (which is very cool by the way), we should. If it's not easy >> I'm fine with keeping the test as it is. My main motivation here is to >> learn about the new framework :) > > ah, I understand now. > No, ignoring all force_tokens at once is not possible at the moment, > but I agree that it would be a nice feature, I think I'll implement it > later. > > Btw, I fear I need more of your help with test_silly_max and > test_iter_max (see 2e5bd737be0c): what do we want to check there? > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > Note that force_token operation is really cheap in the backend. It also does not use a whole lot of space. From hakan at debian.org Fri Apr 15 10:50:25 2011 From: hakan at debian.org (Hakan Ardo) Date: Fri, 15 Apr 2011 10:50:25 +0200 Subject: [pypy-dev] [pypy-svn] pypy default: fixed test_circular In-Reply-To: References: <20110414171414.3B30C2A2043@codespeak.net> <4DA7FA89.2000908@gmail.com> Message-ID: Hi, the point here is that we want max(a,b) to be turned into a single guard while we dont want max(*range(300)) and max(range(300)) to blow up into 300 guards, since that might lead to 2**300 different traces. I'm not sure how to best test this... On Fri, Apr 15, 2011 at 10:38 AM, Antonio Cuni wrote: >> Right. My point was that since we dont care if they are there or not >> the test should not test that they are there and fail if they are not. >> So if there is an easy way to ignore them in this new test_pypy_c >> framework (which is very cool by the way), we should. If it's not easy >> I'm fine with keeping the test as it is. My main motivation here is to >> learn about the new framework :) > > ah, I understand now. > No, ignoring all force_tokens at once is not possible at the moment, > but I agree that it would be a nice feature, I think I'll implement it > later. > > Btw, I fear I need more of your help with test_silly_max and > test_iter_max (see 2e5bd737be0c): what do we want to check there? > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -- H?kan Ard? From anto.cuni at gmail.com Fri Apr 15 11:55:45 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Fri, 15 Apr 2011 11:55:45 +0200 Subject: [pypy-dev] [pypy-svn] pypy default: fixed test_circular In-Reply-To: References: <20110414171414.3B30C2A2043@codespeak.net> <4DA7FA89.2000908@gmail.com> Message-ID: <4DA81621.2020000@gmail.com> On 15/04/11 10:50, Hakan Ardo wrote: > Hi, > the point here is that we want max(a,b) to be turned into a single > guard while we dont want max(*range(300)) and max(range(300)) to blow > up into 300 guards, since that might lead to 2**300 different traces. > I'm not sure how to best test this... can't we just check that the loop contains a residual call to min_max_loop? From hakan at debian.org Fri Apr 15 13:11:53 2011 From: hakan at debian.org (Hakan Ardo) Date: Fri, 15 Apr 2011 13:11:53 +0200 Subject: [pypy-dev] [pypy-svn] pypy default: fixed test_circular In-Reply-To: <4DA81621.2020000@gmail.com> References: <20110414171414.3B30C2A2043@codespeak.net> <4DA7FA89.2000908@gmail.com> <4DA81621.2020000@gmail.com> Message-ID: OK, I also added a check on the guard count. On Fri, Apr 15, 2011 at 11:55 AM, Antonio Cuni wrote: > On 15/04/11 10:50, Hakan Ardo wrote: >> Hi, >> the point here is that we want max(a,b) to be turned into a single >> guard while we dont want max(*range(300)) and max(range(300)) to blow >> up into 300 guards, since that might lead to 2**300 different traces. >> I'm not sure how to best test this... > > can't we just check that the loop contains a residual call to min_max_loop? > -- H?kan Ard? From holger at merlinux.eu Fri Apr 15 21:42:02 2011 From: holger at merlinux.eu (holger krekel) Date: Fri, 15 Apr 2011 19:42:02 +0000 Subject: [pypy-dev] project infrastructure issues In-Reply-To: References: <20110414174739.GX16231@merlinux.eu> Message-ID: <20110415194202.GZ16231@merlinux.eu> On Thu, Apr 14, 2011 at 21:01 -0400, Alex Gaynor wrote: > On Thu, Apr 14, 2011 at 1:47 PM, holger krekel wrote: > > > Hey all, > > > > now that pypy's codespeak subversion usage is basically gone i'd like to > > push for remaining issues related to the pypy infrastructure: > > > > - apache/website > > - buildbot/master > > - roundup/issue tracker > > - mailman/mailing lists pypy-dev/commits/z > > > > Which of them shall we try to move elsewhere? > > > > My preliminary suggestion: > > > > - website -> readthedocs? or other site > > - buildbot -> python.org? or other site > > - issue tracker -> bitbucket issue tracker > > - mailing lists -> google groups or python.org or other site > > > > The "other site" could be a host that i anyway > > need to have for remaining codespeak and merlinux stuff > > and which thus is somewhat guaranteed to work > > mail- and otherwise. Other people could get admin > > access as well, of course. > > > > any suggestions or comments? > > > > best, > > holger > > _______________________________________________ > > pypy-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/pypy-dev > > > > readthedocs seems like the right solution for docs, should just be a matter > of setting up the post-commit hook and adding a cname for docs.pypy.org That would still leave open the question of pypy.org itself i guess. besides, "make" in pypy/doc spews out a lot of errors and warnings for me. Do you know if anybody is caring for completing the transition to sphinx? holger From fijall at gmail.com Fri Apr 15 22:02:30 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 15 Apr 2011 22:02:30 +0200 Subject: [pypy-dev] project infrastructure issues In-Reply-To: <20110415194202.GZ16231@merlinux.eu> References: <20110414174739.GX16231@merlinux.eu> <20110415194202.GZ16231@merlinux.eu> Message-ID: On Fri, Apr 15, 2011 at 9:42 PM, holger krekel wrote: > On Thu, Apr 14, 2011 at 21:01 -0400, Alex Gaynor wrote: >> On Thu, Apr 14, 2011 at 1:47 PM, holger krekel wrote: >> >> > Hey all, >> > >> > now that pypy's codespeak subversion usage is basically gone i'd like to >> > push for remaining issues related to the pypy infrastructure: >> > >> > - apache/website >> > - buildbot/master >> > - roundup/issue tracker >> > - mailman/mailing lists pypy-dev/commits/z >> > >> > Which of them shall we try to move elsewhere? >> > >> > My preliminary suggestion: >> > >> > - website -> readthedocs? or other site >> > - buildbot -> python.org? or other site >> > - issue tracker -> bitbucket issue tracker >> > - mailing lists -> google groups or python.org or other site >> > >> > The "other site" could be a host that i anyway >> > need to have for remaining codespeak and merlinux stuff >> > and which thus is somewhat guaranteed to work >> > mail- and otherwise. ?Other people could get admin >> > access as well, of course. >> > >> > any suggestions or comments? >> > >> > best, >> > holger >> > _______________________________________________ >> > pypy-dev at codespeak.net >> > http://codespeak.net/mailman/listinfo/pypy-dev >> > >> >> readthedocs seems like the right solution for docs, should just be a matter >> of setting up the post-commit hook and adding a cname for docs.pypy.org > > That would still leave open the question of pypy.org itself i guess. > > besides, "make" in pypy/doc spews out a lot of errors and warnings for me. > Do you know if anybody is caring for completing the transition to sphinx? > > holger I think laura does. From matthew at woodcraft.me.uk Sun Apr 17 22:03:16 2011 From: matthew at woodcraft.me.uk (Matthew Woodcraft) Date: Sun, 17 Apr 2011 21:03:16 +0100 Subject: [pypy-dev] Ignore 'pinsrb/w/d' instructions in trackgcroot Message-ID: <20110417200316.GA22864@golux.woodcraft.me.uk> I found I needed the following patch in order to run translation with gcc 4.6 and -march=corei7. -M- --- a/pypy/translator/c/gcc/trackgcroot.py +++ b/pypy/translator/c/gcc/trackgcroot.py @@ -456,7 +456,7 @@ class FunctionGcRootTracker(object): 'inc', 'dec', 'not', 'neg', 'or', 'and', 'sbb', 'adc', 'shl', 'shr', 'sal', 'sar', 'rol', 'ror', 'mul', 'imul', 'div', 'idiv', 'bswap', 'bt', 'rdtsc', - 'punpck', 'pshufd', 'pcmp', 'pand', 'psllw', 'pslld', 'psllq', + 'punpck', 'pshufd', 'pcmp', 'pand', 'psllw', 'pslld', 'psllq', 'pinsr', # zero-extending moves should not produce GC pointers 'movz', ]) From fijall at gmail.com Mon Apr 18 08:07:39 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 18 Apr 2011 08:07:39 +0200 Subject: [pypy-dev] Ignore 'pinsrb/w/d' instructions in trackgcroot In-Reply-To: <20110417200316.GA22864@golux.woodcraft.me.uk> References: <20110417200316.GA22864@golux.woodcraft.me.uk> Message-ID: thanks, commited! On Sun, Apr 17, 2011 at 10:03 PM, Matthew Woodcraft wrote: > I found I needed the following patch in order to run translation with > gcc 4.6 and -march=corei7. > > -M- > > > --- a/pypy/translator/c/gcc/trackgcroot.py > +++ b/pypy/translator/c/gcc/trackgcroot.py > @@ -456,7 +456,7 @@ class FunctionGcRootTracker(object): > ? ? ? ? 'inc', 'dec', 'not', 'neg', 'or', 'and', 'sbb', 'adc', > ? ? ? ? 'shl', 'shr', 'sal', 'sar', 'rol', 'ror', 'mul', 'imul', 'div', 'idiv', > ? ? ? ? 'bswap', 'bt', 'rdtsc', > - ? ? ? ?'punpck', 'pshufd', 'pcmp', 'pand', 'psllw', 'pslld', 'psllq', > + ? ? ? ?'punpck', 'pshufd', 'pcmp', 'pand', 'psllw', 'pslld', 'psllq', 'pinsr', > ? ? ? ? # zero-extending moves should not produce GC pointers > ? ? ? ? 'movz', > ? ? ? ? ]) > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From pypy at pocketnix.org Mon Apr 18 13:44:13 2011 From: pypy at pocketnix.org (pypy at pocketnix.org) Date: Mon, 18 Apr 2011 11:44:13 +0000 Subject: [pypy-dev] project infrastructure issues In-Reply-To: <20110414174739.GX16231@merlinux.eu> References: <20110414174739.GX16231@merlinux.eu> Message-ID: <20110418114413.GA17283@pocketnix.org> On Thu, Apr 14, 2011 at 05:47:39PM +0000, holger krekel wrote: > Hey all, > > now that pypy's codespeak subversion usage is basically gone i'd like to > push for remaining issues related to the pypy infrastructure: > > - apache/website > - buildbot/master > - roundup/issue tracker > - mailman/mailing lists pypy-dev/commits/z > > Which of them shall we try to move elsewhere? > > My preliminary suggestion: > > - website -> readthedocs? or other site > - buildbot -> python.org? or other site > - issue tracker -> bitbucket issue tracker > - mailing lists -> google groups or python.org or other site > > The "other site" could be a host that i anyway > need to have for remaining codespeak and merlinux stuff > and which thus is somewhat guaranteed to work > mail- and otherwise. Other people could get admin > access as well, of course. > Hi I dont mind hosting the website and mailing lists and would be more than willing to grab a cheap dedicated box or vps somewhere to do so. i already manage a number of boxes for my own private use and adding another wont be much more work i should be able to provide some build slaves without issue. these would be on a home dsl plan and may be down for a day or two per year. that said i can easily cache the output on a publicly available server. KVM or linux containers are avalible on the host and i was planning to run at least ubuntu and fedora as build slaves in a container with $GB of ram allocated to them (64bit) ssh access would be avalible. at the very least i am willing to put in some admin time for the pypy project. if there is alternate hardware avalible and you just need someone to man them or set them up let me know Questions: * with Apache is this a standard cgi setup or a wsgi or mod_python setup for any dynamic parts of the site * if there are any other oddities or special setups let me know - Da_Blitz From alex.gaynor at gmail.com Mon Apr 18 15:16:18 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Mon, 18 Apr 2011 09:16:18 -0400 Subject: [pypy-dev] project infrastructure issues In-Reply-To: <20110418114413.GA17283@pocketnix.org> References: <20110414174739.GX16231@merlinux.eu> <20110418114413.GA17283@pocketnix.org> Message-ID: On Mon, Apr 18, 2011 at 7:44 AM, wrote: > On Thu, Apr 14, 2011 at 05:47:39PM +0000, holger krekel wrote: > > Hey all, > > > > now that pypy's codespeak subversion usage is basically gone i'd like to > > push for remaining issues related to the pypy infrastructure: > > > > - apache/website > > - buildbot/master > > - roundup/issue tracker > > - mailman/mailing lists pypy-dev/commits/z > > > > Which of them shall we try to move elsewhere? > > > > My preliminary suggestion: > > > > - website -> readthedocs? or other site > > - buildbot -> python.org? or other site > > - issue tracker -> bitbucket issue tracker > > - mailing lists -> google groups or python.org or other site > > > > The "other site" could be a host that i anyway > > need to have for remaining codespeak and merlinux stuff > > and which thus is somewhat guaranteed to work > > mail- and otherwise. Other people could get admin > > access as well, of course. > > > > Hi > > I dont mind hosting the website and mailing lists and would be more > than willing to grab a cheap dedicated box or vps somewhere to do so. > i already manage a number of boxes for my own private use and adding > another wont be much more work > > i should be able to provide some build slaves without issue. these > would be on a home dsl plan and may be down for a day or two per year. > that said i can easily cache the output on a publicly available > server. > > KVM or linux containers are avalible on the host and i was planning to > run at least ubuntu and fedora as build slaves in a container with $GB > of ram allocated to them (64bit) ssh access would be avalible. > > at the very least i am willing to put in some admin time for the pypy > project. if there is alternate hardware avalible and you just need > someone to man them or set them up let me know > > Questions: > * with Apache is this a standard cgi setup or a wsgi or mod_python setup > for any dynamic parts of the site > > * if there are any other oddities or special setups let me know > > - Da_Blitz > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > For hosting just the website, I'm sure someone like ep.io would be more than willing to help us out if we asked (hell we probably fall under their free limits). Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110418/0b2465e2/attachment-0001.htm From berdario at gmail.com Mon Apr 18 23:14:16 2011 From: berdario at gmail.com (Dario Bertini) Date: Mon, 18 Apr 2011 23:14:16 +0200 Subject: [pypy-dev] Possible sprint in Genova before/after Europython In-Reply-To: <4DA578EA.2080108@gmail.com> References: <4DA578EA.2080108@gmail.com> Message-ID: On 13 April 2011 12:20, Antonio Cuni wrote: > Thus, I ask everybody who is potentially/likely interested in coming to let me > know. > I plan to attend From qbproger at gmail.com Tue Apr 19 00:25:44 2011 From: qbproger at gmail.com (Joe) Date: Mon, 18 Apr 2011 18:25:44 -0400 Subject: [pypy-dev] Problem with large allocation in test Message-ID: I was trying to run the test file: pypy/jit/backend/x86/test/test_rx86_64_auto_encoding.py and was getting the following traceback: http://paste.pocoo.org/show/374129/ If you look at the comment on line 17, it's trying to allocate much more memory than I have. I think it's a total of 21GB, while I only have 4GB. I'm using 64bit OpenSuSE 11.4 for my operating system. I had the kernel setting overcommit_memory set to 0 (which may be part of the problem). Anyway, after I went into ll2ctypes.py and set far_regions to True, I was able to successfully run the original test. I don't think setting far_regions to True is the correct solution to the problem, but fiddling with kernel settings on my system is not ideal either. What would be a better overall solution? If any clarification is needed let me know, Joe From pypy at pocketnix.org Tue Apr 19 00:59:26 2011 From: pypy at pocketnix.org (pypy at pocketnix.org) Date: Mon, 18 Apr 2011 22:59:26 +0000 Subject: [pypy-dev] Problem with large allocation in test In-Reply-To: References: Message-ID: <20110418225926.GB17283@pocketnix.org> On Mon, Apr 18, 2011 at 06:25:44PM -0400, Joe wrote: > I was trying to run the test file: > pypy/jit/backend/x86/test/test_rx86_64_auto_encoding.py > > and was getting the following traceback: > http://paste.pocoo.org/show/374129/ > > If you look at the comment on line 17, it's trying to allocate much > more memory than I have. I think it's a total of 21GB, while I only > have 4GB. I'm using 64bit OpenSuSE 11.4 for my operating system. I > had the kernel setting overcommit_memory set to 0 (which may be part > of the problem). > > Anyway, after I went into ll2ctypes.py and set far_regions to True, I > was able to successfully run the original test. I don't think setting > far_regions to True is the correct solution to the problem, but > fiddling with kernel settings on my system is not ideal either. What > would be a better overall solution? > > If any clarification is needed let me know, > Joe your vm.overcommit_ratio should be set to "50" or 50% by default, as per http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob_plain;f=Documentation/vm/overcommit-accounting;hb=HEAD this means that any allocations of 6GB will automatically be rejected as "not sane" and you should receive the ENOMEM error indicating the kernel cannot satisfy the supplied range there are a couple of ways to fix this * don't allocate so much ram (did anyone test this before on a 64bit host on linux) * change the vm overcommit policy to 1 (allow everything, don't perform sanity checks) * change the overcommit ratio to something that will satisfy the allocation (20GB/4GB ~= 5x, so a value of 600% or 600 should do it) * Make the mapping a rmmap.MAP_PRIVATE and rmmap.PROT_READ only, depending on what you are testing this may not be useful as indicated in the linked kernel documentation YYMV, first option is safe, 2nd option you may want to double check the documentation and the values you are passing, the 3rd option is also safe in that it will allow badly behaved apps to run, not prevent apps from running to change either of these values use the following: * to adjust the allocation policy: sysctl vm.overcommit_memory= * to adjust the ratio: sysctl vm.overcommit_ratio= to print the current values (and save them for restoring them after you have done tweaking them: sysctl vm.overcommit_memory or sysctl vm.overcommit_ratio Hope this is whats causing the issue Da_Blitz From ian.overgard at gmail.com Tue Apr 19 03:15:57 2011 From: ian.overgard at gmail.com (Ian Overgard) Date: Mon, 18 Apr 2011 19:15:57 -0600 Subject: [pypy-dev] rlib parsing Message-ID: Hey guys, I was playing around with using the parser generator in rlib, but I'm having some trouble figuring out how to get it to work in rpython (I did get one working in normal python though). Is there a resource on using it somewhere, or maybe just a few pointers? (I realize it's probably in a pretty beta state, but so far it seems like the only parser-generator that's runnable in rpython without really big changes). I was reading this: http://codespeak.net/pypy/dist/pypy/doc/rlib.html#full-example but it seems to cut off rather abruptly. It seems like you do something along the lines of instantiating one in normal python, then asking it for its string representation and generating a source file from that. Is that accurate? Or did I just manage to confuse myself terribly while reading the prolog example? Thanks! Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110418/0f66c7e2/attachment.htm From santagada at gmail.com Tue Apr 19 04:47:48 2011 From: santagada at gmail.com (Leonardo Santagada) Date: Mon, 18 Apr 2011 23:47:48 -0300 Subject: [pypy-dev] rlib parsing In-Reply-To: References: Message-ID: IIRC you have to parse your grammar definition before your entry point for example in the module level somewhere you do try: t = GFILE.read(mode='U') regexs, rules, ToAST = parse_ebnf(t) except ParseError,e: print e.nice_error_message(filename=str(GFILE),source=t) raise parsef = make_parse_function(regexs, rules, eof=True) then parsef can be used inside your entry point. What I mean is, parse_ebnf and make_parse_function are not RPython, so they need to run before translation take place (remember that the pypy translator runs after import time, and it translates rpython from live python objects in memory). On Mon, Apr 18, 2011 at 10:15 PM, Ian Overgard wrote: > Hey guys, > > I was playing around with using the parser generator in rlib, but I'm having > some trouble figuring out how to get it to work in rpython (I did get one > working in normal python though). Is there a resource on using it somewhere, > or maybe just a few pointers? (I realize it's probably in a pretty beta > state, but so far it seems like the only parser-generator that's runnable in > rpython without really big changes). I was reading this: > http://codespeak.net/pypy/dist/pypy/doc/rlib.html#full-example but it seems > to cut off rather abruptly. > > It seems like you do something along the lines of instantiating one in > normal python, then asking it for its string representation and generating a > source file from that. Is that accurate? Or did I just manage to confuse > myself terribly while reading the prolog example? > > Thanks! > Ian > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -- Leonardo Santagada From fijall at gmail.com Tue Apr 19 08:23:47 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 19 Apr 2011 08:23:47 +0200 Subject: [pypy-dev] Problem with large allocation in test In-Reply-To: <20110418225926.GB17283@pocketnix.org> References: <20110418225926.GB17283@pocketnix.org> Message-ID: On Tue, Apr 19, 2011 at 12:59 AM, wrote: > On Mon, Apr 18, 2011 at 06:25:44PM -0400, Joe wrote: >> I was trying to run the test file: >> pypy/jit/backend/x86/test/test_rx86_64_auto_encoding.py >> >> and was getting the following traceback: >> http://paste.pocoo.org/show/374129/ >> >> If you look at the comment on line 17, it's trying to allocate much >> more memory than I have. ?I think it's a total of 21GB, while I only >> have 4GB. ?I'm using 64bit OpenSuSE 11.4 for my operating system. I >> had the kernel setting overcommit_memory set to 0 (which may be part >> of the problem). >> >> Anyway, after I went into ll2ctypes.py and set far_regions to True, I >> was able to successfully run the original test. ?I don't think setting >> far_regions to True is the correct solution to the problem, but >> fiddling with kernel settings on my system is not ideal either. ?What >> would be a better overall solution? >> >> If any clarification is needed let me know, >> Joe > > > your vm.overcommit_ratio should be set to "50" or 50% by default, as > per > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob_plain;f=Documentation/vm/overcommit-accounting;hb=HEAD > this means that any allocations of 6GB will automatically be rejected > as "not sane" and you should receive the ENOMEM error indicating the > kernel cannot satisfy the supplied range > > there are a couple of ways to fix this > > * don't allocate so much ram (did anyone test this before on a 64bit > host on linux) The reason for this is that we want to test far jumps (exceeding 4G or 2^32 in address space). How can you do it otherwise? > > * change the vm overcommit policy to 1 (allow everything, don't perform > sanity checks) > > * change the overcommit ratio to something that will satisfy the > allocation (20GB/4GB ~= 5x, so a value of 600% or 600 should do it) > > * Make the mapping a rmmap.MAP_PRIVATE and rmmap.PROT_READ only, > depending on what you are testing this may not be useful as indicated > in the linked kernel documentation > > YYMV, first option is safe, 2nd option you may want to double check > the documentation and the values you are passing, the 3rd option is > also safe in that it will allow badly behaved apps to run, not prevent > apps from running > > > to change either of these values use the following: > > * to adjust the allocation policy: > ?sysctl vm.overcommit_memory= > > * to adjust the ratio: > ?sysctl vm.overcommit_ratio= > > to print the current values (and save them for restoring them after > you have done tweaking them: > ?sysctl vm.overcommit_memory > or > ?sysctl vm.overcommit_ratio > > > Hope this is whats causing the issue > Da_Blitz > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From arigo at tunes.org Tue Apr 19 10:10:40 2011 From: arigo at tunes.org (Armin Rigo) Date: Tue, 19 Apr 2011 10:10:40 +0200 Subject: [pypy-dev] Problem with large allocation in test In-Reply-To: References: <20110418225926.GB17283@pocketnix.org> Message-ID: Hi Joe, On Tue, Apr 19, 2011 at 8:23 AM, Maciej Fijalkowski wrote: >>> Anyway, after I went into ll2ctypes.py and set far_regions to True This is a correct workaround, indeed. As you found out the problem comes from the fact that your OS reserves actual RAM from mmap() eagerly. As Maciej points out, it's useful to test e.g. jumps over more than +/-2GB. But the same result could potentially be obtained in a different way: instead of allocating a single block of 20GB, we could mmap 10 blocks of (say) 20MB for the test, but taking care of placing the blocks so that they start 2GB apart from each other. This could be done using pypy.rlib.rmmap.alloc() or a variant. A bient?t, Armin. From vincent.legoll at gmail.com Tue Apr 19 14:02:43 2011 From: vincent.legoll at gmail.com (Vincent Legoll) Date: Tue, 19 Apr 2011 14:02:43 +0200 Subject: [pypy-dev] Problem with large allocation in test In-Reply-To: References: <20110418225926.GB17283@pocketnix.org> Message-ID: Hello On Tue, Apr 19, 2011 at 10:10 AM, Armin Rigo wrote: > This is a correct workaround, indeed. ?As you found out the problem > comes from the fact that your OS reserves actual RAM from mmap() > eagerly. ?As Maciej points out, it's useful to test e.g. jumps over > more than +/-2GB. ?But the same result could potentially be obtained > in a different way: instead of allocating a single block of 20GB, we > could mmap 10 blocks of (say) 20MB for the test, but taking care of > placing the blocks so that they start 2GB apart from each other. ?This > could be done using pypy.rlib.rmmap.alloc() or a variant. Wouldn't a big read only mmap from /dev/zero before the to be jumped adress allow for not really trying to allocate the big chunk of RAM ? That may be a good fix, still allowing the test for far jumps... -- Vincent Legoll From ian.overgard at gmail.com Wed Apr 20 00:51:00 2011 From: ian.overgard at gmail.com (Ian Overgard) Date: Tue, 19 Apr 2011 16:51:00 -0600 Subject: [pypy-dev] rlib parsing In-Reply-To: References: Message-ID: Thanks, that definitely helped. I forgot you could run arbitrary python before the entry point. I've got it parsing now, but the one issue I'm still running into is that the syntax tree that comes back has a lot of junk nodes. The ToAST.transform function will clean them up, but it seems to be not-rpython, and I don't think there's any way I can call it before the entry point (since it's processing the ast, not the grammar). Is that class just hopeless? Or is there some way I can annotate it myself in the code? ( On Mon, Apr 18, 2011 at 8:47 PM, Leonardo Santagada wrote: > IIRC you have to parse your grammar definition before your entry point > > for example in the module level somewhere you do > > try: > t = GFILE.read(mode='U') > regexs, rules, ToAST = parse_ebnf(t) > except ParseError,e: > print e.nice_error_message(filename=str(GFILE),source=t) > raise > > parsef = make_parse_function(regexs, rules, eof=True) > > then parsef can be used inside your entry point. What I mean is, > parse_ebnf and make_parse_function are not RPython, so they need to > run before translation take place (remember that the pypy translator > runs after import time, and it translates rpython from live python > objects in memory). > > On Mon, Apr 18, 2011 at 10:15 PM, Ian Overgard > wrote: > > Hey guys, > > > > I was playing around with using the parser generator in rlib, but I'm > having > > some trouble figuring out how to get it to work in rpython (I did get one > > working in normal python though). Is there a resource on using it > somewhere, > > or maybe just a few pointers? (I realize it's probably in a pretty beta > > state, but so far it seems like the only parser-generator that's runnable > in > > rpython without really big changes). I was reading this: > > http://codespeak.net/pypy/dist/pypy/doc/rlib.html#full-example but it > seems > > to cut off rather abruptly. > > > > It seems like you do something along the lines of instantiating one in > > normal python, then asking it for its string representation and > generating a > > source file from that. Is that accurate? Or did I just manage to confuse > > myself terribly while reading the prolog example? > > > > Thanks! > > Ian > > > > _______________________________________________ > > pypy-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/pypy-dev > > > > > > -- > Leonardo Santagada > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110419/c1f347ce/attachment-0001.htm From santagada at gmail.com Wed Apr 20 07:25:34 2011 From: santagada at gmail.com (Leonardo Santagada) Date: Wed, 20 Apr 2011 02:25:34 -0300 Subject: [pypy-dev] rlib parsing In-Reply-To: References: Message-ID: Sorry I forgot to reply to the list. The first step would be to tell us the error you are getting. But the fastest way to solve your problems, and get help learning how to debug pypy is to join the pypy channel at freenode (IRC) On Tue, Apr 19, 2011 at 8:54 PM, Ian Overgard wrote: > That's what I thought too (my code is exactly like that), but something in > it is causing the translator to break, and the error I get back doesn't seem > to tell me anything I can work with. > > I wouldn't mind jumping in to help fix it, is there maybe some sort of > guide/general tips on how to go about debugging translation issues? So far > I've just been looking at the exceptions and the rpython docs and guessing > it out, but I'm guessing people that have worked with this longer might have > more robust ideas on ways to trace things? > > On Tue, Apr 19, 2011 at 5:24 PM, Leonardo Santagada > wrote: >> >> This code is rpython >> >> t = parsef(code) >> ToAST().transform(t) >> >> Maybe you can look at the tests in the library or the prolog and js >> implementation, both use parsing I think (at least js does). I really >> think that parsing could be polished to be a great tool for people >> wanting to implement a language in pypy. >> >> On Tue, Apr 19, 2011 at 7:51 PM, Ian Overgard >> wrote: >> > Thanks, that definitely helped. I forgot you could run arbitrary python >> > before the entry point. >> > >> > I've got it parsing now, but the one issue I'm still running into is >> > that >> > the syntax tree that comes back has a lot of junk nodes. The >> > ToAST.transform >> > function will clean them up, but it seems to be not-rpython, and I don't >> > think there's any way I can call it before the entry point (since it's >> > processing the ast, not the grammar). >> > >> > Is that class just hopeless? Or is there some way I can annotate it >> > myself >> > in the code? ( >> >> -- >> Leonardo Santagada > > -- Leonardo Santagada From cfbolz at gmx.de Wed Apr 20 11:33:02 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Wed, 20 Apr 2011 11:33:02 +0200 Subject: [pypy-dev] rlib parsing In-Reply-To: References: Message-ID: <4DAEA84E.5020904@gmx.de> On 04/20/2011 12:51 AM, Ian Overgard wrote: > Thanks, that definitely helped. I forgot you could run arbitrary python > before the entry point. > > I've got it parsing now, but the one issue I'm still running into is > that the syntax tree that comes back has a lot of junk nodes. The > ToAST.transform function will clean them up, but it seems to be > not-rpython, and I don't think there's any way I can call it before the > entry point (since it's processing the ast, not the grammar). > > Is that class just hopeless? Or is there some way I can annotate it > myself in the code? ( You could take a look at the test test_translate_ast_visitor in rlib/parsing/test/test_translate.py. It does what you need to do by explicitly calling visit_[initial rule] on the AST visitor. I tried to replace that with a call to transform and it still worked. So I don't know what you are doing differently from that test. Carl Friedrich From arigo at tunes.org Thu Apr 21 12:21:48 2011 From: arigo at tunes.org (Armin Rigo) Date: Thu, 21 Apr 2011 12:21:48 +0200 Subject: [pypy-dev] Problem with large allocation in test In-Reply-To: References: <20110418225926.GB17283@pocketnix.org> Message-ID: Hi Vincent, On Tue, Apr 19, 2011 at 2:02 PM, Vincent Legoll wrote: > Wouldn't a big read only mmap from /dev/zero before the to be jumped > adress allow for not really trying to allocate the big chunk of RAM ? > > That may be a good fix, still allowing the test for far jumps... It's not needed; mmap() can optionally take a position argument, allowing us to choose the position of the small pieces of requested memory. A bient?t, Armin. From pypy at pocketnix.org Thu Apr 21 12:33:58 2011 From: pypy at pocketnix.org (pypy at pocketnix.org) Date: Thu, 21 Apr 2011 10:33:58 +0000 Subject: [pypy-dev] Problem with large allocation in test In-Reply-To: References: <20110418225926.GB17283@pocketnix.org> Message-ID: <20110421103358.GA25510@pocketnix.org> On Thu, Apr 21, 2011 at 12:21:48PM +0200, Armin Rigo wrote: > Hi Vincent, > > On Tue, Apr 19, 2011 at 2:02 PM, Vincent Legoll > wrote: > > Wouldn't a big read only mmap from /dev/zero before the to be jumped > > adress allow for not really trying to allocate the big chunk of RAM ? > > > > That may be a good fix, still allowing the test for far jumps... > > It's not needed; mmap() can optionally take a position argument, > allowing us to choose the position of the small pieces of requested > memory. i have been working on a patch based around this behavior of mmap, currently mmap as called in this test does not allow you to specify the base_addr hint however there is an alloc function as used by the jit that allows you to specify this address indirectly (by setting hint.pos in the same class). it appears only the jit is calling this function and as such i have been working on a patch that i hope to have ready in the next couple of days. i wouldn't mind some feedback on how i am trying to do it i am cloning the alloc function and adding a new parameter to specify a base addr, then ripping out the hint object as i have been unable to find anything that references it directly (with a quick grep, will test to be sure) after there is done i will migrate the test from mmap to the new alloc function and provide hints that satisfy the spacing requirements one this is all working i had intended to dump the Alic function and replace it with the new alloc function and update the callers, from my grepping of the source it appears like it is called once or twice in the jit and should be easy to update i am assuming this is the correct course of action, if i should instead not be replacing the original alloc function and just create a new alloc_with_hint function and keep/use both let me know this has been an intresting way to get fammilar with pypy From arigo at tunes.org Thu Apr 21 13:28:31 2011 From: arigo at tunes.org (Armin Rigo) Date: Thu, 21 Apr 2011 13:28:31 +0200 Subject: [pypy-dev] Problem with large allocation in test In-Reply-To: <20110421103358.GA25510@pocketnix.org> References: <20110418225926.GB17283@pocketnix.org> <20110421103358.GA25510@pocketnix.org> Message-ID: Hi, On Thu, Apr 21, 2011 at 12:33 PM, wrote: > i am assuming this is the correct course of action, if i should > instead not be replacing the original alloc function and just create > a new alloc_with_hint function and keep/use both let me know Indeed, you should keep the original function too: the hint parameter is not used anywhere else in the source code, but it's convenient for debugging because it means that the JIT-generated code will be allocated at a known address, instead of randomly. That's this function's purpose. I'm fine if you create another function. Or if you prefer, you can change the existing function to take an optional base_addr argument. This arguments would default to NULL, and if it is NULL then the function would use the hintp as it does now. A bient?t, Armin. From ian.overgard at gmail.com Thu Apr 21 17:31:42 2011 From: ian.overgard at gmail.com (Ian Overgard) Date: Thu, 21 Apr 2011 09:31:42 -0600 Subject: [pypy-dev] rlib parsing In-Reply-To: <4DAEA84E.5020904@gmx.de> References: <4DAEA84E.5020904@gmx.de> Message-ID: Doh, it turns out you're right, it is valid rpython, the problem was elsewhere. I was assuming the transform function was at fault because the error only happened when that was included in the build. It turns out correlation doesn't equal causation :-) The actual bug was this: [translation:ERROR] Exception': found an operation that always raises AttributeError: generated by a constant operation: getattr(, 'config') Which I managed to fix by putting this lovely hack at the top of my script: class FixConfig: class option: view = False py.test.config = FixConfig Everything seems to be working now. Thanks to both Carl and Leonardo for the help! On Wed, Apr 20, 2011 at 3:33 AM, Carl Friedrich Bolz wrote: > On 04/20/2011 12:51 AM, Ian Overgard wrote: > > Thanks, that definitely helped. I forgot you could run arbitrary python > > before the entry point. > > > > I've got it parsing now, but the one issue I'm still running into is > > that the syntax tree that comes back has a lot of junk nodes. The > > ToAST.transform function will clean them up, but it seems to be > > not-rpython, and I don't think there's any way I can call it before the > > entry point (since it's processing the ast, not the grammar). > > > > Is that class just hopeless? Or is there some way I can annotate it > > myself in the code? ( > > You could take a look at the test test_translate_ast_visitor in > rlib/parsing/test/test_translate.py. It does what you need to do by > explicitly calling visit_[initial rule] on the AST visitor. I tried to > replace that with a call to transform and it still worked. So I don't > know what you are doing differently from that test. > > Carl Friedrich > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110421/6075eb77/attachment.htm From exarkun at twistedmatrix.com Sat Apr 23 01:14:22 2011 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Fri, 22 Apr 2011 23:14:22 -0000 Subject: [pypy-dev] improved cpyext PyArg_ParseTuple s* and t# support Message-ID: <20110422231422.1992.1252208249.divmod.xquotient.920@localhost.localdomain> Hello, There's a pyarg-parsebuffer-new branch in the pypy bitbucket repository that improves PyArg_ParseTuple. It doesn't completely implement either of these, but it adds enough for pyOpenSSL to mostly work. Arbitrary new-style buffer support was too hard for me, so I skipped that. Still, it would be nice to have this branch in 1.5 if possible, since I think with it along with my pyOpenSSL branch, pyOpenSSL becomes mostly functional on PyPy. Can someone take a look and let me know if it's okay to merge to default? Thanks, Jean-Paul From bhartsho at yahoo.com Mon Apr 25 09:45:24 2011 From: bhartsho at yahoo.com (Hart's Antler) Date: Mon, 25 Apr 2011 00:45:24 -0700 (PDT) Subject: [pypy-dev] stackless over ctypes - threading Message-ID: <177297.9081.qm@web114019.mail.gq1.yahoo.com> Hi PyPy Devs, I have an example of using a compiled Rpython extension module in Python using stackless (rcoroutines). From ctypes i can safely read the values of a shared-object (shared between CPython and RPython), but i can not in a thread-safe way set values on the shared object so that RPython can see them. I was wondering what is the best way to handle the thread locking. http://pyppet.blogspot.com/2011/04/stackless-compiled-modules.html thanks, -brett From arigo at tunes.org Mon Apr 25 11:07:55 2011 From: arigo at tunes.org (Armin Rigo) Date: Mon, 25 Apr 2011 11:07:55 +0200 Subject: [pypy-dev] Release 1.5 preparation Message-ID: Hi all, We are starting to prepare the release PyPy 1.5. Note that we are doing it in the "default" branch of mercurial, because it's more convenient to run tests (notably codespeed's). So please, *any* development that you don't explicitly intend to be included in the 1.5 release must go to the branch "post-release-1.5", which we will merge back with "default" after the release is done. A bient?t, Armin. From cfbolz at gmx.de Mon Apr 25 14:31:19 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 25 Apr 2011 14:31:19 +0200 Subject: [pypy-dev] state of std library Message-ID: Hi all, anybody has an idea what the state of PyPy's standard library is? I assume it's at the state of CPython 2.7, right? Then there is the branch merge-stdlib, which seems to merge in the 2.7.1 changes. Why are all the modified directories deleted there? Carl Friedrich From alex.gaynor at gmail.com Mon Apr 25 15:18:12 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Mon, 25 Apr 2011 09:18:12 -0400 Subject: [pypy-dev] state of std library In-Reply-To: References: Message-ID: On Mon, Apr 25, 2011 at 8:31 AM, Carl Friedrich Bolz wrote: > Hi all, > > anybody has an idea what the state of PyPy's standard library is? I > assume it's at the state of CPython 2.7, right? > > Then there is the branch merge-stdlib, which seems to merge in the > 2.7.1 changes. Why are all the modified directories deleted there? > > Carl Friedrich > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > merge-stdlib was a branch Carl Meyer worked on at PyCon. The idea was we kept a single stdlib dir, modified as necessary in that dir, and just used hg to merge up with newer versions of CPython, rather than keeping a -modified dir around. "merge" therefore refers to merging -modified and no -modified, not merging with upstream. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110425/e3d7db2e/attachment.htm From dimaqq at gmail.com Mon Apr 25 21:53:36 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Mon, 25 Apr 2011 12:53:36 -0700 Subject: [pypy-dev] cpyext reference counting and other gc's In-Reply-To: References: Message-ID: Apologies that it took a little long Here is the doc describing the idea and its side effects https://docs.google.com/document/d/1k7t-WIsfKW4tIL9i8-7Y6_9lo18wcsibyDONOF2i_l8/edit?hl=en Since many of you are now busy with release, I'm only asking for quick feedback, especially if I missed something obvious. Thanks, Dima Tisnek On 26 March 2011 10:03, Dima Tisnek wrote: > I have an alternative idea in mind > > I'll write up a doc, stick it on github and share with you guys in a > couple of days > > thanks for a clear answer, I just couldn't figure that out form code easily :P > > d. > > On 26 March 2011 01:33, Amaury Forgeot d'Arc wrote: >> Hi, >> >> 2011/3/26 Dima Tisnek : >>> Hey, I had a look at cpyext recently and saw that reference counting >>> is emulated with, err, reference counting, seeminlgy in the referenced >>> object itself. >>> >>> Does this mean that cpyext would not work with other gc's or is there >>> some wrapping going on behind the scenes? >> >> Cpyext works with all pypy gc's. The PyObject* exposed to C code is actually >> a proxy to the "real" interpreter object; a dict lookup is necessary each time a >> reference crosses the C/pypy boundary. Yes, this is slow. >> >> This is implemented in pypy/module/cpyext/pyobject.py; the main functions are >> create_ref() and from_ref(). >> >> -- >> Amaury Forgeot d'Arc >> > From arigo at tunes.org Tue Apr 26 18:23:30 2011 From: arigo at tunes.org (Armin Rigo) Date: Tue, 26 Apr 2011 18:23:30 +0200 Subject: [pypy-dev] cpyext reference counting and other gc's In-Reply-To: References: Message-ID: Hi Dima, On Mon, Apr 25, 2011 at 9:53 PM, Dima Tisnek wrote: > https://docs.google.com/document/d/1k7t-WIsfKW4tIL9i8-7Y6_9lo18wcsibyDONOF2i_l8/edit?hl=en Can you explain a bit more what are the advantages of the solution you propose, compared to what is already implemented in cpyext? Your description is far too high-level for us to know exactly what you mean. It seems that you want to replace the currently implemented solution with a very different one. I can explain it in a bit more details, but I would first like to hear what goal you are trying to achieve. Here is a quick reply based on guessing. The issue with your version is that Py_INCREF() and Py_DECREF() needs to do a slow dictionary lookup, while ours doesn't. Conversely, I believe that your version doesn't need a dictionary lookup in other cases where ours needs to. However it seems to me that if you add so much overhead to Py_INCREF() and Py_DECREF(), you loose all other speed advantages. A bient?t, Armin. From qbproger at gmail.com Tue Apr 26 23:51:06 2011 From: qbproger at gmail.com (Joe) Date: Tue, 26 Apr 2011 17:51:06 -0400 Subject: [pypy-dev] PyPy Assembler SQRT Patch Message-ID: Attached is a patch to allow pypy to use SQRTSD rather than calling out to libc. This resulted in a 2x speedup, that scaled as the benchmark took longer. When i added another 0 to the end of the benchmark it was still a 2x speedup. benchmark results: http://paste.pocoo.org/show/378122/ I'll be happy to answer any questions about the patch. Joe -------------- next part -------------- A non-text attachment was scrubbed... Name: sqrtsd.patch Type: application/octet-stream Size: 14448 bytes Desc: not available Url : http://codespeak.net/pipermail/pypy-dev/attachments/20110426/ecde1a3d/attachment-0001.obj From bhartsho at yahoo.com Thu Apr 28 02:27:37 2011 From: bhartsho at yahoo.com (Hart's Antler) Date: Wed, 27 Apr 2011 17:27:37 -0700 (PDT) Subject: [pypy-dev] Hybrid gc and stackless not threadsafe Message-ID: <530338.23195.qm@web114020.mail.gq1.yahoo.com> I am using a stackless compiled rpython module in python over ctypes. To make things threadsafe im using named semaphores. The problem is stackless wont work with the refcounting gc that is threadsafe. Sent from Yahoo! Mail on Android -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110427/a0ba5764/attachment.htm From arigo at tunes.org Thu Apr 28 15:34:15 2011 From: arigo at tunes.org (Armin Rigo) Date: Thu, 28 Apr 2011 15:34:15 +0200 Subject: [pypy-dev] stackless over ctypes - threading In-Reply-To: <177297.9081.qm@web114019.mail.gq1.yahoo.com> References: <177297.9081.qm@web114019.mail.gq1.yahoo.com> Message-ID: Hi Brett, On Mon, Apr 25, 2011 at 9:45 AM, Hart's Antler wrote: > I have an example of using a compiled Rpython extension module in Python using stackless (rcoroutines). ?From ctypes i can safely read the values of a shared-object (shared between CPython and RPython), but i can not in a thread-safe way set values on the shared object so that RPython can see them. ?I was wondering what is the best way to handle the thread locking. Sorry, no clue. We don't know exactly what you are doing, but I can tell you that poking around with CPython's ctypes in the GC objects of a translated PyPy will just get you randomness. It might work a bit by chance with today's version of PyPy (e.g. as long as there are not multithreading issues, as long as you are on 32-bits and not 64-bits or vice-versa, as long as you use only the default GC options, etc.). It's not something that we want to support. A bient?t, Armin. From dasdasich at googlemail.com Thu Apr 28 20:55:19 2011 From: dasdasich at googlemail.com (DasIch) Date: Thu, 28 Apr 2011 20:55:19 +0200 Subject: [pypy-dev] Proposal for a common benchmark suite Message-ID: Hello, As announced in my GSoC proposal I'd like to announce which benchmarks I'll use for the benchmark suite I will work on this summer. As of now there are two benchmark suites (that I know of) which receive some sort of attention, those are the ones developed as part of the PyPy project[1] which is used for http://speed.pypy.org and the one initially developed for Unladen Swallow which has been continued by CPython[2]. The PyPy benchmarks contain a lot of interesting benchmarks some explicitly developed for that suite, the CPython benchmarks have an extensive set of microbenchmarks in the pybench package as well as the previously mentioned modifications made to the Unladen Swallow benchmarks. I'd like to "simply" merge both suites so that no changes are lost. However I'd like to leave out the waf benchmark which is part of the PyPy suite, the removal was proposed on pypy-dev for obvious deficits[3]. It will be easier to add a better benchmark later than replacing it at a later point. Unless there is a major issue with this plan I'd like to go forward with this. .. [1]: https://bitbucket.org/pypy/benchmarks .. [2]: http://hg.python.org/benchmarks .. [3]: http://mailrepository.com/pypy-dev.codespeak.net/msg/3627509/ From arigo at tunes.org Sat Apr 30 17:04:41 2011 From: arigo at tunes.org (Armin Rigo) Date: Sat, 30 Apr 2011 17:04:41 +0200 Subject: [pypy-dev] PyPy 1.5 released Message-ID: ====================== PyPy 1.5: Catching Up ====================== We're pleased to announce the 1.5 release of PyPy. This release updates PyPy with the features of CPython 2.7.1, including the standard library. Thus all the features of `CPython 2.6`_ and `CPython 2.7`_ are now supported. It also contains additional performance improvements. You can download it here: http://pypy.org/download.html What is PyPy? ============= PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7.1. It's fast (`pypy 1.5 and cpython 2.6.2`_ performance comparison) due to its integrated tracing JIT compiler. This release includes the features of CPython 2.6 and 2.7. It also includes a large number of small improvements to the tracing JIT compiler. It supports Intel machines running Linux 32/64 or Mac OS X. Windows is beta (it roughly works but a lot of small issues have not been fixed so far). Windows 64 is not yet supported. Numerous speed achievements are described on `our blog`_. Normalized speed charts comparing `pypy 1.5 and pypy 1.4`_ as well as `pypy 1.5 and cpython 2.6.2`_ are available on our benchmark website. The speed improvement over 1.4 seems to be around 25% on average. More highlights =============== - The largest change in PyPy's tracing JIT is adding support for `loop invariant code motion`_, which was mostly done by H?kan Ard?. This feature improves the performance of tight loops doing numerical calculations. - The CPython extension module API has been improved and now supports many more extensions. For information on which one are supported, please refer to our `compatibility wiki`_. - These changes make it possible to support `Tkinter and IDLE`_. - The `cProfile`_ profiler is now working with the JIT. However, it skews the performance in unstudied ways. Therefore it is not yet usable to analyze subtle performance problems (the same is true for CPython of course). - There is an `external fork`_ which includes an RPython version of the ``postgresql``. However, there are no prebuilt binaries for this. - Our developer documentation was moved to Sphinx and cleaned up. (click 'Dev Site' on http://pypy.org/ .) - and many small things :-) Cheers, Carl Friedrich Bolz, Laura Creighton, Antonio Cuni, Maciej Fijalkowski, Amaury Forgeot d'Arc, Alex Gaynor, Armin Rigo and the PyPy team .. _`CPython 2.6`: http://docs.python.org/dev/whatsnew/2.6.html .. _`CPython 2.7`: http://docs.python.org/dev/whatsnew/2.7.html .. _`our blog`: http://morepypy.blogspot.com .. _`pypy 1.5 and pypy 1.4`: http://bit.ly/joPhHo .. _`pypy 1.5 and cpython 2.6.2`: http://bit.ly/mbVWwJ .. _`loop invariant code motion`: http://morepypy.blogspot.com/2011/01/loop-invariant-code-motion.html .. _`compatibility wiki`: https://bitbucket.org/pypy/compatibility/wiki/Home .. _`Tkinter and IDLE`: http://morepypy.blogspot.com/2011/04/using-tkinter-and-idle-with-pypy.html .. _`cProfile`: http://docs.python.org/library/profile.html .. _`external fork`: https://bitbucket.org/alex_gaynor/pypy-postgresql From massimo.sala.71 at gmail.com Sat Apr 30 17:38:19 2011 From: massimo.sala.71 at gmail.com (Massimo Sala) Date: Sat, 30 Apr 2011 17:38:19 +0200 Subject: [pypy-dev] PyPy and memory Message-ID: What about PyPy vs plain CPython ? Please see : http://pushingtheweb.com/2010/06/python-and-tcmalloc/ Is it possible for the maintainers to provide - PyPy - PyPy with tcmalloc so end-users can - give a try - do some tests in their different configs - provide a feedback here on the mailing list ? ciao, Massimo From arigo at tunes.org Sat Apr 30 17:54:31 2011 From: arigo at tunes.org (Armin Rigo) Date: Sat, 30 Apr 2011 17:54:31 +0200 Subject: [pypy-dev] PyPy and memory In-Reply-To: References: Message-ID: Hi Massimo, On Sat, Apr 30, 2011 at 5:38 PM, Massimo Sala wrote: > Is it possible for the maintainers to provide > - PyPy > - PyPy with tcmalloc Feel free to try. You need to get pypy, translate it (cd pypy/translator/goal; ./translate.py -Ojit), and you get the C sources in /tmp/usession-yourname/testing_1. Then you can try to add a #define to rename all malloc() to tcmalloc(), or however it is called. PyPy does not use malloc() to allocate its own objects, but it still uses malloc() to get arenas, so in this point of view it is similar to CPython. A bient?t, Armin. From pypy at pocketnix.org Sun May 1 00:07:35 2011 From: pypy at pocketnix.org (pypy at pocketnix.org) Date: Sat, 30 Apr 2011 22:07:35 +0000 Subject: [pypy-dev] PyPy and memory In-Reply-To: References: Message-ID: <20110430220735.GA24306@pocketnix.org> On Sat, Apr 30, 2011 at 05:38:19PM +0200, Massimo Sala wrote: > What about PyPy vs plain CPython ? > > Please see : > http://pushingtheweb.com/2010/06/python-and-tcmalloc/ > > Is it possible for the maintainers to provide > - PyPy > - PyPy with tcmalloc > > so end-users can > - give a try > - do some tests in their different configs > - provide a feedback here on the mailing list > ? if i remember correctly the memory fragmentation was noted by the dropbox guys in http://pycon.blip.tv/file/4878722/ if i also remember correctly one of the last slides is a shout out to pypy indicating it doesn't have this problem if the video is incorrect let me know and i will dig through my archives for the correct video may have a quick peek at this anyway From rogerb at rogerbinns.com Sun May 1 11:14:35 2011 From: rogerb at rogerbinns.com (Roger Binns) Date: Sun, 01 May 2011 02:14:35 -0700 Subject: [pypy-dev] cpyext: Detecting pypy and other issues Message-ID: <4DBD247B.7040102@rogerbinns.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'm trying to port my APSW C extension to pypy and am encountering various issues. APSW is somewhat similar to pysqlite in that it wraps SQLite, but APSW wraps all of SQLite and provides SQLite's semantics rather than whatever DBAPI says. APSW supports Python 2.3 through 3.2. What is the correct way to detect pypy compilation? My code includes various enhancements and optimisations but they depend on CPython to some degree. These would need to be disabled at compile time. It looks like PYPY_VERSION being #defined will work but I'd rather use the "official" way whatever that is. One example optimisation I do is object reuse. When the refcount is 1 and it is dealloc then they go to a recycle list. Doing this gave several percentage point performance improvements. This can be disabled with a #define. An example of enhanced functionality is an alternative to PyErr_WriteUnraiseable. The CPython implementation is virtually useless (eg no traceback). My version shows the traceback but again depends on CPython internals to get it. http://code.google.com/p/apsw/source/browse/src/util.c#89 I need the api PyRun_StringFlags as it is used to execute a Python string and attach the results to the module. It is called like this: PyRun_StringFlags(char*, Py_file_input, apswdict, apswdict, NULL); I do have a benchmarking tool that mainly exposes the performance improvement of APSW over pysqlite. Most of the work of the wrapper is taking Python types (especially strings) and providing them to the SQLite APIs, and getting data back from SQLite and converting back to Python types. There is a lot of GIL acquisition and release. In order to improve performance there are also various string related caches. My test suite has 99,6% coverage of the C code so I should be able to find all the places where pypy differs. Roger -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk29JHMACgkQmOOfHg372QSTSQCgu3mi8osgcXgpcQ0SpMTJvnVu 7BQAoIxIZsaeq29QCqrJdFRarnvpjay0 =BZfM -----END PGP SIGNATURE----- From arigo at tunes.org Sun May 1 11:36:34 2011 From: arigo at tunes.org (Armin Rigo) Date: Sun, 1 May 2011 11:36:34 +0200 Subject: [pypy-dev] virtualenv 1.6.1 now released Message-ID: Hi, If you are using virtualenv, note that PyPy 1.5 requires virtualenv 1.6.1. It has been released a few hours *after* PyPy 1.5. For those who did not find it, it's there now :-) Armin. From rogerb at rogerbinns.com Sun May 1 12:08:12 2011 From: rogerb at rogerbinns.com (Roger Binns) Date: Sun, 01 May 2011 03:08:12 -0700 Subject: [pypy-dev] cpyext: Detecting pypy and other issues In-Reply-To: <4DBD247B.7040102@rogerbinns.com> References: <4DBD247B.7040102@rogerbinns.com> Message-ID: <4DBD310C.9090806@rogerbinns.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The website is totally silent on how to debug C extension issues. I'm encountering an issue where calling PyObject_GetAttrString results in pypy getting SIGSEGV. The first parameter is the module (return of Py_InitModule3) stored in a static variable. If I add an extra Py_INCREF to init then that goes away. It is behaving as though modules are created with a ref count of zero. Do I have to add an incref for everything that has a static pointer to it? My next problem is a list allocated with PyList_New(0). I later call PyList_GET_SIZE on it and get a SIGSEGV. I put an assertion before that of PyList_Check but that crashes too. Unfortunately I can't really do much with the pypy/gdb output: E RPython traceback: File "implement_1.c", line 52621, in PyList_Check File "implement_5.c", line 14533, in CpyTypedescr_realize Program received signal SIGSEGV, Segmentation fault. 0x000000000041118e in ?? () (gdb) bt #0 0x000000000041118e in ?? () #1 0x00000000004ab890 in ?? () #2 0x0000000000f47bfb in ?? () #3 0x000000000043f31f in PyList_Check () #4 0x00007ffff5410795 in Connection_close_internal (self=0x7ffff7f91030, force=1) at src/connection.c:166 #5 0x00007ffff5410b6d in Connection_close (self=0x7ffff7f91030, args=0x7ffff7f8ee60) at src/connection.c:274 #6 0x0000000000f47bfb in ?? () #7 0x0000000000729236 in ?? () ... #154 0x00007ffff5cfbeff in __libc_start_main (main=0x4118b0, argc=2, ubp_av=0x7fffffffe108, init=, fini=, rtld_fini=, stack_end=0x7fffffffe0f8) at libc-start.c:226 ... Roger -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk29MQcACgkQmOOfHg372QRBUgCfVaXt6c8S7CKkWlmSyzOUknIV FVMAniyQCLw7DWaElMF0AYiabbXqXsyV =1Ecf -----END PGP SIGNATURE----- From holger at merlinux.eu Sun May 1 12:20:14 2011 From: holger at merlinux.eu (holger krekel) Date: Sun, 1 May 2011 10:20:14 +0000 Subject: [pypy-dev] virtualenv 1.6.1 now released In-Reply-To: References: Message-ID: <20110501102014.GA4071@merlinux.eu> Hi Armin, all (also CC Jannis Leidel), On Sun, May 01, 2011 at 11:36 +0200, Armin Rigo wrote: > Hi, > > If you are using virtualenv, note that PyPy 1.5 requires virtualenv > 1.6.1. It has been released a few hours *after* PyPy 1.5. For those > who did not find it, it's there now :-) I still encounter a problem with virtualenv/pypy, see transcript on tannit below. it's basically a recursive use of virtualenv which however works nicely with CPython. best, holger (0)hpk at tannit:~$ virtualenv p0 --no-site-packages New python executable in p0/bin/python Installing distribute.................................................................................................................................................................................done. (0)hpk at tannit:~$ p0/bin/pip install virtualenv Downloading/unpacking virtualenv Using download cache from /home/hpk/.tox/downloads/http%3A%2F%2Fpypi.python.org%2Fpackages%2Fsource%2Fv%2Fvirtualenv%2Fvirtualenv-1.6.1.tar.gz Running setup.py egg_info for package virtualenv warning: no previously-included files matching '*.*' found under directory 'docs/_templates' Installing collected packages: virtualenv Running setup.py install for virtualenv warning: no previously-included files matching '*.*' found under directory 'docs/_templates' Installing virtualenv script to /home/hpk/p0/bin Successfully installed virtualenv Cleaning up... (0)hpk at tannit:~$ p0/bin/virtualenv p1 # cpython works fine Using real prefix '/usr' New python executable in p1/bin/python Overwriting p1/lib/python2.6/distutils/__init__.py with new content Installing setuptools............done. Installing pip...............done. (0)hpk at tannit:~$ p0/bin/virtualenv -p pypy p1 # this is pypy-1.5 and it fails Running virtualenv with interpreter /usr/local/bin/pypy New pypy executable in p1/bin/pypy Installing setuptools........ Complete output from command /home/hpk/p1/bin/pypy -c "#!python \"\"\"Bootstra...sys.argv[1:]) " --always-copy -U setuptools: Traceback (most recent call last): File "app_main.py", line 53, in run_toplevel File "app_main.py", line 526, in run_it File "", line 67, in ImportError: No module named md5 ---------------------------------------- ...Installing setuptools...done. Traceback (most recent call last): File "app_main.py", line 53, in run_toplevel File "/home/hpk/p0/lib/python2.6/site-packages/virtualenv.py", line 1965, in main() File "/home/hpk/p0/lib/python2.6/site-packages/virtualenv.py", line 795, in main never_download=options.never_download) File "/home/hpk/p0/lib/python2.6/site-packages/virtualenv.py", line 895, in create_environment search_dirs=search_dirs, never_download=never_download) File "/home/hpk/p0/lib/python2.6/site-packages/virtualenv.py", line 588, in install_setuptools search_dirs=search_dirs, never_download=never_download) File "/home/hpk/p0/lib/python2.6/site-packages/virtualenv.py", line 562, in _install_req cwd=cwd) File "/home/hpk/p0/lib/python2.6/site-packages/virtualenv.py", line 863, in call_subprocess % (cmd_desc, proc.returncode)) OSError: Command /home/hpk/p1/bin/pypy -c "#!python \"\"\"Bootstra...sys.argv[1:]) " --always-copy -U setuptools failed with error code 1 (0)hpk at tannit:~$ From arigo at tunes.org Sun May 1 16:35:35 2011 From: arigo at tunes.org (Armin Rigo) Date: Sun, 1 May 2011 16:35:35 +0200 Subject: [pypy-dev] cpyext: Detecting pypy and other issues In-Reply-To: <4DBD310C.9090806@rogerbinns.com> References: <4DBD247B.7040102@rogerbinns.com> <4DBD310C.9090806@rogerbinns.com> Message-ID: Hi Roger, A first warning: even if you manage to compile your C extension with PyPy, the interfacing is much slower than with CPython. Keep that in mind, given that according to what you write you care enough about performance to try to squeeze a few percents. On Sun, May 1, 2011 at 12:08 PM, Roger Binns wrote: > It looks like PYPY_VERSION being #defined will work > but I'd rather use the "official" way whatever that is. That's the "official" way. > I need the api PyRun_StringFlags as it is used to execute a Python string > and attach the results to the module. If it's not already provided, then you are welcome to contribute it to PyPy in pypy/module/cpyext/. It is known that we don't have 100% coverage even of Python.h (not to mention other headers like compile.h, which will probably never be supported). > Do I have to add an incref for everything that has a static pointer to it? It depends on details of exactly what you do, but it is likely. The exact rule is, like CPython, that any object is freed as soon as its reference counter goes down to zero; but unlike CPython, this does *not* include places like a reference stored in the module's dictionary. In other words, if it seems to work on CPython but not on PyPy, then it is likely that there is actually a (potentially very rare) bug in the CPython version too. For example, the object you have in the static variable could be meant to be kept alive by being also stored in the module's dictionary. If so, what if a user actually delete it from your module's dictionary? Then you get a crash on CPython too. > My next problem is a list allocated with PyList_New(0). ?I later call PyList_GET_SIZE on it > and get a SIGSEGV. I put an assertion before that of PyList_Check but that crashes too. >?Unfortunately I can't really do much with the pypy/gdb output: You get easier-to-read tracebacks if you run it without the JIT: "pypy --jit threshold=-1". Of course in order to get the details of, say, the local variables, then you need a custom pypy built with debugging symbols. A bient?t, Armin. From arigo at tunes.org Sun May 1 17:22:06 2011 From: arigo at tunes.org (Armin Rigo) Date: Sun, 1 May 2011 17:22:06 +0200 Subject: [pypy-dev] virtualenv 1.6.1 now released In-Reply-To: <20110501102014.GA4071@merlinux.eu> References: <20110501102014.GA4071@merlinux.eu> Message-ID: Hi Holger, On Sun, May 1, 2011 at 12:20 PM, holger krekel wrote: > I still encounter a problem with virtualenv/pypy, see transcript on > tannit below. I don't know virtualenv, but I tried to repeat the commands you typed (with virtualenv 1.6.1 and pypy 1.5 as released, on tannit 64-bit), and it works for me. I can only guess that you have some older virtualenv or some older pypy around. Armin From arigo at tunes.org Sun May 1 17:35:39 2011 From: arigo at tunes.org (Armin Rigo) Date: Sun, 1 May 2011 17:35:39 +0200 Subject: [pypy-dev] virtualenv 1.6.1 now released In-Reply-To: References: <20110501102014.GA4071@merlinux.eu> Message-ID: Hi all, On Sun, May 1, 2011 at 5:22 PM, Armin Rigo wrote: > I can only guess that you have some older virtualenv or some older pypy around. For reference, I just installed now the pypy 1.5 release as the default pypy on tannit and tannit32. A bient?t, Armin. From holger at merlinux.eu Sun May 1 20:07:54 2011 From: holger at merlinux.eu (holger krekel) Date: Sun, 1 May 2011 18:07:54 +0000 Subject: [pypy-dev] virtualenv 1.6.1 now released In-Reply-To: References: <20110501102014.GA4071@merlinux.eu> Message-ID: <20110501180754.GC4071@merlinux.eu> On Sun, May 01, 2011 at 17:22 +0200, Armin Rigo wrote: > Hi Holger, > > On Sun, May 1, 2011 at 12:20 PM, holger krekel wrote: > > I still encounter a problem with virtualenv/pypy, see transcript on > > tannit below. > > I don't know virtualenv, but I tried to repeat the commands you typed > (with virtualenv 1.6.1 and pypy 1.5 as released, on tannit 64-bit), > and it works for me. I can only guess that you have some older > virtualenv or some older pypy around. i use pypy 1.5 from opt. However, /usr/bin/virtualenv on tannit is at version 1.4.5 and maybe that is the problem. sidenote: the released pypy-1.5 shows this: (0)hpk at tannit:~$ /opt/pypy-1.5-linux64/bin/pypy Python 2.7.1 (b590cf6de419, Apr 30 2011, 02:00:34) [PyPy 1.5.0-alpha0 with GCC 4.4.3] on linux2 so the version string is still at "alpha0" FWIW. holger From arigo at tunes.org Sun May 1 20:17:13 2011 From: arigo at tunes.org (Armin Rigo) Date: Sun, 1 May 2011 20:17:13 +0200 Subject: [pypy-dev] virtualenv 1.6.1 now released In-Reply-To: <20110501180754.GC4071@merlinux.eu> References: <20110501102014.GA4071@merlinux.eu> <20110501180754.GC4071@merlinux.eu> Message-ID: Hi Holger, On Sun, May 1, 2011 at 8:07 PM, holger krekel wrote: > i use pypy 1.5 from opt. ?However, /usr/bin/virtualenv on tannit is at > version 1.4.5 and maybe that is the problem. I assumed that you got virtualenv 1.6.1, because you replied to my message that was saying precisely "virtualenv 1.6.1 is needed, use that version". :-) I will update it on tannit too. Armin From holger at merlinux.eu Sun May 1 20:39:14 2011 From: holger at merlinux.eu (holger krekel) Date: Sun, 1 May 2011 18:39:14 +0000 Subject: [pypy-dev] virtualenv 1.6.1 now released In-Reply-To: References: <20110501102014.GA4071@merlinux.eu> <20110501180754.GC4071@merlinux.eu> Message-ID: <20110501183914.GE4071@merlinux.eu> On Sun, May 01, 2011 at 20:17 +0200, Armin Rigo wrote: > Hi Holger, > > On Sun, May 1, 2011 at 8:07 PM, holger krekel wrote: > > i use pypy 1.5 from opt. ?However, /usr/bin/virtualenv on tannit is at > > version 1.4.5 and maybe that is the problem. > > I assumed that you got virtualenv 1.6.1, because you replied to my > message that was saying precisely "virtualenv 1.6.1 is needed, use > that version". :-) there were two virtualenv invocations. The second one was with 1.6.1 and pypy but the first one was with 1.4.5 and cpython. Maybe that kind of nesting is just not supported across venv and python versions. holger > I will update it on tannit too. > > Armin > From rogerb at rogerbinns.com Mon May 2 06:55:50 2011 From: rogerb at rogerbinns.com (Roger Binns) Date: Sun, 01 May 2011 21:55:50 -0700 Subject: [pypy-dev] cpyext: Detecting pypy and other issues In-Reply-To: References: <4DBD247B.7040102@rogerbinns.com> <4DBD310C.9090806@rogerbinns.com> Message-ID: <4DBE3956.90409@rogerbinns.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 05/01/2011 07:35 AM, Armin Rigo wrote: > A first warning: even if you manage to compile your C extension with > PyPy, the interfacing is much slower than with CPython. Keep that in > mind, given that according to what you write you care enough about > performance to try to squeeze a few percents. First you make it work right then you make it work fast :-) Even if it is 10% slower than CPython the rest of pypy will be more than 10% faster so things will still be better overall. Worst case some combination of my code, my tests and pypy will be improved by trying to do the port. >> I need the api PyRun_StringFlags as it is used to execute a Python string >> and attach the results to the module. > > If it's not already provided, then you are welcome to contribute it to > PyPy in pypy/module/cpyext/. It is known that we don't have 100% > coverage even of Python.h (not to mention other headers like > compile.h, which will probably never be supported). It would be helpful if the cpyext site said what to do in these situations. For example add missing functions to a wiki page or create a ticket or send to mailing list etc. ie what actions should developers like me take when discovering things like this that ensures the smoothest running of the project. (Obviously just fixing it is good but we aren't all in a position to do that at the moment.) PyErr_Display is another missing function which is widely used, but never documented in the Python docs for some reason. The pypy site also doesn't say where to communicate for cpyext - I assumed this mailing list was the right place. >> Do I have to add an incref for everything that has a static pointer to it? > > It depends on details of exactly what you do, but it is likely. The > exact rule is, like CPython, that any object is freed as soon as its > reference counter goes down to zero; but unlike CPython, this does > *not* include places like a reference stored in the module's > dictionary. Note my problem is not with contents of the module, but with the module itself. static PyObject *themodule=Py_InitModule3(...); Later in one of the methods called back from Python I call PyObject_AttrString(themodule, "foo") which is crashing. If I incref themodule in init then this problem doesn't occur. The behaviour is as though Py_InitModule3 is setting the starting refcount to zero, or with the module's refcount becoming zero inside pypy which then begs the question of how my callback could have been called. > In other words, if it seems to work on CPython but not on > PyPy, then it is likely that there is actually a (potentially very > rare) bug in the CPython version too. For my tests I build a full debug version of CPython will all memory optimisations disabled (quite a feat since it squirrels away stuff all over the place) and then run everything under valgrind. Combined with 100% coverage in almost all source files I'm reasonably confident that things are coded well, although there may be some luck :-) > For example, the object you > have in the static variable could be meant to be kept alive by being > also stored in the module's dictionary. I did just double check my code and every item that I do PyModule_AddObject with is also incref first. That was some diligent coding however many years ago when I wrote it :-) > You get easier-to-read tracebacks if you run it without the JIT: "pypy > --jit threshold=-1". Of course in order to get the details of, say, > the local variables, then you need a custom pypy built with debugging > symbols. Unfortunately disabling the JIT wasn't much help. My goal is to produce the best bug report possible and use the least amount of the pypy team's time. (It is a lot easier to fix and understand things starting from a good bug report.) What would be most helpful is if the cpyext page gave some instructions to follow in particular making an appropriate pypy and using valgrind. (Usually making valgrind work well requires extra options above and beyond regular debugging builds.) Roger -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk2+OVEACgkQmOOfHg372QTkpQCfc+DL/GFDZCpyMsNEsx5tRhvY fIUAn0z1EQ+NxZLQ8PwaouXXSRy4JCTo =vu+1 -----END PGP SIGNATURE----- From anto.cuni at gmail.com Mon May 2 08:30:18 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Mon, 02 May 2011 08:30:18 +0200 Subject: [pypy-dev] cpyext: Detecting pypy and other issues In-Reply-To: <4DBE3956.90409@rogerbinns.com> References: <4DBD247B.7040102@rogerbinns.com> <4DBD310C.9090806@rogerbinns.com> <4DBE3956.90409@rogerbinns.com> Message-ID: <4DBE4F7A.3030109@gmail.com> Hello Roger, I'm not "the" cpyext expert here, so I'll let the others to answer your specific questions. However: On 02/05/11 06:55, Roger Binns wrote: > Unfortunately disabling the JIT wasn't much help. My goal is to produce the > best bug report possible and use the least amount of the pypy team's time. > (It is a lot easier to fix and understand things starting from a good bug > report.) What would be most helpful is if the cpyext page gave some > instructions to follow in particular making an appropriate pypy and using > valgrind. (Usually making valgrind work well requires extra options above > and beyond regular debugging builds.) you are right, we lack such a document. Do you feel like writing it? :-) (seriously: you are by far in a better position than us for knowing what could or could not help a newcomer. Even if you don't know the answer of all the questions, having a document with XXX that we can fix would be valuable). ciao, Anto From arigo at tunes.org Mon May 2 12:12:07 2011 From: arigo at tunes.org (Armin Rigo) Date: Mon, 2 May 2011 12:12:07 +0200 Subject: [pypy-dev] cpyext: Detecting pypy and other issues In-Reply-To: <4DBE3956.90409@rogerbinns.com> References: <4DBD247B.7040102@rogerbinns.com> <4DBD310C.9090806@rogerbinns.com> <4DBE3956.90409@rogerbinns.com> Message-ID: Hi Roger, On Mon, May 2, 2011 at 6:55 AM, Roger Binns wrote: > The pypy site also doesn't say where to communicate for cpyext - I assumed > this mailing list was the right place. Yes. > ?static PyObject *themodule=Py_InitModule3(...); > > Later in one of the methods called back from Python I call > PyObject_AttrString(themodule, "foo") which is crashing. ?If I incref > themodule in init then this problem doesn't occur. Indeed, it's an edge case. What you must realize is that your module object is of course kept alive in PyPy, even if its refcount drops to zero. More precisely, in PyPy, every object accessed by the C extensions is actually accessed via a "wrapper", which is declared as PyObject. A wrapper holds just the refcount and a reference to the real object, which is not a PyObject at all (it's an object that can be moved by our GC, to start with). When the refcount drops to zero we assume that the C extension doesn't need the object any more, and we free the PyObject wrapper. This may or may not cause the real object within PyPy to be freed. It may even be the case that later we want again to pass a reference to the same object to the C extension; in this case, we make a new wrapper. So the refcount of -- in your case -- the module object is never zero; it's just that as soon as the main function of your C extension module finishes, the refcount is decremented and reaches zero, so the PyObject wrapper is freed. It is arguably a bug in your module, even though in that case it is hard to write an example that triggers the bug, because it is impossible in CPython to free a loaded extension module. The point is that if it is was possible, then doing so in CPython would cause the module to see its refcount drop to zero, and be freed, which makes the pointer in the static global variable invalid. That's what I meant by saying that such issues are, somehow, showing bugs in the C extension module, although admittedly they are often "bugs" that cannot possibly cause an issue in practice. A bient?t, Armin. From dimaqq at gmail.com Mon May 2 20:14:45 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Mon, 2 May 2011 11:14:45 -0700 Subject: [pypy-dev] cpyext reference counting and other gc's In-Reply-To: References: Message-ID: Sure, I started with a basic concept of having as few dependencies in pypy rpython code as possible, that is pypy sees object defined/created in c extension as native. Granted, I didn't check all the details yet, e.g. cpython protocols probably need to be wrapped in cpyext anyway. Dunno if possible in general case, I still need to investigate the hack I'm thinking of here. I looked through code for quite a few cpython c extensions, most make only minimal use of cpython api. Moreover it occurs to me it is important to support sheer number of simple cpython extensions (rationale is that python is commonly used to glue large-ish software modules together). The few extensions that do something non-trivial would probably need to be patched to take full advantage of pypy anyway. The consequence is Py* "symbols" (C source uses those symbolically, we are free to inject macros or define functions) can become quite complex functions. If c extension writer assumes that incref is a very simple operation while repeatedly calling a user-defined function is complex and overoptimizes their extension with this view, the results could be funny :P Please correct me if I misunderstood cpyext, I'd be willing to write up cpyext on pypy wiki. Current cpyext creates a shadow object for every pointer that is passed to or from C and keeps this object for every pointer that is used in C internally, lifetime seen through object's reference count. My concept is to shadow only the reference count. When an object crosses pypy-C boundary, current cpyext has to perform an allocation or lookup anyway, it is rather similar with my proposal, perhaps my proposal is even cheaper here. As far as incref/decref is concerned, only first lookup is potentially slow, subsequent lookups are in cpu cache anyway, moreover if there's significant traffic to the reference count dict, whole dict is probably cached. Another possible advantage is unloading C extension modules as we can attribute privately stored python objects to the module that holds them; although as this is not done in cpython, most modules are probably broken. Cheers, Dima Tisnek p.s. I'd really like to know what are reasons some c extension do not work with cpyext right now. can anyone weigh in on dict/custom data structure lookup, e.g. splay tree vs pointer-chasing linked shadow objects? is there a better data structure than a has table (void* -> ssize_t) On 26 April 2011 09:23, Armin Rigo wrote: > Hi Dima, > > On Mon, Apr 25, 2011 at 9:53 PM, Dima Tisnek wrote: >> https://docs.google.com/document/d/1k7t-WIsfKW4tIL9i8-7Y6_9lo18wcsibyDONOF2i_l8/edit?hl=en > > Can you explain a bit more what are the advantages of the solution you > propose, compared to what is already implemented in cpyext? ?Your > description is far too high-level for us to know exactly what you > mean. > > It seems that you want to replace the currently implemented solution > with a very different one. ?I can explain it in a bit more details, > but I would first like to hear what goal you are trying to achieve. > Here is a quick reply based on guessing. ?The issue with your version > is that Py_INCREF() and Py_DECREF() needs to do a slow dictionary > lookup, while ours doesn't. ?Conversely, I believe that your version > doesn't need a dictionary lookup in other cases where ours needs to. > However it seems to me that if you add so much overhead to Py_INCREF() > and Py_DECREF(), you loose all other speed advantages. > > > A bient?t, > > Armin. > From amauryfa at gmail.com Mon May 2 20:44:12 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Mon, 2 May 2011 20:44:12 +0200 Subject: [pypy-dev] cpyext reference counting and other gc's In-Reply-To: References: Message-ID: Hi, 2011/5/2 Dima Tisnek : > I'd really like to know what are reasons some c extension do not work > with cpyext right now. I can see three kinds of reasons: - Reference count mistakes, that normally don't show up in CPython (but many c extensions do crash when you delete them from sys.modules, then reimport them; cpyext just detect the failure the first time :-) - unsupported functions, either because we never encountered them, or because they are really hard to support in pypy - extensions that play too much with CPython inter nals: numpy, Cython are in this case > can anyone weigh in on dict/custom data structure lookup, e.g. splay > tree vs pointer-chasing linked shadow objects? > is there a better data structure than a has table (void* -> ssize_t) Probably! The hash table is used because it's readily available in RPython. But if you care to provide a RPython implementation of the associative container, I'd be happy to test it. -- Amaury Forgeot d'Arc From dimaqq at gmail.com Mon May 2 20:53:11 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Mon, 2 May 2011 11:53:11 -0700 Subject: [pypy-dev] cpyext reference counting and other gc's In-Reply-To: References: Message-ID: Hi Amaury, thanks for a quick reply, btw, which api functions are hard to support in pypy or why some are? d. On 2 May 2011 11:44, Amaury Forgeot d'Arc wrote: > Hi, > > 2011/5/2 Dima Tisnek : >> I'd really like to know what are reasons some c extension do not work >> with cpyext right now. > > I can see three kinds of reasons: > > - Reference count mistakes, that normally don't show up in CPython > (but many c extensions do crash when you delete them from sys.modules, > then reimport them; cpyext just detect the failure the first time :-) > > - unsupported functions, either because we never encountered them, > or because they are really hard to support in pypy > > - extensions that play too much with CPython inter > nals: numpy, Cython > are in this case > >> can anyone weigh in on dict/custom data structure lookup, e.g. splay >> tree vs pointer-chasing linked shadow objects? >> is there a better data structure than a has table (void* -> ssize_t) > > Probably! The hash table is used because it's readily available in > RPython. But if you care to provide a RPython implementation of > the associative container, I'd be happy to test it. > > -- > Amaury Forgeot d'Arc > From amauryfa at gmail.com Mon May 2 21:09:31 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Mon, 2 May 2011 21:09:31 +0200 Subject: [pypy-dev] cpyext reference counting and other gc's In-Reply-To: References: Message-ID: 2011/5/2 Dima Tisnek : > Hi Amaury, thanks for a quick reply, btw, which api functions are hard > to support in pypy or why some are? Fortunately, there are not so many of them: - PyFile_AsFile(), PyFile_FromFile, because files opened by pypy don't use a FILE* (like python3) - PyThreadState creation and suppression - PyInterpreter creation and suppression - Py_Initialize and Py_Finalize (to embed a python interpreter inside an application) - Some trace and Traceback management functions that are not even documented (but used by Cython :-)) -- Amaury Forgeot d'Arc From rogerb at rogerbinns.com Mon May 2 22:36:48 2011 From: rogerb at rogerbinns.com (Roger Binns) Date: Mon, 02 May 2011 13:36:48 -0700 Subject: [pypy-dev] cpyext: Detecting pypy and other issues In-Reply-To: <4DBE4F7A.3030109@gmail.com> References: <4DBD247B.7040102@rogerbinns.com> <4DBD310C.9090806@rogerbinns.com> <4DBE3956.90409@rogerbinns.com> <4DBE4F7A.3030109@gmail.com> Message-ID: <4DBF15E0.6080005@rogerbinns.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 05/01/2011 11:30 PM, Antonio Cuni wrote: > you are right, we lack such a document. Do you feel like writing it? :-) I'll make an initial stab at it this week. Roger -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk2/FdwACgkQmOOfHg372QSeZQCeJ7P/5VsMxno6j2eunziJtzAC jHIAoIpPgm4jPHOERSKq4OVJLt/lv+MA =uSH6 -----END PGP SIGNATURE----- From rogerb at rogerbinns.com Mon May 2 23:04:48 2011 From: rogerb at rogerbinns.com (Roger Binns) Date: Mon, 02 May 2011 14:04:48 -0700 Subject: [pypy-dev] cpyext: Detecting pypy and other issues In-Reply-To: References: <4DBD247B.7040102@rogerbinns.com> <4DBD310C.9090806@rogerbinns.com> <4DBE3956.90409@rogerbinns.com> Message-ID: <4DBF1C70.6040907@rogerbinns.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 05/02/2011 03:12 AM, Armin Rigo wrote: > It is arguably a bug in your module, The rule seems to be that if the C code saves the value of *any* PyObject* in a static variable then an incref must be done first. I have actually been doing this for everything (including PyTypeObject) except for the module object itself. > The point is that if it is was possible, then doing so in > CPython would cause the module to see its refcount drop to zero, and > be freed, which makes the pointer in the static global variable > invalid. This wouldn't have affected me. There is no way you could have called a function provided by the module if the module itself had been freed. My other issue is happening where the object struct has a PyObject * that is allocated with PyList_New(0) in the _new method. Later on PyList_ methods crash. The debugger is showing a refcount of 140737353674481 so something has gone horribly wrong ... Roger -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk2/HGwACgkQmOOfHg372QT9qgCfWvr9rjt0jsV0r/2wnWk5Tq3Q b+UAniNU0scFslq1TqQeowa/XZYICxh3 =HyOO -----END PGP SIGNATURE----- From rogerb at rogerbinns.com Mon May 2 23:21:46 2011 From: rogerb at rogerbinns.com (Roger Binns) Date: Mon, 02 May 2011 14:21:46 -0700 Subject: [pypy-dev] cpyext reference counting and other gc's In-Reply-To: References: Message-ID: <4DBF206A.70807@rogerbinns.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 > - Some trace and Traceback management functions that are not even > documented (but used by Cython :-)) I use them too. One of the problems with the standard documented C api is that tracebacks do not include any methods implemented in C. For example if a Python method calls a C method which calls a Python method which errors then the traceback won't include the C code. This is very confusing if you don't know why the C code called a Python method. I show this in the doc for my project: http://apidoc.apsw.googlecode.com/hg/exceptions.html#augmented-stack-traces As the doc shows I end up adding in synthetic stack frames so you can clearly see the C code. I also augment the synthetic frames with local variables that can be introspected to find out what is going on. The very bottom of the page shows what a difference that makes. You can see the C code to do it here: http://code.google.com/p/apsw/source/browse/src/traceback.c Note use of functions like PyThreadState_Get() and PyTraceback_here. As long as the signature of AddTraceBackHere can remain the same then I don't care what the body inside is for pypy. Roger -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk2/IGYACgkQmOOfHg372QRzAgCglUdQlOoF3fQBJAAQ7Lf8vRs2 4OMAn1weWlNDNtHHcccB1vjxSvn7wk59 =jTAS -----END PGP SIGNATURE----- From amauryfa at gmail.com Tue May 3 00:08:42 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 3 May 2011 00:08:42 +0200 Subject: [pypy-dev] cpyext reference counting and other gc's In-Reply-To: <4DBF206A.70807@rogerbinns.com> References: <4DBF206A.70807@rogerbinns.com> Message-ID: Hi, 2011/5/2 Roger Binns : > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > >> - Some trace and Traceback management functions that are not even >> documented (but used by Cython :-)) > > I use them too. ?One of the problems with the standard documented C api is > that tracebacks do not include any methods implemented in C. ?For example if > a Python method calls a C method which calls a Python method which errors > then the traceback won't include the C code. ?This is very confusing if you > don't know why the C code called a Python method. > > I show this in the doc for my project: > > ?http://apidoc.apsw.googlecode.com/hg/exceptions.html#augmented-stack-traces > > As the doc shows I end up adding in synthetic stack frames so you can > clearly see the C code. ?I also augment the synthetic frames with local > variables that can be introspected to find out what is going on. ?The very > bottom of the page shows what a difference that makes. > > You can see the C code to do it here: > > ?http://code.google.com/p/apsw/source/browse/src/traceback.c > > Note use of functions like PyThreadState_Get() and PyTraceback_here. ?As > long as the signature of AddTraceBackHere can remain the same then I don't > care what the body inside is for pypy. Yes, we've implemented PyTraceback_here so that it works exactly for this usage. Can you check whether pypy does the right thing for you as well? -- Amaury Forgeot d'Arc From amauryfa at gmail.com Tue May 3 00:08:42 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 3 May 2011 00:08:42 +0200 Subject: [pypy-dev] cpyext reference counting and other gc's In-Reply-To: <4DBF206A.70807@rogerbinns.com> References: <4DBF206A.70807@rogerbinns.com> Message-ID: Hi, 2011/5/2 Roger Binns : > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > >> - Some trace and Traceback management functions that are not even >> documented (but used by Cython :-)) > > I use them too. ?One of the problems with the standard documented C api is > that tracebacks do not include any methods implemented in C. ?For example if > a Python method calls a C method which calls a Python method which errors > then the traceback won't include the C code. ?This is very confusing if you > don't know why the C code called a Python method. > > I show this in the doc for my project: > > ?http://apidoc.apsw.googlecode.com/hg/exceptions.html#augmented-stack-traces > > As the doc shows I end up adding in synthetic stack frames so you can > clearly see the C code. ?I also augment the synthetic frames with local > variables that can be introspected to find out what is going on. ?The very > bottom of the page shows what a difference that makes. > > You can see the C code to do it here: > > ?http://code.google.com/p/apsw/source/browse/src/traceback.c > > Note use of functions like PyThreadState_Get() and PyTraceback_here. ?As > long as the signature of AddTraceBackHere can remain the same then I don't > care what the body inside is for pypy. Yes, we've implemented PyTraceback_here so that it works exactly for this usage. Can you check whether pypy does the right thing for you as well? -- Amaury Forgeot d'Arc From amauryfa at gmail.com Tue May 3 00:12:44 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 3 May 2011 00:12:44 +0200 Subject: [pypy-dev] cpyext: Detecting pypy and other issues In-Reply-To: <4DBF1C70.6040907@rogerbinns.com> References: <4DBD247B.7040102@rogerbinns.com> <4DBD310C.9090806@rogerbinns.com> <4DBE3956.90409@rogerbinns.com> <4DBF1C70.6040907@rogerbinns.com> Message-ID: 2011/5/2 Roger Binns : > There is no way you could have called a > function provided by the module if the module itself had been freed. Are you sure? from mymodule import func del sys.modules['mymodule'] func() -- Amaury Forgeot d'Arc From dmalcolm at redhat.com Mon May 2 23:49:09 2011 From: dmalcolm at redhat.com (David Malcolm) Date: Mon, 02 May 2011 17:49:09 -0400 Subject: [pypy-dev] PyPy 1.5 released In-Reply-To: References: Message-ID: <1304372949.2063.5.camel@radiator.bos.redhat.com> On Sat, 2011-04-30 at 17:04 +0200, Armin Rigo wrote: > ====================== > PyPy 1.5: Catching Up > ====================== > > We're pleased to announce the 1.5 release of PyPy. This release updates > PyPy with the features of CPython 2.7.1, including the standard library. Thus > all the features of `CPython 2.6`_ and `CPython 2.7`_ are now supported. It > also contains additional performance improvements. You can download it here: > > http://pypy.org/download.html FWIW, I've now updated Fedora's RPM packages of PyPy to pypy-1.5. See https://bugzilla.redhat.com/show_bug.cgi?id=701121 for links to the builds for various releases/architectures. Enjoy! Dave From exarkun at twistedmatrix.com Tue May 3 00:51:26 2011 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Mon, 02 May 2011 22:51:26 -0000 Subject: [pypy-dev] cpyext reference counting and other gc's In-Reply-To: References: Message-ID: <20110502225126.1992.2104890452.divmod.xquotient.1011@localhost.localdomain> On 07:09 pm, amauryfa at gmail.com wrote: >2011/5/2 Dima Tisnek : >>Hi Amaury, thanks for a quick reply, btw, which api functions are hard >>to support in pypy or why some are? > >Fortunately, there are not so many of them: >- PyFile_AsFile(), PyFile_FromFile, because files opened by pypy don't >use a FILE* (like python3) Do fdopen(3) and fileno(3) not help here? I can understand how there might be synchronization issues if it were implemented this way... I've never used these CPython APIs myself. You didn't have the memoryview APIs on your list. Does that mean they're easy? Then can you implement them? ;) Jean-Paul From rogerb at rogerbinns.com Tue May 3 01:41:07 2011 From: rogerb at rogerbinns.com (Roger Binns) Date: Mon, 02 May 2011 16:41:07 -0700 Subject: [pypy-dev] cpyext reference counting and other gc's In-Reply-To: References: <4DBF206A.70807@rogerbinns.com> Message-ID: <4DBF4113.1020900@rogerbinns.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 05/02/2011 03:08 PM, Amaury Forgeot d'Arc wrote: > Yes, we've implemented PyTraceback_here so that it works exactly for this usage. > Can you check whether pypy does the right thing for you as well? I will once I can get anything actually working. Currently having problems with pypy crashing. (The code does compile though.) Note that I am perfectly happy to change my code to something that works with pypy - I already had to make it work the different ways that Python 2 and 3 do. Roger -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk2/QQ4ACgkQmOOfHg372QRujACeLLylS1qh8jBCc3lmqNmGHvSn EHEAoI+3P8B8iuCIsmkhfl/0Bn1qdggc =8j1u -----END PGP SIGNATURE----- From rogerb at rogerbinns.com Tue May 3 07:04:38 2011 From: rogerb at rogerbinns.com (Roger Binns) Date: Mon, 02 May 2011 22:04:38 -0700 Subject: [pypy-dev] cpyext reference counting and other gc's In-Reply-To: <20110502225126.1992.2104890452.divmod.xquotient.1011@localhost.localdomain> References: <20110502225126.1992.2104890452.divmod.xquotient.1011@localhost.localdomain> Message-ID: <4DBF8CE6.9050906@rogerbinns.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 05/02/2011 03:51 PM, exarkun at twistedmatrix.com wrote: > On 07:09 pm, amauryfa at gmail.com wrote: >> Fortunately, there are not so many of them: >> - PyFile_AsFile(), PyFile_FromFile, because files opened by pypy don't >> use a FILE* (like python3) > > Do fdopen(3) and fileno(3) not help here? That gets you the OS handle but the FILE* has a whole bunch of other crud inside like a buffer, current offsets and eol handling. If it is just a one time transfer from Python to C or vice versa then it could work, but if both use the file/FILE concurrently then chances are you'll end up with file corruption. Roger -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk2/jOEACgkQmOOfHg372QTLWACdHs8UC5UBeby+kONqFV9Q8Jar sKAAn1EB8cXK8wZYWjOEnInTAYcSSXIQ =9/ZX -----END PGP SIGNATURE----- From fijall at gmail.com Tue May 3 08:50:39 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 3 May 2011 08:50:39 +0200 Subject: [pypy-dev] PyPy 1.5 released In-Reply-To: <1304372949.2063.5.camel@radiator.bos.redhat.com> References: <1304372949.2063.5.camel@radiator.bos.redhat.com> Message-ID: On Mon, May 2, 2011 at 11:49 PM, David Malcolm wrote: > On Sat, 2011-04-30 at 17:04 +0200, Armin Rigo wrote: >> ====================== >> PyPy 1.5: Catching Up >> ====================== >> >> We're pleased to announce the 1.5 release of PyPy. This release updates >> PyPy with the features of CPython 2.7.1, including the standard library. Thus >> all the features of `CPython 2.6`_ and `CPython 2.7`_ are now supported. It >> also contains additional performance improvements. You can download it here: >> >> ? ? http://pypy.org/download.html > > FWIW, I've now updated Fedora's RPM packages of PyPy to pypy-1.5. > > See https://bugzilla.redhat.com/show_bug.cgi?id=701121 for links to the > builds for various releases/architectures. Cool! Thanks. Arch linux was first btw ;-) > > Enjoy! > Dave > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From ondrej at certik.cz Tue May 3 09:22:44 2011 From: ondrej at certik.cz (Ondrej Certik) Date: Tue, 3 May 2011 00:22:44 -0700 Subject: [pypy-dev] speed of sympy tests Message-ID: Hi, I have downloaded the pypy 1.5 binary (with jit) and run tests (only sympy/core, so that it's fast) on Ubuntu Natty, 64bit: ondrej at eagle:~/repos/sympy(master)$ bin/test sympy/core/ ============================= test process starts ============================== executable: /usr/bin/python (2.7.1-final-0) ground types: python sympy/core/tests/test_arit.py[48] ...f.......................................... .. [OK] sympy/core/tests/test_assumptions.py[28] ....f....................... [OK] sympy/core/tests/test_basic.py[10] .......... [OK] sympy/core/tests/test_cache.py[1] . [OK] sympy/core/tests/test_complex.py[13] ............. [OK] sympy/core/tests/test_containers.py[5] ..... [OK] sympy/core/tests/test_count_ops.py[2] .. [OK] sympy/core/tests/test_diff.py[6] ...... [OK] sympy/core/tests/test_equal.py[5] ..... [OK] sympy/core/tests/test_eval.py[8] .....f.. [OK] sympy/core/tests/test_eval_power.py[12] ............ [OK] sympy/core/tests/test_evalf.py[23] ....................... [OK] sympy/core/tests/test_expand.py[6] ...... [OK] sympy/core/tests/test_expr.py[60] .............................................. .............. [OK] sympy/core/tests/test_exprtools.py[4] .... [OK] sympy/core/tests/test_facts.py[11] ........... [OK] sympy/core/tests/test_functions.py[23] .....f................. [OK] sympy/core/tests/test_logic.py[11] ........... [OK] sympy/core/tests/test_match.py[26] ...f...................... [OK] sympy/core/tests/test_numbers.py[47] ........................................... .... [OK] sympy/core/tests/test_operations.py[4] .... [OK] sympy/core/tests/test_priority.py[5] ..... [OK] sympy/core/tests/test_relational.py[7] ....... [OK] sympy/core/tests/test_sets.py[18] .................. [OK] sympy/core/tests/test_subs.py[30] .............................. [OK] sympy/core/tests/test_symbol.py[9] ......... [OK] sympy/core/tests/test_sympify.py[23] ....................... [OK] sympy/core/tests/test_truediv.py[3] ... [OK] sympy/core/tests/test_var.py[5] ..... [OK] ======= tests finished: 448 passed, 5 expected to fail, in 2.90 seconds ======== ondrej at eagle:~/repos/sympy(master)$ ~/Downloads/pypy-c-jit-43780-b590cf6de419-linux64/bin/pypy bin/test sympy/core/ ============================= test process starts ============================== executable: /home/ondrej/Downloads/pypy-c-jit-43780-b590cf6de419-linux64/bin/pypy (2.7.1-final-42) ground types: python sympy/core/tests/test_arit.py[48] ...f.......................................... .. [OK] sympy/core/tests/test_assumptions.py[28] ....f....................... [OK] sympy/core/tests/test_basic.py[10] .......... [OK] sympy/core/tests/test_cache.py[1] . [OK] sympy/core/tests/test_complex.py[13] ............. [OK] sympy/core/tests/test_containers.py[5] ..... [OK] sympy/core/tests/test_count_ops.py[2] .. [OK] sympy/core/tests/test_diff.py[6] ...... [OK] sympy/core/tests/test_equal.py[5] ..... [OK] sympy/core/tests/test_eval.py[8] .....f.. [OK] sympy/core/tests/test_eval_power.py[12] ............ [OK] sympy/core/tests/test_evalf.py[23] ....................... [OK] sympy/core/tests/test_expand.py[6] ...... [OK] sympy/core/tests/test_expr.py[60] ..........F................................... .............. [FAIL] sympy/core/tests/test_exprtools.py[4] .... [OK] sympy/core/tests/test_facts.py[11] ........... [OK] sympy/core/tests/test_functions.py[23] .....f................. [OK] sympy/core/tests/test_logic.py[11] ........... [OK] sympy/core/tests/test_match.py[26] ...f...................... [OK] sympy/core/tests/test_numbers.py[47] ........................................... .... [OK] sympy/core/tests/test_operations.py[4] .... [OK] sympy/core/tests/test_priority.py[5] ..... [OK] sympy/core/tests/test_relational.py[7] ....... [OK] sympy/core/tests/test_sets.py[18] .................. [OK] sympy/core/tests/test_subs.py[30] .............................. [OK] sympy/core/tests/test_symbol.py[9] ......... [OK] sympy/core/tests/test_sympify.py[23] ....................... [OK] sympy/core/tests/test_truediv.py[3] ... [OK] sympy/core/tests/test_var.py[5] ..... [OK] ________________________________________________________________________________ _____________ sympy/core/tests/test_expr.py:test_as_leading_term3 ______________ File "/home/ondrej/repos/sympy/sympy/core/tests/test_expr.py", line 212, in test_as_leading_term3 assert (2*x+pi*x+x**2).as_leading_term(x) == (2+pi)*x AssertionError == tests finished: 447 passed, 1 failed, 5 expected to fail, in 13.87 seconds == DO *NOT* COMMIT! The test failure is probably some minor difference between CPython and PyPy.. However, as you can see, the test execution is 4x slower in PyPy, compared to CPython. Is this the supposed slow down, or is there something specific to sympy? I have also done this test in bin/isympy, CPython: >>> from timeit import default_timer as clock >>> t = clock(); a = expand((1+x+y+z)**20); clock()-t 1.4584941864 PyPy: >>> from timeit import default_timer as clock >>> t = clock(); a = expand((1+x+y+z)**20); clock()-t 4.48898005486 so PyPy is roughly 3x slower here. So 3x or 4x slower, I think that's already very usable. Especially the sandbox pypy is an awesome idea, for serving web apps. The only problem is that I use Cython a lot, which I think is incompatible with PyPy... Ondrej From fijall at gmail.com Tue May 3 09:31:39 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 3 May 2011 09:31:39 +0200 Subject: [pypy-dev] speed of sympy tests In-Reply-To: References: Message-ID: On Tue, May 3, 2011 at 9:22 AM, Ondrej Certik wrote: > Hi, > > I have downloaded the pypy 1.5 binary (with jit) and run tests (only > sympy/core, so that it's fast) on Ubuntu Natty, 64bit: > > > ondrej at eagle:~/repos/sympy(master)$ bin/test sympy/core/ > ============================= test process starts ============================== > executable: ? /usr/bin/python ?(2.7.1-final-0) > ground types: python > > sympy/core/tests/test_arit.py[48] ...f.......................................... > .. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_assumptions.py[28] ....f....................... ? ? ? [OK] > sympy/core/tests/test_basic.py[10] .......... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] > sympy/core/tests/test_cache.py[1] . ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] > sympy/core/tests/test_complex.py[13] ............. ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_containers.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_count_ops.py[2] .. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_diff.py[6] ...... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] > sympy/core/tests/test_equal.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] > sympy/core/tests/test_eval.py[8] .....f.. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] > sympy/core/tests/test_eval_power.py[12] ............ ? ? ? ? ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_evalf.py[23] ....................... ? ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_expand.py[6] ...... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] > sympy/core/tests/test_expr.py[60] .............................................. > .............. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_exprtools.py[4] .... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_facts.py[11] ........... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_functions.py[23] .....f................. ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_logic.py[11] ........... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_match.py[26] ...f...................... ? ? ? ? ? ? ? [OK] > sympy/core/tests/test_numbers.py[47] ........................................... > .... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_operations.py[4] .... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] > sympy/core/tests/test_priority.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_relational.py[7] ....... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_sets.py[18] .................. ? ? ? ? ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_subs.py[30] .............................. ? ? ? ? ? ?[OK] > sympy/core/tests/test_symbol.py[9] ......... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_sympify.py[23] ....................... ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_truediv.py[3] ... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] > sympy/core/tests/test_var.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] > > ======= tests finished: 448 passed, 5 expected to fail, in 2.90 seconds ======== > ondrej at eagle:~/repos/sympy(master)$ > ~/Downloads/pypy-c-jit-43780-b590cf6de419-linux64/bin/pypy bin/test > sympy/core/ > ============================= test process starts ============================== > executable: ? /home/ondrej/Downloads/pypy-c-jit-43780-b590cf6de419-linux64/bin/pypy > ?(2.7.1-final-42) > ground types: python > > sympy/core/tests/test_arit.py[48] ...f.......................................... > .. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_assumptions.py[28] ....f....................... ? ? ? [OK] > sympy/core/tests/test_basic.py[10] .......... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] > sympy/core/tests/test_cache.py[1] . ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] > sympy/core/tests/test_complex.py[13] ............. ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_containers.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_count_ops.py[2] .. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_diff.py[6] ...... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] > sympy/core/tests/test_equal.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] > sympy/core/tests/test_eval.py[8] .....f.. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] > sympy/core/tests/test_eval_power.py[12] ............ ? ? ? ? ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_evalf.py[23] ....................... ? ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_expand.py[6] ...... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] > sympy/core/tests/test_expr.py[60] ..........F................................... > .............. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[FAIL] > sympy/core/tests/test_exprtools.py[4] .... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_facts.py[11] ........... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_functions.py[23] .....f................. ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_logic.py[11] ........... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_match.py[26] ...f...................... ? ? ? ? ? ? ? [OK] > sympy/core/tests/test_numbers.py[47] ........................................... > .... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_operations.py[4] .... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] > sympy/core/tests/test_priority.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_relational.py[7] ....... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_sets.py[18] .................. ? ? ? ? ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_subs.py[30] .............................. ? ? ? ? ? ?[OK] > sympy/core/tests/test_symbol.py[9] ......... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_sympify.py[23] ....................... ? ? ? ? ? ? ? ?[OK] > sympy/core/tests/test_truediv.py[3] ... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] > sympy/core/tests/test_var.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] > > ________________________________________________________________________________ > _____________ sympy/core/tests/test_expr.py:test_as_leading_term3 ______________ > ?File "/home/ondrej/repos/sympy/sympy/core/tests/test_expr.py", line > 212, in test_as_leading_term3 > ? ?assert (2*x+pi*x+x**2).as_leading_term(x) == (2+pi)*x > AssertionError > > == tests finished: 447 passed, 1 failed, 5 expected to fail, in 13.87 seconds == > DO *NOT* COMMIT! > > > > > The test failure is probably some minor difference between CPython and > PyPy.. However, as you can see, the test execution is 4x slower in > PyPy, compared to CPython. > Is this the supposed slow down, or is there something specific to sympy? > > I have also done this test in bin/isympy, CPython: > >>>> from timeit import default_timer as clock >>>> t = clock(); a = expand((1+x+y+z)**20); clock()-t > 1.4584941864 > > PyPy: > >>>> from timeit import default_timer as clock >>>> t = clock(); a = expand((1+x+y+z)**20); clock()-t > 4.48898005486 > > so PyPy is roughly 3x slower here. > > So 3x or 4x slower, I think that's already very usable. Especially the > sandbox pypy is an awesome idea, for serving web apps. The only > problem is that I use Cython a lot, which I think is incompatible with > PyPy... > > Ondrej > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > Hey. I think the test run is somehow within acceptable numbers (in theory tests should run each code once, hence making JIT not really fit well), however I think the other slowdown is not really acceptable. We were looking for a while to include some sympy benchmarks to our benchmark suite, maybe the good time is now :) I'll look at it From ondrej at certik.cz Tue May 3 09:44:05 2011 From: ondrej at certik.cz (Ondrej Certik) Date: Tue, 3 May 2011 00:44:05 -0700 Subject: [pypy-dev] speed of sympy tests In-Reply-To: References: Message-ID: On Tue, May 3, 2011 at 12:31 AM, Maciej Fijalkowski wrote: > On Tue, May 3, 2011 at 9:22 AM, Ondrej Certik wrote: >> Hi, >> >> I have downloaded the pypy 1.5 binary (with jit) and run tests (only >> sympy/core, so that it's fast) on Ubuntu Natty, 64bit: >> >> >> ondrej at eagle:~/repos/sympy(master)$ bin/test sympy/core/ >> ============================= test process starts ============================== >> executable: ? /usr/bin/python ?(2.7.1-final-0) >> ground types: python >> >> sympy/core/tests/test_arit.py[48] ...f.......................................... >> .. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_assumptions.py[28] ....f....................... ? ? ? [OK] >> sympy/core/tests/test_basic.py[10] .......... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >> sympy/core/tests/test_cache.py[1] . ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >> sympy/core/tests/test_complex.py[13] ............. ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_containers.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_count_ops.py[2] .. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_diff.py[6] ...... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >> sympy/core/tests/test_equal.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >> sympy/core/tests/test_eval.py[8] .....f.. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >> sympy/core/tests/test_eval_power.py[12] ............ ? ? ? ? ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_evalf.py[23] ....................... ? ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_expand.py[6] ...... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >> sympy/core/tests/test_expr.py[60] .............................................. >> .............. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_exprtools.py[4] .... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_facts.py[11] ........... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_functions.py[23] .....f................. ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_logic.py[11] ........... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_match.py[26] ...f...................... ? ? ? ? ? ? ? [OK] >> sympy/core/tests/test_numbers.py[47] ........................................... >> .... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_operations.py[4] .... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >> sympy/core/tests/test_priority.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_relational.py[7] ....... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_sets.py[18] .................. ? ? ? ? ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_subs.py[30] .............................. ? ? ? ? ? ?[OK] >> sympy/core/tests/test_symbol.py[9] ......... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_sympify.py[23] ....................... ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_truediv.py[3] ... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >> sympy/core/tests/test_var.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >> >> ======= tests finished: 448 passed, 5 expected to fail, in 2.90 seconds ======== >> ondrej at eagle:~/repos/sympy(master)$ >> ~/Downloads/pypy-c-jit-43780-b590cf6de419-linux64/bin/pypy bin/test >> sympy/core/ >> ============================= test process starts ============================== >> executable: ? /home/ondrej/Downloads/pypy-c-jit-43780-b590cf6de419-linux64/bin/pypy >> ?(2.7.1-final-42) >> ground types: python >> >> sympy/core/tests/test_arit.py[48] ...f.......................................... >> .. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_assumptions.py[28] ....f....................... ? ? ? [OK] >> sympy/core/tests/test_basic.py[10] .......... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >> sympy/core/tests/test_cache.py[1] . ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >> sympy/core/tests/test_complex.py[13] ............. ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_containers.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_count_ops.py[2] .. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_diff.py[6] ...... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >> sympy/core/tests/test_equal.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >> sympy/core/tests/test_eval.py[8] .....f.. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >> sympy/core/tests/test_eval_power.py[12] ............ ? ? ? ? ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_evalf.py[23] ....................... ? ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_expand.py[6] ...... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >> sympy/core/tests/test_expr.py[60] ..........F................................... >> .............. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[FAIL] >> sympy/core/tests/test_exprtools.py[4] .... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_facts.py[11] ........... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_functions.py[23] .....f................. ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_logic.py[11] ........... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_match.py[26] ...f...................... ? ? ? ? ? ? ? [OK] >> sympy/core/tests/test_numbers.py[47] ........................................... >> .... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_operations.py[4] .... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >> sympy/core/tests/test_priority.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_relational.py[7] ....... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_sets.py[18] .................. ? ? ? ? ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_subs.py[30] .............................. ? ? ? ? ? ?[OK] >> sympy/core/tests/test_symbol.py[9] ......... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_sympify.py[23] ....................... ? ? ? ? ? ? ? ?[OK] >> sympy/core/tests/test_truediv.py[3] ... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >> sympy/core/tests/test_var.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >> >> ________________________________________________________________________________ >> _____________ sympy/core/tests/test_expr.py:test_as_leading_term3 ______________ >> ?File "/home/ondrej/repos/sympy/sympy/core/tests/test_expr.py", line >> 212, in test_as_leading_term3 >> ? ?assert (2*x+pi*x+x**2).as_leading_term(x) == (2+pi)*x >> AssertionError >> >> == tests finished: 447 passed, 1 failed, 5 expected to fail, in 13.87 seconds == >> DO *NOT* COMMIT! >> >> >> >> >> The test failure is probably some minor difference between CPython and >> PyPy.. However, as you can see, the test execution is 4x slower in >> PyPy, compared to CPython. >> Is this the supposed slow down, or is there something specific to sympy? >> >> I have also done this test in bin/isympy, CPython: >> >>>>> from timeit import default_timer as clock >>>>> t = clock(); a = expand((1+x+y+z)**20); clock()-t >> 1.4584941864 >> >> PyPy: >> >>>>> from timeit import default_timer as clock >>>>> t = clock(); a = expand((1+x+y+z)**20); clock()-t >> 4.48898005486 >> >> so PyPy is roughly 3x slower here. >> >> So 3x or 4x slower, I think that's already very usable. Especially the >> sandbox pypy is an awesome idea, for serving web apps. The only >> problem is that I use Cython a lot, which I think is incompatible with >> PyPy... >> >> Ondrej >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev >> > > Hey. > > I think the test run is somehow within acceptable numbers (in theory > tests should run each code once, hence making JIT not really fit > well), however I think the other slowdown is not really acceptable. We Well, it is also only run once, so JIT probably doesn't help. > were looking for a while to include some sympy benchmarks to our > benchmark suite, maybe the good time is now :) SymPy uses caching, so maybe that's the problem. It is possible to turn it down though. Ping me if you have any questions, or run into any problems with sympy. Ondrej From holger at merlinux.eu Tue May 3 11:55:00 2011 From: holger at merlinux.eu (holger krekel) Date: Tue, 3 May 2011 09:55:00 +0000 Subject: [pypy-dev] docs on doc.pypy.org Message-ID: <20110503095500.GW4071@merlinux.eu> Hi all, i created a CNAME for our use of pypy.readthedocs.org and so you can now reach pypy docs under the frontend URL of http://doc.pypy.org It might be better to promote this link also from pypy.org so that URLs have a higher chance to remain valid even if we decide to move away from readthedocs some day. Moreover, we could see to have (another) sphinx setup for sphinx setup for our current pypy.org so we can host it there as well, maybe under www.pypy.org -- we need to have a subdomain/CNAME as we cannot redirect the root domain to readthedocs. best, holger From anto.cuni at gmail.com Tue May 3 12:04:39 2011 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 03 May 2011 12:04:39 +0200 Subject: [pypy-dev] docs on doc.pypy.org In-Reply-To: <20110503095500.GW4071@merlinux.eu> References: <20110503095500.GW4071@merlinux.eu> Message-ID: <4DBFD337.4080107@gmail.com> On 03/05/11 11:55, holger krekel wrote: > Hi all, > > i created a CNAME for our use of pypy.readthedocs.org and so you can now > reach pypy docs under the frontend URL of > > http://doc.pypy.org > > It might be better to promote this link also from pypy.org so > that URLs have a higher chance to remain valid even if we > decide to move away from readthedocs some day. yes, good idea. Moreover, I think we should change the link from "Dev Documentation" to just "Documentation", because it contains docs which are generally useful also for users (e.g., the getting started, the papers, etc.) Alternatively, we could have a heavier refactoring of the documentation and have clearly separated sections for "user docs" and "dev docs". > Moreover, we could see to have (another) sphinx setup for sphinx > setup for our current pypy.org so we can host it there as well, maybe > under www.pypy.org -- we need to have a subdomain/CNAME as we cannot > redirect the root domain to readthedocs. I'm not a sphinx expert, but I don't think that the default layout (i.e., with a sidebar on the left) is the best for the home page of pypy.org. ciao, Anto From holger at merlinux.eu Tue May 3 12:12:17 2011 From: holger at merlinux.eu (holger krekel) Date: Tue, 3 May 2011 10:12:17 +0000 Subject: [pypy-dev] docs on doc.pypy.org In-Reply-To: <4DBFD337.4080107@gmail.com> References: <20110503095500.GW4071@merlinux.eu> <4DBFD337.4080107@gmail.com> Message-ID: <20110503101217.GY4071@merlinux.eu> On Tue, May 03, 2011 at 12:04 +0200, Antonio Cuni wrote: > On 03/05/11 11:55, holger krekel wrote: > > Hi all, > > > > i created a CNAME for our use of pypy.readthedocs.org and so you can now > > reach pypy docs under the frontend URL of > > > > http://doc.pypy.org > > > > It might be better to promote this link also from pypy.org so > > that URLs have a higher chance to remain valid even if we > > decide to move away from readthedocs some day. > > yes, good idea. > Moreover, I think we should change the link from "Dev Documentation" to just > "Documentation", because it contains docs which are generally useful also for > users (e.g., the getting started, the papers, etc.) > > Alternatively, we could have a heavier refactoring of the documentation and > have clearly separated sections for "user docs" and "dev docs". > > > Moreover, we could see to have (another) sphinx setup for sphinx > > setup for our current pypy.org so we can host it there as well, maybe > > under www.pypy.org -- we need to have a subdomain/CNAME as we cannot > > redirect the root domain to readthedocs. > > I'm not a sphinx expert, but I don't think that the default layout (i.e., with > a sidebar on the left) is the best for the home page of pypy.org. I think sphinx can be tuned to support pypy.org layout but i don't consider myself a sphinx expert either. Maybe someone feels like helping and trying with https://bitbucket.org/pypy/pypy.org/overview to put it into a sphinx format? holger From holger at merlinux.eu Tue May 3 12:40:12 2011 From: holger at merlinux.eu (holger krekel) Date: Tue, 3 May 2011 10:40:12 +0000 Subject: [pypy-dev] docs on doc.pypy.org In-Reply-To: <20110503101217.GY4071@merlinux.eu> References: <20110503095500.GW4071@merlinux.eu> <4DBFD337.4080107@gmail.com> <20110503101217.GY4071@merlinux.eu> Message-ID: <20110503104012.GA4071@merlinux.eu> On Tue, May 03, 2011 at 10:12 +0000, holger krekel wrote: > On Tue, May 03, 2011 at 12:04 +0200, Antonio Cuni wrote: > > On 03/05/11 11:55, holger krekel wrote: > > > Hi all, > > > > > > i created a CNAME for our use of pypy.readthedocs.org and so you can now > > > reach pypy docs under the frontend URL of > > > > > > http://doc.pypy.org > > > > > > It might be better to promote this link also from pypy.org so > > > that URLs have a higher chance to remain valid even if we > > > decide to move away from readthedocs some day. > > > > yes, good idea. > > Moreover, I think we should change the link from "Dev Documentation" to just > > "Documentation", because it contains docs which are generally useful also for > > users (e.g., the getting started, the papers, etc.) > > > > Alternatively, we could have a heavier refactoring of the documentation and > > have clearly separated sections for "user docs" and "dev docs". > > > > > Moreover, we could see to have (another) sphinx setup for sphinx > > > setup for our current pypy.org so we can host it there as well, maybe > > > under www.pypy.org -- we need to have a subdomain/CNAME as we cannot > > > redirect the root domain to readthedocs. > > > > I'm not a sphinx expert, but I don't think that the default layout (i.e., with > > a sidebar on the left) is the best for the home page of pypy.org. > > I think sphinx can be tuned to support pypy.org layout but i don't > consider myself a sphinx expert either. Maybe someone feels like helping > and trying with https://bitbucket.org/pypy/pypy.org/overview > to put it into a sphinx format? seems like there is a limitation: http://read-the-docs.readthedocs.org/en/latest/install.html#what-s-available which i just hit while playing with rtfd and pytest docs :/ Maybe we can get ourselves whitelisted sometime. holger From fijall at gmail.com Tue May 3 13:20:09 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 3 May 2011 13:20:09 +0200 Subject: [pypy-dev] PyPy Assembler SQRT Patch In-Reply-To: References: Message-ID: Hi. Sorry it took so long to review, long weekend and whatnot. The test for x86 backend actually fails for -1.0 on 64bit. Can you reproduce/deny? On Tue, Apr 26, 2011 at 11:51 PM, Joe wrote: > Attached is a patch to allow pypy to use SQRTSD rather than calling > out to libc. ?This resulted in a 2x speedup, that scaled as the > benchmark took longer. > When i added another 0 to the end of the benchmark it was still a 2x speedup. > > benchmark results: > http://paste.pocoo.org/show/378122/ > > I'll be happy to answer any questions about the patch. > Joe > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From cool-rr at cool-rr.com Tue May 3 13:25:40 2011 From: cool-rr at cool-rr.com (cool-RR) Date: Tue, 3 May 2011 13:25:40 +0200 Subject: [pypy-dev] Allow subclassing `staticmethod` and `classmethod` Message-ID: In my project I made an `abstract_static_method` class: https://github.com/cool-RR/GarlicSim/blob/master/garlicsim/garlicsim/general_misc/abc_tools.py#L7 A similar one has been made in Python 3.2: http://hg.python.org/cpython/file/default/Lib/abc.py#l48 In Pypy this currently cannot be done, because if you subclass `staticmethod` into `abstractstaticmethod`, any instances will have a class of `staticmethod` instead of `abstractstaticmethod`: >>>> class A(staticmethod): .... pass .... >>>> def f(): .... pass .... >>>> a = A(f) >>>> a >>>> type(a) What can be done? I want to have `abstract_static_method` in my code. Thanks, Ram. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110503/9f528de0/attachment-0001.htm From cfbolz at gmx.de Tue May 3 13:36:43 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Tue, 03 May 2011 13:36:43 +0200 Subject: [pypy-dev] Allow subclassing `staticmethod` and `classmethod` In-Reply-To: References: Message-ID: <4DBFE8CB.40300@gmx.de> Hi Ram, On 05/03/2011 01:25 PM, cool-RR wrote: > In my project I made an `abstract_static_method` class: > > https://github.com/cool-RR/GarlicSim/blob/master/garlicsim/garlicsim/general_misc/abc_tools.py#L7 > > A > similar one has been made in Python 3.2: > > http://hg.python.org/cpython/file/default/Lib/abc.py#l48 > > > In Pypy this > currently cannot be done, because if you subclass `staticmethod` into > `abstractstaticmethod`, any instances will have a class > of `staticmethod` instead of `abstractstaticmethod`: > > >>>> class A(staticmethod): > .... pass > .... > >>>> def f(): > .... pass > .... > >>>> a = A(f) > >>>> a > > >>>> type(a) > > > > What can be done? I want to have `abstract_static_method` in my code. This is a bug in PyPy, I fixed it in the repo, tomorrow you can use a fresh nightly build from here: http://buildbot.pypy.org/nightly/trunk/ Carl Friedrich From cool-rr at cool-rr.com Tue May 3 13:38:11 2011 From: cool-rr at cool-rr.com (cool-RR) Date: Tue, 3 May 2011 13:38:11 +0200 Subject: [pypy-dev] Allow subclassing `staticmethod` and `classmethod` In-Reply-To: <4DBFE8CB.40300@gmx.de> References: <4DBFE8CB.40300@gmx.de> Message-ID: Cool. Is there gonna be a 1.5.1 release with stuff like this? On Tue, May 3, 2011 at 1:36 PM, Carl Friedrich Bolz wrote: > Hi Ram, > > On 05/03/2011 01:25 PM, cool-RR wrote: > > In my project I made an `abstract_static_method` class: > > > > > https://github.com/cool-RR/GarlicSim/blob/master/garlicsim/garlicsim/general_misc/abc_tools.py#L7 > > > > < > https://github.com/cool-RR/GarlicSim/blob/master/garlicsim/garlicsim/general_misc/abc_tools.py#L7 > >A > > similar one has been made in Python 3.2: > > > > http://hg.python.org/cpython/file/default/Lib/abc.py#l48 > > > > > > In Pypy this > > currently cannot be done, because if you subclass `staticmethod` into > > `abstractstaticmethod`, any instances will have a class > > of `staticmethod` instead of `abstractstaticmethod`: > > > > >>>> class A(staticmethod): > > .... pass > > .... > > >>>> def f(): > > .... pass > > .... > > >>>> a = A(f) > > >>>> a > > > > >>>> type(a) > > > > > > > > What can be done? I want to have `abstract_static_method` in my code. > > > This is a bug in PyPy, I fixed it in the repo, tomorrow you can use a > fresh nightly build from here: > > http://buildbot.pypy.org/nightly/trunk/ > > Carl Friedrich > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -- Sincerely, Ram Rachum -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110503/b8e8c44f/attachment.htm From rndblnch at gmail.com Tue May 3 15:11:15 2011 From: rndblnch at gmail.com (renaud blanch) Date: Tue, 3 May 2011 15:11:15 +0200 Subject: [pypy-dev] BytecodeCorruption error with pypy-1.5 Message-ID: Hello, First, thank you for the pypy-1.5 release! I quickly tested pypy-1.5 (osx64 binary from your download page) with some code I use as benchmark (some solutions to Google code jam problems), and I got a reproductible "Fatal RPython error: BytecodeCorruption". The minimal code sample that reproduce this behaviour I could came up with is: def what(_="whatever"): return (_ if _ else _) if False else _ what() This code works as expected with any cpython and with pypy-1.4.1 on the same machine (Intel Core 2 Duo iMac/Mac OS X 10.6.7). Should I open a bug on the pypy tracker? renaud From fijall at gmail.com Tue May 3 15:18:30 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 3 May 2011 15:18:30 +0200 Subject: [pypy-dev] speed of sympy tests In-Reply-To: References: Message-ID: On Tue, May 3, 2011 at 9:44 AM, Ondrej Certik wrote: > On Tue, May 3, 2011 at 12:31 AM, Maciej Fijalkowski wrote: >> On Tue, May 3, 2011 at 9:22 AM, Ondrej Certik wrote: >>> Hi, >>> >>> I have downloaded the pypy 1.5 binary (with jit) and run tests (only >>> sympy/core, so that it's fast) on Ubuntu Natty, 64bit: >>> >>> >>> ondrej at eagle:~/repos/sympy(master)$ bin/test sympy/core/ >>> ============================= test process starts ============================== >>> executable: ? /usr/bin/python ?(2.7.1-final-0) >>> ground types: python >>> >>> sympy/core/tests/test_arit.py[48] ...f.......................................... >>> .. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_assumptions.py[28] ....f....................... ? ? ? [OK] >>> sympy/core/tests/test_basic.py[10] .......... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>> sympy/core/tests/test_cache.py[1] . ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>> sympy/core/tests/test_complex.py[13] ............. ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_containers.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_count_ops.py[2] .. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_diff.py[6] ...... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>> sympy/core/tests/test_equal.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>> sympy/core/tests/test_eval.py[8] .....f.. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>> sympy/core/tests/test_eval_power.py[12] ............ ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_evalf.py[23] ....................... ? ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_expand.py[6] ...... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>> sympy/core/tests/test_expr.py[60] .............................................. >>> .............. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_exprtools.py[4] .... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_facts.py[11] ........... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_functions.py[23] .....f................. ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_logic.py[11] ........... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_match.py[26] ...f...................... ? ? ? ? ? ? ? [OK] >>> sympy/core/tests/test_numbers.py[47] ........................................... >>> .... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_operations.py[4] .... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>> sympy/core/tests/test_priority.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_relational.py[7] ....... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_sets.py[18] .................. ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_subs.py[30] .............................. ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_symbol.py[9] ......... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_sympify.py[23] ....................... ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_truediv.py[3] ... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>> sympy/core/tests/test_var.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>> >>> ======= tests finished: 448 passed, 5 expected to fail, in 2.90 seconds ======== >>> ondrej at eagle:~/repos/sympy(master)$ >>> ~/Downloads/pypy-c-jit-43780-b590cf6de419-linux64/bin/pypy bin/test >>> sympy/core/ >>> ============================= test process starts ============================== >>> executable: ? /home/ondrej/Downloads/pypy-c-jit-43780-b590cf6de419-linux64/bin/pypy >>> ?(2.7.1-final-42) >>> ground types: python >>> >>> sympy/core/tests/test_arit.py[48] ...f.......................................... >>> .. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_assumptions.py[28] ....f....................... ? ? ? [OK] >>> sympy/core/tests/test_basic.py[10] .......... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>> sympy/core/tests/test_cache.py[1] . ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>> sympy/core/tests/test_complex.py[13] ............. ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_containers.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_count_ops.py[2] .. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_diff.py[6] ...... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>> sympy/core/tests/test_equal.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>> sympy/core/tests/test_eval.py[8] .....f.. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>> sympy/core/tests/test_eval_power.py[12] ............ ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_evalf.py[23] ....................... ? ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_expand.py[6] ...... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>> sympy/core/tests/test_expr.py[60] ..........F................................... >>> .............. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[FAIL] >>> sympy/core/tests/test_exprtools.py[4] .... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_facts.py[11] ........... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_functions.py[23] .....f................. ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_logic.py[11] ........... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_match.py[26] ...f...................... ? ? ? ? ? ? ? [OK] >>> sympy/core/tests/test_numbers.py[47] ........................................... >>> .... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_operations.py[4] .... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>> sympy/core/tests/test_priority.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_relational.py[7] ....... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_sets.py[18] .................. ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_subs.py[30] .............................. ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_symbol.py[9] ......... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_sympify.py[23] ....................... ? ? ? ? ? ? ? ?[OK] >>> sympy/core/tests/test_truediv.py[3] ... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>> sympy/core/tests/test_var.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>> >>> ________________________________________________________________________________ >>> _____________ sympy/core/tests/test_expr.py:test_as_leading_term3 ______________ >>> ?File "/home/ondrej/repos/sympy/sympy/core/tests/test_expr.py", line >>> 212, in test_as_leading_term3 >>> ? ?assert (2*x+pi*x+x**2).as_leading_term(x) == (2+pi)*x >>> AssertionError >>> >>> == tests finished: 447 passed, 1 failed, 5 expected to fail, in 13.87 seconds == >>> DO *NOT* COMMIT! >>> >>> >>> >>> >>> The test failure is probably some minor difference between CPython and >>> PyPy.. However, as you can see, the test execution is 4x slower in >>> PyPy, compared to CPython. >>> Is this the supposed slow down, or is there something specific to sympy? >>> >>> I have also done this test in bin/isympy, CPython: >>> >>>>>> from timeit import default_timer as clock >>>>>> t = clock(); a = expand((1+x+y+z)**20); clock()-t >>> 1.4584941864 >>> >>> PyPy: >>> >>>>>> from timeit import default_timer as clock >>>>>> t = clock(); a = expand((1+x+y+z)**20); clock()-t >>> 4.48898005486 >>> >>> so PyPy is roughly 3x slower here. >>> >>> So 3x or 4x slower, I think that's already very usable. Especially the >>> sandbox pypy is an awesome idea, for serving web apps. The only >>> problem is that I use Cython a lot, which I think is incompatible with >>> PyPy... >>> >>> Ondrej >>> _______________________________________________ >>> pypy-dev at codespeak.net >>> http://codespeak.net/mailman/listinfo/pypy-dev >>> >> >> Hey. >> >> I think the test run is somehow within acceptable numbers (in theory >> tests should run each code once, hence making JIT not really fit >> well), however I think the other slowdown is not really acceptable. We > > Well, it is also only run once, so JIT probably doesn't help. > >> were looking for a while to include some sympy benchmarks to our >> benchmark suite, maybe the good time is now :) > > > SymPy uses caching, so maybe that's the problem. It is possible to > turn it down though. Ping me if you have any questions, or run into > any problems with sympy. > > Ondrej > That might sound strange but for me, both pypy and cpython runs within 4s. Which python you use? From fijall at gmail.com Tue May 3 15:20:49 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 3 May 2011 15:20:49 +0200 Subject: [pypy-dev] speed of sympy tests In-Reply-To: References: Message-ID: On Tue, May 3, 2011 at 3:18 PM, Maciej Fijalkowski wrote: > On Tue, May 3, 2011 at 9:44 AM, Ondrej Certik wrote: >> On Tue, May 3, 2011 at 12:31 AM, Maciej Fijalkowski wrote: >>> On Tue, May 3, 2011 at 9:22 AM, Ondrej Certik wrote: >>>> Hi, >>>> >>>> I have downloaded the pypy 1.5 binary (with jit) and run tests (only >>>> sympy/core, so that it's fast) on Ubuntu Natty, 64bit: >>>> >>>> >>>> ondrej at eagle:~/repos/sympy(master)$ bin/test sympy/core/ >>>> ============================= test process starts ============================== >>>> executable: ? /usr/bin/python ?(2.7.1-final-0) >>>> ground types: python >>>> >>>> sympy/core/tests/test_arit.py[48] ...f.......................................... >>>> .. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_assumptions.py[28] ....f....................... ? ? ? [OK] >>>> sympy/core/tests/test_basic.py[10] .......... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>>> sympy/core/tests/test_cache.py[1] . ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>>> sympy/core/tests/test_complex.py[13] ............. ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_containers.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_count_ops.py[2] .. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_diff.py[6] ...... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>>> sympy/core/tests/test_equal.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>>> sympy/core/tests/test_eval.py[8] .....f.. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>>> sympy/core/tests/test_eval_power.py[12] ............ ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_evalf.py[23] ....................... ? ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_expand.py[6] ...... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>>> sympy/core/tests/test_expr.py[60] .............................................. >>>> .............. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_exprtools.py[4] .... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_facts.py[11] ........... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_functions.py[23] .....f................. ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_logic.py[11] ........... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_match.py[26] ...f...................... ? ? ? ? ? ? ? [OK] >>>> sympy/core/tests/test_numbers.py[47] ........................................... >>>> .... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_operations.py[4] .... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>>> sympy/core/tests/test_priority.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_relational.py[7] ....... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_sets.py[18] .................. ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_subs.py[30] .............................. ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_symbol.py[9] ......... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_sympify.py[23] ....................... ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_truediv.py[3] ... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>>> sympy/core/tests/test_var.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>>> >>>> ======= tests finished: 448 passed, 5 expected to fail, in 2.90 seconds ======== >>>> ondrej at eagle:~/repos/sympy(master)$ >>>> ~/Downloads/pypy-c-jit-43780-b590cf6de419-linux64/bin/pypy bin/test >>>> sympy/core/ >>>> ============================= test process starts ============================== >>>> executable: ? /home/ondrej/Downloads/pypy-c-jit-43780-b590cf6de419-linux64/bin/pypy >>>> ?(2.7.1-final-42) >>>> ground types: python >>>> >>>> sympy/core/tests/test_arit.py[48] ...f.......................................... >>>> .. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_assumptions.py[28] ....f....................... ? ? ? [OK] >>>> sympy/core/tests/test_basic.py[10] .......... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>>> sympy/core/tests/test_cache.py[1] . ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>>> sympy/core/tests/test_complex.py[13] ............. ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_containers.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_count_ops.py[2] .. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_diff.py[6] ...... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>>> sympy/core/tests/test_equal.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>>> sympy/core/tests/test_eval.py[8] .....f.. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>>> sympy/core/tests/test_eval_power.py[12] ............ ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_evalf.py[23] ....................... ? ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_expand.py[6] ...... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>>> sympy/core/tests/test_expr.py[60] ..........F................................... >>>> .............. ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[FAIL] >>>> sympy/core/tests/test_exprtools.py[4] .... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_facts.py[11] ........... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_functions.py[23] .....f................. ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_logic.py[11] ........... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_match.py[26] ...f...................... ? ? ? ? ? ? ? [OK] >>>> sympy/core/tests/test_numbers.py[47] ........................................... >>>> .... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_operations.py[4] .... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>>> sympy/core/tests/test_priority.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_relational.py[7] ....... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_sets.py[18] .................. ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_subs.py[30] .............................. ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_symbol.py[9] ......... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_sympify.py[23] ....................... ? ? ? ? ? ? ? ?[OK] >>>> sympy/core/tests/test_truediv.py[3] ... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>>> sympy/core/tests/test_var.py[5] ..... ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? [OK] >>>> >>>> ________________________________________________________________________________ >>>> _____________ sympy/core/tests/test_expr.py:test_as_leading_term3 ______________ >>>> ?File "/home/ondrej/repos/sympy/sympy/core/tests/test_expr.py", line >>>> 212, in test_as_leading_term3 >>>> ? ?assert (2*x+pi*x+x**2).as_leading_term(x) == (2+pi)*x >>>> AssertionError >>>> >>>> == tests finished: 447 passed, 1 failed, 5 expected to fail, in 13.87 seconds == >>>> DO *NOT* COMMIT! >>>> >>>> >>>> >>>> >>>> The test failure is probably some minor difference between CPython and >>>> PyPy.. However, as you can see, the test execution is 4x slower in >>>> PyPy, compared to CPython. >>>> Is this the supposed slow down, or is there something specific to sympy? >>>> >>>> I have also done this test in bin/isympy, CPython: >>>> >>>>>>> from timeit import default_timer as clock >>>>>>> t = clock(); a = expand((1+x+y+z)**20); clock()-t >>>> 1.4584941864 >>>> >>>> PyPy: >>>> >>>>>>> from timeit import default_timer as clock >>>>>>> t = clock(); a = expand((1+x+y+z)**20); clock()-t >>>> 4.48898005486 >>>> >>>> so PyPy is roughly 3x slower here. >>>> >>>> So 3x or 4x slower, I think that's already very usable. Especially the >>>> sandbox pypy is an awesome idea, for serving web apps. The only >>>> problem is that I use Cython a lot, which I think is incompatible with >>>> PyPy... >>>> >>>> Ondrej >>>> _______________________________________________ >>>> pypy-dev at codespeak.net >>>> http://codespeak.net/mailman/listinfo/pypy-dev >>>> >>> >>> Hey. >>> >>> I think the test run is somehow within acceptable numbers (in theory >>> tests should run each code once, hence making JIT not really fit >>> well), however I think the other slowdown is not really acceptable. We >> >> Well, it is also only run once, so JIT probably doesn't help. >> >>> were looking for a while to include some sympy benchmarks to our >>> benchmark suite, maybe the good time is now :) >> >> >> SymPy uses caching, so maybe that's the problem. It is possible to >> turn it down though. Ping me if you have any questions, or run into >> any problems with sympy. >> >> Ondrej >> > > That might sound strange but for me, both pypy and cpython runs within > 4s. Which python you use? > CPython 2.7 runs 2x faster than 2.6. Can we get some more real-time communication set up, so we can discuss a bit From arigo at tunes.org Tue May 3 15:30:46 2011 From: arigo at tunes.org (Armin Rigo) Date: Tue, 3 May 2011 15:30:46 +0200 Subject: [pypy-dev] Allow subclassing `staticmethod` and `classmethod` In-Reply-To: References: <4DBFE8CB.40300@gmx.de> Message-ID: Hi Cool-RR, On Tue, May 3, 2011 at 1:38 PM, cool-RR wrote: > Cool. Is there gonna be a 1.5.1 release with stuff like this? Yes, very likely. Note that the 1.5 release is just one of the nightly binaries :-) Armin From alex.gaynor at gmail.com Tue May 3 15:32:39 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Tue, 3 May 2011 09:32:39 -0400 Subject: [pypy-dev] Allow subclassing `staticmethod` and `classmethod` In-Reply-To: References: <4DBFE8CB.40300@gmx.de> Message-ID: On Tue, May 3, 2011 at 9:30 AM, Armin Rigo wrote: > Hi Cool-RR, > > On Tue, May 3, 2011 at 1:38 PM, cool-RR wrote: > > Cool. Is there gonna be a 1.5.1 release with stuff like this? > > Yes, very likely. Note that the 1.5 release is just one of the > nightly binaries :-) > > Armin > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > FYI classmethod works correctly in 1.5 (because I fixed it a while ago because Django uses it). Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110503/7b5417ec/attachment.htm From arigo at tunes.org Tue May 3 16:30:23 2011 From: arigo at tunes.org (Armin Rigo) Date: Tue, 3 May 2011 16:30:23 +0200 Subject: [pypy-dev] PyPy Assembler SQRT Patch In-Reply-To: References: Message-ID: Hi, On Tue, May 3, 2011 at 1:20 PM, Maciej Fijalkowski wrote: > Sorry it took so long to review, long weekend and whatnot. The test > for x86 backend actually fails for -1.0 on 64bit. Can you > reproduce/deny? I fixed it, it was really a bug about FINISH(ConstFloat(...)). But the patch, at least as applied now, does not have the intended effect; SQRTSD is never generated because ll_math_sqrt() is called with a residual call. Joe, can you fix it? I can try to if you don't have the time. A bient?t, Armin. From amauryfa at gmail.com Tue May 3 16:36:08 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 3 May 2011 16:36:08 +0200 Subject: [pypy-dev] BytecodeCorruption error with pypy-1.5 In-Reply-To: References: Message-ID: 2011/5/3 renaud blanch : > I quickly tested pypy-1.5 (osx64 binary from your download page) with > some code I use as benchmark (some solutions to Google code jam > problems), and I got a reproductible "Fatal RPython error: > BytecodeCorruption". > > The minimal code sample that reproduce this behaviour I could came up with is: > > def what(_="whatever"): > ? ?return (_ if _ else _) if False else _ > what() > > > This code works as expected with any cpython and with pypy-1.4.1 on > the same machine (Intel Core 2 Duo iMac/Mac OS X 10.6.7). > > Should I open a bug on the pypy tracker? Yes please ! -- Amaury Forgeot d'Arc From alex.gaynor at gmail.com Tue May 3 18:23:16 2011 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Tue, 3 May 2011 12:23:16 -0400 Subject: [pypy-dev] PyPy Assembler SQRT Patch In-Reply-To: References: Message-ID: On Tue, May 3, 2011 at 10:30 AM, Armin Rigo wrote: > Hi, > > On Tue, May 3, 2011 at 1:20 PM, Maciej Fijalkowski > wrote: > > Sorry it took so long to review, long weekend and whatnot. The test > > for x86 backend actually fails for -1.0 on 64bit. Can you > > reproduce/deny? > > I fixed it, it was really a bug about FINISH(ConstFloat(...)). But > the patch, at least as applied now, does not have the intended effect; > SQRTSD is never generated because ll_math_sqrt() is called with a > residual call. Joe, can you fix it? I can try to if you don't have > the time. > > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > Another thing is it would be useful to test that the SQRTSD is generated, I don't know a good way to do that though. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110503/a767bf13/attachment.htm From arigo at tunes.org Tue May 3 18:28:58 2011 From: arigo at tunes.org (Armin Rigo) Date: Tue, 3 May 2011 18:28:58 +0200 Subject: [pypy-dev] PyPy Assembler SQRT Patch In-Reply-To: References: Message-ID: Hi Alex, On Tue, May 3, 2011 at 6:23 PM, Alex Gaynor wrote: > Another thing is it would be useful to test that the SQRTSD is generated, I > don't know a good way to do that though. In the test, you can hack genop_math_sqrt() in-place to set a flag, and test that the flag is set... And another place it should also be tested is in test_pypy_c_new.py (pypy.module.pypyjit.test_pypy_c), i.e. you should write a test that checks that in a full translated pypy-c, the code corresponding to the Python source "math.sqrt(x)" indeed ends up being a CALL(sqrt_wrapper) and not a CALL(ll_math_sqrt). A bient?t, Armin. From lac at openend.se Tue May 3 18:50:02 2011 From: lac at openend.se (Laura Creighton) Date: Tue, 03 May 2011 18:50:02 +0200 Subject: [pypy-dev] docs on doc.pypy.org In-Reply-To: Message from holger krekel of "Tue, 03 May 2011 10:12:17 -0000." <20110503101217.GY4071@merlinux.eu> References: <20110503095500.GW4071@merlinux.eu> <4DBFD337.4080107@gmail.com> <20110503101217.GY4071@merlinux.eu> Message-ID: <201105031650.p43Go2C8018390@theraft.openend.se> In a message of Tue, 03 May 2011 10:12:17 -0000, holger krekel writes: >> I'm not a sphinx expert, but I don't think that the default layout (i.e >., with >> a sidebar on the left) is the best for the home page of pypy.org. > >I think sphinx can be tuned to support pypy.org layout but i don't >consider myself a sphinx expert either. Maybe someone feels like helping > >and trying with https://bitbucket.org/pypy/pypy.org/overview >to put it into a sphinx format? > >holger Georg Brandl (and you don't get more expert than this) showed up in the channel to volunteer to give us a custom look. Now we only have to decide what it is that we want. Laura From santagada at gmail.com Tue May 3 19:08:49 2011 From: santagada at gmail.com (Leonardo Santagada) Date: Tue, 3 May 2011 14:08:49 -0300 Subject: [pypy-dev] docs on doc.pypy.org In-Reply-To: <201105031650.p43Go2C8018390@theraft.openend.se> References: <20110503095500.GW4071@merlinux.eu> <4DBFD337.4080107@gmail.com> <20110503101217.GY4071@merlinux.eu> <201105031650.p43Go2C8018390@theraft.openend.se> Message-ID: On Tue, May 3, 2011 at 1:50 PM, Laura Creighton wrote: > In a message of Tue, 03 May 2011 10:12:17 -0000, holger krekel writes: >>> I'm not a sphinx expert, but I don't think that the default layout (i.e >>., with >>> a sidebar on the left) is the best for the home page of pypy.org. >> >>I think sphinx can be tuned to support pypy.org layout but i don't >>consider myself a sphinx expert either. ?Maybe someone feels like helping >> >>and trying with https://bitbucket.org/pypy/pypy.org/overview >>to put it into a sphinx format? >> >>holger > > Georg Brandl (and you don't get more expert than this) showed up in > the channel to volunteer to give us a custom look. ?Now we only have to > decide what it is that we want. I found this site themes to be really cheap and good looking: http://themeforest.net/category/site-templates/technology/software http://themeforest.net/category/site-templates/technology Either this or hiring someone I think is the best idea, pypy deserves a great looking website :) -- Leonardo Santagada From holger at merlinux.eu Tue May 3 19:13:05 2011 From: holger at merlinux.eu (holger krekel) Date: Tue, 3 May 2011 17:13:05 +0000 Subject: [pypy-dev] Allow subclassing `staticmethod` and `classmethod` In-Reply-To: References: <4DBFE8CB.40300@gmx.de> Message-ID: <20110503171305.GD4071@merlinux.eu> On Tue, May 03, 2011 at 15:30 +0200, Armin Rigo wrote: > Hi Cool-RR, > > On Tue, May 3, 2011 at 1:38 PM, cool-RR wrote: > > Cool. Is there gonna be a 1.5.1 release with stuff like this? > > Yes, very likely. Note that the 1.5 release is just one of the > nightly binaries :-) modulo a 1.5.1 version string, maybe? :) holger > Armin > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From rogerb at rogerbinns.com Thu May 5 05:54:00 2011 From: rogerb at rogerbinns.com (Roger Binns) Date: Wed, 04 May 2011 20:54:00 -0700 Subject: [pypy-dev] cpyext: Detecting pypy and other issues In-Reply-To: <4DBF15E0.6080005@rogerbinns.com> References: <4DBD247B.7040102@rogerbinns.com> <4DBD310C.9090806@rogerbinns.com> <4DBE3956.90409@rogerbinns.com> <4DBE4F7A.3030109@gmail.com> <4DBF15E0.6080005@rogerbinns.com> Message-ID: <4DC21F58.8000101@rogerbinns.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 05/02/2011 01:36 PM, Roger Binns wrote: > On 05/01/2011 11:30 PM, Antonio Cuni wrote: >> you are right, we lack such a document. Do you feel like writing it? :-) > > I'll make an initial stab at it this week. Here you go: https://docs.google.com/document/pub?id=1Ve9B6Gz5laV63zYFOlb_Lhe8ZBIw2V1756lquott6JI Unfortunately building pypy myself didn't really achieve anything for my own code. PyPy behaved really bizarrely for example claiming functions don't exist in my test code (standard unittest). I get different behaviour if I run under gdb or not. Additionally objects aren't particularly useful. With CPython there are some nice macros supplied with a .gdbinit, especially pyo so you can tell what is going on. Pypy is behaving like a randomized black box! If anyone wants to try for themselves then the code is at http://code.google.com/p/apsw/source/checkout Roger -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk3CH1MACgkQmOOfHg372QQsxACdElcBC7z/um35YWZ9d0fTM212 DkMAniD7ibCOYQcfHIeB8U8aSpMu9EQ6 =RRZh -----END PGP SIGNATURE----- From qbproger at gmail.com Thu May 5 22:39:05 2011 From: qbproger at gmail.com (Joe) Date: Thu, 5 May 2011 16:39:05 -0400 Subject: [pypy-dev] docs on doc.pypy.org In-Reply-To: References: <20110503095500.GW4071@merlinux.eu> <4DBFD337.4080107@gmail.com> <20110503101217.GY4071@merlinux.eu> <201105031650.p43Go2C8018390@theraft.openend.se> Message-ID: If you do a blog post asking for help someone would probably help out. Especially since they'd have pypy.org in their portfolio. It may not turn up anything, but worth a shot. Joe On Tue, May 3, 2011 at 1:08 PM, Leonardo Santagada wrote: > On Tue, May 3, 2011 at 1:50 PM, Laura Creighton wrote: >> In a message of Tue, 03 May 2011 10:12:17 -0000, holger krekel writes: >>>> I'm not a sphinx expert, but I don't think that the default layout (i.e >>>., with >>>> a sidebar on the left) is the best for the home page of pypy.org. >>> >>>I think sphinx can be tuned to support pypy.org layout but i don't >>>consider myself a sphinx expert either. ?Maybe someone feels like helping >>> >>>and trying with https://bitbucket.org/pypy/pypy.org/overview >>>to put it into a sphinx format? >>> >>>holger >> >> Georg Brandl (and you don't get more expert than this) showed up in >> the channel to volunteer to give us a custom look. ?Now we only have to >> decide what it is that we want. > > I found this site themes to be really cheap and good looking: > > http://themeforest.net/category/site-templates/technology/software > http://themeforest.net/category/site-templates/technology > > Either this or hiring someone I think is the best idea, pypy deserves > a great looking website :) > > -- > Leonardo Santagada > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev From dacjames at gmail.com Thu May 5 23:00:08 2011 From: dacjames at gmail.com (Daniel Collins) Date: Thu, 5 May 2011 14:00:08 -0700 Subject: [pypy-dev] docs on doc.pypy.org In-Reply-To: References: <20110503095500.GW4071@merlinux.eu> <4DBFD337.4080107@gmail.com> <20110503101217.GY4071@merlinux.eu> <201105031650.p43Go2C8018390@theraft.openend.se> Message-ID: Hello! After following the project for a long time, I just joined the pypy mailing list looking to help out. I would love to help on the pypy.orgwebsite; admittedly, I am fairly new to web design but I think I could help. I have never contributed to an open source project before so I need a little guidance about how things work here. Thanks, let me know! --Daniel On Thu, May 5, 2011 at 1:39 PM, Joe wrote: > If you do a blog post asking for help someone would probably help out. > Especially since they'd have pypy.org in their portfolio. It may not > turn up anything, but worth a shot. > > Joe > > On Tue, May 3, 2011 at 1:08 PM, Leonardo Santagada > wrote: > > On Tue, May 3, 2011 at 1:50 PM, Laura Creighton wrote: > >> In a message of Tue, 03 May 2011 10:12:17 -0000, holger krekel writes: > >>>> I'm not a sphinx expert, but I don't think that the default layout > (i.e > >>>., with > >>>> a sidebar on the left) is the best for the home page of pypy.org. > >>> > >>>I think sphinx can be tuned to support pypy.org layout but i don't > >>>consider myself a sphinx expert either. Maybe someone feels like > helping > >>> > >>>and trying with https://bitbucket.org/pypy/pypy.org/overview > >>>to put it into a sphinx format? > >>> > >>>holger > >> > >> Georg Brandl (and you don't get more expert than this) showed up in > >> the channel to volunteer to give us a custom look. Now we only have to > >> decide what it is that we want. > > > > I found this site themes to be really cheap and good looking: > > > > http://themeforest.net/category/site-templates/technology/software > > http://themeforest.net/category/site-templates/technology > > > > Either this or hiring someone I think is the best idea, pypy deserves > > a great looking website :) > > > > -- > > Leonardo Santagada > > _______________________________________________ > > pypy-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/pypy-dev > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110505/d35025ad/attachment.htm From fijall at gmail.com Thu May 5 23:13:31 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 5 May 2011 23:13:31 +0200 Subject: [pypy-dev] docs on doc.pypy.org In-Reply-To: References: <20110503095500.GW4071@merlinux.eu> <4DBFD337.4080107@gmail.com> <20110503101217.GY4071@merlinux.eu> <201105031650.p43Go2C8018390@theraft.openend.se> Message-ID: On Thu, May 5, 2011 at 11:00 PM, Daniel Collins wrote: > Hello! > After following the project for a long time, I just joined the pypy mailing > list looking to help out. ?I would love to help on the pypy.org > website;?admittedly, I am fairly new to web design but I think I could help. > > I have never contributed to an open source project before so I need a little > guidance about how things work here. ?Thanks, let me know! > --Daniel It's very simple. You come up with an idea what do you want to work on (this is done, pypy website) so you need to check out an appropriate repository. http://bitbucket.org/pypy/pypy.org then you either create a branch and issue a pull request or create a patch or whatever way is really convinient. Feel free to join #pypy on irc.freenode.net if you have any more questions > > On Thu, May 5, 2011 at 1:39 PM, Joe wrote: >> >> If you do a blog post asking for help someone would probably help out. >> ?Especially since they'd have pypy.org in their portfolio. ?It may not >> turn up anything, but worth a shot. >> >> Joe >> >> On Tue, May 3, 2011 at 1:08 PM, Leonardo Santagada >> wrote: >> > On Tue, May 3, 2011 at 1:50 PM, Laura Creighton wrote: >> >> In a message of Tue, 03 May 2011 10:12:17 -0000, holger krekel writes: >> >>>> I'm not a sphinx expert, but I don't think that the default layout >> >>>> (i.e >> >>>., with >> >>>> a sidebar on the left) is the best for the home page of pypy.org. >> >>> >> >>>I think sphinx can be tuned to support pypy.org layout but i don't >> >>>consider myself a sphinx expert either. ?Maybe someone feels like >> >>> helping >> >>> >> >>>and trying with https://bitbucket.org/pypy/pypy.org/overview >> >>>to put it into a sphinx format? >> >>> >> >>>holger >> >> >> >> Georg Brandl (and you don't get more expert than this) showed up in >> >> the channel to volunteer to give us a custom look. ?Now we only have to >> >> decide what it is that we want. >> > >> > I found this site themes to be really cheap and good looking: >> > >> > http://themeforest.net/category/site-templates/technology/software >> > http://themeforest.net/category/site-templates/technology >> > >> > Either this or hiring someone I think is the best idea, pypy deserves >> > a great looking website :) >> > >> > -- >> > Leonardo Santagada >> > _______________________________________________ >> > pypy-dev at codespeak.net >> > http://codespeak.net/mailman/listinfo/pypy-dev >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From santagada at gmail.com Fri May 6 02:40:47 2011 From: santagada at gmail.com (Leonardo Santagada) Date: Thu, 5 May 2011 21:40:47 -0300 Subject: [pypy-dev] docs on doc.pypy.org In-Reply-To: References: <20110503095500.GW4071@merlinux.eu> <4DBFD337.4080107@gmail.com> <20110503101217.GY4071@merlinux.eu> <201105031650.p43Go2C8018390@theraft.openend.se> Message-ID: On Thu, May 5, 2011 at 6:13 PM, Maciej Fijalkowski wrote: >> I have never contributed to an open source project before so I need a little >> guidance about how things work here. ?Thanks, let me know! >> --Daniel > > It's very simple. You come up with an idea what do you want to work on > (this is done, pypy website) so you need to check out an appropriate > repository. > > http://bitbucket.org/pypy/pypy.org > > then you either create a branch and issue a pull request or create a > patch or whatever way is really convinient. > > Feel free to join #pypy on irc.freenode.net if you have any more questions Sorry to disagree but wouldn't just doing a sketch in photoshop and posting the design here be enough for a start? Then do the html+css+javascript needed and only then think about doing a template, which if you don't know how to do there are plenty of people that knows that and can do it for you in 2 hours tops if you have everything already on html. I think the problem is the design of the website, not coding the template in jinja2. -- Leonardo Santagada From exarkun at twistedmatrix.com Fri May 6 13:57:17 2011 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Fri, 06 May 2011 11:57:17 -0000 Subject: [pypy-dev] docs on doc.pypy.org In-Reply-To: References: <20110503095500.GW4071@merlinux.eu> <4DBFD337.4080107@gmail.com> <20110503101217.GY4071@merlinux.eu> <201105031650.p43Go2C8018390@theraft.openend.se> Message-ID: <20110506115717.1992.541890468.divmod.xquotient.1245@localhost.localdomain> On 12:40 am, santagada at gmail.com wrote: >On Thu, May 5, 2011 at 6:13 PM, Maciej Fijalkowski >wrote: >>>I have never contributed to an open source project before so I need a >>>little >>>guidance about how things work here. ?Thanks, let me know! >>>--Daniel >> >>It's very simple. You come up with an idea what do you want to work on >>(this is done, pypy website) so you need to check out an appropriate >>repository. >> >>http://bitbucket.org/pypy/pypy.org >> >>then you either create a branch and issue a pull request or create a >>patch or whatever way is really convinient. >> >>Feel free to join #pypy on irc.freenode.net if you have any more >>questions > >Sorry to disagree but wouldn't just doing a sketch in photoshop and Sorry, this isn't really very on topic, but photoshop is a stupid, worse-than-useless tool for designing websites. Sketch it in html+css+javascript. If you can't do that, then you haven't really designed anything. Jean-Paul >posting the design here be enough for a start? Then do the >html+css+javascript needed and only then think about doing a template, >which if you don't know how to do there are plenty of people that >knows that and can do it for you in 2 hours tops if you have >everything already on html. > >I think the problem is the design of the website, not coding the >template in jinja2. > >-- >Leonardo Santagada >_______________________________________________ >pypy-dev at codespeak.net >http://codespeak.net/mailman/listinfo/pypy-dev From qbproger at gmail.com Fri May 6 19:29:08 2011 From: qbproger at gmail.com (Joe) Date: Fri, 6 May 2011 13:29:08 -0400 Subject: [pypy-dev] PyPy Assembler SQRT Patch In-Reply-To: References: Message-ID: Attached an updated patch. It actually generates the assembly now to use SQRTSD. It provides about the same performance as the previous patch. Because of that, I feel as though the performance gain can be attributed to the change in error checking. It might be worth splitting out the other math functions and slimming down the error checking. Right now in the new_unary_math_function it generates a math function that does 3 C library calls 2 errno calls and the math function itself. The patch takes this down to the minimal number of checks necessary for math.sqrt, making only an isfinite() call in addition to the sqrt() call best case. Previously it'd call "_error_reset(); c_func(); rposix.get_errno()" and then at least 1 call to isnan() and a call to either isnan() or isinf(). Does this seem like something worth looking into for the other math functions? It wouldn't be as hard as getting the whole assembly instruction working, and seems like it provides the majority of the performance gain. Joe On Tue, May 3, 2011 at 12:28 PM, Armin Rigo wrote: > Hi Alex, > > On Tue, May 3, 2011 at 6:23 PM, Alex Gaynor wrote: >> Another thing is it would be useful to test that the SQRTSD is generated, I >> don't know a good way to do that though. > > In the test, you can hack genop_math_sqrt() in-place to set a flag, > and test that the flag is set... > > And another place it should also be tested is in test_pypy_c_new.py > (pypy.module.pypyjit.test_pypy_c), i.e. you should write a test that > checks that in a full translated pypy-c, the code corresponding to the > Python source "math.sqrt(x)" indeed ends up being a CALL(sqrt_wrapper) > and not a CALL(ll_math_sqrt). > > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -------------- next part -------------- A non-text attachment was scrubbed... Name: sqrtsd.patch Type: application/octet-stream Size: 17795 bytes Desc: not available Url : http://codespeak.net/pipermail/pypy-dev/attachments/20110506/2ec62d2d/attachment-0001.obj From kristjan at ccpgames.com Fri May 6 20:30:55 2011 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Sat, 7 May 2011 02:30:55 +0800 Subject: [pypy-dev] cheerio Message-ID: <2E034B571A5CE44E949B9FCC3B6D24EE6D298E70@exchcn.ccp.ad.local> Hi there. I finally got round to subscribe after the G?teborg sprint. Hi there, to all of you sprinters, thanks for the fun! Meanwhile, the Stackless project urgently feels the need to resolve the RCS system. Any suggestions on how to proceed with moving the stackless repository from the python svn one to Hg? Also, there is the idea we mentioned to have a 2.7 branch that we can freely modify for our mutual purposes, e.g. by fixing some of the unittests and other stuff. Any thoughts on that? -Kristj?n -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110507/10b4de3f/attachment.htm From dimaqq at gmail.com Fri May 6 21:53:45 2011 From: dimaqq at gmail.com (Dima Tisnek) Date: Fri, 6 May 2011 12:53:45 -0700 Subject: [pypy-dev] cheerio In-Reply-To: <2E034B571A5CE44E949B9FCC3B6D24EE6D298E70@exchcn.ccp.ad.local> References: <2E034B571A5CE44E949B9FCC3B6D24EE6D298E70@exchcn.ccp.ad.local> Message-ID: go git, much better 2011/5/6 Kristj?n Valur J?nsson : > Hi there. > > I finally got round to subscribe after the G?teborg sprint.? Hi there, to > all of you sprinters, thanks for the fun! > > > > Meanwhile, the Stackless project urgently feels the need to resolve the RCS > system. > > Any suggestions on how to proceed with moving the stackless repository from > the python svn one to Hg? > > Also, there is the idea we mentioned to have a 2.7 branch that we can freely > modify for our mutual purposes, e.g. by fixing some of the unittests and > other stuff.? Any thoughts on that? > > > > -Kristj?n > > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From Ronny.Pfannschmidt at gmx.de Fri May 6 22:20:42 2011 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Fri, 06 May 2011 22:20:42 +0200 Subject: [pypy-dev] cheerio In-Reply-To: <2E034B571A5CE44E949B9FCC3B6D24EE6D298E70@exchcn.ccp.ad.local> References: <2E034B571A5CE44E949B9FCC3B6D24EE6D298E70@exchcn.ccp.ad.local> Message-ID: <1304713242.4655.6.camel@Klappe2> On Sat, 2011-05-07 at 02:30 +0800, Kristj?n Valur J?nsson wrote: > Hi there. > > I finally got round to subscribe after the G?teborg sprint. Hi there, > to all of you sprinters, thanks for the fun! > > > > Meanwhile, the Stackless project urgently feels the need to resolve > the RCS system. > > Any suggestions on how to proceed with moving the stackless repository > from the python svn one to Hg? that depends mainly on how stackless managed syncing with cpython and if/how you want to record that in the new history also the importance of merges being recorded is a important question its basically choosing the tradeoff between correctness and ease of migration (for pypy we skipped merges and for old files we have to resort to using annotate on svn) > > Also, there is the idea we mentioned to have a 2.7 branch that we can > freely modify for our mutual purposes, e.g. by fixing some of the > unittests and other stuff. Any thoughts on that? there is some talk on splitting the stdlib to a separate repo (which may be more interesting for everyone) -- Ronny > > > > -Kristj?n > > > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part Url : http://codespeak.net/pipermail/pypy-dev/attachments/20110506/06d4c44f/attachment.pgp From qbproger at gmail.com Sat May 7 06:15:50 2011 From: qbproger at gmail.com (Joe) Date: Sat, 7 May 2011 00:15:50 -0400 Subject: [pypy-dev] PyPy Assembler SQRT Patch In-Reply-To: References: Message-ID: My mistake. There was a bug so it appeared as though SQRTSD wasn't being used while running the tests. It turns out the first patch I posted probably worked. After benchmarking, SQRTSD is definitely the cause of the speedup. Sorry for the confusion. Joe On Fri, May 6, 2011 at 1:29 PM, Joe wrote: > Attached an updated patch. ?It actually generates the assembly now to > use SQRTSD. ?It provides about the same performance as the previous > patch. ?Because of that, I feel as though the performance gain can be > attributed to the change in error checking. > > It might be worth splitting out the other math functions and slimming > down the error checking. ?Right now in the new_unary_math_function it > generates a math function that does 3 C library calls 2 errno calls > and the math function itself. ?The patch takes this down to the > minimal number of checks necessary for math.sqrt, making only an > isfinite() call in addition to the sqrt() call best case. ?Previously > it'd call "_error_reset(); c_func(); rposix.get_errno()" and then at > least 1 call to isnan() and a call to either isnan() or isinf(). > > Does this seem like something worth looking into for the other math > functions? ?It wouldn't be as hard as getting the whole assembly > instruction working, and seems like it provides the majority of the > performance gain. > > Joe > > On Tue, May 3, 2011 at 12:28 PM, Armin Rigo wrote: >> Hi Alex, >> >> On Tue, May 3, 2011 at 6:23 PM, Alex Gaynor wrote: >>> Another thing is it would be useful to test that the SQRTSD is generated, I >>> don't know a good way to do that though. >> >> In the test, you can hack genop_math_sqrt() in-place to set a flag, >> and test that the flag is set... >> >> And another place it should also be tested is in test_pypy_c_new.py >> (pypy.module.pypyjit.test_pypy_c), i.e. you should write a test that >> checks that in a full translated pypy-c, the code corresponding to the >> Python source "math.sqrt(x)" indeed ends up being a CALL(sqrt_wrapper) >> and not a CALL(ll_math_sqrt). >> >> >> A bient?t, >> >> Armin. >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev >> > From arigo at tunes.org Sat May 7 11:41:31 2011 From: arigo at tunes.org (Armin Rigo) Date: Sat, 7 May 2011 11:41:31 +0200 Subject: [pypy-dev] PyPy Assembler SQRT Patch In-Reply-To: References: Message-ID: Hi Joe, On Sat, May 7, 2011 at 6:15 AM, Joe wrote: > My mistake. ?There was a bug so it appeared as though SQRTSD wasn't > being used while running the tests. ?It turns out the first patch I > posted probably worked. ?After benchmarking, SQRTSD is definitely the > cause of the speedup. ?Sorry for the confusion. Sorry, the confusion probably came from my side. I missed that running the tests with interp_operations() would not set supports_floats=True :-( That's fixed now. Thank you again for your patience. Armin From estama at gmail.com Sun May 8 03:13:57 2011 From: estama at gmail.com (Elefterios Stamatogiannakis) Date: Sun, 08 May 2011 04:13:57 +0300 Subject: [pypy-dev] Pypy bugs Message-ID: <4DC5EE55.4080004@gmail.com> Hi all, and many many thanks for pypy. Your work on pypy is exceptional. Due to Roger Binns doing the first steps of porting APSW on top of pypy, i've tried my project (madIS: http://code.google.com/p/madis ) with pypy and head APSW, and found the following: In python 2.7.1: In [1]: print dict.__setattr__ In pypy 1.5: In [10]: print dict.__setattr__ In essence in python 2.7.1 dict.__setattr__ is the same as object.__setattr__ whereas in pypy dict.__setattr__ works only with dictionaries, so it throws the following error when called with an object: TypeError: unbound method __setattr__() must be called with dict instance as first argument Above problem was solved by changing all dict.__setattr__ invocations into object.__setattr__ invocations, and now the same code works in both CPython and pypy. The second problem that i found with pypy is this: In [12]: import xml.etree.c xml.etree.cElementTree Above is to show that cElementTree exists in pypy's xml.etree. If then i try to import cElementTree: In [11]: import xml.etree.cElementTree --------------------------------------------------------------------------- ImportError Traceback (most recent call last) /home/estama/programs/pypy-15/bin/ in () /home/estama/programs/pypy-15/lib-python/2.7/xml/etree/cElementTree.pyc in () 1 # Wrapper module for _elementtree 2 ----> 3 from _elementtree import * ImportError: No module named _elementtree Thanks again for pypy, and to Roger Binns for your outstanding work on APSW. lefteris. From drsalists at gmail.com Sun May 8 05:06:00 2011 From: drsalists at gmail.com (Dan Stromberg) Date: Sat, 7 May 2011 20:06:00 -0700 Subject: [pypy-dev] Pypy bugs In-Reply-To: <4DC5EE55.4080004@gmail.com> References: <4DC5EE55.4080004@gmail.com> Message-ID: Hi Elefterios. I don't really work on PyPy, but I've written code that runs on PyPy and CPython 2&3 and Jython, unmodified. I tried IronPython too, but it was too different from the others. On Sat, May 7, 2011 at 6:13 PM, Elefterios Stamatogiannakis < estama at gmail.com> wrote: > Hi all, and many many thanks for pypy. Your work on pypy is exceptional. > > Due to Roger Binns doing the first steps of porting APSW on top of pypy, > i've tried my project (madIS: http://code.google.com/p/madis ) with pypy > and head APSW, and found the following: > > In python 2.7.1: > In [1]: print dict.__setattr__ > > > In pypy 1.5: > In [10]: print dict.__setattr__ > > > In essence in python 2.7.1 dict.__setattr__ is the same as > object.__setattr__ whereas in pypy dict.__setattr__ works only with > dictionaries, so it throws the following error when called with an object: > > TypeError: unbound method __setattr__() must be called with dict > instance as first argument > > Above problem was solved by changing all dict.__setattr__ invocations > into object.__setattr__ invocations, and now the same code works in both > CPython and pypy. > Actually, this feels a bit like it was a portability issue in your code to me. Why should dict's __setattr__ apply to all objects? Doesn't it seem more natural that it would apply to dicts alone, and if it applies to more, that that's an implementation detail? > The second problem that i found with pypy is this: > > In [12]: import xml.etree.c > xml.etree.cElementTree > > Above is to show that cElementTree exists in pypy's xml.etree. If then i > try to import cElementTree: > > In [11]: import xml.etree.cElementTree > --------------------------------------------------------------------------- > ImportError Traceback (most recent call last) > > /home/estama/programs/pypy-15/bin/ in () > > /home/estama/programs/pypy-15/lib-python/2.7/xml/etree/cElementTree.pyc > in () > 1 # Wrapper module for _elementtree > > 2 > ----> 3 from _elementtree import * > > ImportError: No module named _elementtree > > Thanks again for pypy, and to Roger Binns for your outstanding work on > APSW. > PyPy in general doesn't include a bunch of C extension modules; it's written in Python and RPython, so this seems somewhat natural. Your code probably should do something like: try: import xml.etree.cElementTree as element_tree except ImportError: import xml.etree.ElementTree as element_tree ...and then use element_tree.whatever in your code. It's 3 more lines, but more portable. HTH. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110507/8058359d/attachment-0001.htm From rubycoder at gmail.com Sun May 8 09:39:54 2011 From: rubycoder at gmail.com (Gustavo Kuerten) Date: Sun, 8 May 2011 15:39:54 +0800 Subject: [pypy-dev] rlb.parsing left recursion support ? Message-ID: Hi, I noticed that there is a test in test_pypackrat.py which seems to suggest that the parser supports left recursion. a: a ":" | "b"; However running with the ebnf above failed with the error: nonterminal a is in its own follow set(['a']) AssertionError assert not self.has_left_recursion() So, does it or does it not support left recursion ? -- Regards, S -------------- next part -------------- An HTML attachment was scrubbed... URL: http://codespeak.net/pipermail/pypy-dev/attachments/20110508/4d3ea6e2/attachment.htm From estama at gmail.com Sun May 8 12:18:56 2011 From: estama at gmail.com (Elefterios Stamatogiannakis) Date: Sun, 08 May 2011 13:18:56 +0300 Subject: [pypy-dev] Pypy bugs In-Reply-To: References: <4DC5EE55.4080004@gmail.com> Message-ID: <4DC66E10.7060306@gmail.com> On 08/05/11 06:06, Dan Stromberg wrote: > > Hi Elefterios. > > I don't really work on PyPy, but I've written code that runs on PyPy and > CPython 2&3 and Jython, unmodified. I tried IronPython too, but it was > too different from the others. I don't really know how pypy wants to present itself. From reading the site and blog, i got the impression that pypy wishes to become an alternative to (and maybe replace it?) CPython, and not a "parallel universe" Python like Jython and IronPython. If this is indeed true, then pypy has to mirror some of the quirks and historic bugs of CPython, that people have come to depend on. Nevertheless, i don't mind at all, if a porting effort is required to retarget something to also run on pypy. And from what i see, you have a lot more experience in doing this than me :-) . > > On Sat, May 7, 2011 at 6:13 PM, Elefterios Stamatogiannakis > > wrote: > > Hi all, and many many thanks for pypy. Your work on pypy is exceptional. > > Due to Roger Binns doing the first steps of porting APSW on top of pypy, > i've tried my project (madIS: http://code.google.com/p/madis ) with pypy > and head APSW, and found the following: > > In python 2.7.1: > In [1]: print dict.__setattr__ > > > In pypy 1.5: > In [10]: print dict.__setattr__ > > > In essence in python 2.7.1 dict.__setattr__ is the same as > object.__setattr__ whereas in pypy dict.__setattr__ works only with > dictionaries, so it throws the following error when called with an > object: > > TypeError: unbound method __setattr__() must be called with dict > instance as first argument > > Above problem was solved by changing all dict.__setattr__ invocations > into object.__setattr__ invocations, and now the same code works in both > CPython and pypy. > > > Actually, this feels a bit like it was a portability issue in your code > to me. Why should dict's __setattr__ apply to all objects? Doesn't it > seem more natural that it would apply to dicts alone, and if it applies > to more, that that's an implementation detail? You are right, and at first i too thought that CPython's behavior wasn't right. Nevertheless, maybe there is a reason that CPython behaves in this non consistent way, and other programs may hit this too. As it is right now, i prefer pypy's implementation on this, as it makes more sense, and it is easily corrected. > > The second problem that i found with pypy is this: > > In [12]: import xml.etree.c > xml.etree.cElementTree > > Above is to show that cElementTree exists in pypy's xml.etree. If then i > try to import cElementTree: > > In [11]: import xml.etree.cElementTree > --------------------------------------------------------------------------- > ImportError Traceback (most recent > call last) > > /home/estama/programs/pypy-15/bin/ in () > > /home/estama/programs/pypy-15/lib-python/2.7/xml/etree/cElementTree.pyc > in () > 1 # Wrapper module for _elementtree > > 2 > ----> 3 from _elementtree import * > > ImportError: No module named _elementtree > > Thanks again for pypy, and to Roger Binns for your outstanding work > on APSW. > > > PyPy in general doesn't include a bunch of C extension modules; it's > written in Python and RPython, so this seems somewhat natural. Your > code probably should do something like: > > try: > import xml.etree.cElementTree as element_tree > except ImportError: > import xml.etree.ElementTree as element_tree > > ...and then use element_tree.whatever in your code. It's 3 more lines, > but more portable. > > HTH. You are exactly right, and i've more or less used code like the one you presented. My only problem is that i believe that pypy shouldn't have answered at my first example (In [12]:...) above that cElementTree exists in xml.etree .Or maybe it should short circuit cElementTree to point at ElementTree. To declare that something exists, and then when somebody tries to use it, to throw an exception, isn't that nice. l. From arigo at tunes.org Sun May 8 13:17:23 2011 From: arigo at tunes.org (Armin Rigo) Date: Sun, 8 May 2011 13:17:23 +0200 Subject: [pypy-dev] Pypy bugs In-Reply-To: <4DC66E10.7060306@gmail.com> References: <4DC5EE55.4080004@gmail.com> <4DC66E10.7060306@gmail.com> Message-ID: Hi Elefterios, On Sun, May 8, 2011 at 12:18 PM, Elefterios Stamatogiannakis wrote: > You are exactly right, and i've more or less used code like the one you > presented. My only problem is that i believe that pypy shouldn't have > answered at my first example (In [12]:...) above that cElementTree > exists in xml.etree .Or maybe it should short circuit cElementTree to > point at ElementTree. To declare that something exists, and then when > somebody tries to use it, to throw an exception, isn't that nice. Nothing we can do: the module cElementTree is actually written in Python also in CPython --- it is cElementTree.py --- but it import internally "_elementtree". So there is no way PyPy can know that cElementTree cannot be imported before actually trying to. The workaround to write "try: import cXxx; except ImportError: import xxx" is the general solution. It works in other cases too (cProfile, cPickle, cStringIO, etc). And also, it works in case the version of CPython you are running on misses some C extension modules. For example you would get the exact same error as you got on PyPy if you are running on a CPython where the C module "_elementree" was not compiled. A bient?t, Armin. From arigo at tunes.org Sun May 8 13:22:07 2011 From: arigo at tunes.org (Armin Rigo) Date: Sun, 8 May 2011 13:22:07 +0200 Subject: [pypy-dev] rlb.parsing left recursion support ? In-Reply-To: References: Message-ID: Hi Gustavo, On Sun, May 8, 2011 at 9:39 AM, Gustavo Kuerten wrote: > I noticed that there is a test in test_pypackrat.py which seems to suggest > that the parser supports left recursion. > > a: a ":" | "b"; I cannot find this test anywhere. Can you tell more precisely where it is from? I looked in pypy/rlib/parsing/test/test_pypackrat.py. > So, does it or does it not support left recursion ? It does not, as far as I know. Armin From pypy at pocketnix.org Sun May 8 13:22:58 2011 From: pypy at pocketnix.org (Da_Blitz) Date: Sun, 8 May 2011 11:22:58 +0000 Subject: [pypy-dev] Pypy bugs In-Reply-To: <4DC66E10.7060306@gmail.com> References: <4DC5EE55.4080004@gmail.com> <4DC66E10.7060306@gmail.com> Message-ID: <20110508112258.GA14145@pocketnix.org> On Sun, May 08, 2011 at 01:18:56PM +0300, Elefterios Stamatogiannakis wrote: > You are right, and at first i too thought that CPython's behavior wasn't > right. Nevertheless, maybe there is a reason that CPython behaves in > this non consistent way, and other programs may hit this too. > > As it is right now, i prefer pypy's implementation on this, as it makes > more sense, and it is easily corrected. I had a quick look at your code and i don't fully grasp why you didn't just use setattr which in my understanding is the correct way to go about this, touching anything starting with "__" or "_" should be avoided as you should not rely on those attributes, for example when passed in as an argument to a function/method. i understand that you are invoking it on a known object and not a passed argument but i would still apply the same rules and try to avoid relying on direct access to those attributes and instead call them in the proper manner i would have a hard time calling this a pypy bug, i would say the code is too closly coupled to the implemetnation of dict. > You are exactly right, and i've more or less used code like the one you > presented. My only problem is that i believe that pypy shouldn't have > answered at my first example (In [12]:...) above that cElementTree > exists in xml.etree .Or maybe it should short circuit cElementTree to > point at ElementTree. To declare that something exists, and then when > somebody tries to use it, to throw an exception, isn't that nice. looks like the lib is in the lib-python/2.7 and not lib-python/modified-2.7 directory and so it appears to be a side effect of importing the python std-lib as a side note cElementTree is gone in python 3.3 (didn't check the other versions) and most python libs with optional c versions now autodetect if the lib is installed and use that (eg Pickle and cPickle). using the try/except clause is the recommended idiom to avoid this situation when there are multiple versions of the same lib available i also would not call this a pypy bug as you rightly pointed out people may see pypy as a cpython replacment and perhaps noting the above 2 things in the cpython_diffrences.rst file in the doc folder would be a good idea hope i am right, feel free to flame me if i am not From rubycoder at gmail.com Sun May 8 14:53:40 2011 From: rubycoder at gmail.com (Gustavo Kuerten) Date: Sun, 8 May 2011 20:53:40 +0800 Subject: [pypy-dev] rlb.parsing left recursion support ? In-Reply-To: References: Message-ID: Armin, My example is not equivalent to the actual test. But here is the snippet from test_pypackrat.py ?? ?def test_leftrecursion(self): ?? ? ? ?class parser(PackratParser): ?? ? ? ? ? ?""" ?? ? ? ? ? ?b: b 'a' | 'b'; ?? ? ? ? ? ?""" ?? ? ? ?print parser._code ?? ? ? ?p = parser("b") ?? ? ? ?res = p.b() ?? ? ? ?assert res == "b" ?? ? ? ?p = parser("bac") ?? ? ? ?res = p.b() ?? ? ? ?assert p._pos == 2 ?? ? ? ?assert res == "a" ?? ? ? ?p = parser("baaaaaaaaaaaaaac") ?? ? ? ?res = p.b() ?? ? ? ?assert p._pos == 15 ?? ? ? ?assert res == "a" This seems to indicate that it does support?left recursion. 2011/5/8 Armin Rigo > > Hi Gustavo, > > On Sun, May 8, 2011 at 9:39 AM, Gustavo Kuerten wrote: > > I noticed that there is a test in test_pypackrat.py which seems to suggest > > that the parser supports left recursion. > > > > a: a ":" | "b"; > > I cannot find this test anywhere. ?Can you tell more precisely where > it is from? ?I looked in pypy/rlib/parsing/test/test_pypackrat.py. > > > So, does it or does it not support left recursion ? > > It does not, as far as I know. > > > Armin -- Regards, G From arigo at tunes.org Sun May 8 15:07:22 2011 From: arigo at tunes.org (Armin Rigo) Date: Sun, 8 May 2011 15:07:22 +0200 Subject: [pypy-dev] rlb.parsing left recursion support ? In-Reply-To: References: Message-ID: Hi Gustavo, On Sun, May 8, 2011 at 2:53 PM, Gustavo Kuerten wrote: > This seems to indicate that it does support?left recursion. Ah, sorry. I have no real clue, so I can just answer from looking at the source code, which I'm sure you did already for longer than me. There seems to be two classes called PackratParser. You are getting the error from the one in parsing.py, but test_pypackrat uses the one in makepackrat.py. Even if you want to use the class in parsing.py, it takes an argument "check_for_left_recursion" which you can try setting to False. Well, it's probably obvious to you; if so, sorry not to be more useful. Armin. From cfbolz at gmx.de Mon May 9 10:20:18 2011 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 09 May 2011 10:20:18 +0200 Subject: [pypy-dev] rlb.parsing left recursion support ? In-Reply-To: References: Message-ID: <4DC7A3C2.5090500@gmx.de> On 05/08/2011 02:53 PM, Gustavo Kuerten wrote: > Armin, > > My example is not equivalent to the actual test. > But here is the snippet from test_pypackrat.py > > def test_leftrecursion(self): > class parser(PackratParser): > """ > b: b 'a' | 'b'; > """ > print parser._code > p = parser("b") > res = p.b() > assert res == "b" > p = parser("bac") > res = p.b() > assert p._pos == 2 > assert res == "a" > p = parser("baaaaaaaaaaaaaac") > res = p.b() > assert p._pos == 15 > assert res == "a" > > This seems to indicate that it does support left recursion. There are actually two parser generators in rlib/parsing. This is a test for the second, experimental, one. The "official" one does not support left recursion. The second one is not stable enough to be used, and I guess I should remove it from rlib/ at some point given that I also don't plan to continue working on it. Carl Friedrich From holger at merlinux.eu Mon May 9 15:01:53 2011 From: holger at merlinux.eu (holger krekel) Date: Mon, 9 May 2011 13:01:53 +0000 Subject: [pypy-dev] moving to python.org ... Message-ID: <20110509130153.GH15987@merlinux.eu> Hi all, We are today / currently migrating pypy mailing lists to python.org, made possible by the great help from Ralf Hildebrandt. Here are the new "From" addresses: pypy-dev at codespeak.net -> pypy-dev at python.org pypy-issue at codespeak.net -> pypy-issue at python.org pypy-svn at codespeak.net -> pypy-commit at python.org pypy-z at codespeak.net -> pypy-z at python.org Note the move from "pypy-svn" to the "pypy-commit" name. All membership information is retained but you will now get mail from python.org so you may need to adjust your filters. If i do things correctly you can still send mail to the old mail addresses for at least another week in addition to the new ones. best, holger