[pypy-dev] Work plan for PyPy

[Martijn Faassen]

Hey,

Christian Tismer wrote:
> Whatever way we choose, I think it would be best for PyPy
> if we try to be as user-friendly as possible, saying like
> """we believe we must change this and that immediately, but
> we will not leave you in the rain, but actively help porting
> your stuff."""
> 
> This also relates a bit to my former thoughts about things
> which we have donated to the open source. We can, physically,
> just trash anything, of course. But from a point of view
> of users and our growing community, this is a matter of belief
> and trust that we should be careful about.
> 
> I'm not saying that we don't care, these are just my thoughts
> and feelings about what I want to be not questionable.

Thanks Christian for making these points and having these concerns. I 
think that the project is wise to consider these issues once every while.

I only understand a little of the technical discussion. I understand 
that RCTypes has problems, and that people anticipate having to break it 
and reimplement it in a different way in order to solve these problems. 
I understand that one feature that RCTypes offers, namely the ability to 
test your RPython code on CPython, might also go away.

There are currently, as far as I can see it, two reasons to use RPython 
for CPython extensions:

a) benefit from performance gains by rewriting your CPython code into 
RPython. This is what we want to do with the page template interpreter. 
In case of language implementations, this might lead to the ability to 
use other PyPy technology later (such as JIT generation and so on). Some 
others might purely be interested in performance though and aren't 
implementing a language.

b) better ways to write extensions that use C libraries in CPython. The 
typical Pyrex use case, though it can do the former as well. This would 
use RCTypes.

Use case a) probably has less interested people than use case b). 
Eventually JIT technologies will help here, which will make use case a) 
less important.

I happen to be interested in use case a) though. It is quite exciting to 
me that we could have a faster template language interpreter 
implementation that we can use in CPython today, with the potential 
benefits of PyPy for interpreter translation in the future as well. We 
are basically language implementors, one of your core target audiences. 
We just feel more comfortable doing work if we know we have payoffs in 
the near future, i.e. use in CPython. If we were implementing this 
template language in isolation and had no hope we could deploy it in 
production any time soon, we would be less interested.

I am actually quite interested in use case b) as well. While developing 
lxml, I've done quite a bit of work with Pyrex and I can see how RPython 
and RCTypes might potentially help projects like that. I've been taking 
a wait and see attitude here though, as the technology needed more maturing.

As I understand it, the proposed changes wouldn't affect use case a). Or 
is the ability to generate CPython extensions also going to be broken 
due to this work? Please let us know, as if CPython support is going to 
be broken, we would likely stop our efforts for the time being.

Use case b) would certainly be broken due to this work. What's more, I 
understand that testing your extension code in CPython first wouldn't 
work anymore (at least if it uses CTypes). This is a very nice feature 
of RPython that Pyrex doesn't have, by the way. You mentioned that there 
are potential ways to reimplement this feature again, but I'm not sure 
whether you're committing to such implementation.

If I were writing C extension modules with PyPy I'd like to know things 
like:

* what are the reasons this change is going to be made?

* would this benefit just core PyPy or would it benefit extension module 
writers too?

* when do you expect the new way to be finalized?

* will you make a real commitment to make the "Test in CPython first" 
feature work again? Or is did you just sketched out possible ways to 
implement this in the new system and you hope some contributor is going 
to do the work?

* will you make a commitment to make this new way useful and supported 
for extension module writers, or should they invest their time in some 
other technology? I.e. what are the changes a future shift will make me 
change things again?

* How well documented will this be for CPython extension developers? 
Will you market this feature to developers? Things like this indicate 
your commitment in another way. :)

Please note that I'm not actually *asking* for anything but information. 
I'm not telling the PyPy developers they should make commitments and do 
various kinds of work. That's up to you. I'm just trying to guess what 
other developers might want to know.

Of course, the quality of your communication and the answers you 
actually give will make people decide whether to continue to use PyPy or 
move on. This will affect the way your open source community will grow 
and develop.

I would urge the PyPy developers to consider investing in the open 
source community by supporting features like this and promoting them. 
This would mean serving admittedly self-interested developers with short 
term goals. The promise of immediate payoff is a great way to attract 
people to an open source project. If there is the hope of continuing 
payoff (and PyPy has tremendous potential here), some of those new 
people can be expected to become valuable contributors to the project as 
a whole. And yes, I'm self-interested in saying all of this - that's my 
point.

Regards,

Martijn