From brett.calcott at gmail.com  Mon Nov  1 04:44:37 2010
From: brett.calcott at gmail.com (Brett Calcott)
Date: Mon, 1 Nov 2010 14:44:37 +1100
Subject: [Cython] Wrapping C++ - questions and thoughts
Message-ID: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>

Hi,

I'm wrapping a C++ library (http://www.box2d.org/). A python version done
with swig already exists, but I'm trying to make it more pythonic/cythonic,
and preparing it for a future project. It's all going well, and I'm having a
fun time -- cython rocks.

I have a few questions and thoughts about wrapping. Consider wrapping
something simple like this, where things are public:

class Vector2d {
public:
  float x, y
  Vector2d() : x(0.0), y(0.0) {}
  // other useful stuff here
};

I begin by doing this:
extern from "vector2d.h":
  cdef cppclass Vector2d:
     float x, y
     Vector2d()

Here are the questions:
1. Why must I use a *pointer* in my python class?

cdef class PythonVector2d:
  cdef Vector2d *obj

There is a default constructor, and I can happily declare it on the stack in
a function. I'm sure I'm missing something here; is there some danger with
allowing this?

2. How hard would it be to automatically forward member access to a
contained cppclass? Consider the following nasty hack:

cdef class PythonVector2d:
  cdef public float x, y
  cdef Vector2d *obj
  def __cinit__(self, x, y):
    self.x = x
    self.y = y
    self.obj = <Vector 2d*>x

ok. That is a bit nasty (I'm sure someone can tell me lots of reasons why I
shouldn't do this). But it really makes things easy. I don't have to declare
all the properties independently. And when I need the cpp version, I just
pass around pointer.

Something like this would be nice:

cdef class PythonVector2d:
  cdef Vector obj
  properties:
     public obj.x as x
     public obj.y as y

Does this look crazy?

Cheers,
Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20101101/46e2d9da/attachment.htm 

From vitja.makarov at gmail.com  Mon Nov  1 08:46:56 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Mon, 1 Nov 2010 10:46:56 +0300
Subject: [Cython] metaclasses
Message-ID: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>

Hi, all!

I want to see metaclasses in cython. So I look into the code and I see
that classes are generated like this:

class Foo:
   xxx = 111

is transformed in something like:

Foo = Pyx_CreateClass('Foo', bases=(), attrs={})
# and then it does setattr()
Foo.xxx = 111

So nothing to do with metaclasses in Pyx_CreateClass() as attributes
are set after class is actually created ;(
Btw if class dict will be filled before class creation it's easy to
handle metaclass stuff in Pyx_CreateClass.
This this should be hard as PyDict_SetItem() and PyDict_GetItem()
should be used while class body creation

In python it works like this:

# based on Python/ceval.c:build_class()
def create_class(name, bases, attrs):
    if '__metaclass__' in attrs:
        metaclass = attrs.get('__metaclass__')
    elif len(bases) > 0:
        base = bases[0]
        if hasattr(base, '__class__'):
            metaclass = base.__class__
        else:
            metaclass = type(base)
    else '__metaclass__' in globals():
        metaclass = globals().get('__metaclass__')
    else:
        metaclass = type
    return metaclass(name, bases, attrs)

Another tricky way is to transform ClassDefNode, that has
__metaclass__ attribute into two classes:

class Foo(object):
   __metaclass__ = Bar
   xxx = 111

transform into:

Foo = create_class('Foo', (), {})
Foo.xxx = 111
Foo = create_class('Foo', (object,),  Foo.__dict__())

Second way is much easy to implement (I think) adding
MetaclassTransform into pipeline, but seems to be dirty hack.

I want to ask what is the best way to implement this?

vitja.

From stefan_ml at behnel.de  Mon Nov  1 11:38:21 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 01 Nov 2010 11:38:21 +0100
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
Message-ID: <4CCE989D.6070907@behnel.de>

Vitja Makarov, 01.11.2010 08:46:
> I want to see metaclasses in cython.

+1, could you open an enhancement ticket in the bug tracker? I'm surprised 
there isn't one yet.


> So I look into the code and I see that classes are generated like this:
>
> class Foo:
>     xxx = 111
>
> is transformed in something like:
>
> Foo = Pyx_CreateClass('Foo', bases=(), attrs={})
> # and then it does setattr()
> Foo.xxx = 111
>
> So nothing to do with metaclasses in Pyx_CreateClass() as attributes
> are set after class is actually created ;(
> Btw if class dict will be filled before class creation it's easy to
> handle metaclass stuff in Pyx_CreateClass.

Your description seems to indicate that you want this only for Python 
classes, which is fine. I think building the dict before building the class 
is the right way to implement this, as the metaclass is allowed to do 
whatever it likes with its input, including changes to the class dict. So 
it really needs to know the complete dict.

Also see this post:

http://thread.gmane.org/gmane.comp.python.cython.devel/10388

I don't know if the method binding problem has gone away, but it's worth a 
try. If it hasn't, it's worth fixing the way methods work, IMHO, but that 
may not be low hanging fruit.

The post is from this originating thread, where Ryan Kelly proposed ways to 
implement metaclasses:

http://thread.gmane.org/gmane.comp.python.cython.user/1652

I don't think metaclasses for cdef classes are ready to be implemented yet, 
but it should be easier for Python classes.

Also note that there is a new syntax for metaclasses in Py3, which I prefer 
over the Py2 way. So we will eventually have to support both, although the 
Py2 way is easy to emulate for Py3 code.

Stefan

From vitja.makarov at gmail.com  Mon Nov  1 12:58:18 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Mon, 1 Nov 2010 14:58:18 +0300
Subject: [Cython] metaclasses
In-Reply-To: <4CCE989D.6070907@behnel.de>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CCE989D.6070907@behnel.de>
Message-ID: <AANLkTik-nqf-4bAwMmzcTMOCBf=3o8=mxOKrt8MvA59Z@mail.gmail.com>

2010/11/1 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 01.11.2010 08:46:
>> I want to see metaclasses in cython.
>
> +1, could you open an enhancement ticket in the bug tracker? I'm surprised
> there isn't one yet.
>
>

It seems that I don't have write access to trac.

>> So I look into the code and I see that classes are generated like this:
>>
>> class Foo:
>> ? ? xxx = 111
>>
>> is transformed in something like:
>>
>> Foo = Pyx_CreateClass('Foo', bases=(), attrs={})
>> # and then it does setattr()
>> Foo.xxx = 111
>>
>> So nothing to do with metaclasses in Pyx_CreateClass() as attributes
>> are set after class is actually created ;(
>> Btw if class dict will be filled before class creation it's easy to
>> handle metaclass stuff in Pyx_CreateClass.
>
> Your description seems to indicate that you want this only for Python
> classes, which is fine. I think building the dict before building the class
> is the right way to implement this, as the metaclass is allowed to do
> whatever it likes with its input, including changes to the class dict. So
> it really needs to know the complete dict.
>
> Also see this post:
>
> http://thread.gmane.org/gmane.comp.python.cython.devel/10388
>
> I don't know if the method binding problem has gone away, but it's worth a
> try. If it hasn't, it's worth fixing the way methods work, IMHO, but that
> may not be low hanging fruit.
>
> The post is from this originating thread, where Ryan Kelly proposed ways to
> implement metaclasses:
>
> http://thread.gmane.org/gmane.comp.python.cython.user/1652
>
> I don't think metaclasses for cdef classes are ready to be implemented yet,
> but it should be easier for Python classes.
>
> Also note that there is a new syntax for metaclasses in Py3, which I prefer
> over the Py2 way. So we will eventually have to support both, although the
> Py2 way is easy to emulate for Py3 code.
>

This should be done in _Pyx_CreateClass() wrapper?

> Stefan


Yes, I'm mostly interested in metaclasses for "pure" python.
As I understand current PyClassDef implementation is mostly based on Pyrex one.

So is there a way to turn (set|get)attr into (set|get)item to write
dict instead of class object?

vitja.

From stefan_ml at behnel.de  Mon Nov  1 17:53:35 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 01 Nov 2010 17:53:35 +0100
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTik-nqf-4bAwMmzcTMOCBf=3o8=mxOKrt8MvA59Z@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>	<4CCE989D.6070907@behnel.de>
	<AANLkTik-nqf-4bAwMmzcTMOCBf=3o8=mxOKrt8MvA59Z@mail.gmail.com>
Message-ID: <4CCEF08F.70702@behnel.de>

Vitja Makarov, 01.11.2010 12:58:
> 2010/11/1 Stefan Behnel:
>> Vitja Makarov, 01.11.2010 08:46:
>>> I want to see metaclasses in cython.
>> +1, could you open an enhancement ticket in the bug tracker? I'm surprised
>> there isn't one yet.
>>
> It seems that I don't have write access to trac.

It's restricted to authorised users to prevent spam. Please send a htpasswd 
line and a user name to Robert Bradshaw.


>>> So I look into the code and I see that classes are generated like this:
>>>
>>> class Foo:
>>>      xxx = 111
>>>
>>> is transformed in something like:
>>>
>>> Foo = Pyx_CreateClass('Foo', bases=(), attrs={})
>>> # and then it does setattr()
>>> Foo.xxx = 111
>>>
>>> So nothing to do with metaclasses in Pyx_CreateClass() as attributes
>>> are set after class is actually created ;(
>>> Btw if class dict will be filled before class creation it's easy to
>>> handle metaclass stuff in Pyx_CreateClass.
>>
>> Your description seems to indicate that you want this only for Python
>> classes, which is fine.
>
> Yes, I'm mostly interested in metaclasses for "pure" python. As I
> understand current PyClassDef implementation is mostly based on Pyrex
> one.

Correct.


>> I think building the dict before building the class
>> is the right way to implement this, as the metaclass is allowed to do
>> whatever it likes with its input, including changes to the class dict. So
>> it really needs to know the complete dict.
>>
>> Also note that there is a new syntax for metaclasses in Py3, which I prefer
>> over the Py2 way. So we will eventually have to support both, although the
>> Py2 way is easy to emulate for Py3 code.
>
> This should be done in _Pyx_CreateClass() wrapper?

Partly, I think. If the metaclass is defined outside of the class, e.g. 
because it was passed as a keyword argument to the class, or because there 
is a globally defined name __metaclass__, I think it can safely be passed 
as an argument to that function from the calling code. The rest, i.e. 
checking for a locally defined name in the class dict, can be done inside 
of __Pyx_CreateClass(). I don't know how Py3 handles duplicate metaclass 
definitions through keyword *and* __metaclass__, but Cython should do it in 
the same way.


> So is there a way to turn (set|get)attr into (set|get)item to write
> dict instead of class object?

The class body (see PyClassDefNode in Nodes.py) is evaluated in its own 
scope (PyClassScope in Symtab.py). It might (!) work to build a DictNode at 
the end of the class body that contains all names defined in the scope with 
their current value, so that it can be passed into __Pyx_CreateClass(). Not 
sure how to handle "del" here, but I doubt that that currently works anyway.

Does that help? Please ask back on this list when you have any questions.

Stefan

From koepsell at gmail.com  Mon Nov  1 21:14:19 2010
From: koepsell at gmail.com (killian koepsell)
Date: Mon, 1 Nov 2010 13:14:19 -0700
Subject: [Cython] Fwd (from cython-user): pointers to numpy ndarray /
 conversion of numpy array to some type C++ array
Message-ID: <AANLkTinaOGxcnoEJRKR-nQWqcYKG5JPh0okkvb5nTGQJ@mail.gmail.com>

Hi,

I am not sure if my question is appropriate for the cython-dev list.
I thought, this might be a pretty common problem and I am curious
to hear an opinion of the cython developers.

Thanks,
  Kilian


---------- Forwarded message ----------
From: killian koepsell <koepsell at gmail.com>
Date: Sat, Oct 30, 2010 at 3:14 PM
Subject: pointers to numpy ndarray / conversion of numpy array to some
type C++ array
To: cython-users at googlegroups.com


Hi,

I would like to access a large number of numpy ndarrays (of different
size) from cython inside some inner loop. In python I would keep these
arrays inside an object attribute, such as a dictionary or a list of
arrays. However, as I understand it, I would always incure a
significant overhead if I access a python list from within cython
(since the list members are not typed). What is the best way to attach
a number of ndarrays to a python object in a way that they are fast
accessible inside cython without going through the python layer?

I was thinking of keeping a list of pointers in a C++ vector, however
I did not manage to obtain a pointer to a buffer of type e.g.
np.ndarray[y[np.double_t, dim=2]. If I only keep a pointer to the
ndarray.data, I loose the shape and stride information. The workaround
I came up with is to create an auxiliary struct or cppclass that
contains all the required information to access a numpy array (data
pointer, shape, strides) and to keep pointers to these auxiliary
object in a C++ vector. This works fast but seems somewhat cumbersome.
In particular, I lose all the nice cython buffer indexing features and
now have to use pointer arithmetics to access the data. I am wondering
if there is a better solution. Maybe, it is possible to obtain a
pointer to the buffer objects that cython uses internally?

Or alternatively, maybe it is possible to create some type of C++
ndarrays without copying of the data and to store pointers to these
C++ arrays inside a C++ vector? Blitz, opencv, and boost come to mind,
but I haven't used either from inside cython. What would be the
best/easiest way to do that?

Thanks for any suggestions.

Kilian

From robertwb at math.washington.edu  Mon Nov  1 21:14:36 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 1 Nov 2010 13:14:36 -0700
Subject: [Cython] metaclasses
In-Reply-To: <4CCE989D.6070907@behnel.de>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CCE989D.6070907@behnel.de>
Message-ID: <AANLkTin+ofqt6-z+-Z-bnuSmW9eSDL94xDs=D=dT3ic7@mail.gmail.com>

On Mon, Nov 1, 2010 at 3:38 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Vitja Makarov, 01.11.2010 08:46:
>> I want to see metaclasses in cython.
>
> +1, could you open an enhancement ticket in the bug tracker? I'm surprised
> there isn't one yet.

+1 from me too, this'd be great to see. It's probably the largest
missing piece after generators.

>> So I look into the code and I see that classes are generated like this:
>>
>> class Foo:
>> ? ? xxx = 111
>>
>> is transformed in something like:
>>
>> Foo = Pyx_CreateClass('Foo', bases=(), attrs={})
>> # and then it does setattr()
>> Foo.xxx = 111
>>
>> So nothing to do with metaclasses in Pyx_CreateClass() as attributes
>> are set after class is actually created ;(
>> Btw if class dict will be filled before class creation it's easy to
>> handle metaclass stuff in Pyx_CreateClass.
>
> Your description seems to indicate that you want this only for Python
> classes, which is fine. I think building the dict before building the class
> is the right way to implement this, as the metaclass is allowed to do
> whatever it likes with its input, including changes to the class dict. So
> it really needs to know the complete dict.

I agree, we should create the dict first. It'll take some work, but
don't think this'll be that big of a change.

> Also see this post:
>
> http://thread.gmane.org/gmane.comp.python.cython.devel/10388
>
> I don't know if the method binding problem has gone away, but it's worth a
> try. If it hasn't, it's worth fixing the way methods work, IMHO, but that
> may not be low hanging fruit.

It has been fixed, in the sense that it has been implemented. I'm it's
enabled by default yet, but that would be a question of just flipping
a (context-triggered for now) switch.
http://trac.cython.org/cython_trac/ticket/494

> The post is from this originating thread, where Ryan Kelly proposed ways to
> implement metaclasses:
>
> http://thread.gmane.org/gmane.comp.python.cython.user/1652
>
> I don't think metaclasses for cdef classes are ready to be implemented yet,
> but it should be easier for Python classes.
>
> Also note that there is a new syntax for metaclasses in Py3, which I prefer
> over the Py2 way. So we will eventually have to support both, although the
> Py2 way is easy to emulate for Py3 code.
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From robertwb at math.washington.edu  Mon Nov  1 21:44:46 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 1 Nov 2010 13:44:46 -0700
Subject: [Cython] auto __test__ dict
In-Reply-To: <4CCDB7C3.20208@behnel.de>
References: <4CCC6F28.1040705@behnel.de> <4CCD1A7C.3010805@behnel.de>
	<4CCDB7C3.20208@behnel.de>
Message-ID: <AANLkTikBoY5pFx=AO=G7iVPmLqnrhwzjhZ1hTOA=SRO5@mail.gmail.com>

On Sun, Oct 31, 2010 at 11:38 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Stefan Behnel, 31.10.2010 08:27:
>> Stefan Behnel, 30.10.2010 21:16:
>>> can now write something like this for a type that you only use internally
>>> in your module:
>>>
>>> ? ? ? @cython.final
>>> ? ? ? @cython.internal
>>> ? ? ? cdef class _MyInternalType:
>>> ? ? ? ? ?...
>>
>> One problem I spotted was the auto test dict (again). It takes the classes
>> from the module dict if they have a docstring, which now fails for internal
>> types (it would also fail for 'del'-ed types, BTW). I always wondered why
>> it used the module dict anyway - I guess it's just simplicity. Any reason
>> not to rewrite that?
>
> I simplified it now to use the compile time docstrings directly, instead of
> looking them up at runtime. I don't think this breaks anything important,
> as the docstrings of functions and methods cannot be modified anyway. On
> the plus side, it removes a *ton* of ugly C code, and also saves some time
> during module initialisation.

Excellent! I've been wanting to do this myself, eventually when I
found the time. This means that cdef methods could have tested
docstrings as well (though, of course, they couldn't be called
directly).

- Robert

From robertwb at math.washington.edu  Mon Nov  1 21:46:53 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 1 Nov 2010 13:46:53 -0700
Subject: [Cython] Fwd (from cython-user): pointers to numpy ndarray /
 conversion of numpy array to some type C++ array
In-Reply-To: <AANLkTinaOGxcnoEJRKR-nQWqcYKG5JPh0okkvb5nTGQJ@mail.gmail.com>
References: <AANLkTinaOGxcnoEJRKR-nQWqcYKG5JPh0okkvb5nTGQJ@mail.gmail.com>
Message-ID: <AANLkTinNNHJi_=Ssn17atkf4Jsj4EnBbrXGavK2uq8Q6@mail.gmail.com>

Cython-users is the appropriate list for this kind of thing (and a
near super-set of this list, so no need to double-post). Incidentally,
I just gave some comments on that thread over there. Definitely report
back when you get something working.

On Mon, Nov 1, 2010 at 1:14 PM, killian koepsell <koepsell at gmail.com> wrote:
> Hi,
>
> I am not sure if my question is appropriate for the cython-dev list.
> I thought, this might be a pretty common problem and I am curious
> to hear an opinion of the cython developers.
>
> Thanks,
> ?Kilian
>
>
> ---------- Forwarded message ----------
> From: killian koepsell <koepsell at gmail.com>
> Date: Sat, Oct 30, 2010 at 3:14 PM
> Subject: pointers to numpy ndarray / conversion of numpy array to some
> type C++ array
> To: cython-users at googlegroups.com
>
>
> Hi,
>
> I would like to access a large number of numpy ndarrays (of different
> size) from cython inside some inner loop. In python I would keep these
> arrays inside an object attribute, such as a dictionary or a list of
> arrays. However, as I understand it, I would always incure a
> significant overhead if I access a python list from within cython
> (since the list members are not typed). What is the best way to attach
> a number of ndarrays to a python object in a way that they are fast
> accessible inside cython without going through the python layer?
>
> I was thinking of keeping a list of pointers in a C++ vector, however
> I did not manage to obtain a pointer to a buffer of type e.g.
> np.ndarray[y[np.double_t, dim=2]. If I only keep a pointer to the
> ndarray.data, I loose the shape and stride information. The workaround
> I came up with is to create an auxiliary struct or cppclass that
> contains all the required information to access a numpy array (data
> pointer, shape, strides) and to keep pointers to these auxiliary
> object in a C++ vector. This works fast but seems somewhat cumbersome.
> In particular, I lose all the nice cython buffer indexing features and
> now have to use pointer arithmetics to access the data. I am wondering
> if there is a better solution. Maybe, it is possible to obtain a
> pointer to the buffer objects that cython uses internally?
>
> Or alternatively, maybe it is possible to create some type of C++
> ndarrays without copying of the data and to store pointers to these
> C++ arrays inside a C++ vector? Blitz, opencv, and boost come to mind,
> but I haven't used either from inside cython. What would be the
> best/easiest way to do that?
>
> Thanks for any suggestions.
>
> Kilian
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From vitja.makarov at gmail.com  Mon Nov  1 21:53:12 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Mon, 1 Nov 2010 23:53:12 +0300
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTin+ofqt6-z+-Z-bnuSmW9eSDL94xDs=D=dT3ic7@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CCE989D.6070907@behnel.de>
	<AANLkTin+ofqt6-z+-Z-bnuSmW9eSDL94xDs=D=dT3ic7@mail.gmail.com>
Message-ID: <AANLkTikn4VJtcJ3EsESd4iVKNpg42fQK2EAdbAbuBA1o@mail.gmail.com>

Created ticket
http://trac.cython.org/cython_trac/ticket/587

2010/11/1 Robert Bradshaw <robertwb at math.washington.edu>:
> On Mon, Nov 1, 2010 at 3:38 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Vitja Makarov, 01.11.2010 08:46:
>>> I want to see metaclasses in cython.
>>
>> +1, could you open an enhancement ticket in the bug tracker? I'm surprised
>> there isn't one yet.
>
> +1 from me too, this'd be great to see. It's probably the largest
> missing piece after generators.
>
>>> So I look into the code and I see that classes are generated like this:
>>>
>>> class Foo:
>>> ? ? xxx = 111
>>>
>>> is transformed in something like:
>>>
>>> Foo = Pyx_CreateClass('Foo', bases=(), attrs={})
>>> # and then it does setattr()
>>> Foo.xxx = 111
>>>
>>> So nothing to do with metaclasses in Pyx_CreateClass() as attributes
>>> are set after class is actually created ;(
>>> Btw if class dict will be filled before class creation it's easy to
>>> handle metaclass stuff in Pyx_CreateClass.
>>
>> Your description seems to indicate that you want this only for Python
>> classes, which is fine. I think building the dict before building the class
>> is the right way to implement this, as the metaclass is allowed to do
>> whatever it likes with its input, including changes to the class dict. So
>> it really needs to know the complete dict.
>
> I agree, we should create the dict first. It'll take some work, but
> don't think this'll be that big of a change.
>
>> Also see this post:
>>
>> http://thread.gmane.org/gmane.comp.python.cython.devel/10388
>>
>> I don't know if the method binding problem has gone away, but it's worth a
>> try. If it hasn't, it's worth fixing the way methods work, IMHO, but that
>> may not be low hanging fruit.
>
> It has been fixed, in the sense that it has been implemented. I'm it's
> enabled by default yet, but that would be a question of just flipping
> a (context-triggered for now) switch.
> http://trac.cython.org/cython_trac/ticket/494
>
>> The post is from this originating thread, where Ryan Kelly proposed ways to
>> implement metaclasses:
>>
>> http://thread.gmane.org/gmane.comp.python.cython.user/1652
>>
>> I don't think metaclasses for cdef classes are ready to be implemented yet,
>> but it should be easier for Python classes.
>>
>> Also note that there is a new syntax for metaclasses in Py3, which I prefer
>> over the Py2 way. So we will eventually have to support both, although the
>> Py2 way is easy to emulate for Py3 code.
>>
>> Stefan
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From robertwb at math.washington.edu  Mon Nov  1 22:04:56 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 1 Nov 2010 14:04:56 -0700
Subject: [Cython] New cdef class directives: "final" and "internal"
In-Reply-To: <4CCD75A1.8050909@behnel.de>
References: <4CCC6F28.1040705@behnel.de>
	<AANLkTinq_z6=JZN7kU=VNPSXS_J7T8OMVsqq+u4bZHbC@mail.gmail.com>
	<AANLkTikGQ0a4sNzuCPNYPYcpGj8p0bp3BZ-_qZtbKT4r@mail.gmail.com>
	<4CCD0834.20801@behnel.de>
	<AANLkTikZhqONQ4DCE7SB9z-_OX45b61Na3zCA+9B0rzq@mail.gmail.com>
	<AANLkTinQ3sWbo6N=i+T6=96jquvHPOHRMck0fOExZL6j@mail.gmail.com>
	<4CCD75A1.8050909@behnel.de>
Message-ID: <AANLkTimmX5J0NJ5ZkJA0Ftim6sCdSX0aVmhZAiDdUqLd@mail.gmail.com>

On Sun, Oct 31, 2010 at 6:56 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Lisandro Dalcin, 31.10.2010 08:22:
>> On 31 October 2010 10:09, Robert Bradshaw wrote:
>>> What about using the
>>>
>>> [cdef] class A:
>>> ? ? ...
>>>
>>> del A
>>>
>>> idiom that people use for pure Python?
>>
>> I have to admit that this looks even simpler.
>
> Emphasis on "looks". I can see this easily working for Python classes, but
> what would be the semantics for cdef classes? Would it have any impact on
> their accessibility from Cython, and more specifically, from inside of the
> module that defines the type? What if a user externally overrides a Python
> class as cdef class in pure mode? And would it (or should it ever) be
> possible to redefine the type after deleting it? Should Cython prevent this
> for public/api classes, or would you just get the ImportError at runtime
> when it tries to read the type from the module dict?

Personally, I would not be opposed to putting everything into the
__pyx_api object rather than loading anything from the module dict.
That would be safer for one thing.

> Or would you just say "well, it's not really 'del' what's happening here,
> it's something that looks like Python, doesn't have an immediate impact,
> but removes the name from something that you can't see in your code"?
>
> The advantage of a decorator is that it's easy to get right, to read and
> comprehend. I'm not really opposed to using "del" here, but I'm not sure I
> can oversee all the implications of allowing it on cdef types.

I would say that "del" simply removes things from the Python
namespace. The c namespace is fixed at compile time, and would remain
so. This is essentially the case already, though perhaps we disallow
del for cdef classes as they live in both namespaces (don't know).

My main objection is that this decorator is yet another thing that
doesn't, and cannot, have an exact Python equivalent, and there's a
reasonable and widespread Python idiom to use in this case. We'd
probably want to support del on cdef classes for pure Python mode
anyways.

- Robert

From dagss at student.matnat.uio.no  Mon Nov  1 22:14:02 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 01 Nov 2010 22:14:02 +0100
Subject: [Cython] New cdef class directives: "final" and "internal"
In-Reply-To: <4CCD0834.20801@behnel.de>
References: <4CCC6F28.1040705@behnel.de>	<AANLkTinq_z6=JZN7kU=VNPSXS_J7T8OMVsqq+u4bZHbC@mail.gmail.com>	<AANLkTikGQ0a4sNzuCPNYPYcpGj8p0bp3BZ-_qZtbKT4r@mail.gmail.com>
	<4CCD0834.20801@behnel.de>
Message-ID: <4CCF2D9A.6010604@student.matnat.uio.no>

On 10/31/2010 07:09 AM, Stefan Behnel wrote:
> Robert Bradshaw, 31.10.2010 04:34:
>    
>> On Sat, Oct 30, 2010 at 2:12 PM, Lisandro Dalcin wrote:
>>      
>>> On 30 October 2010 22:16, Stefan Behnel wrote:
>>>        
>>>> a while ago, we had agreed that a "final" directive would be nice to have
>>>> for cdef classes. I implemented the basics for that (i.e. "final" types are
>>>> no longer subclassable from Python), and I also added an "internal"
>>>> directive that prevents a type from appearing in the module dict. So you
>>>> can now write something like this for a type that you only use internally
>>>> in your module:
>>>>
>>>>      @cython.final
>>>>      @cython.internal
>>>>      cdef class _MyInternalType:
>>>>         ...
>>>>
>>>> http://trac.cython.org/cython_trac/ticket/263
>>>> http://trac.cython.org/cython_trac/ticket/585
>>>>
>>>>          
>>> Many thanks for this, Stefan!
>>>        
> I'm as happy as you are.
>
>
>    
>>>> I think the "final" directive name was pretty much agreed on, but what
>>>> about the "internal" directive? Any objections to that?
>>>>          
>>> Perhaps "private" ? I still prefer "internal".
>>>        
>> I'd lean towards "private" but that's not a strong preference.
>>      
> I may be mistaken, but it looks like we don't actually use "private"
> anywhere in the language so far. I thought we did, but it currently seems
> to be a purely internal visibility marker. So to me, "internal" makes as
> much sense as "private".
>
> Robert, if you want me to change it, just say so clearly. ;)
>
>
>   >  I also
>   >  don't think such a feature is strictly necessary and complicates the
>   >  language, but wouldn't stand in its way if I'm the only one.
>
> I just wrote a patch for lxml.etree that adds one or both of the decorators
> where they make sense. (I won't apply it right away as the next release is
> hopefully closer than the next release of Cython, but I'll keep it around
> for one of the next releases after Cython 0.13.1.)
>
> It turns out that out of the 117 extension types in lxml.etree, 39 would be
> happy about at least one of the decorators: 30x final and 36x internal. So
> almost every third class is an internal one. Most of them are context
> classes that are passed through C in one way or another and that are used
> by internal callback code. Pure implementation details that no user should
> ever mess around with or even care about.
>    

Well, a lot of pure Python libraries have "pure implementation details 
that no user should ever mess around with or care about". The Python way 
is by convention (like a leading underscore telling people to just stay 
away), not by enforcement.

Is the rationale that it is more important to protect possibly 
non-segfaulting Cython code than always-safe Python code?

I guess I just don't see the usecase...prepend an underscore to 
module-internal class names, and that's it?

Regarding final, I see more of a usecase for individual methods than 
classes. I'm +0 for final on classes (Python doesn't have such a 
feature, why should we?), it is locking down individual methods (for 
possible speedups) that is important to me (in this case we can argue 
why Cython should have something Python does not).

Dag Sverre

From dagss at student.matnat.uio.no  Mon Nov  1 22:40:53 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 01 Nov 2010 22:40:53 +0100
Subject: [Cython] New cdef class directives: "final" and "internal"
In-Reply-To: <4CCF2D9A.6010604@student.matnat.uio.no>
References: <4CCC6F28.1040705@behnel.de>	<AANLkTinq_z6=JZN7kU=VNPSXS_J7T8OMVsqq+u4bZHbC@mail.gmail.com>	<AANLkTikGQ0a4sNzuCPNYPYcpGj8p0bp3BZ-_qZtbKT4r@mail.gmail.com>	<4CCD0834.20801@behnel.de>
	<4CCF2D9A.6010604@student.matnat.uio.no>
Message-ID: <4CCF33E5.4090506@student.matnat.uio.no>

On 11/01/2010 10:14 PM, Dag Sverre Seljebotn wrote:
> On 10/31/2010 07:09 AM, Stefan Behnel wrote:
>    
>> Robert Bradshaw, 31.10.2010 04:34:
>>
>>      
>>> On Sat, Oct 30, 2010 at 2:12 PM, Lisandro Dalcin wrote:
>>>
>>>        
>>>> On 30 October 2010 22:16, Stefan Behnel wrote:
>>>>
>>>>          
>>>>> a while ago, we had agreed that a "final" directive would be nice to have
>>>>> for cdef classes. I implemented the basics for that (i.e. "final" types are
>>>>> no longer subclassable from Python), and I also added an "internal"
>>>>> directive that prevents a type from appearing in the module dict. So you
>>>>> can now write something like this for a type that you only use internally
>>>>> in your module:
>>>>>
>>>>>       @cython.final
>>>>>       @cython.internal
>>>>>       cdef class _MyInternalType:
>>>>>          ...
>>>>>
>>>>> http://trac.cython.org/cython_trac/ticket/263
>>>>> http://trac.cython.org/cython_trac/ticket/585
>>>>>
>>>>>
>>>>>            
>>>> Many thanks for this, Stefan!
>>>>
>>>>          
>> I'm as happy as you are.
>>
>>
>>
>>      
>>>>> I think the "final" directive name was pretty much agreed on, but what
>>>>> about the "internal" directive? Any objections to that?
>>>>>
>>>>>            
>>>> Perhaps "private" ? I still prefer "internal".
>>>>
>>>>          
>>> I'd lean towards "private" but that's not a strong preference.
>>>
>>>        
>> I may be mistaken, but it looks like we don't actually use "private"
>> anywhere in the language so far. I thought we did, but it currently seems
>> to be a purely internal visibility marker. So to me, "internal" makes as
>> much sense as "private".
>>
>> Robert, if you want me to change it, just say so clearly. ;)
>>
>>
>>    >   I also
>>    >   don't think such a feature is strictly necessary and complicates the
>>    >   language, but wouldn't stand in its way if I'm the only one.
>>
>> I just wrote a patch for lxml.etree that adds one or both of the decorators
>> where they make sense. (I won't apply it right away as the next release is
>> hopefully closer than the next release of Cython, but I'll keep it around
>> for one of the next releases after Cython 0.13.1.)
>>
>> It turns out that out of the 117 extension types in lxml.etree, 39 would be
>> happy about at least one of the decorators: 30x final and 36x internal. So
>> almost every third class is an internal one. Most of them are context
>> classes that are passed through C in one way or another and that are used
>> by internal callback code. Pure implementation details that no user should
>> ever mess around with or even care about.
>>
>>      
> Well, a lot of pure Python libraries have "pure implementation details
> that no user should ever mess around with or care about". The Python way
> is by convention (like a leading underscore telling people to just stay
> away), not by enforcement.
>
> Is the rationale that it is more important to protect possibly
> non-segfaulting Cython code than always-safe Python code?
>
> I guess I just don't see the usecase...prepend an underscore to
> module-internal class names, and that's it?
>
> Regarding final, I see more of a usecase for individual methods than
> classes. I'm +0 for final on classes (Python doesn't have such a
> feature, why should we?), it is locking down individual methods (for
> possible speedups) that is important to me (in this case we can argue
> why Cython should have something Python does not).
>    

Although, as you know, there's no lack of concessions in the Cython 
language to "the numerics crowd". If Stefan claims this is important to 
his code and believes he can justify the additional complexity in the 
language, then I'm don't really care to stand in his way. It's not like 
@cython.internal will hurt me.

Dag Sverre

From robertwb at math.washington.edu  Tue Nov  2 07:17:26 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 1 Nov 2010 23:17:26 -0700
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>
Message-ID: <AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>

On Sun, Oct 31, 2010 at 8:44 PM, Brett Calcott <brett.calcott at gmail.com> wrote:
> Hi,
> I'm wrapping a C++ library (http://www.box2d.org/). A python version done
> with swig already exists, but I'm trying to make it more pythonic/cythonic,
> and preparing it for a future project. It's all going well, and I'm having a
> fun time -- cython rocks.

Thanks.

> I have a few questions and thoughts about wrapping. Consider wrapping
> something simple like this, where things are public:
> class Vector2d {
> public:
> ??float x, y
> ??Vector2d() : x(0.0), y(0.0) {}
> ??// other useful stuff here
> };
> I begin by doing this:
> extern from "vector2d.h":
> ??cdef cppclass Vector2d:
> ?? ? float x, y
> ?? ? Vector2d()
> Here are the questions:
> 1. Why must I use a *pointer* in my python class?
> cdef class PythonVector2d:
> ??cdef Vector2d *obj
> There is a default constructor, and I can happily declare it on the stack in
> a function. I'm sure I'm missing something here; is there some danger with
> allowing this?

It's conceivable that we could allow "stack allocated" members in an
object in the presence default constructors, but that would require
modifying the constructor and destructor to manually initialize and
deconstruct the object when the PyObject* is allocated and reclaimed.
Note that stack-allocated local variables are all initialized at the
top of a function and live until the function returns, unlike C++
where you can constrain their scope. Eventually, I'd like to do
something like references where it's all pointers under the hood, but
that's not obvious to the user, and Cython has the ability to perform
stack-allocations if possible.

> 2. How hard would it be to automatically forward member access to a
> contained cppclass? Consider the following nasty hack:
> cdef class PythonVector2d:
> ??cdef public float x, y
> ??cdef Vector2d *obj
> ??def __cinit__(self, x, y):
> ?? ?self.x = x
> ?? ?self.y = y
> ?? ?self.obj = <Vector 2d*>x
> ok. That is a bit nasty (I'm sure someone can tell me lots of reasons why I
> shouldn't do this).

In reading that code, I'm not sure what your intent is.

> But it really makes things easy. I don't have to declare
> all the properties independently. And when I need the cpp version, I just
> pass around pointer.
> Something like this would be nice:
> cdef class PythonVector2d:
> ??cdef Vector obj
> ??properties:
> ?? ? public obj.x as x
> ?? ? public obj.y as y
> Does this look crazy?

A better way to specify "simple" properties would be nice. Another
option would be

    property obj.x as x

or

    property x:
        obj.x

which would create a getter and setter automatically (if the body is
an expression). It would be nice for exposing this to Python. I'm not
a huge fan of doing the automatic-rewriting for C-level access, though
perhaps some eventual macro system could. Cython is not a
wrapper-generator (though it would make sense for a wrapper-generator
to emit Cython code).

- Robert

From stefan_ml at behnel.de  Tue Nov  2 08:32:24 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 02 Nov 2010 08:32:24 +0100
Subject: [Cython] New cdef class directives: "final" and "internal"
In-Reply-To: <AANLkTimmX5J0NJ5ZkJA0Ftim6sCdSX0aVmhZAiDdUqLd@mail.gmail.com>
References: <4CCC6F28.1040705@behnel.de>	<AANLkTinq_z6=JZN7kU=VNPSXS_J7T8OMVsqq+u4bZHbC@mail.gmail.com>	<AANLkTikGQ0a4sNzuCPNYPYcpGj8p0bp3BZ-_qZtbKT4r@mail.gmail.com>	<4CCD0834.20801@behnel.de>	<AANLkTikZhqONQ4DCE7SB9z-_OX45b61Na3zCA+9B0rzq@mail.gmail.com>	<AANLkTinQ3sWbo6N=i+T6=96jquvHPOHRMck0fOExZL6j@mail.gmail.com>	<4CCD75A1.8050909@behnel.de>
	<AANLkTimmX5J0NJ5ZkJA0Ftim6sCdSX0aVmhZAiDdUqLd@mail.gmail.com>
Message-ID: <4CCFBE88.8070502@behnel.de>

Robert Bradshaw, 01.11.2010 22:04:
> On Sun, Oct 31, 2010 at 6:56 AM, Stefan Behnel wrote:
>>> On 31 October 2010 10:09, Robert Bradshaw wrote:
>>>> [cdef] class A:
>>>>      ...
>>>>
>>>> del A
>>
>> Should Cython prevent this
>> for public/api classes, or would you just get the ImportError at runtime
>> when it tries to read the type from the module dict?
>
> Personally, I would not be opposed to putting everything into the
> __pyx_api object rather than loading anything from the module dict.
> That would be safer for one thing.

Sounds better, yes. And with an additional fallback to reading classes from 
the module dict, it would even be backwards compatible.

I think that would be part of the public/api unification that we considered 
a while ago.


> I would say that "del" simply removes things from the Python
> namespace. The c namespace is fixed at compile time, and would remain
> so. This is essentially the case already, though perhaps we disallow
> del for cdef classes as they live in both namespaces (don't know).

See? That's the kind of "don't know" that I meant. I'm not sure either. 
When I see "del TheClass" in a module, do I have to know what kind of class 
this is in order to know if I can continue to refer to it inside the module?


> We'd probably want to support del on cdef classes for pure Python mode
> anyways.

It wouldn't work the same way, though. If you "del" a cdef class in pure 
mode running under Python, its name will disappear. If you "del" it from 
within Cython, it will continue to be accessible under that name and likely 
wouldn't support redefinition either.


> My main objection is that this decorator is yet another thing that
> doesn't, and cannot, have an exact Python equivalent, and there's a
> reasonable and widespread Python idiom to use in this case.

I agree. I'm just saying that it is a well defined and useful feature 
within Cython. Maybe we could really only allow it within Cython modules, 
i.e. use a cdef class modifier instead of a decorator. That would prevent 
its ambiguous use from pure mode and we could continue to disallow "del" on 
cdef classes as that's also ambiguous in pure more (and also mostly useless 
as it doesn't free any resources nor allow redefinition).

Stefan

From stefan_ml at behnel.de  Tue Nov  2 09:19:52 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 02 Nov 2010 09:19:52 +0100
Subject: [Cython] New cdef class directives: "final" and "internal"
In-Reply-To: <4CCF2D9A.6010604@student.matnat.uio.no>
References: <4CCC6F28.1040705@behnel.de>	<AANLkTinq_z6=JZN7kU=VNPSXS_J7T8OMVsqq+u4bZHbC@mail.gmail.com>	<AANLkTikGQ0a4sNzuCPNYPYcpGj8p0bp3BZ-_qZtbKT4r@mail.gmail.com>	<4CCD0834.20801@behnel.de>
	<4CCF2D9A.6010604@student.matnat.uio.no>
Message-ID: <4CCFC9A8.60208@behnel.de>

Dag Sverre Seljebotn, 01.11.2010 22:14:
> On 10/31/2010 07:09 AM, Stefan Behnel wrote:
>> I just wrote a patch for lxml.etree that adds one or both of the decorators
>> where they make sense.
>>
>> It turns out that out of the 117 extension types in lxml.etree, 39 would be
>> happy about at least one of the decorators: 30x final and 36x internal. So
>> almost every third class is an internal one. Most of them are context
>> classes that are passed through C in one way or another and that are used
>> by internal callback code. Pure implementation details that no user should
>> ever mess around with or even care about.
>
> Well, a lot of pure Python libraries have "pure implementation details
> that no user should ever mess around with or care about". The Python way
> is by convention (like a leading underscore telling people to just stay
> away), not by enforcement.
>
> Is the rationale that it is more important to protect possibly
> non-segfaulting Cython code than always-safe Python code?

Yes, I think that's a major issue here. In many cases, it's hard to prevent 
a segfault when users manually instantiate or subclass such a class. Plus, 
it fills up the user visible module dict with internal stuff because you 
can't just "del" a cdef class after use. So you get a huge "help(module)" 
page full of classes out of which one third is useless information.

I'm currently considering ways to make lxml ready for environments like 
Google's App Engine, where it is important to prevent segfaults. Letting 
users instantiate and override arbitrary classes has proven to be an 
unlocked barn door for crashes in the past. It's a lot better now because I 
actually locked down a lot of classes and their methods. But being able to 
move them out of the way entirely would allow me to really concentrate on 
what users are meant to work with.

(I just checked http://dict.leo.org for a good translation of what this 
feels like and didn't find one. But looking up a related word made me find 
"proxy war", which, I think, nicely matches lxml's use of proxy classes ;)


> I guess I just don't see the usecase...prepend an underscore to
> module-internal class names, and that's it?

Most cdef classes in lxml.etree are prefixed with an underscore. That 
doesn't prevent users from instantiating them and filing crash bug reports.


> Regarding final, I see more of a usecase for individual methods than
> classes. I'm +0 for final on classes (Python doesn't have such a
> feature, why should we?)

CPython does. You cannot subclass "bool" and "NoneType". Some classes are 
simply not meant for subclassing, especially proxy classes that map C level 
data structures into Python space.


> it is locking down individual methods (for
> possible speedups) that is important to me (in this case we can argue
> why Cython should have something Python does not).

You may notice that I've written all existing feature requests regarding 
"final". Final methods would really make the language easier to use as you 
could write nicer OO code without sacrificing performance. lxml.etree has a 
lot of C functions that would look much better as methods.

I just didn't want to implement it just now because it's a lot more work 
than the "final" feature on entire classes (which, BTW, really is a feature 
that CPython does not support). I think it would make sense to get the 
Python function refactoring (Python wrapper + C function) done before that. 
But that will take a bit of time as well.

Stefan

From stefan_ml at behnel.de  Tue Nov  2 09:49:39 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 02 Nov 2010 09:49:39 +0100
Subject: [Cython] auto __test__ dict
In-Reply-To: <AANLkTikBoY5pFx=AO=G7iVPmLqnrhwzjhZ1hTOA=SRO5@mail.gmail.com>
References: <4CCC6F28.1040705@behnel.de>
	<4CCD1A7C.3010805@behnel.de>	<4CCDB7C3.20208@behnel.de>
	<AANLkTikBoY5pFx=AO=G7iVPmLqnrhwzjhZ1hTOA=SRO5@mail.gmail.com>
Message-ID: <4CCFD0A3.8010503@behnel.de>

Robert Bradshaw, 01.11.2010 21:44:
> On Sun, Oct 31, 2010 at 11:38 AM, Stefan Behnel wrote:
>>> One problem I spotted was the auto test dict (again).
>>
>> I simplified it now to use the compile time docstrings directly, instead of
>> looking them up at runtime. I don't think this breaks anything important,
>> as the docstrings of functions and methods cannot be modified anyway. On
>> the plus side, it removes a *ton* of ugly C code, and also saves some time
>> during module initialisation.
>
> Excellent! I've been wanting to do this myself, eventually when I
> found the time. This means that cdef methods could have tested
> docstrings as well (though, of course, they couldn't be called
> directly).

That was easy to implement, so I tried it. In lxml.etree, it adds some 10KB 
to the size of the module source code as the docstrings of cdef functions 
wouldn't otherwise end up there. Personally, I'm not sure it's worth it. If 
you can't call a cdef function/method from a doctest directly, why not just 
move the doctest itself to the function or method that you would have to 
call anyway?

Stefan

From stefan_ml at behnel.de  Tue Nov  2 10:02:26 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 02 Nov 2010 10:02:26 +0100
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>
	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>
Message-ID: <4CCFD3A2.7090301@behnel.de>

Robert Bradshaw, 02.11.2010 07:17:
> On Sun, Oct 31, 2010 at 8:44 PM, Brett Calcott wrote:
>> But it really makes things easy. I don't have to declare
>> all the properties independently. And when I need the cpp version, I just
>> pass around pointer.
>> Something like this would be nice:
>> cdef class PythonVector2d:
>>    cdef Vector obj
>>    properties:
>>       public obj.x as x
>>       public obj.y as y
>> Does this look crazy?
>
> A better way to specify "simple" properties would be nice. Another
> option would be
>
>      property obj.x as x
>
> or
>
>      property x:
>          obj.x
>
> which would create a getter and setter automatically (if the body is
> an expression).

For this case, I'd prefer a version entirely without body, maybe even

     property x = obj.x

although the "as" version would also work. However, it's not so clear from 
the above that both setter and getter get generated, and how you would get 
only the getter for read-only access, for example. And why not have a 
"delattr" as well? May make sense in some cases.

Personally, I think that the current way of coding this isn't all that 
cumbersome but a lot more explicit. And it makes you think a bit more about 
the implications of mapping a C++ class field directly to a Python property 
without value checking etc.

Stefan

From vitja.makarov at gmail.com  Tue Nov  2 10:36:36 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 2 Nov 2010 12:36:36 +0300
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTikn4VJtcJ3EsESd4iVKNpg42fQK2EAdbAbuBA1o@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CCE989D.6070907@behnel.de>
	<AANLkTin+ofqt6-z+-Z-bnuSmW9eSDL94xDs=D=dT3ic7@mail.gmail.com>
	<AANLkTikn4VJtcJ3EsESd4iVKNpg42fQK2EAdbAbuBA1o@mail.gmail.com>
Message-ID: <AANLkTinPTHtp=X483QUGsRO58tm2TTokSc02LxaEt3fP@mail.gmail.com>

I tried to implement class dict  filling, patch is attached.
Could you take a look at the patch and tell me am I going right way?


Here is example of metaclass stuff that works, __new__ is decorated
with staticmethod as type() does not turn PyMethods into staticmethods
it works only with PyFunction so PyCFunction will not work too.
__new__ should be converted into staticmethod in Cython


class Base(type):
    @staticmethod
    def __new__(cls, name, bases, attrs):
        print 'Creating class'
        return type.__new__(cls, name, bases, attrs)

class Foo(object):
    __metaclass__ = Base


ps:

with my patch I got:
failures=29, errors=4
-------------- next part --------------
A non-text attachment was scrubbed...
Name: metaclass.diff
Type: text/x-patch
Size: 6838 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20101102/a1735524/attachment.bin 

From vitja.makarov at gmail.com  Tue Nov  2 10:48:18 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 2 Nov 2010 12:48:18 +0300
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTinPTHtp=X483QUGsRO58tm2TTokSc02LxaEt3fP@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CCE989D.6070907@behnel.de>
	<AANLkTin+ofqt6-z+-Z-bnuSmW9eSDL94xDs=D=dT3ic7@mail.gmail.com>
	<AANLkTikn4VJtcJ3EsESd4iVKNpg42fQK2EAdbAbuBA1o@mail.gmail.com>
	<AANLkTinPTHtp=X483QUGsRO58tm2TTokSc02LxaEt3fP@mail.gmail.com>
Message-ID: <AANLkTikESy8G5XWBi=8cvmO0nCJgQg_A9dQ=znrz-XuN@mail.gmail.com>

2010/11/2 Vitja Makarov <vitja.makarov at gmail.com>:
> ps:
>
> with my patch I got:
> failures=29, errors=4
>

First I tried to create method with NULL
PyMethod_New(__pyx_t_2, 0, NULL);

But that doesn't worked for @staticmethod that gives SIGSEGV so I turned it into
PyMethod_New(__pyx_t_2, 0, (PyObject *) &PyBaseObject_Type);

But that doesn't work for simple classes, only for object based...

From dalcinl at gmail.com  Tue Nov  2 12:17:51 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 2 Nov 2010 14:17:51 +0300
Subject: [Cython] auto __test__ dict
In-Reply-To: <4CCFD0A3.8010503@behnel.de>
References: <4CCC6F28.1040705@behnel.de> <4CCD1A7C.3010805@behnel.de>
	<4CCDB7C3.20208@behnel.de>
	<AANLkTikBoY5pFx=AO=G7iVPmLqnrhwzjhZ1hTOA=SRO5@mail.gmail.com>
	<4CCFD0A3.8010503@behnel.de>
Message-ID: <AANLkTikzsMY36n-Cmyh9uJ82a7MRQcF-66vdLtK-id3H@mail.gmail.com>

On 2 November 2010 11:49, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 01.11.2010 21:44:
>> On Sun, Oct 31, 2010 at 11:38 AM, Stefan Behnel wrote:
>>>> One problem I spotted was the auto test dict (again).
>>>
>>> I simplified it now to use the compile time docstrings directly, instead of
>>> looking them up at runtime. I don't think this breaks anything important,
>>> as the docstrings of functions and methods cannot be modified anyway. On
>>> the plus side, it removes a *ton* of ugly C code, and also saves some time
>>> during module initialisation.
>>
>> Excellent! I've been wanting to do this myself, eventually when I
>> found the time. This means that cdef methods could have tested
>> docstrings as well (though, of course, they couldn't be called
>> directly).
>
> That was easy to implement, so I tried it. In lxml.etree, it adds some 10KB
> to the size of the module source code as the docstrings of cdef functions
> wouldn't otherwise end up there. Personally, I'm not sure it's worth it. If
> you can't call a cdef function/method from a doctest directly, why not just
> move the doctest itself to the function or method that you would have to
> call anyway?
>

I agree with Stefan.


-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From robertwb at math.washington.edu  Tue Nov  2 18:53:07 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 2 Nov 2010 10:53:07 -0700
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTikESy8G5XWBi=8cvmO0nCJgQg_A9dQ=znrz-XuN@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CCE989D.6070907@behnel.de>
	<AANLkTin+ofqt6-z+-Z-bnuSmW9eSDL94xDs=D=dT3ic7@mail.gmail.com>
	<AANLkTikn4VJtcJ3EsESd4iVKNpg42fQK2EAdbAbuBA1o@mail.gmail.com>
	<AANLkTinPTHtp=X483QUGsRO58tm2TTokSc02LxaEt3fP@mail.gmail.com>
	<AANLkTikESy8G5XWBi=8cvmO0nCJgQg_A9dQ=znrz-XuN@mail.gmail.com>
Message-ID: <AANLkTino6OWKMbgG41Wvn1vo-jNP=KRvnX2H9ai31rv-@mail.gmail.com>

On Tue, Nov 2, 2010 at 2:48 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2010/11/2 Vitja Makarov <vitja.makarov at gmail.com>:
>> ps:
>>
>> with my patch I got:
>> failures=29, errors=4
>>
>
> First I tried to create method with NULL
> PyMethod_New(__pyx_t_2, 0, NULL);
>
> But that doesn't worked for @staticmethod that gives SIGSEGV so I turned it into
> PyMethod_New(__pyx_t_2, 0, (PyObject *) &PyBaseObject_Type);
>
> But that doesn't work for simple classes, only for object based...

This is the issue that Greg Ewing was mentioning--you can't make a
method until you have a class. You'll have to make (binding) function
objects, which get turned into methods at class creation time.

- Robert

From vitja.makarov at gmail.com  Tue Nov  2 22:20:25 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Wed, 3 Nov 2010 00:20:25 +0300
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTino6OWKMbgG41Wvn1vo-jNP=KRvnX2H9ai31rv-@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CCE989D.6070907@behnel.de>
	<AANLkTin+ofqt6-z+-Z-bnuSmW9eSDL94xDs=D=dT3ic7@mail.gmail.com>
	<AANLkTikn4VJtcJ3EsESd4iVKNpg42fQK2EAdbAbuBA1o@mail.gmail.com>
	<AANLkTinPTHtp=X483QUGsRO58tm2TTokSc02LxaEt3fP@mail.gmail.com>
	<AANLkTikESy8G5XWBi=8cvmO0nCJgQg_A9dQ=znrz-XuN@mail.gmail.com>
	<AANLkTino6OWKMbgG41Wvn1vo-jNP=KRvnX2H9ai31rv-@mail.gmail.com>
Message-ID: <AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>

It worked!

I've enabled binding and used ExprNodes.PyCFunctionNode instead of
ExprNodes.PyCFunctionNode in DefNode.synthesize_assignment_node().

Btw __new__ is still decorated with @staticmethod...
To solve this PyCFunctionNode should be used for __new__ and probably
for all static methods?

2010/11/2 Robert Bradshaw <robertwb at math.washington.edu>:
> On Tue, Nov 2, 2010 at 2:48 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> 2010/11/2 Vitja Makarov <vitja.makarov at gmail.com>:
>>> ps:
>>>
>>> with my patch I got:
>>> failures=29, errors=4
>>>
>>
>> First I tried to create method with NULL
>> PyMethod_New(__pyx_t_2, 0, NULL);
>>
>> But that doesn't worked for @staticmethod that gives SIGSEGV so I turned it into
>> PyMethod_New(__pyx_t_2, 0, (PyObject *) &PyBaseObject_Type);
>>
>> But that doesn't work for simple classes, only for object based...
>
> This is the issue that Greg Ewing was mentioning--you can't make a
> method until you have a class. You'll have to make (binding) function
> objects, which get turned into methods at class creation time.
>
> - Robert
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From robertwb at math.washington.edu  Tue Nov  2 22:46:53 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 2 Nov 2010 14:46:53 -0700
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CCE989D.6070907@behnel.de>
	<AANLkTin+ofqt6-z+-Z-bnuSmW9eSDL94xDs=D=dT3ic7@mail.gmail.com>
	<AANLkTikn4VJtcJ3EsESd4iVKNpg42fQK2EAdbAbuBA1o@mail.gmail.com>
	<AANLkTinPTHtp=X483QUGsRO58tm2TTokSc02LxaEt3fP@mail.gmail.com>
	<AANLkTikESy8G5XWBi=8cvmO0nCJgQg_A9dQ=znrz-XuN@mail.gmail.com>
	<AANLkTino6OWKMbgG41Wvn1vo-jNP=KRvnX2H9ai31rv-@mail.gmail.com>
	<AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>
Message-ID: <AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>

On Tue, Nov 2, 2010 at 2:20 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> It worked!

Excellent! Do you have a patch?

> I've enabled binding and used ExprNodes.PyCFunctionNode instead of
> ExprNodes.PyCFunctionNode in DefNode.synthesize_assignment_node().
>
> Btw __new__ is still decorated with @staticmethod...
> To solve this PyCFunctionNode should be used for __new__ and probably
> for all static methods?

__new__ should be a class method, not a static method, right? This
trick should work fine for static methods. Also, I should note that
__new__ will require a bit more work to implement fully.

> 2010/11/2 Robert Bradshaw <robertwb at math.washington.edu>:
>> On Tue, Nov 2, 2010 at 2:48 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>> 2010/11/2 Vitja Makarov <vitja.makarov at gmail.com>:
>>>> ps:
>>>>
>>>> with my patch I got:
>>>> failures=29, errors=4
>>>>
>>>
>>> First I tried to create method with NULL
>>> PyMethod_New(__pyx_t_2, 0, NULL);
>>>
>>> But that doesn't worked for @staticmethod that gives SIGSEGV so I turned it into
>>> PyMethod_New(__pyx_t_2, 0, (PyObject *) &PyBaseObject_Type);
>>>
>>> But that doesn't work for simple classes, only for object based...
>>
>> This is the issue that Greg Ewing was mentioning--you can't make a
>> method until you have a class. You'll have to make (binding) function
>> objects, which get turned into methods at class creation time.
>>
>> - Robert
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From vitja.makarov at gmail.com  Tue Nov  2 23:09:19 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Wed, 3 Nov 2010 01:09:19 +0300
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CCE989D.6070907@behnel.de>
	<AANLkTin+ofqt6-z+-Z-bnuSmW9eSDL94xDs=D=dT3ic7@mail.gmail.com>
	<AANLkTikn4VJtcJ3EsESd4iVKNpg42fQK2EAdbAbuBA1o@mail.gmail.com>
	<AANLkTinPTHtp=X483QUGsRO58tm2TTokSc02LxaEt3fP@mail.gmail.com>
	<AANLkTikESy8G5XWBi=8cvmO0nCJgQg_A9dQ=znrz-XuN@mail.gmail.com>
	<AANLkTino6OWKMbgG41Wvn1vo-jNP=KRvnX2H9ai31rv-@mail.gmail.com>
	<AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>
	<AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>
Message-ID: <AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>

Hmm... yes patch is here... I run fails on classmethod tests ;( don't
know how to solve this yet but all others are okay.

This patch also includes some tests for __metaclass__ and one more
test for staticmethod

2010/11/3 Robert Bradshaw <robertwb at math.washington.edu>:
> On Tue, Nov 2, 2010 at 2:20 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> It worked!
>
> Excellent! Do you have a patch?
>
>> I've enabled binding and used ExprNodes.PyCFunctionNode instead of
>> ExprNodes.PyCFunctionNode in DefNode.synthesize_assignment_node().
>>
>> Btw __new__ is still decorated with @staticmethod...
>> To solve this PyCFunctionNode should be used for __new__ and probably
>> for all static methods?
>
> __new__ should be a class method, not a static method, right? This
> trick should work fine for static methods. Also, I should note that
> __new__ will require a bit more work to implement fully.
>
>> 2010/11/2 Robert Bradshaw <robertwb at math.washington.edu>:
>>> On Tue, Nov 2, 2010 at 2:48 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>> 2010/11/2 Vitja Makarov <vitja.makarov at gmail.com>:
>>>>> ps:
>>>>>
>>>>> with my patch I got:
>>>>> failures=29, errors=4
>>>>>
>>>>
>>>> First I tried to create method with NULL
>>>> PyMethod_New(__pyx_t_2, 0, NULL);
>>>>
>>>> But that doesn't worked for @staticmethod that gives SIGSEGV so I turned it into
>>>> PyMethod_New(__pyx_t_2, 0, (PyObject *) &PyBaseObject_Type);
>>>>
>>>> But that doesn't work for simple classes, only for object based...
>>>
>>> This is the issue that Greg Ewing was mentioning--you can't make a
>>> method until you have a class. You'll have to make (binding) function
>>> objects, which get turned into methods at class creation time.
>>>
>>> - Robert
>>> _______________________________________________
>>> Cython-dev mailing list
>>> Cython-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/cython-dev
>>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: metaclass.diff
Type: text/x-patch
Size: 7844 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20101103/0874bcb5/attachment.bin 

From vitja.makarov at gmail.com  Tue Nov  2 23:11:54 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Wed, 3 Nov 2010 01:11:54 +0300
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CCE989D.6070907@behnel.de>
	<AANLkTin+ofqt6-z+-Z-bnuSmW9eSDL94xDs=D=dT3ic7@mail.gmail.com>
	<AANLkTikn4VJtcJ3EsESd4iVKNpg42fQK2EAdbAbuBA1o@mail.gmail.com>
	<AANLkTinPTHtp=X483QUGsRO58tm2TTokSc02LxaEt3fP@mail.gmail.com>
	<AANLkTikESy8G5XWBi=8cvmO0nCJgQg_A9dQ=znrz-XuN@mail.gmail.com>
	<AANLkTino6OWKMbgG41Wvn1vo-jNP=KRvnX2H9ai31rv-@mail.gmail.com>
	<AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>
	<AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>
	<AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>
Message-ID: <AANLkTikSi2Awwv31sx3A6BJy=WNSYROo+ceecoygQ0Fk@mail.gmail.com>

2010/11/3 Vitja Makarov <vitja.makarov at gmail.com>:
> Hmm... yes patch is here... I run fails on classmethod tests ;( don't
> know how to solve this yet but all others are okay.
>
> This patch also includes some tests for __metaclass__ and one more
> test for staticmethod
>
> 2010/11/3 Robert Bradshaw <robertwb at math.washington.edu>:
>> On Tue, Nov 2, 2010 at 2:20 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>> It worked!
>>
>> Excellent! Do you have a patch?
>>
>>> I've enabled binding and used ExprNodes.PyCFunctionNode instead of
>>> ExprNodes.PyCFunctionNode in DefNode.synthesize_assignment_node().
>>>
>>> Btw __new__ is still decorated with @staticmethod...
>>> To solve this PyCFunctionNode should be used for __new__ and probably
>>> for all static methods?
>>
>> __new__ should be a class method, not a static method, right? This
>> trick should work fine for static methods. Also, I should note that
>> __new__ will require a bit more work to implement fully.
>>

http://docs.python.org/reference/datamodel.html#basic-customization

"""Called to create a new instance of class cls. __new__() is a static
method (special-cased so you need not declare it as such) that takes
the class of which an instance was requested as its first argument.
The remaining arguments are those passed to the object constructor
expression (the call to the class). The return value of __new__()
should be the new object instance (usually an instance of cls)."""

Btw classmethods are now broken :(

From robertwb at math.washington.edu  Tue Nov  2 23:23:50 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 2 Nov 2010 15:23:50 -0700
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CCE989D.6070907@behnel.de>
	<AANLkTin+ofqt6-z+-Z-bnuSmW9eSDL94xDs=D=dT3ic7@mail.gmail.com>
	<AANLkTikn4VJtcJ3EsESd4iVKNpg42fQK2EAdbAbuBA1o@mail.gmail.com>
	<AANLkTinPTHtp=X483QUGsRO58tm2TTokSc02LxaEt3fP@mail.gmail.com>
	<AANLkTikESy8G5XWBi=8cvmO0nCJgQg_A9dQ=znrz-XuN@mail.gmail.com>
	<AANLkTino6OWKMbgG41Wvn1vo-jNP=KRvnX2H9ai31rv-@mail.gmail.com>
	<AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>
	<AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>
	<AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>
Message-ID: <AANLkTi=_qmWeP1A9wHVWRp2vDTGrOrVfWxoMvHEMQ4vL@mail.gmail.com>

On Tue, Nov 2, 2010 at 3:09 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> Hmm... yes patch is here... I run fails on classmethod tests ;( don't
> know how to solve this yet but all others are okay.

Thanks! Did you want to post it to trac? It's preferable to use "hg
export" so we get some metadata attached to the patch. (You can turn
your Cython into a working directory by doing "make repo".)

> This patch also includes some tests for __metaclass__ and one more
> test for staticmethod

I'll try to find some time to look at the staticmethod issue, but
won't complain if someone beats me to it :).

- Robert

From robertwb at math.washington.edu  Tue Nov  2 23:21:38 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 2 Nov 2010 15:21:38 -0700
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTikSi2Awwv31sx3A6BJy=WNSYROo+ceecoygQ0Fk@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CCE989D.6070907@behnel.de>
	<AANLkTin+ofqt6-z+-Z-bnuSmW9eSDL94xDs=D=dT3ic7@mail.gmail.com>
	<AANLkTikn4VJtcJ3EsESd4iVKNpg42fQK2EAdbAbuBA1o@mail.gmail.com>
	<AANLkTinPTHtp=X483QUGsRO58tm2TTokSc02LxaEt3fP@mail.gmail.com>
	<AANLkTikESy8G5XWBi=8cvmO0nCJgQg_A9dQ=znrz-XuN@mail.gmail.com>
	<AANLkTino6OWKMbgG41Wvn1vo-jNP=KRvnX2H9ai31rv-@mail.gmail.com>
	<AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>
	<AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>
	<AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>
	<AANLkTikSi2Awwv31sx3A6BJy=WNSYROo+ceecoygQ0Fk@mail.gmail.com>
Message-ID: <AANLkTi=bFQwdofb4TPcV53b9w0RtcshvC8Uiv2mJa9i-@mail.gmail.com>

On Tue, Nov 2, 2010 at 3:11 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2010/11/3 Vitja Makarov <vitja.makarov at gmail.com>:
>> Hmm... yes patch is here... I run fails on classmethod tests ;( don't
>> know how to solve this yet but all others are okay.
>>
>> This patch also includes some tests for __metaclass__ and one more
>> test for staticmethod
>>
>> 2010/11/3 Robert Bradshaw <robertwb at math.washington.edu>:
>>> On Tue, Nov 2, 2010 at 2:20 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>> It worked!
>>>
>>> Excellent! Do you have a patch?
>>>
>>>> I've enabled binding and used ExprNodes.PyCFunctionNode instead of
>>>> ExprNodes.PyCFunctionNode in DefNode.synthesize_assignment_node().
>>>>
>>>> Btw __new__ is still decorated with @staticmethod...
>>>> To solve this PyCFunctionNode should be used for __new__ and probably
>>>> for all static methods?
>>>
>>> __new__ should be a class method, not a static method, right? This
>>> trick should work fine for static methods. Also, I should note that
>>> __new__ will require a bit more work to implement fully.
>>>
>
> http://docs.python.org/reference/datamodel.html#basic-customization
>
> """Called to create a new instance of class cls. __new__() is a static
> method (special-cased so you need not declare it as such) that takes
> the class of which an instance was requested as its first argument.
> The remaining arguments are those passed to the object constructor
> expression (the call to the class). The return value of __new__()
> should be the new object instance (usually an instance of cls)."""

I was actually thinking about cdef classes, where __new__ is much more
subtle. I didn't realize it was a static method that takes the class
as a first argument rather than just a class method.

> Btw classmethods are now broken :(

Well, the fix will probably involve cleaning up some hacky
workarounds, which will be good.

- Robert

From vitja.makarov at gmail.com  Tue Nov  2 23:29:53 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Wed, 3 Nov 2010 01:29:53 +0300
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTi=_qmWeP1A9wHVWRp2vDTGrOrVfWxoMvHEMQ4vL@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CCE989D.6070907@behnel.de>
	<AANLkTin+ofqt6-z+-Z-bnuSmW9eSDL94xDs=D=dT3ic7@mail.gmail.com>
	<AANLkTikn4VJtcJ3EsESd4iVKNpg42fQK2EAdbAbuBA1o@mail.gmail.com>
	<AANLkTinPTHtp=X483QUGsRO58tm2TTokSc02LxaEt3fP@mail.gmail.com>
	<AANLkTikESy8G5XWBi=8cvmO0nCJgQg_A9dQ=znrz-XuN@mail.gmail.com>
	<AANLkTino6OWKMbgG41Wvn1vo-jNP=KRvnX2H9ai31rv-@mail.gmail.com>
	<AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>
	<AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>
	<AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>
	<AANLkTi=_qmWeP1A9wHVWRp2vDTGrOrVfWxoMvHEMQ4vL@mail.gmail.com>
Message-ID: <AANLkTinRbxta5dRgciK+TjH2qGuz2pm8yZkao58FF4cR@mail.gmail.com>

2010/11/3 Robert Bradshaw <robertwb at math.washington.edu>:
> On Tue, Nov 2, 2010 at 3:09 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> Hmm... yes patch is here... I run fails on classmethod tests ;( don't
>> know how to solve this yet but all others are okay.
>
> Thanks! Did you want to post it to trac? It's preferable to use "hg
> export" so we get some metadata attached to the patch. (You can turn
> your Cython into a working directory by doing "make repo".)
>

Ok I'll try... I'm not very familar with hg so I have to do commit then export?


>> This patch also includes some tests for __metaclass__ and one more
>> test for staticmethod
>
> I'll try to find some time to look at the staticmethod issue, but
> won't complain if someone beats me to it :).
>

Just one more case when staticmethod is decorator, now it works)

class class4:
    @staticmethod
    def plus1(a):
        return a + 1



> - Robert
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From robertwb at math.washington.edu  Wed Nov  3 00:11:41 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 2 Nov 2010 16:11:41 -0700
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTinRbxta5dRgciK+TjH2qGuz2pm8yZkao58FF4cR@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CCE989D.6070907@behnel.de>
	<AANLkTin+ofqt6-z+-Z-bnuSmW9eSDL94xDs=D=dT3ic7@mail.gmail.com>
	<AANLkTikn4VJtcJ3EsESd4iVKNpg42fQK2EAdbAbuBA1o@mail.gmail.com>
	<AANLkTinPTHtp=X483QUGsRO58tm2TTokSc02LxaEt3fP@mail.gmail.com>
	<AANLkTikESy8G5XWBi=8cvmO0nCJgQg_A9dQ=znrz-XuN@mail.gmail.com>
	<AANLkTino6OWKMbgG41Wvn1vo-jNP=KRvnX2H9ai31rv-@mail.gmail.com>
	<AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>
	<AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>
	<AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>
	<AANLkTi=_qmWeP1A9wHVWRp2vDTGrOrVfWxoMvHEMQ4vL@mail.gmail.com>
	<AANLkTinRbxta5dRgciK+TjH2qGuz2pm8yZkao58FF4cR@mail.gmail.com>
Message-ID: <AANLkTin8bserTjoVXhADG9xjgZzMdOafav4Eb75LyAG4@mail.gmail.com>

On Tue, Nov 2, 2010 at 3:29 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2010/11/3 Robert Bradshaw <robertwb at math.washington.edu>:
>> On Tue, Nov 2, 2010 at 3:09 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>> Hmm... yes patch is here... I run fails on classmethod tests ;( don't
>>> know how to solve this yet but all others are okay.
>>
>> Thanks! Did you want to post it to trac? It's preferable to use "hg
>> export" so we get some metadata attached to the patch. (You can turn
>> your Cython into a working directory by doing "make repo".)
>>
>
> Ok I'll try... I'm not very familar with hg so I have to do commit then export?

Yes, exactly. You might also want to set up a ~/.hgrc file with

[ui]
username = name <email>

for the commit message (otherwise it just uses your logged-in username).

>>> This patch also includes some tests for __metaclass__ and one more
>>> test for staticmethod
>>
>> I'll try to find some time to look at the staticmethod issue, but
>> won't complain if someone beats me to it :).
>>
>
> Just one more case when staticmethod is decorator, now it works)
>
> class class4:
> ? ?@staticmethod
> ? ?def plus1(a):
> ? ? ? ?return a + 1

Good.

- Robert

From vitja.makarov at gmail.com  Wed Nov  3 00:17:42 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Wed, 3 Nov 2010 02:17:42 +0300
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTin8bserTjoVXhADG9xjgZzMdOafav4Eb75LyAG4@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CCE989D.6070907@behnel.de>
	<AANLkTin+ofqt6-z+-Z-bnuSmW9eSDL94xDs=D=dT3ic7@mail.gmail.com>
	<AANLkTikn4VJtcJ3EsESd4iVKNpg42fQK2EAdbAbuBA1o@mail.gmail.com>
	<AANLkTinPTHtp=X483QUGsRO58tm2TTokSc02LxaEt3fP@mail.gmail.com>
	<AANLkTikESy8G5XWBi=8cvmO0nCJgQg_A9dQ=znrz-XuN@mail.gmail.com>
	<AANLkTino6OWKMbgG41Wvn1vo-jNP=KRvnX2H9ai31rv-@mail.gmail.com>
	<AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>
	<AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>
	<AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>
	<AANLkTi=_qmWeP1A9wHVWRp2vDTGrOrVfWxoMvHEMQ4vL@mail.gmail.com>
	<AANLkTinRbxta5dRgciK+TjH2qGuz2pm8yZkao58FF4cR@mail.gmail.com>
	<AANLkTin8bserTjoVXhADG9xjgZzMdOafav4Eb75LyAG4@mail.gmail.com>
Message-ID: <AANLkTinqQd9MhZ57WiS1gdGBWOs5A6hbGQvmrLTUF3F4@mail.gmail.com>

I've attached patch, thank you :)

2010/11/3 Robert Bradshaw <robertwb at math.washington.edu>:
> On Tue, Nov 2, 2010 at 3:29 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> 2010/11/3 Robert Bradshaw <robertwb at math.washington.edu>:
>>> On Tue, Nov 2, 2010 at 3:09 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>>>> Hmm... yes patch is here... I run fails on classmethod tests ;( don't
>>>> know how to solve this yet but all others are okay.
>>>
>>> Thanks! Did you want to post it to trac? It's preferable to use "hg
>>> export" so we get some metadata attached to the patch. (You can turn
>>> your Cython into a working directory by doing "make repo".)
>>>
>>
>> Ok I'll try... I'm not very familar with hg so I have to do commit then export?
>
> Yes, exactly. You might also want to set up a ~/.hgrc file with
>
> [ui]
> username = name <email>
>
> for the commit message (otherwise it just uses your logged-in username).
>
>>>> This patch also includes some tests for __metaclass__ and one more
>>>> test for staticmethod
>>>
>>> I'll try to find some time to look at the staticmethod issue, but
>>> won't complain if someone beats me to it :).
>>>
>>
>> Just one more case when staticmethod is decorator, now it works)
>>
>> class class4:
>> ? ?@staticmethod
>> ? ?def plus1(a):
>> ? ? ? ?return a + 1
>
> Good.
>
> - Robert
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From robertwb at math.washington.edu  Wed Nov  3 00:21:13 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 2 Nov 2010 16:21:13 -0700
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTinqQd9MhZ57WiS1gdGBWOs5A6hbGQvmrLTUF3F4@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CCE989D.6070907@behnel.de>
	<AANLkTin+ofqt6-z+-Z-bnuSmW9eSDL94xDs=D=dT3ic7@mail.gmail.com>
	<AANLkTikn4VJtcJ3EsESd4iVKNpg42fQK2EAdbAbuBA1o@mail.gmail.com>
	<AANLkTinPTHtp=X483QUGsRO58tm2TTokSc02LxaEt3fP@mail.gmail.com>
	<AANLkTikESy8G5XWBi=8cvmO0nCJgQg_A9dQ=znrz-XuN@mail.gmail.com>
	<AANLkTino6OWKMbgG41Wvn1vo-jNP=KRvnX2H9ai31rv-@mail.gmail.com>
	<AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>
	<AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>
	<AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>
	<AANLkTi=_qmWeP1A9wHVWRp2vDTGrOrVfWxoMvHEMQ4vL@mail.gmail.com>
	<AANLkTinRbxta5dRgciK+TjH2qGuz2pm8yZkao58FF4cR@mail.gmail.com>
	<AANLkTin8bserTjoVXhADG9xjgZzMdOafav4Eb75LyAG4@mail.gmail.com>
	<AANLkTinqQd9MhZ57WiS1gdGBWOs5A6hbGQvmrLTUF3F4@mail.gmail.com>
Message-ID: <AANLkTi=gNfVGwz45Rp1LXkL9Dg1K86cC=HJT1CpRZNdF@mail.gmail.com>

On Tue, Nov 2, 2010 at 4:17 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> I've attached patch, thank you :)

Thank you.

- Robert

From brett.calcott at gmail.com  Wed Nov  3 00:36:17 2010
From: brett.calcott at gmail.com (Brett Calcott)
Date: Wed, 3 Nov 2010 10:36:17 +1100
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>
	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>
Message-ID: <AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>

On 2 November 2010 17:17, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Sun, Oct 31, 2010 at 8:44 PM, Brett Calcott <brett.calcott at gmail.com> wrote:
>> 1. Why must I use a *pointer* in my python class?
>> cdef class PythonVector2d:
>> ??cdef Vector2d *obj
>> There is a default constructor, and I can happily declare it on the stack in
>> a function. I'm sure I'm missing something here; is there some danger with
>> allowing this?
>
> It's conceivable that we could allow "stack allocated" members in an
> object in the presence default constructors, but that would require
> modifying the constructor and destructor to manually initialize and
> deconstruct the object when the PyObject* is allocated and reclaimed.

I don't get why you need to manually do it. Doesn't the c++ default
construction just do it for you?

>> 2. How hard would it be to automatically forward member access to a
>> contained cppclass? Consider the following nasty hack:
>> cdef class PythonVector2d:
>> ??cdef public float x, y
>> ??cdef Vector2d *obj
>> ??def __cinit__(self, x, y):
>> ?? ?self.x = x
>> ?? ?self.y = y
>> ?? ?self.obj = <Vector 2d*>x
>> ok. That is a bit nasty (I'm sure someone can tell me lots of reasons why I
>> shouldn't do this).
>
> In reading that code, I'm not sure what your intent is.
>

Nastiness explained: Vector2d is a struct with 2 float members. I
declare 2 float members in the python object, and then get a pointer
to the first of these and cast it to a Vector2d. Assuming that the
struct layout is the same in both of these, then accessing members via
the cython properties will update the same shared bit of memory that
the c++ struct uses.

Brett

From brett.calcott at gmail.com  Wed Nov  3 00:51:51 2010
From: brett.calcott at gmail.com (Brett Calcott)
Date: Wed, 3 Nov 2010 10:51:51 +1100
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <4CCFD3A2.7090301@behnel.de>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>
	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>
	<4CCFD3A2.7090301@behnel.de>
Message-ID: <AANLkTik99s8btrtotC+7UtffcXiihWP7x9112ibZxCtT@mail.gmail.com>

On 2 November 2010 20:02, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 02.11.2010 07:17:
>> On Sun, Oct 31, 2010 at 8:44 PM, Brett Calcott wrote:
>>> cdef class PythonVector2d:
>>> ? ?cdef Vector obj
>>> ? ?properties:
>>> ? ? ? public obj.x as x
>>> ? ? ? public obj.y as y
<--- snip -->
> For this case, I'd prefer a version entirely without body, maybe even
>
> ? ? property x = obj.x
>
> although the "as" version would also work. However, it's not so clear from
> the above that both setter and getter get generated, and how you would get
> only the getter for read-only access, for example. And why not have a
> "delattr" as well? May make sense in some cases.
>

The proposal I made above would handle getter/setter differences using
the syntax that is already established.

properties:
  public obj.x as x
  readonly obj.y as y

I went for a body as it means not repeating the word property over and
over, and it allows renaming. I'm not seeing how delattr would make
sense yet.

> Personally, I think that the current way of coding this isn't all that
> cumbersome but a lot more explicit. And it makes you think a bit more about
> the implications of mapping a C++ class field directly to a Python property
> without value checking etc.
>

I've got structures with something like 10 members that I want to
directly map. I think it starts to look cumbersome then. The proposal
is really just extending an already existing option: you can automate
access to float/int etc using public and readonly of stuff declared in
your extension class. If a c++ library exposes structure members (ie,
they aren't private/protected), and you declare this in your extension
class, providing the same sort of access doesn't seem too much of a
stretch (of course, I have no idea how hard this would be to
implement!)

I should add: I like manually wrapping a c++ library with cython, as I
can make things more pythonic in the process. The automated swig way
of doing things fails to do this.

From robertwb at math.washington.edu  Wed Nov  3 02:16:15 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 2 Nov 2010 18:16:15 -0700
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>
	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>
	<AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>
Message-ID: <AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>

On Tue, Nov 2, 2010 at 4:36 PM, Brett Calcott <brett.calcott at gmail.com> wrote:
> On 2 November 2010 17:17, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>> On Sun, Oct 31, 2010 at 8:44 PM, Brett Calcott <brett.calcott at gmail.com> wrote:
>>> 1. Why must I use a *pointer* in my python class?
>>> cdef class PythonVector2d:
>>> ??cdef Vector2d *obj
>>> There is a default constructor, and I can happily declare it on the stack in
>>> a function. I'm sure I'm missing something here; is there some danger with
>>> allowing this?
>>
>> It's conceivable that we could allow "stack allocated" members in an
>> object in the presence default constructors, but that would require
>> modifying the constructor and destructor to manually initialize and
>> deconstruct the object when the PyObject* is allocated and reclaimed.
>
> I don't get why you need to manually do it. Doesn't the c++ default
> construction just do it for you?

No, because the memory for the class object is allocated deep withing
the Python library with an ordinary malloc, and the pointers just
passed around.

>>> 2. How hard would it be to automatically forward member access to a
>>> contained cppclass? Consider the following nasty hack:
>>> cdef class PythonVector2d:
>>> ??cdef public float x, y
>>> ??cdef Vector2d *obj
>>> ??def __cinit__(self, x, y):
>>> ?? ?self.x = x
>>> ?? ?self.y = y
>>> ?? ?self.obj = <Vector 2d*>x
>>> ok. That is a bit nasty (I'm sure someone can tell me lots of reasons why I
>>> shouldn't do this).
>>
>> In reading that code, I'm not sure what your intent is.
>>
>
> Nastiness explained: Vector2d is a struct with 2 float members. I
> declare 2 float members in the python object, and then get a pointer
> to the first of these and cast it to a Vector2d. Assuming that the
> struct layout is the same in both of these, then accessing members via
> the cython properties will update the same shared bit of memory that
> the c++ struct uses.

If this is just for syntactic sugar, it sound way to messy... I could
see simplifications of the properties syntax to make Python access
easier.

On 2 November 2010 20:02, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 02.11.2010 07:17:
>> On Sun, Oct 31, 2010 at 8:44 PM, Brett Calcott wrote:
>>> cdef class PythonVector2d:
>>>    cdef Vector obj
>>>    properties:
>>>       public obj.x as x
>>>       public obj.y as y
<--- snip -->
> For this case, I'd prefer a version entirely without body, maybe even
>
>     property x = obj.x
>
> although the "as" version would also work. However, it's not so clear from
> the above that both setter and getter get generated, and how you would get
> only the getter for read-only access, for example. And why not have a
> "delattr" as well? May make sense in some cases.

setters, getters, (and delattrs?) would be defined iff they make sense
on the expression in question. One could imagine

    property readonly obj.x as x

or something like that. I'm not a fan of the = sign, as it's no where
else used for declarations (it's always a statement), unless we do
something like

    public x = property(obj.x)

which reads kind of like a default value.

- Robert

From brett.calcott at gmail.com  Wed Nov  3 02:31:12 2010
From: brett.calcott at gmail.com (Brett Calcott)
Date: Wed, 3 Nov 2010 12:31:12 +1100
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>
	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>
	<AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>
	<AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>
Message-ID: <AANLkTik412YxxUx96j_G+fVoshj61c0AMdzQQb0J_qs5@mail.gmail.com>

>>
>> I don't get why you need to manually do it. Doesn't the c++ default
>> construction just do it for you?
>
> No, because the memory for the class object is allocated deep withing
> the Python library with an ordinary malloc, and the pointers just
> passed around.

I see. I should have thought of that. So in-place construction would
have to be done.

>>
>> Nastiness explained: Vector2d is a struct with 2 float members. I
>> declare 2 float members in the python object, and then get a pointer
>> to the first of these and cast it to a Vector2d. Assuming that the
>> struct layout is the same in both of these, then accessing members via
>> the cython properties will update the same shared bit of memory that
>> the c++ struct uses.
>
> If this is just for syntactic sugar, it sound way to messy... I could
> see simplifications of the properties syntax to make Python access
> easier.
>

Yes -- it is way too messy.

.
>
> setters, getters, (and delattrs?) would be defined iff they make sense
> on the expression in question. One could imagine
>
> ? ?property readonly obj.x as x
>
> or something like that. I'm not a fan of the = sign, as it's no where
> else used for declarations (it's always a statement), unless we do
> something like
>

I agree that the = sign is confusing. And both "property" and "as"
should be familiar to python users.

In any case, this would be a nice addition. Where would I look to do
something like this? Is it a big thing?

Brett

From stefan_ml at behnel.de  Wed Nov  3 06:13:00 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 03 Nov 2010 06:13:00 +0100
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>	<AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>
	<AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>
Message-ID: <4CD0EF5C.1080607@behnel.de>

Robert Bradshaw, 03.11.2010 02:16:
> On 2 November 2010 20:02, Stefan Behnel wrote:
>> Robert Bradshaw, 02.11.2010 07:17:
>>> On Sun, Oct 31, 2010 at 8:44 PM, Brett Calcott wrote:
>>>> cdef class PythonVector2d:
>>>>     cdef Vector obj
>>>>     properties:
>>>>        public obj.x as x
>>>>        public obj.y as y
> <--- snip -->
>> For this case, I'd prefer a version entirely without body, maybe even
>>
>>      property x = obj.x
>>
>> although the "as" version would also work. However, it's not so clear from
>> the above that both setter and getter get generated, and how you would get
>> only the getter for read-only access, for example. And why not have a
>> "delattr" as well? May make sense in some cases.
>
> setters, getters, (and delattrs?) would be defined iff they make sense
> on the expression in question. One could imagine
>
>      property readonly obj.x as x
>
> or something like that.

I'm generally ok with such an addition.

What about something like

     cdef class PythonVector2d:
         cdef Vector obj
         cdef public x from obj.x

? I'm wondering if it isn't better to keep the existing attribute syntax 
than to introduce a new property syntax. Although, a counter argument could 
be that it isn't really an attribute...

Next try:

     cdef class PythonVector2d:
         cdef Vector obj:
             property x
             property readonly y as my_y

Makes it clearer where the properties actually come from.

Anyway, I won't object to your plain "property ... as ..." proposal above.

Stefan

From stefan_ml at behnel.de  Wed Nov  3 06:59:05 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 03 Nov 2010 06:59:05 +0100
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTi=bFQwdofb4TPcV53b9w0RtcshvC8Uiv2mJa9i-@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>	<4CCE989D.6070907@behnel.de>	<AANLkTin+ofqt6-z+-Z-bnuSmW9eSDL94xDs=D=dT3ic7@mail.gmail.com>	<AANLkTikn4VJtcJ3EsESd4iVKNpg42fQK2EAdbAbuBA1o@mail.gmail.com>	<AANLkTinPTHtp=X483QUGsRO58tm2TTokSc02LxaEt3fP@mail.gmail.com>	<AANLkTikESy8G5XWBi=8cvmO0nCJgQg_A9dQ=znrz-XuN@mail.gmail.com>	<AANLkTino6OWKMbgG41Wvn1vo-jNP=KRvnX2H9ai31rv-@mail.gmail.com>	<AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>	<AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>	<AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>	<AANLkTikSi2Awwv31sx3A6BJy=WNSYROo+ceecoygQ0Fk@mail.gmail.com>
	<AANLkTi=bFQwdofb4TPcV53b9w0RtcshvC8Uiv2mJa9i-@mail.gmail.com>
Message-ID: <4CD0FA29.7000201@behnel.de>

Robert Bradshaw, 02.11.2010 23:21:
> On Tue, Nov 2, 2010 at 3:11 PM, Vitja Makarov wrote:
>> http://docs.python.org/reference/datamodel.html#basic-customization
>>
>> """Called to create a new instance of class cls. __new__() is a static
>> method (special-cased so you need not declare it as such) that takes
>> the class of which an instance was requested as its first argument.
>> The remaining arguments are those passed to the object constructor
>> expression (the call to the class). The return value of __new__()
>> should be the new object instance (usually an instance of cls)."""
>
> I was actually thinking about cdef classes, where __new__ is much more
> subtle. I didn't realize it was a static method that takes the class
> as a first argument rather than just a class method.

It needs to be to support inheritance. You usually call the __new__ method 
of the superclass to create the instance that you return. So you must be 
able to pass the required class through the hierarchy explicitly.

Stefan

From robertwb at math.washington.edu  Wed Nov  3 07:15:51 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 2 Nov 2010 23:15:51 -0700
Subject: [Cython] metaclasses
In-Reply-To: <4CD0FA29.7000201@behnel.de>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CCE989D.6070907@behnel.de>
	<AANLkTin+ofqt6-z+-Z-bnuSmW9eSDL94xDs=D=dT3ic7@mail.gmail.com>
	<AANLkTikn4VJtcJ3EsESd4iVKNpg42fQK2EAdbAbuBA1o@mail.gmail.com>
	<AANLkTinPTHtp=X483QUGsRO58tm2TTokSc02LxaEt3fP@mail.gmail.com>
	<AANLkTikESy8G5XWBi=8cvmO0nCJgQg_A9dQ=znrz-XuN@mail.gmail.com>
	<AANLkTino6OWKMbgG41Wvn1vo-jNP=KRvnX2H9ai31rv-@mail.gmail.com>
	<AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>
	<AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>
	<AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>
	<AANLkTikSi2Awwv31sx3A6BJy=WNSYROo+ceecoygQ0Fk@mail.gmail.com>
	<AANLkTi=bFQwdofb4TPcV53b9w0RtcshvC8Uiv2mJa9i-@mail.gmail.com>
	<4CD0FA29.7000201@behnel.de>
Message-ID: <AANLkTinNGspAzLoOBdoXcUZgxkxR86i4aQfS2FMrgVYR@mail.gmail.com>

On Tue, Nov 2, 2010 at 10:59 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 02.11.2010 23:21:
>> On Tue, Nov 2, 2010 at 3:11 PM, Vitja Makarov wrote:
>>> http://docs.python.org/reference/datamodel.html#basic-customization
>>>
>>> """Called to create a new instance of class cls. __new__() is a static
>>> method (special-cased so you need not declare it as such) that takes
>>> the class of which an instance was requested as its first argument.
>>> The remaining arguments are those passed to the object constructor
>>> expression (the call to the class). The return value of __new__()
>>> should be the new object instance (usually an instance of cls)."""
>>
>> I was actually thinking about cdef classes, where __new__ is much more
>> subtle. I didn't realize it was a static method that takes the class
>> as a first argument rather than just a class method.
>
> It needs to be to support inheritance. You usually call the __new__ method
> of the superclass to create the instance that you return. So you must be
> able to pass the required class through the hierarchy explicitly.

Good point, I hadn't thought about that.

- Robert

From robertwb at math.washington.edu  Wed Nov  3 07:29:54 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 2 Nov 2010 23:29:54 -0700
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <4CD0EF5C.1080607@behnel.de>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>
	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>
	<AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>
	<AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>
	<4CD0EF5C.1080607@behnel.de>
Message-ID: <AANLkTikTk7UruwF0gjrDU-Q_Y2xUSj7xQD7aoFuJ68v6@mail.gmail.com>

On Tue, Nov 2, 2010 at 10:13 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 03.11.2010 02:16:
>> On 2 November 2010 20:02, Stefan Behnel wrote:
>>> Robert Bradshaw, 02.11.2010 07:17:
>>>> On Sun, Oct 31, 2010 at 8:44 PM, Brett Calcott wrote:
>>>>> cdef class PythonVector2d:
>>>>> ? ? cdef Vector obj
>>>>> ? ? properties:
>>>>> ? ? ? ?public obj.x as x
>>>>> ? ? ? ?public obj.y as y
>> <--- snip -->
>>> For this case, I'd prefer a version entirely without body, maybe even
>>>
>>> ? ? ?property x = obj.x
>>>
>>> although the "as" version would also work. However, it's not so clear from
>>> the above that both setter and getter get generated, and how you would get
>>> only the getter for read-only access, for example. And why not have a
>>> "delattr" as well? May make sense in some cases.
>>
>> setters, getters, (and delattrs?) would be defined iff they make sense
>> on the expression in question. One could imagine
>>
>> ? ? ?property readonly obj.x as x
>>
>> or something like that.
>
> I'm generally ok with such an addition.
>
> What about something like
>
> ? ? cdef class PythonVector2d:
> ? ? ? ? cdef Vector obj
> ? ? ? ? cdef public x from obj.x

I think of "cdef" as something accessible from c. This could be OK it
did c-level attribute renaming as well. (Hmm... maybe I'd be OK with
that, but it feels a bit odd to create this alias on the C level.)

> ? I'm wondering if it isn't better to keep the existing attribute syntax
> than to introduce a new property syntax. Although, a counter argument could
> be that it isn't really an attribute...

Yep. It's nice to look at the class and be able to visualize its struct.

> Next try:
>
> ? ? cdef class PythonVector2d:
> ? ? ? ? cdef Vector obj:
> ? ? ? ? ? ? property x
> ? ? ? ? ? ? property readonly y as my_y
>
> Makes it clearer where the properties actually come from.

This doesn't allow arbitrary expressions. On that note, should we
require an explicit self.obj.x, or is self implicit? (The latter makes
it impossible to refer to non-attributes, and is bad according to the
zen of Python...), so I'd lean against this one.

> Anyway, I won't object to your plain "property ... as ..." proposal above.

Or "property ... from ..." if the name should be first (I don't have a
strong preference, but the as is a more idiomatic keyword to use
here).

- Robert

From stefan_ml at behnel.de  Wed Nov  3 07:56:18 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 03 Nov 2010 07:56:18 +0100
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <AANLkTikTk7UruwF0gjrDU-Q_Y2xUSj7xQD7aoFuJ68v6@mail.gmail.com>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>	<AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>	<AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>	<4CD0EF5C.1080607@behnel.de>
	<AANLkTikTk7UruwF0gjrDU-Q_Y2xUSj7xQD7aoFuJ68v6@mail.gmail.com>
Message-ID: <4CD10792.3000507@behnel.de>

Robert Bradshaw, 03.11.2010 07:29:
> On Tue, Nov 2, 2010 at 10:13 PM, Stefan Behnel wrote:
>> What about something like
>>
>>      cdef class PythonVector2d:
>>          cdef Vector obj
>>          cdef public x from obj.x
>
> I think of "cdef" as something accessible from c. This could be OK it
> did c-level attribute renaming as well. (Hmm... maybe I'd be OK with
> that, but it feels a bit odd to create this alias on the C level.)
>
>> ? I'm wondering if it isn't better to keep the existing attribute syntax
>> than to introduce a new property syntax. Although, a counter argument could
>> be that it isn't really an attribute...
>
> Yep. It's nice to look at the class and be able to visualize its struct.

Agreed.


>> Next try:
>>
>>      cdef class PythonVector2d:
>>          cdef Vector obj:
>>              property x
>>              property readonly y as my_y
>>
>> Makes it clearer where the properties actually come from.
>
> This doesn't allow arbitrary expressions.

Well, you can't allow "arbitrary expressions" since you will end up having 
to assign to them in the setter. You can only allow arbitrary attribute 
paths, which isn't prevented by the above.

We could distinguish between normal and read-only properties, though, and 
allow pretty much arbitrary expressions for the latter. Basically like a 
lambda expression wrapped into a property descriptor.


> On that note, should we
> require an explicit self.obj.x, or is self implicit? (The latter makes
> it impossible to refer to non-attributes, and is bad according to the
> zen of Python...), so I'd lean against this one.

That's a very good idea. Makes it more readable and a lot clearer what 
happens. This also removes the link to C++ as you can access any attribute 
of the type in one way or another.

Would you actually require it to refer to "self", or would you allow 
arbitrary global names? I wouldn't even mind the latter, although maybe 
restricted to static cdef declared names (at least of we generate a setter).


>> Anyway, I won't object to your plain "property ... as ..." proposal above.
>
> Or "property ... from ..." if the name should be first (I don't have a
> strong preference, but the as is a more idiomatic keyword to use
> here).

I think it makes sense to put the name first as the access path can be 
arbitrarily long. It also simplifies the syntax as the keyword "property" 
is then always followed by a name, then either ":" or "from" to distinguish 
the two cases.

We could also use

     property x as if self.obj.x

but I think "from" reads well enough here. ;-)

Stefan

From robertwb at math.washington.edu  Wed Nov  3 08:25:08 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 3 Nov 2010 00:25:08 -0700
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <4CD10792.3000507@behnel.de>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>
	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>
	<AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>
	<AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>
	<4CD0EF5C.1080607@behnel.de>
	<AANLkTikTk7UruwF0gjrDU-Q_Y2xUSj7xQD7aoFuJ68v6@mail.gmail.com>
	<4CD10792.3000507@behnel.de>
Message-ID: <AANLkTikua0kipkGTE0B4VLQUAPi7K5+b11-7Q-TEzJT4@mail.gmail.com>

On Tue, Nov 2, 2010 at 11:56 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 03.11.2010 07:29:
>> On Tue, Nov 2, 2010 at 10:13 PM, Stefan Behnel wrote:
>>> What about something like
>>>
>>> ? ? ?cdef class PythonVector2d:
>>> ? ? ? ? ?cdef Vector obj
>>> ? ? ? ? ?cdef public x from obj.x
>>
>> I think of "cdef" as something accessible from c. This could be OK it
>> did c-level attribute renaming as well. (Hmm... maybe I'd be OK with
>> that, but it feels a bit odd to create this alias on the C level.)
>>
>>> ? I'm wondering if it isn't better to keep the existing attribute syntax
>>> than to introduce a new property syntax. Although, a counter argument could
>>> be that it isn't really an attribute...
>>
>> Yep. It's nice to look at the class and be able to visualize its struct.
>
> Agreed.
>
>
>>> Next try:
>>>
>>> ? ? ?cdef class PythonVector2d:
>>> ? ? ? ? ?cdef Vector obj:
>>> ? ? ? ? ? ? ?property x
>>> ? ? ? ? ? ? ?property readonly y as my_y
>>>
>>> Makes it clearer where the properties actually come from.
>>
>> This doesn't allow arbitrary expressions.
>
> Well, you can't allow "arbitrary expressions" since you will end up having
> to assign to them in the setter. You can only allow arbitrary attribute
> paths, which isn't prevented by the above.
>
> We could distinguish between normal and read-only properties, though, and
> allow pretty much arbitrary expressions for the latter. Basically like a
> lambda expression wrapped into a property descriptor.

Yep, that's what I was thinking. Attribute access paths would probably
be the most common, but no need to restrict ourselves for the getter.

>> On that note, should we
>> require an explicit self.obj.x, or is self implicit? (The latter makes
>> it impossible to refer to non-attributes, and is bad according to the
>> zen of Python...), so I'd lean against this one.
>
> That's a very good idea. Makes it more readable and a lot clearer what
> happens. This also removes the link to C++ as you can access any attribute
> of the type in one way or another.
>
> Would you actually require it to refer to "self", or would you allow
> arbitrary global names? I wouldn't even mind the latter, although maybe
> restricted to static cdef declared names (at least of we generate a setter).

Hmm.... I would at least generate a warning if not assigning to a self
attribute.

>>> Anyway, I won't object to your plain "property ... as ..." proposal above.
>>
>> Or "property ... from ..." if the name should be first (I don't have a
>> strong preference, but the as is a more idiomatic keyword to use
>> here).
>
> I think it makes sense to put the name first as the access path can be
> arbitrarily long. It also simplifies the syntax as the keyword "property"
> is then always followed by a name, then either ":" or "from" to distinguish
> the two cases.

+1

> We could also use
>
> ? ? property x as if self.obj.x
>
> but I think "from" reads well enough here. ;-)

Yeah, I like "from."

- Robert

From robertwb at math.washington.edu  Wed Nov  3 08:26:38 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 3 Nov 2010 00:26:38 -0700
Subject: [Cython] auto __test__ dict
In-Reply-To: <AANLkTikzsMY36n-Cmyh9uJ82a7MRQcF-66vdLtK-id3H@mail.gmail.com>
References: <4CCC6F28.1040705@behnel.de> <4CCD1A7C.3010805@behnel.de>
	<4CCDB7C3.20208@behnel.de>
	<AANLkTikBoY5pFx=AO=G7iVPmLqnrhwzjhZ1hTOA=SRO5@mail.gmail.com>
	<4CCFD0A3.8010503@behnel.de>
	<AANLkTikzsMY36n-Cmyh9uJ82a7MRQcF-66vdLtK-id3H@mail.gmail.com>
Message-ID: <AANLkTinR2dFJWTPH6R+D5NN7j8uKZs1YZCP=C6dnTfhF@mail.gmail.com>

On Tue, Nov 2, 2010 at 4:17 AM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
> On 2 November 2010 11:49, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Robert Bradshaw, 01.11.2010 21:44:
>>> On Sun, Oct 31, 2010 at 11:38 AM, Stefan Behnel wrote:
>>>>> One problem I spotted was the auto test dict (again).
>>>>
>>>> I simplified it now to use the compile time docstrings directly, instead of
>>>> looking them up at runtime. I don't think this breaks anything important,
>>>> as the docstrings of functions and methods cannot be modified anyway. On
>>>> the plus side, it removes a *ton* of ugly C code, and also saves some time
>>>> during module initialisation.
>>>
>>> Excellent! I've been wanting to do this myself, eventually when I
>>> found the time. This means that cdef methods could have tested
>>> docstrings as well (though, of course, they couldn't be called
>>> directly).
>>
>> That was easy to implement, so I tried it. In lxml.etree, it adds some 10KB
>> to the size of the module source code as the docstrings of cdef functions
>> wouldn't otherwise end up there. Personally, I'm not sure it's worth it. If
>> you can't call a cdef function/method from a doctest directly, why not just
>> move the doctest itself to the function or method that you would have to
>> call anyway?
>>
>
> I agree with Stefan.

Sometimes I have a doctest in, e.g., a class body docstring to
(indirectly) test a cdef method. Not common though. On this note,
would it be easy/feasible to only insert docstrings that actually have
tests in them?

- Robert

From robertwb at math.washington.edu  Wed Nov  3 08:31:01 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 3 Nov 2010 00:31:01 -0700
Subject: [Cython] New cdef class directives: "final" and "internal"
In-Reply-To: <4CCFBE88.8070502@behnel.de>
References: <4CCC6F28.1040705@behnel.de>
	<AANLkTinq_z6=JZN7kU=VNPSXS_J7T8OMVsqq+u4bZHbC@mail.gmail.com>
	<AANLkTikGQ0a4sNzuCPNYPYcpGj8p0bp3BZ-_qZtbKT4r@mail.gmail.com>
	<4CCD0834.20801@behnel.de>
	<AANLkTikZhqONQ4DCE7SB9z-_OX45b61Na3zCA+9B0rzq@mail.gmail.com>
	<AANLkTinQ3sWbo6N=i+T6=96jquvHPOHRMck0fOExZL6j@mail.gmail.com>
	<4CCD75A1.8050909@behnel.de>
	<AANLkTimmX5J0NJ5ZkJA0Ftim6sCdSX0aVmhZAiDdUqLd@mail.gmail.com>
	<4CCFBE88.8070502@behnel.de>
Message-ID: <AANLkTikYqBtNp8idRCw_8C+Gqy71Us04Z8a4Q-HGGO7u@mail.gmail.com>

On Tue, Nov 2, 2010 at 12:32 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 01.11.2010 22:04:
>> On Sun, Oct 31, 2010 at 6:56 AM, Stefan Behnel wrote:
>>>> On 31 October 2010 10:09, Robert Bradshaw wrote:
>>>>> [cdef] class A:
>>>>> ? ? ?...
>>>>>
>>>>> del A
>>>
>>> Should Cython prevent this
>>> for public/api classes, or would you just get the ImportError at runtime
>>> when it tries to read the type from the module dict?
>>
>> Personally, I would not be opposed to putting everything into the
>> __pyx_api object rather than loading anything from the module dict.
>> That would be safer for one thing.
>
> Sounds better, yes. And with an additional fallback to reading classes from
> the module dict, it would even be backwards compatible.
>
> I think that would be part of the public/api unification that we considered
> a while ago.
>
>
>> I would say that "del" simply removes things from the Python
>> namespace. The c namespace is fixed at compile time, and would remain
>> so. This is essentially the case already, though perhaps we disallow
>> del for cdef classes as they live in both namespaces (don't know).
>
> See? That's the kind of "don't know" that I meant. I'm not sure either.

I meant I don't know what happens now.

> When I see "del TheClass" in a module, do I have to know what kind of class
> this is in order to know if I can continue to refer to it inside the module?

Good point.

>> We'd probably want to support del on cdef classes for pure Python mode
>> anyways.
>
> It wouldn't work the same way, though. If you "del" a cdef class in pure
> mode running under Python, its name will disappear. If you "del" it from
> within Cython, it will continue to be accessible under that name and likely
> wouldn't support redefinition either.
>
>
>> My main objection is that this decorator is yet another thing that
>> doesn't, and cannot, have an exact Python equivalent, and there's a
>> reasonable and widespread Python idiom to use in this case.
>
> I agree. I'm just saying that it is a well defined and useful feature
> within Cython. Maybe we could really only allow it within Cython modules,
> i.e. use a cdef class modifier instead of a decorator. That would prevent
> its ambiguous use from pure mode and we could continue to disallow "del" on
> cdef classes as that's also ambiguous in pure more (and also mostly useless
> as it doesn't free any resources nor allow redefinition).

To clarify, I'm only slightly opposed to this decorator, so if you
think it's useful (and you've made a good case) I'm OK. Definitely
want final, though unfortunately it seems that final functions and
methods would still have to pass through function pointers (perhaps
with one level less of indirection) for cross-module calls (unless we
did some kind of compile-time linking).

- Robert

From stefan_ml at behnel.de  Wed Nov  3 08:42:06 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 03 Nov 2010 08:42:06 +0100
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <AANLkTikua0kipkGTE0B4VLQUAPi7K5+b11-7Q-TEzJT4@mail.gmail.com>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>	<AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>	<AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>	<4CD0EF5C.1080607@behnel.de>	<AANLkTikTk7UruwF0gjrDU-Q_Y2xUSj7xQD7aoFuJ68v6@mail.gmail.com>	<4CD10792.3000507@behnel.de>
	<AANLkTikua0kipkGTE0B4VLQUAPi7K5+b11-7Q-TEzJT4@mail.gmail.com>
Message-ID: <4CD1124E.5050307@behnel.de>

Robert Bradshaw, 03.11.2010 08:25:
> On Tue, Nov 2, 2010 at 11:56 PM, Stefan Behnel wrote:
>> We could distinguish between normal and read-only properties, though, and
>> allow pretty much arbitrary expressions for the latter. Basically like a
>> lambda expression wrapped into a property descriptor.
>
> Yep, that's what I was thinking. Attribute access paths would probably
> be the most common, but no need to restrict ourselves for the getter.
>
>>> On that note, should we
>>> require an explicit self.obj.x, or is self implicit? (The latter makes
>>> it impossible to refer to non-attributes, and is bad according to the
>>> zen of Python...), so I'd lean against this one.
>>
>> That's a very good idea. Makes it more readable and a lot clearer what
>> happens. This also removes the link to C++ as you can access any attribute
>> of the type in one way or another.
>>
>> Would you actually require it to refer to "self", or would you allow
>> arbitrary global names? I wouldn't even mind the latter, although maybe
>> restricted to static cdef declared names (at least of we generate a setter).
>
> Hmm.... I would at least generate a warning if not assigning to a self
> attribute.

Ok, then let's support arbitrary expressions for read-only properties and 
self-attribute-only expressions for normal (getter+setter) properties.

Should it be

     property x from readonly expr

or

     property readonly x from expr

? I lean towards the first, once again due to the simpler grammar.

Then we get

     property x from readonly anyexpr

and

     property x from self.attribute

(i.e. "from" followed by a dotted name, which may be "readonly", which 
would then be followed by an arbitrary expression)

I don't think we need a "public" for the getter+setter case, "property" is 
clear enough IMHO.

Stefan

From robertwb at math.washington.edu  Wed Nov  3 08:52:56 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 3 Nov 2010 00:52:56 -0700
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <4CD1124E.5050307@behnel.de>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>
	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>
	<AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>
	<AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>
	<4CD0EF5C.1080607@behnel.de>
	<AANLkTikTk7UruwF0gjrDU-Q_Y2xUSj7xQD7aoFuJ68v6@mail.gmail.com>
	<4CD10792.3000507@behnel.de>
	<AANLkTikua0kipkGTE0B4VLQUAPi7K5+b11-7Q-TEzJT4@mail.gmail.com>
	<4CD1124E.5050307@behnel.de>
Message-ID: <AANLkTi=3H0iWCAroDbrpk6pDnAEsJkgxxn4Ai4Vrn-zk@mail.gmail.com>

On Wed, Nov 3, 2010 at 12:42 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 03.11.2010 08:25:
>> On Tue, Nov 2, 2010 at 11:56 PM, Stefan Behnel wrote:
>>> We could distinguish between normal and read-only properties, though, and
>>> allow pretty much arbitrary expressions for the latter. Basically like a
>>> lambda expression wrapped into a property descriptor.
>>
>> Yep, that's what I was thinking. Attribute access paths would probably
>> be the most common, but no need to restrict ourselves for the getter.
>>
>>>> On that note, should we
>>>> require an explicit self.obj.x, or is self implicit? (The latter makes
>>>> it impossible to refer to non-attributes, and is bad according to the
>>>> zen of Python...), so I'd lean against this one.
>>>
>>> That's a very good idea. Makes it more readable and a lot clearer what
>>> happens. This also removes the link to C++ as you can access any attribute
>>> of the type in one way or another.
>>>
>>> Would you actually require it to refer to "self", or would you allow
>>> arbitrary global names? I wouldn't even mind the latter, although maybe
>>> restricted to static cdef declared names (at least of we generate a setter).
>>
>> Hmm.... I would at least generate a warning if not assigning to a self
>> attribute.
>
> Ok, then let's support arbitrary expressions for read-only properties and
> self-attribute-only expressions for normal (getter+setter) properties.
>
> Should it be
>
> ? ? property x from readonly expr
>
> or
>
> ? ? property readonly x from expr
>
> ? I lean towards the first, once again due to the simpler grammar.
>
> Then we get
>
> ? ? property x from readonly anyexpr
>
> and
>
> ? ? property x from self.attribute
>
> (i.e. "from" followed by a dotted name, which may be "readonly", which
> would then be followed by an arbitrary expression)

Sound good to me. It's possible that index expressions could support
setters as well.

> I don't think we need a "public" for the getter+setter case, "property" is
> clear enough IMHO.

Agreed.

- Robert

From stefan_ml at behnel.de  Wed Nov  3 09:01:30 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 03 Nov 2010 09:01:30 +0100
Subject: [Cython] auto __test__ dict
In-Reply-To: <AANLkTinR2dFJWTPH6R+D5NN7j8uKZs1YZCP=C6dnTfhF@mail.gmail.com>
References: <4CCC6F28.1040705@behnel.de>
	<4CCD1A7C.3010805@behnel.de>	<4CCDB7C3.20208@behnel.de>	<AANLkTikBoY5pFx=AO=G7iVPmLqnrhwzjhZ1hTOA=SRO5@mail.gmail.com>	<4CCFD0A3.8010503@behnel.de>	<AANLkTikzsMY36n-Cmyh9uJ82a7MRQcF-66vdLtK-id3H@mail.gmail.com>
	<AANLkTinR2dFJWTPH6R+D5NN7j8uKZs1YZCP=C6dnTfhF@mail.gmail.com>
Message-ID: <4CD116DA.2030204@behnel.de>

Robert Bradshaw, 03.11.2010 08:26:
> On Tue, Nov 2, 2010 at 4:17 AM, Lisandro Dalcin wrote:
>> On 2 November 2010 11:49, Stefan Behnel wrote:
>>> Robert Bradshaw, 01.11.2010 21:44:
>>>> On Sun, Oct 31, 2010 at 11:38 AM, Stefan Behnel wrote:
>>>>>> One problem I spotted was the auto test dict (again).
>>>>>
>>>>> I simplified it now to use the compile time docstrings directly, instead of
>>>>> looking them up at runtime. I don't think this breaks anything important,
>>>>> as the docstrings of functions and methods cannot be modified anyway. On
>>>>> the plus side, it removes a *ton* of ugly C code, and also saves some time
>>>>> during module initialisation.
>>>>
>>>> Excellent! I've been wanting to do this myself, eventually when I
>>>> found the time. This means that cdef methods could have tested
>>>> docstrings as well (though, of course, they couldn't be called
>>>> directly).
>>>
>>> That was easy to implement, so I tried it. In lxml.etree, it adds some 10KB
>>> to the size of the module source code as the docstrings of cdef functions
>>> wouldn't otherwise end up there. Personally, I'm not sure it's worth it. If
>>> you can't call a cdef function/method from a doctest directly, why not just
>>> move the doctest itself to the function or method that you would have to
>>> call anyway?
>>
>> I agree with Stefan.
>
> Sometimes I have a doctest in, e.g., a class body docstring to
> (indirectly) test a cdef method.

Hmm, but putting that doctest into the cdef method's docstring would still 
test it through the class, i.e. it would test the interface of the class, 
not just the cdef method.

I think this is really just a minor issue.


> On this note,
> would it be easy/feasible to only insert docstrings that actually have
> tests in them?

I thought about that, too. You could easily check for ">>> " being in the 
text, that would pretty clearly hint towards a doctest. However, leaving 
everything else out means that it's impossible to run non-doctest based 
docstring tests, e.g. by just putting some kind of test data into the 
docstrings.

What about restricting it to docstrings containing ">>> " by default and 
adding another directive like "autotestdict.all" to include all docstrings?

Maybe also "autotestdict.cdef" to include matching cdef function/method 
docstrings in addition to the matching Python functions?

    autotestdict (default) < autotestdict.cdef < autotestdict.all

I think that would basically cover all use cases, including the less common 
ones.

Stefan

From stefan_ml at behnel.de  Wed Nov  3 09:05:26 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 03 Nov 2010 09:05:26 +0100
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <AANLkTi=3H0iWCAroDbrpk6pDnAEsJkgxxn4Ai4Vrn-zk@mail.gmail.com>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>	<AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>	<AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>	<4CD0EF5C.1080607@behnel.de>	<AANLkTikTk7UruwF0gjrDU-Q_Y2xUSj7xQD7aoFuJ68v6@mail.gmail.com>	<4CD10792.3000507@behnel.de>	<AANLkTikua0kipkGTE0B4VLQUAPi7K5+b11-7Q-TEzJT4@mail.gmail.com>	<4CD1124E.5050307@behnel.de>
	<AANLkTi=3H0iWCAroDbrpk6pDnAEsJkgxxn4Ai4Vrn-zk@mail.gmail.com>
Message-ID: <4CD117C6.4050905@behnel.de>

Robert Bradshaw, 03.11.2010 08:52:
> On Wed, Nov 3, 2010 at 12:42 AM, Stefan Behnel wrote:
>> Ok, then let's support arbitrary expressions for read-only properties and
>> self-attribute-only expressions for normal (getter+setter) properties.
>>
>> Should it be
>>
>>      property x from readonly expr
>>
>> or
>>
>>      property readonly x from expr
>>
>> ? I lean towards the first, once again due to the simpler grammar.
>>
>> Then we get
>>
>>      property x from readonly anyexpr
>>
>> and
>>
>>      property x from self.attribute
>>
>> (i.e. "from" followed by a dotted name, which may be "readonly", which
>> would then be followed by an arbitrary expression)
>
> Sound good to me. It's possible that index expressions could support
> setters as well.

Hmm, right, and they may really make a useful feature. That'll be trickier 
to test, though, as it can no longer be done that easily in the parser (I 
think).

Stefan

From robertwb at math.washington.edu  Wed Nov  3 09:18:56 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 3 Nov 2010 01:18:56 -0700
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <4CD117C6.4050905@behnel.de>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>
	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>
	<AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>
	<AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>
	<4CD0EF5C.1080607@behnel.de>
	<AANLkTikTk7UruwF0gjrDU-Q_Y2xUSj7xQD7aoFuJ68v6@mail.gmail.com>
	<4CD10792.3000507@behnel.de>
	<AANLkTikua0kipkGTE0B4VLQUAPi7K5+b11-7Q-TEzJT4@mail.gmail.com>
	<4CD1124E.5050307@behnel.de>
	<AANLkTi=3H0iWCAroDbrpk6pDnAEsJkgxxn4Ai4Vrn-zk@mail.gmail.com>
	<4CD117C6.4050905@behnel.de>
Message-ID: <AANLkTi=UNMoJQ-npmH50FrCdBdcdk89ZqBtVrSAu7dNg@mail.gmail.com>

On Wed, Nov 3, 2010 at 1:05 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 03.11.2010 08:52:
>> On Wed, Nov 3, 2010 at 12:42 AM, Stefan Behnel wrote:
>>> Ok, then let's support arbitrary expressions for read-only properties and
>>> self-attribute-only expressions for normal (getter+setter) properties.
>>>
>>> Should it be
>>>
>>> ? ? ?property x from readonly expr
>>>
>>> or
>>>
>>> ? ? ?property readonly x from expr
>>>
>>> ? I lean towards the first, once again due to the simpler grammar.
>>>
>>> Then we get
>>>
>>> ? ? ?property x from readonly anyexpr
>>>
>>> and
>>>
>>> ? ? ?property x from self.attribute
>>>
>>> (i.e. "from" followed by a dotted name, which may be "readonly", which
>>> would then be followed by an arbitrary expression)
>>
>> Sound good to me. It's possible that index expressions could support
>> setters as well.
>
> Hmm, right, and they may really make a useful feature. That'll be trickier
> to test, though, as it can no longer be done that easily in the parser (I
> think).

No need to do this in the parser, just say that readonly is required
if it's not an lvalue.

- Robert

From robertwb at math.washington.edu  Wed Nov  3 09:22:03 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 3 Nov 2010 01:22:03 -0700
Subject: [Cython] auto __test__ dict
In-Reply-To: <4CD116DA.2030204@behnel.de>
References: <4CCC6F28.1040705@behnel.de> <4CCD1A7C.3010805@behnel.de>
	<4CCDB7C3.20208@behnel.de>
	<AANLkTikBoY5pFx=AO=G7iVPmLqnrhwzjhZ1hTOA=SRO5@mail.gmail.com>
	<4CCFD0A3.8010503@behnel.de>
	<AANLkTikzsMY36n-Cmyh9uJ82a7MRQcF-66vdLtK-id3H@mail.gmail.com>
	<AANLkTinR2dFJWTPH6R+D5NN7j8uKZs1YZCP=C6dnTfhF@mail.gmail.com>
	<4CD116DA.2030204@behnel.de>
Message-ID: <AANLkTimskYdvEP0DavaZLzma+KLx2dQyxOXC2+ju7Vab@mail.gmail.com>

On Wed, Nov 3, 2010 at 1:01 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 03.11.2010 08:26:
>> On Tue, Nov 2, 2010 at 4:17 AM, Lisandro Dalcin wrote:
>>> On 2 November 2010 11:49, Stefan Behnel wrote:
>>>> Robert Bradshaw, 01.11.2010 21:44:
>>>>> On Sun, Oct 31, 2010 at 11:38 AM, Stefan Behnel wrote:
>>>>>>> One problem I spotted was the auto test dict (again).
>>>>>>
>>>>>> I simplified it now to use the compile time docstrings directly, instead of
>>>>>> looking them up at runtime. I don't think this breaks anything important,
>>>>>> as the docstrings of functions and methods cannot be modified anyway. On
>>>>>> the plus side, it removes a *ton* of ugly C code, and also saves some time
>>>>>> during module initialisation.
>>>>>
>>>>> Excellent! I've been wanting to do this myself, eventually when I
>>>>> found the time. This means that cdef methods could have tested
>>>>> docstrings as well (though, of course, they couldn't be called
>>>>> directly).
>>>>
>>>> That was easy to implement, so I tried it. In lxml.etree, it adds some 10KB
>>>> to the size of the module source code as the docstrings of cdef functions
>>>> wouldn't otherwise end up there. Personally, I'm not sure it's worth it. If
>>>> you can't call a cdef function/method from a doctest directly, why not just
>>>> move the doctest itself to the function or method that you would have to
>>>> call anyway?
>>>
>>> I agree with Stefan.
>>
>> Sometimes I have a doctest in, e.g., a class body docstring to
>> (indirectly) test a cdef method.
>
> Hmm, but putting that doctest into the cdef method's docstring would still
> test it through the class, i.e. it would test the interface of the class,
> not just the cdef method.
>
> I think this is really just a minor issue.

I agree.

>> On this note,
>> would it be easy/feasible to only insert docstrings that actually have
>> tests in them?
>
> I thought about that, too. You could easily check for ">>> " being in the
> text, that would pretty clearly hint towards a doctest. However, leaving
> everything else out means that it's impossible to run non-doctest based
> docstring tests, e.g. by just putting some kind of test data into the
> docstrings.
>
> What about restricting it to docstrings containing ">>> " by default and
> adding another directive like "autotestdict.all" to include all docstrings?
>
> Maybe also "autotestdict.cdef" to include matching cdef function/method
> docstrings in addition to the matching Python functions?
>
> ? ?autotestdict (default) < autotestdict.cdef < autotestdict.all
>
> I think that would basically cover all use cases, including the less common
> ones.

+1. Dotted directives are already supported.

- Robert

From dagss at student.matnat.uio.no  Wed Nov  3 09:24:16 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 03 Nov 2010 09:24:16 +0100
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>	<AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>
	<AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>
Message-ID: <4CD11C30.9090103@student.matnat.uio.no>

On 11/03/2010 02:16 AM, Robert Bradshaw wrote:
> On Tue, Nov 2, 2010 at 4:36 PM, Brett Calcott<brett.calcott at gmail.com>  wrote:
>    
>> On 2 November 2010 17:17, Robert Bradshaw<robertwb at math.washington.edu>  wrote:
>>      
>>> On Sun, Oct 31, 2010 at 8:44 PM, Brett Calcott<brett.calcott at gmail.com>  wrote:
>>>        
>>>> 1. Why must I use a *pointer* in my python class?
>>>> cdef class PythonVector2d:
>>>>    cdef Vector2d *obj
>>>> There is a default constructor, and I can happily declare it on the stack in
>>>> a function. I'm sure I'm missing something here; is there some danger with
>>>> allowing this?
>>>>          
>>> It's conceivable that we could allow "stack allocated" members in an
>>> object in the presence default constructors, but that would require
>>> modifying the constructor and destructor to manually initialize and
>>> deconstruct the object when the PyObject* is allocated and reclaimed.
>>>        
>> I don't get why you need to manually do it. Doesn't the c++ default
>> construction just do it for you?
>>      
> No, because the memory for the class object is allocated deep withing
> the Python library with an ordinary malloc, and the pointers just
> passed around.
>
>    
>>>> 2. How hard would it be to automatically forward member access to a
>>>> contained cppclass? Consider the following nasty hack:
>>>> cdef class PythonVector2d:
>>>>    cdef public float x, y
>>>>    cdef Vector2d *obj
>>>>    def __cinit__(self, x, y):
>>>>      self.x = x
>>>>      self.y = y
>>>>      self.obj =<Vector 2d*>x
>>>> ok. That is a bit nasty (I'm sure someone can tell me lots of reasons why I
>>>> shouldn't do this).
>>>>          
>>> In reading that code, I'm not sure what your intent is.
>>>
>>>        
>> Nastiness explained: Vector2d is a struct with 2 float members. I
>> declare 2 float members in the python object, and then get a pointer
>> to the first of these and cast it to a Vector2d. Assuming that the
>> struct layout is the same in both of these, then accessing members via
>> the cython properties will update the same shared bit of memory that
>> the c++ struct uses.
>>      
> If this is just for syntactic sugar, it sound way to messy... I could
> see simplifications of the properties syntax to make Python access
> easier.
>
> On 2 November 2010 20:02, Stefan Behnel<stefan_ml at behnel.de>  wrote:
>    
>> Robert Bradshaw, 02.11.2010 07:17:
>>      
>>> On Sun, Oct 31, 2010 at 8:44 PM, Brett Calcott wrote:
>>>        
>>>> cdef class PythonVector2d:
>>>>     cdef Vector obj
>>>>     properties:
>>>>        public obj.x as x
>>>>        public obj.y as y
>>>>          
> <--- snip -->
>    
>> For this case, I'd prefer a version entirely without body, maybe even
>>
>>      property x = obj.x
>>
>> although the "as" version would also work. However, it's not so clear from
>> the above that both setter and getter get generated, and how you would get
>> only the getter for read-only access, for example. And why not have a
>> "delattr" as well? May make sense in some cases.
>>      
> setters, getters, (and delattrs?) would be defined iff they make sense
> on the expression in question. One could imagine
>
>      property readonly obj.x as x
>
> or something like that. I'm not a fan of the = sign, as it's no where
> else used for declarations (it's always a statement), unless we do
> something like
>
>      public x = property(obj.x)
>
> which reads kind of like a default value.
>    

FWIW, I'm in favor of something Python-compatible like

cdef class A:
     cdef Vector obj
     x = delegate_property(obj, 'x')

or similar, without throwing in new syntax. But it may be too late and I 
don't care too much. Remember that "x = property(...)" is already in Python.

Dag Sverre

From markflorisson88 at gmail.com  Wed Nov  3 10:34:45 2010
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 3 Nov 2010 10:34:45 +0100
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTinNGspAzLoOBdoXcUZgxkxR86i4aQfS2FMrgVYR@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CCE989D.6070907@behnel.de>
	<AANLkTin+ofqt6-z+-Z-bnuSmW9eSDL94xDs=D=dT3ic7@mail.gmail.com>
	<AANLkTikn4VJtcJ3EsESd4iVKNpg42fQK2EAdbAbuBA1o@mail.gmail.com>
	<AANLkTinPTHtp=X483QUGsRO58tm2TTokSc02LxaEt3fP@mail.gmail.com>
	<AANLkTikESy8G5XWBi=8cvmO0nCJgQg_A9dQ=znrz-XuN@mail.gmail.com>
	<AANLkTino6OWKMbgG41Wvn1vo-jNP=KRvnX2H9ai31rv-@mail.gmail.com>
	<AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>
	<AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>
	<AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>
	<AANLkTikSi2Awwv31sx3A6BJy=WNSYROo+ceecoygQ0Fk@mail.gmail.com>
	<AANLkTi=bFQwdofb4TPcV53b9w0RtcshvC8Uiv2mJa9i-@mail.gmail.com>
	<4CD0FA29.7000201@behnel.de>
	<AANLkTinNGspAzLoOBdoXcUZgxkxR86i4aQfS2FMrgVYR@mail.gmail.com>
Message-ID: <AANLkTi=a8SMk+u7JfLKxujR3p2ZVnjLsSPjrMY8hufbt@mail.gmail.com>

On 3 November 2010 07:15, Robert Bradshaw <robertwb at math.washington.edu>wrote:

> On Tue, Nov 2, 2010 at 10:59 PM, Stefan Behnel <stefan_ml at behnel.de>
> wrote:
> > Robert Bradshaw, 02.11.2010 23:21:
> >> On Tue, Nov 2, 2010 at 3:11 PM, Vitja Makarov wrote:
> >>> http://docs.python.org/reference/datamodel.html#basic-customization
> >>>
> >>> """Called to create a new instance of class cls. __new__() is a static
> >>> method (special-cased so you need not declare it as such) that takes
> >>> the class of which an instance was requested as its first argument.
> >>> The remaining arguments are those passed to the object constructor
> >>> expression (the call to the class). The return value of __new__()
> >>> should be the new object instance (usually an instance of cls)."""
> >>
> >> I was actually thinking about cdef classes, where __new__ is much more
> >> subtle. I didn't realize it was a static method that takes the class
> >> as a first argument rather than just a class method.
> >
> > It needs to be to support inheritance. You usually call the __new__
> method
> > of the superclass to create the instance that you return. So you must be
> > able to pass the required class through the hierarchy explicitly.
>
> Good point, I hadn't thought about that.
>
> - Robert
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

It could also work with a classmethod, because you'd be using super(cls,
cls).__new__(), which also ensures that the subclass is passed as the first
argument to __new__ in superclasses. Of course, type(TheClass).__call__ will
pass the TheClass as the first argument, so if you make __new__ a
classmethod it will need to take the class as two arguments.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20101103/c50c4e15/attachment-0001.htm 

From markflorisson88 at gmail.com  Wed Nov  3 10:36:01 2010
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 3 Nov 2010 10:36:01 +0100
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTi=a8SMk+u7JfLKxujR3p2ZVnjLsSPjrMY8hufbt@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CCE989D.6070907@behnel.de>
	<AANLkTin+ofqt6-z+-Z-bnuSmW9eSDL94xDs=D=dT3ic7@mail.gmail.com>
	<AANLkTikn4VJtcJ3EsESd4iVKNpg42fQK2EAdbAbuBA1o@mail.gmail.com>
	<AANLkTinPTHtp=X483QUGsRO58tm2TTokSc02LxaEt3fP@mail.gmail.com>
	<AANLkTikESy8G5XWBi=8cvmO0nCJgQg_A9dQ=znrz-XuN@mail.gmail.com>
	<AANLkTino6OWKMbgG41Wvn1vo-jNP=KRvnX2H9ai31rv-@mail.gmail.com>
	<AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>
	<AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>
	<AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>
	<AANLkTikSi2Awwv31sx3A6BJy=WNSYROo+ceecoygQ0Fk@mail.gmail.com>
	<AANLkTi=bFQwdofb4TPcV53b9w0RtcshvC8Uiv2mJa9i-@mail.gmail.com>
	<4CD0FA29.7000201@behnel.de>
	<AANLkTinNGspAzLoOBdoXcUZgxkxR86i4aQfS2FMrgVYR@mail.gmail.com>
	<AANLkTi=a8SMk+u7JfLKxujR3p2ZVnjLsSPjrMY8hufbt@mail.gmail.com>
Message-ID: <AANLkTimjYc=4HdLE+VDmUyn0piXPWGZ7zWGqQ0QoabkP@mail.gmail.com>

err, that should be super(TheClass, cls), otherwise you make your class
unsubclassable.

On 3 November 2010 10:34, mark florisson <markflorisson88 at gmail.com> wrote:

>
>
> On 3 November 2010 07:15, Robert Bradshaw <robertwb at math.washington.edu>wrote:
>
>> On Tue, Nov 2, 2010 at 10:59 PM, Stefan Behnel <stefan_ml at behnel.de>
>> wrote:
>> > Robert Bradshaw, 02.11.2010 23:21:
>> >> On Tue, Nov 2, 2010 at 3:11 PM, Vitja Makarov wrote:
>> >>> http://docs.python.org/reference/datamodel.html#basic-customization
>> >>>
>> >>> """Called to create a new instance of class cls. __new__() is a static
>> >>> method (special-cased so you need not declare it as such) that takes
>> >>> the class of which an instance was requested as its first argument.
>> >>> The remaining arguments are those passed to the object constructor
>> >>> expression (the call to the class). The return value of __new__()
>> >>> should be the new object instance (usually an instance of cls)."""
>> >>
>> >> I was actually thinking about cdef classes, where __new__ is much more
>> >> subtle. I didn't realize it was a static method that takes the class
>> >> as a first argument rather than just a class method.
>> >
>> > It needs to be to support inheritance. You usually call the __new__
>> method
>> > of the superclass to create the instance that you return. So you must be
>> > able to pass the required class through the hierarchy explicitly.
>>
>> Good point, I hadn't thought about that.
>>
>> - Robert
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
> It could also work with a classmethod, because you'd be using super(cls,
> cls).__new__(), which also ensures that the subclass is passed as the first
> argument to __new__ in superclasses. Of course, type(TheClass).__call__ will
> pass the TheClass as the first argument, so if you make __new__ a
> classmethod it will need to take the class as two arguments.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20101103/7b06c175/attachment.htm 

From markflorisson88 at gmail.com  Wed Nov  3 10:55:39 2010
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 3 Nov 2010 10:55:39 +0100
Subject: [Cython] Cython debugger
In-Reply-To: <AANLkTikOF4j_7v-AzHFaT=vzp+RTYxEFOBGc1M8HC9Wz@mail.gmail.com>
References: <AANLkTinjhVuH=vqA=EFySaBKh+j6Rk1U-Gn0qDLLoUaB@mail.gmail.com>
	<4C8F3E56.2020705@student.matnat.uio.no>
	<1284482379.13330.333.camel@radiator.bos.redhat.com>
	<AANLkTimzNdwzsRo6vAymnMfAH+uBU7UG+OqPSF=Ugrp2@mail.gmail.com>
	<AANLkTikOF4j_7v-AzHFaT=vzp+RTYxEFOBGc1M8HC9Wz@mail.gmail.com>
Message-ID: <AANLkTi=WSoP5wkC9+0cmSwSCPfLAKFghO9OSwkSx+peb@mail.gmail.com>

Most of the Cython debugger is implemented, it can be pulled from
hg.cython.org/cython-gdb. To use it, you need Cython to export some debug
information, which can be done using  'cython --debug mymod.pyx', or 'python
setup.py build_ext --pyrex-debug'. You can also pass 'pyrex_debug=True' as
an argument to Cython.Distutils.extension.Extension.
You can then start gdb by typing 'cygdb'  (which should be installed as a
script) in your project directory. Then 'help cy' gives an overview of
commands. It currently supports breapoints, stepping, stepping-over,
backtraces, source code listing, going up and down the stack, printing
variables with regard to context, listing locals or globals and running and
continuing the program. If pygments is installed it will colorize source
code which is configurable through cy_* parameters.

It basically works on three levels: the Python, Cython and C level.
Depending on the context cygdb does the right thing. For stepping it
considers the following stack frames relevant: any Python frame, any Cython
frame and any C frame from a C function called directly by Cython user-code.

So, it would be great if some people could test and try it (unit tests are
written but some system testing is always great). It works with gdb 7.2 (the
7.1 python api was too incomplete and broken). So if you guys like it I
could write documentation that explains to Cython users how to install and
use it. Any suggestion or criticism is most welcome!

The Python support (libpython.py) was also modified. I'd like to push this
mainstream (and process any suggestions and criticism) before supplying a
patch to Python.

Kind regards,

Mark

On 15 September 2010 17:49, Robert Bradshaw <robertwb at math.washington.edu>wrote:

> On Wed, Sep 15, 2010 at 2:55 AM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>
> >> It ought to be possible to do something similar with cython code.  It
> >> may not even be necessary to modify cython: perhaps some searching for
> >> locals named "__pyx_*" iirc would get you 70% of the way there?
> >
> > Although that sounds like a wonderful idea, I think there are also
> > issues with that. One issue is that a user must be able to set Cython
> > breakpoints before the Cython module would be loaded, and for that the
> > symbol name would be needed beforehand. Also, I don't know if these
> > mangled names are consistent now and in the future and if you would be
> > able to unambiguously associate a Cython variable name with a mangled
> > name.
>
> Mangled names are deterministic and, though they're not guaranteed to
> be consistent from release to release, almost always are.
>
> >> I can attest that having the prettyprinters enabled does make it much
> >> easier to debug cython code: all of the PyObject* get prettyprinted.
> >
> > I've been looking at the code and this is pretty neat. I did encounter
> > some issues, for instance if you load the script before loading the
> > python interpreter you get this traceback because these types are not
> > defined at that time:
> >
> > Traceback (most recent call last):
> >  File "<string>", line 1, in <module>
> >  File ".libpython.py", line 49, in <module>
> >    _type_size_t = gdb.lookup_type('size_t')
> > RuntimeError: No type named size_t.
> >
> > So I think it would be a good idea to not make that code module-level.
> >
> >>
> >> One other thought: if it's possibly to expose the cython structures in
> >> some meaningful way, perhaps we could change upstream python's gdb hooks
> >> to simply integrate them into the py-* commands I mentioned above? (so
> >> e.g. cython c functions get somehow treated as python frames; currently
> >> I have a test predicate:
> >>  Frame.is_evalframeex()
> >> which perhaps could be generalized?)
> >>
> >> (Not sure; it would complicate the selftests within python itself)
> >>
> >
> > I think it would be hard to make them actual Python frames because
> > creating frames in the inferior process from gdb is probably quite
> > dangerous, and the alternative would be to modify Cython so that it
> > creates Python stack frames (this sounds feasible but I think it might
> > be a little bit of work). However, if this could be done (it would
> > only do so if this 'debug' flag is active), then tracebacks and locals
> > inspection etc wouldn't need special attention and the code would
> > appear as normal Python code (apart from the Code objects obviously).
> > However, this would form a problem for non-primitive C-type Cython
> > variables.
> >
> > So at the very least we could have a 'py-locals' or some such command
> > that would show the value of all the locals (the Python locals would
> > be printed by py-print and C locals by gdb print). For regular python
> > code it would show the locals from the current stack frame. For the
> > Cython part to work we would need information from the Cython compiler
> > because we wouldn't want to list any temporary or irrelevant
> > variables.
>
> Yes, this is what I was thinking, at least in terms of exposing stuff
> to pdb (which is a complementary project). BTW, frames are already
> created for functions when profiling is enabled.
>
> > So I think we should be able to integrate these two projects into one
> > fruitful project, and with proper documentation it could help both
> > regular Python users and Cython users.
>
> +1
>
> - Robert
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20101103/a4065665/attachment.htm 

From markflorisson88 at gmail.com  Wed Nov  3 13:06:54 2010
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 3 Nov 2010 13:06:54 +0100
Subject: [Cython] Cython debugger
In-Reply-To: <AANLkTi=WSoP5wkC9+0cmSwSCPfLAKFghO9OSwkSx+peb@mail.gmail.com>
References: <AANLkTinjhVuH=vqA=EFySaBKh+j6Rk1U-Gn0qDLLoUaB@mail.gmail.com>
	<4C8F3E56.2020705@student.matnat.uio.no>
	<1284482379.13330.333.camel@radiator.bos.redhat.com>
	<AANLkTimzNdwzsRo6vAymnMfAH+uBU7UG+OqPSF=Ugrp2@mail.gmail.com>
	<AANLkTikOF4j_7v-AzHFaT=vzp+RTYxEFOBGc1M8HC9Wz@mail.gmail.com>
	<AANLkTi=WSoP5wkC9+0cmSwSCPfLAKFghO9OSwkSx+peb@mail.gmail.com>
Message-ID: <AANLkTi=R6GJ4+onwG3DGZt2PKtDHmokV5BX0ifrknn0p@mail.gmail.com>

I'd also like to mention that cygdb is a lot more useful when using a python
debug build (or having compiled python with -g). Of course, extensions
should also be build with -g, which is the default I believe.

On 3 November 2010 10:55, mark florisson <markflorisson88 at gmail.com> wrote:

> Most of the Cython debugger is implemented, it can be pulled from
> hg.cython.org/cython-gdb. To use it, you need Cython to export some debug
> information, which can be done using  'cython --debug mymod.pyx', or 'python
> setup.py build_ext --pyrex-debug'. You can also pass 'pyrex_debug=True' as
> an argument to Cython.Distutils.extension.Extension.
> You can then start gdb by typing 'cygdb'  (which should be installed as a
> script) in your project directory. Then 'help cy' gives an overview of
> commands. It currently supports breapoints, stepping, stepping-over,
> backtraces, source code listing, going up and down the stack, printing
> variables with regard to context, listing locals or globals and running and
> continuing the program. If pygments is installed it will colorize source
> code which is configurable through cy_* parameters.
>
> It basically works on three levels: the Python, Cython and C level.
> Depending on the context cygdb does the right thing. For stepping it
> considers the following stack frames relevant: any Python frame, any Cython
> frame and any C frame from a C function called directly by Cython user-code.
>
> So, it would be great if some people could test and try it (unit tests are
> written but some system testing is always great). It works with gdb 7.2 (the
> 7.1 python api was too incomplete and broken). So if you guys like it I
> could write documentation that explains to Cython users how to install and
> use it. Any suggestion or criticism is most welcome!
>
> The Python support (libpython.py) was also modified. I'd like to push this
> mainstream (and process any suggestions and criticism) before supplying a
> patch to Python.
>
> Kind regards,
>
> Mark
>
>
> On 15 September 2010 17:49, Robert Bradshaw <robertwb at math.washington.edu>wrote:
>
>> On Wed, Sep 15, 2010 at 2:55 AM, mark florisson
>> <markflorisson88 at gmail.com> wrote:
>>
>> >> It ought to be possible to do something similar with cython code.  It
>> >> may not even be necessary to modify cython: perhaps some searching for
>> >> locals named "__pyx_*" iirc would get you 70% of the way there?
>> >
>> > Although that sounds like a wonderful idea, I think there are also
>> > issues with that. One issue is that a user must be able to set Cython
>> > breakpoints before the Cython module would be loaded, and for that the
>> > symbol name would be needed beforehand. Also, I don't know if these
>> > mangled names are consistent now and in the future and if you would be
>> > able to unambiguously associate a Cython variable name with a mangled
>> > name.
>>
>> Mangled names are deterministic and, though they're not guaranteed to
>> be consistent from release to release, almost always are.
>>
>> >> I can attest that having the prettyprinters enabled does make it much
>> >> easier to debug cython code: all of the PyObject* get prettyprinted.
>> >
>> > I've been looking at the code and this is pretty neat. I did encounter
>> > some issues, for instance if you load the script before loading the
>> > python interpreter you get this traceback because these types are not
>> > defined at that time:
>> >
>> > Traceback (most recent call last):
>> >  File "<string>", line 1, in <module>
>> >  File ".libpython.py", line 49, in <module>
>> >    _type_size_t = gdb.lookup_type('size_t')
>> > RuntimeError: No type named size_t.
>> >
>> > So I think it would be a good idea to not make that code module-level.
>> >
>> >>
>> >> One other thought: if it's possibly to expose the cython structures in
>> >> some meaningful way, perhaps we could change upstream python's gdb
>> hooks
>> >> to simply integrate them into the py-* commands I mentioned above? (so
>> >> e.g. cython c functions get somehow treated as python frames; currently
>> >> I have a test predicate:
>> >>  Frame.is_evalframeex()
>> >> which perhaps could be generalized?)
>> >>
>> >> (Not sure; it would complicate the selftests within python itself)
>> >>
>> >
>> > I think it would be hard to make them actual Python frames because
>> > creating frames in the inferior process from gdb is probably quite
>> > dangerous, and the alternative would be to modify Cython so that it
>> > creates Python stack frames (this sounds feasible but I think it might
>> > be a little bit of work). However, if this could be done (it would
>> > only do so if this 'debug' flag is active), then tracebacks and locals
>> > inspection etc wouldn't need special attention and the code would
>> > appear as normal Python code (apart from the Code objects obviously).
>> > However, this would form a problem for non-primitive C-type Cython
>> > variables.
>> >
>> > So at the very least we could have a 'py-locals' or some such command
>> > that would show the value of all the locals (the Python locals would
>> > be printed by py-print and C locals by gdb print). For regular python
>> > code it would show the locals from the current stack frame. For the
>> > Cython part to work we would need information from the Cython compiler
>> > because we wouldn't want to list any temporary or irrelevant
>> > variables.
>>
>> Yes, this is what I was thinking, at least in terms of exposing stuff
>> to pdb (which is a complementary project). BTW, frames are already
>> created for functions when profiling is enabled.
>>
>> > So I think we should be able to integrate these two projects into one
>> > fruitful project, and with proper documentation it could help both
>> > regular Python users and Cython users.
>>
>> +1
>>
>> - Robert
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20101103/1e0c46e9/attachment.htm 

From vinjvinj at gmail.com  Wed Nov  3 14:29:25 2010
From: vinjvinj at gmail.com (Vineet Jain)
Date: Wed, 3 Nov 2010 09:29:25 -0400
Subject: [Cython] Importing cython modules and sys.path errors.
Message-ID: <AANLkTikNSDObD80ScNF-1UHJX3Hncf_3w0FTpGu0TNuD@mail.gmail.com>

I have the following directory structure

/module1

/scripts/cythoncode/
    test1.pyx
    test1.pxd
    test2.pyx   [test2.pyx imports test1.pxd]


I can import the module in cython code. However, when I copy the *.so
files from cythoncode to /module1. The following command
    import module1.test2 as test2

raises an error saying that it cannot import test1.pxd. However, when
I add the directory /module1 to the sys.path everything works fine.

Can someone help me understand why import module1.test2 does not work?

Thanks,

Vineet

From vitja.makarov at gmail.com  Wed Nov  3 15:13:16 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Wed, 3 Nov 2010 17:13:16 +0300
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTimjYc=4HdLE+VDmUyn0piXPWGZ7zWGqQ0QoabkP@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CCE989D.6070907@behnel.de>
	<AANLkTin+ofqt6-z+-Z-bnuSmW9eSDL94xDs=D=dT3ic7@mail.gmail.com>
	<AANLkTikn4VJtcJ3EsESd4iVKNpg42fQK2EAdbAbuBA1o@mail.gmail.com>
	<AANLkTinPTHtp=X483QUGsRO58tm2TTokSc02LxaEt3fP@mail.gmail.com>
	<AANLkTikESy8G5XWBi=8cvmO0nCJgQg_A9dQ=znrz-XuN@mail.gmail.com>
	<AANLkTino6OWKMbgG41Wvn1vo-jNP=KRvnX2H9ai31rv-@mail.gmail.com>
	<AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>
	<AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>
	<AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>
	<AANLkTikSi2Awwv31sx3A6BJy=WNSYROo+ceecoygQ0Fk@mail.gmail.com>
	<AANLkTi=bFQwdofb4TPcV53b9w0RtcshvC8Uiv2mJa9i-@mail.gmail.com>
	<4CD0FA29.7000201@behnel.de>
	<AANLkTinNGspAzLoOBdoXcUZgxkxR86i4aQfS2FMrgVYR@mail.gmail.com>
	<AANLkTi=a8SMk+u7JfLKxujR3p2ZVnjLsSPjrMY8hufbt@mail.gmail.com>
	<AANLkTimjYc=4HdLE+VDmUyn0piXPWGZ7zWGqQ0QoabkP@mail.gmail.com>
Message-ID: <AANLkTim0dL=7nR6WiXAYQyMUSyQ=_b7jsBuKCQ8zh-WT@mail.gmail.com>

I've fixed classmethod issue patch is in trac.

2010/11/3 mark florisson <markflorisson88 at gmail.com>:
> err, that should be super(TheClass, cls), otherwise you make your class
> unsubclassable.
>
> On 3 November 2010 10:34, mark florisson <markflorisson88 at gmail.com> wrote:
>>
>>
>> On 3 November 2010 07:15, Robert Bradshaw <robertwb at math.washington.edu>
>> wrote:
>>>
>>> On Tue, Nov 2, 2010 at 10:59 PM, Stefan Behnel <stefan_ml at behnel.de>
>>> wrote:
>>> > Robert Bradshaw, 02.11.2010 23:21:
>>> >> On Tue, Nov 2, 2010 at 3:11 PM, Vitja Makarov wrote:
>>> >>> http://docs.python.org/reference/datamodel.html#basic-customization
>>> >>>
>>> >>> """Called to create a new instance of class cls. __new__() is a
>>> >>> static
>>> >>> method (special-cased so you need not declare it as such) that takes
>>> >>> the class of which an instance was requested as its first argument.
>>> >>> The remaining arguments are those passed to the object constructor
>>> >>> expression (the call to the class). The return value of __new__()
>>> >>> should be the new object instance (usually an instance of cls)."""
>>> >>
>>> >> I was actually thinking about cdef classes, where __new__ is much more
>>> >> subtle. I didn't realize it was a static method that takes the class
>>> >> as a first argument rather than just a class method.
>>> >
>>> > It needs to be to support inheritance. You usually call the __new__
>>> > method
>>> > of the superclass to create the instance that you return. So you must
>>> > be
>>> > able to pass the required class through the hierarchy explicitly.
>>>
>>> Good point, I hadn't thought about that.
>>>
>>> - Robert
>>> _______________________________________________
>>> Cython-dev mailing list
>>> Cython-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>> It could also work with a classmethod, because you'd be using super(cls,
>> cls).__new__(), which also ensures that the subclass is passed as the first
>> argument to __new__ in superclasses. Of course, type(TheClass).__call__ will
>> pass the TheClass as the first argument, so if you make __new__ a
>> classmethod it will need to take the class as two arguments.
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>

From robertwb at math.washington.edu  Wed Nov  3 17:45:19 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 3 Nov 2010 09:45:19 -0700
Subject: [Cython] Importing cython modules and sys.path errors.
In-Reply-To: <AANLkTikNSDObD80ScNF-1UHJX3Hncf_3w0FTpGu0TNuD@mail.gmail.com>
References: <AANLkTikNSDObD80ScNF-1UHJX3Hncf_3w0FTpGu0TNuD@mail.gmail.com>
Message-ID: <AANLkTimug_kdsZW1hCeNfQH8oXTeAJKMyKwo4wo6KF8V@mail.gmail.com>

On Wed, Nov 3, 2010 at 6:29 AM, Vineet Jain <vinjvinj at gmail.com> wrote:
> I have the following directory structure
>
> /module1
>
> /scripts/cythoncode/
> ? ?test1.pyx
> ? ?test1.pxd
> ? ?test2.pyx ? [test2.pyx imports test1.pxd]
>
>
> I can import the module in cython code. However, when I copy the *.so
> files from cythoncode to /module1. The following command
> ? ?import module1.test2 as test2
>
> raises an error saying that it cannot import test1.pxd. However, when
> I add the directory /module1 to the sys.path everything works fine.
>
> Can someone help me understand why import module1.test2 does not work?

To do its static analysis, Cython resolves cimported paths to absolute
imports at compile time. It is recommended to simply put your .pyx and
.pxd files in the same folders as your .py files. (Simpler as well
than maintaining two source trees...) An alternative would be to make
a module1 inside your cythoncode directory.

- Robert

From robertwb at math.washington.edu  Wed Nov  3 18:00:39 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 3 Nov 2010 10:00:39 -0700
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTim0dL=7nR6WiXAYQyMUSyQ=_b7jsBuKCQ8zh-WT@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CCE989D.6070907@behnel.de>
	<AANLkTin+ofqt6-z+-Z-bnuSmW9eSDL94xDs=D=dT3ic7@mail.gmail.com>
	<AANLkTikn4VJtcJ3EsESd4iVKNpg42fQK2EAdbAbuBA1o@mail.gmail.com>
	<AANLkTinPTHtp=X483QUGsRO58tm2TTokSc02LxaEt3fP@mail.gmail.com>
	<AANLkTikESy8G5XWBi=8cvmO0nCJgQg_A9dQ=znrz-XuN@mail.gmail.com>
	<AANLkTino6OWKMbgG41Wvn1vo-jNP=KRvnX2H9ai31rv-@mail.gmail.com>
	<AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>
	<AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>
	<AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>
	<AANLkTikSi2Awwv31sx3A6BJy=WNSYROo+ceecoygQ0Fk@mail.gmail.com>
	<AANLkTi=bFQwdofb4TPcV53b9w0RtcshvC8Uiv2mJa9i-@mail.gmail.com>
	<4CD0FA29.7000201@behnel.de>
	<AANLkTinNGspAzLoOBdoXcUZgxkxR86i4aQfS2FMrgVYR@mail.gmail.com>
	<AANLkTi=a8SMk+u7JfLKxujR3p2ZVnjLsSPjrMY8hufbt@mail.gmail.com>
	<AANLkTimjYc=4HdLE+VDmUyn0piXPWGZ7zWGqQ0QoabkP@mail.gmail.com>
	<AANLkTim0dL=7nR6WiXAYQyMUSyQ=_b7jsBuKCQ8zh-WT@mail.gmail.com>
Message-ID: <AANLkTim6qdY6hmj2VVJJ_rwje5Sup5NQjspZuymVEv8Z@mail.gmail.com>

On Wed, Nov 3, 2010 at 7:13 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> I've fixed classmethod issue patch is in trac.

Very cool. Looks like I only apply the last patch, right?

- Robert

From robertwb at math.washington.edu  Wed Nov  3 17:57:43 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 3 Nov 2010 09:57:43 -0700
Subject: [Cython] Cython debugger
In-Reply-To: <AANLkTi=WSoP5wkC9+0cmSwSCPfLAKFghO9OSwkSx+peb@mail.gmail.com>
References: <AANLkTinjhVuH=vqA=EFySaBKh+j6Rk1U-Gn0qDLLoUaB@mail.gmail.com>
	<4C8F3E56.2020705@student.matnat.uio.no>
	<1284482379.13330.333.camel@radiator.bos.redhat.com>
	<AANLkTimzNdwzsRo6vAymnMfAH+uBU7UG+OqPSF=Ugrp2@mail.gmail.com>
	<AANLkTikOF4j_7v-AzHFaT=vzp+RTYxEFOBGc1M8HC9Wz@mail.gmail.com>
	<AANLkTi=WSoP5wkC9+0cmSwSCPfLAKFghO9OSwkSx+peb@mail.gmail.com>
Message-ID: <AANLkTikrH31rvjyzjS+BWoSnkOZ+x3PUeg2ufqm4c2Wk@mail.gmail.com>

On Wed, Nov 3, 2010 at 2:55 AM, mark florisson
<markflorisson88 at gmail.com> wrote:
> Most of the Cython debugger is implemented, it can be pulled from
> hg.cython.org/cython-gdb.

Cool! Have you recently pulled from main?

> To use it, you need Cython to export some debug
> information, which can be done using ?'cython --debug mymod.pyx', or 'python
> setup.py build_ext --pyrex-debug'. You can also pass 'pyrex_debug=True' as
> an argument to Cython.Distutils.extension.Extension.

How much overhead (runtime, codesize, extra files, etc.) does
compiling with this option incur? Too much to enable by default?

> You can then start gdb by typing 'cygdb'??(which should be installed as a
> script)?in your project directory. Then 'help cy' gives an overview of
> commands. It currently supports breapoints, stepping, stepping-over,
> backtraces, source code listing, going up and down the stack, printing
> variables with regard to context, listing locals or globals and running and
> continuing the program. If pygments is installed it will colorize source
> code which is configurable through cy_* parameters.
> It basically works on three levels: the Python, Cython and C level.
> Depending on the context cygdb does the right thing. For stepping it
> considers the following stack frames relevant: any Python frame, any Cython
> frame and any C frame from a C function called directly by Cython user-code.
> So, it would be great if some people could test and try it (unit tests are
> written but some system testing is always great). It works with gdb 7.2 (the
> 7.1 python api was too incomplete and broken). So if you guys like it I
> could write documentation that explains to Cython users how to install and
> use it.

I'm not a gdb expert, but I definitely want to try this out.
Documentation would be great--either as a wiki or, preferably, the
main documentation (with a little note mentioning it's pending
release). I've given you push access to
http://hg.cython.org/cython-docs

> Any suggestion or criticism is most welcome!
> The Python support (libpython.py) was also modified. I'd like to push this
> mainstream (and process any suggestions and criticism) before supplying a
> patch to Python.
> Kind regards,
> Mark
>
> On 15 September 2010 17:49, Robert Bradshaw <robertwb at math.washington.edu>
> wrote:
>>
>> On Wed, Sep 15, 2010 at 2:55 AM, mark florisson
>> <markflorisson88 at gmail.com> wrote:
>>
>> >> It ought to be possible to do something similar with cython code. ?It
>> >> may not even be necessary to modify cython: perhaps some searching for
>> >> locals named "__pyx_*" iirc would get you 70% of the way there?
>> >
>> > Although that sounds like a wonderful idea, I think there are also
>> > issues with that. One issue is that a user must be able to set Cython
>> > breakpoints before the Cython module would be loaded, and for that the
>> > symbol name would be needed beforehand. Also, I don't know if these
>> > mangled names are consistent now and in the future and if you would be
>> > able to unambiguously associate a Cython variable name with a mangled
>> > name.
>>
>> Mangled names are deterministic and, though they're not guaranteed to
>> be consistent from release to release, almost always are.
>>
>> >> I can attest that having the prettyprinters enabled does make it much
>> >> easier to debug cython code: all of the PyObject* get prettyprinted.
>> >
>> > I've been looking at the code and this is pretty neat. I did encounter
>> > some issues, for instance if you load the script before loading the
>> > python interpreter you get this traceback because these types are not
>> > defined at that time:
>> >
>> > Traceback (most recent call last):
>> > ?File "<string>", line 1, in <module>
>> > ?File ".libpython.py", line 49, in <module>
>> > ? ?_type_size_t = gdb.lookup_type('size_t')
>> > RuntimeError: No type named size_t.
>> >
>> > So I think it would be a good idea to not make that code module-level.
>> >
>> >>
>> >> One other thought: if it's possibly to expose the cython structures in
>> >> some meaningful way, perhaps we could change upstream python's gdb
>> >> hooks
>> >> to simply integrate them into the py-* commands I mentioned above? (so
>> >> e.g. cython c functions get somehow treated as python frames; currently
>> >> I have a test predicate:
>> >> ?Frame.is_evalframeex()
>> >> which perhaps could be generalized?)
>> >>
>> >> (Not sure; it would complicate the selftests within python itself)
>> >>
>> >
>> > I think it would be hard to make them actual Python frames because
>> > creating frames in the inferior process from gdb is probably quite
>> > dangerous, and the alternative would be to modify Cython so that it
>> > creates Python stack frames (this sounds feasible but I think it might
>> > be a little bit of work). However, if this could be done (it would
>> > only do so if this 'debug' flag is active), then tracebacks and locals
>> > inspection etc wouldn't need special attention and the code would
>> > appear as normal Python code (apart from the Code objects obviously).
>> > However, this would form a problem for non-primitive C-type Cython
>> > variables.
>> >
>> > So at the very least we could have a 'py-locals' or some such command
>> > that would show the value of all the locals (the Python locals would
>> > be printed by py-print and C locals by gdb print). For regular python
>> > code it would show the locals from the current stack frame. For the
>> > Cython part to work we would need information from the Cython compiler
>> > because we wouldn't want to list any temporary or irrelevant
>> > variables.
>>
>> Yes, this is what I was thinking, at least in terms of exposing stuff
>> to pdb (which is a complementary project). BTW, frames are already
>> created for functions when profiling is enabled.
>>
>> > So I think we should be able to integrate these two projects into one
>> > fruitful project, and with proper documentation it could help both
>> > regular Python users and Cython users.
>>
>> +1
>>
>> - Robert
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>

From robertwb at math.washington.edu  Wed Nov  3 18:13:16 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 3 Nov 2010 10:13:16 -0700
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTim6qdY6hmj2VVJJ_rwje5Sup5NQjspZuymVEv8Z@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CCE989D.6070907@behnel.de>
	<AANLkTin+ofqt6-z+-Z-bnuSmW9eSDL94xDs=D=dT3ic7@mail.gmail.com>
	<AANLkTikn4VJtcJ3EsESd4iVKNpg42fQK2EAdbAbuBA1o@mail.gmail.com>
	<AANLkTinPTHtp=X483QUGsRO58tm2TTokSc02LxaEt3fP@mail.gmail.com>
	<AANLkTikESy8G5XWBi=8cvmO0nCJgQg_A9dQ=znrz-XuN@mail.gmail.com>
	<AANLkTino6OWKMbgG41Wvn1vo-jNP=KRvnX2H9ai31rv-@mail.gmail.com>
	<AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>
	<AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>
	<AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>
	<AANLkTikSi2Awwv31sx3A6BJy=WNSYROo+ceecoygQ0Fk@mail.gmail.com>
	<AANLkTi=bFQwdofb4TPcV53b9w0RtcshvC8Uiv2mJa9i-@mail.gmail.com>
	<4CD0FA29.7000201@behnel.de>
	<AANLkTinNGspAzLoOBdoXcUZgxkxR86i4aQfS2FMrgVYR@mail.gmail.com>
	<AANLkTi=a8SMk+u7JfLKxujR3p2ZVnjLsSPjrMY8hufbt@mail.gmail.com>
	<AANLkTimjYc=4HdLE+VDmUyn0piXPWGZ7zWGqQ0QoabkP@mail.gmail.com>
	<AANLkTim0dL=7nR6WiXAYQyMUSyQ=_b7jsBuKCQ8zh-WT@mail.gmail.com>
	<AANLkTim6qdY6hmj2VVJJ_rwje5Sup5NQjspZuymVEv8Z@mail.gmail.com>
Message-ID: <AANLkTimWfQO1SsVgDX9yyex=xivq0z7nmw0PJoJBXvFc@mail.gmail.com>

On Wed, Nov 3, 2010 at 10:00 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Wed, Nov 3, 2010 at 7:13 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> I've fixed classmethod issue patch is in trac.
>
> Very cool. Looks like I only apply the last patch, right?

Sweet. All looks well. I've merged it.

- Robert

From markflorisson88 at gmail.com  Wed Nov  3 18:17:55 2010
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 3 Nov 2010 18:17:55 +0100
Subject: [Cython] Cython debugger
In-Reply-To: <AANLkTikrH31rvjyzjS+BWoSnkOZ+x3PUeg2ufqm4c2Wk@mail.gmail.com>
References: <AANLkTinjhVuH=vqA=EFySaBKh+j6Rk1U-Gn0qDLLoUaB@mail.gmail.com>
	<4C8F3E56.2020705@student.matnat.uio.no>
	<1284482379.13330.333.camel@radiator.bos.redhat.com>
	<AANLkTimzNdwzsRo6vAymnMfAH+uBU7UG+OqPSF=Ugrp2@mail.gmail.com>
	<AANLkTikOF4j_7v-AzHFaT=vzp+RTYxEFOBGc1M8HC9Wz@mail.gmail.com>
	<AANLkTi=WSoP5wkC9+0cmSwSCPfLAKFghO9OSwkSx+peb@mail.gmail.com>
	<AANLkTikrH31rvjyzjS+BWoSnkOZ+x3PUeg2ufqm4c2Wk@mail.gmail.com>
Message-ID: <AANLkTinVmXQuuaXA1PSFda+J5W=46n9c-t55zasuj7Kf@mail.gmail.com>

On 3 November 2010 17:57, Robert Bradshaw <robertwb at math.washington.edu>wrote:

> On Wed, Nov 3, 2010 at 2:55 AM, mark florisson
> <markflorisson88 at gmail.com> wrote:
> > Most of the Cython debugger is implemented, it can be pulled from
> > hg.cython.org/cython-gdb.
>
> Cool! Have you recently pulled from main?


Nope, I can try to pull and merge though if you want.


> > To use it, you need Cython to export some debug
> > information, which can be done using  'cython --debug mymod.pyx', or
> 'python
> > setup.py build_ext --pyrex-debug'. You can also pass 'pyrex_debug=True'
> as
> > an argument to Cython.Distutils.extension.Extension.
>
> How much overhead (runtime, codesize, extra files, etc.) does
> compiling with this option incur? Too much to enable by default?


It writes debug information to the 'cython_debug' subdirectory of the build
directory (so build_ext --inplace --pyrex-debug makes for a good combination
of command line options to setup.py). It outputs information in XML so it
needs an etree implementation (either from the xml.etree package, or from
lxml or the regular elementtree package). I believe xml.etree.ElementTree is
new in python 2.5.

In any case, I don't notice any difference in time when compiling with debug
information, but it's xml and it has to list the association between C line
numbers and Cython line numbers. A quick test for a project which generates
8M of C code and ELF binaries has about 1M of debug information. So I'd say
it's not really a bottleneck.


> > You can then start gdb by typing 'cygdb'  (which should be installed as a
> > script) in your project directory. Then 'help cy' gives an overview of
> > commands. It currently supports breapoints, stepping, stepping-over,
> > backtraces, source code listing, going up and down the stack, printing
> > variables with regard to context, listing locals or globals and running
> and
> > continuing the program. If pygments is installed it will colorize source
> > code which is configurable through cy_* parameters.
> > It basically works on three levels: the Python, Cython and C level.
> > Depending on the context cygdb does the right thing. For stepping it
> > considers the following stack frames relevant: any Python frame, any
> Cython
> > frame and any C frame from a C function called directly by Cython
> user-code.
> > So, it would be great if some people could test and try it (unit tests
> are
> > written but some system testing is always great). It works with gdb 7.2
> (the
> > 7.1 python api was too incomplete and broken). So if you guys like it I
> > could write documentation that explains to Cython users how to install
> and
> > use it.
>
> I'm not a gdb expert, but I definitely want to try this out.
> Documentation would be great--either as a wiki or, preferably, the
> main documentation (with a little note mentioning it's pending
> release). I've given you push access to
> http://hg.cython.org/cython-docs


Great! I'll get on it next week.


> > Any suggestion or criticism is most welcome!
> > The Python support (libpython.py) was also modified. I'd like to push
> this
> > mainstream (and process any suggestions and criticism) before supplying a
> > patch to Python.
> > Kind regards,
> > Mark
> >
> > On 15 September 2010 17:49, Robert Bradshaw <
> robertwb at math.washington.edu>
> > wrote:
> >>
> >> On Wed, Sep 15, 2010 at 2:55 AM, mark florisson
> >> <markflorisson88 at gmail.com> wrote:
> >>
> >> >> It ought to be possible to do something similar with cython code.  It
> >> >> may not even be necessary to modify cython: perhaps some searching
> for
> >> >> locals named "__pyx_*" iirc would get you 70% of the way there?
> >> >
> >> > Although that sounds like a wonderful idea, I think there are also
> >> > issues with that. One issue is that a user must be able to set Cython
> >> > breakpoints before the Cython module would be loaded, and for that the
> >> > symbol name would be needed beforehand. Also, I don't know if these
> >> > mangled names are consistent now and in the future and if you would be
> >> > able to unambiguously associate a Cython variable name with a mangled
> >> > name.
> >>
> >> Mangled names are deterministic and, though they're not guaranteed to
> >> be consistent from release to release, almost always are.
> >>
> >> >> I can attest that having the prettyprinters enabled does make it much
> >> >> easier to debug cython code: all of the PyObject* get prettyprinted.
> >> >
> >> > I've been looking at the code and this is pretty neat. I did encounter
> >> > some issues, for instance if you load the script before loading the
> >> > python interpreter you get this traceback because these types are not
> >> > defined at that time:
> >> >
> >> > Traceback (most recent call last):
> >> >  File "<string>", line 1, in <module>
> >> >  File ".libpython.py", line 49, in <module>
> >> >    _type_size_t = gdb.lookup_type('size_t')
> >> > RuntimeError: No type named size_t.
> >> >
> >> > So I think it would be a good idea to not make that code module-level.
> >> >
> >> >>
> >> >> One other thought: if it's possibly to expose the cython structures
> in
> >> >> some meaningful way, perhaps we could change upstream python's gdb
> >> >> hooks
> >> >> to simply integrate them into the py-* commands I mentioned above?
> (so
> >> >> e.g. cython c functions get somehow treated as python frames;
> currently
> >> >> I have a test predicate:
> >> >>  Frame.is_evalframeex()
> >> >> which perhaps could be generalized?)
> >> >>
> >> >> (Not sure; it would complicate the selftests within python itself)
> >> >>
> >> >
> >> > I think it would be hard to make them actual Python frames because
> >> > creating frames in the inferior process from gdb is probably quite
> >> > dangerous, and the alternative would be to modify Cython so that it
> >> > creates Python stack frames (this sounds feasible but I think it might
> >> > be a little bit of work). However, if this could be done (it would
> >> > only do so if this 'debug' flag is active), then tracebacks and locals
> >> > inspection etc wouldn't need special attention and the code would
> >> > appear as normal Python code (apart from the Code objects obviously).
> >> > However, this would form a problem for non-primitive C-type Cython
> >> > variables.
> >> >
> >> > So at the very least we could have a 'py-locals' or some such command
> >> > that would show the value of all the locals (the Python locals would
> >> > be printed by py-print and C locals by gdb print). For regular python
> >> > code it would show the locals from the current stack frame. For the
> >> > Cython part to work we would need information from the Cython compiler
> >> > because we wouldn't want to list any temporary or irrelevant
> >> > variables.
> >>
> >> Yes, this is what I was thinking, at least in terms of exposing stuff
> >> to pdb (which is a complementary project). BTW, frames are already
> >> created for functions when profiling is enabled.
> >>
> >> > So I think we should be able to integrate these two projects into one
> >> > fruitful project, and with proper documentation it could help both
> >> > regular Python users and Cython users.
> >>
> >> +1
> >>
> >> - Robert
> >> _______________________________________________
> >> Cython-dev mailing list
> >> Cython-dev at codespeak.net
> >> http://codespeak.net/mailman/listinfo/cython-dev
> >
> >
> > _______________________________________________
> > Cython-dev mailing list
> > Cython-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/cython-dev
> >
> >
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20101103/b8e61bcd/attachment.htm 

From robertwb at math.washington.edu  Wed Nov  3 18:18:23 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 3 Nov 2010 10:18:23 -0700
Subject: [Cython] Cython 0.13.1
Message-ID: <AANLkTin0LA4pjJfDMogXYYqKxOT9HHrANsFQtnHAvsp-@mail.gmail.com>

I think it's time we start thinking about rolling out another release.
I've got a couple of experimental features in the hopper I'd like to
get out, we've started to merge some of Haoyu's code (perhaps there's
more that should go in), got __metaclass__ support thanks to Vitja,
some cool gdb support from Mark (that I haven't had a chance to look
at, but I've no reason to doubt it's ready or could be soon, and a
bunch of other bugfixes/minor features. How does everyone feel about a
target date of this time next month? This one shouldn't drag on--for
one thing Sage is still in good shape:
https://sage.math.washington.edu:8091/hudson/job/sage-tests/ Any
specific features people want to be sure to get in?

- Robert

From vitja.makarov at gmail.com  Wed Nov  3 18:40:21 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Wed, 3 Nov 2010 20:40:21 +0300
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTimJ0HEdke56LJgjxUpm23KUntpSE5rSaBUA0vXx@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CCE989D.6070907@behnel.de>
	<AANLkTin+ofqt6-z+-Z-bnuSmW9eSDL94xDs=D=dT3ic7@mail.gmail.com>
	<AANLkTikn4VJtcJ3EsESd4iVKNpg42fQK2EAdbAbuBA1o@mail.gmail.com>
	<AANLkTinPTHtp=X483QUGsRO58tm2TTokSc02LxaEt3fP@mail.gmail.com>
	<AANLkTikESy8G5XWBi=8cvmO0nCJgQg_A9dQ=znrz-XuN@mail.gmail.com>
	<AANLkTino6OWKMbgG41Wvn1vo-jNP=KRvnX2H9ai31rv-@mail.gmail.com>
	<AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>
	<AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>
	<AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>
	<AANLkTikSi2Awwv31sx3A6BJy=WNSYROo+ceecoygQ0Fk@mail.gmail.com>
	<AANLkTi=bFQwdofb4TPcV53b9w0RtcshvC8Uiv2mJa9i-@mail.gmail.com>
	<4CD0FA29.7000201@behnel.de>
	<AANLkTinNGspAzLoOBdoXcUZgxkxR86i4aQfS2FMrgVYR@mail.gmail.com>
	<AANLkTi=a8SMk+u7JfLKxujR3p2ZVnjLsSPjrMY8hufbt@mail.gmail.com>
	<AANLkTimjYc=4HdLE+VDmUyn0piXPWGZ7zWGqQ0QoabkP@mail.gmail.com>
	<AANLkTim0dL=7nR6WiXAYQyMUSyQ=_b7jsBuKCQ8zh-WT@mail.gmail.com>
	<AANLkTim6qdY6hmj2VVJJ_rwje5Sup5NQjspZuymVEv8Z@mail.gmail.com>
	<AANLkTimWfQO1SsVgDX9yyex=xivq0z7nmw0PJoJBXvFc@mail.gmail.com>
	<AANLkTimJ0HEdke56LJgjxUpm23KUntpSE5rSaBUA0vXx@mail.gmail.com>
Message-ID: <AANLkTikpzSLL0F+8OoQyibHH60n0Fzh2mQcvzwGOtMNv@mail.gmail.com>

Yes, but global metaclasses and py3 isnt supported... Btw its better than
nothing ;)

03.11.2010 20:13 ???????????? "Robert Bradshaw" <
robertwb at math.washington.edu> ???????:

On Wed, Nov 3, 2010 at 10:00 AM, Robert Bradshaw

<robertwb at math.washington.edu> wrote:

> On Wed, Nov 3, 2010 at 7:13 AM, Vitja Makarov <vitja.makarov at gmail.com>
wrote:
>> I've fixed class...
Sweet. All looks well. I've merged it.


- Robert
_______________________________________________
Cython-dev mailing list
Cython-dev at codespe...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20101103/675905f8/attachment.htm 

From markflorisson88 at gmail.com  Wed Nov  3 18:43:54 2010
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 3 Nov 2010 18:43:54 +0100
Subject: [Cython] Cython debugger
In-Reply-To: <AANLkTikrH31rvjyzjS+BWoSnkOZ+x3PUeg2ufqm4c2Wk@mail.gmail.com>
References: <AANLkTinjhVuH=vqA=EFySaBKh+j6Rk1U-Gn0qDLLoUaB@mail.gmail.com>
	<4C8F3E56.2020705@student.matnat.uio.no>
	<1284482379.13330.333.camel@radiator.bos.redhat.com>
	<AANLkTimzNdwzsRo6vAymnMfAH+uBU7UG+OqPSF=Ugrp2@mail.gmail.com>
	<AANLkTikOF4j_7v-AzHFaT=vzp+RTYxEFOBGc1M8HC9Wz@mail.gmail.com>
	<AANLkTi=WSoP5wkC9+0cmSwSCPfLAKFghO9OSwkSx+peb@mail.gmail.com>
	<AANLkTikrH31rvjyzjS+BWoSnkOZ+x3PUeg2ufqm4c2Wk@mail.gmail.com>
Message-ID: <AANLkTik4WeanKnibsH8PnN5FfWRvCAN2z4f9pR5+APVb@mail.gmail.com>

Performed a clean merge with cython-devel.

On 3 November 2010 17:57, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Wed, Nov 3, 2010 at 2:55 AM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>> Most of the Cython debugger is implemented, it can be pulled from
>> hg.cython.org/cython-gdb.
>
> Cool! Have you recently pulled from main?
>
>> To use it, you need Cython to export some debug
>> information, which can be done using ?'cython --debug mymod.pyx', or 'python
>> setup.py build_ext --pyrex-debug'. You can also pass 'pyrex_debug=True' as
>> an argument to Cython.Distutils.extension.Extension.
>
> How much overhead (runtime, codesize, extra files, etc.) does
> compiling with this option incur? Too much to enable by default?
>
>> You can then start gdb by typing 'cygdb'??(which should be installed as a
>> script)?in your project directory. Then 'help cy' gives an overview of
>> commands. It currently supports breapoints, stepping, stepping-over,
>> backtraces, source code listing, going up and down the stack, printing
>> variables with regard to context, listing locals or globals and running and
>> continuing the program. If pygments is installed it will colorize source
>> code which is configurable through cy_* parameters.
>> It basically works on three levels: the Python, Cython and C level.
>> Depending on the context cygdb does the right thing. For stepping it
>> considers the following stack frames relevant: any Python frame, any Cython
>> frame and any C frame from a C function called directly by Cython user-code.
>> So, it would be great if some people could test and try it (unit tests are
>> written but some system testing is always great). It works with gdb 7.2 (the
>> 7.1 python api was too incomplete and broken). So if you guys like it I
>> could write documentation that explains to Cython users how to install and
>> use it.
>
> I'm not a gdb expert, but I definitely want to try this out.
> Documentation would be great--either as a wiki or, preferably, the
> main documentation (with a little note mentioning it's pending
> release). I've given you push access to
> http://hg.cython.org/cython-docs
>
>> Any suggestion or criticism is most welcome!
>> The Python support (libpython.py) was also modified. I'd like to push this
>> mainstream (and process any suggestions and criticism) before supplying a
>> patch to Python.
>> Kind regards,
>> Mark
>>
>> On 15 September 2010 17:49, Robert Bradshaw <robertwb at math.washington.edu>
>> wrote:
>>>
>>> On Wed, Sep 15, 2010 at 2:55 AM, mark florisson
>>> <markflorisson88 at gmail.com> wrote:
>>>
>>> >> It ought to be possible to do something similar with cython code. ?It
>>> >> may not even be necessary to modify cython: perhaps some searching for
>>> >> locals named "__pyx_*" iirc would get you 70% of the way there?
>>> >
>>> > Although that sounds like a wonderful idea, I think there are also
>>> > issues with that. One issue is that a user must be able to set Cython
>>> > breakpoints before the Cython module would be loaded, and for that the
>>> > symbol name would be needed beforehand. Also, I don't know if these
>>> > mangled names are consistent now and in the future and if you would be
>>> > able to unambiguously associate a Cython variable name with a mangled
>>> > name.
>>>
>>> Mangled names are deterministic and, though they're not guaranteed to
>>> be consistent from release to release, almost always are.
>>>
>>> >> I can attest that having the prettyprinters enabled does make it much
>>> >> easier to debug cython code: all of the PyObject* get prettyprinted.
>>> >
>>> > I've been looking at the code and this is pretty neat. I did encounter
>>> > some issues, for instance if you load the script before loading the
>>> > python interpreter you get this traceback because these types are not
>>> > defined at that time:
>>> >
>>> > Traceback (most recent call last):
>>> > ?File "<string>", line 1, in <module>
>>> > ?File ".libpython.py", line 49, in <module>
>>> > ? ?_type_size_t = gdb.lookup_type('size_t')
>>> > RuntimeError: No type named size_t.
>>> >
>>> > So I think it would be a good idea to not make that code module-level.
>>> >
>>> >>
>>> >> One other thought: if it's possibly to expose the cython structures in
>>> >> some meaningful way, perhaps we could change upstream python's gdb
>>> >> hooks
>>> >> to simply integrate them into the py-* commands I mentioned above? (so
>>> >> e.g. cython c functions get somehow treated as python frames; currently
>>> >> I have a test predicate:
>>> >> ?Frame.is_evalframeex()
>>> >> which perhaps could be generalized?)
>>> >>
>>> >> (Not sure; it would complicate the selftests within python itself)
>>> >>
>>> >
>>> > I think it would be hard to make them actual Python frames because
>>> > creating frames in the inferior process from gdb is probably quite
>>> > dangerous, and the alternative would be to modify Cython so that it
>>> > creates Python stack frames (this sounds feasible but I think it might
>>> > be a little bit of work). However, if this could be done (it would
>>> > only do so if this 'debug' flag is active), then tracebacks and locals
>>> > inspection etc wouldn't need special attention and the code would
>>> > appear as normal Python code (apart from the Code objects obviously).
>>> > However, this would form a problem for non-primitive C-type Cython
>>> > variables.
>>> >
>>> > So at the very least we could have a 'py-locals' or some such command
>>> > that would show the value of all the locals (the Python locals would
>>> > be printed by py-print and C locals by gdb print). For regular python
>>> > code it would show the locals from the current stack frame. For the
>>> > Cython part to work we would need information from the Cython compiler
>>> > because we wouldn't want to list any temporary or irrelevant
>>> > variables.
>>>
>>> Yes, this is what I was thinking, at least in terms of exposing stuff
>>> to pdb (which is a complementary project). BTW, frames are already
>>> created for functions when profiling is enabled.
>>>
>>> > So I think we should be able to integrate these two projects into one
>>> > fruitful project, and with proper documentation it could help both
>>> > regular Python users and Cython users.
>>>
>>> +1
>>>
>>> - Robert
>>> _______________________________________________
>>> Cython-dev mailing list
>>> Cython-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From stefan_ml at behnel.de  Wed Nov  3 19:26:30 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 03 Nov 2010 19:26:30 +0100
Subject: [Cython] Cython 0.13.1
In-Reply-To: <AANLkTin0LA4pjJfDMogXYYqKxOT9HHrANsFQtnHAvsp-@mail.gmail.com>
References: <AANLkTin0LA4pjJfDMogXYYqKxOT9HHrANsFQtnHAvsp-@mail.gmail.com>
Message-ID: <4CD1A956.4020507@behnel.de>

Robert Bradshaw, 03.11.2010 18:18:
> I've got a couple of experimental features in the hopper I'd like to
> get out, we've started to merge some of Haoyu's code (perhaps there's
> more that should go in), got __metaclass__ support thanks to Vitja,

Yep, the metaclass stuff really makes me happy, especially because it fixes 
the long standing problem of reversed class building.

However, given that it hasn't seen much testing and impacts several hugely 
important features of Python/Cython, I'd like to see at least a well tested 
beta before carefully pushing out a minor release.


> some cool gdb support from Mark (that I haven't had a chance to look
> at, but I've no reason to doubt it's ready or could be soon

Same from here. The mail really hinted at some awfully useful features.


> and a
> bunch of other bugfixes/minor features. How does everyone feel about a
> target date of this time next month? This one shouldn't drag on

Fine with me.

I already fixed a couple of bugs that handicapped the CPython 3.2 
regression test suite and (above all) enabled -3 mode in the test runner 
for the Py3 regr tests. It now finds almost 1900 tests, compared to a bit 
more than 1000 a couple of days ago. The improvement is due to less 
failures to compile entire test modules. It's now nearing the test count 
for Py2, meaning that the -3 mode actually starts to work as expected.

https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py3k-c/test/?width=800&height=600

Note that the results of the pyregr tests aren't all that meaningful since 
a huge bunch of them simply tests modules in the standard library or the 
behaviour of builtins. But they also keep discovering long standing bugs in 
Cython.

I also added a way to test the pure Python mode. If you put a .py file into 
the tests/run/ directory, it will be compiled and tested in Cython and 
additionally tested uncompiled by the standard Python doctest module.


> Any specific features people want to be sure to get in?

I'm currently making exec() a function in -3 mode and I'd like to see the 
Python 3 metaclass syntax enabled. I may also give the new property syntax 
a try, but I'm not sure I'll have the time to look into that soon enough to 
get it into 0.13.1. So if others want to give it a go, please do, it 
shouldn't be hard to implement.

Stefan

From stefan_ml at behnel.de  Wed Nov  3 20:38:58 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 03 Nov 2010 20:38:58 +0100
Subject: [Cython] Happy about the C++ support
Message-ID: <4CD1BA52.80607@behnel.de>

Hi,

I'm totally happy about the C++ support, even without using C++ ;)

I just added the following to the list of builtin functions:

     ('exec',       "OOO",  "O",     "__Pyx_PyRun"),
     ('exec',       "OO",   "O",     "__Pyx_PyRun2"),
     ...
     ('iter',       "OO",   "O",     "PyCallIter_New"),
     ('iter',       "O",    "O",     "PyObject_GetIter"),

then applied a tiny opener in Symtab.py, and it just worked!

Thank you for that, this will make so many things easier.

Stefan

From vitja.makarov at gmail.com  Wed Nov  3 21:07:16 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Wed, 3 Nov 2010 23:07:16 +0300
Subject: [Cython] Cython 0.13.1
In-Reply-To: <4CD1A956.4020507@behnel.de>
References: <AANLkTin0LA4pjJfDMogXYYqKxOT9HHrANsFQtnHAvsp-@mail.gmail.com>
	<4CD1A956.4020507@behnel.de>
Message-ID: <AANLkTimFRoVkK7v=Kw37o+OBXxwkML18Fhpdaa9WVNtb@mail.gmail.com>

2010/11/3 Stefan Behnel <stefan_ml at behnel.de>:
> Robert Bradshaw, 03.11.2010 18:18:
>> I've got a couple of experimental features in the hopper I'd like to
>> get out, we've started to merge some of Haoyu's code (perhaps there's
>> more that should go in), got __metaclass__ support thanks to Vitja,
>
> Yep, the metaclass stuff really makes me happy, especially because it fixes
> the long standing problem of reversed class building.
>
> However, given that it hasn't seen much testing and impacts several hugely
> important features of Python/Cython, I'd like to see at least a well tested
> beta before carefully pushing out a minor release.
>
>
>> some cool gdb support from Mark (that I haven't had a chance to look
>> at, but I've no reason to doubt it's ready or could be soon
>
> Same from here. The mail really hinted at some awfully useful features.
>
>
>> and a
>> bunch of other bugfixes/minor features. How does everyone feel about a
>> target date of this time next month? This one shouldn't drag on
>
> Fine with me.
>
> I already fixed a couple of bugs that handicapped the CPython 3.2
> regression test suite and (above all) enabled -3 mode in the test runner
> for the Py3 regr tests. It now finds almost 1900 tests, compared to a bit
> more than 1000 a couple of days ago. The improvement is due to less
> failures to compile entire test modules. It's now nearing the test count
> for Py2, meaning that the -3 mode actually starts to work as expected.
>
> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py3k-c/test/?width=800&height=600
>
> Note that the results of the pyregr tests aren't all that meaningful since
> a huge bunch of them simply tests modules in the standard library or the
> behaviour of builtins. But they also keep discovering long standing bugs in
> Cython.
>
> I also added a way to test the pure Python mode. If you put a .py file into
> the tests/run/ directory, it will be compiled and tested in Cython and
> additionally tested uncompiled by the standard Python doctest module.
>
>
>> Any specific features people want to be sure to get in?
>
> I'm currently making exec() a function in -3 mode and I'd like to see the
> Python 3 metaclass syntax enabled. I may also give the new property syntax
> a try, but I'm not sure I'll have the time to look into that soon enough to
> get it into 0.13.1. So if others want to give it a go, please do, it
> shouldn't be hard to implement.
>

What is about new property syntax?

From robertwb at math.washington.edu  Wed Nov  3 21:18:24 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 3 Nov 2010 13:18:24 -0700
Subject: [Cython] Cython 0.13.1
In-Reply-To: <AANLkTimFRoVkK7v=Kw37o+OBXxwkML18Fhpdaa9WVNtb@mail.gmail.com>
References: <AANLkTin0LA4pjJfDMogXYYqKxOT9HHrANsFQtnHAvsp-@mail.gmail.com>
	<4CD1A956.4020507@behnel.de>
	<AANLkTimFRoVkK7v=Kw37o+OBXxwkML18Fhpdaa9WVNtb@mail.gmail.com>
Message-ID: <AANLkTi=XhRB_vVKN4Sc2TnpPVBZ=EZK_a-HRdxzeORno@mail.gmail.com>

On Wed, Nov 3, 2010 at 1:07 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2010/11/3 Stefan Behnel <stefan_ml at behnel.de>:
>> Robert Bradshaw, 03.11.2010 18:18:
>>> I've got a couple of experimental features in the hopper I'd like to
>>> get out, we've started to merge some of Haoyu's code (perhaps there's
>>> more that should go in), got __metaclass__ support thanks to Vitja,
>>
>> Yep, the metaclass stuff really makes me happy, especially because it fixes
>> the long standing problem of reversed class building.
>>
>> However, given that it hasn't seen much testing and impacts several hugely
>> important features of Python/Cython, I'd like to see at least a well tested
>> beta before carefully pushing out a minor release.
>>
>>
>>> some cool gdb support from Mark (that I haven't had a chance to look
>>> at, but I've no reason to doubt it's ready or could be soon
>>
>> Same from here. The mail really hinted at some awfully useful features.
>>
>>
>>> and a
>>> bunch of other bugfixes/minor features. How does everyone feel about a
>>> target date of this time next month? This one shouldn't drag on
>>
>> Fine with me.
>>
>> I already fixed a couple of bugs that handicapped the CPython 3.2
>> regression test suite and (above all) enabled -3 mode in the test runner
>> for the Py3 regr tests. It now finds almost 1900 tests, compared to a bit
>> more than 1000 a couple of days ago. The improvement is due to less
>> failures to compile entire test modules. It's now nearing the test count
>> for Py2, meaning that the -3 mode actually starts to work as expected.
>>
>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py3k-c/test/?width=800&height=600
>>
>> Note that the results of the pyregr tests aren't all that meaningful since
>> a huge bunch of them simply tests modules in the standard library or the
>> behaviour of builtins. But they also keep discovering long standing bugs in
>> Cython.
>>
>> I also added a way to test the pure Python mode. If you put a .py file into
>> the tests/run/ directory, it will be compiled and tested in Cython and
>> additionally tested uncompiled by the standard Python doctest module.
>>
>>
>>> Any specific features people want to be sure to get in?
>>
>> I'm currently making exec() a function in -3 mode and I'd like to see the
>> Python 3 metaclass syntax enabled. I may also give the new property syntax
>> a try, but I'm not sure I'll have the time to look into that soon enough to
>> get it into 0.13.1. So if others want to give it a go, please do, it
>> shouldn't be hard to implement.
>>
>
> What is about new property syntax?

If it gets implemented and debugged in time, I'm not opposed. It may
be a big change, but it's not in any way backwards incompatible.

- Robert

From stefan_ml at behnel.de  Wed Nov  3 21:29:09 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 03 Nov 2010 21:29:09 +0100
Subject: [Cython] Cython 0.13.1
In-Reply-To: <AANLkTimFRoVkK7v=Kw37o+OBXxwkML18Fhpdaa9WVNtb@mail.gmail.com>
References: <AANLkTin0LA4pjJfDMogXYYqKxOT9HHrANsFQtnHAvsp-@mail.gmail.com>	<4CD1A956.4020507@behnel.de>
	<AANLkTimFRoVkK7v=Kw37o+OBXxwkML18Fhpdaa9WVNtb@mail.gmail.com>
Message-ID: <4CD1C615.6040807@behnel.de>

Vitja Makarov, 03.11.2010 21:07:
> 2010/11/3 Stefan Behnel:
>> I'm currently making exec() a function in -3 mode and I'd like to see the
>> Python 3 metaclass syntax enabled. I may also give the new property syntax
>> a try, but I'm not sure I'll have the time to look into that soon enough to
>> get it into 0.13.1. So if others want to give it a go, please do, it
>> shouldn't be hard to implement.
>
> What is about new property syntax?

http://thread.gmane.org/gmane.comp.python.cython.devel/10411/focus=10459

Stefan

From stefan_ml at behnel.de  Wed Nov  3 21:43:47 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 03 Nov 2010 21:43:47 +0100
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTimWfQO1SsVgDX9yyex=xivq0z7nmw0PJoJBXvFc@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>	<AANLkTinPTHtp=X483QUGsRO58tm2TTokSc02LxaEt3fP@mail.gmail.com>	<AANLkTikESy8G5XWBi=8cvmO0nCJgQg_A9dQ=znrz-XuN@mail.gmail.com>	<AANLkTino6OWKMbgG41Wvn1vo-jNP=KRvnX2H9ai31rv-@mail.gmail.com>	<AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>	<AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>	<AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>	<AANLkTikSi2Awwv31sx3A6BJy=WNSYROo+ceecoygQ0Fk@mail.gmail.com>	<AANLkTi=bFQwdofb4TPcV53b9w0RtcshvC8Uiv2mJa9i-@mail.gmail.com>	<4CD0FA29.7000201@behnel.de>	<AANLkTinNGspAzLoOBdoXcUZgxkxR86i4aQfS2FMrgVYR@mail.gmail.com>	<AANLkTi=a8SMk+u7JfLKxujR3p2ZVnjLsSPjrMY8hufbt@mail.gmail.com>	<AANLkTimjYc=4HdLE+VDmUyn0piXPWGZ7zWGqQ0QoabkP@mail.gmail.com>	<AANLkTim0dL=7nR6WiXAYQyMUSyQ=_b7jsBuKCQ8zh-WT@mail.gmail.com>	<AANLkTim6qdY6hmj2VVJJ_rwje5Sup5NQjspZuymVEv8Z@mail.gmail.com>
	<AANLkTimWfQO1SsVgDX9yyex=xivq0z7nmw0PJoJBXvFc@mail.gmail.com>
Message-ID: <4CD1C983.7000300@behnel.de>

Robert Bradshaw, 03.11.2010 18:13:
> On Wed, Nov 3, 2010 at 10:00 AM, Robert Bradshaw wrote:
>> On Wed, Nov 3, 2010 at 7:13 AM, Vitja Makarov wrote:
>>> I've fixed classmethod issue patch is in trac.
>>
>> Very cool. Looks like I only apply the last patch, right?
>
> Sweet. All looks well. I've merged it.

Hmm, it has a ref-leak and it doesn't work in Py3. Not that well after all.

I actually don't think that "python3+ handle __metaclass__ itself". Why 
should it? It has syntax support for this very feature.

Stefan

From stefan_ml at behnel.de  Wed Nov  3 21:51:14 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 03 Nov 2010 21:51:14 +0100
Subject: [Cython] metaclasses
In-Reply-To: <4CD1C983.7000300@behnel.de>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>	<AANLkTikESy8G5XWBi=8cvmO0nCJgQg_A9dQ=znrz-XuN@mail.gmail.com>	<AANLkTino6OWKMbgG41Wvn1vo-jNP=KRvnX2H9ai31rv-@mail.gmail.com>	<AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>	<AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>	<AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>	<AANLkTikSi2Awwv31sx3A6BJy=WNSYROo+ceecoygQ0Fk@mail.gmail.com>	<AANLkTi=bFQwdofb4TPcV53b9w0RtcshvC8Uiv2mJa9i-@mail.gmail.com>	<4CD0FA29.7000201@behnel.de>	<AANLkTinNGspAzLoOBdoXcUZgxkxR86i4aQfS2FMrgVYR@mail.gmail.com>	<AANLkTi=a8SMk+u7JfLKxujR3p2ZVnjLsSPjrMY8hufbt@mail.gmail.com>	<AANLkTimjYc=4HdLE+VDmUyn0piXPWGZ7zWGqQ0QoabkP@mail.gmail.com>	<AANLkTim0dL=7nR6WiXAYQyMUSyQ=_b7jsBuKCQ8zh-WT@mail.gmail.com>	<AANLkTim6qdY6hmj2VVJJ_rwje5Sup5NQjspZuymVEv8Z@mail.gmail.com>	<AANLkTimWfQO1SsVgDX9yyex=xivq0z7nmw0PJoJBXvFc@mail.gmail.com>
	<4CD1C983.7000300@behnel.de>
Message-ID: <4CD1CB42.1020606@behnel.de>

Stefan Behnel, 03.11.2010 21:43:
> Robert Bradshaw, 03.11.2010 18:13:
>> On Wed, Nov 3, 2010 at 10:00 AM, Robert Bradshaw wrote:
>>> On Wed, Nov 3, 2010 at 7:13 AM, Vitja Makarov wrote:
>>>> I've fixed classmethod issue patch is in trac.
>>>
>>> Very cool. Looks like I only apply the last patch, right?
>>
>> Sweet. All looks well. I've merged it.
>
> Hmm, it has a ref-leak and it doesn't work in Py3. Not that well after all.
>
> I actually don't think that "python3+ handle __metaclass__ itself". Why
> should it? It has syntax support for this very feature.

... and it even has a builtin function to build a class, see

	builtin___build_class__()

right at the top of Python/bltinmodule.c.

Stefan

From vitja.makarov at gmail.com  Wed Nov  3 22:13:32 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 4 Nov 2010 00:13:32 +0300
Subject: [Cython] metaclasses
In-Reply-To: <4CD1C983.7000300@behnel.de>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<AANLkTinPTHtp=X483QUGsRO58tm2TTokSc02LxaEt3fP@mail.gmail.com>
	<AANLkTikESy8G5XWBi=8cvmO0nCJgQg_A9dQ=znrz-XuN@mail.gmail.com>
	<AANLkTino6OWKMbgG41Wvn1vo-jNP=KRvnX2H9ai31rv-@mail.gmail.com>
	<AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>
	<AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>
	<AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>
	<AANLkTikSi2Awwv31sx3A6BJy=WNSYROo+ceecoygQ0Fk@mail.gmail.com>
	<AANLkTi=bFQwdofb4TPcV53b9w0RtcshvC8Uiv2mJa9i-@mail.gmail.com>
	<4CD0FA29.7000201@behnel.de>
	<AANLkTinNGspAzLoOBdoXcUZgxkxR86i4aQfS2FMrgVYR@mail.gmail.com>
	<AANLkTi=a8SMk+u7JfLKxujR3p2ZVnjLsSPjrMY8hufbt@mail.gmail.com>
	<AANLkTimjYc=4HdLE+VDmUyn0piXPWGZ7zWGqQ0QoabkP@mail.gmail.com>
	<AANLkTim0dL=7nR6WiXAYQyMUSyQ=_b7jsBuKCQ8zh-WT@mail.gmail.com>
	<AANLkTim6qdY6hmj2VVJJ_rwje5Sup5NQjspZuymVEv8Z@mail.gmail.com>
	<AANLkTimWfQO1SsVgDX9yyex=xivq0z7nmw0PJoJBXvFc@mail.gmail.com>
	<4CD1C983.7000300@behnel.de>
Message-ID: <AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>

2010/11/3 Stefan Behnel <stefan_ml at behnel.de>:
> Robert Bradshaw, 03.11.2010 18:13:
>> On Wed, Nov 3, 2010 at 10:00 AM, Robert Bradshaw wrote:
>>> On Wed, Nov 3, 2010 at 7:13 AM, Vitja Makarov wrote:
>>>> I've fixed classmethod issue patch is in trac.
>>>
>>> Very cool. Looks like I only apply the last patch, right?
>>
>> Sweet. All looks well. I've merged it.
>
> Hmm, it has a ref-leak and it doesn't work in Py3. Not that well after all.
>

Where is ref-leak? Guess it's somewhere in
PyClassDefNode.generate_execution_code

> I actually don't think that "python3+ handle __metaclass__ itself". Why
> should it? It has syntax support for this very feature.
>

Yes, I didnt knew much about python3.

Now it seems to me that class declaration is more like function call
with special case for metaclass keyword and positional arguments:

class Base(type):
    def __new__(cls, name, bases, attrs, **kwargs):
        print(cls, name, bases, attrs, kwargs)

class Foo(1, 2, 3, metaclass=Base, foo='bar'):
    def hello(self):
        pass

From vitja.makarov at gmail.com  Wed Nov  3 22:20:27 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 4 Nov 2010 00:20:27 +0300
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<AANLkTinPTHtp=X483QUGsRO58tm2TTokSc02LxaEt3fP@mail.gmail.com>
	<AANLkTikESy8G5XWBi=8cvmO0nCJgQg_A9dQ=znrz-XuN@mail.gmail.com>
	<AANLkTino6OWKMbgG41Wvn1vo-jNP=KRvnX2H9ai31rv-@mail.gmail.com>
	<AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>
	<AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>
	<AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>
	<AANLkTikSi2Awwv31sx3A6BJy=WNSYROo+ceecoygQ0Fk@mail.gmail.com>
	<AANLkTi=bFQwdofb4TPcV53b9w0RtcshvC8Uiv2mJa9i-@mail.gmail.com>
	<4CD0FA29.7000201@behnel.de>
	<AANLkTinNGspAzLoOBdoXcUZgxkxR86i4aQfS2FMrgVYR@mail.gmail.com>
	<AANLkTi=a8SMk+u7JfLKxujR3p2ZVnjLsSPjrMY8hufbt@mail.gmail.com>
	<AANLkTimjYc=4HdLE+VDmUyn0piXPWGZ7zWGqQ0QoabkP@mail.gmail.com>
	<AANLkTim0dL=7nR6WiXAYQyMUSyQ=_b7jsBuKCQ8zh-WT@mail.gmail.com>
	<AANLkTim6qdY6hmj2VVJJ_rwje5Sup5NQjspZuymVEv8Z@mail.gmail.com>
	<AANLkTimWfQO1SsVgDX9yyex=xivq0z7nmw0PJoJBXvFc@mail.gmail.com>
	<4CD1C983.7000300@behnel.de>
	<AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>
Message-ID: <AANLkTimMn3q1zpWwmV0p7DUX9ri5eCxCCnFvH1Dwtb9p@mail.gmail.com>

2010/11/4 Vitja Makarov <vitja.makarov at gmail.com>:
> 2010/11/3 Stefan Behnel <stefan_ml at behnel.de>:
>> Robert Bradshaw, 03.11.2010 18:13:
>>> On Wed, Nov 3, 2010 at 10:00 AM, Robert Bradshaw wrote:
>>>> On Wed, Nov 3, 2010 at 7:13 AM, Vitja Makarov wrote:
>>>>> I've fixed classmethod issue patch is in trac.
>>>>
>>>> Very cool. Looks like I only apply the last patch, right?
>>>
>>> Sweet. All looks well. I've merged it.
>>
>> Hmm, it has a ref-leak and it doesn't work in Py3. Not that well after all.
>>
>
> Where is ref-leak? Guess it's somewhere in
> PyClassDefNode.generate_execution_code
>

Oh, there should be Py_DECREF for metaclass in Pyx_CreateClass?

From stefan_ml at behnel.de  Wed Nov  3 22:24:14 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 03 Nov 2010 22:24:14 +0100
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTimMn3q1zpWwmV0p7DUX9ri5eCxCCnFvH1Dwtb9p@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>	<AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>	<AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>	<AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>	<AANLkTikSi2Awwv31sx3A6BJy=WNSYROo+ceecoygQ0Fk@mail.gmail.com>	<AANLkTi=bFQwdofb4TPcV53b9w0RtcshvC8Uiv2mJa9i-@mail.gmail.com>	<4CD0FA29.7000201@behnel.de>	<AANLkTinNGspAzLoOBdoXcUZgxkxR86i4aQfS2FMrgVYR@mail.gmail.com>	<AANLkTi=a8SMk+u7JfLKxujR3p2ZVnjLsSPjrMY8hufbt@mail.gmail.com>	<AANLkTimjYc=4HdLE+VDmUyn0piXPWGZ7zWGqQ0QoabkP@mail.gmail.com>	<AANLkTim0dL=7nR6WiXAYQyMUSyQ=_b7jsBuKCQ8zh-WT@mail.gmail.com>	<AANLkTim6qdY6hmj2VVJJ_rwje5Sup5NQjspZuymVEv8Z@mail.gmail.com>	<AANLkTimWfQO1SsVgDX9yyex=xivq0z7nmw0PJoJBXvFc@mail.gmail.com>	<4CD1C983.7000300@behnel.de>	<AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>
	<AANLkTimMn3q1zpWwmV0p7DUX9ri5eCxCCnFvH1Dwtb9p@mail.gmail.com>
Message-ID: <4CD1D2FE.9090806@behnel.de>

Vitja Makarov, 03.11.2010 22:20:
> 2010/11/4 Vitja Makarov<vitja.makarov at gmail.com>:
>> 2010/11/3 Stefan Behnel<stefan_ml at behnel.de>:
>>> Robert Bradshaw, 03.11.2010 18:13:
>>>> On Wed, Nov 3, 2010 at 10:00 AM, Robert Bradshaw wrote:
>>>>> On Wed, Nov 3, 2010 at 7:13 AM, Vitja Makarov wrote:
>>>>>> I've fixed classmethod issue patch is in trac.
>>>>>
>>>>> Very cool. Looks like I only apply the last patch, right?
>>>>
>>>> Sweet. All looks well. I've merged it.
>>>
>>> Hmm, it has a ref-leak and it doesn't work in Py3. Not that well after all.
>>>
>>
>> Where is ref-leak? Guess it's somewhere in
>> PyClassDefNode.generate_execution_code
>>
>
> Oh, there should be Py_DECREF for metaclass in Pyx_CreateClass?

Yes, I already pushed the fix.

Stefan

From vitja.makarov at gmail.com  Wed Nov  3 22:25:22 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 4 Nov 2010 00:25:22 +0300
Subject: [Cython] metaclasses
In-Reply-To: <4CD1D2FE.9090806@behnel.de>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>
	<AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>
	<AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>
	<AANLkTikSi2Awwv31sx3A6BJy=WNSYROo+ceecoygQ0Fk@mail.gmail.com>
	<AANLkTi=bFQwdofb4TPcV53b9w0RtcshvC8Uiv2mJa9i-@mail.gmail.com>
	<4CD0FA29.7000201@behnel.de>
	<AANLkTinNGspAzLoOBdoXcUZgxkxR86i4aQfS2FMrgVYR@mail.gmail.com>
	<AANLkTi=a8SMk+u7JfLKxujR3p2ZVnjLsSPjrMY8hufbt@mail.gmail.com>
	<AANLkTimjYc=4HdLE+VDmUyn0piXPWGZ7zWGqQ0QoabkP@mail.gmail.com>
	<AANLkTim0dL=7nR6WiXAYQyMUSyQ=_b7jsBuKCQ8zh-WT@mail.gmail.com>
	<AANLkTim6qdY6hmj2VVJJ_rwje5Sup5NQjspZuymVEv8Z@mail.gmail.com>
	<AANLkTimWfQO1SsVgDX9yyex=xivq0z7nmw0PJoJBXvFc@mail.gmail.com>
	<4CD1C983.7000300@behnel.de>
	<AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>
	<AANLkTimMn3q1zpWwmV0p7DUX9ri5eCxCCnFvH1Dwtb9p@mail.gmail.com>
	<4CD1D2FE.9090806@behnel.de>
Message-ID: <AANLkTinWwBGYsiHOkcVM-xxMyPKmN9Bgy3LMmd1QEnx+@mail.gmail.com>

Thank, you! What's about PyClassDefNode.generate_execution_code is
that correct, I was not sure?

2010/11/4 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 03.11.2010 22:20:
>> 2010/11/4 Vitja Makarov<vitja.makarov at gmail.com>:
>>> 2010/11/3 Stefan Behnel<stefan_ml at behnel.de>:
>>>> Robert Bradshaw, 03.11.2010 18:13:
>>>>> On Wed, Nov 3, 2010 at 10:00 AM, Robert Bradshaw wrote:
>>>>>> On Wed, Nov 3, 2010 at 7:13 AM, Vitja Makarov wrote:
>>>>>>> I've fixed classmethod issue patch is in trac.
>>>>>>
>>>>>> Very cool. Looks like I only apply the last patch, right?
>>>>>
>>>>> Sweet. All looks well. I've merged it.
>>>>
>>>> Hmm, it has a ref-leak and it doesn't work in Py3. Not that well after all.
>>>>
>>>
>>> Where is ref-leak? Guess it's somewhere in
>>> PyClassDefNode.generate_execution_code
>>>
>>
>> Oh, there should be Py_DECREF for metaclass in Pyx_CreateClass?
>
> Yes, I already pushed the fix.
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From stefan_ml at behnel.de  Wed Nov  3 22:43:35 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 03 Nov 2010 22:43:35 +0100
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>	<AANLkTino6OWKMbgG41Wvn1vo-jNP=KRvnX2H9ai31rv-@mail.gmail.com>	<AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>	<AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>	<AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>	<AANLkTikSi2Awwv31sx3A6BJy=WNSYROo+ceecoygQ0Fk@mail.gmail.com>	<AANLkTi=bFQwdofb4TPcV53b9w0RtcshvC8Uiv2mJa9i-@mail.gmail.com>	<4CD0FA29.7000201@behnel.de>	<AANLkTinNGspAzLoOBdoXcUZgxkxR86i4aQfS2FMrgVYR@mail.gmail.com>	<AANLkTi=a8SMk+u7JfLKxujR3p2ZVnjLsSPjrMY8hufbt@mail.gmail.com>	<AANLkTimjYc=4HdLE+VDmUyn0piXPWGZ7zWGqQ0QoabkP@mail.gmail.com>	<AANLkTim0dL=7nR6WiXAYQyMUSyQ=_b7jsBuKCQ8zh-WT@mail.gmail.com>	<AANLkTim6qdY6hmj2VVJJ_rwje5Sup5NQjspZuymVEv8Z@mail.gmail.com>	<AANLkTimWfQO1SsVgDX9yyex=xivq0z7nmw0PJoJBXvFc@mail.gmail.com>	<4CD1C983.7000300@behnel.de>
	<AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>
Message-ID: <4CD1D787.4010202@behnel.de>

Vitja Makarov, 03.11.2010 22:13:
> 2010/11/3 Stefan Behnel:
>> I actually don't think that "python3+ handle __metaclass__ itself". Why
>> should it? It has syntax support for this very feature.
>
> Yes, I didnt knew much about python3.
>
> Now it seems to me that class declaration is more like function call
> with special case for metaclass keyword and positional arguments:
>
> class Base(type):
>      def __new__(cls, name, bases, attrs, **kwargs):
>          print(cls, name, bases, attrs, kwargs)
>
> class Foo(1, 2, 3, metaclass=Base, foo='bar'):
>      def hello(self):
>          pass

Yes, I think that's exactly how this should work. Want to give it another try?

It seems that the number of positional arguments is fixed, but you can pass 
any number of keyword arguments, out of which only "metaclass" is special 
cased and removed (from a copy of the dict, BTW). There is also an 
additional protocol if the metaclass has a "__prepare__" class method. See 
"__build_class__".

I think the Python 2 protocol in Cython should work as in Py3. Metaclasses 
in Py2 won't generally support kwargs, but if a user explicitly provides 
them, it's best to assume that that's intentional. In the worst case, there 
will be a runtime TypeError.

So, kwargs should pass through, except for __metaclass__. If both the 
__metaclass__ class field and the metaclass kwarg are passed, I'd lean 
towards ignoring the field and prefer the kwarg, just like Py3 does. It can 
be argued that this is worth a warning, though.

I also think that the parser should extract the metaclass keyword, not the 
runtime code. So, if provided, it should be passed into __Pyx_CreateClass() 
as an argument and not in the kwargs.

Stefan

From stefan_ml at behnel.de  Wed Nov  3 22:46:21 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 03 Nov 2010 22:46:21 +0100
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTinWwBGYsiHOkcVM-xxMyPKmN9Bgy3LMmd1QEnx+@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>	<AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>	<AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>	<AANLkTikSi2Awwv31sx3A6BJy=WNSYROo+ceecoygQ0Fk@mail.gmail.com>	<AANLkTi=bFQwdofb4TPcV53b9w0RtcshvC8Uiv2mJa9i-@mail.gmail.com>	<4CD0FA29.7000201@behnel.de>	<AANLkTinNGspAzLoOBdoXcUZgxkxR86i4aQfS2FMrgVYR@mail.gmail.com>	<AANLkTi=a8SMk+u7JfLKxujR3p2ZVnjLsSPjrMY8hufbt@mail.gmail.com>	<AANLkTimjYc=4HdLE+VDmUyn0piXPWGZ7zWGqQ0QoabkP@mail.gmail.com>	<AANLkTim0dL=7nR6WiXAYQyMUSyQ=_b7jsBuKCQ8zh-WT@mail.gmail.com>	<AANLkTim6qdY6hmj2VVJJ_rwje5Sup5NQjspZuymVEv8Z@mail.gmail.com>	<AANLkTimWfQO1SsVgDX9yyex=xivq0z7nmw0PJoJBXvFc@mail.gmail.com>	<4CD1C983.7000300@behnel.de>	<AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>	<AANLkTimMn3q1zpWwmV0p7DUX9ri5eCxCCnFvH1Dwtb9p@mail.gmail.com>	<4CD1D2FE.9090806@behnel.de>
	<AANLkTinWwBGYsiHOkcVM-xxMyPKmN9Bgy3LMmd1QEnx+@mail.gmail.com>
Message-ID: <4CD1D82D.3010106@behnel.de>

Vitja Makarov, 03.11.2010 22:25:
> What's about PyClassDefNode.generate_execution_code is
> that correct, I was not sure?

It does look a bit funny but that didn't change much from before, so I 
think it's ok.

Stefan

From vitja.makarov at gmail.com  Wed Nov  3 23:04:24 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 4 Nov 2010 01:04:24 +0300
Subject: [Cython] metaclasses
In-Reply-To: <4CD1D82D.3010106@behnel.de>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>
	<AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>
	<AANLkTikSi2Awwv31sx3A6BJy=WNSYROo+ceecoygQ0Fk@mail.gmail.com>
	<AANLkTi=bFQwdofb4TPcV53b9w0RtcshvC8Uiv2mJa9i-@mail.gmail.com>
	<4CD0FA29.7000201@behnel.de>
	<AANLkTinNGspAzLoOBdoXcUZgxkxR86i4aQfS2FMrgVYR@mail.gmail.com>
	<AANLkTi=a8SMk+u7JfLKxujR3p2ZVnjLsSPjrMY8hufbt@mail.gmail.com>
	<AANLkTimjYc=4HdLE+VDmUyn0piXPWGZ7zWGqQ0QoabkP@mail.gmail.com>
	<AANLkTim0dL=7nR6WiXAYQyMUSyQ=_b7jsBuKCQ8zh-WT@mail.gmail.com>
	<AANLkTim6qdY6hmj2VVJJ_rwje5Sup5NQjspZuymVEv8Z@mail.gmail.com>
	<AANLkTimWfQO1SsVgDX9yyex=xivq0z7nmw0PJoJBXvFc@mail.gmail.com>
	<4CD1C983.7000300@behnel.de>
	<AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>
	<AANLkTimMn3q1zpWwmV0p7DUX9ri5eCxCCnFvH1Dwtb9p@mail.gmail.com>
	<4CD1D2FE.9090806@behnel.de>
	<AANLkTinWwBGYsiHOkcVM-xxMyPKmN9Bgy3LMmd1QEnx+@mail.gmail.com>
	<4CD1D82D.3010106@behnel.de>
Message-ID: <AANLkTimQJx9FHvA_Bm7yR0-NCXGvujsd_MUjV8vxdSJ9@mail.gmail.com>

2010/11/4 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 03.11.2010 22:25:
>> What's about PyClassDefNode.generate_execution_code is
>> that correct, I was not sure?
>
> It does look a bit funny but that didn't change much from before, so I
> think it's ok.
>

I'm not sure about

        cenv.namespace_cname = cenv.class_obj_cname = self.dict.result()
and
        cenv.namespace_cname = cenv.class_obj_cname = self.classobj.result()

From vitja.makarov at gmail.com  Wed Nov  3 23:03:25 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 4 Nov 2010 01:03:25 +0300
Subject: [Cython] metaclasses
In-Reply-To: <4CD1D787.4010202@behnel.de>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<AANLkTino6OWKMbgG41Wvn1vo-jNP=KRvnX2H9ai31rv-@mail.gmail.com>
	<AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>
	<AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>
	<AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>
	<AANLkTikSi2Awwv31sx3A6BJy=WNSYROo+ceecoygQ0Fk@mail.gmail.com>
	<AANLkTi=bFQwdofb4TPcV53b9w0RtcshvC8Uiv2mJa9i-@mail.gmail.com>
	<4CD0FA29.7000201@behnel.de>
	<AANLkTinNGspAzLoOBdoXcUZgxkxR86i4aQfS2FMrgVYR@mail.gmail.com>
	<AANLkTi=a8SMk+u7JfLKxujR3p2ZVnjLsSPjrMY8hufbt@mail.gmail.com>
	<AANLkTimjYc=4HdLE+VDmUyn0piXPWGZ7zWGqQ0QoabkP@mail.gmail.com>
	<AANLkTim0dL=7nR6WiXAYQyMUSyQ=_b7jsBuKCQ8zh-WT@mail.gmail.com>
	<AANLkTim6qdY6hmj2VVJJ_rwje5Sup5NQjspZuymVEv8Z@mail.gmail.com>
	<AANLkTimWfQO1SsVgDX9yyex=xivq0z7nmw0PJoJBXvFc@mail.gmail.com>
	<4CD1C983.7000300@behnel.de>
	<AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>
	<4CD1D787.4010202@behnel.de>
Message-ID: <AANLkTikYRv6CPTbSifo-C_gAwU-AG0REC1sPwAHq2DJO@mail.gmail.com>

2010/11/4 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 03.11.2010 22:13:
>> 2010/11/3 Stefan Behnel:
>>> I actually don't think that "python3+ handle __metaclass__ itself". Why
>>> should it? It has syntax support for this very feature.
>>
>> Yes, I didnt knew much about python3.
>>
>> Now it seems to me that class declaration is more like function call
>> with special case for metaclass keyword and positional arguments:
>>
>> class Base(type):
>> ? ? ?def __new__(cls, name, bases, attrs, **kwargs):
>> ? ? ? ? ?print(cls, name, bases, attrs, kwargs)
>>
>> class Foo(1, 2, 3, metaclass=Base, foo='bar'):
>> ? ? ?def hello(self):
>> ? ? ? ? ?pass
>
> Yes, I think that's exactly how this should work. Want to give it another try?
>

Yes, I want to finish this part ;)

> It seems that the number of positional arguments is fixed, but you can pass
> any number of keyword arguments, out of which only "metaclass" is special
> cased and removed (from a copy of the dict, BTW). There is also an
> additional protocol if the metaclass has a "__prepare__" class method. See
> "__build_class__".
>

Yes, all positional arguments are turned into bases. So we can make it this way:

Pyx_CreateClass(PyObject *bases, PyObject *dict, PyObject *name,
PyObject *modname, PyObject *kwargs);

And then

args = (metaclass, bases, name, dict)
PyEval_CallObjectWithKeywords(metaclass, args, kwargs)

> I think the Python 2 protocol in Cython should work as in Py3. Metaclasses
> in Py2 won't generally support kwargs, but if a user explicitly provides
> them, it's best to assume that that's intentional. In the worst case, there
> will be a runtime TypeError.
>

Do you mean that new syntax should be used for Py2?

what about this case:

bases = [Foo, Bar]

class MyClass(*bases):
    pass

Don't see any problem to support this for Py2

> So, kwargs should pass through, except for __metaclass__. If both the
> __metaclass__ class field and the metaclass kwarg are passed, I'd lean
> towards ignoring the field and prefer the kwarg, just like Py3 does. It can
> be argued that this is worth a warning, though.
>

Py3 doesn't support __metaclass__ attribute, so if metaclass is set
that seems to be mostly Py3 so __metaclass__ should be ignored.

> I also think that the parser should extract the metaclass keyword, not the
> runtime code. So, if provided, it should be passed into __Pyx_CreateClass()
> as an argument and not in the kwargs.
>

I think parser can't extract metaclass keyword, try this code:

class Base(type):
    def __new__(cls, name, bases, attrs, **kwargs):
        print(cls, name, bases, attrs, kwargs)

myargs={'metaclass': Base}
class Foo(1, 2, 3, **myargs):
    def hello(self):
        pass
.

From brett.calcott at gmail.com  Thu Nov  4 04:29:40 2010
From: brett.calcott at gmail.com (Brett Calcott)
Date: Thu, 4 Nov 2010 14:29:40 +1100
Subject: [Cython] Cython 0.13.1
In-Reply-To: <4CD1C615.6040807@behnel.de>
References: <AANLkTin0LA4pjJfDMogXYYqKxOT9HHrANsFQtnHAvsp-@mail.gmail.com>
	<4CD1A956.4020507@behnel.de>
	<AANLkTimFRoVkK7v=Kw37o+OBXxwkML18Fhpdaa9WVNtb@mail.gmail.com>
	<4CD1C615.6040807@behnel.de>
Message-ID: <AANLkTimNwGP57A4S1Fr67m_sADJdCaBVCq0a2i7CC11k@mail.gmail.com>

On 4 November 2010 07:29, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> What is about new property syntax?
>
> http://thread.gmane.org/gmane.comp.python.cython.devel/10411/focus=10459
>

+1 for this (I started the thread)

Happy to try and help. But my cython powers are not (yet) strong.

Brett

From brett.calcott at gmail.com  Thu Nov  4 04:24:27 2010
From: brett.calcott at gmail.com (Brett Calcott)
Date: Thu, 4 Nov 2010 14:24:27 +1100
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <4CD11C30.9090103@student.matnat.uio.no>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>
	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>
	<AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>
	<AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>
	<4CD11C30.9090103@student.matnat.uio.no>
Message-ID: <AANLkTimABt0xae_NBG1MVhLGx5-AMf4ps4m7jT2bcQQC@mail.gmail.com>

On 3 November 2010 19:24, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> cdef class A:
> ? ? cdef Vector obj
> ? ? x = delegate_property(obj, 'x')
>
> or similar, without throwing in new syntax. But it may be too late and I
> don't care too much. Remember that "x = property(...)" is already in Python.
>

This looks like an interesting option, especially if it was possible
to write your own reusable "xxxx_property" functions that did "the
right thing" for the particular object/library being wrapped. This
would remove a lot boilerplate code that is normally cut and pasted
for setter/getter functions. So, in a case where the C++ members are
private, but access is provided via camelcase set/get functions,
instead of

property value:
  def __get__(self):
    return self.obj.getValue()
  def __set__(self, int x):
    self.obj.setValue(x)

We could write:
x = camelcase_delegate_property(self.obj, 'value')

Brett

From robertwb at math.washington.edu  Thu Nov  4 04:34:39 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 3 Nov 2010 20:34:39 -0700
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <AANLkTimABt0xae_NBG1MVhLGx5-AMf4ps4m7jT2bcQQC@mail.gmail.com>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>
	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>
	<AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>
	<AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>
	<4CD11C30.9090103@student.matnat.uio.no>
	<AANLkTimABt0xae_NBG1MVhLGx5-AMf4ps4m7jT2bcQQC@mail.gmail.com>
Message-ID: <AANLkTi=5DttGcLp6SEv0w06YVratHpc_CSdCSpoiU-rD@mail.gmail.com>

On Wed, Nov 3, 2010 at 8:24 PM, Brett Calcott <brett.calcott at gmail.com> wrote:
> On 3 November 2010 19:24, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>> cdef class A:
>> ? ? cdef Vector obj
>> ? ? x = delegate_property(obj, 'x')
>>
>> or similar, without throwing in new syntax. But it may be too late and I
>> don't care too much. Remember that "x = property(...)" is already in Python.
>>
>
> This looks like an interesting option, especially if it was possible
> to write your own reusable "xxxx_property" functions that did "the
> right thing" for the particular object/library being wrapped. This
> would remove a lot boilerplate code that is normally cut and pasted
> for setter/getter functions. So, in a case where the C++ members are
> private, but access is provided via camelcase set/get functions,
> instead of
>
> property value:
> ?def __get__(self):
> ? ?return self.obj.getValue()
> ?def __set__(self, int x):
> ? ?self.obj.setValue(x)
>
> We could write:
> x = camelcase_delegate_property(self.obj, 'value')

The one thing that I don't like about this is putting the attribute in
a String.

- Robert

From brett.calcott at gmail.com  Thu Nov  4 04:51:40 2010
From: brett.calcott at gmail.com (Brett Calcott)
Date: Thu, 4 Nov 2010 14:51:40 +1100
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <AANLkTi=5DttGcLp6SEv0w06YVratHpc_CSdCSpoiU-rD@mail.gmail.com>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>
	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>
	<AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>
	<AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>
	<4CD11C30.9090103@student.matnat.uio.no>
	<AANLkTimABt0xae_NBG1MVhLGx5-AMf4ps4m7jT2bcQQC@mail.gmail.com>
	<AANLkTi=5DttGcLp6SEv0w06YVratHpc_CSdCSpoiU-rD@mail.gmail.com>
Message-ID: <AANLkTikpWy1GW-ht-37faSvb3gTkpGPnOkycC9qY-Wyn@mail.gmail.com>

On 4 November 2010 14:34, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Wed, Nov 3, 2010 at 8:24 PM, Brett Calcott <brett.calcott at gmail.com> wrote:
>> On 3 November 2010 19:24, Dag Sverre Seljebotn
>> <dagss at student.matnat.uio.no> wrote:
>>> cdef class A:
>>> ? ? cdef Vector obj
>>> ? ? x = delegate_property(obj, 'x')
>>>
>>> or similar, without throwing in new syntax. But it may be too late and I
>>> don't care too much. Remember that "x = property(...)" is already in Python.
>>>
>>
>> This looks like an interesting option, especially if it was possible
>> to write your own reusable "xxxx_property" functions that did "the
>> right thing" for the particular object/library being wrapped. This
>> would remove a lot boilerplate code that is normally cut and pasted
>> for setter/getter functions. So, in a case where the C++ members are
>> private, but access is provided via camelcase set/get functions,
>> instead of
>>
>> property value:
>> ?def __get__(self):
>> ? ?return self.obj.getValue()
>> ?def __set__(self, int x):
>> ? ?self.obj.setValue(x)
>>
>> We could write:
>> x = camelcase_delegate_property(self.obj, 'value')
>
> The one thing that I don't like about this is putting the attribute in
> a String.
>

Is this possible?
x = member_property(self.obj.x)
y = getset_property(self.obj.getY, self.obj.setY)
(the names of these need work I agree)

But I take it that the general idea would be able to roll your own
when there was some repetitive thing you needed to do, so up to you
what you passed.

For example, the obj pointer in some of the classes I am currently
wrapping can sometimes be set to NULL. So in many cases the property
access needs to check that first, and raise an exception. If I could
write my own getset_with_check_property(...), that would be very cool.

Brett

From strombrg at gmail.com  Thu Nov  4 05:05:14 2010
From: strombrg at gmail.com (Dan Stromberg)
Date: Wed, 3 Nov 2010 21:05:14 -0700
Subject: [Cython] Cython 0.13.1
In-Reply-To: <AANLkTin0LA4pjJfDMogXYYqKxOT9HHrANsFQtnHAvsp-@mail.gmail.com>
References: <AANLkTin0LA4pjJfDMogXYYqKxOT9HHrANsFQtnHAvsp-@mail.gmail.com>
Message-ID: <AANLkTimwab4WPVtuOmp6ahrvobPBR17CUq==QJhaGS+2@mail.gmail.com>

On Wed, Nov 3, 2010 at 10:18 AM, Robert Bradshaw <
robertwb at math.washington.edu> wrote:

> I think it's time we start thinking about rolling out another release.
>

Not a code contribution, so I don't know how much weight this'll carry, but
the two features I miss the most in Cython are yield, and generator
expressions - in that order.  I use them pretty often in CPython, so when I
got to convert a critical section from CPython to Cython, these language
features appear to need more than just type annotations.

-- 
Dan Stromberg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20101103/8fe30cdc/attachment.htm 

From robertwb at math.washington.edu  Thu Nov  4 06:15:05 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 3 Nov 2010 22:15:05 -0700
Subject: [Cython] Cython 0.13.1
In-Reply-To: <AANLkTimwab4WPVtuOmp6ahrvobPBR17CUq==QJhaGS+2@mail.gmail.com>
References: <AANLkTin0LA4pjJfDMogXYYqKxOT9HHrANsFQtnHAvsp-@mail.gmail.com>
	<AANLkTimwab4WPVtuOmp6ahrvobPBR17CUq==QJhaGS+2@mail.gmail.com>
Message-ID: <AANLkTikB49qKgLNJKN5HnbF7S1+t_yvA9ntMNdofwOs1@mail.gmail.com>

On Wed, Nov 3, 2010 at 9:05 PM, Dan Stromberg <strombrg at gmail.com> wrote:
>
> On Wed, Nov 3, 2010 at 10:18 AM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>>
>> I think it's time we start thinking about rolling out another release.
>
> Not a code contribution, so I don't know how much weight this'll carry, but
> the two features I miss the most in Cython are yield, and generator
> expressions - in that order.

The good news is that these are, essentially, one feature.

> I use them pretty often in CPython, so when I
> got to convert a critical section from CPython to Cython, these language
> features appear to need more than just type annotations.

They are non-trivial to implement, but certainly on our todo list. I
doubt by 0.13.1 though.

- Robert

From robertwb at math.washington.edu  Thu Nov  4 06:19:48 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 3 Nov 2010 22:19:48 -0700
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <AANLkTikpWy1GW-ht-37faSvb3gTkpGPnOkycC9qY-Wyn@mail.gmail.com>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>
	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>
	<AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>
	<AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>
	<4CD11C30.9090103@student.matnat.uio.no>
	<AANLkTimABt0xae_NBG1MVhLGx5-AMf4ps4m7jT2bcQQC@mail.gmail.com>
	<AANLkTi=5DttGcLp6SEv0w06YVratHpc_CSdCSpoiU-rD@mail.gmail.com>
	<AANLkTikpWy1GW-ht-37faSvb3gTkpGPnOkycC9qY-Wyn@mail.gmail.com>
Message-ID: <AANLkTi=k_MKp0czueQ+w6JE+60Jtbkg=+9-TNE2L=wnu@mail.gmail.com>

On Wed, Nov 3, 2010 at 8:51 PM, Brett Calcott <brett.calcott at gmail.com> wrote:
> On 4 November 2010 14:34, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>> On Wed, Nov 3, 2010 at 8:24 PM, Brett Calcott <brett.calcott at gmail.com> wrote:
>>> On 3 November 2010 19:24, Dag Sverre Seljebotn
>>> <dagss at student.matnat.uio.no> wrote:
>>>> cdef class A:
>>>> ? ? cdef Vector obj
>>>> ? ? x = delegate_property(obj, 'x')
>>>>
>>>> or similar, without throwing in new syntax. But it may be too late and I
>>>> don't care too much. Remember that "x = property(...)" is already in Python.
>>>>
>>>
>>> This looks like an interesting option, especially if it was possible
>>> to write your own reusable "xxxx_property" functions that did "the
>>> right thing" for the particular object/library being wrapped. This
>>> would remove a lot boilerplate code that is normally cut and pasted
>>> for setter/getter functions. So, in a case where the C++ members are
>>> private, but access is provided via camelcase set/get functions,
>>> instead of
>>>
>>> property value:
>>> ?def __get__(self):
>>> ? ?return self.obj.getValue()
>>> ?def __set__(self, int x):
>>> ? ?self.obj.setValue(x)
>>>
>>> We could write:
>>> x = camelcase_delegate_property(self.obj, 'value')
>>
>> The one thing that I don't like about this is putting the attribute in
>> a String.
>>
>
> Is this possible?
> x = member_property(self.obj.x)
> y = getset_property(self.obj.getY, self.obj.setY)
> (the names of these need work I agree)
>
> But I take it that the general idea would be able to roll your own
> when there was some repetitive thing you needed to do, so up to you
> what you passed.
>
> For example, the obj pointer in some of the classes I am currently
> wrapping can sometimes be set to NULL. So in many cases the property
> access needs to check that first, and raise an exception. If I could
> write my own getset_with_check_property(...), that would be very cool.

If you need more than something simple, is the

property x:
    def __get__(self):
        [some logic here]
        return result

too verbose for you? We're not trying to be perl with magic one-liners.

- Robert

From stefan_ml at behnel.de  Thu Nov  4 06:42:37 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 04 Nov 2010 06:42:37 +0100
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTikYRv6CPTbSifo-C_gAwU-AG0REC1sPwAHq2DJO@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>	<AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>	<AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>	<AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>	<AANLkTikSi2Awwv31sx3A6BJy=WNSYROo+ceecoygQ0Fk@mail.gmail.com>	<AANLkTi=bFQwdofb4TPcV53b9w0RtcshvC8Uiv2mJa9i-@mail.gmail.com>	<4CD0FA29.7000201@behnel.de>	<AANLkTinNGspAzLoOBdoXcUZgxkxR86i4aQfS2FMrgVYR@mail.gmail.com>	<AANLkTi=a8SMk+u7JfLKxujR3p2ZVnjLsSPjrMY8hufbt@mail.gmail.com>	<AANLkTimjYc=4HdLE+VDmUyn0piXPWGZ7zWGqQ0QoabkP@mail.gmail.com>	<AANLkTim0dL=7nR6WiXAYQyMUSyQ=_b7jsBuKCQ8zh-WT@mail.gmail.com>	<AANLkTim6qdY6hmj2VVJJ_rwje5Sup5NQjspZuymVEv8Z@mail.gmail.com>	<AANLkTimWfQO1SsVgDX9yyex=xivq0z7nmw0PJoJBXvFc@mail.gmail.com>	<4CD1C983.7000300@behnel.de>	<AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>	<4CD1D787.4010202@behnel.de>
	<AANLkTikYRv6CPTbSifo-C_gAwU-AG0REC1sPwAHq2DJO@mail.gmail.com>
Message-ID: <4CD247CD.705@behnel.de>

Vitja Makarov, 03.11.2010 23:03:
> 2010/11/4 Stefan Behnel:
>> Vitja Makarov, 03.11.2010 22:13:
>>> Now it seems to me that class declaration is more like function call
>>> with special case for metaclass keyword and positional arguments:
>>>
>>> class Base(type):
>>>       def __new__(cls, name, bases, attrs, **kwargs):
>>>           print(cls, name, bases, attrs, kwargs)
>>>
>>> class Foo(1, 2, 3, metaclass=Base, foo='bar'):
>>>       def hello(self):
>>>           pass
>>
>> Yes, I think that's exactly how this should work. Want to give it another try?
>
> Yes, I want to finish this part ;)

Great! :)

BTW, thanks for doing all this. Even your first patch was pretty good for a 
"first patch".


>> It seems that the number of positional arguments is fixed, but you can pass
>> any number of keyword arguments, out of which only "metaclass" is special
>> cased and removed (from a copy of the dict, BTW). There is also an
>> additional protocol if the metaclass has a "__prepare__" class method. See
>> "__build_class__".
>
> Yes, all positional arguments are turned into bases. So we can make it this way:
>
> Pyx_CreateClass(PyObject *bases, PyObject *dict, PyObject *name,
> PyObject *modname, PyObject *kwargs);
>
> And then
>
> args = (metaclass, bases, name, dict)
> PyEval_CallObjectWithKeywords(metaclass, args, kwargs)

Except for the first 'metaclass' in args (args has length 3). Again, see 
the __build_class__ implementation.


>> I think the Python 2 protocol in Cython should work as in Py3. Metaclasses
>> in Py2 won't generally support kwargs, but if a user explicitly provides
>> them, it's best to assume that that's intentional. In the worst case, there
>> will be a runtime TypeError.
>
> Do you mean that new syntax should be used for Py2?

No, just what happens behind the syntax. A __metaclass__ field can easily 
be mapped to a metaclass keyword argument (or the equivalent representation 
on a PyClassDefNode in Cython) and from that point on, it's the same thing 
in both cases.


>> So, kwargs should pass through, except for __metaclass__. If both the
>> __metaclass__ class field and the metaclass kwarg are passed, I'd lean
>> towards ignoring the field and prefer the kwarg, just like Py3 does. It can
>> be argued that this is worth a warning, though.
>>
> Py3 doesn't support __metaclass__ attribute, so if metaclass is set that
> seems to be mostly Py3 so __metaclass__ should be ignored.

If I'm not mistaken, the __metaclass__ field can always be handled by the 
compiler during type analysis (or maybe in a transform, see the classes in 
ParseTreeTransforms.py, for example). You can't dynamically add an 
attribute to a class at definition time without letting the parser see its 
name. So you just have to check the class body and you'll know if the field 
is there or not.

Cython also has a Python 3 syntax mode (look out for "language_level") in 
which we can disable this field lookup, so that you get semantic 
equivalence with Python 3 even before the code generation phase. So we 
don't need any runtime support for the __metaclass__ field, not even in Py2.


>> I also think that the parser should extract the metaclass keyword, not the
>> runtime code. So, if provided, it should be passed into __Pyx_CreateClass()
>> as an argument and not in the kwargs.
>
> I think parser can't extract metaclass keyword, try this code:
>
> class Base(type):
>      def __new__(cls, name, bases, attrs, **kwargs):
>          print(cls, name, bases, attrs, kwargs)
>
> myargs={'metaclass': Base}
> class Foo(1, 2, 3, **myargs):
>      def hello(self):
>          pass

Well, it can if it's passed explicitly, though.

We already need to support an explicit metaclass to handle the 
__metaclass__ field. In most cases, the keyword argument will be spelled 
out explicitly, so we can reduce the runtime overhead in that case. But 
that sounds like an optimisation, not a required feature.

Stefan

From stefan_ml at behnel.de  Thu Nov  4 06:54:39 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 04 Nov 2010 06:54:39 +0100
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <AANLkTikpWy1GW-ht-37faSvb3gTkpGPnOkycC9qY-Wyn@mail.gmail.com>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>	<AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>	<AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>	<4CD11C30.9090103@student.matnat.uio.no>	<AANLkTimABt0xae_NBG1MVhLGx5-AMf4ps4m7jT2bcQQC@mail.gmail.com>	<AANLkTi=5DttGcLp6SEv0w06YVratHpc_CSdCSpoiU-rD@mail.gmail.com>
	<AANLkTikpWy1GW-ht-37faSvb3gTkpGPnOkycC9qY-Wyn@mail.gmail.com>
Message-ID: <4CD24A9F.70602@behnel.de>

Brett Calcott, 04.11.2010 04:51:
> On 4 November 2010 14:34, Robert Bradshaw wrote:
>> On Wed, Nov 3, 2010 at 8:24 PM, Brett Calcott wrote:
>>> On 3 November 2010 19:24, Dag Sverre Seljebotn  wrote:
>>>> cdef class A:
>>>>      cdef Vector obj
>>>>      x = delegate_property(obj, 'x')
>>>>
>>>> or similar, without throwing in new syntax. But it may be too late and I
>>>> don't care too much. Remember that "x = property(...)" is already in Python.
>>>
>>> This looks like an interesting option, especially if it was possible
>>> to write your own reusable "xxxx_property" functions that did "the
>>> right thing" for the particular object/library being wrapped. This
>>> would remove a lot boilerplate code that is normally cut and pasted
>>> for setter/getter functions. So, in a case where the C++ members are
>>> private, but access is provided via camelcase set/get functions,
>>> instead of
>>>
>>> property value:
>>>   def __get__(self):
>>>     return self.obj.getValue()
>>>   def __set__(self, int x):
>>>     self.obj.setValue(x)
>>>
>>> We could write:
>>> x = camelcase_delegate_property(self.obj, 'value')
>>
>> The one thing that I don't like about this is putting the attribute in
>> a String.
>
> Is this possible?
>     x = member_property(self.obj.x)
>     y = getset_property(self.obj.getY, self.obj.setY)

The latter could be mapped to Python's own property() with a bit of minor 
magic, I think. However, we don't currently implement property() ourselves, 
so a builtin property transmogrification transform would be the first step.

http://trac.cython.org/cython_trac/ticket/264

Apart from that, I agree with Robert that the parser should see the name of 
the field, and that for anything more fancy than a simple expression, the 
spelled-out syntax is the way to go.

Stefan

From vitja.makarov at gmail.com  Thu Nov  4 07:18:15 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 4 Nov 2010 09:18:15 +0300
Subject: [Cython] Cython 0.13.1
In-Reply-To: <AANLkTikB49qKgLNJKN5HnbF7S1+t_yvA9ntMNdofwOs1@mail.gmail.com>
References: <AANLkTin0LA4pjJfDMogXYYqKxOT9HHrANsFQtnHAvsp-@mail.gmail.com>
	<AANLkTimwab4WPVtuOmp6ahrvobPBR17CUq==QJhaGS+2@mail.gmail.com>
	<AANLkTikB49qKgLNJKN5HnbF7S1+t_yvA9ntMNdofwOs1@mail.gmail.com>
Message-ID: <AANLkTint2j_j5wSm8wENPAtWC57iTnfb9Kc5_hR3fLCe@mail.gmail.com>

What about closure for classes and nested classes?

I think it should be easy to implement when function closures are done

def gen():
    a = 1
    class A:
         def __call__(self):
             return a
    return A
print gen()()()

class A:
    class B:
        pass

2010/11/4 Robert Bradshaw <robertwb at math.washington.edu>:
> On Wed, Nov 3, 2010 at 9:05 PM, Dan Stromberg <strombrg at gmail.com> wrote:
>>
>> On Wed, Nov 3, 2010 at 10:18 AM, Robert Bradshaw
>> <robertwb at math.washington.edu> wrote:
>>>
>>> I think it's time we start thinking about rolling out another release.
>>
>> Not a code contribution, so I don't know how much weight this'll carry, but
>> the two features I miss the most in Cython are yield, and generator
>> expressions - in that order.
>
> The good news is that these are, essentially, one feature.
>
>> I use them pretty often in CPython, so when I
>> got to convert a critical section from CPython to Cython, these language
>> features appear to need more than just type annotations.
>
> They are non-trivial to implement, but certainly on our todo list. I
> doubt by 0.13.1 though.
>
> - Robert
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From robertwb at math.washington.edu  Thu Nov  4 07:35:44 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 3 Nov 2010 23:35:44 -0700
Subject: [Cython] Cython 0.13.1
In-Reply-To: <AANLkTint2j_j5wSm8wENPAtWC57iTnfb9Kc5_hR3fLCe@mail.gmail.com>
References: <AANLkTin0LA4pjJfDMogXYYqKxOT9HHrANsFQtnHAvsp-@mail.gmail.com>
	<AANLkTimwab4WPVtuOmp6ahrvobPBR17CUq==QJhaGS+2@mail.gmail.com>
	<AANLkTikB49qKgLNJKN5HnbF7S1+t_yvA9ntMNdofwOs1@mail.gmail.com>
	<AANLkTint2j_j5wSm8wENPAtWC57iTnfb9Kc5_hR3fLCe@mail.gmail.com>
Message-ID: <AANLkTikNx8Ts42DFZESGE-sMxEf2a6JO-nOQ-S3adKwH@mail.gmail.com>

On Wed, Nov 3, 2010 at 11:18 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> What about closure for classes and nested classes?
>
> I think it should be easy to implement when function closures are done

Function closures were done as part of the last release. Your
metaclass refactoring should make something like this much easier to
implement (make the dict, construct the class).

- Robert

> def gen():
> ? ?a = 1
> ? ?class A:
> ? ? ? ? def __call__(self):
> ? ? ? ? ? ? return a
> ? ?return A
> print gen()()()
>
> class A:
> ? ?class B:
> ? ? ? ?pass
>
> 2010/11/4 Robert Bradshaw <robertwb at math.washington.edu>:
>> On Wed, Nov 3, 2010 at 9:05 PM, Dan Stromberg <strombrg at gmail.com> wrote:
>>>
>>> On Wed, Nov 3, 2010 at 10:18 AM, Robert Bradshaw
>>> <robertwb at math.washington.edu> wrote:
>>>>
>>>> I think it's time we start thinking about rolling out another release.
>>>
>>> Not a code contribution, so I don't know how much weight this'll carry, but
>>> the two features I miss the most in Cython are yield, and generator
>>> expressions - in that order.
>>
>> The good news is that these are, essentially, one feature.
>>
>>> I use them pretty often in CPython, so when I
>>> got to convert a critical section from CPython to Cython, these language
>>> features appear to need more than just type annotations.
>>
>> They are non-trivial to implement, but certainly on our todo list. I
>> doubt by 0.13.1 though.
>>
>> - Robert
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From brett.calcott at gmail.com  Thu Nov  4 07:47:07 2010
From: brett.calcott at gmail.com (Brett Calcott)
Date: Thu, 4 Nov 2010 17:47:07 +1100
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <AANLkTi=k_MKp0czueQ+w6JE+60Jtbkg=+9-TNE2L=wnu@mail.gmail.com>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>
	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>
	<AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>
	<AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>
	<4CD11C30.9090103@student.matnat.uio.no>
	<AANLkTimABt0xae_NBG1MVhLGx5-AMf4ps4m7jT2bcQQC@mail.gmail.com>
	<AANLkTi=5DttGcLp6SEv0w06YVratHpc_CSdCSpoiU-rD@mail.gmail.com>
	<AANLkTikpWy1GW-ht-37faSvb3gTkpGPnOkycC9qY-Wyn@mail.gmail.com>
	<AANLkTi=k_MKp0czueQ+w6JE+60Jtbkg=+9-TNE2L=wnu@mail.gmail.com>
Message-ID: <AANLkTik9OEfmg=wUYNNrcGURTA8hjXCt-eUPZMmZpWsh@mail.gmail.com>

On 4 November 2010 16:19, Robert Bradshaw <robertwb at math.washington.edu> wrote:
> On Wed, Nov 3, 2010 at 8:51 PM, Brett Calcott <brett.calcott at gmail.com> wrote:
>> On 4 November 2010 14:34, Robert Bradshaw <robertwb at math.washington.edu> wrote:
>>> On Wed, Nov 3, 2010 at 8:24 PM, Brett Calcott <brett.calcott at gmail.com> wrote:
>>>> On 3 November 2010 19:24, Dag Sverre Seljebotn
>>>> <dagss at student.matnat.uio.no> wrote:
>>>>> cdef class A:
>>>>> ? ? cdef Vector obj
>>>>> ? ? x = delegate_property(obj, 'x')
>>>>>
>>>>> or similar, without throwing in new syntax. But it may be too late and I
>>>>> don't care too much. Remember that "x = property(...)" is already in Python.
>>>>>
>>>>
>>>> This looks like an interesting option, especially if it was possible
>>>> to write your own reusable "xxxx_property" functions that did "the
>>>> right thing" for the particular object/library being wrapped. This
>>>> would remove a lot boilerplate code that is normally cut and pasted
>>>> for setter/getter functions. So, in a case where the C++ members are
>>>> private, but access is provided via camelcase set/get functions,
>>>> instead of
>>>>
>>>> property value:
>>>> ?def __get__(self):
>>>> ? ?return self.obj.getValue()
>>>> ?def __set__(self, int x):
>>>> ? ?self.obj.setValue(x)
>>>>
>>>> We could write:
>>>> x = camelcase_delegate_property(self.obj, 'value')
>>>
>>> The one thing that I don't like about this is putting the attribute in
>>> a String.
>>>
>>
>> Is this possible?
>> x = member_property(self.obj.x)
>> y = getset_property(self.obj.getY, self.obj.setY)
>> (the names of these need work I agree)
>>
>> But I take it that the general idea would be able to roll your own
>> when there was some repetitive thing you needed to do, so up to you
>> what you passed.
>>
>> For example, the obj pointer in some of the classes I am currently
>> wrapping can sometimes be set to NULL. So in many cases the property
>> access needs to check that first, and raise an exception. If I could
>> write my own getset_with_check_property(...), that would be very cool.
>
> If you need more than something simple, is the
>
> property x:
> ? ?def __get__(self):
> ? ? ? ?[some logic here]
> ? ? ? ?return result
>
> too verbose for you? We're not trying to be perl with magic one-liners.
>

I think we've lost the context of my original question. I'm wrapping a
C++ library. If I just had to write one of these functions, then
indeed trying to do it as a one-liner would be silly. But I'm wrapping
lots of them, and this means I'm just cutting and pasting code, and
then replacing names. When I find myself doing that, I think: there
has to be a better way. The problem with perl one-liners is not that
they are short, but that they are often obscure. But a concise and
obvious one line mapping from a C++ member (or member functions) to a
python property would be clearer, more concise, and easier to modify.
It isn't so much the verbosity that is the problem (though brevity
along with clarify is surely a good thing), it is the robotic
repetition of code that I'm opposed to.

Brett

From stefan_ml at behnel.de  Thu Nov  4 09:01:12 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 04 Nov 2010 09:01:12 +0100
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <AANLkTik9OEfmg=wUYNNrcGURTA8hjXCt-eUPZMmZpWsh@mail.gmail.com>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>	<AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>	<AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>	<4CD11C30.9090103@student.matnat.uio.no>	<AANLkTimABt0xae_NBG1MVhLGx5-AMf4ps4m7jT2bcQQC@mail.gmail.com>	<AANLkTi=5DttGcLp6SEv0w06YVratHpc_CSdCSpoiU-rD@mail.gmail.com>	<AANLkTikpWy1GW-ht-37faSvb3gTkpGPnOkycC9qY-Wyn@mail.gmail.com>	<AANLkTi=k_MKp0czueQ+w6JE+60Jtbkg=+9-TNE2L=wnu@mail.gmail.com>
	<AANLkTik9OEfmg=wUYNNrcGURTA8hjXCt-eUPZMmZpWsh@mail.gmail.com>
Message-ID: <4CD26848.2040805@behnel.de>

Brett Calcott, 04.11.2010 07:47:
> On 4 November 2010 16:19, Robert Bradshaw wrote:
>> On Wed, Nov 3, 2010 at 8:51 PM, Brett Calcott wrote:
>>> Is this possible?
>>> x = member_property(self.obj.x)
>>> y = getset_property(self.obj.getY, self.obj.setY)
>>> (the names of these need work I agree)
>>>
>>> But I take it that the general idea would be able to roll your own
>>> when there was some repetitive thing you needed to do, so up to you
>>> what you passed.
>>>
>>> For example, the obj pointer in some of the classes I am currently
>>> wrapping can sometimes be set to NULL. So in many cases the property
>>> access needs to check that first, and raise an exception. If I could
>>> write my own getset_with_check_property(...), that would be very cool.
>>
>> If you need more than something simple, is the
>>
>> property x:
>>     def __get__(self):
>>         [some logic here]
>>         return result
>>
>> too verbose for you? We're not trying to be perl with magic one-liners.
>
> I think we've lost the context of my original question. I'm wrapping a
> C++ library. If I just had to write one of these functions, then
> indeed trying to do it as a one-liner would be silly. But I'm wrapping
> lots of them, and this means I'm just cutting and pasting code, and
> then replacing names. When I find myself doing that, I think: there
> has to be a better way. The problem with perl one-liners is not that
> they are short, but that they are often obscure. But a concise and
> obvious one line mapping from a C++ member (or member functions) to a
> python property would be clearer, more concise, and easier to modify.
> It isn't so much the verbosity that is the problem (though brevity
> along with clarify is surely a good thing), it is the robotic
> repetition of code that I'm opposed to.

You may have missed my other e-mail, but to put it a bit clearer, there are 
three levels where Cython can provide support:

1) map an expression or class field to a property

This is what we were previously discussing in this thread, well in the 
context of your original question.

2) map a pair of setters and getters to a property

This can be done using Python's builtin property() type and further 
optimised (or enabled for C/C++ functions/methods) by Cython.

3) map more complex things to properties

Use the existing property syntax in Cython. It's a bit verbose for some 
special cases, but usually, special cases are not special enough to break 
the rules, unless practicality really beats purity. Point 1) above may or 
may not be seen as such a case.

IMHO, only point 1) is worth discussing as it appears to require new 
syntax. 2) simply needs an implementation and 3) is usable today.

Stefan

From stefan_ml at behnel.de  Thu Nov  4 09:10:36 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 04 Nov 2010 09:10:36 +0100
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <AANLkTikua0kipkGTE0B4VLQUAPi7K5+b11-7Q-TEzJT4@mail.gmail.com>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>	<AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>	<AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>	<4CD0EF5C.1080607@behnel.de>	<AANLkTikTk7UruwF0gjrDU-Q_Y2xUSj7xQD7aoFuJ68v6@mail.gmail.com>	<4CD10792.3000507@behnel.de>
	<AANLkTikua0kipkGTE0B4VLQUAPi7K5+b11-7Q-TEzJT4@mail.gmail.com>
Message-ID: <4CD26A7C.6070408@behnel.de>

Robert Bradshaw, 03.11.2010 08:25:
> On Tue, Nov 2, 2010 at 11:56 PM, Stefan Behnel wrote:
>> Robert Bradshaw, 03.11.2010 07:29:
>>> On that note, should we
>>> require an explicit self.obj.x, or is self implicit? (The latter makes
>>> it impossible to refer to non-attributes, and is bad according to the
>>> zen of Python...), so I'd lean against this one.
>>
>> That's a very good idea. Makes it more readable and a lot clearer what
>> happens. This also removes the link to C++ as you can access any attribute
>> of the type in one way or another.

In another post, I wrote that "special cases are not special enough to 
break the rules", and it made me change my opinion here. It's normal Python 
behaviour to look up names in either the current or the global namespace, 
and potentially in a surrounding closure. The same should apply here.

This means that I'm no longer a supporter of requiring "self" here. "self" 
may be a common convention, but it is not actually defined in the context 
of the property statement. I think it should be

     cdef class MyType:
         cdef SomeStruct the_struct

         property i from the_struct.i

where "the_struct" is looked up in the class scope or surrounding scope, 
i.e. the property expression simply gets evaluated in the normal chain of 
scopes in which it is defined.

Stefan

From stefan_ml at behnel.de  Thu Nov  4 09:39:59 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 04 Nov 2010 09:39:59 +0100
Subject: [Cython] Cython 0.13.1
In-Reply-To: <AANLkTimNwGP57A4S1Fr67m_sADJdCaBVCq0a2i7CC11k@mail.gmail.com>
References: <AANLkTin0LA4pjJfDMogXYYqKxOT9HHrANsFQtnHAvsp-@mail.gmail.com>	<4CD1A956.4020507@behnel.de>	<AANLkTimFRoVkK7v=Kw37o+OBXxwkML18Fhpdaa9WVNtb@mail.gmail.com>	<4CD1C615.6040807@behnel.de>
	<AANLkTimNwGP57A4S1Fr67m_sADJdCaBVCq0a2i7CC11k@mail.gmail.com>
Message-ID: <4CD2715F.7070909@behnel.de>

Brett Calcott, 04.11.2010 04:29:
> On 4 November 2010 07:29, Stefan Behnel wrote:
>>> What is about new property syntax?
>>
>> http://thread.gmane.org/gmane.comp.python.cython.devel/10411/focus=10459
>
> +1 for this (I started the thread)
>
> Happy to try and help. But my cython powers are not (yet) strong.

That's just fine, neither were mine when I first started digging into Pyrex 
to make it do what I wanted it to (*long* before Robert and I renamed 
"Pyrex + a bunch of patches" to Cython). It certainly helps to have a clear 
goal.

I think the discussion is slowly coming to an end. Once it's clear (enough) 
what needs to be done, we will support anyone who wants to implement it. 
Currently, it looks like there are two things that can be done, see my 
latest posts in that thread.

Stefan

From Paul.Connell at disney.com  Thu Nov  4 10:36:03 2010
From: Paul.Connell at disney.com (Connell, Paul)
Date: Thu, 4 Nov 2010 09:36:03 +0000
Subject: [Cython] Cython debugger
In-Reply-To: <AANLkTi=WSoP5wkC9+0cmSwSCPfLAKFghO9OSwkSx+peb@mail.gmail.com>
References: <AANLkTinjhVuH=vqA=EFySaBKh+j6Rk1U-Gn0qDLLoUaB@mail.gmail.com>
	<4C8F3E56.2020705@student.matnat.uio.no>
	<1284482379.13330.333.camel@radiator.bos.redhat.com>
	<AANLkTimzNdwzsRo6vAymnMfAH+uBU7UG+OqPSF=Ugrp2@mail.gmail.com>
	<AANLkTikOF4j_7v-AzHFaT=vzp+RTYxEFOBGc1M8HC9Wz@mail.gmail.com>
	<AANLkTi=WSoP5wkC9+0cmSwSCPfLAKFghO9OSwkSx+peb@mail.gmail.com>
Message-ID: <DA536BC5C16ADD4D850A9522DB86E01E51C1D4FBF9@SM-GBLO-VXM01A.emea.wdpr.disney.com>

Hi

I just pulled the tip out of hg.cython.org/cython-gdb as I?m excited to try this out.

However, when running ?cython ?cplus ?debug? on a pyx that I know is correct and works, I always get the following output:

  File "C:\Python26\lib\xml\etree\ElementTree.py", line 777, in _raise_serialization_error
    "cannot serialize %r (type %s)" % (text, type(text).__name__)
TypeError: cannot serialize None (type NoneType)


I have not yet had the opportunity to try reducing my pyx (it is fairly large and getting larger every hour), but is this any kind of known problem ? what might be causing it?

Here?s the full traceback:

  File "/projects/ctg/build/bin/cython.py", line 3, in <module>
    main(command_line = 1)
  File "C:\Python26\lib\site-packages\Cython\Compiler\Main.py", line 780, in main
    result = compile(sources, options)
  File "C:\Python26\lib\site-packages\Cython\Compiler\Main.py", line 755, in compile
    return compile_multiple(source, options)
  File "C:\Python26\lib\site-packages\Cython\Compiler\Main.py", line 727, in compile_multiple
    result = run_pipeline(source, options)
  File "C:\Python26\lib\site-packages\Cython\Compiler\Main.py", line 596, in run_pipeline
    err, enddata = context.run_pipeline(pipeline, source)
  File "C:\Python26\lib\site-packages\Cython\Compiler\Main.py", line 237, in run_pipeline
    data = phase(data)
  File "C:\Python26\lib\site-packages\Cython\Compiler\Main.py", line 160, in generate_pyx_code
    module_node.process_implementation(options, result)
  File "C:\Python26\lib\site-packages\Cython\Compiler\ModuleNode.py", line 72, in process_implementation
    self.generate_c_code(env, options, result)
  File "C:\Python26\lib\site-packages\Cython\Compiler\ModuleNode.py", line 302,in generate_c_code
    self._serialize_lineno_map(env, rootwriter)
  File "C:\Python26\lib\site-packages\Cython\Compiler\ModuleNode.py", line 327,in _serialize_lineno_map
    tb.serialize()
  File "C:\Python26\lib\site-packages\Cython\Debugger\DebugWriter.py", line 71,in serialize
    et.write(os.path.join(self.output_dir, fn), encoding="UTF-8", **kw)
  File "C:\Python26\lib\xml\etree\ElementTree.py", line 663, in write
    self._write(file, self._root, encoding, {})
  File "C:\Python26\lib\xml\etree\ElementTree.py", line 707, in _write
    self._write(file, n, encoding, namespaces)
  File "C:\Python26\lib\xml\etree\ElementTree.py", line 698, in _write
    _escape_attrib(v, encoding)))
  File "C:\Python26\lib\xml\etree\ElementTree.py", line 830, in _escape_attrib
    _raise_serialization_error(text)
  File "C:\Python26\lib\xml\etree\ElementTree.py", line 777, in _raise_serializa
tion_error
    "cannot serialize %r (type %s)" % (text, type(text).__name__)
TypeError: cannot serialize None (type NoneType)

Regards,
Paul

From: cython-dev-bounces at codespeak.net [mailto:cython-dev-bounces at codespeak.net] On Behalf Of mark florisson
Sent: 03 November 2010 09:56
To: cython-dev at codespeak.net
Subject: Re: [Cython] Cython debugger

Most of the Cython debugger is implemented, it can be pulled from hg.cython.org/cython-gdb<http://hg.cython.org/cython-gdb>. To use it, you need Cython to export some debug information, which can be done using  'cython --debug mymod.pyx', or 'python setup.py build_ext --pyrex-debug'. You can also pass 'pyrex_debug=True' as an argument to Cython.Distutils.extension.Extension.
You can then start gdb by typing 'cygdb'  (which should be installed as a script) in your project directory. Then 'help cy' gives an overview of commands. It currently supports breapoints, stepping, stepping-over, backtraces, source code listing, going up and down the stack, printing variables with regard to context, listing locals or globals and running and continuing the program. If pygments is installed it will colorize source code which is configurable through cy_* parameters.

It basically works on three levels: the Python, Cython and C level. Depending on the context cygdb does the right thing. For stepping it considers the following stack frames relevant: any Python frame, any Cython frame and any C frame from a C function called directly by Cython user-code.

So, it would be great if some people could test and try it (unit tests are written but some system testing is always great). It works with gdb 7.2 (the 7.1 python api was too incomplete and broken). So if you guys like it I could write documentation that explains to Cython users how to install and use it. Any suggestion or criticism is most welcome!

The Python support (libpython.py) was also modified. I'd like to push this mainstream (and process any suggestions and criticism) before supplying a patch to Python.

Kind regards,

Mark

On 15 September 2010 17:49, Robert Bradshaw <robertwb at math.washington.edu<mailto:robertwb at math.washington.edu>> wrote:
On Wed, Sep 15, 2010 at 2:55 AM, mark florisson
<markflorisson88 at gmail.com<mailto:markflorisson88 at gmail.com>> wrote:

>> It ought to be possible to do something similar with cython code.  It
>> may not even be necessary to modify cython: perhaps some searching for
>> locals named "__pyx_*" iirc would get you 70% of the way there?
>
> Although that sounds like a wonderful idea, I think there are also
> issues with that. One issue is that a user must be able to set Cython
> breakpoints before the Cython module would be loaded, and for that the
> symbol name would be needed beforehand. Also, I don't know if these
> mangled names are consistent now and in the future and if you would be
> able to unambiguously associate a Cython variable name with a mangled
> name.
Mangled names are deterministic and, though they're not guaranteed to
be consistent from release to release, almost always are.

>> I can attest that having the prettyprinters enabled does make it much
>> easier to debug cython code: all of the PyObject* get prettyprinted.
>
> I've been looking at the code and this is pretty neat. I did encounter
> some issues, for instance if you load the script before loading the
> python interpreter you get this traceback because these types are not
> defined at that time:
>
> Traceback (most recent call last):
>  File "<string>", line 1, in <module>
>  File ".libpython.py", line 49, in <module>
>    _type_size_t = gdb.lookup_type('size_t')
> RuntimeError: No type named size_t.
>
> So I think it would be a good idea to not make that code module-level.
>
>>
>> One other thought: if it's possibly to expose the cython structures in
>> some meaningful way, perhaps we could change upstream python's gdb hooks
>> to simply integrate them into the py-* commands I mentioned above? (so
>> e.g. cython c functions get somehow treated as python frames; currently
>> I have a test predicate:
>>  Frame.is_evalframeex()
>> which perhaps could be generalized?)
>>
>> (Not sure; it would complicate the selftests within python itself)
>>
>
> I think it would be hard to make them actual Python frames because
> creating frames in the inferior process from gdb is probably quite
> dangerous, and the alternative would be to modify Cython so that it
> creates Python stack frames (this sounds feasible but I think it might
> be a little bit of work). However, if this could be done (it would
> only do so if this 'debug' flag is active), then tracebacks and locals
> inspection etc wouldn't need special attention and the code would
> appear as normal Python code (apart from the Code objects obviously).
> However, this would form a problem for non-primitive C-type Cython
> variables.
>
> So at the very least we could have a 'py-locals' or some such command
> that would show the value of all the locals (the Python locals would
> be printed by py-print and C locals by gdb print). For regular python
> code it would show the locals from the current stack frame. For the
> Cython part to work we would need information from the Cython compiler
> because we wouldn't want to list any temporary or irrelevant
> variables.
Yes, this is what I was thinking, at least in terms of exposing stuff
to pdb (which is a complementary project). BTW, frames are already
created for functions when profiling is enabled.

> So I think we should be able to integrate these two projects into one
> fruitful project, and with proper documentation it could help both
> regular Python users and Cython users.
+1

- Robert
_______________________________________________
Cython-dev mailing list
Cython-dev at codespeak.net<mailto:Cython-dev at codespeak.net>
http://codespeak.net/mailman/listinfo/cython-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20101104/e04ad9b8/attachment-0001.htm 

From brett.calcott at gmail.com  Thu Nov  4 10:59:53 2010
From: brett.calcott at gmail.com (Brett Calcott)
Date: Thu, 4 Nov 2010 20:59:53 +1100
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <4CD26848.2040805@behnel.de>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>
	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>
	<AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>
	<AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>
	<4CD11C30.9090103@student.matnat.uio.no>
	<AANLkTimABt0xae_NBG1MVhLGx5-AMf4ps4m7jT2bcQQC@mail.gmail.com>
	<AANLkTi=5DttGcLp6SEv0w06YVratHpc_CSdCSpoiU-rD@mail.gmail.com>
	<AANLkTikpWy1GW-ht-37faSvb3gTkpGPnOkycC9qY-Wyn@mail.gmail.com>
	<AANLkTi=k_MKp0czueQ+w6JE+60Jtbkg=+9-TNE2L=wnu@mail.gmail.com>
	<AANLkTik9OEfmg=wUYNNrcGURTA8hjXCt-eUPZMmZpWsh@mail.gmail.com>
	<4CD26848.2040805@behnel.de>
Message-ID: <AANLkTikHZkf343pPeS-j4QceM6S-F4bG=Zjz_c8AKVfF@mail.gmail.com>

On 4 November 2010 19:01, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Brett Calcott, 04.11.2010 07:47:
>> On 4 November 2010 16:19, Robert Bradshaw wrote:
>>> On Wed, Nov 3, 2010 at 8:51 PM, Brett Calcott wrote:
>>>> Is this possible?
>>>> x = member_property(self.obj.x)
>>>> y = getset_property(self.obj.getY, self.obj.setY)
>>>> (the names of these need work I agree)
>>>>
>>>> But I take it that the general idea would be able to roll your own
>>>> when there was some repetitive thing you needed to do, so up to you
>>>> what you passed.
>>>>
>>>> For example, the obj pointer in some of the classes I am currently
>>>> wrapping can sometimes be set to NULL. So in many cases the property
>>>> access needs to check that first, and raise an exception. If I could
>>>> write my own getset_with_check_property(...), that would be very cool.
>>>
>>> If you need more than something simple, is the
>>>
>>> property x:
>>> ? ? def __get__(self):
>>> ? ? ? ? [some logic here]
>>> ? ? ? ? return result
>>>
>>> too verbose for you? We're not trying to be perl with magic one-liners.
>>
>> I think we've lost the context of my original question. I'm wrapping a
>> C++ library. If I just had to write one of these functions, then
>> indeed trying to do it as a one-liner would be silly. But I'm wrapping
>> lots of them, and this means I'm just cutting and pasting code, and
>> then replacing names. When I find myself doing that, I think: there
>> has to be a better way. The problem with perl one-liners is not that
>> they are short, but that they are often obscure. But a concise and
>> obvious one line mapping from a C++ member (or member functions) to a
>> python property would be clearer, more concise, and easier to modify.
>> It isn't so much the verbosity that is the problem (though brevity
>> along with clarify is surely a good thing), it is the robotic
>> repetition of code that I'm opposed to.
>
> You may have missed my other e-mail, but to put it a bit clearer, there are
> three levels where Cython can provide support:
>
> 1) map an expression or class field to a property
>
> This is what we were previously discussing in this thread, well in the
> context of your original question.
>
> 2) map a pair of setters and getters to a property
>
> This can be done using Python's builtin property() type and further
> optimised (or enabled for C/C++ functions/methods) by Cython.
>
> 3) map more complex things to properties
>
> Use the existing property syntax in Cython. It's a bit verbose for some
> special cases, but usually, special cases are not special enough to break
> the rules, unless practicality really beats purity. Point 1) above may or
> may not be seen as such a case.
>
> IMHO, only point 1) is worth discussing as it appears to require new
> syntax. 2) simply needs an implementation and 3) is usable today.
>

That is nice and clear. So I started talking about 1, but I've
wandered into talking about 3. That seems right. But I think Dag's
suggestion made me think that these cases were really not so
different. As I understand it, all of the cases we're talking about
can currently be handled by the existing property syntax. Dag's
suggestion was to add a specific function (not a syntax addition) to
enable case 1 (above). I thought that something more flexible could be
defined that both enabled 1, and also let users do 3 more easily
themselves. Case 2 would be another form of the same thing. So the
suggestion is allow the ability to write your own specialised
property() functions. Doing so would make cases 1/2/3 all just special
cases of one functionality.

Brett

From markflorisson88 at gmail.com  Thu Nov  4 11:20:17 2010
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 4 Nov 2010 11:20:17 +0100
Subject: [Cython] Cython debugger
In-Reply-To: <DA536BC5C16ADD4D850A9522DB86E01E51C1D4FBF9@SM-GBLO-VXM01A.emea.wdpr.disney.com>
References: <AANLkTinjhVuH=vqA=EFySaBKh+j6Rk1U-Gn0qDLLoUaB@mail.gmail.com>
	<4C8F3E56.2020705@student.matnat.uio.no>
	<1284482379.13330.333.camel@radiator.bos.redhat.com>
	<AANLkTimzNdwzsRo6vAymnMfAH+uBU7UG+OqPSF=Ugrp2@mail.gmail.com>
	<AANLkTikOF4j_7v-AzHFaT=vzp+RTYxEFOBGc1M8HC9Wz@mail.gmail.com>
	<AANLkTi=WSoP5wkC9+0cmSwSCPfLAKFghO9OSwkSx+peb@mail.gmail.com>
	<DA536BC5C16ADD4D850A9522DB86E01E51C1D4FBF9@SM-GBLO-VXM01A.emea.wdpr.disney.com>
Message-ID: <AANLkTin3Eq=5BKhx85W_PF1huyyBJ2EzdXzb+Q_Bv3H=@mail.gmail.com>

Hey Paul,

thanks for the feedback, I hadn't tried the --cplus option yet.
Apparently I was using an attribute (that indicates where to write the
C file) that is set to None when writing C++ files. I changed that and
gave it a quick test, so hopefully it's fixed now :)

Kind regards,

Mark

On 4 November 2010 10:36, Connell, Paul <Paul.Connell at disney.com> wrote:
> Hi
>
>
>
> I just pulled the tip out of hg.cython.org/cython-gdb as I?m excited to try
> this out.
>
>
>
> However, when running ?cython ?cplus ?debug? on a pyx that I know is correct
> and works, I always get the following output:
>
>
>
> ? File "C:\Python26\lib\xml\etree\ElementTree.py", line 777, in
> _raise_serialization_error
>
> ??? "cannot serialize %r (type %s)" % (text, type(text).__name__)
>
> TypeError: cannot serialize None (type NoneType)
>
>
>
>
>
> I have not yet had the opportunity to try reducing my pyx (it is fairly
> large and getting larger every hour), but is this any kind of known problem
> ? what might be causing it?
>
>
>
> Here?s the full traceback:
>
>
>
> ? File "/projects/ctg/build/bin/cython.py", line 3, in <module>
>
> ??? main(command_line = 1)
>
> ? File "C:\Python26\lib\site-packages\Cython\Compiler\Main.py", line 780, in
> main
>
> ??? result = compile(sources, options)
>
> ? File "C:\Python26\lib\site-packages\Cython\Compiler\Main.py", line 755, in
> compile
>
> ??? return compile_multiple(source, options)
>
> ? File "C:\Python26\lib\site-packages\Cython\Compiler\Main.py", line 727, in
> compile_multiple
>
> ??? result = run_pipeline(source, options)
>
> ? File "C:\Python26\lib\site-packages\Cython\Compiler\Main.py", line 596, in
> run_pipeline
>
> ??? err, enddata = context.run_pipeline(pipeline, source)
>
> ? File "C:\Python26\lib\site-packages\Cython\Compiler\Main.py", line 237, in
> run_pipeline
>
> ??? data = phase(data)
>
> ? File "C:\Python26\lib\site-packages\Cython\Compiler\Main.py", line 160, in
> generate_pyx_code
>
> ??? module_node.process_implementation(options, result)
>
> ? File "C:\Python26\lib\site-packages\Cython\Compiler\ModuleNode.py", line
> 72, in process_implementation
>
> ??? self.generate_c_code(env, options, result)
>
> ? File "C:\Python26\lib\site-packages\Cython\Compiler\ModuleNode.py", line
> 302,in generate_c_code
>
> ??? self._serialize_lineno_map(env, rootwriter)
>
> ? File "C:\Python26\lib\site-packages\Cython\Compiler\ModuleNode.py", line
> 327,in _serialize_lineno_map
>
> ??? tb.serialize()
>
> ? File "C:\Python26\lib\site-packages\Cython\Debugger\DebugWriter.py", line
> 71,in serialize
>
> ??? et.write(os.path.join(self.output_dir, fn), encoding="UTF-8", **kw)
>
> ? File "C:\Python26\lib\xml\etree\ElementTree.py", line 663, in write
>
> ??? self._write(file, self._root, encoding, {})
>
> ? File "C:\Python26\lib\xml\etree\ElementTree.py", line 707, in _write
>
> ??? self._write(file, n, encoding, namespaces)
>
> ? File "C:\Python26\lib\xml\etree\ElementTree.py", line 698, in _write
>
> ??? _escape_attrib(v, encoding)))
>
> ? File "C:\Python26\lib\xml\etree\ElementTree.py", line 830, in
> _escape_attrib
>
> ??? _raise_serialization_error(text)
>
> ? File "C:\Python26\lib\xml\etree\ElementTree.py", line 777, in
> _raise_serializa
>
> tion_error
>
> ??? "cannot serialize %r (type %s)" % (text, type(text).__name__)
>
> TypeError: cannot serialize None (type NoneType)
>
>
>
> Regards,
>
> Paul
>
>
>
> From: cython-dev-bounces at codespeak.net
> [mailto:cython-dev-bounces at codespeak.net] On Behalf Of mark florisson
> Sent: 03 November 2010 09:56
> To: cython-dev at codespeak.net
> Subject: Re: [Cython] Cython debugger
>
>
>
> Most of the Cython debugger is implemented, it can be pulled from
> hg.cython.org/cython-gdb. To use it, you need Cython to export some debug
> information, which can be done using ?'cython --debug mymod.pyx', or 'python
> setup.py build_ext --pyrex-debug'. You can also pass 'pyrex_debug=True' as
> an argument to Cython.Distutils.extension.Extension.
>
> You can then start gdb by typing 'cygdb'??(which should be installed as a
> script)?in your project directory. Then 'help cy' gives an overview of
> commands. It currently supports breapoints, stepping, stepping-over,
> backtraces, source code listing, going up and down the stack, printing
> variables with regard to context, listing locals or globals and running and
> continuing the program. If pygments is installed it will colorize source
> code which is configurable through cy_* parameters.
>
>
>
> It basically works on three levels: the Python, Cython and C level.
> Depending on the context cygdb does the right thing. For stepping it
> considers the following stack frames relevant: any Python frame, any Cython
> frame and any C frame from a C function called directly by Cython user-code.
>
>
>
> So, it would be great if some people could test and try it (unit tests are
> written but some system testing is always great). It works with gdb 7.2 (the
> 7.1 python api was too incomplete and broken). So if you guys like it I
> could write documentation that explains to Cython users how to install and
> use it. Any suggestion or criticism is most welcome!
>
>
>
> The Python support (libpython.py) was also modified. I'd like to push this
> mainstream (and process any suggestions and criticism) before supplying a
> patch to Python.
>
>
>
> Kind regards,
>
>
>
> Mark
>
>
>
> On 15 September 2010 17:49, Robert Bradshaw <robertwb at math.washington.edu>
> wrote:
>
> On Wed, Sep 15, 2010 at 2:55 AM, mark florisson
> <markflorisson88 at gmail.com> wrote:
>
>>> It ought to be possible to do something similar with cython code. ?It
>>> may not even be necessary to modify cython: perhaps some searching for
>>> locals named "__pyx_*" iirc would get you 70% of the way there?
>>
>> Although that sounds like a wonderful idea, I think there are also
>> issues with that. One issue is that a user must be able to set Cython
>> breakpoints before the Cython module would be loaded, and for that the
>> symbol name would be needed beforehand. Also, I don't know if these
>> mangled names are consistent now and in the future and if you would be
>> able to unambiguously associate a Cython variable name with a mangled
>> name.
>
> Mangled names are deterministic and, though they're not guaranteed to
> be consistent from release to release, almost always are.
>
>>> I can attest that having the prettyprinters enabled does make it much
>>> easier to debug cython code: all of the PyObject* get prettyprinted.
>>
>> I've been looking at the code and this is pretty neat. I did encounter
>> some issues, for instance if you load the script before loading the
>> python interpreter you get this traceback because these types are not
>> defined at that time:
>>
>> Traceback (most recent call last):
>> ?File "<string>", line 1, in <module>
>> ?File ".libpython.py", line 49, in <module>
>> ? ?_type_size_t = gdb.lookup_type('size_t')
>> RuntimeError: No type named size_t.
>>
>> So I think it would be a good idea to not make that code module-level.
>>
>>>
>>> One other thought: if it's possibly to expose the cython structures in
>>> some meaningful way, perhaps we could change upstream python's gdb hooks
>>> to simply integrate them into the py-* commands I mentioned above? (so
>>> e.g. cython c functions get somehow treated as python frames; currently
>>> I have a test predicate:
>>> ?Frame.is_evalframeex()
>>> which perhaps could be generalized?)
>>>
>>> (Not sure; it would complicate the selftests within python itself)
>>>
>>
>> I think it would be hard to make them actual Python frames because
>> creating frames in the inferior process from gdb is probably quite
>> dangerous, and the alternative would be to modify Cython so that it
>> creates Python stack frames (this sounds feasible but I think it might
>> be a little bit of work). However, if this could be done (it would
>> only do so if this 'debug' flag is active), then tracebacks and locals
>> inspection etc wouldn't need special attention and the code would
>> appear as normal Python code (apart from the Code objects obviously).
>> However, this would form a problem for non-primitive C-type Cython
>> variables.
>>
>> So at the very least we could have a 'py-locals' or some such command
>> that would show the value of all the locals (the Python locals would
>> be printed by py-print and C locals by gdb print). For regular python
>> code it would show the locals from the current stack frame. For the
>> Cython part to work we would need information from the Cython compiler
>> because we wouldn't want to list any temporary or irrelevant
>> variables.
>
> Yes, this is what I was thinking, at least in terms of exposing stuff
> to pdb (which is a complementary project). BTW, frames are already
> created for functions when profiling is enabled.
>
>> So I think we should be able to integrate these two projects into one
>> fruitful project, and with proper documentation it could help both
>> regular Python users and Cython users.
>
> +1
>
> - Robert
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>

From vitja.makarov at gmail.com  Thu Nov  4 12:18:12 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 4 Nov 2010 14:18:12 +0300
Subject: [Cython] metaclasses
In-Reply-To: <4CD247CD.705@behnel.de>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>
	<AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>
	<AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>
	<AANLkTikSi2Awwv31sx3A6BJy=WNSYROo+ceecoygQ0Fk@mail.gmail.com>
	<AANLkTi=bFQwdofb4TPcV53b9w0RtcshvC8Uiv2mJa9i-@mail.gmail.com>
	<4CD0FA29.7000201@behnel.de>
	<AANLkTinNGspAzLoOBdoXcUZgxkxR86i4aQfS2FMrgVYR@mail.gmail.com>
	<AANLkTi=a8SMk+u7JfLKxujR3p2ZVnjLsSPjrMY8hufbt@mail.gmail.com>
	<AANLkTimjYc=4HdLE+VDmUyn0piXPWGZ7zWGqQ0QoabkP@mail.gmail.com>
	<AANLkTim0dL=7nR6WiXAYQyMUSyQ=_b7jsBuKCQ8zh-WT@mail.gmail.com>
	<AANLkTim6qdY6hmj2VVJJ_rwje5Sup5NQjspZuymVEv8Z@mail.gmail.com>
	<AANLkTimWfQO1SsVgDX9yyex=xivq0z7nmw0PJoJBXvFc@mail.gmail.com>
	<4CD1C983.7000300@behnel.de>
	<AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>
	<4CD1D787.4010202@behnel.de>
	<AANLkTikYRv6CPTbSifo-C_gAwU-AG0REC1sPwAHq2DJO@mail.gmail.com>
	<4CD247CD.705@behnel.de>
Message-ID: <AANLkTimEmE9=xiN6=PDNEgRo=P_=ECxE=n7Eh0aYs81g@mail.gmail.com>

Here is my first try with metaclasses in py3. It does not handle
__prepare__ I don't understand what function should I call to create
cell... and what is that func

2010/11/4 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 03.11.2010 23:03:
>> 2010/11/4 Stefan Behnel:
>>> Vitja Makarov, 03.11.2010 22:13:
>>>> Now it seems to me that class declaration is more like function call
>>>> with special case for metaclass keyword and positional arguments:
>>>>
>>>> class Base(type):
>>>> ? ? ? def __new__(cls, name, bases, attrs, **kwargs):
>>>> ? ? ? ? ? print(cls, name, bases, attrs, kwargs)
>>>>
>>>> class Foo(1, 2, 3, metaclass=Base, foo='bar'):
>>>> ? ? ? def hello(self):
>>>> ? ? ? ? ? pass
>>>
>>> Yes, I think that's exactly how this should work. Want to give it another try?
>>
>> Yes, I want to finish this part ;)
>
> Great! :)
>
> BTW, thanks for doing all this. Even your first patch was pretty good for a
> "first patch".
>
>
>>> It seems that the number of positional arguments is fixed, but you can pass
>>> any number of keyword arguments, out of which only "metaclass" is special
>>> cased and removed (from a copy of the dict, BTW). There is also an
>>> additional protocol if the metaclass has a "__prepare__" class method. See
>>> "__build_class__".
>>
>> Yes, all positional arguments are turned into bases. So we can make it this way:
>>
>> Pyx_CreateClass(PyObject *bases, PyObject *dict, PyObject *name,
>> PyObject *modname, PyObject *kwargs);
>>
>> And then
>>
>> args = (metaclass, bases, name, dict)
>> PyEval_CallObjectWithKeywords(metaclass, args, kwargs)
>
> Except for the first 'metaclass' in args (args has length 3). Again, see
> the __build_class__ implementation.
>
>
>>> I think the Python 2 protocol in Cython should work as in Py3. Metaclasses
>>> in Py2 won't generally support kwargs, but if a user explicitly provides
>>> them, it's best to assume that that's intentional. In the worst case, there
>>> will be a runtime TypeError.
>>
>> Do you mean that new syntax should be used for Py2?
>
> No, just what happens behind the syntax. A __metaclass__ field can easily
> be mapped to a metaclass keyword argument (or the equivalent representation
> on a PyClassDefNode in Cython) and from that point on, it's the same thing
> in both cases.
>
>
>>> So, kwargs should pass through, except for __metaclass__. If both the
>>> __metaclass__ class field and the metaclass kwarg are passed, I'd lean
>>> towards ignoring the field and prefer the kwarg, just like Py3 does. It can
>>> be argued that this is worth a warning, though.
>>>
>> Py3 doesn't support __metaclass__ attribute, so if metaclass is set that
>> seems to be mostly Py3 so __metaclass__ should be ignored.
>
> If I'm not mistaken, the __metaclass__ field can always be handled by the
> compiler during type analysis (or maybe in a transform, see the classes in
> ParseTreeTransforms.py, for example). You can't dynamically add an
> attribute to a class at definition time without letting the parser see its
> name. So you just have to check the class body and you'll know if the field
> is there or not.
>
> Cython also has a Python 3 syntax mode (look out for "language_level") in
> which we can disable this field lookup, so that you get semantic
> equivalence with Python 3 even before the code generation phase. So we
> don't need any runtime support for the __metaclass__ field, not even in Py2.
>
>
>>> I also think that the parser should extract the metaclass keyword, not the
>>> runtime code. So, if provided, it should be passed into __Pyx_CreateClass()
>>> as an argument and not in the kwargs.
>>
>> I think parser can't extract metaclass keyword, try this code:
>>
>> class Base(type):
>> ? ? ?def __new__(cls, name, bases, attrs, **kwargs):
>> ? ? ? ? ?print(cls, name, bases, attrs, kwargs)
>>
>> myargs={'metaclass': Base}
>> class Foo(1, 2, 3, **myargs):
>> ? ? ?def hello(self):
>> ? ? ? ? ?pass
>
> Well, it can if it's passed explicitly, though.
>
> We already need to support an explicit metaclass to handle the
> __metaclass__ field. In most cases, the keyword argument will be spelled
> out explicitly, so we can reduce the runtime overhead in that case. But
> that sounds like an optimisation, not a required feature.
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: metaclass-py3.patch
Type: text/x-patch
Size: 11802 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20101104/eb5f145a/attachment.bin 

From Paul.Connell at disney.com  Thu Nov  4 12:25:16 2010
From: Paul.Connell at disney.com (Connell, Paul)
Date: Thu, 4 Nov 2010 11:25:16 +0000
Subject: [Cython] Cython debugger
In-Reply-To: <AANLkTin3Eq=5BKhx85W_PF1huyyBJ2EzdXzb+Q_Bv3H=@mail.gmail.com>
References: <AANLkTinjhVuH=vqA=EFySaBKh+j6Rk1U-Gn0qDLLoUaB@mail.gmail.com>
	<4C8F3E56.2020705@student.matnat.uio.no>
	<1284482379.13330.333.camel@radiator.bos.redhat.com>
	<AANLkTimzNdwzsRo6vAymnMfAH+uBU7UG+OqPSF=Ugrp2@mail.gmail.com>
	<AANLkTikOF4j_7v-AzHFaT=vzp+RTYxEFOBGc1M8HC9Wz@mail.gmail.com>
	<AANLkTi=WSoP5wkC9+0cmSwSCPfLAKFghO9OSwkSx+peb@mail.gmail.com>
	<DA536BC5C16ADD4D850A9522DB86E01E51C1D4FBF9@SM-GBLO-VXM01A.emea.wdpr.disney.com>
	<AANLkTin3Eq=5BKhx85W_PF1huyyBJ2EzdXzb+Q_Bv3H=@mail.gmail.com>
Message-ID: <DA536BC5C16ADD4D850A9522DB86E01E51C1D4FCC8@SM-GBLO-VXM01A.emea.wdpr.disney.com>

Hi Mark

Thanks for the quick response.

Actually I was also able to reproduce the bug on a test case which does not use the --cplus flag, but it seems fixed now in both cases.

Now, onwards with trying cygdb out!

Cheers,
Paul

-----Original Message-----
From: cython-dev-bounces at codespeak.net [mailto:cython-dev-bounces at codespeak.net] On Behalf Of mark florisson
Sent: 04 November 2010 10:20
To: cython-dev at codespeak.net
Subject: Re: [Cython] Cython debugger

Hey Paul,

thanks for the feedback, I hadn't tried the --cplus option yet.
Apparently I was using an attribute (that indicates where to write the C file) that is set to None when writing C++ files. I changed that and gave it a quick test, so hopefully it's fixed now :)

Kind regards,

Mark

On 4 November 2010 10:36, Connell, Paul <Paul.Connell at disney.com> wrote:
> Hi
>
>
>
> I just pulled the tip out of hg.cython.org/cython-gdb as I?m excited 
> to try this out.
>
>
>
> However, when running ?cython ?cplus ?debug? on a pyx that I know is 
> correct and works, I always get the following output:
>
>
>
> ? File "C:\Python26\lib\xml\etree\ElementTree.py", line 777, in 
> _raise_serialization_error
>
> ??? "cannot serialize %r (type %s)" % (text, type(text).__name__)
>
> TypeError: cannot serialize None (type NoneType)
>
>
>
>
>
> I have not yet had the opportunity to try reducing my pyx (it is 
> fairly large and getting larger every hour), but is this any kind of 
> known problem ? what might be causing it?
>
>
>
> Here?s the full traceback:
>
>
>
> ? File "/projects/ctg/build/bin/cython.py", line 3, in <module>
>
> ??? main(command_line = 1)
>
> ? File "C:\Python26\lib\site-packages\Cython\Compiler\Main.py", line 
> 780, in main
>
> ??? result = compile(sources, options)
>
> ? File "C:\Python26\lib\site-packages\Cython\Compiler\Main.py", line 
> 755, in compile
>
> ??? return compile_multiple(source, options)
>
> ? File "C:\Python26\lib\site-packages\Cython\Compiler\Main.py", line 
> 727, in compile_multiple
>
> ??? result = run_pipeline(source, options)
>
> ? File "C:\Python26\lib\site-packages\Cython\Compiler\Main.py", line 
> 596, in run_pipeline
>
> ??? err, enddata = context.run_pipeline(pipeline, source)
>
> ? File "C:\Python26\lib\site-packages\Cython\Compiler\Main.py", line 
> 237, in run_pipeline
>
> ??? data = phase(data)
>
> ? File "C:\Python26\lib\site-packages\Cython\Compiler\Main.py", line 
> 160, in generate_pyx_code
>
> ??? module_node.process_implementation(options, result)
>
> ? File "C:\Python26\lib\site-packages\Cython\Compiler\ModuleNode.py", 
> line 72, in process_implementation
>
> ??? self.generate_c_code(env, options, result)
>
> ? File "C:\Python26\lib\site-packages\Cython\Compiler\ModuleNode.py", 
> line 302,in generate_c_code
>
> ??? self._serialize_lineno_map(env, rootwriter)
>
> ? File "C:\Python26\lib\site-packages\Cython\Compiler\ModuleNode.py", 
> line 327,in _serialize_lineno_map
>
> ??? tb.serialize()
>
> ? File "C:\Python26\lib\site-packages\Cython\Debugger\DebugWriter.py", 
> line 71,in serialize
>
> ??? et.write(os.path.join(self.output_dir, fn), encoding="UTF-8", 
> **kw)
>
> ? File "C:\Python26\lib\xml\etree\ElementTree.py", line 663, in write
>
> ??? self._write(file, self._root, encoding, {})
>
> ? File "C:\Python26\lib\xml\etree\ElementTree.py", line 707, in _write
>
> ??? self._write(file, n, encoding, namespaces)
>
> ? File "C:\Python26\lib\xml\etree\ElementTree.py", line 698, in _write
>
> ??? _escape_attrib(v, encoding)))
>
> ? File "C:\Python26\lib\xml\etree\ElementTree.py", line 830, in 
> _escape_attrib
>
> ??? _raise_serialization_error(text)
>
> ? File "C:\Python26\lib\xml\etree\ElementTree.py", line 777, in 
> _raise_serializa
>
> tion_error
>
> ??? "cannot serialize %r (type %s)" % (text, type(text).__name__)
>
> TypeError: cannot serialize None (type NoneType)
>
>
>
> Regards,
>
> Paul
>
>
>
> From: cython-dev-bounces at codespeak.net 
> [mailto:cython-dev-bounces at codespeak.net] On Behalf Of mark florisson
> Sent: 03 November 2010 09:56
> To: cython-dev at codespeak.net
> Subject: Re: [Cython] Cython debugger
>
>
>
> Most of the Cython debugger is implemented, it can be pulled from 
> hg.cython.org/cython-gdb. To use it, you need Cython to export some 
> debug information, which can be done using ?'cython --debug 
> mymod.pyx', or 'python setup.py build_ext --pyrex-debug'. You can also 
> pass 'pyrex_debug=True' as an argument to Cython.Distutils.extension.Extension.
>
> You can then start gdb by typing 'cygdb'??(which should be installed 
> as a
> script)?in your project directory. Then 'help cy' gives an overview of 
> commands. It currently supports breapoints, stepping, stepping-over, 
> backtraces, source code listing, going up and down the stack, printing 
> variables with regard to context, listing locals or globals and 
> running and continuing the program. If pygments is installed it will 
> colorize source code which is configurable through cy_* parameters.
>
>
>
> It basically works on three levels: the Python, Cython and C level.
> Depending on the context cygdb does the right thing. For stepping it 
> considers the following stack frames relevant: any Python frame, any 
> Cython frame and any C frame from a C function called directly by Cython user-code.
>
>
>
> So, it would be great if some people could test and try it (unit tests 
> are written but some system testing is always great). It works with 
> gdb 7.2 (the
> 7.1 python api was too incomplete and broken). So if you guys like it 
> I could write documentation that explains to Cython users how to 
> install and use it. Any suggestion or criticism is most welcome!
>
>
>
> The Python support (libpython.py) was also modified. I'd like to push 
> this mainstream (and process any suggestions and criticism) before 
> supplying a patch to Python.
>
>
>
> Kind regards,
>
>
>
> Mark
>
>
>
> On 15 September 2010 17:49, Robert Bradshaw 
> <robertwb at math.washington.edu>
> wrote:
>
> On Wed, Sep 15, 2010 at 2:55 AM, mark florisson 
> <markflorisson88 at gmail.com> wrote:
>
>>> It ought to be possible to do something similar with cython code. ?
>>> It may not even be necessary to modify cython: perhaps some 
>>> searching for locals named "__pyx_*" iirc would get you 70% of the way there?
>>
>> Although that sounds like a wonderful idea, I think there are also 
>> issues with that. One issue is that a user must be able to set Cython 
>> breakpoints before the Cython module would be loaded, and for that 
>> the symbol name would be needed beforehand. Also, I don't know if 
>> these mangled names are consistent now and in the future and if you 
>> would be able to unambiguously associate a Cython variable name with 
>> a mangled name.
>
> Mangled names are deterministic and, though they're not guaranteed to 
> be consistent from release to release, almost always are.
>
>>> I can attest that having the prettyprinters enabled does make it 
>>> much easier to debug cython code: all of the PyObject* get prettyprinted.
>>
>> I've been looking at the code and this is pretty neat. I did 
>> encounter some issues, for instance if you load the script before 
>> loading the python interpreter you get this traceback because these 
>> types are not defined at that time:
>>
>> Traceback (most recent call last):
>> ?File "<string>", line 1, in <module>
>> ?File ".libpython.py", line 49, in <module>
>> ? ?_type_size_t = gdb.lookup_type('size_t')
>> RuntimeError: No type named size_t.
>>
>> So I think it would be a good idea to not make that code module-level.
>>
>>>
>>> One other thought: if it's possibly to expose the cython structures 
>>> in some meaningful way, perhaps we could change upstream python's 
>>> gdb hooks to simply integrate them into the py-* commands I 
>>> mentioned above? (so e.g. cython c functions get somehow treated as 
>>> python frames; currently I have a test predicate:
>>> ?Frame.is_evalframeex()
>>> which perhaps could be generalized?)
>>>
>>> (Not sure; it would complicate the selftests within python itself)
>>>
>>
>> I think it would be hard to make them actual Python frames because 
>> creating frames in the inferior process from gdb is probably quite 
>> dangerous, and the alternative would be to modify Cython so that it 
>> creates Python stack frames (this sounds feasible but I think it 
>> might be a little bit of work). However, if this could be done (it 
>> would only do so if this 'debug' flag is active), then tracebacks and 
>> locals inspection etc wouldn't need special attention and the code 
>> would appear as normal Python code (apart from the Code objects obviously).
>> However, this would form a problem for non-primitive C-type Cython 
>> variables.
>>
>> So at the very least we could have a 'py-locals' or some such command 
>> that would show the value of all the locals (the Python locals would 
>> be printed by py-print and C locals by gdb print). For regular python 
>> code it would show the locals from the current stack frame. For the 
>> Cython part to work we would need information from the Cython 
>> compiler because we wouldn't want to list any temporary or irrelevant 
>> variables.
>
> Yes, this is what I was thinking, at least in terms of exposing stuff 
> to pdb (which is a complementary project). BTW, frames are already 
> created for functions when profiling is enabled.
>
>> So I think we should be able to integrate these two projects into one 
>> fruitful project, and with proper documentation it could help both 
>> regular Python users and Cython users.
>
> +1
>
> - Robert
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>
>
_______________________________________________
Cython-dev mailing list
Cython-dev at codespeak.net
http://codespeak.net/mailman/listinfo/cython-dev

From vitja.makarov at gmail.com  Thu Nov  4 12:37:25 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 4 Nov 2010 14:37:25 +0300
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTimEmE9=xiN6=PDNEgRo=P_=ECxE=n7Eh0aYs81g@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<AANLkTi=-=OQ8_o8NpmDvb5kr--+fd9Sq54K6vVm+4YbA@mail.gmail.com>
	<AANLkTimuj3Y+Cn_YxdvThcZMTcS=WkAYNXezQi+YunfT@mail.gmail.com>
	<AANLkTikBd6Uj58_uKVgcpSSm4UAoNgZz1wpTzGMG5Xg2@mail.gmail.com>
	<AANLkTikSi2Awwv31sx3A6BJy=WNSYROo+ceecoygQ0Fk@mail.gmail.com>
	<AANLkTi=bFQwdofb4TPcV53b9w0RtcshvC8Uiv2mJa9i-@mail.gmail.com>
	<4CD0FA29.7000201@behnel.de>
	<AANLkTinNGspAzLoOBdoXcUZgxkxR86i4aQfS2FMrgVYR@mail.gmail.com>
	<AANLkTi=a8SMk+u7JfLKxujR3p2ZVnjLsSPjrMY8hufbt@mail.gmail.com>
	<AANLkTimjYc=4HdLE+VDmUyn0piXPWGZ7zWGqQ0QoabkP@mail.gmail.com>
	<AANLkTim0dL=7nR6WiXAYQyMUSyQ=_b7jsBuKCQ8zh-WT@mail.gmail.com>
	<AANLkTim6qdY6hmj2VVJJ_rwje5Sup5NQjspZuymVEv8Z@mail.gmail.com>
	<AANLkTimWfQO1SsVgDX9yyex=xivq0z7nmw0PJoJBXvFc@mail.gmail.com>
	<4CD1C983.7000300@behnel.de>
	<AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>
	<4CD1D787.4010202@behnel.de>
	<AANLkTikYRv6CPTbSifo-C_gAwU-AG0REC1sPwAHq2DJO@mail.gmail.com>
	<4CD247CD.705@behnel.de>
	<AANLkTimEmE9=xiN6=PDNEgRo=P_=ECxE=n7Eh0aYs81g@mail.gmail.com>
Message-ID: <AANLkTimOXxppAQ6Fb5DSCTyRoY9ZJghhV5=sM3i0N6ic@mail.gmail.com>

I can call prepare and then update dict?

2010/11/4 Vitja Makarov <vitja.makarov at gmail.com>:
> Here is my first try with metaclasses in py3. It does not handle
> __prepare__ I don't understand what function should I call to create
> cell... and what is that func
>
> 2010/11/4 Stefan Behnel <stefan_ml at behnel.de>:
>> Vitja Makarov, 03.11.2010 23:03:
>>> 2010/11/4 Stefan Behnel:
>>>> Vitja Makarov, 03.11.2010 22:13:
>>>>> Now it seems to me that class declaration is more like function call
>>>>> with special case for metaclass keyword and positional arguments:
>>>>>
>>>>> class Base(type):
>>>>> ? ? ? def __new__(cls, name, bases, attrs, **kwargs):
>>>>> ? ? ? ? ? print(cls, name, bases, attrs, kwargs)
>>>>>
>>>>> class Foo(1, 2, 3, metaclass=Base, foo='bar'):
>>>>> ? ? ? def hello(self):
>>>>> ? ? ? ? ? pass
>>>>
>>>> Yes, I think that's exactly how this should work. Want to give it another try?
>>>
>>> Yes, I want to finish this part ;)
>>
>> Great! :)
>>
>> BTW, thanks for doing all this. Even your first patch was pretty good for a
>> "first patch".
>>
>>
>>>> It seems that the number of positional arguments is fixed, but you can pass
>>>> any number of keyword arguments, out of which only "metaclass" is special
>>>> cased and removed (from a copy of the dict, BTW). There is also an
>>>> additional protocol if the metaclass has a "__prepare__" class method. See
>>>> "__build_class__".
>>>
>>> Yes, all positional arguments are turned into bases. So we can make it this way:
>>>
>>> Pyx_CreateClass(PyObject *bases, PyObject *dict, PyObject *name,
>>> PyObject *modname, PyObject *kwargs);
>>>
>>> And then
>>>
>>> args = (metaclass, bases, name, dict)
>>> PyEval_CallObjectWithKeywords(metaclass, args, kwargs)
>>
>> Except for the first 'metaclass' in args (args has length 3). Again, see
>> the __build_class__ implementation.
>>
>>
>>>> I think the Python 2 protocol in Cython should work as in Py3. Metaclasses
>>>> in Py2 won't generally support kwargs, but if a user explicitly provides
>>>> them, it's best to assume that that's intentional. In the worst case, there
>>>> will be a runtime TypeError.
>>>
>>> Do you mean that new syntax should be used for Py2?
>>
>> No, just what happens behind the syntax. A __metaclass__ field can easily
>> be mapped to a metaclass keyword argument (or the equivalent representation
>> on a PyClassDefNode in Cython) and from that point on, it's the same thing
>> in both cases.
>>
>>
>>>> So, kwargs should pass through, except for __metaclass__. If both the
>>>> __metaclass__ class field and the metaclass kwarg are passed, I'd lean
>>>> towards ignoring the field and prefer the kwarg, just like Py3 does. It can
>>>> be argued that this is worth a warning, though.
>>>>
>>> Py3 doesn't support __metaclass__ attribute, so if metaclass is set that
>>> seems to be mostly Py3 so __metaclass__ should be ignored.
>>
>> If I'm not mistaken, the __metaclass__ field can always be handled by the
>> compiler during type analysis (or maybe in a transform, see the classes in
>> ParseTreeTransforms.py, for example). You can't dynamically add an
>> attribute to a class at definition time without letting the parser see its
>> name. So you just have to check the class body and you'll know if the field
>> is there or not.
>>
>> Cython also has a Python 3 syntax mode (look out for "language_level") in
>> which we can disable this field lookup, so that you get semantic
>> equivalence with Python 3 even before the code generation phase. So we
>> don't need any runtime support for the __metaclass__ field, not even in Py2.
>>
>>
>>>> I also think that the parser should extract the metaclass keyword, not the
>>>> runtime code. So, if provided, it should be passed into __Pyx_CreateClass()
>>>> as an argument and not in the kwargs.
>>>
>>> I think parser can't extract metaclass keyword, try this code:
>>>
>>> class Base(type):
>>> ? ? ?def __new__(cls, name, bases, attrs, **kwargs):
>>> ? ? ? ? ?print(cls, name, bases, attrs, kwargs)
>>>
>>> myargs={'metaclass': Base}
>>> class Foo(1, 2, 3, **myargs):
>>> ? ? ?def hello(self):
>>> ? ? ? ? ?pass
>>
>> Well, it can if it's passed explicitly, though.
>>
>> We already need to support an explicit metaclass to handle the
>> __metaclass__ field. In most cases, the keyword argument will be spelled
>> out explicitly, so we can reduce the runtime overhead in that case. But
>> that sounds like an optimisation, not a required feature.
>>
>> Stefan
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>

From stefan_ml at behnel.de  Thu Nov  4 12:46:05 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 04 Nov 2010 12:46:05 +0100
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTimOXxppAQ6Fb5DSCTyRoY9ZJghhV5=sM3i0N6ic@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>	<AANLkTikSi2Awwv31sx3A6BJy=WNSYROo+ceecoygQ0Fk@mail.gmail.com>	<AANLkTi=bFQwdofb4TPcV53b9w0RtcshvC8Uiv2mJa9i-@mail.gmail.com>	<4CD0FA29.7000201@behnel.de>	<AANLkTinNGspAzLoOBdoXcUZgxkxR86i4aQfS2FMrgVYR@mail.gmail.com>	<AANLkTi=a8SMk+u7JfLKxujR3p2ZVnjLsSPjrMY8hufbt@mail.gmail.com>	<AANLkTimjYc=4HdLE+VDmUyn0piXPWGZ7zWGqQ0QoabkP@mail.gmail.com>	<AANLkTim0dL=7nR6WiXAYQyMUSyQ=_b7jsBuKCQ8zh-WT@mail.gmail.com>	<AANLkTim6qdY6hmj2VVJJ_rwje5Sup5NQjspZuymVEv8Z@mail.gmail.com>	<AANLkTimWfQO1SsVgDX9yyex=xivq0z7nmw0PJoJBXvFc@mail.gmail.com>	<4CD1C983.7000300@behnel.de>	<AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>	<4CD1D787.4010202@behnel.de>	<AANLkTikYRv6CPTbSifo-C_gAwU-AG0REC1sPwAHq2DJO@mail.gmail.com>	<4CD247CD.705@behnel.de>	<AANLkTimEmE9=xiN6=PDNEgRo=P_=ECxE=n7Eh0aYs81g@mail.gmail.com>
	<AANLkTimOXxppAQ6Fb5DSCTyRoY9ZJghhV5=sM3i0N6ic@mail.gmail.com>
Message-ID: <4CD29CFD.7030309@behnel.de>

Vitja Makarov, 04.11.2010 12:37:
> I can call prepare and then update dict?

Sorry, what?

Please provide some more context in your mails, and *please*, stop top-posting.

Stefan

From vitja.makarov at gmail.com  Thu Nov  4 12:50:46 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 4 Nov 2010 14:50:46 +0300
Subject: [Cython] metaclasses
In-Reply-To: <4CD29CFD.7030309@behnel.de>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<AANLkTikSi2Awwv31sx3A6BJy=WNSYROo+ceecoygQ0Fk@mail.gmail.com>
	<AANLkTi=bFQwdofb4TPcV53b9w0RtcshvC8Uiv2mJa9i-@mail.gmail.com>
	<4CD0FA29.7000201@behnel.de>
	<AANLkTinNGspAzLoOBdoXcUZgxkxR86i4aQfS2FMrgVYR@mail.gmail.com>
	<AANLkTi=a8SMk+u7JfLKxujR3p2ZVnjLsSPjrMY8hufbt@mail.gmail.com>
	<AANLkTimjYc=4HdLE+VDmUyn0piXPWGZ7zWGqQ0QoabkP@mail.gmail.com>
	<AANLkTim0dL=7nR6WiXAYQyMUSyQ=_b7jsBuKCQ8zh-WT@mail.gmail.com>
	<AANLkTim6qdY6hmj2VVJJ_rwje5Sup5NQjspZuymVEv8Z@mail.gmail.com>
	<AANLkTimWfQO1SsVgDX9yyex=xivq0z7nmw0PJoJBXvFc@mail.gmail.com>
	<4CD1C983.7000300@behnel.de>
	<AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>
	<4CD1D787.4010202@behnel.de>
	<AANLkTikYRv6CPTbSifo-C_gAwU-AG0REC1sPwAHq2DJO@mail.gmail.com>
	<4CD247CD.705@behnel.de>
	<AANLkTimEmE9=xiN6=PDNEgRo=P_=ECxE=n7Eh0aYs81g@mail.gmail.com>
	<AANLkTimOXxppAQ6Fb5DSCTyRoY9ZJghhV5=sM3i0N6ic@mail.gmail.com>
	<4CD29CFD.7030309@behnel.de>
Message-ID: <AANLkTinP8Osz1YKkLyuDhPT=vzyc6iscnOCoi2h0f+Rv@mail.gmail.com>

2010/11/4 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 04.11.2010 12:37:
>> I can call prepare and then update dict?
>
> Sorry, what?
>
> Please provide some more context in your mails, and *please*, stop top-posting.
>

Ok, sorry.
That's all about __prepare__ where should it be called and how to join
dict with ns?

I see  one solution (in Pyx_CreateClass) :
1. call __preapre__
2. update returned namespace with values from dict
3. call metaclass with this namespace instead of dict

From stefan_ml at behnel.de  Thu Nov  4 13:15:39 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 04 Nov 2010 13:15:39 +0100
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTinP8Osz1YKkLyuDhPT=vzyc6iscnOCoi2h0f+Rv@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>	<AANLkTi=bFQwdofb4TPcV53b9w0RtcshvC8Uiv2mJa9i-@mail.gmail.com>	<4CD0FA29.7000201@behnel.de>	<AANLkTinNGspAzLoOBdoXcUZgxkxR86i4aQfS2FMrgVYR@mail.gmail.com>	<AANLkTi=a8SMk+u7JfLKxujR3p2ZVnjLsSPjrMY8hufbt@mail.gmail.com>	<AANLkTimjYc=4HdLE+VDmUyn0piXPWGZ7zWGqQ0QoabkP@mail.gmail.com>	<AANLkTim0dL=7nR6WiXAYQyMUSyQ=_b7jsBuKCQ8zh-WT@mail.gmail.com>	<AANLkTim6qdY6hmj2VVJJ_rwje5Sup5NQjspZuymVEv8Z@mail.gmail.com>	<AANLkTimWfQO1SsVgDX9yyex=xivq0z7nmw0PJoJBXvFc@mail.gmail.com>	<4CD1C983.7000300@behnel.de>	<AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>	<4CD1D787.4010202@behnel.de>	<AANLkTikYRv6CPTbSifo-C_gAwU-AG0REC1sPwAHq2DJO@mail.gmail.com>	<4CD247CD.705@behnel.de>	<AANLkTimEmE9=xiN6=PDNEgRo=P_=ECxE=n7Eh0aYs81g@mail.gmail.com>	<AANLkTimOXxppAQ6Fb5DSCTyRoY9ZJghhV5=sM3i0N6ic@mail.gmail.com>	<4CD29CFD.7030309@behnel.de>
	<AANLkTinP8Osz1YKkLyuDhPT=vzyc6iscnOCoi2h0f+Rv@mail.gmail.com>
Message-ID: <4CD2A3EB.4030601@behnel.de>

Vitja Makarov, 04.11.2010 12:50:
> 2010/11/4 Stefan Behnel<stefan_ml at behnel.de>:
>> Vitja Makarov, 04.11.2010 12:37:
>>> I can call prepare and then update dict?
>>
>> Sorry, what?
>>
>> Please provide some more context in your mails, and *please*, stop top-posting.
>>
>
> Ok, sorry.
> That's all about __prepare__ where should it be called and how to join
> dict with ns?
>
> I see  one solution (in Pyx_CreateClass) :
> 1. call __preapre__
> 2. update returned namespace with values from dict
> 3. call metaclass with this namespace instead of dict

Ah, ok. A took a deeper look at the way __build_class__ works, then frowned 
a couple of times and moved over to reading the PEP, which explains stuff 
that __build_class__ doesn't even seem to do.

http://www.python.org/dev/peps/pep-3115/

Apparently, __prepare__ is meant to be executed *before* even evaluating 
the class body:

"""
__prepare__ returns a dictionary-like object which is used to store
      the class member definitions during evaluation of the class body.
      In other words, the class body is evaluated as a function block
      (just like it is now), except that the local variables dictionary
      is replaced by the dictionary returned from __prepare__. This
      dictionary object can be a regular dictionary or a custom mapping
      type.
"""

So it's basically what gives you the dict that you assign the class members 
to (as you did in your original patch).

I do not see why we should currently implement this, given that 
__build_class__ doesn't seem to follow that protocol either. It would just 
slow down the evaluation of the class body as we could no longer rely on 
the class dict being a dict.

I think we can just ignore that part of the protocol for now - I doubt that 
many people will use it (or even know about it). Maybe worth a feature 
request on the bug tracker, but not more.

Stefan

From markflorisson88 at gmail.com  Thu Nov  4 14:01:13 2010
From: markflorisson88 at gmail.com (mark florisson)
Date: Thu, 4 Nov 2010 14:01:13 +0100
Subject: [Cython] metaclasses
In-Reply-To: <4CD2A3EB.4030601@behnel.de>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<AANLkTi=bFQwdofb4TPcV53b9w0RtcshvC8Uiv2mJa9i-@mail.gmail.com>
	<4CD0FA29.7000201@behnel.de>
	<AANLkTinNGspAzLoOBdoXcUZgxkxR86i4aQfS2FMrgVYR@mail.gmail.com>
	<AANLkTi=a8SMk+u7JfLKxujR3p2ZVnjLsSPjrMY8hufbt@mail.gmail.com>
	<AANLkTimjYc=4HdLE+VDmUyn0piXPWGZ7zWGqQ0QoabkP@mail.gmail.com>
	<AANLkTim0dL=7nR6WiXAYQyMUSyQ=_b7jsBuKCQ8zh-WT@mail.gmail.com>
	<AANLkTim6qdY6hmj2VVJJ_rwje5Sup5NQjspZuymVEv8Z@mail.gmail.com>
	<AANLkTimWfQO1SsVgDX9yyex=xivq0z7nmw0PJoJBXvFc@mail.gmail.com>
	<4CD1C983.7000300@behnel.de>
	<AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>
	<4CD1D787.4010202@behnel.de>
	<AANLkTikYRv6CPTbSifo-C_gAwU-AG0REC1sPwAHq2DJO@mail.gmail.com>
	<4CD247CD.705@behnel.de>
	<AANLkTimEmE9=xiN6=PDNEgRo=P_=ECxE=n7Eh0aYs81g@mail.gmail.com>
	<AANLkTimOXxppAQ6Fb5DSCTyRoY9ZJghhV5=sM3i0N6ic@mail.gmail.com>
	<4CD29CFD.7030309@behnel.de>
	<AANLkTinP8Osz1YKkLyuDhPT=vzyc6iscnOCoi2h0f+Rv@mail.gmail.com>
	<4CD2A3EB.4030601@behnel.de>
Message-ID: <AANLkTinHaapMihFN0vHaRD2o3xKVD6Ows-JjD7W_6mzx@mail.gmail.com>

On 4 November 2010 13:15, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Vitja Makarov, 04.11.2010 12:50:
>> 2010/11/4 Stefan Behnel<stefan_ml at behnel.de>:
>>> Vitja Makarov, 04.11.2010 12:37:
>>>> I can call prepare and then update dict?
>>>
>>> Sorry, what?
>>>
>>> Please provide some more context in your mails, and *please*, stop top-posting.
>>>
>>
>> Ok, sorry.
>> That's all about __prepare__ where should it be called and how to join
>> dict with ns?
>>
>> I see ?one solution (in Pyx_CreateClass) :
>> 1. call __preapre__
>> 2. update returned namespace with values from dict
>> 3. call metaclass with this namespace instead of dict
>
> Ah, ok. A took a deeper look at the way __build_class__ works, then frowned
> a couple of times and moved over to reading the PEP, which explains stuff
> that __build_class__ doesn't even seem to do.
>
> http://www.python.org/dev/peps/pep-3115/
>
> Apparently, __prepare__ is meant to be executed *before* even evaluating
> the class body:
>
> """
> __prepare__ returns a dictionary-like object which is used to store
> ? ? ?the class member definitions during evaluation of the class body.
> ? ? ?In other words, the class body is evaluated as a function block
> ? ? ?(just like it is now), except that the local variables dictionary
> ? ? ?is replaced by the dictionary returned from __prepare__. This
> ? ? ?dictionary object can be a regular dictionary or a custom mapping
> ? ? ?type.
> """
>
> So it's basically what gives you the dict that you assign the class members
> to (as you did in your original patch).
>
> I do not see why we should currently implement this, given that
> __build_class__ doesn't seem to follow that protocol either. It would just
> slow down the evaluation of the class body as we could no longer rely on
> the class dict being a dict.
>
> I think we can just ignore that part of the protocol for now - I doubt that
> many people will use it (or even know about it). Maybe worth a feature
> request on the bug tracker, but not more.

As the PEP mentions, __prepare__ is especially useful for establishing
in what order members are defined. To quote:

"    In particular, there is an important body of use cases where it
     would be useful to preserve the order in which a class members are
     declared. Ordinary Python objects store their members in a
     dictionary, in which ordering is unimportant, and members are
     accessed strictly by name. However, Python is often used to
     interface with external systems in which the members are organized
     according to an implicit ordering. Examples include declaration of C
     structs; COM objects; Automatic translation of Python classes into
     IDL or database schemas, such as used in an ORM; and so on. "

If it's a hassle to implement then users could always just write this
stuff in pure python though (as they have to do currently).

> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From stefan_ml at behnel.de  Thu Nov  4 14:17:50 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 04 Nov 2010 14:17:50 +0100
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTinHaapMihFN0vHaRD2o3xKVD6Ows-JjD7W_6mzx@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>	<AANLkTinNGspAzLoOBdoXcUZgxkxR86i4aQfS2FMrgVYR@mail.gmail.com>	<AANLkTi=a8SMk+u7JfLKxujR3p2ZVnjLsSPjrMY8hufbt@mail.gmail.com>	<AANLkTimjYc=4HdLE+VDmUyn0piXPWGZ7zWGqQ0QoabkP@mail.gmail.com>	<AANLkTim0dL=7nR6WiXAYQyMUSyQ=_b7jsBuKCQ8zh-WT@mail.gmail.com>	<AANLkTim6qdY6hmj2VVJJ_rwje5Sup5NQjspZuymVEv8Z@mail.gmail.com>	<AANLkTimWfQO1SsVgDX9yyex=xivq0z7nmw0PJoJBXvFc@mail.gmail.com>	<4CD1C983.7000300@behnel.de>	<AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>	<4CD1D787.4010202@behnel.de>	<AANLkTikYRv6CPTbSifo-C_gAwU-AG0REC1sPwAHq2DJO@mail.gmail.com>	<4CD247CD.705@behnel.de>	<AANLkTimEmE9=xiN6=PDNEgRo=P_=ECxE=n7Eh0aYs81g@mail.gmail.com>	<AANLkTimOXxppAQ6Fb5DSCTyRoY9ZJghhV5=sM3i0N6ic@mail.gmail.com>	<4CD29CFD.7030309@behnel.de>	<AANLkTinP8Osz1YKkLyuDhPT=vzyc6iscnOCoi2h0f+Rv@mail.gmail.com>	<4CD2A3EB.4030601@behnel.de>
	<AANLkTinHaapMihFN0vHaRD2o3xKVD6Ows-JjD7W_6mzx@mail.gmail.com>
Message-ID: <4CD2B27E.6080805@behnel.de>

mark florisson, 04.11.2010 14:01:
> On 4 November 2010 13:15, Stefan Behnel wrote:
>> http://www.python.org/dev/peps/pep-3115/
>>
>> Apparently, __prepare__ is meant to be executed *before* even evaluating
>> the class body
>> So it's basically what gives you the dict that you assign the class members
>> to (as you did in your original patch).
>>
>> I do not see why we should currently implement this, given that
>> __build_class__ doesn't seem to follow that protocol either. It would just
>> slow down the evaluation of the class body as we could no longer rely on
>> the class dict being a dict.
>>
>> I think we can just ignore that part of the protocol for now - I doubt that
>> many people will use it (or even know about it). Maybe worth a feature
>> request on the bug tracker, but not more.
>
> As the PEP mentions, __prepare__ is especially useful for establishing
> in what order members are defined. To quote:
>[...]
> If it's a hassle to implement then users could always just write this
> stuff in pure python though (as they have to do currently).

Well, "pure Python" in the sense that the whole class needs to be put into 
a Python module. Changing the class dict needs runtime support as it must 
happen when the code evaluation starts running into the class body. It 
cannot be done from within the class building function, which is executed 
at the end of the class body.

This implies that it only works with metaclasses that really are defined as 
part of the class declaration, i.e. in Py3 syntax. A __metaclass__ field 
will be detected too late in the class building cycle to make this work.

I'm not really opposed to implementing this. I'm just saying that it would 
complicate things for an extremely rare use case. If someone wants to do 
it, why not.

BTW, I'll ask on python-dev about the way __build_class__ works here.

Stefan

From stefan_ml at behnel.de  Thu Nov  4 14:30:45 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 04 Nov 2010 14:30:45 +0100
Subject: [Cython] metaclasses
In-Reply-To: <4CD2B27E.6080805@behnel.de>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>	<AANLkTi=a8SMk+u7JfLKxujR3p2ZVnjLsSPjrMY8hufbt@mail.gmail.com>	<AANLkTimjYc=4HdLE+VDmUyn0piXPWGZ7zWGqQ0QoabkP@mail.gmail.com>	<AANLkTim0dL=7nR6WiXAYQyMUSyQ=_b7jsBuKCQ8zh-WT@mail.gmail.com>	<AANLkTim6qdY6hmj2VVJJ_rwje5Sup5NQjspZuymVEv8Z@mail.gmail.com>	<AANLkTimWfQO1SsVgDX9yyex=xivq0z7nmw0PJoJBXvFc@mail.gmail.com>	<4CD1C983.7000300@behnel.de>	<AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>	<4CD1D787.4010202@behnel.de>	<AANLkTikYRv6CPTbSifo-C_gAwU-AG0REC1sPwAHq2DJO@mail.gmail.com>	<4CD247CD.705@behnel.de>	<AANLkTimEmE9=xiN6=PDNEgRo=P_=ECxE=n7Eh0aYs81g@mail.gmail.com>	<AANLkTimOXxppAQ6Fb5DSCTyRoY9ZJghhV5=sM3i0N6ic@mail.gmail.com>	<4CD29CFD.7030309@behnel.de>	<AANLkTinP8Osz1YKkLyuDhPT=vzyc6iscnOCoi2h0f+Rv@mail.gmail.com>	<4CD2A3EB.4030601@behnel.de>	<AANLkTinHaapMihFN0vHaRD2o3xKVD6Ows-JjD7W_6mzx@mail.gmail.com>
	<4CD2B27E.6080805@behnel.de>
Message-ID: <4CD2B585.8040601@behnel.de>

Stefan Behnel, 04.11.2010 14:17:
> BTW, I'll ask on python-dev about the way __build_class__ works here.

;) writing e-mails about a problem without sending them away is the best 
way to understand the topic.

Ok, so, __build_class__ is actually ok, because it *is* the point where the 
class body is evaluated. So, __prepare__ is really called before the 
evaluation of the body (which is done in a "function" call near the end) 
and the metaclass is called afterwards.

We could essentially do the same thing in Cython. At the point where the 
class dict is now getting instantiated, we need to put the code of the 
first part of __build_class__, i.e. look up the metaclass from the keywords 
and find and call its __prepare__ method. Then, evaluate the class body 
into the class namespace object that __prepare__ returned, followed by the 
call to __Pyx_CreateClass().

If we don't find a metaclass in the keywords, we do not support __prepare__ 
but try to look up the __metaclass__ field in the class namespace during 
__Pyx_CreateClass().

Stefan


From vitja.makarov at gmail.com  Thu Nov  4 18:36:04 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 4 Nov 2010 20:36:04 +0300
Subject: [Cython] metaclasses
In-Reply-To: <4CD2B585.8040601@behnel.de>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<AANLkTi=a8SMk+u7JfLKxujR3p2ZVnjLsSPjrMY8hufbt@mail.gmail.com>
	<AANLkTimjYc=4HdLE+VDmUyn0piXPWGZ7zWGqQ0QoabkP@mail.gmail.com>
	<AANLkTim0dL=7nR6WiXAYQyMUSyQ=_b7jsBuKCQ8zh-WT@mail.gmail.com>
	<AANLkTim6qdY6hmj2VVJJ_rwje5Sup5NQjspZuymVEv8Z@mail.gmail.com>
	<AANLkTimWfQO1SsVgDX9yyex=xivq0z7nmw0PJoJBXvFc@mail.gmail.com>
	<4CD1C983.7000300@behnel.de>
	<AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>
	<4CD1D787.4010202@behnel.de>
	<AANLkTikYRv6CPTbSifo-C_gAwU-AG0REC1sPwAHq2DJO@mail.gmail.com>
	<4CD247CD.705@behnel.de>
	<AANLkTimEmE9=xiN6=PDNEgRo=P_=ECxE=n7Eh0aYs81g@mail.gmail.com>
	<AANLkTimOXxppAQ6Fb5DSCTyRoY9ZJghhV5=sM3i0N6ic@mail.gmail.com>
	<4CD29CFD.7030309@behnel.de>
	<AANLkTinP8Osz1YKkLyuDhPT=vzyc6iscnOCoi2h0f+Rv@mail.gmail.com>
	<4CD2A3EB.4030601@behnel.de>
	<AANLkTinHaapMihFN0vHaRD2o3xKVD6Ows-JjD7W_6mzx@mail.gmail.com>
	<4CD2B27E.6080805@behnel.de> <4CD2B585.8040601@behnel.de>
Message-ID: <AANLkTik4qhsVhHhCQRyheKpDzoGGQD-q5r=bJUi2_VV_@mail.gmail.com>

2010/11/4 Stefan Behnel <stefan_ml at behnel.de>:
> Stefan Behnel, 04.11.2010 14:17:
>> BTW, I'll ask on python-dev about the way __build_class__ works here.
>
> ;) writing e-mails about a problem without sending them away is the best
> way to understand the topic.
>
> Ok, so, __build_class__ is actually ok, because it *is* the point where the
> class body is evaluated. So, __prepare__ is really called before the
> evaluation of the body (which is done in a "function" call near the end)
> and the metaclass is called afterwards.
>
> We could essentially do the same thing in Cython. At the point where the
> class dict is now getting instantiated, we need to put the code of the
> first part of __build_class__, i.e. look up the metaclass from the keywords
> and find and call its __prepare__ method. Then, evaluate the class body
> into the class namespace object that __prepare__ returned, followed by the
> call to __Pyx_CreateClass().
>
> If we don't find a metaclass in the keywords, we do not support __prepare__
> but try to look up the __metaclass__ field in the class namespace during
> __Pyx_CreateClass().
>
> Stefan
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

I have attached patch (in trac) for python3 metaclass support,
__prepare__ now works this way:

if metaclass is defined py3 way and it has __prepare__, namespace is
merged into dict.
This is wrong but it's much easy to implement, right behavior could be
imlemented later.

This could be done this way:

1. PyClassDefNode detect that class definition is py3 style and has
metaclass keyword defined, it sets some flag
2. then code generation should work this way:

name = <class name>
bases = <tuple of bases>
metaclass = <metaclass>
mkw = <metaclass keywords dict>

if hasattr(metaclass, '__prepare__'):
    dict = __Pyx_PrepareClass(metaclass, bases, name, mkw)
else:
    dict = {}
# now fill dict with attributes

margs = tuple(name, bases, dict)
klass = PyEval_CallObjectWithKeywords(metaclass, margs, mkw)


This is interesting to implement, I'll try...

From Chris.Barker at noaa.gov  Thu Nov  4 17:55:15 2010
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 04 Nov 2010 09:55:15 -0700
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <AANLkTik9OEfmg=wUYNNrcGURTA8hjXCt-eUPZMmZpWsh@mail.gmail.com>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>
	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>
	<AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>
	<AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>
	<4CD11C30.9090103@student.matnat.uio.no>
	<AANLkTimABt0xae_NBG1MVhLGx5-AMf4ps4m7jT2bcQQC@mail.gmail.com>
	<AANLkTi=5DttGcLp6SEv0w06YVratHpc_CSdCSpoiU-rD@mail.gmail.com>
	<AANLkTikpWy1GW-ht-37faSvb3gTkpGPnOkycC9qY-Wyn@mail.gmail.com>
	<AANLkTi=k_MKp0czueQ+w6JE+60Jtbkg=+9-TNE2L=wnu@mail.gmail.com>
	<AANLkTik9OEfmg=wUYNNrcGURTA8hjXCt-eUPZMmZpWsh@mail.gmail.com>
Message-ID: <4CD2E573.80907@noaa.gov>

On 11/3/10 11:47 PM, Brett Calcott wrote:
> If I just had to write one of these functions, then
> indeed trying to do it as a one-liner would be silly. But I'm wrapping
> lots of them, and this means I'm just cutting and pasting code, and
> then replacing names. When I find myself doing that, I think: there
> has to be a better way.

yup. However, it seems to me that the reason you are cutting and pasting 
code is that you have followed a consistent convention in your code 
(convention over configuration, as the Rails people say), which is a 
good thing. However, to me, the alternative to your cutting and pasting 
is not to try to come up with a compact way to express your convention 
in Cython, but rather code-generation.

One day, we may have a mature Cython code generator for wrapping C/C++ 
(and it may be worth looking at what has been done along those lines). 
But in the meantime, you could pretty quickly whip out a code generator 
that works for your particular use-case. Maybe use a general purpose 
template system (Cheetah, for instance) would help?

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov

From robertwb at math.washington.edu  Thu Nov  4 19:28:14 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 4 Nov 2010 11:28:14 -0700
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <4CD2E573.80907@noaa.gov>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>
	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>
	<AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>
	<AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>
	<4CD11C30.9090103@student.matnat.uio.no>
	<AANLkTimABt0xae_NBG1MVhLGx5-AMf4ps4m7jT2bcQQC@mail.gmail.com>
	<AANLkTi=5DttGcLp6SEv0w06YVratHpc_CSdCSpoiU-rD@mail.gmail.com>
	<AANLkTikpWy1GW-ht-37faSvb3gTkpGPnOkycC9qY-Wyn@mail.gmail.com>
	<AANLkTi=k_MKp0czueQ+w6JE+60Jtbkg=+9-TNE2L=wnu@mail.gmail.com>
	<AANLkTik9OEfmg=wUYNNrcGURTA8hjXCt-eUPZMmZpWsh@mail.gmail.com>
	<4CD2E573.80907@noaa.gov>
Message-ID: <AANLkTi=TgqmVmizSc34BJMMZVWH_Ko-7CS7NQtmUEPkr@mail.gmail.com>

On Thu, Nov 4, 2010 at 9:55 AM, Christopher Barker
<Chris.Barker at noaa.gov> wrote:
> On 11/3/10 11:47 PM, Brett Calcott wrote:
>> If I just had to write one of these functions, then
>> indeed trying to do it as a one-liner would be silly. But I'm wrapping
>> lots of them, and this means I'm just cutting and pasting code, and
>> then replacing names. When I find myself doing that, I think: there
>> has to be a better way.
>
> yup. However, it seems to me that the reason you are cutting and pasting
> code is that you have followed a consistent convention in your code
> (convention over configuration, as the Rails people say), which is a
> good thing. However, to me, the alternative to your cutting and pasting
> is not to try to come up with a compact way to express your convention
> in Cython, but rather code-generation.
>
> One day, we may have a mature Cython code generator for wrapping C/C++
> (and it may be worth looking at what has been done along those lines).
> But in the meantime, you could pretty quickly whip out a code generator
> that works for your particular use-case. Maybe use a general purpose
> template system (Cheetah, for instance) would help?

+1

- Robert

From robert.kern at gmail.com  Thu Nov  4 21:47:50 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Thu, 04 Nov 2010 15:47:50 -0500
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <4CD2E573.80907@noaa.gov>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>	<AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>	<AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>	<4CD11C30.9090103@student.matnat.uio.no>	<AANLkTimABt0xae_NBG1MVhLGx5-AMf4ps4m7jT2bcQQC@mail.gmail.com>	<AANLkTi=5DttGcLp6SEv0w06YVratHpc_CSdCSpoiU-rD@mail.gmail.com>	<AANLkTikpWy1GW-ht-37faSvb3gTkpGPnOkycC9qY-Wyn@mail.gmail.com>	<AANLkTi=k_MKp0czueQ+w6JE+60Jtbkg=+9-TNE2L=wnu@mail.gmail.com>	<AANLkTik9OEfmg=wUYNNrcGURTA8hjXCt-eUPZMmZpWsh@mail.gmail.com>
	<4CD2E573.80907@noaa.gov>
Message-ID: <iav65n$1n3$1@dough.gmane.org>

On 11/4/10 11:55 AM, Christopher Barker wrote:

> One day, we may have a mature Cython code generator for wrapping C/C++
> (and it may be worth looking at what has been done along those lines).
> But in the meantime, you could pretty quickly whip out a code generator
> that works for your particular use-case. Maybe use a general purpose
> template system (Cheetah, for instance) would help?

FWIW, Tempita has a tiny implementation that could be dropped into Cython but is 
as full-featured as you could want for code generation. You would want to avoid 
Cheetah or the other larger templating packages. Many of their Web-focused 
features would be a hindrance for this use.

   http://pythonpaste.org/tempita/

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From stefan_ml at behnel.de  Thu Nov  4 22:28:31 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 04 Nov 2010 22:28:31 +0100
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <4CD2E573.80907@noaa.gov>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>	<AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>	<AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>	<4CD11C30.9090103@student.matnat.uio.no>	<AANLkTimABt0xae_NBG1MVhLGx5-AMf4ps4m7jT2bcQQC@mail.gmail.com>	<AANLkTi=5DttGcLp6SEv0w06YVratHpc_CSdCSpoiU-rD@mail.gmail.com>	<AANLkTikpWy1GW-ht-37faSvb3gTkpGPnOkycC9qY-Wyn@mail.gmail.com>	<AANLkTi=k_MKp0czueQ+w6JE+60Jtbkg=+9-TNE2L=wnu@mail.gmail.com>	<AANLkTik9OEfmg=wUYNNrcGURTA8hjXCt-eUPZMmZpWsh@mail.gmail.com>
	<4CD2E573.80907@noaa.gov>
Message-ID: <4CD3257F.5060608@behnel.de>

Christopher Barker, 04.11.2010 17:55:
> One day, we may have a mature Cython code generator for wrapping C/C++
> (and it may be worth looking at what has been done along those lines).

The code generation has never been the main problem. That would be parsing 
arbitrary header files (preferably without adding huge dependencies like 
gccxml or clang) and figuring out what's relevant enough to merit ending up 
in the generated code. Once that's done, generating a Cython .pxd file 
and/or "mostly usable" code skeleton from it is rather straight forward.

Stefan

From Chris.Barker at noaa.gov  Thu Nov  4 22:40:32 2010
From: Chris.Barker at noaa.gov (Christopher Barker)
Date: Thu, 04 Nov 2010 14:40:32 -0700
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <4CD3257F.5060608@behnel.de>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>
	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>
	<AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>
	<AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>
	<4CD11C30.9090103@student.matnat.uio.no>
	<AANLkTimABt0xae_NBG1MVhLGx5-AMf4ps4m7jT2bcQQC@mail.gmail.com>
	<AANLkTi=5DttGcLp6SEv0w06YVratHpc_CSdCSpoiU-rD@mail.gmail.com>
	<AANLkTikpWy1GW-ht-37faSvb3gTkpGPnOkycC9qY-Wyn@mail.gmail.com>
	<AANLkTi=k_MKp0czueQ+w6JE+60Jtbkg=+9-TNE2L=wnu@mail.gmail.com>
	<AANLkTik9OEfmg=wUYNNrcGURTA8hjXCt-eUPZMmZpWsh@mail.gmail.com>
	<4CD2E573.80907@noaa.gov> <4CD3257F.5060608@behnel.de>
Message-ID: <4CD32850.1090809@noaa.gov>

On 11/4/10 2:28 PM, Stefan Behnel wrote:
> Christopher Barker, 04.11.2010 17:55:
>> One day, we may have a mature Cython code generator for wrapping C/C++
>> (and it may be worth looking at what has been done along those lines).
>
> The code generation has never been the main problem. That would be parsing
> arbitrary header files (preferably without adding huge dependencies like
> gccxml or clang) and figuring out what's relevant enough to merit ending up
> in the generated code. Once that's done, generating a Cython .pxd file
> and/or "mostly usable" code skeleton from it is rather straight forward.

well, when I wrote "code generation", I meant to imply the whole kit and 
kaboodle.

What about Doxygen for the parsing? Or still too huge a dependency?

Anyway, part of my point is that if you have a bunch of code that 
follows conventions, auto-generating cython that, for instance, turns 
every GetThis-SetThis pair into a python property would be pretty easy.

Making it fully general is another story.

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov

From robertwb at math.washington.edu  Thu Nov  4 22:50:54 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 4 Nov 2010 14:50:54 -0700
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <4CD32850.1090809@noaa.gov>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>
	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>
	<AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>
	<AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>
	<4CD11C30.9090103@student.matnat.uio.no>
	<AANLkTimABt0xae_NBG1MVhLGx5-AMf4ps4m7jT2bcQQC@mail.gmail.com>
	<AANLkTi=5DttGcLp6SEv0w06YVratHpc_CSdCSpoiU-rD@mail.gmail.com>
	<AANLkTikpWy1GW-ht-37faSvb3gTkpGPnOkycC9qY-Wyn@mail.gmail.com>
	<AANLkTi=k_MKp0czueQ+w6JE+60Jtbkg=+9-TNE2L=wnu@mail.gmail.com>
	<AANLkTik9OEfmg=wUYNNrcGURTA8hjXCt-eUPZMmZpWsh@mail.gmail.com>
	<4CD2E573.80907@noaa.gov> <4CD3257F.5060608@behnel.de>
	<4CD32850.1090809@noaa.gov>
Message-ID: <AANLkTi=pW_B+QN-PujHrT1AMABNX1AJE9Oj2zFSir=0S@mail.gmail.com>

On Thu, Nov 4, 2010 at 2:40 PM, Christopher Barker
<Chris.Barker at noaa.gov> wrote:
> On 11/4/10 2:28 PM, Stefan Behnel wrote:
>> Christopher Barker, 04.11.2010 17:55:
>>> One day, we may have a mature Cython code generator for wrapping C/C++
>>> (and it may be worth looking at what has been done along those lines).
>>
>> The code generation has never been the main problem. That would be parsing
>> arbitrary header files (preferably without adding huge dependencies like
>> gccxml or clang) and figuring out what's relevant enough to merit ending up
>> in the generated code. Once that's done, generating a Cython .pxd file
>> and/or "mostly usable" code skeleton from it is rather straight forward.
>
> well, when I wrote "code generation", I meant to imply the whole kit and
> kaboodle.
>
> What about Doxygen for the parsing? Or still too huge a dependency?

I would not be opposed to having a separate dependency for the code
parser/generator--once one has the .pxd files those can be shipped and
used elsewhere. For simple cases (e.g. basic C header files) a
relatively lightweight thing would be nice.

> Anyway, part of my point is that if you have a bunch of code that
> follows conventions, auto-generating cython that, for instance, turns
> every GetThis-SetThis pair into a python property would be pretty easy.
>
> Making it fully general is another story.

Yep. Another general solution would be a full lisp-like macro system,
but that's a more design and implementation work than would take place
in the near future.

- Robert

From brett.calcott at gmail.com  Thu Nov  4 23:17:51 2010
From: brett.calcott at gmail.com (Brett Calcott)
Date: Fri, 5 Nov 2010 09:17:51 +1100
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <4CD2E573.80907@noaa.gov>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>
	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>
	<AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>
	<AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>
	<4CD11C30.9090103@student.matnat.uio.no>
	<AANLkTimABt0xae_NBG1MVhLGx5-AMf4ps4m7jT2bcQQC@mail.gmail.com>
	<AANLkTi=5DttGcLp6SEv0w06YVratHpc_CSdCSpoiU-rD@mail.gmail.com>
	<AANLkTikpWy1GW-ht-37faSvb3gTkpGPnOkycC9qY-Wyn@mail.gmail.com>
	<AANLkTi=k_MKp0czueQ+w6JE+60Jtbkg=+9-TNE2L=wnu@mail.gmail.com>
	<AANLkTik9OEfmg=wUYNNrcGURTA8hjXCt-eUPZMmZpWsh@mail.gmail.com>
	<4CD2E573.80907@noaa.gov>
Message-ID: <AANLkTinr6G0HsQLyeXECE6mu+61HpTnCwvs+j-uhO7j0@mail.gmail.com>

On 5 November 2010 03:55, Christopher Barker <Chris.Barker at noaa.gov> wrote:
> On 11/3/10 11:47 PM, Brett Calcott wrote:
>> If I just had to write one of these functions, then
>> indeed trying to do it as a one-liner would be silly. But I'm wrapping
>> lots of them, and this means I'm just cutting and pasting code, and
>> then replacing names. When I find myself doing that, I think: there
>> has to be a better way.
>
> yup. However, it seems to me that the reason you are cutting and pasting
> code is that you have followed a consistent convention in your code
> (convention over configuration, as the Rails people say), which is a
> good thing. However, to me, the alternative to your cutting and pasting
> is not to try to come up with a compact way to express your convention
> in Cython, but rather code-generation.
>
> One day, we may have a mature Cython code generator for wrapping C/C++
> (and it may be worth looking at what has been done along those lines).
> But in the meantime, you could pretty quickly whip out a code generator
> that works for your particular use-case. Maybe use a general purpose
> template system (Cheetah, for instance) would help?
>

I may be guilty of thinking of cython as a "code-generator" :(

From brett.calcott at gmail.com  Thu Nov  4 23:25:06 2010
From: brett.calcott at gmail.com (Brett Calcott)
Date: Fri, 5 Nov 2010 09:25:06 +1100
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <4CD3257F.5060608@behnel.de>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>
	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>
	<AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>
	<AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>
	<4CD11C30.9090103@student.matnat.uio.no>
	<AANLkTimABt0xae_NBG1MVhLGx5-AMf4ps4m7jT2bcQQC@mail.gmail.com>
	<AANLkTi=5DttGcLp6SEv0w06YVratHpc_CSdCSpoiU-rD@mail.gmail.com>
	<AANLkTikpWy1GW-ht-37faSvb3gTkpGPnOkycC9qY-Wyn@mail.gmail.com>
	<AANLkTi=k_MKp0czueQ+w6JE+60Jtbkg=+9-TNE2L=wnu@mail.gmail.com>
	<AANLkTik9OEfmg=wUYNNrcGURTA8hjXCt-eUPZMmZpWsh@mail.gmail.com>
	<4CD2E573.80907@noaa.gov> <4CD3257F.5060608@behnel.de>
Message-ID: <AANLkTikWcstj2Td1BFHeMTZcAywh+MNA9GJn0-ufysMQ@mail.gmail.com>

On 5 November 2010 08:28, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Christopher Barker, 04.11.2010 17:55:
>> One day, we may have a mature Cython code generator for wrapping C/C++
>> (and it may be worth looking at what has been done along those lines).
>
> The code generation has never been the main problem. That would be parsing
> arbitrary header files (preferably without adding huge dependencies like
> gccxml or clang) and figuring out what's relevant enough to merit ending up
> in the generated code. Once that's done, generating a Cython .pxd file
> and/or "mostly usable" code skeleton from it is rather straight forward.
>

Getting the pxd files automated would be nice. But I *like*
constructing the pyx files by hand, as I don't want to simply copy the
c/c++ interface (like swig does). It means you can rethink how the
structure of a library best works in a python-like way. The problem
I'm having (which I'm getting solutions too - thanks!) is that a large
proportion of python code ends up being dumb property mapping stuff.

In any case, thanks for listening, and the suggestions. I hope to be
hanging around for a while.

Brett

From robertwb at math.washington.edu  Thu Nov  4 23:37:57 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 4 Nov 2010 15:37:57 -0700
Subject: [Cython] Wrapping C++ - questions and thoughts
In-Reply-To: <AANLkTikWcstj2Td1BFHeMTZcAywh+MNA9GJn0-ufysMQ@mail.gmail.com>
References: <AANLkTi=wP+PFeLpKpQxoL7+-+jgnW76=FikcLmKjyEVq@mail.gmail.com>
	<AANLkTinYg3N9uyeQgKU0GwN56S-Ta7pQdYX8j6n+K1MN@mail.gmail.com>
	<AANLkTikoAM7QpTK1+XMAj6fUsdy_Ti6oCT1i2NhdW=qJ@mail.gmail.com>
	<AANLkTik8ZpvSNCh7mMd3dVH1ee_BBshW=fWNKSMOoasm@mail.gmail.com>
	<4CD11C30.9090103@student.matnat.uio.no>
	<AANLkTimABt0xae_NBG1MVhLGx5-AMf4ps4m7jT2bcQQC@mail.gmail.com>
	<AANLkTi=5DttGcLp6SEv0w06YVratHpc_CSdCSpoiU-rD@mail.gmail.com>
	<AANLkTikpWy1GW-ht-37faSvb3gTkpGPnOkycC9qY-Wyn@mail.gmail.com>
	<AANLkTi=k_MKp0czueQ+w6JE+60Jtbkg=+9-TNE2L=wnu@mail.gmail.com>
	<AANLkTik9OEfmg=wUYNNrcGURTA8hjXCt-eUPZMmZpWsh@mail.gmail.com>
	<4CD2E573.80907@noaa.gov> <4CD3257F.5060608@behnel.de>
	<AANLkTikWcstj2Td1BFHeMTZcAywh+MNA9GJn0-ufysMQ@mail.gmail.com>
Message-ID: <AANLkTimiDpeAUMibTfCTZoS32s8_+zXP-2pPTmXHkocL@mail.gmail.com>

On Thu, Nov 4, 2010 at 3:25 PM, Brett Calcott <brett.calcott at gmail.com> wrote:

> I may be guilty of thinking of cython as a "code-generator" :(

Well, aren't all compilers :). There's a distinction between code
generators and wrapper generators (which you seem to understand). Of
course automatically generating .pxd files for use from Cython is a
slightly different thing that would be very handy to have.

> On 5 November 2010 08:28, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Christopher Barker, 04.11.2010 17:55:
>>> One day, we may have a mature Cython code generator for wrapping C/C++
>>> (and it may be worth looking at what has been done along those lines).
>>
>> The code generation has never been the main problem. That would be parsing
>> arbitrary header files (preferably without adding huge dependencies like
>> gccxml or clang) and figuring out what's relevant enough to merit ending up
>> in the generated code. Once that's done, generating a Cython .pxd file
>> and/or "mostly usable" code skeleton from it is rather straight forward.
>>
>
> Getting the pxd files automated would be nice. But I *like*
> constructing the pyx files by hand, as I don't want to simply copy the
> c/c++ interface (like swig does). It means you can rethink how the
> structure of a library best works in a python-like way.

Yes, I think this is extremely valuable too.

> The problem
> I'm having (which I'm getting solutions too - thanks!) is that a large
> proportion of python code ends up being dumb property mapping stuff.

Yeah. I don't think we've hit on a nice Pythonic way to do this yet,
but I agree it could be better. If there's a lot, I might be tempted
to write a one-off script generating this portion of the file--as
mentioned, much easier than getting the general case right.

> In any case, thanks for listening, and the suggestions. I hope to be
> hanging around for a while.

Good.

- Robert

From brett.calcott at gmail.com  Fri Nov  5 05:56:23 2010
From: brett.calcott at gmail.com (Brett Calcott)
Date: Fri, 5 Nov 2010 15:56:23 +1100
Subject: [Cython] parsing gotcha
Message-ID: <AANLkTinL8K+DUQ7VSvGSaZm5Fk0W3SZx1zgp0uROQCuC@mail.gmail.com>

To create pxd files that wrap a C++ interface, I often just cutting
and pasting from the headers then cleaning up the code. A few of the
headers use "def" as a variable name, like this:
cdef extern from "test.h":
    cdef cppclass Test:
        void func(int def)

It took me a while to figure out that the variable name was the
problem (my brain was blanking them as they're not even needed). Maybe
the parse error could be more better than:
/Users/brett/Sandbox/test.pyx:3:22: Expected ')'

Brett

From brett.calcott at gmail.com  Fri Nov  5 06:05:48 2010
From: brett.calcott at gmail.com (Brett Calcott)
Date: Fri, 5 Nov 2010 16:05:48 +1100
Subject: [Cython] __add__ problems
Message-ID: <AANLkTinQyj_RYmwPNdEkKNXOUYnCaYwLFuqzgEMFN7BP@mail.gmail.com>

What am I doing wrong here. The same code compiles as "def add", but
not as "def __add__"?

cdef extern from "box2d/Common/b2Math.h":
    cdef cppclass b2Vec2:
        float x, y
        b2Vec2()
        b2Vec2(float x, float y)
        b2Vec2 operator+(b2Vec2 v)

cdef class Vec2:
    cdef b2Vec2 *obj
    def __cinit__(self, float x=0.0, float y=0.0):
        self.obj = new b2Vec2(x, y)

    def __dealloc__(self):
        del self.obj

    # This compiles
    def add(self, Vec2 other):
        ret = Vec2()
        ret.obj[0] = self.obj[0] + other.obj[0]
        return ret

    # This does not
    def __add__(self, Vec2 other):
        ret = Vec2()
        ret.obj[0] = self.obj[0] + other.obj[0]  # error here
        return ret

error is "test.pyx:25:33: Cannot convert Python object to 'b2Vec2' "

From robertwb at math.washington.edu  Fri Nov  5 06:54:44 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 4 Nov 2010 22:54:44 -0700
Subject: [Cython] __add__ problems
In-Reply-To: <AANLkTinQyj_RYmwPNdEkKNXOUYnCaYwLFuqzgEMFN7BP@mail.gmail.com>
References: <AANLkTinQyj_RYmwPNdEkKNXOUYnCaYwLFuqzgEMFN7BP@mail.gmail.com>
Message-ID: <AANLkTim4PyHKuhjnpO_TktE+wxFpjJv71gnjG3wrpsSk@mail.gmail.com>

On Thu, Nov 4, 2010 at 10:05 PM, Brett Calcott <brett.calcott at gmail.com> wrote:
> What am I doing wrong here. The same code compiles as "def add", but
> not as "def __add__"?
>
> cdef extern from "box2d/Common/b2Math.h":
> ? ?cdef cppclass b2Vec2:
> ? ? ? ?float x, y
> ? ? ? ?b2Vec2()
> ? ? ? ?b2Vec2(float x, float y)
> ? ? ? ?b2Vec2 operator+(b2Vec2 v)
>
> cdef class Vec2:
> ? ?cdef b2Vec2 *obj
> ? ?def __cinit__(self, float x=0.0, float y=0.0):
> ? ? ? ?self.obj = new b2Vec2(x, y)
>
> ? ?def __dealloc__(self):
> ? ? ? ?del self.obj
>
> ? ?# This compiles
> ? ?def add(self, Vec2 other):
> ? ? ? ?ret = Vec2()
> ? ? ? ?ret.obj[0] = self.obj[0] + other.obj[0]
> ? ? ? ?return ret
>
> ? ?# This does not
> ? ?def __add__(self, Vec2 other):
> ? ? ? ?ret = Vec2()
> ? ? ? ?ret.obj[0] = self.obj[0] + other.obj[0] ?# error here
> ? ? ? ?return ret
>
> error is "test.pyx:25:33: Cannot convert Python object to 'b2Vec2' "

See http://docs.cython.org/src/userguide/special_methods.html#special-methods

From brett.calcott at gmail.com  Fri Nov  5 07:20:52 2010
From: brett.calcott at gmail.com (Brett Calcott)
Date: Fri, 5 Nov 2010 17:20:52 +1100
Subject: [Cython] __add__ problems
In-Reply-To: <AANLkTim4PyHKuhjnpO_TktE+wxFpjJv71gnjG3wrpsSk@mail.gmail.com>
References: <AANLkTinQyj_RYmwPNdEkKNXOUYnCaYwLFuqzgEMFN7BP@mail.gmail.com>
	<AANLkTim4PyHKuhjnpO_TktE+wxFpjJv71gnjG3wrpsSk@mail.gmail.com>
Message-ID: <AANLkTinNjvGGg7RQ-3Qtpq1zh+6DNmbkwHLJDCp+Nj-j@mail.gmail.com>

oh. I see. No self. I had read that page too.

From stefan_ml at behnel.de  Fri Nov  5 08:22:04 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 05 Nov 2010 08:22:04 +0100
Subject: [Cython] parsing gotcha
In-Reply-To: <AANLkTinL8K+DUQ7VSvGSaZm5Fk0W3SZx1zgp0uROQCuC@mail.gmail.com>
References: <AANLkTinL8K+DUQ7VSvGSaZm5Fk0W3SZx1zgp0uROQCuC@mail.gmail.com>
Message-ID: <4CD3B09C.8080602@behnel.de>

Brett Calcott, 05.11.2010 05:56:
> To create pxd files that wrap a C++ interface, I often just cutting
> and pasting from the headers then cleaning up the code. A few of the
> headers use "def" as a variable name, like this:
> cdef extern from "test.h":
>      cdef cppclass Test:
>          void func(int def)
>
> It took me a while to figure out that the variable name was the
> problem (my brain was blanking them as they're not even needed). Maybe
> the parse error could be more better than:
> /Users/brett/Sandbox/test.pyx:3:22: Expected ')'

Right, that's not very telling. I pushed a fix that shows at least what was 
found instead, i.e. you'd now get

     /Users/brett/Sandbox/test.pyx:3:22: Expected ')', found 'def'

Stefan

From mistobaan at gmail.com  Sat Nov  6 01:50:15 2010
From: mistobaan at gmail.com (Fabrizio Milo aka misto)
Date: Fri, 5 Nov 2010 17:50:15 -0700
Subject: [Cython] Cython Error on Macos X snow leopard
Message-ID: <AANLkTinH0cm=iyHmcqbjxVX4bOAvqOHOg1ethcXxW9r5@mail.gmail.com>

Plain macos install

easy_install
cython <file>

Traceback (most recent call last):
  File "/usr/local/bin/cython", line 8, in <module>
    load_entry_point('Cython==0.13', 'console_scripts', 'cython')()
  File "/Library/Python/2.6/site-packages/Cython-0.13-py2.6-macosx-10.6-universal.egg/Cython/Compiler/Main.py",
line 759, in setuptools_main
    return main(command_line = 1)
  File "/Library/Python/2.6/site-packages/Cython-0.13-py2.6-macosx-10.6-universal.egg/Cython/Compiler/Main.py",
line 776, in main
    result = compile(sources, options)
  File "/Library/Python/2.6/site-packages/Cython-0.13-py2.6-macosx-10.6-universal.egg/Cython/Compiler/Main.py",
line 751, in compile
    return compile_multiple(source, options)
  File "/Library/Python/2.6/site-packages/Cython-0.13-py2.6-macosx-10.6-universal.egg/Cython/Compiler/Main.py",
line 723, in compile_multiple
    result = run_pipeline(source, options)
  File "/Library/Python/2.6/site-packages/Cython-0.13-py2.6-macosx-10.6-universal.egg/Cython/Compiler/Main.py",
line 593, in run_pipeline
    err, enddata = context.run_pipeline(pipeline, source)
  File "/Library/Python/2.6/site-packages/Cython-0.13-py2.6-macosx-10.6-universal.egg/Cython/Compiler/Main.py",
line 234, in run_pipeline
    data = phase(data)
  File "Visitor.py", line 276, in
Cython.Compiler.Visitor.CythonTransform.__call__
(/var/folders/F4/F4bjdeyFHqSkP82qVpZM2U+++TQ/-Tmp-/easy_install-f4XQDC/Cython-0.13/Cython/Compiler/Visitor.c:4924)
  File "Visitor.py", line 259, in
Cython.Compiler.Visitor.VisitorTransform.__call__
(/var/folders/F4/F4bjdeyFHqSkP82qVpZM2U+++TQ/-Tmp-/easy_install-f4XQDC/Cython-0.13/Cython/Compiler/Visitor.c:4684)
  File "Visitor.py", line 28, in
Cython.Compiler.Visitor.BasicVisitor.visit
(/var/folders/F4/F4bjdeyFHqSkP82qVpZM2U+++TQ/-Tmp-/easy_install-f4XQDC/Cython-0.13/Cython/Compiler/Visitor.c:1178)
  File "/Library/Python/2.6/site-packages/Cython-0.13-py2.6-macosx-10.6-universal.egg/Cython/Compiler/ParseTreeTransforms.py",
line 1146, in visit_ModuleNode
    node.body.analyse_expressions(node.scope)
  File "/Library/Python/2.6/site-packages/Cython-0.13-py2.6-macosx-10.6-universal.egg/Cython/Compiler/Nodes.py",
line 346, in analyse_expressions
    stat.analyse_expressions(env)
  File "/Library/Python/2.6/site-packages/Cython-0.13-py2.6-macosx-10.6-universal.egg/Cython/Compiler/Nodes.py",
line 346, in analyse_expressions
    stat.analyse_expressions(env)
  File "/Library/Python/2.6/site-packages/Cython-0.13-py2.6-macosx-10.6-universal.egg/Cython/Compiler/Nodes.py",
line 3270, in analyse_expressions
    self.analyse_types(env)
  File "/Library/Python/2.6/site-packages/Cython-0.13-py2.6-macosx-10.6-universal.egg/Cython/Compiler/Nodes.py",
line 3364, in analyse_types
    self.rhs = self.rhs.coerce_to(self.lhs.type, env)
  File "/Library/Python/2.6/site-packages/Cython-0.13-py2.6-macosx-10.6-universal.egg/Cython/Compiler/ExprNodes.py",
line 572, in coerce_to
    src = CoerceFromPyTypeNode(dst_type, src, env)
  File "/Library/Python/2.6/site-packages/Cython-0.13-py2.6-macosx-10.6-universal.egg/Cython/Compiler/ExprNodes.py",
line 6634, in __init__
    if not result_type.create_from_py_utility_code(env):
AttributeError: 'UnspecifiedType' object has no attribute
'create_from_py_utility_code'


-- 
--------------------------
Luck favors the prepared mind. (Pasteur)

From robertwb at math.washington.edu  Sat Nov  6 04:12:58 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 5 Nov 2010 20:12:58 -0700
Subject: [Cython] Cython Error on Macos X snow leopard
In-Reply-To: <AANLkTinH0cm=iyHmcqbjxVX4bOAvqOHOg1ethcXxW9r5@mail.gmail.com>
References: <AANLkTinH0cm=iyHmcqbjxVX4bOAvqOHOg1ethcXxW9r5@mail.gmail.com>
Message-ID: <AANLkTi=Qh3pdy9TvxQtigr0+U1-QapWb=mY8H=-u4=8_@mail.gmail.com>

Could you provide the code?

On Fri, Nov 5, 2010 at 5:50 PM, Fabrizio Milo aka misto
<mistobaan at gmail.com> wrote:
> Plain macos install
>
> easy_install
> cython <file>
>
> Traceback (most recent call last):
> ?File "/usr/local/bin/cython", line 8, in <module>
> ? ?load_entry_point('Cython==0.13', 'console_scripts', 'cython')()
> ?File "/Library/Python/2.6/site-packages/Cython-0.13-py2.6-macosx-10.6-universal.egg/Cython/Compiler/Main.py",
> line 759, in setuptools_main
> ? ?return main(command_line = 1)
> ?File "/Library/Python/2.6/site-packages/Cython-0.13-py2.6-macosx-10.6-universal.egg/Cython/Compiler/Main.py",
> line 776, in main
> ? ?result = compile(sources, options)
> ?File "/Library/Python/2.6/site-packages/Cython-0.13-py2.6-macosx-10.6-universal.egg/Cython/Compiler/Main.py",
> line 751, in compile
> ? ?return compile_multiple(source, options)
> ?File "/Library/Python/2.6/site-packages/Cython-0.13-py2.6-macosx-10.6-universal.egg/Cython/Compiler/Main.py",
> line 723, in compile_multiple
> ? ?result = run_pipeline(source, options)
> ?File "/Library/Python/2.6/site-packages/Cython-0.13-py2.6-macosx-10.6-universal.egg/Cython/Compiler/Main.py",
> line 593, in run_pipeline
> ? ?err, enddata = context.run_pipeline(pipeline, source)
> ?File "/Library/Python/2.6/site-packages/Cython-0.13-py2.6-macosx-10.6-universal.egg/Cython/Compiler/Main.py",
> line 234, in run_pipeline
> ? ?data = phase(data)
> ?File "Visitor.py", line 276, in
> Cython.Compiler.Visitor.CythonTransform.__call__
> (/var/folders/F4/F4bjdeyFHqSkP82qVpZM2U+++TQ/-Tmp-/easy_install-f4XQDC/Cython-0.13/Cython/Compiler/Visitor.c:4924)
> ?File "Visitor.py", line 259, in
> Cython.Compiler.Visitor.VisitorTransform.__call__
> (/var/folders/F4/F4bjdeyFHqSkP82qVpZM2U+++TQ/-Tmp-/easy_install-f4XQDC/Cython-0.13/Cython/Compiler/Visitor.c:4684)
> ?File "Visitor.py", line 28, in
> Cython.Compiler.Visitor.BasicVisitor.visit
> (/var/folders/F4/F4bjdeyFHqSkP82qVpZM2U+++TQ/-Tmp-/easy_install-f4XQDC/Cython-0.13/Cython/Compiler/Visitor.c:1178)
> ?File "/Library/Python/2.6/site-packages/Cython-0.13-py2.6-macosx-10.6-universal.egg/Cython/Compiler/ParseTreeTransforms.py",
> line 1146, in visit_ModuleNode
> ? ?node.body.analyse_expressions(node.scope)
> ?File "/Library/Python/2.6/site-packages/Cython-0.13-py2.6-macosx-10.6-universal.egg/Cython/Compiler/Nodes.py",
> line 346, in analyse_expressions
> ? ?stat.analyse_expressions(env)
> ?File "/Library/Python/2.6/site-packages/Cython-0.13-py2.6-macosx-10.6-universal.egg/Cython/Compiler/Nodes.py",
> line 346, in analyse_expressions
> ? ?stat.analyse_expressions(env)
> ?File "/Library/Python/2.6/site-packages/Cython-0.13-py2.6-macosx-10.6-universal.egg/Cython/Compiler/Nodes.py",
> line 3270, in analyse_expressions
> ? ?self.analyse_types(env)
> ?File "/Library/Python/2.6/site-packages/Cython-0.13-py2.6-macosx-10.6-universal.egg/Cython/Compiler/Nodes.py",
> line 3364, in analyse_types
> ? ?self.rhs = self.rhs.coerce_to(self.lhs.type, env)
> ?File "/Library/Python/2.6/site-packages/Cython-0.13-py2.6-macosx-10.6-universal.egg/Cython/Compiler/ExprNodes.py",
> line 572, in coerce_to
> ? ?src = CoerceFromPyTypeNode(dst_type, src, env)
> ?File "/Library/Python/2.6/site-packages/Cython-0.13-py2.6-macosx-10.6-universal.egg/Cython/Compiler/ExprNodes.py",
> line 6634, in __init__
> ? ?if not result_type.create_from_py_utility_code(env):
> AttributeError: 'UnspecifiedType' object has no attribute
> 'create_from_py_utility_code'
>
>
> --
> --------------------------
> Luck favors the prepared mind. (Pasteur)
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From baihaoyu at gmail.com  Sat Nov  6 08:39:08 2010
From: baihaoyu at gmail.com (Haoyu Bai)
Date: Sat, 6 Nov 2010 15:39:08 +0800
Subject: [Cython] gsoc-haoyu merge candidates for 0.13.1?
In-Reply-To: <4CCBC370.2090604@behnel.de>
References: <4C6E763A.80602@behnel.de> <4CCBC370.2090604@behnel.de>
Message-ID: <AANLkTi=V7aQQti0TXQ+UN7WVT=eZf5k77t_yrxinJZev@mail.gmail.com>

On Sat, Oct 30, 2010 at 3:04 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> so, what's the status? I wouldn't like to see this dangling around for too
> long. There's a lot of interesting and nice stuff to merge here.
>
>
> Stefan Behnel, 20.08.2010 14:34:
>>
>> your GSoC branch has been around for a while now. Given that 0.13 will be
>> out soon (soon-isher than ever before), is there anything that you
>> consider
>> safe enough to get merged into mainline after the release?
>>
>> IMHO, any feature patch is a candidate, especially when it depends on
>> currently illegal syntax (i.e. not impacting existing code). Everything
>> that changes current behaviour is worth considering if it fixes a bug or
>> Python compatibility issue but may have to wait for 0.14 if it impacts
>> code
>> that currently works with 0.13.
>
> Haoyu, you uploaded patches to Rietvelt. However, they don't include the hg
> metadata, so I can't just apply them. Is it easy enough for you (or Craig)
> to provide complete exported patches for each ticket?
>
>
>> ?From a quick look, I think #488 (ellipsis) is safe, but I'd like to see
>> some more syntax tests - I bet there are some in Python's own test suite.
>
> Is the space issue resolved now?
>
>
>> Ticket #487 (multiple 'with' statements) looks nice, but clearly lacks
>> test. I can't even see a test that makes sure the context managers are
>> executed in the correct nesting order, neither can I see anything that
>> tests the chaining of cleanup actions during exception propagation, let
>> alone partial propagation in cases where one of the context managers
>> swallows an exception. Again, Python's test suite will provide hints on
>> better tests here.
>>
>> Ticket #490 (nonlocal), while I'd be happy to get it in, seems too big a
>> change to go into 0.13.1.
>>
>> It seems you also have a fix for #477. That would be another candidate for
>> 0.13.1. Note that the related test case doesn't actually test that the
>> argument typing has the expected effect. This could be done using a tree
>> assertion based on coercion nodes.
>
> Do we have more complete tests for these now?
>

Hi,

Thanks for getting these merged! It's great that now I can move my
focus to the other patches. I think #477 (@cython.locals for cdef
function) will be the next one. I'll try to cleanup and add more tests
for #542 (relative import) and #423 (explicit exception chaining
syntax) as well.

> I also took another quick look at the relative import code. Currently, we
> have the restriction that Cython requires (and consequently has) knowledge
> about the exact package layout for an extension module. I wonder if we
> shouldn't just use that information for relative imports.
>
> Stefan
>

The relative import feature I implemented is Python's runtime relative
import. I'm not sure whether Cython need compile-time relative import
and how that should be done.

Thanks!
-- 
Haoyu BAI
School of Computing,
National University of Singapore.

From vitja.makarov at gmail.com  Sat Nov  6 21:37:50 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 6 Nov 2010 23:37:50 +0300
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTik4qhsVhHhCQRyheKpDzoGGQD-q5r=bJUi2_VV_@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<AANLkTi=a8SMk+u7JfLKxujR3p2ZVnjLsSPjrMY8hufbt@mail.gmail.com>
	<AANLkTimjYc=4HdLE+VDmUyn0piXPWGZ7zWGqQ0QoabkP@mail.gmail.com>
	<AANLkTim0dL=7nR6WiXAYQyMUSyQ=_b7jsBuKCQ8zh-WT@mail.gmail.com>
	<AANLkTim6qdY6hmj2VVJJ_rwje5Sup5NQjspZuymVEv8Z@mail.gmail.com>
	<AANLkTimWfQO1SsVgDX9yyex=xivq0z7nmw0PJoJBXvFc@mail.gmail.com>
	<4CD1C983.7000300@behnel.de>
	<AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>
	<4CD1D787.4010202@behnel.de>
	<AANLkTikYRv6CPTbSifo-C_gAwU-AG0REC1sPwAHq2DJO@mail.gmail.com>
	<4CD247CD.705@behnel.de>
	<AANLkTimEmE9=xiN6=PDNEgRo=P_=ECxE=n7Eh0aYs81g@mail.gmail.com>
	<AANLkTimOXxppAQ6Fb5DSCTyRoY9ZJghhV5=sM3i0N6ic@mail.gmail.com>
	<4CD29CFD.7030309@behnel.de>
	<AANLkTinP8Osz1YKkLyuDhPT=vzyc6iscnOCoi2h0f+Rv@mail.gmail.com>
	<4CD2A3EB.4030601@behnel.de>
	<AANLkTinHaapMihFN0vHaRD2o3xKVD6Ows-JjD7W_6mzx@mail.gmail.com>
	<4CD2B27E.6080805@behnel.de> <4CD2B585.8040601@behnel.de>
	<AANLkTik4qhsVhHhCQRyheKpDzoGGQD-q5r=bJUi2_VV_@mail.gmail.com>
Message-ID: <AANLkTik1H1z-R23Kc6e7FgdMhGdX7JPX5iy=j33oS-J3@mail.gmail.com>

2010/11/4 Vitja Makarov <vitja.makarov at gmail.com>:
> 2010/11/4 Stefan Behnel <stefan_ml at behnel.de>:
>> Stefan Behnel, 04.11.2010 14:17:
>>> BTW, I'll ask on python-dev about the way __build_class__ works here.
>>
>> ;) writing e-mails about a problem without sending them away is the best
>> way to understand the topic.
>>
>> Ok, so, __build_class__ is actually ok, because it *is* the point where the
>> class body is evaluated. So, __prepare__ is really called before the
>> evaluation of the body (which is done in a "function" call near the end)
>> and the metaclass is called afterwards.
>>
>> We could essentially do the same thing in Cython. At the point where the
>> class dict is now getting instantiated, we need to put the code of the
>> first part of __build_class__, i.e. look up the metaclass from the keywords
>> and find and call its __prepare__ method. Then, evaluate the class body
>> into the class namespace object that __prepare__ returned, followed by the
>> call to __Pyx_CreateClass().
>>
>> If we don't find a metaclass in the keywords, we do not support __prepare__
>> but try to look up the __metaclass__ field in the class namespace during
>> __Pyx_CreateClass().
>>
>> Stefan
>>
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
> I have attached patch (in trac) for python3 metaclass support,
> __prepare__ now works this way:
>
> if metaclass is defined py3 way and it has __prepare__, namespace is
> merged into dict.
> This is wrong but it's much easy to implement, right behavior could be
> imlemented later.
>
> This could be done this way:
>
> 1. PyClassDefNode detect that class definition is py3 style and has
> metaclass keyword defined, it sets some flag
> 2. then code generation should work this way:
>
> name = <class name>
> bases = <tuple of bases>
> metaclass = <metaclass>
> mkw = <metaclass keywords dict>
>
> if hasattr(metaclass, '__prepare__'):
> ? ?dict = __Pyx_PrepareClass(metaclass, bases, name, mkw)
> else:
> ? ?dict = {}
> # now fill dict with attributes
>
> margs = tuple(name, bases, dict)
> klass = PyEval_CallObjectWithKeywords(metaclass, margs, mkw)
>
>
> This is interesting to implement, I'll try...
>

I'm now trying to implement this:

Is it correct:

If parser detect use of kwargs in class definition it creates Py3ClassDefNode

I add this classes:
Nodes.Py3ClassDefNode   - ClassDefNode replacement for python3 style classes
ExprNodes.Py3ClassNode - ClassNode replacement for python3 style classes

ExprNdoes.Py3MetaclassNode - contains metaclass
   Holds value returned by __Pyx_Py3MetaclassGet(bases, mkw)

ExprNodes.Py3NamespaceNode - contains namespace object
   Value returned by __Pyx_Py3MetaclassPrepare(metaclass, base, name,
mkw), that calls __prepare__ if any.

From stefan_ml at behnel.de  Sun Nov  7 09:18:42 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 07 Nov 2010 09:18:42 +0100
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTik1H1z-R23Kc6e7FgdMhGdX7JPX5iy=j33oS-J3@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>	<AANLkTim0dL=7nR6WiXAYQyMUSyQ=_b7jsBuKCQ8zh-WT@mail.gmail.com>	<AANLkTim6qdY6hmj2VVJJ_rwje5Sup5NQjspZuymVEv8Z@mail.gmail.com>	<AANLkTimWfQO1SsVgDX9yyex=xivq0z7nmw0PJoJBXvFc@mail.gmail.com>	<4CD1C983.7000300@behnel.de>	<AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>	<4CD1D787.4010202@behnel.de>	<AANLkTikYRv6CPTbSifo-C_gAwU-AG0REC1sPwAHq2DJO@mail.gmail.com>	<4CD247CD.705@behnel.de>	<AANLkTimEmE9=xiN6=PDNEgRo=P_=ECxE=n7Eh0aYs81g@mail.gmail.com>	<AANLkTimOXxppAQ6Fb5DSCTyRoY9ZJghhV5=sM3i0N6ic@mail.gmail.com>	<4CD29CFD.7030309@behnel.de>	<AANLkTinP8Osz1YKkLyuDhPT=vzyc6iscnOCoi2h0f+Rv@mail.gmail.com>	<4CD2A3EB.4030601@behnel.de>	<AANLkTinHaapMihFN0vHaRD2o3xKVD6Ows-JjD7W_6mzx@mail.gmail.com>	<4CD2B27E.6080805@behnel.de>
	<4CD2B585.8040601@behnel.de>	<AANLkTik4qhsVhHhCQRyheKpDzoGGQD-q5r=bJUi2_VV_@mail.gmail.com>
	<AANLkTik1H1z-R23Kc6e7FgdMhGdX7JPX5iy=j33oS-J3@mail.gmail.com>
Message-ID: <4CD660E2.9040800@behnel.de>

Vitja Makarov, 06.11.2010 21:37:
> I'm now trying to implement this:
>
> Is it correct:
>
> If parser detect use of kwargs in class definition it creates Py3ClassDefNode
>
> I add this classes:
> Nodes.Py3ClassDefNode   - ClassDefNode replacement for python3 style classes
> ExprNodes.Py3ClassNode - ClassNode replacement for python3 style classes
>
> ExprNdoes.Py3MetaclassNode - contains metaclass
>     Holds value returned by __Pyx_Py3MetaclassGet(bases, mkw)
>
> ExprNodes.Py3NamespaceNode - contains namespace object
>     Value returned by __Pyx_Py3MetaclassPrepare(metaclass, base, name,
> mkw), that calls __prepare__ if any.

Why do you think these new classes are necessary?

Stefan

From vitja.makarov at gmail.com  Sun Nov  7 09:41:24 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sun, 7 Nov 2010 11:41:24 +0300
Subject: [Cython] metaclasses
In-Reply-To: <4CD660E2.9040800@behnel.de>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<AANLkTim0dL=7nR6WiXAYQyMUSyQ=_b7jsBuKCQ8zh-WT@mail.gmail.com>
	<AANLkTim6qdY6hmj2VVJJ_rwje5Sup5NQjspZuymVEv8Z@mail.gmail.com>
	<AANLkTimWfQO1SsVgDX9yyex=xivq0z7nmw0PJoJBXvFc@mail.gmail.com>
	<4CD1C983.7000300@behnel.de>
	<AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>
	<4CD1D787.4010202@behnel.de>
	<AANLkTikYRv6CPTbSifo-C_gAwU-AG0REC1sPwAHq2DJO@mail.gmail.com>
	<4CD247CD.705@behnel.de>
	<AANLkTimEmE9=xiN6=PDNEgRo=P_=ECxE=n7Eh0aYs81g@mail.gmail.com>
	<AANLkTimOXxppAQ6Fb5DSCTyRoY9ZJghhV5=sM3i0N6ic@mail.gmail.com>
	<4CD29CFD.7030309@behnel.de>
	<AANLkTinP8Osz1YKkLyuDhPT=vzyc6iscnOCoi2h0f+Rv@mail.gmail.com>
	<4CD2A3EB.4030601@behnel.de>
	<AANLkTinHaapMihFN0vHaRD2o3xKVD6Ows-JjD7W_6mzx@mail.gmail.com>
	<4CD2B27E.6080805@behnel.de> <4CD2B585.8040601@behnel.de>
	<AANLkTik4qhsVhHhCQRyheKpDzoGGQD-q5r=bJUi2_VV_@mail.gmail.com>
	<AANLkTik1H1z-R23Kc6e7FgdMhGdX7JPX5iy=j33oS-J3@mail.gmail.com>
	<4CD660E2.9040800@behnel.de>
Message-ID: <AANLkTimCCTQoSmgi91Xz_QVm_4uLBXkhd=fwGuzX6-Y7@mail.gmail.com>

2010/11/7 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 06.11.2010 21:37:
>> I'm now trying to implement this:
>>
>> Is it correct:
>>
>> If parser detect use of kwargs in class definition it creates Py3ClassDefNode
>>
>> I add this classes:
>> Nodes.Py3ClassDefNode ? - ClassDefNode replacement for python3 style classes
>> ExprNodes.Py3ClassNode - ClassNode replacement for python3 style classes
>>
>> ExprNdoes.Py3MetaclassNode - contains metaclass
>> ? ? Holds value returned by __Pyx_Py3MetaclassGet(bases, mkw)
>>
>> ExprNodes.Py3NamespaceNode - contains namespace object
>> ? ? Value returned by __Pyx_Py3MetaclassPrepare(metaclass, base, name,
>> mkw), that calls __prepare__ if any.
>
> Why do you think these new classes are necessary?
>

I think this objects are required to represent metaclass and namespace.
If we want real support for __prepare__ (simple version is attached in trac)

1. extract metaclass from bases and keywordargs
2. create namespace by calling metaclass __preapre__
3. fill namespace with class body
4. create class

I don't want to reuse ClassDefNode as it differs too much. To many if/else.
So Py3ClassNode should hold metaclass and namespace objects, I add
Py3MetaclassNode and Py3NamespaceNode.

Can I use PyTempNode for metaclass and namespace?

Is there another way?

From vitja.makarov at gmail.com  Sun Nov  7 12:39:22 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sun, 7 Nov 2010 14:39:22 +0300
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTimCCTQoSmgi91Xz_QVm_4uLBXkhd=fwGuzX6-Y7@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<AANLkTim0dL=7nR6WiXAYQyMUSyQ=_b7jsBuKCQ8zh-WT@mail.gmail.com>
	<AANLkTim6qdY6hmj2VVJJ_rwje5Sup5NQjspZuymVEv8Z@mail.gmail.com>
	<AANLkTimWfQO1SsVgDX9yyex=xivq0z7nmw0PJoJBXvFc@mail.gmail.com>
	<4CD1C983.7000300@behnel.de>
	<AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>
	<4CD1D787.4010202@behnel.de>
	<AANLkTikYRv6CPTbSifo-C_gAwU-AG0REC1sPwAHq2DJO@mail.gmail.com>
	<4CD247CD.705@behnel.de>
	<AANLkTimEmE9=xiN6=PDNEgRo=P_=ECxE=n7Eh0aYs81g@mail.gmail.com>
	<AANLkTimOXxppAQ6Fb5DSCTyRoY9ZJghhV5=sM3i0N6ic@mail.gmail.com>
	<4CD29CFD.7030309@behnel.de>
	<AANLkTinP8Osz1YKkLyuDhPT=vzyc6iscnOCoi2h0f+Rv@mail.gmail.com>
	<4CD2A3EB.4030601@behnel.de>
	<AANLkTinHaapMihFN0vHaRD2o3xKVD6Ows-JjD7W_6mzx@mail.gmail.com>
	<4CD2B27E.6080805@behnel.de> <4CD2B585.8040601@behnel.de>
	<AANLkTik4qhsVhHhCQRyheKpDzoGGQD-q5r=bJUi2_VV_@mail.gmail.com>
	<AANLkTik1H1z-R23Kc6e7FgdMhGdX7JPX5iy=j33oS-J3@mail.gmail.com>
	<4CD660E2.9040800@behnel.de>
	<AANLkTimCCTQoSmgi91Xz_QVm_4uLBXkhd=fwGuzX6-Y7@mail.gmail.com>
Message-ID: <AANLkTi=ZDvsiw5wWMM52sS1YcWL_3wZxZL-_4DMm_ORE@mail.gmail.com>

2010/11/7 Vitja Makarov <vitja.makarov at gmail.com>:
> 2010/11/7 Stefan Behnel <stefan_ml at behnel.de>:
>> Vitja Makarov, 06.11.2010 21:37:
>>> I'm now trying to implement this:
>>>
>>> Is it correct:
>>>
>>> If parser detect use of kwargs in class definition it creates Py3ClassDefNode
>>>
>>> I add this classes:
>>> Nodes.Py3ClassDefNode ? - ClassDefNode replacement for python3 style classes
>>> ExprNodes.Py3ClassNode - ClassNode replacement for python3 style classes
>>>
>>> ExprNdoes.Py3MetaclassNode - contains metaclass
>>> ? ? Holds value returned by __Pyx_Py3MetaclassGet(bases, mkw)
>>>
>>> ExprNodes.Py3NamespaceNode - contains namespace object
>>> ? ? Value returned by __Pyx_Py3MetaclassPrepare(metaclass, base, name,
>>> mkw), that calls __prepare__ if any.
>>
>> Why do you think these new classes are necessary?
>>
>
> I think this objects are required to represent metaclass and namespace.
> If we want real support for __prepare__ (simple version is attached in trac)
>
> 1. extract metaclass from bases and keywordargs
> 2. create namespace by calling metaclass __preapre__
> 3. fill namespace with class body
> 4. create class
>
> I don't want to reuse ClassDefNode as it differs too much. To many if/else.
> So Py3ClassNode should hold metaclass and namespace objects, I add
> Py3MetaclassNode and Py3NamespaceNode.
>
> Can I use PyTempNode for metaclass and namespace?
>
> Is there another way?
>

Another way:

if py3_metaclass:
     mkw = keyword_args.copy()
     mkw.update(starstar_args)
     metaclass = py3_metaclass_get(bases, mkw)
     dict = py3_namespace_prepare(metaclass, bases, name, mkw)
else:
     dict = dict()

# construct class body here...
dict['foo'] = 'bar'

if py3_metaclass:
    classobj = py3_metaclass_new(metaclass, name, bases, dict, mkw, modname)
else:
    classobj = create_class(bases, dict, name, modname)

2 new object types here: metaclass, namespace.
How can this all be done without creating new Node types for metaclass and dict?

From stefan_ml at behnel.de  Sun Nov  7 21:39:06 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 07 Nov 2010 21:39:06 +0100
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTimCCTQoSmgi91Xz_QVm_4uLBXkhd=fwGuzX6-Y7@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>	<AANLkTim6qdY6hmj2VVJJ_rwje5Sup5NQjspZuymVEv8Z@mail.gmail.com>	<AANLkTimWfQO1SsVgDX9yyex=xivq0z7nmw0PJoJBXvFc@mail.gmail.com>	<4CD1C983.7000300@behnel.de>	<AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>	<4CD1D787.4010202@behnel.de>	<AANLkTikYRv6CPTbSifo-C_gAwU-AG0REC1sPwAHq2DJO@mail.gmail.com>	<4CD247CD.705@behnel.de>	<AANLkTimEmE9=xiN6=PDNEgRo=P_=ECxE=n7Eh0aYs81g@mail.gmail.com>	<AANLkTimOXxppAQ6Fb5DSCTyRoY9ZJghhV5=sM3i0N6ic@mail.gmail.com>	<4CD29CFD.7030309@behnel.de>	<AANLkTinP8Osz1YKkLyuDhPT=vzyc6iscnOCoi2h0f+Rv@mail.gmail.com>	<4CD2A3EB.4030601@behnel.de>	<AANLkTinHaapMihFN0vHaRD2o3xKVD6Ows-JjD7W_6mzx@mail.gmail.com>	<4CD2B27E.6080805@behnel.de>
	<4CD2B585.8040601@behnel.de>	<AANLkTik4qhsVhHhCQRyheKpDzoGGQD-q5r=bJUi2_VV_@mail.gmail.com>	<AANLkTik1H1z-R23Kc6e7FgdMhGdX7JPX5iy=j33oS-J3@mail.gmail.com>	<4CD660E2.9040800@behnel.de>
	<AANLkTimCCTQoSmgi91Xz_QVm_4uLBXkhd=fwGuzX6-Y7@mail.gmail.com>
Message-ID: <4CD70E6A.1080101@behnel.de>

Vitja Makarov, 07.11.2010 09:41:
> 2010/11/7 Stefan Behnel:
>> Vitja Makarov, 06.11.2010 21:37:
>>> I'm now trying to implement this:
>>>
>>> Is it correct:
>>>
>>> If parser detect use of kwargs in class definition it creates Py3ClassDefNode
>>>
>>> I add this classes:
>>> Nodes.Py3ClassDefNode   - ClassDefNode replacement for python3 style classes
>>> ExprNodes.Py3ClassNode - ClassNode replacement for python3 style classes
>>>
>>> ExprNdoes.Py3MetaclassNode - contains metaclass
>>>      Holds value returned by __Pyx_Py3MetaclassGet(bases, mkw)
>>>
>>> ExprNodes.Py3NamespaceNode - contains namespace object
>>>      Value returned by __Pyx_Py3MetaclassPrepare(metaclass, base, name,
>>> mkw), that calls __prepare__ if any.
>>
>> Why do you think these new classes are necessary?
>>
>
> I think this objects are required to represent metaclass and namespace.
> If we want real support for __prepare__ (simple version is attached in trac)

Let me take a closer look at your current patch, I'll try to find some time 
tomorrow.


> 1. extract metaclass from bases and keywordargs
> 2. create namespace by calling metaclass __preapre__
> 3. fill namespace with class body
> 4. create class
>
> I don't want to reuse ClassDefNode as it differs too much. To many if/else.
> So Py3ClassNode should hold metaclass and namespace objects, I add
> Py3MetaclassNode and Py3NamespaceNode.
>
> Can I use PyTempNode for metaclass and namespace?

Temporary variables are handled mostly automatically in Cython. I'll try to 
figure out what's needed here.

Stefan

From markflorisson88 at gmail.com  Sun Nov  7 21:40:02 2010
From: markflorisson88 at gmail.com (mark florisson)
Date: Sun, 7 Nov 2010 21:40:02 +0100
Subject: [Cython] Cython debugger
In-Reply-To: <AANLkTikrH31rvjyzjS+BWoSnkOZ+x3PUeg2ufqm4c2Wk@mail.gmail.com>
References: <AANLkTinjhVuH=vqA=EFySaBKh+j6Rk1U-Gn0qDLLoUaB@mail.gmail.com>
	<4C8F3E56.2020705@student.matnat.uio.no>
	<1284482379.13330.333.camel@radiator.bos.redhat.com>
	<AANLkTimzNdwzsRo6vAymnMfAH+uBU7UG+OqPSF=Ugrp2@mail.gmail.com>
	<AANLkTikOF4j_7v-AzHFaT=vzp+RTYxEFOBGc1M8HC9Wz@mail.gmail.com>
	<AANLkTi=WSoP5wkC9+0cmSwSCPfLAKFghO9OSwkSx+peb@mail.gmail.com>
	<AANLkTikrH31rvjyzjS+BWoSnkOZ+x3PUeg2ufqm4c2Wk@mail.gmail.com>
Message-ID: <AANLkTikvw2pyxx2VTD=s1URk_EOAhsE56Ce8wK9+wB8p@mail.gmail.com>

> I'm not a gdb expert, but I definitely want to try this out.
> Documentation would be great--either as a wiki or, preferably, the
> main documentation (with a little note mentioning it's pending
> release). I've given you push access to
> http://hg.cython.org/cython-docs

A "Debugging your Cython program" page was added to the docs. If
someone feels certain topics need more elaboration, please let me
know!

From stefan_ml at behnel.de  Sun Nov  7 21:44:37 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 07 Nov 2010 21:44:37 +0100
Subject: [Cython] Cython debugger
In-Reply-To: <AANLkTikvw2pyxx2VTD=s1URk_EOAhsE56Ce8wK9+wB8p@mail.gmail.com>
References: <AANLkTinjhVuH=vqA=EFySaBKh+j6Rk1U-Gn0qDLLoUaB@mail.gmail.com>	<4C8F3E56.2020705@student.matnat.uio.no>	<1284482379.13330.333.camel@radiator.bos.redhat.com>	<AANLkTimzNdwzsRo6vAymnMfAH+uBU7UG+OqPSF=Ugrp2@mail.gmail.com>	<AANLkTikOF4j_7v-AzHFaT=vzp+RTYxEFOBGc1M8HC9Wz@mail.gmail.com>	<AANLkTi=WSoP5wkC9+0cmSwSCPfLAKFghO9OSwkSx+peb@mail.gmail.com>	<AANLkTikrH31rvjyzjS+BWoSnkOZ+x3PUeg2ufqm4c2Wk@mail.gmail.com>
	<AANLkTikvw2pyxx2VTD=s1URk_EOAhsE56Ce8wK9+wB8p@mail.gmail.com>
Message-ID: <4CD70FB5.3010304@behnel.de>

mark florisson, 07.11.2010 21:40:
>> I'm not a gdb expert, but I definitely want to try this out.
>> Documentation would be great--either as a wiki or, preferably, the
>> main documentation (with a little note mentioning it's pending
>> release). I've given you push access to
>> http://hg.cython.org/cython-docs
>
> A "Debugging your Cython program" page was added to the docs. If
> someone feels certain topics need more elaboration, please let me
> know!

For those who want to read the latest version, it's here:

https://sage.math.washington.edu:8091/hudson/job/cython-docs/doclinks/1/src/userguide/debugging.html

Stefan

From matej at laitl.cz  Mon Nov  8 01:13:01 2010
From: matej at laitl.cz (=?UTF-8?B?TWF0xJtqIExhaXRs?=)
Date: Mon, 8 Nov 2010 01:13:01 +0100
Subject: [Cython] Cython 0.13.1 - bugfix for ticket 583 (Wrong code
	generated)
Message-ID: <AANLkTinaDcV0Vu0E4NPB2xsZdCHj3VAzaiPSg=NkLOXT@mail.gmail.com>

Hi,

On Wed, Nov 3, 2010 at 18:18, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> I think it's time we start thinking about rolling out another release.
> I've got a couple of experimental features in the hopper I'd like to
> get out, we've started to merge some of Haoyu's code (perhaps there's
> more that should go in), got __metaclass__ support thanks to Vitja,
> some cool gdb support from Mark (that I haven't had a chance to look
> at, but I've no reason to doubt it's ready or could be soon, and a
> bunch of other bugfixes/minor features. How does everyone feel about a
> target date of this time next month? This one shouldn't drag on--for
> one thing Sage is still in good shape:
> https://sage.math.washington.edu:8091/hudson/job/sage-tests/ Any
> specific features people want to be sure to get in?

I tried hard and luckily produced partial fix for bug 583 (Wrong C
code generated (cdef class < cdef class inheritance, cpdef methods,
pure python mode), details are in the ticket:
http://trac.cython.org/cython_trac/ticket/583

Sadly, I'm clueless how to fix it completely. It would be great if a
fix for this bug could make it into 0.13.1 as my code [1] depends on
it.

[1] PyBayes - emerging library for bayesian filtering and decision making

Regards,
              Mat?j Laitl

From stefan_ml at behnel.de  Mon Nov  8 09:03:17 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 08 Nov 2010 09:03:17 +0100
Subject: [Cython] Cython 0.13.1 - bugfix for ticket 583 (Wrong code
	generated)
In-Reply-To: <AANLkTinaDcV0Vu0E4NPB2xsZdCHj3VAzaiPSg=NkLOXT@mail.gmail.com>
References: <AANLkTinaDcV0Vu0E4NPB2xsZdCHj3VAzaiPSg=NkLOXT@mail.gmail.com>
Message-ID: <4CD7AEC5.7090805@behnel.de>

Mat?j Laitl, 08.11.2010 01:13:
> I tried hard and luckily produced partial fix for bug 583 (Wrong C
> code generated (cdef class<  cdef class inheritance, cpdef methods,
> pure python mode), details are in the ticket:
> http://trac.cython.org/cython_trac/ticket/583
>
> Sadly, I'm clueless how to fix it completely. It would be great if a
> fix for this bug could make it into 0.13.1 as my code [1] depends on
> it.

Your patch is a good hint, I'm looking into it.

Stefan

From stefan_ml at behnel.de  Mon Nov  8 11:59:55 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 08 Nov 2010 11:59:55 +0100
Subject: [Cython] Cython 0.13.1 - bugfix for ticket 583 (Wrong code
	generated)
In-Reply-To: <4CD7AEC5.7090805@behnel.de>
References: <AANLkTinaDcV0Vu0E4NPB2xsZdCHj3VAzaiPSg=NkLOXT@mail.gmail.com>
	<4CD7AEC5.7090805@behnel.de>
Message-ID: <4CD7D82B.80607@behnel.de>

Stefan Behnel, 08.11.2010 09:03:
> Mat?j Laitl, 08.11.2010 01:13:
>> I tried hard and luckily produced partial fix for bug 583 (Wrong C
>> code generated (cdef class<   cdef class inheritance, cpdef methods,
>> pure python mode), details are in the ticket:
>> http://trac.cython.org/cython_trac/ticket/583
>>
>> Sadly, I'm clueless how to fix it completely. It would be great if a
>> fix for this bug could make it into 0.13.1 as my code [1] depends on
>> it.
>
> Your patch is a good hint, I'm looking into it.

What you do in your patch would provide a quick fix.

Personally, I think the right way to really fix this would be to finally 
refactor def functions into cdef function wrappers. That way, overriding 
pure def functions in .pxd files would simply enable the underlying cdef 
function as being callable, instead of needing to build a new one from 
scratch. And we could even override def functions as pure cdef functions by 
simply dropping the wrapper entirely.

Stefan

From dsdale24 at gmail.com  Mon Nov  8 15:34:54 2010
From: dsdale24 at gmail.com (Darren Dale)
Date: Mon, 8 Nov 2010 09:34:54 -0500
Subject: [Cython] >=python-2.6 property pattern not supported
Message-ID: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>

Python-2.6 extended the definition of property to include getter,
setter, and deleter methods that can be used as decorators. This
allows a really nice pattern for declaring properties without
populating the namespace with a bunch of private methods:

class Foo(object):

    _bar = None
    @property
    def bar(self):
        return self._bar
    @bar.setter
    def bar(self, val):
        self._bar = val

What happens is @bar.setter makes a copy of bar, updates the copy to
use the decorated method as the setter, and returns the copy, which is
then bound to the name of the decorated function. The decorated
function has to be called bar, so that the updated copy is bound to
bar, replacing the original.

Attempting to cythonize this example yields an error:

$ python setup.py build_ext --inplace
running build_ext
cythoning test_property.pyx to test_property.c

Error converting Pyrex file to C:
------------------------------------------------------------
...
    _bar = None
    @property
    def bar(self):
        return self._bar
    @bar.setter
    def bar(self, val):
   ^
------------------------------------------------------------

/Users/darren/temp/test/test_property.pyx:11:4: 'bar' already declared
building 'test_property' extension
/usr/bin/gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -O2
-DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes
-I/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
-c test_property.c -o
build/temp.macosx-10.6-x86_64-2.7/test_property.o
test_property.c:1:2: error: #error Do not use this file, it is the
result of a failed Cython compilation.
error: command '/usr/bin/gcc-4.2' failed with exit status 1


Would it be possible for Cython to support this pattern? I think it
will be increasingly common. For example: I recently pointed it out on
the PyQt and PySide mailing lists. In both cases, the developers were
pretty enthusiastic about the pattern once they were aware of it. The
most recent versions of PyQt and dip now support it for their own
properties, and the PySide developers are working on implementing it.

Thanks,
Darren

From stefan_ml at behnel.de  Mon Nov  8 15:51:10 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 08 Nov 2010 15:51:10 +0100
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTi=ZDvsiw5wWMM52sS1YcWL_3wZxZL-_4DMm_ORE@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>	<AANLkTimWfQO1SsVgDX9yyex=xivq0z7nmw0PJoJBXvFc@mail.gmail.com>	<4CD1C983.7000300@behnel.de>	<AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>	<4CD1D787.4010202@behnel.de>	<AANLkTikYRv6CPTbSifo-C_gAwU-AG0REC1sPwAHq2DJO@mail.gmail.com>	<4CD247CD.705@behnel.de>	<AANLkTimEmE9=xiN6=PDNEgRo=P_=ECxE=n7Eh0aYs81g@mail.gmail.com>	<AANLkTimOXxppAQ6Fb5DSCTyRoY9ZJghhV5=sM3i0N6ic@mail.gmail.com>	<4CD29CFD.7030309@behnel.de>	<AANLkTinP8Osz1YKkLyuDhPT=vzyc6iscnOCoi2h0f+Rv@mail.gmail.com>	<4CD2A3EB.4030601@behnel.de>	<AANLkTinHaapMihFN0vHaRD2o3xKVD6Ows-JjD7W_6mzx@mail.gmail.com>	<4CD2B27E.6080805@behnel.de>
	<4CD2B585.8040601@behnel.de>	<AANLkTik4qhsVhHhCQRyheKpDzoGGQD-q5r=bJUi2_VV_@mail.gmail.com>	<AANLkTik1H1z-R23Kc6e7FgdMhGdX7JPX5iy=j33oS-J3@mail.gmail.com>	<4CD660E2.9040800@behnel.de>	<AANLkTimCCTQoSmgi91Xz_QVm_4uLBXkhd=fwGuzX6-Y7@mail.gmail.com>
	<AANLkTi=ZDvsiw5wWMM52sS1YcWL_3wZxZL-_4DMm_ORE@mail.gmail.com>
Message-ID: <4CD80E5E.6020307@behnel.de>

Hi,

I looked through your code and pushed a slightly cleaned up version of your 
latest patch.

Vitja Makarov, 07.11.2010 12:39:
> 2010/11/7 Vitja Makarov:
>> 2010/11/7 Stefan Behnel:
>>> Vitja Makarov, 06.11.2010 21:37:
>>>> I'm now trying to implement this:
>>>>
>>>> Is it correct:
>>>>
>>>> If parser detect use of kwargs in class definition it creates Py3ClassDefNode
>>>>
>>>> I add this classes:
>>>> Nodes.Py3ClassDefNode   - ClassDefNode replacement for python3 style classes
>>>> ExprNodes.Py3ClassNode - ClassNode replacement for python3 style classes
>>>>
>>>> ExprNdoes.Py3MetaclassNode - contains metaclass
>>>>      Holds value returned by __Pyx_Py3MetaclassGet(bases, mkw)
>>>>
>>>> ExprNodes.Py3NamespaceNode - contains namespace object
>>>>      Value returned by __Pyx_Py3MetaclassPrepare(metaclass, base, name,
>>>> mkw), that calls __prepare__ if any.
>>>
>>> Why do you think these new classes are necessary?
>>
>> I think this objects are required to represent metaclass and namespace.
>> If we want real support for __prepare__ (simple version is attached in trac)
>>
>> 1. extract metaclass from bases and keywordargs
>> 2. create namespace by calling metaclass __preapre__
>> 3. fill namespace with class body
>> 4. create class
>>
>> I don't want to reuse ClassDefNode as it differs too much. To many if/else.
>> So Py3ClassNode should hold metaclass and namespace objects, I add
>> Py3MetaclassNode and Py3NamespaceNode.
>>
>> Can I use PyTempNode for metaclass and namespace?
>>
>> Is there another way?
>
> Another way:
>
> if py3_metaclass:
>       mkw = keyword_args.copy()
>       mkw.update(starstar_args)
>       metaclass = py3_metaclass_get(bases, mkw)
>       dict = py3_namespace_prepare(metaclass, bases, name, mkw)
> else:
>       dict = dict()
>
> # construct class body here...
> dict['foo'] = 'bar'
>
> if py3_metaclass:
>      classobj = py3_metaclass_new(metaclass, name, bases, dict, mkw, modname)
> else:
>      classobj = create_class(bases, dict, name, modname)
>
> 2 new object types here: metaclass, namespace.

Yes, I think that's the way to go. The two new nodes should go into 
ExprNodes, maybe named "PyClassMetaclassNode" and "PyClassNamespaceNode". 
Both must have their "is_temp" set to True.

The metaclass node will look up the metaclass in the type keywords and use 
it as its node result value.

The namespace node will be used by ExprNodes.ClassNode just like before but 
should do the "prepare() or dict()" dance above. It obviously needs a 
reference to the metaclass node to do that, in order to access its result 
value.

Using two new node classes here is the easiest way to do it as you only 
need to pass around references to them. Otherwise, you'd have to manually 
allocate temporary variables and pass around their names. That's not any 
harder to do, but it's a lot more ugly.

Note the disposal code call at the end of 
PyClassDefNode.generate_execution_code(). You need to do that for both 
nodes here, that will automatically take care to properly clean up the 
Python references to their results (based on the value of their "is_temp" 
attribute).

Stefan

From stefan_ml at behnel.de  Mon Nov  8 16:05:22 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 08 Nov 2010 16:05:22 +0100
Subject: [Cython] >=python-2.6 property pattern not supported
In-Reply-To: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>
Message-ID: <4CD811B2.1010108@behnel.de>

Darren Dale, 08.11.2010 15:34:
> Python-2.6 extended the definition of property to include getter,
> setter, and deleter methods that can be used as decorators. This
> allows a really nice pattern for declaring properties without
> populating the namespace with a bunch of private methods:
>
> class Foo(object):
>
>      _bar = None
>      @property
>      def bar(self):
>          return self._bar
>      @bar.setter
>      def bar(self, val):
>          self._bar = val
>
> What happens is @bar.setter makes a copy of bar, updates the copy to
> use the decorated method as the setter, and returns the copy, which is
> then bound to the name of the decorated function. The decorated
> function has to be called bar, so that the updated copy is bound to
> bar, replacing the original.
>
> Attempting to cythonize this example yields an error:
>
> $ python setup.py build_ext --inplace
> running build_ext
> cythoning test_property.pyx to test_property.c
>
> Error converting Pyrex file to C:
> ------------------------------------------------------------
> ...
>      _bar = None
>      @property
>      def bar(self):
>          return self._bar
>      @bar.setter
>      def bar(self, val):
>     ^
> ------------------------------------------------------------
>
> /Users/darren/temp/test/test_property.pyx:11:4: 'bar' already declared
>
> Would it be possible for Cython to support this pattern?

The problem is not the property behaviour, it's the mapping from a Python 
function/method to a C function, which must have a unique name. Currently, 
functions are unique within one namespace, be it the module namespace or a 
class body namespace. They are actually looked up statically, not 
dynamically in definition order, so the fix is not as easy as adding a 
counted ID to the C-level name.

The above might actually work if you change one of the method names, i.e. 
either the getter or setter name. If it works as you say, the setter name 
shouldn't have any impact on the result.

Stefan

From stefan_ml at behnel.de  Mon Nov  8 16:07:16 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 08 Nov 2010 16:07:16 +0100
Subject: [Cython] >=python-2.6 property pattern not supported
In-Reply-To: <4CD811B2.1010108@behnel.de>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>
	<4CD811B2.1010108@behnel.de>
Message-ID: <4CD81224.7030409@behnel.de>

Stefan Behnel, 08.11.2010 16:05:
> The problem is not the property behaviour, it's the mapping from a Python
> function/method to a C function, which must have a unique name. Currently,
> functions are unique within one namespace, be it the module namespace or a
> class body namespace. They are actually looked up statically, not
> dynamically in definition order, so the fix is not as easy as adding a
> counted ID to the C-level name.

Here's the related bug report, BTW:

http://trac.cython.org/cython_trac/ticket/489

Stefan

From dsdale24 at gmail.com  Mon Nov  8 16:24:09 2010
From: dsdale24 at gmail.com (Darren Dale)
Date: Mon, 8 Nov 2010 10:24:09 -0500
Subject: [Cython] >=python-2.6 property pattern not supported
In-Reply-To: <4CD811B2.1010108@behnel.de>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>
	<4CD811B2.1010108@behnel.de>
Message-ID: <AANLkTi=MxmQQ_3a0vU7J867krhJJAhWotpBEGNhOGr+b@mail.gmail.com>

On Mon, Nov 8, 2010 at 10:05 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Darren Dale, 08.11.2010 15:34:
>> Python-2.6 extended the definition of property to include getter,
>> setter, and deleter methods that can be used as decorators. This
>> allows a really nice pattern for declaring properties without
>> populating the namespace with a bunch of private methods:
>>
>> class Foo(object):
>>
>> ? ? ?_bar = None
>> ? ? ?@property
>> ? ? ?def bar(self):
>> ? ? ? ? ?return self._bar
>> ? ? ?@bar.setter
>> ? ? ?def bar(self, val):
>> ? ? ? ? ?self._bar = val
>>
>> What happens is @bar.setter makes a copy of bar, updates the copy to
>> use the decorated method as the setter, and returns the copy, which is
>> then bound to the name of the decorated function. The decorated
>> function has to be called bar, so that the updated copy is bound to
>> bar, replacing the original.
>>
>> Attempting to cythonize this example yields an error:
>>
>> $ python setup.py build_ext --inplace
>> running build_ext
>> cythoning test_property.pyx to test_property.c
>>
>> Error converting Pyrex file to C:
>> ------------------------------------------------------------
>> ...
>> ? ? ?_bar = None
>> ? ? ?@property
>> ? ? ?def bar(self):
>> ? ? ? ? ?return self._bar
>> ? ? ?@bar.setter
>> ? ? ?def bar(self, val):
>> ? ? ^
>> ------------------------------------------------------------
>>
>> /Users/darren/temp/test/test_property.pyx:11:4: 'bar' already declared
>>
>> Would it be possible for Cython to support this pattern?
>
> The problem is not the property behaviour, it's the mapping from a Python
> function/method to a C function, which must have a unique name. Currently,
> functions are unique within one namespace, be it the module namespace or a
> class body namespace. They are actually looked up statically, not
> dynamically in definition order, so the fix is not as easy as adding a
> counted ID to the C-level name.
>
> The above might actually work if you change one of the method names, i.e.
> either the getter or setter name. If it works as you say, the setter name
> shouldn't have any impact on the result.

Sorry, I should have emphasized the point about the importance of the
setter name. It *must* be the same as the original property.

class Foo(object):

    _bar = None
    @property
    def bar(self):
        return self._bar
    @bar.setter
    def bar_(self, val):
        self._bar = val

That produces a class with two properties: bar and bar_. Foo.bar is a
property with a getter but no setter. Foo.bar_ is a property with a
getter and a setter.

Darren

From stefan_ml at behnel.de  Mon Nov  8 17:01:53 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 08 Nov 2010 17:01:53 +0100
Subject: [Cython] >=python-2.6 property pattern not supported
In-Reply-To: <AANLkTi=MxmQQ_3a0vU7J867krhJJAhWotpBEGNhOGr+b@mail.gmail.com>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>	<4CD811B2.1010108@behnel.de>
	<AANLkTi=MxmQQ_3a0vU7J867krhJJAhWotpBEGNhOGr+b@mail.gmail.com>
Message-ID: <4CD81EF1.2080000@behnel.de>

Darren Dale, 08.11.2010 16:24:
> On Mon, Nov 8, 2010 at 10:05 AM, Stefan Behnel wrote:
>> Darren Dale, 08.11.2010 15:34:
>>> Python-2.6 extended the definition of property to include getter,
>>> setter, and deleter methods that can be used as decorators. This
>>> allows a really nice pattern for declaring properties without
>>> populating the namespace with a bunch of private methods:
>>>
>>> class Foo(object):
>>>
>>>       _bar = None
>>>       @property
>>>       def bar(self):
>>>           return self._bar
>>>       @bar.setter
>>>       def bar(self, val):
>>>           self._bar = val
>>>
>>> What happens is @bar.setter makes a copy of bar, updates the copy to
>>> use the decorated method as the setter, and returns the copy, which is
>>> then bound to the name of the decorated function. The decorated
>>> function has to be called bar, so that the updated copy is bound to
>>> bar, replacing the original.
>>>
>>> Attempting to cythonize this example yields an error:
>>>
>>> $ python setup.py build_ext --inplace
>>> running build_ext
>>> cythoning test_property.pyx to test_property.c
>>>
>>> Error converting Pyrex file to C:
>>> ------------------------------------------------------------
>>> ...
>>>       _bar = None
>>>       @property
>>>       def bar(self):
>>>           return self._bar
>>>       @bar.setter
>>>       def bar(self, val):
>>>      ^
>>> ------------------------------------------------------------
>>>
>>> /Users/darren/temp/test/test_property.pyx:11:4: 'bar' already declared
>>>
>>> Would it be possible for Cython to support this pattern?
>>
>> The problem is not the property behaviour, it's the mapping from a Python
>> function/method to a C function, which must have a unique name. Currently,
>> functions are unique within one namespace, be it the module namespace or a
>> class body namespace. They are actually looked up statically, not
>> dynamically in definition order, so the fix is not as easy as adding a
>> counted ID to the C-level name.
>>
>> The above might actually work if you change one of the method names, i.e.
>> either the getter or setter name. If it works as you say, the setter name
>> shouldn't have any impact on the result.
>
> Sorry, I should have emphasized the point about the importance of the
> setter name. It *must* be the same as the original property.
>
> class Foo(object):
>
>      _bar = None
>      @property
>      def bar(self):
>          return self._bar
>      @bar.setter
>      def bar_(self, val):
>          self._bar = val
>
> That produces a class with two properties: bar and bar_. Foo.bar is a
> property with a getter but no setter. Foo.bar_ is a property with a
> getter and a setter.

Renaming the getter works for me, though:

     class Foo(object):
         _bar = None

         @property
         def bar_(self):
             return self._bar

         @bar_.setter
         def bar(self, val):
             self._bar = val

         del bar_

     f = Foo()
     print f.bar
     f.bar = 5
     print f.bar

     try:
         Foo.bar_
     except AttributeError:
         pass
     except:
         print "hoppsa!"

I admit that it's ugly to do a "del bar_" at the end, but it's not more 
than a work-around after all...

Stefan

From dsdale24 at gmail.com  Mon Nov  8 19:13:22 2010
From: dsdale24 at gmail.com (Darren Dale)
Date: Mon, 8 Nov 2010 13:13:22 -0500
Subject: [Cython] >=python-2.6 property pattern not supported
In-Reply-To: <4CD81EF1.2080000@behnel.de>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>
	<4CD811B2.1010108@behnel.de>
	<AANLkTi=MxmQQ_3a0vU7J867krhJJAhWotpBEGNhOGr+b@mail.gmail.com>
	<4CD81EF1.2080000@behnel.de>
Message-ID: <AANLkTinW7Lmy4apTLvuJBk5G0pTTAFFVdiGtDfX-7DKs@mail.gmail.com>

On Mon, Nov 8, 2010 at 11:01 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Darren Dale, 08.11.2010 16:24:
>> On Mon, Nov 8, 2010 at 10:05 AM, Stefan Behnel wrote:
>>> Darren Dale, 08.11.2010 15:34:
>>>> Python-2.6 extended the definition of property to include getter,
>>>> setter, and deleter methods that can be used as decorators. This
>>>> allows a really nice pattern for declaring properties without
>>>> populating the namespace with a bunch of private methods:
>>>>
>>>> class Foo(object):
>>>>
>>>> ? ? ? _bar = None
>>>> ? ? ? @property
>>>> ? ? ? def bar(self):
>>>> ? ? ? ? ? return self._bar
>>>> ? ? ? @bar.setter
>>>> ? ? ? def bar(self, val):
>>>> ? ? ? ? ? self._bar = val
>>>>
>>>> What happens is @bar.setter makes a copy of bar, updates the copy to
>>>> use the decorated method as the setter, and returns the copy, which is
>>>> then bound to the name of the decorated function. The decorated
>>>> function has to be called bar, so that the updated copy is bound to
>>>> bar, replacing the original.
>>>>
>>>> Attempting to cythonize this example yields an error:
>>>>
>>>> $ python setup.py build_ext --inplace
>>>> running build_ext
>>>> cythoning test_property.pyx to test_property.c
>>>>
>>>> Error converting Pyrex file to C:
>>>> ------------------------------------------------------------
>>>> ...
>>>> ? ? ? _bar = None
>>>> ? ? ? @property
>>>> ? ? ? def bar(self):
>>>> ? ? ? ? ? return self._bar
>>>> ? ? ? @bar.setter
>>>> ? ? ? def bar(self, val):
>>>> ? ? ?^
>>>> ------------------------------------------------------------
>>>>
>>>> /Users/darren/temp/test/test_property.pyx:11:4: 'bar' already declared
>>>>
>>>> Would it be possible for Cython to support this pattern?
>>>
>>> The problem is not the property behaviour, it's the mapping from a Python
>>> function/method to a C function, which must have a unique name. Currently,
>>> functions are unique within one namespace, be it the module namespace or a
>>> class body namespace. They are actually looked up statically, not
>>> dynamically in definition order, so the fix is not as easy as adding a
>>> counted ID to the C-level name.
>>>
>>> The above might actually work if you change one of the method names, i.e.
>>> either the getter or setter name. If it works as you say, the setter name
>>> shouldn't have any impact on the result.
>>
>> Sorry, I should have emphasized the point about the importance of the
>> setter name. It *must* be the same as the original property.
>>
>> class Foo(object):
>>
>> ? ? ?_bar = None
>> ? ? ?@property
>> ? ? ?def bar(self):
>> ? ? ? ? ?return self._bar
>> ? ? ?@bar.setter
>> ? ? ?def bar_(self, val):
>> ? ? ? ? ?self._bar = val
>>
>> That produces a class with two properties: bar and bar_. Foo.bar is a
>> property with a getter but no setter. Foo.bar_ is a property with a
>> getter and a setter.
>
> Renaming the getter works for me, though:
>
> ? ? class Foo(object):
> ? ? ? ? _bar = None
>
> ? ? ? ? @property
> ? ? ? ? def bar_(self):
> ? ? ? ? ? ? return self._bar
>
> ? ? ? ? @bar_.setter
> ? ? ? ? def bar(self, val):
> ? ? ? ? ? ? self._bar = val
>
> ? ? ? ? del bar_
>
> ? ? f = Foo()
> ? ? print f.bar
> ? ? f.bar = 5
> ? ? print f.bar
>
> ? ? try:
> ? ? ? ? Foo.bar_
> ? ? except AttributeError:
> ? ? ? ? pass
> ? ? except:
> ? ? ? ? print "hoppsa!"
>
> I admit that it's ugly to do a "del bar_" at the end, but it's not more
> than a work-around after all...

Oh, that's what you meant. I suppose that is a reasonable workaround.
Is issue 489 solvable?

From robertwb at math.washington.edu  Mon Nov  8 19:38:08 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 8 Nov 2010 10:38:08 -0800
Subject: [Cython] >=python-2.6 property pattern not supported
In-Reply-To: <AANLkTinW7Lmy4apTLvuJBk5G0pTTAFFVdiGtDfX-7DKs@mail.gmail.com>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>
	<4CD811B2.1010108@behnel.de>
	<AANLkTi=MxmQQ_3a0vU7J867krhJJAhWotpBEGNhOGr+b@mail.gmail.com>
	<4CD81EF1.2080000@behnel.de>
	<AANLkTinW7Lmy4apTLvuJBk5G0pTTAFFVdiGtDfX-7DKs@mail.gmail.com>
Message-ID: <AANLkTimqt4V0-8SqZOann2dy2CiEswPQpV3efecBZ=Kk@mail.gmail.com>

On Mon, Nov 8, 2010 at 10:13 AM, Darren Dale <dsdale24 at gmail.com> wrote:
> On Mon, Nov 8, 2010 at 11:01 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Darren Dale, 08.11.2010 16:24:
>>> On Mon, Nov 8, 2010 at 10:05 AM, Stefan Behnel wrote:
>>>> Darren Dale, 08.11.2010 15:34:
>>>>> Python-2.6 extended the definition of property to include getter,
>>>>> setter, and deleter methods that can be used as decorators. This
>>>>> allows a really nice pattern for declaring properties without
>>>>> populating the namespace with a bunch of private methods:
>>>>>
>>>>> class Foo(object):
>>>>>
>>>>> ? ? ? _bar = None
>>>>> ? ? ? @property
>>>>> ? ? ? def bar(self):
>>>>> ? ? ? ? ? return self._bar
>>>>> ? ? ? @bar.setter
>>>>> ? ? ? def bar(self, val):
>>>>> ? ? ? ? ? self._bar = val
>>>>>
>>>>> What happens is @bar.setter makes a copy of bar, updates the copy to
>>>>> use the decorated method as the setter, and returns the copy, which is
>>>>> then bound to the name of the decorated function. The decorated
>>>>> function has to be called bar, so that the updated copy is bound to
>>>>> bar, replacing the original.
>>>>>
>>>>> Attempting to cythonize this example yields an error:
>>>>>
>>>>> $ python setup.py build_ext --inplace
>>>>> running build_ext
>>>>> cythoning test_property.pyx to test_property.c
>>>>>
>>>>> Error converting Pyrex file to C:
>>>>> ------------------------------------------------------------
>>>>> ...
>>>>> ? ? ? _bar = None
>>>>> ? ? ? @property
>>>>> ? ? ? def bar(self):
>>>>> ? ? ? ? ? return self._bar
>>>>> ? ? ? @bar.setter
>>>>> ? ? ? def bar(self, val):
>>>>> ? ? ?^
>>>>> ------------------------------------------------------------
>>>>>
>>>>> /Users/darren/temp/test/test_property.pyx:11:4: 'bar' already declared
>>>>>
>>>>> Would it be possible for Cython to support this pattern?
>>>>
>>>> The problem is not the property behaviour, it's the mapping from a Python
>>>> function/method to a C function, which must have a unique name. Currently,
>>>> functions are unique within one namespace, be it the module namespace or a
>>>> class body namespace. They are actually looked up statically, not
>>>> dynamically in definition order, so the fix is not as easy as adding a
>>>> counted ID to the C-level name.
>>>>
>>>> The above might actually work if you change one of the method names, i.e.
>>>> either the getter or setter name. If it works as you say, the setter name
>>>> shouldn't have any impact on the result.
>>>
>>> Sorry, I should have emphasized the point about the importance of the
>>> setter name. It *must* be the same as the original property.
>>>
>>> class Foo(object):
>>>
>>> ? ? ?_bar = None
>>> ? ? ?@property
>>> ? ? ?def bar(self):
>>> ? ? ? ? ?return self._bar
>>> ? ? ?@bar.setter
>>> ? ? ?def bar_(self, val):
>>> ? ? ? ? ?self._bar = val
>>>
>>> That produces a class with two properties: bar and bar_. Foo.bar is a
>>> property with a getter but no setter. Foo.bar_ is a property with a
>>> getter and a setter.
>>
>> Renaming the getter works for me, though:
>>
>> ? ? class Foo(object):
>> ? ? ? ? _bar = None
>>
>> ? ? ? ? @property
>> ? ? ? ? def bar_(self):
>> ? ? ? ? ? ? return self._bar
>>
>> ? ? ? ? @bar_.setter
>> ? ? ? ? def bar(self, val):
>> ? ? ? ? ? ? self._bar = val
>>
>> ? ? ? ? del bar_
>>
>> ? ? f = Foo()
>> ? ? print f.bar
>> ? ? f.bar = 5
>> ? ? print f.bar
>>
>> ? ? try:
>> ? ? ? ? Foo.bar_
>> ? ? except AttributeError:
>> ? ? ? ? pass
>> ? ? except:
>> ? ? ? ? print "hoppsa!"
>>
>> I admit that it's ugly to do a "del bar_" at the end, but it's not more
>> than a work-around after all...
>
> Oh, that's what you meant. I suppose that is a reasonable workaround.
> Is issue 489 solvable?

Yes, but it won't be a quick fix.

- Robert

From matej at laitl.cz  Tue Nov  9 01:06:27 2010
From: matej at laitl.cz (=?UTF-8?B?TWF0xJtqIExhaXRs?=)
Date: Tue, 9 Nov 2010 01:06:27 +0100
Subject: [Cython] Cython 0.13.1 - bugfix for ticket 583 (Wrong code
	generated)
In-Reply-To: <4CD7D82B.80607@behnel.de>
References: <AANLkTinaDcV0Vu0E4NPB2xsZdCHj3VAzaiPSg=NkLOXT@mail.gmail.com>
	<4CD7AEC5.7090805@behnel.de> <4CD7D82B.80607@behnel.de>
Message-ID: <AANLkTinU2gRtvYw7xfkU1Fab=G-xPCSgvX+__72nwagu@mail.gmail.com>

On Mon, Nov 8, 2010 at 11:59, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Stefan Behnel, 08.11.2010 09:03:
>> Mat?j Laitl, 08.11.2010 01:13:
>>> Sadly, I'm clueless how to fix it completely. It would be great if a
>>> fix for this bug could make it into 0.13.1 as my code [1] depends on
>>> it.
>>
>> Your patch is a good hint, I'm looking into it.
>
> What you do in your patch would provide a quick fix.
>
> Personally, I think the right way to really fix this would be to finally
> refactor def functions into cdef function wrappers. That way, overriding
> pure def functions in .pxd files would simply enable the underlying cdef
> function as being callable, instead of needing to build a new one from
> scratch. And we could even override def functions as pure cdef functions by
> simply dropping the wrapper entirely.

That would be great in long-term view, but I see you've commited more
elaborate fix for the bug - that's equally good for now. Thanks for
that, Stefan! I'll have an opportunity to test your patch on
moderately complicated code tomorrow, I'll report back if something
goes wrong.

      Mat?j

From stefan_ml at behnel.de  Tue Nov  9 08:30:29 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 09 Nov 2010 08:30:29 +0100
Subject: [Cython] Cython 0.13.1 - bugfix for ticket 583 (Wrong code
	generated)
In-Reply-To: <AANLkTinU2gRtvYw7xfkU1Fab=G-xPCSgvX+__72nwagu@mail.gmail.com>
References: <AANLkTinaDcV0Vu0E4NPB2xsZdCHj3VAzaiPSg=NkLOXT@mail.gmail.com>	<4CD7AEC5.7090805@behnel.de>
	<4CD7D82B.80607@behnel.de>
	<AANLkTinU2gRtvYw7xfkU1Fab=G-xPCSgvX+__72nwagu@mail.gmail.com>
Message-ID: <4CD8F895.2010005@behnel.de>

Mat?j Laitl, 09.11.2010 01:06:
> On Mon, Nov 8, 2010 at 11:59, Stefan Behnel wrote:
>> Stefan Behnel, 08.11.2010 09:03:
>>> Mat?j Laitl, 08.11.2010 01:13:
>>>> Sadly, I'm clueless how to fix it completely. It would be great if a
>>>> fix for this bug could make it into 0.13.1 as my code [1] depends on
>>>> it.
>>>
>>> Your patch is a good hint, I'm looking into it.
>>
>> What you do in your patch would provide a quick fix.
>>
>> Personally, I think the right way to really fix this would be to finally
>> refactor def functions into cdef function wrappers. That way, overriding
>> pure def functions in .pxd files would simply enable the underlying cdef
>> function as being callable, instead of needing to build a new one from
>> scratch. And we could even override def functions as pure cdef functions by
>> simply dropping the wrapper entirely.
>
> That would be great in long-term view, but I see you've commited more
> elaborate fix for the bug - that's equally good for now. Thanks for
> that, Stefan! I'll have an opportunity to test your patch on
> moderately complicated code tomorrow, I'll report back if something
> goes wrong.

You should report back even if it works for you. It's more fun to hear 
about things that work than about things that don't work.

Stefan

From matej at laitl.cz  Tue Nov  9 14:17:40 2010
From: matej at laitl.cz (=?UTF-8?B?TWF0xJtqIExhaXRs?=)
Date: Tue, 9 Nov 2010 14:17:40 +0100
Subject: [Cython] Cython 0.13.1 - bugfix for ticket 583 (Wrong code
	generated)
In-Reply-To: <4CD8F895.2010005@behnel.de>
References: <AANLkTinaDcV0Vu0E4NPB2xsZdCHj3VAzaiPSg=NkLOXT@mail.gmail.com>
	<4CD7AEC5.7090805@behnel.de> <4CD7D82B.80607@behnel.de>
	<AANLkTinU2gRtvYw7xfkU1Fab=G-xPCSgvX+__72nwagu@mail.gmail.com>
	<4CD8F895.2010005@behnel.de>
Message-ID: <AANLkTi=npyr-738QPuaYxy-LrRHt+gO+gvDRTJxpYkQf@mail.gmail.com>

On Tue, Nov 9, 2010 at 08:30, Stefan Behnel <stefan_ml at behnel.de> wrote:
> You should report back even if it works for you. It's more fun to hear
> about things that work than about things that don't work.

In that case I won't disappoint you - the fix works perfectly even
when -O2 is passed to gcc - something that failed badly with the quick
fix. :-)

       Mat?j

From Paul.Connell at disney.com  Tue Nov  9 16:59:40 2010
From: Paul.Connell at disney.com (Connell, Paul)
Date: Tue, 9 Nov 2010 15:59:40 +0000
Subject: [Cython] Cython gdb on Windows
Message-ID: <DA536BC5C16ADD4D850A9522DB86E01E51C1F542EF@SM-GBLO-VXM01A.emea.wdpr.disney.com>

Hi

I have been playing with Cython gdb on Windows and finding it far from straightforward, as is often the case with *nix centric software. So here are my experiences, and I wonder if anyone else has had better or at least different experiences.

The only gdb I was able to find with Python support is the one with Mingw/MSys. I had tried to build my own, but with no success as far as python support is concerned. I may revisit this at some point.

The first problem I encountered upon running cygdb.py was, predictably, a posix-vs-windows path issue. I was able to kludge around this by hacking cygdb.py:

import posixpath
#...
pattern = posixpath.join(path_to_debug_info, 
                               'cython_debug/cython_debug_info_*')
        debug_files = glob.glob(pattern)
        new_debug_files=[]
        for df in debug_files:
            new_debug_files.append(string.replace(df, "\\", "/"))
        debug_files = new_debug_files
        if not debug_files:
            usage()
            sys.exit('No debug files were found in %s. Aborting.' % (
                    os.path.abspath(path_to_debug_info)))


otherwise cygdb tries to load e.g. .cython_debugcython_debug_info_mymodule as all the backslashes have vanished.

The next problem was with the xml parser. I found I had to manually add

from xml.parsers import expat


cygdb now goes through the motions of working, although I am against another brick wall now in that my modules and python installation are all 64 bit, whereas mingw/msys is 32 bit. This isn't cygdb's fault though...

I hope some of this is useful. I've not yet got a useable cygdb, but hopefully am getting there.

Paul

From markflorisson88 at gmail.com  Tue Nov  9 17:15:48 2010
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 9 Nov 2010 17:15:48 +0100
Subject: [Cython] Cython gdb on Windows
In-Reply-To: <DA536BC5C16ADD4D850A9522DB86E01E51C1F542EF@SM-GBLO-VXM01A.emea.wdpr.disney.com>
References: <DA536BC5C16ADD4D850A9522DB86E01E51C1F542EF@SM-GBLO-VXM01A.emea.wdpr.disney.com>
Message-ID: <AANLkTikdVJPoa0UbzKBqLMSy2LQm6kS2kq9PR+Hic=c0@mail.gmail.com>

Hey Paul,

On 9 November 2010 16:59, Connell, Paul <Paul.Connell at disney.com> wrote:
> Hi
>
> I have been playing with Cython gdb on Windows and finding it far from straightforward, as is often the case with *nix centric software. So here are my experiences, and I wonder if anyone else has had better or at least different experiences.
>
> The only gdb I was able to find with Python support is the one with Mingw/MSys. I had tried to build my own, but with no success as far as python support is concerned. I may revisit this at some point.

I don't have any experience with building gdb on windows, so I can't
really help in that regard. What I can say is, if your program is
compatible with Cygwin you could try to use gdb from there.

> The first problem I encountered upon running cygdb.py was, predictably, a posix-vs-windows path issue. I was able to kludge around this by hacking cygdb.py:
>
> import posixpath
> #...
> pattern = posixpath.join(path_to_debug_info,
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 'cython_debug/cython_debug_info_*')
> ? ? ? ?debug_files = glob.glob(pattern)
> ? ? ? ?new_debug_files=[]
> ? ? ? ?for df in debug_files:
> ? ? ? ? ? ?new_debug_files.append(string.replace(df, "\\", "/"))
> ? ? ? ?debug_files = new_debug_files
> ? ? ? ?if not debug_files:
> ? ? ? ? ? ?usage()
> ? ? ? ? ? ?sys.exit('No debug files were found in %s. Aborting.' % (
> ? ? ? ? ? ? ? ? ? ?os.path.abspath(path_to_debug_info)))
>
>
> otherwise cygdb tries to load e.g. .cython_debugcython_debug_info_mymodule as all the backslashes have vanished.

Regarding the path issue, I guess the code should have read
'os.path.join(path_to_debug_info, "cython_debug",
"cython_debug_info_*")'. I will try to fix and test it later today or
early tomorrow.

> The next problem was with the xml parser. I found I had to manually add
>
> from xml.parsers import expat
>

Why did you have to do that? I don't believe xml.parsers.expat
provides an ElementTree interface, so this would break all the code.
Cygdb needs an ElementTree implementation, of which there are several,
and it tries all implementations that are out there. Python2.5 comes
with xml.etree.ElementTree in the stdlib, and cygdb actually requires
python 2.5 or higher at this moment so it should not be an issue.

> cygdb now goes through the motions of working, although I am against another brick wall now in that my modules and python installation are all 64 bit, whereas mingw/msys is 32 bit. This isn't cygdb's fault though...
>
> I hope some of this is useful. I've not yet got a useable cygdb, but hopefully am getting there.

Yes, this is kind of unfortunate that windows is always a problem.
However, note that the cygdb tool is by no means needed to use the
Cython debugger. You can also start gdb manually, and issue the
command 'python from Cython.Debugger import libcython' and then import
the debug information with the 'cy import' command. Of course, cygdb
should and will be fixed, but it basically requires a sane
environment: gdb 7.2, python 2.5 and a compatible C compiler.

> Paul
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

Kind regards,

Mark

From stefan at sun.ac.za  Tue Nov  9 18:37:39 2010
From: stefan at sun.ac.za (=?ISO-8859-1?Q?St=E9fan_van_der_Walt?=)
Date: Tue, 9 Nov 2010 19:37:39 +0200
Subject: [Cython] Pure Python definition of ndarray / object array
Message-ID: <AANLkTinzmeP3ao_OBMnz2Wrz=NA6bn8Lfhd16kEEH7uA@mail.gmail.com>

Hi all,

A while ago there was a question on how to define ndarrays in a pure
Python annotated module.  I thought it should be pretty easy to get
this working by simply defining a new type in Shadow.py, and using
syntax such as

@python.locals(arr=cython.ndarray(dtype=float, ndim=2))

but now I have no idea what to return from cython.ndarray.__call__,
since the ndarray type is normally cimported from numpy.pxd.  Any
tips?

Regards
St?fan

From Paul.Connell at disney.com  Wed Nov 10 10:39:51 2010
From: Paul.Connell at disney.com (Connell, Paul)
Date: Wed, 10 Nov 2010 09:39:51 +0000
Subject: [Cython] Cython gdb on Windows
In-Reply-To: <AANLkTikdVJPoa0UbzKBqLMSy2LQm6kS2kq9PR+Hic=c0@mail.gmail.com>
References: <DA536BC5C16ADD4D850A9522DB86E01E51C1F542EF@SM-GBLO-VXM01A.emea.wdpr.disney.com>
	<AANLkTikdVJPoa0UbzKBqLMSy2LQm6kS2kq9PR+Hic=c0@mail.gmail.com>
Message-ID: <DA536BC5C16ADD4D850A9522DB86E01E51C1F54401@SM-GBLO-VXM01A.emea.wdpr.disney.com>



-----Original Message-----
From: cython-dev-bounces at codespeak.net [mailto:cython-dev-bounces at codespeak.net] On Behalf Of mark florisson
Sent: 09 November 2010 16:16
To: cython-dev at codespeak.net
Subject: Re: [Cython] Cython gdb on Windows

> The next problem was with the xml parser. I found I had to manually 
> add
>
> from xml.parsers import expat
>

> Why did you have to do that? I don't believe xml.parsers.expat provides an ElementTree interface, so this would break all the code.
> Cygdb needs an ElementTree implementation, of which there are several, and it tries all implementations that are out there. > Python2.5 comes with xml.etree.ElementTree in the stdlib, and cygdb actually requires python 2.5 or higher at this moment so it >should not be an issue.

Python complained that expat was missing. So I imported it. This is with Python 2.7 by the way.


> cygdb now goes through the motions of working, although I am against another brick wall now in that my modules and python installation are all 64 bit, whereas mingw/msys is 32 bit. This isn't cygdb's fault though...
>
> I hope some of this is useful. I've not yet got a useable cygdb, but hopefully am getting there.

>Yes, this is kind of unfortunate that windows is always a problem.

It certainly is...


Cheers,
Paul

From markflorisson88 at gmail.com  Wed Nov 10 10:47:38 2010
From: markflorisson88 at gmail.com (mark florisson)
Date: Wed, 10 Nov 2010 10:47:38 +0100
Subject: [Cython] Cython gdb on Windows
In-Reply-To: <DA536BC5C16ADD4D850A9522DB86E01E51C1F54401@SM-GBLO-VXM01A.emea.wdpr.disney.com>
References: <DA536BC5C16ADD4D850A9522DB86E01E51C1F542EF@SM-GBLO-VXM01A.emea.wdpr.disney.com>
	<AANLkTikdVJPoa0UbzKBqLMSy2LQm6kS2kq9PR+Hic=c0@mail.gmail.com>
	<DA536BC5C16ADD4D850A9522DB86E01E51C1F54401@SM-GBLO-VXM01A.emea.wdpr.disney.com>
Message-ID: <AANLkTikp_DsCJm83t4mzsSCaCgerVAv0BD7OhXid1XtD@mail.gmail.com>

On 10 November 2010 10:39, Connell, Paul <Paul.Connell at disney.com> wrote:
>
>
> -----Original Message-----
> From: cython-dev-bounces at codespeak.net [mailto:cython-dev-bounces at codespeak.net] On Behalf Of mark florisson
> Sent: 09 November 2010 16:16
> To: cython-dev at codespeak.net
> Subject: Re: [Cython] Cython gdb on Windows
>
>> The next problem was with the xml parser. I found I had to manually
>> add
>>
>> from xml.parsers import expat
>>
>
>> Why did you have to do that? I don't believe xml.parsers.expat provides an ElementTree interface, so this would break all the code.
>> Cygdb needs an ElementTree implementation, of which there are several, and it tries all implementations that are out there. > Python2.5 comes with xml.etree.ElementTree in the stdlib, and cygdb actually requires python 2.5 or higher at this moment so it >should not be an issue.
>
> Python complained that expat was missing. So I imported it. This is with Python 2.7 by the way.
>

Weird, this shouldn't happen. Could you paste a full traceback so I
know which implementation it tried to import and what happened? In any
case, it sounds like something was not installed correctly.

>> cygdb now goes through the motions of working, although I am against another brick wall now in that my modules and python installation are all 64 bit, whereas mingw/msys is 32 bit. This isn't cygdb's fault though...
>>
>> I hope some of this is useful. I've not yet got a useable cygdb, but hopefully am getting there.
>
>>Yes, this is kind of unfortunate that windows is always a problem.
>
> It certainly is...
>
>
> Cheers,
> Paul
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From vinjvinj at gmail.com  Thu Nov 11 00:48:26 2010
From: vinjvinj at gmail.com (Vineet Jain)
Date: Wed, 10 Nov 2010 18:48:26 -0500
Subject: [Cython] How do I declare numpy arrays with multiple dimensions and
	different types.
Message-ID: <AANLkTi=j-+BevpePJZsgL4bX+EyJ7xDT4CETzr30D7h_@mail.gmail.com>

This I undestand:

cdef test(numpy.ndarray[numpy.int32_t, ndim=2] a):

However what if a is of type (numpy.int32, numpy.float64), how would I
declare it using numpy.ndaray?

Thanks,

VIneet

From stefan_ml at behnel.de  Thu Nov 11 06:19:31 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 11 Nov 2010 06:19:31 +0100
Subject: [Cython] How do I declare numpy arrays with multiple dimensions
 and	different types.
In-Reply-To: <AANLkTi=j-+BevpePJZsgL4bX+EyJ7xDT4CETzr30D7h_@mail.gmail.com>
References: <AANLkTi=j-+BevpePJZsgL4bX+EyJ7xDT4CETzr30D7h_@mail.gmail.com>
Message-ID: <4CDB7CE3.9090605@behnel.de>

Vineet Jain, 11.11.2010 00:48:
> This I undestand:
>
> cdef test(numpy.ndarray[numpy.int32_t, ndim=2] a):
>
> However what if a is of type (numpy.int32, numpy.float64), how would I
> declare it using numpy.ndaray?

Hi,

note that this question is appropriate for the cython-users mailing list, 
not for the cython-dev (as in "core development") mailing list.

Stefan

From dagss at student.matnat.uio.no  Thu Nov 11 07:10:20 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 11 Nov 2010 07:10:20 +0100
Subject: [Cython] How do I declare numpy arrays with multiple dimensions
 and	different types.
In-Reply-To: <AANLkTi=j-+BevpePJZsgL4bX+EyJ7xDT4CETzr30D7h_@mail.gmail.com>
References: <AANLkTi=j-+BevpePJZsgL4bX+EyJ7xDT4CETzr30D7h_@mail.gmail.com>
Message-ID: <4CDB88CC.7060504@student.matnat.uio.no>

On 11/11/2010 12:48 AM, Vineet Jain wrote:
> This I undestand:
>
> cdef test(numpy.ndarray[numpy.int32_t, ndim=2] a):
>
> However what if a is of type (numpy.int32, numpy.float64), how would I
> declare it using numpy.ndaray?
>    

Use a struct as the data type. Look in proceedings of SciPy 2009 conference.

Dag Sverre

From qed777 at gmail.com  Fri Nov 12 01:10:55 2010
From: qed777 at gmail.com (Mitesh Patel)
Date: Thu, 11 Nov 2010 18:10:55 -0600
Subject: [Cython] Runaway processes on sage.math?
Message-ID: <4CDC860F.2050102@gmail.com>

I noticed two resource-intensive processes on sage.math.washington.edu:

top - 16:08:27 up 87 days,  3:21, 27 users,  load average: 11.79, 10.26,
13.84
Tasks: 906 total,  12 running, 894 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.5%us, 12.0%sy, 31.7%ni, 52.2%id,  3.4%wa,  0.1%hi,  0.2%si,
0.0%st
Mem:  132294748k total, 130767708k used,  1527040k free,   179412k buffers
Swap: 71681988k total, 42344884k used, 29337104k free,  3898360k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

20313 scoder    35  15 73.3g  56g 4416 R  100 44.9  49:17.86 python3
24407 scoder    35  15 73.4g  56g 4856 R  100 45.1  49:49.77 python3
[...]

Are these runaway Hudson jobs that should be killed?


From robertwb at math.washington.edu  Fri Nov 12 05:11:57 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 11 Nov 2010 20:11:57 -0800
Subject: [Cython] Runaway processes on sage.math?
In-Reply-To: <4CDC860F.2050102@gmail.com>
References: <4CDC860F.2050102@gmail.com>
Message-ID: <AANLkTimcGKde5Ym2Rp86S-RO9mBZoEcRCOuJEX5G87RJ@mail.gmail.com>

Yes. I've killed them.

On Thu, Nov 11, 2010 at 4:10 PM, Mitesh Patel <qed777 at gmail.com> wrote:
> I noticed two resource-intensive processes on sage.math.washington.edu:
>
> top - 16:08:27 up 87 days, ?3:21, 27 users, ?load average: 11.79, 10.26,
> 13.84
> Tasks: 906 total, ?12 running, 894 sleeping, ? 0 stopped, ? 0 zombie
> Cpu(s): ?0.5%us, 12.0%sy, 31.7%ni, 52.2%id, ?3.4%wa, ?0.1%hi, ?0.2%si,
> 0.0%st
> Mem: ?132294748k total, 130767708k used, ?1527040k free, ? 179412k buffers
> Swap: 71681988k total, 42344884k used, 29337104k free, ?3898360k cached
>
> ?PID USER ? ? ?PR ?NI ?VIRT ?RES ?SHR S %CPU %MEM ? ?TIME+ ?COMMAND
>
> 20313 scoder ? ?35 ?15 73.3g ?56g 4416 R ?100 44.9 ?49:17.86 python3
> 24407 scoder ? ?35 ?15 73.4g ?56g 4856 R ?100 45.1 ?49:49.77 python3
> [...]
>
> Are these runaway Hudson jobs that should be killed?
>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From stefan_ml at behnel.de  Fri Nov 12 07:07:05 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 12 Nov 2010 07:07:05 +0100
Subject: [Cython] Runaway processes on sage.math?
In-Reply-To: <AANLkTimcGKde5Ym2Rp86S-RO9mBZoEcRCOuJEX5G87RJ@mail.gmail.com>
References: <4CDC860F.2050102@gmail.com>
	<AANLkTimcGKde5Ym2Rp86S-RO9mBZoEcRCOuJEX5G87RJ@mail.gmail.com>
Message-ID: <4CDCD989.2020902@behnel.de>

Robert Bradshaw, 12.11.2010 05:11:
> On Thu, Nov 11, 2010 at 4:10 PM, Mitesh Patel wrote:
>> I noticed two resource-intensive processes on sage.math.washington.edu:
>>
>> top - 16:08:27 up 87 days,  3:21, 27 users,  load average: 11.79, 10.26,
>> 13.84
>> Tasks: 906 total,  12 running, 894 sleeping,   0 stopped,   0 zombie
>> Cpu(s):  0.5%us, 12.0%sy, 31.7%ni, 52.2%id,  3.4%wa,  0.1%hi,  0.2%si,
>> 0.0%st
>> Mem:  132294748k total, 130767708k used,  1527040k free,   179412k buffers
>> Swap: 71681988k total, 42344884k used, 29337104k free,  3898360k cached
>>
>>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>
>> 20313 scoder    35  15 73.3g  56g 4416 R  100 44.9  49:17.86 python3
>> 24407 scoder    35  15 73.4g  56g 4856 R  100 45.1  49:49.77 python3
>> [...]
>>
>> Are these runaway Hudson jobs that should be killed?
>
> Yes. I've killed them.

Seems to be reproducible. I'll take a look.

Stefan

From vitja.makarov at gmail.com  Fri Nov 12 08:06:42 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Fri, 12 Nov 2010 10:06:42 +0300
Subject: [Cython] >=python-2.6 property pattern not supported
In-Reply-To: <4CD811B2.1010108@behnel.de>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>
	<4CD811B2.1010108@behnel.de>
Message-ID: <AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>

2010/11/8 Stefan Behnel <stefan_ml at behnel.de>:
> Darren Dale, 08.11.2010 15:34:
>> Python-2.6 extended the definition of property to include getter,
>> setter, and deleter methods that can be used as decorators. This
>> allows a really nice pattern for declaring properties without
>> populating the namespace with a bunch of private methods:
>>
>> class Foo(object):
>>
>> ? ? ?_bar = None
>> ? ? ?@property
>> ? ? ?def bar(self):
>> ? ? ? ? ?return self._bar
>> ? ? ?@bar.setter
>> ? ? ?def bar(self, val):
>> ? ? ? ? ?self._bar = val
>>
>> What happens is @bar.setter makes a copy of bar, updates the copy to
>> use the decorated method as the setter, and returns the copy, which is
>> then bound to the name of the decorated function. The decorated
>> function has to be called bar, so that the updated copy is bound to
>> bar, replacing the original.
>>
>> Attempting to cythonize this example yields an error:
>>
>> $ python setup.py build_ext --inplace
>> running build_ext
>> cythoning test_property.pyx to test_property.c
>>
>> Error converting Pyrex file to C:
>> ------------------------------------------------------------
>> ...
>> ? ? ?_bar = None
>> ? ? ?@property
>> ? ? ?def bar(self):
>> ? ? ? ? ?return self._bar
>> ? ? ?@bar.setter
>> ? ? ?def bar(self, val):
>> ? ? ^
>> ------------------------------------------------------------
>>
>> /Users/darren/temp/test/test_property.pyx:11:4: 'bar' already declared
>>
>> Would it be possible for Cython to support this pattern?
>
> The problem is not the property behaviour, it's the mapping from a Python
> function/method to a C function, which must have a unique name. Currently,
> functions are unique within one namespace, be it the module namespace or a
> class body namespace. They are actually looked up statically, not
> dynamically in definition order, so the fix is not as easy as adding a
> counted ID to the C-level name.
>

Is not it safe to add ID to C-function in pure python class scope?
I see problem in module scope when it turns functions into list of
methoddef that are passed to PyModule_Init.

vitja.

From stefan_ml at behnel.de  Fri Nov 12 08:12:19 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 12 Nov 2010 08:12:19 +0100
Subject: [Cython] >=python-2.6 property pattern not supported
In-Reply-To: <AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>	<4CD811B2.1010108@behnel.de>
	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>
Message-ID: <4CDCE8D3.70002@behnel.de>

Vitja Makarov, 12.11.2010 08:06:
> 2010/11/8 Stefan Behnel:
>> The problem is not the property behaviour, it's the mapping from a Python
>> function/method to a C function, which must have a unique name. Currently,
>> functions are unique within one namespace, be it the module namespace or a
>> class body namespace. They are actually looked up statically, not
>> dynamically in definition order, so the fix is not as easy as adding a
>> counted ID to the C-level name.
>
> Is not it safe to add ID to C-function in pure python class scope?
> I see problem in module scope when it turns functions into list of
> methoddef that are passed to PyModule_Init.

As I said, it's not *enough* to fix the name.

Stefan

From vitja.makarov at gmail.com  Fri Nov 12 08:52:02 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Fri, 12 Nov 2010 10:52:02 +0300
Subject: [Cython] >=python-2.6 property pattern not supported
In-Reply-To: <4CDCE8D3.70002@behnel.de>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>
	<4CD811B2.1010108@behnel.de>
	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>
	<4CDCE8D3.70002@behnel.de>
Message-ID: <AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>

2010/11/12 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 12.11.2010 08:06:
>> 2010/11/8 Stefan Behnel:
>>> The problem is not the property behaviour, it's the mapping from a Python
>>> function/method to a C function, which must have a unique name. Currently,
>>> functions are unique within one namespace, be it the module namespace or a
>>> class body namespace. They are actually looked up statically, not
>>> dynamically in definition order, so the fix is not as easy as adding a
>>> counted ID to the C-level name.
>>
>> Is not it safe to add ID to C-function in pure python class scope?
>> I see problem in module scope when it turns functions into list of
>> methoddef that are passed to PyModule_Init.
>
> As I said, it's not *enough* to fix the name.
>

Can please you explain the problem? What more should be done?

I see problem in decorators here:
clsas Foo(object):
    @property
    def foo(self)
    ....
    @foo.setter
    def foo(self, val):
    ....

Setter Cython first sets foo item, then get it back and decorate with
setter attribute:

dict['foo'] = Foo_foo_method_000
dict['foo'] = property(dict['foo'])
dict['foo'] = Foo_foo_method_001
dict['foo'] = dict['foo'].setter(dict['foo'])

Am I right here? I see problem in decorators, attribute assignment here.

From stefan_ml at behnel.de  Fri Nov 12 09:02:17 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 12 Nov 2010 09:02:17 +0100
Subject: [Cython] >=python-2.6 property pattern not supported
In-Reply-To: <AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>	<4CD811B2.1010108@behnel.de>	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>	<4CDCE8D3.70002@behnel.de>
	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>
Message-ID: <4CDCF489.3000108@behnel.de>

Vitja Makarov, 12.11.2010 08:52:
> 2010/11/12 Stefan Behnel:
>> Vitja Makarov, 12.11.2010 08:06:
>>> 2010/11/8 Stefan Behnel:
>>>> The problem is not the property behaviour, it's the mapping from a Python
>>>> function/method to a C function, which must have a unique name. Currently,
>>>> functions are unique within one namespace, be it the module namespace or a
>>>> class body namespace. They are actually looked up statically, not
>>>> dynamically in definition order, so the fix is not as easy as adding a
>>>> counted ID to the C-level name.
>>>
>>> Is not it safe to add ID to C-function in pure python class scope?
>>> I see problem in module scope when it turns functions into list of
>>> methoddef that are passed to PyModule_Init.
>>
>> As I said, it's not *enough* to fix the name.
>
> Can please you explain the problem? What more should be done?
>
> I see problem in decorators here:
> clsas Foo(object):
>      @property
>      def foo(self)
>      ....
>      @foo.setter
>      def foo(self, val):
>      ....
>
> Setter Cython first sets foo item, then get it back and decorate with
> setter attribute:
>
> dict['foo'] = Foo_foo_method_000
> dict['foo'] = property(dict['foo'])
> dict['foo'] = Foo_foo_method_001
> dict['foo'] = dict['foo'].setter(dict['foo'])
>
> Am I right here? I see problem in decorators, attribute assignment here.

This is not related to properties. What I meant by this:

"""
Currently, functions are unique within one namespace, be it the module 
namespace or a class body namespace. They are actually looked up 
statically, not dynamically in definition order
"""

is that you simply cannot currently define a function more than once within 
a namespace because it is looked up statically. So, when you redefine it, 
its Python name would get overwritten in the symbol table. So you couldn't 
refer to the previously defined function anymore from that point on, which 
means that you cannot even use it in the code that gets executed before the 
redefinition. The symbol table is not dynamically adaptive to the position 
in the code, it's just a plain table that is global to a namespace.

This is a compiler, not an interpreter. Things don't simply happen in the 
order they are executed in the code.

Does that explain it?

Stefan

From brett.calcott at gmail.com  Fri Nov 12 12:56:59 2010
From: brett.calcott at gmail.com (Brett Calcott)
Date: Fri, 12 Nov 2010 22:56:59 +1100
Subject: [Cython] [Numpy-discussion] Cython distutils
In-Reply-To: <AANLkTikR7Xi69uCzVP29FqC8LmY_FwhFx9iuSo6VPvND@mail.gmail.com>
References: <AANLkTikHurmxH9_htLmTXVNaQ8K3D7hYB6iMy-Th3VvA@mail.gmail.com>
	<AANLkTinjqhVQokMM_P8uJnoY3X0A=kZq1j5+bwg4vmFN@mail.gmail.com>
	<AANLkTinU+xQ7UYSuF6qOef_S060rny3+TgL64qsL33+4@mail.gmail.com>
	<AANLkTimFmW0BmkYj46ZZ7eG91uQVgU7eg6ky0WVQwOhH@mail.gmail.com>
	<AANLkTikR7Xi69uCzVP29FqC8LmY_FwhFx9iuSo6VPvND@mail.gmail.com>
Message-ID: <AANLkTimhL=uYH6AWNgD53=YCrfDDy+LaECbr7J8aj=kX@mail.gmail.com>

On 15 September 2010 22:58, David Cournapeau <cournape at gmail.com> wrote:
> The next version of bento (to be released soon), will have relatively
> decent support for cython, in particular chaining with other builder
> (so that you could say chain .pyx.in -> .pyx -> .c -> .o -> .so) will
> be possible without any monkey patch or other moronic operations.
>

Is this version out yet? I see there is a test for a pyx file, but it
doesn't work on my Mac (cloned from github today)

bentomaker: Error: bentomaker crashed (uncaught exception <class
'yaku.task_manager.NoHookException'>: No rule defined for extension
'.pyx').

Brett

From dagss at student.matnat.uio.no  Fri Nov 12 13:13:16 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 12 Nov 2010 13:13:16 +0100
Subject: [Cython] [Numpy-discussion] Cython distutils
In-Reply-To: <AANLkTimhL=uYH6AWNgD53=YCrfDDy+LaECbr7J8aj=kX@mail.gmail.com>
References: <AANLkTikHurmxH9_htLmTXVNaQ8K3D7hYB6iMy-Th3VvA@mail.gmail.com>	<AANLkTinjqhVQokMM_P8uJnoY3X0A=kZq1j5+bwg4vmFN@mail.gmail.com>	<AANLkTinU+xQ7UYSuF6qOef_S060rny3+TgL64qsL33+4@mail.gmail.com>	<AANLkTimFmW0BmkYj46ZZ7eG91uQVgU7eg6ky0WVQwOhH@mail.gmail.com>	<AANLkTikR7Xi69uCzVP29FqC8LmY_FwhFx9iuSo6VPvND@mail.gmail.com>
	<AANLkTimhL=uYH6AWNgD53=YCrfDDy+LaECbr7J8aj=kX@mail.gmail.com>
Message-ID: <4CDD2F5C.40409@student.matnat.uio.no>

On 11/12/2010 12:56 PM, Brett Calcott wrote:
> On 15 September 2010 22:58, David Cournapeau<cournape at gmail.com>  wrote:
>    
>> The next version of bento (to be released soon), will have relatively
>> decent support for cython, in particular chaining with other builder
>> (so that you could say chain .pyx.in ->  .pyx ->  .c ->  .o ->  .so) will
>> be possible without any monkey patch or other moronic operations.
>>
>>      
> Is this version out yet? I see there is a test for a pyx file, but it
> doesn't work on my Mac (cloned from github today)
>
> bentomaker: Error: bentomaker crashed (uncaught exception<class
> 'yaku.task_manager.NoHookException'>: No rule defined for extension
> '.pyx').
>    

You simply need to tell bento (or rather yaku) to use the "cython" tool. 
See here:

examples/hooks/customization/bscript

Dag Sverre

From brett.calcott at gmail.com  Fri Nov 12 13:18:08 2010
From: brett.calcott at gmail.com (Brett Calcott)
Date: Fri, 12 Nov 2010 23:18:08 +1100
Subject: [Cython] [Numpy-discussion] Cython distutils
In-Reply-To: <4CDD2F5C.40409@student.matnat.uio.no>
References: <AANLkTikHurmxH9_htLmTXVNaQ8K3D7hYB6iMy-Th3VvA@mail.gmail.com>
	<AANLkTinjqhVQokMM_P8uJnoY3X0A=kZq1j5+bwg4vmFN@mail.gmail.com>
	<AANLkTinU+xQ7UYSuF6qOef_S060rny3+TgL64qsL33+4@mail.gmail.com>
	<AANLkTimFmW0BmkYj46ZZ7eG91uQVgU7eg6ky0WVQwOhH@mail.gmail.com>
	<AANLkTikR7Xi69uCzVP29FqC8LmY_FwhFx9iuSo6VPvND@mail.gmail.com>
	<AANLkTimhL=uYH6AWNgD53=YCrfDDy+LaECbr7J8aj=kX@mail.gmail.com>
	<4CDD2F5C.40409@student.matnat.uio.no>
Message-ID: <AANLkTinZfjBcWMy-27sjWCxiRUsf7TCME1XgqBnyQyCk@mail.gmail.com>

On 12 November 2010 23:13, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> You simply need to tell bento (or rather yaku) to use the "cython" tool.
> See here:
>
> examples/hooks/customization/bscript

That is exactly what generated the error:

camponotus:customization(master) brett$ pwd
/Users/brett/Source/Bento/examples/hooks/customization
camponotus:customization(master) brett$ bentomaker configure
Looking for gcc... yes
Detecting distutils CC exec ... cc
Detecting CC type... gcc
camponotus:customization(master) brett$ bentomaker build
bentomaker: Error: bentomaker crashed (uncaught exception <class
'yaku.task_manager.NoHookException'>: No rule defined for extension
'.pyx').
Please report this on bento issue tracker:
    http://github.com/cournape/bento/issues
You can get a full traceback by setting BENTOMAKER_DEBUG=1
camponotus:customization(master) brett$

If it is working for others, then I must have something set up wrong.
I just git cloned and then used python setup.py install.

Brett

From cournape at gmail.com  Fri Nov 12 14:01:40 2010
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 12 Nov 2010 22:01:40 +0900
Subject: [Cython] [Numpy-discussion] Cython distutils
In-Reply-To: <AANLkTimhL=uYH6AWNgD53=YCrfDDy+LaECbr7J8aj=kX@mail.gmail.com>
References: <AANLkTikHurmxH9_htLmTXVNaQ8K3D7hYB6iMy-Th3VvA@mail.gmail.com>
	<AANLkTinjqhVQokMM_P8uJnoY3X0A=kZq1j5+bwg4vmFN@mail.gmail.com>
	<AANLkTinU+xQ7UYSuF6qOef_S060rny3+TgL64qsL33+4@mail.gmail.com>
	<AANLkTimFmW0BmkYj46ZZ7eG91uQVgU7eg6ky0WVQwOhH@mail.gmail.com>
	<AANLkTikR7Xi69uCzVP29FqC8LmY_FwhFx9iuSo6VPvND@mail.gmail.com>
	<AANLkTimhL=uYH6AWNgD53=YCrfDDy+LaECbr7J8aj=kX@mail.gmail.com>
Message-ID: <AANLkTi=S_J7bgDpy==6+fr=A9mCPpUtQuJpQSXJ-cAzw@mail.gmail.com>

On Fri, Nov 12, 2010 at 8:56 PM, Brett Calcott <brett.calcott at gmail.com> wrote:
> On 15 September 2010 22:58, David Cournapeau <cournape at gmail.com> wrote:
>> The next version of bento (to be released soon), will have relatively
>> decent support for cython, in particular chaining with other builder
>> (so that you could say chain .pyx.in -> .pyx -> .c -> .o -> .so) will
>> be possible without any monkey patch or other moronic operations.
>>
>
> Is this version out yet?

Yes

> I see there is a test for a pyx file, but it
> doesn't work on my Mac (cloned from github today)
>
> bentomaker: Error: bentomaker crashed (uncaught exception <class
> 'yaku.task_manager.NoHookException'>: No rule defined for extension
> '.pyx').

This is because you are using an unreleased version. Basically, bento
git repo include a copy of yaku repo, and I only make sure they are
synchronized for the releases. Otherwise, you must use yaku from its
own repository, something like:

git clone git://github.com/cournape/yaku.git
# add yaku package to your sys.path
export BENTO_UNBUNDLE_YAKU=1 # tell bento not to use its bundled yaku
# go into the directory with your bento example: it should work

arguably, this should be documented somewhere,

cheers,

David

From cournape at gmail.com  Fri Nov 12 14:12:19 2010
From: cournape at gmail.com (David Cournapeau)
Date: Fri, 12 Nov 2010 22:12:19 +0900
Subject: [Cython] [Numpy-discussion] Cython distutils
In-Reply-To: <AANLkTi=S_J7bgDpy==6+fr=A9mCPpUtQuJpQSXJ-cAzw@mail.gmail.com>
References: <AANLkTikHurmxH9_htLmTXVNaQ8K3D7hYB6iMy-Th3VvA@mail.gmail.com>
	<AANLkTinjqhVQokMM_P8uJnoY3X0A=kZq1j5+bwg4vmFN@mail.gmail.com>
	<AANLkTinU+xQ7UYSuF6qOef_S060rny3+TgL64qsL33+4@mail.gmail.com>
	<AANLkTimFmW0BmkYj46ZZ7eG91uQVgU7eg6ky0WVQwOhH@mail.gmail.com>
	<AANLkTikR7Xi69uCzVP29FqC8LmY_FwhFx9iuSo6VPvND@mail.gmail.com>
	<AANLkTimhL=uYH6AWNgD53=YCrfDDy+LaECbr7J8aj=kX@mail.gmail.com>
	<AANLkTi=S_J7bgDpy==6+fr=A9mCPpUtQuJpQSXJ-cAzw@mail.gmail.com>
Message-ID: <AANLkTinCDEdKn10LJUVLyJU-atOR3JGVKGkCzLjYC7tF@mail.gmail.com>

On Fri, Nov 12, 2010 at 10:01 PM, David Cournapeau <cournape at gmail.com> wrote:
> On Fri, Nov 12, 2010 at 8:56 PM, Brett Calcott <brett.calcott at gmail.com> wrote:
>> On 15 September 2010 22:58, David Cournapeau <cournape at gmail.com> wrote:
>>> The next version of bento (to be released soon), will have relatively
>>> decent support for cython, in particular chaining with other builder
>>> (so that you could say chain .pyx.in -> .pyx -> .c -> .o -> .so) will
>>> be possible without any monkey patch or other moronic operations.
>>>
>>
>> Is this version out yet?
>
> Yes
>
>> I see there is a test for a pyx file, but it
>> doesn't work on my Mac (cloned from github today)
>>
>> bentomaker: Error: bentomaker crashed (uncaught exception <class
>> 'yaku.task_manager.NoHookException'>: No rule defined for extension
>> '.pyx').

Hm, I should have checked - this is indeed broken, even if you use the
last version in master branch from both bento and yaku. This is a
persistence issue, but I don't understand the issue exactly yet. WIll
look into it.

Note that the build tool itself supports the case fine: if you try the
example from yaku repo, it should work fine (I checked it this time):

https://github.com/cournape/yaku/blob/master/examples/chaining/build.py

cheers,

David

From stefan_ml at behnel.de  Fri Nov 12 16:24:37 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 12 Nov 2010 16:24:37 +0100
Subject: [Cython] What to do about large integer literals and constants?
Message-ID: <4CDD5C35.4080401@behnel.de>

Hi,

one of the CPython regression tests (test_long_future in Py2.7) failed 
because it used the constant expression "1L << 40000". We had this problem 
before, Cython currently calculates the result in the compiler and writes 
it literally into the C source. When I disable the folding for constants of 
that size, it actually writes "PyInt_FromLong(1L << 40000)", which is not a 
bit better.

I found this old thread related to this topic but not much more

http://comments.gmane.org/gmane.comp.python.cython.devel/2449

The main problem here is that we cannot make hard assumptions about the 
target storage type in C. We currently assume (more or less) that a 'long' 
is at least 32bit, but if it happens to be 64bit, it can hold much larger 
constants natively, and we can't know that at code generation time. So our 
best bet is to play safe and use Python computation for things that may not 
necessarily fit the target type. And, yes, my fellow friends of the math, 
this implies a major performance regression in the case that Cython cannot 
know that it actually will fit at C compilation time.

However, instead of changing the constant folding here, I think it would be 
better to implement type inference for integer literals. It can try to find 
a suitable type for a (folded or original) literal, potentially suggesting 
PyLong if we think there isn't a C type to handle it.

The main problem with this approach is that disabling type inference 
explicitly will bring code back to suffering from the above problem, which 
would surely be unexpected for users. So we might have to implement 
something similar at least for the type coercion of integer literals (to 
change literals into PyLong if a large constant coerces to a Python type).

Does this make sense? Any better ideas?

Stefan

From vitja.makarov at gmail.com  Fri Nov 12 16:28:25 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Fri, 12 Nov 2010 18:28:25 +0300
Subject: [Cython] >=python-2.6 property pattern not supported
In-Reply-To: <4CDCF489.3000108@behnel.de>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>
	<4CD811B2.1010108@behnel.de>
	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>
	<4CDCE8D3.70002@behnel.de>
	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>
	<4CDCF489.3000108@behnel.de>
Message-ID: <AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>

2010/11/12 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 12.11.2010 08:52:
>> 2010/11/12 Stefan Behnel:
>>> Vitja Makarov, 12.11.2010 08:06:
>>>> 2010/11/8 Stefan Behnel:
>>>>> The problem is not the property behaviour, it's the mapping from a Python
>>>>> function/method to a C function, which must have a unique name. Currently,
>>>>> functions are unique within one namespace, be it the module namespace or a
>>>>> class body namespace. They are actually looked up statically, not
>>>>> dynamically in definition order, so the fix is not as easy as adding a
>>>>> counted ID to the C-level name.
>>>>
>>>> Is not it safe to add ID to C-function in pure python class scope?
>>>> I see problem in module scope when it turns functions into list of
>>>> methoddef that are passed to PyModule_Init.
>>>
>>> As I said, it's not *enough* to fix the name.
>>
>> Can please you explain the problem? What more should be done?
>>
>> I see problem in decorators here:
>> clsas Foo(object):
>> ? ? ?@property
>> ? ? ?def foo(self)
>> ? ? ?....
>> ? ? ?@foo.setter
>> ? ? ?def foo(self, val):
>> ? ? ?....
>>
>> Setter Cython first sets foo item, then get it back and decorate with
>> setter attribute:
>>
>> dict['foo'] = Foo_foo_method_000
>> dict['foo'] = property(dict['foo'])
>> dict['foo'] = Foo_foo_method_001
>> dict['foo'] = dict['foo'].setter(dict['foo'])
>>
>> Am I right here? I see problem in decorators, attribute assignment here.
>
> This is not related to properties. What I meant by this:
>
> """
> Currently, functions are unique within one namespace, be it the module
> namespace or a class body namespace. They are actually looked up
> statically, not dynamically in definition order
> """
>
> is that you simply cannot currently define a function more than once within
> a namespace because it is looked up statically. So, when you redefine it,
> its Python name would get overwritten in the symbol table. So you couldn't
> refer to the previously defined function anymore from that point on, which
> means that you cannot even use it in the code that gets executed before the
> redefinition. The symbol table is not dynamically adaptive to the position
> in the code, it's just a plain table that is global to a namespace.
>
> This is a compiler, not an interpreter. Things don't simply happen in the
> order they are executed in the code.
>
> Does that explain it?
>

Can you show me where is static lookup performed? Is this cython level
lookup or C-level?
Can you show me example?

Btw, if we see that function is redefined we can mark it as
"overrided" and use special more dynamic rules to handle that (create
it in runtime,  don't declare it in methods list and so on)

From stefan_ml at behnel.de  Fri Nov 12 21:36:18 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 12 Nov 2010 21:36:18 +0100
Subject: [Cython] What to do about large integer literals and constants?
In-Reply-To: <4CDD5C35.4080401@behnel.de>
References: <4CDD5C35.4080401@behnel.de>
Message-ID: <4CDDA542.4040400@behnel.de>

Stefan Behnel, 12.11.2010 16:24:
> one of the CPython regression tests (test_long_future in Py2.7) failed
> because it used the constant expression "1L<<  40000". We had this problem
> before, Cython currently calculates the result in the compiler and writes
> it literally into the C source. When I disable the folding for constants of
> that size, it actually writes "PyInt_FromLong(1L<<  40000)", which is not a
> bit better.
>
> I found this old thread related to this topic but not much more
>
> http://comments.gmane.org/gmane.comp.python.cython.devel/2449
>
> The main problem here is that we cannot make hard assumptions about the
> target storage type in C. We currently assume (more or less) that a 'long'
> is at least 32bit, but if it happens to be 64bit, it can hold much larger
> constants natively, and we can't know that at code generation time. So our
> best bet is to play safe and use Python computation for things that may not
> necessarily fit the target type. And, yes, my fellow friends of the math,
> this implies a major performance regression in the case that Cython cannot
> know that it actually will fit at C compilation time.
>
> However, instead of changing the constant folding here, I think it would be
> better to implement type inference for integer literals. It can try to find
> a suitable type for a (folded or original) literal, potentially suggesting
> PyLong if we think there isn't a C type to handle it.
>
> The main problem with this approach is that disabling type inference
> explicitly will bring code back to suffering from the above problem, which
> would surely be unexpected for users. So we might have to implement
> something similar at least for the type coercion of integer literals (to
> change literals into PyLong if a large constant coerces to a Python type).
>
> Does this make sense? Any better ideas?

I opened a ticket for this:

http://trac.cython.org/cython_trac/ticket/592

I also have an "almost ready" solution that basically leaves obvious C 
literals (enum names or numeric literals ending with "U" or "LL") unchanged 
and only checks that Python compatible literals are <= 32bit. If they are 
not, they are promoted to Python objects. Here's the code:

     def find_suitable_type_for_value(self):
         if self.constant_result is constant_value_not_set:
             try:
                 self.calculate_constant_result()
             except ValueError:
                 pass
         if self.constant_result in (constant_value_not_set,
                                     not_a_constant) or \
                self.unsigned or self.longness == 'LL':
             # clearly a C literal (including enum names etc.)
             rank = (self.longness == 'LL') and 2 or 1
             suitable_type = PyrexTypes.modifiers_and_name_to_type[
                                        not self.unsigned, rank, "int"]
             if self.type:
                 suitable_type = PyrexTypes.widest_numeric_type(
                     suitable_type, self.type)
         else:
             # C literal or Python literal - split at 32bit boundary
             if self.constant_result >= -2**31 and \
                        self.constant_result < 2**31:
                 if self.type and self.type.is_int:
                     suitable_type = self.type
                 else:
                     suitable_type = PyrexTypes.c_long_type
             else:
                 suitable_type = PyrexTypes.py_object_type
         return suitable_type

Comments on the above so far?

I also reworked the constant folding code quite a bit to make it work more 
nicely with changes like the above.

Would a change like this be suitable for 0.13.1 or rather 0.14?

Stefan

From brett.calcott at gmail.com  Fri Nov 12 21:52:18 2010
From: brett.calcott at gmail.com (Brett Calcott)
Date: Sat, 13 Nov 2010 07:52:18 +1100
Subject: [Cython] [Numpy-discussion] Cython distutils
In-Reply-To: <AANLkTinCDEdKn10LJUVLyJU-atOR3JGVKGkCzLjYC7tF@mail.gmail.com>
References: <AANLkTikHurmxH9_htLmTXVNaQ8K3D7hYB6iMy-Th3VvA@mail.gmail.com>
	<AANLkTinjqhVQokMM_P8uJnoY3X0A=kZq1j5+bwg4vmFN@mail.gmail.com>
	<AANLkTinU+xQ7UYSuF6qOef_S060rny3+TgL64qsL33+4@mail.gmail.com>
	<AANLkTimFmW0BmkYj46ZZ7eG91uQVgU7eg6ky0WVQwOhH@mail.gmail.com>
	<AANLkTikR7Xi69uCzVP29FqC8LmY_FwhFx9iuSo6VPvND@mail.gmail.com>
	<AANLkTimhL=uYH6AWNgD53=YCrfDDy+LaECbr7J8aj=kX@mail.gmail.com>
	<AANLkTi=S_J7bgDpy==6+fr=A9mCPpUtQuJpQSXJ-cAzw@mail.gmail.com>
	<AANLkTinCDEdKn10LJUVLyJU-atOR3JGVKGkCzLjYC7tF@mail.gmail.com>
Message-ID: <AANLkTi=mA55ePuVyiCvA9twzvckAk+t+xX+C=pSnAFDg@mail.gmail.com>

On 13 November 2010 00:12, David Cournapeau <cournape at gmail.com> wrote:
>
> Note that the build tool itself supports the case fine: if you try the
> example from yaku repo, it should work fine (I checked it this time):
>
> https://github.com/cournape/yaku/blob/master/examples/chaining/build.py
>

To get it to install required this:

diff --git a/setup.py b/setup.py
index 062884d..deee0db 100644
--- a/setup.py
+++ b/setup.py
@@ -21,5 +21,5 @@ setup(name="yaku",
     author="David Cournapeau",
     author_email="cournape at gmail.com",
     license="BSD",
-    packages=["yaku.tools", "yaku", "yaku.compat"],
+    packages=["yaku.tools", "yaku", "yaku.compat", "yaku.conftests"],
 )

The build works fine, but trying to use the generated hello.so:
[chaining/build]|4> import hello
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)

/Users/brett/Source/yaku/examples/chaining/build/<ipython console> in <module>()

ImportError: dlopen(./hello.so, 2): no suitable image found.  Did find:
	./hello.so: mach-o, but wrong architecture

I guess this is some problem with Mac universal builds. How does yaku
figure out what version (32/64 etc) to build?

Brett

From dagss at student.matnat.uio.no  Fri Nov 12 21:55:49 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 12 Nov 2010 21:55:49 +0100
Subject: [Cython] What to do about large integer literals and constants?
In-Reply-To: <4CDD5C35.4080401@behnel.de>
References: <4CDD5C35.4080401@behnel.de>
Message-ID: <4CDDA9D5.3040105@student.matnat.uio.no>

On 11/12/2010 04:24 PM, Stefan Behnel wrote:
> Hi,
>
> one of the CPython regression tests (test_long_future in Py2.7) failed
> because it used the constant expression "1L<<  40000". We had this problem
> before, Cython currently calculates the result in the compiler and writes
> it literally into the C source. When I disable the folding for constants of
> that size, it actually writes "PyInt_FromLong(1L<<  40000)", which is not a
> bit better.
>
> I found this old thread related to this topic but not much more
>
> http://comments.gmane.org/gmane.comp.python.cython.devel/2449
>
> The main problem here is that we cannot make hard assumptions about the
> target storage type in C. We currently assume (more or less) that a 'long'
> is at least 32bit, but if it happens to be 64bit, it can hold much larger
> constants natively, and we can't know that at code generation time. So our
> best bet is to play safe and use Python computation for things that may not
> necessarily fit the target type. And, yes, my fellow friends of the math,
> this implies a major performance regression in the case that Cython cannot
> know that it actually will fit at C compilation time.
>
> However, instead of changing the constant folding here, I think it would be
> better to implement type inference for integer literals. It can try to find
> a suitable type for a (folded or original) literal, potentially suggesting
> PyLong if we think there isn't a C type to handle it.
>
> The main problem with this approach is that disabling type inference
> explicitly will bring code back to suffering from the above problem, which
> would surely be unexpected for users. So we might have to implement
> something similar at least for the type coercion of integer literals (to
> change literals into PyLong if a large constant coerces to a Python type).
>
> Does this make sense? Any better ideas?
>    

This isn't really another proposal, just a related thought:

For my own code, I would like to have a directive 
@cython.hugeints(False) or similar, that says that all integer values 
can be assumed to stay within the machine range, whatever that is. 
"range(n)" would be assumed to also stay within 32/64 bit, and so on.

I almost exclusively use integers for accessing items in arrays or count 
number of samples or such things. I never use large integers, and I 
could rather safely just set this directive at the top of my files. I 
think this may fit very many programmers.

Dag Sverre

From stefan_ml at behnel.de  Fri Nov 12 22:07:15 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 12 Nov 2010 22:07:15 +0100
Subject: [Cython] >=python-2.6 property pattern not supported
In-Reply-To: <AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>	<4CD811B2.1010108@behnel.de>	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>	<4CDCE8D3.70002@behnel.de>	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>	<4CDCF489.3000108@behnel.de>
	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>
Message-ID: <4CDDAC83.9030903@behnel.de>

Vitja Makarov, 12.11.2010 16:28:
> 2010/11/12 Stefan Behnel:
>> you simply cannot currently define a function more than once within
>> a namespace because it is looked up statically. So, when you redefine it,
>> its Python name would get overwritten in the symbol table. So you couldn't
>> refer to the previously defined function anymore from that point on, which
>> means that you cannot even use it in the code that gets executed before the
>> redefinition. The symbol table is not dynamically adaptive to the position
>> in the code, it's just a plain table that is global to a namespace.
>>
>> This is a compiler, not an interpreter. Things don't simply happen in the
>> order they are executed in the code.
>>
>> Does that explain it?
>>
>
> Can you show me where is static lookup performed?

See the Entry class in Symtab.py, it holds the information about declared 
names and their types.


> Is this cython level lookup or C-level?

Cython. Mostly during type analysis, but potentially also in the type 
declaration phase and other places.


> Can you show me example?

You will find tons of places that call "lookup()" (recursively) or 
"lookup_here()" (non-recusively) on the current scope. Take a look at the 
three call node classes, for example.


> Btw, if we see that function is redefined we can mark it as
> "overrided" and use special more dynamic rules to handle that (create
> it in runtime,  don't declare it in methods list and so on)

Sure. I didn't say we lack solutions. It just needs to be done. And this 
isn't trivial enough to be done within a couple of days, so no-one has 
started working on it.

Stefan

From stefan_ml at behnel.de  Fri Nov 12 22:08:09 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 12 Nov 2010 22:08:09 +0100
Subject: [Cython] metaclasses
In-Reply-To: <4CD80E5E.6020307@behnel.de>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>	<4CD1C983.7000300@behnel.de>	<AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>	<4CD1D787.4010202@behnel.de>	<AANLkTikYRv6CPTbSifo-C_gAwU-AG0REC1sPwAHq2DJO@mail.gmail.com>	<4CD247CD.705@behnel.de>	<AANLkTimEmE9=xiN6=PDNEgRo=P_=ECxE=n7Eh0aYs81g@mail.gmail.com>	<AANLkTimOXxppAQ6Fb5DSCTyRoY9ZJghhV5=sM3i0N6ic@mail.gmail.com>	<4CD29CFD.7030309@behnel.de>	<AANLkTinP8Osz1YKkLyuDhPT=vzyc6iscnOCoi2h0f+Rv@mail.gmail.com>	<4CD2A3EB.4030601@behnel.de>	<AANLkTinHaapMihFN0vHaRD2o3xKVD6Ows-JjD7W_6mzx@mail.gmail.com>	<4CD2B27E.6080805@behnel.de>	<4CD2B585.8040601@behnel.de>	<AANLkTik4qhsVhHhCQRyheKpDzoGGQD-q5r=bJUi2_VV_@mail.gmail.com>	<AANLkTik1H1z-R23Kc6e7FgdMhGdX7JPX5iy=j33oS-J3@mail.gmail.com>	<4CD660E2.9040800@behnel.de>	<AANLkTimCCTQoSmgi91Xz_QVm_4uLBXkhd=fwGuzX6-Y7@mail.gmail.com>	<AANLkTi=ZDvsiw5wWMM52sS1YcWL_3wZxZL-_4DMm_ORE@mail.gmail.com>
	<4CD80E5E.6020307@behnel.de>
Message-ID: <4CDDACB9.7010903@behnel.de>

Stefan Behnel, 08.11.2010 15:51:
> Hi,
>
> I looked through your code and pushed a slightly cleaned up version of your
> latest patch.
>
> Vitja Makarov, 07.11.2010 12:39:
>> 2010/11/7 Vitja Makarov:
>>> 2010/11/7 Stefan Behnel:
>>>> Vitja Makarov, 06.11.2010 21:37:
>>>>> I'm now trying to implement this:
>>>>>
>>>>> Is it correct:
>>>>>
>>>>> If parser detect use of kwargs in class definition it creates Py3ClassDefNode
>>>>>
>>>>> I add this classes:
>>>>> Nodes.Py3ClassDefNode   - ClassDefNode replacement for python3 style classes
>>>>> ExprNodes.Py3ClassNode - ClassNode replacement for python3 style classes
>>>>>
>>>>> ExprNdoes.Py3MetaclassNode - contains metaclass
>>>>>       Holds value returned by __Pyx_Py3MetaclassGet(bases, mkw)
>>>>>
>>>>> ExprNodes.Py3NamespaceNode - contains namespace object
>>>>>       Value returned by __Pyx_Py3MetaclassPrepare(metaclass, base, name,
>>>>> mkw), that calls __prepare__ if any.
>>>>
>>>> Why do you think these new classes are necessary?
>>>
>>> I think this objects are required to represent metaclass and namespace.
>>> If we want real support for __prepare__ (simple version is attached in trac)
>>>
>>> 1. extract metaclass from bases and keywordargs
>>> 2. create namespace by calling metaclass __preapre__
>>> 3. fill namespace with class body
>>> 4. create class
>>>
>>> I don't want to reuse ClassDefNode as it differs too much. To many if/else.
>>> So Py3ClassNode should hold metaclass and namespace objects, I add
>>> Py3MetaclassNode and Py3NamespaceNode.
>>>
>>> Can I use PyTempNode for metaclass and namespace?
>>>
>>> Is there another way?
>>
>> Another way:
>>
>> if py3_metaclass:
>>        mkw = keyword_args.copy()
>>        mkw.update(starstar_args)
>>        metaclass = py3_metaclass_get(bases, mkw)
>>        dict = py3_namespace_prepare(metaclass, bases, name, mkw)
>> else:
>>        dict = dict()
>>
>> # construct class body here...
>> dict['foo'] = 'bar'
>>
>> if py3_metaclass:
>>       classobj = py3_metaclass_new(metaclass, name, bases, dict, mkw, modname)
>> else:
>>       classobj = create_class(bases, dict, name, modname)
>>
>> 2 new object types here: metaclass, namespace.
>
> Yes, I think that's the way to go. The two new nodes should go into
> ExprNodes, maybe named "PyClassMetaclassNode" and "PyClassNamespaceNode".
> Both must have their "is_temp" set to True.
>
> The metaclass node will look up the metaclass in the type keywords and use
> it as its node result value.
>
> The namespace node will be used by ExprNodes.ClassNode just like before but
> should do the "prepare() or dict()" dance above. It obviously needs a
> reference to the metaclass node to do that, in order to access its result
> value.
>
> Using two new node classes here is the easiest way to do it as you only
> need to pass around references to them. Otherwise, you'd have to manually
> allocate temporary variables and pass around their names. That's not any
> harder to do, but it's a lot more ugly.
>
> Note the disposal code call at the end of
> PyClassDefNode.generate_execution_code(). You need to do that for both
> nodes here, that will automatically take care to properly clean up the
> Python references to their results (based on the value of their "is_temp"
> attribute).

Any news here? Anything I can help with?

Stefan

From robertwb at math.washington.edu  Fri Nov 12 22:09:12 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 12 Nov 2010 13:09:12 -0800
Subject: [Cython] What to do about large integer literals and constants?
In-Reply-To: <4CDD5C35.4080401@behnel.de>
References: <4CDD5C35.4080401@behnel.de>
Message-ID: <AANLkTi=hZjM=mXzenM1wdoQ5hz8WDtvNPYx0XNsVsdKu@mail.gmail.com>

On Fri, Nov 12, 2010 at 7:24 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> one of the CPython regression tests (test_long_future in Py2.7) failed
> because it used the constant expression "1L << 40000". We had this problem
> before, Cython currently calculates the result in the compiler and writes
> it literally into the C source. When I disable the folding for constants of
> that size, it actually writes "PyInt_FromLong(1L << 40000)", which is not a
> bit better.
>
> I found this old thread related to this topic but not much more
>
> http://comments.gmane.org/gmane.comp.python.cython.devel/2449
>
> The main problem here is that we cannot make hard assumptions about the
> target storage type in C. We currently assume (more or less) that a 'long'
> is at least 32bit, but if it happens to be 64bit, it can hold much larger
> constants natively, and we can't know that at code generation time. So our
> best bet is to play safe and use Python computation for things that may not
> necessarily fit the target type. And, yes, my fellow friends of the math,
> this implies a major performance regression in the case that Cython cannot
> know that it actually will fit at C compilation time.
>
> However, instead of changing the constant folding here, I think it would be
> better to implement type inference for integer literals. It can try to find
> a suitable type for a (folded or original) literal, potentially suggesting
> PyLong if we think there isn't a C type to handle it.
>
> The main problem with this approach is that disabling type inference
> explicitly will bring code back to suffering from the above problem, which
> would surely be unexpected for users. So we might have to implement
> something similar at least for the type coercion of integer literals (to
> change literals into PyLong if a large constant coerces to a Python type).
>
> Does this make sense? Any better ideas?

I remember talking with Craig Citro about this earlier this year, and
choosing an arbitrary cutoff for inference of literals had odd
side-effects. (Perhaps most of this was due to trying to accommodate
arithmetic.) I think if the user writes

    cdef long a = 1 << 47

this should emit pure C, but if they write

    a = 1 << 47

then I'm OK with infering this to be a Python object (which, I should
note, it currently does, though as a side note I'm not convinced we
should but such *huge* pre-computed literals in the source code...).
Perhaps this is just about intermediate results, which is much
trickier. What was the exact code snippet?

- Robert

From robertwb at math.washington.edu  Fri Nov 12 22:11:03 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 12 Nov 2010 13:11:03 -0800
Subject: [Cython] What to do about large integer literals and constants?
In-Reply-To: <4CDDA9D5.3040105@student.matnat.uio.no>
References: <4CDD5C35.4080401@behnel.de>
	<4CDDA9D5.3040105@student.matnat.uio.no>
Message-ID: <AANLkTik4Db_Z37pYfh+Wp9_oS0YnbivMeuA7ammhjLmo@mail.gmail.com>

On Fri, Nov 12, 2010 at 12:55 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> On 11/12/2010 04:24 PM, Stefan Behnel wrote:
>> Hi,
>>
>> one of the CPython regression tests (test_long_future in Py2.7) failed
>> because it used the constant expression "1L<< ?40000". We had this problem
>> before, Cython currently calculates the result in the compiler and writes
>> it literally into the C source. When I disable the folding for constants of
>> that size, it actually writes "PyInt_FromLong(1L<< ?40000)", which is not a
>> bit better.
>>
>> I found this old thread related to this topic but not much more
>>
>> http://comments.gmane.org/gmane.comp.python.cython.devel/2449
>>
>> The main problem here is that we cannot make hard assumptions about the
>> target storage type in C. We currently assume (more or less) that a 'long'
>> is at least 32bit, but if it happens to be 64bit, it can hold much larger
>> constants natively, and we can't know that at code generation time. So our
>> best bet is to play safe and use Python computation for things that may not
>> necessarily fit the target type. And, yes, my fellow friends of the math,
>> this implies a major performance regression in the case that Cython cannot
>> know that it actually will fit at C compilation time.
>>
>> However, instead of changing the constant folding here, I think it would be
>> better to implement type inference for integer literals. It can try to find
>> a suitable type for a (folded or original) literal, potentially suggesting
>> PyLong if we think there isn't a C type to handle it.
>>
>> The main problem with this approach is that disabling type inference
>> explicitly will bring code back to suffering from the above problem, which
>> would surely be unexpected for users. So we might have to implement
>> something similar at least for the type coercion of integer literals (to
>> change literals into PyLong if a large constant coerces to a Python type).
>>
>> Does this make sense? Any better ideas?
>>
>
> This isn't really another proposal, just a related thought:
>
> For my own code, I would like to have a directive
> @cython.hugeints(False) or similar, that says that all integer values
> can be assumed to stay within the machine range, whatever that is.
> "range(n)" would be assumed to also stay within 32/64 bit, and so on.
>
> I almost exclusively use integers for accessing items in arrays or count
> number of samples or such things. I never use large integers, and I
> could rather safely just set this directive at the top of my files. I
> think this may fit very many programmers.

Whatever we do, +1 to making it easily toggle-able, as this will make
debugging unexpected overflows much easier.

- Robert

From vitja.makarov at gmail.com  Fri Nov 12 22:33:22 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 13 Nov 2010 00:33:22 +0300
Subject: [Cython] metaclasses
In-Reply-To: <4CDDACB9.7010903@behnel.de>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CD1C983.7000300@behnel.de>
	<AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>
	<4CD1D787.4010202@behnel.de>
	<AANLkTikYRv6CPTbSifo-C_gAwU-AG0REC1sPwAHq2DJO@mail.gmail.com>
	<4CD247CD.705@behnel.de>
	<AANLkTimEmE9=xiN6=PDNEgRo=P_=ECxE=n7Eh0aYs81g@mail.gmail.com>
	<AANLkTimOXxppAQ6Fb5DSCTyRoY9ZJghhV5=sM3i0N6ic@mail.gmail.com>
	<4CD29CFD.7030309@behnel.de>
	<AANLkTinP8Osz1YKkLyuDhPT=vzyc6iscnOCoi2h0f+Rv@mail.gmail.com>
	<4CD2A3EB.4030601@behnel.de>
	<AANLkTinHaapMihFN0vHaRD2o3xKVD6Ows-JjD7W_6mzx@mail.gmail.com>
	<4CD2B27E.6080805@behnel.de> <4CD2B585.8040601@behnel.de>
	<AANLkTik4qhsVhHhCQRyheKpDzoGGQD-q5r=bJUi2_VV_@mail.gmail.com>
	<AANLkTik1H1z-R23Kc6e7FgdMhGdX7JPX5iy=j33oS-J3@mail.gmail.com>
	<4CD660E2.9040800@behnel.de>
	<AANLkTimCCTQoSmgi91Xz_QVm_4uLBXkhd=fwGuzX6-Y7@mail.gmail.com>
	<AANLkTi=ZDvsiw5wWMM52sS1YcWL_3wZxZL-_4DMm_ORE@mail.gmail.com>
	<4CD80E5E.6020307@behnel.de> <4CDDACB9.7010903@behnel.de>
Message-ID: <AANLkTimh+B3XTBgOu0=J=-WQnj14F6w10bLBe01-BgtA@mail.gmail.com>

2010/11/13 Stefan Behnel <stefan_ml at behnel.de>:
> Stefan Behnel, 08.11.2010 15:51:
>> Hi,
>>
>> I looked through your code and pushed a slightly cleaned up version of your
>> latest patch.
>>
>> Vitja Makarov, 07.11.2010 12:39:
>>> 2010/11/7 Vitja Makarov:
>>>> 2010/11/7 Stefan Behnel:
>>>>> Vitja Makarov, 06.11.2010 21:37:
>>>>>> I'm now trying to implement this:
>>>>>>
>>>>>> Is it correct:
>>>>>>
>>>>>> If parser detect use of kwargs in class definition it creates Py3ClassDefNode
>>>>>>
>>>>>> I add this classes:
>>>>>> Nodes.Py3ClassDefNode ? - ClassDefNode replacement for python3 style classes
>>>>>> ExprNodes.Py3ClassNode - ClassNode replacement for python3 style classes
>>>>>>
>>>>>> ExprNdoes.Py3MetaclassNode - contains metaclass
>>>>>> ? ? ? Holds value returned by __Pyx_Py3MetaclassGet(bases, mkw)
>>>>>>
>>>>>> ExprNodes.Py3NamespaceNode - contains namespace object
>>>>>> ? ? ? Value returned by __Pyx_Py3MetaclassPrepare(metaclass, base, name,
>>>>>> mkw), that calls __prepare__ if any.
>>>>>
>>>>> Why do you think these new classes are necessary?
>>>>
>>>> I think this objects are required to represent metaclass and namespace.
>>>> If we want real support for __prepare__ (simple version is attached in trac)
>>>>
>>>> 1. extract metaclass from bases and keywordargs
>>>> 2. create namespace by calling metaclass __preapre__
>>>> 3. fill namespace with class body
>>>> 4. create class
>>>>
>>>> I don't want to reuse ClassDefNode as it differs too much. To many if/else.
>>>> So Py3ClassNode should hold metaclass and namespace objects, I add
>>>> Py3MetaclassNode and Py3NamespaceNode.
>>>>
>>>> Can I use PyTempNode for metaclass and namespace?
>>>>
>>>> Is there another way?
>>>
>>> Another way:
>>>
>>> if py3_metaclass:
>>> ? ? ? ?mkw = keyword_args.copy()
>>> ? ? ? ?mkw.update(starstar_args)
>>> ? ? ? ?metaclass = py3_metaclass_get(bases, mkw)
>>> ? ? ? ?dict = py3_namespace_prepare(metaclass, bases, name, mkw)
>>> else:
>>> ? ? ? ?dict = dict()
>>>
>>> # construct class body here...
>>> dict['foo'] = 'bar'
>>>
>>> if py3_metaclass:
>>> ? ? ? classobj = py3_metaclass_new(metaclass, name, bases, dict, mkw, modname)
>>> else:
>>> ? ? ? classobj = create_class(bases, dict, name, modname)
>>>
>>> 2 new object types here: metaclass, namespace.
>>
>> Yes, I think that's the way to go. The two new nodes should go into
>> ExprNodes, maybe named "PyClassMetaclassNode" and "PyClassNamespaceNode".
>> Both must have their "is_temp" set to True.
>>
>> The metaclass node will look up the metaclass in the type keywords and use
>> it as its node result value.
>>
>> The namespace node will be used by ExprNodes.ClassNode just like before but
>> should do the "prepare() or dict()" dance above. It obviously needs a
>> reference to the metaclass node to do that, in order to access its result
>> value.
>>
>> Using two new node classes here is the easiest way to do it as you only
>> need to pass around references to them. Otherwise, you'd have to manually
>> allocate temporary variables and pass around their names. That's not any
>> harder to do, but it's a lot more ugly.
>>
>> Note the disposal code call at the end of
>> PyClassDefNode.generate_execution_code(). You need to do that for both
>> nodes here, that will automatically take care to properly clean up the
>> Python references to their results (based on the value of their "is_temp"
>> attribute).
>
> Any news here? Anything I can help with?
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

No time this week, I'll try on weekend.
Think there will be three new nodes:

 * PyClassKeywordArgsNode - will hold keyword args (keywords and
starstar), need this because dict should be used in __prepare__ and
class creation.
 * PyClassMetaclassNode
 * PyClassNamespaceNode

Also new utility code will be added.
Btw that doesn't seems to be a big change ;)

From stefan_ml at behnel.de  Fri Nov 12 22:46:15 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 12 Nov 2010 22:46:15 +0100
Subject: [Cython] What to do about large integer literals and constants?
In-Reply-To: <AANLkTi=hZjM=mXzenM1wdoQ5hz8WDtvNPYx0XNsVsdKu@mail.gmail.com>
References: <4CDD5C35.4080401@behnel.de>
	<AANLkTi=hZjM=mXzenM1wdoQ5hz8WDtvNPYx0XNsVsdKu@mail.gmail.com>
Message-ID: <4CDDB5A7.3000105@behnel.de>

Robert Bradshaw, 12.11.2010 22:09:
> On Fri, Nov 12, 2010 at 7:24 AM, Stefan Behnel wrote:
>>
>> one of the CPython regression tests (test_long_future in Py2.7) failed
>> because it used the constant expression "1L<<  40000". We had this problem
>> before, Cython currently calculates the result in the compiler and writes
>> it literally into the C source. When I disable the folding for constants of
>> that size, it actually writes "PyInt_FromLong(1L<<  40000)", which is not a
>> bit better.
>>
>> I found this old thread related to this topic but not much more
>>
>> http://comments.gmane.org/gmane.comp.python.cython.devel/2449
>>
>> The main problem here is that we cannot make hard assumptions about the
>> target storage type in C. We currently assume (more or less) that a 'long'
>> is at least 32bit, but if it happens to be 64bit, it can hold much larger
>> constants natively, and we can't know that at code generation time. So our
>> best bet is to play safe and use Python computation for things that may not
>> necessarily fit the target type. And, yes, my fellow friends of the math,
>> this implies a major performance regression in the case that Cython cannot
>> know that it actually will fit at C compilation time.
>>
>> However, instead of changing the constant folding here, I think it would be
>> better to implement type inference for integer literals. It can try to find
>> a suitable type for a (folded or original) literal, potentially suggesting
>> PyLong if we think there isn't a C type to handle it.
>>
>> The main problem with this approach is that disabling type inference
>> explicitly will bring code back to suffering from the above problem, which
>> would surely be unexpected for users. So we might have to implement
>> something similar at least for the type coercion of integer literals (to
>> change literals into PyLong if a large constant coerces to a Python type).
>>
>> Does this make sense? Any better ideas?
>
> I remember talking with Craig Citro about this earlier this year, and
> choosing an arbitrary cutoff for inference of literals had odd
> side-effects.

Like what? We always try to keep up Python semantics, so it shouldn't 
really matter if we do arithmetic in C space or Python space.


> I think if the user writes
>
>      cdef long a = 1<<  47
>
> this should emit pure C

In this case, the only difference in semantics is if we simply let the C 
compiler truncate the value or raise an OverflowError. But I admit that we 
do not currently do that for smaller types either, e.g.

      cdef char a = 1 <<  20

would likely continue to wrap around in C. I'm actually fine with that and 
wouldn't mind coercing an inferred Python integer back into a C integer on 
request, like in the case above. That would even be efficient once we have 
folded the expression into an integer constant. But you'd still get a 
warning from the C compiler here.

Think of a case like this:

     cdef int a = (1 << 200) >> 199

Just an odd way of writing 2, but you may get the idea. This can safely be 
computed in Python space and the result fed back into C space, but 
calculating it entirely in C space will assign 0.


> but if they write
>
>      a = 1<<  47
>
> then I'm OK with infering this to be a Python object (which, I should
> note, it currently does, though as a side note I'm not convinced we
> should but such *huge* pre-computed literals in the source code...).

Well, it's exactly the intent to make literals pass through C code safely. 
That's not currently the case because the C compiler can truncate values at 
a platform specific bit length. Moving potentially unsafe integer sizes 
into Python space would solve that, obviously at the price of lower 
performance but higher Python compliance.


> Perhaps this is just about intermediate results, which is much
> trickier. What was the exact code snippet?

Something like

     a = 1L << 40000

in a .py file. Pretty much filled my screen on output. ;-)

Stefan

From stefan_ml at behnel.de  Fri Nov 12 23:20:00 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 12 Nov 2010 23:20:00 +0100
Subject: [Cython] What to do about large integer literals and constants?
In-Reply-To: <4CDDA542.4040400@behnel.de>
References: <4CDD5C35.4080401@behnel.de> <4CDDA542.4040400@behnel.de>
Message-ID: <4CDDBD90.4050804@behnel.de>

Stefan Behnel, 12.11.2010 21:36:
> Stefan Behnel, 12.11.2010 16:24:
>> one of the CPython regression tests (test_long_future in Py2.7) failed
>> because it used the constant expression "1L<<   40000". We had this problem
>> before, Cython currently calculates the result in the compiler and writes
>> it literally into the C source. When I disable the folding for constants of
>> that size, it actually writes "PyInt_FromLong(1L<<   40000)", which is not a
>> bit better.
>>
>> I found this old thread related to this topic but not much more
>>
>> http://comments.gmane.org/gmane.comp.python.cython.devel/2449
>>
>> The main problem here is that we cannot make hard assumptions about the
>> target storage type in C. We currently assume (more or less) that a 'long'
>> is at least 32bit, but if it happens to be 64bit, it can hold much larger
>> constants natively, and we can't know that at code generation time. So our
>> best bet is to play safe and use Python computation for things that may not
>> necessarily fit the target type. And, yes, my fellow friends of the math,
>> this implies a major performance regression in the case that Cython cannot
>> know that it actually will fit at C compilation time.
>>
>> However, instead of changing the constant folding here, I think it would be
>> better to implement type inference for integer literals. It can try to find
>> a suitable type for a (folded or original) literal, potentially suggesting
>> PyLong if we think there isn't a C type to handle it.
>>
>> The main problem with this approach is that disabling type inference
>> explicitly will bring code back to suffering from the above problem, which
>> would surely be unexpected for users. So we might have to implement
>> something similar at least for the type coercion of integer literals (to
>> change literals into PyLong if a large constant coerces to a Python type).
>>
>> Does this make sense? Any better ideas?
>
> I opened a ticket for this:
>
> http://trac.cython.org/cython_trac/ticket/592
>
> I also have an "almost ready" solution that basically leaves obvious C
> literals (enum names or numeric literals ending with "U" or "LL") unchanged
> and only checks that Python compatible literals are<= 32bit. If they are
> not, they are promoted to Python objects. Here's the code:
>
>       def find_suitable_type_for_value(self):
>           if self.constant_result is constant_value_not_set:
>               try:
>                   self.calculate_constant_result()
>               except ValueError:
>                   pass
>           if self.constant_result in (constant_value_not_set,
>                                       not_a_constant) or \
>                  self.unsigned or self.longness == 'LL':
>               # clearly a C literal (including enum names etc.)
>               rank = (self.longness == 'LL') and 2 or 1
>               suitable_type = PyrexTypes.modifiers_and_name_to_type[
>                                          not self.unsigned, rank, "int"]
>               if self.type:
>                   suitable_type = PyrexTypes.widest_numeric_type(
>                       suitable_type, self.type)
>           else:
>               # C literal or Python literal - split at 32bit boundary
>               if self.constant_result>= -2**31 and \
>                          self.constant_result<  2**31:
>                   if self.type and self.type.is_int:
>                       suitable_type = self.type
>                   else:
>                       suitable_type = PyrexTypes.c_long_type
>               else:
>                   suitable_type = PyrexTypes.py_object_type
>           return suitable_type
>
> Comments on the above so far?
>
> I also reworked the constant folding code quite a bit to make it work more
> nicely with changes like the above.

I pushed several constant folding improvements in separate commits. The 
code above is included here:

http://hg.cython.org/cython-devel/rev/ec85b2d8aefb

That's the one that I consider worth some serious testing because it may 
have a behavioural impact on code. It should be the only one to roll back 
if this change is not accepted.

There's also the follow-up commit for the constant folding transformation

http://hg.cython.org/cython-devel/rev/346813017914

but I think that should be mostly safe, even without the preceding change.


> Would a change like this be suitable for 0.13.1 or rather 0.14?

Stefan

From robertwb at math.washington.edu  Sat Nov 13 05:17:28 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 12 Nov 2010 20:17:28 -0800
Subject: [Cython] What to do about large integer literals and constants?
In-Reply-To: <4CDDBD90.4050804@behnel.de>
References: <4CDD5C35.4080401@behnel.de> <4CDDA542.4040400@behnel.de>
	<4CDDBD90.4050804@behnel.de>
Message-ID: <AANLkTik7qV39TobamWxQ=UoV76wSK2rGT8xR0kFptQKs@mail.gmail.com>

On Fri, Nov 12, 2010 at 2:20 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Stefan Behnel, 12.11.2010 21:36:
>> Stefan Behnel, 12.11.2010 16:24:
>>> one of the CPython regression tests (test_long_future in Py2.7) failed
>>> because it used the constant expression "1L<< ? 40000". We had this problem
>>> before, Cython currently calculates the result in the compiler and writes
>>> it literally into the C source. When I disable the folding for constants of
>>> that size, it actually writes "PyInt_FromLong(1L<< ? 40000)", which is not a
>>> bit better.
>>>
>>> I found this old thread related to this topic but not much more
>>>
>>> http://comments.gmane.org/gmane.comp.python.cython.devel/2449
>>>
>>> The main problem here is that we cannot make hard assumptions about the
>>> target storage type in C. We currently assume (more or less) that a 'long'
>>> is at least 32bit, but if it happens to be 64bit, it can hold much larger
>>> constants natively, and we can't know that at code generation time. So our
>>> best bet is to play safe and use Python computation for things that may not
>>> necessarily fit the target type. And, yes, my fellow friends of the math,
>>> this implies a major performance regression in the case that Cython cannot
>>> know that it actually will fit at C compilation time.
>>>
>>> However, instead of changing the constant folding here, I think it would be
>>> better to implement type inference for integer literals. It can try to find
>>> a suitable type for a (folded or original) literal, potentially suggesting
>>> PyLong if we think there isn't a C type to handle it.
>>>
>>> The main problem with this approach is that disabling type inference
>>> explicitly will bring code back to suffering from the above problem, which
>>> would surely be unexpected for users. So we might have to implement
>>> something similar at least for the type coercion of integer literals (to
>>> change literals into PyLong if a large constant coerces to a Python type).
>>>
>>> Does this make sense? Any better ideas?
>>
>> I opened a ticket for this:
>>
>> http://trac.cython.org/cython_trac/ticket/592
>>
>> I also have an "almost ready" solution that basically leaves obvious C
>> literals (enum names or numeric literals ending with "U" or "LL") unchanged
>> and only checks that Python compatible literals are<= 32bit. If they are
>> not, they are promoted to Python objects. Here's the code:
>>
>> ? ? ? def find_suitable_type_for_value(self):
>> ? ? ? ? ? if self.constant_result is constant_value_not_set:
>> ? ? ? ? ? ? ? try:
>> ? ? ? ? ? ? ? ? ? self.calculate_constant_result()
>> ? ? ? ? ? ? ? except ValueError:
>> ? ? ? ? ? ? ? ? ? pass
>> ? ? ? ? ? if self.constant_result in (constant_value_not_set,
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? not_a_constant) or \
>> ? ? ? ? ? ? ? ? ?self.unsigned or self.longness == 'LL':
>> ? ? ? ? ? ? ? # clearly a C literal (including enum names etc.)
>> ? ? ? ? ? ? ? rank = (self.longness == 'LL') and 2 or 1
>> ? ? ? ? ? ? ? suitable_type = PyrexTypes.modifiers_and_name_to_type[
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?not self.unsigned, rank, "int"]
>> ? ? ? ? ? ? ? if self.type:
>> ? ? ? ? ? ? ? ? ? suitable_type = PyrexTypes.widest_numeric_type(
>> ? ? ? ? ? ? ? ? ? ? ? suitable_type, self.type)
>> ? ? ? ? ? else:
>> ? ? ? ? ? ? ? # C literal or Python literal - split at 32bit boundary
>> ? ? ? ? ? ? ? if self.constant_result>= -2**31 and \
>> ? ? ? ? ? ? ? ? ? ? ? ? ?self.constant_result< ?2**31:
>> ? ? ? ? ? ? ? ? ? if self.type and self.type.is_int:
>> ? ? ? ? ? ? ? ? ? ? ? suitable_type = self.type
>> ? ? ? ? ? ? ? ? ? else:
>> ? ? ? ? ? ? ? ? ? ? ? suitable_type = PyrexTypes.c_long_type
>> ? ? ? ? ? ? ? else:
>> ? ? ? ? ? ? ? ? ? suitable_type = PyrexTypes.py_object_type
>> ? ? ? ? ? return suitable_type
>>
>> Comments on the above so far?
>>
>> I also reworked the constant folding code quite a bit to make it work more
>> nicely with changes like the above.
>
> I pushed several constant folding improvements in separate commits. The
> code above is included here:
>
> http://hg.cython.org/cython-devel/rev/ec85b2d8aefb
>
> That's the one that I consider worth some serious testing because it may
> have a behavioural impact on code. It should be the only one to roll back
> if this change is not accepted.

Too bad Sage just recently died in the build bot. I'll try to see
what's up with that.

> There's also the follow-up commit for the constant folding transformation
>
> http://hg.cython.org/cython-devel/rev/346813017914
>
> but I think that should be mostly safe, even without the preceding change.
>
>
>> Would a change like this be suitable for 0.13.1 or rather 0.14?

I'd say so, but I'm not 100% sure. Could you be more precise about
what changed? Before, if you had something like

    a = 1000000000000000000
    b = 1 << 100

it worked just fine. It's clear you did some major re-writing here
though. Would it be fair to say the (post-changes) semantics of
integer constants (literals or compile time expressions) are that they
are C ints if they are less than 32 bits, and Python object otherwise?
In both cases coercible to the other depending on the context (e.g.
assignment to a cdef'd variable)?

- Robert

From stefan_ml at behnel.de  Sat Nov 13 07:28:10 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 13 Nov 2010 07:28:10 +0100
Subject: [Cython] What to do about large integer literals and constants?
In-Reply-To: <AANLkTik7qV39TobamWxQ=UoV76wSK2rGT8xR0kFptQKs@mail.gmail.com>
References: <4CDD5C35.4080401@behnel.de>
	<4CDDA542.4040400@behnel.de>	<4CDDBD90.4050804@behnel.de>
	<AANLkTik7qV39TobamWxQ=UoV76wSK2rGT8xR0kFptQKs@mail.gmail.com>
Message-ID: <4CDE2FFA.6010108@behnel.de>

Robert Bradshaw, 13.11.2010 05:17:
> On Fri, Nov 12, 2010 at 2:20 PM, Stefan Behnel wrote:
>>> Would a change like this be suitable for 0.13.1 or rather 0.14?
>
> I'd say so, but I'm not 100% sure. Could you be more precise about
> what changed? Before, if you had something like
>
>      a = 1000000000000000000
>      b = 1<<  100
>
> it worked just fine.

There were apparently cases where the simple large integer assignments 
above wouldn't work. I already gave the example of

     huge = 1L << 40000


> It's clear you did some major re-writing here
> though. Would it be fair to say the (post-changes) semantics of
> integer constants (literals or compile time expressions) are that they
> are C ints if they are less than 32 bits, and Python object otherwise?
> In both cases coercible to the other depending on the context (e.g.
> assignment to a cdef'd variable)?

Yes, I think that's basically it. I consider it much safer now because 
large integer constants will not just accidentally end up in the C code.

Plus, the constant folding code now keeps track of the signedness and 
longness of literals, so if we end up coercing back to C, the literal will 
be well written.

And negative signs were not constant folded, so

     a = -(1 << 100)

didn't work for me.

Stefan

From stefan_ml at behnel.de  Sat Nov 13 11:00:31 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 13 Nov 2010 11:00:31 +0100
Subject: [Cython] What to do about large integer literals and constants?
In-Reply-To: <4CDDA542.4040400@behnel.de>
References: <4CDD5C35.4080401@behnel.de> <4CDDA542.4040400@behnel.de>
Message-ID: <4CDE61BF.3020705@behnel.de>

Stefan Behnel, 12.11.2010 21:36:
> solution that basically leaves obvious C
> literals (enum names or numeric literals ending with "U" or "LL") unchanged
> and only checks that Python compatible literals are<= 32bit.

On a related note, should we treat "123L" literals as explicit C long 
literals in -3 mode? After all, the "L" suffix is only ambiguous in Cython 
2 mode (and clearly a Python literal suffix in Python 2 .py files).

Stefan

From vitja.makarov at gmail.com  Sat Nov 13 14:15:20 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 13 Nov 2010 16:15:20 +0300
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTimh+B3XTBgOu0=J=-WQnj14F6w10bLBe01-BgtA@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CD1C983.7000300@behnel.de>
	<AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>
	<4CD1D787.4010202@behnel.de>
	<AANLkTikYRv6CPTbSifo-C_gAwU-AG0REC1sPwAHq2DJO@mail.gmail.com>
	<4CD247CD.705@behnel.de>
	<AANLkTimEmE9=xiN6=PDNEgRo=P_=ECxE=n7Eh0aYs81g@mail.gmail.com>
	<AANLkTimOXxppAQ6Fb5DSCTyRoY9ZJghhV5=sM3i0N6ic@mail.gmail.com>
	<4CD29CFD.7030309@behnel.de>
	<AANLkTinP8Osz1YKkLyuDhPT=vzyc6iscnOCoi2h0f+Rv@mail.gmail.com>
	<4CD2A3EB.4030601@behnel.de>
	<AANLkTinHaapMihFN0vHaRD2o3xKVD6Ows-JjD7W_6mzx@mail.gmail.com>
	<4CD2B27E.6080805@behnel.de> <4CD2B585.8040601@behnel.de>
	<AANLkTik4qhsVhHhCQRyheKpDzoGGQD-q5r=bJUi2_VV_@mail.gmail.com>
	<AANLkTik1H1z-R23Kc6e7FgdMhGdX7JPX5iy=j33oS-J3@mail.gmail.com>
	<4CD660E2.9040800@behnel.de>
	<AANLkTimCCTQoSmgi91Xz_QVm_4uLBXkhd=fwGuzX6-Y7@mail.gmail.com>
	<AANLkTi=ZDvsiw5wWMM52sS1YcWL_3wZxZL-_4DMm_ORE@mail.gmail.com>
	<4CD80E5E.6020307@behnel.de> <4CDDACB9.7010903@behnel.de>
	<AANLkTimh+B3XTBgOu0=J=-WQnj14F6w10bLBe01-BgtA@mail.gmail.com>
Message-ID: <AANLkTiny6S029uJvzenkMgLqfAsFT9g1M9eFK-y=g+Ys@mail.gmail.com>

2010/11/13 Vitja Makarov <vitja.makarov at gmail.com>:
> 2010/11/13 Stefan Behnel <stefan_ml at behnel.de>:
>> Stefan Behnel, 08.11.2010 15:51:
>>> Hi,
>>>
>>> I looked through your code and pushed a slightly cleaned up version of your
>>> latest patch.
>>>
>>> Vitja Makarov, 07.11.2010 12:39:
>>>> 2010/11/7 Vitja Makarov:
>>>>> 2010/11/7 Stefan Behnel:
>>>>>> Vitja Makarov, 06.11.2010 21:37:
>>>>>>> I'm now trying to implement this:
>>>>>>>
>>>>>>> Is it correct:
>>>>>>>
>>>>>>> If parser detect use of kwargs in class definition it creates Py3ClassDefNode
>>>>>>>
>>>>>>> I add this classes:
>>>>>>> Nodes.Py3ClassDefNode ? - ClassDefNode replacement for python3 style classes
>>>>>>> ExprNodes.Py3ClassNode - ClassNode replacement for python3 style classes
>>>>>>>
>>>>>>> ExprNdoes.Py3MetaclassNode - contains metaclass
>>>>>>> ? ? ? Holds value returned by __Pyx_Py3MetaclassGet(bases, mkw)
>>>>>>>
>>>>>>> ExprNodes.Py3NamespaceNode - contains namespace object
>>>>>>> ? ? ? Value returned by __Pyx_Py3MetaclassPrepare(metaclass, base, name,
>>>>>>> mkw), that calls __prepare__ if any.
>>>>>>
>>>>>> Why do you think these new classes are necessary?
>>>>>
>>>>> I think this objects are required to represent metaclass and namespace.
>>>>> If we want real support for __prepare__ (simple version is attached in trac)
>>>>>
>>>>> 1. extract metaclass from bases and keywordargs
>>>>> 2. create namespace by calling metaclass __preapre__
>>>>> 3. fill namespace with class body
>>>>> 4. create class
>>>>>
>>>>> I don't want to reuse ClassDefNode as it differs too much. To many if/else.
>>>>> So Py3ClassNode should hold metaclass and namespace objects, I add
>>>>> Py3MetaclassNode and Py3NamespaceNode.
>>>>>
>>>>> Can I use PyTempNode for metaclass and namespace?
>>>>>
>>>>> Is there another way?
>>>>
>>>> Another way:
>>>>
>>>> if py3_metaclass:
>>>> ? ? ? ?mkw = keyword_args.copy()
>>>> ? ? ? ?mkw.update(starstar_args)
>>>> ? ? ? ?metaclass = py3_metaclass_get(bases, mkw)
>>>> ? ? ? ?dict = py3_namespace_prepare(metaclass, bases, name, mkw)
>>>> else:
>>>> ? ? ? ?dict = dict()
>>>>
>>>> # construct class body here...
>>>> dict['foo'] = 'bar'
>>>>
>>>> if py3_metaclass:
>>>> ? ? ? classobj = py3_metaclass_new(metaclass, name, bases, dict, mkw, modname)
>>>> else:
>>>> ? ? ? classobj = create_class(bases, dict, name, modname)
>>>>
>>>> 2 new object types here: metaclass, namespace.
>>>
>>> Yes, I think that's the way to go. The two new nodes should go into
>>> ExprNodes, maybe named "PyClassMetaclassNode" and "PyClassNamespaceNode".
>>> Both must have their "is_temp" set to True.
>>>
>>> The metaclass node will look up the metaclass in the type keywords and use
>>> it as its node result value.
>>>
>>> The namespace node will be used by ExprNodes.ClassNode just like before but
>>> should do the "prepare() or dict()" dance above. It obviously needs a
>>> reference to the metaclass node to do that, in order to access its result
>>> value.
>>>
>>> Using two new node classes here is the easiest way to do it as you only
>>> need to pass around references to them. Otherwise, you'd have to manually
>>> allocate temporary variables and pass around their names. That's not any
>>> harder to do, but it's a lot more ugly.
>>>
>>> Note the disposal code call at the end of
>>> PyClassDefNode.generate_execution_code(). You need to do that for both
>>> nodes here, that will automatically take care to properly clean up the
>>> Python references to their results (based on the value of their "is_temp"
>>> attribute).
>>
>> Any news here? Anything I can help with?
>>
>> Stefan
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
> No time this week, I'll try on weekend.
> Think there will be three new nodes:
>
> ?* PyClassKeywordArgsNode - will hold keyword args (keywords and
> starstar), need this because dict should be used in __prepare__ and
> class creation.
> ?* PyClassMetaclassNode
> ?* PyClassNamespaceNode
>
> Also new utility code will be added.
> Btw that doesn't seems to be a big change ;)
>

I've added patch in trac.

It introduces new nodes:

Py3ClassNode
KeywordArgsNode
PyClassBasesNode
PyClassMetaclassNode
PyClassNamespaceNode

From vitja.makarov at gmail.com  Sat Nov 13 14:29:00 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 13 Nov 2010 16:29:00 +0300
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTiny6S029uJvzenkMgLqfAsFT9g1M9eFK-y=g+Ys@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CD1C983.7000300@behnel.de>
	<AANLkTin98tkGXDAWM06EtFxV1Uhh+4pm+Lh0QNararDB@mail.gmail.com>
	<4CD1D787.4010202@behnel.de>
	<AANLkTikYRv6CPTbSifo-C_gAwU-AG0REC1sPwAHq2DJO@mail.gmail.com>
	<4CD247CD.705@behnel.de>
	<AANLkTimEmE9=xiN6=PDNEgRo=P_=ECxE=n7Eh0aYs81g@mail.gmail.com>
	<AANLkTimOXxppAQ6Fb5DSCTyRoY9ZJghhV5=sM3i0N6ic@mail.gmail.com>
	<4CD29CFD.7030309@behnel.de>
	<AANLkTinP8Osz1YKkLyuDhPT=vzyc6iscnOCoi2h0f+Rv@mail.gmail.com>
	<4CD2A3EB.4030601@behnel.de>
	<AANLkTinHaapMihFN0vHaRD2o3xKVD6Ows-JjD7W_6mzx@mail.gmail.com>
	<4CD2B27E.6080805@behnel.de> <4CD2B585.8040601@behnel.de>
	<AANLkTik4qhsVhHhCQRyheKpDzoGGQD-q5r=bJUi2_VV_@mail.gmail.com>
	<AANLkTik1H1z-R23Kc6e7FgdMhGdX7JPX5iy=j33oS-J3@mail.gmail.com>
	<4CD660E2.9040800@behnel.de>
	<AANLkTimCCTQoSmgi91Xz_QVm_4uLBXkhd=fwGuzX6-Y7@mail.gmail.com>
	<AANLkTi=ZDvsiw5wWMM52sS1YcWL_3wZxZL-_4DMm_ORE@mail.gmail.com>
	<4CD80E5E.6020307@behnel.de> <4CDDACB9.7010903@behnel.de>
	<AANLkTimh+B3XTBgOu0=J=-WQnj14F6w10bLBe01-BgtA@mail.gmail.com>
	<AANLkTiny6S029uJvzenkMgLqfAsFT9g1M9eFK-y=g+Ys@mail.gmail.com>
Message-ID: <AANLkTimQsDT7YRbZBSXEeZ9pbvN=xD9Mov+n7-rpcRt3@mail.gmail.com>

2010/11/13 Vitja Makarov <vitja.makarov at gmail.com>:
> 2010/11/13 Vitja Makarov <vitja.makarov at gmail.com>:
>> 2010/11/13 Stefan Behnel <stefan_ml at behnel.de>:
>>> Stefan Behnel, 08.11.2010 15:51:
>>>> Hi,
>>>>
>>>> I looked through your code and pushed a slightly cleaned up version of your
>>>> latest patch.
>>>>
>>>> Vitja Makarov, 07.11.2010 12:39:
>>>>> 2010/11/7 Vitja Makarov:
>>>>>> 2010/11/7 Stefan Behnel:
>>>>>>> Vitja Makarov, 06.11.2010 21:37:
>>>>>>>> I'm now trying to implement this:
>>>>>>>>
>>>>>>>> Is it correct:
>>>>>>>>
>>>>>>>> If parser detect use of kwargs in class definition it creates Py3ClassDefNode
>>>>>>>>
>>>>>>>> I add this classes:
>>>>>>>> Nodes.Py3ClassDefNode ? - ClassDefNode replacement for python3 style classes
>>>>>>>> ExprNodes.Py3ClassNode - ClassNode replacement for python3 style classes
>>>>>>>>
>>>>>>>> ExprNdoes.Py3MetaclassNode - contains metaclass
>>>>>>>> ? ? ? Holds value returned by __Pyx_Py3MetaclassGet(bases, mkw)
>>>>>>>>
>>>>>>>> ExprNodes.Py3NamespaceNode - contains namespace object
>>>>>>>> ? ? ? Value returned by __Pyx_Py3MetaclassPrepare(metaclass, base, name,
>>>>>>>> mkw), that calls __prepare__ if any.
>>>>>>>
>>>>>>> Why do you think these new classes are necessary?
>>>>>>
>>>>>> I think this objects are required to represent metaclass and namespace.
>>>>>> If we want real support for __prepare__ (simple version is attached in trac)
>>>>>>
>>>>>> 1. extract metaclass from bases and keywordargs
>>>>>> 2. create namespace by calling metaclass __preapre__
>>>>>> 3. fill namespace with class body
>>>>>> 4. create class
>>>>>>
>>>>>> I don't want to reuse ClassDefNode as it differs too much. To many if/else.
>>>>>> So Py3ClassNode should hold metaclass and namespace objects, I add
>>>>>> Py3MetaclassNode and Py3NamespaceNode.
>>>>>>
>>>>>> Can I use PyTempNode for metaclass and namespace?
>>>>>>
>>>>>> Is there another way?
>>>>>
>>>>> Another way:
>>>>>
>>>>> if py3_metaclass:
>>>>> ? ? ? ?mkw = keyword_args.copy()
>>>>> ? ? ? ?mkw.update(starstar_args)
>>>>> ? ? ? ?metaclass = py3_metaclass_get(bases, mkw)
>>>>> ? ? ? ?dict = py3_namespace_prepare(metaclass, bases, name, mkw)
>>>>> else:
>>>>> ? ? ? ?dict = dict()
>>>>>
>>>>> # construct class body here...
>>>>> dict['foo'] = 'bar'
>>>>>
>>>>> if py3_metaclass:
>>>>> ? ? ? classobj = py3_metaclass_new(metaclass, name, bases, dict, mkw, modname)
>>>>> else:
>>>>> ? ? ? classobj = create_class(bases, dict, name, modname)
>>>>>
>>>>> 2 new object types here: metaclass, namespace.
>>>>
>>>> Yes, I think that's the way to go. The two new nodes should go into
>>>> ExprNodes, maybe named "PyClassMetaclassNode" and "PyClassNamespaceNode".
>>>> Both must have their "is_temp" set to True.
>>>>
>>>> The metaclass node will look up the metaclass in the type keywords and use
>>>> it as its node result value.
>>>>
>>>> The namespace node will be used by ExprNodes.ClassNode just like before but
>>>> should do the "prepare() or dict()" dance above. It obviously needs a
>>>> reference to the metaclass node to do that, in order to access its result
>>>> value.
>>>>
>>>> Using two new node classes here is the easiest way to do it as you only
>>>> need to pass around references to them. Otherwise, you'd have to manually
>>>> allocate temporary variables and pass around their names. That's not any
>>>> harder to do, but it's a lot more ugly.
>>>>
>>>> Note the disposal code call at the end of
>>>> PyClassDefNode.generate_execution_code(). You need to do that for both
>>>> nodes here, that will automatically take care to properly clean up the
>>>> Python references to their results (based on the value of their "is_temp"
>>>> attribute).
>>>
>>> Any news here? Anything I can help with?
>>>
>>> Stefan
>>> _______________________________________________
>>> Cython-dev mailing list
>>> Cython-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/cython-dev
>>>
>>
>> No time this week, I'll try on weekend.
>> Think there will be three new nodes:
>>
>> ?* PyClassKeywordArgsNode - will hold keyword args (keywords and
>> starstar), need this because dict should be used in __prepare__ and
>> class creation.
>> ?* PyClassMetaclassNode
>> ?* PyClassNamespaceNode
>>
>> Also new utility code will be added.
>> Btw that doesn't seems to be a big change ;)
>>
>
> I've added patch in trac.
>
> It introduces new nodes:
>
> Py3ClassNode
> KeywordArgsNode
> PyClassBasesNode
> PyClassMetaclassNode
> PyClassNamespaceNode
>

Here is test case that fails now, because of decorators implementation.
Bar attribute is assigned twice.

class ODict(dict):
    def __init__(self):
        dict.__init__(self)
        self._order = []
        dict.__setitem__(self, '_order', self._order)

    def __setitem__(self, key, value):
        dict.__setitem__(self, key, value)
        self._order.append(key)

class Base(type):
    @staticmethod
    def __prepare__(*args, **kwargs):
        return ODict()

class Foo(metaclass=Base):
    """
    >>> Foo._order
    ['__module__', '__doc__', 'bar']
    """
    @property
    def bar(self):
        return 0

From stefan_ml at behnel.de  Sat Nov 13 15:40:20 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 13 Nov 2010 15:40:20 +0100
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTiny6S029uJvzenkMgLqfAsFT9g1M9eFK-y=g+Ys@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>	<AANLkTikYRv6CPTbSifo-C_gAwU-AG0REC1sPwAHq2DJO@mail.gmail.com>	<4CD247CD.705@behnel.de>	<AANLkTimEmE9=xiN6=PDNEgRo=P_=ECxE=n7Eh0aYs81g@mail.gmail.com>	<AANLkTimOXxppAQ6Fb5DSCTyRoY9ZJghhV5=sM3i0N6ic@mail.gmail.com>	<4CD29CFD.7030309@behnel.de>	<AANLkTinP8Osz1YKkLyuDhPT=vzyc6iscnOCoi2h0f+Rv@mail.gmail.com>	<4CD2A3EB.4030601@behnel.de>	<AANLkTinHaapMihFN0vHaRD2o3xKVD6Ows-JjD7W_6mzx@mail.gmail.com>	<4CD2B27E.6080805@behnel.de>
	<4CD2B585.8040601@behnel.de>	<AANLkTik4qhsVhHhCQRyheKpDzoGGQD-q5r=bJUi2_VV_@mail.gmail.com>	<AANLkTik1H1z-R23Kc6e7FgdMhGdX7JPX5iy=j33oS-J3@mail.gmail.com>	<4CD660E2.9040800@behnel.de>	<AANLkTimCCTQoSmgi91Xz_QVm_4uLBXkhd=fwGuzX6-Y7@mail.gmail.com>	<AANLkTi=ZDvsiw5wWMM52sS1YcWL_3wZxZL-_4DMm_ORE@mail.gmail.com>	<4CD80E5E.6020307@behnel.de>
	<4CDDACB9.7010903@behnel.de>	<AANLkTimh+B3XTBgOu0=J=-WQnj14F6w10bLBe01-BgtA@mail.gmail.com>
	<AANLkTiny6S029uJvzenkMgLqfAsFT9g1M9eFK-y=g+Ys@mail.gmail.com>
Message-ID: <4CDEA354.6020002@behnel.de>

Vitja Makarov, 13.11.2010 14:15:
> I've added patch in trac.

Thanks, I'll give it a try.

Stefan

From stefan_ml at behnel.de  Sat Nov 13 15:42:26 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 13 Nov 2010 15:42:26 +0100
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTimQsDT7YRbZBSXEeZ9pbvN=xD9Mov+n7-rpcRt3@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>	<4CD247CD.705@behnel.de>	<AANLkTimEmE9=xiN6=PDNEgRo=P_=ECxE=n7Eh0aYs81g@mail.gmail.com>	<AANLkTimOXxppAQ6Fb5DSCTyRoY9ZJghhV5=sM3i0N6ic@mail.gmail.com>	<4CD29CFD.7030309@behnel.de>	<AANLkTinP8Osz1YKkLyuDhPT=vzyc6iscnOCoi2h0f+Rv@mail.gmail.com>	<4CD2A3EB.4030601@behnel.de>	<AANLkTinHaapMihFN0vHaRD2o3xKVD6Ows-JjD7W_6mzx@mail.gmail.com>	<4CD2B27E.6080805@behnel.de>
	<4CD2B585.8040601@behnel.de>	<AANLkTik4qhsVhHhCQRyheKpDzoGGQD-q5r=bJUi2_VV_@mail.gmail.com>	<AANLkTik1H1z-R23Kc6e7FgdMhGdX7JPX5iy=j33oS-J3@mail.gmail.com>	<4CD660E2.9040800@behnel.de>	<AANLkTimCCTQoSmgi91Xz_QVm_4uLBXkhd=fwGuzX6-Y7@mail.gmail.com>	<AANLkTi=ZDvsiw5wWMM52sS1YcWL_3wZxZL-_4DMm_ORE@mail.gmail.com>	<4CD80E5E.6020307@behnel.de>
	<4CDDACB9.7010903@behnel.de>	<AANLkTimh+B3XTBgOu0=J=-WQnj14F6w10bLBe01-BgtA@mail.gmail.com>	<AANLkTiny6S029uJvzenkMgLqfAsFT9g1M9eFK-y=g+Ys@mail.gmail.com>
	<AANLkTimQsDT7YRbZBSXEeZ9pbvN=xD9Mov+n7-rpcRt3@mail.gmail.com>
Message-ID: <4CDEA3D2.5090007@behnel.de>

Vitja Makarov, 13.11.2010 14:29:
> Here is test case that fails now, because of decorators implementation.
> Bar attribute is assigned twice.
>
> class ODict(dict):
>      def __init__(self):
>          dict.__init__(self)
>          self._order = []
>          dict.__setitem__(self, '_order', self._order)
>
>      def __setitem__(self, key, value):
>          dict.__setitem__(self, key, value)
>          self._order.append(key)
>
> class Base(type):
>      @staticmethod
>      def __prepare__(*args, **kwargs):
>          return ODict()
>
> class Foo(metaclass=Base):
>      """
>      >>>  Foo._order
>      ['__module__', '__doc__', 'bar']
>      """
>      @property
>      def bar(self):
>          return 0

That's unrelated to the feature itself, so it's ok for now.

Could you open a trac ticket for this?

Stefan

From stefan_ml at behnel.de  Sat Nov 13 15:57:29 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 13 Nov 2010 15:57:29 +0100
Subject: [Cython] metaclasses
In-Reply-To: <4CDEA354.6020002@behnel.de>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>	<4CD247CD.705@behnel.de>	<AANLkTimEmE9=xiN6=PDNEgRo=P_=ECxE=n7Eh0aYs81g@mail.gmail.com>	<AANLkTimOXxppAQ6Fb5DSCTyRoY9ZJghhV5=sM3i0N6ic@mail.gmail.com>	<4CD29CFD.7030309@behnel.de>	<AANLkTinP8Osz1YKkLyuDhPT=vzyc6iscnOCoi2h0f+Rv@mail.gmail.com>	<4CD2A3EB.4030601@behnel.de>	<AANLkTinHaapMihFN0vHaRD2o3xKVD6Ows-JjD7W_6mzx@mail.gmail.com>	<4CD2B27E.6080805@behnel.de>	<4CD2B585.8040601@behnel.de>	<AANLkTik4qhsVhHhCQRyheKpDzoGGQD-q5r=bJUi2_VV_@mail.gmail.com>	<AANLkTik1H1z-R23Kc6e7FgdMhGdX7JPX5iy=j33oS-J3@mail.gmail.com>	<4CD660E2.9040800@behnel.de>	<AANLkTimCCTQoSmgi91Xz_QVm_4uLBXkhd=fwGuzX6-Y7@mail.gmail.com>	<AANLkTi=ZDvsiw5wWMM52sS1YcWL_3wZxZL-_4DMm_ORE@mail.gmail.com>	<4CD80E5E.6020307@behnel.de>	<4CDDACB9.7010903@behnel.de>	<AANLkTimh+B3XTBgOu0=J=-WQnj14F6w10bLBe01-BgtA@mail.gmail.com>	<AANLkTiny6S029uJvzenkMgLqfAsFT9g1M9eFK-y=g+Ys@mail.gmail.com>
	<4CDEA354.6020002@behnel.de>
Message-ID: <4CDEA759.5030302@behnel.de>

Stefan Behnel, 13.11.2010 15:40:
> Vitja Makarov, 13.11.2010 14:15:
>> I've added patch in trac.
>
> Thanks, I'll give it a try.

It lacks tests for passing **kwargs to the metaclass, and ISTM that this 
feature can't work. Could you add the missing tests?

Stefan

From stefan_ml at behnel.de  Sat Nov 13 16:06:58 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 13 Nov 2010 16:06:58 +0100
Subject: [Cython] metaclasses
In-Reply-To: <4CDEA759.5030302@behnel.de>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>	<4CD247CD.705@behnel.de>	<AANLkTimEmE9=xiN6=PDNEgRo=P_=ECxE=n7Eh0aYs81g@mail.gmail.com>	<AANLkTimOXxppAQ6Fb5DSCTyRoY9ZJghhV5=sM3i0N6ic@mail.gmail.com>	<4CD29CFD.7030309@behnel.de>	<AANLkTinP8Osz1YKkLyuDhPT=vzyc6iscnOCoi2h0f+Rv@mail.gmail.com>	<4CD2A3EB.4030601@behnel.de>	<AANLkTinHaapMihFN0vHaRD2o3xKVD6Ows-JjD7W_6mzx@mail.gmail.com>	<4CD2B27E.6080805@behnel.de>	<4CD2B585.8040601@behnel.de>	<AANLkTik4qhsVhHhCQRyheKpDzoGGQD-q5r=bJUi2_VV_@mail.gmail.com>	<AANLkTik1H1z-R23Kc6e7FgdMhGdX7JPX5iy=j33oS-J3@mail.gmail.com>	<4CD660E2.9040800@behnel.de>	<AANLkTimCCTQoSmgi91Xz_QVm_4uLBXkhd=fwGuzX6-Y7@mail.gmail.com>	<AANLkTi=ZDvsiw5wWMM52sS1YcWL_3wZxZL-_4DMm_ORE@mail.gmail.com>	<4CD80E5E.6020307@behnel.de>	<4CDDACB9.7010903@behnel.de>	<AANLkTimh+B3XTBgOu0=J=-WQnj14F6w10bLBe01-BgtA@mail.gmail.com>	<AANLkTiny6S029uJvzenkMgLqfAsFT9g1M9eFK-y=g+Ys@mail.gmail.com>	<4CDEA354.6020002@behnel.de>
	<4CDEA759.5030302@behnel.de>
Message-ID: <4CDEA992.7010300@behnel.de>

Stefan Behnel, 13.11.2010 15:57:
> Stefan Behnel, 13.11.2010 15:40:
>> Vitja Makarov, 13.11.2010 14:15:
>>> I've added patch in trac.
>>
>> Thanks, I'll give it a try.
>
> It lacks tests for passing **kwargs to the metaclass, and ISTM that this
> feature can't work. Could you add the missing tests?

More comments in the bug tracker.

Stefan

From vitja.makarov at gmail.com  Sat Nov 13 16:35:32 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 13 Nov 2010 18:35:32 +0300
Subject: [Cython] metaclasses
In-Reply-To: <4CDEA992.7010300@behnel.de>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CD247CD.705@behnel.de>
	<AANLkTimEmE9=xiN6=PDNEgRo=P_=ECxE=n7Eh0aYs81g@mail.gmail.com>
	<AANLkTimOXxppAQ6Fb5DSCTyRoY9ZJghhV5=sM3i0N6ic@mail.gmail.com>
	<4CD29CFD.7030309@behnel.de>
	<AANLkTinP8Osz1YKkLyuDhPT=vzyc6iscnOCoi2h0f+Rv@mail.gmail.com>
	<4CD2A3EB.4030601@behnel.de>
	<AANLkTinHaapMihFN0vHaRD2o3xKVD6Ows-JjD7W_6mzx@mail.gmail.com>
	<4CD2B27E.6080805@behnel.de> <4CD2B585.8040601@behnel.de>
	<AANLkTik4qhsVhHhCQRyheKpDzoGGQD-q5r=bJUi2_VV_@mail.gmail.com>
	<AANLkTik1H1z-R23Kc6e7FgdMhGdX7JPX5iy=j33oS-J3@mail.gmail.com>
	<4CD660E2.9040800@behnel.de>
	<AANLkTimCCTQoSmgi91Xz_QVm_4uLBXkhd=fwGuzX6-Y7@mail.gmail.com>
	<AANLkTi=ZDvsiw5wWMM52sS1YcWL_3wZxZL-_4DMm_ORE@mail.gmail.com>
	<4CD80E5E.6020307@behnel.de> <4CDDACB9.7010903@behnel.de>
	<AANLkTimh+B3XTBgOu0=J=-WQnj14F6w10bLBe01-BgtA@mail.gmail.com>
	<AANLkTiny6S029uJvzenkMgLqfAsFT9g1M9eFK-y=g+Ys@mail.gmail.com>
	<4CDEA354.6020002@behnel.de> <4CDEA759.5030302@behnel.de>
	<4CDEA992.7010300@behnel.de>
Message-ID: <AANLkTimmgWR6J92Sg33ECbZo1wAkF-HFYELADVhEEtMr@mail.gmail.com>

2010/11/13 Stefan Behnel <stefan_ml at behnel.de>:
> Stefan Behnel, 13.11.2010 15:57:
>> Stefan Behnel, 13.11.2010 15:40:
>>> Vitja Makarov, 13.11.2010 14:15:
>>>> I've added patch in trac.
>>>
>>> Thanks, I'll give it a try.
>>
>> It lacks tests for passing **kwargs to the metaclass, and ISTM that this
>> feature can't work. Could you add the missing tests?
>
> More comments in the bug tracker.
>

Ok, I'll add. What's that ISTM?

From vitja.makarov at gmail.com  Sat Nov 13 17:01:07 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 13 Nov 2010 19:01:07 +0300
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTimmgWR6J92Sg33ECbZo1wAkF-HFYELADVhEEtMr@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CD247CD.705@behnel.de>
	<AANLkTimEmE9=xiN6=PDNEgRo=P_=ECxE=n7Eh0aYs81g@mail.gmail.com>
	<AANLkTimOXxppAQ6Fb5DSCTyRoY9ZJghhV5=sM3i0N6ic@mail.gmail.com>
	<4CD29CFD.7030309@behnel.de>
	<AANLkTinP8Osz1YKkLyuDhPT=vzyc6iscnOCoi2h0f+Rv@mail.gmail.com>
	<4CD2A3EB.4030601@behnel.de>
	<AANLkTinHaapMihFN0vHaRD2o3xKVD6Ows-JjD7W_6mzx@mail.gmail.com>
	<4CD2B27E.6080805@behnel.de> <4CD2B585.8040601@behnel.de>
	<AANLkTik4qhsVhHhCQRyheKpDzoGGQD-q5r=bJUi2_VV_@mail.gmail.com>
	<AANLkTik1H1z-R23Kc6e7FgdMhGdX7JPX5iy=j33oS-J3@mail.gmail.com>
	<4CD660E2.9040800@behnel.de>
	<AANLkTimCCTQoSmgi91Xz_QVm_4uLBXkhd=fwGuzX6-Y7@mail.gmail.com>
	<AANLkTi=ZDvsiw5wWMM52sS1YcWL_3wZxZL-_4DMm_ORE@mail.gmail.com>
	<4CD80E5E.6020307@behnel.de> <4CDDACB9.7010903@behnel.de>
	<AANLkTimh+B3XTBgOu0=J=-WQnj14F6w10bLBe01-BgtA@mail.gmail.com>
	<AANLkTiny6S029uJvzenkMgLqfAsFT9g1M9eFK-y=g+Ys@mail.gmail.com>
	<4CDEA354.6020002@behnel.de> <4CDEA759.5030302@behnel.de>
	<4CDEA992.7010300@behnel.de>
	<AANLkTimmgWR6J92Sg33ECbZo1wAkF-HFYELADVhEEtMr@mail.gmail.com>
Message-ID: <AANLkTinjxdf5R8zUj7H+HvBnKmNoE3E7bM6i95Z_mG1C@mail.gmail.com>

2010/11/13 Vitja Makarov <vitja.makarov at gmail.com>:
> 2010/11/13 Stefan Behnel <stefan_ml at behnel.de>:
>> Stefan Behnel, 13.11.2010 15:57:
>>> Stefan Behnel, 13.11.2010 15:40:
>>>> Vitja Makarov, 13.11.2010 14:15:
>>>>> I've added patch in trac.
>>>>
>>>> Thanks, I'll give it a try.
>>>
>>> It lacks tests for passing **kwargs to the metaclass, and ISTM that this
>>> feature can't work. Could you add the missing tests?
>>
>> More comments in the bug tracker.
>>
>
> Ok, I'll add. What's that ISTM?
>

KeywordArgsNode was buggy I've attached new patch, that solves the
issue and add some more test cases.

From stefan_ml at behnel.de  Sat Nov 13 18:49:42 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 13 Nov 2010 18:49:42 +0100
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTimmgWR6J92Sg33ECbZo1wAkF-HFYELADVhEEtMr@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>	<AANLkTimOXxppAQ6Fb5DSCTyRoY9ZJghhV5=sM3i0N6ic@mail.gmail.com>	<4CD29CFD.7030309@behnel.de>	<AANLkTinP8Osz1YKkLyuDhPT=vzyc6iscnOCoi2h0f+Rv@mail.gmail.com>	<4CD2A3EB.4030601@behnel.de>	<AANLkTinHaapMihFN0vHaRD2o3xKVD6Ows-JjD7W_6mzx@mail.gmail.com>	<4CD2B27E.6080805@behnel.de>
	<4CD2B585.8040601@behnel.de>	<AANLkTik4qhsVhHhCQRyheKpDzoGGQD-q5r=bJUi2_VV_@mail.gmail.com>	<AANLkTik1H1z-R23Kc6e7FgdMhGdX7JPX5iy=j33oS-J3@mail.gmail.com>	<4CD660E2.9040800@behnel.de>	<AANLkTimCCTQoSmgi91Xz_QVm_4uLBXkhd=fwGuzX6-Y7@mail.gmail.com>	<AANLkTi=ZDvsiw5wWMM52sS1YcWL_3wZxZL-_4DMm_ORE@mail.gmail.com>	<4CD80E5E.6020307@behnel.de>
	<4CDDACB9.7010903@behnel.de>	<AANLkTimh+B3XTBgOu0=J=-WQnj14F6w10bLBe01-BgtA@mail.gmail.com>	<AANLkTiny6S029uJvzenkMgLqfAsFT9g1M9eFK-y=g+Ys@mail.gmail.com>	<4CDEA354.6020002@behnel.de>
	<4CDEA759.5030302@behnel.de>	<4CDEA992.7010300@behnel.de>
	<AANLkTimmgWR6J92Sg33ECbZo1wAkF-HFYELADVhEEtMr@mail.gmail.com>
Message-ID: <4CDECFB6.2080501@behnel.de>

Vitja Makarov, 13.11.2010 16:35:
> 2010/11/13 Stefan Behnel:
>> Stefan Behnel, 13.11.2010 15:57:
>>> Stefan Behnel, 13.11.2010 15:40:
>>>> Vitja Makarov, 13.11.2010 14:15:
>>>>> I've added patch in trac.
>>>>
>>>> Thanks, I'll give it a try.
>>>
>>> It lacks tests for passing **kwargs to the metaclass, and ISTM that this
>>> feature can't work. Could you add the missing tests?
>>
>> More comments in the bug tracker.
>>
>
> Ok, I'll add. What's that ISTM?

It Seems To Me. :)

Stefan

From vitja.makarov at gmail.com  Sat Nov 13 19:04:01 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 13 Nov 2010 21:04:01 +0300
Subject: [Cython] metaclasses
In-Reply-To: <4CDECFB6.2080501@behnel.de>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<AANLkTimOXxppAQ6Fb5DSCTyRoY9ZJghhV5=sM3i0N6ic@mail.gmail.com>
	<4CD29CFD.7030309@behnel.de>
	<AANLkTinP8Osz1YKkLyuDhPT=vzyc6iscnOCoi2h0f+Rv@mail.gmail.com>
	<4CD2A3EB.4030601@behnel.de>
	<AANLkTinHaapMihFN0vHaRD2o3xKVD6Ows-JjD7W_6mzx@mail.gmail.com>
	<4CD2B27E.6080805@behnel.de> <4CD2B585.8040601@behnel.de>
	<AANLkTik4qhsVhHhCQRyheKpDzoGGQD-q5r=bJUi2_VV_@mail.gmail.com>
	<AANLkTik1H1z-R23Kc6e7FgdMhGdX7JPX5iy=j33oS-J3@mail.gmail.com>
	<4CD660E2.9040800@behnel.de>
	<AANLkTimCCTQoSmgi91Xz_QVm_4uLBXkhd=fwGuzX6-Y7@mail.gmail.com>
	<AANLkTi=ZDvsiw5wWMM52sS1YcWL_3wZxZL-_4DMm_ORE@mail.gmail.com>
	<4CD80E5E.6020307@behnel.de> <4CDDACB9.7010903@behnel.de>
	<AANLkTimh+B3XTBgOu0=J=-WQnj14F6w10bLBe01-BgtA@mail.gmail.com>
	<AANLkTiny6S029uJvzenkMgLqfAsFT9g1M9eFK-y=g+Ys@mail.gmail.com>
	<4CDEA354.6020002@behnel.de> <4CDEA759.5030302@behnel.de>
	<4CDEA992.7010300@behnel.de>
	<AANLkTimmgWR6J92Sg33ECbZo1wAkF-HFYELADVhEEtMr@mail.gmail.com>
	<4CDECFB6.2080501@behnel.de>
Message-ID: <AANLkTimfSV=mJKmtZdxP3UN3H309mfNvTe43gjUdTv6F@mail.gmail.com>

2010/11/13 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 13.11.2010 16:35:
>> 2010/11/13 Stefan Behnel:
>>> Stefan Behnel, 13.11.2010 15:57:
>>>> Stefan Behnel, 13.11.2010 15:40:
>>>>> Vitja Makarov, 13.11.2010 14:15:
>>>>>> I've added patch in trac.
>>>>>
>>>>> Thanks, I'll give it a try.
>>>>
>>>> It lacks tests for passing **kwargs to the metaclass, and ISTM that this
>>>> feature can't work. Could you add the missing tests?
>>>
>>> More comments in the bug tracker.
>>>
>>
>> Ok, I'll add. What's that ISTM?
>
> It Seems To Me. :)
>

Yeah, that was completely broken, sorry ;)

From vinjvinj at gmail.com  Sat Nov 13 19:22:42 2010
From: vinjvinj at gmail.com (Vineet Jain)
Date: Sat, 13 Nov 2010 13:22:42 -0500
Subject: [Cython] compiler crash error
Message-ID: <AANLkTi=xGz-STEcKtDGZQkjdkujBZ=XAXiG_z+28kggP@mail.gmail.com>

I'm trying to use a struct defintion declared in another package:

import PriceDBCython as PriceDBCython
cimport PriceDBCython as PriceDBCython

def transform():
    cdef i, numRows
    cdef PriceDBCython.PriceRow * dataRowsCython


Here is the traceback:


def transform():
    cdef i, numRows
    cdef PriceDBCython.PriceRow * dataRowsCython
        ^
------------------------------------------------------------

/host/trading/pytrade/scripts/transformsqlitedb/transformdbcython.pyx:47:9:
'PriceRow' is not a type identifier

Error converting Pyrex file to C:
------------------------------------------------------------
...
    numpy.float32_t close
    numpy.float32_t volume
    numpy.float32_t bid
    numpy.float32_t offer

def transform():
^
------------------------------------------------------------

/host/trading/pytrade/scripts/transformsqlitedb/transformdbcython.pyx:45:0:
Compiler crash in IntroduceBufferAuxiliaryVars

ModuleNode.body = StatListNode(transformdbcython.pyx:3:0)
StatListNode.stats[22] = StatListNode(transformdbcython.pyx:45:0)
StatListNode.stats[0] = DefNode(transformdbcython.pyx:45:0,
    modifiers = [...]/0,
    name = u'transform',
    reqd_kw_flags_cname = '0')

Compiler crash traceback from this point on:
  File "Visitor.py", line 179, in
Cython.Compiler.Visitor.TreeVisitor.visitchild
(/tmp/easy_install-jnjvFc/Cython-0.13/Cython/Compiler/Visitor.c:3407)
  File "Visitor.py", line 28, in
Cython.Compiler.Visitor.BasicVisitor.visit
(/tmp/easy_install-jnjvFc/Cython-0.13/Cython/Compiler/Visitor.c:1178)
  File "/usr/local/lib/python2.6/dist-packages/Cython-0.13-py2.6-linux-x86_64.egg/Cython/Compiler/Buffer.py",
line 106, in visit_FuncDefNode
    self.handle_scope(node, node.local_scope)
  File "/usr/local/lib/python2.6/dist-packages/Cython-0.13-py2.6-linux-x86_64.egg/Cython/Compiler/Buffer.py",
line 61, in handle_scope
    if entry.type.dtype.is_ptr:
AttributeError: 'NoneType' object has no attribute 'is_ptr'
building 'transformdbcython' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
-Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c
transformdbcython.c -o build/temp.linux-x86_64-2.6/transformdbcython.o
transformdbcython.c:1: error: #error Do not use this file, it is the
result of a failed Cython compilation.
error: command 'gcc' failed with exit status 1

From robertwb at math.washington.edu  Sat Nov 13 21:51:01 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 13 Nov 2010 12:51:01 -0800
Subject: [Cython] What to do about large integer literals and constants?
In-Reply-To: <4CDE61BF.3020705@behnel.de>
References: <4CDD5C35.4080401@behnel.de> <4CDDA542.4040400@behnel.de>
	<4CDE61BF.3020705@behnel.de>
Message-ID: <AANLkTi=B6d3=wuubcwh0NiYaXk_XX7Crvx2DeZ0pVCg6@mail.gmail.com>

On Sat, Nov 13, 2010 at 2:00 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Stefan Behnel, 12.11.2010 21:36:
>> solution that basically leaves obvious C
>> literals (enum names or numeric literals ending with "U" or "LL") unchanged
>> and only checks that Python compatible literals are<= 32bit.
>
> On a related note, should we treat "123L" literals as explicit C long
> literals in -3 mode? After all, the "L" suffix is only ambiguous in Cython
> 2 mode (and clearly a Python literal suffix in Python 2 .py files).

Yes, that's kind of what I was thinking as well.

- Robert

From robertwb at math.washington.edu  Sat Nov 13 22:02:02 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 13 Nov 2010 13:02:02 -0800
Subject: [Cython] compiler crash error
In-Reply-To: <AANLkTi=xGz-STEcKtDGZQkjdkujBZ=XAXiG_z+28kggP@mail.gmail.com>
References: <AANLkTi=xGz-STEcKtDGZQkjdkujBZ=XAXiG_z+28kggP@mail.gmail.com>
Message-ID: <AANLkTikspu_dBTX9AUjR+vmoe96ZXMLdPPTiVcFNt-xZ@mail.gmail.com>

On Sat, Nov 13, 2010 at 10:22 AM, Vineet Jain <vinjvinj at gmail.com> wrote:
> I'm trying to use a struct defintion declared in another package:
>
> import PriceDBCython as PriceDBCython
> cimport PriceDBCython as PriceDBCython
>
> def transform():
> ? ?cdef i, numRows
> ? ?cdef PriceDBCython.PriceRow * dataRowsCython
>
>
> Here is the traceback:
>
>
> def transform():
> ? ?cdef i, numRows
> ? ?cdef PriceDBCython.PriceRow * dataRowsCython
> ? ? ? ?^
> ------------------------------------------------------------
>
> /host/trading/pytrade/scripts/transformsqlitedb/transformdbcython.pyx:47:9:
> 'PriceRow' is not a type identifier
>
> Error converting Pyrex file to C:
> ------------------------------------------------------------
> ...
> ? ?numpy.float32_t close
> ? ?numpy.float32_t volume
> ? ?numpy.float32_t bid
> ? ?numpy.float32_t offer
>
> def transform():
> ^
> ------------------------------------------------------------
>
> /host/trading/pytrade/scripts/transformsqlitedb/transformdbcython.pyx:45:0:
> Compiler crash in IntroduceBufferAuxiliaryVars
>
> ModuleNode.body = StatListNode(transformdbcython.pyx:3:0)
> StatListNode.stats[22] = StatListNode(transformdbcython.pyx:45:0)
> StatListNode.stats[0] = DefNode(transformdbcython.pyx:45:0,
> ? ?modifiers = [...]/0,
> ? ?name = u'transform',
> ? ?reqd_kw_flags_cname = '0')
>
> Compiler crash traceback from this point on:
> ?File "Visitor.py", line 179, in
> Cython.Compiler.Visitor.TreeVisitor.visitchild
> (/tmp/easy_install-jnjvFc/Cython-0.13/Cython/Compiler/Visitor.c:3407)
> ?File "Visitor.py", line 28, in
> Cython.Compiler.Visitor.BasicVisitor.visit
> (/tmp/easy_install-jnjvFc/Cython-0.13/Cython/Compiler/Visitor.c:1178)
> ?File "/usr/local/lib/python2.6/dist-packages/Cython-0.13-py2.6-linux-x86_64.egg/Cython/Compiler/Buffer.py",
> line 106, in visit_FuncDefNode
> ? ?self.handle_scope(node, node.local_scope)
> ?File "/usr/local/lib/python2.6/dist-packages/Cython-0.13-py2.6-linux-x86_64.egg/Cython/Compiler/Buffer.py",
> line 61, in handle_scope
> ? ?if entry.type.dtype.is_ptr:
> AttributeError: 'NoneType' object has no attribute 'is_ptr'
> building 'transformdbcython' extension
> gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
> -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c
> transformdbcython.c -o build/temp.linux-x86_64-2.6/transformdbcython.o
> transformdbcython.c:1: error: #error Do not use this file, it is the
> result of a failed Cython compilation.
> error: command 'gcc' failed with exit status 1
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

Thanks for the report. Can you try placing the cimport first? (Though
it shouldn't matter, currently it does.) In either case, it shouldn't
be crashing--we'll look into this.

- Robert

From vinjvinj at gmail.com  Sun Nov 14 05:45:21 2010
From: vinjvinj at gmail.com (Vineet Jain)
Date: Sat, 13 Nov 2010 23:45:21 -0500
Subject: [Cython] Another compiler crash.
Message-ID: <AANLkTinind1y95uFHfb8D1bLD4JahxH8J=C6bii4AaH7@mail.gmail.com>

PriceRow is a struct declared in a pxd file. I'm not including it
since I assumed that declarations from the pxd file are automatically
included.

Vineet


l = fcntllock()

def insertSorted(h5pyFile, timeFormatStr, dataRowsPython):
    cdef object[PriceRow] dataRows=dataRowsPython
    cdef PriceRow * dataRowsCython = <PriceRow *>&(dataRows[0])
        ^
------------------------------------------------------------

PriceDBCython.pyx:40:9: 'PriceRow' is not a type identifier

Error converting Pyrex file to C:
------------------------------------------------------------
...

dbconfig = DeploymentConfig.db
from pytrade.util.misc import fcntllock
l = fcntllock()

def insertSorted(h5pyFile, timeFormatStr, dataRowsPython):
^
------------------------------------------------------------

/host/trading/pytrade/modules/pytrade/db/h5py/PriceDBCython.pyx:38:0:
Compiler crash in IntroduceBufferAuxiliaryVars

ModuleNode.body = StatListNode(PriceDBCython.pyx:20:0)
StatListNode.stats[15] = StatListNode(PriceDBCython.pyx:38:0)
StatListNode.stats[0] = DefNode(PriceDBCython.pyx:38:0,
    modifiers = [...]/0,
    name = u'insertSorted',
    num_required_args = 3,
    reqd_kw_flags_cname = '0')

Compiler crash traceback from this point on:
  File "Visitor.py", line 179, in
Cython.Compiler.Visitor.TreeVisitor.visitchild
(/tmp/easy_install-jnjvFc/Cython-0.13/Cython/Compiler/Visitor.c:3407)
  File "Visitor.py", line 28, in
Cython.Compiler.Visitor.BasicVisitor.visit
(/tmp/easy_install-jnjvFc/Cython-0.13/Cython/Compiler/Visitor.c:1178)
  File "/usr/local/lib/python2.6/dist-packages/Cython-0.13-py2.6-linux-x86_64.egg/Cython/Compiler/Buffer.py",
line 106, in visit_FuncDefNode
    self.handle_scope(node, node.local_scope)
  File "/usr/local/lib/python2.6/dist-packages/Cython-0.13-py2.6-linux-x86_64.egg/Cython/Compiler/Buffer.py",
line 61, in handle_scope
    if entry.type.dtype.is_ptr:
AttributeError: 'NoneType' object has no attribute 'is_ptr'
building 'modules.pytrade.db.h5py.PriceDBCython' extension

From robertwb at math.washington.edu  Sun Nov 14 06:25:37 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sat, 13 Nov 2010 21:25:37 -0800
Subject: [Cython] Another compiler crash.
In-Reply-To: <AANLkTinind1y95uFHfb8D1bLD4JahxH8J=C6bii4AaH7@mail.gmail.com>
References: <AANLkTinind1y95uFHfb8D1bLD4JahxH8J=C6bii4AaH7@mail.gmail.com>
Message-ID: <AANLkTinBnHAstHO4gKYEHyfZtnUtBAsj+kHN0PVWpqKM@mail.gmail.com>

On Sat, Nov 13, 2010 at 8:45 PM, Vineet Jain <vinjvinj at gmail.com> wrote:
> PriceRow is a struct declared in a pxd file. I'm not including it
> since I assumed that declarations from the pxd file are automatically
> included.

The error below indicates that it's not finding it--could you send us
enough of a source snippet to reproduce this error?

> l = fcntllock()
>
> def insertSorted(h5pyFile, timeFormatStr, dataRowsPython):
> ? ?cdef object[PriceRow] dataRows=dataRowsPython
> ? ?cdef PriceRow * dataRowsCython = <PriceRow *>&(dataRows[0])
> ? ? ? ?^
> ------------------------------------------------------------
>
> PriceDBCython.pyx:40:9: 'PriceRow' is not a type identifier
>
> Error converting Pyrex file to C:
> ------------------------------------------------------------
> ...
>
> dbconfig = DeploymentConfig.db
> from pytrade.util.misc import fcntllock
> l = fcntllock()
>
> def insertSorted(h5pyFile, timeFormatStr, dataRowsPython):
> ^
> ------------------------------------------------------------
>
> /host/trading/pytrade/modules/pytrade/db/h5py/PriceDBCython.pyx:38:0:
> Compiler crash in IntroduceBufferAuxiliaryVars
>
> ModuleNode.body = StatListNode(PriceDBCython.pyx:20:0)
> StatListNode.stats[15] = StatListNode(PriceDBCython.pyx:38:0)
> StatListNode.stats[0] = DefNode(PriceDBCython.pyx:38:0,
> ? ?modifiers = [...]/0,
> ? ?name = u'insertSorted',
> ? ?num_required_args = 3,
> ? ?reqd_kw_flags_cname = '0')
>
> Compiler crash traceback from this point on:
> ?File "Visitor.py", line 179, in
> Cython.Compiler.Visitor.TreeVisitor.visitchild
> (/tmp/easy_install-jnjvFc/Cython-0.13/Cython/Compiler/Visitor.c:3407)
> ?File "Visitor.py", line 28, in
> Cython.Compiler.Visitor.BasicVisitor.visit
> (/tmp/easy_install-jnjvFc/Cython-0.13/Cython/Compiler/Visitor.c:1178)
> ?File "/usr/local/lib/python2.6/dist-packages/Cython-0.13-py2.6-linux-x86_64.egg/Cython/Compiler/Buffer.py",
> line 106, in visit_FuncDefNode
> ? ?self.handle_scope(node, node.local_scope)
> ?File "/usr/local/lib/python2.6/dist-packages/Cython-0.13-py2.6-linux-x86_64.egg/Cython/Compiler/Buffer.py",
> line 61, in handle_scope
> ? ?if entry.type.dtype.is_ptr:
> AttributeError: 'NoneType' object has no attribute 'is_ptr'
> building 'modules.pytrade.db.h5py.PriceDBCython' extension
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From vitja.makarov at gmail.com  Sun Nov 14 07:23:13 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sun, 14 Nov 2010 09:23:13 +0300
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTimfSV=mJKmtZdxP3UN3H309mfNvTe43gjUdTv6F@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<AANLkTimOXxppAQ6Fb5DSCTyRoY9ZJghhV5=sM3i0N6ic@mail.gmail.com>
	<4CD29CFD.7030309@behnel.de>
	<AANLkTinP8Osz1YKkLyuDhPT=vzyc6iscnOCoi2h0f+Rv@mail.gmail.com>
	<4CD2A3EB.4030601@behnel.de>
	<AANLkTinHaapMihFN0vHaRD2o3xKVD6Ows-JjD7W_6mzx@mail.gmail.com>
	<4CD2B27E.6080805@behnel.de> <4CD2B585.8040601@behnel.de>
	<AANLkTik4qhsVhHhCQRyheKpDzoGGQD-q5r=bJUi2_VV_@mail.gmail.com>
	<AANLkTik1H1z-R23Kc6e7FgdMhGdX7JPX5iy=j33oS-J3@mail.gmail.com>
	<4CD660E2.9040800@behnel.de>
	<AANLkTimCCTQoSmgi91Xz_QVm_4uLBXkhd=fwGuzX6-Y7@mail.gmail.com>
	<AANLkTi=ZDvsiw5wWMM52sS1YcWL_3wZxZL-_4DMm_ORE@mail.gmail.com>
	<4CD80E5E.6020307@behnel.de> <4CDDACB9.7010903@behnel.de>
	<AANLkTimh+B3XTBgOu0=J=-WQnj14F6w10bLBe01-BgtA@mail.gmail.com>
	<AANLkTiny6S029uJvzenkMgLqfAsFT9g1M9eFK-y=g+Ys@mail.gmail.com>
	<4CDEA354.6020002@behnel.de> <4CDEA759.5030302@behnel.de>
	<4CDEA992.7010300@behnel.de>
	<AANLkTimmgWR6J92Sg33ECbZo1wAkF-HFYELADVhEEtMr@mail.gmail.com>
	<4CDECFB6.2080501@behnel.de>
	<AANLkTimfSV=mJKmtZdxP3UN3H309mfNvTe43gjUdTv6F@mail.gmail.com>
Message-ID: <AANLkTi=SSS+CY0h0rob9gnc88YYf7_-a8KY3NX5n_AuK@mail.gmail.com>

2010/11/13 Vitja Makarov <vitja.makarov at gmail.com>:
> 2010/11/13 Stefan Behnel <stefan_ml at behnel.de>:
>> Vitja Makarov, 13.11.2010 16:35:
>>> 2010/11/13 Stefan Behnel:
>>>> Stefan Behnel, 13.11.2010 15:57:
>>>>> Stefan Behnel, 13.11.2010 15:40:
>>>>>> Vitja Makarov, 13.11.2010 14:15:
>>>>>>> I've added patch in trac.
>>>>>>
>>>>>> Thanks, I'll give it a try.
>>>>>
>>>>> It lacks tests for passing **kwargs to the metaclass, and ISTM that this
>>>>> feature can't work. Could you add the missing tests?
>>>>
>>>> More comments in the bug tracker.
>>>>
>>>
>>> Ok, I'll add. What's that ISTM?
>>
>> It Seems To Me. :)
>>
>
> Yeah, that was completely broken, sorry ;)
>

Is it safe to use keyword_args in KeywordArgsNode instead of copying.

vitja.

From stefan_ml at behnel.de  Sun Nov 14 08:18:06 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 14 Nov 2010 08:18:06 +0100
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTi=SSS+CY0h0rob9gnc88YYf7_-a8KY3NX5n_AuK@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>	<4CD2A3EB.4030601@behnel.de>	<AANLkTinHaapMihFN0vHaRD2o3xKVD6Ows-JjD7W_6mzx@mail.gmail.com>	<4CD2B27E.6080805@behnel.de>
	<4CD2B585.8040601@behnel.de>	<AANLkTik4qhsVhHhCQRyheKpDzoGGQD-q5r=bJUi2_VV_@mail.gmail.com>	<AANLkTik1H1z-R23Kc6e7FgdMhGdX7JPX5iy=j33oS-J3@mail.gmail.com>	<4CD660E2.9040800@behnel.de>	<AANLkTimCCTQoSmgi91Xz_QVm_4uLBXkhd=fwGuzX6-Y7@mail.gmail.com>	<AANLkTi=ZDvsiw5wWMM52sS1YcWL_3wZxZL-_4DMm_ORE@mail.gmail.com>	<4CD80E5E.6020307@behnel.de>
	<4CDDACB9.7010903@behnel.de>	<AANLkTimh+B3XTBgOu0=J=-WQnj14F6w10bLBe01-BgtA@mail.gmail.com>	<AANLkTiny6S029uJvzenkMgLqfAsFT9g1M9eFK-y=g+Ys@mail.gmail.com>	<4CDEA354.6020002@behnel.de>
	<4CDEA759.5030302@behnel.de>	<4CDEA992.7010300@behnel.de>	<AANLkTimmgWR6J92Sg33ECbZo1wAkF-HFYELADVhEEtMr@mail.gmail.com>	<4CDECFB6.2080501@behnel.de>	<AANLkTimfSV=mJKmtZdxP3UN3H309mfNvTe43gjUdTv6F@mail.gmail.com>
	<AANLkTi=SSS+CY0h0rob9gnc88YYf7_-a8KY3NX5n_AuK@mail.gmail.com>
Message-ID: <4CDF8D2E.4000405@behnel.de>

Vitja Makarov, 14.11.2010 07:23:
> Is it safe to use keyword_args in KeywordArgsNode instead of copying.

It's a dict that gets created internally by Cython, so the only case that 
can go wrong is when it's passed into user code and reused afterwards in a 
user visible way. Here, it can get passed into user code more than once, so 
the question is if CPython propagates changes to it or not. Cython should 
behave the same.

You can test for it being safe by defining a metaclass with __prepare__ in 
both Python (doctest code) and Cython, modify the keywords dict in all 
possible places, and check if there is a difference in behaviour between 
the two.

Stefan

From vitja.makarov at gmail.com  Sun Nov 14 08:56:15 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sun, 14 Nov 2010 10:56:15 +0300
Subject: [Cython] metaclasses
In-Reply-To: <4CDF8D2E.4000405@behnel.de>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CD2A3EB.4030601@behnel.de>
	<AANLkTinHaapMihFN0vHaRD2o3xKVD6Ows-JjD7W_6mzx@mail.gmail.com>
	<4CD2B27E.6080805@behnel.de> <4CD2B585.8040601@behnel.de>
	<AANLkTik4qhsVhHhCQRyheKpDzoGGQD-q5r=bJUi2_VV_@mail.gmail.com>
	<AANLkTik1H1z-R23Kc6e7FgdMhGdX7JPX5iy=j33oS-J3@mail.gmail.com>
	<4CD660E2.9040800@behnel.de>
	<AANLkTimCCTQoSmgi91Xz_QVm_4uLBXkhd=fwGuzX6-Y7@mail.gmail.com>
	<AANLkTi=ZDvsiw5wWMM52sS1YcWL_3wZxZL-_4DMm_ORE@mail.gmail.com>
	<4CD80E5E.6020307@behnel.de> <4CDDACB9.7010903@behnel.de>
	<AANLkTimh+B3XTBgOu0=J=-WQnj14F6w10bLBe01-BgtA@mail.gmail.com>
	<AANLkTiny6S029uJvzenkMgLqfAsFT9g1M9eFK-y=g+Ys@mail.gmail.com>
	<4CDEA354.6020002@behnel.de> <4CDEA759.5030302@behnel.de>
	<4CDEA992.7010300@behnel.de>
	<AANLkTimmgWR6J92Sg33ECbZo1wAkF-HFYELADVhEEtMr@mail.gmail.com>
	<4CDECFB6.2080501@behnel.de>
	<AANLkTimfSV=mJKmtZdxP3UN3H309mfNvTe43gjUdTv6F@mail.gmail.com>
	<AANLkTi=SSS+CY0h0rob9gnc88YYf7_-a8KY3NX5n_AuK@mail.gmail.com>
	<4CDF8D2E.4000405@behnel.de>
Message-ID: <AANLkTi=X9CD+RTUP4RXWrpjpQ=xnHMbo7RrYU_gid7Py@mail.gmail.com>

2010/11/14 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 14.11.2010 07:23:
>> Is it safe to use keyword_args in KeywordArgsNode instead of copying.
>
> It's a dict that gets created internally by Cython, so the only case that
> can go wrong is when it's passed into user code and reused afterwards in a
> user visible way. Here, it can get passed into user code more than once, so
> the question is if CPython propagates changes to it or not. Cython should
> behave the same.
>
> You can test for it being safe by defining a metaclass with __prepare__ in
> both Python (doctest code) and Cython, modify the keywords dict in all
> possible places, and check if there is a difference in behaviour between
> the two.
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>


That seems to be okay. I've added new patch on trac it includes:
 * KeywordArgNode optimization
 * one more test case too see that kwargs aren't changed

vitja.

From vinjvinj at gmail.com  Sun Nov 14 14:31:11 2010
From: vinjvinj at gmail.com (Vineet Jain)
Date: Sun, 14 Nov 2010 08:31:11 -0500
Subject: [Cython] Another compiler crash.
In-Reply-To: <AANLkTinBnHAstHO4gKYEHyfZtnUtBAsj+kHN0PVWpqKM@mail.gmail.com>
References: <AANLkTinind1y95uFHfb8D1bLD4JahxH8J=C6bii4AaH7@mail.gmail.com>
	<AANLkTinBnHAstHO4gKYEHyfZtnUtBAsj+kHN0PVWpqKM@mail.gmail.com>
Message-ID: <AANLkTimLznpw0=rn+8E+HdfESqc+Px9XXpKeZ9o=Ai8T@mail.gmail.com>

Thanks. After playing with it for some time, the error went away after
I replaced

cimport StructModulePxd

with

cimport <path.to.pxdfile.>StructModulePxd

It seems that the compiler will not find a pxd file with a different
name (and no corresponding pyx file) is in the same directory as where
you are trying to import it. If I replace the cimport with a fully
qualified path then it works.

Vineet




On Sun, Nov 14, 2010 at 12:25 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Sat, Nov 13, 2010 at 8:45 PM, Vineet Jain <vinjvinj at gmail.com> wrote:
>> PriceRow is a struct declared in a pxd file. I'm not including it
>> since I assumed that declarations from the pxd file are automatically
>> included.
>
> The error below indicates that it's not finding it--could you send us
> enough of a source snippet to reproduce this error?
>
>> l = fcntllock()
>>
>> def insertSorted(h5pyFile, timeFormatStr, dataRowsPython):
>> ? ?cdef object[PriceRow] dataRows=dataRowsPython
>> ? ?cdef PriceRow * dataRowsCython = <PriceRow *>&(dataRows[0])
>> ? ? ? ?^
>> ------------------------------------------------------------
>>
>> PriceDBCython.pyx:40:9: 'PriceRow' is not a type identifier
>>
>> Error converting Pyrex file to C:
>> ------------------------------------------------------------
>> ...
>>
>> dbconfig = DeploymentConfig.db
>> from pytrade.util.misc import fcntllock
>> l = fcntllock()
>>
>> def insertSorted(h5pyFile, timeFormatStr, dataRowsPython):
>> ^
>> ------------------------------------------------------------
>>
>> /host/trading/pytrade/modules/pytrade/db/h5py/PriceDBCython.pyx:38:0:
>> Compiler crash in IntroduceBufferAuxiliaryVars
>>
>> ModuleNode.body = StatListNode(PriceDBCython.pyx:20:0)
>> StatListNode.stats[15] = StatListNode(PriceDBCython.pyx:38:0)
>> StatListNode.stats[0] = DefNode(PriceDBCython.pyx:38:0,
>> ? ?modifiers = [...]/0,
>> ? ?name = u'insertSorted',
>> ? ?num_required_args = 3,
>> ? ?reqd_kw_flags_cname = '0')
>>
>> Compiler crash traceback from this point on:
>> ?File "Visitor.py", line 179, in
>> Cython.Compiler.Visitor.TreeVisitor.visitchild
>> (/tmp/easy_install-jnjvFc/Cython-0.13/Cython/Compiler/Visitor.c:3407)
>> ?File "Visitor.py", line 28, in
>> Cython.Compiler.Visitor.BasicVisitor.visit
>> (/tmp/easy_install-jnjvFc/Cython-0.13/Cython/Compiler/Visitor.c:1178)
>> ?File "/usr/local/lib/python2.6/dist-packages/Cython-0.13-py2.6-linux-x86_64.egg/Cython/Compiler/Buffer.py",
>> line 106, in visit_FuncDefNode
>> ? ?self.handle_scope(node, node.local_scope)
>> ?File "/usr/local/lib/python2.6/dist-packages/Cython-0.13-py2.6-linux-x86_64.egg/Cython/Compiler/Buffer.py",
>> line 61, in handle_scope
>> ? ?if entry.type.dtype.is_ptr:
>> AttributeError: 'NoneType' object has no attribute 'is_ptr'
>> building 'modules.pytrade.db.h5py.PriceDBCython' extension
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From stefan_ml at behnel.de  Sun Nov 14 16:12:35 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 14 Nov 2010 16:12:35 +0100
Subject: [Cython] Another compiler crash.
In-Reply-To: <AANLkTimLznpw0=rn+8E+HdfESqc+Px9XXpKeZ9o=Ai8T@mail.gmail.com>
References: <AANLkTinind1y95uFHfb8D1bLD4JahxH8J=C6bii4AaH7@mail.gmail.com>	<AANLkTinBnHAstHO4gKYEHyfZtnUtBAsj+kHN0PVWpqKM@mail.gmail.com>
	<AANLkTimLznpw0=rn+8E+HdfESqc+Px9XXpKeZ9o=Ai8T@mail.gmail.com>
Message-ID: <4CDFFC63.6030203@behnel.de>

Vineet Jain, 14.11.2010 14:31:
> Thanks. After playing with it for some time, the error went away after
> I replaced
>
> cimport StructModulePxd
>
> with
>
> cimport<path.to.pxdfile.>StructModulePxd
>
> It seems that the compiler will not find a pxd file with a different
> name (and no corresponding pyx file) is in the same directory as where
> you are trying to import it. If I replace the cimport with a fully
> qualified path then it works.

Maybe you forgot to add an __init__.py file to the package that holds your 
source files?

Stefan

From vinjvinj at gmail.com  Sun Nov 14 16:30:59 2010
From: vinjvinj at gmail.com (Vineet Jain)
Date: Sun, 14 Nov 2010 10:30:59 -0500
Subject: [Cython] Another compiler crash.
In-Reply-To: <4CDFFC63.6030203@behnel.de>
References: <AANLkTinind1y95uFHfb8D1bLD4JahxH8J=C6bii4AaH7@mail.gmail.com>
	<AANLkTinBnHAstHO4gKYEHyfZtnUtBAsj+kHN0PVWpqKM@mail.gmail.com>
	<AANLkTimLznpw0=rn+8E+HdfESqc+Px9XXpKeZ9o=Ai8T@mail.gmail.com>
	<4CDFFC63.6030203@behnel.de>
Message-ID: <AANLkTi=AGW9Fw6J=nzabrTCbaFQJ0eTLwDEKuaE2sqP-@mail.gmail.com>

No. There is __init__.py in the folder. Otherwise the fully qualified
path would not work either since it is no longer a python module.

Vineet


On Sun, Nov 14, 2010 at 10:12 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Vineet Jain, 14.11.2010 14:31:
>> Thanks. After playing with it for some time, the error went away after
>> I replaced
>>
>> cimport StructModulePxd
>>
>> with
>>
>> cimport<path.to.pxdfile.>StructModulePxd
>>
>> It seems that the compiler will not find a pxd file with a different
>> name (and no corresponding pyx file) is in the same directory as where
>> you are trying to import it. If I replace the cimport with a fully
>> qualified path then it works.
>
> Maybe you forgot to add an __init__.py file to the package that holds your
> source files?
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From stefan_ml at behnel.de  Sun Nov 14 16:40:49 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 14 Nov 2010 16:40:49 +0100
Subject: [Cython] Another compiler crash.
In-Reply-To: <AANLkTi=AGW9Fw6J=nzabrTCbaFQJ0eTLwDEKuaE2sqP-@mail.gmail.com>
References: <AANLkTinind1y95uFHfb8D1bLD4JahxH8J=C6bii4AaH7@mail.gmail.com>	<AANLkTinBnHAstHO4gKYEHyfZtnUtBAsj+kHN0PVWpqKM@mail.gmail.com>	<AANLkTimLznpw0=rn+8E+HdfESqc+Px9XXpKeZ9o=Ai8T@mail.gmail.com>	<4CDFFC63.6030203@behnel.de>
	<AANLkTi=AGW9Fw6J=nzabrTCbaFQJ0eTLwDEKuaE2sqP-@mail.gmail.com>
Message-ID: <4CE00301.5060405@behnel.de>

[fixed order of replies]

Vineet Jain, 14.11.2010 16:30:
> On Sun, Nov 14, 2010 at 10:12 AM, Stefan Behnel wrote:
>> Vineet Jain, 14.11.2010 14:31:
>>> Thanks. After playing with it for some time, the error went away after
>>> I replaced
>>>
>>> cimport StructModulePxd
>>>
>>> with
>>>
>>> cimport<path.to.pxdfile.>StructModulePxd
>>>
>>> It seems that the compiler will not find a pxd file with a different
>>> name (and no corresponding pyx file) is in the same directory as where
>>> you are trying to import it. If I replace the cimport with a fully
>>> qualified path then it works.
>>
>> Maybe you forgot to add an __init__.py file to the package that holds your
>> source files?
>>
> No. There is __init__.py in the folder. Otherwise the fully qualified
> path would not work either since it is no longer a python module.

Then I suggest you use the standard Python package layout for your code.

Stefan

From vinjvinj at gmail.com  Sun Nov 14 17:05:35 2010
From: vinjvinj at gmail.com (Vineet Jain)
Date: Sun, 14 Nov 2010 11:05:35 -0500
Subject: [Cython] Another compiler crash.
In-Reply-To: <4CE00301.5060405@behnel.de>
References: <AANLkTinind1y95uFHfb8D1bLD4JahxH8J=C6bii4AaH7@mail.gmail.com>
	<AANLkTinBnHAstHO4gKYEHyfZtnUtBAsj+kHN0PVWpqKM@mail.gmail.com>
	<AANLkTimLznpw0=rn+8E+HdfESqc+Px9XXpKeZ9o=Ai8T@mail.gmail.com>
	<4CDFFC63.6030203@behnel.de>
	<AANLkTi=AGW9Fw6J=nzabrTCbaFQJ0eTLwDEKuaE2sqP-@mail.gmail.com>
	<4CE00301.5060405@behnel.de>
Message-ID: <AANLkTimj-iK8SEFvt3Z8RVw9peNdw+eyTVgfXXSt-LtJ@mail.gmail.com>

> Then I suggest you use the standard Python package layout for your code.

That is what I'm going to do from now. As a side note, even with still
a lot of non optimized python code I'm getting a 5x speed up. So cool.

One quick question. I have a bunch of arrays which are of mixed types
with no names ("i4,f4,f4"). I don't need to access them by names and
am fine with getting them as array[0,1]. Do I always have to create a
cython struct definition for them and access the arrays by name in
cython?

Thanks,

Vineet

From vitja.makarov at gmail.com  Mon Nov 15 09:12:45 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Mon, 15 Nov 2010 11:12:45 +0300
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTi=X9CD+RTUP4RXWrpjpQ=xnHMbo7RrYU_gid7Py@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
	<4CD2A3EB.4030601@behnel.de>
	<AANLkTinHaapMihFN0vHaRD2o3xKVD6Ows-JjD7W_6mzx@mail.gmail.com>
	<4CD2B27E.6080805@behnel.de> <4CD2B585.8040601@behnel.de>
	<AANLkTik4qhsVhHhCQRyheKpDzoGGQD-q5r=bJUi2_VV_@mail.gmail.com>
	<AANLkTik1H1z-R23Kc6e7FgdMhGdX7JPX5iy=j33oS-J3@mail.gmail.com>
	<4CD660E2.9040800@behnel.de>
	<AANLkTimCCTQoSmgi91Xz_QVm_4uLBXkhd=fwGuzX6-Y7@mail.gmail.com>
	<AANLkTi=ZDvsiw5wWMM52sS1YcWL_3wZxZL-_4DMm_ORE@mail.gmail.com>
	<4CD80E5E.6020307@behnel.de> <4CDDACB9.7010903@behnel.de>
	<AANLkTimh+B3XTBgOu0=J=-WQnj14F6w10bLBe01-BgtA@mail.gmail.com>
	<AANLkTiny6S029uJvzenkMgLqfAsFT9g1M9eFK-y=g+Ys@mail.gmail.com>
	<4CDEA354.6020002@behnel.de> <4CDEA759.5030302@behnel.de>
	<4CDEA992.7010300@behnel.de>
	<AANLkTimmgWR6J92Sg33ECbZo1wAkF-HFYELADVhEEtMr@mail.gmail.com>
	<4CDECFB6.2080501@behnel.de>
	<AANLkTimfSV=mJKmtZdxP3UN3H309mfNvTe43gjUdTv6F@mail.gmail.com>
	<AANLkTi=SSS+CY0h0rob9gnc88YYf7_-a8KY3NX5n_AuK@mail.gmail.com>
	<4CDF8D2E.4000405@behnel.de>
	<AANLkTi=X9CD+RTUP4RXWrpjpQ=xnHMbo7RrYU_gid7Py@mail.gmail.com>
Message-ID: <AANLkTimCJisGxBdNt8Ho5U=KvSRyT2OjPxFWKVmzkAsq@mail.gmail.com>

2010/11/14 Vitja Makarov <vitja.makarov at gmail.com>:
> 2010/11/14 Stefan Behnel <stefan_ml at behnel.de>:
>> Vitja Makarov, 14.11.2010 07:23:
>>> Is it safe to use keyword_args in KeywordArgsNode instead of copying.
>>
>> It's a dict that gets created internally by Cython, so the only case that
>> can go wrong is when it's passed into user code and reused afterwards in a
>> user visible way. Here, it can get passed into user code more than once, so
>> the question is if CPython propagates changes to it or not. Cython should
>> behave the same.
>>
>> You can test for it being safe by defining a metaclass with __prepare__ in
>> both Python (doctest code) and Cython, modify the keywords dict in all
>> possible places, and check if there is a difference in behaviour between
>> the two.
>>
>> Stefan
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
> That seems to be okay. I've added new patch on trac it includes:
> ?* KeywordArgNode optimization
> ?* one more test case too see that kwargs aren't changed
>
> vitja.
>

I also want to use KeywordArgsNode in GeneralCallNode.

KeywordArgsNode should be extended to check for multiple values in
keyword_args and starstar_args,
not sure that is a good idea, because of speed. But cpython works this way.

This code works in cython but fails in python:
def foo(**kwargs):
    return kwargs
foo(foo=1, **{'foo': 1})

-- 
vitja.

From dsdale24 at gmail.com  Mon Nov 15 18:04:39 2010
From: dsdale24 at gmail.com (Darren Dale)
Date: Mon, 15 Nov 2010 12:04:39 -0500
Subject: [Cython] possible bug report: combination of with/in
Message-ID: <AANLkTim_GhrXyTPi1zwG9bLWb6QpbVfcLfo7zHMWxqhK@mail.gmail.com>

I can cythonize this code:

def test():
    sentinal = '3'
    with open('foo.txt', 'w') as f:
        sentinal == '1' or sentinal == '2'

or this:

def test():
    sentinal = '3'
    with open('foo.txt', 'w') as f:
        pass
    sentinal in ('1', '2')

but not this:

def test():
    sentinal = '3'
    with open('foo.txt', 'w') as f:
        sentinal in ('1', '2')


$ cython test_open.pyx

Error converting Pyrex file to C:
------------------------------------------------------------
...
def test():
    sentinal = '3'
    with open('foo.txt', 'w') as f:
   ^
------------------------------------------------------------

/Users/darren/temp/test/test_open.pyx:3:4: Compiler crash in
AnalyseExpressionsTransform

ModuleNode.body = StatListNode(test_open.pyx:1:0)
StatListNode.stats[0] = DefNode(test_open.pyx:1:0,
    modifiers = [...]/0,
    name = 'test',
    reqd_kw_flags_cname = '0')
File 'Nodes.py', line 343, in analyse_expressions:
StatListNode(test_open.pyx:2:4)
File 'Nodes.py', line 343, in analyse_expressions:
StatListNode(test_open.pyx:3:4)
File 'Nodes.py', line 4914, in analyse_expressions:
TryFinallyStatNode(test_open.pyx:3:4,
    preserve_exception = 1)
File 'Nodes.py', line 343, in analyse_expressions:
StatListNode(test_open.pyx:3:4)
File 'Nodes.py', line 4608, in analyse_expressions:
TryExceptStatNode(test_open.pyx:3:4)
File 'Nodes.py', line 343, in analyse_expressions:
StatListNode(test_open.pyx:3:4)
File 'Nodes.py', line 343, in analyse_expressions:
StatListNode(test_open.pyx:3:4)
File 'Nodes.py', line 3243, in analyse_expressions:
ExprStatNode(test_open.pyx:3:4)
File 'ExprNodes.py', line 308, in analyse_expressions:
EvalWithTempExprNode(test_open.pyx:3:4,
    use_managed_ref = True)
File 'UtilNodes.py', line 239, in analyse_types:
EvalWithTempExprNode(test_open.pyx:3:4,
    use_managed_ref = True)
File 'ExprNodes.py', line 5664, in analyse_types:
BoolBinopNode(test_open.pyx:3:4,
    operator = 'or',
    use_managed_ref = True)
File 'ExprNodes.py', line 4892, in analyse_types:
TypecastNode(test_open.pyx:3:4,
    use_managed_ref = True)
File 'ExprNodes.py', line 6138, in analyse_types:
PrimaryCmpNode(test_open.pyx:3:4,
    operator = '==',
    use_managed_ref = True)
File 'ExprNodes.py', line 5840, in is_cpp_comparison:
PrimaryCmpNode(test_open.pyx:3:4,
    operator = '==',
    use_managed_ref = True)

Compiler crash traceback from this point on:
  File "/Users/darren/.local/lib/python3.1/site-packages/Cython/Compiler/ExprNodes.py",
line 5841, in is_cpp_comparison
    return self.operand1.type.is_cpp_class or self.operand2.type.is_cpp_class
AttributeError: 'NoneType' object has no attribute 'is_cpp_class'

From vinjvinj at gmail.com  Mon Nov 15 20:23:56 2010
From: vinjvinj at gmail.com (Vineet Jain)
Date: Mon, 15 Nov 2010 14:23:56 -0500
Subject: [Cython] small test program works in python but raises exception
	through cython
Message-ID: <AANLkTi=hABQoLkTemxGowqU1Q2Frxq_eKYjV=j_iD6ky@mail.gmail.com>

I posted this on cython-users and got not response. Any idea what is
wrong here? short test program is below:


#######test.py and testcython.pyx
import os
import h5py
import numpy
os.popen("rm myfile.hdf5")
f = h5py.File('myfile.hdf5')
dset = f.create_dataset("MyDataset", (10, 10), 'i')
dset[...] = 42
indexDest = f["MyDataset"]
index = numpy.zeros(dtype=indexDest.dtype, shape=indexDest.shape)
index = indexDest[:]
f.close()
print index



>>> import testcython

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "testcython.pyx", line 13, in init
modules.pytrade.db.h5py.testcython
(modules/pytrade/db/h5py/testcython.c:725)
    index = indexDest[:]
  File "/usr/lib/python2.6/dist-packages/h5py/highlevel.py", line
1200, in __getitem__
    selection = sel.select(self.shape, args, dsid=self.id)
  File "/usr/lib/python2.6/dist-packages/h5py/selections.py", line 94,
in select
    sel[args]
  File "/usr/lib/python2.6/dist-packages/h5py/selections.py", line
261, in __getitem__
    start, count, step, scalar = _handle_simple(self.shape,args)
  File "/usr/lib/python2.6/dist-packages/h5py/selections.py", line
512, in _handle_simple
    x,y,z = _translate_slice(arg, length)
  File "/usr/lib/python2.6/dist-packages/h5py/selections.py", line
564, in _translate_slice
    raise ValueError("Stop index %s out of range (1-%d)" % (stop, length))
ValueError: Stop index 9223372036854775807 out of range (1-10)




>>> import test

[[42 42 42 42 42 42 42 42 42 42]
 [42 42 42 42 42 42 42 42 42 42]
 [42 42 42 42 42 42 42 42 42 42]
 [42 42 42 42 42 42 42 42 42 42]
 [42 42 42 42 42 42 42 42 42 42]
 [42 42 42 42 42 42 42 42 42 42]
 [42 42 42 42 42 42 42 42 42 42]
 [42 42 42 42 42 42 42 42 42 42]
 [42 42 42 42 42 42 42 42 42 42]
 [42 42 42 42 42 42 42 42 42 42]]

From stefan_ml at behnel.de  Tue Nov 16 08:40:28 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 16 Nov 2010 08:40:28 +0100
Subject: [Cython] small test program works in python but raises
 exception through cython
In-Reply-To: <AANLkTi=hABQoLkTemxGowqU1Q2Frxq_eKYjV=j_iD6ky@mail.gmail.com>
References: <AANLkTi=hABQoLkTemxGowqU1Q2Frxq_eKYjV=j_iD6ky@mail.gmail.com>
Message-ID: <4CE2356C.3070604@behnel.de>

Vineet Jain, 15.11.2010 20:23:
> I posted this on cython-users and got not response.

cython-users is generally the right address for user questions. It also has 
a larger audience than the core developer list.

I think the reason you got no response so far is that reproducing it seems 
to require both h5py and numpy, which not every Cython developer has 
installed. It is not clear if the problem is related to Cython, it may well 
be a bug in h5py, a usage problem on your side or whatever.

Stefan

From stefan_ml at behnel.de  Tue Nov 16 12:46:34 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 16 Nov 2010 12:46:34 +0100
Subject: [Cython] Can't push to cython-devel
Message-ID: <4CE26F1A.4000607@behnel.de>

Hi,

when I try to hg push to cython-devel, I keep getting this:

"""
Traceback (most recent call last):
   File "/home/cython/www/hg/hgwebdir.cgi", line 43, in &lt;module&gt;
     wsgicgi.launch(wsgiapplication(make_web_app))
   File 
"/home/sage/sage_install/stable/local/lib/python2.6/mercurial/hgweb/wsgicgi.py", 
line 69, in launch
     for chunk in content:
   File 
"/home/sage/sage_install/stable/local/lib/python2.6/mercurial/util.py", 
line 310, in increasingchunks
     for chunk in source:
   File 
"/home/sage/sage_install/stable/local/lib/python2.6/mercurial/templater.py", line 
148, in expand
     item = iters[0].next()
   File 
"/home/sage/sage_install/stable/local/lib/python2.6/mercurial/templater.py", line 
122, in _process
     % (key, format))
SyntaxError: Error expanding 'sessionvars%urlparameter'
"""

Trac seems to be broken, too. Has anything changed on the Cython VM since 
yesterday that could trigger this?

Stefan

From vitja.makarov at gmail.com  Tue Nov 16 15:35:14 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 16 Nov 2010 17:35:14 +0300
Subject: [Cython] [bug] inf loop
Message-ID: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>

Hi!

I'm now trying to implement nested classes and found some errors in code.

genv = env
while genv.is_py_class_scope or genv.is_c_class_scope:
    genv = env.outer_scope

Can cause infinty loop. Small patch is attached.

vitja.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fix-genv-loops.patch
Type: text/x-patch
Size: 919 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20101116/957684ec/attachment.bin 

From robertwb at math.washington.edu  Tue Nov 16 18:18:26 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 16 Nov 2010 09:18:26 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <4CE26F1A.4000607@behnel.de>
References: <4CE26F1A.4000607@behnel.de>
Message-ID: <AANLkTi=oUpLpoBi5XZ1mR7h3Z3DdoFRXwsx3n7dFRhsH@mail.gmail.com>

Not that I know of. The sagemath.org site seems to be working fine.
Did someone upgrade the Python/Sage install there?

On Tue, Nov 16, 2010 at 3:46 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Hi,
>
> when I try to hg push to cython-devel, I keep getting this:
>
> """
> Traceback (most recent call last):
> ? File "/home/cython/www/hg/hgwebdir.cgi", line 43, in &lt;module&gt;
> ? ? wsgicgi.launch(wsgiapplication(make_web_app))
> ? File
> "/home/sage/sage_install/stable/local/lib/python2.6/mercurial/hgweb/wsgicgi.py",
> line 69, in launch
> ? ? for chunk in content:
> ? File
> "/home/sage/sage_install/stable/local/lib/python2.6/mercurial/util.py",
> line 310, in increasingchunks
> ? ? for chunk in source:
> ? File
> "/home/sage/sage_install/stable/local/lib/python2.6/mercurial/templater.py", line
> 148, in expand
> ? ? item = iters[0].next()
> ? File
> "/home/sage/sage_install/stable/local/lib/python2.6/mercurial/templater.py", line
> 122, in _process
> ? ? % (key, format))
> SyntaxError: Error expanding 'sessionvars%urlparameter'
> """
>
> Trac seems to be broken, too. Has anything changed on the Cython VM since
> yesterday that could trigger this?
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From markflorisson88 at gmail.com  Tue Nov 16 19:44:11 2010
From: markflorisson88 at gmail.com (mark florisson)
Date: Tue, 16 Nov 2010 19:44:11 +0100
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTi=oUpLpoBi5XZ1mR7h3Z3DdoFRXwsx3n7dFRhsH@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de>
	<AANLkTi=oUpLpoBi5XZ1mR7h3Z3DdoFRXwsx3n7dFRhsH@mail.gmail.com>
Message-ID: <AANLkTinw1KLD+nLmnPw+2d_DF1zrReEzhejJ_COq29fv@mail.gmail.com>

On 16 November 2010 18:18, Robert Bradshaw <robertwb at math.washington.edu>wrote:

> Not that I know of. The sagemath.org site seems to be working fine.
> Did someone upgrade the Python/Sage install there?
>
> On Tue, Nov 16, 2010 at 3:46 AM, Stefan Behnel <stefan_ml at behnel.de>
> wrote:
> > Hi,
> >
> > when I try to hg push to cython-devel, I keep getting this:
> >
> > """
> > Traceback (most recent call last):
> >   File "/home/cython/www/hg/hgwebdir.cgi", line 43, in &lt;module&gt;
> >     wsgicgi.launch(wsgiapplication(make_web_app))
> >   File
> >
> "/home/sage/sage_install/stable/local/lib/python2.6/mercurial/hgweb/wsgicgi.py",
> > line 69, in launch
> >     for chunk in content:
> >   File
> > "/home/sage/sage_install/stable/local/lib/python2.6/mercurial/util.py",
> > line 310, in increasingchunks
> >     for chunk in source:
> >   File
> >
> "/home/sage/sage_install/stable/local/lib/python2.6/mercurial/templater.py",
> line
> > 148, in expand
> >     item = iters[0].next()
> >   File
> >
> "/home/sage/sage_install/stable/local/lib/python2.6/mercurial/templater.py",
> line
> > 122, in _process
> >     % (key, format))
> > SyntaxError: Error expanding 'sessionvars%urlparameter'
> > """
> >
> > Trac seems to be broken, too. Has anything changed on the Cython VM since
> > yesterday that could trigger this?
> >
> > Stefan
> > _______________________________________________
> > Cython-dev mailing list
> > Cython-dev at codespeak.net
> > http://codespeak.net/mailman/listinfo/cython-dev
> >
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>


I have the same problem when trying to push to cython-gdb.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20101116/d6326fe8/attachment.htm 

From stefan_ml at behnel.de  Wed Nov 17 08:14:40 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 17 Nov 2010 08:14:40 +0100
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <4CE26F1A.4000607@behnel.de>
References: <4CE26F1A.4000607@behnel.de>
Message-ID: <4CE380E0.7070403@behnel.de>

Stefan Behnel, 16.11.2010 12:46:
> when I try to hg push to cython-devel, I keep getting this:
>
> """
> Traceback (most recent call last):
> SyntaxError: Error expanding 'sessionvars%urlparameter'
> """
>
> Trac seems to be broken, too. Has anything changed on the Cython VM since
> yesterday that could trigger this?

hg is back to normal since William cleaned up the disk space. trac is still 
down, it seems.

Robert, since you set up most of the infrastructure (IIRC), could you take 
an attempt at the migration to boxen?

Stefan

From robertwb at math.washington.edu  Wed Nov 17 08:34:18 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 16 Nov 2010 23:34:18 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <4CE380E0.7070403@behnel.de>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
Message-ID: <AANLkTi=Jf4hGPjf+V68OF8M2Rjhhg2SZUF6Or66U7L+A@mail.gmail.com>

On Tue, Nov 16, 2010 at 11:14 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Stefan Behnel, 16.11.2010 12:46:
>> when I try to hg push to cython-devel, I keep getting this:
>>
>> """
>> Traceback (most recent call last):
>> SyntaxError: Error expanding 'sessionvars%urlparameter'
>> """
>>
>> Trac seems to be broken, too. Has anything changed on the Cython VM since
>> yesterday that could trigger this?
>
> hg is back to normal since William cleaned up the disk space. trac is still
> down, it seems.
>
> Robert, since you set up most of the infrastructure (IIRC), could you take
> an attempt at the migration to boxen?

Yes, I'll do that.

- Robert

From robertwb at math.washington.edu  Wed Nov 17 08:40:26 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 16 Nov 2010 23:40:26 -0800
Subject: [Cython] possible bug report: combination of with/in
In-Reply-To: <AANLkTim_GhrXyTPi1zwG9bLWb6QpbVfcLfo7zHMWxqhK@mail.gmail.com>
References: <AANLkTim_GhrXyTPi1zwG9bLWb6QpbVfcLfo7zHMWxqhK@mail.gmail.com>
Message-ID: <AANLkTikOqk0XRNDk2WnyO5tU79kSd16rWs1o=iNP+MPE@mail.gmail.com>

Yep, looks like a bug to me.

On Mon, Nov 15, 2010 at 9:04 AM, Darren Dale <dsdale24 at gmail.com> wrote:
> I can cythonize this code:
>
> def test():
> ? ?sentinal = '3'
> ? ?with open('foo.txt', 'w') as f:
> ? ? ? ?sentinal == '1' or sentinal == '2'
>
> or this:
>
> def test():
> ? ?sentinal = '3'
> ? ?with open('foo.txt', 'w') as f:
> ? ? ? ?pass
> ? ?sentinal in ('1', '2')
>
> but not this:
>
> def test():
> ? ?sentinal = '3'
> ? ?with open('foo.txt', 'w') as f:
> ? ? ? ?sentinal in ('1', '2')
>
>
> $ cython test_open.pyx
>
> Error converting Pyrex file to C:
> ------------------------------------------------------------
> ...
> def test():
> ? ?sentinal = '3'
> ? ?with open('foo.txt', 'w') as f:
> ? ^
> ------------------------------------------------------------
>
> /Users/darren/temp/test/test_open.pyx:3:4: Compiler crash in
> AnalyseExpressionsTransform
>
> ModuleNode.body = StatListNode(test_open.pyx:1:0)
> StatListNode.stats[0] = DefNode(test_open.pyx:1:0,
> ? ?modifiers = [...]/0,
> ? ?name = 'test',
> ? ?reqd_kw_flags_cname = '0')
> File 'Nodes.py', line 343, in analyse_expressions:
> StatListNode(test_open.pyx:2:4)
> File 'Nodes.py', line 343, in analyse_expressions:
> StatListNode(test_open.pyx:3:4)
> File 'Nodes.py', line 4914, in analyse_expressions:
> TryFinallyStatNode(test_open.pyx:3:4,
> ? ?preserve_exception = 1)
> File 'Nodes.py', line 343, in analyse_expressions:
> StatListNode(test_open.pyx:3:4)
> File 'Nodes.py', line 4608, in analyse_expressions:
> TryExceptStatNode(test_open.pyx:3:4)
> File 'Nodes.py', line 343, in analyse_expressions:
> StatListNode(test_open.pyx:3:4)
> File 'Nodes.py', line 343, in analyse_expressions:
> StatListNode(test_open.pyx:3:4)
> File 'Nodes.py', line 3243, in analyse_expressions:
> ExprStatNode(test_open.pyx:3:4)
> File 'ExprNodes.py', line 308, in analyse_expressions:
> EvalWithTempExprNode(test_open.pyx:3:4,
> ? ?use_managed_ref = True)
> File 'UtilNodes.py', line 239, in analyse_types:
> EvalWithTempExprNode(test_open.pyx:3:4,
> ? ?use_managed_ref = True)
> File 'ExprNodes.py', line 5664, in analyse_types:
> BoolBinopNode(test_open.pyx:3:4,
> ? ?operator = 'or',
> ? ?use_managed_ref = True)
> File 'ExprNodes.py', line 4892, in analyse_types:
> TypecastNode(test_open.pyx:3:4,
> ? ?use_managed_ref = True)
> File 'ExprNodes.py', line 6138, in analyse_types:
> PrimaryCmpNode(test_open.pyx:3:4,
> ? ?operator = '==',
> ? ?use_managed_ref = True)
> File 'ExprNodes.py', line 5840, in is_cpp_comparison:
> PrimaryCmpNode(test_open.pyx:3:4,
> ? ?operator = '==',
> ? ?use_managed_ref = True)
>
> Compiler crash traceback from this point on:
> ?File "/Users/darren/.local/lib/python3.1/site-packages/Cython/Compiler/ExprNodes.py",
> line 5841, in is_cpp_comparison
> ? ?return self.operand1.type.is_cpp_class or self.operand2.type.is_cpp_class
> AttributeError: 'NoneType' object has no attribute 'is_cpp_class'
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From stefan_ml at behnel.de  Wed Nov 17 08:47:22 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 17 Nov 2010 08:47:22 +0100
Subject: [Cython] possible bug report: combination of with/in
In-Reply-To: <AANLkTikOqk0XRNDk2WnyO5tU79kSd16rWs1o=iNP+MPE@mail.gmail.com>
References: <AANLkTim_GhrXyTPi1zwG9bLWb6QpbVfcLfo7zHMWxqhK@mail.gmail.com>
	<AANLkTikOqk0XRNDk2WnyO5tU79kSd16rWs1o=iNP+MPE@mail.gmail.com>
Message-ID: <4CE3888A.80003@behnel.de>

Robert Bradshaw, 17.11.2010 08:40:
> On Mon, Nov 15, 2010 at 9:04 AM, Darren Dale wrote:
>> I can cythonize this code:
>>
>> def test():
>>     sentinal = '3'
>>     with open('foo.txt', 'w') as f:
>>         sentinal == '1' or sentinal == '2'
>>
>> or this:
>>
>> def test():
>>     sentinal = '3'
>>     with open('foo.txt', 'w') as f:
>>         pass
>>     sentinal in ('1', '2')
>>
>> but not this:
>>
>> def test():
>>     sentinal = '3'
>>     with open('foo.txt', 'w') as f:
>>         sentinal in ('1', '2')
>>
>>
>> $ cython test_open.pyx
>>
>> Error converting Pyrex file to C:
>> ------------------------------------------------------------
>> ...
>> def test():
>>     sentinal = '3'
>>     with open('foo.txt', 'w') as f:
>>    ^
>> ------------------------------------------------------------
>>
>> /Users/darren/temp/test/test_open.pyx:3:4: Compiler crash in
>> AnalyseExpressionsTransform
>>
> Yep, looks like a bug to me.

In general, it is a bug if the compiler crashes on whatever code, instead 
of printing a suitable error message. Please file a bug report.

Stefan

From robertwb at math.washington.edu  Wed Nov 17 09:08:09 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 17 Nov 2010 00:08:09 -0800
Subject: [Cython] possible bug report: combination of with/in
In-Reply-To: <4CE3888A.80003@behnel.de>
References: <AANLkTim_GhrXyTPi1zwG9bLWb6QpbVfcLfo7zHMWxqhK@mail.gmail.com>
	<AANLkTikOqk0XRNDk2WnyO5tU79kSd16rWs1o=iNP+MPE@mail.gmail.com>
	<4CE3888A.80003@behnel.de>
Message-ID: <AANLkTi=8j9dy59Vfacm+kN9PVp6r++FRKZYoMcaHXas3@mail.gmail.com>

On Tue, Nov 16, 2010 at 11:47 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 17.11.2010 08:40:
>> On Mon, Nov 15, 2010 at 9:04 AM, Darren Dale wrote:
>>> I can cythonize this code:
>>>
>>> def test():
>>> ? ? sentinal = '3'
>>> ? ? with open('foo.txt', 'w') as f:
>>> ? ? ? ? sentinal == '1' or sentinal == '2'
>>>
>>> or this:
>>>
>>> def test():
>>> ? ? sentinal = '3'
>>> ? ? with open('foo.txt', 'w') as f:
>>> ? ? ? ? pass
>>> ? ? sentinal in ('1', '2')
>>>
>>> but not this:
>>>
>>> def test():
>>> ? ? sentinal = '3'
>>> ? ? with open('foo.txt', 'w') as f:
>>> ? ? ? ? sentinal in ('1', '2')
>>>
>>>
>>> $ cython test_open.pyx
>>>
>>> Error converting Pyrex file to C:
>>> ------------------------------------------------------------
>>> ...
>>> def test():
>>> ? ? sentinal = '3'
>>> ? ? with open('foo.txt', 'w') as f:
>>> ? ?^
>>> ------------------------------------------------------------
>>>
>>> /Users/darren/temp/test/test_open.pyx:3:4: Compiler crash in
>>> AnalyseExpressionsTransform
>>>
>> Yep, looks like a bug to me.
>
> In general, it is a bug if the compiler crashes on whatever code, instead
> of printing a suitable error message.

I agree, I was just stating that I confirmed it.

> Please file a bug report.

As soon as I get trac up and running...

- Robert

From robertwb at math.washington.edu  Wed Nov 17 09:16:11 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 17 Nov 2010 00:16:11 -0800
Subject: [Cython] possible bug report: combination of with/in
In-Reply-To: <AANLkTi=8j9dy59Vfacm+kN9PVp6r++FRKZYoMcaHXas3@mail.gmail.com>
References: <AANLkTim_GhrXyTPi1zwG9bLWb6QpbVfcLfo7zHMWxqhK@mail.gmail.com>
	<AANLkTikOqk0XRNDk2WnyO5tU79kSd16rWs1o=iNP+MPE@mail.gmail.com>
	<4CE3888A.80003@behnel.de>
	<AANLkTi=8j9dy59Vfacm+kN9PVp6r++FRKZYoMcaHXas3@mail.gmail.com>
Message-ID: <AANLkTikjKprL838nE9uWwR97T-jAvx2vp_Vw=rUFAiXx@mail.gmail.com>

On Wed, Nov 17, 2010 at 12:08 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Tue, Nov 16, 2010 at 11:47 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Robert Bradshaw, 17.11.2010 08:40:
>>> On Mon, Nov 15, 2010 at 9:04 AM, Darren Dale wrote:
>>>> I can cythonize this code:
>>>>
>>>> def test():
>>>> ? ? sentinal = '3'
>>>> ? ? with open('foo.txt', 'w') as f:
>>>> ? ? ? ? sentinal == '1' or sentinal == '2'
>>>>
>>>> or this:
>>>>
>>>> def test():
>>>> ? ? sentinal = '3'
>>>> ? ? with open('foo.txt', 'w') as f:
>>>> ? ? ? ? pass
>>>> ? ? sentinal in ('1', '2')
>>>>
>>>> but not this:
>>>>
>>>> def test():
>>>> ? ? sentinal = '3'
>>>> ? ? with open('foo.txt', 'w') as f:
>>>> ? ? ? ? sentinal in ('1', '2')
>>>>
>>>>
>>>> $ cython test_open.pyx
>>>>
>>>> Error converting Pyrex file to C:
>>>> ------------------------------------------------------------
>>>> ...
>>>> def test():
>>>> ? ? sentinal = '3'
>>>> ? ? with open('foo.txt', 'w') as f:
>>>> ? ?^
>>>> ------------------------------------------------------------
>>>>
>>>> /Users/darren/temp/test/test_open.pyx:3:4: Compiler crash in
>>>> AnalyseExpressionsTransform
>>>>
>>> Yep, looks like a bug to me.
>>
>> In general, it is a bug if the compiler crashes on whatever code, instead
>> of printing a suitable error message.
>
> I agree, I was just stating that I confirmed it.
>
>> Please file a bug report.
>
> As soon as I get trac up and running...

http://trac.cython.org/cython_trac/ticket/595

From stefan_ml at behnel.de  Wed Nov 17 09:20:04 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 17 Nov 2010 09:20:04 +0100
Subject: [Cython] gcc warnings in Py3 C++ compilation
Message-ID: <4CE39034.2030902@behnel.de>

Hi,

we get warnings in basically all test modules compiled under Python 3 in 
C++ mode, e.g.

https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-py3k-cpp/605/parsed_console/

They all look like this:

"""
withstat.cpp:10002: warning: missing initializer for member 
?PyModuleDef_Base::m_init?
withstat.cpp:10002: warning: missing initializer for member 
?PyModuleDef_Base::m_index?
withstat.cpp:10002: warning: missing initializer for member 
?PyModuleDef_Base::m_copy?
"""

Can one of the C++ gurus please check if there's anything we can do about 
these? Given that this only appears in Py3, it's most likely a problem in 
CPython ...

Stefan

From stefan_ml at behnel.de  Wed Nov 17 09:21:06 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 17 Nov 2010 09:21:06 +0100
Subject: [Cython] possible bug report: combination of with/in
In-Reply-To: <AANLkTikjKprL838nE9uWwR97T-jAvx2vp_Vw=rUFAiXx@mail.gmail.com>
References: <AANLkTim_GhrXyTPi1zwG9bLWb6QpbVfcLfo7zHMWxqhK@mail.gmail.com>	<AANLkTikOqk0XRNDk2WnyO5tU79kSd16rWs1o=iNP+MPE@mail.gmail.com>	<4CE3888A.80003@behnel.de>	<AANLkTi=8j9dy59Vfacm+kN9PVp6r++FRKZYoMcaHXas3@mail.gmail.com>
	<AANLkTikjKprL838nE9uWwR97T-jAvx2vp_Vw=rUFAiXx@mail.gmail.com>
Message-ID: <4CE39072.5030207@behnel.de>

Robert Bradshaw, 17.11.2010 09:16:
> On Wed, Nov 17, 2010 at 12:08 AM, Robert Bradshaw  wrote:
>> On Tue, Nov 16, 2010 at 11:47 PM, Stefan Behnel wrote:
>>> Please file a bug report.
>>
>> As soon as I get trac up and running...
>
> http://trac.cython.org/cython_trac/ticket/595

Thanks a lot, Robert!

Stefan

From fperez.net at gmail.com  Wed Nov 17 09:41:14 2010
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 17 Nov 2010 00:41:14 -0800
Subject: [Cython] possible bug report: combination of with/in
In-Reply-To: <AANLkTim_GhrXyTPi1zwG9bLWb6QpbVfcLfo7zHMWxqhK@mail.gmail.com>
References: <AANLkTim_GhrXyTPi1zwG9bLWb6QpbVfcLfo7zHMWxqhK@mail.gmail.com>
Message-ID: <AANLkTi=x2YnCgKicg=o3GKeHk2ZNua2w921UySAteXWp@mail.gmail.com>

Hey Darren,

On Mon, Nov 15, 2010 at 9:04 AM, Darren Dale <dsdale24 at gmail.com> wrote:
> but not this:
>
> def test():
> ? ?sentinal = '3'
> ? ?with open('foo.txt', 'w') as f:
> ? ? ? ?sentinal in ('1', '2')
>

as a possible workaround until the actual bug is fixed, remember that
'x in y' is mostly sugar for 'y.__contains__(x)',

In [1]: a = ('1', '2')

In [2]: '3' in a
Out[2]: False

In [3]: a.__contains__('3')
Out[3]: False

so you may be able to use the uglier form for now.

Also worth mentioning that if you assign the tuple to a variable, the
bug doesn't show up:

dreamweaver[~]> cat bad.pyx
def test():
   sentinel = '3'
   with open('foo.txt', 'w') as f:
      a = ('1', '2')
      sentinel in a

dreamweaver[~]> cython bad.pyx
dreamweaver[~]> # no crash, though I didn't test that it behaves correctly.


Cheers,

f

From robertwb at math.washington.edu  Wed Nov 17 09:59:51 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 17 Nov 2010 00:59:51 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <4CE380E0.7070403@behnel.de>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
Message-ID: <AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>

On Tue, Nov 16, 2010 at 11:14 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Stefan Behnel, 16.11.2010 12:46:
>> when I try to hg push to cython-devel, I keep getting this:
>>
>> """
>> Traceback (most recent call last):
>> SyntaxError: Error expanding 'sessionvars%urlparameter'
>> """
>>
>> Trac seems to be broken, too. Has anything changed on the Cython VM since
>> yesterday that could trigger this?
>
> hg is back to normal since William cleaned up the disk space. trac is still
> down, it seems.
>
> Robert, since you set up most of the infrastructure (IIRC), could you take
> an attempt at the migration to boxen?

So, I've been thinking about this, and one of the reasons we're using
the current setup is that it requires very little administration from
me, as I am essentially leaching of the sagemath.org infrastructure.
Moving things to boxen would be a bit more administration on our part,
but still not too bad, and I know William is a happy enough Cython
user to be fine with continuing to host us :). However, I'm wondering
if this would be a ripe occasion to making the leap to something like
http://code.google.com. Currently, our infrastructure consists of

1. The web site
2. Trac
3. Wiki
4. Repositories
5. Buildbot
6. Mailing lists

Currently we're hosting 1-5, and 6 is being hosted by codespeak.net
(for cython-dev) and google (for cython-users). I think it may be
worth considering moving 2-4 elsewhere, as there is little loss and
they are the higher-maintenance (from an administrative point of view)
items, and features such as code review tools would be nice to have as
well. Trying to use launchpad was painful, so this decision shouldn't
be taken lightly, but I think we can do better. Of course
code.google.com isn't the only option, but even trying to be unbiased
about it I think it's a very good option, and
http://www.dataliberation.org/google/code-project-hosting factors into
it as well.

In terms of the website itself, I think we should keep hosting that
(for maximum flexibility, and it's easy to administer), and there
isn't much of an option for 5 (though steps should be taken to reduce
its load).

Thoughts?

- Robert

From matthew.brett at gmail.com  Wed Nov 17 10:19:35 2010
From: matthew.brett at gmail.com (Matthew Brett)
Date: Wed, 17 Nov 2010 01:19:35 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
Message-ID: <AANLkTi=L081GsSjnbCFAOQA2r6=z+T7WT_nqyJaVDaHt@mail.gmail.com>

Hi,

> However, I'm wondering
> if this would be a ripe occasion to making the leap to something like
> http://code.google.com.

Oh - dear - I'm afraid I have said this on several mailing lists, but
Google code is the most aggressive in blocking access from 'forbidden
countries' such as Cuba, where I often work [1].   Only Google code
and Sourceforge do this; and Sourceforge allows you to opt out.   I
think this is against the spirit of open source software, and it has
made it harder for me to advocate the use of open-source in general
and python in particular.   I hope very much you consider using a more
open provider.

Best,

Matthew

[1] http://en.wikipedia.org/wiki/Comparison_of_open_source_software_hosting_facilities

From stefan_ml at behnel.de  Wed Nov 17 10:20:12 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 17 Nov 2010 10:20:12 +0100
Subject: [Cython] possible bug report: combination of with/in
In-Reply-To: <AANLkTi=x2YnCgKicg=o3GKeHk2ZNua2w921UySAteXWp@mail.gmail.com>
References: <AANLkTim_GhrXyTPi1zwG9bLWb6QpbVfcLfo7zHMWxqhK@mail.gmail.com>
	<AANLkTi=x2YnCgKicg=o3GKeHk2ZNua2w921UySAteXWp@mail.gmail.com>
Message-ID: <4CE39E4C.4090304@behnel.de>

Fernando Perez, 17.11.2010 09:41:
> On Mon, Nov 15, 2010 at 9:04 AM, Darren Dale wrote:
>> def test():
>>     sentinal = '3'
>>     with open('foo.txt', 'w') as f:
>>         sentinal in ('1', '2')
>
> as a possible workaround until the actual bug is fixed, remember that
> 'x in y' is mostly sugar for 'y.__contains__(x)',

I agree that it provides a work-around, but it's worth knowing that there's 
a major performance difference. The 'in' operator is a lot faster, 
especially on literal sequences, where Cython explodes it into separate 
comparisons. Even in CPython, 'in' is much faster than looking up and 
calling the bound Python method.


> Also worth mentioning that if you assign the tuple to a variable, the
> bug doesn't show up:
>
> dreamweaver[~]>  cat bad.pyx
> def test():
>     sentinel = '3'
>     with open('foo.txt', 'w') as f:
>        a = ('1', '2')
>        sentinel in a
>
> dreamweaver[~]>  cython bad.pyx
> dreamweaver[~]>  # no crash, though I didn't test that it behaves correctly.

I think this is a better work-around. It's still slower in Cython (even 
though I just implemented interning of constant tuples for 0.13.1), but it 
should be much faster than calling __contains__().

Stefan

From stefan_ml at behnel.de  Wed Nov 17 10:24:52 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 17 Nov 2010 10:24:52 +0100
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTi=L081GsSjnbCFAOQA2r6=z+T7WT_nqyJaVDaHt@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de>
	<4CE380E0.7070403@behnel.de>	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<AANLkTi=L081GsSjnbCFAOQA2r6=z+T7WT_nqyJaVDaHt@mail.gmail.com>
Message-ID: <4CE39F64.9010909@behnel.de>

Matthew Brett, 17.11.2010 10:19:
>> However, I'm wondering
>> if this would be a ripe occasion to making the leap to something like
>> http://code.google.com.
>
> Oh - dear - I'm afraid I have said this on several mailing lists, but
> Google code is the most aggressive in blocking access from 'forbidden
> countries' such as Cuba, where I often work [1].   Only Google code
> and Sourceforge do this; and Sourceforge allows you to opt out.   I
> think this is against the spirit of open source software, and it has
> made it harder for me to advocate the use of open-source in general
> and python in particular.   I hope very much you consider using a more
> open provider.

Hear hear.

Stefan

From vitja.makarov at gmail.com  Wed Nov 17 11:33:00 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Wed, 17 Nov 2010 13:33:00 +0300
Subject: [Cython] [bug] inf loop
In-Reply-To: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>
References: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>
Message-ID: <AANLkTi=bWs0X8peV0fQQeQx4_okvduQue_f5nJD75Nta@mail.gmail.com>

2010/11/16 Vitja Makarov <vitja.makarov at gmail.com>:
> Hi!
>
> I'm now trying to implement nested classes and found some errors in code.
>
> genv = env
> while genv.is_py_class_scope or genv.is_c_class_scope:
> ? ?genv = env.outer_scope
>
> Can cause infinty loop. Small patch is attached.
>
> vitja.
>

I've created ticket for #596 for class closures.

From dagss at student.matnat.uio.no  Wed Nov 17 11:33:24 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 17 Nov 2010 11:33:24 +0100
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
Message-ID: <4CE3AF74.40703@student.matnat.uio.no>

On 11/17/2010 09:59 AM, Robert Bradshaw wrote:
> On Tue, Nov 16, 2010 at 11:14 PM, Stefan Behnel<stefan_ml at behnel.de>  wrote:
>    
>> Stefan Behnel, 16.11.2010 12:46:
>>      
>>> when I try to hg push to cython-devel, I keep getting this:
>>>
>>> """
>>> Traceback (most recent call last):
>>> SyntaxError: Error expanding 'sessionvars%urlparameter'
>>> """
>>>
>>> Trac seems to be broken, too. Has anything changed on the Cython VM since
>>> yesterday that could trigger this?
>>>        
>> hg is back to normal since William cleaned up the disk space. trac is still
>> down, it seems.
>>
>> Robert, since you set up most of the infrastructure (IIRC), could you take
>> an attempt at the migration to boxen?
>>      
> So, I've been thinking about this, and one of the reasons we're using
> the current setup is that it requires very little administration from
> me, as I am essentially leaching of the sagemath.org infrastructure.
> Moving things to boxen would be a bit more administration on our part,
> but still not too bad, and I know William is a happy enough Cython
> user to be fine with continuing to host us :). However, I'm wondering
> if this would be a ripe occasion to making the leap to something like
> http://code.google.com. Currently, our infrastructure consists of
>
> 1. The web site
> 2. Trac
> 3. Wiki
> 4. Repositories
> 5. Buildbot
> 6. Mailing lists
>
> Currently we're hosting 1-5, and 6 is being hosted by codespeak.net
> (for cython-dev) and google (for cython-users). I think it may be
> worth considering moving 2-4 elsewhere, as there is little loss and
> they are the higher-maintenance (from an administrative point of view)
> items, and features such as code review tools would be nice to have as
> well. Trying to use launchpad was painful, so this decision shouldn't
> be taken lightly, but I think we can do better. Of course
> code.google.com isn't the only option, but even trying to be unbiased
> about it I think it's a very good option, and
> http://www.dataliberation.org/google/code-project-hosting factors into
> it as well.
>    

Well, my personal favorite would be github :-), which is simply growing 
more and more awesome. I'll just make a short case and then let the case 
rest, not wanting an all-out flame war on VCSes :-)

Bitbucket would be more natural for us, but I haven't used bitbucket 
much, so I don't know how it compares. Anybody?

At any rate, I don't think github should be ruled out:

  - There is a plugin for Mercurial to deal with git repositories: 
http://hg-git.github.com/, so we don't need to officially switch to git 
(I'd switch personally, but that's another matter)

  - Static web site content (you just create a repository which it 
serves as static content, e.g, http://cournape.github.com/Bento/)

  - Not only wiki, but one using a git repository as backend

  - github pull requests! (Make your own branch with a fix, and then 
send a request that bundles this branch together with commenting, 
tickets etc.)

  - Nice visualization tools for seeing what exists in other people's 
branches that has not been merged yet

  - Most things are saved in git repositories, so it doesn't lock any 
data in either. Not sure about what backend tickets use.

Dag Sverre

From dagss at student.matnat.uio.no  Wed Nov 17 11:39:37 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 17 Nov 2010 11:39:37 +0100
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <4CE3AF74.40703@student.matnat.uio.no>
References: <4CE26F1A.4000607@behnel.de>
	<4CE380E0.7070403@behnel.de>	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<4CE3AF74.40703@student.matnat.uio.no>
Message-ID: <4CE3B0E9.7090304@student.matnat.uio.no>

On 11/17/2010 11:33 AM, Dag Sverre Seljebotn wrote:
> On 11/17/2010 09:59 AM, Robert Bradshaw wrote:
>    
>> On Tue, Nov 16, 2010 at 11:14 PM, Stefan Behnel<stefan_ml at behnel.de>   wrote:
>>
>>      
>>> Stefan Behnel, 16.11.2010 12:46:
>>>
>>>        
>>>> when I try to hg push to cython-devel, I keep getting this:
>>>>
>>>> """
>>>> Traceback (most recent call last):
>>>> SyntaxError: Error expanding 'sessionvars%urlparameter'
>>>> """
>>>>
>>>> Trac seems to be broken, too. Has anything changed on the Cython VM since
>>>> yesterday that could trigger this?
>>>>
>>>>          
>>> hg is back to normal since William cleaned up the disk space. trac is still
>>> down, it seems.
>>>
>>> Robert, since you set up most of the infrastructure (IIRC), could you take
>>> an attempt at the migration to boxen?
>>>
>>>        
>> So, I've been thinking about this, and one of the reasons we're using
>> the current setup is that it requires very little administration from
>> me, as I am essentially leaching of the sagemath.org infrastructure.
>> Moving things to boxen would be a bit more administration on our part,
>> but still not too bad, and I know William is a happy enough Cython
>> user to be fine with continuing to host us :). However, I'm wondering
>> if this would be a ripe occasion to making the leap to something like
>> http://code.google.com. Currently, our infrastructure consists of
>>
>> 1. The web site
>> 2. Trac
>> 3. Wiki
>> 4. Repositories
>> 5. Buildbot
>> 6. Mailing lists
>>
>> Currently we're hosting 1-5, and 6 is being hosted by codespeak.net
>> (for cython-dev) and google (for cython-users). I think it may be
>> worth considering moving 2-4 elsewhere, as there is little loss and
>> they are the higher-maintenance (from an administrative point of view)
>> items, and features such as code review tools would be nice to have as
>> well. Trying to use launchpad was painful, so this decision shouldn't
>> be taken lightly, but I think we can do better. Of course
>> code.google.com isn't the only option, but even trying to be unbiased
>> about it I think it's a very good option, and
>> http://www.dataliberation.org/google/code-project-hosting factors into
>> it as well.
>>
>>      
> Well, my personal favorite would be github :-), which is simply growing
> more and more awesome. I'll just make a short case and then let the case
> rest, not wanting an all-out flame war on VCSes :-)
>
> Bitbucket would be more natural for us, but I haven't used bitbucket
> much, so I don't know how it compares. Anybody?
>    

Forgetting about the git aspect for a moment, I'm strongly in favor of a 
DVCS-centered site. That is, +1 to bitbucket, github, gitorious, ++, and 
-1 to SourceForge, Google Code, etc.

It is simply much nicer to let people create a personal feature/bugfix 
branch with the click of a button, and work with pull requests of those 
branches within a website and tracker designed around that, than having 
to send patches around manually in the ticket system.

One ticket, one branch :-)

Dag Sverre

From dmalcolm at redhat.com  Wed Nov 17 17:01:03 2010
From: dmalcolm at redhat.com (David Malcolm)
Date: Wed, 17 Nov 2010 11:01:03 -0500
Subject: [Cython] gcc warnings in Py3 C++ compilation
In-Reply-To: <4CE39034.2030902@behnel.de>
References: <4CE39034.2030902@behnel.de>
Message-ID: <1290009663.16384.3.camel@radiator.bos.redhat.com>

On Wed, 2010-11-17 at 09:20 +0100, Stefan Behnel wrote:
> Hi,
> 
> we get warnings in basically all test modules compiled under Python 3 in 
> C++ mode, e.g.
> 
> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-py3k-cpp/605/parsed_console/
> 
> They all look like this:
> 
> """
> withstat.cpp:10002: warning: missing initializer for member 
> ?PyModuleDef_Base::m_init?
> withstat.cpp:10002: warning: missing initializer for member 
> ?PyModuleDef_Base::m_index?
> withstat.cpp:10002: warning: missing initializer for member 
> ?PyModuleDef_Base::m_copy?
> """
> 
> Can one of the C++ gurus please check if there's anything we can do about 
> these? Given that this only appears in Py3, it's most likely a problem in 
> CPython ...

Looks like:
  http://bugs.python.org/issue9518


Hope this is helpful
Dave








From stefan_ml at behnel.de  Wed Nov 17 15:26:15 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 17 Nov 2010 15:26:15 +0100
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <4CE3B0E9.7090304@student.matnat.uio.no>
References: <4CE26F1A.4000607@behnel.de>	<4CE380E0.7070403@behnel.de>	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>	<4CE3AF74.40703@student.matnat.uio.no>
	<4CE3B0E9.7090304@student.matnat.uio.no>
Message-ID: <4CE3E607.2040801@behnel.de>

Dag Sverre Seljebotn, 17.11.2010 11:39:
> On 11/17/2010 11:33 AM, Dag Sverre Seljebotn wrote:
>> On 11/17/2010 09:59 AM, Robert Bradshaw wrote:
>>> So, I've been thinking about this, and one of the reasons we're using
>>> the current setup is that it requires very little administration from
>>> me, as I am essentially leaching of the sagemath.org infrastructure.
>>> Moving things to boxen would be a bit more administration on our part,
>>> but still not too bad, and I know William is a happy enough Cython
>>> user to be fine with continuing to host us :). However, I'm wondering
>>> if this would be a ripe occasion to making the leap to something like
>>> http://code.google.com. Currently, our infrastructure consists of
>>>
>>> 1. The web site
>>> 2. Trac
>>> 3. Wiki
>>> 4. Repositories
>>> 5. Buildbot
>>> 6. Mailing lists
>>>
>>> Currently we're hosting 1-5, and 6 is being hosted by codespeak.net
>>> (for cython-dev) and google (for cython-users). I think it may be
>>> worth considering moving 2-4 elsewhere, as there is little loss and
>>> they are the higher-maintenance (from an administrative point of view)
>>> items, and features such as code review tools would be nice to have as
>>> well.

I see the serious advantage of simplifying the access to bug tracker, wiki, 
etc. for new users.

However, regarding the "little loss", does anyone have experience in 
migrating trac tickets to a different tracker? There's a lot of information 
in the tracker that I don't want to loose, and I wouldn't like to look at 
two trackers to figure out if a bug has been reported.


> Forgetting about the git aspect for a moment, I'm strongly in favor of a
> DVCS-centered site. That is, +1 to bitbucket, github, gitorious, ++, and
> -1 to SourceForge, Google Code, etc.

I guess it's bitbucket then? Switching the VCS just because of a project 
hosting site sounds like more trouble than I'm currently after.

Stefan

From stefan_ml at behnel.de  Wed Nov 17 15:55:09 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 17 Nov 2010 15:55:09 +0100
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
Message-ID: <4CE3ECCD.3030605@behnel.de>

Robert Bradshaw, 17.11.2010 09:59:
> 1. The web site
> 5. Buildbot
>
> In terms of the website itself, I think we should keep hosting that
> (for maximum flexibility, and it's easy to administer)

+1


> and there
> isn't much of an option for 5 (though steps should be taken to reduce
> its load).

I'm a big fan of timely commit feedback. Hudson is already down to 6 build 
threads, so it takes a while now to finish the test suites. Also, the less 
parallel builds we run, the longer it disturbs other programs on the same 
machine.

Could you describe a little where you think the current pain points are? Is 
it more about the number of builds per day or the overall load during a 
single build? I/O or CPU load? My impression is that the /scratch/hudson 
mount point doesn't have very fast I/O, which lets the builds take longer 
than necessary.

A problem that I see is that Hudson only checks for changes every 10 
minutes in order to keep the load of the (fairly slow) Cython VM 
acceptable. So if a new commit arrives shortly after triggering the builds, 
it will continue to run the tests for 10 minutes before it starts over and 
does it all again. I tried changing that with the commit hook script but 
didn't get a response.

I don't know if bitbucket supports commit hooks, BTW.

One way to get relief is to reduce the currently 9 jobs that all query the 
cython-devel repo for changes. I can try to change that by adding a 
dedicated source build job that does the polling and triggers the real 
build jobs that just download the sdist with no hg interaction. That moves 
the hg change history one step further away from the build result on the 
Hudson job page, but that won't hurt anyone, I guess.

Stefan

From robertwb at math.washington.edu  Wed Nov 17 21:07:01 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 17 Nov 2010 12:07:01 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <4CE3E607.2040801@behnel.de>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<4CE3AF74.40703@student.matnat.uio.no>
	<4CE3B0E9.7090304@student.matnat.uio.no>
	<4CE3E607.2040801@behnel.de>
Message-ID: <AANLkTi=zvAX0hodpvWDd=8Wy9aTV6m=Q8NCrw7Su_f0s@mail.gmail.com>

On Wed, Nov 17, 2010 at 6:26 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Dag Sverre Seljebotn, 17.11.2010 11:39:
>> On 11/17/2010 11:33 AM, Dag Sverre Seljebotn wrote:
>>> On 11/17/2010 09:59 AM, Robert Bradshaw wrote:
>>>> So, I've been thinking about this, and one of the reasons we're using
>>>> the current setup is that it requires very little administration from
>>>> me, as I am essentially leaching of the sagemath.org infrastructure.
>>>> Moving things to boxen would be a bit more administration on our part,
>>>> but still not too bad, and I know William is a happy enough Cython
>>>> user to be fine with continuing to host us :). However, I'm wondering
>>>> if this would be a ripe occasion to making the leap to something like
>>>> http://code.google.com. Currently, our infrastructure consists of
>>>>
>>>> 1. The web site
>>>> 2. Trac
>>>> 3. Wiki
>>>> 4. Repositories
>>>> 5. Buildbot
>>>> 6. Mailing lists
>>>>
>>>> Currently we're hosting 1-5, and 6 is being hosted by codespeak.net
>>>> (for cython-dev) and google (for cython-users). I think it may be
>>>> worth considering moving 2-4 elsewhere, as there is little loss and
>>>> they are the higher-maintenance (from an administrative point of view)
>>>> items, and features such as code review tools would be nice to have as
>>>> well.
>
> I see the serious advantage of simplifying the access to bug tracker, wiki,
> etc. for new users.

And for us old-timers too :)

> However, regarding the "little loss", does anyone have experience in
> migrating trac tickets to a different tracker? There's a lot of information
> in the tracker that I don't want to loose, and I wouldn't like to look at
> two trackers to figure out if a bug has been reported.

+1, this was one of my criteria too (hence the link to
http://www.dataliberation.org/google/code-project-hosting , which I
consider a critical part being open). However, the unfortunate
geographic limitations seem to be a blocker. It looks like GitHub and
BitBucket have APIs for importing/exporting issues as well, so we'd
certainly migrate all our data. The wikis for all are stored in a DVCS
repository, which is nice. Wiki history isn't as important, but I'd
see if it wasn't too hard to scrape/grab.

>> Forgetting about the git aspect for a moment, I'm strongly in favor of a
>> DVCS-centered site. That is, +1 to bitbucket, github, gitorious, ++, and
>> -1 to SourceForge, Google Code, etc.

For sure. If Google Code didn't support the DVCS "click to clone"
model I wouldn't have considered it.

> I guess it's bitbucket then? Switching the VCS just because of a project
> hosting site sounds like more trouble than I'm currently after.

The biggest drawback of bitbucket is the lack of integrated code
review--if we're moving projects we might as well move to something
that's a step up in this area. I would be happy switching to Git to
use GitHub--my primary issue with it compared to mercurial is the
steep(er) learning curve, but this is less of an issue for a rather
technical project like ours.

- Robert

From dagss at student.matnat.uio.no  Wed Nov 17 21:16:40 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 17 Nov 2010 21:16:40 +0100
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <4CE3E607.2040801@behnel.de>
References: <4CE26F1A.4000607@behnel.de>	<4CE380E0.7070403@behnel.de>	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>	<4CE3AF74.40703@student.matnat.uio.no>	<4CE3B0E9.7090304@student.matnat.uio.no>
	<4CE3E607.2040801@behnel.de>
Message-ID: <4CE43828.1050502@student.matnat.uio.no>

On 11/17/2010 03:26 PM, Stefan Behnel wrote:
> Dag Sverre Seljebotn, 17.11.2010 11:39:
>    
>> On 11/17/2010 11:33 AM, Dag Sverre Seljebotn wrote:
>>      
>>> On 11/17/2010 09:59 AM, Robert Bradshaw wrote:
>>>        
>>>> So, I've been thinking about this, and one of the reasons we're using
>>>> the current setup is that it requires very little administration from
>>>> me, as I am essentially leaching of the sagemath.org infrastructure.
>>>> Moving things to boxen would be a bit more administration on our part,
>>>> but still not too bad, and I know William is a happy enough Cython
>>>> user to be fine with continuing to host us :). However, I'm wondering
>>>> if this would be a ripe occasion to making the leap to something like
>>>> http://code.google.com. Currently, our infrastructure consists of
>>>>
>>>> 1. The web site
>>>> 2. Trac
>>>> 3. Wiki
>>>> 4. Repositories
>>>> 5. Buildbot
>>>> 6. Mailing lists
>>>>
>>>> Currently we're hosting 1-5, and 6 is being hosted by codespeak.net
>>>> (for cython-dev) and google (for cython-users). I think it may be
>>>> worth considering moving 2-4 elsewhere, as there is little loss and
>>>> they are the higher-maintenance (from an administrative point of view)
>>>> items, and features such as code review tools would be nice to have as
>>>> well.
>>>>          
> I see the serious advantage of simplifying the access to bug tracker, wiki,
> etc. for new users.
>
> However, regarding the "little loss", does anyone have experience in
> migrating trac tickets to a different tracker? There's a lot of information
> in the tracker that I don't want to loose, and I wouldn't like to look at
> two trackers to figure out if a bug has been reported.
>    

+1, migrating the tracker in full automatically is a must, otherwise we 
can just as well stick to the old one. Of course, we can move the 
repositories before moving the tracker, it's not like they can become 
more disconnected than they already are (well, except that we may want 
to patch some URLs in the strings).

I'm hoping we should be able to find scripts for this. All I can find in 
5 minutes of Googling is the import APIs though. Both bitbucket and 
github has APIs, but we should look closer make sure we can both import 
and export tickets easily before moving the tracker IMO. I assume 
there's exporters for Trac and that some scripting can be done to move 
the content in not too many hours.

>    
>> Forgetting about the git aspect for a moment, I'm strongly in favor of a
>> DVCS-centered site. That is, +1 to bitbucket, github, gitorious, ++, and
>> -1 to SourceForge, Google Code, etc.
>>      
> I guess it's bitbucket then? Switching the VCS just because of a project
> hosting site sounds like more trouble than I'm currently after.
>    

I haven't really used bitbucket. Only one I know who tried it is Kurt; 
do you have an opinion? Did you switch to github only because you wanted 
to switch VCS, or because of the site also?

Switching the repositories over is a no-brainer as it likely takes 15 
minutes, but it'd be nice to know if the tracker is up to scratch etc. 
before thinking about 1)-3).

Dag Sverre

From robertwb at math.washington.edu  Wed Nov 17 21:20:55 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 17 Nov 2010 12:20:55 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <4CE3ECCD.3030605@behnel.de>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<4CE3ECCD.3030605@behnel.de>
Message-ID: <AANLkTinmqoAdXuJtZ8mxMVQwaPmoKs=cefnGZ=DZ2s4h@mail.gmail.com>

On Wed, Nov 17, 2010 at 6:55 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 17.11.2010 09:59:
>> 1. The web site
>> 5. Buildbot
>>
>> In terms of the website itself, I think we should keep hosting that
>> (for maximum flexibility, and it's easy to administer)
>
> +1
>
>
>> and there
>> isn't much of an option for 5 (though steps should be taken to reduce
>> its load).
>
> I'm a big fan of timely commit feedback.

Me too, the build bot has been great.

> Hudson is already down to 6 build
> threads, so it takes a while now to finish the test suites. Also, the less
> parallel builds we run, the longer it disturbs other programs on the same
> machine.

Yes, though it disturbs them less. I think we'll be able to bump it
past 6 threads in the long run, it's just that things have been pretty
bad lately.

> Could you describe a little where you think the current pain points are? Is
> it more about the number of builds per day or the overall load during a
> single build? I/O or CPU load? My impression is that the /scratch/hudson
> mount point doesn't have very fast I/O, which lets the builds take longer
> than necessary.

I/O is a big pain point. /scratch/hudson isn't very fast IO, but it's
faster than the nsf mount. CPU peaks pretty badly too.

> A problem that I see is that Hudson only checks for changes every 10
> minutes in order to keep the load of the (fairly slow) Cython VM
> acceptable. So if a new commit arrives shortly after triggering the builds,
> it will continue to run the tests for 10 minutes before it starts over and
> does it all again. I tried changing that with the commit hook script but
> didn't get a response.

I looked and wasn't sure how to do it, contacted Harald Schilly (the
de-facto admin) and he wasn't sure, and then we started thinking about
migration, so this got put on hold.

The current system is particularly bad as you or I happen to push just
frequently enough to keep 12+ threads going at full tilt for the
entire time we're working on Cython.

> I don't know if bitbucket supports commit hooks, BTW.
>
> One way to get relief is to reduce the currently 9 jobs that all query the
> cython-devel repo for changes. I can try to change that by adding a
> dedicated source build job that does the polling and triggers the real
> build jobs that just download the sdist with no hg interaction. That moves
> the hg change history one step further away from the build result on the
> Hudson job page, but that won't hurt anyone, I guess.

I was actually thinking the exact same thing. Sdist is actually rather
io-intensive, but I'd imagine all jobs pulling locally (disk-to-disk)
triggered a single job that pulls from the web interface would be a
big improvement. Two other things that might help out would be (1)
doing much of the building on boxen's RAM disk and (2) having a
cascading system where, for example, 2.3 and 2.6 were fired off, and
only if all their tests passed would 2.4-2.6 run. (It seems like
there's a lot of wasteful testing going on for the non-version
specific failures.)

- Robert

From dagss at student.matnat.uio.no  Wed Nov 17 21:26:23 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 17 Nov 2010 21:26:23 +0100
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTi=zvAX0hodpvWDd=8Wy9aTV6m=Q8NCrw7Su_f0s@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de>
	<4CE380E0.7070403@behnel.de>	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>	<4CE3AF74.40703@student.matnat.uio.no>	<4CE3B0E9.7090304@student.matnat.uio.no>	<4CE3E607.2040801@behnel.de>
	<AANLkTi=zvAX0hodpvWDd=8Wy9aTV6m=Q8NCrw7Su_f0s@mail.gmail.com>
Message-ID: <4CE43A6F.6020104@student.matnat.uio.no>

On 11/17/2010 09:07 PM, Robert Bradshaw wrote:
> On Wed, Nov 17, 2010 at 6:26 AM, Stefan Behnel<stefan_ml at behnel.de>  wrote:
>    
>> Dag Sverre Seljebotn, 17.11.2010 11:39:
>>      
>>> On 11/17/2010 11:33 AM, Dag Sverre Seljebotn wrote:
>>>        
>>>> On 11/17/2010 09:59 AM, Robert Bradshaw wrote:
>>>>          
>>>>> So, I've been thinking about this, and one of the reasons we're using
>>>>> the current setup is that it requires very little administration from
>>>>> me, as I am essentially leaching of the sagemath.org infrastructure.
>>>>> Moving things to boxen would be a bit more administration on our part,
>>>>> but still not too bad, and I know William is a happy enough Cython
>>>>> user to be fine with continuing to host us :). However, I'm wondering
>>>>> if this would be a ripe occasion to making the leap to something like
>>>>> http://code.google.com. Currently, our infrastructure consists of
>>>>>
>>>>> 1. The web site
>>>>> 2. Trac
>>>>> 3. Wiki
>>>>> 4. Repositories
>>>>> 5. Buildbot
>>>>> 6. Mailing lists
>>>>>
>>>>> Currently we're hosting 1-5, and 6 is being hosted by codespeak.net
>>>>> (for cython-dev) and google (for cython-users). I think it may be
>>>>> worth considering moving 2-4 elsewhere, as there is little loss and
>>>>> they are the higher-maintenance (from an administrative point of view)
>>>>> items, and features such as code review tools would be nice to have as
>>>>> well.
>>>>>            
>> I see the serious advantage of simplifying the access to bug tracker, wiki,
>> etc. for new users.
>>      
> And for us old-timers too :)
>
>    
>> However, regarding the "little loss", does anyone have experience in
>> migrating trac tickets to a different tracker? There's a lot of information
>> in the tracker that I don't want to loose, and I wouldn't like to look at
>> two trackers to figure out if a bug has been reported.
>>      
> +1, this was one of my criteria too (hence the link to
> http://www.dataliberation.org/google/code-project-hosting , which I
> consider a critical part being open). However, the unfortunate
> geographic limitations seem to be a blocker. It looks like GitHub and
> BitBucket have APIs for importing/exporting issues as well, so we'd
> certainly migrate all our data. The wikis for all are stored in a DVCS
> repository, which is nice. Wiki history isn't as important, but I'd
> see if it wasn't too hard to scrape/grab.
>
>    
>>> Forgetting about the git aspect for a moment, I'm strongly in favor of a
>>> DVCS-centered site. That is, +1 to bitbucket, github, gitorious, ++, and
>>> -1 to SourceForge, Google Code, etc.
>>>        
> For sure. If Google Code didn't support the DVCS "click to clone"
> model I wouldn't have considered it.
>    

Ah ok, shows my ignorance.

>    
>> I guess it's bitbucket then? Switching the VCS just because of a project
>> hosting site sounds like more trouble than I'm currently after.
>>      
> The biggest drawback of bitbucket is the lack of integrated code
> review--if we're moving projects we might as well move to something
> that's a step up in this area. I would be happy switching to Git to
> use GitHub--my primary issue with it compared to mercurial is the
> steep(er) learning curve, but this is less of an issue for a rather
> technical project like ours.
>    


To quote Kurt: "The deciding factor, ultimately, was the hg-git 
mercurial plugin <http://hg-git.github.com/>, which allows someone to 
use a mercurial client with a git repository,/ with improved branch 
functionality/."

Using a git repo allows people to use either hg or git as their 
front-end, whichever they prefer. With a mercurial repo you can't 
currently use git transparently as the front-end (although, if Cython 
doesn't switch, I'll have to find a non-transparent way of working in 
git and move patches to hg -- after a couple of weeks there's no going 
back.)

http://fortrancython.wordpress.com/2010/08/25/fwrap-has-moved-to-github/


Dag Sverre
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/cython-dev/attachments/20101117/d3146efd/attachment.htm 

From dagss at student.matnat.uio.no  Wed Nov 17 21:35:48 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 17 Nov 2010 21:35:48 +0100
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTinmqoAdXuJtZ8mxMVQwaPmoKs=cefnGZ=DZ2s4h@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de>
	<4CE380E0.7070403@behnel.de>	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>	<4CE3ECCD.3030605@behnel.de>
	<AANLkTinmqoAdXuJtZ8mxMVQwaPmoKs=cefnGZ=DZ2s4h@mail.gmail.com>
Message-ID: <4CE43CA4.7010600@student.matnat.uio.no>

On 11/17/2010 09:20 PM, Robert Bradshaw wrote:
> On Wed, Nov 17, 2010 at 6:55 AM, Stefan Behnel<stefan_ml at behnel.de>  wrote:
>    
>> Robert Bradshaw, 17.11.2010 09:59:
>>      
>>> 1. The web site
>>> 5. Buildbot
>>>
>>> In terms of the website itself, I think we should keep hosting that
>>> (for maximum flexibility, and it's easy to administer)
>>>        
>> +1
>>      


An overall comment on Hudson: AFAIK, there's not very many buildbots set 
up for the SciPy-related projects, and they could probably use one.

I.e., long term, perhaps one could save some work by teaming up, share 
the workload, find some dedicated hardware etc.

I don't know nearly enough about current operations to judging the 
merits of this idea, I'm just throwing it out there.

Or, perhaps snakebite.org will become available for us eventually?

Dag Sverre

From dagss at student.matnat.uio.no  Wed Nov 17 22:15:22 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Wed, 17 Nov 2010 22:15:22 +0100
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <4CE3E607.2040801@behnel.de>
References: <4CE26F1A.4000607@behnel.de>	<4CE380E0.7070403@behnel.de>	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>	<4CE3AF74.40703@student.matnat.uio.no>	<4CE3B0E9.7090304@student.matnat.uio.no>
	<4CE3E607.2040801@behnel.de>
Message-ID: <4CE445EA.80209@student.matnat.uio.no>

On 11/17/2010 03:26 PM, Stefan Behnel wrote:
>
>> Forgetting about the git aspect for a moment, I'm strongly in favor of a
>> DVCS-centered site. That is, +1 to bitbucket, github, gitorious, ++, and
>> -1 to SourceForge, Google Code, etc.
>>      
> I guess it's bitbucket then? Switching the VCS just because of a project
> hosting site sounds like more trouble than I'm currently after.
>    

I'm not sure if it's any trouble at all. Converting cython-devel to a 
git repository on github took me about 15 minutes from I hit Google.

https://github.com/dagss/cython

It's just for testing, but you can perhaps see if hg-git allows you to 
completely transparently use the repo in the way you are used to, with hg:

http://hg-git.github.com/

The conversion tool (hg-fast-export) has a few options that I didn't 
read up on.

Dag Sverre

From stefan_ml at behnel.de  Wed Nov 17 23:08:31 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 17 Nov 2010 23:08:31 +0100
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <4CE43CA4.7010600@student.matnat.uio.no>
References: <4CE26F1A.4000607@behnel.de>	<4CE380E0.7070403@behnel.de>	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>	<4CE3ECCD.3030605@behnel.de>	<AANLkTinmqoAdXuJtZ8mxMVQwaPmoKs=cefnGZ=DZ2s4h@mail.gmail.com>
	<4CE43CA4.7010600@student.matnat.uio.no>
Message-ID: <4CE4525F.3080408@behnel.de>

Dag Sverre Seljebotn, 17.11.2010 21:35:
> Or, perhaps snakebite.org will become available for us eventually?

IIRC, snakebite has been "becoming available" for more than a year now. Not 
so long ago, the last thing I read was that they are still fighting issues 
here and there. I keep having my doubts that this will change any time soon.

Stefan

From stefan_ml at behnel.de  Thu Nov 18 00:01:24 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 18 Nov 2010 00:01:24 +0100
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTinmqoAdXuJtZ8mxMVQwaPmoKs=cefnGZ=DZ2s4h@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de>
	<4CE380E0.7070403@behnel.de>	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>	<4CE3ECCD.3030605@behnel.de>
	<AANLkTinmqoAdXuJtZ8mxMVQwaPmoKs=cefnGZ=DZ2s4h@mail.gmail.com>
Message-ID: <4CE45EC4.9060906@behnel.de>

Robert Bradshaw, 17.11.2010 21:20:
> On Wed, Nov 17, 2010 at 6:55 AM, Stefan Behnel wrote:
>> One way to get relief is to reduce the currently 9 jobs that all query the
>> cython-devel repo for changes. I can try to change that by adding a
>> dedicated source build job that does the polling and triggers the real
>> build jobs that just download the sdist with no hg interaction.

Done.

I've also disabled the hg polling in the cython-haoyu-build-* jobs.


> I was actually thinking the exact same thing. Sdist is actually rather
> io-intensive, but I'd imagine all jobs pulling locally (disk-to-disk)
> triggered a single job that pulls from the web interface would be a
> big improvement.

There's one sdist job now that finishes within seconds. It triggers all 
cython-devel-build-py* jobs that only copy over the sdist and build from 
it. Hudson's fingerprinting will keep track of the origin of the sources.

We also get all test results aggregated in one place that way:

https://sage.math.washington.edu:8091/hudson/job/cython-devel-sdist/lastBuild/aggregatedTestReport/


> Two other things that might help out would be (1)
> doing much of the building on boxen's RAM disk and (2) having a
> cascading system where, for example, 2.3 and 2.6 were fired off, and
> only if all their tests passed would 2.4-2.6 run. (It seems like
> there's a lot of wasteful testing going on for the non-version
> specific failures.)

Yes, I agree. (2) can be done, but let's see where the sdist change gets us 
so far.

Stefan

From robertwb at math.washington.edu  Thu Nov 18 00:20:31 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 17 Nov 2010 15:20:31 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <4CE45EC4.9060906@behnel.de>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<4CE3ECCD.3030605@behnel.de>
	<AANLkTinmqoAdXuJtZ8mxMVQwaPmoKs=cefnGZ=DZ2s4h@mail.gmail.com>
	<4CE45EC4.9060906@behnel.de>
Message-ID: <AANLkTimPnFqXxdKHVUn6M9j10-LRq07Hsg0vrOH6iiUo@mail.gmail.com>

On Wed, Nov 17, 2010 at 3:01 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 17.11.2010 21:20:
>> On Wed, Nov 17, 2010 at 6:55 AM, Stefan Behnel wrote:
>>> One way to get relief is to reduce the currently 9 jobs that all query the
>>> cython-devel repo for changes. I can try to change that by adding a
>>> dedicated source build job that does the polling and triggers the real
>>> build jobs that just download the sdist with no hg interaction.
>
> Done.
>
> I've also disabled the hg polling in the cython-haoyu-build-* jobs.
>
>
>> I was actually thinking the exact same thing. Sdist is actually rather
>> io-intensive, but I'd imagine all jobs pulling locally (disk-to-disk)
>> triggered a single job that pulls from the web interface would be a
>> big improvement.
>
> There's one sdist job now that finishes within seconds. It triggers all
> cython-devel-build-py* jobs that only copy over the sdist and build from
> it. Hudson's fingerprinting will keep track of the origin of the sources.
>
> We also get all test results aggregated in one place that way:
>
> https://sage.math.washington.edu:8091/hudson/job/cython-devel-sdist/lastBuild/aggregatedTestReport/

Cool. Thanks.

>> Two other things that might help out would be (1)
>> doing much of the building on boxen's RAM disk and (2) having a
>> cascading system where, for example, 2.3 and 2.6 were fired off, and
>> only if all their tests passed would 2.4-2.6 run. (It seems like
>> there's a lot of wasteful testing going on for the non-version
>> specific failures.)
>
> Yes, I agree. (2) can be done, but let's see where the sdist change gets us
> so far.

Sure.

- Robert

From robertwb at math.washington.edu  Thu Nov 18 05:22:52 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Wed, 17 Nov 2010 20:22:52 -0800
Subject: [Cython] [bug] inf loop
In-Reply-To: <AANLkTi=bWs0X8peV0fQQeQx4_okvduQue_f5nJD75Nta@mail.gmail.com>
References: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>
	<AANLkTi=bWs0X8peV0fQQeQx4_okvduQue_f5nJD75Nta@mail.gmail.com>
Message-ID: <AANLkTinxq+W8kOb=xdv9LUjKBfkAG0-+g-GW5LCbSExw@mail.gmail.com>

On Wed, Nov 17, 2010 at 2:33 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2010/11/16 Vitja Makarov <vitja.makarov at gmail.com>:
>> Hi!
>>
>> I'm now trying to implement nested classes and found some errors in code.
>>
>> genv = env
>> while genv.is_py_class_scope or genv.is_c_class_scope:
>> ? ?genv = env.outer_scope
>>
>> Can cause infinty loop. Small patch is attached.
>>
>> vitja.
>>
>
> I've created ticket for #596 for class closures.

Very cool.

- Robert

From craigcitro at gmail.com  Thu Nov 18 09:13:15 2010
From: craigcitro at gmail.com (Craig Citro)
Date: Thu, 18 Nov 2010 00:13:15 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
Message-ID: <AANLkTin2SsA2Siwfsq+Zg6V3U9q7TEDRdJ1y9iKnRAMh@mail.gmail.com>

I'm a little late to this party, it seems, but I wanted to throw in my vote.

> 1. The web site
> 2. Trac
> 3. Wiki
> 4. Repositories
> 5. Buildbot
> 6. Mailing lists
>

I think these + code review make for a pretty good list, and I'd fight
hard to claim code review integrated with a bug tracker should be a
requirement. Personally, I'm a big +1 on moving everything to either
github or code.google.com, which seems like the best of the choices.
If people really think the geographic restrictions are a problem, I
guess I'm fine with dropping code.google.com off the list -- but just
for the sake of completeness, are there any features code.google.com
has that github doesn't? (Or vice-versa?) Are they things we'd ever be
likely to notice?

I'm also more than happy to switch to git for Cython, especially in
light of the hg-on-git stuff Dag mentioned. My sense is that github
definitely seems to be the "new hotness" -- I'll admit that there's
something about this I don't "get," but if that just means more people
might notice Cython, I'd say that's a win. I have one or two questions
to put out there:

 * Does github have any sort of mailing list support? I'd love to see
cython-dev migrated to something with better searchability, linking,
archives, etc -- I don't know that there's any special link between
google groups and code.google.com, so this is really orthogonal to the
project hosting question. I know the decision to use codespeak was
made as an explicit decision against google groups, but my
understanding is that whatever objections there were are now gone. Is
that still the case?

 * Has anyone actually been an admin on a github project? The tools
sure sound good from the outside, but are they fairly pleasant to use?
Trading writing our own apache configs for regularly fighting with,
say, flaky web interfaces is decidedly *not* a win. (I've done some
minor admin on a code.google.com project, and generally had good
experiences.)

 * Are there a bunch of examples of projects that host basically
everything out of github? I'd be curious to check one out, just for
the sake of it.

 * I do have one minor worry about github, which may have something to
do with me not understanding why it's such a phenomenon. The whole
model seems to really take the D in DVCS to the extreme -- so much so
that I often have a hard time deciding what the "central" repo for a
project is (in cases where there is such a thing) just based on what I
see on github, or how to get there when I happen to end up in a random
person's branch (say from a google search). My vague impression is
that the model for code.google.com is "project-centric," whereas the
github model is much more "developer-centric." Is there something I'm
missing?

-cc

From stefan_ml at behnel.de  Thu Nov 18 09:38:24 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 18 Nov 2010 09:38:24 +0100
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTin2SsA2Siwfsq+Zg6V3U9q7TEDRdJ1y9iKnRAMh@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de>
	<4CE380E0.7070403@behnel.de>	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<AANLkTin2SsA2Siwfsq+Zg6V3U9q7TEDRdJ1y9iKnRAMh@mail.gmail.com>
Message-ID: <4CE4E600.2050302@behnel.de>

Craig Citro, 18.11.2010 09:13:
>   * Does github have any sort of mailing list support? I'd love to see
> cython-dev migrated

-1


> to something with better searchability, linking, archives, etc

What's wrong with all the archives of the current mailing list? Doesn't at 
least one of them suite your way of using a ML archive? Gmane alone has 
three different web interfaces.

And in what way is "linking" different from "posting a link" or "referring 
to an archive post"? I never had any problem with either of those.


> -- I don't know that there's any special link between
> google groups and code.google.com, so this is really orthogonal to the
> project hosting question. I know the decision to use codespeak was
> made as an explicit decision against google groups, but my
> understanding is that whatever objections there were are now gone. Is
> that still the case?

 From comparing the ML related comments on cython-dev and cython-users, I 
got the impression that it wasn't that a good idea to have a mailing list 
on google-groups. I remember reading things like "I thought top-posting was 
the right thing to do on google-groups" more than once. An environment that 
doesn't help people to do the right thing doesn't feel welcoming to me. And 
I really hate the way google-groups presents archived threads. Makes them 
totally hard to read.

Stefan

From fperez.net at gmail.com  Thu Nov 18 09:45:45 2010
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 18 Nov 2010 00:45:45 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <4CE3E607.2040801@behnel.de>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<4CE3AF74.40703@student.matnat.uio.no>
	<4CE3B0E9.7090304@student.matnat.uio.no>
	<4CE3E607.2040801@behnel.de>
Message-ID: <AANLkTimEVH_wD-D38eP6Ubc8yjiLYb3Tc4AiH18BAV=a@mail.gmail.com>

On Wed, Nov 17, 2010 at 6:26 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> However, regarding the "little loss", does anyone have experience in
> migrating trac tickets to a different tracker? There's a lot of information
> in the tracker that I don't want to loose, and I wouldn't like to look at
> two trackers to figure out if a bug has been reported.
>
>

I migrated the ipython bug history from launchpad to github. If you
guys decide to go to github, ping me for the code (you can ignore the
half that pulled from LP, but reuse the half that pushed).

Cheers,

f

From fperez.net at gmail.com  Thu Nov 18 09:50:53 2010
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 18 Nov 2010 00:50:53 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <4CE3AF74.40703@student.matnat.uio.no>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<4CE3AF74.40703@student.matnat.uio.no>
Message-ID: <AANLkTi=+AktpDE=yBrXFQWKbC+NKLc4AQfwYTuFhSzxL@mail.gmail.com>

On Wed, Nov 17, 2010 at 2:33 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
>
> ?- Static web site content (you just create a repository which it
> serves as static content, e.g, http://cournape.github.com/Bento/)

If you go down the github route, I'd suggest a slightly different
alternative than github's default for the creation of static content.
Their approach uses a hidden DAG in the repo called gh-pages, which
eternally pollutes your repo with all static html you generate.  It's
easy with a tiny bit of scripting to create instead the gh-pages
content in a *separate repo* completely decoupled from the main one.
That's what we do here:

https://github.com/fperez/datarray  # main repo
http://fperez.github.com/datarray-doc/  # gh-pages in separate repo.

The tool that does the job is this trivial script:

https://github.com/fperez/datarray/blob/master/doc/gh-pages.py

I'll be happy to provide pointers if you want them.

I just wanted to let you know so that you avoid what I think is a
poor, but easy to work around, design choice of github's pages
implementation.

Regards,

f

From fperez.net at gmail.com  Thu Nov 18 10:00:21 2010
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 18 Nov 2010 01:00:21 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTin2SsA2Siwfsq+Zg6V3U9q7TEDRdJ1y9iKnRAMh@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<AANLkTin2SsA2Siwfsq+Zg6V3U9q7TEDRdJ1y9iKnRAMh@mail.gmail.com>
Message-ID: <AANLkTi=KhpaC4rm-M-UkzrJUjOTMX5XRaH_uvhPzEV=T@mail.gmail.com>

On Thu, Nov 18, 2010 at 12:13 AM, Craig Citro <craigcitro at gmail.com> wrote:
> but just
> for the sake of completeness, are there any features code.google.com
> has that github doesn't? (Or vice-versa?) Are they things we'd ever be
> likely to notice?

I say this as a flat out github fan, who's thrilled we moved for
IPython and everything else I'm involved with.  But... the github bug
tracker is *vastly* inferior to google's.  I keep hoping that, with
how great github is getting very quickly, they'll soon roll out
something to improve the bug tracker, because it's frankly painfully
primitive.

Whoever thought out the google code bug tracker was a smart cookie.
It relies on what appears to be a plain labels system, but with very
simple conventions on how to declare the labels, it basically encodes
a full key-value system on which queries can later be made, reports,
etc.  It's a really elegant piece of work:

http://code.google.com/p/support/wiki/IssueTracker

The github tracker is dumb as a brick in comparison, and their search
doesn't even work right.  It's frankly embarrassing, and it sticks out
as a sore thumb compared to how great pretty much everything else
there is.

It's only the fact that the contrast is so stark, that makes me hope
it won't last :)

Regards,

f

From fperez.net at gmail.com  Thu Nov 18 10:06:26 2010
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 18 Nov 2010 01:06:26 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTin2SsA2Siwfsq+Zg6V3U9q7TEDRdJ1y9iKnRAMh@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<AANLkTin2SsA2Siwfsq+Zg6V3U9q7TEDRdJ1y9iKnRAMh@mail.gmail.com>
Message-ID: <AANLkTi=E+eCf5Rs_MiNA=E60SCyh3NqZhAaD1BbRQJrb@mail.gmail.com>

On Thu, Nov 18, 2010 at 12:13 AM, Craig Citro <craigcitro at gmail.com> wrote:
> ?* Has anyone actually been an admin on a github project? The tools
> sure sound good from the outside, but are they fairly pleasant to use?
> Trading writing our own apache configs for regularly fighting with,
> say, flaky web interfaces is decidedly *not* a win. (I've done some
> minor admin on a code.google.com project, and generally had good
> experiences.)

Modulo my irritation with their bug tracker, everything else so far
has been pretty pleasant.  And I guess I also dislike their default
gh-pages implementation, but as indicated above, the workaround was so
easy that I don't think about it anymore.

> ?* I do have one minor worry about github, which may have something to
> do with me not understanding why it's such a phenomenon. The whole
> model seems to really take the D in DVCS to the extreme -- so much so
> that I often have a hard time deciding what the "central" repo for a
> project is (in cases where there is such a thing) just based on what I
> see on github, or how to get there when I happen to end up in a random
> person's branch (say from a google search). My vague impression is
> that the model for code.google.com is "project-centric," whereas the
> github model is much more "developer-centric." Is there something I'm
> missing?

This was a very real issue, but their implementation of
'organizations' is extremely good, and takes care of it.  IPython for
example is an organization:

http://github.com/ipython/

that hosts the official ipython repo:

http://github.com/ipython/ipython/

It has teams, repos, and permissions can be assigned to teams as
needed.  The setup is very, very easy and pleasant to work with.  It's
similar to launchpad's project/team concept but actually done right:
LP allowed the Unix user/group permission model leak out into their
website design, and it's a disaster (you can't assign individuals
permissions to share a branch without making a whole new team -aka
Unix group- for it).  In contrast, github's organization setup makes
it very easy to have separate repos in a project (say core
development, website sources, old branches, etc), have separate teams
that work on each repo, and have the individual developers keep their
personal branches from which they do pull requests into the official
repo.

HTH,

f

From dagss at student.matnat.uio.no  Thu Nov 18 10:18:25 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 18 Nov 2010 10:18:25 +0100
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <4CE4E600.2050302@behnel.de>
References: <4CE26F1A.4000607@behnel.de>	<4CE380E0.7070403@behnel.de>	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>	<AANLkTin2SsA2Siwfsq+Zg6V3U9q7TEDRdJ1y9iKnRAMh@mail.gmail.com>
	<4CE4E600.2050302@behnel.de>
Message-ID: <4CE4EF61.5070601@student.matnat.uio.no>

On 11/18/2010 09:38 AM, Stefan Behnel wrote:
> Craig Citro, 18.11.2010 09:13:
>    
>>    * Does github have any sort of mailing list support? I'd love to see
>> cython-dev migrated
>>      
> -1
>
>    
I agree, no sense to take the trouble until we need to. But once we do, 
http://librelist.com seems pretty decent from a first glance.

Dag Sverre

From dagss at student.matnat.uio.no  Thu Nov 18 10:24:06 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 18 Nov 2010 10:24:06 +0100
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTi=+AktpDE=yBrXFQWKbC+NKLc4AQfwYTuFhSzxL@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de>
	<4CE380E0.7070403@behnel.de>	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>	<4CE3AF74.40703@student.matnat.uio.no>
	<AANLkTi=+AktpDE=yBrXFQWKbC+NKLc4AQfwYTuFhSzxL@mail.gmail.com>
Message-ID: <4CE4F0B6.6020208@student.matnat.uio.no>

On 11/18/2010 09:50 AM, Fernando Perez wrote:
> On Wed, Nov 17, 2010 at 2:33 AM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no>  wrote:
>    
>>   - Static web site content (you just create a repository which it
>> serves as static content, e.g, http://cournape.github.com/Bento/)
>>      
> If you go down the github route, I'd suggest a slightly different
> alternative than github's default for the creation of static content.
> Their approach uses a hidden DAG in the repo called gh-pages, which
> eternally pollutes your repo with all static html you generate.  It's
> easy with a tiny bit of scripting to create instead the gh-pages
> content in a *separate repo* completely decoupled from the main one.
> That's what we do here:
>    

Also, for the main page (http://fperez.github.com/ ) there's always a 
seperate repo, right? The gh-pages branch is just for serving content in 
folders that has the same name as one of your repos?

I agree, I'm surprised they didn't think of anything better here. Thanks 
for the workaround.

Dag Sverre

From dagss at student.matnat.uio.no  Thu Nov 18 10:27:33 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 18 Nov 2010 10:27:33 +0100
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTi=KhpaC4rm-M-UkzrJUjOTMX5XRaH_uvhPzEV=T@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de>
	<4CE380E0.7070403@behnel.de>	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>	<AANLkTin2SsA2Siwfsq+Zg6V3U9q7TEDRdJ1y9iKnRAMh@mail.gmail.com>
	<AANLkTi=KhpaC4rm-M-UkzrJUjOTMX5XRaH_uvhPzEV=T@mail.gmail.com>
Message-ID: <4CE4F185.4070102@student.matnat.uio.no>

On 11/18/2010 10:00 AM, Fernando Perez wrote:
> On Thu, Nov 18, 2010 at 12:13 AM, Craig Citro<craigcitro at gmail.com>  wrote:
>    
>> but just
>> for the sake of completeness, are there any features code.google.com
>> has that github doesn't? (Or vice-versa?) Are they things we'd ever be
>> likely to notice?
>>      
> I say this as a flat out github fan, who's thrilled we moved for
> IPython and everything else I'm involved with.  But... the github bug
> tracker is *vastly* inferior to google's.  I keep hoping that, with
> how great github is getting very quickly, they'll soon roll out
> something to improve the bug tracker, because it's frankly painfully
> primitive.
>
> Whoever thought out the google code bug tracker was a smart cookie.
> It relies on what appears to be a plain labels system, but with very
> simple conventions on how to declare the labels, it basically encodes
> a full key-value system on which queries can later be made, reports,
> etc.  It's a really elegant piece of work:
>
> http://code.google.com/p/support/wiki/IssueTracker
>
> The github tracker is dumb as a brick in comparison, and their search
> doesn't even work right.  It's frankly embarrassing, and it sticks out
> as a sore thumb compared to how great pretty much everything else
> there is.
>
> It's only the fact that the contrast is so stark, that makes me hope
> it won't last :)
>    

Hmm. It seems that GitHub itself uses Lighthouse (lighthouseapp.com), 
and that there's some Lighthous/Github integration.

Lighthouse appears to be free for open source projects, although it is a 
bit of work to check how well the Github integration works.

Dag Sverre

From stefan_ml at behnel.de  Thu Nov 18 15:24:37 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 18 Nov 2010 15:24:37 +0100
Subject: [Cython] [bug] inf loop
In-Reply-To: <AANLkTi=bWs0X8peV0fQQeQx4_okvduQue_f5nJD75Nta@mail.gmail.com>
References: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>
	<AANLkTi=bWs0X8peV0fQQeQx4_okvduQue_f5nJD75Nta@mail.gmail.com>
Message-ID: <4CE53725.5050707@behnel.de>

Vitja Makarov, 17.11.2010 11:33:
> I've created ticket for #596 for class closures.

The patch shows issues when running the CPython regression tests, e.g. with 
the "--sys-pyregr" option (assuming you have the CPython "test" package 
installed).

Several modules fail to compile at the C level, and the "test_call" test 
doesn't terminate in Py2.7.

Stefan

From matthew.brett at gmail.com  Thu Nov 18 18:36:26 2010
From: matthew.brett at gmail.com (Matthew Brett)
Date: Thu, 18 Nov 2010 09:36:26 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTin2SsA2Siwfsq+Zg6V3U9q7TEDRdJ1y9iKnRAMh@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<AANLkTin2SsA2Siwfsq+Zg6V3U9q7TEDRdJ1y9iKnRAMh@mail.gmail.com>
Message-ID: <AANLkTinJm1z5_+2_maTn2GTDy5jB51d=NdGz_jhb+xPq@mail.gmail.com>

Hi,

On Thu, Nov 18, 2010 at 12:13 AM, Craig Citro <craigcitro at gmail.com> wrote:
...
> If people really think the geographic restrictions are a problem, I
> guess I'm fine with dropping code.google.com off the list

I'm sorry to return to this but 'geographical restrictions' is a
euphemism.  The term you want here is 'political restrictions'.  And
yes, I do 'really' think they are a problem, in principle.  I think
most open-source programmers care a lot about the freedom implied in
code sharing, and it matters when that freedom is eroded or
compromised.

Best,

Matthew

From vitja.makarov at gmail.com  Thu Nov 18 19:03:09 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 18 Nov 2010 21:03:09 +0300
Subject: [Cython] [bug] inf loop
In-Reply-To: <4CE53725.5050707@behnel.de>
References: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>
	<AANLkTi=bWs0X8peV0fQQeQx4_okvduQue_f5nJD75Nta@mail.gmail.com>
	<4CE53725.5050707@behnel.de>
Message-ID: <AANLkTi=s_tFgLxGXnb5xQxyjwc8gyquTuUBOGkNgr-d5@mail.gmail.com>

2010/11/18 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 17.11.2010 11:33:
>> I've created ticket for #596 for class closures.
>
> The patch shows issues when running the CPython regression tests, e.g. with
> the "--sys-pyregr" option (assuming you have the CPython "test" package
> installed).
>
> Several modules fail to compile at the C level, and the "test_call" test
> doesn't terminate in Py2.7.
>

I'll take a look. Do you mean test_capi doesn't terminate?

From vitja.makarov at gmail.com  Thu Nov 18 20:33:36 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 18 Nov 2010 22:33:36 +0300
Subject: [Cython] [bug] inf loop
In-Reply-To: <AANLkTi=s_tFgLxGXnb5xQxyjwc8gyquTuUBOGkNgr-d5@mail.gmail.com>
References: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>
	<AANLkTi=bWs0X8peV0fQQeQx4_okvduQue_f5nJD75Nta@mail.gmail.com>
	<4CE53725.5050707@behnel.de>
	<AANLkTi=s_tFgLxGXnb5xQxyjwc8gyquTuUBOGkNgr-d5@mail.gmail.com>
Message-ID: <AANLkTimAYXqVOBWU_Bdr_ihP9_s5g3iEvZf0TJgmcVDN@mail.gmail.com>

2010/11/18 Vitja Makarov <vitja.makarov at gmail.com>:
> 2010/11/18 Stefan Behnel <stefan_ml at behnel.de>:
>> Vitja Makarov, 17.11.2010 11:33:
>>> I've created ticket for #596 for class closures.
>>
>> The patch shows issues when running the CPython regression tests, e.g. with
>> the "--sys-pyregr" option (assuming you have the CPython "test" package
>> installed).
>>
>> Several modules fail to compile at the C level, and the "test_call" test
>> doesn't terminate in Py2.7.
>>
>
> I'll take a look. Do you mean test_capi doesn't terminate?
>

Btw test_capi doesn't get reached w/o my patch.
On the other hand TestPendingCalls

hang too, I'm not sure why, but try this code:

import _testcapi
import time
import random

def callback():
    print 'callback'

for i in xrange(64):
    time.sleep(random.random()*0.02)
    print 'Try', i
    while True:
        if _testcapi._pending_threadfunc(callback):
            break


Fails too, it seems to me to be depended on GIL which is never
released in Cython code.
So I think that for now it's better to disable test_capi.

Didn't yet take time to look at other issues.

-- 
vitja.

From stefan_ml at behnel.de  Thu Nov 18 20:52:57 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 18 Nov 2010 20:52:57 +0100
Subject: [Cython] [bug] inf loop
In-Reply-To: <AANLkTimAYXqVOBWU_Bdr_ihP9_s5g3iEvZf0TJgmcVDN@mail.gmail.com>
References: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>	<AANLkTi=bWs0X8peV0fQQeQx4_okvduQue_f5nJD75Nta@mail.gmail.com>	<4CE53725.5050707@behnel.de>	<AANLkTi=s_tFgLxGXnb5xQxyjwc8gyquTuUBOGkNgr-d5@mail.gmail.com>
	<AANLkTimAYXqVOBWU_Bdr_ihP9_s5g3iEvZf0TJgmcVDN@mail.gmail.com>
Message-ID: <4CE58419.7070000@behnel.de>

Vitja Makarov, 18.11.2010 20:33:
> 2010/11/18 Vitja Makarov:
>> 2010/11/18 Stefan Behnel:
>>> Vitja Makarov, 17.11.2010 11:33:
>>>> I've created ticket for #596 for class closures.
>>>
>>> The patch shows issues when running the CPython regression tests, e.g. with
>>> the "--sys-pyregr" option (assuming you have the CPython "test" package
>>> installed).
>>>
>>> Several modules fail to compile at the C level, and the "test_call" test
>>> doesn't terminate in Py2.7.
>>>
>>
>> I'll take a look. Do you mean test_capi doesn't terminate?

Ah, right. It hang *after* running test_call, but only showed the name of 
the next running test after I killed it.

https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py27-c/130/console


> Btw test_capi doesn't get reached w/o my patch.
> On the other hand TestPendingCalls
>
> hang too, I'm not sure why, but try this code:
>
> import _testcapi
> import time
> import random
>
> def callback():
>      print 'callback'
>
> for i in xrange(64):
>      time.sleep(random.random()*0.02)
>      print 'Try', i
>      while True:
>          if _testcapi._pending_threadfunc(callback):
>              break
>
>
> Fails too, it seems to me to be depended on GIL which is never
> released in Cython code.
> So I think that for now it's better to disable test_capi.

Right, sorry for the noise. Should have looked closer. It's not the first 
pyregr test that I disable, and likely won't be the last one either.


> Didn't yet take time to look at other issues.

The C compiler problems looked like closure issues, so they are more likely 
to be related.

Stefan

From vitja.makarov at gmail.com  Thu Nov 18 21:17:09 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 18 Nov 2010 23:17:09 +0300
Subject: [Cython] [bug] inf loop
In-Reply-To: <4CE58419.7070000@behnel.de>
References: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>
	<AANLkTi=bWs0X8peV0fQQeQx4_okvduQue_f5nJD75Nta@mail.gmail.com>
	<4CE53725.5050707@behnel.de>
	<AANLkTi=s_tFgLxGXnb5xQxyjwc8gyquTuUBOGkNgr-d5@mail.gmail.com>
	<AANLkTimAYXqVOBWU_Bdr_ihP9_s5g3iEvZf0TJgmcVDN@mail.gmail.com>
	<4CE58419.7070000@behnel.de>
Message-ID: <AANLkTi=v8BVSLduQkqmmmOZ1sKp9XSsx5XeMcU8FR7MA@mail.gmail.com>

2010/11/18 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 18.11.2010 20:33:
>> 2010/11/18 Vitja Makarov:
>>> 2010/11/18 Stefan Behnel:
>>>> Vitja Makarov, 17.11.2010 11:33:
>>>>> I've created ticket for #596 for class closures.
>>>>
>>>> The patch shows issues when running the CPython regression tests, e.g. with
>>>> the "--sys-pyregr" option (assuming you have the CPython "test" package
>>>> installed).
>>>>
>>>> Several modules fail to compile at the C level, and the "test_call" test
>>>> doesn't terminate in Py2.7.
>>>>
>>>
>>> I'll take a look. Do you mean test_capi doesn't terminate?
>
> Ah, right. It hang *after* running test_call, but only showed the name of
> the next running test after I killed it.
>
> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py27-c/130/console
>
>
>> Btw test_capi doesn't get reached w/o my patch.
>> On the other hand TestPendingCalls
>>
>> hang too, I'm not sure why, but try this code:
>>
>> import _testcapi
>> import time
>> import random
>>
>> def callback():
>> ? ? ?print 'callback'
>>
>> for i in xrange(64):
>> ? ? ?time.sleep(random.random()*0.02)
>> ? ? ?print 'Try', i
>> ? ? ?while True:
>> ? ? ? ? ?if _testcapi._pending_threadfunc(callback):
>> ? ? ? ? ? ? ?break
>>
>>
>> Fails too, it seems to me to be depended on GIL which is never
>> released in Cython code.
>> So I think that for now it's better to disable test_capi.
>
> Right, sorry for the noise. Should have looked closer. It's not the first
> pyregr test that I disable, and likely won't be the last one either.
>
>
>> Didn't yet take time to look at other issues.
>
> The C compiler problems looked like closure issues, so they are more likely
> to be related.
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>


Is compile problem logged somewhere?

I'm now running tests against class closure patch. But it would take a while.
Can you please point me?

-- 
vitja.

From stefan_ml at behnel.de  Thu Nov 18 21:25:16 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 18 Nov 2010 21:25:16 +0100
Subject: [Cython] [bug] inf loop
In-Reply-To: <AANLkTi=v8BVSLduQkqmmmOZ1sKp9XSsx5XeMcU8FR7MA@mail.gmail.com>
References: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>	<AANLkTi=bWs0X8peV0fQQeQx4_okvduQue_f5nJD75Nta@mail.gmail.com>	<4CE53725.5050707@behnel.de>	<AANLkTi=s_tFgLxGXnb5xQxyjwc8gyquTuUBOGkNgr-d5@mail.gmail.com>	<AANLkTimAYXqVOBWU_Bdr_ihP9_s5g3iEvZf0TJgmcVDN@mail.gmail.com>	<4CE58419.7070000@behnel.de>
	<AANLkTi=v8BVSLduQkqmmmOZ1sKp9XSsx5XeMcU8FR7MA@mail.gmail.com>
Message-ID: <4CE58BAC.6020105@behnel.de>

Vitja Makarov, 18.11.2010 21:17:
> 2010/11/18 Stefan Behnel:
>> The C compiler problems looked like closure issues, so they are more likely
>> to be related.
>
> Is compile problem logged somewhere?

You can find most of them here:

https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py26-c/704/parsed_console/

Look out for "has no member named".

Stefan

From craigcitro at gmail.com  Thu Nov 18 21:41:25 2010
From: craigcitro at gmail.com (Craig Citro)
Date: Thu, 18 Nov 2010 12:41:25 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTinJm1z5_+2_maTn2GTDy5jB51d=NdGz_jhb+xPq@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<AANLkTin2SsA2Siwfsq+Zg6V3U9q7TEDRdJ1y9iKnRAMh@mail.gmail.com>
	<AANLkTinJm1z5_+2_maTn2GTDy5jB51d=NdGz_jhb+xPq@mail.gmail.com>
Message-ID: <AANLkTimsQ+ezBTps9TRRnVueoqX6aad_7FxuAVekdqD2@mail.gmail.com>

> I'm sorry to return to this but 'geographical restrictions' is a
> euphemism. ?The term you want here is 'political restrictions'. ?And
> yes, I do 'really' think they are a problem, in principle. ?I think
> most open-source programmers care a lot about the freedom implied in
> code sharing, and it matters when that freedom is eroded or
> compromised.
>

Yep, you're right, they're political restrictions, and "geographic
restrictions" was a euphemism. I'm a huge believer in open source, but
personally, these political restrictions don't bother me at all.
However, you and Stefan had already both chimed in saying they did
bother you, so I'm very happy to pick another hosting option with no
argument, which is what I think I said above. I generally loathe
political debate, and I think cython-dev is a wildly inappropriate
place for it, so I was happy to stick with trying to avoid it
altogether -- hence the wording that tried to sidestep the political
issues. I'm just trying to figure out what (if anything) we're giving
up by dropping code.google.com from the list.

-cc

From matthew.brett at gmail.com  Thu Nov 18 22:03:48 2010
From: matthew.brett at gmail.com (Matthew Brett)
Date: Thu, 18 Nov 2010 13:03:48 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTimsQ+ezBTps9TRRnVueoqX6aad_7FxuAVekdqD2@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<AANLkTin2SsA2Siwfsq+Zg6V3U9q7TEDRdJ1y9iKnRAMh@mail.gmail.com>
	<AANLkTinJm1z5_+2_maTn2GTDy5jB51d=NdGz_jhb+xPq@mail.gmail.com>
	<AANLkTimsQ+ezBTps9TRRnVueoqX6aad_7FxuAVekdqD2@mail.gmail.com>
Message-ID: <AANLkTikX6tC2_tmAFkiiNvOW4L3pWUpujM147k1eYw5t@mail.gmail.com>

Hi,

> Yep, you're right, they're political restrictions, and "geographic
> restrictions" was a euphemism. I'm a huge believer in open source, but
> personally, these political restrictions don't bother me at all.
> However, you and Stefan had already both chimed in saying they did
> bother you, so I'm very happy to pick another hosting option with no
> argument, which is what I think I said above. I generally loathe
> political debate, and I think cython-dev is a wildly inappropriate
> place for it.

If we see a powerful country trying to exclude less powerful countries
from what we believe to be valid discourse, then, to say and do
nothing is an act of passive agreement that not all of us can accept.

We state our principles, and we try to follow them; when we agree, we
do not have to discuss them.  The principles under which we operate
are one of the great strengths of the open-source idea and the energy
that we give to our projects.

Best,

Matthew

From fperez.net at gmail.com  Thu Nov 18 22:14:53 2010
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 18 Nov 2010 13:14:53 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <4CE4F0B6.6020208@student.matnat.uio.no>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<4CE3AF74.40703@student.matnat.uio.no>
	<AANLkTi=+AktpDE=yBrXFQWKbC+NKLc4AQfwYTuFhSzxL@mail.gmail.com>
	<4CE4F0B6.6020208@student.matnat.uio.no>
Message-ID: <AANLkTikUjGwhn38z3ACMhop-KaRUZnVzuEEWEmVsVR_U@mail.gmail.com>

On Thu, Nov 18, 2010 at 1:24 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Also, for the main page (http://fperez.github.com/ ) there's always a
> seperate repo, right? The gh-pages branch is just for serving content in
> folders that has the same name as one of your repos?

Correct.

> I agree, I'm surprised they didn't think of anything better here. Thanks
> for the workaround.

My pleasure, I'll be happy to help if you guys end up going this way
and have any questions.

f

From fperez.net at gmail.com  Thu Nov 18 22:17:08 2010
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 18 Nov 2010 13:17:08 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <4CE4F185.4070102@student.matnat.uio.no>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<AANLkTin2SsA2Siwfsq+Zg6V3U9q7TEDRdJ1y9iKnRAMh@mail.gmail.com>
	<AANLkTi=KhpaC4rm-M-UkzrJUjOTMX5XRaH_uvhPzEV=T@mail.gmail.com>
	<4CE4F185.4070102@student.matnat.uio.no>
Message-ID: <AANLkTi=Yoj4+sHyc3vkKx_N2siJMtfa5q2WRt_B69F2L@mail.gmail.com>

On Thu, Nov 18, 2010 at 1:27 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Hmm. It seems that GitHub itself uses Lighthouse (lighthouseapp.com),
> and that there's some Lighthous/Github integration.
>
> Lighthouse appears to be free for open source projects, although it is a
> bit of work to check how well the Github integration works.

I think you're mistaken: github offers webhooks integration with
lighthouse, but their own default issues implementation doesn't seem
to be a lighthouse one. LH seems to have milestones, proper queries,
attachments, issues-by-email, etc., all features sorely missing from
the GH issues implementation (in its current state).

Cheers,

f

From stefan_ml at behnel.de  Thu Nov 18 22:18:54 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 18 Nov 2010 22:18:54 +0100
Subject: [Cython] metaclasses
In-Reply-To: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
References: <AANLkTik80DQ9A-vSwa_aBMSPr37CHRyztscgWgqTLcO1@mail.gmail.com>
Message-ID: <4CE5983E.9040403@behnel.de>

Vitja Makarov, 01.11.2010 08:46:
> I want to see metaclasses in cython.

I just pushed your latest patch to cython-devel (and added a couple of 
cleanups).

Thanks a lot, you did a really good job here!

Stefan

From dagss at student.matnat.uio.no  Thu Nov 18 22:42:00 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Thu, 18 Nov 2010 22:42:00 +0100
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTi=Yoj4+sHyc3vkKx_N2siJMtfa5q2WRt_B69F2L@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de>
	<4CE380E0.7070403@behnel.de>	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>	<AANLkTin2SsA2Siwfsq+Zg6V3U9q7TEDRdJ1y9iKnRAMh@mail.gmail.com>	<AANLkTi=KhpaC4rm-M-UkzrJUjOTMX5XRaH_uvhPzEV=T@mail.gmail.com>	<4CE4F185.4070102@student.matnat.uio.no>
	<AANLkTi=Yoj4+sHyc3vkKx_N2siJMtfa5q2WRt_B69F2L@mail.gmail.com>
Message-ID: <4CE59DA8.2040001@student.matnat.uio.no>

On 11/18/2010 10:17 PM, Fernando Perez wrote:
> On Thu, Nov 18, 2010 at 1:27 AM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no>  wrote:
>    
>> Hmm. It seems that GitHub itself uses Lighthouse (lighthouseapp.com),
>> and that there's some Lighthous/Github integration.
>>
>> Lighthouse appears to be free for open source projects, although it is a
>> bit of work to check how well the Github integration works.
>>      
> I think you're mistaken: github offers webhooks integration with
> lighthouse, but their own default issues implementation doesn't seem
> to be a lighthouse one. LH seems to have milestones, proper queries,
> attachments, issues-by-email, etc., all features sorely missing from
> the GH issues implementation (in its current state).
>    

If was referring to the GitHub development team using Lighthouse 
internally for their own purposes when developing the underlying 
software of Github (saw that in a blog post from 2008..).

So a) they're aware of the limitations, b) but perhaps they consider it 
out of scope?

Dag Sverre

From craigcitro at gmail.com  Thu Nov 18 22:48:40 2010
From: craigcitro at gmail.com (Craig Citro)
Date: Thu, 18 Nov 2010 13:48:40 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTikX6tC2_tmAFkiiNvOW4L3pWUpujM147k1eYw5t@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<AANLkTin2SsA2Siwfsq+Zg6V3U9q7TEDRdJ1y9iKnRAMh@mail.gmail.com>
	<AANLkTinJm1z5_+2_maTn2GTDy5jB51d=NdGz_jhb+xPq@mail.gmail.com>
	<AANLkTimsQ+ezBTps9TRRnVueoqX6aad_7FxuAVekdqD2@mail.gmail.com>
	<AANLkTikX6tC2_tmAFkiiNvOW4L3pWUpujM147k1eYw5t@mail.gmail.com>
Message-ID: <AANLkTikUfDnPZQ86XU3wFmjYHJpbVZzF01OJn+2vSXXn@mail.gmail.com>

>> argument, which is what I think I said above. I generally loathe
>> political debate, and I think cython-dev is a wildly inappropriate
>> place for it.
>

I'm really confused here: I've agreed from the beginning to eliminate
what I thought was a reasonable candidate from the running because you
had an issue with it, with *no argument whatsoever*. I explained
clearly that I don't want to get into a political debate, because I
don't think our personal politics have anything to do with what
features github has that code.google.com doesn't. Last I checked, I
was entitled to my opinions just as much as you are to yours. I really
don't get how spouting political rhetoric seemed like it was really
going to add to this conversation in any way.

This has been an excellent reminder of why I try to avoid getting
involved in mailing list threads in the first place -- as always, I
regret sending the first email at all. Thanks for helping to keep my
experiences consistent.

-cc

From vitja.makarov at gmail.com  Thu Nov 18 22:54:49 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Fri, 19 Nov 2010 00:54:49 +0300
Subject: [Cython] [bug] inf loop
In-Reply-To: <4CE58BAC.6020105@behnel.de>
References: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>
	<AANLkTi=bWs0X8peV0fQQeQx4_okvduQue_f5nJD75Nta@mail.gmail.com>
	<4CE53725.5050707@behnel.de>
	<AANLkTi=s_tFgLxGXnb5xQxyjwc8gyquTuUBOGkNgr-d5@mail.gmail.com>
	<AANLkTimAYXqVOBWU_Bdr_ihP9_s5g3iEvZf0TJgmcVDN@mail.gmail.com>
	<4CE58419.7070000@behnel.de>
	<AANLkTi=v8BVSLduQkqmmmOZ1sKp9XSsx5XeMcU8FR7MA@mail.gmail.com>
	<4CE58BAC.6020105@behnel.de>
Message-ID: <AANLkTik68aQ3SQ9Se=KmNpAU3jkQopxdw5E39L7SSMfm@mail.gmail.com>

2010/11/18 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 18.11.2010 21:17:
>> 2010/11/18 Stefan Behnel:
>>> The C compiler problems looked like closure issues, so they are more likely
>>> to be related.
>>
>> Is compile problem logged somewhere?
>
> You can find most of them here:
>
> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py26-c/704/parsed_console/
>
> Look out for "has no member named".
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

Disabling this tests seems to make regression tests finish up.

pyregr.test_socket
pyregr.test_threading

-- 
vitja.

From greg.ewing at canterbury.ac.nz  Thu Nov 18 23:54:11 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 19 Nov 2010 11:54:11 +1300
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <4CE4EF61.5070601@student.matnat.uio.no>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<AANLkTin2SsA2Siwfsq+Zg6V3U9q7TEDRdJ1y9iKnRAMh@mail.gmail.com>
	<4CE4E600.2050302@behnel.de> <4CE4EF61.5070601@student.matnat.uio.no>
Message-ID: <4CE5AE93.9020500@canterbury.ac.nz>

Dag Sverre Seljebotn wrote:

> http://librelist.com seems pretty decent from a first glance.

Hmmm, it appears the way that outfit works is that you get
subscribed to a list automatically just by sending a message
to it.

That sounds like an invitation for spam to me. When I run
a list, I prefer to set it up so that only subscribers can
post, with subscription requiring confirmation.

They claim to do spam filtering, but I'm not sure I'd like
to rely solely on that.

-- 
Greg


From robertwb at math.washington.edu  Fri Nov 19 01:08:31 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 18 Nov 2010 16:08:31 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTi=KhpaC4rm-M-UkzrJUjOTMX5XRaH_uvhPzEV=T@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<AANLkTin2SsA2Siwfsq+Zg6V3U9q7TEDRdJ1y9iKnRAMh@mail.gmail.com>
	<AANLkTi=KhpaC4rm-M-UkzrJUjOTMX5XRaH_uvhPzEV=T@mail.gmail.com>
Message-ID: <AANLkTinwZwf6NFGX+GBXMnvH8rK8Nu-AFua=kxHVZXrj@mail.gmail.com>

On Thu, Nov 18, 2010 at 1:00 AM, Fernando Perez <fperez.net at gmail.com> wrote:
> On Thu, Nov 18, 2010 at 12:13 AM, Craig Citro <craigcitro at gmail.com> wrote:
>> but just
>> for the sake of completeness, are there any features code.google.com
>> has that github doesn't? (Or vice-versa?) Are they things we'd ever be
>> likely to notice?
>
> I say this as a flat out github fan, who's thrilled we moved for
> IPython and everything else I'm involved with. ?But... the github bug
> tracker is *vastly* inferior to google's. ?I keep hoping that, with
> how great github is getting very quickly, they'll soon roll out
> something to improve the bug tracker, because it's frankly painfully
> primitive.
>
> Whoever thought out the google code bug tracker was a smart cookie.
> It relies on what appears to be a plain labels system, but with very
> simple conventions on how to declare the labels, it basically encodes
> a full key-value system on which queries can later be made, reports,
> etc. ?It's a really elegant piece of work:
>
> http://code.google.com/p/support/wiki/IssueTracker
>
> The github tracker is dumb as a brick in comparison, and their search
> doesn't even work right. ?It's frankly embarrassing, and it sticks out
> as a sore thumb compared to how great pretty much everything else
> there is.
>
> It's only the fact that the contrast is so stark, that makes me hope
> it won't last :)

Thanks for your detailed analysis about the pros and cons of github.
I've played around with the issue tracker there and a google code and
come to the conclusion that

    trac > google code > github

for this piece of the puzzle, but feel like I've only scratched the
surface of the google code site, while reaching the limits of the
github one. The thing I like about trac is the easy visualization of a
milestone (the histogram-by-component is nice to look at, as is the
grouping in the reports--does anyone know how to do that in the above
two systems). However, the easy clone-review-push model of getting
code in is far more important to me.

- Robert

From robertwb at math.washington.edu  Fri Nov 19 01:33:32 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 18 Nov 2010 16:33:32 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <4CE4E600.2050302@behnel.de>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<AANLkTin2SsA2Siwfsq+Zg6V3U9q7TEDRdJ1y9iKnRAMh@mail.gmail.com>
	<4CE4E600.2050302@behnel.de>
Message-ID: <AANLkTi=gsW02Zq6tRanZMxTYpe08czBfgwj4Q-Twt13R@mail.gmail.com>

On Thu, Nov 18, 2010 at 12:38 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Craig Citro, 18.11.2010 09:13:
>> ? * Does github have any sort of mailing list support? I'd love to see
>> cython-dev migrated
>
> -1
>
>
>> to something with better searchability, linking, archives, etc
>
> What's wrong with all the archives of the current mailing list? Doesn't at
> least one of them suite your way of using a ML archive? Gmane alone has
> three different web interfaces.
>
> And in what way is "linking" different from "posting a link" or "referring
> to an archive post"? I never had any problem with either of those.

There's no canonical link for a message or a thread, most of the
archives chop threads at month boundaries, and I, for one, don't like
the treaded views they offer near as much.

>> -- I don't know that there's any special link between
>> google groups and code.google.com, so this is really orthogonal to the
>> project hosting question. I know the decision to use codespeak was
>> made as an explicit decision against google groups, but my
>> understanding is that whatever objections there were are now gone. Is
>> that still the case?
>
> ?From comparing the ML related comments on cython-dev and cython-users, I
> got the impression that it wasn't that a good idea to have a mailing list
> on google-groups. I remember reading things like "I thought top-posting was
> the right thing to do on google-groups" more than once. An environment that
> doesn't help people to do the right thing doesn't feel welcoming to me. And
> I really hate the way google-groups presents archived threads. Makes them
> totally hard to read.

I find google groups easier to administer and use, but I consider most
of these to be matters of taste and opinion (I prefer bottom posting
as well, though not to the same extreme). So for now we have a
compromise between our -users and -dev lists, and though we'd both
like to move them to the same place (in opposite directions) it seems
to be working fine. In particular, I don't think the mailing lists
need to be tied to the other infrastructure (whereas having issue
trackers and code review in the same place is really handy...)

- Robert

From robertwb at math.washington.edu  Fri Nov 19 01:35:45 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 18 Nov 2010 16:35:45 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <4CE3E607.2040801@behnel.de>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<4CE3AF74.40703@student.matnat.uio.no>
	<4CE3B0E9.7090304@student.matnat.uio.no>
	<4CE3E607.2040801@behnel.de>
Message-ID: <AANLkTin+XOU-dizQPQTNKKn-fus96=Yt53RkmcJ_8=d9@mail.gmail.com>

On Wed, Nov 17, 2010 at 6:26 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Dag Sverre Seljebotn, 17.11.2010 11:39:
>> On 11/17/2010 11:33 AM, Dag Sverre Seljebotn wrote:
>>> On 11/17/2010 09:59 AM, Robert Bradshaw wrote:
>>>> So, I've been thinking about this, and one of the reasons we're using
>>>> the current setup is that it requires very little administration from
>>>> me, as I am essentially leaching of the sagemath.org infrastructure.
>>>> Moving things to boxen would be a bit more administration on our part,
>>>> but still not too bad, and I know William is a happy enough Cython
>>>> user to be fine with continuing to host us :). However, I'm wondering
>>>> if this would be a ripe occasion to making the leap to something like
>>>> http://code.google.com. Currently, our infrastructure consists of
>>>>
>>>> 1. The web site
>>>> 2. Trac
>>>> 3. Wiki
>>>> 4. Repositories
>>>> 5. Buildbot
>>>> 6. Mailing lists
>>>>
>>>> Currently we're hosting 1-5, and 6 is being hosted by codespeak.net
>>>> (for cython-dev) and google (for cython-users). I think it may be
>>>> worth considering moving 2-4 elsewhere, as there is little loss and
>>>> they are the higher-maintenance (from an administrative point of view)
>>>> items, and features such as code review tools would be nice to have as
>>>> well.
>
> I see the serious advantage of simplifying the access to bug tracker, wiki,
> etc. for new users.
>
> However, regarding the "little loss", does anyone have experience in
> migrating trac tickets to a different tracker? There's a lot of information
> in the tracker that I don't want to loose, and I wouldn't like to look at
> two trackers to figure out if a bug has been reported.
>
>
>> Forgetting about the git aspect for a moment, I'm strongly in favor of a
>> DVCS-centered site. That is, +1 to bitbucket, github, gitorious, ++, and
>> -1 to SourceForge, Google Code, etc.
>
> I guess it's bitbucket then? Switching the VCS just because of a project
> hosting site sounds like more trouble than I'm currently after.

Would you be opposed to switching? In other words, is that a
deal-breaker, or simply not worth doing assuming there were no great
advantage?

- Robert

From ondrej at certik.cz  Fri Nov 19 02:57:29 2010
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 18 Nov 2010 17:57:29 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTi=L081GsSjnbCFAOQA2r6=z+T7WT_nqyJaVDaHt@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<AANLkTi=L081GsSjnbCFAOQA2r6=z+T7WT_nqyJaVDaHt@mail.gmail.com>
Message-ID: <AANLkTi=DqgZ03wB9XiJ++_N1zNB3=3LfpYUVxYWE3fHD@mail.gmail.com>

On Wed, Nov 17, 2010 at 1:19 AM, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
>> However, I'm wondering
>> if this would be a ripe occasion to making the leap to something like
>> http://code.google.com.
>
> Oh - dear - I'm afraid I have said this on several mailing lists, but
> Google code is the most aggressive in blocking access from 'forbidden
> countries' such as Cuba, where I often work [1]. ? Only Google code
> and Sourceforge do this; and Sourceforge allows you to opt out. ? I
> think this is against the spirit of open source software, and it has
> made it harder for me to advocate the use of open-source in general
> and python in particular. ? I hope very much you consider using a more
> open provider.

Are you happy with sympy switching to github? Now all these sites are
hosted at github:

http://sympy.org/
http://docs.sympy.org/
https://github.com/sympy/sympy

Does this work from Cuba? Indeed, I got couple requests from Cuba,
that they can't access sympy, and it truly sucks.

Ondrej

From ondrej at certik.cz  Fri Nov 19 03:05:44 2010
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 18 Nov 2010 18:05:44 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTi=+AktpDE=yBrXFQWKbC+NKLc4AQfwYTuFhSzxL@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<4CE3AF74.40703@student.matnat.uio.no>
	<AANLkTi=+AktpDE=yBrXFQWKbC+NKLc4AQfwYTuFhSzxL@mail.gmail.com>
Message-ID: <AANLkTimATefKRkvj6PKzgLfO6YyCRr6gh56O7H9JkuOp@mail.gmail.com>

On Thu, Nov 18, 2010 at 12:50 AM, Fernando Perez <fperez.net at gmail.com> wrote:
> On Wed, Nov 17, 2010 at 2:33 AM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>>
>> ?- Static web site content (you just create a repository which it
>> serves as static content, e.g, http://cournape.github.com/Bento/)
>
> If you go down the github route, I'd suggest a slightly different
> alternative than github's default for the creation of static content.
> Their approach uses a hidden DAG in the repo called gh-pages, which
> eternally pollutes your repo with all static html you generate. ?It's
> easy with a tiny bit of scripting to create instead the gh-pages
> content in a *separate repo* completely decoupled from the main one.
> That's what we do here:
>
> https://github.com/fperez/datarray ?# main repo
> http://fperez.github.com/datarray-doc/ ?# gh-pages in separate repo.
>
> The tool that does the job is this trivial script:
>
> https://github.com/fperez/datarray/blob/master/doc/gh-pages.py
>
> I'll be happy to provide pointers if you want them.
>
> I just wanted to let you know so that you avoid what I think is a
> poor, but easy to work around, design choice of github's pages
> implementation.

Two improvements over your setup:

* you can have your own domain point to the pages (e.g. docs.sympy.org
are hosted at github servers)

* you can set the "gh-pages" as the default (and only) branch in the
datarray-doc, just like this: https://github.com/sympy/sympy_doc, and
then this little issue becomes a non-issue, because you just pull and
push into the repo and don't worry about anything (there is no master
branch, just the gh-pages branch, and as a result, you just do "git
pull, git push origin")


Ondrej

From ondrej at certik.cz  Fri Nov 19 03:49:09 2010
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 18 Nov 2010 18:49:09 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
Message-ID: <AANLkTimRqTQs6E47_Uy6SVXuzLBi6ek4MMgmaAdH5Sfs@mail.gmail.com>

On Wed, Nov 17, 2010 at 12:59 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Tue, Nov 16, 2010 at 11:14 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> Stefan Behnel, 16.11.2010 12:46:
>>> when I try to hg push to cython-devel, I keep getting this:
>>>
>>> """
>>> Traceback (most recent call last):
>>> SyntaxError: Error expanding 'sessionvars%urlparameter'
>>> """
>>>
>>> Trac seems to be broken, too. Has anything changed on the Cython VM since
>>> yesterday that could trigger this?
>>
>> hg is back to normal since William cleaned up the disk space. trac is still
>> down, it seems.
>>
>> Robert, since you set up most of the infrastructure (IIRC), could you take
>> an attempt at the migration to boxen?
>
> So, I've been thinking about this, and one of the reasons we're using
> the current setup is that it requires very little administration from
> me, as I am essentially leaching of the sagemath.org infrastructure.
> Moving things to boxen would be a bit more administration on our part,
> but still not too bad, and I know William is a happy enough Cython
> user to be fine with continuing to host us :). However, I'm wondering
> if this would be a ripe occasion to making the leap to something like
> http://code.google.com. Currently, our infrastructure consists of
>
> 1. The web site
> 2. Trac
> 3. Wiki
> 4. Repositories
> 5. Buildbot
> 6. Mailing lists
>
> Currently we're hosting 1-5, and 6 is being hosted by codespeak.net
> (for cython-dev) and google (for cython-users). I think it may be
> worth considering moving 2-4 elsewhere, as there is little loss and
> they are the higher-maintenance (from an administrative point of view)
> items, and features such as code review tools would be nice to have as
> well. Trying to use launchpad was painful, so this decision shouldn't
> be taken lightly, but I think we can do better. Of course
> code.google.com isn't the only option, but even trying to be unbiased
> about it I think it's a very good option, and
> http://www.dataliberation.org/google/code-project-hosting factors into
> it as well.
>
> In terms of the website itself, I think we should keep hosting that
> (for maximum flexibility, and it's easy to administer), and there
> isn't much of an option for 5 (though steps should be taken to reduce
> its load).

Robert has asked me to reply what I think about google code and
github, since sympy has used google code since the beginning, e.g.
last couple years, so I wrote this blog post about comparing Google
Code and GitHub:

http://ondrejcertik.blogspot.com/2010/11/google-code-vs-github-for-hosting.html

Ondrej

From robertwb at math.washington.edu  Fri Nov 19 04:00:56 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 18 Nov 2010 19:00:56 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTikX6tC2_tmAFkiiNvOW4L3pWUpujM147k1eYw5t@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<AANLkTin2SsA2Siwfsq+Zg6V3U9q7TEDRdJ1y9iKnRAMh@mail.gmail.com>
	<AANLkTinJm1z5_+2_maTn2GTDy5jB51d=NdGz_jhb+xPq@mail.gmail.com>
	<AANLkTimsQ+ezBTps9TRRnVueoqX6aad_7FxuAVekdqD2@mail.gmail.com>
	<AANLkTikX6tC2_tmAFkiiNvOW4L3pWUpujM147k1eYw5t@mail.gmail.com>
Message-ID: <AANLkTimTgxF2oxOsfkxjdc4fKGHNK3MbbZ0t1dZSDY-4@mail.gmail.com>

On Thu, Nov 18, 2010 at 1:03 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
>> Yep, you're right, they're political restrictions, and "geographic
>> restrictions" was a euphemism. I'm a huge believer in open source, but
>> personally, these political restrictions don't bother me at all.
>> However, you and Stefan had already both chimed in saying they did
>> bother you, so I'm very happy to pick another hosting option with no
>> argument, which is what I think I said above. I generally loathe
>> political debate, and I think cython-dev is a wildly inappropriate
>> place for it.
>
> If we see a powerful country trying to exclude less powerful countries
> from what we believe to be valid discourse, then, to say and do
> nothing is an act of passive agreement that not all of us can accept.
>
> We state our principles, and we try to follow them; when we agree, we
> do not have to discuss them. ?The principles under which we operate
> are one of the great strengths of the open-source idea and the energy
> that we give to our projects.

I'm not sure which "we" you are referring to here, but you made your
point quite clearly and it was acknowledged. "Political restrictions"
and other non-technical issues are perfectly valid objections to bring
up, but belaboring the point, especially as someone who has never (as
far as I can tell) participated in the development process and hence
used the tools in question, doesn't help your case.

To be clear, every potential solution involves giving up some freedom.
Each site has terms of service agreements that are pages long, and
even hosting it ourselves isn't completely unencumbered (software
licenses agreements (yes, including the GPL), contracts with the ISP,
possible legal restrictions, etc.) You hopefully already know
this--you've chosen to use a gmail account to post this. Personally,
my top (but certainly not only) criteria are

    1. Lack of Lock-in
    2. Productivity

Everyone has different criteria and priorities, which is why I started
this conversation instead of just moving everything over and making an
announcement :).

-Robert

From robertwb at math.washington.edu  Fri Nov 19 04:10:57 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Thu, 18 Nov 2010 19:10:57 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTimRqTQs6E47_Uy6SVXuzLBi6ek4MMgmaAdH5Sfs@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<AANLkTimRqTQs6E47_Uy6SVXuzLBi6ek4MMgmaAdH5Sfs@mail.gmail.com>
Message-ID: <AANLkTi=PQtq0kP+a-x6Qxz1FFTvLtOyPY2=kKWm_3xUf@mail.gmail.com>

On Thu, Nov 18, 2010 at 6:49 PM, Ondrej Certik <ondrej at certik.cz> wrote:
> On Wed, Nov 17, 2010 at 12:59 AM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> On Tue, Nov 16, 2010 at 11:14 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>> Stefan Behnel, 16.11.2010 12:46:
>>>> when I try to hg push to cython-devel, I keep getting this:
>>>>
>>>> """
>>>> Traceback (most recent call last):
>>>> SyntaxError: Error expanding 'sessionvars%urlparameter'
>>>> """
>>>>
>>>> Trac seems to be broken, too. Has anything changed on the Cython VM since
>>>> yesterday that could trigger this?
>>>
>>> hg is back to normal since William cleaned up the disk space. trac is still
>>> down, it seems.
>>>
>>> Robert, since you set up most of the infrastructure (IIRC), could you take
>>> an attempt at the migration to boxen?
>>
>> So, I've been thinking about this, and one of the reasons we're using
>> the current setup is that it requires very little administration from
>> me, as I am essentially leaching of the sagemath.org infrastructure.
>> Moving things to boxen would be a bit more administration on our part,
>> but still not too bad, and I know William is a happy enough Cython
>> user to be fine with continuing to host us :). However, I'm wondering
>> if this would be a ripe occasion to making the leap to something like
>> http://code.google.com. Currently, our infrastructure consists of
>>
>> 1. The web site
>> 2. Trac
>> 3. Wiki
>> 4. Repositories
>> 5. Buildbot
>> 6. Mailing lists
>>
>> Currently we're hosting 1-5, and 6 is being hosted by codespeak.net
>> (for cython-dev) and google (for cython-users). I think it may be
>> worth considering moving 2-4 elsewhere, as there is little loss and
>> they are the higher-maintenance (from an administrative point of view)
>> items, and features such as code review tools would be nice to have as
>> well. Trying to use launchpad was painful, so this decision shouldn't
>> be taken lightly, but I think we can do better. Of course
>> code.google.com isn't the only option, but even trying to be unbiased
>> about it I think it's a very good option, and
>> http://www.dataliberation.org/google/code-project-hosting factors into
>> it as well.
>>
>> In terms of the website itself, I think we should keep hosting that
>> (for maximum flexibility, and it's easy to administer), and there
>> isn't much of an option for 5 (though steps should be taken to reduce
>> its load).
>
> Robert has asked me to reply what I think about google code and
> github, since sympy has used google code since the beginning, e.g.
> last couple years, so I wrote this blog post about comparing Google
> Code and GitHub:
>
> http://ondrejcertik.blogspot.com/2010/11/google-code-vs-github-for-hosting.html

Thanks! To clarify, you never used google's distributed module of push
and pull requests? (hg, not git) Granted, I think github (and perhaps
because of git) gets this done smoother. What are the chances they'll
fix their issue tracker? :(

- Robert

From ondrej at certik.cz  Fri Nov 19 04:25:20 2010
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 18 Nov 2010 19:25:20 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTi=PQtq0kP+a-x6Qxz1FFTvLtOyPY2=kKWm_3xUf@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<AANLkTimRqTQs6E47_Uy6SVXuzLBi6ek4MMgmaAdH5Sfs@mail.gmail.com>
	<AANLkTi=PQtq0kP+a-x6Qxz1FFTvLtOyPY2=kKWm_3xUf@mail.gmail.com>
Message-ID: <AANLkTinSJMv_Mr+Wfegr3yNuGJfFO6dLm09-itNA2Y08@mail.gmail.com>

On Thu, Nov 18, 2010 at 7:10 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Thu, Nov 18, 2010 at 6:49 PM, Ondrej Certik <ondrej at certik.cz> wrote:
>> On Wed, Nov 17, 2010 at 12:59 AM, Robert Bradshaw
>> <robertwb at math.washington.edu> wrote:
>>> On Tue, Nov 16, 2010 at 11:14 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>>>> Stefan Behnel, 16.11.2010 12:46:
>>>>> when I try to hg push to cython-devel, I keep getting this:
>>>>>
>>>>> """
>>>>> Traceback (most recent call last):
>>>>> SyntaxError: Error expanding 'sessionvars%urlparameter'
>>>>> """
>>>>>
>>>>> Trac seems to be broken, too. Has anything changed on the Cython VM since
>>>>> yesterday that could trigger this?
>>>>
>>>> hg is back to normal since William cleaned up the disk space. trac is still
>>>> down, it seems.
>>>>
>>>> Robert, since you set up most of the infrastructure (IIRC), could you take
>>>> an attempt at the migration to boxen?
>>>
>>> So, I've been thinking about this, and one of the reasons we're using
>>> the current setup is that it requires very little administration from
>>> me, as I am essentially leaching of the sagemath.org infrastructure.
>>> Moving things to boxen would be a bit more administration on our part,
>>> but still not too bad, and I know William is a happy enough Cython
>>> user to be fine with continuing to host us :). However, I'm wondering
>>> if this would be a ripe occasion to making the leap to something like
>>> http://code.google.com. Currently, our infrastructure consists of
>>>
>>> 1. The web site
>>> 2. Trac
>>> 3. Wiki
>>> 4. Repositories
>>> 5. Buildbot
>>> 6. Mailing lists
>>>
>>> Currently we're hosting 1-5, and 6 is being hosted by codespeak.net
>>> (for cython-dev) and google (for cython-users). I think it may be
>>> worth considering moving 2-4 elsewhere, as there is little loss and
>>> they are the higher-maintenance (from an administrative point of view)
>>> items, and features such as code review tools would be nice to have as
>>> well. Trying to use launchpad was painful, so this decision shouldn't
>>> be taken lightly, but I think we can do better. Of course
>>> code.google.com isn't the only option, but even trying to be unbiased
>>> about it I think it's a very good option, and
>>> http://www.dataliberation.org/google/code-project-hosting factors into
>>> it as well.
>>>
>>> In terms of the website itself, I think we should keep hosting that
>>> (for maximum flexibility, and it's easy to administer), and there
>>> isn't much of an option for 5 (though steps should be taken to reduce
>>> its load).
>>
>> Robert has asked me to reply what I think about google code and
>> github, since sympy has used google code since the beginning, e.g.
>> last couple years, so I wrote this blog post about comparing Google
>> Code and GitHub:
>>
>> http://ondrejcertik.blogspot.com/2010/11/google-code-vs-github-for-hosting.html
>
> Thanks! To clarify, you never used google's distributed module of push
> and pull requests? (hg, not git) Granted, I think github (and perhaps

I never used it, because we switched to git (from mercurial) before
they implemented it (for mercurial).

> because of git) gets this done smoother. What are the chances they'll
> fix their issue tracker? :(

I think quite high, the main question is when, and that I don't know.
Just to clarify, it's not *that* bad, for example we use it quite
extensively here:

https://github.com/hpfem/hermes/issues
https://github.com/hpfem/femhub/issues

so it's usable, but obviously, for someone coming from Google Code,
it's a step back.

Ondrej

From fperez.net at gmail.com  Fri Nov 19 04:51:32 2010
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 18 Nov 2010 19:51:32 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTimATefKRkvj6PKzgLfO6YyCRr6gh56O7H9JkuOp@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<4CE3AF74.40703@student.matnat.uio.no>
	<AANLkTi=+AktpDE=yBrXFQWKbC+NKLc4AQfwYTuFhSzxL@mail.gmail.com>
	<AANLkTimATefKRkvj6PKzgLfO6YyCRr6gh56O7H9JkuOp@mail.gmail.com>
Message-ID: <AANLkTikz_=jzA-10+hQAJZ=QaKi+=a238HNY3Kc_-Ujn@mail.gmail.com>

On Thu, Nov 18, 2010 at 6:05 PM, Ondrej Certik <ondrej at certik.cz> wrote:
>
> * you can have your own domain point to the pages (e.g. docs.sympy.org
> are hosted at github servers)

Yup, I didn't bother but it's easy to do.

> * you can set the "gh-pages" as the default (and only) branch in the
> datarray-doc, just like this: https://github.com/sympy/sympy_doc, and
> then this little issue becomes a non-issue, because you just pull and
> push into the repo and don't worry about anything (there is no master
> branch, just the gh-pages branch, and as a result, you just do "git
> pull, git push origin")

Yes, that's precisely what the script I pointed to does.  It automates
the process of building the docs in that separate repo with gh-pages
as its only branch.

In practice it works very well, and there's no pollution of the
project repo with the gh-pages builds.

Cheers,

f

From ondrej at certik.cz  Fri Nov 19 05:11:53 2010
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 18 Nov 2010 20:11:53 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTikz_=jzA-10+hQAJZ=QaKi+=a238HNY3Kc_-Ujn@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<4CE3AF74.40703@student.matnat.uio.no>
	<AANLkTi=+AktpDE=yBrXFQWKbC+NKLc4AQfwYTuFhSzxL@mail.gmail.com>
	<AANLkTimATefKRkvj6PKzgLfO6YyCRr6gh56O7H9JkuOp@mail.gmail.com>
	<AANLkTikz_=jzA-10+hQAJZ=QaKi+=a238HNY3Kc_-Ujn@mail.gmail.com>
Message-ID: <AANLkTimByBUBhwuJSz7Ok97u+UL3vF-Pt14t8PxzML45@mail.gmail.com>

On Thu, Nov 18, 2010 at 7:51 PM, Fernando Perez <fperez.net at gmail.com> wrote:
> On Thu, Nov 18, 2010 at 6:05 PM, Ondrej Certik <ondrej at certik.cz> wrote:
>>
>> * you can have your own domain point to the pages (e.g. docs.sympy.org
>> are hosted at github servers)
>
> Yup, I didn't bother but it's easy to do.
>
>> * you can set the "gh-pages" as the default (and only) branch in the
>> datarray-doc, just like this: https://github.com/sympy/sympy_doc, and
>> then this little issue becomes a non-issue, because you just pull and
>> push into the repo and don't worry about anything (there is no master
>> branch, just the gh-pages branch, and as a result, you just do "git
>> pull, git push origin")
>
> Yes, that's precisely what the script I pointed to does. ?It automates
> the process of building the docs in that separate repo with gh-pages
> as its only branch.

Well, but if I go here:

https://github.com/fperez/datarray-doc/branches

it says the main repo is "master", and "gh-pages" is another branch.
If I go here:

https://github.com/sympy/sympy_doc/branches

there is only one main branch, gh-pages. That's what I was trying to point out.

>
> In practice it works very well, and there's no pollution of the
> project repo with the gh-pages builds.

Yep, it's not an issue.

Ondrej

From stefan_ml at behnel.de  Fri Nov 19 05:34:35 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 19 Nov 2010 05:34:35 +0100
Subject: [Cython] [bug] inf loop
In-Reply-To: <AANLkTik68aQ3SQ9Se=KmNpAU3jkQopxdw5E39L7SSMfm@mail.gmail.com>
References: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>	<AANLkTi=bWs0X8peV0fQQeQx4_okvduQue_f5nJD75Nta@mail.gmail.com>	<4CE53725.5050707@behnel.de>	<AANLkTi=s_tFgLxGXnb5xQxyjwc8gyquTuUBOGkNgr-d5@mail.gmail.com>	<AANLkTimAYXqVOBWU_Bdr_ihP9_s5g3iEvZf0TJgmcVDN@mail.gmail.com>	<4CE58419.7070000@behnel.de>	<AANLkTi=v8BVSLduQkqmmmOZ1sKp9XSsx5XeMcU8FR7MA@mail.gmail.com>	<4CE58BAC.6020105@behnel.de>
	<AANLkTik68aQ3SQ9Se=KmNpAU3jkQopxdw5E39L7SSMfm@mail.gmail.com>
Message-ID: <4CE5FE5B.30108@behnel.de>

Vitja Makarov, 18.11.2010 22:54:
> 2010/11/18 Stefan Behnel:
>> Vitja Makarov, 18.11.2010 21:17:
>>> 2010/11/18 Stefan Behnel:
>>>> The C compiler problems looked like closure issues, so they are more likely
>>>> to be related.
>>>
>>> Is compile problem logged somewhere?
>>
>> You can find most of them here:
>>
>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py26-c/704/parsed_console/
>>
>> Look out for "has no member named".
>
> Disabling this tests seems to make regression tests finish up.
>
> pyregr.test_socket
> pyregr.test_threading

Ok, I disabled them.

Stefan

From stefan_ml at behnel.de  Fri Nov 19 05:48:53 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 19 Nov 2010 05:48:53 +0100
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTin+XOU-dizQPQTNKKn-fus96=Yt53RkmcJ_8=d9@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de>
	<4CE380E0.7070403@behnel.de>	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>	<4CE3AF74.40703@student.matnat.uio.no>	<4CE3B0E9.7090304@student.matnat.uio.no>	<4CE3E607.2040801@behnel.de>
	<AANLkTin+XOU-dizQPQTNKKn-fus96=Yt53RkmcJ_8=d9@mail.gmail.com>
Message-ID: <4CE601B5.4090304@behnel.de>

Robert Bradshaw, 19.11.2010 01:35:
> On Wed, Nov 17, 2010 at 6:26 AM, Stefan Behnel wrote:
>> Dag Sverre Seljebotn, 17.11.2010 11:39:
>>> Forgetting about the git aspect for a moment, I'm strongly in favor of a
>>> DVCS-centered site. That is, +1 to bitbucket, github, gitorious, ++, and
>>> -1 to SourceForge, Google Code, etc.
>>
>> I guess it's bitbucket then? Switching the VCS just because of a project
>> hosting site sounds like more trouble than I'm currently after.
>
> Would you be opposed to switching? In other words, is that a
> deal-breaker, or simply not worth doing assuming there were no great
> advantage?

Let's say, I'm -0 on switching to git. I never really used it except for a 
couple of times where I had to and that already gave me a scratching itch 
in my neck that told me I was touching something crude. That feeling may be 
wrong, but maybe not. ;-)

However, I'm already happily using hgsvn to talk to SVN, so as long as the 
history can be preserved 100% (which is likely the case), and as long as I 
(and others) have the choice to do the switch or not, I'll be fine.

Stefan

From stefan_ml at behnel.de  Fri Nov 19 05:55:23 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 19 Nov 2010 05:55:23 +0100
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTi=gsW02Zq6tRanZMxTYpe08czBfgwj4Q-Twt13R@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de>
	<4CE380E0.7070403@behnel.de>	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>	<AANLkTin2SsA2Siwfsq+Zg6V3U9q7TEDRdJ1y9iKnRAMh@mail.gmail.com>	<4CE4E600.2050302@behnel.de>
	<AANLkTi=gsW02Zq6tRanZMxTYpe08czBfgwj4Q-Twt13R@mail.gmail.com>
Message-ID: <4CE6033B.6040707@behnel.de>

Robert Bradshaw, 19.11.2010 01:33:
> So for now we have a
> compromise between our -users and -dev lists, and though we'd both
> like to move them to the same place (in opposite directions) it seems
> to be working fine. In particular, I don't think the mailing lists
> need to be tied to the other infrastructure (whereas having issue
> trackers and code review in the same place is really handy...)

+1, although I don't see it as a "compromise" (in the sense that I'd see a 
reason to have them in the same place).

Stefan

From matthew.brett at gmail.com  Fri Nov 19 07:13:28 2010
From: matthew.brett at gmail.com (Matthew Brett)
Date: Thu, 18 Nov 2010 22:13:28 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTi=DqgZ03wB9XiJ++_N1zNB3=3LfpYUVxYWE3fHD@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<AANLkTi=L081GsSjnbCFAOQA2r6=z+T7WT_nqyJaVDaHt@mail.gmail.com>
	<AANLkTi=DqgZ03wB9XiJ++_N1zNB3=3LfpYUVxYWE3fHD@mail.gmail.com>
Message-ID: <AANLkTi=JVPLZMTY6U1K45AqG=4+_rNRQZNqcjSv6UNJc@mail.gmail.com>

Hi,

On Thu, Nov 18, 2010 at 5:57 PM, Ondrej Certik <ondrej at certik.cz> wrote:
> On Wed, Nov 17, 2010 at 1:19 AM, Matthew Brett <matthew.brett at gmail.com> wrote:
>> Hi,
>>
>>> However, I'm wondering
>>> if this would be a ripe occasion to making the leap to something like
>>> http://code.google.com.
>>
>> Oh - dear - I'm afraid I have said this on several mailing lists, but
>> Google code is the most aggressive in blocking access from 'forbidden
>> countries' such as Cuba, where I often work [1]. ? Only Google code
>> and Sourceforge do this; and Sourceforge allows you to opt out. ? I
>> think this is against the spirit of open source software, and it has
>> made it harder for me to advocate the use of open-source in general
>> and python in particular. ? I hope very much you consider using a more
>> open provider.
>
> Are you happy with sympy switching to github? Now all these sites are
> hosted at github:
>
> http://sympy.org/
> http://docs.sympy.org/
> https://github.com/sympy/sympy
>
> Does this work from Cuba? Indeed, I got couple requests from Cuba,
> that they can't access sympy, and it truly sucks.

Yes, I've used github extensively from Cuba, including sympy  - I
haven't come across any restrictions.  Is it possible they were trying
to get to something on the Google code site?  You can't even get to
the web pages there, so it can be difficult to work out what you can't
see, if you see what I mean.

See you,

Matthew

From matthew.brett at gmail.com  Fri Nov 19 07:15:34 2010
From: matthew.brett at gmail.com (Matthew Brett)
Date: Thu, 18 Nov 2010 22:15:34 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTimTgxF2oxOsfkxjdc4fKGHNK3MbbZ0t1dZSDY-4@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<AANLkTin2SsA2Siwfsq+Zg6V3U9q7TEDRdJ1y9iKnRAMh@mail.gmail.com>
	<AANLkTinJm1z5_+2_maTn2GTDy5jB51d=NdGz_jhb+xPq@mail.gmail.com>
	<AANLkTimsQ+ezBTps9TRRnVueoqX6aad_7FxuAVekdqD2@mail.gmail.com>
	<AANLkTikX6tC2_tmAFkiiNvOW4L3pWUpujM147k1eYw5t@mail.gmail.com>
	<AANLkTimTgxF2oxOsfkxjdc4fKGHNK3MbbZ0t1dZSDY-4@mail.gmail.com>
Message-ID: <AANLkTikgNWobmA6fd9+g05Dck2JAQGgY=iBAv5z8vEQ4@mail.gmail.com>

> I'm not sure which "we" you are referring to here, but you made your
> point quite clearly and it was acknowledged. "Political restrictions"
> and other non-technical issues are perfectly valid objections to bring
> up, but belaboring the point, especially as someone who has never (as
> far as I can tell) participated in the development process and hence
> used the tools in question, doesn't help your case.

Ouch.

> To be clear, every potential solution involves giving up some freedom.
> Each site has terms of service agreements that are pages long, and
> even hosting it ourselves isn't completely unencumbered (software
> licenses agreements (yes, including the GPL), contracts with the ISP,
> possible legal restrictions, etc.) You hopefully already know
> this--you've chosen to use a gmail account to post this. Personally,
> my top (but certainly not only) criteria are
>
> ? ?1. Lack of Lock-in
> ? ?2. Productivity
>
> Everyone has different criteria and priorities, which is why I started
> this conversation instead of just moving everything over and making an
> announcement :).

Yes, I'm sorry if I was implying you weren't being open to discussion;
I'm very glad of the atmosphere y'all have created here,

Best,

Matthew

From ondrej at certik.cz  Fri Nov 19 07:19:05 2010
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 18 Nov 2010 22:19:05 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTi=JVPLZMTY6U1K45AqG=4+_rNRQZNqcjSv6UNJc@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<AANLkTi=L081GsSjnbCFAOQA2r6=z+T7WT_nqyJaVDaHt@mail.gmail.com>
	<AANLkTi=DqgZ03wB9XiJ++_N1zNB3=3LfpYUVxYWE3fHD@mail.gmail.com>
	<AANLkTi=JVPLZMTY6U1K45AqG=4+_rNRQZNqcjSv6UNJc@mail.gmail.com>
Message-ID: <AANLkTimXVe=bbbjHepOm6ySgh--vg2PpPdyfG1LdLJAD@mail.gmail.com>

On Thu, Nov 18, 2010 at 10:13 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
> Hi,
>
> On Thu, Nov 18, 2010 at 5:57 PM, Ondrej Certik <ondrej at certik.cz> wrote:
>> On Wed, Nov 17, 2010 at 1:19 AM, Matthew Brett <matthew.brett at gmail.com> wrote:
>>> Hi,
>>>
>>>> However, I'm wondering
>>>> if this would be a ripe occasion to making the leap to something like
>>>> http://code.google.com.
>>>
>>> Oh - dear - I'm afraid I have said this on several mailing lists, but
>>> Google code is the most aggressive in blocking access from 'forbidden
>>> countries' such as Cuba, where I often work [1]. ? Only Google code
>>> and Sourceforge do this; and Sourceforge allows you to opt out. ? I
>>> think this is against the spirit of open source software, and it has
>>> made it harder for me to advocate the use of open-source in general
>>> and python in particular. ? I hope very much you consider using a more
>>> open provider.
>>
>> Are you happy with sympy switching to github? Now all these sites are
>> hosted at github:
>>
>> http://sympy.org/
>> http://docs.sympy.org/
>> https://github.com/sympy/sympy
>>
>> Does this work from Cuba? Indeed, I got couple requests from Cuba,
>> that they can't access sympy, and it truly sucks.
>
> Yes, I've used github extensively from Cuba, including sympy ?- I
> haven't come across any restrictions. ?Is it possible they were trying
> to get to something on the Google code site? ?You can't even get to
> the web pages there, so it can be difficult to work out what you can't
> see, if you see what I mean.

They wrote me a private email, asking how to download sympy sources,
and at that time, I just pointed a debian page with the latest
release, that worked in Cuba. Now we use github for the latest git, so
it's a non issue.

Ondrej

From dagss at student.matnat.uio.no  Fri Nov 19 07:50:02 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 19 Nov 2010 07:50:02 +0100
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTikX6tC2_tmAFkiiNvOW4L3pWUpujM147k1eYw5t@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de>
	<4CE380E0.7070403@behnel.de>	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>	<AANLkTin2SsA2Siwfsq+Zg6V3U9q7TEDRdJ1y9iKnRAMh@mail.gmail.com>	<AANLkTinJm1z5_+2_maTn2GTDy5jB51d=NdGz_jhb+xPq@mail.gmail.com>	<AANLkTimsQ+ezBTps9TRRnVueoqX6aad_7FxuAVekdqD2@mail.gmail.com>
	<AANLkTikX6tC2_tmAFkiiNvOW4L3pWUpujM147k1eYw5t@mail.gmail.com>
Message-ID: <4CE61E1A.4020501@student.matnat.uio.no>

On 11/18/2010 10:03 PM, Matthew Brett wrote:
> Hi,
>
>    
>> Yep, you're right, they're political restrictions, and "geographic
>> restrictions" was a euphemism. I'm a huge believer in open source, but
>> personally, these political restrictions don't bother me at all.
>> However, you and Stefan had already both chimed in saying they did
>> bother you, so I'm very happy to pick another hosting option with no
>> argument, which is what I think I said above. I generally loathe
>> political debate, and I think cython-dev is a wildly inappropriate
>> place for it.
>>      
> If we see a powerful country trying to exclude less powerful countries
> from what we believe to be valid discourse, then, to say and do
> nothing is an act of passive agreement that not all of us can accept.
>
> We state our principles, and we try to follow them; when we agree, we
> do not have to discuss them.  The principles under which we operate
> are one of the great strengths of the open-source idea and the energy
> that we give to our projects.
>    

Others have said what really needs to be said, but I'd really like to 
make a point that it is worth bearing in mind that each person always 
has an individual set of reasons for contributing his or her time to 
open source development.

Some do it because of politics or principles about freedom. Some just 
wants to learn. Some just wants a challenge. Some just wants to improve 
the tool to do research more efficiently, or to make money faster.

Cython is not even under the GPL but Apache, so there's an invitation to 
join for pretty much whatever reason. We can't really assume anything 
about any of the political views of the authors just by them being 
Cython contributors alone.

(But, a priori, one can perhaps speculate that people would be more 
willing to work on Cython for pragmatic reasons alone than, say, Gnome 
or KDE, which are more going heads on against commercial alternatives.)

Dag Sverre


From matthew.brett at gmail.com  Fri Nov 19 08:11:48 2010
From: matthew.brett at gmail.com (Matthew Brett)
Date: Thu, 18 Nov 2010 23:11:48 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <4CE61E1A.4020501@student.matnat.uio.no>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<AANLkTin2SsA2Siwfsq+Zg6V3U9q7TEDRdJ1y9iKnRAMh@mail.gmail.com>
	<AANLkTinJm1z5_+2_maTn2GTDy5jB51d=NdGz_jhb+xPq@mail.gmail.com>
	<AANLkTimsQ+ezBTps9TRRnVueoqX6aad_7FxuAVekdqD2@mail.gmail.com>
	<AANLkTikX6tC2_tmAFkiiNvOW4L3pWUpujM147k1eYw5t@mail.gmail.com>
	<4CE61E1A.4020501@student.matnat.uio.no>
Message-ID: <AANLkTi=nUB8H7dJF7pRBcr1dt+ahh28sgjJXRkkokNaw@mail.gmail.com>

On Thu, Nov 18, 2010 at 10:50 PM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
>>> Yep, you're right, they're political restrictions, and "geographic
>>> restrictions" was a euphemism. I'm a huge believer in open source, but
>>> personally, these political restrictions don't bother me at all.
>>> However, you and Stefan had already both chimed in saying they did
>>> bother you, so I'm very happy to pick another hosting option with no
>>> argument, which is what I think I said above. I generally loathe
>>> political debate, and I think cython-dev is a wildly inappropriate
>>> place for it.
>>>
>> If we see a powerful country trying to exclude less powerful countries
>> from what we believe to be valid discourse, then, to say and do
>> nothing is an act of passive agreement that not all of us can accept.
>>
>> We state our principles, and we try to follow them; when we agree, we
>> do not have to discuss them. ?The principles under which we operate
>> are one of the great strengths of the open-source idea and the energy
>> that we give to our projects.
>>
>
> Others have said what really needs to be said, but I'd really like to
> make a point that it is worth bearing in mind that each person always
> has an individual set of reasons for contributing his or her time to
> open source development.
>
> Some do it because of politics or principles about freedom. Some just
> wants to learn. Some just wants a challenge. Some just wants to improve
> the tool to do research more efficiently, or to make money faster.
>
> Cython is not even under the GPL but Apache, so there's an invitation to
> join for pretty much whatever reason. We can't really assume anything
> about any of the political views of the authors just by them being
> Cython contributors alone.

Oh dear - I'm sorry, I could not resist the temptation to reply.

There's a difference between politics and principles.  I don't like
politics very much, but I care a lot about principles, and my
experience has been that the same is true of many open source coders.
 It's a founding principle of open source that we (I mean, open source
developers) do not discriminate against persons or groups [1].   I
feel strongly about that, and I hope that y'all do to.

I'm not claiming any right to have a say, other than the right of any
person to appeal to the basic principles of open source.

See you,

Matthew

From matthew.brett at gmail.com  Fri Nov 19 08:13:10 2010
From: matthew.brett at gmail.com (Matthew Brett)
Date: Thu, 18 Nov 2010 23:13:10 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTi=nUB8H7dJF7pRBcr1dt+ahh28sgjJXRkkokNaw@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<AANLkTin2SsA2Siwfsq+Zg6V3U9q7TEDRdJ1y9iKnRAMh@mail.gmail.com>
	<AANLkTinJm1z5_+2_maTn2GTDy5jB51d=NdGz_jhb+xPq@mail.gmail.com>
	<AANLkTimsQ+ezBTps9TRRnVueoqX6aad_7FxuAVekdqD2@mail.gmail.com>
	<AANLkTikX6tC2_tmAFkiiNvOW4L3pWUpujM147k1eYw5t@mail.gmail.com>
	<4CE61E1A.4020501@student.matnat.uio.no>
	<AANLkTi=nUB8H7dJF7pRBcr1dt+ahh28sgjJXRkkokNaw@mail.gmail.com>
Message-ID: <AANLkTikfKS-epW7WLqkQ-Nu-vtT8qbPjR6SxgzzhxrOt@mail.gmail.com>

Oops - late night missing reference:

On Thu, Nov 18, 2010 at 11:11 PM, Matthew Brett <matthew.brett at gmail.com> wrote:
> On Thu, Nov 18, 2010 at 10:50 PM, Dag Sverre Seljebotn
...
> ?It's a founding principle of open source that we (I mean, open source
> developers) do not discriminate against persons or groups [1].

[1] http://www.opensource.org/docs/osd

Best,

Matthew

From vitja.makarov at gmail.com  Fri Nov 19 08:27:47 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Fri, 19 Nov 2010 10:27:47 +0300
Subject: [Cython] [bug] inf loop
In-Reply-To: <4CE5FE5B.30108@behnel.de>
References: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>
	<AANLkTi=bWs0X8peV0fQQeQx4_okvduQue_f5nJD75Nta@mail.gmail.com>
	<4CE53725.5050707@behnel.de>
	<AANLkTi=s_tFgLxGXnb5xQxyjwc8gyquTuUBOGkNgr-d5@mail.gmail.com>
	<AANLkTimAYXqVOBWU_Bdr_ihP9_s5g3iEvZf0TJgmcVDN@mail.gmail.com>
	<4CE58419.7070000@behnel.de>
	<AANLkTi=v8BVSLduQkqmmmOZ1sKp9XSsx5XeMcU8FR7MA@mail.gmail.com>
	<4CE58BAC.6020105@behnel.de>
	<AANLkTik68aQ3SQ9Se=KmNpAU3jkQopxdw5E39L7SSMfm@mail.gmail.com>
	<4CE5FE5B.30108@behnel.de>
Message-ID: <AANLkTi=9sAz7Oy4yHRKR-AXWcwgEngmWd4ONYKS_gyD7@mail.gmail.com>

2010/11/19 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 18.11.2010 22:54:
>> 2010/11/18 Stefan Behnel:
>>> Vitja Makarov, 18.11.2010 21:17:
>>>> 2010/11/18 Stefan Behnel:
>>>>> The C compiler problems looked like closure issues, so they are more likely
>>>>> to be related.
>>>>
>>>> Is compile problem logged somewhere?
>>>
>>> You can find most of them here:
>>>
>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py26-c/704/parsed_console/
>>>
>>> Look out for "has no member named".
>>
>> Disabling this tests seems to make regression tests finish up.
>>
>> pyregr.test_socket
>> pyregr.test_threading
>
> Ok, I disabled them.
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>


New patch fixes #537, where are all that gcc errors came from.
It seems that patch enable many new tests that break test runner
itself, in threads.

With latest patch runtests couldn't exit because of thread lock
(join?), here is gdb traceback:
I don't know how to figure out which test causes this. Should we
disable some more thread-related tests.
Or as workaround use _exit() in testrunner

(gdb) bt
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
#1  0x00000000004dcac8 in PyThread_acquire_lock (lock=0x6d43c60,
waitflag=128) at Python/thread_pthread.h:309
#2  0x00000000004e00d2 in lock_PyThread_acquire_lock (self=0x1d113790,
args=<value optimised out>) at ./Modules/threadmodule.c:51
#3  0x00000000004a860a in call_function (f=0x3136af0, throwflag=<value
optimised out>) at Python/ceval.c:4012
#4  PyEval_EvalFrameEx (f=0x3136af0, throwflag=<value optimised out>)
at Python/ceval.c:2665
#5  0x00000000004a9e81 in PyEval_EvalCodeEx (co=0x3f7b430,
globals=<value optimised out>, locals=<value optimised out>, args=0x2,
argcount=0, kws=<value optimised out>,
    kwcount=0, defs=0x2c53668, defcount=1, closure=0x0) at Python/ceval.c:3252
#6  0x00000000004a8134 in fast_function (f=0x3c34650, throwflag=<value
optimised out>) at Python/ceval.c:4108
#7  call_function (f=0x3c34650, throwflag=<value optimised out>) at
Python/ceval.c:4033
#8  PyEval_EvalFrameEx (f=0x3c34650, throwflag=<value optimised out>)
at Python/ceval.c:2665
#9  0x00000000004a9e81 in PyEval_EvalCodeEx (co=0x3f60530,
globals=<value optimised out>, locals=<value optimised out>, args=0x2,
argcount=0, kws=<value optimised out>,
    kwcount=0, defs=0x3d239e8, defcount=1, closure=0x0) at Python/ceval.c:3252
#10 0x00000000004a8134 in fast_function (f=0x18172230,
throwflag=<value optimised out>) at Python/ceval.c:4108
#11 call_function (f=0x18172230, throwflag=<value optimised out>) at
Python/ceval.c:4033
#12 PyEval_EvalFrameEx (f=0x18172230, throwflag=<value optimised out>)
at Python/ceval.c:2665
#13 0x00000000004a9e81 in PyEval_EvalCodeEx (co=0x3f60eb0,
globals=<value optimised out>, locals=<value optimised out>,
args=0x23a6c68, argcount=0, kws=<value optimised out>,
    kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3252
#14 0x000000000050aa4e in function_call (func=0x3d12e60,
arg=0x23a6c50, kw=0x0) at Objects/funcobject.c:526
#15 0x00000000004195c7 in PyObject_Call (func=0x3d12e60, arg=0x80,
kw=0x0) at Objects/abstract.c:2522
#16 0x000000000042222f in instancemethod_call (func=0x3d12e60,
arg=0x23a6c50, kw=0x0) at Objects/classobject.c:2578
#17 0x000000000041c133 in PyObject_Call (callable=0x2e36550,
args=0x7f599bbc1050) at Objects/abstract.c:2522
#18 call_function_tail (callable=0x2e36550, args=0x7f599bbc1050) at
Objects/abstract.c:2554
#19 0x000000000041f340 in PyObject_CallMethod (o=<value optimised
out>, name=0x53f786 "_shutdown", format=0x5285b0 "") at
Objects/abstract.c:2631
#20 0x00000000004ca089 in wait_for_thread_shutdown () at Python/pythonrun.c:1696
#21 Py_Finalize () at Python/pythonrun.c:401
#22 0x00000000004c9810 in Py_Exit () at Python/pythonrun.c:1753
#23 handle_system_exit () at Python/pythonrun.c:1127
#24 0x00000000004c9e15 in PyErr_PrintEx (set_sys_last_vars=<value
optimised out>) at Python/pythonrun.c:1137
#25 0x00000000004cac40 in PyRun_SimpleFileExFlags (fp=<value optimised
out>, filename=0x7fff5f1d8f91 "runtests.py", closeit=1,
flags=0x7fff5f1d7270) at Python/pythonrun.c:940
#26 0x0000000000414b7a in Py_Main (argc=-1682423728, argv=<value
optimised out>) at Modules/main.c:599
#27 0x00007f599ae04d8e in __libc_start_main (main=<value optimised
out>, argc=<value optimised out>, ubp_av=<value optimised out>,
init=<value optimised out>,
    fini=<value optimised out>, rtld_fini=<value optimised out>,
stack_end=0x7fff5f1d7388) at libc-start.c:226
#28 0x0000000000413cd9 in _start ()
(gdb) info threads
  3 Thread 0x7f59857cb710 (LWP 17342)  sem_wait () at
../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  2 Thread 0x7f597cf7e710 (LWP 17397)  0x00007f599aec52e3 in select ()
at ../sysdeps/unix/syscall-template.S:82
* 1 Thread 0x7f599bc02700 (LWP 8116)  sem_wait () at
../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
(gdb) thread 2
[Switching to thread 2 (Thread 0x7f597cf7e710 (LWP 17397))]#0
0x00007f599aec52e3 in select () at
../sysdeps/unix/syscall-template.S:82
82	../sysdeps/unix/syscall-template.S: ??? ?????? ????? ??? ????????.
	in ../sysdeps/unix/syscall-template.S
(gdb) bt
#0  0x00007f599aec52e3 in select () at ../sysdeps/unix/syscall-template.S:82
#1  0x00007f596b8e2f18 in ?? () from /usr/lib/libtcl8.5.so.0
#2  0x00007f599b7fa971 in start_thread (arg=<value optimised out>) at
pthread_create.c:304
#3  0x00007f599aecc94d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#4  0x0000000000000000 in ?? ()
(gdb) thread 3
[Switching to thread 3 (Thread 0x7f59857cb710 (LWP 17342))]#0
sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
86	../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S: ??? ?????? ?????
??? ????????.
	in ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S
(gdb) bt
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
#1  0x00000000004dcac8 in PyThread_acquire_lock (lock=0x14816d80,
waitflag=128) at Python/thread_pthread.h:309
#2  0x00000000004e00d2 in lock_PyThread_acquire_lock (self=0x26964370,
args=<value optimised out>) at ./Modules/threadmodule.c:51
#3  0x00000000004a860a in call_function (f=0x24b11010,
throwflag=<value optimised out>) at Python/ceval.c:4012
#4  PyEval_EvalFrameEx (f=0x24b11010, throwflag=<value optimised out>)
at Python/ceval.c:2665
#5  0x00000000004a9e81 in PyEval_EvalCodeEx (co=0x3f7b430,
globals=<value optimised out>, locals=<value optimised out>,
args=0x250b8a58, argcount=0, kws=<value optimised out>,
    kwcount=0, defs=0x2c53668, defcount=1, closure=0x0) at Python/ceval.c:3252
#6  0x00000000004a8134 in fast_function (f=0x250b88d0,
throwflag=<value optimised out>) at Python/ceval.c:4108
#7  call_function (f=0x250b88d0, throwflag=<value optimised out>) at
Python/ceval.c:4033
#8  PyEval_EvalFrameEx (f=0x250b88d0, throwflag=<value optimised out>)
at Python/ceval.c:2665
#9  0x00000000004a9e81 in PyEval_EvalCodeEx (co=0x3f7bdb0,
globals=<value optimised out>, locals=<value optimised out>, args=0x2,
argcount=0, kws=<value optimised out>,
    kwcount=0, defs=0x3d23ae8, defcount=1, closure=0x0) at Python/ceval.c:3252
#10 0x000000000050aa4e in function_call (func=0x3d120c8,
arg=0x26b704d0, kw=0x0) at Objects/funcobject.c:526
#11 0x00000000004195c7 in PyObject_Call (func=0x3d120c8, arg=0x80,
kw=0x0) at Objects/abstract.c:2522
#12 0x000000000042222f in instancemethod_call (func=0x3d120c8,
arg=0x26b704d0, kw=0x0) at Objects/classobject.c:2578
#13 0x00000000004195c7 in PyObject_Call (func=0x26b72370, arg=0x80,
kw=0x0) at Objects/abstract.c:2522
#14 0x00007f596cb85f36 in ?? ()
#15 0x0000000000000000 in ?? ()

-- 
vitja.

From fperez.net at gmail.com  Fri Nov 19 08:47:16 2010
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 18 Nov 2010 23:47:16 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTimByBUBhwuJSz7Ok97u+UL3vF-Pt14t8PxzML45@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<4CE3AF74.40703@student.matnat.uio.no>
	<AANLkTi=+AktpDE=yBrXFQWKbC+NKLc4AQfwYTuFhSzxL@mail.gmail.com>
	<AANLkTimATefKRkvj6PKzgLfO6YyCRr6gh56O7H9JkuOp@mail.gmail.com>
	<AANLkTikz_=jzA-10+hQAJZ=QaKi+=a238HNY3Kc_-Ujn@mail.gmail.com>
	<AANLkTimByBUBhwuJSz7Ok97u+UL3vF-Pt14t8PxzML45@mail.gmail.com>
Message-ID: <AANLkTikvPuTi3-GVHFRx5mYD-BWKyK3jxQY_58Ud5s9M@mail.gmail.com>

On Thu, Nov 18, 2010 at 8:11 PM, Ondrej Certik <ondrej at certik.cz> wrote:
> Well, but if I go here:
>
> https://github.com/fperez/datarray-doc/branches
>
> it says the main repo is "master", and "gh-pages" is another branch.

Ah, I just forgot to delete master, that's all.  It's actually empty
(it might have just the first commit in it).  I'm not too picky with
branch cleanup, I guess :)

Cheers,

f

From fperez.net at gmail.com  Fri Nov 19 08:56:04 2010
From: fperez.net at gmail.com (Fernando Perez)
Date: Thu, 18 Nov 2010 23:56:04 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTinSJMv_Mr+Wfegr3yNuGJfFO6dLm09-itNA2Y08@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<AANLkTimRqTQs6E47_Uy6SVXuzLBi6ek4MMgmaAdH5Sfs@mail.gmail.com>
	<AANLkTi=PQtq0kP+a-x6Qxz1FFTvLtOyPY2=kKWm_3xUf@mail.gmail.com>
	<AANLkTinSJMv_Mr+Wfegr3yNuGJfFO6dLm09-itNA2Y08@mail.gmail.com>
Message-ID: <AANLkTiku_Z0cd=mEaeNa4u54tP=LGM+EH2CHSqCLQ=1Y@mail.gmail.com>

On Thu, Nov 18, 2010 at 7:25 PM, Ondrej Certik <ondrej at certik.cz> wrote:
> Just to clarify, it's not *that* bad, for example we use it quite
> extensively here:
>
> https://github.com/hpfem/hermes/issues
> https://github.com/hpfem/femhub/issues
>
> so it's usable, but obviously, for someone coming from Google Code,
> it's a step back.

Talking today with Min about this, he made a very good point: it's not
that bad when working on a specific issue, but it's annoying when
doing 'high level' work for a project where you want to do things that
operate on the whole bug list: searching, milestone targeting,
reporting, etc.

And for some reason, issue search is simply broken.  Witness searching
the ipython issues for something liike 'gtk':

https://github.com/ipython/ipython/issuesearch?state=open&q=gtk

a bunch of issues are listed at the top that don't mention gtk
anywhere.  Those issues seem to appear no matter what you search for,
it's really weird.

In the github support site there's a lot of complaining about this,
but no fix yet...

I have to say that at the end of the day, we're *thrilled* to use
github.  The interface is fluid, the features are very well thought
out, the organization/team setup is excellent and the code review
machinery is just brilliant, and very, very useful in actual practice.
 Reviewing pull requests is basically as easy as can be, with
near-zero tool-induced overhead and almost everything you could want
for a great code review right at your fingertips.  Here's what a
complex, active review looks like, for example:

https://github.com/ipython/ipython/pull/179

The impact of the excellent integration of all these features vastly
outweighs the annoyances of the issue tracker.  And since it's obvious
they have a really good team behind the site, and people have been
complaining about this for a while, I'm going to trust it's in their
private pipeline of fixes and we'll see some improvements at some
point.

I just wanted to give you some useful, detailed points from real
experience so you can make the decision that best fits cython.  Don't
hesitate to bug for specifics if you need any.

Regards,

f

From dagss at student.matnat.uio.no  Fri Nov 19 09:22:01 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 19 Nov 2010 09:22:01 +0100
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTiku_Z0cd=mEaeNa4u54tP=LGM+EH2CHSqCLQ=1Y@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de>
	<4CE380E0.7070403@behnel.de>	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>	<AANLkTimRqTQs6E47_Uy6SVXuzLBi6ek4MMgmaAdH5Sfs@mail.gmail.com>	<AANLkTi=PQtq0kP+a-x6Qxz1FFTvLtOyPY2=kKWm_3xUf@mail.gmail.com>	<AANLkTinSJMv_Mr+Wfegr3yNuGJfFO6dLm09-itNA2Y08@mail.gmail.com>
	<AANLkTiku_Z0cd=mEaeNa4u54tP=LGM+EH2CHSqCLQ=1Y@mail.gmail.com>
Message-ID: <4CE633A9.8020007@student.matnat.uio.no>

On 11/19/2010 08:56 AM, Fernando Perez wrote:
> On Thu, Nov 18, 2010 at 7:25 PM, Ondrej Certik<ondrej at certik.cz>  wrote:
>    
>> Just to clarify, it's not *that* bad, for example we use it quite
>> extensively here:
>>
>> https://github.com/hpfem/hermes/issues
>> https://github.com/hpfem/femhub/issues
>>
>> so it's usable, but obviously, for someone coming from Google Code,
>> it's a step back.
>>      
> Talking today with Min about this, he made a very good point: it's not
> that bad when working on a specific issue, but it's annoying when
> doing 'high level' work for a project where you want to do things that
> operate on the whole bug list: searching, milestone targeting,
> reporting, etc.
>
> And for some reason, issue search is simply broken.  Witness searching
> the ipython issues for something liike 'gtk':
>
> https://github.com/ipython/ipython/issuesearch?state=open&q=gtk
>
> a bunch of issues are listed at the top that don't mention gtk
> anywhere.  Those issues seem to appear no matter what you search for,
> it's really weird.
>
> In the github support site there's a lot of complaining about this,
> but no fix yet...
>
> I have to say that at the end of the day, we're *thrilled* to use
> github.  The interface is fluid, the features are very well thought
> out, the organization/team setup is excellent and the code review
> machinery is just brilliant, and very, very useful in actual practice.
>   Reviewing pull requests is basically as easy as can be, with
> near-zero tool-induced overhead and almost everything you could want
> for a great code review right at your fingertips.  Here's what a
> complex, active review looks like, for example:
>
> https://github.com/ipython/ipython/pull/179
>
> The impact of the excellent integration of all these features vastly
> outweighs the annoyances of the issue tracker.  And since it's obvious
> they have a really good team behind the site, and people have been
> complaining about this for a while, I'm going to trust it's in their
> private pipeline of fixes and we'll see some improvements at some
> point.
>    
> I just wanted to give you some useful, detailed points from real
> experience so you can make the decision that best fits cython.  Don't
> hesitate to bug for specifics if you need any.
>    

Out of ignorance: Just how interlinked is the code review process with 
the issue tracker? Do you typically link code review up to tickets in 
some way, or are pull requests completely standalone?

With e.g. Google Code it seems we'd need to host the web page and 
perhaps the wiki elsewhere (as Ondrej noted, the wiki on Google Code is 
only editable by project members), so hosting the bug tracker somewhere 
else and use github for the rest doesn't seem so bad.

Dag Sverre

From fperez.net at gmail.com  Fri Nov 19 09:37:18 2010
From: fperez.net at gmail.com (Fernando Perez)
Date: Fri, 19 Nov 2010 00:37:18 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <4CE633A9.8020007@student.matnat.uio.no>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<AANLkTimRqTQs6E47_Uy6SVXuzLBi6ek4MMgmaAdH5Sfs@mail.gmail.com>
	<AANLkTi=PQtq0kP+a-x6Qxz1FFTvLtOyPY2=kKWm_3xUf@mail.gmail.com>
	<AANLkTinSJMv_Mr+Wfegr3yNuGJfFO6dLm09-itNA2Y08@mail.gmail.com>
	<AANLkTiku_Z0cd=mEaeNa4u54tP=LGM+EH2CHSqCLQ=1Y@mail.gmail.com>
	<4CE633A9.8020007@student.matnat.uio.no>
Message-ID: <AANLkTi=aM9uTZXrBVdRift-7NjxnPArEziqt_wEJgX1d@mail.gmail.com>

On Fri, Nov 19, 2010 at 12:22 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> Out of ignorance: Just how interlinked is the code review process with
> the issue tracker? Do you typically link code review up to tickets in
> some way, or are pull requests completely standalone?

When a pull request is made, an issue is automatically created by
github that links to the pull request.  E.g.:

https://github.com/ipython/ipython/pull/199
https://github.com/ipython/ipython/issues/199

If the pull request is merged *without rebasing* (by following
something like the 3 steps given at the bottom of the request page) ,
github automatically notices that the commits from the request went
in, and closes both the request and the issue for you without  the
need for any extra action.  The act of pushing causes the closing of
all.

If you do rebase, github can't tell that you did merge the contents,
so you'll have to click manually on the close button yourself (but
closing the request auto-closes the associated bug history as well).

Pull request pages remain available even after closed, so any
discussion that takes place during a pull request (which often
involves useful design discussions) remains accessible later on.  This
is something I think is very useful and I'm glad they thought of it
(e.g. on launchpad, closed merge requests were very hard to find: they
didn't get deleted, but there was nothing in the interface to find
them unless you *manually* had made a bug for it to fish it back out
later. Awful).

In summary, pull/issue linking is:
- automatic for opening,
- automatic for no-rebase closing
-if rebase did happen, you only need to manually close one for both to
get closed.

> With e.g. Google Code it seems we'd need to host the web page and
> perhaps the wiki elsewhere (as Ondrej noted, the wiki on Google Code is
> only editable by project members), so hosting the bug tracker somewhere
> else and use github for the rest doesn't seem so bad.

It's worth noting what Ondrej pointed out in his blog review: since
github lets you host both the wiki and the webpages with more flexible
controls (everything is a repo), you could get the whole thing on
github.

One more nice thing about having the issues at github, is that any
commit that closes a bug only needs to say 'closes gh-NNN'  or 'closes
#NNN', and github will

- automatically close the bug
- make a link at the closed bug page to the commit page
- make a link at the commit entry to the bug

See for example:

https://github.com/ipython/ipython/issues/148
https://github.com/ipython/ipython/commit/a55b74a629425e3344c8bad7b5994ebfc5b12085

This is very handy, because you know that the moment that commit gets
pushed, everything will be automatically get taken care of.

This good integration between commits and issues is a point in favor
of keeping the issue hosting at github, despite the limnitations I
complained about earlier.

HTH,

f

From dagss at student.matnat.uio.no  Fri Nov 19 09:46:35 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Fri, 19 Nov 2010 09:46:35 +0100
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTi=aM9uTZXrBVdRift-7NjxnPArEziqt_wEJgX1d@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de>
	<4CE380E0.7070403@behnel.de>	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>	<AANLkTimRqTQs6E47_Uy6SVXuzLBi6ek4MMgmaAdH5Sfs@mail.gmail.com>	<AANLkTi=PQtq0kP+a-x6Qxz1FFTvLtOyPY2=kKWm_3xUf@mail.gmail.com>	<AANLkTinSJMv_Mr+Wfegr3yNuGJfFO6dLm09-itNA2Y08@mail.gmail.com>	<AANLkTiku_Z0cd=mEaeNa4u54tP=LGM+EH2CHSqCLQ=1Y@mail.gmail.com>	<4CE633A9.8020007@student.matnat.uio.no>
	<AANLkTi=aM9uTZXrBVdRift-7NjxnPArEziqt_wEJgX1d@mail.gmail.com>
Message-ID: <4CE6396B.3030706@student.matnat.uio.no>

On 11/19/2010 09:37 AM, Fernando Perez wrote:
> On Fri, Nov 19, 2010 at 12:22 AM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no>  wrote:
>    
>> Out of ignorance: Just how interlinked is the code review process with
>> the issue tracker? Do you typically link code review up to tickets in
>> some way, or are pull requests completely standalone?
>>      
> When a pull request is made, an issue is automatically created by
> github that links to the pull request.  E.g.:
>
> https://github.com/ipython/ipython/pull/199
> https://github.com/ipython/ipython/issues/199
>
> If the pull request is merged *without rebasing* (by following
> something like the 3 steps given at the bottom of the request page) ,
> github automatically notices that the commits from the request went
> in, and closes both the request and the issue for you without  the
> need for any extra action.  The act of pushing causes the closing of
> all.
>
> If you do rebase, github can't tell that you did merge the contents,
> so you'll have to click manually on the close button yourself (but
> closing the request auto-closes the associated bug history as well).
>
> Pull request pages remain available even after closed, so any
> discussion that takes place during a pull request (which often
> involves useful design discussions) remains accessible later on.  This
> is something I think is very useful and I'm glad they thought of it
> (e.g. on launchpad, closed merge requests were very hard to find: they
> didn't get deleted, but there was nothing in the interface to find
> them unless you *manually* had made a bug for it to fish it back out
> later. Awful).
>
> In summary, pull/issue linking is:
> - automatic for opening,
> - automatic for no-rebase closing
> -if rebase did happen, you only need to manually close one for both to
> get closed.
>
>    
>> With e.g. Google Code it seems we'd need to host the web page and
>> perhaps the wiki elsewhere (as Ondrej noted, the wiki on Google Code is
>> only editable by project members), so hosting the bug tracker somewhere
>> else and use github for the rest doesn't seem so bad.
>>      
> It's worth noting what Ondrej pointed out in his blog review: since
> github lets you host both the wiki and the webpages with more flexible
> controls (everything is a repo), you could get the whole thing on
> github.
>
> One more nice thing about having the issues at github, is that any
> commit that closes a bug only needs to say 'closes gh-NNN'  or 'closes
> #NNN', and github will
>
> - automatically close the bug
> - make a link at the closed bug page to the commit page
> - make a link at the commit entry to the bug
>
> See for example:
>
> https://github.com/ipython/ipython/issues/148
> https://github.com/ipython/ipython/commit/a55b74a629425e3344c8bad7b5994ebfc5b12085
>
> This is very handy, because you know that the moment that commit gets
> pushed, everything will be automatically get taken care of.
>
> This good integration between commits and issues is a point in favor
> of keeping the issue hosting at github, despite the limnitations I
> complained about earlier.
>    

Thanks again. Seems to me that if the limitations of GitHub is primarily 
in getting reports (one can always create tags for each milestone, tags 
for each component, and so on). And with an open API, I assume that 
making reports is something one could do on a seperate page outside of 
GitHub's control (say, on appspot.google.com).

As long as I had a simple webapp that would take all tags starting with 
"Milestone" and group tickets by them, that would fill 80% of my needs 
from Trac. But that seems like a half hour script once one knows the 
API. Of course, I don't want that to be *my* half hour... I'd love to 
stumble over somebody's extensive suite of reporting tools for GitHub 
issues :-)

Dag Sverre

From fperez.net at gmail.com  Fri Nov 19 09:52:02 2010
From: fperez.net at gmail.com (Fernando Perez)
Date: Fri, 19 Nov 2010 00:52:02 -0800
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <4CE6396B.3030706@student.matnat.uio.no>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<AANLkTimRqTQs6E47_Uy6SVXuzLBi6ek4MMgmaAdH5Sfs@mail.gmail.com>
	<AANLkTi=PQtq0kP+a-x6Qxz1FFTvLtOyPY2=kKWm_3xUf@mail.gmail.com>
	<AANLkTinSJMv_Mr+Wfegr3yNuGJfFO6dLm09-itNA2Y08@mail.gmail.com>
	<AANLkTiku_Z0cd=mEaeNa4u54tP=LGM+EH2CHSqCLQ=1Y@mail.gmail.com>
	<4CE633A9.8020007@student.matnat.uio.no>
	<AANLkTi=aM9uTZXrBVdRift-7NjxnPArEziqt_wEJgX1d@mail.gmail.com>
	<4CE6396B.3030706@student.matnat.uio.no>
Message-ID: <AANLkTi=t7gyZU3vH2N5At739sH-coCPqZXFwM1xJJw0-@mail.gmail.com>

On Fri, Nov 19, 2010 at 12:46 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:

> Thanks again. Seems to me that if the limitations of GitHub is primarily
> in getting reports (one can always create tags for each milestone, tags
> for each component, and so on). And with an open API, I assume that
> making reports is something one could do on a seperate page outside of
> GitHub's control (say, on appspot.google.com).
>
> As long as I had a simple webapp that would take all tags starting with
> "Milestone" and group tickets by them, that would fill 80% of my needs
> from Trac. But that seems like a half hour script once one knows the
> API. Of course, I don't want that to be *my* half hour... I'd love to
> stumble over somebody's extensive suite of reporting tools for GitHub
> issues :-)

Absolutely.  And a lot of the work has already been done, there are
two python projects that already wrap access to the github api:

https://github.com/ask/python-github2
https://github.com/jsmits/github-cli

The second is more oriented towards command-line use and not very nice
to use programatically (aside bash scripting), I tried both and
settled on using the first when I wrote my code to do
launchpad->github bug migration.  With that, writing a small reporting
tool would be pretty trivial.

Cheers,

f

From vitja.makarov at gmail.com  Fri Nov 19 16:36:31 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Fri, 19 Nov 2010 18:36:31 +0300
Subject: [Cython] [bug] inf loop
In-Reply-To: <AANLkTi=9sAz7Oy4yHRKR-AXWcwgEngmWd4ONYKS_gyD7@mail.gmail.com>
References: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>
	<AANLkTi=bWs0X8peV0fQQeQx4_okvduQue_f5nJD75Nta@mail.gmail.com>
	<4CE53725.5050707@behnel.de>
	<AANLkTi=s_tFgLxGXnb5xQxyjwc8gyquTuUBOGkNgr-d5@mail.gmail.com>
	<AANLkTimAYXqVOBWU_Bdr_ihP9_s5g3iEvZf0TJgmcVDN@mail.gmail.com>
	<4CE58419.7070000@behnel.de>
	<AANLkTi=v8BVSLduQkqmmmOZ1sKp9XSsx5XeMcU8FR7MA@mail.gmail.com>
	<4CE58BAC.6020105@behnel.de>
	<AANLkTik68aQ3SQ9Se=KmNpAU3jkQopxdw5E39L7SSMfm@mail.gmail.com>
	<4CE5FE5B.30108@behnel.de>
	<AANLkTi=9sAz7Oy4yHRKR-AXWcwgEngmWd4ONYKS_gyD7@mail.gmail.com>
Message-ID: <AANLkTikMz4KD72MhycysTKY+UBKMPBKOoLqiiW056hdR@mail.gmail.com>

19 ?????? 2010??. 10:27 ???????????? Vitja Makarov
<vitja.makarov at gmail.com> ???????:
> 2010/11/19 Stefan Behnel <stefan_ml at behnel.de>:
>> Vitja Makarov, 18.11.2010 22:54:
>>> 2010/11/18 Stefan Behnel:
>>>> Vitja Makarov, 18.11.2010 21:17:
>>>>> 2010/11/18 Stefan Behnel:
>>>>>> The C compiler problems looked like closure issues, so they are more likely
>>>>>> to be related.
>>>>>
>>>>> Is compile problem logged somewhere?
>>>>
>>>> You can find most of them here:
>>>>
>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py26-c/704/parsed_console/
>>>>
>>>> Look out for "has no member named".
>>>
>>> Disabling this tests seems to make regression tests finish up.
>>>
>>> pyregr.test_socket
>>> pyregr.test_threading
>>
>> Ok, I disabled them.
>>
>> Stefan
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>
>
>
> New patch fixes #537, where are all that gcc errors came from.
> It seems that patch enable many new tests that break test runner
> itself, in threads.
>
> With latest patch runtests couldn't exit because of thread lock
> (join?), here is gdb traceback:
> I don't know how to figure out which test causes this. Should we
> disable some more thread-related tests.
> Or as workaround use _exit() in testrunner
>
> (gdb) bt
> #0 ?sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
> #1 ?0x00000000004dcac8 in PyThread_acquire_lock (lock=0x6d43c60,
> waitflag=128) at Python/thread_pthread.h:309
> #2 ?0x00000000004e00d2 in lock_PyThread_acquire_lock (self=0x1d113790,
> args=<value optimised out>) at ./Modules/threadmodule.c:51
> #3 ?0x00000000004a860a in call_function (f=0x3136af0, throwflag=<value
> optimised out>) at Python/ceval.c:4012
> #4 ?PyEval_EvalFrameEx (f=0x3136af0, throwflag=<value optimised out>)
> at Python/ceval.c:2665
> #5 ?0x00000000004a9e81 in PyEval_EvalCodeEx (co=0x3f7b430,
> globals=<value optimised out>, locals=<value optimised out>, args=0x2,
> argcount=0, kws=<value optimised out>,
> ? ?kwcount=0, defs=0x2c53668, defcount=1, closure=0x0) at Python/ceval.c:3252
> #6 ?0x00000000004a8134 in fast_function (f=0x3c34650, throwflag=<value
> optimised out>) at Python/ceval.c:4108
> #7 ?call_function (f=0x3c34650, throwflag=<value optimised out>) at
> Python/ceval.c:4033
> #8 ?PyEval_EvalFrameEx (f=0x3c34650, throwflag=<value optimised out>)
> at Python/ceval.c:2665
> #9 ?0x00000000004a9e81 in PyEval_EvalCodeEx (co=0x3f60530,
> globals=<value optimised out>, locals=<value optimised out>, args=0x2,
> argcount=0, kws=<value optimised out>,
> ? ?kwcount=0, defs=0x3d239e8, defcount=1, closure=0x0) at Python/ceval.c:3252
> #10 0x00000000004a8134 in fast_function (f=0x18172230,
> throwflag=<value optimised out>) at Python/ceval.c:4108
> #11 call_function (f=0x18172230, throwflag=<value optimised out>) at
> Python/ceval.c:4033
> #12 PyEval_EvalFrameEx (f=0x18172230, throwflag=<value optimised out>)
> at Python/ceval.c:2665
> #13 0x00000000004a9e81 in PyEval_EvalCodeEx (co=0x3f60eb0,
> globals=<value optimised out>, locals=<value optimised out>,
> args=0x23a6c68, argcount=0, kws=<value optimised out>,
> ? ?kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3252
> #14 0x000000000050aa4e in function_call (func=0x3d12e60,
> arg=0x23a6c50, kw=0x0) at Objects/funcobject.c:526
> #15 0x00000000004195c7 in PyObject_Call (func=0x3d12e60, arg=0x80,
> kw=0x0) at Objects/abstract.c:2522
> #16 0x000000000042222f in instancemethod_call (func=0x3d12e60,
> arg=0x23a6c50, kw=0x0) at Objects/classobject.c:2578
> #17 0x000000000041c133 in PyObject_Call (callable=0x2e36550,
> args=0x7f599bbc1050) at Objects/abstract.c:2522
> #18 call_function_tail (callable=0x2e36550, args=0x7f599bbc1050) at
> Objects/abstract.c:2554
> #19 0x000000000041f340 in PyObject_CallMethod (o=<value optimised
> out>, name=0x53f786 "_shutdown", format=0x5285b0 "") at
> Objects/abstract.c:2631
> #20 0x00000000004ca089 in wait_for_thread_shutdown () at Python/pythonrun.c:1696
> #21 Py_Finalize () at Python/pythonrun.c:401
> #22 0x00000000004c9810 in Py_Exit () at Python/pythonrun.c:1753
> #23 handle_system_exit () at Python/pythonrun.c:1127
> #24 0x00000000004c9e15 in PyErr_PrintEx (set_sys_last_vars=<value
> optimised out>) at Python/pythonrun.c:1137
> #25 0x00000000004cac40 in PyRun_SimpleFileExFlags (fp=<value optimised
> out>, filename=0x7fff5f1d8f91 "runtests.py", closeit=1,
> flags=0x7fff5f1d7270) at Python/pythonrun.c:940
> #26 0x0000000000414b7a in Py_Main (argc=-1682423728, argv=<value
> optimised out>) at Modules/main.c:599
> #27 0x00007f599ae04d8e in __libc_start_main (main=<value optimised
> out>, argc=<value optimised out>, ubp_av=<value optimised out>,
> init=<value optimised out>,
> ? ?fini=<value optimised out>, rtld_fini=<value optimised out>,
> stack_end=0x7fff5f1d7388) at libc-start.c:226
> #28 0x0000000000413cd9 in _start ()
> (gdb) info threads
> ?3 Thread 0x7f59857cb710 (LWP 17342) ?sem_wait () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
> ?2 Thread 0x7f597cf7e710 (LWP 17397) ?0x00007f599aec52e3 in select ()
> at ../sysdeps/unix/syscall-template.S:82
> * 1 Thread 0x7f599bc02700 (LWP 8116) ?sem_wait () at
> ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
> (gdb) thread 2
> [Switching to thread 2 (Thread 0x7f597cf7e710 (LWP 17397))]#0
> 0x00007f599aec52e3 in select () at
> ../sysdeps/unix/syscall-template.S:82
> 82 ? ? ?../sysdeps/unix/syscall-template.S: ??? ?????? ????? ??? ????????.
> ? ? ? ?in ../sysdeps/unix/syscall-template.S
> (gdb) bt
> #0 ?0x00007f599aec52e3 in select () at ../sysdeps/unix/syscall-template.S:82
> #1 ?0x00007f596b8e2f18 in ?? () from /usr/lib/libtcl8.5.so.0
> #2 ?0x00007f599b7fa971 in start_thread (arg=<value optimised out>) at
> pthread_create.c:304
> #3 ?0x00007f599aecc94d in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #4 ?0x0000000000000000 in ?? ()
> (gdb) thread 3
> [Switching to thread 3 (Thread 0x7f59857cb710 (LWP 17342))]#0
> sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
> 86 ? ? ?../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S: ??? ?????? ?????
> ??? ????????.
> ? ? ? ?in ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S
> (gdb) bt
> #0 ?sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
> #1 ?0x00000000004dcac8 in PyThread_acquire_lock (lock=0x14816d80,
> waitflag=128) at Python/thread_pthread.h:309
> #2 ?0x00000000004e00d2 in lock_PyThread_acquire_lock (self=0x26964370,
> args=<value optimised out>) at ./Modules/threadmodule.c:51
> #3 ?0x00000000004a860a in call_function (f=0x24b11010,
> throwflag=<value optimised out>) at Python/ceval.c:4012
> #4 ?PyEval_EvalFrameEx (f=0x24b11010, throwflag=<value optimised out>)
> at Python/ceval.c:2665
> #5 ?0x00000000004a9e81 in PyEval_EvalCodeEx (co=0x3f7b430,
> globals=<value optimised out>, locals=<value optimised out>,
> args=0x250b8a58, argcount=0, kws=<value optimised out>,
> ? ?kwcount=0, defs=0x2c53668, defcount=1, closure=0x0) at Python/ceval.c:3252
> #6 ?0x00000000004a8134 in fast_function (f=0x250b88d0,
> throwflag=<value optimised out>) at Python/ceval.c:4108
> #7 ?call_function (f=0x250b88d0, throwflag=<value optimised out>) at
> Python/ceval.c:4033
> #8 ?PyEval_EvalFrameEx (f=0x250b88d0, throwflag=<value optimised out>)
> at Python/ceval.c:2665
> #9 ?0x00000000004a9e81 in PyEval_EvalCodeEx (co=0x3f7bdb0,
> globals=<value optimised out>, locals=<value optimised out>, args=0x2,
> argcount=0, kws=<value optimised out>,
> ? ?kwcount=0, defs=0x3d23ae8, defcount=1, closure=0x0) at Python/ceval.c:3252
> #10 0x000000000050aa4e in function_call (func=0x3d120c8,
> arg=0x26b704d0, kw=0x0) at Objects/funcobject.c:526
> #11 0x00000000004195c7 in PyObject_Call (func=0x3d120c8, arg=0x80,
> kw=0x0) at Objects/abstract.c:2522
> #12 0x000000000042222f in instancemethod_call (func=0x3d120c8,
> arg=0x26b704d0, kw=0x0) at Objects/classobject.c:2578
> #13 0x00000000004195c7 in PyObject_Call (func=0x26b72370, arg=0x80,
> kw=0x0) at Objects/abstract.c:2522
> #14 0x00007f596cb85f36 in ?? ()
> #15 0x0000000000000000 in ?? ()
>

Still can't find which one test have thread left.

Here is workaround:

import threading
import time
import os

class XXX(threading.Thread):
    def run(self):
        print ('Bad thread')
        while True:
            time.sleep(1)
XXX().start()

time.sleep(.1)

if len(threading._active) > 1:
    import warnings
    print("Warning more then one threads left, forcing _exit()")
    os._exit(0)


I'm not sure it is a good idea to add this into runtests.py

From dsdale24 at gmail.com  Fri Nov 19 22:30:36 2010
From: dsdale24 at gmail.com (Darren Dale)
Date: Fri, 19 Nov 2010 16:30:36 -0500
Subject: [Cython] Can't push to cython-devel
In-Reply-To: <AANLkTiku_Z0cd=mEaeNa4u54tP=LGM+EH2CHSqCLQ=1Y@mail.gmail.com>
References: <4CE26F1A.4000607@behnel.de> <4CE380E0.7070403@behnel.de>
	<AANLkTimBKZVaO6qpH9NDvfdG4VF6y+QCFc44bOtkKr1d@mail.gmail.com>
	<AANLkTimRqTQs6E47_Uy6SVXuzLBi6ek4MMgmaAdH5Sfs@mail.gmail.com>
	<AANLkTi=PQtq0kP+a-x6Qxz1FFTvLtOyPY2=kKWm_3xUf@mail.gmail.com>
	<AANLkTinSJMv_Mr+Wfegr3yNuGJfFO6dLm09-itNA2Y08@mail.gmail.com>
	<AANLkTiku_Z0cd=mEaeNa4u54tP=LGM+EH2CHSqCLQ=1Y@mail.gmail.com>
Message-ID: <AANLkTimPxjbM8ho7AuJ3_SD8wYAX7azF63RE_Z1Kiz3K@mail.gmail.com>

On Fri, Nov 19, 2010 at 2:56 AM, Fernando Perez <fperez.net at gmail.com> wrote:
> On Thu, Nov 18, 2010 at 7:25 PM, Ondrej Certik <ondrej at certik.cz> wrote:
>> Just to clarify, it's not *that* bad, for example we use it quite
>> extensively here:
>>
>> https://github.com/hpfem/hermes/issues
>> https://github.com/hpfem/femhub/issues
>>
>> so it's usable, but obviously, for someone coming from Google Code,
>> it's a step back.
>
> Talking today with Min about this, he made a very good point: it's not
> that bad when working on a specific issue, but it's annoying when
> doing 'high level' work for a project where you want to do things that
> operate on the whole bug list: searching, milestone targeting,
> reporting, etc.
>
> And for some reason, issue search is simply broken.

Some context: http://support.github.com/discussions/issues-issues/330-issue-tracker-usability
Note how quickly they closed discussion of the issue. That gives me
some idea of how aware they are of the community's opinion of their
issue tracker.

Darren

From robertwb at math.washington.edu  Sat Nov 20 00:03:21 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 19 Nov 2010 15:03:21 -0800
Subject: [Cython] GitHub for Cython
Message-ID: <AANLkTinFLOtcHvdUj-9SnX=7uZELoH9W74zRfy37df=z@mail.gmail.com>

It seems that the clear consensus, or at least the one thing that we
can all be happy about, is moving to GitHub (crossing our fingers that
the issue tracker improves quickly). I've created
https://github.com/cython , but haven't started to migrate the rest of
the stuff over. I'm thinking we can still stick with the push-to-main
option for the core developers (which seems to still be appropriate
for our project right now, and also works well with stuff like the
build-bot), but can easily fork for experimental work, reviews, and
anyone else who wants to contribute.

Fortunately, with a DVCS, we don't have to "flip the switch" all at
once. I'll be migrating the site off of the VM and the trac/wiki out
to GitHub soonish (and this will probably be a flip-the-switch kind of
thing), but would welcome any help. For those who need admin access,
let me know what your usernames are.

- Robert

From vitja.makarov at gmail.com  Sat Nov 20 00:10:34 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 20 Nov 2010 02:10:34 +0300
Subject: [Cython] GitHub for Cython
In-Reply-To: <AANLkTinFLOtcHvdUj-9SnX=7uZELoH9W74zRfy37df=z@mail.gmail.com>
References: <AANLkTinFLOtcHvdUj-9SnX=7uZELoH9W74zRfy37df=z@mail.gmail.com>
Message-ID: <AANLkTik6T6PY0sysSSK=8wD6WL8rovfmctNFbHHNmy9j@mail.gmail.com>

2010/11/20 Robert Bradshaw <robertwb at math.washington.edu>:
> It seems that the clear consensus, or at least the one thing that we
> can all be happy about, is moving to GitHub (crossing our fingers that
> the issue tracker improves quickly). I've created
> https://github.com/cython , but haven't started to migrate the rest of
> the stuff over. I'm thinking we can still stick with the push-to-main
> option for the core developers (which seems to still be appropriate
> for our project right now, and also works well with stuff like the
> build-bot), but can easily fork for experimental work, reviews, and
> anyone else who wants to contribute.
>
> Fortunately, with a DVCS, we don't have to "flip the switch" all at
> once. I'll be migrating the site off of the VM and the trac/wiki out
> to GitHub soonish (and this will probably be a flip-the-switch kind of
> thing), but would welcome any help. For those who need admin access,
> let me know what your usernames are.
>
> - Robert
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

Don't forget to update wiki for new git tricks ;)
And what about all that docs located on trac?

-- 
vitja.

From robertwb at math.washington.edu  Sat Nov 20 00:22:20 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 19 Nov 2010 15:22:20 -0800
Subject: [Cython] GitHub for Cython
In-Reply-To: <AANLkTik6T6PY0sysSSK=8wD6WL8rovfmctNFbHHNmy9j@mail.gmail.com>
References: <AANLkTinFLOtcHvdUj-9SnX=7uZELoH9W74zRfy37df=z@mail.gmail.com>
	<AANLkTik6T6PY0sysSSK=8wD6WL8rovfmctNFbHHNmy9j@mail.gmail.com>
Message-ID: <AANLkTikMK1OiZqLUYNF6gu3v2mocSAq168vpL8GOw8rK@mail.gmail.com>

On Fri, Nov 19, 2010 at 3:10 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2010/11/20 Robert Bradshaw <robertwb at math.washington.edu>:
>> It seems that the clear consensus, or at least the one thing that we
>> can all be happy about, is moving to GitHub (crossing our fingers that
>> the issue tracker improves quickly). I've created
>> https://github.com/cython , but haven't started to migrate the rest of
>> the stuff over. I'm thinking we can still stick with the push-to-main
>> option for the core developers (which seems to still be appropriate
>> for our project right now, and also works well with stuff like the
>> build-bot), but can easily fork for experimental work, reviews, and
>> anyone else who wants to contribute.
>>
>> Fortunately, with a DVCS, we don't have to "flip the switch" all at
>> once. I'll be migrating the site off of the VM and the trac/wiki out
>> to GitHub soonish (and this will probably be a flip-the-switch kind of
>> thing), but would welcome any help. For those who need admin access,
>> let me know what your usernames are.
>
> Don't forget to update wiki for new git tricks ;)

One great thing about GitHub is that they'll provide all this goodness
for us :).

> And what about all that docs located on trac?

The track and wiki will be migrated as faithfully as possible--this
was one of the criteria.

- Robert

From fperez.net at gmail.com  Sat Nov 20 03:26:23 2010
From: fperez.net at gmail.com (Fernando Perez)
Date: Fri, 19 Nov 2010 18:26:23 -0800
Subject: [Cython] GitHub for Cython
In-Reply-To: <AANLkTinFLOtcHvdUj-9SnX=7uZELoH9W74zRfy37df=z@mail.gmail.com>
References: <AANLkTinFLOtcHvdUj-9SnX=7uZELoH9W74zRfy37df=z@mail.gmail.com>
Message-ID: <AANLkTinVrx3vqE9Nx7Rnj6bo4kvf+BVjNV3Y6Tqk_HCi@mail.gmail.com>

On Fri, Nov 19, 2010 at 3:03 PM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> It seems that the clear consensus, or at least the one thing that we
> can all be happy about, is moving to GitHub (crossing our fingers that

Great news, glad to hear that!

> the issue tracker improves quickly). I've created
> https://github.com/cython

Minor note: I'd suggest that instead of having

https://github.com/cython/cython

be a fork of

https://github.com/dagss/cython

you seed it with a new repo (which can be a local clone of Dag's), so
that the fork tree starts at the official repo and not one particular
devleoper's.

It's a minor point, but I always find it nicer to look at organization
repos that start there, rather than starting at some random
developer's repo.  It also reduces the chances that people
accidentally fork from the other one, creating a less obvious network
in the long run.

Regards,

f

From ondrej at certik.cz  Sat Nov 20 05:23:22 2010
From: ondrej at certik.cz (Ondrej Certik)
Date: Fri, 19 Nov 2010 20:23:22 -0800
Subject: [Cython] GitHub for Cython
In-Reply-To: <AANLkTinVrx3vqE9Nx7Rnj6bo4kvf+BVjNV3Y6Tqk_HCi@mail.gmail.com>
References: <AANLkTinFLOtcHvdUj-9SnX=7uZELoH9W74zRfy37df=z@mail.gmail.com>
	<AANLkTinVrx3vqE9Nx7Rnj6bo4kvf+BVjNV3Y6Tqk_HCi@mail.gmail.com>
Message-ID: <AANLkTikdThiYUK2GOOjyS6fmXLQg5hDQ8jobkSeSSwZt@mail.gmail.com>

On Fri, Nov 19, 2010 at 6:26 PM, Fernando Perez <fperez.net at gmail.com> wrote:
> On Fri, Nov 19, 2010 at 3:03 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> It seems that the clear consensus, or at least the one thing that we
>> can all be happy about, is moving to GitHub (crossing our fingers that
>
> Great news, glad to hear that!
>
>> the issue tracker improves quickly). I've created
>> https://github.com/cython
>
> Minor note: I'd suggest that instead of having
>
> https://github.com/cython/cython
>
> be a fork of
>
> https://github.com/dagss/cython
>
> you seed it with a new repo (which can be a local clone of Dag's), so
> that the fork tree starts at the official repo and not one particular
> devleoper's.
>
> It's a minor point, but I always find it nicer to look at organization
> repos that start there, rather than starting at some random
> developer's repo. ?It also reduces the chances that people
> accidentally fork from the other one, creating a less obvious network
> in the long run.

Yes, that was a mistake we did with sympy. I consider this a bug of
GitHub, that there is no simple way to fix this, unless the developer
agrees to delete his repo and reclone.

Ondrej

From robertwb at math.washington.edu  Sat Nov 20 07:19:23 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Fri, 19 Nov 2010 22:19:23 -0800
Subject: [Cython] GitHub for Cython
In-Reply-To: <AANLkTinVrx3vqE9Nx7Rnj6bo4kvf+BVjNV3Y6Tqk_HCi@mail.gmail.com>
References: <AANLkTinFLOtcHvdUj-9SnX=7uZELoH9W74zRfy37df=z@mail.gmail.com>
	<AANLkTinVrx3vqE9Nx7Rnj6bo4kvf+BVjNV3Y6Tqk_HCi@mail.gmail.com>
Message-ID: <AANLkTinSF-tjR87jYsARVYOebQ3cT7eJkupn8XvX+rcP@mail.gmail.com>

On Fri, Nov 19, 2010 at 6:26 PM, Fernando Perez <fperez.net at gmail.com> wrote:
> On Fri, Nov 19, 2010 at 3:03 PM, Robert Bradshaw
> <robertwb at math.washington.edu> wrote:
>> It seems that the clear consensus, or at least the one thing that we
>> can all be happy about, is moving to GitHub (crossing our fingers that
>
> Great news, glad to hear that!
>
>> the issue tracker improves quickly). I've created
>> https://github.com/cython
>
> Minor note: I'd suggest that instead of having
>
> https://github.com/cython/cython
>
> be a fork of
>
> https://github.com/dagss/cython
>
> you seed it with a new repo (which can be a local clone of Dag's), so
> that the fork tree starts at the official repo and not one particular
> devleoper's.
>
> It's a minor point, but I always find it nicer to look at organization
> repos that start there, rather than starting at some random
> developer's repo. ?It also reduces the chances that people
> accidentally fork from the other one, creating a less obvious network
> in the long run.

Good point, I'll do that. I agree, much harder to fix in retrospect...

- Robert

From vitja.makarov at gmail.com  Sat Nov 20 10:43:34 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 20 Nov 2010 12:43:34 +0300
Subject: [Cython] [bug] inf loop
In-Reply-To: <AANLkTikMz4KD72MhycysTKY+UBKMPBKOoLqiiW056hdR@mail.gmail.com>
References: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>
	<AANLkTi=bWs0X8peV0fQQeQx4_okvduQue_f5nJD75Nta@mail.gmail.com>
	<4CE53725.5050707@behnel.de>
	<AANLkTi=s_tFgLxGXnb5xQxyjwc8gyquTuUBOGkNgr-d5@mail.gmail.com>
	<AANLkTimAYXqVOBWU_Bdr_ihP9_s5g3iEvZf0TJgmcVDN@mail.gmail.com>
	<4CE58419.7070000@behnel.de>
	<AANLkTi=v8BVSLduQkqmmmOZ1sKp9XSsx5XeMcU8FR7MA@mail.gmail.com>
	<4CE58BAC.6020105@behnel.de>
	<AANLkTik68aQ3SQ9Se=KmNpAU3jkQopxdw5E39L7SSMfm@mail.gmail.com>
	<4CE5FE5B.30108@behnel.de>
	<AANLkTi=9sAz7Oy4yHRKR-AXWcwgEngmWd4ONYKS_gyD7@mail.gmail.com>
	<AANLkTikMz4KD72MhycysTKY+UBKMPBKOoLqiiW056hdR@mail.gmail.com>
Message-ID: <AANLkTi=669wu-DaLBNV8T4ryn5cbfvmr7vczB7HH3V6R@mail.gmail.com>

19 ?????? 2010??. 18:36 ???????????? Vitja Makarov
<vitja.makarov at gmail.com> ???????:
> 19 ?????? 2010??. 10:27 ???????????? Vitja Makarov
> <vitja.makarov at gmail.com> ???????:
>> 2010/11/19 Stefan Behnel <stefan_ml at behnel.de>:
>>> Vitja Makarov, 18.11.2010 22:54:
>>>> 2010/11/18 Stefan Behnel:
>>>>> Vitja Makarov, 18.11.2010 21:17:
>>>>>> 2010/11/18 Stefan Behnel:
>>>>>>> The C compiler problems looked like closure issues, so they are more likely
>>>>>>> to be related.
>>>>>>
>>>>>> Is compile problem logged somewhere?
>>>>>
>>>>> You can find most of them here:
>>>>>
>>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py26-c/704/parsed_console/
>>>>>
>>>>> Look out for "has no member named".
>>>>
>>>> Disabling this tests seems to make regression tests finish up.
>>>>
>>>> pyregr.test_socket
>>>> pyregr.test_threading
>>>
>>> Ok, I disabled them.
>>>
>>> Stefan
>>> _______________________________________________
>>> Cython-dev mailing list
>>> Cython-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/cython-dev
>>>
>>
>>
>> New patch fixes #537, where are all that gcc errors came from.
>> It seems that patch enable many new tests that break test runner
>> itself, in threads.
>>
>> With latest patch runtests couldn't exit because of thread lock
>> (join?), here is gdb traceback:
>> I don't know how to figure out which test causes this. Should we
>> disable some more thread-related tests.
>> Or as workaround use _exit() in testrunner
>>
>> (gdb) bt
>> #0 ?sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
>> #1 ?0x00000000004dcac8 in PyThread_acquire_lock (lock=0x6d43c60,
>> waitflag=128) at Python/thread_pthread.h:309
>> #2 ?0x00000000004e00d2 in lock_PyThread_acquire_lock (self=0x1d113790,
>> args=<value optimised out>) at ./Modules/threadmodule.c:51
>> #3 ?0x00000000004a860a in call_function (f=0x3136af0, throwflag=<value
>> optimised out>) at Python/ceval.c:4012
>> #4 ?PyEval_EvalFrameEx (f=0x3136af0, throwflag=<value optimised out>)
>> at Python/ceval.c:2665
>> #5 ?0x00000000004a9e81 in PyEval_EvalCodeEx (co=0x3f7b430,
>> globals=<value optimised out>, locals=<value optimised out>, args=0x2,
>> argcount=0, kws=<value optimised out>,
>> ? ?kwcount=0, defs=0x2c53668, defcount=1, closure=0x0) at Python/ceval.c:3252
>> #6 ?0x00000000004a8134 in fast_function (f=0x3c34650, throwflag=<value
>> optimised out>) at Python/ceval.c:4108
>> #7 ?call_function (f=0x3c34650, throwflag=<value optimised out>) at
>> Python/ceval.c:4033
>> #8 ?PyEval_EvalFrameEx (f=0x3c34650, throwflag=<value optimised out>)
>> at Python/ceval.c:2665
>> #9 ?0x00000000004a9e81 in PyEval_EvalCodeEx (co=0x3f60530,
>> globals=<value optimised out>, locals=<value optimised out>, args=0x2,
>> argcount=0, kws=<value optimised out>,
>> ? ?kwcount=0, defs=0x3d239e8, defcount=1, closure=0x0) at Python/ceval.c:3252
>> #10 0x00000000004a8134 in fast_function (f=0x18172230,
>> throwflag=<value optimised out>) at Python/ceval.c:4108
>> #11 call_function (f=0x18172230, throwflag=<value optimised out>) at
>> Python/ceval.c:4033
>> #12 PyEval_EvalFrameEx (f=0x18172230, throwflag=<value optimised out>)
>> at Python/ceval.c:2665
>> #13 0x00000000004a9e81 in PyEval_EvalCodeEx (co=0x3f60eb0,
>> globals=<value optimised out>, locals=<value optimised out>,
>> args=0x23a6c68, argcount=0, kws=<value optimised out>,
>> ? ?kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3252
>> #14 0x000000000050aa4e in function_call (func=0x3d12e60,
>> arg=0x23a6c50, kw=0x0) at Objects/funcobject.c:526
>> #15 0x00000000004195c7 in PyObject_Call (func=0x3d12e60, arg=0x80,
>> kw=0x0) at Objects/abstract.c:2522
>> #16 0x000000000042222f in instancemethod_call (func=0x3d12e60,
>> arg=0x23a6c50, kw=0x0) at Objects/classobject.c:2578
>> #17 0x000000000041c133 in PyObject_Call (callable=0x2e36550,
>> args=0x7f599bbc1050) at Objects/abstract.c:2522
>> #18 call_function_tail (callable=0x2e36550, args=0x7f599bbc1050) at
>> Objects/abstract.c:2554
>> #19 0x000000000041f340 in PyObject_CallMethod (o=<value optimised
>> out>, name=0x53f786 "_shutdown", format=0x5285b0 "") at
>> Objects/abstract.c:2631
>> #20 0x00000000004ca089 in wait_for_thread_shutdown () at Python/pythonrun.c:1696
>> #21 Py_Finalize () at Python/pythonrun.c:401
>> #22 0x00000000004c9810 in Py_Exit () at Python/pythonrun.c:1753
>> #23 handle_system_exit () at Python/pythonrun.c:1127
>> #24 0x00000000004c9e15 in PyErr_PrintEx (set_sys_last_vars=<value
>> optimised out>) at Python/pythonrun.c:1137
>> #25 0x00000000004cac40 in PyRun_SimpleFileExFlags (fp=<value optimised
>> out>, filename=0x7fff5f1d8f91 "runtests.py", closeit=1,
>> flags=0x7fff5f1d7270) at Python/pythonrun.c:940
>> #26 0x0000000000414b7a in Py_Main (argc=-1682423728, argv=<value
>> optimised out>) at Modules/main.c:599
>> #27 0x00007f599ae04d8e in __libc_start_main (main=<value optimised
>> out>, argc=<value optimised out>, ubp_av=<value optimised out>,
>> init=<value optimised out>,
>> ? ?fini=<value optimised out>, rtld_fini=<value optimised out>,
>> stack_end=0x7fff5f1d7388) at libc-start.c:226
>> #28 0x0000000000413cd9 in _start ()
>> (gdb) info threads
>> ?3 Thread 0x7f59857cb710 (LWP 17342) ?sem_wait () at
>> ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
>> ?2 Thread 0x7f597cf7e710 (LWP 17397) ?0x00007f599aec52e3 in select ()
>> at ../sysdeps/unix/syscall-template.S:82
>> * 1 Thread 0x7f599bc02700 (LWP 8116) ?sem_wait () at
>> ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
>> (gdb) thread 2
>> [Switching to thread 2 (Thread 0x7f597cf7e710 (LWP 17397))]#0
>> 0x00007f599aec52e3 in select () at
>> ../sysdeps/unix/syscall-template.S:82
>> 82 ? ? ?../sysdeps/unix/syscall-template.S: ??? ?????? ????? ??? ????????.
>> ? ? ? ?in ../sysdeps/unix/syscall-template.S
>> (gdb) bt
>> #0 ?0x00007f599aec52e3 in select () at ../sysdeps/unix/syscall-template.S:82
>> #1 ?0x00007f596b8e2f18 in ?? () from /usr/lib/libtcl8.5.so.0
>> #2 ?0x00007f599b7fa971 in start_thread (arg=<value optimised out>) at
>> pthread_create.c:304
>> #3 ?0x00007f599aecc94d in clone () at
>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
>> #4 ?0x0000000000000000 in ?? ()
>> (gdb) thread 3
>> [Switching to thread 3 (Thread 0x7f59857cb710 (LWP 17342))]#0
>> sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
>> 86 ? ? ?../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S: ??? ?????? ?????
>> ??? ????????.
>> ? ? ? ?in ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S
>> (gdb) bt
>> #0 ?sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
>> #1 ?0x00000000004dcac8 in PyThread_acquire_lock (lock=0x14816d80,
>> waitflag=128) at Python/thread_pthread.h:309
>> #2 ?0x00000000004e00d2 in lock_PyThread_acquire_lock (self=0x26964370,
>> args=<value optimised out>) at ./Modules/threadmodule.c:51
>> #3 ?0x00000000004a860a in call_function (f=0x24b11010,
>> throwflag=<value optimised out>) at Python/ceval.c:4012
>> #4 ?PyEval_EvalFrameEx (f=0x24b11010, throwflag=<value optimised out>)
>> at Python/ceval.c:2665
>> #5 ?0x00000000004a9e81 in PyEval_EvalCodeEx (co=0x3f7b430,
>> globals=<value optimised out>, locals=<value optimised out>,
>> args=0x250b8a58, argcount=0, kws=<value optimised out>,
>> ? ?kwcount=0, defs=0x2c53668, defcount=1, closure=0x0) at Python/ceval.c:3252
>> #6 ?0x00000000004a8134 in fast_function (f=0x250b88d0,
>> throwflag=<value optimised out>) at Python/ceval.c:4108
>> #7 ?call_function (f=0x250b88d0, throwflag=<value optimised out>) at
>> Python/ceval.c:4033
>> #8 ?PyEval_EvalFrameEx (f=0x250b88d0, throwflag=<value optimised out>)
>> at Python/ceval.c:2665
>> #9 ?0x00000000004a9e81 in PyEval_EvalCodeEx (co=0x3f7bdb0,
>> globals=<value optimised out>, locals=<value optimised out>, args=0x2,
>> argcount=0, kws=<value optimised out>,
>> ? ?kwcount=0, defs=0x3d23ae8, defcount=1, closure=0x0) at Python/ceval.c:3252
>> #10 0x000000000050aa4e in function_call (func=0x3d120c8,
>> arg=0x26b704d0, kw=0x0) at Objects/funcobject.c:526
>> #11 0x00000000004195c7 in PyObject_Call (func=0x3d120c8, arg=0x80,
>> kw=0x0) at Objects/abstract.c:2522
>> #12 0x000000000042222f in instancemethod_call (func=0x3d120c8,
>> arg=0x26b704d0, kw=0x0) at Objects/classobject.c:2578
>> #13 0x00000000004195c7 in PyObject_Call (func=0x26b72370, arg=0x80,
>> kw=0x0) at Objects/abstract.c:2522
>> #14 0x00007f596cb85f36 in ?? ()
>> #15 0x0000000000000000 in ?? ()
>>
>
> Still can't find which one test have thread left.
>
> Here is workaround:
>
> import threading
> import time
> import os
>
> class XXX(threading.Thread):
> ? ?def run(self):
> ? ? ? ?print ('Bad thread')
> ? ? ? ?while True:
> ? ? ? ? ? ?time.sleep(1)
> XXX().start()
>
> time.sleep(.1)
>
> if len(threading._active) > 1:
> ? ?import warnings
> ? ?print("Warning more then one threads left, forcing _exit()")
> ? ?os._exit(0)
>
>
> I'm not sure it is a good idea to add this into runtests.py
>

Btw I think we need a way to find blocking thread.

wait_for_thread_shutdown() calls threading._shutdown and we can hook
it and test is there NonDaemon threads left and try to join
them with timeout. If it fails call _exit and print error message.

One more problem: how to make it portable between python versions.
--
vitja.

From vitja.makarov at gmail.com  Sat Nov 20 11:05:40 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 20 Nov 2010 13:05:40 +0300
Subject: [Cython] [bug] inf loop
In-Reply-To: <AANLkTi=669wu-DaLBNV8T4ryn5cbfvmr7vczB7HH3V6R@mail.gmail.com>
References: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>
	<AANLkTi=bWs0X8peV0fQQeQx4_okvduQue_f5nJD75Nta@mail.gmail.com>
	<4CE53725.5050707@behnel.de>
	<AANLkTi=s_tFgLxGXnb5xQxyjwc8gyquTuUBOGkNgr-d5@mail.gmail.com>
	<AANLkTimAYXqVOBWU_Bdr_ihP9_s5g3iEvZf0TJgmcVDN@mail.gmail.com>
	<4CE58419.7070000@behnel.de>
	<AANLkTi=v8BVSLduQkqmmmOZ1sKp9XSsx5XeMcU8FR7MA@mail.gmail.com>
	<4CE58BAC.6020105@behnel.de>
	<AANLkTik68aQ3SQ9Se=KmNpAU3jkQopxdw5E39L7SSMfm@mail.gmail.com>
	<4CE5FE5B.30108@behnel.de>
	<AANLkTi=9sAz7Oy4yHRKR-AXWcwgEngmWd4ONYKS_gyD7@mail.gmail.com>
	<AANLkTikMz4KD72MhycysTKY+UBKMPBKOoLqiiW056hdR@mail.gmail.com>
	<AANLkTi=669wu-DaLBNV8T4ryn5cbfvmr7vczB7HH3V6R@mail.gmail.com>
Message-ID: <AANLkTik4_eiTqB99-vHQzQow8b9wWYcFk5xYPVDKL+15@mail.gmail.com>

20 ?????? 2010??. 12:43 ???????????? Vitja Makarov
<vitja.makarov at gmail.com> ???????:
> 19 ?????? 2010??. 18:36 ???????????? Vitja Makarov
> <vitja.makarov at gmail.com> ???????:
>> 19 ?????? 2010??. 10:27 ???????????? Vitja Makarov
>> <vitja.makarov at gmail.com> ???????:
>>> 2010/11/19 Stefan Behnel <stefan_ml at behnel.de>:
>>>> Vitja Makarov, 18.11.2010 22:54:
>>>>> 2010/11/18 Stefan Behnel:
>>>>>> Vitja Makarov, 18.11.2010 21:17:
>>>>>>> 2010/11/18 Stefan Behnel:
>>>>>>>> The C compiler problems looked like closure issues, so they are more likely
>>>>>>>> to be related.
>>>>>>>
>>>>>>> Is compile problem logged somewhere?
>>>>>>
>>>>>> You can find most of them here:
>>>>>>
>>>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py26-c/704/parsed_console/
>>>>>>
>>>>>> Look out for "has no member named".
>>>>>
>>>>> Disabling this tests seems to make regression tests finish up.
>>>>>
>>>>> pyregr.test_socket
>>>>> pyregr.test_threading
>>>>
>>>> Ok, I disabled them.
>>>>
>>>> Stefan
>>>> _______________________________________________
>>>> Cython-dev mailing list
>>>> Cython-dev at codespeak.net
>>>> http://codespeak.net/mailman/listinfo/cython-dev
>>>>
>>>
>>>
>>> New patch fixes #537, where are all that gcc errors came from.
>>> It seems that patch enable many new tests that break test runner
>>> itself, in threads.
>>>
>>> With latest patch runtests couldn't exit because of thread lock
>>> (join?), here is gdb traceback:
>>> I don't know how to figure out which test causes this. Should we
>>> disable some more thread-related tests.
>>> Or as workaround use _exit() in testrunner
>>>
>>> (gdb) bt
>>> #0 ?sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
>>> #1 ?0x00000000004dcac8 in PyThread_acquire_lock (lock=0x6d43c60,
>>> waitflag=128) at Python/thread_pthread.h:309
>>> #2 ?0x00000000004e00d2 in lock_PyThread_acquire_lock (self=0x1d113790,
>>> args=<value optimised out>) at ./Modules/threadmodule.c:51
>>> #3 ?0x00000000004a860a in call_function (f=0x3136af0, throwflag=<value
>>> optimised out>) at Python/ceval.c:4012
>>> #4 ?PyEval_EvalFrameEx (f=0x3136af0, throwflag=<value optimised out>)
>>> at Python/ceval.c:2665
>>> #5 ?0x00000000004a9e81 in PyEval_EvalCodeEx (co=0x3f7b430,
>>> globals=<value optimised out>, locals=<value optimised out>, args=0x2,
>>> argcount=0, kws=<value optimised out>,
>>> ? ?kwcount=0, defs=0x2c53668, defcount=1, closure=0x0) at Python/ceval.c:3252
>>> #6 ?0x00000000004a8134 in fast_function (f=0x3c34650, throwflag=<value
>>> optimised out>) at Python/ceval.c:4108
>>> #7 ?call_function (f=0x3c34650, throwflag=<value optimised out>) at
>>> Python/ceval.c:4033
>>> #8 ?PyEval_EvalFrameEx (f=0x3c34650, throwflag=<value optimised out>)
>>> at Python/ceval.c:2665
>>> #9 ?0x00000000004a9e81 in PyEval_EvalCodeEx (co=0x3f60530,
>>> globals=<value optimised out>, locals=<value optimised out>, args=0x2,
>>> argcount=0, kws=<value optimised out>,
>>> ? ?kwcount=0, defs=0x3d239e8, defcount=1, closure=0x0) at Python/ceval.c:3252
>>> #10 0x00000000004a8134 in fast_function (f=0x18172230,
>>> throwflag=<value optimised out>) at Python/ceval.c:4108
>>> #11 call_function (f=0x18172230, throwflag=<value optimised out>) at
>>> Python/ceval.c:4033
>>> #12 PyEval_EvalFrameEx (f=0x18172230, throwflag=<value optimised out>)
>>> at Python/ceval.c:2665
>>> #13 0x00000000004a9e81 in PyEval_EvalCodeEx (co=0x3f60eb0,
>>> globals=<value optimised out>, locals=<value optimised out>,
>>> args=0x23a6c68, argcount=0, kws=<value optimised out>,
>>> ? ?kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3252
>>> #14 0x000000000050aa4e in function_call (func=0x3d12e60,
>>> arg=0x23a6c50, kw=0x0) at Objects/funcobject.c:526
>>> #15 0x00000000004195c7 in PyObject_Call (func=0x3d12e60, arg=0x80,
>>> kw=0x0) at Objects/abstract.c:2522
>>> #16 0x000000000042222f in instancemethod_call (func=0x3d12e60,
>>> arg=0x23a6c50, kw=0x0) at Objects/classobject.c:2578
>>> #17 0x000000000041c133 in PyObject_Call (callable=0x2e36550,
>>> args=0x7f599bbc1050) at Objects/abstract.c:2522
>>> #18 call_function_tail (callable=0x2e36550, args=0x7f599bbc1050) at
>>> Objects/abstract.c:2554
>>> #19 0x000000000041f340 in PyObject_CallMethod (o=<value optimised
>>> out>, name=0x53f786 "_shutdown", format=0x5285b0 "") at
>>> Objects/abstract.c:2631
>>> #20 0x00000000004ca089 in wait_for_thread_shutdown () at Python/pythonrun.c:1696
>>> #21 Py_Finalize () at Python/pythonrun.c:401
>>> #22 0x00000000004c9810 in Py_Exit () at Python/pythonrun.c:1753
>>> #23 handle_system_exit () at Python/pythonrun.c:1127
>>> #24 0x00000000004c9e15 in PyErr_PrintEx (set_sys_last_vars=<value
>>> optimised out>) at Python/pythonrun.c:1137
>>> #25 0x00000000004cac40 in PyRun_SimpleFileExFlags (fp=<value optimised
>>> out>, filename=0x7fff5f1d8f91 "runtests.py", closeit=1,
>>> flags=0x7fff5f1d7270) at Python/pythonrun.c:940
>>> #26 0x0000000000414b7a in Py_Main (argc=-1682423728, argv=<value
>>> optimised out>) at Modules/main.c:599
>>> #27 0x00007f599ae04d8e in __libc_start_main (main=<value optimised
>>> out>, argc=<value optimised out>, ubp_av=<value optimised out>,
>>> init=<value optimised out>,
>>> ? ?fini=<value optimised out>, rtld_fini=<value optimised out>,
>>> stack_end=0x7fff5f1d7388) at libc-start.c:226
>>> #28 0x0000000000413cd9 in _start ()
>>> (gdb) info threads
>>> ?3 Thread 0x7f59857cb710 (LWP 17342) ?sem_wait () at
>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
>>> ?2 Thread 0x7f597cf7e710 (LWP 17397) ?0x00007f599aec52e3 in select ()
>>> at ../sysdeps/unix/syscall-template.S:82
>>> * 1 Thread 0x7f599bc02700 (LWP 8116) ?sem_wait () at
>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
>>> (gdb) thread 2
>>> [Switching to thread 2 (Thread 0x7f597cf7e710 (LWP 17397))]#0
>>> 0x00007f599aec52e3 in select () at
>>> ../sysdeps/unix/syscall-template.S:82
>>> 82 ? ? ?../sysdeps/unix/syscall-template.S: ??? ?????? ????? ??? ????????.
>>> ? ? ? ?in ../sysdeps/unix/syscall-template.S
>>> (gdb) bt
>>> #0 ?0x00007f599aec52e3 in select () at ../sysdeps/unix/syscall-template.S:82
>>> #1 ?0x00007f596b8e2f18 in ?? () from /usr/lib/libtcl8.5.so.0
>>> #2 ?0x00007f599b7fa971 in start_thread (arg=<value optimised out>) at
>>> pthread_create.c:304
>>> #3 ?0x00007f599aecc94d in clone () at
>>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
>>> #4 ?0x0000000000000000 in ?? ()
>>> (gdb) thread 3
>>> [Switching to thread 3 (Thread 0x7f59857cb710 (LWP 17342))]#0
>>> sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
>>> 86 ? ? ?../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S: ??? ?????? ?????
>>> ??? ????????.
>>> ? ? ? ?in ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S
>>> (gdb) bt
>>> #0 ?sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
>>> #1 ?0x00000000004dcac8 in PyThread_acquire_lock (lock=0x14816d80,
>>> waitflag=128) at Python/thread_pthread.h:309
>>> #2 ?0x00000000004e00d2 in lock_PyThread_acquire_lock (self=0x26964370,
>>> args=<value optimised out>) at ./Modules/threadmodule.c:51
>>> #3 ?0x00000000004a860a in call_function (f=0x24b11010,
>>> throwflag=<value optimised out>) at Python/ceval.c:4012
>>> #4 ?PyEval_EvalFrameEx (f=0x24b11010, throwflag=<value optimised out>)
>>> at Python/ceval.c:2665
>>> #5 ?0x00000000004a9e81 in PyEval_EvalCodeEx (co=0x3f7b430,
>>> globals=<value optimised out>, locals=<value optimised out>,
>>> args=0x250b8a58, argcount=0, kws=<value optimised out>,
>>> ? ?kwcount=0, defs=0x2c53668, defcount=1, closure=0x0) at Python/ceval.c:3252
>>> #6 ?0x00000000004a8134 in fast_function (f=0x250b88d0,
>>> throwflag=<value optimised out>) at Python/ceval.c:4108
>>> #7 ?call_function (f=0x250b88d0, throwflag=<value optimised out>) at
>>> Python/ceval.c:4033
>>> #8 ?PyEval_EvalFrameEx (f=0x250b88d0, throwflag=<value optimised out>)
>>> at Python/ceval.c:2665
>>> #9 ?0x00000000004a9e81 in PyEval_EvalCodeEx (co=0x3f7bdb0,
>>> globals=<value optimised out>, locals=<value optimised out>, args=0x2,
>>> argcount=0, kws=<value optimised out>,
>>> ? ?kwcount=0, defs=0x3d23ae8, defcount=1, closure=0x0) at Python/ceval.c:3252
>>> #10 0x000000000050aa4e in function_call (func=0x3d120c8,
>>> arg=0x26b704d0, kw=0x0) at Objects/funcobject.c:526
>>> #11 0x00000000004195c7 in PyObject_Call (func=0x3d120c8, arg=0x80,
>>> kw=0x0) at Objects/abstract.c:2522
>>> #12 0x000000000042222f in instancemethod_call (func=0x3d120c8,
>>> arg=0x26b704d0, kw=0x0) at Objects/classobject.c:2578
>>> #13 0x00000000004195c7 in PyObject_Call (func=0x26b72370, arg=0x80,
>>> kw=0x0) at Objects/abstract.c:2522
>>> #14 0x00007f596cb85f36 in ?? ()
>>> #15 0x0000000000000000 in ?? ()
>>>
>>
>> Still can't find which one test have thread left.
>>
>> Here is workaround:
>>
>> import threading
>> import time
>> import os
>>
>> class XXX(threading.Thread):
>> ? ?def run(self):
>> ? ? ? ?print ('Bad thread')
>> ? ? ? ?while True:
>> ? ? ? ? ? ?time.sleep(1)
>> XXX().start()
>>
>> time.sleep(.1)
>>
>> if len(threading._active) > 1:
>> ? ?import warnings
>> ? ?print("Warning more then one threads left, forcing _exit()")
>> ? ?os._exit(0)
>>
>>
>> I'm not sure it is a good idea to add this into runtests.py
>>
>
> Btw I think we need a way to find blocking thread.
>
> wait_for_thread_shutdown() calls threading._shutdown and we can hook
> it and test is there NonDaemon threads left and try to join
> them with timeout. If it fails call _exit and print error message.
>
> One more problem: how to make it portable between python versions.
> --
> vitja.
>

2.5, 2.6, 2.7, 3.x use threading._shutdown

2.3,2.4 use sys.exitfunc

-- 
vitja.

From vitja.makarov at gmail.com  Sun Nov 21 00:02:15 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sun, 21 Nov 2010 02:02:15 +0300
Subject: [Cython] [bug] inf loop
In-Reply-To: <AANLkTik4_eiTqB99-vHQzQow8b9wWYcFk5xYPVDKL+15@mail.gmail.com>
References: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>
	<AANLkTi=bWs0X8peV0fQQeQx4_okvduQue_f5nJD75Nta@mail.gmail.com>
	<4CE53725.5050707@behnel.de>
	<AANLkTi=s_tFgLxGXnb5xQxyjwc8gyquTuUBOGkNgr-d5@mail.gmail.com>
	<AANLkTimAYXqVOBWU_Bdr_ihP9_s5g3iEvZf0TJgmcVDN@mail.gmail.com>
	<4CE58419.7070000@behnel.de>
	<AANLkTi=v8BVSLduQkqmmmOZ1sKp9XSsx5XeMcU8FR7MA@mail.gmail.com>
	<4CE58BAC.6020105@behnel.de>
	<AANLkTik68aQ3SQ9Se=KmNpAU3jkQopxdw5E39L7SSMfm@mail.gmail.com>
	<4CE5FE5B.30108@behnel.de>
	<AANLkTi=9sAz7Oy4yHRKR-AXWcwgEngmWd4ONYKS_gyD7@mail.gmail.com>
	<AANLkTikMz4KD72MhycysTKY+UBKMPBKOoLqiiW056hdR@mail.gmail.com>
	<AANLkTi=669wu-DaLBNV8T4ryn5cbfvmr7vczB7HH3V6R@mail.gmail.com>
	<AANLkTik4_eiTqB99-vHQzQow8b9wWYcFk5xYPVDKL+15@mail.gmail.com>
Message-ID: <AANLkTimi1pkHt7FP-LssbvDUVmTWM0mLCjNu9iB19+hC@mail.gmail.com>

20 ?????? 2010??. 13:05 ???????????? Vitja Makarov
<vitja.makarov at gmail.com> ???????:
> 20 ?????? 2010??. 12:43 ???????????? Vitja Makarov
> <vitja.makarov at gmail.com> ???????:
>> 19 ?????? 2010??. 18:36 ???????????? Vitja Makarov
>> <vitja.makarov at gmail.com> ???????:
>>> 19 ?????? 2010??. 10:27 ???????????? Vitja Makarov
>>> <vitja.makarov at gmail.com> ???????:
>>>> 2010/11/19 Stefan Behnel <stefan_ml at behnel.de>:
>>>>> Vitja Makarov, 18.11.2010 22:54:
>>>>>> 2010/11/18 Stefan Behnel:
>>>>>>> Vitja Makarov, 18.11.2010 21:17:
>>>>>>>> 2010/11/18 Stefan Behnel:
>>>>>>>>> The C compiler problems looked like closure issues, so they are more likely
>>>>>>>>> to be related.
>>>>>>>>
>>>>>>>> Is compile problem logged somewhere?
>>>>>>>
>>>>>>> You can find most of them here:
>>>>>>>
>>>>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py26-c/704/parsed_console/
>>>>>>>
>>>>>>> Look out for "has no member named".
>>>>>>
>>>>>> Disabling this tests seems to make regression tests finish up.
>>>>>>
>>>>>> pyregr.test_socket
>>>>>> pyregr.test_threading
>>>>>
>>>>> Ok, I disabled them.
>>>>>
>>>>> Stefan
>>>>> _______________________________________________
>>>>> Cython-dev mailing list
>>>>> Cython-dev at codespeak.net
>>>>> http://codespeak.net/mailman/listinfo/cython-dev
>>>>>
>>>>
>>>>
>>>> New patch fixes #537, where are all that gcc errors came from.
>>>> It seems that patch enable many new tests that break test runner
>>>> itself, in threads.
>>>>
>>>> With latest patch runtests couldn't exit because of thread lock
>>>> (join?), here is gdb traceback:
>>>> I don't know how to figure out which test causes this. Should we
>>>> disable some more thread-related tests.
>>>> Or as workaround use _exit() in testrunner
>>>>
>>>> (gdb) bt
>>>> #0 ?sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
>>>> #1 ?0x00000000004dcac8 in PyThread_acquire_lock (lock=0x6d43c60,
>>>> waitflag=128) at Python/thread_pthread.h:309
>>>> #2 ?0x00000000004e00d2 in lock_PyThread_acquire_lock (self=0x1d113790,
>>>> args=<value optimised out>) at ./Modules/threadmodule.c:51
>>>> #3 ?0x00000000004a860a in call_function (f=0x3136af0, throwflag=<value
>>>> optimised out>) at Python/ceval.c:4012
>>>> #4 ?PyEval_EvalFrameEx (f=0x3136af0, throwflag=<value optimised out>)
>>>> at Python/ceval.c:2665
>>>> #5 ?0x00000000004a9e81 in PyEval_EvalCodeEx (co=0x3f7b430,
>>>> globals=<value optimised out>, locals=<value optimised out>, args=0x2,
>>>> argcount=0, kws=<value optimised out>,
>>>> ? ?kwcount=0, defs=0x2c53668, defcount=1, closure=0x0) at Python/ceval.c:3252
>>>> #6 ?0x00000000004a8134 in fast_function (f=0x3c34650, throwflag=<value
>>>> optimised out>) at Python/ceval.c:4108
>>>> #7 ?call_function (f=0x3c34650, throwflag=<value optimised out>) at
>>>> Python/ceval.c:4033
>>>> #8 ?PyEval_EvalFrameEx (f=0x3c34650, throwflag=<value optimised out>)
>>>> at Python/ceval.c:2665
>>>> #9 ?0x00000000004a9e81 in PyEval_EvalCodeEx (co=0x3f60530,
>>>> globals=<value optimised out>, locals=<value optimised out>, args=0x2,
>>>> argcount=0, kws=<value optimised out>,
>>>> ? ?kwcount=0, defs=0x3d239e8, defcount=1, closure=0x0) at Python/ceval.c:3252
>>>> #10 0x00000000004a8134 in fast_function (f=0x18172230,
>>>> throwflag=<value optimised out>) at Python/ceval.c:4108
>>>> #11 call_function (f=0x18172230, throwflag=<value optimised out>) at
>>>> Python/ceval.c:4033
>>>> #12 PyEval_EvalFrameEx (f=0x18172230, throwflag=<value optimised out>)
>>>> at Python/ceval.c:2665
>>>> #13 0x00000000004a9e81 in PyEval_EvalCodeEx (co=0x3f60eb0,
>>>> globals=<value optimised out>, locals=<value optimised out>,
>>>> args=0x23a6c68, argcount=0, kws=<value optimised out>,
>>>> ? ?kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3252
>>>> #14 0x000000000050aa4e in function_call (func=0x3d12e60,
>>>> arg=0x23a6c50, kw=0x0) at Objects/funcobject.c:526
>>>> #15 0x00000000004195c7 in PyObject_Call (func=0x3d12e60, arg=0x80,
>>>> kw=0x0) at Objects/abstract.c:2522
>>>> #16 0x000000000042222f in instancemethod_call (func=0x3d12e60,
>>>> arg=0x23a6c50, kw=0x0) at Objects/classobject.c:2578
>>>> #17 0x000000000041c133 in PyObject_Call (callable=0x2e36550,
>>>> args=0x7f599bbc1050) at Objects/abstract.c:2522
>>>> #18 call_function_tail (callable=0x2e36550, args=0x7f599bbc1050) at
>>>> Objects/abstract.c:2554
>>>> #19 0x000000000041f340 in PyObject_CallMethod (o=<value optimised
>>>> out>, name=0x53f786 "_shutdown", format=0x5285b0 "") at
>>>> Objects/abstract.c:2631
>>>> #20 0x00000000004ca089 in wait_for_thread_shutdown () at Python/pythonrun.c:1696
>>>> #21 Py_Finalize () at Python/pythonrun.c:401
>>>> #22 0x00000000004c9810 in Py_Exit () at Python/pythonrun.c:1753
>>>> #23 handle_system_exit () at Python/pythonrun.c:1127
>>>> #24 0x00000000004c9e15 in PyErr_PrintEx (set_sys_last_vars=<value
>>>> optimised out>) at Python/pythonrun.c:1137
>>>> #25 0x00000000004cac40 in PyRun_SimpleFileExFlags (fp=<value optimised
>>>> out>, filename=0x7fff5f1d8f91 "runtests.py", closeit=1,
>>>> flags=0x7fff5f1d7270) at Python/pythonrun.c:940
>>>> #26 0x0000000000414b7a in Py_Main (argc=-1682423728, argv=<value
>>>> optimised out>) at Modules/main.c:599
>>>> #27 0x00007f599ae04d8e in __libc_start_main (main=<value optimised
>>>> out>, argc=<value optimised out>, ubp_av=<value optimised out>,
>>>> init=<value optimised out>,
>>>> ? ?fini=<value optimised out>, rtld_fini=<value optimised out>,
>>>> stack_end=0x7fff5f1d7388) at libc-start.c:226
>>>> #28 0x0000000000413cd9 in _start ()
>>>> (gdb) info threads
>>>> ?3 Thread 0x7f59857cb710 (LWP 17342) ?sem_wait () at
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
>>>> ?2 Thread 0x7f597cf7e710 (LWP 17397) ?0x00007f599aec52e3 in select ()
>>>> at ../sysdeps/unix/syscall-template.S:82
>>>> * 1 Thread 0x7f599bc02700 (LWP 8116) ?sem_wait () at
>>>> ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
>>>> (gdb) thread 2
>>>> [Switching to thread 2 (Thread 0x7f597cf7e710 (LWP 17397))]#0
>>>> 0x00007f599aec52e3 in select () at
>>>> ../sysdeps/unix/syscall-template.S:82
>>>> 82 ? ? ?../sysdeps/unix/syscall-template.S: ??? ?????? ????? ??? ????????.
>>>> ? ? ? ?in ../sysdeps/unix/syscall-template.S
>>>> (gdb) bt
>>>> #0 ?0x00007f599aec52e3 in select () at ../sysdeps/unix/syscall-template.S:82
>>>> #1 ?0x00007f596b8e2f18 in ?? () from /usr/lib/libtcl8.5.so.0
>>>> #2 ?0x00007f599b7fa971 in start_thread (arg=<value optimised out>) at
>>>> pthread_create.c:304
>>>> #3 ?0x00007f599aecc94d in clone () at
>>>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
>>>> #4 ?0x0000000000000000 in ?? ()
>>>> (gdb) thread 3
>>>> [Switching to thread 3 (Thread 0x7f59857cb710 (LWP 17342))]#0
>>>> sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
>>>> 86 ? ? ?../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S: ??? ?????? ?????
>>>> ??? ????????.
>>>> ? ? ? ?in ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S
>>>> (gdb) bt
>>>> #0 ?sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
>>>> #1 ?0x00000000004dcac8 in PyThread_acquire_lock (lock=0x14816d80,
>>>> waitflag=128) at Python/thread_pthread.h:309
>>>> #2 ?0x00000000004e00d2 in lock_PyThread_acquire_lock (self=0x26964370,
>>>> args=<value optimised out>) at ./Modules/threadmodule.c:51
>>>> #3 ?0x00000000004a860a in call_function (f=0x24b11010,
>>>> throwflag=<value optimised out>) at Python/ceval.c:4012
>>>> #4 ?PyEval_EvalFrameEx (f=0x24b11010, throwflag=<value optimised out>)
>>>> at Python/ceval.c:2665
>>>> #5 ?0x00000000004a9e81 in PyEval_EvalCodeEx (co=0x3f7b430,
>>>> globals=<value optimised out>, locals=<value optimised out>,
>>>> args=0x250b8a58, argcount=0, kws=<value optimised out>,
>>>> ? ?kwcount=0, defs=0x2c53668, defcount=1, closure=0x0) at Python/ceval.c:3252
>>>> #6 ?0x00000000004a8134 in fast_function (f=0x250b88d0,
>>>> throwflag=<value optimised out>) at Python/ceval.c:4108
>>>> #7 ?call_function (f=0x250b88d0, throwflag=<value optimised out>) at
>>>> Python/ceval.c:4033
>>>> #8 ?PyEval_EvalFrameEx (f=0x250b88d0, throwflag=<value optimised out>)
>>>> at Python/ceval.c:2665
>>>> #9 ?0x00000000004a9e81 in PyEval_EvalCodeEx (co=0x3f7bdb0,
>>>> globals=<value optimised out>, locals=<value optimised out>, args=0x2,
>>>> argcount=0, kws=<value optimised out>,
>>>> ? ?kwcount=0, defs=0x3d23ae8, defcount=1, closure=0x0) at Python/ceval.c:3252
>>>> #10 0x000000000050aa4e in function_call (func=0x3d120c8,
>>>> arg=0x26b704d0, kw=0x0) at Objects/funcobject.c:526
>>>> #11 0x00000000004195c7 in PyObject_Call (func=0x3d120c8, arg=0x80,
>>>> kw=0x0) at Objects/abstract.c:2522
>>>> #12 0x000000000042222f in instancemethod_call (func=0x3d120c8,
>>>> arg=0x26b704d0, kw=0x0) at Objects/classobject.c:2578
>>>> #13 0x00000000004195c7 in PyObject_Call (func=0x26b72370, arg=0x80,
>>>> kw=0x0) at Objects/abstract.c:2522
>>>> #14 0x00007f596cb85f36 in ?? ()
>>>> #15 0x0000000000000000 in ?? ()
>>>>
>>>
>>> Still can't find which one test have thread left.
>>>
>>> Here is workaround:
>>>
>>> import threading
>>> import time
>>> import os
>>>
>>> class XXX(threading.Thread):
>>> ? ?def run(self):
>>> ? ? ? ?print ('Bad thread')
>>> ? ? ? ?while True:
>>> ? ? ? ? ? ?time.sleep(1)
>>> XXX().start()
>>>
>>> time.sleep(.1)
>>>
>>> if len(threading._active) > 1:
>>> ? ?import warnings
>>> ? ?print("Warning more then one threads left, forcing _exit()")
>>> ? ?os._exit(0)
>>>
>>>
>>> I'm not sure it is a good idea to add this into runtests.py
>>>
>>
>> Btw I think we need a way to find blocking thread.
>>
>> wait_for_thread_shutdown() calls threading._shutdown and we can hook
>> it and test is there NonDaemon threads left and try to join
>> them with timeout. If it fails call _exit and print error message.
>>
>> One more problem: how to make it portable between python versions.
>> --
>> vitja.
>>
>
> 2.5, 2.6, 2.7, 3.x use threading._shutdown
>
> 2.3,2.4 use sys.exitfunc
>
> --
> vitja.
>

I added patch to runthreads to check thread state at exit.

http://trac.cython.org/cython_trac/attachment/ticket/596/runtests-check-threads.patch

-- 
vitja.

From stefan_ml at behnel.de  Sun Nov 21 08:08:08 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 21 Nov 2010 08:08:08 +0100
Subject: [Cython] [bug] inf loop
In-Reply-To: <AANLkTimi1pkHt7FP-LssbvDUVmTWM0mLCjNu9iB19+hC@mail.gmail.com>
References: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>	<AANLkTi=bWs0X8peV0fQQeQx4_okvduQue_f5nJD75Nta@mail.gmail.com>	<4CE53725.5050707@behnel.de>	<AANLkTi=s_tFgLxGXnb5xQxyjwc8gyquTuUBOGkNgr-d5@mail.gmail.com>	<AANLkTimAYXqVOBWU_Bdr_ihP9_s5g3iEvZf0TJgmcVDN@mail.gmail.com>	<4CE58419.7070000@behnel.de>	<AANLkTi=v8BVSLduQkqmmmOZ1sKp9XSsx5XeMcU8FR7MA@mail.gmail.com>	<4CE58BAC.6020105@behnel.de>	<AANLkTik68aQ3SQ9Se=KmNpAU3jkQopxdw5E39L7SSMfm@mail.gmail.com>	<4CE5FE5B.30108@behnel.de>	<AANLkTi=9sAz7Oy4yHRKR-AXWcwgEngmWd4ONYKS_gyD7@mail.gmail.com>	<AANLkTikMz4KD72MhycysTKY+UBKMPBKOoLqiiW056hdR@mail.gmail.com>	<AANLkTi=669wu-DaLBNV8T4ryn5cbfvmr7vczB7HH3V6R@mail.gmail.com>	<AANLkTik4_eiTqB99-vHQzQow8b9wWYcFk5xYPVDKL+15@mail.gmail.com>
	<AANLkTimi1pkHt7FP-LssbvDUVmTWM0mLCjNu9iB19+hC@mail.gmail.com>
Message-ID: <4CE8C558.40206@behnel.de>

Vitja Makarov, 21.11.2010 00:02:
> I added patch to runthreads to check thread state at exit.
>
> http://trac.cython.org/cython_trac/attachment/ticket/596/runtests-check-threads.patch

This isn't really related to ticket #596 (just like the fix for ticket #537 
wasn't). In any case, this should be done after each test, not at 
interpreter exit time. It sounds wrong if any thread (even a deamon thread) 
is still alive after running a test.

Stefan

From vitja.makarov at gmail.com  Sun Nov 21 09:29:22 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sun, 21 Nov 2010 11:29:22 +0300
Subject: [Cython] [bug] inf loop
In-Reply-To: <4CE8C558.40206@behnel.de>
References: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>
	<AANLkTi=bWs0X8peV0fQQeQx4_okvduQue_f5nJD75Nta@mail.gmail.com>
	<4CE53725.5050707@behnel.de>
	<AANLkTi=s_tFgLxGXnb5xQxyjwc8gyquTuUBOGkNgr-d5@mail.gmail.com>
	<AANLkTimAYXqVOBWU_Bdr_ihP9_s5g3iEvZf0TJgmcVDN@mail.gmail.com>
	<4CE58419.7070000@behnel.de>
	<AANLkTi=v8BVSLduQkqmmmOZ1sKp9XSsx5XeMcU8FR7MA@mail.gmail.com>
	<4CE58BAC.6020105@behnel.de>
	<AANLkTik68aQ3SQ9Se=KmNpAU3jkQopxdw5E39L7SSMfm@mail.gmail.com>
	<4CE5FE5B.30108@behnel.de>
	<AANLkTi=9sAz7Oy4yHRKR-AXWcwgEngmWd4ONYKS_gyD7@mail.gmail.com>
	<AANLkTikMz4KD72MhycysTKY+UBKMPBKOoLqiiW056hdR@mail.gmail.com>
	<AANLkTi=669wu-DaLBNV8T4ryn5cbfvmr7vczB7HH3V6R@mail.gmail.com>
	<AANLkTik4_eiTqB99-vHQzQow8b9wWYcFk5xYPVDKL+15@mail.gmail.com>
	<AANLkTimi1pkHt7FP-LssbvDUVmTWM0mLCjNu9iB19+hC@mail.gmail.com>
	<4CE8C558.40206@behnel.de>
Message-ID: <AANLkTi=TeaPuSNDANi2PK5E+h3BYJ5KyR+paMXzJRn=x@mail.gmail.com>

2010/11/21 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 21.11.2010 00:02:
>> I added patch to runthreads to check thread state at exit.
>>
>> http://trac.cython.org/cython_trac/attachment/ticket/596/runtests-check-threads.patch
>
> This isn't really related to ticket #596 (just like the fix for ticket #537
> wasn't). In any case, this should be done after each test, not at
> interpreter exit time. It sounds wrong if any thread (even a deamon thread)
> is still alive after running a test.
>

Sure that's a different issue.

You've pushed check_thread_termination() it seems to me that it only
checks, and don't help python interpreter to exit.
So we can find this tests and disable. Or hook threading._shutdown and
allow python quit.

Think we should open new ticket for thread issue, it seems to be
related to GIL, right?
I don't know how to describe that in general way.

-- 
vitja.

From stefan_ml at behnel.de  Sun Nov 21 10:27:30 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 21 Nov 2010 10:27:30 +0100
Subject: [Cython] [bug] inf loop
In-Reply-To: <AANLkTi=TeaPuSNDANi2PK5E+h3BYJ5KyR+paMXzJRn=x@mail.gmail.com>
References: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>	<AANLkTi=bWs0X8peV0fQQeQx4_okvduQue_f5nJD75Nta@mail.gmail.com>	<4CE53725.5050707@behnel.de>	<AANLkTi=s_tFgLxGXnb5xQxyjwc8gyquTuUBOGkNgr-d5@mail.gmail.com>	<AANLkTimAYXqVOBWU_Bdr_ihP9_s5g3iEvZf0TJgmcVDN@mail.gmail.com>	<4CE58419.7070000@behnel.de>	<AANLkTi=v8BVSLduQkqmmmOZ1sKp9XSsx5XeMcU8FR7MA@mail.gmail.com>	<4CE58BAC.6020105@behnel.de>	<AANLkTik68aQ3SQ9Se=KmNpAU3jkQopxdw5E39L7SSMfm@mail.gmail.com>	<4CE5FE5B.30108@behnel.de>	<AANLkTi=9sAz7Oy4yHRKR-AXWcwgEngmWd4ONYKS_gyD7@mail.gmail.com>	<AANLkTikMz4KD72MhycysTKY+UBKMPBKOoLqiiW056hdR@mail.gmail.com>	<AANLkTi=669wu-DaLBNV8T4ryn5cbfvmr7vczB7HH3V6R@mail.gmail.com>	<AANLkTik4_eiTqB99-vHQzQow8b9wWYcFk5xYPVDKL+15@mail.gmail.com>	<AANLkTimi1pkHt7FP-LssbvDUVmTWM0mLCjNu9iB19+hC@mail.gmail.com>	<4CE8C558.40206@behnel.de>
	<AANLkTi=TeaPuSNDANi2PK5E+h3BYJ5KyR+paMXzJRn=x@mail.gmail.com>
Message-ID: <4CE8E602.7070706@behnel.de>

Vitja Makarov, 21.11.2010 09:29:
> 2010/11/21 Stefan Behnel:
>> Vitja Makarov, 21.11.2010 00:02:
>>> I added patch to runthreads to check thread state at exit.
>>>
>>> http://trac.cython.org/cython_trac/attachment/ticket/596/runtests-check-threads.patch
>>
>> This isn't really related to ticket #596 (just like the fix for ticket #537
>> wasn't). In any case, this should be done after each test, not at
>> interpreter exit time. It sounds wrong if any thread (even a deamon thread)
>> is still alive after running a test.
>>
>
> Sure that's a different issue.
>
> You've pushed check_thread_termination() it seems to me that it only
> checks, and don't help python interpreter to exit.
> So we can find this tests and disable. Or hook threading._shutdown and
> allow python quit.

You're right, I think we need both. I pushed a fix that (I hope) handles 
both cases.


> Think we should open new ticket for thread issue

The test runner hasn't really been a reason to open tickets before. When we 
find bugs, we just fix them.


> it seems to be related to GIL, right?

No, just to threading in general. It's not even a problem with the compiler 
(i.e. what users would normally see), just with the test support. The 
support for running CPython regression tests is really rudimentary. We just 
compile what we find and let unittest look for anything it thinks it can 
deal with. To do it right, we'd have to emulate CPython's own test runner 
to make sure we ignore abstract test classes and the like. We may have to 
do that at some point, now that we are getting closer and closer to the 
point that most tests should "just work". But as long as it's enough to 
disable a hand full of test modules, I don't feel obliged to put much 
effort into this. After all, most of the tests just heavily deal with the 
standard library, so they really aren't that telling w.r.t. Cython.

Stefan

From vitja.makarov at gmail.com  Sun Nov 21 10:51:45 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sun, 21 Nov 2010 12:51:45 +0300
Subject: [Cython] [bug] inf loop
In-Reply-To: <4CE8E602.7070706@behnel.de>
References: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>
	<AANLkTi=bWs0X8peV0fQQeQx4_okvduQue_f5nJD75Nta@mail.gmail.com>
	<4CE53725.5050707@behnel.de>
	<AANLkTi=s_tFgLxGXnb5xQxyjwc8gyquTuUBOGkNgr-d5@mail.gmail.com>
	<AANLkTimAYXqVOBWU_Bdr_ihP9_s5g3iEvZf0TJgmcVDN@mail.gmail.com>
	<4CE58419.7070000@behnel.de>
	<AANLkTi=v8BVSLduQkqmmmOZ1sKp9XSsx5XeMcU8FR7MA@mail.gmail.com>
	<4CE58BAC.6020105@behnel.de>
	<AANLkTik68aQ3SQ9Se=KmNpAU3jkQopxdw5E39L7SSMfm@mail.gmail.com>
	<4CE5FE5B.30108@behnel.de>
	<AANLkTi=9sAz7Oy4yHRKR-AXWcwgEngmWd4ONYKS_gyD7@mail.gmail.com>
	<AANLkTikMz4KD72MhycysTKY+UBKMPBKOoLqiiW056hdR@mail.gmail.com>
	<AANLkTi=669wu-DaLBNV8T4ryn5cbfvmr7vczB7HH3V6R@mail.gmail.com>
	<AANLkTik4_eiTqB99-vHQzQow8b9wWYcFk5xYPVDKL+15@mail.gmail.com>
	<AANLkTimi1pkHt7FP-LssbvDUVmTWM0mLCjNu9iB19+hC@mail.gmail.com>
	<4CE8C558.40206@behnel.de>
	<AANLkTi=TeaPuSNDANi2PK5E+h3BYJ5KyR+paMXzJRn=x@mail.gmail.com>
	<4CE8E602.7070706@behnel.de>
Message-ID: <AANLkTi=NPumJy=ZPyu9L7fR0jsKYcWfBnc4UEEkN=3e3@mail.gmail.com>

2010/11/21 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 21.11.2010 09:29:
>> 2010/11/21 Stefan Behnel:
>>> Vitja Makarov, 21.11.2010 00:02:
>>>> I added patch to runthreads to check thread state at exit.
>>>>
>>>> http://trac.cython.org/cython_trac/attachment/ticket/596/runtests-check-threads.patch
>>>
>>> This isn't really related to ticket #596 (just like the fix for ticket #537
>>> wasn't). In any case, this should be done after each test, not at
>>> interpreter exit time. It sounds wrong if any thread (even a deamon thread)
>>> is still alive after running a test.
>>>
>>
>> Sure that's a different issue.
>>
>> You've pushed check_thread_termination() it seems to me that it only
>> checks, and don't help python interpreter to exit.
>> So we can find this tests and disable. Or hook threading._shutdown and
>> allow python quit.
>
> You're right, I think we need both. I pushed a fix that (I hope) handles
> both cases.
>
>
if __name__ == '__main__':
    try:
        main()
    except Exception:
        traceback.print_exc()
        try:
            check_thread_termination(ignore_seen=False)
        except PendingThreadsError:
            # normal program exit won't kill the threads, do it the
hard way here
            os._exit(1)

check_thread_termination() should be called in both exception and
normal case, so why do not place check in finally?
So now it doesn't work. Why don't you want to hook it to
threading._shutdown and sys.exitcall?

>> Think we should open new ticket for thread issue
>
> The test runner hasn't really been a reason to open tickets before. When we
> find bugs, we just fix them.
>
>
>> it seems to be related to GIL, right?
>
> No, just to threading in general. It's not even a problem with the compiler
> (i.e. what users would normally see), just with the test support. The
> support for running CPython regression tests is really rudimentary. We just
> compile what we find and let unittest look for anything it thinks it can
> deal with. To do it right, we'd have to emulate CPython's own test runner
> to make sure we ignore abstract test classes and the like. We may have to
> do that at some point, now that we are getting closer and closer to the
> point that most tests should "just work". But as long as it's enough to
> disable a hand full of test modules, I don't feel obliged to put much
> effort into this. After all, most of the tests just heavily deal with the
> standard library, so they really aren't that telling w.r.t. Cython.
>

When I mean gil I mean that gil may be never released in cython code.

-- 
vitja.

From stefan_ml at behnel.de  Sun Nov 21 11:01:14 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 21 Nov 2010 11:01:14 +0100
Subject: [Cython] threading related test issues (was: [bug] inf loop)
In-Reply-To: <AANLkTi=NPumJy=ZPyu9L7fR0jsKYcWfBnc4UEEkN=3e3@mail.gmail.com>
References: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>	<4CE53725.5050707@behnel.de>	<AANLkTi=s_tFgLxGXnb5xQxyjwc8gyquTuUBOGkNgr-d5@mail.gmail.com>	<AANLkTimAYXqVOBWU_Bdr_ihP9_s5g3iEvZf0TJgmcVDN@mail.gmail.com>	<4CE58419.7070000@behnel.de>	<AANLkTi=v8BVSLduQkqmmmOZ1sKp9XSsx5XeMcU8FR7MA@mail.gmail.com>	<4CE58BAC.6020105@behnel.de>	<AANLkTik68aQ3SQ9Se=KmNpAU3jkQopxdw5E39L7SSMfm@mail.gmail.com>	<4CE5FE5B.30108@behnel.de>	<AANLkTi=9sAz7Oy4yHRKR-AXWcwgEngmWd4ONYKS_gyD7@mail.gmail.com>	<AANLkTikMz4KD72MhycysTKY+UBKMPBKOoLqiiW056hdR@mail.gmail.com>	<AANLkTi=669wu-DaLBNV8T4ryn5cbfvmr7vczB7HH3V6R@mail.gmail.com>	<AANLkTik4_eiTqB99-vHQzQow8b9wWYcFk5xYPVDKL+15@mail.gmail.com>	<AANLkTimi1pkHt7FP-LssbvDUVmTWM0mLCjNu9iB19+hC@mail.gmail.com>	<4CE8C558.40206@behnel.de>	<AANLkTi=TeaPuSNDANi2PK5E+h3BYJ5KyR+paMXzJRn=x@mail.gmail.com>	<4CE8E602.7070706@behnel.de>
	<AANLkTi=NPumJy=ZPyu9L7fR0jsKYcWfBnc4UEEkN=3e3@mail.gmail.com>
Message-ID: <4CE8EDEA.6080508@behnel.de>

Vitja Makarov, 21.11.2010 10:51:
> if __name__ == '__main__':
>      try:
>          main()
>      except Exception:
>          traceback.print_exc()
>          try:
>              check_thread_termination(ignore_seen=False)
>          except PendingThreadsError:
>              # normal program exit won't kill the threads, do it the
> hard way here
>              os._exit(1)
>
> check_thread_termination() should be called in both exception and
> normal case, so why do not place check in finally?

Note that this exception handler is only ever reached when an exception 
terminates the test runner's main() function. The other (more important) 
cases are handled by the exit code at the end of main().


> So now it doesn't work. Why don't you want to hook it to
> threading._shutdown and sys.exitcall?

I wouldn't mind doing that, but my feeling is that it makes things less 
predictable.


> When I mean gil I mean that gil may be never released in cython code.

Ah, ok. Yes, that may be worth a ticket, although I think having code 
depend on the GIL being released arbitrarily has a bit of a code smell. It 
should never happen in Cython code, although it may be worth discussing for 
plain Python modules.

Stefan

From robertwb at math.washington.edu  Sun Nov 21 11:13:40 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Sun, 21 Nov 2010 02:13:40 -0800
Subject: [Cython] WIki migration
Message-ID: <AANLkTimOLb+vQf4knzcp6tnBX2WP+zmSVmRi4SSKDk++@mail.gmail.com>

I've started to look at migrating our wiki to github (or, more
specifically, a git repo full of markdown/ReST text documents) and it
looks like it'll be fairly easy to extract the data from moinmoin,
including history. Does anyone know of any good moinmoin to ReST
translators?

- Robert

From vitja.makarov at gmail.com  Sun Nov 21 11:27:45 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sun, 21 Nov 2010 13:27:45 +0300
Subject: [Cython] threading related test issues (was: [bug] inf loop)
In-Reply-To: <4CE8EDEA.6080508@behnel.de>
References: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>
	<4CE53725.5050707@behnel.de>
	<AANLkTi=s_tFgLxGXnb5xQxyjwc8gyquTuUBOGkNgr-d5@mail.gmail.com>
	<AANLkTimAYXqVOBWU_Bdr_ihP9_s5g3iEvZf0TJgmcVDN@mail.gmail.com>
	<4CE58419.7070000@behnel.de>
	<AANLkTi=v8BVSLduQkqmmmOZ1sKp9XSsx5XeMcU8FR7MA@mail.gmail.com>
	<4CE58BAC.6020105@behnel.de>
	<AANLkTik68aQ3SQ9Se=KmNpAU3jkQopxdw5E39L7SSMfm@mail.gmail.com>
	<4CE5FE5B.30108@behnel.de>
	<AANLkTi=9sAz7Oy4yHRKR-AXWcwgEngmWd4ONYKS_gyD7@mail.gmail.com>
	<AANLkTikMz4KD72MhycysTKY+UBKMPBKOoLqiiW056hdR@mail.gmail.com>
	<AANLkTi=669wu-DaLBNV8T4ryn5cbfvmr7vczB7HH3V6R@mail.gmail.com>
	<AANLkTik4_eiTqB99-vHQzQow8b9wWYcFk5xYPVDKL+15@mail.gmail.com>
	<AANLkTimi1pkHt7FP-LssbvDUVmTWM0mLCjNu9iB19+hC@mail.gmail.com>
	<4CE8C558.40206@behnel.de>
	<AANLkTi=TeaPuSNDANi2PK5E+h3BYJ5KyR+paMXzJRn=x@mail.gmail.com>
	<4CE8E602.7070706@behnel.de>
	<AANLkTi=NPumJy=ZPyu9L7fR0jsKYcWfBnc4UEEkN=3e3@mail.gmail.com>
	<4CE8EDEA.6080508@behnel.de>
Message-ID: <AANLkTimje+qhxohk6JSsJHoK-TcRMxw1rvGZ-GXiTMGy@mail.gmail.com>

2010/11/21 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 21.11.2010 10:51:
>> if __name__ == '__main__':
>> ? ? ?try:
>> ? ? ? ? ?main()
>> ? ? ?except Exception:
>> ? ? ? ? ?traceback.print_exc()
>> ? ? ? ? ?try:
>> ? ? ? ? ? ? ?check_thread_termination(ignore_seen=False)
>> ? ? ? ? ?except PendingThreadsError:
>> ? ? ? ? ? ? ?# normal program exit won't kill the threads, do it the
>> hard way here
>> ? ? ? ? ? ? ?os._exit(1)
>>
>> check_thread_termination() should be called in both exception and
>> normal case, so why do not place check in finally?
>
> Note that this exception handler is only ever reached when an exception
> terminates the test runner's main() function. The other (more important)
> cases are handled by the exit code at the end of main().
>
>
>> So now it doesn't work. Why don't you want to hook it to
>> threading._shutdown and sys.exitcall?
>
> I wouldn't mind doing that, but my feeling is that it makes things less
> predictable.
>

Yeah I see. The problem is check_thread_termination() ignores
ignore_seen argument.

>
>> When I mean gil I mean that gil may be never released in cython code.
>
> Ah, ok. Yes, that may be worth a ticket, although I think having code
> depend on the GIL being released arbitrarily has a bit of a code smell. It
> should never happen in Cython code, although it may be worth discussing for
> plain Python modules.
>

Would 'with nogil' help here?

-- 
vitja.

From stefan_ml at behnel.de  Sun Nov 21 11:40:03 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 21 Nov 2010 11:40:03 +0100
Subject: [Cython] threading related test issues
In-Reply-To: <AANLkTimje+qhxohk6JSsJHoK-TcRMxw1rvGZ-GXiTMGy@mail.gmail.com>
References: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>	<AANLkTimAYXqVOBWU_Bdr_ihP9_s5g3iEvZf0TJgmcVDN@mail.gmail.com>	<4CE58419.7070000@behnel.de>	<AANLkTi=v8BVSLduQkqmmmOZ1sKp9XSsx5XeMcU8FR7MA@mail.gmail.com>	<4CE58BAC.6020105@behnel.de>	<AANLkTik68aQ3SQ9Se=KmNpAU3jkQopxdw5E39L7SSMfm@mail.gmail.com>	<4CE5FE5B.30108@behnel.de>	<AANLkTi=9sAz7Oy4yHRKR-AXWcwgEngmWd4ONYKS_gyD7@mail.gmail.com>	<AANLkTikMz4KD72MhycysTKY+UBKMPBKOoLqiiW056hdR@mail.gmail.com>	<AANLkTi=669wu-DaLBNV8T4ryn5cbfvmr7vczB7HH3V6R@mail.gmail.com>	<AANLkTik4_eiTqB99-vHQzQow8b9wWYcFk5xYPVDKL+15@mail.gmail.com>	<AANLkTimi1pkHt7FP-LssbvDUVmTWM0mLCjNu9iB19+hC@mail.gmail.com>	<4CE8C558.40206@behnel.de>	<AANLkTi=TeaPuSNDANi2PK5E+h3BYJ5KyR+paMXzJRn=x@mail.gmail.com>	<4CE8E602.7070706@behnel.de>	<AANLkTi=NPumJy=ZPyu9L7fR0jsKYcWfBnc4UEEkN=3e3@mail.gmail.com>	<4CE8EDEA.6080508@behnel.de>
	<AANLkTimje+qhxohk6JSsJHoK-TcRMxw1rvGZ-GXiTMGy@mail.gmail.com>
Message-ID: <4CE8F703.6090708@behnel.de>

Vitja Makarov, 21.11.2010 11:27:
> 2010/11/21 Stefan Behnel:
>> Vitja Makarov, 21.11.2010 10:51:
>>> if __name__ == '__main__':
>>>       try:
>>>           main()
>>>       except Exception:
>>>           traceback.print_exc()
>>>           try:
>>>               check_thread_termination(ignore_seen=False)
>>>           except PendingThreadsError:
>>>               # normal program exit won't kill the threads, do it the
>>> hard way here
>>>               os._exit(1)
>>>
>>> check_thread_termination() should be called in both exception and
>>> normal case, so why do not place check in finally?
>>
>> Note that this exception handler is only ever reached when an exception
>> terminates the test runner's main() function. The other (more important)
>> cases are handled by the exit code at the end of main().
>>
>>
>>> So now it doesn't work. Why don't you want to hook it to
>>> threading._shutdown and sys.exitcall?
>>
>> I wouldn't mind doing that, but my feeling is that it makes things less
>> predictable.
>
> Yeah I see. The problem is check_thread_termination() ignores
> ignore_seen argument.

:-] good catch, thanks!


>>> When I mean gil I mean that gil may be never released in cython code.
>>
>> Ah, ok. Yes, that may be worth a ticket, although I think having code
>> depend on the GIL being released arbitrarily has a bit of a code smell. It
>> should never happen in Cython code, although it may be worth discussing for
>> plain Python modules.
>
> Would 'with nogil' help here?

Sure, a

      with nogil: pass

will free the GIL and potentially allow other threads to take over.

Stefan

From stefan_ml at behnel.de  Sun Nov 21 17:07:54 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 21 Nov 2010 17:07:54 +0100
Subject: [Cython] class closures
In-Reply-To: <AANLkTi=bWs0X8peV0fQQeQx4_okvduQue_f5nJD75Nta@mail.gmail.com>
References: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>
	<AANLkTi=bWs0X8peV0fQQeQx4_okvduQue_f5nJD75Nta@mail.gmail.com>
Message-ID: <4CE943DA.4010703@behnel.de>

Vitja Makarov, 17.11.2010 11:33:
> I've created ticket for #596 for class closures.

Just a quick follow-up here, your patch is now in. It brought the test 
count for the CPython regression test suite up from ~2300 to ~4600. Most of 
the newly discovered tests pass. Looks like conditional or inner classes 
are a very common feature in test suites.

Stefan

From Paul.Connell at disney.com  Mon Nov 22 10:03:07 2010
From: Paul.Connell at disney.com (Connell, Paul)
Date: Mon, 22 Nov 2010 09:03:07 +0000
Subject: [Cython] GitHub for Cython
In-Reply-To: <AANLkTinFLOtcHvdUj-9SnX=7uZELoH9W74zRfy37df=z@mail.gmail.com>
References: <AANLkTinFLOtcHvdUj-9SnX=7uZELoH9W74zRfy37df=z@mail.gmail.com>
Message-ID: <DA536BC5C16ADD4D850A9522DB86E01E51C2082D2C@SM-GBLO-VXM01A.emea.wdpr.disney.com>

This might be nitpicking, but "a Python to C compiler" is a questionable tagline for what Cython is/does... That's a description of something like shedskin isn't it?


-----Original Message-----
From: cython-dev-bounces at codespeak.net [mailto:cython-dev-bounces at codespeak.net] On Behalf Of Robert Bradshaw
Sent: 19 November 2010 23:03
To: cython-dev at codespeak.net
Subject: [Cython] GitHub for Cython

It seems that the clear consensus, or at least the one thing that we can all be happy about, is moving to GitHub (crossing our fingers that the issue tracker improves quickly). I've created https://github.com/cython , but haven't started to migrate the rest of the stuff over. I'm thinking we can still stick with the push-to-main option for the core developers (which seems to still be appropriate for our project right now, and also works well with stuff like the build-bot), but can easily fork for experimental work, reviews, and anyone else who wants to contribute.

Fortunately, with a DVCS, we don't have to "flip the switch" all at once. I'll be migrating the site off of the VM and the trac/wiki out to GitHub soonish (and this will probably be a flip-the-switch kind of thing), but would welcome any help. For those who need admin access, let me know what your usernames are.


- Robert



From stefan_ml at behnel.de  Mon Nov 22 11:49:49 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 22 Nov 2010 11:49:49 +0100
Subject: [Cython] GitHub for Cython
In-Reply-To: <DA536BC5C16ADD4D850A9522DB86E01E51C2082D2C@SM-GBLO-VXM01A.emea.wdpr.disney.com>
References: <AANLkTinFLOtcHvdUj-9SnX=7uZELoH9W74zRfy37df=z@mail.gmail.com>
	<DA536BC5C16ADD4D850A9522DB86E01E51C2082D2C@SM-GBLO-VXM01A.emea.wdpr.disney.com>
Message-ID: <4CEA4ACD.5070907@behnel.de>

Connell, Paul, 22.11.2010 10:03:
> Robert Bradshaw, 19 November 2010 23:03
>> https://github.com/cython
>
> This might be nitpicking, but "a Python to C compiler" is a questionable
> tagline for what Cython is/does... That's a description of something
> like shedskin isn't it?

Right, Cython is a lot more than that. I find the "C-extensions for Python" 
from our homepage a lot broader, the idea being that basically everything 
you could do with a C extension to CPython, you can do better with Cython.

Stefan

From dagss at student.matnat.uio.no  Mon Nov 22 11:58:08 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Mon, 22 Nov 2010 11:58:08 +0100
Subject: [Cython] GitHub for Cython
In-Reply-To: <4CEA4ACD.5070907@behnel.de>
References: <AANLkTinFLOtcHvdUj-9SnX=7uZELoH9W74zRfy37df=z@mail.gmail.com>	<DA536BC5C16ADD4D850A9522DB86E01E51C2082D2C@SM-GBLO-VXM01A.emea.wdpr.disney.com>
	<4CEA4ACD.5070907@behnel.de>
Message-ID: <4CEA4CC0.4090305@student.matnat.uio.no>

On 11/22/2010 11:49 AM, Stefan Behnel wrote:
> Connell, Paul, 22.11.2010 10:03:
>    
>> Robert Bradshaw, 19 November 2010 23:03
>>      
>>> https://github.com/cython
>>>        
>> This might be nitpicking, but "a Python to C compiler" is a questionable
>> tagline for what Cython is/does... That's a description of something
>> like shedskin isn't it?
>>      
> Right, Cython is a lot more than that. I find the "C-extensions for Python"
> from our homepage a lot broader, the idea being that basically everything
> you could do with a C extension to CPython, you can do better with Cython.
>    

Problem is that some users may not know what "C-extensions for Python" 
is about. "Python to C compiler" communicates to a much broader range of 
people, who may feel that Python is too slow but don't quite know how to 
go about it.

Considering the purpose of that subject line, I feel the latter is 
better. People who know what a C-extension means will often already have 
heard about Cython and not bother to read the line.

(Also, as is, "C-extensions for Python" sounds like we're a repository 
of extensions, so at least a little rewording is needed.)

I don't feel "Python to C compiler" is wrong. We just plug into and use 
the API provided by CPython.  shedskin also uses a library in a sense, 
although one reimplemented (from scratch?), and that may run a little 
faster. Although I can see the argument that it may make certain people 
think that we can magically turn *any* Python code into lightning fast C 
code...

Dag Sverre

From vitja.makarov at gmail.com  Mon Nov 22 19:26:48 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Mon, 22 Nov 2010 21:26:48 +0300
Subject: [Cython] class closures
In-Reply-To: <4CE943DA.4010703@behnel.de>
References: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>
	<AANLkTi=bWs0X8peV0fQQeQx4_okvduQue_f5nJD75Nta@mail.gmail.com>
	<4CE943DA.4010703@behnel.de>
Message-ID: <AANLkTikuCr=GPMFDmxra7HBMqiuRu8rV+eMoToAwZpDE@mail.gmail.com>

2010/11/21 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 17.11.2010 11:33:
>> I've created ticket for #596 for class closures.
>
> Just a quick follow-up here, your patch is now in. It brought the test
> count for the CPython regression test suite up from ~2300 to ~4600. Most of
> the newly discovered tests pass. Looks like conditional or inner classes
> are a very common feature in test suites.
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

Good news! May be it is time to enable closure_class_T596 testcase?

-- 
vitja.

From robertwb at math.washington.edu  Mon Nov 22 19:42:17 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 22 Nov 2010 10:42:17 -0800
Subject: [Cython] class closures
In-Reply-To: <4CE943DA.4010703@behnel.de>
References: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>
	<AANLkTi=bWs0X8peV0fQQeQx4_okvduQue_f5nJD75Nta@mail.gmail.com>
	<4CE943DA.4010703@behnel.de>
Message-ID: <AANLkTikAb6nr0LhBDzf68DW-uuTUJhMRgQwhPhFeKYeU@mail.gmail.com>

On Sun, Nov 21, 2010 at 8:07 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Vitja Makarov, 17.11.2010 11:33:
>> I've created ticket for #596 for class closures.
>
> Just a quick follow-up here, your patch is now in. It brought the test
> count for the CPython regression test suite up from ~2300 to ~4600. Most of
> the newly discovered tests pass. Looks like conditional or inner classes
> are a very common feature in test suites.

Wow, very cool. Thanks Stefan and Vitja! For reference,
https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py27-c/
Any common themes in the failing ones (besides the obvious, e.g.
generators).

- Robert

From robertwb at math.washington.edu  Mon Nov 22 19:59:07 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 22 Nov 2010 10:59:07 -0800
Subject: [Cython] GitHub for Cython
In-Reply-To: <4CEA4CC0.4090305@student.matnat.uio.no>
References: <AANLkTinFLOtcHvdUj-9SnX=7uZELoH9W74zRfy37df=z@mail.gmail.com>
	<DA536BC5C16ADD4D850A9522DB86E01E51C2082D2C@SM-GBLO-VXM01A.emea.wdpr.disney.com>
	<4CEA4ACD.5070907@behnel.de> <4CEA4CC0.4090305@student.matnat.uio.no>
Message-ID: <AANLkTin50qnuqME+fzbo2DJ6jtzR9c=v_m9F_7D1ks3y@mail.gmail.com>

On Mon, Nov 22, 2010 at 2:58 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> On 11/22/2010 11:49 AM, Stefan Behnel wrote:
>> Connell, Paul, 22.11.2010 10:03:
>>
>>> Robert Bradshaw, 19 November 2010 23:03
>>>
>>>> https://github.com/cython
>>>>
>>> This might be nitpicking, but "a Python to C compiler" is a questionable
>>> tagline for what Cython is/does... That's a description of something
>>> like shedskin isn't it?
>>>
>> Right, Cython is a lot more than that. I find the "C-extensions for Python"
>> from our homepage a lot broader, the idea being that basically everything
>> you could do with a C extension to CPython, you can do better with Cython.
>>
>
> Problem is that some users may not know what "C-extensions for Python"
> is about. "Python to C compiler" communicates to a much broader range of
> people, who may feel that Python is too slow but don't quite know how to
> go about it.
>
> Considering the purpose of that subject line, I feel the latter is
> better. People who know what a C-extension means will often already have
> heard about Cython and not bother to read the line.

This is exactly whey I wrote the current tagline--I know people who've
written (good) Cython code that wouldn't even know what you were
talking about if you said "C-extension."

> (Also, as is, "C-extensions for Python" sounds like we're a repository
> of extensions, so at least a little rewording is needed.)
>
> I don't feel "Python to C compiler" is wrong. We just plug into and use
> the API provided by CPython. ?shedskin also uses a library in a sense,
> although one reimplemented (from scratch?), and that may run a little
> faster. Although I can see the argument that it may make certain people
> think that we can magically turn *any* Python code into lightning fast C
> code...

True. If someone were to ask me, in one word, what Cython was, I would
call it a compiler. I'm open to suggestions, but I don't think
"C-extension" captures the idea (and attention) well for the intended
audience. Also, though it's a bit of an abuse of the term, I think of
"C-extension module" as a handcoded alternative to "Cython extension
module" or "SWIG extension module" or (eventual) "frap extension
module."

- Robert

From ondrej at certik.cz  Tue Nov 23 01:23:36 2010
From: ondrej at certik.cz (Ondrej Certik)
Date: Mon, 22 Nov 2010 16:23:36 -0800
Subject: [Cython] GitHub for Cython
In-Reply-To: <AANLkTin50qnuqME+fzbo2DJ6jtzR9c=v_m9F_7D1ks3y@mail.gmail.com>
References: <AANLkTinFLOtcHvdUj-9SnX=7uZELoH9W74zRfy37df=z@mail.gmail.com>
	<DA536BC5C16ADD4D850A9522DB86E01E51C2082D2C@SM-GBLO-VXM01A.emea.wdpr.disney.com>
	<4CEA4ACD.5070907@behnel.de>
	<4CEA4CC0.4090305@student.matnat.uio.no>
	<AANLkTin50qnuqME+fzbo2DJ6jtzR9c=v_m9F_7D1ks3y@mail.gmail.com>
Message-ID: <AANLkTikKvkPF7uZFQYOt3YhHM5Vy6Ehr4fM=3VxCyr67@mail.gmail.com>

On Mon, Nov 22, 2010 at 10:59 AM, Robert Bradshaw
<robertwb at math.washington.edu> wrote:
> On Mon, Nov 22, 2010 at 2:58 AM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no> wrote:
>> On 11/22/2010 11:49 AM, Stefan Behnel wrote:
>>> Connell, Paul, 22.11.2010 10:03:
>>>
>>>> Robert Bradshaw, 19 November 2010 23:03
>>>>
>>>>> https://github.com/cython
>>>>>
>>>> This might be nitpicking, but "a Python to C compiler" is a questionable
>>>> tagline for what Cython is/does... That's a description of something
>>>> like shedskin isn't it?
>>>>
>>> Right, Cython is a lot more than that. I find the "C-extensions for Python"
>>> from our homepage a lot broader, the idea being that basically everything
>>> you could do with a C extension to CPython, you can do better with Cython.
>>>
>>
>> Problem is that some users may not know what "C-extensions for Python"
>> is about. "Python to C compiler" communicates to a much broader range of
>> people, who may feel that Python is too slow but don't quite know how to
>> go about it.
>>
>> Considering the purpose of that subject line, I feel the latter is
>> better. People who know what a C-extension means will often already have
>> heard about Cython and not bother to read the line.
>
> This is exactly whey I wrote the current tagline--I know people who've
> written (good) Cython code that wouldn't even know what you were
> talking about if you said "C-extension."
>
>> (Also, as is, "C-extensions for Python" sounds like we're a repository
>> of extensions, so at least a little rewording is needed.)
>>
>> I don't feel "Python to C compiler" is wrong. We just plug into and use
>> the API provided by CPython. ?shedskin also uses a library in a sense,
>> although one reimplemented (from scratch?), and that may run a little
>> faster. Although I can see the argument that it may make certain people
>> think that we can magically turn *any* Python code into lightning fast C
>> code...
>
> True. If someone were to ask me, in one word, what Cython was, I would
> call it a compiler. I'm open to suggestions, but I don't think
> "C-extension" captures the idea (and attention) well for the intended
> audience. Also, though it's a bit of an abuse of the term, I think of
> "C-extension module" as a handcoded alternative to "Cython extension
> module" or "SWIG extension module" or (eventual) "frap extension
> module."

I use Cython a lot, pretty much everyday for my work and research.

I view Cython as a "a Python to C compiler". Just like shedskin. I
think that the two projects are competing, that's for sure. Cython is
more robust, and more general, maybe in some cases a bit slower
without helping it by hand. But that's fine, just the fact, that they
are competing doesn't mean they can't use the same project
description.

Ondrej

From stefan_ml at behnel.de  Tue Nov 23 10:34:32 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 23 Nov 2010 10:34:32 +0100
Subject: [Cython] results of CPython regression tests (was: class
	closures)
In-Reply-To: <AANLkTikAb6nr0LhBDzf68DW-uuTUJhMRgQwhPhFeKYeU@mail.gmail.com>
References: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>	<AANLkTi=bWs0X8peV0fQQeQx4_okvduQue_f5nJD75Nta@mail.gmail.com>	<4CE943DA.4010703@behnel.de>
	<AANLkTikAb6nr0LhBDzf68DW-uuTUJhMRgQwhPhFeKYeU@mail.gmail.com>
Message-ID: <4CEB8AA8.70108@behnel.de>

Robert Bradshaw, 22.11.2010 19:42:
> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py27-c/
> Any common themes in the failing ones (besides the obvious, e.g.
> generators).

It's certainly worth looking through the results. I already noticed a 
couple of failures that indicate bugs in Cython. One is (once again) due to 
the tricky str handling rules in Cython and inconsistent usage of 
EncodedString in the code base. I'm currently looking into that.

There also seems to be a problem with the test runner that makes the 
py3regr tests hang (with --no-fork or XML result output). Apparently, 
certain test failures leak into subsequent test runs. Running the two tests 
"test_compileall" and "test_gzip" hangs reliable for me in py3k, after a 
compiler crash in the first. The weird thing about it is that the gzip test 
doesn't normally compile because it uses a genexpr in combination with 
"sorted()", but it compiles without errors after the crash in the 
compileall test. I faintly remember that we have a way to disable 
termination on compile errors, maybe that's not getting properly cleaned up 
here.

My current way to work around this is to implement "sorted(genexpr)" as 
inlined genexpr. ;)

Stefan

From nahuel.defosse at gmail.com  Tue Nov 23 16:14:35 2010
From: nahuel.defosse at gmail.com (=?ISO-8859-1?Q?Nahuel_Defoss=E9?=)
Date: Tue, 23 Nov 2010 12:14:35 -0300
Subject: [Cython] Wrapping C structs in Python Objects
Message-ID: <AANLkTimutuwkxKkFD2tX_y7EnTsiJcw8UZJka9iHgkF8@mail.gmail.com>

Hi everyone.
I'm quite new at Cython, but I'm impressed how fast I've produced some
usable code, it's a great project.

I'm trying to wrap a C library for regular expressions. It's called
Oniguruma. You might wonder why
would I like to reinvent the wheel when Python's re module is
reasonably fast and easy to use. Well,
my goal is to support oniguruma regexes since Textmate's syntax
definitions are not Python's re compatible.

Textmate has a lot of plugins, or bundles as they call them. Most of
these bundles are Open Source, but
there's no Open Source editor which supports them. Our aim is to
create text editor which supports these
bundles, and a key element is to be able to use Oniguruma regexes.


I've followed the tutorial, and I've translated most of the .h strcut
definitions. I want to wrap these structs in
python objects [0], I've read cdefs objects, but I'm not sure if
cython will automatically convert a C struct pointer, returned
by a C function, into a Python object, with some custom methods.


Just in case there's anybody interested in the project, here's the repo [1].

In particular the code I'm working on is this [2] based on this .h file [3].

Thanks
Nahuel

[0] http://docs.cython.org/src/tutorial/cdef_classes.html
[1] https://github.com/D3f0/prymatex
[2] https://github.com/D3f0/prymatex/blob/master/src/prymatex/lib/oniguruma/pyonig.pyx
[3] https://github.com/D3f0/prymatex/blob/master/src/prymatex/lib/oniguruma/includes/oniguruma.h

From stefan_ml at behnel.de  Tue Nov 23 16:33:06 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 23 Nov 2010 16:33:06 +0100
Subject: [Cython] Wrapping C structs in Python Objects
In-Reply-To: <AANLkTimutuwkxKkFD2tX_y7EnTsiJcw8UZJka9iHgkF8@mail.gmail.com>
References: <AANLkTimutuwkxKkFD2tX_y7EnTsiJcw8UZJka9iHgkF8@mail.gmail.com>
Message-ID: <4CEBDEB2.907@behnel.de>

Hi,

Nahuel Defoss?, 23.11.2010 16:14:
> I'm quite new at Cython, but I'm impressed how fast I've produced some
> usable code, it's a great project.

Happy to hear that, and welcome!


> I'm trying to wrap a C library for regular expressions.

Note that your message is better suited to the cython-users mailing list,
not the cython-dev (as in "core developers") mailing list.

Stefan

From nahuel.defosse at gmail.com  Tue Nov 23 16:35:01 2010
From: nahuel.defosse at gmail.com (=?ISO-8859-1?Q?Nahuel_Defoss=E9?=)
Date: Tue, 23 Nov 2010 12:35:01 -0300
Subject: [Cython] Wrapping C structs in Python Objects
In-Reply-To: <4CEBDEB2.907@behnel.de>
References: <AANLkTimutuwkxKkFD2tX_y7EnTsiJcw8UZJka9iHgkF8@mail.gmail.com>
	<4CEBDEB2.907@behnel.de>
Message-ID: <AANLkTikwHRSo9dO01agsBN+9qvda_MUHnjBPLjpz61vQ@mail.gmail.com>

Thanks, sorry. I'll re post there.

2010/11/23 Stefan Behnel <stefan_ml at behnel.de>:
> Hi,
>
> Nahuel Defoss?, 23.11.2010 16:14:
>> I'm quite new at Cython, but I'm impressed how fast I've produced some
>> usable code, it's a great project.
>
> Happy to hear that, and welcome!
>
>
>> I'm trying to wrap a C library for regular expressions.
>
> Note that your message is better suited to the cython-users mailing list,
> not the cython-dev (as in "core developers") mailing list.
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From stefan_ml at behnel.de  Tue Nov 23 17:23:10 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 23 Nov 2010 17:23:10 +0100
Subject: [Cython] GitHub for Cython
In-Reply-To: <AANLkTinSF-tjR87jYsARVYOebQ3cT7eJkupn8XvX+rcP@mail.gmail.com>
References: <AANLkTinFLOtcHvdUj-9SnX=7uZELoH9W74zRfy37df=z@mail.gmail.com>	<AANLkTinVrx3vqE9Nx7Rnj6bo4kvf+BVjNV3Y6Tqk_HCi@mail.gmail.com>
	<AANLkTinSF-tjR87jYsARVYOebQ3cT7eJkupn8XvX+rcP@mail.gmail.com>
Message-ID: <4CEBEA6E.6050106@behnel.de>

Robert Bradshaw, 20.11.2010 07:19:
> On Fri, Nov 19, 2010 at 6:26 PM, Fernando Perez wrote:
>> https://github.com/cython/cython
>
> Good point, I'll do that.

Ok, so if that's the new leading repo, what will become of the old repos on 
hg.cython.org?

Currently, there are changes in hg.cython.org (including some of my own) 
that are not on github. I tried cloning the github repo using hg-git and 
then pulling from my local repo, but that doesn't produce anything useful. 
So I currently have no idea how I'm supposed to migrate changes between the 
two in a somewhat automatic way.

So, what's the plan? Will the hg repo be kept up-to-date (and who or what 
will do that)? Or will it be taken down completely?

Stefan

From robertwb at math.washington.edu  Tue Nov 23 18:42:53 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 23 Nov 2010 09:42:53 -0800
Subject: [Cython] GitHub for Cython
In-Reply-To: <4CEBEA6E.6050106@behnel.de>
References: <AANLkTinFLOtcHvdUj-9SnX=7uZELoH9W74zRfy37df=z@mail.gmail.com>
	<AANLkTinVrx3vqE9Nx7Rnj6bo4kvf+BVjNV3Y6Tqk_HCi@mail.gmail.com>
	<AANLkTinSF-tjR87jYsARVYOebQ3cT7eJkupn8XvX+rcP@mail.gmail.com>
	<4CEBEA6E.6050106@behnel.de>
Message-ID: <AANLkTim6=fzV0JG=GWPJxoB82O+ZqmNY+QDn+8B2aaXX@mail.gmail.com>

On Tue, Nov 23, 2010 at 8:23 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Robert Bradshaw, 20.11.2010 07:19:
>> On Fri, Nov 19, 2010 at 6:26 PM, Fernando Perez wrote:
>>> https://github.com/cython/cython
>>
>> Good point, I'll do that.
>
> Ok, so if that's the new leading repo, what will become of the old repos on
> hg.cython.org?
>
> Currently, there are changes in hg.cython.org (including some of my own)
> that are not on github. I tried cloning the github repo using hg-git and
> then pulling from my local repo, but that doesn't produce anything useful.
> So I currently have no idea how I'm supposed to migrate changes between the
> two in a somewhat automatic way.

I've been able to pull from the one and push to the other, but I don't
know what all the ins and outs are. I just pushed everything up into
the git one.

> So, what's the plan? Will the hg repo be kept up-to-date (and who or what
> will do that)? Or will it be taken down completely?

I would say for the moment the hg repo is still the "live" one and the
git one is just something to play around with. When everyone is ready,
I'll make the hg ones read-only and do one last pull/sync from there.

As for the long-term history of the hg ones, I could probably migrate
them to boxen and sync them via cron, but I don't know if it's really
worth keeping them alive. Probably depends on the effort, which I'm
hoping will be minimal. (We'll probably at least keep them up until
we've fully migrated the issues/wiki, as there are still links into
it.) We'll also need to redirect hudson.

- Robert

From dalcinl at gmail.com  Tue Nov 23 19:34:27 2010
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Tue, 23 Nov 2010 15:34:27 -0300
Subject: [Cython] [cython-users] complex type typedef problem in
	cython-generated C code
In-Reply-To: <d8e7937b-baf0-4cd4-a0f9-5294d721fadc@a37g2000yqi.googlegroups.com>
References: <d8e7937b-baf0-4cd4-a0f9-5294d721fadc@a37g2000yqi.googlegroups.com>
Message-ID: <AANLkTin+iwP+N06xxtCQM4Fx-mxFZQ0ouuHY_qpLpXaY@mail.gmail.com>

On 21 November 2010 08:49, cornail <kornel.jahn at gmail.com> wrote:
> Hello,
>
> I tried to compile cython pyx file using distutils but gcc didn't like
> the resulting C code (no error message from Cython).
> It gave the following error message:
>
> running build_ext
> building 'vector_slepian_ext_cython' extension
> gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -
> Wstrict-prototypes -fPIC -I/usr/lib/python2.6/dist-packages/numpy/core/
> include -I/usr/local/include/ -I. -I/usr/include/python2.6 -c
> vector_slepian_ext_cython.c -o build/temp.linux-x86_64-2.6/
> vector_slepian_ext_cython.o -ffast-math
> vector_slepian_ext_cython.c:362: error: two or more data types in
> declaration specifiers
> error: command 'gcc' failed with exit status 1
>
> The corresponding C code fragment is:
>
> #if CYTHON_CCOMPLEX
> ?#ifdef __cplusplus
> ? ?typedef ::std::complex< npy_float64 > __pyx_t_npy_float64_complex;
> ?#else
> ? ?typedef npy_float64 _Complex __pyx_t_npy_float64_complex; /* THIS
> IS THE PROBLEMATIC LINE 362 */
> ?#endif
> #else
> ? ?typedef struct { npy_float64 real, imag; }
> __pyx_t_npy_float64_complex;
> #endif
>
> In my pyx file, I both import and cimport numpy, and use both "double
> complex" type for calculations and the np.complex128_t type for
> ndarrays.
>
> Version information:
> Python 2.6.6
> Cython 0.13
> Numpy 1.3.0
> GCC (Ubuntu/Linaro 4.4.4-14ubuntu5) 4.4.5
>
> Tell me please, if you need some more information. I would appreciate,
> if someone could solve this for me or suggest a workaround.
>
> Thank you very much in advance!
>
> Kornel JAHN
>

It seems that we are generating bad code.

As a quick workaround, could you use "double" instead of "npy_float64" ?

Robert, any idea of how could we fix this? It seems that we cannot
support arbitrary typedefs for the real type...




-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169

From stefan_ml at behnel.de  Tue Nov 23 23:22:30 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 23 Nov 2010 23:22:30 +0100
Subject: [Cython] >=python-2.6 property pattern not supported
In-Reply-To: <4CDDAC83.9030903@behnel.de>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>	<4CD811B2.1010108@behnel.de>	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>	<4CDCE8D3.70002@behnel.de>	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>	<4CDCF489.3000108@behnel.de>	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>
	<4CDDAC83.9030903@behnel.de>
Message-ID: <4CEC3EA6.9040002@behnel.de>

Stefan Behnel, 12.11.2010 22:07:
> Vitja Makarov, 12.11.2010 16:28:
>> 2010/11/12 Stefan Behnel:
>>> you simply cannot currently define a function more than once within
>>> a namespace because it is looked up statically. So, when you redefine it,
>>> its Python name would get overwritten in the symbol table. So you couldn't
>>> refer to the previously defined function anymore from that point on, which
>>> means that you cannot even use it in the code that gets executed before the
>>> redefinition. The symbol table is not dynamically adaptive to the position
>>> in the code, it's just a plain table that is global to a namespace.
>>>
>>> This is a compiler, not an interpreter. Things don't simply happen in the
>>> order they are executed in the code.
>>>
>>> Does that explain it?
>>
>> Can you show me where is static lookup performed?
>
> See the Entry class in Symtab.py, it holds the information about declared
> names and their types.
>
>> Is this cython level lookup or C-level?
>
> Cython. Mostly during type analysis, but potentially also in the type
> declaration phase and other places.
>
>> Can you show me example?
>
> You will find tons of places that call "lookup()" (recursively) or
> "lookup_here()" (non-recusively) on the current scope. Take a look at the
> three call node classes, for example.
>
>> Btw, if we see that function is redefined we can mark it as
>> "overrided" and use special more dynamic rules to handle that (create
>> it in runtime,  don't declare it in methods list and so on)
>
> Sure. I didn't say we lack solutions. It just needs to be done. And this
> isn't trivial enough to be done within a couple of days, so no-one has
> started working on it.

I took a deeper look and came to the conclusion that it will be easier to 
do this at the same time as the split of Python functions into a Python 
variable name, a Python function wrapper and a C function.

The trivial hack to allow name redefinitions almost works when using a 
unique cname. However, the problem is that the symtab entry of the function 
node is the same for the function definition and the Python variable. 
Ideally, the function definition entry shouldn't be stored in the symtab 
dict at all, and the name should be declared independently with a simple 
variable entry of py_object_type (unless already declared, in which case 
the variable entry can be reused but not the declaration entry).

Stefan

From stefan_ml at behnel.de  Wed Nov 24 09:02:57 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 24 Nov 2010 09:02:57 +0100
Subject: [Cython] GitHub for Cython
In-Reply-To: <AANLkTim6=fzV0JG=GWPJxoB82O+ZqmNY+QDn+8B2aaXX@mail.gmail.com>
References: <AANLkTinFLOtcHvdUj-9SnX=7uZELoH9W74zRfy37df=z@mail.gmail.com>	<AANLkTinVrx3vqE9Nx7Rnj6bo4kvf+BVjNV3Y6Tqk_HCi@mail.gmail.com>	<AANLkTinSF-tjR87jYsARVYOebQ3cT7eJkupn8XvX+rcP@mail.gmail.com>	<4CEBEA6E.6050106@behnel.de>
	<AANLkTim6=fzV0JG=GWPJxoB82O+ZqmNY+QDn+8B2aaXX@mail.gmail.com>
Message-ID: <4CECC6B1.7090106@behnel.de>

Robert Bradshaw, 23.11.2010 18:42:
> I would say for the moment the hg repo is still the "live" one and the
> git one is just something to play around with. When everyone is ready,
> I'll make the hg ones read-only and do one last pull/sync from there.

Ok.


> We'll also need to redirect hudson.

I already installed the git plugin, so, theoretically, all we have to do is 
change the VCS setup for the cython-devel-sdist job when we do the final 
switch. (well, and reconfigure the cython-release-* jobs at some point)

Stefan

From stefan_ml at behnel.de  Wed Nov 24 09:22:08 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 24 Nov 2010 09:22:08 +0100
Subject: [Cython] results of CPython regression tests
In-Reply-To: <4CEB8AA8.70108@behnel.de>
References: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>	<AANLkTi=bWs0X8peV0fQQeQx4_okvduQue_f5nJD75Nta@mail.gmail.com>	<4CE943DA.4010703@behnel.de>	<AANLkTikAb6nr0LhBDzf68DW-uuTUJhMRgQwhPhFeKYeU@mail.gmail.com>
	<4CEB8AA8.70108@behnel.de>
Message-ID: <4CECCB30.7020702@behnel.de>

Stefan Behnel, 23.11.2010 10:34:
> There also seems to be a problem with the test runner that makes the
> py3regr tests hang (with --no-fork or XML result output). Apparently,
> certain test failures leak into subsequent test runs. Running the two tests
> "test_compileall" and "test_gzip" hangs reliable for me in py3k, after a
> compiler crash in the first. The weird thing about it is that the gzip test
> doesn't normally compile because it uses a genexpr in combination with
> "sorted()", but it compiles without errors after the crash in the
> compileall test. I faintly remember that we have a way to disable
> termination on compile errors, maybe that's not getting properly cleaned up
> here.

I don't currently have much time to debug this, so in case someone wants to 
throw in another pair of eyes, the following reliably fails for me in the 
latest py3k:

python3.2 runtests.py -3 --no-cleanup -vv --sys-pyregr --no-fork \
           --no-file --no-unit --no-doctest --no-cpp 'test_(dict|dis)$'

Both tests should fail in the compiler ("generators not supported"), but 
when test_dict is run before test_dis, the second fails in gcc.

Stefan

From nahuel.defosse at gmail.com  Wed Nov 24 13:20:33 2010
From: nahuel.defosse at gmail.com (=?ISO-8859-1?Q?Nahuel_Defoss=E9?=)
Date: Wed, 24 Nov 2010 09:20:33 -0300
Subject: [Cython] Passing char pointers to a function
Message-ID: <AANLkTin+MQvLzqS+s347=Djuk_WNrwk0FZS7=kvc2=Ra@mail.gmail.com>

Hi everyone,

I want to call a C function which accepts 2 char * pointers, like this:

int my_function( char *start, char *end);

These pointers hold the start and end position of a long string which
has to be splited.

Then, there's another function which will return slices of a initial call to
my_funciton, ala strtok.

int my_iterate( char **start, char **end);

How do I define a Python class which can hold this strucutre, and iterate
over it? I can't figure out how to handle this kind of pointers in Cython.

Thanks

From stefan_ml at behnel.de  Wed Nov 24 13:24:33 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 24 Nov 2010 13:24:33 +0100
Subject: [Cython] Passing char pointers to a function
In-Reply-To: <AANLkTin+MQvLzqS+s347=Djuk_WNrwk0FZS7=kvc2=Ra@mail.gmail.com>
References: <AANLkTin+MQvLzqS+s347=Djuk_WNrwk0FZS7=kvc2=Ra@mail.gmail.com>
Message-ID: <4CED0401.608@behnel.de>

Nahuel Defoss?, 24.11.2010 13:20:
> I want to call a C function which accepts 2 char * pointers, like this:
>
> int my_function( char *start, char *end);
>
> These pointers hold the start and end position of a long string which
> has to be splited.
>
> Then, there's another function which will return slices of a initial call to
> my_funciton, ala strtok.
>
> int my_iterate( char **start, char **end);
>
> How do I define a Python class which can hold this strucutre, and iterate
> over it? I can't figure out how to handle this kind of pointers in Cython.

Hmm, wrong list again?

Stefan

From nahuel.defosse at gmail.com  Wed Nov 24 13:26:09 2010
From: nahuel.defosse at gmail.com (=?ISO-8859-1?Q?Nahuel_Defoss=E9?=)
Date: Wed, 24 Nov 2010 09:26:09 -0300
Subject: [Cython] Passing char pointers to a function
In-Reply-To: <4CED0401.608@behnel.de>
References: <AANLkTin+MQvLzqS+s347=Djuk_WNrwk0FZS7=kvc2=Ra@mail.gmail.com>
	<4CED0401.608@behnel.de>
Message-ID: <AANLkTi=eiaW5+DrbZcMP40U=ZHnV9SHDCXmKBtMYXCV2@mail.gmail.com>

Oh, I haven't updated my address book, sorry.
It won't happen again.

2010/11/24 Stefan Behnel <stefan_ml at behnel.de>:
> Nahuel Defoss?, 24.11.2010 13:20:
>> I want to call a C function which accepts 2 char * pointers, like this:
>>
>> int my_function( char *start, char *end);
>>
>> These pointers hold the start and end position of a long string which
>> has to be splited.
>>
>> Then, there's another function which will return slices of a initial call to
>> my_funciton, ala strtok.
>>
>> int my_iterate( char **start, char **end);
>>
>> How do I define a Python class which can hold this strucutre, and iterate
>> over it? I can't figure out how to handle this kind of pointers in Cython.
>
> Hmm, wrong list again?
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

From stefan_ml at behnel.de  Wed Nov 24 22:05:47 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 24 Nov 2010 22:05:47 +0100
Subject: [Cython] results of CPython regression tests
In-Reply-To: <4CECCB30.7020702@behnel.de>
References: <AANLkTik6BHGamLMSo1PFaGj_y7TF5d4tUyF7Syh5UcuS@mail.gmail.com>	<AANLkTi=bWs0X8peV0fQQeQx4_okvduQue_f5nJD75Nta@mail.gmail.com>	<4CE943DA.4010703@behnel.de>	<AANLkTikAb6nr0LhBDzf68DW-uuTUJhMRgQwhPhFeKYeU@mail.gmail.com>	<4CEB8AA8.70108@behnel.de>
	<4CECCB30.7020702@behnel.de>
Message-ID: <4CED7E2B.1050405@behnel.de>

Stefan Behnel, 24.11.2010 09:22:
> Stefan Behnel, 23.11.2010 10:34:
>> There also seems to be a problem with the test runner that makes the
>> py3regr tests hang (with --no-fork or XML result output). Apparently,
>> certain test failures leak into subsequent test runs. Running the two tests
>> "test_compileall" and "test_gzip" hangs reliable for me in py3k, after a
>> compiler crash in the first. The weird thing about it is that the gzip test
>> doesn't normally compile because it uses a genexpr in combination with
>> "sorted()", but it compiles without errors after the crash in the
>> compileall test. I faintly remember that we have a way to disable
>> termination on compile errors, maybe that's not getting properly cleaned up
>> here.
>
> I don't currently have much time to debug this, so in case someone wants to
> throw in another pair of eyes, the following reliably fails for me in the
> latest py3k:
>
> python3.2 runtests.py -3 --no-cleanup -vv --sys-pyregr --no-fork \
>             --no-file --no-unit --no-doctest --no-cpp 'test_(dict|dis)$'
>
> Both tests should fail in the compiler ("generators not supported"), but
> when test_dict is run before test_dis, the second fails in gcc.

I found a bit of time to fix this. The pyregr tests in py3k are runnable 
again and we are up from ~2000 to ~3000 tests in total. Since the 
compilation now properly fails for generators and genexprs, the test count 
in Py2.7 went down from ~4600 to ~3000 tests as well.

https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py3k-c/test/?width=800&height=600

https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py27-c/test/?width=800&height=600

I still think that the test results are worth looking at. There may be some 
low hanging fruit left for 0.13.1.

Stefan

From vitja.makarov at gmail.com  Thu Nov 25 13:39:22 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 25 Nov 2010 15:39:22 +0300
Subject: [Cython] Closure optimization
Message-ID: <AANLkTinog+AYj36G_RaoSUJm7116HPLSJ+YJQztkYvMU@mail.gmail.com>

Hi, I've tried to optimize closures.

I think I should create ticket for this.

Closure optimizations:
 * Don't add entries to closure if they will not be later used
 * Don't add outer scope to closure if it will not be used
 * Don't pass closure scope to functions that don't use it
 * Don't create empty closure classes.
 * If closure class contains outer scope only pass outer scope
directly (called is_passthrought)

Please review attached patch.

vitja.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: closure-optimization.patch
Type: text/x-patch
Size: 16801 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20101125/8ae90790/attachment.bin 

From stefan_ml at behnel.de  Thu Nov 25 14:49:11 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 25 Nov 2010 14:49:11 +0100
Subject: [Cython] Closure optimization
In-Reply-To: <AANLkTinog+AYj36G_RaoSUJm7116HPLSJ+YJQztkYvMU@mail.gmail.com>
References: <AANLkTinog+AYj36G_RaoSUJm7116HPLSJ+YJQztkYvMU@mail.gmail.com>
Message-ID: <4CEE6957.301@behnel.de>

Vitja Makarov, 25.11.2010 13:39:
> Hi, I've tried to optimize closures.
>
> I think I should create ticket for this.

Yes, please do. There doesn't seem to be one yet.


> Closure optimizations:
>   * Don't add entries to closure if they will not be later used
>   * Don't add outer scope to closure if it will not be used
>   * Don't pass closure scope to functions that don't use it
>   * Don't create empty closure classes.
>   * If closure class contains outer scope only pass outer scope
> directly (called is_passthrought)
>
> Please review attached patch.

I took a quick look through it and it looks good. The tests still pass, 
although I'm not very confident in the closure tests that we have, given 
that there aren't many that involve typed closure variables or type 
inference. But at least those that are there didn't break.

Anyway, I think the patch is good enough to push it on to Hudson to see 
what it has to say about it.

Thanks!

Stefan

From stefan_ml at behnel.de  Thu Nov 25 15:30:59 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 25 Nov 2010 15:30:59 +0100
Subject: [Cython] Closure optimization
In-Reply-To: <4CEE6957.301@behnel.de>
References: <AANLkTinog+AYj36G_RaoSUJm7116HPLSJ+YJQztkYvMU@mail.gmail.com>
	<4CEE6957.301@behnel.de>
Message-ID: <4CEE7323.8030000@behnel.de>

Stefan Behnel, 25.11.2010 14:49:
> Vitja Makarov, 25.11.2010 13:39:
>> Hi, I've tried to optimize closures.
>>
>> I think I should create ticket for this.
>
> Yes, please do. There doesn't seem to be one yet.
>
>
>> Closure optimizations:
>>    * Don't add entries to closure if they will not be later used
>>    * Don't add outer scope to closure if it will not be used
>>    * Don't pass closure scope to functions that don't use it
>>    * Don't create empty closure classes.
>>    * If closure class contains outer scope only pass outer scope
>> directly (called is_passthrought)
>>
>> Please review attached patch.
>
> I took a quick look through it and it looks good. The tests still pass,
> although I'm not very confident in the closure tests that we have, given
> that there aren't many that involve typed closure variables or type
> inference. But at least those that are there didn't break.
>
> Anyway, I think the patch is good enough to push it on to Hudson to see
> what it has to say about it.

Hudson says: no degradation at all. :)

Thanks again, this is a huge improvement.

Stefan

From vitja.makarov at gmail.com  Thu Nov 25 15:31:30 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Thu, 25 Nov 2010 17:31:30 +0300
Subject: [Cython] Closure optimization
In-Reply-To: <4CEE6957.301@behnel.de>
References: <AANLkTinog+AYj36G_RaoSUJm7116HPLSJ+YJQztkYvMU@mail.gmail.com>
	<4CEE6957.301@behnel.de>
Message-ID: <AANLkTimEX9GSwAkKdZbPEh2UOdfV8RQ8+Sad0GzUx2N_@mail.gmail.com>

2010/11/25 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 25.11.2010 13:39:
>> Hi, I've tried to optimize closures.
>>
>> I think I should create ticket for this.
>
> Yes, please do. There doesn't seem to be one yet.
>

Ticket is now here http://trac.cython.org/cython_trac/ticket/599

>
>> Closure optimizations:
>> ? * Don't add entries to closure if they will not be later used
>> ? * Don't add outer scope to closure if it will not be used
>> ? * Don't pass closure scope to functions that don't use it
>> ? * Don't create empty closure classes.
>> ? * If closure class contains outer scope only pass outer scope
>> directly (called is_passthrought)
>>
>> Please review attached patch.
>
> I took a quick look through it and it looks good. The tests still pass,
> although I'm not very confident in the closure tests that we have, given
> that there aren't many that involve typed closure variables or type
> inference. But at least those that are there didn't break.
>

Closure tests helped very much to implement all this stuff.
Type inference should be okay because it's already done for inner entry.
And closure class entries are created with defined type.

I'm not sure (about my PC load) but is seems that whole pyregr test
takes a little bit less time.

> Anyway, I think the patch is good enough to push it on to Hudson to see
> what it has to say about it.
>
> Thanks!
>
> Stefan

Interesting to see.

-- 
vitja.

From stefan_ml at behnel.de  Thu Nov 25 17:01:24 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 25 Nov 2010 17:01:24 +0100
Subject: [Cython] Closure optimization
In-Reply-To: <AANLkTimEX9GSwAkKdZbPEh2UOdfV8RQ8+Sad0GzUx2N_@mail.gmail.com>
References: <AANLkTinog+AYj36G_RaoSUJm7116HPLSJ+YJQztkYvMU@mail.gmail.com>	<4CEE6957.301@behnel.de>
	<AANLkTimEX9GSwAkKdZbPEh2UOdfV8RQ8+Sad0GzUx2N_@mail.gmail.com>
Message-ID: <4CEE8854.3030107@behnel.de>

Vitja Makarov, 25.11.2010 15:31:
> Closure tests helped very much to implement all this stuff.

I bet they did.


> I'm not sure (about my PC load) but is seems that whole pyregr test
> takes a little bit less time.

Here's a little artificial benchmark:

     def deep_inner():
         cdef int x = 1
         def f():
             def g():
                 def h():
                     return x+1
                 return h
             return g()
         return f()

** Python 2.7:

Before your patch:

$ python -m timeit -s 'from closure_bench import deep_inner as f' 'f()'
1000000 loops, best of 3: 0.313 usec per loop
$ python -m timeit -s 'from closure_bench import deep_inner as f' 'f()()'
1000000 loops, best of 3: 0.348 usec per loop

After your patch:

$ python -m timeit -s 'from closure_bench import deep_inner as f' 'f()'
1000000 loops, best of 3: 0.234 usec per loop
$ python -m timeit -s 'from closure_bench import deep_inner as f' 'f()()'
1000000 loops, best of 3: 0.266 usec per loop

** Python 3.2:

Before your patch:

$ python3 -m timeit -s 'from closure_bench import deep_inner as f' 'f()'
1000000 loops, best of 3: 0.299 usec per loop
$ python3 -m timeit -s 'from closure_bench import deep_inner as f' 'f()()'
1000000 loops, best of 3: 0.336 usec per loop

After your patch:

$ python3 -m timeit -s 'from closure_bench import deep_inner as f' 'f()'
1000000 loops, best of 3: 0.204 usec per loop
$ python3 -m timeit -s 'from closure_bench import deep_inner as f' 'f()()'
1000000 loops, best of 3: 0.247 usec per loop


That's a lot better, I'd say.

Stefan

From mhansen at gmail.com  Thu Nov 25 22:16:03 2010
From: mhansen at gmail.com (Mike Hansen)
Date: Thu, 25 Nov 2010 13:16:03 -0800
Subject: [Cython] Bug with ExprNodes.CoerceFromPyTypeNode
Message-ID: <AANLkTinxXTFgGarO0S03WZkqVdQJEfbB0EM9RPTSPtP3@mail.gmail.com>

Hello,

I'm trying to build a Cython extension and get the following traceback

  ...
  File "/opt/sage/local/lib/python2.6/site-packages/Cython/Compiler/Nodes.py",
line 4499, in generate_execution_code
    from_py_node =
ExprNodes.CoerceFromPyTypeNode(self.loopvar_node.type, target_node,
None)
  File "/opt/sage/local/lib/python2.6/site-packages/Cython/Compiler/ExprNodes.py",
line 6634, in __init__
    if not result_type.create_from_py_utility_code(env):
  File "/opt/sage/local/lib/python2.6/site-packages/Cython/Compiler/PyrexTypes.py",
line 279, in create_from_py_utility_code
    env.use_utility_code(self.from_py_utility_code)
AttributeError: 'NoneType' object has no attribute 'use_utility_code'

What should be passed into ExprNodes.CoerceFromPyTypeNode for env
instead of None?

--Mike

From stefan_ml at behnel.de  Thu Nov 25 22:30:42 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 25 Nov 2010 22:30:42 +0100
Subject: [Cython] Bug with ExprNodes.CoerceFromPyTypeNode
In-Reply-To: <AANLkTinxXTFgGarO0S03WZkqVdQJEfbB0EM9RPTSPtP3@mail.gmail.com>
References: <AANLkTinxXTFgGarO0S03WZkqVdQJEfbB0EM9RPTSPtP3@mail.gmail.com>
Message-ID: <4CEED582.9090808@behnel.de>

Mike Hansen, 25.11.2010 22:16:
> I'm trying to build a Cython extension and get the following traceback
>
>    ...
>    File "/opt/sage/local/lib/python2.6/site-packages/Cython/Compiler/Nodes.py",
> line 4499, in generate_execution_code
>      from_py_node =
> ExprNodes.CoerceFromPyTypeNode(self.loopvar_node.type, target_node,
> None)
>    File "/opt/sage/local/lib/python2.6/site-packages/Cython/Compiler/ExprNodes.py",
> line 6634, in __init__
>      if not result_type.create_from_py_utility_code(env):
>    File "/opt/sage/local/lib/python2.6/site-packages/Cython/Compiler/PyrexTypes.py",
> line 279, in create_from_py_utility_code
>      env.use_utility_code(self.from_py_utility_code)
> AttributeError: 'NoneType' object has no attribute 'use_utility_code'
>
> What should be passed into ExprNodes.CoerceFromPyTypeNode for env
> instead of None?

Wow, thanks for catching that. There's a bunch of evil code there that 
seriously needs to get ripped out.

As a (very ugly) work-around, something like 
"self.loopvar_node.entry.scope" *might* work instead of "None".

Stefan

From stefan_ml at behnel.de  Thu Nov 25 22:35:24 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 25 Nov 2010 22:35:24 +0100
Subject: [Cython] Bug with ExprNodes.CoerceFromPyTypeNode
In-Reply-To: <4CEED582.9090808@behnel.de>
References: <AANLkTinxXTFgGarO0S03WZkqVdQJEfbB0EM9RPTSPtP3@mail.gmail.com>
	<4CEED582.9090808@behnel.de>
Message-ID: <4CEED69C.6010208@behnel.de>

Stefan Behnel, 25.11.2010 22:30:
> Mike Hansen, 25.11.2010 22:16:
>> I'm trying to build a Cython extension and get the following traceback
>>
>>     ...
>>     File "/opt/sage/local/lib/python2.6/site-packages/Cython/Compiler/Nodes.py",
>> line 4499, in generate_execution_code
>>       from_py_node =
>> ExprNodes.CoerceFromPyTypeNode(self.loopvar_node.type, target_node,
>> None)
>>     File "/opt/sage/local/lib/python2.6/site-packages/Cython/Compiler/ExprNodes.py",
>> line 6634, in __init__
>>       if not result_type.create_from_py_utility_code(env):
>>     File "/opt/sage/local/lib/python2.6/site-packages/Cython/Compiler/PyrexTypes.py",
>> line 279, in create_from_py_utility_code
>>       env.use_utility_code(self.from_py_utility_code)
>> AttributeError: 'NoneType' object has no attribute 'use_utility_code'
>>
>> What should be passed into ExprNodes.CoerceFromPyTypeNode for env
>> instead of None?
>
> Wow, thanks for catching that. There's a bunch of evil code there that
> seriously needs to get ripped out.

http://trac.cython.org/cython_trac/ticket/601

Stefan

From vitja.makarov at gmail.com  Sat Nov 27 08:24:56 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 27 Nov 2010 10:24:56 +0300
Subject: [Cython] >=python-2.6 property pattern not supported
In-Reply-To: <4CEC3EA6.9040002@behnel.de>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>
	<4CD811B2.1010108@behnel.de>
	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>
	<4CDCE8D3.70002@behnel.de>
	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>
	<4CDCF489.3000108@behnel.de>
	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>
	<4CDDAC83.9030903@behnel.de> <4CEC3EA6.9040002@behnel.de>
Message-ID: <AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>

2010/11/24 Stefan Behnel <stefan_ml at behnel.de>:
> Stefan Behnel, 12.11.2010 22:07:
>> Vitja Makarov, 12.11.2010 16:28:
>>> 2010/11/12 Stefan Behnel:
>>>> you simply cannot currently define a function more than once within
>>>> a namespace because it is looked up statically. So, when you redefine it,
>>>> its Python name would get overwritten in the symbol table. So you couldn't
>>>> refer to the previously defined function anymore from that point on, which
>>>> means that you cannot even use it in the code that gets executed before the
>>>> redefinition. The symbol table is not dynamically adaptive to the position
>>>> in the code, it's just a plain table that is global to a namespace.
>>>>
>>>> This is a compiler, not an interpreter. Things don't simply happen in the
>>>> order they are executed in the code.
>>>>
>>>> Does that explain it?
>>>
>>> Can you show me where is static lookup performed?
>>
>> See the Entry class in Symtab.py, it holds the information about declared
>> names and their types.
>>
>>> Is this cython level lookup or C-level?
>>
>> Cython. Mostly during type analysis, but potentially also in the type
>> declaration phase and other places.
>>
>>> Can you show me example?
>>
>> You will find tons of places that call "lookup()" (recursively) or
>> "lookup_here()" (non-recusively) on the current scope. Take a look at the
>> three call node classes, for example.
>>
>>> Btw, if we see that function is redefined we can mark it as
>>> "overrided" and use special more dynamic rules to handle that (create
>>> it in runtime, ?don't declare it in methods list and so on)
>>
>> Sure. I didn't say we lack solutions. It just needs to be done. And this
>> isn't trivial enough to be done within a couple of days, so no-one has
>> started working on it.
>
> I took a deeper look and came to the conclusion that it will be easier to
> do this at the same time as the split of Python functions into a Python
> variable name, a Python function wrapper and a C function.
>

How does that work? What is Python function wrapper?
Is that described in generators CEP?

> The trivial hack to allow name redefinitions almost works when using a
> unique cname. However, the problem is that the symtab entry of the function
> node is the same for the function definition and the Python variable.
> Ideally, the function definition entry shouldn't be stored in the symtab
> dict at all, and the name should be declared independently with a simple
> variable entry of py_object_type (unless already declared, in which case
> the variable entry can be reused but not the declaration entry).
>
> Stefan

I'm now trying to make "module scope lambda" work.
That's almost done.

I see that needs_assignment_synthesis() should be forced to create
PyMethodDef struct.
Now I do it this way:
def needs_assignment_synthesis(self, ...):
.....
             if self.name == '<lambda>':
.....

Seems like a dirty hack.

Also lambda function is defined in module methods table as
__pyx_lambda_funcdef4lala_lambda1,
Is it safe not to add lambda cname to pyfunc_entries in
Scope.declare_lambda_function()?

-- 
vitja.

From stefan_ml at behnel.de  Sat Nov 27 09:00:31 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 27 Nov 2010 09:00:31 +0100
Subject: [Cython] splitting up Python functions (was: >=python-2.6
 property pattern not supported)
In-Reply-To: <AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>	<4CD811B2.1010108@behnel.de>	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>	<4CDCE8D3.70002@behnel.de>	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>	<4CDCF489.3000108@behnel.de>	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>	<4CDDAC83.9030903@behnel.de>
	<4CEC3EA6.9040002@behnel.de>
	<AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>
Message-ID: <4CF0BA9F.9060304@behnel.de>

Vitja Makarov, 27.11.2010 08:24:
> 2010/11/24 Stefan Behnel:
>>>>> you simply cannot currently define a function more than once within
>>>>> a namespace because it is looked up statically. So, when you redefine it,
>>>>> its Python name would get overwritten in the symbol table. So you couldn't
>>>>> refer to the previously defined function anymore from that point on, which
>>>>> means that you cannot even use it in the code that gets executed before the
>>>>> redefinition. The symbol table is not dynamically adaptive to the position
>>>>> in the code, it's just a plain table that is global to a namespace.
>>
>> I took a deeper look and came to the conclusion that it will be easier to
>> do this at the same time as the split of Python functions into a Python
>> variable name, a Python function wrapper and a C function.
>
> How does that work? What is Python function wrapper?
> Is that described in generators CEP?

Yes, it's described in that CEP. Currently, cpdef functions create a 
DefNode internally. The idea is to actually represent *all* Python 
functions (and methods) as a C function with the correct signature, and a 
Python wrapper function that does the argument unpacking and return value 
conversion. There shouldn't be a performance difference as the C compiler 
can inline the C function into the wrapper if it thinks it's worth it. 
However, this has several advantages:

* it moves the lengthy argument unpacking and error handling code out of 
the actual function body, thus making the C code more readable.

* it restricts the inner C function code to what is really needed 
(including temp variables, visible arguments and exit points), which may 
make it easier for the C compiler to optimise it.

* it makes the inner C function directly callable, thus making it easier to 
implement things like cpdef functions and generators on top.

* it (likely) simplifies the code in the compiler, as the DefNode will get 
stripped down to handling the argument unpacking and Python name assignment 
(and maybe some glue stuff), and can delegate everything else to a plain 
CFuncDefNode.

This has been on the list for a while now, but no-one has started any 
serious work.

Stefan

From stefan_ml at behnel.de  Sat Nov 27 09:10:06 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 27 Nov 2010 09:10:06 +0100
Subject: [Cython] module level lambda functions (was: >=python-2.6
 property pattern not supported)
In-Reply-To: <AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>	<4CD811B2.1010108@behnel.de>	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>	<4CDCE8D3.70002@behnel.de>	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>	<4CDCF489.3000108@behnel.de>	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>	<4CDDAC83.9030903@behnel.de>
	<4CEC3EA6.9040002@behnel.de>
	<AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>
Message-ID: <4CF0BCDE.1020407@behnel.de>

Hi,

please start a new thread when switching to a new topic.

Vitja Makarov, 27.11.2010 08:24:
> I'm now trying to make "module scope lambda" work.
> That's almost done.
>
> I see that needs_assignment_synthesis() should be forced to create
> PyMethodDef struct.

I would be happier if we could get rid of PyMethodDef in the long term, but 
it's clearly easier to keep it for now.


> Now I do it this way:
> def needs_assignment_synthesis(self, ...):
> .....
>               if self.name == '<lambda>':
> .....
>
> Seems like a dirty hack.

Yes. Instead, use a new flag "is_lambda" on the entry.


> Also lambda function is defined in module methods table as
> __pyx_lambda_funcdef4lala_lambda1,
> Is it safe not to add lambda cname to pyfunc_entries in
> Scope.declare_lambda_function()?

I don't know, you can try. In any case, it needs to do everything that a 
normal Python function does, except for adding its Python name to the 
defining scope. That's a rather small distinction.

Stefan

From vitja.makarov at gmail.com  Sat Nov 27 09:17:58 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 27 Nov 2010 11:17:58 +0300
Subject: [Cython] module level lambda functions (was: >=python-2.6
 property pattern not supported)
In-Reply-To: <4CF0BCDE.1020407@behnel.de>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>
	<4CD811B2.1010108@behnel.de>
	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>
	<4CDCE8D3.70002@behnel.de>
	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>
	<4CDCF489.3000108@behnel.de>
	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>
	<4CDDAC83.9030903@behnel.de> <4CEC3EA6.9040002@behnel.de>
	<AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>
	<4CF0BCDE.1020407@behnel.de>
Message-ID: <AANLkTi=XkcQwgBnrzjEn=3O42-t4RVgk8Ty+zcA_G+gB@mail.gmail.com>

2010/11/27 Stefan Behnel <stefan_ml at behnel.de>:
> Hi,
>
> please start a new thread when switching to a new topic.
>
> Vitja Makarov, 27.11.2010 08:24:
>> I'm now trying to make "module scope lambda" work.
>> That's almost done.
>>
>> I see that needs_assignment_synthesis() should be forced to create
>> PyMethodDef struct.
>
> I would be happier if we could get rid of PyMethodDef in the long term, but
> it's clearly easier to keep it for now.
>
>
>> Now I do it this way:
>> def needs_assignment_synthesis(self, ...):
>> .....
>> ? ? ? ? ? ? ? if self.name == '<lambda>':
>> .....
>>
>> Seems like a dirty hack.
>
> Yes. Instead, use a new flag "is_lambda" on the entry.
>

This check is performed in DefNode so I prefer to call it
force_assignment_synthesis it could be used later for module level
functions.

Or do you mean that "is_lambda" should be set after
declare_lambda_function() and then checked in DefNode?

>
>> Also lambda function is defined in module methods table as
>> __pyx_lambda_funcdef4lala_lambda1,
>> Is it safe not to add lambda cname to pyfunc_entries in
>> Scope.declare_lambda_function()?
>
> I don't know, you can try. In any case, it needs to do everything that a
> normal Python function does, except for adding its Python name to the
> defining scope. That's a rather small distinction.
>
> Stefan

Seems to work fine. No regressions.

-- 
vitja.

From stefan_ml at behnel.de  Sat Nov 27 09:26:40 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 27 Nov 2010 09:26:40 +0100
Subject: [Cython] module level lambda functions
In-Reply-To: <AANLkTi=XkcQwgBnrzjEn=3O42-t4RVgk8Ty+zcA_G+gB@mail.gmail.com>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>	<4CD811B2.1010108@behnel.de>	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>	<4CDCE8D3.70002@behnel.de>	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>	<4CDCF489.3000108@behnel.de>	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>	<4CDDAC83.9030903@behnel.de>
	<4CEC3EA6.9040002@behnel.de>	<AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>	<4CF0BCDE.1020407@behnel.de>
	<AANLkTi=XkcQwgBnrzjEn=3O42-t4RVgk8Ty+zcA_G+gB@mail.gmail.com>
Message-ID: <4CF0C0C0.2050702@behnel.de>

Vitja Makarov, 27.11.2010 09:17:
> 2010/11/27 Stefan Behnel<stefan_ml at behnel.de>:
>> Hi,
>>
>> please start a new thread when switching to a new topic.
>>
>> Vitja Makarov, 27.11.2010 08:24:
>>> I'm now trying to make "module scope lambda" work.
>>> That's almost done.
>>>
>>> I see that needs_assignment_synthesis() should be forced to create
>>> PyMethodDef struct.
>>
>> I would be happier if we could get rid of PyMethodDef in the long term, but
>> it's clearly easier to keep it for now.
>>
>>
>>> Now I do it this way:
>>> def needs_assignment_synthesis(self, ...):
>>> .....
>>>                if self.name == '<lambda>':
>>> .....
>>>
>>> Seems like a dirty hack.
>>
>> Yes. Instead, use a new flag "is_lambda" on the entry.
>
> This check is performed in DefNode so I prefer to call it
> force_assignment_synthesis it could be used later for module level
> functions.
>
> Or do you mean that "is_lambda" should be set after
> declare_lambda_function() and then checked in DefNode?

Yes. As I said, the difference is small, so a flag and a couple of checks 
in the right places will do.


>>> Also lambda function is defined in module methods table as
>>> __pyx_lambda_funcdef4lala_lambda1,
>>> Is it safe not to add lambda cname to pyfunc_entries in
>>> Scope.declare_lambda_function()?
>>
>> I don't know, you can try. In any case, it needs to do everything that a
>> normal Python function does, except for adding its Python name to the
>> defining scope. That's a rather small distinction.
>
> Seems to work fine. No regressions.

Ok. IIRC, lambda functions are already kept in a different list. I wonder 
if that was a good idea. Maybe a flag right on the DefNode would have been 
a better way to do it.

Stefan

From vitja.makarov at gmail.com  Sat Nov 27 09:28:06 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 27 Nov 2010 11:28:06 +0300
Subject: [Cython] module level lambda functions (was: >=python-2.6
 property pattern not supported)
In-Reply-To: <AANLkTi=XkcQwgBnrzjEn=3O42-t4RVgk8Ty+zcA_G+gB@mail.gmail.com>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>
	<4CD811B2.1010108@behnel.de>
	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>
	<4CDCE8D3.70002@behnel.de>
	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>
	<4CDCF489.3000108@behnel.de>
	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>
	<4CDDAC83.9030903@behnel.de> <4CEC3EA6.9040002@behnel.de>
	<AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>
	<4CF0BCDE.1020407@behnel.de>
	<AANLkTi=XkcQwgBnrzjEn=3O42-t4RVgk8Ty+zcA_G+gB@mail.gmail.com>
Message-ID: <AANLkTi=TFj8v_wOOCiQrJA2iGuBmWtVDDXtV+M25pPqh@mail.gmail.com>

2010/11/27 Vitja Makarov <vitja.makarov at gmail.com>:
> 2010/11/27 Stefan Behnel <stefan_ml at behnel.de>:
>> Hi,
>>
>> please start a new thread when switching to a new topic.
>>
>> Vitja Makarov, 27.11.2010 08:24:
>>> I'm now trying to make "module scope lambda" work.
>>> That's almost done.
>>>
>>> I see that needs_assignment_synthesis() should be forced to create
>>> PyMethodDef struct.
>>
>> I would be happier if we could get rid of PyMethodDef in the long term, but
>> it's clearly easier to keep it for now.
>>
>>
>>> Now I do it this way:
>>> def needs_assignment_synthesis(self, ...):
>>> .....
>>> ? ? ? ? ? ? ? if self.name == '<lambda>':
>>> .....
>>>
>>> Seems like a dirty hack.
>>
>> Yes. Instead, use a new flag "is_lambda" on the entry.
>>
>
> This check is performed in DefNode so I prefer to call it
> force_assignment_synthesis it could be used later for module level
> functions.
>
> Or do you mean that "is_lambda" should be set after
> declare_lambda_function() and then checked in DefNode?
>
>>
>>> Also lambda function is defined in module methods table as
>>> __pyx_lambda_funcdef4lala_lambda1,
>>> Is it safe not to add lambda cname to pyfunc_entries in
>>> Scope.declare_lambda_function()?
>>
>> I don't know, you can try. In any case, it needs to do everything that a
>> normal Python function does, except for adding its Python name to the
>> defining scope. That's a rather small distinction.
>>
>> Stefan
>
> Seems to work fine. No regressions.
>
> --
> vitja.
>

I've attached patch to ticket #308, not sure right place for it.
Btw lambda at module scope gives +29 points in pyregr.

-- 
vitja.

From stefan_ml at behnel.de  Sat Nov 27 09:30:07 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 27 Nov 2010 09:30:07 +0100
Subject: [Cython] module level lambda functions
In-Reply-To: <4CF0C0C0.2050702@behnel.de>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>	<4CD811B2.1010108@behnel.de>	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>	<4CDCE8D3.70002@behnel.de>	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>	<4CDCF489.3000108@behnel.de>	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>	<4CDDAC83.9030903@behnel.de>	<4CEC3EA6.9040002@behnel.de>	<AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>	<4CF0BCDE.1020407@behnel.de>	<AANLkTi=XkcQwgBnrzjEn=3O42-t4RVgk8Ty+zcA_G+gB@mail.gmail.com>
	<4CF0C0C0.2050702@behnel.de>
Message-ID: <4CF0C18F.7060406@behnel.de>

Stefan Behnel, 27.11.2010 09:26:
> Vitja Makarov, 27.11.2010 09:17:
>> 2010/11/27 Stefan Behnel:
>>> Vitja Makarov, 27.11.2010 08:24:
>>>> Also lambda function is defined in module methods table as
>>>> __pyx_lambda_funcdef4lala_lambda1,
>>>> Is it safe not to add lambda cname to pyfunc_entries in
>>>> Scope.declare_lambda_function()?
>>>
>>> I don't know, you can try. In any case, it needs to do everything that a
>>> normal Python function does, except for adding its Python name to the
>>> defining scope. That's a rather small distinction.
>>
>> Seems to work fine. No regressions.
>
> Ok. IIRC, lambda functions are already kept in a different list. I wonder
> if that was a good idea. Maybe a flag right on the DefNode would have been
> a better way to do it.

Rethink this a bit, a flag on the entry seems more useful here...

Stefan

From vitja.makarov at gmail.com  Sat Nov 27 09:35:30 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 27 Nov 2010 11:35:30 +0300
Subject: [Cython] module level lambda functions
In-Reply-To: <4CF0C18F.7060406@behnel.de>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>
	<4CD811B2.1010108@behnel.de>
	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>
	<4CDCE8D3.70002@behnel.de>
	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>
	<4CDCF489.3000108@behnel.de>
	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>
	<4CDDAC83.9030903@behnel.de> <4CEC3EA6.9040002@behnel.de>
	<AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>
	<4CF0BCDE.1020407@behnel.de>
	<AANLkTi=XkcQwgBnrzjEn=3O42-t4RVgk8Ty+zcA_G+gB@mail.gmail.com>
	<4CF0C0C0.2050702@behnel.de> <4CF0C18F.7060406@behnel.de>
Message-ID: <AANLkTikpGSoLo1WKWMLZ=wizTxEzKcDi+oe0krsKne6h@mail.gmail.com>

2010/11/27 Stefan Behnel <stefan_ml at behnel.de>:
> Stefan Behnel, 27.11.2010 09:26:
>> Vitja Makarov, 27.11.2010 09:17:
>>> 2010/11/27 Stefan Behnel:
>>>> Vitja Makarov, 27.11.2010 08:24:
>>>>> Also lambda function is defined in module methods table as
>>>>> __pyx_lambda_funcdef4lala_lambda1,
>>>>> Is it safe not to add lambda cname to pyfunc_entries in
>>>>> Scope.declare_lambda_function()?
>>>>
>>>> I don't know, you can try. In any case, it needs to do everything that a
>>>> normal Python function does, except for adding its Python name to the
>>>> defining scope. That's a rather small distinction.
>>>
>>> Seems to work fine. No regressions.
>>
>> Ok. IIRC, lambda functions are already kept in a different list. I wonder
>> if that was a good idea. Maybe a flag right on the DefNode would have been
>> a better way to do it.
>
> Rethink this a bit, a flag on the entry seems more useful here...
>

I think so too ;) Btw more generic flag will be needed someday.

-- 
vitja.

From vitja.makarov at gmail.com  Sat Nov 27 09:54:49 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 27 Nov 2010 11:54:49 +0300
Subject: [Cython] module level lambda functions
In-Reply-To: <AANLkTikpGSoLo1WKWMLZ=wizTxEzKcDi+oe0krsKne6h@mail.gmail.com>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>
	<4CD811B2.1010108@behnel.de>
	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>
	<4CDCE8D3.70002@behnel.de>
	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>
	<4CDCF489.3000108@behnel.de>
	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>
	<4CDDAC83.9030903@behnel.de> <4CEC3EA6.9040002@behnel.de>
	<AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>
	<4CF0BCDE.1020407@behnel.de>
	<AANLkTi=XkcQwgBnrzjEn=3O42-t4RVgk8Ty+zcA_G+gB@mail.gmail.com>
	<4CF0C0C0.2050702@behnel.de> <4CF0C18F.7060406@behnel.de>
	<AANLkTikpGSoLo1WKWMLZ=wizTxEzKcDi+oe0krsKne6h@mail.gmail.com>
Message-ID: <AANLkTin-u+MSKoKO2ykqTw2Vao2rORt8Pt++javBPBFX@mail.gmail.com>

2010/11/27 Vitja Makarov <vitja.makarov at gmail.com>:
> 2010/11/27 Stefan Behnel <stefan_ml at behnel.de>:
>> Stefan Behnel, 27.11.2010 09:26:
>>> Vitja Makarov, 27.11.2010 09:17:
>>>> 2010/11/27 Stefan Behnel:
>>>>> Vitja Makarov, 27.11.2010 08:24:
>>>>>> Also lambda function is defined in module methods table as
>>>>>> __pyx_lambda_funcdef4lala_lambda1,
>>>>>> Is it safe not to add lambda cname to pyfunc_entries in
>>>>>> Scope.declare_lambda_function()?
>>>>>
>>>>> I don't know, you can try. In any case, it needs to do everything that a
>>>>> normal Python function does, except for adding its Python name to the
>>>>> defining scope. That's a rather small distinction.
>>>>
>>>> Seems to work fine. No regressions.
>>>
>>> Ok. IIRC, lambda functions are already kept in a different list. I wonder
>>> if that was a good idea. Maybe a flag right on the DefNode would have been
>>> a better way to do it.
>>
>> Rethink this a bit, a flag on the entry seems more useful here...
>>
>
> I think so too ;) Btw more generic flag will be needed someday.
>

http://trac.cython.org/cython_trac/ticket/308

Attached patch adds is_lambda to Entry and set this flag in
declare_lambda_function() then DefNode.needs_assignment_synthesis()
check for this flag. Seems rather simple.

-- 
vitja.

From stefan_ml at behnel.de  Sat Nov 27 10:08:36 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 27 Nov 2010 10:08:36 +0100
Subject: [Cython] module level lambda functions
In-Reply-To: <AANLkTi=TFj8v_wOOCiQrJA2iGuBmWtVDDXtV+M25pPqh@mail.gmail.com>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>	<4CD811B2.1010108@behnel.de>	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>	<4CDCE8D3.70002@behnel.de>	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>	<4CDCF489.3000108@behnel.de>	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>	<4CDDAC83.9030903@behnel.de>
	<4CEC3EA6.9040002@behnel.de>	<AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>	<4CF0BCDE.1020407@behnel.de>	<AANLkTi=XkcQwgBnrzjEn=3O42-t4RVgk8Ty+zcA_G+gB@mail.gmail.com>
	<AANLkTi=TFj8v_wOOCiQrJA2iGuBmWtVDDXtV+M25pPqh@mail.gmail.com>
Message-ID: <4CF0CA94.1050001@behnel.de>

Vitja Makarov, 27.11.2010 09:28:
> I've attached patch to ticket #308, not sure right place for it.

Not quite the right place. That ticket is about keeping module level C 
names local to the module *instance* instead of the module's C code. Looks 
like the lambda reference is actually unrelated after all.

Could you create a new ticket for this? It's better to have a ticket for a 
single feature that we can close (with a name on it) when it's done.

Then, please add a test that involves typed cdef names at the module level 
and also a case where Cython can infer expression types (unicode indexing 
maybe?). Having these things work at the Python level is great, but for the 
Cython language, we always need to make sure that name lookups, type 
inference etc. work as expected, and these are scope dependent. Those are 
things that CPython's regression tests cannot cover.

You can also type parameters in lambda expressions, see the tests in 
run/lambda_T195.pyx. That's not related to this patch, but you can use it 
in your tests to check that type inference/analysis works correctly here.


> Btw lambda at module scope gives +29 points in pyregr.

Yes, I noticed that a couple of pyregr tests failed because of that. Nice. :)

Stefan

From vitja.makarov at gmail.com  Sat Nov 27 17:11:46 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Sat, 27 Nov 2010 19:11:46 +0300
Subject: [Cython] module level lambda functions
In-Reply-To: <4CF0CA94.1050001@behnel.de>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>
	<4CD811B2.1010108@behnel.de>
	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>
	<4CDCE8D3.70002@behnel.de>
	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>
	<4CDCF489.3000108@behnel.de>
	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>
	<4CDDAC83.9030903@behnel.de> <4CEC3EA6.9040002@behnel.de>
	<AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>
	<4CF0BCDE.1020407@behnel.de>
	<AANLkTi=XkcQwgBnrzjEn=3O42-t4RVgk8Ty+zcA_G+gB@mail.gmail.com>
	<AANLkTi=TFj8v_wOOCiQrJA2iGuBmWtVDDXtV+M25pPqh@mail.gmail.com>
	<4CF0CA94.1050001@behnel.de>
Message-ID: <AANLkTi=Hp5v3NXZ1QCQeuBU7v6kzKoSHxrUS5xiAy8uQ@mail.gmail.com>

2010/11/27 Stefan Behnel <stefan_ml at behnel.de>:
> Vitja Makarov, 27.11.2010 09:28:
>> I've attached patch to ticket #308, not sure right place for it.
>
> Not quite the right place. That ticket is about keeping module level C
> names local to the module *instance* instead of the module's C code. Looks
> like the lambda reference is actually unrelated after all.
>
> Could you create a new ticket for this? It's better to have a ticket for a
> single feature that we can close (with a name on it) when it's done.
>
> Then, please add a test that involves typed cdef names at the module level
> and also a case where Cython can infer expression types (unicode indexing
> maybe?). Having these things work at the Python level is great, but for the
> Cython language, we always need to make sure that name lookups, type
> inference etc. work as expected, and these are scope dependent. Those are
> things that CPython's regression tests cannot cover.
>
> You can also type parameters in lambda expressions, see the tests in
> run/lambda_T195.pyx. That's not related to this patch, but you can use it
> in your tests to check that type inference/analysis works correctly here.
>
>
>> Btw lambda at module scope gives +29 points in pyregr.
>
> Yes, I noticed that a couple of pyregr tests failed because of that. Nice. :)
>
> Stefan
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>

I've created ticket http://trac.cython.org/cython_trac/ticket/603

And added some more tests but I'm not sure that they cover all the cases.
Can you please add some?

-- 
vitja.

From stefan_ml at behnel.de  Sat Nov 27 19:53:51 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 27 Nov 2010 19:53:51 +0100
Subject: [Cython] module level lambda functions
In-Reply-To: <AANLkTi=Hp5v3NXZ1QCQeuBU7v6kzKoSHxrUS5xiAy8uQ@mail.gmail.com>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>	<4CD811B2.1010108@behnel.de>	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>	<4CDCE8D3.70002@behnel.de>	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>	<4CDCF489.3000108@behnel.de>	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>	<4CDDAC83.9030903@behnel.de>
	<4CEC3EA6.9040002@behnel.de>	<AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>	<4CF0BCDE.1020407@behnel.de>	<AANLkTi=XkcQwgBnrzjEn=3O42-t4RVgk8Ty+zcA_G+gB@mail.gmail.com>	<AANLkTi=TFj8v_wOOCiQrJA2iGuBmWtVDDXtV+M25pPqh@mail.gmail.com>	<4CF0CA94.1050001@behnel.de>
	<AANLkTi=Hp5v3NXZ1QCQeuBU7v6kzKoSHxrUS5xiAy8uQ@mail.gmail.com>
Message-ID: <4CF153BF.6030502@behnel.de>

Vitja Makarov, 27.11.2010 17:11:
> 2010/11/27 Stefan Behnel:
>> Vitja Makarov, 27.11.2010 09:28:
>>> I've attached patch to ticket #308, not sure right place for it.
>>
>> Not quite the right place. That ticket is about keeping module level C
>> names local to the module *instance* instead of the module's C code. Looks
>> like the lambda reference is actually unrelated after all.
>>
>> Could you create a new ticket for this? It's better to have a ticket for a
>> single feature that we can close (with a name on it) when it's done.
>>
>> Then, please add a test that involves typed cdef names at the module level
>> and also a case where Cython can infer expression types (unicode indexing
>> maybe?). Having these things work at the Python level is great, but for the
>> Cython language, we always need to make sure that name lookups, type
>> inference etc. work as expected, and these are scope dependent. Those are
>> things that CPython's regression tests cannot cover.
>>
>> You can also type parameters in lambda expressions, see the tests in
>> run/lambda_T195.pyx. That's not related to this patch, but you can use it
>> in your tests to check that type inference/analysis works correctly here.
>
> I've created ticket http://trac.cython.org/cython_trac/ticket/603
>
> And added some more tests but I'm not sure that they cover all the cases.
> Can you please add some?

I think what you have there is just fine and passes nicely.

Thanks!

Stefan

From stefan_ml at behnel.de  Sat Nov 27 21:58:31 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 27 Nov 2010 21:58:31 +0100
Subject: [Cython] Bug with ExprNodes.CoerceFromPyTypeNode
In-Reply-To: <4CEED69C.6010208@behnel.de>
References: <AANLkTinxXTFgGarO0S03WZkqVdQJEfbB0EM9RPTSPtP3@mail.gmail.com>	<4CEED582.9090808@behnel.de>
	<4CEED69C.6010208@behnel.de>
Message-ID: <4CF170F7.3080709@behnel.de>

Stefan Behnel, 25.11.2010 22:35:
> Stefan Behnel, 25.11.2010 22:30:
>> Mike Hansen, 25.11.2010 22:16:
>>> I'm trying to build a Cython extension and get the following traceback
>>>
>>>      ...
>>>      File "/opt/sage/local/lib/python2.6/site-packages/Cython/Compiler/Nodes.py",
>>> line 4499, in generate_execution_code
>>>        from_py_node =
>>> ExprNodes.CoerceFromPyTypeNode(self.loopvar_node.type, target_node,
>>> None)
>>>      File "/opt/sage/local/lib/python2.6/site-packages/Cython/Compiler/ExprNodes.py",
>>> line 6634, in __init__
>>>        if not result_type.create_from_py_utility_code(env):
>>>      File "/opt/sage/local/lib/python2.6/site-packages/Cython/Compiler/PyrexTypes.py",
>>> line 279, in create_from_py_utility_code
>>>        env.use_utility_code(self.from_py_utility_code)
>>> AttributeError: 'NoneType' object has no attribute 'use_utility_code'
>>>
>>> What should be passed into ExprNodes.CoerceFromPyTypeNode for env
>>> instead of None?
>>
>> Wow, thanks for catching that. There's a bunch of evil code there that
>> seriously needs to get ripped out.
>
> http://trac.cython.org/cython_trac/ticket/601

BTW, could you show us the relevant sections of the code that you are 
compiling there? A test case would be great.

Stefan

From mhansen at gmail.com  Sat Nov 27 22:28:14 2010
From: mhansen at gmail.com (Mike Hansen)
Date: Sat, 27 Nov 2010 13:28:14 -0800
Subject: [Cython] Bug with ExprNodes.CoerceFromPyTypeNode
In-Reply-To: <4CF170F7.3080709@behnel.de>
References: <AANLkTinxXTFgGarO0S03WZkqVdQJEfbB0EM9RPTSPtP3@mail.gmail.com>
	<4CEED582.9090808@behnel.de> <4CEED69C.6010208@behnel.de>
	<4CF170F7.3080709@behnel.de>
Message-ID: <AANLkTimGkXd-_UYo2PYE-VivHYDtki64DSeu2kUXNiid@mail.gmail.com>

On Sat, Nov 27, 2010 at 12:58 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> BTW, could you show us the relevant sections of the code that you are
> compiling there? A test case would be great.

If I run pdb.post_mortem(), is there an easy way to move up the stack
to somewhere where I can find what code it was trying to compile? The
file is somewhat large so I don't want to do it by trial and error.

--Mike

From stefan_ml at behnel.de  Sun Nov 28 08:36:11 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 28 Nov 2010 08:36:11 +0100
Subject: [Cython] Bug with ExprNodes.CoerceFromPyTypeNode
In-Reply-To: <AANLkTimGkXd-_UYo2PYE-VivHYDtki64DSeu2kUXNiid@mail.gmail.com>
References: <AANLkTinxXTFgGarO0S03WZkqVdQJEfbB0EM9RPTSPtP3@mail.gmail.com>	<4CEED582.9090808@behnel.de>
	<4CEED69C.6010208@behnel.de>	<4CF170F7.3080709@behnel.de>
	<AANLkTimGkXd-_UYo2PYE-VivHYDtki64DSeu2kUXNiid@mail.gmail.com>
Message-ID: <4CF2066B.9070303@behnel.de>

Mike Hansen, 27.11.2010 22:28:
> On Sat, Nov 27, 2010 at 12:58 PM, Stefan Behnel wrote:
>> BTW, could you show us the relevant sections of the code that you are
>> compiling there? A test case would be great.
>
> If I run pdb.post_mortem(), is there an easy way to move up the stack
> to somewhere where I can find what code it was trying to compile? The
> file is somewhat large so I don't want to do it by trial and error.

Ah, right, I think you don't get an AST trace during code generation. 
That's too bad.

But you can just go back to the frame in Nodes.py where it is trying to 
handle the ForFromStatNode and print its self.pos. That will give you the 
file and line/column number.

Stefan

From mhansen at gmail.com  Sun Nov 28 09:43:15 2010
From: mhansen at gmail.com (Mike Hansen)
Date: Sun, 28 Nov 2010 00:43:15 -0800
Subject: [Cython] Bug with ExprNodes.CoerceFromPyTypeNode
In-Reply-To: <4CF2066B.9070303@behnel.de>
References: <AANLkTinxXTFgGarO0S03WZkqVdQJEfbB0EM9RPTSPtP3@mail.gmail.com>
	<4CEED582.9090808@behnel.de> <4CEED69C.6010208@behnel.de>
	<4CF170F7.3080709@behnel.de>
	<AANLkTimGkXd-_UYo2PYE-VivHYDtki64DSeu2kUXNiid@mail.gmail.com>
	<4CF2066B.9070303@behnel.de>
Message-ID: <AANLkTimKV9wEWBQzc4zPrAhWO9WQFYVzMVGj-fi6f77N@mail.gmail.com>

On Sat, Nov 27, 2010 at 11:36 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Ah, right, I think you don't get an AST trace during code generation.
> That's too bad.
>
> But you can just go back to the frame in Nodes.py where it is trying to
> handle the ForFromStatNode and print its self.pos. That will give you the
> file and line/column number.

Thanks!  Here is a minimal example of where it succeeds and where it fails:

cdef extern from "globals.h":
    ctypedef unsigned long Ulong

cdef extern from "list.h":
    Ulong size()
    unsigned long size2()

def fail():
    cdef j = 0
    for j from 0 <= j < size():
        print j

def succeed():
    cdef j = 0
    for j from 0 <= j < size2():
        print j

--Mike

From stefan_ml at behnel.de  Sun Nov 28 16:49:44 2010
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 28 Nov 2010 16:49:44 +0100
Subject: [Cython] Bug with ExprNodes.CoerceFromPyTypeNode
In-Reply-To: <AANLkTimKV9wEWBQzc4zPrAhWO9WQFYVzMVGj-fi6f77N@mail.gmail.com>
References: <AANLkTinxXTFgGarO0S03WZkqVdQJEfbB0EM9RPTSPtP3@mail.gmail.com>	<4CEED582.9090808@behnel.de>
	<4CEED69C.6010208@behnel.de>	<4CF170F7.3080709@behnel.de>	<AANLkTimGkXd-_UYo2PYE-VivHYDtki64DSeu2kUXNiid@mail.gmail.com>	<4CF2066B.9070303@behnel.de>
	<AANLkTimKV9wEWBQzc4zPrAhWO9WQFYVzMVGj-fi6f77N@mail.gmail.com>
Message-ID: <4CF27A18.5090807@behnel.de>

Mike Hansen, 28.11.2010 09:43:
> Here is a minimal example of where it succeeds and where it fails:
>
> cdef extern from "globals.h":
>      ctypedef unsigned long Ulong
>
> cdef extern from "list.h":
>      Ulong size()
>      unsigned long size2()
>
> def fail():
>      cdef j = 0
>      for j from 0<= j<  size():
>          print j
>
> def succeed():
>      cdef j = 0
>      for j from 0<= j<  size2():
>          print j

Thanks!

Given that you explicitly Python typed the loop variables in your example, 
I assume you are aware that a C typed loop variable shouldn't hit this problem?

Stefan

From mhansen at gmail.com  Sun Nov 28 19:43:10 2010
From: mhansen at gmail.com (Mike Hansen)
Date: Sun, 28 Nov 2010 10:43:10 -0800
Subject: [Cython] Bug with ExprNodes.CoerceFromPyTypeNode
In-Reply-To: <4CF27A18.5090807@behnel.de>
References: <AANLkTinxXTFgGarO0S03WZkqVdQJEfbB0EM9RPTSPtP3@mail.gmail.com>
	<4CEED582.9090808@behnel.de> <4CEED69C.6010208@behnel.de>
	<4CF170F7.3080709@behnel.de>
	<AANLkTimGkXd-_UYo2PYE-VivHYDtki64DSeu2kUXNiid@mail.gmail.com>
	<4CF2066B.9070303@behnel.de>
	<AANLkTimKV9wEWBQzc4zPrAhWO9WQFYVzMVGj-fi6f77N@mail.gmail.com>
	<4CF27A18.5090807@behnel.de>
Message-ID: <AANLkTim=M6SBTvYNf1tz4r70gVvv_8ZjzS9f2hY1Xpbj@mail.gmail.com>

On Sun, Nov 28, 2010 at 7:49 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Given that you explicitly Python typed the loop variables in your example,
> I assume you are aware that a C typed loop variable shouldn't hit this problem?

Yeah, I had noticed that after the fact.

--Mike

From vitja.makarov at gmail.com  Mon Nov 29 21:18:48 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Mon, 29 Nov 2010 23:18:48 +0300
Subject: [Cython] Redefinition of Python functions (ticket #489)
Message-ID: <AANLkTikAHDpWm1=VZEcBjpT5Vd421Y+Ej6BYLAkfirCR@mail.gmail.com>

Hi!

Attached patch tries to solve the issue.

How it works:

- `declare_pyfunction` accepts new argument,  called `allow_redefine`
which means that this function can be redefined.
This flag is set when function isn't cpdef wrapper. If allow_redefine
is False old declare_pyfunction is called.
 - When scope.declare_pyfunction() is called it checks was this name
already defined
   - it's okay if it was defined with py_object_type
   - if it was defined with unspecified_type set entry type to py_object_type
   - if entry type has other type fallback to old declare_pyfunction() behavior
   - if it wasn't create dummy entry with py_object_type
   - Then create real private entry for function with is_lambda flag
set (should be renamed to something more generic, any ideas?)
- ID is added to function cname.

ps:
I've noticed interesting thing, this code compiles:

def foo():
    pass
foo = 1

but this not:
foo = 1
def foo():
    pass

-- 
vitja.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: allow-function-redefine.patch
Type: text/x-patch
Size: 6949 bytes
Desc: not available
Url : http://codespeak.net/pipermail/cython-dev/attachments/20101129/534deeb5/attachment.bin 

From greg.ewing at canterbury.ac.nz  Mon Nov 29 21:27:26 2010
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 30 Nov 2010 09:27:26 +1300
Subject: [Cython] splitting up Python functions
In-Reply-To: <4CF0BA9F.9060304@behnel.de>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>
	<4CD811B2.1010108@behnel.de>
	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>
	<4CDCE8D3.70002@behnel.de>
	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>
	<4CDCF489.3000108@behnel.de>
	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>
	<4CDDAC83.9030903@behnel.de> <4CEC3EA6.9040002@behnel.de>
	<AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>
	<4CF0BA9F.9060304@behnel.de>
Message-ID: <4CF40CAE.6070609@canterbury.ac.nz>

Stefan Behnel wrote:

> * it makes the inner C function directly callable, thus making it easier to 
> implement things like cpdef functions and generators on top.

If you mention the name of such a function without calling it,
does it refer to the C function or the Python function?

-- 
Greg

From robertwb at math.washington.edu  Tue Nov 30 04:56:07 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 29 Nov 2010 19:56:07 -0800
Subject: [Cython] splitting up Python functions
In-Reply-To: <4CF40CAE.6070609@canterbury.ac.nz>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>
	<4CD811B2.1010108@behnel.de>
	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>
	<4CDCE8D3.70002@behnel.de>
	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>
	<4CDCF489.3000108@behnel.de>
	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>
	<4CDDAC83.9030903@behnel.de> <4CEC3EA6.9040002@behnel.de>
	<AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>
	<4CF0BA9F.9060304@behnel.de> <4CF40CAE.6070609@canterbury.ac.nz>
Message-ID: <AANLkTi=yndmiv0+DhzFTAOmA20sHK6zCxUVCx6ouqrbZ@mail.gmail.com>

On Mon, Nov 29, 2010 at 12:27 PM, Greg Ewing
<greg.ewing at canterbury.ac.nz> wrote:
> Stefan Behnel wrote:
>
>> * it makes the inner C function directly callable, thus making it easier to
>> implement things like cpdef functions and generators on top.
>
> If you mention the name of such a function without calling it,
> does it refer to the C function or the Python function?

That would depend on the context in which it's being used.
Essentially, the as_variable attribute would be set, allowing it to
coerce to a Python object if need be.

- Robert

From robertwb at math.washington.edu  Tue Nov 30 05:33:00 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 29 Nov 2010 20:33:00 -0800
Subject: [Cython] module level lambda functions (was: >=python-2.6
 property pattern not supported)
In-Reply-To: <AANLkTi=TFj8v_wOOCiQrJA2iGuBmWtVDDXtV+M25pPqh@mail.gmail.com>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>
	<4CD811B2.1010108@behnel.de>
	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>
	<4CDCE8D3.70002@behnel.de>
	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>
	<4CDCF489.3000108@behnel.de>
	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>
	<4CDDAC83.9030903@behnel.de> <4CEC3EA6.9040002@behnel.de>
	<AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>
	<4CF0BCDE.1020407@behnel.de>
	<AANLkTi=XkcQwgBnrzjEn=3O42-t4RVgk8Ty+zcA_G+gB@mail.gmail.com>
	<AANLkTi=TFj8v_wOOCiQrJA2iGuBmWtVDDXtV+M25pPqh@mail.gmail.com>
Message-ID: <AANLkTikTf1x_9=LiHt3fbc45nP8E=zFtRdcAh54S3i2q@mail.gmail.com>

On Sat, Nov 27, 2010 at 12:28 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2010/11/27 Vitja Makarov <vitja.makarov at gmail.com>:
>> 2010/11/27 Stefan Behnel <stefan_ml at behnel.de>:
>>> Hi,
>>>
>>> please start a new thread when switching to a new topic.
>>>
>>> Vitja Makarov, 27.11.2010 08:24:
>>>> I'm now trying to make "module scope lambda" work.
>>>> That's almost done.
>>>>
>>>> I see that needs_assignment_synthesis() should be forced to create
>>>> PyMethodDef struct.
>>>
>>> I would be happier if we could get rid of PyMethodDef in the long term, but
>>> it's clearly easier to keep it for now.
>>>
>>>
>>>> Now I do it this way:
>>>> def needs_assignment_synthesis(self, ...):
>>>> .....
>>>> ? ? ? ? ? ? ? if self.name == '<lambda>':
>>>> .....
>>>>
>>>> Seems like a dirty hack.
>>>
>>> Yes. Instead, use a new flag "is_lambda" on the entry.
>>>
>>
>> This check is performed in DefNode so I prefer to call it
>> force_assignment_synthesis it could be used later for module level
>> functions.
>>
>> Or do you mean that "is_lambda" should be set after
>> declare_lambda_function() and then checked in DefNode?
>>
>>>
>>>> Also lambda function is defined in module methods table as
>>>> __pyx_lambda_funcdef4lala_lambda1,
>>>> Is it safe not to add lambda cname to pyfunc_entries in
>>>> Scope.declare_lambda_function()?
>>>
>>> I don't know, you can try. In any case, it needs to do everything that a
>>> normal Python function does, except for adding its Python name to the
>>> defining scope. That's a rather small distinction.
>>>
>>> Stefan
>>
>> Seems to work fine. No regressions.
>>
>> --
>> vitja.
>>
>
> I've attached patch to ticket #308, not sure right place for it.
> Btw lambda at module scope gives +29 points in pyregr.
>

Wow, it's amazing how much can happen when you go away for a few days
of vacation. Thanks!

- Robert

From vitja.makarov at gmail.com  Tue Nov 30 06:20:31 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 30 Nov 2010 08:20:31 +0300
Subject: [Cython] splitting up Python functions
In-Reply-To: <AANLkTi=yndmiv0+DhzFTAOmA20sHK6zCxUVCx6ouqrbZ@mail.gmail.com>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>
	<4CD811B2.1010108@behnel.de>
	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>
	<4CDCE8D3.70002@behnel.de>
	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>
	<4CDCF489.3000108@behnel.de>
	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>
	<4CDDAC83.9030903@behnel.de> <4CEC3EA6.9040002@behnel.de>
	<AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>
	<4CF0BA9F.9060304@behnel.de> <4CF40CAE.6070609@canterbury.ac.nz>
	<AANLkTi=yndmiv0+DhzFTAOmA20sHK6zCxUVCx6ouqrbZ@mail.gmail.com>
Message-ID: <AANLkTimkDb_Z2BtG2w7MuYin7CSm84HU8EVwagsS7Dow@mail.gmail.com>

2010/11/30 Robert Bradshaw <robertwb at math.washington.edu>:
> On Mon, Nov 29, 2010 at 12:27 PM, Greg Ewing
> <greg.ewing at canterbury.ac.nz> wrote:
>> Stefan Behnel wrote:
>>
>>> * it makes the inner C function directly callable, thus making it easier to
>>> implement things like cpdef functions and generators on top.
>>
>> If you mention the name of such a function without calling it,
>> does it refer to the C function or the Python function?
>
> That would depend on the context in which it's being used.
> Essentially, the as_variable attribute would be set, allowing it to
> coerce to a Python object if need be.
>

I see problem with closures here where should scope object be created
in C function or in wrapper?

-- 
vitja.

From vitja.makarov at gmail.com  Tue Nov 30 06:23:11 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Tue, 30 Nov 2010 08:23:11 +0300
Subject: [Cython] module level lambda functions (was: >=python-2.6
 property pattern not supported)
In-Reply-To: <AANLkTikTf1x_9=LiHt3fbc45nP8E=zFtRdcAh54S3i2q@mail.gmail.com>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>
	<4CD811B2.1010108@behnel.de>
	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>
	<4CDCE8D3.70002@behnel.de>
	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>
	<4CDCF489.3000108@behnel.de>
	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>
	<4CDDAC83.9030903@behnel.de> <4CEC3EA6.9040002@behnel.de>
	<AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>
	<4CF0BCDE.1020407@behnel.de>
	<AANLkTi=XkcQwgBnrzjEn=3O42-t4RVgk8Ty+zcA_G+gB@mail.gmail.com>
	<AANLkTi=TFj8v_wOOCiQrJA2iGuBmWtVDDXtV+M25pPqh@mail.gmail.com>
	<AANLkTikTf1x_9=LiHt3fbc45nP8E=zFtRdcAh54S3i2q@mail.gmail.com>
Message-ID: <AANLkTikv48+GztSvO81eamauDtp4GqEWxqMQOaWSOuNb@mail.gmail.com>

2010/11/30 Robert Bradshaw <robertwb at math.washington.edu>:
> On Sat, Nov 27, 2010 at 12:28 AM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
>> 2010/11/27 Vitja Makarov <vitja.makarov at gmail.com>:
>>> 2010/11/27 Stefan Behnel <stefan_ml at behnel.de>:
>>>> Hi,
>>>>
>>>> please start a new thread when switching to a new topic.
>>>>
>>>> Vitja Makarov, 27.11.2010 08:24:
>>>>> I'm now trying to make "module scope lambda" work.
>>>>> That's almost done.
>>>>>
>>>>> I see that needs_assignment_synthesis() should be forced to create
>>>>> PyMethodDef struct.
>>>>
>>>> I would be happier if we could get rid of PyMethodDef in the long term, but
>>>> it's clearly easier to keep it for now.
>>>>
>>>>
>>>>> Now I do it this way:
>>>>> def needs_assignment_synthesis(self, ...):
>>>>> .....
>>>>> ? ? ? ? ? ? ? if self.name == '<lambda>':
>>>>> .....
>>>>>
>>>>> Seems like a dirty hack.
>>>>
>>>> Yes. Instead, use a new flag "is_lambda" on the entry.
>>>>
>>>
>>> This check is performed in DefNode so I prefer to call it
>>> force_assignment_synthesis it could be used later for module level
>>> functions.
>>>
>>> Or do you mean that "is_lambda" should be set after
>>> declare_lambda_function() and then checked in DefNode?
>>>
>>>>
>>>>> Also lambda function is defined in module methods table as
>>>>> __pyx_lambda_funcdef4lala_lambda1,
>>>>> Is it safe not to add lambda cname to pyfunc_entries in
>>>>> Scope.declare_lambda_function()?
>>>>
>>>> I don't know, you can try. In any case, it needs to do everything that a
>>>> normal Python function does, except for adding its Python name to the
>>>> defining scope. That's a rather small distinction.
>>>>
>>>> Stefan
>>>
>>> Seems to work fine. No regressions.
>>>
>>> --
>>> vitja.
>>>
>>
>> I've attached patch to ticket #308, not sure right place for it.
>> Btw lambda at module scope gives +29 points in pyregr.
>>
>
> Wow, it's amazing how much can happen when you go away for a few days
> of vacation. Thanks!
>
> - Robert

Thank you!

Btw there is one more issue with lambdas, #605. Now it gives gcc
compilation error. It seems that we forget about this case.

-- 
vitja.

From robertwb at math.washington.edu  Tue Nov 30 06:35:45 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Mon, 29 Nov 2010 21:35:45 -0800
Subject: [Cython] splitting up Python functions
In-Reply-To: <AANLkTimkDb_Z2BtG2w7MuYin7CSm84HU8EVwagsS7Dow@mail.gmail.com>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>
	<4CD811B2.1010108@behnel.de>
	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>
	<4CDCE8D3.70002@behnel.de>
	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>
	<4CDCF489.3000108@behnel.de>
	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>
	<4CDDAC83.9030903@behnel.de> <4CEC3EA6.9040002@behnel.de>
	<AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>
	<4CF0BA9F.9060304@behnel.de> <4CF40CAE.6070609@canterbury.ac.nz>
	<AANLkTi=yndmiv0+DhzFTAOmA20sHK6zCxUVCx6ouqrbZ@mail.gmail.com>
	<AANLkTimkDb_Z2BtG2w7MuYin7CSm84HU8EVwagsS7Dow@mail.gmail.com>
Message-ID: <AANLkTimK65o16+3DXPM81RMHSDK3UvxOuP5YjrpUyvVD@mail.gmail.com>

On Mon, Nov 29, 2010 at 9:20 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:
> 2010/11/30 Robert Bradshaw <robertwb at math.washington.edu>:
>> On Mon, Nov 29, 2010 at 12:27 PM, Greg Ewing
>> <greg.ewing at canterbury.ac.nz> wrote:
>>> Stefan Behnel wrote:
>>>
>>>> * it makes the inner C function directly callable, thus making it easier to
>>>> implement things like cpdef functions and generators on top.
>>>
>>> If you mention the name of such a function without calling it,
>>> does it refer to the C function or the Python function?
>>
>> That would depend on the context in which it's being used.
>> Essentially, the as_variable attribute would be set, allowing it to
>> coerce to a Python object if need be.
>>
>
> I see problem with closures here where should scope object be created
> in C function or in wrapper?

The only handle you can get of a closure object is a Python one.
(Well, eventually we want to have cdef closures, but we're not there
yet, and wouldn't be compatible with cdef functions--we'd probably
expose them as structs with a state and function pointer attributes.)

- Robert

From dagss at student.matnat.uio.no  Tue Nov 30 11:12:33 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 30 Nov 2010 11:12:33 +0100
Subject: [Cython] splitting up Python functions
In-Reply-To: <AANLkTimK65o16+3DXPM81RMHSDK3UvxOuP5YjrpUyvVD@mail.gmail.com>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>	<4CD811B2.1010108@behnel.de>	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>	<4CDCE8D3.70002@behnel.de>	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>	<4CDCF489.3000108@behnel.de>	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>	<4CDDAC83.9030903@behnel.de>
	<4CEC3EA6.9040002@behnel.de>	<AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>	<4CF0BA9F.9060304@behnel.de>
	<4CF40CAE.6070609@canterbury.ac.nz>	<AANLkTi=yndmiv0+DhzFTAOmA20sHK6zCxUVCx6ouqrbZ@mail.gmail.com>	<AANLkTimkDb_Z2BtG2w7MuYin7CSm84HU8EVwagsS7Dow@mail.gmail.com>
	<AANLkTimK65o16+3DXPM81RMHSDK3UvxOuP5YjrpUyvVD@mail.gmail.com>
Message-ID: <4CF4CE11.5040707@student.matnat.uio.no>

On 11/30/2010 06:35 AM, Robert Bradshaw wrote:
> On Mon, Nov 29, 2010 at 9:20 PM, Vitja Makarov<vitja.makarov at gmail.com>  wrote:
>    
>> 2010/11/30 Robert Bradshaw<robertwb at math.washington.edu>:
>>      
>>> On Mon, Nov 29, 2010 at 12:27 PM, Greg Ewing
>>> <greg.ewing at canterbury.ac.nz>  wrote:
>>>        
>>>> Stefan Behnel wrote:
>>>>
>>>>          
>>>>> * it makes the inner C function directly callable, thus making it easier to
>>>>> implement things like cpdef functions and generators on top.
>>>>>            
>>>> If you mention the name of such a function without calling it,
>>>> does it refer to the C function or the Python function?
>>>>          
>>> That would depend on the context in which it's being used.
>>> Essentially, the as_variable attribute would be set, allowing it to
>>> coerce to a Python object if need be.
>>>
>>>        
>> I see problem with closures here where should scope object be created
>> in C function or in wrapper?
>>      
> The only handle you can get of a closure object is a Python one.
> (Well, eventually we want to have cdef closures, but we're not there
> yet, and wouldn't be compatible with cdef functions--we'd probably
> expose them as structs with a state and function pointer attributes.)
>    

Just for reference: Carl recently made me aware that ctypes contain some 
code to proxy Python functions to C function pointers. And apparently it 
contains a small compiler that creates new CPU code on demand for this. 
I'm not sure how well that would be exposed for our purposes, if it has 
a C API or not. (I haven't really looked at this myself.)

Being able to create a closure in Cython and pass it as an ordinary C 
function callback, without having to manage the context manually, would 
be a really cool feature! And with such a mini-compiler it is possible.

(Same idea applies to a concept of "pointer to bound cdef method")

Dag Sverre

From robertwb at math.washington.edu  Tue Nov 30 19:28:06 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 30 Nov 2010 10:28:06 -0800
Subject: [Cython] splitting up Python functions
In-Reply-To: <4CF4CE11.5040707@student.matnat.uio.no>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>
	<4CD811B2.1010108@behnel.de>
	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>
	<4CDCE8D3.70002@behnel.de>
	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>
	<4CDCF489.3000108@behnel.de>
	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>
	<4CDDAC83.9030903@behnel.de> <4CEC3EA6.9040002@behnel.de>
	<AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>
	<4CF0BA9F.9060304@behnel.de> <4CF40CAE.6070609@canterbury.ac.nz>
	<AANLkTi=yndmiv0+DhzFTAOmA20sHK6zCxUVCx6ouqrbZ@mail.gmail.com>
	<AANLkTimkDb_Z2BtG2w7MuYin7CSm84HU8EVwagsS7Dow@mail.gmail.com>
	<AANLkTimK65o16+3DXPM81RMHSDK3UvxOuP5YjrpUyvVD@mail.gmail.com>
	<4CF4CE11.5040707@student.matnat.uio.no>
Message-ID: <AANLkTimC=0cTaWTR54wpTGjYdsdkLVAXir3vY+uGhV_h@mail.gmail.com>

On Tue, Nov 30, 2010 at 2:12 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> On 11/30/2010 06:35 AM, Robert Bradshaw wrote:
>> On Mon, Nov 29, 2010 at 9:20 PM, Vitja Makarov<vitja.makarov at gmail.com> ?wrote:
>>
>>> 2010/11/30 Robert Bradshaw<robertwb at math.washington.edu>:
>>>
>>>> On Mon, Nov 29, 2010 at 12:27 PM, Greg Ewing
>>>> <greg.ewing at canterbury.ac.nz> ?wrote:
>>>>
>>>>> Stefan Behnel wrote:
>>>>>
>>>>>
>>>>>> * it makes the inner C function directly callable, thus making it easier to
>>>>>> implement things like cpdef functions and generators on top.
>>>>>>
>>>>> If you mention the name of such a function without calling it,
>>>>> does it refer to the C function or the Python function?
>>>>>
>>>> That would depend on the context in which it's being used.
>>>> Essentially, the as_variable attribute would be set, allowing it to
>>>> coerce to a Python object if need be.
>>>>
>>>>
>>> I see problem with closures here where should scope object be created
>>> in C function or in wrapper?
>>>
>> The only handle you can get of a closure object is a Python one.
>> (Well, eventually we want to have cdef closures, but we're not there
>> yet, and wouldn't be compatible with cdef functions--we'd probably
>> expose them as structs with a state and function pointer attributes.)
>>
>
> Just for reference: Carl recently made me aware that ctypes contain some
> code to proxy Python functions to C function pointers. And apparently it
> contains a small compiler that creates new CPU code on demand for this.
> I'm not sure how well that would be exposed for our purposes, if it has
> a C API or not. (I haven't really looked at this myself.)
>
> Being able to create a closure in Cython and pass it as an ordinary C
> function callback, without having to manage the context manually, would
> be a really cool feature! And with such a mini-compiler it is possible.
>
> (Same idea applies to a concept of "pointer to bound cdef method")

Wow, that's a pretty interesting idea. Does this limit the
applicability to certain architectures? Of course even if the context
needs to be handled separately, we could make it easier than it is
now.

- Robert

From dagss at student.matnat.uio.no  Tue Nov 30 19:52:21 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 30 Nov 2010 19:52:21 +0100
Subject: [Cython] splitting up Python functions
In-Reply-To: <AANLkTimC=0cTaWTR54wpTGjYdsdkLVAXir3vY+uGhV_h@mail.gmail.com>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>	<4CD811B2.1010108@behnel.de>	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>	<4CDCE8D3.70002@behnel.de>	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>	<4CDCF489.3000108@behnel.de>	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>	<4CDDAC83.9030903@behnel.de>
	<4CEC3EA6.9040002@behnel.de>	<AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>	<4CF0BA9F.9060304@behnel.de>
	<4CF40CAE.6070609@canterbury.ac.nz>	<AANLkTi=yndmiv0+DhzFTAOmA20sHK6zCxUVCx6ouqrbZ@mail.gmail.com>	<AANLkTimkDb_Z2BtG2w7MuYin7CSm84HU8EVwagsS7Dow@mail.gmail.com>	<AANLkTimK65o16+3DXPM81RMHSDK3UvxOuP5YjrpUyvVD@mail.gmail.com>	<4CF4CE11.5040707@student.matnat.uio.no>
	<AANLkTimC=0cTaWTR54wpTGjYdsdkLVAXir3vY+uGhV_h@mail.gmail.com>
Message-ID: <4CF547E5.1080409@student.matnat.uio.no>

On 11/30/2010 07:28 PM, Robert Bradshaw wrote:
> On Tue, Nov 30, 2010 at 2:12 AM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no>  wrote:
>    
>> On 11/30/2010 06:35 AM, Robert Bradshaw wrote:
>>      
>>> On Mon, Nov 29, 2010 at 9:20 PM, Vitja Makarov<vitja.makarov at gmail.com>    wrote:
>>>
>>>        
>>>> 2010/11/30 Robert Bradshaw<robertwb at math.washington.edu>:
>>>>
>>>>          
>>>>> On Mon, Nov 29, 2010 at 12:27 PM, Greg Ewing
>>>>> <greg.ewing at canterbury.ac.nz>    wrote:
>>>>>
>>>>>            
>>>>>> Stefan Behnel wrote:
>>>>>>
>>>>>>
>>>>>>              
>>>>>>> * it makes the inner C function directly callable, thus making it easier to
>>>>>>> implement things like cpdef functions and generators on top.
>>>>>>>
>>>>>>>                
>>>>>> If you mention the name of such a function without calling it,
>>>>>> does it refer to the C function or the Python function?
>>>>>>
>>>>>>              
>>>>> That would depend on the context in which it's being used.
>>>>> Essentially, the as_variable attribute would be set, allowing it to
>>>>> coerce to a Python object if need be.
>>>>>
>>>>>
>>>>>            
>>>> I see problem with closures here where should scope object be created
>>>> in C function or in wrapper?
>>>>
>>>>          
>>> The only handle you can get of a closure object is a Python one.
>>> (Well, eventually we want to have cdef closures, but we're not there
>>> yet, and wouldn't be compatible with cdef functions--we'd probably
>>> expose them as structs with a state and function pointer attributes.)
>>>
>>>        
>> Just for reference: Carl recently made me aware that ctypes contain some
>> code to proxy Python functions to C function pointers. And apparently it
>> contains a small compiler that creates new CPU code on demand for this.
>> I'm not sure how well that would be exposed for our purposes, if it has
>> a C API or not. (I haven't really looked at this myself.)
>>
>> Being able to create a closure in Cython and pass it as an ordinary C
>> function callback, without having to manage the context manually, would
>> be a really cool feature! And with such a mini-compiler it is possible.
>>
>> (Same idea applies to a concept of "pointer to bound cdef method")
>>      
> Wow, that's a pretty interesting idea. Does this limit the
> applicability to certain architectures? Of course even if the context
> needs to be handled separately, we could make it easier than it is
> now.
>    

Spending five more minutes on this, ctypes uses the libffi library 
(which is simply bundled with Python, although I haven't probed into 
whether this means it will always be available in a form we can link 
with). It appears to be very user-friendly, and has specific routines 
for creating C closures.

Google it to see platform availability. Apparently closures are not 
available on every platform, but a grep through the source for 
FFI_CLOSURES seem to indicated that the "moxie", "m32r" and "m68k" 
platforms are the only ones without closure support. I can live with 
that :-) Looks to me like libffi lies at the core of a lot of FFI out 
there so that support is rather good.

At any rate, it seems tempting to just make Cython cdef closures simply 
be libffi closures.

Dag Sverre

From robertwb at math.washington.edu  Tue Nov 30 20:26:27 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 30 Nov 2010 11:26:27 -0800
Subject: [Cython] splitting up Python functions
In-Reply-To: <4CF547E5.1080409@student.matnat.uio.no>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>
	<4CD811B2.1010108@behnel.de>
	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>
	<4CDCE8D3.70002@behnel.de>
	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>
	<4CDCF489.3000108@behnel.de>
	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>
	<4CDDAC83.9030903@behnel.de> <4CEC3EA6.9040002@behnel.de>
	<AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>
	<4CF0BA9F.9060304@behnel.de> <4CF40CAE.6070609@canterbury.ac.nz>
	<AANLkTi=yndmiv0+DhzFTAOmA20sHK6zCxUVCx6ouqrbZ@mail.gmail.com>
	<AANLkTimkDb_Z2BtG2w7MuYin7CSm84HU8EVwagsS7Dow@mail.gmail.com>
	<AANLkTimK65o16+3DXPM81RMHSDK3UvxOuP5YjrpUyvVD@mail.gmail.com>
	<4CF4CE11.5040707@student.matnat.uio.no>
	<AANLkTimC=0cTaWTR54wpTGjYdsdkLVAXir3vY+uGhV_h@mail.gmail.com>
	<4CF547E5.1080409@student.matnat.uio.no>
Message-ID: <AANLkTi=f7U9Xx5WgFkOuz38kqw6Rr=V5E0y-PpMk0cki@mail.gmail.com>

On Tue, Nov 30, 2010 at 10:52 AM, Dag Sverre Seljebotn
<dagss at student.matnat.uio.no> wrote:
> On 11/30/2010 07:28 PM, Robert Bradshaw wrote:
>> On Tue, Nov 30, 2010 at 2:12 AM, Dag Sverre Seljebotn
>> <dagss at student.matnat.uio.no> ?wrote:
>>
>>> On 11/30/2010 06:35 AM, Robert Bradshaw wrote:
>>>
>>>> On Mon, Nov 29, 2010 at 9:20 PM, Vitja Makarov<vitja.makarov at gmail.com> ? ?wrote:
>>>>
>>>>
>>>>> 2010/11/30 Robert Bradshaw<robertwb at math.washington.edu>:
>>>>>
>>>>>
>>>>>> On Mon, Nov 29, 2010 at 12:27 PM, Greg Ewing
>>>>>> <greg.ewing at canterbury.ac.nz> ? ?wrote:
>>>>>>
>>>>>>
>>>>>>> Stefan Behnel wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> * it makes the inner C function directly callable, thus making it easier to
>>>>>>>> implement things like cpdef functions and generators on top.
>>>>>>>>
>>>>>>>>
>>>>>>> If you mention the name of such a function without calling it,
>>>>>>> does it refer to the C function or the Python function?
>>>>>>>
>>>>>>>
>>>>>> That would depend on the context in which it's being used.
>>>>>> Essentially, the as_variable attribute would be set, allowing it to
>>>>>> coerce to a Python object if need be.
>>>>>>
>>>>>>
>>>>>>
>>>>> I see problem with closures here where should scope object be created
>>>>> in C function or in wrapper?
>>>>>
>>>>>
>>>> The only handle you can get of a closure object is a Python one.
>>>> (Well, eventually we want to have cdef closures, but we're not there
>>>> yet, and wouldn't be compatible with cdef functions--we'd probably
>>>> expose them as structs with a state and function pointer attributes.)
>>>>
>>>>
>>> Just for reference: Carl recently made me aware that ctypes contain some
>>> code to proxy Python functions to C function pointers. And apparently it
>>> contains a small compiler that creates new CPU code on demand for this.
>>> I'm not sure how well that would be exposed for our purposes, if it has
>>> a C API or not. (I haven't really looked at this myself.)
>>>
>>> Being able to create a closure in Cython and pass it as an ordinary C
>>> function callback, without having to manage the context manually, would
>>> be a really cool feature! And with such a mini-compiler it is possible.
>>>
>>> (Same idea applies to a concept of "pointer to bound cdef method")
>>>
>> Wow, that's a pretty interesting idea. Does this limit the
>> applicability to certain architectures? Of course even if the context
>> needs to be handled separately, we could make it easier than it is
>> now.
>>
>
> Spending five more minutes on this, ctypes uses the libffi library
> (which is simply bundled with Python, although I haven't probed into
> whether this means it will always be available in a form we can link
> with). It appears to be very user-friendly, and has specific routines
> for creating C closures.
>
> Google it to see platform availability. Apparently closures are not
> available on every platform, but a grep through the source for
> FFI_CLOSURES seem to indicated that the "moxie", "m32r" and "m68k"
> platforms are the only ones without closure support. I can live with
> that :-) Looks to me like libffi lies at the core of a lot of FFI out
> there so that support is rather good.

Very cool. I'm completely fine with relying on this library (bundled
with Python) for our use. The only downside is that documentation
seems a bit sparse, but I'm sure we can figure it out.

> At any rate, it seems tempting to just make Cython cdef closures simply
> be libffi closures.

Yep. This would give a nice way to get bound cdef methods as well
(though I could see value in being able to go back to the unbound
version, not sure how easy that would be).

- Robert

From dagss at student.matnat.uio.no  Tue Nov 30 20:30:18 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 30 Nov 2010 20:30:18 +0100
Subject: [Cython] splitting up Python functions
In-Reply-To: <AANLkTi=f7U9Xx5WgFkOuz38kqw6Rr=V5E0y-PpMk0cki@mail.gmail.com>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>	<4CD811B2.1010108@behnel.de>	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>	<4CDCE8D3.70002@behnel.de>	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>	<4CDCF489.3000108@behnel.de>	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>	<4CDDAC83.9030903@behnel.de>
	<4CEC3EA6.9040002@behnel.de>	<AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>	<4CF0BA9F.9060304@behnel.de>
	<4CF40CAE.6070609@canterbury.ac.nz>	<AANLkTi=yndmiv0+DhzFTAOmA20sHK6zCxUVCx6ouqrbZ@mail.gmail.com>	<AANLkTimkDb_Z2BtG2w7MuYin7CSm84HU8EVwagsS7Dow@mail.gmail.com>	<AANLkTimK65o16+3DXPM81RMHSDK3UvxOuP5YjrpUyvVD@mail.gmail.com>	<4CF4CE11.5040707@student.matnat.uio.no>	<AANLkTimC=0cTaWTR54wpTGjYdsdkLVAXir3vY+uGhV_h@mail.gmail.com>	<4CF547E5.1080409@student.matnat.uio.no>
	<AANLkTi=f7U9Xx5WgFkOuz38kqw6Rr=V5E0y-PpMk0cki@mail.gmail.com>
Message-ID: <4CF550CA.3090106@student.matnat.uio.no>

On 11/30/2010 08:26 PM, Robert Bradshaw wrote:
> On Tue, Nov 30, 2010 at 10:52 AM, Dag Sverre Seljebotn
> <dagss at student.matnat.uio.no>  wrote:
>    
>> On 11/30/2010 07:28 PM, Robert Bradshaw wrote:
>>      
>>> On Tue, Nov 30, 2010 at 2:12 AM, Dag Sverre Seljebotn
>>> <dagss at student.matnat.uio.no>    wrote:
>>>
>>>        
>>>> On 11/30/2010 06:35 AM, Robert Bradshaw wrote:
>>>>
>>>>          
>>>>> On Mon, Nov 29, 2010 at 9:20 PM, Vitja Makarov<vitja.makarov at gmail.com>      wrote:
>>>>>
>>>>>
>>>>>            
>>>>>> 2010/11/30 Robert Bradshaw<robertwb at math.washington.edu>:
>>>>>>
>>>>>>
>>>>>>              
>>>>>>> On Mon, Nov 29, 2010 at 12:27 PM, Greg Ewing
>>>>>>> <greg.ewing at canterbury.ac.nz>      wrote:
>>>>>>>
>>>>>>>
>>>>>>>                
>>>>>>>> Stefan Behnel wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                  
>>>>>>>>> * it makes the inner C function directly callable, thus making it easier to
>>>>>>>>> implement things like cpdef functions and generators on top.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                    
>>>>>>>> If you mention the name of such a function without calling it,
>>>>>>>> does it refer to the C function or the Python function?
>>>>>>>>
>>>>>>>>
>>>>>>>>                  
>>>>>>> That would depend on the context in which it's being used.
>>>>>>> Essentially, the as_variable attribute would be set, allowing it to
>>>>>>> coerce to a Python object if need be.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>                
>>>>>> I see problem with closures here where should scope object be created
>>>>>> in C function or in wrapper?
>>>>>>
>>>>>>
>>>>>>              
>>>>> The only handle you can get of a closure object is a Python one.
>>>>> (Well, eventually we want to have cdef closures, but we're not there
>>>>> yet, and wouldn't be compatible with cdef functions--we'd probably
>>>>> expose them as structs with a state and function pointer attributes.)
>>>>>
>>>>>
>>>>>            
>>>> Just for reference: Carl recently made me aware that ctypes contain some
>>>> code to proxy Python functions to C function pointers. And apparently it
>>>> contains a small compiler that creates new CPU code on demand for this.
>>>> I'm not sure how well that would be exposed for our purposes, if it has
>>>> a C API or not. (I haven't really looked at this myself.)
>>>>
>>>> Being able to create a closure in Cython and pass it as an ordinary C
>>>> function callback, without having to manage the context manually, would
>>>> be a really cool feature! And with such a mini-compiler it is possible.
>>>>
>>>> (Same idea applies to a concept of "pointer to bound cdef method")
>>>>
>>>>          
>>> Wow, that's a pretty interesting idea. Does this limit the
>>> applicability to certain architectures? Of course even if the context
>>> needs to be handled separately, we could make it easier than it is
>>> now.
>>>
>>>        
>> Spending five more minutes on this, ctypes uses the libffi library
>> (which is simply bundled with Python, although I haven't probed into
>> whether this means it will always be available in a form we can link
>> with). It appears to be very user-friendly, and has specific routines
>> for creating C closures.
>>
>> Google it to see platform availability. Apparently closures are not
>> available on every platform, but a grep through the source for
>> FFI_CLOSURES seem to indicated that the "moxie", "m32r" and "m68k"
>> platforms are the only ones without closure support. I can live with
>> that :-) Looks to me like libffi lies at the core of a lot of FFI out
>> there so that support is rather good.
>>      
> Very cool. I'm completely fine with relying on this library (bundled
> with Python) for our use. The only downside is that documentation
> seems a bit sparse, but I'm sure we can figure it out.
>    

There's an "info" file in the doc subdir in the tarball, not sure if you 
saw that. It has some examples of use at least.

Dag Sverre


>    
>> At any rate, it seems tempting to just make Cython cdef closures simply
>> be libffi closures.
>>      
> Yep. This would give a nice way to get bound cdef methods as well
> (though I could see value in being able to go back to the unbound
> version, not sure how easy that would be).
>
> - Robert
> _______________________________________________
> Cython-dev mailing list
> Cython-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/cython-dev
>    


From vitja.makarov at gmail.com  Tue Nov 30 22:04:27 2010
From: vitja.makarov at gmail.com (Vitja Makarov)
Date: Wed, 1 Dec 2010 00:04:27 +0300
Subject: [Cython] splitting up Python functions
In-Reply-To: <4CF550CA.3090106@student.matnat.uio.no>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>
	<4CD811B2.1010108@behnel.de>
	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>
	<4CDCE8D3.70002@behnel.de>
	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>
	<4CDCF489.3000108@behnel.de>
	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>
	<4CDDAC83.9030903@behnel.de> <4CEC3EA6.9040002@behnel.de>
	<AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>
	<4CF0BA9F.9060304@behnel.de> <4CF40CAE.6070609@canterbury.ac.nz>
	<AANLkTi=yndmiv0+DhzFTAOmA20sHK6zCxUVCx6ouqrbZ@mail.gmail.com>
	<AANLkTimkDb_Z2BtG2w7MuYin7CSm84HU8EVwagsS7Dow@mail.gmail.com>
	<AANLkTimK65o16+3DXPM81RMHSDK3UvxOuP5YjrpUyvVD@mail.gmail.com>
	<4CF4CE11.5040707@student.matnat.uio.no>
	<AANLkTimC=0cTaWTR54wpTGjYdsdkLVAXir3vY+uGhV_h@mail.gmail.com>
	<4CF547E5.1080409@student.matnat.uio.no>
	<AANLkTi=f7U9Xx5WgFkOuz38kqw6Rr=V5E0y-PpMk0cki@mail.gmail.com>
	<4CF550CA.3090106@student.matnat.uio.no>
Message-ID: <AANLkTinCpz_Y1zDtEREsdLtjfSQurJpjaMcsLLP1tZ8d@mail.gmail.com>

2010/11/30 Dag Sverre Seljebotn <dagss at student.matnat.uio.no>:
> On 11/30/2010 08:26 PM, Robert Bradshaw wrote:
>> On Tue, Nov 30, 2010 at 10:52 AM, Dag Sverre Seljebotn
>> <dagss at student.matnat.uio.no> ?wrote:
>>
>>> On 11/30/2010 07:28 PM, Robert Bradshaw wrote:
>>>
>>>> On Tue, Nov 30, 2010 at 2:12 AM, Dag Sverre Seljebotn
>>>> <dagss at student.matnat.uio.no> ? ?wrote:
>>>>
>>>>
>>>>> On 11/30/2010 06:35 AM, Robert Bradshaw wrote:
>>>>>
>>>>>
>>>>>> On Mon, Nov 29, 2010 at 9:20 PM, Vitja Makarov<vitja.makarov at gmail.com> ? ? ?wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> 2010/11/30 Robert Bradshaw<robertwb at math.washington.edu>:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On Mon, Nov 29, 2010 at 12:27 PM, Greg Ewing
>>>>>>>> <greg.ewing at canterbury.ac.nz> ? ? ?wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Stefan Behnel wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> * it makes the inner C function directly callable, thus making it easier to
>>>>>>>>>> implement things like cpdef functions and generators on top.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> If you mention the name of such a function without calling it,
>>>>>>>>> does it refer to the C function or the Python function?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> That would depend on the context in which it's being used.
>>>>>>>> Essentially, the as_variable attribute would be set, allowing it to
>>>>>>>> coerce to a Python object if need be.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> I see problem with closures here where should scope object be created
>>>>>>> in C function or in wrapper?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> The only handle you can get of a closure object is a Python one.
>>>>>> (Well, eventually we want to have cdef closures, but we're not there
>>>>>> yet, and wouldn't be compatible with cdef functions--we'd probably
>>>>>> expose them as structs with a state and function pointer attributes.)
>>>>>>
>>>>>>
>>>>>>
>>>>> Just for reference: Carl recently made me aware that ctypes contain some
>>>>> code to proxy Python functions to C function pointers. And apparently it
>>>>> contains a small compiler that creates new CPU code on demand for this.
>>>>> I'm not sure how well that would be exposed for our purposes, if it has
>>>>> a C API or not. (I haven't really looked at this myself.)
>>>>>
>>>>> Being able to create a closure in Cython and pass it as an ordinary C
>>>>> function callback, without having to manage the context manually, would
>>>>> be a really cool feature! And with such a mini-compiler it is possible.
>>>>>
>>>>> (Same idea applies to a concept of "pointer to bound cdef method")
>>>>>
>>>>>
>>>> Wow, that's a pretty interesting idea. Does this limit the
>>>> applicability to certain architectures? Of course even if the context
>>>> needs to be handled separately, we could make it easier than it is
>>>> now.
>>>>
>>>>
>>> Spending five more minutes on this, ctypes uses the libffi library
>>> (which is simply bundled with Python, although I haven't probed into
>>> whether this means it will always be available in a form we can link
>>> with). It appears to be very user-friendly, and has specific routines
>>> for creating C closures.
>>>
>>> Google it to see platform availability. Apparently closures are not
>>> available on every platform, but a grep through the source for
>>> FFI_CLOSURES seem to indicated that the "moxie", "m32r" and "m68k"
>>> platforms are the only ones without closure support. I can live with
>>> that :-) Looks to me like libffi lies at the core of a lot of FFI out
>>> there so that support is rather good.
>>>
>> Very cool. I'm completely fine with relying on this library (bundled
>> with Python) for our use. The only downside is that documentation
>> seems a bit sparse, but I'm sure we can figure it out.
>>
>
> There's an "info" file in the doc subdir in the tarball, not sure if you
> saw that. It has some examples of use at least.
>
> Dag Sverre
>
>
>>
>>> At any rate, it seems tempting to just make Cython cdef closures simply
>>> be libffi closures.
>>>
>> Yep. This would give a nice way to get bound cdef methods as well
>> (though I could see value in being able to go back to the unbound
>> version, not sure how easy that would be).
>>
>> - Robert
>> _______________________________________________
>> Cython-dev mailing list
>> Cython-dev at codespeak.net
>> http://codespeak.net/mailman/listinfo/cython-dev
>>

I don't like the idea to add one more external dependency. But if that
really helps its okay.

Also libffi is not supported on blackfin architecture :(

Here is simple example from libffi documentation:
     #include <stdio.h>
     #include <ffi.h>

     /* Acts like puts with the file given at time of enclosure. */
     void puts_binding(ffi_cif *cif, unsigned int *ret, void* args[],
                       FILE *stream)
     {
       *ret = fputs(*(char **)args[0], stream);
     }

     int main()
     {
       ffi_cif cif;
       ffi_type *args[1];
       ffi_closure *closure;

       int (*bound_puts)(char *);
       int rc;

       /* Allocate closure and bound_puts */
       closure = ffi_closure_alloc(sizeof(ffi_closure), &bound_puts);

       if (closure)
         {
           /* Initialize the argument info vectors */
           args[0] = &ffi_type_pointer;

           /* Initialize the cif */
           if (ffi_prep_cif(&cif, FFI_DEFAULT_ABI, 1,
                            &ffi_type_uint, args) == FFI_OK)
             {
               /* Initialize the closure, setting stream to stdout */
               if (ffi_prep_closure_loc(closure, &cif, puts_binding,
                                        stdout, bound_puts) == FFI_OK)
                 {
                   rc = bound_puts("Hello World!");
                   /* rc now holds the result of the call to fputs */
                 }
             }
         }

       /* Deallocate both closure, and bound_puts */
       ffi_closure_free(closure);

       return 0;
     }

I don't actually understand how do you want to use libffi in Cython.
Can you provide some usecase, please?

vitja.

From robertwb at math.washington.edu  Tue Nov 30 22:31:15 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 30 Nov 2010 13:31:15 -0800
Subject: [Cython] splitting up Python functions
In-Reply-To: <AANLkTinCpz_Y1zDtEREsdLtjfSQurJpjaMcsLLP1tZ8d@mail.gmail.com>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>
	<4CD811B2.1010108@behnel.de>
	<AANLkTin9g8JGQjQybJaVeUGqGsjJ9cpVbktL+7ZPSrE+@mail.gmail.com>
	<4CDCE8D3.70002@behnel.de>
	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>
	<4CDCF489.3000108@behnel.de>
	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>
	<4CDDAC83.9030903@behnel.de> <4CEC3EA6.9040002@behnel.de>
	<AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>
	<4CF0BA9F.9060304@behnel.de> <4CF40CAE.6070609@canterbury.ac.nz>
	<AANLkTi=yndmiv0+DhzFTAOmA20sHK6zCxUVCx6ouqrbZ@mail.gmail.com>
	<AANLkTimkDb_Z2BtG2w7MuYin7CSm84HU8EVwagsS7Dow@mail.gmail.com>
	<AANLkTimK65o16+3DXPM81RMHSDK3UvxOuP5YjrpUyvVD@mail.gmail.com>
	<4CF4CE11.5040707@student.matnat.uio.no>
	<AANLkTimC=0cTaWTR54wpTGjYdsdkLVAXir3vY+uGhV_h@mail.gmail.com>
	<4CF547E5.1080409@student.matnat.uio.no>
	<AANLkTi=f7U9Xx5WgFkOuz38kqw6Rr=V5E0y-PpMk0cki@mail.gmail.com>
	<4CF550CA.3090106@student.matnat.uio.no>
	<AANLkTinCpz_Y1zDtEREsdLtjfSQurJpjaMcsLLP1tZ8d@mail.gmail.com>
Message-ID: <AANLkTin2Rh6WNCcYrV0nWPDGhgqrCVnZqLC4qYMzpGZ3@mail.gmail.com>

On Tue, Nov 30, 2010 at 1:04 PM, Vitja Makarov <vitja.makarov at gmail.com> wrote:

> I don't like the idea to add one more external dependency. But if that
> really helps its okay.

It's not an external dependency if it ships with Python. Actually,
ctypes didn't ship with Python until 2.5, so that may be an issue. I'm
thinking we'd only use it where we couldn't do it otherwise.

> Also libffi is not supported on blackfin architecture :(

Well, it'd probably be a longer time before we supported blackfin if
we were to roll our own... Do you have any other alternatives? I
suppose we could compile a fixed number of dummy functions

    int foo1(int arg1, int arg2) { return func_array[1](data_array[1],
arg1, arg2); }

but then we'd have to decide the upper limit ahead of time, and it
seems rather messy and wasteful.

> Here is simple example from libffi documentation:

...

> I don't actually understand how do you want to use libffi in Cython.
> Can you provide some usecase, please?

Suppose you have a c library that takes a function pointer callback,
and you want to use a Python callable as the callback. To do this you
would create a closure that translates the arguments and has the
Python callable as the state. Another usecase would be for actual
closures, e.g.

    cdef double (*add_n(double n))(double):
        cdef f(double a):
            return a + n
        return &f

    cdef double (*my_func)(double) = add_n(3)

(I have to admit that C function pointer syntax is pretty atrocious...
perhaps we can offer a better alternative?) They'd be useful for bound
cdef methods as well, so one could do something like


cdef extern from ...:
     double integrate(double (*f)(double), double a, double b)

cdef class MyEvaluator:
    cdef double eval(double):
        raise NotImplementedError # override

cdef MyEvaluator f = ...
print integrate(f.eval, 0, 10)

It's possible that we could handle much of this ourselves with a
struct containing a (raw) function pointer and void* state, but we'd
need something more to pass things to external libraries.

- Robert

From dagss at student.matnat.uio.no  Tue Nov 30 22:36:52 2010
From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn)
Date: Tue, 30 Nov 2010 22:36:52 +0100
Subject: [Cython] splitting up Python functions
In-Reply-To: <AANLkTinCpz_Y1zDtEREsdLtjfSQurJpjaMcsLLP1tZ8d@mail.gmail.com>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>	<4CDCE8D3.70002@behnel.de>	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>	<4CDCF489.3000108@behnel.de>	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>	<4CDDAC83.9030903@behnel.de>
	<4CEC3EA6.9040002@behnel.de>	<AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>	<4CF0BA9F.9060304@behnel.de>
	<4CF40CAE.6070609@canterbury.ac.nz>	<AANLkTi=yndmiv0+DhzFTAOmA20sHK6zCxUVCx6ouqrbZ@mail.gmail.com>	<AANLkTimkDb_Z2BtG2w7MuYin7CSm84HU8EVwagsS7Dow@mail.gmail.com>	<AANLkTimK65o16+3DXPM81RMHSDK3UvxOuP5YjrpUyvVD@mail.gmail.com>	<4CF4CE11.5040707@student.matnat.uio.no>	<AANLkTimC=0cTaWTR54wpTGjYdsdkLVAXir3vY+uGhV_h@mail.gmail.com>	<4CF547E5.1080409@student.matnat.uio.no>	<AANLkTi=f7U9Xx5WgFkOuz38kqw6Rr=V5E0y-PpMk0cki@mail.gmail.com>	<4CF550CA.3090106@student.matnat.uio.no>
	<AANLkTinCpz_Y1zDtEREsdLtjfSQurJpjaMcsLLP1tZ8d@mail.gmail.com>
Message-ID: <4CF56E74.5050605@student.matnat.uio.no>

On 11/30/2010 10:04 PM, Vitja Makarov wrote:
> 2010/11/30 Dag Sverre Seljebotn<dagss at student.matnat.uio.no>:
>    
>> On 11/30/2010 08:26 PM, Robert Bradshaw wrote:
>>      
>>> On Tue, Nov 30, 2010 at 10:52 AM, Dag Sverre Seljebotn
>>> <dagss at student.matnat.uio.no>    wrote:
>>>
>>>        
>>>> On 11/30/2010 07:28 PM, Robert Bradshaw wrote:
>>>>
>>>>          
>>>>> On Tue, Nov 30, 2010 at 2:12 AM, Dag Sverre Seljebotn
>>>>> <dagss at student.matnat.uio.no>      wrote:
>>>>>
>>>>>
>>>>>            
>>>>>> On 11/30/2010 06:35 AM, Robert Bradshaw wrote:
>>>>>>
>>>>>>
>>>>>>              
>>>>>>> On Mon, Nov 29, 2010 at 9:20 PM, Vitja Makarov<vitja.makarov at gmail.com>        wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>                
>>>>>>>> 2010/11/30 Robert Bradshaw<robertwb at math.washington.edu>:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                  
>>>>>>>>> On Mon, Nov 29, 2010 at 12:27 PM, Greg Ewing
>>>>>>>>> <greg.ewing at canterbury.ac.nz>        wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                    
>>>>>>>>>> Stefan Behnel wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                      
>>>>>>>>>>> * it makes the inner C function directly callable, thus making it easier to
>>>>>>>>>>> implement things like cpdef functions and generators on top.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>                        
>>>>>>>>>> If you mention the name of such a function without calling it,
>>>>>>>>>> does it refer to the C function or the Python function?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>                      
>>>>>>>>> That would depend on the context in which it's being used.
>>>>>>>>> Essentially, the as_variable attribute would be set, allowing it to
>>>>>>>>> coerce to a Python object if need be.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>                    
>>>>>>>> I see problem with closures here where should scope object be created
>>>>>>>> in C function or in wrapper?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>                  
>>>>>>> The only handle you can get of a closure object is a Python one.
>>>>>>> (Well, eventually we want to have cdef closures, but we're not there
>>>>>>> yet, and wouldn't be compatible with cdef functions--we'd probably
>>>>>>> expose them as structs with a state and function pointer attributes.)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>                
>>>>>> Just for reference: Carl recently made me aware that ctypes contain some
>>>>>> code to proxy Python functions to C function pointers. And apparently it
>>>>>> contains a small compiler that creates new CPU code on demand for this.
>>>>>> I'm not sure how well that would be exposed for our purposes, if it has
>>>>>> a C API or not. (I haven't really looked at this myself.)
>>>>>>
>>>>>> Being able to create a closure in Cython and pass it as an ordinary C
>>>>>> function callback, without having to manage the context manually, would
>>>>>> be a really cool feature! And with such a mini-compiler it is possible.
>>>>>>
>>>>>> (Same idea applies to a concept of "pointer to bound cdef method")
>>>>>>
>>>>>>
>>>>>>              
>>>>> Wow, that's a pretty interesting idea. Does this limit the
>>>>> applicability to certain architectures? Of course even if the context
>>>>> needs to be handled separately, we could make it easier than it is
>>>>> now.
>>>>>
>>>>>
>>>>>            
>>>> Spending five more minutes on this, ctypes uses the libffi library
>>>> (which is simply bundled with Python, although I haven't probed into
>>>> whether this means it will always be available in a form we can link
>>>> with). It appears to be very user-friendly, and has specific routines
>>>> for creating C closures.
>>>>
>>>> Google it to see platform availability. Apparently closures are not
>>>> available on every platform, but a grep through the source for
>>>> FFI_CLOSURES seem to indicated that the "moxie", "m32r" and "m68k"
>>>> platforms are the only ones without closure support. I can live with
>>>> that :-) Looks to me like libffi lies at the core of a lot of FFI out
>>>> there so that support is rather good.
>>>>
>>>>          
>>> Very cool. I'm completely fine with relying on this library (bundled
>>> with Python) for our use. The only downside is that documentation
>>> seems a bit sparse, but I'm sure we can figure it out.
>>>
>>>        
>> There's an "info" file in the doc subdir in the tarball, not sure if you
>> saw that. It has some examples of use at least.
>>
>> Dag Sverre
>>
>>
>>      
>>>        
>>>> At any rate, it seems tempting to just make Cython cdef closures simply
>>>> be libffi closures.
>>>>
>>>>          
>>> Yep. This would give a nice way to get bound cdef methods as well
>>> (though I could see value in being able to go back to the unbound
>>> version, not sure how easy that would be).
>>>
>>> - Robert
>>> _______________________________________________
>>> Cython-dev mailing list
>>> Cython-dev at codespeak.net
>>> http://codespeak.net/mailman/listinfo/cython-dev
>>>
>>>        
> I don't like the idea to add one more external dependency. But if that
> really helps its okay.
>    

If we can manage to link with the libffi shipped with CPython it's not 
really adding a dependency... Also, this is for a feature that's not in 
Cython today, so it's not like we're breaking anything anyway.

> Also libffi is not supported on blackfin architecture :(
>    

I'm not familiar with that architecture. Do you know if CPython is 
supported on it? (If so I assume with disabled ctypes?)

> Here is simple example from libffi documentation:
>       #include<stdio.h>
>       #include<ffi.h>
>
>       /* Acts like puts with the file given at time of enclosure. */
>       void puts_binding(ffi_cif *cif, unsigned int *ret, void* args[],
>                         FILE *stream)
>       {
>         *ret = fputs(*(char **)args[0], stream);
>       }
>
>       int main()
>       {
>         ffi_cif cif;
>         ffi_type *args[1];
>         ffi_closure *closure;
>
>         int (*bound_puts)(char *);
>         int rc;
>
>         /* Allocate closure and bound_puts */
>         closure = ffi_closure_alloc(sizeof(ffi_closure),&bound_puts);
>
>         if (closure)
>           {
>             /* Initialize the argument info vectors */
>             args[0] =&ffi_type_pointer;
>
>             /* Initialize the cif */
>             if (ffi_prep_cif(&cif, FFI_DEFAULT_ABI, 1,
>                              &ffi_type_uint, args) == FFI_OK)
>               {
>                 /* Initialize the closure, setting stream to stdout */
>                 if (ffi_prep_closure_loc(closure,&cif, puts_binding,
>                                          stdout, bound_puts) == FFI_OK)
>                   {
>                     rc = bound_puts("Hello World!");
>                     /* rc now holds the result of the call to fputs */
>                   }
>               }
>           }
>
>         /* Deallocate both closure, and bound_puts */
>         ffi_closure_free(closure);
>
>         return 0;
>       }
>
> I don't actually understand how do you want to use libffi in Cython.
> Can you provide some usecase, please?
>    

Sure. I want to do this:

cdef extern from "mylib.h":
     cdef double integrate(double from, double to_, double 
(*func)(double x)) # Not sure, I can never get this right

cdef extern from "math.h":
     cdef double sin(double)

def integrate_scaled_sin(from, to_, double a):
     cdef double scaled_sin(double x):
         return sin(a * x) # all C
     return integrate(from, to_, &scaled_sin)

AND, since mylib.h is written by your standard numerical scientist, it's 
not a stretch to assume that it doesn't allow for a context pointer, 
like it should have.

I don't really understand the example fully, but it looks like you'd

a) Have some normal static "pyx_scaled_sin" function that would fill the 
role of puts_binding above; reading "a" from its argument list

b) Inside the implementation of integrate_scaled_sin, one would use 
ffi_closure_alloc and ffi_prep_closure_loc to bind the local variables 
and get a simple pointer "double (*func)(double)".

c) This pointer can be passed directly to the "integrate" function 
above. When called, the libffi magic kicks in and calls pyx_scaled_sin 
with the correct (enclosured) arguments.

Dag Sverre


From robert.kern at gmail.com  Tue Nov 30 22:57:32 2010
From: robert.kern at gmail.com (Robert Kern)
Date: Tue, 30 Nov 2010 15:57:32 -0600
Subject: [Cython] splitting up Python functions
In-Reply-To: <AANLkTin2Rh6WNCcYrV0nWPDGhgqrCVnZqLC4qYMzpGZ3@mail.gmail.com>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>	<4CDCF489.3000108@behnel.de>	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>	<4CDDAC83.9030903@behnel.de>
	<4CEC3EA6.9040002@behnel.de>	<AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>	<4CF0BA9F.9060304@behnel.de>
	<4CF40CAE.6070609@canterbury.ac.nz>	<AANLkTi=yndmiv0+DhzFTAOmA20sHK6zCxUVCx6ouqrbZ@mail.gmail.com>	<AANLkTimkDb_Z2BtG2w7MuYin7CSm84HU8EVwagsS7Dow@mail.gmail.com>	<AANLkTimK65o16+3DXPM81RMHSDK3UvxOuP5YjrpUyvVD@mail.gmail.com>	<4CF4CE11.5040707@student.matnat.uio.no>	<AANLkTimC=0cTaWTR54wpTGjYdsdkLVAXir3vY+uGhV_h@mail.gmail.com>	<4CF547E5.1080409@student.matnat.uio.no>	<AANLkTi=f7U9Xx5WgFkOuz38kqw6Rr=V5E0y-PpMk0cki@mail.gmail.com>	<4CF550CA.3090106@student.matnat.uio.no>	<AANLkTinCpz_Y1zDtEREsdLtjfSQurJpjaMcsLLP1tZ8d@mail.gmail.com>
	<AANLkTin2Rh6WNCcYrV0nWPDGhgqrCVnZqLC4qYMzpGZ3@mail.gmail.com>
Message-ID: <id3s0c$br0$1@dough.gmane.org>

On 11/30/10 3:31 PM, Robert Bradshaw wrote:
> On Tue, Nov 30, 2010 at 1:04 PM, Vitja Makarov<vitja.makarov at gmail.com>  wrote:
>
>> I don't like the idea to add one more external dependency. But if that
>> really helps its okay.
>
> It's not an external dependency if it ships with Python. Actually,
> ctypes didn't ship with Python until 2.5, so that may be an issue. I'm
> thinking we'd only use it where we couldn't do it otherwise.

ctypes/libffi is deliberately considered an optional component of Python for 
several reasons, one of which is the lack of support on certain platforms. I 
would say that it exists in a somewhat unique middle ground between "ships with 
Python" and "external dependency".

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
  that is made terrible by our own mad attempt to interpret it as though it had
  an underlying truth."
   -- Umberto Eco


From robertwb at math.washington.edu  Tue Nov 30 23:01:57 2010
From: robertwb at math.washington.edu (Robert Bradshaw)
Date: Tue, 30 Nov 2010 14:01:57 -0800
Subject: [Cython] splitting up Python functions
In-Reply-To: <id3s0c$br0$1@dough.gmane.org>
References: <AANLkTikAzxMQCTbcOsr04CD1_ybof+fZkY6_S1cGeVPb@mail.gmail.com>
	<AANLkTin5kk6PN39SToBEvyoz9AkhEZNzUDTzFNDDG8i=@mail.gmail.com>
	<4CDCF489.3000108@behnel.de>
	<AANLkTi=TVvQZLkdVWGV9Sa_mesKq2H2OhJq8eAYEvAQ+@mail.gmail.com>
	<4CDDAC83.9030903@behnel.de> <4CEC3EA6.9040002@behnel.de>
	<AANLkTintaofiAiqruLuG5kMBDtxdi7NwdQVJkNtDHsrL@mail.gmail.com>
	<4CF0BA9F.9060304@behnel.de> <4CF40CAE.6070609@canterbury.ac.nz>
	<AANLkTi=yndmiv0+DhzFTAOmA20sHK6zCxUVCx6ouqrbZ@mail.gmail.com>
	<AANLkTimkDb_Z2BtG2w7MuYin7CSm84HU8EVwagsS7Dow@mail.gmail.com>
	<AANLkTimK65o16+3DXPM81RMHSDK3UvxOuP5YjrpUyvVD@mail.gmail.com>
	<4CF4CE11.5040707@student.matnat.uio.no>
	<AANLkTimC=0cTaWTR54wpTGjYdsdkLVAXir3vY+uGhV_h@mail.gmail.com>
	<4CF547E5.1080409@student.matnat.uio.no>
	<AANLkTi=f7U9Xx5WgFkOuz38kqw6Rr=V5E0y-PpMk0cki@mail.gmail.com>
	<4CF550CA.3090106@student.matnat.uio.no>
	<AANLkTinCpz_Y1zDtEREsdLtjfSQurJpjaMcsLLP1tZ8d@mail.gmail.com>
	<AANLkTin2Rh6WNCcYrV0nWPDGhgqrCVnZqLC4qYMzpGZ3@mail.gmail.com>
	<id3s0c$br0$1@dough.gmane.org>
Message-ID: <AANLkTikzaMTKUq3C2hzRmMNZKSZZHjEHUL6VGLSJKLjE@mail.gmail.com>

On Tue, Nov 30, 2010 at 1:57 PM, Robert Kern <robert.kern at gmail.com> wrote:
> On 11/30/10 3:31 PM, Robert Bradshaw wrote:
>> On Tue, Nov 30, 2010 at 1:04 PM, Vitja Makarov<vitja.makarov at gmail.com> ?wrote:
>>
>>> I don't like the idea to add one more external dependency. But if that
>>> really helps its okay.
>>
>> It's not an external dependency if it ships with Python. Actually,
>> ctypes didn't ship with Python until 2.5, so that may be an issue. I'm
>> thinking we'd only use it where we couldn't do it otherwise.
>
> ctypes/libffi is deliberately considered an optional component of Python for
> several reasons, one of which is the lack of support on certain platforms. I
> would say that it exists in a somewhat unique middle ground between "ships with
> Python" and "external dependency".

Thanks for the perspective. Are there any good alternatives
(especially that would be less of an "external" dependency)?

- Robert