From hjh at alterras.de  Sun Sep  3 19:07:07 2006
From: hjh at alterras.de (=?ISO-8859-1?Q?Hans-J=FCrgen?= Hay)
Date: Sun, 03 Sep 2006 19:07:07 +0200
Subject: [lxml-dev] Pure Text Elements as XSLT results impossible?
Message-ID: <1157303227.14688.44.camel@hera.local>

XSLT transforms that should result in a TEXT only Element silently
result in some kind of 'None' result class with the text seemingly lost.
Propably caused by Text Elements beeing second class in ElementTree. 
This seems to be very wrong or am I.

Regards
Hans

from lxml import etree
xsl = """<?xml version="1.0"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="text" />
<xsl:template match="/">
   TestText 
</xsl:template>
</xsl:stylesheet>"""
trans = etree.XSLT(etree.XML(xsl))
str(etree.tostring(trans(etree.XML('<test/>'))))

--->> 'None' insted of 'TestText'
----------------------------------------------------------------------
from lxml import etree
xsl = """<?xml version="1.0"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:template match="/">
   <dummy/>  TestText
</xsl:template>
</xsl:stylesheet>"""
trans = etree.XSLT(etree.XML(xsl))
etree.tostring(trans(etree.XML('<test/>')))

-->> '<dummy/>  TestText'  (works)
----------------------------------------------------------------------
from lxml import etree
xsl = """<?xml version="1.0"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:template match="/">
  TestText <dummy/>
</xsl:template>
</xsl:stylesheet>"""
trans = etree.XSLT(etree.XML(xsl))
etree.tostring(trans(etree.XML('<test/>')))

--->> '<dummy/>' ('TestText' in front missing)





From Holger.Joukl at LBBW.de  Mon Sep  4 12:07:06 2006
From: Holger.Joukl at LBBW.de (Holger Joukl)
Date: Mon, 4 Sep 2006 12:07:06 +0200
Subject: [lxml-dev] [objectify] inconsistent element types for different
	element access orders
In-Reply-To: <44F52B80.9080500@gkec.informatik.tu-darmstadt.de>
Message-ID: <OF4CC28437.044AACFC-ONC12571DF.00333009-C12571DF.0037BABB@LBBW.de>

Hi,
depending on how one accesses objectified elements there can
be differences in the resulting element type:

>>> root = objectify.Element('root')
>>> sub = objectify.Element('root')
>>> root.sub = sub
>>> root.sub.x = 1
>>> del root.sub.x
>>> print root
root = None [ObjectifiedElement]
    sub = '' [StringElement]

This yields a StringElement root.sub because root.sub has no element
contents, does have a parent element but not any children.
Whereas

>>> root = objectify.Element('root')
>>> sub = objectify.Element('root')
>>> root.sub = sub
>>> root.sub.x = 1
>>> print root
root = None [ObjectifiedElement]
    sub = None [ObjectifiedElement]
        x = 1 [IntElement]
>>> del root.sub.x
>>> print root
root = None [ObjectifiedElement]
    sub = None [ObjectifiedElement]
>>>

yields an ObjectifiedElement root.sub because I already accessed root.sub
before deleting its child x, thus making it an ObjectifiedElement in the
etree node proxy because at that time it had children.

I'm not sure how to address this problem. For my use case it is desirable
for
- empty content leaf elements to be StringElements, just like it is today:
E.g. when parsing from xml s.th. like '<root><s/></root>' then s should be
a StringElement (empty string, leaf node).
Also when assigning an empty string in objectify this should end up in a
StringElement:
>>> root.s = ''
>>> print root
root = None [ObjectifiedElement]
    s = '' [StringElement]
>>>
- a "structural" element (this is what I use ObjectifiedElements for - they
are supposed to potentially have children) to remain like it is even if
its children get deleted

The problem also manifests in this use case:
>>> root = objectify.Element('root')
>>> root.sub = objectify.Element('whatever')
>>> print root
root = None [ObjectifiedElement]
    sub = '' [StringElement]
>>>
where I would rather have root.sub to be an ObjectifiedElement.

And I'm also the one to blame for the current behaviour because I proposed
parts of the class lookup order to Stefan :-)

Some thoughts:
- maybe disallow DataElements to have children, i.e. disabling __setattr__
and alike for DataElements? Then ObjectifiedElements would need to have
an accessible (string) pyvalue in contrast to current behaviour
- maybe change the time an object is actually registered in the node proxy?
- add an additional "structural" element class that is basically just an
ObjectifiedElement but has an artificial pytype to make it retain its type
and
can be produced by a factory similar to objectify.DataElement?
- just not care about a StringElement acting as a structural element as it
can currently have children too (though it supports the string API parts on
top
of the ObjectifiedElement basic API)?

Greetings,
Holger


Der Inhalt dieser E-Mail ist vertraulich. Falls Sie nicht der angegebene
Empf?nger sind oder falls diese E-Mail irrt?mlich an Sie adressiert wurde,
verst?ndigen Sie bitte den Absender sofort und l?schen Sie die E-Mail
sodann. Das unerlaubte Kopieren sowie die unbefugte ?bermittlung sind nicht
gestattet. Die Sicherheit von ?bermittlungen per E-Mail kann nicht
garantiert werden. Falls Sie eine Best?tigung w?nschen, fordern Sie bitte
den Inhalt der E-Mail als Hardcopy an.

The contents of this  e-mail are confidential. If you are not the named
addressee or if this transmission has been addressed to you in error,
please notify the sender immediately and then delete this e-mail.  Any
unauthorized copying and transmission is forbidden. E-Mail transmission
cannot be guaranteed to be secure. If verification is required, please
request a hard copy version.



From behnel_ml at gkec.informatik.tu-darmstadt.de  Tue Sep  5 15:54:54 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Tue, 05 Sep 2006 15:54:54 +0200
Subject: [lxml-dev] Pure Text Elements as XSLT results impossible?
In-Reply-To: <1157303227.14688.44.camel@hera.local>
References: <1157303227.14688.44.camel@hera.local>
Message-ID: <44FD81AE.4060800@gkec.informatik.tu-darmstadt.de>

Hi Hans-J?rgen,

Hans-J?rgen Hay wrote:
> XSLT transforms that should result in a TEXT only Element silently
> result in some kind of 'None' result class with the text seemingly lost.
> Propably caused by Text Elements beeing second class in ElementTree. 
> 
> from lxml import etree
> xsl = """<?xml version="1.0"?>
> <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
> <xsl:output method="text" />
> <xsl:template match="/">
>    TestText 
> </xsl:template>
> </xsl:stylesheet>"""
> trans = etree.XSLT(etree.XML(xsl))
> str(etree.tostring(trans(etree.XML('<test/>'))))
> 
> --->> 'None' insted of 'TestText'
> ----------------------------------------------------------------------
> from lxml import etree
> xsl = """<?xml version="1.0"?>
> <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
> <xsl:template match="/">
>    <dummy/>  TestText
> </xsl:template>
> </xsl:stylesheet>"""
> trans = etree.XSLT(etree.XML(xsl))
> etree.tostring(trans(etree.XML('<test/>')))
> 
> -->> '<dummy/>  TestText'  (works)
> ----------------------------------------------------------------------
> from lxml import etree
> xsl = """<?xml version="1.0"?>
> <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
> <xsl:template match="/">
>   TestText <dummy/>
> </xsl:template>
> </xsl:stylesheet>"""
> trans = etree.XSLT(etree.XML(xsl))
> etree.tostring(trans(etree.XML('<test/>')))
> 
> --->> '<dummy/>' ('TestText' in front missing)

looks like you forgot to test

str(trans(...))

which is the usage suggested by the documentation.

Stefan

From pawel at praterm.com.pl  Tue Sep  5 17:26:15 2006
From: pawel at praterm.com.pl (=?ISO-8859-2?Q?Pawe=B3_Pa=B3ucha?=)
Date: Tue, 05 Sep 2006 17:26:15 +0200
Subject: [lxml-dev] File name encoding problems
Message-ID: <44FD9717.30802@praterm.com.pl>

Hi!
I'm not sure if it's really an lxml problem, but it looks like...
I'm using lxml version 1.0.3 (from Debian package). My default system 
encoding is ISO-8859-2. I have a simple program:

#!/usr/bin/python
# -*- coding: iso-8859-2 -*-

import lxml.etree
d = lxml.etree.parse('/tmp/?.xml')

The problem is with letter '?' (or any other non-ascii letter) in file 
name. While running program I get something like that:

Traceback (most recent call last):
   File "./p.py", line 6, in ?
     d = lxml.etree.parse('/tmp/??d?.xml')
   File "etree.pyx", line 1615, in etree.parse
   File "parser.pxi", line 687, in etree._parseDocument
   File "apihelpers.pxi", line 343, in etree._utf8
AssertionError: All strings must be Unicode or ASCII

If I try to write path using UTF-8 encoding, the file cannot be found 
(because the path with UTF-8 encoded name does not exists).
This did not happen with python 2.3 - the problem is only with 
python2.4. There's a workaround - reading file to StringIO object and 
then parsing XML from that object. It works fine but it's silly.

I would be very grateful for any suggestions.

Pawe? Pa?ucha

From behnel_ml at gkec.informatik.tu-darmstadt.de  Tue Sep  5 21:48:32 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Tue, 05 Sep 2006 21:48:32 +0200
Subject: [lxml-dev] [objectify] inconsistent element types for different
 element access orders
In-Reply-To: <OF4CC28437.044AACFC-ONC12571DF.00333009-C12571DF.0037BABB@LBBW.de>
References: <OF4CC28437.044AACFC-ONC12571DF.00333009-C12571DF.0037BABB@LBBW.de>
Message-ID: <44FDD490.9000303@gkec.informatik.tu-darmstadt.de>

Hi Holger,

Holger Joukl wrote:
> depending on how one accesses objectified elements there can
> be differences in the resulting element type:
> 
>>>> root = objectify.Element('root')
>>>> sub = objectify.Element('root')
>>>> root.sub = sub
>>>> root.sub.x = 1
>>>> del root.sub.x
>>>> print root
> root = None [ObjectifiedElement]
>     sub = '' [StringElement]
> 
> This yields a StringElement root.sub because root.sub has no element
> contents, does have a parent element but not any children.
> Whereas
> 
>>>> root = objectify.Element('root')
>>>> sub = objectify.Element('root')
>>>> root.sub = sub
>>>> root.sub.x = 1
>>>> print root
> root = None [ObjectifiedElement]
>     sub = None [ObjectifiedElement]
>         x = 1 [IntElement]
>>>> del root.sub.x
>>>> print root
> root = None [ObjectifiedElement]
>     sub = None [ObjectifiedElement]
> 
> yields an ObjectifiedElement root.sub because I already accessed root.sub
> before deleting its child x, thus making it an ObjectifiedElement in the
> etree node proxy because at that time it had children.

It's even worse:

  >>> print sub
  root = None [ObjectifiedElement]
  >>> root.sub = sub
  >>> print root
  root = None [ObjectifiedElement]
      sub = '' [StringElement]
  >>> root.sub.x = 1
  >>> print root
  root = None [ObjectifiedElement]
      sub = '' [StringElement]
          x = 1 [IntElement]

This is pretty wrong. The thing that bothers me is that there should not
actually be a permanent Python reference to root.sub, which would normally
mean that the object should get recreated each time it is accessed. But as the
last command shows, that is not the case.


> I'm not sure how to address this problem. For my use case it is desirable
> for
[snip]
> - a "structural" element (this is what I use ObjectifiedElements for - they
> are supposed to potentially have children) to remain like it is even if
> its children get deleted

That's only a problem if you access the Python reference of the child itself
afterwards, which you normally wouldn't if it's a pure structural element.


> The problem also manifests in this use case:
>>>> root = objectify.Element('root')
>>>> root.sub = objectify.Element('whatever')
>>>> print root
> root = None [ObjectifiedElement]
>     sub = '' [StringElement]
> where I would rather have root.sub to be an ObjectifiedElement.

Sure, but I'd figure that's a rare use case anyway. And if you need it, there
are enough ways to get around it, from parsing to ObjectPath.


> Some thoughts:
> - maybe disallow DataElements to have children, i.e. disabling __setattr__
> and alike for DataElements? Then ObjectifiedElements would need to have
> an accessible (string) pyvalue in contrast to current behaviour

Not a good idea. In that case, things like this would potentially stop working:

  >>> root = objectify.Element('root')
  >>> root.sub = objectify.Element('whatever')
  >>> root.sub.sub = ...

Reason: as it stands now, root.sub would become a StringElement, which would
not accept any children.


> - maybe change the time an object is actually registered in the node proxy?

It's difficult to avoid instantiating element objects when setting and
modifying content. The main reason is that if we don't have a proxy, we have
to clean up the element ourselves, which means code duplication and/or a
tighter code coupling between etree and objectify.


> - add an additional "structural" element class that is basically just an
> ObjectifiedElement but has an artificial pytype to make it retain its type
> and can be produced by a factory similar to objectify.DataElement?

Hmm, we could potentially allow "ObjectifiedElement" as pytype, though I'd
prefer waiting for a really good reason to do that.


> - just not care about a StringElement acting as a structural element as it
> can currently have children too (though it supports the string API parts on
> top of the ObjectifiedElement basic API)?

That leads to the problem I pointed out at the top. What is your actual
reasoning for requiring that empty leaf elements should be StringElements? I
mean, you could always make them StringElements explicitly by setting

  >>> root.a.b.c.d = ''

and you can always explicitly access their String value with ".text".

If we removed that special case, leaf elements that contain strings would
always be StringElements and empty leaves and internal elements would always
be ObjectifiedElements.

That would not change the fact that elements keep their type as long as there
is a Python reference to them, but it would work in a few more cases than it
does now.

Stefan


From Holger.Joukl at LBBW.de  Wed Sep  6 09:44:34 2006
From: Holger.Joukl at LBBW.de (Holger Joukl)
Date: Wed, 6 Sep 2006 09:44:34 +0200
Subject: [lxml-dev] [objectify] inconsistent element types for different
	element access orders
In-Reply-To: <44FDD490.9000303@gkec.informatik.tu-darmstadt.de>
Message-ID: <OF9959F839.F6A29A86-ONC12571E1.002877E9-C12571E1.002AADFB@LBBW.de>

Hi Stefan,

lxml-dev-bounces at codespeak.net schrieb am 05.09.2006 21:48:32:

>
> It's even worse:
>
>   >>> print sub
>   root = None [ObjectifiedElement]
>   >>> root.sub = sub
>   >>> print root
>   root = None [ObjectifiedElement]
>       sub = '' [StringElement]
>   >>> root.sub.x = 1
>   >>> print root
>   root = None [ObjectifiedElement]
>       sub = '' [StringElement]
>           x = 1 [IntElement]
>
> This is pretty wrong. The thing that bothers me is that there should not
> actually be a permanent Python reference to root.sub, which would
normally
> mean that the object should get recreated each time it is accessed. But
as the
> last command shows, that is not the case.
>
> [...]
>
> > The problem also manifests in this use case:
> >>>> root = objectify.Element('root')
> >>>> root.sub = objectify.Element('whatever')
> >>>> print root
> > root = None [ObjectifiedElement]
> >     sub = '' [StringElement]
> > where I would rather have root.sub to be an ObjectifiedElement.
>
> Sure, but I'd figure that's a rare use case anyway. And if you need it,
there
> are enough ways to get around it, from parsing to ObjectPath.

This was just another way to describe the behaviour you put out above.
I want it to be an ObjectifiedElement because I know I'll put children
in it later.

 > > Some thoughts:
> > - maybe disallow DataElements to have children, i.e. disabling
__setattr__
> > and alike for DataElements? Then ObjectifiedElements would need to have
> > an accessible (string) pyvalue in contrast to current behaviour
>
> Not a good idea. In that case, things like this would potentially
> stop working:
>
>   >>> root = objectify.Element('root')
>   >>> root.sub = objectify.Element('whatever')
>   >>> root.sub.sub = ...
>
> Reason: as it stands now, root.sub would become a StringElement, which
would
> not accept any children.
>
>
> > - maybe change the time an object is actually registered in the node
proxy?
>
> It's difficult to avoid instantiating element objects when setting and
> modifying content. The main reason is that if we don't have a proxy, we
have
> to clean up the element ourselves, which means code duplication and/or a
> tighter code coupling between etree and objectify.
>
>
> > - add an additional "structural" element class that is basically just
an
> > ObjectifiedElement but has an artificial pytype to make it retain its
type
> > and can be produced by a factory similar to objectify.DataElement?
>
> Hmm, we could potentially allow "ObjectifiedElement" as pytype, though
I'd
> prefer waiting for a really good reason to do that.

I know, it's not nice. But right now I can't think of another way to force
a leaf to be an ObjectifiedElement.

> > - just not care about a StringElement acting as a structural element as
it
> > can currently have children too (though it supports the string API
parts on
> > top of the ObjectifiedElement basic API)?
>
> That leads to the problem I pointed out at the top. What is your actual
> reasoning for requiring that empty leaf elements should be
StringElements? I
> mean, you could always make them StringElements explicitly by setting
>
>   >>> root.a.b.c.d = ''

Wouldn't this end up in d being an ObjectifiedElement if the logic (empty
leaves
are StringElements) changed?

> and you can always explicitly access their String value with ".text".
>
> If we removed that special case, leaf elements that contain strings would
> always be StringElements and empty leaves and internal elements would
always
> be ObjectifiedElements.
>
> That would not change the fact that elements keep their type as long as
there
> is a Python reference to them, but it would work in a few more cases than
it
> does now.
>

When parsing from XML I need '<root><s>some string</s></root>' to behave
like
'<root><s></s></root>'. For someone processing the data "s" should always
act like a (possibly empty) string.
Your solution would only work for me if ObjectifiedElement got a .pyval
attribute,
too, and its .text was not None but rather '' if no text content is in the
node,
and probably also needed the String API parts.
Much of this stems from the fact the ElementTree elt.text returns None if
there
is no element text instead of '' (but I guess this won't change :-)

Holger


Der Inhalt dieser E-Mail ist vertraulich. Falls Sie nicht der angegebene
Empf?nger sind oder falls diese E-Mail irrt?mlich an Sie adressiert wurde,
verst?ndigen Sie bitte den Absender sofort und l?schen Sie die E-Mail
sodann. Das unerlaubte Kopieren sowie die unbefugte ?bermittlung sind nicht
gestattet. Die Sicherheit von ?bermittlungen per E-Mail kann nicht
garantiert werden. Falls Sie eine Best?tigung w?nschen, fordern Sie bitte
den Inhalt der E-Mail als Hardcopy an.

The contents of this  e-mail are confidential. If you are not the named
addressee or if this transmission has been addressed to you in error,
please notify the sender immediately and then delete this e-mail.  Any
unauthorized copying and transmission is forbidden. E-Mail transmission
cannot be guaranteed to be secure. If verification is required, please
request a hard copy version.



From behnel_ml at gkec.informatik.tu-darmstadt.de  Wed Sep  6 11:27:43 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Wed, 06 Sep 2006 11:27:43 +0200
Subject: [lxml-dev] [objectify] inconsistent element types for different
 element access orders
In-Reply-To: <OF9959F839.F6A29A86-ONC12571E1.002877E9-C12571E1.002AADFB@LBBW.de>
References: <OF9959F839.F6A29A86-ONC12571E1.002877E9-C12571E1.002AADFB@LBBW.de>
Message-ID: <44FE948F.7030702@gkec.informatik.tu-darmstadt.de>



Holger Joukl wrote:
> Stefan Behnel wrote:
>> That leads to the problem I pointed out at the top. What is your actual
>> reasoning for requiring that empty leaf elements should be
>> StringElements? I
>> mean, you could always make them StringElements explicitly by setting
>>
>>   >>> root.a.b.c.d = ''
> 
> Wouldn't this end up in d being an ObjectifiedElement if the logic (empty
> leaves are StringElements) changed?

No. The value is an empty string, not an empty value. So there is text content
in there, it's just of length zero.


> When parsing from XML I need '<root><s>some string</s></root>' to behave
> like
> '<root><s></s></root>'. For someone processing the data "s" should always
> act like a (possibly empty) string.
> Your solution would only work for me if ObjectifiedElement got a .pyval
> attribute,
> too, and its .text was not None but rather '' if no text content is in the
> node,
> and probably also needed the String API parts.


> Much of this stems from the fact the ElementTree elt.text returns None if
> there is no element text instead of '' (but I guess this won't change :-)

It returns '' if the value is '' and it returns None if there is no value.
That already changed to adapt to ET's own behaviour. The parser sees "<a/>"
and "<a></a>" as not having a value. So you will never get an empty string
back from a parsed tree. However, if you set it to '', lxml will continue to
return an empty string and objectify will determine that it is a StringElement.

Maybe you could get by with wrapper functions that add the '' for leafs where
required?

Stefan

From behnel_ml at gkec.informatik.tu-darmstadt.de  Wed Sep  6 11:37:34 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Wed, 06 Sep 2006 11:37:34 +0200
Subject: [lxml-dev] File name encoding problems
In-Reply-To: <44FD9717.30802@praterm.com.pl>
References: <44FD9717.30802@praterm.com.pl>
Message-ID: <44FE96DE.7030101@gkec.informatik.tu-darmstadt.de>

Hi Pawe?,

first thing to note: lxml uses UTF-8 internally, also for filenames, as
libxml2 requires a char sequence for their representation. If your system
can't handle that, we'll have to figure out a way to make it work.

This part of lxml is not much tested, so it would be nice if you could help us
in getting this straight.

Pawe? Pa?ucha wrote:
> I'm using lxml version 1.0.3 (from Debian package). My default system 
> encoding is ISO-8859-2. I have a simple program:
> 
> #!/usr/bin/python
> # -*- coding: iso-8859-2 -*-
> 
> import lxml.etree
> d = lxml.etree.parse('/tmp/?.xml')

Ok, so you're using 8-bit encoded filenames.


> Traceback (most recent call last):
>    File "./p.py", line 6, in ?
>      d = lxml.etree.parse('/tmp/??d?.xml')
>    File "etree.pyx", line 1615, in etree.parse
>    File "parser.pxi", line 687, in etree._parseDocument
>    File "apihelpers.pxi", line 343, in etree._utf8
> AssertionError: All strings must be Unicode or ASCII

Right, I guess that treating the filename with the _utf8() function is not the
right thing to do for 8-bit strings. We should have a separate way of treating
filenames. I'll look into it.


> If I try to write path using UTF-8 encoding, the file cannot be found 
> (because the path with UTF-8 encoded name does not exists).

I assume you get the same file-not-found error if you pass the filename as
unicode string?

   d = lxml.etree.parse(u'/tmp/?.xml')


> This did not happen with python 2.3 - the problem is only with 
> python2.4. There's a workaround - reading file to StringIO object and 
> then parsing XML from that object. It works fine but it's silly.

You can also pass an opened file object.

Stefan


From behnel_ml at gkec.informatik.tu-darmstadt.de  Wed Sep  6 13:09:34 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Wed, 06 Sep 2006 13:09:34 +0200
Subject: [lxml-dev] File name encoding problems
In-Reply-To: <44FE96DE.7030101@gkec.informatik.tu-darmstadt.de>
References: <44FD9717.30802@praterm.com.pl>
	<44FE96DE.7030101@gkec.informatik.tu-darmstadt.de>
Message-ID: <44FEAC6E.1080604@gkec.informatik.tu-darmstadt.de>

Hi Pawe?,

Stefan Behnel wrote:
> first thing to note: lxml uses UTF-8 internally, also for filenames, as
> libxml2 requires a char sequence for their representation. If your system
> can't handle that, we'll have to figure out a way to make it work.
> 
> I guess that treating the filename with the _utf8() function is not the
> right thing to do for 8-bit strings. We should have a separate way of treating
> filenames. I'll look into it.

Here's a patch that might fix your problem. However, it's against the current
trunk (i.e. 1.1 beta), as fixing this problem requires a behavioural change
that will not make it into 1.0.

The web page has information on how to build lxml on Linux, it's pretty easy.

Stefan

-------------- next part --------------
A non-text attachment was scrubbed...
Name: filename_encoding.patch
Type: text/x-patch
Size: 11597 bytes
Desc: not available
Url : http://codespeak.net/pipermail/lxml-dev/attachments/20060906/f43db195/attachment.bin 

From Holger.Joukl at LBBW.de  Wed Sep  6 13:25:08 2006
From: Holger.Joukl at LBBW.de (Holger Joukl)
Date: Wed, 6 Sep 2006 13:25:08 +0200
Subject: [lxml-dev] [objectify] inconsistent element types for different
	element access orders
In-Reply-To: <44FE948F.7030702@gkec.informatik.tu-darmstadt.de>
Message-ID: <OF4BAFE920.B9A94327-ONC12571E1.003CDA53-C12571E1.003EDFA0@LBBW.de>

Stefan Behnel <behnel_ml at gkec.informatik.tu-darmstadt.de> schrieb am
06.09.2006 11:27:43:

> It returns '' if the value is '' and it returns None if there is no
value.
> That already changed to adapt to ET's own behaviour. The parser sees
"<a/>"
> and "<a></a>" as not having a value. So you will never get an empty
string
> back from a parsed tree. However, if you set it to '', lxml will continue
to
> return an empty string and objectify will determine that it is a
> StringElement.
>
> Maybe you could get by with wrapper functions that add the '' for leafs
where
> required?
>

Hm, I could of course "stringify" all empty leaves after parsing, given
that my users aren't accessing the etree/objectify APIs e.g. fromstring()
directly.
But I'd have to iterate over the whole tree for this.

BTW: What do you think about adding .encode(...) to StringElement?

Something we've discussed before: Would it make sense to allow an
ObjectifiedElement instance to change its element.text internally,
like e.g. in its _init() method?
Or do you think it is better to stay explicit, loop over the
tree and replace elements as needed?
My use case is the DatetimeElement class I'm using where I will
probably want to change the text to iso format datetime.

Holger

Der Inhalt dieser E-Mail ist vertraulich. Falls Sie nicht der angegebene
Empf?nger sind oder falls diese E-Mail irrt?mlich an Sie adressiert wurde,
verst?ndigen Sie bitte den Absender sofort und l?schen Sie die E-Mail
sodann. Das unerlaubte Kopieren sowie die unbefugte ?bermittlung sind nicht
gestattet. Die Sicherheit von ?bermittlungen per E-Mail kann nicht
garantiert werden. Falls Sie eine Best?tigung w?nschen, fordern Sie bitte
den Inhalt der E-Mail als Hardcopy an.

The contents of this  e-mail are confidential. If you are not the named
addressee or if this transmission has been addressed to you in error,
please notify the sender immediately and then delete this e-mail.  Any
unauthorized copying and transmission is forbidden. E-Mail transmission
cannot be guaranteed to be secure. If verification is required, please
request a hard copy version.



From oliver.schoenborn at utoronto.ca  Wed Sep  6 17:00:49 2006
From: oliver.schoenborn at utoronto.ca (Oliver Schoenborn)
Date: Wed, 06 Sep 2006 11:00:49 -0400
Subject: [lxml-dev] help: lxml causes wxpython app to crash at exit
Message-ID: <44FEE2A1.60003@utoronto.ca>

Hi, just wondering if anyone else has had same problem and what solution 
they found:

   1. system is RHEL 4, with python 2.3, and latest wxpython from rpm on
      wxpython website; python and wxpython work fine (demo etc)
   2. latest lxml (http://codespeak.net/lxml/) 1.0.3 installed (with
      latest libxml2 and libxslt built from source) and also built lxml
      from source (with latest pyrex etc)
   3. when I open pyalamode (a wx app with a python shell window), then
      do "from lxml import etree" then exit, a seg fault occurs; the
      core dump says it happens during finalization of the python
      interpreter, possibly when terminating some thread.
   4. same thing happens from my own wx app (if I uncomment a line that
      says "from lxml import etree" there is no segfault after exit)
   5. I can use lxml for validation etc from within the wx app, all is
      fine; the problem is *only* when exiting the wx app
   6. this segfault does NOT happen on Windows,
   7. it does NOT happen if I do the above from within python shell in xterm
   8. the problem does NOT happen when just do "import lxml", lxml.etree
      is the problem; etree is a binding to some C code in lxml/etree.c;
      this file is generated by pyrex

Upgrading to python 2.4 will not be easy so I want to make sure I 
exhaust other solutions first.

Thanks,

Oliver


From pawel at praterm.com.pl  Wed Sep  6 22:50:23 2006
From: pawel at praterm.com.pl (=?ISO-8859-2?Q?Pawe=B3_Pa=B3ucha?=)
Date: Wed, 06 Sep 2006 22:50:23 +0200
Subject: [lxml-dev] Building debian packages
Message-ID: <44FF348F.6070605@praterm.com.pl>


Hi, I've got a small patch against current SVN to file doc/build.txt:
* info about changing md5sum in *.dsc file
* command `dpkg -x ...` is incorrect - it should be `dpkg-source -x ...`

Pawe?

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: build-deb.patch
Url: http://codespeak.net/pipermail/lxml-dev/attachments/20060906/a858d812/attachment.diff 

From behnel_ml at gkec.informatik.tu-darmstadt.de  Wed Sep  6 21:21:24 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Wed, 06 Sep 2006 21:21:24 +0200
Subject: [lxml-dev] help: lxml causes wxpython app to crash at exit
In-Reply-To: <44FEE2A1.60003@utoronto.ca>
References: <44FEE2A1.60003@utoronto.ca>
Message-ID: <44FF1FB3.3010305@gkec.informatik.tu-darmstadt.de>

Hi Oliver,

Oliver Schoenborn wrote:
> Hi, just wondering if anyone else has had same problem and what solution 
> they found:
> 
>    1. system is RHEL 4, with python 2.3, and latest wxpython from rpm on
>       wxpython website; python and wxpython work fine (demo etc)
>    2. latest lxml (http://codespeak.net/lxml/) 1.0.3 installed (with
>       latest libxml2 and libxslt built from source) and also built lxml
>       from source (with latest pyrex etc)
>    3. when I open pyalamode (a wx app with a python shell window), then
>       do "from lxml import etree" then exit, a seg fault occurs; the
>       core dump says it happens during finalization of the python
>       interpreter, possibly when terminating some thread.
>    4. same thing happens from my own wx app (if I uncomment a line that
>       says "from lxml import etree" there is no segfault after exit)
>    5. I can use lxml for validation etc from within the wx app, all is
>       fine; the problem is *only* when exiting the wx app
>    6. this segfault does NOT happen on Windows,
>    7. it does NOT happen if I do the above from within python shell in xterm
>    8. the problem does NOT happen when just do "import lxml", lxml.etree
>       is the problem; etree is a binding to some C code in lxml/etree.c;
>       this file is generated by pyrex
> 
> Upgrading to python 2.4 will not be easy so I want to make sure I 
> exhaust other solutions first.

You use a lot of C extensions and libraries and even compiled some of them
from source, so the problem can be virtually everywhere. Since I used wxPython
for a project once, I'd usually suspect the error in there first. The
experience at the time made me learn and appreciate Qt.

You can try running the Python interpreter under Valgrind control to see where
the segfault occurs. There is a command line in lxml/doc/valgrind.txt.

Stefan



From behnel_ml at gkec.informatik.tu-darmstadt.de  Wed Sep  6 22:28:39 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Wed, 06 Sep 2006 22:28:39 +0200
Subject: [lxml-dev] [objectify] inconsistent element types for different
 element access orders
In-Reply-To: <OF4BAFE920.B9A94327-ONC12571E1.003CDA53-C12571E1.003EDFA0@LBBW.de>
References: <OF4BAFE920.B9A94327-ONC12571E1.003CDA53-C12571E1.003EDFA0@LBBW.de>
Message-ID: <44FF2F77.1010106@gkec.informatik.tu-darmstadt.de>

Hi Holger,

Holger Joukl wrote:
> Hm, I could of course "stringify" all empty leaves after parsing, given
> that my users aren't accessing the etree/objectify APIs e.g. fromstring()
> directly.

That's a matter of documentation. The best way is to write a small Python
wrapper around the objectify module and have them import /that/.

      from lxml.objectify import *
      from lxml import objectify

      def fromstring(xml):
           return _fixItUp( objectify.fromstring(xml) )


> But I'd have to iterate over the whole tree for this.

Sure, it's much easier with the right class in place. And having all elements
instantiated during iteration isn't quite the most efficient thing ever. You
could reduce the effort with a smart XPath expression, though.

One thing that comes to my mind is that we could add support for replacing the
default type classes used by ObjectifyElementClassLookup. We could add keyword
arguments so that you could say

      lookup = ObjectifyElementClassLookup(StringElement=MyStringElementClass)

That would currently work for String-, None- and ObjectifiedElement only, as
the others use the data type registry. Maybe we should rather support
something like "default_data_class" and "default_tree_class" (and keep the
NoneElement, which is only used in a well defined case anyway).

Then again, what about "empty_class", "xsi_nil_class" and "tree_class"? Any
preference or comments?


> BTW: What do you think about adding .encode(...) to StringElement?

Python's string objects have 35 documented methods, most of which we could
implement (although some of them, like "index" and "find" already have a
different meaning in etree/objectify). If we consider implementing one, we
should rather have all of them in place. Don't know if it's worth it. As the
documentation says, if you want a real string, use ".text".


> Something we've discussed before: Would it make sense to allow an
> ObjectifiedElement instance to change its element.text internally,
> like e.g. in its _init() method?

I think it would be a good idea to add a method "__setText(s)" to
ObjectifiedDataElement. That would make it available to subclasses and at the
same time make it clear that it is *no* public API.

Stefan



From Holger.Joukl at LBBW.de  Fri Sep  8 10:22:16 2006
From: Holger.Joukl at LBBW.de (Holger Joukl)
Date: Fri, 8 Sep 2006 10:22:16 +0200
Subject: [lxml-dev] [objectify] inconsistent element types for different
	element access orders
In-Reply-To: <44FF2F77.1010106@gkec.informatik.tu-darmstadt.de>
Message-ID: <OF92EA38A6.6CCFB5C3-ONC12571E3.002AEF11-C12571E3.002E21EC@LBBW.de>

Hi Stefan,

lxml-dev-bounces at codespeak.net schrieb am 06.09.2006 22:28:39:

> One thing that comes to my mind is that we could add support for
replacing the
> default type classes used by ObjectifyElementClassLookup. We could add
keyword
> arguments so that you could say
>
>       lookup =
ObjectifyElementClassLookup(StringElement=MyStringElementClass)
>
> That would currently work for String-, None- and ObjectifiedElement only,
as
> the others use the data type registry. Maybe we should rather support
> something like "default_data_class" and "default_tree_class" (and keep
the
> NoneElement, which is only used in a well defined case anyway).
>
> Then again, what about "empty_class", "xsi_nil_class" and "tree_class"?
Any
> preference or comments?

I'm perfectly happy with the current solution except for setattr-ing a
'structural element' and wanting this to remain instead of becoming a
StringElement. I don't quite see how a different default data class or
different tree class achieve this?
So I'm back to suggesting a TreeElement() factory (not the best name,
maybe)
returning an ObjectifiedElement with a new pytype='ObjectifiedElement'
which keeps it from becoming a string. I think that's still nicer than
"stringifying" every single empty leaf when parsing from XML.

> > BTW: What do you think about adding .encode(...) to StringElement?
>
> Python's string objects have 35 documented methods, most of which we
could
> implement (although some of them, like "index" and "find" already have a
> different meaning in etree/objectify). If we consider implementing one,
we
> should rather have all of them in place. Don't know if it's worth it. As
the
> documentation says, if you want a real string, use ".text".

Guess you're right, let's keep things simple. I was worried about e.g.
printing
to stdout but then again someone should probably convert all data to
unicode and then
encode to his preferred encoding, as he'll have to deal with numbers and
stuff
anyway (which don't have an .encode method, either).

> > Something we've discussed before: Would it make sense to allow an
> > ObjectifiedElement instance to change its element.text internally,
> > like e.g. in its _init() method?
>
> I think it would be a good idea to add a method "__setText(s)" to
> ObjectifiedDataElement. That would make it available to subclasses and at
the
> same time make it clear that it is *no* public API.

+1 for that.
If somebody wants a user to be able to modify in place he can then also
add a public method to his data classes.

Holger





Der Inhalt dieser E-Mail ist vertraulich. Falls Sie nicht der angegebene
Empf?nger sind oder falls diese E-Mail irrt?mlich an Sie adressiert wurde,
verst?ndigen Sie bitte den Absender sofort und l?schen Sie die E-Mail
sodann. Das unerlaubte Kopieren sowie die unbefugte ?bermittlung sind nicht
gestattet. Die Sicherheit von ?bermittlungen per E-Mail kann nicht
garantiert werden. Falls Sie eine Best?tigung w?nschen, fordern Sie bitte
den Inhalt der E-Mail als Hardcopy an.

The contents of this  e-mail are confidential. If you are not the named
addressee or if this transmission has been addressed to you in error,
please notify the sender immediately and then delete this e-mail.  Any
unauthorized copying and transmission is forbidden. E-Mail transmission
cannot be guaranteed to be secure. If verification is required, please
request a hard copy version.



From behnel_ml at gkec.informatik.tu-darmstadt.de  Thu Sep  7 17:48:30 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Thu, 07 Sep 2006 17:48:30 +0200
Subject: [lxml-dev] File name encoding problems
In-Reply-To: <44FEBDBF.2020202@praterm.com.pl>
References: <44FD9717.30802@praterm.com.pl>
	<44FE96DE.7030101@gkec.informatik.tu-darmstadt.de>
	<44FEAC6E.1080604@gkec.informatik.tu-darmstadt.de>
	<44FEBDBF.2020202@praterm.com.pl>
Message-ID: <45003F4E.8030403@gkec.informatik.tu-darmstadt.de>

Hi Pawe?,

Pawe? Pa?ucha wrote:
> Your patch is almost ok, I mean it produces:
> 
>  File "p.py", line 5, in ?
>     d = lxml.etree.parse('/tmp/?.xml')
>   File "etree.pyx", line 1774, in etree.parse
>   File "parser.pxi", line 878, in etree._parseDocument
>   File "parser.pxi", line 798, in etree._parseDocFromFile
>   File "parser.pxi", line 512, in etree._BaseParser._parseDocFromFile
>   File "parser.pxi", line 583, in etree._handleParseResult
>   File "etree.pyx", line 195, in etree._ExceptionContext._raise_if_stored
>   File "parser.pxi", line 275, in etree._parser_resolve_from_python
>   File "apihelpers.pxi", line 492, in etree.funicode
> UnicodeDecodeError: 'utf8' codec can't decode bytes in position 5-8: invalid data
> 
> but if I change line 275 in parse.pxi (etree._parser_resolve_from_python) from:
> 	url = funicode(c_url)
> to:
> 	url = c_url
> 
> parsing works ok. Does it make sense?

It's not a perfect solution, as it may mean that the user ends up with a UTF-8
encoded URL to resolve. That's rarely a problem, as most URLs are plain ASCII
anyway, except for filenames. URLs are passed as UTF-8 encoded strings if they
occurred in a document. The funicode() call is meant to handle that.

Filenames are passed as byte strings encoded for the local filesystem if they
were provided by the user and are UTF-8 encoded if they originate from a
document, so that makes custom resolvers more difficult to handle. At least,
it's always a byte encoded string, it's just not obvious to the user which
encoding was used...

As it works with ASCII URLs and also supports the case of locally encoded
filenames, I think it's an acceptable tradeoff. I'll put it into the trunk.

Stefan



From behnel_ml at gkec.informatik.tu-darmstadt.de  Thu Sep  7 18:19:19 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Thu, 07 Sep 2006 18:19:19 +0200
Subject: [lxml-dev] File name encoding problems
In-Reply-To: <44FEACD8.1090805@praterm.com.pl>
References: <44FD9717.30802@praterm.com.pl>
	<44FE96DE.7030101@gkec.informatik.tu-darmstadt.de>
	<44FEACD8.1090805@praterm.com.pl>
Message-ID: <45004687.80501@gkec.informatik.tu-darmstadt.de>

Hi,

Pawe? Pa?ucha wrote:
> Stefan Behnel wrote:
> 
>> first thing to note: lxml uses UTF-8 internally, also for filenames, as
>> libxml2 requires a char sequence for their representation. If your system
>> can't handle that, we'll have to figure out a way to make it work.
> 
> Looking at the code of libxml2, filenames are passed as 'const char *' and not
> 'const xmlChar *'.

with "char*" == "xmlChar*", but that's not what I meant.


>> Right, I guess that treating the filename with the _utf8() function is not the
>> right thing to do for 8-bit strings. We should have a separate way of treating
>> filenames. I'll look into it.
> 
> I'm not sure, but perhaps the only way is to leave this as user responsibilty,
> and pass filenames without any modifications. There's no way to check what
> encoding is used for file name.

At least, there is sys.getfilesystemencoding(), which tells us what it
*should* be.


> As for libc open() it's just a NULL terminated
> data. But I don't know if Python can handle it...

Python just passes it on.


>> I assume you get the same file-not-found error if you pass the filename as
>> unicode string?
>>
>>    d = lxml.etree.parse(u'/tmp/?.xml')
> 
> Yes, btw - the file-not-found error is just "IOError%" - not very helpful message.

Might have been because of the same problem as the exception you got from
funicode().


>>> This did not happen with python 2.3 - the problem is only with 
>>> python2.4. There's a workaround - reading file to StringIO object and 
>>> then parsing XML from that object. It works fine but it's silly.
>> You can also pass an opened file object.
> 
> No ;-) That's strange but code like that:
> 
> f = open('/tmp/?.xml')
> d = lxml.etree.parse(f)
> 
> behaves the same way (assertion) - it looks like you check the file name
> remembered in file object.

Possibly just another side effect of funicode. :)

I fixed some more places where this was used, so here is a new patch.

Stefan

PS: please remember replying to the list also, so that this discussion gets a)
archived and b) looked at by others.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: filename_encoding.patch
Type: text/x-patch
Size: 15945 bytes
Desc: not available
Url : http://codespeak.net/pipermail/lxml-dev/attachments/20060907/aa9334c8/attachment-0001.bin 

From behnel_ml at gkec.informatik.tu-darmstadt.de  Thu Sep  7 19:47:57 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Thu, 07 Sep 2006 19:47:57 +0200
Subject: [lxml-dev] Building debian packages
In-Reply-To: <44FF348F.6070605@praterm.com.pl>
References: <44FF348F.6070605@praterm.com.pl>
Message-ID: <45005B4D.2060208@gkec.informatik.tu-darmstadt.de>

Czesc Pawe?,

Pawe? Pa?ucha wrote:
> Hi, I've got a small patch against current SVN to file doc/build.txt:
> * info about changing md5sum in *.dsc file
> * command `dpkg -x ...` is incorrect - it should be `dpkg-source -x ...`

Dzienki, I'll add it.

Stefan


From behnel_ml at gkec.informatik.tu-darmstadt.de  Sat Sep  9 08:14:27 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Sat, 09 Sep 2006 08:14:27 +0200
Subject: [lxml-dev] 1.0.4 released
Message-ID: <45025BC3.7080504@gkec.informatik.tu-darmstadt.de>

Hi all,

I just released lxml 1.0.4 to cheeseshop. It's a bug-fix release that fixes a
crash bug in the replace() function. It also adds an extend() method to
Element to make it more list-like.

1.1 final will still have to wait a little longer until the filename issue has
settled. If I find the time, it will be out next week. I know, I keep saying
things like that, but we had a couple of bug reports last week, so I'm happy
that we can solve some more problems before the release.

Have fun,
Stefan

From pawel at praterm.com.pl  Sat Sep  9 10:23:38 2006
From: pawel at praterm.com.pl (=?ISO-8859-2?Q?Pawe=B3_Pa=B3ucha?=)
Date: Sat, 09 Sep 2006 10:23:38 +0200
Subject: [lxml-dev] File name encoding problems
In-Reply-To: <45004687.80501@gkec.informatik.tu-darmstadt.de>
References: <44FD9717.30802@praterm.com.pl>
	<44FE96DE.7030101@gkec.informatik.tu-darmstadt.de>
	<44FEACD8.1090805@praterm.com.pl>
	<45004687.80501@gkec.informatik.tu-darmstadt.de>
Message-ID: <45027A0A.4090909@praterm.com.pl>


> 
> I fixed some more places where this was used, so here is a new patch.
> 
> Stefan

> --- src/lxml/xslt.pxi	(Revision 31765)
> +++ src/lxml/xslt.pxi	(Arbeitskopie)

Should I try to use this patch against Revision 31765 or better a current
(32104) SVN Revision?

Pawe? Pa?ucha

From pawel at praterm.com.pl  Sat Sep  9 11:33:54 2006
From: pawel at praterm.com.pl (=?ISO-8859-2?Q?Pawe=B3_Pa=B3ucha?=)
Date: Sat, 09 Sep 2006 11:33:54 +0200
Subject: [lxml-dev] File name encoding problems
In-Reply-To: <45027A0A.4090909@praterm.com.pl>
References: <44FD9717.30802@praterm.com.pl>	<44FE96DE.7030101@gkec.informatik.tu-darmstadt.de>	<44FEACD8.1090805@praterm.com.pl>	<45004687.80501@gkec.informatik.tu-darmstadt.de>
	<45027A0A.4090909@praterm.com.pl>
Message-ID: <45028A82.9030906@praterm.com.pl>

Pawe? Pa?ucha wrote:

>> I fixed some more places where this was used, so here is a new patch.
>>
>> Stefan
> 
>> --- src/lxml/xslt.pxi	(Revision 31765)
>> +++ src/lxml/xslt.pxi	(Arbeitskopie)
> 
> Should I try to use this patch against Revision 31765 or better a current
> (32104) SVN Revision?

I tried both versions but they both have the same problem with 'url =
funicode(c_url)' resulting in:

d = lxml.etree.parse('/tmp/?.xml')

File "p.py", line 5, in ?
    d = lxml.etree.parse('/tmp/?.xml')
  File "etree.pyx", line 1774, in etree.parse
  File "parser.pxi", line 878, in etree._parseDocument
  File "parser.pxi", line 798, in etree._parseDocFromFile
  File "parser.pxi", line 512, in etree._BaseParser._parseDocFromFile
  File "parser.pxi", line 579, in etree._handleParseResult
  File "etree.pyx", line 195, in etree._ExceptionContext._raise_if_stored
  File "parser.pxi", line 275, in etree._parser_resolve_from_python
  File "apihelpers.pxi", line 492, in etree.funicode
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 5-8: invalid data

When I try

d = lxml.etree.parse('/tmp/?.xml'.encode(utf8))

the result is:

File "p.py", line 5, in ?
    d = lxml.etree.parse(('/tmp/?.xml').encode('utf8'))
  File "etree.pyx", line 1774, in etree.parse
  File "parser.pxi", line 878, in etree._parseDocument
  File "parser.pxi", line 798, in etree._parseDocFromFile
  File "parser.pxi", line 512, in etree._BaseParser._parseDocFromFile
  File "parser.pxi", line 582, in etree._handleParseResult
  File "parser.pxi", line 548, in etree._raiseParseError
IOError: Error reading file '/tmp/??.xml': failed to load external entity
"/tmp/??.xml"

The good news is that:

f = open('/tmp/?.xml')
d = lxml.etree.parse(f)

now works ok (at least at SVN 32104)

Pawe? Pa?ucha

From behnel_ml at gkec.informatik.tu-darmstadt.de  Sat Sep  9 20:22:58 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Sat, 09 Sep 2006 20:22:58 +0200
Subject: [lxml-dev] File name encoding problems
In-Reply-To: <45028A82.9030906@praterm.com.pl>
References: <44FD9717.30802@praterm.com.pl>	<44FE96DE.7030101@gkec.informatik.tu-darmstadt.de>	<44FEACD8.1090805@praterm.com.pl>	<45004687.80501@gkec.informatik.tu-darmstadt.de>
	<45027A0A.4090909@praterm.com.pl> <45028A82.9030906@praterm.com.pl>
Message-ID: <45030682.20301@gkec.informatik.tu-darmstadt.de>

Hi,

Pawe? Pa?ucha wrote:
>>> I fixed some more places where this was used, so here is a new patch.
>>>
> I tried both versions but they both have the same problem with 'url =
> funicode(c_url)' resulting in:
> 
> d = lxml.etree.parse('/tmp/?.xml')
> 
> File "p.py", line 5, in ?
>     d = lxml.etree.parse('/tmp/?.xml')
>   File "etree.pyx", line 1774, in etree.parse
>   File "parser.pxi", line 878, in etree._parseDocument
>   File "parser.pxi", line 798, in etree._parseDocFromFile
>   File "parser.pxi", line 512, in etree._BaseParser._parseDocFromFile
>   File "parser.pxi", line 579, in etree._handleParseResult
>   File "etree.pyx", line 195, in etree._ExceptionContext._raise_if_stored
>   File "parser.pxi", line 275, in etree._parser_resolve_from_python
>   File "apihelpers.pxi", line 492, in etree.funicode
> UnicodeDecodeError: 'utf8' codec can't decode bytes in position 5-8: invalid data

As I said, the problem is that URLs can originate from libxml2 (as UTF-8) and
from the user (as encoded 8-bit strings). In "_parser_resolve_from_python()",
however, we cannot distinguish between the two cases and thus cannot easily
find out about the encoding used. However, we have to if we want to provide a
consistent interface to the user.


> When I try
> 
> d = lxml.etree.parse('/tmp/?.xml'.encode(utf8))
> 
> the result is:
> 
> File "p.py", line 5, in ?
>     d = lxml.etree.parse(('/tmp/?.xml').encode('utf8'))
>   File "etree.pyx", line 1774, in etree.parse
>   File "parser.pxi", line 878, in etree._parseDocument
>   File "parser.pxi", line 798, in etree._parseDocFromFile
>   File "parser.pxi", line 512, in etree._BaseParser._parseDocFromFile
>   File "parser.pxi", line 582, in etree._handleParseResult
>   File "parser.pxi", line 548, in etree._raiseParseError
> IOError: Error reading file '/tmp/??.xml': failed to load external entity
> "/tmp/??.xml"

That's the expected result, as the file you want to refer to does not have
that name.


> The good news is that:
> 
> f = open('/tmp/?.xml')
> d = lxml.etree.parse(f)
> 
> now works ok (at least at SVN 32104)

Yes, I fixed that. And I'll have to see what I can do about the problem above.
I'm considering to distinguish between the case where the URL in question is
the URL of the document that is being parsed and the case where a different
URL (like a DTD) is requested. The second case usually means that the URL
originated from libxml2 (i.e. is in UTF-8) and the first case means that the
user is in charge (i.e. we'd better not recode the string). But I don't know
if that's sufficient...

Stefan


From behnel_ml at gkec.informatik.tu-darmstadt.de  Sat Sep  9 20:51:31 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Sat, 09 Sep 2006 20:51:31 +0200
Subject: [lxml-dev] File name encoding problems
In-Reply-To: <45030682.20301@gkec.informatik.tu-darmstadt.de>
References: <44FD9717.30802@praterm.com.pl>	<44FE96DE.7030101@gkec.informatik.tu-darmstadt.de>	<44FEACD8.1090805@praterm.com.pl>	<45004687.80501@gkec.informatik.tu-darmstadt.de>	<45027A0A.4090909@praterm.com.pl>
	<45028A82.9030906@praterm.com.pl>
	<45030682.20301@gkec.informatik.tu-darmstadt.de>
Message-ID: <45030D33.207@gkec.informatik.tu-darmstadt.de>



Stefan Behnel wrote:
> Pawe? Pa?ucha wrote:
>>>> I fixed some more places where this was used, so here is a new patch.
>>>>
>> I tried both versions but they both have the same problem with 'url =
>> funicode(c_url)' resulting in:
>>
>> d = lxml.etree.parse('/tmp/?.xml')
>>
>> File "p.py", line 5, in ?
>>     d = lxml.etree.parse('/tmp/?.xml')
>>   File "etree.pyx", line 1774, in etree.parse
>>   File "parser.pxi", line 878, in etree._parseDocument
>>   File "parser.pxi", line 798, in etree._parseDocFromFile
>>   File "parser.pxi", line 512, in etree._BaseParser._parseDocFromFile
>>   File "parser.pxi", line 579, in etree._handleParseResult
>>   File "etree.pyx", line 195, in etree._ExceptionContext._raise_if_stored
>>   File "parser.pxi", line 275, in etree._parser_resolve_from_python
>>   File "apihelpers.pxi", line 492, in etree.funicode
>> UnicodeDecodeError: 'utf8' codec can't decode bytes in position 5-8: invalid data
> 
> As I said, the problem is that URLs can originate from libxml2 (as UTF-8) and
> from the user (as encoded 8-bit strings). In "_parser_resolve_from_python()",
> however, we cannot distinguish between the two cases and thus cannot easily
> find out about the encoding used. However, we have to if we want to provide a
> consistent interface to the user.
> 
> I'm considering to distinguish between the case where the URL in question is
> the URL of the document that is being parsed and the case where a different
> URL (like a DTD) is requested. The second case usually means that the URL
> originated from libxml2 (i.e. is in UTF-8) and the first case means that the
> user is in charge (i.e. we'd better not recode the string). But I don't know
> if that's sufficient...

Still not sure, but I committed the following patch for now. Please test if it
works for you.

Stefan

Index: src/lxml/parser.pxi
===================================================================
--- src/lxml/parser.pxi (Revision 32103)
+++ src/lxml/parser.pxi (Arbeitskopie)
@@ -271,12 +271,16 @@
     try:
         if c_url is NULL:
             url = None
+        elif c_context.myDoc is NULL or c_context.myDoc.URL is NULL:
+            # parsing a main document, so URL was passed verbatimly by user
+            url = c_url
         else:
+            # parsing a related document (DTD etc.) => UTF-8 encoded URL
             url = funicode(c_url)
         if c_pubid is NULL:
             pubid = None
         else:
-            pubid = funicode(c_pubid)
+            pubid = funicode(c_pubid) # always UTF-8

         doc_ref = context._resolvers.resolve(url, pubid, context)
         if doc_ref is None:


From pawel at praterm.com.pl  Sat Sep  9 23:35:22 2006
From: pawel at praterm.com.pl (Pawe? Pa?ucha)
Date: Sat, 09 Sep 2006 23:35:22 +0200
Subject: [lxml-dev] File name encoding problems
In-Reply-To: <45030D33.207@gkec.informatik.tu-darmstadt.de>
References: <44FD9717.30802@praterm.com.pl>
	<44FE96DE.7030101@gkec.informatik.tu-darmstadt.de>
	<44FEACD8.1090805@praterm.com.pl>
	<45004687.80501@gkec.informatik.tu-darmstadt.de>
	<45027A0A.4090909@praterm.com.pl> <45028A82.9030906@praterm.com.pl>
	<45030682.20301@gkec.informatik.tu-darmstadt.de>
	<45030D33.207@gkec.informatik.tu-darmstadt.de>
Message-ID: <20060909233522.k0dl19gaes8gsg40@www.szarp.com.pl>

Quoting Stefan Behnel <behnel_ml at gkec.informatik.tu-darmstadt.de>:

> Still not sure, but I committed the following patch for now. Please   
> test if it
> works for you.

>      try:
>          if c_url is NULL:
>              url = None
> +        elif c_context.myDoc is NULL or c_context.myDoc.URL is NULL:
> +            # parsing a main document, so URL was passed verbatimly by user
> +            url = c_url
>          else:
> +            # parsing a related document (DTD etc.) => UTF-8 encoded URL
>              url = funicode(c_url)

I will try it, but what about something like:

try:
   url = funicode(c_url)
except UnicodeDecodeError:
   url = c_url

?
Pawe?

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.


From behnel_ml at gkec.informatik.tu-darmstadt.de  Sun Sep 10 09:32:04 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Sun, 10 Sep 2006 09:32:04 +0200
Subject: [lxml-dev] File name encoding problems
In-Reply-To: <20060909233522.k0dl19gaes8gsg40@www.szarp.com.pl>
References: <44FD9717.30802@praterm.com.pl>	<44FE96DE.7030101@gkec.informatik.tu-darmstadt.de>	<44FEACD8.1090805@praterm.com.pl>	<45004687.80501@gkec.informatik.tu-darmstadt.de>	<45027A0A.4090909@praterm.com.pl>
	<45028A82.9030906@praterm.com.pl>	<45030682.20301@gkec.informatik.tu-darmstadt.de>	<45030D33.207@gkec.informatik.tu-darmstadt.de>
	<20060909233522.k0dl19gaes8gsg40@www.szarp.com.pl>
Message-ID: <4503BF74.40708@gkec.informatik.tu-darmstadt.de>



Pawe? Pa?ucha wrote:
> Quoting Stefan Behnel <behnel_ml at gkec.informatik.tu-darmstadt.de>:
> 
>> Still not sure, but I committed the following patch for now. Please 
>> test if it
>> works for you.
> 
>>      try:
>>          if c_url is NULL:
>>              url = None
>> +        elif c_context.myDoc is NULL or c_context.myDoc.URL is NULL:
>> +            # parsing a main document, so URL was passed verbatimly
>> by user
>> +            url = c_url
>>          else:
>> +            # parsing a related document (DTD etc.) => UTF-8 encoded URL
>>              url = funicode(c_url)
> 
> I will try it, but what about something like:
> 
> try:
>   url = funicode(c_url)
> except UnicodeDecodeError:
>   url = c_url

what if the URL contains characters in a local encoding that the UTF-8 decoder
can accidentally decode?

Stefan


From pawel at praterm.com.pl  Sun Sep 10 12:12:43 2006
From: pawel at praterm.com.pl (=?ISO-8859-2?Q?Pawe=B3_Pa=B3ucha?=)
Date: Sun, 10 Sep 2006 12:12:43 +0200
Subject: [lxml-dev] File name encoding problems
In-Reply-To: <4503BF74.40708@gkec.informatik.tu-darmstadt.de>
References: <44FD9717.30802@praterm.com.pl>	<44FE96DE.7030101@gkec.informatik.tu-darmstadt.de>	<44FEACD8.1090805@praterm.com.pl>	<45004687.80501@gkec.informatik.tu-darmstadt.de>	<45027A0A.4090909@praterm.com.pl>
	<45028A82.9030906@praterm.com.pl>	<45030682.20301@gkec.informatik.tu-darmstadt.de>	<45030D33.207@gkec.informatik.tu-darmstadt.de>
	<20060909233522.k0dl19gaes8gsg40@www.szarp.com.pl>
	<4503BF74.40708@gkec.informatik.tu-darmstadt.de>
Message-ID: <4503E51B.9090407@praterm.com.pl>

Stefan Behnel wrote:

>> I will try it, but what about something like:
>>
>> try:
>>   url = funicode(c_url)
>> except UnicodeDecodeError:
>>   url = c_url
> 
> what if the URL contains characters in a local encoding that the UTF-8 decoder
> can accidentally decode?

Probably you are right ;-) Anyway, SVN version 32115 works just fine, both
calling parse() with file name and with already opened file. Thanks a lot!

Pawe?

From ashish.kulkarni at kalyptorisk.com  Tue Sep 12 09:15:14 2006
From: ashish.kulkarni at kalyptorisk.com (Ashish Kulkarni)
Date: Tue, 12 Sep 2006 12:45:14 +0530
Subject: [lxml-dev] Static win32 builds for lxml 1.0.4
Message-ID: <2AB7346A3227A74BB97F9A0D79E3E65A03ED4A@mailserver.kalyptorisk.com>

Hello,

I finally got the VC++ Toolkit 2003 compiler, and have made static builds for lxml 1.0.4. They are available at:

http://puggy.symonds.net/~ashish/downloads/

Regards,
ashish

From nico at tekNico.net  Tue Sep 12 12:46:08 2006
From: nico at tekNico.net (Nicola Larosa)
Date: Tue, 12 Sep 2006 12:46:08 +0200
Subject: [lxml-dev] Unicode munging in element tag and text
Message-ID: <45068FF0.8020500@tekNico.net>

Hi all, thanks for a great library. :-)


I found a rather peculiar behavior in Unicode object handling for element
tag and text. It looks like they get converted to a plain string if they
only contains ASCII chars, but not always. ElementTree instead always keeps
them as Unicode objects.

>>> from lxml.etree import Element as lxElem
>>> from elementtree.ElementTree import Element as etElem


1) Let's first build an element from a Unicode object with ASCII chars;
only ElementTree keeps it as Unicode:

>>> lx = lxElem(u'ascii')
>>> et = etElem(u'ascii')
>>> lx.tag
'ascii'
>>> et.tag
u'ascii'

while when the Unicode object contains non-ASCII chars, both libraries
correctly keep it as Unicode:

>>> lx = lxElem(u'm?r?th?n?sc??')
>>> et = etElem(u'm?r?th?n?sc??')
>>> lx.tag
u'm\xf2r\xe8th\xe0n\xe0sc\xec\xec'
>>> et.tag
u'm\xf2r\xe8th\xe0n\xe0sc\xec\xec'


2) The same happens for the element text; ASCII:

>>> lx.text = u'ascii'
>>> et.text = u'ascii'
>>> lx.text
'ascii'
>>> et.text
u'ascii'

non-ASCII:

>>> lx.text = u'm?r?th?n?sc??'
>>> et.text = u'm?r?th?n?sc??'
>>> lx.text
u'm\xf2r\xe8th\xe0n\xe0sc\xec\xec'
>>> et.text
u'm\xf2r\xe8th\xe0n\xe0sc\xec\xec'


3) OTOH, when directly setting the element tag, lxml keeps the Unicode
object too:

>>> lx.tag = u'ascii'
>>> et.tag = u'ascii'
>>> lx.tag
u'ascii'
>>> et.tag
u'ascii'


while both libraries keep working correctly when using non-ASCII chars:

>>> lx.tag = u'm?r?th?n?sc??'
>>> et.tag = u'm?r?th?n?sc??'
>>> lx.tag
u'm\xf2r\xe8th\xe0n\xe0sc\xec\xec'
>>> et.tag
u'm\xf2r\xe8th\xe0n\xe0sc\xec\xec'


This inconsistent behavior does not seem intentional. In my opinion, in the
cases 1) and 2) lxml should work as it already does in the case 3), and as
ElementTree always does.

Thanks again.


-- 
Nicola Larosa - http://www.tekNico.net/

There is more money being spent on breast implants and Viagra today than
on Alzheimer's research. This means that by 2040, there should be a large
elderly population with perky boobs and huge erections and absolutely no
recollection of what to do with them. -- David Icke, April 2006



From fredrik at pythonware.com  Tue Sep 12 13:27:23 2006
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Tue, 12 Sep 2006 13:27:23 +0200
Subject: [lxml-dev] Unicode munging in element tag and text
In-Reply-To: <45068FF0.8020500@tekNico.net>
References: <45068FF0.8020500@tekNico.net>
Message-ID: <ee65it$3sc$1@sea.gmane.org>

Nicola Larosa wrote:

> This inconsistent behavior does not seem intentional. In my opinion, in the
> cases 1) and 2) lxml should work as it already does in the case 3), and as
> ElementTree always does.

in Python 2.X, Unicode strings are compatible with 8-bit ASCII-only 
strings, so the lxml.etree behaviour is perfectly acceptable.  I see no 
reason to force an implementation that doesn't use Python objects for 
its internal storage to be forced to keep track of the original type.

(especially not since the Unicode string type will disappear in Python 
3.0; all strings will be able to hold Unicode data).

</F>


From behnel_ml at gkec.informatik.tu-darmstadt.de  Tue Sep 12 18:23:39 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Tue, 12 Sep 2006 18:23:39 +0200
Subject: [lxml-dev] Unicode munging in element tag and text
In-Reply-To: <45068FF0.8020500@tekNico.net>
References: <45068FF0.8020500@tekNico.net>
Message-ID: <4506DF0B.5070604@gkec.informatik.tu-darmstadt.de>

Hi Nicola,

Nicola Larosa wrote:
> This inconsistent behavior does not seem intentional. In my opinion, in the
> cases 1) and 2) lxml should work as it already does in the case 3), and as
> ElementTree always does.

At least under Python 2.x, lxml.etree will continue to return unicode or plain
strings depending on their content. Internally, everything is stored as UTF-8,
so this is for performance reasons as we can avoid unicode conversion for
plain ASCII strings (which are very common, just think of numeric data, dates,
etc.).

This may change in Python 3.x, but then, there may be more to change, so
that's not in our scope for now.

Stefan


From nico at tekNico.net  Tue Sep 12 19:00:34 2006
From: nico at tekNico.net (Nicola Larosa)
Date: Tue, 12 Sep 2006 19:00:34 +0200
Subject: [lxml-dev] Unicode munging in element tag and text
In-Reply-To: <4506DF0B.5070604@gkec.informatik.tu-darmstadt.de>
References: <45068FF0.8020500@tekNico.net>
	<4506DF0B.5070604@gkec.informatik.tu-darmstadt.de>
Message-ID: <4506E7B2.20808@tekNico.net>

Stefan Behnel wrote:
> At least under Python 2.x, lxml.etree will continue to return unicode or plain
> strings depending on their content. Internally, everything is stored as UTF-8,
> so this is for performance reasons as we can avoid unicode conversion for
> plain ASCII strings (which are very common, just think of numeric data, dates,
> etc.).

Any benchmarks supporting this decision?


-- 
Nicola Larosa - http://www.tekNico.net/

There is more money being spent on breast implants and Viagra today than
on Alzheimer's research. This means that by 2040, there should be a large
elderly population with perky boobs and huge erections and absolutely no
recollection of what to do with them. -- David Icke, April 2006



From fredrik at pythonware.com  Tue Sep 12 19:19:52 2006
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Tue, 12 Sep 2006 19:19:52 +0200
Subject: [lxml-dev] Unicode munging in element tag and text
In-Reply-To: <4506E7B2.20808@tekNico.net>
References: <45068FF0.8020500@tekNico.net>	<4506DF0B.5070604@gkec.informatik.tu-darmstadt.de>
	<4506E7B2.20808@tekNico.net>
Message-ID: <ee6q7o$jn3$1@sea.gmane.org>

Nicola Larosa wrote:

> Any benchmarks supporting this decision?

Are you trying to use the "premature optimization is evil" argument 
against people who's spent more time than anyone else on optimizing 
Python's string subsystem? ;-)

</F>


From nico at tekNico.net  Wed Sep 13 00:28:31 2006
From: nico at tekNico.net (Nicola Larosa)
Date: Wed, 13 Sep 2006 00:28:31 +0200
Subject: [lxml-dev] Unicode munging in element tag and text
In-Reply-To: <ee6q7o$jn3$1@sea.gmane.org>
References: <45068FF0.8020500@tekNico.net>	<4506DF0B.5070604@gkec.informatik.tu-darmstadt.de>	<4506E7B2.20808@tekNico.net>
	<ee6q7o$jn3$1@sea.gmane.org>
Message-ID: <4507348F.7050001@tekNico.net>

Fredrik Lundh wrote:
> Are you trying to use the "premature optimization is evil" argument 
> against people who's spent more time than anyone else on optimizing 
> Python's string subsystem? ;-)

I, for one, welcome our new Iceland Sprint overlords. ;-P


-- 
Nicola Larosa - http://www.tekNico.net/

Many software developers have become hostage to the development
frameworks that they utilise. In turn, many frameworks have made
session state a fundamental building block of web development
because it permits sloppy design. -- Alan Dean, April 2006



From behnel_ml at gkec.informatik.tu-darmstadt.de  Tue Sep 12 21:59:53 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Tue, 12 Sep 2006 21:59:53 +0200
Subject: [lxml-dev] Static win32 builds for lxml 1.0.4
In-Reply-To: <2AB7346A3227A74BB97F9A0D79E3E65A03ED4A@mailserver.kalyptorisk.com>
References: <2AB7346A3227A74BB97F9A0D79E3E65A03ED4A@mailserver.kalyptorisk.com>
Message-ID: <450711B9.7080208@gkec.informatik.tu-darmstadt.de>

Hi Ashish,

Ashish Kulkarni wrote:
> I finally got the VC++ Toolkit 2003 compiler, and have made static builds
> for lxml 1.0.4.

Great, I uploaded them. Thanks for contributing!

Stefan



From behnel_ml at gkec.informatik.tu-darmstadt.de  Wed Sep 13 09:19:14 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Wed, 13 Sep 2006 09:19:14 +0200
Subject: [lxml-dev] Unicode munging in element tag and text
In-Reply-To: <4506E7B2.20808@tekNico.net>
References: <45068FF0.8020500@tekNico.net>	<4506DF0B.5070604@gkec.informatik.tu-darmstadt.de>
	<4506E7B2.20808@tekNico.net>
Message-ID: <4507B0F2.5080009@gkec.informatik.tu-darmstadt.de>

Hi Nicola,

Nicola Larosa wrote:
> Stefan Behnel wrote:
>> At least under Python 2.x, lxml.etree will continue to return unicode or plain
>> strings depending on their content. Internally, everything is stored as UTF-8,
>> so this is for performance reasons as we can avoid unicode conversion for
>> plain ASCII strings (which are very common, just think of numeric data, dates,
>> etc.).
> 
> Any benchmarks supporting this decision?

Pretty short question for a long answer.

Your third point was that .tag returned the original type. This is done
through caching the original input, which avoids some 95% of the work required
to rebuild it on each access (last time I ran the benchmark, at least). This
means, it is 95% faster if a program frequently accesses the same tag name.

We could instead recreate a string for the result to make it fit the behaviour
of .text, but why if it's not more than overhead? As Fredrik said, plain
strings and unicode strings are compatible, no need to convert one into the
other for normal string operations. It's actually your fault if you waste
memory and processing time by passing a unicode string where a plain string
would do.

As for the first two points, skipping through a string to see if any non-ASCII
characters are in there is trivial and fast (7-bit vs. 8-bit), creating a
plain string from it means allocating the same amount of memory, copying the
string (which most likely is in the processor cache already by then) using a
platform-optimised memcpy (or whatever, note that we already know the length
of the string by then) and then create a Python object for it. Converting it
to unicode means allocating two or four times the memory, doing a
per-character conversion step by step (from multi-byte UTF-8) and then create
a Python object for it.

I didn't do much benchmarking here, but given the "95%" result above (meaning,
the majority of work is the actual string instantiation), I simply assume that
avoiding to do the character conversion is worth it if ASCII content is frequent.

Stefan

From behnel_ml at gkec.informatik.tu-darmstadt.de  Wed Sep 13 10:05:04 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Wed, 13 Sep 2006 10:05:04 +0200
Subject: [lxml-dev] lxml 1.1 released
Message-ID: <4507BBB0.4090903@gkec.informatik.tu-darmstadt.de>

Hi all,

I'm proud to announce the release of lxml 1.1. It is a major new release that
builds upon the changes in 1.0.4 and introduces many new features compared to
the 1.0 series.

The main improvements are:

* threading support
* XPath axis iteration
* iterparse() and iterwalk()
* configurable Element class lookup methods
* lxml.objectify - a data binding API on top of lxml.etree

The complete changelog is below.

lxml 1.1 has been tested in an alpha and beta version and received various bug
fixes before the final release. It is currently considered stable and ready
for production use, whereas the 1.0 series are considered rock-stable and may
still be the right thing to choose for more conservative environments.

Bug-fix releases and minor improvements to the 1.1 and 1.0 series will
continue to become available at need. That way, 1.1 will soon be considered as
stable as 1.0 to become its the main line successor.

Given the many helpful egg contributions we received for past releases, I hope
that 1.1 (and 1.0.4) will be as well supported in that regard to help in
further increasing our community.

Thanks to everyone who helped in getting 1.1 done, in getting bugs fixed and
helping others in using it.

Have fun,
Stefan


Changes since 1.0.4:


1.1 (2006-09-13)
================

Features added
--------------

* Comments and processing instructions return '<!-- coment -->' and
  '<?pi-target content?>' for repr()

* Parsers are now the preferred (and default) place where element class lookup
  schemes should be registered.  Namespace lookup is no longer supported by
  default.

Bugs fixed
----------

* filenames with local 8-bit encoding were not supported

* 1.1beta did not compile under Python 2.3

* ignore unknown 'pyval' attribute values in objectify

* objectify.ObjectifiedElement.addattr() failed to accept Elements and Lists

* objectify.ObjectPath.setattr() failed to accept Elements and Lists


1.1beta (2006-08-08)
====================

Features added
--------------

* Support for Python 2.5 beta

* Unlock the GIL for deep copying documents and for XPath()

* New ``compact`` keyword argument for parsing read-only documents

* Support for parser options in iterparse()

* The ``namespace`` axis is supported in XPath and returns (prefix, URI)
  tuples

* The XPath expression "/" now returns an empty list instead of raising an
  exception

* XML-Object API on top of lxml (lxml.objectify)

* Customizable Element class lookup:

  * different pre-implemented lookup mechanisms

  * support for externally provided lookup functions

* Support for processing instructions (ET-like, not compatible)

* Public C-level API for independent extension modules

Bugs fixed
----------

* XPathSyntaxError now inherits from XPathError

* Threading race conditions in RelaxNG and XMLSchema

* Crash when mixing elements from XSLT results into other trees, concurrent
  XSLT is only allowed when the stylesheet was parsed in the main thread

* The EXSLT ``regexp:match`` function now works as defined (except for some
  differences in the regular expression syntax)

* Setting element.text to '' returned None on request, not the empty string

* ``iterparse()`` could crash on long XML files

* Creating documents no longer copies the parser for later URL resolving.  For
  performance reasons, only a reference is kept.  Resolver updates on the
  parser will now be reflected by documents that were parsed before the
  change.  Although this should rarely become visible, it is a behavioral
  change from 1.0.


1.1alpha (2006-06-27)
=====================

Features added
--------------

* Module level ``iterwalk()`` function as 'iterparse' for trees

* Module level ``iterparse()`` function similar to ElementTree (see
  documentation for differences)

* Element.nsmap property returns a mapping of all namespace prefixes known at
  the Element to their namespace URI

* Reentrant threading support in RelaxNG, XMLSchema and XSLT

* Threading support in parsers and serializers:

  * All in-memory operations (tostring, parse(StringIO), etc.) free the GIL

  * File operations (on file names) free the GIL

  * Reading from file-like objects frees the GIL and reacquires it for reading

  * Serialisation to file-like objects is single-threaded (high lock overhead)

* Element iteration over XPath axes:

  * Element.iterdescendants() iterates over the descendants of an element

  * Element.iterancestors() iterates over the ancestors of an element (from
    parent to parent)

  * Element.itersiblings() iterates over either the following or preceding
    siblings of an element

  * Element.iterchildren() iterates over the children of an element in either
    direction

  * All iterators support the ``tag`` keyword argument to restrict the
    generated elements

* Element.getnext() and Element.getprevious() return the direct siblings of an
  element

From ashish.kulkarni at kalyptorisk.com  Thu Sep 14 07:07:24 2006
From: ashish.kulkarni at kalyptorisk.com (Ashish Kulkarni)
Date: Thu, 14 Sep 2006 10:37:24 +0530
Subject: [lxml-dev] Static win32 builds for lxml 1.1
In-Reply-To: <450711B9.7080208@gkec.informatik.tu-darmstadt.de>
Message-ID: <2AB7346A3227A74BB97F9A0D79E3E65A03EDC2@mailserver.kalyptorisk.com>

Hello,

I've made static win32 builds for lxml 1.1. However, I faced a few problems so I'm just mentioning them here.

The builds are available at http://puggy.symonds.net/~ashish/downloads/

side-note: the static build is now 2.1MB, compared to the 1.2MB for 1.0.4.

1] the lxml.etree extension failed to compile

etree.c
src\lxml\etree.c(49440) : error C2137: empty character constant

The code at the location is in function __Pyx_ImportModuleCApi, and looks like this:

if (*t->s == '<NUL>')

Apparently, MSVC chokes on the raw <NUL> character (gcc doesn't have a problem with it). Ideally such a character should not be present -- it should use something like '\x0', which is what I did. If this is correct, then it should be fixed either in the Pyrex code, the patched Pyrex or swig (wherever the problem originated).

2] the lxml.objectify extension failed to link

The lxml.objectify extension is using the xmlNanoHTTP/xmlNanoFTP functions from libxml2, which in turn require linking to wsock32. The solution is to add that to the list of libraries.

# This is called if the '--static' option is passed
def setupStaticBuild():
    cflags = [
        "-I..\\libxml2-2.6.23.win32\\include",
        "-I..\\libxslt-1.1.15.win32\\include",
        "-I..\\zlib-1.2.3.win32\\include",
        "-I..\\iconv-1.9.1.win32\\include"
    ]
    xslt_libs = [
        "..\\libxml2-2.6.23.win32\\lib\\libxml2_a.lib",
        "..\\libxslt-1.1.15.win32\\lib\\libxslt_a.lib",
        "..\\libxslt-1.1.15.win32\\lib\\libexslt_a.lib",
        "..\\zlib-1.2.3.win32\\lib\\zlib.lib",
        "..\\iconv-1.9.1.win32\\lib\\iconv_a.lib",
        "wsock32.lib"	# required for xmlNano* methods
    ]
    result = (cflags, xslt_libs)
    return result

The documentation should ideally be updated to reflect this.

Regards,
ashish


From behnel_ml at gkec.informatik.tu-darmstadt.de  Thu Sep 14 19:58:14 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Thu, 14 Sep 2006 19:58:14 +0200
Subject: [lxml-dev] Static win32 builds for lxml 1.1
In-Reply-To: <2AB7346A3227A74BB97F9A0D79E3E65A03EDC2@mailserver.kalyptorisk.com>
References: <2AB7346A3227A74BB97F9A0D79E3E65A03EDC2@mailserver.kalyptorisk.com>
Message-ID: <45099836.1000804@gkec.informatik.tu-darmstadt.de>

Hi Ashish,

Ashish Kulkarni wrote:
> I've made static win32 builds for lxml 1.1. However, I faced a few problems so I'm just mentioning them here.

Thanks for binaries and feedback! Both are very helpful.


> side-note: the static build is now 2.1MB, compared to the 1.2MB for 1.0.4.

That's the disadvantage of a static build: each binary is statically linked
against all libraries, so now it's twice the size as we have lxml.etree and
lxml.objectify as modules.


> 1] the lxml.etree extension failed to compile
> 
> etree.c
> src\lxml\etree.c(49440) : error C2137: empty character constant
> 
> The code at the location is in function __Pyx_ImportModuleCApi, and looks like this:
> 
> if (*t->s == '<NUL>')

Right. I found that in my Pyrex patch. It actually says "\0", but that's
parsed and converted by Python already, so it ends up in the C source as
0-byte. Changing it to "\\0" will fix it (see patch below).


> 2] the lxml.objectify extension failed to link
> 
> The lxml.objectify extension is using the xmlNanoHTTP/xmlNanoFTP functions from libxml2,

Not on my machine. I can't see them use any FTP functions whatsoever. Does
etree depend on those on your machine, too?


> which in turn require linking to wsock32. The solution is to add that to the list of libraries.
> 
> # This is called if the '--static' option is passed
> def setupStaticBuild():
>     cflags = [
>         "-I..\\libxml2-2.6.23.win32\\include",
>         "-I..\\libxslt-1.1.15.win32\\include",
>         "-I..\\zlib-1.2.3.win32\\include",
>         "-I..\\iconv-1.9.1.win32\\include"
>     ]
>     xslt_libs = [
>         "..\\libxml2-2.6.23.win32\\lib\\libxml2_a.lib",
>         "..\\libxslt-1.1.15.win32\\lib\\libxslt_a.lib",
>         "..\\libxslt-1.1.15.win32\\lib\\libexslt_a.lib",
>         "..\\zlib-1.2.3.win32\\lib\\zlib.lib",
>         "..\\iconv-1.9.1.win32\\lib\\iconv_a.lib",
>         "wsock32.lib"	# required for xmlNano* methods
>     ]
>     result = (cflags, xslt_libs)
>     return result
> 
> The documentation should ideally be updated to reflect this.

Not sure if that's really required. If etree doesn't use those functions, I
can't see why objectify should.

Stefan


Index: Pyrex/Compiler/Nodes.py
===================================================================
--- Pyrex/Compiler/Nodes.py     (Revision 31073)
+++ Pyrex/Compiler/Nodes.py     (Arbeitskopie)
@@ -4149,7 +4149,7 @@
 static int __Pyx_ImportModuleCApi(__Pyx_CApiTabEntry *t) {
     __Pyx_CApiTabEntry *api_t;
     while (t->s) {
-        if (*t->s == '\0')
+        if (*t->s == '\\0')
             continue; /* shortcut for erased string entries */
         api_t = %(API_TAB)s;
         while ((api_t->s) && (strcmp(api_t->s, t->s) < 0))


From behnel_ml at gkec.informatik.tu-darmstadt.de  Thu Sep 14 19:29:07 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Thu, 14 Sep 2006 19:29:07 +0200
Subject: [lxml-dev] [objectify] inconsistent element types for different
 element access orders
In-Reply-To: <OF92EA38A6.6CCFB5C3-ONC12571E3.002AEF11-C12571E3.002E21EC@LBBW.de>
References: <OF92EA38A6.6CCFB5C3-ONC12571E3.002AEF11-C12571E3.002E21EC@LBBW.de>
Message-ID: <45099163.4050005@gkec.informatik.tu-darmstadt.de>

Hi Holger,

I didn't wait for this to settle for 1.1.0, but it can become available in
1.1.1 if we see it fit.


Holger Joukl wrote:
> Stefan Behnel wrote:
>> One thing that comes to my mind is that we could add support for
>> replacing the default type classes used by ObjectifyElementClassLookup.
>> We could add keyword arguments so that you could say
>>
>>       lookup =
> ObjectifyElementClassLookup(StringElement=MyStringElementClass)
>> That would currently work for String-, None- and ObjectifiedElement only,
>> as the others use the data type registry. Maybe we should rather support
>> something like "default_data_class" and "default_tree_class" (and keep
>> the NoneElement, which is only used in a well defined case anyway).

I chose "tree_class" and "empty_data_class" now. I think that's sufficiently
telling.


> I'm perfectly happy with the current solution except for setattr-ing a
> 'structural element' and wanting this to remain instead of becoming a
> StringElement. I don't quite see how a different default data class or
> different tree class achieve this?

Well, the idea is that you can change the default for empty data classes
(remember that it's a pretty arbitrary decision to default to StringElement
here) and also use subclasses of ObjectifiedElement for the tree structure.

However, if you want StringElement in some cases and ObjectifiedElement in
other cases, that's difficult to achieve at the Python level, as it would
require passing information about the C node to allow taking the decision.


> So I'm back to suggesting a TreeElement() factory (not the best name,
> maybe)
> returning an ObjectifiedElement with a new pytype='ObjectifiedElement'
> which keeps it from becoming a string. I think that's still nicer than
> "stringifying" every single empty leaf when parsing from XML.

What about adding the attribute in objectify.Element()? You can't normally
change the data value of an Element itself, so the only real reason why you
would call objectify.Element() is to create a structural element (usually a
root node).

I called the corresponding pytype value "TREE" for now, I think it's unlikely
that someone would use that as custom type name.

Stefan


From lxml at holloway.co.nz  Fri Sep 15 00:13:29 2006
From: lxml at holloway.co.nz (Matthew Cruickshank)
Date: Fri, 15 Sep 2006 10:13:29 +1200
Subject: [lxml-dev] Handling of <xsl:message terminate="yes"> ...
Message-ID: <4509D409.3060809@holloway.co.nz>

Hi, I've been writing this XML chain/pipeline - sort of a multi-pipeline 
version of Apache Cocoon and lxml has been great -- very fast!  I'm 
having some problems with <xsl:message> and I don't know how to read 
errors caused by that. They terminate the processing but I can't find 
the error message text (from within the <xsl:message> tag) anywhere... 
it doesn't seem to be in error_log and no exception is thrown.

Any ideas -- am I looking in the wrong place?

I'm using <xsl:message> to assert things about document structure and 
runtime parameters (things that couldn't be asserted in RelaxNG, I think).


.Matthew Cruickshank
http://docvert.org << MSWord to HTML or any XML


From ashish.kulkarni at kalyptorisk.com  Fri Sep 15 07:21:53 2006
From: ashish.kulkarni at kalyptorisk.com (Ashish Kulkarni)
Date: Fri, 15 Sep 2006 10:51:53 +0530
Subject: [lxml-dev] Static win32 builds for lxml 1.1
In-Reply-To: <45099836.1000804@gkec.informatik.tu-darmstadt.de>
Message-ID: <2AB7346A3227A74BB97F9A0D79E3E65A03EE05@mailserver.kalyptorisk.com>

 
Hi,

>> 2] the lxml.objectify extension failed to link
>> 
>> The lxml.objectify extension is using the xmlNanoHTTP/xmlNanoFTP functions from libxml2,
>
>Not on my machine. I can't see them use any FTP functions whatsoever. Does
>etree depend on those on your machine, too?
>
>

Well, etree doesn't depend on it, but objectify does. I looked at the code, and probably it's because objectify includes some namespace URLs while etree doesn't. This may automatically be triggering the inclusion of the xmlNano* methods during a static build -- that's a guess, I don't know enough of lxml or libxml2 to be more certain :-)

I've attached the build log of the compilation.

Regards,
Ashish

==================== BUILD LOG ==================== 
Building lxml version 1.1
*NOTE*: Trying to build without Pyrex, needs pre-generated 'src/lxml/etree.c' !
running build
running build_py
creating build
creating build\lib.win32-2.4
creating build\lib.win32-2.4\lxml
copying src\lxml\elementlib.py -> build\lib.win32-2.4\lxml
copying src\lxml\sax.py -> build\lib.win32-2.4\lxml
copying src\lxml\_elementpath.py -> build\lib.win32-2.4\lxml
copying src\lxml\__init__.py -> build\lib.win32-2.4\lxml
running build_ext
building 'lxml.etree' extension
creating build\temp.win32-2.4
creating build\temp.win32-2.4\Release
creating build\temp.win32-2.4\Release\src
creating build\temp.win32-2.4\Release\src\lxml
C:\Program Files\Microsoft Visual C++ Toolkit 2003\bin\cl.exe /c /nologo /Ox /MD /W3 /GX /DNDEBUG -IC:\DevTools\python24\include -IC:\DevTools\python24\PC /Tcsrc/lxml/etree.c /Fobuild\temp.win32-2.4\Release\src/lxml/etree.obj -w -I..\libxml2-2.6.23.win32\include -I..\libxslt-1.1.15.win32\include -I..\zlib-1.2.3.win32\include -I..\iconv-1.9.1.win32\include
cl : Command line warning D4025 : overriding '/W3' with '/w'
etree.c
C:\Program Files\Microsoft Visual C++ Toolkit 2003\bin\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\DevTools\python24\libs /LIBPATH:C:\DevTools\python24\PCBuild /EXPORT:initetree build\temp.win32-2.4\Release\src/lxml/etree.obj /OUT:build\lib.win32-2.4\lxml\etree.pyd /IMPLIB:build\temp.win32-2.4\Release\src/lxml\etree.lib ..\libxml2-2.6.23.win32\lib\libxml2_a.lib ..\libxslt-1.1.15.win32\lib\libxslt_a.lib ..\libxslt-1.1.15.win32\lib\libexslt_a.lib ..\zlib-1.2.3.win32\lib\zlib.lib ..\iconv-1.9.1.win32\lib\iconv_a.lib
   Creating library build\temp.win32-2.4\Release\src/lxml\etree.lib and object build\temp.win32-2.4\Release\src/lxml\etree.exp
etree.obj : warning LNK4217: locally defined symbol _xmlFree imported in function ___pyx_f_5etree__bugFixURL
etree.obj : warning LNK4217: locally defined symbol _xsltDocDefaultLoader imported in function _initetree
etree.obj : warning LNK4217: locally defined symbol _xsltLibxsltVersion imported in function _initetree
building 'lxml.objectify' extension
C:\Program Files\Microsoft Visual C++ Toolkit 2003\bin\cl.exe /c /nologo /Ox /MD /W3 /GX /DNDEBUG -IC:\DevTools\python24\include -IC:\DevTools\python24\PC /Tcsrc/lxml/objectify.c /Fobuild\temp.win32-2.4\Release\src/lxml/objectify.obj -w -I..\libxml2-2.6.23.win32\include -I..\libxslt-1.1.15.win32\include -I..\zlib-1.2.3.win32\include -I..\iconv-1.9.1.win32\include
cl : Command line warning D4025 : overriding '/W3' with '/w'
objectify.c
C:\Program Files\Microsoft Visual C++ Toolkit 2003\bin\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\DevTools\python24\libs /LIBPATH:C:\DevTools\python24\PCBuild /EXPORT:initobjectify build\temp.win32-2.4\Release\src/lxml/objectify.obj/OUT:build\lib.win32-2.4\lxml\objectify.pyd /IMPLIB:build\temp.win32-2.4\Release\src/lxml\objectify.lib ..\libxml2-2.6.23.win32\lib\libxml2_a.lib ..\libxslt-1.1.15.win32\lib\libxslt_a.lib ..\libxslt-1.1.15.win32\lib\libexslt_a.lib ..\zlib-1.2.3.win32\lib\zlib.lib ..\iconv-1.9.1.win32\lib\iconv_a.lib
   Creating library build\temp.win32-2.4\Release\src/lxml\objectify.lib and object build\temp.win32-2.4\Release\src/lxml\objectify.exp
libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__WSAGetLastError at 0 referenced in function _socket_errno
libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__WSACleanup at 0 referenced in function _xmlNanoHTTPCleanup
libxml2_a.lib(nanoftp.obj) : error LNK2001: unresolved external symbol __imp__WSACleanup at 0
libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__closesocket at 4 referenced in function _xmlNanoHTTPFreeCtxt
libxml2_a.lib(nanoftp.obj) : error LNK2019: unresolved external symbol __imp__closesocket at 4 referenced in function _xmlNanoFTPNewCtxt
libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__select at 20 referenced in function _xmlNanoHTTPSend
libxml2_a.lib(nanoftp.obj) : error LNK2001: unresolved external symbol __imp__select at 20
libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__send at 16 referenced in function _xmlNanoHTTPSend
libxml2_a.lib(nanoftp.obj) : error LNK2001: unresolved external symbol __imp__send at 16
libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__recv at 16 referenced in function _xmlNanoHTTPRecv
libxml2_a.lib(nanoftp.obj) : error LNK2001: unresolved external symbol __imp__recv at 16
libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__getsockopt at 20 referenced in function _xmlNanoHTTPConnectAttempt
libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol ___WSAFDIsSet at 8 referenced in function _xmlNanoHTTPConnectAttempt
libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__connect at 12 referenced in function _xmlNanoHTTPConnectAttempt
libxml2_a.lib(nanoftp.obj) : error LNK2001: unresolved external symbol __imp__connect at 12
libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__ioctlsocket at 12 referenced in function _xmlNanoHTTPConnectAttempt
libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__socket at 12 referenced in function _xmlNanoHTTPConnectAttempt
libxml2_a.lib(nanoftp.obj) : error LNK2001: unresolved external symbol __imp__socket at 12
libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__htons at 4 referenced in function _xmlNanoHTTPConnectHost
libxml2_a.lib(nanoftp.obj) : error LNK2001: unresolved external symbol __imp__htons at 4
libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__gethostbyname at 4 referenced in function _xmlNanoHTTPConnectHost
libxml2_a.lib(nanoftp.obj) : error LNK2001: unresolved external symbol __imp__gethostbyname at 4
libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__WSAStartup at 8 referenced in function _xmlNanoHTTPInit
libxml2_a.lib(nanoftp.obj) : error LNK2001: unresolved external symbol __imp__WSAStartup at 8
libxml2_a.lib(nanoftp.obj) : error LNK2019: unresolved external symbol __imp__listen at 8 referenced in function _xmlNanoFTPGetConnection
libxml2_a.lib(nanoftp.obj) : error LNK2019: unresolved external symbol __imp__bind at 12 referenced in function _xmlNanoFTPGetConnection
libxml2_a.lib(nanoftp.obj) : error LNK2019: unresolved external symbol __imp__getsockname at 12 referenced in function _xmlNanoFTPGetConnection
build\lib.win32-2.4\lxml\objectify.pyd : fatal error LNK1120: 17 unresolved externals
error: command '"C:\Program Files\Microsoft Visual C++ Toolkit 2003\bin\link.exe"' failed with exit status 1120

==================== END ==================== 


From Holger.Joukl at LBBW.de  Fri Sep 15 10:06:33 2006
From: Holger.Joukl at LBBW.de (Holger Joukl)
Date: Fri, 15 Sep 2006 10:06:33 +0200
Subject: [lxml-dev] [objectify] inconsistent element types for different
	element access orders
In-Reply-To: <45099163.4050005@gkec.informatik.tu-darmstadt.de>
Message-ID: <OFDF28C80C.1904ED06-ONC12571EA.002710A8-C12571EA.002C8C04@LBBW.de>

Hello Stefan,
first of all congrats for bringing out the 1.1 release! We are currently
working hard on basing our toolkit on lxml.objectify and so far it works
like a charm.
Especially the ObjectPath functionality and the ease of hooking custom
element
classes into the lookup mechanism is great.

I really think anybody considering to use amara.bindery or gnosis.objectify
should have a good look at lxml.objectify.

Stefan Behnel <behnel_ml at gkec.informatik.tu-darmstadt.de> schrieb am
14.09.2006 19:29:07:

> Hi Holger,
>
> I didn't wait for this to settle for 1.1.0, but it can become available
in
> 1.1.1 if we see it fit.

Not in a hurry :-)

> Holger Joukl wrote:
> > Stefan Behnel wrote:
> >> One thing that comes to my mind is that we could add support for
> >> replacing the default type classes used by
ObjectifyElementClassLookup.
> >> We could add keyword arguments so that you could say
> >>
> >>       lookup =
> > ObjectifyElementClassLookup(StringElement=MyStringElementClass)
> >> That would currently work for String-, None- and ObjectifiedElement
only,
> >> as the others use the data type registry. Maybe we should rather
support
> >> something like "default_data_class" and "default_tree_class" (and keep
> >> the NoneElement, which is only used in a well defined case anyway).
>
> I chose "tree_class" and "empty_data_class" now. I think that's
sufficiently
> telling.

I'm still not quite sure what you mean by that. In my words: That will
allow for
customizing the behaviour when encountering
- an empty leaf node (empty_data_class gets chosen)
- an empty tree node = a node that contains no text but has children
(tree_class gets chosen)
This is to not force an objectify user to follow our arbitrary (though with
good reason ;-) decision to use StringElement for empty leaves.
Right?

> However, if you want StringElement in some cases and ObjectifiedElement
in
> other cases, that's difficult to achieve at the Python level, as it would
> require passing information about the C node to allow taking the
decision.
>
>
> > So I'm back to suggesting a TreeElement() factory (not the best name,
> > maybe)
> > returning an ObjectifiedElement with a new pytype='ObjectifiedElement'
> > which keeps it from becoming a string. I think that's still nicer than
> > "stringifying" every single empty leaf when parsing from XML.
>
> What about adding the attribute in objectify.Element()? You can't
normally
> change the data value of an Element itself, so the only real reason why
you
> would call objectify.Element() is to create a structural element (usually
a
> root node).
>
> I called the corresponding pytype value "TREE" for now, I think it's
unlikely
> that someone would use that as custom type name.
>
> Stefan

Great, I'll try it out. But I'm still voting for a TreeElement() factory
as I'll have to write s.th. like this anyway:

def TreeElement():
    return objectify.Element('tree', {objectify.PYTYPE_ATTRIBUTE: 'TREE'})

And such a factory complemented the objectify module interface nicely,
given there's also the DataElement function, imho.

Btw. I've found that I rather often use
ElementBase.__len__(<someObjectifiedObj>)
to get the childcount/find out if an element has children.
What do you think about adding a hasChildren() or countChildren() function
to objectify?

Regards, Holger

Der Inhalt dieser E-Mail ist vertraulich. Falls Sie nicht der angegebene
Empf?nger sind oder falls diese E-Mail irrt?mlich an Sie adressiert wurde,
verst?ndigen Sie bitte den Absender sofort und l?schen Sie die E-Mail
sodann. Das unerlaubte Kopieren sowie die unbefugte ?bermittlung sind nicht
gestattet. Die Sicherheit von ?bermittlungen per E-Mail kann nicht
garantiert werden. Falls Sie eine Best?tigung w?nschen, fordern Sie bitte
den Inhalt der E-Mail als Hardcopy an.

The contents of this  e-mail are confidential. If you are not the named
addressee or if this transmission has been addressed to you in error,
please notify the sender immediately and then delete this e-mail.  Any
unauthorized copying and transmission is forbidden. E-Mail transmission
cannot be guaranteed to be secure. If verification is required, please
request a hard copy version.



From Holger.Joukl at LBBW.de  Fri Sep 15 11:45:03 2006
From: Holger.Joukl at LBBW.de (Holger Joukl)
Date: Fri, 15 Sep 2006 11:45:03 +0200
Subject: [lxml-dev] [objectify] inconsistent element types for different
	element access orders
In-Reply-To: <20060915.CcY.88727500@groupware>
Message-ID: <OF023A5ADE.803A9135-ONC12571EA.0033D5E8-C12571EA.003590AF@LBBW.de>

Hi Stefan,

"Stefan Behnel" <behnel at dvs1.informatik.tu-darmstadt.de> schrieb am
15.09.2006 11:11:24:

> > I really think anybody considering to use amara.bindery or
gnosis.objectify
> > should have a good look at lxml.objectify.
>
> Hear, hear! :)
>
> Can we quote you on our web page?
>
> Like: "Google uses Python, but for critical stuff, my bank depends on
lxml,
> because ..."

Yes you can, and I'd also be willing to write a short success story or
s.th. like that.
But let's just delay any quoting or success stories for now, until we have
actually
realized the success ;-)

> > > I chose "tree_class" and "empty_data_class" now. I think that's
> > > sufficiently telling.
> >
> > I'm still not quite sure what you mean by that. In my words: That will
> > allow for
> > customizing the behaviour when encountering
> > - an empty leaf node (empty_data_class gets chosen)
> > - an empty tree node = a node that contains no text but has children
> > (tree_class gets chosen)
>
> Right. There are a few more rules:
>   has no parent -> tree
>   has xsi:nil attribute -> NullElement
>   has parsable type -> type class
>
>
> > This is to not force an objectify user to follow our arbitrary (though
with
> > good reason ;-) decision to use StringElement for empty leaves.
> > Right?
>
> For that, yes, and for easily replacing the inner tree classes. I think
that
> can be a pretty helpful thing if you want to extend the API. You can now
> replace the type classes through PyType, and the inner tree class and the
> default leaf class through the lookup mechanism. I think that's all you
might
> need to extend objectify.

Understood, thanks.

> > > What about adding the attribute in objectify.Element()? You can't
> > > normally
> > > change the data value of an Element itself, so the only real reason
why
> > > you
> > > would call objectify.Element() is to create a structural element
(usually
> > > a root node).
> > >
> > > I called the corresponding pytype value "TREE" for now, I think it's
> > > unlikely that someone would use that as custom type name.
> >
> > Great, I'll try it out. But I'm still voting for a TreeElement()
factory
> > as I'll have to write s.th. like this anyway:
> >
> > def TreeElement():
> >     return objectify.Element('tree', {objectify.PYTYPE_ATTRIBUTE:
'TREE'})
>
> No, that's redundant. Just use
>
>     objectify.Element('tree')

Fine.

Just wanted to try it out and got this error:

>>> objectify.Element('foo', {'a': '1'})
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "objectify.pyx", line 1481, in objectify.Element
NameError: python

I don't really understand that as python.<whatever> is used all over the
place.
Has this s.th. to with python being cimported and so far having only
been used in the cdef'ed stuff but now it shows up in a def'ed module
function?
I must admit I haven't actually looked very much at pyrex itself. Will now
try to add an import python somewhere...

> I'll add a countchildren() method, then.
>
> Stefan

Thanks,
Holger

Der Inhalt dieser E-Mail ist vertraulich. Falls Sie nicht der angegebene
Empf?nger sind oder falls diese E-Mail irrt?mlich an Sie adressiert wurde,
verst?ndigen Sie bitte den Absender sofort und l?schen Sie die E-Mail
sodann. Das unerlaubte Kopieren sowie die unbefugte ?bermittlung sind nicht
gestattet. Die Sicherheit von ?bermittlungen per E-Mail kann nicht
garantiert werden. Falls Sie eine Best?tigung w?nschen, fordern Sie bitte
den Inhalt der E-Mail als Hardcopy an.

The contents of this  e-mail are confidential. If you are not the named
addressee or if this transmission has been addressed to you in error,
please notify the sender immediately and then delete this e-mail.  Any
unauthorized copying and transmission is forbidden. E-Mail transmission
cannot be guaranteed to be secure. If verification is required, please
request a hard copy version.



From behnel_ml at gkec.informatik.tu-darmstadt.de  Fri Sep 15 17:42:56 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Fri, 15 Sep 2006 17:42:56 +0200
Subject: [lxml-dev] [objectify] inconsistent element types for different
 element access orders
In-Reply-To: <OF023A5ADE.803A9135-ONC12571EA.0033D5E8-C12571EA.003590AF@LBBW.de>
References: <OF023A5ADE.803A9135-ONC12571EA.0033D5E8-C12571EA.003590AF@LBBW.de>
Message-ID: <450ACA00.2050401@gkec.informatik.tu-darmstadt.de>

Hi Holger,

Holger Joukl wrote:
> Stefan Behnel wrote:
>> Just use
>>
>>     objectify.Element('tree')
> 
> Fine.
> 
> Just wanted to try it out and got this error:
> 
>>>> objectify.Element('foo', {'a': '1'})
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
>   File "objectify.pyx", line 1481, in objectify.Element
> NameError: python
> 
> I don't really understand that as python.<whatever> is used all over the
> place.

Pyrex gives you that when you use a C function that's not declared. In this
case, it does not even exist. The real name is PyDict_Size().

Should be fixed now. DataElement had the same bug, BTW, I had copied the code
from there.

Stefan

From behnel_ml at gkec.informatik.tu-darmstadt.de  Fri Sep 15 17:57:31 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Fri, 15 Sep 2006 17:57:31 +0200
Subject: [lxml-dev] [objectify] inconsistent element types for different
 element access orders
In-Reply-To: <OFDF28C80C.1904ED06-ONC12571EA.002710A8-C12571EA.002C8C04@LBBW.de>
References: <OFDF28C80C.1904ED06-ONC12571EA.002710A8-C12571EA.002C8C04@LBBW.de>
Message-ID: <450ACD6B.90105@gkec.informatik.tu-darmstadt.de>

## forgot to send this to the list also ##


Hi Holger,

Holger Joukl (Holger.Joukl at LBBW.de) wrote:
> > first of all congrats for bringing out the 1.1 release! We are currently
> > working hard on basing our toolkit on lxml.objectify and so far it works
> > like a charm.

Cool, that's good to know.


> > Especially the ObjectPath functionality and the ease of hooking custom
> > element classes into the lookup mechanism is great.

 :)  Custom element classes were one of the first thing I implemented when I
came to lxml. And I really like the way they now fit into the new lookup
framework.


> > I really think anybody considering to use amara.bindery or gnosis.objectify
> > should have a good look at lxml.objectify.

Hear, hear!  :)

Can we quote you on our web page?

Like: "Google uses Python, but for critical stuff, my bank depends on lxml,
because ..."


>> > > I chose "tree_class" and "empty_data_class" now. I think that's
>> > > sufficiently telling.
> >
> > I'm still not quite sure what you mean by that. In my words: That will
> > allow for
> > customizing the behaviour when encountering
> > - an empty leaf node (empty_data_class gets chosen)
> > - an empty tree node = a node that contains no text but has children
> > (tree_class gets chosen)

Right. There are a few more rules:
  has no parent -> tree
  has xsi:nil attribute -> NullElement
  has parsable type -> type class


> > This is to not force an objectify user to follow our arbitrary (though with
> > good reason  ;-)  decision to use StringElement for empty leaves.
> > Right?

For that, yes, and for easily replacing the inner tree classes. I think that
can be a pretty helpful thing if you want to extend the API. You can now
replace the type classes through PyType, and the inner tree class and the
default leaf class through the lookup mechanism. I think that's all you might
need to extend objectify.


>> > > What about adding the attribute in objectify.Element()? You can't
>> > > normally
>> > > change the data value of an Element itself, so the only real reason why
>> > > you
>> > > would call objectify.Element() is to create a structural element (usually
>> > > a root node).
>> > >
>> > > I called the corresponding pytype value "TREE" for now, I think it's
>> > > unlikely that someone would use that as custom type name.
> >
> > Great, I'll try it out. But I'm still voting for a TreeElement() factory
> > as I'll have to write s.th. like this anyway:
> >
> > def TreeElement():
> >     return objectify.Element('tree', {objectify.PYTYPE_ATTRIBUTE: 'TREE'})

No, that's redundant. Just use

    objectify.Element('tree')

As I said, the main reason to call Element() is to create a tree element. If
you want a data element, call DataElement(). So, no TreeElement() needed.


> > Btw. I've found that I rather often use
> > ElementBase.__len__(<someObjectifiedObj>)
> > to get the childcount/find out if an element has children.
> > What do you think about adding a hasChildren() or countChildren() function
> > to objectify?

Interesting. I didn't realise there isn't really a way to find out about the
children. They are in dir(), but only together with all methods etc. There's
.getchildren(), but that builds all children, so it's less efficient than just
counting them.

A countchildren() method on ObjectifiedElement would match getchildren(), but
it also adds another name that cannot be used to look up children. Maybe
"countchildren" is a good one in that regard, though, as it's not really a
good name for an XML tag.

I'm -0 on haschildren(), though, as you can always call countchildren() if you
expect the number to be small (note that this is about data binding, so very
large documents are unlikely already) or use iterchildren().next() if you
expect it to be really large. Child traversal is so fast that it shouldn't
make too much of a difference if you count 100 children or only the first one.
The method call overhead may even be the dominating factor here...

I'll add a countchildren() method, then.

Stefan



From behnel_ml at gkec.informatik.tu-darmstadt.de  Fri Sep 15 21:45:35 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Fri, 15 Sep 2006 21:45:35 +0200
Subject: [lxml-dev] Handling of <xsl:message terminate="yes"> ...
In-Reply-To: <4509D409.3060809@holloway.co.nz>
References: <4509D409.3060809@holloway.co.nz>
Message-ID: <450B02DF.10905@gkec.informatik.tu-darmstadt.de>

Hi,

Matthew Cruickshank wrote:
> I've been writing this XML chain/pipeline - sort of a multi-pipeline 
> version of Apache Cocoon and lxml has been great -- very fast!

Thanks for sharing that. Sounds like an interesting project. Any chance it
will be open-source? Feel free to post to the list when there is anything to see.


> I'm having some problems with <xsl:message> and I don't know how to read 
> errors caused by that. They terminate the processing but I can't find 
> the error message text (from within the <xsl:message> tag) anywhere... 
> it doesn't seem to be in error_log and no exception is thrown.

Right. I checked that. It was actually a deeper bug that prevented XSLT error
messages in general from turning up in the local error log of XSLT objects.
Should be fixed now (both trunk and 1.1 branch). A side effect is that some of
the error messages from XSLT are much clearer now.

Another side effect of the fix is that lxml.etree no longer compiles without
libxslt (which was possible in 1.0 by commenting out the line "include
'xslt.pxi'" in etree.pyx). Not too much of a drawback, though...


> Any ideas -- am I looking in the wrong place?
> 
> I'm using <xsl:message> to assert things about document structure and 
> runtime parameters (things that couldn't be asserted in RelaxNG, I think).

Well, 'document structure' is pretty much the thing that RelaxNG was made for,
so I don't see why XSLT should be any better. And you might also consider
generating (or modifying, parametrising, ...) an RNG on the fly, e.g. through
an XSLT.

What do you mean with "runtime parameters"? XSLT parameters?

Stefan


From sidnei at enfoldsystems.com  Sun Sep 17 04:13:08 2006
From: sidnei at enfoldsystems.com (Sidnei da Silva)
Date: Sat, 16 Sep 2006 23:13:08 -0300
Subject: [lxml-dev] Stylesheet Processing Instruction
Message-ID: <20060917021308.GK8537@cotia>

Hi there,

Someone asked me if lxml would handle a 'Stylesheet Processing
Instruction', which seems to be the way to embed the stylesheet into
the XML to be transformed. ie, if you use the said instruction and
open the XML in the browser (IE and Firefox?) the browser
automatically applies the transform.

Since the 'xsltproc' command also seems to do this, from it's man
page, I expected lxml to do as well, but didn't actually try.

So, can anyone confirm/deny if it's supported?

-- 
Sidnei da Silva
Enfold Systems                http://enfoldsystems.com
Fax +1 832 201 8856     Office +1 713 942 2377 Ext 214

From faassen at infrae.com  Mon Sep 18 13:13:18 2006
From: faassen at infrae.com (Martijn Faassen)
Date: Mon, 18 Sep 2006 13:13:18 +0200
Subject: [lxml-dev] egg package upload commands
Message-ID: <450E7F4E.2030802@infrae.com>

Hi there,

I wonder whether I'm doing something wrong, as previously this worked. 
To verify I'm not doing something stupid, could someone please say this 
is the proper sequence for uploading the lxml egg?

download the tgz from codespeak.net. This contains the .c file that has 
been created by our special version of Pyrex.

Then unpack it, and:

$ python2.4 setup.py build
[go to lib dir, strip the .so files]
$ python2.4 setup bdist_egg
$ python2.4 setup upload

The last step fails with the following error:

error: No dist file created in earlier command

Given that I get this with another package too, this means one of two 
things: I'm doing something wrong, or my version of setuptools has a bug...

Regards,

Martijn



From behnel_ml at gkec.informatik.tu-darmstadt.de  Mon Sep 18 07:40:59 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Mon, 18 Sep 2006 07:40:59 +0200
Subject: [lxml-dev] XSLT profiling
Message-ID: <450E316B.9040202@gkec.informatik.tu-darmstadt.de>

Hi,

while I was looking through the libxslt API to check for stylesheet PI
support, I stumbled over the profiling support that I had completely forgotten
about. The result is that lxml.etree can now profile stylesheet runs. If you
pass 'profile_run=True' to the transform call, the new 'xslt_profile' property
of the result tree will contain an ElementTree with profiling data for each
template, similar to the following (a 1-template XSLT result):

<profile>
 <template rank="1" match="/" name="" mode="" calls="1" time="1" average="1"/>
</profile>

Hope it's useful,
Stefan


From behnel_ml at gkec.informatik.tu-darmstadt.de  Mon Sep 18 06:10:04 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Mon, 18 Sep 2006 06:10:04 +0200
Subject: [lxml-dev] Stylesheet Processing Instruction
In-Reply-To: <20060917021308.GK8537@cotia>
References: <20060917021308.GK8537@cotia>
Message-ID: <450E1C1C.1070201@gkec.informatik.tu-darmstadt.de>

Hi Sidnei,

Sidnei da Silva wrote:
> Someone asked me if lxml would handle a 'Stylesheet Processing
> Instruction', which seems to be the way to embed the stylesheet into
> the XML to be transformed. ie, if you use the said instruction and
> open the XML in the browser (IE and Firefox?) the browser
> automatically applies the transform.
> 
> Since the 'xsltproc' command also seems to do this, from it's man
> page, I expected lxml to do as well, but didn't actually try.

You can't always infer the features of lxml from the command line tool
xsltproc (and vice versa) as both serve different purposes.

That said, I tried to pass a stylesheet to xsltproc using a PI and couldn't
manage to start it without passing a separate stylesheet document, so I
currently assume the man page to be wrong in that regard. Might be a good idea
to ask the same question on the libxslt mailing list.

lxml does not currently support this, first point being that we don't have an
API for running an XSLT without a stylesheet document. And after a quick
glance through the libxslt API, I currently do not see a trivial way to
implement that.

Stefan


From behnel_ml at gkec.informatik.tu-darmstadt.de  Mon Sep 18 16:00:13 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Mon, 18 Sep 2006 16:00:13 +0200
Subject: [lxml-dev] egg package upload commands
In-Reply-To: <450E7F4E.2030802@infrae.com>
References: <450E7F4E.2030802@infrae.com>
Message-ID: <450EA66D.8010306@gkec.informatik.tu-darmstadt.de>



Martijn Faassen wrote:
> I wonder whether I'm doing something wrong, as previously this worked. 
> To verify I'm not doing something stupid, could someone please say this 
> is the proper sequence for uploading the lxml egg?
> 
> download the tgz from codespeak.net. This contains the .c file that has 
> been created by our special version of Pyrex.
> 
> Then unpack it, and:
> 
> $ python2.4 setup.py build
> [go to lib dir, strip the .so files]
> $ python2.4 setup bdist_egg
> $ python2.4 setup upload

You have to run "bdist_egg upload" on one command line as the upload takes the
files that were created in the same run.

I'm planning to release a 1.1.1 pretty soon, BTW, as there were a couple of
bugs in lxml.objectify (and there are a few nice new features on the trunk,
too). So, no hurry with eggs for 1.1.

Stefan


From sidnei at enfoldsystems.com  Mon Sep 18 17:27:33 2006
From: sidnei at enfoldsystems.com (Sidnei da Silva)
Date: Mon, 18 Sep 2006 12:27:33 -0300
Subject: [lxml-dev] Stylesheet Processing Instruction
In-Reply-To: <450E1C1C.1070201@gkec.informatik.tu-darmstadt.de>
References: <20060917021308.GK8537@cotia>
	<450E1C1C.1070201@gkec.informatik.tu-darmstadt.de>
Message-ID: <20060918152733.GC3853@cotia>

On Mon, Sep 18, 2006 at 06:10:04AM +0200, Stefan Behnel wrote:
| You can't always infer the features of lxml from the command line tool
| xsltproc (and vice versa) as both serve different purposes.
| 
| That said, I tried to pass a stylesheet to xsltproc using a PI and couldn't
| manage to start it without passing a separate stylesheet document, so I
| currently assume the man page to be wrong in that regard. Might be a good idea
| to ask the same question on the libxslt mailing list.

Weird, it works for me here. I've used the example on this page:

  http://www.xml.com/lpt/a/1102

Saved the XML as 'square.xml', saved the stylesheet as
'squareAsHTML.xsl' in the same folder then ran 'xsltproc square.xml'
and it did perform the transformation just fine.

| lxml does not currently support this, first point being that we don't have an
| API for running an XSLT without a stylesheet document. And after a quick
| glance through the libxslt API, I currently do not see a trivial way to
| implement that.

Maybe if I can point out the lines that does it in the xsltproc
source? I'm trying to find those right now.

-- 
Sidnei da Silva
Enfold Systems                http://enfoldsystems.com
Fax +1 832 201 8856     Office +1 713 942 2377 Ext 214

From sidnei at enfoldsystems.com  Mon Sep 18 17:56:45 2006
From: sidnei at enfoldsystems.com (Sidnei da Silva)
Date: Mon, 18 Sep 2006 12:56:45 -0300
Subject: [lxml-dev] Stylesheet Processing Instruction
In-Reply-To: <20060918152733.GC3853@cotia>
References: <20060917021308.GK8537@cotia>
	<450E1C1C.1070201@gkec.informatik.tu-darmstadt.de>
	<20060918152733.GC3853@cotia>
Message-ID: <20060918155645.GD3853@cotia>

| Maybe if I can point out the lines that does it in the xsltproc
| source? I'm trying to find those right now.

Looks like this is the trick:

                cur = xsltLoadStylesheetPI(style);
                if (cur != NULL) {
                    /* it is an embedded stylesheet */
                    xsltProcess(style, cur, argv[i]);
                    xsltFreeStylesheet(cur);
                    cur = NULL;
                    goto done;
                }

-- 
Sidnei da Silva
Enfold Systems                http://enfoldsystems.com
Fax +1 832 201 8856     Office +1 713 942 2377 Ext 214

From faassen at infrae.com  Mon Sep 18 18:23:49 2006
From: faassen at infrae.com (Martijn Faassen)
Date: Mon, 18 Sep 2006 18:23:49 +0200
Subject: [lxml-dev] egg package upload commands
In-Reply-To: <450EA66D.8010306@gkec.informatik.tu-darmstadt.de>
References: <450E7F4E.2030802@infrae.com>
	<450EA66D.8010306@gkec.informatik.tu-darmstadt.de>
Message-ID: <450EC815.2080708@infrae.com>

Stefan Behnel wrote:
> 
> Martijn Faassen wrote:
>> I wonder whether I'm doing something wrong, as previously this worked. 
>> To verify I'm not doing something stupid, could someone please say this 
>> is the proper sequence for uploading the lxml egg?
>>
>> download the tgz from codespeak.net. This contains the .c file that has 
>> been created by our special version of Pyrex.
>>
>> Then unpack it, and:
>>
>> $ python2.4 setup.py build
>> [go to lib dir, strip the .so files]
>> $ python2.4 setup bdist_egg
>> $ python2.4 setup upload
> 
> You have to run "bdist_egg upload" on one command line as the upload takes the
> files that were created in the same run.

Argh, this slipped my mind. Okay, that explains it, thanks! We should 
write this in some file somewhere...

> I'm planning to release a 1.1.1 pretty soon, BTW, as there were a couple of
> bugs in lxml.objectify (and there are a few nice new features on the trunk,
> too). So, no hurry with eggs for 1.1.

Okay.

Regards,

Martijn


From faassen at infrae.com  Mon Sep 18 18:34:31 2006
From: faassen at infrae.com (Martijn Faassen)
Date: Mon, 18 Sep 2006 18:34:31 +0200
Subject: [lxml-dev] Stylesheet Processing Instruction
In-Reply-To: <20060918155645.GD3853@cotia>
References: <20060917021308.GK8537@cotia>	<450E1C1C.1070201@gkec.informatik.tu-darmstadt.de>	<20060918152733.GC3853@cotia>
	<20060918155645.GD3853@cotia>
Message-ID: <450ECA97.6050507@infrae.com>

Sidnei da Silva wrote:
> | Maybe if I can point out the lines that does it in the xsltproc
> | source? I'm trying to find those right now.
> 
> Looks like this is the trick:
> 
>                 cur = xsltLoadStylesheetPI(style);
>                 if (cur != NULL) {
>                     /* it is an embedded stylesheet */
>                     xsltProcess(style, cur, argv[i]);
>                     xsltFreeStylesheet(cur);
>                     cur = NULL;
>                     goto done;
>                 }


It would of course be good if we didn't do this by default and that this 
should be enabled explicitly, as it's a potential security concern to 
pull in stylesheets from URLs. I'm not sure right now what the API would 
be like at all though in the first place.

Regards,

Martijn

From sidnei at enfoldsystems.com  Tue Sep 19 02:30:21 2006
From: sidnei at enfoldsystems.com (Sidnei da Silva)
Date: Mon, 18 Sep 2006 21:30:21 -0300
Subject: [lxml-dev] Stylesheet Processing Instruction
In-Reply-To: <450ECA97.6050507@infrae.com>
References: <20060917021308.GK8537@cotia>
	<450E1C1C.1070201@gkec.informatik.tu-darmstadt.de>
	<20060918152733.GC3853@cotia> <20060918155645.GD3853@cotia>
	<450ECA97.6050507@infrae.com>
Message-ID: <20060919003021.GQ3853@cotia>

On Mon, Sep 18, 2006 at 06:34:31PM +0200, Martijn Faassen wrote:
| It would of course be good if we didn't do this by default and that this 
| should be enabled explicitly, as it's a potential security concern to 
| pull in stylesheets from URLs. I'm not sure right now what the API would 
| be like at all though in the first place.

So, supposing I come up with a reasonable API it wouldn't be hard at
all to add it? For a start, how's XInclude handled today? Or maybe it
isn't?

-- 
Sidnei da Silva
Enfold Systems                http://enfoldsystems.com
Fax +1 832 201 8856     Office +1 713 942 2377 Ext 214

From lxml at holloway.co.nz  Tue Sep 19 02:55:18 2006
From: lxml at holloway.co.nz (Matthew Cruickshank)
Date: Tue, 19 Sep 2006 12:55:18 +1200
Subject: [lxml-dev] Handling of <xsl:message terminate="yes"> ...
In-Reply-To: <450B02DF.10905@gkec.informatik.tu-darmstadt.de>
References: <4509D409.3060809@holloway.co.nz>
	<450B02DF.10905@gkec.informatik.tu-darmstadt.de>
Message-ID: <450F3FF6.3030601@holloway.co.nz>

Stefan Behnel wrote:
> Right. I checked that. It was actually a deeper bug that prevented 
> XSLT error
> messages in general from turning up in the local error log of XSLT 
> objects.
>   

Cool, thanks for fixing that! :)

> Well, 'document structure' is pretty much the thing that RelaxNG was 
> made for,
> [..]
> What do you mean with "runtime parameters"? XSLT parameters?
>   

Yes, XSLT parameters. Most of it is in XML and could be done in either 
RelaxNG or XSLT, but there are parts of the XML that are only 
transformable with the right XSLT parameters.

> Thanks for sharing that. Sounds like an interesting project. Any 
> chance it
> will be open-source? Feel free to post to the list when there is 
> anything to see.
It's for the company I work for so I don't know (it's not a proprietary 
advantage -- we just use it to build our site, so perhaps we could share 
it). As well as the pipeline it's quite nice in that it generates 
XHTML/XUL from the same source files, with high-level widgets like tab 
controls, hierarchical menus, and as-you-type validation. Here's a blog 
entry I wrote about it...  
http://plutoblog.onesquared.net/2006/08/10/bear-patrol-2/


.Matthew Cruickshank
http://holloway.co.nz/






-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20060919/1bf97fd8/attachment.htm 

From sidnei at enfoldsystems.com  Tue Sep 19 04:02:40 2006
From: sidnei at enfoldsystems.com (Sidnei da Silva)
Date: Mon, 18 Sep 2006 23:02:40 -0300
Subject: [lxml-dev] Stylesheet Processing Instruction
In-Reply-To: <20060919003021.GQ3853@cotia>
References: <20060917021308.GK8537@cotia>
	<450E1C1C.1070201@gkec.informatik.tu-darmstadt.de>
	<20060918152733.GC3853@cotia> <20060918155645.GD3853@cotia>
	<450ECA97.6050507@infrae.com> <20060919003021.GQ3853@cotia>
Message-ID: <20060919020239.GR3853@cotia>

On Mon, Sep 18, 2006 at 09:30:21PM -0300, Sidnei da Silva wrote:
| On Mon, Sep 18, 2006 at 06:34:31PM +0200, Martijn Faassen wrote:
| | It would of course be good if we didn't do this by default and that this 
| | should be enabled explicitly, as it's a potential security concern to 
| | pull in stylesheets from URLs. I'm not sure right now what the API would 
| | be like at all though in the first place.

FWIW, I think the API should basically be a 'loadStylesheetFromPI'
function that takes a tree and returns the same kind of object that
etree.XSLT() returns if it can load the stylesheet or None if it can't.

I actually tried to go ahead and create a patch for this but lxml
doesn't even compile here.

On Win32, with Pyrex 0.9.4.1 (with the patch applied) I get this error:

 src\lxml\etree.c(49704) : error C2137: empty character constant
 error: command '"C:\Arquivos de programas\Microsoft Visual C++ Toolkit 2003\bin\cl.exe"' failed with exit status 2

The line in question has:

          if (*t->s == '^@')
              continue; /* shortcut for erased string entries */

On Ubuntu Edgy, with Pyrex 0.9.3 it doesn't work either.

Maybe a non-released version of Pyrex is needed?

-- 
Sidnei da Silva
Enfold Systems                http://enfoldsystems.com
Fax +1 832 201 8856     Office +1 713 942 2377 Ext 214

From behnel_ml at gkec.informatik.tu-darmstadt.de  Tue Sep 19 08:40:10 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Tue, 19 Sep 2006 08:40:10 +0200
Subject: [lxml-dev] Stylesheet Processing Instruction
In-Reply-To: <20060919020239.GR3853@cotia>
References: <20060917021308.GK8537@cotia>	<450E1C1C.1070201@gkec.informatik.tu-darmstadt.de>	<20060918152733.GC3853@cotia>
	<20060918155645.GD3853@cotia>	<450ECA97.6050507@infrae.com>
	<20060919003021.GQ3853@cotia> <20060919020239.GR3853@cotia>
Message-ID: <450F90CA.6060902@gkec.informatik.tu-darmstadt.de>

Hi Sidnei, Martijn,

Sidnei da Silva wrote:
> On Mon, Sep 18, 2006 at 09:30:21PM -0300, Sidnei da Silva wrote:
> | On Mon, Sep 18, 2006 at 06:34:31PM +0200, Martijn Faassen wrote:
> | | It would of course be good if we didn't do this by default and that this 
> | | should be enabled explicitly, as it's a potential security concern to 
> | | pull in stylesheets from URLs. I'm not sure right now what the API would 
> | | be like at all though in the first place.

I think the right API would simple be a call to ElementTree.xslt() without
stylesheet. Maybe we should then raise an exception if no PI is found.


> I actually tried to go ahead and create a patch for this but lxml
> doesn't even compile here.
> 
> On Win32, with Pyrex 0.9.4.1 (with the patch applied) I get this error:
> 
>  src\lxml\etree.c(49704) : error C2137: empty character constant
>  error: command '"C:\Arquivos de programas\Microsoft Visual C++ Toolkit 2003\bin\cl.exe"' failed with exit status 2
> 
> The line in question has:
> 
>           if (*t->s == '^@')
>               continue; /* shortcut for erased string entries */

That's a known bug. Please use the updated Pyrex version from the lxml SVN
(see build.txt).


> On Ubuntu Edgy, with Pyrex 0.9.3 it doesn't work either.

Sure, it can't handle some of the newer lxml features (and may even have some
bugs that prevent it from compiling lxml at all, even older versions).


Stefan

From sidnei at enfoldsystems.com  Tue Sep 19 09:00:43 2006
From: sidnei at enfoldsystems.com (Sidnei da Silva)
Date: Tue, 19 Sep 2006 04:00:43 -0300
Subject: [lxml-dev] Stylesheet Processing Instruction
In-Reply-To: <450F90CA.6060902@gkec.informatik.tu-darmstadt.de>
References: <20060917021308.GK8537@cotia>
	<450E1C1C.1070201@gkec.informatik.tu-darmstadt.de>
	<20060918152733.GC3853@cotia> <20060918155645.GD3853@cotia>
	<450ECA97.6050507@infrae.com> <20060919003021.GQ3853@cotia>
	<20060919020239.GR3853@cotia>
	<450F90CA.6060902@gkec.informatik.tu-darmstadt.de>
Message-ID: <20060919070042.GT3853@cotia>

On Tue, Sep 19, 2006 at 08:40:10AM +0200, Stefan Behnel wrote:
| I think the right API would simple be a call to ElementTree.xslt() without
| stylesheet. Maybe we should then raise an exception if no PI is found.

Yeah, that would work great. And you also can pass an access_control
argument if you wanted to. I've tried going through a similar route
(after sorting out the right version of Pyrex) but maybe I'm on the
wrong track. 

I've added a process_pi=True to XSLT() and then called:

  XSLT(self, process_pi=True) 

from ElementTree.xslt() and then in XSLT.__init__() I did:

        if process_pi:
            c_style = xslt.xsltLoadStylesheetPI(c_doc)
        else:
            c_style = xslt.xsltParseStylesheetDoc(c_doc)

However that didn't work for some reason. Maybe c_doc at that point
doesn't have the processing instruction anymore for some reason.

| >           if (*t->s == '^@')
| >               continue; /* shortcut for erased string entries */
| 
| That's a known bug. Please use the updated Pyrex version from the lxml SVN
| (see build.txt).

Ok, will do.

-- 
Sidnei da Silva
Enfold Systems                http://enfoldsystems.com
Fax +1 832 201 8856     Office +1 713 942 2377 Ext 214

From faassen at infrae.com  Tue Sep 19 12:27:55 2006
From: faassen at infrae.com (Martijn Faassen)
Date: Tue, 19 Sep 2006 12:27:55 +0200
Subject: [lxml-dev] Stylesheet Processing Instruction
In-Reply-To: <450F90CA.6060902@gkec.informatik.tu-darmstadt.de>
References: <20060917021308.GK8537@cotia>	<450E1C1C.1070201@gkec.informatik.tu-darmstadt.de>	<20060918152733.GC3853@cotia>	<20060918155645.GD3853@cotia>	<450ECA97.6050507@infrae.com>	<20060919003021.GQ3853@cotia>
	<20060919020239.GR3853@cotia>
	<450F90CA.6060902@gkec.informatik.tu-darmstadt.de>
Message-ID: <450FC62B.2070905@infrae.com>

Stefan Behnel wrote:
> Hi Sidnei, Martijn,
> 
> Sidnei da Silva wrote:
>> On Mon, Sep 18, 2006 at 09:30:21PM -0300, Sidnei da Silva wrote:
>> | On Mon, Sep 18, 2006 at 06:34:31PM +0200, Martijn Faassen wrote:
>> | | It would of course be good if we didn't do this by default and that this 
>> | | should be enabled explicitly, as it's a potential security concern to 
>> | | pull in stylesheets from URLs. I'm not sure right now what the API would 
>> | | be like at all though in the first place.
> 
> I think the right API would simple be a call to ElementTree.xslt() without
> stylesheet. Maybe we should then raise an exception if no PI is found.

I'd prefer something more explicit myself. I find it a bit dangerous to 
stop requiring arguments. And then, if you forget the argument, it'll 
load off stylesheets from some URL somewhere? I'd prefer a separate 
function to execute the PI-based stylesheet which could indeed have the 
behavior you suggest. I think it is also valuable to have a way to get 
to the styleshet object directly, in case someone wants to cache or 
modify this stylesheet.

Regards,

Martijn

From sidnei at enfoldsystems.com  Tue Sep 19 15:20:43 2006
From: sidnei at enfoldsystems.com (Sidnei da Silva)
Date: Tue, 19 Sep 2006 10:20:43 -0300
Subject: [lxml-dev] Stylesheet Processing Instruction
In-Reply-To: <450FC62B.2070905@infrae.com>
References: <20060917021308.GK8537@cotia>
	<450E1C1C.1070201@gkec.informatik.tu-darmstadt.de>
	<20060918152733.GC3853@cotia> <20060918155645.GD3853@cotia>
	<450ECA97.6050507@infrae.com> <20060919003021.GQ3853@cotia>
	<20060919020239.GR3853@cotia>
	<450F90CA.6060902@gkec.informatik.tu-darmstadt.de>
	<450FC62B.2070905@infrae.com>
Message-ID: <20060919132042.GU3853@cotia>

On Tue, Sep 19, 2006 at 12:27:55PM +0200, Martijn Faassen wrote:
| I'd prefer something more explicit myself. I find it a bit dangerous to 
| stop requiring arguments. And then, if you forget the argument, it'll 
| load off stylesheets from some URL somewhere? I'd prefer a separate 
| function to execute the PI-based stylesheet which could indeed have the 
| behavior you suggest. I think it is also valuable to have a way to get 
| to the styleshet object directly, in case someone wants to cache or 
| modify this stylesheet.

That was my original proposal *wink*.

-- 
Sidnei da Silva
Enfold Systems                http://enfoldsystems.com
Fax +1 832 201 8856     Office +1 713 942 2377 Ext 214

From faassen at infrae.com  Tue Sep 19 15:42:11 2006
From: faassen at infrae.com (Martijn Faassen)
Date: Tue, 19 Sep 2006 15:42:11 +0200
Subject: [lxml-dev] Stylesheet Processing Instruction
In-Reply-To: <20060919132042.GU3853@cotia>
References: <20060917021308.GK8537@cotia>	<450E1C1C.1070201@gkec.informatik.tu-darmstadt.de>	<20060918152733.GC3853@cotia>
	<20060918155645.GD3853@cotia>	<450ECA97.6050507@infrae.com>
	<20060919003021.GQ3853@cotia>	<20060919020239.GR3853@cotia>	<450F90CA.6060902@gkec.informatik.tu-darmstadt.de>	<450FC62B.2070905@infrae.com>
	<20060919132042.GU3853@cotia>
Message-ID: <450FF3B3.1060004@infrae.com>

Sidnei da Silva wrote:
> On Tue, Sep 19, 2006 at 12:27:55PM +0200, Martijn Faassen wrote:
> | I'd prefer something more explicit myself. I find it a bit dangerous to 
> | stop requiring arguments. And then, if you forget the argument, it'll 
> | load off stylesheets from some URL somewhere? I'd prefer a separate 
> | function to execute the PI-based stylesheet which could indeed have the 
> | behavior you suggest. I think it is also valuable to have a way to get 
> | to the styleshet object directly, in case someone wants to cache or 
> | modify this stylesheet.
> 
> That was my original proposal *wink*.

I was just listing use cases and various API worries I have - your 
original proposal was not backed up by such. I find that this helps 
making more clear what API we really want.

I thinking having an API to pick up the PI automatically would be 
convenient, but I don't think it should be 'just leave off the parameter 
of a well-known method' to make it kick in. I also think there are 
usecases beyond just executing the transformation, listed in my previous 
mail, which make your original proposal attractive.

Regards,

Martijn

From sidnei at enfoldsystems.com  Tue Sep 19 15:53:43 2006
From: sidnei at enfoldsystems.com (Sidnei da Silva)
Date: Tue, 19 Sep 2006 10:53:43 -0300
Subject: [lxml-dev] Stylesheet Processing Instruction
In-Reply-To: <450FF3B3.1060004@infrae.com>
References: <450E1C1C.1070201@gkec.informatik.tu-darmstadt.de>
	<20060918152733.GC3853@cotia> <20060918155645.GD3853@cotia>
	<450ECA97.6050507@infrae.com> <20060919003021.GQ3853@cotia>
	<20060919020239.GR3853@cotia>
	<450F90CA.6060902@gkec.informatik.tu-darmstadt.de>
	<450FC62B.2070905@infrae.com> <20060919132042.GU3853@cotia>
	<450FF3B3.1060004@infrae.com>
Message-ID: <20060919135343.GV3853@cotia>

On Tue, Sep 19, 2006 at 03:42:11PM +0200, Martijn Faassen wrote:
| >That was my original proposal *wink*.
| 
| I was just listing use cases and various API worries I have - your 
| original proposal was not backed up by such. I find that this helps 
| making more clear what API we really want.

I agree with that. I admire you for doing this because it's
something I'm not good at.

| I thinking having an API to pick up the PI automatically would be 
| convenient, but I don't think it should be 'just leave off the parameter 
| of a well-known method' to make it kick in. I also think there are 
| usecases beyond just executing the transformation, listed in my previous 
| mail, which make your original proposal attractive.

I'm glad we have someone like you Martijn. I've been away from lxml
for a couple months only and I'm exceedingly happy to see how it has
evolved in the meantime. And it's not different from any other project
that you have been involved.

Personally, I wish people would hear you more than they do. A certain
known application server out there would be in a much better position
these days.

-- 
Sidnei da Silva
Enfold Systems                http://enfoldsystems.com
Fax +1 832 201 8856     Office +1 713 942 2377 Ext 214

From faassen at infrae.com  Tue Sep 19 17:35:56 2006
From: faassen at infrae.com (Martijn Faassen)
Date: Tue, 19 Sep 2006 17:35:56 +0200
Subject: [lxml-dev] Stylesheet Processing Instruction
In-Reply-To: <20060919135343.GV3853@cotia>
References: <450E1C1C.1070201@gkec.informatik.tu-darmstadt.de>	<20060918152733.GC3853@cotia>
	<20060918155645.GD3853@cotia>	<450ECA97.6050507@infrae.com>
	<20060919003021.GQ3853@cotia>	<20060919020239.GR3853@cotia>	<450F90CA.6060902@gkec.informatik.tu-darmstadt.de>	<450FC62B.2070905@infrae.com>
	<20060919132042.GU3853@cotia>	<450FF3B3.1060004@infrae.com>
	<20060919135343.GV3853@cotia>
Message-ID: <45100E5C.2000502@infrae.com>

Sidnei da Silva wrote:
> On Tue, Sep 19, 2006 at 03:42:11PM +0200, Martijn Faassen wrote:
[snip
> I'm glad we have someone like you Martijn. I've been away from lxml
> for a couple months only and I'm exceedingly happy to see how it has
> evolved in the meantime. And it's not different from any other project
> that you have been involved.

I can't take a lot of credit for that; that's been Stephan Behnel's work 
mostly, so he deserves the praise. I currently just hang out and give 
API comments here and there.

> Personally, I wish people would hear you more than they do. A certain
> known application server out there would be in a much better position
> these days.

Thanks for the compliments, I'm a bit embarassed now, but it's nice to 
hear. :)

It's hard to say whether Zope would be in a much better position; I 
don't think my words go unheeded that much anyway so I share in the 
blame for any problems. I'm in the Zope Foundation board now, after all. 
Anyway, we're not in a terrible position and we're getting better.

Regards,

Martijn


From behnel_ml at gkec.informatik.tu-darmstadt.de  Tue Sep 19 18:14:04 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Tue, 19 Sep 2006 18:14:04 +0200
Subject: [lxml-dev] Stylesheet Processing Instruction
In-Reply-To: <450FF3B3.1060004@infrae.com>
References: <20060917021308.GK8537@cotia>	<450E1C1C.1070201@gkec.informatik.tu-darmstadt.de>	<20060918152733.GC3853@cotia>
	<20060918155645.GD3853@cotia>	<450ECA97.6050507@infrae.com>
	<20060919003021.GQ3853@cotia>	<20060919020239.GR3853@cotia>	<450F90CA.6060902@gkec.informatik.tu-darmstadt.de>	<450FC62B.2070905@infrae.com>
	<20060919132042.GU3853@cotia> <450FF3B3.1060004@infrae.com>
Message-ID: <4510174C.3010209@gkec.informatik.tu-darmstadt.de>

Hi Martijn,

Martijn Faassen wrote:
> I was just listing use cases and various API worries I have - your
> original proposal was not backed up by such. I find that this helps
> making more clear what API we really want.

Right.


> I thinking having an API to pick up the PI automatically would be
> convenient, but I don't think it should be 'just leave off the parameter
> of a well-known method' to make it kick in.

True, that's not very explicit.


> I also think there are
> usecases beyond just executing the transformation, listed in my previous
> mail, which make your original proposal attractive.

Ok, what we are getting at in that case is that we provide a convenience
method for the following code:

  tree = etree.parse(<some XML with XSLT PI>)
  root = tree.getroot()
  pi = root.getprevious()
  ... read 'href' value from pi.text if 'type' is XSL ...
  transform = etree.XSLT(etree.parse(href))
  transform(tree)

The PI attribute parsing part is maybe the most tricky one.

Questions:

* is it worth special casing?
* are there other cases where PIs can or should be special cased?
* should we maybe extend the PI class for parsing PI parameters? (-> attrib?)
  would that put us into a position where special casing is no longer
  required?

Any comments?

Stefan

From sidnei at enfoldsystems.com  Tue Sep 19 18:22:41 2006
From: sidnei at enfoldsystems.com (Sidnei da Silva)
Date: Tue, 19 Sep 2006 13:22:41 -0300
Subject: [lxml-dev] Stylesheet Processing Instruction
In-Reply-To: <4510174C.3010209@gkec.informatik.tu-darmstadt.de>
References: <20060918152733.GC3853@cotia> <20060918155645.GD3853@cotia>
	<450ECA97.6050507@infrae.com> <20060919003021.GQ3853@cotia>
	<20060919020239.GR3853@cotia>
	<450F90CA.6060902@gkec.informatik.tu-darmstadt.de>
	<450FC62B.2070905@infrae.com> <20060919132042.GU3853@cotia>
	<450FF3B3.1060004@infrae.com>
	<4510174C.3010209@gkec.informatik.tu-darmstadt.de>
Message-ID: <20060919162241.GZ3853@cotia>

On Tue, Sep 19, 2006 at 06:14:04PM +0200, Stefan Behnel wrote:
| Ok, what we are getting at in that case is that we provide a convenience
| method for the following code:

I think there's an extra step in there:

|   tree = etree.parse(<some XML with XSLT PI>)
|   root = tree.getroot()
|   pi = root.getprevious()
|   ... read 'href' value from pi.text if 'type' is XSL ...
+   ... resolve 'href' through resolvers
|   transform = etree.XSLT(etree.parse(href))
|   transform(tree)

I believe a CachingResolver then could be used for the caching that
Martijn envisioned.

| The PI attribute parsing part is maybe the most tricky one.
| 
| Questions:
| 
| * is it worth special casing?

Well, since libxml2 provides such an API, there's probably value in
it. In fact, we should probably look at the source for
xsltLoadStylesheetPI to see if it does anything more than what you're
proposing.

| * are there other cases where PIs can or should be special cased?

For stylesheets specifically, I've saw an example where the stylesheet
href was a fragment identifier ('#stylesheet') and the stylesheet was
then *in the same XML* instead of a separate file.

| * should we maybe extend the PI class for parsing PI parameters? (-> attrib?)
|   would that put us into a position where special casing is no longer
|   required?

That I can't comment on. I have never really used PI other than for
stylesheets. In fact, all the googling that I did seems to indicate
it's pretty much the only thing it's used for these days.

-- 
Sidnei da Silva
Enfold Systems                http://enfoldsystems.com
Fax +1 832 201 8856     Office +1 713 942 2377 Ext 214

From sidnei at enfoldsystems.com  Tue Sep 19 18:27:32 2006
From: sidnei at enfoldsystems.com (Sidnei da Silva)
Date: Tue, 19 Sep 2006 13:27:32 -0300
Subject: [lxml-dev] Stylesheet Processing Instruction
In-Reply-To: <20060919162241.GZ3853@cotia>
References: <20060918155645.GD3853@cotia> <450ECA97.6050507@infrae.com>
	<20060919003021.GQ3853@cotia> <20060919020239.GR3853@cotia>
	<450F90CA.6060902@gkec.informatik.tu-darmstadt.de>
	<450FC62B.2070905@infrae.com> <20060919132042.GU3853@cotia>
	<450FF3B3.1060004@infrae.com>
	<4510174C.3010209@gkec.informatik.tu-darmstadt.de>
	<20060919162241.GZ3853@cotia>
Message-ID: <20060919162732.GA3853@cotia>

On Tue, Sep 19, 2006 at 01:22:41PM -0300, Sidnei da Silva wrote:
| That I can't comment on. I have never really used PI other than for
| stylesheets. In fact, all the googling that I did seems to indicate
| it's pretty much the only thing it's used for these days.

Actually, let me correct that. Docbook makes extensive use of
processing instructions.

-- 
Sidnei da Silva
Enfold Systems                http://enfoldsystems.com
Fax +1 832 201 8856     Office +1 713 942 2377 Ext 214

From behnel_ml at gkec.informatik.tu-darmstadt.de  Tue Sep 19 19:55:42 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Tue, 19 Sep 2006 19:55:42 +0200
Subject: [lxml-dev] Stylesheet Processing Instruction
In-Reply-To: <20060919162732.GA3853@cotia>
References: <20060918155645.GD3853@cotia> <450ECA97.6050507@infrae.com>
	<20060919003021.GQ3853@cotia> <20060919020239.GR3853@cotia>
	<450F90CA.6060902@gkec.informatik.tu-darmstadt.de>
	<450FC62B.2070905@infrae.com> <20060919132042.GU3853@cotia>
	<450FF3B3.1060004@infrae.com>
	<4510174C.3010209@gkec.informatik.tu-darmstadt.de>
	<20060919162241.GZ3853@cotia> <20060919162732.GA3853@cotia>
Message-ID: <45102F1E.7040202@gkec.informatik.tu-darmstadt.de>

Hi,

Sidnei da Silva wrote:
> On Tue, Sep 19, 2006 at 01:22:41PM -0300, Sidnei da Silva wrote:
> | That I can't comment on. I have never really used PI other than for
> | stylesheets. In fact, all the googling that I did seems to indicate
> | it's pretty much the only thing it's used for these days.
> 
> Actually, let me correct that. Docbook makes extensive use of
> processing instructions.

So? Do they use 'normal' XML attributes?

If yes, then providing a special implementation for PI.attrib might be the
right thing to do.

Stefan

From behnel_ml at gkec.informatik.tu-darmstadt.de  Tue Sep 19 19:58:36 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Tue, 19 Sep 2006 19:58:36 +0200
Subject: [lxml-dev] Stylesheet Processing Instruction
In-Reply-To: <20060919162241.GZ3853@cotia>
References: <20060918152733.GC3853@cotia> <20060918155645.GD3853@cotia>
	<450ECA97.6050507@infrae.com> <20060919003021.GQ3853@cotia>
	<20060919020239.GR3853@cotia>
	<450F90CA.6060902@gkec.informatik.tu-darmstadt.de>
	<450FC62B.2070905@infrae.com> <20060919132042.GU3853@cotia>
	<450FF3B3.1060004@infrae.com>
	<4510174C.3010209@gkec.informatik.tu-darmstadt.de>
	<20060919162241.GZ3853@cotia>
Message-ID: <45102FCC.3040703@gkec.informatik.tu-darmstadt.de>

Hi Sidnei,

Sidnei da Silva wrote:
> On Tue, Sep 19, 2006 at 06:14:04PM +0200, Stefan Behnel wrote:
> | Ok, what we are getting at in that case is that we provide a convenience
> | method for the following code:
> 
> I think there's an extra step in there:
> 
> |   tree = etree.parse(<some XML with XSLT PI>)
> |   root = tree.getroot()
> |   pi = root.getprevious()
> |   ... read 'href' value from pi.text if 'type' is XSL ...
> +   ... resolve 'href' through resolvers
> |   transform = etree.XSLT(etree.parse(href))
> |   transform(tree)

No, that's already done by parse(), see resolvers.txt.


> | The PI attribute parsing part is maybe the most tricky one.
> | 
> | Questions:
> | 
> | * is it worth special casing?
> 
> Well, since libxml2 provides such an API, there's probably value in
> it.

Bad argument. libxml2 and libxslt have all sorts of redundant APIs.


> In fact, we should probably look at the source for
> xsltLoadStylesheetPI to see if it does anything more than what you're
> proposing.
> 
> | * are there other cases where PIs can or should be special cased?
> 
> For stylesheets specifically, I've saw an example where the stylesheet
> href was a fragment identifier ('#stylesheet') and the stylesheet was
> then *in the same XML* instead of a separate file.

That could easily be handled via Python resolvers.

So, given the fact that the libxml2 API can't actually resolve URLs through
Python resolvers, we should consider not using it and instead making the way I
sketched above a bit smoother.

Stefan

From sidnei at enfoldsystems.com  Tue Sep 19 20:03:22 2006
From: sidnei at enfoldsystems.com (Sidnei da Silva)
Date: Tue, 19 Sep 2006 15:03:22 -0300
Subject: [lxml-dev] Stylesheet Processing Instruction
In-Reply-To: <45102F1E.7040202@gkec.informatik.tu-darmstadt.de>
References: <20060919003021.GQ3853@cotia> <20060919020239.GR3853@cotia>
	<450F90CA.6060902@gkec.informatik.tu-darmstadt.de>
	<450FC62B.2070905@infrae.com> <20060919132042.GU3853@cotia>
	<450FF3B3.1060004@infrae.com>
	<4510174C.3010209@gkec.informatik.tu-darmstadt.de>
	<20060919162241.GZ3853@cotia> <20060919162732.GA3853@cotia>
	<45102F1E.7040202@gkec.informatik.tu-darmstadt.de>
Message-ID: <20060919180322.GB3853@cotia>

On Tue, Sep 19, 2006 at 07:55:42PM +0200, Stefan Behnel wrote:
| So? Do they use 'normal' XML attributes?

Apparently yes.

| If yes, then providing a special implementation for PI.attrib might be the
| right thing to do.

+1 then.

-- 
Sidnei da Silva
Enfold Systems                http://enfoldsystems.com
Fax +1 832 201 8856     Office +1 713 942 2377 Ext 214

From sidnei at enfoldsystems.com  Tue Sep 19 20:06:13 2006
From: sidnei at enfoldsystems.com (Sidnei da Silva)
Date: Tue, 19 Sep 2006 15:06:13 -0300
Subject: [lxml-dev] Stylesheet Processing Instruction
In-Reply-To: <45102FCC.3040703@gkec.informatik.tu-darmstadt.de>
References: <450ECA97.6050507@infrae.com> <20060919003021.GQ3853@cotia>
	<20060919020239.GR3853@cotia>
	<450F90CA.6060902@gkec.informatik.tu-darmstadt.de>
	<450FC62B.2070905@infrae.com> <20060919132042.GU3853@cotia>
	<450FF3B3.1060004@infrae.com>
	<4510174C.3010209@gkec.informatik.tu-darmstadt.de>
	<20060919162241.GZ3853@cotia>
	<45102FCC.3040703@gkec.informatik.tu-darmstadt.de>
Message-ID: <20060919180613.GC3853@cotia>

On Tue, Sep 19, 2006 at 07:58:36PM +0200, Stefan Behnel wrote:
| > I think there's an extra step in there:
| > 
| > |   tree = etree.parse(<some XML with XSLT PI>)
| > |   root = tree.getroot()
| > |   pi = root.getprevious()
| > |   ... read 'href' value from pi.text if 'type' is XSL ...
| > +   ... resolve 'href' through resolvers
| > |   transform = etree.XSLT(etree.parse(href))
| > |   transform(tree)
| 
| No, that's already done by parse(), see resolvers.txt.

Oh, sorry. Didn't spot that.

| > In fact, we should probably look at the source for
| > xsltLoadStylesheetPI to see if it does anything more than what you're
| > proposing.
| > 
| > | * are there other cases where PIs can or should be special cased?
| > 
| > For stylesheets specifically, I've saw an example where the stylesheet
| > href was a fragment identifier ('#stylesheet') and the stylesheet was
| > then *in the same XML* instead of a separate file.
| 
| That could easily be handled via Python resolvers.
| 
| So, given the fact that the libxml2 API can't actually resolve URLs through
| Python resolvers, we should consider not using it and instead making the way I
| sketched above a bit smoother.

Sure, why not. Martijn, what's your opinion?

-- 
Sidnei da Silva
Enfold Systems                http://enfoldsystems.com
Fax +1 832 201 8856     Office +1 713 942 2377 Ext 214

From Holger.Joukl at LBBW.de  Wed Sep 20 10:47:45 2006
From: Holger.Joukl at LBBW.de (Holger Joukl)
Date: Wed, 20 Sep 2006 10:47:45 +0200
Subject: [lxml-dev] [objectify] LongElement pytype registration misses
	type_check
In-Reply-To: <450ACD6B.90105@gkec.informatik.tu-darmstadt.de>
Message-ID: <OF408A497B.19D9524D-ONC12571EF.002FCEC4-C12571EF.003077FE@LBBW.de>

Hi,
the type registration of LongElement in _registerPyTypes
in objectify.pyx (line 833 in my not totally up-to-date
1.1 checkout)
should be changed from

    pytype = PyType('long', None, LongElement)
to
    pytype = PyType('long', long, LongElement)

to activate its actual usefulness.

Funny I never noticed that before.

Cheers, Holger

Der Inhalt dieser E-Mail ist vertraulich. Falls Sie nicht der angegebene
Empf?nger sind oder falls diese E-Mail irrt?mlich an Sie adressiert wurde,
verst?ndigen Sie bitte den Absender sofort und l?schen Sie die E-Mail
sodann. Das unerlaubte Kopieren sowie die unbefugte ?bermittlung sind nicht
gestattet. Die Sicherheit von ?bermittlungen per E-Mail kann nicht
garantiert werden. Falls Sie eine Best?tigung w?nschen, fordern Sie bitte
den Inhalt der E-Mail als Hardcopy an.

The contents of this  e-mail are confidential. If you are not the named
addressee or if this transmission has been addressed to you in error,
please notify the sender immediately and then delete this e-mail.  Any
unauthorized copying and transmission is forbidden. E-Mail transmission
cannot be guaranteed to be secure. If verification is required, please
request a hard copy version.



From faassen at infrae.com  Wed Sep 20 11:35:00 2006
From: faassen at infrae.com (Martijn Faassen)
Date: Wed, 20 Sep 2006 11:35:00 +0200
Subject: [lxml-dev] Stylesheet Processing Instruction
In-Reply-To: <45102FCC.3040703@gkec.informatik.tu-darmstadt.de>
References: <20060918152733.GC3853@cotia>
	<20060918155645.GD3853@cotia>	<450ECA97.6050507@infrae.com>
	<20060919003021.GQ3853@cotia>	<20060919020239.GR3853@cotia>	<450F90CA.6060902@gkec.informatik.tu-darmstadt.de>	<450FC62B.2070905@infrae.com>
	<20060919132042.GU3853@cotia>	<450FF3B3.1060004@infrae.com>	<4510174C.3010209@gkec.informatik.tu-darmstadt.de>	<20060919162241.GZ3853@cotia>
	<45102FCC.3040703@gkec.informatik.tu-darmstadt.de>
Message-ID: <45110B44.50105@infrae.com>

Stefan Behnel wrote:
[snip]
>> For stylesheets specifically, I've saw an example where the stylesheet
>> href was a fragment identifier ('#stylesheet') and the stylesheet was
>> then *in the same XML* instead of a separate file.
> 
> That could easily be handled via Python resolvers.
> 
> So, given the fact that the libxml2 API can't actually resolve URLs through
> Python resolvers, we should consider not using it and instead making the way I
> sketched above a bit smoother.

I'm all for going through Python resolvers. It makes it much easier to 
be sure we're secure as well, as in that it isn't plucking off URLs from 
the web when we don't want it to. I realize though that there are 
libxml2 options to turn this off and on. What's the current situation of 
our usage of these anyway?

Regards,

Martijn

From Holger.Joukl at LBBW.de  Wed Sep 20 15:37:21 2006
From: Holger.Joukl at LBBW.de (Holger Joukl)
Date: Wed, 20 Sep 2006 15:37:21 +0200
Subject: [lxml-dev] [objectify] doc patch
In-Reply-To: <450ACD6B.90105@gkec.informatik.tu-darmstadt.de>
Message-ID: <OF08231BBB.E49EC61E-ONC12571EF.00496F45-C12571EF.004AFBA9@LBBW.de>


Hi,
find attached a little doc patch proposal for objectify.txt, laying out
the data type determination in (even greater) detail, plus
minor additions for the additional data class workings.

I also suggest changing doc/mkhtml.py:
36,37c36,37
<     command = ('%s --stylesheet=%s --link-stylesheet %s > %s' %
<                (script, stylesheet_url, source_path, dest_path))
---
>     command = ('%s %s --stylesheet=%s --link-stylesheet %s > %s' %
>                (sys.executable, script, stylesheet_url, source_path,
dest_path))

which allows for running things like
PYTHON=/apps/pydev/gcc/3.4.4/bin/python2.4 make -e  html
with the non-standard interpreter path actually getting picked up for
rest2html invocation.

Holger


(See attached file: objectify.txt.diff.txt)

Der Inhalt dieser E-Mail ist vertraulich. Falls Sie nicht der angegebene
Empf?nger sind oder falls diese E-Mail irrt?mlich an Sie adressiert wurde,
verst?ndigen Sie bitte den Absender sofort und l?schen Sie die E-Mail
sodann. Das unerlaubte Kopieren sowie die unbefugte ?bermittlung sind nicht
gestattet. Die Sicherheit von ?bermittlungen per E-Mail kann nicht
garantiert werden. Falls Sie eine Best?tigung w?nschen, fordern Sie bitte
den Inhalt der E-Mail als Hardcopy an.

The contents of this  e-mail are confidential. If you are not the named
addressee or if this transmission has been addressed to you in error,
please notify the sender immediately and then delete this e-mail.  Any
unauthorized copying and transmission is forbidden. E-Mail transmission
cannot be guaranteed to be secure. If verification is required, please
request a hard copy version.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: objectify.txt.diff.txt
Type: application/octet-stream
Size: 6357 bytes
Desc: not available
Url : http://codespeak.net/pipermail/lxml-dev/attachments/20060920/a6fe83c8/attachment.obj 

From behnel_ml at gkec.informatik.tu-darmstadt.de  Wed Sep 20 16:51:20 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Wed, 20 Sep 2006 16:51:20 +0200
Subject: [lxml-dev] [objectify] LongElement pytype registration misses
 type_check
In-Reply-To: <OF408A497B.19D9524D-ONC12571EF.002FCEC4-C12571EF.003077FE@LBBW.de>
References: <OF408A497B.19D9524D-ONC12571EF.002FCEC4-C12571EF.003077FE@LBBW.de>
Message-ID: <45115568.60600@gkec.informatik.tu-darmstadt.de>



Holger Joukl wrote:
> the type registration of LongElement in _registerPyTypes
> in objectify.pyx (line 833 in my not totally up-to-date
> 1.1 checkout)
> should be changed from
> 
>     pytype = PyType('long', None, LongElement)
> to
>     pytype = PyType('long', long, LongElement)
> 
> to activate its actual usefulness.
> 
> Funny I never noticed that before.

You didn't notice it because it is not used. As Python's int() accepts both
ints and longs, there is no use in parsing values with both. It is only used
for resolving the 'long' pytype hint, not for parsing longs. That's why the
type check function is None.

Stefan


From Holger.Joukl at LBBW.de  Wed Sep 20 17:11:07 2006
From: Holger.Joukl at LBBW.de (Holger Joukl)
Date: Wed, 20 Sep 2006 17:11:07 +0200
Subject: [lxml-dev] [objectify] LongElement pytype registration misses
	type_check
In-Reply-To: <45115568.60600@gkec.informatik.tu-darmstadt.de>
Message-ID: <OF89B207D9.201A13B5-ONC12571EF.005306F3-C12571EF.00539109@LBBW.de>

Hi Stefan,

lxml-dev-bounces at codespeak.net schrieb am 20.09.2006 16:51:20:

> Holger Joukl wrote:
> > the type registration of LongElement in _registerPyTypes
> > in objectify.pyx (line 833 in my not totally up-to-date
> > 1.1 checkout)
> > should be changed from
> >
> >     pytype = PyType('long', None, LongElement)
> > to
> >     pytype = PyType('long', long, LongElement)
> >
> > to activate its actual usefulness.
> >
> > Funny I never noticed that before.
>
> You didn't notice it because it is not used. As Python's int() accepts
both
> ints and longs, there is no use in parsing values with both. It is only
used
> for resolving the 'long' pytype hint, not for parsing longs. That's why
the
> type check function is None.

Yeah right. But if it was changed you'd be able to do
>>> root = objectify.Element('root')
>>> root.myLong = '2L'
>>> print objectify.dump(root)
root = None [ObjectifiedElement]
    myLong = 2L [LongElement]
>>>

Still, you could of course achieve such a result by using DataElement.

Holger

Der Inhalt dieser E-Mail ist vertraulich. Falls Sie nicht der angegebene
Empf?nger sind oder falls diese E-Mail irrt?mlich an Sie adressiert wurde,
verst?ndigen Sie bitte den Absender sofort und l?schen Sie die E-Mail
sodann. Das unerlaubte Kopieren sowie die unbefugte ?bermittlung sind nicht
gestattet. Die Sicherheit von ?bermittlungen per E-Mail kann nicht
garantiert werden. Falls Sie eine Best?tigung w?nschen, fordern Sie bitte
den Inhalt der E-Mail als Hardcopy an.

The contents of this  e-mail are confidential. If you are not the named
addressee or if this transmission has been addressed to you in error,
please notify the sender immediately and then delete this e-mail.  Any
unauthorized copying and transmission is forbidden. E-Mail transmission
cannot be guaranteed to be secure. If verification is required, please
request a hard copy version.



From behnel_ml at gkec.informatik.tu-darmstadt.de  Wed Sep 20 17:29:34 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Wed, 20 Sep 2006 17:29:34 +0200
Subject: [lxml-dev] [objectify] LongElement pytype registration misses
 type_check
In-Reply-To: <OF89B207D9.201A13B5-ONC12571EF.005306F3-C12571EF.00539109@LBBW.de>
References: <OF89B207D9.201A13B5-ONC12571EF.005306F3-C12571EF.00539109@LBBW.de>
Message-ID: <45115E5E.7050203@gkec.informatik.tu-darmstadt.de>

Hi Holger,

Holger Joukl wrote:
>>>> root = objectify.Element('root')
>>>> root.myLong = '2L'
>>>> print objectify.dump(root)
> root = None [ObjectifiedElement]
>     myLong = 2L [LongElement]

Right, int() can't handle that. So it makes sense to allow long().

Stefan


From behnel_ml at gkec.informatik.tu-darmstadt.de  Wed Sep 20 17:46:39 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Wed, 20 Sep 2006 17:46:39 +0200
Subject: [lxml-dev] [objectify] doc patch
In-Reply-To: <OF08231BBB.E49EC61E-ONC12571EF.00496F45-C12571EF.004AFBA9@LBBW.de>
References: <OF08231BBB.E49EC61E-ONC12571EF.00496F45-C12571EF.004AFBA9@LBBW.de>
Message-ID: <4511625F.1020202@gkec.informatik.tu-darmstadt.de>

Hi Holger,

Holger Joukl wrote:
> find attached a little doc patch proposal for objectify.txt, laying out
> the data type determination in (even greater) detail, plus
> minor additions for the additional data class workings.

I had to change a few passages, but mainly applied it as you wrote it. Please
look through objectify.txt in the trunk (or 1.1 branch) to see if it fits your
intention.


> I also suggest changing doc/mkhtml.py:
> 36,37c36,37
> <     command = ('%s --stylesheet=%s --link-stylesheet %s > %s' %
> <                (script, stylesheet_url, source_path, dest_path))
> ---
>>     command = ('%s %s --stylesheet=%s --link-stylesheet %s > %s' %
>>                (sys.executable, script, stylesheet_url, source_path,
> dest_path))

I applied that, too.

Thanks for contributing.

Stefan

From behnel_ml at gkec.informatik.tu-darmstadt.de  Wed Sep 20 22:27:01 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Wed, 20 Sep 2006 22:27:01 +0200
Subject: [lxml-dev] Stylesheet Processing Instruction
In-Reply-To: <4510174C.3010209@gkec.informatik.tu-darmstadt.de>
References: <20060917021308.GK8537@cotia>	<450E1C1C.1070201@gkec.informatik.tu-darmstadt.de>	<20060918152733.GC3853@cotia>	<20060918155645.GD3853@cotia>	<450ECA97.6050507@infrae.com>	<20060919003021.GQ3853@cotia>	<20060919020239.GR3853@cotia>	<450F90CA.6060902@gkec.informatik.tu-darmstadt.de>	<450FC62B.2070905@infrae.com>	<20060919132042.GU3853@cotia>
	<450FF3B3.1060004@infrae.com>
	<4510174C.3010209@gkec.informatik.tu-darmstadt.de>
Message-ID: <4511A415.9080705@gkec.informatik.tu-darmstadt.de>

Hi,

so, back to this proposal.

Stefan Behnel wrote:
>   tree = etree.parse(<some XML with XSLT PI>)
>   root = tree.getroot()
>   pi = root.getprevious()
>   ... read 'href' value from pi.text if 'type' is XSL ...
>   transform = etree.XSLT(etree.parse(href))
>   transform(tree)

What should this become? I think the beginning is ok so far:

   tree = etree.parse(<some XML with XSLT PI>)
   root = tree.getroot()
   pi = root.getprevious()

Maybe we should think about better PI support here. Currently, there is no way
to add a PI before the root note, for example. So there should be a better way
for handling them in general.

Then, if we change PI.attrib/.get/.set to work on the PI text (not sure how to
do this right), we could end up with this:

   href = pi.get("href")
   type = pi.get("type")
   if type == "text/xsl":
       ...

That's not too bad, I'd say. And then the rest becomes:

   transform = etree.XSLT(etree.parse(href))
   transform(tree)

which supports Python resolvers and access control as usual. The case of an
inner-document reference could be handled as follows:

   href = pi.get("href")
   if href[:1] == '#':
       xslt_root = root.xpath("//xsl:stylesheet[@xml:id = $href]",
                              {"xsl":"..."}, href=href)
       transform = etree.XSLT(xslt_root)

or maybe even with XMLDTDID() and the like.

We could also think about a custom element class for the PI, such as an
XSLProcessingInstruction. The infrastructure for this is mostly in place
already. That would let the usage look like this:

    xslt_pi = root.getprevious() # "xsl-transform" PI
    xslt_doc = xslt_pi.parseXSL(parser=...)
    transform = etree.XSLT(xslt_doc)

Same support for parsers, resolvers and access control, but a much simpler API
that would mainly hide the above code.

Any comments?

Stefan



From behnel_ml at gkec.informatik.tu-darmstadt.de  Wed Sep 20 21:58:13 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Wed, 20 Sep 2006 21:58:13 +0200
Subject: [lxml-dev] Stylesheet Processing Instruction
In-Reply-To: <45110B44.50105@infrae.com>
References: <20060918152733.GC3853@cotia>
	<20060918155645.GD3853@cotia>	<450ECA97.6050507@infrae.com>
	<20060919003021.GQ3853@cotia>	<20060919020239.GR3853@cotia>	<450F90CA.6060902@gkec.informatik.tu-darmstadt.de>	<450FC62B.2070905@infrae.com>
	<20060919132042.GU3853@cotia>	<450FF3B3.1060004@infrae.com>	<4510174C.3010209@gkec.informatik.tu-darmstadt.de>	<20060919162241.GZ3853@cotia>
	<45102FCC.3040703@gkec.informatik.tu-darmstadt.de>
	<45110B44.50105@infrae.com>
Message-ID: <45119D55.70406@gkec.informatik.tu-darmstadt.de>

Hi Martijn,

Martijn Faassen wrote:
> Stefan Behnel wrote:
> [snip]
>>> For stylesheets specifically, I've saw an example where the stylesheet
>>> href was a fragment identifier ('#stylesheet') and the stylesheet was
>>> then *in the same XML* instead of a separate file.
>>
>> That could easily be handled via Python resolvers.

Actually, libxslt has special code for this in its xsltLoadStylesheetPI
function, so that would be the easiest way to support this. However, the
function also modifies the document in-place, so we'd have to deep-copy the
document before the call in order to keep the stylesheet portion from being
modified. Not the most beautiful thing ever...


>> So, given the fact that the libxml2 API can't actually resolve URLs
>> through Python resolvers,

I double checked this now and it really seems to be the case that libxslt uses
the normal resolver callback when loading the PI stylesheet, but we are left
without context in this case, so it's hard to determine which set of Python
resolvers should actually be used.

Normally, when we load documents for XSLT, we have always an XSLT object
around that sets up the required context. The libxslt function for PI
resolving only passes a URL, so there's no place to keep a context.


>> we should consider not using it and instead making
>> the way I sketched above a bit smoother.
> 
> I'm all for going through Python resolvers. It makes it much easier to
> be sure we're secure as well, as in that it isn't plucking off URLs from
> the web when we don't want it to.

libxslt respects the default security prefs, so we could set up the access
rules at a per-thread level. Not very satisfactory, IMHO, and the same problem
as with the resolvers above. It would be too bad if we ended up calling a
method (or utility function) on an ElementTree object that relied on global
settings for security settings and URL resolving. We can do much better with
an XSLT object here.


> I realize though that there are
> libxml2 options to turn this off and on. What's the current situation of
> our usage of these anyway?

XSLT accepts an XSLTAccessControl object in the constructor that holds the
security settings: file access read/write, directory creation and network
access. I think that's a nice and simple API. I'll also add the respective
keyword argument to ET.xslt(), now that I'm at it.

So, I feel pretty much convinced that we should side-step the libxslt function
and brew our own replacement.

Stefan


From behnel_ml at gkec.informatik.tu-darmstadt.de  Thu Sep 21 10:49:19 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Thu, 21 Sep 2006 10:49:19 +0200
Subject: [lxml-dev] lxml 1.1.1 released
Message-ID: <4512520F.5000506@gkec.informatik.tu-darmstadt.de>

Hi all,

lxml 1.1.1 is on cheeseshop. It's mainly a bugfix release for the 1.1 series.
Changelog follows:

Features added

    * XSLT profiling support (profile_run keyword)
    * countchildren() method on objectify.ObjectifiedElement
    * Support custom elements for tree nodes in lxml.objectify

Bugs fixed

    * Error messages from XSLT did not reach XSLT.error_log
    * Factories objectify.Element() and objectify.DataElement() were missing
      attrib and nsmap keyword arguments
    * Changing the default parser in lxml.objectify did not update the
      factories Element() and DataElement()
    * Let lxml.objectify.Element() always generate tree elements (not data
      elements)
    * Build under Windows failed ('0' bug in patched Pyrex version)

I uploaded X86-64 eggs for both Python 2.4 and 2.5, and I'd be very happy to
see more contributions by others. Now is the right time to bring 1.1 to the
masses. :)

Have fun,
Stefan

From faassen at infrae.com  Thu Sep 21 10:52:06 2006
From: faassen at infrae.com (Martijn Faassen)
Date: Thu, 21 Sep 2006 10:52:06 +0200
Subject: [lxml-dev] lxml 1.1.1 released
In-Reply-To: <4512520F.5000506@gkec.informatik.tu-darmstadt.de>
References: <4512520F.5000506@gkec.informatik.tu-darmstadt.de>
Message-ID: <451252B6.2030400@infrae.com>

Stefan Behnel wrote:
[snip]
> I uploaded X86-64 eggs for both Python 2.4 and 2.5, and I'd be very happy to
> see more contributions by others. Now is the right time to bring 1.1 to the
> masses. :)

I went looking for the tarball, but noticed the website wasn't updated. 
I can find the tarball in the cheeseshop but just to let you know.

Regards,

Martijn

From faassen at infrae.com  Thu Sep 21 10:56:14 2006
From: faassen at infrae.com (Martijn Faassen)
Date: Thu, 21 Sep 2006 10:56:14 +0200
Subject: [lxml-dev] lxml 1.1.1 released
In-Reply-To: <4512520F.5000506@gkec.informatik.tu-darmstadt.de>
References: <4512520F.5000506@gkec.informatik.tu-darmstadt.de>
Message-ID: <451253AE.9010901@infrae.com>

Stefan Behnel wrote:
[snip]
> I uploaded X86-64 eggs for both Python 2.4 and 2.5, and I'd be very happy to
> see more contributions by others. Now is the right time to bring 1.1 to the
> masses. :)

linux-i686 eggs for Python 2.3 and Python 2.4 uploaded.

Regards,

Martijn

From sidnei at enfoldsystems.com  Thu Sep 21 15:43:49 2006
From: sidnei at enfoldsystems.com (Sidnei da Silva)
Date: Thu, 21 Sep 2006 10:43:49 -0300
Subject: [lxml-dev] Stylesheet Processing Instruction
In-Reply-To: <4511A415.9080705@gkec.informatik.tu-darmstadt.de>
References: <20060918155645.GD3853@cotia> <450ECA97.6050507@infrae.com>
	<20060919003021.GQ3853@cotia> <20060919020239.GR3853@cotia>
	<450F90CA.6060902@gkec.informatik.tu-darmstadt.de>
	<450FC62B.2070905@infrae.com> <20060919132042.GU3853@cotia>
	<450FF3B3.1060004@infrae.com>
	<4510174C.3010209@gkec.informatik.tu-darmstadt.de>
	<4511A415.9080705@gkec.informatik.tu-darmstadt.de>
Message-ID: <20060921134349.GR3853@cotia>

On Wed, Sep 20, 2006 at 10:27:01PM +0200, Stefan Behnel wrote:
| What should this become? I think the beginning is ok so far:
| 
|    tree = etree.parse(<some XML with XSLT PI>)
|    root = tree.getroot()
|    pi = root.getprevious()

One problem here: there might be more than one PI before the root node
I believe. So you would have to look at all of them.

| Maybe we should think about better PI support here. Currently, there is no way
| to add a PI before the root note, for example. So there should be a better way
| for handling them in general.

I agree with that.

| Then, if we change PI.attrib/.get/.set to work on the PI text (not sure how to
| do this right), we could end up with this:
| 
|    href = pi.get("href")
|    type = pi.get("type")
|    if type == "text/xsl":
|        ...
| 
| That's not too bad, I'd say. And then the rest becomes:
| 
|    transform = etree.XSLT(etree.parse(href))
|    transform(tree)
| 
| which supports Python resolvers and access control as usual. The case of an
| inner-document reference could be handled as follows:
| 
|    href = pi.get("href")
|    if href[:1] == '#':
|        xslt_root = root.xpath("//xsl:stylesheet[@xml:id = $href]",
|                               {"xsl":"..."}, href=href)
|        transform = etree.XSLT(xslt_root)
| 
| or maybe even with XMLDTDID() and the like.

Indeed. That's easier than I thought.

| We could also think about a custom element class for the PI, such as an
| XSLProcessingInstruction. The infrastructure for this is mostly in place
| already. That would let the usage look like this:
| 
|     xslt_pi = root.getprevious() # "xsl-transform" PI
|     xslt_doc = xslt_pi.parseXSL(parser=...)
|     transform = etree.XSLT(xslt_doc)
| 
| Same support for parsers, resolvers and access control, but a much simpler API
| that would mainly hide the above code.

That would be perfect. The only issue is possibly multiple PI's before
the root node and how to find the ones that are xsl-transform.

-- 
Sidnei da Silva
Enfold Systems                http://enfoldsystems.com
Fax +1 832 201 8856     Office +1 713 942 2377 Ext 214

From Holger.Joukl at LBBW.de  Thu Sep 21 16:54:45 2006
From: Holger.Joukl at LBBW.de (Holger Joukl)
Date: Thu, 21 Sep 2006 16:54:45 +0200
Subject: [lxml-dev] [objectify] missing __rmul__ for StringElement?
In-Reply-To: <4512520F.5000506@gkec.informatik.tu-darmstadt.de>
Message-ID: <OF9C98D80F.907C6CA0-ONC12571F0.00502620-C12571F0.00521155@LBBW.de>

Hi,
>>> root = objectify.Element('root')
>>> root.s = "words don't come easy"
>>> root.s * 2
"words don't come easywords don't come easy"
>>> 2 * root.s
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "objectify.pyx", line 639, in objectify.StringElement.__mul__
TypeError: invalid types for * operator

Whereas:
>>> root.s + " to me "
"words don't come easy to me "
>>> " to me " + root.s
" to me words don't come easy"
>>>

Seems like StringElement should get an __rmul__ method, too.
Though I don't understand why the __mul__ method gets invoked here
in the first place, shouldn't that be int.__mul__? Or is this
the last resort, after wanting to use int.__mul__, not finding
an __int__ conversion and no __rmul__?

What about NumberElements? Need they too get all the __r<something>__
stuff?
Or rather not because for them coercion kicks in?

Holger

Der Inhalt dieser E-Mail ist vertraulich. Falls Sie nicht der angegebene
Empf?nger sind oder falls diese E-Mail irrt?mlich an Sie adressiert wurde,
verst?ndigen Sie bitte den Absender sofort und l?schen Sie die E-Mail
sodann. Das unerlaubte Kopieren sowie die unbefugte ?bermittlung sind nicht
gestattet. Die Sicherheit von ?bermittlungen per E-Mail kann nicht
garantiert werden. Falls Sie eine Best?tigung w?nschen, fordern Sie bitte
den Inhalt der E-Mail als Hardcopy an.

The contents of this  e-mail are confidential. If you are not the named
addressee or if this transmission has been addressed to you in error,
please notify the sender immediately and then delete this e-mail.  Any
unauthorized copying and transmission is forbidden. E-Mail transmission
cannot be guaranteed to be secure. If verification is required, please
request a hard copy version.



From ashish.kulkarni at kalyptorisk.com  Fri Sep 22 07:35:29 2006
From: ashish.kulkarni at kalyptorisk.com (Ashish Kulkarni)
Date: Fri, 22 Sep 2006 11:05:29 +0530
Subject: [lxml-dev] lxml 1.1.1 win32 builds for 2.4 and 2.5
Message-ID: <2AB7346A3227A74BB97F9A0D79E3E65A03EFE1@mailserver.kalyptorisk.com>

Hello, 

The builds are up at the usual location:

http://puggy.symonds.net/~ashish/downloads/

Please note that the requirement for linking in wsock32.lib is still present in both 2.4 and 2.5. It should at least be mentioned as a possible requirement in the documentation.

Regards,
ashish

From behnel_ml at gkec.informatik.tu-darmstadt.de  Fri Sep 22 18:48:39 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Fri, 22 Sep 2006 18:48:39 +0200
Subject: [lxml-dev] Stylesheet Processing Instruction
In-Reply-To: <20060921134349.GR3853@cotia>
References: <20060918155645.GD3853@cotia> <450ECA97.6050507@infrae.com>
	<20060919003021.GQ3853@cotia> <20060919020239.GR3853@cotia>
	<450F90CA.6060902@gkec.informatik.tu-darmstadt.de>
	<450FC62B.2070905@infrae.com> <20060919132042.GU3853@cotia>
	<450FF3B3.1060004@infrae.com>
	<4510174C.3010209@gkec.informatik.tu-darmstadt.de>
	<4511A415.9080705@gkec.informatik.tu-darmstadt.de>
	<20060921134349.GR3853@cotia>
Message-ID: <451413E7.5010205@gkec.informatik.tu-darmstadt.de>

Hi Sidnei,

Sidnei da Silva wrote:
> On Wed, Sep 20, 2006 at 10:27:01PM +0200, Stefan Behnel wrote:
> | What should this become? I think the beginning is ok so far:
> | 
> |    tree = etree.parse(<some XML with XSLT PI>)
> |    root = tree.getroot()
> |    pi = root.getprevious()
> 
> One problem here: there might be more than one PI before the root node
> I believe. So you would have to look at all of them.

Sure, but you'd have to do that anyway. Just use a loop.


> | Maybe we should think about better PI support here. Currently, there is no way
> | to add a PI before the root note, for example. So there should be a better way
> | for handling them in general.
> 
> I agree with that.

Ok, but that's independent of any special support for XSLT-PIs. So there are
three topics here:

* better support for handling PIs in general
* support for parsing attribute-like character sequences in PIs
* special support the "xml-stylesheet" PI


> | We could also think about a custom element class for the PI, such as an
> | XSLProcessingInstruction. The infrastructure for this is mostly in place
> | already. That would let the usage look like this:
> | 
> |     xslt_pi = root.getprevious() # "xsl-transform" PI
> |     xslt_doc = xslt_pi.parseXSL(parser=...)
> |     transform = etree.XSLT(xslt_doc)
> | 
> | Same support for parsers, resolvers and access control, but a much simpler API
> | that would mainly hide the above code.
> 
> That would be perfect. The only issue is possibly multiple PI's before
> the root node and how to find the ones that are xsl-transform.

hasattr(pi, 'parseXSL')

would be a good start, I guess.

I committed an implementation of the above to the SVN trunk, please check it
out to see if it fits your expectations.

Stefan

From behnel_ml at gkec.informatik.tu-darmstadt.de  Fri Sep 22 18:55:03 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Fri, 22 Sep 2006 18:55:03 +0200
Subject: [lxml-dev] lxml 1.1.1 win32 builds for 2.4 and 2.5
In-Reply-To: <2AB7346A3227A74BB97F9A0D79E3E65A03EFE1@mailserver.kalyptorisk.com>
References: <2AB7346A3227A74BB97F9A0D79E3E65A03EFE1@mailserver.kalyptorisk.com>
Message-ID: <45141567.3070802@gkec.informatik.tu-darmstadt.de>

Hi Ashish,

Ashish Kulkarni wrote:
> The builds are up at the usual location

Thanks a lot!

> Please note that the requirement for linking in wsock32.lib is still
> present in both 2.4 and 2.5. It should at least be mentioned as a possible
> requirement in the documentation.

Hmmm, ok, I guess I should mention that on the web page.

Thanks again,
Stefan

From sidnei at awkly.org  Fri Sep 22 18:58:10 2006
From: sidnei at awkly.org (Sidnei da Silva)
Date: Fri, 22 Sep 2006 13:58:10 -0300
Subject: [lxml-dev] Testing lxml on Windows [was: Re: lxml 1.1.1 win32
	builds for 2.4 and 2.5]
In-Reply-To: <45141567.3070802@gkec.informatik.tu-darmstadt.de>
References: <2AB7346A3227A74BB97F9A0D79E3E65A03EFE1@mailserver.kalyptorisk.com>
	<45141567.3070802@gkec.informatik.tu-darmstadt.de>
Message-ID: <20060922165810.GA6687@cotia>

Since this thread is about Windows builds, I thought it would be nice
to mention that I'm in the process of setting up a build slave on
Windows 2003 to test lxml with Python 2.5 and trunk.

-- 
Sidnei da Silva
Enfold Systems                http://enfoldsystems.com
Fax +1 832 201 8856     Office +1 713 942 2377 Ext 214

From behnel_ml at gkec.informatik.tu-darmstadt.de  Fri Sep 22 19:17:50 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Fri, 22 Sep 2006 19:17:50 +0200
Subject: [lxml-dev] Testing lxml on Windows
In-Reply-To: <20060922165810.GA6687@cotia>
References: <2AB7346A3227A74BB97F9A0D79E3E65A03EFE1@mailserver.kalyptorisk.com>
	<45141567.3070802@gkec.informatik.tu-darmstadt.de>
	<20060922165810.GA6687@cotia>
Message-ID: <45141ABE.8030301@gkec.informatik.tu-darmstadt.de>

Hi Sidnei,

Sidnei da Silva wrote:
> Since this thread is about Windows builds, I thought it would be nice
> to mention that I'm in the process of setting up a build slave on
> Windows 2003 to test lxml with Python 2.5 and trunk.

Way cool! How is the build triggered? Note that you can monitor check-ins via
a mailing list:

http://codespeak.net/mailman/listinfo/lxml-checkins

Is the build result publicly available?

Stefan


From sidnei at awkly.org  Fri Sep 22 19:50:11 2006
From: sidnei at awkly.org (Sidnei da Silva)
Date: Fri, 22 Sep 2006 14:50:11 -0300
Subject: [lxml-dev] Testing lxml on Windows
In-Reply-To: <45141ABE.8030301@gkec.informatik.tu-darmstadt.de>
References: <2AB7346A3227A74BB97F9A0D79E3E65A03EFE1@mailserver.kalyptorisk.com>
	<45141567.3070802@gkec.informatik.tu-darmstadt.de>
	<20060922165810.GA6687@cotia>
	<45141ABE.8030301@gkec.informatik.tu-darmstadt.de>
Message-ID: <20060922175011.GA4121@cotia>

On Fri, Sep 22, 2006 at 07:17:50PM +0200, Stefan Behnel wrote:
| Way cool! How is the build triggered?

Anytime a checkin is made to python2.5 or trunk.

| Note that you can monitor check-ins via
| a mailing list:
| 
| http://codespeak.net/mailman/listinfo/lxml-checkins
| 
| Is the build result publicly available?

You mean the built binary or the build logs? We could possibly email
it somewhere too.

-- 
Sidnei da Silva
Enfold Systems                http://enfoldsystems.com
Fax +1 832 201 8856     Office +1 713 942 2377 Ext 214

From behnel_ml at gkec.informatik.tu-darmstadt.de  Sat Sep 23 08:01:29 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Sat, 23 Sep 2006 08:01:29 +0200
Subject: [lxml-dev] [objectify] missing __rmul__ for StringElement?
In-Reply-To: <OF9C98D80F.907C6CA0-ONC12571F0.00502620-C12571F0.00521155@LBBW.de>
References: <OF9C98D80F.907C6CA0-ONC12571F0.00502620-C12571F0.00521155@LBBW.de>
Message-ID: <4514CDB9.4090203@gkec.informatik.tu-darmstadt.de>

Hi Holger,

Holger Joukl wrote:
> Hi,
>>>> root = objectify.Element('root')
>>>> root.s = "words don't come easy"
>>>> root.s * 2
> "words don't come easywords don't come easy"
>>>> 2 * root.s
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
>   File "objectify.pyx", line 639, in objectify.StringElement.__mul__
> TypeError: invalid types for * operator

True, that was missing. Just added it. Thanks.



> Whereas:
>>>> root.s + " to me "
> "words don't come easy to me "
>>>> " to me " + root.s
> " to me words don't come easy"
> 
> Seems like StringElement should get an __rmul__ method, too.
> Though I don't understand why the __mul__ method gets invoked here
> in the first place, shouldn't that be int.__mul__?

See the Pyrex documentation on this topic, section on "arithmetic methods":

http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/special_methods.html

Stefan

From greg.steffensen at gmail.com  Sun Sep 24 04:46:57 2006
From: greg.steffensen at gmail.com (Greg Steffensen)
Date: Sat, 23 Sep 2006 22:46:57 -0400
Subject: [lxml-dev] xinclude bug?
Message-ID: <438a23610609231946v159e2b59y446520e29f5b6441@mail.gmail.com>

I'm working on a project that will use lxml's xinclude functionality to
insert the contents of python files into an xml document and have noticed a
possible bug.  When you xinclude with the parse attribute set to "text", the
text frequently (though not always) gets loaded into multiple adjacent text
nodes, so that if you access the text attribute of a containing element, you
only get part of the actual text.  You can verify this by calling

element.xpath('text()')

on the container... you get back a list with multiple elements.  Is this how
things are supposed to work?

Also, escaping seems to occur in strings accessed from the "text" attribute
of xincluded content, but not in strings retrieved via xpath, as described
above.  Is there a reliable way to reverse the escaping process, so that the
original contents of the xincluded file can be retrieved?  I assume that
xml.sax.saxutils.unescape() would work, but don't know for sure.

I've pasted some example code below to demonstrate the seemingly broken
"text" attribute use and the different escaping styles.  Thanks,

Greg

doc.xml
-----------------------------
<?xml version="1.0" encoding="UTF-8"?>
<doc xmlns:xi="http://www.w3.org/2001/XInclude">
    <xi:include href="doc.py" parse="text"/>
</doc>


doc.py
---------------------------
#!/usr/bin/python

s1 = '3 < 4'
s2 = "hello;"


test.py
--------------------------
from lxml import etree
tree = lxml.parse('doc.xml')
tree.xinclude ()
root = tree.getroot()
print repr(root.text)
print '----'
print root.xpath('text()')
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20060923/e34dc0c1/attachment.htm 

From sidnei at awkly.org  Sun Sep 24 14:31:16 2006
From: sidnei at awkly.org (Sidnei da Silva)
Date: Sun, 24 Sep 2006 09:31:16 -0300
Subject: [lxml-dev] Testing lxml on Windows [was: Re: lxml 1.1.1 win32
	builds for 2.4 and 2.5]
In-Reply-To: <20060922165810.GA6687@cotia>
References: <2AB7346A3227A74BB97F9A0D79E3E65A03EFE1@mailserver.kalyptorisk.com>
	<45141567.3070802@gkec.informatik.tu-darmstadt.de>
	<20060922165810.GA6687@cotia>
Message-ID: <20060924123116.GB4337@cotia>

On Fri, Sep 22, 2006 at 01:58:10PM -0300, Sidnei da Silva wrote:
| Since this thread is about Windows builds, I thought it would be nice
| to mention that I'm in the process of setting up a build slave on
| Windows 2003 to test lxml with Python 2.5 and trunk.

One question though: 

 - Should I test the latest release or 'trunk'?

Opinions?

-- 
Sidnei da Silva
Enfold Systems                http://enfoldsystems.com
Fax +1 832 201 8856     Office +1 713 942 2377 Ext 214

From behnel_ml at gkec.informatik.tu-darmstadt.de  Sun Sep 24 17:36:23 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Sun, 24 Sep 2006 17:36:23 +0200
Subject: [lxml-dev] Testing lxml on Windows [was: Re: lxml 1.1.1 win32
 builds for 2.4 and 2.5]
In-Reply-To: <20060924123116.GB4337@cotia>
References: <2AB7346A3227A74BB97F9A0D79E3E65A03EFE1@mailserver.kalyptorisk.com>
	<45141567.3070802@gkec.informatik.tu-darmstadt.de>
	<20060922165810.GA6687@cotia> <20060924123116.GB4337@cotia>
Message-ID: <4516A5F7.50304@gkec.informatik.tu-darmstadt.de>

Hi Sidnei,

Sidnei da Silva wrote:
> On Fri, Sep 22, 2006 at 01:58:10PM -0300, Sidnei da Silva wrote:
> | Since this thread is about Windows builds, I thought it would be nice
> | to mention that I'm in the process of setting up a build slave on
> | Windows 2003 to test lxml with Python 2.5 and trunk.
> 
> One question though: 
> 
>  - Should I test the latest release or 'trunk'?

well, any problem with testing both? I mean, you can compile "branch/lxml-1.1"
and "trunk" independently. Or, if you manage to make it commit-mail triggered,
you can see in the mail where the commit was ("lxml/trunk/..." or
"lxml/branch/...") and then trigger only the corresponding build. Don't know
how much work it would be to set that up, though...

Could you give us some more detail how this will work?

Stefan


From behnel_ml at gkec.informatik.tu-darmstadt.de  Sun Sep 24 17:59:15 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Sun, 24 Sep 2006 17:59:15 +0200
Subject: [lxml-dev] xinclude bug?
In-Reply-To: <438a23610609231946v159e2b59y446520e29f5b6441@mail.gmail.com>
References: <438a23610609231946v159e2b59y446520e29f5b6441@mail.gmail.com>
Message-ID: <4516AB53.7000401@gkec.informatik.tu-darmstadt.de>

Hi Greg,

thanks for reporting this.

Greg Steffensen wrote:
> When you xinclude with the parse attribute set
> to "text", the text frequently (though not always) gets loaded into
> multiple adjacent text nodes, so that if you access the text attribute
> of a containing element, you only get part of the actual text.  You can
> verify this by calling
> 
> element.xpath('text()')
> 
> on the container... you get back a list with multiple elements.  Is this
> how things are supposed to work?

That's not quite what your example below shows. You take this document:

> <?xml version="1.0" encoding="UTF-8"?>
> <doc xmlns:xi="http://www.w3.org/2001/XInclude">
>     <xi:include href="doc.py " parse="text"/>
> </doc>

Note the whitespace around the include element. What I get for the XPath call
after running the include is:

['\n    ', '#!/usr/bin/python\n\ns1 = \'3 < 4\'\ns2 = "hello;"\n', '\n']

So the new text was correctly added between the two existing text nodes. Now,
what happens internally is that libxml2 adds special xinclude nodes around the
included part as a kind of marker. So, when we collect text nodes for the
".text" property, we stop at the xinclude nodes and only regard the text
before them. This results in what you see for the text property:

'\n    '

I consider this a bug in lxml. I think we should step over xinclude nodes when
collecting text content.


> Also, escaping seems to occur in strings accessed from the "text"
> attribute of xincluded content, but not in strings retrieved via xpath,

I'm not quite sure what you mean here. Could you give an example? I mean, the
above XPath result has normal text content, not escaped in any way.

Stefan

From greg.steffensen at gmail.com  Mon Sep 25 00:25:18 2006
From: greg.steffensen at gmail.com (Greg Steffensen)
Date: Sun, 24 Sep 2006 18:25:18 -0400
Subject: [lxml-dev] xinclude bug?
In-Reply-To: <438a23610609241524u6ca833cdo71b830dce6483ba1@mail.gmail.com>
References: <438a23610609231946v159e2b59y446520e29f5b6441@mail.gmail.com>
	<4516AB53.7000401@gkec.informatik.tu-darmstadt.de>
	<438a23610609241524u6ca833cdo71b830dce6483ba1@mail.gmail.com>
Message-ID: <438a23610609241525y11a19d29heb8ebdc625f98429@mail.gmail.com>

In retrospect my example was poorly chosen, but fortunately, you got the
gist.  The escaping inconsistency I thought I'd noticed now seems to only
occur when using etree.tostring, which replaces "<" and ">" in text nodes
with their entities.  That seems like correct behavior after all.  Thanks,

Greg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20060924/5c49741e/attachment.htm 

From sidnei at enfoldsystems.com  Mon Sep 25 02:25:43 2006
From: sidnei at enfoldsystems.com (Sidnei da Silva)
Date: Sun, 24 Sep 2006 21:25:43 -0300
Subject: [lxml-dev] Testing lxml on Windows [was: Re: lxml 1.1.1 win32
	builds for 2.4 and 2.5]
In-Reply-To: <4516A5F7.50304@gkec.informatik.tu-darmstadt.de>
References: <2AB7346A3227A74BB97F9A0D79E3E65A03EFE1@mailserver.kalyptorisk.com>
	<45141567.3070802@gkec.informatik.tu-darmstadt.de>
	<20060922165810.GA6687@cotia> <20060924123116.GB4337@cotia>
	<4516A5F7.50304@gkec.informatik.tu-darmstadt.de>
Message-ID: <20060925002543.GE4337@cotia>

On Sun, Sep 24, 2006 at 05:36:23PM +0200, Stefan Behnel wrote:
| well, any problem with testing both?

Other than the time required to set it up, nope.

| I mean, you can compile "branch/lxml-1.1"
| and "trunk" independently.

Actually I was refering to tarball. But if we use the branch instead
then I agree that not much extra work is needed.

Maybe if I can do 'easy_install lxml=1.1 develop' (or whatever is the
right incantation) and get a branch checkout? That would be a ton
easier.

| Or, if you manage to make it commit-mail triggered,
| you can see in the mail where the commit was ("lxml/trunk/..." or
| "lxml/branch/...") and then trigger only the corresponding build. Don't know
| how much work it would be to set that up, though...

I will ask Grig.

| Could you give us some more detail how this will work?

 http://www.python.org/dev/buildbot/community/trunk/
 http://www.python.org/dev/buildbot/community/2.5/

Look for 'x86 Windows 2003'. Right now the second 'test' step is
running an empty test.bat script. I need to fill that with whatever it
takes to do a checkout, build and test for lxml for whatever branches
should be tested with python 2.5 and trunk (2.6). The test runs will
be triggered by python checkins for now, and potentially can be
extended to run on lxml checkins too.

-- 
Sidnei da Silva
Enfold Systems                http://enfoldsystems.com
Fax +1 832 201 8856     Office +1 713 942 2377 Ext 214

From behnel_ml at gkec.informatik.tu-darmstadt.de  Mon Sep 25 08:21:43 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Mon, 25 Sep 2006 08:21:43 +0200
Subject: [lxml-dev] xinclude bug?
In-Reply-To: <4516AB53.7000401@gkec.informatik.tu-darmstadt.de>
References: <438a23610609231946v159e2b59y446520e29f5b6441@mail.gmail.com>
	<4516AB53.7000401@gkec.informatik.tu-darmstadt.de>
Message-ID: <45177577.4040800@gkec.informatik.tu-darmstadt.de>

Hi again,

Stefan Behnel wrote:
>> <?xml version="1.0" encoding="UTF-8"?>
>> <doc xmlns:xi="http://www.w3.org/2001/XInclude">
>>     <xi:include href="doc.py " parse="text"/>
>> </doc>
> 
> Note the whitespace around the include element. What I get for the XPath call
> after running the include is:
> 
> ['\n    ', '#!/usr/bin/python\n\ns1 = \'3 < 4\'\ns2 = "hello;"\n', '\n']
> 
> So the new text was correctly added between the two existing text nodes. Now,
> what happens internally is that libxml2 adds special xinclude nodes around the
> included part as a kind of marker. So, when we collect text nodes for the
> ".text" property, we stop at the xinclude nodes and only regard the text
> before them. This results in what you see for the text property:
> 
> '\n    '
> 
> I consider this a bug in lxml. I think we should step over xinclude nodes when
> collecting text content.

I fixed this. When collecting text nodes, we now step over xinclude nodes and
continue.

Stefan



From sidnei at enfoldsystems.com  Mon Sep 25 22:07:51 2006
From: sidnei at enfoldsystems.com (Sidnei da Silva)
Date: Mon, 25 Sep 2006 17:07:51 -0300
Subject: [lxml-dev] Stylesheet Processing Instruction
In-Reply-To: <451413E7.5010205@gkec.informatik.tu-darmstadt.de>
References: <20060919003021.GQ3853@cotia> <20060919020239.GR3853@cotia>
	<450F90CA.6060902@gkec.informatik.tu-darmstadt.de>
	<450FC62B.2070905@infrae.com> <20060919132042.GU3853@cotia>
	<450FF3B3.1060004@infrae.com>
	<4510174C.3010209@gkec.informatik.tu-darmstadt.de>
	<4511A415.9080705@gkec.informatik.tu-darmstadt.de>
	<20060921134349.GR3853@cotia>
	<451413E7.5010205@gkec.informatik.tu-darmstadt.de>
Message-ID: <20060925200751.GJ4356@cotia>

On Fri, Sep 22, 2006 at 06:48:39PM +0200, Stefan Behnel wrote:
| I committed an implementation of the above to the SVN trunk, please check it
| out to see if it fits your expectations.

Looks like you only wrote tests for the embedded id inclusion. Would
be nice to add a test for filename/url inclusion just to make
sure. Other than that, looks great to me.

-- 
Sidnei da Silva
Enfold Systems                http://enfoldsystems.com
Fax +1 832 201 8856     Office +1 713 942 2377 Ext 214

From sidnei at enfoldsystems.com  Tue Sep 26 00:09:13 2006
From: sidnei at enfoldsystems.com (Sidnei da Silva)
Date: Mon, 25 Sep 2006 19:09:13 -0300
Subject: [lxml-dev] Failure to Compile on Windows
Message-ID: <20060925220913.GL4356@cotia>

I've got pretty far with setting up the dependencies and all but after
that the compilation failed with this error. Anyone have a clue? Looks
like it's missing some .h that 'nano http' depends on.

C:\Arquivos de programas\Microsoft Visual C++ Toolkit 2003\bin\link.exe /DLL /no logo /INCREMENTAL:NO /LIBPATH:c:\Python24\libs /LIBPATH:c:\Python24\PCBuild /EXPORT:initobjectify build\temp.win32-2.4\Release\src/lxml/objectify.obj /OUT:build\lib.win32-2.4\lxml\objectify.pyd /IMPLIB:build\temp.win32-2.4\Release\src/lxml\objectify.lib ..\libxml2-2.6.26.win32\lib\libxml2_a.lib ..\libxslt-1.1.17.win32\lib\libxslt_a.lib ..\libxslt-1.1.17.win32\lib\libexslt_a.lib ..\zlib-1.2.3.win32\lib\zlib.lib ..\iconv-1.9.2.win32\lib\iconv_a.lib
   Creating library build\temp.win32-2.4\Release\src/lxml\objectify.lib and object build\temp.win32-2.4\Release\src/lxml\objectify.exp
libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__WSAGetLastError at 0 referenced in function _socket_errno
libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__WSACleanup at 0 referenced in function _xmlNanoHTTPCleanup
libxml2_a.lib(nanoftp.obj) : error LNK2001: unresolved external symbol __imp__WSACleanup at 0
libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__closesocket at 4 referenced in function _xmlNanoHTTPFreeCtxt
libxml2_a.lib(nanoftp.obj) : error LNK2019: unresolved external symbol __imp__closesocket at 4 referenced in function _xmlNanoFTPNewCtxt
libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__select at 20 referenced in function _xmlNanoHTTPSend
libxml2_a.lib(nanoftp.obj) : error LNK2001: unresolved external symbol __imp__select at 20
libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__send at 16 referenced in function _xmlNanoHTTPSend
libxml2_a.lib(nanoftp.obj) : error LNK2001: unresolved external symbol __imp__send at 16
libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__recv at 16 referenced in function _xmlNanoHTTPRecv
libxml2_a.lib(nanoftp.obj) : error LNK2001: unresolved external symbol __imp__recv at 16
libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__getsockopt at 20 referenced in function _xmlNanoHTTPConnectAttempt
libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol ___WSAFDIsSet at 8 referenced in function _xmlNanoHTTPConnectAttempt
libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__connect at 12 referenced in function _xmlNanoHTTPConnectAttempt
libxml2_a.lib(nanoftp.obj) : error LNK2001: unresolved external symbol __imp__connect at 12
libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__ioctlsocket at 12 referenced in function _xmlNanoHTTPConnectAttempt
libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__socket at 12 referenced in function _xmlNanoHTTPConnectAttempt
libxml2_a.lib(nanoftp.obj) : error LNK2001: unresolved external symbol __imp__socket at 12
libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__htons at 4 referenced in function _xmlNanoHTTPConnectHost
libxml2_a.lib(nanoftp.obj) : error LNK2001: unresolved external symbol __imp__htons at 4
libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__gethostbyname at 4 referenced in function _xmlNanoHTTPConnectHost
libxml2_a.lib(nanoftp.obj) : error LNK2001: unresolved external symbol __imp__gethostbyname at 4
libxml2_a.lib(nanohttp.obj) : error LNK2019: unresolved external symbol __imp__WSAStartup at 8 referenced in function _xmlNanoHTTPInit
libxml2_a.lib(nanoftp.obj) : error LNK2001: unresolved external symbol __imp__WSAStartup at 8
libxml2_a.lib(nanoftp.obj) : error LNK2019: unresolved external symbol __imp__listen at 8 referenced in function _xmlNanoFTPGetConnection
libxml2_a.lib(nanoftp.obj) : error LNK2019: unresolved external symbol __imp__bind at 12 referenced in function _xmlNanoFTPGetConnection
libxml2_a.lib(nanoftp.obj) : error LNK2019: unresolved external symbol __imp__getsockname at 12 referenced in function _xmlNanoFTPGetConnection
build\lib.win32-2.4\lxml\objectify.pyd : fatal error LNK1120: 17 unresolved externals
error: command '"C:\Arquivos de programas\Microsoft Visual C++ Toolkit 2003\bin\link.exe"' failed with exit status 1120


-- 
Sidnei da Silva
Enfold Systems                http://enfoldsystems.com
Fax +1 832 201 8856     Office +1 713 942 2377 Ext 214

From sidnei at enfoldsystems.com  Tue Sep 26 04:56:06 2006
From: sidnei at enfoldsystems.com (Sidnei da Silva)
Date: Mon, 25 Sep 2006 23:56:06 -0300
Subject: [lxml-dev] Failure to Compile on Windows
In-Reply-To: <20060925220913.GL4356@cotia>
References: <20060925220913.GL4356@cotia>
Message-ID: <20060926025606.GP4356@cotia>

On Mon, Sep 25, 2006 at 07:09:13PM -0300, Sidnei da Silva wrote:
| I've got pretty far with setting up the dependencies and all but after
| that the compilation failed with this error. Anyone have a clue? Looks
| like it's missing some .h that 'nano http' depends on.

FWIW, I've got past that. It was missing Ws2_32.lib.

I've managed to build lxml with python trunk, but when running the
tests I've got a fatal error due to a negative refcount to a tuple
(!).

I think I will disable running lxml tests with python trunk, unless
someone wants to digg into this.

-- 
Sidnei da Silva
Enfold Systems                http://enfoldsystems.com
Fax +1 832 201 8856     Office +1 713 942 2377 Ext 214

From sidnei at enfoldsystems.com  Tue Sep 26 05:55:44 2006
From: sidnei at enfoldsystems.com (Sidnei da Silva)
Date: Tue, 26 Sep 2006 00:55:44 -0300
Subject: [lxml-dev] Failure to Compile on Windows
In-Reply-To: <20060926025606.GP4356@cotia>
References: <20060925220913.GL4356@cotia> <20060926025606.GP4356@cotia>
Message-ID: <20060926035544.GQ4356@cotia>

On Mon, Sep 25, 2006 at 11:56:06PM -0300, Sidnei da Silva wrote:
| I've managed to build lxml with python trunk, but when running the
| tests I've got a fatal error due to a negative refcount to a tuple
| (!).
| 
| I think I will disable running lxml tests with python trunk, unless
| someone wants to digg into this.

And, to my surprise, it fails with the same refcount issue with
python2.5-branch.

-- 
Sidnei da Silva
Enfold Systems                http://enfoldsystems.com
Fax +1 832 201 8856     Office +1 713 942 2377 Ext 214

From desai at mcs.anl.gov  Tue Sep 26 16:00:06 2006
From: desai at mcs.anl.gov (Narayan Desai)
Date: Tue, 26 Sep 2006 14:00:06 +0000 (UTC)
Subject: [lxml-dev] strange problem with lxml-1.1.1 and py2.5
Message-ID: <loom.20060926T155853-694@post.gmane.org>

I am running into a strange problem when using python2.5 and
lxml-1.1.1. calling findall causing a non-sensical exception to be
raised. I know that this code is fine; it works perfectly with
python2.4. 

The code in my app calls findall on an Element.

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/Bcfg2/Server/Core.py", line 175, in Service
    self.handles[event.requestID].HandleEvent(event)
  File "/usr/lib/python2.5/site-packages/Bcfg2/Server/Plugin.py", line 301, in
HandleEvent
    DirectoryBacked.HandleEvent(self, event)
  File "/usr/lib/python2.5/site-packages/Bcfg2/Server/Plugin.py", line 119, in
HandleEvent
    self.AddEntry(event.filename)
  File "/usr/lib/python2.5/site-packages/Bcfg2/Server/Plugin.py", line 109, in
AddEntry
    self.entries[name].HandleEvent()
  File "/usr/lib/python2.5/site-packages/Bcfg2/Server/Plugin.py", line 267, in
HandleEvent
    self.pnode = self.__node__(xdata, self.items)
  File "/usr/lib/python2.5/site-packages/Bcfg2/Server/Plugins/Pkgmgr.py", line
22, in __init__
    Bcfg2.Server.Plugin.INode.__init__(self, data, pdict, parent)
  File "/usr/lib/python2.5/site-packages/Bcfg2/Server/Plugin.py", line 218, in
__init__
    self.children.append(mytype(item, idict, self))
  File "/usr/lib/python2.5/site-packages/Bcfg2/Server/Plugins/Pkgmgr.py", line
25, in __init__
    for pkg in data.findall('./Package'):
  File "etree.pyx", line 1164, in etree._Element.findall
  File "/usr/lib/python2.5/site-packages/lxml/_elementpath.py", line 193, in findall
    return _compile(path).findall(element)
  File "/usr/lib/python2.5/site-packages/lxml/_elementpath.py", line 155, in findall
    for node in node:
  File "etree.pyx", line 1446, in etree._ElementIterator.__next__
TypeError: exceptions must be strings, classes, or instances, not type

I have not tried svn, though I could if it would be useful.
thanks..
 -nld


From ltucker at openplans.org  Tue Sep 26 18:09:55 2006
From: ltucker at openplans.org (Luke Tucker)
Date: Tue, 26 Sep 2006 12:09:55 -0400
Subject: [lxml-dev] problems accessing style tag text
Message-ID: <1159286995.24787.10.camel@ltucker>




Hey, 

I was trying to get at the contents of a style tag parsed with 
etree.HTML and it doesn't seem to be accessible as the text
attribute of the element. This appears to be related to the insertion 
of the CDATA marker around the text. Maybe there is something obvious 
that I'm missing. I tried this against a 1.1 release and a recent
checkout of the trunk.  I appreciate any advice you all can offer. 

The following demonstrates what I'm seeing: 

>>> from lxml import etree
>>> text = '<html><head><style>foo</style></head></html>'
>>> html = etree.HTML(text)
>>> print html[0][0]
<Element style at 2b97a66f9140>
>>> print html[0][0].text
None
>>> etree.tostring(html[0][0])
'<style><![CDATA[foo]]></style>'
>>> len(html[0][0])
0

Thanks, 

- Luke 


From behnel_ml at gkec.informatik.tu-darmstadt.de  Tue Sep 26 18:19:00 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Tue, 26 Sep 2006 18:19:00 +0200
Subject: [lxml-dev] strange problem with lxml-1.1.1 and py2.5
In-Reply-To: <loom.20060926T155853-694@post.gmane.org>
References: <loom.20060926T155853-694@post.gmane.org>
Message-ID: <451952F4.40602@gkec.informatik.tu-darmstadt.de>

Hi,

Narayan Desai wrote:
> I am running into a strange problem when using python2.5 and
> lxml-1.1.1. calling findall causing a non-sensical exception to be
> raised. I know that this code is fine; it works perfectly with
> python2.4. 

>   File "/usr/lib/python2.5/site-packages/lxml/_elementpath.py", line 155, in findall
>     for node in node:
>   File "etree.pyx", line 1446, in etree._ElementIterator.__next__
> TypeError: exceptions must be strings, classes, or instances, not type

Quick question: did you compile it yourself or use the egg? If you compiled
it, I assume you used the Pyrex version provided by lxml? (see build.txt)

Stefan


From desai at mcs.anl.gov  Tue Sep 26 19:22:23 2006
From: desai at mcs.anl.gov (Narayan Desai)
Date: Tue, 26 Sep 2006 17:22:23 +0000 (UTC)
Subject: [lxml-dev] strange problem with lxml-1.1.1 and py2.5
References: <loom.20060926T155853-694@post.gmane.org>
	<451952F4.40602@gkec.informatik.tu-darmstadt.de>
Message-ID: <loom.20060926T192012-274@post.gmane.org>

Stefan Behnel <behnel_ml <at> gkec.informatik.tu-darmstadt.de> writes:

> Quick question: did you compile it yourself or use the egg? If you compiled
> it, I assume you used the Pyrex version provided by lxml? (see build.txt)

I built it myself. This is probably the issue. I will followup if using an
up-to-date pyrex does not fix the issue.
Thanks for the quick response
 -nld




From behnel_ml at gkec.informatik.tu-darmstadt.de  Tue Sep 26 22:04:34 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Tue, 26 Sep 2006 22:04:34 +0200
Subject: [lxml-dev] problems accessing style tag text
In-Reply-To: <1159286995.24787.10.camel@ltucker>
References: <1159286995.24787.10.camel@ltucker>
Message-ID: <451987D2.6080303@gkec.informatik.tu-darmstadt.de>

Hi,

thanks for reporting this and for the excellent test case.

Luke Tucker wrote:
> I was trying to get at the contents of a style tag parsed with 
> etree.HTML and it doesn't seem to be accessible as the text
> attribute of the element. This appears to be related to the insertion 
> of the CDATA marker around the text. Maybe there is something obvious 
> that I'm missing. I tried this against a 1.1 release and a recent
> checkout of the trunk.  I appreciate any advice you all can offer. 
> 
> The following demonstrates what I'm seeing: 
> 
>>>> from lxml import etree
>>>> text = '<html><head><style>foo</style></head></html>'
>>>> html = etree.HTML(text)
>>>> print html[0][0].text
> None
>>>> etree.tostring(html[0][0])
> '<style><![CDATA[foo]]></style>'

The thing is that lxml always sets up the parser to convert CDATA sections to
normal text nodes, which simplifies the internal handling of text quite a bit.
However, the HTML parser does not have such a parser option, so CDATA sections
slip through in this case. Especially, as the libxml2 HTML parser generates
them explicitly for script content and CSS.

I consider this a bug that should be fixed.

It looks like we can prevent the CDATA generation by modifying the SAX parser
function table of libxml2 in place (set the cdataBlock function to NULL).
That's how libxml2 handles the XML_PARSE_NOCDATA flag internally and it seems
to work for HTML just fine. I'll commit it to both the trunk and the 1.1 branch.

Stefan



From behnel_ml at gkec.informatik.tu-darmstadt.de  Tue Sep 26 22:26:25 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Tue, 26 Sep 2006 22:26:25 +0200
Subject: [lxml-dev] Testing lxml on Windows [was: Re: lxml 1.1.1 win32
 builds for 2.4 and 2.5]
In-Reply-To: <20060925002543.GE4337@cotia>
References: <2AB7346A3227A74BB97F9A0D79E3E65A03EFE1@mailserver.kalyptorisk.com>
	<45141567.3070802@gkec.informatik.tu-darmstadt.de>
	<20060922165810.GA6687@cotia> <20060924123116.GB4337@cotia>
	<4516A5F7.50304@gkec.informatik.tu-darmstadt.de>
	<20060925002543.GE4337@cotia>
Message-ID: <45198CF1.10004@gkec.informatik.tu-darmstadt.de>

Hi Sidnei,

Sidnei da Silva wrote:
> On Sun, Sep 24, 2006 at 05:36:23PM +0200, Stefan Behnel wrote:
> | well, any problem with testing both?
> 
> Other than the time required to set it up, nope.

Cool, that would allow us to see if anything breaks in the stable series or
the development version. And if the builds were up on the web or in an ftp
directory somewhere, we could simply point windows users there if they want to
test a bug fix and the like.


> | I mean, you can compile "branch/lxml-1.1"
> | and "trunk" independently.
> 
> Actually I was refering to tarball. But if we use the branch instead
> then I agree that not much extra work is needed.

A simple "svn up" should do the trick after having checked out the respective
SVN directory once.


> Maybe if I can do 'easy_install lxml=1.1 develop' (or whatever is the
> right incantation) and get a branch checkout? That would be a ton
> easier.

Sadly, cheeseshop can't handle this. But SVN access should work just fine.


> | Could you give us some more detail how this will work?
> 
>  http://www.python.org/dev/buildbot/community/trunk/
>  http://www.python.org/dev/buildbot/community/2.5/
> 
> Look for 'x86 Windows 2003'. Right now the second 'test' step is
> running an empty test.bat script. I need to fill that with whatever it
> takes to do a checkout, build and test for lxml

That would be

	cd HD\path\to\lxml-checkout
	svn up
	HD\path\to\specific\python setup.py build_ext -i
	HD\path\to\specific\python test.py

If you want, you can also add a

	HD\path\to\specific\python setup.py bdist_egg

at the end to additionally test the egg build.


> The test runs will
> be triggered by python checkins for now, and potentially can be
> extended to run on lxml checkins too.

That should even be enough. As Python checkins tend to happen much more
frequently than lxml checkins, lxml will also be rebuilt just a while after
the last checkin.

Stefan


From faassen at infrae.com  Wed Sep 27 11:11:51 2006
From: faassen at infrae.com (Martijn Faassen)
Date: Wed, 27 Sep 2006 11:11:51 +0200
Subject: [lxml-dev] problems accessing style tag text
In-Reply-To: <451987D2.6080303@gkec.informatik.tu-darmstadt.de>
References: <1159286995.24787.10.camel@ltucker>
	<451987D2.6080303@gkec.informatik.tu-darmstadt.de>
Message-ID: <451A4057.90409@infrae.com>

Stefan Behnel wrote:
[snip]
> I consider this a bug that should be fixed.
> 
> It looks like we can prevent the CDATA generation by modifying the SAX parser
> function table of libxml2 in place (set the cdataBlock function to NULL).
> That's how libxml2 handles the XML_PARSE_NOCDATA flag internally and it seems
> to work for HTML just fine. I'll commit it to both the trunk and the 1.1 branch.

Cool, +1 on preventing CDATA nodes and such. CDATA is evil and should 
not appear in the API. :) Perhaps this also warrants an issue in libxml2 
so that eventually this is fixed upstream for the HTML parser?

Regards,

Martijn

From achimkern at hirschmanngmbh.com  Wed Sep 27 13:01:37 2006
From: achimkern at hirschmanngmbh.com (Achim Kern)
Date: Wed, 27 Sep 2006 13:01:37 +0200
Subject: [lxml-dev] Building Problems
Message-ID: <87fyed4lpq.fsf@hirschmanngmbh.com>

Hello everybody,

during googeling on how to write easier xml datastores with python I
just found our project. Especialy the objectify modules impressed
me. So to test things I wanted to install it. Unfortunatly I can not
use the provided debian package as there is only one for version 1.03
not including the objectify extension. So I downloaded the source of
1.1.1 from codespeak.net extracted it and that's what it did.

# tar -xzf lxml-1.1.1.tgz
# ce lxml-1.1.1
# make clean test. 
python setup.py  build_ext -i
Building lxml version 1.1
running build_ext
python test.py -p -v 
Traceback (most recent call last):
  File "test.py", line 591, in ?
    exitcode = main(sys.argv)
  File "test.py", line 554, in main
    test_cases = get_test_cases(test_files, cfg, tracer=tracer)
  File "test.py", line 254, in get_test_cases
    module = import_module(file, cfg, tracer=tracer)
  File "test.py", line 197, in import_module
    mod = __import__(modname)
  File "/tmp/lxml-1.1/src/lxml/tests/test_objectify.py", line 16, in ?
    from lxml import objectify
ImportError: /tmp/lxml-1.1/src/lxml/objectify.so: undefined symbol: previousElement
make: *** [test_inplace] Error 1

It builds just ok but when it comes to test it crashes. There seams to
be something missing. I have installed libxml2 in version 2.6.26 and
libxslt in version 1.1.17. I investigated the problem a little further
with ldd and it gives me:

# ldd -d src/lxml/objectify.so 
undefined symbol: PyTraceBack_Type      (src/lxml/objectify.so)
undefined symbol: _Py_NoneStruct        (src/lxml/objectify.so)
undefined symbol: PyString_Type (src/lxml/objectify.so)
undefined symbol: previousElement       (src/lxml/objectify.so)
undefined symbol: PyExc_SystemError     (src/lxml/objectify.so)
undefined symbol: PyExc_ValueError      (src/lxml/objectify.so)
undefined symbol: PyExc_TypeError       (src/lxml/objectify.so)
undefined symbol: PyBaseString_Type     (src/lxml/objectify.so)
undefined symbol: PyExc_IndexError      (src/lxml/objectify.so)
undefined symbol: PyTuple_Type  (src/lxml/objectify.so)
undefined symbol: PyExc_AttributeError  (src/lxml/objectify.so)
undefined symbol: PyClass_Type  (src/lxml/objectify.so)
undefined symbol: nextElement   (src/lxml/objectify.so)
undefined symbol: PyList_Type   (src/lxml/objectify.so)
undefined symbol: PyExc_NotImplementedError     (src/lxml/objectify.so)
undefined symbol: PyBool_Type   (src/lxml/objectify.so)
undefined symbol: PyInstance_Type       (src/lxml/objectify.so)
undefined symbol: PyType_Type   (src/lxml/objectify.so)
undefined symbol: PyObject_GC_Del       (src/lxml/objectify.so)
undefined symbol: PyExc_NameError       (src/lxml/objectify.so)
        linux-gate.so.1 =>  (0xffffe000)
        libexslt.so.0 => /usr/lib/libexslt.so.0 (0xa7eb2000)
        libxslt.so.1 => /usr/lib/libxslt.so.1 (0xa7e80000)
        libxml2.so.2 => /usr/lib/libxml2.so.2 (0xa7d67000)
        libpthread.so.0 => /lib/tls/libpthread.so.0 (0xa7d55000)
        libc.so.6 => /lib/tls/libc.so.6 (0xa7c23000)
        libgcrypt.so.11 => /usr/lib/libgcrypt.so.11 (0xa7bd2000)
        libgpg-error.so.0 => /usr/lib/libgpg-error.so.0 (0xa7bcd000)
        libm.so.6 => /lib/tls/libm.so.6 (0xa7ba8000)
        libdl.so.2 => /lib/tls/libdl.so.2 (0xa7ba4000)
        libz.so.1 => /usr/lib/libz.so.1 (0xa7b90000)
        /lib/ld-linux.so.2 (0x75555000)
        libnsl.so.1 => /lib/tls/libnsl.so.1 (0xa7b7a000)

Can anybody give me a hint where previousElement is defined? Or did I
do something wrong?

Thanks in advance.

Achim Kern

PS: if responding please put me into the cc, as I am currently not
able to receive mails from mailinglists. Thank you.

From Holger.Joukl at LBBW.de  Wed Sep 27 15:32:46 2006
From: Holger.Joukl at LBBW.de (Holger Joukl)
Date: Wed, 27 Sep 2006 15:32:46 +0200
Subject: [lxml-dev] [objectify] StringElement: Implementing string methods,
	revisited
In-Reply-To: <87fyed4lpq.fsf@hirschmanngmbh.com>
Message-ID: <OF875BE0A2.020A86F0-ONC12571F6.004708F5-C12571F6.004A6985@LBBW.de>

Hi,
I know we've discussed that before but I ran into it again:
The StringElement class mirrors only a small subset of native
python string methods. Converting my codebase to lxml.objectify
I now have to substitute every occurance of <obj>.<some string method>
with <obj>.pyval.<some string method>.
Automating this depends on developing sed magic for all of these methods.

I now wonder if it wouldn't make sense to rather add the string methods (*)
to StringElement instead of putting work into developing the substitution
scripts, making StringElement much more feature rich.
I don't think that having the methods has any negative impact on
StringElement
regarding child access as
- data elements should normally not have child elements (although this is
not enforced)
- you can always access child elements with names that conflict with
element
methods using the find() method.

I could add that code and tests for it and post a patch.

The one reason to not do this might be the rationale "when operating
on objectify string elements always use <obj>.pyval" and avoid the
additional
code. But then again I think all these methods would rather just return
_strValueOf(self).<method>(...), thus being rather maintenance-robust.

Stefan?

Holger

(*) minus the sequence protocol stuff, of course

Der Inhalt dieser E-Mail ist vertraulich. Falls Sie nicht der angegebene
Empf?nger sind oder falls diese E-Mail irrt?mlich an Sie adressiert wurde,
verst?ndigen Sie bitte den Absender sofort und l?schen Sie die E-Mail
sodann. Das unerlaubte Kopieren sowie die unbefugte ?bermittlung sind nicht
gestattet. Die Sicherheit von ?bermittlungen per E-Mail kann nicht
garantiert werden. Falls Sie eine Best?tigung w?nschen, fordern Sie bitte
den Inhalt der E-Mail als Hardcopy an.

The contents of this  e-mail are confidential. If you are not the named
addressee or if this transmission has been addressed to you in error,
please notify the sender immediately and then delete this e-mail.  Any
unauthorized copying and transmission is forbidden. E-Mail transmission
cannot be guaranteed to be secure. If verification is required, please
request a hard copy version.



From ianb at colorstudy.com  Wed Sep 27 17:10:03 2006
From: ianb at colorstudy.com (Ian Bicking)
Date: Wed, 27 Sep 2006 10:10:03 -0500
Subject: [lxml-dev] Testing lxml on Windows [was: Re: lxml 1.1.1 win32
 builds for 2.4 and 2.5]
In-Reply-To: <45198CF1.10004@gkec.informatik.tu-darmstadt.de>
References: <2AB7346A3227A74BB97F9A0D79E3E65A03EFE1@mailserver.kalyptorisk.com>	<45141567.3070802@gkec.informatik.tu-darmstadt.de>	<20060922165810.GA6687@cotia>
	<20060924123116.GB4337@cotia>	<4516A5F7.50304@gkec.informatik.tu-darmstadt.de>	<20060925002543.GE4337@cotia>
	<45198CF1.10004@gkec.informatik.tu-darmstadt.de>
Message-ID: <451A944B.9070103@colorstudy.com>

Stefan Behnel wrote:
>> Maybe if I can do 'easy_install lxml=1.1 develop' (or whatever is the
>> right incantation) and get a branch checkout? That would be a ton
>> easier.
> 
> Sadly, cheeseshop can't handle this. But SVN access should work just fine.

You can kind of manage this.  You need a link like this (possibly in the 
long_description that gets put on the Cheese Shop):

   http://codespeak.net/svn/lxml/branch/threading#egg=lxml-threading

Then you can do:

   easy_install -e -b . lxml==threading

And it will check out (as "editable": -e) the branch.  You'll have to do 
"python setup.py develop" on your own after that.



-- 
Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org

From ltucker at openplans.org  Wed Sep 27 17:24:18 2006
From: ltucker at openplans.org (Luke Tucker)
Date: Wed, 27 Sep 2006 11:24:18 -0400
Subject: [lxml-dev] problems accessing style tag text
In-Reply-To: <451987D2.6080303@gkec.informatik.tu-darmstadt.de>
References: <1159286995.24787.10.camel@ltucker>
	<451987D2.6080303@gkec.informatik.tu-darmstadt.de>
Message-ID: <1159370658.6139.17.camel@ltucker>


Thanks for the prompt fix, I appreciate it. I updated this morning, and
it is working correctly for me now. 

- Luke 

On Tue, 2006-09-26 at 22:04 +0200, Stefan Behnel wrote:

> 
> The thing is that lxml always sets up the parser to convert CDATA sections to
> normal text nodes, which simplifies the internal handling of text quite a bit.
> However, the HTML parser does not have such a parser option, so CDATA sections
> slip through in this case. Especially, as the libxml2 HTML parser generates
> them explicitly for script content and CSS.
> 
> I consider this a bug that should be fixed.
> 
> It looks like we can prevent the CDATA generation by modifying the SAX parser
> function table of libxml2 in place (set the cdataBlock function to NULL).
> That's how libxml2 handles the XML_PARSE_NOCDATA flag internally and it seems
> to work for HTML just fine. I'll commit it to both the trunk and the 1.1 branch.
> 



From behnel_ml at gkec.informatik.tu-darmstadt.de  Wed Sep 27 17:59:50 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Wed, 27 Sep 2006 17:59:50 +0200
Subject: [lxml-dev] Building Problems
In-Reply-To: <87fyed4lpq.fsf@hirschmanngmbh.com>
References: <87fyed4lpq.fsf@hirschmanngmbh.com>
Message-ID: <451A9FF6.3010108@gkec.informatik.tu-darmstadt.de>

Hi Achim,

Achim Kern wrote:
> during googeling on how to write easier xml datastores with python I
> just found our project. Especialy the objectify modules impressed
> me. So to test things I wanted to install it. Unfortunatly I can not
> use the provided debian package as there is only one for version 1.03
> not including the objectify extension. So I downloaded the source of
> 1.1.1 from codespeak.net extracted it and that's what it did.
> 
> # tar -xzf lxml-1.1.1.tgz
> # ce lxml-1.1.1
> # make clean test. 
> python setup.py  build_ext -i
> Building lxml version 1.1

1.1? Not 1.1.1?


> running build_ext
> python test.py -p -v

You did build it, right? I assume this is a second try after already having
built it once.

Did you do "make clean" in between? That removes the ".c" files, which means
you need a special Pyrex version to rebuild it. See "doc/build.txt". If you
only unpack the tgz and build from that, you should not need Pyrex as the ".c"
files are included.

Please retry the above with a clean setup and if that still fails, send a
complete copy of your attempted commands and the resulting output to the list.

Stefan

From behnel_ml at gkec.informatik.tu-darmstadt.de  Wed Sep 27 18:05:03 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Wed, 27 Sep 2006 18:05:03 +0200
Subject: [lxml-dev] [objectify] StringElement: Implementing string
 methods, revisited
In-Reply-To: <OF875BE0A2.020A86F0-ONC12571F6.004708F5-C12571F6.004A6985@LBBW.de>
References: <OF875BE0A2.020A86F0-ONC12571F6.004708F5-C12571F6.004A6985@LBBW.de>
Message-ID: <451AA12F.2040801@gkec.informatik.tu-darmstadt.de>

Hi Holger,

Holger Joukl wrote:
> I know we've discussed that before but I ran into it again:
> The StringElement class mirrors only a small subset of native
> python string methods. Converting my codebase to lxml.objectify
> I now have to substitute every occurance of <obj>.<some string method>
> with <obj>.pyval.<some string method>.
> Automating this depends on developing sed magic for all of these methods.
> 
> I now wonder if it wouldn't make sense to rather add the string methods (*)
> to StringElement instead of putting work into developing the substitution
> scripts, making StringElement much more feature rich.
> I don't think that having the methods has any negative impact on
> StringElement
> regarding child access as
> - data elements should normally not have child elements (although this is
> not enforced)
> - you can always access child elements with names that conflict with
> element
> methods using the find() method.
> 
> I could add that code and tests for it and post a patch.

Feel free to figure out which methods make sense (and can be supported) and
then post a list or even a patch if you like.


> The one reason to not do this might be the rationale "when operating
> on objectify string elements always use <obj>.pyval" and avoid the
> additional
> code. But then again I think all these methods would rather just return
> _strValueOf(self).<method>(...), thus being rather maintenance-robust.

Maintenance is not my main concern. The problem is that we provide an
incomplete interface here, so it's "kinda compatible, but not quite", which I
consider worse than "no string methods there". I fear that the choice of
methods may look too arbitrary to understand.

But as I said, feel free to convince me.

Stefan


From tseaver at palladion.com  Wed Sep 27 20:25:59 2006
From: tseaver at palladion.com (Tres Seaver)
Date: Wed, 27 Sep 2006 14:25:59 -0400
Subject: [lxml-dev] Building Problems
In-Reply-To: <451A9FF6.3010108@gkec.informatik.tu-darmstadt.de>
References: <87fyed4lpq.fsf@hirschmanngmbh.com>
	<451A9FF6.3010108@gkec.informatik.tu-darmstadt.de>
Message-ID: <451AC237.9090204@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stefan Behnel wrote:
> Hi Achim,
> 
> Achim Kern wrote:
>> during googeling on how to write easier xml datastores with python I
>> just found our project. Especialy the objectify modules impressed
>> me. So to test things I wanted to install it. Unfortunatly I can not
>> use the provided debian package as there is only one for version 1.03
>> not including the objectify extension. So I downloaded the source of
>> 1.1.1 from codespeak.net extracted it and that's what it did.
>>
>> # tar -xzf lxml-1.1.1.tgz
>> # ce lxml-1.1.1
>> # make clean test. 
>> python setup.py  build_ext -i
>> Building lxml version 1.1
> 
> 1.1? Not 1.1.1?
> 
> 
>> running build_ext
>> python test.py -p -v
> 
> You did build it, right? I assume this is a second try after already having
> built it once.
> 
> Did you do "make clean" in between? That removes the ".c" files, which means
> you need a special Pyrex version to rebuild it.

That sounds as thought it should be the result of 'make distclean';
make clean should remove .o files, etc., but leave the tree in a state
in which 'make' can be re-run safely.


Tres.
- --
===================================================================
Tres Seaver          +1 202-558-7113          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD4DBQFFGsI3+gerLs4ltQ4RAhcYAJY28BThKHuNMa7cp0hIezmNIcgFAJwLxi+1
3Lw3DNsAokeEkyT/EZRBmw==
=JlHg
-----END PGP SIGNATURE-----


From behnel_ml at gkec.informatik.tu-darmstadt.de  Thu Sep 28 08:51:37 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Thu, 28 Sep 2006 08:51:37 +0200
Subject: [lxml-dev] Building Problems
In-Reply-To: <451AC237.9090204@palladion.com>
References: <87fyed4lpq.fsf@hirschmanngmbh.com>	<451A9FF6.3010108@gkec.informatik.tu-darmstadt.de>
	<451AC237.9090204@palladion.com>
Message-ID: <451B70F9.3070703@gkec.informatik.tu-darmstadt.de>

Hi Tres,

Tres Seaver wrote:
> Stefan Behnel wrote:
>> Did you do "make clean" in between? That removes the ".c" files, which means
>> you need a special Pyrex version to rebuild it.
> 
> That sounds as though it should be the result of 'make distclean';
> make clean should remove .o files, etc., but leave the tree in a state
> in which 'make' can be re-run safely.

Well, it can be re-run safely, if you have Pyrex installed.

The thing is, as long as you don't modify anything in the sources, there is
not much of a reason to run "make clean", but if you do changes, you need
Pyrex anyway to regenerate the ".c" files. So there is much to win from
distinguishing "make clean" and "make distclean".

Stefan


From faassen at infrae.com  Thu Sep 28 13:16:08 2006
From: faassen at infrae.com (Martijn Faassen)
Date: Thu, 28 Sep 2006 13:16:08 +0200
Subject: [lxml-dev] a buildout for lxml
Message-ID: <451BAEF8.1020101@infrae.com>

Hi there,

I've just created what's called a "buildout" for lxml. Buildout is a new 
system for constructing Python applications and its constituent parts. 
More information can be found here:

http://cheeseshop.python.org/pypi/zc.buildout/

Now before you read all that (it's fairly technical), let me explain 
what problem the lxml buildout tries to solve.

Yesterday I tried using lxml's XMLSchema validator against an XML file 
and a schema. It gave a validation error, but to the best of my 
knowledge, the XML file should've been valid. Thinking it might perhaps 
be a bug that was fixed in more recent versions of libxml2, I set out to 
build and install libxml2 and libxslt locally, compiled lxml against it, 
and confirmed it: the document does validate against the schema with the 
most recent release of libxml2.

Now downloading, building and installing libxml2 and libxslt and linking 
lxml against it is quite a bit of work. Buildout to the rescue! Now it's 
just this:

$ svn co https://infrae.com/svn/buildout/lxml-recent/trunk lxml-recent
$ cd lxml-recent
$ python bootstrap/bootstrap.py
$ bin/buildout

and then wait. It'll download, configure and compile libxml2, libxslt, 
will downoad the lxml source and compile it against the libraries.

In the end you end up with a script that invokes the Python interpreter 
with the right paths, so you can import the new lxml:

$ bin/lxmlpython

Though you can also manipulate the PYTHONPATH yourself in your scripts 
to put the lxml development egg on it, and then you can use the regular 
Python you used to run bootstrap.py with.

To see how this works, you can take a look at buildout.cfg. This
specifies how things are put together.

Note that 'zc.recipe.cmmi' is a 'buildout recipe'. Recipes are 
components responsible for deployment of particular bits of an application.

If zc.recipe.cmmi were released as an egg onto the cheeseshop, buildout 
would go out and download and install the recipe from there. Since it's 
not on the cheeseshop yet, I've installed it as a development-egg and 
included it as an svn:external, but this can be fixed by wrapping that 
into an egg.

Many buildout recipes exist: this one does 'configure; make; make 
install', hence 'cmmi'. There are also recipes that build and install 
Zope 3, recipes to build tricky Python modules that don't work as an egg 
yet, and so on. New recipes can be developed. At Infrae's we've created 
recipes that can deploy OpenOffice in server mode, so that it can be 
used through UNO for document conversion purposes. This expanding pool 
of recipes will allow us to put together extremely complicated 
applications in a reproducable fashion. The idea of buildout is to have 
control - to know exactly what is installed and not to have to install 
everything into site-packages.

buildout uses setuptools and easy_install internally, so the whole egg 
story just works; it expands on that.

I think this way of building lxml has more benefits than just my single 
case. It'll allow one to deploy lxml against a known-good version of 
libxml2 and libxslt, at least on a linux system. It should also help 
testing and developing lxml - it becomes a lot more easy to test and 
develop against different versions of libxml2.

I've placed this buildout in infrae.com SVN for now (checkout command 
above), but if people think this is interesting, I can integrate this 
into the lxml trunk. This can be done without harming the current way of 
working, but it'll allow someone to test against any version of libxml2 
and libxslt a lot more easily than one can do now.

Comments?

Regards,

Martijn


From behnel_ml at gkec.informatik.tu-darmstadt.de  Thu Sep 28 19:51:20 2006
From: behnel_ml at gkec.informatik.tu-darmstadt.de (Stefan Behnel)
Date: Thu, 28 Sep 2006 19:51:20 +0200
Subject: [lxml-dev] Stylesheet Processing Instruction
In-Reply-To: <20060925200751.GJ4356@cotia>
References: <20060919003021.GQ3853@cotia> <20060919020239.GR3853@cotia>
	<450F90CA.6060902@gkec.informatik.tu-darmstadt.de>
	<450FC62B.2070905@infrae.com> <20060919132042.GU3853@cotia>
	<450FF3B3.1060004@infrae.com>
	<4510174C.3010209@gkec.informatik.tu-darmstadt.de>
	<4511A415.9080705@gkec.informatik.tu-darmstadt.de>
	<20060921134349.GR3853@cotia>
	<451413E7.5010205@gkec.informatik.tu-darmstadt.de>
	<20060925200751.GJ4356@cotia>
Message-ID: <451C0B98.4030308@gkec.informatik.tu-darmstadt.de>

Hi Sidnei,

Sidnei da Silva wrote:
> On Fri, Sep 22, 2006 at 06:48:39PM +0200, Stefan Behnel wrote:
> | I committed an implementation of the above to the SVN trunk, please check it
> | out to see if it fits your expectations.
> 
> Looks like you only wrote tests for the embedded id inclusion. Would
> be nice to add a test for filename/url inclusion just to make
> sure.

Added now, which made me reconsider the return value. As it resembles the
parse() function/method, parseXSL() will now always return an ElementTree,
regardless of the source of the stylesheet.

Stefan



From cabraham at openplans.org  Fri Sep 29 17:03:51 2006
From: cabraham at openplans.org (Chris Abraham)
Date: Fri, 29 Sep 2006 11:03:51 -0400
Subject: [lxml-dev] lxml replace() deletes tail
Message-ID: <451D35D7.4040306@openplans.org>

Hi,
We have a question about the etree.replace() function.  We found that it
doesn't preserve the tail text from the replaced node when inserting a
new node.  Perhaps this is the intended behavior, but, to us, it was
unexpected.  In the example below, notice how the "tail" text is deleted
when the <p> is replaced:

>>> tree = etree.HTML("<html><body>text
before<p>textin</p>tail</body></html>")
>>> newel = etree.HTML("<html><body><div>new</div></body></html>")
>>> tree[0].replace(tree[0][0], newel[0][0])
>>> etree.tostring(tree)
'<html><body>text before<div>new</div></body></html>'

Any ideas or comments?  Thanks.
Luke Tucker & Chris Abraham