From elliottslaughter at gmail.com  Wed Jul  1 00:04:26 2009
From: elliottslaughter at gmail.com (Elliott Slaughter)
Date: Tue, 30 Jun 2009 15:04:26 -0700
Subject: [lxml-dev] parsing DTDs - listing of valid elements
Message-ID: <42c0ab790906301504x3eddea6at21c32c3b17f0aa1f@mail.gmail.com>

Hi,

I'm trying to get the elements in a DTD. Since these internals are not
exported in the Python interface of lxml.etree, I am trying to write a
Cython extension to do so, as previously suggested on this mailing list (see
link below).

http://codespeak.net/pipermail/lxml-dev/2009-January/004298.html

To quote the message, "all you'd really need is the internal _c_dtd field of
the DTD class, which you could cimport". I'm wondering exactly how I am
supposed to do that (my attempts so far are described below). It would also
be nice to know if the last attempt to do so was successful or not.

Thanks. Any help would be appreciated.



Here is what I've tried so far (on Python 2.5.4, Cython 0.11.2, Windows):

The DTD class is not declared in etreepublic.pxd, so I can't just "cimport
etreepublic". The actual DTD class definition is in dtd.pxi, as stated in
the message. But I can't just "include 'dtd.pxi' " because it inherits from
the _Validator class in lxml.etree.pyx . And I can't "cimport lxml.etree"
because there is no file lxml.etree.pxd.

I tried writing a lxml.etree.pxd file to circumvent these barriers (which
was thoroughly confusing because _Validator contains an _ErrorLog which made
me search through several other files...), but even when I got the entire
thing to compile, it failed to load in Python:

>>> import mydtd
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "lxml.etree.pxd", line 3, in mydtd (mydtd.c:513)
    cdef class _LogEntry:
ValueError: lxml.etree._LogEntry does not appear to be the correct type
object

I have attached my lxml.etree.pxd in case I made any mistakes, in the event
that this method can be made to work.

-- 
Elliott Slaughter

"Don't worry about what anybody else is going to do. The best way to predict
the future is to invent it." - Alan Kay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20090630/0a7b98d7/attachment.htm 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: lxml.etree.pxd
Type: application/octet-stream
Size: 602 bytes
Desc: not available
Url : http://codespeak.net/pipermail/lxml-dev/attachments/20090630/0a7b98d7/attachment.obj 

From kris at cs.ucsb.edu  Wed Jul  1 00:18:31 2009
From: kris at cs.ucsb.edu (kristian kvilekval)
Date: Tue, 30 Jun 2009 15:18:31 -0700
Subject: [lxml-dev] Binary egg for Mac OS X
In-Reply-To: <1246087234.3475.28.camel@krispc.sd.cox.net>
References: <op.uuh979ydvhba00@plym.local> <4A2112AE.8040903@behnel.de>
	<op.uu6hmuvdvhba00@mobile-166-129-055-119.mycingular.net>
	<4A2CA7AE.8030508@behnel.de>
	<b4ef227c0906080150mac0037dk395515c8445d8e13@mail.gmail.com>
	<e909ed11f3e611bb7a91c4cb3393a414.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<97EFCE41-4A6E-4CB2-8593-CE7CF5F5A2D3@inquant.de>
	<4A34DD37.8000207@behnel.de> <4A3B1D56.3000007@behnel.de>
	<op.uvraxstivhba00@mobile-166-129-245-182.mycingular.net>
	<op.uvxvsfduvhba00@mobile-166-129-132-075.mycingular.net>
	<4A41AD28.4010202@behnel.de>	<1245882881.10362.239.camel@loup.ece.ucsb.edu>
	<op.uv1x73qnvhba00@plym.local>  <4A45C2F0.5030802@behnel.de>
	<1246087234.3475.28.camel@krispc.sd.cox.net>
Message-ID: <1246400311.10362.293.camel@loup.ece.ucsb.edu>

Hi,

    We are critical need of an installable 
recent lxml egg for Mac OS 10.5.7 with python 2.5

Could you please post the building instructions you used to 
create the eggs for python 2.4

Thanks,
Kris




From tseaver at palladion.com  Wed Jul  1 01:15:32 2009
From: tseaver at palladion.com (Tres Seaver)
Date: Tue, 30 Jun 2009 19:15:32 -0400
Subject: [lxml-dev] Binary egg for Mac OS X
In-Reply-To: <1246400311.10362.293.camel@loup.ece.ucsb.edu>
References: <op.uuh979ydvhba00@plym.local>
	<4A2112AE.8040903@behnel.de>	<op.uu6hmuvdvhba00@mobile-166-129-055-119.mycingular.net>	<4A2CA7AE.8030508@behnel.de>	<b4ef227c0906080150mac0037dk395515c8445d8e13@mail.gmail.com>	<e909ed11f3e611bb7a91c4cb3393a414.squirrel@groupware.dvs.informatik.tu-darmstadt.de>	<97EFCE41-4A6E-4CB2-8593-CE7CF5F5A2D3@inquant.de>	<4A34DD37.8000207@behnel.de>
	<4A3B1D56.3000007@behnel.de>	<op.uvraxstivhba00@mobile-166-129-245-182.mycingular.net>	<op.uvxvsfduvhba00@mobile-166-129-132-075.mycingular.net>	<4A41AD28.4010202@behnel.de>	<1245882881.10362.239.camel@loup.ece.ucsb.edu>	<op.uv1x73qnvhba00@plym.local>
	<4A45C2F0.5030802@behnel.de>	<1246087234.3475.28.camel@krispc.sd.cox.net>
	<1246400311.10362.293.camel@loup.ece.ucsb.edu>
Message-ID: <h2e6am$fg1$1@ger.gmane.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

kristian kvilekval wrote:
> Hi,
> 
>     We are critical need of an installable 
> recent lxml egg for Mac OS 10.5.7 with python 2.5
> 
> Could you please post the building instructions you used to 
> create the eggs for python 2.4

This is what I use to build lxml in CentOS4, which has the
"too-old-libxml2-and-libxslt" problem:

 $ wget http://pypi.python.org/packages/source/l/lxml/lxml-2.2.2.tar.gz
 ...
 $ tar xzf lxml-2.2.2.tar.gz
 $ cd lxml-2.2.2
 $ /path/to/python setup.py bdist_egg --static-deps


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKSpyU+gerLs4ltQ4RAjrpAJ9OSCMg2OrVviezfDnAdg9krNqcnQCfdBne
3MOjrE76sMTqtjcrDMatLcE=
=Ihsx
-----END PGP SIGNATURE-----


From elliottslaughter at gmail.com  Wed Jul  1 02:29:19 2009
From: elliottslaughter at gmail.com (Elliott Slaughter)
Date: Tue, 30 Jun 2009 17:29:19 -0700
Subject: [lxml-dev] parsing DTDs - listing of valid elements
In-Reply-To: <42c0ab790906301504x3eddea6at21c32c3b17f0aa1f@mail.gmail.com>
References: <42c0ab790906301504x3eddea6at21c32c3b17f0aa1f@mail.gmail.com>
Message-ID: <42c0ab790906301729w326e3f5ahbbf15ea4d920a914@mail.gmail.com>

Please ignore my previous message; I solved my own problem by finding an XML
schema for what I need to do.

Sorry for the noise.

On Tue, Jun 30, 2009 at 3:04 PM, Elliott Slaughter <
elliottslaughter at gmail.com> wrote:

> Hi,
>
> I'm trying to get the elements in a DTD. Since these internals are not
> exported in the Python interface of lxml.etree, I am trying to write a
> Cython extension to do so, as previously suggested on this mailing list (see
> link below).
>
> http://codespeak.net/pipermail/lxml-dev/2009-January/004298.html
>
> To quote the message, "all you'd really need is the internal _c_dtd field
> of the DTD class, which you could cimport". I'm wondering exactly how I am
> supposed to do that (my attempts so far are described below). It would also
> be nice to know if the last attempt to do so was successful or not.
>
> Thanks. Any help would be appreciated.
>
>
>
> Here is what I've tried so far (on Python 2.5.4, Cython 0.11.2, Windows):
>
> The DTD class is not declared in etreepublic.pxd, so I can't just "cimport
> etreepublic". The actual DTD class definition is in dtd.pxi, as stated in
> the message. But I can't just "include 'dtd.pxi' " because it inherits from
> the _Validator class in lxml.etree.pyx . And I can't "cimport lxml.etree"
> because there is no file lxml.etree.pxd.
>
> I tried writing a lxml.etree.pxd file to circumvent these barriers (which
> was thoroughly confusing because _Validator contains an _ErrorLog which made
> me search through several other files...), but even when I got the entire
> thing to compile, it failed to load in Python:
>
> >>> import mydtd
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "lxml.etree.pxd", line 3, in mydtd (mydtd.c:513)
>     cdef class _LogEntry:
> ValueError: lxml.etree._LogEntry does not appear to be the correct type
> object
>
> I have attached my lxml.etree.pxd in case I made any mistakes, in the event
> that this method can be made to work.
>
> --
> Elliott Slaughter
>
> "Don't worry about what anybody else is going to do. The best way to
> predict the future is to invent it." - Alan Kay
>



-- 
Elliott Slaughter

"Don't worry about what anybody else is going to do. The best way to predict
the future is to invent it." - Alan Kay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20090630/3a28afc0/attachment.htm 

From stefan_ml at behnel.de  Wed Jul  1 07:37:05 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 01 Jul 2009 07:37:05 +0200
Subject: [lxml-dev] parsing DTDs - listing of valid elements
In-Reply-To: <42c0ab790906301729w326e3f5ahbbf15ea4d920a914@mail.gmail.com>
References: <42c0ab790906301504x3eddea6at21c32c3b17f0aa1f@mail.gmail.com>
	<42c0ab790906301729w326e3f5ahbbf15ea4d920a914@mail.gmail.com>
Message-ID: <4A4AF601.7010900@behnel.de>

Hi,

Elliott Slaughter wrote:
> Please ignore my previous message; I solved my own problem by finding an XML
> schema for what I need to do.

Note that you can always use trang to convert a DTD to an XML Schema.

http://www.thaiopensource.com/relaxng/trang.html

If all you need is a list of allowed elements, the required logic to
extract that from the schema shouldn't be too hard to figure out. Although
I wonder if RelaxNG wouldn't be easier to work on.

Stefan

From stefan_ml at behnel.de  Wed Jul  1 08:14:57 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 01 Jul 2009 08:14:57 +0200
Subject: [lxml-dev] parsing DTDs - listing of valid elements
In-Reply-To: <42c0ab790906301504x3eddea6at21c32c3b17f0aa1f@mail.gmail.com>
References: <42c0ab790906301504x3eddea6at21c32c3b17f0aa1f@mail.gmail.com>
Message-ID: <4A4AFEE1.9080707@behnel.de>

Hi,

Elliott Slaughter wrote:
> I'm trying to get the elements in a DTD. Since these internals are not
> exported in the Python interface of lxml.etree, I am trying to write a
> Cython extension to do so, as previously suggested on this mailing list (see
> link below).
> 
> http://codespeak.net/pipermail/lxml-dev/2009-January/004298.html
> 
> To quote the message, "all you'd really need is the internal _c_dtd field of
> the DTD class, which you could cimport". I'm wondering exactly how I am
> supposed to do that
> [...]
> Here is what I've tried so far (on Python 2.5.4, Cython 0.11.2, Windows):
> 
> The DTD class is not declared in etreepublic.pxd, so I can't just "cimport
> etreepublic". The actual DTD class definition is in dtd.pxi, as stated in
> the message. But I can't just "include 'dtd.pxi' " because it inherits from
> the _Validator class in lxml.etree.pyx . And I can't "cimport lxml.etree"
> because there is no file lxml.etree.pxd.

True. So your only chance is to write one yourself. And yes, it needs to be
called "lxml.etree.pxd".


> I tried writing a lxml.etree.pxd file to circumvent these barriers (which
> was thoroughly confusing because _Validator contains an _ErrorLog which made
> me search through several other files...),

All you should really need is this:

	cimport tree

	cdef class _Validator:
	    cdef object _error_log

	cdef class DTD(_Validator):
	    cdef tree.xmlDtd* _c_dtd

Cython needs to know the exact /layout/ of the classes that you use (at
least if they are not exported as C header files), but it doesn't need to
know the exact class types of attributes. "object" will do just fine if you
don't care.

I know that this is harder than necessary (thanks for bringing this up,
BTW), but that's just because _DTD isn't an 'officially' C-exported type,
just like all other schema types.

Stefan


From maruadventurer at gmail.com  Thu Jul  2 04:08:33 2009
From: maruadventurer at gmail.com (john mcginnis)
Date: Wed, 1 Jul 2009 21:08:33 -0500
Subject: [lxml-dev] Really dumb question
Message-ID: <c1017c2a0907011908g4cc9e1dsfd8e7645250c30ed@mail.gmail.com>

I have completed a build with easy_install. Looking at site-packages I also
see a lxml directory that exists. What I don't find however is a lxml.py
module anywhere.

If you have a successful build should there not be a lxml.py module
somewhere?

Thanks.

JohnMc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20090701/6dcd978c/attachment-0001.htm 

From stefan_ml at behnel.de  Thu Jul  2 06:18:02 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 02 Jul 2009 06:18:02 +0200
Subject: [lxml-dev] Really dumb question
In-Reply-To: <c1017c2a0907011908g4cc9e1dsfd8e7645250c30ed@mail.gmail.com>
References: <c1017c2a0907011908g4cc9e1dsfd8e7645250c30ed@mail.gmail.com>
Message-ID: <4A4C34FA.3060008@behnel.de>


john mcginnis wrote:
> I have completed a build with easy_install. Looking at site-packages I also
> see a lxml directory that exists. What I don't find however is a lxml.py
> module anywhere.
> 
> If you have a successful build should there not be a lxml.py module
> somewhere?

No, lxml is a package, not a module. When you import "lxml", Python will
load "lxml/__init__.py" instead.

Stefan


From friedel at translate.org.za  Fri Jul  3 15:48:44 2009
From: friedel at translate.org.za (F Wolff)
Date: Fri, 03 Jul 2009 15:48:44 +0200
Subject: [lxml-dev] space normalisation for .text and .tail
Message-ID: <1246628924.19137.298.camel@localhost>

Hallo all

On 2009-03-24 I wrote about space normalisation with reference to the
xml:space attribute, and the string() and normalize-string() functions
in xpath. I solved my problem in code, partly due to slightly changing
requirements.

Now I need to do similar magic, but need to handle the text nodes
separately, without descending into child nodes.

>From the xpath document:
> The string-value of an element node is the concatenation of the
> string-values of all text node descendants of the element node in
> document order.
...which is not what I need to do in this case.

Is there a way to apply the normalize-text() to a node's .text or .tail
only? Is there another way to obtain the same result? From the looks of
it, there is no reliable way that I can normalise correctly in code,
since I won't know if a newline (for example) was given as a newline or
as a character reference, and this should influence the normalisation.

Any help is appreciated.

Friedel


--
Recently on my blog:
http://translate.org.za/blogs/friedel/en/content/presentation-afrilex-alasa-2009


From foolistbar at googlemail.com  Sat Jul  4 11:13:48 2009
From: foolistbar at googlemail.com (Geoffrey Sneddon)
Date: Sat, 4 Jul 2009 11:13:48 +0200
Subject: [lxml-dev] lxml.html, now with ignored namespaces!
In-Reply-To: <4A45ACBE.40107@behnel.de>
References: <mailman.39154.1245697305.27370.lxml-dev@codespeak.net>
	<4A40278E.3020805@chantofwaves.com> <4A45ACBE.40107@behnel.de>
Message-ID: <847184AA-5B2B-4C97-8F8F-F50FCD04C8F0@googlemail.com>


On 27 Jun 2009, at 07:23, Stefan Behnel wrote:

>> The output:
>> -----
>> <html xmlns="http://www.w3.org/TR/1999/REC-html-in-xml"
>> cs="http://something.com/cs" xml:lang="en"
>> lang="en"><head><title>Help!</title></head><body><p>My namespaces are
>> going to disappear!</p><p content="fruit">FRUIT</p></body></html>
>> -----
>
> That's because HTML parsers are not namespace aware. Namespaces are  
> simply
> not defined for HTML. But if you get a difference on different  
> systems, I'd
> still suspect the reason to be different libxml2 versions. There's  
> nothing
> lxml can do about this.

It should still be outputting an element with a name of "cs:content",  
it shouldn't be dropping the "cs:", as, as you say, there are not  
namespaces in HTML, so it has no meaning.

My basic advice to the OP would be to use html5lib, which is far  
slower, but does cope with this fine.


--
Geoffrey Sneddon
<http://gsnedders.com/>


From stefan_ml at behnel.de  Sat Jul  4 12:03:01 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 04 Jul 2009 12:03:01 +0200
Subject: [lxml-dev] lxml.html, now with ignored namespaces!
In-Reply-To: <847184AA-5B2B-4C97-8F8F-F50FCD04C8F0@googlemail.com>
References: <mailman.39154.1245697305.27370.lxml-dev@codespeak.net>
	<4A40278E.3020805@chantofwaves.com> <4A45ACBE.40107@behnel.de>
	<847184AA-5B2B-4C97-8F8F-F50FCD04C8F0@googlemail.com>
Message-ID: <4A4F28D5.2020008@behnel.de>

Hi,

Geoffrey Sneddon wrote:
>>> The output:
>>> -----
>>> <html xmlns="http://www.w3.org/TR/1999/REC-html-in-xml"
>>> cs="http://something.com/cs" xml:lang="en"
>>> lang="en"><head><title>Help!</title></head><body><p>My namespaces are
>>> going to disappear!</p><p content="fruit">FRUIT</p></body></html>
>>> -----
>
> My basic advice to the OP would be to use html5lib, which is far slower,
> but does cope with this fine.

Well, as I said, it just depends on the version of libxml2 that you are using.

>>> from lxml import etree
>>> print "lxml.etree:       ", etree.LXML_VERSION
lxml.etree:        (2, 2, 2, 0)
>>> print "libxml used:      ", etree.LIBXML_VERSION
libxml used:       (2, 6, 32)

>>> from lxml.html import fromstring

>>> document = fromstring("""<!DOCTYPE html PUBLIC "-//W3C//DTD
... XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd"><html
... xmlns="http://www.w3.org/TR/1999/REC-html-in-xml"
... xmlns:cs="http://something.com/cs" xml:lang="en"
... lang="en"><head><title>Help!</title></head><body><p>My namespaces are
... going to disappear!</p><p cs:content='fruit'>FRUIT</p></body></html>
... """)

>>> print parser.tostring(document)
<html xmlns="http://www.w3.org/TR/1999/REC-html-in-xml"
xmlns:cs="http://something.com/cs" xml:lang="en"
lang="en"><head><title>Help!</title></head><body><p>My namespaces are
going to disappear!</p><p cs:content="fruit">FRUIT</p></body></html>


Stefan

From stefan_ml at behnel.de  Sat Jul  4 13:24:00 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 04 Jul 2009 13:24:00 +0200
Subject: [lxml-dev] space normalisation for .text and .tail
In-Reply-To: <1246628924.19137.298.camel@localhost>
References: <1246628924.19137.298.camel@localhost>
Message-ID: <4A4F3BD0.7070108@behnel.de>

Hi,

F Wolff wrote:
> On 2009-03-24 I wrote about space normalisation with reference to the
> xml:space attribute, and the string() and normalize-string() functions
> in xpath. I solved my problem in code, partly due to slightly changing
> requirements.
> 
> Now I need to do similar magic, but need to handle the text nodes
> separately, without descending into child nodes.
> 
>>From the xpath document:
>> The string-value of an element node is the concatenation of the
>> string-values of all text node descendants of the element node in
>> document order.
> ...which is not what I need to do in this case.
> 
> Is there a way to apply the normalize-text() to a node's .text or .tail
> only? Is there another way to obtain the same result?

Well, lxml will not allow you to modify individual text nodes that the
parser created next to each other for whatever reason (likely due to
implementation details), even if XPath allows you to get your hands on them
using "text()". The text/tail properties are as deep down as it gets.


> From the looks of
> it, there is no reliable way that I can normalise correctly in code,
> since I won't know if a newline (for example) was given as a newline or
> as a character reference, and this should influence the normalisation.

Why is that? XML parsers will always replace character references by their
Unicode character value, and there is no way XPath could see them. If you
need that information for your algorithm, you will have to parse the XML
byte stream yourself. Neither the XML infoset nor the XPath data model
provide this.

Stefan

From friedel at translate.org.za  Tue Jul  7 09:29:51 2009
From: friedel at translate.org.za (F Wolff)
Date: Tue, 07 Jul 2009 09:29:51 +0200
Subject: [lxml-dev] space normalisation for .text and .tail
In-Reply-To: <4A4F3BD0.7070108@behnel.de>
References: <1246628924.19137.298.camel@localhost> <4A4F3BD0.7070108@behnel.de>
Message-ID: <1246951791.15406.14.camel@localhost.localdomain>

Op Sa, 2009-07-04 om 13:24 +0200 skryf Stefan Behnel:
> Hi,
> 
> F Wolff wrote:
> > On 2009-03-24 I wrote about space normalisation with reference to the
> > xml:space attribute, and the string() and normalize-string() functions
> > in xpath. I solved my problem in code, partly due to slightly changing
> > requirements.
> > 
> > Now I need to do similar magic, but need to handle the text nodes
> > separately, without descending into child nodes.
> > 
> >>From the xpath document:
> >> The string-value of an element node is the concatenation of the
> >> string-values of all text node descendants of the element node in
> >> document order.
> > ...which is not what I need to do in this case.
> > 
> > Is there a way to apply the normalize-text() to a node's .text or .tail
> > only? Is there another way to obtain the same result?
> 
> Well, lxml will not allow you to modify individual text nodes that the
> parser created next to each other for whatever reason (likely due to
> implementation details), even if XPath allows you to get your hands on them
> using "text()". The text/tail properties are as deep down as it gets.

Sorry, let me rephrase: I don't need to alter the internal XML
structure, I just want to obtain normalised versions of the .text
and .tail nodes in a tree with text and xml elements intertwined.

For example:
<a>
    Moo
    <b>
    Mew
    </b>
    bla    bla
</a>

In this case I'm looking or a way to obtain the strings "Moo", "Mew",
and "bla bla" (with the the spaces normalised).  XPath's
normalize-text() can give me "Moo Mew bla bla", but I still want access
to each .text and .tail separately normalised.

> 
> > From the looks of
> > it, there is no reliable way that I can normalise correctly in code,
> > since I won't know if a newline (for example) was given as a newline or
> > as a character reference, and this should influence the normalisation.
> 
> Why is that? XML parsers will always replace character references by their
> Unicode character value, and there is no way XPath could see them. If you
> need that information for your algorithm, you will have to parse the XML
> byte stream yourself. Neither the XML infoset nor the XPath data model
> provide this.
> 
> Stefan

My understanding was that the normalisation does not touch entities, and
that the following two is not equivalent when normalised;
<a>&#10;</a>

vs.

<a>
</a>

...but playing now with normalize-string() it seems that they are
equivalent.

Would it be possible to normalise correctly in code in all cases?

Thank you for the help.
Friedel


--
Recently on my blog:
http://translate.org.za/blogs/friedel/en/content/presentation-afrilex-alasa-2009


From herve.cauwelier at free.fr  Thu Jul  9 20:35:10 2009
From: herve.cauwelier at free.fr (=?UTF-8?B?SGVydsOpIENhdXdlbGllcg==?=)
Date: Thu, 09 Jul 2009 20:35:10 +0200
Subject: [lxml-dev] blather question about XML declaration
Message-ID: <4A56385E.1000001@free.fr>

Hi,

I just wonder why lxml is producing XML declarations such as

  <?xml version='1.0' encoding='UTF-8'?>

or sometimes

  <?xml version=&apos;1.0&apos; encoding=&apos;UTF-8&apos;?>

instead of the most common (to my knowledge)

    <?xml version="1.0" encoding="UTF-8"?>

For now I don't ask tostring() to print it, I concatenate my own with
the tree serialisation.

Thanks,

Herv?

From stefan_ml at behnel.de  Thu Jul  9 22:03:28 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 09 Jul 2009 22:03:28 +0200
Subject: [lxml-dev] blather question about XML declaration
In-Reply-To: <4A56385E.1000001@free.fr>
References: <4A56385E.1000001@free.fr>
Message-ID: <4A564D10.7010504@behnel.de>

Hi,

Herv? Cauwelier wrote:
> I just wonder why lxml is producing XML declarations such as
> 
>   <?xml version='1.0' encoding='UTF-8'?>
> 
> or sometimes
> 
>   <?xml version=&apos;1.0&apos; encoding=&apos;UTF-8&apos;?>

Except that the latter is impossible to get out of the serialiser.


> instead of the most common (to my knowledge)
> 
>     <?xml version="1.0" encoding="UTF-8"?>

Why should this be more common? Just because Java doesn't understand
strings in single quotes?

The XML spec allows both:

http://www.w3.org/TR/REC-xml/#sec-prolog-dtd

and I see no reason to use '"'. It just degrades readability.


> For now I don't ask tostring() to print it, I concatenate my own with
> the tree serialisation.

Should work as long as you only serialise to UTF-8 (in which case the
declaration is optional anyway).

Stefan


From stefan_ml at behnel.de  Fri Jul 10 06:49:12 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 10 Jul 2009 06:49:12 +0200
Subject: [lxml-dev] bizarre crashes under FreeBSD
Message-ID: <4A56C848.2030202@behnel.de>

Hi all,

I got a bug report on crashes under FreeBSD that are not reproducible on
other systems. The (varying) problem descriptions sound rather bizarre and
unlikely. They are all threading related, one even regards the thread-local
storage in Python.

https://bugs.launchpad.net/bugs/397516

Does anyone have experience with threading on FreeBSD? Any ideas where
these problems may arise from? Could anyone test the attached test script
on other FreeBSD systems?

Thanks for any hints.

Stefan


From stefan_ml at behnel.de  Sat Jul 11 21:12:40 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sat, 11 Jul 2009 21:12:40 +0200
Subject: [lxml-dev] bizarre crashes under FreeBSD
In-Reply-To: <4A56C848.2030202@behnel.de>
References: <4A56C848.2030202@behnel.de>
Message-ID: <4A58E428.8070908@behnel.de>


Stefan Behnel wrote:
> I got a bug report on crashes under FreeBSD that are not reproducible on
> other systems. The (varying) problem descriptions sound rather bizarre and
> unlikely. They are all threading related, one even regards the thread-local
> storage in Python.
> 
> https://bugs.launchpad.net/bugs/397516
> 
> Does anyone have experience with threading on FreeBSD? Any ideas where
> these problems may arise from? Could anyone test the attached test script
> on other FreeBSD systems?

As a quick follow-up: it seems that static building fixes this. No idea
what the original problem was, though...

Stefan


From jonas.esp at googlemail.com  Sun Jul 12 19:09:15 2009
From: jonas.esp at googlemail.com (Jonas)
Date: Sun, 12 Jul 2009 17:09:15 +0000
Subject: [lxml-dev] complex node
Message-ID: <20cf13940907121009m5488fd55q5a04b60e32409161@mail.gmail.com>

Hi,

I'm trying to parsing a xml file (in python) to convert it to
dictionaries and then to JSON.

This is the file:
http://unicode.org/cldr/data/common/main/es.xml
and I've got all nodes except dates', which is enought complex.

Any idea to get it? because I'm very tired of try it.

From stefan_ml at behnel.de  Mon Jul 13 08:16:16 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 13 Jul 2009 08:16:16 +0200
Subject: [lxml-dev] complex node
In-Reply-To: <20cf13940907121009m5488fd55q5a04b60e32409161@mail.gmail.com>
References: <20cf13940907121009m5488fd55q5a04b60e32409161@mail.gmail.com>
Message-ID: <4A5AD130.9080104@behnel.de>


Jonas wrote:
> I'm trying to parsing a xml file (in python) to convert it to
> dictionaries and then to JSON.
> 
> This is the file:
> http://unicode.org/cldr/data/common/main/es.xml
> and I've got all nodes except dates', which is enought complex.
> 
> Any idea to get it? because I'm very tired of try it.

What's your question?

- how to find the 'dates' element in a parsed tree?
- how to traverse the subtree of the 'dates' element?
- how to extract the data from the parsed tree?
- how to extract the data using iterparse()?
- how to map the data to a dictionary?
- how to map the data to a dictionary while parsing?
- how to map the data to a recursive structure of dictionaries?
- how to find a dictionary mapping that works well with JSON?
- how to find a dictionary mapping that works well in the use case that
  I didn't tell you about?
- ... something else ?

Stefan

From stefan_ml at behnel.de  Mon Jul 13 08:24:15 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 13 Jul 2009 08:24:15 +0200
Subject: [lxml-dev] lxml binary egg redux
In-Reply-To: <05CD06C6-9260-4D5F-BD10-CB87B9B00465@inquant.de>
References: <ea39de390906301704t5e85c943rf680529ef1af4702@mail.gmail.com>
	<ea39de390907050009v68a42334w99acb25247796b9f@mail.gmail.com>
	<b4ef227c0907061322v3bd959b4gd79c87799e6bb89@mail.gmail.com>
	<189D48B6-52BA-4C47-9922-3467060BCAD7@inquant.de>
	<5af21ed10907090716t76324991pc7a4839ff4fdf5c4@mail.gmail.com>
	<ea39de390907120135g21a34722m6516eaa4de94083@mail.gmail.com>
	<CF03F0E3-BD94-48B7-9C7A-8DCB067FA2F4@inquant.de>
	<4A5A398F.5050105@behnel.de>
	<05CD06C6-9260-4D5F-BD10-CB87B9B00465@inquant.de>
Message-ID: <4A5AD30F.8020000@behnel.de>


Stephan Eletzhofer wrote:
> Am 12.07.2009 um 21:29 schrieb Stefan Behnel:
>> Stephan Eletzhofer wrote:
>>> I've built 2.4, 2.5 and 2.6 versions of LXML 2.2.2 and uploaded them.
>>
>> Any reason why the 2.5 egg is so much bigger than the others?
> 
> Hmm:
> 
> $ ls -lah lib.macosx-10.5-i386-2.?/lxml/*.so
> -rwxr-xr-x 1 seletz staff 3,0M 12. Jul 11:09
> lib.macosx-10.5-i386-2.4/lxml/etree.so
> -rwxr-xr-x 1 seletz staff 1,7M 12. Jul 11:09
> lib.macosx-10.5-i386-2.4/lxml/objectify.so
> -rwxr-xr-x 1 seletz staff 5,5M 12. Jul 21:15
> lib.macosx-10.5-i386-2.5/lxml/etree.so
> -rwxr-xr-x 1 seletz staff 3,0M 12. Jul 21:16
> lib.macosx-10.5-i386-2.5/lxml/objectify.so
> -rwxr-xr-x 1 seletz staff 2,8M 12. Jul 21:20
> lib.macosx-10.5-i386-2.6/lxml/etree.so
> -rwxr-xr-x 1 seletz staff 1,5M 12. Jul 21:20
> lib.macosx-10.5-i386-2.6/lxml/objectify.so
> 
> I've built the 2.5 version using the Apple-supplied python 2.5, the
> other ones using the
> macports variants. I've no idea about the sizes, really. :p

I guess the macports builds are platform specific, while the Apple-builds
are fat eggs, i.e. they would work on both PPC and x86.

Given that Apple's platform Python is 2.5, I guess it's ok to provide the
other eggs only for macports. And I also guess that 'fatness' isn't really
a future-proof requirement, either...

Anyway, thanks for the contribution!

Stefan

From p.oberndoerfer at urheberrecht.org  Mon Jul 13 22:56:00 2009
From: p.oberndoerfer at urheberrecht.org (Pascal Oberndoerfer)
Date: Mon, 13 Jul 2009 22:56:00 +0200
Subject: [lxml-dev] building lxml for OS X 10.4 on PPC?
Message-ID: <4A5B9F60.6070108@urheberrecht.org>

Hello,

I am desperately trying to build lxml on a Mac PPC with OS X 10.4. It is
not the most current plattform, but I am stuck with it...

I tried "STATIC_DEPS= true easy_install lxml", which works perfectly on
all Intel machines I have access to. But unfortunately there seems to be
a problem with libiconv on the PPC side.

Has anyone else had success?  I searched the archives for any hints on
building with this combination, but didn't find a solution/suggestions.

Should I try something similar to "./configure --without-iconv"? But how
would I pass this to easy_install? Or maybe try a local static build
including not only libxml2 and libxslt but libiconv as well?

Any help is greatly appreciated!

and sorry for bothering you with these newbee questions.

Pascal

From manu3d at gmail.com  Mon Jul 13 13:35:33 2009
From: manu3d at gmail.com (Emanuele D'Arrigo)
Date: Mon, 13 Jul 2009 12:35:33 +0100
Subject: [lxml-dev] Targeted XSL transformations
Message-ID: <915dc91d0907130435k7aad6e3aq2726b87ad59bd59e@mail.gmail.com>

Hi everybody, newbie here, typing from an unusually sunny UK.

I've been using lxml with great pleasure for a few months now and I'd like
to thank the main authors and the rest of the lxml community for the work
that has gone into it. From my perspective it's a well done API, with many
useful features and it's easy to use.

I was wondering if I could pick this mailing-list's collective brain as my
own seems to be insufficient for the matter at hand.

Let's say I have parsed an xml document, representing xhtml data, into an
ElementTree, and I have applied one or more xsl transformations to add style
information to it.

Now, let's say I want to add a whole subtree with a top-element of class
"sub" to it. The subtree to be added might have been partially styled
already, i.e. because it comes from an xml file that has its own
xml-stylesheet processing instructions. However, some styling can occurr
only in the context of the whole tree, i.e. because the document-wide xsl
transformation file specifically establishes that elements of class "sub"
must have a yellow background. Furthermore, the same, xsl file asserts that
if the document contains an element of class "sub" the document's background
color must be purple rather than blue.

My fundamental question in this context is: how do I avoid re-applying the
xsl transformation to the whole ElementTree and only apply the bits of the
transformation that are necessary, due to the change in the tree?



a Manu typing from a now much more normal, cloudy UK - [sigh]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20090713/f624848b/attachment.htm 

From jonas.esp at googlemail.com  Mon Jul 13 15:46:47 2009
From: jonas.esp at googlemail.com (Jonas)
Date: Mon, 13 Jul 2009 13:46:47 +0000
Subject: [lxml-dev] complex node
In-Reply-To: <4A5AD130.9080104@behnel.de>
References: <20cf13940907121009m5488fd55q5a04b60e32409161@mail.gmail.com>
	<4A5AD130.9080104@behnel.de>
Message-ID: <20cf13940907130646w1ceea04free9f0a938723e933@mail.gmail.com>

2009/7/13 Stefan Behnel <stefan_ml at behnel.de>:
>
> Jonas wrote:
>> I'm trying to parsing a xml file (in python) to convert it to
>> dictionaries and then to JSON.
>>
>> This is the file:
>> http://unicode.org/cldr/data/common/main/es.xml
>> and I've got all nodes except dates', which is enought complex.
>>
>> Any idea to get it? because I'm very tired of try it.
>
> What's your question?
>
> - how to find the 'dates' element in a parsed tree?
> - how to traverse the subtree of the 'dates' element?
> - how to extract the data from the parsed tree?
> - how to extract the data using iterparse()?
> - how to map the data to a dictionary?
> - how to map the data to a dictionary while parsing?
> - how to map the data to a recursive structure of dictionaries?
This is my main problem. But
checking i.e. if each element has attributes to manage it

> - how to find a dictionary mapping that works well with JSON?
> - how to find a dictionary mapping that works well in the use case that
> ?I didn't tell you about?
> - ... something else ?


* For if anybody is interested here there is an awesome conversor from
XML to JSON
http://code.google.com/p/xmltojson/

From mike at it-loops.com  Tue Jul 14 07:46:30 2009
From: mike at it-loops.com (Michael Guntsche)
Date: Tue, 14 Jul 2009 07:46:30 +0200
Subject: [lxml-dev] =?utf-8?q?building_lxml_for_OS_X_10=2E4_on_PPC=3F?=
In-Reply-To: <4A5B9F60.6070108@urheberrecht.org>
References: <4A5B9F60.6070108@urheberrecht.org>
Message-ID: <8d93cb3af9d1da196b2b8b0000336a8d@localhost>

On Mon, 13 Jul 2009 22:56:00 +0200, Pascal Oberndoerfer
<p.oberndoerfer at urheberrecht.org> wrote:
> Hello,
> 
> I am desperately trying to build lxml on a Mac PPC with OS X 10.4. It is
> not the most current plattform, but I am stuck with it...
> 
> I tried "STATIC_DEPS= true easy_install lxml", which works perfectly on
> all Intel machines I have access to. But unfortunately there seems to be
> a problem with libiconv on the PPC side.

I built lxml several months ago on my Tiger G5 without any problems. What
error do you get exactly?


Kind regards,
Michael

From stefan_ml at behnel.de  Tue Jul 14 21:16:55 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 14 Jul 2009 21:16:55 +0200
Subject: [lxml-dev] complex node
In-Reply-To: <20cf13940907130646w1ceea04free9f0a938723e933@mail.gmail.com>
References: <20cf13940907121009m5488fd55q5a04b60e32409161@mail.gmail.com>	
	<4A5AD130.9080104@behnel.de>
	<20cf13940907130646w1ceea04free9f0a938723e933@mail.gmail.com>
Message-ID: <4A5CD9A7.4010304@behnel.de>

Hi,

Jonas wrote:
> 2009/7/13 Stefan Behnel wrote:
>> Jonas wrote:
>>> I'm trying to parsing a xml file (in python) to convert it to
>>> dictionaries and then to JSON.
>>>
>>> This is the file:
>>> http://unicode.org/cldr/data/common/main/es.xml
>>> and I've got all nodes except dates', which is enought complex.
>>>
>>> Any idea to get it? because I'm very tired of try it.
>> What's your question?
>>
>> - how to find the 'dates' element in a parsed tree?
>> - how to traverse the subtree of the 'dates' element?
>> - how to extract the data from the parsed tree?
>> - how to extract the data using iterparse()?
>> - how to map the data to a dictionary?
>> - how to map the data to a dictionary while parsing?
>> - how to map the data to a recursive structure of dictionaries?
>
> This is my main problem. But
> checking i.e. if each element has attributes to manage it

What about providing some more background and detail so that we can
understand the problem you are trying to solve? So far, I have no idea
about the target data structure that you want to construct, or what
information in the XML document you consider important.

This might be worth reading:

http://www.catb.org/~esr/faqs/smart-questions.html

Stefan


From stefan_ml at behnel.de  Tue Jul 14 21:31:04 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 14 Jul 2009 21:31:04 +0200
Subject: [lxml-dev] Targeted XSL transformations
In-Reply-To: <915dc91d0907130435k7aad6e3aq2726b87ad59bd59e@mail.gmail.com>
References: <915dc91d0907130435k7aad6e3aq2726b87ad59bd59e@mail.gmail.com>
Message-ID: <4A5CDCF8.3000602@behnel.de>

Hi,

Emanuele D'Arrigo wrote:
> I've been using lxml with great pleasure for a few months now and I'd like
> to thank the main authors and the rest of the lxml community for the work
> that has gone into it. From my perspective it's a well done API, with many
> useful features and it's easy to use.

Happy to hear that. :)


> Let's say I have parsed an xml document, representing xhtml data, into an
> ElementTree, and I have applied one or more xsl transformations to add style
> information to it.
> 
> Now, let's say I want to add a whole subtree with a top-element of class
> "sub" to it. The subtree to be added might have been partially styled
> already, i.e. because it comes from an xml file that has its own
> xml-stylesheet processing instructions. However, some styling can occurr
> only in the context of the whole tree, i.e. because the document-wide xsl
> transformation file specifically establishes that elements of class "sub"
> must have a yellow background. Furthermore, the same, xsl file asserts that
> if the document contains an element of class "sub" the document's background
> color must be purple rather than blue.
> 
> My fundamental question in this context is: how do I avoid re-applying the
> xsl transformation to the whole ElementTree and only apply the bits of the
> transformation that are necessary, due to the change in the tree?

Hmm, to me, this description contains a bit too many uncertainties. If you
can control the input documents, I'd rather try to make the stylesheets
distinct and apply them in a progressing order on a clean input, instead of
trying to figure out which XSLTs you may have to run on a document.

If you can't do that, you can try to make the XSLTs idempotent, so that
they do not break already styled documents or subtrees. But you should
definitely try to let different stylesheets do independent things.

My preferred solution, however: if you can avoid putting any style
information into the (X)HTML document (or at least the style bits that
interfere in multiple XSL docs), and move it out into a CSS file, I think
you can get yourself out of a lot of hassle. Many of the decisions that you
would take in XSLT to put style information in the right places can be left
to the browser's CSS engine.

Stefan

From lei at ipac.caltech.edu  Tue Jul 14 22:11:25 2009
From: lei at ipac.caltech.edu (Mary Lei)
Date: Tue, 14 Jul 2009 13:11:25 -0700
Subject: [lxml-dev] lxml cssselect
Message-ID: <4A5CE66D.305@ipac.caltech.edu>

Can I use cssselect to select a set of input tags
with image

  <input name="etsstop" value="1" src="nph-bgExec_files/top.png" 
type="image">

for img in root.cssselect('input'):
...


works fine but
for img in root.cssselect('input type="image"'):
...
sends lxml spinnning.

-- 
Mary Lei

Software Testing
IPAC-NExScl

Rm: KS-233
MS: 220-6
Phone: 395-1998


From ted at milo.com  Tue Jul 14 22:16:02 2009
From: ted at milo.com (Ted Dziuba)
Date: Tue, 14 Jul 2009 13:16:02 -0700
Subject: [lxml-dev] lxml cssselect
In-Reply-To: <4A5CE66D.305@ipac.caltech.edu>
References: <4A5CE66D.305@ipac.caltech.edu>
Message-ID: <6451ccbf0907141316v1b7aa0cu82cbaf8c74ee7243@mail.gmail.com>

I don't think that's a valid CSS selector.  Try xpath:

root.xpath("//input[@type = 'image']")

ted

On Tue, Jul 14, 2009 at 1:11 PM, Mary Lei <lei at ipac.caltech.edu> wrote:

> Can I use cssselect to select a set of input tags
> with image
>
>  <input name="etsstop" value="1" src="nph-bgExec_files/top.png"
> type="image">
>
> for img in root.cssselect('input'):
> ...
>
>
> works fine but
> for img in root.cssselect('input type="image"'):
> ...
> sends lxml spinnning.
>
> --
> Mary Lei
>
> Software Testing
> IPAC-NExScl
>
> Rm: KS-233
> MS: 220-6
> Phone: 395-1998
>
> _______________________________________________
> lxml-dev mailing list
> lxml-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/lxml-dev
>



-- 
Ted Dziuba
Co-Founder and Engineer

Milo.com, Inc.
165 University Avenue
Palo Alto, CA, 94301
http://milo.com

Cell: (609)-665-2639
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20090714/9b9e3c43/attachment.htm 

From lostlogic at lostlogicx.com  Wed Jul 15 02:20:20 2009
From: lostlogic at lostlogicx.com (Brandon Low)
Date: Tue, 14 Jul 2009 17:20:20 -0700
Subject: [lxml-dev] lxml cssselect
In-Reply-To: <4A5CE66D.305@ipac.caltech.edu>
References: <4A5CE66D.305@ipac.caltech.edu>
Message-ID: <20090715002020.GS1135@lostlogicx.com>

root.cssselect('input[type="image"]')

is the syntax you need

--Brandon

On 2009-07-14 (Tue) at 13:11:25 -0700, Mary Lei wrote:
> Can I use cssselect to select a set of input tags
> with image
> 
>   <input name="etsstop" value="1" src="nph-bgExec_files/top.png" 
> type="image">
> 
> for img in root.cssselect('input'):
> ...
> 
> 
> works fine but
> for img in root.cssselect('input type="image"'):
> ...
> sends lxml spinnning.
> 
> -- 
> Mary Lei
> 
> Software Testing
> IPAC-NExScl
> 
> Rm: KS-233
> MS: 220-6
> Phone: 395-1998
> 
> _______________________________________________
> lxml-dev mailing list
> lxml-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/lxml-dev

From stefan_ml at behnel.de  Wed Jul 15 08:36:56 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 15 Jul 2009 08:36:56 +0200
Subject: [lxml-dev] lxml cssselect
In-Reply-To: <20090715002020.GS1135@lostlogicx.com>
References: <4A5CE66D.305@ipac.caltech.edu>
	<20090715002020.GS1135@lostlogicx.com>
Message-ID: <4A5D7908.7050900@behnel.de>

Hi,

Brandon Low wrote:
> On 2009-07-14 (Tue) at 13:11:25 -0700, Mary Lei wrote:
>> Can I use cssselect to select a set of input tags
>> with image
>>
>>   <input name="etsstop" value="1" src="nph-bgExec_files/top.png" 
>> type="image">
>>
>> for img in root.cssselect('input'):
>> ...
>>
>> works fine but
>> for img in root.cssselect('input type="image"'):
>> ...
>> sends lxml spinnning.
>
> root.cssselect('input[type="image"]')
>
> is the syntax you need

Right, here are the docs:

http://www.w3.org/TR/CSS2/selector.html#matching-attrs

Stefan


From manu3d at gmail.com  Thu Jul 16 01:08:51 2009
From: manu3d at gmail.com (Emanuele D'Arrigo)
Date: Thu, 16 Jul 2009 00:08:51 +0100
Subject: [lxml-dev] Targeted XSL transformations
In-Reply-To: <4A5CDCF8.3000602@behnel.de>
References: <915dc91d0907130435k7aad6e3aq2726b87ad59bd59e@mail.gmail.com> 
	<4A5CDCF8.3000602@behnel.de>
Message-ID: <915dc91d0907151608o28747c4di7cd58334888be195@mail.gmail.com>

Stefan, thank you for your reply.

2009/7/14 Stefan Behnel <stefan_ml at behnel.de>

>  > My fundamental question in this context is: how do I avoid re-applying
> the
> > xsl transformation to the whole ElementTree and only apply the bits of
> the
> > transformation that are necessary, due to the change in the tree?
>
> Hmm, to me, this description contains a bit too many uncertainties. If you
> can control the input documents, I'd rather try to make the stylesheets
> distinct and apply them in a progressing order on a clean input, instead of
> trying to figure out which XSLTs you may have to run on a document.


Hmm... the intention IS to apply distinct stylesheets in progressing order
on a clean input. But eventually that input must be used in the application
and that's when the application might add pieces that require styling and
might change the style of other elements that are already in the tree.

I.e., suppose I'm starting with just a node <aRoot /> and an xslt file that
gives the attribute color="blue" to it and the attribute color="orange" to
any of its children - when any exists. After the initial transformation I
then have <aRoot color="blue" />. Now, let's suppose the application then
does the following:

1) changes the color attribute of <aRoot /> from blue to red.
2) adds to <aRoot /> an <aChild /> node that needs to be transformed
according to the initial xslt file.

If I now apply the initial xslt to the whole tree the child node will
correctly acquire its orange color attribute, but at the same time the root
node will go back to its initial blue color when it shouldn't be touched.
-Ideally- there should be a way to apply only the transformations that are
due to the addition of the child element. This might very well be impossible
or impractical. I'm just asking if that's the case.

If you can't do that, you can try to make the XSLTs idempotent, so that
> they do not break already styled documents or subtrees. But you should
> definitely try to let different stylesheets do independent things.


Uhm... I think I understand what you mean. In the example above you are
talking about splitting the stylesheet so that the initial stylesheet is
applied and then, only when a child is added, a second stylesheet is applied
bringing only transformations that are related to the addition of the child.
Hmmm... I wonder if this can be made to work in all circumstances. I.e. in
the example above if I apply the same orange-coloring stylesheet every time
I add a child I might end up overwriting the color of children that in the
meantime has been changed from the default orange.


> My preferred solution, however: if you can avoid putting any style
> information into the (X)HTML document (or at least the style bits that
> interfere in multiple XSL docs), and move it out into a CSS file, I think
> you can get yourself out of a lot of hassle. Many of the decisions that you
> would take in XSLT to put style information in the right places can be left
> to the browser's CSS engine.


I wish I could leave that to something indeed rather than having to
implement it myself. Unfortunately, although my original post referred to
xhtml, in practice I'm not dealing with a browser nor with xhtml. Instead
I'm attempting to write a GUI system based on Mozilla's XUL.

I initially thought of using CSS but, I seem to understand, lxml only allows
me to -select- elements in the tree via css. This is quite short from
generating the cascade of css rules and transforming the tree accordingly.
For this reason I'm currently oriented toward XSLT: once a stylesheet is
ready it's quite easy to create a transform object out of it and apply it to
an xml object.

What do you think?

Manu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20090716/5eedb658/attachment.htm 

From manu3d at gmail.com  Thu Jul 16 01:15:38 2009
From: manu3d at gmail.com (Emanuele D'Arrigo)
Date: Thu, 16 Jul 2009 00:15:38 +0100
Subject: [lxml-dev] objectify and processing instructions
Message-ID: <915dc91d0907151615q76dc9363yb4135a7d36e5210d@mail.gmail.com>

Hi everybody,

having a piece of xml such as:

<?xsl-transform href="test.xsl" type="text/xsl"?>
<aRoot />

I can use:

xmlObject = objectify.fromstring()
piElement = xmlObject.getprevious()

but the returned object (differently from an xml object generated with
etree.fromstring()), is not a processing instruction but an
ObjectifiedElement, and doesn't seem to give me access to the href and type
attributes. I'm aware that they are actually a single pseudo-attribute, but
how do I access it?

Manu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20090716/599f8520/attachment.htm 

From qhlonline at 163.com  Thu Jul 16 07:06:04 2009
From: qhlonline at 163.com (qhlonline)
Date: Thu, 16 Jul 2009 13:06:04 +0800 (CST)
Subject: [lxml-dev] About the position of html parsing by HTML Target parser
Message-ID: <2514262.666501247720764592.JavaMail.coremail@bj163app16.163.com>

Hi, all
  I am parsing html files with lxml target parser, now I wan't to know when I have reached some HTML tag, how can I know the position of the HTML document I am parsing? Is there any callbacks in target parser who can tell me the total stream length I have parsed? 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20090716/9efbea4a/attachment.htm 

From stefan_ml at behnel.de  Thu Jul 16 07:11:30 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 16 Jul 2009 07:11:30 +0200
Subject: [lxml-dev] Targeted XSL transformations
In-Reply-To: <915dc91d0907151608o28747c4di7cd58334888be195@mail.gmail.com>
References: <915dc91d0907130435k7aad6e3aq2726b87ad59bd59e@mail.gmail.com>
	<4A5CDCF8.3000602@behnel.de>
	<915dc91d0907151608o28747c4di7cd58334888be195@mail.gmail.com>
Message-ID: <4A5EB682.6030501@behnel.de>

Hi,

just a quick answer to the last bit (skipping over your mail somewhat).

Emanuele D'Arrigo wrote:
> 2009/7/14 Stefan Behnel wrote:
>> My preferred solution, however: if you can avoid putting any style
>> information into the (X)HTML document (or at least the style bits that
>> interfere in multiple XSL docs), and move it out into a CSS file, I think
>> you can get yourself out of a lot of hassle. Many of the decisions that you
>> would take in XSLT to put style information in the right places can be left
>> to the browser's CSS engine.
> 
> I wish I could leave that to something indeed rather than having to
> implement it myself. Unfortunately, although my original post referred to
> xhtml, in practice I'm not dealing with a browser nor with xhtml. Instead
> I'm attempting to write a GUI system based on Mozilla's XUL.

How does that discourage CSS?


> I initially thought of using CSS but, I seem to understand, lxml only allows
> me to -select- elements in the tree via css. This is quite short from
> generating the cascade of css rules and transforming the tree accordingly.

I didn't say "generate" them - just write them down. There's also a package
called "cssutils" which you might (or might not) find useful for your needs.

http://cthedot.de/cssutils/


> For this reason I'm currently oriented toward XSLT: once a stylesheet is
> ready it's quite easy to create a transform object out of it and apply it to
> an xml object.

It's common to use both: XSLT for structural tree transformations, and CSS
for applying style information. That keeps the details of content structure
and visual representation somewhat separated, which tends to be a rather
big advantage.

Stefan

From stefan_ml at behnel.de  Thu Jul 16 07:14:40 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 16 Jul 2009 07:14:40 +0200
Subject: [lxml-dev] objectify and processing instructions
In-Reply-To: <915dc91d0907151615q76dc9363yb4135a7d36e5210d@mail.gmail.com>
References: <915dc91d0907151615q76dc9363yb4135a7d36e5210d@mail.gmail.com>
Message-ID: <4A5EB740.50305@behnel.de>

Hi,

Emanuele D'Arrigo wrote:
> having a piece of xml such as:
> 
> <?xsl-transform href="test.xsl" type="text/xsl"?>
> <aRoot />
> 
> I can use:
> 
> xmlObject = objectify.fromstring()
> piElement = xmlObject.getprevious()
> 
> but the returned object (differently from an xml object generated with
> etree.fromstring()), is not a processing instruction but an
> ObjectifiedElement, and doesn't seem to give me access to the href and type
> attributes. I'm aware that they are actually a single pseudo-attribute, but
> how do I access it?

Hmmm, without looking into it myself, this sounds like a bug to me. It
shouldn't return an ObjectifiedElement for Processing instructions, a
normal etree PI element should work here (given that there are no children,
for example).

Stefan

From dglick at gmail.com  Thu Jul 16 08:28:00 2009
From: dglick at gmail.com (David Glick)
Date: Thu, 16 Jul 2009 06:28:00 +0000 (UTC)
Subject: [lxml-dev] binary egg problems with OSX 10.5 / official Python Mac
	Installer
Message-ID: <loom.20090716T061929-612@post.gmane.org>

I just tried out the OSX binary egg of lxml 2.2.2 for the first time. When I
try to import lxml.etree, I get the following traceback:

 from lxml import etree
ImportError:
dlopen(/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-pac
kages/lxml-2.2.2-py2.6-macosx-10.5-i386.egg/lxml/etree.so,
2): Library not loaded: /opt/local/lib/libiconv.2.dylib
  Referenced from:
/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/l
xml-2.2.2-py2.6-macosx-10.5-i386.egg/lxml/etree.so
  Reason: Incompatible library version: etree.so requires version 8.0.0 or
later, but libiconv.2.dylib provides version 7.0.0

I presume this is because I'm using Python 2.6 from the official Python Mac
Installer rather than from macports, so I don't have the right version of
libiconv in my macports directory where the binary egg is trying to link
against it. Does that sound like an accurate diagnosis? Does the binary egg
require installing Python 2.6 via macports?

I'm working around this by building the egg myself, but wanted to make sure
you're aware of it.

peace,
David Glick


From manu3d at gmail.com  Thu Jul 16 10:45:45 2009
From: manu3d at gmail.com (Emanuele D'Arrigo)
Date: Thu, 16 Jul 2009 09:45:45 +0100
Subject: [lxml-dev] Targeted XSL transformations
In-Reply-To: <4A5EB682.6030501@behnel.de>
References: <915dc91d0907130435k7aad6e3aq2726b87ad59bd59e@mail.gmail.com> 
	<4A5CDCF8.3000602@behnel.de>
	<915dc91d0907151608o28747c4di7cd58334888be195@mail.gmail.com> 
	<4A5EB682.6030501@behnel.de>
Message-ID: <915dc91d0907160145s43bb854dy893cc754412238b0@mail.gmail.com>

2009/7/16 Stefan Behnel <stefan_ml at behnel.de>

> (skipping over your mail somewhat).


Oh, thank you for that! =D


> Emanuele D'Arrigo wrote:
> > I wish I could leave that to something indeed rather than having to
> > implement it myself. Unfortunately, although my original post referred to
> > xhtml, in practice I'm not dealing with a browser nor with xhtml. Instead
> > I'm attempting to write a GUI system based on Mozilla's XUL.
>
> How does that discourage CSS?


It isn't XUL or me working on a GUI system that does. But lxml does not have
an etree.CSS() function which generates a CSS transformation, does it?
However, it does have an etree.XSLT() function which generates an xslt
transformation. Isn't it?

I didn't say "generate" them - just write them down. There's also a package
> called "cssutils" which you might (or might not) find useful for your
> needs.
>
> http://cthedot.de/cssutils/


Thank you for this! I did have a look and it has all sorts of nice things.
Still, it doesn't seem to have functionality ordering the CSS rules
according to the cascade nor there is functionality to apply all those rules
at once to an xml/xhtml object. Does it?

It's common to use both: XSLT for structural tree transformations, and CSS
> for applying style information. That keeps the details of content structure
> and visual representation somewhat separated, which tends to be a rather
> big advantage.


Totally agree. But in my case the content structure is already separate from
the visual representation. It is stored in a XUL file with no style
information. Only when I apply the xsl transformations XUL content gains
style (and locale, in a separate xsl transformation) information.

I somehow suspect I wasn't particularly clear in my explanation of the
problem. Was I?

Manu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20090716/4156f745/attachment.htm 

From stefan_ml at behnel.de  Thu Jul 16 11:41:04 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 16 Jul 2009 11:41:04 +0200
Subject: [lxml-dev] Targeted XSL transformations
In-Reply-To: <915dc91d0907160145s43bb854dy893cc754412238b0@mail.gmail.com>
References: <915dc91d0907130435k7aad6e3aq2726b87ad59bd59e@mail.gmail.com>
	<4A5CDCF8.3000602@behnel.de>
	<915dc91d0907151608o28747c4di7cd58334888be195@mail.gmail.com>
	<4A5EB682.6030501@behnel.de>
	<915dc91d0907160145s43bb854dy893cc754412238b0@mail.gmail.com>
Message-ID: <4A5EF5B0.3070507@behnel.de>


Emanuele D'Arrigo wrote:
> 2009/7/16 Stefan Behnel wrote:
>> Emanuele D'Arrigo wrote:
>>> I wish I could leave that to something indeed rather than having to
>>> implement it myself. Unfortunately, although my original post referred to
>>> xhtml, in practice I'm not dealing with a browser nor with xhtml. Instead
>>> I'm attempting to write a GUI system based on Mozilla's XUL.
>>
>> How does that discourage CSS?
> 
> It isn't XUL or me working on a GUI system that does. But lxml does not have
> an etree.CSS() function which generates a CSS transformation, does it?
> However, it does have an etree.XSLT() function which generates an xslt
> transformation. Isn't it?

That's what I meant: you don't need that. CSS doesn't do tree
transformations, it only cares about layout and 'visual effects'. The idea
is to build up a document that contains the complete content, and then
associate a CSS file with it that defines where the different parts of the
content appear and how they should look like (colour, fonts, etc.).

You can use XSLT to aggregate the content into a document, but you'd write
up a CSS file manually (or with a suitable CSS editor or whatever) to
define the visual layout of the presentation. Do you have a need to do that
programmatically?

Stefan


From manu3d at gmail.com  Thu Jul 16 12:54:53 2009
From: manu3d at gmail.com (Emanuele D'Arrigo)
Date: Thu, 16 Jul 2009 11:54:53 +0100
Subject: [lxml-dev] Targeted XSL transformations
In-Reply-To: <4A5EF5B0.3070507@behnel.de>
References: <915dc91d0907130435k7aad6e3aq2726b87ad59bd59e@mail.gmail.com> 
	<4A5CDCF8.3000602@behnel.de>
	<915dc91d0907151608o28747c4di7cd58334888be195@mail.gmail.com> 
	<4A5EB682.6030501@behnel.de>
	<915dc91d0907160145s43bb854dy893cc754412238b0@mail.gmail.com> 
	<4A5EF5B0.3070507@behnel.de>
Message-ID: <915dc91d0907160354j7e145213i93bd33ff1e632d1@mail.gmail.com>

2009/7/16 Stefan Behnel <stefan_ml at behnel.de>

>  > It isn't XUL or me working on a GUI system that does. But lxml does not
> have
> > an etree.CSS() function which generates a CSS transformation, does it?
> > However, it does have an etree.XSLT() function which generates an xslt
> > transformation. Isn't it?
>
> That's what I meant: you don't need that. CSS doesn't do tree
> transformations, it only cares about layout and 'visual effects'. The idea
> is to build up a document that contains the complete content, and then
> associate a CSS file with it that defines where the different parts of the
> content appear and how they should look like (colour, fonts, etc.).


Hmmm. I might be starting to grasp what you are suggesting. You are
suggesting to load the xml/xul content separately from the css information
and associate the two together only as needed, inside the application, as I
build the actual objects that are described by the xul content and are
eventually displayed to screen. This effectively means that the css doesn't
act on the xml/xul content file nor the generated ElementTree, but instead
it acts directly on the display objects that are created out of it. It acts
on the objects that are eventually displayed rather than the xml objects
generated by etree or objectify. Hmmm. That means I'd have to create my own
css selectors that act on the display objects rather than the elements in an
ElementTree object. And of course I'd have to call upon the right selectors
as the relationships of the display objects changes - the original problem I
am facing even with xslt providing style information. Right now I can't
quite think how to do that. But it's certainly something worth investigating
further. Thank you!


> You can use XSLT to aggregate the content into a document, but you'd write
> up a CSS file manually (or with a suitable CSS editor or whatever) to
> define the visual layout of the presentation. Do you have a need to do that
> programmatically?


Programmatically what I need is to be able to add composite GUI elements to
an interface. I.e. a composite GUI element might have text labels, text
input fields and buttons. The description of the element in terms of its
sub-components is in a XUL file but the style is dependent on 1) some
default style information 2) any overriding style information that has been
added or modified at runtime, i.e. by the user.

When I add a composite GUI element to an interface (i.e. a new record in a
table) it's easy enough to add it to the tree of objects in the GUI, but in
terms of style I'd need to find out which style information applies to it
given the nature of the added element and its location in the tree. In a
sense, rather than selecting the element given a (css) rule, I'd need to
select the applicable rules from the element and its position.

Furthermore I'd need to find a way to apply any rule that the addition,
removal or modification of an element has made applicable to other elements.
I.e. the rule E F {color:red} does not apply until the element E has an
element F as a child. When I add that child I must somehow retrieve that
rule and apply it to E. Right now I don't quite know how to do that.

But I realise I am digressing into non-lxml related issues. I apologize if
I'm abusing of your and other's patience. Admittedly there are various parts
of the problem statement that remain unclear even to me. Attempting to
explain it to you and the list does help a little, hence my potentially
unwelcome verbosity.

Thank you again.

Manu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20090716/b0fd855e/attachment.htm 

From stefan_ml at behnel.de  Thu Jul 16 13:31:39 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 16 Jul 2009 13:31:39 +0200
Subject: [lxml-dev] Targeted XSL transformations
In-Reply-To: <915dc91d0907160354j7e145213i93bd33ff1e632d1@mail.gmail.com>
References: <915dc91d0907130435k7aad6e3aq2726b87ad59bd59e@mail.gmail.com>
	<4A5CDCF8.3000602@behnel.de>	<915dc91d0907151608o28747c4di7cd58334888be195@mail.gmail.com>
	<4A5EB682.6030501@behnel.de>	<915dc91d0907160145s43bb854dy893cc754412238b0@mail.gmail.com>
	<4A5EF5B0.3070507@behnel.de>
	<915dc91d0907160354j7e145213i93bd33ff1e632d1@mail.gmail.com>
Message-ID: <4A5F0F9B.8090601@behnel.de>


Emanuele D'Arrigo wrote:
> Hmmm. I might be starting to grasp what you are suggesting. You are
> suggesting to load the xml/xul content separately from the css information
> and associate the two together only as needed, inside the application, as I
> build the actual objects that are described by the xul content and are
> eventually displayed to screen. This effectively means that the css doesn't
> act on the xml/xul content file nor the generated ElementTree, but instead
> it acts directly on the display objects that are created out of it. It acts
> on the objects that are eventually displayed rather than the xml objects
> generated by etree or objectify.

Sort of, yes.


> Hmmm. That means I'd have to create my own
> css selectors that act on the display objects rather than the elements in an
> ElementTree object.

What? Why? I thought you used the Mozilla engine for displaying the XUL
pages? They are perfectly capable of handling CSS. Actually, CSS is the
only way to map style information onto a XUL document. I'm only suggesting
to put it into a separate file rather than into the XUL document itself.

Asking Google for "xul css" immediately brought up this, for example,
although there may well be some better tutorials out there:

https://developer.mozilla.org/en/introduction_to_xul


> Programmatically what I need is to be able to add composite GUI elements to
> an interface. I.e. a composite GUI element might have text labels, text
> input fields and buttons. The description of the element in terms of its
> sub-components is in a XUL file but the style is dependent on 1) some
> default style information 2) any overriding style information that has been
> added or modified at runtime, i.e. by the user.
>
> When I add a composite GUI element to an interface (i.e. a new record in a
> table) it's easy enough to add it to the tree of objects in the GUI, but in
> terms of style I'd need to find out which style information applies to it
> given the nature of the added element and its location in the tree. In a
> sense, rather than selecting the element given a (css) rule, I'd need to
> select the applicable rules from the element and its position.

And that needs to be decided at runtime? Aren't there any general rules
like "when element X appears within the panel named Y, give it style Z"?

What you can generate with XSLT (or using whatever type of tree
manipulation) is the element X, which now appears in the panel Y and thus
picks up the style that was predefined in the CSS file for exactly this
case: "Y > X { ... }".

I doubt that you really have requirements that you cannot express
statically (i.e. in advance) using CSS selectors. If you really need to
distinguish identical elements depending on some logic in the XSLT driven
document generation process, just give them different IDs and use those in
the CSS file.


> Furthermore I'd need to find a way to apply any rule that the addition,
> removal or modification of an element has made applicable to other elements.
> I.e. the rule E F {color:red} does not apply until the element E has an
> element F as a child. When I add that child I must somehow retrieve that
> rule and apply it to E. Right now I don't quite know how to do that.

That's exactly what the CSS engine is there for. Keep moving, nothing to do
here.


> But I realise I am digressing into non-lxml related issues. I apologize if
> I'm abusing of your and other's patience. Admittedly there are various parts
> of the problem statement that remain unclear even to me. Attempting to
> explain it to you and the list does help a little, hence my potentially
> unwelcome verbosity.

What I think you need to understand is the difference between the document
status of an element (e.g. descendant of Z, child of X, @id="Y",
@enabled="true") and the visual representation (visible, green, large,
...). The mapping between those two is what CSS is designed for. And the
CSS engine that sits right inside your XUL layout engine will do that
mapping for you whenever either the document or the style information changes.

Stefan


From manu3d at gmail.com  Thu Jul 16 14:10:27 2009
From: manu3d at gmail.com (Emanuele D'Arrigo)
Date: Thu, 16 Jul 2009 13:10:27 +0100
Subject: [lxml-dev] Targeted XSL transformations
In-Reply-To: <4A5F0F9B.8090601@behnel.de>
References: <915dc91d0907130435k7aad6e3aq2726b87ad59bd59e@mail.gmail.com> 
	<4A5CDCF8.3000602@behnel.de>
	<915dc91d0907151608o28747c4di7cd58334888be195@mail.gmail.com> 
	<4A5EB682.6030501@behnel.de>
	<915dc91d0907160145s43bb854dy893cc754412238b0@mail.gmail.com> 
	<4A5EF5B0.3070507@behnel.de>
	<915dc91d0907160354j7e145213i93bd33ff1e632d1@mail.gmail.com> 
	<4A5F0F9B.8090601@behnel.de>
Message-ID: <915dc91d0907160510t22df6264m437be462ac332ea1@mail.gmail.com>

2009/7/16 Stefan Behnel <stefan_ml at behnel.de>

> What? Why? I thought you used the Mozilla engine for displaying the XUL
> pages?


Ahem. No. I am using the XUL -format- as a description of interfaces. But
I'm implementing my own layout engine in a completely different rendering
framework (Panda3d <http:/www.panda3d.com>). Sorry if this wasn't clear from
my previous messages.

> When I add a composite GUI element to an interface (i.e. a new record in a
> > table) it's easy enough to add it to the tree of objects in the GUI, but
> in
> > terms of style I'd need to find out which style information applies to it
> > given the nature of the added element and its location in the tree. In a
> > sense, rather than selecting the element given a (css) rule, I'd need to
> > select the applicable rules from the element and its position.
>
> And that needs to be decided at runtime? Aren't there any general rules
> like "when element X appears within the panel named Y, give it style Z"?


Not only. Some rules are applicable to an element even though they are
specific to that element. I.e. ".myClass" applies to any element of class
"myClass". And that might not be that difficult to find. But there are some
rules that specify that IF element Y has a child element X you should apply
style Z (to Y, not X). To find those rules sounds a little harder.



> I doubt that you really have requirements that you cannot express
> statically (i.e. in advance) using CSS selectors. If you really need to
> distinguish identical elements depending on some logic in the XSLT driven
> document generation process, just give them different IDs and use those in
> the CSS file.


Yes, true, I might be attempting to cover all cases programmatically when
some sensible use of css/xslt might simplify the implementation.

 What I think you need to understand is the difference between the document
> status of an element (e.g. descendant of Z, child of X, @id="Y",
> @enabled="true") and the visual representation (visible, green, large,
> ...). The mapping between those two is what CSS is designed for. And the
> CSS engine that sits right inside your XUL layout engine will do that
> mapping for you whenever either the document or the style information
> changes.


Except, as I mentioned at the very beginning and is the crux of the
discussion, I do not have a css engine nor a xul layout engine. I am making
those more or less from scratch.

Manu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20090716/bb5e5bbc/attachment-0001.htm 

From qhlonline at 163.com  Thu Jul 16 15:31:10 2009
From: qhlonline at 163.com (qhlonline)
Date: Thu, 16 Jul 2009 21:31:10 +0800 (CST)
Subject: [lxml-dev] About lxml Target parser
Message-ID: <27417447.901721247751070945.JavaMail.coremail@bj163app72.163.com>

Hi ,all
   I have read part of the source file Saxparser.pxi, May be the two classes named _SaxParserTarget and _SaxParserContext have close relationship with Target Parser. A function _handleSaxData has been defined to deal with HTML data part. But why the data returned is only the text part between tags? I wan't to know how much characters of the HTML file have been parsed with the target parser, when a "start_element" event comes.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20090716/41226d05/attachment.htm 

From dfedoruk at gmail.com  Thu Jul 16 16:22:28 2009
From: dfedoruk at gmail.com (Dmitri Fedoruk)
Date: Thu, 16 Jul 2009 18:22:28 +0400
Subject: [lxml-dev] bizarre crashes under FreeBSD
In-Reply-To: <4A56C848.2030202@behnel.de>
References: <4A56C848.2030202@behnel.de>
Message-ID: <a36e1d4b0907160722s3c52cba9xa7bb04f1f088730d@mail.gmail.com>

Greetings,

> Does anyone have experience with threading on FreeBSD? Any ideas where
> these problems may arise from? Could anyone test the attached test script
> on other FreeBSD systems?
So far I can only reproduce this:

FreeBSD 7.0-RELEASE i386
lxml.etree:        (2, 2, -199, 59835)
libxml used:       (2, 6, 32)
libxml compiled:   (2, 6, 30)
libxslt used:      (1, 1, 22)
libxslt compiled:  (1, 1, 22)

Coredump with the following backtrace:
#0  0x2853eb1b in __pyx_f_4lxml_5etree__forwardError ()
   from /usr/local/lib/python2.5/site-packages/lxml-2.2alpha1-py2.5-freebsd-7.0-RELEASE-i386.egg/lxml/etree.so
[New Thread 0x28302b00 (LWP 100819)]
[New Thread 0x28302a00 (LWP 100646)]
[New Thread 0x28302900 (LWP 100572)]
[New Thread 0x28302800 (LWP 100561)]
[New Thread 0x28302700 (LWP 100524)]
[New Thread 0x28302600 (LWP 100522)]
[New Thread 0x28302500 (LWP 100500)]
[New Thread 0x28302400 (LWP 100444)]
[New Thread 0x28302300 (LWP 100335)]
[New Thread 0x28302200 (LWP 100254)]
[New Thread 0x28301100 (LWP 100182)]
(gdb) bt
#0  0x2853eb1b in __pyx_f_4lxml_5etree__forwardError ()
   from /usr/local/lib/python2.5/site-packages/lxml-2.2alpha1-py2.5-freebsd-7.0-RELEASE-i386.egg/lxml/etree.so
#1  0x28641699 in __xmlRaiseError () from /usr/local/lib/libxml2.so.5
#2  0x2867dc1b in htmlParseErr () from /usr/local/lib/libxml2.so.5
#3  0x2867fe3d in htmlParseEndTag () from /usr/local/lib/libxml2.so.5
#4  0x28684de0 in htmlParseContent () from /usr/local/lib/libxml2.so.5
#5  0x286849a9 in htmlParseElement () from /usr/local/lib/libxml2.so.5
#6  0x28684e3e in htmlParseContent () from /usr/local/lib/libxml2.so.5
#7  0x286849a9 in htmlParseElement () from /usr/local/lib/libxml2.so.5
#8  0x28684e3e in htmlParseContent () from /usr/local/lib/libxml2.so.5
#9  0x286849a9 in htmlParseElement () from /usr/local/lib/libxml2.so.5
#10 0x28684e3e in htmlParseContent () from /usr/local/lib/libxml2.so.5
#11 0x286849a9 in htmlParseElement () from /usr/local/lib/libxml2.so.5
#12 0x28684e3e in htmlParseContent () from /usr/local/lib/libxml2.so.5
#13 0x28685299 in htmlParseDocument () from /usr/local/lib/libxml2.so.5
#14 0x28685460 in htmlDoRead () from /usr/local/lib/libxml2.so.5
#15 0x285ac36b in __pyx_f_4lxml_5etree_11_BaseParser__parseDoc ()
   from /usr/local/lib/python2.5/site-packages/lxml-2.2alpha1-py2.5-freebsd-7.0-RELEASE-i386.egg/lxml/etree.so
#16 0x285b128b in __pyx_f_4lxml_5etree__parseDoc ()
   from /usr/local/lib/python2.5/site-packages/lxml-2.2alpha1-py2.5-freebsd-7.0-RELEASE-i386.egg/lxml/etree.so
#17 0x285b24f2 in __pyx_f_4lxml_5etree__parseMemoryDocument ()
   from /usr/local/lib/python2.5/site-packages/lxml-2.2alpha1-py2.5-freebsd-7.0-RELEASE-i386.egg/lxml/etree.so
#18 0x285b28b3 in __pyx_pf_4lxml_5etree_fromstring ()
   from /usr/local/lib/python2.5/site-packages/lxml-2.2alpha1-py2.5-freebsd-7.0-RELEASE-i386.egg/lxml/etree.so
#19 0x08059cd7 in PyObject_Call ()
#20 0x080af811 in PyEval_EvalFrameEx ()
#21 0x080b21c9 in PyEval_EvalCodeEx ()
#22 0x080edd5c in PyClassMethod_New ()
#23 0x08059cd7 in PyObject_Call ()
#24 0x080af811 in PyEval_EvalFrameEx ()
#25 0x080b21c9 in PyEval_EvalCodeEx ()
#26 0x080b0cc0 in PyEval_EvalFrameEx ()
#27 0x080b1858 in PyEval_EvalFrameEx ()
#28 0x080b21c9 in PyEval_EvalCodeEx ()
#29 0x080eddd1 in PyClassMethod_New ()
#30 0x08059cd7 in PyObject_Call ()
#31 0x0805f111 in PyClass_IsSubclass ()
#32 0x08059cd7 in PyObject_Call ()
#33 0x080abddc in PyEval_CallObjectWithKeywords ()
#34 0x080d3f38 in initthread ()
#35 0x28174b1f in pthread_getprio () from /lib/libthr.so.3

Another machine, FreeBSD 7.0 amd64,
lxml.etree:        (2, 0, 7, 0)
libxml used:       (2, 6, 32)
ml compiled:   (2, 6, 32)
libxslt used:      (1, 1, 24)
libxslt compiled:  (1, 1, 24) :

#0  0x00000000004394d5 in PyObject_GetAttr ()
[New Thread 0x705780 (LWP 100540)]
[New Thread 0x705600 (LWP 100539)]
[New Thread 0x705180 (LWP 100538)]
[New Thread 0x705000 (LWP 100537)]
[New Thread 0x704d00 (LWP 100536)]
[New Thread 0x704e80 (LWP 100535)]
[New Thread 0x705900 (LWP 100534)]
[New Thread 0x705a80 (LWP 100533)]
[New Thread 0x705480 (LWP 100532)]
[New Thread 0x705300 (LWP 100193)]
[New Thread 0x702180 (LWP 101248)]
(gdb)  bt
#0  0x00000000004394d5 in PyObject_GetAttr ()
#1  0x0000000800ecd2a3 in __pyx_f_4lxml_5etree_13_BaseErrorLog ()
   from /usr/local/lib/python2.5/site-packages/lxml-2.0.7-py2.5-freebsd-7.0-20081001-SNAP-amd64.egg/lxml/etree.so
#2  0x0000000800ed8b20 in __pyx_f_4lxml_5etree__forwardError ()
   from /usr/local/lib/python2.5/site-packages/lxml-2.0.7-py2.5-freebsd-7.0-20081001-SNAP-amd64.egg/lxml/etree.so
#3  0x0000000801305910 in __xmlRaiseError () from /usr/local/lib/libxml2.so.5
#4  0x000000080133e3a6 in htmlParseErr () from /usr/local/lib/libxml2.so.5
#5  0x00000008013404b0 in htmlParseEndTag () from /usr/local/lib/libxml2.so.5
#6  0x00000008013452c2 in htmlParseContent () from /usr/local/lib/libxml2.so.5
#7  0x0000000801344f8f in htmlParseElement () from /usr/local/lib/libxml2.so.5
#8  0x0000000801345320 in htmlParseContent () from /usr/local/lib/libxml2.so.5
#9  0x0000000801344f8f in htmlParseElement () from /usr/local/lib/libxml2.so.5
#10 0x0000000801345320 in htmlParseContent () from /usr/local/lib/libxml2.so.5
#11 0x0000000801344f8f in htmlParseElement () from /usr/local/lib/libxml2.so.5
#12 0x0000000801345320 in htmlParseContent () from /usr/local/lib/libxml2.so.5
#13 0x0000000801344f8f in htmlParseElement () from /usr/local/lib/libxml2.so.5
#14 0x0000000801345320 in htmlParseContent () from /usr/local/lib/libxml2.so.5
#15 0x00000008013457bf in htmlParseDocument () from /usr/local/lib/libxml2.so.5
#16 0x000000080134596c in htmlDoRead () from /usr/local/lib/libxml2.so.5
#17 0x0000000800edeb44 in __pyx_f_4lxml_5etree_11_BaseParser__parseDoc ()
   from /usr/local/lib/python2.5/site-packages/lxml-2.0.7-py2.5-freebsd-7.0-20081001-SNAP-amd64.egg/lxml/etree.so
#18 0x0000000800f22fba in __pyx_f_4lxml_5etree__parseDoc ()
   from /usr/local/lib/python2.5/site-packages/lxml-2.0.7-py2.5-freebsd-7.0-20081001-SNAP-amd64.egg/lxml/etree.so
#19 0x0000000800f24181 in __pyx_f_4lxml_5etree__parseMemoryDocument ()
   from /usr/local/lib/python2.5/site-packages/lxml-2.0.7-py2.5-freebsd-7.0-20081001-SNAP-amd64.egg/lxml/etree.so
#20 0x0000000800f28e75 in __pyx_pf_4lxml_5etree_HTML ()
   from /usr/local/lib/python2.5/site-packages/lxml-2.0.7-py2.5-freebsd-7.0-20081001-SNAP-amd64.egg/lxml/etree.so
#21 0x0000000000415173 in PyObject_Call ()
#22 0x000000000046e5f0 in PyEval_EvalFrameEx ()
#23 0x00000000004710ac in PyEval_EvalCodeEx ()
#24 0x00000000004af191 in PyClassMethod_New ()
#25 0x0000000000415173 in PyObject_Call ()
#26 0x000000000046e5f0 in PyEval_EvalFrameEx ()
#27 0x00000000004710ac in PyEval_EvalCodeEx ()
#28 0x000000000046fbe4 in PyEval_EvalFrameEx ()
#29 0x00000000004708e4 in PyEval_EvalFrameEx ()
#30 0x00000000004708e4 in PyEval_EvalFrameEx ()
#31 0x00000000004710ac in PyEval_EvalCodeEx ()
#32 0x00000000004af1f8 in PyClassMethod_New ()
#33 0x0000000000415173 in PyObject_Call ()
#34 0x000000000041adad in PyClass_IsSubclass ()
#35 0x0000000000415173 in PyObject_Call ()
#36 0x000000000046aa02 in PyEval_CallObjectWithKeywords ()
#37 0x0000000000494cad in initthread ()
#38 0x000000080093fa27 in pthread_getprio () from /lib/libthr.so.3
#39 0x00007fffffb5b000 in ?? ()

But no ideas about the reason, sorry

Dmitri

From stefan_ml at behnel.de  Thu Jul 16 16:53:00 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 16 Jul 2009 16:53:00 +0200
Subject: [lxml-dev] About the position of html parsing by HTML Target
 parser
In-Reply-To: <2514262.666501247720764592.JavaMail.coremail@bj163app16.163.com>
References: <2514262.666501247720764592.JavaMail.coremail@bj163app16.163.com>
Message-ID: <4A5F3ECC.1060104@behnel.de>


qhlonline wrote:
> Hi, all I am parsing html files with lxml target parser, now I wan't to
> know when I have reached some HTML tag, how can I know the position of
> the HTML document I am parsing?

These are two different requirements. Do you really need the line/character
information here? Isn't the structural position enough?


> Is there any callbacks in target parser
> who can tell me the total stream length I have parsed?

Not that I know of. Same as in ElementTree, I'd say.

Stefan

From stefan_ml at behnel.de  Thu Jul 16 16:56:06 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 16 Jul 2009 16:56:06 +0200
Subject: [lxml-dev] About lxml Target parser
In-Reply-To: <27417447.901721247751070945.JavaMail.coremail@bj163app72.163.com>
References: <27417447.901721247751070945.JavaMail.coremail@bj163app72.163.com>
Message-ID: <4A5F3F86.2060508@behnel.de>


qhlonline wrote:
> I have read part of the source file Saxparser.pxi, May be the two
> classes named _SaxParserTarget and _SaxParserContext have close
> relationship with Target Parser.

Yes, they are (somewhat) related.


> A function _handleSaxData has been
> defined to deal with HTML data part. But why the data returned is only
> the text part between tags?

Because that's how the SAX interface is defined (and how XML defines text
content, BTW).


> I wan't to know how much characters of the
> HTML file have been parsed with the target parser, when a
> "start_element" event comes.

I replied in your other thread about the same topic.

Stefan

From p.oberndoerfer at urheberrecht.org  Thu Jul 16 19:59:09 2009
From: p.oberndoerfer at urheberrecht.org (Pascal Oberndoerfer)
Date: Thu, 16 Jul 2009 19:59:09 +0200
Subject: [lxml-dev] building lxml for OS X 10.4 on PPC?
In-Reply-To: <8d93cb3af9d1da196b2b8b0000336a8d@localhost>
References: <4A5B9F60.6070108@urheberrecht.org>
	<8d93cb3af9d1da196b2b8b0000336a8d@localhost>
Message-ID: <4A5F6A6D.1000509@urheberrecht.org>

Michael Guntsche schrieb:

> On Mon, 13 Jul 2009 22:56:00 +0200, Pascal Oberndoerfer
> <p.oberndoerfer at urheberrecht.org> wrote:
>> Hello,
>>
>> I am desperately trying to build lxml on a Mac PPC with OS X 10.4. It is
>> not the most current plattform, but I am stuck with it...
>>
>> I tried "STATIC_DEPS= true easy_install lxml", which works perfectly on
>> all Intel machines I have access to. But unfortunately there seems to be
>> a problem with libiconv on the PPC side.
> 
> I built lxml several months ago on my Tiger G5 without any problems. What
> error do you get exactly?
> 
> 

I get the following "last lines" (below) during compiling libxml2, where
I suppose the important bit is:

> /usr/libexec/gcc/i686-apple-darwin8/4.0.1/ld:
> warning /usr/lib/libiconv.dylib cputype (18,
> architecture ppc) does not match cputype (7)
> for specified -arch flag: i386 (file not loaded)

This is a PPC G5, 2Ghz, 1GB, 10.4.11, XCode 2.5.

Maybe I got an incompatible libiconv installed by a previous version of
Apple's DevTools? I am really wildly guessing here...

Thanks a lot.

Pascal


<snip>

> (cd .libs && rm -f testdso.la && ln -s ../testdso.la testdso.la)
> gcc -DHAVE_CONFIG_H -I. -I./include -I./include  -D_REENTRANT      -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -O2 -pedantic -W -Wformat -
> Wunused -Wimplicit -Wreturn-type -Wswitch -Wcomment -Wtrigraphs -Wformat -Wchar-subscripts -Wuninitialized -Wparentheses -Wshadow -Wpointer-arith -Wcast-alig
> n -Wwrite-strings -Waggregate-return -Wstrict-prototypes -Wmissing-prototypes -Wnested-externs -Winline -Wredundant-decls -c xmllint.c
> /bin/sh ./libtool --tag=CC   --mode=link gcc  -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -O2 -pedantic -W -Wformat -Wunused -Wimplicit -W
> return-type -Wswitch -Wcomment -Wtrigraphs -Wformat -Wchar-subscripts -Wuninitialized -Wparentheses -Wshadow -Wpointer-arith -Wcast-align -Wwrite-strings -Wa
> ggregate-return -Wstrict-prototypes -Wmissing-prototypes -Wnested-externs -Winline -Wredundant-decls  -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10
> .4u.sdk -o xmllint xmllint.o ./libxml2.la -lpthread -lz -liconv -lm  
> gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -O2 -pedantic -W -Wformat -Wunused -Wimplicit -Wreturn-type -Wswitch -Wcomment -Wtrigraphs
>  -Wformat -Wchar-subscripts -Wuninitialized -Wparentheses -Wshadow -Wpointer-arith -Wcast-align -Wwrite-strings -Waggregate-return -Wstrict-prototypes -Wmiss
> ing-prototypes -Wnested-externs -Winline -Wredundant-decls -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -o xmllint xmllint.o  ./.libs/libxm
> l2.a -lpthread -lz /usr/lib/libiconv.dylib -lm  
> /usr/libexec/gcc/i686-apple-darwin8/4.0.1/ld: for architecture i386
> /usr/libexec/gcc/i686-apple-darwin8/4.0.1/ld: warning /usr/lib/libiconv.dylib cputype (18, architecture ppc) does not match cputype (7) for specified -arch f
> lag: i386 (file not loaded)
> /usr/libexec/gcc/i686-apple-darwin8/4.0.1/ld: Undefined symbols:
> _libiconv
> _libiconv_close
> _libiconv_open
> collect2: ld returned 1 exit status
> lipo: can't open input file: /var/tmp//cc5Ydgsq.out (No such file or directory)
> make[2]: *** [xmllint] Error 1
> make[1]: *** [all-recursive] Error 1
> make: *** [all] Error 2
> Traceback (most recent call last):
>   File "/Library/Frameworks/Python.framework/Versions/Current/bin/easy_install", line 8, in <module>
>     load_entry_point('setuptools==0.6c9', 'console_scripts', 'easy_install')()
>   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg/setuptools/command/easy_install.py", line 1671, in main
>   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg/setuptools/command/easy_install.py", line 1659, in with_ei_usage
>   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg/setuptools/command/easy_install.py", line 1675, in <lambda>
>   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/core.py", line 151, in setup
>     dist.run_commands()
>   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", line 974, in run_commands
>     self.run_command(cmd)
>   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", line 994, in run_command
>     cmd_obj.run()
>   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg/setuptools/command/easy_install.py", line 211, in run
>   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg/setuptools/command/easy_install.py", line 446, in easy_install
>   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg/setuptools/command/easy_install.py", line 476, in install_item
>   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg/setuptools/command/easy_install.py", line 655, in install_eggs
>   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg/setuptools/command/easy_install.py", line 930, in build_and_install
>   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg/setuptools/command/easy_install.py", line 919, in run_setup
>   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg/setuptools/sandbox.py", line 27, in run_setup
>   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg/setuptools/sandbox.py", line 63, in run
>   File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg/setuptools/sandbox.py", line 29, in <lambda>
>   File "setup.py", line 115, in <module>
>   File "/tmp/easy_install-O_UuTG/lxml-2.2.2/setupinfo.py", line 50, in ext_modules
>   File "/tmp/easy_install-O_UuTG/lxml-2.2.2/buildlibxml.py", line 200, in build_libxml2xslt
>   File "/tmp/easy_install-O_UuTG/lxml-2.2.2/buildlibxml.py", line 158, in call_subprocess
> Exception: Command "make" returned code 2



From sidnei at enfoldsystems.com  Thu Jul 16 21:35:51 2009
From: sidnei at enfoldsystems.com (Sidnei da Silva)
Date: Thu, 16 Jul 2009 16:35:51 -0300
Subject: [lxml-dev] Compilation of lxml on Windows 64-bit
In-Reply-To: <789d27b10810170810q50ccabd7v1c95312d6286b0de@mail.gmail.com>
References: <789d27b10810020656j147558e9yb52af3bfafd2ca17@mail.gmail.com> 
	<a7a2b76b0810020706w792c5c97qa7cd12a519333615@mail.gmail.com> 
	<789d27b10810150943v1e156111r64dde24aeb53d03d@mail.gmail.com> 
	<789d27b10810170810q50ccabd7v1c95312d6286b0de@mail.gmail.com>
Message-ID: <a7a2b76b0907161235t49f425dft99566e4e8077e602@mail.gmail.com>

Hi Hanni,

I finally managed to compile lxml for x64 myself. Since now I have a
working environment capable of generating those, I will start
uploading x64 binaries to PyPI.

Would be awesome if you could give those lxml 2.2.1 binaries a try and
let me know if it works for you. They are completely statically linked
so no extra dependencies should be needed.

   http://drop.io/ouequws

-- Sidnei

From mike at it-loops.com  Thu Jul 16 23:04:48 2009
From: mike at it-loops.com (Michael Guntsche)
Date: Thu, 16 Jul 2009 23:04:48 +0200
Subject: [lxml-dev] building lxml for OS X 10.4 on PPC?
In-Reply-To: <4A5F6A6D.1000509@urheberrecht.org>
References: <4A5B9F60.6070108@urheberrecht.org>
	<8d93cb3af9d1da196b2b8b0000336a8d@localhost>
	<4A5F6A6D.1000509@urheberrecht.org>
Message-ID: <38E808CD-2715-41AE-BF4B-3618FA57F484@it-loops.com>


On Jul 16, 2009, at 19:59, Pascal Oberndoerfer wrote:
> I get the following "last lines" (below) during compiling libxml2,  
> where
> I suppose the important bit is:
>
>> /usr/libexec/gcc/i686-apple-darwin8/4.0.1/ld:
>> warning /usr/lib/libiconv.dylib cputype (18,
>> architecture ppc) does not match cputype (7)
>> for specified -arch flag: i386 (file not loaded)
>

Ok, I was able to reproduce the problem. I also have a quick fix but  
it's more a hack than anything else.
First the problem. During the creation of the static libxml.a library  
libtool also creates an libxml2.la file.
libtool finds /usr/lib/libiconv.la which defines /usr/lib/ 
libiconv.dylib above. Now this file is no fat-binary and just has the  
PCC architecture.
As a workaround rename /usr/lib/libiconv.la or move it somewhere else,  
build lxml and move/rename the file back. This way the "correct"  
libiconv.dylib will be used.

I think the most proper fix would be to build libiconv as part of the  
static-deps as well. Stefan what do you think?

Kind regards,
Michael

From p.oberndoerfer at urheberrecht.org  Fri Jul 17 01:18:25 2009
From: p.oberndoerfer at urheberrecht.org (Pascal Oberndoerfer)
Date: Fri, 17 Jul 2009 01:18:25 +0200
Subject: [lxml-dev] building lxml for OS X 10.4 on PPC?
In-Reply-To: <38E808CD-2715-41AE-BF4B-3618FA57F484@it-loops.com>
References: <4A5B9F60.6070108@urheberrecht.org>
	<8d93cb3af9d1da196b2b8b0000336a8d@localhost>
	<4A5F6A6D.1000509@urheberrecht.org>
	<38E808CD-2715-41AE-BF4B-3618FA57F484@it-loops.com>
Message-ID: <4A5FB541.2070804@urheberrecht.org>

Michael Guntsche schrieb:
> 
> On Jul 16, 2009, at 19:59, Pascal Oberndoerfer wrote:
>> I get the following "last lines" (below) during compiling libxml2, where
>> I suppose the important bit is:
>>
>>> /usr/libexec/gcc/i686-apple-darwin8/4.0.1/ld:
>>> warning /usr/lib/libiconv.dylib cputype (18,
>>> architecture ppc) does not match cputype (7)
>>> for specified -arch flag: i386 (file not loaded)
>>
> 
> Ok, I was able to reproduce the problem. I also have a quick fix but
> it's more a hack than anything else.
> First the problem. During the creation of the static libxml.a library
> libtool also creates an libxml2.la file.
> libtool finds /usr/lib/libiconv.la which defines /usr/lib/libiconv.dylib
> above. Now this file is no fat-binary and just has the PCC architecture.
> As a workaround rename /usr/lib/libiconv.la or move it somewhere else,
> build lxml and move/rename the file back. This way the "correct"
> libiconv.dylib will be used.
> 
> I think the most proper fix would be to build libiconv as part of the
> static-deps as well. Stefan what do you think?
> 
> Kind regards,
> Michael
> 

This allowed me to build on a G4 PPC and a G5 PPC Mac. Thanks!

The very last step: "Building against libxml2/libxslt in the following
directory:
/private/tmp/easy_install-W57UyT/lxml-2.2.2/build/tmp/libxml2/lib" takes
quite long. Is this expected behaviour?

I will do some testing over the weekend. Please let me know if I can do
some testing etc.

Best regards & again: thanks!

Pascal

From qhlonline at 163.com  Fri Jul 17 05:15:55 2009
From: qhlonline at 163.com (qhlonline)
Date: Fri, 17 Jul 2009 11:15:55 +0800 (CST)
Subject: [lxml-dev] About the position of html parsing by HTML Target
 parser
In-Reply-To: <4A5F3ECC.1060104@behnel.de>
References: <4A5F3ECC.1060104@behnel.de>
	<2514262.666501247720764592.JavaMail.coremail@bj163app16.163.com>
Message-ID: <15125988.136891247800555967.JavaMail.coremail@bj163app61.163.com>

Stefan Behnel" <stefan_ml at behnel.de>
>
>qhlonline wrote:
>> Hi, all I am parsing html files with lxml target parser, now I wan't to
>> know when I have reached some HTML tag, how can I know the position of
>> the HTML document I am parsing?
>
>These are two different requirements. Do you really need the line/character
>information here? Isn't the structural position enough?
>
I have to know the real parsing position when some special tags found by target parser. Is the 'structural position ' means information about which line and which column, like that in Parsing Error Report? I think they are helpless in compute the parsing stream length. In libxml2 source file SAX2.c there is an callback interface (charactersSAXFunc) for character event:
    hdlr->characters = xmlSAX2Characters  
The event handler has a 'len' parameter which tells current parsed HTML stream length. and I noticed that lxml source Saxparser.pxi there is a function defination:
      cdef void _handleSaxData(void* ctxt, char* c_data, int data_len) with gil:
   It works just as processer of the sax.character event. How can I change the lxml source code of target parser to add sax.character event processing to it with 'data_len' parameter? Not the default 'data' function in target parser of couse, It has no parameter like 'data_len' and its 'data' parameter is only the text between an element, not the whole parsed string.
>> Is there any callbacks in target parser
>> who can tell me the total stream length I have parsed?
>
>Not that I know of. Same as in ElementTree, I'd say.
>
>Stefan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20090717/1f213c46/attachment.htm 

From qhlonline at 163.com  Fri Jul 17 05:34:16 2009
From: qhlonline at 163.com (qhlonline)
Date: Fri, 17 Jul 2009 11:34:16 +0800 (CST)
Subject: [lxml-dev] About the position of html parsing by HTML Target
 parser
In-Reply-To: <4A5F3ECC.1060104@behnel.de>
References: <4A5F3ECC.1060104@behnel.de>
	<2514262.666501247720764592.JavaMail.coremail@bj163app16.163.com>
Message-ID: <452989.142481247801656118.JavaMail.coremail@bj163app72.163.com>

2009-07-16?"Stefan Behnel" <stefan_ml at behnel.de> 
>
>qhlonline wrote:
>> Hi, all I am parsing html files with lxml target parser, now I wan't to
>> know when I have reached some HTML tag, how can I know the position of
>> the HTML document I am parsing?
>
>These are two different requirements. Do you really need the line/character
>information here? Isn't the structural position enough?
>
>
>> Is there any callbacks in target parser
>> who can tell me the total stream length I have parsed?
>
>Not that I know of. Same as in ElementTree, I'd say.
>
>Stefan

If there are some way for me to get the parsing context, and if I can access this structure directly, may be this problem can get solved.  In libxml2 there is a defination of "struct _xmlParserCtxt". This structure have a member "long nbChars; " , It is just the "number of xmlChar processed" .    
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20090717/5c1329fc/attachment.htm 

From stefan_ml at behnel.de  Fri Jul 17 08:53:38 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 17 Jul 2009 08:53:38 +0200
Subject: [lxml-dev] About the position of html parsing by HTML Target
 parser
In-Reply-To: <15125988.136891247800555967.JavaMail.coremail@bj163app61.163.com>
References: <4A5F3ECC.1060104@behnel.de>	<2514262.666501247720764592.JavaMail.coremail@bj163app16.163.com>
	<15125988.136891247800555967.JavaMail.coremail@bj163app61.163.com>
Message-ID: <4A601FF2.8080604@behnel.de>


qhlonline wrote:
> I have to know the real parsing position when some special tags found by
> target parser.

Interesting requirement. I wonder who designs XML formats where you have to
know the stream position to read them. Do you actually mean the bytes
position or the character position?


> Is the 'structural position ' means information about which
> line and which column, like that in Parsing Error Report?

No, with "structural position" I meant the position of the element within
the tree structure, such as the unique path from the root element to the
currently parsed element.


> In libxml2 source file
> SAX2.c there is an callback interface (charactersSAXFunc) for character event:
>
>     hdlr->characters = xmlSAX2Characters
>
> The event handler has a 'len' parameter which tells current parsed HTML
> stream length. and I noticed that lxml source Saxparser.pxi there is a
> function defination:
> 
>       cdef void _handleSaxData(void* ctxt, char* c_data, int data_len) with gil:
>
> It works just as processer of the sax.character event. How can I change
> the lxml source code of target parser to add sax.character event
> processing to it with 'data_len' parameter?

You don't have to. Just take the string that you receive in .data(), encode
it as UTF-8, and take its len(). However, that doesn't help you with your
problem, as it is not the information you are looking for.

Stefan


From stefan_ml at behnel.de  Fri Jul 17 09:00:44 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 17 Jul 2009 09:00:44 +0200
Subject: [lxml-dev] About the position of html parsing by HTML Target
 parser
In-Reply-To: <452989.142481247801656118.JavaMail.coremail@bj163app72.163.com>
References: <4A5F3ECC.1060104@behnel.de>	<2514262.666501247720764592.JavaMail.coremail@bj163app16.163.com>
	<452989.142481247801656118.JavaMail.coremail@bj163app72.163.com>
Message-ID: <4A60219C.7050806@behnel.de>


qhlonline wrote:
> If there are some way for me to get the parsing context, and if I can
> access this structure directly, may be this problem can get solved.  In
> libxml2 there is a defination of "struct _xmlParserCtxt". This structure
> have a member "long nbChars; " , It is just the "number of xmlChar
> processed" .

You could subtype the XMLParser class in Cython. That's not trivial, since
it's not exported at the C-API level. You'll have to redefine the class
hierarchy in a separate lxml.etree.pxd file to do that. Note that you only
need to access the _parser_context and (maybe) _push_parser_context. The
other object type fields in the classes can be set to type "object" instead
of their real type.

But remember that the type isn't public. Future lxml versions may change
it, which means that you will have to adapt your code.

That said, I still do not understand why you need the character stream
position for parsing. Could you elaborate on that?

Stefan


From stefan_ml at behnel.de  Fri Jul 17 09:02:46 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 17 Jul 2009 09:02:46 +0200
Subject: [lxml-dev] building lxml for OS X 10.4 on PPC?
In-Reply-To: <4A5FB541.2070804@urheberrecht.org>
References: <4A5B9F60.6070108@urheberrecht.org>	<8d93cb3af9d1da196b2b8b0000336a8d@localhost>	<4A5F6A6D.1000509@urheberrecht.org>	<38E808CD-2715-41AE-BF4B-3618FA57F484@it-loops.com>
	<4A5FB541.2070804@urheberrecht.org>
Message-ID: <4A602216.9020606@behnel.de>


Pascal Oberndoerfer wrote:
> The very last step: "Building against libxml2/libxslt in the following
> directory:
> /private/tmp/easy_install-W57UyT/lxml-2.2.2/build/tmp/libxml2/lib" takes
> quite long. Is this expected behaviour?

Yes. The generated C source files are pretty huge.

Stefan

From stefan_ml at behnel.de  Fri Jul 17 09:04:46 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 17 Jul 2009 09:04:46 +0200
Subject: [lxml-dev] building lxml for OS X 10.4 on PPC?
In-Reply-To: <38E808CD-2715-41AE-BF4B-3618FA57F484@it-loops.com>
References: <4A5B9F60.6070108@urheberrecht.org>	<8d93cb3af9d1da196b2b8b0000336a8d@localhost>	<4A5F6A6D.1000509@urheberrecht.org>
	<38E808CD-2715-41AE-BF4B-3618FA57F484@it-loops.com>
Message-ID: <4A60228E.3090607@behnel.de>


Michael Guntsche wrote:
> I think the most proper fix would be to build libiconv as part of the  
> static-deps as well. Stefan what do you think?

Yes, I agree that that should at least be an option. Want to give it a try?

Stefan

From qhlonline at 163.com  Fri Jul 17 09:23:56 2009
From: qhlonline at 163.com (qhlonline)
Date: Fri, 17 Jul 2009 15:23:56 +0800 (CST)
Subject: [lxml-dev] About the position of html parsing by HTML Target
 parser
In-Reply-To: <4A601FF2.8080604@behnel.de>
References: <4A601FF2.8080604@behnel.de> <4A5F3ECC.1060104@behnel.de>
	<2514262.666501247720764592.JavaMail.coremail@bj163app16.163.com>
	<15125988.136891247800555967.JavaMail.coremail@bj163app61.163.com>
Message-ID: <25627354.246611247815436083.JavaMail.coremail@bj163app69.163.com>


2009-07-17?"Stefan Behnel" <stefan_ml at behnel.de> ?
>
>qhlonline wrote:
>> I have to know the real parsing position when some special tags found by
>> target parser.
>
>Interesting requirement. I wonder who designs XML formats where you 
>know the stream position to read them. Do you actually mean the bytes
>position or the character position?
We are parsing some HTML files. I don't think HTML parser has any obligation to provide position infomation too. But Our leader requires position information for Traget events. The position is not necessarily accurate, but it must be character position, like how much Bytes of the HTML file have been parsed .
>
>
>> Is the 'structural position ' means information about which
>> line and which column, like that in Parsing Error Report?
>
>No, with "structural position" I meant the position of the element within
>the tree structure, such as the unique path from the root element to the
>currently parsed element.
>
>
>> In libxml2 source file
>> SAX2.c there is an callback interface (charactersSAXFunc) for character event:
>>
>>     hdlr->characters = xmlSAX2Characters
>>
>> The event handler has a 'len' parameter which tells current parsed HTML
>> stream length. and I noticed that lxml source Saxparser.pxi there is a
>> function defination:
>> 
>>       cdef void _handleSaxData(void* ctxt, char* c_data, int data_len) with gil:
>>
>> It works just as processer of the sax.character event. How can I change
>> the lxml source code of target parser to add sax.character event
>> processing to it with 'data_len' parameter?
>
>You don't have to. Just take the string that you receive in .data(), encode
>it as UTF-8, and take its len(). However, that doesn't help you with your
>problem, as it is not the information you are looking for.
>
Yes, I have tested for libxml2 library directly. I have defined my function according to the character event handlers' arg-list of libxml2. The result shows that its 'len' argument only shows the string length of text between closed tags. It doesn't help to solve my problem. The worse, I found its "ctxt" parameter, whose type is "xmlParserCtxtPtr ", is NULL. That means I can't get ParserContext now. The xmlParserCtxt structer has a member showing it's current parsing position. Now I am trying to know, If a lxml target parser (or if a self-defined libxml2 sax parser) generate a "xmlParserCtextPtr" like ParserContext? How Can I get it? I guess may be I have steped on the wrong way. I don't know which I should focus on ( libxml2 or lxml ) to solve my problem, would you give me some suggestion?
>Stefan
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20090717/bbbc5562/attachment-0001.htm 

From hanni.ali at gmail.com  Fri Jul 17 09:36:47 2009
From: hanni.ali at gmail.com (Hanni Ali)
Date: Fri, 17 Jul 2009 08:36:47 +0100
Subject: [lxml-dev] Compilation of lxml on Windows 64-bit
In-Reply-To: <a7a2b76b0907161235t49f425dft99566e4e8077e602@mail.gmail.com>
References: <789d27b10810020656j147558e9yb52af3bfafd2ca17@mail.gmail.com>
	<a7a2b76b0810020706w792c5c97qa7cd12a519333615@mail.gmail.com>
	<789d27b10810150943v1e156111r64dde24aeb53d03d@mail.gmail.com>
	<789d27b10810170810q50ccabd7v1c95312d6286b0de@mail.gmail.com>
	<a7a2b76b0907161235t49f425dft99566e4e8077e602@mail.gmail.com>
Message-ID: <789d27b10907170036k69984df0oe9b767c603d90aea@mail.gmail.com>

Hi Sidnei,

Seems to work well for me here, I will install it across our dev environment
to give it some thorought testing time, but it looks good to me.

Cheers,

Hanni

2009/7/16 Sidnei da Silva <sidnei at enfoldsystems.com>

> Hi Hanni,
>
> I finally managed to compile lxml for x64 myself. Since now I have a
> working environment capable of generating those, I will start
> uploading x64 binaries to PyPI.
>
> Would be awesome if you could give those lxml 2.2.1 binaries a try and
> let me know if it works for you. They are completely statically linked
> so no extra dependencies should be needed.
>
>   http://drop.io/ouequws
>
> -- Sidnei
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20090717/e9feb31c/attachment.htm 

From ndudfield at gmail.com  Fri Jul 17 12:48:24 2009
From: ndudfield at gmail.com (Nicholas Dudfield)
Date: Fri, 17 Jul 2009 18:48:24 +0800
Subject: [lxml-dev] About the position of html parsing by HTML Target
	parser
Message-ID: <cf6355060907170348m5ec44c58o80bdebbcb635ca52@mail.gmail.com>

Wow, someone else with this requirement. I was meaning to post to the
list about this. I'm using lxml to implement a XPath / CSS selection
plugin for a python extensible editor. I'd like to have a mapping of
view buffer regions to xml nodes.   The workaround I used to get the
exact character position was to use the feed interface, a character at
a time and manually monitor bytestream position. It's fairly slow
though. I'd like to implement this in CYthon or use whatever
underlying facility there is to speed it up.

You can see some screen casts at this forum thread which should make
it more obvious what I mean re: css / xpath document selections:
http://www.sublimetext.com/forum/viewtopic.php?f=5&t=547

Cheers.

From qhlonline at 163.com  Fri Jul 17 14:52:18 2009
From: qhlonline at 163.com (qhlonline)
Date: Fri, 17 Jul 2009 20:52:18 +0800 (CST)
Subject: [lxml-dev] Fw:Re:Re: About the position of html parsing by HTML
 Target parser
Message-ID: <21282282.401601247835138011.JavaMail.coremail@bj163app59.163.com>

 
 
 
 
Re:Re: [lxml-dev] About the position of html parsing by HTML Target parser




2009-07-17?"Nicholas Dudfield" <ndudfield at gmail.com> 
>Wow, someone else with this requirement. I was meaning to post to the
>list about this. I'm using lxml to implement a XPath / CSS selection
>plugin for a python extensible editor. I'd like to have a mapping of
>view buffer regions to xml nodes.   The workaround I used to get the
>exact character position was to use the feed interface, a character at
>a time and manually monitor bytestream position. It's fairly slow
>though. I'd like to implement this in CYthon or use whatever
>underlying facility there is to speed it up.
>
Thank you for your suggestion. I have another idea that I can cumulate total characters of tags and text parsed when I encounter an element, that will mean I have to set a counter to add characters got by startElement function and data function of target parser?This is not an accurate result although. 
    But the key problem is that we need high parsing speed too. I mean we should get the position value during the parsing process. The libxml2's ParsingContext does provide a value of current parsing position. Now I wan't to read it in lxml. So I think I have to define a callback function in libxml2 to access to the value and then alter part of lxml pxi source to receive the value in target parser. I don't know whether this will do, but I am trying. Thank you for your suggestion again! 
>You can see some screen casts at this forum thread which should make
>it more obvious what I mean re: css / xpath document selections:
>http://www.sublimetext.com/forum/viewtopic.php?f=5&t=547
>
>Cheers.
>_______________________________________________
>lxml-dev mailing list
>lxml-dev at codespeak.net
>http://codespeak.net/mailman/listinfo/lxml-dev





-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20090717/1c3f56dd/attachment.htm 

From cz at gocept.com  Sun Jul 19 10:25:34 2009
From: cz at gocept.com (Christian Zagrodnick)
Date: Sun, 19 Jul 2009 10:25:34 +0200
Subject: [lxml-dev] Weakref support?
Message-ID: <h3ul9u$rpr$1@ger.gmane.org>

Hoi,

would it be possible to add weakref support to lxml trees (objectify in 
particular)?

http://docs.python.org/extending/newtypes.html#weakref-support

Regards,
-- 
Christian Zagrodnick ? cz at gocept.com
gocept gmbh & co. kg ? forsterstra?e 29 ? 06112 halle (saale) ? germany
http://gocept.com ? tel +49 345 1229889 4 ? fax +49 345 1229889 1
Zope and Plone consulting and development



From stefan_ml at behnel.de  Sun Jul 19 15:25:52 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 19 Jul 2009 15:25:52 +0200
Subject: [lxml-dev] Weakref support?
In-Reply-To: <h3ul9u$rpr$1@ger.gmane.org>
References: <h3ul9u$rpr$1@ger.gmane.org>
Message-ID: <4A631EE0.3080200@behnel.de>


Christian Zagrodnick wrote:
> would it be possible to add weakref support to lxml trees (objectify in 
> particular)?

Possible: yes. You can simply add a "__weakref__" attribute to the _Element
class (or just the ObjectifiedElement class) and Cython will do the rest.

http://docs.cython.org/docs/extension_types.html#making-extension-types-weak-referenceable

However, making the class weak-referenceable will add another bit to the
size of each Element instance, which you will notice when you do lots of
tree work in Python space. I'd prefer avoiding that for a class as vital as
_Element, that's why it isn't currently there. I also don't know if it has
any performance implications if a class is weak-referenceable. Does anyone
know if there is any impact on the garbage collector, for example?

At least the memory size isn't that a big issue. When you look at the
class, you may notice that there already is some stuff in there that isn't
100% required (e.g. the cached tag name). Adding 4/8 bytes to it may not be
that bad.

Is there a specific use case you have in mind?

Stefan


From stefan_ml at behnel.de  Sun Jul 19 20:31:56 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 19 Jul 2009 20:31:56 +0200
Subject: [lxml-dev] About the position of html parsing by HTML Target
 parser
In-Reply-To: <cf6355060907170348m5ec44c58o80bdebbcb635ca52@mail.gmail.com>
References: <cf6355060907170348m5ec44c58o80bdebbcb635ca52@mail.gmail.com>
Message-ID: <4A63669C.4050404@behnel.de>


Nicholas Dudfield wrote:
> Wow, someone else with this requirement. I was meaning to post to the
> list about this. I'm using lxml to implement a XPath / CSS selection
> plugin for a python extensible editor. I'd like to have a mapping of
> view buffer regions to xml nodes.

At least the line is available from the "sourceline" property of an element
- although only up to 65536:

http://bugzilla.gnome.org/show_bug.cgi?id=325533

If you are in a position to whitespace-clean and pretty-print the XML
document, that would give you a simple mapping from elements to document
positions that you can exploit at the application level. Even if you can't,
that would still give you a usable model to work with that you could match
with the original stream to find the 'real' positions.


> The workaround I used to get the
> exact character position was to use the feed interface, a character at
> a time and manually monitor bytestream position. It's fairly slow
> though. I'd like to implement this in CYthon or use whatever
> underlying facility there is to speed it up.

Speeding up this approach is pretty much futile IMHO. The parser gains
speed from efficient I/O and memory management. Passing a byte at a time
totally counters that (and even then it's only a *byte* at a time, not a
*character* at a time).

What I could imagine to do instead is to traverse the element tree and to
do an incremental text search for each element tag (i.e. the regexp
"<tagname\s") to recover the exact original positions. That would also
allow you to work at the character level rather than the byte level
(assuming that the editor works at that level, too). Searching for the
above regexp is safe as "<" cannot occur anywhere in the XML data stream
except for a tag start/end or comment/PI (ok, minus DTDs, but that's easy
to catch using the line number of the root element).

I will also check if there is a way to provide the position at the (target)
parser level, but that needs to fit the current interface. And I currently
do not have much time to dig into this.

Stefan

From qhlonline at 163.com  Mon Jul 20 05:22:00 2009
From: qhlonline at 163.com (qhlonline)
Date: Mon, 20 Jul 2009 11:22:00 +0800 (CST)
Subject: [lxml-dev] About the position of html parsing by HTML Target
 parser
In-Reply-To: <4A63669C.4050404@behnel.de>
References: <4A63669C.4050404@behnel.de>
	<cf6355060907170348m5ec44c58o80bdebbcb635ca52@mail.gmail.com>
Message-ID: <9148036.360291248060120782.JavaMail.coremail@bj163app72.163.com>

 
2009-07-20?"Stefan Behnel" <stefan_ml at behnel.de> ?
>
>Nicholas Dudfield wrote:
>> Wow, someone else with this requirement. I was meaning to post to the
>> list about this. I'm using lxml to implement a XPath / CSS selection
>> plugin for a python extensible editor. I'd like to have a mapping of
>> view buffer regions to xml nodes.
>
>At least the line is available from the "sourceline" property of an element
>- although only up to 65536:
>
>http://bugzilla.gnome.org/show_bug.cgi?id=325533
>
>If you are in a position to whitespace-clean and pretty-print the XML
>document, that would give you a simple mapping from elements to document
>positions that you can exploit at the application level. Even if you can't,
>that would still give you a usable model to work with that you could match
>with the original stream to find the 'real' positions.
>
>
>> The workaround I used to get the
>> exact character position was to use the feed interface, a character at
>> a time and manually monitor bytestream position. It's fairly slow
>> though. I'd like to implement this in CYthon or use whatever
>> underlying facility there is to speed it up.
>
>Speeding up this approach is pretty much futile IMHO. The parser gains
>speed from efficient I/O and memory management. Passing a byte at a time
>totally counters that (and even then it's only a *byte* at a time, not a
>*character* at a time).
>
>What I could imagine to do instead is to traverse the element tree and to
>do an incremental text search for each element tag (i.e. the regexp
>"<tagname\s") to recover the exact original positions. That would also
>allow you to work at the character level rather than the byte level
>(assuming that the editor works at that level, too). Searching for the
>above regexp is safe as "<" cannot occur anywhere in the XML data stream
>except for a tag start/end or comment/PI (ok, minus DTDs, but that's easy
>to catch using the line number of the root element).
>
>I will also check if there is a way to provide the position at the (target)
>parser level, but that needs to fit the current interface. And I currently
>do not have much time to dig into this.
>
>Stefan
>_______________________________________________
>lxml-dev mailing list
>lxml-dev at codespeak.net
>http://codespeak.net/mailman/listinfo/lxml-dev

Hi, all
   I have tried to alter the libxml2 source to add a callback telling the current position when an element parsed. Now I wan't to know this position in lxml target parser. I don't know where and how to alter lxml source to add target parser event defination and event handler defination, so that my libxml2 new event(callback) can be processed. and I have nerver compile cython source before. Can any body give me some suggestion? 
    Thank you!
Best Regards
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20090720/193a6136/attachment-0001.htm 

From qhlonline at 163.com  Mon Jul 20 05:48:52 2009
From: qhlonline at 163.com (qhlonline)
Date: Mon, 20 Jul 2009 11:48:52 +0800 (CST)
Subject: [lxml-dev] About the position of html parsing by HTML Target
 parser
In-Reply-To: <4A60219C.7050806@behnel.de>
References: <4A60219C.7050806@behnel.de> <4A5F3ECC.1060104@behnel.de>
	<2514262.666501247720764592.JavaMail.coremail@bj163app16.163.com>
	<452989.142481247801656118.JavaMail.coremail@bj163app72.163.com>
Message-ID: <13861823.379461248061732358.JavaMail.coremail@bj163app72.163.com>


2009-07-17?"Stefan Behnel" <stefan_ml at behnel.de> ?
>
>qhlonline wrote:
>> If there are some way for me to get the parsing context, and if I can
>> access this structure directly, may be this problem can get solved.  In
>> libxml2 there is a defination of "struct _xmlParserCtxt". This structure
>> have a member "long nbChars; " , It is just the "number of xmlChar
>> processed" .
>
>You could subtype the XMLParser class in Cython. That's not trivial, since
>it's not exported at the C-API level. You'll have to redefine the class
>hierarchy in a separate lxml.etree.pxd file to do that. Note that you only
>need to access the _parser_context and (maybe) _push_parser_context. The
>other object type fields in the classes can be set to type "object" instead
>of their real type.
>
>But remember that the type isn't public. Future lxml versions may change
>it, which means that you will have to adapt your code.
>
I have changed the libxml2 code to add a new callback telling the current position when an element was seen. I think this can avoid the direct access to parser context. But I am now thinking of how to change target parser to let it access the newly defined callback on python level. I even did't know where to find the target related lxml source code. nor do I know whether my idea is feasble. Is target parser inherited from the TreeBuilder class? can It be changed? and how? I am in urgent need of more and deeper lxml source information.
>That said, I still do not understand why you need the character stream
>position for parsing. Could you elaborate on that?
Well, the position information is usefull. Some outside source of HTML document is declared in a seperate file, like <style> 'css' file. We may get the HTML document and its related source on net concurrently. But the outside source should be inserted in the proper position of HTML document in out application after parse. so the related tag position is usefull now.
>
>Stefan
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20090720/20ddd96b/attachment.htm 

From cz at gocept.com  Mon Jul 20 08:02:47 2009
From: cz at gocept.com (Christian Zagrodnick)
Date: Mon, 20 Jul 2009 08:02:47 +0200
Subject: [lxml-dev] Weakref support?
References: <h3ul9u$rpr$1@ger.gmane.org> <4A631EE0.3080200@behnel.de>
Message-ID: <h411a6$2j9$1@ger.gmane.org>

On 2009-07-19 15:25:52 +0200, Stefan Behnel <stefan_ml at behnel.de> said:

> 
> Christian Zagrodnick wrote:
>> would it be possible to add weakref support to lxml trees (objectify in
>> particular)?
> 
> Possible: yes. You can simply add a "__weakref__" attribute to the _Element
> class (or just the ObjectifiedElement class) and Cython will do the rest.
> 
> http://docs.cython.org/docs/extension_types.html#making-extension-types-weak-referenceable

However, 
> 
> making the class weak-referenceable will add another bit to the
> size of each Element instance, which you will notice when you do lots of
> tree work in Python space. I'd prefer avoiding that for a class as vital as
> _Element, that's why it isn't currently there. I also don't know if it has
> any performance implications if a class is weak-referenceable. Does anyone
> know if there is any impact on the garbage collector, for example?
> 
> At least the memory size isn't that a big issue. When you look at the
> class, you may notice that there already is some stuff in there that isn't
> 100% required (e.g. the cached tag name). Adding 4/8 bytes to it may not be
> that bad.

Of course there are a lot of elements. So being cautious is good.

> 
> Is there a specific use case you have in mind?

Actually all I'd need is a reference to the tree. I don't think 
weak-refs to all elements are necessary or useful. Well, maybe they 
are, but I don't need them :)

Anyway, if you think it is to risky, it won't kill me to go w/o weak refs. :)

Regards,
-- 
Christian Zagrodnick ? cz at gocept.com
gocept gmbh & co. kg ? forsterstra?e 29 ? 06112 halle (saale) ? germany
http://gocept.com ? tel +49 345 1229889 4 ? fax +49 345 1229889 1
Zope and Plone consulting and development



From stefan_ml at behnel.de  Mon Jul 20 09:06:00 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 20 Jul 2009 09:06:00 +0200
Subject: [lxml-dev] About the position of html parsing by HTML Target
 parser
In-Reply-To: <9148036.360291248060120782.JavaMail.coremail@bj163app72.163.com>
References: <4A63669C.4050404@behnel.de>	<cf6355060907170348m5ec44c58o80bdebbcb635ca52@mail.gmail.com>
	<9148036.360291248060120782.JavaMail.coremail@bj163app72.163.com>
Message-ID: <4A641758.1020303@behnel.de>


qhlonline wrote:
> I have tried to alter the libxml2 source to add a callback telling the 
> current position when an element parsed.

Note that something that requires patching libxml2 will not make it into an
lxml release.

As you noted before, the parser context already provides this information
at any time, not only when parsing elements. So adding a callback for it is
not a sensible approach.

I'm not even sure what this position means exactly. Is it (1) the byte
position in the original (undecoded) data stream, (2) the byte position in
the UTF-8 encoded parse stream, or (3) the character position in the XML
stream?

According to the libxml2 docs:

	long nbChars : number of xmlChar processed

This sounds like it's the second information. That would not be useful and
shouldn't get exposed in lxml's API as it's rather error prone to rely on
it: works for ASCII and UTF-8, obviously, may work for some other encodings
depending on the data, but fails for most other streams. OTOH, the first
and the third information /might/ be of interest, depending on your use
case, but are not easily recovered from the information that the parser
provides.


> I have nerver compile cython source before. Can any body give me some
> suggestion?

If you just change lxml's sources, running setup.py will build it just as
before. All you need to do is install Cython 0.11 or later.

http://codespeak.net/lxml/build.html

Stefan

From stefan_ml at behnel.de  Mon Jul 20 09:11:42 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 20 Jul 2009 09:11:42 +0200
Subject: [lxml-dev] About the position of html parsing by HTML Target
 parser
In-Reply-To: <13861823.379461248061732358.JavaMail.coremail@bj163app72.163.com>
References: <4A60219C.7050806@behnel.de>
	<4A5F3ECC.1060104@behnel.de>	<2514262.666501247720764592.JavaMail.coremail@bj163app16.163.com>	<452989.142481247801656118.JavaMail.coremail@bj163app72.163.com>
	<13861823.379461248061732358.JavaMail.coremail@bj163app72.163.com>
Message-ID: <4A6418AE.8020409@behnel.de>


qhlonline wrote:
> 2009-07-17??"Stefan Behnel" wrote??
>> That said, I still do not understand why you need the character stream
>> position for parsing. Could you elaborate on that?
>
> Well, the position information is usefull. Some outside source of HTML
> document is declared in a seperate file, like <style> 'css' file. We may
> get the HTML document and its related source on net concurrently. But
> the outside source should be inserted in the proper position of HTML
> document in out application after parse. so the related tag position is
> usefull now.

I still don't understand what you need the stream position for. If you want
to inject data into the tree, just find the right element and do so.

Or did you mean that you are actually doing a /textual/ replacement here?

Stefan

From stefan_ml at behnel.de  Mon Jul 20 09:48:04 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 20 Jul 2009 09:48:04 +0200
Subject: [lxml-dev] About the position of html parsing by HTML Target
 parser
In-Reply-To: <cf6355060907191845r405b24dfr4eadf394c5eedd26@mail.gmail.com>
References: <cf6355060907170348m5ec44c58o80bdebbcb635ca52@mail.gmail.com>	
	<4A63669C.4050404@behnel.de>
	<cf6355060907191845r405b24dfr4eadf394c5eedd26@mail.gmail.com>
Message-ID: <4A642134.7070206@behnel.de>

[bringing this back to the list]

Nicholas Dudfield wrote:
>>> even then it's only a *byte* at a time, not a *character* at a time).
> 
> I have been feeding it a unicode character at a time and tested it
> with some Chinese characters.

Ah, sure, that works.


>> Searching for the
>> above regexp is safe as "<" cannot occur anywhere in the XML data stream
>> except for a tag start/end or comment/PI
> 
> or in embedded <script></script> with CDATA eg
> $("someSelector").html('<p> ...')

Right, I keep forgetting that CDATA is evil. But you can check for CDATA as
well.


> For clarification, you are saying skip the target parser and feeding
> process and just iterate over the elements and regex for the tags?
> something like "<tagname\s"  but accounting for tags such as <br/> etc

I'd try that, yes. Should be a lot faster as you only handle the elements
in your code, not the text content, for example. And you always know
exactly what the next opening/closing tag must be.

You can special case self-closing tags by checking for contained text and
children. If there isn't anything in them, be prepared to find the next
opening element before finding a closing tag. Having the parsed tree
available gives you a lot of information that you can exploit.

You may also consider using something like ahocorasick to search for all
opening tags at once (note that lxml makes the namespace prefix available
for each Element), plus "<![CDATA". Then double-check the result by
traversing the tree and fixing up any false positives 'somehow'.

http://hkn.eecs.berkeley.edu/~dyoo/python/ahocorasick/


>> I will also check if there is a way to provide the position at the (target)
>> parser level, but that needs to fit the current interface. And I currently
>> do not have much time to dig into this.
> 
> That would be awesomely ideal :)

... although not that helpful, as I just noted in another post. So I don't
think this is a viable solution for now.

Stefan

From stefan_ml at behnel.de  Mon Jul 20 09:21:50 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 20 Jul 2009 09:21:50 +0200
Subject: [lxml-dev] Weakref support?
In-Reply-To: <h411a6$2j9$1@ger.gmane.org>
References: <h3ul9u$rpr$1@ger.gmane.org> <4A631EE0.3080200@behnel.de>
	<h411a6$2j9$1@ger.gmane.org>
Message-ID: <4A641B0E.6090303@behnel.de>


Christian Zagrodnick wrote:
> On 2009-07-19 15:25:52 +0200, Stefan Behnel said:
>> Is there a specific use case you have in mind?
> 
> Actually all I'd need is a reference to the tree. I don't think 
> weak-refs to all elements are necessary or useful. Well, maybe they 
> are, but I don't need them :)

That sounds better. :)

Feel free to add a

	cdef object __weakref__

to the _ElementTree class (below the _context_node attribute) and see if
that works for you. Please report back if it turns out be useful.

Stefan

From qhlonline at 163.com  Mon Jul 20 10:06:43 2009
From: qhlonline at 163.com (qhlonline)
Date: Mon, 20 Jul 2009 16:06:43 +0800 (CST)
Subject: [lxml-dev] About the position of html parsing by HTML Target
 parser
In-Reply-To: <4A641758.1020303@behnel.de>
References: <4A641758.1020303@behnel.de> <4A63669C.4050404@behnel.de>
	<cf6355060907170348m5ec44c58o80bdebbcb635ca52@mail.gmail.com>
	<9148036.360291248060120782.JavaMail.coremail@bj163app72.163.com>
Message-ID: <7854912.670601248077203627.JavaMail.coremail@bj163app105.163.com>


2009-07-20?"Stefan Behnel" <stefan_ml at behnel.de> ?
>
>qhlonline wrote:
>> I have tried to alter the libxml2 source to add a callback telling the 
>> current position when an element parsed.
>
>Note that something that requires patching libxml2 will not make it into an
>lxml release.
>
>As you noted before, the parser context already provides this information
>at any time, not only when parsing elements. So adding a callback for it is
>not a sensible approach.
>
>I'm not even sure what this position means exactly. Is it (1) the byte
>position in the original (undecoded) data stream, (2) the byte position in
>the UTF-8 encoded parse stream, or (3) the character position in the XML
>stream?
>
My change is taking place on the 'htmlParseStartTag' function in HTMLparser.c source file, I think may be its a UTF-8 stream.
 
>According to the libxml2 docs:
>
> long nbChars : number of xmlChar processed
>
>This sounds like it's the second information. That would not be useful and
>shouldn't get exposed in lxml's API as it's rather error prone to rely on
>it: works for ASCII and UTF-8, obviously, may work for some other encodings
>depending on the data, but fails for most other streams. OTOH, the first
>and the third information /might/ be of interest, depending on your use
>case, but are not easily recovered from the information that the parser
>provides.
This positon may not be precise after some encoding changement form other encoding to UTF-8, but I think it can meet our needs according to my leader's requirement.
>
>> I have nerver compile cython source before. Can any body give me some
>> suggestion?
>
>If you just change lxml's sources, running setup.py will build it just as
>before. All you need to do is install Cython 0.11 or later.
>
Thank you for your help! 
>http://codespeak.net/lxml/build.html
>
>Stefan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20090720/115ce0eb/attachment-0001.htm 

From qhlonline at 163.com  Mon Jul 20 10:15:09 2009
From: qhlonline at 163.com (qhlonline)
Date: Mon, 20 Jul 2009 16:15:09 +0800 (CST)
Subject: [lxml-dev] About the position of html parsing by HTML Target
 parser
In-Reply-To: <4A6418AE.8020409@behnel.de>
References: <4A6418AE.8020409@behnel.de> <4A60219C.7050806@behnel.de>
	<4A5F3ECC.1060104@behnel.de>
	<2514262.666501247720764592.JavaMail.coremail@bj163app16.163.com>
	<452989.142481247801656118.JavaMail.coremail@bj163app72.163.com>
	<13861823.379461248061732358.JavaMail.coremail@bj163app72.163.com>
Message-ID: <1474551.678631248077709167.JavaMail.coremail@bj163app105.163.com>


 
2009-07-20?"Stefan Behnel" <stefan_ml at behnel.de> ?
>
>qhlonline wrote:
>> 2009-07-17?"Stefan Behnel" wrote?
>>> That said, I still do not understand why you need the character stream
>>> position for parsing. Could you elaborate on that?
>>
>> Well, the position information is usefull. Some outside source of HTML
>> document is declared in a seperate file, like <style> 'css' file. We may
>> get the HTML document and its related source on net concurrently. But
>> the outside source should be inserted in the proper position of HTML
>> document in out application after parse. so the related tag position is
>> usefull now.
>
>I still don't understand what you need the stream position for. If you want
>to inject data into the tree, just find the right element and do so.
>
>Or did you mean that you are actually doing a /textual/ replacement here?
Yes, I think if we use the ordinary parsing mode, things may be easy. But my leader has used the target parser of lxml. It generate no DOM tree at all, so I can't inject data to a constructed tree directly. 
>
>Stefan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20090720/11ea41fd/attachment.htm 

From qhlonline at 163.com  Mon Jul 20 10:27:48 2009
From: qhlonline at 163.com (qhlonline)
Date: Mon, 20 Jul 2009 16:27:48 +0800 (CST)
Subject: [lxml-dev] About the position of html parsing by HTML Target
 parser
In-Reply-To: <4A641758.1020303@behnel.de>
References: <4A641758.1020303@behnel.de> <4A63669C.4050404@behnel.de>
	<cf6355060907170348m5ec44c58o80bdebbcb635ca52@mail.gmail.com>
	<9148036.360291248060120782.JavaMail.coremail@bj163app72.163.com>
Message-ID: <32282956.691001248078468280.JavaMail.coremail@bj163app105.163.com>


2009-07-20?"Stefan Behnel" stefan_ml at behnel.de?
>
>qhlonline wrote:
>> I have tried to alter the libxml2 source to add a callback telling the 
>> current position when an element parsed.
>
>Note that something that requires patching libxml2 will not make it into an
>lxml release.
>
>As you noted before, the parser context already provides this information
>at any time, not only when parsing elements. So adding a callback for it is
>not a sensible approach.
>
>I'm not even sure what this position means exactly. Is it (1) the byte
>position in the original (undecoded) data stream, (2) the byte position in
>the UTF-8 encoded parse stream, or (3) the character position in the XML
>stream?
>
>According to the libxml2 docs:
>
> long nbChars : number of xmlChar processed
>
>This sounds like it's the second information. That would not be useful and
>shouldn't get exposed in lxml's API as it's rather error prone to rely on
>it: works for ASCII and UTF-8, obviously, may work for some other encodings
>depending on the data, but fails for most other streams. OTOH, the first
>and the third information /might/ be of interest, depending on your use
>case, but are not easily recovered from the information that the parser
>provides.
>
>
>> I have nerver compile cython source before. Can any body give me some
>> suggestion?
>
>If you just change lxml's sources, running setup.py will build it just as
>before. All you need to do is install Cython 0.11 or later.
>
>http://codespeak.net/lxml/build.html
>
>Stefan

Now the key problem for me is I don't konw whether can I and how to change the lxml target parser defination to let it support my new callback in libxml2, I suspect that the Treebuilder class in Saxparser.pxi is the base class of target parser,because it support functions like 'start', 'end', 'close',  'data', just like the target parser, But I am not sure, because this class seems to be used to bulid a ElmentTree or dom for parserd HTML document from its name and has no relationship with target parser. Am I steping the wrong place?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20090720/c719b72a/attachment.htm 

From stefan_ml at behnel.de  Mon Jul 20 10:37:25 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 20 Jul 2009 10:37:25 +0200
Subject: [lxml-dev] About the position of html parsing by HTML Target
 parser
In-Reply-To: <1474551.678631248077709167.JavaMail.coremail@bj163app105.163.com>
References: <4A6418AE.8020409@behnel.de>
	<4A60219C.7050806@behnel.de>	<4A5F3ECC.1060104@behnel.de>	<2514262.666501247720764592.JavaMail.coremail@bj163app16.163.com>	<452989.142481247801656118.JavaMail.coremail@bj163app72.163.com>	<13861823.379461248061732358.JavaMail.coremail@bj163app72.163.com>
	<1474551.678631248077709167.JavaMail.coremail@bj163app105.163.com>
Message-ID: <4A642CC5.2030904@behnel.de>


qhlonline wrote:
> 2009-07-20?"Stefan Behnel" wrote:
>> did you mean that you are actually doing a /textual/ replacement here?
>
> Yes, I think if we use the ordinary parsing mode, things may be easy.
> But my leader has used the target parser of lxml.

"leader" meaning your supervisor, I suppose? The person who gave you the task?


> It generate no DOM tree at all

Then what does it do? Any chance you could provide enough information to
actually *solve* your problem?

I'm really getting tired of pulling every little bit of required
information out of your responses myself. Keeping it that way is a sure
path towards being ignored.


> so I can't inject data to a constructed tree directly.

Can't you just transform the data inside the callback? (assuming that there
is something behind the callback that receives it...)

Normally, I would expect some kind of SAX-like event chain that receives
callbacks from the parser, transforms it in some way, and sends it
somewhere else. Then I'd modify the data along that path to include the
content I want.

Stefan

From ndudfield at gmail.com  Mon Jul 20 10:53:57 2009
From: ndudfield at gmail.com (Nicholas Dudfield)
Date: Mon, 20 Jul 2009 16:53:57 +0800
Subject: [lxml-dev] About the position of html parsing by HTML Target
	parser
In-Reply-To: <4A642134.7070206@behnel.de>
References: <cf6355060907170348m5ec44c58o80bdebbcb635ca52@mail.gmail.com>
	<4A63669C.4050404@behnel.de>
	<cf6355060907191845r405b24dfr4eadf394c5eedd26@mail.gmail.com>
	<4A642134.7070206@behnel.de>
Message-ID: <cf6355060907200153u1049f88bnf016e2226e48373d@mail.gmail.com>

> [bringing this back to the list]

Sorry about that :)

> You may also consider using something like ahocorasick

ahocorasick doesnt seem to work with native unicode, which makes it
about as useful (for my particular purpose anyway) as the parser
context uft8 stream positions :(

Is there any fundamental reason why an xml parser couldn't work with
native unicode? ie an abstract character stream? I'm completely
clueless when it comes to parsers.

From stefan_ml at behnel.de  Mon Jul 20 13:00:27 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 20 Jul 2009 13:00:27 +0200
Subject: [lxml-dev] About the position of html parsing by HTML Target
 parser
In-Reply-To: <cf6355060907200153u1049f88bnf016e2226e48373d@mail.gmail.com>
References: <cf6355060907170348m5ec44c58o80bdebbcb635ca52@mail.gmail.com>	<4A63669C.4050404@behnel.de>	<cf6355060907191845r405b24dfr4eadf394c5eedd26@mail.gmail.com>	<4A642134.7070206@behnel.de>
	<cf6355060907200153u1049f88bnf016e2226e48373d@mail.gmail.com>
Message-ID: <4A644E4B.4070309@behnel.de>


Nicholas Dudfield wrote:
> ahocorasick doesnt seem to work with native unicode,

Ah, great. :-/ Never tried it myself, just read about it more than once on
c.l.py. Looking at the code, it actually reuses an existing C
implementation, that's why.


> which makes it
> about as useful (for my particular purpose anyway) as the parser
> context uft8 stream positions :(

So the editor you are working with (which one is it, BTW?) gives you
unicode strings? I would have expected it to work with byte buffers
internally. Or maybe that would be considered an implementation detail that
doesn't show at the API level.


> Is there any fundamental reason why an xml parser couldn't work with
> native unicode? ie an abstract character stream? I'm completely
> clueless when it comes to parsers.

Not a fundamental reason, but it's a lot simpler and faster to parse XML
streams in UTF-8 than in any other encoding, and it's also more efficient
to parse them as a UTF-8 byte stream than as a Unicode character stream,
especially in C. You basically read one byte and immediately know if it
represents a control character or not. Unicode characters require 4 bytes
here to represent all possible code points.

Also, UTF-8 is the internal representation format used inside of libxml2
anyway, just for the same reason. So the parser of libxml2 first encodes
the stream to UTF-8 (at the I/O stream buffer layer) and then processes it
without further modifications.

Stefan

From ndudfield at gmail.com  Mon Jul 20 15:05:09 2009
From: ndudfield at gmail.com (Nicholas Dudfield)
Date: Mon, 20 Jul 2009 21:05:09 +0800
Subject: [lxml-dev] About the position of html parsing by HTML Target
	parser
In-Reply-To: <4A644E4B.4070309@behnel.de>
References: <cf6355060907170348m5ec44c58o80bdebbcb635ca52@mail.gmail.com>
	<4A63669C.4050404@behnel.de>
	<cf6355060907191845r405b24dfr4eadf394c5eedd26@mail.gmail.com>
	<4A642134.7070206@behnel.de>
	<cf6355060907200153u1049f88bnf016e2226e48373d@mail.gmail.com>
	<4A644E4B.4070309@behnel.de>
Message-ID: <cf6355060907200605x2333a17cs762955fad01c0028@mail.gmail.com>

>>> So the editor you are working with (which one is it, BTW?)

SublimeText:  http://www.sublimetext.com/features

It's a windows only, closed source editor which however has quite a
few redeeming features, including a Python (2.5) API. All buffer
access returns native unicode and the substr(pt1, pt2) indexing is
character based rather than byte.

>>> Or maybe that would be considered an implementation detail that doesn't show at the API level.

No idea what encoding it represents characters internally with.

>>> So the parser of libxml2 first encodes the stream to UTF-8 (at the I/O stream buffer layer) and then processes it

When you say it converts first to utf8 internally, does that include
recoding xml entities as well? ie A file already utf8 encoded may not
necessarily maintain the bytestream after the first stage of
processing?  eg {'<p>&quot;</p>' : "<p>'</p>"}


ps.  Thanks very much for your time ( and lxml! )

From stefan_ml at behnel.de  Mon Jul 20 15:59:17 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 20 Jul 2009 15:59:17 +0200
Subject: [lxml-dev] About the position of html parsing by HTML Target
 parser
In-Reply-To: <cf6355060907200605x2333a17cs762955fad01c0028@mail.gmail.com>
References: <cf6355060907170348m5ec44c58o80bdebbcb635ca52@mail.gmail.com>	<4A63669C.4050404@behnel.de>	<cf6355060907191845r405b24dfr4eadf394c5eedd26@mail.gmail.com>	<4A642134.7070206@behnel.de>	<cf6355060907200153u1049f88bnf016e2226e48373d@mail.gmail.com>	<4A644E4B.4070309@behnel.de>
	<cf6355060907200605x2333a17cs762955fad01c0028@mail.gmail.com>
Message-ID: <4A647835.9040303@behnel.de>


Nicholas Dudfield wrote:
>> So the editor you are working with (which one is it, BTW?)
> 
> SublimeText:  http://www.sublimetext.com/features
> 
> It's a windows only, closed source editor which however has quite a
> few redeeming features, including a Python (2.5) API. All buffer
> access returns native unicode and the substr(pt1, pt2) indexing is
> character based rather than byte.

Sounds like a sensible API design.

There is a special thing about parsing from Python unicode strings, BTW.
Basically, lxml figures out the platform specific encoding that CPython
uses internally (at startup time), and then passes the plain unicode string
buffer to libxml2 together with the correct decoding selector. Thus,
libxml2 will first recode the UCS2/UCS4 encoded byte sequence into UTF-8,
and then parse that.


>> So the parser of libxml2 first encodes the stream to UTF-8 (at the I/O stream buffer layer) and then processes it
> 
> When you say it converts first to utf8 internally, does that include
> recoding xml entities as well? ie A file already utf8 encoded may not
> necessarily maintain the bytestream after the first stage of
> processing?  eg {'<p>&quot;</p>' : "<p>'</p>"}

No, the codec layer only recodes the characters, so you'd get a UTF-8
encoded "&quot;" byte string as result. The rest is handled by the XML
parser layer, which sees the "&" and considers it the start of an
entity/char reference.

Stefan


From ndudfield at gmail.com  Mon Jul 20 17:34:03 2009
From: ndudfield at gmail.com (Nicholas Dudfield)
Date: Mon, 20 Jul 2009 23:34:03 +0800
Subject: [lxml-dev] About the position of html parsing by HTML Target
	parser
In-Reply-To: <4A647835.9040303@behnel.de>
References: <cf6355060907170348m5ec44c58o80bdebbcb635ca52@mail.gmail.com>
	<4A63669C.4050404@behnel.de>
	<cf6355060907191845r405b24dfr4eadf394c5eedd26@mail.gmail.com>
	<4A642134.7070206@behnel.de>
	<cf6355060907200153u1049f88bnf016e2226e48373d@mail.gmail.com>
	<4A644E4B.4070309@behnel.de>
	<cf6355060907200605x2333a17cs762955fad01c0028@mail.gmail.com>
	<4A647835.9040303@behnel.de>
Message-ID: <cf6355060907200834y374545c0p57732cc941fd188c@mail.gmail.com>

>>> No, the codec layer only recodes the characters, so you'd get a UTF-8 encoded "&quot;" byte string as result.

Makes sense. I was hoping that was the case :)

Asking more regarding possibilities than your personal inclinations
and time allowances, would it be very difficult to add a `character`
index attribute (startchpos, endchpos etc) to nodes?  I was think
something akin to the sourceline attributes Would this be possible
purely from lxml/cython land or would libxml2 need to be patched?
Being a novice programmer I have little C or Cython experience however
it would be an interesting and motivating project to learn on.

As `unicode` is the future of python `text` a character based index
would be useful (admittedly for not that many uses) regardless of
encoding?  For my use case, character based positions would be
perfect.

Please forgive any misconceptions driving boneheaded questions.

From jlovell at nwesd.org  Mon Jul 20 18:45:45 2009
From: jlovell at nwesd.org (John Lovell)
Date: Mon, 20 Jul 2009 09:45:45 -0700
Subject: [lxml-dev] lxml.objectify.deannotate refuses to clean nil nodes
In-Reply-To: <4A2A2D94.20607@behnel.de>
References: <8dbf5ac80906010911g13f04a6ek14bf0510c5db80da@mail.gmail.com>
	<8dbf5ac80906010921r41427cedt49b084011d856d9@mail.gmail.com>
	<20090602075902.62950@gmx.net> <4A257C51.2050203@behnel.de>
	<20090603065858.115690@gmx.net> <4A2627DA.5000104@behnel.de>
	<20090604124227.28860@gmx.net>
	<36741390b07e0cf6fc7f93834031fb3c.squirrel@groupware.dvs.informatik.tu-darmstadt.de>
	<BBACAC0139BC3F4A996A25A71248730E0A6CDEDCE9@AGENCY-EM-01.esd189.org>
	<8dbf5ac80906041023y3e9a3f69pe2b21311da14d7b2@mail.gmail.com>
	<4A2A2D94.20607@behnel.de>
Message-ID: <BBACAC0139BC3F4A996A25A71248730E0A6DFB91DC@AGENCY-EM-01.esd189.org>

Stefan:

Has their been any action on this?  I really want to help and I am really swamped.

Sorry,

John W. Lovell
Web Applications Engineer
Northwest Educational Service District
1601 R Avenue
Anacortes, WA 98221
(360) 299-4086
jlovell at nwesd.org
 
www.nwesd.org
Together We Can ...


-----Original Message-----
From: Stefan Behnel [mailto:stefan_ml at behnel.de] 
Sent: Saturday, June 06, 2009 1:49 AM
To: Robert Pierce; John Lovell
Cc: lxml-dev at codespeak.net
Subject: Re: [lxml-dev] lxml.objectify.deannotate refuses to clean nil nodes

Hi,

Robert Pierce wrote:
> On Thu, Jun 4, 2009 at 8:30 AM, John Lovell wrote:
>> Stefan Behnel wrote:
>>> - strip_attributes(tree, *attribute_names)
>>>   remove all named attributes from a tree
>>>
>>> - strip_elements(tree, *element_names)
>>>   remove all named elements from a tree, including their subtrees (alt:
>>>   "strip_subtrees")
>>>
>>> - strip_tags(tree, *element_names)
>>>   remove all named elements from a tree, merging their children and text
>>>   content into their parents

Done:

https://codespeak.net/viewvc/?view=rev&revision=65612
https://codespeak.net/viewvc/lxml/trunk/src/lxml/cleanup.pxi?view=markup&pathrev=65612

>>> Comments?
>>
>> My comments would be: brilliant, useful, wonderful!
>
> I agree!  Solves my problem and then some.

Since you two seem to be very happy about this feature, what about writing up some docs/doctests for it?

A new section here sounds like the right place:

http://codespeak.net/svn/lxml/trunk/doc/api.txt
-> http://codespeak.net/lxml/api.html

Maybe the tutorial could also benefit from a short reference.

Holger, could you replace the current deannotate() implementation in lxml.objectify and add the xsl:nil cleanup option as we discussed? I expect it to be a little slower than before due to the more general implementation. If you have some code at your hands to benchmark it, please do.

Unless Ian (or someone else) beats me to it, I'll also look through lxml.html next week to check for places where this can be used. For example, clean.py looks like an obvious candidate.

Stefan

From stefan_ml at behnel.de  Mon Jul 20 19:43:36 2009
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 20 Jul 2009 19:43:36 +0200
Subject: [lxml-dev] About the position of html parsing by HTML Target
 parser
In-Reply-To: <cf6355060907200834y374545c0p57732cc941fd188c@mail.gmail.com>
References: <cf6355060907170348m5ec44c58o80bdebbcb635ca52@mail.gmail.com>	
	<4A63669C.4050404@behnel.de>	
	<cf6355060907191845r405b24dfr4eadf394c5eedd26@mail.gmail.com>	
	<4A642134.7070206@behnel.de>	
	<cf6355060907200153u1049f88bnf016e2226e48373d@mail.gmail.com>	
	<4A644E4B.4070309@behnel.de>	
	<cf6355060907200605x2333a17cs762955fad01c0028@mail.gmail.com>	
	<4A647835.9040303@behnel.de>
	<cf6355060907200834y374545c0p57732cc941fd188c@mail.gmail.com>
Message-ID: <4A64ACC8.9010103@behnel.de>


Nicholas Dudfield wrote:
> Asking more regarding possibilities than your personal inclinations
> and time allowances, would it be very difficult to add a `character`
> index attribute (startchpos, endchpos etc) to nodes?  I was think
> something akin to the sourceline attributes Would this be possible
> purely from lxml/cython land or would libxml2 need to be patched?

The latter. Here's what libxml2 knows about a node:

http://xmlsoft.org/html/libxml-tree.html#xmlNode

So it doesn't remember any character positions, and it only knows source
line numbers up to 65535 (because of memory considerations).


> Being a novice programmer I have little C or Cython experience however
> it would be an interesting and motivating project to learn on.
> 
> As `unicode` is the future of python `text` a character based index
> would be useful (admittedly for not that many uses) regardless of
> encoding?  For my use case, character based positions would be
> perfect.

Certainly. However, it does require some work to recover this information
from inside the parser framework (due to recoding), so I doubt that it's
worth adding such a feature for 'general' use.

If you want to dig into this, you'll have to start reading through the
source code of libxml2 to figure out where this information could become
available. "xmlio.c" might be good place to start, as it implements the I/O
routines that copy between encoded buffers.


> Please forgive any misconceptions driving boneheaded questions.

It's always fine to ask.

Stefan

From ndudfield at gmail.com  Tue Jul 21 14:25:19 2009
From: ndudfield at gmail.com (Nicholas Dudfield)
Date: Tue, 21 Jul 2009 20:25:19 +0800
Subject: [lxml-dev] About the position of html parsing by HTML Target
	parser
In-Reply-To: <4A64ACC8.9010103@behnel.de>
References: <cf6355060907170348m5ec44c58o80bdebbcb635ca52@mail.gmail.com>
	<4A63669C.4050404@behnel.de>
	<cf6355060907191845r405b24dfr4eadf394c5eedd26@mail.gmail.com>
	<4A642134.7070206@behnel.de>
	<cf6355060907200153u1049f88bnf016e2226e48373d@mail.gmail.com>
	<4A644E4B.4070309@behnel.de>
	<cf6355060907200605x2333a17cs762955fad01c0028@mail.gmail.com>
	<4A647835.9040303@behnel.de>
	<cf6355060907200834y374545c0p57732cc941fd188c@mail.gmail.com>
	<4A64ACC8.9010103@behnel.de>
Message-ID: <cf6355060907210525i66f34003j5a7252d4e102cd57@mail.gmail.com>

> If you want to dig into this, you'll have to start reading through the
> source code of libxml2 to figure out where this information could become
> available. "xmlio.c" might be good place to start, as it implements the I/O
> routines that copy between encoded buffers.


Thanks again
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20090721/39b7781c/attachment.htm 

From qhlonline at 163.com  Wed Jul 22 11:42:48 2009
From: qhlonline at 163.com (qhlonline)
Date: Wed, 22 Jul 2009 17:42:48 +0800 (CST)
Subject: [lxml-dev] About the position of html parsing by HTML Target
 parser
In-Reply-To: <4A641758.1020303@behnel.de>
References: <4A641758.1020303@behnel.de> <4A63669C.4050404@behnel.de>
	<cf6355060907170348m5ec44c58o80bdebbcb635ca52@mail.gmail.com>
	<9148036.360291248060120782.JavaMail.coremail@bj163app72.163.com>
Message-ID: <12721869.818541248255768387.JavaMail.coremail@bj163app29.163.com>

2009-07-20?"Stefan Behnel" <stefan_ml at behnel.de> ?
>
>qhlonline wrote:
>> I have tried to alter the libxml2 source to add a callback telling the 
>> current position when an element parsed.
>
>Note that something that requires patching libxml2 will not make it into an
>lxml release.
>
>As you noted before, the parser context already provides this information
>at any time, not only when parsing elements. So adding a callback for it is
>not a sensible approach.
>
>I'm not even sure what this position means exactly. Is it (1) the byte
>position in the original (undecoded) data stream, (2) the byte position in
>the UTF-8 encoded parse stream, or (3) the character position in the XML
>stream?
>
>According to the libxml2 docs:
>
>	long nbChars : number of xmlChar processed
>
>This sounds like it's the second information. That would not be useful and
>shouldn't get exposed in lxml's API as it's rather error prone to rely on
>it: works for ASCII and UTF-8, obviously, may work for some other encodings
>depending on the data, but fails for most other streams. OTOH, the first
>and the third information /might/ be of interest, depending on your use
>case, but are not easily recovered from the information that the parser
>provides.
>
>
>> I have nerver compile cython source before. Can any body give me some
>> suggestion?
>
>If you just change lxml's sources, running setup.py will build it just as
>before. All you need to do is install Cython 0.11 or later.
>
>http://codespeak.net/lxml/build.html
>
>Stefan

Thanks for Stefan's suggestion?I have finished the alternation of libxml2 and lxml code to add callback which can fetch the current position of HTML tag when parsing with Target parser.
   Firstly, I have changed the libxml2 library, There are three files I shoud deal with:
?"parser.h", where all callbacks are declared, I have added my event declaration here:
typedef void (*getCurrentPosFunc) (void *ctx,
				const xmlChar *tagname,
				const unsigned long current_position );
 Then add this event to Handler structure defination, in struct _xmlSAXHandler and struct _xmlSAXHandlerV1, add the statement as:
   getCurrentPosFunc getCurrentPos;
Note, In struct _xmlSAXHandlerV1, the event shoud also be added in. I have thought of only the structure _xmlSAXHandler had been used since it was SAX2 and I didn't add anything to _xmlSAXHandlerV1, as a result, the changed lxml library will always report "Segmentation fault" error, Until I add my event to the "V1" structure. 
 Then the file "SAX2.c", There are three changes at all. The former two changes are adding event handler to parser init functions: in function xmlSAX2InitHtmlDefaultSAXHandler and function xmlSAXVersion, add statement as follow:
   hdlr->getCurrentPos= xmlGetCurrentPosition;
Then the third change, I have write the xmlGetCurrentPosition function myself, It is simplely a statement with ";", doing nothing at all.I add this function to avoid my self defined event affecting libxml2's DOM parsers.
   The third file to deal with is "HTMLparser.c", in the function htmlParseStartTag, add my event handler here to get the position:
   ctxt->sax->getCurrentPos(ctxt->userData, name, ctxt->nbChars);
The third parameter is the current position. Althouth I have changed the normal HTML parsing function, It will not affect the DOM parser, because I have defined an event handler doing nothing in the former file.
   Recompile the libxml2 library and install it, Now libxml2 support getting SAX parser's position.
   Then some source of lxml files should also be altered: 
   In "xmlparser.pxd", adding getCurrentPos event declaration according to that in "parser.h", add an event to xmlSAXHandler in this file too.
   In "saxparser.pxi", Add new event in enum _SaxParserEvents:
   SAX_EVENT_POSITION = 64 
   Then add source code to class _SaxParserTarget and class _SaxParserContext according to context events, It is not very hard. At last in this source file, add a function to convert  C event handler to Python event handler. Because my event handler parameters are integers and string, They are supported by Cython of automatic converting, So it is not hard either.
  To support lxml target parser, I have modified the third file "parsertarget.pxi". In this file , In defination of class _PythonSaxParserTarget, adding a member as follow:
    cdef object _target_position
In this class I have changed the __init__ function to add a new event handler called "position" to all target parsers. and then add a handler as member function in this class, with the statement like : 
       return self._target_position(tag?current_position)
At last, recompile the lxml library and install it:
      python setup.py build
      python setup.py install
  Now the new lxml library support a new event handler called "position", It takes two parameters?a tagname and its position in the HTML file being parsed.
    This method is usefull for my task of getting a target parser support parsing position. But I don't know whether my change will take in new bugs, and, as Stefan has said, if lxml library or libxml2 library upgrades, all changes should be redo. So it is not suggested if no same requirement.
   ?And if somebody has deep understanding about libxml2 library?I wan't to ask a question. During my changing the libxml2 source, When I did not add my self-defined event handler to _xmlSAXHandlerV1 structure and function xmlSAXVersion, libxml2 will still support self-defined SAX parser, But it didn't support DOM parsers. It will report "Segmentation fault". GDB tracking found that in function  xmlNewDocPI of "tree.c" had called the xmlStrdup function which raised the "Segmentation fault". Why? and lxml's target parser will also raise this signal, Is there any relationship?
   ?Thanks.
 Best regards!

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20090722/56c9c3bd/attachment.htm 

From hillshum at gmail.com  Thu Jul 23 02:17:21 2009
From: hillshum at gmail.com (Hilton Shumway)
Date: Wed, 22 Jul 2009 18:17:21 -0600
Subject: [lxml-dev] Compile for Linux-PowerPC Problems
Message-ID: <1adeddf0907221717i6bdb8c39qf82de6c030dafd6a@mail.gmail.com>

Hello, I'm trying to get lxml to compile on my g3 iBook with Xubuntu Jaunty
without success.
I get quite a bit of errors compiling. I pastebined a selection of them at
http://omploader.org/vMjBzNw They are all quite similar to those ones there.

Does anyone know what might be the issue?
Hilton

Send me all the forwards/chain mails you want. No guaranties I will freak
out, cry, continue, or otherwise do anything with (or even read) them
though.
A selected quote: A wise man will make more opportunities than he finds. -
Sir Francis Bacon
A random quote:
Joan Crawford  - "I, Joan Crawford, I believe in the dollar. Everything I
earn, I spend." -
http://www.brainyquote.com/quotes/authors/j/joan_crawford.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20090722/c0d76b2f/attachment-0001.htm 

From qhlonline at 163.com  Thu Jul 23 08:03:08 2009
From: qhlonline at 163.com (qhlonline)
Date: Thu, 23 Jul 2009 14:03:08 +0800 (CST)
Subject: [lxml-dev] About the position of html parsing by HTML Target
 parser
In-Reply-To: <12721869.818541248255768387.JavaMail.coremail@bj163app29.163.com>
References: <12721869.818541248255768387.JavaMail.coremail@bj163app29.163.com>
	<4A641758.1020303@behnel.de> <4A63669C.4050404@behnel.de>
	<cf6355060907170348m5ec44c58o80bdebbcb635ca52@mail.gmail.com>
	<9148036.360291248060120782.JavaMail.coremail@bj163app72.163.com>
Message-ID: <10791685.218701248328988576.JavaMail.coremail@bj163app87.163.com>




2009-07-22?qhlonline <qhlonline at 163.com> ?

2009-07-20?"Stefan Behnel" <stefan_ml at behnel.de> ?
>
>qhlonline wrote:
>> I have tried to alter the libxml2 source to add a callback telling the 
>> current position when an element parsed.
>
>Note that something that requires patching libxml2 will not make it into an
>lxml release.
>
>As you noted before, the parser context already provides this information
>at any time, not only when parsing elements. So adding a callback for it is
>not a sensible approach.
>
>I'm not even sure what this position means exactly. Is it (1) the byte
>position in the original (undecoded) data stream, (2) the byte position in
>the UTF-8 encoded parse stream, or (3) the character position in the XML
>stream?
>
>According to the libxml2 docs:
>
>	long nbChars : number of xmlChar processed
>
>This sounds like it's the second information. That would not be useful and
>shouldn't get exposed in lxml's API as it's rather error prone to rely on
>it: works for ASCII and UTF-8, obviously, may work for some other encodings
>depending on the data, but fails for most other streams. OTOH, the first
>and the third information /might/ be of interest, depending on your use
>case, but are not easily recovered from the information that the parser
>provides.
>
>
>> I have nerver compile cython source before. Can any body give me some
>> suggestion?
>
>If you just change lxml's sources, running setup.py will build it just as
>before. All you need to do is install Cython 0.11 or later.
>
>http://codespeak.net/lxml/build.html
>
>Stefan

Thanks for Stefan's suggestion?I have finished the alternation of libxml2 and lxml code to add callback which can fetch the current position of HTML tag when parsing with Target parser.
   Firstly, I have changed the libxml2 library, There are three files I shoud deal with:
?"parser.h", where all callbacks are declared, I have added my event declaration here:
typedef void (*getCurrentPosFunc) (void *ctx,
				const xmlChar *tagname,
				const unsigned long current_position );
 Then add this event to Handler structure defination, in struct _xmlSAXHandler and struct _xmlSAXHandlerV1, add the statement as:
   getCurrentPosFunc getCurrentPos;
Note, In struct _xmlSAXHandlerV1, the event shoud also be added in. I have thought of only the structure _xmlSAXHandler had been used since it was SAX2 and I didn't add anything to _xmlSAXHandlerV1, as a result, the changed lxml library will always report "Segmentation fault" error, Until I add my event to the "V1" structure. 
 Then the file "SAX2.c", There are three changes at all. The former two changes are adding event handler to parser init functions: in function xmlSAX2InitHtmlDefaultSAXHandler and function xmlSAXVersion, add statement as follow:
   hdlr->getCurrentPos= xmlGetCurrentPosition;
Then the third change, I have write the xmlGetCurrentPosition function myself, It is simplely a statement with ";", doing nothing at all.I add this function to avoid my self defined event affecting libxml2's DOM parsers.
   The third file to deal with is "HTMLparser.c", in the function htmlParseStartTag, add my event handler here to get the position:
   ctxt->sax->getCurrentPos(ctxt->userData, name, ctxt->nbChars);
The third parameter is the current position. Althouth I have changed the normal HTML parsing function, It will not affect the DOM parser, because I have defined an event handler doing nothing in the former file.
   Recompile the libxml2 library and install it, Now libxml2 support getting SAX parser's position.
   Then some source of lxml files should also be altered: 
   In "xmlparser.pxd", adding getCurrentPos event declaration according to that in "parser.h", add an event to xmlSAXHandler in this file too.
   In "saxparser.pxi", Add new event in enum _SaxParserEvents:
   SAX_EVENT_POSITION = 64 
   Then add source code to class _SaxParserTarget and class _SaxParserContext according to context events, It is not very hard. At last in this source file, add a function to convert  C event handler to Python event handler. Because my event handler parameters are integers and string, They are supported by Cython of automatic converting, So it is not hard either.
  To support lxml target parser, I have modified the third file "parsertarget.pxi". In this file , In defination of class _PythonSaxParserTarget, adding a member as follow:
    cdef object _target_position
In this class I have changed the __init__ function to add a new event handler called "position" to all target parsers. and then add a handler as member function in this class, with the statement like : 
       return self._target_position(tag?current_position)
At last, recompile the lxml library and install it:
      python setup.py build
      python setup.py install
  Now the new lxml library support a new event handler called "position", It takes two parameters?a tagname and its position in the HTML file being parsed.
    This method is usefull for my task of getting a target parser support parsing position. But I don't know whether my change will take in new bugs, and, as Stefan has said, if lxml library or libxml2 library upgrades, all changes should be redo. So it is not suggested if no same requirement.
   ?And if somebody has deep understanding about libxml2 library?I wan't to ask a question. During my changing the libxml2 source, When I did not add my self-defined event handler to _xmlSAXHandlerV1 structure and function xmlSAXVersion, libxml2 will still support self-defined SAX parser, But it didn't support DOM parsers. It will report "Segmentation fault". GDB tracking found that in function  xmlNewDocPI of "tree.c" had called the xmlStrdup function which raised the "Segmentation fault". Why? and lxml's target parser will also raise this signal, Is there any relationship?
   ?Thanks.
 Best regards!





Hi all?
    The modification on libxml2 file "SAX2.c" and "SAX2.h" are not necessary for lxml target parser to support my position event. I add this just to avoid affecting libxml2 DOM parser. But experiment had shown that even if we do nothing on those tow files, The parsing process will be fine. Namely, In libxml2 I only have to add event defination and activate it in htmlParseStartTag function with the position parameter I need. Other works can be done in lxml target parser's callback named "position".

Best regards? 
 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20090723/45f0ced4/attachment.htm 

From daniel.albeseder at tttech.com  Thu Jul 23 14:30:07 2009
From: daniel.albeseder at tttech.com (Daniel Albeseder)
Date: Thu, 23 Jul 2009 14:30:07 +0200
Subject: [lxml-dev] Jython and XPointer support
Message-ID: <1248352207.20640.107.camel@tttdal>

Hi!

I wonder if there is, or will be any Jython support of lxml? We are
currently evaluating several possibilities to support several
XML-features in Python and Jython together. This includes XPath, XSLT,
XPointer. The last one is supported by libxml2 but I have not found any
support inside lxml. Have I missed something? Will there be XPointer
support in lxml in the future?

About Jython: There is of course the possibility to use JNA to access
the libxml2 library natively, but I do not comprehend how to access lxml
from within Jython, since it is not a "normal" shared library, but a
shared object created for CPython in Cython. Does anyone have any hint
about that?

Best regards
Daniel



From arubinst at library.umass.edu  Thu Jul 23 17:42:17 2009
From: arubinst at library.umass.edu (Aaron Rubinstein)
Date: Thu, 23 Jul 2009 11:42:17 -0400
Subject: [lxml-dev] Parsing namespace declarations
Message-ID: <4A6884D9.5070006@library.umass.edu>

Hello,

I'm trying to test for the presence of a specific namespace declaration 
in my document root but the namespace declarations don't seem to appear 
in the root.attrib dictionary.

Here is an example of my doc root:

<mods:mods version="3.3" xmlns:mods="http://www.loc.gov/mods/v3" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xsi:schemaLocation="http://www.loc.gov/mods/v3 
http://www.loc.gov/standards/mods/v3/mods-3-3.xsd" 
xmlns:xlink="http://www.w3.org/1999/xlink">

Does lxml provide a way to parse the namespace declarations?

Thanks for your help!



From yassin at ezbakhe.es  Fri Jul 24 00:19:47 2009
From: yassin at ezbakhe.es (Yassin Ezbakhe)
Date: Fri, 24 Jul 2009 00:19:47 +0200
Subject: [lxml-dev] lxml
Message-ID: <5d8599fb0907231519k2b00fe81v7dd1bc177801bf9f@mail.gmail.com>

Hi, I'm using Python 2.6.2 (Windows) and lxml 2.2.2.

When I run the following piece of code, it gets stuck.

import lxml.etree as et
s = "<root><a><b/><c/></a><d/><e/></root>"
xml = et.fromstring(s)
print xml[0] # prints element a
print xml[2] # prints element e
print xml[3] # it should raise an out of range exception, but it gets stuck

What is the problem? In the original ElementTree implementation, I get an
IndexError.

Thanks,
Yassin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20090724/63fe7d3d/attachment.htm 

From friedel at translate.org.za  Fri Jul 24 11:18:09 2009
From: friedel at translate.org.za (F Wolff)
Date: Fri, 24 Jul 2009 11:18:09 +0200
Subject: [lxml-dev] lxml
In-Reply-To: <5d8599fb0907231519k2b00fe81v7dd1bc177801bf9f@mail.gmail.com>
References: <5d8599fb0907231519k2b00fe81v7dd1bc177801bf9f@mail.gmail.com>
Message-ID: <1248427089.4814.58.camel@localhost.localdomain>

Op Vr, 2009-07-24 om 00:19 +0200 skryf Yassin Ezbakhe:
> 
> Hi, I'm using Python 2.6.2 (Windows) and lxml 2.2.2.
> 
> 
> When I run the following piece of code, it gets stuck.
> 
> 
> import lxml.etree as et
> s = "<root><a><b/><c/></a><d/><e/></root>"
> xml = et.fromstring(s)
> print xml[0] # prints element a
> print xml[2] # prints element e
> print xml[3] # it should raise an out of range exception, but it gets
> stuck
> 
> 
> What is the problem? In the original ElementTree implementation, I get
> an IndexError.

Hallo Yassin

I get the correct IndexError with lxml 2.1.5 in case that helps.

Friedel


--
Recently on my blog:
http://translate.org.za/blogs/friedel/en/content/firefox-35-released


From adam at varud.com  Fri Jul 24 16:30:03 2009
From: adam at varud.com (Adam Nelson)
Date: Fri, 24 Jul 2009 14:30:03 +0000 (UTC)
Subject: [lxml-dev] Getting 'user-visible' text from HTML
Message-ID: <loom.20090724T142530-109@post.gmane.org>

Is there a shortcut method (or even a pasted script) that allows lxml to get all 
the 'user-visible' text?  

I'm writing a screen scraper that then takes that text and looks for 
banned words next to an 
advertiser's content - and therefore I need to run a regular 
expression on everything a user 
might see (including meta keywords, etc...) but I don't care
 about the actual tags 
themselves, or urls, etc...

Right now, I'm just doing the regex on the entire HTML block.

Thanks,
Adam


From ted at milo.com  Fri Jul 24 18:00:41 2009
From: ted at milo.com (Ted Dziuba)
Date: Fri, 24 Jul 2009 09:00:41 -0700
Subject: [lxml-dev] Getting 'user-visible' text from HTML
In-Reply-To: <loom.20090724T142530-109@post.gmane.org>
References: <loom.20090724T142530-109@post.gmane.org>
Message-ID: <6451ccbf0907240900o5334a133s1a24987bbf3bd458@mail.gmail.com>

So, a user will likely never see meta keywords displayed because they're an
attribute of the <meta> tag.  However, if you just want all of the text
contained within a document, try this:

from lxml import html
tree = html.fromstring(text_of_html)
all_text = tree.text_content()

And then run your regexes against all_text.  One caveat is that
text_content() recursively gives you all text children of the nodes, so it
will pull in JavaScript contained within <script> tags.  If that's a
problem, you can come up with some minor hackery to pull <script> nodes out
of the tree.

Ted


On Fri, Jul 24, 2009 at 7:30 AM, Adam Nelson <adam at varud.com> wrote:

> Is there a shortcut method (or even a pasted script) that allows lxml to
> get all
> the 'user-visible' text?
>
> I'm writing a screen scraper that then takes that text and looks for
> banned words next to an
> advertiser's content - and therefore I need to run a regular
> expression on everything a user
> might see (including meta keywords, etc...) but I don't care
>  about the actual tags
> themselves, or urls, etc...
>
> Right now, I'm just doing the regex on the entire HTML block.
>
> Thanks,
> Adam
>
> _______________________________________________
> lxml-dev mailing list
> lxml-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/lxml-dev
>



-- 
Ted Dziuba
Co-Founder and Engineer

Milo.com, Inc.
165 University Avenue
Palo Alto, CA, 94301
http://milo.com

Cell: (609)-665-2639
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://codespeak.net/pipermail/lxml-dev/attachments/20090724/681a343d/attachment.htm 

From terry_n_brown at yahoo.com  Fri Jul 24 18:05:16 2009
From: terry_n_brown at yahoo.com (Terry Brown)
Date: Fri, 24 Jul 2009 11:05:16 -0500
Subject: [lxml-dev] Getting 'user-visible' text from HTML
In-Reply-To: <loom.20090724T142530-109@post.gmane.org>
References: <loom.20090724T142530-109@post.gmane.org>
Message-ID: <20090724110516.500d74e0@nrri.umn.edu>

On Fri, 24 Jul 2009 14:30:03 +0000 (UTC)
Adam Nelson <adam at varud.com> wrote:

> Is there a shortcut method (or even a pasted script) that allows lxml
> to get all the 'user-visible' text?  

doc.xpath("//text()") should return a list of every piece of text in
the html.

Cheers -Terry

> I'm writing a screen scraper that then takes that text and looks for 
> banned words next to an 
> advertiser's content - and therefore I need to run a regular 
> expression on everything a user 
> might see (including meta keywords, etc...) but I don't care
>  about the actual tags 
> themselves, or urls, etc...
> 
> Right now, I'm just doing the regex on the entire HTML block.
> 
> Thanks,
> Adam
> 
> _______________________________________________
> lxml-dev mailing list
> lxml-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/lxml-dev
> 


From jkrukoff at ltgc.com  Fri Jul 24 22:19:25 2009
From: jkrukoff at ltgc.com (John Krukoff)
Date: Fri, 24 Jul 2009 14:19:25 -0600
Subject: [lxml-dev] docinfo.doctype doesn't include internal entities?
In-Reply-To: <49E97750.1010903@behnel.de>
References: <a7a2b76b0904171948w4572155q733f4ce3005e1d80@mail.gmail.com>
	<49E97750.1010903@behnel.de>
Message-ID: <1248466765.30250.38.camel@localhost.localdomain>

On Sat, 2009-04-18 at 08:46 +0200, Stefan Behnel wrote:
> Sidnei da Silva wrote:
> > I am looking for a way to output internal entities that have been
> > parsed from the original document when writing out a tree, but
> > apparently this is not exposed in any attribute.
> > 
> > Here's an example:
> > 
> > {{{
> > import lxml.etree
> > 
> > document = """<?xml version="1.0"?>
> >   <!DOCTYPE application [
> >     <!ENTITY nbsp "\&#160;">
> >   ]>
> >   <application>&nbsp;</application>
> > """
> > 
> > 
> > tree = lxml.etree.fromstring(document)
> > print tree.getroottree().docinfo.doctype
> > }}}
> > 
> > I would expect this to output:
> > {{{
> >   <!DOCTYPE application [
> >     <!ENTITY nbsp "\&#160;">
> >   ]>
> > }}}
> > 
> > But instead it gives me:
> > 
> > {{{
> >   <!DOCTYPE application>
> > }}}
> > 
> > Is it a bug or I'm not looking at the right place?
> 
> What you are looking for is the internal subset of the document, which is
> not (really) part of the DOCTYPE itself. It's available through the
> "docinfo.internalDTD" property. However, lxml.etree doesn't expose the
> content of the DTD, so this is currently only usable for validation (i.e.
> not very helpful in your case).
> 
> What you could try is to parse the document without resolving the entities,
> then traverse the Entity elements and collect their names in a set. That
> will not give you the resolved entity values, though...
> 
> I think it would be nice if tostring() could serialise DTDs, but I doubt
> that there are so many use cases for that. In your case, you'd then have to
> parse the DTD yourself, which you could also do by clearing the root node
> and serialising the document to unicode.
> 
> Stefan
> _______________________________________________
> lxml-dev mailing list
> lxml-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/lxml-dev

Hello,

I'm sorry to be resurrecting an ancient thread, but it seemed the
easiest way to bring up the fact that I've recently come up with a use
case for exactly the feature mentioned here, namely serializing internal
DTD subsets.

I've been writing a round trip converter for a personal XML shorthand,
and internal DTD subsets are the only thing I haven't been able to come
up with a good workaround for being able to pull out of the original
document, do my modifications, and create an identical new document
from. So far the best I've been able to do is destructively modify a
copy of the document to the point where I've a reasonable chance of
writing my own string parser to pull out the internal DTD subset, a
parser which is looking unfortunately complicated to be able to deal
with multiple top level elements (comments being the most common for
me).

Really, I'd be happy in the simplest case if docinfo.doctype included
the internal DTD subset exactly as defined in the original document
(parsed or not), as then it'd be reasonably easy to at least stick
things back together at the string level. Is this the kind of thing
you'd accept a wishlist bugtracker item on?

Somewhat related, I was surprised to discover that the TreeBuilder API
doesn't deal well with additional top level elements. For example, 

Python 2.6.2 (r262:71600, May 29 2009, 09:48:09) 
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from lxml import etree
>>> t = etree.TreeBuilder( )
>>> t.comment( 'first' )
<!--first-->
>>> t.start( 'element', {} )
<Element element at b773aa7c>
>>> e = t.end( 'element' )
>>> t.comment( 'last' )
<!--last-->
>>> t.close( )
<!--last-->

I was expecting that the TreeBuilder class was actually for creating XML
documents, but I guess it's actually meant for XML fragments? I expected
this:

>>> etree.tostring( e )
'<element/>'

But when I went to retrieve the root tree, I was surprised that my other
top level elements were missing.

>>> etree.tostring( e.getroottree( ) )
'<element/>'

But since it also allows you to create multiple top level elements with
start and end, it obviously doesn't care about the restrictions of
creating an XML document. This actually mattered to me, as my XML here
makes heavy use of top level comments for documentation.

Anyway, not a big deal, I figured I'd just keep track of all the top
level parts myself, and create an ElementTree manually. Only, it doesn't
look like there's any way to add top level elements like doctype
information or comments or processing instructions without first
serializing all the parts to strings, sticking them together, and
running them back through the parser again. I can't find any way to
duplicate what the parser does through the API, and am hoping I'm just
missing some obscure corner of the ElementTree API that would let me
build this programatically.

Sure, I can read them using ElementTree.getroot( ).itersiblings( ), but
I couldn't find any way to create them or doctype information without
resorting to string parsing.

So, yeah, really just a diary of my misunderstandings of the TreeBuilder
API, and my attempts to work around it. Hopefully I'm missing something
obvious.

I should also add a note here, in thanks of all the effort you've put
into lxml. I've been using it daily for over 2 years now, and I can't
imagine programming XML with python using anything else. Even the
original ElementTree seems limited in comparison now, much less ending
up in javascript looking at DOM code. It's been the most useful and best
supported library I depend on. Thank you.

-- 
John Krukoff <jkrukoff at ltgc.com>
Land Title Guarantee Company


From daniel.albeseder at tttech.com  Mon Jul 27 10:49:25 2009
From: daniel.albeseder at tttech.com (Daniel Albeseder)
Date: Mon, 27 Jul 2009 10:49:25 +0200
Subject: [lxml-dev] id function of xpath and parseid
Message-ID: <1248684566.20640.137.camel@tttdal>

Hi!

I wonder why there is no parseid function inside the objectify module?
Is there a reason, why this is only in etree?

Additionally I wonder, how the `id` function of XPath does work with
lxml. I created a schema-aware parser, which reads an XML-file, where
some attributes are declared as xs:ID inside the schema. However the
`xpath` method always returns an empty list of nodes, even if the IDs
given are inside the XML.

Example:
----

from lxml import etree

xsd = \
"""
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema">
	<xsd:element name="root" type="rootType"/>
	<xsd:complexType name="rootType">
		<xsd:sequence>
			<xsd:element name="foo" type="fooType" maxOccurs="unbounded"/>
		</xsd:sequence>
		<xsd:attribute name="id" type="xsd:ID" use="required"/>
	</xsd:complexType>
	<xsd:complexType name="fooType">
		<xsd:attribute name="id" type="xsd:ID" use="optional"/>
	</xsd:complexType>
</xsd:schema>
"""
xml = """<root id="a"><foo id="b"/><foo id="bar"/><foo/></root>"""

schema = etree.XMLSchema (etree.XML (xsd))
parser = etree.XMLParser (schema = schema)
root   = etree.XML (xml, parser = parser)

for i in ["a", "b", "bar"] :
    print root.xpath ("id('%s')" % i)

print "---"

print root.get ("id")
for el in root :
    print el.get ("id")

---

Returns just:

[]
[]
[]
---
a
b
bar
None

where the first three lists should not be empty IMHO. Can anyone help
me?

Best regards
Daniel



From cz at gocept.com  Mon Jul 27 10:51:27 2009
From: cz at gocept.com (Christian Zagrodnick)
Date: Mon, 27 Jul 2009 10:51:27 +0200
Subject: [lxml-dev] Weakref support?
References: <h3ul9u$rpr$1@ger.gmane.org> <4A631EE0.3080200@behnel.de>
	<h411a6$2j9$1@ger.gmane.org> <4A641B0E.6090303@behnel.de>
Message-ID: <h4jpqf$331$1@ger.gmane.org>

On 2009-07-20 09:21:50 +0200, Stefan Behnel <stefan_ml at behnel.de> said:

> 
> Christian Zagrodnick wrote:
>> On 2009-07-19 15:25:52 +0200, Stefan Behnel said:
>>> Is there a specific use case you have in mind?
>> 
>> Actually all I'd need is a reference to the tree. I don't think
>> weak-refs to all elements are necessary or useful. Well, maybe they
>> are, but I don't need them :)
> 
> That sounds better. :)
> 
> Feel free to add a
> 
> 	cdef object __weakref__
> 
> to the _ElementTree class (below the _context_node attribute) and see if
> that works for you. Please report back if it turns out be useful.

Hm. Creating a weakref to the tree (getroottree()) works. But since the 
etree itself doesn't keep a reference to that object but creates new 
onces for each call it doesn't help.

I suppose there is no way of changeing that behaviour, is there?


-- 
Christian Zagrodnick ? cz at gocept.com
gocept gmbh & co. kg ? forsterstra?e 29 ? 06112 halle (saale) ? germany
http://gocept.com ? tel +49 345 1229889 4 ? fax +49 345 1229889 1
Zope and Plone consulting and development



From cz at gocept.com  Mon Jul 27 11:12:09 2009
From: cz at gocept.com (Christian Zagrodnick)
Date: Mon, 27 Jul 2009 11:12:09 +0200
Subject: [lxml-dev] Weakref support?
In-Reply-To: <4A641B0E.6090303@behnel.de>
References: <h3ul9u$rpr$1@ger.gmane.org> <4A631EE0.3080200@behnel.de>
	<h411a6$2j9$1@ger.gmane.org> <4A641B0E.6090303@behnel.de>
Message-ID: <399DAB2B-B8B5-4019-8157-0CEE0F492A8D@gocept.com>


On 20.07.2009, at 09:21, Stefan Behnel wrote:

>
> Christian Zagrodnick wrote:
>> On 2009-07-19 15:25:52 +0200, Stefan Behnel said:
>>> Is there a specific use case you have in mind?
>>
>> Actually all I'd need is a reference to the tree. I don't think
>> weak-refs to all elements are necessary or useful. Well, maybe they
>> are, but I don't need them :)
>
> That sounds better. :)

Okay, just for the fun of it I added weakref to element.

With   : Ran 1025 tests in 15.731s
Without: Ran 1025 tests in 15.584s

In the latter case the weakref test fails of course. Patch attached.

I don't know what the memory implications are, though?


-- 
Christian Zagrodnick ? cz at gocept.com
gocept gmbh & co. kg ? forsterstra?e 29 ? 06112 halle (saale) ? germany
http://gocept.com ? tel +49 345 1229889 0 ? fax +49 345 1229889 1
Zope and Plone consulting and development
-------------- next part --------------
A non-text attachment was scrubbed...
Name: weakref.patch
Type: application/octet-stream
Size: 1270 bytes
Desc: not available
Url : http://codespeak.net/pipermail/lxml-dev/attachments/20090727/b247e43c/attachment-0001.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2474 bytes
Desc: not available
Url : http://codespeak.net/pipermail/lxml-dev/attachments/20090727/b247e43c/attachment-0001.bin 

From daniel.albeseder at tttech.com  Mon Jul 27 11:43:17 2009
From: daniel.albeseder at tttech.com (Daniel Albeseder)
Date: Mon, 27 Jul 2009 11:43:17 +0200
Subject: [lxml-dev] id function of xpath and parseid
In-Reply-To: <1248684566.20640.137.camel@tttdal>
References: <1248684566.20640.137.camel@tttdal>
Message-ID: <1248687797.20640.143.camel@tttdal>

On Mon, 2009-07-27 at 10:49 +0200, Daniel Albeseder wrote:

> Additionally I wonder, how the `id` function of XPath does work with
> lxml. I created a schema-aware parser, which reads an XML-file, where
> some attributes are declared as xs:ID inside the schema. However the
> `xpath` method always returns an empty list of nodes, even if the IDs
> given are inside the XML.

I just found in the archive, that it only seems to work with either an
xml:id attribute or a given DTD. However since XML Schema is the modern
way to restrict XML-files, and XML schema also has the xsd:ID attribute
type, I see not reason, why this does not work.

The XPath specification does talk about DTDs, thats correct, but since
this specification is older than the XML Schema document and XML schema
is designed to be a superset of DTD (and even to replace it), it sounds
strange, that some features only work for DTDs.

Best regards
Daniel




From bwooster47 at gmail.com  Thu Jul 30 19:47:42 2009
From: bwooster47 at gmail.com (B Wooster)
Date: Thu, 30 Jul 2009 13:47:42 -0400
Subject: [lxml-dev] Whitespace foiling pretty_print - any fix?
Message-ID: <204abd770907301047t1c805ab8k30ce8e7b6821e9cf@mail.gmail.com>

f an XML element has whitespace along with sub elements, pretty_print
does not work well.

This is probably a XML whitespace issue, but is there any way to work
around this issue?

Here's example code and the output it prints, and what I would like to get:

from lxml import etree as ET
root = ET.XML("""<a>
       <b>test</b>
       </a>""")
ET.SubElement(root, "c")
ET.SubElement(root, "d")
print ET.tostring(root, pretty_print=True)
# This prints:
"""<a>
       <b>test</b>
       <c/><d/></a>
"""
# Would like:
"""<a>
       <b>test</b>
       <c/>
       <d/>
  </a>
"""

From cz at gocept.com  Fri Jul 31 07:57:17 2009
From: cz at gocept.com (Christian Zagrodnick)
Date: Fri, 31 Jul 2009 07:57:17 +0200
Subject: [lxml-dev] Bug with whitespace in namespaces
Message-ID: <h4u13t$79j$1@ger.gmane.org>

Hoi,

it is possible to create invalid XML with lxml:

>>> import lxml.etree
>>> import lxml.objectify
>>> xml = lxml.objectify.XML('<a/>')
>>> xml.set('{a b}c', 'foo')             # This should fail!
>>> lxml.etree.tostring(xml)
'<a xmlns:ns0="a b" ns0:c="foo"/>'
>>> lxml.objectify.fromstring(lxml.etree.tostring(xml))
Traceback (most recent call last):
  ...
  File "parser.pxi", line 625, in lxml.etree._handleParseResult 
(src/lxml/lxml.etree.c:64741)
  File "parser.pxi", line 565, in lxml.etree._raiseParseError 
(src/lxml/lxml.etree.c:64084)
lxml.etree.XMLSyntaxError: xmlns:ns0: 'a b' is not a valid URI, line 1, 
column 13
>>>


Regards,
-- 
Christian Zagrodnick ? cz at gocept.com
gocept gmbh & co. kg ? forsterstra?e 29 ? 06112 halle (saale) ? germany
http://gocept.com ? tel +49 345 1229889 4 ? fax +49 345 1229889 1
Zope and Plone consulting and development



From cz at gocept.com  Fri Jul 31 07:59:34 2009
From: cz at gocept.com (Christian Zagrodnick)
Date: Fri, 31 Jul 2009 07:59:34 +0200
Subject: [lxml-dev] Parsing namespace declarations
References: <4A6884D9.5070006@library.umass.edu>
Message-ID: <h4u186$79j$2@ger.gmane.org>

On 2009-07-23 17:42:17 +0200, Aaron Rubinstein 
<arubinst at library.umass.edu> said:

> Hello,
> 
> I'm trying to test for the presence of a specific namespace declaration
> in my document root but the namespace declarations don't seem to appear
> in the root.attrib dictionary.
> 
> Here is an example of my doc root:
> 
> <mods:mods version="3.3" xmlns:mods="http://www.loc.gov/mods/v3"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://www.loc.gov/mods/v3
> http://www.loc.gov/standards/mods/v3/mods-3-3.xsd"
> xmlns:xlink="http://www.w3.org/1999/xlink">
> 
> Does lxml provide a way to parse the namespace declarations?

Aren't those in the nsmap attribute? Or was it namespaces?


-- 
Christian Zagrodnick ? cz at gocept.com
gocept gmbh & co. kg ? forsterstra?e 29 ? 06112 halle (saale) ? germany
http://gocept.com ? tel +49 345 1229889 4 ? fax +49 345 1229889 1
Zope and Plone consulting and development



From daniel.albeseder at tttech.com  Fri Jul 31 13:17:53 2009
From: daniel.albeseder at tttech.com (Daniel Albeseder)
Date: Fri, 31 Jul 2009 13:17:53 +0200
Subject: [lxml-dev] Strange segmentation fault if class inherited from
 objectify.ObjectifiedElement
Message-ID: <1249039073.20640.371.camel@tttdal>

I just tried this and got an segmentation fault :-(

Python 2.6.2 (release26-maint, Apr 19 2009, 01:58:18) 
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from lxml import objectify
>>> from lxml import etree
>>> 
>>> print etree.LXML_VERSION, etree.LIBXML_VERSION
(2, 2, 2, 0) (2, 6, 32)
>>> 
>>> class test (objectify.ObjectifiedElement) :
...     pass
... 
>>> good = objectify.Element ("abc")
>>> print type (good), repr (good)
<type 'lxml.objectify.ObjectifiedElement'> <Element abc at 7f0797587fc8>
>>> 
>>> bad  = test ("abc")
Segmentation fault

I read that you have to instantiate a parser with objectify.makeparser
and use the method makeelement from there. However I would expect that
at least not Segmentation Fault happens, if someone is using the normal
inheritance as used to be.

regards
Daniel



From herve.cauwelier at free.fr  Fri Jul 31 17:39:45 2009
From: herve.cauwelier at free.fr (=?UTF-8?B?SGVydsOpIENhdXdlbGllcg==?=)
Date: Fri, 31 Jul 2009 17:39:45 +0200
Subject: [lxml-dev] automatic attribute unicode decode?
Message-ID: <4A731041.4020908@free.fr>

Hi,

I'm quite puzzled by the following excerpt:

>>> from lxml import etree
>>> r = etree.fromstring('<root toto="fran?ais" titi="ascii" tata="1"/>'
>>> r.attrib
{'titi': 'ascii', 'toto': u'fran\xe7ais', 'tata': '1'}

In a bare document with no encoding declaration, lxml has decoded itself
a string that did not match the ascii table (what heuristic did it
use?). Now I have three attributes of two different types. I wonder why
the integer was not decoded. ;-)

I actually found this in a real-world document with encoding and
namespaces (An ODF xml part).

Is this a bug to report and how to circumvent it?

Thanks,

Herv?

From jlovell at nwesd.org  Fri Jul 31 18:42:01 2009
From: jlovell at nwesd.org (John Lovell)
Date: Fri, 31 Jul 2009 09:42:01 -0700
Subject: [lxml-dev] automatic attribute unicode decode?
In-Reply-To: <4A731041.4020908@free.fr>
References: <4A731041.4020908@free.fr>
Message-ID: <BBACAC0139BC3F4A996A25A71248730E0A6DFB9213@AGENCY-EM-01.esd189.org>

Herv?:

I keep hearing that LXML defaults to UTF-8 so that is probably the heuristic used.

Good luck,

John W. Lovell
Web Applications Engineer
Northwest Educational Service District
1601 R Avenue
Anacortes, WA 98221
(360) 299-4086
jlovell at nwesd.org
 
www.nwesd.org
Together We Can ...
 

-----Original Message-----
From: lxml-dev-bounces at codespeak.net [mailto:lxml-dev-bounces at codespeak.net] On Behalf Of Herv? Cauwelier
Sent: Friday, July 31, 2009 8:40 AM
To: lxml-dev at codespeak.net
Subject: [lxml-dev] automatic attribute unicode decode?

Hi,

I'm quite puzzled by the following excerpt:

>>> from lxml import etree
>>> r = etree.fromstring('<root toto="fran?ais" titi="ascii" tata="1"/>'
>>> r.attrib
{'titi': 'ascii', 'toto': u'fran\xe7ais', 'tata': '1'}

In a bare document with no encoding declaration, lxml has decoded itself a string that did not match the ascii table (what heuristic did it use?). Now I have three attributes of two different types. I wonder why the integer was not decoded. ;-)

I actually found this in a real-world document with encoding and namespaces (An ODF xml part).

Is this a bug to report and how to circumvent it?

Thanks,

Herv?
_______________________________________________
lxml-dev mailing list
lxml-dev at codespeak.net
http://codespeak.net/mailman/listinfo/lxml-dev