[lxml-dev] XSLT and threading

[Stefan Behnel]

Hi,

eric casteleijn wrote:
> Martijn Faassen wrote:
>> Hey Stefan,
>>
>> Yeah, we realize you need a test case. Eric worked quite hard to try
>> to isolate the problem, but no luck yet so far. Hopefully we'll be
>> able to tell you more next week. The situation seems to occur when
>> custom resolvers are involved, and it's conceivable it's not really a
>> threading problem at all.
> 
> I *finally* succeeded in building a small (semi) reliable test that
> shows the behavior. In the end, it has nothing to do with custom
> resolvers or even xsl:imports, which had me barking up quite a few wrong
> trees. Enclosed is a small python file, which when run, segfaults pretty
> reliably for me.
> 
> I don't think it's doing anything special anymore except the threading.
> If anyone can help with this, I will be very grateful!

I investigated this a little more. The way lxml now handles passing trees
between threads is that it migrates all tag names from the dictionary of the
thread that created the tree to the dictionary of the target thread.

It seems that there is more that has to be done here. Your test specifically
crashes when trying to free the "xml" special name (or maybe "xmlns", not
sure). Breakpointing on xmlDictLookup() showed me that libxml2 also stores
(some?) namespace URIs and prefixes in the dictionary, at least during
parsing. It doesn't do this when declaring namespaces later on with
xmlNewNs(). How's that for consistency...

So the fix is even a bit more involved than I initially thought. It may take a
bit to figure out how to get this straight.

Stefan