[lxml-dev] lxml + mod_python: cannot unmarshal code objects in restricted execution mode

[Stefan Behnel]

David Danier wrote:
>>> As this takes some time I will post
>>> the result later.
>> Please do.
> 
> Seems to work properly. But I'm not really sure how bad
> "main_interpreter" is polluted now.

I wouldn't expect much (namespace) polution - unless there's real evidence
that this can become a problem.

And a crash is definitely a more important problem than namespace polution.


>> No way. There's a reason why it is there which is the same why we use it: it's
>> simple and usable. Using anything else would mean a lot of rewriting.
> 
> Thats sad. What are the chances that patches addressing this problem are
> accepted?
> (Must review the code first, but I would really like a clean solution here)

We always accept patches as long as there is general interest and/or a good
motivation behind them. But threading is pretty much an issue by itself in
lxml.etree. And the "simplified API" gives you a way to just say "release GIL
- call to libxml2 - acquire GIL" and "acquire GIL - run callback code - free
GIL". That's as easy as it can get - especially since Cython has support for
the latter nowadays. It is very unlikely that this can get any "cleaner" by
changing the thread-lock calls.


>> You might want to try compiling lxml with "--without-threading", though, which
>> disables concurrency support completely (i.e. not more GIL freeing).
> 
> Works, too. But I'm not really sure it it is a good idea to do so, as
> Py_NewInterpreter seems to create a thread, see
> http://www.python.org/doc/current/api/initialization.html#l2h-820. But I
> think this might not be a problem if not using a threaded Apache-MPM.

What this options does is that lxml.etree stops freeing the GIL internally
when calling into libxml2, which simply disables any concurrency as it keeps
the GIL until execution returns to Python code. Especially the (simplified)
Thread API is no longer used, so there should no longer be any threading issues.

Stefan