[py-dev] execnet hangs

[holger krekel]

Hi Armin, 

On Fri, Apr 08, 2005 at 11:52 +0100, Armin Rigo wrote:
> On Fri, Apr 08, 2005 at 12:41:16PM +0200, holger krekel wrote:
> > Full ack.  However, nobody forces us to deadlock if we are 
> > exceeding the number of maximum threads. 
> 
> My point is that perfectly reasonable examples cause deadlocks because of this
> "feature".  Detecting the problem and throwing an exception is not a good
> solution either.  It causes nice small programs to start crashing when you
> scale them up.
> 
> I don't see why we can't just document: "remote_exec() creates a new thread on
> the remote side", and be done with it.  A program needs to seriously screw up
> to produce a ton of calls to remote_exec() and starve the resources; it is
> just the same as a program that, by error, calls thread.start_new_thread() far
> too many times.  It could occur, but I've never seen that problem in practice
> and I'm definitely on the opinion that any default bound like 10 -- even if
> you can disable it -- is going to cause bugs.  (And 10 is worse than 2 because
> you need to scale more before you see the bug.)
> 
> I don't mind if you want to enforce a bound in some situations but the default
> should be no bound.

Ok, i am fine with making maxthreads=None (meaning unbound) the default. 
But allowing to bound the number of threads makes sense as well
(unless it is too hard to implement :-).  Or could we actually detect 
the "can't create a thread anymore" case on the remote side and raise 
a ThreadsExhausted exception to the remote_exec() caller [*]? Seems to be 
the best of all worlds ...

    holger

[*] which one would only notice when trying to channel.*() something 
    because remote_exec() is fully asynchronous.  It is helpful to get an 
    exception instead of just blocking and not knowing why, isn't it?