Interrupt interruptible foreign calls on HS exit

[Edward Z. Yang]

Hello Andreas,

Yes, it seems that deleteAllThreads() is too late.

The problem is that you need to defer the major GC until all the logical
threads (now including interruptible FFI threads) have terminated
themselves.  This happens automatically for Haskell threads, because
they are either (1) chilling in a run queue, in which case we don't
need to do anything, or (2) being processed by a capability, in which
case the GC will wait until the capability is done processing the thread
and gets the memo (since it needs to acquire all locks.)

However, in the case of safe FFI calls, these give up the capability
before going off to C-land, so trying to acquire all the capabilities
won't block on the FFI calls coming back.  So you'll need to do
something else.

It's not necessarily clear what the best course of action here is: if
an interruptible thread fails to be interrupted, should we keep waiting?
(Maybe we should; that's how we handle non-preemptible Haskell threads
too.)  In any case, you'll need some way of waiting on the FFI calls,
which might need some more gyrations as well.

Cheers,
Edward

Excerpts from Andreas Voellmy's message of 2014-08-02 21:28:31 +0100:
> I tried to go ahead and call throwTo() instead of throwToSingleThreaded()
> for threads BlockedOnCCall_Interruptible state during the shutdown
> sequence. Unfortunately something goes wrong with this change. I haven't
> tracked it down yet, but it looks like the following happens...
> 
> hs_exit() eventually result in a call to scheduleDoGC(), which does
> acquireAllCapabilities()
> and then deleteAllThreads() interrupts interruptible foreign calls. Those
> foreign calls come back and call waitForReturnCapability() but get stuck
> here:
> 
> if (!task->wakeup) waitCondition(&task->cond, &task->lock);
> 
> I guess the scheduleDoGC is blocking the interrupted Haskell threads from
> finishing.
> 
> One possible fix is to have the returning foreign call see that we are in
> the exit sequence and avoid trying to return to the Haskell caller - I
> guess it can just exit. I tried adding some code in resumeThread() to exit
> if sched_state is SCHED_INTERRUPTING or SCHED_SHUTTING_DOWN, but this
> caused more trouble, so it seems that it's not a simple change.
> 
> 
> 
> 
> 
> On Sat, Aug 2, 2014 at 1:55 PM, Andreas Voellmy <andreas.voellmy at gmail.com>
> wrote:
> 
> > Thanks Edward! Another question...
> >
> > deleteThread() calls throwToSingleThreaded(). I can update this so that it
> > also calls throwToSingleThreaded() in the case
> > of BlockedOnCCall_Interruptible (currently it explicitly excludes this
> > case), but this doesn't solve the problem, because throwToSingleThreaded()
> > doesn't seem to interrupt blocked calls at all. That functionality is in
> > throwTo(), which is not called by throwToSingleThreaded(). Why are we using
> > throwToSingleThreaded() in deleteThread() rather than throwTo()? Can I
> > switch deleteThread() to use throwTo()? Or should I use throwTo() in
> > deleteThread() only for the special case of BlockedOnCCall_Interruptible?
> >  Or should throwToSingleThreaded() be updated to do the same thing that
> > throwTo does for the case of BlockedOnCCall_Interruptible?
> >
> > Thanks,
> > Andi
> >
> >
> > On Wed, Jul 30, 2014 at 6:57 PM, Edward Z. Yang <ezyang at mit.edu> wrote:
> >
> >> Recalling when I implemented this functionality, I think not
> >> interrupting threads in the exit sequence was just an oversight,
> >> and I think we could implement it.  Seems reasonable to me.
> >>
> >> Edward
> >>
> >> Excerpts from Andreas Voellmy's message of 2014-07-30 23:49:24 +0100:
> >> > Hi GHCers,
> >> >
> >> > I've been looking into issue #9284, which boils down to getting certain
> >> > foreign calls issued by HS threads to finish (i.e. return) in the exit
> >> > sequence of forkProcess.
> >> >
> >> > There are several options for solving the particular problem in #9284;
> >> one
> >> > option is to issue the particular foreign calls causing that issue as
> >> > "interruptible" and then have the exit sequence interrupt interruptible
> >> > foreign calls.
> >> >
> >> > The exit sequence, starting from hs_exit(), goes through hs_exit_(),
> >> > exitScheduler(), scheduleDoGC(), deleteAllThreads(), and then
> >> > deleteThread(), where deleteThread is this:
> >> >
> >> > static void
> >> > deleteThread (Capability *cap STG_UNUSED, StgTSO *tso)
> >> > {
> >> >     // NOTE: must only be called on a TSO that we have exclusive
> >> >     // access to, because we will call throwToSingleThreaded() below.
> >> >     // The TSO must be on the run queue of the Capability we own, or
> >> >     // we must own all Capabilities.
> >> >   if (tso->why_blocked != BlockedOnCCall &&
> >> > tso->why_blocked != BlockedOnCCall_Interruptible) {
> >> >         throwToSingleThreaded(tso->cap,tso,NULL);
> >> >     }
> >> > }
> >> >
> >> > So it looks like interruptible foreign calls are not interrupted in the
> >> > exit sequence.
> >> >
> >> > Is there a good reason why we have this behavior? Could we change it to
> >> > interrupt TSO's with why_blocked == BlockedOnCCall_Interruptible in the
> >> > exit sequence?
> >> >
> >> > Thanks,
> >> > Andi
> >> >
> >> > P.S. It looks like this was introduced in commit
> >> > 83d563cb9ede0ba792836e529b1e2929db926355.
> >>
> >
> >