Proposed change to ForeignPtr

[Alastair Reid]

> I appreciate it's not Malcolm Wallace's fault that NHC can't
> implement a form of dynamic wrapper which can be used at all times.
> However since I do not think this problem is going to go away, would
> it not be better to put it in the specification but admit that NHC
> can't do it?  Otherwise the FFI specification will be very good (as
> it is) if you want to call languages like C, but will be inadequate
> if you want to communicate with other languages which also know
> about things like concurrency and garbage-collection, such as
> Haskell or even Java.

I think the two of you are getting caught up in implementation details
(which always seem fixable) instead of looking at fundamentals (which
tend not to be).

The issue is that allowing finalizers to be Haskell code or to call
Haskell code forces us to add preemptive concurrency to every Haskell
implementation.  Hugs has cooperative concurrency but that isn't
enough, it has to be preemptive.

To see why, suppose Hugs is busy calculating sum [1..1000000] and a
garbage collection finds it has to run a Haskell function, it can't
run it straight away because that would preempt the original
execution.  

Likewise, it can't run it immediately that the GC ends - that's still
preemption.

Hugs is probably busy evaluating a + or (:) or the like when the GC
happens, what if we wait until that finishes and then run the
finalizers?  Maybe.  The world is in a cleaner state at this point but
it's still tricky.  And, again, you've just added most of the
machinery required for implementing preemptive concurrency.

Hugs does allow context switches in the IO monad.  Could we allow it
there?  Maybe, we could.  It wouldn't do you much good though because
Haskell can spend arbitrarily long running pure code so your
finalizers might not get run for a long time or maybe never.


So what's the objection to adding preemptive concurrency?

1) The goal was to be able to call foreign code and have it call us.
   It seems like there's something wrong if doing that forces you to
   add concurrency.

2) Preemptive concurrency is one of those things that you can't just
   paint on later - you really have to rewrite the whole thing from
   scratch because it's just too hard to find all the pieces that
   make the assumption that they are single-threaded.

3) Adding preemptive concurrency guarantees a steady source of hard to
   repeat bugs - people aren't very good at writing concurrent code.

Cooperative concurrency is much easier to handle because context
switches happen at times when both tasks are ready for it: the sender
because it knows it is passing control over, the receiver because it
put itself in a clean state before it handed over control.


You ask how garbage collection can work if you can't invoke it at
arbitrary times?  You ask what use foreign export is if you can't call
the exported function at arbitrary times?
[I hope I'm paraphrasing your questions appropriately]

Both questions start by assuming that you have Haskell and Foosh [or
whatever your other language is called] running in separate threads
(otherwise things wouldn't happen at arbitrary times).  Put them in a
single thread and there's no problem: you're not doing things at
arbitrary times, you're doing them at quite specific times: Foosh had
control and it called into Haskell or Haskell had control and it
called into Foosh.  

In other words, the problems you raise come from wanting preemptive
concurrency so it's no wonder that the ffi extension doesn't address
them.  We could start working on a concurrency extension but that's a
different beast.

That said, we probably do need a few hooks to make Haskell talk to
other GCs better.  See the paper I pointed at this afternoon for
a sketch of what you need if you're to avoid space leaks.

--
Alastair Reid                 alastair at reid-consulting-uk.ltd.uk  
Reid Consulting (UK) Limited  http://www.reid-consulting-uk.ltd.uk/alastair/