Fwd: Garbage collection

[Jost Berthold]

Hi Facundo,

You are completely right, the CAF named "g" might be accessed at any 
time during the program execution. Parallel Haskell systems with 
distributed heap (and runtime-supported serialisation) need to keep all 
CAFS alive for this reason.

Some comments inline along your mail:

>   While working in the StaticPointers language extension [1], we
> found we have some unusual CAFs which can be accessed after some
> periods of time where there is no reference to them.
>
>   For instance, the following program when compiled contains no
> reference to `g`. `g` is actually looked up at runtime in symbol
> tables via the call to `deRefStaticPtr`.
> g :: String
> g = "hello"
>
> main =
>   deRefStaticPtr (static g) >>= putStrLn

The bad scenario is certainly one where CAF g (a static thunk) is 
evaluated during execution (i.e. turned into an indirection into the 
heap), and then garbage-collected, as it might not be referenced by any 
(runnable) thread.
This GC does not revert the indirection into a thunk. Why should it, 
there are no references to it, right? ;-)

So technically, your example might need to involve using g (and forceful 
GC at a certain point during execution):

main = putStrLn g >> performGC >>
        deRefStaticPtr (static g) >>= putStrLn
	
> Desugars to:
>
> g :: String
> g = "hello"
>
> main =
     putStrLn g >> performGC >>
>   deRefStaticPtr (StaticPtr (StaticName "" "Main" "g")) >>= putStrLn

During performGC, there would be no reference to g from any thread's stack.
I am of course assuming that g is indeed a thunk, and not statically 
evaluated to a string during compilation (I am unsure whether GHC would 
do that).

> In principle, there is nothing stopping the garbage collector from
> reclaiming the closure of `g` before it is dynamically looked up.

Maybe a stupid question, sorry: The RemoteTable generated using 
template-haskell in CH without XStaticPointers would keep CAFs alive. So 
the XStaticPointers extension does not entail using such a table?

> We are considering using StablePtrs to preserve `g`. So the code
> desugars instead to:
>
> g :: String
> g = "hello"
>
> main =
>   deRefStaticPtr (let x = StaticPtr (StaticName "" "Main" "g")
>                          in unsafePerformIO $ newStablePtr g >> return x
>                         ) >>= putStrLn
>

Another question: Would it be sufficient to desugar "static g" to
	g `seq` StaticPtr(StaticName "" "Main" "g")
instead of introducing a stable ptr and all that?
AFter all, g is a CAF, so it is anyway "stable" in some sense, as long 
as it is alive.

However, I conjecture that this only fixes the one-node test, not the 
actual use case (sending "static" stuff over the wire).

Finally, there is a flag keepCAFs in the runtime which you can set to 
secure the CAFs for the entire run. The parallel runtimes for Eden and 
GUM (as well as my "packman" serialisation) do this.

Yes, obviously, this opens a memory leak. It would be nice to not "keep" 
but "revert" the CAFs (ghci does that) but on a "by-need" basis when a 
simple GC cannot reclaim enough space; this would plug the mem.leak. 
This requires a modification to the GHC runtime system, and it is 
unclear _which_ CAFs to prefer when starting to revert. But I think it 
would be a more generally useful feature.
However, this discussion (runtime/GC features) leads us straight out of 
the design goals of "-XStaticPointers", I guess...

Best regards,
Jost