[Haskell-cafe] Re: FFI and heap memory usage limit

[Duncan Coutts]

On Fri, 2009-06-26 at 13:00 +0100, Simon Marlow wrote:

> Maybe bzlib allocates using malloc()?  That would not be tracked by 
> GHC's memory management, but could cause OOM.

Yes it does.

> Another problem is that if you ask for a large amount of memory in one 
> go, the request is usually honoured immediately, and then we GC shortly 
> afterward.  If this is the problem for you, please submit a ticket and 
> I'll see whether it can be changed.  You could work around it by calling 
> System.Mem.performGC just before allocating the memory.

What I'd like is a way to inform the GC about how much memory there is
attached to a ForeignPtr. The amount would contribute to the GC's
decision on when to collect. This would solve the bzlib case and also
the same problem in other bindings like cairo (big pixmaps) and gtk2hs
(lots and lots of medium-sized foreign objects).

Obviously it's not a total solution since not all bindings can easily
discover the size of the foreign allocations, but in the case of zlib,
bzlib, cairo and gtk2hs it would be possible. It also does not have to
be 100% accurate.

For implementation, perhaps keep in the rts an extra foreign memory
count (which gets used by the GC in its accounting). Then on a foreign
allocation we add to the count. An extra finaliser on the ForeignPtr
could then decrement the foreign memory count. Another, more local,
approach might be to keep the accounting info in a data structure
directly, eg:

data ForeignAccounting#

which would contain a count. This type would be recognised by the GC.
These could be embedded into data structures anywhere to account for the
size of foreign allocations, in particular into the ForeignPtrContents.

Duncan