[Haskell-cafe] Re: FFI and heap memory usage limit

[Don Stewart]

duncan.coutts:
> On Fri, 2009-06-26 at 13:00 +0100, Simon Marlow wrote:
> 
> > Maybe bzlib allocates using malloc()?  That would not be tracked by 
> > GHC's memory management, but could cause OOM.
> 
> Yes it does.
> 
> > Another problem is that if you ask for a large amount of memory in one 
> > go, the request is usually honoured immediately, and then we GC shortly 
> > afterward.  If this is the problem for you, please submit a ticket and 
> > I'll see whether it can be changed.  You could work around it by calling 
> > System.Mem.performGC just before allocating the memory.
> 
> What I'd like is a way to inform the GC about how much memory there is
> attached to a ForeignPtr. The amount would contribute to the GC's
> decision on when to collect. This would solve the bzlib case and also
> the same problem in other bindings like cairo (big pixmaps) and gtk2hs
> (lots and lots of medium-sized foreign objects).
> 
> Obviously it's not a total solution since not all bindings can easily
> discover the size of the foreign allocations, but in the case of zlib,
> bzlib, cairo and gtk2hs it would be possible. It also does not have to
> be 100% accurate.
> 
> For implementation, perhaps keep in the rts an extra foreign memory
> count (which gets used by the GC in its accounting). Then on a foreign
> allocation we add to the count. An extra finaliser on the ForeignPtr
> could then decrement the foreign memory count. Another, more local,
> approach might be to keep the accounting info in a data structure
> directly, eg:
> 
> data ForeignAccounting#
> 
> which would contain a count. This type would be recognised by the GC.
> These could be embedded into data structures anywhere to account for the
> size of foreign allocations, in particular into the ForeignPtrContents.
> 


Sounds like the solution we thought through a few months ago. I'd
support this -- it would be useful.

-- Don