[Haskell-cafe] Re: FFI and heap memory usage limit

[Simon Marlow]

On 22/06/2009 23:48, Bulat Ziganshin wrote:
> Hello Marcin,
>
> Tuesday, June 23, 2009, 2:31:13 AM, you wrote:
>
>>          Now this took an odd turn, because the simulation started crashing with
>> out-of-memory errors _after_ completing (during bz2 compression). I'm fairly
>> certain this is a GC/FFI bug, because increasing the max heap didn't help.
>> Moving the bz2 compression to a separate process provided a reasonable
>> solution.
>>          What I think is happening is that after the simulation completes, almost all
>> of the available memory (within the -M limit) is filled with garbage. Then I
>> run bzlib which tries to allocate more memory (from behind FFI?) to compress
>> the results, which in turn causes an out-of-memory error instead of
>> triggering a GC collection.
>
> i can propose a quick fix - alloc 10 mb using allocBytes before
> starting your algorithm, and free it just before starting bzlib. it
> may help
>
> i agree that this looks like a deficiency of memory allocator. it's
> better to write at ghc-users maillist (or at least make a copy to
> Simon Marlow) to attract attention to your message

Maybe bzlib allocates using malloc()?  That would not be tracked by 
GHC's memory management, but could cause OOM.

Another problem is that if you ask for a large amount of memory in one 
go, the request is usually honoured immediately, and then we GC shortly 
afterward.  If this is the problem for you, please submit a ticket and 
I'll see whether it can be changed.  You could work around it by calling 
System.Mem.performGC just before allocating the memory.

Cheers,
	Simon