ghc releasing memory during compilation

[Simon Marlow]

Duncan Coutts wrote:
> On Mon, 2006-03-13 at 12:47 +0000, Simon Marlow wrote:
> 
> 
>>>Am I right in thinking that ghc's rts can free memory back to the system
>>>when the heap pressure reduces (at least if it's doing a compacting GC)?
>>
>>No, not at the moment.  One thing we planned to do but never got around 
>>to is to use madvise() to improve swapping behaviour when memory is 
>>tight (see the thread that Bulat refferred to).
>>
>>
>>>In this case if it can do so, it should be quite dramatic. It'd ought to
>>>be able to go from 400Mb back down to just a few Mb or so.
>>
>>Yes it ought to.
>>
>>A related problem is that the block allocator is really stupid.  It 
>>makes no attempt to reduce fragmentation, and in fact freeing blocks is 
>>O(n) because the freelist is kept sorted in address order.  This isn't 
>>usually an issue, although it has been reported to be noticeable with 
>>very large residencies (500M+).  It's on my list to fix at some point.
>>
>>I mention this because freeing all that memory might not be possible if 
>>it is highly fragmented.
> 
> 
> Ah, I hoped that if we were using the compacting GC then it might be
> able to defragment (since it is copying anyway) and thus free large
> contiguous blocks, eg munmap()ing whole MBlocks.

The compacting collector doesn't necessarily compact into contiguous 
memory, it just compacts towards the front of a chain of blocks.  The 
blocks may be anywhere in the address space.  If you're unlucky, every 
MBlock will have a few live blocks in it, and you won't be able to free 
a thing.  (although you could still madvise(MADV_FREE) the unused memory).

I believe the usage pattern of the GC means that this doesn't happen 
much in practice.  Lots of pinned objects and arrays will tend to 
increase fragmentation, because their memory isn't recycled during GC. 
If someone got the graphical front end back up again, we could have a 
look at the fragmentation in real time :)

Cheers,
	Simon