[Haskell-cafe] help optimizing memory usage for a program

[Bulat Ziganshin]

Hello Manlio,

Monday, March 2, 2009, 6:30:51 PM, you wrote:

> The process-data-1 program parse the entire dataset using about 1.4 GB
> of memory (3x increment).

> This is strange.
> The memory required is proportional to the number of ratings.
> It may be IntMap the culprit, or the garbage collector than does not 
> release the memory to the operating system (or, worse, does not 
> deallocate all used temporary memory).

ghc has 2 garbage collectors. first is copying, usied by default
(because it's faster): when previously allocated memory block
overflows, it collects used data by copying them into *new* data block
allocated from OS and then use old data block to satisfy further
memory requests.

let's calculate. if at GC moment your program has allocated 100 mb of
memory and only 50 mb was not a garbage, then memory usage will be 150
mb

another GC is compacting one - it's about 2x slower, but collects data
in-place. moreover, you may set up"growing factor". with a g.f. of
1.5, for example, memory will be collected once heap will become 1.5x
larger than real memory usage after last GC. this effectively
guarantees that memory overhead will never be over this factor

look at GHC manual, RTS switches section. and last - GHC never returns
memory to the OS, there should be ticket on this


-- 
Best regards,
 Bulat                            mailto:Bulat.Ziganshin at gmail.com