[Haskell-cafe] How on Earth Do You Reason about Space?

[Brandon Moore]

Wait, do ByteStrings show up on a heap profile, if the space is
allocated with malloc?

Anyway, I think my tests still show that the memory used by the
process doesn't grow simply by adding more data, if you
are no longer added keys to the map.



----- Original Message -----
> From: Brandon Moore <brandon_m_moore at yahoo.com>
> To: Aleksandar Dimitrov <aleks.dimitrov at googlemail.com>; "haskell-cafe at haskell.org" <haskell-cafe at haskell.org>
> Cc: 
> Sent: Tuesday, May 31, 2011 1:43 PM
> Subject: Re: [Haskell-cafe] How on Earth Do You Reason about Space?
> 
> I can't reproduce heap usage growing with the
> size of the input file.
> 
> I made a word list from Project Gutenberg's
> copy of "War and Peace" by
> 
> tr -sc '[[:alpha:]]' '\n' < pg2600.txt > words.txt
> 
> Using 1, 25, or 1000 repetitions of this ~3MB wordlist
> shows about 100MB of address space used according
> to top, and no more than 5MB or so of haskell heap
> used according to the memory profile, with a flat
> memory profile.
> 
> 
> Is your memory usage growing with the size of the input
> file, or the size of the histogram?
> 
> I was worried data sharing might mean your keys
> retain entire 64K chunks of the input. However, it
> seems enumLines depends on the StringLike ByteString
> instance, which just converts to and from String.
> That can't be efficient, but I suppose it avoids excessive sharing.
> 
> The other thing that occurs to me is that the total size of
> your keys would also be approximately the size of the input
> file if you were using plain text without each word split onto
> a separate line.
> 
> Brandon
>