[Haskell-cafe] Increasing memory use in stream computation
Thu Oct 10 13:02:32 UTC 2013
(Sorry for the long email.)
Summary: why does the attached program have non-constant memory use?
==== Introduction ====
I've written a program to do a big computation. Unfortunately, the computation
takes a very long time (expectedly), and the memory use increases slowly
(unexpectedly), until it fills up the entire memory and swap space of the
computer (many gigabytes).
The rough structure of the program is:
? create a long (up to 20 million) list of objects;
? compute a number for each of those objects;
? compute the sum of the resulting list.
I switched the intermediate data structure from a list to a Stream (from the
stream-fusion package), hoping to fix the memory issue. It decreased both the
memory use and the rate of its increase, but after a long time, the program
still uses up all available memory.
==== A simple program
After many hours of cutting down my program, I now have a small program
(attached) that shows the same behaviour. It uses only said stream-fusion
package, and vector. (I haven't yet tried to cut out the use of vector. I hope
it is irrelevant, because all vectors are of fixed small size.)
I compile the program with ghc-7.6.1 using
> ghc --make -threaded -rtsopts -with-rtsopts="-M1G -K128M" -O2 -main-is
> Test.main Test
The rts options may not be strictly necessary: I added them at some point to
allow the use of multiple cores, and to prevent the program from crashing the
machine by using all available memory.
When running the program, the resident memory quickly grows to about 3.5 MB
(which I am fine with); this stays constant for a long time, but after about 7
minutes, it starts to grow further. The growth is slow, but I really would
hope this program to run in constant memory.
==== The code ====
Note that I added an instance for Monad Stream, using concatMap. This is
implicitly used in the definition of the big stream.
The source of Data.Stream contains many alternative implementations of concat
and concatMap, and alludes to the difficulty of making it fuse properly. Could
it be that the fusion did not succeed in this case?
Thanks for any help!
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 895 bytes
Desc: not available
More information about the Haskell-Cafe