[Haskell-cafe] Slow IO

[Ketil Malde]

Daniel Fischer <daniel.is.fischer at web.de> writes:

> Maybe I've misused the word segfault.

I think so.  A segfault is the operating-system complaining about an
illegal memory access.  If you get them from Haskell, it is likely a
bug in the compiler or run-time system (or you were using unsafeAt, or
FFI). 

> The programme consumed more and more memory (according to top),
> kswapd started to have a higher CPU-percentage than my programme,
> programme died, system yelling 'Speicherzugriffsfehler', top displays 
> 'kswapd<defunct>'.

I find that swapping the GHC heap is not productive, so I tend to
limit memory to available RAM.  You can do that either by limiting
available memory to process in the shell (ulimit or limit), or by
specifying RTS options to the haskell program (typically +RTS -Mxxx,
where xxx is 80-90% of physical RAM).

>From the GHC docs (6.4.2):

   -Msize

    [Default: unlimited] Set the maximum heap size to size bytes. The

I thought the default was set according to limits? 

    heap normally grows and shrinks according to the memory
    requirements of the program. The only reason for having this
    option is to stop the heap growing without bound and filling up
    all the available swap space, which at the least will result in
    the program being summarily killed by the operating system. 

In my experience, a program which thrashes severely without heap
limits can run fine if you just limit the heap.  So it's not as much
an issue of 'filling up swap' vs. 'killed by the OS', but 'takes
forever, making the system unresponsive in the process' vs. 'tries
hard to complete in a timely fashion, or halts with an error'.  I much
prefer the latter.

> However the problem might be too little lazyness, because if I explicitly read 
> the file line by line, memory usage remains low enough -- but ByteString is 
> *much* faster anyway.

Right.  You should still try to consume the file lazily, and make sure
the data can get discarded and GC'ed when you have no further use for
it.  But a String is something like 8 or 12 bytes per character, a
ByteString gets you down to 1.

-k
-- 
If I haven't seen further, it is by standing in the footprints of giants