[Haskell-cafe] Slow IO
Donald Bruce Stewart
dons at cse.unsw.edu.au
Tue Sep 12 20:58:30 EDT 2006
> Am Dienstag, 12. September 2006 22:26 schrieben Sie:
> > Daniel Fischer wrote:
> > > The programme consumed more and more memory (according to top),
> > > kswapd started to have a higher CPU-percentage than my programme,
> > > programme died, system yelling 'Speicherzugriffsfehler', top displays
> > > 'kswapd<defunct>'.
> > > I believe that means my programme demanded more memory than I have
> > > available (only 256MB RAM + 800MB swap). Is that a segfault or what is
> > > the correct term?
> > >
> > > That is probably due to (apart from the stupidity of my IO-code) the
> > > large overhead of Haskell lists.
> > Most certainly not. I'm pretty sure this is to a bug in your code.
> > Something retains a data structure which is actually unneeded. Probably
> Apparently. And my money is on a load of lines from the file (of which I need
> only the first and last Char).
> > a case of "foldl" where "foldl'" should be used or a "try" in Parsec
> > code where it should be left out or a lot of "updateWiths" to a Map,
> > etc. Or it could be a bad choice of data structure. I bet, it's the
> > map you're using to represent the graph (which you don't even need to
> > represent at all, btw).
> No foldl nor parsec around. I represent the graph as a
> UArray (Char,Char) Int
> (I've switched to Int for the index type, too, when tuning the code), so that
> shouldn't use much memory (array size is 676).
> The array is built via accumArray, I hope that's sufficiently efficient
> (though now I use unsafeAccumArrayUArray, that's faster).
> How could I solve the problem without representing the graph in some way?
> Possibly that could be done more efficiently than I do it, but I can't imagine
> how to do it without representing the graph in some data structure.
> > > So the chunk of the file which easily fits into my
> > > RAM in ByteString form is too large as a list of ordinary Strings.
> > The chunk of file should never need to fit into RAM. If that's a
> > problem, you also forgot to prime a crucial "foldl".
> Forgive the stupid question, but where if not RAM would the chunk currently
> processed reside?
I agree. Some problems simply require you to hold large strings in
memory. And for those, [Char] conks out around 5-10M (try reversing a
More information about the Haskell-Cafe