[Haskell-cafe] Slow IO

[Donald Bruce Stewart]

daniel.is.fischer:
> Am Dienstag, 12. September 2006 22:26 schrieben Sie:
> > Daniel Fischer wrote:
> > > The programme consumed more and more memory (according to top),
> > > kswapd started to have a higher CPU-percentage than my programme,
> > > programme died, system yelling 'Speicherzugriffsfehler', top displays
> > > 'kswapd<defunct>'.
> > > I believe that means my programme demanded more memory than I have
> > > available (only 256MB RAM + 800MB swap). Is that a segfault or what is
> > > the correct term?
> > >
> > > That is probably due to (apart from the stupidity of my IO-code) the
> > > large overhead of Haskell lists.
> >
> > Most certainly not.  I'm pretty sure this is to a bug in your code.
> > Something retains a data structure which is actually unneeded.  Probably
> 
> Apparently. And my money is on a load of lines from the file (of which I need 
> only the first and last Char).
> 
> > a case of "foldl" where "foldl'" should be used or a "try" in Parsec
> > code where it should be left out or a lot of "updateWiths" to a Map,
> > etc.  Or it could be a bad choice of data structure.  I bet, it's the
> > map you're using to represent the graph (which you don't even need to
> > represent at all, btw).
> 
> No foldl nor parsec around. I represent the graph as a
> 
> UArray (Char,Char) Int 
> 
> (I've switched to Int for the index type, too, when tuning the code), so that 
> shouldn't use much memory (array size is 676).
> The array is built via accumArray, I hope that's sufficiently efficient
> (though now I use unsafeAccumArrayUArray, that's faster).
> 
> How could I solve the problem without representing the graph in some way?
> Possibly that could be done more efficiently than I do it, but I can't imagine 
> how to do it without representing the graph in some data structure.
> >
> > > So the chunk of the file which easily fits into my
> > > RAM in ByteString form is too large as a list of ordinary Strings.
> >
> > The chunk of file should never need to fit into RAM.  If that's a
> > problem, you also forgot to prime a crucial "foldl".
> >
> 
> Forgive the stupid question, but where if not RAM would the chunk currently 
> processed reside?

I agree. Some problems simply require you to hold large strings in
memory. And for those, [Char] conks out around 5-10M (try reversing a
10M [Char]).

-- Don