ByteStrings and the ram barrier

[Donald Bruce Stewart]

bulat.ziganshin:
> Hello Donald,
> 
> Friday, May 12, 2006, 10:07:47 AM, you wrote:
> > Last week Duncan Coutts had the bright idea to write
> > Data.ByteString.Lazy, a newtyped [ByteString], which would allow
> > constant space processing of chunks of strict ByteString arrays.
> 
> i have somewhat similar idea - implement getContents variany that
> returns list of LINES in file, represented as [ByteString]
> 
> >     Data.ByteString.Lazy (5M chunks)
> >         constant heap, total time 11.44 minutes, 34% cpu
> 
> try to use smaller chunks, don't forget that P4 has 512-1024k cache

In fact, here's a graph of time versus chunk size, for my Pentium M
laptop, with a 256k L2 cache:

    http://www.cse.unsw.edu.au/~dons/tmp/chunksize_v_cache.png

Lesson: chunksize should be between 0.5 and 1x the L2 cache, and things
get much worse as soon as you go beyond your cache size.

-- Don