ByteStrings and the ram barrier

[Donald Bruce Stewart]

bulat.ziganshin:
> Hello Donald,
> 
> Friday, May 12, 2006, 10:07:47 AM, you wrote:
> > Last week Duncan Coutts had the bright idea to write
> > Data.ByteString.Lazy, a newtyped [ByteString], which would allow
> > constant space processing of chunks of strict ByteString arrays.
> 
> i have somewhat similar idea - implement getContents variany that
> returns list of LINES in file, represented as [ByteString]
> 
> >     Data.ByteString.Lazy (5M chunks)
> >         constant heap, total time 11.44 minutes, 34% cpu
> 
> try to use smaller chunks, don't forget that P4 has 512-1024k cache

After tuning the chunk size to the cache size, the filter 'e' test now
runs in:
    4.52 minutes, 53% cpu.

Which I think is pretty spectacular for 10 gigabytes of data.

-- Don