ByteStrings and the ram barrier

Donald Bruce Stewart dons at
Fri May 12 23:06:02 EDT 2006

> Hello Donald,
> Friday, May 12, 2006, 10:07:47 AM, you wrote:
> > Last week Duncan Coutts had the bright idea to write
> > Data.ByteString.Lazy, a newtyped [ByteString], which would allow
> > constant space processing of chunks of strict ByteString arrays.
> i have somewhat similar idea - implement getContents variany that
> returns list of LINES in file, represented as [ByteString]
> >     Data.ByteString.Lazy (5M chunks)
> >         constant heap, total time 11.44 minutes, 34% cpu
> try to use smaller chunks, don't forget that P4 has 512-1024k cache

In fact, here's a graph of time versus chunk size, for my Pentium M
laptop, with a 256k L2 cache:

Lesson: chunksize should be between 0.5 and 1x the L2 cache, and things
get much worse as soon as you go beyond your cache size.

-- Don

More information about the Libraries mailing list