ByteStrings and the ram barrier
Donald Bruce Stewart
dons at cse.unsw.edu.au
Fri May 12 23:35:35 EDT 2006
> Hello Donald,
> Friday, May 12, 2006, 10:07:47 AM, you wrote:
> > Last week Duncan Coutts had the bright idea to write
> > Data.ByteString.Lazy, a newtyped [ByteString], which would allow
> > constant space processing of chunks of strict ByteString arrays.
> i have somewhat similar idea - implement getContents variany that
> returns list of LINES in file, represented as [ByteString]
> > Data.ByteString.Lazy (5M chunks)
> > constant heap, total time 11.44 minutes, 34% cpu
> try to use smaller chunks, don't forget that P4 has 512-1024k cache
After tuning the chunk size to the cache size, the filter 'e' test now
4.52 minutes, 53% cpu.
Which I think is pretty spectacular for 10 gigabytes of data.
More information about the Libraries