[Haskell-cafe] newbie question about list performance

[Jules Bean]

John Lato wrote:
> I'm working
> with moderate-sized files (tens to hundreds of MBs) that have some
> ascii header data followed by a bunch of 32-bit ints.

> but I don't know if [Int32] is actually the best choice.  It seems to me
> that something like a lazy list of strict arrays (analogous to a lazy
> bytestring) would be better.  

Depends on your data access pattern. If you access the words strictly 
linearly, from the beginning of the file to the end, and that's all, 
then [Int32] is absolutely fine. A list is a data-structure equivalent 
of a for loop; it's the correct structure if you are dealing with things 
linearly or nearly-linearly. If you were using adjacent words together, 
that would be fine too (as in, e.g., zip xs (tail xs)).

If your data access pattern is more scattered or random-access in style, 
then [Int32] does not scale well to 10s of MBs. If you keep the data 
around, the overhead for [] is inappropriate (around 600-800% memory 
usage overhead on [Int32]) and its performance guarantees are not good 
either, for random access. In this case, as a first approximation, I 
would be inclined to try a library which simple backended onto lazy 
bytestring. For example the 'index' operation to fetch a single word 
would fetch four bytes and bit-twiddle them into a word. If that doesn't 
give the high speed you're after, then perhaps something *like* LBS, 
i.e. foreignptr behind the scenes, but directly accessing word-at-a-time.

Jules