ByteStrings and the ram barrier

[Tomasz Zielonka]

On Fri, May 12, 2006 at 04:07:47PM +1000, Donald Bruce Stewart wrote:
> The theory is that we'd be able to efficiently process large data, where
> both ByteString and [Char] fails, and open a new range of applications
> that could be handled successfully with Haskell.

My large data files are already divided into reasonably sized chunks
and I think this approach is quite widespread - at least Google also
processes much of their data in chunks.

To process my data with Haskell, I would have to be able to decode it
into records with efficiency close to what I achieve in C++ (say, at
least 80% as fast). Until now I managed to get 33% of it in reasonably
pure Haskell, which is not that bad IMO. However, I feel that I've hit a
barrier now and will have to use raw Ptr's or FFI. Maybe I could try
pushing it through the haskell-cafe optimisation ;-)

Anyway, the point is that large data tends to be divided into smaller
chunks not only because it's impossible to load the whole file into
memory, but also to allow random access, to help distributing the
computation over many computers, etc. So, I am not sure Haskell would
gain that much by being able to process terabytes of data in one go.

On the other hand, this is quite cool and I am probably wrong, being
concentrated on my needs.

Best regards
Tomasz