Binary IO

[Glynn Clements]

David Brown wrote:

> >> And hGet/PutWord8 are fast enough for most situations.
> >
> > Are you certain? Which interesting applications do you know
> > that read and write one byte at a time? I can't speak for
> > "most" situations, but in those situations where I needed
> > binary I/O this API would have been impracticable.
> 
> I'm going to chime in agreement with the disagreement.
> 
> I have one application which uses single character binary I/O.  Because it  
> works on small files, it borders on usefuleness.  As time goes on, the  
> size of these inputs grows, and now the tool has become nearly worthless.
> 
> I seem to run into this kind of thing a lot with Haskell.  Dominic's ASN  
> library is useful to me, however I'm finding I'll probably just have to  
> use it as a template for new code.  It works fine for what designed for,  
> parsing keys and such, but I'm looking to use it to represent gigabytes of  
> data.  Processing data like that one Word8 at a time isn't practical.

Personally, I doubt that Haskell will ever be practical for processing
very large amounts of data (e.g. larger than your system's RAM).

When processing large amounts of data, rule #1 is to do as little as
possible to most of the data; don't even read it into memory if you
can avoid it (e.g. create an index and use lseek() as much as
possible). You certainly don't want to be boxing/unboxing gigabytes.

-- 
Glynn Clements <glynn at gclements.plus.com>