[Haskell-cafe] Patterns for processing large but finite streams

[dm-list-haskell-cafe at scs.stanford.edu]

At Fri, 1 Jul 2011 09:39:32 +0400,
Eugene Kirpichov wrote:
> 
> Hi,
> 
> I'm rewriting timeplot to avoid holding the whole input in memory, and
> naturally a problem arises:
> 
> How to represent large but finite streams and functions that process
> them, returning other streams or some kinds of aggregate values?
> 
> Examples:
> * Adjacent differences of a stream of numbers
> * Given a stream of numbers with times, split it into buckets by time
> of given width and produce a stream of (bucket, 50%,75% and 90%
> quantiles in this bucket)
> * Sum a stream of numbers
> 
> Is this, perhaps, what comonads are for? Or iteratees?

Sounds like a good job for iteratees.  Summing a stream of numbers is
just an Iteratee.  Transcoding a stream into another stream is a job
for an Inum (Iteratee-enumerator) or enumeratee, depending on which
package's nomenclature you use.  You have three implementations to
choose from:

 - http://hackage.haskell.org/package/iteratee    (original)

 - http://hackage.haskell.org/package/enumerator  (John Milikin's re-write)

 - http://hackage.haskell.org/package/iterIO      (my 3rd-generation attempt)

David