[Haskell-cafe] Conduits vs. lazy byte strings

[John Wiegley]

>>>>> Mike Meyer <mwm at mired.org> writes:

> The current version uses a conduit that just reads the data to a sinkLbs to
> get a lazy bytestring, which is then processed.

sinkLbs reads the entire contents into memory, so this is the exact opposite
of what you want.

> Someone want to tell me if I correctly understand things and the itch is
> just leftover imperative thinking, or the itch is right and I need to fix
> the code?

You should write a Conduit or a Sink which will do the processing you need.

By default, you'll receive "chunks" at each call to "await".  If you need
lines, there is a linesUnbounded Conduit (as of conduit 1.1), but it still
reads whole chunks into memory at a time (I believe the default chunk size is
32k)?  But that's the same behavior as plain lazy I/O.

Once your Conduit or Sink (i.e., Consumer) finds the data it needs, it should
simply end, and not call await anymore.  This will inform upstream that
processing and done and that all finalizers should be executed.

John