[Haskell-cafe] Design suggestion for Data.Binary.Defer

[David Roundy]

On Tue, Jun 17, 2008 at 07:55:51AM +0100, Neil Mitchell wrote:
> Hi,

Hello!

> David:
> > Is there any reason not to just write all lazy fields of variable size
> >  in a deferred manner?
> 
> I completely hadn't though of this! You will loose a bit of time, for
> example reading fields which were previously not deferred will require
> a file seek, but the effort/thought/efficiency trade-off is good. I
> think I will go with this.

Actually, you ought to be able to pretty easily remove this tradeoff
by introducing a strict read function as a new method in your class.
So anyone who wants to strictly read lazy data can use that function
instead of the lazy one.  The writing of data is unaffected... or
rather, it *is* affected, but only in the sense that writing deferred
data cannot be as lazy as the writing of undeferred data.  e.g. when
writing a deferred list, you can't output the first word until you've
already read through the entire list in order to find out how long it
is.  That could actually be quite a problem for code that relies on
laziness to handle massive amounts of data.  :( Lazy reading seems to
require strict writing, while lazy writing requires strict reading!

So perhaps you want a pair of read functions, one strict, one lazy,
and a pair of write functions.  That doesn't provide any sort of
fine-grained control, but at least allows streaming in either
direction.  You could use the same data format for either strictly or
lazily pickled data as long as you have a reserved value for the
length-to-be-skipped, so the lazy reader could tell if it reaches data
that whose length it doesn't know.

David