[Haskell-cafe] Too much strictness in binary-0.5.0.2

[Duncan Coutts]

On Sat, 2009-09-19 at 00:36 +0400, Khudyakov Alexey wrote:
> Hello
> 
> I run into problems with new binary package. Following function reads a list 
> of elements one by one until end of stream. List is very long (won't fit into 
> memory).

Sorry for the late reply.

> In binary-0.5.0.1 and earlier it read list lazily. Now it seems that it tries 
> to read whole list to memory. Program does not produce any output and memory 
> usage steadily grows.

Yes. We decided that having the Get monad as a lazy state monad just
doesn't make sense. It makes this one use case work, but causes more
problems generally. Certainly we need to be able to do lazy binary
deserialisation too, but our feeling is that it should be more explicit
and integrate better with error handling. Using a lazy state monad gives
us neither.

> > getStream :: Get a -> Get [a]
> > getStream getter = do
> >   empty <- isEmpty
> >   if empty
> >     then return []
> >     else do x <- getter
> >             xs <- getStream getter
> >             return (x:xs)
> 
> How could I add laziness to this function to revert to old behavior.

You can make it explicitly lazy using something like:

getStream :: Get a -> BS.ByteString -> [a]
getStream getter bs = unfoldr step (bs, 0)
  where
    step (bs, off) = case runGetState getOne bs off of
      (Nothing, _,   _   ) -> Nothing
      (Just x,  bs', off') -> Just (x, (bs', off'))

    getOne = do
      empty <- isEmpty
      if empty
        then return Nothing
        else fmap Just getter

Note the distinct lack of error handling, but if we had runGetState
return an Either then you could still make this code lazy, but you'd
have to decide what to do when you got a decoding error. You could at
this point translate it into a pure error, or enrich your output stream
type to account for the possibility of errors.

Note also that if we changed runGetState to do any error handling that
this would also break the lazy state monad properties that you were
previously relying upon.

Duncan