[Haskell-cafe] reading file content with conduit

Alexander V Vershilov alexander.vershilov at gmail.com
Fri Nov 2 12:15:36 UTC 2018


Hello Fabien,

in your example, the problem is that sourceFile gives you a bytestring
chunk that contains
more than one record and decoding function does not return a list of
values, instead
return only one. So you need to pass the leftover data to the next
decoding round.

You may consider using
https://hackage.haskell.org/package/binary-conduit package that
does that (see https://hackage.haskell.org/package/binary-conduit-1.3.1/docs/src/Data.Conduit.Serialization.Binary.html#conduitGet)

--
Best regards, Alexander.


On Fri, 2 Nov 2018 at 15:09, Fabien R <theedge456 at free.fr> wrote:
>
> Hello,
> I have a strange behaviour with my code when reading data from a file saved by sinkFile.
> I only see the first record of the file, although the file seems to contain several records.
>
> Any hints ?
>
> -- built in a sandbox with GHC 8.2.2, base 4.10.1.0, binary 0.8.5.1, bytestring 0.8.10.2, conduit 1.3.1
> {-# LANGUAGE DeriveGeneric #-}
> import qualified Data.ByteString as DB
> import qualified Data.ByteString.Lazy as DBL
> import qualified Data.Binary as DBI
> import GHC.Generics (Generic)
> import Conduit
> import Data.Int (Int64)
>
> data MySubRec = MySubRec { sr1 :: Float,
>                            sr2 :: Float }
>              deriving (Generic, Show)
>
> data MyRec = MyRec { r1 :: Int64,
>                      r2 :: String,
>                      r3 :: [MySubRec],
>                      r4 :: [MySubRec]
>                      }
>              deriving (Generic, Show)
>
> instance DBI.Binary MySubRec
> instance DBI.Binary MyRec
>
> es = MySubRec { sr1 =1.0, sr2 =1000.5 }
> myList = repeat es
> e1 = MyRec{ r1=1, r2="e1", r3=take 2 myList, r4=take 1 myList}
> e2 = MyRec{ r1=2, r2="e2", r3=take 2 myList, r4=take 1 myList}
> myData = concat $ repeat [e1,e2]
>
> dataToBs :: Monad m =>
>             ConduitT MyRec DB.ByteString m ()
> dataToBs = do
>                             d <- await
>                             case d of
>                              Just bs -> do
>                                 yield $ DBL.toStrict $ DBI.encode bs
>                                 dataToBs
>                              _ -> return ()
> bsToData :: Monad m =>
>             ConduitT DB.ByteString MyRec m ()
> bsToData = do
>                             d <- await
>                             case d of
>                              Just bs -> do
>                                 yield $ DBI.decode $ DBL.fromStrict bs
>                                 bsToData
>                              _ -> return ()
> main = do
>         runConduitRes $ yieldMany (take 10 myData) .| dataToBs .|  sinkFile "/tmp/res.bin"
>         runConduitRes $ sourceFile "/tmp/res.bin" .| bsToData .| mapM_C (liftIO . putStrLn . show)
> _______________________________________________
> Haskell-Cafe mailing list
> To (un)subscribe, modify options or view archives go to:
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
> Only members subscribed via the mailman list are allowed to post.



-- 
Alexander


More information about the Haskell-Cafe mailing list