[Haskell-cafe] two problems with Data.Binary and Data.ByteString
kr.angelov at gmail.com
Wed Nov 5 10:17:37 EST 2008
I had the same problem (stack overflow). The solution was to change
the >>= operator in the Get monad. Currently it is:
m >>= k = Get (\s -> let (a, s') = unGet m s
in unGet (k a) s')
but I changed it to:
m >>= k = Get (\s -> case unGet m s of
(a, s') -> unGet (k a) s')
It seems that the bind operator is lazy and this caused the stack overflow.
I have also another problem. Every Int and Word is stored as 64-bit
value and this expands the output file a lot. I have a lot of integers
and most of them are < 128 but not all of them. I changed the
serialization so that the Int and Word are serialized in a variable
number of bytes. Without this change the binary serialization was even
worse than the textual serialization that we had before. The file was
almost full with zeros.
I just haven't time to prepare a patch and to send it for review but
if other people have the same problem I will do it.
On Wed, Aug 13, 2008 at 1:13 AM, Tim Newsham <newsham at lava.net> wrote:
> I have a program that read in and populated a large data structure and
> then saved it out with Data.Binary and Data.ByteString.Lazy.Char8:
> saveState db = B.writeFile stateFile =<<
> encode <$> atomically (readTVar db)
> when I go to read this in later I get a stack overflow:
> loadState db = do
> d <- decode <$> B.readFile stateFile
> atomically $ writeTVar db d
> Stack space overflow: current size 8388608 bytes.
> Use `+RTS -Ksize' to increase it.
> or from ghci:
> d <- liftM decode
> "savedState.bin") :: IO InstrsDb
> fromList *** Exception: stack overflow
> The data type I'm storing is a Map (of maps):
> type DailyDb = M.Map Date Daily
> type InstrsDb = M.Map String DailyDb
> What's going on here? Why is the system capable of building and saving
> the data but not in reading and umarhsalling it? What is the proper way
> to track down where the exception is happening? Any debugging tips?
> I also noticed another issue while testing. If my program loads
> the data at startup by calling loadState then all later calls to
> saveState give an error:
> Log: savedState.bin: openFile: resource busy (file is locked)
> this does not occur if the program wasnt loaded. My best guess here
> is that B.readFile isnt completing and closing the file for some
> reason. Is there a good way to force this?
> Tim Newsham
> Haskell-Cafe mailing list
> Haskell-Cafe at haskell.org
More information about the Haskell-Cafe