[Haskell-beginners] Writing huge result of `show` to file results in out of memory

[Norbert Melzer]

I do not think that `show` or `read` are good candidates for serialising
data into a file. I'd suggest to use a dedicated serialisation library, to
write out proper JSON/YAML/XML/binary.

Mahdi Dibaiee <mdibaiee at aol.com> schrieb am Do., 27. Okt. 2016 um 10:53 Uhr:

Hi,

So I have a data instance which contains a few big matrices, and I want to
save my instance to a file so I can `read` it back later
to avoid the long computation every time (I'm training a recurrent neural
network).

First attempt, the simplest method:

  writeFile "rnn" (show dataInstance)

It starts to take all of the memory and then bails out with `out-of-memory`
error.

So I wrote a function to write the string chunk by chunk, without
buffering, here is the code:
https://github.com/mdibaiee/sibe/blob/728df02fbdd6f134af107c098f5477094c61ea76/examples/recurrent.hs#L52-L64

Copy/pasted from the link:

  saveRecurrent :: FilePath -> String -> Int -> IO ()
  saveRecurrent path str chunkSize = do
    handle <- openFile path AppendMode
    hSetBuffering handle NoBuffering
    loop handle str
    hClose handle
    where
      loop _ [] = return ()
      loop handle s = do
        hPutStr handle $ take chunkSize s
        hFlush handle
        loop handle $ drop chunkSize s

But it doesn't work either, I still get `out-of-memory` errors. From
what I understand, this should work, but it isn't.

I asked on IRC and someone said "Show is not lazy *enough*", if that's
the case, I would appreciate an explanation of that.


Thanks,

Mahdi


_______________________________________________
Beginners mailing list
Beginners at haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/beginners
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/beginners/attachments/20161028/47857015/attachment.html>