[Haskell-beginners] Writing huge result of `show` to file results in out of memory

[Mahdi Dibaiee]

Hi,

So I have a data instance which contains a few big matrices, and I want to save my instance to a file so I can `read` it back later
to avoid the long computation every time (I'm training a recurrent neural network).

First attempt, the simplest method:

  writeFile "rnn" (show dataInstance)

It starts to take all of the memory and then bails out with `out-of-memory` error.

So I wrote a function to write the string chunk by chunk, without buffering, here is the code:
https://github.com/mdibaiee/sibe/blob/728df02fbdd6f134af107c098f5477094c61ea76/examples/recurrent.hs#L52-L64

Copy/pasted from the link:

  saveRecurrent :: FilePath -> String -> Int -> IO ()
  saveRecurrent path str chunkSize = do
    handle <- openFile path AppendMode
    hSetBuffering handle NoBuffering
    loop handle str
    hClose handle
    where
      loop _ [] = return ()
      loop handle s = do
        hPutStr handle $ take chunkSize s
        hFlush handle
        loop handle $ drop chunkSize s

But it doesn't work either, I still get `out-of-memory` errors. From what I understand, this should work, but it isn't.
I asked on IRC and someone said "Show is not lazy enough", if that's the case, I would appreciate an explanation of that.

Thanks,
Mahdi


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/beginners/attachments/20161027/f2453c5b/attachment.html>