[Haskell-cafe] help understanding zlib space leak

[diego souza]

Dear haskellers,

I'd like assistance figuring out a strange space leak using zlib
package.

To make it easier to reproduce I've come up with the following snippet
that pretty much resumes up the problem I'm trying to solve:

> -- omitting imports and function signatures
> encode = compress . L.fromChunks
>  
> main = do { hSetBinaryMode stdout True
>           ; hSetBinaryMode stdin True
>           ; hSetBuffering stdin NoBuffering
>           ; hSetBuffering stdout NoBuffering
>           ; loop []
>           }
>  where loop buff
>          | length buff == 64 = L.hPut stdout (encode buff) >> loop []
>          | otherwise         = do { eof <- hIsEOF stdin
>                                   ; when (not eof) (fmap (: buff) (B.hGetSome stdin 512) >>= loop)
>                                   }

N.B.: Removing the `compress' function from the above code also removes the
      space leak.

Now, feeding the above program with ~8GB worth of binary data:

> $ uname -a
Linux mephisto.localhost.localdomain 3.8.7-1-ARCH #1 SMP PREEMPT Sat Apr 13 09:01:47 CEST 2013 x86_64 GNU/Linux
> $ ghc --version
The Glorious Glasgow Haskell Compilation System, version 7.6.3
> $ ghc -W -Wall -rtsopts --make -O2 test.hs
> $ sudo dd if=/dev/sda bs=4K count=2048K | ./test +RTS -M1M -s >/dev/null
...
8589934592 bytes (8.6 GB) copied, 277.263 s, 31.0 MB/s
...
               2 MB total memory in use (0 MB lost due to fragmentation)
...

Which is fine. However, monitoring the RSS size is a different story:

> $ while pidof test; do ps -o rss= -p $(pidof test); done | tail
16967
25620
16967
25628
16967
25628
16967
25628
16967  # ~16M
    0

I know the RSS usually overestimates the memory consumption but the
problem is that it is forever growing.

The following I found very intriguing:

  * `+RTS -hc` gives me no hint about whats wrong [at least I couldn't see one];
  * `+RTS -M1M` doesn't produce an error;
  * removing the `compress' functions makes the problem disappear;

I couldn't figure these out and I don't think this is matter of
strictness, though. Has anyone seen this before?

Thanks in advance,
~dsouza