[Haskell-cafe] iteratee-compress space leak?

Maciej Piechotka uzytkownik2 at gmail.com
Sun Feb 20 16:10:26 CET 2011


On Fri, 2011-02-18 at 17:27 +0300, Michael A Baikov wrote:
> I am trying to play with iteratee making parser for squid log files, but found that my code do not run in constant space when it tries to process compressed log files. So i simplified my code down to this snippet:
> 
> import Data.ByteString (ByteString)
> import Data.Iteratee as I
> import Data.Iteratee.Char
> import Data.Iteratee.ZLib
> import System
> 
> main = do
>         args <- getArgs
>         let fname     = args !! 0
>         let blockSize = read $ args !! 1
> 
>         fileDriver (leak blockSize) fname >>= print
> 
> leak :: Int -> Iteratee ByteString IO ()
> leak blockSize = joinIM $ enumInflate GZip defaultDecompressParams chunkedRead
>     where
>         consChunk :: Iteratee ByteString IO String
>         consChunk = (joinI $ I.take blockSize I.length) >>= return . show
> 
>         chunkedRead :: Iteratee ByteString IO ()
>         chunkedRead = joinI $ convStream consChunk printLines
> 
> 
> First argument - file name (/var/log/messages.1.gz will do)
> second - size of block to consume input. with low size (10 bytes) of consumed blocks it leaks very fast, with larger blocks (~10000) it works almost without leaks.
> 
> So. Is it bugs within my code, or iteratee-compress should behave differently?

After looking into problem (or rather onto your code) - the problem have
nothing to do with iteratee-compress I believe. I get similar behaviour
and results when I replace "joinIM $ enumInflate GZip
defaultDecompressParams chunkedRead" by chunkedRead. (The memory is
smaller but it is due to decompression not iteratee fault).

Regards
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://www.haskell.org/pipermail/haskell-cafe/attachments/20110220/611cad83/attachment.pgp>


More information about the Haskell-Cafe mailing list