[Haskell-beginners] Space leak while reading from a file?

[Bob Ippolito]

I haven't looked closely but I suspect if you use foldM to build your Map
rather than untilM it might consume less memory since you won't be
allocating this big list.

BUT… the real reason why the memory usage is so much higher than you expect
is because slicing a Data.Text is an O(1) operation that shares the
underlying buffer between the original and the slice. Calling T.words will
ensure that the full original line stays around. You can see this in the
implementation:
http://hackage.haskell.org/package/text-0.11.2.0/docs/src/Data-Text.html#words

This behavior is documented somewhat near there in the docs, but I think it
should really be a top-level thing:
http://hackage.haskell.org/package/text-0.11.2.0/docs/Data-Text.html#g:18

I don't know what function to use to force the array to be copied,
hopefully there is one! Erlang's binaries work similarly and there is a
copy function for them for exactly this purpose:
http://www.erlang.org/doc/man/binary.html

-bob



On Sun, May 11, 2014 at 4:24 AM, Jan Snajder <jan.snajder at fer.hr> wrote:

> Dear all,
>
> I'm trying to implement a simple file-based database. I apparently have
> a space leak, but I have no clue where it comes from.
>
> Here's the file-based database implementation:
> http://pastebin.com/QqiqcXFw
>
> The idea to have a database table in a single textual file. One line
> equals one table row. The fields within a row are whitespace separated.
> The first field is the key. Because I'd like to work with large files, I
> don't want to load the whole file into memory. Instead, I'd like to be
> able to fetch the rows on demand, by keys. Thus I first create an index
> that links keys to file seeks. I use the readerT to add the index to the
> IO monad.
>
> For testing, I use a dummy table produced as follows:
>
> import System.IO
> import Text.Printf
> import Control.Monad
>
> row = unwords [printf "field%03d" (i::Int) | i <- [1..999]]
>
> main = do
>   forM_ [1..250000] $ \i ->
>      putStrLn $ printf "row%06d %s" (i::Int) row
>
> This generates a 2.1G textual file, which I store on my disk.
>
> The testing code:
>
> import FileDB
> import qualified Data.Text as T
> import Text.Printf
> import Control.Applicative
> import Control.Monad
> import Control.Monad.Trans
> import System.IO
> import System.Environment
>
> main = do
>   (f:_) <- getArgs
>   t <- openTable f
>   runDB t $ do
>     ks <- getKeys
>     liftIO $ do
>       putStrLn . printf "%d keys read" $ length ks
>       putStrLn "Press any key to continue..."
>       getChar
>     forM_ ks $ \k -> do
>       Just r <- getRow k
>       liftIO . putStrLn $ printf "Row \"%s\" has %d fields"
>         (T.unpack k) (length r)
>
> When I run the test on the 2.1GB file, the whole program consumes 10GB.
>
> 6GB seem to be allocated after the index is built (just before entering
> the forM_ function). The remaining 4GB are allocated while fetching all
> the rows.
>
> I find both things difficult to explain.
>
> 6GB seems too much for the index. Each key is 9 characters (stored as
> Data.Text), and I have 250K such keys in a Data.Map. Should this really
> add up to 6GB?
>
> Also, I have no idea why fetching all the rows, one by one, should
> consume any additional memory. Each row is fetched and its length is
> computed and printed out. I see no reason for the rows to be retained in
> the memory.
>
> Here's the memory allocation summary:
>
> > 1,093,931,338,632 bytes allocated in the heap
> >    2,225,144,704 bytes copied during GC
> >    4,533,898,000 bytes maximum residency (26 sample(s))
> >    3,080,926,336 bytes maximum slop
> >            10004 MB total memory in use (0 MB lost due to fragmentation)
> >
> >                                     Tot time (elapsed)  Avg pause  Max
> pause
> >   Gen  0     2171739 colls,     0 par   45.29s   45.26s     0.0000s
>  0.0030s
> >   Gen  1        26 colls,     0 par    1.50s    1.53s     0.0589s
> 0.7087s
> >
> >   INIT    time    0.00s  (  0.00s elapsed)
> >   MUT     time  279.92s  (284.85s elapsed)
> >   GC      time   46.80s  ( 46.79s elapsed)
> >   EXIT    time    0.68s  (  0.71s elapsed)
> >   Total   time  327.40s  (332.35s elapsed)
> >
> >   %GC     time      14.3%  (14.1% elapsed)
> >
> >   Alloc rate    3,908,073,170 bytes per MUT second
> >
> >   Productivity  85.7% of total user, 84.4% of total elapsed
>
>
> Btw., I don't get the "bytes allocated in the heap" figure, which is
> approx. 1000 GB (?).
>
> I'm obviously doing something wrong here. I'd be thankful for any help.
>
> Best,
> Jan
> _______________________________________________
> Beginners mailing list
> Beginners at haskell.org
> http://www.haskell.org/mailman/listinfo/beginners
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/beginners/attachments/20140512/a535b500/attachment-0001.html>