[Haskell-beginners] Space leak while reading from a file?

Mon May 12 18:34:49 UTC 2014

This Chrome plugin helps avoid the Hackage version problem btw:
https://chrome.google.com/webstore/detail/hackage-fu/dnpldbohleinhdgfnhlkofpgkdcfcfmf?hl=en-US

On Mon, May 12, 2014 at 1:32 PM, Bob Ippolito <bob at redivi.com> wrote:

> I don't know why I ended up looking at an old version of Data.Text but the
> copy function has been around for a year, since 0.11.3.0.
> http://hackage.haskell.org/package/text-1.1.1.2/docs/Data-Text.html#copy– you'll want to use it for this use case.
>
>
> On Mon, May 12, 2014 at 9:54 AM, Bob Ippolito <bob at redivi.com> wrote:
>
>> I haven't looked closely but I suspect if you use foldM to build your Map
>> rather than untilM it might consume less memory since you won't be
>> allocating this big list.
>>
>> BUT… the real reason why the memory usage is so much higher than you
>> expect is because slicing a Data.Text is an O(1) operation that shares the
>> underlying buffer between the original and the slice. Calling T.words will
>> ensure that the full original line stays around. You can see this in the
>> implementation:
>>
>> http://hackage.haskell.org/package/text-0.11.2.0/docs/src/Data-Text.html#words
>>
>> This behavior is documented somewhat near there in the docs, but I think
>> it should really be a top-level thing:
>> http://hackage.haskell.org/package/text-0.11.2.0/docs/Data-Text.html#g:18
>>
>> I don't know what function to use to force the array to be copied,
>> hopefully there is one! Erlang's binaries work similarly and there is a
>> copy function for them for exactly this purpose:
>> http://www.erlang.org/doc/man/binary.html
>>
>> -bob
>>
>>
>>
>> On Sun, May 11, 2014 at 4:24 AM, Jan Snajder <jan.snajder at fer.hr> wrote:
>>
>>> Dear all,
>>>
>>> I'm trying to implement a simple file-based database. I apparently have
>>> a space leak, but I have no clue where it comes from.
>>>
>>> Here's the file-based database implementation:
>>> http://pastebin.com/QqiqcXFw
>>>
>>> The idea to have a database table in a single textual file. One line
>>> equals one table row. The fields within a row are whitespace separated.
>>> The first field is the key. Because I'd like to work with large files, I
>>> don't want to load the whole file into memory. Instead, I'd like to be
>>> able to fetch the rows on demand, by keys. Thus I first create an index
>>> that links keys to file seeks. I use the readerT to add the index to the
>>> IO monad.
>>>
>>> For testing, I use a dummy table produced as follows:
>>>
>>> import System.IO
>>> import Text.Printf
>>> import Control.Monad
>>>
>>> row = unwords [printf "field%03d" (i::Int) | i <- [1..999]]
>>>
>>> main = do
>>>   forM_ [1..250000] $ \i ->
>>>      putStrLn $ printf "row%06d %s" (i::Int) row
>>>
>>> This generates a 2.1G textual file, which I store on my disk.
>>>
>>> The testing code:
>>>
>>> import FileDB
>>> import qualified Data.Text as T
>>> import Text.Printf
>>> import Control.Applicative
>>> import Control.Monad
>>> import Control.Monad.Trans
>>> import System.IO
>>> import System.Environment
>>>
>>> main = do
>>>   (f:_) <- getArgs
>>>   t <- openTable f
>>>   runDB t $ do
>>>     ks <- getKeys
>>>     liftIO $ do
>>>       putStrLn . printf "%d keys read" $ length ks
>>>       putStrLn "Press any key to continue..."
>>>       getChar
>>>     forM_ ks $ \k -> do
>>>       Just r <- getRow k
>>>       liftIO . putStrLn $ printf "Row \"%s\" has %d fields"
>>>         (T.unpack k) (length r)
>>>
>>> When I run the test on the 2.1GB file, the whole program consumes 10GB.
>>>
>>> 6GB seem to be allocated after the index is built (just before entering
>>> the forM_ function). The remaining 4GB are allocated while fetching all
>>> the rows.
>>>
>>> I find both things difficult to explain.
>>>
>>> 6GB seems too much for the index. Each key is 9 characters (stored as
>>> Data.Text), and I have 250K such keys in a Data.Map. Should this really
>>> add up to 6GB?
>>>
>>> Also, I have no idea why fetching all the rows, one by one, should
>>> consume any additional memory. Each row is fetched and its length is
>>> computed and printed out. I see no reason for the rows to be retained in
>>> the memory.
>>>
>>> Here's the memory allocation summary:
>>>
>>> > 1,093,931,338,632 bytes allocated in the heap
>>> >    2,225,144,704 bytes copied during GC
>>> >    4,533,898,000 bytes maximum residency (26 sample(s))
>>> >    3,080,926,336 bytes maximum slop
>>> >            10004 MB total memory in use (0 MB lost due to
>>> fragmentation)
>>> >
>>> >                                     Tot time (elapsed)  Avg pause  Max
>>> pause
>>> >   Gen  0     2171739 colls,     0 par   45.29s   45.26s     0.0000s
>>>  0.0030s
>>> >   Gen  1        26 colls,     0 par    1.50s    1.53s     0.0589s
>>> 0.7087s
>>> >
>>> >   INIT    time    0.00s  (  0.00s elapsed)
>>> >   MUT     time  279.92s  (284.85s elapsed)
>>> >   GC      time   46.80s  ( 46.79s elapsed)
>>> >   EXIT    time    0.68s  (  0.71s elapsed)
>>> >   Total   time  327.40s  (332.35s elapsed)
>>> >
>>> >   %GC     time      14.3%  (14.1% elapsed)
>>> >
>>> >   Alloc rate    3,908,073,170 bytes per MUT second
>>> >
>>> >   Productivity  85.7% of total user, 84.4% of total elapsed
>>>
>>>
>>> Btw., I don't get the "bytes allocated in the heap" figure, which is
>>> approx. 1000 GB (?).
>>>
>>> I'm obviously doing something wrong here. I'd be thankful for any help.
>>>
>>> Best,
>>> Jan
>>> _______________________________________________
>>> Beginners mailing list
>>> Beginners at haskell.org
>>> http://www.haskell.org/mailman/listinfo/beginners
>>>
>>
>>
>
> _______________________________________________
> Beginners mailing list
> Beginners at haskell.org
> http://www.haskell.org/mailman/listinfo/beginners
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/beginners/attachments/20140512/38f0e751/attachment-0001.html>