[Haskell-cafe] possible memory leak in uvector 0.1.0.3
manlio_perillo at libero.it
Tue Mar 3 12:20:47 EST 2009
Claus Reinke ha scritto:
>>>> I split the string in lines, then map some functions on each line to
>>>> parse the data, and finally calling toU, for converting to an UArr.
>>> Just to make sure (code fragments or, better, reduced examples
>>> would make it easier to see what the discussion is about): are you
>>> forcing the UArr to be constructed before putting it into the Map?
>> parse handle =
>> contents <- S.hGetContents handle
>> let v = map singleton' $ ratings contents
>> let m = foldl1' (unionWith appendU) v
>> v `seq` return $! m
>> -- Build a Map with a single movie rating
>> singleton' :: (Word32, Word8) -> MovieRatings
>> singleton' (id, rate) =
>> singleton (fromIntegral $ id) (singletonU $ pairS (id, rate))
> That helps to make things clearer, I think. One issue is
> the nature of Maps (strict in keys, non-strict in values).
> - neither singleton nor unionWith are strict in the Map values, so
> nothing here forces the evaluation of rate or construction of UArr
But, as I have written, in one of my tests I also tried rnf to force
rnf v `seq` rnf m `seq` return m
instance NFData a => NFData (Data.IntMap.IntMap a) where
rnf = rnf . Data.IntMap.toList
Isn't this sufficient?
> - seq on a list only forces the first node of the list ((:),,_|_),
> so (v `seq`) isn't likely to help much. Also, you probably
> do not want to force the whole list of singletons before
> builing the Map, you want the singletons to be constructed and
> consumed incrementally.
>> type Rating = Word32 :*: Word8
>> type MovieRatings = IntMap (UArr Rating) -- UArr from uvector
> A standard trick to keep Map values evaluated by construction
> is to make the availability of keys dependent on their values, eg
> (singleton key) $! value. That won't help with unionWith and
> the appendUs, but it should allow the source string references
> to be dropped early, as the singletons are constructed.
Tried; but, even using union instead of unionWith, the memory grows fast
Just to check if the culprit is IntMap, isn't possible to write a test
program that build a big IntMap (UArr Int) ?
Right now I don't have the time.
More information about the Haskell-Cafe