[Haskell-cafe] possible memory leak in uvector 0.1.0.3

[Manlio Perillo]

Claus Reinke ha scritto:
>>>> I split the string in lines, then map some functions on each line to 
>>>> parse the data, and finally calling toU, for converting to an UArr.
>>>
>>> Just to make sure (code fragments or, better, reduced examples
>>> would make it easier to see what the discussion is about): are you 
>>> forcing the UArr to be constructed before putting it into the Map?
>>
>> parse handle =
>>   contents <- S.hGetContents handle
>>   let v =  map singleton' $ ratings contents
>>   let m = foldl1' (unionWith appendU) v
>>   v `seq` return $! m
>>
>>   where
>>     -- Build a Map with a single movie rating
>>     singleton' :: (Word32, Word8) -> MovieRatings
>>     singleton' (id, rate) =
>>       singleton (fromIntegral $ id) (singletonU $ pairS (id, rate))
> 
> That helps to make things clearer, I think. One issue is
> the nature of Maps (strict in keys, non-strict in values).
> 
> - neither singleton nor unionWith are strict in the Map values, so
>    nothing here forces the evaluation of rate or construction    of UArr
> 

But, as I have written, in one of my tests I also tried rnf to force 
evaluation:
rnf v `seq` rnf m `seq` return m

where:

instance NFData a => NFData (Data.IntMap.IntMap a) where
     rnf = rnf . Data.IntMap.toList


Isn't this sufficient?

> [...]
> - seq on a list only forces the first node of the list ((:),[],_|_),
>    so (v `seq`) isn't likely to help much. Also, you probably
>    do not want to force the whole list of singletons before
>    builing the Map, you want the singletons to be constructed    and 
> consumed incrementally.
> 

Right, thanks.

> 
>> type Rating = Word32 :*: Word8
>> type MovieRatings = IntMap (UArr Rating) -- UArr from uvector
> 
> A standard trick to keep Map values evaluated by construction
> is to make the availability of keys dependent on their values, eg
> (singleton key) $! value. That won't help with unionWith and
> the appendUs, but it should allow the source string references
> to be dropped early, as the singletons are constructed.
> 

Tried; but, even using union instead of unionWith, the memory grows fast 
as before.


Just to check if the culprit is IntMap, isn't possible to write a test 
program that build a big IntMap (UArr Int) ?

Right now I don't have the time.



Manlio