[Haskell-cafe] Measuring memory usage

[Claude Heiland-Allen]

On 2018-06-29 15:14, Vlatko Basic wrote:
> Indeed bang solves the issue. I didn't try it because the docs says
> value doesn't have to be forced for validateFunc (which is used for
> value), but obviously only to whnf.

I think the issue is something to do with the two default 
implementations for rnf in the NFData class.  Historically, `rnf a = seq 
a ()` was the default implementation (ie just WHNF), but more recently 
there is a Generic-based version that should automatically reduce to 
normal form.  I don't know why the Generic version is either 1. not used 
at all, or 2. not working properly, but I suspect lack of instance 
Generic (HashMap k v), or possibly instance Generic1/2 MapData (if they 
are things?), may have something to do with it.  I don't know why there 
is no instance, but maybe it would allow breaking internal data 
structure invariants?


Claude

> 
> Thanks. :-)
> Been wasting whole morning on this.
> 
> 
> -------- Original Message  --------
> Subject: Re: [Haskell-cafe] Measuring memory usage
> From: Claude Heiland-Allen <claude at mathr.co.uk>
> To: haskell-cafe at haskell.org
> Date: 29/06/18 15:37
> 
>> Hi Vlatko,
>> 
>> On 29/06/18 13:31, Vlatko Basic wrote:
>>> 
>>> Hello,
>>> 
>>> I've come to some strange results using Weigh package.
>>> 
>>> It shows that HashMap inside 'data' is using much, much more memory.
>>> 
>> This seems to be astrictness issue - you may be measuring the size of 
>> a thunk instead of the resulting evaluated data.
>> 
>> To confirm that this is the case, you can replace:
>> 
>> data MapData k v = MapData (HashMap k v) deriving Generic
>> 
>> with
>> 
>> data MapData k v = MapData !(HashMap k v) deriving Generic
>> 
>> Or replace:
>> 
>>    value "MapData"       (MapData $ mkHMList full)
>> 
>> with
>> 
>>    value "MapData"       (MapData $! mkHMList full)
>> 
>> Either of these changes gave me results like this:
>> 
>> Case           Allocated  GCs
>> HashMap          262,824    0
>> HashMap half      58,536    0
>> HashMap third     17,064    0
>> MapData          263,416    0
>> 
>> The real issue seems to be NFData not doing what you expect. I'm not 
>> sure what the generic NFData instance is supposed to do, as there is 
>> no instance Generic (HashMap k v), so maybe you need to write your own 
>> rnf if you don't like either of the above workarounds.
>> 
>> Claude
>>> 
>>> The strange thing is that I'm seeing too large mem usage in my app as 
>>> well (several "MapData" like in records), and trying to figure out 
>>> with 'weigh' what's keeping the mem.
>>> 
>>> Noticed that when I change the code to use HashMap directly (not 
>>> inside 'data', that's the only change), the mem usage observed with 
>>> top drops down for ~60M, from 850M to 790M.
>>> 
>>> 
>>> These are the test results for 10K, 5K and 3.3K items for "data 
>>> MapData k v = MapData (HashMap k v)" (at the end is the full runnable 
>>> example.)
>>> 
>>> Case           Allocated  GCs
>>> HashMap          262,824    0
>>> HashMap half      58,536    0
>>> HashMap third     17,064    0
>>> MapData        4,242,208    4
>>> 
>>> I tested by changing the order, disabling all but one etc., and the 
>>> results were the same. Same 'weigh' behaviour with IntMap and Map.
>>> 
>>> 
>>> So, if anyone knows and has some experience with such issues, my 
>>> questions are:
>>> 
>>> 1. Is 'weigh' package reliable/usable, at least to some extent? (the 
>>> results do show diff between full, half and third)
>>> 
>>> 2. How do you measure mem consumptions of your large data/records?
>>> 
>>> 3. If the results are even approximately valid, what could cause such 
>>> large discrepancies with 'data'?
>>> 
>>> 4. Is there a way to see if some record has been freed from memory, 
>>> GCed?
>>> 
>>> 
>>> 
>>> module Main where
>>> 
>>> import Prelude
>>> 
>>> import Control.DeepSeq     (NFData)
>>> import Data.HashMap.Strict (HashMap, fromList)
>>> import GHC.Generics        (Generic)
>>> import Weigh               (mainWith, value)
>>> 
>>> 
>>> data MapData k v = MapData (HashMap k v) deriving Generic
>>> instance (NFData k, NFData v) => NFData (MapData k v)
>>> 
>>> full, half, third :: Int
>>> full  = 10000
>>> half  =  5000
>>> third =  3333
>>> 
>>> main :: IO ()
>>> main = mainWith $ do
>>>   value "HashMap"       (          mkHMList full)
>>>   value "HashMap half"  (          mkHMList half)
>>>   value "HashMap third" (          mkHMList third)
>>>   value "MapData"       (MapData $ mkHMList full)
>>> 
>>> mkHMList :: Int -> HashMap Int String
>>> mkHMList n = fromList . zip [1..n] $ replicate n "some text"
>>> 
>>> 

-- 
https://mathr.co.uk