<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="background-color: rgb(255, 255, 255); color: rgb(0, 0,
0);" text="#000000" bgcolor="#FFFFFF">
Good advice for eventlog. I read about it long ago and completely
forgot we have it. :-)<br>
<p><br>
</p>
<br>
<blockquote type="cite"
cite="mid:f7152f39-c6bb-5d60-4635-2e3fc9e1ff8b@iohk.io"
style="border-left: 2px solid #330033 !important; border-right:
2px solid #330033 !important; padding: 0px 15px 0px 15px; margin:
8px 2px;"><!--[if !IE]><DIV style="border-left: 2px solid #330033; border-right: 2px solid #330033; padding: 0px 15px; margin: 2px 0px;"><![endif]--><span
class="headerSpan" style="color:#000000;">
<div class="moz-cite-prefix">-------- Original Message --------<br>
Subject: Re: [Haskell-cafe] Measuring memory usage<br>
From: Vanessa McHale <a class="moz-txt-link-rfc2396E" href="mailto:vanessa.mchale@iohk.io"><vanessa.mchale@iohk.io></a><br>
To: <a class="moz-txt-link-abbreviated" href="mailto:haskell-cafe@haskell.org">haskell-cafe@haskell.org</a><br>
Date: 29/06/18 15:34<br>
</div>
<br>
<br>
</span>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<p>You might consider profiling your application or making an
event log instead. The event log should show GC activity and the
heap profile should show memory usage more accurately. <br>
</p>
<br>
<div class="moz-cite-prefix">On 06/29/2018 07:31 AM, Vlatko Basic
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:5e82210c-6aa4-a7d0-fce6-2668df343214@gmail.com"
style="border-left: 2px solid #330033 !important; border-right:
2px solid #330033 !important; padding: 0px 15px 0px 15px;
margin: 8px 2px;"><!--[if !IE]><DIV style="border-left: 2px solid #330033; border-right: 2px solid #330033; padding: 0px 15px; margin: 2px 0px;"><![endif]-->
<meta http-equiv="content-type" content="text/html;
charset=utf-8">
<style id="EHTipGlobalStyle">.EHTipToolTip * {background: inherit;font-family: inherit;font-size: inherit;font-size-adjust: none;font-stretch: normal;line-height: inherit;font-variant: normal;border: 0px;text-transform: inherit;color: inherit;font-style: inherit;text-decoration: inherit;margin: 0px 0px 0px 0px;padding: 0px 0px 0px 0px;float: none;display: inline;cursor: default;}
.EHTipReplacer, .EHTipKey, .EHTipAudio {cursor: pointer;}
.EHTipToolTip hr {margin: 0.4em 0;display: block;border: 1px inset;}
.EHTipTranslation {font-style: normal;}
.EHTipTranslation a {color: #000099;font-style: normal;text-decoration: none;}
.EHTipTranslation a:hover {background: inherit;color: #000000;text-decoration: underline;}
</style>
<p>Hello,</p>
<p>I've come to some strange results using Weigh package. <br>
</p>
<p>It shows that HashMap inside 'data' is using much, much more
memory. <br>
</p>
<p>The strange thing is that I'm seeing too large mem usage in
my app as well (several "MapData" like in records), and trying
to figure out with 'weigh' what's keeping the mem. <br>
</p>
<p>Noticed that when I change the code to use HashMap directly
(not inside 'data', that's the only change), the mem usage
observed with top drops down for ~60M, from 850M to 790M.<br>
</p>
<br>
<p>These are the test results for 10K, 5K and 3.3K items for
"data MapData k v = MapData (HashMap k v)" (at the end is the
full runnable example.)<br>
</p>
<p><font size="-1"><tt>Case Allocated GCs</tt><tt><br>
</tt><tt>HashMap 262,824 0</tt><tt><br>
</tt><tt>HashMap half 58,536 0</tt><tt><br>
</tt><tt>HashMap third 17,064 0</tt><tt><br>
</tt><tt>MapData 4,242,208 4</tt></font></p>
<p>I tested by changing the order, disabling all but one etc.,
and the results were the same. Same 'weigh' behaviour with
IntMap and Map.<br>
</p>
<p><br>
</p>
<p>So, if anyone knows and has some experience with such issues,
my questions are:</p>
<p>1. Is 'weigh' package reliable/usable, at least to some
extent? (the results do show diff between full, half and
third)</p>
<p>2. How do you measure mem consumptions of your large
data/records?</p>
<p>3. If the results are even approximately valid, what could
cause such large discrepancies with 'data'?</p>
<p>4. Is there a way to see if some record has been freed from
memory, GCed?<br>
</p>
<br>
<p><br>
</p>
<p><font size="-1"><tt>module Main where</tt><tt><br>
</tt><tt><br>
</tt><tt>import Prelude</tt><tt><br>
</tt><tt><br>
</tt><tt>import Control.DeepSeq (NFData)</tt><tt><br>
</tt><tt>import Data.HashMap.Strict (HashMap, fromList)</tt><tt><br>
</tt><tt>import GHC.Generics (Generic)</tt><tt><br>
</tt><tt>import Weigh (mainWith, value)</tt><tt><br>
</tt><tt><br>
</tt><tt><br>
</tt><tt>data MapData k v = MapData (HashMap k v) deriving
Generic</tt><tt><br>
</tt><tt>instance (NFData k, NFData v) => NFData (MapData
k v)</tt><tt><br>
</tt><tt><br>
</tt><tt>full, half, third :: Int</tt><tt><br>
</tt><tt>full = 10000</tt><tt><br>
</tt><tt>half = 5000</tt><tt><br>
</tt><tt>third = 3333</tt><tt><br>
</tt><tt><br>
</tt><tt>main :: IO ()</tt><tt><br>
</tt><tt>main = mainWith $ do</tt><tt><br>
</tt><tt> value "HashMap" ( mkHMList full)</tt><tt><br>
</tt><tt> value "HashMap half" ( mkHMList half)</tt><tt><br>
</tt><tt> value "HashMap third" ( mkHMList third)</tt><tt><br>
</tt><tt> value "MapData" (MapData $ mkHMList full)</tt><tt><br>
</tt><tt><br>
</tt><tt>mkHMList :: Int -> HashMap Int String</tt><tt><br>
</tt><tt>mkHMList n = fromList . zip [1..n] $ replicate n
"some text"</tt><tt><br>
</tt><tt><br>
</tt><tt><br>
</tt></font><br>
</p>
<!--'"--><br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
<a class="moz-txt-link-freetext" href="http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe" moz-do-not-send="true">http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe</a>
Only members subscribed via the mailman list are allowed to post.</pre>
<!--[if !IE]></DIV><![endif]--></blockquote>
<!--'"--><br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Haskell-Cafe mailing list
To (un)subscribe, modify options or view archives go to:
<a class="moz-txt-link-freetext" href="http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe">http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe</a>
Only members subscribed via the mailman list are allowed to post.</pre>
<!--[if !IE]></DIV><![endif]--></blockquote>
<br>
</body>
</html>