heap profiling

[Claus Reinke]

> On Sat, Jun 19, 2010 at 8:46 AM, Claus Reinke <claus.reinke at talk21.com> 
> wrote:
>>> I put the simple version at
>>> http://hpaste.org/fastcgi/hpaste.fcgi/view?id=26329#a26329
>>>
>>> This one displays much better performance with DList + Writer.Strict
>>> than List + StrictWriter so I guess it's not too surprising.  However,
>>> *something* is still generating a lot of lag
>>
>> Some lag is to be expected: logging is about creating
>> and recording data for later use. You'd need  to annotate
>
> Indeed, though I don't think this is the case, because I get lots of
> lag even when no logs are written.

In the part you deleted I mentioned one source of lag that does
not disappear when no logs are written, and a way of using
profiling cost centers to track down other sources (the ones
I mentioned accounted for 2/3 of lag in the "profile_logging"
profile, according to "-hblag -hc -L40" on SCC-annotated
source).

> My conclusion is that my contrived tests are not useful because the
> whole point is to get a controlled situation by simplifying it, but
> even the simple version displays mysterious behaviour.

It is sometimes difficult to produce a simplified version that
has strictly fewer problems than the full version, especially
if the simple version is not derived from the full version but
from simplifying assumptions. Nevertheless, the simpler
version has less code, and it is therefore helpful to try and
understand its behaviour before returning to full complexity.

In particular, it is important to figure out which problematic
aspects of the simple version also apply to the full version
and which problems can be ignored because they cannot
occur in the full version.

> I should go back to poking at the production version.  Even
> if I don't understand the changes I'm making, at least when
> I get a speedup I'm more sure it translates to a real speed
> up in the application.

That way lies madness. Not recommended unless you are
producing code that noone is ever going to look at again,
and even then you'll be writing similar code in the future,
so it is better to put in the energy once to puzzle out
those mysterious details!-)

Claus