Test performance impact (was: The dreaded M-R)

[John Hughes]

Summary: 2 programs failed to compile due to type errors (anna, gg).
One program did 19% more allocation, a few other programs increased
allocation very slightly (<2%).

            pic         +0.28%   +19.27%      0.02

  

Thanks, that was interesting. A follow-up question: pic has a space bug. 
How long will it take you
to find and fix it? And how come speed improved slightly in many 
cases--that seems counter-
intuitive.

Let me make clear that what concerns me is not the impact of the M-R on 
space and time
performance on average. What concerns me is the difficulty of debugging 
performance
problems. Even if we Haskell programmers usually consider correctness 
first and
performance later, when we want to actually use our beautiful code, we 
need to find and
fix at least the worst performance problems.

At that point, I'm reading the code and trying to understand--with the 
help of profiling
tools--why it performs the way it does. I need the performance behaviour 
of the code
to be *obvious*. When I pull out a common sub-expression into a 
let-binding, I want
to *know* that it will be evaluated at most once. When I see a 
collection of variable
bindings, likewise, I want to *know* that they are computed once only. 
When I see
a seq on such a variable, I want to *know* that any pointers in its 
unevaluated form
are thereby dropped. When I introduce or remove overloading in a 
function definition,
I want to *know* that I have not thereby changed the sharing properties 
of a variable
binding somewhere else, potentially introducing a space or time leak.  I 
want to know
these things by looking at the code in front of me, not by running the 
compiler with the
right flags and reading its warning messages. Unexpected behaviour at 
this point is a trap,
that can lead to many wasted hours as one reads the same code again and 
again, assuming it
behaves as expected, only to eventually convince oneself, by long 
examination of profiles,
that it cannot be doing so. If the unexpected behaviour is rare, that 
only means that when it
*does* happen, it will take all the longer to find and fix. I find 
languages with such traps
intensely frustrating to work with. If I can't even tell, *by reading 
it*, what my code means,
then what hope have I got?

There are plenty who will argue, quite reasonably, that lazy evaluation 
in itself makes
understanding performance behaviour unreasonably difficult; that strict 
languages
should be preferred because the space and time behaviour is much more 
obvious.
I don't agree, because I find the software engineering benefits of lazy 
evaluation to
be so large, that it's worth the cost in making performance reasoning 
more difficult
--but I accept there is a trade-off there. I don't see benefits on 
anything like the same scale
from dropping the M-R.

Today's M-R has the merit that it at least makes it evident, in the 
code, where
sharing may be lost. It's often a minor irritant, occasionally a bit 
more than minor.
Were it to be removed with no replacement, then the irritation would go 
away, and
the signs are that most of the time, performance would not change much. But
occasionally--just occasionally--there would be a space or time bug 
caused by
loss of sharing *somewhere* in a large body of code, and searching for 
it, we
would descend into nightmare. I'll put up with the irritation any day, to be
sure of being spared the nightmare.

John
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.haskell.org//pipermail/haskell-prime/attachments/20060202/f8daec83/attachment-0001.htm