Test performance impact (was: The dreaded M-R)
rjmh at cs.chalmers.se
Thu Feb 2 04:51:58 EST 2006
Summary: 2 programs failed to compile due to type errors (anna, gg).
One program did 19% more allocation, a few other programs increased
allocation very slightly (<2%).
pic +0.28% +19.27% 0.02
Thanks, that was interesting. A follow-up question: pic has a space bug.
How long will it take you
to find and fix it? And how come speed improved slightly in many
cases--that seems counter-
Let me make clear that what concerns me is not the impact of the M-R on
space and time
performance on average. What concerns me is the difficulty of debugging
problems. Even if we Haskell programmers usually consider correctness
performance later, when we want to actually use our beautiful code, we
need to find and
fix at least the worst performance problems.
At that point, I'm reading the code and trying to understand--with the
help of profiling
tools--why it performs the way it does. I need the performance behaviour
of the code
to be *obvious*. When I pull out a common sub-expression into a
let-binding, I want
to *know* that it will be evaluated at most once. When I see a
collection of variable
bindings, likewise, I want to *know* that they are computed once only.
When I see
a seq on such a variable, I want to *know* that any pointers in its
are thereby dropped. When I introduce or remove overloading in a
I want to *know* that I have not thereby changed the sharing properties
of a variable
binding somewhere else, potentially introducing a space or time leak. I
want to know
these things by looking at the code in front of me, not by running the
compiler with the
right flags and reading its warning messages. Unexpected behaviour at
this point is a trap,
that can lead to many wasted hours as one reads the same code again and
again, assuming it
behaves as expected, only to eventually convince oneself, by long
examination of profiles,
that it cannot be doing so. If the unexpected behaviour is rare, that
only means that when it
*does* happen, it will take all the longer to find and fix. I find
languages with such traps
intensely frustrating to work with. If I can't even tell, *by reading
it*, what my code means,
then what hope have I got?
There are plenty who will argue, quite reasonably, that lazy evaluation
in itself makes
understanding performance behaviour unreasonably difficult; that strict
should be preferred because the space and time behaviour is much more
I don't agree, because I find the software engineering benefits of lazy
be so large, that it's worth the cost in making performance reasoning
--but I accept there is a trade-off there. I don't see benefits on
anything like the same scale
from dropping the M-R.
Today's M-R has the merit that it at least makes it evident, in the
sharing may be lost. It's often a minor irritant, occasionally a bit
more than minor.
Were it to be removed with no replacement, then the irritation would go
the signs are that most of the time, performance would not change much. But
occasionally--just occasionally--there would be a space or time bug
loss of sharing *somewhere* in a large body of code, and searching for
would descend into nightmare. I'll put up with the irritation any day, to be
sure of being spared the nightmare.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Haskell-prime