[Haskell-cafe] Re: [Haskell] [Fwd: Re: Computer Language Shootout]

[Kirsten Chevalier]

On 1/25/07, Donald Bruce Stewart <dons at cse.unsw.edu.au> wrote:
> The degradation is due to two things:
>
>     * several entries have been disqualified (some fairly, some unfairly)
>        Fix: fix is to submit more
>
>     * the shootout haskellers stopped submitting once it was clear we'd
>       need bytestrings to do signifcantly better.
>
>        Fix: bytestring is now on the shootout machine, so we can resume
>
> I don't think there is any fundamental issue exposed, other than that we
> stopped updating entries, while other languages decided they should try
> harder.

I have to disagree with this. That is, I don't object to Don's
explanation of why the shootout entries degraded in this particular
case, but I do think that Andrzej was right to point this out:
"Perhaps making a collective effort towards benchmarking Haskell programs and
analyzing the results in some methodic way could prove helpful?"
and I think that he *is* pointing out a fundamental issue here.

Maybe it's just me, but as someone who has some amount of experience
with implementing optimizations for Haskell, I find it nearly
impossible to precisely understand and measure exactly how those
optimizations improve (or degrade) program performance. The tools we
have just aren't adequate. And so, at least for me, implementing
optimizations feels more like fumbling around in the dark than like
using anything approaching the scientific method.

This is, of course, because everybody has limited resources and
implementing good profiling tools for implementors hasn't been a high
priority. And Alexey Rodriguez's recent work on using hardware
counters is a step in the right direction. But, I think the time is
now to put more effort into profiling and benchmarks for Haskell.

Cheers,
Kirsten

-- 
Kirsten Chevalier* chevalier at alum.wellesley.edu *Often in error, never in doubt
"I wish people weren't so set on being themselves, when that means being a
bastard." -- Robertson Davies