[Haskell-cafe] Shootout favouring C

[Sebastian Sylvan]

On 1/16/06, Isaac Gouy <igouy at yahoo.com> wrote:
> > > Shootout favouring C
> > > On 1/16/06, Daniel Fischer wrote:
> > > Is it only my machime, or can you confirm that for
> > > the Ackermann benchmark, it's very good for C that
> they chose
> > > 9 and not a larger value?
>
> > Sebastian Sylvan <sebastian.sylvan at gmail.com> wrote:
> > This is interesting. Hopefully it's not intentional,
>
> Pardon my rudeness but this really is getting a bit
> much!
>
> Please keep to the true spirit of fictional crime
> writing and provide a motive for these evil characters
> who will stop at nothing to make Haskell seem some
> worse than C.
>

I think you're being a bit paranoid here, nobody's claiming that
there's some hidden agenda, just that there are improvements that
could be made to increase accuracy and fairness.

Surely you can see how a benchmark where the top three entries all
have the same reported time would be better if run for a longer period
of time?
The larger problem sets you run your benchmarks on, the more accurate
the results will be. In this particular case, the results could even
be substantially different (if Daniel's results were to carry over the
the systems used in the shootout).

Do you not agree that if the results of a benchmark significantly
changes when increasing the problem set slightly, it's better to run
at a larger problem set?
I really don't understand why you get so upset when someone points out
improvements that could be made to the benchmarks. Isn't it better to
get more accurate (and fair) results?

This type of responses from you will certainly not help if you're
concerned that people think there is a deliberate effort to make C
look good. Why would you be against improving the accuracy of the
benchmarks in cases where that would make C look worse?

Ideally every benchmark should be run at a problem size large enough
so that the rankings of the entries don't change substantially when
increasing it (this size could be found automatically). That's just
plain old common sense for benchmarking. Nothing to take personally
and get upset about.
In practice this may not be feasible for all of the benchmarks (I
assume the shootout machines have better things to do than to run
benchmarks 24/7) so it may be best to have a ceiling of a few seconds
or so for the fastest benchmark.


/S

--
Sebastian Sylvan
+46(0)736-818655
UIN: 44640862