[Haskell-cafe] Shootout favouring C

[Daniel Fischer]

Am Montag, 16. Januar 2006 18:56 schrieb Isaac Gouy:
> > > Shootout favouring C
> > > On 1/16/06, Daniel Fischer wrote:
> > > Is it only my machime, or can you confirm that for
> > > the Ackermann benchmark, it's very good for C that
>
> they chose
>
> > > 9 and not a larger value?
> >
> > Sebastian Sylvan <sebastian.sylvan at gmail.com> wrote:
> > This is interesting. Hopefully it's not intentional,

Although it'd give a great thread, just think what you'd have to do.
You'd have to write _good_ programmes in many languages and then try a lot of 
different inputs...

Probably, the Ack(3,9) benchmark was just taken over from the times when it 
took serious time to calculate.

However, I'm curious. Is the behaviour I reported peculiar to my flavour of 
linux, or does the same trend show on other machines?
Would you be so kind to test and report?
>
> Pardon my rudeness but this really is getting a bit
> much!
>
> Please keep to the true spirit of fictional crime
> writing and provide a motive for these evil characters
> who will stop at nothing to make Haskell seem some
> worse than C.

Jealousy?
>
> > but it's quite obvious that for benchmarks where the fastest
> > time is only a few fractions of a second, languages with more
> > complex runtime systems will be unfairly slow due to the
> > startup cost.
>
> Sebastian perhaps you'd like to provide something more
> substantive than "quite obvious".

Well, if initializing the run-time environment takes as long as for Java
(boy, I just timed Hello, World: 0.34s!!!!), no matter how fast the actual 
work is being done, you can't catch sufficiently fast languages with 
lightweight RTEs in those cases -- and I dare say Java does suffer from that 
even more than GHC.

> > There is already a startup benchmark in there
>
> Yes and if we make the huge assumption that it means
> anything at all, then we are being unfair to Haskell
> by 0.002s on every test - we only show measurements to
> 0.01s!

And you take the average of how many runs?
From my experience, even if I run nothing else, for tasks which take around 
0.1s, timings vary by up to 0.03s (and if the system starts some real work 
while the task is underway, it can be more).
So I don't consider timings of such short tasks very reliable and second 
Sebastians suggestion that all benchmarks should take at least -- well, I 
don't know how much -- 0.5s, 1s for the fastest contenders.
And that's not because apparently -- it might be violently different on other 
systems -- Haskell would gain ground over C if the Ackermann benchmark would 
be increased (though I'm happy if it does).
And though I've no reason to suppose it would help Haskell, for the same 
reasons, I'd like the fannkuch benchmark changed to Pfannkuchen(10).
>
> > In other words I'd prefer if all benchmarks are
> > reconfigured to target an execution time of at least a few
> > seconds for the fastest benchmarks.
>
> We run the Haskell regex-dna programs for 2500s -
> isn't that long enough?

Phew, 'tis.
The old song, not every language is good at everything...
>
> Let me join Simon Marlow in congratulating those who
> are using the Shootout to advertise what Haskell can
> do, by the straightforward approach of contributing
> faster, smaller, more elegant programs.

Agreed. Unfortunately, often the desire for speed wrecks elegance.
>
> best wishes, Isaac
>
Cheers,
Daniel