[Haskell-cafe] Re: OCaml list sees abysmal Language Shootout results

[Greg Buchholz]

Malcolm Wallace wrote:
> For instance, the shootout often requires that a task be carried out N
> times, to make the timings large enough to measure.  In all the naive
> Haskell implementations of these tasks, Haskell wins by a mile.  Why?
> Because the language quite reasonably says that if you multiply
> a matrix by itself N times, but only use the result of the last
> multiplication, well it is jolly well not going to bother computing
> the first (N-1) identical multiplications - what a waste of time!
> 
> So is it fair to compare the default lazy Haskell solution with all
> the eager solutions out there that laboriously do all this unnecessary
> work?  Apparently not, so we have gone to all kinds of trouble to slow
> the Haskell solution down, make it over-strict, do the work N times,
> and thereby have a "fair" performance test.  Huh.

    Well, the shootout appears to have two types of tests.  Each
individual test is labeled as either being implemented in the "Same Way"
or doing the "Same Thing".  I agree that the "Same Way" test are usually
too synthetic and too geared toward measuring artifacts of imperative
programming which aren't appropriate in Haskell.  But there are also the
"Same Thing" tests that are free to be coded in any style at all.  And
of the "Same Thing" tests, the largest slowdown in the GHC programs is
caused by lazy I/O (I think using better I/O routines would fix
"Reverse a File" and "Statistical Moments").  To get GHC to come out on
top of the CRAPS Scorecard, we need to emphasize the "Same Thing" tests
and downplay the "Same Way" tests as well as properly weighting lines of
code vs. memory consumption and speed.  Here's one way to do it...

    http://makeashorterlink.com/?O5BD42089

What we probably need to do is create some new tests which aren't as
phony and convince the powers-that-be to drop some of the synthetic
tests or convert them to "Same Thing" tests (I think the "Sum a Column
of Integers" and "Spell Checker" would be good candidates to convert).

Just for reference, here's a list of the "Same Thing" tests:

Simple TCP/IP Server
Matrix Multiplication
Statistical Moments
Process Instantiation
Reverse A File
Ring of Messages
Count Lines/Words/Chars
Word Frequency  


Greg Buchholz