Building GHC for performance testing

[Herbert Valerio Riedel]

On 2014-03-06 at 10:50:09 +0100, Johan Tibell wrote:
> I'd like to set up a performance build bot for GHC, but before I can do
> that I need a script that reliably builds GHC and runs nofib. Do we have
> such a script? Here's a strawman proposal for one:
>
>     cabal install happy alex
>     git clone git://git.haskell.org/ghc.git
>     cd ghc
>     ./sync-all --nofib get
>     perl boot
>     ./configure
>     make
>     cd nofib
>     make clean
>     make boot
>     make -k mode=slow
>
> Questions:
>
>  * Does this look sensible?

If the script is supposed to start in an empty environment, then it
looks good to me

>  * Is there a way to only build and run a subset of the benchmarks?
>  * Are there any tweaks to mk/build.mk we can do to make the build faster
> without compromising the results?

You should disable LaTeX generation, and maybe also haddock generation;
and possibly remove the libraries/dph folder (unless that's becomes part
of the benchmarks)

how often would you run the benchmarks? once a day?
 
>  * Is there a way to do this in a cheap throwaway VM like travis-ci does?
> Could such a VM already provide GHC and the required libs to make the whole
> thing hermetic?

Btw, does benchmarking inside a VM provide good enough measurements?
E.g. I'd be worried from noise due to vSMP scheduling (as GHC's GC is
rather sensible to proper SMP scheduling) and maybe from noise coming
from other VMs trashing the caches.