GHC Performance Tsar
nicolas at incubaid.com
Fri Nov 30 17:20:13 CET 2012
On Fri, 2012-11-30 at 15:51 +0000, Tim Watson wrote:
> Could we not configure travis-ci to run the benchmarks for us or
> something like that? A simple (free) ci setup would be easier than
> finding a pair of hands to do this regularly I would've thought.
AFAIK Travis uses some IAAS service (EC2 if I'm not mistaken) to execute
CI jobs. Experience has shown benchmark results from VMs (especially
running on some public/shared IAAS service) are rather useless: there's
huge variance between results, even when executing the exact same
binaries, depending on host CPU and IO load (the latter even in case
your benchmark doesn't perform any IO itself), time of day (which
influences load), actual hardware your VM gets deployed on,...
Just my .02,
More information about the Glasgow-haskell-users