Attempt at a real world benchmark

[Joachim Breitner]

Hi,

Am Freitag, den 09.12.2016, 13:54 +0800 schrieb Moritz Angermann:
> > I am not sure what you are saying. Are you proposing the maintain a
> > benchmark set outside GHC, or did you get the impression that I am
> > proposing it?
> 
> Yes, that’s what *I* am proposing for the reasons I mentioned; one I
> did not yet mention is time. Running nofib takes time, adding more time
> consuming performance tests would reduce their likelihood of being run
> in my experience.  As I see this as being almost completely scriptable,
> this could live outside of ghc i think. 

I don’t think the running time of nofib is a constraint at the moment,
and I expect most who run nofib to happily let it run for a few minutes
more in order to get more meaningful results.

> 
> > But maybe it is ok if it part of nofib, and hence of GHC, so that every
> > breaking change in GHC can immediately be accounted for in the
> > benchmark code.
> > 
> > A nice side effect of this might be that GHC developers can get a
> > better idea of how much code their change breaks.
> 
> I’m not much a fan of this, but that’s just my opinion :-)

What is the alternative? Keep updating the libraries? But libraries
change APIs. Then you need to keep updating the program itself? That
seems to be too many moving parts for a benchmark suite.


> > > Something I’ve recently had some success with was dumping measurements
> > > into influxdb[1] (or a similar data point collections service) and hook
> > > that up to grafana[2] for visualization.
> > 
> > Nice! Although these seem to be tailored for data-over-time, not
> > data-over-commit. This mismatch in the data model was part of the
> > motivation for me to create gipeda, which powers
> > https://perf.haskell.org/ghc/
> 
> Assuming we confine this to a particular branch, or discriminate by branch,
> commits would be measured in sequence anyway, and the timestamp could be the
> time of the reporting of the measurement, and the respective ghc commit hash
> end up being an annotation. While this is not very pretty (and I would hope
> that grafana has some other ability to enrich the hover-tooltips) it could
> present a flexible solution without requiring additional engineering effort.
> 
> However, if gipeda is sufficient, please ignore my comment :)

Oh, we could certainly do better here! (But it serves my purposes for
now, so I’ll stick to it until someone sets up something better, in
which case I happily dump my code.)

Greetings,
Joachim

-- 
Joachim “nomeata” Breitner
  mail at joachim-breitner.de • https://www.joachim-breitner.de/
  XMPP: nomeata at joachim-breitner.de • OpenPGP-Key: 0xF0FBF51F
  Debian Developer: nomeata at debian.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20161209/458367e4/attachment.sig>