Attempt at a real world benchmark
mail at joachim-breitner.de
Fri Dec 9 14:26:43 UTC 2016
Am Freitag, den 09.12.2016, 13:54 +0800 schrieb Moritz Angermann:
> > I am not sure what you are saying. Are you proposing the maintain a
> > benchmark set outside GHC, or did you get the impression that I am
> > proposing it?
> Yes, that’s what *I* am proposing for the reasons I mentioned; one I
> did not yet mention is time. Running nofib takes time, adding more time
> consuming performance tests would reduce their likelihood of being run
> in my experience. As I see this as being almost completely scriptable,
> this could live outside of ghc i think.
I don’t think the running time of nofib is a constraint at the moment,
and I expect most who run nofib to happily let it run for a few minutes
more in order to get more meaningful results.
> > But maybe it is ok if it part of nofib, and hence of GHC, so that every
> > breaking change in GHC can immediately be accounted for in the
> > benchmark code.
> > A nice side effect of this might be that GHC developers can get a
> > better idea of how much code their change breaks.
> I’m not much a fan of this, but that’s just my opinion :-)
What is the alternative? Keep updating the libraries? But libraries
change APIs. Then you need to keep updating the program itself? That
seems to be too many moving parts for a benchmark suite.
> > > Something I’ve recently had some success with was dumping measurements
> > > into influxdb (or a similar data point collections service) and hook
> > > that up to grafana for visualization.
> > Nice! Although these seem to be tailored for data-over-time, not
> > data-over-commit. This mismatch in the data model was part of the
> > motivation for me to create gipeda, which powers
> > https://perf.haskell.org/ghc/
> Assuming we confine this to a particular branch, or discriminate by branch,
> commits would be measured in sequence anyway, and the timestamp could be the
> time of the reporting of the measurement, and the respective ghc commit hash
> end up being an annotation. While this is not very pretty (and I would hope
> that grafana has some other ability to enrich the hover-tooltips) it could
> present a flexible solution without requiring additional engineering effort.
> However, if gipeda is sufficient, please ignore my comment :)
Oh, we could certainly do better here! (But it serves my purposes for
now, so I’ll stick to it until someone sets up something better, in
which case I happily dump my code.)
Joachim “nomeata” Breitner
mail at joachim-breitner.de • https://www.joachim-breitner.de/
XMPP: nomeata at joachim-breitner.de • OpenPGP-Key: 0xF0FBF51F
Debian Developer: nomeata at debian.org
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 833 bytes
Desc: This is a digitally signed message part
More information about the ghc-devs