Proposal: Add a benchmark section to .cabal files

[Duncan Coutts]

On Mon, 2011-10-03 at 22:59 -0700, Johan Tibell wrote:

> Proposal
> ========
> 
> Step 1. Add basic support for building simple benchmark suites e.g. modelled
> after test suite sections:
> 
> Name:           foo
> Version:        1.0
> License:        BSD3
> Cabal-Version:  >= 1.9.2
> Build-Type:     Simple
> 
> Benchmark bench-foo
>     type:           stdio-1.0
>     main-is:        bench-foo.hs
>     build-depends:  base
> 
> This doesn't add much on top of normal executable sections, except that it
> tells Cabal that this isn't an executable it should build by default or
> install.

Just to point out that the closest workaround at the moment is to add an
executable but to mark it as not buildable by default.

> Step 2. Add a detailed benchmark interface that allows the benchmark to
> associate a real valued *measure* (optionally with upper and lower bounds)
> with a benchmark name. Benchmark naming would work like test naming (e.g.
> group/subgroup/name). This would allow us to track performance over time.
> 
> Step 2 needs to be fleshed out some more, but I think we can get quite a lot
> of bang-for-the-buck with step 1.

In general, yes I think benchmarking is a sensible follow-on from test
suites. We can follow the same approach of specifying a protocol so that
we can add more sophisticated interfaces later as we gain experience wit
them.

Duncan