[Hackage] #215: Overhaul support for package's tests

[Hackage]

#215: Overhaul support for package's tests
------------------------------+---------------------------------------------
  Reporter:  duncan           |        Owner:         
      Type:  task             |       Status:  new    
  Priority:  normal           |    Milestone:         
 Component:  Cabal library    |      Version:  1.2.3.0
  Severity:  normal           |     Keywords:         
Difficulty:  project(> week)  |   Ghcversion:  6.8.2  
  Platform:                   |  
------------------------------+---------------------------------------------
 The current support for running package's testsuites is almost uniformly
 unused and packages instead implement ad-hoc solutions which makes it hard
 for users to run the testsuites and impossible to automate on hackage.

 The current support is so trivial anyone who takes this task on can ignore
 it and start with a clean slate.

 A [http://www.haskell.org/pipermail/cabal-devel/2008-January/001661.html
 thread on the issue].

 Here are some ideas for possible requirements:
  * Support tests even in the `Simple` build system. Do not require using
 the `build-type: Custom` just to be able to use tests.
  * Define a standard interface for returning test results from a testsuite
 to Cabal so they can be included into a build report and could be
 published on the hackage website or some other custom in-house test
 server.
  * Support for above interface in `QuickCheck` and `HUnit`.
  * Support for HPC when running testsuite programs, and gathering coverage
 results, again for reporting purposes.
  * We want to be able to run tests on a library that is not installed yet.
 This will rely on support in Cabal for registering a package inplace in a
 local package db, and building test code against that.
  * What about performance tests? What format should that use for
 reporting?

 The emphasis is on automation. A lot of interesting information becomes
 available with enough automation. For example we could compare on hackage
 which packages have more comprehensive testsuites (measured by HPC
 coverage). Or we could track performance over time. Once the data is
 available there is a lot we could do with it. It's not just for a public
 hackage server, developers could use this privately too.

-- 
Ticket URL: <http://hackage.haskell.org/trac/hackage/ticket/215>
Hackage <http://haskell.org/cabal/>
Hackage: Cabal and related projects