[GHC] #12758: Bring sanity to our performance testsuite

[GHC]

#12758: Bring sanity to our performance testsuite
-------------------------------------+-------------------------------------
        Reporter:  bgamari           |                Owner:  (none)
            Type:  task              |               Status:  new
        Priority:  high              |            Milestone:  8.8.1
       Component:  Test Suite        |              Version:  8.0.1
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):  Phab:D3758
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by davide):

 In the case of expected changes in performance metrics, we still need the
 tests to pass. This requires some **convenient** mechanism by which the
 author can explicitly inform the CI of that change. The current plan is
 for the author to indicate such changes in the commit message in the
 following form:

 {{{
 Metric  (Increase | Decrease)  ['metric' | \['metrics',..\]]
 [\((test_env|way)='abc',...\)]: TestName01, TestName02, ...
 }}}

 e.g.

 {{{
 # All metrics decreased for test T1234.
 Metric Decrease: T1234

 # max_bytes_used decreased for test T1234.
 Metric Decrease 'max_bytes_used': T1234

 # bytes allocated and peak_megabytes_allocated increased for tests T005
 and T006.
 Metric Increase ['bytes allocated', 'peak_megabytes_allocated']:
     T005,
     T00

 # All metrics decreased for test T200 in the x86_linux environment
 Metric Decrease (test_env = 'x86_linux'): T200
 }}}

-- 
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12758#comment:16>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler