[GHC] #12758: Bring sanity to our performance testsuite

[GHC]

#12758: Bring sanity to our performance testsuite
-------------------------------------+-------------------------------------
        Reporter:  bgamari           |                Owner:  (none)
            Type:  task              |               Status:  new
        Priority:  high              |            Milestone:  8.8.1
       Component:  Test Suite        |              Version:  8.0.1
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):  Phab:D3758,
       Wiki Page:                    |  Phab:D5059
-------------------------------------+-------------------------------------

Comment (by Ben Gamari <ben@…>):

 In [changeset:"932cd41d8c7984c767c1b3b58e05146f69cc5c15/ghc"
 932cd41d/ghc]:
 {{{
 #!CommitTicketReference repository="ghc"
 revision="932cd41d8c7984c767c1b3b58e05146f69cc5c15"
 testsuite: Save performance metrics in git notes.

 This patch makes the following improvement:
   - Automatically records test metrics (per test environment) so that
     the programmer need not supply nor update expected values in *.T
     files.
     - On expected metric changes, the programmer need only indicate the
       direction of change in the git commit message.
   - Provides a simple python tool "perf_notes.py" to compare metrics
     over time.

 Issues:
   - Using just the previous commit allows performance to drift with each
     commit.
     - Currently we allow drift as we have a preference for minimizing
       false positives.
     - Some possible alternatives include:
       - Use metrics from a fixed commit per test: the last commit that
         allowed a change in performance (else the oldest metric)
       - Or use some sort of aggregate since the last commit that allowed
         a change in performance (else all available metrics)
       - These alternatives may result in a performance issue (with the
         test driver) having to heavily search git commits/notes.
   - Run locally, performance tests will trivially pass unless the tests
     were run locally on the previous commit. This is often not the case
     e.g.  after pulling recent changes.

 Previously, *.T files contain statements such as:
 ```
 stats_num_field('peak_megabytes_allocated', (2, 1))
 compiler_stats_num_field('bytes allocated',
                          [(wordsize(64), 165890392, 10)])
 ```
 This required the programmer to give the expected values and a tolerance
 deviation (percentage). With this patch, the above statements are
 replaced with:
 ```
 collect_stats('peak_megabytes_allocated', 5)
 collect_compiler_stats('bytes allocated', 10)
 ```
 So that programmer must only enter which metrics to test and a tolerance
 deviation. No expected value is required. CircleCI will then run the
 tests per test environment and record the metrics to a git note for that
 commit and push them to the git.haskell.org ghc repo. Metrics will be
 compared to the previous commit. If they are different by the tolerance
 deviation from the *.T file, then the corresponding test will fail. By
 adding to the git commit message e.g.
 ```
  # Metric (In|De)crease <metric(s)> <options>: <tests>
 Metric Increase ['bytes allocated', 'peak_megabytes_allocated'] \
          (test_env='linux_x86', way='default'):
     Test012, Test345
 Metric Decrease 'bytes allocated':
     Test678
 Metric Increase:
     Test711
 ```
 This will allow the noted changes (letting the test pass). Note that by
 omitting metrics or options, the change will apply to all possible
 metrics/options (i.e. in the above, an increase for all metrics in all
 test environments is allowed for Test711)

 phabricator will use the message in the description

 Reviewers: bgamari, hvr

 Reviewed By: bgamari

 Subscribers: rwbarton, carter

 GHC Trac Issues: #12758

 Differential Revision: https://phabricator.haskell.org/D5059
 }}}

-- 
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/12758#comment:19>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler