[commit: ghc] wip/t-12758_circleci_push_perf_notes: testsuite: Save performance metrics in git notes. (a4c7ddd)

[git at git.haskell.org]

Repository : ssh://git@git.haskell.org/ghc

On branch  : wip/t-12758_circleci_push_perf_notes
Link       : http://ghc.haskell.org/trac/ghc/changeset/a4c7ddd48929b394bc4aa4a684cf114285507822/ghc

>---------------------------------------------------------------

commit a4c7ddd48929b394bc4aa4a684cf114285507822
Author: David Eichmann <davide at Well-Typed.com>
Date:   Fri Oct 19 16:31:39 2018 +0100

    testsuite: Save performance metrics in git notes.
    
    Summary:
    This patch makes the following improvement:
      - Automatically records test metrics (per test environment) so that the programmer need not supply nor update expected values in *.T files.
        - On expected metric changes, the programmer need only indicate the direction of change in the git commit message.
      - Provides a simple python tool "perf_notes.py" to compare metrics over time.
    
    Issues:
      - Using just the previous commit allows performance to drift with each commit.
        - Currently we allow drift as we have a preference for minimizing false positives.
        - Some possible alternatives include:
          - Use metrics from a fixed commit per test: the last commit that allowed a change in performance (else the oldest metric)
          - Or use some sort of aggregate since the last commit that allowed a change in performance (else all available metrics)
          - These alternatives may result in a performance issue (with the test driver) having to heavily search git commits/notes.
      - Run locally, performance tests will trivially pass unless the tests were run locally on the previous commit. This is often not the case e.g. after pulling recent changes.
    
    Previously, *.T files contain statements such as:
    ```
    stats_num_field('peak_megabytes_allocated', (2, 1))
    compiler_stats_num_field('bytes allocated', [(wordsize(64), 165890392, 10)])
    ```
    This required the programmer to give the expected values and a tolerance deviation (percentage). With this patch, the above statements are replaced with:
    ```
    collect_stats('peak_megabytes_allocated', 5)
    collect_compiler_stats('bytes allocated', 10)
    ```
    So that programmer must only enter which metrics to test and a tolerance deviation. No expected value is required. CircleCI will then run the tests per test environment and record the metrics to a git note for that commit and push them to the git.haskell.org ghc repo. Metrics will be compared to the previous commit. If they are different by the tolerance deviation from the *.T file, then the corresponding test will fail. By adding to the git commit message e.g.
    ```
    # Metric (In|De)crease <metric(s)> <options>: <tests>
    Metric Increase ['bytes allocated', 'peak_megabytes_allocated'] (test_env='linux_x86', way='default'):
        Test012, Test345
    Metric Decrease 'bytes allocated':
        Test678
    Metric Increase:
        Test711
    ```
    This will allow the noted changes (letting the test pass). Note that by omitting metrics or options, the change will apply to all possible metrics/options (i.e. in the above, an increase for all metrics in all test environments is allowed for Test711)
    
    phabricator will use the message in the description
    
    Reviewers: bgamari, hvr
    
    Reviewed By: bgamari
    
    Subscribers: rwbarton, carter
    
    GHC Trac Issues: #12758
    
    Differential Revision: https://phabricator.haskell.org/D5059


>---------------------------------------------------------------

a4c7ddd48929b394bc4aa4a684cf114285507822
 .circleci/config.yml                           |  132 ++-
 libraries/Cabal                                |    2 +-
 libraries/base/tests/all.T                     |   18 +-
 libraries/deepseq                              |    2 +-
 libraries/stm                                  |    2 +-
 libraries/text                                 |    2 +-
 nofib                                          |    2 +-
 testsuite/driver/README.md                     |  133 +++
 testsuite/driver/perf_notes.py                 |  378 +++++++++
 testsuite/driver/runtests.py                   |   63 +-
 testsuite/driver/testglobals.py                |   30 +-
 testsuite/driver/testlib.py                    |  228 +++--
 testsuite/driver/testutil.py                   |   18 +
 testsuite/mk/test.mk                           |    8 +
 testsuite/tests/callarity/perf/all.T           |   12 +-
 testsuite/tests/deriving/perf/all.T            |   11 +-
 testsuite/tests/perf/compiler/all.T            | 1051 ++----------------------
 testsuite/tests/perf/haddock/all.T             |  183 +----
 testsuite/tests/perf/join_points/all.T         |   16 +-
 testsuite/tests/perf/should_run/all.T          |  362 ++------
 testsuite/tests/perf/space_leaks/all.T         |   75 +-
 testsuite/tests/pmcheck/should_compile/all.T   |   27 +-
 testsuite/tests/primops/should_run/all.T       |    6 +-
 testsuite/tests/simplCore/should_compile/all.T |    3 +-
 testsuite/tests/simplStg/should_run/all.T      |    5 +-
 utils/haddock                                  |    2 +-
 utils/hsc2hs                                   |    2 +-
 27 files changed, 1019 insertions(+), 1754 deletions(-)

Diff suppressed because of size. To see it, use:

    git diff-tree --root --patch-with-stat --no-color --find-copies-harder --ignore-space-at-eol --cc a4c7ddd48929b394bc4aa4a684cf114285507822