[commit: ghc] wip/perf-testsuite: Add more documentation and a README to the testsuite driver. (fae73dc)

[git at git.haskell.org]

Repository : ssh://git@git.haskell.org/ghc

On branch  : wip/perf-testsuite
Link       : http://ghc.haskell.org/trac/ghc/changeset/fae73dccc87c780e28b48546cfe238b60645dabe/ghc

>---------------------------------------------------------------

commit fae73dccc87c780e28b48546cfe238b60645dabe
Author: Jared Weakly <jweakly at pdx.edu>
Date:   Fri Sep 1 14:53:27 2017 -0700

    Add more documentation and a README to the testsuite driver.
    
    Additionally, add the modified make file that was left out of the last
    commit by mistake.


>---------------------------------------------------------------

fae73dccc87c780e28b48546cfe238b60645dabe
 testsuite/driver/README.md     | 24 ++++++++++++++++++++++--
 testsuite/driver/perf_notes.py |  8 ++++----
 testsuite/driver/runtests.py   | 13 -------------
 3 files changed, 26 insertions(+), 19 deletions(-)

diff --git a/testsuite/driver/README.md b/testsuite/driver/README.md
index 7b4b045..4417fb5 100644
--- a/testsuite/driver/README.md
+++ b/testsuite/driver/README.md
@@ -9,8 +9,9 @@ For more detailed documentation, please see [here][1].
 
 1. Entry points of the testsuite
 2. Quick overview of program parts
-3. Important Types
-4. Quick answers for "how do I do X"?
+3. How to use the comparison tool
+4. Important Types
+5. Quick answers for "how do I do X"?
 
 
 ## Entry Points of the Testsuite
@@ -40,6 +41,25 @@ The relevant bits of the directory tree are as such:
 ├── tests                    -- Main tests directory.
 ```
 
+## How to Use the Comparison Tool
+
+The comparison tool exists in `/driver/perf_notes.py`.
+
+When the testsuite is ran, the performance metrics of the performance tests are saved automatically in a local git note that will be attached to the commit.
+The comparison tool is designed to help analyze the performance of the compiler across commits using this performance information.
+
+Currently, it can only be ran by executing the file directly, like so:
+```
+$ python3 perf_notes.py (arguments go here)
+```
+
+If you run `perf_notes.py -h` you will see a description of all of the arguments and how to use them.
+The optional arguments exist to filter the output to include only commits that you're interested in.
+The most typical usage of this tool will likely be running `perf_notes.py HEAD 'HEAD~1' '(commit hash)' ...`
+
+The way the performance metrics are stored in git notes remains strictly local to the machine;
+as such, performance metrics will not exist for a commit until you checkout that commit and run the testsuite (or test).
+
 ## Important Types
 
 * metric
diff --git a/testsuite/driver/perf_notes.py b/testsuite/driver/perf_notes.py
index 075651f..0ff9d2d 100644
--- a/testsuite/driver/perf_notes.py
+++ b/testsuite/driver/perf_notes.py
@@ -5,8 +5,8 @@
 #
 # This file will be a utility to help facilitate the comparison of performance
 # metrics across arbitrary commits. The file will produce a table comparing
-# metrics between measurements taken for given commits in the environment given
-# by --test-env.
+# metrics between measurements taken for given commits in the environment
+# (which defaults to 'local' if not given by --test-env).
 #
 
 import argparse
@@ -22,10 +22,10 @@ def parse_arguments():
     parser.add_argument("--test-env",
                         help="The given test environment to be compared.")
     parser.add_argument("--test-name",
-                        help="Optional: If given, filters table to include only \
+                        help="If given, filters table to include only \
                         tests matching the given regular expression.")
     parser.add_argument("--min-delta",type=float,
-                        help="Optional: Display only tests where the relative \
+                        help="Display only tests where the relative \
                         spread is greater than the given value. \
                         This will not be run if you only pass in one commit.")
     parser.add_argument("--add-note", nargs=3,
diff --git a/testsuite/driver/runtests.py b/testsuite/driver/runtests.py
index da2ba3b..59d9e45 100644
--- a/testsuite/driver/runtests.py
+++ b/testsuite/driver/runtests.py
@@ -327,19 +327,6 @@ else:
     # Write our accumulated metrics into the git notes for this commit.
     note = subprocess.check_output(["git","notes","--ref=perf","append","-m", "\n".join(config.accumulate_metrics)])
 
-    # This here is loading up all of the git notes into memory.
-    # It's most likely in the wrong spot and I haven't fully fleshed out
-    # where exactly I'm putting this and how I'm refactoring the performance
-    # test running logic.
-    # Currently this is useful for debugging, at least.
-    if config.use_git_notes:
-            note = subprocess.check_output(["git","notes","--ref=perf","append","-m", "\n".join(config.accumulate_metrics)])
-            parse_git_notes('perf') # Should this be hardcoded? Most likely not...
-
-    # Write our accumulated metrics into the git notes for this commit.
-    if config.use_git_notes:
-            note = subprocess.check_output(["git","notes","--ref=perf","append","-m", "\n".join(config.accumulate_metrics)])
-
     if config.summary_file:
         with open(config.summary_file, 'w') as file:
             summary(t, file)