[commit: ghc] wip/T16042: testsuite: Warn if test will never be run (b8ef54f)

[git at git.haskell.org]

Repository : ssh://git@git.haskell.org/ghc

On branch  : wip/T16042
Link       : http://ghc.haskell.org/trac/ghc/changeset/b8ef54f90c7171f023fb311db6d74d1864911b69/ghc

>---------------------------------------------------------------

commit b8ef54f90c7171f023fb311db6d74d1864911b69
Author: Ben Gamari <ben at smart-cactus.org>
Date:   Sun Dec 16 10:43:11 2018 -0500

    testsuite: Warn if test will never be run
    
    We currently use the test's func to determine whether it is a compile-time or a
    run-time test. This means that run_command tests and the like are considered to
    be neither run-time nor compile-time tests and only get run in the normal way.
    This is quite surprising (see #16042) and has meant that some tests have gone
    silently un-run in the past. Warn when this happens.
    
    Frankly, this design is a bit unfortunate but changing it is a bit more work
    than I have time to invest at the moment.


>---------------------------------------------------------------

b8ef54f90c7171f023fb311db6d74d1864911b69
 testsuite/driver/testlib.py                  | 13 +++++++++++++
 testsuite/tests/codeGen/should_compile/all.T |  2 +-
 2 files changed, 14 insertions(+), 1 deletion(-)

diff --git a/testsuite/driver/testlib.py b/testsuite/driver/testlib.py
index 1c6668d..0d37358 100644
--- a/testsuite/driver/testlib.py
+++ b/testsuite/driver/testlib.py
@@ -762,6 +762,19 @@ def test_common_work(watcher, name, opts, func, args):
             else:
                 all_ways = []
         else:
+            if getTestOpts().only_ways is not None and getTestOpts().only_ways != ['normal']:
+                # This is all a bit unfortunate: We use the test's func to determine
+                # whether it is a compile-time or a run-time test. This means that
+                # run_command tests and the like are considered to be neither
+                # run-time nor compile-time tests and only get run in the normal way.
+                # This is quite surprising (see #16042) and has meant that some
+                # tests have gone silently un-run in the past. Warn when this happens.
+                err = '''
+                this test is neither a run-time nor a compile-time test yet and
+                consequently can only run in the 'normal' testsuite way.
+                However, it is configured to run only in the %s ways.
+                ''' % getTestOpts().only_ways
+                framework_fail(name, 'whole-test', err)
             all_ways = ['normal']
 
         # A test itself can request extra ways by setting opts.extra_ways
diff --git a/testsuite/tests/codeGen/should_compile/all.T b/testsuite/tests/codeGen/should_compile/all.T
index 7217d93..e3d0bbc 100644
--- a/testsuite/tests/codeGen/should_compile/all.T
+++ b/testsuite/tests/codeGen/should_compile/all.T
@@ -45,7 +45,7 @@ test('T14999',
 # Verify that we optimize away redundant jumps for unordered comparisons.
 test('T15196',
   [ unless(arch('x86_64'),skip),
-    only_ways('normal'),
+    only_ways(['normal']),
   ], run_command, ['$MAKE -s --no-print-directory T15196'])
 
 test('T15723',