[Git][ghc/ghc][wip/backport-fragile] 7 commits: testsuite: Introduce fragile modifier

[Ben Gamari]

Ben Gamari pushed to branch wip/backport-fragile at Glasgow Haskell Compiler / GHC


Commits:
175d49e6 by Ben Gamari at 2019-06-12T00:07:25Z
testsuite: Introduce fragile modifier

Now since we have been a bit more stringent in testsuite cleanliness we
have been marking a lot of tests as fragile using the `skip` modifier.
However, this unfortunately means that we lose the association with the
ticket number documenting the fragility.

Here we introduce `fragile` and `fragile_for` to retain this
information.

(cherry picked from commit 4ca271d1880a6f4c5f49869de7f1920a2073adb6)

- - - - -
18ffc386 by Ben Gamari at 2019-06-12T00:07:25Z
testsuite: Mark heapprof001 as fragile on i386

(cherry picked from commit 910185a3eb5fd2148e42d39f6374ab03d098b682)

- - - - -
a9a34082 by Ben Gamari at 2019-06-12T00:07:25Z
testsuite: Mark heapprof001 as fragile on all platforms

See #15382.

(cherry picked from commit 23fc615679072a6fa433460a92f597af2ae388b2)

- - - - -
9c385294 by Ben Gamari at 2019-06-12T00:07:25Z
base: Mark CPUTime001 as fragile

As noted in #16224, CPUTime001 has been quite problematic, reporting
non-monotonic timestamps in CI. Unfortunately I've been unable to
reproduce this locally.

(cherry picked from commit 1a3420cabdcf6d7d90c154681230f1150604c097)

- - - - -
c993dee9 by Ben Gamari at 2019-06-12T00:07:25Z
testsuite: Mark T13167 as fragile in threaded2

As noted in #16536.

(cherry picked from commit b351004702c1a595bcedfa3ffeb4f816d5fd8503)

- - - - -
5993703c by Ben Gamari at 2019-06-12T00:07:25Z
testsuite: Fix fragile_for test modifier

(cherry picked from commit 658199cce0aabeed77f3bbbbde6abc0c5c3cc83d)

- - - - -
5e6f261a by Ben Gamari at 2019-06-12T00:07:25Z
testsuite: Mark threadstatus-T9333 as fragile in ghci way

As noted in #16555.

(cherry picked from commit 64b1684da09ddb3dc480bd0370adc7b002657a39)

- - - - -


4 changed files:

- libraries/base/tests/all.T
- testsuite/driver/testlib.py
- testsuite/tests/concurrent/should_run/all.T
- testsuite/tests/profiling/should_run/all.T


Changes:

=====================================
libraries/base/tests/all.T
=====================================
@@ -20,7 +20,7 @@ test('tempfiles', normal, compile_and_run, [''])
 test('fixed', normal, compile_and_run, [''])
 test('quotOverflow', normal, compile_and_run, [''])
 test('assert', exit_code(1), compile_and_run, ['-fno-ignore-asserts'])
-test('CPUTime001', normal, compile_and_run, [''])
+test('CPUTime001', fragile(16224), compile_and_run, [''])
 test('readLitChar',   normal, compile_and_run, [''])
 test('unicode001',
      when(platform('i386-unknown-openbsd'), expect_fail),
@@ -231,5 +231,5 @@ test('T3474',
 test('T14425', normal, compile_and_run, [''])
 test('T10412', normal, compile_and_run, [''])
 test('T13896', normal, compile_and_run, [''])
-test('T13167', normal, compile_and_run, [''])
+test('T13167', fragile_for(16536, ['threaded2']), compile_and_run, [''])
 test('T15349', [exit_code(1), expect_broken_for(15349, 'ghci')], compile_and_run, [''])


=====================================
testsuite/driver/testlib.py
=====================================
@@ -225,11 +225,35 @@ def _expect_pass(way):
 
 # -----
 
+def fragile( bug ):
+    """
+    Indicates that the test should be skipped due to fragility documented in
+    the given ticket.
+    """
+    def helper( name, opts, bug=bug ):
+        record_broken(name, opts, bug)
+        opts.skip = True
+
+    return helper
+
+def fragile_for( bug, ways ):
+    """
+    Indicates that the test should be skipped due to fragility in the given
+    test ways as documented in the given ticket.
+    """
+    def helper( name, opts, bug=bug, ways=ways ):
+        record_broken(name, opts, bug)
+        opts.omit_ways += ways
+
+    return helper
+
+# -----
+
 def omit_ways( ways ):
     return lambda name, opts, w=ways: _omit_ways( name, opts, w )
 
 def _omit_ways( name, opts, ways ):
-    opts.omit_ways = ways
+    opts.omit_ways += ways
 
 # -----
 


=====================================
testsuite/tests/concurrent/should_run/all.T
=====================================
@@ -89,7 +89,7 @@ test('T7970', normal, compile_and_run, [''])
 test('AtomicPrimops', normal, compile_and_run, [''])
 
 # test uses 2 threads and yield, scheduling can vary with threaded2
-test('threadstatus-9333', [omit_ways(['threaded2'])], compile_and_run, [''])
+test('threadstatus-9333', [fragile_for(16555, ['ghci']), omit_ways(['threaded2'])], compile_and_run, [''])
 
 test('T9379', normal, compile_and_run, [''])
 


=====================================
testsuite/tests/profiling/should_run/all.T
=====================================
@@ -24,7 +24,7 @@ expect_broken_for_10037 = expect_broken_for(
 
 test('heapprof001',
      [when(have_profiling(), extra_ways(extra_prof_ways)),
-      when(arch('i386'), expect_broken_for(15382, ['prof_hc_hb'])),
+      fragile(15382),
       extra_run_opts('7')],
      compile_and_run, [''])
 



View it on GitLab: https://gitlab.haskell.org/ghc/ghc/compare/086814df9fb89a9e1fb1e0f736d3a3ebedd48135...5e6f261aee196eb5984d192dcb01710b070452b3

-- 
View it on GitLab: https://gitlab.haskell.org/ghc/ghc/compare/086814df9fb89a9e1fb1e0f736d3a3ebedd48135...5e6f261aee196eb5984d192dcb01710b070452b3
You're receiving this email because of your account on gitlab.haskell.org.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-commits/attachments/20190611/487bd291/attachment-0001.html>