[GHC] #11165: Testsuite framework failures are too easy to ignore and too hard to find

[GHC]

#11165: Testsuite framework failures are too easy to ignore and too hard to find
-------------------------------------+-------------------------------------
           Reporter:  bgamari        |             Owner:
               Type:  bug            |            Status:  new
           Priority:  normal         |         Milestone:
          Component:  Test Suite     |           Version:  7.10.2
           Keywords:                 |  Operating System:  Unknown/Multiple
       Architecture:                 |   Type of failure:  None/Unknown
  Unknown/Multiple                   |
          Test Case:                 |        Blocked By:
           Blocking:                 |   Related Tickets:
Differential Rev(s):                 |         Wiki Page:
-------------------------------------+-------------------------------------
 The testsuite driver really should do a better job of drawing the user's
 attention to framework failures. The only evidence in the testsuite
 summary that something went wrong is a non-zero "framework failures"
 count.

 If you notice that this count is non-zero then you need to sift manually
 through the initial log output produced by the driver and find the
 culprit. It's just generally terrible.

 Testsuite failures are failures, we should at least treat them as such and
 include them in the `TEST=...` message in the testsuite summary so they
 can be easily reproduced. Ideally we would also save the exception that
 caused the failure and show it in the summary as well.

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11165>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler