[GHC] #11980: Testsuite: run each test in its own /tmp directory, after copying required files

[GHC]

#11980: Testsuite: run each test in its own /tmp directory, after copying required
files
-------------------------------------+-------------------------------------
        Reporter:  thomie            |                Owner:  thomie
            Type:  task              |               Status:  patch
        Priority:  normal            |            Milestone:  8.2.1
       Component:  Test Suite        |              Version:  8.0.1
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):  Phab:D1187
       Wiki Page:                    |
-------------------------------------+-------------------------------------
Changes (by thomie):

 * status:  new => patch
 * differential:   => Phab:D1187


@@ -33,7 +33,1 @@
- building mk/ghc-config first.
-
- There are 2 problems with this:
- * you don't expect a call to `make clean` to start building stuff. Even
- less so in a pristine source distribution.
- * most of the time it doesn't clean all files, because lots of tests are
- missing are have incomplete `extra_clean`s.
+ building mk/ghc-config first. This will all become much simpler.

New description:

 == Plan ==
 For each TEST:
 * create a directory `<testdir>` inside `/tmp`
 * link/copy all source files that the test needs into `<testdir>`
 * run the test inside `<testdir>`
 * delete `<testdir>`

 == Benefits ==

 * running tests in parallel always works, even when two tests write to the
 same file (for example .hi/.o files, from sharing a .hs file). This should
 reduce the number of unsuspected and annoying validate failures. There is
 no longer any need to specify `-outputdir` explicitly.

 * there is no longer any need to add entries to the perpetually out-of-
 date `testsuite/.gitignore`.

 * there is no longer any need to specify `extra_clean` or `clean_cmd`
 setup functions for tests. Sometimes you //will// have to specify which
 extra files a test needs, using the (new) `extra_files` setup function,
 but it will be immediately clear when to do so (i.e. the test won't run if
 you don't). By default only files with the name `<testname>*` are copied
 to `<testdir>`.

 * it could become possible to run //ways// for a single test in parallel
 (i.e. `make TEST=<testname> slow` would become faster)

 * `clean`ing the testsuite won't be necessary (except for a few files in
 `testsuite/mk` perhaps)

 Currently the only way to clean the testsuite is via `make CLEANUP=1
 CLEAN_ONLY=1`, which requires a call to the testsuite, which requires
 building mk/ghc-config first. This will all become much simpler.

--

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/11980#comment:1>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler