making ./validate run tests on all CPUs by default

[Sergei Trofimovich]

On Wed, 13 Aug 2014 11:39:56 +0200
Tuncer Ayaz <tuncer.ayaz at gmail.com> wrote:

> On Tue, Aug 12, 2014 at 10:31 PM, Sergei Trofimovich wrote:
> > Good evening all!
> >
> > Currently when being  ran './validate' script (without any parameters):
> > - builds ghc using 2 parallel jobs
> > - runs testsuite using 2 parallel jobs
> >
> > I propose to change the default value to amount of available CPUs:
> > - build ghc using N+1 parallel jobs
> > - run testsuite using N+1 parallel jobs
> >
> > Pros:
> > - first-time users will get faster ./validate
> > - seasoned users will need less tweaking for buildbots
> >
> > Cons:
> > - for imbalanced boxes (32 cores, 8GB RAM) it might
> >    be quite painful to drag box out of swap
> >
> > What do you think about it?
> 
> Isn't the memory use also a problem on boxes with a much lower
> number of cores (e.g. 7.8 space leak(s))?
> 
> On one machine with 1GB per core, I had to limit cabal install's
> parallelism when using 7.8.

It's true in general, but I would not expect such a massive growth
on ghc source. Current -Rghc-timing shows ~300MBs per ghc process
on amd64.

The fallout examples are HsSyn and cabal's PackageDescription modules.

ghc's build system is a bit different from Cabal's:
- Cabal runs one 'ghc --make' instance for a whole package.
  It leads to massive RAM usage in case of a multitude of modules
  (highlighting-kate and qthaskell come to mind).
- ghc's buld system uses one 'ghc -c' execution for a single .hs file (roughly)

> Assuming the patch goes in, is there a way to limit parallel jobs
> on the command line?

The mechanism to set limit manually is the same as before:
    CPUS=8 ./validate

It's the default that is proposed to be changed.

-- 

  Sergei
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: not available
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20140813/1ab6e4e8/attachment.sig>