Plan of Attack for Parallel Builds

[Johan Tibell]

Hi Frank,

Thanks for reaching out and gathering input.

On Tue, Mar 29, 2011 at 9:54 PM, Frank Murphy <anirishduck at gmail.com> wrote:
> - Parallelize executeInstallPlan. When given a target load average as a flag it
>  will determine whether it should spawn a worker (if below the target load
>  average) or wait. If waiting, it will listen to all worker status channels
>  and print out their current build status and the load average. Once a worker
>  exits, it will again check the load average and spawn a new thread if
>  necessary.

I think the most important setting is the number of worker threads
(e.g. -jX). Load average sounds like a cool idea but I don't know how
well it'll work in practice. Gentoo's Portage uses it so you might
snoop around there for more info.

> - Rewrite install.*Package and their callees to use the CHP (Communicating
>  Haskell Process) monad where possible. Use channels to communicate build
>  status back to the main thread.

CHP might be a bit overkill, an MVar and a Chan or two should be
enough. At least start simple.

> - It might be necessary to parse the output of external builds in some way so
>  that meaningful status can be communicated back to the user.

I'm not sure this is worth it and even possible in the general case. See below.

> - Add a default parallel build log path template. Allow the user to specify one
>  on the command line to override the default.

I'm not quite sure what you mean here. Do you mean that we'd write
"cabal install" logs to e.g. .cabal/logs or something along those
lines?

> - On single-threaded (sequential) builds, revert to the old output style.

Sounds good. One possible policy would be: If you run "cabal build",
you get the old output format (and a single threaded build). If you
run "cabal install", you get the new output format, regardless of if
the build runs in parallel or not.

What do people think? Is it worth displaying all the build output for
"cabal install" in the single threaded case? Does the user care to see
it? Perhaps it's good for debugging to let single threaded "cabal
install" show the old output (i.e. if a parellel build fails, run the
single threaded one to get more output).

>  On multi-threaded builds, display the current status of all running builds, load
>  averages and nothing else. Possible output:
>
> Resolving dependencies...
> Building derive-2.3.0.2...                                            [17 of 58]
> Building regex-base-0.93.1...                                           [1 of 4]
> Building dyre-0.8.6...                                                  [5 of 7]
> Configuring xdg-basedir-0.2...                                     [in progress]
>
>                                                  Dependencies Built:  [0 of 9]
>                                                        Load Average: [3.4/4.0]
>                                                                Running 4 Jobs.

Cabal allows packages to use any build system they want (e.g. make),
which means that we can't know the progress of a single build in the
general case. Today, Cabal simply shows the stdout of the build
process, whatever it is. This means that we cannot show progress of
individual packages. I suggest something like (take from Gentoo's
Portage):

Building (1 of 9) derive-2.3.0.2...
Building (2 of 9) regex-base-0.93.1...
Building (3 of 9) dyre-0.8.6...
Building (4 of 9) aeson-0.3.2.1...
Building (5 of 9) binary-0.5.0.2...
Installing derive-2.3.0.2
Installing regex-base-0.93.1
Building (6 of 9) text-0.11.0.6...
Installing dyre-0.8.6
Jobs: 3 of 9 complete, 3 running               Load avg: 3.44, 1.46, 0.69

We could perhaps make a special case for the Simple build type and
parse the GHC output and show progress on individual builds. I don't
think it's worth it, at least not initially.

> A possible error message might look like:
>
> derive-2.3.0.2 failed during the building phase.
> Log stored in /home/frank/cabal/logs/build/derive-2.3.0.2.log

For build failures I think we should output the content of the log
file to stdout (as one chunk, using a lock to avoid interleaving).
This will make it quicker for users to get to the build failure. For
successful builds I don't think we need to output more than in the
example above.

Cheers,
Johan