Observation on Hadrian's relative performance re current buildsystem

Herbert Valerio Riedel hvriedel at gmail.com
Fri Nov 17 13:08:11 UTC 2017

Hello GHC devs,

I took the opportunity to give Hadrian a test-run to see whether it
could live up to the big promise of delivering a "more scalable, faster"
system than the current GNU Make based system. Unfortunately, my
preliminary results don't back this claim, and actually make Hadrian
appear to be significantly slower.


Here's the summary of the results:

                         | Hadrian   | GNU Make  |
| Compiling `hadrian`    | 5m25s     | 0         |
| (one-time setup)       |           |           |
| build "all" at -j8     | 38m       | 33m       |
| no-op build at -j8     | 10.977s   | 3.258s    |
| "clean"                | 21s       | 51s       |

So, Hadrian is ~5 minutes than GNU Make (or even ~10 minutes if
you also count the one-time setup cost).

And what I personally consider a bit annoying is that it's ~3 times
slower detecting; i.e. you have to wait 11s for Hadrian to detect
there's nothing to be done which compared to GNU Make (which currently
needlessly re-runs Sphinx; so it could be even faster!) is very
noticeable to me.

There's a silver-lining though, deleting files is the part which is a
lot more costly in the GNU Make system currently since artifacts are
spread over several (scroll to the end of this email) subfolders
there. Whereas Hadrian did something we should have done for the GNU
Make system as well (and probably would have done sooner or later anyway
in order to support the srcdir!=buildir scheme that people are used to
from GNU Autotools projects); Hadrian places build-artifacts into a few
top-level folders, and so cleaning up is trivial and requires to unlink
only a few folders from the filesystem.


At the very least, I'd expect Hadrian to be as fast as the GNU Make
system (and ideally beat it, not the least as this was besides
maintainability its big promise), but so far it doesn't seem to deliver
that promise for me.

It could easily be that I'm comparing apples to oranges here or that
I've otherwise overlooked something, so let me describe how I came to
this conclusion:

I tried this on an reasonably idle Linux workstation with an Intel(R)
Core(TM) i7-3770 CPU @ 3.40GHz CPU, and with 32GiB RAM (i.e. the
filesystem content was well-cached into memory; NB: a ghc source tree +
compiled artifacts takes up about 4GiB on the filesystem).

I've started from a fresh Git clone, i.e.

  git clone --recursive git://git.haskell.org/ghc.git

followed by


At this point, we're at the common point from which both the Hadrian and
the GNU Make build-system would start diverging:

For the Hadrian build-system, we need to pay for a one-time setup, since
we need to build the `hadrian` executable (which requires to build the
in-tree lib:Cabal as an in-place library):

  $ time ./hadrian/build.sh --help
  real  5m25.992s
  user  6m19.196s
  sys   0m6.079s

I'm not too worried about this part, as there's a few tricks by which we
could likely bring that down to about 2 minutes or so, and we mostly pay
this setup-cost, when lib:Cabal and/or `hadrian` changes and requires to
be recompiled.

Now, after having made sure that the `hadrian` executable is fresh, I
started the actual build:

    $ time ./hadrian/build.sh -j8
    shakeArgsWith                        0.000s    0%                           
    Function shake                       0.178s    0%                           
    Database read                        0.000s    0%                           
    With database                        0.000s    0%                           
    Running rules                     2338.398s   99%  =========================
    Pool finished (5261 threads, 8 max)  0.002s    0%                           
    Lint checking                        0.111s    0%                           
    Total                             2338.690s  100%                           
    Build completed in 38:59m
    real 38m59.626s
    user 219m7.421s
    sys  11m7.584s

then I immediately re-issued the same command to test how long it takes
to perform a no-op build:

    $ time ./hadrian/build.sh -j8
    Up to date
    Up to date
    shakeArgsWith                        0.000s    0%                           
    Function shake                       0.183s    1%                           
    Database read                        0.144s    1%                           
    With database                        0.241s    2%                           
    Running rules                        9.379s   93%  =========================
    Pool finished (4165 threads, 8 max)  0.004s    0%                           
    Lint checking                        0.101s    1%                           
    Total                               10.051s  100%                           
    Build completed in 0:11m
    real 0m10.977s
    user 0m19.399s
    sys  0m2.443s

Same without -j8:

    $ time ./hadrian/build.sh 
    Up to date
    Up to date
    shakeArgsWith                     0.000s    0%                           
    Function shake                    0.175s    2%                           
    Database read                     0.123s    1%                           
    With database                     0.197s    2%                           
    Running rules                     7.710s   92%  =========================
    Pool finished (1 threads, 1 max)  0.003s    0%                           
    Lint checking                     0.084s    1%                           
    Total                             8.293s  100%                           
    Build completed in 0:09m
    real 0m9.196s
    user 0m8.656s
    sys	0m0.724s

And finally clean it again:

    $ sync
    $ time ./hadrian/build.sh clean
    Up to date
    Up to date
    | Remove directory _build/stage0
    | Remove directory _build/stage1
    | Remove directory _build/stage2
    | Remove directory _build/stage3
    | Remove directory inplace/bin
    | Remove directory inplace/lib
    | Remove directory sdistprep
    | Remove Hadrian files...
    | Remove directory _build/generated
    | Done. 
    shakeArgsWith                     0.000s    0%                           
    Function shake                    0.175s    0%                           
    Database read                     0.114s    0%                           
    With database                     0.205s    0%                           
    Running rules                    20.037s   97%  ======================== 
    Pool finished (1 threads, 1 max)  0.002s    0%                           
    Lint checking                     0.006s    0%                           
    Total                            20.540s  100%                           
    Build completed in 0:21m
    real 0m21.426s
    user 0m1.415s
    sys	0m1.045s


Running the full build via our rusty GNU Make system:

    $ time make V=0 -j8
    real 33m30.801s
    user 157m49.520s
    sys  6m49.289s

A no-op build:

    $ time make V=0 -j8
    ===--- building phase 0
    make --no-print-directory -f ghc.mk phase=0 phase_0_builds
    make[1]: Nothing to be done for 'phase_0_builds'.
    ===--- building phase 1
    make --no-print-directory -f ghc.mk phase=1 phase_1_builds
    make[1]: Nothing to be done for 'phase_1_builds'.
    ===--- building final phase
    make --no-print-directory -f ghc.mk phase=final all
    make -C utils/haddock/doc html SPHINX_BUILD=/usr/bin/sphinx-build
    /usr/bin/sphinx-build -b html . .build-html
    Running Sphinx v1.3.6
    loading translations [en]... done
    loading pickled environment... done
    building [mo]: targets for 0 po files that are out of date
    building [html]: targets for 0 source files that are out of date
    updating environment: 0 added, 0 changed, 0 removed
    looking for now-outdated files... none found
    no targets are out of date.
    build succeeded.
    cp -R utils/haddock/doc/.build-html utils/haddock/doc/haddock
    real	0m3.258s
    user	0m3.134s
    sys	0m0.283s

And finally `make clean`:

    $ sync
    $ time make clean
    make --no-print-directory -f ghc.mk clean CLEANING=YES
    "rm" -rf inplace/bin inplace/lib  
    "rm" -rf utils/touchy/dist  
    "rm" -rf inplace/lib/bin/touchy  
    "rm" -rf utils/unlit/dist  
    "rm" -rf inplace/lib/bin/unlit  
    "rm" -rf utils/unlit/dist-install  
    "rm" -rf utils/hp2ps/dist  
    "rm" -rf inplace/bin/hp2ps  
    "rm" -rf inplace/lib/bin/hp2ps  
    "rm" -rf utils/hp2ps/dist-install  
    "rm" -rf driver/split/dist inplace/lib/bin/ghc-split  
    "rm" -rf utils/genprimopcode/dist  
    "rm" -rf inplace/bin/genprimopcode  
    "rm" -rf  libffi/build libffi/stamp.ffi.static-shared.configure libffi/stamp.ffi.static-shared.build libffi/stamp.ffi.static-shared.install libffi/dist-install  
    "rm" -rf utils/deriveConstants/dist  
    "rm" -rf inplace/bin/deriveConstants  
    "rm" -rf  includes/ghcautoconf.h includes/ghcplatform.h includes/ghcversion.h  
    "rm" -rf rts/dist  
    "rm" -rf bindisttest/"install   dir"  bindisttest/HelloWorld bindisttest/HelloWorld.o bindisttest/HelloWorld.hi bindisttest/output  
    "rm" -rf utils/genapply/dist  
    "rm" -rf inplace/bin/genapply  
    "rm" -rf  libraries/integer-gmp/include/ghc-gmp.h libraries/integer-gmp/gmp/config.mk libraries/integer-gmp/gmp/libgmp.a libraries/integer-gmp/gmp/gmp.h libraries/integer-gmp/gmp/gmpbuild libraries/integer-gmp/gmp/gmp-6.1.2  
    "rm" -rf libraries/integer-gmp/gmp/objs  
    "rm" -rf libraries/integer-gmp/gmp/gmpbuild  
    "rm" -rf utils/haddock/dist  
    "rm" -rf inplace/bin/haddock  
    "rm" -rf inplace/lib/bin/haddock  
    "rm" -rf compiler/stage1  
    "rm" -rf compiler/stage2  
    "rm" -rf compiler/stage3  
    "rm" -rf utils/hsc2hs/dist  
    "rm" -rf inplace/bin/hsc2hs  
    "rm" -rf inplace/lib/bin/hsc2hs  
    "rm" -rf utils/hsc2hs/dist-install  
    "rm" -rf utils/ghc-pkg/dist  
    "rm" -rf inplace/bin/ghc-pkg  
    "rm" -rf inplace/lib/bin/ghc-pkg  
    "rm" -rf utils/ghc-pkg/dist-install  
    "rm" -rf utils/ghctags/dist-install  
    "rm" -rf inplace/bin/ghctags  
    "rm" -rf utils/check-api-annotations/dist-install  
    "rm" -rf inplace/bin/check-api-annotations  
    "rm" -rf utils/check-ppr/dist-install  
    "rm" -rf inplace/bin/check-ppr  
    "rm" -rf  utils/ghc-cabal/dist bootstrapping  
    "rm" -rf utils/ghc-cabal/dist-install  
    "rm" -rf utils/hpc/dist-install  
    "rm" -rf inplace/bin/hpc  
    "rm" -rf inplace/lib/bin/hpc  
    "rm" -rf utils/runghc/dist-install  
    "rm" -rf inplace/bin/runghc  
    "rm" -rf inplace/lib/bin/runghc  
    "rm" -rf ghc/stage1  
    "rm" -rf inplace/bin/ghc-stage1  
    "rm" -rf inplace/lib/bin/ghc-stage1  
    "rm" -rf ghc/stage2  
    "rm" -rf inplace/bin/ghc-stage2  
    "rm" -rf inplace/lib/bin/ghc-stage2  
    "rm" -rf ghc/stage3  
    "rm" -rf docs/users_guide/.doctrees-html/ docs/users_guide/.doctrees-pdf/ docs/users_guide/build-html/ docs/users_guide/build-pdf/ docs/users_guide/users_guide.pdf  
    "rm" -rf docs/users_guide/.doctrees-man/ docs/users_guide/build-man/  
    "rm" -rf utils/count_lines/dist inplace/bin/count_lines  
    "rm" -rf utils/compare_sizes/dist-install  
    "rm" -rf iserv/stage2  
    "rm" -rf inplace/lib/bin/ghc-iserv  
    "rm" -rf iserv/stage2_p  
    "rm" -rf inplace/lib/bin/ghc-iserv-prof  
    "rm" -rf iserv/stage2_dyn  
    "rm" -rf inplace/lib/bin/ghc-iserv-dyn  
    "rm" -f libraries/integer-gmp/include/HsIntegerGmp.h libraries/base/include/EventConfig.h mk/config.mk.old mk/project.mk.old compiler/ghc.cabal.old includes/GHCConstants.h includes/DerivedConstants.h includes/ghcautoconf.h includes/ghcplatform.h includes/ghcversion.h utils/ghc-pkg/Version.hs compiler/prelude/primops.txt   
    "rm" -rf includes/dist-derivedconstants  
    "rm" -rf inplace/bin  
    "rm" -rf inplace/lib  
    "rm" -rf libraries/bootstrapping.conf  
    "rm" -f mk/are-validating.mk  
    "rm" -rf libraries/ghc-boot-th/dist-install  
    "rm" -rf libraries/ghc-boot/dist-install  
    "rm" -rf libraries/ghci/dist-install  
    "rm" -rf libraries/base/dist-install  
    "rm" -rf libraries/ghc-prim/dist-install  
    "rm" -rf libraries/integer-gmp/dist-install  
    "rm" -rf libraries/integer-simple/dist-install  
    "rm" -rf libraries/template-haskell/dist-install  
    "rm" -rf libraries/array/dist-install  
    "rm" -rf libraries/binary/dist-install  
    "rm" -rf libraries/bytestring/dist-install  
    "rm" -rf libraries/Cabal/Cabal/dist-install  
    "rm" -rf libraries/ghc-compact/dist-install  
    "rm" -rf libraries/containers/dist-install  
    "rm" -rf libraries/deepseq/dist-install  
    "rm" -rf libraries/directory/dist-install  
    "rm" -rf libraries/filepath/dist-install  
    "rm" -rf libraries/haskeline/dist-install  
    "rm" -rf libraries/hpc/dist-install  
    "rm" -rf libraries/mtl/dist-install  
    "rm" -rf libraries/parsec/dist-install  
    "rm" -rf libraries/pretty/dist-install  
    "rm" -rf libraries/process/dist-install  
    "rm" -rf libraries/terminfo/dist-install  
    "rm" -rf libraries/text/dist-install  
    "rm" -rf libraries/time/dist-install  
    "rm" -rf libraries/transformers/dist-install  
    "rm" -rf libraries/unix/dist-install  
    "rm" -rf libraries/Win32/dist-install  
    "rm" -rf libraries/xhtml/dist-install  
    "rm" -rf libraries/parallel/dist-install  
    "rm" -rf libraries/stm/dist-install  
    "rm" -rf libraries/random/dist-install  
    "rm" -rf libraries/primitive/dist-install  
    "rm" -rf libraries/vector/dist-install  
    "rm" -rf libraries/dph/dph-base/dist-install  
    "rm" -rf libraries/dph/dph-prim-interface/dist-install  
    "rm" -rf libraries/dph/dph-prim-seq/dist-install  
    "rm" -rf libraries/dph/dph-prim-par/dist-install  
    "rm" -rf libraries/dph/dph-lifted-base/dist-install  
    "rm" -rf libraries/dph/dph-lifted-boxed/dist-install  
    "rm" -rf libraries/dph/dph-lifted-copy/dist-install  
    "rm" -rf libraries/dph/dph-lifted-vseg/dist-install  
    "rm" -rf libraries/ghc-boot-th/dist-boot  
    "rm" -rf libraries/ghc-boot/dist-boot  
    "rm" -rf libraries/ghci/dist-boot  
    "rm" -rf libraries/base/dist-boot  
    "rm" -rf libraries/ghc-prim/dist-boot  
    "rm" -rf libraries/integer-gmp/dist-boot  
    "rm" -rf libraries/integer-simple/dist-boot  
    "rm" -rf libraries/template-haskell/dist-boot  
    "rm" -rf libraries/array/dist-boot  
    "rm" -rf libraries/binary/dist-boot  
    "rm" -rf libraries/bytestring/dist-boot  
    "rm" -rf libraries/Cabal/Cabal/dist-boot  
    "rm" -rf libraries/ghc-compact/dist-boot  
    "rm" -rf libraries/containers/dist-boot  
    "rm" -rf libraries/deepseq/dist-boot  
    "rm" -rf libraries/directory/dist-boot  
    "rm" -rf libraries/filepath/dist-boot  
    "rm" -rf libraries/haskeline/dist-boot  
    "rm" -rf libraries/hpc/dist-boot  
    "rm" -rf libraries/mtl/dist-boot  
    "rm" -rf libraries/parsec/dist-boot  
    "rm" -rf libraries/pretty/dist-boot  
    "rm" -rf libraries/process/dist-boot  
    "rm" -rf libraries/terminfo/dist-boot  
    "rm" -rf libraries/text/dist-boot  
    "rm" -rf libraries/time/dist-boot  
    "rm" -rf libraries/transformers/dist-boot  
    "rm" -rf libraries/unix/dist-boot  
    "rm" -rf libraries/Win32/dist-boot  
    "rm" -rf libraries/xhtml/dist-boot  
    "rm" -rf libraries/parallel/dist-boot  
    "rm" -rf libraries/stm/dist-boot  
    "rm" -rf libraries/random/dist-boot  
    "rm" -rf libraries/primitive/dist-boot  
    "rm" -rf libraries/vector/dist-boot  
    "rm" -rf libraries/dph/dph-base/dist-boot  
    "rm" -rf libraries/dph/dph-prim-interface/dist-boot  
    "rm" -rf libraries/dph/dph-prim-seq/dist-boot  
    "rm" -rf libraries/dph/dph-prim-par/dist-boot  
    "rm" -rf libraries/dph/dph-lifted-base/dist-boot  
    "rm" -rf libraries/dph/dph-lifted-boxed/dist-boot  
    "rm" -rf libraries/dph/dph-lifted-copy/dist-boot  
    "rm" -rf libraries/dph/dph-lifted-vseg/dist-boot  
    "rm" -rf  libraries/ghc-boot-th/dist  libraries/ghc-boot/dist  libraries/ghci/dist  libraries/base/dist  libraries/ghc-prim/dist  libraries/integer-gmp/dist  libraries/integer-simple/dist  libraries/template-haskell/dist  libraries/array/dist  libraries/binary/dist  libraries/bytestring/dist  libraries/Cabal/Cabal/dist  libraries/ghc-compact/dist  libraries/containers/dist  libraries/deepseq/dist  libraries/directory/dist  libraries/filepath/dist  libraries/haskeline/dist  libraries/hpc/dist  libraries/mtl/dist  libraries/parsec/dist  libraries/pretty/dist  libraries/process/dist  libraries/terminfo/dist  libraries/text/dist  libraries/time/dist  libraries/transformers/dist  libraries/unix/dist  libraries/Win32/dist  libraries/xhtml/dist  libraries/parallel/dist  libraries/stm/dist  libraries/random/dist  libraries/primitive/dist  libraries/vector/dist  libraries/dph/dph-base/dist  libraries/dph/dph-prim-interface/dist  libraries/dph/dph-prim-seq/dist  libraries/dph/dph-prim-par/dist  libraries/dph/dph-lifted-base/dist  libraries/dph/dph-lifted-boxed/dist  libraries/dph/dph-lifted-copy/dist  libraries/dph/dph-lifted-vseg/dist  
    "rm" -f libraries/base/base.buildinfo libraries/integer-gmp/integer-gmp.buildinfo libraries/terminfo/terminfo.buildinfo libraries/unix/unix.buildinfo  
    "rm" -f  libraries/ghc-boot-th/config.log  libraries/ghc-boot/config.log  libraries/ghci/config.log  libraries/base/config.log  libraries/ghc-prim/config.log  libraries/integer-gmp/config.log  libraries/integer-simple/config.log  libraries/template-haskell/config.log  libraries/array/config.log  libraries/binary/config.log  libraries/bytestring/config.log  libraries/Cabal/Cabal/config.log  libraries/ghc-compact/config.log  libraries/containers/config.log  libraries/deepseq/config.log  libraries/directory/config.log  libraries/filepath/config.log  libraries/haskeline/config.log  libraries/hpc/config.log  libraries/mtl/config.log  libraries/parsec/config.log  libraries/pretty/config.log  libraries/process/config.log  libraries/terminfo/config.log  libraries/text/config.log  libraries/time/config.log  libraries/transformers/config.log  libraries/unix/config.log  libraries/Win32/config.log  libraries/xhtml/config.log  libraries/parallel/config.log  libraries/stm/config.log  libraries/random/config.log  libraries/primitive/config.log  libraries/vector/config.log  libraries/dph/dph-base/config.log  libraries/dph/dph-prim-interface/config.log  libraries/dph/dph-prim-seq/config.log  libraries/dph/dph-prim-par/config.log  libraries/dph/dph-lifted-base/config.log  libraries/dph/dph-lifted-boxed/config.log  libraries/dph/dph-lifted-copy/config.log  libraries/dph/dph-lifted-vseg/config.log  
    "rm" -f  libraries/ghc-boot-th/config.status  libraries/ghc-boot/config.status  libraries/ghci/config.status  libraries/base/config.status  libraries/ghc-prim/config.status  libraries/integer-gmp/config.status  libraries/integer-simple/config.status  libraries/template-haskell/config.status  libraries/array/config.status  libraries/binary/config.status  libraries/bytestring/config.status  libraries/Cabal/Cabal/config.status  libraries/ghc-compact/config.status  libraries/containers/config.status  libraries/deepseq/config.status  libraries/directory/config.status  libraries/filepath/config.status  libraries/haskeline/config.status  libraries/hpc/config.status  libraries/mtl/config.status  libraries/parsec/config.status  libraries/pretty/config.status  libraries/process/config.status  libraries/terminfo/config.status  libraries/text/config.status  libraries/time/config.status  libraries/transformers/config.status  libraries/unix/config.status  libraries/Win32/config.status  libraries/xhtml/config.status  libraries/parallel/config.status  libraries/stm/config.status  libraries/random/config.status  libraries/primitive/config.status  libraries/vector/config.status  libraries/dph/dph-base/config.status  libraries/dph/dph-prim-interface/config.status  libraries/dph/dph-prim-seq/config.status  libraries/dph/dph-prim-par/config.status  libraries/dph/dph-lifted-base/config.status  libraries/dph/dph-lifted-boxed/config.status  libraries/dph/dph-lifted-copy/config.status  libraries/dph/dph-lifted-vseg/config.status  
    "rm" -f libraries/base/include/HsBaseConfig.h libraries/process/include/HsProcessConfig.h libraries/unix/include/HsUnixConfig.h  
    "rm" -rf libraries/dist-haddock  
    "rm" -rf bindistprep/  
    test ! -d testsuite || make -C testsuite clean
    make[1]: Entering directory '/stuff3/work/GHC2/ghc/testsuite'
    make -C ./timeout clean
    make[2]: Entering directory '/stuff3/work/GHC2/ghc/testsuite/timeout'
    test ! -f Setup || ./Setup clean
    rm -f -rf install-inplace
    rm -f -f calibrate.out
    rm -f -f Setup Setup.exe Setup.hi Setup.o
    make[2]: Leaving directory '/stuff3/work/GHC2/ghc/testsuite/timeout'
    rm -f -f mk/*.o
    rm -f -f mk/*.hi
    rm -f -f mk/ghcconfig*.mk
    rm -f -f mk/ghc-config mk/ghc-config.exe
    rm -f -f driver/*.pyc
    make[1]: Leaving directory '/stuff3/work/GHC2/ghc/testsuite'
    real	0m50.990s
    user	0m0.496s
    sys	0m1.582s


More information about the ghc-devs mailing list