Observation on Hadrian's relative performance re current buildsystem
Herbert Valerio Riedel
hvriedel at gmail.com
Fri Nov 17 13:08:11 UTC 2017
Hello GHC devs,
I took the opportunity to give Hadrian a test-run to see whether it
could live up to the big promise of delivering a "more scalable, faster"
system than the current GNU Make based system. Unfortunately, my
preliminary results don't back this claim, and actually make Hadrian
appear to be significantly slower.
----
Here's the summary of the results:
| Hadrian | GNU Make |
+------------------------+-----------+-----------+
| Compiling `hadrian` | 5m25s | 0 |
| (one-time setup) | | |
+------------------------+-----------+-----------+
| build "all" at -j8 | 38m | 33m |
+------------------------+-----------+-----------+
| no-op build at -j8 | 10.977s | 3.258s |
+------------------------+-----------+-----------+
| "clean" | 21s | 51s |
+------------------------+-----------+-----------+
So, Hadrian is ~5 minutes than GNU Make (or even ~10 minutes if
you also count the one-time setup cost).
And what I personally consider a bit annoying is that it's ~3 times
slower detecting; i.e. you have to wait 11s for Hadrian to detect
there's nothing to be done which compared to GNU Make (which currently
needlessly re-runs Sphinx; so it could be even faster!) is very
noticeable to me.
There's a silver-lining though, deleting files is the part which is a
lot more costly in the GNU Make system currently since artifacts are
spread over several (scroll to the end of this email) subfolders
there. Whereas Hadrian did something we should have done for the GNU
Make system as well (and probably would have done sooner or later anyway
in order to support the srcdir!=buildir scheme that people are used to
from GNU Autotools projects); Hadrian places build-artifacts into a few
top-level folders, and so cleaning up is trivial and requires to unlink
only a few folders from the filesystem.
----
At the very least, I'd expect Hadrian to be as fast as the GNU Make
system (and ideally beat it, not the least as this was besides
maintainability its big promise), but so far it doesn't seem to deliver
that promise for me.
It could easily be that I'm comparing apples to oranges here or that
I've otherwise overlooked something, so let me describe how I came to
this conclusion:
I tried this on an reasonably idle Linux workstation with an Intel(R)
Core(TM) i7-3770 CPU @ 3.40GHz CPU, and with 32GiB RAM (i.e. the
filesystem content was well-cached into memory; NB: a ghc source tree +
compiled artifacts takes up about 4GiB on the filesystem).
I've started from a fresh Git clone, i.e.
git clone --recursive git://git.haskell.org/ghc.git
followed by
./boot
./configure
At this point, we're at the common point from which both the Hadrian and
the GNU Make build-system would start diverging:
For the Hadrian build-system, we need to pay for a one-time setup, since
we need to build the `hadrian` executable (which requires to build the
in-tree lib:Cabal as an in-place library):
$ time ./hadrian/build.sh --help
...
real 5m25.992s
user 6m19.196s
sys 0m6.079s
I'm not too worried about this part, as there's a few tricks by which we
could likely bring that down to about 2 minutes or so, and we mostly pay
this setup-cost, when lib:Cabal and/or `hadrian` changes and requires to
be recompiled.
Now, after having made sure that the `hadrian` executable is fresh, I
started the actual build:
$ time ./hadrian/build.sh -j8
...
shakeArgsWith 0.000s 0%
Function shake 0.178s 0%
Database read 0.000s 0%
With database 0.000s 0%
Running rules 2338.398s 99% =========================
Pool finished (5261 threads, 8 max) 0.002s 0%
Lint checking 0.111s 0%
Total 2338.690s 100%
Build completed in 38:59m
real 38m59.626s
user 219m7.421s
sys 11m7.584s
then I immediately re-issued the same command to test how long it takes
to perform a no-op build:
$ time ./hadrian/build.sh -j8
Up to date
Up to date
shakeArgsWith 0.000s 0%
Function shake 0.183s 1%
Database read 0.144s 1%
With database 0.241s 2%
Running rules 9.379s 93% =========================
Pool finished (4165 threads, 8 max) 0.004s 0%
Lint checking 0.101s 1%
Total 10.051s 100%
Build completed in 0:11m
real 0m10.977s
user 0m19.399s
sys 0m2.443s
Same without -j8:
$ time ./hadrian/build.sh
Up to date
Up to date
shakeArgsWith 0.000s 0%
Function shake 0.175s 2%
Database read 0.123s 1%
With database 0.197s 2%
Running rules 7.710s 92% =========================
Pool finished (1 threads, 1 max) 0.003s 0%
Lint checking 0.084s 1%
Total 8.293s 100%
Build completed in 0:09m
real 0m9.196s
user 0m8.656s
sys 0m0.724s
And finally clean it again:
$ sync
$ time ./hadrian/build.sh clean
Up to date
Up to date
| Remove directory _build/stage0
| Remove directory _build/stage1
| Remove directory _build/stage2
| Remove directory _build/stage3
| Remove directory inplace/bin
| Remove directory inplace/lib
| Remove directory sdistprep
| Remove Hadrian files...
| Remove directory _build/generated
| Done.
shakeArgsWith 0.000s 0%
Function shake 0.175s 0%
Database read 0.114s 0%
With database 0.205s 0%
Running rules 20.037s 97% ========================
Pool finished (1 threads, 1 max) 0.002s 0%
Lint checking 0.006s 0%
Total 20.540s 100%
Build completed in 0:21m
real 0m21.426s
user 0m1.415s
sys 0m1.045s
----
Running the full build via our rusty GNU Make system:
$ time make V=0 -j8
real 33m30.801s
user 157m49.520s
sys 6m49.289s
A no-op build:
$ time make V=0 -j8
===--- building phase 0
make --no-print-directory -f ghc.mk phase=0 phase_0_builds
make[1]: Nothing to be done for 'phase_0_builds'.
===--- building phase 1
make --no-print-directory -f ghc.mk phase=1 phase_1_builds
make[1]: Nothing to be done for 'phase_1_builds'.
===--- building final phase
make --no-print-directory -f ghc.mk phase=final all
make -C utils/haddock/doc html SPHINX_BUILD=/usr/bin/sphinx-build
/usr/bin/sphinx-build -b html . .build-html
Running Sphinx v1.3.6
loading translations [en]... done
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 0 source files that are out of date
updating environment: 0 added, 0 changed, 0 removed
looking for now-outdated files... none found
no targets are out of date.
build succeeded.
cp -R utils/haddock/doc/.build-html utils/haddock/doc/haddock
real 0m3.258s
user 0m3.134s
sys 0m0.283s
And finally `make clean`:
$ sync
$ time make clean
make --no-print-directory -f ghc.mk clean CLEANING=YES
"rm" -rf inplace/bin inplace/lib
"rm" -rf utils/touchy/dist
"rm" -rf inplace/lib/bin/touchy
"rm" -rf utils/unlit/dist
"rm" -rf inplace/lib/bin/unlit
"rm" -rf utils/unlit/dist-install
"rm" -rf utils/hp2ps/dist
"rm" -rf inplace/bin/hp2ps
"rm" -rf inplace/lib/bin/hp2ps
"rm" -rf utils/hp2ps/dist-install
"rm" -rf driver/split/dist inplace/lib/bin/ghc-split
"rm" -rf utils/genprimopcode/dist
"rm" -rf inplace/bin/genprimopcode
"rm" -rf libffi/build libffi/stamp.ffi.static-shared.configure libffi/stamp.ffi.static-shared.build libffi/stamp.ffi.static-shared.install libffi/dist-install
"rm" -rf utils/deriveConstants/dist
"rm" -rf inplace/bin/deriveConstants
"rm" -rf includes/ghcautoconf.h includes/ghcplatform.h includes/ghcversion.h
"rm" -rf rts/dist
"rm" -rf bindisttest/"install dir" bindisttest/HelloWorld bindisttest/HelloWorld.o bindisttest/HelloWorld.hi bindisttest/output
"rm" -rf utils/genapply/dist
"rm" -rf inplace/bin/genapply
"rm" -rf libraries/integer-gmp/include/ghc-gmp.h libraries/integer-gmp/gmp/config.mk libraries/integer-gmp/gmp/libgmp.a libraries/integer-gmp/gmp/gmp.h libraries/integer-gmp/gmp/gmpbuild libraries/integer-gmp/gmp/gmp-6.1.2
"rm" -rf libraries/integer-gmp/gmp/objs
"rm" -rf libraries/integer-gmp/gmp/gmpbuild
"rm" -rf utils/haddock/dist
"rm" -rf inplace/bin/haddock
"rm" -rf inplace/lib/bin/haddock
"rm" -rf compiler/stage1
"rm" -rf compiler/stage2
"rm" -rf compiler/stage3
"rm" -rf utils/hsc2hs/dist
"rm" -rf inplace/bin/hsc2hs
"rm" -rf inplace/lib/bin/hsc2hs
"rm" -rf utils/hsc2hs/dist-install
"rm" -rf utils/ghc-pkg/dist
"rm" -rf inplace/bin/ghc-pkg
"rm" -rf inplace/lib/bin/ghc-pkg
"rm" -rf utils/ghc-pkg/dist-install
"rm" -rf utils/ghctags/dist-install
"rm" -rf inplace/bin/ghctags
"rm" -rf utils/check-api-annotations/dist-install
"rm" -rf inplace/bin/check-api-annotations
"rm" -rf utils/check-ppr/dist-install
"rm" -rf inplace/bin/check-ppr
"rm" -rf utils/ghc-cabal/dist bootstrapping
"rm" -rf utils/ghc-cabal/dist-install
"rm" -rf utils/hpc/dist-install
"rm" -rf inplace/bin/hpc
"rm" -rf inplace/lib/bin/hpc
"rm" -rf utils/runghc/dist-install
"rm" -rf inplace/bin/runghc
"rm" -rf inplace/lib/bin/runghc
"rm" -rf ghc/stage1
"rm" -rf inplace/bin/ghc-stage1
"rm" -rf inplace/lib/bin/ghc-stage1
"rm" -rf ghc/stage2
"rm" -rf inplace/bin/ghc-stage2
"rm" -rf inplace/lib/bin/ghc-stage2
"rm" -rf ghc/stage3
"rm" -rf docs/users_guide/.doctrees-html/ docs/users_guide/.doctrees-pdf/ docs/users_guide/build-html/ docs/users_guide/build-pdf/ docs/users_guide/users_guide.pdf
"rm" -rf docs/users_guide/.doctrees-man/ docs/users_guide/build-man/
"rm" -rf utils/count_lines/dist inplace/bin/count_lines
"rm" -rf utils/compare_sizes/dist-install
"rm" -rf iserv/stage2
"rm" -rf inplace/lib/bin/ghc-iserv
"rm" -rf iserv/stage2_p
"rm" -rf inplace/lib/bin/ghc-iserv-prof
"rm" -rf iserv/stage2_dyn
"rm" -rf inplace/lib/bin/ghc-iserv-dyn
"rm" -f libraries/integer-gmp/include/HsIntegerGmp.h libraries/base/include/EventConfig.h mk/config.mk.old mk/project.mk.old compiler/ghc.cabal.old includes/GHCConstants.h includes/DerivedConstants.h includes/ghcautoconf.h includes/ghcplatform.h includes/ghcversion.h utils/ghc-pkg/Version.hs compiler/prelude/primops.txt
"rm" -rf includes/dist-derivedconstants
"rm" -rf inplace/bin
"rm" -rf inplace/lib
"rm" -rf libraries/bootstrapping.conf
"rm" -f mk/are-validating.mk
"rm" -rf libraries/ghc-boot-th/dist-install
"rm" -rf libraries/ghc-boot/dist-install
"rm" -rf libraries/ghci/dist-install
"rm" -rf libraries/base/dist-install
"rm" -rf libraries/ghc-prim/dist-install
"rm" -rf libraries/integer-gmp/dist-install
"rm" -rf libraries/integer-simple/dist-install
"rm" -rf libraries/template-haskell/dist-install
"rm" -rf libraries/array/dist-install
"rm" -rf libraries/binary/dist-install
"rm" -rf libraries/bytestring/dist-install
"rm" -rf libraries/Cabal/Cabal/dist-install
"rm" -rf libraries/ghc-compact/dist-install
"rm" -rf libraries/containers/dist-install
"rm" -rf libraries/deepseq/dist-install
"rm" -rf libraries/directory/dist-install
"rm" -rf libraries/filepath/dist-install
"rm" -rf libraries/haskeline/dist-install
"rm" -rf libraries/hpc/dist-install
"rm" -rf libraries/mtl/dist-install
"rm" -rf libraries/parsec/dist-install
"rm" -rf libraries/pretty/dist-install
"rm" -rf libraries/process/dist-install
"rm" -rf libraries/terminfo/dist-install
"rm" -rf libraries/text/dist-install
"rm" -rf libraries/time/dist-install
"rm" -rf libraries/transformers/dist-install
"rm" -rf libraries/unix/dist-install
"rm" -rf libraries/Win32/dist-install
"rm" -rf libraries/xhtml/dist-install
"rm" -rf libraries/parallel/dist-install
"rm" -rf libraries/stm/dist-install
"rm" -rf libraries/random/dist-install
"rm" -rf libraries/primitive/dist-install
"rm" -rf libraries/vector/dist-install
"rm" -rf libraries/dph/dph-base/dist-install
"rm" -rf libraries/dph/dph-prim-interface/dist-install
"rm" -rf libraries/dph/dph-prim-seq/dist-install
"rm" -rf libraries/dph/dph-prim-par/dist-install
"rm" -rf libraries/dph/dph-lifted-base/dist-install
"rm" -rf libraries/dph/dph-lifted-boxed/dist-install
"rm" -rf libraries/dph/dph-lifted-copy/dist-install
"rm" -rf libraries/dph/dph-lifted-vseg/dist-install
"rm" -rf libraries/ghc-boot-th/dist-boot
"rm" -rf libraries/ghc-boot/dist-boot
"rm" -rf libraries/ghci/dist-boot
"rm" -rf libraries/base/dist-boot
"rm" -rf libraries/ghc-prim/dist-boot
"rm" -rf libraries/integer-gmp/dist-boot
"rm" -rf libraries/integer-simple/dist-boot
"rm" -rf libraries/template-haskell/dist-boot
"rm" -rf libraries/array/dist-boot
"rm" -rf libraries/binary/dist-boot
"rm" -rf libraries/bytestring/dist-boot
"rm" -rf libraries/Cabal/Cabal/dist-boot
"rm" -rf libraries/ghc-compact/dist-boot
"rm" -rf libraries/containers/dist-boot
"rm" -rf libraries/deepseq/dist-boot
"rm" -rf libraries/directory/dist-boot
"rm" -rf libraries/filepath/dist-boot
"rm" -rf libraries/haskeline/dist-boot
"rm" -rf libraries/hpc/dist-boot
"rm" -rf libraries/mtl/dist-boot
"rm" -rf libraries/parsec/dist-boot
"rm" -rf libraries/pretty/dist-boot
"rm" -rf libraries/process/dist-boot
"rm" -rf libraries/terminfo/dist-boot
"rm" -rf libraries/text/dist-boot
"rm" -rf libraries/time/dist-boot
"rm" -rf libraries/transformers/dist-boot
"rm" -rf libraries/unix/dist-boot
"rm" -rf libraries/Win32/dist-boot
"rm" -rf libraries/xhtml/dist-boot
"rm" -rf libraries/parallel/dist-boot
"rm" -rf libraries/stm/dist-boot
"rm" -rf libraries/random/dist-boot
"rm" -rf libraries/primitive/dist-boot
"rm" -rf libraries/vector/dist-boot
"rm" -rf libraries/dph/dph-base/dist-boot
"rm" -rf libraries/dph/dph-prim-interface/dist-boot
"rm" -rf libraries/dph/dph-prim-seq/dist-boot
"rm" -rf libraries/dph/dph-prim-par/dist-boot
"rm" -rf libraries/dph/dph-lifted-base/dist-boot
"rm" -rf libraries/dph/dph-lifted-boxed/dist-boot
"rm" -rf libraries/dph/dph-lifted-copy/dist-boot
"rm" -rf libraries/dph/dph-lifted-vseg/dist-boot
"rm" -rf libraries/ghc-boot-th/dist libraries/ghc-boot/dist libraries/ghci/dist libraries/base/dist libraries/ghc-prim/dist libraries/integer-gmp/dist libraries/integer-simple/dist libraries/template-haskell/dist libraries/array/dist libraries/binary/dist libraries/bytestring/dist libraries/Cabal/Cabal/dist libraries/ghc-compact/dist libraries/containers/dist libraries/deepseq/dist libraries/directory/dist libraries/filepath/dist libraries/haskeline/dist libraries/hpc/dist libraries/mtl/dist libraries/parsec/dist libraries/pretty/dist libraries/process/dist libraries/terminfo/dist libraries/text/dist libraries/time/dist libraries/transformers/dist libraries/unix/dist libraries/Win32/dist libraries/xhtml/dist libraries/parallel/dist libraries/stm/dist libraries/random/dist libraries/primitive/dist libraries/vector/dist libraries/dph/dph-base/dist libraries/dph/dph-prim-interface/dist libraries/dph/dph-prim-seq/dist libraries/dph/dph-prim-par/dist libraries/dph/dph-lifted-base/dist libraries/dph/dph-lifted-boxed/dist libraries/dph/dph-lifted-copy/dist libraries/dph/dph-lifted-vseg/dist
"rm" -f libraries/base/base.buildinfo libraries/integer-gmp/integer-gmp.buildinfo libraries/terminfo/terminfo.buildinfo libraries/unix/unix.buildinfo
"rm" -f libraries/ghc-boot-th/config.log libraries/ghc-boot/config.log libraries/ghci/config.log libraries/base/config.log libraries/ghc-prim/config.log libraries/integer-gmp/config.log libraries/integer-simple/config.log libraries/template-haskell/config.log libraries/array/config.log libraries/binary/config.log libraries/bytestring/config.log libraries/Cabal/Cabal/config.log libraries/ghc-compact/config.log libraries/containers/config.log libraries/deepseq/config.log libraries/directory/config.log libraries/filepath/config.log libraries/haskeline/config.log libraries/hpc/config.log libraries/mtl/config.log libraries/parsec/config.log libraries/pretty/config.log libraries/process/config.log libraries/terminfo/config.log libraries/text/config.log libraries/time/config.log libraries/transformers/config.log libraries/unix/config.log libraries/Win32/config.log libraries/xhtml/config.log libraries/parallel/config.log libraries/stm/config.log libraries/random/config.log libraries/primitive/config.log libraries/vector/config.log libraries/dph/dph-base/config.log libraries/dph/dph-prim-interface/config.log libraries/dph/dph-prim-seq/config.log libraries/dph/dph-prim-par/config.log libraries/dph/dph-lifted-base/config.log libraries/dph/dph-lifted-boxed/config.log libraries/dph/dph-lifted-copy/config.log libraries/dph/dph-lifted-vseg/config.log
"rm" -f libraries/ghc-boot-th/config.status libraries/ghc-boot/config.status libraries/ghci/config.status libraries/base/config.status libraries/ghc-prim/config.status libraries/integer-gmp/config.status libraries/integer-simple/config.status libraries/template-haskell/config.status libraries/array/config.status libraries/binary/config.status libraries/bytestring/config.status libraries/Cabal/Cabal/config.status libraries/ghc-compact/config.status libraries/containers/config.status libraries/deepseq/config.status libraries/directory/config.status libraries/filepath/config.status libraries/haskeline/config.status libraries/hpc/config.status libraries/mtl/config.status libraries/parsec/config.status libraries/pretty/config.status libraries/process/config.status libraries/terminfo/config.status libraries/text/config.status libraries/time/config.status libraries/transformers/config.status libraries/unix/config.status libraries/Win32/config.status libraries/xhtml/config.status libraries/parallel/config.status libraries/stm/config.status libraries/random/config.status libraries/primitive/config.status libraries/vector/config.status libraries/dph/dph-base/config.status libraries/dph/dph-prim-interface/config.status libraries/dph/dph-prim-seq/config.status libraries/dph/dph-prim-par/config.status libraries/dph/dph-lifted-base/config.status libraries/dph/dph-lifted-boxed/config.status libraries/dph/dph-lifted-copy/config.status libraries/dph/dph-lifted-vseg/config.status
"rm" -f libraries/base/include/HsBaseConfig.h libraries/process/include/HsProcessConfig.h libraries/unix/include/HsUnixConfig.h
"rm" -rf libraries/dist-haddock
"rm" -rf bindistprep/
test ! -d testsuite || make -C testsuite clean
make[1]: Entering directory '/stuff3/work/GHC2/ghc/testsuite'
make -C ./timeout clean
make[2]: Entering directory '/stuff3/work/GHC2/ghc/testsuite/timeout'
test ! -f Setup || ./Setup clean
rm -f -rf install-inplace
rm -f -f calibrate.out
rm -f -f Setup Setup.exe Setup.hi Setup.o
make[2]: Leaving directory '/stuff3/work/GHC2/ghc/testsuite/timeout'
rm -f -f mk/*.o
rm -f -f mk/*.hi
rm -f -f mk/ghcconfig*.mk
rm -f -f mk/ghc-config mk/ghc-config.exe
rm -f -f driver/*.pyc
make[1]: Leaving directory '/stuff3/work/GHC2/ghc/testsuite'
real 0m50.990s
user 0m0.496s
sys 0m1.582s
--
More information about the ghc-devs
mailing list