[GHC] #13426: compile-time memory-usage regression for DynFlags between GHC 8.0 and GHC 8.2

[GHC]

#13426: compile-time memory-usage regression for DynFlags between GHC 8.0 and GHC
8.2
-------------------------------------+-------------------------------------
           Reporter:  hvr            |             Owner:  (none)
               Type:  bug            |            Status:  new
           Priority:  high           |         Milestone:  8.2.1
          Component:  Compiler       |           Version:  8.1
           Keywords:                 |  Operating System:  Unknown/Multiple
       Architecture:                 |   Type of failure:  Compile-time
  Unknown/Multiple                   |  performance bug
          Test Case:                 |        Blocked By:
           Blocking:                 |   Related Tickets:
Differential Rev(s):                 |         Wiki Page:
-------------------------------------+-------------------------------------
 I notice that buildbots were having a hard-time building GHC 8.2
 snapshots... so I investigated...

 One thing that jumped at me was the memory usage while compiling DynFlags
 which is the worst memory-offender with 4777MiB of resident size:

 {{{
 "inplace/bin/ghc-stage1" -hisuf hi -osuf  o -hcsuf hc -static  -H32m -O
 -Wall   -Iincludes -Iincludes/dist -Iincludes/dist-derivedconstants/header
 -Iincludes/dist-ghcconstants/header   -this-unit-id ghc-8.2.0.201
 70313 -hide-all-packages -i -icompiler/backpack -icompiler/basicTypes
 -icompiler/cmm -icompiler/codeGen -icompiler/coreSyn -icompiler/deSugar
 -icompiler/ghci -icompiler/hsSyn -icompiler/iface -icompiler/llvmGen -
 icompiler/main -icompiler/nativeGen -icompiler/parser -icompiler/prelude
 -icompiler/profiling -icompiler/rename -icompiler/simplCore
 -icompiler/simplStg -icompiler/specialise -icompiler/stgSyn
 -icompiler/stranal
 -icompiler/typecheck -icompiler/types -icompiler/utils
 -icompiler/vectorise -icompiler/stage2/build -Icompiler/stage2/build
 -icompiler/stage2/build/./autogen -Icompiler/stage2/build/./autogen
 -Icompiler/. -Icompi
 ler/parser -Icompiler/utils -Icompiler/../rts/dist/build -Icompiler/stage2
 -optP-DGHCI -optP-include
 -optPcompiler/stage2/build/./autogen/cabal_macros.h -package-id
 base-4.10.0.0 -package-id deepseq-1.4.3.0 -pa
 ckage-id directory-1.3.0.2 -package-id process-1.4.3.0 -package-id
 bytestring-0.10.8.2 -package-id binary-0.8.4.1 -package-id time-1.8.0.1
 -package-id containers-0.5.10.2 -package-id array-0.5.1.2 -package-id fil
 epath-1.4.1.1 -package-id template-haskell-2.12.0.0 -package-id
 hpc-0.6.0.3 -package-id transformers-0.5.2.0 -package-id ghc-
 boot-8.2.0.20170313 -package-id ghc-boot-th-8.2.0.20170313 -package-id
 ghci-8.2.0.20170
 313 -package-id hoopl-3.10.2.2 -package-id unix-2.7.2.1 -package-id
 terminfo-0.4.0.2 -Wall -fno-warn-name-shadowing -this-unit-id ghc
 -XHaskell2010 -optc-DTHREADED_RTS -DGHCI_TABLES_NEXT_TO_CODE -DSTAGE=2
 -Rghc-t
 iming -O2  -no-user-package-db -rtsopts      -Wnoncanonical-monad-
 instances  -odir compiler/stage2/build -hidir compiler/stage2/build
 -stubdir compiler/stage2/build   -dynamic-too -c compiler/main/DynFlags.hs
 -o
 compiler/stage2/build/DynFlags.o -dyno
 compiler/stage2/build/DynFlags.dyn_o
 }}}

 {{{
 <<ghc: 63014533832 bytes, 4032 GCs, 395131448/1973969928 avg/max bytes
 residency (24 samples), 4777M in use, 0.000 INIT (0.000 elapsed), 32.207
 MUT (33.221 elapsed), 26.682 GC (26.685 elapsed) :ghc>>
 }}}


 Then I did two clean full builds of the `ghc-8.0` and `ghc-8.2` branch,
 and collected the -Rghc-timings output into simple 3-columns .csv files;

 Here's are the respective top-12 entries for GHC 8.0 and GHC 8.2
 respectively:

 {{{#!bash
 $ sort -t, -k2n build.80.csv | tail -n12
 "inplace/bin/ghc-stage1",560,compiler/stage2/build/TcRnDriver.o
 "inplace/bin/ghc-stage1",572,compiler/stage2/build/TcRnDriver.p_o
 "inplace/bin/ghc-stage1",572,compiler/stage2/build/TcSplice.o
 "inplace/bin/ghc-stage1",847,compiler/stage2/build/OptCoercion.p_o
 "/opt/ghc/bin/ghc",979,compiler/stage1/build/OptCoercion.o
 "/opt/ghc/bin/ghc",1036,compiler/stage1/build/DynFlags.o
 "/opt/ghc/bin/ghc",1040,compiler/stage1/build/X86/CodeGen.o
 "inplace/bin/ghc-stage1",1119,compiler/stage2/build/DynFlags.p_o
 "inplace/bin/ghc-stage1",1123,compiler/stage2/build/DynFlags.o
 "inplace/bin/ghc-stage1",1178,compiler/stage2/build/OptCoercion.o
 "inplace/bin/ghc-stage1",1233,compiler/stage2/build/X86/CodeGen.o
 "inplace/bin/ghc-stage1",1282,compiler/stage2/build/X86/CodeGen.p_o
 }}}

 {{{#!bash
 $ sort -t, -k2n build.82.csv | tail -n12
 "inplace/bin/ghc-stage1",891,compiler/stage2/build/CmmParse.p_o
 "/opt/ghc/bin/ghc",1053,compiler/stage1/build/DynFlags.o
 "/opt/ghc/bin/ghc",1058,compiler/stage1/build/OptCoercion.o
 "inplace/bin/ghc-stage1",1284,compiler/stage2/build/HsExpr.p_o
 "inplace/bin/ghc-stage1",1313,compiler/stage2/build/HsExpr.o
 "/opt/ghc/bin/ghc",1394,compiler/stage1/build/X86/CodeGen.o
 "inplace/bin/ghc-stage1",1495,compiler/stage2/build/X86/CodeGen.o
 "inplace/bin/ghc-stage1",1605,compiler/stage2/build/X86/CodeGen.p_o
 "inplace/bin/ghc-stage1",1620,compiler/stage2/build/OptCoercion.o
 "inplace/bin/ghc-stage1",1733,compiler/stage2/build/OptCoercion.p_o
 "inplace/bin/ghc-stage1",3989,compiler/stage2/build/DynFlags.p_o
 "inplace/bin/ghc-stage1",4777,compiler/stage2/build/DynFlags.o
 }}}


 What's apparent from this data:

  - GHC stage0 (which was GHC 8.0.2) is able to build DynFlags.o for GHC
 8.0 and GHC 8.2 w/ roughtly the same amount of max resident memory, i.e. 1
 GiB

  - **GHC 8.2 stage1 however needs 4 times the amount of memory more than
 GHC 8.0 to build the same `DynFlags.hs` module!**

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/13426>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler