On CI

Thu Mar 18 17:37:28 UTC 2021

To be clear: All performance tests that run as part of CI measure
allocations only. No wall clock time.
Those measurements are (mostly) deterministic and reproducible between
compiles of the same worktree and not impacted by thermal issues/hardware
at all.

Am Do., 18. März 2021 um 18:09 Uhr schrieb davean <davean at xkcd.com>:

> That really shouldn't be near system noise for a well constructed
> performance test. You might be seeing things like thermal issues, etc
> though - good benchmarking is a serious subject.
> Also we're not talking wall clock tests, we're talking specific metrics.
> The machines do tend to be bare metal, but many of these are entirely CPU
> performance independent, memory timing independent, etc. Well not quite but
> that's a longer discussion.
>
> The investigation of Haskell code performance is a very good thing to do
> BTW, but you'd still want to avoid regressions in the improvements you
> made. How well we can do that and the cost of it is the primary issue here.
>
> -davean
>
>
> On Wed, Mar 17, 2021 at 6:22 PM Karel Gardas <karel.gardas at centrum.cz>
> wrote:
>
>> On 3/17/21 4:16 PM, Andreas Klebinger wrote:
>> > Now that isn't really an issue anyway I think. The question is rather is
>> > 2% a large enough regression to worry about? 5%? 10%?
>>
>> 5-10% is still around system noise even on lightly loaded workstation.
>> Not sure if CI is not run on some shared cloud resources where it may be
>> even higher.
>>
>> I've done simple experiment of pining ghc compiling ghc-cabal and I've
>> been able to "speed" it up by 5-10% on W-2265.
>>
>> Also following this CI/performance regs discussion I'm not entirely sure
>> if  this is not just a witch-hunt hurting/beating mostly most active GHC
>> developers. Another idea may be to give up on CI doing perf reg testing
>> at all and invest saved resources into proper investigation of
>> GHC/Haskell programs performance. Not sure, if this would not be more
>> beneficial longer term.
>>
>> Just one random number thrown to the ring. Linux's perf claims that
>> nearly every second L3 cache access on the example above ends with cache
>> miss. Is it a good number or bad number? See stats below (perf stat -d
>> on ghc with +RTS -T -s -RTS').
>>
>> Good luck to anybody working on that!
>>
>> Karel
>>
>>
>> Linking utils/ghc-cabal/dist/build/tmp/ghc-cabal ...
>>   61,020,836,136 bytes allocated in the heap
>>    5,229,185,608 bytes copied during GC
>>      301,742,768 bytes maximum residency (19 sample(s))
>>        3,533,000 bytes maximum slop
>>              840 MiB total memory in use (0 MB lost due to fragmentation)
>>
>>                                      Tot time (elapsed)  Avg pause  Max
>> pause
>>   Gen  0      2012 colls,     0 par    5.725s   5.731s     0.0028s
>> 0.1267s
>>   Gen  1        19 colls,     0 par    1.695s   1.696s     0.0893s
>> 0.2636s
>>
>>   TASKS: 4 (1 bound, 3 peak workers (3 total), using -N1)
>>
>>   SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)
>>
>>   INIT    time    0.000s  (  0.000s elapsed)
>>   MUT     time   27.849s  ( 32.163s elapsed)
>>   GC      time    7.419s  (  7.427s elapsed)
>>   EXIT    time    0.000s  (  0.010s elapsed)
>>   Total   time   35.269s  ( 39.601s elapsed)
>>
>>   Alloc rate    2,191,122,004 bytes per MUT second
>>
>>   Productivity  79.0% of total user, 81.2% of total elapsed
>>
>>
>>  Performance counter stats for
>> '/export/home/karel/sfw/ghc-8.10.3/bin/ghc -H32m -O -Wall -optc-Wall -O0
>> -hide-all-packages -package ghc-prim -package base -package binary
>> -package array -package transformers -package time -package containers
>> -package bytestring -package deepseq -package process -package pretty
>> -package directory -package filepath -package template-haskell -package
>> unix --make utils/ghc-cabal/Main.hs -o
>> utils/ghc-cabal/dist/build/tmp/ghc-cabal -no-user-package-db -Wall
>> -fno-warn-unused-imports -fno-warn-warnings-deprecations
>> -DCABAL_VERSION=3,4,0,0 -DBOOTSTRAPPING -odir bootstrapping -hidir
>> bootstrapping libraries/Cabal/Cabal/Distribution/Fields/Lexer.hs
>> -ilibraries/Cabal/Cabal -ilibraries/binary/src -ilibraries/filepath
>> -ilibraries/hpc -ilibraries/mtl -ilibraries/text/src
>> libraries/text/cbits/cbits.c -Ilibraries/text/include
>> -ilibraries/parsec/src +RTS -T -s -RTS':
>>
>>          39,632.99 msec task-clock                #    0.999 CPUs
>> utilized
>>             17,191      context-switches          #    0.434 K/sec
>>
>>                  0      cpu-migrations            #    0.000 K/sec
>>
>>            899,930      page-faults               #    0.023 M/sec
>>
>>    177,636,979,975      cycles                    #    4.482 GHz
>>               (87.54%)
>>    181,945,795,221      instructions              #    1.02  insn per
>> cycle           (87.59%)
>>     34,033,574,511      branches                  #  858.718 M/sec
>>               (87.42%)
>>      1,664,969,299      branch-misses             #    4.89% of all
>> branches          (87.48%)
>>     41,522,737,426      L1-dcache-loads           # 1047.681 M/sec
>>               (87.53%)
>>      2,675,319,939      L1-dcache-load-misses     #    6.44% of all
>> L1-dcache hits    (87.48%)
>>        372,370,395      LLC-loads                 #    9.395 M/sec
>>               (87.49%)
>>        173,614,140      LLC-load-misses           #   46.62% of all
>> LL-cache hits     (87.46%)
>>
>>       39.663103602 seconds time elapsed
>>
>>       38.288158000 seconds user
>>        1.358263000 seconds sys
>> _______________________________________________
>> ghc-devs mailing list
>> ghc-devs at haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20210318/833ffcb8/attachment.html>