On CI
davean
davean at xkcd.com
Thu Mar 18 17:39:51 UTC 2021
I left the wiggle room for things like longer wall time causing more time
events in the IO Manager/RTS which can be a thermal/HW issue.
They're small and indirect though
-davean
On Thu, Mar 18, 2021 at 1:37 PM Sebastian Graf <sgraf1337 at gmail.com> wrote:
> To be clear: All performance tests that run as part of CI measure
> allocations only. No wall clock time.
> Those measurements are (mostly) deterministic and reproducible between
> compiles of the same worktree and not impacted by thermal issues/hardware
> at all.
>
> Am Do., 18. März 2021 um 18:09 Uhr schrieb davean <davean at xkcd.com>:
>
>> That really shouldn't be near system noise for a well constructed
>> performance test. You might be seeing things like thermal issues, etc
>> though - good benchmarking is a serious subject.
>> Also we're not talking wall clock tests, we're talking specific metrics.
>> The machines do tend to be bare metal, but many of these are entirely CPU
>> performance independent, memory timing independent, etc. Well not quite but
>> that's a longer discussion.
>>
>> The investigation of Haskell code performance is a very good thing to do
>> BTW, but you'd still want to avoid regressions in the improvements you
>> made. How well we can do that and the cost of it is the primary issue here.
>>
>> -davean
>>
>>
>> On Wed, Mar 17, 2021 at 6:22 PM Karel Gardas <karel.gardas at centrum.cz>
>> wrote:
>>
>>> On 3/17/21 4:16 PM, Andreas Klebinger wrote:
>>> > Now that isn't really an issue anyway I think. The question is rather
>>> is
>>> > 2% a large enough regression to worry about? 5%? 10%?
>>>
>>> 5-10% is still around system noise even on lightly loaded workstation.
>>> Not sure if CI is not run on some shared cloud resources where it may be
>>> even higher.
>>>
>>> I've done simple experiment of pining ghc compiling ghc-cabal and I've
>>> been able to "speed" it up by 5-10% on W-2265.
>>>
>>> Also following this CI/performance regs discussion I'm not entirely sure
>>> if this is not just a witch-hunt hurting/beating mostly most active GHC
>>> developers. Another idea may be to give up on CI doing perf reg testing
>>> at all and invest saved resources into proper investigation of
>>> GHC/Haskell programs performance. Not sure, if this would not be more
>>> beneficial longer term.
>>>
>>> Just one random number thrown to the ring. Linux's perf claims that
>>> nearly every second L3 cache access on the example above ends with cache
>>> miss. Is it a good number or bad number? See stats below (perf stat -d
>>> on ghc with +RTS -T -s -RTS').
>>>
>>> Good luck to anybody working on that!
>>>
>>> Karel
>>>
>>>
>>> Linking utils/ghc-cabal/dist/build/tmp/ghc-cabal ...
>>> 61,020,836,136 bytes allocated in the heap
>>> 5,229,185,608 bytes copied during GC
>>> 301,742,768 bytes maximum residency (19 sample(s))
>>> 3,533,000 bytes maximum slop
>>> 840 MiB total memory in use (0 MB lost due to fragmentation)
>>>
>>> Tot time (elapsed) Avg pause Max
>>> pause
>>> Gen 0 2012 colls, 0 par 5.725s 5.731s 0.0028s
>>> 0.1267s
>>> Gen 1 19 colls, 0 par 1.695s 1.696s 0.0893s
>>> 0.2636s
>>>
>>> TASKS: 4 (1 bound, 3 peak workers (3 total), using -N1)
>>>
>>> SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)
>>>
>>> INIT time 0.000s ( 0.000s elapsed)
>>> MUT time 27.849s ( 32.163s elapsed)
>>> GC time 7.419s ( 7.427s elapsed)
>>> EXIT time 0.000s ( 0.010s elapsed)
>>> Total time 35.269s ( 39.601s elapsed)
>>>
>>> Alloc rate 2,191,122,004 bytes per MUT second
>>>
>>> Productivity 79.0% of total user, 81.2% of total elapsed
>>>
>>>
>>> Performance counter stats for
>>> '/export/home/karel/sfw/ghc-8.10.3/bin/ghc -H32m -O -Wall -optc-Wall -O0
>>> -hide-all-packages -package ghc-prim -package base -package binary
>>> -package array -package transformers -package time -package containers
>>> -package bytestring -package deepseq -package process -package pretty
>>> -package directory -package filepath -package template-haskell -package
>>> unix --make utils/ghc-cabal/Main.hs -o
>>> utils/ghc-cabal/dist/build/tmp/ghc-cabal -no-user-package-db -Wall
>>> -fno-warn-unused-imports -fno-warn-warnings-deprecations
>>> -DCABAL_VERSION=3,4,0,0 -DBOOTSTRAPPING -odir bootstrapping -hidir
>>> bootstrapping libraries/Cabal/Cabal/Distribution/Fields/Lexer.hs
>>> -ilibraries/Cabal/Cabal -ilibraries/binary/src -ilibraries/filepath
>>> -ilibraries/hpc -ilibraries/mtl -ilibraries/text/src
>>> libraries/text/cbits/cbits.c -Ilibraries/text/include
>>> -ilibraries/parsec/src +RTS -T -s -RTS':
>>>
>>> 39,632.99 msec task-clock # 0.999 CPUs
>>> utilized
>>> 17,191 context-switches # 0.434 K/sec
>>>
>>> 0 cpu-migrations # 0.000 K/sec
>>>
>>> 899,930 page-faults # 0.023 M/sec
>>>
>>> 177,636,979,975 cycles # 4.482 GHz
>>> (87.54%)
>>> 181,945,795,221 instructions # 1.02 insn per
>>> cycle (87.59%)
>>> 34,033,574,511 branches # 858.718 M/sec
>>> (87.42%)
>>> 1,664,969,299 branch-misses # 4.89% of all
>>> branches (87.48%)
>>> 41,522,737,426 L1-dcache-loads # 1047.681 M/sec
>>> (87.53%)
>>> 2,675,319,939 L1-dcache-load-misses # 6.44% of all
>>> L1-dcache hits (87.48%)
>>> 372,370,395 LLC-loads # 9.395 M/sec
>>> (87.49%)
>>> 173,614,140 LLC-load-misses # 46.62% of all
>>> LL-cache hits (87.46%)
>>>
>>> 39.663103602 seconds time elapsed
>>>
>>> 38.288158000 seconds user
>>> 1.358263000 seconds sys
>>> _______________________________________________
>>> ghc-devs mailing list
>>> ghc-devs at haskell.org
>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>>
>> _______________________________________________
>> ghc-devs mailing list
>> ghc-devs at haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20210318/f0d7cb84/attachment.html>
More information about the ghc-devs
mailing list