Perf metrics

[Ben Gamari]

Simon Peyton Jones via ghc-devs <ghc-devs at haskell.org> writes:

> When I run `sh validate -legacy`, towards the end I see stuff like this
>
> Performance Metrics (test environment: local):
>
>
>
> Conversions(normal)       runtime/bytes allocated                        107696.000
>
> DeriveNull(normal)        runtime/bytes allocated                     112050960.000
>
> InlineArrayAlloc(normal)  runtime/bytes allocated                    1600041088.000
>
> InlineByteArrayAlloc(normal)  runtime/bytes allocated                    1440041088.000
>
> InlineCloneArrayAlloc(normal)  runtime/bytes allocated                    1600041248.000
>
> ManyAlternatives(normal)  compile_time/bytes allocated                840985728.000
>
> ManyConstructors(normal)  compile_time/bytes allocated               4540766560.000
>
> MethSharing(normal)       runtime/peak_megabytes_allocated                    2.000
>
> MethSharing(normal)       runtime/bytes allocated                     480098136.000
>
> MultiLayerModules(normal)  compile_time/bytes allocated               5856970504.000
>
> ...
> It is intermingled with other apparently unrelated output.

Hmm, it really should be a distinct block of output. Are you saying that
you are seeing lines of unrelated output interspersed in the performance
metrics table?

> What should I conclude form this? Is it good or bad? By what amount
> have these figures changed, and relative to what?

Below the performance metrics table you should see a blurb of text like
the following:


    Missing Baseline Metrics these metrics trivially pass because a
    baseline (expected value) cannot be recovered from previous git
    commits. This may be due to HEAD having new tests or having expected
    changes, the presence of expected changes since the last run of the
    tests, and/or the latest test run being too old.

          MultiLayerModules
          ...

    If the tests exist on the previous commit (And are configured to run
    with the same ways), then check out that commit and run the tests to
    generate the missing metrics. Alternatively, a baseline may be
    recovered from ci results once fetched:

          git fetch https://gitlab.haskell.org/ghc/ghc-performance-notes.git refs/notes/perf:refs/notes/ci/perf

The suggested command will fetch up-to-date performance metrics from the
metrics repository (populated by CI). If you then run the testsuite
again you will see output comparing each test's output to the baseline
from CI. For instance,

    Performance Metrics (test environment: local):

    MultiLayerModules(normal)  compile_time/bytes allocated               5710260920.000
                               (baseline @ HEAD~1)                        5726848340.000  [unchanged 0.3%]

The "unchanged" here refers to the fact that the +0.3% observed change
is within the indicated acceptance window of the test.


> How can I run a single perf test?

Perf tests are treated like any other test. A single test can be run
under Hadrian with the following:

    ./hadrian/build.cabal.sh test --build-root=_validatebuild \
      --flavour=Validate --only=MultiLayerModules

Cheers,

- Ben
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20191219/4acdf8a1/attachment.sig>