Problem with compiler perf tests

[Ömer Sinan Ağacan]

Hi,

Currently we have a bunch of tests in testsuite/tests/perf/compiler for keeping
compile time allocations, max residency etc. in the expected ranges and avoid
introducing accidental compile time performance regressions.

This has a problem: we expect every MR to keep the compile time stats in the
specified ranges, but sometimes a patch fixes an issue, or does something right
(removes hacks/refactors bad code etc.) but also increases the numbers because
sometimes doing it right means doing more work or keeping more things in memory
(e.g. !1747, !2100 which is required by !1304).

We then spend hours/days trying to shave a few bytes off in those patches,
because the previous hacky/buggy code set the standards. It doesn't make sense
to compare bad/buggy code with good code and expect them to do the same thing.

Second problem is that it forces the developer to focus on a tiny part of the
compiler to reduce the numbers to the where they were. If they looked at the big
picture instead it might be possible to see rooms of improvements in other
places that could be possibly lead to much more efficient use of the developer
time.

I think what we should do instead is that once it's clear that the patch did not
introduce *accidental* increases in numbers (e.g. in !2100 I checked and
explained the increase in average residency, and showed that the increase makes
sense and is not a leak) and it's the right thing to do, we should merge it, and
track the performance issues in another issue. The CI should still run perf
tests, but those should be allowed to fail.

Any opinions on this?

Ömer