[Git][ghc/ghc][master] 15 commits: ci: make lint-ci-config job fast again
Marge Bot (@marge-bot)
gitlab at gitlab.haskell.org
Sat Apr 1 01:28:26 UTC 2023
Marge Bot pushed to branch master at Glasgow Haskell Compiler / GHC
Commits:
6d6a37a8 by Cheng Shao at 2023-03-30T18:42:56+00:00
ci: make lint-ci-config job fast again
We don't pin our nixpkgs revision and tracks the default
nixpkgs-unstable channel anyway. Instead of using
haskell.packages.ghc924, we should be using haskell.packages.ghc92 to
maximize the binary cache hit rate and make lint-ci-config job fast
again. Also bumps the nix docker image to the latest revision.
- - - - -
ef1548c4 by Cheng Shao at 2023-03-30T18:42:56+00:00
ci: ensure that all non-i386 pipelines do parallel xz compression
We can safely enable parallel xz compression for non-i386 pipelines.
However, previously we didn't export XZ_OPT, so the xz process won't
see it if XZ_OPT hasn't already been set in the current job.
- - - - -
20432d16 by Cheng Shao at 2023-03-30T18:42:56+00:00
ci: unset CROSS_EMULATOR for js job
- - - - -
4a24dbbe by Cheng Shao at 2023-03-30T18:42:56+00:00
ci: fix lint-testsuite job
The list_broken make target will transitively depend on the
calibrate.out target, which used STAGE1_GHC instead of TEST_HC. It
really should be TEST_HC since that's what get passed in the gitlab CI
config.
- - - - -
cea56ccc by Cheng Shao at 2023-03-30T18:42:56+00:00
ci: use alpine3_17-wasm image for wasm jobs
Bump the ci-images dependency and use the new alpine3_17-wasm docker
image for wasm jobs.
- - - - -
79d0cb32 by Ben Gamari at 2023-03-30T18:43:53+00:00
testsuite/driver: Add basic support for testing cross-compilers
- - - - -
e7392b4e by Ben Gamari at 2023-03-30T18:43:53+00:00
testsuite/driver: Normalize away differences in ghc executable name
- - - - -
ee160d06 by Ben Gamari at 2023-03-30T18:43:53+00:00
hadrian: Pass CROSS_EMULATOR to runtests.py
- - - - -
30c84511 by Cheng Shao at 2023-03-30T18:43:53+00:00
testsuite: don't add optllvm way for wasm32
- - - - -
f1beee36 by Cheng Shao at 2023-03-30T18:43:53+00:00
testsuite: normalize the .wasm extension
- - - - -
a984a103 by Cheng Shao at 2023-03-30T18:43:53+00:00
testsuite: strip the cross ghc prefix in output and error message
- - - - -
f7478d95 by Cheng Shao at 2023-03-30T18:43:53+00:00
testsuite: handle target executable extension
- - - - -
8fe8b653 by Cheng Shao at 2023-03-30T18:43:53+00:00
testsuite: mypy typing error fixes
This patch fixes some mypy typing errors which weren't caught in
previous linting jobs.
- - - - -
0149f32f by Cheng Shao at 2023-03-30T18:43:53+00:00
testsuite: use context variable instead of thread-local variable
This patch changes a thread-local variable to context variable
instead, which works as intended when the testsuite transitions to use
asyncio & coroutines instead of multi-threading to concurrently run
test cases. Note that this also raises the minimum Python version to
3.7.
- - - - -
ea853ff0 by Cheng Shao at 2023-03-30T18:43:53+00:00
testsuite: asyncify the testsuite driver
This patch refactors the testsuite driver, gets rid of multi-threading
logic for running test cases concurrently, and uses asyncio &
coroutines instead. This is not yak shaving for its own sake; the
previous multi-threading logic is prone to livelock/deadlock
conditions for some reason, even if the total number of threads is
bounded to a thread pool's capacity.
The asyncify change is an internal implementation detail of the
testsuite driver and does not impact most GHC maintainers out there.
The patch does not touch the .T files, test cases can be
added/modified the exact same way as before.
- - - - -
15 changed files:
- .gitlab-ci.yml
- .gitlab/ci.sh
- .gitlab/gen_ci.hs
- .gitlab/generate_job_metadata
- .gitlab/generate_jobs
- .gitlab/jobs.yaml
- hadrian/src/Settings/Builders/RunTest.hs
- testsuite/config/ghc
- testsuite/driver/my_typing.py
- testsuite/driver/runtests.py
- testsuite/driver/testglobals.py
- testsuite/driver/testlib.py
- testsuite/driver/testutil.py
- − testsuite/driver/typing_stubs.py
- testsuite/timeout/Makefile
Changes:
=====================================
.gitlab-ci.yml
=====================================
@@ -2,7 +2,7 @@ variables:
GIT_SSL_NO_VERIFY: "1"
# Commit of ghc/ci-images repository from which to pull Docker images
- DOCKER_REV: 572353e0644044fe3a5465bba4342a9a0b0eb60e
+ DOCKER_REV: b58ecd021c2533f0f0d0b1c9109200a69506d2b7
# Sequential version number of all cached things.
# Bump to invalidate GitLab CI cache.
@@ -264,7 +264,7 @@ lint-author:
- *drafts-can-fail-lint
lint-ci-config:
- image: nixos/nix:2.12.0
+ image: nixos/nix:2.14.1
extends: .lint
# We don't need history/submodules in this job
variables:
@@ -979,7 +979,7 @@ project-version:
.ghcup-metadata:
stage: deploy
- image: "nixos/nix:2.12.0"
+ image: nixos/nix:2.14.1
dependencies: null
tags:
- x86_64-linux
@@ -1084,4 +1084,3 @@ ghcup-metadata-testing-release:
rules:
- if: '$RELEASE_JOB == "yes"'
when: manual
-
=====================================
.gitlab/ci.sh
=====================================
@@ -514,7 +514,7 @@ function build_hadrian() {
# hadrian calls tar/xz to produce bindist, there's no other build
# work taking place.
if [[ "${CI_JOB_NAME:-}" != *"i386"* ]]; then
- XZ_OPT="${XZ_OPT:-} -T$cores"
+ export XZ_OPT="${XZ_OPT:-} -T$cores"
fi
if [[ -n "${REINSTALL_GHC:-}" ]]; then
@@ -606,6 +606,8 @@ function test_hadrian() {
return
# special case for JS backend
elif [ -n "${CROSS_TARGET:-}" ] && [ "${CROSS_EMULATOR:-}" == "js-emulator" ]; then
+ # The JS backend doesn't support CROSS_EMULATOR logic yet
+ unset CROSS_EMULATOR
# run "hadrian test" directly, not using the bindist, even though it did get installed.
# This is a temporary solution, See !9515 for the status of hadrian support.
run_hadrian \
=====================================
.gitlab/gen_ci.hs
=====================================
@@ -114,6 +114,7 @@ data LinuxDistro
| Ubuntu1804
| Centos7
| Alpine
+ | AlpineWasm
| Rocky8
deriving (Eq)
@@ -279,6 +280,7 @@ distroName Ubuntu1804 = "ubuntu18_04"
distroName Ubuntu2004 = "ubuntu20_04"
distroName Centos7 = "centos7"
distroName Alpine = "alpine3_12"
+distroName AlpineWasm = "alpine3_17-wasm"
distroName Rocky8 = "rocky8"
opsysName :: Opsys -> String
@@ -646,7 +648,6 @@ job arch opsys buildConfig = NamedJob { name = jobName, jobInfo = Job {..} }
, "bash .gitlab/ci.sh test_hadrian" ]
| otherwise
= [ "find libraries -name config.sub -exec cp config.sub {} \\;" | Darwin == opsys ] ++
- [ "sudo apk del --purge glibc*" | opsys == Linux Alpine, isNothing $ crossTarget buildConfig ] ++
[ "sudo chown ghc:ghc -R ." | Linux {} <- [opsys]] ++
[ ".gitlab/ci.sh setup"
, ".gitlab/ci.sh configure"
@@ -929,7 +930,7 @@ job_groups =
. setVariable "HADRIAN_ARGS" "--docs=none"
. delVariable "INSTALL_CONFIGURE_ARGS"
)
- $ validateBuilds Amd64 (Linux Alpine) cfg
+ $ validateBuilds Amd64 (Linux AlpineWasm) cfg
wasm_build_config =
(crossConfig "wasm32-wasi" NoEmulatorNeeded Nothing)
@@ -992,4 +993,3 @@ write_result as obj =
[] -> B.putStrLn
(fp:_) -> B.writeFile fp)
(A.encode obj)
-
=====================================
.gitlab/generate_job_metadata
=====================================
@@ -1,5 +1,5 @@
#! /usr/bin/env nix-shell
-#!nix-shell -i bash -p cabal-install "haskell.packages.ghc924.ghcWithPackages (pkgs: with pkgs; [aeson])" git jq
+#!nix-shell -i bash -p cabal-install "haskell.packages.ghc92.ghcWithPackages (pkgs: with pkgs; [aeson])" git jq
cd "$(dirname "${BASH_SOURCE[0]}")"
cabal run gen_ci -- metadata jobs-metadata.json
=====================================
.gitlab/generate_jobs
=====================================
@@ -1,5 +1,5 @@
#!/usr/bin/env nix-shell
-#!nix-shell -i bash -p cabal-install "haskell.packages.ghc924.ghcWithPackages (pkgs: with pkgs; [aeson])" git jq
+#!nix-shell -i bash -p cabal-install "haskell.packages.ghc92.ghcWithPackages (pkgs: with pkgs; [aeson])" git jq
# shellcheck shell=bash
=====================================
.gitlab/jobs.yaml
=====================================
@@ -544,17 +544,17 @@
"XZ_OPT": "-9"
}
},
- "nightly-x86_64-linux-alpine3_12-cross_wasm32-wasi-release+fully_static": {
+ "nightly-x86_64-linux-alpine3_12-int_native-validate+fully_static": {
"after_script": [
".gitlab/ci.sh save_cache",
".gitlab/ci.sh clean",
"cat ci_timings"
],
- "allow_failure": false,
+ "allow_failure": true,
"artifacts": {
"expire_in": "8 weeks",
"paths": [
- "ghc-x86_64-linux-alpine3_12-cross_wasm32-wasi-release+fully_static.tar.xz",
+ "ghc-x86_64-linux-alpine3_12-int_native-validate+fully_static.tar.xz",
"junit.xml"
],
"reports": {
@@ -595,17 +595,18 @@
"x86_64-linux"
],
"variables": {
- "BIGNUM_BACKEND": "gmp",
- "BIN_DIST_NAME": "ghc-x86_64-linux-alpine3_12-cross_wasm32-wasi-release+fully_static",
- "BUILD_FLAVOUR": "release+fully_static",
- "CONFIGURE_ARGS": "--disable-ld-override --with-intree-gmp --with-system-libffi",
- "CROSS_TARGET": "wasm32-wasi",
- "HADRIAN_ARGS": "--docs=none",
- "TEST_ENV": "x86_64-linux-alpine3_12-cross_wasm32-wasi-release+fully_static",
+ "BIGNUM_BACKEND": "native",
+ "BIN_DIST_NAME": "ghc-x86_64-linux-alpine3_12-int_native-validate+fully_static",
+ "BROKEN_TESTS": "encoding004 T10458 ghcilink002 linker_unload_native",
+ "BUILD_FLAVOUR": "validate+fully_static",
+ "CONFIGURE_ARGS": "--disable-ld-override ",
+ "HADRIAN_ARGS": "--docs=no-sphinx",
+ "INSTALL_CONFIGURE_ARGS": "--disable-ld-override",
+ "TEST_ENV": "x86_64-linux-alpine3_12-int_native-validate+fully_static",
"XZ_OPT": "-9"
}
},
- "nightly-x86_64-linux-alpine3_12-int_native-cross_wasm32-wasi-release+fully_static": {
+ "nightly-x86_64-linux-alpine3_12-validate+fully_static": {
"after_script": [
".gitlab/ci.sh save_cache",
".gitlab/ci.sh clean",
@@ -615,7 +616,7 @@
"artifacts": {
"expire_in": "8 weeks",
"paths": [
- "ghc-x86_64-linux-alpine3_12-int_native-cross_wasm32-wasi-release+fully_static.tar.xz",
+ "ghc-x86_64-linux-alpine3_12-validate+fully_static.tar.xz",
"junit.xml"
],
"reports": {
@@ -656,27 +657,28 @@
"x86_64-linux"
],
"variables": {
- "BIGNUM_BACKEND": "native",
- "BIN_DIST_NAME": "ghc-x86_64-linux-alpine3_12-int_native-cross_wasm32-wasi-release+fully_static",
- "BUILD_FLAVOUR": "release+fully_static",
- "CONFIGURE_ARGS": "--disable-ld-override --with-intree-gmp --with-system-libffi",
- "CROSS_TARGET": "wasm32-wasi",
- "HADRIAN_ARGS": "--docs=none",
- "TEST_ENV": "x86_64-linux-alpine3_12-int_native-cross_wasm32-wasi-release+fully_static",
+ "BIGNUM_BACKEND": "gmp",
+ "BIN_DIST_NAME": "ghc-x86_64-linux-alpine3_12-validate+fully_static",
+ "BROKEN_TESTS": "encoding004 T10458 ghcilink002 linker_unload_native",
+ "BUILD_FLAVOUR": "validate+fully_static",
+ "CONFIGURE_ARGS": "--disable-ld-override ",
+ "HADRIAN_ARGS": "--docs=no-sphinx",
+ "INSTALL_CONFIGURE_ARGS": "--disable-ld-override",
+ "TEST_ENV": "x86_64-linux-alpine3_12-validate+fully_static",
"XZ_OPT": "-9"
}
},
- "nightly-x86_64-linux-alpine3_12-int_native-validate+fully_static": {
+ "nightly-x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static": {
"after_script": [
".gitlab/ci.sh save_cache",
".gitlab/ci.sh clean",
"cat ci_timings"
],
- "allow_failure": true,
+ "allow_failure": false,
"artifacts": {
"expire_in": "8 weeks",
"paths": [
- "ghc-x86_64-linux-alpine3_12-int_native-validate+fully_static.tar.xz",
+ "ghc-x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static.tar.xz",
"junit.xml"
],
"reports": {
@@ -685,14 +687,14 @@
"when": "always"
},
"cache": {
- "key": "x86_64-linux-alpine3_12-$CACHE_REV",
+ "key": "x86_64-linux-alpine3_17-wasm-$CACHE_REV",
"paths": [
"cabal-cache",
"toolchain"
]
},
"dependencies": [],
- "image": "registry.gitlab.haskell.org/ghc/ci-images/x86_64-linux-alpine3_12:$DOCKER_REV",
+ "image": "registry.gitlab.haskell.org/ghc/ci-images/x86_64-linux-alpine3_17-wasm:$DOCKER_REV",
"needs": [
{
"artifacts": false,
@@ -706,7 +708,6 @@
}
],
"script": [
- "sudo apk del --purge glibc*",
"sudo chown ghc:ghc -R .",
".gitlab/ci.sh setup",
".gitlab/ci.sh configure",
@@ -718,18 +719,17 @@
"x86_64-linux"
],
"variables": {
- "BIGNUM_BACKEND": "native",
- "BIN_DIST_NAME": "ghc-x86_64-linux-alpine3_12-int_native-validate+fully_static",
- "BROKEN_TESTS": "encoding004 T10458 ghcilink002 linker_unload_native",
- "BUILD_FLAVOUR": "validate+fully_static",
- "CONFIGURE_ARGS": "--disable-ld-override ",
- "HADRIAN_ARGS": "--docs=no-sphinx",
- "INSTALL_CONFIGURE_ARGS": "--disable-ld-override",
- "TEST_ENV": "x86_64-linux-alpine3_12-int_native-validate+fully_static",
+ "BIGNUM_BACKEND": "gmp",
+ "BIN_DIST_NAME": "ghc-x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static",
+ "BUILD_FLAVOUR": "release+fully_static",
+ "CONFIGURE_ARGS": "--with-intree-gmp --with-system-libffi",
+ "CROSS_TARGET": "wasm32-wasi",
+ "HADRIAN_ARGS": "--docs=none",
+ "TEST_ENV": "x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static",
"XZ_OPT": "-9"
}
},
- "nightly-x86_64-linux-alpine3_12-unreg-cross_wasm32-wasi-release+fully_static": {
+ "nightly-x86_64-linux-alpine3_17-wasm-int_native-cross_wasm32-wasi-release+fully_static": {
"after_script": [
".gitlab/ci.sh save_cache",
".gitlab/ci.sh clean",
@@ -739,7 +739,7 @@
"artifacts": {
"expire_in": "8 weeks",
"paths": [
- "ghc-x86_64-linux-alpine3_12-unreg-cross_wasm32-wasi-release+fully_static.tar.xz",
+ "ghc-x86_64-linux-alpine3_17-wasm-int_native-cross_wasm32-wasi-release+fully_static.tar.xz",
"junit.xml"
],
"reports": {
@@ -748,14 +748,14 @@
"when": "always"
},
"cache": {
- "key": "x86_64-linux-alpine3_12-$CACHE_REV",
+ "key": "x86_64-linux-alpine3_17-wasm-$CACHE_REV",
"paths": [
"cabal-cache",
"toolchain"
]
},
"dependencies": [],
- "image": "registry.gitlab.haskell.org/ghc/ci-images/x86_64-linux-alpine3_12:$DOCKER_REV",
+ "image": "registry.gitlab.haskell.org/ghc/ci-images/x86_64-linux-alpine3_17-wasm:$DOCKER_REV",
"needs": [
{
"artifacts": false,
@@ -780,17 +780,17 @@
"x86_64-linux"
],
"variables": {
- "BIGNUM_BACKEND": "gmp",
- "BIN_DIST_NAME": "ghc-x86_64-linux-alpine3_12-unreg-cross_wasm32-wasi-release+fully_static",
+ "BIGNUM_BACKEND": "native",
+ "BIN_DIST_NAME": "ghc-x86_64-linux-alpine3_17-wasm-int_native-cross_wasm32-wasi-release+fully_static",
"BUILD_FLAVOUR": "release+fully_static",
- "CONFIGURE_ARGS": "--disable-ld-override --enable-unregisterised --with-intree-gmp --with-system-libffi",
+ "CONFIGURE_ARGS": "--with-intree-gmp --with-system-libffi",
"CROSS_TARGET": "wasm32-wasi",
"HADRIAN_ARGS": "--docs=none",
- "TEST_ENV": "x86_64-linux-alpine3_12-unreg-cross_wasm32-wasi-release+fully_static",
+ "TEST_ENV": "x86_64-linux-alpine3_17-wasm-int_native-cross_wasm32-wasi-release+fully_static",
"XZ_OPT": "-9"
}
},
- "nightly-x86_64-linux-alpine3_12-validate+fully_static": {
+ "nightly-x86_64-linux-alpine3_17-wasm-unreg-cross_wasm32-wasi-release+fully_static": {
"after_script": [
".gitlab/ci.sh save_cache",
".gitlab/ci.sh clean",
@@ -800,7 +800,7 @@
"artifacts": {
"expire_in": "8 weeks",
"paths": [
- "ghc-x86_64-linux-alpine3_12-validate+fully_static.tar.xz",
+ "ghc-x86_64-linux-alpine3_17-wasm-unreg-cross_wasm32-wasi-release+fully_static.tar.xz",
"junit.xml"
],
"reports": {
@@ -809,14 +809,14 @@
"when": "always"
},
"cache": {
- "key": "x86_64-linux-alpine3_12-$CACHE_REV",
+ "key": "x86_64-linux-alpine3_17-wasm-$CACHE_REV",
"paths": [
"cabal-cache",
"toolchain"
]
},
"dependencies": [],
- "image": "registry.gitlab.haskell.org/ghc/ci-images/x86_64-linux-alpine3_12:$DOCKER_REV",
+ "image": "registry.gitlab.haskell.org/ghc/ci-images/x86_64-linux-alpine3_17-wasm:$DOCKER_REV",
"needs": [
{
"artifacts": false,
@@ -830,7 +830,6 @@
}
],
"script": [
- "sudo apk del --purge glibc*",
"sudo chown ghc:ghc -R .",
".gitlab/ci.sh setup",
".gitlab/ci.sh configure",
@@ -843,13 +842,12 @@
],
"variables": {
"BIGNUM_BACKEND": "gmp",
- "BIN_DIST_NAME": "ghc-x86_64-linux-alpine3_12-validate+fully_static",
- "BROKEN_TESTS": "encoding004 T10458 ghcilink002 linker_unload_native",
- "BUILD_FLAVOUR": "validate+fully_static",
- "CONFIGURE_ARGS": "--disable-ld-override ",
- "HADRIAN_ARGS": "--docs=no-sphinx",
- "INSTALL_CONFIGURE_ARGS": "--disable-ld-override",
- "TEST_ENV": "x86_64-linux-alpine3_12-validate+fully_static",
+ "BIN_DIST_NAME": "ghc-x86_64-linux-alpine3_17-wasm-unreg-cross_wasm32-wasi-release+fully_static",
+ "BUILD_FLAVOUR": "release+fully_static",
+ "CONFIGURE_ARGS": "--enable-unregisterised --with-intree-gmp --with-system-libffi",
+ "CROSS_TARGET": "wasm32-wasi",
+ "HADRIAN_ARGS": "--docs=none",
+ "TEST_ENV": "x86_64-linux-alpine3_17-wasm-unreg-cross_wasm32-wasi-release+fully_static",
"XZ_OPT": "-9"
}
},
@@ -2459,7 +2457,6 @@
}
],
"script": [
- "sudo apk del --purge glibc*",
"sudo chown ghc:ghc -R .",
".gitlab/ci.sh setup",
".gitlab/ci.sh configure",
@@ -2523,7 +2520,6 @@
}
],
"script": [
- "sudo apk del --purge glibc*",
"sudo chown ghc:ghc -R .",
".gitlab/ci.sh setup",
".gitlab/ci.sh configure",
@@ -3521,7 +3517,7 @@
"TEST_ENV": "x86_64-freebsd13-validate"
}
},
- "x86_64-linux-alpine3_12-cross_wasm32-wasi-release+fully_static": {
+ "x86_64-linux-alpine3_12-validate+fully_static": {
"after_script": [
".gitlab/ci.sh save_cache",
".gitlab/ci.sh clean",
@@ -3531,7 +3527,7 @@
"artifacts": {
"expire_in": "2 weeks",
"paths": [
- "ghc-x86_64-linux-alpine3_12-cross_wasm32-wasi-release+fully_static.tar.xz",
+ "ghc-x86_64-linux-alpine3_12-validate+fully_static.tar.xz",
"junit.xml"
],
"reports": {
@@ -3573,15 +3569,16 @@
],
"variables": {
"BIGNUM_BACKEND": "gmp",
- "BIN_DIST_NAME": "ghc-x86_64-linux-alpine3_12-cross_wasm32-wasi-release+fully_static",
- "BUILD_FLAVOUR": "release+fully_static",
- "CONFIGURE_ARGS": "--disable-ld-override --with-intree-gmp --with-system-libffi",
- "CROSS_TARGET": "wasm32-wasi",
- "HADRIAN_ARGS": "--docs=none",
- "TEST_ENV": "x86_64-linux-alpine3_12-cross_wasm32-wasi-release+fully_static"
+ "BIN_DIST_NAME": "ghc-x86_64-linux-alpine3_12-validate+fully_static",
+ "BROKEN_TESTS": "encoding004 T10458 ghcilink002 linker_unload_native",
+ "BUILD_FLAVOUR": "validate+fully_static",
+ "CONFIGURE_ARGS": "--disable-ld-override ",
+ "HADRIAN_ARGS": "--docs=no-sphinx",
+ "INSTALL_CONFIGURE_ARGS": "--disable-ld-override",
+ "TEST_ENV": "x86_64-linux-alpine3_12-validate+fully_static"
}
},
- "x86_64-linux-alpine3_12-validate+fully_static": {
+ "x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static": {
"after_script": [
".gitlab/ci.sh save_cache",
".gitlab/ci.sh clean",
@@ -3591,7 +3588,7 @@
"artifacts": {
"expire_in": "2 weeks",
"paths": [
- "ghc-x86_64-linux-alpine3_12-validate+fully_static.tar.xz",
+ "ghc-x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static.tar.xz",
"junit.xml"
],
"reports": {
@@ -3600,14 +3597,14 @@
"when": "always"
},
"cache": {
- "key": "x86_64-linux-alpine3_12-$CACHE_REV",
+ "key": "x86_64-linux-alpine3_17-wasm-$CACHE_REV",
"paths": [
"cabal-cache",
"toolchain"
]
},
"dependencies": [],
- "image": "registry.gitlab.haskell.org/ghc/ci-images/x86_64-linux-alpine3_12:$DOCKER_REV",
+ "image": "registry.gitlab.haskell.org/ghc/ci-images/x86_64-linux-alpine3_17-wasm:$DOCKER_REV",
"needs": [
{
"artifacts": false,
@@ -3621,7 +3618,6 @@
}
],
"script": [
- "sudo apk del --purge glibc*",
"sudo chown ghc:ghc -R .",
".gitlab/ci.sh setup",
".gitlab/ci.sh configure",
@@ -3634,13 +3630,12 @@
],
"variables": {
"BIGNUM_BACKEND": "gmp",
- "BIN_DIST_NAME": "ghc-x86_64-linux-alpine3_12-validate+fully_static",
- "BROKEN_TESTS": "encoding004 T10458 ghcilink002 linker_unload_native",
- "BUILD_FLAVOUR": "validate+fully_static",
- "CONFIGURE_ARGS": "--disable-ld-override ",
- "HADRIAN_ARGS": "--docs=no-sphinx",
- "INSTALL_CONFIGURE_ARGS": "--disable-ld-override",
- "TEST_ENV": "x86_64-linux-alpine3_12-validate+fully_static"
+ "BIN_DIST_NAME": "ghc-x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static",
+ "BUILD_FLAVOUR": "release+fully_static",
+ "CONFIGURE_ARGS": "--with-intree-gmp --with-system-libffi",
+ "CROSS_TARGET": "wasm32-wasi",
+ "HADRIAN_ARGS": "--docs=none",
+ "TEST_ENV": "x86_64-linux-alpine3_17-wasm-cross_wasm32-wasi-release+fully_static"
}
},
"x86_64-linux-deb10-int_native-validate": {
=====================================
hadrian/src/Settings/Builders/RunTest.hs
=====================================
@@ -213,6 +213,7 @@ runTestBuilderArgs = builder Testsuite ? do
(testEnv, testMetricsFile) <- expr . liftIO $
(,) <$> lookupEnv "TEST_ENV" <*> lookupEnv "METRICS_FILE"
perfBaseline <- expr . liftIO $ lookupEnv "PERF_BASELINE_COMMIT"
+ targetWrapper <- expr . liftIO $ lookupEnv "CROSS_EMULATOR"
threads <- shakeThreads <$> expr getShakeOptions
top <- expr $ topDirectory
@@ -282,6 +283,7 @@ runTestBuilderArgs = builder Testsuite ? do
, case perfBaseline of
Just commit | not (null commit) -> arg ("--perf-baseline=" ++ commit)
_ -> mempty
+ , emitWhenSet targetWrapper $ \cmd -> arg ("--target-wrapper=" ++ cmd)
, emitWhenSet testEnv $ \env -> arg ("--test-env=" ++ env)
, emitWhenSet testMetricsFile $ \file -> arg ("--metrics-file=" ++ file)
, getTestArgs -- User-provided arguments from command line.
=====================================
testsuite/config/ghc
=====================================
@@ -70,7 +70,7 @@ if windows:
config.other_ways += winio_ways
# LLVM
-if not config.unregisterised and not config.arch == "javascript" and config.have_llvm:
+if not config.unregisterised and not config.arch in {"wasm32", "javascript"} and config.have_llvm:
config.compile_ways.append('optllvm')
config.run_ways.append('optllvm')
=====================================
testsuite/driver/my_typing.py
=====================================
@@ -8,13 +8,8 @@ The testsuite driver can be typechecked using mypy [1].
[1] http://mypy-lang.org/
"""
-try:
- from typing import *
- import typing
-except:
- # The backwards compatibility stubs must live in another module lest
- # mypy complains.
- from typing_stubs import * # type: ignore
+from typing import *
+import typing
####################################################
@@ -23,14 +18,7 @@ except:
# N.B. mypy appears to typecheck as though the "then" clause of if structures
# is taken. We exploit this below.
-# TextIO is missing on some older Pythons.
-if 'TextIO' not in globals():
- try:
- from typing import TextIO
- except ImportError:
- TextIO = None # type: ignore
-else:
- TextIO = None # type: ignore
+from typing import TextIO
####################################################
=====================================
testsuite/driver/runtests.py
=====================================
@@ -26,10 +26,10 @@ from pathlib import Path
# So we import it here first, so that the testsuite doesn't appear to fail.
import subprocess
-from concurrent.futures import ThreadPoolExecutor
+import asyncio
from testutil import getStdout, str_warn, str_info, print_table, shorten_metric_name
-from testglobals import getConfig, ghc_env, getTestRun, TestConfig, \
+from testglobals import getConfig, ghc_env, TestConfig, t, \
TestOptions, brokens, PerfMetric
from my_typing import TestName
from perf_notes import MetricChange, GitRef, inside_git_repo, is_worktree_dirty, format_perf_stat, get_abbrev_hash_length, is_commit_hash
@@ -71,6 +71,7 @@ parser.add_argument("--config", action='append', help="config field")
parser.add_argument("--rootdir", action='append', help="root of tree containing tests (default: .)")
parser.add_argument("--metrics-file", help="file in which to save (append) the performance test metrics. If omitted, git notes will be used.")
parser.add_argument("--summary-file", help="file in which to save the (human-readable) summary")
+parser.add_argument("--target-wrapper", help="wrapper executable to use when executing binaries compiled for the target")
parser.add_argument("--no-print-summary", action="store_true", help="should we print the summary?")
parser.add_argument("--only", action="append", help="just this test (can be give multiple --only= flags)")
parser.add_argument("--way", action="append", help="just this way")
@@ -119,6 +120,7 @@ hasMetricsFile = config.metrics_file is not None
config.summary_file = args.summary_file
config.no_print_summary = args.no_print_summary
config.baseline_commit = args.perf_baseline
+config.target_wrapper = args.target_wrapper
if args.top:
config.top = args.top
@@ -307,7 +309,7 @@ if windows:
path = format_path(path)
ghc_env['PATH'] = os.pathsep.join([path, ghc_env.get("PATH", "")])
-testopts_local.x = TestOptions()
+testopts_ctx_var.set(TestOptions())
# if timeout == -1 then we try to calculate a sensible value
if config.timeout == -1:
@@ -337,8 +339,6 @@ t_files = list(findTFiles(config.rootdirs))
print('Found', len(t_files), '.T files...')
-t = getTestRun() # type: TestRun
-
# Avoid cmd.exe built-in 'date' command on Windows
t.start_time = datetime.datetime.now()
@@ -484,25 +484,28 @@ if config.list_broken:
else:
# Now run all the tests
try:
- with ThreadPoolExecutor(max_workers=config.threads) as executor:
+ async def run_parallelTests():
+ sem = asyncio.Semaphore(config.threads)
+ ts = []
+
for oneTest in parallelTests:
if stopping():
break
- oneTest(executor)
+ ts.append(oneTest(sem))
+
+ return await asyncio.gather(*ts)
- # wait for parallel tests to finish
- if not stopping():
- executor.shutdown(wait=True)
+ # wait for parallel tests to finish
+ asyncio.run(run_parallelTests())
# Run the following tests purely sequential
- with ThreadPoolExecutor(max_workers=1) as executor:
+ async def run_aloneTests():
for oneTest in aloneTests:
if stopping():
break
- oneTest(executor)
+ await oneTest(None)
- if not stopping():
- executor.shutdown(wait=True)
+ asyncio.run(run_aloneTests())
except KeyboardInterrupt:
pass
@@ -523,13 +526,13 @@ else:
groups[m.stat.metric].append(m)
- for metric_name, stats in groups.items():
+ for metric_name, stats in groups.items(): # type: ignore
heading = 'Metrics: %s' % metric_name
print()
print(heading)
print('-' * len(heading))
print()
- tabulate_metrics(stats)
+ tabulate_metrics(stats) # type: ignore
else:
print("\nNone collected.")
print("")
=====================================
testsuite/driver/testglobals.py
=====================================
@@ -178,6 +178,11 @@ class TestConfig:
# threads
self.threads = 1
+ # An optional executable used to wrap target code execution
+ # When set tests which aren't marked with TestConfig.cross_okay
+ # are skipped.
+ self.target_wrapper = None
+
# tests which should be considered to be broken during this testsuite
# run.
self.broken_tests = set() # type: Set[TestName]
@@ -304,9 +309,6 @@ class TestRun:
global t
t = TestRun()
-def getTestRun() -> TestRun:
- return t
-
# -----------------------------------------------------------------------------
# Information about the current test
@@ -448,6 +450,12 @@ class TestOptions:
# Should we copy the files of symlink the files for the test?
self.copy_files = False
+ # Should the test be run in a cross-compiled tree?
+ # None: infer from test function
+ # True: run when --target-wrapper is set
+ # False: do not run in cross-compiled trees
+ self.cross_okay = None # type: Optional[bool]
+
# The extra hadrian dependencies we need for this particular test
self.hadrian_deps = set(["test:ghc"]) # type: Set[str]
=====================================
testsuite/driver/testlib.py
=====================================
@@ -36,7 +36,8 @@ from my_typing import *
from threading import Timer
from collections import OrderedDict
-import threading
+import asyncio
+import contextvars
global wantToStop
wantToStop = False
@@ -80,15 +81,19 @@ def get_all_ways() -> Set[WayName]:
# Options valid for the current test only (these get reset to
# testdir_testopts after each test).
-global testopts_local
-testopts_local = threading.local()
+global testopts_ctx_var
+testopts_ctx_var = contextvars.ContextVar('testopts_ctx_var') # type: ignore
def getTestOpts() -> TestOptions:
- return testopts_local.x
+ return testopts_ctx_var.get()
def setLocalTestOpts(opts: TestOptions) -> None:
- global testopts_local
- testopts_local.x = opts
+ global testopts_ctx_var
+ testopts_ctx_var.set(opts)
+
+def isCross() -> bool:
+ """ Are we testing a cross-compiler? """
+ return config.target_wrapper is not None
def isCompilerStatsTest() -> bool:
opts = getTestOpts()
@@ -205,14 +210,8 @@ def have_library(lib: str) -> bool:
print(cmd_line)
- p = subprocess.Popen(cmd_line,
- stdout=subprocess.PIPE,
- stderr=subprocess.PIPE,
- env=ghc_env)
- # read from stdout and stderr to avoid blocking due to
- # buffers filling
- p.communicate()
- r = p.wait()
+ cp = subprocess.run(cmd_line, capture_output=True, env=ghc_env)
+ r = cp.returncode
got_it = r == 0
have_lib_cache[lib] = got_it
@@ -255,7 +254,7 @@ def req_dynamic_hs( name, opts ):
opts.expect = 'fail'
def req_interp( name, opts ):
- if not config.have_interp:
+ if not config.have_interp or isCross():
opts.expect = 'fail'
# JS backend doesn't provide an interpreter yet
js_skip(name, opts)
@@ -633,6 +632,11 @@ KNOWN_OPERATING_SYSTEMS = set([
'solaris2',
])
+def exe_extension() -> str:
+ if config.arch == 'wasm32':
+ return '.wasm'
+ return ''
+
def opsys( os: str ) -> bool:
assert os in KNOWN_OPERATING_SYSTEMS
return config.os == os
@@ -1021,8 +1025,11 @@ parallelTests = []
aloneTests = []
allTestNames = set([]) # type: Set[TestName]
-def runTest(executor, opts, name: TestName, func, args):
- return executor.submit(test_common_work, name, opts, func, args)
+async def runTest(sem, opts, name: TestName, func, args):
+ if sem is None:
+ return await test_common_work(name, opts, func, args)
+ async with sem:
+ return await test_common_work(name, opts, func, args)
# name :: String
# setup :: [TestOpt] -> IO ()
@@ -1060,7 +1067,7 @@ def test(name: TestName,
if name in config.broken_tests:
myTestOpts.expect = 'fail'
- thisTest = lambda executor: runTest(executor, myTestOpts, name, func, args)
+ thisTest = lambda sem: runTest(sem, myTestOpts, name, func, args)
if myTestOpts.alone:
aloneTests.append(thisTest)
else:
@@ -1080,7 +1087,7 @@ do_not_copy = ('.hi', '.o', '.dyn_hi'
, '.dyn_o', '.out'
,'.hi-boot', '.o-boot') # 12112
-def test_common_work(name: TestName, opts,
+async def test_common_work(name: TestName, opts,
func, args) -> None:
try:
t.total_tests += 1
@@ -1098,14 +1105,21 @@ def test_common_work(name: TestName, opts,
all_ways = [WayName('ghci')]
else:
all_ways = []
+ if isCross():
+ opts.cross_okay = False
elif func in [makefile_test, run_command]:
# makefile tests aren't necessarily runtime or compile-time
# specific. Assume we can run them in all ways. See #16042 for what
# happened previously.
all_ways = config.compile_ways + config.run_ways
+ if isCross():
+ opts.cross_okay = False
else:
all_ways = [WayName('normal')]
+ if isCross() and opts.cross_okay is False:
+ opts.skip = True
+
# A test itself can request extra ways by setting opts.extra_ways
all_ways = list(OrderedDict.fromkeys(all_ways + [way for way in opts.extra_ways if way not in all_ways]))
@@ -1176,7 +1190,7 @@ def test_common_work(name: TestName, opts,
if stopping():
break
try:
- do_test(name, way, func, args, files)
+ await do_test(name, way, func, args, files)
except KeyboardInterrupt:
stopNow()
except Exception as e:
@@ -1200,9 +1214,9 @@ def test_common_work(name: TestName, opts,
except Exception as e:
framework_fail(name, None, 'Unhandled exception: ' + str(e))
-def do_test(name: TestName,
+async def do_test(name: TestName,
way: WayName,
- func: Callable[..., PassFail],
+ func: Callable[..., Awaitable[PassFail]],
args,
files: Set[str]
) -> None:
@@ -1271,7 +1285,7 @@ def do_test(name: TestName,
if opts.pre_cmd:
stdout_path = in_testdir(name, 'pre_cmd_stdout')
stderr_path = in_testdir(name, 'pre_cmd_stderr')
- exit_code = runCmd('cd "{0}" && {1}'.format(opts.testdir, override_options(opts.pre_cmd)),
+ exit_code = await runCmd('cd "{0}" && {1}'.format(opts.testdir, override_options(opts.pre_cmd)),
stdout = stdout_path,
stderr = stderr_path,
print_output = config.verbose >= 3)
@@ -1287,7 +1301,7 @@ def do_test(name: TestName,
# Don't continue and try to run the test if the pre_cmd fails.
return
- result = func(*[name,way] + args)
+ result = await func(*[name,way] + args)
if opts.expect not in ['pass', 'fail', 'missing-lib']:
framework_fail(name, way, 'bad expected ' + opts.expect)
@@ -1366,20 +1380,20 @@ def framework_warn(name: TestName, way: WayName, reason: str) -> None:
# altogether by using the setup function ignore_stdout instead of
# run_command.
-def run_command( name, way, cmd ):
- return simple_run( name, '', override_options(cmd), '' )
+async def run_command( name, way, cmd ):
+ return await simple_run( name, '', override_options(cmd), '' )
-def makefile_test( name, way, target=None ):
+async def makefile_test( name, way, target=None ):
if target is None:
target = name
cmd = '$MAKE -s --no-print-directory {target}'.format(target=target)
- return run_command(name, way, cmd)
+ return await run_command(name, way, cmd)
# -----------------------------------------------------------------------------
# GHCi tests
-def ghci_script( name, way, script):
+async def ghci_script( name, way, script):
flags = ' '.join(get_compiler_flags())
way_flags = ' '.join(config.way_flags[way])
@@ -1390,54 +1404,54 @@ def ghci_script( name, way, script):
# NB: put way_flags before flags so that flags in all.T can override others
getTestOpts().stdin = script
- return simple_run( name, way, cmd, getTestOpts().extra_run_opts )
+ return await simple_run( name, way, cmd, getTestOpts().extra_run_opts )
# -----------------------------------------------------------------------------
# Compile-only tests
-def compile( name, way, extra_hc_opts ):
- return do_compile( name, way, False, None, [], [], extra_hc_opts )
+async def compile( name, way, extra_hc_opts ):
+ return await do_compile( name, way, False, None, [], [], extra_hc_opts )
-def compile_fail( name, way, extra_hc_opts ):
- return do_compile( name, way, True, None, [], [], extra_hc_opts )
+async def compile_fail( name, way, extra_hc_opts ):
+ return await do_compile( name, way, True, None, [], [], extra_hc_opts )
-def backpack_typecheck( name, way, extra_hc_opts ):
- return do_compile( name, way, False, None, [], [], "-fno-code -fwrite-interface " + extra_hc_opts, backpack=True )
+async def backpack_typecheck( name, way, extra_hc_opts ):
+ return await do_compile( name, way, False, None, [], [], "-fno-code -fwrite-interface " + extra_hc_opts, backpack=True )
-def backpack_typecheck_fail( name, way, extra_hc_opts ):
- return do_compile( name, way, True, None, [], [], "-fno-code -fwrite-interface " + extra_hc_opts, backpack=True )
+async def backpack_typecheck_fail( name, way, extra_hc_opts ):
+ return await do_compile( name, way, True, None, [], [], "-fno-code -fwrite-interface " + extra_hc_opts, backpack=True )
-def backpack_compile( name, way, extra_hc_opts ):
- return do_compile( name, way, False, None, [], [], extra_hc_opts, backpack=True )
+async def backpack_compile( name, way, extra_hc_opts ):
+ return await do_compile( name, way, False, None, [], [], extra_hc_opts, backpack=True )
-def backpack_compile_fail( name, way, extra_hc_opts ):
- return do_compile( name, way, True, None, [], [], extra_hc_opts, backpack=True )
+async def backpack_compile_fail( name, way, extra_hc_opts ):
+ return await do_compile( name, way, True, None, [], [], extra_hc_opts, backpack=True )
-def backpack_run( name, way, extra_hc_opts ):
- return compile_and_run__( name, way, None, [], extra_hc_opts, backpack=True )
+async def backpack_run( name, way, extra_hc_opts ):
+ return await compile_and_run__( name, way, None, [], extra_hc_opts, backpack=True )
-def multimod_compile( name, way, top_mod, extra_hc_opts ):
- return do_compile( name, way, False, top_mod, [], [], extra_hc_opts )
+async def multimod_compile( name, way, top_mod, extra_hc_opts ):
+ return await do_compile( name, way, False, top_mod, [], [], extra_hc_opts )
-def multimod_compile_fail( name, way, top_mod, extra_hc_opts ):
- return do_compile( name, way, True, top_mod, [], [], extra_hc_opts )
+async def multimod_compile_fail( name, way, top_mod, extra_hc_opts ):
+ return await do_compile( name, way, True, top_mod, [], [], extra_hc_opts )
-def multimod_compile_filter( name, way, top_mod, extra_hc_opts, filter_with, suppress_stdout=True ):
- return do_compile( name, way, False, top_mod, [], [], extra_hc_opts, filter_with=filter_with, suppress_stdout=suppress_stdout )
+async def multimod_compile_filter( name, way, top_mod, extra_hc_opts, filter_with, suppress_stdout=True ):
+ return await do_compile( name, way, False, top_mod, [], [], extra_hc_opts, filter_with=filter_with, suppress_stdout=suppress_stdout )
-def multiunit_compile( name, way, units, extra_hc_opts ):
- return do_compile( name, way, False, None, [], units, extra_hc_opts )
+async def multiunit_compile( name, way, units, extra_hc_opts ):
+ return await do_compile( name, way, False, None, [], units, extra_hc_opts )
-def multiunit_compile_fail( name, way, units, extra_hc_opts ):
- return do_compile( name, way, True, None, [], units, extra_hc_opts )
+async def multiunit_compile_fail( name, way, units, extra_hc_opts ):
+ return await do_compile( name, way, True, None, [], units, extra_hc_opts )
-def multi_compile( name, way, top_mod, extra_mods, extra_hc_opts ):
- return do_compile( name, way, False, top_mod, extra_mods, [], extra_hc_opts)
+async def multi_compile( name, way, top_mod, extra_mods, extra_hc_opts ):
+ return await do_compile( name, way, False, top_mod, extra_mods, [], extra_hc_opts)
-def multi_compile_fail( name, way, top_mod, extra_mods, extra_hc_opts ):
- return do_compile( name, way, True, top_mod, extra_mods, [], extra_hc_opts)
+async def multi_compile_fail( name, way, top_mod, extra_mods, extra_hc_opts ):
+ return await do_compile( name, way, True, top_mod, extra_mods, [], extra_hc_opts)
-def do_compile(name: TestName,
+async def do_compile(name: TestName,
way: WayName,
should_fail: bool,
top_mod: Optional[Path],
@@ -1448,12 +1462,12 @@ def do_compile(name: TestName,
) -> PassFail:
# print 'Compile only, extra args = ', extra_hc_opts
- result = extras_build( way, extra_mods, extra_hc_opts )
+ result = await extras_build( way, extra_mods, extra_hc_opts )
if badResult(result):
return result
extra_hc_opts = result.hc_opts
- result = simple_build(name, way, extra_hc_opts, should_fail, top_mod, units, False, True, **kwargs)
+ result = await simple_build(name, way, extra_hc_opts, should_fail, top_mod, units, False, True, **kwargs)
if badResult(result):
return result
@@ -1466,7 +1480,7 @@ def do_compile(name: TestName,
actual_stderr_file = add_suffix(name, 'comp.stderr')
diff_file_name = in_testdir(add_suffix(name, 'comp.diff'))
- if not compare_outputs(way, 'stderr',
+ if not await compare_outputs(way, 'stderr',
join_normalisers(getTestOpts().extra_errmsg_normaliser,
normalise_errmsg),
expected_stderr_file, actual_stderr_file,
@@ -1482,14 +1496,14 @@ def do_compile(name: TestName,
# no problems found, this test passed
return passed()
-def compile_cmp_asm(name: TestName,
+async def compile_cmp_asm(name: TestName,
way: WayName,
ext: str,
extra_hc_opts: str
) -> PassFail:
if extra_hc_opts:
print('Compile only, extra args = ', extra_hc_opts)
- result = simple_build(name + '.' + ext, way, '-keep-s-files -O ' + extra_hc_opts, False, None, [], False, False)
+ result = await simple_build(name + '.' + ext, way, '-keep-s-files -O ' + extra_hc_opts, False, None, [], False, False)
if badResult(result):
return result
@@ -1501,7 +1515,7 @@ def compile_cmp_asm(name: TestName,
expected_asm_file = find_expected_file(name, 'asm')
actual_asm_file = add_suffix(name, 's')
- if not compare_outputs(way, 'asm',
+ if not await compare_outputs(way, 'asm',
join_normalisers(normalise_errmsg, normalise_asm),
expected_asm_file, actual_asm_file):
return failBecause('asm mismatch')
@@ -1509,7 +1523,7 @@ def compile_cmp_asm(name: TestName,
# no problems found, this test passed
return passed()
-def compile_grep_asm(name: TestName,
+async def compile_grep_asm(name: TestName,
way: WayName,
ext: str,
is_substring: bool,
@@ -1517,7 +1531,7 @@ def compile_grep_asm(name: TestName,
) -> PassFail:
if extra_hc_opts:
print('Compile and grep asm, extra args = ', extra_hc_opts)
- result = simple_build(name + '.' + ext, way, '-keep-s-files -O ' + extra_hc_opts, False, None, [], False, False)
+ result = await simple_build(name + '.' + ext, way, '-keep-s-files -O ' + extra_hc_opts, False, None, [], False, False)
if badResult(result):
return result
@@ -1533,13 +1547,13 @@ def compile_grep_asm(name: TestName,
# no problems found, this test passed
return passed()
-def compile_grep_core(name: TestName,
+async def compile_grep_core(name: TestName,
way: WayName,
extra_hc_opts: str
) -> PassFail:
if extra_hc_opts:
print('Compile only, extra args = ', extra_hc_opts)
- result = simple_build(name + '.hs', way, '-ddump-to-file -dsuppress-all -ddump-simpl -O ' + extra_hc_opts, False, None, [], False, False)
+ result = await simple_build(name + '.hs', way, '-ddump-to-file -dsuppress-all -ddump-simpl -O ' + extra_hc_opts, False, None, [], False, False)
if badResult(result):
return result
@@ -1557,7 +1571,7 @@ def compile_grep_core(name: TestName,
# -----------------------------------------------------------------------------
# Compile-and-run tests
-def compile_and_run__(name: TestName,
+async def compile_and_run__(name: TestName,
way: WayName,
top_mod: Path,
extra_mods: List[str],
@@ -1566,38 +1580,38 @@ def compile_and_run__(name: TestName,
) -> PassFail:
# print 'Compile and run, extra args = ', extra_hc_opts
- result = extras_build( way, extra_mods, extra_hc_opts )
+ result = await extras_build( way, extra_mods, extra_hc_opts )
if badResult(result):
return result
extra_hc_opts = result.hc_opts
assert extra_hc_opts is not None
if way.startswith('ghci'): # interpreted...
- return interpreter_run(name, way, extra_hc_opts, top_mod)
+ return await interpreter_run(name, way, extra_hc_opts, top_mod)
else: # compiled...
- result = simple_build(name, way, extra_hc_opts, False, top_mod, [], True, True, backpack = backpack)
+ result = await simple_build(name, way, extra_hc_opts, False, top_mod, [], True, True, backpack = backpack)
if badResult(result):
return result
- cmd = './' + name;
+ cmd = './' + name + exe_extension()
# we don't check the compiler's stderr for a compile-and-run test
- return simple_run( name, way, cmd, getTestOpts().extra_run_opts )
+ return await simple_run( name, way, cmd, getTestOpts().extra_run_opts )
-def compile_and_run( name, way, extra_hc_opts ):
- return compile_and_run__( name, way, None, [], extra_hc_opts)
+async def compile_and_run( name, way, extra_hc_opts ):
+ return await compile_and_run__( name, way, None, [], extra_hc_opts)
-def multimod_compile_and_run( name, way, top_mod, extra_hc_opts ):
- return compile_and_run__( name, way, top_mod, [], extra_hc_opts)
+async def multimod_compile_and_run( name, way, top_mod, extra_hc_opts ):
+ return await compile_and_run__( name, way, top_mod, [], extra_hc_opts)
-def multi_compile_and_run( name, way, top_mod, extra_mods, extra_hc_opts ):
- return compile_and_run__( name, way, top_mod, extra_mods, extra_hc_opts)
+async def multi_compile_and_run( name, way, top_mod, extra_mods, extra_hc_opts ):
+ return await compile_and_run__( name, way, top_mod, extra_mods, extra_hc_opts)
def stats( name, way, stats_file ):
opts = getTestOpts()
return check_stats(name, way, in_testdir(stats_file), opts.stats_range_fields)
-def static_stats( name, way, stats_file ):
+async def static_stats( name, way, stats_file ):
opts = getTestOpts()
return check_stats(name, way, in_statsdir(stats_file), opts.stats_range_fields)
@@ -1683,9 +1697,9 @@ def check_stats(name: TestName,
# -----------------------------------------------------------------------------
# Build a single-module program
-def extras_build( way, extra_mods, extra_hc_opts ):
+async def extras_build( way, extra_mods, extra_hc_opts ):
for mod, opts in extra_mods:
- result = simple_build(mod, way, opts + ' ' + extra_hc_opts, False, None, [], False, False)
+ result = await simple_build(mod, way, opts + ' ' + extra_hc_opts, False, None, [], False, False)
if not (mod.endswith('.hs') or mod.endswith('.lhs')):
extra_hc_opts += ' %s' % Path(mod).with_suffix('.o')
if badResult(result):
@@ -1693,7 +1707,7 @@ def extras_build( way, extra_mods, extra_hc_opts ):
return passed(hc_opts=extra_hc_opts)
-def simple_build(name: Union[TestName, str],
+async def simple_build(name: Union[TestName, str],
way: WayName,
extra_hc_opts: str,
should_fail: bool,
@@ -1765,7 +1779,7 @@ def simple_build(name: Union[TestName, str],
if filter_with != '':
cmd = cmd + ' | ' + filter_with
- exit_code = runCmd(cmd, None, stdout, stderr, opts.compile_timeout_multiplier)
+ exit_code = await runCmd(cmd, None, stdout, stderr, opts.compile_timeout_multiplier)
actual_stderr_path = in_testdir(name, 'comp.stderr')
@@ -1799,7 +1813,7 @@ def simple_build(name: Union[TestName, str],
# from /dev/null. Route output to testname.run.stdout and
# testname.run.stderr. Returns the exit code of the run.
-def simple_run(name: TestName, way: WayName, prog: str, extra_run_opts: str) -> Any:
+async def simple_run(name: TestName, way: WayName, prog: str, extra_run_opts: str) -> Any:
opts = getTestOpts()
# figure out what to use for stdin
@@ -1831,7 +1845,10 @@ def simple_run(name: TestName, way: WayName, prog: str, extra_run_opts: str) ->
stats_args = ''
# Put extra_run_opts last: extra_run_opts('+RTS foo') should work.
- cmd = ' '.join([prog, stats_args, my_rts_flags, extra_run_opts])
+ args = [prog, stats_args, my_rts_flags, extra_run_opts]
+ if config.target_wrapper is not None:
+ args = [config.target_wrapper] + args
+ cmd = ' '.join(args)
if opts.cmd_wrapper is not None:
cmd = opts.cmd_wrapper(cmd)
@@ -1839,7 +1856,7 @@ def simple_run(name: TestName, way: WayName, prog: str, extra_run_opts: str) ->
cmd = 'cd "{opts.testdir}" && {cmd}'.format(**locals())
# run the command
- exit_code = runCmd(cmd, stdin_arg, stdout_arg, stderr_arg, opts.run_timeout_multiplier)
+ exit_code = await runCmd(cmd, stdin_arg, stdout_arg, stderr_arg, opts.run_timeout_multiplier)
# check the exit code
if exit_code != opts.exit_code:
@@ -1850,11 +1867,11 @@ def simple_run(name: TestName, way: WayName, prog: str, extra_run_opts: str) ->
message = format_bad_exit_code_message(exit_code)
return failBecause(message)
- if not (opts.ignore_stderr or stderr_ok(name, way) or opts.combined_output):
+ if not (opts.ignore_stderr or await stderr_ok(name, way) or opts.combined_output):
return failBecause('bad stderr',
stderr=read_stderr(name),
stdout=read_stdout(name))
- if not (opts.ignore_stdout or stdout_ok(name, way)):
+ if not (opts.ignore_stdout or await stdout_ok(name, way)):
return failBecause('bad stdout',
stderr=read_stderr(name),
stdout=read_stdout(name))
@@ -1863,9 +1880,9 @@ def simple_run(name: TestName, way: WayName, prog: str, extra_run_opts: str) ->
check_prof = '-p' in my_rts_flags
# exit_code > 127 probably indicates a crash, so don't try to run hp2ps.
- if check_hp and (exit_code <= 127 or exit_code == 251) and not check_hp_ok(name):
+ if check_hp and (exit_code <= 127 or exit_code == 251) and not await check_hp_ok(name):
return failBecause('bad heap profile')
- if check_prof and not check_prof_ok(name, way):
+ if check_prof and not await check_prof_ok(name, way):
return failBecause('bad profile')
# Check runtime stats if desired.
@@ -1881,7 +1898,7 @@ def rts_flags(way: WayName) -> str:
# -----------------------------------------------------------------------------
# Run a program in the interpreter and check its output
-def interpreter_run(name: TestName,
+async def interpreter_run(name: TestName,
way: WayName,
extra_hc_opts: str,
top_mod: Path
@@ -1932,7 +1949,7 @@ def interpreter_run(name: TestName,
cmd = 'cd "{opts.testdir}" && {cmd}'.format(**locals())
- exit_code = runCmd(cmd, script, stdout, stderr, opts.run_timeout_multiplier)
+ exit_code = await runCmd(cmd, script, stdout, stderr, opts.run_timeout_multiplier)
# split the stdout into compilation/program output
split_file(stdout, delimiter,
@@ -1955,11 +1972,11 @@ def interpreter_run(name: TestName,
# ToDo: if the sub-shell was killed by ^C, then exit
- if not (opts.ignore_stderr or stderr_ok(name, way)):
+ if not (opts.ignore_stderr or await stderr_ok(name, way)):
return failBecause('bad stderr',
stderr=read_stderr(name),
stdout=read_stdout(name))
- elif not (opts.ignore_stdout or stdout_ok(name, way)):
+ elif not (opts.ignore_stdout or await stdout_ok(name, way)):
return failBecause('bad stdout',
stderr=read_stderr(name),
stdout=read_stdout(name))
@@ -1995,7 +2012,7 @@ def get_compiler_flags() -> List[str]:
return flags
-def stdout_ok(name: TestName, way: WayName) -> bool:
+async def stdout_ok(name: TestName, way: WayName) -> bool:
actual_stdout_file = add_suffix(name, 'run.stdout')
expected_stdout_file = find_expected_file(name, 'stdout')
@@ -2006,7 +2023,7 @@ def stdout_ok(name: TestName, way: WayName) -> bool:
actual_stdout_path = in_testdir(actual_stdout_file)
return check_stdout(actual_stdout_path, extra_norm)
- return compare_outputs(way, 'stdout', extra_norm,
+ return await compare_outputs(way, 'stdout', extra_norm,
expected_stdout_file, actual_stdout_file)
def read_stdout( name: TestName ) -> str:
@@ -2022,11 +2039,11 @@ def dump_stdout( name: TestName ) -> None:
print("Stdout (", name, "):")
safe_print(s)
-def stderr_ok(name: TestName, way: WayName) -> bool:
+async def stderr_ok(name: TestName, way: WayName) -> bool:
actual_stderr_file = add_suffix(name, 'run.stderr')
expected_stderr_file = find_expected_file(name, 'stderr')
- return compare_outputs(way, 'stderr',
+ return await compare_outputs(way, 'stderr',
join_normalisers(normalise_errmsg, getTestOpts().extra_errmsg_normaliser), \
expected_stderr_file, actual_stderr_file,
whitespace_normaliser=normalise_whitespace)
@@ -2086,20 +2103,20 @@ def write_file(f: Path, s: str) -> None:
# Another solution would be to open files in binary mode always, and
# operate on bytes.
-def check_hp_ok(name: TestName) -> bool:
+async def check_hp_ok(name: TestName) -> bool:
opts = getTestOpts()
# do not qualify for hp2ps because we should be in the right directory
hp2psCmd = 'cd "{opts.testdir}" && {{hp2ps}} {name}'.format(**locals())
- hp2psResult = runCmd(hp2psCmd, print_output=True)
+ hp2psResult = await runCmd(hp2psCmd, print_output=True)
actual_ps_path = in_testdir(name, 'ps')
if hp2psResult == 0:
if actual_ps_path.exists():
- if does_ghostscript_work():
- gsResult = runCmd(genGSCmd(actual_ps_path))
+ if await does_ghostscript_work():
+ gsResult = await runCmd(genGSCmd(actual_ps_path))
if (gsResult == 0):
return True
else:
@@ -2114,7 +2131,7 @@ def check_hp_ok(name: TestName) -> bool:
print("hp2ps error when processing heap profile for " + name)
return False
-def check_prof_ok(name: TestName, way: WayName) -> bool:
+async def check_prof_ok(name: TestName, way: WayName) -> bool:
expected_prof_file = find_expected_file(name, 'prof.sample')
expected_prof_path = in_testdir(expected_prof_file)
@@ -2123,7 +2140,7 @@ def check_prof_ok(name: TestName, way: WayName) -> bool:
if not expected_prof_path.exists():
return True
- actual_prof_file = add_suffix(name, 'prof')
+ actual_prof_file = add_suffix(name + exe_extension(), 'prof')
actual_prof_path = in_testdir(actual_prof_file)
if not actual_prof_path.exists():
@@ -2134,7 +2151,7 @@ def check_prof_ok(name: TestName, way: WayName) -> bool:
print("%s is empty" % actual_prof_path)
return(False)
- return compare_outputs(way, 'prof', normalise_prof,
+ return await compare_outputs(way, 'prof', normalise_prof,
expected_prof_file, actual_prof_file,
whitespace_normaliser=normalise_whitespace)
@@ -2142,7 +2159,7 @@ def check_prof_ok(name: TestName, way: WayName) -> bool:
# new output. Returns true if output matched or was accepted, false
# otherwise. See Note [Output comparison] for the meaning of the
# normaliser and whitespace_normaliser parameters.
-def compare_outputs(way: WayName,
+async def compare_outputs(way: WayName,
kind: str,
normaliser: OutputNormalizer,
expected_file, actual_file, diff_file=None,
@@ -2180,7 +2197,7 @@ def compare_outputs(way: WayName,
if config.verbose >= 1 and _expect_pass(way):
# See Note [Output comparison].
- r = runCmd('diff -uw "{0}" "{1}"'.format(null2unix_null(expected_normalised_path),
+ r = await runCmd('diff -uw "{0}" "{1}"'.format(null2unix_null(expected_normalised_path),
actual_normalised_path),
stdout=diff_file,
print_output=True)
@@ -2188,7 +2205,7 @@ def compare_outputs(way: WayName,
# If for some reason there were no non-whitespace differences,
# then do a full diff
if r == 0:
- r = runCmd('diff -u "{0}" "{1}"'.format(null2unix_null(expected_normalised_path),
+ r = await runCmd('diff -u "{0}" "{1}"'.format(null2unix_null(expected_normalised_path),
actual_normalised_path),
stdout=diff_file,
print_output=True)
@@ -2333,11 +2350,25 @@ def normalise_errmsg(s: str) -> str:
s = normalise_callstacks(s)
s = normalise_type_reps(s)
+ # normalise slashes, minimise Windows/Unix filename differences
+ s = re.sub('\\\\', '/', s)
+
+ # Normalize the name of the GHC executable. Specifically,
+ # this catches the cases that:
+ #
+ # * In cross-compilers ghc's executable name may include
+ # a target prefix (e.g. `aarch64-linux-gnu-ghc`)
+ # * On Windows the executable name may mention the
+ # versioned name (e.g. `ghc-9.2.1`)
+ s = re.sub(Path(config.compiler).name + ':', 'ghc:', s)
+
# If somefile ends in ".exe" or ".exe:", zap ".exe" (for Windows)
# the colon is there because it appears in error messages; this
# hacky solution is used in place of more sophisticated filename
# mangling
s = re.sub('([^\\s])\\.exe', '\\1', s)
+ # Same thing for .wasm modules generated by the Wasm backend
+ s = re.sub('([^\\s])\\.wasm', '\\1', s)
# Same thing for .jsexe directories generated by the JS backend
s = re.sub('([^\\s])\\.jsexe', '\\1', s)
@@ -2401,6 +2432,9 @@ def normalise_errmsg(s: str) -> str:
# clang may warn about unused argument when used as assembler
s = re.sub('.*warning: argument unused during compilation:.*\n', '', s)
+ # strip the cross prefix if any
+ s = re.sub('^([^:]+-)?ghc:', 'ghc:', s)
+
return s
# normalise a .prof file, so that we can reasonably compare it against
@@ -2459,6 +2493,7 @@ def normalise_slashes_( s: str ) -> str:
def normalise_exe_( s: str ) -> str:
s = re.sub(r'\.exe', '', s)
+ s = re.sub(r'\.wasm', '', s)
s = re.sub(r'\.jsexe', '', s)
return s
@@ -2468,9 +2503,11 @@ def normalise_output( s: str ) -> str:
s = modify_lines(s, lambda l: re.sub(' error:', '', l))
s = modify_lines(s, lambda l: re.sub(' Warning:', ' warning:', l))
# Remove a .exe extension (for Windows)
+ # and .wasm extension (for the Wasm backend)
# and .jsexe extension (for the JS backend)
# This can occur in error messages generated by the program.
s = re.sub('([^\\s])\\.exe', '\\1', s)
+ s = re.sub('([^\\s])\\.wasm', '\\1', s)
s = re.sub('([^\\s])\\.jsexe', '\\1', s)
s = normalise_callstacks(s)
s = normalise_type_reps(s)
@@ -2490,6 +2527,9 @@ def normalise_output( s: str ) -> str:
# clang may warn about unused argument when used as assembler
s = re.sub('.*warning: argument unused during compilation:.*\n', '', s)
+ # strip the cross prefix if any
+ s = re.sub('^([^:]+-)?ghc:', 'ghc:', s)
+
return s
def normalise_asm( s: str ) -> str:
@@ -2527,7 +2567,7 @@ def dump_file(f: Path):
except Exception:
print('')
-def runCmd(cmd: str,
+async def runCmd(cmd: str,
stdin: Union[None, Path]=None,
stdout: Union[None, Path]=None,
stderr: Union[None, int, Path]=None,
@@ -2544,9 +2584,9 @@ def runCmd(cmd: str,
stdout_buffer = b''
stderr_buffer = b''
- hStdErr = subprocess.PIPE
- if stderr is subprocess.STDOUT:
- hStdErr = subprocess.STDOUT
+ hStdErr = asyncio.subprocess.PIPE
+ if stderr is asyncio.subprocess.STDOUT:
+ hStdErr = asyncio.subprocess.STDOUT
try:
# cmd is a complex command in Bourne-shell syntax
@@ -2554,13 +2594,9 @@ def runCmd(cmd: str,
# Hence it must ultimately be run by a Bourne shell. It's timeout's job
# to invoke the Bourne shell
- r = subprocess.Popen([timeout_prog, timeout, cmd],
- stdin=stdin_file,
- stdout=subprocess.PIPE,
- stderr=hStdErr,
- env=ghc_env)
+ proc = await asyncio.create_subprocess_exec(timeout_prog, timeout, cmd, stdin=stdin_file, stdout=asyncio.subprocess.PIPE, stderr=hStdErr, env=ghc_env)
- stdout_buffer, stderr_buffer = r.communicate()
+ stdout_buffer, stderr_buffer = await proc.communicate()
finally:
if stdin_file:
stdin_file.close()
@@ -2580,13 +2616,13 @@ def runCmd(cmd: str,
if isinstance(stderr, Path):
stderr.write_bytes(stderr_buffer)
- if r.returncode == 98:
+ if proc.returncode == 98:
# The python timeout program uses 98 to signal that ^C was pressed
stopNow()
- if r.returncode == 99 and getTestOpts().exit_code != 99:
+ if proc.returncode == 99 and getTestOpts().exit_code != 99:
# Only print a message when timeout killed the process unexpectedly.
if_verbose(1, 'Timeout happened...killed process "{0}"...\n'.format(cmd))
- return r.returncode
+ return proc.returncode # type: ignore
# Each message should be kept lowercase
def exit_code_specific_message(exit_code: int) -> str:
@@ -2605,34 +2641,45 @@ def format_bad_exit_code_message(exit_code: int) -> str:
def genGSCmd(psfile: Path) -> str:
return '{{gs}} -dNODISPLAY -dBATCH -dQUIET -dNOPAUSE "{0}"'.format(psfile)
- at memoize
-def does_ghostscript_work() -> bool:
+does_ghostscript_work_cache = None
+
+async def does_ghostscript_work() -> bool:
"""
Detect whether Ghostscript is functional.
"""
+ global does_ghostscript_work_cache
+ if does_ghostscript_work_cache is not None:
+ return does_ghostscript_work_cache
+
def gsNotWorking(reason: str) -> None:
print("GhostScript not available for hp2ps tests:", reason)
if config.gs is None:
+ does_ghostscript_work_cache = False
return False
try:
- if runCmd(genGSCmd(config.top / 'config' / 'good.ps')) != 0:
+ if await runCmd(genGSCmd(config.top / 'config' / 'good.ps')) != 0:
gsNotWorking("gs can't process good input")
+ does_ghostscript_work_cache = False
return False
except Exception as e:
gsNotWorking('error invoking gs on bad input: %s' % e)
+ does_ghostscript_work_cache = False
return False
try:
cmd = genGSCmd(config.top / 'config' / 'bad.ps') + ' >/dev/null 2>&1'
- if runCmd(cmd) == 0:
+ if await runCmd(cmd) == 0:
gsNotWorking('gs accepts bad input')
+ does_ghostscript_work_cache = False
return False
except Exception as e:
gsNotWorking('error invoking gs on bad input: %s' % e)
+ does_ghostscript_work_cache = False
return False
+ does_ghostscript_work_cache = True
return True
def add_suffix( name: Union[str, Path], suffix: str ) -> Path:
=====================================
testsuite/driver/testutil.py
=====================================
@@ -29,9 +29,9 @@ def passed(hc_opts=None) -> PassFail:
hc_opts=hc_opts)
def failBecause(reason: str,
- tag: str=None,
- stderr: str=None,
- stdout: str=None
+ tag: Optional[str]=None,
+ stderr: Optional[str]=None,
+ stdout: Optional[str]=None
) -> PassFail:
return PassFail(passed=False, reason=reason, tag=tag,
stderr=stderr, stdout=stdout, hc_opts=None)
@@ -49,15 +49,14 @@ def str_info(s: str) -> str:
def getStdout(cmd_and_args: List[str]):
# Can't use subprocess.check_output, since we also verify that
# no stderr was produced
- p = subprocess.Popen([strip_quotes(cmd_and_args[0])] + cmd_and_args[1:],
- stdout=subprocess.PIPE,
- stderr=subprocess.PIPE)
- (stdout, stderr) = p.communicate()
- r = p.wait()
+ cp = subprocess.run([strip_quotes(cmd_and_args[0])] + cmd_and_args[1:], capture_output=True)
+ r = cp.returncode
+ stdout = cp.stdout
+ stderr = cp.stderr
if r != 0:
raise Exception("Command failed: " + str(cmd_and_args))
if stderr:
- raise Exception("stderr from command: %s\nStdOut(%s):\n%s\n%s\nOutput(%s):\n%s\n%s\n" % (cmd_and_args,str(len(stdout)), stdout, stdout.decode('utf-8'), str(len(stderr)), stderr, stderr.decode('utf-8')))
+ raise Exception("stderr from command: %s\nStdOut(%s):\n%r\n%s\nOutput(%s):\n%r\n%s\n" % (cmd_and_args,str(len(stdout)), stdout, stdout.decode('utf-8'), str(len(stderr)), stderr, stderr.decode('utf-8')))
return stdout.decode('utf-8')
def lndir(srcdir: Path, dstdir: Path, force_copy=False):
=====================================
testsuite/driver/typing_stubs.py deleted
=====================================
@@ -1,23 +0,0 @@
-# Stub definitions for things provided by the `typing` package for use by older
-# Python versions which don't ship with `typing`.
-
-import collections
-
-class Dummy:
- def __getitem__(self, *args):
- return None
-
-List = Dummy()
-Tuple = Dummy()
-Set = Dummy()
-TextIO = Dummy()
-Iterator = Dummy()
-Callable = Dummy()
-Optional = Dummy()
-Dict = Dummy()
-Union = Dummy()
-Any = Dummy()
-
-NewType = lambda name, ty: ty
-def NamedTuple(name, fields):
- return collections.namedtuple(name, [field[0] for field in fields])
=====================================
testsuite/timeout/Makefile
=====================================
@@ -51,11 +51,7 @@ boot all :: calibrate.out $(TIMEOUT_PROGRAM)
calibrate.out:
$(RM) -f TimeMe.o TimeMe.hi TimeMe TimeMe.exe
- $(PYTHON) calibrate '$(STAGE1_GHC)' > $@
-# We use stage 1 to do the calibration, as stage 2 may not exist.
-# This isn't necessarily the compiler we'll be running the testsuite
-# with, but it's really the performance of the machine that we're
-# interested in
+ $(PYTHON) calibrate '$(TEST_HC)' > $@
endif
endif
@@ -66,4 +62,3 @@ clean distclean maintainer-clean:
$(RM) -rf install-inplace
$(RM) -f calibrate.out
$(RM) -f Setup Setup.exe Setup.hi Setup.o
-
View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/6f885e6575eb741556d6e198d1a9dbdadf10307b...ea853ff066afb4d4f2271b24be898693e2a3e18d
--
View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/6f885e6575eb741556d6e198d1a9dbdadf10307b...ea853ff066afb4d4f2271b24be898693e2a3e18d
You're receiving this email because of your account on gitlab.haskell.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-commits/attachments/20230331/500a03be/attachment-0001.html>
More information about the ghc-commits
mailing list