[Git][ghc/ghc][master] 8 commits: ghcup-metadata: Don't override existing metadata if version already exists.
Marge Bot (@marge-bot)
gitlab at gitlab.haskell.org
Fri May 26 19:27:17 UTC 2023
Marge Bot pushed to branch master at Glasgow Haskell Compiler / GHC
Commits:
05a1b626 by Matthew Pickering at 2023-05-26T15:26:58-04:00
ghcup-metadata: Don't override existing metadata if version already exists.
If a nightly pipeline runs twice for some reason for the same version
then we really don't want to override an existing entry with new
bindists. This could cause ABI compatability issues for users or break
ghcup's caching logic.
- - - - -
fcbcb3cc by Matthew Pickering at 2023-05-26T15:26:58-04:00
ghcup-metadata: Use proper API url for bindist download
Previously we were using links from the web interface, but it's more
robust and future-proof to use the documented links to the artifacts.
https://docs.gitlab.com/ee/api/job_artifacts.html
- - - - -
5b59c8fe by Matthew Pickering at 2023-05-26T15:26:58-04:00
ghcup-metadata: Set Nightly and LatestNightly tags
The latest nightly release needs the LatestNightly tag, and all other
nightly releases need the Nightly tag. Therefore when the metadata is
updated we need to replace all LatestNightly with Nightly.`
- - - - -
914e1468 by Matthew Pickering at 2023-05-26T15:26:58-04:00
ghcup-metadata: Download nightly metadata for correct date
The metadata now lives in https://gitlab.haskell.org/ghc/ghcup-metadata
with one metadata file per year. When we update the metadata we download
and update the right file for the current year.
- - - - -
16cf7d2e by Matthew Pickering at 2023-05-26T15:26:58-04:00
ghcup-metadata: Download metadata and update for correct year
something about pipeline date
- - - - -
14792c4b by Matthew Pickering at 2023-05-26T15:26:58-04:00
ghcup-metadata: Don't skip CI
On a push we now have a CI job which updates gitlab pages with the
metadata files.
- - - - -
1121bdd8 by Matthew Pickering at 2023-05-26T15:26:59-04:00
ghcup-metadata: Add --date flag to specify the release date
The ghcup-metadata now has a viReleaseDay field which needs to be
populated with the day of the release.
- - - - -
bc478bee by Matthew Pickering at 2023-05-26T15:26:59-04:00
ghcup-metadata: Add dlOutput field
ghcup now requires us to add this field which specifies where it should
download the bindist to. See
https://gitlab.haskell.org/ghc/ghcup-metadata/-/issues/1 for some more
discussion.
- - - - -
3 changed files:
- .gitlab-ci.yml
- .gitlab/rel_eng/mk-ghcup-metadata/README.mkd
- .gitlab/rel_eng/mk-ghcup-metadata/mk_ghcup_metadata.py
Changes:
=====================================
.gitlab-ci.yml
=====================================
@@ -998,8 +998,9 @@ project-version:
# Calculate the project version
- . ./version.sh
- # Download existing ghcup metadata
- - nix shell --extra-experimental-features nix-command --extra-experimental-features flakes nixpkgs#wget -c wget "https://gitlab.haskell.org/ghc/ghcup-metadata/-/raw/updates/ghcup-0.0.7.yaml"
+ # Download existing ghcup metadata for the correct year
+ - PipelineYear="$(date -d $CI_PIPELINE_CREATED_AT +%Y)"
+ - nix shell nixpkgs#wget -c wget "https://ghc.gitlab.haskell.org/ghcup-metadata/ghcup-nightlies-$PipelineYear-0.0.7.yaml" -O ghcup-0.0.7.yaml
- .gitlab/generate_job_metadata
@@ -1044,7 +1045,7 @@ ghcup-metadata-nightly:
artifacts: false
- job: project-version
script:
- - nix shell --extra-experimental-features nix-command -f .gitlab/rel_eng -c ghcup-metadata --metadata ghcup-0.0.7.yaml --pipeline-id="$CI_PIPELINE_ID" --version="$ProjectVersion" > "metadata_test.yaml"
+ - nix shell --extra-experimental-features nix-command -f .gitlab/rel_eng -c ghcup-metadata --metadata ghcup-0.0.7.yaml --date="$(date -d $CI_PIPELINE_CREATED_AT +%Y-%M-%d)" --pipeline-id="$CI_PIPELINE_ID" --version="$ProjectVersion" > "metadata_test.yaml"
rules:
- if: $NIGHTLY
@@ -1063,14 +1064,15 @@ ghcup-metadata-nightly-push:
artifacts: true
script:
- git clone https://gitlab.haskell.org/ghc/ghcup-metadata.git
- - cp metadata_test.yaml ghcup-metadata/ghcup-0.0.7.yaml
+ - PipelineYear="$(date -d $CI_PIPELINE_CREATED_AT +%Y)"
+ - cp metadata_test.yaml "ghcup-metadata/ghcup-nightlies-$PipelineYear-0.0.7.yaml"
- cd ghcup-metadata
- git config user.email "ghc-ci at gitlab-haskell.org"
- git config user.name "GHC GitLab CI"
- git remote add gitlab_origin https://oauth2:$PROJECT_PUSH_TOKEN@gitlab.haskell.org/ghc/ghcup-metadata.git
- git add .
- git commit -m "Update metadata"
- - git push gitlab_origin HEAD:updates -o ci.skip
+ - git push gitlab_origin HEAD:updates
rules:
# Only run the update on scheduled nightly pipelines, ie once a day
- if: $NIGHTLY && $CI_PIPELINE_SOURCE == "schedule" && $CI_COMMIT_BRANCH == "master"
@@ -1080,7 +1082,7 @@ ghcup-metadata-release:
# No explicit needs for release pipeline as we assume we need everything and everything will pass.
extends: .ghcup-metadata
script:
- - nix shell --extra-experimental-features nix-command -f .gitlab/rel_eng -c ghcup-metadata --release-mode --metadata ghcup-0.0.7.yaml --pipeline-id="$CI_PIPELINE_ID" --version="$ProjectVersion" > "metadata_test.yaml"
+ - nix shell --extra-experimental-features nix-command -f .gitlab/rel_eng -c ghcup-metadata --release-mode --metadata ghcup-0.0.7.yaml --date="$(date -d $CI_PIPELINE_CREATED_AT +%Y-%M-%d)" --pipeline-id="$CI_PIPELINE_ID" --version="$ProjectVersion" > "metadata_test.yaml"
rules:
- if: '$RELEASE_JOB == "yes"'
=====================================
.gitlab/rel_eng/mk-ghcup-metadata/README.mkd
=====================================
@@ -18,6 +18,7 @@ options:
--release-mode Generate metadata which points to downloads folder
--fragment Output the generated fragment rather than whole modified file
--version VERSION Version of the GHC compiler
+ --date DATE Date of the compiler release
```
The script also requires the `.gitlab/jobs-metadata.yaml` file which can be generated
=====================================
.gitlab/rel_eng/mk-ghcup-metadata/mk_ghcup_metadata.py
=====================================
@@ -63,7 +63,8 @@ eprint(f"Supported platforms: {job_mapping.keys()}")
# Artifact precisely specifies a job what the bindist to download is called.
class Artifact(NamedTuple):
job_name: str
- name: str
+ download_name: str
+ output_name: str
subdir: str
# Platform spec provides a specification which is agnostic to Job
@@ -72,8 +73,14 @@ class PlatformSpec(NamedTuple):
name: str
subdir: str
-source_artifact = Artifact('source-tarball', 'ghc-{version}-src.tar.xz', 'ghc-{version}' )
-test_artifact = Artifact('source-tarball', 'ghc-{version}-testsuite.tar.xz', 'ghc-{version}' )
+source_artifact = Artifact('source-tarball'
+ , 'ghc-{version}-src.tar.xz'
+ , 'ghc-{version}-src.tar.xz'
+ , 'ghc-{version}' )
+test_artifact = Artifact('source-tarball'
+ , 'ghc-{version}-testsuite.tar.xz'
+ , 'ghc-{version}-testsuite.tar.xz'
+ , 'ghc-{version}' )
def debian(arch, n):
return linux_platform(arch, "{arch}-linux-deb{n}".format(arch=arch, n=n))
@@ -104,7 +111,7 @@ def linux_platform(arch, opsys):
return PlatformSpec( opsys, 'ghc-{version}-{arch}-unknown-linux'.format(version="{version}", arch=arch) )
-base_url = 'https://gitlab.haskell.org/ghc/ghc/-/jobs/{job_id}/artifacts/raw/{artifact_name}'
+base_url = 'https://gitlab.haskell.org/api/v4/projects/1/jobs/{job_id}/artifacts/{artifact_name}'
hash_cache = {}
@@ -129,7 +136,7 @@ def download_and_hash(url):
def mk_one_metadata(release_mode, version, job_map, artifact):
job_id = job_map[artifact.job_name].id
- url = base_url.format(job_id=job_id, artifact_name=urllib.parse.quote_plus(artifact.name.format(version=version)))
+ url = base_url.format(job_id=job_id, artifact_name=urllib.parse.quote_plus(artifact.download_name.format(version=version)))
# In --release-mode, the URL in the metadata needs to point into the downloads folder
# rather then the pipeline.
@@ -143,10 +150,13 @@ def mk_one_metadata(release_mode, version, job_map, artifact):
eprint(f"Bindist URL: {url}")
eprint(f"Download URL: {final_url}")
- # Download and hash from the release pipeline, this must not change anyway during upload.
+ #Download and hash from the release pipeline, this must not change anyway during upload.
h = download_and_hash(url)
- res = { "dlUri": final_url, "dlSubdir": artifact.subdir.format(version=version), "dlHash" : h }
+ res = { "dlUri": final_url
+ , "dlSubdir": artifact.subdir.format(version=version)
+ , "dlOutput": artifact.output_name.format(version=version)
+ , "dlHash" : h }
eprint(res)
return res
@@ -155,10 +165,14 @@ def mk_one_metadata(release_mode, version, job_map, artifact):
def mk_from_platform(pipeline_type, platform):
info = job_mapping[platform.name][pipeline_type]
eprint(f"From {platform.name} / {pipeline_type} selecting {info['name']}")
- return Artifact(info['name'] , f"{info['jobInfo']['bindistName']}.tar.xz", platform.subdir)
+ return Artifact(info['name']
+ , f"{info['jobInfo']['bindistName']}.tar.xz"
+ , "ghc-{version}-{pn}.tar.xz".format(version="{version}", pn=platform.name)
+ , platform.subdir)
+
# Generate the new metadata for a specific GHC mode etc
-def mk_new_yaml(release_mode, version, pipeline_type, job_map):
+def mk_new_yaml(release_mode, version, date, pipeline_type, job_map):
def mk(platform):
eprint("\n=== " + platform.name + " " + ('=' * (75 - len(platform.name))))
return mk_one_metadata(release_mode, version, job_map, mk_from_platform(pipeline_type, platform))
@@ -227,7 +241,14 @@ def mk_new_yaml(release_mode, version, pipeline_type, job_map):
else:
change_log = "https://gitlab.haskell.org"
- return { "viTags": ["Latest", "TODO_base_version"]
+ if release_mode:
+ tags = ["Latest", "TODO_base_version"]
+ else:
+ tags = ["LatestNightly"]
+
+
+ return { "viTags": tags
+ , "viReleaseDay": date
# Check that this link exists
, "viChangeLog": change_log
, "viSourceDL": source
@@ -239,6 +260,15 @@ def mk_new_yaml(release_mode, version, pipeline_type, job_map):
}
+def setNightlyTags(ghcup_metadata):
+ for version in ghcup_metadata['ghcupDownloads']['GHC']:
+ if "LatestNightly" in ghcup_metadata['ghcupDownloads']['GHC'][version]["viTags"]:
+ ghcup_metadata['ghcupDownloads']['GHC'][version]["viTags"].remove("LatestNightly")
+ ghcup_metadata['ghcupDownloads']['GHC'][version]["viTags"].append("Nightly")
+
+
+
+
def main() -> None:
import argparse
@@ -249,6 +279,7 @@ def main() -> None:
parser.add_argument('--fragment', action='store_true', help='Output the generated fragment rather than whole modified file')
# TODO: We could work out the --version from the project-version CI job.
parser.add_argument('--version', required=True, type=str, help='Version of the GHC compiler')
+ parser.add_argument('--date', required=True, type=str, help='Date of the compiler release')
args = parser.parse_args()
project = gl.projects.get(1, lazy=True)
@@ -269,17 +300,21 @@ def main() -> None:
eprint(f"Pipeline Type: {pipeline_type}")
- new_yaml = mk_new_yaml(args.release_mode, args.version, pipeline_type, job_map)
+ new_yaml = mk_new_yaml(args.release_mode, args.version, args.date, pipeline_type, job_map)
if args.fragment:
print(yaml.dump({ args.version : new_yaml }))
else:
with open(args.metadata, 'r') as file:
ghcup_metadata = yaml.safe_load(file)
+ if args.version in ghcup_metadata['ghcupDownloads']['GHC']:
+ raise RuntimeError("Refusing to override existing version in metadata")
+ setNightlyTags(ghcup_metadata)
ghcup_metadata['ghcupDownloads']['GHC'][args.version] = new_yaml
print(yaml.dump(ghcup_metadata))
+
if __name__ == '__main__':
main()
View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/34b44f7d22883ef89784c727c25f0dae225be8d2...bc478bee6e7e46bcf30212ab94545a83bdeb0203
--
View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/34b44f7d22883ef89784c727c25f0dae225be8d2...bc478bee6e7e46bcf30212ab94545a83bdeb0203
You're receiving this email because of your account on gitlab.haskell.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-commits/attachments/20230526/59581c50/attachment-0001.html>
More information about the ghc-commits
mailing list