[WIP] Fix runner exit code shell

.github/workflows/architecture.md

@@ -0,0 +1,106 @@
+# Swift Package Test Workflow Architecture
+
+## Goals
+
+The Swift Package Test workflow is intended to give maintainers of open-source Swift packages a low-friction way to give contributors confidence in the code they write. This means making testing against a range of Swift versions and platforms as simple as possible, and making the results as accessible as possible.
+
+The following key features are treated as requirements by the unified architecture.
+
+### 1. GitHub-release-based versioning
+
+Workflows and actions must be versioned using GitHub releases to guarantee stability to adopters. Properly tagged SemVer will allow even breaking changes to be rolled out safely. This method of versioning also lets adopters use Dependabot to automatically open PRs which bump their pinned references.
+
+### 2. No skipped jobs
+
+No reusable workflow offered should show skipped jobs. Skipped jobs add visual noise, look confusing to contributors and imply that something is configured incorrectly.
+
+### 3. Scripts must be cloned not curled
+
+Some jobs rely on scripts to execute their functionality. This is often preferred over inline scripts in the workflow definition `yml` files since it gives a better developer experience to the maintainers of those scripts. However, one downside of this approach is that these scripts are not present when a reusable workflow is executed from another repository. It is important that a unified solution checks the scripts out explicitly rather than curling them, since curling frequently runs into rate limiting with GitHub's API.
+
+### 4. Custom matrix builds
+
+Some packages need additional checks, such as their own integration tests or a custom script. Those checks often need the same matrix of builds that the recommended test workflow, `package_test.yml`, uses. Hence, a unified solution should offer lower level primitives that can execute a matrix. Furthermore, it should provide a workflow with a matrix that is already configured with the recommended Swift versions and platforms that just executes a command across them.
+
+### 5. Detect minimum version
+
+Any matrix that is generated should take the tools-version of the package manifest into consideration to automatically remove unsupported Swift versions. This matters most for newly released packages, which often support only the latest Swift version.
+
+## High level design
+
+Provide one workflow which can be adopted to run a range of common test and build configurations on a variety of platforms against multiple Swift versions. This workflow sits atop a "matrix generation" layer which takes inputs and produces a canonical work definition (in YAML or JSON). That definition is then expanded into one job per entry, each executing one slice of the work. Benchmarking is a further consumer of the same layer rather than part of the recommended test workflow.
+
+The design is intended to be layered, so that adopters may use the whole stack or, where the top-level workflow does not offer the customization they need, drop down a level and supply that part themselves:
+
+* Use the `package_test.yml` workflow for the full suite of conveniences
+* Use a custom workflow (perhaps with custom inputs) which calls `toolchain_matrix.yml` for the matrix and `execute_matrix.yml` to run its own command across it
+* Generate the work definition in YAML or JSON by some other means, or hard-code it into a workflow, and pass it to `execute_matrix.yml`
+
+## Components of the design
+
+* `package_test.yml`: top-level workflow for the full suite of conveniences
+* `benchmarks.yml`: top-level workflow for benchmarking, consuming the same matrix layer
+* `toolchain_matrix.yml`: produces a matrix of toolchains with no command attached, for callers supplying their own
+* `execute_matrix.yml`: expands a work definition into one job per entry and dispatches each to its platform. FreeBSD runs in a VM step here rather than through a job runner
+* `generate-matrix.swift`: produces the canonical work definition YAML or JSON. It reads the workflow's inputs from the environment and fails the run rather than emitting a matrix that would silently drop a job the caller asked for
+* `job-runner-linux.sh`, `job-runner-macos.sh`, `job-runner-windows.ps1`: helper scripts which execute the defined work. They handle the complexity of running inside/outside Docker and installing Swift if required.
+
+## Swift version labels
+
+A version list names toolchains with labels: a version such as `6.3`, `nightly-release`, `nightly-main`, or a position counted back from the current release, `current`, `current-1` and `current-2`. A job's name carries the label the caller wrote. Job names are required status checks, so the default lists use the relative labels: a new Swift release changes which versions they name and leaves the checks alone. A caller who writes versions out gets those names, and the two spellings can be mixed in one list.
+
+The workflow supplies what the relative labels resolve against, in `swift_current_release` and `swift_nightly_release`. They are inputs with defaults in the workflow files rather than values in the generator, so a workflow pinned to a release tag keeps the versions it was released with. Setting `swift_current_release` holds a repository at an earlier release without changing its check names.
+
+## The matrix definition
+
+`execute_matrix.yml` takes the work definition as a YAML string (JSON is valid YAML, so a JSON document works too) in `matrix_yaml_string`. `generate-matrix.swift` writes it, and a caller can write it by hand or generate it some other way. It is a map with one key, `config`, holding a list of entries. Each entry becomes one job, named by its `name`. An empty list is valid and runs no jobs. A missing or non-list `config`, an unknown `platform`, or an entry left with no `command` fails before any job starts.
+
+Fields every entry carries:
+
+| Field | Type | Meaning |
+| --- | --- | --- |
+| `platform` | `Linux`, `macOS`, `Windows` or `FreeBSD` | Which runner script the job dispatches to. |
+| `name` | string | The job name, which is also the status check name. |
+| `runner` | list of strings | The `runs-on` labels. |
+| `command` | string | The command to run. An entry that omits it takes `execute_matrix.yml`'s `command` input. |
+| `setup_command` | string | Run before the command. Defaults to the `setup_command` input. |
+| `command_arguments` | list of strings | Appended to the command. Defaults to the `command_arguments` input. |
+| `timeout_minutes` | number | The job's timeout. An entry that omits it takes `execute_matrix.yml`'s `job_timeout` input. |
+| `env` | map | Environment variables for the job. The `env` input supplies defaults and an entry's own keys win. |
+
+The toolchain is described by one further key, by platform:
+
+* **Linux and Windows** use `swift_build`:
+  * `swift_version` is the label, such as `6.3`, `nightly-release` or `nightly-main`. It is the only required key.
+  * `toolchain` and `swiftly` are the resolved forms of a nightly (for example `nightly-6.4.x` and `6.4.x-snapshot`). Leave them out for a release.
+  * `container` (`image`, and optionally `dockerfile`, `capabilities`, `security_options`) runs the job in a Docker container. Leave it out to run on the runner.
+  * `sdk` (`type`, one of `static-linux`, `wasm`, `embedded-wasm`, `android`; plus `triples` and, for Android, `ndk_version`) installs a Swift SDK and runs the command as the build with it. An entry cannot carry both `sdk` and `container`.
+* **macOS** uses `xcode_build`:
+  * `swift_version` or `xcode_version` selects the Xcode, and `swiftly_toolchain` adds a swiftly-managed toolchain under it.
+  * `targets` is a list of `platform`, `scheme`, `build_destination`, `test_destination`, `build` and `test`, for building and testing on simulators.
+  * `debug_output` prints the `xcodebuild` output.
+* **FreeBSD** uses `freebsd`: `os_version`, `swift_version`, `swift_url`, `build_flags` and `env_vars`.
+
+Linux entries may also set `android_emulator: true` to run the Android emulator tests after an Android SDK build.
+
+For example, one release and one nightly on Linux:
+
+```yaml
+config:
+  - name: Linux Swift 6.3
+    platform: Linux
+    runner: [ubuntu-latest]
+    command: swift test
+    swift_build:
+      swift_version: "6.3"
+  - name: Linux Swift nightly-release
+    platform: Linux
+    runner: [ubuntu-latest]
+    command: swift test
+    swift_build:
+      swift_version: nightly-release
+      toolchain: nightly-6.4.x
+      swiftly: 6.4.x-snapshot
+```
+
+The shortest way to see every field a platform takes is to run `generate-matrix.swift` with the relevant `ENABLE_*` variable set and read its output.

.github/workflows/benchmarks.yml

@@ -0,0 +1,204 @@
+name: Benchmarks
+
+permissions:
+  contents: read
+
+on:
+  workflow_call:
+    inputs:
+      benchmark_package_paths:
+        type: string
+        description: 'JSON array of benchmarking package paths to run in sequence on each runner, e.g. ''["Benchmarks/A", "Benchmarks/B"]''. Empty or [] runs the package in the repository root.'
+        default: "[]"
+      swift_package_arguments:
+        type: string
+        description: "Additional arguments passed to swift package (e.g. --disable-sandbox)."
+        default: ""
+      swift_current_release:
+        type: string
+        description: "The Swift release that `current` names in a version list, and that `current-1` and `current-2` count back from. Job names use the relative label, so a new release moves the versions under the same checks. Changing it holds a repository's versions at a release; changing a version list names versions explicitly."
+        default: "6.4"
+      swift_nightly_release:
+        type: string
+        description: "The branch that `nightly-release` names, in the spelling upstream publishes its nightlies under."
+        default: "6.4.x"
+      linux_swift_versions:
+        type: string
+        description: "Linux Swift version list (JSON array). Empty uses current-2, current-1, current, nightly-release and nightly-main."
+        default: ""
+      enable_linux:
+        type: boolean
+        description: "Run the benchmarks on Linux."
+        default: true
+      enable_macos:
+        type: boolean
+        description: "Run the benchmarks on macOS."
+        default: false
+      macos_swift_versions:
+        type: string
+        description: "macOS Swift version list (JSON array) for benchmarks. Resolved through the runners' Xcode symlinks. Empty, with macos_xcode_versions also empty, uses the generator's list of release versions."
+        default: ""
+      macos_xcode_versions:
+        type: string
+        description: "macOS Xcode version list (JSON array) for benchmarks. Combined with macos_swift_versions rather than replaced by it."
+        default: ""
+      macos_runner_pool:
+        type: string
+        description: "The self-hosted runner pool for macOS benchmark jobs."
+        default: "general"
+      linux_env_vars:
+        type: string
+        description: "Environment variables for Linux jobs as JSON."
+        default: "{}"
+      macos_env_vars:
+        type: string
+        description: "Environment variables for macOS jobs as JSON."
+        default: "{}"
+      minimum_swift_version:
+        type: string
+        description: "Minimum Swift version. Empty auto-detects from Package.swift, 'none' disables filtering, or name a version explicitly."
+        default: ""
+      name:
+        type: string
+        description: "Name used for the concurrency group. Set this when a workflow calls benchmarks.yml more than once, or the calls cancel each other."
+        default: "benchmarks"
+      job_timeout:
+        type: number
+        description: "Timeout in minutes for each job."
+        default: 60
+      needs_token:
+        type: boolean
+        description: "Whether to provide GITHUB_TOKEN to jobs."
+        default: false
+      enable_cross_pr_testing:
+        type: boolean
+        description: "Whether PRs can be tested together with linked PRs mentioned in the PR description."
+        default: false
+
+      workflows_repository:
+        type: string
+        description: "Repository to take the matrix scripts from. Point this at a fork to test a change to the workflows before it lands; it carries no version, so Dependabot has only the `uses:` line to bump."
+        default: "swiftlang/github-workflows"
+      workflows_ref:
+        type: string
+        description: "Ref to take the scripts from. Empty uses workflows_repository's default branch, which is correct for a released version; set it when workflows_repository is a fork whose default branch does not carry the change."
+        default: ""
+jobs:
+  generate-matrix:
+    name: Generate benchmark matrix
+    runs-on: ubuntu-latest
+    outputs:
+      matrix_yaml: ${{ steps.generate.outputs.matrix_yaml }}
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v7
+        with:
+          persist-credentials: false
+      - name: Resolve the workflows source
+        id: workflows_source
+        shell: bash
+        env:
+          WORKFLOWS_REPOSITORY: ${{ inputs.workflows_repository }}
+          WORKFLOWS_REF: ${{ inputs.workflows_ref }}
+        run: |
+          set -euo pipefail
+
+          # Empty uses the default branch.
+          echo "ref=$WORKFLOWS_REF" >> $GITHUB_OUTPUT
+
+          if [ "$GITHUB_REPOSITORY" = "$WORKFLOWS_REPOSITORY" ]; then
+            echo "needs_checkout=false" >> $GITHUB_OUTPUT
+            echo "root_directory=$GITHUB_WORKSPACE" >> $GITHUB_OUTPUT
+          else
+            echo "needs_checkout=true" >> $GITHUB_OUTPUT
+            echo "root_directory=$GITHUB_WORKSPACE/github-workflows" >> $GITHUB_OUTPUT
+          fi
+      - name: Checkout the workflows repository
+        if: ${{ steps.workflows_source.outputs.needs_checkout == 'true' }}
+        uses: actions/checkout@v7
+        with:
+          repository: ${{ inputs.workflows_repository }}
+          ref: ${{ steps.workflows_source.outputs.ref }}
+          path: github-workflows
+          persist-credentials: false
+      - name: Generate matrix
+        id: generate
+        env:
+          WORKFLOWS_CHECKOUT: ${{ steps.workflows_source.outputs.root_directory }}
+          LINUX_USER_ENV_VARS: ${{ inputs.linux_env_vars }}
+          MACOS_USER_ENV_VARS: ${{ inputs.macos_env_vars }}
+          BENCHMARK_PACKAGE_PATHS: ${{ inputs.benchmark_package_paths }}
+          SWIFT_PACKAGE_ARGUMENTS: ${{ inputs.swift_package_arguments }}
+          ENABLE_LINUX: ${{ inputs.enable_linux }}
+          ENABLE_MACOS: ${{ inputs.enable_macos }}
+          ENABLE_WINDOWS: "false"
+          ENABLE_MACOS_SWIFTLY: "false"
+          ENABLE_FREEBSD: "false"
+          ENABLE_LINUX_STATIC_SDK_BUILD: "false"
+          ENABLE_WASM_SDK_BUILD: "false"
+          ENABLE_EMBEDDED_WASM_SDK_BUILD: "false"
+          ENABLE_ANDROID_SDK_BUILD: "false"
+          ENABLE_ANDROID_EMULATOR_TESTS: "false"
+          ENABLE_CXX_INTEROP: "false"
+          LINUX_SWIFT_VERSIONS: ${{ inputs.linux_swift_versions }}
+          MACOS_SWIFT_VERSIONS: ${{ inputs.macos_swift_versions }}
+          MACOS_XCODE_VERSIONS: ${{ inputs.macos_xcode_versions }}
+          MACOS_RUNNER_POOL: ${{ inputs.macos_runner_pool }}
+          MINIMUM_SWIFT_VERSION: ${{ inputs.minimum_swift_version }}
+          SWIFT_CURRENT_RELEASE: ${{ inputs.swift_current_release }}
+          SWIFT_NIGHTLY_RELEASE: ${{ inputs.swift_nightly_release }}
+          LINUX_SETUP_COMMAND: |
+            git config --global --add safe.directory "$(pwd)"
+            # The job runs unprivileged on the runner but as root in a container,
+            # where sudo is not installed.
+            if command -v sudo >/dev/null 2>&1; then
+              sudo apt-get update -y -q && sudo apt-get install -y -q libjemalloc-dev jq
+            else
+              apt-get update -y -q && apt-get install -y -q libjemalloc-dev jq
+            fi
+          MACOS_SETUP_COMMAND: |
+            git config --global --add safe.directory "$(pwd)"
+            brew install jemalloc
+        run: |
+          set -euo pipefail
+
+          # The threshold script reads the package paths from the environment, so
+          # they are added to the caller's own variables rather than replacing
+          # them.
+          with_benchmark_paths() {
+            jq -cn \
+              --argjson user "$(echo "${1:-}" | yq -o=json '. // {}')" \
+              --arg paths "$BENCHMARK_PACKAGE_PATHS" \
+              '$user + {BENCHMARK_PACKAGE_PATHS: $paths}'
+          }
+          LINUX_ENV_VARS=$(with_benchmark_paths "$LINUX_USER_ENV_VARS")
+          MACOS_ENV_VARS=$(with_benchmark_paths "$MACOS_USER_ENV_VARS")
+          export LINUX_ENV_VARS MACOS_ENV_VARS
+
+          # Single-quoted so ${SCRIPTS_ROOT} survives into the matrix entry and is
+          # expanded on the runner, where the scripts directory is known.
+          threshold_command='${SCRIPTS_ROOT}/check-benchmark-thresholds.sh'
+          if [ -n "$SWIFT_PACKAGE_ARGUMENTS" ]; then
+            threshold_command="$threshold_command $SWIFT_PACKAGE_ARGUMENTS"
+          fi
+          LINUX_COMMAND="$threshold_command"
+          MACOS_COMMAND="$threshold_command"
+          export LINUX_COMMAND MACOS_COMMAND
+
+          matrix_yaml=$("${WORKFLOWS_CHECKOUT}/.github/workflows/scripts/matrix/generate-matrix.swift")
+          echo "matrix_yaml<<EOF" >> $GITHUB_OUTPUT
+          echo "$matrix_yaml" >> $GITHUB_OUTPUT
+          echo "EOF" >> $GITHUB_OUTPUT
+
+  benchmarks:
+    name: Benchmarks
+    needs: generate-matrix
+    uses: ./.github/workflows/execute_matrix.yml
+    with:
+      name: ${{ inputs.name }}
+      matrix_yaml_string: ${{ needs.generate-matrix.outputs.matrix_yaml }}
+      workflows_repository: ${{ inputs.workflows_repository }}
+      workflows_ref: ${{ inputs.workflows_ref }}
+      job_timeout: ${{ inputs.job_timeout }}
+      needs_token: ${{ inputs.needs_token }}
+      enable_cross_pr_testing: ${{ inputs.enable_cross_pr_testing }}