Release and Publishing

Audience: Maintainer

How a maintainer cuts a release: tag → publish (build, push, sign, attest, and package + push + sign the chart) → verify → record digests. This is the maintainer runbook for producing a release. Operators consuming a release pin the published digests at install time — see tenant-onboarding.md and the chart README.

What a release produces

Operators installing a release pin the published digests at install time — see install.md § Pin images by digest.

A release is a vX.Y.Z git tag plus its outputs:

• The five first-party images — gmc, agc, proxy, worker, wrapper — pushed to GHCR (ghcr.io/actions-gateway/<name>), each tagged vX.Y.Z and by long commit SHA. Each is multi-arch (linux/amd64 + linux/arm64): the pushed artifact is an OCI image index, and the digest recorded everywhere (run summary, release notes, chart pins) is the index digest — the kubelet resolves the per-arch manifest from it at pull time, so one pinned digest schedules on both amd64 and arm64 (e.g. Graviton) nodes.
• A keyless cosign signature on every image (sigstore/Fulcio via GitHub Actions OIDC — no signing key, no stored secret), signed recursively — the index and each per-arch manifest — and an SPDX-JSON SBOM per architecture attached as a keyless cosign attestation to that architecture's manifest.
• A signed SLSA build-provenance attestation on every image (actions/attest-build-provenance), attached to the index digest as an OCI referrer. It is generated through the same keyless path as the signatures — the publish workflow's GitHub OIDC identity → a short-lived Fulcio cert → Rekor — so the provenance is authenticated (it records the workflow, repo, commit, and trigger that produced the image and cannot be forged by a pusher). This reaches SLSA Build L2; buildx's own unsigned default provenance is disabled in favour of it. Consumers verify it with gh attestation verify or cosign verify-attestation (see step 3).
• The Helm chart, packaged and pushed as an OCI artifact to oci://ghcr.io/actions-gateway/charts/actions-gateway, with its version and appVersion set to the release tag and a keyless cosign signature from the same Fulcio/Rekor flow as the images. Operators install it straight from the registry (helm install … oci://…) with the published image digests pinned — no git clone of the chart. OCI (over a gh-pages chart repo) is chosen so the chart reuses the images' registry, login, and keyless-signing path; Artifact Hub (see Chart.yaml annotations) indexes the OCI ref for discoverability.
• The opt-in v2 CRD chart, packaged and pushed alongside the main chart to oci://ghcr.io/actions-gateway/charts/actions-gateway-crds-v2, with the same version derivation and keyless signature. It ships only the v2alpha1 (actions-gateway.com) CRDs — separated from the main chart because the large pod-template CRDs would otherwise push the main chart's Helm release Secret past its 1 MiB limit (Q149). Operators install it only when adopting the v2 API. Both chart packages are produced by the same chart-publish job.
• The signed v2 CRD manifest (Q276), a pre-rendered actions-gateway-crds-v2.yaml (rendered for the default gmc-system namespace) attached to the tag's GitHub Release with a keyless cosign blob signature (sign-blob → a Sigstore bundle, actions-gateway-crds-v2.yaml.cosign.bundle). This is the helm-free manual install path — the v2 CRD chart is too large to helm install, so operators kubectl apply --server-side -f …/releases/download/<tag>/actions-gateway-crds-v2.yaml. The chart-publish job renders, signs, and uploads it (widening the job to contents: write).
• The gag-migrate CLI binaries (Q306), cross-compiled by chart-publish for the operator platform matrix (linux/amd64, linux/arm64, darwin/amd64, darwin/arm64, windows/amd64) via scripts/release/build-migrate-binaries.sh and attached to the GitHub Release as gag-migrate-<tag>-<os>-<arch> assets. A single SHA256SUMS manifest is keyless cosign sign-blob-signed (SHA256SUMS.cosign.bundle) — the same no-secret Fulcio/Rekor path as the v2 CRD manifest — so one signature covers the whole set (verify the manifest signature, then sha256sum -c). This is the one-shot v1→v2 migration tool, previously source-build-only.
• The GitHub Release itself (Q293), composed by chart-publish: the five image index digests, the make verify-release command, a generated changelog, and a tag-derived --prerelease flag. It is created only if the tag has no Release yet, so a maintainer's pre-tag curated notes are never clobbered.

Both the image and chart work are automated by the publish.yml workflow, which triggers on the v* tag push (the chart-publish job runs after every image leg succeeds). The maintainer's job is to cut the tag and verify the result.

Signing and chart publish are exercised for the first time on the first real v* tag. Pull-request CI builds each image and generates its SBOM, but it does not push, sign, attest, or publish the chart (those need a registry push and the publish workflow's OIDC identity). The verification step below is therefore not optional on the first release — it is the only thing that proves the signing and chart-publish paths work.

One-time setup (first release only)

1. GHCR package visibility. The first publish creates the ghcr.io/actions-gateway/{gmc,agc,proxy,worker,wrapper} image packages and the ghcr.io/actions-gateway/charts/actions-gateway chart package. They inherit the repository's visibility and may start private. For third parties to run cosign verify / helm pull (and for an air-gapped operator to pull), set each package to public in the org's GHCR package settings, or keep them private and distribute pull credentials. Verification by this project's CI and by anyone with pull access works either way — but the released-chart upgrade gate (make chart-released-upgrade-check, run by every e2e CI pass) pulls the chart and the released gmc image anonymously, so those two packages must be public for PR CI to stay green.
2. Workflow permissions are already declared in publish.yml (packages: write to push, id-token: write for keyless cosign and provenance, attestations: write for the build-provenance attestation). No repo secret is required — that is the point of keyless signing.

When to cut

Everything below this section is how to cut a release. This section is whether to — the question the rest of the page assumes has already been answered.

Start from the record, not from memory. The delete-on-done Queue erases delivered work from STATUS.md by design, so nothing in the backlog shows what has piled up since the last tag:

scripts/release/release-delta.sh

It reports, for <last stable tag>..origin/main, the commits by Conventional Commit type (breaking ones called out), the Queue rows closed in that window, the API diffstat that is the semver signal, and the operator-facing pages touched. Pass an explicit FROM (and optionally TO) to look at a different window.

The triggers. Any one of these is reason enough to scope a release; none of them is automatic, and the report is the input to the judgement rather than a substitute for it:

• A security fix users cannot get any other way. Cut promptly, as a patch off the release branch if main carries unrelated risk (Patch releases and backports).
• A headline capability landed. Scope a minor around it — one thing an adopter would upgrade for, with the rest of the window riding along. That is the 1.3 pattern: worker right-sizing was the headline, and three dozen other changes shipped underneath it.
• User-visible fixes accumulated with no feature. A patch release. The bar is that an operator running the current tag is hitting something already fixed, not that the fix count crossed a number.
• Internal-only churn. Wait. Refactors, test coverage, CI work, and docs the site publishes continuously do not need a tag to reach anyone.

The counterweight — a tag is not free. Every GA cut spends the release-candidate dogfood validation: a real GKE cluster, a live e2e matrix, and a maintainer watching it. That cost is what makes "enough" a real bar rather than a formality — it is also why the answer to a thin delta is wait, not cut a small one.

Once you decide to cut, the question changes. From that point the report stops being the view: write the release plan doc, open it with a scope ledger, and let the ledger's -gate rows answer "is it done?" until the tag.

Release sequence

1. Pre-flight

• main is green: unit/integration/e2e and security-scan.yml all passing on the commit you are about to tag. Run make check locally as a final gate.
• Choose the version vX.Y.Z (semver). The tag must match v* or publish.yml will refuse to publish.
• Review the API surface this tag publishes for the first time. A field, enum value, or default costs a rename to change before it ships and a conversion shim plus a deprecation window afterwards, so the tag is the moment the cheap window closes. Nothing lints this — it is judgement, not a gate that can pass or fail mechanically.

scripts/release/api-surface-since.sh

Apply the checklist in api-review.md to each addition it lists, record the verdict in the release's plan doc, and file anything deferred as a Queue row carrying this release's gate label. "Ship as-is, deliberately" is a valid and common outcome — the point is that the shape is chosen rather than frozen by default.

This step exists because it nearly did not happen: Q476 renamed capacityGate.mode: On days before 1.3.0 would have published it, and only because the question came up in an unrelated conversation.

• Review the operator-facing caveats this tag publishes, and curate the notes if there are any. publish.yml writes a generated body from the commit log, which never says "this upgrade needs a manual step first". Richer notes are opt-in and must be created before you push the tag (§ 5), so the decision belongs here, not after.

scripts/release/operator-caveats-since.sh

It needs no bookkeeping to stay current: the doc-update matrix already requires an operator-visible change to land in docs/operations/, so the diff of those pages since the last tag already is the list — this only makes it scannable (added sections, bold-lead bullets, anything marked BREAKING) instead of a release-sized raw diff.

Reading it is judgement: a clarification is not a caveat, and the script cannot tell them apart. What it guarantees is that you have seen them. Carry the real ones into a curated body:

gh release create vX.Y.Z --draft --notes-file <file>

This step exists because the alternative is remembering. The v1.2.0→next window accumulated a required pre-upgrade kubectl apply, a removed values key that fails the render, and a rollback that re-arms a cluster-wide outage — each recorded correctly in docs/operations/upgrade.md by the change that introduced it, and each invisible to anyone reading a generated changelog.

• Reconcile docs/roadmap.md and docs/features.md against docs/STATUS.md before you tag. The same freeze that applies to the announce bar applies here: a stable tag deploys that tag's docs wholesale, so a stale roadmap is published permanently under that version. make roadmap-check catches the mechanical half — a roadmap bullet naming a deleted Queue row, or one sitting in the wrong section — and it runs in CI. What it cannot catch is the move itself: work that shipped this cycle needs a docs/features.md line, and a Deferred row describing a capability an adopter would ask about belongs in Exploring. A 2026-07-25 audit found six of seven near-term items already shipped.
• Optional: refresh the docs-site announce bar's highlight. The banner in overrides/main.html is the "vX.Y.Z is here" strip at the top of every page on the site. Its version needs no action: it is derived from the git tags at build time, so a stable tag names itself automatically (website.md § The announce bar). What you may want to update is the one-line highlight after it, and the highlight_for version that guards it. Leave both alone and the banner reads "vX.Y.Z is here. Read the release notes.", which is correct but plainer; update them together and it leads with this release's headline. Land the change before you tag either way.

This used to be a manual version bump, and every stable tag to date missed it: v1.0.0 shipped saying "Alpha, pre-1.0", and v1.1.0 and v1.2.0 both said "v1.0.0 is here". Preview what the tag will render, standing in the version you are about to cut (without it the local build resolves the current newest tag, which is still the previous release):

GAG_DOCS_RELEASE=vX.Y.Z make docs-build && awk '/md-banner/,/<\/aside>/' site/index.html

publish.yml enforces the result via its announce-bar job, which every publishing job depends on: it builds the site at the tag and fails the release if the rendered banner does not name it, before any image is pushed. Prereleases are exempt (they publish no docs), as is a backport tag cut after a newer minor, since the banner advertises the newest release by design.

(A docs_ref seed of an already-cut release pins overrides/ to the current checkout, so re-seeding refreshes the highlight on past tags too. See website.md § Seeding.)

Validate the release candidate on dogfood

Before promoting a release-candidate line to a stable vX.Y.Z tag, validate the latest RC functionally on the dogfood cluster. main-green covers unit/integration/kind-e2e, but publishing an image the pipeline signed is not the same as proving it runs jobs — this gate exercises real GAG-provisions-runners-on-GKE behaviour the CI tiers can't observe. Run it before every GA (vX.Y.Z) cut; skip it only for an RC-to-RC or a patch tag that changes nothing an operator runs.

The dogfood scripts pin GAG to any published ref via GAG_IMAGE_TAG, which resolves both as an image tag (ghcr.io/actions-gateway/{gmc,agc,proxy,wrapper}:<ref>) and as a git ref (for the matching CRDs) — an RC tag satisfies both by construction.

One command runs the whole gate, and it runs for the better part of an hour — a green v1.3.0-rc.4 run took 39 minutes end to end — with nothing to type at it after the first confirmation. validate-release.sh bakes in all the env and ordering below — deploy → route CI → on-demand e2e → dispatch the e2e matrix (run-scoped routing) → CRD smoke → teardown — is idempotent, and self-cleans back to 0 nodes on exit (success or failure). On failure it first dumps a cluster snapshot (nodes, pods, unhealthy-pod detail, events) to the gate's output, because the teardown's scale-to-0 evicts every pod and destroys the evidence — read the Failure diagnostics section of a failed run's log (e.g. the FailedScheduling events) instead of re-running the gate to watch it fail again. The legs it runs are documented at the end of this section, and are the recovery path if one needs re-running by hand.

If you just merged something, the gate waits before it spends anything. The gate's dispatched run enters the e2e workflow's per-ref concurrency group, whose single pending slot the next push to main would cancel it out of — and the latest e2e-test.yml run is usually the still-running push-run of the merge you just made. The gate settles the lane up front — before the node scale-up, the deploy, and the e2e AGC — so a collision costs a wait, not a cluster cycle. It polls for up to E2E_WAIT_TIMEOUT seconds (default 1800), then fails with the run id; E2E_WAIT_TIMEOUT=0 fails immediately instead of waiting.

The gate also checks every local tool it needs up front — including the pinned cosign the final CRD-smoke leg verifies with (make cosign downloads it to .build/cosign; COSIGN=<path> overrides) — so a missing binary fails the run before it spends anything, not 25 minutes in.

Run it detached; the sentinel reports it back

This is the default path. Nobody should spend an hour watching a terminal for a gate built to be walked away from. Launch it as a background task — an agent session's background task, or nohup … & by hand — from a checkout of any post-Q74 ref, with PROJECT/CLUSTER/ZONE/REPO exported (App IDs auto-resolved; the one-time scripts/dogfood/e2e-setup.sh must have run once):

ASSUME_YES=1 PROJECT=… CLUSTER=… ZONE=… REPO=… \
  nohup scripts/dogfood/validate-release.sh vX.Y.Z-rc.N >tmp/validate-release.log 2>&1 &

ASSUME_YES=1 is required when detaching. The gate confirms the resolved target once before it spends anything, and a detached run has no stdin to answer with — without it the gate exits 1 immediately, having done nothing. Leave it off when you run the gate in your own terminal, so the confirmation still gates a fat-finger.

Then launch the sentinel as a second background task. It is what turns a silent hour into a report:

bash scripts/dogfood/release-sentinel.sh

It sleeps, and exits when there is something to say — a phase transition, a verdict, or a gate that has gone quiet for RELEASE_SENTINEL_STALL seconds (default 1200). That exit is the report: it carries the phase, both clocks, the latest e2e heartbeat, and what to do next. Relaunch it after each report until a verdict arrives — in an agent session, the exit is what wakes the session, and the relay-and-relaunch loop is the session's job. Reporting is therefore driven by what the gate does, not by a clock: nothing is spent on an interval where nothing changed. Knobs: RELEASE_SENTINEL_INTERVAL (poll seconds, default 30 — it bounds how quickly a transition is noticed, never how often anything is reported), RELEASE_SENTINEL_TIMEOUT (watch budget, default 7200), RELEASE_SENTINEL_STALL.

The sentinel's exit is a wake, never a verdict — every event exits 0. The verdict is the gate's own exit status, and the failure diagnostics are in the gate's log, not in the report.

Where is it right now?

Reading back an hour of log to answer that is the wrong shape, so the gate keeps its event stream rendered as one object in tmp/release-validation-status.json, rewritten atomically after every phase transition and every relayed e2e heartbeat. This is what the sentinel reads, and it answers the same question directly for a human:

jq . tmp/release-validation-status.json

gate is preflight (settling the e2e lane, nothing spent yet), running, passed, or failed; phase, elapsed, phaseElapsed and idle say where and for how long; heartbeat carries the newest relayed spec line; failure names the phase that broke — the one that broke first, not the teardown that followed it. scripts/dogfood/release-status.sh [stream-file] renders the same object from any stream, including one whose gate process is gone. RELEASE_STATUS_FILE= disables the file.

Underneath it, each phase transition is appended as one JSON line to tmp/release-validation-progress.jsonl — the stream both renderers read, and inspectable directly (tail -f) without disturbing the run. Set RELEASE_PROGRESS_FILE= to disable both files; the gate's own output is unaffected.

Running it in your own terminal instead

Equally supported, and equally legible — drop the nohup/&/ASSUME_YES=1 and answer the confirmation:

PROJECT=… CLUSTER=… ZONE=… REPO=… scripts/dogfood/validate-release.sh vX.Y.Z-rc.N

The gate narrates itself while it runs. Each phase is announced as it starts (==> [e2e] Running the e2e matrix on GAG runners), and the e2e leg — the long one, ~25 minutes while runners autoscale in — relays the dispatched run's own spec heartbeat into your terminal every 30 s:

[e2e t+04:12] 31/73 specs | 29 ok, 1 failed, 1 skipped | running: E2E_GMC_Isolation cross... (3m58s)

The run URL is printed before the watch begins if you would rather follow it in a browser. When the e2e leg finishes — pass or fail — the run's JUnit report is rendered into your terminal: counts, every failing spec with its message, and the ten slowest specs. A red gate names the specs that failed without you having to open the run.

None of that narration is a terminal redraw. The gate's progress output is append-only with no cursor control, by design — which is why a detached run's captured log carries the same phase lines and heartbeats rather than a screenful of escape sequences, and why three identical heartbeat counts in a row are readable as a stall.

The legs the gate runs

What follows documents what validate-release.sh does, and is the recovery path if a leg needs re-running by hand. From a detached checkout of the RC tag (git switch --detach vX.Y.Z-rc.N):

1. Deploy the RC to dogfood. setup.sh needs APP_ID, INSTALLATION_ID, and ASSUME_YES=1 exported alongside GAG_IMAGE_TAG — it reads the GitHub App private key from the macOS keychain (not from an env var), so run it on a macOS host that has that keychain entry. The cluster sits at 0 nodes at rest, so setup.sh's GMC-rollout wait has nothing to schedule on and times out — that is expected; scripts/dogfood/start.sh then scales the system pool to one node and completes the rollout. So: GAG_IMAGE_TAG=vX.Y.Z-rc.N APP_ID=… INSTALLATION_ID=… ASSUME_YES=1 scripts/dogfood/setup.sh (a timed-out rollout wait here is fine), then scripts/dogfood/start.sh. Run the one-time scripts/dogfood/e2e-setup.sh first if the e2e node pool / GitHub App Secret aren't set up yet. The cluster, context pinning, and prod-guard cautions are in gke-dogfood.md.
2. Run the e2e job matrix on GAG runners. This is two moves, not one. scripts/dogfood/e2e-start.sh spins up the on-demand e2e tenant's AGC — it does not start a run and does not touch routing. Trigger the matrix by dispatching a run with routing scoped to it:

gh workflow run e2e-test.yml --ref main -f runner='"gag-ci-e2e"'

(same for e2e-calico.yml). Only that dispatched run lands on the RC's GAG-provisioned runners; every concurrent PR and merge keeps its normal hosted runners. Do not reach for the repo-wide GAG_E2E_RUNNER variable here — flipping it routes every e2e job in the window, and a caught job wedged main CI when the teardown deleted the AGC under it (2026-07-31; the variable remains only as an E2E_ROUTE_VAR=1 opt-in for a standing dogfood soak). Node contention: the on-demand e2e AGC (~500m CPU) does not fit on the single e2-standard-2 system node beside the always-on CI AGCs (the CI AGC goes Pending/Insufficient cpu), so temporarily add a system node (e.g. scale default-pool to 2) for the duration of the e2e leg and scale it back after. Require the matrix green — this is GAG running its own CI end-to-end on the RC images. 3. Smoke the signed v2 CRD asset. Download the RC release's actions-gateway-crds-v2.yaml + .cosign.bundle, cosign verify-blob against the publish identity (step 3 below), kubectl apply --server-side it, and assert the five v2 CRDs register — the helm-free install path operators actually use. 4. Assert the sizing profiles actuated. A profile that silently falls back to Static still provisions a healthy pod and still runs the matrix green, so without this leg every other check reports success while the release's headline feature sits inert. sizing_leg treats the two profiles differently on purpose:

Profile	Tenant	Behaviour
NodeShare	gag-dogfood-e2e	Hard failure. It needs no sample history, so it must report sizingProfileState: Active and derive the envelope's per-worker share. Anything else is a defect.
Throughput	gag-dogfood	Reported, never fatal. It needs ≥20 samples per template container, supplied by the CI tenant's ordinary traffic — not by this gate's ~7-job matrix.
Binpack	—	Not re-asserted; live-validated 2026-07-25.

When Throughput reports NOT VALIDATED THIS RUN, read the state before reaching for the sample count — the two non-Active states are different problems, and the leg prints which one you have:

sizingProfileState	What it means	Fix
empty	spec.sizing is not on the live RunnerSet — a deploy gap, not a sample gap. A CR edit reaches the cluster only through setup.sh's apply_cr or a direct patch; scripts/dogfood/start.sh resizes the pool and routes CI but never applies CRs, so no start can deploy it.	Re-run scripts/dogfood/setup.sh, or kubectl patch runnersets.v2alpha1.actions-gateway.com/ci in gag-dogfood.
AwaitingSamples	The profile is deployed but a template container is below the threshold. Check the sampleCounts the leg prints for which one.	Let ordinary CI traffic run ~20 jobs per container.

Sample history needs no advance planning: the sampler tracks every worker pod regardless of spec.sizing, and the aggregate re-seeds from the persisted status.sizingRecommendation — so samples accrue without the profile configured and survive a stop/start rather than being re-earned.

1. Tear down. scripts/dogfood/e2e-stop.sh, then scripts/dogfood/stop.sh (dogfood scales to 0 at rest).

A red matrix, a failed CRD smoke, or a dead NodeShare profile is a stop-ship for the GA tag: fix forward and cut a new RC — never promote a known-bad RC to a stable tag.

2. Tag and push

git switch main && git pull --ff-only
git tag -a vX.Y.Z -m "Release vX.Y.Z"
git push origin vX.Y.Z

Pushing the tag starts publish.yml. Watch it:

gh run watch "$(gh run list --workflow=publish.yml --branch=vX.Y.Z -L1 --json databaseId -q '.[0].databaseId')"

A workflow_dispatch run with a tag input publishes the same way without a git tag — use it to dry-run the pipeline against a throwaway vX.Y.Z-rc1 tag.

The docs site publishes this release too (Q238). A stable vX.Y.Z tag also triggers pages.yml, which deploys the release's docs as a new version on actions-gateway.com and moves the stable alias + the site's default root redirect to it — so the public docs default to this release, with the unreleased main docs kept behind an opt-in dev version. Prerelease tags (0.x, -rc/-alpha/-beta) do not deploy the site. The versioned-docs model, and the one-time mike seeding of releases cut before it landed, are documented in website.md § Versioned deploy.

3. Verify the publish

Confirm every image and the chart was signed by this workflow before announcing the release. The one-command check uses the pinned cosign (make downloads COSIGN_VERSION — the same version publish.yml signs with — into .build/):

make verify-release VERSION=vX.Y.Z

A broken or missing chart publish now reddens PR CI, not just operators. CI's released-chart upgrade gate (testing.md § The released-chart upgrade gate) discovers the highest stable vX.Y.Z tag on the repo and helm pulls that chart version from GHCR on every e2e run. Pushing a stable tag whose chart-publish job failed therefore fails e2e on every subsequent PR until the publish is repaired (re-run the publish.yml run for the tag) or the tag is removed. Prerelease (-rc) tags are ignored by the gate.

This verifies the five image signatures (gmc, agc, proxy, worker, wrapper) plus the chart (whose tag is X.Y.Z, without the leading v) against the publish workflow's keyless identity. It needs no credentials once the GHCR packages are public. The equivalent explicit commands (and SBOM attestation retrieval) live in security-operations.md § Image provenance; each is a cosign verify --certificate-identity-regexp '…/publish\.yml@refs/tags/v.*$' --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' <ref>.

make verify-release covers the OCI artifacts (images + both charts) but not the GitHub Release's signed v2 CRD manifest asset (Q276), which is a blob signature — verify it against the same identity with verify-blob. Download the manifest and its bundle from the release, then:

cosign verify-blob --bundle actions-gateway-crds-v2.yaml.cosign.bundle \
  --certificate-identity-regexp '^https://github.com/actions-gateway/github-actions-gateway/\.github/workflows/publish\.yml@refs/tags/v.*$' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  actions-gateway-crds-v2.yaml >/dev/null && echo OK

A cosign verify failure is a stop-ship: do not announce the release until it passes. Spot-check one SBOM attestation too so the attestation path is exercised — SBOM attestations are bound to the per-arch manifest digests, not the index, so resolve one first (the full command set is in security-operations.md § Retrieve and inspect the SBOM):

digest="$(docker buildx imagetools inspect ghcr.io/actions-gateway/gmc:vX.Y.Z --raw \
  | jq -r '.manifests[] | select(.platform.os == "linux" and .platform.architecture == "amd64") | .digest')"
cosign verify-attestation --type spdxjson \
  --certificate-identity-regexp '^https://github.com/actions-gateway/github-actions-gateway/\.github/workflows/publish\.yml@refs/tags/v.*$' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  "ghcr.io/actions-gateway/gmc@${digest}" >/dev/null && echo OK

Also spot-check that the index actually carries both platforms (docker buildx imagetools inspect ghcr.io/actions-gateway/gmc:vX.Y.Z should list linux/amd64 and linux/arm64 manifests).

Finally, confirm the build-provenance attestation is present and was minted by this workflow. The attestation binds to the index digest (unlike the per-arch SBOMs), so a tag reference resolves correctly:

# Verifies the signed SLSA provenance against the publish workflow's identity.
gh attestation verify oci://ghcr.io/actions-gateway/gmc:vX.Y.Z \
  --repo actions-gateway/github-actions-gateway \
  --signer-workflow actions-gateway/github-actions-gateway/.github/workflows/publish.yml

Exit 0 alone does not prove this ran. gh attestation verify writes its summary only to a terminal — redirected to a file or captured in a variable it prints nothing, so a real verification and a silent no-op look identical. When you are not reading the output live, ask for something assertable and check it:

gh attestation verify oci://ghcr.io/actions-gateway/gmc:vX.Y.Z \
  --repo actions-gateway/github-actions-gateway \
  --signer-workflow actions-gateway/github-actions-gateway/.github/workflows/publish.yml \
  --format json \
  | jq -r '.[0].verificationResult.signature.certificate
           | "\(.buildSignerURI)\n\(.sourceRepositoryDigest)"'

The workflow URI must end publish.yml@refs/tags/vX.Y.Z for this tag, and the digest must be the commit you tagged. That is the check — not the status.

The equivalent cosign command and the predicate-inspection one-liner are in security-operations.md § Verify build provenance. A provenance verification failure is the same stop-ship signal as a cosign verify failure.

4. Record the published digests

publish.yml writes each image's immutable ghcr.io/.../<name>@sha256:… ref to the run summary (the "Record published digest" step) and into the GitHub Release notes (step 5). These are the multi-arch index digests — the single ref that serves both amd64 and arm64 nodes. Operators pin the workload to the digest (gmc, agc, proxy, worker, wrapper), not the mutable vX.Y.Z tag. You can also resolve a digest directly:

docker buildx imagetools inspect ghcr.io/actions-gateway/gmc:vX.Y.Z \
  --format '{{json .Manifest.Digest}}'

5. Cut the GitHub Release

publish.yml creates the GitHub Release itself (Q293) — no manual step. The chart-publish job's "Compose and create the GitHub Release" step writes the body with the five name@sha256:… index digests, the make verify-release VERSION=vX.Y.Z command, and a generated changelog (previous-tag compare link), and sets --prerelease from the tag (0.x or a -rc/-alpha/-beta suffix ⇒ prerelease; a stable ≥1.0.0 tag ⇒ latest). So the default flow for this step is: nothing — verify the auto-created Release looks right.

The step only creates a Release when the tag has none yet, so it never clobbers curated notes. If you want richer notes (highlights, upgrade caveats), create the Release before pushing the tag — e.g. gh release create vX.Y.Z --draft --notes-file … — and the pipeline will leave your body untouched while still attaching the signed v2 CRD manifest asset.

Writing the curated notes

A minor release accumulates more than a generated changelog can convey, and the two inputs that feed it, operator-caveats-since.sh and the commit log, both mislead in specific ways. What follows is the method, written after v1.3.0.

Author the notes in docs/releases/, not in a scratch file. One file per stable tag, vX.Y.Z.md, holding the release body verbatim — no front matter, and no title heading — the Releases page already renders the tag name as the page's <h1>, so a # vX.Y.Z in the body duplicates it. Publish from it:

gh release edit vX.Y.Z --notes-file docs/releases/vX.Y.Z.md

v1.3.0's notes were drafted under tmp/ and edited straight on the Release. By the time they were right they had been through a wrong count, a dead anchor, 46 forced line breaks, two mismatched PR numbers, and a caveat that never said "GHES" — every one caught by hand, none by review, and none of it reviewable because the text was not in a diff. In-repo makes each fix a diff and each published body reproducible from a commit.

These files are excluded from the docs site on every version, dev included: they are written for github.com's renderer, which the site is not. The exclusion is spelled out in four places that must agree — mkdocs.yml, two env: blocks and one export in pages.yml, and scripts/docs/docs-preview.sh.

Past bodies are retrievable, so the previous release is your template:

gh release view vX.Y.Z --json body --jq .body

Use it to seed a new file when a tag predates this convention.

Notes answer "what is in it" and "what must I do". The docs answer "how" and "why". Every explanation that can live in upgrade.md or an operations/ page should, behind a link. v1.3.0's first draft ran ~1000 words of prose; cutting it to links lost nothing. Link a Highlight from its bold lead, and link group headings rather than every line.

That rule shortens the prose. It does not shorten the notes: v1.3.0 shipped 2100 words and 25 links, because enumerations kept being added — a feature list, a fix list, the API surface, the condition reasons. Prose is what gets cut; lists are what a reader actually searches. Fold the lists (below) and the length costs nothing.

The section skeleton

v1.3.0 arrived at this order after several passes. It is ordered by what a reader needs first, not by what took the most work:

Section	Answers	Notes
(one-line tagline)	what this project is	for the reader who arrived from a search result
(danger banner)	is anything here going to hurt me	a GFM alert, above the fold; see below
Highlights	why upgrade	3–5, each linked from its bold lead
Upgrading	what must I do	numbered steps; say which are guarded
Deprecations	what is going away	see below
Everything since <prev>	did my bug get fixed	folded lists with counts
API and metric surface	what changed in the contract	CRDs and metrics; see below
Validation	why should I believe you	receipts, not adjectives
Project and tooling	is this project healthy	contributor-facing; last for a reason
Security	is there anything I must patch for	state it even when the answer is no
Verifying this release	how do I check the artifacts	the make verify-release line

Say something about security even when there is nothing to report. Silence reads as an omission to the one reader scanning specifically for it. State plainly that no advisory accompanies the release, then list what it does carry — dependency security bumps, and any fix that hardens credential or trust handling without patching a reported vulnerability. Name the scanning gates and when they run. v1.3.0 had no CVE and still warranted the section.

Lead with a danger banner, and make it an alert. GFM alerts — > [!WARNING], > [!CAUTION], > [!NOTE] — render as real coloured callouts in a release body (verified; the render check below counts them). v1.3.0 opens with a [!WARNING] naming the two required upgrade steps and the asymmetric rollback, because those are the only things that can hurt an operator who reads no further. Use [!NOTE] for a scope caveat and [!CAUTION] for the one thing that is genuinely destructive. Three alerts is a lot; more and none of them read as urgent.

Write a Deprecations section even when nothing is removed — saying so is the point, since "deprecation" reads as "removal" to a skimmer. For each notice give the removal version, the migration path, and whether the apiserver actually warns. v1.3.0 deprecated v2alpha1 (warns on every apply) alongside v1alpha1, which is removal-slated and emits no warning at all — so nothing reminds an operator it is going away. That asymmetry is exactly what a reader cannot discover for themselves.

Diff every surface an operator can see, not just the CRDs. Each of these is enumerable, and each hides in a different file, so a review that reads only the Go diff misses most of them. v1.3.0 shipped five: CRD fields, metric names, Kubernetes Event reasons, condition reasons, and configuration (chart values, env tunables, CLI flags). Diff each between the two tags mechanically rather than reading the changelog for them — the Event reasons and the metrics had no enumeration at all until they were diffed, and the notes had already been through several reviews.

Two traps. A rename reads as a removal when the extraction is scoped to one directory: env vars first appeared to have 17 removals, all of which were code moving out of cmd/; re-running repo-wide showed zero. Adjacent string arguments read as the same thing: recordEvent(obj, type, reason, action, …) puts a reason and an action side by side, so ProvisionWorker and ApplyAGCAutoscaler both survived extraction as reasons until each call site was checked. Always report "none removed" when it is true — operators are looking for exactly that.

Diff docs/ as well, and link what is new from where it is actionable. A new operator page is the strongest signal of a capability the notes forgot, and a heavily grown one shows where the release's real weight landed:

git diff --name-status v<prev>..origin/main -- docs/operations/ | grep '^A'
git diff --stat     v<prev>..origin/main -- docs/operations/ | sort -t'|' -k2 -rn | head

v1.3.0 added three operator guides and grew troubleshooting.md by 36 sections. Link a new guide from the bullet it serves rather than from a documentation inventory — resourcequota-sizing.md belongs on the quota-accounting upgrade note, where an operator hits the problem it solves. A guide with no feature to attach to goes in the contributor-facing section with one line on why it exists.

Give the API surface its own section, and lead it with any new CRD. A new kind is not a field: the chart installs chart-root crds/ on a fresh install only, so a new CRD is the reason "apply the CRDs" is step 1 of Upgrading, and the two must cross-reference. Then fold the rest — new spec fields, new status fields, new condition reasons — grouped by kind, counted like any other fold. v1.3.0 listed 28 new condition reasons this way. Until a generated API reference exists, say which artifact is authoritative (kubectl explain, or the signed CRD asset).

Validation is receipts, not adjectives. "Thoroughly tested" is worth nothing. Link the run, quote the counts, and quote a value measured at the layer that matters: v1.3.0 cites 73/73 specs with a run link, and a derived 1500m observed on the pod where the templates asked for 2 and 3 CPU. Ship the receipt wherever a claim is made, not only in this section — a feature line that links its own PR is a receipt too.

Check the validation story against the plan doc, not memory. This is the section a sceptical reader checks first, so a wrong detail here costs more than anywhere else. v1.3.0's draft claimed no candidate had ever cleared the gate and that rc.5 was the first to return a verdict; the plan doc records rc.4 passing the day before. The true version was better anyway — five candidates, three aborted, rc.4 passed without catching a live worker pod, rc.5 caught one — and it is checkable, which the flattering version was not.

Keep a contributor-facing section, and put it last. Release, CI, docs-site, and tooling work ships in no image and no chart, so it does not belong in the change lists. It still belongs in the notes: it is what a reader evaluating the project's health is looking for. Fold it, label it as not user-facing, and let it sit below everything an operator needs.

The caveats script reports headings that changed, not headings that are new. A section edited in this window is listed exactly like one added in it. Test each before repeating it:

git show <prev-tag>:docs/operations/upgrade.md | grep -qF "### <heading>" \
  && echo "pre-existing" || echo "new in this window"

v1.3.0 listed two BREAKING headings and neither was a caveat for an upgrading operator: priorityTiers was already in v1.2.0 (a pre-1.0 change), and capacityGate.mode's removed values had only ever existed on main. Repeating them unexamined would have sent operators after migrations they did not need.

Promote by danger, not by label. The most hazardous item in v1.3.0 carried no BREAKING heading at all: the PriorityClass allowlist CRD apply, which affects every install and whose rollback re-arms a cluster-wide outage. Read for consequence, not for keywords.

Distinguish "breaking" from "guarded migration". If skipping a step stops the upgrade with a message naming the fix, it is a required migration and saying "breaking change" overstates it. If a wrong path fails silently, say so loudly. State which of the two you mean.

Enumerate from the commit log, filtered to what ships. Conventional Commit subjects are already terse diagnoses, so they need only the prefix and Q-ID removed:

git log --format='%s' <prev-tag>..HEAD \
  | grep -E '^(feat|fix)\((agc|gmc|proxy|worker|api|scaleset|chart|broker|wrapper|admission|provisioner|metrics|scalesetlistener|migrate|observability)\)'

That scope list is a guess, so reconcile what it dropped — never just run it. An allow-list silently omits any scope nobody thought of, and the result still looks like a complete list. Print the residue and read every scope in it:

git log --format='%s' <prev-tag>..HEAD | grep -E '^(feat|fix)' > /tmp/all
# ...run the filter above into /tmp/kept, then:
grep -vxF -f /tmp/kept /tmp/all | sed -E 's/^((feat|fix)\([^)]*\)).*/\1/' | sort | uniq -c | sort -rn

v1.3.0's first pass kept 57 of 132 and dropped seven shipping fixes: two fix(scalesetlistener) (the pattern had scaleset, which does not match it), a compound fix(agc,gmc), three fix(migrate) for the shipped gag-migrate binary, and one fix(observability). Nothing in the output said so. The last three scopes above were added only after that reconciliation — assume the list is still incomplete for the next release.

Exclude the ci, dogfood, test, docs, and build scopes, which ship in no image or chart. Say in the notes that you excluded them, so a reader does not read hundreds of commits as the user-visible change count. Keep the trailing (#NNNN): GitHub auto-links a bare #NNNN in a release body, so every line becomes traceable for free.

Passing the scope filter does not make a commit shippable — check the paths. The filter matches a scope string, and scopes are reused. v1.3.0 listed "Attribute usage rows to the machine that measured them" as a feature; the commit is feat(metrics) and touches only claude-usage/, the Claude Code usage tooling, because that module and the product's Prometheus metrics share a scope. Assert the paths for every cited PR instead of reading the subjects:

git show --stat --format='' --name-only <sha> \
  | grep -qE '^(api|broker|cmd|githubapp|scaleset|charts)/' || echo "NOT PRODUCT"

It was one entry in 64, and no amount of re-reading the list would have found it.

Cite the commit that did the work, not the one that filed it. A Q-numbered backlog row and its implementation have near-identical subjects, so a docs(plan) commit reads exactly like the fix. v1.3.0 cited #988 — docs(plan): file and scope Q507 — under the label of #1008, the gate itself; a reader clicking through would have landed on a planning row. Resolve each number to its title before shipping, and look for the same work cited twice under two numbers.

One fact per line, especially next to a procedure. Distinct operator-facing changes run together into a paragraph read as background, and a paragraph sitting under a numbered list reads as a footnote to it. v1.3.0's Upgrading section closed with three unrelated changes — quota accounting, a dropped proxy label, a new apiserver warning — in one sentence-run below its two numbered steps; as a bulleted list with a bold lead each, the same words are scannable. Prose is for framing. Anything a reader might need to act on individually gets its own line.

Fold long lists. <details><summary> renders on the Releases page and keeps the top scannable. It is also the only lever against truncation: the Releases index collapses a long body behind a "read more" link, and a fold counts as its one summary line while collapsed. v1.3.0 hit that limit and was folded back under it — eight folds. If the index is truncating, the fix is another fold, not a cut.

Pick the next fold by measuring collapsed height, not by eye. Sum each section's bytes with <details>…</details> bodies excluded; the biggest sections are rarely the ones that feel long. v1.3.0 measured 9.5k collapsed, and the third-largest section was Validation — not an obvious candidate, since Upgrading and Highlights are larger but must both stay open.

When the content being folded is evidence, put the evidence in the <summary>. A fold whose summary reads "Validation details" hides the receipt; one that reads 73/73 e2e specs on Kata microVM workers, on live GKE — the four legs, and what none of them assert is the receipt, and a reader who never expands it has still seen the number. That is the exception to the count-in-the-summary convention: enumerations carry a count, evidence carries the finding.

Count what you list — and count the unit in the label. State a count in a <summary> and it will be wrong the moment you curate the list. v1.3.0's draft claimed 25 features and listed 23. Subtler: its "New spec fields (10)" had ten bullets carrying thirteen fields, because three bullets grouped related ones (.minRequests / .maxRequests / .limitHeadroomPercent). Every other fold counted the noun in its own label; that one silently switched to bullets. Count what the label says, then re-count after every edit — mechanically, not by eye.

Caveat anything a validation run did not exercise. The dogfood gate runs against github.com, so it says nothing about GHES. A feature list that reads as finished support overclaims. Check docs/plan/archive/ for the feature's own "what this will not verify" section before describing it. Then ask whether an unexercised feature belongs in Highlights at all — v1.3.0 kept GHES there and paid for it with a caveat in three places.

A caveat is a claim, so measure it before writing it. Understating coverage is as wrong as overstating it, and easier to do accidentally because it feels safe. v1.3.0's draft said the capacity gate had "unit and envtest coverage only"; the repo has a live-cluster-autoscaler test for its matcher (#929) and 305 lines of e2e proving a quota-blocked job redelivers (#1028). Before writing "only tested at tier X", grep for the feature at every higher tier — and if the true statement is narrower than the tidy one, ship the narrow statement. What survived here was "the release gate does not assert it", which is checkable.

A caveat must survive being read alone. Every line in a folded list is read out of its heading's context — by search, by a linked anchor, by a skimmer. The v1.3.0 draft said "Untested against a real appliance" under a GHES heading, which says nothing at all once the heading scrolls away; it shipped as "Untested against a real GHES appliance". Name the subject inside the caveat, and repeat the caveat at each place the feature is claimed rather than relying on proximity.

Link the versioned docs site, not main and not blob/. A reader of these notes should land on that release's instructions, and the site publishes a build per stable tag: https://actions-gateway.com/X.Y.Z/operations/…. Mind the form — the site drops the leading v exactly like the chart does. v1.3.0 shipped 18 such links. They 404 until the docs deploy for the tag completes, which is expected while the Release is still a draft.

Verify every link and anchor. Nothing checks them for you: make doc-links skips external URLs by design, and every link in a notes file is absolute. Anchors are the usual failure, and the built site is the authoritative oracle — read the ids MkDocs actually emitted rather than re-deriving a slug by hand, which gets punctuation, backticks, and parenthesised clauses wrong:

make docs-build
grep -oE 'id="[^"]*"' site/operations/upgrade/index.html | sed 's/id="//;s/"//'

Better, resolve every link in the file against site/ in one pass. Whatever you write, include a URL you know is broken and confirm it is reported: a checker that silently resolves nothing looks exactly like a clean file. v1.3.0's notes were verified this way — 20 site URLs resolved, plus one deliberately bogus anchor that the same run flagged.

Do not hard-wrap. GitHub renders a release body with comment-flavour GFM, where a single newline becomes <br>. Keep every paragraph, blockquote, and list continuation on one line. v1.3.0's hard-wrapped draft rendered 46 of them.

Check this against the renderer, not the source. gh release view --json body returns the raw Markdown, which never contains <br> however badly it is wrapped, so grepping that is a check that cannot fail. Render it the way GitHub will:

gh api -X POST /markdown -f mode=gfm -f "text=$(cat docs/releases/vX.Y.Z.md)" \
  | grep -c '<br>'

mode=gfm is the comment flavour; mode=markdown is not, and reports 0 on a hard-wrapped file. The same render confirms the rest of the GitHub-only markup survived — expect one markdown-alert-* class per > [!…] block, one <details> per fold, and no literal [! anywhere.

In-page anchors do not work in a release body. Release-body headings carry no id, so [Upgrading](#upgrading) is a dead link. Refer to a section by name in bold instead. Verify on a published release rather than trusting this — the page does emit ids, but only on GitHub's own chrome:

curl -sS https://github.com/<owner>/<repo>/releases/tag/<tag> \
  | grep -oE '<h[1-6][^>]*>' | grep -c 'id='

So a table of contents is not available, and should not be faked. An unlinked list of section names is dead weight that costs collapsed height against the truncation limit while navigating nothing. The folds already serve that role: a collapsed <details> is a labelled one-line entry, so a body with ten folds reads as an outline whether or not the reader expands any of them. Navigation comes from section order and the danger banner, not from a ToC. (The in-repo copy under docs/releases/ does get GitHub's auto-generated file outline for free, which is a second reason not to hand-roll one.)

Watch the chart-version form. Images are tagged vX.Y.Z, charts X.Y.Z. A copy-pasteable helm command with a v in it fails.

Run the deslop skill over the draft before publishing. Release notes are the most-read prose the project ships.

Before publishing: the mechanical checks

Every rule above that can be checked by a machine, in the order they are cheapest to run. None of these is a substitute for reading the notes — but each one caught a defect in v1.3.0 that several careful readings had not.

Check	How	What it catches
Fold counts	recount every <summary> against its bullets, counting the noun the label names	a count that drifted during curation
Enumerations	reconcile each surface fold against the tag-to-tag diff both ways	a name listed that no longer exists, or shipped and never listed
Citations resolve	gh pr view <n> --json title for each #NNNN	a planning commit cited as the implementation
Citations ship	git show --stat --name-only <sha> against product paths	a non-product commit listed as a feature
Anchors	resolve every site URL against a built site/, with one planted bad anchor	a heading that moved, and a checker that silently resolves nothing
Rendering	gh api -X POST /markdown -f mode=gfm	hard-wrapped <br>s, alerts that did not render, literal [!
Published body	re-fetch and diff against the file	an edit made on the Release and not in the repo

Two habits make the difference. Plant a known failure in anything that reports "all clear" — a checker with a broken query and a clean file produce identical output. And report the negative when it is true: "16 metrics added, none removed" and "13 spec fields, nothing removed" are what an operator is actually scanning for, and neither is worth stating unless it was measured.

6. Chart version & metadata

The chart-publish job sets the published chart's version and appVersion to the release tag (with the leading v stripped, since chart SemVer forbids it), so there is no manual Chart.yaml version bump to remember — the in-repo version/appVersion are dev placeholders the pipeline overrides at package time. The prerelease annotation is likewise derived from the tag now, so nothing here needs a hand-flip. The remaining items below are one-time setup or guardrails, not per-release steps:

• Prerelease annotation — derived, not hand-flipped (Q293). Chart.yaml carries artifacthub.io/prerelease, but its committed value is a dev placeholder: publish.yml overrides it with yq before helm package, setting "true" for a 0.x or -rc/-alpha/-beta tag and "false" for a stable ≥1.0.0 tag (same test that sets the Release's --prerelease flag). There is no flip PR to land before or after a cut. The v2 CRD chart is stamped the same way.
• Artifact Hub listing. Discoverability metadata (description, keywords, prerelease flag) ships in the chart's own annotations. Ownership verification uses artifacthub-repo.yml at the repo root — register the OCI repository in the Artifact Hub control panel, copy the assigned repositoryID into that file, and push it to the registry as the repository-metadata OCI artifact (the file's header documents the exact steps). This is a one-time control-panel action, not part of publish.yml.
• Empty values.yaml digests. Do not commit real sha256:… digests into values.yaml. The empty digest fields are the secure default: an unconfigured install fails closed (the GMC rejects floating AGC/proxy tags at startup) until the operator pins a real digest at install time. Baking a digest into the shipped chart would defeat that fail-closed posture and immediately go stale. The published digests belong in the release notes (step 5), which is where the operator copies them from.

7. Hand off to operators

Operators install/upgrade straight from the published OCI chart with the digests pinned via --set, exactly as install.md § Pin images by digest and upgrade.md document (X.Y.Z is the release tag without the v):

helm install gag oci://ghcr.io/actions-gateway/charts/actions-gateway --version X.Y.Z \
  --set gmc.image.digest=sha256:<gmc> \
  --set agc.image.digest=sha256:<agc> \
  --set proxy.image.digest=sha256:<proxy> \
  --set wrapper.image.digest=sha256:<wrapper>

Patch releases and backports

A patch release (vX.Y.Z+1) is bugfix-only by SemVer. That has a release- engineering consequence: do not tag a patch from main once main has merged features headed for the next minor — doing so ships those unreleased features in the patch's images and chart, and advertises them in the patch's docs (the site builds each version from its tag). Tag a patch from the released line instead:

• Ephemeral branch off the tag (branchless-friendly). For a one-off patch: git switch --detach vX.Y.Z, git switch -c release-X.Y, cherry-pick the fix, tag vX.Y.(Z+1), push the tag. Delete the branch afterward if you don't maintain the line.
• Long-lived release branch. If you support multiple minor lines at once, keep a release-X.Y branch at the minor's tag and land backported fixes on it.

You only need a branch/backport when main isn't itself the intended patch — i.e. when it has diverged past the release with content you don't want in the patch. If main is clean and ready to ship, that's the next minor (vX.(Y+1).0), not a patch.

A release line carries its own validation harness. validate-release.sh and the dogfood scripts under it live in this repo, and the gate runs the scripts in your checkout against the artifacts of the tag. A release-X.Y branch cut from vX.Y.0 therefore inherits whatever harness that commit had — so a fix to the gate only reaches a patch line if it was in the minor tag, or is cherry-picked like any other fix.

The consequence is a rule for the minor cut, not the patch: tag vX.Y.0 from a commit that carries the harness fixes you want the line to keep. Cutting it from an older RC's commit to make the tag byte-match the validated artifact is the trap — it strands every gate fix found during that release's validation, which is exactly when gate fixes tend to be found. v1.3.0 is the worked example: rc.5's validation produced two harness fixes (Q627 and the e2e watch deadline), and a release-1.3 branch cut from rc.5's commit would reproduce both on every future v1.3.x.

Validating an RC whose product code matches main is unaffected: the images and CRDs come from the tag, the harness from your checkout, so the two move independently by design.

The docs site follows automatically: mike builds each version from its own tag, so a patch tagged off the release line publishes docs with only that line's content — no unreleased features. And the site's stable alias + default root redirect move only to the highest released version, so a backport patch released after a newer minor updates its own version's docs without demoting the site (see website.md § Versioned deploy). No docs-specific branch is ever required beyond what the release itself needs.

Rollback

A release is just a tag and a set of immutable, digest-addressed images — nothing is destructive. To roll an installed release back, re-pin the previous digests and helm rollback/helm upgrade; the procedure and post-rollback validation are in upgrade.md. A bad tag can be superseded by a higher patch release; do not retag an existing vX.Y.Z (it would break the digest↔tag binding consumers rely on).

The worker images: wrapper and worker

publish.yml builds and signs two worker-related images, both holding the same cmd/worker wrapper that feeds the job payload into Runner.Worker:

• ghcr.io/actions-gateway/wrapper — a ~2 MB FROM scratch image with just the wrapper binary. The GMC forwards it to every AGC (WRAPPER_IMAGE), whose provisioner injects it into each worker pod — as a read-only OCI image volume (K8s ≥ 1.33) or via an initContainer below that — so the runner container can be the unmodified upstream ghcr.io/actions/actions-runner (or any tenant workerImage). This is what makes DefaultWorkerImage (still the digest-pinned upstream actions-runner, runner version locked to agent.version — see building.md) actually run jobs (Q235, plan).
• ghcr.io/actions-gateway/worker — the full upstream actions-runner + the wrapper as ENTRYPOINT (~520 MB). Kept as an optional batteries-included image; unnecessary once injection is enabled, since the runner image is the upstream one with the wrapper injected. Retiring it is tracked separately.

Only wrapper is digest-pinned in the chart (wrapper.image.digest, like agc/proxy), so a release must publish the wrapper image and pin its digest for the default install to run jobs. The worker image has no chart image block — it is the optional batteries-included image a tenant opts into via its per-RunnerGroup workerImage, not a chart-provisioned one — so nothing in the chart pins it.

PR CI vs publish — what runs where

Stage	Build image	Generate SBOM	Push to GHCR	Sign + SBOM-attest	Provenance attest
Pull request (security-scan.yml)	✅	✅ (artifact)	—	—	—
Release tag (publish.yml)	✅	✅ (attached)	✅	✅ keyless	✅ keyless (SLSA L2)

PR CI proves the image builds and the SBOM generates so those paths can't silently break; signing, SBOM attestation, and build-provenance attestation are all first exercised on a real v* tag, which is why step 3 verification matters on every release.

publish.yml also runs one pre-publish gate, announce-bar, that every publishing job depends on. It builds the docs site at the tag and asserts the rendered banner names it (see Pre-flight), so a docs-site banner advertising the wrong version stops the release before an image, chart, or GitHub Release exists, rather than after.

Supply-chain integrity of the pipeline itself

The publish job holds packages: write + id-token: write + attestations: write: its ambient OIDC identity is the release trust root. A hijacked upstream action tag executing in that job could push and keyless-sign malicious images as the legitimate publish identity. Three controls keep the pipeline itself trustworthy.

Actions are pinned to full commit SHAs

Every uses: across .github/workflows/ is pinned to a full 40-char commit SHA with a trailing # vX.Y.Z comment for readability — never a floating tag (@v4) or branch. A tag is mutable: whoever controls the upstream repo can repoint it at new code, which would then run inside the privileged publish job. A SHA is immutable. The runtime tool downloads in the publish path are pinned the same way: cosign via sigstore/cosign-installer with an explicit cosign-release (kept in step with COSIGN_VERSION in the Makefile so a local make verify-release uses the same version that signed), and syft via the syft-version input on anchore/sbom-action/download-syft (the action is SHA-pinned, but syft itself is a runtime download).

Bumping a pinned action. Dependabot's github-actions ecosystem (.github/dependabot.yml) opens weekly PRs that bump both the SHA and the # vX.Y.Z comment, so the pins don't rot — review and merge those like any dependency PR. To pin or bump by hand, resolve the tag to its commit SHA and keep the comment in sync:

gh api repos/<owner>/<action>/commits/<tag> --jq '.sha'
# -> uses: <owner>/<action>@<sha> # <tag>

syft-version is not Dependabot-managed (it's a tool download, not an action ref) — bump it by hand in publish.yml when you bump the anchore/sbom-action SHA. actionlint (CI lint job) keeps SHA-pinned uses: lint-clean.

Signing identity is tags-only

Releases are cut by pushing a v* tag, so a legitimate keyless signature's Fulcio certificate records publish.yml running from refs/tags/vX.Y.Z. Two layers enforce that a signature can only ever be a tag signature:

• publish.yml refuses to run from a non-tag ref. Both publish jobs' "Resolve publish tag" step rejects any GITHUB_REF that isn't refs/tags/…, so a workflow_dispatch run from a branch can't even reach the sign step.
• make verify-release only accepts a tags identity. The --certificate-identity-regexp is anchored to …/publish\.yml@refs/tags/v.*$ (sourced from release_identity_regexp in scripts/lib/common.sh), so a signature minted from refs/heads/… is rejected even if one were somehow produced. The scripts/release/verify-release-test.sh assertions (run by make check and CI) guard that the regexp stays tags-only.

Together these close the hole where repo-write could dispatch publish.yml from a scratch branch, overwrite a released GHCR version tag, and still pass verification.

Build inputs and the signer binary are integrity-checked

The first two controls protect who runs the pipeline and how signatures are trusted; this one protects what goes into the signed artifacts and the tool that verifies them.

• Vendored dependencies are gated against go.sum. Images build with go build -mod=vendor, but -mod=vendor only checks vendor/modules.txt consistency — it never verifies that the vendored source matches the hashes in go.sum. A malicious or accidental edit under vendor/ (or tools/vendor/) would otherwise compile straight into the signed release images. The vendor-check job (in unit-test.yml, single source of truth make vendor-check → scripts/go/vendor-check.sh) re-runs the workspace-vendor flow — which re-fetches every module verified against go.sum — and fails on any diff against the committed trees. A Dependabot go.mod bump legitimately fails this gate until a follow-up vendor sync lands; that is the intended signal (see go-workspaces.md § Changing dependencies).
• The cosign verify binary is checksummed. GitHub release assets are mutable for an existing tag, so a raw download of the release verifier can't be trusted on its own. The publish pipeline obtains cosign via the SHA-pinned sigstore/cosign-installer action (which performs its own signature verification); the local verify path (make verify-release → scripts/release/download-cosign.sh) pins the expected SHA256 per platform in-repo and refuses to install a binary whose bytes don't match. Bumping COSIGN_VERSION must add the new digests to that script (it fails closed on an unpinned version) — the same deliberate-pin discipline as KIND_BINARY_SHA256 in e2e-test.yml.