Skip to content

Troubleshooting Guide

Audience: SRE, Platform engineer

Each section below covers a specific failure mode: symptoms, likely cause, diagnostic commands, and resolution steps.

Table of Contents

How to Validate a Fresh Deployment

Run these checks immediately after deploying a new tenant gateway or upgrading existing components.

# 1. Check ActionsGateway status
kubectl get actionsgateway -n <namespace> -o yaml | grep -A 20 status:

# 2. Confirm the AGC pod is running
kubectl get deploy -n <namespace> actions-gateway-controller
kubectl logs -n <namespace> deploy/actions-gateway-controller --tail=50

# 3. Confirm the proxy pool is healthy
kubectl get deploy -n <namespace> actions-gateway-proxy
kubectl get hpa -n <namespace>

# 4. Confirm RunnerGroup resources exist
kubectl get runnergroup -n <namespace>

# 5. Check for condition errors
kubectl get actionsgateway -n <namespace> -o jsonpath='{.status.conditions}' | jq .

Expected state after a healthy deployment:

  • ActionsGateway condition Ready=True.
  • ActionsGateway condition AGCAvailable=True.
  • ActionsGateway condition ProxyAvailable=True.
  • AGC Deployment: READY 1/1.
  • Proxy Deployment: READY count >= minReplicas.
  • HPA: TARGETS showing a CPU percentage (not <unknown>).
  • Each RunnerGroup has at least one listener session (actions_gateway_active_sessions > 0).

Helm Render Fails: gmc.image Must Be Pinned by Digest

Symptoms. helm install, helm upgrade, helm lint, or helm template of the actions-gateway chart fails immediately with (here for gmc; the message names whichever image is unpinned — gmc, agc, proxy, or wrapper):

Error: execution error at (actions-gateway/templates/deployment.yaml:...):
gmc.image must be pinned by digest: set gmc.image.digest=sha256:<64 hex digits>
(see docs/operations/install.md, "Pin images by digest").
DEV/TEST ONLY: set allowFloatingImageTags=true to allow a floating tag.

Cause. One of the four image digests (gmc, agc, proxy, or wrapper) is empty in the release values. The chart enforces digest pinning of all four images at render time (secure by default): an empty digest must never silently fall back to a mutable :latest tag — for the GMC's own image nothing at runtime validates it, and for the AGC/proxy/wrapper images an unpinned tag would otherwise only surface later as a GMC crash-loop. Common ways to get here:

  • A fresh install without --set <image>.image.digest=sha256:<…> for one of the four (a forgotten wrapper.image.digest is the most common).
  • A helm upgrade with a values file (or --reset-values) that omits a digest. (--reuse-values carries the previously pinned digests forward.)
  • Offline rendering (helm template / helm lint) without supplying all four digests.

Resolution.

  • Production: pin the digest published for the release you are installing (see release.md for where digests are recorded):
helm upgrade --install gag charts/actions-gateway \
  --namespace gmc-system \
  --set gmc.image.digest=sha256:<gmc> \
  --set agc.image.digest=sha256:<agc> \
  --set proxy.image.digest=sha256:<proxy> \
  --set wrapper.image.digest=sha256:<wrapper>
  • Dev/test only: --set allowFloatingImageTags=true lets all four images render from a floating tag and disables the GMC's startup digest check on the AGC/proxy/wrapper images. Never use it in production.
  • Offline rendering: any well-formed digest satisfies the check, e.g. --set-string gmc.image.digest=sha256:1111111111111111111111111111111111111111111111111111111111111111 (repeat for agc/proxy/wrapper).

All four images are validated by the chart at render time. The AGC/proxy/wrapper images are additionally re-checked by the GMC at startup (a floating tag there crash-loops the GMC — see install.md § Pin images by digest), a second layer that also covers non-chart deployments.

GMC Pods Rejected: insufficient quota to match these scopes (PriorityClass)

Symptoms. After helm install, the GMC Deployment never reaches Ready. There are no GMC pods, and the ReplicaSet emits a FailedCreate event:

kubectl describe replicaset -n gmc-system -l app.kubernetes.io/name=gmc
# Events:
#   Warning  FailedCreate  ...  Error creating: pods "gmc-controller-manager-..." is forbidden:
#   insufficient quota to match these scopes:
#   [{PriorityClass In [system-node-critical system-cluster-critical]}]

Cause. The cluster's API server enables the restricted PriorityClass admission config (GKE Standard does this by default), which permits system-node-critical / system-cluster-critical pods only in a namespace carrying a ResourceQuota whose scopeSelector matches those classes. The GMC runs with priorityClassName: system-cluster-critical by default — a deliberate secure default that protects the control plane from eviction — so without a permitting quota in the install namespace, the apiserver rejects every GMC pod.

Resolution. The chart ships the permitting quota by default, so this should not occur on a current chart. If you hit it:

  • Confirm the quota is enabled. The chart renders <namePrefix>-critical-pods (default gmc-critical-pods) when systemCriticalPriorityQuota.enabled=true (the default) and priorityClassName is a system-critical class:
kubectl get resourcequota -n gmc-system gmc-critical-pods

If it is missing, you likely installed with --set systemCriticalPriorityQuota.enabled=false. Re-run the install/upgrade without that override (it defaults to true). See install.md § GKE and other restricted-PriorityClass clusters. - Do not work around the rejection by clearing priorityClassName — that removes the GMC's eviction protection (a security regression). Keep system-cluster-critical and let the quota permit it. - If you manage the quota out-of-band (e.g. a cluster-wide policy), ensure it exists in the install namespace and its scopeSelector matches the system-critical classes before installing.

GMC Not Provisioning Tenant Resources

Symptoms. An ActionsGateway CR was applied but nothing has been created in the tenant namespace: no AGC Deployment, no proxy Deployment, no RunnerGroup resources.

Likely causes. - GMC pod is not running or not the leader. - GMC lacks permission to write to the tenant namespace (RBAC misconfiguration during initial GMC install). - The ActionsGateway CR failed admission validation (check for validation errors in kubectl apply output or Events).

Diagnostics.

# Check whether the GMC is running and has a leader
kubectl get lease -n gmc-system
kubectl get pods -n gmc-system

# Check GMC logs for reconcile errors
kubectl logs -n gmc-system deploy/gmc-controller-manager --tail=100 | grep -i error

# Check events on the ActionsGateway CR
kubectl describe actionsgateway -n <namespace> <name>

# Check the conditions — Degraded names the failing provisioning step
kubectl get actionsgateway -n <namespace> <name> -o jsonpath='{.status.conditions}' | jq .

Reading the Degraded condition. When a reconcile fails partway through provisioning, the GMC sets Degraded=True (reason ProvisioningFailed) on the ActionsGateway and names the failing step in the message — e.g. provisioning failed at step "proxy Deployment + Service": .... The reconcile returns immediately on that error, so the other conditions (ProxyAvailable, AGCAvailable) may be stale; Degraded is the authoritative signal of which step is stuck. It clears (Degraded=False, reason ReconcileSucceeded) automatically once a reconcile completes all steps. Read it directly:

kubectl get actionsgateway -n <namespace> <name> \
  -o jsonpath='{range .status.conditions[?(@.type=="Degraded")]}{.status} {.reason}: {.message}{"\n"}{end}'

Resolution. - If the GMC pod is not running, restore it from its Deployment. - If RBAC is missing, re-run helm upgrade --install of the chart (RBAC ships with it). - If the admission webhook is rejecting the CR, fix the CR spec and re-apply. - If Degraded=True, fix the underlying problem named by the failing step (e.g. a conflicting hand-created resource, missing permission, or exhausted quota) — also cross-check the controller_runtime_reconcile_errors_total metric and the full error in the GMC logs. The GMC's reconciler retries with backoff and clears Degraded on the next successful reconcile.

ActionsGateway Reports RunnerGroupsDegraded

Symptoms. kubectl get actionsgateway shows a RunnerGroupsDegraded=True condition, or the actions_gateway_runnergroups_degraded gauge is 1. The gateway infrastructure itself (proxy, AGC) may still be Ready=True — this condition rolls child RunnerGroup health up to the gateway so you don't have to inspect each group individually.

Cause. One or more of the gateway's owned RunnerGroups reports an impairing condition — CredentialUnavailable (the AGC can't obtain an installation token), Degraded (an unhealthy/unauthorized listener session), RunnerVersionTooOld (GitHub rejects the configured runner version), or WorkersUnschedulable (worker pods can't be scheduled). Advisory capacity/throughput conditions (WorkerQuotaPressure/WorkerQuotaExceeded, RateLimited) are deliberately not rolled up here — they have their own signals. RunnerGroupsDegraded does not gate Ready: the gateway can keep serving healthy groups while one is impaired.

Diagnostics.

# Read the rollup — its message names the impaired groups and their tripped conditions.
kubectl get actionsgateway -n <namespace> <name> \
  -o jsonpath='{range .status.conditions[?(@.type=="RunnerGroupsDegraded")]}{.status} {.reason}: {.message}{"\n"}{end}'

# Drill into a named RunnerGroup's own conditions.
kubectl get runnergroup -n <namespace> <runner-group> -o jsonpath='{.status.conditions}' | jq .

Resolution. Resolve the underlying per-group condition, then the rollup clears automatically on the next reconcile (the GMC watches the owned RunnerGroups): - CredentialUnavailable → see GitHub App Secret Misconfiguration. - Degraded / RunnerVersionTooOld → see AGC CrashLoopBackOff or Not Acquiring Jobs. - WorkersUnschedulable → see RunnerGroup Reports WorkersUnschedulable.

v2 ActionsGateway: RunnerSetsDegraded. The v2 ActionsGateway reports the same rollup as RunnerSetsDegraded (Q304), rolling child RunnerSet health up to the gateway. A set counts as impaired on either of two axes: it is not Ready for a non-transient reason — a reference or provisioning failure, which a v2 RunnerSet folds into Ready=False (anything but the benign startup NoActiveSessions) — or any of its abnormal-is-True impairing conditions is True: Degraded (revoked or invalid credentials), CredentialUnavailable, RunnerVersionTooOld, or WorkersUnschedulable. The second axis matters because the shared listener pushes Degraded/RunnerVersionTooOld independently of Ready (Q330): a classic set whose sessions are all rejected as unauthorized converges to the benign Ready=NoActiveSessions while Degraded=True sits on its own condition. Advisory throughput signals (RateLimited, the WorkerQuota ladder) are deliberately excluded so the rollup does not flap on normal load. The message names the impaired sets and their tripped signals; a set targeting a different gateway is never counted. It is advisory (does not gate Ready) and clears automatically once the children recover (the GMC watches bound RunnerSets). Read it with:

kubectl get actionsgateway -n <namespace> <name> \
  -o jsonpath='{range .status.conditions[?(@.type=="RunnerSetsDegraded")]}{.status} {.reason}: {.message}{"\n"}{end}'

Runners Never Appear Online — AGC unknown authority Through the Egress Proxy

Symptoms. A freshly onboarded tenant reaches Ready=True with the proxy (PROXYREADY) and AGC pods running, but no runner ever appears in the repo/org runner list, the RunnerGroup shows ActiveSessions empty/0, the gateway reports RunnerGroupsDegraded, and the AGC log repeats:

EnsureAgents failed ... register agent: generate jit config:
Post "https://api.github.com/.../actions/runners/generate-jitconfig":
proxyconnect tcp: tls: failed to verify certificate:
x509: certificate signed by unknown authority

The installation-token fetch succeeds (initial token acquired appears in the log just before the failures), so this is specific to agent registration, not credentials — distinguishing it from GitHub App Secret Misconfiguration.

Cause. The AGC routes its own GitHub API traffic through the per-tenant egress proxy, whose serving certificate is the GMC's self-signed per-tenant CA. In v1.1.0-rc.4 and earlier, the AGC's runner-registration HTTP client was constructed before that proxy CA was added to the process trust store, so it trusted only the system roots and could not complete the TLS handshake to the proxy. It affects any proxied AGC — every v1alpha1 ActionsGateway, and a v2 RunnerSet whenever an EgressProxy is attached. Direct-egress (no proxy) tenants are unaffected (which is why it does not reproduce without a proxy).

Resolution. Upgrade the AGC image to a release after v1.1.0-rc.4, where the registration client is built lazily — after the proxy CA is trusted. There is no clean configuration workaround on rc.4 for a proxied tenant; switching the tenant to direct egress sidesteps the bug but gives up the proxy's egress isolation, so prefer the upgrade.

RunnerGroup Reports WorkersUnschedulable

Symptoms. kubectl get runnergroup shows a WorkersUnschedulable=True condition, or the actions_gateway_workers_unschedulable gauge is 1. Jobs are acquired but never start; worker pods sit Pending. Each pod the reaper eventually gives up on also emits a WorkersUnschedulable Warning event and a WorkerPodStuckPending event on the RunnerGroup.

Cause. The Kubernetes scheduler cannot place the group's worker pods on any node — PodScheduled=False with reason Unschedulable. Typical reasons: no node has enough allocatable CPU/memory for the pod's requests, the pod's nodeSelector / affinity matches no node, or every candidate node carries a taint the pod does not tolerate. The condition trips once a pod has been Pending+Unschedulable for longer than the scheduling grace (half the group's pendingPodDeadline), giving an early warning before the reaper deletes the pod at the full deadline.

This is not a quota problem: a ResourceQuota rejection blocks pod admission so the pod is never created — that path is the separate WorkerQuotaExceeded condition. The two never both fire for the same cause.

v2 RunnerSet. The same WorkersUnschedulable condition is set on a v2 RunnerSet (Q303) with identical semantics — swap runnergroup for runnerset in the commands below. (The actions_gateway_workers_unschedulable gauge is still emitted only for v1 RunnerGroups; on a RunnerSet, read the condition directly.)

Diagnostics.

# Read the condition — its message names the stuck pods and the scheduler verdict.
kubectl get runnergroup -n <namespace> <runner-group> \
  -o jsonpath='{range .status.conditions[?(@.type=="WorkersUnschedulable")]}{.status} {.reason}: {.message}{"\n"}{end}'

# Inspect a stuck worker pod's scheduler events for the exact reason.
kubectl describe pod -n <namespace> <worker-pod>   # look for "FailedScheduling"

Resolution. Match the scheduler verdict to the fix: - Insufficient cpu/memory → add nodes / scale the cluster autoscaler, or lower the worker pod's resource requests in the group's podTemplate. - node(s) didn't match nodeSelector / affinity → correct the podTemplate's nodeSelector/affinity, or label the intended nodes. - node(s) had untolerated taint → add the matching toleration to the podTemplate, or remove the taint from the target nodes.

The condition clears automatically on the next reconcile once a worker pod schedules successfully.

Worker / Proxy / AGC Pods Rejected by a Cluster Policy Engine

Symptoms. Pods never appear at all (not even Pending): a Deployment stays at zero ready replicas, or no worker pod is created for an acquired job. The owning controller's events or logs show an admission denial naming Kyverno or OPA Gatekeeper, for example:

admission webhook "validate.kyverno.svc-fail" denied the request:
... validation error: ... rule require-drop-all failed

Cause. A cluster-wide admission policy rejects the GAG pod for violating a rule it does not satisfy. The usual culprits: a policy requiring drop: [ALL] capabilities or allowPrivilegeEscalation: false on all pods (the default baseline worker profile sets neither — baseline CI relies on in-job sudo); a readOnlyRootFilesystem requirement (no worker profile sets it — the runner needs a writable root filesystem); a registry allowlist that omits GAG's registries; or a "require resource limits" rule (AGC v1alpha1 pods carry none).

This is distinct from WorkersUnschedulable (scheduler can't place a created pod) and from WorkerQuotaExceeded (ResourceQuota blocks admission): here the policy engine blocks pod creation before either applies.

Diagnostics.

# Worker path: the owning RunnerGroup surfaces the create error in its conditions.
kubectl get runnergroup -n <namespace> <runner-group> -o yaml | less   # status.conditions / events
# Proxy / AGC path: the GMC logs the failed apply.
kubectl logs -n <gmc-install-namespace> deploy/<gmc-manager> | grep -i "denied\|forbidden\|policy"
# Confirm which policy fired.
kubectl get cpol,polr -A           # Kyverno ClusterPolicies + PolicyReports
kubectl get constraints            # Gatekeeper

Resolution. Reconcile the cluster policy with GAG's real pod posture — see the admission-policies compatibility matrix, which states per policy class whether GAG complies or what to allow. In short: - Add GAG's registries to your allowlist (ghcr.io/actions-gateway/* for the control plane, ghcr.io/actions/actions-runner for the default worker). - For drop-ALL / no-privilege-escalation requirements: have tenants set securityProfile: restricted (which satisfies them), or apply the scoped exception samples so baseline workers pass. - For readOnlyRootFilesystem: exempt worker pods (no profile can satisfy it). - For "require limits" on AGC v1alpha1: migrate the tenant to a v2alpha1 ActionsGateway, or exempt AGC pods.

ActionsGateway Reports EgressRulesStale

Symptoms. kubectl get actionsgateway shows an EgressRulesStale=True condition, or the actions_gateway_egress_rules_stale gauge is 1. Optionally, jobs intermittently fail to reach newly-rotated GitHub endpoints.

Cause. The GMC refreshes each managed proxy NetworkPolicy's egress allowlist from api.github.com/meta on a ~24h cycle. If that refresh loop stalls (GitHub meta API unreachable, persistent fetch errors), the allowlist freezes. GitHub periodically rotates its published IP ranges, so a frozen allowlist eventually drops egress to the new ranges silently. The condition trips when the last successful refresh is older than the staleness window (just over two refresh cycles), so a single missed/slow refresh does not false-trip it. It is advisory and does not gate Ready — existing egress keeps working until GitHub rotates. It is only evaluated for gateways whose proxy NetworkPolicy is gateway-managed (spec.proxy.managedNetworkPolicy unset or true).

Diagnostics.

# Read the condition — its message reports how long ago the last refresh succeeded.
kubectl get actionsgateway -n <namespace> <name> \
  -o jsonpath='{range .status.conditions[?(@.type=="EgressRulesStale")]}{.status} {.reason}: {.message}{"\n"}{end}'

# Inspect the GMC log for the refresh loop's errors.
kubectl logs -n <gmc-namespace> deploy/<gmc> | grep -i "ip range"

# Confirm the GMC can reach the GitHub meta API from its pod.
kubectl exec -n <gmc-namespace> deploy/<gmc> -- wget -qO- https://api.github.com/meta >/dev/null && echo ok

Resolution. Restore the GMC's reachability to api.github.com (egress policy, DNS, proxy). The actions_gateway_ip_range_updates_total counter resumes incrementing on the next successful refresh, and the condition clears automatically within the re-check cadence (a fraction of the staleness window). If GitHub's meta API is down, no action is needed beyond waiting — the allowlist is still valid until GitHub rotates.

Tenant Namespace Missing the Managed-Tenant Marker Label

Symptoms. An ActionsGateway never becomes Ready. kubectl describe shows a Warning event with reason NamespaceMarkerMissing, and the GMC log reports a Forbidden error stamping Pod Security Admission labels, citing the namespace-psa-guard admission policy. This is common immediately after upgrading a cluster whose tenant namespaces predate the policy (see Upgrade — Migration Notes).

Cause. The GMC's cluster-wide namespaces:patch grant is gated by the namespace-psa-guard ValidatingAdmissionPolicy, which denies the GMC any namespace that is not labelled actions-gateway.github.com/tenant: "true". The label confines the grant to managed tenants so a compromised GMC cannot relabel kube-system PSA (see Security §5.1/§5.3). The GMC never sets this label itself — a trusted administrator must apply it. The same marker also gates the gmc-tenant-resource-guard policy, which confines every tenant-resource write (Deployments, Secrets, RoleBindings, …) to marked namespaces; provisioning fails at the PSA-stamping step first, so NamespaceMarkerMissing is the signal you will see, but applying the label clears both gates.

Diagnostics.

# Confirm the warning event
kubectl describe actionsgateway -n <namespace> <name> | grep -A2 NamespaceMarkerMissing

# Check whether the marker label is present
kubectl get namespace <namespace> \
  -o jsonpath='{.metadata.labels.actions-gateway\.github\.com/tenant}'   # want: true

# Confirm both policies and their bindings are installed
kubectl get validatingadmissionpolicy gmc-namespace-psa-guard gmc-tenant-resource-guard
kubectl get validatingadmissionpolicybinding gmc-namespace-psa-guard-binding gmc-tenant-resource-guard-binding

Resolution. Apply the marker label as an administrator, then the GMC reconciler retries automatically:

kubectl label namespace <namespace> actions-gateway.github.com/tenant=true

If the GMC's ServiceAccount is installed under a non-default namespace or name, also confirm the policy's matchConditions username (system:serviceaccount:gmc-system:gmc-controller-manager) matches your install.

ActionsGateway Stuck Deleting (Teardown Blocked on a Failing Delete)

Symptoms. You deleted an ActionsGateway, but the CR does not disappear: kubectl get actionsgateway -n <namespace> still lists it with a non-empty metadata.deletionTimestamp, and kubectl describe shows a repeating Warning event with reason TeardownIncomplete. Some tenant resources (e.g. the AGC Deployment, RoleBinding, or a ServiceAccount) are still present in the namespace.

Cause. Teardown is fail-closed by design (Q125): the GMC keeps the cleanup finalizer on the CR and requeues until it can confirm every owned resource is deleted (or already gone). If a delete keeps failing — most often an API-server error, or a Forbidden from an admission policy or revoked RBAC — the finalizer is retained on purpose so a live, credentialed AGC Deployment is never orphaned by a half-finished teardown. A NotFound is treated as success, so an already-deleted resource never blocks convergence.

This applies to both API versions (Q328): the v1 (actions-gateway.github.com) gateway holds the actions-gateway.github.com/gmc-cleanup finalizer; the v2 (actions-gateway.com) gateway holds its own cleanup finalizer and additionally verifies each child is actually gone after its delete is accepted — a child held by another controller's finalizer (its deletionTimestamp is set but the object lingers) also keeps the teardown open, with the lingering child named in the TeardownIncomplete event message. The v2 per-tenant metrics Secrets are the one exception: they are removed by owner-reference garbage collection (the GMC deliberately holds no delete permission on Secrets), so they never appear in the event.

Diagnostics.

# Confirm the CR is mid-deletion and which resources remain
kubectl get actionsgateway -n <namespace> <name> -o jsonpath='{.metadata.deletionTimestamp}{"\n"}{.metadata.finalizers}{"\n"}'
kubectl describe actionsgateway -n <namespace> <name> | grep -A3 TeardownIncomplete

# The event message names the namespace and the underlying error; also check the GMC log
kubectl logs -n gmc-system deploy/gmc-controller-manager --tail=50 | grep -i "delete resource during teardown"

Resolution. Fix the underlying delete failure — restore API-server health, or re-grant the GMC the delete verb / re-apply the gmc-tenant-resource-guard marker if the namespace lost its actions-gateway.github.com/tenant=true label (the policy gates DELETE too, so an unmarked namespace blocks teardown). The reconciler retries on its own backoff and removes the finalizer automatically once every delete is confirmed. Do not manually strip the finalizer to force the CR away — that re-introduces the orphaned-AGC failure mode the fail-closed behaviour exists to prevent; clear the real delete error instead.

AGC CrashLoopBackOff or Not Acquiring Jobs

Symptoms. The AGC pod is restarting repeatedly, or it is running but actions_gateway_active_sessions is zero and actions_gateway_jobs_acquired_total is not incrementing even when jobs are queued.

Likely causes. - GitHub App Secret is missing, malformed, or contains an invalid private key. - GitHub App installationId or appId is wrong. - The proxy pool is not reachable from the AGC (network policy or proxy pod not ready). - The AGC binary was built with an incompatible runner version (GitHub returns 400 on session creation).

Diagnostics.

# Check pod status and restarts
kubectl get pod -n <namespace> -l app=actions-gateway-controller

# Check logs for startup errors
kubectl logs -n <namespace> deploy/actions-gateway-controller

# Check that the referenced Secret exists and has the right keys
kubectl get secret -n <namespace> <gitHubAppRef.name>
kubectl get secret -n <namespace> <gitHubAppRef.name> -o jsonpath='{.data}' | jq 'keys'
# Expected keys: appId, installationId, privateKey

# Test proxy reachability — the AGC image is distroless (no shell, no curl),
# so spawn an ephemeral curl pod in the same namespace and use the same proxy URL.
kubectl run nettest-$$ -n <namespace> --rm -it --restart=Never \
  --image=curlimages/curl:latest \
  --overrides='{"spec":{"automountServiceAccountToken":false,"containers":[{"name":"c","image":"curlimages/curl:latest","command":["sh","-c","curl -x https://actions-gateway-proxy:8080 -sI https://api.github.com"]}]}}'

# Check RunnerGroup conditions
kubectl get runnergroup -n <namespace> -o yaml | grep -A 10 conditions

# Check RunnerGroup events — the AGC emits Warning events for the common failures.
kubectl describe runnergroup -n <namespace> <name>
# Look for:
#   TokenUnavailable          — GitHub App installation token could not be fetched (Secret/appId/installationId).
#   AgentPoolError            — agent Secret provisioning (EnsureAgents) failed.
#   ListenerStartFailed       — listener goroutines could not be (re)started.
#   AgentDeregistrationFailed — agent Secret cleanup on scale-down/delete failed.
#   RunnerVersionTooOld       — session creation rejected: the runner version is too old for GitHub (Q170).
#   SessionUnauthorized       — session creation rejected as unauthorized: agent credentials invalid/revoked (Q170).
#   JobAcquisitionFailed      — a delivered job could not be acquired from GitHub; it stays queued for redelivery (Q170).
#   NoActiveSessions / ListenerActive — Ready condition transitions.

Resolution. - If the Secret is missing or has wrong keys, recreate it. See Getting Started — GitHub App Secret. - If the private key format is wrong, ensure it is a PEM-encoded key starting with -----BEGIN RSA PRIVATE KEY----- (PKCS#1) or -----BEGIN PRIVATE KEY----- (PKCS#8, RSA or Ed25519). The Secret stringData.privateKey must include the full key including header and footer lines. - If the runner version is outdated, update workerImage in the RunnerGroup spec (or the AGC's --worker-image flag). Watch for RunnerGroup conditions with reason VersionTooOld. - If appId or installationId are wrong, update the Secret.

RunnerGroup ActiveSessions Exceeds maxListeners

Symptoms. kubectl get runnergroup -n <namespace> -o jsonpath='{.items[*].status.activeSessions}' reports a value greater than the group's spec.maxListeners, typically climbing by one after each broker or network outage. GitHub shows more concurrent runner sessions for the group than the configured ceiling.

What happened. On AGC versions without the Q100 fix, a recoverable crash of the permanent baseline listener left the active count at zero for the duration of the restart backoff; a reconcile firing inside that window started a second permanent baseline on top of the pending restart. Permanent listeners are restarted after every recoverable exit and are exempt from the maxListeners ceiling, so each repeat of the race ratchets the session count up by one, indefinitely. Fixed versions make the multiplexer start idempotent, so the race cannot stack baselines.

Resolution. - Upgrade the AGC image to a version with the Q100 fix. - To clear excess listeners immediately on an affected version, restart the AGC Deployment (kubectl rollout restart deploy/actions-gateway-controller -n <namespace>). Listener sessions are in-memory; the restarted AGC re-creates exactly one baseline per RunnerGroup.

RunnerGroup Stops Serving Jobs With Stale Ready=True

Symptoms. A RunnerGroup stops servicing queued jobs even though the AGC pod is healthy, while status.activeSessions and the Ready=True condition still report the group as operational. kubectl get runnergroup -n <namespace> -o jsonpath='{.status.activeSessions}' shows a stale nonzero value that does not match the (zero) sessions GitHub sees for the group.

What happened. The permanent baseline listener exited non-retriably — e.g. GitHub returned 401 Unauthorized on session creation for a credential it considers dead. The multiplexer does not auto-restart a non-retriable exit (that restart is reserved for recoverable crashes), so the in-memory listener count drops to zero. On AGC versions without the Q137 fix the RunnerGroup was only re-reconciled on a watch event (a RunnerGroup edit or a worker-pod lifecycle event) or the 10-hour informer resync, so with no such event the dead baseline — and the status written just before it died — could persist for hours.

Resolution. - Upgrade the AGC image to a version with the Q137 fix. Fixed versions requeue the RunnerGroup on a bounded interval while the listener count is below the desired ceiling, so the reconciler re-runs its zero-listener recovery and revives the baseline within seconds; status.activeSessions and Ready then track reality again. - To recover immediately on an affected version, trigger a reconcile by editing the RunnerGroup (e.g. a no-op annotation change) or restart the AGC Deployment (kubectl rollout restart deploy/actions-gateway-controller -n <namespace>); the restarted AGC re-creates one baseline per RunnerGroup from scratch. - If the baseline keeps exiting non-retriably after revival, the underlying credential or runner-version problem is real — check kubectl describe runnergroup for Degraded / Unauthorized / VersionTooOld conditions and resolve per the AGC CrashLoopBackOff or Not Acquiring Jobs section.

Listener Stalls for Minutes After a Black-Holed Broker Connection

Symptoms. One of a RunnerGroup's sessions stops picking up jobs for minutes at a stretch even though the AGC pod is healthy, the broker is reachable, and other sessions in the same group keep working. The stall typically follows a network event that silently drops an established connection — a firewall/NAT idle-timeout that discards packets without sending a RST, an egress-proxy failover, or a broker-side hang — so the long-poll's TCP connection is black-holed: accepted but never answered. actions_gateway_message_poll_errors_total{reason="timeout"} increments when an affected listener recovers.

What happened. The broker GetMessage long-poll holds the connection open for ~50s waiting for a job. On AGC versions without the Q108 fix the broker client had no response-header deadline, so a black-holed connection blocked the listener goroutine inside a single GetMessage call until the operating system's TCP timeout expired — minutes — during which that listener served no jobs. Fixed versions give the broker client a ResponseHeaderTimeout sized just above the 50s hold: a healthy long-poll is never cut short, but a black-holed connection is torn down a few seconds past the hold, classified as a benign "no message, retry", and the listener immediately opens a fresh long-poll.

Resolution. - Upgrade the AGC image to a version with the Q108 fix. No configuration is required; the bound is built in. - A steady stream of actions_gateway_message_poll_errors_total{reason="timeout"} after upgrade indicates the network is repeatedly black-holing broker connections (rather than wedging a listener). Investigate the egress path — proxy/NAT idle timeouts shorter than the 50s long-poll hold are the usual cause; raise the idle timeout above ~60s so healthy long-polls are not severed mid-hold.

Reconcile or Token Mint Hangs on a Slow GitHub Endpoint

Symptoms. An AGC or GMC operation that calls a GitHub REST endpoint — installation-token mint, runner registration (generate-jitconfig), rerun-failed-jobs, or the GMC's api.github.com/meta IP-range fetch — appears to stall, and on a fixed version the logs now show a prompt context deadline exceeded / Client.Timeout exceeded error instead. These are short request/response calls, distinct from the broker long-poll above.

What happened. Before the Q138 fix these clients fell back to http.DefaultClient, which has no timeout: a peer that accepted the TCP connection but was slow — or never — to send response headers wedged the calling goroutine (a reconcile or a token fetch) until the multi-minute OS TCP timeout. Fixed versions build these clients with a bounded default (an overall request timeout plus a transport response-header timeout), so a slow GitHub endpoint fails fast and retriably rather than stalling the work. The broker long-poll is the one deliberate exception — it is bounded by the response-header deadline above, not an overall timeout.

Resolution. - Upgrade to a version with the Q138 fix. No configuration is required; the bound is built in. - Repeated timeout errors point at the egress path to api.github.com / *.githubusercontent.com (proxy, NAT, or DNS latency), not the gateway — investigate connectivity to those hosts.

Orphaned RunnerGroup After Removing It From the Spec

Symptoms. A runner group was removed from (or reordered within) spec.runnerGroups on an ActionsGateway, but a RunnerGroup for it still exists and keeps running listeners and worker pods. kubectl get runnergroup -n <namespace> lists more groups than the CR now declares:

# Owner-labelled RunnerGroups for a gateway vs. what the spec now declares
kubectl get runnergroup -n <namespace> -l actions-gateway/owner-name=<gateway-name>
kubectl get actionsgateway <gateway-name> -n <namespace> -o jsonpath='{range .spec.runnerGroups[*]}{.runnerLabels[0]}{"\n"}{end}'

What happened. On GMC versions without the Q101 fix, reconciliation only created/patched the groups currently in the spec and never deleted the ones removed — and because groups were keyed by list index, a remove or reorder could orphan a RunnerGroup CR that kept serving jobs until the entire ActionsGateway was deleted.

Resolution. - Upgrade the GMC to a version with the Q101 fix. Fixed versions reconcile spec.runnerGroups to the desired set: after applying the declared groups, the GMC prunes any owner-labelled RunnerGroup no longer in the spec, and keys pruning on owner labels (not list index) so a reorder never orphans a group. A subsequent reconcile (edit the CR, or wait for the next resync) cleans up any pre-existing orphans automatically. - To remove a stranded group immediately on an affected version, delete its RunnerGroup directly: kubectl delete runnergroup <name> -n <namespace>. The AGC's RunnerGroup cleanup stops its listeners and cascades to its worker pods. Confirm you are deleting an orphan (its runnerLabels are not in the current ActionsGateway spec), not a live group.

Proxy NetworkPolicy Has an Empty GitHub Allowlist

Symptoms. On a freshly provisioned tenant, all proxy egress to GitHub is silently dropped: curl through the proxy times out (no 502), the AGC cannot acquire jobs, and token refresh fails. The proxy NetworkPolicy exists but its ipBlock egress peers are empty.

Likely cause. The IP Range Reconciler's initial api.github.com/meta fetch failed or stalled at GMC startup. The cached ranges seed each proxy NetworkPolicy's ipBlock allowlist; until the first fetch lands, the allowlist is empty. The reconciler retries the initial fetch on a capped exponential backoff (1s→30s), so a transient outage normally self-heals within seconds — but a sustained inability to reach api.github.com from the GMC pod (egress firewall, DNS, or a long GitHub outage) leaves the allowlist empty until connectivity returns.

For direct-egress gateways (no defaultProxyRef) the same GitHub allowlist lives on the <gateway>-workload and <gateway>-agc policies instead of a proxy policy, so the empty-allowlist symptom applies to worker and AGC egress there. A gateway that has already been programmed keeps its allowlist across a GMC restart: the per-gateway reconcile preserves an existing direct-egress policy's rules while the cache is still warming, rather than rebuilding it from the empty cache (which would have blanked the allowlist for the seconds until the first fetch lands — a window that widened under node CPU pressure and caused release-asset downloads to time out right after a restart). Only a first-ever provision with a not-yet-populated cache shows the empty allowlist, and it self-heals on the first fetch.

Diagnostics.

# Proxied gateway: inspect the proxy NetworkPolicy's GitHub ipBlock egress peers — empty means the cache never populated.
kubectl get networkpolicy -n <namespace> actions-gateway-proxy \
  -o jsonpath='{.spec.egress[*].to[*].ipBlock.cidr}'

# Direct-egress gateway: check the workload (and AGC) policy instead.
kubectl get networkpolicy -n <namespace> <gateway>-workload \
  -o jsonpath='{.spec.egress[*].to[*].ipBlock.cidr}'

# Look for retry warnings in the GMC log.
kubectl logs -n gmc-system deploy/gmc-controller-manager \
  | grep -i "GitHub IP-range"

Resolution. - Confirm the GMC pod itself can reach api.github.com (corporate egress firewall, DNS, or proxy in front of the cluster). The reconciler retries automatically; once connectivity is restored the next successful fetch patches every existing NetworkPolicy. - If the tenant manages its own egress policy (Cilium/Calico FQDN rules), set spec.proxy.managedNetworkPolicy: false so the reconciler leaves the policy alone.

Worker Pods Stuck Pending

Symptoms. Jobs are acquired (actions_gateway_jobs_acquired_total increments) but worker pods remain in Pending state for more than 60 seconds. pod_creation_latency_seconds p95 exceeds the 15s SLO target.

Likely causes. - Namespace ResourceQuota is exhausted — no pod slot, CPU request, or memory request available. - No node has enough capacity for the pod's requested resources (GPU nodes may be at capacity). - PriorityClass referenced in priorityTiers does not exist. - Image pull is slow due to a large image on a cold node (expected; see SLO targets in Appendix A).

Diagnostics.

# Check quota usage
kubectl describe resourcequota -n <namespace>

# Describe a stuck pod to see the scheduling event
kubectl describe pod -n <namespace> <worker-pod-name>
# Look for: "Insufficient cpu", "Insufficient memory", "Insufficient nvidia.com/gpu",
#           "no nodes available to schedule pods", "didn't match PodDisruptionBudget"

# Check whether the PriorityClass exists
kubectl get priorityclass <priorityClassName>

# Check node capacity
kubectl describe nodes | grep -A 5 "Allocated resources"

Resolution. - If quota is exhausted: raise the platform-owned ResourceQuota on the namespace (kubectl edit resourcequota -n <namespace> <quota-name>) or reduce maxWorkers / last-tier threshold. - If no GPU nodes are available: check node autoscaler status or provision additional nodes. - If a PriorityClass is missing: create it (cluster-admin action) or remove the tier reference. - If image pull is slow (first job on a cold node): this is expected. If it exceeds the p99 SLO (60s), consider pre-pulling the image via a DaemonSet or enabling image streaming.

Deadline. A pod that stays Pending is not held forever: after pendingPodDeadline (default 10m, per-RunnerGroup) the AGC deletes it to free the concurrency-ceiling slot it was holding — see the next runbook section. Diagnose a stuck pod (kubectl describe pod) before the deadline reaps it, or raise pendingPodDeadline temporarily while debugging.

Worker Pod Reaped While Pending (WorkerPodStuckPending)

Symptoms. A Warning Event with reason WorkerPodStuckPending appears on the RunnerGroup (kubectl describe runnergroup -n <namespace>), actions_gateway_worker_pods_reaped_total{reason="pending_deadline"} increments, and the job the pod was created for is cancelled by GitHub (it never started, so its lock lapsed). The worker pod itself is gone.

What happened. The pod stayed Pending longer than the RunnerGroup's pendingPodDeadline (default 10m), so the AGC deleted it. A permanently Pending pod would otherwise hold one of the group's concurrency-ceiling slots forever — the ceiling counts Pending pods. The deadline is a capacity-protection mechanism, not a retry mechanism: the job is not re-queued automatically.

Likely causes. - workerImage (or the podTemplate container image) does not exist or is not pullable from the cluster — ErrImagePull / ImagePullBackOff. - podTemplate scheduling constraints (nodeSelector, tolerations, GPU resources) that no node satisfies. - Node autoscaler provisioning slower than the deadline (common for GPU node pools).

Diagnostics.

# The reap event names the deleted pod and the deadline that fired
kubectl get events -n <namespace> --field-selector reason=WorkerPodStuckPending

# Rate of reaps per group
# PromQL: rate(actions_gateway_worker_pods_reaped_total{reason="pending_deadline"}[1h])

# Reproduce the pull/scheduling failure before the next reap:
# trigger a job, then describe the new Pending pod within the deadline window
kubectl get pods -n <namespace> -l actions-gateway/runner-group=<group> -w
kubectl describe pod -n <namespace> <worker-pod-name>

Resolution. - Fix the unpullable image or unsatisfiable scheduling constraint — that is the root cause; the reap is the messenger. - If scheduling is legitimately slow (autoscaled GPU nodes), raise spec.pendingPodDeadline on the RunnerGroup (or the matching runnerGroups[] entry of the ActionsGateway CR) above the worst-case node-provisioning time, e.g. pendingPodDeadline: "30m". - Re-run the cancelled workflow from the GitHub UI once the cause is fixed.

Scale-Set Job Stranded by a Stale Runner Record (Runner-Name 409)

Symptoms. On the scale-set path (acquisitionProtocol: ScaleSet), one job is never picked up while others in the same RunnerSet run fine. The AGC logs repeat scaleset: runner name conflict for a single jobID, and — on versions before the fix — end in scaleset: runner name conflict persists, skipping job. Restarting the AGC clears it. In the repo/org runner list, offline records named <scaleSet>-<jobID> (and <scaleSet>-<jobID>-1, -2, -3) accumulate over time.

What happened. The scale-set listener pre-registers each worker's runner under a deterministic <scaleSet>-<jobID> name via generatejitconfig. When a worker pod is reaped while still Pending (see above), it never came online, so GitHub does not auto-remove its runner record — the record lingers offline holding that name. Every later re-provision of the same jobID derives the same name and 409s, so the job is stranded until the AGC restarts.

Fix. Current versions self-heal: on the base-name 409 the listener deletes the stale record (REST DELETE .../actions/runners/{id}) and re-registers under the same name, so the job provisions and the offline records stop piling up. A record still running a job (422) is left in place and the job takes a fresh suffixed name instead. No operator action is needed on a fixed version.

Manual cleanup (older versions, or to clear an existing backlog). Delete the offline records — they re-register on the next run:

# List self-hosted runner records (repo-scoped shown; use /orgs/<org>/... for org scope)
gh api /repos/<owner>/<repo>/actions/runners --paginate \
  | jq -r '.runners[] | select(.status=="offline") | "\(.id)\t\(.name)"'

# Delete each offline record by id (skip any that are online or busy)
gh api -X DELETE /repos/<owner>/<repo>/actions/runners/<id>

Only delete records that are offline and not busy; an online record is a live listener or a running worker.

Worker Pods Stuck Running After the Job Finished (Mesh Sidecar)

Symptoms. Worker pods sit Running with a not-ready container count (READY 1/2) long after their job completed; completedPodTTL never deletes them; over time the RunnerGroup wedges at maxWorkers and new jobs stop being picked up even though no job is actually executing. kubectl get pod -o jsonpath='{.spec.containers[*].name}' shows a second container such as istio-proxy or linkerd-proxy.

What happened. A service-mesh sidecar was injected into the worker pod. GAG worker pods run to completion: the slot is freed and the pod reaped only when the pod reaches a terminal phase (Succeeded/Failed), which requires every container to exit. A classic mesh sidecar never exits on its own, so the pod stays Running forever and falls through both reaper paths (completedPodTTL covers terminal pods; pendingPodDeadline covers Pending pods — neither covers a stuck Running pod).

Resolution. Opt the GAG tenant namespace out of the mesh, or — if mesh membership is mandatory — switch to native sidecars (Kubernetes 1.28+) or a sidecar-less/ambient data plane. The full per-mesh configuration (Istio sidecar + ambient, Linkerd, Cilium, generic) is in Running GAG Alongside a Service Mesh. Note that mesh opt-out/exclusion annotations set on the RunnerGroup podTemplate are not honored — GAG strips arbitrary worker-pod-template metadata; configure the mesh at the namespace level instead.

A mesh sidecar is injected at admission, so it is not in the RunnerTemplate and GAG cannot warn about it ahead of time — the runtime symptom above is the only signal. A build/DinD sidecar you declare in the template is different: GAG detects it and warns proactively — see the next section.

RunnerSet Reports PossibleReapBlockingSidecar (Build/DinD Sidecar in the Template)

Symptoms. A RunnerSet reports the advisory condition PossibleReapBlockingSidecar=True (kubectl get runnerset <name> -n <ns> -o jsonpath='{.status.conditions}'), the actions_gateway_reap_blocking_sidecar_templates gauge is > 0, and/or kubectl apply of the RunnerTemplate/ClusterRunnerTemplate printed a Warning: naming a sidecar container. Left unfixed, the symptom is the same READY 1/2 stranding as the mesh-sidecar case above: worker pods linger after the job and the set wedges at maxWorkers.

What happened. The resolved worker template carries a regular (non-native) sidecar container besides the runner container — e.g. a docker:dind daemon or a BuildKit sidecar declared under spec.containers[]. A pod terminates only when every regular container exits, so a sidecar that runs for the life of the job keeps the pod from reaping (Q249). The condition, gauge, and admission warning are advisory only — they never block template creation or gate the set's Ready — because the "runs forever" property can't be proven from a pod spec.

Resolution.

  • Preferred — convert to a native sidecar. Move the sidecar to spec.initContainers[] with restartPolicy: Always (Kubernetes ≥ 1.29). The kubelet tears it down when the runner container exits, so the pod completes on its own. The template shape is in In-runner image builds § Sidecar containers must be native sidecars.
  • If the sidecar genuinely exits on its own when the job ends, acknowledge it in the template's actions-gateway.com/self-exiting-sidecars annotation (a comma-separated name-list). This silences the warning, the condition, and the gauge for the named containers only — a newly added, unacknowledged sidecar still warns.
# See which containers a set's condition is flagging
kubectl get runnerset <name> -n <namespace> \
  -o jsonpath='{.status.conditions[?(@.type=="PossibleReapBlockingSidecar")].message}'
# Which templates are flagged, fleet-wide
# PromQL: actions_gateway_reap_blocking_sidecar_templates > 0

Job-Lifecycle Events on a RunnerGroup / RunnerSet

What this is. Beyond WorkerPodStuckPending (above), the AGC records Warning Kubernetes Events on the owning RunnerGroup (v1alpha1) or RunnerSet (v2alpha1) when a job-lifecycle transition fails terminally (Q170). They are the event-based companion to the always-present metrics and status conditions — surfacing the same incident in kubectl describe, kubectl get events, and any event watcher, without a Prometheus query. Each Reason mirrors the corresponding metric name so you can correlate the two.

Events are recorded on a transition / terminal outcome, not on every reconcile or every requeue, so they do not spam. (The cluster's event recorder additionally aggregates repeats of the same reason+message into one event with a count.) An event is recorded on the owner's next reconcile, so it can trail the underlying metric by a few seconds; the metric is the real-time signal.

Reason Type Meaning Where to look next
JobAcquisitionFailed Warning A delivered job could not be acquired from GitHub (acquirejob failed); the job stays queued at GitHub for redelivery. AGC CrashLoopBackOff or Not Acquiring Jobs
RunnerVersionTooOld Warning Session creation was rejected permanently because the runner version is too old for GitHub. Also sets the RunnerVersionTooOld condition. AGC CrashLoopBackOff or Not Acquiring Jobs
SessionUnauthorized Warning Session creation was rejected as unauthorized — the agent credentials are invalid or revoked. Also sets the Degraded condition. GitHub App Secret Misconfiguration
QuotaRetriesExhausted Warning Worker pod creation was abandoned after exhausting the namespace ResourceQuota retry budget (maxQuotaRetries). Jobs Failing Due to Namespace ResourceQuota Exhaustion
EvictionRetriesExhausted Warning An evicted worker pod's auto-retry budget (maxEvictionRetries) is exhausted; a manual re-run is required. Evicted Worker Pods Exhausting Retry Budget

Diagnostics.

# All AGC-emitted Warning events on one owner, newest last.
kubectl describe runnergroup -n <namespace> <name>          # v1alpha1
kubectl describe runnerset   -n <namespace> <name>          # v2alpha1

# Filter the namespace event stream by a specific reason.
kubectl get events -n <namespace> --field-selector reason=EvictionRetriesExhausted

Proxy Pool Not Scaling

Symptoms. The HPA for the proxy pool shows TARGETS: <unknown>/60% and the replica count does not increase under load.

Likely cause. resources.requests.cpu is unset or zero for proxy pods. The Kubernetes Horizontal Pod Autoscaler (HPA) computes CPU utilization as (current_cpu_usage / requested_cpu). If requests.cpu is zero, the denominator is undefined and the HPA emits <unknown> for the target metric and stops scaling entirely.

Diagnostics.

# Check HPA status
kubectl describe hpa -n <namespace> actions-gateway-proxy

# Check proxy pod resource requests
kubectl get pod -n <namespace> -l app=actions-gateway-proxy -o jsonpath='{.items[0].spec.containers[0].resources}'

# Check metrics-server is running
kubectl get pods -n kube-system -l k8s-app=metrics-server

Resolution.

Ensure spec.proxy.resources.requests.cpu is set to a non-zero value in the ActionsGateway spec. The default is 10m. If you explicitly set resources without including requests.cpu, the whole resources block is replaced and defaults are lost — set all four sub-fields explicitly:

proxy:
  resources:
    requests:
      cpu: "10m"
      memory: "32Mi"
    limits:
      cpu: "100m"
      memory: "64Mi"

After updating the spec, patch the proxy Deployment or trigger a rollout; the HPA will start computing utilization on the next metrics scrape cycle (~30s).

Second likely cause: the namespace ResourceQuota won't admit the replicas the HPA wants. The HPA computes utilization correctly but the proxy Deployment cannot create more pods because the platform-owned namespace ResourceQuota is the hard cap. Under load the pool wedges below its target and the Deployment/ReplicaSet logs FailedCreate ... exceeded quota events instead of scaling out.

The GMC surfaces this as two non-blocking conditions on the ActionsGateway (neither gates Ready — the pool keeps serving at its current scale), each also exported as a gauge for alerting:

Condition / metric Meaning Action
ProxyQuotaPressure (warning) — actions_gateway_proxy_quota_pressure The pool can't grow to maxReplicas within the quota's remaining headroom (hard − used). Load-dependent. Raise the quota or lower maxReplicas before the next spike.
ProxyQuotaExceeded (error) — actions_gateway_proxy_quota_exceeded Replica creates are being rejected now (Deployment ReplicaFailure with exceeded quota). Raise the quota now; the pool is degraded below the HPA's target. 
# Read both conditions (Exceeded supersedes Pressure when firing).
kubectl get actionsgateway -n <namespace> <name> \
  -o jsonpath='{range .status.conditions[?(@.type=="ProxyQuotaPressure")]}{.type}={.status} {.reason}: {.message}{"\n"}{end}'
kubectl describe actionsgateway -n <namespace> <name>

Resolve by either raising the platform-owned quota (kubectl edit resourcequota -n <namespace> <quota-name>) to admit the configured maxReplicas, or lowering spec.proxy.maxReplicas to fit. Editing the quota's .spec.hard re-triggers reconciliation immediately; the conditions clear on the next reconcile.

Proxy Tunnel Closed Mid-Stream — Idle or Lifetime Cap

Symptoms. A worker job logs a connection reset, EOF, or broken pipe from the GitHub SDK / curl / git, with no proxy 502 response. The proxy pod itself is healthy and serving other tunnels.

Likely cause. The proxy enforces two per-tunnel deadlines on the CONNECT relay (M-18, 2026-05-31):

  • Idle timeout — 5 minutes of no data in either direction. A long-poll against the GitHub API or a stalled SDK call hits this first.
  • Hard lifetime cap — 6 hours absolute, regardless of activity. A continuous artifact stream or Twirp log relay that exceeds this is torn down even with traffic flowing.

These are not bugs. They bound goroutine and file-descriptor exhaustion from slow or stuck clients. The healthy case (an actively-used GitHub API call) completes in seconds and does not trip either cap.

Diagnostics.

The proxy serves /metrics over mutual TLS on :8443 (not :8081, which now carries only the plaintext /healthz + /readyz probes). Scraping requires the per-tenant scraper client certificate the GMC publishes — see Metrics scrape returns a TLS / connection error for how to fetch the bundle. With the bundle written to scraper.crt / scraper.key / metrics-ca.crt:

ns=<namespace>
# Distribution of tunnel lifetimes; a heavy tail near 21600s (6h) or
# a spike at 300s (5m idle) indicates clients hitting the caps.
curl -s --cert scraper.crt --key scraper.key --cacert metrics-ca.crt \
  "https://actions-gateway-proxy.$ns.svc:8443/metrics" | \
  grep actions_gateway_proxy_tunnel_duration_seconds_bucket

# Active vs. total tunnels — healthy ratio is "active << total".
curl -s --cert scraper.crt --key scraper.key --cacert metrics-ca.crt \
  "https://actions-gateway-proxy.$ns.svc:8443/metrics" | \
  grep -E 'actions_gateway_proxy_connections_(active|total)'

Resolution.

For idle hits: examine the workflow step that stalls. A workflow sleep-ing inside a long-running curl --connect-timeout 0 or a misconfigured webhook receiver are typical causes. The fix is usually in the workflow, not the proxy.

For lifetime-cap hits: split very long-running uploads or streams across multiple HTTP requests. The 6h cap is a safety net for stuck connections; a legitimately-long single stream should be rare.

To change the defaults during an incident, patch the proxy Deployment with environment overrides — note that there is no env-var knob today; defaults are baked into the Server struct and require a code change to adjust. File a Queue item if a tenant repeatedly hits either cap on a legitimate workload.

Metrics scrape returns a TLS / connection error

Symptoms. Prometheus (or a manual curl) of a per-tenant proxy or AGC /metrics endpoint fails with one of:

  • remote error: tls: certificate required / bad certificate — no client cert, or one signed by the wrong CA.
  • connection refused on :8081/metrics — the metrics endpoint moved to :8443 (mTLS); :8081 now serves only /healthz + /readyz.
  • context deadline exceeded / no route — the scraper namespace is not labelled metrics: enabled, so the NetworkPolicy drops the connection before the handshake.

Cause. The proxy and AGC serve /metrics over mutual TLS on :8443 (Q69). A scraper must (1) connect from a namespace labelled metrics: enabled and (2) present a client certificate signed by the per-tenant metrics CA the GMC issues. Both halves are required.

Resolution.

  1. Label the monitoring namespace so the NetworkPolicy admits it: 
    kubectl label namespace <prometheus-namespace> metrics=enabled
  2. Fetch the scraper client bundle the GMC publishes in each tenant namespace and point the scrape at :8443 with scheme: https: 
    ns=<tenant-namespace>
kubectl get secret actions-gateway-metrics-client -n "$ns" -o jsonpath='{.data.tls\.crt}' | base64 -d > scraper.crt
kubectl get secret actions-gateway-metrics-client -n "$ns" -o jsonpath='{.data.tls\.key}' | base64 -d > scraper.key
kubectl get secret actions-gateway-metrics-client -n "$ns" -o jsonpath='{.data.ca\.crt}'  | base64 -d > metrics-ca.crt
curl -s --cert scraper.crt --key scraper.key --cacert metrics-ca.crt \
  "https://actions-gateway-proxy.$ns.svc:8443/metrics" | head
    Delete the extracted key file when finished. For a ServiceMonitor, mount the bundle and reference it from tlsConfig (cert/keySecret/ca).
  3. If the cert is rejected after a CA rotation, the GMC re-issues the whole bundle ~30 days before expiry but pods read certs at startup — restart the proxy/AGC pods (and re-fetch the client bundle) after a rotation.

RateLimited Condition on ActionsGateway

Symptoms. kubectl get actionsgateway shows a RateLimited=True condition. actions_gateway_active_sessions is at or near the per-installation budget.

Likely cause. The GitHub App installation's API budget (15,000 GET /message requests/hour) is exhausted. This occurs when the sum of maxListeners across all RunnerGroups simultaneously bursts to their ceiling for a sustained period.

SLO threshold. A RateLimited condition lasting more than 1 minute during non-peak hours indicates the installation is over budget. Durations exceeding 10 minutes during business hours should page on-call.

Diagnostics.

# Check the condition
kubectl get actionsgateway -n <namespace> <name> -o jsonpath='{.status.conditions}' | jq .

# Check active sessions vs. budget
# Budget: ~208 sessions (15000/hr ÷ 72 polls/session/hr)
# Metric: actions_gateway_active_sessions{namespace="<namespace>"}

# Check per-RunnerGroup maxListeners sum
kubectl get runnergroup -n <namespace> -o jsonpath='{.items[*].spec.maxListeners}'

Resolution. - If a burst is temporary and below 10 minutes: no action required, the condition will clear as the burst subsides. RateLimited=True (reason SustainedRateLimit) is set only after GET /message has been answered 429 for over 10 minutes, and clears to RateLimited=False (reason PollingHealthy) on the first successful poll once the budget recovers — you do not need to restart the AGC to clear a stale condition. - If maxListeners values are set higher than needed, reduce them. - If the tenant's RunnerGroup count × maxListeners sustainably exceeds the installation budget, shard to a second ActionsGateway CR with a new GitHub App installation. See Appendix E §E.6.

GitHub App Secret Misconfiguration

Symptoms. AGC logs show errors like failed to create installation token, private key: RSA key parse error, or 401 Unauthorized. The ActionsGateway condition AGCAvailable=False with reason CredentialError. When the AGC cannot obtain an installation token while reconciling a RunnerGroup, that group also reports CredentialUnavailable=True (reason TokenUnavailable) in its status — surfacing the failure in kubectl get runnergroup/describe, not only as a TokenUnavailable Event. The condition clears (CredentialUnavailable=False, reason CredentialAvailable) on the next reconcile once a token is obtained. Read it with:

kubectl get runnergroup -n <namespace> <name> \
  -o jsonpath='{range .status.conditions[?(@.type=="CredentialUnavailable")]}{.status} {.reason}: {.message}{"\n"}{end}'

Common misconfigurations.

Error message Likely cause
private key: RSA key parse error / no PEM block found PEM key is corrupted — extra whitespace, missing or extra newlines, CRLF line endings, hand-paste damage, or an unsupported block type (e.g. OPENSSH/EC, which fail with unsupported PEM block type). Both PKCS#1 (-----BEGIN RSA PRIVATE KEY-----) and PKCS#8 (-----BEGIN PRIVATE KEY-----) are accepted, so PKCS#8 is not a wrong format.
401 Unauthorized on token exchange appId or installationId is wrong.
404 Not Found on token exchange The GitHub App is not installed in the target organization or the installationId does not match.
422 Unprocessable Entity The App lacks the Actions: Read and Administration: Read permissions.

Diagnostics.

# Check Secret keys exist and are non-empty
kubectl get secret -n <namespace> <name> -o jsonpath='{.data.appId}' | base64 -d
kubectl get secret -n <namespace> <name> -o jsonpath='{.data.installationId}' | base64 -d
kubectl get secret -n <namespace> <name> -o jsonpath='{.data.privateKey}' | base64 -d | head -1
# Expected first line: -----BEGIN RSA PRIVATE KEY----- (PKCS#1)
#                  or: -----BEGIN PRIVATE KEY----- (PKCS#8, RSA or Ed25519)

# Verify the App ID and installation ID match the GitHub App
# GitHub UI: Settings → Developer settings → GitHub Apps → <app> → General (App ID)
# GitHub UI: Settings → Developer settings → GitHub Apps → <app> → Install App (Installation ID in URL)

Resolution. Re-create the Secret with correct values. To trigger a rolling update on the AGC Deployment after fixing the Secret, change gitHubAppRef.name in the ActionsGateway spec to reference the new Secret name (the GMC will roll the AGC Deployment automatically) or manually restart the Deployment:

kubectl rollout restart deploy/actions-gateway-controller -n <namespace>

See Getting Started — Rotating GitHub App Credentials for the full rotation procedure.

Token Refresh Errors Spiking

Symptoms. actions_gateway_token_refresh_errors_total is increasing. GitHub App installation tokens expire after one hour; if refresh fails, new sessions cannot be established once the token expires.

Likely causes. - GitHub API is temporarily unavailable or returning 5xx errors. - The GitHub App private key was rotated but the Secret was not updated. - Network path from AGC to GitHub API is down (proxy pool issue).

Diagnostics.

# Check the error rate
# Metric: rate(actions_gateway_token_refresh_errors_total[5m])

# Check AGC logs for the error detail
kubectl logs -n <namespace> deploy/actions-gateway-controller | grep "token refresh"

# Test connectivity to GitHub via the tenant proxy (AGC is distroless — use an
# ephemeral curl pod in the same namespace; it picks up the same NetworkPolicy egress).
kubectl run nettest-$$ -n <namespace> --rm -it --restart=Never \
  --image=curlimages/curl:latest \
  --overrides='{"spec":{"automountServiceAccountToken":false,"containers":[{"name":"c","image":"curlimages/curl:latest","command":["sh","-c","curl -x https://actions-gateway-proxy:8080 -sI https://api.github.com/app"]}]}}'

Resolution. - If GitHub is temporarily unavailable: the AGC's exponential back-off retry (5s → 60s cap) will recover automatically. Monitor until the error rate returns to zero. - If the private key was rotated: update the Secret. See Getting Started — Rotating GitHub App Credentials. - If the proxy is unreachable: see Proxy Pool Not Scaling and the network connectivity section below.

SLO. Token refresh errors should stay below 1 per hour per tenant. Above this rate, begin investigating immediately. In-flight sessions will fail at the next reconnection once the token expires (~1 hour).

RenewJob Failures Rising

Symptoms. actions_gateway_renew_job_errors_total is increasing. Jobs may start being cancelled by GitHub before completion. On current versions, a definitively lost lock also increments actions_gateway_renew_job_teardowns_total and the AGC self-cancels the worker (see the self-cancel note below).

Likely causes. - Network connectivity issues between the AGC and GitHub (via proxy). - GitHub API is temporarily unavailable. - The runner job lock window expired before the renewer could refresh (AGC was slow or restarting). - AGC versions before the Q247 fix renewed with the wrong job identifier (the broker envelope's MessageID instead of the job's RunnerRequestID), so every renewal returned an error and no lock was ever refreshed. The tell is a sustained, non-transient error rate that tracks the acquired-job rate — every job that outlives GitHub's lock window (roughly one renewal interval) is then recycled and redelivered to a sibling session, so you also see duplicate worker pods for one job and completed jobs that fail with conclusion: failure, no failed step, no logs, and a TaskOrchestrationJobNotFoundException at CompleteJobAsync. Long jobs expose it; sub-window jobs finish before the lock lapses. - AGC versions before the Q247 residual fix ran each RenewJob call inline with no per-call timeout. Under heavy worker-node load (CPU/egress saturation) a single /renewjob call can black-hole — the connection is accepted but never answered — and, because the next renewal cannot start until the call returns, that one hung call starves every subsequent renewal. The tell is a long job failing at exactly GitHub's ~10-minute lock window (the initial lock TTL, never refreshed) with a single worker pod that keeps running past the cutoff — distinct from the wrong-jobId signature above, which produces duplicate pods. Fixed versions bound each renewal with the control-plane timeout, so a hung call aborts (one renew_job_errors_total increment) and the loop renews on schedule. - AGC versions before the Q247 auth fix renewed with the broker session (OAuth) token instead of the job-scoped token GitHub issues in the acquirejob response (the SystemVssConnection endpoint's AccessToken). GitHub accepts the session token to claim a job but rejects it for renewing that job's lock, so every renewal returns 401 {"source":"actions-run-service","errorMessage":"Not authorized for this job"} from the very first call. The tell is identical to the residual signature — a long job failing at exactly the ~10-minute lock window with a single worker pod — but the AGC log shows every RenewJob returning that specific 401 (not a timeout, not a wrong jobId). Fixed versions present the job-scoped token, so renewals return 200 and the lock is refreshed.

Self-cancel on a definitively lost lock (current behavior, Q254). On a lock the renewer can prove is unrecoverably lost, the AGC no longer lets the worker run on to completion as an orphan pod. Two triggers:

  • Definitive job-gone. The run service answers /renewjob with 404/410 (the job's lock no longer exists — GitHub recycled or reassigned it). A single such response is enough.
  • Sustained failure run. Renewal fails for 5 consecutive intervals (~5 min at the default 60s cadence) — a network partition or a persistently unreachable run service. This is well past any single transient blip, and it tears down before the ~10-minute lock TTL lapses.

On either trigger the AGC cancels the job's context so the worker pod tears down promptly, logs a distinct error line (job lock definitively lost; cancelling worker …), and increments actions_gateway_renew_job_teardowns_total{reason="job_not_found"|"consecutive_failures"}. Tearing the orphan down before the lock lapses closes the residual window in which GitHub could recycle the job and redeliver it to a sibling session (a duplicate worker pod for one job). A single/transient renewal failure still stays non-fatal and is retried.

Diagnostics.

# Check recent error rate
# Metric: rate(actions_gateway_renew_job_errors_total[5m])

# Check AGC logs for renewal errors and job IDs
kubectl logs -n <namespace> deploy/actions-gateway-controller | grep "renewjob"

# Definitive-loss teardowns (worker self-cancelled), split by reason
# Metric: sum by (reason) (rate(actions_gateway_renew_job_teardowns_total[15m]))
kubectl logs -n <namespace> deploy/actions-gateway-controller | grep "job lock definitively lost"

# Confirm the proxy pool is healthy
kubectl get pods -n <namespace> -l app=actions-gateway-proxy

Resolution. - Sustained errors on every job (the Q247 signature above): upgrade to a gateway version with the Q247 fix, which renews by RunnerRequestID. On affected versions no renewal succeeds, so there is no interim mitigation short of the upgrade — jobs longer than the lock window keep failing. - A long job failing at exactly the ~10-minute lock window with a single (not duplicate) worker pod (the Q247 residual signature above): upgrade to a gateway version that bounds each renewal call with the control-plane timeout. Reducing worker-node CPU/egress pressure (which is what makes a renewal call black-hole) lowers the odds on affected versions but is not a reliable mitigation — the upgrade is the fix. - Every renewal returning 401 "Not authorized for this job" from the first call (the Q247 auth signature above): upgrade to a gateway version that renews with the job-scoped token from the acquirejob response. On affected versions no renewal is authorized, so there is no interim mitigation short of the upgrade — jobs longer than the lock window keep failing. - Transient GitHub API errors: the renewer retries; monitor until the rate returns to zero. - Proxy pool unhealthy: fix the proxy pool (see Proxy Pool Not Scaling). - If the AGC restarted mid-job: jobs whose lock expired will have been cancelled by GitHub. These require manual re-run. Check actions_gateway_eviction_retries_exhausted_total for any jobs that were also evicted. - actions_gateway_renew_job_teardowns_total rising (a self-cancel, Q254 behavior above): the worker was torn down because its lock was definitively lost — this is the AGC avoiding an orphan pod, not a new fault. Investigate the underlying cause via the split reason: reason="job_not_found" means GitHub reassigned/recycled the job (often downstream of a lock that already lapsed for one of the Q247 reasons, or genuine cancellation); reason="consecutive_failures" means renewal was unreachable for ~5 min — treat it like a sustained error rate above (proxy/egress or GitHub reachability). The affected job is re-run by GitHub on the sibling session that picks it up.

Each renewjob error is a warning, not an immediate job failure — GitHub grants ~10 minutes per renewal window. A single transient error on a long-running job is rarely fatal; a sustained error on every job is the Q247 signature above, not a transient blip. When a lock is definitively lost, current versions self-cancel the worker (Q254 behavior above) rather than orphaning it.

Sessions Stuck in 401/EOF GetMessage Loops (Tenant Throughput Decays to Zero)

Symptoms. On gateway versions without the Q114 self-heal (≤ the M4 validation build): - AGC logs fill with repeating GetMessage error ... decode response: EOF and later broker: unauthorized (HTTP 401) lines for the same session, backing off forever. - The repo/org runner list (gh api .../actions/runners) loses one runner after each completed job, and the registrations never come back. - RunnerGroup status.activeSessions decays over time; after roughly maxListeners completed jobs, queued workflow jobs wait forever even though the AGC pod is healthy.

Cause. GitHub deletes a JIT-registered runner record once it acquires a job (single-use runners). Pre-fix AGC versions keep polling the dead session with the dead agent's credentials instead of re-registering, so every completed job permanently burns one listener slot (M4 §12, bug 2).

Diagnostics.

# Repeating EOF/401 poll errors
kubectl logs -n <namespace> deploy/actions-gateway-controller | grep -E "decode response: EOF|unauthorized"

# Listener slots remaining
kubectl get runnergroup -n <namespace> -o jsonpath='{.items[*].status.activeSessions}'

# On fixed versions, recycles should be happening instead:
# Metric: rate(actions_gateway_agent_recycles_total[15m])  — roughly tracks job completions
# Metric: actions_gateway_agent_recycle_errors_total       — should stay flat

Resolution. - Upgrade to a gateway version with the Q114 self-heal. Fixed versions re-register each agent after every job (actions_gateway_agent_recycles_total{trigger="post_job"}) and heal stale sessions discovered after a restart (trigger="stale_session" / "startup"); no per-job operator action is needed. - Interim manual recovery on pre-fix versions: delete the RunnerGroup's agent Secrets and restart the AGC so it registers a fresh pool:

kubectl delete secret -n <namespace> -l actions-gateway/runner-group=<group>
kubectl rollout restart deploy/actions-gateway-controller -n <namespace>

Expect 409 Already exists registration errors for any agent that never ran a job — its record survives server-side under an ID the AGC no longer knows. Delete the survivor from GitHub first: find its ID with gh api '.../actions/runners?name=<group>-<index>', then gh api -X DELETE .../actions/runners/<id>. Fixed versions resolve this 409 automatically.

On fixed versions, a sustained rise of actions_gateway_agent_recycle_errors_total means the AGC cannot re-register agents (registration API unreachable, installation token failures, or revoked GitHub App runner-administration permission) — listener capacity shrinks until recycles succeed. Check AGC logs for recycle errors and verify the App's runner permissions.

Concurrent Job Burst Serializes to ~1 Worker (Recycle Blocked on a Still-Running Runner)

Symptoms. Each job runs green individually, but bursting a whole CI matrix onto the gateway at once serializes to roughly one running worker even with ample node room (nodes well under capacity, zero Pending pods). Queued jobs sit unstarted until GitHub cancels them at its ~15-minute unstarted-job timeout; an already-running job whose token is invalidated dies with RenewJob 401 "Not authorized for this job".

Cause. After a single-use JIT runner completes a job, GitHub auto-removes the ephemeral runner record — but for a few to tens of seconds it still reports the runner as running and answers a delete with 422 "Runner … is currently running a job and cannot be deleted". Because the AGC re-registers each agent under a stable name, the lingering record also makes the re-registration conflict (409). Under a burst, many agents recycle at once and hit this window together. On gateway versions before the Q259 fix the AGC treated the 422 as fatal: the post-job recycle failed, the listener goroutine exited, and a non-permanent replacement listener is not restarted — so every completed job permanently dropped a polling slot until only the permanent baseline remained. GitHub then had one online runner to dispatch to, so it dispatched ~1 job at a time.

Diagnostics.

# Recycle errors climbing in lockstep with a burst (pre-fix: fatal 422s)
kubectl logs -n <namespace> deploy/actions-gateway-controller | grep -iE "currently running|recycle"

# Metric: actions_gateway_agent_recycle_errors_total — spikes during the burst on pre-fix versions
# Metric: actions_gateway_active_sessions             — collapses toward 1 as replacements exit

Resolution. - Upgrade to a gateway version with the Q259 fix. Fixed versions treat the 422 "currently running" as transient: Pool.Recycle retries the re-registration with a bounded, jittered backoff (waiting for GitHub to release the just-consumed runner) instead of failing, so the listener keeps its polling slot and concurrency is sustained. A recycle that finally succeeds after the wait increments actions_gateway_agent_recycles_total{trigger="post_job"} as usual. - On fixed versions, actions_gateway_agent_recycle_errors_total still rises only if a runner never releases within the retry bound — a genuine fault (registration API unreachable, or the runner is truly wedged running server-side), not the normal post-job race. Investigate as in the section above.

Related seam — the freshly recycled record's token exchange (Q267). A recycle that does re-register successfully then exchanges the new record's client credential for a broker OAuth token. For a brief window between generate-jitconfig creating the record and GitHub's OAuth service recognizing it, that exchange returns a transient 400 "Registration … was not found". On versions before the Q267 fix this 400 was fatal on the recycle path — the listener exited and churned yet another record, and at a wide maxListeners under sustained burst the exits multiplied stale records and held the online pool near zero even though agent_recycles_total kept climbing. The tell is AGC logs showing broker token exchange rejected … "Registration … was not found" recurring while active_sessions stays low. Fixed versions ride it out with a bounded, jittered retry of the same fresh credential (no re-registration), counted by actions_gateway_broker_token_propagation_retries_total; a listener stays online through the propagation lag. A sustained rate on that counter is expected during wide-pool bursts and is benign; investigate only if it climbs alongside agent_recycle_errors_total or active_sessions fails to recover after the burst drains.

If the burst still serializes to ~1 worker after the Q259 fix, see the next section (Q260) — a distinct duplicate-acquisition cause.

Concurrent Job Burst Serializes to ~1 Worker (Duplicate Job Acquisition)

Symptoms. As above, a whole CI matrix bursted onto the gateway serializes to roughly one running worker — but this variant persists even on a gateway with the Q259 recycle fix. The distinguishing tell is in the AGC logs: several sessions of the same RunnerGroup (distinct agentIndex / sessionId) all fail provisioning the same job with provisioner: create Secret job-<planid>: secrets "…" already exists. In GitHub's runner list the losing runners show busy but offline with no worker pod; their sessions then die. Remaining jobs sit in_progress (assigned to the now-dead duplicate runners) until GitHub's ~15-minute unstarted-job timeout.

Cause. Under a burst, GitHub's broker fans one job out to several sibling long-poll sessions of one RunnerGroup — as separate RunnerJobRequest messages with distinct RunnerRequestIDs but one shared plan ID. On gateway versions before the Q260 fix, every recipient independently ran acquirejob — succeeding and marking its runner busy — and then entered the provisioner, where the per-job worker Secret name is derived from the job's plan ID. The first session created the Secret; the rest collided (already exists), failed provisioning, and died with their runner slot already consumed. Net effect: one worker runs the job while the other slots are burned, collapsing the pool to the baseline listener. This is distinct from the Q259 post-job recycle churn (which may still be visible as a secondary 422 "currently running" signal).

A second, late-redelivery variant collides on the worker Pod rather than the Secret: provisioner: create Pod runner-<group>-<planid>: pods "…" already exists. Here the winning session already ran the job to completion, but its terminal worker pod lingers for completedPodTTL before the reaper GCs it. A gateway version that freed the plan-ID claim on completion (rather than on pod GC) would let a late GitHub redelivery of that same plan ID pass the claim gate, re-provision, and collide on the winner's still-present Completed pod.

Diagnostics.

# Multiple sessions provisioning the SAME job (the duplicate-acquisition signature)
# — the Secret variant (burst) and the Pod variant (late redelivery)
kubectl logs -n <namespace> deploy/actions-gateway-controller | grep -iE "create (Secret|Pod).*already exists|duplicate job delivery"

# Metric: actions_gateway_jobs_duplicate_delivery_total — on fixed versions, this
#   rises (deliveries safely deduplicated) INSTEAD of runner slots being burned.
# Metric: actions_gateway_active_sessions — collapses toward 1 on pre-fix versions.

Resolution. - Upgrade to a gateway version with the Q260 fix. Fixed versions dedup a job across the RunnerGroup's sibling sessions on its plan ID — the identity that collapses across the fan-out and names the worker Secret. Because the plan ID is only returned by acquirejob, a sibling still acquires, but then finds the plan ID already claimed by another session in the same AGC and skips provisioning, recycling its consumed runner back online (slot reclaimed) instead of colliding on the shared job-<planid> Secret. Each such skip increments actions_gateway_jobs_duplicate_delivery_total; a steady low rate of that counter during bursts is the fix working as intended. (The first Q260 fix keyed on RunnerRequestID before acquirejob, but siblings get distinct request ids, so it did not collapse the fan-out — upgrade past it.) - For the late-redelivery Pod variant, the same fixed versions hold the plan-ID claim for completedPodTTL past job completion — until the winner's terminal worker pod has been reaped — so a late redelivery is deduped (counted on actions_gateway_jobs_duplicate_delivery_total) rather than colliding on the lingering pod. No operator action; a lower completedPodTTL shortens both the retained pod and the claim linger together. - No operator action is needed on fixed versions — the dedup is automatic and per-RunnerGroup. If the burst still serializes with jobs_duplicate_delivery_total flat and jobs_acquired_total not climbing, the bottleneck is elsewhere (worker-node capacity, namespace ResourceQuota, or the Q259 recycle path) — work through those sections.

Known limitation — redelivery accounting. The dedup gate is post-acquirejob (the plan ID is only known then), so a deduplicated sibling has already claimed its own per-delivery job assignment at GitHub before it skips provisioning. Left untouched, GitHub still expects a runner to start that assignment and cancels the whole job at its ~15-minute unstarted-job timeout — even after the winning sibling ran the job to completion. The tell: a job whose pod logged Job completed with no renewal errors is nonetheless reported cancelled on GitHub. The mitigation (AGC_FANOUT_COMPLETION, Q260 Option A) is on by default: when the winner's job finishes, it fans a completejob out to every deduped sibling delivery — keyed on each sibling's own delivery job ID, with the winner's pod-phase-proxy result (a Failed pod → failed, else succeeded) — and a late redelivery arriving during the linger window is resolved with the same result; all tracked by actions_gateway_abandoned_delivery_completions_total. The dogfood re-route #5 experiment (2026-07-04) live-confirmed it: the run service's completion is per-delivery, not plan-ID-scoped — completejob on a sibling's own job ID resolves only that assignment (returns OK), while the winner's own delivery still carries the real workflow result reported by its runner binary, so a green sibling proxy cannot mask a red workflow. Previously-wedged concurrent jobs concluded green, the recycle 422 cleared per job on winner completion, and no job cancelled at the ~15-minute timeout. Opt out with AGC_FANOUT_COMPLETION=false. See docs/plan/q260-fanout-completion-reconciliation.md.

Related failure mode — deduped-loser slot stranding (Q266). Because a deduped sibling already ran acquirejob, GitHub holds its runner as assigned to the job. That runner's recycle therefore 422s ("runner is currently running a job and cannot be deleted") for the winner's entire runtime — the loser's 422 only clears when the winner fans completejob out to that delivery on completion (above). On gateway versions before Q266 the loser recycled eagerly, blew through the bounded recycle backoff (which is sized for the seconds-long Q259 post-job race, not a whole job runtime), gave up, and exited the listener — and a non-permanent replacement is never restarted, so under sustained fan-out burst enough losers stranded and exited to collapse the pool (observed 2/8 online at dogfood re-route #5). Fixed versions defer each loser's recycle until its winner concludes, holding the slot (but freeing its worker-capacity reservation, since the loser provisions no pod) instead of recycling into a guaranteed 422; the loser then recycles in place and resumes polling. Each defer increments actions_gateway_fanout_loser_recycle_deferred_total with outcome="winner_concluded". A sustained outcome="fallback_timeout" rate means winners are not concluding within the ~15-minute bound (a stuck-winner class — investigate the winners' pods/renewals), not a loser problem. Requires AGC_FANOUT_COMPLETION enabled (the default) — with it off, losers fall back to the eager-recycle path and the collapse can recur.

Network Connectivity Failures

Symptoms. The AGC cannot reach GitHub through the proxy. Logs show connection refused, dial tcp: i/o timeout, or proxy: no response from proxy.

Likely causes. - The proxy pod is not running or not ready. - HTTP_PROXY/HTTPS_PROXY environment variables are incorrect (wrong Service name or port). - actions-gateway-workload NetworkPolicy is blocking the AGC-to-proxy egress path (e.g. proxy ClusterIP changed after a recreate and the rule wasn't reconciled). - actions-gateway-proxy NetworkPolicy is blocking the proxy's egress to GitHub (IP ranges stale or managedNetworkPolicy: false with no replacement rule). - actions-gateway-controller NetworkPolicy is missing — AGC can't reach the K8s API server, so token refresh and webhook health checks fail before any GitHub traffic.

Diagnostics.

# Check proxy pod status
kubectl get pods -n <namespace> -l app=actions-gateway-proxy

# Verify the proxy Service exists and has endpoints
kubectl get svc -n <namespace> actions-gateway-proxy
kubectl get endpoints -n <namespace> actions-gateway-proxy

# Check the AGC container's HTTPS_PROXY env var (distroless — inspect spec, not the running process)
kubectl get pod -n <namespace> -l app=actions-gateway-controller \
  -o jsonpath='{range .items[0].spec.containers[?(@.name=="agc")].env[?(@.name=="HTTPS_PROXY")]}{.name}={.value}{"\n"}{end}'

# Test proxy connectivity using an ephemeral curl pod in the same namespace
kubectl run nettest-$$ -n <namespace> --rm -it --restart=Never \
  --image=curlimages/curl:latest \
  --overrides='{"spec":{"automountServiceAccountToken":false,"containers":[{"name":"c","image":"curlimages/curl:latest","command":["sh","-c","curl -v -x https://actions-gateway-proxy:8080 https://api.github.com 2>&1 | head -20"]}]}}'

# Check NetworkPolicy rules — there are three: workload, agc, proxy
kubectl get networkpolicy -n <namespace>
# Expected: actions-gateway-workload, actions-gateway-controller, actions-gateway-proxy
kubectl describe networkpolicy -n <namespace>

# Check the IP range refresh metric
# Metric: actions_gateway_ip_range_updates_total{namespace="<namespace>"}

Resolution. - If the proxy pod is down: check its logs and restart if necessary. - If the NetworkPolicy egress rules are stale: trigger a manual refresh by temporarily setting spec.proxy.managedNetworkPolicy: false and back to true, or wait for the 24-hour automatic refresh cycle. Check the GitHub meta API for current IP ranges: curl https://api.github.com/meta | jq .actions. - If the NO_PROXY list is missing the cluster service CIDR: update spec.proxy.noProxyCIDRs to include your cluster's service CIDR (see the noProxyCIDRs field documentation in §3.1).

AGC Cannot Reach the Kubernetes API Server (NetworkPolicy + post-DNAT port mismatch)

Symptoms. AGC logs show dial tcp 10.96.0.1:443: i/o timeout (or similar) when calling the K8s API server. The actions-gateway-controller NetworkPolicy appears to allow port 443, yet the connection is silently dropped. Most often surfaces in kind, but possible on any cluster where the kubernetes Service backends listen on a port other than 443.

Cause. NetworkPolicy enforcement evaluates packets after kube-proxy's DNAT. When a pod connects to kubernetes.default.svc (ClusterIP 10.96.0.1:443), kube-proxy DNATs the destination to the apiserver's actual Endpoints address — in kind, that's <node-ip>:6443. The policy controller sees the post-DNAT destination port (6443), and an NP rule that allows only port 443 doesn't match. This is the port-axis equivalent of the ipBlock: <ClusterIP>/32 trap that bit the proxy NP in PR #59.

Diagnostics.

# 1. Confirm the apiserver Endpoints port. If it's 6443, the AGC NP must allow 6443.
kubectl get endpointslice -n default -l kubernetes.io/service-name=kubernetes \
  -o jsonpath='{.items[0].ports[0].port}{"\n"}'

# 2. Confirm the AGC NetworkPolicy actually allows both 443 and 6443.
kubectl get networkpolicy -n <namespace> actions-gateway-controller -o yaml \
  | yq '.spec.egress[].ports[].port' | sort -u

# 3. If the cluster uses kindnet / kube-network-policies, check the verdict log
#    on the node hosting the AGC pod. Look for lines like:
#      "Pod is not allowed to connect to port" pod="<ns>/<agc-pod>" port=6443
kubectl get pod -n <namespace> -l app=actions-gateway-controller \
  -o jsonpath='{.items[0].spec.nodeName}{"\n"}'
kubectl logs -n kube-system -l app=kindnet --tail=200 --field-selector spec.nodeName=<node-name>

Resolution. Ensure buildAGCNetworkPolicy allows both port 443 (production Service shape) and port 6443 (kind / Endpoints-on-6443 clusters). The shipped policy does this. If you see this on a custom build or a hand-edited NP, add the 6443 rule. The diagnosis writeup at docs/development/networkpolicy-port-matching.md has a minimal repro and the reasoning behind allowing both ports.

If you see the same symptom for an ingress-type rule or for a different Service whose backend port differs from the Service port, the same fix applies: list both ports, or omit the port restriction on that rule.

DNS Times Out Under the Egress NetworkPolicy (GKE Dataplane V2 / NodeLocal DNSCache)

Symptoms. A tenant's AGC pod crash-loops on startup, unable to resolve api.github.com:

token fetch: Post "https://api.github.com/app/installations/<id>/access_tokens":
  dial tcp: lookup api.github.com: i/o timeout
startup failed: initial token fetch: context deadline exceeded

From a pod in the tenant namespace (which the GMC egress NetworkPolicy governs) DNS times out, while the same lookup from a pod in a namespace with no GAG policy (e.g. default) succeeds — so cluster DNS is healthy and the egress policy is the cause. Direct TCP to a GitHub IP on 443 from the tenant pod still works (the GitHub-CIDR egress rule is fine); only DNS is broken. This was first hit on the first live GAG install on GKE (Q224) running on a GKE Standard cluster with Dataplane V2 (Cilium) and NodeLocal DNSCache enabled.

Cause (Q229). On GKE Dataplane V2, NodeLocal DNSCache does not give pods a link-local resolver address — pods' resolv.conf still points at the kube-dns ClusterIP. GKE installs a RedirectService (networking.gke.io/v1alpha1, spec.redirect.type: nodelocaldns) that drives a Cilium Local Redirect: traffic to the kube-dns ClusterIP is transparently redirected to the per-node node-local-dns pod. Cilium enforces the egress NetworkPolicy against that redirect backend's identity — and the backend is node-local-dns (k8s-app: node-local-dns), not kube-dns. The GMC's DNS egress rule selected only k8s-app: kube-dns pods plus the link-local block 169.254.0.0/16; neither matches a node-local-dns pod (on Dataplane V2 it runs with -setupinterface=false, so it has a regular pod IP and no link-local address), so DNS is dropped.

A supplemental NetworkPolicy allowing egress to the kube-dns ClusterIP CIDR does not help: Cilium matches ipBlock/CIDR egress only against the external (world) identity, never against in-cluster destinations such as a ClusterIP-backed pod. The selector path is the only one that works.

Resolution. Upgrade to a GAG build that includes the fix — the GMC-generated DNS egress rule now carries a third peer selecting k8s-app: node-local-dns in kube-system, alongside the kube-dns selector and the link-local block. This is a strict, minimal widening (still cluster DNS only, port 53 only — it preserves the DNS-exfiltration containment of § Security) and is harmless on clusters without NodeLocal DNSCache, where the selector simply matches no pod. Re-validated live (2026-07-07) on a fresh GKE Standard / Dataplane V2 cluster with NodeLocal DNSCache: from a pod governed by the GMC egress NetworkPolicy, nslookup github.com resolves through the node-local-dns peer while non-DNS non-allowlisted egress stays blocked. Confirm the shipped policy:

# The DNS (port-53) egress rule must list BOTH k8s-app: kube-dns and
# k8s-app: node-local-dns as peers.
kubectl get networkpolicy <gateway>-workload -n <tenant-ns> -o yaml | grep -A3 'k8s-app'

To verify resolution end-to-end from inside the tenant's policy, run an ephemeral pod carrying the workload label and resolve through the kube-dns ClusterIP:

kubectl run dnstest -n <tenant-ns> --image=busybox:1.36 \
  --labels='actions-gateway/component=workload' --restart=Never -- sleep 300
kubectl exec -n <tenant-ns> dnstest -- nslookup api.github.com
kubectl delete pod dnstest -n <tenant-ns>

If you are pinned to an older GAG build and cannot upgrade immediately, the same effect can be obtained by setting spec.proxy.managedNetworkPolicy: false and supplying your own egress policy that allows port-53 to node-local-dns — but prefer upgrading, since a hand-managed policy must then track GitHub IP-range rotation itself.

Worker Pod Runner.Worker Fails TLS Handshake With UntrustedRoot

Symptoms. Worker pod logs (look at the runner container) contain repeated lines like:

System.Security.Authentication.AuthenticationException: The remote certificate is invalid because of errors in the certificate chain: UntrustedRoot

emitted from JobExtension connectivity checks, ResultServer init, JobServerQueue log uploads, the GitHubActionsService log-blob fetch, or RunServer.CompleteJobAsync. The runner retries for ~3 minutes, then exits 1. The AGC then logs worker pod completed phase=Failed, renewjob starts returning 401 Not authorized for this job, and the GitHub workflow concludes cancelled even though the actual job steps may have run.

Cause. Runner.Worker's .NET HttpClient is validating the egress proxy's TLS cert and the worker pod's trust store does not include the cert-manager-issued self-signed CA that signed it. This is the worker-side mirror of the AGC's proxy-CA pinning (§5.2 "Cross-Tenant Proxy CA Trust"): the AGC mounts the CA explicitly so its outbound HTTPS works; worker pods must do the same.

The AGC's pod provisioner is supposed to project the per-tenant actions-gateway-proxy-tls Secret into every worker pod at /etc/actions-gateway/proxy-ca/tls.crt and set PROXY_CA_CERT_PATH so the worker entrypoint wrapper builds a combined SSL_CERT_FILE bundle before exec'ing Runner.Worker. UntrustedRoot means one of those steps did not happen.

Diagnostics.

# 1. Inspect a failed worker pod's spec — the Secret volume must exist.
kubectl get pod -n <namespace> <worker-pod-name> -o yaml \
  | yq '.spec.volumes[] | select(.name=="proxy-ca")'
# Expected: a Secret volume with secretName: actions-gateway-proxy-tls and Items: [{key: tls.crt, path: tls.crt}]
# If empty: the AGC was deployed without PROXY_TLS_SECRET_NAME.

# 2. Confirm the AGC has the PROXY_TLS_SECRET_NAME env wired.
kubectl get pod -n <namespace> -l app=actions-gateway-controller \
  -o jsonpath='{range .items[0].spec.containers[?(@.name=="agc")].env[?(@.name=="PROXY_TLS_SECRET_NAME")]}{.name}={.value}{"\n"}{end}'
# Expected: PROXY_TLS_SECRET_NAME=actions-gateway-proxy-tls
# Empty means the GMC needs to roll the AGC Deployment (likely an upgrade across the 5h boundary).

# 3. Confirm the worker container's PROXY_CA_CERT_PATH env.
kubectl get pod -n <namespace> <worker-pod-name> -o yaml \
  | yq '.spec.containers[] | select(.name=="runner") | .env[] | select(.name=="PROXY_CA_CERT_PATH")'

# 4. Confirm the proxy TLS Secret exists and contains tls.crt.
kubectl get secret -n <namespace> actions-gateway-proxy-tls \
  -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject -issuer -dates

Resolution. - If the worker pod has no proxy-ca volume: ensure the AGC was started with PROXY_TLS_SECRET_NAME=actions-gateway-proxy-tls (the GMC injects this automatically — if it's missing, the GMC needs to roll the AGC Deployment, e.g. by bumping ag.Spec or restarting the GMC). - If the volume is present but the wrapper logs nothing about proxy CA trust installed: check that PROXY_CA_CERT_PATH is set on the runner container and the mounted file is non-empty. An empty/missing file is tolerated as a no-op, which silently leaves the runner with no proxy trust — the wrapper log line no proxy CA cert mounted; skipping trust-store install distinguishes this case from a wrapper that ran the install successfully. - If the proxy TLS Secret is missing or the cert has expired: the GMC's cert-manager integration (§2.1 "Proxy Deployer") owns rotation; check the GMC's logs for issuer errors. As a fallback, deleting the Secret triggers reissuance. - If the issue persists after the volume and env are correct: confirm the proxy pod is presenting the cert signed by the CA in the Secret — kubectl exec into a curl pod in the same namespace and run `openssl s_client -connect actions-gateway-proxy:8080 -showcerts