Skip to content

Production Runbook

Audience: SRE

For initial setup steps see Getting Started. For detailed symptom → diagnosis steps see Troubleshooting.

Day-2 Operations

Adding a Tenant

  1. Ensure the tenant namespace exists: kubectl get namespace <namespace>.
  2. Have the tenant create the GitHub App Secret in their namespace. See Getting Started §3.
  3. Have the tenant create the gateway CR(s) — the recommended v2 ActionsGateway + RunnerSet, or the legacy v1 ActionsGateway. See Getting Started §4.
  4. Confirm the GMC has provisioned resources within ~30 seconds: 
    kubectl get actionsgateway -n <namespace>
kubectl get deploy,hpa,networkpolicy,resourcequota -n <namespace>
  5. Confirm the Ready=True condition on the ActionsGateway CR.

No cluster-admin involvement is required after initial GMC deployment.

Adjusting Tenant Quota

The namespace ResourceQuota is platform-owned — it is not a field on the ActionsGateway CR. Edit the ResourceQuota object on the tenant namespace directly (or through your GitOps / tenant-operator stack, if that is what manages it):

kubectl edit resourcequota -n <namespace> <quota-name>
# Update spec.hard values, save and exit

The change takes effect immediately. Running jobs are not interrupted; the new quota applies on the next pod creation attempt. The gateway reads remaining quota and reacts to exhaustion (it fast-cancels and reruns quota-blocked jobs) but never writes the quota itself.

Scaling maxListeners

kubectl edit actionsgateway -n <namespace> <name>
# Update spec.runnerGroups[N].maxListeners

The GMC propagates the change to the RunnerGroup CR. The AGC reconciles the new ceiling on its next reconcile cycle (a few seconds). No restart needed.

Rotating GitHub App Credentials

See Getting Started — Rotating GitHub App Credentials for the full procedure.

In brief: create a new Secret with the new private key, then change spec.gitHubAppRef.name in the ActionsGateway CR to reference the new Secret. The GMC detects the Secret reference change and rolls the AGC Deployment. Do not update the existing Secret in-place; the GMC does not watch Secret contents, only the reference.

Alerting

Reference the SLO targets in Appendix A for threshold derivation.

The alerts below cover availability and SLO breaches. For abuse and compromise detection (eviction-retry loops, proxy slowloris, credential harvesting), see security-operations.md.

Which Metrics to Alert On

Metric Recommended threshold Severity Notes
actions_gateway_token_refresh_errors_total rate > 1/hour per namespace Page Token expiry causes session failures within ~1 hour
actions_gateway_renew_job_errors_total rate > 5/minute per namespace Page Sustained failures cancel running jobs
actions_gateway_pod_creation_latency_seconds p95 > 15s Ticket SLO target from Appendix A
actions_gateway_pod_creation_latency_seconds p99 > 60s Page Indicates scheduling stall or quota exhaustion
actions_gateway_eviction_retries_exhausted_total rate > 0 Ticket Each increment requires a manual re-run
actions_gateway_active_sessions = 0 for a RunnerGroup Page No listener polling; jobs queue indefinitely
controller_runtime_reconcile_errors_total rate > 1/5min Ticket Persistent reconcile failure; resources may be stale
ActionsGateway condition RateLimited=True duration > 10 minutes Page Installation is over API budget
Proxy HPA TARGETS: <unknown> any Ticket HPA metric broken; autoscaling not working
AGC pod OOMKilled any Page AGC has no active sessions while restarting

Page-Worthy vs. Ticket-Worthy

Page (requires immediate response, typically < 15 minutes): - active_sessions = 0 — no jobs can be acquired until fixed. - renew_job_errors_total rate high — jobs will be cancelled. - token_refresh_errors_total spiking — token will expire within ~1 hour. - pod_creation_latency p99 > 60s — scheduling is stalled. - RateLimited condition > 10 minutes — installation is over budget. - AGC pod in OOMKilled / CrashLoopBackOff.

Ticket (respond within next business day): - pod_creation_latency p95 > 15s — degraded but jobs are completing. - eviction_retries_exhausted_total incrementing — jobs require manual re-run. - reconcile_errors_total non-zero — investigate before it becomes a page. - HPA metric unknown — autoscaling broken; proxy may not handle burst load.

Alert Rule Reference

Every alert shipped in the reference PrometheusRule (reproduced in Recommended Alert Rules) carries a runbook_url annotation that resolves to the matching entry below, so on-call lands on a response procedure rather than just the alert's summary/description. Severity classes follow Page-Worthy vs. Ticket-Worthy.

ActionsGatewayNoActiveSessions

Page. The AGC has no open long-poll sessions, so no jobs are acquired and the queue backs up indefinitely. Restore sessions with active_sessions Flatlining at Zero.

ActionsGatewayTokenRefreshErrors

Page. GitHub App token refresh has been failing; sessions will fail once the current token expires (~1 hour). See Token Refresh Errors Spiking.

ActionsGatewayRenewJobErrors

Page. RenewJob is failing at a sustained rate, so running jobs may be cancelled by GitHub. See RenewJob Failures Rising.

ActionsGatewayPodCreationLatencyP99

Page. p99 pod-creation latency has breached the 60s SLO, indicating a scheduling stall or quota exhaustion. Triage with pod_creation_latency_seconds p95 > 15s.

ActionsGatewayPodCreationLatencyP95

Ticket. p95 pod-creation latency has breached the 15s SLO — degraded but jobs still complete. Triage with pod_creation_latency_seconds p95 > 15s.

ActionsGatewayEvictionRetriesExhausted

Ticket. A job's eviction-retry budget is exhausted and the job requires a manual re-run. See Evicted Worker Pods Exhausting Retry Budget.

ActionsGatewayQuotaRetriesExhausted

Ticket. A job's quota-retry budget is exhausted — worker pod creation kept being rejected by the namespace ResourceQuota — and the job was abandoned and requires a manual re-run. Raise the quota or lower maxWorkers — see Jobs Failing Due to Namespace ResourceQuota Exhaustion and Adjusting Tenant Quota.

ActionsGatewayWorkerQuotaExceeded

Page. The namespace ResourceQuota is rejecting worker pods, so acquired jobs cannot schedule. Raise the quota or lower maxWorkers — see Jobs Failing Due to Namespace ResourceQuota Exhaustion and Adjusting Tenant Quota.

ActionsGatewayProxyQuotaExceeded

Page. The ResourceQuota is holding the egress proxy pool below the HPA's target. Raise the quota or lower proxy.maxReplicas — see Proxy Pool Not Scaling.

ActionsGatewayQuotaPressure

Ticket. A proxy or worker pool cannot reach its configured ceiling within the namespace ResourceQuota headroom. Plan a quota increase before the next load spike — see Adjusting Tenant Quota and Jobs Failing Due to Namespace ResourceQuota Exhaustion.

ActionsGatewayReconcileErrors

Ticket. A controller is logging sustained reconcile errors and owned resources may be stale. See GMC Not Provisioning Tenant Resources.

ActionsGatewayWorkersUnschedulable

Page. Worker pods are stuck Pending past the scheduling grace because the scheduler cannot place them (no matching node / affinity / taints — not quota); capacity is not materializing. See RunnerGroup Reports WorkersUnschedulable and Worker Pods Stuck Pending.

ActionsGatewayEgressRulesStale

Page. The gateway's GitHub egress IP-range allowlist has not refreshed within the staleness window; the proxy NetworkPolicy may drift from GitHub's published ranges. See ActionsGateway Reports EgressRulesStale.

ActionsGatewayAgentRecycleErrors

Ticket. Single-use JIT agent re-registration is failing; sustained growth shrinks listener capacity and decays tenant throughput job by job. See Concurrent Job Burst Serializes to ~1 Worker (Recycle Blocked on a Still-Running Runner).

ActionsGatewayFanoutFallbackTimeout

Ticket. Deduped fan-out losers are recycling on the fallback timeout because their winner never concluded within the bound — a class of stuck winners. Investigate long-running or wedged winning jobs; see Concurrent Job Burst Serializes to ~1 Worker (Duplicate Job Acquisition).

ActionsGatewayAbandonedDeliveryErrors

Ticket. The winner of a fanned-out job is failing to issue completejob on a deduped sibling delivery; affected jobs may be cancelled at GitHub's ~15-minute unstarted-job timeout. See Concurrent Job Burst Serializes to ~1 Worker (Duplicate Job Acquisition).

ActionsGatewayScaleSetProvisioningStalled

Page. The scale-set acquisition tier (the default protocol) is receiving JobAssigned messages but has provisioned no worker pods — the tier is wedged and acquired jobs will not start. A ScaleSet-protocol RunnerSet emits no actions_gateway_active_sessions, so this demand-vs-supply signal is the scale-set analog of active_sessions Flatlining at Zero. Respond with Scale-set provisioning stalled.

ActionsGatewayScaleSetProvisionErrors

Ticket. The scale-set tier is failing to provision worker pods (JIT-config mint or pod create) at a sustained rate. A transient failure retries on a later poll; a sustained rate means provisioning is degraded. Check the run service's generate-jitconfig responses and namespace quota headroom, then triage as for Scale-set provisioning stalled.

ActionsGatewayProxyConnectDenied

Ticket. The egress proxy is refusing CONNECT requests to destinations off the egress allowlist at a sustained rate — a Server-Side Request Forgery (SSRF) / egress-policy signal. Every increment is an explicit allowlist denial (sharper than dial_errors, which also counts transient failures to allowed hosts).

  1. Identify the denied destinations from the proxy's CONNECT destination not allowed warning logs, which record the rejected host: kubectl logs -n <namespace> deploy/<proxy-deployment> | grep "destination not allowed".
  2. If the destinations are unexpected, treat it as SSRF probing or a compromised workload — inspect the workflow acquiring the affected runner and correlate with proxy_dial_errors_total.
  3. If a legitimate egress target is missing, add it to the platform-owned egress allowlist rather than to the workflow — see security-operations.md § Worker egress destinations.

SLO Breach Response

pod_creation_latency_seconds p95 > 15s

  1. Check for quota exhaustion: kubectl describe resourcequota -n <namespace>.
  2. Check for pending pods: kubectl get pods -n <namespace> | grep Pending.
  3. Describe a pending pod for scheduling events: kubectl describe pod -n <namespace> <pod>.
  4. If quota is exhausted: raise the platform-owned ResourceQuota on the namespace (kubectl edit resourcequota -n <namespace> <quota-name>) or wait for running pods to complete.
  5. If no schedulable nodes: check node autoscaler or provision capacity.
  6. If PriorityClass is missing: create it. See Troubleshooting — Worker Pods Stuck Pending.

active_sessions Flatlining at Zero

  1. Check AGC pod status: kubectl get pod -n <namespace> -l app=actions-gateway-controller.
  2. Check AGC logs: kubectl logs -n <namespace> deploy/actions-gateway-controller --tail=100.
  3. Check RunnerGroup conditions: kubectl get runnergroup -n <namespace> -o yaml.
  4. If pod is CrashLoopBackOff or Error: see Troubleshooting — AGC CrashLoopBackOff.
  5. If pod is running but sessions are zero: check for token errors (see Token Refresh Errors) and network connectivity (see Network Connectivity Failures).

jobs_acquired_total Stops Incrementing

  1. Verify jobs are actually queued: check the GitHub Actions UI for the repository.
  2. Check active_sessions — if zero, restore sessions first (see above).
  3. Check RateLimited condition — if true, reduce session load or wait for the burst to subside.
  4. Check message_poll_errors_total — persistent poll errors indicate a broken GitHub connection.
  5. If sessions are active and no errors, the queue may simply be empty.

Scale-set provisioning stalled

The scale-set tier is the default protocol; it emits no active_sessions gauge, so a wedge shows up as scaleset_jobs_assigned_total climbing while scaleset_jobs_provisioned_total stays flat.

  1. Confirm the wedge: scaleset_jobs_assigned_total is rising but scaleset_jobs_provisioned_total is not, for the affected namespace/runner_set.
  2. Check the provision-error rate: rate(actions_gateway_scaleset_provision_errors_total[5m]). A non-zero rate points at JIT-config mint or pod-create failures — inspect the AGC logs (kubectl logs -n <namespace> deploy/actions-gateway-controller --tail=100) for generate-jitconfig errors and worker-pod create rejections.
  3. Check the worker-pod ResourceQuota (see Adjusting Tenant Quota) and the WorkerQuotaExceeded / WorkersUnschedulable conditions — a full quota or an unschedulable pod stalls provisioning with no provision error.
  4. If provision errors are zero and quota is healthy, check the listener session itself: the RunnerSet's Ready/RateLimited/Degraded conditions (kubectl get runnerset -n <namespace> -o yaml) surface a rate-limited or unauthorized scale-set session (Q325).
  5. If the queue is genuinely empty, scaleset_jobs_assigned_total will also be flat — the alert only fires when assignment is active, so a flat-both state is benign.

Incident Response

For restoring a deleted or corrupted ActionsGateway CR, a lost tenant namespace, or a full cluster, see the dedicated Backup, Restore, and Disaster Recovery guide. The CR is the source of truth; deleting it cascades to the resources the GMC owns, and re-applying it reconciles them back.

GitHub App Key Compromise

Immediate steps (< 5 minutes):

  1. Revoke the compromised private key in the GitHub App settings (Settings → Developer settings → GitHub Apps → <app> → Private keys → Revoke).
  2. The AGC's token refresh will fail within minutes of revocation; sessions will become invalid.

Restoration steps:

  1. Generate a new private key from the GitHub App settings page and download the .pem file.
  2. Create a new Secret with the new key: 
    kubectl create secret generic <new-secret-name> \
  --from-literal=appId=<appId> \
  --from-literal=installationId=<installationId> \
  --from-file=privateKey=<path-to-new-key.pem> \
  -n <namespace>
  3. Update the ActionsGateway CR to reference the new Secret: 
    kubectl patch actionsgateway -n <namespace> <name> \
  --type=merge -p '{"spec":{"gitHubAppRef":{"name":"<new-secret-name>"}}}'
  4. Confirm the AGC Deployment has rolled and the new pod is healthy: 
    kubectl rollout status deploy/actions-gateway-controller -n <namespace>
  5. Confirm actions_gateway_token_refresh_errors_total is no longer incrementing.
  6. Delete the old Secret once confirmed healthy, and delete the downloaded .pem file from disk (shred -u <path> on Linux, rm -P <path> on macOS) — the key now lives only in the Kubernetes Secret.

Scope assessment. The compromised key could have been used to acquire installation tokens (scoped to Actions: Read, Administration: Read). Check GitHub's audit log for unusual API activity from the App installation: Settings → Organizations → <org> → Audit log → filter by the App name.

AGC Total Failure

If the AGC pod is destroyed and cannot restart (e.g. node failure without rescheduling, OOM loop):

  1. In-flight jobs whose renewjob loop has lapsed will be cancelled by GitHub. There is no automatic recovery for these — they require manual re-run.
  2. Queued jobs (not yet acquired) will be redelivered by GitHub to the next healthy session within ~2 minutes of the AGC restarting.
  3. To force restart: kubectl rollout restart deploy/actions-gateway-controller -n <namespace>.
  4. Monitor actions_gateway_active_sessions — it should reach 1 per RunnerGroup within a few seconds of the pod starting.

State that persists: All RunnerGroup CRs, Secrets, and Kubernetes resources are durable. The AGC reconstructs all in-memory state (session registry, per-job renewers) from scratch on restart. The only non-recoverable state is in-flight job locks that expire during the blackout window.

GMC Total Failure

If the GMC pod is unavailable:

  1. Existing tenant gateways continue operating normally. The GMC is not in the data plane; it only responds to ActionsGateway CR changes. Provisioned AGCs, proxies, and RunnerGroups are not affected.
  2. New ActionsGateway CRs will not be provisioned until the GMC recovers.
  3. Spec changes to existing ActionsGateway CRs will not be reconciled until the GMC recovers.
  4. To restore: kubectl rollout restart deploy/gmc-controller-manager -n gmc-system.
  5. On recovery, the GMC reconciles all ActionsGateway CRs idempotently — it compares desired vs. actual state and only applies changes. No resources are duplicated or deleted.

On-Call Handoff Checklist

Before handing off to the next on-call:

  • [ ] All ActionsGateway conditions Ready=True across active tenant namespaces.
  • [ ] No sustained RateLimited conditions.
  • [ ] active_sessions > 0 for all active RunnerGroups.
  • [ ] token_refresh_errors_total rate is zero (or below 1/hour).
  • [ ] renew_job_errors_total rate is zero.
  • [ ] No pods in CrashLoopBackOff or OOMKilled state.
  • [ ] No open incidents or unresolved pages.
  • [ ] Any eviction_retries_exhausted_total increments from the shift are documented and re-runs are queued.