Appendix A — Capacity Targets & SLOs¶

← Glossary | Back to index | Next: Appendix B — Worker Isolation →

The following targets are conservative defaults derived from the architectural constraints in §2 and §3.5. They are intended as starting points to be refined against real production data; operators are expected to override them based on their cluster size, GitHub plan, and workload profile.

Latency SLOs (per-job, per-tenant)¶

Metric	Target	Source	Note
Pod-creation latency (p95)	≤ 15s	`actions_gateway_pod_creation_latency_seconds`	From worker pod creation to runner container start (scheduling + image pull). Dominated by image pull on cold nodes; sub-second on warm.
Pod-creation latency (p99)	≤ 60s	`actions_gateway_pod_creation_latency_seconds`	Tolerates cold-start image pull.
Session reacquisition after Actions Gateway Controller (AGC) restart	≤ 2 min	derived	Equal to GitHub's redelivery window; jobs redelivered within this window suffer no observable disruption.
Token refresh failure budget	< 1 / hour	`actions_gateway_token_refresh_errors_total`	Anything above this rate indicates either GitHub API instability or a credential problem.

Capacity Targets (per-AGC pod, single tenant)¶

Resource	Target	Rationale
Concurrent virtual sessions (peak burst)	≤ 1,000	Memory-bound burst ceiling: each goroutine stack + HTTP buffer + token-manager indirection averages ~60 KiB resident (a deliberately conservative sizing figure — the AGC's own per-session structures measure ~12.6 KiB on the classic tier and ~7.7 KiB per scale set on the scale-set tier, see Per-session memory & density); 1,000 sessions ≈ 60 MiB at peak. Steady-state cost is 1 session per RunnerGroup, far below this ceiling for typical deployments.
Memory request	2 GiB	Sized for the peak burst ceiling of 1,000 concurrent goroutines (~60 MiB) with 4× safety margin for Go runtime overhead, heap churn, and reconcile storms. Actual steady-state resident size will be much smaller.
Memory limit	4 GiB	Allows transient bursts during reconcile storms without triggering OOM.
CPU request	500m	Predominantly I/O-bound; request reflects baseline scheduling weight rather than steady CPU draw.
CPU limit	2 (cores)	Permits short bursts during reconcile churn or token refresh contention without throttling.

Capacity Targets (per GitHub App installation)¶

Resource	Target	Source
Concurrent sessions per installation	≤ 250	Bounded by §3.5 rate-limit math: ~72 message polls/hr/session against the 15,000/hr installation budget.
Sustained `RateLimited` condition	< 1 min	Anything longer indicates the operator is over budget and should shard across installations.

Capacity Targets (per proxy pod)¶

Resource	Target	Note
Concurrent CONNECT tunnels	≤ 500	File-descriptor-bound; tune the proxy pod `ulimit nofile` if increasing.
CPU request / limit	10m / 500m	Defaults per `ProxyConfig`. The 500m limit (not 100m) keeps the pod from throttling before the HPA's 60%-utilization signal trips under CONNECT load. Adjust upward if HPA lag is observed under bursty load.
Memory request / limit	32 MiB / 64 MiB	Stateless CONNECT proxies have a small footprint; these defaults survive 500 concurrent tunnels with headroom.

Tenant-Aggregate Capacity (single `ActionsGateway`)¶

Resource	Target	Note
Active jobs (worker pods)	≤ 250	Conservative default governed by the platform-owned namespace `ResourceQuota`, `maxWorkers`, or the last `priorityTiers` threshold — whichever is most restrictive. Not rate-limit-bounded under the adaptive listener model; increase this ceiling by adjusting the namespace ResourceQuota and per-`RunnerGroup` concurrency controls.
Aggregate namespace ResourceQuota	20 CPU / 40Gi memory / 50 pods	Conservative starting allocation. Platform-owned (set on the namespace, not the CR). Adjust against observed job CPU/memory profiles.

Per-session memory & density (measured)¶

The peak-burst sizing above uses a deliberately conservative ~60 KiB/session. To pin the AGC's actual per-session overhead — the figure behind the density-versus-pod-per-runner claim — TestAGCPerSessionMemory (cmd/agc/test/load/mem_test.go, make mem-profile) isolates it locally, with no cluster, no real broker, and no in-process broker stub (the stub inflated the earlier ~127 KiB figure because its server side runs in the same process).

Methodology. The probe drives the real multiplexing core (listener.Multiplexer + agentpool.Pool + per-goroutine broker.Client) but replaces the broker stub with memTransport, an in-process http.RoundTripper that answers the OAuth/CreateSession/GetMessage calls with canned responses and no server, socket, or per-session server-side state. GET …/message parks the caller on its request context, so each of the 1,000 started listeners rests in exactly one goroutine blocked in its long-poll — the steady idle-session state. It then takes a three-point heap+stack differential: shared infra only → plus N pooled agents and empty multiplexers → plus all N goroutines parked. The last delta (mFull − mAgents) is the marginal cost of one more concurrent session and excludes both the agent pool and the fake k8s client's retained Secrets (an apiserver-side cost in production, not AGC memory).

Result (1,000 sessions, Go on darwin/arm64):

Component	Per session
listener goroutine stack	~8.1 KiB
heap (`broker.Client` + live session state: sessionID, AES key, scoped logger)	~4.1 KiB
AGC-only total (measured)	~12.2 KiB

The pre-registered agent struct (Ed25519 key + credentials, no JIT blob in this path) adds sub-KiB on top; the agent Secret itself is apiserver-resident in production. The measured ~12.2 KiB is ~5× below the ~60 KiB design estimate — the gap is the per-connection HTTP transport buffers that an active long-poll holds in production, which the in-process transport omits. The design estimate is therefore confirmed as a conservative upper bound.

Re-measured 2026-09-12 on the same harness and machine: 12.57 KiB/session (8.32 KiB stack, 4.25 KiB heap), so the figure is stable.

The scale-set tier measures a different unit, and a smaller one¶

The probe above drives the classic tier, where one listener goroutine holds one virtual runner session, so per-session and per-goroutine are the same quantity. The scale-set tier is the default protocol and had never been measured (Q722). TestScaleSetPerListenerMemory (cmd/agc/test/load/scaleset_mem_test.go, make scaleset-mem-profile) isolates it the same way, against scalesetMemTransport rather than scalesettest, whose httptest.Server would hold a parked goroutine and its read/write buffers per session in the same process.

The unit differs, which is the substance rather than a detail. One scale-set Listener holds one scale set's acquisition session and multiplexes every job assigned to that set through it, so the resident cost scales with the RunnerSet count, not with the number of concurrent jobs.

Result (200 scale sets, Go on darwin/arm64, 2026-09-12):

Component	Per scale set
listener goroutine stack	~4.8 KiB
heap (live session state)	~2.9 KiB
AGC-only total (measured)	~7.7 KiB

The scaleset.Client and Listener structs add ~1.2 KiB per set, built before any session opens and so held out of the marginal figure.

So the scale-set tier is cheaper per session and needs far fewer sessions: a tenant running one RunnerSet holds one long poll however many jobs are in flight, where the classic tier holds one per concurrent runner. The probe asserts every one of its 200 polls is actually parked before sampling, since a goroutine count cannot tell a resting long poll from one spinning in a backoff retry and the figure divides by the set count.

Density versus ARC — a pod-count argument, not a memory ratio. The honest comparison against ARC scale-set mode is structural: ARC runs one always-on listener pod per scale set (a Go binary, cmd/ghalistener in actions/actions-runner-controller, built on the same official github.com/actions/scaleset client library this repo tracks), each costing a pod slot, a cluster IP, a scheduling unit, an image pull, an upgrade surface, and a Go runtime baseline. GAG runs every listener as a goroutine in one shared AGC pod per tenant — N runner sets is N pods and N cluster IPs there, 1 pod and 1 cluster IP here, at ~7.7 KiB of measured AGC state per scale set on the default scale-set tier (~12.6 KiB per session on the classic tier).

Why no memory ratio is published. Earlier revisions published a "~4,000×" figure derived from a "~256 MiB .NET listener" baseline. That baseline was retired (#781): ARC's scale-set listener is the Go ghalistener, not the .NET Runner.Listener (which runs inside the runner pod, a different component), and the gha-runner-scale-set chart ships no default listener resource requests or limits — so there was no measured denominator to ratio against. If a memory comparison is reintroduced, it must benchmark an actual ARC ghalistener pod at a stated ARC version and cite that measurement.

These numbers should still be re-derived once two consecutive weeks of production telemetry are available. Treat the locally-measured figures as validated lower bounds on efficiency, not as a production-scale contract.

Validation status. The session-multiplexing core has been load-tested, and its per-session memory is now pinned. The in-process harness (cmd/agc/test/load/, Q13; make load-test-quick) holds ~1,000 concurrent virtual sessions in a single AGC — a representative run sustained avg 998/1,000 with zero goroutine leak and 1.0 re-registrations per job (the single-use model under load). The faithful results from that tier are the sustained-session count, the no-leak guarantee, and the re-registration rate; it deliberately stubs the apiserver, registrar, and broker, so it does not speak to real apiserver/GitHub latency or worker-pod scheduling. The earlier ~127 KiB/session figure was an upper bound inflated by the in-process broker stub; the stub-free probe above (Q181) isolates the AGC's own structures at ~12.2 KiB/session, well under the ~60 KiB conservative design bound. One caveat remains: the real-cluster, real-GitHub scale run — worker-pod scheduling and cross-tenant network at full concurrency — is still deferred. Operators should size against their own observed telemetry rather than treat these ceilings as proven.