Appendix H — v2 API Decomposition¶
← Optional Future Enhancements | Back to index
Status: shipped —
v2alpha1, served besidev1alpha1. This appendix is the design source of truth for thev2alpha1API (actions-gateway.comgroup) that replaces the monolithicActionsGateway+RunnerGroupmodel. All milestones M1–M5 have landed: the five kinds, their GMC/AGC reconcilers, multiple gateways per namespace, the namespace-scoped security profile, and the one-shot v1→v2 migration tool are all built.v2alpha1is an alpha API served side by side withv1alpha1during the coexistence window — nothing in the shippedv1alpha1API changes, and tenants migrate on their own schedule via the migration tool, a deliberate tool-assisted fan-out (see §H.11). Milestone sequencing and the itemized task record are in the v2 API plan.
H.1. Why decompose¶
Three independent pressures all trace back to the same root cause — the current API is a monolith that aggregates large pod templates, couples the egress proxy to the tenant 1:1, and assumes one gateway per namespace.
-
Pod templates are growing past comfortable object sizes. Docker-in-Docker and sysbox runner images need large
PodTemplateSpecs (init containers, sidecars, volumes, security context). Today the template is embedded inRunnerGroupSpec, and a bootstrap copy can also be embedded inline inActionsGateway.spec.runnerGroups[]. Several runner groups, each with a fat template, marshalled into one CR is the failure mode that approaches etcd's ~1.5 MB per-object limit. The limit is per object, so the fix is to stop the aggregation and enable reuse, not to shrink fields. -
Multiple runner sets should be able to share one egress proxy. The proxy is currently created per
ActionsGatewayand is not independently addressable. There is no way to point several runner sets at one proxy pool, or to run a single platform-operated pool. -
Tenants want everything in one namespace and the freedom to rebalance. The current rule is one
ActionsGatewayper namespace. A tenant running several GitHub orgs, or wanting to shufflemaxWorkers/priorityTiersacross runner sets against a single namespaceResourceQuota, cannot do so without spreading across namespaces.
Non-goals¶
- Not a behavior change. v2 re-shapes the API; the runtime semantics (job
acquisition, worker provisioning, quota/PSA enforcement, egress restriction) are
preserved.
v2alpha1tracks v1 behavior wherever a field is unchanged. - Not an in-place conversion. The v1→v2 split is a fan-out handled by a migration tool (§H.11), not a conversion webhook.
- Not the admin policy layer. Tiered admin policy (singleton/class) is explicitly deferred (§H.14).
- Not cross-namespace sharing on day one. Same-namespace
EgressProxysharing ships first; cross-namespace consent + CA distribution follow on demand (§H.9).
Risks¶
- Two APIs in flight. Serving
v1alpha1+v2alpha1means dual maintenance until v1 removal — bounded by the coexistence window and the behavior-parity non-goal above. - Migration is fan-out-on-create. Operators run a deliberate one-shot tool, not a silent upgrade — mitigated by dry-run-by-default and a documented runbook.
- Multi-gateway naming collisions. Per-gateway derived names under a 52-char
cap (§H.6); the webhook-enforced
maxLengthmakes the limit discoverable, not a runtime surprise.
H.2. Design principles¶
- Split data objects from controller objects. Two kinds are reconciled into running infrastructure (verbs); two are reusable data referenced by name (nouns). The large pod template lives only in a data object, so it never co-bloats anything that gets aggregated.
- Reference, don't embed; reference, don't own. Shared objects are named by reference. Referrers never set owner references on referents (that would cascade-delete a shared object when any one referrer goes away).
- GitOps-friendly: no apply-ordering requirements. Referential integrity is a runtime condition, not an admission gate. Applying a whole directory at once must converge regardless of object order.
- Secure by default, optional by layering. The egress proxy becomes optional to lower onboarding cost, but the egress restriction it sat on top of stays mandatory. Cross-namespace sharing requires explicit provider consent.
- Simplest shape that solves a problem we have today; forward-compatible for the rest. Build only the abstraction a current pressure forces. Where a future need is foreseeable, shape the schema so the abstraction is additive when its trigger fires — a new kind, or a new optional field with a default — never a second breaking migration. Deferred abstractions are recorded with the concrete trigger that would revive them (the admin policy layer in §H.14 is the worked example).
H.3. The CRD set¶
Two controller kinds (ActionsGateway, RunnerSet) and two data kinds
(RunnerTemplate/ClusterRunnerTemplate, EgressProxy). Boxes are kinds;
arrows are references (a RunnerSet points at the objects it uses). Per-kind
fields are in §H.4.
┌──────────────────────────────┐
│ ActionsGateway │ GitHub binding + AGC control plane
│ (1..N per namespace) │
└──────────────▲───────────────┘
│ gatewayRef
┌──────────────┴───────────────┐
│ RunnerSet │ scheduling / quota (small object)
│ scheduling / quota │
└──────┬────────────────┬──────┘
│ templateRef └───────────────────────┐ proxyRef? (optional;
▼ ▼ else gateway.defaultProxyRef)
┌──────┴───────────────────────┐ ┌────────────┴─────────────────┐
│ RunnerTemplate / │ │ EgressProxy (optional) │
│ ClusterRunnerTemplate │ │ shared egress proxy pool │
│ pod shape (large); reusable │ │ sharing? → cross-ns consent │
└──────────────────────────────┘ └──────────────────────────────┘
This mirrors the Gateway API pattern (GatewayClass → Gateway → route
attachment by reference) and ARC's split of scheduling (AutoscalingRunnerSet)
from pod shape, rather than introducing a novel structure.
Runtime view — what the kinds become¶
The diagram above is the static shape; this is what those kinds reconcile into at runtime. The GMC (cluster-scoped) provisions the per-tenant control plane; the AGC (one per gateway, in the tenant namespace) provisions worker pods; all GitHub traffic egresses through the proxy pool's stable per-tenant IP.
GMC · cluster controller
│ reconciles ActionsGateway → AGC Deployment ┐ creates
│ reconciles EgressProxy → Proxy pool ┘ + owns
▼
┌──────────────────────┐ reconciles RunnerSet ┌──────────────────────┐
│ AGC · per-gateway │ ──── → creates pods ───► │ Worker pods │
│ controller │ (one per job) │ ephemeral · per job │
└──────────┬───────────┘ └──────────┬───────────┘
│ control-plane long-poll │ job egress
└──────────────────────┬──────────────────────────┘
▼
┌────────────────────────────┐ stable ┌──────────────────┐
│ Proxy pool │ per-tenant │ GitHub │
│ routes all GitHub egress │ ─── IP ───► │ broker + API │
└────────────────────────────┘ └──────────────────┘
Multiple ActionsGateways may share one namespace; each AGC reconciles only the
RunnerSets whose gatewayRef targets it.
H.4. Spec sketches¶
// ActionsGateway — GitHub identity + AGC control plane only.
// Now permitted 1..N per namespace.
type ActionsGatewaySpec struct {
Credentials GitHubCredentials `json:"credentials"` // discriminated union: githubApp today, workloadIdentity additive (Q196/Q197, §H.15)
GitHubURL string `json:"githubURL"` // immutable (CEL oldSelf)
DefaultTemplateRef *ObjectRef `json:"defaultTemplateRef"` // optional (Q172): inherited by RunnerSets with no templateRef
Tracing TracingConfig `json:"tracing"` // unchanged
// REMOVED vs v1alpha1: SecurityProfile string → PSA is namespace-scoped, so it
// is owned at the namespace (GMC-guarded), not per gateway. See §H.16 #7.
// (Fallback if co-located differing profiles are ever needed: keep it here and
// resolve by most-restrictive-wins.)
// DefaultProxyRef names an EgressProxy used for AGC control-plane egress and
// inherited by RunnerSets that do not set their own proxyRef. Optional:
// unset means the control plane egresses directly (subject to NetworkPolicy).
// Same-namespace unless the target EgressProxy grants cross-namespace use.
DefaultProxyRef *ObjectRef `json:"defaultProxyRef,omitempty"`
// AGCResources optionally tunes the per-gateway AGC container CPU/memory
// requests/limits (Q171). Additive, per-key overlay of the platform default
// (requests cpu:500m/mem:2Gi, limits cpu:2/mem:4Gi — the Appendix A sizing);
// unset ⇒ that default unchanged. See §H.4 note below.
AGCResources *corev1.ResourceRequirements `json:"agcResources,omitempty"`
// REMOVED vs v1alpha1: Proxy ProxyConfig → standalone EgressProxy
// REMOVED vs v1alpha1: RunnerGroups []RunnerGroupSpec → explicit RunnerSet objects
}
// EgressProxy — standalone, optionally shared proxy pool (was ActionsGateway.spec.proxy).
type EgressProxySpec struct {
MinReplicas, MaxReplicas *int32
TargetCPUUtilizationPercentage *int32
Resources corev1.ResourceRequirements
NoProxyCIDRs []string
ManagedNetworkPolicy *bool
// LogLevel is the per-pool verbosity knob (info|debug, default info) — v1
// parity (Q327). Threaded to the proxy container as LOG_LEVEL; changing it
// rolls the pool. In v1 the single ActionsGateway.spec.logLevel covered both
// the AGC and its inline proxy; in v2 the decomposed kinds each carry their
// own (ActionsGateway.spec.logLevel for the AGC, this field for the pool).
// +kubebuilder:validation:Enum=info;debug
// +kubebuilder:default=info
LogLevel string `json:"logLevel,omitempty"`
// EgressPolicyMode is TENANT INTENT for how the GMC expresses the GitHub egress
// allowlist: CIDR (default; standard NetworkPolicy + 24h IPRangeReconcile, works on
// every CNI) or FQDN (by hostname, via a CNI-native DNS-aware policy). For FQDN
// intent the OPERATOR picks the mechanism with the GMC --fqdn-policy-backend flag
// (none|cilium|calico|gke) — none (default) rejects FQDN intent at admission. The
// deprecated CiliumFQDN/CalicoFQDN values pin their namesake backend (Q245). FQDN
// modes are fail-closed: the standard NetworkPolicy still default-denies GitHub
// egress, so an unenforced FQDN policy keeps egress denied rather than opening it
// (Q208). No effect when ManagedNetworkPolicy is false.
// +kubebuilder:validation:Enum=CIDR;FQDN;CiliumFQDN;CalicoFQDN
// +kubebuilder:default=CIDR
EgressPolicyMode EgressPolicyMode `json:"egressPolicyMode,omitempty"`
// Sharing controls cross-namespace reference. nil ⇒ same-namespace only
// (default, secure). Consent lives on the provider (proxy owner) side.
Sharing *ProxySharing `json:"sharing,omitempty"`
}
type ProxySharing struct {
// AllowedNamespaces lists namespaces permitted to reference this proxy.
// Alternatively a NamespaceSelector may be offered.
AllowedNamespaces []string `json:"allowedNamespaces,omitempty"`
}
// RunnerTemplate (namespaced) and ClusterRunnerTemplate (cluster-scoped, identical
// spec) — the only object permitted to be large. Isolated so it never co-bloats
// a controller object, and reusable across many RunnerSets. The cluster-scoped
// variant lets the platform own golden privileged templates (DinD/sysbox) once.
type RunnerTemplateSpec struct {
PodTemplate corev1.PodTemplateSpec // the big field
WorkerImage string
// Reserved-pod-field rejection (serviceAccountName, hostPID/hostNetwork/hostIPC,
// automountServiceAccountToken, proxy env vars) moves to THIS object's webhook.
}
// RunnerSet — small scheduling/quota binder (was RunnerGroup; podTemplate removed,
// references added). See H.6 for the rename rationale.
type RunnerSetSpec struct {
GatewayRef ObjectRef // which GitHub connection (was implicit via namespace)
TemplateRef *ObjectRef // RunnerTemplate | ClusterRunnerTemplate; optional (Q172): unset ⇒ gateway.defaultTemplateRef ⇒ the single cluster-default ClusterRunnerTemplate ⇒ TemplateNotFound
ProxyRef *ObjectRef // EgressProxy; nil ⇒ gateway.defaultProxyRef; both nil ⇒ direct egress
RunnerLabels []string
MaxListeners int32
MaxWorkers *int32
PriorityTiers []PriorityTier
// lifecycle tunables (eviction/quota retries, TTLs, deadlines) — unchanged from RunnerGroup
}
Why templateRef and proxyRef are both optional — but resolve differently.
They look parallel but the fallback differs. An unset proxyRef has a well-defined
behavior — direct egress, still NetworkPolicy-restricted — so the dependency can
simply be dropped (Q168, shipped): both proxyRef and defaultProxyRef unset
resolves to direct egress, with proxyMode: Direct + an EgressUnattributed
condition in status. A RunnerSet with no template has no such drop-the-dependency
fallback — the AGC cannot synthesize a worker pod without a pod shape — so instead of
a behavior it resolves a default template (Q172, shipped). templateRef was
required at GA; it has been relaxed to optional-with-a-default — a backward-compatible
required → optional change, so a set that sets templateRef behaves exactly as before.
The resolution chain for an unset templateRef (runtime, fail-closed, §H.7) is:
rs.spec.templateRef(explicit) —status.templateSource: TemplateRef.- else
ActionsGateway.spec.defaultTemplateRef— per-gateway default (may name aRunnerTemplateor aClusterRunnerTemplate);templateSource: GatewayDefault. - else the single cluster-default
ClusterRunnerTemplate— the one markedactions-gateway.com/is-default-template: "true"(theStorageClassdefault-class pattern);templateSource: ClusterDefault. - else
Ready=False/TemplateNotFound— fail-closed, no worker wiring, never a synthesized phantom pod.
At most one cluster-default — enforced at runtime, not admission. The marker lives
only on the cluster-scoped ClusterRunnerTemplate (platform-authored: a tenant cannot
self-elect a namespaced RunnerTemplate cluster-wide). If two are marked, the AGC
fails closed Ready=False/AmbiguousDefault (naming the conflicts) rather than
silently picking one — stricter than upstream StorageClass's newest-wins. The
≤1 invariant is checked at resolution time in the AGC reconciler, not at admission,
for the same reason all reference integrity is runtime here (§H.7): it is a cross-object
invariant single-object CEL cannot express, and an admission reject would break GitOps
apply-ordering. The trade-off — admission would give earlier feedback — is accepted; the
condition surfaces the moment a RunnerSet actually depends on the ambiguous default,
and clears the moment one default is demoted (the ClusterRunnerTemplate watch
re-enqueues). A defaultTemplateRef/templateRef that names a missing template still
fails closed (TemplateNotFound), exactly like a missing proxy fails closed
(ProxyNotFound); only an entirely-unset reference falls through to the next rung.
Per-gateway AGC resources — agcResources (Q171, shipped). The AGC control-plane container is sized by an optional ActionsGateway.spec.agcResources of the standard corev1.ResourceRequirements shape. It is an additive, per-key overlay of the platform default — the Appendix A sizing (requests {cpu: 500m, memory: 2Gi}, limits {cpu: 2, memory: 4Gi}): the GMC stamps the default and replaces only the request/limit keys the tenant sets, so an unset field reproduces the default unchanged (non-breaking) and a value that sets one knob keeps the default for the rest. There is no admission-time floor on the values — sizing guidance and the recommended floor (don't set a memory limit below the working set; don't request more than a node/quota can schedule) are operator-owned in tenant-onboarding. v1alpha1 has no equivalent field; its AGC carries no GMC-stamped resources (unchanged).
Worked example — minimal proxy-less onboarding (three objects)¶
apiVersion: actions-gateway.com/v2alpha1
kind: ActionsGateway
metadata: { name: acme, namespace: team-a }
spec:
credentials: # discriminated union (Q196, §H.15)
type: GitHubApp # discriminator; workloadIdentity is the additive 2nd member (Q197)
githubApp: { name: acme-github-app } # LocalSecretReference, same namespace
githubURL: https://github.com/acme
# Pod Security level is owned at the namespace (GMC-guarded), not here — see §H.16 #7.
---
apiVersion: actions-gateway.com/v2alpha1
kind: RunnerTemplate
metadata: { name: default, namespace: team-a }
spec:
podTemplate:
spec:
containers:
- name: runner
resources: { requests: { cpu: "1", memory: 2Gi } }
---
apiVersion: actions-gateway.com/v2alpha1
kind: RunnerSet
metadata: { name: linux, namespace: team-a }
spec:
gatewayRef: { name: acme }
templateRef: { name: default } # kind defaults to RunnerTemplate; set kind: ClusterRunnerTemplate for a platform golden template
acquisitionProtocol: Classic # multi-label ⇒ Classic (deprecated); the default is now ScaleSet (Q264 P5), which takes exactly one runnerLabel
runnerLabels: [self-hosted, linux]
maxListeners: 10
maxWorkers: 50
# no proxyRef and no ActionsGateway.spec.defaultProxyRef ⇒ direct egress
# (RunnerSet status reports proxyMode: Direct + an EgressUnattributed condition)
Adding a shared egress proxy is one more object plus a defaultProxyRef:
apiVersion: actions-gateway.com/v2alpha1
kind: EgressProxy
metadata: { name: shared, namespace: team-a }
spec: { minReplicas: 2, maxReplicas: 10 }
# then set on the gateway: spec.defaultProxyRef: { name: shared }
# every RunnerSet under that gateway inherits it unless it sets its own proxyRef
This shows the renamed group, the noun/verb split, and that proxy-less is the
minimal path — the EgressProxy and RunnerTemplate reuse only appear when a
tenant actually needs a shared proxy or a second runner shape.
H.5. How each pressure is resolved¶
- Object size. The large
PodTemplateSpeclives only inRunnerTemplate/ClusterRunnerTemplate.ActionsGatewayandRunnerSetbecome small, fixed-size objects, and nothing embeds a template anymore. Reuse means one 40 KB sysbox template exists once and is referenced N times instead of copied into N runner sets; aClusterRunnerTemplatelets the platform own it once cluster-wide. - Shared egress. Any number of
RunnerSets pointproxyRefat oneEgressProxy; settingdefaultProxyRefon the gateway makes every runner set inherit it — one tenant proxy, many runner sets. - One namespace, free rebalancing. Multiple
ActionsGateways andRunnerSets are permitted per namespace, all drawing on the single namespaceResourceQuota. A tenant rebalances by editing smallRunnerSetobjects (maxWorkers/priorityTiers) — no template churn, no new namespaces. PriorityClasses are already cluster-shared, so tiers compose across runner sets.
H.6. Naming and length budgets¶
The one rename worth insisting on: RunnerGroup → RunnerSet. "Runner
group" is already a first-class GitHub concept (runner groups gate which repos
may use which runners), so the current kind name collides with the domain.
RunnerSet also aligns with ARC's AutoscalingRunnerSet/EphemeralRunnerSet.
| New kind | Short | Scope | Role | Derives | Label value |
|---|---|---|---|---|---|
ActionsGateway |
ag |
ns | GitHub binding + AGC control plane | <ag>-agc Deploy/SA/Role |
…/gateway |
RunnerSet |
rs |
ns | scheduling/quota (was RunnerGroup) |
worker pod generateName=<rs>- |
…/runner-set |
RunnerTemplate |
rt |
ns | pod shape (the large object) | — (referenced) | …/runner-template |
ClusterRunnerTemplate |
crt |
cluster | platform golden templates | — | …/runner-template |
EgressProxy |
ep |
ns | proxy pool | <ep>-proxy Deploy/Svc/HPA/PDB |
…/egress-proxy |
Length constraints that actually bite (RFC 1123): object names ≤ 253, but
label values ≤ 63 and Service names ≤ 63. These CR names become both
selector label values and <name>-<suffix> Service names, so the
Service-name path is tightest:
EgressProxy→<ep>-proxyService ⇒ name ≤ 57 (reserve-proxy).RunnerSet→ worker podgenerateNameplus the random tail, and the name is also a label value ⇒ ≤ 63, practically ≤ ~57 for headroom.- Recommendation: put an explicit
maxLengthof 52 on every v2 CR name (leaves 11 for any derived suffix, stays well under 63 as a label value) and document it in the CRD field comment so it is discoverable, not a runtime surprise.
Field naming freezes at GA — do the pass now¶
JSON field names are part of the API contract and become permanent at v2. Do
the naming pass during M1 while names are still cheap to change under v2alpha1:
- Acronym/brand casing — decided.
githubis one lowercased word; trailing initialisms stay uppercase:githubURL,githubAppRef(k8s-consistent withclusterIP,targetCPUUtilizationPercentage). v1'sgitHubURL/gitHubAppRefcasing is not carried over. Apply the rule to every field and freeze it. - References are uniform.
gatewayRef/templateRef/proxyRef/githubAppRefshare one…Refsuffix and theObjectRef/LocalSecretReferenceshapes. - List fields are plural (
runnerLabels,priorityTiers).
Field movement, v1alpha1 → v2alpha1:
| v1alpha1 | v2alpha1 |
|---|---|
RunnerGroup (kind) |
RunnerSet (kind) |
RunnerGroup.spec.podTemplate + workerImage |
RunnerTemplate.spec (+ ClusterRunnerTemplate) |
RunnerGroup.spec.{runnerLabels,maxListeners,maxWorkers,priorityTiers, lifecycle} |
RunnerSet.spec (unchanged) |
ActionsGateway.spec.proxy |
EgressProxy (kind) |
ActionsGateway.spec.runnerGroups |
removed (explicit RunnerSet objects) |
| — | RunnerSet.spec.{gatewayRef,templateRef,proxyRef}; ActionsGateway.spec.{defaultProxyRef,defaultTemplateRef,agcResources} |
H.7. Reference integrity — runtime conditions, not admission¶
Requiring referents to exist at admission time would force apply ordering, which
breaks GitOps: Argo CD/Flux (and kubectl apply -f dir/) submit the whole set
at once and rely on eventual consistency. A webhook that denies a RunnerSet
because its RunnerTemplate has not synced yet turns a normal reconcile into a
failed sync. So responsibilities split by what admission is actually good at:
- Webhook keeps (static, order-independent): structural/shape validation, the
cross-field rule (
maxWorkers == priorityTiers[last].threshold), reserved-pod-field rejection onRunnerTemplate, namemaxLength, reference well-formedness, and whether a cross-namespace reference is permitted by operator policy at all.
Reserved-pod-field split (M2, implemented). The scalar pod-level reserved
fields (serviceAccountName, host{PID,Network,IPC},
automountServiceAccountToken) are CRD CEL rules (M1). The per-container checks
that exceed the CEL cost budget — an unbounded containers-array walk — are the
GMC-hosted validating webhook (M2): it rejects the AGC-injected egress-proxy env
vars (HTTP_PROXY/HTTPS_PROXY/NO_PROXY/PROXY_CA_CERT_PATH, the variables v1
silently overwrote at pod-build time) on every container of both template
kinds. Privileged containers are rejected on the namespaced RunnerTemplate
(a tenant must not self-author a privileged worker shape) but allowed on the
cluster-scoped ClusterRunnerTemplate — that kind is platform-authored (tenants
cannot create cluster-scoped objects) and exists precisely to hold golden
privileged templates (DinD/sysbox, §H.6). A RunnerTemplate carries no
securityProfile, so a v1-style profile-aware privileged decision is
impossible at the template layer; Pod Security Admission — which stamps the
namespace's enforcement level from the effective securityProfile (§H.16 #7) —
stays the runtime backstop for both kinds, so allowing privileged on the
cluster-scoped kind is no weaker than v1.
- Runtime condition (existence/referential): does the template exist, does
the proxy exist, does the cross-namespace grant exist. The controller watches
referents and re-reconciles when they appear:
RunnerSet status:
Ready: False
Reason: TemplateNotFound # or ProxyNotFound; ProxyShareNotGranted arrives with M4 (§H.9)
Message: "RunnerTemplate 'dind-large' not found in namespace 'team-a'"
Wire it with a watch + enqueue mapper (template/proxy/grant → referencing
RunnerSets) so the set flips to Ready the moment the missing object syncs.
This stays fail-closed: the controller only creates wiring (worker pods, the
cross-namespace NetworkPolicy egress, the proxy CA mount) once both the reference
and any required grant resolve. A RunnerSet pointing at a not-yet-granted
cross-namespace proxy simply sits NotReady — no traffic is ever permitted in
the gap, so moving the grant check to runtime opens no window.
Fast feedback without the hard reject: the admission webhook may return a
non-blocking warning (admission responses carry warnings[] without denying)
when a reference looks dangling at apply time. The operator sees
Warning: RunnerTemplate 'dind-large' not found immediately, the object is still
admitted, and the runtime condition remains authoritative.
Make gatewayRef a CRD field selector. Under multi-gateway, each AGC must
list/watch only the RunnerSets whose gatewayRef targets its gateway. Declaring
spec.gatewayRef.name a selectableField (CRD field selectors, KEP-4358) runs
that filter server-side instead of fetching every RunnerSet and filtering
in-process. Additive, but the watch should be designed around it from the start.
Status & condition contract — uniform across all five kinds. Every kind
carries status.conditions (listType=map keyed on type),
status.observedGeneration, and a Ready condition with the same polarity and a
shared reason vocabulary; messages name the specific blocker
(RunnerTemplate 'dind-large' not found), never a generic string. Pin this as the
contract in M1 rather than letting each reconciler invent its own.
Alongside the persisted conditions, each reconciler emits Kubernetes Events on the
object for the meaningful transitions — provisioning start, success, and failure;
credential (mTLS/TLS cert) issuance and rotation; and degraded/recovered — so
kubectl describe reflects what happened, not just the current condition set (Q305).
Events are transition-gated (emitted only when a condition's status actually changes, or
once per object for a lifecycle-unique step like the provisioning-start signal), never on
every reconcile, mirroring the v1 controllers' Event discipline. The operator-facing Event
reason vocabulary is tabulated in
troubleshooting.md.
Child-health rollup onto the ActionsGateway (Q304). The GMC's ActionsGateway
reconciler rolls the health of the RunnerSets bound to a gateway up onto a
RunnerSetsDegraded condition — the v2 counterpart of v1's RunnerGroupsDegraded, and
the operator's single pane. Because v2 RunnerSets are not owned by the gateway (they
only reference it via spec.gatewayRef, §H.8), the
binding is resolved by matching gatewayRef.name within the namespace — the same
scoping the AGC applies server-side via its spec.gatewayRef.name field selector, not
owner labels. A set is impaired (not serving jobs) on either of two axes: its Ready
is False for a non-transient reason — a reference did not resolve or a provisioning
step failed, anything but the benign startup NoActiveSessions, which is how v2 surfaces
the reference/provisioning failures that had no standalone condition — or any of its
abnormal-is-True impairing conditions is True (v2alpha1.ImpairingConditionTypes():
Degraded, CredentialUnavailable, RunnerVersionTooOld, WorkersUnschedulable). The
second axis is load-bearing because the shared listener pushes Degraded/RunnerVersionTooOld
onto the RunnerSet independently of Ready (Q330): a classic set whose sessions are all
rejected as unauthorized converges to Ready=NoActiveSessions while Degraded=True sits on
its own condition, so an Ready-only rollup would silently miss it. The advisory conditions
(RateLimited, the WorkerQuota ladder, EgressUnattributed, PossibleReapBlockingSidecar)
are excluded so the rollup does not flap on normal load. Like the v1 rollup it is
advisory — it does not gate Ready, since the gateway's own AGC control plane can be
healthy while one tenant's set is impaired. The GMC watches bound RunnerSets (predicated
on a change to a set's impaired signature, dropping high-frequency activeSessions/
pendingJobs churn) so a child's health change refreshes the parent promptly.
H.8. Ownership, GC, and deletion¶
Shared objects must not be owner-referenced by their referrers:
EgressProxyis standalone and owns its own children (the proxy Deployment/Service/HPA/PDB/NetworkPolicy, a self-signed proxy TLS Secret, and the metrics-mTLS bundle below, reconciled by the GMC). Nothing owns theEgressProxy. Each child is derived as<ep>-proxy(the TLS Secret as<ep>-proxy-tls) and carries the per-EgressProxyidentity labelactions-gateway.com/egress-proxy: <name>; every Deployment / Service / PDB / HPA / NetworkPolicy selector and the pod anti-affinity key on that label. This is load-bearing twice: it keeps multiple proxy pools in one namespace selector-isolated (v1 could assume one proxy per namespace), and because each pool is now its own Deployment, proxy metrics carry the proxy identity automatically. Same-namespace only at M2; cross-namespace sharing is M4.- Proxy metrics-mTLS + ServiceMonitor (Q324, at classic parity). The proxy
serves
/metricsover mutual TLS on:8443, requiring a scraper client cert signed by thisEgressProxy's own metrics CA (never shared with the AGC or another tenant). The GMC issues a per-EgressProxyPKI and writes a server bundle Secret<ep>-metrics-tls(mounted into the proxy) and a scraper client bundle<ep>-metrics-client(published for monitoring); the NetworkPolicy admits the:8443scrape only frommetrics: enablednamespaces. A<ep>-proxy-metricsServiceMonitoris provisioned when--enable-tenant-service-monitorsis on (gracefulServiceMonitorCRDMissinghandling otherwise). This is secure-by-default parity with the classic proxy — without the mounted bundle the proxy binary would fall back to serving/metricsplaintext on the health port, so mounting it is what prevents an unauthenticated-metrics regression. Both metrics Secrets carry an owner reference and are GC-delegated (the GMC holds no Secret delete verb). RunnerTemplateis pure data — no children, nothing owns it.ActionsGatewayteardown is fail-closed, not GC-trusting (Q328, ports v1's Q125). The gateway's namespaced children all carry a controller owner reference, so cascade GC would eventually remove them — but a transient GC failure is silent and unretried, and the ClusterRunnerTemplate ClusterRoleBinding (cluster-scoped, cannot be owned by a namespaced CR) has no GC at all. So the gateway holds a cleanup finalizer and its delete path deletes every child explicitly, then verifies each one is gone: a delete error or a child lingering under a foreign finalizer retains the gateway's finalizer, emits aTeardownIncompleteWarning event naming the blocker, and requeues until teardown is verifiably clean. The metrics mTLS Secrets are the one GC-delegated exception (the GMC holds no delete verb on Secrets by design). BoundRunnerSets are referrers, not children — they degrade per the next bullet. The gateway reconciler is also serialized (MaxConcurrentReconciles: 1), matching v1's single-writer assumption.- Deletion degrades, it does not block — and uses no finalizer at all.
Hard-blocking deletion of a still-referenced shared object via finalizer would
fight GitOps prune the same way an ordering webhook does; Kubernetes' own
finalizer guidance also warns that finalizers on shared/referenced objects are
a common cause of stuck-
Terminatingresources. So allow the delete and flip referrers toReady=False, Reason=TemplateDeleted(ProxyDeletedfor a deletedEgressProxy; same fail-closed behavior — no template ⇒ no new pods) via the referent→referrer watch. The deletion-specific reason is distinguished from*NotFoundby the referrer's own status markers (templateSource/proxyMode: Proxied) under an unchanged spec generation. Do not keep a finalizer even for bookkeeping:.status.referencedByis computable from the same informer/watch without taking on a finalizer that can block deletion.
H.9. Cross-namespace proxy sharing¶
Default is same-namespace. Cross-namespace sharing uses provider consent: the
owner of the EgressProxy publishes that it is shareable (via
spec.sharing.allowedNamespaces or a namespace selector). Naming a
cross-namespace proxy from the consumer side is not sufficient. This mirrors the
consent handshake of Gateway API's ReferenceGrant (GA / v1), where the grant
lives in the target (provider) namespace and a consumer-side name alone never
authorizes the reference.
v2 ships the inline allowlist only. It needs no Gateway API CRDs installed
(lower onboarding), and honoring a ReferenceGrant when Gateway API is present
can be added later without a breaking change. The load-bearing principle taken
from the precedent — consent is always provider-side — holds for both.
A shared proxy is a shared egress identity — it is for cooperating tenants or a platform-operated central pool, not mutually-distrusting tenants, because sharing surrenders the per-tenant egress attribution the proxy exists to provide.
Cross-namespace sharing forces two mechanisms that same-namespace sharing does not, and these are the bulk of the implementation cost:
- NetworkPolicy on both sides. The GMC must add egress (consumer workers → provider proxy Service) and ingress (provider proxy ← consumer namespaces) whenever a grant is active. Today both policies assume the proxy is colocated.
- Proxy TLS CA distribution — a ConfigMap, not a secret. A cross-namespace
consumer needs the proxy's CA certificate (public) to validate the tunnel —
never the private key, which stays in the proxy namespace. So this is trust
distribution, not secret replication: the GMC writes the CA as a ConfigMap
into only the granted consumer namespaces, scoped by the same grant that
authorizes the reference. This follows the cert-manager trust-manager
pattern (selector-scoped bundle sync); if trust-manager is installed, the CA
may instead be published as a
Bundle. No new secret-distribution mechanism is required — the earlier framing of this as a "secret" overstated the cost.
H.10. The egress proxy becomes optional¶
The proxy earns its keep for stable per-tenant egress IPs (GitHub IP-allowlisting, common with Enterprise Managed Users), egress attribution / incident containment, and avoiding shared-NAT throttling when many tenants reach GitHub from one IP. A small single-tenant cluster whose node egress IPs are already acceptable needs none of that.
So proxyRef/defaultProxyRef are both optional; unset ⇒ direct egress.
Onboarding collapses to three objects — one ActionsGateway, one
RunnerTemplate, one RunnerSet — with no proxy object at all. This is shipped
(Q168): a v2 ActionsGateway/RunnerSet with no proxy reaches Ready with direct
egress; the worked example in §H.4 is valid as written.
Secure-by-default guardrail (signed off). The proxy bundles two properties: egress identity (IP attribution) and egress restriction (traffic can only reach GitHub). Dropping the proxy drops identity, but it does not drop restriction — this trade was raised and signed off, and the shipped behavior holds the line:
- The NetworkPolicy egress lockdown stays mandatory and on by default even with no proxy — default-deny egress, allow only DNS + GitHub CIDRs (+ kube API for the AGC). Direct egress is still IP-restricted egress; there is no proxy-less mode in which a worker or AGC can reach arbitrary internet.
- The managed GitHub-IP refresh loop, which previously hung off the proxy, now
runs at the gateway level: the GMC's
IPRangeReconcilerpatches each direct-egress gateway's AGC + workload NetworkPolicies (as well as the proxied EgressProxy policies) so the direct-egress allowlist stays current as GitHub rotates ranges.
With those two in place, defaulting the proxy off loses only per-tenant IP identity — a property a subset of tenants need and opt into by attaching a proxy — not the egress containment baseline. Defaulting off the restriction would be a security regression and is out of scope. See the secure-by-default principle for the rule this satisfies.
Live enforcement is proven, not just shaped (Q178). Envtest proves the
direct-egress NetworkPolicies carry the right shape but has no CNI, so it cannot
prove the lockdown is enforced. The E2E_V2_DirectEgress kind e2e closes that gap:
a proxy-less worker pod reaches api.github.com directly (positive, both CNI legs)
while a connection from the same workload network context to a non-GitHub
destination is dropped by the default-deny egress NetworkPolicy (negative,
Calico-only — kindnet does not enforce egress drops, so the block self-skips there).
See §7.3 of the test plan.
Two refinements keep direct egress auditable, not silently inferred (both shipped):
- Direct egress is a structurally explicit state. An unset
proxyRefresolves to direct egress, and the gateway/runner-set status recordsproxyMode: Directrather than leaving "no proxy" to be inferred from an absent field. - Surface the attribution trade. The proxy-less gateway and runner set carry an
advisory
EgressUnattributedcondition (True), so an operator sees at a glance that this workload has no per-tenant egress identity — the property they opted out of — without grepping specs. It is advisory only and never gates Ready.
Composition bonus. §H.9 and §H.10 combine: a platform team runs one shared
EgressProxy in a central namespace, grants it to the EMU/allowlist tenants who
need stable IPs, and everyone else runs proxy-less.
H.11. Migration (v2, tool-assisted)¶
A conversion webhook cannot do this migration. Conversion webhooks convert one
object in place; they cannot create sibling objects. Splitting one
ActionsGateway (with inline proxy + bootstrap groups) into ActionsGateway +
EgressProxy + N RunnerTemplates + N RunnerSets is a fan-out on create,
which no conversion webhook can express. Therefore:
- Serve
v1alpha1andv2alpha1side by side (no automatic conversion of the split fields). - Ship a one-shot migration tool (a
kubectlplugin or subcommand) that reads v1 CRs and emits the v2 object set — extracting each inlinepodTemplateinto aRunnerTemplate, the inlineproxyinto anEgressProxy, and rewriting references. Dry-run to manifests by default; apply on--apply. - Deprecate
v1alpha1after a release or two. The cutover is deliberate, not silent, because the migration is fan-out-on-create.
Shipped (M5, Q165). The tool is gag-migrate (core in
cmd/gmc/internal/migrate, CLI in cmd/gmc/migrate). It resolves the latent
ambiguities §H.17 flags: reuse is detected by content-addressing the built
RunnerTemplateSpec (podTemplate and workerImage), so K identical templates
collapse to one object; maxListeners is pinned to the v1 effective value (not v2's
new default); defaultProxyRef is always wired so egress never silently goes direct;
standalone RunnerGroup CRs win over inline bootstrap entries; and the
securityProfile relocates onto the namespace label (most-restrictive-wins). The
operator runbook is migration-v1-to-v2.md and
the v1alpha1 deprecation notice.
v1alpha1 (one monolith) one-shot tool v2alpha1 (fan-out)
┌──────────────────────────────┐ ┌──────────────────────────────┐
│ ActionsGateway │ ┌──► │ ActionsGateway · identity │
│ ├ credentials · githubURL │ │ └──────────────────────────────┘
│ ├ spec.proxy (inline) │ ┌───────────┐ │ ┌──────────────────────────────┐
│ └ spec.runnerGroups[] │──►│ migration │──┼──► │ EgressProxy │
│ (inline podTemplates) │ │ tool │ │ └──────────────────────────────┘
└──────────────────────────────┘ │ dry-run → │ │ ┌──────────────────────────────┐
reads v1 │ --apply │ ├──► │ RunnerTemplate × N │
└───────────┘ │ └──────────────────────────────┘
fan-out on create│ ┌──────────────────────────────┐
└──► │ RunnerSet × N │
└──────────────────────────────┘
A conversion webhook can't create those sibling objects — which is exactly why the migration is a tool, not a webhook.
The v1→v2 fan-out is one-time. Once on v2alpha1, the API graduates in place
v2alpha1 → v2beta1 → v2 via a conversion webhook (a thing a conversion webhook
can do, since the kinds no longer change shape) — see the
graduation path
in the implementation plan for the per-hop mechanics.
H.12. Folding in the grandfathered label-value alignment (Q147)¶
Two shipped keys still carry boolean-looking "true" values that predate the
no-boolean label convention and are
grandfathered only because changing them is breaking:
actions-gateway.github.com/tenant: "true"— the managed-tenant marker, matched as== 'true'by thenamespace-psa-guardandtenant-resource-guardValidatingAdmissionPolicyobjects, the onboarding scripts, and operator runbooks.actions-gateway.github.com/allow-profile-downgrade: "true"— the downgrade opt-in annotation, matched by the GMC validating webhook.
Aligning them (→ tenant: managed, allow-profile-downgrade: allowed, following the
existing privileged-profile: allowed precedent) is a breaking change to deployed
clusters: it touches VAP CEL, onboarding, runbooks, and the label/annotation on every
live tenant namespace. The convention doc therefore defers it to "a separate,
deliberate migration." The v2 cutover is that migration — it is already breaking,
already ships a migration tool (§H.11), and already
reworks the same VAPs and onboarding for multi-gateway-per-namespace. Folding Q147 in
here costs almost nothing extra and avoids a second, standalone breaking migration
later.
The key prefixes migrate too, not just the values. The v2
API group rename moves these keys off
the actions-gateway.github.com/ domain to actions-gateway.com/ (e.g.
actions-gateway.com/tenant), together with the other domain-prefixed identifiers
— privileged-profile, agentpool-cleanup, gmc-cleanup, the version label, and
the finalizer names. This is the same class of breaking change as the value
alignment and rides the same dual-read window: every consumer accepts either
domain and either value until v1alpha1 removal, and the migration tool
relabels in one pass. Renaming the API group but leaving the labels on the old
domain would be a permanent inconsistency, so the prefixes move with the group.
Both keys survive into v2 unchanged in meaning — the tenant marker still confines
the GMC's namespace writes under multi-gateway-per-namespace, and
allow-profile-downgrade still guards ActionsGateway PSA downgrades — so the cutover
changes only their values, not their role.
Dual-read window (coincident with the v1/v2 coexistence window). Q147 needs a dual-read migration so live namespaces are not broken mid-cutover:
- While
v1alpha1andv2alpha1are served side by side, every consumer of these values — both VAPs and the downgrade webhook — accepts either"true"(legacy) or the new keyword. Reads are dual; writes prefer the new keyword. - The migration tool (§H.11) relabels the namespace marker and rewrites the annotation to the new keyword as part of the same one-shot pass that emits the v2 object set, so no separate operator action is required.
- When
v1alpha1is deprecated and removed, drop the"true"arm from the VAPs and webhook. The dual-read window closes exactly whenv1alpha1serving does.
This stays fail-closed throughout: the CEL/webhook checks already treat any
non-sentinel value as "not granted", so accepting a second sentinel during the window
never widens a grant — at worst a namespace is briefly un-aligned and the
already-applied "true" keeps working until the tool relabels it.
Shipped (M5, Q165). The dual-read spans all four consumers: the
namespace-psa-guard, tenant-resource-guard, and namespace-security-profile-guard
ValidatingAdmissionPolicy objects (dual-marked in M3a/M3b), and the v1 GMC
ActionsGateway validating webhook (M5) — whose validatePrivilegedEligibility now
accepts the grant label on either domain and whose validateSecurityProfileTransition
accepts the downgrade opt-in on either domain and either value keyword. The
migration tool relabels the namespace markers additively (it adds the v2 keys and
keeps the v1 keys), so a still-running v1 gateway in a relabeled namespace is never
stranded. The window closes when v1alpha1 is removed.
H.13. What adopting this changes¶
This proposal, if accepted, touches more than the API types. Non-exhaustive impact list, to be turned into plan-doc scope when scheduled:
- API: new
v2alpha1groupactions-gateway.comwith five CRD kinds (ActionsGateway,RunnerSet,RunnerTemplate,ClusterRunnerTemplate,EgressProxy) + generated CRDs/deepcopy/RBAC. - GMC: multi-gateway-per-namespace requires every derived resource to be
keyed by gateway name (not the namespace-singleton assumption today); the
gmc-tenant-resource-guardpolicy must still confine writes; newEgressProxyreconciler; cross-namespace CA distribution. - AGC:
RunnerSetreconciler resolvestemplateRef/proxyRefat runtime with watches; reserved-field webhook moves toRunnerTemplate. - Docs: §3.1 CRD schemas, Appendix E (RunnerGroup design), and the operator-facing docs per the doc-update matrix.
- Migration tool + its tests.
- Label keys + values (Q147 + group rename): both the domain prefix
(
actions-gateway.github.com/*→actions-gateway.com/*, plus finalizer names) and the grandfatheredtenant/allow-profile-downgrade"true"values migrate during the cutover (§H.12) — VAP CEL, onboarding scripts, runbooks, and the convention doc's "grandfathered" note all update, and one dual-read window covers both, riding the v1alpha1 serving window.
H.13.1. CRD packaging and conditional GMC startup¶
The five actions-gateway.com/v2alpha1 CRDs ship in a separate, opt-in Helm
chart (actions-gateway-crds-v2), not the main actions-gateway chart. The
split keeps the main chart's Helm release Secret under Kubernetes' 1 MiB object
limit — the five v2 CRDs' OpenAPI schemas are large enough that bundling them
would push it over.
So a supported install can have the GMC running without the v2 CRDs present. The
GMC makes the opt-in real on the controller side: at startup it queries the REST
mapper for the ActionsGateway and EgressProxy v2 kinds and only registers the
v2 controllers (ActionsGatewayV2Reconciler, EgressProxyReconciler,
NamespacePSAReconciler) and the IP-range reconciler's v2 NetworkPolicy refresh
passes when both are served. On a v1-only install it logs one info line and
starts v1-only — rather than registering source.Kind watches against absent
kinds, which would spin a "no matches for kind" retry loop and make the
IP-range reconcile log a list error every tick. v1alpha1 reconciliation is
unaffected either way.
Detection is a one-shot startup check, so installing the CRD chart into a running
v1-only cluster requires a GMC restart to enable the v2 controllers. The
RunnerTemplate validating webhooks are registered unconditionally (they never
fire until a RunnerTemplate/ClusterRunnerTemplate exists, which requires the
CRDs), so the webhook path needs no restart. The operator-facing runbook for the
"v2 objects not reconciling after installing the CRD chart" symptom lives in
troubleshooting.md.
H.13.2. Install/upgrade lifecycle — apply-render, not helm install (Q276)¶
Splitting the CRDs into their own chart (§H.13.1) keeps the main chart under the
1 MiB release-Secret limit, but the v2 chart itself is still too large to helm
install: rendered it is ~2.5 MB, and Helm stores a release as
base64(gzip(json(release))) — where json(release) embeds both the rendered
manifest and a base64-inflated copy of the chart source — inside one Secret the
apiserver caps at 1 MiB. Reconstructing that encoding for the five-CRD chart lands at
~1.10 MiB stored, ~0.1 MiB over the ceiling. So the chart is applied from its
render, not installed:
helm template actions-gateway-crds-v2 <chart-or-oci-ref> --namespace gmc-system \
| kubectl apply --server-side -f -
Decision: apply-render is the supported, deliberate install/upgrade path — not a
stopgap. Server-side apply is declarative and idempotent, so install and upgrade are
the same command (re-run it to carry CRD field changes); it also clears kubectl's
256 KB client-side-apply annotation ceiling, which the two ~1.16 MB RunnerTemplate
CRDs blow past on their own. This is the mainstream practice for large CRDs (cert-
manager, Crossplane, Istio, Gateway API all ship CRDs to be applied out-of-band and
warn against Helm-managing them). The templates carry helm.sh/resource-policy: keep
so an operator who fronts them with a GitOps tool that does build a Helm release
keeps the CRDs on prune/uninstall.
To keep the manual path a single command with no helm or chart checkout, each release
attaches a pre-rendered, cosign-signed actions-gateway-crds-v2.yaml (rendered for
the default gmc-system namespace) to its GitHub Release, so a manual install is just
kubectl apply --server-side -f <release-url>/actions-gateway-crds-v2.yaml. It is
keyless-signed via the same Fulcio/Rekor path as the images and charts (sign-blob → a
Sigstore bundle verified with cosign verify-blob --bundle), which also answers the
"release assets are mutable" caveat. The asset covers the default-namespace case; a GMC
in a non-default namespace, or a GitOps render, still uses the helm template --set … |
kubectl apply --server-side path (the conversion webhook clientConfig namespace bakes
in at render time). Argo CD renders the chart itself and never builds a release Secret,
so it is unaffected by the 1 MiB limit; Flux HelmRelease does build one and uses the
rendered manifest or a Kustomization instead.
The properties this forgoes — helm upgrade revision history, helm rollback, helm
uninstall cleanup — are low-value for CRDs specifically and partly anti-patterns:
rolling a CRD schema backward can strand stored objects (removing a served/storage
version or field is a deliberate migration, never a casual rollback), and cascading
helm uninstall of a CRD deletes every custom resource with it (data loss — exactly
what resource-policy: keep prevents). The operator-facing upgrade/rollback/uninstall
runbook is install.md § the v2 API CRDs.
Alternatives, each measured by reconstructing Helm's stored-Secret encoding:
| Option | Stored release Secret | helm install |
helm upgrade |
Verdict |
|---|---|---|---|---|
| Apply-render (chosen) | n/a — no release created | n/a | re-apply, server-side | Chosen. Uniform install≡upgrade; clears both the 1 MiB and 256 KB limits; GitOps-friendly; ecosystem norm. |
| Single templated chart (today's chart shape, helm-installed) | ~1.10 MiB | ❌ fails | — | Over the ceiling — this is the Q276 problem. |
crds/ directory convention |
~0.72 MiB | ✅ installs | ❌ never upgrades/deletes | Rejected: Helm's crds/ dir is create-only by design, so schema changes never propagate on upgrade — trades the size problem for a silent no-upgrade problem on an evolving API. |
| Split large CRDs into per-CRD charts | ~0.50 MiB each large CRD; ~0.10 MiB for the three small ones together | ✅ installs | ✅ | Rejected: technically viable and the spec.conversion wiring is per-CRD so it survives the split, but it multiplies one artifact into three published/versioned charts and three releases (an umbrella chart re-aggregates into one Secret and lands back over the limit), buying a Helm CRD lifecycle you should not use (see above) at real packaging + operator-UX cost. |
Trim rendered size (drop descriptions / x-kubernetes-preserve-unknown-fields) |
shrinks, but | partial | partial | Rejected: dropping descriptions degrades kubectl explain; replacing the embedded PodTemplateSpec schema with a preserve-unknown-fields blob drops server-side validation of the pod template — a secure-by-default regression. The structural Pod schema is inherently large regardless. |
Fold-back into the main chart is gated on v2 reaching a single served version,
not on v1alpha1 removal. Measured against the same encoding: with v2 down to one
version (v2beta1 only — v2alpha1 and the conversion webhook retired), the folded
main chart stores at ~0.65 MiB, ~0.4 MiB under the ceiling. It fits because a
single-version v2 CRD set (~1.3 MB) is about the same footprint as the two v1 CRDs it
replaces (~1.3 MB), and today's main chart already carries those and installs cleanly
(~0.62 MiB stored). The trap is the coexistence window: while v2 serves both
v2beta1 and v2alpha1 (which is what forces the conversion webhook), that set alone
stores at ~1.26 MiB — over the limit before a single controller is added, which is
exactly why the v2 CRDs live in their own chart today. So the clean fold-back sequence
is two removals, both required: retire v1alpha1, and drop v2alpha1 (retiring
the conversion webhook), leaving one served version — then the now single-version v2
CRDs fold into the main chart with headroom to spare. Until both land, the opt-in chart
plus apply-render stands.
H.14. Admin policy layer — deferred until tiering is real¶
The decomposition above mirrors Gateway API's Gateway → route attachment but
stops one level short: there is no cluster-scoped, admin-owned object — no
GatewayClass equivalent. Today the admin/tenant boundary is real but lives
outside the API, scattered across mechanisms that cannot be RBAC'd, audited, or
GitOps'd as objects:
| Policy | Where it lives today |
|---|---|
| Which PriorityClasses a tenant may name | --allowed-priority-classes GMC flag |
Whether privileged profile is allowed |
namespace label …/privileged-profile=allowed |
| Default worker image | --worker-image GMC flag |
| Reserved namespaces | Go constant + POD_NAMESPACE |
| Namespace ResourceQuota | platform-stamped, out-of-band |
Promoting this into a first-class API object turns the boundary into a clean RBAC
split (admin writes the policy kind; tenants cannot) and makes "what is this
tenant allowed to do?" a kubectl get away. But it is not a problem we have
today, and the abstraction is addable without a second breaking change — so v2
does not ship it. This section records the capability ladder and the exact
trigger, so the decision is captured rather than rediscovered.
The capability ladder¶
| Layer | Expresses | Breaks when |
|---|---|---|
| Flags (today) | one global policy, cluster-wide | can't vary per tenant at all (except the one bolted-on privileged namespace-label gate) |
| Singleton policy object | one global policy, but declarative / auditable / RBAC'd | still uniform — every tenant gets the same rules |
| Singleton + namespace labels | one global policy plus a few independent per-tenant dials ("privileged iff namespace has label X") | the per-tenant variation becomes multi-dimensional and correlated |
Class (ActionsGatewayClass) |
named bundles of correlated policy, tenant-selectable, RBAC-gated on which class may be referenced | — |
A single per-tenant escape hatch (like privileged-allowed) does not need a class — a namespace label the admin controls handles it, which is already how v1 works. The singleton + the occasional label dial gets you a long way.
Promote flags → singleton when¶
The singleton carries the same policy as the flags — one uniform policy, no tiers — so it buys only the rung-2 wins, not tiering. Promote when any of:
- GitOps — policy must be managed declaratively / changed without a controller redeploy.
- RBAC separation — the people who set policy must be distinct from the people who own the controller Deployment (a platform-policy team vs. the SRE who deploys the GMC).
- Audit/compliance — "show me, as a cluster object, exactly what tenants are allowed" is an actual requirement.
If none of those bite, flags are simpler and equally forward-compatible.
The trigger for the class¶
Introduce ActionsGatewayClass only when both hold:
- ≥2 distinct policy bundles must coexist in one cluster — e.g. an "internal/trusted" tier (DinD allowed, broad registries, proxy optional) vs an "external/untrusted" tier (restricted-only, platform registry only, proxy mandatory): multiple policy dimensions that travel together as a tier; and
- either those tiers are spread across enough namespaces that encoding each as a combination of namespace labels becomes an audit/maintenance liability (N namespaces × M labels that should just say "tier = A"), or you want tenants to self-select a tier with RBAC deciding which they may pick — which labels cannot express, because tenants do not control namespace labels.
Smell signs the trigger has arrived: the onboarding runbook grows a "pick your tier, then apply these K labels" step (that step is a class waiting to be born); a request like "team X gets privileged + registries A&B, team Y neither"; or a self-service "request the privileged tier" flow.
Why deferring costs nothing¶
Every rung of the ladder is an additive transition — none is a breaking migration:
- flags → singleton: add the
ActionsGatewayPolicykind; the controller prefers it when present and the flags remain as fallback/defaults. - singleton → class: add the
ActionsGatewayClasskind, and addActionsGateway.spec.gatewayClassNameas an optional field whose unset value means "the default class / the old singleton"; the singleton simply becomes the default class.
So deferring either step buys no future breaking migration. The one constraint to honor now: whatever policy lands in v2 — flags or a singleton — must be shaped so a future singleton/class could carry the identical schema field-for-field. Don't paint the policy into a corner a later rung couldn't inherit.
v2 decision¶
v2 keeps the controller flags. A singleton/class earns its keep only at the triggers above, none of which is a problem we have today, and every rung is additive — so promoting later costs nothing, while building now would be abstraction ahead of need. The single obligation v2 carries is to shape the flag-backed policy so a future singleton/class inherits its schema field-for- field. Ship neither the singleton nor the class. Promote to the singleton at the flags→singleton trigger; introduce the class at the two-part class trigger.
H.15. Other breaking changes worth batching¶
v2 is the one window where breaking changes are cheap (we are already rewriting the schema and shipping a migration tool). A few small changes are only possible at a major break, or are awkward to add later — batch them in, but only the ones that fix a problem we have today.
Decided for v2 (today's problem, break-only or break-cheapest):
- Drop the
SecretReference.namespacefootgun. It is reserved-but-validated- empty and reads like a cross-namespace reference that does not exist. Replace with a name-onlyLocalSecretReference. Removing a field is break-only. - Per-field immutability via CEL
XValidation(oldSelf):githubURLimmutable — rebinding a running gateway's GitHub org is a footgun;credentials.githubApp.namemutable — it is the credential-rotation path. Adding immutability later is itself breaking, so it is fixed at v2. - API group rename →
actions-gateway.com. The group isactions-gateway.github.com, which suffixes a domain the project does not control — against the k8s convention of using a domain you own. The project ownsactions-gateway.com, so v2 renames the group to it. Changing the group touches every CRD (and every CR, RBAC rule, VAP, and manifest that names it), so it can only happen at a major break — it rides the v2 cutover and its migration tool. The label/annotation key prefixes, the version label, and finalizer names carry the same domain and rename with it (actions-gateway.github.com/*→actions-gateway.com/*), on the Q147 dual-read window so live namespaces are not broken mid-cutover (§H.12). - Cheap usability while regenerating:
additionalPrinterColumns(Ready, profile, active sessions), resourcecategories, and the short names from §H.6. maxListenersdefault →10(was1in code; matches the design). Confirmed against the AGC listenerMultiplexer/Runsource: the pool keeps a permanent baseline of one poller and demand-spawns extra pollers only as jobs are acquired (a job-holding goroutine is busy, not polling, for the job's whole duration), with non-baseline pollers idle-exiting after 50 empty polls. SomaxListenersis a concurrency ceiling with a baseline of 1, not a steady-state count: a higher default costs nothing at idle, while1serializes job pickup per group (the busy baseline leaves no poller, andSpawnReplacementis a no-op at the ceiling). The real resource guards (maxWorkers+ namespaceResourceQuota) still bind, so the higher default regresses no safety property. v2 sets the default to10.
Opportunistic (take if it falls out of the rewrite; not a sign-off item):
-
Webhook → CEL migration. v2 targets a newer k8s floor, so checks that are webhook-only today because CEL could not express them on k8s ≤1.30 (singleton, GitHub-URL structure, cross-field rules) can become structural/CEL. Every check moved out of the fail-closed validating webhook is one fewer thing whose outage blocks all admission — an availability and operability win, best taken during the schema rewrite.
-
Credentials as a discriminated union — shipped in
v2alpha1(Q196/Q197). A flatworkloadIdentityRefsibling is mechanically additive, but additive into a permanently worse shape: oncegithubAppRefis top-level under beta it can never move under a parent without a breaking change + storage migration. Sincealpha → betais the last free break and workload identity is on-strategy (removes the App key from the cluster — the secure-by-default direction), the credential is nested under an explicit-discriminatorspec.credentialsparent inv2alpha1— a free reshape while alpha carries no stability contract, so the beta cut inherits the right shape and the conversion webhook (Q74) round-trips it as an identity for the credentials block.spec.credentialsis a discriminated union keyed bycredentials.type(+unionDiscriminator):githubApp(a name-onlyLocalSecretReference, the possession model) andworkloadIdentity(the no-PEM delegation model, Q197). The union's "exactly the named member is set" invariant is enforced by a per-member CELiffrule that each new member extends — never an N-way "exactly one of" that grows with the union.workloadIdentity(Q197) shipped as the second member: an external signer signs the App JWT so the App private key never enters the cluster (MVP = HashiCorp Vault transit + Vault Kubernetes auth, behind agithubapp.Signerinterface so cloud KMS providers add without another breaking change). Adding it validated the shape against a real second consumer, the whole point of fixing the shape before the freeze. See 05-security.md §5.7 for the trust model. Plan + schema sketch: v2beta1.md.
Explicitly NOT now (shape for additive later, do not build):
- Admin policy class — §H.14.
- Worker-image registry allowlist — a real security control, but only needed once there are untrusted tenants to restrict. It belongs in the admin policy schema and is enforced when that layer arrives; do not add a standalone tenant field for it now.
H.16. Open questions / sign-off needed¶
Recommended (pending ratification)¶
Each carries a recommendation grounded in precedent (Gateway API, ARC, cert-manager trust-manager, Kubernetes finalizer guidance); ratify or override.
- Multi-gateway-per-namespace — naming, AGC scoping, ownership. Verified
against
gmc-tenant-resource-guard(cmd/gmc/config/admission-policy/tenant-resource-guard.yaml): the GMC-confinement VAP keys on the namespacetenant=truemarker, not on resource names, so it already scales to N gateways per namespace and needs no change. The real work is three controller-side changes: - (a) Per-gateway naming — every derived resource becomes
<ag>-<suffix>(<ag>-agc,<ep>-proxy, workergenerateName=<rs>-) under the §H.6 52-char cap, so two gateways in one namespace never collide on a fixed name. - (b) Per-gateway AGC scoping — N gateways ⇒ N AGC Deployments in one
namespace, so each AGC must reconcile only the
RunnerSets whosegatewayReftargets it — the one genuinely new controller behavior, without which N controllers fight over the same objects. - (c) Per-gateway ownership — each
ActionsGatewayowner-refs its own children so deleting one gateway GCs only its resources, not a neighbor's.
(Optional defense-in-depth: also require a GMC managed-by label on writes; not
needed for correctness since the VAP already confines by namespace.) Precedent:
ARC runs multiple scale sets per namespace, names = CR prefix + fixed suffix.
The core build of v2 — naming + watch-scoping + ownership, not a policy rewrite.
Implemented (M3b, Q167). All three: (a) every AGC child is named <ag>-<suffix>
(<ag>-agc, <ag>-worker, <ag>-workload, <ag>-agc-metrics-{tls,client}) under
the §H.6 52-char cap, <ag>-agc doubling as the pod app label / NetworkPolicy /
Service selector so two AGC Deployments never adopt each other's pods; (b) the GMC
stamps GATEWAY_NAME on each AGC Deployment and the AGC scopes its RunnerSet
informer with a server-side spec.gatewayRef.name field selector (KEP-4358, k8s ≥
1.31) plus a defense-in-depth reconcile guard; (c) per-gateway names + owner refs
GC only the deleted gateway's children. The gmc-tenant-resource-guard VAP is
unchanged. Closing the M3a deferral, the AGC also gains least-privilege
cluster-scoped read of ClusterRunnerTemplate via a per-gateway ClusterRoleBinding
(shipped agc-clusterrunnertemplate-reader ClusterRole; GMC holds only bind),
deleted explicitly on teardown since a cluster-scoped object cannot own-ref a
namespaced gateway. Envtest (both suites) + a kind e2e (E2E_V2_MultiGateway) prove
the scoping isolation and per-gateway GC.
2. Cross-namespace proxy CA distribution → ConfigMap, not secret. The CA is a
public certificate, so the GMC distributes it as a ConfigMap into only the
granted consumer namespaces (trust-manager pattern; §H.9).
No secret-replication subsystem is needed — recommend dropping that as a
blocker.
3. Optional proxy — ✅ guardrail as in §H.10:
egress restriction stays mandatory, managed-IP refresh relocates, plus an
explicit proxyMode: Direct status and an EgressUnattributed advisory
condition so the attribution trade is auditable.
4. Sharing model — ship the inline allowedNamespaces allowlist only for
v2; ReferenceGrant support is additive later. Consent stays provider-side
either way (§H.9).
5. Deletion semantics — degrade-not-block with no finalizer at all
(§H.8); referencedBy is computed from the
watch. Confirm no operator relies on hard deletion protection.
6. Q147 label-value keywords — ratify tenant: managed and
allow-profile-downgrade: allowed (symmetric with privileged-profile:
allowed), with the dual-read window closing only at v1alpha1 removal
(§H.12).
7. Multi-gateway securityProfile composition — ✅ move it off the gateway.
The contention is self-inflicted. Pod Security Admission is a namespace-scoped
control in Kubernetes, and v1 hung it on a sub-namespace object
(ActionsGateway.spec.securityProfile) — so under multi-gateway (#1) two
ActionsGateways in one namespace fight over the single namespace PSA label.
The fix deletes the question instead of answering it: securityProfile
becomes a namespace-scoped concern, not a per-gateway field. Drop
SecurityProfile from ActionsGatewaySpec (a cheap follow-up to the just-merged
M1 v2alpha1 types — alpha, no compatibility guarantee) and let the namespace
own its Pod Security level, GMC-guarded exactly as today: the downgrade-protection
and privileged-eligibility machinery (securityProfileRank,
validateSecurityProfileTransition, the allow-profile-downgrade keyword) stays,
now keyed once per namespace instead of per gateway. Co-located gateways
therefore always share one posture; tenants that need different postures use
different namespaces — the natural PSA isolation boundary anyway. Land the
field-home change no later than M3a (Q164), where
ActionsGateway is first reconciled, so M3a reads the profile from its new home
rather than building per-gateway logic that M3b would rip out. Migration is
unaffected (one v1 namespace → one gateway → one profile).
Implemented mechanism (M3a, Q175). The namespace-side selector is the label
actions-gateway.com/security-profile: baseline|restricted|privileged
(SecurityProfileLabel); absent on a managed tenant namespace ⇒ baseline (secure
default). Two GMC-side pieces realize the guarantee, and crucially the relocation
makes the guard simpler than v1, not just relocated:
- NamespacePSAReconciler (GMC) watches managed v2 tenant namespaces and stamps
the six pod-security.kubernetes.io/* labels from the profile label via
Server-Side Apply (the v1 applyNamespacePSA stamping logic, now keyed once per
namespace and decoupled from any gateway's lifecycle). The PSA labels exist as
soon as the namespace is a managed tenant, with or without a gateway.
- gmc-namespace-security-profile-guard ValidatingAdmissionPolicy reproduces the
v1 webhook's three guarantees — enum, no-silent-downgrade (requires the
allow-profile-downgrade annotation), and privileged eligibility (requires the
platform privileged-profile=allowed label) — none weaker than v1. v1 needed a
Go validating webhook because the downgrade/eligibility checks read a different
object (the namespace) than the one admitted (the ActionsGateway). Now that the
profile lives on the namespace, both checks act on the same object the admission
is about, so they collapse into a VAP — in-process, no webhook-pod availability
dependency, fail-closed (failurePolicy: Fail, validationActions: [Deny]), and
consistent with the existing namespace-psa-guard/tenant-resource-guard pattern.
The downgrade check is skipped on CREATE (no prior state), so a namespace may be
created directly at any eligible profile. Both guard policies, plus the two existing
ones, dual-read the v1 (actions-gateway.github.com/tenant=true) and v2
(actions-gateway.com/tenant=managed) markers during coexistence (§H.12),
so the GMC can stamp PSA and provision in v2 tenant namespaces.
Fallback — only if a concrete need for co-located differing profiles
emerges: keep securityProfile per-gateway and resolve the namespace label by
most-restrictive-wins — runtime composition (max securityProfileRank across
the namespace's gateways), surfaced via a per-gateway EffectiveSecurityProfile
condition reporting the resolved profile and whether a sibling raised it. It is
secure-by-default (the label only ever rises) and fits
§H.7's "runtime
conditions, not admission" stance. Reject the other two rules: all-must-agree /
reject-on-conflict needs cross-object admission (inspect sibling gateways —
exactly what §H.7 avoids, and awkward in single-object CEL); off-label drops
namespace PSA enforcement entirely, a secure-by-default regression. Edge for the
fallback only: deleting the gateway that forced restricted drops the label to
the next-highest — a downgrade-by-deletion that interacts with
validateSecurityProfileTransition / allow-profile-downgrade; decide whether
that auto-downgrade needs the guard or is acceptable (no remaining gateway
requested the stricter level).
Resolved¶
- Admin policy layer (§H.14) — ✅ v2 keeps the controller flags. Neither the singleton nor the class ships in v2; each is deferred behind a documented trigger (flags→singleton, then the two-part class trigger), and every rung is an additive, non-breaking transition. v2's only obligation is to shape the flag-backed policy so a future singleton/class inherits its schema field-for-field.
- API group rename (§H.15) — ✅ Yes, rename to
actions-gateway.com. The project owns the domain; the change rides the v2 cutover and its migration tool. Break-only, so it happens here or never. - Per-field immutability (§H.15) — ✅
githubURLimmutable,credentials.githubApp.namemutable. maxListenersdefault (§H.15) — ✅10(was1in code). Verified against the AGC listener source:maxListenersis a concurrency ceiling with a baseline-of-1 + demand-spawn + idle-shutdown, so a higher default is free at idle and1needlessly serializes job pickup;maxWorkers/quota remain the binding resource guards. (Closed Q162.)
H.17. Migration correctness — the fan-out's untested invariants¶
The migration (§H.11) is the first and only
place two invariants this design asserts are tested against real v1 data. Both
are stated confidently above; neither has been exercised. They are acceptance
criteria the M5 tool (Q165) must meet — and, because they are
pure data-shape questions, they can be validated before M5 by a mapping over
representative v1 fixtures, surfacing any v2 schema gap at alpha-rewrite cost
instead of post-adoption cost. (The v2alpha1 types do not exist yet, so this is a
fixtures-and-asserted-output exercise, not runnable tool code, until M1 lands.)
Invariant 1 — "no behavior change" (the non-goal most at risk)¶
The fan-out changes field defaults and optionality, so "v2 tracks v1 behavior" is false unless the mapping actively compensates:
- Proxy must not silently become direct egress. v1 always routes through the
proxy; in v2 an unset
proxyRefand unsetdefaultProxyRef⇒ direct egress (§H.10). The mapping must setdefaultProxyRefon every migrated gateway, or migration regresses both behavior and the secure-by-default egress identity. Acceptance: a proxied v1 tenant migrates to a proxied v2 tenant, never toproxyMode: Direct. maxListeners1 → 10. v1 unset = 1; v2 unset = 10 (§H.15). The mapping must either pinmaxListeners: 1to preserve the v1 concurrency ceiling or consciously accept the change — not inherit the new default by omission. Decide and encode it.- v1 data must be admissible under v2 CEL. The cross-field rule
maxWorkers == priorityTiers[last].thresholdand the reserved-pod-field rejection (now onRunnerTemplate, §H.7) must accept every object the mapping emits. Real-apiserver defaulting applied to a round-trippedPodTemplateSpeccan introduce a field the source lacked, so this is an envtest check, not a pure-Go transform check.
Invariant 2 — "reuse" (the object-size justification)¶
v2's headline benefit is "one template exists once, referenced N times"
(§H.5). That benefit is realized only if the
migration detects reuse: K RunnerGroups sharing an identical podTemplate must
collapse to one RunnerTemplate, not K copies. Template equality is non-trivial
(deep-equal over a defaulted PodTemplateSpec; whether the separate workerImage
dedups with the template or independently). Acceptance: a tenant with K
identical-template groups migrates to one RunnerTemplate + K RunnerSets. If the
mapping emits K templates, v2 delivers zero object-size win for migrated tenants
— the benefit evaporates for the exact population it targets.
Latent v1 ambiguities the fan-out forces to a decision¶
- Standalone vs. inline runner groups. v1 has both a standalone
RunnerGroupCRD and inlineActionsGateway.spec.runnerGroups[]bootstrap copies. The mapping must define which is authoritative when both name the same group, and whether they merge or collide. v1 never reconciled the two representations; v2 forces the choice. - Naming the extracted objects. The
EgressProxypulled fromspec.proxyand eachRunnerTemplatepulled from a group need generated names under the §H.6 52-char cap — a naming scheme distinct from the runtime per-gateway derivation, and one that can collide.
These criteria turn the migration from "does it run" into "does it preserve what the
design promised." The proxy-default regression and the securityProfile composition
gap (§H.16 #7) are the two worth finding now,
at alpha cost, rather than from an adopter.