docs(adr,k8s): clarify RWX enables free scheduling, RWO requires affinity
- RWX volumes allow job pods to be scheduled on any cluster node - RWO volumes require affinity to pin job pods to runner's node - Remove ACTIONS_RUNNER_USE_KUBE_SCHEDULER from RWX migration steps - Emphasize resource utilization benefits of RWX free scheduling Co-authored-by: Sisyphus <[email protected]>
This commit is contained in:
@@ -8,13 +8,13 @@
|
||||
|
||||
The Kubernetes hook implementation for GitHub Actions runners requires access to the runner's working directory (`_work`) within the dynamically created job pods. This shared access is typically managed via Persistent Volume Claims (PVCs).
|
||||
|
||||
Regardless of the storage strategy, job pods are always constrained to run on the same node as the runner pod to ensure consistent access to the local environment and state. The choice of volume access mode determines operational flexibility and multi-pod access capability rather than pod placement.
|
||||
The choice of storage strategy significantly impacts pod scheduling. While ReadWriteOnce (RWO) volumes require job pods to be co-located on the same node as the runner pod, ReadWriteMany (RWX) volumes allow job pods to be scheduled freely across the cluster.
|
||||
|
||||
Depending on the storage provider and cluster configuration, operators may choose between `ReadWriteMany` (RWX) or `ReadWriteOnce` (RWO) access modes. RWX is preferred because it allows multiple pods to access the volume simultaneously, providing greater operational flexibility for future scaling or monitoring scenarios. RWO restricts volume access to a single pod at a time, locking the volume to that pod's specific node.
|
||||
Depending on the storage provider and cluster configuration, operators may choose between `ReadWriteMany` (RWX) or `ReadWriteOnce` (RWO) access modes. RWX is preferred because it allows the Kubernetes scheduler to place job pods on any available node, improving resource utilization and cluster flexibility. RWO restricts volume access to a single node at a time, requiring all pods using the volume to be pinned to that specific node.
|
||||
|
||||
## Decision
|
||||
|
||||
We have decided to establish `ReadWriteMany` (RWX) as the preferred storage strategy for the Kubernetes hook. While job pods remain pinned to the runner's node, RWX provides superior operational flexibility by allowing multiple pods (such as sidecars or auxiliary tools) to access the same volume without storage-imposed locking constraints.
|
||||
We have decided to establish `ReadWriteMany` (RWX) as the preferred storage strategy for the Kubernetes hook. RWX provides superior operational flexibility by enabling free scheduling of job pods across the cluster, as the shared volume is accessible from any node. This decoupling of job pods from the runner's node allows for better resource distribution and reduces the risk of node-level resource exhaustion.
|
||||
|
||||
For environments where RWX is unavailable or undesirable, we support a `ReadWriteOnce` (RWO) fallback strategy. This fallback is implemented using node affinity to ensure that job pods are scheduled onto the same node as the runner pod that holds the RWO volume.
|
||||
|
||||
@@ -39,9 +39,9 @@ We explicitly do **not** recommend the use of `spec.nodeName` for operator-drive
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Flexibility:** RWX users benefit from the ability to have multiple pods access the volume simultaneously, simplifying future operational extensions.
|
||||
- **Node Coupling:** All users are coupled to the node where the runner pod is running. The hook ensures job pods are scheduled on the same node to maintain workspace integrity.
|
||||
- **Configuration:** Operators must be aware of the `ACTIONS_RUNNER_USE_KUBE_SCHEDULER` toggle when moving from RWX to RWO. This toggle controls whether the hook uses `nodeName` (bypassing the scheduler) or node affinity (using the scheduler) to pin the pod to the runner's node.
|
||||
- **Flexibility:** RWX users benefit from the ability to schedule job pods on any node in the cluster, maximizing resource utilization.
|
||||
- **Node Coupling:** RWO users remain coupled to the node where the runner pod is running. The hook ensures job pods are scheduled on the same node via affinity to maintain workspace integrity.
|
||||
- **Configuration:** Operators must be aware of the `ACTIONS_RUNNER_USE_KUBE_SCHEDULER` toggle when using RWO. This toggle controls whether the hook uses `nodeName` (bypassing the scheduler) or node affinity (using the scheduler) to pin the pod to the runner's node. RWX configurations do not require this toggle for basic operation.
|
||||
|
||||
## Migration Guidance
|
||||
|
||||
|
||||
@@ -36,12 +36,12 @@ rules:
|
||||
The K8s hooks require a shared volume between the runner pod and the job pods to share the workspace and other internal directories.
|
||||
|
||||
### RWX (Recommended)
|
||||
The preferred way to configure storage is using a `ReadWriteMany` (RWX) Persistent Volume Claim. While job pods are always pinned to the runner's node, RWX provides better operational flexibility by allowing multiple pods to access the same workspace simultaneously.
|
||||
The preferred way to configure storage is using a `ReadWriteMany` (RWX) Persistent Volume Claim. RWX allows the Kubernetes scheduler to place job pods on any node in the cluster, maximizing resource availability and flexibility.
|
||||
|
||||
To migrate from RWO to RWX:
|
||||
1. Provision a new `ReadWriteMany` StorageClass if one is not available.
|
||||
2. Update your PVC definition to use `accessModes: [ReadWriteMany]`.
|
||||
3. Set `ACTIONS_RUNNER_USE_KUBE_SCHEDULER=true` to enable the scheduler-based node pinning (via affinity) instead of the default `nodeName` pinning.
|
||||
3. Remove the `ACTIONS_RUNNER_USE_KUBE_SCHEDULER` environment variable, as affinity is no longer required for pod placement.
|
||||
|
||||
### RWO Fallback (Affinity-based)
|
||||
If `ReadWriteMany` storage is not available, you can use `ReadWriteOnce` (RWO) storage. In this mode, all job pods must be scheduled on the same node as the runner pod that owns the PVC.
|
||||
|
||||
Reference in New Issue
Block a user