Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion roles/github_runner_arc/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ A pruner DaemonSet (`scripts/node-cache-pruner.sh`, from the autoscaler image, o
- **The credential must be able to read cluster roles.** Release 1.31.0 installed the pruner with a cluster-wide `ClusterRole` and `ClusterRoleBinding` named `node-cache-pruner`; every run now looks for them by name and removes them, binding first, only if they are what 1.31.0 created (a role that lists pods and nothing else, not aggregated, and a binding of it to the pruner's account in the platform namespace and nothing else; 1.31.0 set no labels, so the content is the identity). The lookup needs the credential the role reaches the cluster with to be allowed to `get` ClusterRoles and ClusterRoleBindings (cluster-scoped, read-only), also on a fleet that never ran 1.31.0, and `delete` on them only when it finds the 1.31.0 objects. A role of that name whose binding is not the one 1.31.0 created (the platform namespace changed, another subject was added) is left in place with the binding and reported once per run with a line starting `NODE_CACHE_LEGACY_ACCESS_LEFT`, so a dangling binding can never regain cluster-wide pod listing; delete them by hand if they are not wanted. The cleanup goes in a later release (ExaDev/github-runner#92). A flag that skipped the lookup on a fleet that never ran 1.31.0 would be one more setting to maintain, so there is none.
- **Upgrading from 1.31.0:** the host path root is now validated whether or not any profile has a node cache, so a configuration whose `github_runner_arc_node_cache_host_path` does not end in `node-cache` (the default does) fails validation even if it uses no node-cache capability; set the path to a conforming one (a new path starts an empty cache, and the old directory is yours to delete).
- **A cluster that ran release 1.31.0** has a cluster-wide grant to list pods named `node-cache-pruner`; the role removes the 1.31.0 objects where it finds them (binding, then role), subject to the paragraph above: only what matches what 1.31.0 created, and a role whose binding is not the one 1.31.0 created is left with it and reported.
- **Burst nodes are not pruned.** The pruner runs only on nodes that carry `github_runner_arc_node_label_key`, and a burst node carries the burst label and taint but, by design, never the eligibility label (checked against how Nexsys-Analytics/ci-runners labels its burst nodes: `docs/spec.md` section 2, "never the eligibility label"). A profile that overflows onto burst nodes therefore also gets a cache directory there, which nothing empties. That is harmless where burst nodes are what they are meant to be, short-lived instances removed once idle, since the directory goes with the node; if burst nodes live long enough for a cache to outgrow their disk, give them the eligibility label so the pruner runs there.
- **Burst nodes are not pruned.** The pruner runs only on nodes that carry `github_runner_arc_node_label_key`, and a burst node carries the burst label and taint but, by design, never the eligibility label. A profile that overflows onto burst nodes therefore also gets a cache directory there, which nothing empties. That is harmless where burst nodes are what they are meant to be, short-lived instances removed once idle, since the directory goes with the node; if burst nodes live long enough for a cache to outgrow their disk, give them the eligibility label so the pruner runs there.
- **A bad setting stops the pruner before it does anything.** The pruner's start-up checks end it with status 2 and a message naming the variable, and the DaemonSet's pods then crash-loop with that message in their log. The role refuses the numeric settings it templates by the same pattern and maxima (the size limit, the hard limit factor and the check interval: positive integers without sign, leading zero or fraction, up to 1073741824 GiB, 1024 and 86400), so only a value changed some other way, such as by editing the DaemonSet, reaches that check.
- **A failure never stops the others:** a size that cannot be read skips that profile for the round, and a pod list that fails or is not a pod list means no pruning (`NODE_CACHE_UNPRUNABLE` when it is refused as Forbidden, `NODE_CACHE_LIST_FAILED` otherwise), since an unknown set of pods may be using the cache.

Expand Down
Loading