The Afternoon the Cluster Ate Itself

Problem

On the afternoon of May 12, seven worker nodes on our self-managed RKE2 QA cluster went under disk pressure at nearly the same time. The kubelet started evicting pods, the evicted pods rescheduled onto the surviving nodes and pushed them toward the same edge, and within the hour the cluster was in a full eviction cascade.

Then the part that turned an incident into a siege: Harbor — the in-cluster registry — lost its scheduling home. Every pod trying to recover needed to pull an image from a registry that was down because of the same incident it was needed to fix.

Context

The cluster ran a rapid-QA tier: 28 namespaces of full product environments, in-cluster CI (Tekton building through Harbor), GitOps via Argo CD, and Longhorn for storage — with Longhorn's data path sharing the root filesystem with the kubelet, no reservation between them, and no ephemeral-storage limits on any workload. The cluster had been drifting toward its capacity edge for weeks; disk pressure was the tripwire, not the disease.

Investigation

The first hours were a catalogue of reasonable fixes that didn't work, and the dead ends are the real story:

Pausing Argo CD applications didn't shed a single pod. Sync pause stops Git-driven changes — the Deployment controllers already in the cluster kept respawning everything regardless.
Image pruning freed 0 bytes. Every image on the tight nodes was still referenced by a Pending pod waiting to run. The garbage collector was technically correct and completely useless.
Cordoning the tightest nodes made things worse. Load concentrated on fewer nodes, container churn spiked, and kubelets started failing PLEG (pod lifecycle event) health checks — which flapped the nodes NotReady and triggered more rescheduling.
Restarting rke2-agent couldn't unstick a hung kubelet. Restarted agents came back, went NotReady again within minutes, and on the worst nodes the kubelet hung silently for 30+ minutes on startup, relisting the wreckage before reporting anything.

The signal that finally pointed somewhere useful: crictl showed enormous piles of orphaned pod sandboxes on the affected nodes — dead pods the kubelet had lost track of, still holding their writable layers.

Root Cause

Two conditions had to be true at once. First, no ephemeral-storage limits anywhere: any pod could write unbounded data to the node disk, and across 28 busy namespaces, many did. Second, the cluster was already at its capacity edge, so the eviction wave had nowhere to land — recovery required free pod slots and free disk, and the incident consumed both. Harbor going down was the multiplier: the recovery path itself depended on a component inside the blast radius.

Solution

What actually worked, in order:

crictl rmp on orphaned sandboxes across the affected nodes reclaimed 33–47GB per node — the single biggest source of relief.
Reboots instead of service restarts. A hung kubelet doesn't get better by restarting the agent above it; a clean reboot took wedged nodes from a 30-minute silent hang to a normal startup. We rebooted the stuck workers in parallel rather than one cautious node at a time.
Scaling five dev namespaces to zero freed roughly 390 pod slots in one move. With actual headroom available, Harbor scheduled within seconds — and once the registry was back, everything else could finally pull images and recover on its own.

Total time from first eviction to a stable cluster: about four hours.

Technical Decisions

Load shedding over node surgery. The instinct is to fix nodes one by one; the win came from reducing what the cluster was being asked to run. Dev namespaces were the designated sacrifice — we made that an explicit, pre-agreed lever afterward.
Parallel reboots were the right risk. With Longhorn replicas spread across nodes, rebooting several workers at once risked volume availability — but the volumes were already degraded, and serial reboots would have stretched a 4-hour incident into a full day.
We stopped trusting "restart the agent" as a remediation for anything kubelet-side and wrote it down: hung kubelet → reboot, no intermediate steps.

Lessons Learned

Your recovery path can't depend on the thing that's down. An in-cluster registry is fine — until the incident that takes it down is the one where every recovering pod needs it. We documented which cluster components are recovery dependencies and what the fallback is for each.
Unbounded ephemeral storage is a cluster-wide contract violation waiting to happen. The aftermath: ephemeral-storage LimitRanges on every tenant namespace, so one workload's disk appetite becomes that pod's problem, not the node's.
Orphaned sandboxes are silent disk debt. A weekly cleanup CronJob now handles what crictl found the hard way.
A cluster at its capacity edge has no recovery capacity. Headroom isn't waste — it's the resource incidents run on.