The 150GB Volume That Weighed 190GB

Problem

Every build in the cluster started failing with I/O errors at once. The trail led to Harbor, the in-cluster registry, whose storage volume had gone read-only — and from there to a node whose 196GB Longhorn disk was at 100%.

The confusing part: the registry volume was provisioned at 150GB and wasn't even full at the volume level. Nominal usage looked like roughly 60%. The disk underneath it disagreed, violently.

Context

The registry volume lived on Longhorn, the cluster's distributed block storage. Two configuration facts set the stage: the storage class allowed 200% over-provisioning (you can promise more than you physically have, betting volumes won't all fill), and there were zero snapshot-cleanup RecurringJobs configured anywhere. Longhorn creates system snapshots as part of normal replica operations — rebuilds, expansions — and something has to eventually merge and purge them.

Nothing ever did.

Investigation

Inspecting the replica directory on the full disk showed where the missing ~100GB lived: a long chain of snapshots marked as removed but never actually purged. In Longhorn's copy-on-write model, a "removed" snapshot's blocks stay on disk until the chain is coalesced. The volume's live data was 150GB nominal; the physical replica — live data plus the ghost blocks of every dead snapshot under it — weighed about 190GB. On a 196GB disk.

Three traps revealed themselves during the fix, each nastier than the diagnosis:

• Over-provisioning had hidden the problem for months. Every dashboard that showed volume-level usage said ~60%. The only number that mattered — physical bytes on the one disk that was full — wasn't on any dashboard we looked at.
• Snapshot coalescing needs temporary space roughly equal to the snapshot being merged. On a full disk, the cleanup operation itself re-faults. The cure required the one resource the disease had consumed.
• A replica rebuild spreads the bloat. Longhorn's natural recovery response — rebuild the replica elsewhere — would copy the entire physical chain, ghost blocks included, onto a second node's disk. The failure mode was contagious.

Meanwhile the obvious-looking fixes were all at the wrong layer: Harbor's own garbage collection and tag pruning operate on registry artifacts inside the filesystem. No amount of registry GC can free blocks pinned by the storage layer underneath the filesystem. And the cluster as a whole had 2.4TB free — this was never a capacity problem, it was a distribution problem concentrated on one disk.

Root Cause

Dead snapshots with no garbage collector. Longhorn faithfully marked snapshots as removed and then — with no RecurringJob configured to purge and coalesce — kept every block forever. Over-provisioning ensured the debt could grow far past the physical disk before anything complained, and the first complaint was ext4 aborting to read-only mid-write.

Solution

In order of operations, because the order was the hard part:

1. Grow the disk first. Coalescing needed working room that didn't exist; expanding the underlying disk gave the purge space to run without re-faulting.
2. Manually salvage the snapshot chain on the registry volume — walk the removed snapshots, purge, and coalesce until physical size converged back toward nominal.
3. Recover the filesystem and bring Harbor back; builds resumed.
4. Make the garbage collector exist: daily snapshot-cleanup RecurringJobs on every volume, so "removed" once again means "will actually be deleted."

Technical Decisions

• Grow-then-clean over clean-in-place: attempting the coalesce on the full disk was tempting (no infra change needed) and would have failed exactly the way the traps predicted. Buying space first turned a risky salvage into a boring one.
• No replica rebuilds until the chain was clean — deliberately holding off Longhorn's own recovery instinct so the bloat wouldn't colonize a second node.
• Registry-layer GC was left alone during the incident. It was the wrong tool, and running it mid-salvage would only have churned the filesystem the storage layer was trying to stabilize.

Lessons Learned

• Every layer needs its own garbage collector — and needs you to verify it's actually configured. Harbor GC cleans Harbor. Nothing cleans Longhorn snapshots unless you schedule it.
• Over-provisioning is deferred debt with a hidden due date. It's a fine tool, but only alongside monitoring of physical per-disk usage — nominal volume usage is a comforting fiction.
• Cleanup operations have resource costs too. Any recovery that needs free space, memory, or connections must be run before the resource is exhausted — which means alerting thresholds have to leave room for the cure, not just the diagnosis.
• "The cluster has plenty of space" and "this disk is full" are both true. Aggregate capacity metrics hide exactly the failures that matter.