Production incidents written up honestly: the problem, the investigation, the root cause, and what I'd do differently. No sanitized success stories — the value is in the dead ends.
SEV-2● Resolvedsame dayAug 6, 2026
A production OCR worker failed every batch with PermissionError — in a directory where every file was rwxrwxrwx, while the file it 'failed' to copy sat fully written on disk. shutil.copy is two operations, and the second one needs ownership, not write permission.
cause shutil.copy chmods the file it just wrote→fix shutil.copyfile — copy bytes, not metadata→lesson write permission ≠ chmod permission
Python · NFS · docker-compose · RabbitMQ—5 min read
SEV-2● Resolvedmonths in the makingJul 27, 2026
Backend views started throwing NoneType errors on Redis reads, and the Redis pod's logs were spotless — because OOMKill is a SIGKILL, and dead processes don't write farewell notes.
cause no-TTL job keys + RDB reload of the bloat→fix purge stale keys; real fix is TTLs at write time→lesson check Last State, not logs — SIGKILL leaves no trace
Redis · Kubernetes · Django—3 min read
SEV-1● ResolvedunrecordedJun 8, 2026
Zero snapshot-cleanup jobs plus 200% over-provisioning let dead Longhorn snapshots pin ~100GB of ghost blocks under the Harbor registry volume. It filled a 196GB disk, the filesystem went read-only, and every build in the cluster started failing with I/O errors.
cause snapshots marked removed but never purged→fix daily RecurringJobs + grow disk + manual salvage→lesson every layer needs its own garbage collector
Longhorn · Harbor · Kaniko · ext4—4 min read
SEV-1● Resolved~4 hoursMay 12, 2026
Seven workers hit disk pressure, and every standard fix made it worse: restarted agents flapped back NotReady in minutes, kubelets hung silently for 30+ minutes on startup, and Harbor — the registry every recovering pod needed — went down with the cluster it runs on.
cause no ephemeral-storage limits + cluster at capacity edge→fix reboot (not restart) + scale dev namespaces to 0→lesson your recovery path can't depend on the thing that's down
RKE2 · Longhorn · Harbor · Argo CD · kubelet—4 min read