The Crash That Never Reaches the Logs

There's a class of production incident where the application is obviously failing, the dependency it blames looks perfectly healthy, and the dependency's logs are clean. Not "clean except for a warning" — spotless. If you've been paged for one of these, there's a good chance you were looking at an OOMKill and didn't know it yet.

Why the logs are innocent

When a container exceeds its memory limit, the kernel's OOM killer sends SIGKILL. Not SIGTERM, which a process can catch to flush buffers and write a farewell note — SIGKILL, which cannot be caught, handled, or delayed. The process stops existing between one instruction and the next.

Then Kubernetes does exactly what it's designed to do: restarts the container. And here's the trap — kubectl logs shows you the current process's output. The fresh process started clean and has nothing to report. The victim's testimony died with it. The crash never reaches the logs because, mechanically, it can't.

Where the evidence actually lives

The kill is recorded — just not where habit sends you first.

$ kubectl describe pod redis-0
...
Last State:  Terminated
  Reason:    OOMKilled
  Exit Code: 137
  Finished:  Mon, 27 Jul 2026 14:31:07 +0600
Restart Count: 13

Three signals, in the order I check them:

1. Last State: Terminated / Reason: OOMKilled — the previous container's fate, preserved by the kubelet. This is the smoking gun, one describe away.
2. Exit code 137 — 128 + 9, i.e. killed by signal 9. Even when the Reason field is less explicit, 137 means SIGKILL, and in a container with a memory limit, SIGKILL usually means the OOM killer.
3. Restart count — a pod with 13 restarts is telling you this isn't an event, it's a loop. The kernel's own record (dmesg on the node, cgroup OOM events) confirms it when you need the forensic version.

Corollary worth internalizing: kubectl logs --previous exists and shows the prior container's output — useful for crashes that do manage to log. For OOMKills it mostly proves the point: the previous log just stops mid-sentence.

Why it loops

The incident that taught me to write this up had a particularly vicious shape: Redis with months of no-TTL job payloads (10–24MB each) walking into its 1Gi limit. The OOMKill wasn't the end — Redis persisted via RDB snapshots, so every restart reloaded ~1GB of the same bloat from disk and got killed again within minutes.

That's the general trap for any stateful process: persistence exists to protect your data, and it protects your pathological state with exactly the same fidelity. Restart-based recovery only works if what gets restored isn't what caused the kill.

Preventing the class, not the instance

The fix for one OOMKill is a purge. The fix for the class is at write time and in monitoring:

• Every cache or job-state key gets a TTL at the point of write. An unbounded key set with a bounded memory limit is a crash with a long fuse. Make the working set self-limiting.
• Configure the memory policy deliberately. For a cache, an eviction policy (allkeys-lru or similar) turns memory pressure into cache misses instead of kills. For job state — where eviction means data loss — you need TTLs and alerting instead; noeviction just chooses the kernel as your eviction policy.
• Alert on trajectory, not just failure: working set as a percentage of limit, and restart counts. When we finally checked the fleet after one environment's incident, three more were sitting at 63–82% of their limit with the same write pattern — same crash, scheduled for later. Trajectory alerts buy you the boring version of the incident.
• Right-size limits from measured working sets, not folklore. A limit you never revisit as the workload grows is a tripwire you set for your future self.

The takeaway

A clean log is not evidence of a healthy process. It can equally be evidence of a process that died too fast to speak. When an application blames a dependency whose logs are suspiciously spotless: kubectl describe, Last State, exit code, restart count — then decide who's actually failing.