IL-0067 BC Government · 2025-07-09kubernetes
Klamm runs on OpenShift with a Redis cluster as its cache. On 9 July 2025 a Laravel worker produced a large I/O spike, which led to a surge of requests to Redis. Redis's append-only file, the log it writes every change to so it can rebuild its data after a restart, grew quickly and filled the persistent volume of the Redis master, whose storage limit request was 50Mi.
When fsync of the AOF failed for lack of space, Redis could not write its buffered changes to disk and the process stopped, so the container crashed. The last writes to the AOF had not completed cleanly, which left the file corrupt. On each restart Redis tried to load the AOF to recover its data, could not parse it, and crashed again. With the cache down, queue workers timed out and the application went down.
The team backed off the StatefulSet and the deployment, deleted the corrupted PVC and rebuilt it, and restarted both. Rebuilding the PVC lost the data on it. OpenShift events from the incident were not retained.
The Klamm production application was completely unavailable for about 30 minutes outside working hours. Klamm's queue workers timed out because they could not reach the Redis cache, and the services that depend on it were badly affected.
The Redis master's logs showed "Fail to fsync the AOF file: No space left on device". After the Redis container stopped, every restart failed while loading the append-only file (AOF) with an "invalid byte sequence" error, and the pod kept crashing without becoming ready.
Symptom Service outageContainer crash loopPersistent volume full
Root cause Capacity exhaustion
–
–
IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025same root cause · shared In-memory cache
IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hoursMagic Pages · 2025same root cause · shared Persistent volumes and stateful sets
IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recoverHoneycomb · 2023 · sev1same root cause · shared In-memory cache
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.