IL-0064 Moonlight · 2019-01-18kubernetesgcp
Moonlight's website and API had intermittent outages from January 18 to January 22, ending in periods of complete downtime. The application ran on a managed Kubernetes cluster on Google Cloud with three nodes, and every web-facing application had three replicas. The team expected one replica per node, so that two nodes could fail before the site went down.
The scheduler did not spread the replicas. It sometimes put all three website pods on one node, together with other CPU-heavy work such as server-side rendering. Those pods drove the node to 100% CPU, the node's kernel panicked, and the node stopped responding. The scheduler then moved all the crashed pods to another node, which went through the same cycle, and the effects compounded. What started with the Redis pod ended with every web-serving pod offline.
The team first assumed a Google Cloud network outage, because Google had reported networking problems at the time. Google Cloud Support identified the link between sustained full CPU and kernel panics. Adding anti-affinity rules to all major deployments, so their pods are spread across nodes, restored the site. The cause of the kernel panics themselves was still being investigated with Google when the post was written.
On Friday, January 18, Moonlight's API began returning occasional errors that looked like connection problems with its Redis database. The API checks Redis on every authenticated request to validate the session. At the same time the Kubernetes monitoring reported some nodes and pods as unresponsive.
Over the weekend, as traffic dropped, the errors mostly went away. On Tuesday morning the website was completely offline, with no external traffic reaching the cluster at all.
The team first blamed Google Cloud: Google had reported networking service disruptions, and another person on Twitter described similar symptoms. Google Cloud Support escalated the case to its support engineers, who found the real pattern: whenever a node ran at 100% CPU for a sustained period, the virtual machine hit a kernel panic and crashed.
The failure grew over the days. At first only the Redis pod was crashing; later every pod serving web traffic went offline. Exponential back-off rules made the periods of downtime longer.
Symptom High error rateIntermittent connection failuresNode NotReady
Root cause Capacity exhaustion
API errors that looked like Redis connectivity problems, and Kubernetes monitoring that showed nodes and pods as unresponsive. Google Cloud Support later found the link between sustained 100% CPU and kernel panics.
Three replicas give no fault tolerance if the scheduler places them all on one node; anti-affinity rules make the spread explicit. A node failure that comes from the workload itself moves with the workload when the scheduler reschedules it.
IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025same root cause · shared Pod scheduling and placement, Application database
IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation이언철 · 2024same root cause · shared Pod scheduling and placement
IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024same root cause · shared Application database
IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recoverHoneycomb · 2023 · sev1same root cause · shared Application database
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Application database
IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999LINE · 2020same root cause · shared Linux kernel
IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instancesDatadog · 2019same root cause · shared Application database
IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was downUniverse · 2017same root cause · shared Pod scheduling and placement