IL-0093 UK Ministry of Justice · 2020-08-14kubernetes
The cluster ran six replicas of the ingress controller. The leader pod among them had exhausted its shared memory, and the other replicas could not connect to it to get the latest nginx configuration. Two replicas were crash-looping, and following the runbook by restarting more replicas only added more crash-looping pods, four of six in the end.
Restarting the leader ingress controller pod let the other replicas connect and fetch the latest nginx.config file, and all of them returned to ready and running. The report gives the time to repair as 37 minutes and the time to resolve as 55 minutes.
At 10:43 on 14 August 2020 two of the six ingress controller pods in the cluster were in a crash loop. Restarting those pods did not fix them.
The normal runbook said to recycle all the pods. After pods 4 and 5 were restarted they also began to crash-loop, so four of the six replicas were down. Restarting the remaining pods risked leaving all six in a crash loop and every ingress in the cluster down. The incident was declared at 11:01.
The pod that had not yet been looked at or restarted was the leader ingress controller pod, and it had exhausted its shared memory.
Symptom Container crash loop
Root cause Capacity exhaustion
Two of the six ingress controller pods were seen crash-looping at 10:43; the incident was declared at 11:01.
–
–
IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app servicesBackpackr · 2025same root cause · shared Ingress controller
IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifestZalando · 2018same root cause · shared Ingress controller
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.