Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

Ingress controller

인그레스 컨트롤러 · ingress

25 cases · 2017 to 2026

Root cause Software defect6 Breaking change in an upgrade5 Operator error4 Capacity exhaustion3 Faulty deployment3 Configuration change2 Cloud provider outage1 DNS or network fault1

Contributing factor Unreviewed defaults6 Unverified assumption5 Excessive blast radius5 Monitoring blind spot5

Symptom Service outage13 High error rate7 Stale endpoint routing6 Deploys failing3

Organisation UK Ministry of Justice5 2i2c2 DFDS2 OpenAI2 Backpackr

Cases, by severity

IL-0098 After 2i2c moved from ingress-nginx to nginx-ingress, a hub's Ingress took priority over cert-manager's temporary Ingress for the same host, so new hubs could not get TLS certificates2i2c · 2026
Root cause Breaking change in an upgrade
IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applicationsUK Ministry of Justice · 2026
Root cause Breaking change in an upgrade

View on the graph

Unrepresentative staging5

Change with a delayed effect4

Single point of failure4

No safeguard on destructive change3

Ignored early signal2

Overlapping failures2

Automation amplification1

Harmful manual action1

No configuration tests1

No staged rollout1

Missed in review1

Deferred upgrades1

Intermittent connection failures3

Intermittent timeouts3

Resources deleted3

Database overload2

Probe failures2

Autoscaler misbehaviour1

Scale-up failure1

Connection resets1

Control plane unresponsive1

Container crash loop1

Monitoring outage1

Misrouted requests1

Stale or inconsistent API reads1

Unauthorised access or a foreign workload1

Load imbalance1

1

Lotte E-commerce1

IL-0188 Rolling updates on EKS returned 502 errors for about 30 seconds because Kubernetes deleted old pods as soon as new pods passed readiness, before the ALB had marked the new pod targets healthy

Yonghyun Kim · 2025

Root cause Faulty deployment

IL-0095 An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour2i2c · 2025

Root cause Operator error

IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app servicesBackpackr · 2025

Root cause Capacity exhaustion

IL-0053 Unauthenticated ingress-nginx admission webhook let anyone on the pod network run code and read every Secret in the cluster (IngressNightmare, CVE-2025-1974)Wiz · 2025

Root cause Software defect

IL-0081 API requests timed out during deployment rollouts because the ingress-nginx enable-serial-reloads option held back endpoint updates while a long NGINX reload was runningCresta · 2025

Root cause Software defect

IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start queryOpenAI · 2024

Root cause Breaking change in an upgrade

IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replacedHoneycomb · 2024

Root cause Breaking change in an upgrade

IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALBLotte E-commerce · 2022

Root cause DNS or network fault

IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on NginxLINE+ · 2022

Root cause Configuration change

IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgradedOpenAI · 2022

Root cause Operator error

IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404DFDS · 2021

Root cause Software defect

IL-0111 Switching the main Kubernetes ingress controller from Traefik v1 to v2 stopped X-Forwarded headers being passed, which broke the Okta login flow for every service using Okta and stopped DFDS online bookings for over an hourDFDS · 2021

Root cause Configuration change

IL-0090 After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and changeUK Ministry of Justice · 2021

Root cause Breaking change in an upgrade

IL-0086 A deployment meant for a CircleCI test Kubernetes cluster also removed Traefik and Calico resources from the production cluster, taking down the CircleCI UI and runner.circleci.comCircleCI · 2021

Root cause Faulty deployment

IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dnsUK Ministry of Justice · 2020

Root cause Operator error

IL-0088 A validating admission webhook added by a new ingress controller module version made the live-1 cluster reject every new Ingress with an x509 errorUK Ministry of Justice · 2020

Root cause Faulty deployment

IL-0093 Ingress controller pods crash-looped one after another because the leader pod had exhausted its shared memoryUK Ministry of Justice · 2020

Root cause Capacity exhaustion

IL-0062 Intermittent 502 errors behind a GKE Ingress because pods shut down before the load balancer stopped sending them trafficRavelin · 2019

Root cause Software defect

IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP orderSkyscanner · 2019

Root cause Software defect

IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPAloveholidays · 2019

Root cause Cloud provider outage

IL-0054 All ingress traffic lost while draining a GKE node pool because the load balancer only targeted new nodes without ingress-nginx podsDevOps Hof · 2019

Root cause Operator error

IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifestZalando · 2018

Root cause Capacity exhaustion

IL-0072 High etcd write latency made the API servers return stale data, and the production cluster flipped between two views of its stateNordstrom · 2017

Root cause Software defect