IL-0011 Skyscanner · 2019-04-23 (published)kubernetesnetworking
Skyscanner exposes services through the open source HAProxy ingress controller, a Go wrapper that watches the Kubernetes API server for pods and services, generates the HAProxy configuration from a template with Go's text/template library, and reloads HAProxy. It creates one HAProxy backend per service. When a graceful reload is not possible the controller restarts the HAProxy process, so round-robin load balancing starts again from the top of each server list. The controller runs as a DaemonSet, about 80 instances on the largest clusters, behind an ELB that spreads traffic across them.
In v0.3 the data structure holding each backend's servers sorted them, so every HAProxy instance listed pods in the same order, and every non-graceful restart sent all instances back to the same first pod. The workaround was a default annotation on all services to use the Service virtual IP instead of pod IPs, which left load balancing to Kubernetes but meant HAProxy could no longer mark individual pods unhealthy. When the v0.5 documentation introduced sort-backends with a default of false, the team read the code, tested briefly and upgraded to v0.6, returning to pod IPs. The load test ran on a fairly static cluster, so it never produced the frequent configuration rewrites and non-graceful restarts of the production clusters.
The Go code did shuffle the servers, and kept them shuffled right up to templating. The template, lightly modified by Skyscanner from upstream, iterated with range over a map keyed by pod_ip:port. In Go code, range over a map has no guaranteed order, but text/template has visited maps with comparable keys in sorted key order since Go 1.0, so every instance listed pods by IP in ascending order. On the Sunday the team stopped the damage by switching the affected service back to its Service virtual IP through a config change in Slingshot, the internal deployment tool. The fix was a small controller patch that keeps a slice in insertion order and a template that iterates over that slice and looks each entry up in the map. After testing in sandbox clusters and rolling it out to production, the request graphs evened out. The upstream pull request broke functionality Skyscanner did not use, because the project had no test suite, so the maintainer shipped an alternative fix in the next release and offered a backport to v0.6.
On a Sunday, a team using one of Skyscanner's Kubernetes clusters manually paged the out-of-hours Kubernetes support because their service showed elevated latency, worst at the p90 and maximum levels.
After some joint digging and a few false starts, the engineers found that the request queues of a few pods of the service were full and those pods refused further requests, while other pods of the same service received far less. Prometheus queries showed the pattern the platform team called "hot-podding": traffic that came in through the ingress layer was spread unevenly, with some pods consistently getting a much higher request rate than others.
The imbalance fed itself, in what the team described as a "spiral of failures". A pod with a full queue correctly marked itself unready, which removed it from its HAProxy backend, which caused a configuration reload, which made the next pod in the list the overloaded one. Some ingress controller containers became so overloaded with requests that they were OOMKilled, which put the remaining ones under even more stress.
There had been warnings. Users had reported hot-podding back on v0.3 of the ingress controller, and after the upgrade to v0.6 one squad again pointed out unusual request distribution across its pods. The platform team filed that as a curiosity: its load test, its reading of the code and the fact that only one squad had reported it made it look like a low priority.
Symptom High latencyLoad imbalance
Root cause Software defect
A cluster user manually paged the out-of-hours Kubernetes support about elevated latency; Prometheus queries then showed the uneven request distribution across pods.
The team already knew that sorted backends cause hot-podding and still did not test the new flag well enough to see that it did not work; changes to cluster components need tests aimed at the exact behaviour that matters, under conditions like production. A keyword can behave differently in different parts of a language: range over a map is unordered in Go code but sorted in text/template. A small cluster component can do outsized harm to every user of the cluster and, in the end, to customers.
IL-0053 Unauthenticated ingress-nginx admission webhook let anyone on the pod network run code and read every Secret in the cluster (IngressNightmare, CVE-2025-1974)Wiz · 2025same root cause · shared Ingress controller
IL-0081 API requests timed out during deployment rollouts because the ingress-nginx enable-serial-reloads option held back endpoint updates while a long NGINX reload was runningCresta · 2025same root cause · shared Ingress controller
IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404DFDS · 2021same root cause · shared Ingress controller
IL-0062 Intermittent 502 errors behind a GKE Ingress because pods shut down before the load balancer stopped sending them trafficRavelin · 2019same root cause · shared Ingress controller
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Ingress controller