IL-0203 Honeycomb · 2024-04-09kubernetesaws
On April 9, 2024, event ingestion and the UI of Honeycomb's EU1 instance were unavailable from 22:00:10 to 22:11:30 UTC. Earlier that day Honeycomb had begun rolling out a routine system update to its Kubernetes cluster. The update had been tested extensively in development and staging, but in production it interacted badly with other components of the cluster and made the AWS Load Balancer Controller and other pods fail sporadically.
With the controller down, new pods were no longer registered with the ALB. The next deployment replaced the old pods one at a time, the ALB kept sending traffic only to the old pods, and when the last of them was terminated the ingestion service and the UI service lost all their traffic. Honeycomb restarted the AWS Load Balancer Controller to restore service and rolled back the system update so the failure would not come back. The exact cause of the interaction was still being investigated when the write-up was posted.
Event ingestion and the Honeycomb UI in EU1 became unavailable at 22:00:10 UTC on April 9, 2024, and the status page reported that the EU1 instance was down. Both the ingestion service and the service behind the web UI stopped at roughly the same moment.
The outage came at the end of an ordinary deployment. Honeycomb deploys by terminating pods and starting new ones one at a time. The new pods were working, but no traffic was forwarded to them; service broke off when the last old pod was terminated.
The reason was that the AWS Load Balancer Controller, which registers each new pod with the Application Load Balancer (ALB) so the ALB starts sending it traffic, had failed earlier. The routine system update rolled out to the cluster that day made the controller and other pods fail sporadically.
Root cause Breaking change in an upgrade
The first status post, saying the EU1 instance was down, went up at 15:08 PDT (22:08 UTC), about 8 minutes after the outage began. There was no alerting on the AWS Load Balancer Controller.
A rolling deployment can look healthy while the component that wires new pods into the load balancer is broken; the failure only shows when the last old pod goes away. Honeycomb now treats the AWS Load Balancer Controller as a critical component that needs its own alerting, so deployments can be paused before they remove the last pods that still receive traffic. Extensive testing in development and staging did not reveal the bad interaction.
IL-0098 After 2i2c moved from ingress-nginx to nginx-ingress, a hub's Ingress took priority over cert-manager's temporary Ingress for the same host, so new hubs could not get TLS certificates2i2c · 2026same root cause · shared Ingress controller
IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applicationsUK Ministry of Justice · 2026same root cause · shared Ingress controller
IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress trafficAltinity · 2025same root cause · shared Cloud load balancer
IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start queryOpenAI · 2024same root cause · shared Ingress controller
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Ingress controller