Cases6 cases

1 to 5 of 6

1. IL-0098 After 2i2c moved from ingress-nginx to nginx-ingress, a hub's Ingress took priority over cert-manager's temporary Ingress for the same host, so new hubs could not get TLS certificates

   2i2c2026

   Symptoms

   An attempt to launch a new hub failed because its ingress could not obtain a new TLS certificate. The day before, the same engineer had recreated the support deployment on the 2i2c-aws-us cluster and seen its TLS certificate orders stuck pending.

   Misrouted requestsDeploys failing

   Trigger

   Creating new hub deployments that needed new TLS certificates on clusters recently migrated from ingress-nginx to nginx-ingress.

   Root cause

   Breaking change in an upgrade

   Component

   Ingress controllercert-manager

   Contributing factor

   Change with a delayed effect

   Detection

   An attempt to launch a new hub failed; the engineer had seen certificate orders stuck pending on another cluster the day before and declared an incident.

   Remediation

   • annotated the Ingress objects that need TLS so that cert-manager could edit them in place to complete the ACME challenge

   Action items

   • Follow-up tracked in https://github.com/2i2c-org/infrastructure/issues/8016
   • Follow-up tracked in https://github.com/2i2c-org/infrastructure/issues/3278

   Lessons

   Swapping one ingress controller for another can change how Ingress objects that share a host are combined. cert-manager's temporary challenge Ingress depends on that behaviour, so the breakage shows up when a deployment needs a new certificate. 2i2c notes that it got lucky: it had already learned of this possible cause while testing on an internal cluster, and changing the cert-manager configuration was quick.

   View case
2. IL-0405 An expired AWS Load Balancer Controller webhook certificate left no healthy targets and took BSEE SaaS down

   PortSwigger2026

   Symptoms

   From 10:00 UTC on 11 January 2026 every BSEE SaaS instance in PortSwigger's US and Canada regions was unavailable: customers could not open the Web UI or start scans. The platform team learned of it only when customers reported it through support at 09:52 UTC the next day.

   Service outageTLS certificate errors

   Trigger

   The upgrade of the AWS Load Balancer Controller from v1.14.1 to v1.16.0 on 28 November 2025 left an orphaned webhook TLS secret that cert-manager could not renew, and the certificate in it expired on 8 January 2026.

   Root cause

   Breaking change in an upgrade

   Component

   Ingress controllerAdmission webhookcert-managerHelmConfigMaps and SecretsCloud load balancer

   Contributing factor

   Change with a delayed effectMonitoring blind spot

   Detection

   Customers reported the outage through support at 09:52 UTC on 12 January 2026, almost 24 hours after impact began, and the platform team began investigating at 10:00 UTC.

   Remediation

   • Deleted the orphaned aws-load-balancer-tls secrets in the US and Canada clusters
   • Redeployed the AWS Load Balancer Controller configuration with Helm
   • Verified that new certificates were generated and the webhooks worked again

   Action items

   • Audit all clusters for orphaned secrets that have no owning Certificate resource and resolve them
   • Add high-level application access alerting for each SaaS cluster from an external probe

   View case
3. IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replaced

   Honeycomb2024

   Symptoms

   Event ingestion and the Honeycomb UI in EU1 became unavailable at 22:00:10 UTC on April 9, 2024, and the status page reported that the EU1 instance was down. Both the ingestion service and the service behind the web UI stopped at roughly the same moment.

   Service outageStale endpoint routing

   Trigger

   A routine system update to the Kubernetes cluster, rolled out earlier on April 9, followed by a regular deployment that replaced pods one at a time.

   Root cause

   Breaking change in an upgrade

   Component

   Ingress controllerCloud load balancer

   Contributing factor

   Monitoring blind spotUnrepresentative stagingChange with a delayed effect

   Detection

   The first status post, saying the EU1 instance was down, went up at 15:08 PDT (22:08 UTC), about 8 minutes after the outage began. There was no alerting on the AWS Load Balancer Controller.

   Remediation

   • Restarted the AWS Load Balancer Controller
   • Rolled back the Kubernetes cluster system update

   Action items

   • Find the exact cause of the failure so the system update can be deployed safely
   • Add monitoring and alerting on the AWS Load Balancer Controller so on-call engineers learn of a failure quickly and can pause deployments

   Lessons

   A rolling deployment can look healthy while the component that wires new pods into the load balancer is broken; the failure only shows when the last old pod goes away. Honeycomb now treats the AWS Load Balancer Controller as a critical component that needs its own alerting, so deployments can be paused before they remove the last pods that still receive traffic. Extensive testing in development and staging did not reveal the bad interaction.

   View case
4. IL-0513 PyPI backends on Kubernetes were unreachable over HTTPS for 20 minutes when the Ingress ELB lost its HTTPS listener

   Python Software Foundation2022

   Symptoms

   From 14:43 to 15:03 UTC on 24 October 2022, during regular maintenance of the Kubernetes cluster that hosts PyPI, PyPI's backends could not be reached over HTTPS. The CDN could not fetch pages, uploads to uploads.pypi.org were impossible, and other services such as the legacy file redirect service were interrupted.

   Service outage

   Trigger

   Regular maintenance did a rolling restart of every node in the cluster to roll out upgrades and new configuration; as the new Kubernetes API server hosts came up, Kubernetes validated and refreshed the Ingress configurations.

   Root cause

   Configuration change

   Component

   Ingress controllerCloud load balancer

   Contributing factor

   Configuration not in codeChange with a delayed effect

   Remediation

   • Pointed the Ingress configuration at the new ACM TLS certificate, and Kubernetes restored the HTTPS listener on the ELB

   Action items

   • Investigate managing the hostnames on the ACM TLS certificates for PyPI's Ingress through Kubernetes resources instead of the AWS console

   Lessons

   When every resource is managed through the Kubernetes API, drift between the desired state and reality is less likely to build up and then surface during later maintenance.

   View case
5. IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on Nginx

   LINE Plus2022

   Symptoms

   The graph of WebSocket connections to messaging-hub, the messaging platform used by the Japanese food delivery service Demaecan, showed sudden deep drops even outside peak traffic. The drops kept coming back at irregular intervals. Messages still reached clients, because messaging-hub switches to Firebase Cloud Messaging when a WebSocket is gone.

   Connection resetsDatabase overloadIntermittent connection failuresHigh latency

   Trigger

   Another service that shared the same L7 load balancer deployed its own load balancer configuration in the Verda dashboard, which reloaded HAProxy; 30 minutes later, at the hard-stop-after limit, the old HAProxy process exited and closed every WebSocket connection it still held.

   Root cause

   Configuration change

   Component

   HAProxyCloud load balancerconntrack tableIngress controller

   Contributing factor

   Change with a delayed effectExcessive blast radiusRetry stormUnverified assumption

   Detection

   The developer noticed deep, sudden drops in the WebSocket connection graph that were not tied to peak traffic and kept recurring irregularly. The Nginx packet drops were found through network monitoring.

   Remediation

   • separated the Merchant system's token issuance API into its own deployment so reconnect traffic did not delay other Merchant APIs
   • scaled out Nginx with pod anti-affinity and a label selector so each pod ran on a different node, which stopped the nf_conntrack packet drops
   • raised Ingress Nginx proxy-read-timeout and proxy-send-timeout and the load balancer timeout to 3600s, which did not stop the drops
   • moved messaging-hub from the shared L7 load balancer to an L4 load balancer, which ended the WebSocket drops

   Lessons

   A reload of a shared load balancer by a different team can cut long-lived connections, and with hard-stop-after the cut comes 30 minutes later, far from the change that caused it. The author's lesson is not to narrow the scope of the search too early: trusting that the infrastructure was fine kept the investigation inside the team's own systems for days. A reconnect wave from many WebSocket clients can overload the systems that issue connection tokens and fill nf_conntrack on proxy nodes. Scaling Nginx out only helps for conntrack if the pods land on different nodes, and anti-affinity leaves extra replicas Pending once they outnumber the nodes. That messaging-hub problems can still load the connected systems remains an open problem.

   View case

By root cause3