Incident Library
Applied filters ComponentCloud load balancerSymptomService outageClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases8 cases

8 to 8 of 8

  1. IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dns
    UK Ministry of Justice2020
    Symptoms

    At 18:27 on 21 September 2020 the Cloud Platform team detected that some services running on the live-1 production cluster had gone down, and some users could not reach them. The incident was declared at 18:40.

    Service outageResources deletedMonitoring outageStale endpoint routing
    Trigger

    A script for deleting a test cluster was triggered while the kube context was pointing at the live-1 production cluster instead of the test cluster.

    Root cause
    Operator error
    Component
    Ingress controllerCredential broker for podsexternal-dnsCloud load balancerPrometheus
    Contributing factor
    No safeguard on destructive change
    Detection

    The first detection was at 18:27, and the incident was declared 13 minutes later, at 18:40.

    Remediation
    • restored the default ingress controller together with external-dns and kiam, which it depends on
    • corrected the owner field in the Formbuilder Route 53 TXT records so external-dns pointed their A records at the new NLB
    • restored the remaining components with a targeted apply
    • ran the apply pipeline to restore all certificates, ServiceMonitors and Prometheus rules from the environment repository
    View case
Check your cluster for this

These 8 cases involve Cloud load balancer and Ingress controller. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause5

Faulty deployment1 case

Component
Cloud load balancerIngress controllerPod termination
Contributing factor
Unverified assumptionUnreviewed defaultsMonitoring blind spot
Organisation
Yonghyun Kim
  1. IL-0188 Rolling updates on EKS returned 502 errors for about 30 seconds because Kubernetes deleted old pods as soon as new pods passed readiness, before the ALB had marked the new pod targets healthy
    Yonghyun Kim2025

Breaking change in an upgrade2 cases

Component
Cloud load balancer2Ingress controller1kube-proxy1Linux kernel1Node operating system image1
Contributing factor
Change with a delayed effect1Unpatched known bug1Monitoring blind spot1No staged rollout1Unrepresentative staging1
Organisation
Altinity1Honeycomb1
  1. IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress traffic
    Altinity2025
  2. IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replaced
    Honeycomb2024

Configuration change3 cases

Component
Cloud load balancer3Configuration templating1Ingress controller1Terraform1AWS VPC CNI1
Contributing factor
Unverified assumption2Excessive blast radius1Monitoring blind spot1No configuration tests1No safeguard on destructive change1No staged rollout1Missed in review1
Organisation
DFDS2Grafana Labs1
  1. IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud services
    Grafana Labs2025
  2. IL-0111 Switching the main Kubernetes ingress controller from Traefik v1 to v2 stopped X-Forwarded headers being passed, which broke the Okta login flow for every service using Okta and stopped DFDS online bookings for over an hour
    DFDS2021
  3. IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503
    DFDS2021

Software defect1 case

Component
Cloud load balancerIngress controller
Contributing factor
Excessive blast radiusChange with a delayed effectNo configuration tests
Organisation
DFDS
  1. IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404
    DFDS2021

Operator error1 case

Component
Cloud load balancerCredential broker for podsexternal-dnsIngress controllerPrometheus
Contributing factor
No safeguard on destructive change
Organisation
UK Ministry of Justice
  1. IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dns
    UK Ministry of Justice2020