Incident Library
Applied filters ComponentCloud load balancerFactorNo safeguard on destructive changeDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud services
    Grafana Labs2025
    Symptoms

    At 08:41 UTC on 18 February 2025, several Grafana Labs application teams noticed that some of their endpoints were no longer available. The investigation quickly found that the load balancers for those endpoints were gone.

    Service outageResources deleted
    Trigger

    A change to the merge policy of a Crossplane composition, made so that the default TLS policy of the load balancers could be overridden, was applied to development, staging and production at the same time.

    Root cause
    Configuration change
    Component
    Configuration templatingCloud load balancer
    Contributing factor
    Unverified assumptionNo staged rolloutExcessive blast radiusNo safeguard on destructive change
    Detection

    Application teams noticed at 08:41 UTC that some of their endpoints were no longer available.

    Remediation
    • rolled back the Crossplane composition change
    • recreated the affected Kubernetes Services so that Kubernetes recreated their load balancers
    Action items
    • Add a CI check that blocks a deployment to more than one wave, so any exception to the dev, staging, canary, production order is caught
    • Give higher priority to a more sophisticated wave design that was already under discussion
    • Enable deletion protection on the load balancers, one layer below Crossplane
    • Enable deletion protection on other critical cloud provider resources
    • Develop the Crossplane Managed Resources exporter for change validation and visibility, now in beta testing within Platform engineering; simulated against the change that caused this outage, it shows a clear diff
    • Complete and test support for reconfiguring the TLS policy so application teams can deploy it
    Lessons

    A change that looked small and low-risk had a side effect that only a full end-to-end test through Crossplane and the cloud provider's load balancer controller would have shown; seeing the intended override work was not that test. Staged rollout and blast-radius limits that exist only as process can be skipped, so Grafana Labs moved to enforce waves in CI and to add deletion protection on the load balancers themselves, beneath the Crossplane layer.

    View case
  2. IL-0473 Jenkins infrastructure went fully down after a pod IP range fix deleted every AKS node pool and could not recreate them
    Jenkins project2023
    Symptoms

    On 7 July 2023 from 11:05 to 15:25 UTC many public Jenkins infrastructure services were completely down, including get.jenkins.io, plugins.jenkins.io, accounts.jenkins.io, javadoc.jenkins.io and incrementals.jenkins.io, while ci.jenkins.io, issues.jenkins.io, repo.jenkins-ci.org and www.jenkins.io were partly or completely disrupted.

    Network unreachableResources deletedCluster or node provisioning stallService outage
    Trigger

    After the upgrade to Kubernetes 1.25 changed the range of the pods' originating IP and LDAP access was blocked, a fix (Azure PR 431) deployed at 10:55 UTC to set a proper set of pod IP ranges removed all node pools of the cluster.

    Root cause
    Configuration change
    Component
    Node poolCNI pluginCloud load balancer
    Contributing factor
    Unverified assumptionNo staged rolloutNo safeguard on destructive change
    Detection

    After the upgrade, at 10:30 UTC, the team noticed that services inside the cluster, such as accounts.jenkins.io, could not reach the LDAP service.

    Remediation
    • Created the cluster again from scratch with a fixed network setup
    • Set a custom pod CIDR in a range separate from the virtual network
    • Moved the services and their DNS records to new public IPs
    • Pushed a pull request that recorded the manual rebuild, including the IP setup, in code
    Action items
    • Protect the public IPs from deletion with a Management Lock
    • Keep the public IPs in a separate resource group and point the Kubernetes-managed load balancers to it with the service.beta.kubernetes.io/azure-load-balancer-resource-group annotation
    • Improve network diagrams and documentation so possible overlaps are visible when preparing operations
    • Avoid changing the configuration of all AKS node pools at once
    Lessons

    The team said that changing one node pool at a time would have shown the problem after the first pool and avoided a full outage, and that the public IPs should change as little as possible because users behind corporate firewalls with allow-lists depend on them. The team also noted that it could rebuild, in under 4 hours, the cluster that hosts almost all public services.

    View case
  3. IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dns
    UK Ministry of Justice2020
    Symptoms

    At 18:27 on 21 September 2020 the Cloud Platform team detected that some services running on the live-1 production cluster had gone down, and some users could not reach them. The incident was declared at 18:40.

    Service outageResources deletedMonitoring outageStale endpoint routing
    Trigger

    A script for deleting a test cluster was triggered while the kube context was pointing at the live-1 production cluster instead of the test cluster.

    Root cause
    Operator error
    Component
    Ingress controllerCredential broker for podsexternal-dnsCloud load balancerPrometheus
    Contributing factor
    No safeguard on destructive change
    Detection

    The first detection was at 18:27, and the incident was declared 13 minutes later, at 18:40.

    Remediation
    • restored the default ingress controller together with external-dns and kiam, which it depends on
    • corrected the owner field in the Formbuilder Route 53 TXT records so external-dns pointed their A records at the new NLB
    • restored the remaining components with a targeted apply
    • ran the apply pipeline to restore all certificates, ServiceMonitors and Prometheus rules from the environment repository
    View case
Check your cluster for this

These 3 cases involve Cloud load balancer and CNI plugin. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Configuration change2 cases

Component
Cloud load balancer2CNI plugin1Configuration templating1Node pool1
Contributing factor
Unverified assumption2No safeguard on destructive change2No staged rollout2Excessive blast radius1
Organisation
Grafana Labs1Jenkins project1
  1. IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud services
    Grafana Labs2025
  2. IL-0473 Jenkins infrastructure went fully down after a pod IP range fix deleted every AKS node pool and could not recreate them
    Jenkins project2023

Operator error1 case

Component
Cloud load balancerCredential broker for podsexternal-dnsIngress controllerPrometheus
Contributing factor
No safeguard on destructive change
Organisation
UK Ministry of Justice
  1. IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dns
    UK Ministry of Justice2020