Incident Library
Applied filters ComponentCloud load balancerFactorUnverified assumptionOrganisationGrafana LabsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud services
    Grafana Labs2025
    Symptoms

    At 08:41 UTC on 18 February 2025, several Grafana Labs application teams noticed that some of their endpoints were no longer available. The investigation quickly found that the load balancers for those endpoints were gone.

    Service outageResources deleted
    Trigger

    A change to the merge policy of a Crossplane composition, made so that the default TLS policy of the load balancers could be overridden, was applied to development, staging and production at the same time.

    Root cause
    Configuration change
    Component
    Configuration templatingCloud load balancer
    Contributing factor
    Unverified assumptionNo staged rolloutExcessive blast radiusNo safeguard on destructive change
    Detection

    Application teams noticed at 08:41 UTC that some of their endpoints were no longer available.

    Remediation
    • rolled back the Crossplane composition change
    • recreated the affected Kubernetes Services so that Kubernetes recreated their load balancers
    Action items
    • Add a CI check that blocks a deployment to more than one wave, so any exception to the dev, staging, canary, production order is caught
    • Give higher priority to a more sophisticated wave design that was already under discussion
    • Enable deletion protection on the load balancers, one layer below Crossplane
    • Enable deletion protection on other critical cloud provider resources
    • Develop the Crossplane Managed Resources exporter for change validation and visibility, now in beta testing within Platform engineering; simulated against the change that caused this outage, it shows a clear diff
    • Complete and test support for reconfiguring the TLS policy so application teams can deploy it
    Lessons

    A change that looked small and low-risk had a side effect that only a full end-to-end test through Crossplane and the cloud provider's load balancer controller would have shown; seeing the intended override work was not that test. Staged rollout and blast-radius limits that exist only as process can be skipped, so Grafana Labs moved to enforce waves in CI and to add deletion protection on the load balancers themselves, beneath the Crossplane layer.

    View case
Check your cluster for this

This case involves Cloud load balancer and Configuration templating. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change1 case

Component
Cloud load balancerConfiguration templating
Contributing factor
Unverified assumptionExcessive blast radiusNo safeguard on destructive changeNo staged rollout
Organisation
Grafana Labs
  1. IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud services
    Grafana Labs2025