IL-0192 Grafana Labs · 2025-02-18kubernetesnetworking
An engineer was upgrading the TLS policy version on the cloud load balancers used by many Grafana Cloud services, including metrics and logs ingestion. The engineer tested the new configuration by hand in the development environment and confirmed it could be applied without redeploying the load balancers. Grafana Labs manages many cloud provider resources from Kubernetes with Crossplane, so the code change was made in a Crossplane composition: its merge policy was changed so that the default TLS policy could be overridden. The engineer saw the override survive reconciliation and took that as enough of a test, so the full redeployment and test in development was skipped. The pull request was reviewed by a peer before it was merged.
The new merge policy also overrode a default value from the base configuration that was never meant to change. Many Kubernetes Services switched from type LoadBalancer to type ClusterIP, and Kubernetes destroyed their load balancers. About 25% of Grafana Cloud services were affected for about 150 minutes. Grafana Labs normally rolls changes out in waves, first development, then staging, then canary and production in steps, observing each wave, typically for at least 24 hours, with what is effectively a feature flag deciding which wave gets a change. This change had no such flag and was applied manually, and nothing stopped it from reaching development, staging and production at once. Deletion protection at the Crossplane level did not help, because the Crossplane resources were not deleted; they were changed so that the objects underneath became invalid.
The team rolled back the code change and recreated the affected Kubernetes Services so that Kubernetes created the load balancers again. Most applications were fully recovered by 11:10 UTC. It was not a security incident, and no customer data was leaked or exposed.
At 08:41 UTC on 18 February 2025, several Grafana Labs application teams noticed that some of their endpoints were no longer available. The investigation quickly found that the load balancers for those endpoints were gone.
The load balancers served many services, including the ingestion of metrics and logs. Customers lost access to Grafana Cloud services, and customers whose agents could not buffer for the length of the outage may have lost metrics or logs before they reached Grafana Cloud.
The missing load balancers traced back to a recent configuration change made while updating the TLS policies on the load balancers. The Crossplane resources still existed; what had changed were the Kubernetes Services underneath, many of which were now of type ClusterIP instead of LoadBalancer.
Symptom Service outageResources deleted
Root cause Configuration change
Application teams noticed at 08:41 UTC that some of their endpoints were no longer available.
A change that looked small and low-risk had a side effect that only a full end-to-end test through Crossplane and the cloud provider's load balancer controller would have shown; seeing the intended override work was not that test. Staged rollout and blast-radius limits that exist only as process can be skipped, so Grafana Labs moved to enforce waves in CI and to add deletion protection on the load balancers themselves, beneath the Crossplane layer.
IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on NginxLINE+ · 2022same root cause · shared Cloud load balancer
IL-0191 A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rolloutGrafana Labs · 2022same root cause · shared Configuration templating
IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environmentInfluxData · 2022same root cause · shared Configuration templating
IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errorsGitLab · 2022same root cause · shared Cloud load balancer
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Cloud load balancer
IL-0010 A templating change that dropped its {{ }} delimiters made Argo CD delete all 478 Skyscanner services in every regionSkyscanner · 2021same root cause · shared Configuration templating
IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503DFDS · 2021same root cause · shared Cloud load balancer
IL-0052 A cryptocurrency miner ran on every node of the development and staging clusters through a public, unauthenticated Weave Scope dashboardJW Player · 2019same root cause · shared Cloud load balancer