Cases2 cases

1 to 2 of 2

1. IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud services

   Grafana Labs2025

   Symptoms

   At 08:41 UTC on 18 February 2025, several Grafana Labs application teams noticed that some of their endpoints were no longer available. The investigation quickly found that the load balancers for those endpoints were gone.

   Service outageResources deleted

   Trigger

   A change to the merge policy of a Crossplane composition, made so that the default TLS policy of the load balancers could be overridden, was applied to development, staging and production at the same time.

   Root cause

   Configuration change

   Component

   Configuration templatingCloud load balancer

   Contributing factor

   Unverified assumptionNo staged rolloutExcessive blast radiusNo safeguard on destructive change

   Detection

   Application teams noticed at 08:41 UTC that some of their endpoints were no longer available.

   Remediation

   • rolled back the Crossplane composition change
   • recreated the affected Kubernetes Services so that Kubernetes recreated their load balancers

   Action items

   • Add a CI check that blocks a deployment to more than one wave, so any exception to the dev, staging, canary, production order is caught
   • Give higher priority to a more sophisticated wave design that was already under discussion
   • Enable deletion protection on the load balancers, one layer below Crossplane
   • Enable deletion protection on other critical cloud provider resources
   • Develop the Crossplane Managed Resources exporter for change validation and visibility, now in beta testing within Platform engineering; simulated against the change that caused this outage, it shows a clear diff
   • Complete and test support for reconfiguring the TLS policy so application teams can deploy it

   Lessons

   A change that looked small and low-risk had a side effect that only a full end-to-end test through Crossplane and the cloud provider's load balancer controller would have shown; seeing the intended override work was not that test. Staged rollout and blast-radius limits that exist only as process can be skipped, so Grafana Labs moved to enforce waves in CI and to add deletion protection on the load balancers themselves, beneath the Crossplane layer.

   View case
2. IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP order

   Skyscanner2019

   Symptoms

   On a Sunday, a team using one of Skyscanner's Kubernetes clusters manually paged the out-of-hours Kubernetes support because their service showed elevated latency, worst at the p90 and maximum levels.

   High latencyLoad imbalanceOOMKilledIntermittent connection failures

   Trigger

   Upgrading the HAProxy ingress controller to v0.6 and going back from Service virtual IPs to individual pod IPs as backends, on the understanding that the sort-backends flag introduced in v0.5, false by default, meant backends were randomly shuffled.

   Root cause

   Software defect

   Component

   Ingress controllerHAProxyConfiguration templating

   Contributing factor

   Unrepresentative stagingUnverified assumptionIgnored early signal

   Detection

   A cluster user manually paged the out-of-hours Kubernetes support about elevated latency; Prometheus queries then showed the uneven request distribution across pods.

   Remediation

   • switched the affected service from pod IPs back to the Service virtual IP through a config change in the internal deployment tool Slingshot
   • patched the controller to keep each backend's servers in an insertion-ordered slice and changed the template to iterate over that slice
   • tested the patch in sandbox clusters and rolled it out to the production clusters

   Action items

   • Run the internal controller patch in production and return to pod IPs as the default backends
   • Raise the problem upstream with a GitHub issue and pull request
   • Test changes to cluster components against the specific behaviour and conditions that matter

   Lessons

   The team already knew that sorted backends cause hot-podding and still did not test the new flag well enough to see that it did not work; changes to cluster components need tests aimed at the exact behaviour that matters, under conditions like production. A keyword can behave differently in different parts of a language: range over a map is unordered in Go code but sorted in text/template. A small cluster component can do outsized harm to every user of the cluster and, in the end, to customers.

   View case

By root cause2