Cases3 cases

1 to 3 of 3

1. IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud services

   Grafana Labs2025

   Symptoms

   At 08:41 UTC on 18 February 2025, several Grafana Labs application teams noticed that some of their endpoints were no longer available. The investigation quickly found that the load balancers for those endpoints were gone.

   Service outageResources deleted

   Trigger

   A change to the merge policy of a Crossplane composition, made so that the default TLS policy of the load balancers could be overridden, was applied to development, staging and production at the same time.

   Root cause

   Configuration change

   Component

   Configuration templatingCloud load balancer

   Contributing factor

   Unverified assumptionNo staged rolloutExcessive blast radiusNo safeguard on destructive change

   Detection

   Application teams noticed at 08:41 UTC that some of their endpoints were no longer available.

   Remediation

   • rolled back the Crossplane composition change
   • recreated the affected Kubernetes Services so that Kubernetes recreated their load balancers

   Action items

   • Add a CI check that blocks a deployment to more than one wave, so any exception to the dev, staging, canary, production order is caught
   • Give higher priority to a more sophisticated wave design that was already under discussion
   • Enable deletion protection on the load balancers, one layer below Crossplane
   • Enable deletion protection on other critical cloud provider resources
   • Develop the Crossplane Managed Resources exporter for change validation and visibility, now in beta testing within Platform engineering; simulated against the change that caused this outage, it shows a clear diff
   • Complete and test support for reconfiguring the TLS policy so application teams can deploy it

   Lessons

   A change that looked small and low-risk had a side effect that only a full end-to-end test through Crossplane and the cloud provider's load balancer controller would have shown; seeing the intended override work was not that test. Staged rollout and blast-radius limits that exist only as process can be skipped, so Grafana Labs moved to enforce waves in CI and to add deletion protection on the load balancers themselves, beneath the Crossplane layer.

   View case
2. IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1

   DigitalOcean2024

   Symptoms

   At 13:02 UTC on 2024-01-23 DigitalOcean posted that, as of 12:18 UTC, its engineers were investigating an issue with Kubernetes clusters in the FRA1 region and that users might see errors when communicating with their clusters. The control planes of the affected DOKS (DigitalOcean Kubernetes) clusters had become inaccessible.

   Resources deletedControl plane unresponsive

   Trigger

   An incorrect configuration was applied to an internal infrastructure cluster in FRA1 at 12:21 UTC during a maintenance operation.

   Root cause

   Configuration change

   Component

   etcdKubernetes API server

   Contributing factor

   Automation amplificationNo safeguard on destructive changeExcessive blast radius

   Detection

   Engineers were investigating errors with Kubernetes clusters in FRA1 as of 12:18 UTC and identified the incorrect configuration as the cause at 12:36.

   Remediation

   • restored each affected cluster's etcd from its latest twice-daily backup
   • restored the stateless control plane components after etcd
   • patched the recovery mechanism after unexpected problems during recovery
   • found and fixed an additional small bug during recovery
   • contacted affected customers directly with further instructions, since some might need to reconfigure Kubernetes resources

   Action items

   • Prevent automatic deletion of infrastructure cluster resources under any circumstances in the production environment

   Lessons

   When managed control planes run as resources in another cluster, one configuration mistake in that cluster can delete hundreds of control planes at once through ordinary automation. The etcd backups limited the damage, but their twice-daily schedule set the worst case at 12 hours of lost control plane state, and the recovery tooling itself needed patching during the incident.

   View case
3. IL-0010 A templating change that dropped its {{ }} delimiters made Argo CD delete all 478 Skyscanner services in every region

   Skyscanner2021

   Symptoms

   On 25 August 2021, skyscanner.net was unavailable and the Skyscanner mobile apps could not work, for travellers and business partners everywhere at the same time. The full global outage lasted four and a half hours.

   Service outageResources deleted

   Trigger

   A peer-reviewed change to a rarely changed root-level file of the cells templating configuration, intended as a no-op, that left out the {{ }} delimiters; it was submitted around 15:30 GMT, merged to master and rolled out automatically to every region at once.

   Root cause

   Configuration change

   Component

   Argo CDConfiguration templating

   Contributing factor

   Missed in reviewNo staged rolloutNo configuration testsUntested backupAutomation amplificationExcessive blast radius

   Remediation

   • restored the correct configuration for each cluster and let Argo CD reconcile the cluster back to it
   • restored services through the backup and restoration process
   • brought one region back first with the critical services and served all traffic from it from around 20:30 GMT
   • restored and verified the remaining regions and non-critical services over Thursday and Friday

   Action items

   • Avoid global configuration deploys, including rare change paths like this one
   • Treat templated configuration as code and add testing and linting
   • Plan and wargame more drastic failure scenarios in runbooks
   • Verify backup and restore processes, including access to the backups and the loss of a region
   • Refactor runbooks so they are clear and, where appropriate, idempotent for a stressed engineer in the early morning
   • Reconsider how much configuration is templated across regions, weighing configuration drift against automated global rollout

   Lessons

   Global configuration changes had been prevented in many cases, but not in this rarely used path. Configuration that gains templating and logic has become code and needs tests and at least linting. The failure scenarios and runbooks were not pessimistic enough, a backup only counts once it has been restored, and runbooks need care so they work for a stressed engineer at night. Automation that templates configuration across regions trades drift for the risk of rolling one mistake out everywhere. An experienced incident commander who kept the response calm and blameless made a large difference.

   View case

By root cause1