IL-0010 Skyscanner · 2021-08-25kubernetesaws
Skyscanner runs its services in cells. A cell sits in one region and consists of several Kubernetes clusters in each availability zone, traffic is prioritised inter-AZ, then inter-region, before going cross-region in a failure mode, and services run in an n+2 configuration so that 100% of traffic can be served with one cluster down and one drained. All of it runs on AWS spot instances. Clusters are kept in shape by GitOps: they reconcile themselves to the configuration in Git through Argo CD rather than a central system pushing changes.
At around 15:30 GMT an engineer submitted a change to the templating of this configuration. It was peer reviewed, merged and rolled out automatically. The edited file sat at the root of the cells configuration and rolls out regional configuration, so it went everywhere at once. Without the {{ }} delimiters the templating did not run, and every namespace that used the file was reapplied and corrupted. At 16:00 GMT Argo CD reconciled the clusters to this configuration and, finding no valid namespaces, deleted all 478 services worldwide.
GitOps also made the recovery possible: once the correct configuration was back in place for a cluster, the cluster reconfigured itself and came back up. The team knew the backup and restoration process well, although an IAM policy change had made the backups hard to obtain at a critical moment. They restored a single region first and gave priority to critical services, so that by around 20:30 GMT skyscanner.net was serving travellers again with all traffic coming from that one region. Engineers were then sent home to rest, and over Thursday and Friday the other regions and the non-critical services were brought back and verified.
On 25 August 2021, skyscanner.net was unavailable and the Skyscanner mobile apps could not work, for travellers and business partners everywhere at the same time. The full global outage lasted four and a half hours.
Underneath, from 16:00 GMT, Argo CD, the deployment tool that keeps each Kubernetes cluster in line with the configuration in Git, was deleting workloads. The configuration it had just received contained no valid namespaces, so it began deleting all 478 services in all namespaces, across every availability zone and every region. Skyscanner's cells architecture is designed to keep serving with one cluster failed and another drained for maintenance, and to shift traffic between availability zones and regions on failure, but the same deletion ran in every cluster.
The change behind it had been submitted about half an hour earlier, at around 15:30 GMT, then peer reviewed, merged and rolled out automatically. It was an edit to a root-level file of the cells templating configuration, meant as a no-op in preparation for a later change. The edit left out the {{ }} template delimiters, so the templating no longer applied and every namespace built from that file, which was all of them, was reapplied in a corrupted form. The file was rarely changed, but because it rolls out regional configuration, the system pushed it to every region immediately.
Symptom Service outageResources deleted
Root cause Configuration change
–
Global configuration changes had been prevented in many cases, but not in this rarely used path. Configuration that gains templating and logic has become code and needs tests and at least linting. The failure scenarios and runbooks were not pessimistic enough, a backup only counts once it has been restored, and runbooks need care so they work for a stressed engineer at night. Automation that templates configuration across regions trades drift for the risk of rolling one mistake out everywhere. An experienced incident commander who kept the response calm and blameless made a large difference.
IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environmentInfluxData · 2022same root cause · shared Configuration templating, Argo CD
IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud servicesGrafana Labs · 2025same root cause · shared Configuration templating
IL-0191 A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rolloutGrafana Labs · 2022same root cause · shared Configuration templating
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.