Cases2 cases

1 to 2 of 2

1. IL-0010 A templating change that dropped its {{ }} delimiters made Argo CD delete all 478 Skyscanner services in every region

   Skyscanner2021

   Symptoms

   On 25 August 2021, skyscanner.net was unavailable and the Skyscanner mobile apps could not work, for travellers and business partners everywhere at the same time. The full global outage lasted four and a half hours.

   Service outageResources deleted

   Trigger

   A peer-reviewed change to a rarely changed root-level file of the cells templating configuration, intended as a no-op, that left out the {{ }} delimiters; it was submitted around 15:30 GMT, merged to master and rolled out automatically to every region at once.

   Root cause

   Configuration change

   Component

   Argo CDConfiguration templating

   Contributing factor

   Missed in reviewNo staged rolloutNo configuration testsUntested backupAutomation amplificationExcessive blast radius

   Remediation

   • restored the correct configuration for each cluster and let Argo CD reconcile the cluster back to it
   • restored services through the backup and restoration process
   • brought one region back first with the critical services and served all traffic from it from around 20:30 GMT
   • restored and verified the remaining regions and non-critical services over Thursday and Friday

   Action items

   • Avoid global configuration deploys, including rare change paths like this one
   • Treat templated configuration as code and add testing and linting
   • Plan and wargame more drastic failure scenarios in runbooks
   • Verify backup and restore processes, including access to the backups and the loss of a region
   • Refactor runbooks so they are clear and, where appropriate, idempotent for a stressed engineer in the early morning
   • Reconsider how much configuration is templated across regions, weighing configuration drift against automated global rollout

   Lessons

   Global configuration changes had been prevented in many cases, but not in this rarely used path. Configuration that gains templating and logic has become code and needs tests and at least linting. The failure scenarios and runbooks were not pessimistic enough, a backup only counts once it has been restored, and runbooks need care so they work for a stressed engineer at night. Automation that templates configuration across regions trades drift for the risk of rolling one mistake out everywhere. An experienced incident commander who kept the response calm and blameless made a large difference.

   View case
2. IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP order

   Skyscanner2019

   Symptoms

   On a Sunday, a team using one of Skyscanner's Kubernetes clusters manually paged the out-of-hours Kubernetes support because their service showed elevated latency, worst at the p90 and maximum levels.

   High latencyLoad imbalanceOOMKilledIntermittent connection failures

   Trigger

   Upgrading the HAProxy ingress controller to v0.6 and going back from Service virtual IPs to individual pod IPs as backends, on the understanding that the sort-backends flag introduced in v0.5, false by default, meant backends were randomly shuffled.

   Root cause

   Software defect

   Component

   Ingress controllerHAProxyConfiguration templating

   Contributing factor

   Unrepresentative stagingUnverified assumptionIgnored early signal

   Detection

   A cluster user manually paged the out-of-hours Kubernetes support about elevated latency; Prometheus queries then showed the uneven request distribution across pods.

   Remediation

   • switched the affected service from pod IPs back to the Service virtual IP through a config change in the internal deployment tool Slingshot
   • patched the controller to keep each backend's servers in an insertion-ordered slice and changed the template to iterate over that slice
   • tested the patch in sandbox clusters and rolled it out to the production clusters

   Action items

   • Run the internal controller patch in production and return to pod IPs as the default backends
   • Raise the problem upstream with a GitHub issue and pull request
   • Test changes to cluster components against the specific behaviour and conditions that matter

   Lessons

   The team already knew that sorted backends cause hot-podding and still did not test the new flag well enough to see that it did not work; changes to cluster components need tests aimed at the exact behaviour that matters, under conditions like production. A keyword can behave differently in different parts of a language: range over a map is unordered in Go code but sorted in text/template. A small cluster component can do outsized harm to every user of the cluster and, in the end, to customers.

   View case

By root cause2