Incident Library

Organisation

Skyscanner

Skyscanner · skyscanner

2 cases · 2019 to 2021

Root cause Configuration change1Software defect1
Component Configuration templating2Argo CD1HAProxy1Ingress controller1
Contributing factor Unverified assumption1Automation amplification1Untested backup1Excessive blast radius1Ignored early signal1No configuration tests1No staged rollout1Missed in review1Unrepresentative staging1
Symptom High latency1Intermittent connection failures1OOMKilled1Resources deleted1Service outage1Load imbalance1

Cases

  1. IL-0010 A templating change that dropped its {{ }} delimiters made Argo CD delete all 478 Skyscanner services in every regionSkyscanner · 2021

    Root cause Configuration change

  2. IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP orderSkyscanner · 2019

    Root cause Software defect