Incident Library

Organisation

Airbnb

Airbnb · airbnb

3 cases · 2019 to 2020

Root cause Capacity exhaustion1Faulty deployment1Software defect1
Component Admission webhook1Client-side DNS resolver and cache1Horizontal pod autoscaler1kube-dns1kube-proxy1Service discovery1Telemetry agent1
Contributing factor Unverified assumption1Excessive blast radius1Unreviewed defaults1Change with a delayed effect1Ignored early signal1Monitoring blind spot1
Symptom Deploys failing1DNS resolution failures1High error rate1High latency1Intermittent timeouts1Probe failures1Service outage1Throughput degradation1Load imbalance1

Cases

  1. IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two podsAirbnb · 2020

    Root cause Software defect

  2. IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of trafficAirbnb · 2019

    Root cause Faulty deployment

  3. IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 secondsAirbnb · 2019

    Root cause Capacity exhaustion