Incident Library

Organisation

Slack

Slack · slack

4 cases · 2020 to 2022

Root cause Software defect2Capacity exhaustion1Cloud provider outage1
Component Service discovery2In-memory cache1Automated capacity manager1Database1Authoritative DNS servers and records1HAProxy1Transit gateway or network interconnect1Recursive DNS resolver service1Terraform1
Contributing factor Automation amplification3Unverified assumption1Change during peak hours1Circular dependency1Inefficient query1Limited debug access1Harmful manual action1No alert on limit1Retry storm1Ineffective rollback1Unrepresentative staging1Dependence on a warm cache1
Symptom High error rate3Scale-up failure1Cloud API rate limit exhausted1Database overload1DNS resolution failures1File descriptor exhaustion1High latency1Monitoring outage1Packet loss1Service outage1Stale endpoint routing1Load imbalance1

Cases

  1. IL-0212 Many Slack users could not connect because a Vitess keyspace for channel membership was overloaded by cache-miss queriesSlack · 2022

    Root cause Capacity exhaustion

  2. IL-0250 slack.com stopped resolving on some DNSSEC-validating resolvers for about 24 hours after DNSSEC signing was turned onSlack · 2021

    Root cause Software defect

  3. IL-0220 An overloaded AWS Transit Gateway dropped packets on 4 January 2021, and Slack's web tier saturated and went downSlack · 2021

    Root cause Cloud provider outage

  4. IL-0251 Most of Slack's HAProxy load balancers routed to a stale, shrinking set of webapp backends and served HTTP 503 errorsSlack · 2020

    Root cause Software defect