Incident Library

IL-0220 Slack · 2021-01-04awsnetworking

An overloaded AWS Transit Gateway dropped packets on 4 January 2021, and Slack's web tier saturated and went down

Slack runs its services in separate AWS accounts and dedicated VPCs and links the VPCs through AWS Transit Gateways (TGWs) used as hubs. TGWs are managed by AWS and are meant to scale without any action from the customer. On 4 January 2021 one of Slack's TGWs became overloaded and dropped packets. Slack's dashboard and alerting services ran in a different VPC from their backend databases, so they depended on the TGWs and failed with them. At the 7am PST mini-peak the network problems saturated the web tier and Slack became unavailable.

The web tier scales on CPU utilisation and on the use of Apache worker threads. Before 7:00am PST the threads spent more time waiting on the network, CPU utilisation dropped and automation downscaled the web tier; then thread use rose as the network got worse and automation tried to add 1,200 servers between 7:01am and 7:15am PST. provision-service talked to internal Slack systems and AWS APIs over the same degraded network, used more resources than usual, and under the load of so many simultaneous instances hit the Linux open files limit and an AWS quota limit. Most new instances were not fully provisioned, the broken instances reached the autoscaling-group size limits, and no new dashboard service instances could be provisioned either. Responders disabled the downscaling, cleared up the broken instances and repaired provision-service. Once provision-service was working again around 8:15am PST, healthy instances entered service, and by 9:15am PST Slack was degraded rather than down. The load balancers' panic mode, which spreads requests across all instances when many fail health checks, together with retries and circuit breaking, kept requests served. It still took an hour to provision enough instances to bring error rates down to a low level, because the unhealthy network meant more instances than usual were needed and each took longer to provision.

AWS engineers, alerted to the packet drops by AWS's own internal monitoring, increased the TGW capacity manually. By 10:40am PST that change had rolled out across all Availability Zones, and the network, error rates and latency returned to normal.

Severity
–
Scope
Slack messaging and its web tier: rising errors, then Slack unavailable, then degraded with higher latency and errors until the network recovered at 10:40am PST
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

During the Americas' morning of 4 January 2021, an external monitoring service paged Slack because error rates were creeping up. While the investigation was starting, Slack's own dashboarding and alerting service became unavailable, so responders had to work without their usual dashboards and alerts.

The infrastructure looked generally up, but there were signs of widespread network degradation, which Slack escalated to AWS. At 6.57am PST 99% of Slack messages were being sent successfully, against a usual success rate above 99.999%. Responders worked from internal consoles, status pages, command line tools, the logging infrastructure and direct queries to the metrics backends.

At the 7am PST mini-peak, when reminders and other automation send messages at the top of the hour, the network problems and the load saturated the web tier. Packet loss grew as load increased, calls from the web tier to its backends became much slower, and Slack became unavailable. Automation marked instances that could not reach their backends as unhealthy and tried to replace them, and the autoscaling system downscaled the web tier, so engineers investigating on those instances had their SSH sessions cut off as the instances were deprovisioned.

The web tier scales on CPU utilisation and on the use of Apache worker threads. Before 7:00am PST threads spent more time waiting on the network, CPU utilisation dropped and some automated downscaling followed; then thread use rose as the network got worse. Responders first rolled back some changes pushed out that day, which turned out not to be the cause. The automated scale-up of 1,200 servers between 7:01am and 7:15am PST did not add serving capacity: provision-service, which configures and tests new instances, hit the Linux open files limit and an AWS quota limit, most new instances were never fully provisioned, and the broken instances filled the autoscaling groups up to their size limits. No new dashboard service instances could be provisioned either, because provision-service was overloaded. The load balancing tier showed an extremely high rate of health check failures to the web application instances.

Symptom High error rateMonitoring outagePacket lossService outageScale-up failureFile descriptor exhaustionCloud API rate limit exhausted

Root cause

Root cause Cloud provider outage

Component Transit gateway or network interconnectAutomated capacity manager
Contributing factor Circular dependencyAutomation amplification
Trigger
On the first working day after the holidays, Slack's traffic went from the quietest time of the year to one of its biggest days overnight, with cold client caches pulling more data than usual, and one of Slack's AWS Transit Gateways did not scale fast enough for it.

Detection

An external monitoring service paged Slack when error rates started creeping up during the Americas' morning. AWS engineers were separately alerted to the packet drops by AWS's internal monitoring.

Remediation

Action items

Lessons

Monitoring has to stay as independent as possible of the infrastructure it watches, or it fails when it is needed most. Here the dashboard and alerting services failed because they ran in a different VPC from their databases and so depended on the Transit Gateways. provision-service had been load tested before, but this event exceeded the earlier scale-ups.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Transit gateway or network interconnect, Automated capacity manager among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.