Incident Library
Applied filters ComponentTransit gateway or network interconnectSymptomCloud API rate limit exhaustedFactorAutomation amplificationClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0220 An overloaded AWS Transit Gateway dropped packets on 4 January 2021, and Slack's web tier saturated and went down
    Slack2021
    Symptoms

    During the Americas' morning of 4 January 2021, an external monitoring service paged Slack because error rates were creeping up. While the investigation was starting, Slack's own dashboarding and alerting service became unavailable, so responders had to work without their usual dashboards and alerts.

    High error rateMonitoring outagePacket lossService outageScale-up failureFile descriptor exhaustionCloud API rate limit exhausted
    Trigger

    On the first working day after the holidays, Slack's traffic went from the quietest time of the year to one of its biggest days overnight, with cold client caches pulling more data than usual, and one of Slack's AWS Transit Gateways did not scale fast enough for it.

    Root cause
    Cloud provider outage
    Component
    Transit gateway or network interconnectAutomated capacity manager
    Contributing factor
    Circular dependencyAutomation amplification
    Detection

    An external monitoring service paged Slack when error rates started creeping up during the Americas' morning. AWS engineers were separately alerted to the packet drops by AWS's internal monitoring.

    Remediation
    • Disabled automated downscaling of the web tier to keep serving capacity and allow investigation
    • Cleared up the broken, partly provisioned instances that filled the autoscaling groups
    • Repaired provision-service so new web tier instances could be provisioned and enter service
    • AWS engineers increased the Transit Gateway capacity manually across all Availability Zones
    Action items
    • AWS reviews its Transit Gateway scaling algorithms for large packet-per-second increases
    • Request a preemptive upscaling of the Transit Gateways at the end of the next holiday season
    • Run the dashboard service instances in the same VPC as their database to remove the Transit Gateway dependency
    • Load test provision-service regularly
    • Reevaluate health-checking and autoscaling configurations so they cannot overload provision-service again
    Lessons

    Monitoring has to stay as independent as possible of the infrastructure it watches, or it fails when it is needed most. Here the dashboard and alerting services failed because they ran in a different VPC from their databases and so depended on the Transit Gateways. provision-service had been load tested before, but this event exceeded the earlier scale-ups.

    View case
Check your cluster for this

This case involves Automated capacity manager and Transit gateway or network interconnect. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Cloud provider outage1 case

Component
Automated capacity managerTransit gateway or network interconnect
Contributing factor
Automation amplificationCircular dependency
Organisation
Slack
  1. IL-0220 An overloaded AWS Transit Gateway dropped packets on 4 January 2021, and Slack's web tier saturated and went down
    Slack2021