Incident Library
Applied filters Root causeCloud provider outageComponentTransit gateway or network interconnectSymptomHigh error rateClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0220 An overloaded AWS Transit Gateway dropped packets on 4 January 2021, and Slack's web tier saturated and went down
    Slack2021
    Symptoms

    During the Americas' morning of 4 January 2021, an external monitoring service paged Slack because error rates were creeping up. While the investigation was starting, Slack's own dashboarding and alerting service became unavailable, so responders had to work without their usual dashboards and alerts.

    High error rateMonitoring outagePacket lossService outageScale-up failureFile descriptor exhaustionCloud API rate limit exhausted
    Trigger

    On the first working day after the holidays, Slack's traffic went from the quietest time of the year to one of its biggest days overnight, with cold client caches pulling more data than usual, and one of Slack's AWS Transit Gateways did not scale fast enough for it.

    Root cause
    Cloud provider outage
    Component
    Transit gateway or network interconnectAutomated capacity manager
    Contributing factor
    Circular dependencyAutomation amplification
    Detection

    An external monitoring service paged Slack when error rates started creeping up during the Americas' morning. AWS engineers were separately alerted to the packet drops by AWS's internal monitoring.

    Remediation
    • Disabled automated downscaling of the web tier to keep serving capacity and allow investigation
    • Cleared up the broken, partly provisioned instances that filled the autoscaling groups
    • Repaired provision-service so new web tier instances could be provisioned and enter service
    • AWS engineers increased the Transit Gateway capacity manually across all Availability Zones
    Action items
    • AWS reviews its Transit Gateway scaling algorithms for large packet-per-second increases
    • Request a preemptive upscaling of the Transit Gateways at the end of the next holiday season
    • Run the dashboard service instances in the same VPC as their database to remove the Transit Gateway dependency
    • Load test provision-service regularly
    • Reevaluate health-checking and autoscaling configurations so they cannot overload provision-service again
    Lessons

    Monitoring has to stay as independent as possible of the infrastructure it watches, or it fails when it is needed most. Here the dashboard and alerting services failed because they ran in a different VPC from their databases and so depended on the Transit Gateways. provision-service had been load tested before, but this event exceeded the earlier scale-ups.

    View case
  2. IL-0362 PagerDuty could not dispatch notifications for 18 minutes when a peering point shared by two AWS regions degraded
    PagerDuty2013sev1
    Symptoms

    On Saturday 13 April 2013 at 8:11am Pacific Time, PagerDuty's on-call engineer was paged about one node of the notification dispatch system, and an attempt at 8:13 to bring the node back failed. Customers saw delayed notifications and 500 errors on PagerDuty's API endpoints.

    High latencyHigh error rateIntermittent connection failures
    Trigger

    At 7:57am Pacific Time, according to AWS, a peering point in Northern California began to degrade, causing connectivity problems for the two AWS regions that PagerDuty used as two of its three datacenters.

    Root cause
    Cloud provider outage
    Component
    Transit gateway or network interconnectCoordination and key-value store
    Contributing factor
    Single point of failureUnverified assumptionMonitoring blind spot
    Detection

    The PagerDuty on-call engineer was paged at 8:11am about an issue with one node of the notification dispatch system, and at 8:18 monitoring detected multiple-provider failures for notifications.

    Remediation
    • Restarted most of the failed coordinator processes by hand
    • The notification dispatch system reached quorum again and processed all queued notifications
    Action items
    • Add more logging and aggregate logs into a single searchable source
    • Add a process watcher that restarts failed coordinator processes automatically
    • Build a dashboard that shows inter-host connectivity
    • Train engineering staff on Cassandra and ZooKeeper
    • Investigate moving off one of the AWS regions to a new hosting provider and datacenter that avoids a single point of failure
    Lessons

    Datacenters chosen to be independent and physically separate can still share a network peering point, and a design built to survive the loss of one datacenter did not survive losing two at once.

    View case
Check your cluster for this

These 2 cases involve Transit gateway or network interconnect and Automated capacity manager. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Cloud provider outage2 cases

Component
Transit gateway or network interconnect2Automated capacity manager1Coordination and key-value store1
Contributing factor
Unverified assumption1Automation amplification1Circular dependency1Monitoring blind spot1Single point of failure1
Organisation
PagerDuty1Slack1
  1. IL-0220 An overloaded AWS Transit Gateway dropped packets on 4 January 2021, and Slack's web tier saturated and went down
    Slack2021
  2. IL-0362 PagerDuty could not dispatch notifications for 18 minutes when a peering point shared by two AWS regions degraded
    PagerDuty2013