Incident Library
Applied filters Root causeCloud provider outageSymptomMonitoring outageClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases4 cases

1 to 4 of 4

  1. IL-0220 An overloaded AWS Transit Gateway dropped packets on 4 January 2021, and Slack's web tier saturated and went down
    Slack2021
    Symptoms

    During the Americas' morning of 4 January 2021, an external monitoring service paged Slack because error rates were creeping up. While the investigation was starting, Slack's own dashboarding and alerting service became unavailable, so responders had to work without their usual dashboards and alerts.

    High error rateMonitoring outagePacket lossService outageScale-up failureFile descriptor exhaustionCloud API rate limit exhausted
    Trigger

    On the first working day after the holidays, Slack's traffic went from the quietest time of the year to one of its biggest days overnight, with cold client caches pulling more data than usual, and one of Slack's AWS Transit Gateways did not scale fast enough for it.

    Root cause
    Cloud provider outage
    Component
    Transit gateway or network interconnectAutomated capacity manager
    Contributing factor
    Circular dependencyAutomation amplification
    Detection

    An external monitoring service paged Slack when error rates started creeping up during the Americas' morning. AWS engineers were separately alerted to the packet drops by AWS's internal monitoring.

    Remediation
    • Disabled automated downscaling of the web tier to keep serving capacity and allow investigation
    • Cleared up the broken, partly provisioned instances that filled the autoscaling groups
    • Repaired provision-service so new web tier instances could be provisioned and enter service
    • AWS engineers increased the Transit Gateway capacity manually across all Availability Zones
    Action items
    • AWS reviews its Transit Gateway scaling algorithms for large packet-per-second increases
    • Request a preemptive upscaling of the Transit Gateways at the end of the next holiday season
    • Run the dashboard service instances in the same VPC as their database to remove the Transit Gateway dependency
    • Load test provision-service regularly
    • Reevaluate health-checking and autoscaling configurations so they cannot overload provision-service again
    Lessons

    Monitoring has to stay as independent as possible of the infrastructure it watches, or it fails when it is needed most. Here the dashboard and alerting services failed because they ran in a different VPC from their databases and so depended on the Transit Gateways. provision-service had been load tested before, but this event exceeded the earlier scale-ups.

    View case
  2. IL-0505 kubectl logs failed on GKE with "No SSH tunnels currently open" and mybinder.org build logs stopped reaching users
    Binder Project2019
    Symptoms

    On 20 February 2019 at 08:03 CET, `kubectl logs -f <pod>` stopped working for every pod in the mybinder.org cluster on GKE. Because pod logs could not be read, build logs were no longer forwarded to users, although the builds themselves ran normally.

    Monitoring outageDeploys failing
    Root cause
    Cloud provider outage
    Component
    GKE control planeKubernetes API server
    Contributing factor
    Limited debug access
    Remediation
    • Cordoned node gke-prod-a-users-b921bb88-rd5d so it would be removed from the pool once empty
    • Drained the node and removed it from the pool, after which kubectl logs worked again
    Action items
    • Test that kubectl logs works and notify when it does not
    • Add the command for retrieving API server logs to the SRE guide
    • Describe in the SRE guide how to identify suspicious node networking status
    Lessons

    The root cause was very difficult to identify, partly because it appears to sit in the Kubernetes master or the GKE service itself. Capturing `kubectl describe node` for the suspicious node would have allowed a comparison with healthy nodes, and a node that reports no ExternalIP or InternalIP in its status was likely a direct symptom of the problem and can be used as a diagnostic in the future.

    View case
  3. IL-0588 AppOptics metrics APIs returned 5xx as Kafka partitions stayed leaderless after an AWS power event hit their ZooKeeper
    AppOptics2018
    Symptoms

    On May 31, 2018 from 21:53 UTC the metrics APIs of AppOptics and Librato returned a 5xx for a majority of incoming metric data. APM metrics and traces sent from AppOptics APM were affected, and many alert notifications were delayed.

    High error rateFailover failureMonitoring outage
    Trigger

    A power event in one of AWS's availability zones in us-east-1 cut off nodes in several critical parts of the AppOptics and Librato infrastructure, including the ZooKeeper node that was then the leader of the ensemble coordinating the Kafka brokers.

    Root cause
    Cloud provider outage
    Component
    Message brokerCoordination and key-value store
    Contributing factor
    Single point of failureExcessive blast radiusMonitoring blind spot
    Remediation
    • Restarted one ZooKeeper node by hand to force a leader election, after which the ensemble regained quorum
    • Restarted every Kafka broker in a rolling restart until all partitions were back online
    • Worked through the backlog, including the backlog of alerts, and resumed normal operations
    Action items
    • Isolate the Kafka clusters better across availability zones and regions
    • Stop sharing one ZooKeeper ensemble across Kafka clusters
    • Run game days on this failure and check whether newer Kafka versions fix it
    • Put in place more checks that notify the team earlier about subsystems that need extra help to recover
    Lessons

    Running services across several availability zones and more than one Kafka cluster did not isolate the pipeline from the failure of one zone, because partitions of different clusters sat in the same zones and the clusters shared one ZooKeeper ensemble. More operational monitoring of the subsystems that needed help to recover would have led to a faster recovery.

    View case
  4. IL-0399 Applications on GOV.UK PaaS in AWS eu-west-1 failed DNS lookups intermittently for about two hours overnight
    Government Digital Service2016
    Symptoms

    On 15 December 2016 from 1.17am, VMs of the GOV.UK Platform as a Service (PaaS) in AWS eu-west-1 logged DNS resolution errors, and applications deployed on the PaaS failed DNS queries intermittently. Applications could fail to connect to their databases, backend services and APIs, and some end users of those applications would have seen intermittent slowness and errors between 1am and 3.17am.

    DNS resolution failuresIntermittent connection failuresMonitoring outage
    Root cause
    Cloud provider outage
    Component
    VPC DNS resolver
    Contributing factor
    Signal buried in noise
    Detection

    Datadog and Pingdom alerts fired during the night, but no one was on out-of-hours support. A PaaS team member noticed the warnings at 8am when starting work and checked monitoring to confirm the issue was over.

    Action items
    • Stop the monitoring system from sending emails when disk monitoring data is not being received
    • Make notifying tenants about outages easier and quicker
    • Implement alert notifications for out-of-hours support, shaped by this incident
    • Provide 24/7 support for the PaaS in line with the needs of adopting services
    View case
Check your cluster for this

These 4 cases involve Automated capacity manager and Coordination and key-value store. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Cloud provider outage4 cases

Component
Automated capacity manager1Coordination and key-value store1GKE control plane1Transit gateway or network interconnect1Kubernetes API server1Message broker1VPC DNS resolver1
Contributing factor
Signal buried in noise1Automation amplification1Excessive blast radius1Circular dependency1Limited debug access1Monitoring blind spot1Single point of failure1
Organisation
AppOptics1Binder Project1Government Digital Service1Slack1
  1. IL-0220 An overloaded AWS Transit Gateway dropped packets on 4 January 2021, and Slack's web tier saturated and went down
    Slack2021
  2. IL-0505 kubectl logs failed on GKE with "No SSH tunnels currently open" and mybinder.org build logs stopped reaching users
    Binder Project2019
  3. IL-0588 AppOptics metrics APIs returned 5xx as Kafka partitions stayed leaderless after an AWS power event hit their ZooKeeper
    AppOptics2018
  4. IL-0399 Applications on GOV.UK PaaS in AWS eu-west-1 failed DNS lookups intermittently for about two hours overnight
    Government Digital Service2016