Incident Library
Applied filters Root causeCapacity exhaustionSymptomMonitoring outageOrganisationAmazon Web ServicesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0384 Amazon Kinesis front-end servers in US-EAST-1 could not route requests, so Kinesis and services built on it failed
    Amazon Web Services2020
    Symptoms

    At 5:15 AM PST on 25 November 2020, alarms fired for errors on putting and getting Amazon Kinesis records in the Northern Virginia (US-EAST-1) Region. From the same time CloudWatch, which processes metric and log data through Kinesis Data Streams, and CloudWatch Events and EventBridge returned more errors.

    High error rateHigh latencyMonitoring outage
    Trigger

    A relatively small addition of capacity to the Kinesis front-end fleet, made between 2:44 AM and 3:47 AM PST.

    Root cause
    Capacity exhaustion
    Component
    Message brokerServerless function platformTelemetry agent
    Contributing factor
    Unknown limitExcessive blast radiusCircular dependency
    Detection

    Alarms for errors on putting and getting Kinesis records fired at 5:15 AM PST, and the teams began reviewing logs.

    Remediation
    • Removed the newly added front-end capacity
    • Added a configuration so front-end servers load bootstrap data from the authoritative metadata store instead of neighbouring servers
    • Restarted the front-end fleet gradually, a few hundred servers per hour from 10:07 AM PST
    • Added Cognito capacity to buffer more calls to Kinesis
    • Deployed a change that reduced Cognito's dependency on Kinesis
    • Mitigated the memory contention on the Lambda invocation hosts
    • Informed customers through the backup Service Health Dashboard tool, the Personal Health Dashboard and a global banner
    Action items
    • Move the Kinesis front-end to larger CPU and memory servers so fewer servers, and fewer threads per server, are needed
    • Add fine-grained alarming for thread consumption
    • Finish testing a higher operating system thread count limit
    • Greatly improve the cold-start time of the front-end fleet
    • Move the front-end server cache to a dedicated fleet
    • Move a few large AWS services such as CloudWatch to a separate, partitioned front-end fleet
    • Accelerate the cellularization of the front-end fleet to match the back-end
    • Change the Cognito webservers so they sustain Kinesis API errors without exhausting their buffers
    • Keep 3 hours of metric data in the CloudWatch local metrics data store, first in US-EAST-1 and then globally
    • Train support engineers regularly on the backup tool for posting to the Service Health Dashboard
    Lessons

    In the Kinesis front-end, the number of threads each server keeps grows directly with the number of servers in the fleet, so adding servers can push every server past an operating system limit at the same time. AWS sees cellularization of the front-end, already done for the back-end, as the way to keep the shard-map cache within a tested and operated range and to guard against unknown scaling limits.

    View case
Check your cluster for this

This case involves Message broker and Serverless function platform. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
Message brokerServerless function platformTelemetry agent
Contributing factor
Excessive blast radiusCircular dependencyUnknown limit
Organisation
Amazon Web Services
  1. IL-0384 Amazon Kinesis front-end servers in US-EAST-1 could not route requests, so Kinesis and services built on it failed
    Amazon Web Services2020