Incident Library

IL-0384 Amazon Web Services · 2020-11-25awsdatabaseobservability

Amazon Kinesis front-end servers in US-EAST-1 could not route requests, so Kinesis and services built on it failed

Kinesis Data Streams spreads streams over many back-end cell-clusters through sharding owned by a front-end fleet of servers, which handles authentication, throttling and routing of requests to the right shards. Each front-end server keeps a cache called the shard-map with membership and shard ownership of the back-end clusters, built from calls to a membership microservice, configuration in DynamoDB and messages from the other front-end servers. For those messages each front-end server creates an operating system thread for every other server in the fleet, and existing servers take up to an hour to learn about new ones. Between 2:44 and 3:47 AM PST on 25 November 2020 AWS added a relatively small amount of capacity to the front-end fleet. As the servers learned of the new members, every server in the fleet went over the maximum number of threads allowed by an operating system configuration, shard-map construction could not finish, and the front-end servers were left with useless shard-maps that could not route requests to the back-end clusters. The first alarms fired at 5:15 AM PST.

AWS removed the new capacity as a precaution and by 7:51 AM PST saw that every likely fix meant changing the configuration of every front-end server and restarting the whole fleet. That restart had to be slow: building the shard-map competes for resources with serving requests, so servers brought back too fast would be slow, could be judged unhealthy and removed, and would set recovery back. In parallel AWS added a configuration so that front-end servers load their data at bootstrap from the authoritative metadata store instead of from neighbouring front-end servers. At 9:39 AM PST AWS confirmed the thread limit as the cause. Rather than raise the operating system limit without more testing, AWS relied on the completed capacity removal to keep the thread count under the limit and went ahead with the restart. The first servers took traffic at 10:07 AM PST, the fleet of many thousands of servers could only be brought back a few hundred per hour, errors fell steadily from noon, and Kinesis was fully back to normal at 10:23 PM PST.

Services that use Kinesis failed with it. In Cognito, the long Kinesis outage set off a latent bug in local buffering that made the Cognito webservers block on backlogged Kinesis buffers; extra Cognito capacity helped at first, errors rose sharply by 7:01 AM PST, a change that reduced Cognito's dependency on Kinesis began deploying at 10:15 AM, and Cognito was normal by 2:18 PM PST. CloudWatch, which processes metrics and logs through Kinesis, could not process most metrics from 5:15 AM, began to recover at 5:47 PM and fully recovered by 10:31 PM PST, with backfilling over the following hours; reactive AutoScaling policies were delayed until 5:47 PM PST. Lambda invocations must publish metrics to CloudWatch, and the local buffers of the Lambda metric agents grew until they caused memory contention on the hosts that run invocations, raising errors from 6:15 AM until engineers mitigated it at 10:36 AM PST. CloudWatch Events and EventBridge had API errors and delayed events from 5:15 AM, which affected ECS and EKS cluster provisioning, scaling and task de-provisioning until most issues were resolved by 4:15 PM PST. The Service Health Dashboard posting tool depends on Cognito, so AWS used the slower backup tool, the Personal Health Dashboard and a global banner to inform customers.

Severity
–
Scope
Kinesis Data Streams in US-EAST-1 and AWS services that use it: Cognito User Pools and Identity Pools, CloudWatch metrics, logs and alarms, reactive AutoScaling, Lambda invocations, CloudWatch Events and EventBridge, and ECS and EKS cluster provisioning, scaling and task de-provisioning; Service Health Dashboard updates were delayed
Users affected
–
Region
US-EAST-1 (Northern Virginia)
Time to detect
–
Time to resolve
17h 8m

Symptoms

At 5:15 AM PST on 25 November 2020, alarms fired for errors on putting and getting Amazon Kinesis records in the Northern Virginia (US-EAST-1) Region. From the same time CloudWatch, which processes metric and log data through Kinesis Data Streams, and CloudWatch Events and EventBridge returned more errors.

Errors appeared in every kind of call made by both existing and new front-end servers, and many of them had nothing to do with the front-end capacity added that morning and would probably have stayed if it were removed. The variety of errors made side effects hard to tell apart from the cause and slowed diagnosis. The new capacity was removed anyway as a precaution. By 7:51 AM PST the team had narrowed the cause to a couple of candidates, the leading one an issue that seemed to create memory pressure, but the cause confirmed at 9:39 AM PST was not memory pressure.

CloudWatch had more errors and latency on the PutMetricData and PutLogEvents APIs, alarms went into the INSUFFICIENT_DATA state, and most metrics were not processed, which left gaps in CloudWatch metrics. Cognito User Pools and Identity Pools had more API failures and latency, so external users could not authenticate or get temporary AWS credentials. Lambda function invocations had more errors from 6:15 AM PST, reactive AutoScaling policies were delayed, and CloudWatch Events and EventBridge had API errors and delayed event processing, which affected cluster provisioning, scaling and task de-provisioning in ECS and EKS.

AWS could not update the Service Health Dashboard at first, because the tool used to post updates itself uses Cognito. The backup tool worked but is more manual and less familiar to support operators, so early updates were delayed.

Symptom High error rateHigh latencyMonitoring outage

Root cause

Root cause Capacity exhaustion

Component Message brokerServerless function platformTelemetry agent
Contributing factor Unknown limitExcessive blast radiusCircular dependency
Trigger
A relatively small addition of capacity to the Kinesis front-end fleet, made between 2:44 AM and 3:47 AM PST.

Detection

Alarms for errors on putting and getting Kinesis records fired at 5:15 AM PST, and the teams began reviewing logs.

Remediation

Action items

Lessons

In the Kinesis front-end, the number of threads each server keeps grows directly with the number of servers in the fleet, so adding servers can push every server past an operating system limit at the same time. AWS sees cellularization of the front-end, already done for the back-end, as the way to keep the shard-map cache within a tested and operated range and to guard against unknown scaling limits.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Message broker, Serverless function platform, Telemetry agent among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.