Incident Library

IL-0535 Amazon Web Services · 2024-07-30awsdatabaseobservability

An internal Kinesis Data Streams cell in US-EAST-1 degraded, raising errors in CloudWatch Logs, S3 events and Lambda

Many AWS services buffer data through Amazon Kinesis Data Streams; CloudWatch Logs, for example, buffers incoming log streams in it before writing them to its own storage. Kinesis Data Streams is split into cells, each made of independent subsystems spread over several Availability Zones, and every region has cells used only by AWS services and separate cells for customers. Over the previous couple of years AWS had been moving cells to a new architecture with a cell management system that monitors host health and spreads Kinesis shards across healthy hosts. The system had been tested extensively and had run in many cells for months, and internal cells in US-EAST-1 had been upgraded without issue. One of these internal cells had a workload unlike the others: a very large number of very low-throughput shards.

At 9:09 AM PDT on July 30th, 2024 a routine deployment began in one Availability Zone of that cell. The deployment takes each host out of service, deploys, runs health checks and brings it back, and the cell management system moves work off each host being removed. It balances work by throughput and other I/O dimensions, so it did not spread the many low-throughput shards across all healthy hosts, and a few hosts received a very large number of them. The status messages these hosts send to the cell management system list every shard the host processes, so the messages became very large, could not be sent and processed in time, and delayed the status messages of other hosts. The cell management system then wrongly marked healthy hosts as unhealthy and moved their shards elsewhere. The spike of shard redistribution overloaded another component that provisions secure connections to the Kinesis data plane subsystems, and from 2:45 PM PDT requests to the cell had elevated latency and errors. Other Kinesis cells in US-EAST-1 and other regions were unaffected.

At 4:25 PM PDT engineers deployed a change that shed load by reducing incoming stream volume from less time-sensitive internal workloads, and error rates and latency began to improve at 5:39 PM PDT. Engineers added capacity to the subsystem that provisions secure connections; by 5:55 PM PDT the cell could establish secure connections again, most requests succeeded at 6:04 PM PDT, more capacity was added at 6:32 PM PDT, and by 7:21 PM PDT the vast majority of requests were processed normally. The cell recovered fully at 9:37 PM PDT, and the dependent services worked through their backlogs until July 31st and August 1st.

Severity
–
Scope
One internal Kinesis Data Streams cell, and through it CloudWatch Logs, Amazon Data Firehose, S3 event delivery, Amazon ECS, AWS Lambda, API Gateway logging, Amazon Redshift, Amazon MWAA and AWS Glue
Users affected
–
Region
Northern Virginia (US-EAST-1)
Time to detect
–
Time to resolve
6h 52m

Symptoms

From 2:45 PM PDT on July 30th, 2024, several AWS services in the Northern Virginia (US-EAST-1) Region had higher latency and error rates: CloudWatch Logs, Amazon Data Firehose, the event framework that publishes Amazon S3 events, Amazon ECS, AWS Lambda, Amazon Redshift and AWS Glue. All of them used one internal cell of Amazon Kinesis Data Streams, whose requests had started to see elevated latency and errors.

Each service showed it differently. Amazon Data Firehose had more failures on PutRecord and PutRecordBatch calls and was slower to deliver and to create or delete delivery streams, without losing data. S3 event delivery was delayed until 7:21 PM PDT. ECS tasks that used the awslogs log driver in its default blocking mode were blocked when logs could not be sent to CloudWatch Logs and then could not answer health checks. Lambda function executions were missing their CloudWatch Logs, so customers could not tell whether their functions ran successfully, and functions that called CloudWatch Logs or Firehose directly had more errors or latency. API Gateway logging failed. Redshift connections through Query Editor v2 and JDBC, ODBC or Python drivers failed intermittently, and MWAA and Glue had slow and failing API calls.

Recovery was gradual. Error rates first improved at 5:39 PM PDT, most requests to the cell succeeded by 7:21 PM PDT and the cell was normal at 9:37 PM PDT. Backlogs took much longer: CloudWatch Logs processed the newest log streams first and finished the older backlog at 5:50 AM PDT on July 31st, when logs, metrics filtered from log events and CloudWatch alarms on delayed metrics were normal again, and the backlog of S3 events finished at 2:38 AM PDT on August 1st.

Symptom High latencyHigh error rateMonitoring outageProbe failures

Root cause

Root cause Software defect

Component Message brokerServerless function platformObject storage
Contributing factor Unrepresentative stagingAutomation amplification
Trigger
At 9:09 AM PDT on July 30th a routine deployment started in one Availability Zone of the Kinesis cell, and taking hosts in and out of service during the deployment set off the failure.

Detection

–

Remediation

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Message broker, Serverless function platform, Object storage among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.