Cases7 cases

4 to 7 of 7

1. IL-0535 An internal Kinesis Data Streams cell in US-EAST-1 degraded, raising errors in CloudWatch Logs, S3 events and Lambda

   Amazon Web Services2024

   Symptoms

   From 2:45 PM PDT on July 30th, 2024, several AWS services in the Northern Virginia (US-EAST-1) Region had higher latency and error rates: CloudWatch Logs, Amazon Data Firehose, the event framework that publishes Amazon S3 events, Amazon ECS, AWS Lambda, Amazon Redshift and AWS Glue. All of them used one internal cell of Amazon Kinesis Data Streams, whose requests had started to see elevated latency and errors.

   High latencyHigh error rateMonitoring outageProbe failures

   Trigger

   At 9:09 AM PDT on July 30th a routine deployment started in one Availability Zone of the Kinesis cell, and taking hosts in and out of service during the deployment set off the failure.

   Root cause

   Software defect

   Component

   Message brokerServerless function platformObject storage

   Contributing factor

   Unrepresentative stagingAutomation amplification

   Remediation

   • Deployed a change that shed load by reducing incoming stream volume from less time-sensitive internal workloads
   • Added capacity to the subsystem that provisions secure connections for Kinesis requests
   • Added more capacity at 6:32 PM PDT to further improve connection provisioning

   Action items

   • Keep the added connection provisioning capacity, load-shedding tooling and changed connection limits in place in US-EAST-1
   • Apply the same changes in every other region that has cells with the new Kinesis Data Streams architecture
   • Update the cell management system to better manage the workload that triggered the issue

   View case
2. IL-0280 Kafka brokers in many New Relic cells ran out of disk, leaving US data collection and alerting unusable for hours

   New Relic2021

   Symptoms

   On 29 July 2021, data collection and alerting in New Relic's US region became unusable for many customers for several hours. It began with one unresponsive Kafka broker in a single cell of NRDB, New Relic's telemetry data platform, and over the following hours Kafka brokers in more and more cells failed as their disks filled up.

   Node disk fullThroughput degradationMonitoring outage

   Trigger

   Responders used an incident response tool to extend Kafka data retention for the degraded cell so that its backlog would not be lost. The retention increase was larger than usual, and the tool applied it to all cells of the same type, not only the affected one.

   Root cause

   Operator error

   Component

   Message broker

   Contributing factor

   Harmful manual actionNo staged rolloutSignal buried in noiseUnverified assumption

   Detection

   Engineers saw a Kafka broker in one cell become unresponsive on the morning of 29 July. Later a monitoring alert warned that disk space was running low, but it fired together with unrelated warnings from several other systems and was missed.

   Remediation

   • Restarted the unresponsive Kafka broker by hand, while the automated restart was also running
   • Resolved the disk space problem on the Kafka brokers, after which service recovered slowly

   Lessons

   Cell isolation is the strongest protection against a widespread disruption, and emergency tools have to respect it just as normal operations do. Emergency response tools must be safe to use in chaotic situations without losing their ability to work. Engineers' attention is limited, so incident response processes and alerts need regular review to direct it at the most important signal.

   View case
3. IL-0384 Amazon Kinesis front-end servers in US-EAST-1 could not route requests, so Kinesis and services built on it failed

   Amazon Web Services2020

   Symptoms

   At 5:15 AM PST on 25 November 2020, alarms fired for errors on putting and getting Amazon Kinesis records in the Northern Virginia (US-EAST-1) Region. From the same time CloudWatch, which processes metric and log data through Kinesis Data Streams, and CloudWatch Events and EventBridge returned more errors.

   High error rateHigh latencyMonitoring outage

   Trigger

   A relatively small addition of capacity to the Kinesis front-end fleet, made between 2:44 AM and 3:47 AM PST.

   Root cause

   Capacity exhaustion

   Component

   Message brokerServerless function platformTelemetry agent

   Contributing factor

   Unknown limitExcessive blast radiusCircular dependency

   Detection

   Alarms for errors on putting and getting Kinesis records fired at 5:15 AM PST, and the teams began reviewing logs.

   Remediation

   • Removed the newly added front-end capacity
   • Added a configuration so front-end servers load bootstrap data from the authoritative metadata store instead of neighbouring servers
   • Restarted the front-end fleet gradually, a few hundred servers per hour from 10:07 AM PST
   • Added Cognito capacity to buffer more calls to Kinesis
   • Deployed a change that reduced Cognito's dependency on Kinesis
   • Mitigated the memory contention on the Lambda invocation hosts
   • Informed customers through the backup Service Health Dashboard tool, the Personal Health Dashboard and a global banner

   Action items

   • Move the Kinesis front-end to larger CPU and memory servers so fewer servers, and fewer threads per server, are needed
   • Add fine-grained alarming for thread consumption
   • Finish testing a higher operating system thread count limit
   • Greatly improve the cold-start time of the front-end fleet
   • Move the front-end server cache to a dedicated fleet
   • Move a few large AWS services such as CloudWatch to a separate, partitioned front-end fleet
   • Accelerate the cellularization of the front-end fleet to match the back-end
   • Change the Cognito webservers so they sustain Kinesis API errors without exhausting their buffers
   • Keep 3 hours of metric data in the CloudWatch local metrics data store, first in US-EAST-1 and then globally
   • Train support engineers regularly on the backup tool for posting to the Service Health Dashboard

   Lessons

   In the Kinesis front-end, the number of threads each server keeps grows directly with the number of servers in the fleet, so adding servers can push every server past an operating system limit at the same time. AWS sees cellularization of the front-end, already done for the back-end, as the way to keep the shard-map cache within a tested and operated range and to guard against unknown scaling limits.

   View case
4. IL-0588 AppOptics metrics APIs returned 5xx as Kafka partitions stayed leaderless after an AWS power event hit their ZooKeeper

   AppOptics2018

   Symptoms

   On May 31, 2018 from 21:53 UTC the metrics APIs of AppOptics and Librato returned a 5xx for a majority of incoming metric data. APM metrics and traces sent from AppOptics APM were affected, and many alert notifications were delayed.

   High error rateFailover failureMonitoring outage

   Trigger

   A power event in one of AWS's availability zones in us-east-1 cut off nodes in several critical parts of the AppOptics and Librato infrastructure, including the ZooKeeper node that was then the leader of the ensemble coordinating the Kafka brokers.

   Root cause

   Cloud provider outage

   Component

   Message brokerCoordination and key-value store

   Contributing factor

   Single point of failureExcessive blast radiusMonitoring blind spot

   Remediation

   • Restarted one ZooKeeper node by hand to force a leader election, after which the ensemble regained quorum
   • Restarted every Kafka broker in a rolling restart until all partitions were back online
   • Worked through the backlog, including the backlog of alerts, and resumed normal operations

   Action items

   • Isolate the Kafka clusters better across availability zones and regions
   • Stop sharing one ZooKeeper ensemble across Kafka clusters
   • Run game days on this failure and check whether newer Kafka versions fix it
   • Put in place more checks that notify the team earlier about subsystems that need extra help to recover

   Lessons

   Running services across several availability zones and more than one Kafka cluster did not isolate the pipeline from the failure of one zone, because partitions of different clusters sat in the same zones and the clusters shared one ZooKeeper ensemble. More operational monitoring of the subsystems that needed help to recover would have led to a faster recovery.

   View case

By root cause5