Cases2 cases

1 to 2 of 2

1. IL-0384 Amazon Kinesis front-end servers in US-EAST-1 could not route requests, so Kinesis and services built on it failed

   Amazon Web Services2020

   Symptoms

   At 5:15 AM PST on 25 November 2020, alarms fired for errors on putting and getting Amazon Kinesis records in the Northern Virginia (US-EAST-1) Region. From the same time CloudWatch, which processes metric and log data through Kinesis Data Streams, and CloudWatch Events and EventBridge returned more errors.

   High error rateHigh latencyMonitoring outage

   Trigger

   A relatively small addition of capacity to the Kinesis front-end fleet, made between 2:44 AM and 3:47 AM PST.

   Root cause

   Capacity exhaustion

   Component

   Message brokerServerless function platformTelemetry agent

   Contributing factor

   Unknown limitExcessive blast radiusCircular dependency

   Detection

   Alarms for errors on putting and getting Kinesis records fired at 5:15 AM PST, and the teams began reviewing logs.

   Remediation

   • Removed the newly added front-end capacity
   • Added a configuration so front-end servers load bootstrap data from the authoritative metadata store instead of neighbouring servers
   • Restarted the front-end fleet gradually, a few hundred servers per hour from 10:07 AM PST
   • Added Cognito capacity to buffer more calls to Kinesis
   • Deployed a change that reduced Cognito's dependency on Kinesis
   • Mitigated the memory contention on the Lambda invocation hosts
   • Informed customers through the backup Service Health Dashboard tool, the Personal Health Dashboard and a global banner

   Action items

   • Move the Kinesis front-end to larger CPU and memory servers so fewer servers, and fewer threads per server, are needed
   • Add fine-grained alarming for thread consumption
   • Finish testing a higher operating system thread count limit
   • Greatly improve the cold-start time of the front-end fleet
   • Move the front-end server cache to a dedicated fleet
   • Move a few large AWS services such as CloudWatch to a separate, partitioned front-end fleet
   • Accelerate the cellularization of the front-end fleet to match the back-end
   • Change the Cognito webservers so they sustain Kinesis API errors without exhausting their buffers
   • Keep 3 hours of metric data in the CloudWatch local metrics data store, first in US-EAST-1 and then globally
   • Train support engineers regularly on the backup tool for posting to the Service Health Dashboard

   Lessons

   In the Kinesis front-end, the number of threads each server keeps grows directly with the number of servers in the fleet, so adding servers can push every server past an operating system limit at the same time. AWS sees cellularization of the front-end, already done for the back-end, as the way to keep the shard-map cache within a tested and operated range and to guard against unknown scaling limits.

   View case
2. IL-0588 AppOptics metrics APIs returned 5xx as Kafka partitions stayed leaderless after an AWS power event hit their ZooKeeper

   AppOptics2018

   Symptoms

   On May 31, 2018 from 21:53 UTC the metrics APIs of AppOptics and Librato returned a 5xx for a majority of incoming metric data. APM metrics and traces sent from AppOptics APM were affected, and many alert notifications were delayed.

   High error rateFailover failureMonitoring outage

   Trigger

   A power event in one of AWS's availability zones in us-east-1 cut off nodes in several critical parts of the AppOptics and Librato infrastructure, including the ZooKeeper node that was then the leader of the ensemble coordinating the Kafka brokers.

   Root cause

   Cloud provider outage

   Component

   Message brokerCoordination and key-value store

   Contributing factor

   Single point of failureExcessive blast radiusMonitoring blind spot

   Remediation

   • Restarted one ZooKeeper node by hand to force a leader election, after which the ensemble regained quorum
   • Restarted every Kafka broker in a rolling restart until all partitions were back online
   • Worked through the backlog, including the backlog of alerts, and resumed normal operations

   Action items

   • Isolate the Kafka clusters better across availability zones and regions
   • Stop sharing one ZooKeeper ensemble across Kafka clusters
   • Run game days on this failure and check whether newer Kafka versions fix it
   • Put in place more checks that notify the team earlier about subsystems that need extra help to recover

   Lessons

   Running services across several availability zones and more than one Kafka cluster did not isolate the pipeline from the failure of one zone, because partitions of different clusters sat in the same zones and the clusters shared one ZooKeeper ensemble. More operational monitoring of the subsystems that needed help to recover would have led to a faster recovery.

   View case

By root cause2