Cases8 cases

7 to 8 of 8

1. IL-0535 An internal Kinesis Data Streams cell in US-EAST-1 degraded, raising errors in CloudWatch Logs, S3 events and Lambda

   Amazon Web Services2024

   Symptoms

   From 2:45 PM PDT on July 30th, 2024, several AWS services in the Northern Virginia (US-EAST-1) Region had higher latency and error rates: CloudWatch Logs, Amazon Data Firehose, the event framework that publishes Amazon S3 events, Amazon ECS, AWS Lambda, Amazon Redshift and AWS Glue. All of them used one internal cell of Amazon Kinesis Data Streams, whose requests had started to see elevated latency and errors.

   High latencyHigh error rateMonitoring outageProbe failures

   Trigger

   At 9:09 AM PDT on July 30th a routine deployment started in one Availability Zone of the Kinesis cell, and taking hosts in and out of service during the deployment set off the failure.

   Root cause

   Software defect

   Component

   Message brokerServerless function platformObject storage

   Contributing factor

   Unrepresentative stagingAutomation amplification

   Remediation

   • Deployed a change that shed load by reducing incoming stream volume from less time-sensitive internal workloads
   • Added capacity to the subsystem that provisions secure connections for Kinesis requests
   • Added more capacity at 6:32 PM PDT to further improve connection provisioning

   Action items

   • Keep the added connection provisioning capacity, load-shedding tooling and changed connection limits in place in US-EAST-1
   • Apply the same changes in every other region that has cells with the new Kinesis Data Streams architecture
   • Update the cell management system to better manage the workload that triggered the issue

   View case
2. IL-0382 Linode Object Storage in Newark and Atlanta returned 503 Slow Down to all clients under very high inbound traffic

   Linode2022

   Symptoms

   On 19 and 20 October 2022, Linode Object Storage in Newark and Atlanta slowed down, became less available and rate limited many requests with "503 - Slow Down" HTTP responses, after the clusters received extremely high inbound traffic in both request count and bandwidth.

   High error rateMemory pressure

   Trigger

   Extremely high inbound traffic, in both total requests and throughput, reached the Newark and Atlanta Object Storage clusters on 19 and 20 October 2022.

   Root cause

   Capacity exhaustion

   Component

   Object storage

   Contributing factor

   Unpatched known bugExcessive blast radius

   Remediation

   • Fixed the edge rate limiting so it enforces the advertised 750 requests per second per bucket
   • Adjusted Ceph tunables to stop runaway memory use during high-pressure storage events
   • Moved internal Object Storage components hit by resource contention to less busy, higher-performance infrastructure
   • Adjusted performance tunables on internal components and back-end clusters so customers were not rate limited too early

   Action items

   • Refine throughput and bandwidth throttling
   • Scale up the internal Object Storage infrastructure to spread load more efficiently
   • Look for further hardware performance improvements for the internal Object Storage infrastructure
   • Tune the load balancing algorithms to avoid hotspots in the clusters

   Lessons

   Linode's per-bucket rate limiting at the edge is meant to keep excess load off the back end; when it works as intended, the back-end limits, which throttle every client alike, should never come into play.

   View case

By root cause6