Cases19 cases

18 to 19 of 19

1. IL-0273 A power disruption took GitHub's Redis clusters offline, the app could not boot, and GitHub served 503s for 2 hours

   GitHub2016

   Symptoms

   At 00:23 UTC on 28 January 2016 GitHub became unavailable and returned HTTP 503 responses with the unicorn error page. On-call engineers received alerts for unreachable servers and a spike in exceptions related to Redis connectivity, after slightly over 25% of the servers and several network devices in the primary data center rebooted.

   Service outage

   Trigger

   At 00:23 UTC a brief disruption in the systems that supply power to the servers and equipment of GitHub's primary data center rebooted slightly over 25% of the servers and several network devices.

   Root cause

   Dependency failure

   Component

   In-memory cacheData center power and cooling

   Contributing factor

   Unpatched known bugSingle point of failureGap in automated testsCircular dependency

   Detection

   The reboots generated alerts to several on-call engineers, starting with notifications for unreachable servers and a spike in exceptions related to Redis connectivity.

   Remediation

   • Worked with on-site facilities technicians to drain flea power from the failed servers and start them cold so their disks were visible
   • Rebuilt the affected Redis clusters on alternate hardware
   • Restored the Redis cluster data onto standby equipment and brought the Redis-server processes back online
   • Removed the maintenance page after the HAProxy load balancers returned the recovered application servers to the pool

   Action items

   • Update the server firmware with the known issue across the fleet
   • Make tooling open issues automatically when new firmware updates are available, so their changelogs are reviewed
   • Add tests that make sure application processes start even when certain external systems are unavailable
   • Improve circuit breakers so functionality degrades gracefully when backend services are down
   • Review the availability requirements of internal systems such as server provisioning so they are on par with user-facing systems
   • Strengthen communication between teams during incidents
   • Predefine escalation strategies for all-hands incidents
   • Improve status messaging to users during incidents

   Lessons

   Dependencies between components fail in new ways unless they are tested, and a power loss cannot be fully prevented, so the goal is fast and reliable recovery. The application should start without non-essential backends, and internal systems needed to recover from an outage, such as server provisioning, must be as reliable as the system being recovered.

   View case
2. IL-0282 DynamoDB in US-East reached about 55% request errors as storage servers overloaded its metadata service

   Amazon Web Services2015

   Symptoms

   Early on Sunday 20 September 2015, the error rate of customer requests to Amazon DynamoDB in the US-East Region rose far beyond any level of the previous 3 years by 2:37am PDT and settled at about 55%. Some other AWS services in the region that depend on DynamoDB were affected as well.

   High error rateDatabase overload

   Trigger

   At 2:19am PDT a brief network disruption affected a portion of DynamoDB's storage servers, and they all requested their membership data from the metadata service at the same time.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   No capacity planningMonitoring blind spotRetry stormCircular dependencyExcessive blast radius

   Detection

   For the errors that remained after the main recovery, a growing number of customer support cases on Monday about tables stuck in the updating or deleting stage and higher than normal error rates showed that the low overall error rate was hitting some customers hard.

   Remediation

   • Paused requests to the metadata service, which cut the retries and relieved its load
   • Added significant capacity to the metadata service once it could answer administrative requests
   • Turned requests to the metadata service back on and returned storage servers to the customer request path
   • Restored a metadata partition that was still not taking its full traffic, ending the remaining errors on Monday

   Action items

   • Significantly increased the capacity of the metadata service, which AWS had already done by the time of the summary
   • Add stricter monitoring on performance dimensions such as membership size, to plan capacity ahead
   • Reduce the rate at which storage nodes request membership data
   • Lengthen the time allowed to process membership queries
   • Segment DynamoDB into many metadata service instances, each serving only part of the storage server fleet
   • Change SQS metadata caching so send and receive keep working without long access to the metadata table
   • Partition work across the Auto Scaling servers to process a backlog of scaling jobs in parallel
   • Prune older scaling activities that have been superseded
   • Increase the capacity available to process scaling activities
   • Change CloudWatch caching of DynamoDB group membership data to refresh only the smallest set of metrics
   • Deliver CloudWatch metrics faster through write-through caching
   • Roll out the shorter timeout on the DynamoDB-dependent API call in Console login

   Lessons

   A low overall error rate can hide disproportionately high error rates for a few customers, and AWS should have posted to the dashboard sooner. A call that does not need to succeed, like the DynamoDB-dependent API call in Console login, should fail quickly instead of blocking on a very long timeout. Splitting the metadata service into many instances, each serving part of the fleet, is meant to contain the impact of software, capacity or infrastructure failures.

   View case

By root cause8