Cases2 cases

1 to 2 of 2

1. IL-0333 A downsized RDS PostgreSQL database maxed out its CPU at US peak and failing ELB health checks took buildkite.com down

   Buildkite2016

   Symptoms

   On 22 August 2016 at 17:21 UTC buildkite.com went offline: nobody could log in, view build logs or read the documentation, although builds kept running and GitHub and Bitbucket pull request statuses kept updating. The Buildkite team in Melbourne was asleep and woke at 21:00 UTC, almost 4 hours later, to emails, tweets and Slack messages saying Buildkite was down.

   Service outageDatabase overloadProbe failuresCloud API rate limit exhausted

   Trigger

   Over a weekend maintenance window, Buildkite downgraded its m4.10xlarge Multi-AZ RDS PostgreSQL database to an r3.2xlarge Multi-AZ instance to cut its AWS bill, and moved to a centralised pgbouncer to pool connections. The first US weekday peak after the change pushed the new instance's CPU to its maximum at about 16:00 UTC.

   Root cause

   Capacity exhaustion

   Component

   DatabaseCloud load balancerLiveness and readiness probesNode launch template

   Contributing factor

   Unverified assumptionHealth check tied to a dependencyAutomation amplificationCircular dependencyPages not deliveredOverlapping failures

   Detection

   New Relic, Pingdom, Datadog, Calibre and CloudWatch raised PagerDuty alerts, but the on-call phone call never came and only silent push notifications arrived. The team noticed at 21:00 UTC when they woke to emails, tweets and Slack messages from users.

   Remediation

   • Moved servers from the Agent API to the dashboard ELB to bring buildkite.com back online
   • Switched the Auto Scaling Groups to EC2 health checks and locked in desired counts
   • Upgraded the database to an instance type that could handle the load, using the aws-cli
   • Built a new AMI that did not depend on buildkite.com/_secret/version and rolled out servers that could self-deploy
   • Removed the centralised pgbouncer and went back to pgbouncer on each host

   Action items

   • Plan load testing after significant infrastructure changes without affecting other users of the system
   • Make the ELB health check return OK when the app answers, without checking database connectivity
   • Consider a separate health check endpoint for Pingdom that checks the database and other dependencies
   • Store the latest deployed revision as a file on S3 for newly launched servers to self-deploy from
   • Test by hand that PagerDuty calls reach every on-call team member
   • Investigate regular company-wide tests of everyone's paging setup
   • Ask PagerDuty about calling members when they go on call or requiring an immediate phone rule

   Lessons

   Buildkite listed six lessons. Watch AWS credits, because billing reports show $0 while credits are spent, so the real cost is hidden and big changes end up rushed. Load test after significant infrastructure changes. Think about what a health check should do and what happens when every server behind an ELB fails it. Do not make new servers look up the deployed revision from the service they are replacing. Make sure every on-call member has the right PagerDuty and phone settings before going on call. When servers churn in AWS, stop health checks and autoscaling, for example with the Auto Scaling feature to suspend and resume processes, so the EC2 request limit is not used up.

   View case
2. IL-0282 DynamoDB in US-East reached about 55% request errors as storage servers overloaded its metadata service

   Amazon Web Services2015

   Symptoms

   Early on Sunday 20 September 2015, the error rate of customer requests to Amazon DynamoDB in the US-East Region rose far beyond any level of the previous 3 years by 2:37am PDT and settled at about 55%. Some other AWS services in the region that depend on DynamoDB were affected as well.

   High error rateDatabase overload

   Trigger

   At 2:19am PDT a brief network disruption affected a portion of DynamoDB's storage servers, and they all requested their membership data from the metadata service at the same time.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   No capacity planningMonitoring blind spotRetry stormCircular dependencyExcessive blast radius

   Detection

   For the errors that remained after the main recovery, a growing number of customer support cases on Monday about tables stuck in the updating or deleting stage and higher than normal error rates showed that the low overall error rate was hitting some customers hard.

   Remediation

   • Paused requests to the metadata service, which cut the retries and relieved its load
   • Added significant capacity to the metadata service once it could answer administrative requests
   • Turned requests to the metadata service back on and returned storage servers to the customer request path
   • Restored a metadata partition that was still not taking its full traffic, ending the remaining errors on Monday

   Action items

   • Significantly increased the capacity of the metadata service, which AWS had already done by the time of the summary
   • Add stricter monitoring on performance dimensions such as membership size, to plan capacity ahead
   • Reduce the rate at which storage nodes request membership data
   • Lengthen the time allowed to process membership queries
   • Segment DynamoDB into many metadata service instances, each serving only part of the storage server fleet
   • Change SQS metadata caching so send and receive keep working without long access to the metadata table
   • Partition work across the Auto Scaling servers to process a backlog of scaling jobs in parallel
   • Prune older scaling activities that have been superseded
   • Increase the capacity available to process scaling activities
   • Change CloudWatch caching of DynamoDB group membership data to refresh only the smallest set of metrics
   • Deliver CloudWatch metrics faster through write-through caching
   • Roll out the shorter timeout on the DynamoDB-dependent API call in Console login

   Lessons

   A low overall error rate can hide disproportionately high error rates for a few customers, and AWS should have posted to the dashboard sooner. A call that does not need to succeed, like the DynamoDB-dependent API call in Console login, should fail quickly instead of blocking on a very long timeout. Splitting the metadata service into many instances, each serving part of the fleet, is meant to contain the impact of software, capacity or infrastructure failures.

   View case

By root cause1