Cases10 cases

6 to 10 of 10

1. IL-0229 bunny.net's DNS servers crashed in a loop on a corrupted routing database file and could not fetch a fix from its CDN

   bunny.net2021

   Symptoms

   On 22 June 2021, within minutes of an update released at 8:25 AM UTC, bunny.net's global fleet of close to 100 DNS servers kept crashing and restarting, more than 60% of traffic was lost, hundreds of Gbits of throughput disappeared, and over 750,000 websites were affected. After 10 minutes the team realised that the DNS servers were restarting and dying and that nothing could bring them back up in that state.

   Container crash loopDNS resolution failuresService outageMisrouted requestsLoad imbalance

   Trigger

   At 8:25 AM UTC bunny.net released an update meant to reduce the download size of the SmartEdge optimization database, and it uploaded a corrupted file to Edge Storage.

   Root cause

   Faulty deployment

   Component

   Authoritative DNS servers and recordsCDN and edge networkObject storage

   Contributing factor

   Circular dependencySingle point of failureAutomation amplificationIneffective rollbackHarmful manual action

   Detection

   About 10 minutes after the release, the team realised that the DNS servers were restarting and dying.

   Remediation

   • Deployed by hand an update that disabled the SmartEdge system on the DNS nodes
   • Routed all requests through Frankfurt instead of Madrid on a couple of DNS servers
   • Migrated the deployment systems and files to a third-party cloud storage service
   • Rewired the deployment system and DNS software to the new storage and deployed
   • Synced the databases and deployed the newest file sets through the third-party service

   Action items

   • Phase out the BinaryPack library as a hot-fix
   • Test third-party libraries more extensively
   • Migrate the internal APIs to an independent third-party service
   • Find how to stop non-critical software from becoming a single point of failure across clusters
   • Make the DNS system keep a local copy of all backup data with automatic failure detection

   Lessons

   bunny.net had built its systems to rely on each other, including the critical parts of its internal infrastructure, so one corrupted file brought down several layers of redundancy with no way back. A part of the infrastructure treated as non-critical turned out to be a single point of failure for several clusters at once, which canary deployments did not catch. Building infrastructure inside its own ecosystem lets one failure cascade through all of it. In the rush to fix the outage the team made typos and mistakes, and the final deployment came after it calmed down and double-checked everything.

   View case
2. IL-0220 An overloaded AWS Transit Gateway dropped packets on 4 January 2021, and Slack's web tier saturated and went down

   Slack2021

   Symptoms

   During the Americas' morning of 4 January 2021, an external monitoring service paged Slack because error rates were creeping up. While the investigation was starting, Slack's own dashboarding and alerting service became unavailable, so responders had to work without their usual dashboards and alerts.

   High error rateMonitoring outagePacket lossService outageScale-up failureFile descriptor exhaustionCloud API rate limit exhausted

   Trigger

   On the first working day after the holidays, Slack's traffic went from the quietest time of the year to one of its biggest days overnight, with cold client caches pulling more data than usual, and one of Slack's AWS Transit Gateways did not scale fast enough for it.

   Root cause

   Cloud provider outage

   Component

   Transit gateway or network interconnectAutomated capacity manager

   Contributing factor

   Circular dependencyAutomation amplification

   Detection

   An external monitoring service paged Slack when error rates started creeping up during the Americas' morning. AWS engineers were separately alerted to the packet drops by AWS's internal monitoring.

   Remediation

   • Disabled automated downscaling of the web tier to keep serving capacity and allow investigation
   • Cleared up the broken, partly provisioned instances that filled the autoscaling groups
   • Repaired provision-service so new web tier instances could be provisioned and enter service
   • AWS engineers increased the Transit Gateway capacity manually across all Availability Zones

   Action items

   • AWS reviews its Transit Gateway scaling algorithms for large packet-per-second increases
   • Request a preemptive upscaling of the Transit Gateways at the end of the next holiday season
   • Run the dashboard service instances in the same VPC as their database to remove the Transit Gateway dependency
   • Load test provision-service regularly
   • Reevaluate health-checking and autoscaling configurations so they cannot overload provision-service again

   Lessons

   Monitoring has to stay as independent as possible of the infrastructure it watches, or it fails when it is needed most. Here the dashboard and alerting services failed because they ran in a different VPC from their databases and so depended on the Transit Gateways. provision-service had been load tested before, but this event exceeded the earlier scale-ups.

   View case
3. IL-0534 Amazon S3 in US-EAST-1 could not serve GET, LIST, PUT or DELETE requests for hours while its index subsystem restarted

   Amazon Web Services2017

   Symptoms

   From 9:37AM PST on February 28th, 2017, Amazon S3 in the Northern Virginia (US-EAST-1) Region could not serve requests. AWS services in the region that store data in S3 failed with it, including the S3 console, new Amazon EC2 instance launches, Amazon EBS volumes that needed data from an S3 snapshot, and AWS Lambda.

   Service outage

   Trigger

   While debugging an S3 billing system that was progressing more slowly than expected, an authorized S3 team member followed an established playbook to remove a small number of servers from one S3 subsystem used by the billing process, and entered one input of the command incorrectly.

   Root cause

   Operator error

   Component

   Object storageCloud block storageServerless function platform

   Contributing factor

   No safeguard on destructive changeExcessive blast radiusCircular dependency

   Remediation

   • Restarted the index subsystem, which served GET, LIST and DELETE again from 12:26PM PST
   • Restarted the placement subsystem after the index subsystem worked, restoring PUT by 1:54PM PST

   Action items

   • Change the capacity removal tool to remove capacity more slowly
   • Add safeguards that stop capacity removal when it would take a subsystem below its minimum required capacity
   • Audit other operational tools for similar safety checks
   • Improve the recovery time of key S3 subsystems and start the planned further partitioning of the index subsystem into cells now
   • Run the SHD administration console across multiple AWS regions

   View case
4. IL-0333 A downsized RDS PostgreSQL database maxed out its CPU at US peak and failing ELB health checks took buildkite.com down

   Buildkite2016

   Symptoms

   On 22 August 2016 at 17:21 UTC buildkite.com went offline: nobody could log in, view build logs or read the documentation, although builds kept running and GitHub and Bitbucket pull request statuses kept updating. The Buildkite team in Melbourne was asleep and woke at 21:00 UTC, almost 4 hours later, to emails, tweets and Slack messages saying Buildkite was down.

   Service outageDatabase overloadProbe failuresCloud API rate limit exhausted

   Trigger

   Over a weekend maintenance window, Buildkite downgraded its m4.10xlarge Multi-AZ RDS PostgreSQL database to an r3.2xlarge Multi-AZ instance to cut its AWS bill, and moved to a centralised pgbouncer to pool connections. The first US weekday peak after the change pushed the new instance's CPU to its maximum at about 16:00 UTC.

   Root cause

   Capacity exhaustion

   Component

   DatabaseCloud load balancerLiveness and readiness probesNode launch template

   Contributing factor

   Unverified assumptionHealth check tied to a dependencyAutomation amplificationCircular dependencyPages not deliveredOverlapping failures

   Detection

   New Relic, Pingdom, Datadog, Calibre and CloudWatch raised PagerDuty alerts, but the on-call phone call never came and only silent push notifications arrived. The team noticed at 21:00 UTC when they woke to emails, tweets and Slack messages from users.

   Remediation

   • Moved servers from the Agent API to the dashboard ELB to bring buildkite.com back online
   • Switched the Auto Scaling Groups to EC2 health checks and locked in desired counts
   • Upgraded the database to an instance type that could handle the load, using the aws-cli
   • Built a new AMI that did not depend on buildkite.com/_secret/version and rolled out servers that could self-deploy
   • Removed the centralised pgbouncer and went back to pgbouncer on each host

   Action items

   • Plan load testing after significant infrastructure changes without affecting other users of the system
   • Make the ELB health check return OK when the app answers, without checking database connectivity
   • Consider a separate health check endpoint for Pingdom that checks the database and other dependencies
   • Store the latest deployed revision as a file on S3 for newly launched servers to self-deploy from
   • Test by hand that PagerDuty calls reach every on-call team member
   • Investigate regular company-wide tests of everyone's paging setup
   • Ask PagerDuty about calling members when they go on call or requiring an immediate phone rule

   Lessons

   Buildkite listed six lessons. Watch AWS credits, because billing reports show $0 while credits are spent, so the real cost is hidden and big changes end up rushed. Load test after significant infrastructure changes. Think about what a health check should do and what happens when every server behind an ELB fails it. Do not make new servers look up the deployed revision from the service they are replacing. Make sure every on-call member has the right PagerDuty and phone settings before going on call. When servers churn in AWS, stop health checks and autoscaling, for example with the Auto Scaling feature to suspend and resume processes, so the EC2 request limit is not used up.

   View case
5. IL-0273 A power disruption took GitHub's Redis clusters offline, the app could not boot, and GitHub served 503s for 2 hours

   GitHub2016

   Symptoms

   At 00:23 UTC on 28 January 2016 GitHub became unavailable and returned HTTP 503 responses with the unicorn error page. On-call engineers received alerts for unreachable servers and a spike in exceptions related to Redis connectivity, after slightly over 25% of the servers and several network devices in the primary data center rebooted.

   Service outage

   Trigger

   At 00:23 UTC a brief disruption in the systems that supply power to the servers and equipment of GitHub's primary data center rebooted slightly over 25% of the servers and several network devices.

   Root cause

   Dependency failure

   Component

   In-memory cacheData center power and cooling

   Contributing factor

   Unpatched known bugSingle point of failureGap in automated testsCircular dependency

   Detection

   The reboots generated alerts to several on-call engineers, starting with notifications for unreachable servers and a spike in exceptions related to Redis connectivity.

   Remediation

   • Worked with on-site facilities technicians to drain flea power from the failed servers and start them cold so their disks were visible
   • Rebuilt the affected Redis clusters on alternate hardware
   • Restored the Redis cluster data onto standby equipment and brought the Redis-server processes back online
   • Removed the maintenance page after the HAProxy load balancers returned the recovered application servers to the pool

   Action items

   • Update the server firmware with the known issue across the fleet
   • Make tooling open issues automatically when new firmware updates are available, so their changelogs are reviewed
   • Add tests that make sure application processes start even when certain external systems are unavailable
   • Improve circuit breakers so functionality degrades gracefully when backend services are down
   • Review the availability requirements of internal systems such as server provisioning so they are on par with user-facing systems
   • Strengthen communication between teams during incidents
   • Predefine escalation strategies for all-hands incidents
   • Improve status messaging to users during incidents

   Lessons

   Dependencies between components fail in new ways unless they are tested, and a power loss cannot be fully prevented, so the goal is fast and reliable recovery. The application should start without non-essential backends, and internal systems needed to recover from an outage, such as server provisioning, must be as reliable as the system being recovered.

   View case

By root cause7