Cases1 case

1 to 1 of 1

1. IL-0273 A power disruption took GitHub's Redis clusters offline, the app could not boot, and GitHub served 503s for 2 hours

   GitHub2016

   Symptoms

   At 00:23 UTC on 28 January 2016 GitHub became unavailable and returned HTTP 503 responses with the unicorn error page. On-call engineers received alerts for unreachable servers and a spike in exceptions related to Redis connectivity, after slightly over 25% of the servers and several network devices in the primary data center rebooted.

   Service outage

   Trigger

   At 00:23 UTC a brief disruption in the systems that supply power to the servers and equipment of GitHub's primary data center rebooted slightly over 25% of the servers and several network devices.

   Root cause

   Dependency failure

   Component

   In-memory cacheData center power and cooling

   Contributing factor

   Unpatched known bugSingle point of failureGap in automated testsCircular dependency

   Detection

   The reboots generated alerts to several on-call engineers, starting with notifications for unreachable servers and a spike in exceptions related to Redis connectivity.

   Remediation

   • Worked with on-site facilities technicians to drain flea power from the failed servers and start them cold so their disks were visible
   • Rebuilt the affected Redis clusters on alternate hardware
   • Restored the Redis cluster data onto standby equipment and brought the Redis-server processes back online
   • Removed the maintenance page after the HAProxy load balancers returned the recovered application servers to the pool

   Action items

   • Update the server firmware with the known issue across the fleet
   • Make tooling open issues automatically when new firmware updates are available, so their changelogs are reviewed
   • Add tests that make sure application processes start even when certain external systems are unavailable
   • Improve circuit breakers so functionality degrades gracefully when backend services are down
   • Review the availability requirements of internal systems such as server provisioning so they are on par with user-facing systems
   • Strengthen communication between teams during incidents
   • Predefine escalation strategies for all-hands incidents
   • Improve status messaging to users during incidents

   Lessons

   Dependencies between components fail in new ways unless they are tested, and a power loss cannot be fully prevented, so the goal is fast and reliable recovery. The application should start without non-essential backends, and internal systems needed to recover from an outage, such as server provisioning, must be as reliable as the system being recovered.

   View case

By root cause1