Incident Library
Applied filters Root causeDependency failureFactorCircular dependencyOrganisationGitHubClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0273 A power disruption took GitHub's Redis clusters offline, the app could not boot, and GitHub served 503s for 2 hours
    GitHub2016
    Symptoms

    At 00:23 UTC on 28 January 2016 GitHub became unavailable and returned HTTP 503 responses with the unicorn error page. On-call engineers received alerts for unreachable servers and a spike in exceptions related to Redis connectivity, after slightly over 25% of the servers and several network devices in the primary data center rebooted.

    Service outage
    Trigger

    At 00:23 UTC a brief disruption in the systems that supply power to the servers and equipment of GitHub's primary data center rebooted slightly over 25% of the servers and several network devices.

    Root cause
    Dependency failure
    Component
    In-memory cacheData center power and cooling
    Contributing factor
    Unpatched known bugSingle point of failureGap in automated testsCircular dependency
    Detection

    The reboots generated alerts to several on-call engineers, starting with notifications for unreachable servers and a spike in exceptions related to Redis connectivity.

    Remediation
    • Worked with on-site facilities technicians to drain flea power from the failed servers and start them cold so their disks were visible
    • Rebuilt the affected Redis clusters on alternate hardware
    • Restored the Redis cluster data onto standby equipment and brought the Redis-server processes back online
    • Removed the maintenance page after the HAProxy load balancers returned the recovered application servers to the pool
    Action items
    • Update the server firmware with the known issue across the fleet
    • Make tooling open issues automatically when new firmware updates are available, so their changelogs are reviewed
    • Add tests that make sure application processes start even when certain external systems are unavailable
    • Improve circuit breakers so functionality degrades gracefully when backend services are down
    • Review the availability requirements of internal systems such as server provisioning so they are on par with user-facing systems
    • Strengthen communication between teams during incidents
    • Predefine escalation strategies for all-hands incidents
    • Improve status messaging to users during incidents
    Lessons

    Dependencies between components fail in new ways unless they are tested, and a power loss cannot be fully prevented, so the goal is fast and reliable recovery. The application should start without non-essential backends, and internal systems needed to recover from an outage, such as server provisioning, must be as reliable as the system being recovered.

    View case
Check your cluster for this

This case involves In-memory cache and Data center power and cooling. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Dependency failure1 case

Component
In-memory cacheData center power and cooling
Contributing factor
Circular dependencyUnpatched known bugSingle point of failureGap in automated tests
Organisation
GitHub
  1. IL-0273 A power disruption took GitHub's Redis clusters offline, the app could not boot, and GitHub served 503s for 2 hours
    GitHub2016