IL-0273 GitHub · 2016-01-28database

A power disruption took GitHub's Redis clusters offline, the app could not boot, and GitHub served 503s for 2 hours

A short power disruption in GitHub's primary data center rebooted slightly over a quarter of the servers. The servers that did not come back all shared one hardware class: after the power cycle they could not see their own physical drives, a known firmware issue. These servers spanned many racks and rows, so several Redis clusters lost every member at once even though their members sat in different racks. GitHub's application code had come to have a hard dependency on Redis in its boot path, so application processes failed to start while Redis was offline, and the site returned 503 responses. Early on, the team suspected the internal network and raised DDoS defenses before tracing the offline servers to a single hardware class.

One group of engineers worked with on-site facilities technicians to drain the flea power from the failed servers and start them from a cold state so the disks became visible. Another group rebuilt the affected Redis clusters on alternate hardware, slowed because a number of crucial internal systems ran on the offline hardware, which made provisioning new servers more difficult. Once the Redis data was restored on standby equipment and the Redis-server processes were back, the application processes recovered, the HAProxy load balancers returned the application servers to the backend pool, and the maintenance page was removed two hours and six minutes after the outage began. The following hours went into confirming that all systems worked normally and that no data had been lost.

Severity: –
Scope: GitHub was unavailable and returned HTTP 503 responses for two hours and six minutes
Users affected: all GitHub users
Region: –
Time to detect: –
Time to resolve: 2h 6m

Symptoms

At 00:23 UTC on 28 January 2016 GitHub became unavailable and returned HTTP 503 responses with the unicorn error page. On-call engineers received alerts for unreachable servers and a spike in exceptions related to Redis connectivity, after slightly over 25% of the servers and several network devices in the primary data center rebooted.

The load balancers and many frontend application servers were unaffected, but the systems they depend on were not available. Many of GitHub's ChatOps systems ran on servers that had rebooted, which caused confusion and delay at the start of the response, and status.github.com was not set to red until 00:32 UTC, eight minutes after the site became inaccessible.

The unreachable servers, the Redis connectivity errors and more connection attempts than usual pointed first to a problem in the internal network, and the team spent time early on bringing up DDoS defenses and checking network health; a DDoS attack turned out not to be the cause. Because several Redis clusters could not be reached on any of their members, engineers compared device uptimes across the facility: some servers reported uptimes of a few minutes while the network equipment had not rebooted, and all the servers still offline belonged to one hardware class.

As time passed, application processes were not starting as expected. The process tables and logs on the application servers showed that the processes failed to start because the Redis clusters were offline, which left the site without backend capacity. Remote console screenshots of the failed servers showed boot failures because the physical drives were no longer recognized.

Symptom Service outage

Root cause

Root cause Dependency failure

Component In-memory cache Data center power and cooling

Contributing factor Unpatched known bug Single point of failure Gap in automated tests Circular dependency

Trigger: At 00:23 UTC a brief disruption in the systems that supply power to the servers and equipment of GitHub's primary data center rebooted slightly over 25% of the servers and several network devices.

Detection

The reboots generated alerts to several on-call engineers, starting with notifications for unreachable servers and a spike in exceptions related to Redis connectivity.

Remediation

Worked with on-site facilities technicians to drain flea power from the failed servers and start them cold so their disks were visible
Rebuilt the affected Redis clusters on alternate hardware
Restored the Redis cluster data onto standby equipment and brought the Redis-server processes back online
Removed the maintenance page after the HAProxy load balancers returned the recovered application servers to the pool

Action items

Update the server firmware with the known issue across the fleet
Make tooling open issues automatically when new firmware updates are available, so their changelogs are reviewed
Add tests that make sure application processes start even when certain external systems are unavailable
Improve circuit breakers so functionality degrades gracefully when backend services are down
Review the availability requirements of internal systems such as server provisioning so they are on par with user-facing systems
Strengthen communication between teams during incidents
Predefine escalation strategies for all-hands incidents
Improve status messaging to users during incidents

Lessons

Dependencies between components fail in new ways unless they are tested, and a power loss cannot be fully prevented, so the goal is fast and reliable recovery. The application should start without non-essential backends, and internal systems needed to recover from an outage, such as server provisioning, must be as reliable as the system being recovered.

Sources

https://github.blog/news-insights/company-news/january-28th-incident-report/ · blog · archived copy

Related cases

IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returnedNebius · 2026
same root cause · shared Data center power and cooling
IL-0408 OpenRouter's API returned 500s and false 401 "User not found" errors when its API key cache dropped and came back emptyOpenRouter · 2026
same root cause · shared In-memory cache
IL-0134 A power event at Lambda's AUS01 data center rebooted 246 GPU nodes, all spine switches and the VAST storage, failing customer workloads and leaving stale NFS mounts on VMsLambda · 2026 · sev1
same root cause · shared Data center power and cooling
IL-0135 A power source switch during UPS maintenance cut power to a row of GPU servers and shut down Crusoe Cloud VMs in us-northcentral1Crusoe Cloud · 2023
same root cause · shared Data center power and cooling
IL-0255 Basecamp's two redundant links between data centers dropped together and a Redis replica resync loop stalled chat37signals · 2020
same root cause · shared In-memory cache

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, In-memory cache, Data center power and cooling among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0273
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0273/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.