Cases1 case

1 to 1 of 1

1. IL-0200 Newly provisioned Kubernetes nodes ran an incompatible etcd version and broke cluster consensus in several GitHub production clusters, causing 5XX errors on Git operations and page loads

   GitHub2026

   Symptoms

   From 01:36 UTC on 12 March 2026, GitHub.com returned elevated error rates for Git operations, web requests and related services. At 01:54 UTC the GitHub status page said it was investigating reports of impacted performance for some services.

   High error rate

   Trigger

   During a planned infrastructure upgrade, a configuration issue made newly provisioned Kubernetes nodes run an incompatible version of etcd.

   Root cause

   Configuration change

   Component

   etcd

   Contributing factor

   Monitoring blind spot

   Detection

   The status page opened the incident at 01:54 UTC, 18 minutes after errors began, citing reports of impacted performance.

   Remediation

   • paused deployments for the duration of the incident
   • identified the newly provisioned nodes running the incompatible etcd version and removed them, which restored cluster consensus
   • ran a validation deploy that confirmed all systems were healthy before resuming normal operations

   Action items

   • Enforce version compatibility programmatically during node replacements
   • Add monitoring that detects split-brain conditions earlier
   • Update the recovery tooling to reduce restoration time

   View case

By root cause1