Incident Library
Applied filters SymptomHigh error rateFactorMonitoring blind spotOrganisationGitHubClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0200 Newly provisioned Kubernetes nodes ran an incompatible etcd version and broke cluster consensus in several GitHub production clusters, causing 5XX errors on Git operations and page loads
    GitHub2026
    Symptoms

    From 01:36 UTC on 12 March 2026, GitHub.com returned elevated error rates for Git operations, web requests and related services. At 01:54 UTC the GitHub status page said it was investigating reports of impacted performance for some services.

    High error rate
    Trigger

    During a planned infrastructure upgrade, a configuration issue made newly provisioned Kubernetes nodes run an incompatible version of etcd.

    Root cause
    Configuration change
    Component
    etcd
    Contributing factor
    Monitoring blind spot
    Detection

    The status page opened the incident at 01:54 UTC, 18 minutes after errors began, citing reports of impacted performance.

    Remediation
    • paused deployments for the duration of the incident
    • identified the newly provisioned nodes running the incompatible etcd version and removed them, which restored cluster consensus
    • ran a validation deploy that confirmed all systems were healthy before resuming normal operations
    Action items
    • Enforce version compatibility programmatically during node replacements
    • Add monitoring that detects split-brain conditions earlier
    • Update the recovery tooling to reduce restoration time
    View case
Check your cluster for this

This case involves etcd. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change1 case

Component
etcd
Contributing factor
Monitoring blind spot
Organisation
GitHub
  1. IL-0200 Newly provisioned Kubernetes nodes ran an incompatible etcd version and broke cluster consensus in several GitHub production clusters, causing 5XX errors on Git operations and page loads
    GitHub2026