Incident Library
Applied filters SymptomHigh error rateOrganisationGitHubClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0200 Newly provisioned Kubernetes nodes ran an incompatible etcd version and broke cluster consensus in several GitHub production clusters, causing 5XX errors on Git operations and page loads
    GitHub2026
    Symptoms

    From 01:36 UTC on 12 March 2026, GitHub.com returned elevated error rates for Git operations, web requests and related services. At 01:54 UTC the GitHub status page said it was investigating reports of impacted performance for some services.

    High error rate
    Trigger

    During a planned infrastructure upgrade, a configuration issue made newly provisioned Kubernetes nodes run an incompatible version of etcd.

    Root cause
    Configuration change
    Component
    etcd
    Contributing factor
    Monitoring blind spot
    Detection

    The status page opened the incident at 01:54 UTC, 18 minutes after errors began, citing reports of impacted performance.

    Remediation
    • paused deployments for the duration of the incident
    • identified the newly provisioned nodes running the incompatible etcd version and removed them, which restored cluster consensus
    • ran a validation deploy that confirmed all systems were healthy before resuming normal operations
    Action items
    • Enforce version compatibility programmatically during node replacements
    • Add monitoring that detects split-brain conditions earlier
    • Update the recovery tooling to reduce restoration time
    View case
  2. IL-0201 A framework upgrade caused kernel panics in GitHub's Kubernetes infrastructure, and GitHub.com web and API requests, mostly for issues and pull requests, failed intermittently for about 16 hours
    GitHub2025
    Symptoms

    From 18:05 UTC on March 17, 2025, GitHub.com web and API requests failed now and then for a small percentage of users, mostly requests about issues and pull requests. At 18:39 UTC the status page reported degraded performance for Issues.

    High error rateIntermittent timeouts
    Trigger

    A framework upgrade rolled out to GitHub's Kubernetes infrastructure caused kernel panics.

    Root cause
    Breaking change in an upgrade
    Component
    Linux kernel
    Contributing factor
    –
    Detection

    The status page opened the incident at 18:39 UTC, 34 minutes after the failures began, as an investigation of reports of degraded performance for Issues.

    Remediation
    • Downgraded the framework upgrade that caused the kernel panics
    • Disabled the problematic feature of the framework
    Action items
    • Investigate why the upgrade caused the kernel panics
    • Take steps to temporarily prevent the problem from coming back
    • Work out longer-term patch plans
    • Improve observability so that similar classes of problems can be reacted to quickly
    View case
Check your cluster for this

These 2 cases involve etcd and Linux kernel. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Configuration change1 case

Component
etcd
Contributing factor
Monitoring blind spot
Organisation
GitHub
  1. IL-0200 Newly provisioned Kubernetes nodes ran an incompatible etcd version and broke cluster consensus in several GitHub production clusters, causing 5XX errors on Git operations and page loads
    GitHub2026

Breaking change in an upgrade1 case

Component
Linux kernel
Contributing factor
–
Organisation
GitHub
  1. IL-0201 A framework upgrade caused kernel panics in GitHub's Kubernetes infrastructure, and GitHub.com web and API requests, mostly for issues and pull requests, failed intermittently for about 16 hours
    GitHub2025