Incident Library
Applied filters Root causeConfiguration changeComponentetcdClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0200 Newly provisioned Kubernetes nodes ran an incompatible etcd version and broke cluster consensus in several GitHub production clusters, causing 5XX errors on Git operations and page loads
    GitHub2026
    Symptoms

    From 01:36 UTC on 12 March 2026, GitHub.com returned elevated error rates for Git operations, web requests and related services. At 01:54 UTC the GitHub status page said it was investigating reports of impacted performance for some services.

    High error rate
    Trigger

    During a planned infrastructure upgrade, a configuration issue made newly provisioned Kubernetes nodes run an incompatible version of etcd.

    Root cause
    Configuration change
    Component
    etcd
    Contributing factor
    Monitoring blind spot
    Detection

    The status page opened the incident at 01:54 UTC, 18 minutes after errors began, citing reports of impacted performance.

    Remediation
    • paused deployments for the duration of the incident
    • identified the newly provisioned nodes running the incompatible etcd version and removed them, which restored cluster consensus
    • ran a validation deploy that confirmed all systems were healthy before resuming normal operations
    Action items
    • Enforce version compatibility programmatically during node replacements
    • Add monitoring that detects split-brain conditions earlier
    • Update the recovery tooling to reduce restoration time
    View case
  2. IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1
    DigitalOcean2024
    Symptoms

    At 13:02 UTC on 2024-01-23 DigitalOcean posted that, as of 12:18 UTC, its engineers were investigating an issue with Kubernetes clusters in the FRA1 region and that users might see errors when communicating with their clusters. The control planes of the affected DOKS (DigitalOcean Kubernetes) clusters had become inaccessible.

    Resources deletedControl plane unresponsive
    Trigger

    An incorrect configuration was applied to an internal infrastructure cluster in FRA1 at 12:21 UTC during a maintenance operation.

    Root cause
    Configuration change
    Component
    etcdKubernetes API server
    Contributing factor
    Automation amplificationNo safeguard on destructive changeExcessive blast radius
    Detection

    Engineers were investigating errors with Kubernetes clusters in FRA1 as of 12:18 UTC and identified the incorrect configuration as the cause at 12:36.

    Remediation
    • restored each affected cluster's etcd from its latest twice-daily backup
    • restored the stateless control plane components after etcd
    • patched the recovery mechanism after unexpected problems during recovery
    • found and fixed an additional small bug during recovery
    • contacted affected customers directly with further instructions, since some might need to reconfigure Kubernetes resources
    Action items
    • Prevent automatic deletion of infrastructure cluster resources under any circumstances in the production environment
    Lessons

    When managed control planes run as resources in another cluster, one configuration mistake in that cluster can delete hundreds of control planes at once through ordinary automation. The etcd backups limited the damage, but their twice-daily schedule set the worst case at 12 hours of lost control plane state, and the recovery tooling itself needed patching during the incident.

    View case
Check your cluster for this

These 2 cases involve etcd and Kubernetes API server. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change2 cases

Component
etcd2Kubernetes API server1
Contributing factor
Automation amplification1Excessive blast radius1Monitoring blind spot1No safeguard on destructive change1
Organisation
DigitalOcean1GitHub1
  1. IL-0200 Newly provisioned Kubernetes nodes ran an incompatible etcd version and broke cluster consensus in several GitHub production clusters, causing 5XX errors on Git operations and page loads
    GitHub2026
  2. IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1
    DigitalOcean2024