Incident Library

Organisation

GitHub

GitHub · github

3 cases · 2020 to 2026

Root cause Configuration change1DNS or network fault1Breaking change in an upgrade1
Component etcd1Linux kernel1Container memory limit1Container registry and its credentials1
Contributing factor Harmful manual action1Monitoring blind spot1Overlapping failures1Single point of failure1
Symptom High error rate2Image pull failures1Intermittent timeouts1OOMKilled1Service outage1

Cases

  1. IL-0200 Newly provisioned Kubernetes nodes ran an incompatible etcd version and broke cluster consensus in several GitHub production clusters, causing 5XX errors on Git operations and page loadsGitHub · 2026

    Root cause Configuration change

  2. IL-0201 A framework upgrade caused kernel panics in GitHub's Kubernetes infrastructure, and GitHub.com web and API requests, mostly for issues and pull requests, failed intermittently for about 16 hoursGitHub · 2025

    Root cause Breaking change in an upgrade

  3. IL-0199 A non-essential container exceeding its memory limit made GitHub's production pods unavailable, and their restarts failed because a DNS maintenance change had left the clusters unable to reach the image registryGitHub · 2020

    Root cause DNS or network fault