Incident Library
Applied filters Root causeSoftware defectSymptomLock contention or deadlockFactorNo headroomClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0385 GitHub MySQL read replicas hit a semaphore deadlock during a schema migration's table rename and kept crash-recovering
    GitHub2021
    Symptoms

    On 27 November 2021 at 20:40 UTC, a significant part of GitHub's MySQL read replicas entered a semaphore deadlock while a schema migration on a large MySQL table ran its final step, the rename that moves the updated table into place. Core GitHub services, including GitHub Actions, API Requests, Codespaces, Git Operations, Issues, GitHub Packages, GitHub Pages, Pull Requests and Webhooks, became degraded.

    Lock contention or deadlockDatabase overload
    Trigger

    The final rename step of a schema migration on a large MySQL table. Schema migrations are a common task at GitHub and often take weeks to complete.

    Root cause
    Software defect
    Component
    DatabaseSchema migration
    Contributing factor
    No headroomNo staged rollout
    Remediation
    • Promoted every healthy internal replica into the production path to add read capacity
    • Removed production traffic from the broken replicas until each had processed the table rename
    • Moved the recovered replicas back into production to restore capacity
    Action items
    • Continue functional partitioning so migrations can run in canary mode on a single shard
    • Update internal procedures to increase how much each cluster is over-provisioned
    • Pause schema migrations until it is known how to safeguard against this failure
    • Keep investigating the specific failure scenario
    • Test the migration tooling and classify ways to improve it for this kind of scenario
    View case
Check your cluster for this

This case involves Database and Schema migration. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
DatabaseSchema migration
Contributing factor
No headroomNo staged rollout
Organisation
GitHub
  1. IL-0385 GitHub MySQL read replicas hit a semaphore deadlock during a schema migration's table rename and kept crash-recovering
    GitHub2021