Incident Library
Applied filters Root causeBreaking change in an upgradeFactorIgnored early signalClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0236 After an Istio 1.15 to 1.24 upgrade, half of Wikimedia's ML model servers returned 5XX for about a day
    Wikimedia Foundation2026
    Symptoms

    From 17:30 UTC on 23 February 2026, right after the ml-serve-codfw Kubernetes cluster was upgraded, 5XX responses appeared in the metrics of a subset of Wikimedia's machine learning model servers and alerts started to fire. About half of the running model servers were affected and stayed unreachable for roughly 24 hours.

    High error rate
    Trigger

    The two production Machine Learning Kubernetes clusters were upgraded to version 1.31, which included upgrading Istio from 1.15 to 1.24; ml-serve-codfw was done at 17:21 UTC on 23 February and ml-serve-eqiad between 08:00 and 11:00 UTC on 24 February.

    Root cause
    Breaking change in an upgrade
    Component
    IstioModel inference server
    Contributing factor
    Ignored early signal
    Detection

    Users of the affected model servers first reported the issue in the talk-to-machine-learning Slack channel. Alerts for HTTP 5XX had fired in the Machine Learning IRC channel right after the codfw upgrade, but nobody on the team noticed them.

    Remediation
    • Deployed an Istio hot patch that restored the broken functionality and fixed most of the HTTP 500 errors
    • Deployed the permanent fix to all affected clusters after verifying it on ml-staging-codfw
    Action items
    • Deliver alerts to the relevant Slack channels and watch IRC or alerts.wikimedia.org
    • Add more SLOs and fix the existing ones, whose Istio metrics changed
    Lessons

    The problem could have been caught on the staging cluster first, or by ad-hoc testing within the team after each cluster upgrade, which had been requested when the upgrades were announced. Not everyone involved knew that httpbb tests existed, and nobody on the Machine Learning team saw the alerts that fired right after the codfw upgrade. Recovery was fairly quick once users reported the problem, helped by having the right person online to find the Istio hot patch.

    View case
  2. IL-0294 CircleCI Docker and machine jobs stalled for nearly 12 hours after a VM cleanup queue lost all its RabbitMQ consumers
    CircleCI2021
    Symptoms

    From 13:30 UTC on 21 May 2021, CircleCI jobs on Docker, the Machine executor (including Windows, Mac, Arm and GPU) and remote Docker were delayed or blocked. Engineers had been paged at 13:00 for excessive delays in starting jobs, and the alerts showed CircleCI could not provision some types and classes of VM because it was hitting its CPU quotas in GCP.

    Throughput degradationCloud API rate limit exhaustedDatabase overload
    Trigger

    The upgrade of the RabbitMQ cluster from 3.8.9 to 3.8.16 on 20 May 2021, which brought in a 15 minute consumer acknowledgement timeout for all queue types.

    Root cause
    Breaking change in an upgrade
    Component
    Message brokerDatabase
    Contributing factor
    Change with a delayed effectIgnored early signalRetry stormExcessive blast radiusOverlapping failures
    Detection

    Engineers were paged at 13:00 UTC on 21 May for excessive delays in starting jobs. Earlier, a 02:31 notification that a VM destroyer queue had no consumers was handled by restarting pods, and a 10:11 alert that a VM destroy queue was backing up notified once at its threshold and did not page.

    Remediation
    • Blocked Linux machine jobs from entering the work queue
    • Had GCP raise the CPU quotas
    • Deleted old VMs by hand
    • Cancelled some workflows to free resources
    • Reset the VM services deployment so the VM destroy queues had consumers again
    • Scaled up the RDS database instance, at 18:20 and again at 19:15
    • Turned down VM provisioning and blocked remote-docker jobs to shed load
    • Scaled down all pods that created and destroyed VMs
    • Deployed a revised distribution backend that kept remote-docker jobs out of the provisioning queues
    • Brought provisioning and distribution back one component at a time and reversed the mitigation code changes
    Action items
    • Make VM provisioning more robust to changes in load so it can recover from thundering herds
    • Make VM provisioning independent so its failure does not affect Docker jobs
    • Review and improve observability of RabbitMQ and the production queues
    • Batch requests to avoid hitting GCP API rate limits
    • Improve incident response for long-running incidents
    View case
Check your cluster for this

These 2 cases involve Database and Model inference server. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade2 cases

Component
Database1Model inference server1Istio1Message broker1
Contributing factor
Ignored early signal2Excessive blast radius1Change with a delayed effect1Overlapping failures1Retry storm1
Organisation
CircleCI1Wikimedia Foundation1
  1. IL-0236 After an Istio 1.15 to 1.24 upgrade, half of Wikimedia's ML model servers returned 5XX for about a day
    Wikimedia Foundation2026
  2. IL-0294 CircleCI Docker and machine jobs stalled for nearly 12 hours after a VM cleanup queue lost all its RabbitMQ consumers
    CircleCI2021