Incident Library
Applied filters ComponentModel inference serverFactorIgnored early signalClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0236 After an Istio 1.15 to 1.24 upgrade, half of Wikimedia's ML model servers returned 5XX for about a day
    Wikimedia Foundation2026
    Symptoms

    From 17:30 UTC on 23 February 2026, right after the ml-serve-codfw Kubernetes cluster was upgraded, 5XX responses appeared in the metrics of a subset of Wikimedia's machine learning model servers and alerts started to fire. About half of the running model servers were affected and stayed unreachable for roughly 24 hours.

    High error rate
    Trigger

    The two production Machine Learning Kubernetes clusters were upgraded to version 1.31, which included upgrading Istio from 1.15 to 1.24; ml-serve-codfw was done at 17:21 UTC on 23 February and ml-serve-eqiad between 08:00 and 11:00 UTC on 24 February.

    Root cause
    Breaking change in an upgrade
    Component
    IstioModel inference server
    Contributing factor
    Ignored early signal
    Detection

    Users of the affected model servers first reported the issue in the talk-to-machine-learning Slack channel. Alerts for HTTP 5XX had fired in the Machine Learning IRC channel right after the codfw upgrade, but nobody on the team noticed them.

    Remediation
    • Deployed an Istio hot patch that restored the broken functionality and fixed most of the HTTP 500 errors
    • Deployed the permanent fix to all affected clusters after verifying it on ml-staging-codfw
    Action items
    • Deliver alerts to the relevant Slack channels and watch IRC or alerts.wikimedia.org
    • Add more SLOs and fix the existing ones, whose Istio metrics changed
    Lessons

    The problem could have been caught on the staging cluster first, or by ad-hoc testing within the team after each cluster upgrade, which had been requested when the upgrades were announced. Not everyone involved knew that httpbb tests existed, and nobody on the Machine Learning team saw the alerts that fired right after the codfw upgrade. Recovery was fairly quick once users reported the problem, helped by having the right person online to find the Istio hot patch.

    View case
Check your cluster for this

This case involves Model inference server and Istio. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade1 case

Component
Model inference serverIstio
Contributing factor
Ignored early signal
Organisation
Wikimedia Foundation
  1. IL-0236 After an Istio 1.15 to 1.24 upgrade, half of Wikimedia's ML model servers returned 5XX for about a day
    Wikimedia Foundation2026