IL-0236 Wikimedia Foundation · 2026-02-23kubernetesnetworkingai-infra

After an Istio 1.15 to 1.24 upgrade, half of Wikimedia's ML model servers returned 5XX for about a day

In Wikimedia's ML serving clusters Istio acts both as the ingress gateway and as a mesh sidecar, a transparent proxy that carries all ingress and egress traffic of the model servers. Model servers call services such as the MediaWiki API in one of two ways: some name the MediaWiki API VIP in their code with the right Host header, for example mw-api-ro.discovery.wmnet with Host header en.wikipedia.org, and others simply call a domain such as en.wikipedia.org and rely on the transparent proxy to route it. The ML team had never chosen one of the two as the standard. Istio 1.24 validates the "host" attribute of VirtualService definitions more strictly than 1.15: when two or more VirtualServices have overlapping host values, only one of them is used and the others are discarded. After the upgrade, the model servers that used the transparent proxy could no longer reach the MediaWiki API and returned 5XX.

The codfw cluster broke at 17:30 UTC on 23 February, but the alerts were missed, and the eqiad cluster was upgraded the next morning and broke at 11:00 UTC. Investigation began at 11:30 UTC on 24 February after users reported the problem in Slack. With the right Wikimedia technical staff member online to help, the team found an Istio hot patch that restored the broken behaviour, and deploying it at 16:00 UTC fixed most of the HTTP 500 errors. A permanent fix was filed on 27 February and verified on ml-staging-codfw, and was deployed to all affected clusters at 13:17 UTC on 3 March. The incident metadata gives the start as 17:20 UTC on 23 February and the end as 12:50 UTC on 24 February.

Severity: –
Scope: A subset of ML model servers, about half of those running, were unreachable or returned 5XX for about 24 hours, first in codfw and the next day in eqiad; the Search Update pipeline, the Revised Tone model server and the ML cache storage were affected
Users affected: Consumers of the affected model servers
Region: codfw and eqiad
Time to detect: 16h 45m
Time to resolve: 22h 30m

Symptoms

From 17:30 UTC on 23 February 2026, right after the ml-serve-codfw Kubernetes cluster was upgraded, 5XX responses appeared in the metrics of a subset of Wikimedia's machine learning model servers and alerts started to fire. About half of the running model servers were affected and stayed unreachable for roughly 24 hours.

The affected model servers were article-models, articletopic-outlink, revertrisk, revision-models, revscoring-articlequality, revscoring-articletopic, revscoring-draftquality, revscoring-drafttopic, revscoring-editquality-damaging and revscoring-editquality-goodfaith, all of them calling the MediaWiki API through the Istio sidecar's transparent proxy. The alerts pointed to HTTP 5XX errors for each affected service; the error volume was moderate compared with other Wikimedia services. The Search Update pipeline, which calls these model servers, the Revised Tone model server and the Machine Learning cache storage were also affected.

The alerts went to the Machine Learning team's IRC channel and nobody noticed them. At 10:15 UTC on 24 February consumers of the model servers reported the codfw problems in the talk-to-machine-learning Slack channel. By then the ml-serve-eqiad cluster was being upgraded the same way, and when that finished at 11:00 UTC the services in eqiad failed too.

Symptom High error rate

Root cause

Root cause Breaking change in an upgrade

Component Istio Model inference server

Contributing factor Ignored early signal

Trigger: The two production Machine Learning Kubernetes clusters were upgraded to version 1.31, which included upgrading Istio from 1.15 to 1.24; ml-serve-codfw was done at 17:21 UTC on 23 February and ml-serve-eqiad between 08:00 and 11:00 UTC on 24 February.

Detection

Users of the affected model servers first reported the issue in the talk-to-machine-learning Slack channel. Alerts for HTTP 5XX had fired in the Machine Learning IRC channel right after the codfw upgrade, but nobody on the team noticed them.

Remediation

Deployed an Istio hot patch that restored the broken functionality and fixed most of the HTTP 500 errors
Deployed the permanent fix to all affected clusters after verifying it on ml-staging-codfw

Action items

Deliver alerts to the relevant Slack channels and watch IRC or alerts.wikimedia.org
Add more SLOs and fix the existing ones, whose Istio metrics changed

Lessons

The problem could have been caught on the staging cluster first, or by ad-hoc testing within the team after each cluster upgrade, which had been requested when the upgrades were announced. Not everyone involved knew that httpbb tests existed, and nobody on the Machine Learning team saw the alerts that fired right after the codfw upgrade. Recovery was fairly quick once users reported the problem, helped by having the right person online to find the Istio hot patch.

Sources

https://wikitech.wikimedia.org/wiki/Incidents/2026-02-23_ml-serve · runbook · archived copy

References

Related cases

IL-0180 A GKE control plane auto-upgrade from 1.22 to 1.23 removed the v1beta1 Ingress and ValidatingWebhookConfiguration APIs that Istio 1.7.4 still used, and Istio stopped working in productionSunwoo Park · 2023
same root cause · shared Istio

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Istio, Model inference server among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0236
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0236/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.