Cases4 cases

1 to 4 of 4

1. IL-0236 After an Istio 1.15 to 1.24 upgrade, half of Wikimedia's ML model servers returned 5XX for about a day

   Wikimedia Foundation2026

   Symptoms

   From 17:30 UTC on 23 February 2026, right after the ml-serve-codfw Kubernetes cluster was upgraded, 5XX responses appeared in the metrics of a subset of Wikimedia's machine learning model servers and alerts started to fire. About half of the running model servers were affected and stayed unreachable for roughly 24 hours.

   High error rate

   Trigger

   The two production Machine Learning Kubernetes clusters were upgraded to version 1.31, which included upgrading Istio from 1.15 to 1.24; ml-serve-codfw was done at 17:21 UTC on 23 February and ml-serve-eqiad between 08:00 and 11:00 UTC on 24 February.

   Root cause

   Breaking change in an upgrade

   Component

   IstioModel inference server

   Contributing factor

   Ignored early signal

   Detection

   Users of the affected model servers first reported the issue in the talk-to-machine-learning Slack channel. Alerts for HTTP 5XX had fired in the Machine Learning IRC channel right after the codfw upgrade, but nobody on the team noticed them.

   Remediation

   • Deployed an Istio hot patch that restored the broken functionality and fixed most of the HTTP 500 errors
   • Deployed the permanent fix to all affected clusters after verifying it on ml-staging-codfw

   Action items

   • Deliver alerts to the relevant Slack channels and watch IRC or alerts.wikimedia.org
   • Add more SLOs and fix the existing ones, whose Istio metrics changed

   Lessons

   The problem could have been caught on the staging cluster first, or by ad-hoc testing within the team after each cluster upgrade, which had been requested when the upgrades were announced. Not everyone involved knew that httpbb tests existed, and nobody on the Machine Learning team saw the alerts that fired right after the codfw upgrade. Recovery was fairly quick once users reported the problem, helped by having the right person online to find the Istio hot patch.

   View case
2. IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNS

   Adevinta2023

   Symptoms

   On a Friday afternoon a tenant of SCHIP, Adevinta's multi-tenant Kubernetes runtime, reported a spike of HTTP 5xx errors in some of its services. The drops in quality of service came back periodically through the following days, each one short, and were reported mostly around midnight.

   High error rateDNS resolution failuresHigh latency

   Root cause

   DNS or network fault

   Component

   CoreDNSClient-side DNS resolver and cacheTelemetry agent

   Contributing factor

   Unreviewed defaultsUnknown limitIgnored early signalUnverified assumption

   Detection

   A tenant reported a spike of HTTP 5xx errors in some of its services; the DNS Latency SLI had already been degraded for some time without being acted on.

   Remediation

   • raised the dnsmasq concurrent query limit from 150 to 1500
   • added a CoreDNS header block that sets the ra and aa flags and clears rd, so dnsmasq caches cluster.local answers
   • configured dnsmasq on each node to answer NXDOMAIN for unresolvable search-domain names instead of forwarding them to CoreDNS
   • changed the fluent-bit log router hostname to a fully qualified name
   • asked the affected tenants to use fully qualified names for internal calls

   Lessons

   The DNS Latency SLI had shown the problem long before anyone looked at it. An SLO only protects you when its SLIs are trusted and a drop is investigated. The investigation took a month partly because several on-call groups worked on it separately, without one place for observations and hypotheses, and sometimes chased hypotheses already discarded. Disprove one hypothesis before starting the next, and collect data the metrics system lacks: DNStop showed the garbage lookups that no dashboard did. With ndots:5, short service names multiply DNS traffic; a node-local cache only helps if it can cache the cluster DNS answers at all.

   View case
3. IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two pods

   Airbnb2020

   Symptoms

   A production service at Airbnb showed a high error rate and could not be deployed. The deploy failed with a cryptic Kubernetes error about being unable to access invalid index 5.

   High error rateThroughput degradationDeploys failing

   Trigger

   A change to the mutating admission controller, which had been rolled out within the previous month, had gone in seven days before the incident surfaced; when it had to delete two items from an array in a spec, it generated an invalid JSON patch that removed index 2 and then index 3.

   Root cause

   Software defect

   Component

   Admission webhookHorizontal pod autoscaler

   Contributing factor

   Change with a delayed effectMonitoring blind spotIgnored early signal

   Detection

   The affected service reported a high error rate and its deploy failed; the slow degradation of other services over the previous seven days had gone unnoticed.

   Remediation

   • deleted the mutating admission controller from the affected cluster

   Action items

   • Create alerts from the existing metrics that would have shown the problem earlier

   Lessons

   Keep in mind that a mutating admission controller exists in the cluster; a recently added one is easy to forget during an incident. Search for the exact error message early instead of working on the visible symptoms. Page a teammate when progress stalls.

   View case
4. IL-0049 Partial DNS outage after CoreDNS scaled down and kube-proxy failed to delete stale conntrack entries

   Preply2020

   Symptoms

   Prometheus detected a high rate of 5xx responses on three services, called A, B and C in the write-up, and paged the on-call engineers through VictorOps at 22:18 EET. The 5xx rate was also visible in Kibana.

   DNS resolution failuresHigh error rateStale endpoint routing

   Trigger

   Because load in the cluster was low, the CoreDNS autoscaler reduced the number of CoreDNS pods from 3 to 2.

   Root cause

   DNS or network fault

   Component

   CoreDNSkube-proxyconntrack table

   Contributing factor

   Automation amplificationIgnored early signalNo caching

   Detection

   Prometheus alerted on a high 5xx rate on services A, B and C and paged on-call through VictorOps.

   Remediation

   • reverted the latest release of the affected service
   • a regular deploy created new nodes, the CoreDNS autoscaler added CoreDNS pods, and the conntrack table was rewritten

   Action items

   • Disable the autoscaler for CoreDNS
   • Run a DNS cache server in production
   • Monitor conntrack usage

   Lessons

   Scaling CoreDNS down is not free: if kube-proxy fails to clear conntrack entries for the removed pod, clients keep sending DNS queries to it. Known DNS weaknesses in a cluster should be prioritised before they cause an outage, and fixes that only treat consequences leave the underlying conntrack problem in place.

   View case

By root cause3