Cases14 cases

1 to 5 of 14

1. IL-0200 Newly provisioned Kubernetes nodes ran an incompatible etcd version and broke cluster consensus in several GitHub production clusters, causing 5XX errors on Git operations and page loads

   GitHub2026

   Symptoms

   From 01:36 UTC on 12 March 2026, GitHub.com returned elevated error rates for Git operations, web requests and related services. At 01:54 UTC the GitHub status page said it was investigating reports of impacted performance for some services.

   High error rate

   Trigger

   During a planned infrastructure upgrade, a configuration issue made newly provisioned Kubernetes nodes run an incompatible version of etcd.

   Root cause

   Configuration change

   Component

   etcd

   Contributing factor

   Monitoring blind spot

   Detection

   The status page opened the incident at 01:54 UTC, 18 minutes after errors began, citing reports of impacted performance.

   Remediation

   • paused deployments for the duration of the incident
   • identified the newly provisioned nodes running the incompatible etcd version and removed them, which restored cluster consensus
   • ran a validation deploy that confirmed all systems were healthy before resuming normal operations

   Action items

   • Enforce version compatibility programmatically during node replacements
   • Add monitoring that detects split-brain conditions earlier
   • Update the recovery tooling to reduce restoration time

   View case
2. IL-0188 Rolling updates on EKS returned 502 errors for about 30 seconds because Kubernetes deleted old pods as soon as new pods passed readiness, before the ALB had marked the new pod targets healthy

   Yonghyun Kim2025

   Symptoms

   Every rolling update of the team's microservices on Amazon EKS, behind an Application Load Balancer, was followed by intermittent 502 and 503 errors for users. A Slack message at 3 a.m. read "배포는 성공했는데, 서비스는 죽었다" (the deployment succeeded, but the service died). Every pod was Ready and Datadog was green while users saw errors.

   High error rateService outage

   Trigger

   A rolling update of a Deployment served through an ALB that registers pod IPs as targets, with no pod readiness gates.

   Root cause

   Faulty deployment

   Component

   Cloud load balancerIngress controllerPod termination

   Contributing factor

   Unverified assumptionUnreviewed defaultsMonitoring blind spot

   Detection

   A Slack message at 3 a.m. reported that the deployment had succeeded but the service was down; Datadog dashboards were green at the time.

   Remediation

   • labelled the namespace with elbv2.k8s.aws/pod-readiness-gate-inject=enabled so the AWS Load Balancer Controller injects pod readiness gates and Kubernetes waits for the ALB target to be healthy before deleting old pods

   Lessons

   A pod being Ready in Kubernetes does not mean the load balancer will send it traffic. Without a readiness gate, a rolling update can delete the old pods while the ALB still considers the new targets unhealthy, and dashboards that track pod status stay green during the errors. Making pods wait longer on shutdown reduced the errors but did not fix the start-up gap and made every deployment much slower; the readiness gate ties the pod's Ready state to the ALB's view of the target.

   View case
3. IL-0413 LLM inference engines timed out or ran out of resources under load in about 60% of 156 severe incidents at Microsoft

   Microsoft2025

   Symptoms

   In Microsoft's cloud LLM inference service, the inference engines themselves failed in 94 of the 156 high-severity incidents from April to June 2025 (≈60%). Requests timed out or ran into resource exhaustion under load, and callers got HTTP 408, 429 and 5xx errors.

   Intermittent timeoutsHigh error rateHigh latency

   Trigger

   Bursty load, traffic with unusually high sampling parameters and long-running requests pushed the inference engines into resource exhaustion and timeouts.

   Root cause

   Capacity exhaustion

   Component

   Model inference server

   Contributing factor

   Monitoring blind spot

   Detection

   Across the 156 incidents, 115 (≈74%) were auto-detected, meaning health probes or DRI pipelines alerted before customers reported the problem; the others were not.

   Remediation

   • Enabled dynamic pushback at the load balancer with caps on pending tokens and requests
   • Tuned request batching and GPU/CPU replica counts
   • Improved gRPC backoff and error mapping
   • Limited sampling parameters and throttled per client
   • Enabled streaming, tuned request batching and capped pending tokens against engine connection timeouts on long-running requests
   • Increased capacity, routed traffic to healthy endpoints and restarted nodes, the actions applied mostly to inference engine failures

   Action items

   • Optimise serving for long-running requests and high sampling regimes, with connection liveness and GPU capacity-aware routing as the main levers
   • Expand health-probe-driven failover and GPU capacity-aware routing so traffic moves off degraded endpoints on its own
   • Forecast token and throughput growth and automate capacity increase decisions
   • Deploy adaptive alerting and expand engine-level telemetry to close detection gaps

   Lessons

   Hotfixes restore correctness but lengthen time to mitigate, while traffic routing and node rebalancing mitigate faster under overload and timeout patterns. Even with AIOps tooling a meaningful share of incidents still needs manual detection or judgement, so health-probe-driven traffic routing, restarts and capacity increases are the decisions to automate first.

   View case
4. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services

   OpenAI2024

   Symptoms

   Alerts fired at 3:13 PM PST on 11 December 2024, and from 3:16 PM every OpenAI service was significantly degraded or completely unavailable, including ChatGPT, the API and Sora. The status page reported API calls returning errors and difficulties logging in to platform.openai.com and ChatGPT. Customer impact peaked at 3:40 PM.

   Service outageHigh error rateControl plane unresponsiveDNS resolution failures

   Trigger

   At 2:23 PM PST a change introducing a new telemetry service, which collects detailed Kubernetes control plane metrics, was merged and the deployment pipeline started. Between 2:51 PM and 3:20 PM the change was applied to all clusters.

   Root cause

   Configuration change

   Component

   Kubernetes API serverClient-side DNS resolver and cacheTelemetry agent

   Contributing factor

   Unrepresentative stagingNo staged rolloutCache-masked failureMonitoring blind spotLimited debug access

   Detection

   OpenAI's detection tools worked: alerts fired at 3:13 PM PST, a few minutes before customer impact began at 3:16 PM, and the root cause was identified at 3:16 PM.

   Remediation

   • Scaled clusters down to reduce the total Kubernetes API load
   • Blocked network access to the Kubernetes admin APIs so no new expensive requests arrived
   • Scaled up the Kubernetes API servers
   • Removed the telemetry service once enough control was back
   • Moved traffic from impacted clusters to healthy ones
   • Intervened manually in clusters where simultaneous resource downloads saturated resource limits

   Action items

   • Put every infrastructure configuration change through a phased rollout with continuous monitoring of both the service workloads and the clusters, including the Kubernetes control plane
   • Run fault injection tests in which the data plane runs without the control plane for longer, and tests that roll out bad changes on purpose to check detection and rollback
   • Build break-glass mechanisms so engineers can reach the Kubernetes API servers in any circumstances
   • Decouple the Kubernetes data plane from the control plane so that service discovery for critical workloads does not depend on the control plane
   • Speed up recovery with better caching and dynamic rate limiters for resources needed at cluster startup, and regular drills that rapidly replace an entire cluster

   Lessons

   Several systems and processes failed at once and interacted in unexpected ways: tests did not catch the change's effect on the Kubernetes control plane, DNS caching put a delay between the change and the failures, and being locked out of the control plane made remediation very slow. Because service discovery used Kubernetes DNS, the control plane was carrying load for product traffic even though the data plane was meant to be independent of it.

   View case
5. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes

   Reddit2023

   Symptoms

   About two minutes after an engineer started the Kubernetes 1.23 to 1.24 upgrade of the cluster running Reddit's legacy stack, just after 19:00 UTC, the site stopped. The CDN edge request graph, which is kept separate from the cluster, showed traffic by status collapsing. Users saw the error page, a homepage with no posts, or plain errors.

   Service outageHigh error rateDNS resolution failuresMonitoring outagePods stuck in ContainerCreatingControl plane lag

   Trigger

   An in-place upgrade of the legacy cluster from Kubernetes 1.23 to 1.24; in 1.24 the node-role.kubernetes.io/master label is removed even from running clusters.

   Root cause

   Breaking change in an upgrade

   Component

   CalicoKubernetes API serverAdmission webhooketcd

   Contributing factor

   Configuration not in codeUnrepresentative stagingUntested backupMonitoring blind spotSingle point of failure

   Detection

   The site broke about two minutes into the upgrade; an incident was opened immediately and responders were on the call within three minutes, working from CDN edge graphs because the cluster's own metrics were gone.

   Remediation

   • deleted the Open Policy Agent webhook configurations to stop API server timeouts
   • terminated all worker nodes of the cluster
   • restored the control plane from the pre-upgrade backup onto the first control plane node
   • switched off the cluster autoscaler until the control plane was stable
   • generated new TLS certificates so replacement control plane nodes could join etcd
   • brought traffic back in steps from 1% to 100%
   • restarted legacy services that had been stopped by hand

   Action items

   • Codify cluster configuration so every choice has a record, and document in detail what cannot be codified
   • Standardise components across clusters and replace bespoke configurations
   • Build tooling to manage the lifecycle of whole clusters so they can be recreated or replicated
   • Add pre-production clusters that can be tested with confidence
   • Improve the backup restore process and make cluster upgrades safe

   Lessons

   Configuration that exists only in a live cluster, edited by hand through a special tool, can be forgotten, as the route reflector setup was, and then break on an upgrade nobody connects it to. A Kubernetes upgrade cannot be reverted without a restore, so the restore procedure has to be kept current and practised. Consistent, codified clusters are what make upgrades testable; bespoke clusters keep producing outages.

   View case

By root cause5