Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases52 cases

15 to 19 of 52

  1. IL-0119 ChatGPT web service pods crash-looped on failing health checks after a Cosmos DB failure, leaving too few healthy pods to serve the web layer
    OpenAI2025
    Symptoms

    From 3:30am to 7:08am PT, chatgpt.com and a small number of OpenAI APIs returned elevated error rates. The status page classed the incident as a full outage.

    High error rateContainer crash loopProbe failures
    Trigger

    A failure of Cosmos DB in ChatGPT, at OpenAI's service provider.

    Root cause
    Cloud provider outage
    Component
    Application database
    Contributing factor
    Health check tied to a dependency
    Remediation
    • the service provider recovered the affected Cosmos DB databases at 4:25am PT
    • fixed the web server's health checking logic, which ended the pod crash loop
    Action items
    • Change the web systems to degrade more gracefully when they fail, including concurrency control and better load shedding
    • Make the database layer more resilient to service provider outages (described as already underway)
    Lessons

    The database came back at 4:25am PT, yet users saw errors until 7:08am PT: the web tier's own health checks kept removing pods after the dependency had recovered. The planned changes target the web tier's behaviour under failure, with concurrency control and load shedding, rather than the database alone.

    View case
  2. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server
    Render2024
    Symptoms

    In January 2022 an etcd memory spike overloaded the Kubernetes control plane of one of Render's clusters in Frankfurt. At first only operations that need the control plane failed: new deploys and cron job runs.

    Control plane unresponsiveDeploys failingDNS resolution failuresHigh error rate
    Trigger

    A memory spike in etcd overloaded the cluster's control plane.

    Root cause
    Capacity exhaustion
    Component
    etcdCoreDNSKubernetes API server
    Contributing factor
    Unreviewed defaultsHarmful manual action
    Action items
    • Run CoreDNS on data plane nodes instead of control plane nodes
    • Run etcd on dedicated nodes
    Lessons

    CoreDNS can keep answering from its in-memory cache while the API server is down, but it cannot initialise without it. With CoreDNS on the control plane nodes, a reboot or restart during a control plane outage turned a failure of deploys and cron jobs into a failure of requests to services that were already running. The core problem was that the data plane was not isolated enough from the control plane: even without the reboot, the etcd memory spike alone could have forced CoreDNS to restart.

    View case
  3. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
    OpenAI2024
    Symptoms

    Alerts fired at 3:13 PM PST on 11 December 2024, and from 3:16 PM every OpenAI service was significantly degraded or completely unavailable, including ChatGPT, the API and Sora. The status page reported API calls returning errors and difficulties logging in to platform.openai.com and ChatGPT. Customer impact peaked at 3:40 PM.

    Service outageHigh error rateControl plane unresponsiveDNS resolution failures
    Trigger

    At 2:23 PM PST a change introducing a new telemetry service, which collects detailed Kubernetes control plane metrics, was merged and the deployment pipeline started. Between 2:51 PM and 3:20 PM the change was applied to all clusters.

    Root cause
    Configuration change
    Component
    Kubernetes API serverClient-side DNS resolver and cacheTelemetry agent
    Contributing factor
    Unrepresentative stagingNo staged rolloutCache-masked failureMonitoring blind spotLimited debug access
    Detection

    OpenAI's detection tools worked: alerts fired at 3:13 PM PST, a few minutes before customer impact began at 3:16 PM, and the root cause was identified at 3:16 PM.

    Remediation
    • Scaled clusters down to reduce the total Kubernetes API load
    • Blocked network access to the Kubernetes admin APIs so no new expensive requests arrived
    • Scaled up the Kubernetes API servers
    • Removed the telemetry service once enough control was back
    • Moved traffic from impacted clusters to healthy ones
    • Intervened manually in clusters where simultaneous resource downloads saturated resource limits
    Action items
    • Put every infrastructure configuration change through a phased rollout with continuous monitoring of both the service workloads and the clusters, including the Kubernetes control plane
    • Run fault injection tests in which the data plane runs without the control plane for longer, and tests that roll out bad changes on purpose to check detection and rollback
    • Build break-glass mechanisms so engineers can reach the Kubernetes API servers in any circumstances
    • Decouple the Kubernetes data plane from the control plane so that service discovery for critical workloads does not depend on the control plane
    • Speed up recovery with better caching and dynamic rate limiters for resources needed at cluster startup, and regular drills that rapidly replace an entire cluster
    Lessons

    Several systems and processes failed at once and interacted in unexpected ways: tests did not catch the change's effect on the Kubernetes control plane, DNS caching put a delay between the change and the failures, and being locked out of the control plane made remediation very slow. Because service discovery used Kubernetes DNS, the control plane was carrying load for product traffic even though the data plane was meant to be independent of it.

    View case
  4. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters
    OpenAI2024
    Symptoms

    From 10:15 am PT on 25 November 2024 a large share of both API and ChatGPT traffic failed with timeouts and 503 errors.

    High error rateHigh latencyControl plane unresponsive
    Trigger

    A global change to Kubernetes namespace labels caused the networking layer to recompute metadata. The change had rolled out to the small clusters without problems.

    Root cause
    Configuration change
    Component
    Kubernetes API server
    Contributing factor
    Excessive blast radiusLimited debug access
    Remediation
    • moved workloads out of the impacted clusters by hand
    • with the cloud provider, scaled up the number of control plane nodes while scaling down the number of data plane nodes, which let the control planes recover
    Action items
    • Lock namespace changes so this incident cannot recur
    • Freeze certain types of infrastructure deploys
    • Allowlist identities to prevent a recurrence of this class of incidents
    • Improve the GPU load balancing logic so traffic is re-routed automatically and more effectively when possible
    • Improve change management for fleet-wide infrastructure changes
    View case
  5. IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start query
    OpenAI2024
    Symptoms

    On August 16, 2024 from 11:38 AM to 1:16 PM PT, the OpenAI primary API was unreliable: ChatGPT conversations succeeded less often, and logging in and creating accounts were affected. The status page called it elevated error rates for ChatGPT and the Platform API. The incident came in two waves, of 44 and 15 minutes.

    High error rateDatabase overload
    Trigger

    A scheduled maintenance combined with an upgrade to the ingress of OpenAI's user-facing clusters.

    Root cause
    Breaking change in an upgrade
    Component
    Ingress controllerApplication database
    Contributing factor
    No staged rolloutAutomation amplification
    Action items
    • Mitigate the networking control plane regression and validate that control plane restarts do not interfere with the clusters' data plane
    • Change service software to start faster and remove the first-start query, relieving the persistence layer
    • Deploy configuration changes that optimise the networking control plane's effect on the clusters' ability to handle traffic
    • Remove the expensive database query from critical startup paths
    • Add monitoring and alerting for networking control plane issues
    • Introduce staged rollouts with longer soak time for infrastructure changes
    • Audit systems for other slow queries that may affect service start time
    Lessons

    A brief connectivity loss turned into a long outage because recovery itself was expensive: the services that restarted overwhelmed one backend persistence store with a heavy first-start query. Keeping costly queries off startup paths shortens recovery, and staged rollouts with longer soak time limit how many systems an infrastructure regression reaches before it is caught.

    View case
Check your cluster for this

These 52 cases involve Application database and Kubernetes API server. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause9

Configuration change11 cases

Component
Cloud load balancer2Kubernetes API server2Telemetry agent2Argo CD1Configuration templating1ConfigMaps and Secrets1Container runtime1Application database1Client-side DNS resolver and cache1etcd1HAProxy1Helm1kubelet1Application logging library1Container memory limit1Persistent volumes and stateful sets1Pod priority and preemption1Prometheus1Terraform1AWS VPC CNI1
Contributing factor
Excessive blast radius4Monitoring blind spot4Unverified assumption3Automation amplification3Limited debug access2No staged rollout2Missed in review2Unreviewed defaults1Change with a delayed effect1Cache-masked failure1No alert on limit1No configuration tests1No resource requests or limits1No runbook1No safeguard on destructive change1No headroom1Partial rollout1Retry storm1Unrepresentative staging1
Organisation
OpenAI3DFDS1GitHub1GitLab1Grafana Labs1InfluxData1MUSINSA1NU.nl1Pusher1
  1. IL-0179 With DEBUG logging and synchronous log output, a high-traffic service filled the 64 KB stdout pipe, so request threads blocked in write() while holding database connections and HikariCP ran dry, failing requests in 30-second bursts
    MUSINSA2026
  2. IL-0200 Newly provisioned Kubernetes nodes ran an incompatible etcd version and broke cluster consensus in several GitHub production clusters, causing 5XX errors on Git operations and page loads
    GitHub2026
  3. IL-0122 An invalid value in a configuration shared by several ChatGPT services put pods into crash loops and raised error rates
    OpenAI2025
  4. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
    OpenAI2024
  5. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters
    OpenAI2024
  6. IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environment
    InfluxData2022
  7. IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errors
    GitLab2022
  8. IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503
    DFDS2021
  9. IL-0004 A new default pod priority let a new Cortex cluster preempt every production ingester in a cascade, taking Grafana Cloud Hosted Prometheus down
    Grafana Labs2019
  10. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods
    Pusher2019
  11. IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memory
    NU.nl2019

Faulty deployment3 cases

Component
In-memory cache1Cloud load balancer1Application database1Ingress controller1Pod termination1
Contributing factor
Monitoring blind spot2Unverified assumption1Automation amplification1Excessive blast radius1Change during peak hours1Unreviewed defaults1No runbook1No safeguard on destructive change1Health check tied to a dependency1Retry storm1
Organisation
Microsoft Azure1OpenAI1Yonghyun Kim1
  1. IL-0136 A deployment that cleared an internal service's caches at peak traffic sent 20 times the normal load to the Azure OpenAI metadata backend in Sweden Central, and health checks that depended on its database restarted instances in a loop for almost six hours
    Microsoft Azure2026
  2. IL-0188 Rolling updates on EKS returned 502 errors for about 30 seconds because Kubernetes deleted old pods as soon as new pods passed readiness, before the ALB had marked the new pod targets healthy
    Yonghyun Kim2025
  3. IL-0121 A faulty code change to a control plane cluster meant for low-priority workloads degraded the production inference workloads it also managed
    OpenAI2025

Breaking change in an upgrade5 cases

Component
Ingress controller2Admission webhook1Calico1Application database1etcd1Kubernetes API server1Linux kernel1Node operating system image1systemd-networkd1Web application firewall rules1
Contributing factor
No staged rollout2Unrepresentative staging2Automation amplification1Untested backup1Configuration not in code1Limited debug access1Monitoring blind spot1Single point of failure1
Organisation
OpenAI2GitHub1Reddit1UK Ministry of Justice1
  1. IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applications
    UK Ministry of Justice2026
  2. IL-0120 A daily scheduled OS update restarted systemd-networkd on cloud-hosted GPU nodes, which conflicted with OpenAI's networking agent and removed all routes, cutting capacity for ChatGPT and the API
    OpenAI2025
  3. IL-0201 A framework upgrade caused kernel panics in GitHub's Kubernetes infrastructure, and GitHub.com web and API requests, mostly for issues and pull requests, failed intermittently for about 16 hours
    GitHub2025
  4. IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start query
    OpenAI2024
  5. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
    Reddit2023

Software defect12 cases

Component
Cloud load balancer3kubelet3etcd2Model request routing2Admission webhook1Automated capacity manager1Container CPU limit1Application database1Authoritative DNS servers and records1Horizontal pod autoscaler1Model inference server1Ingress controller1Kubernetes API server1Linkerd1Linux kernel1Container memory limit1Node pool1Serverless function platform1
Contributing factor
Monitoring blind spot4Automation amplification3Change with a delayed effect3Excessive blast radius2Unpatched known bug2Unknown limit2No safeguard on destructive change2No headroom2Unreviewed defaults1Ignored early signal1Harmful manual action1No alert on limit1No configuration tests1No runbook1Overlapping failures1Partial rollout1Retry storm1Ineffective rollback1Single point of failure1
Organisation
Amazon Web Services2Google Cloud2OpenAI2Airbnb1AirMap1DFDS1Grafana Labs1Monzo1Omio1
  1. IL-0124 Capacity changes queued while a circuit breaker was engaged ran all at once when it was released, taking multiple OpenAI inference engines out of service and raising API error rates
    OpenAI2026
  2. IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1
    Amazon Web Services2025
  3. IL-0131 A new kind of GenerateContent request made Vertex AI prediction API servers segfault, and once enough of them restarted at the same time the serving fleet could not carry the load
    Google Cloud2024
  4. IL-0123 A response buffer allocated anew in a loop made OpenAI's routing layer nodes hit their memory limits and fail readiness checks on a morning with more completions than any day before, causing 502 and 503 errors across all models
    OpenAI2023
  5. IL-0151 A latent defect in the AWS Lambda Frontend, set off when one us-east-1 cell scaled past a size never reached before, raised errors for Lambda and for services that depend on it, including new EKS cluster provisioning
    Amazon Web Services2023
  6. IL-0132 A faulty resource cleanup process marked Vertex AI prediction endpoints as deleted worldwide, undeploying customers' models and failing predictions for 4 hours 20 minutes
    Google Cloud2022
  7. IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404
    DFDS2021
  8. IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two pods
    Airbnb2020
  9. IL-0190 Cortex distributors kept a stuck connection to a dead etcd leader after a Kubernetes upgrade terminated its node, failing writes to Hosted Prometheus for 12 minutes
    Grafana Labs2020
  10. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts
    Omio2020
  11. IL-0074 All AirMap platform services down as worker nodes went NotReady and pods crash-looped on network and DNS failures
    AirMap2018
  12. IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outage
    Monzo2017

Capacity exhaustion12 cases

Component
Application database4Kubernetes API server4CoreDNS3Node pool3In-memory cache2etcd2Container memory limit2Pod scheduling and placement2CI system1conntrack table1Credential broker for pods1Client-side DNS resolver and cache1GKE control plane1HAProxy1Ingress controller1kube-proxy1kubelet1Linux kernel1Service discovery1Telemetry agent1
Contributing factor
Retry storm4Single point of failure4Unreviewed defaults3Unknown limit3Unverified assumption2Automation amplification2Excessive blast radius2Monitoring blind spot2No alert on limit2No capacity planning2Change during peak hours1Harmful manual action1No caching1No staged rollout1Ineffective rollback1Unrepresentative staging1
Organisation
Zalando3Civis Analytics1GitLab1Google Cloud1Honeycomb1loveholidays1Moonlight1PostHog1RebelMouse1Render1
  1. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries
    PostHog2025
  2. IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouse
    RebelMouse2025
  3. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server
    Render2024
  4. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023
  5. IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worse
    Google Cloud2022
  6. IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primary
    GitLab2022
  7. IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resets
    loveholidays2020
  8. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit
    Zalando2019
  9. IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was down
    Moonlight2019
  10. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API
    Civis Analytics2019
  11. IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queries
    Zalando2019
  12. IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifest
    Zalando2018

Certificate or credential expiry1 case

Component
Cluster autoscalerManaged Kubernetes control plane and provider API
Contributing factor
Change with a delayed effect
Organisation
Linode
  1. IL-0156 Revoking old certificates after a rotation left a certificate key mismatch between API endpoints and the cluster controller, and LKE cluster management and autoscaling in Atlanta returned 5xx errors
    Linode2025

Cloud provider outage3 cases

Component
Application database3AKS control plane1Node root disk1Persistent volumes and stateful sets1
Contributing factor
Unverified assumption1Excessive blast radius1Unpatched known bug1Overlapping failures1Health check tied to a dependency1Single point of failure1
Organisation
Grafana Labs1Microsoft Azure1OpenAI1
  1. IL-0119 ChatGPT web service pods crash-looped on failing health checks after a Cosmos DB failure, leaving too few healthy pods to serve the web layer
    OpenAI2025
  2. IL-0145 A power sag and chiller failure in one Australia East availability zone took down the AKS control plane's compute and its non-zone-redundant SQL Database operation queue, blocking AKS create, update and delete operations in Australia East and Australia Southeast
    Microsoft Azure2023
  3. IL-0193 A GCP persistent disk degradation took down a Cassandra StatefulSet whose OrderedReady policy blocked recovery, and Metrictank could not reconnect to the recreated pods, causing a 23-hour Graphite query outage
    Grafana Labs2019

DNS or network fault3 cases

Component
conntrack table2CoreDNS2kube-proxy2Cloud load balancer1Client-side DNS resolver and cache1Horizontal pod autoscaler1Ingress controller1Pod termination1Telemetry agent1
Contributing factor
Unreviewed defaults2Ignored early signal2Unverified assumption1Automation amplification1Unknown limit1No caching1
Organisation
Adevinta1Lotte E-commerce1Preply1
  1. IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNS
    Adevinta2023
  2. IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALB
    Lotte E-commerce2022
  3. IL-0049 Partial DNS outage after CoreDNS scaled down and kube-proxy failed to delete stale conntrack entries
    Preply2020

Operator error2 cases

Component
Application database1Ingress controller1Model hub1
Contributing factor
No safeguard on destructive change2No headroom1Overlapping failures1Deferred upgrades1
Organisation
GitLab1OpenAI1
  1. IL-0207 A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all traffic
    GitLab2022
  2. IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgraded
    OpenAI2022