Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases52 cases

10 to 14 of 52

  1. IL-0122 An invalid value in a configuration shared by several ChatGPT services put pods into crash loops and raised error rates
    OpenAI2025
    Symptoms

    From 7:43 PM to 8:38 PM PDT on July 15, 2025, a small percentage of ChatGPT users saw more errors in their conversations.

    Container crash loopHigh error rate
    Trigger

    An internal engineer applied a configuration change that contained an invalid value.

    Root cause
    Configuration change
    Component
    –
    Contributing factor
    Excessive blast radiusNo staged rollout
    Remediation
    • reverted the configuration to the correct value
    • started a rolling restart of the affected pods to force them to resynchronise
    Action items
    • Roll out future critical configuration changes for the service to a single cluster first, with automated health guards that stop the rollout if availability drops
    • Move to a service-specific configuration value instead of one shared between services
    • Improve deployment safeguards
    View case
  2. IL-0121 A faulty code change to a control plane cluster meant for low-priority workloads degraded the production inference workloads it also managed
    OpenAI2025
    Symptoms

    Between 2:10 PM and 3:05 PM PDT on July 9, 2025, success rates of the gpt-4.1-mini model in the API dropped for about 20 minutes, and ChatGPT search returned many more errors than usual for about 35 minutes.

    High error rate
    Trigger

    A faulty code change was deployed to a control plane cluster whose main job is managing low-priority workloads.

    Root cause
    Faulty deployment
    Component
    –
    Contributing factor
    Excessive blast radiusNo safeguard on destructive change
    Remediation
    • rolled back the faulty code change
    Action items
    • Add safeguards that prevent production inference workloads from being managed by the low-priority control plane cluster
    • Strengthen testing for deploying high-impact control plane changes
    • Move all existing production inference workloads under the production control plane cluster
    • Build a circuit breaker that stops large-scale unintended changes
    View case
  3. IL-0120 A daily scheduled OS update restarted systemd-networkd on cloud-hosted GPU nodes, which conflicted with OpenAI's networking agent and removed all routes, cutting capacity for ChatGPT and the API
    OpenAI2025
    Symptoms

    At 11:36 PM PDT on 9 June, alerts fired across multiple OpenAI services as a large number of cloud-hosted GPU nodes lost network connectivity, and the engineering team was escalated to immediately. Losing those nodes cut the capacity available to serve ChatGPT and the API.

    High error rateGPU unavailable
    Trigger

    A routine daily scheduled update of the host operating system on the cloud-hosted GPU servers restarted systemd-networkd on the affected nodes.

    Root cause
    Breaking change in an upgrade
    Component
    systemd-networkdNode operating system image
    Contributing factor
    No staged rolloutLimited debug access
    Detection

    Alerts fired immediately across multiple services when the infrastructure began losing network connectivity at 11:36 PM PDT, and the engineering team was escalated to at once.

    Remediation
    • re-imaged the affected GPU virtual machines
    • deployed recovery automation to restart the remaining impacted nodes
    • halted the background update mechanisms on the GPU VMs
    Action items
    • Disable automatic daily updates on GPU VMs so updates run in a controlled, scheduled way
    • Update system configurations to prevent conflicts between systemd-networkd and the networking service
    • Audit VM configurations across the fleet to find and reduce similar risks
    • Improve recovery speed for critical infrastructure such as GPU VMs and clusters
    • Run regular disaster recovery drills
    Lessons

    An unattended operating system update that restarts systemd-networkd can wipe the routes another networking agent manages, and it hits every node that takes the update at about the same time. Without break-glass tooling to restore connectivity, nodes that lost their network had to be re-imaged, which stretched recovery over many hours.

    View case
  4. IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouse
    RebelMouse2025
    Symptoms

    At 13:42 UTC on 19 May 2025, RebelMouse's monitoring tools reported a higher error rate and degraded performance across services. A routine reroll of the EKS nodes had started 27 minutes earlier, at 13:15 UTC.

    DNS resolution failuresHigh latencyHigh error rateService outage
    Trigger

    At 13:15 UTC RebelMouse started its regular reroll of EKS nodes to apply security patches. The planned, automatic termination of an EC2 instance terminated one of the CoreDNS pods.

    Root cause
    Capacity exhaustion
    Component
    CoreDNSNode pool
    Contributing factor
    Change during peak hours
    Detection

    Monitoring tools detected an increased error rate and degraded service performance at 13:42 UTC, and the tech team started investigating at 13:44 UTC.

    Remediation
    • rescheduled the affected CoreDNS pod onto a new EC2 instance
    • scaled up the CoreDNS deployment to more replicas so DNS load was spread evenly across all pods
    Action items
    • Add scale-in protection so CoreDNS pods are not terminated unexpectedly during peak load
    • Review the CoreDNS configuration so that vertical scaling of the pods works
    • Hold regular capacity reviews so the CoreDNS deployment is provisioned for peak load
    View case
  5. IL-0201 A framework upgrade caused kernel panics in GitHub's Kubernetes infrastructure, and GitHub.com web and API requests, mostly for issues and pull requests, failed intermittently for about 16 hours
    GitHub2025
    Symptoms

    From 18:05 UTC on March 17, 2025, GitHub.com web and API requests failed now and then for a small percentage of users, mostly requests about issues and pull requests. At 18:39 UTC the status page reported degraded performance for Issues.

    High error rateIntermittent timeouts
    Trigger

    A framework upgrade rolled out to GitHub's Kubernetes infrastructure caused kernel panics.

    Root cause
    Breaking change in an upgrade
    Component
    Linux kernel
    Contributing factor
    –
    Detection

    The status page opened the incident at 18:39 UTC, 34 minutes after the failures began, as an investigation of reports of degraded performance for Issues.

    Remediation
    • Downgraded the framework upgrade that caused the kernel panics
    • Disabled the problematic feature of the framework
    Action items
    • Investigate why the upgrade caused the kernel panics
    • Take steps to temporarily prevent the problem from coming back
    • Work out longer-term patch plans
    • Improve observability so that similar classes of problems can be reacted to quickly
    View case
Check your cluster for this

These 52 cases involve Application database and Kubernetes API server. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause9

Configuration change11 cases

Component
Cloud load balancer2Kubernetes API server2Telemetry agent2Argo CD1Configuration templating1ConfigMaps and Secrets1Container runtime1Application database1Client-side DNS resolver and cache1etcd1HAProxy1Helm1kubelet1Application logging library1Container memory limit1Persistent volumes and stateful sets1Pod priority and preemption1Prometheus1Terraform1AWS VPC CNI1
Contributing factor
Excessive blast radius4Monitoring blind spot4Unverified assumption3Automation amplification3Limited debug access2No staged rollout2Missed in review2Unreviewed defaults1Change with a delayed effect1Cache-masked failure1No alert on limit1No configuration tests1No resource requests or limits1No runbook1No safeguard on destructive change1No headroom1Partial rollout1Retry storm1Unrepresentative staging1
Organisation
OpenAI3DFDS1GitHub1GitLab1Grafana Labs1InfluxData1MUSINSA1NU.nl1Pusher1
  1. IL-0179 With DEBUG logging and synchronous log output, a high-traffic service filled the 64 KB stdout pipe, so request threads blocked in write() while holding database connections and HikariCP ran dry, failing requests in 30-second bursts
    MUSINSA2026
  2. IL-0200 Newly provisioned Kubernetes nodes ran an incompatible etcd version and broke cluster consensus in several GitHub production clusters, causing 5XX errors on Git operations and page loads
    GitHub2026
  3. IL-0122 An invalid value in a configuration shared by several ChatGPT services put pods into crash loops and raised error rates
    OpenAI2025
  4. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
    OpenAI2024
  5. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters
    OpenAI2024
  6. IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environment
    InfluxData2022
  7. IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errors
    GitLab2022
  8. IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503
    DFDS2021
  9. IL-0004 A new default pod priority let a new Cortex cluster preempt every production ingester in a cascade, taking Grafana Cloud Hosted Prometheus down
    Grafana Labs2019
  10. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods
    Pusher2019
  11. IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memory
    NU.nl2019

Faulty deployment3 cases

Component
In-memory cache1Cloud load balancer1Application database1Ingress controller1Pod termination1
Contributing factor
Monitoring blind spot2Unverified assumption1Automation amplification1Excessive blast radius1Change during peak hours1Unreviewed defaults1No runbook1No safeguard on destructive change1Health check tied to a dependency1Retry storm1
Organisation
Microsoft Azure1OpenAI1Yonghyun Kim1
  1. IL-0136 A deployment that cleared an internal service's caches at peak traffic sent 20 times the normal load to the Azure OpenAI metadata backend in Sweden Central, and health checks that depended on its database restarted instances in a loop for almost six hours
    Microsoft Azure2026
  2. IL-0188 Rolling updates on EKS returned 502 errors for about 30 seconds because Kubernetes deleted old pods as soon as new pods passed readiness, before the ALB had marked the new pod targets healthy
    Yonghyun Kim2025
  3. IL-0121 A faulty code change to a control plane cluster meant for low-priority workloads degraded the production inference workloads it also managed
    OpenAI2025

Breaking change in an upgrade5 cases

Component
Ingress controller2Admission webhook1Calico1Application database1etcd1Kubernetes API server1Linux kernel1Node operating system image1systemd-networkd1Web application firewall rules1
Contributing factor
No staged rollout2Unrepresentative staging2Automation amplification1Untested backup1Configuration not in code1Limited debug access1Monitoring blind spot1Single point of failure1
Organisation
OpenAI2GitHub1Reddit1UK Ministry of Justice1
  1. IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applications
    UK Ministry of Justice2026
  2. IL-0120 A daily scheduled OS update restarted systemd-networkd on cloud-hosted GPU nodes, which conflicted with OpenAI's networking agent and removed all routes, cutting capacity for ChatGPT and the API
    OpenAI2025
  3. IL-0201 A framework upgrade caused kernel panics in GitHub's Kubernetes infrastructure, and GitHub.com web and API requests, mostly for issues and pull requests, failed intermittently for about 16 hours
    GitHub2025
  4. IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start query
    OpenAI2024
  5. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
    Reddit2023

Software defect12 cases

Component
Cloud load balancer3kubelet3etcd2Model request routing2Admission webhook1Automated capacity manager1Container CPU limit1Application database1Authoritative DNS servers and records1Horizontal pod autoscaler1Model inference server1Ingress controller1Kubernetes API server1Linkerd1Linux kernel1Container memory limit1Node pool1Serverless function platform1
Contributing factor
Monitoring blind spot4Automation amplification3Change with a delayed effect3Excessive blast radius2Unpatched known bug2Unknown limit2No safeguard on destructive change2No headroom2Unreviewed defaults1Ignored early signal1Harmful manual action1No alert on limit1No configuration tests1No runbook1Overlapping failures1Partial rollout1Retry storm1Ineffective rollback1Single point of failure1
Organisation
Amazon Web Services2Google Cloud2OpenAI2Airbnb1AirMap1DFDS1Grafana Labs1Monzo1Omio1
  1. IL-0124 Capacity changes queued while a circuit breaker was engaged ran all at once when it was released, taking multiple OpenAI inference engines out of service and raising API error rates
    OpenAI2026
  2. IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1
    Amazon Web Services2025
  3. IL-0131 A new kind of GenerateContent request made Vertex AI prediction API servers segfault, and once enough of them restarted at the same time the serving fleet could not carry the load
    Google Cloud2024
  4. IL-0123 A response buffer allocated anew in a loop made OpenAI's routing layer nodes hit their memory limits and fail readiness checks on a morning with more completions than any day before, causing 502 and 503 errors across all models
    OpenAI2023
  5. IL-0151 A latent defect in the AWS Lambda Frontend, set off when one us-east-1 cell scaled past a size never reached before, raised errors for Lambda and for services that depend on it, including new EKS cluster provisioning
    Amazon Web Services2023
  6. IL-0132 A faulty resource cleanup process marked Vertex AI prediction endpoints as deleted worldwide, undeploying customers' models and failing predictions for 4 hours 20 minutes
    Google Cloud2022
  7. IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404
    DFDS2021
  8. IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two pods
    Airbnb2020
  9. IL-0190 Cortex distributors kept a stuck connection to a dead etcd leader after a Kubernetes upgrade terminated its node, failing writes to Hosted Prometheus for 12 minutes
    Grafana Labs2020
  10. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts
    Omio2020
  11. IL-0074 All AirMap platform services down as worker nodes went NotReady and pods crash-looped on network and DNS failures
    AirMap2018
  12. IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outage
    Monzo2017

Capacity exhaustion12 cases

Component
Application database4Kubernetes API server4CoreDNS3Node pool3In-memory cache2etcd2Container memory limit2Pod scheduling and placement2CI system1conntrack table1Credential broker for pods1Client-side DNS resolver and cache1GKE control plane1HAProxy1Ingress controller1kube-proxy1kubelet1Linux kernel1Service discovery1Telemetry agent1
Contributing factor
Retry storm4Single point of failure4Unreviewed defaults3Unknown limit3Unverified assumption2Automation amplification2Excessive blast radius2Monitoring blind spot2No alert on limit2No capacity planning2Change during peak hours1Harmful manual action1No caching1No staged rollout1Ineffective rollback1Unrepresentative staging1
Organisation
Zalando3Civis Analytics1GitLab1Google Cloud1Honeycomb1loveholidays1Moonlight1PostHog1RebelMouse1Render1
  1. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries
    PostHog2025
  2. IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouse
    RebelMouse2025
  3. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server
    Render2024
  4. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023
  5. IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worse
    Google Cloud2022
  6. IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primary
    GitLab2022
  7. IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resets
    loveholidays2020
  8. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit
    Zalando2019
  9. IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was down
    Moonlight2019
  10. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API
    Civis Analytics2019
  11. IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queries
    Zalando2019
  12. IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifest
    Zalando2018

Certificate or credential expiry1 case

Component
Cluster autoscalerManaged Kubernetes control plane and provider API
Contributing factor
Change with a delayed effect
Organisation
Linode
  1. IL-0156 Revoking old certificates after a rotation left a certificate key mismatch between API endpoints and the cluster controller, and LKE cluster management and autoscaling in Atlanta returned 5xx errors
    Linode2025

Cloud provider outage3 cases

Component
Application database3AKS control plane1Node root disk1Persistent volumes and stateful sets1
Contributing factor
Unverified assumption1Excessive blast radius1Unpatched known bug1Overlapping failures1Health check tied to a dependency1Single point of failure1
Organisation
Grafana Labs1Microsoft Azure1OpenAI1
  1. IL-0119 ChatGPT web service pods crash-looped on failing health checks after a Cosmos DB failure, leaving too few healthy pods to serve the web layer
    OpenAI2025
  2. IL-0145 A power sag and chiller failure in one Australia East availability zone took down the AKS control plane's compute and its non-zone-redundant SQL Database operation queue, blocking AKS create, update and delete operations in Australia East and Australia Southeast
    Microsoft Azure2023
  3. IL-0193 A GCP persistent disk degradation took down a Cassandra StatefulSet whose OrderedReady policy blocked recovery, and Metrictank could not reconnect to the recreated pods, causing a 23-hour Graphite query outage
    Grafana Labs2019

DNS or network fault3 cases

Component
conntrack table2CoreDNS2kube-proxy2Cloud load balancer1Client-side DNS resolver and cache1Horizontal pod autoscaler1Ingress controller1Pod termination1Telemetry agent1
Contributing factor
Unreviewed defaults2Ignored early signal2Unverified assumption1Automation amplification1Unknown limit1No caching1
Organisation
Adevinta1Lotte E-commerce1Preply1
  1. IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNS
    Adevinta2023
  2. IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALB
    Lotte E-commerce2022
  3. IL-0049 Partial DNS outage after CoreDNS scaled down and kube-proxy failed to delete stale conntrack entries
    Preply2020

Operator error2 cases

Component
Application database1Ingress controller1Model hub1
Contributing factor
No safeguard on destructive change2No headroom1Overlapping failures1Deferred upgrades1
Organisation
GitLab1OpenAI1
  1. IL-0207 A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all traffic
    GitLab2022
  2. IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgraded
    OpenAI2022