Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases21 cases

4 to 8 of 21

  1. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours
    Wealthy2025sev1
    Symptoms

    Nodes that already existed kept running, but every new node, spot or standard, on either v1.32 or v1.33, failed to initialise and could not join the cluster. Spot nodes that were preempted and recreated never came back, so services degraded progressively: background workers on spot instances, reporting services and order allocation stopped completely, and the APIs on existing nodes kept answering but could not scale for load spikes.

    Scale-up failureContainer crash loopService outageDNS resolution failures
    Trigger

    Google upgraded the GKE control plane from v1.32.x to v1.33.2 at the version deprecation deadline, although auto-upgrade was disabled on the node pools.

    Root cause
    Breaking change in an upgrade
    Component
    GKE control planeNode poolClient-side DNS resolver and cachePersistent volumes and stateful setsConfigMaps and Secrets
    Contributing factor
    Deferred upgradesSingle point of failureIneffective rollbackNo headroom
    Remediation
    • Created a new GKE cluster at v1.33, the version of the forced upgrade
    • Upgraded Istio to 1.27.1 for Kubernetes 1.33 compatibility
    • Updated the cert-manager configuration for Kubernetes 1.33
    • Migrated all production workloads to the new cluster and verified pod scheduling, DNS resolution, PVC mounting and external integrations
    • Decommissioned the broken cluster
    Action items
    • Configure all clusters to use the GKE stable release channel
    • Implement a separate management cluster for Argo CD, monitoring and other control components (Q4 2025)
    • Review spot instance usage for critical workloads and add on-demand fallback pools
    • Document the SRE upgrade procedure: management cluster first, production after a monitoring period of several days
    • Document a Kubernetes, Istio and cert-manager compatibility matrix
    • Create an automated upgrade testing pipeline
    • Reserve static IPs for quick migrations
    • Pin versions of Istio, cert-manager, CSI and DNS addons
    • Stay current with stable releases and regularly check GKE alerts and announcements to avoid deprecation deadlines
    Lessons

    GKE upgrades the control plane at a version's deprecation deadline even when auto-upgrade is disabled, so staying inside the supported version window is how to avoid a forced upgrade. A control plane upgrade can leave existing nodes healthy while every new node fails, and spot nodes that cannot rejoin after preemption turn this into cascading failures. Relying heavily on spot instances for critical workers turned a node join failure into a progressive outage. Kubernetes upgrades have to be planned together with dependent components such as Istio and cert-manager, and without a standby cluster on the same provider there was no immediate failover; the only recovery left was building a new cluster.

    View case
  2. IL-0122 An invalid value in a configuration shared by several ChatGPT services put pods into crash loops and raised error rates
    OpenAI2025
    Symptoms

    From 7:43 PM to 8:38 PM PDT on July 15, 2025, a small percentage of ChatGPT users saw more errors in their conversations.

    Container crash loopHigh error rate
    Trigger

    An internal engineer applied a configuration change that contained an invalid value.

    Root cause
    Configuration change
    Component
    –
    Contributing factor
    Excessive blast radiusNo staged rollout
    Remediation
    • reverted the configuration to the correct value
    • started a rolling restart of the affected pods to force them to resynchronise
    Action items
    • Roll out future critical configuration changes for the service to a single cluster first, with automated health guards that stop the rollout if availability drops
    • Move to a service-specific configuration value instead of one shared between services
    • Improve deployment safeguards
    View case
  3. IL-0067 Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restart
    BC Government2025sev1
    Symptoms

    The Klamm production application was completely unavailable for about 30 minutes outside working hours. Klamm's queue workers timed out because they could not reach the Redis cache, and the services that depend on it were badly affected.

    Service outageContainer crash loopPersistent volume full
    Trigger

    A large, unexpected spike in I/O from a Klamm Laravel worker sharply increased requests to the Redis cache.

    Root cause
    Capacity exhaustion
    Component
    Persistent volumes and stateful setsIn-memory cache
    Contributing factor
    No capacity planning
    Remediation
    • backed off the Redis StatefulSet and the deployment
    • deleted the corrupted PVC and rebuilt it
    • restarted the StatefulSet and the deployment
    Action items
    • Increase the Klamm Redis cache volume to 100Mi
    • Integrate Sysdig monitoring on the namespace
    • Enable replica pods on the Redis cluster
    View case
  4. IL-0103 During an EKS node AMI update, Istio sidecars crash-looped because hundreds of pods restarting at once overloaded the external certificate authority behind istiod
    Tetrate (unnamed U.S. federal customer)2025sev1
    Symptoms

    While the customer was updating the AMIs of its Amazon EKS nodes across more than 30 clusters running Istio, sidecar-injected workloads in one cluster failed to start and showed CrashLoopBackOff.

    Container crash loopProbe failures
    Trigger

    Updating the EKS node AMIs, which started hundreds of sidecar-injected pods at the same time.

    Root cause
    Dependency failure
    Component
    IstioAdmission webhookNode operating system image
    Contributing factor
    Unreviewed defaultsUnknown limit
    Detection

    Sidecar-injected workloads in the updated cluster failed to start and showed CrashLoopBackOff; the customer raised a Severity 1 ticket and a Tetrate engineer was reviewing logs within six minutes.

    Remediation
    • scaled istiod down temporarily from 40 replicas to 1 to slow Istio's internal processes and ease the load on the CA
    • deleted the CertificateSigningRequests stuck in the Approved, Failed state
    • restarted the pods gradually in a controlled order instead of all at once
    Lessons

    When a mesh hands certificate signing to an external CA, every pod start becomes a signing request to that CA, so a mass restart such as a node image rollout can overload it. Sidecars then abort under istiod's 10-second default CA timeout even though the CSRs are eventually signed. Restarting pods in controlled batches keeps the request rate within what the CA can handle. The author's point is that diagnosing such a failure means looking at the whole service mesh ecosystem, including its external dependencies, not only Istio.

    View case
  5. IL-0119 ChatGPT web service pods crash-looped on failing health checks after a Cosmos DB failure, leaving too few healthy pods to serve the web layer
    OpenAI2025
    Symptoms

    From 3:30am to 7:08am PT, chatgpt.com and a small number of OpenAI APIs returned elevated error rates. The status page classed the incident as a full outage.

    High error rateContainer crash loopProbe failures
    Trigger

    A failure of Cosmos DB in ChatGPT, at OpenAI's service provider.

    Root cause
    Cloud provider outage
    Component
    Database
    Contributing factor
    Health check tied to a dependency
    Remediation
    • the service provider recovered the affected Cosmos DB databases at 4:25am PT
    • fixed the web server's health checking logic, which ended the pod crash loop
    Action items
    • Change the web systems to degrade more gracefully when they fail, including concurrency control and better load shedding
    • Make the database layer more resilient to service provider outages (described as already underway)
    Lessons

    The database came back at 4:25am PT, yet users saw errors until 7:08am PT: the web tier's own health checks kept removing pods after the dependency had recovered. The planned changes target the web tier's behaviour under failure, with concurrency control and load shedding, rather than the database alone.

    View case
Check your cluster for this

These 21 cases involve Database and In-memory cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause8

Certificate or credential expiry3 cases

Component
Admission webhook1Calico1cert-manager1etcd1Kubernetes API server1kubelet1Policy engine1Secrets store1
Contributing factor
Unreviewed defaults2Monitoring blind spot2Unverified assumption1Ignored early signal1Harmful manual action1Single point of failure1Deferred upgrades1
Organisation
ckck8031DFDS1GoCardless1
  1. IL-0183 kubeadm's one-year leaf certificates expired, so kube-apiserver could not reach etcd, restarted in a loop and left the cluster unreachable
    ckck8032026
  2. IL-0107 Kyverno stuck in a crash loop behind a webhook with failurePolicy fail blocked every change to Kubernetes Service objects for 88 minutes
    DFDS2022
  3. IL-0460 Vault served an expired TLS certificate and almost every GoCardless service pod crash-looped, a 22-minute full outage
    GoCardless2020

Capacity exhaustion8 cases

Component
In-memory cache3Database3Ingress controller2Container memory limit2Calico1Helm1Jobs and CronJobs1JupyterHub1kubelet1Linux kernel1Node pool1Persistent volumes and stateful sets1Pod scheduling and placement1Prometheus1Telemetry agent1
Contributing factor
Harmful manual action3No resource requests or limits3No alert on limit2No capacity planning2Retry storm2Single point of failure2Automation amplification1Excessive blast radius1Change with a delayed effect1Monitoring blind spot1No runbook1No staged rollout1Ineffective rollback1Unrepresentative staging1
Organisation
BC Government1Binder Project1DFDS1Honeycomb1LINE1PostHog1UK Ministry of Justice1Wikimedia Foundation1
  1. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries
    PostHog2025
  2. IL-0067 Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restart
    BC Government2025
  3. IL-0232 Calico pods in the codfw WikiKube cluster hit their resource limits and crash-looped, taking nodes out one by one
    Wikimedia Foundation2024
  4. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day
    DFDS2024
  5. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023
  6. IL-0093 Ingress controller pods crash-looped one after another because the leader pod had exhausted its shared memory
    UK Ministry of Justice2020
  7. IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999
    LINE2020
  8. IL-0503 mybinder.org launches failed after a GKE node ran out of memory and its nginx-ingress proxy pod crash-looped
    Binder Project2017

Dependency failure2 cases

Component
Admission webhook1Liveness and readiness probes1Istio1Node operating system image1
Contributing factor
Unreviewed defaults2Automation amplification1Excessive blast radius1Unknown limit1
Organisation
Tetrate (unnamed U.S. federal customer)1Yonghyun Kim1
  1. IL-0187 A brief Spring Cloud Config Server downtime failed every client application's liveness probe, because the actuator health check included the config server, and Kubernetes restarted the pods in a kill loop
    Yonghyun Kim2025
  2. IL-0103 During an EKS node AMI update, Istio sidecars crash-looped because hundreds of pods restarting at once overloaded the external certificate authority behind istiod
    Tetrate (unnamed U.S. federal customer)2025

Breaking change in an upgrade2 cases

Component
ConfigMaps and Secrets1Client-side DNS resolver and cache1GKE control plane1Karpenter1Node operating system image1Node pool1Persistent volumes and stateful sets1
Contributing factor
No headroom1Ineffective rollback1Single point of failure1Deferred upgrades1Unpinned version1
Organisation
PortSwigger1Wealthy1
  1. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours
    Wealthy2025
  2. IL-0404 All BSEE SaaS customer pods on EKS crash-looped after Karpenter moved nodes to Bottlerocket v1.26.0, which broke Java
    PortSwigger2024

Configuration change1 case

Component
–
Contributing factor
Excessive blast radiusNo staged rollout
Organisation
OpenAI
  1. IL-0122 An invalid value in a configuration shared by several ChatGPT services put pods into crash loops and raised error rates
    OpenAI2025

Cloud provider outage1 case

Component
Database
Contributing factor
Health check tied to a dependency
Organisation
OpenAI
  1. IL-0119 ChatGPT web service pods crash-looped on failing health checks after a Cosmos DB failure, leaving too few healthy pods to serve the web layer
    OpenAI2025

Faulty deployment2 cases

Component
CI system1ConfigMaps and Secrets1Helm1Node pool1Progressive delivery controller1
Contributing factor
Unverified assumption1Automation amplification1Change started before off-hours1Change with a delayed effect1No alert on limit1Partial rollout1Single point of failure1
Organisation
Gunosy1Kubernetes project1
  1. IL-0437 Old pods from an unfinished Argo Rollouts canary could not start with a Helm-updated ConfigMap and died in a cascade
    Gunosy2023
  2. IL-0467 Boskos crash-looped after a node drain started it from a buggy StatefulSet image, failing Kubernetes PR jobs for 9 hours
    Kubernetes project2019

Software defect2 cases

Component
kubelet1Telemetry agent1
Contributing factor
Unpatched known bug1Limited debug access1
Organisation
AirMap1Hyperconnect1
  1. IL-0174 After an upgrade to Fluent Bit 2.0, logging sidecars that used the Kafka output plugin kept restarting with SIGSEGV because the plugin read a corrupted record as a map without checking its type
    Hyperconnect2023
  2. IL-0074 All AirMap platform services down as worker nodes went NotReady and pods crash-looped on network and DNS failures
    AirMap2018