Incident Library
Applied filters SymptomHigh latencyFactorRetry stormClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases8 cases

8 to 8 of 8

  1. IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worse
    Google Cloud2022
    Symptoms

    From 03:03 US/Pacific on 14 June 2022, calls to the GKE API in us-central1 through gcloud and the Google Cloud Console were slow and returned errors. Retrying sometimes succeeded.

    High error rateHigh latencyControl plane lag
    Trigger

    An unexpected increase in backend traffic to the GKE API in us-central1.

    Root cause
    Capacity exhaustion
    Component
    GKE control plane
    Contributing factor
    Retry storm
    Remediation
    • scaled up the GKE control plane to handle the increased traffic
    Lessons

    Client retries against an overloaded management API add to the overload. When the cloud provider's control plane is degraded, running workloads may be fine while cluster operations fail, and the status page offered no workaround beyond advising customers to use another region if they could.

    View case
Check your cluster for this

These 8 cases involve Application database and In-memory cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause4

Faulty deployment1 case

Component
In-memory cacheApplication database
Contributing factor
Automation amplificationChange during peak hoursMonitoring blind spotNo runbookHealth check tied to a dependencyRetry storm
Organisation
Microsoft Azure
  1. IL-0136 A deployment that cleared an internal service's caches at peak traffic sent 20 times the normal load to the Azure OpenAI metadata backend in Sweden Central, and health checks that depended on its database restarted instances in a loop for almost six hours
    Microsoft Azure2026

Configuration change2 cases

Component
In-memory cache1Cloud load balancer1conntrack table1HAProxy1Ingress controller1Container memory limit1Persistent volumes and stateful sets1
Contributing factor
Unverified assumption2Retry storm2Excessive blast radius1Change with a delayed effect1No health probes1No headroom1Single point of failure1
Organisation
AdGuard VPN1LINE+1
  1. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks
    AdGuard VPN2025
  2. IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on Nginx
    LINE+2022

Capacity exhaustion3 cases

Component
In-memory cache2Application database2GKE control plane1Node pool1Pod scheduling and placement1
Contributing factor
Retry storm3Automation amplification1No alert on limit1No staged rollout1Ineffective rollback1Single point of failure1Unrepresentative staging1
Organisation
Google Cloud1Honeycomb1PostHog1
  1. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries
    PostHog2025
  2. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023
  3. IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worse
    Google Cloud2022

Software defect2 cases

Component
Application database2Cloud load balancer1Cluster autoscaler1Authoritative DNS servers and records1Model hub1
Contributing factor
Retry storm2Automation amplification1No runbook1No headroom1Overlapping failures1
Organisation
Amazon Web Services1Hugging Face1
  1. IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1
    Amazon Web Services2025
  2. IL-0024 Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three times
    Hugging Face2024