Incident Library
Applied filters SymptomHigh latencyClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases36 cases

24 to 28 of 36

  1. IL-0209 A too-high HPA CPU target kept GitLab.com web pods from scaling up as traffic moved from VMs to Kubernetes, breaching the web latency SLO for 15 minutes
    GitLab2021
    Symptoms

    The web service breached its latency SLO at 07:30 UTC and its Apdex dropped: some requests to the GitLab.com web front end slowed down for about 15 minutes. This happened while web traffic was being shifted from virtual machines to the web pods on Kubernetes, and the pods did not scale up as the traffic arrived.

    High latencyAutoscaler misbehaviour
    Trigger

    Shifting web traffic from VMs to Kubernetes pods during the GitLab.com web migration.

    Root cause
    Configuration change
    Component
    Horizontal pod autoscaler
    Contributing factor
    Unverified assumption
    Detection

    The web latency SLO breach at 07:30 UTC; an engineer declared the incident in Slack at 07:40.

    Remediation
    • Lowered the target CPU utilisation of the web pods' autoscaler to a more conservative value (merge request gitlab-com/gl-infra/k8s-workloads/gitlab-com!1149)
    Action items
    • Adjust the target utilisation for the web pods (gitlab-com/gl-infra/k8s-workloads/gitlab-com!1149)
    Lessons

    When traffic is moved onto pods that rely on CPU-based autoscaling, a CPU target set too high can keep the pods from scaling as the traffic arrives; GitLab fixed it by setting a more conservative target.

    View case
  2. IL-0205 The GKE cluster autoscaler could not add nodes to GitLab.com's production cluster because Google Cloud ran out of SSD persistent disks in us-east1
    GitLab2021
    Symptoms

    At 08:05 UTC on 2021-06-21 the API service's Apdex (the share of requests that finish within a target time) and Rails request queuing began trending the wrong way. At 09:01 the team found the API deployment unhealthy: it needed more pods for the day's growing traffic, but no new nodes could be added to give it pod capacity. Users saw slow responses; the review found no sign that error rates rose.

    Scale-up failureHigh latencyPods PendingDeploys failing
    Trigger

    Google Cloud ran out of SSD persistent disk stock in us-east1, the region GitLab.com runs in, while the API service needed more nodes for rising user traffic.

    Root cause
    Cloud provider outage
    Component
    Cluster autoscalerNode pool
    Contributing factor
    Monitoring blind spotLimited debug accessOverlapping failures
    Detection

    There was no alert on resource availability. Apdex and Rails queuing began trending the wrong way at 08:05 UTC and an engineer declared the incident in Slack at 08:43; finding the cause took a lot of log digging: cluster autoscaler errors about node pools in backoff, then ZONE_RESOURCE_POOL_EXHAUSTED at the instance-group level, and finally Google Cloud support naming SSD as the exhausted resource.

    Remediation
    • added a node pool that uses pd-standard disks
    • scaled the new node pool by hand across all three zones to relieve the API
    • brought more pd-standard node pools online for the other blocked workloads
    • stopped production deployments and opened a Google Cloud support case about the cluster autoscaler not working with the new node pools
    • changed node pools and nodeSelectors so the remaining workloads ran on pd-standard nodes
    • removed the SSD-backed node pools after a day without problems
    Action items
    • Move Kubernetes services to pd-standard disks (delivery#1837)
    • Capture resource exhaustion errors as a metric
    Lessons

    A cloud region can run out of one specific resource, here SSD persistent disks, and the only error may be a generic ZONE_RESOURCE_POOL_EXHAUSTED that does not name it. GitLab could neither monitor nor reserve resource availability, so detection took 22 hours and the cause came from Google Cloud support. The workloads ran fine on pd-standard disks. A second problem found on the way, a cluster autoscaler that does not fall back to another matching node pool when one cannot scale, slowed the move to the new node pools.

    View case
  3. IL-0029 Slow user-facing services from CPU throttling on containers that used far less CPU than their limit
    Buffer2020
    Symptoms

    Services on Buffer's kops-managed cluster on AWS (about 60 nodes and 1500 containers) answered more slowly than they should have. The container throttling metric showed that many containers were being throttled whatever their CPU usage was, including containers that never came close to their CPU limit.

    CPU throttlingHigh latencyNode NotReady
    Trigger

    CPU limits on user-facing services, enforced through CFS quota by a Linux kernel with a bug that throttled containers even when they were well under their limit.

    Root cause
    Software defect
    Component
    Container CPU limitLinux kernelNode operating system image
    Contributing factor
    Unpatched known bugUnverified assumption
    Detection

    While reviewing the container throttling metric, the team noticed many containers being throttled even though their CPU usage stayed far below their limits.

    Remediation
    • removed CPU limits from all services directly or indirectly on the users' critical path
    • moved those services to dedicated tainted nodes and gave them matching tolerations
    • set each service's CPU request to its peak usage over a few months plus a margin of more than 20%, and set memory requests the same way
    Action items
    • Move nodes to a kernel that contains the CFS throttling fix (4.19 or later), which for Buffer means a kops version whose host image is Ubuntu 20.04
    • After the kernel upgrade, check whether throttling is fully gone and update the post
    Lessons

    A CPU limit can slow a service down even when the service uses a small part of it, because of how the kernel enforces CFS quota on older kernels; the throttling metric shows it, the CPU usage graph does not. Upgrading the kernel is the safer fix. Removing limits works for latency, but only together with honest CPU and memory requests, isolation of the unbounded services, and monitoring of node usage, since unbounded containers can starve the kubelet.

    View case
  4. IL-0056 Calls from EKS pods to services behind an NLB in another VPC failed intermittently because the AWS VPC CNI did source NAT on the node
    MindTickle2020
    Symptoms

    On the Saturday the production cluster was moved from kops to EKS, the platform became very slow and too unstable to keep running, and the team reverted.

    Intermittent connection failuresHigh latency
    Trigger

    Production was moved from a kops cluster using Weave networking to a new EKS cluster using the AWS VPC CNI, without external SNAT enabled.

    Root cause
    DNS or network fault
    Component
    AWS VPC CNI
    Contributing factor
    Unreviewed defaultsUnverified assumption
    Detection

    The team lead saw the platform behaving very slowly after the switch to EKS and phoned the author.

    Remediation
    • reverted production to the kops cluster
    • set AWS_VPC_K8S_CNI_EXTERNALSNAT=true in the AWS VPC CNI
    Action items
    • Monitor and alert on the AWS VPC CNI
    Lessons

    In a networking problem every component in the stack is a suspect; ruling out EKS early cost a lot of time. A DaemonSet that pinged every component from every other one exposed the anomaly that led to the fix. The team had only skimmed the AWS CNI documentation before the migration; knowing how the plugin works internally would have led to better judgements and hypotheses while debugging.

    View case
  5. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts
    Omio2020
    Symptoms

    For about six months, applications on Omio's Google Kubernetes Engine clusters stalled at random: they got stuck, stopped answering health check requests or lost network connections, and nothing the team looked at explained why.

    CPU throttlingProbe failuresHigh error rateHigh latencyIntermittent timeoutsIntermittent connection failures
    Trigger

    Multi-threaded workloads running with CPU limits on nodes whose kernel did not yet have the fix for a known CFS quota bug that causes unnecessary throttling.

    Root cause
    Software defect
    Component
    Container CPU limitLinux kernelkubelet
    Contributing factor
    Unknown limitMonitoring blind spotUnpatched known bug
    Detection

    Months of random stalls and failed health checks led the team to read the cgroup cpu.stat throttling counters inside the pods, which showed far more throttling than the limits should allow.

    Remediation
    • removed CPU limits until the kernel with the CFS bug fix was deployed across all clusters
    Action items
    • Deploy the kernel with the CFS quota bug fix across all clusters
    • Monitor utilisation, saturation and error metrics, API latency and error rates to confirm the change behaves as expected
    Lessons

    A CPU limit is a time quota per 100 ms period, not a share of cores, so a multi-threaded container can be stopped for most of each period long before it uses the cores it was given, and a kernel bug can make that worse. Throttling is invisible in standard Kubernetes metrics and has to be read from the cgroup counters. Removing limits trades away protection against a container that takes a whole node, which is only acceptable when runtimes are tuned.

    View case
Check your cluster for this

These 36 cases involve Application database and In-memory cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause7

Faulty deployment1 case

Component
In-memory cacheApplication database
Contributing factor
Automation amplificationChange during peak hoursMonitoring blind spotNo runbookHealth check tied to a dependencyRetry storm
Organisation
Microsoft Azure
  1. IL-0136 A deployment that cleared an internal service's caches at peak traffic sent 20 times the normal load to the Azure OpenAI metadata backend in Sweden Central, and health checks that depended on its database restarted instances in a loop for almost six hours
    Microsoft Azure2026

Software defect10 cases

Component
Cloud load balancer2Container CPU limit2Application database2Ingress controller2Linux kernel2Automated capacity manager1Cluster autoscaler1Configuration templating1Credential broker for pods1Authoritative DNS servers and records1HAProxy1Model inference server1kubelet1Model hub1Node operating system image1Serverless function platform1
Contributing factor
Unverified assumption2Automation amplification2Unpatched known bug2Unknown limit2Monitoring blind spot2No headroom2Retry storm2Change with a delayed effect1Ignored early signal1No runbook1No safeguard on destructive change1Overlapping failures1Partial rollout1Unrepresentative staging1
Organisation
Amazon Web Services2Google Cloud2Buffer1Cresta1Hugging Face1Omio1OpenAI1Skyscanner1
  1. IL-0124 Capacity changes queued while a circuit breaker was engaged ran all at once when it was released, taking multiple OpenAI inference engines out of service and raising API error rates
    OpenAI2026
  2. IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1
    Amazon Web Services2025
  3. IL-0081 API requests timed out during deployment rollouts because the ingress-nginx enable-serial-reloads option held back endpoint updates while a long NGINX reload was running
    Cresta2025
  4. IL-0131 A new kind of GenerateContent request made Vertex AI prediction API servers segfault, and once enough of them restarted at the same time the serving fleet could not carry the load
    Google Cloud2024
  5. IL-0024 Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three times
    Hugging Face2024
  6. IL-0141 gke-metadata-server versions 0.4.272 to 0.4.280 flooded Cloud Logging with "Unable to sync sandbox" debug logs on GKE clusters with Workload Identity, using up logging quota and adding billable usage
    Google Cloud2023
  7. IL-0151 A latent defect in the AWS Lambda Frontend, set off when one us-east-1 cell scaled past a size never reached before, raised errors for Lambda and for services that depend on it, including new EKS cluster provisioning
    Amazon Web Services2023
  8. IL-0029 Slow user-facing services from CPU throttling on containers that used far less CPU than their limit
    Buffer2020
  9. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts
    Omio2020
  10. IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP order
    Skyscanner2019

Capacity exhaustion11 cases

Component
CoreDNS3Application database3Node pool3In-memory cache2Container CPU limit2Client-side DNS resolver and cache2Horizontal pod autoscaler2Container memory limit2Pod scheduling and placement2etcd1GKE control plane1Karpenter1Kubernetes API server1kube-dns1kube-proxy1kubelet1Language runtime settings1Prometheus1Service discovery1Telemetry agent1
Contributing factor
Unreviewed defaults3Monitoring blind spot3Retry storm3Automation amplification2Single point of failure2Unverified assumption1Change during peak hours1Unknown limit1Harmful manual action1No alert on limit1No caching1No capacity planning1No staged rollout1Ineffective rollback1Unrepresentative staging1
Organisation
Airbnb1Backpackr1Civis Analytics1GitLab1Google Cloud1Honeycomb1Kevin1PostHog1RebelMouse1Temporal1이언철1
  1. IL-0181 A traffic spike from an external community post pushed product API pods into CPU throttling under a 1-core limit, and an e-commerce product page slowed down while every pod stayed Running
    Kevin2026
  2. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries
    PostHog2025
  3. IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouse
    RebelMouse2025
  4. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down
    Backpackr2025
  5. IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation
    이언철2024
  6. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023
  7. IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percent
    Temporal2023
  8. IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worse
    Google Cloud2022
  9. IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primary
    GitLab2022
  10. IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 seconds
    Airbnb2019
  11. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API
    Civis Analytics2019

Configuration change7 cases

Component
Cloud load balancer2HAProxy2Container memory limit2In-memory cache1conntrack table1Credential broker for pods1Helm1Horizontal pod autoscaler1Ingress controller1Kubernetes API server1kubelet1Persistent volumes and stateful sets1Prometheus1
Contributing factor
Unverified assumption3Excessive blast radius3Unreviewed defaults2Retry storm2Automation amplification1Change with a delayed effect1Limited debug access1Monitoring blind spot1No alert on limit1No health probes1No resource requests or limits1No headroom1Single point of failure1
Organisation
GitLab2Adevinta1AdGuard VPN1LINE+1NU.nl1OpenAI1
  1. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks
    AdGuard VPN2025
  2. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters
    OpenAI2024
  3. IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on Nginx
    LINE+2022
  4. IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errors
    GitLab2022
  5. IL-0209 A too-high HPA CPU target kept GitLab.com web pods from scaling up as traffic moved from VMs to Kubernetes, breaching the web latency SLO for 15 minutes
    GitLab2021
  6. IL-0189 A JVM service moved from EC2 to Kubernetes answered up to ten times slower because KIAM's 15-minute credentials made the AWS Java SDK fetch new credentials on every request
    Adevinta2019
  7. IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memory
    NU.nl2019

Cloud provider outage2 cases

Component
Cluster autoscaler1Kubernetes API server1Node pool1
Contributing factor
Overlapping failures2Limited debug access1Monitoring blind spot1No capacity planning1No headroom1
Organisation
GitLab1Namespace1
  1. IL-0084 Packet drops on a bare metal provider's vSwitch forced Namespace to rebuild its fra3 region and left global capacity short, and the next day slow NVMe writes on a control plane node drove the Kubernetes API server to very high CPU
    Namespace2024
  2. IL-0205 The GKE cluster autoscaler could not add nodes to GitLab.com's production cluster because Google Cloud ran out of SSD persistent disks in us-east1
    GitLab2021

DNS or network fault4 cases

Component
conntrack table2Client-side DNS resolver and cache2Linux kernel2AWS VPC CNI2CoreDNS1Node operating system image1Telemetry agent1
Contributing factor
Unreviewed defaults4Unverified assumption2Ignored early signal1Unpatched known bug1Unknown limit1
Organisation
MindTickle2Adevinta1Prezi1
  1. IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNS
    Adevinta2023
  2. IL-0056 Calls from EKS pods to services behind an NLB in another VPC failed intermittently because the AWS VPC CNI did source NAT on the node
    MindTickle2020
  3. IL-0057 Higher p99 latency after moving a reverse proxy to EKS, from outgoing connections delayed by a SNAT conntrack race
    Prezi2019
  4. IL-0047 Pod-to-pod requests delayed by 5, 10 or 15 seconds because of DNS lookups lost to a conntrack race
    MindTickle2019

Dependency failure1 case

Component
Container runtimeCredential broker for podsService discoveryTelemetry agent
Contributing factor
Automation amplificationExcessive blast radiusUnpatched known bugSingle point of failure
Organisation
Target
  1. IL-0073 Intermittent Kafka connectivity overloaded Docker daemons, set off a node rescheduling cascade and poisoned the Consul gossip mesh
    Target2019