Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases36 cases

21 to 25 of 36

  1. IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worse
    Google Cloud2022
    Symptoms

    From 03:03 US/Pacific on 14 June 2022, calls to the GKE API in us-central1 through gcloud and the Google Cloud Console were slow and returned errors. Retrying sometimes succeeded.

    High error rateHigh latencyControl plane lag
    Trigger

    An unexpected increase in backend traffic to the GKE API in us-central1.

    Root cause
    Capacity exhaustion
    Component
    GKE control plane
    Contributing factor
    Retry storm
    Remediation
    • scaled up the GKE control plane to handle the increased traffic
    Lessons

    Client retries against an overloaded management API add to the overload. When the cloud provider's control plane is degraded, running workloads may be fine while cluster operations fail, and the status page offered no workaround beyond advising customers to use another region if they could.

    View case
  2. IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primary
    GitLab2022
    Symptoms

    On 2022-04-25 the Sidekiq Apdex (the share of background jobs that finish within a target time) dropped, and an engineer declared an incident in Slack at 15:13 UTC. Users saw intermittent slowness across GitLab.com, and asynchronous work such as pipeline creation and import and export took unusually long.

    High latencyDatabase overloadOOMKilledHigh error rate
    Trigger

    The Consul agent in GitLab.com's GKE pods had memory demand spikes that did not fit in the pods' 500MB memory limit, and the agent was killed again and again.

    Root cause
    Capacity exhaustion
    Component
    Container memory limitService discoveryApplication database
    Contributing factor
    –
    Detection

    An engineer declared the incident in Slack at 15:13 UTC after the Sidekiq Apdex dropped; the second event was found when monitoring detected an elevated error rate at 01:21 UTC on 2022-04-26.

    Remediation
    • planned at 16:13 to raise the per-user connection limit for the gitlab database user and the pgbouncer backend pools for the main database; changes were applied at 17:26 and confirmed at 17:47
    • raised the pod memory limits from 500MB to 1GB so the Consul agents survive memory spikes
    Action items
    • Monitor the Consul agents on GKE pods to see whether the 1GB memory limit keeps them healthy
    • Investigate the Consul memory demand spikes and what can be done beyond raising the hard memory limit
    Lessons

    When the local Consul agent that Rails uses for database service discovery keeps dying, Rails does not fail; it sends every query to the last known primary. What engineers saw was database saturation and slow background jobs, which pointed the investigation first at a recent pgbouncer pool change. The memory limit of the Consul agent in every pod therefore decided the load on the primary database.

    View case
  3. IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errors
    GitLab2022
    Symptoms

    At 07:53 UTC on 2022-03-31 the rate of 50x errors on GitLab.com went up, and at 07:54 PagerDuty paged the engineer on call (EOC) about the rising error rate. Users saw slow pages and intermittent 50x error pages. The incident lists the Web, API, Git and Pages services as affected.

    High error rateHigh latencyIntermittent timeouts
    Trigger

    A k8s-workloads/gitlab-com pipeline deployed merge request !1643, a GitLab chart bump that changed the port names of Kubernetes Services and Deployments (related to the Prometheus service and pod monitors), starting with the canary environment gprd-cny at 07:49 UTC.

    Root cause
    Configuration change
    Component
    HelmCloud load balancerHAProxy
    Contributing factor
    Automation amplification
    Detection

    A PagerDuty alert on the rising error rate paged the EOC at 07:54 UTC, one minute after the 50x errors went up. The engineer deploying the change noticed the errors and linked them to it.

    Remediation
    • drained the canary environment and re-enabled it after twenty minutes
    • cancelled the deployment jobs for clusters c and d before they started
    • drained gprd-us-east1-d in HAProxy and re-enabled it five minutes later
    • raised the minimum replica count above 100 (merge request !1650)
    • drained gprd-us-east1-c in HAProxy, deployed the change to it, and re-enabled it
    • reverted the minimum replica increase (merge request !1652)
    Action items
    • Implement graceful draining of a whole regional cluster
    • Create a Chatops command that puts PagerDuty into maintenance mode
    Lessons

    Renaming ports on Kubernetes Services and Deployments behind a GCP load balancer made the load balancer reconfigure, and requests were slow or timed out until the new configuration had fully applied, which took minutes. GitLab notes that alerting worked well. The follow-up work is a way to drain a whole regional cluster gracefully and a Chatops command to put PagerDuty into maintenance mode.

    View case
  4. IL-0209 A too-high HPA CPU target kept GitLab.com web pods from scaling up as traffic moved from VMs to Kubernetes, breaching the web latency SLO for 15 minutes
    GitLab2021
    Symptoms

    The web service breached its latency SLO at 07:30 UTC and its Apdex dropped: some requests to the GitLab.com web front end slowed down for about 15 minutes. This happened while web traffic was being shifted from virtual machines to the web pods on Kubernetes, and the pods did not scale up as the traffic arrived.

    High latencyAutoscaler misbehaviour
    Trigger

    Shifting web traffic from VMs to Kubernetes pods during the GitLab.com web migration.

    Root cause
    Configuration change
    Component
    Horizontal pod autoscaler
    Contributing factor
    Unverified assumption
    Detection

    The web latency SLO breach at 07:30 UTC; an engineer declared the incident in Slack at 07:40.

    Remediation
    • Lowered the target CPU utilisation of the web pods' autoscaler to a more conservative value (merge request gitlab-com/gl-infra/k8s-workloads/gitlab-com!1149)
    Action items
    • Adjust the target utilisation for the web pods (gitlab-com/gl-infra/k8s-workloads/gitlab-com!1149)
    Lessons

    When traffic is moved onto pods that rely on CPU-based autoscaling, a CPU target set too high can keep the pods from scaling as the traffic arrives; GitLab fixed it by setting a more conservative target.

    View case
  5. IL-0205 The GKE cluster autoscaler could not add nodes to GitLab.com's production cluster because Google Cloud ran out of SSD persistent disks in us-east1
    GitLab2021
    Symptoms

    At 08:05 UTC on 2021-06-21 the API service's Apdex (the share of requests that finish within a target time) and Rails request queuing began trending the wrong way. At 09:01 the team found the API deployment unhealthy: it needed more pods for the day's growing traffic, but no new nodes could be added to give it pod capacity. Users saw slow responses; the review found no sign that error rates rose.

    Scale-up failureHigh latencyPods PendingDeploys failing
    Trigger

    Google Cloud ran out of SSD persistent disk stock in us-east1, the region GitLab.com runs in, while the API service needed more nodes for rising user traffic.

    Root cause
    Cloud provider outage
    Component
    Cluster autoscalerNode pool
    Contributing factor
    Monitoring blind spotLimited debug accessOverlapping failures
    Detection

    There was no alert on resource availability. Apdex and Rails queuing began trending the wrong way at 08:05 UTC and an engineer declared the incident in Slack at 08:43; finding the cause took a lot of log digging: cluster autoscaler errors about node pools in backoff, then ZONE_RESOURCE_POOL_EXHAUSTED at the instance-group level, and finally Google Cloud support naming SSD as the exhausted resource.

    Remediation
    • added a node pool that uses pd-standard disks
    • scaled the new node pool by hand across all three zones to relieve the API
    • brought more pd-standard node pools online for the other blocked workloads
    • stopped production deployments and opened a Google Cloud support case about the cluster autoscaler not working with the new node pools
    • changed node pools and nodeSelectors so the remaining workloads ran on pd-standard nodes
    • removed the SSD-backed node pools after a day without problems
    Action items
    • Move Kubernetes services to pd-standard disks (delivery#1837)
    • Capture resource exhaustion errors as a metric
    Lessons

    A cloud region can run out of one specific resource, here SSD persistent disks, and the only error may be a generic ZONE_RESOURCE_POOL_EXHAUSTED that does not name it. GitLab could neither monitor nor reserve resource availability, so detection took 22 hours and the cause came from Google Cloud support. The workloads ran fine on pd-standard disks. A second problem found on the way, a cluster autoscaler that does not fall back to another matching node pool when one cannot scale, slowed the move to the new node pools.

    View case
Check your cluster for this

These 36 cases involve Application database and In-memory cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause7

Faulty deployment1 case

Component
In-memory cacheApplication database
Contributing factor
Automation amplificationChange during peak hoursMonitoring blind spotNo runbookHealth check tied to a dependencyRetry storm
Organisation
Microsoft Azure
  1. IL-0136 A deployment that cleared an internal service's caches at peak traffic sent 20 times the normal load to the Azure OpenAI metadata backend in Sweden Central, and health checks that depended on its database restarted instances in a loop for almost six hours
    Microsoft Azure2026

Software defect10 cases

Component
Cloud load balancer2Container CPU limit2Application database2Ingress controller2Linux kernel2Automated capacity manager1Cluster autoscaler1Configuration templating1Credential broker for pods1Authoritative DNS servers and records1HAProxy1Model inference server1kubelet1Model hub1Node operating system image1Serverless function platform1
Contributing factor
Unverified assumption2Automation amplification2Unpatched known bug2Unknown limit2Monitoring blind spot2No headroom2Retry storm2Change with a delayed effect1Ignored early signal1No runbook1No safeguard on destructive change1Overlapping failures1Partial rollout1Unrepresentative staging1
Organisation
Amazon Web Services2Google Cloud2Buffer1Cresta1Hugging Face1Omio1OpenAI1Skyscanner1
  1. IL-0124 Capacity changes queued while a circuit breaker was engaged ran all at once when it was released, taking multiple OpenAI inference engines out of service and raising API error rates
    OpenAI2026
  2. IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1
    Amazon Web Services2025
  3. IL-0081 API requests timed out during deployment rollouts because the ingress-nginx enable-serial-reloads option held back endpoint updates while a long NGINX reload was running
    Cresta2025
  4. IL-0131 A new kind of GenerateContent request made Vertex AI prediction API servers segfault, and once enough of them restarted at the same time the serving fleet could not carry the load
    Google Cloud2024
  5. IL-0024 Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three times
    Hugging Face2024
  6. IL-0141 gke-metadata-server versions 0.4.272 to 0.4.280 flooded Cloud Logging with "Unable to sync sandbox" debug logs on GKE clusters with Workload Identity, using up logging quota and adding billable usage
    Google Cloud2023
  7. IL-0151 A latent defect in the AWS Lambda Frontend, set off when one us-east-1 cell scaled past a size never reached before, raised errors for Lambda and for services that depend on it, including new EKS cluster provisioning
    Amazon Web Services2023
  8. IL-0029 Slow user-facing services from CPU throttling on containers that used far less CPU than their limit
    Buffer2020
  9. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts
    Omio2020
  10. IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP order
    Skyscanner2019

Capacity exhaustion11 cases

Component
CoreDNS3Application database3Node pool3In-memory cache2Container CPU limit2Client-side DNS resolver and cache2Horizontal pod autoscaler2Container memory limit2Pod scheduling and placement2etcd1GKE control plane1Karpenter1Kubernetes API server1kube-dns1kube-proxy1kubelet1Language runtime settings1Prometheus1Service discovery1Telemetry agent1
Contributing factor
Unreviewed defaults3Monitoring blind spot3Retry storm3Automation amplification2Single point of failure2Unverified assumption1Change during peak hours1Unknown limit1Harmful manual action1No alert on limit1No caching1No capacity planning1No staged rollout1Ineffective rollback1Unrepresentative staging1
Organisation
Airbnb1Backpackr1Civis Analytics1GitLab1Google Cloud1Honeycomb1Kevin1PostHog1RebelMouse1Temporal1이언철1
  1. IL-0181 A traffic spike from an external community post pushed product API pods into CPU throttling under a 1-core limit, and an e-commerce product page slowed down while every pod stayed Running
    Kevin2026
  2. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries
    PostHog2025
  3. IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouse
    RebelMouse2025
  4. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down
    Backpackr2025
  5. IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation
    이언철2024
  6. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023
  7. IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percent
    Temporal2023
  8. IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worse
    Google Cloud2022
  9. IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primary
    GitLab2022
  10. IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 seconds
    Airbnb2019
  11. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API
    Civis Analytics2019

Configuration change7 cases

Component
Cloud load balancer2HAProxy2Container memory limit2In-memory cache1conntrack table1Credential broker for pods1Helm1Horizontal pod autoscaler1Ingress controller1Kubernetes API server1kubelet1Persistent volumes and stateful sets1Prometheus1
Contributing factor
Unverified assumption3Excessive blast radius3Unreviewed defaults2Retry storm2Automation amplification1Change with a delayed effect1Limited debug access1Monitoring blind spot1No alert on limit1No health probes1No resource requests or limits1No headroom1Single point of failure1
Organisation
GitLab2Adevinta1AdGuard VPN1LINE+1NU.nl1OpenAI1
  1. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks
    AdGuard VPN2025
  2. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters
    OpenAI2024
  3. IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on Nginx
    LINE+2022
  4. IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errors
    GitLab2022
  5. IL-0209 A too-high HPA CPU target kept GitLab.com web pods from scaling up as traffic moved from VMs to Kubernetes, breaching the web latency SLO for 15 minutes
    GitLab2021
  6. IL-0189 A JVM service moved from EC2 to Kubernetes answered up to ten times slower because KIAM's 15-minute credentials made the AWS Java SDK fetch new credentials on every request
    Adevinta2019
  7. IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memory
    NU.nl2019

Cloud provider outage2 cases

Component
Cluster autoscaler1Kubernetes API server1Node pool1
Contributing factor
Overlapping failures2Limited debug access1Monitoring blind spot1No capacity planning1No headroom1
Organisation
GitLab1Namespace1
  1. IL-0084 Packet drops on a bare metal provider's vSwitch forced Namespace to rebuild its fra3 region and left global capacity short, and the next day slow NVMe writes on a control plane node drove the Kubernetes API server to very high CPU
    Namespace2024
  2. IL-0205 The GKE cluster autoscaler could not add nodes to GitLab.com's production cluster because Google Cloud ran out of SSD persistent disks in us-east1
    GitLab2021

DNS or network fault4 cases

Component
conntrack table2Client-side DNS resolver and cache2Linux kernel2AWS VPC CNI2CoreDNS1Node operating system image1Telemetry agent1
Contributing factor
Unreviewed defaults4Unverified assumption2Ignored early signal1Unpatched known bug1Unknown limit1
Organisation
MindTickle2Adevinta1Prezi1
  1. IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNS
    Adevinta2023
  2. IL-0056 Calls from EKS pods to services behind an NLB in another VPC failed intermittently because the AWS VPC CNI did source NAT on the node
    MindTickle2020
  3. IL-0057 Higher p99 latency after moving a reverse proxy to EKS, from outgoing connections delayed by a SNAT conntrack race
    Prezi2019
  4. IL-0047 Pod-to-pod requests delayed by 5, 10 or 15 seconds because of DNS lookups lost to a conntrack race
    MindTickle2019

Dependency failure1 case

Component
Container runtimeCredential broker for podsService discoveryTelemetry agent
Contributing factor
Automation amplificationExcessive blast radiusUnpatched known bugSingle point of failure
Organisation
Target
  1. IL-0073 Intermittent Kafka connectivity overloaded Docker daemons, set off a node rescheduling cascade and poisoned the Consul gossip mesh
    Target2019