Incident Library
Applied filters ComponentHorizontal pod autoscalerDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases9 cases

7 to 9 of 9

  1. IL-0209 A too-high HPA CPU target kept GitLab.com web pods from scaling up as traffic moved from VMs to Kubernetes, breaching the web latency SLO for 15 minutes
    GitLab2021
    Symptoms

    The web service breached its latency SLO at 07:30 UTC and its Apdex dropped: some requests to the GitLab.com web front end slowed down for about 15 minutes. This happened while web traffic was being shifted from virtual machines to the web pods on Kubernetes, and the pods did not scale up as the traffic arrived.

    High latencyAutoscaler misbehaviour
    Trigger

    Shifting web traffic from VMs to Kubernetes pods during the GitLab.com web migration.

    Root cause
    Configuration change
    Component
    Horizontal pod autoscaler
    Contributing factor
    Unverified assumption
    Detection

    The web latency SLO breach at 07:30 UTC; an engineer declared the incident in Slack at 07:40.

    Remediation
    • Lowered the target CPU utilisation of the web pods' autoscaler to a more conservative value (merge request gitlab-com/gl-infra/k8s-workloads/gitlab-com!1149)
    Action items
    • Adjust the target utilisation for the web pods (gitlab-com/gl-infra/k8s-workloads/gitlab-com!1149)
    Lessons

    When traffic is moved onto pods that rely on CPU-based autoscaling, a CPU target set too high can keep the pods from scaling as the traffic arrives; GitLab fixed it by setting a more conservative target.

    View case
  2. IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two pods
    Airbnb2020
    Symptoms

    A production service at Airbnb showed a high error rate and could not be deployed. The deploy failed with a cryptic Kubernetes error about being unable to access invalid index 5.

    High error rateThroughput degradationDeploys failing
    Trigger

    A change to the mutating admission controller, which had been rolled out within the previous month, had gone in seven days before the incident surfaced; when it had to delete two items from an array in a spec, it generated an invalid JSON patch that removed index 2 and then index 3.

    Root cause
    Software defect
    Component
    Admission webhookHorizontal pod autoscaler
    Contributing factor
    Change with a delayed effectMonitoring blind spotIgnored early signal
    Detection

    The affected service reported a high error rate and its deploy failed; the slow degradation of other services over the previous seven days had gone unnoticed.

    Remediation
    • deleted the mutating admission controller from the affected cluster
    Action items
    • Create alerts from the existing metrics that would have shown the problem earlier
    Lessons

    Keep in mind that a mutating admission controller exists in the cluster; a recently added one is easy to forget during an incident. Search for the exact error message early instead of working on the visible symptoms. Page a teammate when progress stalls.

    View case
  3. IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPA
    loveholidays2019
    Symptoms

    On the evening of a long-overdue upgrade of a loveholidays production GKE cluster from 1.10.X to 1.12.5-gke.10, planned to take 2 to 2.5 hours, the master upgrade and the first two 7-node NodePools went smoothly. After that the NodePool upgrades slowed to a crawl. Each node is cordoned, drained and upgraded in turn. Even a 14-node cluster at about 5 minutes per node took over an hour, and with about 100 more nodes to go, some of them taking over 15 minutes to drain, the sequential drains were too slow to finish at a reasonable hour.

    Upgrade stallPods PendingScale-up failureService outageAutoscaler misbehaviour
    Trigger

    A manual upgrade of the GKE masters and NodePools from 1.10.X to 1.12.5-gke.10, two versions at once, on 19 March 2019; GKE's release notes of 14 March had already said that version was no longer available for new clusters or master upgrades because masters showed elevated error rates, but it could still be selected.

    Root cause
    Cloud provider outage
    Component
    GKE control planeNode poolCluster autoscalerIngress controllerHorizontal pod autoscaler
    Contributing factor
    Unrepresentative stagingOverlapping failuresIgnored early signal
    Detection

    The slow drains, zombie nodes and stuck NodePool were seen by the engineer running the upgrade; the broken Ingress was noticed after the upgrade and explained by Google support; the HPA problem was noticed the next day.

    Remediation
    • drained the remaining NodePool nodes ahead of the upgrade with kubectl drain --ignore-daemonsets --delete-local-data
    • deleted the zombie instances from the managed instance group with gcloud compute instance-groups managed delete-instances
    • cancelled the stuck UPGRADE_NODES operation with gcloud beta container operations cancel
    • used kubectl port-forward in place of the broken internal Ingresses
    • scaled down the older of two active ReplicaSets, or triggered a dummy deployment, to bring the HPA back
    Action items
    • Alert when a HorizontalPodAutoscaler is at 70% and at 100% of its capacity
    • Alert when a deployment has had two active ReplicaSets for 30 minutes and for 60 minutes
    • Re-evaluate termination grace periods that were set to arbitrary large values
    • Read the GKE release notes before every cluster upgrade
    • Use auto-upgrade in non-production environments, and manual cluster upgrades with automatic node upgrades in production
    • Keep environments as identical as possible, including resource names
    Lessons

    Read the release notes before upgrading: the target version had been withdrawn five days earlier. Drain time depends on pod configuration, so emptyDir volumes, long grace periods and missing disruption budgets turn a NodePool upgrade into hours. Pre-production only catches problems like the ingress bug if it matches production down to resource names. Auto-upgrades in non-production surface problems early, and a mix of manual cluster upgrades and automatic node upgrades in production keeps clusters current without much effort.

    View case
Check your cluster for this

These 9 cases involve Horizontal pod autoscaler and CoreDNS. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause5

Capacity exhaustion4 cases

Component
Horizontal pod autoscaler4CoreDNS3Container CPU limit2Client-side DNS resolver and cache1Karpenter1Container memory limit1Node pool1Pod scheduling and placement1Pod termination1
Contributing factor
Automation amplification2Monitoring blind spot2Unverified assumption1Unreviewed defaults1Harmful manual action1No resource requests or limits1No headroom1Retry storm1Unrepresentative staging1
Organisation
Buildkite1Kevin1MonotaRO1이언철1
  1. IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatch
    Buildkite2026
  2. IL-0181 A traffic spike from an external community post pushed product API pods into CPU throttling under a 1-core limit, and an e-commerce product page slowed down while every pod stayed Running
    Kevin2026
  3. IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation
    이언철2024
  4. IL-0433 Shifting API traffic from EC2 to EKS on Fargate overloaded two CoreDNS pods and caused mass name-resolution errors
    MonotaRO2023

Cloud provider outage2 cases

Component
Cluster autoscaler2GKE control plane2Horizontal pod autoscaler2Cloud load balancer1Ingress controller1Node pool1
Contributing factor
Overlapping failures2Ignored early signal1Harmful manual action1No headroom1Retry storm1Unrepresentative staging1
Organisation
loveholidays1Twingate1
  1. IL-0542 Twingate authentication and authorization requests failed at higher rates after traffic moved onto a smaller GKE cluster
    Twingate2025
  2. IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPA
    loveholidays2019

DNS or network fault1 case

Component
Cloud load balancerconntrack tableHorizontal pod autoscalerIngress controllerkube-proxyPod termination
Contributing factor
Unreviewed defaults
Organisation
Lotte E-commerce
  1. IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALB
    Lotte E-commerce2022

Configuration change1 case

Component
Horizontal pod autoscaler
Contributing factor
Unverified assumption
Organisation
GitLab
  1. IL-0209 A too-high HPA CPU target kept GitLab.com web pods from scaling up as traffic moved from VMs to Kubernetes, breaching the web latency SLO for 15 minutes
    GitLab2021

Software defect1 case

Component
Admission webhookHorizontal pod autoscaler
Contributing factor
Change with a delayed effectIgnored early signalMonitoring blind spot
Organisation
Airbnb
  1. IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two pods
    Airbnb2020