Incident Library
Applied filters SymptomIntermittent connection failuresFactorHarmful manual actionDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation
    이언철2024
    Symptoms

    When traffic changed sharply, CoreDNS came under heavy load, and the cluster saw network problems and a sudden rise in latency. The horizontal pod autoscaler then added pods, database connections surged, and the impact spread to the whole service.

    High latencyIntermittent connection failuresAutoscaler misbehaviour
    Trigger

    Sharp changes in traffic loaded the small number of CoreDNS pods beyond what they could handle.

    Root cause
    Capacity exhaustion
    Component
    CoreDNSKarpenterHorizontal pod autoscalerPod scheduling and placement
    Contributing factor
    Automation amplificationHarmful manual action
    Remediation
    • Set up the Cluster Proportional Autoscaler for CoreDNS with nodesPerReplica and coresPerReplica chosen for the instance types in use
    • Spread CoreDNS with soft topologySpreadConstraints (ScheduleAnyway) instead of anti-affinity so Karpenter could consolidate nodes
    • Added alert thresholds on CoreDNS error rate, panics, request duration and request rate
    • Deployed NodeLocal DNSCache so each node answers DNS from a local cache and forwards misses to the main CoreDNS
    Lessons

    A fixed, small number of CoreDNS pods cannot follow a cluster whose nodes and pods change quickly with traffic, and the damage spreads: DNS latency triggers autoscaling, which in turn floods the database with connections. Simply adding many CoreDNS replicas created a different problem, because with tainted nodes and a maxSkew: 1 spread constraint the pods piled onto a few nodes and blocked Karpenter consolidation. Scaling CoreDNS in proportion to the cluster and putting a DNS cache on every node handled both.

    View case
  2. IL-0230 kube-proxy sync failures during a Kubernetes upgrade corrupted node iptables, and CircleCI jobs stopped starting
    CircleCI2023
    Symptoms

    On 14 March 2023 at about 18:00 UTC, communication errors rose on one CircleCI service, and pipelines for all customers began to start late until jobs did not run at all. Within an hour the errors had spread to many services and to every function of the application, including the CircleCI UI and API.

    Service outageIntermittent connection failuresDNS resolution failures
    Trigger

    At about 17:40 UTC on 14 March the infrastructure team resumed the upgrade of the main production Kubernetes cluster with a changed approach for some components, after a first attempt on 13 March had been rolled back. The ongoing upgrade left kubelet and kube-proxy at mismatched versions.

    Root cause
    Breaking change in an upgrade
    Component
    kube-proxykubeletCoreDNSClient-side DNS resolver and cache
    Contributing factor
    Partial rolloutIneffective rollbackMonitoring blind spotHarmful manual action
    Detection

    At 18:00 UTC engineers saw communication errors rise on one service, then job processing delays climbing and job queues backing up across several services. The kube-proxy "Sync failed" errors were found only in the analysis after the incident.

    Remediation
    • Restarted every node in the cluster so each node built a fresh iptables ruleset
    • Restarted a separate pool of dependencies so jobs using contexts or the machine executor ran again
    • Added capacity to work through the backlog of jobs
    • Restarted the RabbitMQ nodes to fix a few corrupted queues
    Action items
    • Improve the Kubernetes upgrade process to limit the risk of future upgrades
    • Add metrics and alerts for kube-proxy sync errors
    • Evaluate ways to make the RabbitMQ infrastructure more resilient
    Lessons

    No single trigger caused the incident. Because networking underlies everything in the cluster, Kubernetes upgrades carry inherent risk, and CircleCI's risk mitigation did not hold up when several events interacted.

    View case
Check your cluster for this

These 2 cases involve CoreDNS and Client-side DNS resolver and cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Capacity exhaustion1 case

Component
CoreDNSHorizontal pod autoscalerKarpenterPod scheduling and placement
Contributing factor
Automation amplificationHarmful manual action
Organisation
이언철
  1. IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation
    이언철2024

Breaking change in an upgrade1 case

Component
CoreDNSClient-side DNS resolver and cachekube-proxykubelet
Contributing factor
Harmful manual actionMonitoring blind spotPartial rolloutIneffective rollback
Organisation
CircleCI
  1. IL-0230 kube-proxy sync failures during a Kubernetes upgrade corrupted node iptables, and CircleCI jobs stopped starting
    CircleCI2023