Incident Library
Applied filters Root causeCapacity exhaustionSymptomHigh latencyFactorAutomation amplificationClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation
    이언철2024
    Symptoms

    When traffic changed sharply, CoreDNS came under heavy load, and the cluster saw network problems and a sudden rise in latency. The horizontal pod autoscaler then added pods, database connections surged, and the impact spread to the whole service.

    High latencyIntermittent connection failuresAutoscaler misbehaviour
    Trigger

    Sharp changes in traffic loaded the small number of CoreDNS pods beyond what they could handle.

    Root cause
    Capacity exhaustion
    Component
    CoreDNSKarpenterHorizontal pod autoscalerPod scheduling and placement
    Contributing factor
    Automation amplificationHarmful manual action
    Remediation
    • Set up the Cluster Proportional Autoscaler for CoreDNS with nodesPerReplica and coresPerReplica chosen for the instance types in use
    • Spread CoreDNS with soft topologySpreadConstraints (ScheduleAnyway) instead of anti-affinity so Karpenter could consolidate nodes
    • Added alert thresholds on CoreDNS error rate, panics, request duration and request rate
    • Deployed NodeLocal DNSCache so each node answers DNS from a local cache and forwards misses to the main CoreDNS
    Lessons

    A fixed, small number of CoreDNS pods cannot follow a cluster whose nodes and pods change quickly with traffic, and the damage spreads: DNS latency triggers autoscaling, which in turn floods the database with connections. Simply adding many CoreDNS replicas created a different problem, because with tainted nodes and a maxSkew: 1 spread constraint the pods piled onto a few nodes and blocked Karpenter consolidation. Scaling CoreDNS in proportion to the cluster and putting a DNS cache on every node handled both.

    View case
  2. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023sev1
    Symptoms

    Standard alerting paged Honeycomb's on-call engineers about the performance and stability of event ingestion, and the automated load-shedding mechanisms tripped and sent further notifications. The status page described a higher than normal error rate and response time on the ingestion service.

    High error rateHigh latencyContainer crash loopService outageDatabase overload
    Trigger

    An abnormally large number of queries arrived from many separate connections within a few minutes.

    Root cause
    Capacity exhaustion
    Component
    Application databaseIn-memory cache
    Contributing factor
    Retry stormAutomation amplification
    Detection

    Standard alerting notified the on-call engineers about ingestion performance and stability, and the automated load-shedding mechanisms added notifications when they tripped. The first status post, about alerts on a higher than normal error rate and response time, went up at 18:28 PST (02:28 UTC), 13 minutes after the 02:15 UTC start.

    Remediation
    • Scaled the ingest fleet up manually and aggressively
    • Cut off most ingest for a few minutes so the ingest hosts could come back with enough capacity
    • Temporarily lowered the rate limits for the most aggressive traffic sources until their traffic subsided
    Action items
    • Keep the ingest fleet scaled up to prevent a repeat while longer-term fixes are worked on
    • Decide on the projects that will address these failure paths
    Lessons

    Load shedding and auto-scaling were in place, yet a burst that arrived faster than the cache could spread its writes still took the ingest hosts down. During recovery, Kubernetes' crash loop back-off worked against the operators: the hosts that had failed stayed offline while the fleet needed every host. Clients that retry whole OTel batches after a partial drop can multiply traffic many times over, so per-source rate limits were what finally stabilised ingest.

    View case
Check your cluster for this

These 2 cases involve In-memory cache and CoreDNS. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion2 cases

Component
In-memory cache1CoreDNS1Application database1Horizontal pod autoscaler1Karpenter1Pod scheduling and placement1
Contributing factor
Automation amplification2Harmful manual action1Retry storm1
Organisation
Honeycomb1이언철1
  1. IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation
    이언철2024
  2. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023