Incident Library
Applied filters Root causeDNS or network faultSymptomHigh error rateFactorAutomation amplificationClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0049 Partial DNS outage after CoreDNS scaled down and kube-proxy failed to delete stale conntrack entries
    Preply2020
    Symptoms

    Prometheus detected a high rate of 5xx responses on three services, called A, B and C in the write-up, and paged the on-call engineers through VictorOps at 22:18 EET. The 5xx rate was also visible in Kibana.

    DNS resolution failuresHigh error rateStale endpoint routing
    Trigger

    Because load in the cluster was low, the CoreDNS autoscaler reduced the number of CoreDNS pods from 3 to 2.

    Root cause
    DNS or network fault
    Component
    CoreDNSkube-proxyconntrack table
    Contributing factor
    Automation amplificationIgnored early signalNo caching
    Detection

    Prometheus alerted on a high 5xx rate on services A, B and C and paged on-call through VictorOps.

    Remediation
    • reverted the latest release of the affected service
    • a regular deploy created new nodes, the CoreDNS autoscaler added CoreDNS pods, and the conntrack table was rewritten
    Action items
    • Disable the autoscaler for CoreDNS
    • Run a DNS cache server in production
    • Monitor conntrack usage
    Lessons

    Scaling CoreDNS down is not free: if kube-proxy fails to clear conntrack entries for the removed pod, clients keep sending DNS queries to it. Known DNS weaknesses in a cluster should be prioritised before they cause an outage, and fixes that only treat consequences leave the underlying conntrack problem in place.

    View case
Check your cluster for this

This case involves conntrack table and CoreDNS. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

DNS or network fault1 case

Component
conntrack tableCoreDNSkube-proxy
Contributing factor
Automation amplificationIgnored early signalNo caching
Organisation
Preply
  1. IL-0049 Partial DNS outage after CoreDNS scaled down and kube-proxy failed to delete stale conntrack entries
    Preply2020