Incident Library
Applied filters Root causeDNS or network faultFactorIgnored early signalClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNS
    Adevinta2023
    Symptoms

    On a Friday afternoon a tenant of SCHIP, Adevinta's multi-tenant Kubernetes runtime, reported a spike of HTTP 5xx errors in some of its services. The drops in quality of service came back periodically through the following days, each one short, and were reported mostly around midnight.

    High error rateDNS resolution failuresHigh latency
    Root cause
    DNS or network fault
    Component
    CoreDNSClient-side DNS resolver and cacheTelemetry agent
    Contributing factor
    Unreviewed defaultsUnknown limitIgnored early signalUnverified assumption
    Detection

    A tenant reported a spike of HTTP 5xx errors in some of its services; the DNS Latency SLI had already been degraded for some time without being acted on.

    Remediation
    • raised the dnsmasq concurrent query limit from 150 to 1500
    • added a CoreDNS header block that sets the ra and aa flags and clears rd, so dnsmasq caches cluster.local answers
    • configured dnsmasq on each node to answer NXDOMAIN for unresolvable search-domain names instead of forwarding them to CoreDNS
    • changed the fluent-bit log router hostname to a fully qualified name
    • asked the affected tenants to use fully qualified names for internal calls
    Lessons

    The DNS Latency SLI had shown the problem long before anyone looked at it. An SLO only protects you when its SLIs are trusted and a drop is investigated. The investigation took a month partly because several on-call groups worked on it separately, without one place for observations and hypotheses, and sometimes chased hypotheses already discarded. Disprove one hypothesis before starting the next, and collect data the metrics system lacks: DNStop showed the garbage lookups that no dashboard did. With ndots:5, short service names multiply DNS traffic; a node-local cache only helps if it can cache the cluster DNS answers at all.

    View case
  2. IL-0049 Partial DNS outage after CoreDNS scaled down and kube-proxy failed to delete stale conntrack entries
    Preply2020
    Symptoms

    Prometheus detected a high rate of 5xx responses on three services, called A, B and C in the write-up, and paged the on-call engineers through VictorOps at 22:18 EET. The 5xx rate was also visible in Kibana.

    DNS resolution failuresHigh error rateStale endpoint routing
    Trigger

    Because load in the cluster was low, the CoreDNS autoscaler reduced the number of CoreDNS pods from 3 to 2.

    Root cause
    DNS or network fault
    Component
    CoreDNSkube-proxyconntrack table
    Contributing factor
    Automation amplificationIgnored early signalNo caching
    Detection

    Prometheus alerted on a high 5xx rate on services A, B and C and paged on-call through VictorOps.

    Remediation
    • reverted the latest release of the affected service
    • a regular deploy created new nodes, the CoreDNS autoscaler added CoreDNS pods, and the conntrack table was rewritten
    Action items
    • Disable the autoscaler for CoreDNS
    • Run a DNS cache server in production
    • Monitor conntrack usage
    Lessons

    Scaling CoreDNS down is not free: if kube-proxy fails to clear conntrack entries for the removed pod, clients keep sending DNS queries to it. Known DNS weaknesses in a cluster should be prioritised before they cause an outage, and fixes that only treat consequences leave the underlying conntrack problem in place.

    View case
Check your cluster for this

These 2 cases involve CoreDNS and conntrack table. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

DNS or network fault2 cases

Component
CoreDNS2conntrack table1Client-side DNS resolver and cache1kube-proxy1Telemetry agent1
Contributing factor
Ignored early signal2Unverified assumption1Automation amplification1Unreviewed defaults1Unknown limit1No caching1
Organisation
Adevinta1Preply1
  1. IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNS
    Adevinta2023
  2. IL-0049 Partial DNS outage after CoreDNS scaled down and kube-proxy failed to delete stale conntrack entries
    Preply2020