Incident Library
Applied filters SymptomHigh error rateFactorIgnored early signalOrganisationAdevintaClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNS
    Adevinta2023
    Symptoms

    On a Friday afternoon a tenant of SCHIP, Adevinta's multi-tenant Kubernetes runtime, reported a spike of HTTP 5xx errors in some of its services. The drops in quality of service came back periodically through the following days, each one short, and were reported mostly around midnight.

    High error rateDNS resolution failuresHigh latency
    Root cause
    DNS or network fault
    Component
    CoreDNSClient-side DNS resolver and cacheTelemetry agent
    Contributing factor
    Unreviewed defaultsUnknown limitIgnored early signalUnverified assumption
    Detection

    A tenant reported a spike of HTTP 5xx errors in some of its services; the DNS Latency SLI had already been degraded for some time without being acted on.

    Remediation
    • raised the dnsmasq concurrent query limit from 150 to 1500
    • added a CoreDNS header block that sets the ra and aa flags and clears rd, so dnsmasq caches cluster.local answers
    • configured dnsmasq on each node to answer NXDOMAIN for unresolvable search-domain names instead of forwarding them to CoreDNS
    • changed the fluent-bit log router hostname to a fully qualified name
    • asked the affected tenants to use fully qualified names for internal calls
    Lessons

    The DNS Latency SLI had shown the problem long before anyone looked at it. An SLO only protects you when its SLIs are trusted and a drop is investigated. The investigation took a month partly because several on-call groups worked on it separately, without one place for observations and hypotheses, and sometimes chased hypotheses already discarded. Disprove one hypothesis before starting the next, and collect data the metrics system lacks: DNStop showed the garbage lookups that no dashboard did. With ndots:5, short service names multiply DNS traffic; a node-local cache only helps if it can cache the cluster DNS answers at all.

    View case
Check your cluster for this

This case involves CoreDNS and Client-side DNS resolver and cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

DNS or network fault1 case

Component
CoreDNSClient-side DNS resolver and cacheTelemetry agent
Contributing factor
Unverified assumptionUnreviewed defaultsIgnored early signalUnknown limit
Organisation
Adevinta
  1. IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNS
    Adevinta2023