Cases2 cases

1 to 2 of 2

1. IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNS

   Adevinta2023

   Symptoms

   On a Friday afternoon a tenant of SCHIP, Adevinta's multi-tenant Kubernetes runtime, reported a spike of HTTP 5xx errors in some of its services. The drops in quality of service came back periodically through the following days, each one short, and were reported mostly around midnight.

   High error rateDNS resolution failuresHigh latency

   Root cause

   DNS or network fault

   Component

   CoreDNSClient-side DNS resolver and cacheTelemetry agent

   Contributing factor

   Unreviewed defaultsUnknown limitIgnored early signalUnverified assumption

   Detection

   A tenant reported a spike of HTTP 5xx errors in some of its services; the DNS Latency SLI had already been degraded for some time without being acted on.

   Remediation

   • raised the dnsmasq concurrent query limit from 150 to 1500
   • added a CoreDNS header block that sets the ra and aa flags and clears rd, so dnsmasq caches cluster.local answers
   • configured dnsmasq on each node to answer NXDOMAIN for unresolvable search-domain names instead of forwarding them to CoreDNS
   • changed the fluent-bit log router hostname to a fully qualified name
   • asked the affected tenants to use fully qualified names for internal calls

   Lessons

   The DNS Latency SLI had shown the problem long before anyone looked at it. An SLO only protects you when its SLIs are trusted and a drop is investigated. The investigation took a month partly because several on-call groups worked on it separately, without one place for observations and hypotheses, and sometimes chased hypotheses already discarded. Disprove one hypothesis before starting the next, and collect data the metrics system lacks: DNStop showed the garbage lookups that no dashboard did. With ndots:5, short service names multiply DNS traffic; a node-local cache only helps if it can cache the cluster DNS answers at all.

   View case
2. IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP order

   Skyscanner2019

   Symptoms

   On a Sunday, a team using one of Skyscanner's Kubernetes clusters manually paged the out-of-hours Kubernetes support because their service showed elevated latency, worst at the p90 and maximum levels.

   High latencyLoad imbalanceOOMKilledIntermittent connection failures

   Trigger

   Upgrading the HAProxy ingress controller to v0.6 and going back from Service virtual IPs to individual pod IPs as backends, on the understanding that the sort-backends flag introduced in v0.5, false by default, meant backends were randomly shuffled.

   Root cause

   Software defect

   Component

   Ingress controllerHAProxyConfiguration templating

   Contributing factor

   Unrepresentative stagingUnverified assumptionIgnored early signal

   Detection

   A cluster user manually paged the out-of-hours Kubernetes support about elevated latency; Prometheus queries then showed the uneven request distribution across pods.

   Remediation

   • switched the affected service from pod IPs back to the Service virtual IP through a config change in the internal deployment tool Slingshot
   • patched the controller to keep each backend's servers in an insertion-ordered slice and changed the template to iterate over that slice
   • tested the patch in sandbox clusters and rolled it out to the production clusters

   Action items

   • Run the internal controller patch in production and return to pod IPs as the default backends
   • Raise the problem upstream with a GitHub issue and pull request
   • Test changes to cluster components against the specific behaviour and conditions that matter

   Lessons

   The team already knew that sorted backends cause hot-podding and still did not test the new flag well enough to see that it did not work; changes to cluster components need tests aimed at the exact behaviour that matters, under conditions like production. A keyword can behave differently in different parts of a language: range over a map is unordered in Go code but sorted in text/template. A small cluster component can do outsized harm to every user of the cluster and, in the end, to customers.

   View case

By root cause2