Incident Library

Organisation

Adevinta

Adevinta · adevinta

3 cases · 2019 to 2024

Root cause Capacity exhaustion1Configuration change1DNS or network fault1
Component CoreDNS1Credential broker for pods1Client-side DNS resolver and cache1Karpenter1VPC subnet or pod address range1Telemetry agent1AWS VPC CNI1
Contributing factor Unreviewed defaults3Unverified assumption1Ignored early signal1Unknown limit1No capacity planning1
Symptom High latency2DNS resolution failures1High error rate1IP address exhaustion1Pods Pending1

Cases

  1. IL-0007 The AWS VPC CNI could not assign IP addresses to pods as an EKS cluster ran out of IPv4 spaceAdevinta · 2024

    Root cause Capacity exhaustion

  2. IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNSAdevinta · 2023

    Root cause DNS or network fault

  3. IL-0189 A JVM service moved from EC2 to Kubernetes answered up to ten times slower because KIAM's 15-minute credentials made the AWS Java SDK fetch new credentials on every requestAdevinta · 2019

    Root cause Configuration change