Incident Library

Organisation

DFDS

DFDS · dfds

8 cases · 2020 to 2024

Root cause Configuration change4Software defect2Capacity exhaustion1Certificate or credential expiry1
Component Cloud load balancer3Terraform3Admission webhook2Ingress controller2Node launch template2Telemetry agent2AWS VPC CNI2Container runtime1Application database1Kubernetes API server1kubelet1Linux kernel1Container memory limit1Node operating system image1Node pool1Policy engine1
Contributing factor Monitoring blind spot5No configuration tests4Unverified assumption2Automation amplification2Excessive blast radius2Unreviewed defaults2Harmful manual action2Change with a delayed effect1Ignored early signal1No health probes1No resource requests or limits1No runbook1No staged rollout1Overlapping failures1Missed in review1Deferred upgrades1Unrepresentative staging1
Symptom Service outage4Container crash loop2Deploys failing2High error rate2Monitoring outage2Node NotReady2Control plane lag1Intermittent connection failures1Intermittent timeouts1OOMKilled1Probe failures1Resources deleted1Upgrade stall1

Cases

  1. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024

    Root cause Capacity exhaustion

  2. IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodesDFDS · 2023 · sev1

    Root cause Configuration change

  3. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeoutsDFDS · 2022

    Root cause Software defect

  4. IL-0107 Kyverno stuck in a crash loop behind a webhook with failurePolicy fail blocked every change to Kubernetes Service objects for 88 minutesDFDS · 2022

    Root cause Certificate or credential expiry

  5. IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404DFDS · 2021

    Root cause Software defect

  6. IL-0111 Switching the main Kubernetes ingress controller from Traefik v1 to v2 stopped X-Forwarded headers being passed, which broke the Okta login flow for every service using Okta and stopped DFDS online bookings for over an hourDFDS · 2021

    Root cause Configuration change

  7. IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503DFDS · 2021

    Root cause Configuration change

  8. IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time outDFDS · 2020

    Root cause Configuration change