Incident Library
Applied filters DomainKubernetesOrganisationDFDSClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases9 cases

9 to 9 of 9

  1. IL-0365 A Terraform change to auto-scaling groups detached all worker nodes from both ALBs and cut off every cluster service
    DFDS2020
    Symptoms

    On 9 June 2020 at 12:25 CET a CloudWatch alert fired on a high count of 5XX responses from the AWS Application Load Balancers (ALBs) of DFDS's Hellman Kubernetes cluster, and at 12:28 the SRE team verified that no ingress traffic was reaching the cluster. Every service running inside the cluster was unreachable from outside it, because both ALBs, the authenticated one and the one without authentication, no longer had any Kubernetes worker nodes attached.

    Service outage
    Trigger

    Planned maintenance to split the Hellman cluster's auto-scaling groups by availability zone applied Terraform changes at 12:20 CET that removed the ALB instance attachments.

    Root cause
    Configuration change
    Component
    TerraformCloud load balancerNode launch template
    Contributing factor
    Unrepresentative stagingSignal buried in noiseMonitoring blind spot
    Detection

    A CloudWatch alert detected a high count of 5XX responses for the ALBs, and the lack of ingress traffic was then verified.

    Remediation
    • Attached the worker node instances to the ALBs by hand
    • Implemented a permanent fix within the hour
    Action items
    • Implement alerts on healthy and unhealthy ALB targets for more precise and immediate notice
    • Reduce alert noise further
    • Implement host name resolution for Goldpinger
    Lessons

    General alert noise is still too high, there was no alert on healthy and unhealthy ALB targets, and test clusters should check ingress through the ALBs.

    View case
Check your cluster for this

These 9 cases involve Cloud load balancer and Terraform. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause4

Capacity exhaustion1 case

Component
DatabaseContainer memory limitTelemetry agent
Contributing factor
Excessive blast radiusHarmful manual actionMonitoring blind spotNo resource requests or limits
Organisation
DFDS
  1. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day
    DFDS2024

Configuration change5 cases

Component
Cloud load balancer3Terraform3Node launch template2Admission webhook1Container runtime1Ingress controller1Kubernetes API server1Node pool1Telemetry agent1AWS VPC CNI1
Contributing factor
Monitoring blind spot3No configuration tests3Unverified assumption2Unrepresentative staging2Signal buried in noise1Automation amplification1Unreviewed defaults1Harmful manual action1No runbook1Missed in review1
Organisation
DFDS5
  1. IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodes
    DFDS2023
  2. IL-0111 Switching the main Kubernetes ingress controller from Traefik v1 to v2 stopped X-Forwarded headers being passed, which broke the Okta login flow for every service using Okta and stopped DFDS online bookings for over an hour
    DFDS2021
  3. IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503
    DFDS2021
  4. IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time out
    DFDS2020
  5. IL-0365 A Terraform change to auto-scaling groups detached all worker nodes from both ALBs and cut off every cluster service
    DFDS2020

Software defect2 cases

Component
Cloud load balancer1Ingress controller1kubelet1Node launch template1Linux kernel1Node operating system image1Terraform1AWS VPC CNI1
Contributing factor
Automation amplification1Excessive blast radius1Change with a delayed effect1Monitoring blind spot1No configuration tests1No health probes1No staged rollout1Overlapping failures1
Organisation
DFDS2
  1. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts
    DFDS2022
  2. IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404
    DFDS2021

Certificate or credential expiry1 case

Component
Admission webhookPolicy engine
Contributing factor
Unreviewed defaultsIgnored early signalMonitoring blind spotDeferred upgrades
Organisation
DFDS
  1. IL-0107 Kyverno stuck in a crash loop behind a webhook with failurePolicy fail blocked every change to Kubernetes Service objects for 88 minutes
    DFDS2022