Incident Library
Applied filters Root causeConfiguration changeComponentNode launch templateFactorSignal buried in noiseClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0365 A Terraform change to auto-scaling groups detached all worker nodes from both ALBs and cut off every cluster service
    DFDS2020
    Symptoms

    On 9 June 2020 at 12:25 CET a CloudWatch alert fired on a high count of 5XX responses from the AWS Application Load Balancers (ALBs) of DFDS's Hellman Kubernetes cluster, and at 12:28 the SRE team verified that no ingress traffic was reaching the cluster. Every service running inside the cluster was unreachable from outside it, because both ALBs, the authenticated one and the one without authentication, no longer had any Kubernetes worker nodes attached.

    Service outage
    Trigger

    Planned maintenance to split the Hellman cluster's auto-scaling groups by availability zone applied Terraform changes at 12:20 CET that removed the ALB instance attachments.

    Root cause
    Configuration change
    Component
    TerraformCloud load balancerNode launch template
    Contributing factor
    Unrepresentative stagingSignal buried in noiseMonitoring blind spot
    Detection

    A CloudWatch alert detected a high count of 5XX responses for the ALBs, and the lack of ingress traffic was then verified.

    Remediation
    • Attached the worker node instances to the ALBs by hand
    • Implemented a permanent fix within the hour
    Action items
    • Implement alerts on healthy and unhealthy ALB targets for more precise and immediate notice
    • Reduce alert noise further
    • Implement host name resolution for Goldpinger
    Lessons

    General alert noise is still too high, there was no alert on healthy and unhealthy ALB targets, and test clusters should check ingress through the ALBs.

    View case
Check your cluster for this

This case involves Cloud load balancer and Node launch template. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change1 case

Component
Cloud load balancerNode launch templateTerraform
Contributing factor
Signal buried in noiseMonitoring blind spotUnrepresentative staging
Organisation
DFDS
  1. IL-0365 A Terraform change to auto-scaling groups detached all worker nodes from both ALBs and cut off every cluster service
    DFDS2020