Incident Library
Applied filters Root causeSoftware defectComponentIngress controllerOrganisationDFDSClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404
    DFDS2021
    Symptoms

    At 08:00 UTC on 2021-09-06 every service in DFDS's Kubernetes cluster answered with 404. Investigation started five minutes later.

    Service outageHigh error rateProbe failures
    Trigger

    The automated Kubernetes node rollover on 2021-09-06 restarted all the Traefik v2 pods, five days after a malformed IngressRoute manifest had been applied.

    Root cause
    Software defect
    Component
    Ingress controllerCloud load balancer
    Contributing factor
    Change with a delayed effectNo configuration testsExcessive blast radius
    Detection

    All services returned 404 from 08:00 UTC and root cause investigation started at 08:05. Time to detection was 5 minutes.

    Remediation
    • changed the ALB health check path, which made the targets healthy but did not stop the 404s
    • rolled back by hand to Traefik v1 as the main ingress controller
    • deleted the malformed IngressRoute by hand and confirmed the Traefik v2 bug
    • applied the changes upstream
    Action items
    • Upgrade Traefik to v2.5, where the bug is fixed, and verify that it removes the bug
    • Change the Traefik v2 health check path to something more robust
    • Set up an external liveness probe on the ingress controller
    • Consider workload segmentation to limit the blast zone
    • Look into monitoring the Traefik v2 API for rapid changes
    • Look into smoke testing during and after node rollover
    Lessons

    A malformed IngressRoute did nothing visible for five days and then took down all ingress when every Traefik v2 pod had restarted during a node rollover. Because Traefik v2 failed gracefully, the first signals pointed at the ALB health checks instead of at Traefik. On incident handling, the team concluded that appointing an incident manager could have helped, that the war room meeting should be recorded and a log book kept, that people not involved should only make suggestions in writing, and that a short first round of sharing thoughts and gut feelings is useful.

    View case
Check your cluster for this

This case involves Cloud load balancer and Ingress controller. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
Cloud load balancerIngress controller
Contributing factor
Excessive blast radiusChange with a delayed effectNo configuration tests
Organisation
DFDS
  1. IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404
    DFDS2021