Incident Library
Applied filters Root causeConfiguration changeFactorUnverified assumptionOrganisationDFDSClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503
    DFDS2021
    Symptoms

    At 08:29 UTC on 22 June 2021 alerts fired because all services in the Kubernetes cluster were returning 503. No inbound traffic was reaching the cluster. A customer reported the problem at 08:32.

    Service outageHigh error rate
    Trigger

    The pipeline replaced the AWS VPC CNI as part of a refactoring that moved EKS add-ons from kubectl to native Terraform management.

    Root cause
    Configuration change
    Component
    AWS VPC CNICloud load balancerTerraform
    Contributing factor
    Unverified assumptionNo configuration tests
    Detection

    Alerts fired at 08:29 UTC, eight minutes after the VPC CNI was replaced; a customer reported the issue at 08:32.

    Remediation
    • committed a fix that reverted the infrastructure-modules reference
    • re-triggered the pipeline after it failed on an AWS API timeout
    • let the pipeline reattach the target groups to the ALB
    Action items
    • Revert the ALB de-registration delay to the default of 300 seconds
    • Update the roll-over script to set the ALB de-registration delay to zero temporarily before a roll-over
    • Create an ingress test suite
    Lessons

    The change to the de-registration delay had been weighed mainly for performance. Critical pipeline runs should be watched while they run. Everyone should receive the alerts and react to them, and alert messages that were unclear should be reworded.

    View case
  2. IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time out
    DFDS2020
    Symptoms

    At 14:41 CET on 4 November 2020 users reported timeouts when deploying to Hellman, the production Kubernetes cluster. The timeouts were confirmed at 15:04.

    Deploys failingControl plane lag
    Trigger

    During an Alcide workshop that began at 13:00 CET, an Alcide admission controller was applied to the production cluster at 13:13 in the belief that it would not disrupt anything.

    Root cause
    Configuration change
    Component
    Admission webhookKubernetes API server
    Contributing factor
    No configuration testsUnverified assumptionMonitoring blind spot
    Detection

    Users reported timeouts when deploying to the Hellman cluster at 14:41 CET; the timeouts were verified at 15:04.

    Remediation
    • toggled the Alcide admission controller to verify that it caused the timeouts
    • permanently deleted the Alcide admission controller
    Action items
    • Monitor Kubernetes deployments for timeouts
    • Monitor 5xx status codes from the Kubernetes API server
    Lessons

    New components are applied in sandbox clusters, not in production. The team needs enough Alcide knowledge to maintain the component until it is fully deprecated.

    View case
Check your cluster for this

These 2 cases involve Admission webhook and Cloud load balancer. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change2 cases

Component
Admission webhook1Cloud load balancer1Kubernetes API server1Terraform1AWS VPC CNI1
Contributing factor
Unverified assumption2No configuration tests2Monitoring blind spot1
Organisation
DFDS2
  1. IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503
    DFDS2021
  2. IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time out
    DFDS2020