Incident Library
Applied filters Root causeConfiguration changeClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases35 cases

11 to 15 of 35

  1. IL-0153 DOKS worker nodes went NotReady at 06:00 UTC on three days because a systemd timer in the Debian image ran apt upgrade and killed it mid kernel upgrade, breaking node networking
    DigitalOcean2023
    Symptoms

    Customers of DigitalOcean Managed Kubernetes (DOKS) saw worker nodes go NotReady between 06:00 and 07:00 UTC and stay unavailable, taking down the services on them, until the nodes were replaced or rebooted; some nodes needed several replacements or reboots. The first sporadic reports on October 8 said replacing the node fixed it. The same thing happened again around 06:00 UTC on October 9, more widely and across many regions, and again on October 10. Only clusters on versions 1.27.6-do.0 and 1.28.2-do.0 were affected.

    Node NotReadyService outage
    Trigger

    A systemd timer in the DigitalOcean Debian image started apt upgrade daily at 06:00 UTC, with a random delay of up to 60 minutes.

    Root cause
    Configuration change
    Component
    Node operating system imageLinux kernelkubelet
    Contributing factor
    Unreviewed defaultsLimited debug accessChange with a delayed effect
    Detection

    Sporadic customer reports of failed worker nodes arrived at 07:43 UTC on October 8; internal incident response started at 07:10 UTC on October 9 after broader impact across regions.

    Remediation
    • Advised customers to replace or reboot affected worker nodes
    • Arranged SSH and console access to worker nodes while they were healthy, and live-debugged an affected node on October 10
    • Released a DaemonSet to all affected clusters that disables the apt upgrade systemd timer on every worker node
    • Released new node images without the auto-update process
    Action items
    • Disable the timer during node provisioning, then remove the DaemonSet from all clusters
    • Add a check to the internal conformance test suite for new cluster versions that the auto-upgrade process and other unwanted timers stay disabled
    • Support logging into worker nodes even when normal access paths are broken
    Lessons

    An operating system auto-update timer left enabled in a node image can change the kernel and systemd underneath Kubernetes, and a timeout that kills the upgrade half-way can leave a node without networking. Failures that start at the same time every day point to a scheduled job on the nodes, and systemd timers belong on that list next to cron jobs. When the failure takes out a node's network, an investigation that depends on logging into the node stalls; access paths that work while the node is broken shorten diagnosis.

    View case
  2. IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on device
    UK Ministry of Justice2023
    Symptoms

    At 13:42 the RootVolUtilisation-Critical alert fired in the high-priority alerts channel. At 14:03 a user reported image pull errors with the message "no space left on device".

    Node disk fullImage pull failuresPod evictionPods stuck in ContainerCreatingIP address exhaustion
    Trigger

    Upgrading the EKS Terraform module to version 18, which changed the node launch template, and draining workloads onto the new nodes it created.

    Root cause
    Configuration change
    Component
    Node root diskNode launch templateNode poolAWS VPC CNICoreDNS
    Contributing factor
    Missed in reviewNo configuration testsNo alert on limit
    Detection

    The RootVolUtilisation-Critical alert fired in the high-priority alerts channel at 13:42, and a user reported "no space left on device" image pull errors at 14:03.

    Remediation
    • Uncordoned the old default node group and drained the new nodes to move workloads back
    • Identified the launch template change that cut the root volume to 20 GB and applied a fix on the manager and live clusters
    • Drained the old node group onto the fixed nodes
    Action items
    • Update the runbook to compare launch template changes during an EKS module upgrade
    • Create a test setup that pulls images of different sizes, like live
    • Update the RootVolUtilisation alert runbook to check the disk space configuration
    • Scale CoreDNS dynamically based on the number of nodes
    • Investigate whether IPv6 can solve the IP prefix starvation problem
    • Add drift testing to identify when a terraform plan shows a change to the launch template
    • Set up logging for the CNI and ipamd logs and alerts on errors related to IP prefix starvation
    Lessons

    The write-up lists review actions rather than lessons. They aim at two gaps: the launch template changes brought by a module upgrade were not compared or drift-tested before rollout, and there were no CNI or ipamd logs or alerts to show the IP prefix starvation the team suspected during the drain.

    View case
  3. IL-0017 AWS VPC CNI stopped allocating pod IP addresses on six nodes of the live cluster, leaving new pods in ContainerCreating
    UK Ministry of Justice2023
    Symptoms

    At 08:15 a user posted in the #ask-cloud-platform channel that new deployments were not starting: their pods stayed in ContainerCreating.

    Pods stuck in ContainerCreatingIP address exhaustion
    Root cause
    Configuration change
    Component
    AWS VPC CNINode pool
    Contributing factor
    No configuration tests
    Detection

    A user reported new deployments stuck in ContainerCreating in the #ask-cloud-platform channel at 08:15; the incident was declared at 09:31.

    Remediation
    • Added new nodes and cordoned and drained the six affected nodes
    • Enabled prefix delegation in the VPC CNI on the live cluster after comparing its CNI settings with a 1.23 test cluster
    Action items
    • Add a test or check that confirms IP address allocation is working as expected (#4669)
    Lessons

    The write-up states no lessons beyond its review action, which is to add a check that IP address allocation works.

    View case
  4. IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods together
    UK Ministry of Justice2023
    Symptoms

    At 10:23 on 6 June 2023 a user of the Cloud Platform reported that all of their production pods had been deleted at the same moment. Several users received Pingdom alerts that their applications were down for a few minutes.

    Service outage
    Trigger

    The team changed the instance type of the cluster's worker nodes through Terraform.

    Root cause
    Configuration change
    Component
    Node poolTerraform
    Contributing factor
    Excessive blast radiusNo runbook
    Detection

    Users reported that their production pods had been deleted at once and that Pingdom alerts showed their applications down; the incident was declared at 11:00.

    Action items
    • Ask AWS, through a support ticket, for the steps to change the node instance type without an outage
    • Plan how to make instance type changes through Terraform without downtime and add a runbook with the steps for changing the node instance type
    View case
  5. IL-0068 Anonymous access to the Kubernetes API server used to plant a ClusterRoleBinding backdoor and a cryptominer DaemonSet
    Aqua Security2023
    Symptoms

    Aqua's research team Nautilus recorded the attack on one of its Kubernetes honeypots. What appeared in the cluster was a DaemonSet running the image kuberntesio/kube-controller:1.0.1 from Docker Hub on every node, using the nodes' CPU to mine Monero.

    Unauthorised access or a foreign workloadResources deleted
    Trigger

    An API server misconfigured to accept unauthenticated requests from anonymous users with privileges.

    Root cause
    Configuration change
    Component
    Kubernetes API serverRBAC and service accountsContainer registry and its credentials
    Contributing factor
    Monitoring blind spot
    Detection

    Aqua recorded the attacker's API requests on its honeypot; planted AWS access keys raised a beacon when the attacker tried to use them.

    Lessons

    Fixing an exposed API server is not enough once an attacker has been in: a ClusterRoleBinding to an attacker-created ServiceAccount keeps working after anonymous access is closed. Attackers name these objects and images after real Kubernetes components so they blend into the API audit logs and pass for legitimate deployments. Aqua recommends scanning clusters for vulnerabilities, exposed secrets and misconfigurations with a tool such as Trivy.

    View case
Check your cluster for this

These 35 cases involve Cloud load balancer and Kubernetes API server. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change35 cases

Component
Cloud load balancer6Kubernetes API server6Node pool6Configuration templating4Terraform4AWS VPC CNI4kubelet3Container memory limit3Telemetry agent3Admission webhook2Argo CD2Container runtime2etcd2GKE control plane2HAProxy2Ingress controller2Node launch template2Persistent volumes and stateful sets2In-memory cache1CI system1Cluster dashboard1CNI plugin1ConfigMaps and Secrets1conntrack table1CoreDNS1Credential broker for pods1Application database1Client-side DNS resolver and cache1Helm1Horizontal pod autoscaler1Jobs and CronJobs1Linux kernel1Application logging library1Node root disk1Node operating system image1Node iptables rules1Pod priority and preemption1Prometheus1RBAC and service accounts1Container registry and its credentials1Service discovery1
Contributing factor
Excessive blast radius11Monitoring blind spot10Unverified assumption9Automation amplification8Unreviewed defaults7No configuration tests6Missed in review6No safeguard on destructive change4No staged rollout4Unrepresentative staging4Change with a delayed effect3Limited debug access3No runbook3Retry storm3Ignored early signal2No alert on limit2No health probes2No headroom2Untested backup1Cache-masked failure1Harmful manual action1No resource requests or limits1Overlapping failures1Partial rollout1Single point of failure1Unpinned version1
Organisation
DFDS4OpenAI4Grafana Labs3UK Ministry of Justice3DigitalOcean2GitLab2Adevinta1AdGuard VPN1Aqua Security1CoW Protocol1GitHub1Google Cloud1Hans H. Kim1InfluxData1Jetstack1JW Player1LINE+1MUSINSA1NU.nl1Pusher1Skyscanner1Spotify1Target1
  1. IL-0179 With DEBUG logging and synchronous log output, a high-traffic service filled the 64 KB stdout pipe, so request threads blocked in write() while holding database connections and HikariCP ran dry, failing requests in 30-second bursts
    MUSINSA2026
  2. IL-0200 Newly provisioned Kubernetes nodes ran an incompatible etcd version and broke cluster consensus in several GitHub production clusters, causing 5XX errors on Git operations and page loads
    GitHub2026
  3. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks
    AdGuard VPN2025
  4. IL-0122 An invalid value in a configuration shared by several ChatGPT services put pods into crash loops and raised error rates
    OpenAI2025
  5. IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud services
    Grafana Labs2025
  6. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
    OpenAI2024
  7. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters
    OpenAI2024
  8. IL-0130 Deployments of large models from Vertex Model Garden failed with "Model server never became ready" in most regions for 13 days after a storage provisioning configuration change
    Google Cloud2024
  9. IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hours
    CoW Protocol2024
  10. IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1
    DigitalOcean2024
  11. IL-0153 DOKS worker nodes went NotReady at 06:00 UTC on three days because a systemd timer in the Debian image ran apt upgrade and killed it mid kernel upgrade, breaking node networking
    DigitalOcean2023
  12. IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on device
    UK Ministry of Justice2023
  13. IL-0017 AWS VPC CNI stopped allocating pod IP addresses on six nodes of the live cluster, leaving new pods in ContainerCreating
    UK Ministry of Justice2023
  14. IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods together
    UK Ministry of Justice2023
  15. IL-0068 Anonymous access to the Kubernetes API server used to plant a ClusterRoleBinding backdoor and a cryptominer DaemonSet
    Aqua Security2023
  16. IL-0186 Pods on one Kubernetes worker node could not reach external HTTP and HTTPS because storage components had added nat PREROUTING rules that redirected ports 80 and 443 to local ports
    Hans H. Kim2023
  17. IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodes
    DFDS2023
  18. IL-0116 A CNI configuration change incompatible with a hardware variant that exists only in production cut network connectivity in an OpenAI GPU cluster and took text-davinci-003 fully down
    OpenAI2023
  19. IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on Nginx
    LINE+2022
  20. IL-0191 A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rollout
    Grafana Labs2022
  21. IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environment
    InfluxData2022
  22. IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errors
    GitLab2022
  23. IL-0111 Switching the main Kubernetes ingress controller from Traefik v1 to v2 stopped X-Forwarded headers being passed, which broke the Okta login flow for every service using Okta and stopped DFDS online bookings for over an hour
    DFDS2021
  24. IL-0010 A templating change that dropped its {{ }} delimiters made Argo CD delete all 478 Skyscanner services in every region
    Skyscanner2021
  25. IL-0209 A too-high HPA CPU target kept GitLab.com web pods from scaling up as traffic moved from VMs to Kubernetes, breaching the web latency SLO for 15 minutes
    GitLab2021
  26. IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503
    DFDS2021
  27. IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time out
    DFDS2020
  28. IL-0189 A JVM service moved from EC2 to Kubernetes answered up to ten times slower because KIAM's 15-minute credentials made the AWS Java SDK fetch new credentials on every request
    Adevinta2019
  29. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node
    Jetstack2019
  30. IL-0004 A new default pod priority let a new Cortex cluster preempt every production ingester in a cascade, taking Grafana Cloud Hosted Prometheus down
    Grafana Labs2019
  31. IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second one
    Spotify2019
  32. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods
    Pusher2019
  33. IL-0052 A cryptocurrency miner ran on every node of the development and staging clusters through a public, unauthenticated Weave Scope dashboard
    JW Player2019
  34. IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memory
    NU.nl2019
  35. IL-0055 A failing CronJob with the default concurrency policy piled up thousands of restarting pods and took down a cluster
    Target2018