Cases23 cases

1 to 5 of 23

1. IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodes

   DigitalOcean2026

   Symptoms

   From 20:31 UTC on July 9, 2026, DigitalOcean reported a problem with Kubernetes deployments in its NYC1 region. Application workloads could see intermittent DNS failures and NodeNotReady events.

   DNS resolution failuresIntermittent timeoutsNode NotReady

   Root cause

   Capacity exhaustion

   Component

   CoreDNSNode pool

   Contributing factor

   Unknown limit

   Lessons

   Cluster DNS is sensitive to latency, and CoreDNS on shared-CPU nodes can time out intermittently. DigitalOcean's advice is to place CoreDNS on dedicated-CPU node pools and to run enough CoreDNS replicas.

   View case
2. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours

   Magic Pages2025

   Symptoms

   At around 09:33 UTC one of the three control plane nodes of the Kubernetes cluster that hosts Magic Pages went offline. Looking at the node showed that it had run out of memory.

   Service outageNode NotReadyCloud API rate limit exhausted

   Trigger

   One of the three control plane nodes ran out of memory and went offline at about 09:33 UTC; the attempts to replace it with hetzner-k3s failed while the cluster's own components were exhausting the Hetzner API rate limit.

   Root cause

   Capacity exhaustion

   Component

   Cloud controller managerPersistent volumes and stateful sets

   Contributing factor

   Unknown limitRetry stormMonitoring blind spotLimited debug accessSingle point of failureNo runbook

   Detection

   The founder found the control plane node offline at about 09:33 UTC. Monitoring alerts about unreachable customer websites started around the time the replacement attempts failed, and about half an hour of investigation identified the exhausted Hetzner API rate limit.

   Remediation

   • powered off all servers of the Kubernetes cluster to stop the API requests so the Hetzner rate limit could recover
   • waited until ratelimit-remaining had built up a buffer before starting recovery
   • restarted the control plane servers
   • started the essential Longhorn components by hand so Longhorn could heal and make storage volumes available
   • restarted the remaining websites by hand

   Action items

   • Monitor the Hetzner API call rate coming from the Kubernetes cluster nodes themselves to get early warning of high usage
   • Get a higher Hetzner API rate limit; Hetzner support was contacted
   • Investigate how to reduce the cluster's reliance on a single API, including alternatives to hetzner-k3s
   • Document and test a procedure for recovering Longhorn after a full cluster shutdown

   Lessons

   The running cluster, not only the deployment tooling, consumed the cloud provider's API quota, and in a failure it consumed it fast enough that the quota never recovered. The default limit of 3600 requests per hour was not enough for failure and recovery scenarios, and API usage from inside the cluster was not monitored. A tool that hides the provider's API errors slows diagnosis. After a full, non-graceful shutdown, Longhorn needed manual intervention to come back, which lengthened the outage.

   View case
3. IL-0154 Line cards on an SFO3 core router rebooted together and halved backbone capacity, and DOKS nodes whose guest networking did not recover stayed NotReady with degraded API servers until all affected nodes in the SFO3 DOKS fleet were recycled

   DigitalOcean2024

   Symptoms

   From 22:25 to 22:50 UTC every DigitalOcean service in the SFO3 region had network connectivity problems, as traffic between the datacenter and the backbone dropped. Traffic through the router was back by 22:30 and SFO3's control plane systems had reconnected by 22:50, and most services recovered then.

   Node NotReadyControl plane unresponsiveIntermittent connection failures

   Trigger

   Most of the line cards on one SFO3 core router rebooted at the same time at 22:21 UTC because of hardware errors on the device.

   Root cause

   DNS or network fault

   Component

   Kubernetes API serverkubelet

   Contributing factor

   –

   Remediation

   • The crashed line cards came back online and traffic on the core router was restored
   • Manually recycled all affected nodes across the SFO3 DOKS fleet

   Action items

   • Work with the router vendor's support team to find the root cause of the line card crash
   • Upgrade software on the core routers in SFO3
   • Raise thresholds for automated remediation so affected nodes are restored without manual action

   Lessons

   A short network fault can leave Kubernetes nodes broken long after the network itself is healthy, when the guest networking on the machines does not recover by itself. Checking node health and the API servers after the network returns, and having automated remediation that is allowed to replace many nodes at once, shortens that tail.

   View case
4. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day

   DFDS2024

   Symptoms

   At 10:01 UTC on 2024-03-13 a Kubernetes node entered NotReady state, with CPU requests at 78% and memory requests at 82%. An alert fired in Slack at 10:03 and was picked up at 10:05. An attempt to SSH into the node at 10:13 failed because it was unresponsive; it started responding again at 10:19.

   Node NotReadyContainer crash loopOOMKilledMonitoring outage

   Trigger

   Pods of a dev container that used Hangfire to trigger jobs failed with OOM and crash-looped on a node where a Grafana Agent pod was already using a lot of memory.

   Root cause

   Capacity exhaustion

   Component

   Telemetry agentContainer memory limitApplication database

   Contributing factor

   Excessive blast radiusNo resource requests or limitsHarmful manual actionMonitoring blind spot

   Detection

   A Slack alert fired two minutes after the node went NotReady and was picked up two minutes later; the second node's NotReady at 15:03 also raised a Slack alert.

   Remediation

   • marked the EC2 instance unhealthy so a new node started
   • cordoned and force-drained the NotReady node and terminated its EC2 instance
   • removed the new LimitRange from the Grafana Agent namespace and restarted the Grafana Agent pod
   • scaled the dev container to 0 replicas, after which EC2 network traffic returned to normal
   • after the second NotReady, scaled the dev container to 0 replicas again and added a new node
   • emailed the team that owns the dev container, which confirmed it was working on a fix

   Action items

   • Move grafana-agent to the monitoring node group
   • Add alerts for when bandwidth across the cluster reaches a certain threshold
   • Investigate whether memory limits can be set on the grafana-agent Helm chart
   • Investigate grafana-agent memory usage
   • Ask Grafana support whether grafana-agent should use a PVC
   • Consider pushing node logs to Loki with limited retention
   • Create a wiki page with common metrics and log queries to check when investigating issues with Kubernetes nodes

   Lessons

   Dev workloads mixed with production let one crash-looping dev container take down nodes that production also used, and a monitoring agent outside its dedicated node group added to the memory pressure. Scaling the suspect container back up before its owners had fixed it brought the failure back on the replacement node. The team also noted that action items from earlier post-mortems need follow-up, and that knowledge of useful CloudWatch metrics had not been shared, which made the diagnosis harder.

   View case
5. IL-0021 GKE GPU nodes on Container-Optimized OS could not download NVIDIA drivers (403 Forbidden) and came up without working GPUs

   Google Cloud2024

   Symptoms

   From 08:00 US/Pacific on 12 March 2024, GKE GPU nodes running Container-Optimized OS (COS) that were newly created or recreated came up with GPU drivers that did not work, so workloads that used the GPUs could not run. GKE showed elevated errors.

   GPU unavailableNode NotReady

   Root cause

   Dependency failure

   Component

   GPU driverNode operating system imageNode pool

   Contributing factor

   Automation amplification

   Detection

   The report does not say how Google detected the issue. Affected users saw "Failed to download GPU driver installer, status: 403 Forbidden" on the GPU node when installing the driver.

   Remediation

   • Halted automatic node recreations, which attempt GPU driver downloads, until the issue was mitigated
   • Worked with the driver partner to resolve access to the driver download bucket

   Lessons

   The status page report states no lessons or follow-up actions.

   View case

By root cause8