Cases2 cases

1 to 2 of 2

1. IL-0154 Line cards on an SFO3 core router rebooted together and halved backbone capacity, and DOKS nodes whose guest networking did not recover stayed NotReady with degraded API servers until all affected nodes in the SFO3 DOKS fleet were recycled

   DigitalOcean2024

   Symptoms

   From 22:25 to 22:50 UTC every DigitalOcean service in the SFO3 region had network connectivity problems, as traffic between the datacenter and the backbone dropped. Traffic through the router was back by 22:30 and SFO3's control plane systems had reconnected by 22:50, and most services recovered then.

   Node NotReadyControl plane unresponsiveIntermittent connection failures

   Trigger

   Most of the line cards on one SFO3 core router rebooted at the same time at 22:21 UTC because of hardware errors on the device.

   Root cause

   DNS or network fault

   Component

   Kubernetes API serverkubelet

   Contributing factor

   –

   Remediation

   • The crashed line cards came back online and traffic on the core router was restored
   • Manually recycled all affected nodes across the SFO3 DOKS fleet

   Action items

   • Work with the router vendor's support team to find the root cause of the line card crash
   • Upgrade software on the core routers in SFO3
   • Raise thresholds for automated remediation so affected nodes are restored without manual action

   Lessons

   A short network fault can leave Kubernetes nodes broken long after the network itself is healthy, when the guest networking on the machines does not recover by itself. Checking node health and the API servers after the network returns, and having automated remediation that is allowed to replace many nodes at once, shortens that tail.

   View case
2. IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1

   DigitalOcean2024

   Symptoms

   At 13:02 UTC on 2024-01-23 DigitalOcean posted that, as of 12:18 UTC, its engineers were investigating an issue with Kubernetes clusters in the FRA1 region and that users might see errors when communicating with their clusters. The control planes of the affected DOKS (DigitalOcean Kubernetes) clusters had become inaccessible.

   Resources deletedControl plane unresponsive

   Trigger

   An incorrect configuration was applied to an internal infrastructure cluster in FRA1 at 12:21 UTC during a maintenance operation.

   Root cause

   Configuration change

   Component

   etcdKubernetes API server

   Contributing factor

   Automation amplificationNo safeguard on destructive changeExcessive blast radius

   Detection

   Engineers were investigating errors with Kubernetes clusters in FRA1 as of 12:18 UTC and identified the incorrect configuration as the cause at 12:36.

   Remediation

   • restored each affected cluster's etcd from its latest twice-daily backup
   • restored the stateless control plane components after etcd
   • patched the recovery mechanism after unexpected problems during recovery
   • found and fixed an additional small bug during recovery
   • contacted affected customers directly with further instructions, since some might need to reconfigure Kubernetes resources

   Action items

   • Prevent automatic deletion of infrastructure cluster resources under any circumstances in the production environment

   Lessons

   When managed control planes run as resources in another cluster, one configuration mistake in that cluster can delete hundreds of control planes at once through ordinary automation. The etcd backups limited the damage, but their twice-daily schedule set the worst case at 12 hours of lost control plane state, and the recovery tooling itself needed patching during the incident.

   View case

By root cause2