Cases2 cases

1 to 2 of 2

1. IL-0563 Linode servers lost connectivity in all data centers as route servers built incomplete BGP route tables

   Linode2025

   Symptoms

   At about 16:38 UTC on 17 March 2025, internal alerts fired across multiple Linode data centers because various servers had been unreachable for an extended period. At the same time customers began reporting problems with several products.

   Network unreachableService outage

   Trigger

   A release of Akamai Compute's internal backend API component was rolling out, and during the rollout the API gave inconsistent responses to the route servers.

   Root cause

   Faulty deployment

   Component

   BGP routingNetwork control plane (SDN)

   Contributing factor

   Partial rollout

   Detection

   Internal alerts fired across multiple data centers showing servers unreachable for an extended period, and customers reported problems with multiple products at the same time.

   Remediation

   • Restarted the route servers in all data centers in phases, starting with the most affected regions

   Action items

   • Hold backend API releases until there is a better way to make sure route data for route servers is contiguous

   View case
2. IL-0437 Old pods from an unfinished Argo Rollouts canary could not start with a Helm-updated ConfigMap and died in a cascade

   Gunosy2023

   Symptoms

   During a traffic surge from a breaking news alert, pods of a Gunosy service on Kubernetes could not start because the containers could not read their settings from the ConfigMap. The pods that kept running took the extra load and were restarted, failed to start again, and pod after pod died.

   Container crash loop

   Trigger

   A breaking news alert increased traffic and pods were scaled out automatically while a release that removed an old database's settings from the ConfigMap had been rolled out to only one canary pod.

   Root cause

   Faulty deployment

   Component

   ConfigMaps and SecretsHelmProgressive delivery controller

   Contributing factor

   Partial rolloutAutomation amplification

   Action items

   • Configure Argo Rollouts to roll out automatically, with setWeight 1, a one-hour pause and then setWeight 100
   • Add the "helm.sh/resource-policy": keep annotation so Helm keeps past versions of the ConfigMap
   • Build and use a Helm plugin, helm-clear, that keeps only the last few ConfigMap versions by the revision number in their names, set to keep 5 in CD

   Lessons

   Gunosy holds that a review that only resolves to be careful next time would soon be followed by the same failure, because human attention cannot be relied on, so each incident gets a permanent fix built into the system. Keeping past ConfigMap versions also made it possible to roll back quickly without going through CI/CD, which a ConfigMap change had previously prevented.

   View case

By root cause1