Incident Library

IL-0437 Gunosy · 2023-06-20 (published)kubernetes

Old pods from an unfinished Argo Rollouts canary could not start with a Helm-updated ConfigMap and died in a cascade

The service runs on Kubernetes, keeps its settings in a ConfigMap, manages its YAML with Helm and releases through Argo Rollouts canaries: one pod is released first, and if it is fine all pods are rolled out. Earlier on the day of the incident, before the failure began, a release removed the settings of an old database, which was being retired, from both the ConfigMap and the code. The rollout after the first canary pod was forgotten, so only one pod ran the new version. With a Helm release the old and new pod versions both exist, but by default only the latest ConfigMap is kept. This normally causes no trouble because the server reads the ConfigMap only at startup.

When a breaking news alert raised traffic, pods scaled out automatically, and because the rollout had not happened, the scaled pods were the old version reading the new ConfigMap. The old code needed the old database's settings, which no longer existed, so the new pods could not start. The existing pods were overloaded and restarted, reread the ConfigMap on restart, could not start either, and the pods died one after another.

Gunosy names two root causes: forgetting the manual rollout, and startup failures because only the latest ConfigMap was kept. Automatic rollout had not been configured when Argo Rollouts was introduced, out of concern that releases would go out without a person holding the lever, but over several years forgotten rollouts had become the more common problem.

Severity
–
Scope
Pods of the service failed to start and died in a cascade during a traffic surge from a breaking news alert
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

During a traffic surge from a breaking news alert, pods of a Gunosy service on Kubernetes could not start because the containers could not read their settings from the ConfigMap. The pods that kept running took the extra load and were restarted, failed to start again, and pod after pod died.

The pods added by the automatic scaling were the old version of the application, still running because an earlier canary release had not been rolled out. The old code read an old database and needed the old settings, but only the new ConfigMap existed, so every new or restarted old pod failed at startup.

Symptom Container crash loop

Root cause

Root cause Faulty deployment

Component ConfigMaps and SecretsHelmProgressive delivery controller
Contributing factor Partial rolloutAutomation amplification
Trigger
A breaking news alert increased traffic and pods were scaled out automatically while a release that removed an old database's settings from the ConfigMap had been rolled out to only one canary pod.

Detection

–

Remediation

–

Action items

Lessons

Gunosy holds that a review that only resolves to be careful next time would soon be followed by the same failure, because human attention cannot be relied on, so each incident gets a permanent fix built into the system. Keeping past ConfigMap versions also made it possible to roll back quickly without going through CI/CD, which a ConfigMap change had previously prevented.

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, ConfigMaps and Secrets, Helm, Progressive delivery controller among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.