IL-0009 InfluxData · 2022-06-24 (published)kubernetes
InfluxDB Cloud runs on Kubernetes and ships 5 to 15 changes to production on a typical workday through Argo CD. Argo CD also manages the definitions of its own Applications and AppProjects across clusters, the "app of apps" pattern. Configuration is written in jsonnet and rendered to YAML, and at the time all resources of one application were rendered into a single YAML file. Objects are named (app name)-(cluster name) and live under env/(cluster name)/(app name)/yml. Because of an automation error, the new objects in the pull request were named app01-cluster01 instead of app02-cluster01, so both the existing and the new definition ended up in the same file. Argo CD applied the objects in file order, the last definition won, and the original app01 Application, InfluxDB Cloud's core workload, was replaced and deleted.
Recovery was done by hand in a fixed order. The team restored etcd, which holds metadata on a dedicated cluster separate from the Kubernetes control plane, first. They then recreated the missing Kubernetes objects so the retained PersistentVolumes became available again, and recreated the StatefulSets. Zookeeper volumes were restored from an hourly Velero backup. Kafka's StatefulSet was rebuilt one pod at a time with health and readiness checks disabled, because Kafka does not report ready until its leader is up. Etcd and Kafka could have been restored in parallel, but the team did them one after the other to keep the procedure correct. With Kafka acting as the write-ahead log, writes were accepted again before storage worked, and the write path was given more replicas and higher memory requests and limits to absorb the backlog from Telegraf and other clients. Storage pods were recreated, downloaded their data from object storage, reindexed it and then replayed the unprocessed write-ahead log.
In the last phase the team reduced replicas of some storage services so they started faster, re-enabled Argo CD to create the objects still missing, turned queries and dashboards back on, restored the normal replica counts, enabled scheduled tasks and ran final QA checks. From the merge to full functionality took just under six hours, and no data was lost: clients that retry, such as Telegraf by default, wrote their data once the service came back.
Right after a pull request to the infrastructure-as-code repository was merged, every API endpoint of one InfluxDB Cloud production cluster began returning 404 errors. Writes and reads failed alike: no one could collect data, scheduled tasks failed to run and external queries did not work.
Behind the 404s, the production environment itself was gone. Argo CD, the continuous delivery tool that makes each Kubernetes cluster match the YAML in Git, had deleted the Application that held InfluxDB Cloud's core workload, along with the resources it managed: the StatefulSets and PersistentVolumeClaims of the storage engine, Kafka and Zookeeper among them. At the same time it created a workload that was not meant to run on that cluster. The underlying PersistentVolumes and cloud disks of the data volumes survived because they used reclaimPolicy: Retain, but the Zookeeper volumes were lost.
The cause was hard to see in the merged change. The pull request looked like it only added objects, including a new Argo CD Application and AppProject, but those objects carried the name of the existing production Application, and the problem was hard to spot because it involved an Argo CD collision between a project and an application name.
The first instinct was to revert the merged change. The team started the revert and stopped almost at once, because Argo CD would then have created a brand-new, empty instance of the application, without the metadata about users, dashboards and tasks and without customer data. Monitoring and alerting had not flagged the problem right away, which is why the reflex was a rollback rather than a planned recovery, and there was no runbook for losing part or all of an InfluxDB Cloud instance.
Symptom Service outageResources deleted
Root cause Configuration change
Monitoring and alerting did not detect the issue right away, so the team's first response was to try to roll the change back rather than plan a recovery. The write-up does not say how the team found out.
The biggest gap was that monitoring did not notice the deletion quickly, which pushed the team toward a reflex rollback; in a stateful system a revert can create a new, empty instance instead of bringing the old one back. Retained volumes and clients that retry writes are what kept data loss at zero. Rendering many objects into one file let an overwrite of an existing object look like an addition, so generated configuration needs one object per file and duplicate detection that compares objects the way Kubernetes does. Stateful resources need protection from pruning, and that protection must cover their Namespace too, or deleting the namespace removes everything inside it.
IL-0010 A templating change that dropped its {{ }} delimiters made Argo CD delete all 478 Skyscanner services in every regionSkyscanner · 2021same root cause · shared Configuration templating, Argo CD
IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checksAdGuard VPN · 2025same root cause · shared Persistent volumes and stateful sets
IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud servicesGrafana Labs · 2025same root cause · shared Configuration templating
IL-0191 A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rolloutGrafana Labs · 2022same root cause · shared Configuration templating
IL-0010 linked by the editorhttps://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.