IL-0196 Palark · 2020-03-13 (published)kubernetes
Palark runs Redis for a customer's large website on Kubernetes with the Redis Operator by Spotahome, which builds a Redis StatefulSet, a Sentinel Deployment and ConfigMaps from a RedisFailover custom resource. One installation was used as a persistent store without TTLs; its .rdb snapshot was over 5 GB and the single Redis pod used about 16 GB of RAM at peak while saving to disk, so at first only one replica fit on the nodes.
After new servers were added, the team raised the replica count to two to make the database more reliable and, in the same change, cut the memory requests and limits from 22 GB to 19 GB so the pod could be rescheduled more easily. The resource change forced a rolling update of the StatefulSet. Because the operator's readiness probe was a plain ping, the new replica counted as ready before it held any data, the rolling update restarted the only pod with data, Sentinel promoted the empty replica, and replication from that empty master erased the original data. The result was downtime and a restore from backup.
The author names the operator's readiness probe as the principal cause and opened an issue with the operator project; the developers answered with a fix, which Palark tested. The author also points out that with this operator a single-replica setup was safer against data loss than a multi-replica one, because a master restart while a new replica is joining loses the data. A later analysis of the dump with redis-memory-analyzer found 27,402,900 keys, 16,236,021 of them (59.24%) under one key pattern, and the developers found over 18 million excess keys, which they attributed to a wrong architectural decision in the application.
Right after an updated RedisFailover resource was rolled out, the large Redis database lost all its data and the service went down; the data had to be restored from a backup taken just before the change.
The sequence, as reconstructed afterwards: the StatefulSet grew to two pods and the new pod joined as a slave and began replicating. The operator's readiness probe, which only sends a plain ping, marked the new pod ready almost at once. Because the memory requests and limits had also changed, the StatefulSet rolling update then restarted the one pod that held the data.
The Sentinels saw the master disappear and promoted the new, still empty replica to master. At that moment the old master had hardly managed the initial BGSAVE needed for a full resynchronisation. When the rolling update finished, the restarted original pod connected as a replica of the empty master and synchronised from it immediately, which wiped its data.
Symptom Data lossService outage
Root cause Software defect
–
Changing the replica count and the pod resources in one rollout turned a seemingly harmless change into a rolling restart of the only pod with data while the new replica was still empty.
A readiness probe that only checks that Redis answers a ping does not tell whether a replica has finished syncing. Operators for critical stateful components need to be evaluated for maturity before production use, and the dump.rdb file should be backed up before any change to the Redis cluster.
IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1Amazon Web Services · 2025same root cause · shared Application database
IL-0024 Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three timesHugging Face · 2024same root cause · shared Application database
IL-0123 A response buffer allocated anew in a loop made OpenAI's routing layer nodes hit their memory limits and fail readiness checks on a morning with more completions than any day before, causing 502 and 503 errors across all modelsOpenAI · 2023same root cause · shared Container memory limit
IL-0072 High etcd write latency made the API servers return stale data, and the production cluster flipped between two views of its stateNordstrom · 2017same root cause · shared Persistent volumes and stateful sets
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.