Incident Library

IL-0241 Stripe · 2019-07-10database

A database shard could not elect a new primary, and the Stripe API failed most requests in two periods on one day

Stripe splits data by kind into database clusters and by quantity into shards, and each shard has several redundant nodes. Three months before the incident Stripe upgraded its databases to a new minor version after testing in a quality assurance environment and a phased rollout from less critical to more critical clusters. The version ran well in production, through many successful failovers, but it brought a fault in the database's failover system that showed only when several nodes were stalled. On 10 July one shard was in that state when its primary failed, and the shard could not elect a new primary. The team found that the fastest fix, forcing a new election, required restarting the database cluster; after all nodes were restarted at 17:00 UTC an election succeeded, and the API recovered at 17:02 UTC, 27 minutes after the start.

Looking for the root cause, the team found a code path in the new minor version of the election protocol that was likely behind the fault, and at 20:42 UTC rolled all shards of the cluster back to the previous version of the protocol, which took four minutes. At 21:14 UTC shards became unavailable again. The rolled-back election protocol interacted badly with a configuration setting recently introduced on the production shards, and the result was CPU starvation on every affected shard. Once the CPU starvation was seen, the team found the cause, updated the production configuration and restarted the cluster's nodes by 22:34 UTC, then ramped traffic back up, services needed for user-initiated API requests first, and the API fully recovered at 22:47 UTC.

Severity
–
Scope
Stripe API: a substantial majority of requests failed from 16:35 to 17:02 UTC and from 21:14 to 22:47 UTC
Users affected
Stripe users; those with five or more failed POST requests were sent an email
Region
–
Time to detect
1m
Time to resolve
6h 12m

Symptoms

On 10 July 2019 the Stripe API failed a substantial majority of requests twice, from 16:35 to 17:02 UTC and from 21:14 to 22:47 UTC. In the first period automated monitoring detected within a minute that a database cluster had failed to elect a new primary.

The first period began when the primary node of the database cluster failed and one shard could not elect a new primary. Without a primary the shard accepted no writes, and applications that write to it began to time out. Many applications used this shard, the API among them, so the stalled writes starved the API of compute resources. Four days earlier two nodes had stalled for reasons not yet known: they stopped emitting the metrics that report their replication lag but kept answering active health checks as healthy.

At 21:14 UTC automated alerts reported that some shards of the cluster were unavailable, including the shard from the first period, and the database cluster showed high CPU usage. The symptoms looked the same as before, so the team at first assumed the same fault had recurred on several shards and followed the playbook that had worked earlier. The second period had a different cause, and CPU resource contention also slowed the configuration change that ended it.

Symptom High error rateFailover failureWrites rejectedIntermittent timeoutsSustained high node CPU

Root cause

Root cause Software defect

Component DatabaseDatabase failover manager
Contributing factor Change with a delayed effectMonitoring blind spotExcessive blast radiusHarmful manual actionUnverified assumptionOverlapping failures
Trigger
The primary node of a database cluster failed at 16:35 UTC while one shard had several stalled nodes. The second period was set off by rolling the election protocol back to the previous minor version at 20:42 UTC.

Detection

Automated monitoring detected the failed election within a minute, and the team was alerted at 16:36 UTC. The second period was announced at 21:14 UTC by automated alerts that some shards in the cluster were unavailable.

Remediation

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Database failover manager among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.