IL-0215 Twilio · 2013-07-18database
Twilio's billing Redis master lost all balances and went read-only, and auto-recharge charged customer cards repeatedly
Twilio's billing system keeps in-flight account balances in an in-memory Redis cluster with a single master and several slaves spread over data centers, and keeps an independent double-bookkeeping record of all balances in a separate relational datastore. When network connectivity was lost at 1:35 AM PDT, all slaves reconnected and requested full synchronization together, and by 2:39 AM PDT the load on the master's host made services that depend on it fail. On-call engineers saw the extreme load and restarted the redis process on the master, a misdiagnosis. On restart Redis read an incorrect configuration file: it tried to recover from a non-existent AOF file instead of the binary snapshot, dropped all balance data, and booted as a slave of itself, which put it in read-only mode.
With every balance at zero and no way to write balances, any billable usage, such as 1 cent for an SMS message or a phone call, made the billing system try to recharge the credit card of accounts with auto-recharge enabled, again and again. Monitoring reported the anomaly at 3:28 AM PDT and the billing system was shut down at 4:10 AM PDT. Suspended accounts were restored by 6:24 AM PDT, and the double-bookkeeping record was used to restore the lost balances. Billing was turned back on at 11:58 AM PDT, but the anomalies came back at 12:36 PM PDT for another 0.3% of customers, and billing was shut down again while suspended accounts were reactivated.
Refunds for the erroneous charges started at 2:57 PM PDT on 18 July and were finished at 3:00 PM PDT on 19 July, and all affected accounts received a credit of 10% of their last 30 days of Twilio spend. The original Redis cluster was replaced and the incorrect configuration corrected. From 7:10 PM PDT on 19 July billing was activated progressively for groups of accounts, it was active for all customers at 8:14 PM PDT, and the all-clear was given at 9:15 PM PDT.
- Severity
- –
- Scope
- Twilio billing: payments did not update balances, auto-recharge charged cards repeatedly and some accounts were suspended; usage reports for all customers were delayed in reflecting prices while billing was offline; voice and SMS were not affected
- Users affected
- 1.4% of Twilio customers (1.1% in the first wave, 0.3% in the recurrence)
- Region
- –
- Time to detect
- 1h 7m
- Time to resolve
- 43h 40m
Symptoms
At 2:39 AM PDT (9:39 UTC) on 18 July 2013, services that relied on the redis-master of Twilio's billing system began to fail because the load on its host had become extreme. At 2:42 AM PDT monitoring alerted the on-call engineers to a failure in the Redis cluster.
The load had built up for about an hour. At 1:35 AM PDT network connectivity was lost between all billing redis-slaves and the redis-master, and every slave reconnected and requested a full synchronization with the master at the same moment. Serving those full sync requests degraded the master, and the slaves timed out on their connections to it.
Seeing the extreme load, the on-call engineers misdiagnosed the redis process on the redis-master as needing a restart. On restart the redis-master read an incorrect configuration file, dropped all balance data and booted as a slave of itself in read-only mode, so the billing system could not update account balances.
At 3:28 AM PDT, after the redis-master had been restarted, the billing system monitoring reported anomalous activity: erroneous credit card charges and, in some cases, account suspensions, affecting 1.1% of all Twilio customers. Credit card payments did not update account balances, auto-recharge tried to charge the same card many times, and some accounts were suspended when a recharge hit a card that the repeated billing had deactivated. Usage reports for all customers were delayed in reflecting the prices of billable items while billing was offline. Voice and SMS kept working, but prices were not calculated for the calls and messages made during the incident.
The billing system was brought back online at 11:58 AM PDT after the balances had been restored, and at 12:36 PM PDT monitoring detected the same billing anomalies again, affecting another 0.3% of customers, so billing was taken offline a second time.
Root cause
Root cause Configuration change
- Trigger
- At 1:35 AM PDT on 18 July a loss of network connectivity disconnected all billing redis-slaves from the redis-master at once, and all of them requested a full synchronization at the same time. On-call engineers then restarted the overloaded redis-master.
Detection
At 2:42 AM PDT monitoring alerted the on-call engineers to a failure in the Redis cluster. At 3:28 AM PDT the billing system monitoring reported the anomalous charges, and at 12:36 PM PDT monitoring detected the recurrence.
Remediation
- Restarted the redis-master to relieve the high load, which loaded a wrong configuration
- Took the billing system offline to stop further credit card charges
- Restored service to all suspended accounts
- Restored the lost account balances from the independent double-bookkeeping record in the relational datastore
- Took billing offline again after the recurrence and reactivated the newly suspended accounts
- Refunded all erroneous credit card charges
- Gave all affected accounts a credit of 10% of their last 30 days of spend
- Replaced the original Redis cluster and corrected the redis-master configuration
- Activated the billing system progressively for groups of customer accounts
Action items
- Disable Redis restarts on the redis-master and recover the master by pivoting a slave instead
- Add fail-safes so the billing system neither suspends accounts nor charges cards when balances are missing or cannot be written
- Validate the billing system against the double-bookkeeping databases in real time
Lessons
Losing the in-flight balances and the ability to write balances at the same time exposed a flaw in the auto-recharge system: it failed in a dangerous way, charging cards and suspending accounts instead of stopping.
Sources
Related cases
IL-0430Mackerel's Redis stopped answering during a replication setup, keepalived failed over by mistake, and Mackerel degradedHatena · 2018same root cause · shared In-memory cache, Database replication
IL-0063VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checksAdGuard VPN · 2025same root cause · shared In-memory cache
IL-0392MariaDB replication of section x1 between eqiad and codfw broke on a missing heartbeat row during a cookbook runWikimedia Foundation · 2024same root cause · shared Database replication
IL-0374DigitalOcean creates and management actions failed globally when its service discovery datastore lost quorumDigitalOcean · 2024same root cause · shared Database replication
IL-0331An RDS MySQL Multi-AZ failover lost 5 seconds of committed data and broke replication on all five read replicasRazorpay · 2020same root cause · shared Database replication
IL-0292App Engine Memcache was down in all regions for 1 hour 50 minutes when its global datacenter mapping could not be readGoogle Cloud · 2017same root cause · shared In-memory cache
IL-0325DigitalOcean's control panel and API were down for 4 hours 56 minutes after the production primary database was deletedDigitalOcean · 2017same root cause · shared Database replication
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0215
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0215/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.