Cases2 cases

1 to 2 of 2

1. IL-0241 A database shard could not elect a new primary, and the Stripe API failed most requests in two periods on one day

   Stripe2019

   Symptoms

   On 10 July 2019 the Stripe API failed a substantial majority of requests twice, from 16:35 to 17:02 UTC and from 21:14 to 22:47 UTC. In the first period automated monitoring detected within a minute that a database cluster had failed to elect a new primary.

   High error rateFailover failureWrites rejectedIntermittent timeoutsSustained high node CPU

   Trigger

   The primary node of a database cluster failed at 16:35 UTC while one shard had several stalled nodes. The second period was set off by rolling the election protocol back to the previous minor version at 20:42 UTC.

   Root cause

   Software defect

   Component

   DatabaseDatabase failover manager

   Contributing factor

   Change with a delayed effectMonitoring blind spotExcessive blast radiusHarmful manual actionUnverified assumptionOverlapping failures

   Detection

   Automated monitoring detected the failed election within a minute, and the team was alerted at 16:36 UTC. The second period was announced at 21:14 UTC by automated alerts that some shards in the cluster were unavailable.

   Remediation

   • Restarted all nodes of the database cluster, which forced a successful primary election
   • Updated the production configuration and restarted the database cluster's nodes
   • Ramped traffic back up, putting services for user-initiated API requests first

   Action items

   • Alert when nodes stop reporting their replication lag
   • Alert when a shard enters a state that could trigger the election fault
   • Work with the database maintainers on a fix for the underlying fault
   • Add circuit breaking on failed operations to particular database clusters
   • Use fault isolation to contain the impact of a single failed shard
   • Limit the resources used by clients that keep retrying failed requests
   • Add procedures and tooling that make rapid configuration changes during incident response safer

   View case
2. IL-0387 Mailgun MongoDB secondaries were overloaded by rising connections, and a rollback then moved reads onto the primary too

   Mailgun2017

   Symptoms

   On 12 January 2017 at 14:09 UTC, Mailgun engineers began responding to alerts for the click and open tracking services and the primary MongoDB clusters in one production region. Within ten minutes the engineers found that the MongoDB secondary servers, which serve most reads, were under heavy load and slow to answer, which also caused the tracking errors. The status page reported intermittent timeouts on the Control Panel, API, SMTP and tracking links.

   Database overloadIntermittent timeoutsHigh error rate

   Trigger

   A gradual increase in connections to the MongoDB secondary servers, which coincided with several MongoDB configuration changes made the day before. A rollback of those changes then also introduced, by mistake, a change that moved the API and SMTP services from the secondaries to the primary.

   Root cause

   unknown

   Component

   Database

   Contributing factor

   Harmful manual actionSingle point of failure

   Detection

   Alerts for the click and open tracking services and the primary MongoDB clusters at 14:09 UTC.

   Remediation

   • Rolled back the previous day's MongoDB configuration changes through several pull requests from 15:15 UTC
   • Resized the MongoDB servers to handle more connections
   • Deployed connection limits at the MongoDB server level
   • Limited the number of connections the services could open to the MongoDB primary
   • Redeployed the configuration that makes services prefer secondary servers for reads

   Action items

   • Run the tracking infrastructure across two independent regions
   • Limit the number of connections every service can open to the database
   • Permanently increase the size of the primary Mongo cluster and roll the change out to the other data centers
   • Let click redirection keep working when downstream services fail

   View case

By root cause2