Cases2 cases

1 to 2 of 2

1. IL-0335 A bugcheck in the primary SQL Server put the Stack Exchange network into read-only mode and then offline for 12 minutes

   Stack Exchange2017

   Symptoms

   On January 24, 2017 at 17:53 UTC the Stack Exchange network, including Stack Overflow, went into read-only mode for about 5 minutes and then went offline for 12 minutes. It took the team 2 minutes to notice the problem, 5 minutes to locate its source and 10 minutes to restore service.

   Writes rejectedService outageFailover failureData loss

   Trigger

   At 17:53 UTC the primary SQL Server, NY-SQL02, triggered a bugcheck in the SQL Server process.

   Root cause

   unknown

   Component

   Database

   Contributing factor

   Single point of failure

   Remediation

   • Restarted the SQL service on NY-SQL02, which brought the network back online in read-only mode
   • Put the sites back into read-write mode after a sanity check of SQL health
   • Brought the network online without waiting for the memory dump to complete
   • Took NY-SQL02 out of production for testing, including a memory test, and ran the normal load on NY-SQL01
   • Brought the SQL cluster up to 2016 SP1 CU1

   View case
2. IL-0387 Mailgun MongoDB secondaries were overloaded by rising connections, and a rollback then moved reads onto the primary too

   Mailgun2017

   Symptoms

   On 12 January 2017 at 14:09 UTC, Mailgun engineers began responding to alerts for the click and open tracking services and the primary MongoDB clusters in one production region. Within ten minutes the engineers found that the MongoDB secondary servers, which serve most reads, were under heavy load and slow to answer, which also caused the tracking errors. The status page reported intermittent timeouts on the Control Panel, API, SMTP and tracking links.

   Database overloadIntermittent timeoutsHigh error rate

   Trigger

   A gradual increase in connections to the MongoDB secondary servers, which coincided with several MongoDB configuration changes made the day before. A rollback of those changes then also introduced, by mistake, a change that moved the API and SMTP services from the secondaries to the primary.

   Root cause

   unknown

   Component

   Database

   Contributing factor

   Harmful manual actionSingle point of failure

   Detection

   Alerts for the click and open tracking services and the primary MongoDB clusters at 14:09 UTC.

   Remediation

   • Rolled back the previous day's MongoDB configuration changes through several pull requests from 15:15 UTC
   • Resized the MongoDB servers to handle more connections
   • Deployed connection limits at the MongoDB server level
   • Limited the number of connections the services could open to the MongoDB primary
   • Redeployed the configuration that makes services prefer secondary servers for reads

   Action items

   • Run the tracking infrastructure across two independent regions
   • Limit the number of connections every service can open to the database
   • Permanently increase the size of the primary Mongo cluster and roll the change out to the other data centers
   • Let click redirection keep working when downstream services fail

   View case

By root cause1