Cases1 case

1 to 1 of 1

1. IL-0335 A bugcheck in the primary SQL Server put the Stack Exchange network into read-only mode and then offline for 12 minutes

   Stack Exchange2017

   Symptoms

   On January 24, 2017 at 17:53 UTC the Stack Exchange network, including Stack Overflow, went into read-only mode for about 5 minutes and then went offline for 12 minutes. It took the team 2 minutes to notice the problem, 5 minutes to locate its source and 10 minutes to restore service.

   Writes rejectedService outageFailover failureData loss

   Trigger

   At 17:53 UTC the primary SQL Server, NY-SQL02, triggered a bugcheck in the SQL Server process.

   Root cause

   unknown

   Component

   Database

   Contributing factor

   Single point of failure

   Remediation

   • Restarted the SQL service on NY-SQL02, which brought the network back online in read-only mode
   • Put the sites back into read-write mode after a sanity check of SQL health
   • Brought the network online without waiting for the memory dump to complete
   • Took NY-SQL02 out of production for testing, including a memory test, and ran the normal load on NY-SQL01
   • Brought the SQL cluster up to 2016 SP1 CU1

   View case

By root cause1