IL-0335 Stack Exchange · 2017-01-24database
A bugcheck in the primary SQL Server put the Stack Exchange network into read-only mode and then offline for 12 minutes
A bugcheck in the SQL Server process on the primary NY-SQL02 first put it into a read-only state. Stack Exchange relies on application-level logic, not automatic SQL Server failover, to keep the sites online by querying the standby SQL servers read-only, but a bug had disabled that logic, so Stack Overflow and the rest of the network went fully offline. Restarting the SQL service on NY-SQL02 brought the network back in read-only mode, and after a sanity check of the SQL server's health the sites were put back into read-write mode, about 10 minutes after the source was found. About 3.5 seconds of uncommitted transactions were rolled back. The reason for the bugcheck was not known when the postmortem was written, but logs pointed to a possibly bad DIMM. Given the server's 384GB of memory, the team brought the network back online instead of waiting for the memory dump to complete. NY-SQL02 was then taken out of production for tests including a memory test, NY-SQL01 took all of the normal load, and the SQL cluster was brought up to 2016 SP1 CU1.
- Severity
- –
- Scope
- The Stack Exchange network, including Stack Overflow, read-only for about 5 minutes and then offline for 12 minutes; about 3.5 seconds of uncommitted transactions may have been lost
- Users affected
- –
- Region
- –
- Time to detect
- 2m
- Time to resolve
- 17m
Symptoms
On January 24, 2017 at 17:53 UTC the Stack Exchange network, including Stack Overflow, went into read-only mode for about 5 minutes and then went offline for 12 minutes. It took the team 2 minutes to notice the problem, 5 minutes to locate its source and 10 minutes to restore service.
The primary SQL Server, NY-SQL02, triggered a bugcheck in the SQL Server process, which first left it read-only. Stack Exchange does not allow automatic SQL Server failovers; application logic is meant to keep the sites up by reading from the standby SQL servers in read-only mode. A bug had disabled that application-level failover, so when the primary failed the whole network went offline. About 3.5 seconds of uncommitted transactions were rolled back and may have been lost.
Symptom Writes rejectedService outageFailover failureData loss
Root cause
Root cause unknown
- Trigger
- At 17:53 UTC the primary SQL Server, NY-SQL02, triggered a bugcheck in the SQL Server process.
Detection
–
Remediation
- Restarted the SQL service on NY-SQL02, which brought the network back online in read-only mode
- Put the sites back into read-write mode after a sanity check of SQL health
- Brought the network online without waiting for the memory dump to complete
- Took NY-SQL02 out of production for testing, including a memory test, and ran the normal load on NY-SQL01
- Brought the SQL cluster up to 2016 SP1 CU1
Action items
–
Lessons
–
Sources
Related cases
IL-0545Wasabi API calls in all regions returned HTTP 5XX when the message queue stopped spreading requests over database nodesWasabi · 2024same root cause · shared Database
IL-0441A ZOZO SQL Server database stalled and applications hit timeout storms whenever batch executions peaked in salesZOZO · 2021same root cause · shared Database
IL-0387Mailgun MongoDB secondaries were overloaded by rising connections, and a rollback then moved reads onto the primary tooMailgun · 2017same root cause · shared Database
IL-0393Load on the BBC programme metadata database surged and metadata requests failed, leaving BBC iPlayer down over a weekendBBC · 2014same root cause · shared Database
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0335
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0335/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.