Cases16 cases

13 to 16 of 16

1. IL-0335 A bugcheck in the primary SQL Server put the Stack Exchange network into read-only mode and then offline for 12 minutes

   Stack Exchange2017

   Symptoms

   On January 24, 2017 at 17:53 UTC the Stack Exchange network, including Stack Overflow, went into read-only mode for about 5 minutes and then went offline for 12 minutes. It took the team 2 minutes to notice the problem, 5 minutes to locate its source and 10 minutes to restore service.

   Writes rejectedService outageFailover failureData loss

   Trigger

   At 17:53 UTC the primary SQL Server, NY-SQL02, triggered a bugcheck in the SQL Server process.

   Root cause

   unknown

   Component

   Database

   Contributing factor

   Single point of failure

   Remediation

   • Restarted the SQL service on NY-SQL02, which brought the network back online in read-only mode
   • Put the sites back into read-write mode after a sanity check of SQL health
   • Brought the network online without waiting for the memory dump to complete
   • Took NY-SQL02 out of production for testing, including a memory test, and ran the normal load on NY-SQL01
   • Brought the SQL cluster up to 2016 SP1 CU1

   View case
2. IL-0520 logs.cloud.gov trapped users in a login redirect loop because its full Redis session store stopped saving sessions

   cloud.gov2016

   Symptoms

   At 7:12pm Eastern Time on 5 July 2016, logs.cloud.gov, the cloud.gov log search service, sent users into a login redirect loop, so they could not log in to the service.

   Service outageWrites rejected

   Root cause

   Capacity exhaustion

   Component

   In-memory cacheElasticsearch or OpenSearch

   Contributing factor

   Overlapping failuresMonitoring blind spot

   Remediation

   • Moved Redis on the logsearch service to a much larger memory instance so it could keep logs while Elasticsearch was down
   • Corrected the Elasticsearch memory limits and the /tmp noexec hardening setting and redeployed the cluster
   • Redeployed the Kibana instance and verified the logs once the cluster deployment was complete

   Action items

   • Add a failure alert that checks whether logs are coming through
   • Make Redis auto-scaling or highly available so that failures in more fragile downstream components do not cause data loss
   • Extend the incident response plan to start with events that are not security-specific
   • Create a chaos bot for the logs.cloud.gov service

   View case
3. IL-0215 Twilio's billing Redis master lost all balances and went read-only, and auto-recharge charged customer cards repeatedly

   Twilio2013

   Symptoms

   At 2:39 AM PDT (9:39 UTC) on 18 July 2013, services that relied on the redis-master of Twilio's billing system began to fail because the load on its host had become extreme. At 2:42 AM PDT monitoring alerted the on-call engineers to a failure in the Redis cluster.

   Database overloadData lossWrites rejected

   Trigger

   At 1:35 AM PDT on 18 July a loss of network connectivity disconnected all billing redis-slaves from the redis-master at once, and all of them requested a full synchronization at the same time. On-call engineers then restarted the overloaded redis-master.

   Root cause

   Configuration change

   Component

   In-memory cacheDatabase replication

   Contributing factor

   Retry stormHarmful manual actionUnsafe failure behaviour

   Detection

   At 2:42 AM PDT monitoring alerted the on-call engineers to a failure in the Redis cluster. At 3:28 AM PDT the billing system monitoring reported the anomalous charges, and at 12:36 PM PDT monitoring detected the recurrence.

   Remediation

   • Restarted the redis-master to relieve the high load, which loaded a wrong configuration
   • Took the billing system offline to stop further credit card charges
   • Restored service to all suspended accounts
   • Restored the lost account balances from the independent double-bookkeeping record in the relational datastore
   • Took billing offline again after the recurrence and reactivated the newly suspended accounts
   • Refunded all erroneous credit card charges
   • Gave all affected accounts a credit of 10% of their last 30 days of spend
   • Replaced the original Redis cluster and corrected the redis-master configuration
   • Activated the billing system progressively for groups of customer accounts

   Action items

   • Disable Redis restarts on the redis-master and recover the master by pivoting a slave instead
   • Add fail-safes so the billing system neither suspends accounts nor charges cards when balances are missing or cannot be written
   • Validate the billing system against the double-bookkeeping databases in real time

   Lessons

   Losing the in-flight balances and the ability to write balances at the same time exposed a flaw in the auto-recharge system: it failed in a dangerous way, charging cards and suspending accounts instead of stopping.

   View case
4. IL-0320 Postmark's primary MySQL disk filled but MySQL kept running, so MySQL-MMM never failed over and sending was delayed

   Postmark2011

   Symptoms

   At about 8:15am EDT Postmark's primary MySQL server ran out of disk space, and email sending, the API and the web interface were delayed. Disk space alerts fired, but the space had grown much faster than usual and MySQL was already struggling when the alerts were noticed. MySQL kept running while it failed to write some data to disk, so MySQL-MMM, which normally detects problems and fails over to a healthy server, never triggered a failover; the team spent about 30 minutes trying to free logs and disk space before failing over by hand.

   Node disk fullWrites rejectedFailover failure

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase failover manager

   Contributing factor

   –

   Detection

   Per-server monitoring raised disk space alerts, but the disk filled so quickly that MySQL was already struggling by the time the alerts were noticed.

   Remediation

   • Tried to free up logs and disk space on the primary server
   • Failed over manually to the second MySQL server
   • Recovered the jobs that had failed
   • Sent the emails that each server had queued locally while the database was unavailable

   Action items

   • Upgrade the MySQL cluster with a focus on availability and performance
   • Make the alerts more sensitive to catch problems sooner

   Lessons

   Looking back, Postmark said it should have failed over to the second server right away instead of first making sure replication was healthy.

   View case

By root cause7