Cases8 cases

3 to 7 of 8

1. IL-0270 Discord's ScyllaDB authentication cluster overloaded during an OS upgrade, and logins failed for about 50 minutes

   Discord2023

   Symptoms

   On 6 November 2023 at 10:52 AM PST, when an upgrade script drained all connections from zone [c] of the scylla-authentication ScyllaDB cluster, Discord's availability, measured as the share of HTTP requests that return a 200-level response, fell at once to about 25%. Users could not connect, log in, read or send messages.

   Service outageDatabase overloadHigh latency

   Trigger

   During an OS upgrade that takes one zone of a ScyllaDB cluster offline at a time, the upgrade script drained all connections from zone [c] of scylla-authentication at 10:52 AM, while one node in the just-upgraded zone [b] had a degraded NVMe Local SSD.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   No headroomOverlapping failuresDependence on a warm cacheUnverified assumptionIgnored early signalRetry storm

   Detection

   An incident response call had already started when the first alerts fired at 10:56 AM, four minutes after the start. The time spent by downstream service graph showed the authentication service spending far more time on user requests than normal, and the authentication alerts fired moments later.

   Remediation

   • Turned off sending typing events and marking messages as read to shed load from the authentication cluster
   • Waited for the zone [c] RAID rebuild instead of restarting nodes, and the two zone [c] nodes came back online at 11:41 AM
   • Restarted the two zone [b] nodes one at a time

   Action items

   • Double the size of the scylla-authentication cluster so each node takes half as many requests
   • Rebuild scylla-authentication-b-2 with new, non-degraded disks
   • Build a way to detect disk degradation that is too subtle to affect overall latency but still threatens reliability
   • Move upgrade operations into a more automatic control plane for the Scylla clusters that analyses cluster state, such as cache hit rate
   • Increase the redundancy of the scylla-authentication cluster; the options being explored, not yet decided, are a higher replication factor or a hot-standby cluster in another region

   Lessons

   Taking a whole zone out of a three-zone cluster leaves no margin, so one additional slow node in another zone was enough to overload it. For some Scylla clusters the row cache hit rate of restarted nodes must reach a safe level before the next zone goes down, and a short zonal test beforehand gave false confidence. The small but critical authentication cluster needs to be run differently from Discord's much larger Scylla clusters, with more redundancy.

   View case
2. IL-0217 Honeycomb's main MySQL database locked up under cache-miss reads and ran out of connections, a 68-minute total outage

   Honeycomb2023

   Symptoms

   Hours after a routine switch between two clusters of Retriever, Honeycomb's storage and query engine, late on 24 July 2023, the internal SLO for Shepherd, the ingest service, began to burn slowly. The performance was only marginally slow and inconsistent, with no good explanation, so engineers treated it as minor and left it for the morning.

   High latencyDatabase overloadLock contention or deadlockService outage

   Trigger

   Late on 24 July Honeycomb switched between two Retriever clusters to avoid a subtle bug, and the new cluster stopped writing the last-written timestamps the schema cache depends on. On the morning of 25 July a feature flag flip meant to move those writes back stopped the remaining writes entirely.

   Root cause

   Software defect

   Component

   DatabaseIn-memory cache

   Contributing factor

   Dependence on a warm cacheIgnored early signalHarmful manual actionUnverified assumption

   Detection

   Hours after the cluster switch, the internal SLO for the ingest service Shepherd started burning slowly. Engineers left it for the morning, when a chart showed that Retriever's database calls updating the last-written records had stopped.

   Remediation

   • Flipped only the writes back to the other Retriever cluster with a feature flag, which did not fix the problem
   • Set up circuit breakers that denied all Shepherd traffic with a 5xx error to protect the database
   • Failed the hard-locked database over to a replica
   • Restarted all Retriever hosts so they could provide data for the cache
   • Set the last-written timestamp of every schema with traffic in the last day to now, so the schema store reloaded the cache
   • Removed the circuit breaker and restarted the query hosts that failed internal checks

   Action items

   • Finish the Retriever cluster migration and remove all code that could disable writes behind the cache
   • Explore ways to strengthen the cache further
   • Lower the contention put on the database during update storms from new schemas
   • Stabilise the performance costs of some operations in the system
   • Study instrumentation and experimentation that detect and handle such edge cases better

   Lessons

   Locally good decisions combined into the outage: the query engine wrote the timestamps because it already saw all the data, the shared cache helped scale ingest and untangle the main database, the cluster switch was done to avoid a bug, frequent deploys hid the feature flag bug, pausing deploys during the investigation kept hosts from restarting, and past near-misses made the team expect a slow failover. Feature flags, frequent deploys, pausing deploys during incidents and learning from near-misses all contributed, although they are among the practices Honeycomb trusts most. For a rare lockup of this kind a database failover is the fastest recovery. Several recent Honeycomb incidents started from trying to avoid a subtle bug.

   View case
3. IL-0488 Kit's RDS MySQL write latency rose 100x and running threads jumped from 9 to 900, backing up Sidekiq job queues

   Kit2022

   Symptoms

   At 5:20 am EST on December 24, 2022, Kit's infrastructure team got its first page because the number of MySQL threads running on its RDS MySQL primary in us-east-2 had jumped 100x, from 9 to 900, at a time of day when Kit's traffic is very low. Sidekiq queues were backing up.

   Database overloadThroughput degradation

   Trigger

   An EBS interruption on the standby of the Multi-AZ RDS instance, reported by AWS as an LSE at 02:12:47 UTC on 2022-12-24, started a replaceSecondaryHostAndVolumesTask that replaced the secondary host and volumes.

   Root cause

   Cloud provider outage

   Component

   DatabaseCloud block storageDatabase replication

   Contributing factor

   Ignored early signal

   Detection

   A page fired at 5:20 am EST because MySQL threads running jumped from 9 to 900. An AWS RSS feed alert in Slack about degraded EC2 and EBS in use2-az1 had arrived the evening before and was not acted on.

   Remediation

   • Scaled the RDS storage from 12.5 TB to 13.75 TB, which put the instance through "optimizing storage" and brought write latency back to 1-2 ms

   Lessons

   Multi-AZ RDS keeps data consistent and durable, but synchronous replication means each write also waits for the standby, so a degraded EBS volume on the standby while it is being replaced slows writes on the primary. Large databases recover more slowly from such events than small ones. A single-AZ instance would not have been hit by a standby problem, but would have suffered hours of downtime if its own zone failed, and that trade-off has to be weighed when deploying production RDS instances.

   View case
4. IL-0294 CircleCI Docker and machine jobs stalled for nearly 12 hours after a VM cleanup queue lost all its RabbitMQ consumers

   CircleCI2021

   Symptoms

   From 13:30 UTC on 21 May 2021, CircleCI jobs on Docker, the Machine executor (including Windows, Mac, Arm and GPU) and remote Docker were delayed or blocked. Engineers had been paged at 13:00 for excessive delays in starting jobs, and the alerts showed CircleCI could not provision some types and classes of VM because it was hitting its CPU quotas in GCP.

   Throughput degradationCloud API rate limit exhaustedDatabase overload

   Trigger

   The upgrade of the RabbitMQ cluster from 3.8.9 to 3.8.16 on 20 May 2021, which brought in a 15 minute consumer acknowledgement timeout for all queue types.

   Root cause

   Breaking change in an upgrade

   Component

   Message brokerDatabase

   Contributing factor

   Change with a delayed effectIgnored early signalRetry stormExcessive blast radiusOverlapping failures

   Detection

   Engineers were paged at 13:00 UTC on 21 May for excessive delays in starting jobs. Earlier, a 02:31 notification that a VM destroyer queue had no consumers was handled by restarting pods, and a 10:11 alert that a VM destroy queue was backing up notified once at its threshold and did not page.

   Remediation

   • Blocked Linux machine jobs from entering the work queue
   • Had GCP raise the CPU quotas
   • Deleted old VMs by hand
   • Cancelled some workflows to free resources
   • Reset the VM services deployment so the VM destroy queues had consumers again
   • Scaled up the RDS database instance, at 18:20 and again at 19:15
   • Turned down VM provisioning and blocked remote-docker jobs to shed load
   • Scaled down all pods that created and destroyed VMs
   • Deployed a revised distribution backend that kept remote-docker jobs out of the provisioning queues
   • Brought provisioning and distribution back one component at a time and reversed the mitigation code changes

   Action items

   • Make VM provisioning more robust to changes in load so it can recover from thundering herds
   • Make VM provisioning independent so its failure does not affect Docker jobs
   • Review and improve observability of RabbitMQ and the production queues
   • Batch requests to avoid hitting GCP API rate limits
   • Improve incident response for long-running incidents

   View case
5. IL-0211 A Mandrill PostgreSQL shard shut down at transaction ID wraparound and email sending was impaired for about 40 hours

   Mailchimp2019

   Symptoms

   From 05:35 UTC on 4 February 2019 Mandrill, Mailchimp's transactional email service, sent only 80% of queued emails, because one of its PostgreSQL shards, shard4, stopped accepting writes. When Mandrill engineers opened an investigation at 11:58 UTC, they saw a sharply elevated load on shard4.

   Writes rejectedThroughput degradationLoad imbalanceNode disk full

   Trigger

   At 05:35 UTC on 4 February the transaction ID counter of shard4, the busiest Mandrill PostgreSQL shard, reached its upper limit and PostgreSQL shut the database down to protect against wraparound.

   Root cause

   Capacity exhaustion

   Component

   DatabasePostgreSQL vacuum

   Contributing factor

   Ignored early signalNo alert on limitSignal buried in noiseRetry stormMonitoring blind spot

   Detection

   The first exception was logged overnight but was buried by unrelated exceptions in the centralized error reporting. Mandrill engineers opened an investigation at 11:58 UTC and, while looking for the source of the rising load on shard4, found the wraparound at 12:56 UTC.

   Remediation

   • Put shard4 into standalone mode and started VACUUM at 14:16 UTC on 4 February
   • Stopped and restarted the vacuum with settings that made it run faster
   • Started a dump and restore of the database without the Search and Url tables while the vacuum ran
   • Replaced storage volumes on the Mandrill app servers whose disks were filling with queued jobs and logs
   • Moved the job-queue tables from shard4 to other live shards
   • Truncated the Search and Url tables on the locked database at 19:12 UTC on 5 February, after which the vacuum finished in about an hour
   • Started sending from the backlog at 21:36 UTC
   • Added alerting for XID wraparound by 22:40 UTC

   Lessons

   Mailchimp named the factors that slowed its response: knowledge of and access to Mandrill systems and monitoring rested with a few people, Mandrill logging needed a locally writable disk, logs of malformed API requests were hard to separate from real application errors, job health was hard to see, and the disk space problem made partial deployments for triage risky. What helped was trained incident commanders with clear roles (incident commanders in 3-hour rotations, scribes, technical liaisons and points of contact between teams), periodic summaries with a list of recent and open decisions, and taking time in a video meeting to reassess, which led to the truncation. Running in the cloud let the team expand storage quickly, and the team was lucky that the largest table holding up the vacuum held data it could delete.

   View case

By root cause4