Cases8 cases

3 to 7 of 8

1. IL-0270 Discord's ScyllaDB authentication cluster overloaded during an OS upgrade, and logins failed for about 50 minutes

   Discord2023

   Symptoms

   On 6 November 2023 at 10:52 AM PST, when an upgrade script drained all connections from zone [c] of the scylla-authentication ScyllaDB cluster, Discord's availability, measured as the share of HTTP requests that return a 200-level response, fell at once to about 25%. Users could not connect, log in, read or send messages.

   Service outageDatabase overloadHigh latency

   Trigger

   During an OS upgrade that takes one zone of a ScyllaDB cluster offline at a time, the upgrade script drained all connections from zone [c] of scylla-authentication at 10:52 AM, while one node in the just-upgraded zone [b] had a degraded NVMe Local SSD.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   No headroomOverlapping failuresDependence on a warm cacheUnverified assumptionIgnored early signalRetry storm

   Detection

   An incident response call had already started when the first alerts fired at 10:56 AM, four minutes after the start. The time spent by downstream service graph showed the authentication service spending far more time on user requests than normal, and the authentication alerts fired moments later.

   Remediation

   • Turned off sending typing events and marking messages as read to shed load from the authentication cluster
   • Waited for the zone [c] RAID rebuild instead of restarting nodes, and the two zone [c] nodes came back online at 11:41 AM
   • Restarted the two zone [b] nodes one at a time

   Action items

   • Double the size of the scylla-authentication cluster so each node takes half as many requests
   • Rebuild scylla-authentication-b-2 with new, non-degraded disks
   • Build a way to detect disk degradation that is too subtle to affect overall latency but still threatens reliability
   • Move upgrade operations into a more automatic control plane for the Scylla clusters that analyses cluster state, such as cache hit rate
   • Increase the redundancy of the scylla-authentication cluster; the options being explored, not yet decided, are a higher replication factor or a hot-standby cluster in another region

   Lessons

   Taking a whole zone out of a three-zone cluster leaves no margin, so one additional slow node in another zone was enough to overload it. For some Scylla clusters the row cache hit rate of restarted nodes must reach a safe level before the next zone goes down, and a short zonal test beforehand gave false confidence. The small but critical authentication cluster needs to be run differently from Discord's much larger Scylla clusters, with more redundancy.

   View case
2. IL-0567 Aurora MySQL behind Cookie Run: OvenBreak hit 100% CPU after a sales-ranking feature launch, and players could not play

   Devsisters2023

   Symptoms

   After further bug-fix deployments and reconnecting users on the launch day of the package sales ranking feature, a database alarm fired and the CPU of the Cookie Run: OvenBreak AWS Aurora MySQL database was at 100%. Database connections then failed, and players could not log in to or play the game.

   Database overloadService outage

   Trigger

   The package sales ranking feature was deployed, and further bug-fix deployments followed that reconnected users; each server update left more servers with empty caches that ran the ranking query against the database.

   Root cause

   Faulty deployment

   Component

   Database

   Contributing factor

   Inefficient queryDependence on a warm cacheUnrepresentative staging

   Detection

   A database alarm fired, and the CPU graph of the Aurora MySQL database showed 100%.

   Remediation

   • Put the game servers into maintenance
   • Added a switch for the package sales ranking feature and turned the feature off, which restored the service
   • Rewrote the ranking query to group products that share a sale start time and to drop the current-time condition
   • Moved the ranking query to a scheduler server that runs it periodically and stores one shared cached result
   • Turned the package sales ranking feature back on after both fixes

   Lessons

   A heavy query combined with per-server caches in a distributed server setup caused a problem nobody expected, so the points likely to cause trouble need to be found in advance. During an outage, restoring the service comes before finding the cause, and a switch that turns a feature off, built in ahead of time, is a safety device that helps run a stable service. The online ALTER itself went well with Percona Toolkit and was used again for later schema changes.

   View case
3. IL-0217 Honeycomb's main MySQL database locked up under cache-miss reads and ran out of connections, a 68-minute total outage

   Honeycomb2023

   Symptoms

   Hours after a routine switch between two clusters of Retriever, Honeycomb's storage and query engine, late on 24 July 2023, the internal SLO for Shepherd, the ingest service, began to burn slowly. The performance was only marginally slow and inconsistent, with no good explanation, so engineers treated it as minor and left it for the morning.

   High latencyDatabase overloadLock contention or deadlockService outage

   Trigger

   Late on 24 July Honeycomb switched between two Retriever clusters to avoid a subtle bug, and the new cluster stopped writing the last-written timestamps the schema cache depends on. On the morning of 25 July a feature flag flip meant to move those writes back stopped the remaining writes entirely.

   Root cause

   Software defect

   Component

   DatabaseIn-memory cache

   Contributing factor

   Dependence on a warm cacheIgnored early signalHarmful manual actionUnverified assumption

   Detection

   Hours after the cluster switch, the internal SLO for the ingest service Shepherd started burning slowly. Engineers left it for the morning, when a chart showed that Retriever's database calls updating the last-written records had stopped.

   Remediation

   • Flipped only the writes back to the other Retriever cluster with a feature flag, which did not fix the problem
   • Set up circuit breakers that denied all Shepherd traffic with a 5xx error to protect the database
   • Failed the hard-locked database over to a replica
   • Restarted all Retriever hosts so they could provide data for the cache
   • Set the last-written timestamp of every schema with traffic in the last day to now, so the schema store reloaded the cache
   • Removed the circuit breaker and restarted the query hosts that failed internal checks

   Action items

   • Finish the Retriever cluster migration and remove all code that could disable writes behind the cache
   • Explore ways to strengthen the cache further
   • Lower the contention put on the database during update storms from new schemas
   • Stabilise the performance costs of some operations in the system
   • Study instrumentation and experimentation that detect and handle such edge cases better

   Lessons

   Locally good decisions combined into the outage: the query engine wrote the timestamps because it already saw all the data, the shared cache helped scale ingest and untangle the main database, the cluster switch was done to avoid a bug, frequent deploys hid the feature flag bug, pausing deploys during the investigation kept hosts from restarting, and past near-misses made the team expect a slow failover. Feature flags, frequent deploys, pausing deploys during incidents and learning from near-misses all contributed, although they are among the practices Honeycomb trusts most. For a rare lockup of this kind a database failover is the fastest recovery. Several recent Honeycomb incidents started from trying to avoid a subtle bug.

   View case
4. IL-0212 Many Slack users could not connect because a Vitess keyspace for channel membership was overloaded by cache-miss queries

   Slack2022

   Symptoms

   Just after 6 a.m. Pacific Time on 22 February 2022, user tickets about problems connecting to Slack, problems reported by internal users, and pages and alerts to several engineering teams all arrived at almost the same moment. Many users could not connect: the Slack client could not finish booting, the step that fetches channel lists, preferences and recent conversations, and showed an error page.

   Database overloadHigh error rateHigh latency

   Trigger

   A percentage-based rollout step upgraded the Consul agent on 25% of the fleet, followed by a slow sequential restart of the agents. Each restart on a Memcached node removed it from the Consul service catalog, so Mcrib, the cache-tier control plane, replaced it with an empty spare or flushed it when it rejoined, and the cache hit rate dropped as Slack reached its daily peak traffic.

   Root cause

   Capacity exhaustion

   Component

   DatabaseIn-memory cacheService discovery

   Contributing factor

   Dependence on a warm cacheInefficient queryAutomation amplificationRetry stormChange during peak hours

   Detection

   User tickets about connection problems, reports from internal users and pages and alerts to several engineering teams arrived almost at the same time, just after 6 a.m. Pacific Time.

   Remediation

   • Paused the Consul agent restart operation
   • Throttled client boot requests to reduce load on the database tier
   • Cut the client boot rate limit again after a too-large increase and then raised it in smaller steps
   • Changed the GDM membership query to read from Vitess only the data missing from Memcached
   • Let the query read from replicas as well as Vitess primaries
   • Raised the client boot rate limit slowly back to normal traffic levels as the caches refilled

   Action items

   • Change the process for Consul rollouts so upgrades do not trigger this failure mode again
   • Change the Mcrib control loop so a brief network outage or partition of cache nodes does not set off the same cascading failure
   • Make the GDM membership query read from a table sharded by channel
   • Check the other cache-fronted database queries for high-volume scatter queries
   • Explore longer-term ways to make the caching tier more resilient

   Lessons

   Slack describes performance bottlenecks, a cascading failure, dependence on a warm cache and a change as the trigger. Once the system had passed the tipping point, lower cache hit rates caused high database load that in turn kept the cache from filling, and a system in that state stays there until an outside intervention, a large cut in load or an increase in capacity, changes it. Mcrib contributed not by being wrong but by being faster than the old Consul lock scheme at replacing Memcached nodes, which increased churn in the cache tier. Client retries still added load despite backoff and jitter. A brief network partition of the cache nodes could set off the same failure, and the interaction between Consul and the caching architecture is now more widely understood.

   View case
5. IL-0297 Mezmo users could not sign in and log indexing lagged after the auth cache's Redis servers were OOM killed

   Mezmo2019

   Symptoms

   On 15 May 2019, alerts and customer reports showed that the Mezmo web application did not let users sign in, and the status page reported an indexing delay: incoming logs reached live tail and search late.

   Service outageOOMKilledDatabase overloadThroughput degradation

   Trigger

   A traffic spike that the tool tracking batches of logs was not sized to handle.

   Root cause

   Capacity exhaustion

   Component

   In-memory cacheDatabase

   Contributing factor

   Dependence on a warm cache

   Detection

   Alerts and customer reports in the early morning hours triggered the escalation procedure and woke the on-call engineers.

   Remediation

   • Resized specific parts of the cluster to handle a much higher load

   Action items

   • Retool the batch tracking system to reduce its direct impact on the Redis clusters
   • Add observability instrumentation around the batch tracking feature set
   • Improve monitoring for Redis
   • Remove the bottlenecks in the pipeline that were involved in this incident

   Lessons

   Mezmo took from the incident that deep instrumentation, tracing and observability across the platform matter more as features scale up.

   View case

By root cause4