Cases7 cases

1 to 5 of 7

1. IL-0376 A maintenance update set off needless full syncs between Upstash Redis replicas in eu-west-1, slowing the databases

   Upstash2025

   Symptoms

   On 25 February 2025, regional Upstash Redis databases in AWS eu-west-1 slowed down after a maintenance update, with very high CPU load on databases that have a replica in that region. Latency stayed high until about 21:23 UTC, even after the affected databases were unthrottled at 15:06 UTC.

   High latencySustained high node CPU

   Trigger

   A maintenance update to the regional Upstash Redis databases in AWS eu-west-1.

   Root cause

   Software defect

   Component

   In-memory cacheDatabase replication

   Contributing factor

   Automation amplification

   Remediation

   • Added more resources to the regional eu-west-1 cluster
   • Unthrottled the affected databases at 15:06 UTC and enabled more throughput

   Action items

   • Deploy a fix that stops regional databases from starting unnecessary full synchronisations
   • Contact regional database users about migrating to Upstash Global, which does not have this issue

   View case
2. IL-0216 Wikimedia on-wiki search was slow or rejected after a datacenter switchover overloaded one Elasticsearch node in codfw

   Wikimedia Foundation2023

   Symptoms

   Right after the primary datacenter was switched from EQIAD to CODFW at 14:00 UTC on 20 September 2023, on-wiki search on all Wikimedia wikis became delayed or unavailable for a subset of users. At 14:08 the alert "CirrusSearch codfw 95th percentile latency - more_like" fired and reached IRC and the Search Platform email list as "PROBLEM alert - graphite1005/CirrusSearch codfw 95th percentile latency - more_like is CRITICAL".

   High latencyHigh error rateLoad imbalanceMisrouted requests

   Trigger

   At 14:00 UTC the primary datacenter was switched from EQIAD to CODFW. more_like queries that were expected to go to eqiad, the non-primary datacenter, went to codfw instead, apparently because RESTBase sent them as POST rather than GET.

   Root cause

   Software defect

   Component

   Elasticsearch or OpenSearchIn-memory cache

   Contributing factor

   Unverified assumptionDependence on a warm cacheNo runbook

   Detection

   Automated monitoring caught the problem, but not quickly enough to prevent user-facing impact: the Grafana alert "CirrusSearch codfw 95th percentile latency - more_like" went to IRC and the Search Platform email list at 14:08. No one was paged; SREs were already watching closely because of the datacenter switchover. At 14:14 Hashar pinged #wikimedia-search and engineers began troubleshooting.

   Remediation

   • Banned the overloaded node elastic2044 from the Elasticsearch cluster

   Action items

   • Work out which Elasticsearch steps are needed before a datacenter switchover
   • Update the Elasticsearch section of the datacenter switchover documentation
   • Review the suggested switchover improvements in T285347 and decide whether to implement them
   • Make the RESTBase /related/{article} endpoint send GET instead of POST so the CirrusSearch query cache stays warm in both datacenters
   • Review the shard balance of busy wikis, especially enwiki_content

   Lessons

   Monitoring caught the problem, but not quickly enough to prevent user-facing impact. The heightened attention during the switchover meant the impact was recognised immediately, and banning a single node was quick and easy. The mitigation was not guided by an existing runbook, and the shards on the overloaded host were not checked before it was banned.

   View case
3. IL-0217 Honeycomb's main MySQL database locked up under cache-miss reads and ran out of connections, a 68-minute total outage

   Honeycomb2023

   Symptoms

   Hours after a routine switch between two clusters of Retriever, Honeycomb's storage and query engine, late on 24 July 2023, the internal SLO for Shepherd, the ingest service, began to burn slowly. The performance was only marginally slow and inconsistent, with no good explanation, so engineers treated it as minor and left it for the morning.

   High latencyDatabase overloadLock contention or deadlockService outage

   Trigger

   Late on 24 July Honeycomb switched between two Retriever clusters to avoid a subtle bug, and the new cluster stopped writing the last-written timestamps the schema cache depends on. On the morning of 25 July a feature flag flip meant to move those writes back stopped the remaining writes entirely.

   Root cause

   Software defect

   Component

   DatabaseIn-memory cache

   Contributing factor

   Dependence on a warm cacheIgnored early signalHarmful manual actionUnverified assumption

   Detection

   Hours after the cluster switch, the internal SLO for the ingest service Shepherd started burning slowly. Engineers left it for the morning, when a chart showed that Retriever's database calls updating the last-written records had stopped.

   Remediation

   • Flipped only the writes back to the other Retriever cluster with a feature flag, which did not fix the problem
   • Set up circuit breakers that denied all Shepherd traffic with a 5xx error to protect the database
   • Failed the hard-locked database over to a replica
   • Restarted all Retriever hosts so they could provide data for the cache
   • Set the last-written timestamp of every schema with traffic in the last day to now, so the schema store reloaded the cache
   • Removed the circuit breaker and restarted the query hosts that failed internal checks

   Action items

   • Finish the Retriever cluster migration and remove all code that could disable writes behind the cache
   • Explore ways to strengthen the cache further
   • Lower the contention put on the database during update storms from new schemas
   • Stabilise the performance costs of some operations in the system
   • Study instrumentation and experimentation that detect and handle such edge cases better

   Lessons

   Locally good decisions combined into the outage: the query engine wrote the timestamps because it already saw all the data, the shared cache helped scale ingest and untangle the main database, the cluster switch was done to avoid a bug, frequent deploys hid the feature flag bug, pausing deploys during the investigation kept hosts from restarting, and past near-misses made the team expect a slow failover. Feature flags, frequent deploys, pausing deploys during incidents and learning from near-misses all contributed, although they are among the practices Honeycomb trusts most. For a rare lockup of this kind a database failover is the fastest recovery. Several recent Honeycomb incidents started from trying to avoid a subtle bug.

   View case
4. IL-0295 Pusher's Channels publish API on the mt1 cluster was largely unavailable for 11 minutes while a Redis master failed over

   Pusher2022

   Symptoms

   At 10:42 UTC on 11 April 2022, the API for publishing messages to Pusher Channels on the mt1 cluster became largely unavailable, and from about 10:44 requests to the MT1 API returned more errors.

   Service outageHigh latencySplit brain

   Trigger

   An engineer restarted the master node of an mt1 Redis group to apply a larger replication buffer, after restarting both replicas of that group.

   Root cause

   Software defect

   Component

   In-memory cacheDatabase failover manager

   Contributing factor

   Unverified assumptionUnrepresentative staging

   Detection

   Pusher saw error rates rise for requests to the MT1 API from about 10:44 UTC.

   Action items

   • Fix the Redis shutdown script to check that failover has completed and the node is no longer a master before terminating

   View case
5. IL-0213 Roblox was down for 73 hours because key-value writes on its single Consul cluster slowed from under 300 ms to 2 seconds

   Roblox2021

   Symptoms

   On the afternoon of 28 October 2021 (13:37 PST), Vault performance at Roblox degraded and a single Consul server showed high CPU load. Consul cluster metrics showed that writes to its key-value (KV) store had a 50th percentile latency of 2 seconds instead of the usual under 300 ms. Players were not yet affected.

   High latencySustained high node CPUService outageMonitoring outage

   Trigger

   On 27 October at 14:00, one day before the outage, Roblox enabled the Consul streaming feature on the backend service responsible for traffic routing and, to prepare for end-of-year traffic, added 50% more nodes to traffic routing.

   Root cause

   Software defect

   Component

   Coordination and key-value storeService discoveryIn-memory cache

   Contributing factor

   Single point of failureOverlapping failuresMonitoring blind spotHarmful manual actionChange with a delayed effect

   Detection

   Roblox engineers began investigating when Vault performance degraded and a single Consul server had high CPU load, before players were affected. Consul cluster metrics then showed elevated KV write latency.

   Remediation

   • Replaced one Consul cluster node on the suspicion of degraded hardware
   • Replaced all Consul nodes with 128-core machines with faster NVMe SSDs
   • Shut down the Consul cluster and reset its state from a snapshot taken at the start of the outage, with access blocked by iptables
   • Blocked external traffic, disabled all non-essential Consul use, scaled services down and lowered health check frequency from 60 seconds to 10 minutes
   • Moved the Consul cluster back to 64-core servers
   • Disabled the streaming feature for all Consul systems, which brought KV write latency back to 300ms
   • Kept the slow Consul servers from staying elected as leader
   • Redeployed the caching system and fixed its wrong scheduling data and failed deployments
   • Restarted the stopped services at the right capacity levels
   • Let players back in with DNS steering in roughly 10% steps while checking database load, cache performance and stability

   Action items

   • HashiCorp builds laboratory benchmarks and scale tests that reproduce the streaming contention
   • HashiCorp improves the design of streaming so it does not contend under extreme load
   • Compact the BoltDB database with existing BoltDB tooling
   • Remove the circular dependency between the telemetry systems and Consul
   • Extend telemetry with targeted alerts on Consul and BoltDB performance and with visibility into traffic between services and Consul
   • Run the backend in an additional, geographically distinct data center and in multiple availability zones
   • Split critical services into their own dedicated Consul clusters
   • Move data that core services keep directly in the Consul KV store to a more suitable storage system
   • Delete obsolete KV data from Consul
   • Upgrade to a Consul version that replaces BoltDB with bbolt, in Q1
   • Redesign cache deployment so the cache system can be brought up quickly from a standing start
   • Deploy Nomad enhancements that make it easier to turn up large jobs after a long period of unavailability
   • Use mechanisms for faster machine configuration changes
   • Reintroduce streaming only after a new implementation has been tested at Roblox's scale and workload

   Lessons

   Two mostly unrelated problems deep inside Consul, contention in streaming under very high read and write load and BoltDB freelist growth, were hard to diagnose, and that difficulty was the main reason the outage lasted so long. A single Consul cluster serving every workload made the impact larger, and monitoring that depended on Consul hid the cause when it was needed most. Faster hardware did not help and may have hurt, because the contention was worse on NUMA machines. HashiCorp had benchmarked streaming at a similar scale, but the problem needed both a large number of streams and a high churn rate. Restarting a platform from a fully stopped state, including warming caches, took a long time and needed tools built for bootstrapping rather than incremental change.

   View case

By root cause1