Cases2 cases

1 to 2 of 2

1. IL-0213 Roblox was down for 73 hours because key-value writes on its single Consul cluster slowed from under 300 ms to 2 seconds

   Roblox2021

   Symptoms

   On the afternoon of 28 October 2021 (13:37 PST), Vault performance at Roblox degraded and a single Consul server showed high CPU load. Consul cluster metrics showed that writes to its key-value (KV) store had a 50th percentile latency of 2 seconds instead of the usual under 300 ms. Players were not yet affected.

   High latencySustained high node CPUService outageMonitoring outage

   Trigger

   On 27 October at 14:00, one day before the outage, Roblox enabled the Consul streaming feature on the backend service responsible for traffic routing and, to prepare for end-of-year traffic, added 50% more nodes to traffic routing.

   Root cause

   Software defect

   Component

   Coordination and key-value storeService discoveryIn-memory cache

   Contributing factor

   Single point of failureOverlapping failuresMonitoring blind spotHarmful manual actionChange with a delayed effect

   Detection

   Roblox engineers began investigating when Vault performance degraded and a single Consul server had high CPU load, before players were affected. Consul cluster metrics then showed elevated KV write latency.

   Remediation

   • Replaced one Consul cluster node on the suspicion of degraded hardware
   • Replaced all Consul nodes with 128-core machines with faster NVMe SSDs
   • Shut down the Consul cluster and reset its state from a snapshot taken at the start of the outage, with access blocked by iptables
   • Blocked external traffic, disabled all non-essential Consul use, scaled services down and lowered health check frequency from 60 seconds to 10 minutes
   • Moved the Consul cluster back to 64-core servers
   • Disabled the streaming feature for all Consul systems, which brought KV write latency back to 300ms
   • Kept the slow Consul servers from staying elected as leader
   • Redeployed the caching system and fixed its wrong scheduling data and failed deployments
   • Restarted the stopped services at the right capacity levels
   • Let players back in with DNS steering in roughly 10% steps while checking database load, cache performance and stability

   Action items

   • HashiCorp builds laboratory benchmarks and scale tests that reproduce the streaming contention
   • HashiCorp improves the design of streaming so it does not contend under extreme load
   • Compact the BoltDB database with existing BoltDB tooling
   • Remove the circular dependency between the telemetry systems and Consul
   • Extend telemetry with targeted alerts on Consul and BoltDB performance and with visibility into traffic between services and Consul
   • Run the backend in an additional, geographically distinct data center and in multiple availability zones
   • Split critical services into their own dedicated Consul clusters
   • Move data that core services keep directly in the Consul KV store to a more suitable storage system
   • Delete obsolete KV data from Consul
   • Upgrade to a Consul version that replaces BoltDB with bbolt, in Q1
   • Redesign cache deployment so the cache system can be brought up quickly from a standing start
   • Deploy Nomad enhancements that make it easier to turn up large jobs after a long period of unavailability
   • Use mechanisms for faster machine configuration changes
   • Reintroduce streaming only after a new implementation has been tested at Roblox's scale and workload

   Lessons

   Two mostly unrelated problems deep inside Consul, contention in streaming under very high read and write load and BoltDB freelist growth, were hard to diagnose, and that difficulty was the main reason the outage lasted so long. A single Consul cluster serving every workload made the impact larger, and monitoring that depended on Consul hid the cause when it was needed most. Faster hardware did not help and may have hurt, because the contention was worse on NUMA machines. HashiCorp had benchmarked streaming at a similar scale, but the problem needed both a large number of streams and a high churn rate. Restarting a platform from a fully stopped state, including warming caches, took a long time and needed tools built for bootstrapping rather than incremental change.

   View case
2. IL-0275 Parse.ly's Kafka brokers hit network limits and failed one after another, stopping the analytics pipeline for hours

   Parse.ly2015

   Symptoms

   On Thursday night, 26 March 2015, and again on Friday, 27 March, Parse.ly's data processing backend stopped. Monitoring tools such as Munin, Sentry and Pingdom reported failures widely, PagerDuty woke the on-call engineer, and no new analytics data appeared in the dashboards at dash.parsely.com and preview.parsely.com for several hours.

   Service outageDNS resolution failures

   Trigger

   Data volume reached a point where the Kafka brokers, running on modest Amazon EC2 nodes without high network capacity, hit operating system network limits under the outbound read traffic and became unavailable.

   Root cause

   Capacity exhaustion

   Component

   Message brokerService discovery

   Contributing factor

   Monitoring blind spotUnknown limitNo headroomHarmful manual action

   Detection

   Munin, Sentry and Pingdom reported the failures widely, Graphite graphs showed them, and PagerDuty woke the on-call engineer when the first outages started in the night.

   Remediation

   • Added new brokers to expand the Kafka cluster
   • Corrected the configuration of the new brokers and deployed them to the cluster, after which network utilization per node dropped and processing resumed
   • Turned the downstream consumer systems back on slowly through Apache Storm and reprocessed the backlog
   • Began restoring the data to customer accounts from the raw data archived in Amazon S3

   Lessons

   CPU and disk I/O are good signs of machine health, but creeping network utilization is harder to monitor, because 100% of network capacity is not well defined on cloud VMs that share host machines. When every node of a cluster runs close to the same limit, losing one node can take down the whole cluster, so failure planning should cover an entire subsystem going offline, not just one or two machines. Keeping raw data archival in simple files and S3 uploads, separate from the Kafka stream, meant there was no risk of permanent data loss.

   View case

By root cause2