Cases3 cases

1 to 3 of 3

1. IL-0213 Roblox was down for 73 hours because key-value writes on its single Consul cluster slowed from under 300 ms to 2 seconds

   Roblox2021

   Symptoms

   On the afternoon of 28 October 2021 (13:37 PST), Vault performance at Roblox degraded and a single Consul server showed high CPU load. Consul cluster metrics showed that writes to its key-value (KV) store had a 50th percentile latency of 2 seconds instead of the usual under 300 ms. Players were not yet affected.

   High latencySustained high node CPUService outageMonitoring outage

   Trigger

   On 27 October at 14:00, one day before the outage, Roblox enabled the Consul streaming feature on the backend service responsible for traffic routing and, to prepare for end-of-year traffic, added 50% more nodes to traffic routing.

   Root cause

   Software defect

   Component

   Coordination and key-value storeService discoveryIn-memory cache

   Contributing factor

   Single point of failureOverlapping failuresMonitoring blind spotHarmful manual actionChange with a delayed effect

   Detection

   Roblox engineers began investigating when Vault performance degraded and a single Consul server had high CPU load, before players were affected. Consul cluster metrics then showed elevated KV write latency.

   Remediation

   • Replaced one Consul cluster node on the suspicion of degraded hardware
   • Replaced all Consul nodes with 128-core machines with faster NVMe SSDs
   • Shut down the Consul cluster and reset its state from a snapshot taken at the start of the outage, with access blocked by iptables
   • Blocked external traffic, disabled all non-essential Consul use, scaled services down and lowered health check frequency from 60 seconds to 10 minutes
   • Moved the Consul cluster back to 64-core servers
   • Disabled the streaming feature for all Consul systems, which brought KV write latency back to 300ms
   • Kept the slow Consul servers from staying elected as leader
   • Redeployed the caching system and fixed its wrong scheduling data and failed deployments
   • Restarted the stopped services at the right capacity levels
   • Let players back in with DNS steering in roughly 10% steps while checking database load, cache performance and stability

   Action items

   • HashiCorp builds laboratory benchmarks and scale tests that reproduce the streaming contention
   • HashiCorp improves the design of streaming so it does not contend under extreme load
   • Compact the BoltDB database with existing BoltDB tooling
   • Remove the circular dependency between the telemetry systems and Consul
   • Extend telemetry with targeted alerts on Consul and BoltDB performance and with visibility into traffic between services and Consul
   • Run the backend in an additional, geographically distinct data center and in multiple availability zones
   • Split critical services into their own dedicated Consul clusters
   • Move data that core services keep directly in the Consul KV store to a more suitable storage system
   • Delete obsolete KV data from Consul
   • Upgrade to a Consul version that replaces BoltDB with bbolt, in Q1
   • Redesign cache deployment so the cache system can be brought up quickly from a standing start
   • Deploy Nomad enhancements that make it easier to turn up large jobs after a long period of unavailability
   • Use mechanisms for faster machine configuration changes
   • Reintroduce streaming only after a new implementation has been tested at Roblox's scale and workload

   Lessons

   Two mostly unrelated problems deep inside Consul, contention in streaming under very high read and write load and BoltDB freelist growth, were hard to diagnose, and that difficulty was the main reason the outage lasted so long. A single Consul cluster serving every workload made the impact larger, and monitoring that depended on Consul hid the cause when it was needed most. Faster hardware did not help and may have hurt, because the contention was worse on NUMA machines. HashiCorp had benchmarked streaming at a similar scale, but the problem needed both a large number of streams and a high churn rate. Restarting a platform from a fully stopped state, including warming caches, took a long time and needed tools built for bootstrapping rather than incremental change.

   View case
2. IL-0588 AppOptics metrics APIs returned 5xx as Kafka partitions stayed leaderless after an AWS power event hit their ZooKeeper

   AppOptics2018

   Symptoms

   On May 31, 2018 from 21:53 UTC the metrics APIs of AppOptics and Librato returned a 5xx for a majority of incoming metric data. APM metrics and traces sent from AppOptics APM were affected, and many alert notifications were delayed.

   High error rateFailover failureMonitoring outage

   Trigger

   A power event in one of AWS's availability zones in us-east-1 cut off nodes in several critical parts of the AppOptics and Librato infrastructure, including the ZooKeeper node that was then the leader of the ensemble coordinating the Kafka brokers.

   Root cause

   Cloud provider outage

   Component

   Message brokerCoordination and key-value store

   Contributing factor

   Single point of failureExcessive blast radiusMonitoring blind spot

   Remediation

   • Restarted one ZooKeeper node by hand to force a leader election, after which the ensemble regained quorum
   • Restarted every Kafka broker in a rolling restart until all partitions were back online
   • Worked through the backlog, including the backlog of alerts, and resumed normal operations

   Action items

   • Isolate the Kafka clusters better across availability zones and regions
   • Stop sharing one ZooKeeper ensemble across Kafka clusters
   • Run game days on this failure and check whether newer Kafka versions fix it
   • Put in place more checks that notify the team earlier about subsystems that need extra help to recover

   Lessons

   Running services across several availability zones and more than one Kafka cluster did not isolate the pipeline from the failure of one zone, because partitions of different clusters sat in the same zones and the clusters shared one ZooKeeper ensemble. More operational monitoring of the subsystems that needed help to recover would have led to a faster recovery.

   View case
3. IL-0362 PagerDuty could not dispatch notifications for 18 minutes when a peering point shared by two AWS regions degraded

   PagerDuty2013sev1

   Symptoms

   On Saturday 13 April 2013 at 8:11am Pacific Time, PagerDuty's on-call engineer was paged about one node of the notification dispatch system, and an attempt at 8:13 to bring the node back failed. Customers saw delayed notifications and 500 errors on PagerDuty's API endpoints.

   High latencyHigh error rateIntermittent connection failures

   Trigger

   At 7:57am Pacific Time, according to AWS, a peering point in Northern California began to degrade, causing connectivity problems for the two AWS regions that PagerDuty used as two of its three datacenters.

   Root cause

   Cloud provider outage

   Component

   Transit gateway or network interconnectCoordination and key-value store

   Contributing factor

   Single point of failureUnverified assumptionMonitoring blind spot

   Detection

   The PagerDuty on-call engineer was paged at 8:11am about an issue with one node of the notification dispatch system, and at 8:18 monitoring detected multiple-provider failures for notifications.

   Remediation

   • Restarted most of the failed coordinator processes by hand
   • The notification dispatch system reached quorum again and processed all queued notifications

   Action items

   • Add more logging and aggregate logs into a single searchable source
   • Add a process watcher that restarts failed coordinator processes automatically
   • Build a dashboard that shows inter-host connectivity
   • Train engineering staff on Cassandra and ZooKeeper
   • Investigate moving off one of the AWS regions to a new hosting provider and datacenter that avoids a single point of failure

   Lessons

   Datacenters chosen to be independent and physically separate can still share a network peering point, and a design built to survive the loss of one datacenter did not survive losing two at once.

   View case

By root cause2