Incident Library

IL-0213 Roblox · 2021-10-28database

Roblox was down for 73 hours because key-value writes on its single Consul cluster slowed from under 300 ms to 2 seconds

Roblox runs over 18,000 servers and 170,000 containers in its own data centers on the HashiStack: Nomad schedules containers, Vault holds secrets, and Consul provides service discovery, health checks, session locking and a KV store. One Consul cluster of 5 voters and 5 read-only non-voters, with a leader elected by Raft, served all of the backend. In the months before, Roblox had upgraded from Consul 1.9 to 1.10 for its streaming feature and had been enabling streaming service by service; on 27 October it was enabled for traffic routing, whose node count also grew by 50%. Under that combination of very high read and write load, streaming, which uses fewer Go channels than long polling, concentrated contention on a single Go channel and blocked writes. The dual-socket NUMA 128-core servers that responders moved to made this contention worse.

From 28 October the team, joined by HashiCorp engineers, tried four diagnoses: degraded hardware, too much traffic, thousands of containers reconnecting, and the move to 128-core machines. Replacing nodes, restoring the cluster from a snapshot behind iptables, disabling all non-essential Consul use, and returning to 64-core servers each failed to bring KV write latency down. On 30 October perf reports and flame graphs pointed to the streaming code paths; streaming was disabled for all Consul systems, and when the change finished propagating at 15:51 the 50th percentile KV write latency fell to 300ms. Some leaders were still slow, so the team kept those servers from staying elected. HashiCorp found the reason after the outage: Consul stores its Raft log in BoltDB, whose file never shrinks after old entries are deleted, and under Roblox's workload the freelist of free pages grew so large that every 16kB append also wrote a 7.8MB freelist to disk.

With Consul stable 54 hours into the outage, the team redeployed the caching system, which normally handles 1B requests per second. Its scheduling data in the Consul KV store was wrong, likely because of the snapshot reset, the job scheduler kept placing cache jobs on an unhealthy node it saw as empty, and the cache deployment tool was not built to bootstrap a large cluster from nothing. By 05:00 on 31 October, 61 hours in, Consul and the caches were healthy. Services were restarted at the right capacity, and from 10:00 players were let in with DNS steering in roughly 10% steps while database load, cache performance and stability were checked. At 16:45, 73 hours after the start, all players had access. No user data was lost.

Severity
–
Scope
Roblox fully down: online players fell to 50% of normal at 16:35 on 28 October and the system then went fully down, until all players were let back in at 16:45 on 31 October; no user data was lost
Users affected
Roblox players; fifty million players use Roblox every day
Region
Roblox data centers
Time to detect
–
Time to resolve
73h

Symptoms

On the afternoon of 28 October 2021 (13:37 PST), Vault performance at Roblox degraded and a single Consul server showed high CPU load. Consul cluster metrics showed that writes to its key-value (KV) store had a 50th percentile latency of 2 seconds instead of the usual under 300 ms. Players were not yet affected.

At 16:35 the number of online players fell to 50% of normal, and the platform then went fully down. Roblox services find each other through Consul, and Nomad and Vault depend on it as well, so with Consul unhealthy servers struggled to connect, new containers could not be scheduled and production secrets could not be fetched. The monitoring that would have shown the cause also depended on Consul, so responders lacked telemetry.

The first fixes treated the symptom as slow hardware and then as too much load. Replacing one Consul node, then moving the whole cluster to machines with 128 cores and faster NVMe SSDs, left the cluster reporting that most nodes could not keep up with writes, with KV write latency still around 2 seconds. A new symptom appeared: the Consul leader was regularly out of sync with the other voters. Resetting the cluster from a snapshot behind an iptables block looked healthy until the block was lifted and internal services' discovery and health check traffic returned; KV write latency climbed back to 2 seconds and services that depended on Consul began marking themselves unhealthy. Scaling services down to single-digit instances and cutting health checks from every 60 seconds to every 10 minutes also held only until 02:00 on 30 October, and going back to 64-core servers did not help.

Debug logs and operating system metrics then showed Consul KV writes blocked for long periods, and a perf report showed most time spent in kernel spin locks on the Streaming subscription code path. After streaming was disabled, some newly elected leaders still showed the old latency. BoltDB statistics later showed a 4.2GB Raft log store holding only 489MB of data, with 3.8GB of empty space and a 7.8MB freelist of nearly a million free page ids written to disk on every log append; full TCP buffers and TCP zero windows went with 2-3s write times on the unhealthy leaders.

Symptom High latencySustained high node CPUService outageMonitoring outage

Root cause

Root cause Software defect

Component Coordination and key-value storeService discoveryIn-memory cache
Contributing factor Single point of failureOverlapping failuresMonitoring blind spotHarmful manual actionChange with a delayed effect
Trigger
On 27 October at 14:00, one day before the outage, Roblox enabled the Consul streaming feature on the backend service responsible for traffic routing and, to prepare for end-of-year traffic, added 50% more nodes to traffic routing.

Detection

Roblox engineers began investigating when Vault performance degraded and a single Consul server had high CPU load, before players were affected. Consul cluster metrics then showed elevated KV write latency.

Remediation

Action items

Lessons

Two mostly unrelated problems deep inside Consul, contention in streaming under very high read and write load and BoltDB freelist growth, were hard to diagnose, and that difficulty was the main reason the outage lasted so long. A single Consul cluster serving every workload made the impact larger, and monitoring that depended on Consul hid the cause when it was needed most. Faster hardware did not help and may have hurt, because the contention was worse on NUMA machines. HashiCorp had benchmarked streaming at a similar scale, but the problem needed both a large number of streams and a high churn rate. Restarting a platform from a fully stopped state, including warming caches, took a long time and needed tools built for bootstrapping rather than incremental change.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Coordination and key-value store, Service discovery, In-memory cache among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.