Incident Library

Organisation

Roblox

Roblox · roblox

1 case · 2021

Root cause Software defect
Component In-memory cacheCoordination and key-value storeService discovery
Contributing factor Change with a delayed effectHarmful manual actionMonitoring blind spotOverlapping failuresSingle point of failure
Symptom High latencyMonitoring outageSustained high node CPUService outage

Cases

  1. IL-0213 Roblox was down for 73 hours because key-value writes on its single Consul cluster slowed from under 300 ms to 2 secondsRoblox · 2021

    Root cause Software defect