Incident Library

IL-0276 Parse.ly · 2016-06-09database

Chef restarted two of three ZooKeeper nodes at once, and Parse.ly's real-time data stopped for about 3 hours

Parse.ly manages its fleet of over 200 production servers with Chef, which can upgrade software, change configuration cluster-wide and restart services when their configuration changes. To keep restarts from happening everywhere at once, Chef runs are jittered across machines, each node running at a different minute of the hour. One of Parse.ly's three-node ZooKeeper clusters, spread across data centers, holds a small amount of configuration for real-time data delivery; it keeps serving with one node down but refuses requests when two are down. A regular Chef run restarted two of its nodes at the same time, the cluster lost quorum, and the services that relied on it failed in a way that could not recover on its own. With hundreds of nodes, two runs coinciding is a real probability even at random minutes. Around 1pm, without yet knowing the root cause, the team chose a cold start: clearing a few hours of data from the live systems and bringing the clusters down and up from scratch. The pipeline worked again by 2pm and real-time functionality was restored by 2:30pm. The root cause was confirmed in the following days from monitoring data and system logs. No data was lost; the data from the 3-hour window arrived late through the historical rebuild.

Severity
–
Scope
Parse.ly real-time data delivery to the dashboard stopped for about 3 hours; the data from that window was delivered late and none was lost
Users affected
Parse.ly customers
Region
–
Time to detect
–
Time to resolve
3h

Symptoms

On Thursday 9 June 2016 at 11:30am ET, real-time data suddenly stopped arriving in Parse.ly's dashboard. Only the data delivery monitors reported a failure; no hardware or machine had failed and no other system reported anything, so the cause was at first unclear.

In the war room some engineers restarted the cluster machines, which seemed to have no effect, while others read system logs and monitors. A monitoring graph later showed Chef run events lining up with the moment the real-time data lines dropped to zero.

Symptom Service outage

Root cause

Root cause Faulty deployment

Component Coordination and key-value store
Contributing factor Unverified assumptionNo staged rollout
Trigger
A regular Chef run that applied a configuration change restarted the service on two of the three nodes of a ZooKeeper cluster at the same moment.

Detection

Parse.ly's data delivery monitors reported that data had stopped flowing to the dashboard; no other system reported a failure.

Remediation

Action items

Lessons

Automatic routine service restarts are dangerous for services that form a cluster across nodes, and jittering configuration runs only makes simultaneous restarts unlikely, not impossible, once a fleet has hundreds of nodes. A system designed to survive single-node failures still needs a recovery plan for multi-node failures.

Sources

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Coordination and key-value store among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.