Incident Library
Applied filters FactorUnverified assumptionOrganisationParse.lyClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0276 Chef restarted two of three ZooKeeper nodes at once, and Parse.ly's real-time data stopped for about 3 hours
    Parse.ly2016
    Symptoms

    On Thursday 9 June 2016 at 11:30am ET, real-time data suddenly stopped arriving in Parse.ly's dashboard. Only the data delivery monitors reported a failure; no hardware or machine had failed and no other system reported anything, so the cause was at first unclear.

    Service outage
    Trigger

    A regular Chef run that applied a configuration change restarted the service on two of the three nodes of a ZooKeeper cluster at the same moment.

    Root cause
    Faulty deployment
    Component
    Coordination and key-value store
    Contributing factor
    Unverified assumptionNo staged rollout
    Detection

    Parse.ly's data delivery monitors reported that data had stopped flowing to the dashboard; no other system reported a failure.

    Remediation
    • Cleared a few hours of data from the live systems and cold-started the clusters, taking them offline and bringing them back from scratch
    • Delivered the data from the outage window late through the historical rebuild process
    Action items
    • Separate production configuration changes from production node restarts
    • Always keep a plan B for rapid recovery in systems and processes, instead of treating multi-node failures as impossible
    Lessons

    Automatic routine service restarts are dangerous for services that form a cluster across nodes, and jittering configuration runs only makes simultaneous restarts unlikely, not impossible, once a fleet has hundreds of nodes. A system designed to survive single-node failures still needs a recovery plan for multi-node failures.

    View case
Check your cluster for this

This case involves Coordination and key-value store. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment1 case

Component
Coordination and key-value store
Contributing factor
Unverified assumptionNo staged rollout
Organisation
Parse.ly
  1. IL-0276 Chef restarted two of three ZooKeeper nodes at once, and Parse.ly's real-time data stopped for about 3 hours
    Parse.ly2016