Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0275 Parse.ly's Kafka brokers hit network limits and failed one after another, stopping the analytics pipeline for hours
    Parse.ly2015
    Symptoms

    On Thursday night, 26 March 2015, and again on Friday, 27 March, Parse.ly's data processing backend stopped. Monitoring tools such as Munin, Sentry and Pingdom reported failures widely, PagerDuty woke the on-call engineer, and no new analytics data appeared in the dashboards at dash.parsely.com and preview.parsely.com for several hours.

    Service outageDNS resolution failures
    Trigger

    Data volume reached a point where the Kafka brokers, running on modest Amazon EC2 nodes without high network capacity, hit operating system network limits under the outbound read traffic and became unavailable.

    Root cause
    Capacity exhaustion
    Component
    Message brokerService discovery
    Contributing factor
    Monitoring blind spotUnknown limitNo headroomHarmful manual action
    Detection

    Munin, Sentry and Pingdom reported the failures widely, Graphite graphs showed them, and PagerDuty woke the on-call engineer when the first outages started in the night.

    Remediation
    • Added new brokers to expand the Kafka cluster
    • Corrected the configuration of the new brokers and deployed them to the cluster, after which network utilization per node dropped and processing resumed
    • Turned the downstream consumer systems back on slowly through Apache Storm and reprocessed the backlog
    • Began restoring the data to customer accounts from the raw data archived in Amazon S3
    Lessons

    CPU and disk I/O are good signs of machine health, but creeping network utilization is harder to monitor, because 100% of network capacity is not well defined on cloud VMs that share host machines. When every node of a cluster runs close to the same limit, losing one node can take down the whole cluster, so failure planning should cover an entire subsystem going offline, not just one or two machines. Keeping raw data archival in simple files and S3 uploads, separate from the Kafka stream, meant there was no risk of permanent data loss.

    View case
Check your cluster for this

This case involves Message broker and Service discovery. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
Message brokerService discovery
Contributing factor
Unknown limitHarmful manual actionMonitoring blind spotNo headroom
Organisation
Parse.ly
  1. IL-0275 Parse.ly's Kafka brokers hit network limits and failed one after another, stopping the analytics pipeline for hours
    Parse.ly2015