IL-0275 Parse.ly · 2015-03-26databaseaws

Parse.ly's Kafka brokers hit network limits and failed one after another, stopping the analytics pipeline for hours

Parse.ly's JavaScript tracker sends thousands of requests per second to its data collection servers, which forward the data to Apache Kafka. Kafka feeds the backend of the production dashboard, the backend of the new preview dashboard and in-house monitoring, and Parse.ly had also started running parallel beta, staging and production plants that each read partial or full streams of the data. Customer growth and these parallel consumers drove outbound traffic on the brokers' network interfaces up, to over 250 Mbps of reads against 20 Mbps of writes. The brokers ran on relatively modest EC2 nodes, and at some point they hit network limits, which Parse.ly believes were enforced by Linux, Amazon's Xen hypervisor, the host's network hardware or a mix of them. Because every broker was close to the limit, the failure of one at peak load moved its consumers to the others and the whole cluster failed in a cascade.

The fix was to expand the Kafka cluster, but the production plant ran Kafka 0.7.2 and an earlier version of Parse.ly's own Python client library, pykafka, whose consumers discover brokers and manage consumer groups through ZooKeeper. Kafka 0.7.2 has no rebalancing, and the newer tools for checking topic replication and moving topics onto new brokers were not available. When the new brokers joined, a race condition in the Chef recipe that builds Kafka nodes had bootstrapped them with the wrong host name. The brokers registered those host names in ZooKeeper, the clients tried to reach hosts that did not exist and treated the Kafka cluster as unavailable, and the pixel publishers crashed with host resolution errors, stopping all data processing.

After the correct configuration was put in place and the new nodes were deployed, network utilization per node dropped and data processing resumed. The team turned the downstream consumers back on slowly through Apache Storm, using Kafka consumer offsets to estimate the remaining lag, and had to reprocess a large backlog after about 3 hours offline. No data was lost: raw data kept being collected and archived to Amazon S3 outside the Kafka path, and the data was being restored to customer accounts from it. Customers saw bad real-time data in the dashboard on Thursday night and Friday afternoon and historical reports sent on Monday that were missing a few hours.

Severity: –
Scope: Parse.ly data processing for dash.parsely.com and preview.parsely.com: no new data in the dashboards for several hours, bad real-time data and Monday reports missing a few hours of data; no data was lost
Users affected: existing customers of the production analytics product and select beta customers of the preview product
Region: –
Time to detect: –
Time to resolve: –

Symptoms

On Thursday night, 26 March 2015, and again on Friday, 27 March, Parse.ly's data processing backend stopped. Monitoring tools such as Munin, Sentry and Pingdom reported failures widely, PagerDuty woke the on-call engineer, and no new analytics data appeared in the dashboards at dash.parsely.com and preview.parsely.com for several hours.

The JavaScript tracker kept loading on customer sites and the data collection servers kept receiving requests, but the data no longer flowed from Apache Kafka into the real-time processing stream. Both backends, the production dashboard and the new preview dashboard built on the Mage time series engine, read from the same Kafka brokers, so both stopped. Graphite graphs confirmed the failures. The second failure came a few hours after the first.

The Kafka brokers themselves became unavailable. CPU and disk latency on the broker machines were well within limits, which was what the team mostly watched, but a Munin graph showed the network utilization of a typical Kafka node creeping up over time: about 20 Mbps was written to the brokers while more than 250 Mbps was read from them. When one broker failed, the others had to serve more consumers and more network traffic, and the whole cluster failed in turn.

When new brokers were added to expand the cluster, several of the Kafka writing services, called the "pixel publishers", crashed across the cluster with network host resolution errors, which stopped data processing everywhere. The Kafka clients reported the Kafka cluster as unavailable because they were trying to connect to hosts that did not exist.

Symptom Service outage DNS resolution failures

Root cause

Root cause Capacity exhaustion

Component Message broker Service discovery

Contributing factor Monitoring blind spot Unknown limit No headroom Harmful manual action

Trigger: Data volume reached a point where the Kafka brokers, running on modest Amazon EC2 nodes without high network capacity, hit operating system network limits under the outbound read traffic and became unavailable.

Detection

Munin, Sentry and Pingdom reported the failures widely, Graphite graphs showed them, and PagerDuty woke the on-call engineer when the first outages started in the night.

Remediation

Added new brokers to expand the Kafka cluster
Corrected the configuration of the new brokers and deployed them to the cluster, after which network utilization per node dropped and processing resumed
Turned the downstream consumer systems back on slowly through Apache Storm and reprocessed the backlog
Began restoring the data to customer accounts from the raw data archived in Amazon S3

Action items

–

Lessons

CPU and disk I/O are good signs of machine health, but creeping network utilization is harder to monitor, because 100% of network capacity is not well defined on cloud VMs that share host machines. When every node of a cluster runs close to the same limit, losing one node can take down the whole cluster, so failure planning should cover an entire subsystem going offline, not just one or two machines. Keeping raw data archival in simple files and S3 uploads, separate from the Kafka stream, meant there was no risk of permanent data loss.

Sources

https://www.parse.ly/kafkapocalypse/ · blog · archived copy

Related cases

IL-0353 Fly.io edge proxies in US west coast regions were overloaded and flapping, with lazy loader delays of secondsFly.io · 2026
same root cause · shared Service discovery
IL-0390 PostHog US Cloud event ingestion slowed to delays of up to 2 days as writes to the Persons Postgres table stalledPostHog · 2025
same root cause · shared Message broker
IL-0544 Wasabi's database cache stopped responding when its queueing service filled up, and S3 failed in all regionsWasabi · 2024
same root cause · shared Message broker
IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primaryGitLab · 2022
same root cause · shared Service discovery
IL-0212 Many Slack users could not connect because a Vitess keyspace for channel membership was overloaded by cache-miss queriesSlack · 2022
same root cause · shared Service discovery
IL-0384 Amazon Kinesis front-end servers in US-EAST-1 could not route requests, so Kinesis and services built on it failedAmazon Web Services · 2020
same root cause · shared Message broker

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Message broker, Service discovery among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0275
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0275/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.