Incident Library

IL-0280 New Relic · 2021-07-29database

Kafka brokers in many New Relic cells ran out of disk, leaving US data collection and alerting unusable for hours

NRDB, New Relic's telemetry data platform, is built from isolated cells. Each cell runs a Kafka cluster, an internet-facing ingestion service, data pipeline services and an NRDB cluster, so that a failure in one cell reaches only a share of customers. Kafka carries all customer data through each stage of the pipeline and buffers it when a stage slows down. A broker that becomes unresponsive without leaving the cluster must be restarted, or rebuilt on a new instance attached to the existing storage; this normally happens automatically, and when the automation fails it is triggered by hand. Infrastructure is managed as code: normal changes roll out through CI/CD pipelines to one cell, then some, then all, and a separate set of tools built on the same infrastructure lets engineers act quickly during incidents.

On the morning of 29 July 2021 a Kafka broker in one cell became unresponsive. Engineers started a manual restart at the same time as the automated one, which kept the cluster degraded longer and blurred the cause. To keep the growing backlog from being lost, responders extended Kafka retention with an emergency tool. The increase was larger than usual, and because Kafka retention is normally kept the same across the environment, the tool applied it to every cell. In earlier incidents troubleshooting had finished and the emergency retention had been reverted long before it caused trouble, but this time the unclear root cause and the longer recovery led the team to keep it in place. Over the next hours broker disks filled, brokers failed in a few cells and then in many, and the low-disk alert was lost among unrelated warnings. Data collection and alerting were unusable for many customers in the US region for multiple hours, and recovery was slow even after the disk space problem was resolved.

New Relic listed five safety layers that failed. The emergency tool was deliberately simple and gave no warning that a large retention increase could be dangerous. An engineer used the wrong settings in a high-stress situation, with a tool that had been used dozens of times without a problem and therefore seemed safe. Fast troubleshooting had always kept extended retention from reaching a critical level, and this time the investigation took longer. The disk space alert was missed. Retention changes bypassed cell isolation because they applied automatically to all cells of the same type.

Severity
–
Scope
Data collection and alerting were unusable for many customers for multiple hours; Kafka brokers failed across many NRDB cells and data processing was widely disrupted
Users affected
many customers in the US region
Region
US region
Time to detect
–
Time to resolve
–

Symptoms

On 29 July 2021, data collection and alerting in New Relic's US region became unusable for many customers for several hours. It began with one unresponsive Kafka broker in a single cell of NRDB, New Relic's telemetry data platform, and over the following hours Kafka brokers in more and more cells failed as their disks filled up.

Every NRDB cell has its own Kafka cluster, which carries all customer data through the processing, evaluation and storage stages and holds data back when one stage slows down. A broker that stops responding but stays in the cluster stalls the pipeline data queued behind it. While the first cell was degraded, a backlog of unprocessed data built up in it.

After responders extended Kafka data retention to keep that backlog from being lost, the broker disks filled slowly over the next few hours. Brokers failed first in a few cells, then in many more, and data processing was disrupted across the platform. Once the disk space problem was fixed, service still came back slowly because the Kafka failures were so widespread.

The automatic broker remediation had recently failed to trigger in similar cases, so engineers restarted the unresponsive broker by hand. The automated restart had in fact started at the same moment. Both finished safely, but the two competing processes kept the cluster degraded for longer than expected and confused the team about the real cause. A monitoring alert for low disk space did fire, but it arrived together with unrelated warnings from several other systems and was missed.

Symptom Node disk fullThroughput degradationMonitoring outage

Root cause

Root cause Operator error

Component Message broker
Contributing factor Harmful manual actionNo staged rolloutSignal buried in noiseUnverified assumption
Trigger
Responders used an incident response tool to extend Kafka data retention for the degraded cell so that its backlog would not be lost. The retention increase was larger than usual, and the tool applied it to all cells of the same type, not only the affected one.

Detection

Engineers saw a Kafka broker in one cell become unresponsive on the morning of 29 July. Later a monitoring alert warned that disk space was running low, but it fired together with unrelated warnings from several other systems and was missed.

Remediation

Action items

–

Lessons

Cell isolation is the strongest protection against a widespread disruption, and emergency tools have to respect it just as normal operations do. Emergency response tools must be safe to use in chaotic situations without losing their ability to work. Engineers' attention is limited, so incident response processes and alerts need regular review to direct it at the most important signal.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Message broker among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.