Cases2 cases

1 to 2 of 2

1. IL-0214 PagerDuty's Kafka brokers ran out of JVM heap one after another, and the event APIs rejected events with 502 responses

   PagerDuty2025

   Symptoms

   When a Kafka cluster of PagerDuty's US Service Regions began failing at 03:53 UTC on 28 August 2025, the first alerts pointed to one failing broker. More brokers then ran out of memory, and PagerDuty's event APIs began rejecting incoming events with a response status code of 502 while notifications, webhooks, chat integrations and the REST API degraded.

   Memory pressureHigh error rateThroughput degradation

   Trigger

   A new feature that recorded API key usage in a Kafka topic was being rolled out by configuration: 1% on 21 August, 5% and then 25% on 27 August, and 75% on 28 August. A logical error in the feature created a new Kafka producer for every API request instead of reusing one producer.

   Root cause

   Faulty deployment

   Component

   Message broker

   Contributing factor

   Monitoring blind spotSignal buried in noiseChange with a delayed effectExcessive blast radius

   Detection

   Alerts pointed to one failing Kafka broker at the start of the first incident, and alerts fired again at 16:38 UTC when Kafka performance degraded for the second time. Critical alerts about core API errors were buried among webhook pages.

   Remediation

   • Expanded the Kafka cluster and removed the first failing broker
   • Doubled the Kafka broker heap size and applied it with a rolling restart across the cluster
   • Added status page updates by hand through backup procedures when automated publishing failed
   • Recovered Kafka-dependent services, some by manual intervention, and backfilled lost data with scripts
   • In the second incident, doubled the heap again with another rolling restart
   • Rolled back the API usage feature with a pull request that stopped it from publishing Kafka messages

   Action items

   • Remove the dependency that blocked status page automation and simplify and republish the manual runbook
   • Run live drills regularly so status updates go out even when parts of the platform are degraded
   • Expand JVM and Kafka monitoring of heap, garbage collection and producer and consumer health
   • Strengthen service dependency mapping so cascading failures are easier to trace
   • Collect customer impact metrics into incident workflows automatically
   • Add stricter change management guardrails, with readiness checklists and slower ramp-up windows for major rollouts
   • Improve recovery mechanisms in the streaming systems
   • Automate communication workflows such as status page updates, internal and external syncs and reminders
   • Test major incident workflows monthly with chaos engineering, such as status page failover
   • Define clearer incident severity levels and roles, including dedicated communications liaisons

   Lessons

   PagerDuty found its message processing more intricate than it had thought: some services had to be restarted to pick up changes, some API activity produced far more messages than expected, and it was hard to tell which parts of the system put heavy load on Kafka. Because producers and JVM heap were barely observed, responders could not see the first failure until several downstream systems were affected, and no one on the call held a model of the whole system. Enabling the feature through configuration rather than a separate deployment was meant as a safety step but made the outage harder to link to the change. Runbooks and backfill scripts were imperfect but gave responders a head start, and the second incident was mitigated in about 50 minutes by repeating the first incident's steps.

   View case
2. IL-0280 Kafka brokers in many New Relic cells ran out of disk, leaving US data collection and alerting unusable for hours

   New Relic2021

   Symptoms

   On 29 July 2021, data collection and alerting in New Relic's US region became unusable for many customers for several hours. It began with one unresponsive Kafka broker in a single cell of NRDB, New Relic's telemetry data platform, and over the following hours Kafka brokers in more and more cells failed as their disks filled up.

   Node disk fullThroughput degradationMonitoring outage

   Trigger

   Responders used an incident response tool to extend Kafka data retention for the degraded cell so that its backlog would not be lost. The retention increase was larger than usual, and the tool applied it to all cells of the same type, not only the affected one.

   Root cause

   Operator error

   Component

   Message broker

   Contributing factor

   Harmful manual actionNo staged rolloutSignal buried in noiseUnverified assumption

   Detection

   Engineers saw a Kafka broker in one cell become unresponsive on the morning of 29 July. Later a monitoring alert warned that disk space was running low, but it fired together with unrelated warnings from several other systems and was missed.

   Remediation

   • Restarted the unresponsive Kafka broker by hand, while the automated restart was also running
   • Resolved the disk space problem on the Kafka brokers, after which service recovered slowly

   Lessons

   Cell isolation is the strongest protection against a widespread disruption, and emergency tools have to respect it just as normal operations do. Emergency response tools must be safe to use in chaotic situations without losing their ability to work. Engineers' attention is limited, so incident response processes and alerts need regular review to direct it at the most important signal.

   View case

By root cause2