Incident Library

IL-0214 PagerDuty · 2025-08-28database

PagerDuty's Kafka brokers ran out of JVM heap one after another, and the event APIs rejected events with 502 responses

Kafka is the backbone of PagerDuty's asynchronous architecture. Kafka brokers keep metadata and state for every active producer, identified by a producer ID. A feature giving customers more auditability of API key usage published to a new Kafka topic at roughly the volume of API requests, hundreds of millions per day. It used the pekko-connectors-kafka Scala library, a framework rarely used at PagerDuty, and passing Kafka settings to it instead of a pre-created producer made it create a new producer on every send(). As the rollout grew, the brokers tracked millions of extra producers and their heap use rose slowly but steadily; given how fast producers were created, tuning transaction.max.timeout.ms or producer.id.expiration.ms would not have helped.

At 03:53 UTC on 28 August the brokers exhausted their JVM heap, starting with one broker and spreading over the cluster. Responders expanded the cluster and removed the first broker, then doubled the heap with a rolling restart, which stabilised Kafka. The rest of the first incident went into recovering the many services that depend on Kafka; several needed manual intervention, and the ksqlDB stream management runbooks lacked verification steps, which led to data loss that had to be backfilled with scripts that had not been tested recently. Status page updates were drafted but did not appear, so backup procedures were used to add them by hand. All services were back to normal by 10:10 UTC.

The team that owned the feature did not know it was linked to the outage and raised its rollout to 75%. At 16:38 UTC Kafka degraded again; on-call responders doubled the heap again with a rolling restart and limited customer impact to about 50 minutes. Having seen the pattern repeat, responders connected the heap growth to the gradual rollout and rolled the feature back with a pull request that stopped it from publishing to Kafka. All services were fully restored by 20:24 UTC, and later analysis of the code and logs found the producer-per-request error.

Severity
–
Scope
Event ingestion and asynchronous processing: up to about 95% of API events rejected with 502 at the peak for 38 minutes, 18.87% of create requests and 4.35% of update requests failed with 5xx, notifications, webhooks, integrations, chat messages and the REST API delayed, duplicated or failing; no accepted events lost
Users affected
some customers in the US Service Regions; fewer than 10% of customers saw errors in non-event incident creation and 6.06% of users could not acknowledge or resolve incidents through the mobile app
Region
US Service Regions
Time to detect
–
Time to resolve
6h 17m

Symptoms

When a Kafka cluster of PagerDuty's US Service Regions began failing at 03:53 UTC on 28 August 2025, the first alerts pointed to one failing broker. More brokers then ran out of memory, and PagerDuty's event APIs began rejecting incoming events with a response status code of 502 while notifications, webhooks, chat integrations and the REST API degraded.

Inside Kafka, the brokers came under heavy JVM memory pressure, went into GC thrashing and finally exhausted the JVM heap. The problem started on one broker and quickly spread across the whole cluster, until services could not talk to Kafka at all and errors chained through the downstream services. The heap was filling because the brokers were tracking nearly 4.2 million extra producers per hour at peak, 84 times the usual number of new producers, but there was little visibility into tracked producers and JVM heap usage. Of the 19 high-urgency pages during the incident, 18 were about webhooks, which hid the signals about core API errors, and the public status page was not updated for the first two hours because its automated publishing failed.

Customers saw up to about 95% of API events rejected with 502 at the peak, for 38 minutes, 18.87% of create requests and 4.35% of update requests answered with 5xx, about 23% of notifications delayed by 5 minutes or more, delayed or dropped integration events for Jira, ServiceNow, Salesforce and Zendesk, duplicated webhooks and chat messages, and 400 errors in the iOS and Android apps when viewing incidents. No accepted events were lost; messages held in a transactional outbox were published once Kafka was back, which caused the delays.

Responders first suspected a hardware problem on the single broker, so they expanded the cluster and removed that broker. Only when more brokers ran out of memory did they treat it as a systemic software problem. Doubling the heap stabilised Kafka, but the trigger was not evident, so nothing was rolled back. Later that day the feature behind it was raised from 25% to 75% of traffic, the enlarged heap filled within about two hours, and at 16:38 UTC alerts fired again as Kafka performance degraded.

Symptom Memory pressureHigh error rateThroughput degradation

Root cause

Root cause Faulty deployment

Component Message broker
Contributing factor Monitoring blind spotSignal buried in noiseChange with a delayed effectExcessive blast radius
Trigger
A new feature that recorded API key usage in a Kafka topic was being rolled out by configuration: 1% on 21 August, 5% and then 25% on 27 August, and 75% on 28 August. A logical error in the feature created a new Kafka producer for every API request instead of reusing one producer.

Detection

Alerts pointed to one failing Kafka broker at the start of the first incident, and alerts fired again at 16:38 UTC when Kafka performance degraded for the second time. Critical alerts about core API errors were buried among webhook pages.

Remediation

Action items

Lessons

PagerDuty found its message processing more intricate than it had thought: some services had to be restarted to pick up changes, some API activity produced far more messages than expected, and it was hard to tell which parts of the system put heavy load on Kafka. Because producers and JVM heap were barely observed, responders could not see the first failure until several downstream systems were affected, and no one on the call held a model of the whole system. Enabling the feature through configuration rather than a separate deployment was meant as a safety step but made the outage harder to link to the change. Runbooks and backfill scripts were imperfect but gave responders a head start, and the second incident was mitigated in about 50 minutes by repeating the first incident's steps.

Sources

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Message broker among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.