Graph
Cases4 cases
1 to 4 of 4
IL-0214PagerDuty's Kafka brokers ran out of JVM heap one after another, and the event APIs rejected events with 502 responsesPagerDuty2025- Symptoms
When a Kafka cluster of PagerDuty's US Service Regions began failing at 03:53 UTC on 28 August 2025, the first alerts pointed to one failing broker. More brokers then ran out of memory, and PagerDuty's event APIs began rejecting incoming events with a response status code of 502 while notifications, webhooks, chat integrations and the REST API degraded.
- Trigger
A new feature that recorded API key usage in a Kafka topic was being rolled out by configuration: 1% on 21 August, 5% and then 25% on 27 August, and 75% on 28 August. A logical error in the feature created a new Kafka producer for every API request instead of reusing one producer.
- Root cause
- Component
- Contributing factor
- Detection
Alerts pointed to one failing Kafka broker at the start of the first incident, and alerts fired again at 16:38 UTC when Kafka performance degraded for the second time. Critical alerts about core API errors were buried among webhook pages.
- Remediation
- Expanded the Kafka cluster and removed the first failing broker
- Doubled the Kafka broker heap size and applied it with a rolling restart across the cluster
- Added status page updates by hand through backup procedures when automated publishing failed
- Recovered Kafka-dependent services, some by manual intervention, and backfilled lost data with scripts
- In the second incident, doubled the heap again with another rolling restart
- Rolled back the API usage feature with a pull request that stopped it from publishing Kafka messages
- Action items
- Remove the dependency that blocked status page automation and simplify and republish the manual runbook
- Run live drills regularly so status updates go out even when parts of the platform are degraded
- Expand JVM and Kafka monitoring of heap, garbage collection and producer and consumer health
- Strengthen service dependency mapping so cascading failures are easier to trace
- Collect customer impact metrics into incident workflows automatically
- Add stricter change management guardrails, with readiness checklists and slower ramp-up windows for major rollouts
- Improve recovery mechanisms in the streaming systems
- Automate communication workflows such as status page updates, internal and external syncs and reminders
- Test major incident workflows monthly with chaos engineering, such as status page failover
- Define clearer incident severity levels and roles, including dedicated communications liaisons
- Lessons
PagerDuty found its message processing more intricate than it had thought: some services had to be restarted to pick up changes, some API activity produced far more messages than expected, and it was hard to tell which parts of the system put heavy load on Kafka. Because producers and JVM heap were barely observed, responders could not see the first failure until several downstream systems were affected, and no one on the call held a model of the whole system. Enabling the feature through configuration rather than a separate deployment was meant as a safety step but made the outage harder to link to the change. Runbooks and backfill scripts were imperfect but gave responders a head start, and the second incident was mitigated in about 50 minutes by repeating the first incident's steps.
IL-0294CircleCI Docker and machine jobs stalled for nearly 12 hours after a VM cleanup queue lost all its RabbitMQ consumersCircleCI2021- Symptoms
From 13:30 UTC on 21 May 2021, CircleCI jobs on Docker, the Machine executor (including Windows, Mac, Arm and GPU) and remote Docker were delayed or blocked. Engineers had been paged at 13:00 for excessive delays in starting jobs, and the alerts showed CircleCI could not provision some types and classes of VM because it was hitting its CPU quotas in GCP.
- Trigger
The upgrade of the RabbitMQ cluster from 3.8.9 to 3.8.16 on 20 May 2021, which brought in a 15 minute consumer acknowledgement timeout for all queue types.
- Root cause
- Component
- Contributing factor
- Detection
Engineers were paged at 13:00 UTC on 21 May for excessive delays in starting jobs. Earlier, a 02:31 notification that a VM destroyer queue had no consumers was handled by restarting pods, and a 10:11 alert that a VM destroy queue was backing up notified once at its threshold and did not page.
- Remediation
- Blocked Linux machine jobs from entering the work queue
- Had GCP raise the CPU quotas
- Deleted old VMs by hand
- Cancelled some workflows to free resources
- Reset the VM services deployment so the VM destroy queues had consumers again
- Scaled up the RDS database instance, at 18:20 and again at 19:15
- Turned down VM provisioning and blocked remote-docker jobs to shed load
- Scaled down all pods that created and destroyed VMs
- Deployed a revised distribution backend that kept remote-docker jobs out of the provisioning queues
- Brought provisioning and distribution back one component at a time and reversed the mitigation code changes
- Action items
- Make VM provisioning more robust to changes in load so it can recover from thundering herds
- Make VM provisioning independent so its failure does not affect Docker jobs
- Review and improve observability of RabbitMQ and the production queues
- Batch requests to avoid hitting GCP API rate limits
- Improve incident response for long-running incidents
- Symptoms
At 5:15 AM PST on 25 November 2020, alarms fired for errors on putting and getting Amazon Kinesis records in the Northern Virginia (US-EAST-1) Region. From the same time CloudWatch, which processes metric and log data through Kinesis Data Streams, and CloudWatch Events and EventBridge returned more errors.
- Trigger
A relatively small addition of capacity to the Kinesis front-end fleet, made between 2:44 AM and 3:47 AM PST.
- Root cause
- Component
- Contributing factor
- Detection
Alarms for errors on putting and getting Kinesis records fired at 5:15 AM PST, and the teams began reviewing logs.
- Remediation
- Removed the newly added front-end capacity
- Added a configuration so front-end servers load bootstrap data from the authoritative metadata store instead of neighbouring servers
- Restarted the front-end fleet gradually, a few hundred servers per hour from 10:07 AM PST
- Added Cognito capacity to buffer more calls to Kinesis
- Deployed a change that reduced Cognito's dependency on Kinesis
- Mitigated the memory contention on the Lambda invocation hosts
- Informed customers through the backup Service Health Dashboard tool, the Personal Health Dashboard and a global banner
- Action items
- Move the Kinesis front-end to larger CPU and memory servers so fewer servers, and fewer threads per server, are needed
- Add fine-grained alarming for thread consumption
- Finish testing a higher operating system thread count limit
- Greatly improve the cold-start time of the front-end fleet
- Move the front-end server cache to a dedicated fleet
- Move a few large AWS services such as CloudWatch to a separate, partitioned front-end fleet
- Accelerate the cellularization of the front-end fleet to match the back-end
- Change the Cognito webservers so they sustain Kinesis API errors without exhausting their buffers
- Keep 3 hours of metric data in the CloudWatch local metrics data store, first in US-EAST-1 and then globally
- Train support engineers regularly on the backup tool for posting to the Service Health Dashboard
- Lessons
In the Kinesis front-end, the number of threads each server keeps grows directly with the number of servers in the fleet, so adding servers can push every server past an operating system limit at the same time. AWS sees cellularization of the front-end, already done for the back-end, as the way to keep the shard-map cache within a tested and operated range and to guard against unknown scaling limits.
IL-0588AppOptics metrics APIs returned 5xx as Kafka partitions stayed leaderless after an AWS power event hit their ZooKeeperAppOptics2018- Symptoms
On May 31, 2018 from 21:53 UTC the metrics APIs of AppOptics and Librato returned a 5xx for a majority of incoming metric data. APM metrics and traces sent from AppOptics APM were affected, and many alert notifications were delayed.
- Trigger
A power event in one of AWS's availability zones in us-east-1 cut off nodes in several critical parts of the AppOptics and Librato infrastructure, including the ZooKeeper node that was then the leader of the ensemble coordinating the Kafka brokers.
- Root cause
- Component
- Contributing factor
- Remediation
- Restarted one ZooKeeper node by hand to force a leader election, after which the ensemble regained quorum
- Restarted every Kafka broker in a rolling restart until all partitions were back online
- Worked through the backlog, including the backlog of alerts, and resumed normal operations
- Action items
- Isolate the Kafka clusters better across availability zones and regions
- Stop sharing one ZooKeeper ensemble across Kafka clusters
- Run game days on this failure and check whether newer Kafka versions fix it
- Put in place more checks that notify the team earlier about subsystems that need extra help to recover
- Lessons
Running services across several availability zones and more than one Kafka cluster did not isolate the pipeline from the failure of one zone, because partitions of different clusters sat in the same zones and the clusters shared one ZooKeeper ensemble. More operational monitoring of the subsystems that needed help to recover would have led to a faster recovery.
These 4 cases involve Message broker and Coordination and key-value store. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.
By root cause4
Faulty deployment1 case
- Component
- Contributing factor
- Organisation
Breaking change in an upgrade1 case
- Component
- Contributing factor
- Organisation
Capacity exhaustion1 case
- Contributing factor
- Organisation
Cloud provider outage1 case
- Contributing factor
- Organisation