Cases5 cases

1 to 5 of 5

1. IL-0214 PagerDuty's Kafka brokers ran out of JVM heap one after another, and the event APIs rejected events with 502 responses

   PagerDuty2025

   Symptoms

   When a Kafka cluster of PagerDuty's US Service Regions began failing at 03:53 UTC on 28 August 2025, the first alerts pointed to one failing broker. More brokers then ran out of memory, and PagerDuty's event APIs began rejecting incoming events with a response status code of 502 while notifications, webhooks, chat integrations and the REST API degraded.

   Memory pressureHigh error rateThroughput degradation

   Trigger

   A new feature that recorded API key usage in a Kafka topic was being rolled out by configuration: 1% on 21 August, 5% and then 25% on 27 August, and 75% on 28 August. A logical error in the feature created a new Kafka producer for every API request instead of reusing one producer.

   Root cause

   Faulty deployment

   Component

   Message broker

   Contributing factor

   Monitoring blind spotSignal buried in noiseChange with a delayed effectExcessive blast radius

   Detection

   Alerts pointed to one failing Kafka broker at the start of the first incident, and alerts fired again at 16:38 UTC when Kafka performance degraded for the second time. Critical alerts about core API errors were buried among webhook pages.

   Remediation

   • Expanded the Kafka cluster and removed the first failing broker
   • Doubled the Kafka broker heap size and applied it with a rolling restart across the cluster
   • Added status page updates by hand through backup procedures when automated publishing failed
   • Recovered Kafka-dependent services, some by manual intervention, and backfilled lost data with scripts
   • In the second incident, doubled the heap again with another rolling restart
   • Rolled back the API usage feature with a pull request that stopped it from publishing Kafka messages

   Action items

   • Remove the dependency that blocked status page automation and simplify and republish the manual runbook
   • Run live drills regularly so status updates go out even when parts of the platform are degraded
   • Expand JVM and Kafka monitoring of heap, garbage collection and producer and consumer health
   • Strengthen service dependency mapping so cascading failures are easier to trace
   • Collect customer impact metrics into incident workflows automatically
   • Add stricter change management guardrails, with readiness checklists and slower ramp-up windows for major rollouts
   • Improve recovery mechanisms in the streaming systems
   • Automate communication workflows such as status page updates, internal and external syncs and reminders
   • Test major incident workflows monthly with chaos engineering, such as status page failover
   • Define clearer incident severity levels and roles, including dedicated communications liaisons

   Lessons

   PagerDuty found its message processing more intricate than it had thought: some services had to be restarted to pick up changes, some API activity produced far more messages than expected, and it was hard to tell which parts of the system put heavy load on Kafka. Because producers and JVM heap were barely observed, responders could not see the first failure until several downstream systems were affected, and no one on the call held a model of the whole system. Enabling the feature through configuration rather than a separate deployment was meant as a safety step but made the outage harder to link to the change. Runbooks and backfill scripts were imperfect but gave responders a head start, and the second incident was mitigated in about 50 minutes by repeating the first incident's steps.

   View case
2. IL-0552 CircleCI's archival database ran out of storage, slowing workflow and job starts and the display of workflows in the UI

   CircleCI2025

   Symptoms

   At about 23:00 UTC on 21 January 2025, an automated alert reported that the CircleCI database instance holding archived data was almost out of free storage space. From 23:50 UTC customers saw slow starts and cancellations of workflows and jobs, and delays and difficulty viewing workflows in the CircleCI UI, although pipelines for a specific project could still be viewed.

   High latencyIntermittent timeoutsLog floodNode disk fullCloud API rate limit exhausted

   Trigger

   A code change to a function in the archival database on 16 January 2025 unintentionally made it write an excessive number of log messages, and the database filled its storage.

   Root cause

   Faulty deployment

   Component

   DatabaseCI system

   Contributing factor

   Change with a delayed effectExcessive blast radiusOverlapping failures

   Detection

   An automated alert at about 23:00 UTC warned that the archival database was almost out of free storage, before customers were affected. The growing workflows queue was not linked to it at first because a separate alert about request volume fired at about the same time.

   Remediation

   • Scaled up the event consumer that processes the workflows queue at 23:44 UTC
   • Promoted a read replica of the archival database to a standalone primary at 23:55 UTC

   Action items

   • Fix the database function so it no longer writes an excessive number of log messages
   • Add a max duration to the workflows retry policy for archiving so it fails earlier than the default timeout
   • Move workflow archival to an event-based model that decouples it from the workflows service
   • Add forecast and anomaly monitoring for databases that alerts on unusual activity before it reaches critical levels

   Lessons

   The storage alert on the archival database did fire, but the database filled much faster than forecast, so there was little time to act before customers were affected. Because the workflows service called the archival service synchronously, a problem in the archival database slowed the whole workflows service.

   View case
3. IL-0571 STORES reservation returned bursts of 504 errors when a smart-list query held a database process for over 20 minutes

   STORES2024

   Symptoms

   About 20 minutes into a batch that rewrote the preset smart-list data of STORES 予約 (STORES reservation), an alert fired for a sudden rise in 504 errors (Gateway Timeout). The work had started a little after 03:00 UTC, and the metrics watched during it, such as web server response latency and the load of the writer database, had shown no large change.

   High error rateDatabase overload

   Trigger

   The preset smart lists 初回予約の顧客 (first-reservation customers) and 要フォロー顧客 (customers needing follow-up), created more than a month earlier, were rewritten with new search parameters after the search specification changed, which switched them onto the new search query.

   Root cause

   Faulty deployment

   Component

   Database

   Contributing factor

   Inefficient queryChange with a delayed effectUnrepresentative stagingMonitoring blind spotUnverified assumption

   Detection

   An alert on a sudden increase in 504 errors (Gateway Timeout), about 20 minutes after the preset rewrite batch started.

   Action items

   • Rework the database design so the inline-view search logic is no longer used
   • Add a test that checks this API's EXPLAIN output for scans larger than expected
   • Provide an environment where the timing of this logic can be measured easily

   Lessons

   Rushing to meet the schedule led to both a failure that affected users and a later release. In a multi-tenant database an EXPLAIN that shows index use can still hide large scans, tests need the case where one query is fast and the next is slow, the metrics to watch during a data change are those of the instance that serves the affected queries, and metrics sent at the end of a request are lost for requests that never finish.

   View case
4. IL-0483 Replicate's website and API returned slow responses and HTTP 500s for two hours as PostgreSQL connections ran out

   Replicate2023

   Symptoms

   From 05:45 UTC on 11 May 2023, for about two hours, many Replicate customers got slow responses or HTTP 500 errors from the API and the replicate.com website and had trouble running predictions. Monitoring paged an engineer at 05:46 UTC, a minute after the database connection pool became fully saturated.

   High error rateHigh latencyLock contention or deadlock

   Trigger

   A little after 05:00 UTC on 11 May, normal variations in traffic pushed the stacking INSERT queries past their tipping point.

   Root cause

   Faulty deployment

   Component

   Database

   Contributing factor

   Change with a delayed effect

   Detection

   Monitoring detected the saturated database connection pool almost immediately and paged an engineer at 05:46 UTC.

   Remediation

   • Switched off all asynchronous prediction update features with a feature flag at 07:31 UTC, after which the database recovered

   Action items

   • Keep the asynchronous prediction update feature disabled and redesign it so it no longer sends stacked INSERTs
   • Review lock contention and other database hotspots that could cause similar problems
   • Publish detailed reports after major incidents

   Lessons

   The system ran in a degraded state that looked successful until it tipped into a full failure. The same asynchronous update work that triggered the outage also partly protected customers who used webhooks, and the stacked INSERTs came directly from a latency improvement: removing 100-150ms brought two updates of one prediction to the database almost together.

   View case
5. IL-0467 Boskos crash-looped after a node drain started it from a buggy StatefulSet image, failing Kubernetes PR jobs for 9 hours

   Kubernetes project2019

   Symptoms

   Overnight into 26 March 2019 (PDT), every Prow test job that needed a GCP project from Boskos, the Kubernetes test-infra service that hands out GCP projects to test jobs, failed. The required job pull-kubernetes-e2e-gce failed on every run, so no pull request could be merged in kubernetes/kubernetes for just over nine hours, and other projects saw similar failures.

   Container crash loopService outage

   Trigger

   At 23:04 PDT the node running the only Boskos pod was drained to be replaced with a 1.13 node during the upgrade of the Prow clusters from Kubernetes 1.11 to 1.13. The replacement pod started from the StatefulSet's current spec, which named a newer Boskos image containing the bug.

   Root cause

   Faulty deployment

   Component

   CI systemNode pool

   Contributing factor

   Change with a delayed effectUnverified assumptionSingle point of failureNo alert on limitChange started before off-hours

   Detection

   Users reported the failure on GitHub at 02:50 and on Slack at 05:16 and 06:27, and an engineer saw the Slack reports on waking at 07:16. The Boskos monitoring dashboard showed the failure, but no alert caught it.

   Remediation

   • Built and deployed a new Boskos image with the bug fixed
   • Cleaned up the GCP projects that had been left open

   Action items

   • Change the Boskos configuration so that deployments actually take effect (#11956)
   • Automate Boskos deploys (#11957)
   • Improve alerting on Boskos status (#11953)

   Lessons

   The write-up lists what went poorly: a ten-hour upgrade was started shortly after 17:00 when on-call coverage ended at 18:00; deployments were not checked to confirm that something had actually rolled out; the Boskos StatefulSet was not set up to perform updates, so changing its spec left the old pod running; the December change appears never to have been tested; and Boskos was deployed by hand, unlike most of Prow, which left more room for error. Once people were available, the problem was fixed quickly.

   View case

By root cause1