Cases3 cases

1 to 3 of 3

1. IL-0572 About 10 million Pub/Sub messages backed up and Merpay accounting data was not written to Cloud Spanner for over a day

   Mercari2026

   Symptoms

   From about 6:00 on the last business day of a month, Cloud Pub/Sub messages of the Merpay accounting event pipeline began to back up, new accounting data stopped being registered in Cloud Spanner, and the accounting system intermittently could not accept new data. The backlog grew to about 10 million messages.

   Throughput degradationDatabase overloadAutoscaler misbehaviour

   Trigger

   More than a month earlier the Cloud Functions of the pipeline had been moved from 1st gen to 2nd gen (based on Cloud Run) without setting --max-instances in the deploy command, which lowered the scale limit from effectively unlimited, the 1st gen default, to 100, the 2nd gen default. The month-end peak, which on that day continued for about 40 minutes, reached that limit.

   Root cause

   Breaking change in an upgrade

   Component

   Serverless function platformMessage brokerDatabase

   Contributing factor

   Unreviewed defaultsUnknown limitChange with a delayed effectMonitoring blind spot

   Remediation

   • Re-ingested the messages saved to GCS by the Dataflow subscription, avoiding data loss

   Action items

   • Set max-instances and the other scaling parameters of Cloud Run explicitly in the deploy definition, raising max-instances from 100 to 1000 in steps
   • Set min-instances to 1 to avoid cold starts, raise concurrency from 1 to 10, and set CPU to 1 and memory from 512MB to 1Gi
   • Use the dual CPU scaling mode of spanner-autoscaler v0.8.0 with a new Total CPU threshold of 70%
   • Lower the High priority CPU threshold from 60% to 55% so scale-up starts earlier
   • Forbid Spanner scale-down between 5:00 and 7:00
   • Reserve Spanner processing units on a schedule for periods with predictable load
   • Alert in Datadog when the Cloud Run instance count reaches 40% and 60% of its limit
   • Alert in Datadog when undelivered Pub/Sub messages exceed a threshold, lowering it step by step

   Lessons

   When moving to a new generation of a managed service, defaults that the team never changed can surface as problems under load, so parameters that only matter at peak, such as scale limits, concurrency and timeouts, should be set explicitly at every migration. An outage rarely has one cause; the fact that the previous month passed under the same conditions pointed to traffic continuity, and prevention needs to look at the trigger, the amplification and what turned it into damage. Capacity settings and the monitoring that shows when they are approached have to be designed together, and teams that run systems they inherited should review these settings and assumptions regularly.

   View case
2. IL-0390 PostHog US Cloud event ingestion slowed to delays of up to 2 days as writes to the Persons Postgres table stalled

   PostHog2025

   Symptoms

   From 11 November 2025 at 16:02 UTC, PostHog's event ingestion pipeline for US Cloud became severely degraded, and events for all customers in the US region were processed with delays from 10 minutes up to 2 days. Writes to the Postgres database that stores Person information could take multiple seconds per update, and the pipeline slowed until no standard scaling option helped.

   Throughput degradationHigh latency

   Trigger

   The 32-bit OID space used to track TOASTed values of the Persons table's properties column, a JSONb field that often exceeds the 2KB threshold for out-of-line storage, was nearly exhausted.

   Root cause

   Capacity exhaustion

   Component

   DatabaseMessage broker

   Contributing factor

   Unknown limitMonitoring blind spotNo runbook

   Remediation

   • Created a new partitioned Persons table with a fresh TOAST table and OID space
   • Added database triggers to keep the old and new Persons tables in sync for live writes
   • Copied the existing Person data from the old table to the new one with a backfill script
   • Changed ingestion and application code to write to the new table and read from it first, with fallback to the old table
   • Deployed the web app changes to production under manual control rather than a fully automated rollout
   • Scaled ingestion workers to process the accumulated backlog
   • Paused ingestion when the MSK local disk reached about 85%
   • Reduced the local retention of the affected Kafka topic
   • Resumed ingestion after confirming there was enough disk headroom
   • Moved the backfill into Dagster to finish it over the weekend

   Action items

   • Add metrics and alerts for heavy disk read amplification per write
   • Add metrics and alerts for TOAST table statistics
   • Add metrics and alerts for other Postgres engine limits that matter at large scale
   • Write runbooks for engine-level Postgres limits, with decision trees for migrating versus repairing in place
   • Create customer communication runbooks with a clear escalation path and redundancies
   • Evaluate other data stores for the persons database
   • Complete the reconsolidation on the new Persons tables
   • Improve the backfill tooling and the safeguards for Kafka tiered storage
   • Design and operate Person-like tables so they avoid similar limits

   Lessons

   TOAST OID exhaustion degrades writes long before the hard limit of about 4 billion OIDs, it is rare even at large scale, and standard dashboards for CPU, memory, IOPS and lock contention did not immediately point at it, so diagnosis took a day and a half without monitoring of TOAST growth, OID usage or disk reads per write. Many core features depended on timely updates to the persons table, so its slowdown reached a wide part of the product. The first backfill approach lacked visibility for a long migration, the Kafka tiered storage settings and alerts were not tuned for a large backlog, and the people who normally handle customer communications were all unavailable.

   View case
3. IL-0214 PagerDuty's Kafka brokers ran out of JVM heap one after another, and the event APIs rejected events with 502 responses

   PagerDuty2025

   Symptoms

   When a Kafka cluster of PagerDuty's US Service Regions began failing at 03:53 UTC on 28 August 2025, the first alerts pointed to one failing broker. More brokers then ran out of memory, and PagerDuty's event APIs began rejecting incoming events with a response status code of 502 while notifications, webhooks, chat integrations and the REST API degraded.

   Memory pressureHigh error rateThroughput degradation

   Trigger

   A new feature that recorded API key usage in a Kafka topic was being rolled out by configuration: 1% on 21 August, 5% and then 25% on 27 August, and 75% on 28 August. A logical error in the feature created a new Kafka producer for every API request instead of reusing one producer.

   Root cause

   Faulty deployment

   Component

   Message broker

   Contributing factor

   Monitoring blind spotSignal buried in noiseChange with a delayed effectExcessive blast radius

   Detection

   Alerts pointed to one failing Kafka broker at the start of the first incident, and alerts fired again at 16:38 UTC when Kafka performance degraded for the second time. Critical alerts about core API errors were buried among webhook pages.

   Remediation

   • Expanded the Kafka cluster and removed the first failing broker
   • Doubled the Kafka broker heap size and applied it with a rolling restart across the cluster
   • Added status page updates by hand through backup procedures when automated publishing failed
   • Recovered Kafka-dependent services, some by manual intervention, and backfilled lost data with scripts
   • In the second incident, doubled the heap again with another rolling restart
   • Rolled back the API usage feature with a pull request that stopped it from publishing Kafka messages

   Action items

   • Remove the dependency that blocked status page automation and simplify and republish the manual runbook
   • Run live drills regularly so status updates go out even when parts of the platform are degraded
   • Expand JVM and Kafka monitoring of heap, garbage collection and producer and consumer health
   • Strengthen service dependency mapping so cascading failures are easier to trace
   • Collect customer impact metrics into incident workflows automatically
   • Add stricter change management guardrails, with readiness checklists and slower ramp-up windows for major rollouts
   • Improve recovery mechanisms in the streaming systems
   • Automate communication workflows such as status page updates, internal and external syncs and reminders
   • Test major incident workflows monthly with chaos engineering, such as status page failover
   • Define clearer incident severity levels and roles, including dedicated communications liaisons

   Lessons

   PagerDuty found its message processing more intricate than it had thought: some services had to be restarted to pick up changes, some API activity produced far more messages than expected, and it was hard to tell which parts of the system put heavy load on Kafka. Because producers and JVM heap were barely observed, responders could not see the first failure until several downstream systems were affected, and no one on the call held a model of the whole system. Enabling the feature through configuration rather than a separate deployment was meant as a safety step but made the outage harder to link to the change. Runbooks and backfill scripts were imperfect but gave responders a head start, and the second incident was mitigated in about 50 minutes by repeating the first incident's steps.

   View case

By root cause3