Cases3 cases

1 to 3 of 3

1. IL-0572 About 10 million Pub/Sub messages backed up and Merpay accounting data was not written to Cloud Spanner for over a day

   Mercari2026

   Symptoms

   From about 6:00 on the last business day of a month, Cloud Pub/Sub messages of the Merpay accounting event pipeline began to back up, new accounting data stopped being registered in Cloud Spanner, and the accounting system intermittently could not accept new data. The backlog grew to about 10 million messages.

   Throughput degradationDatabase overloadAutoscaler misbehaviour

   Trigger

   More than a month earlier the Cloud Functions of the pipeline had been moved from 1st gen to 2nd gen (based on Cloud Run) without setting --max-instances in the deploy command, which lowered the scale limit from effectively unlimited, the 1st gen default, to 100, the 2nd gen default. The month-end peak, which on that day continued for about 40 minutes, reached that limit.

   Root cause

   Breaking change in an upgrade

   Component

   Serverless function platformMessage brokerDatabase

   Contributing factor

   Unreviewed defaultsUnknown limitChange with a delayed effectMonitoring blind spot

   Remediation

   • Re-ingested the messages saved to GCS by the Dataflow subscription, avoiding data loss

   Action items

   • Set max-instances and the other scaling parameters of Cloud Run explicitly in the deploy definition, raising max-instances from 100 to 1000 in steps
   • Set min-instances to 1 to avoid cold starts, raise concurrency from 1 to 10, and set CPU to 1 and memory from 512MB to 1Gi
   • Use the dual CPU scaling mode of spanner-autoscaler v0.8.0 with a new Total CPU threshold of 70%
   • Lower the High priority CPU threshold from 60% to 55% so scale-up starts earlier
   • Forbid Spanner scale-down between 5:00 and 7:00
   • Reserve Spanner processing units on a schedule for periods with predictable load
   • Alert in Datadog when the Cloud Run instance count reaches 40% and 60% of its limit
   • Alert in Datadog when undelivered Pub/Sub messages exceed a threshold, lowering it step by step

   Lessons

   When moving to a new generation of a managed service, defaults that the team never changed can surface as problems under load, so parameters that only matter at peak, such as scale limits, concurrency and timeouts, should be set explicitly at every migration. An outage rarely has one cause; the fact that the previous month passed under the same conditions pointed to traffic continuity, and prevention needs to look at the trigger, the amplification and what turned it into damage. Capacity settings and the monitoring that shows when they are approached have to be designed together, and teams that run systems they inherited should review these settings and assumptions regularly.

   View case
2. IL-0390 PostHog US Cloud event ingestion slowed to delays of up to 2 days as writes to the Persons Postgres table stalled

   PostHog2025

   Symptoms

   From 11 November 2025 at 16:02 UTC, PostHog's event ingestion pipeline for US Cloud became severely degraded, and events for all customers in the US region were processed with delays from 10 minutes up to 2 days. Writes to the Postgres database that stores Person information could take multiple seconds per update, and the pipeline slowed until no standard scaling option helped.

   Throughput degradationHigh latency

   Trigger

   The 32-bit OID space used to track TOASTed values of the Persons table's properties column, a JSONb field that often exceeds the 2KB threshold for out-of-line storage, was nearly exhausted.

   Root cause

   Capacity exhaustion

   Component

   DatabaseMessage broker

   Contributing factor

   Unknown limitMonitoring blind spotNo runbook

   Remediation

   • Created a new partitioned Persons table with a fresh TOAST table and OID space
   • Added database triggers to keep the old and new Persons tables in sync for live writes
   • Copied the existing Person data from the old table to the new one with a backfill script
   • Changed ingestion and application code to write to the new table and read from it first, with fallback to the old table
   • Deployed the web app changes to production under manual control rather than a fully automated rollout
   • Scaled ingestion workers to process the accumulated backlog
   • Paused ingestion when the MSK local disk reached about 85%
   • Reduced the local retention of the affected Kafka topic
   • Resumed ingestion after confirming there was enough disk headroom
   • Moved the backfill into Dagster to finish it over the weekend

   Action items

   • Add metrics and alerts for heavy disk read amplification per write
   • Add metrics and alerts for TOAST table statistics
   • Add metrics and alerts for other Postgres engine limits that matter at large scale
   • Write runbooks for engine-level Postgres limits, with decision trees for migrating versus repairing in place
   • Create customer communication runbooks with a clear escalation path and redundancies
   • Evaluate other data stores for the persons database
   • Complete the reconsolidation on the new Persons tables
   • Improve the backfill tooling and the safeguards for Kafka tiered storage
   • Design and operate Person-like tables so they avoid similar limits

   Lessons

   TOAST OID exhaustion degrades writes long before the hard limit of about 4 billion OIDs, it is rare even at large scale, and standard dashboards for CPU, memory, IOPS and lock contention did not immediately point at it, so diagnosis took a day and a half without monitoring of TOAST growth, OID usage or disk reads per write. Many core features depended on timely updates to the persons table, so its slowdown reached a wide part of the product. The first backfill approach lacked visibility for a long migration, the Kafka tiered storage settings and alerts were not tuned for a large backlog, and the people who normally handle customer communications were all unavailable.

   View case
3. IL-0591 Gitpod workspaces in all regions became IO bound and unresponsive for days after the ws39 workspace cluster release

   Gitpod2022

   Symptoms

   From April 11, 2022 at 13:17 UTC customers and Gitpod staff reported that Gitpod workspaces were intermittently becoming unresponsive. CPU usage of nodes and pods looked normal, with nothing maxed out, and at 13:58 UTC the team found that the workspaces were IO bound rather than CPU bound.

   Throughput degradationSustained high node CPUNode NotReadyData loss

   Trigger

   The ws39 generation of workspace clusters, released to all regional clusters on April 8th at 20:08 UTC and to the XL clusters on April 9th at 17:45 UTC, split the 1TB disk into smaller RAID0 disks, changed CPU limiting and workspace networking, and moved to faster machine types.

   Root cause

   Faulty deployment

   Component

   Node root diskContainer CPU limit

   Contributing factor

   No staged rolloutGap in automated testsUnrepresentative stagingUnknown limitOverlapping failuresIneffective rollbackHarmful manual actionLimited debug access

   Detection

   Customers and Gitpod staff reported on April 11th at 13:17 UTC that workspaces were intermittently unresponsive, and the team opened an incident in Slack through its incident.io integration. On April 17th the team noticed increased CPU load on the workspace nodes and opened an internal incident.

   Remediation

   • Shifted workspace starts back to the ws38 clusters on April 11th
   • Built ws39a clusters with c2d machines and local SSDs and moved traffic to them
   • Requested a limit increase from Let's Encrypt
   • Routed EU XL traffic to the regional EU cluster and built ws-eu39bxl to replace the crashed ws-eu39xl
   • Cleaned up workspaces stuck in a stopping state so users could start them again
   • Released ws40 with 1TB boot disks to get more IO bandwidth
   • Removed the workspaces that consumed excessive CPU time for crypto-mining
   • Blocked the fraudulent users behind the crypto-mining from using Gitpod
   • Rolled back the ws-daemon change that broke CPU limiting and shipped the fix in the ws34 clusters
   • Blocked ACME HTTP requests to workspace ports so Let's Encrypt rate limits would not be hit again
   • Rolled out IO limiting with the ws41xl and ws41 clusters and moved traffic to them step by step

   Action items

   • Invest in test automation so that test results stop releases like this one
   • Vary the interval between traffic shifts to new clusters, with more time for larger releases
   • Add a risk assessment stage, to be automated, that governs changeset size and rollout speed of workspace cluster rollouts
   • Reduce the impact teams have on each other
   • Add review criteria that find backwards incompatible changes before pull requests are merged, and split such pull requests
   • Recommend rollback as the primary means of mitigation in the incident procedures, and look at automating it
   • Make the status page show the workspace service as down when it is, and update the playbooks
   • Keep incident communication in a single Slack channel and Google Meet
   • Put 10% of the engineering staff on the on-call rotation from May 9th and use that time on observability, alerting and stability

   Lessons

   Gitpod concluded that it should have committed to rolling back on April 11th instead of building and shipping fixes, which kept customers on degraded workspaces longer than necessary. A large release so soon after the previous one needed a risk assessment and slower traffic shifts, and backwards incompatible changes that block a rollback need to be found before pull requests are merged. Failure-free operations require experience with failure.

   View case

By root cause3