Cases4 cases

1 to 4 of 4

1. IL-0572 About 10 million Pub/Sub messages backed up and Merpay accounting data was not written to Cloud Spanner for over a day

   Mercari2026

   Symptoms

   From about 6:00 on the last business day of a month, Cloud Pub/Sub messages of the Merpay accounting event pipeline began to back up, new accounting data stopped being registered in Cloud Spanner, and the accounting system intermittently could not accept new data. The backlog grew to about 10 million messages.

   Throughput degradationDatabase overloadAutoscaler misbehaviour

   Trigger

   More than a month earlier the Cloud Functions of the pipeline had been moved from 1st gen to 2nd gen (based on Cloud Run) without setting --max-instances in the deploy command, which lowered the scale limit from effectively unlimited, the 1st gen default, to 100, the 2nd gen default. The month-end peak, which on that day continued for about 40 minutes, reached that limit.

   Root cause

   Breaking change in an upgrade

   Component

   Serverless function platformMessage brokerDatabase

   Contributing factor

   Unreviewed defaultsUnknown limitChange with a delayed effectMonitoring blind spot

   Remediation

   • Re-ingested the messages saved to GCS by the Dataflow subscription, avoiding data loss

   Action items

   • Set max-instances and the other scaling parameters of Cloud Run explicitly in the deploy definition, raising max-instances from 100 to 1000 in steps
   • Set min-instances to 1 to avoid cold starts, raise concurrency from 1 to 10, and set CPU to 1 and memory from 512MB to 1Gi
   • Use the dual CPU scaling mode of spanner-autoscaler v0.8.0 with a new Total CPU threshold of 70%
   • Lower the High priority CPU threshold from 60% to 55% so scale-up starts earlier
   • Forbid Spanner scale-down between 5:00 and 7:00
   • Reserve Spanner processing units on a schedule for periods with predictable load
   • Alert in Datadog when the Cloud Run instance count reaches 40% and 60% of its limit
   • Alert in Datadog when undelivered Pub/Sub messages exceed a threshold, lowering it step by step

   Lessons

   When moving to a new generation of a managed service, defaults that the team never changed can surface as problems under load, so parameters that only matter at peak, such as scale limits, concurrency and timeouts, should be set explicitly at every migration. An outage rarely has one cause; the fact that the previous month passed under the same conditions pointed to traffic continuity, and prevention needs to look at the trigger, the amplification and what turned it into damage. Capacity settings and the monitoring that shows when they are approached have to be designed together, and teams that run systems they inherited should review these settings and assumptions regularly.

   View case
2. IL-0316 Railway's control plane slowed and deployments stopped after a PgBouncer version jump used abnormal CPU and memory

   Railway2025

   Symptoms

   At 11:48 UTC on 22 September 2025, minutes after Railway had resolved a dashboard outage, its control plane showed abnormal resource consumption and elevated API latency. A large backlog of deployments queued during the first outage had started all at once, and the control plane began returning elevated database errors.

   High latencyIntermittent timeoutsDeploys failing

   Trigger

   After a dashboard outage caused by a schema change was fixed, a large backlog of queued deployments started at the same time and put the control plane and its database, behind a newly upgraded PgBouncer, under heavy load.

   Root cause

   Breaking change in an upgrade

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   Unpinned versionNo staged rolloutMonitoring blind spotUnreviewed defaults

   Detection

   Users reported at 11:05 UTC that the project canvas failed to load; the control plane's abnormal resource consumption and elevated API latency were seen at 11:48 UTC and opened as a new incident.

   Remediation

   • Deployed a fix for the API failure caused by the schema change within 15 minutes
   • Suspended deployments for non-Pro users so Pro users could deploy and the queue could drain
   • Suspended all deployments, Pro included, for up to 20 minutes during peak instability
   • Scaled up the control plane database's CPU, memory, connection limits and IOPS
   • Rolled PgBouncer back to the previous stable version, which brought all metrics back to nominal
   • Re-enabled deployments for all non-Pro users
   • Fixed the edge case in which private networks and their endpoints were missing from the dashboard

   Action items

   • Re-engineer parts of the control plane to be more resilient, moving large parts of its storage layer away from Postgres
   • Add CI validation that keeps disruptive schema changes out of production and automated schema compatibility checks before backend deployments
   • Monitor and alert on PgBouncer node CPU, memory and connection pool utilization
   • Upgrade PgBouncer in phases, instance by instance, instead of everywhere at once

   Lessons

   Railway believes that without the breaking schema change reaching production the cascade would not have happened, that monitoring PgBouncer's own CPU, memory and pool metrics would have shown the performance regression earlier and shortened recovery, and that upgrading PgBouncer instances one at a time would have exposed the version-specific problem before it reached every instance.

   View case
3. IL-0498 Clerk's Postgres database had load spikes every 15 minutes and requests failed and slowed intermittently for four days

   Clerk2025

   Symptoms

   At 13:08 UTC on 15 September 2025, Clerk's internal monitoring alerted on significant request failures and increased latency. Over the following days the request failures and latency spikes kept coming back while the Postgres database stayed unhealthy, with repeated spikes in database load.

   High error rateHigh latencyDatabase overloadConnection pool exhaustion

   Trigger

   At 05:30 UTC on 14 September 2025 the cloud provider automatically upgraded Clerk's Postgres database to a new minor version, which changed how Postgres handled connection locks.

   Root cause

   Breaking change in an upgrade

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   Unpinned versionRetry stormMonitoring blind spotOverlapping failuresHarmful manual action

   Detection

   Internal monitoring alerted on significant request failures and increased latency at 13:08 UTC on 15 September. Shortly after the upgrade on 14 September it had already shown an increased but still acceptable database load.

   Remediation

   • Optimised several expensive queries, refined critical indexes and re-indexed the database
   • Blocked abusive traffic more aggressively for a time
   • Performed a manual minor version upgrade and increased database capacity
   • Added jitter to the connection pooling configuration so connections no longer recycled in step

   Action items

   • Evaluate other database providers that give more control over upgrades and downgrades
   • Isolate more services from each other, for example Sign Up and Sign In infrastructure from the rest of the system
   • Keep optimising the most costly database queries
   • Evaluate sharding and multi-tenant infrastructure to isolate one application from others

   Lessons

   Clerk's review found that connection metrics sampled every 60 seconds could not show a recycling pattern that lasted seconds and repeated every 15 minutes, that an attack and background jobs with similar periods pulled attention away from the database layer, that the Backend API looked healthy only because it manages far fewer connections, and that the release freeze removed the staggering of connection creation that normal deploys provide.

   View case
4. IL-0308 Most RevenueCat backend systems failed after cutover to a new Aurora PostgreSQL 14 cluster as queries became very slow

   RevenueCat2022

   Symptoms

   At 9:34 UTC on November 23, 2022, RevenueCat started moving traffic from its Aurora PostgreSQL 10.x cluster to a new Aurora PostgreSQL 14.x cluster. By 9:39 UTC all traffic was on the new cluster, latency was bad, and the RevenueCat dashboard and the endpoints used by the RevenueCat SDK stopped working.

   Database overloadHigh latencyService outageConnection pool exhaustion

   Trigger

   At 9:34 UTC production traffic was switched from the Aurora PostgreSQL 10.x cluster to the new Aurora PostgreSQL 14.x cluster, before ANALYZE had run on its largest tables.

   Root cause

   Breaking change in an upgrade

   Component

   DatabasePostgreSQL vacuumDatabase replicationDatabase proxy and connection poolerQuery planner and table statistics

   Contributing factor

   Unrepresentative stagingUnverified assumptionAutomation amplificationMonitoring blind spot

   Detection

   At 9:39 UTC, with all traffic on the new cluster, latency looked bad and the infrastructure team began to investigate.

   Remediation

   • Added a statement timeout to the database user so queries could not hold a connection too long
   • Shed 90% of traffic by returning 503s from the load balancer
   • Dropped indexes that the planner was using wrongly and that were not needed
   • Stopped accepting Apple webhooks for a time
   • Rolled out a release without New Relic to stop its EXPLAIN queries
   • Ran ANALYZE by hand on the biggest tables
   • Raised the values of the affected primary key sequences with a prepared script
   • Stopped the integration event dispatcher by hand
   • Replayed the webhooks that were dispatched but not recorded
   • Restored integration event dispatching, ETLs and other non-web server jobs

   Action items

   • Consider change freezes during business-critical periods
   • Explain better how the SDK behaves during backend outages
   • Tell customers about maintenance and outages in the developer dashboard and by direct email to technical contacts
   • Replay a subset of write traffic in migration dry runs to mimic the cutover
   • Move costly processes to asynchronous work to keep the API fast and stable
   • Re-evaluate table indexes, including unused ones the database may choose for queries
   • Show the load of each endpoint and system in the database dashboards
   • Involve support and customer communications staff during outages

   Lessons

   The new cluster had not been analyzed, and the tests missed it because the test cluster had been up long enough to gather statistics. Dry runs replayed no write traffic, on the belief that writes are simpler than reads, and the team assumed sequence state was replicated with the data. Rolling back at the start would have been better than fixing forward. The migration, timed to finish before Thanksgiving and Black Friday, should have been delayed until the week after, because the US had entered a sensitive period and part of the US support team was already on holiday.

   View case

By root cause1