Incident Library

IL-0572 Mercari · 2026-06-16 (published)gcpdatabase

About 10 million Pub/Sub messages backed up and Merpay accounting data was not written to Cloud Spanner for over a day

Merpay's accounting system collects accounting events published by internal microservices and records them in Cloud Spanner. The pipeline is asynchronous: senders do not wait for a response, Pub/Sub pushes the events to Cloud Run, and the capacity of Cloud Run and Spanner sets the throughput of the whole pipeline. Load concentrates at month end and month start, with spikes of tens of times normal traffic. Accounting data is the basis for reconciliation with the sending services, merchant settlement and monthly closing, so delays hit the business directly. A separate pull subscription read by Dataflow also saves every message to Cloud Storage (GCS).

More than a month before the incident the team moved the Cloud Functions from 1st gen to 2nd gen without specifying --max-instances, so the scale limit fell from effectively unlimited to 100 with no change to code or deploy scripts. 100 instances were enough in normal times, so nothing showed for a while, and without monitoring nobody noticed that some months had already come close to the limit. Three factors combined: the limit of 100 started the backlog, the continuous traffic of the day kept it from clearing, and eventMaxAge turned it into dropped messages. The Spanner autoscaler, the open-source mercari/spanner-autoscaler, watched only High priority CPU and had no way to scale on Total CPU, so it did not add capacity while Total CPU sat at 100%.

Data loss was avoided by re-ingesting the messages saved in GCS, but data that had not reached Spanner stayed unreconciled, and because this fell on a closing day the business impact was wide. Afterwards the team set every scaling parameter of Cloud Run explicitly in the deploy definition, raised max-instances to 1000 in steps with Spanner load in mind, and raised concurrency after confirming the handler was safe for concurrent requests. The SRE team that maintains spanner-autoscaler added a dual CPU scaling mode that scales when either High priority or Total CPU crosses its threshold, released as v0.8.0, and new Datadog monitors watch the Cloud Run instance count and undelivered Pub/Sub messages.

Severity
–
Scope
Merpay accounting event pipeline: new accounting data was not registered in Cloud Spanner and stayed unreconciled, so it could not go on to merchant settlement and monthly closing; falling on a closing day, this caused wide business impact
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

From about 6:00 on the last business day of a month, Cloud Pub/Sub messages of the Merpay accounting event pipeline began to back up, new accounting data stopped being registered in Cloud Spanner, and the accounting system intermittently could not accept new data. The backlog grew to about 10 million messages.

Once the backlog started it fed itself. Cloud Run, which receives the messages through a push subscription, was capped at maxInstances=100 and could not keep up; the capped instances writing all at once pushed Spanner CPU up, each write took longer, throughput fell further and the backlog grew. Messages that waited longer than eventMaxAge, the maximum time a message is kept before it is discarded, were dropped without ever reaching Spanner. The system recovered and failed again repeatedly, and it took more than a full day to settle.

Spanner had an autoscaler, but during the incident its Processing Units (PU, Spanner's unit of compute capacity) did not scale up at all even though CPU was high. Total CPU was stuck at 100% while High priority CPU alone did not stay above the autoscaler's 60% threshold. No alert warned that Cloud Run had reached its scale limit.

The conditions had looked the same at the previous month-end peak after the migration, without a failure: maxInstances was 100, eventMaxAge was the same 21 minutes and total traffic was about 17 to 19 GB in 3 hours. The difference was continuity. At the earlier peak short troughs between bursts let Spanner CPU and the unacknowledged messages reset, while on the incident day the peak ran for about 40 minutes without a trough.

Symptom Throughput degradationDatabase overloadAutoscaler misbehaviour

Root cause

Root cause Breaking change in an upgrade

Component Serverless function platformMessage brokerDatabase
Contributing factor Unreviewed defaultsUnknown limitChange with a delayed effectMonitoring blind spot
Trigger
More than a month earlier the Cloud Functions of the pipeline had been moved from 1st gen to 2nd gen (based on Cloud Run) without setting --max-instances in the deploy command, which lowered the scale limit from effectively unlimited, the 1st gen default, to 100, the 2nd gen default. The month-end peak, which on that day continued for about 40 minutes, reached that limit.

Detection

–

Remediation

Action items

Lessons

When moving to a new generation of a managed service, defaults that the team never changed can surface as problems under load, so parameters that only matter at peak, such as scale limits, concurrency and timeouts, should be set explicitly at every migration. An outage rarely has one cause; the fact that the previous month passed under the same conditions pointed to traffic continuity, and prevention needs to look at the trigger, the amplification and what turned it into damage. Capacity settings and the monitoring that shows when they are approached have to be designed together, and teams that run systems they inherited should review these settings and assumptions regularly.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Serverless function platform, Message broker, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.