Incident Library

IL-0366 Google Cloud · 2026-10-08gcpdatabase

Cloud Storage uploads and writes in us-central1 slowed and failed with HTTP 503 as traffic piled onto a few backends

Google Cloud Storage spreads incoming data over a large, distributed set of storage backend nodes. During the incident window, some backend nodes in us-central1 were short of capacity because traffic had grown. A configuration update to the Cloud Storage load balancing layer added a traffic sorting optimization meant to address local capacity constraints, but it unintentionally bypassed the normal capacity-aware routing logic. A small fraction of backend nodes with little free serving capacity then received a very large share of incoming write traffic and ran out of resources, and the backend services that ingest object data began to time out. Uploads and writes routed to those nodes became slow and failed. At the same time, a separate problem in the underlying metadata database service caused additional metadata processing errors for part of the incident.

Engineers detected the problem at 04:50 from system telemetry. Adding emergency capacity was blocked by regional capacity limits, so engineers drained traffic from the most overloaded backend nodes, deployed a configuration change that disabled the traffic sorting optimization, and rolled back a recent update to the metadata database service. The first occurrence ended at 16:30 on 8 October, after 12 hours, 30 minutes. On 9 October an incomplete rollback of an upload planner feature broke the load balancing again and overloaded a few smaller backend nodes in us-central1-b from 04:05 to 09:46. That recurrence needed more traffic redirection and draining, and it ended when engineers fully reverted the load balancing configuration change, which brought back capacity-aware routing and an even spread of traffic across the region. The drained nodes were returned to service once traffic and resource use were back to their normal baseline. The total duration was 18 hours, 11 minutes.

Severity
–
Scope
Cloud Storage upload and write operations (WriteObject) in us-central1, with a minor impact on reads during the first occurrence
Users affected
–
Region
us-central1 (recurrence in us-central1-b)
Time to detect
50m
Time to resolve
29h 46m

Symptoms

From 04:00 US/Pacific on 8 October 2026, Google Cloud Storage upload and write operations in the us-central1 region became slow and failed. Clients mostly saw HTTP 503 UNAVAILABLE errors on WriteObject operations and client-side timeouts.

Write operations that did succeed took much longer than normal. During the first occurrence a smaller share of read operations also saw higher latency and errors. Cloud Storage in every other region and all other Google Cloud services were not affected.

The status page updates posted on 8 October gave 05:19 PDT as the start of the problem, and the final report gives 04:00. Error rates in Google's internal monitoring began to improve at 10:57 PDT after the first mitigations, and the first occurrence ended at 16:30. The problem came back on 9 October from 04:05 to 09:46, this time in the us-central1-b zone.

Google engineers first tried to add emergency capacity to the region, but systemic regional capacity limits blocked it. Draining the most overloaded backend nodes and disabling a traffic sorting optimization gave relief, but only for a while: the load imbalance returned the next day.

Symptom High latencyHigh error rateLoad imbalance

Root cause

Root cause Configuration change

Component Object storageDatabase
Contributing factor No headroomOverlapping failures
Trigger
A configuration update to the Cloud Storage load balancing layer turned on a traffic sorting optimization that was meant to deal with local capacity constraints. The recurrence on 9 October came from an incomplete rollback of an upload planner feature.

Detection

Google engineering teams monitoring system telemetry detected the problem at 04:50 US/Pacific on 8 October, 50 minutes after it began.

Remediation

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Object storage, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.