IL-0406 Google Cloud · 2026-02-27gcpai-infra

Vertex AI Gemini API returned 429 and 503 overload errors for all Gemini models on the global endpoint and in US regions

A configuration change to a safety filtering service that supports all Gemini models enabled code paths that, for some specific requests, ended in service disruption and capacity loss for the safety filtering service. Callers of the Vertex AI Gemini API then got overload errors for their queries, 429 for PayGo and 503 for Provisioned Throughput, on the global endpoint and in US regions. Dialogflow CX, Agent Assist, the Google Cloud Support AI agent and Customer Experience Agent Studio, which depend on the Gemini APIs, were affected as well.

Google's automated monitoring alerted engineers at 04:54 US/Pacific, 17 minutes after the start. Engineers found the faulty configuration change, rolled back to the previous stable configuration and added capacity to the service to stabilise it. Full restoration was confirmed at 06:35 as the rollback propagated and servers became healthy, 1 hour 58 minutes after the start.

Severity: –
Scope: Vertex AI Gemini API requests to all Gemini models, and downstream Google Cloud Support, Agent Assist, Dialogflow CX and Customer Experience Agent Studio
Users affected: PayGo customers (mostly 429 Resource Exhausted) and Provisioned Throughput customers (503 Service Unavailable)
Region: Global endpoint, us-central1, us-east4 and other US regions
Time to detect: 17m
Time to resolve: 1h 58m

Symptoms

From 04:37 US/Pacific on 27 February 2026, requests to Vertex AI Gemini API models through the global endpoint and in US regions failed at increased rates with overload errors. PayGo customers mostly got 429 Resource Exhausted, and Provisioned Throughput (PT) customers got 503 Service Unavailable.

Every Vertex AI Gemini API model was affected, including gemini-2.5-flash, gemini-2.5-flash-lite, gemini-2.5-pro, gemini-3.0-flash-preview, gemini-3.0-pro-preview, gemini-2.0-flash and gemini-2.0-flash-lite, and some users saw elevated error rates for particular models in US regions. Services that rely on the Gemini APIs were hit downstream: Google Cloud Support and its AI agent, Agent Assist, Dialogflow CX and Customer Experience Agent Studio. The global endpoint, us-central1, us-east4 and other US regions were affected.

Most errors stopped at 06:00 for PT customers and at 06:20 for PayGo customers, and full restoration was confirmed at 06:35. The first status update, posted the same morning, gave the window as 04:36 to 06:45 PST.

Symptom High error rate

Root cause

Root cause Configuration change

Component Content safety filter

Contributing factor –

Trigger: A configuration change to the safety filtering service that supports all Gemini models.

Detection

Google's automated monitoring system alerted engineers at 04:54 US/Pacific, and they started investigating at once.

Remediation

Rolled back the faulty configuration change to the previous stable configuration
Added more capacity to the safety filtering service to stabilise it

Action items

Reinforce rollout processes with mandatory validation checkpoints
Improve alerting to monitor critical dependencies more closely

Lessons

–

Sources

https://status.cloud.google.com/incidents/41E5S3mkTGDfkZuJZH5k · official-status · archived copy

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Content safety filter among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0406
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0406/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.