Incident Library

IL-0243 Google Cloud · 2021-11-16gcpnetworking

Google's global External Proxy Load Balancing returned 404 errors for most HTTP/S endpoints for 33 minutes

A configuration pipeline carries customer configuration rules to Google External Proxy Load Balancing (GCLB). A bug added to it six months earlier allowed a race condition that, in very rare cases, could corrupt a configuration file near the end of the pipeline, past its validation checks. A Google engineer found the bug on 12 November, and Google declared an internal high-priority incident. Google froze part of the configuration system to make the race even less likely and, since the race had existed for months and Black Friday and Cyber Monday were close, chose a controlled rollout of fixes over a same-day emergency patch. Patch A closed the race; patch B added input validation to the binary that receives the configuration. Both were verified by 13 November and began rolling out gradually on 15 November, and patch B finished that evening.

On 16 November at 09:35 PT, with patch A within 30 minutes of finishing, the race occurred in an unpatched cluster. The corruption took a form different from the input errors seen in testing, so patch B did not reject it, and GCLB endpoints served 404 errors globally. Automated alerting notified engineers at 09:50 PT, together with customer support requests, and a rollback to the last known good configuration completed at 10:08 PT and ended the 404 errors. To avoid a repeat, Google suspended customer configuration changes from 10:04 PT, validated the fix, added proctoring and monitoring, and resumed configuration pushes at 11:28 PT, for a total impact of 1 hour and 53 minutes.

Severity
–
Scope
HTTP/S endpoints behind Google External Proxy Load Balancing returned 404 errors, including Cloud Run, App Engine, App Engine Flex, Cloud Functions, Apigee and Firebase behind GCLB; load balancer configuration changes did not take effect until 11:28 PT
Users affected
–
Region
Global; the preliminary report lists us-central, europe-west1 and global
Time to detect
15m
Time to resolve
1h 53m

Symptoms

From 09:35 to 10:08 PT on 16 November 2021, most endpoints served by Google's global External Proxy Load Balancing (GCLB) returned Google 404 errors to HTTP/S requests. For about another 1 hour and 20 minutes customers could not change their load balancing configuration.

Services behind GCLB lost traffic as well. Customers using serverless network endpoint groups on GCLB in front of Cloud Run, App Engine, App Engine Flex or Cloud Functions, and customers of Apigee and Firebase, got 404 errors. The preliminary report lists a 25% drop in Cloud Run traffic in us-central1, an 80% drop in App Engine traffic in us-central and europe-west1, failed App Engine Flex deployments and 404 errors for Apigee runtime requests.

During the partial recovery traffic was served again, but changes to External Proxy Load Balancers did not take effect, because Google had suspended customer-initiated configuration changes from 10:04 PT until configuration pushes resumed at 11:28 PT.

Symptom Service outageHigh error rate

Root cause

Root cause Software defect

Component Cloud load balancerServerless function platform
Contributing factor Unpatched known bugExcessive blast radius
Trigger
On 16 November 2021 a race condition in the configuration pipeline occurred in a cluster that patch A had not yet reached, and pushed a corrupted configuration file to GCLB.

Detection

Automated alerting notified Google engineers at 09:50 PT, at the same time as customer support requests came in.

Remediation

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Cloud load balancer, Serverless function platform among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.