Incident Library

IL-0338 Google Cloud · 2018-07-17gcpnetworking

Google Cloud HTTP(S) and TCP/SSL Proxy load balancers returned 502s or reset connections on 33% to 87% of requests

Google's Global Load Balancers are built on two tiers of Google Front Ends (GFE). The first tier answers requests as close to the user as possible and forwards them over Google's network to a second tier of GFEs placed close to the backend service. The GFE team was adding security and performance features to the second-tier GFE code; the features were in the code base but not yet in service, and one of them contained a bug that made the GFE restart. Neither testing nor the initial rollout had caught the bug.

At 12:17 PDT on 17 July 2018 a configuration change in production began to set off the bug intermittently, and the affected second-tier GFEs restarted again and again. Restarts take time, so second-tier capacity shrank: some requests were answered, others were cut off, which showed as connection resets, or refused for lack of capacity while GFEs came back. Engineers were alerted within 3 minutes, found the root cause at 12:44 and reverted the configuration change, after which the GFEs stopped restarting and traffic returned to normal by 12:55.

Severity
–
Scope
Google Cloud HTTP(S) Load Balancer, TCP/SSL Proxy Load Balancer and the services behind them, including App Engine serving, Cloud Functions, the Stackdriver web UI, Dialogflow, the Cloud Support Portal and API, GKE Ingress and Cloud Storage traffic through Cloud Load Balancers; Cloud CDN cache hits dropped 70%
Users affected
customers using Google Cloud App Engine, HTTP(S) Load Balancer or TCP/SSL Proxy Load Balancers
Region
–
Time to detect
2m
Time to resolve
38m

Symptoms

From 12:17 to 12:49 PDT on 17 July 2018, Google Cloud HTTP(S) Load Balancers answered a share of requests with 502 return codes, and connections to TCP/SSL Proxy Load Balancers were reset. The share of failed requests moved between 33% and 87% during those 32 minutes, and automated monitoring alerted Google engineers at 12:19.

Google services that serve through Cloud HTTP Load Balancing were affected for the whole period, including App Engine application serving, Cloud Functions, the Stackdriver web UI, Dialogflow and the Cloud Support Portal and API. Services on Google Kubernetes Engine that use the Ingress resource returned 502s, and Cloud Storage traffic served through Cloud Load Balancers was hit as well. Cloud CDN cache hits fell by 70%, because fewer Cloud CDN URLs were referenced by the services behind the load balancers and Cloud CDN could not validate stale entries or add new content on cache misses. Applications that used direct VM access or Network Load Balancing were not affected.

After the fix took effect at 12:49 the 502 rate fell back to normal quickly, but latency stayed degraded for several more minutes while traffic returned and caches warmed. Serving was fully recovered at 12:55.

Symptom High error rateConnection resetsHigh latency

Root cause

Root cause Software defect

Component Cloud load balancerCDN and edge network
Contributing factor Gap in automated testsExcessive blast radius
Trigger
A configuration change in the production environment intermittently set off a bug in a new feature of the second-layer Google Front Ends.

Detection

Automated monitoring alerted Google's engineering team at 12:19 PDT, two minutes after the 502s began.

Remediation

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Cloud load balancer, CDN and edge network among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.