Incident Library

IL-0395 Google Cloud · 2017-04-05gcpnetworking

Google Cloud HTTP(S) Load Balancer gave 502s on 25% of requests for 22 minutes after reverting to a stale configuration

A master server controls the process that pushes configuration to the Google Cloud HTTP(S) Load Balancers. When mastership changes, the new master starts its next push by testing the configuration on a subset of load balancers; if the test passes the configuration goes to all load balancers globally, and if it fails the master reverts all load balancers to the last "known good" configuration. On 5 April 2017 one master replica lost access to Google's distributed file system, could not read recent configuration files, and then took over mastership. Its test of the latest configuration failed, so it reverted every HTTP(S) Load Balancer to the newest configuration it could read, which was far out of date. Google describes this as a bug in the configuration update process.

The stale configuration deleted many configurations on the Google Frontend servers, which started garbage collection to free their memory. The garbage collection took so much CPU that the servers failed health checks and were restarted, and requests served by those servers got 502 errors. Engineers were paged at 01:22 PDT and at 01:34 switched the configuration update process to a different master server, which ended the errors for most services within one minute. Configuration updates to the HTTP(S) Load Balancer then stayed paused until 05:16 while the root cause was confirmed, which made App Engine Flexible deployments fail in that window.

Severity
–
Scope
Google Cloud HTTP(S) Load Balancer: 25% of requests failed with 502 for 22 minutes, 100% on some recently modified load balancers; App Engine Flexible deployments failed during the 3 hours 41 minutes configuration pause
Users affected
–
Region
–
Time to detect
9m
Time to resolve
22m

Symptoms

On 5 April 2017 from 01:13 to 01:35 PDT, 25% of requests to the Google Cloud HTTP(S) Load Balancer failed, and clients received 502 errors for the failed requests. Some HTTP(S) Load Balancers that had been modified recently failed 100% of their requests.

The errors came from Google Frontend servers that were restarting. After loading a configuration from which many configurations had been deleted, the Google Frontend servers spent a large share of their CPU on garbage collection to free the memory of the deleted configurations, failed their health checks and were restarted. Every client request served by a restarting server got a 502.

After the errors ended, Google paused all configuration changes to the HTTP(S) Load Balancer for three hours and 41 minutes while engineers confirmed the root cause. During that pause, deployments of App Engine Flexible apps failed.

Symptom High error rateProbe failures

Root cause

Root cause Software defect

Component Cloud load balancer
Contributing factor Automation amplification
Trigger
One replica of the master servers that control the HTTP(S) Load Balancer configuration update process lost access to Google's distributed file system and could not read recent configuration files, and mastership then passed to that replica.

Detection

Google engineers were paged at 01:22 PDT, nine minutes after the errors began.

Remediation

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Cloud load balancer among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.