Incident Library

IL-0222 Google Cloud · 2019-06-02gcpnetworking

Google Cloud US regions had heavy packet loss and some went unreachable after the network control plane was descheduled

In each Google datacenter location, the network control plane runs on several instances of Google's cluster management software, so the failure of one instance does not affect network capacity. The cluster management software also automates maintenance events, and every job is labelled with what to do during one: move to another machine, or stop and be rescheduled afterwards. Three things combined. Network control plane jobs and their supporting infrastructure in the affected regions were configured to stop during a maintenance event. The cluster management instances running the network control plane were eligible for a rare maintenance event type. And the software that starts maintenance events had a bug that let it deschedule several independent software clusters at once, even in different physical locations.

When the maintenance event started at 11:45 in one location, the automation descheduled the network control jobs and their supporting infrastructure in multiple locations. The network is designed to fail static, that is, to keep running for a while without its control plane, and it ran normally for several minutes. Then BGP routing between specific affected locations was withdrawn, which sharply cut network capacity and made some Google Cloud regions inaccessible; user impact began between 11:47 and 11:49. Packet loss lasted between 3 hours 19 minutes and 4 hours 25 minutes depending on the region, and Compute Engine, App Engine, Cloud Endpoints, Cloud Interconnect, Cloud VPN, Cloud Console, Stackdriver Metrics, Cloud Pub/Sub, BigQuery, regional Cloud Spanner instances, regional Cloud Storage buckets and G Suite in those regions were affected.

By 13:01 the root cause was known and engineers halted the maintenance automation. Bringing the network control plane back took longer because, in the locations where every instance had been descheduled, its configuration data was lost and had to be rebuilt and redistributed during a large network event. The new configuration began rolling out at 14:03, while other Google teams moved their services' traffic away from the affected regions. Network capacity started to recover at 15:19 and full service resumed at 16:10 US/Pacific. Cloud VPN in us-west2, northamerica-northeast1 and southamerica-east1 needed extra work to bring its control plane out of a fail-safe state and recovered at 16:30. Cloud Endpoints configuration pushes failed for 50% of workflows until 4 June 11:30, and after that still took up to an hour to take effect.

Severity
–
Scope
Google Cloud services on Google's US network, including Compute Engine, App Engine, Cloud Endpoints, Cloud Interconnect, Cloud VPN, Cloud Console, Stackdriver, Cloud Pub/Sub, BigQuery, Cloud Spanner and Cloud Storage, plus G Suite and other Google services
Users affected
–
Region
us-central1, us-east1, us-east4, us-west2, northamerica-northeast1, southamerica-east1
Time to detect
2m
Time to resolve
4h 25m

Symptoms

On Sunday 2 June 2019, with user impact beginning between 11:47 and 11:49 US/Pacific, Google Cloud projects in several US regions saw elevated packet loss from network congestion, and some regions became unreachable. Customers saw higher latency, intermittent errors and lost connectivity to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1 and southamerica-east1, while us-west1, Europe and Asia had no regional congestion.

Compute Engine instances in us-east4, us-west2, northamerica-northeast1 and southamerica-east1 were inaccessible for the whole incident. Between instances, us-east1 had up to 33% packet loss from 11:38 to 12:17 and up to 8% until 14:50, and us-central1 had a 9% spike that subsided by 12:05. Compute Engine admin operations returned an average of 1.2% errors. Cloud Interconnect attachments in the four inaccessible regions reported 50% to 100% packet loss and some BGP sessions went down, attachments connecting through east coast Interconnects saw 10-50% loss from backbone congestion, and Cloud VPN gateways in those regions were unreachable, while us-central1 VPN endpoints reported 25% packet loss and us-east1 endpoints 10%.

Services that depend on Google's US network failed with it. App Engine apps in the four regions were unavailable, us-central saw a 23.2% drop in requests per second, and requests that did not reach App Engine returned client timeout errors; BigQuery jobs.insert calls had a 27% error rate between 11:47 and 12:02; Cloud Pub/Sub publish and subscribe requests failed in the affected regions; Cloud Spanner in us-east4, us-west2 and northamerica-northeast1 was unavailable from 11:48 to 15:44; Cloud Console pages loaded slowly, partially or not at all; and Cloud Storage average error rates were 96.2% for us-west2, 79.3% for southamerica-east1, 62.4% for us-east4 and 43.4% for northamerica-northeast1 buckets. Stackdriver Monitoring had a 5-10% drop in requests per second, login failures to its frontend averaged 8.4%, UI components hit a 3.5% error rate when loading data, and monitoring statistics were inconsistent during the disruption. Cloud Endpoints configuration pushes failed for 50% of workflows until 4 June 11:30, and new Endpoints services returned 500 errors unless the ESP flag service_control_network_fail_open was enabled.

Google engineers were alerted two minutes after the failure began, but debugging was slowed because their tools competed for the congested network. The backup plans for that case, engineers travelling to secure facilities and lowering the priority of less critical traffic classes, added time, and the damage to tooling made it hard at first to measure impact and tell customers accurately.

On the status page, the first notice at 12:53 said Google was investigating an issue with Google Cloud Networking. The 13:45 notice reported high levels of network congestion in the eastern USA affecting multiple services in Google Cloud, G Suite and YouTube, and the 15:04 notice said the congestion was continuing.

Symptom Packet lossNetwork unreachableHigh latencyHigh error rateMonitoring outage

Root cause

Root cause Software defect

Component Network control plane (SDN)BGP routingBackbone network (WAN)
Contributing factor Excessive blast radiusCircular dependencyLimited debug accessIneffective rollback
Trigger
At 11:45 US/Pacific a relatively rare type of datacenter maintenance event started in one physical location, and the automation that runs it descheduled logical clusters that also held network control plane jobs in other physical locations.

Detection

Google engineers were alerted to the failure two minutes after it began and started the incident process used for the most serious production incidents. The first customer notice appeared on the status page at 12:53 US/Pacific.

Remediation

Action items

Lessons

–

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Network control plane (SDN), BGP routing, Backbone network (WAN) among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.