Incident Library
Applied filters Root causeFaulty deploymentComponentNetwork control plane (SDN)SymptomStale endpoint routingClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0530 Virtual router controllers in us-central1 ran out of memory in a loop, cutting networking for new and migrated VMs
    Google Cloud2023
    Symptoms

    On 5 October 2023 in us-central1, newly created VMs waited a long time before their networking worked, and VMs that had been live migrated lost connectivity for a long time after the migration. Networking telemetry showed cross-region packet loss to and from us-central1 from 03:08 US/Pacific. Health check state of VMs reached load balancers late.

    Packet lossNetwork unreachableOOMKilledStale endpoint routingCluster or node provisioning stall
    Trigger

    A management plane behaviour change, meant to decouple the processing of API updates to Instance Groups and Network Endpoint Groups used as load balancer backends, was deployed in us-central1 after rolling out to several other regions without incident.

    Root cause
    Faulty deployment
    Component
    Network control plane (SDN)Cloud load balancer
    Contributing factor
    Excessive blast radiusNo alert on limit
    Detection

    Engineers were alerted to slowness in the virtual network control plane in us-central1 at 21:45 US/Pacific on 4 October, before the packet loss started, and to controller failures from memory exhaustion at 02:11 on 5 October. Networking telemetry showed cross-region packet loss from 03:08.

    Remediation
    • Increased the memory allocated to the virtual network router controllers
    • Rolled back the management plane change in us-central1
    • Restarted by hand the routers that were still not accepting updates
    Action items
    • Alert proactively on memory risks and unexpected memory increases
    • Refactor the deployment configuration so engineers can reallocate memory much faster
    • Re-evaluate existing practices and add safety mechanisms for API reconciliation
    • Make management plane changes more visible across teams so the changes can be correlated faster
    • Adjust the deployment footprint so a regional API change is less likely to exhaust memory across the region at once
    • Optimise memory use in the traffic routers and their controllers
    View case
Check your cluster for this

This case involves Cloud load balancer and Network control plane (SDN). kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment1 case

Component
Cloud load balancerNetwork control plane (SDN)
Contributing factor
Excessive blast radiusNo alert on limit
Organisation
Google Cloud
  1. IL-0530 Virtual router controllers in us-central1 ran out of memory in a loop, cutting networking for new and migrated VMs
    Google Cloud2023