Cases2 cases

1 to 2 of 2

1. IL-0530 Virtual router controllers in us-central1 ran out of memory in a loop, cutting networking for new and migrated VMs

   Google Cloud2023

   Symptoms

   On 5 October 2023 in us-central1, newly created VMs waited a long time before their networking worked, and VMs that had been live migrated lost connectivity for a long time after the migration. Networking telemetry showed cross-region packet loss to and from us-central1 from 03:08 US/Pacific. Health check state of VMs reached load balancers late.

   Packet lossNetwork unreachableOOMKilledStale endpoint routingCluster or node provisioning stall

   Trigger

   A management plane behaviour change, meant to decouple the processing of API updates to Instance Groups and Network Endpoint Groups used as load balancer backends, was deployed in us-central1 after rolling out to several other regions without incident.

   Root cause

   Faulty deployment

   Component

   Network control plane (SDN)Cloud load balancer

   Contributing factor

   Excessive blast radiusNo alert on limit

   Detection

   Engineers were alerted to slowness in the virtual network control plane in us-central1 at 21:45 US/Pacific on 4 October, before the packet loss started, and to controller failures from memory exhaustion at 02:11 on 5 October. Networking telemetry showed cross-region packet loss from 03:08.

   Remediation

   • Increased the memory allocated to the virtual network router controllers
   • Rolled back the management plane change in us-central1
   • Restarted by hand the routers that were still not accepting updates

   Action items

   • Alert proactively on memory risks and unexpected memory increases
   • Refactor the deployment configuration so engineers can reallocate memory much faster
   • Re-evaluate existing practices and add safety mechanisms for API reconciliation
   • Make management plane changes more visible across teams so the changes can be correlated faster
   • Adjust the deployment footprint so a regional API change is less likely to exhaust memory across the region at once
   • Optimise memory use in the traffic routers and their controllers

   View case
2. IL-0251 Most of Slack's HAProxy load balancers routed to a stale, shrinking set of webapp backends and served HTTP 503 errors

   Slack2020

   Symptoms

   At 4:45pm Pacific on 12 May 2020, Slack was alerted that it was serving more HTTP 503 errors than normal. A subset of the webapp fleet was under heavy load while the rest of the webapp instances were not.

   High error rateLoad imbalanceStale endpoint routing

   Trigger

   At the end of the US business day the webapp tier began its usual scale-down, and autoscaling terminated the oldest instances first, which were the only backends left in most HAProxy instances' state.

   Root cause

   Software defect

   Component

   HAProxyService discovery

   Contributing factor

   Automation amplificationNo alert on limit

   Detection

   An alert for more HTTP 503 errors than normal. The alerting meant to catch stale HAProxy state was not working.

   Remediation

   • Did a rolling restart of the HAProxy fleet

   Action items

   • Move all ingress load balancing from HAProxy to Envoy Proxy, with endpoint discovery through an xDS control plane

   Lessons

   The alerting for this exact failure had stopped working, and nobody noticed because the HAProxy stack had run for a long time without needing changes, so few engineers worked with its monitoring and alerting. A slot limit that had been ample a few months earlier was reached after pandemic-driven growth and one large autoscaling event.

   View case

By root cause2