Cases8 cases

3 to 7 of 8

1. IL-0233 A VPC control plane release broke IP alias routes in Nebius eu-west1, and Managed Kubernetes pods lost connectivity

   Nebius2025

   Symptoms

   On 13 March 2025 practically every pod of Nebius Managed Services for Kubernetes in the eu-west1 region lost network connectivity for at least some time between 13:36 and 18:11 UTC. Packets for the affected IP alias routes were not processed by the data plane. The eu-north1 region was not affected.

   Network unreachable

   Trigger

   A release of a VPC control plane component in eu-west1, deployed to canaries from 13:36 UTC and to all other nodes from 14:35 UTC, changed route distinguisher values for a new feature. The change made a long-standing bug in the handling of route announcements take effect for the update messages of Managed Kubernetes pods.

   Root cause

   Software defect

   Component

   Network control plane (SDN)BGP routing

   Contributing factor

   Ineffective rollbackGap in automated testsMonitoring blind spot

   Remediation

   • Restarted the host VPC component on the affected nodes one node at a time
   • Restarted the host VPC component on the remaining nodes in parallel once the restart was known to fix the problem

   Action items

   • Fix the bug in the VPC host component itself
   • Add an alert on a diverging number of routes
   • Add an alert on the number of drops in the data plane
   • Develop a way to reset endpoints without restarting the host VPC component
   • Move the VPC's persistent e2e test virtual machines to canary hosts
   • Add persistent e2e tests for Managed Services for Kubernetes
   • Run e2e tests more often
   • Add a deployment stage in which non-critical internal services receive updates first

   Lessons

   The failure needed both an existing IP alias at release time and a particular order of node updates, and neither the testing installation nor the canaries met both, so neither caught it. Missing alerts on technical metrics that deviate from normal, here the number of routes and the number of drops, made the response much slower.

   View case
2. IL-0340 Cloudflare's 1.1.1.1 resolver returned SERVFAIL to some valid DNS queries for four hours

   Cloudflare2023

   Symptoms

   From 07:00 to 11:00 UTC on 4 October 2023, some users of Cloudflare's 1.1.1.1 DNS resolver received SERVFAIL responses to valid queries. This reached users of WARP and Zero Trust and of third-party DNS resolvers that use 1.1.1.1, and the first external reports of unexpected SERVFAILs arrived at 7:57 UTC.

   DNS resolution failuresHigh error rate

   Trigger

   On 21 September 2023, as a planned change in root zone management, a ZONEMD resource record, in effect a checksum of the zone's contents, was added to the root zone for the first time. On 4 October at 07:00 UTC the DNSSEC signatures in the last root zone copy that 1.1.1.1 had loaded expired.

   Root cause

   Software defect

   Component

   Recursive DNS resolver service

   Contributing factor

   Cache-masked failureUnsafe failure behaviourMonitoring blind spotGap in automated tests

   Detection

   External reports of unexpected SERVFAILs started coming in at 7:57 UTC, and Cloudflare declared an internal incident at 8:03 UTC. Nothing had alerted on static_zone serving a stale root zone during the two weeks before.

   Remediation

   • Deployed an override rule meant to stop 1.1.1.1 from answering with the stale root zone, which did not cover forwarded queries
   • Stopped 1.1.1.1 from preloading the root zone file entirely

   Action items

   • Add alerts for when static_zone serves a stale root zone file
   • Re-evaluate how the root zone is ingested and distributed internally so new record types are handled
   • Improve test coverage for what happens when the root zone fails to parse
   • Limit how long stale copies of the root zone can be used, following RFC 8806

   Lessons

   Cloudflare's main lesson is never to assume an input will not change. Modern systems pull in long chains of libraries that may not be ready when their input changes, so they need tests that catch regressions and components that fail gracefully on new input, and format changes in DNS and BGP should be expected to have an impact. Cloudflare also writes that with better monitoring of the stale root zone, the existing caching would have prevented any impact.

   View case
3. IL-0253 Firefox froze and could not load any web content for close to two hours, stuck in an HTTP/3 request to Telemetry

   Mozilla2022

   Symptoms

   On 13 January 2022, shortly after 07:28 UTC, Mozilla saw a spike in crashes reported through the Firefox crash reporter and received several reports, from inside and outside Mozilla, of the browser hanging. For close to two hours Firefox was unusable for users worldwide and could not load web content.

   Service outage

   Trigger

   At 07:28 UTC on 13 January 2022 GCP deployed an unannounced change that made HTTP/3 the default for load balancers set to "Automatic (default)", so some Firefox connections to Mozilla's services, including Telemetry, started using HTTP/3 instead of HTTP/2.

   Root cause

   Software defect

   Component

   Cloud load balancerHTTP and RPC client library

   Contributing factor

   Unreviewed defaultsGap in automated tests

   Detection

   A spike in crashes reported through the Firefox crash reporter, and reports of browser hangs from inside and outside Mozilla.

   Remediation

   • Explicitly disabled HTTP/3 on the GCP load balancers at 09:12 UTC

   Action items

   • Work with GCP to improve the situation around unannounced changes such as the HTTP/3 default
   • Review all service configurations to avoid similar mistakes
   • Run more system tests with different HTTP versions
   • Find ways to make the browser more resilient to such problems and to speed up incident response

   Lessons

   The hang needed three things together: a provider-side change to the HTTP version, a Rust component calling the network stack through viaduct, and a case-sensitive header lookup deep in the HTTP/3 code. Leaving the load balancers on an automatic default let the provider's change apply without any action from Mozilla, and an announcement would likely have led to a controlled test in staging first. A change of HTTP version and an extra networking layer such as viaduct are significant enough to be covered by system tests.

   View case
4. IL-0345 Many Render services in the Oregon region returned 503 errors for about four hours

   Render2020

   Symptoms

   Between 13:34 and 17:33 PST on Sunday 6 December 2020, many Render services in the Oregon region returned 503 (Service Unavailable) errors at an elevated rate, and the Render Dashboard was affected. Render's monitoring alerted the team at once. The status page reported upstream load balancers having trouble connecting to the rest of the infrastructure, and after an apparent fix the connectivity error rates rose again before the final fix.

   High error rate

   Trigger

   At 13:34 PST a customer API hosted in Oregon, one of the services with the highest request rates in the region, began holding incoming requests until they timed out.

   Root cause

   Software defect

   Component

   Cloud load balancer

   Contributing factor

   Excessive blast radiusGap in automated tests

   Detection

   Render's monitoring system notified the team immediately.

   Remediation

   • Rolled out a fix to the load balancing algorithm across all load balancers

   Action items

   • Add mechanisms that isolate services from each other in the routing and load balancing layers
   • Increase test coverage of the load balancing code
   • Add synthetic failure states to the continuous integration test suites
   • Put more focus on chaos engineering

   View case
5. IL-0338 Google Cloud HTTP(S) and TCP/SSL Proxy load balancers returned 502s or reset connections on 33% to 87% of requests

   Google Cloud2018

   Symptoms

   From 12:17 to 12:49 PDT on 17 July 2018, Google Cloud HTTP(S) Load Balancers answered a share of requests with 502 return codes, and connections to TCP/SSL Proxy Load Balancers were reset. The share of failed requests moved between 33% and 87% during those 32 minutes, and automated monitoring alerted Google engineers at 12:19.

   High error rateConnection resetsHigh latency

   Trigger

   A configuration change in the production environment intermittently set off a bug in a new feature of the second-layer Google Front Ends.

   Root cause

   Software defect

   Component

   Cloud load balancerCDN and edge network

   Contributing factor

   Gap in automated testsExcessive blast radius

   Detection

   Automated monitoring alerted Google's engineering team at 12:19 PDT, two minutes after the 502s began.

   Remediation

   • Reverted the configuration change at 12:44 PDT, which stopped the GFE restarts

   Action items

   • Add safeguards that disable features not yet in service
   • Harden the GFE testing stack against latent bugs in production binaries that make a task restart
   • Isolate the shards of GFE pools further from each other to reduce the scope of failures
   • Build a consolidated dashboard of all configuration changes for GFE pools

   View case

By root cause1