Cases1 case
1 to 1 of 1
IL-0253Firefox froze and could not load any web content for close to two hours, stuck in an HTTP/3 request to TelemetryMozilla2022- Symptoms
On 13 January 2022, shortly after 07:28 UTC, Mozilla saw a spike in crashes reported through the Firefox crash reporter and received several reports, from inside and outside Mozilla, of the browser hanging. For close to two hours Firefox was unusable for users worldwide and could not load web content.
- Trigger
At 07:28 UTC on 13 January 2022 GCP deployed an unannounced change that made HTTP/3 the default for load balancers set to "Automatic (default)", so some Firefox connections to Mozilla's services, including Telemetry, started using HTTP/3 instead of HTTP/2.
- Root cause
- Component
- Contributing factor
- Detection
A spike in crashes reported through the Firefox crash reporter, and reports of browser hangs from inside and outside Mozilla.
- Remediation
- Explicitly disabled HTTP/3 on the GCP load balancers at 09:12 UTC
- Action items
- Work with GCP to improve the situation around unannounced changes such as the HTTP/3 default
- Review all service configurations to avoid similar mistakes
- Run more system tests with different HTTP versions
- Find ways to make the browser more resilient to such problems and to speed up incident response
- Lessons
The hang needed three things together: a provider-side change to the HTTP version, a Rust component calling the network stack through viaduct, and a case-sensitive header lookup deep in the HTTP/3 code. Leaving the load balancers on an automatic default let the provider's change apply without any action from Mozilla, and an announcement would likely have led to a controlled test in staging first. A change of HTTP version and an extra networking layer such as viaduct are significant enough to be covered by system tests.
This case involves Cloud load balancer and HTTP and RPC client library. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.