Incident Library

IL-0253 Mozilla · 2022-01-13gcpnetworking

Firefox froze and could not load any web content for close to two hours, stuck in an HTTP/3 request to Telemetry

Firefox relies on Mozilla-run services for updates, telemetry, certificate management and crash reporting, hosted by several cloud providers behind load balancers. On Google Cloud Platform these load balancers have an HTTP/3 setting with the states "Enabled", "Disabled" and "Automatic (default)", and Mozilla's were on "Automatic (default)". When GCP made HTTP/3 the default, Firefox, which uses HTTP/3 whenever a server supports it, began sending some requests to these services over HTTP/3. The most critical services, such as updates, use a beConservative flag that keeps experimental technology such as HTTP/3 out of their connections.

All HTTP/3 traffic goes through Necko, the Firefox networking stack. Rust components reach Necko through an intermediate library called viaduct, which lower-cases every header. For HTTP/3 uploads, the higher-level Necko API checks for a Content-Length header without regard to case and adds one if it is missing, while the lower-level HTTP/3 code looks it up case-sensitively. Telemetry was the only Rust component in Firefox Desktop that used the network stack and added a Content-Length header. The HTTP/3 code did not find the lower-cased header, considered the request complete while its body was still unsent, and then looped forever because it only stopped when nothing was left to send. The loop ran on the single socket thread, so the whole browser lost network access.

Mozilla explicitly disabled HTTP/3 on its GCP load balancers at 09:12 UTC, which unblocked users before the root cause was understood. Mozilla named three contributing factors: GCP's change was not announced, the "Automatic (default)" setting let it apply automatically, and the continuous integration system did not cover HTTP/3 combined with viaduct on Firefox Desktop, since the HTTP/3 tests covered the protocol and Necko only as web content uses them.

Severity
–
Scope
Firefox could not load web content; the browser hung on its single network socket thread
Users affected
Firefox users worldwide
Region
worldwide
Time to detect
–
Time to resolve
1h 44m

Symptoms

On 13 January 2022, shortly after 07:28 UTC, Mozilla saw a spike in crashes reported through the Firefox crash reporter and received several reports, from inside and outside Mozilla, of the browser hanging. For close to two hours Firefox was unusable for users worldwide and could not load web content.

The backlog of unprocessed crash reports grew to close to 300K. Incident response found that the browser was hanging inside a network request to one of the Firefox internal services. All network requests in Firefox go through one socket thread, so that one stuck request blocked every other network request. Users who had disabled Telemetry did not see the problem.

The team first looked for a change of its own and found that no updates or configuration changes had shipped. HTTP/3 had been enabled since Firefox 88 and was in active use on popular websites, so it did not look like a new factor. Suspecting an invisible change by a cloud provider, the team checked its load balancer settings, which had not changed, and then found in logs that the load balancers of the Telemetry service were serving HTTP/3 connections, which they had not done before. Tests with various tools and remote services, even against the Telemetry staging server, did not reproduce the hang; only Firefox itself did, against the staging server.

Symptom Service outage

Root cause

Root cause Software defect

Component Cloud load balancerHTTP and RPC client library
Contributing factor Unreviewed defaultsGap in automated tests
Trigger
At 07:28 UTC on 13 January 2022 GCP deployed an unannounced change that made HTTP/3 the default for load balancers set to "Automatic (default)", so some Firefox connections to Mozilla's services, including Telemetry, started using HTTP/3 instead of HTTP/2.

Detection

A spike in crashes reported through the Firefox crash reporter, and reports of browser hangs from inside and outside Mozilla.

Remediation

Action items

Lessons

The hang needed three things together: a provider-side change to the HTTP version, a Rust component calling the network stack through viaduct, and a case-sensitive header lookup deep in the HTTP/3 code. Leaving the load balancers on an automatic default let the provider's change apply without any action from Mozilla, and an announcement would likely have led to a controlled test in staging first. A change of HTTP version and an extra networking layer such as viaduct are significant enough to be covered by system tests.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Cloud load balancer, HTTP and RPC client library among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.