Incident Library

IL-0584 Monicle · 2024-12-04 (published)gcpnetworking

Cloud CDN at Osaka served corrupted JavaScript files, and a single-page app failed with a SyntaxError in western Japan

Monicle served a single-page app through Google Cloud Load Balancing and Cloud CDN, with Cloud Run as the backend. The app delivers a large JavaScript bundle and HTML with some initial data rendered on the server, and provides its APIs over GraphQL. In 2024 some users in western Japan could not use the app because the JavaScript files they received were corrupted, while the files in the container image were correct. A war room was opened as soon as the reports came in. Redeploying the container images did not help. Comparing CDN locations showed the corruption at KIX (Osaka) and not at NRT (Tokyo).

The team first purged the Cloud CDN cache. While investigating, the team noticed that Cloud CDN was enabled on Cloud Load Balancing but content compression was not. Reasoning that compression could change the outcome if the fault was in the transfer from Cloud Load Balancing to Cloud CDN or in how Cloud CDN stored objects in its cache, the team enabled compression_mode. After that the system worked for users again and the incident was closed; users also reported faster page loads because the JavaScript transfer was smaller. The team searched Issue Tracker and Cloud Forums and found no matching reports, and enabling compression resolved the problem as a result.

Severity
–
Scope
A single-page app served through Cloud Load Balancing and Cloud CDN stopped working in the browser with a SyntaxError for some users
Users affected
Some users in western Japan
Region
Western Japan (Cloud CDN at KIX, Osaka)
Time to detect
–
Time to resolve
–

Symptoms

Some users reported that a Monicle web service could not be used. The service is a single-page app (SPA) served through Google Cloud Load Balancing and Cloud CDN, and in the affected browsers the app stopped with a SyntaxError. Users in eastern Japan were not affected; users in western Japan were.

The JavaScript files the system delivered were corrupted: the same code appeared twice in a file. The files inside the container image were intact. The file sizes recorded in the logs, among other signs, pointed the team at the CDN. A developer who happened to be working remotely in Okinawa could reproduce the failure, confirmed that the problem was in the SPA's JavaScript files and saved a copy of a corrupted file.

The failure reproduced when the files came from Cloud CDN at KIX (Osaka) and did not reproduce when they came from NRT (Tokyo).

Before the CDN was suspected, the product's development team rebuilt and redeployed the Cloud Run container images and examined the images that might be faulty. This neither restored the service nor showed the cause.

Symptom Response quality degradation

Root cause

Root cause Cloud provider outage

Component CDN and edge network
Contributing factor –
Trigger
–

Detection

Some users reported that they could not use the system, and a war room was opened right away because of the product and the extent of the impact.

Remediation

Action items

Lessons

The author states that there is no specific countermeasure for this incident yet. The lesson drawn is to keep in mind which parts of the service are single points of failure and to be ready to act quickly on failures in PaaS and SaaS products.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, CDN and edge network among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.