Cases1 case

1 to 1 of 1

1. IL-0584 Cloud CDN at Osaka served corrupted JavaScript files, and a single-page app failed with a SyntaxError in western Japan

   Monicle2024

   Symptoms

   Some users reported that a Monicle web service could not be used. The service is a single-page app (SPA) served through Google Cloud Load Balancing and Cloud CDN, and in the affected browsers the app stopped with a SyntaxError. Users in eastern Japan were not affected; users in western Japan were.

   Response quality degradation

   Root cause

   Cloud provider outage

   Component

   CDN and edge network

   Contributing factor

   –

   Detection

   Some users reported that they could not use the system, and a war room was opened right away because of the product and the extent of the impact.

   Remediation

   • Purged the Cloud CDN cache after the corruption was seen at KIX (Osaka) but not at NRT (Tokyo)
   • Enabled compression_mode for Cloud CDN content on Cloud Load Balancing, after which the app worked for users again

   Action items

   • Stay aware of which parts of the service are single points of failure, remove them where the cost is justified, and prepare to respond quickly to failures in PaaS and SaaS products

   Lessons

   The author states that there is no specific countermeasure for this incident yet. The lesson drawn is to keep in mind which parts of the service are single points of failure and to be ready to act quickly on failures in PaaS and SaaS products.

   View case

By root cause1