Cases1 case

1 to 1 of 1

1. IL-0491 Squarespace's CDN origin fleet fell below its healthy node threshold, and images, scripts and stylesheets failed to load

   Squarespace2019

   Symptoms

   At 12:10 AM ET on Tuesday, February 12, 2019, Squarespace's monitoring alerted the on-call engineer for the CDN origin fleet that the fleet had fallen below a safe number of healthy nodes. Requests to static.squarespace.com and static1.squarespace.com failed, so images, scripts and style sheets did not load for some sites, and the status page later reported elevated error rates serving media and stylesheets.

   High error rate

   Trigger

   At 10:30 PM ET on February 11, an engineer started a maintenance operation across the CDN origin fleet. The operation kept running in the background after the engineer believed it had been terminated.

   Root cause

   Operator error

   Component

   CDN and edge network

   Contributing factor

   Excessive blast radiusMonitoring blind spot

   Detection

   The monitoring system alerted the CDN origin fleet's on-call engineer at 12:10 AM ET when the fleet fell below a safe number of healthy nodes.

   Remediation

   • Restored the CDN origin fleet, partially by 2:15 AM ET and to normal behaviour at 2:20 AM ET

   Action items

   • Revise the incident-response playbooks with checks and actions for this fleet, starting with an assessment of customer impact and the immediate involvement of an Incident Commander
   • Improve external monitoring to assess customer impact of issues with the CDN partners and origin services more quickly
   • Revise operational policies and add safeguards for issuing maintenance commands, including limiting the share of the fleet a command can operate on
   • Reduce the need for direct manual maintenance operations
   • Create independent failure domains and route requests automatically around unhealthy parts of the system

   View case

By root cause1