Incident Library
Applied filters Root causeOperator errorSymptomHigh error rateOrganisationSquarespaceClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0491 Squarespace's CDN origin fleet fell below its healthy node threshold, and images, scripts and stylesheets failed to load
    Squarespace2019
    Symptoms

    At 12:10 AM ET on Tuesday, February 12, 2019, Squarespace's monitoring alerted the on-call engineer for the CDN origin fleet that the fleet had fallen below a safe number of healthy nodes. Requests to static.squarespace.com and static1.squarespace.com failed, so images, scripts and style sheets did not load for some sites, and the status page later reported elevated error rates serving media and stylesheets.

    High error rate
    Trigger

    At 10:30 PM ET on February 11, an engineer started a maintenance operation across the CDN origin fleet. The operation kept running in the background after the engineer believed it had been terminated.

    Root cause
    Operator error
    Component
    CDN and edge network
    Contributing factor
    Excessive blast radiusMonitoring blind spot
    Detection

    The monitoring system alerted the CDN origin fleet's on-call engineer at 12:10 AM ET when the fleet fell below a safe number of healthy nodes.

    Remediation
    • Restored the CDN origin fleet, partially by 2:15 AM ET and to normal behaviour at 2:20 AM ET
    Action items
    • Revise the incident-response playbooks with checks and actions for this fleet, starting with an assessment of customer impact and the immediate involvement of an Incident Commander
    • Improve external monitoring to assess customer impact of issues with the CDN partners and origin services more quickly
    • Revise operational policies and add safeguards for issuing maintenance commands, including limiting the share of the fleet a command can operate on
    • Reduce the need for direct manual maintenance operations
    • Create independent failure domains and route requests automatically around unhealthy parts of the system
    View case
Check your cluster for this

This case involves CDN and edge network. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Operator error1 case

Component
CDN and edge network
Contributing factor
Excessive blast radiusMonitoring blind spot
Organisation
Squarespace
  1. IL-0491 Squarespace's CDN origin fleet fell below its healthy node threshold, and images, scripts and stylesheets failed to load
    Squarespace2019