IL-0491 Squarespace · 2019-02-12networking
Squarespace's CDN origin fleet fell below its healthy node threshold, and images, scripts and stylesheets failed to load
Squarespace serves site media, scripts and stylesheets through global CDN providers in front of its own CDN origin fleet. A maintenance operation that an engineer started across the whole origin fleet at 10:30 PM ET on February 11 continued in the background after the engineer thought it had stopped, and at 12:10 AM it caused an outage of the CDN origin service. Assets that the CDN providers had cached were still served, but other requests to static.squarespace.com and static1.squarespace.com failed.
The on-call engineer was alerted only when the fleet fell below its critical availability threshold, and the customer impact was underestimated until about 1:22 AM, when restarts had failed and the issue was escalated. An Incident Commander took over communication at 1:30 AM. Further remediation steps restored partial service by 2:15 AM and normal behaviour at 2:20 AM, and the incident was declared resolved at 3:03 AM after a period of healthy operation. The cause was found in an incident investigation the next morning.
- Severity
- –
- Scope
- Images, scripts and style sheets on static.squarespace.com and static1.squarespace.com did not load for some Squarespace sites whose assets were not in the CDN providers' cache
- Users affected
- some Squarespace sites and their visitors
- Region
- –
- Time to detect
- 0m
- Time to resolve
- 2h 10m
Symptoms
At 12:10 AM ET on Tuesday, February 12, 2019, Squarespace's monitoring alerted the on-call engineer for the CDN origin fleet that the fleet had fallen below a safe number of healthy nodes. Requests to static.squarespace.com and static1.squarespace.com failed, so images, scripts and style sheets did not load for some sites, and the status page later reported elevated error rates serving media and stylesheets.
Many sites were not affected, because Squarespace's global CDN providers kept serving cached assets. The team first misjudged how much customers were affected and did not bring in an Incident Commander. Basic restarts of the affected service did not help, and only by 1:22 AM, after escalation to more team members, did the severity become clear; an Incident Commander was paged at 1:30 AM and posted on the status page.
Symptom High error rate
Root cause
Root cause Operator error
- Trigger
- At 10:30 PM ET on February 11, an engineer started a maintenance operation across the CDN origin fleet. The operation kept running in the background after the engineer believed it had been terminated.
Detection
The monitoring system alerted the CDN origin fleet's on-call engineer at 12:10 AM ET when the fleet fell below a safe number of healthy nodes.
Remediation
- Restored the CDN origin fleet, partially by 2:15 AM ET and to normal behaviour at 2:20 AM ET
Action items
- Revise the incident-response playbooks with checks and actions for this fleet, starting with an assessment of customer impact and the immediate involvement of an Incident Commander
- Improve external monitoring to assess customer impact of issues with the CDN partners and origin services more quickly
- Revise operational policies and add safeguards for issuing maintenance commands, including limiting the share of the fleet a command can operate on
- Reduce the need for direct manual maintenance operations
- Create independent failure domains and route requests automatically around unhealthy parts of the system
Lessons
–
Sources
- https://status.squarespace.com/incidents/pbn8bwh6w8wk · official-status
Related cases
–
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0491
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0491/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.