Incident Library

Organisation

Square

Square · square

1 case · 2023

Root cause Configuration change
Component Authoritative DNS servers and recordsNode iptables rules
Contributing factor Circular dependencyCache-masked failureNo staged rollout
Symptom DNS resolution failuresService outage

Cases

  1. IL-0254 Square's internal DNS servers failed and Square products, services and payments were unavailable for about 15 hoursSquare · 2023

    Root cause Configuration change