Incident Library
Applied filters SymptomDNS resolution failuresFactorCache-masked failureOrganisationSquareClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0254 Square's internal DNS servers failed and Square products, services and payments were unavailable for about 15 hours
    Square2023
    Symptoms

    From 1:54 PM ET on 7 September 2023, Square products and services were unavailable, and at 2:02 PM ET engineers were notified of infrastructure issues and began investigating DNS. Square Online websites stayed up, but their customers could not make payments.

    DNS resolution failuresService outage
    Trigger

    At 11:04 AM ET a host-based firewall rule change was deployed to enable region communication, which increased the size of the firewall rules on nodes. Combined with a DNS service upgrade, it put unexpected load on the internal DNS servers. The timeline also lists a DNS zone change at 1:56 PM ET.

    Root cause
    Configuration change
    Component
    Authoritative DNS servers and recordsNode iptables rules
    Contributing factor
    No staged rolloutCache-masked failureCircular dependency
    Detection

    Engineers were notified of infrastructure issues at 2:02 PM ET, eight minutes after services became unavailable.

    Remediation
    • Shed network traffic to the DNS servers and started bringing up new DNS servers by hand
    • Deployed networking changes by hand to re-enable the authorization and access services
    • Deployed the networking changes by hand to all data centers
    • Rebuilt the DNS servers
    • Built a new configuration from the reverted firewall ruleset and pushed it to the DNS hosts
    Action items
    • Move the DNS infrastructure onto isolated infrastructure
    • Add monitoring and optimizations for critical networking infrastructure
    • Optimize dependencies between the deploy and platform infrastructure where feasible
    • Expand support for Offline Mode and improve communication about when it is available
    View case
Check your cluster for this

This case involves Authoritative DNS servers and records and Node iptables rules. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change1 case

Component
Authoritative DNS servers and recordsNode iptables rules
Contributing factor
Circular dependencyCache-masked failureNo staged rollout
Organisation
Square
  1. IL-0254 Square's internal DNS servers failed and Square products, services and payments were unavailable for about 15 hours
    Square2023