IL-0254 Square · 2023-09-07networking

Square's internal DNS servers failed and Square products, services and payments were unavailable for about 15 hours

Square runs in multiple data center regions, and its services find their dependencies and serve requests through DNS and mesh-based routing, so without DNS Square products, internal tools and services cannot talk to each other. A central service manages Square's host-based firewall policy and pushes it to nodes, which expand the policy into firewall rules; it uses an accelerated rollout strategy to adapt quickly to changes in the environment. On 7 September 2023 a small policy change expanded into a much larger ruleset. The large ruleset made nodes unstable, and combined with the traffic pattern of DNS and a DNS service upgrade it made the internal DNS servers start failing requests. Once node-based DNS caches expired, services could not reach their dependencies, and the outage reached internal tools needed for recovery, which lengthened it.

Engineers recovered internal access first, then deployed networking changes by hand to re-enable authorization and access services and then to all data centers, while shedding traffic to the DNS servers and bringing up new ones. Adding DNS capacity did not help. The DNS servers were rebuilt from 10:06 PM ET, and a configuration based on the reverted ruleset was pushed to the DNS hosts from 11:52 PM ET. All DNS servers were healthy at 12:55 AM ET on 8 September, payment traffic had fully recovered by 2:40 AM ET, edge routing by 3:12 AM ET, and the status page incident was resolved at 5:19 AM ET. Offline-mode payments, whose processing Square deferred as a precaution, were fully processed by 1:57 PM ET on 8 September. A forensic analysis ruled out a cyberattack and found no evidence of a data breach or loss.

Severity: –
Scope: All Square products and services, internal tools and payment processing; Square Online websites stayed up but could not take payments, and offline-mode payments were processed hours later
Users affected: Square sellers and their customers
Region: –
Time to detect: 8m
Time to resolve: 15h 25m

Symptoms

From 1:54 PM ET on 7 September 2023, Square products and services were unavailable, and at 2:02 PM ET engineers were notified of infrastructure issues and began investigating DNS. Square Online websites stayed up, but their customers could not make payments.

Square's internal DNS servers were failing requests. Services kept working until the DNS caches on their nodes expired; after that they could not reach their dependencies, and external requests failed. Many of the services used for troubleshooting and recovery were also affected, so from 2:52 PM ET engineers first had to recover internal access and tooling. Internal access services came back at 6:29 PM ET, and only then could engineers work in parallel on the authorization and control plane services; the deployment pipeline recovered at 8:36 PM ET.

At 3:56 PM ET the team shed network traffic to the DNS servers and started bringing up new DNS servers by hand. Increasing DNS service capacity at 6:00 PM ET did not help. DNS recovered only after the DNS servers were rebuilt and a configuration built from the reverted firewall ruleset was pushed to the DNS hosts from 11:52 PM ET; all DNS servers were healthy at 12:55 AM ET on 8 September.

Symptom DNS resolution failures Service outage

Root cause

Root cause Configuration change

Component Authoritative DNS servers and records Node iptables rules

Contributing factor No staged rollout Cache-masked failure Circular dependency

Trigger: At 11:04 AM ET a host-based firewall rule change was deployed to enable region communication, which increased the size of the firewall rules on nodes. Combined with a DNS service upgrade, it put unexpected load on the internal DNS servers. The timeline also lists a DNS zone change at 1:56 PM ET.

Detection

Engineers were notified of infrastructure issues at 2:02 PM ET, eight minutes after services became unavailable.

Remediation

Shed network traffic to the DNS servers and started bringing up new DNS servers by hand
Deployed networking changes by hand to re-enable the authorization and access services
Deployed the networking changes by hand to all data centers
Rebuilt the DNS servers
Built a new configuration from the reverted firewall ruleset and pushed it to the DNS hosts

Action items

Move the DNS infrastructure onto isolated infrastructure
Add monitoring and optimizations for critical networking infrastructure
Optimize dependencies between the deploy and platform infrastructure where feasible
Expand support for Offline Mode and improve communication about when it is available

Lessons

–

Sources

https://developer.squareup.com/blog/incident-summary-2023-09-07/ · blog · archived copy

Related cases

IL-0512 xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failedMIT Open Learning · 2026 · sev1
same root cause · shared Authoritative DNS servers and records
IL-0502 The JupyterHub at temple.2i2c.cloud stopped resolving in DNS after a wildcard record was added for a new cluster2i2c · 2025
same root cause · shared Authoritative DNS servers and records
IL-0533 Names in chinacloudapp.cn and chinacloudsites.cn stopped resolving as DNS caches expired, breaking Azure China servicesMicrosoft Azure · 2024
same root cause · shared Authoritative DNS servers and records
IL-0549 Media, attachments, Git LFS and build artifacts failed for Atlassian in us-west-1 when an old network stack was removedAtlassian · 2023
same root cause · shared Authoritative DNS servers and records
IL-0186 Pods on one Kubernetes worker node could not reach external HTTP and HTTPS because storage components had added nat PREROUTING rules that redirected ports 80 and 443 to local portsHans H. Kim · 2023
same root cause · shared Node iptables rules
IL-0363 static.crates.io did not resolve for 10-15 minutes during a Terraform deployment, so Rust builds could not fetch cratesRust Project · 2023
same root cause · shared Authoritative DNS servers and records

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Authoritative DNS servers and records, Node iptables rules among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0254
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0254/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.