Cases11 cases

4 to 8 of 11

1. IL-0260 zoom.us stopped resolving for about two hours after GoDaddy Registry placed a server block on the domain

   Zoom2025

   Symptoms

   On 16 April 2025 customers in the US and International regions could not reach Zoom services because lookups of the zoom.us domain failed. Starting, joining or scheduling a meeting needs a DNS lookup, so none of those actions completed; people already in a Zoom meeting or on a Zoom Phone call were not affected.

   DNS resolution failuresService outage

   Trigger

   A communication error between Zoom's domain registrar, Markmonitor, and GoDaddy Registry led GoDaddy Registry to place a server block on zoom.us by mistake, which shut the domain down.

   Root cause

   Dependency failure

   Component

   Authoritative DNS servers and records

   Contributing factor

   No safeguard on destructive change

   Remediation

   • Identified and removed the server block on zoom.us together with Markmonitor and GoDaddy
   • Asked users who still had connection problems to flush their DNS cache and reconnect

   Action items

   • Put a registry lock on zoom.us that restricts server block commands on the domain
   • Implement additional technical changes and processes with GoDaddy Registry and Markmonitor so the issue does not recur

   View case
2. IL-0348 Scaleway's FR-PAR region lost public internet connectivity when one 100 Gbps backbone link was saturated

   Scaleway2025

   Symptoms

   On 10 March 2025 at 16:22 UTC, public internet connectivity in Scaleway's FR-PAR region broke for all products. Most internet traffic between the FR-PAR-2 availability zone and the rest of the PAR region was blackholed for about five minutes, while VPC traffic and Dedibox RPN kept working.

   Packet lossService outageHigh latency

   Trigger

   While Scaleway's network team connected a new data center to the FR-PAR backbone, a human mistake in a router configuration rerouted more than 1 Tbps of traffic between OpCore DC3 and DC5 through a single 100 Gbps link.

   Root cause

   Operator error

   Component

   Backbone network (WAN)Object storageAuthoritative DNS servers and records

   Contributing factor

   Automation amplification

   Detection

   The router configuration error was detected quickly and rolled back within four minutes.

   Remediation

   • Rolled back the router configuration, restoring the correct configuration at 16:27 UTC

   Action items

   • Harden the peer review and production communication process for changes
   • Add failsafes and tests before network configuration is applied at backbone level
   • Deploy the fix for the Object Storage synchronisation software issue

   View case
3. IL-0269 Octez Connect services went down for hours when its only DNS provider failed and the nodes' records stopped resolving

   Octez Connect2025

   Symptoms

   On 5 February 2025 at 04:34 the monitoring system flagged DNS resolution problems on the first Octez Connect node and raised a Pager Duty alert. Every server that relied on the same DNS provider went down, and users could not reach the services until the provider restored DNS.

   DNS resolution failuresService outage

   Trigger

   An outage at Octez Connect's DNS provider, metanet.ch, stopped the DNS records of its nodes from resolving.

   Root cause

   Dependency failure

   Component

   Authoritative DNS servers and records

   Contributing factor

   Single point of failureLimited debug access

   Detection

   The monitoring system flagged DNS resolution problems on the first node at 04:34 and raised a Pager Duty alert.

   Remediation

   • Escalated the DNS resolution alert through Pager Duty
   • Confirmed the provider's outage by repeatedly calling its hotline and watching its status page
   • Emailed the provider's support, which replied at 08:20 that it was looking into the issue
   • Tested DNS resolution and confirmed that DNS had been restarted
   • Posted a public update on Slack and Twitter announcing the recovery

   Action items

   • Configure a second DNS provider alongside the first
   • Build a public status page for real-time outage information
   • Review and improve internal escalation procedures for third-party outages

   Lessons

   Relying on a single DNS provider made it a single point of failure that brought down every connected service, and the provider's outage also slowed communication with it, which lengthened the downtime. Octez Connect concluded that it needs redundant DNS providers.

   View case
4. IL-0531 Azure Database for MySQL Single Server and MariaDB in West US rejected every new connection for a batch of servers

   Microsoft Azure2024

   Symptoms

   From 04:26 UTC on 19 April 2024, every new connection to a subset of Azure Database for MySQL - Single Server and Azure Database for MariaDB servers in West US failed, and retries did not help. The first report came at 04:46 UTC from Azure Databricks, which told the MySQL team that multiple databases were unreachable.

   Service outageIntermittent timeouts

   Trigger

   At 04:26 UTC, during planned maintenance in West US, the DNS update that moves new incoming connections for a batch of MySQL and MariaDB servers to a new gateway ring was made, and a procedural error pointed it at a newly built gateway ring whose configuration was incomplete.

   Root cause

   Operator error

   Component

   DatabaseAuthoritative DNS servers and records

   Contributing factor

   Monitoring blind spot

   Detection

   Service telemetry did not detect the failure because the new gateway ring had no telemetry or monitoring configured yet. At 04:46 UTC Azure Databricks reported to the MySQL team that multiple databases were unreachable.

   Remediation

   • Rolled back the DNS change and moved all connections back to the previous working gateway ring after validation

   Action items

   • Add checks, testing and sign-off procedures before switching production gateway rings
   • Add Azure Databricks alerts that detect multiple databases becoming unavailable within a short time

   View case
5. IL-0254 Square's internal DNS servers failed and Square products, services and payments were unavailable for about 15 hours

   Square2023

   Symptoms

   From 1:54 PM ET on 7 September 2023, Square products and services were unavailable, and at 2:02 PM ET engineers were notified of infrastructure issues and began investigating DNS. Square Online websites stayed up, but their customers could not make payments.

   DNS resolution failuresService outage

   Trigger

   At 11:04 AM ET a host-based firewall rule change was deployed to enable region communication, which increased the size of the firewall rules on nodes. Combined with a DNS service upgrade, it put unexpected load on the internal DNS servers. The timeline also lists a DNS zone change at 1:56 PM ET.

   Root cause

   Configuration change

   Component

   Authoritative DNS servers and recordsNode iptables rules

   Contributing factor

   No staged rolloutCache-masked failureCircular dependency

   Detection

   Engineers were notified of infrastructure issues at 2:02 PM ET, eight minutes after services became unavailable.

   Remediation

   • Shed network traffic to the DNS servers and started bringing up new DNS servers by hand
   • Deployed networking changes by hand to re-enable the authorization and access services
   • Deployed the networking changes by hand to all data centers
   • Rebuilt the DNS servers
   • Built a new configuration from the reverted firewall ruleset and pushed it to the DNS hosts

   Action items

   • Move the DNS infrastructure onto isolated infrastructure
   • Add monitoring and optimizations for critical networking infrastructure
   • Optimize dependencies between the deploy and platform infrastructure where feasible
   • Expand support for Offline Mode and improve communication about when it is available

   View case

By root cause5