Incident Library
Applied filters ComponentAuthoritative DNS servers and recordsSymptomService outageClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases11 cases

4 to 8 of 11

  1. IL-0260 zoom.us stopped resolving for about two hours after GoDaddy Registry placed a server block on the domain
    Zoom2025
    Symptoms

    On 16 April 2025 customers in the US and International regions could not reach Zoom services because lookups of the zoom.us domain failed. Starting, joining or scheduling a meeting needs a DNS lookup, so none of those actions completed; people already in a Zoom meeting or on a Zoom Phone call were not affected.

    DNS resolution failuresService outage
    Trigger

    A communication error between Zoom's domain registrar, Markmonitor, and GoDaddy Registry led GoDaddy Registry to place a server block on zoom.us by mistake, which shut the domain down.

    Root cause
    Dependency failure
    Component
    Authoritative DNS servers and records
    Contributing factor
    No safeguard on destructive change
    Remediation
    • Identified and removed the server block on zoom.us together with Markmonitor and GoDaddy
    • Asked users who still had connection problems to flush their DNS cache and reconnect
    Action items
    • Put a registry lock on zoom.us that restricts server block commands on the domain
    • Implement additional technical changes and processes with GoDaddy Registry and Markmonitor so the issue does not recur
    View case
  2. IL-0348 Scaleway's FR-PAR region lost public internet connectivity when one 100 Gbps backbone link was saturated
    Scaleway2025
    Symptoms

    On 10 March 2025 at 16:22 UTC, public internet connectivity in Scaleway's FR-PAR region broke for all products. Most internet traffic between the FR-PAR-2 availability zone and the rest of the PAR region was blackholed for about five minutes, while VPC traffic and Dedibox RPN kept working.

    Packet lossService outageHigh latency
    Trigger

    While Scaleway's network team connected a new data center to the FR-PAR backbone, a human mistake in a router configuration rerouted more than 1 Tbps of traffic between OpCore DC3 and DC5 through a single 100 Gbps link.

    Root cause
    Operator error
    Component
    Backbone network (WAN)Object storageAuthoritative DNS servers and records
    Contributing factor
    Automation amplification
    Detection

    The router configuration error was detected quickly and rolled back within four minutes.

    Remediation
    • Rolled back the router configuration, restoring the correct configuration at 16:27 UTC
    Action items
    • Harden the peer review and production communication process for changes
    • Add failsafes and tests before network configuration is applied at backbone level
    • Deploy the fix for the Object Storage synchronisation software issue
    View case
  3. IL-0269 Octez Connect services went down for hours when its only DNS provider failed and the nodes' records stopped resolving
    Octez Connect2025
    Symptoms

    On 5 February 2025 at 04:34 the monitoring system flagged DNS resolution problems on the first Octez Connect node and raised a Pager Duty alert. Every server that relied on the same DNS provider went down, and users could not reach the services until the provider restored DNS.

    DNS resolution failuresService outage
    Trigger

    An outage at Octez Connect's DNS provider, metanet.ch, stopped the DNS records of its nodes from resolving.

    Root cause
    Dependency failure
    Component
    Authoritative DNS servers and records
    Contributing factor
    Single point of failureLimited debug access
    Detection

    The monitoring system flagged DNS resolution problems on the first node at 04:34 and raised a Pager Duty alert.

    Remediation
    • Escalated the DNS resolution alert through Pager Duty
    • Confirmed the provider's outage by repeatedly calling its hotline and watching its status page
    • Emailed the provider's support, which replied at 08:20 that it was looking into the issue
    • Tested DNS resolution and confirmed that DNS had been restarted
    • Posted a public update on Slack and Twitter announcing the recovery
    Action items
    • Configure a second DNS provider alongside the first
    • Build a public status page for real-time outage information
    • Review and improve internal escalation procedures for third-party outages
    Lessons

    Relying on a single DNS provider made it a single point of failure that brought down every connected service, and the provider's outage also slowed communication with it, which lengthened the downtime. Octez Connect concluded that it needs redundant DNS providers.

    View case
  4. IL-0531 Azure Database for MySQL Single Server and MariaDB in West US rejected every new connection for a batch of servers
    Microsoft Azure2024
    Symptoms

    From 04:26 UTC on 19 April 2024, every new connection to a subset of Azure Database for MySQL - Single Server and Azure Database for MariaDB servers in West US failed, and retries did not help. The first report came at 04:46 UTC from Azure Databricks, which told the MySQL team that multiple databases were unreachable.

    Service outageIntermittent timeouts
    Trigger

    At 04:26 UTC, during planned maintenance in West US, the DNS update that moves new incoming connections for a batch of MySQL and MariaDB servers to a new gateway ring was made, and a procedural error pointed it at a newly built gateway ring whose configuration was incomplete.

    Root cause
    Operator error
    Component
    DatabaseAuthoritative DNS servers and records
    Contributing factor
    Monitoring blind spot
    Detection

    Service telemetry did not detect the failure because the new gateway ring had no telemetry or monitoring configured yet. At 04:46 UTC Azure Databricks reported to the MySQL team that multiple databases were unreachable.

    Remediation
    • Rolled back the DNS change and moved all connections back to the previous working gateway ring after validation
    Action items
    • Add checks, testing and sign-off procedures before switching production gateway rings
    • Add Azure Databricks alerts that detect multiple databases becoming unavailable within a short time
    View case
  5. IL-0254 Square's internal DNS servers failed and Square products, services and payments were unavailable for about 15 hours
    Square2023
    Symptoms

    From 1:54 PM ET on 7 September 2023, Square products and services were unavailable, and at 2:02 PM ET engineers were notified of infrastructure issues and began investigating DNS. Square Online websites stayed up, but their customers could not make payments.

    DNS resolution failuresService outage
    Trigger

    At 11:04 AM ET a host-based firewall rule change was deployed to enable region communication, which increased the size of the firewall rules on nodes. Combined with a DNS service upgrade, it put unexpected load on the internal DNS servers. The timeline also lists a DNS zone change at 1:56 PM ET.

    Root cause
    Configuration change
    Component
    Authoritative DNS servers and recordsNode iptables rules
    Contributing factor
    No staged rolloutCache-masked failureCircular dependency
    Detection

    Engineers were notified of infrastructure issues at 2:02 PM ET, eight minutes after services became unavailable.

    Remediation
    • Shed network traffic to the DNS servers and started bringing up new DNS servers by hand
    • Deployed networking changes by hand to re-enable the authorization and access services
    • Deployed the networking changes by hand to all data centers
    • Rebuilt the DNS servers
    • Built a new configuration from the reverted firewall ruleset and pushed it to the DNS hosts
    Action items
    • Move the DNS infrastructure onto isolated infrastructure
    • Add monitoring and optimizations for critical networking infrastructure
    • Optimize dependencies between the deploy and platform infrastructure where feasible
    • Expand support for Offline Mode and improve communication about when it is available
    View case
Check your cluster for this

These 11 cases involve Authoritative DNS servers and records and CDN and edge network. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause5

Configuration change3 cases

Component
Authoritative DNS servers and records3CDN and edge network1cert-manager1Node iptables rules1
Contributing factor
Cache-masked failure2Unverified assumption1Circular dependency1Unreviewed defaults1Change with a delayed effect1Monitoring blind spot1No configuration tests1No staged rollout1
Organisation
2i2c1MIT Open Learning1Square1
  1. IL-0512 xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failed
    MIT Open Learning2026
  2. IL-0502 The JupyterHub at temple.2i2c.cloud stopped resolving in DNS after a wildcard record was added for a new cluster
    2i2c2025
  3. IL-0254 Square's internal DNS servers failed and Square products, services and payments were unavailable for about 15 hours
    Square2023

Software defect1 case

Component
CDN and edge networkAuthoritative DNS servers and records
Contributing factor
Excessive blast radiusChange with a delayed effectIneffective rollbackGap in automated tests
Organisation
Microsoft Azure
  1. IL-0226 Azure Front Door edge servers crashed worldwide on incompatible configuration metadata, causing timeouts and DNS errors
    Microsoft Azure2025

Dependency failure2 cases

Component
Authoritative DNS servers and records2
Contributing factor
Limited debug access1No safeguard on destructive change1Single point of failure1
Organisation
Octez Connect1Zoom1
  1. IL-0260 zoom.us stopped resolving for about two hours after GoDaddy Registry placed a server block on the domain
    Zoom2025
  2. IL-0269 Octez Connect services went down for hours when its only DNS provider failed and the nodes' records stopped resolving
    Octez Connect2025

Operator error3 cases

Component
Authoritative DNS servers and records3Backbone network (WAN)2BGP routing1Database1Object storage1
Contributing factor
Automation amplification2Circular dependency1Limited debug access1Monitoring blind spot1
Organisation
Meta1Microsoft Azure1Scaleway1
  1. IL-0348 Scaleway's FR-PAR region lost public internet connectivity when one 100 Gbps backbone link was saturated
    Scaleway2025
  2. IL-0531 Azure Database for MySQL Single Server and MariaDB in West US rejected every new connection for a batch of servers
    Microsoft Azure2024
  3. IL-0221 A backbone maintenance command disconnected all Facebook data centers, and its DNS servers withdrew their BGP routes
    Meta2021

Faulty deployment2 cases

Component
Authoritative DNS servers and records2In-memory cache1CDN and edge network1Database failover manager1Object storage1
Contributing factor
Automation amplification1Circular dependency1Harmful manual action1Overlapping failures1Ineffective rollback1Single point of failure1
Organisation
bunny.net1Redis1
  1. IL-0229 bunny.net's DNS servers crashed in a loop on a corrupted routing database file and could not fetch a fix from its CDN
    bunny.net2021
  2. IL-0278 A weekly update put a Redis Cloud management process into an infinite loop, and most nodes of a GCP cluster failed
    Redis2016