Incident Library
Applied filters ComponentAuthoritative DNS servers and recordsSymptomService outageClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases11 cases

10 to 11 of 11

  1. IL-0229 bunny.net's DNS servers crashed in a loop on a corrupted routing database file and could not fetch a fix from its CDN
    bunny.net2021
    Symptoms

    On 22 June 2021, within minutes of an update released at 8:25 AM UTC, bunny.net's global fleet of close to 100 DNS servers kept crashing and restarting, more than 60% of traffic was lost, hundreds of Gbits of throughput disappeared, and over 750,000 websites were affected. After 10 minutes the team realised that the DNS servers were restarting and dying and that nothing could bring them back up in that state.

    Container crash loopDNS resolution failuresService outageMisrouted requestsLoad imbalance
    Trigger

    At 8:25 AM UTC bunny.net released an update meant to reduce the download size of the SmartEdge optimization database, and it uploaded a corrupted file to Edge Storage.

    Root cause
    Faulty deployment
    Component
    Authoritative DNS servers and recordsCDN and edge networkObject storage
    Contributing factor
    Circular dependencySingle point of failureAutomation amplificationIneffective rollbackHarmful manual action
    Detection

    About 10 minutes after the release, the team realised that the DNS servers were restarting and dying.

    Remediation
    • Deployed by hand an update that disabled the SmartEdge system on the DNS nodes
    • Routed all requests through Frankfurt instead of Madrid on a couple of DNS servers
    • Migrated the deployment systems and files to a third-party cloud storage service
    • Rewired the deployment system and DNS software to the new storage and deployed
    • Synced the databases and deployed the newest file sets through the third-party service
    Action items
    • Phase out the BinaryPack library as a hot-fix
    • Test third-party libraries more extensively
    • Migrate the internal APIs to an independent third-party service
    • Find how to stop non-critical software from becoming a single point of failure across clusters
    • Make the DNS system keep a local copy of all backup data with automatic failure detection
    Lessons

    bunny.net had built its systems to rely on each other, including the critical parts of its internal infrastructure, so one corrupted file brought down several layers of redundancy with no way back. A part of the infrastructure treated as non-critical turned out to be a single point of failure for several clusters at once, which canary deployments did not catch. Building infrastructure inside its own ecosystem lets one failure cascade through all of it. In the rush to fix the outage the team made typos and mistakes, and the final deployment came after it calmed down and double-checked everything.

    View case
  2. IL-0278 A weekly update put a Redis Cloud management process into an infinite loop, and most nodes of a GCP cluster failed
    Redis2016
    Symptoms

    On 8 February 2016 at 17:45 UTC, four nodes of the Redis Cloud cluster us-central1-1-1.gce.garantiadata.com on Google Cloud Platform stopped responding, four minutes after a first node had failed. The databases whose endpoints end in us-central1-1-1.gce.garantiadata.com went down.

    Service outageFailover failure
    Trigger

    The weekly upgrade of Redis Cloud's non-critical cluster components reached its first Google Cloud Platform cluster, us-central1-1-1.gce.garantiadata.com, at 17:30 UTC.

    Root cause
    Faulty deployment
    Component
    In-memory cacheDatabase failover managerAuthoritative DNS servers and records
    Contributing factor
    Overlapping failures
    Remediation
    • Shut down and terminated all nodes of the cluster
    • Deleted all of the cluster's endpoints from its DNS zone, removing by hand the entries left by throttled deletes
    • Provisioned new cluster nodes from scratch without the weekly update
    • Reconnected the original nodes' storage to the new nodes so persistence-enabled databases could be recovered
    • Bootstrapped and launched the databases, recovering the ones with persistence from storage
    Action items
    • Suspend all automatic weekly upgrades until a procedure that prevents this kind of outage is in place
    • Revalidate the QA processes for DevOps code deployments
    • Improve the tooling so it handles throttling of DNS requests
    View case
Check your cluster for this

These 11 cases involve Authoritative DNS servers and records and CDN and edge network. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause5

Configuration change3 cases

Component
Authoritative DNS servers and records3CDN and edge network1cert-manager1Node iptables rules1
Contributing factor
Cache-masked failure2Unverified assumption1Circular dependency1Unreviewed defaults1Change with a delayed effect1Monitoring blind spot1No configuration tests1No staged rollout1
Organisation
2i2c1MIT Open Learning1Square1
  1. IL-0512 xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failed
    MIT Open Learning2026
  2. IL-0502 The JupyterHub at temple.2i2c.cloud stopped resolving in DNS after a wildcard record was added for a new cluster
    2i2c2025
  3. IL-0254 Square's internal DNS servers failed and Square products, services and payments were unavailable for about 15 hours
    Square2023

Software defect1 case

Component
CDN and edge networkAuthoritative DNS servers and records
Contributing factor
Excessive blast radiusChange with a delayed effectIneffective rollbackGap in automated tests
Organisation
Microsoft Azure
  1. IL-0226 Azure Front Door edge servers crashed worldwide on incompatible configuration metadata, causing timeouts and DNS errors
    Microsoft Azure2025

Dependency failure2 cases

Component
Authoritative DNS servers and records2
Contributing factor
Limited debug access1No safeguard on destructive change1Single point of failure1
Organisation
Octez Connect1Zoom1
  1. IL-0260 zoom.us stopped resolving for about two hours after GoDaddy Registry placed a server block on the domain
    Zoom2025
  2. IL-0269 Octez Connect services went down for hours when its only DNS provider failed and the nodes' records stopped resolving
    Octez Connect2025

Operator error3 cases

Component
Authoritative DNS servers and records3Backbone network (WAN)2BGP routing1Database1Object storage1
Contributing factor
Automation amplification2Circular dependency1Limited debug access1Monitoring blind spot1
Organisation
Meta1Microsoft Azure1Scaleway1
  1. IL-0348 Scaleway's FR-PAR region lost public internet connectivity when one 100 Gbps backbone link was saturated
    Scaleway2025
  2. IL-0531 Azure Database for MySQL Single Server and MariaDB in West US rejected every new connection for a batch of servers
    Microsoft Azure2024
  3. IL-0221 A backbone maintenance command disconnected all Facebook data centers, and its DNS servers withdrew their BGP routes
    Meta2021

Faulty deployment2 cases

Component
Authoritative DNS servers and records2In-memory cache1CDN and edge network1Database failover manager1Object storage1
Contributing factor
Automation amplification1Circular dependency1Harmful manual action1Overlapping failures1Ineffective rollback1Single point of failure1
Organisation
bunny.net1Redis1
  1. IL-0229 bunny.net's DNS servers crashed in a loop on a corrupted routing database file and could not fetch a fix from its CDN
    bunny.net2021
  2. IL-0278 A weekly update put a Redis Cloud management process into an infinite loop, and most nodes of a GCP cluster failed
    Redis2016