Cases3 cases

1 to 3 of 3

1. IL-0229 bunny.net's DNS servers crashed in a loop on a corrupted routing database file and could not fetch a fix from its CDN

   bunny.net2021

   Symptoms

   On 22 June 2021, within minutes of an update released at 8:25 AM UTC, bunny.net's global fleet of close to 100 DNS servers kept crashing and restarting, more than 60% of traffic was lost, hundreds of Gbits of throughput disappeared, and over 750,000 websites were affected. After 10 minutes the team realised that the DNS servers were restarting and dying and that nothing could bring them back up in that state.

   Container crash loopDNS resolution failuresService outageMisrouted requestsLoad imbalance

   Trigger

   At 8:25 AM UTC bunny.net released an update meant to reduce the download size of the SmartEdge optimization database, and it uploaded a corrupted file to Edge Storage.

   Root cause

   Faulty deployment

   Component

   Authoritative DNS servers and recordsCDN and edge networkObject storage

   Contributing factor

   Circular dependencySingle point of failureAutomation amplificationIneffective rollbackHarmful manual action

   Detection

   About 10 minutes after the release, the team realised that the DNS servers were restarting and dying.

   Remediation

   • Deployed by hand an update that disabled the SmartEdge system on the DNS nodes
   • Routed all requests through Frankfurt instead of Madrid on a couple of DNS servers
   • Migrated the deployment systems and files to a third-party cloud storage service
   • Rewired the deployment system and DNS software to the new storage and deployed
   • Synced the databases and deployed the newest file sets through the third-party service

   Action items

   • Phase out the BinaryPack library as a hot-fix
   • Test third-party libraries more extensively
   • Migrate the internal APIs to an independent third-party service
   • Find how to stop non-critical software from becoming a single point of failure across clusters
   • Make the DNS system keep a local copy of all backup data with automatic failure detection

   Lessons

   bunny.net had built its systems to rely on each other, including the critical parts of its internal infrastructure, so one corrupted file brought down several layers of redundancy with no way back. A part of the infrastructure treated as non-critical turned out to be a single point of failure for several clusters at once, which canary deployments did not catch. Building infrastructure inside its own ecosystem lets one failure cascade through all of it. In the rush to fix the outage the team made typos and mistakes, and the final deployment came after it calmed down and double-checked everything.

   View case
2. IL-0360 DNS queries to bunny.net timed out after a failed routing update took two of four nameserver clusters offline

   bunny.net2021

   Symptoms

   On 26 May 2021, after routine maintenance at about 20:40 UTC, bunny.net saw an abnormal level of load on one of its DNS clusters, NS cluster 1, and DNS queries started to time out. About 10% of traffic was affected for roughly 30 minutes, although bunny.net runs three independent anycast networks. The timeouts made resolvers send even more queries, and that extra load gradually overloaded one of the nameserver clusters.

   DNS resolution failures

   Trigger

   During the maintenance, bunny.net deployed a routing update to its DNS system meant to improve performance and reliability, and an issue with a different system made the update deploy incorrectly.

   Root cause

   Faulty deployment

   Component

   Authoritative DNS servers and records

   Contributing factor

   Retry stormNo headroom

   Detection

   bunny.net detected an abnormal level of load on one of its DNS clusters after the maintenance.

   Action items

   • Add quality control and monitoring steps so a failed DNS routing update does not happen again
   • Install additional capacity in all nameserver clusters so that each one can handle the full load
   • Work with the team so the human error behind the incident does not happen again

   View case
3. IL-0278 A weekly update put a Redis Cloud management process into an infinite loop, and most nodes of a GCP cluster failed

   Redis2016

   Symptoms

   On 8 February 2016 at 17:45 UTC, four nodes of the Redis Cloud cluster us-central1-1-1.gce.garantiadata.com on Google Cloud Platform stopped responding, four minutes after a first node had failed. The databases whose endpoints end in us-central1-1-1.gce.garantiadata.com went down.

   Service outageFailover failure

   Trigger

   The weekly upgrade of Redis Cloud's non-critical cluster components reached its first Google Cloud Platform cluster, us-central1-1-1.gce.garantiadata.com, at 17:30 UTC.

   Root cause

   Faulty deployment

   Component

   In-memory cacheDatabase failover managerAuthoritative DNS servers and records

   Contributing factor

   Overlapping failures

   Remediation

   • Shut down and terminated all nodes of the cluster
   • Deleted all of the cluster's endpoints from its DNS zone, removing by hand the entries left by throttled deletes
   • Provisioned new cluster nodes from scratch without the weekly update
   • Reconnected the original nodes' storage to the new nodes so persistence-enabled databases could be recovered
   • Bootstrapped and launched the databases, recovering the ones with persistence from storage

   Action items

   • Suspend all automatic weekly upgrades until a procedure that prevents this kind of outage is in place
   • Revalidate the QA processes for DevOps code deployments
   • Improve the tooling so it handles throttling of DNS requests

   View case

By root cause1