Incident Library

IL-0229 bunny.net · 2021-06-22networking

bunny.net's DNS servers crashed in a loop on a corrupted routing database file and could not fetch a fix from its CDN

SmartEdge, bunny.net's routing system, periodically syncs large database files to the DNS nodes, using the Edge Storage platform to distribute them around the world through Bunny CDN. A few weeks earlier SmartEdge had switched from JSON to the binary serialization library BinaryPack to reduce memory, traffic and garbage collector allocations. The corrupted file crashed the DNS process through a stack overflow in BinaryPack. Because the deployment systems and SmartEdge both depended on the CDN that the DNS failure had taken down, the rollback and later fixes could not reach the DNS servers, and auto-healing restarts kept loading the broken data. A manual update at 8:45 that disabled SmartEdge on the DNS nodes made the servers download corrupted GeoDNS databases, so all requests went to the small Madrid PoP, and 100 servers restarting in a loop crashed the central API.

At around 9:40 the team prepared a routing update that sent all requests to Frankfurt, the biggest region, instead of Madrid, and deployed it by hand to a couple of DNS servers, which brought back a portion of the traffic. The team then moved all deployment systems and files to a third-party cloud storage service, and at 10:15 rewired the deployment system and DNS software to use it and deployed. Traffic was back by 10:30, but in the rush an incorrect GeoDNS database version had been deployed and requests still went to Madrid. At 10:45 the team synced the databases and deployed the newest file sets through the third-party service. With SmartEdge off, storage was pushed to its limits serving uncached data, and the platform stabilised at 11:00 in recovery mode. The outage lasted more than two hours and affected over 750,000 websites.

Severity
–
Scope
bunny.net DNS, CDN, storage, optimizer service, API, dashboard and logging; over 60% of traffic lost
Users affected
over 750,000 websites
Region
global
Time to detect
10m
Time to resolve
2h 35m

Symptoms

On 22 June 2021, within minutes of an update released at 8:25 AM UTC, bunny.net's global fleet of close to 100 DNS servers kept crashing and restarting, more than 60% of traffic was lost, hundreds of Gbits of throughput disappeared, and over 750,000 websites were affected. After 10 minutes the team realised that the DNS servers were restarting and dying and that nothing could bring them back up in that state.

The DNS software was designed to work with or without the SmartEdge data and to ignore exceptions, but the corrupted file made the BinaryPack serialization library fail with a stack overflow exception that bypassed all exception handling and ended the process. Each time auto-healing restarted a DNS server, it loaded the broken deployment and crashed again. At 8:35 a few servers were still struggling to answer, and traffic had dropped to 100Gbit. It was the lowest traffic point of the day.

Rolling back all SmartEdge updates right away did not help, because SmartEdge and the deployment systems deliver data to the DNS servers through Edge Storage and Bunny CDN, and the DNS failure had taken most of the CDN capacity down. After a manual update at 8:45 disabled SmartEdge, the DNS servers downloaded corrupted GeoDNS databases through the failing CDN and sent all requests to Madrid, one of the smallest PoPs, which was quickly overwhelmed.

With 100 servers restarting in a loop, the central API began to crash, and servers that were brought back failed to start properly. After several attempts to re-establish the networking, the team gave up on that idea. The ripple took down the CDN, the storage, the optimizer service, the API and the dashboard, and finally the logging service.

Symptom Container crash loopDNS resolution failuresService outageMisrouted requestsLoad imbalance

Root cause

Root cause Faulty deployment

Component Authoritative DNS servers and recordsCDN and edge networkObject storage
Contributing factor Circular dependencySingle point of failureAutomation amplificationIneffective rollbackHarmful manual action
Trigger
At 8:25 AM UTC bunny.net released an update meant to reduce the download size of the SmartEdge optimization database, and it uploaded a corrupted file to Edge Storage.

Detection

About 10 minutes after the release, the team realised that the DNS servers were restarting and dying.

Remediation

Action items

Lessons

bunny.net had built its systems to rely on each other, including the critical parts of its internal infrastructure, so one corrupted file brought down several layers of redundancy with no way back. A part of the infrastructure treated as non-critical turned out to be a single point of failure for several clusters at once, which canary deployments did not catch. Building infrastructure inside its own ecosystem lets one failure cascade through all of it. In the rush to fix the outage the team made typos and mistakes, and the final deployment came after it calmed down and double-checked everything.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Authoritative DNS servers and records, CDN and edge network, Object storage among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.