IL-0360 bunny.net · 2021-05-26networking

DNS queries to bunny.net timed out after a failed routing update took two of four nameserver clusters offline

During routine maintenance on 26 May 2021, bunny.net deployed a routing update to its DNS system. Because of an issue with a different system the update did not deploy correctly, and 2 of the 4 nameserver clusters became unavailable. The remaining two clusters received more load than expected at the reduced capacity and began to time out queries; resolvers then retried with even more queries, which gradually overloaded one of the nameserver clusters. About 10% of traffic was affected for about 30 minutes, the status page marked the issue resolved at 21:23 UTC, and bunny.net gave the cause in an update on 27 May, calling it a human error. bunny.net then added capacity to all clusters so that each one can carry the full load.

Severity: –
Scope: DNS queries to bunny.net nameservers, about 10% of traffic
Users affected: –
Region: –
Time to detect: –
Time to resolve: 43m

Symptoms

On 26 May 2021, after routine maintenance at about 20:40 UTC, bunny.net saw an abnormal level of load on one of its DNS clusters, NS cluster 1, and DNS queries started to time out. About 10% of traffic was affected for roughly 30 minutes, although bunny.net runs three independent anycast networks. The timeouts made resolvers send even more queries, and that extra load gradually overloaded one of the nameserver clusters.

Symptom DNS resolution failures

Root cause

Root cause Faulty deployment

Component Authoritative DNS servers and records

Contributing factor Retry storm No headroom

Trigger: During the maintenance, bunny.net deployed a routing update to its DNS system meant to improve performance and reliability, and an issue with a different system made the update deploy incorrectly.

Detection

bunny.net detected an abnormal level of load on one of its DNS clusters after the maintenance.

Remediation

–

Action items

Add quality control and monitoring steps so a failed DNS routing update does not happen again
Install additional capacity in all nameserver clusters so that each one can handle the full load
Work with the team so the human error behind the incident does not happen again

Lessons

–

Sources

https://status.bunny.net/incidents/4p37r98cl4vc · official-status · archived copy

Related cases

IL-0229 bunny.net's DNS servers crashed in a loop on a corrupted routing database file and could not fetch a fix from its CDNbunny.net · 2021
same root cause · shared Authoritative DNS servers and records
IL-0278 A weekly update put a Redis Cloud management process into an infinite loop, and most nodes of a GCP cluster failedRedis · 2016
same root cause · shared Authoritative DNS servers and records

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Authoritative DNS servers and records among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0360
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0360/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.