Incident Library
Applied filters FactorRetry stormOrganisationbunny.netClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0360 DNS queries to bunny.net timed out after a failed routing update took two of four nameserver clusters offline
    bunny.net2021
    Symptoms

    On 26 May 2021, after routine maintenance at about 20:40 UTC, bunny.net saw an abnormal level of load on one of its DNS clusters, NS cluster 1, and DNS queries started to time out. About 10% of traffic was affected for roughly 30 minutes, although bunny.net runs three independent anycast networks. The timeouts made resolvers send even more queries, and that extra load gradually overloaded one of the nameserver clusters.

    DNS resolution failures
    Trigger

    During the maintenance, bunny.net deployed a routing update to its DNS system meant to improve performance and reliability, and an issue with a different system made the update deploy incorrectly.

    Root cause
    Faulty deployment
    Component
    Authoritative DNS servers and records
    Contributing factor
    Retry stormNo headroom
    Detection

    bunny.net detected an abnormal level of load on one of its DNS clusters after the maintenance.

    Action items
    • Add quality control and monitoring steps so a failed DNS routing update does not happen again
    • Install additional capacity in all nameserver clusters so that each one can handle the full load
    • Work with the team so the human error behind the incident does not happen again
    View case
Check your cluster for this

This case involves Authoritative DNS servers and records. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment1 case

Component
Authoritative DNS servers and records
Contributing factor
No headroomRetry storm
Organisation
bunny.net
  1. IL-0360 DNS queries to bunny.net timed out after a failed routing update took two of four nameserver clusters offline
    bunny.net2021