Incident Library
Applied filters ComponentAuthoritative DNS servers and recordsFactorSingle point of failureClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0269 Octez Connect services went down for hours when its only DNS provider failed and the nodes' records stopped resolving
    Octez Connect2025
    Symptoms

    On 5 February 2025 at 04:34 the monitoring system flagged DNS resolution problems on the first Octez Connect node and raised a Pager Duty alert. Every server that relied on the same DNS provider went down, and users could not reach the services until the provider restored DNS.

    DNS resolution failuresService outage
    Trigger

    An outage at Octez Connect's DNS provider, metanet.ch, stopped the DNS records of its nodes from resolving.

    Root cause
    Dependency failure
    Component
    Authoritative DNS servers and records
    Contributing factor
    Single point of failureLimited debug access
    Detection

    The monitoring system flagged DNS resolution problems on the first node at 04:34 and raised a Pager Duty alert.

    Remediation
    • Escalated the DNS resolution alert through Pager Duty
    • Confirmed the provider's outage by repeatedly calling its hotline and watching its status page
    • Emailed the provider's support, which replied at 08:20 that it was looking into the issue
    • Tested DNS resolution and confirmed that DNS had been restarted
    • Posted a public update on Slack and Twitter announcing the recovery
    Action items
    • Configure a second DNS provider alongside the first
    • Build a public status page for real-time outage information
    • Review and improve internal escalation procedures for third-party outages
    Lessons

    Relying on a single DNS provider made it a single point of failure that brought down every connected service, and the provider's outage also slowed communication with it, which lengthened the downtime. Octez Connect concluded that it needs redundant DNS providers.

    View case
  2. IL-0229 bunny.net's DNS servers crashed in a loop on a corrupted routing database file and could not fetch a fix from its CDN
    bunny.net2021
    Symptoms

    On 22 June 2021, within minutes of an update released at 8:25 AM UTC, bunny.net's global fleet of close to 100 DNS servers kept crashing and restarting, more than 60% of traffic was lost, hundreds of Gbits of throughput disappeared, and over 750,000 websites were affected. After 10 minutes the team realised that the DNS servers were restarting and dying and that nothing could bring them back up in that state.

    Container crash loopDNS resolution failuresService outageMisrouted requestsLoad imbalance
    Trigger

    At 8:25 AM UTC bunny.net released an update meant to reduce the download size of the SmartEdge optimization database, and it uploaded a corrupted file to Edge Storage.

    Root cause
    Faulty deployment
    Component
    Authoritative DNS servers and recordsCDN and edge networkObject storage
    Contributing factor
    Circular dependencySingle point of failureAutomation amplificationIneffective rollbackHarmful manual action
    Detection

    About 10 minutes after the release, the team realised that the DNS servers were restarting and dying.

    Remediation
    • Deployed by hand an update that disabled the SmartEdge system on the DNS nodes
    • Routed all requests through Frankfurt instead of Madrid on a couple of DNS servers
    • Migrated the deployment systems and files to a third-party cloud storage service
    • Rewired the deployment system and DNS software to the new storage and deployed
    • Synced the databases and deployed the newest file sets through the third-party service
    Action items
    • Phase out the BinaryPack library as a hot-fix
    • Test third-party libraries more extensively
    • Migrate the internal APIs to an independent third-party service
    • Find how to stop non-critical software from becoming a single point of failure across clusters
    • Make the DNS system keep a local copy of all backup data with automatic failure detection
    Lessons

    bunny.net had built its systems to rely on each other, including the critical parts of its internal infrastructure, so one corrupted file brought down several layers of redundancy with no way back. A part of the infrastructure treated as non-critical turned out to be a single point of failure for several clusters at once, which canary deployments did not catch. Building infrastructure inside its own ecosystem lets one failure cascade through all of it. In the rush to fix the outage the team made typos and mistakes, and the final deployment came after it calmed down and double-checked everything.

    View case
  3. IL-0359 Queries to Gandi LiveDNS timed out when two crashed anycast DNS nodes kept announcing all three name server addresses
    Gandi2020
    Symptoms

    On 16 July 2020 between 13:26 and 13:48 UTC, DNS queries to two nodes of Gandi's LiveDNS platform failed with a timeout, and retries against the other name servers of a domain failed as well. The status page reported impact for FR customers.

    DNS resolution failures
    Trigger

    A software bug in Gandi's DNS server software triggered a fault and stopped the server on two LiveDNS nodes.

    Root cause
    Software defect
    Component
    Authoritative DNS servers and recordsBGP routing
    Contributing factor
    Single point of failureSignal buried in noiseOverlapping failures
    Detection

    Monitoring alerts fired on the DNS servers, but they were first read as false positives among the noise from an earlier internal network incident.

    Action items
    • Fix the way a node is disabled when its DNS server is not working correctly
    • Fix the node configuration and add monitoring so that no node announces several clusters
    • Rework monitoring and internal training and organisation
    • Deploy the fix for the bug in the DNS server software
    View case
Check your cluster for this

These 3 cases involve Authoritative DNS servers and records and BGP routing. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause3

Dependency failure1 case

Component
Authoritative DNS servers and records
Contributing factor
Limited debug accessSingle point of failure
Organisation
Octez Connect
  1. IL-0269 Octez Connect services went down for hours when its only DNS provider failed and the nodes' records stopped resolving
    Octez Connect2025

Faulty deployment1 case

Component
CDN and edge networkAuthoritative DNS servers and recordsObject storage
Contributing factor
Automation amplificationCircular dependencyHarmful manual actionIneffective rollbackSingle point of failure
Organisation
bunny.net
  1. IL-0229 bunny.net's DNS servers crashed in a loop on a corrupted routing database file and could not fetch a fix from its CDN
    bunny.net2021

Software defect1 case

Component
BGP routingAuthoritative DNS servers and records
Contributing factor
Signal buried in noiseOverlapping failuresSingle point of failure
Organisation
Gandi
  1. IL-0359 Queries to Gandi LiveDNS timed out when two crashed anycast DNS nodes kept announcing all three name server addresses
    Gandi2020