Cases2 cases

1 to 2 of 2

1. IL-0269 Octez Connect services went down for hours when its only DNS provider failed and the nodes' records stopped resolving

   Octez Connect2025

   Symptoms

   On 5 February 2025 at 04:34 the monitoring system flagged DNS resolution problems on the first Octez Connect node and raised a Pager Duty alert. Every server that relied on the same DNS provider went down, and users could not reach the services until the provider restored DNS.

   DNS resolution failuresService outage

   Trigger

   An outage at Octez Connect's DNS provider, metanet.ch, stopped the DNS records of its nodes from resolving.

   Root cause

   Dependency failure

   Component

   Authoritative DNS servers and records

   Contributing factor

   Single point of failureLimited debug access

   Detection

   The monitoring system flagged DNS resolution problems on the first node at 04:34 and raised a Pager Duty alert.

   Remediation

   • Escalated the DNS resolution alert through Pager Duty
   • Confirmed the provider's outage by repeatedly calling its hotline and watching its status page
   • Emailed the provider's support, which replied at 08:20 that it was looking into the issue
   • Tested DNS resolution and confirmed that DNS had been restarted
   • Posted a public update on Slack and Twitter announcing the recovery

   Action items

   • Configure a second DNS provider alongside the first
   • Build a public status page for real-time outage information
   • Review and improve internal escalation procedures for third-party outages

   Lessons

   Relying on a single DNS provider made it a single point of failure that brought down every connected service, and the provider's outage also slowed communication with it, which lengthened the downtime. Octez Connect concluded that it needs redundant DNS providers.

   View case
2. IL-0221 A backbone maintenance command disconnected all Facebook data centers, and its DNS servers withdrew their BGP routes

   Meta2021

   Symptoms

   On 4 October 2021 Facebook's platforms went down worldwide: every connection in Facebook's global backbone network was lost, and the data centers were cut off from each other and from the internet. The rest of the internet could no longer find Facebook's servers, because Facebook's authoritative DNS servers had become unreachable.

   Service outageDNS resolution failuresNetwork unreachable

   Trigger

   During routine maintenance of the backbone, a command was issued to assess the availability of global backbone capacity, and it unintentionally took down all connections in the backbone network.

   Root cause

   Operator error

   Component

   Backbone network (WAN)BGP routingAuthoritative DNS servers and records

   Contributing factor

   Automation amplificationCircular dependencyLimited debug access

   Remediation

   • Sent engineers onsite to the data centers to debug the issue and restart systems
   • Brought the backbone network back online across the data center regions
   • Brought services back gradually, managing the rising load to avoid crashes from a traffic surge

   Action items

   • Find ways to simulate the global backbone being taken offline in storm drills
   • Strengthen testing, drills and overall resilience

   Lessons

   The hardening that protects Facebook's data centers against unauthorized access slowed recovery from an outage caused by Facebook's own error, and the author judges that tradeoff worth keeping. Storm drills prepared the teams to bring load back safely, although no drill had yet simulated the global backbone going offline.

   View case

By root cause2