IL-0269 Octez Connect · 2025-02-05networking

Octez Connect services went down for hours when its only DNS provider failed and the nodes' records stopped resolving

Octez Connect relied on a single DNS provider for all of its servers. At 04:34 on 5 February 2025 the DNS records for the first node stopped resolving, a Pager Duty alert went out, and an engineer started investigating at 04:40. By 04:52 the team had confirmed that the provider was having an outage, and because there was no fallback provider the failure spread across all nodes. The provider's hotline, status page and ticket system were all down or unresponsive, and an email to its support address got no autoresponse, which left the team with few channels to get the problem resolved; it kept calling the hotline and watching the provider's status page. The provider replied at 08:20, DNS had been restarted by the time tests at 08:30 showed resolution working on all nodes, and recovery was announced at 08:40.

Severity: –
Scope: all servers relying on the DNS provider; users could not access the services
Users affected: users of the Octez Connect services
Region: –
Time to detect: 0m
Time to resolve: 3h 56m

Symptoms

On 5 February 2025 at 04:34 the monitoring system flagged DNS resolution problems on the first Octez Connect node and raised a Pager Duty alert. Every server that relied on the same DNS provider went down, and users could not reach the services until the provider restored DNS.

Diagnostics at 04:50 pointed to DNS, and at 04:52 the team confirmed that the DNS provider itself was having an outage. The provider's own channels were failing too: the hotline did not answer at 04:55, its status page was down at 05:00, its ticket system was unresponsive at 05:05, and an email to [email protected] at 05:10 got no autoresponse. From 05:20 the team explored other ways to resolve the problem and kept calling the hotline.

The provider answered by email at 08:20 that it was looking into the issue. At 08:30 tests showed DNS resolution working again on all nodes, and at 08:40 the team announced the recovery on Slack and Twitter.

Symptom DNS resolution failures Service outage

Root cause

Root cause Dependency failure

Component Authoritative DNS servers and records

Contributing factor Single point of failure Limited debug access

Trigger: An outage at Octez Connect's DNS provider, metanet.ch, stopped the DNS records of its nodes from resolving.

Detection

The monitoring system flagged DNS resolution problems on the first node at 04:34 and raised a Pager Duty alert.

Remediation

Escalated the DNS resolution alert through Pager Duty
Confirmed the provider's outage by repeatedly calling its hotline and watching its status page
Emailed the provider's support, which replied at 08:20 that it was looking into the issue
Tested DNS resolution and confirmed that DNS had been restarted
Posted a public update on Slack and Twitter announcing the recovery

Action items

Configure a second DNS provider alongside the first
Build a public status page for real-time outage information
Review and improve internal escalation procedures for third-party outages

Lessons

Relying on a single DNS provider made it a single point of failure that brought down every connected service, and the provider's outage also slowed communication with it, which lengthened the downtime. Octez Connect concluded that it needs redundant DNS providers.

Sources

https://octez-connect.tezos.com/post-mortem-4 · blog

Related cases

IL-0260 zoom.us stopped resolving for about two hours after GoDaddy Registry placed a server block on the domainZoom · 2025
same root cause · shared Authoritative DNS servers and records
IL-0162 LKE users could not reach their cluster control planes or the LKE Dashboard because an earlier database update made the internal LKE DNS service hangLinode · 2025
same root cause · shared Authoritative DNS servers and records

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Authoritative DNS servers and records among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0269
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0269/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.